Diving deeper into Scala’s Option, Try & Either
This is a continuation of my earlier post Scala for Beginners: How to really use Option. This topic is based on a recent presentation I gave to my local Austin Scala Enthusiasts Meetup. The video and slides for that can by found on my website.
When I was writing the first article, I was trying to create a very simple example with some hypothetical getEmail function to fetch an email address from a customer database and a sendEmail function that would send an email invitation to someone. I found that, despite my attempt to keep things simple, it became hard to articulate realistic implementations using pattern matching and .map.
I’m going to go back to the slightly more complicated example that I used in my presentation. We’re still focusing on the hypothetical use case where we want to send out email invitations. We’ll start with a Contact class that defines an integer contact ID, optionally provides an email address, and has a list of friends referenced by this same integer contact ID.
case class Contact(id: Int, name: String, phone: Option[String],
email: Option[String], friendIds: List[Int])We are going to assume there are some API functions available to fetch a contact record and another function that sends an invitation email and returns whether that email delivery was successful. The lookup function will look like this:
def dbLookup(id: Int): Try[Contact]It is designed to wrap the database lookup process and return a failure (wrapping an exception) if either the contact ID can’t be found in the database or if there’s any sort of network IO problem.
The email delivery function will look a little different (let’s say it was written by someone else with a different style):
def sendEmail(email: String): BooleanThe return value is simply a boolean representing a successful email delivery.
What we want to do is to write an inviteFriends method on our Contact case class that uses these two utility functions to lookup all of the friends so we can get their Contact records and send emails to those records that have email addresses. We’ve got a variety of mechanisms for representing “issues” with this supposedly simple process.
- The database lookup may not be successful (id not present or other technical problems) handled by Try.
- Once we look up a contact, the record may not have an email address, handled by Option.
- The email delivery system may fail, and it only reports this by returning false.
The Imperative Approach
A hidden agenda in this article and its predecessor is to give a feel for Functional Programming, which is a major aspect of the Scala language. Granted, you can write Scala code much like you would Java, with vars and loops and leaning on imperative O-O patterns, but the language really sings if you can wrap your head around FP, and I would claim that the use of things like Option is heavily rooted in FP.
So let’s look at how we would write this code using a traditional Imperative approach. (BTW, this is a method on the Contacts case class, so friendIds refers to the that List[Int] in its fields. If you want to see the fuller code for this example, see this GitHub page.
def inviteFriends: List[Int] = {
var successes: List[Int] = List.empty
for (id <- friendIds) {
val record = dbLookup(id)
if (record.isSuccess) {
val maybeEmail = record.get.email
if (maybeEmail.isDefined) {
val send = sendEmail(maybeEmail.get)
if (send) successes = id :: successes
}
}
}
successes
}This is actually pretty straightforward. (Well, if you’re a beginner to Scala, the id :: successes may look weird, but that’s how to prepend an element onto a list.) We’re looping through the friends, and if we can lookup the friend record and if the record has an email address and if we’re successful at sending the invite, then we’ll add the id to the list record of successful deliveries. We are using that typical .get method to extract elements from the Option or a Try, but we are testing beforehand that this won’t throw an exception.
Now we’re going to try to stick to Functional Programming principles, which simply means (in my view of FP):
- No mutable state—i.e. no
vars. - No looping (BTW: if you really want to do this broadly, you’ll need to get comfortable with recursion, and for large collections, you’ll have to understand when and how to do tail recursion.)
We’re going to start with the friendIds list and transform it into a list of successful deliveries.
def inviteFriends: List[Int] = {
val successes = friendIds.map { id =>
dbLookup(id) match {
case Failure(exception) => None
case Success(friend) => {
friend.email match {
case Some(email) => if (sendEmail(email)) Some(id) else None
case None => None
}
}
}
}
successes.filter(_.isDefined).map(_.get)
}We’re using the pattern matching to cleanly articulate the Somes and Nones of the Option and the Successes and Failures of Try, and we’re using it to unwrap the data from the Somes and Successes in a way that avoids the test-and-call-unsafe-get approach that could introduce bugs if we don’t get it exactly right.
The only slight hitch is that the successes list ends up having the type signature of List[Option[Int]] so I’ve still got to first shrink the list into one that only has known values (via .filter(_.isDefined)) and then converting the Some[Int] values into straight Int values via that unsafe .get.
Happy and Unhappy Paths
By the way, it’s worth thinking about the problem as a set of “happy” and “unhappy” paths.
When we think of performing a task (i.e. send an invite to all your listed friends) we first think of writing code in terms of the happy paths, but we’ve learned the hard way that we really have to spend a lot of time and effort thinking about and handling the unhappy paths. What you’ll see at the end of this whole journey will be that we can write code that focuses on the happy paths while being provably robust with regard to all those unhappy paths. Even with our nice-looking imperative programming approach, we saw a lot of code dedicated to those unhappy paths!
Okay, I’m going to be honest: the pattern matching approach is somewhat readable, but if I’m going to convince an experienced programmer to abandon imperative programming, I haven’t done my job yet. We’re not done with our journey.
Map to the Rescue
I don’t think my previous article did justice in showing how Map helps us solve our problems.
As a quick reminder: we can use .map in Option, Try or Either to perform some task on their contents. If I have a Contact variable friend and that variable has an email address, I can apply the sendEmail function to that address, effectively transforming the address into the true/false state indicating success. If friend.email has no contents, then the map operation doesn’t do anything.
As you can see, the pattern matching code gets converted into something much cleaner and more succinct. We’re letting Option make sure the happy path and unhappy paths are taken care of, but in terms of code you’re really just looking at the happy path.
“Wow! Cool!” you say, “I’m ready to go!” Armed with this, you’re going to be ready to rewrite the example in a cleaner fashion that just focuses on the happy path! Well, let me just say we’ve got a little bit more of a journey to go on, but let’s try it and see how things look. If we try to put .map to work to write for the happy paths, let’s see what we get:
def inviteFriends: List[Int] = {
val successes = friendIds.map { id =>
dbLookup(id).map { contact =>
contact.email.map { address =>
sendEmail(address)
}
}
}
??? // successes is now of type List[Try[Option[Boolean]]]
}If we write this code in our IDE (you can find the sample code here if you want to try it in your own environment) our compiler will tell us that success is now a List[Try[Option[Boolean]]] where we wanted a list of ints reflecting the successful ids. First, there’s a relatively easy fix so our list holds the friendIds rather than just booleans:
def inviteFriends: List[Int] = {
val successes = friendIds.map { id =>
dbLookup(id).map { contact =>
contact.email.map { address =>
if (sendEmail(address)) Some(id) else None
}
}
}
??? // successes is now of type List[Try[Option[Option[Int]]]]
}But now our nesting structure is even one layer deeper? Now, I could hack a solution together to “test and unsafe-get” to return my desired List[Int]:
def inviteFriends: List[Int] = {
val successes = friendIds.map { id =>
dbLookup(id).map { contact =>
contact.email.map { address =>
if (sendEmail(address)) Some(id) else None
}
}
}
success.filter(s => s.isSuccess && s.get.isDefined &&
s.get.get.isDefined)
.map(_.get.get.get)
}So we’ve tested that the outermost Try was a success and both of the nested Options are defined. (Well, we’ve filtered down our list so that only those are left.) And knowing that we can call .get three times in a row, we can get the inner value.
Boxes within Boxes (“It’s turtles all the way down!”)
Think of Option or Try or Either all as boxes. (You’ve heard me quietly refer to them as “typeclasses”, which is sort of the same thing.)
When we call .map, we are essentially taking a function and running it on the insides of the box.
The problem that we have is that the functions that we are executing provide boxes of their own! dbLookup(id) creates a Try box, contact.email is an Option box, and we just tested if sendEmail succeeded, and if it did, we created another Option box around the id!
FlatMap to the Rescue!
Because this situation is so common, .map has its own special sibling called .flatMap designed to solve this problem.
.FlatMap expects a function that creates a boxed value and “flattens” things so you only have one layer instead of two.
Let’s look at a simple example. Let’s say we have a case class Person that has an optional age, and we’ll make a simple database (a map) that map some IDs (1 and 2) to two people (Nitin and Rosemary) and where only one person has an age available.
Calling .get on this db Map returns an Option[Person] so if you say db.get(3) you’ll get None rather than an error. (For an unsafe version that doesn’t box the value, you would write db(3) and it would throw a java.util.NoSuchElementException. So don’t think that every .get method is unsafe. In the case of Map, it’s the safe one!)
So if we want to get an age based on an id, we would write it like db.get(id).map(_.age). Let’s test it on ID value of 1, 2 and 3:
Okay, for ID 1 (Nitin) we have an age. For ID 2 (Rosemary) we have a person, but no age, so Some(None) makes sense, but we would rather have it just be None. For ID 3, we have a simple None. If we just replace .map with .flatMap, we’ll get what we want:
Our nested boxes, went away and we now have a function that takes an ID and returns an Option[Int] for the age as we would want.
Warning: Only use flatMap for the same kinds of boxes!
The one thing you have to be careful about is to make sure you’re working with the same kind of boxes. Option has defined a flatMap function that knows how to deal with functions that create new Options, but we can’t expect it to know how to deal with other boxes, like Try or Either… or even ones we haven’t talked about here, like Future!
In our primary example, we have a mix of Try and Options, so we have to figure out how to deal with that. For some typeclasses (boxes) there are convenient converters to handle the transformation for you. As an example, Try happens to have a convenient .toOption method that will transform it into an Option. Of course, in doing that, you will lose the exception wrapped in a Failure result, since Failure will get converted into None, but that’s okay with what we’re trying to do.
Almost Done!
Let’s try to put all this together now. We know we’ll use .toOption to convert the Try box into an Option so we can flatMap everything together.
def inviteFriends: List[Int] = {
val successes = friendIds
.flatMap(id => dbLookup(id).toOption
.flatMap(contact => contact.email)
.filter(sendEmail).map(_ => id))
successes
}By the way, you might not have noticed, but it actually doesn’t look like I kept my boxes all the same because I’m first calling .flatMap on a List, but the other boxes are Options!! What’s going on here?
Well, two things. First, List (and any Scala class implementing Traversable) is actually a typeclass, meaning it’s a box of it’s own. But instead of having List[T] hold a single T value, it holds any number of T values—even zero!
And if you think about it, Lists’s .map method does the same thing as the other boxes: it applies a function to the contents of the box. For Lists, .flatMap is used when your function could create other lists!
So the second thing going on here is that Option[T] also extends Traversable so it essentially can be treated as a List with a length of exactly zero or one. I recently even wrote a blog post about this interesting fact when I came to discover it.
So now we have this really short and succinct way to write our inviteFriends method, and I can even shorten it into a one-liner:
A Scala wizard may look at that and think it’s nice a clean and short. But is this really easier to understand than that original imperative code I showed in the very beginning, with it’s looping construct? I don’t think I deserve to be let off the hook quite yet. In the third and final entry in this series, I’m going to introduce for-comprehensions and even explain my evil genius agenda underlying this whole journey I just took you on.