Bottom of the Rabbit Hole: for-comprehensions and monads

This is the third in a series of articles intended for Scala Beginners to understand how to write clean code using Option and Try and Either. Part one introduced these typeclasses and demonstrated how to avoid the awkward “test and unsafe-get” approach by using pattern matching of “map”. Part two introduced a slightly more involved example, and showed the problems encountered by nested map transformations that flatMap was designed to fix.

We were working on an example where we started with a case class Contact:

case class Contact(id: Int, name: String, phone: Option[String], 
                   email: Option[String], friendIds: List[Int])

And we had two utility functions: dbLookup that would take a contact ID and (if it’s found and assuming no system error happens) returns a Try[Contact]

def dbLookup(id: Int): Try[Contact] = ???

And finally, there’s a sendEmail function that sends an invitation email and simply returns a boolean to indicate if the delivery was successful.

The objective was to write an inviteFriends method that would send email invitations to a contact’s friends, returning a List[Int] of the contact IDs of people to whom the invitations were successfully delivered.

We started with a typical imperative-style approach that reads pretty straightforward:

def inviteFriends: List[Int] = {

    var successes: List[Int] = List.empty

    for (id <- friendIds) {
      val record = dbLookup(id)
      if (record.isSuccess) {
        val maybeEmail = record.get.email
        if (maybeEmail.isDefined) {
          val send = sendEmail(maybeEmail.get)
          if (send) successes = id :: successes
        }
      }
    }

    successes
}

…and managed to rewrite it into a succinct “one-liner” using Functional Programming and leveraging flatMap:

def inviteFriends: List[Int] =
   friendIds
     .flatMap(id => dbLookup(id).toOption
       .flatMap(contact => contact.email)
       .filter(sendEmail).map(_ => id))

But the aim of this series is not only to teach a classic imperative-style programmer how to do things “the Scala way” (e.g. no vars, no loops) but to also convince that programmer that the Scala way doesn’t have to be too terse and hard to understand.

Introducing for-comprehensions

Scala has an interesting thing called a for-comprehension. What’s interesting is that it’s one of these things we call “syntactic sugar”, which means it’s not actually a new language function, but rather a convenient way of writing things that gets rewritten behind the scenes behind the compiler.

In the imperative example, we had a loop that started with

for (id <- friendIds) {
  ???
}

And anyone with a familiarity with most modern languages will recognize this as a way of iterating through elements in a collection. Our for-comprehension looks almost exactly the same way. Here’s actually the way we’re going to rewrite our FP function:

def inviteFriends: List[Int] = {
  
  val successes = for {
    id <- friendIds
    contact <- dbLookup(id).toOption
    emailAddr <- contact.email
    if sendEmail(email)
  } yield(id)
  
  successes
}

…or more succinctly:

def inviteFriends: List[Int] = {
  for {
    id <- friendIds
    contact <- dbLookup(id).toOption
    emailAddr <- contact.email
    if sendEmail(emailAddr)
  } yield(id)
}

This should be just about as easy as the imperative style to understand:

• We iterate through the friendIds, calling each one id.
• We call dbLookup on the id, and convert its resulting Try into an Option, assigning it’s wrapped value to something named contact. (In other words, if the dbLookup doesn’t throw an error, its returned Contact object will be assigned to contact.
• If the optionalcontact.email field has a value, it will be assigned to emailAddr.
• And if sendEmail returns true…
• We’ll yield the id from the top of our loop logic.

What is actually happening behind the scenes is that each of these item <- box lines is really being rewritten behind-the-scenes into box.flatMap(item => ???). The if sendEmail(emailAddr) line is getting rewritten into a .filter method to filter out unwanted items (i.e. pare down the list).

There’s just one last bit in for-comprehensions that I’m not showing in this example: simple .map operations. But I can explain that right now pretty easily. Let’s say for example that one of the steps wasn’t creating a new “nested box” as I called them in the last article. Pretend for a moment that the email field of the Contact case class is actually just a String instead of an Option[String]. So there would be one layer fewer of unboxing we would have to do, and this:

val successes = friendIds
  .flatMap(id => dbLookup(id).toOption
    .flatMap(contact => contact.email)
    .filter(sendEmail).map(_ => id))

…looked more like this:

val successes = friendIds
  .flatMap(id => dbLookup(id).toOption
    .map(contact => contact.email)
    .filter(sendEmail).map(_ => id))

In that case, our for-comprehension would be written as:

def inviteFriends: List[Int] = {
  for {
    id <- friendIds
    contact <- dbLookup(id).toOption
    emailAddr = contact.email
    if sendEmail(emailAddr)
  } yield(id)
}

When I saw this, it looked weird, because it looked like I was doing a val assignment without the val keyword. But the reality is this is just the third type of grammar in the for-comprehension. So to summarize:

• id <- friendIds rewrites to a flatMap
• emailAddr = contact.email rewrites to a map
• if sendEmail(emailAddr) rewrites to a filter—note the lack of parenthesis around the condition
• And finally, for yield(???) the contents of the yield is a function is put into a final .map operation

So there I finally leave you with the cleaned, FP-compliant, readable code. The code focuses on all the “happy paths” which makes your code arguably cleaner than even the imperative approach, while elegantly handling all the unhappy-paths.

The only thing you have to do is make sure your boxes stay all the same general type: all Trys or all Eithers or all Options or all Lists (Traversables) with the caveat that Options can automatically behave like lists.

Unveiling my Evil Master Plan

You just learned about Monads.

For the last eight or nine years, I’ve heard advanced programming “wizards” talk about these things called Monads as though it were part of a precious secret thing. For the last two years or so I’ve been listened to talks and videos and read books, trying to understand Monads.

A couple months ago I finally got it, and honestly it wasn’t until I started preparing my talk for my local Austin Scala Enthusiasts Meetup (the video can be found at that link) that I really finally got it. As they say: you really don’t learn something until you try to teach it.

So here we go:

• A monad is a kind of typeclass that “boxes” any other type.
• It has .map that applies functions to the contents of the boxes.
• It has .flatMap for functions that create their own boxes in order to avoid nesting problems.
• Option, Try, Either and even Future can be considered good examples of monads. (For the latter, let’s not talk about referential transparency, please.)
• They isolate “pure functions” from problematic contexts like asynchronous operations, exception handling, missing data, etc.

If you spend any time programming in Scala, you’ll start using them without even realizing it.

Hope you enjoyed your journey. You’re now at the bottom of the rabbit hole. (Or should I dig into Monoids and Applicatives next? Mwah hah hah hah!)