Diving deeper into Rust’s Option & Result: Solving the Nesting Problem
In the last article, we talked about Rust’s Option and Result type classes. We showed how idiomatically best to act on their contents with pattern matching as well as using the .map operator to transform their contents.
In this second article of our three-part series, we’re going to introduce a fundamental Functional Programming concept that, on its surface, may not seem that groundbreaking. But you’ll soon see how powerful it is and how crucial it is at untangling code that could otherwise get really unwieldy.
We previously were working with an example where we were reading a temperature sensor’s measurement and, if we were successful at getting a reading, sending the results over the network to some “edge server” that collects the sensor telemetry. We started with the sensor reading being stored as an Option<Reading> and modified the example to use the more typical Result<Reading,&str> where we could represent a simple error message with a string description.
The Option variation of our example looked like this:
fn take_temp(sensor: &TempSensor) -> bool {
match sensor.get_reading() {
Some(measurement) => send_telemetry(measurement),
None => false,
}
}And the Result variation looked like this:
fn take_temp(sensor: &TempSensor) -> Result<(),&str> {
match sensor.get_reading() {
Ok(measurement) => if send_telemetry(measurement) {
Ok(())
} else {
Err("Could not deliver reading telemetry")
},
Err(e) => Err(e),
}
}We’re going to add one more layer to our puzzle by considering whether we were able to locate and configure the temperature sensor on our I²C bus. For the simple Option variation, let’s see what happens when we just change our sensor: &TempSensor parameter type to &Option<TempSensor>.
fn take_temp(sensor: &Option<TempSensor>) -> bool {
match sensor {
Some(s) => {
match (*s).get_reading() {
Some(measurement) => {
send_telemetry(measurement)
},
None => false,
}
},
None => false,
}
}This got complex really quickly! I was even surprised that I had to explicitly dereference the sensor in the first pattern match. Let’s see what happens when we try to graduate this to Result-land…
fn take_temp(sensor: &Result<TempSensor,&'static str>) -> Result<(),&'static str> {
match sensor {
Ok(s) => match (*s).get_reading() {
Ok(measurement) => {
let success = send_telemetry(measurement);
if success {
Ok(())
} else {
Err("Could not deliver telemetry" )
}
}
Err(e) => Err(e),
},
Err(e) => Err(*e)
}
}Quick side-note: let’s clean up the de-referencing!
On both these Option and Result examples, it’s been a pain dealing with de-referencing the wrapped Ok() and Err() values. Both type classes have a nice .as_ref() method that will make life much more pleasant. If you have an object with type &Option<T> , calling .as_ref() will transform it conveniently to Option<&T>, and similarly it can convert a &Result<T,E> into a Result<&T,&E>. We will call it on the outer-most pattern matched value:
fn take_temp(sensor: &Result<TempSensor,&'static str>) -> Result<(),&'static str> {
match sensor.as_ref() {
Ok(s) => match s.get_reading() {
Ok(measurement) => {
let success = send_telemetry(measurement);
if success {
Ok(())
} else {
Err("Could not deliver telemetry" )
}
}
Err(e) => Err(e),
},
Err(e) => Err(*e)
}
}It’s not much of a difference, but I find it more readable, and I personally think it’s easier to think of an Option<&T> than an &Option<T> for some reason. Note that I still had to manually de-reference the outermost error passing, because my IDE hinting was showing the value as Err(e: &&str) => Err(e). (If you come to Rust from C/C++, this is probably easy to understand. If you come from a garbage-collecting language like Java or Scala, these ampersands and asterisks can get really confusing!)
By the way, one more side note for beginners: it may seem confusing that I’m explicitly passing along &'static str values. Since this example comes from an Embedded Rust example, I’m assuming we’re living in “no-std” land and don’t have the luxury of the whole “alloc” framework where dynamically allocated strings live. If you wanted to take this into a Rust world with the std library, you might be passing real string objects and using String::from("Could not deliver telemetry") instead.
Happy & Unhappy Paths
It’s worthwhile to think about the problem as a series of “happy” and “unhappy” paths. The happy paths are where we’re writing code that is thinking about our actual business logic. It’s the stuff where we’re thinking about solving our problem at hand or doing what our application was meant to do.
The unhappy paths represent all of those “what if” scenarios that we have to consider and handle, without which we’re likely to have our application crash (panic) or behave in an otherwise buggy manner. In the above code, only Unhappy Path 3 represents any meaningful logic where we are converting the false return value of the send_telemetry() method a more meaningful error message, i.e. converting a boolean response into the Result<(),&str> type the function needs to return. The other paths are essential but basically meaningless boilerplate. What you’ll see at the end of this whole journey will be that we can write code that focuses on the happy paths while being provably robust with regard to all those unhappy paths.
Map to the Rescue?
In the last article, I presented an option of using the .map() function to apply business logic to the “happy path”. For the simple “Option” flavor of our application, we had the following:
fn get_reading3(sensor: &TempSensor) -> bool {
sensor.get_reading().map(send_telemetry).is_some_and(identity)
}Here you saw that out logic was able to focus on the Happy Path with calling the .get_reading and send_telemetry functions. The potential None value from a potential reading failure automatically merged with the potential false value from a potential telemetry failure. Can we use .map to help us now? Let’s trying writing this with our additional sensor allocation failure case:
fn take_temp(sensor: &Option<TempSensor>) -> bool {
sensor.as_ref()
.map(|s| s.get_reading() // returns Option<Reading>
.map(send_telemetry) // now Option<Option<bool>>
)
}Unfortunately, we were wanting a boolean return value and instead we’re getting an Option<Option<bool>>. Let’s try the same thing in the Result paradigm:
fn take_temp(sensor: &Result<TempSensor,&'static str>) -> Result<(),&'static str> {
sensor.as_ref()
.map(|s| s.get_reading()
.map(send_telemetry)
.map(|success| if success { Ok(()) } else { Err("Could not send telemetry")})
)
}If I try to compile this, the compiler tells me we’re returning a Result<Result<Result<(),&str>,&str,&&_> instead of the desired Result<(),&'static str> type. It would be nice if the .map function could be used here, but we’re getting a nesting challenge.
Thinking about Type Classes
Let’s pause and look at this conceptually. I’ve used the term “type class” here and there without giving it a proper definition, and I distinctly remember feeling confused in my early Scala days when people would use the term.
As we see with both Option<T> and Result<T,E>, we are completely free to put any sort of value or object inside. I like to think of it as just a box or container that provides some special representation.
The .map operation is how we apply a function to the contents of the boxes. However, there’s a problem that we’ve just found ourselves running into where you try to apply .map to a function that returns a value in it’s own “box”. We get a nesting problem that starts to resemble the classic Russian Dolls.
There is a special version of map that deals with this nesting problem. In other languages (notably Scala) this would be called “flatMap”. For whatever annoying reason, the Rust language community decided to use the term .and_then() . In fact, in my IDE when I start typing the function’s name, the quick tool-tip says .and_then() (alias flatmap).
Here’s what flat-map (i.e. .and_then()) does: if you have a type class like Option or Result that acts as a “box” for some value of type T and you have a function that takes a value of type T and creates potentially another box—where this box within a box would create the nesting problem we’ve been seeing in the above examples—it flattens things out so you just get a single-layer result.
So if I have a value foo: Option<T> and a function fn bar(T) -> Option<S>, then I can call foo.and_then(bar) and the result will be of type Option<S>. If either the original foo or the result of the bar() function call would result in a value of None, the whole thing flattens to None. If both foo and the function bar() result in a Some() result, then the whole thing results to a Some(S) result rather than the Some(Some(S)) result that .map would have resulted in.
And we can see this works nicely:
fn take_temp(sensor: &Option<TempSensor>) -> bool {
sensor.as_ref()
.and_then(|s| s.get_reading().map(send_telemetry))
.is_some_and(identity)
}By shifting from .map() into .and_then(), we were able to work with an Option<Reading> instead of an Option<Option<Reading>>. I admit there’s still some weirdness around the last line. This comes from the fact that I decided to create this example where the send_telemetry() function was using a boolean value to represent success or failure, so we are needing to use the .is_some_and(identity) to flatten the nested happy paths (some reading exists + telemetry delivery was successful) into a single boolean. If send_telemetry() had returned some sort of Option instead, we could have called .and_then() again.
And in fact, this is how things look when we elevate this example to the world of Result:
fn take_temp(sensor: &Result<TempSensor,&'static str>) -> Result<(),&'static str> {
sensor.as_ref().map_err(|e| *e)
.and_then(|s| s.get_reading())
.map(send_telemetry)
.and_then(|s| if s { Ok(()) } else { Err("Could not send telemetry") })
}There’s still a little complexity in this example—notably the little .map_err call I had to put in there. The .as_ref() call transforms a &Result<T,E> into a Result<&T,&E> which means that my &Result<TempSensor,&str> got converted into a funky Result<&TempSensor,&&str> and I had to de-reference the error to get it back into a proper &str reference. Note that .map_err behaves just like we’ve seen with .map, applying a function only to the error branch while leaving the success branch (the happy path) alone. You’ll find this a very useful tool in cases where you need to apply some transformations to an error type in order to conform your logic to an different expected return type.
Notice also the fact that we have both .map and .and_then intermingled here in this example. This is very typical where you have a series of transformation steps, some of which “box” their results into Option or Result and others which just do straight transformations with no boxing. (That is, transformations where there’s no expected edge-cases that we have to handle with None or Err.)
One quick last note that you need to remember: the .and_then (flat map) operation only works with boxes of the same type. You can’t intermingle functions that create Option values with ones that create Result values. If you do, you’ll need to “convert” one to the other—for example converting Some(value) into Ok(value) and converting None into Err(some error message). By the way, Option has methods .ok_or and .ok_or_else to support this very conversion, while Result has the method .ok() to convert the other way into an Option.
Quick Detour: Collections
There’s another interesting quirk with the way Rust handles map and “flat map”. In Scala, there are more examples of these magic type class boxes. In addition to Option and Result (which is called “Either” in Scala), we can do the same things with all of the collection classes.
It turns out that Rust has similar (albeit quirky) capabilities. Let’s say I have a vector my_list: Vec<i32> of integers and I want to apply a transformation such as squaring to each value. I can do that like this:
let my_list = vec!(1, 2, 3, 4, 5);
let squares: Vec<i32> = my_list.into_iter().map(|x| x * x).collect();If I print out the result of squares, I’ll get [1, 4, 9, 16, 15], as one would expect. So we see that anything that can be transformed into an iterator gets the useful .map operation.
Now, what if I have a transformation that doesn’t just transform an integer into another integer (e.g. squaring) but it transforms an integer into another vector? For a simple example, let’s say we want to just repeat each value twice. We could write that as a lambda function |x| vec!(x,x).
If Rust’s iterators have .map(fn) just like Option and Result, do they also have .and_then? The answer is a little quirky: this time we have the properly named .flat_map(fn).
let doubles: Vec<i32> = squares.into_iter().flat_map(|x| vec!(x,x)).collect();If I print out doubles, I’ll see [1, 1, 4, 4, 9, 9, 16, 16, 25, 25]! So instead of dealing with a vector of vectors—that same classic nesting problem—it all gets flattened out into a single long vector.
Treating Option and Result as Collections
There’s still more interesting stuff down this particular rabbit hole. It turns out that Option and Result can both be treated as a sort of nested collection. If you wanted, you could call .into_iter on one, and the body of your iterator would only run if the Option value was Some(T) or the Result value was an Ok(T).
Using the previous simple example, if (for some reason) I wanted to filter out odd numbers from my doubles vector, I could use create a function that converted even numbers into Some(x) and odd numbers into None. Rather than using .map to get a Vec<Option<i32>> , I could call .flat_map to compress the vector into one with only the even values:
let evens: Vec<i32> = doubles
.into_iter()
.flat_map(|x| if x % 2 == 0 { Some(x) } else { None })
.collect();If you print out the evens value, you’ll get [4, 4, 16, 16]. The example is obviously a little contrived—it would be better to use .into_iter().filter(predicate) instead. But it’s not unusual in practice to find yourself with a collection of Option values where you’re only interested in working with the non-None elements.
A better example is if you have a collection of sensors represented by Result<Sensor,E> and you want to reduce it to a collection of only those that were configured properly in an Ok<Sensor> state. We can represent that with the following simple code fragment:
let sensor1: Result<&str,&str> = Ok("MCP9808");
let sensor2: Result<&str,&str> = Ok("TMP36");
let sensor3: Result<&str,&str> = Err("Bad Sensor");
let good_sensors: Vec<&str> = [sensor1,sensor2,sensor3]
.into_iter().flatten().collect();
println!("{:?}", good_sensors);If you run it, you’ll see the expected ["MCP9808","TMP36"]. Two things to note here:
- We already had a series of nested collections, so rather than doing some transformation with
.flat_map, we’re just calling.flattento de-nest things. - You’ve seen me provide the explicit
Vec<T>types in the last few examples. That’s because.collectis actually too general for the compiler to know exactly what type to collect the values into, so it needs us to provide the target type. We can either do it in the variable definition, or we can use what’s called the turbofish like this:
let good_sensors = [sensor1,sensor2,sensor3]
.into_iter().flatten().collect::<Vec<_>>();As a final snippet of code example, here’s a very realistic scenario where we might want to define a function that takes a slice of many Result<TempSensor,&str> sensors, calls .get_reading on each one, and finally just returns a vector of the successful readings.
fn take_readings(sensors: &[Result<TempSensor, &'static str>]) -> Vec<Reading> {
sensors
.iter()
.flatten()
.flat_map(|s| s.get_reading())
.collect()
}By the way, you should notice that this time I used .iter() instead of .into_iter(). The difference between these is that .into_iter() actually consumes the original collection’s items so the borrow checker won’t let you use them later on. In this particular use case, I know that calling .get_reading() on my sensor is just going to create a simple Reading struct. In fact, I’m likely to want to take multiple readings in the future, so .into_iter() would have gotten me in trouble!
That’s it for this article! We’ve gone over a lot of important concepts already, but we still haven’t gotten to the bottom of this rabbit hole. By now, you’re probably feeling more comfortable with .map and either .and_then or .flat_map, but you’re probably not feeling all that excited about writing code that reads so weirdly. For an FP enthusiast like me, reading these chains of transformers is perfectly natural, but most people would find it too hard to understand.
Before you give up and go back to writing all those nested match statements, I want to show you one more cool thing that will provide the best of both worlds! I’m also going to shed the funky &'static str error types and examine the idiomatic approach that real Rust developers take when it comes to proper error types. (It’s something that I found really weird as someone used to Java/Scala.) Read all about it in the final article of this mini-series!