Introducing Scopes in ZIO 2.0

Guidance for the sudden switch from ZManaged (updated for 2.0-RC6)

Say Goodbye to ZManaged and Hello to Scope!

Like so many people, I have been eagerly awaiting the release of ZIO 2.0. As an interested newcomer to ZIO, I’ve been waiting for the right moment to really “dip my toes” into giving ZIO a series effort. My current projects haven’t needed the mission-critical stability that ZIO offers, but I have been working on problems that could benefit from taking Streaming approaches, and there are some occasional problems that my developers would turn to the Akka actor model for, and I’ve wanted to try tackling these things with the ZIO paradigm.

When I started watching presentations about the upcoming ZIO 2.0, I felt like “This is it! This is the moment!” Honestly, the biggest deal to me was the new ZLayer design with the new “Service 2.0 Pattern” as seen in this excellent presentation by Kit Langton from ZIO World 2021. (I’ll be honest, I was also excited about the new approach to ZStream, but a lot of those changes were under-the-hood, and I’ve already written some good applications with ZIO 1.0 Streams.)

When ZIO 2.0.0-RC1 was released, and they announced that there would a further simplification by removing the Has keyword, I was elated. I started writing my day-to-day tasks using ZIO, intentionally finding ways to use ZManaged and ZLayer for my resources and dependencies—even when it was a bit of overkill. I waited eagerly for the day when 2.0 would finally be out.

Then about two weeks ago I saw issue #31 of ZIO News drop in my email and my jaw dropped!

The big reveal at ZIO World 2022 is a HUGE simplification to resource management in ZIO 2.0 that brings new levels of simplicity, power, & performance.

Adam Fraser and John De Goes discovered ZIO Environment is now powerful enough to provide resource-safety by itself!

I had actually run into this the day before when I had bumped my ZIO version to the new RC3 and my project stopped compiling because my ZManaged instances would no longer resolve, but that had happened at the end of the day, and I hadn’t had time to investigate.

Author’s Update: the ZIO World 2022 presentation with this “big reveal” just got posted to YouTube. When you have 45 minutes to spare, this will give you the real in-depth explanation. It’s definitely worth watching.

Pondering the implications

This is definitely a dramatic 11th hour change to ZIO, and I know there are a lot of people like me who have been eagerly awaiting 2.0 for a year. And I can only imagine all the people working on ZIO ecosystem projects who have been rushing to get their 2.0-compatible releases ready. I’m sure there was some pain removing all the Has references from code and documentation for RC1, but factoring out all the ZManaged!?

There was a backward-compatible “patch” made available: ZManaged was not completely deleted from the ZIO library, but moved to a separate dependency, so if you choose to include a new zio-managed dependency into your project, your existing code will be able to run as is.

But still, it’s a pretty amazing change if you consider that the ZIO website’s documentation still has pages of ZManaged-focused content. That’s why I decided to write this article: to provide a stop-gap so people Googling “ZIO Scope” will be able to grok these changes without panicking.

(Re)Working a Simple Example

I’ve created a simple sample project here in github in case you want to play around with this yourself. The project has two modules: “old” with ZIO 2.0.0-RC2 as its dependency and “new” which uses ZIO 2.0.0-RC4.

Let’s build a trivial example of ZManaged in ZIO 2.0.0-RC2 and then migrate it to use Scope in RC4. The simplest kind of resource that you would want to use an acquire/release pattern on is something that reads from the filesystem. Our application is simply going to count the total number of lines in the file.

A convenient way to do this is to use the typical Source.fromFile method in the Scala library which returns a BufferedSource object from which we can call _.getLines().size to get the line count. So we can start with:

val fileReader =
  ZManaged.acquireReleaseAttemptWith(Source.fromFile("build.sbt"))
                                    (_.close())

If we use our IDE to inspect the type and/or add the type annotation, we’ll see that fileReader has the type ZManaged[Any, Throwable, BufferedSource]. If we want to use this managed fileReader directly, we could just write something like this:

val getLineCount: ZIO[Any, Throwable, Int] = fileReader.use { f =>
  Task.attemptBlocking(f.getLines().size())
}

Author’s note: I had originally written this article with the simpler Task.attempt() method calls, because I didn’t want to distract the beginner with the more complicated call. But I realize this isn’t a good idea: people new to ZIO should get into the habit of thinking about whether any ZIO effect is likely to be blocking. Interestingly, as I fixed this in my code, I found that the ZManaged objects lacked .attemptBlockingIO that you’ll see me using in the later section where I use Scopes.

The .use method provides our managed fileReader resource all the logic that needs to be run before it can be closed, which is very analogous to how you would use scala.util.Using in Scala 2.13+. The resulting value is a ZIO effect that will return the file line count while automatically handling the acquire/release logic.

By the way, for defining the managed fileReader I also like the alternative form:

val fileReader = ZManaged.fromAutoCloseable {
  ZIO.attemptBlocking(Source.fromFile("build.sbt"))
}

…which can be used for any resources that implement the Java AutoCloseable interface. (Which should be just about anything that closes its resources with _.close()!)

Managed Resources with Layers

The easier way to work with Managed resources that aren’t just going to be immediate one-off acquire/release implementations is to use Layers. So let’s create an alternative fileLayer object like this:

val fileLayer: ZLayer[Any, Throwable, Iterator[String] =
  fileReader.map(_.getLines()).toLayer

I’m fancying-up my layer a bit by returning a more actionable Iterator[String] instead of this weird BufferedSource. (I’d had to use BufferedSource because it’s the thing that exposed the .close() method that I needed to release the resource.) But the important thing is that you typically would create a ZLayer by simply calling .toLayer on the ZManaged object. What I get is a resource that I can pass to any ZIO objects, and it auto-magically comes with the guarantee that when it’s done, it will be properly released.

Now let me write a ZIO object that depends on finding an Iterator[String] in the environment:

val countLinesProgram: ZIO[Iterator[String], Throwable, Int] = for {
    data <- ZIO.service[Iterator[String]]
    size <- ZIO.attempt(data.size)
  } yield size

I can show both mechanisms working with the following ZIOrun method:

override def run: ZIO[ZEnv, Any, Any] = {
 for {
    data <- fileReader.use(f =>
              Task.attemptBlocking(f.getLines().size))
   data2 <- countLinesProgram.provideLayer(fileLayer)
       _ <- Console.printLine(s"$data, $data2")
 } yield data
}

Here I’m just using the first direct use of the ZManaged fileReader as well as my new countLinesProgram that I provide the new fileLayer to. If I run this, I’ll just get the simple output 26, 26.

Shifting to “Scope”

Starting with 2.0.0-RC3, the entire ZManaged class is gone, so I don’t have the ZManaged.acquireReleaseWith or ZManaged.fromAutoCloseable methods, nor can I call .toLayer on a ZManaged anymore. What do I do?

Well, it’s actually pretty trivial: the first two methods have now moved to the base ZIO object. So our previous fileReader example can be written as:

val fileReader = 
       ZIO.fromAutoCloseable(
         ZIO.attemptBlockingIO(Source.fromFile("build.sbt")))

Author’s Note: ZIO 2.0-RC6 introduced some clean-ups, getting rid of some redundant methods. I originally had Task.attemptBlockingIO in my example, but now the only idiomatic method is ZIO.attemptBlockingIO. It’s not difficult, and I agree with the idea of trimming 2.0 to be succinct, but it’s going to add to the number of things you’ll have to rework if you’re migrating ZIO 1.0 code.

There’s something interesting here, however: if we check the compiler, we will see that our fileReader has the type signature ZIO[Scope, IOException, BufferedSource]. But what is this Scope resource requirement?

Pretty simply: it just lets us know that our ZIO object is depending on an environment resource that will need to be managed (released) at some point. If I compose this fileReader with other ZIO effects, that composition will itself retain the Scope resource dependency. For example, a ZIO effect “read” like this:

val read = fileReader.flatMap(f => 
                       ZIO.attemptBlockingIO(f.getLines().size))

…will have a type signature of ZIO[Scope, IOException, Int]. So how do I make this Scope go away? Well, I can write an alternative form of a “read” effect that says “Once this effect is done running, I know the scoped dependencies can be released because I don’t need them anymore.” by doing this:

val read = ZIO.scoped { 
  fileReader.flatMap(f => ZIO.attemptBlockingIO(f.getLines().size))
}

Here, the new read effect gets the type signature ZIO[Any, IOException, Int] that we’re used to.

(Actually, we have another improvement over the ZManaged scenario, because the expected error channel is the more specific IOException rather that just Throwable, which we get from the convenient .attemptBlockingIO method that ZManaged had lacked.)

Scoped Layers

(Warning: I’m going to start with a counterexample. I actually did this while writing the article, and Adam Frasier was kind enough to point out what I did.) Recreating the fileLayer from the ZManaged example is pretty straightforward: I can just call .toLayer on my fileReader object:

val fileLayer: ZLayer[Scope, IOException, Iterator[String]] = 
  fileReader.map(_.getLines()).toLayer

Another Author’s note: the .toLayer method has also now been deprecated. In the next section I show why this was a mistake in the first place, but for completeness, I have to convert the above snippet to the following:

val fileLayer: ZLayer[Scope, IOException, Iterator[String]] =
  ZLayer.fromZIO(fileReader.map(_.getLines()))

Interestingly, the resulting ZLayer gets the Scope requirement, so if we provide this layer to our countLinesProgram, like this:

val countLinesProgram: ZIO[Iterator[String], IOException, Int] = for {
 data <- ZIO.service[Iterator[String]]
 size <- ZIO.attemptBlockingIO(data.size)
} yield size
val toRun: ZIO[Scope, IOException, Int] =
  countLinesProgram.provideLayer(fileLayer)

…we will find that toRun inherits the same Scope reference.

Important: the better way to construct the ZLayer!

When I defined my fileLayer object by calling .toLayer on my scoped fileReader object, I was just throwing a scoped ZIO object into a layer, thus the inherited Scope requirement.

The more idiomatic way to create a ZLayer out of my scoped object is to use ZLayer.scoped instead of .toLayer, like this:

val fileLayer: ZLayer[Any, IOException, Iterator[String]] = 
  ZLayer.scoped(fileReader.map(_.getLines())

We can see that the Scope resource has gone away, and we’re back to having our R time revert back to Any.

We will find that the new fileLayer has a type ZLayer[Any, Throwable, Iterator[String]], so similarly, ZLayer has a better understanding of how to manage its own resources.

(Note: you can see here the way to “map” a layer’s target A to a different type B is to call .project(f: A => B). A useful tidbit to remember!)

Running the program

Recall that the final ZManaged run program was written as follows:

override def run: ZIO[Any, Throwable, Int] = {
 for {
   data <- fileReader.use(f => Task.attempt(f.getLines().size))
   data2 <- countLinesProgram.provideLayer(fileLayer)
   _ <- Console.printLine(s"$data, $data2")
 } yield data
}

But if fileReader and my first (anti-patterned) fileLayer both had Scope dependencies, the run program would also inherit the Scope environment type. There are two way to get run back to a ZIO[Any, Throwable, Int].

First, we can wrap ZIO.scoped() around each step that uses a scoped resource, like this:

override def run: ZIO[Any, IOException, Int] = {
  for {
    data  <- ZIO.scoped(fileReader.flatMap(f =>
              Task.attemptBlockingIO(f.getLines().size)) )  
    data2 <- ZIO.scoped(countLinesProgram.provideLayer(fileLayer))
        _ <- Console.printLine(s"$data, $data2")
  } yield data
}

…or we could wrap everything in ZIO.scoped like this:

override def run: ZIO[Any, IOException, Int] = {
  ZIO.scoped {
    for {
      data  <- fileReader.flatMap(f =>
               Task.attemptBlockingIO(f.getLines().size))
      data2 <- countLinesProgram.provideLayer(fileLayer)
          _ <- Console.printLine(s"$data, $data2")
    } yield data
  }
}

When reviewing this article, Adam Frasier gave this piece of advice comparing these two options—having two smaller scopes versus one big scope:

[It’s about] how early you want to close the resources. All other things being equal earlier is better, but sometimes we need them open longer to do something with them.

Finally, it is now possible to just let all the Scope dependencies percolate up to the top-level ZIOAppDefault.run method, so I can actually omit the ZIO.scoped call to get this simpler form:

override def run: ZIO[Scope, IOException, Int] = {
  for {
    data  <- fileReader.flatMap(f => 
              Task.attemptBlockingIO(f.getLines().size))
    data2 <- countLinesProgram.provideLayer(fileLayer)
          _ <- Console.printLine(s"$data, $data2")
  } yield data
}

If you want, you can just let the top-level run method manage the Scope resolution. For a simple application, there’s nothing wrong with that, but remember Adam’s overall advice: it’s prudent to resolve your scopes sooner rather than later.

In conclusion…

I’ll let you judge for yourself. These eleventh-hour changes require me to re-wrap my brain around the ZIO idiosyncrasies, and I’m sure it’ll cause people some complications as they figure out how to migrate, but reflecting on the following notes from the ZIO News announcement:

First, Managed is Yet Another Thing to teach to developers. Many new ZIO developers try to avoid using Managed, because they are not sure exactly what it’s for or how it differs from ZIO. Those who use it, sometimes wonder when to use ZIO versus Managed.

Second, all the methods on ZIO must be manually and painstakingly re-implemented on Managed, but with much more complex implementations due to the complications of handling resource safety in the presence of concurrency. In practice, ZIO still has more methods than Managed.

Third, Managed is a layer over ZIO, and is slower than ZIO itself, because of the additional complications & wrapping. Unlike other approaches, ZIO uses an executable encoding, so it’s able to avoid double-interpretation, but it still has measurable overhead over ZIO.

Like so many other aspects of ZIO 2.0, it looks like the developers have had another one of those epiphanies that allows the codebase to be smaller, faster, more succinct, and more elegant. And after all, that’s the kind of thing a good Scala programmer should always strive for, so I think this is going to be well worth the wait.

One last shameless plug: last week I just published a blog article with my overall assessment of ZIO 2.0 and an attempt to describe its value to the layperson. If you’re interested, head over here for a read.