Grokking ZIO

After a long evolutionary journey, the 2.0 release is a great time to jump in.

I have been learning and using Scala for about eight years now, ever since I first became involved with a Spark-based project. My own journey has been fun and enjoyable, and I was incredibly lucky to have a local Austin Meetup for Scala Enthusiasts where I leaned (scratching my head a long along the way) about things like implicits and higher-kinded types and all that monoid/monad stuff.

While learning those latter things and puzzling over whether to use Cats or Scalaz, I first came across some really enjoyable articles and talks by John De Goes. (I think I was trying to understand what applicatives were when I came across this video.) It was late 2017, and John was talking about putting some time into Scalaz 8, so I kept an eye on any mention of that work. In June 2018, he announced that he wanted his work to be able to work with in either the cats or scalaz worlds, so he moved it into a standalone project called ZIO.

About two years later, on August 3, 2020, the 1.0 release of ZIO was announced. I was still curious about “what it was all about”, but I was still focusing all my energy understanding monads and tagless finals, and I wasn’t really writing much code that needed to worry about working with concurrency or effects, so with limited bandwidth, I would read the main documentation pages on the website and not much beyond that. I had a feeling it was something pretty cool, and I noticed that there seemed to be a lot of energy and enthusiasm in a growing development community (always a great feeling!)

Really, I was hoping someone in my monthly Scala Meetup would do a presentation on ZIO so I could listen and ask questions and decide if it was worth more investment. I said as much when we had a discussion on future topics, and the head organizer of the group suggested I learn ZIO and give a presentation.

Let’s say that this article is the equivalent of that. If you haven’t been following ZIO closely but want a better sense of it, and especially if you’re wondering if the upcoming 2.0 release is the right time to dive in, I’m writing this for you. I’m not going to try to teach ZIO or walk through any concrete examples, but I’ll try to provide as many references for further study as I can.

What is ZIO? (In layman’s terms)

ZIO represents a philosophical approach to developing reliable, fault-tolerant, concurrent applications. It is meant to apply all the power of Functional Programming (and Object-oriented Programming!) that the Scala language has to offer. The ZIO approach is meant to lead to scalable, reliable, performant application development.

That may sound a bit vague and superlative. So let me dig in a little deeper.

Modeling Effects in FP

ZIO is not unique in the Scala landscape when it comes to working with effects. Programmers have been working with Cats Effect and Monix for years. Cats Effect had its first commits in April 2017, and Monix dates back to 2014, and there’s even this seminal book by Sam Halliday, Functional Programming for Mortals, that showed how to use the tagless-final pattern to solve these problems with Scalaz.

The basic idea is that FP purists deal in a realm of predictability and repeatability: a function given a single input should always give the same output. If you have unpredictable things like IO and external resources and other systems that can’t be deterministically modeled, a lot of the mathematically provable qualities of FP go out the window, unless you can model these external “effects” in a unified way.

If you read articles or listen to talks about effects programming, you’ll often hear the same anecdote about how horrible Scala’s Future[T] class is and how much it falls short of its promise. (Pun not intended.) In short, the moment you define a Future, you instantly kick off an uninterruptible thread-consuming process. In a very simple use case, it seems like a wonderful tool, but if you’re dealing with a lot of them or you start developing complex scenarios, the level of pain mounts quickly.

The takeaway: even if Scala is known as an FP language, Future is not a proper FP way to deal with concurrency.

The proper “effects-oriented” approach is to use Scala, our super-typesafe FP language, to construct the complete logic and “algebra” of your application, including nuances of predicting and dealing with the possibilities of failure, in a way that describes how your system should work, and then at the end of everything, to pass that well-defined system to your environment.

So whereas a Future[T] immediately kicks off a process that hopes to sometime return an object of type T, a ZIO Task[T] just describes the process that aspires to return a T, but which might also fail. If your Future[T] fails, you can’t automatically retry it—it is already spent. Instead you have to go back to whatever system created it and hope you can make another one. Alternatively a Task[T] (or the equivalent Cats Effect IO) could be composed to retry or repeat or delay or whatever. And that composability is amazingly powerful.

Core Capabilities and Evolution of ZIO

What I think is the most interesting and compelling about ZIO (apart from the vibrant community) the story of its evolution. In the preface of the in-progress book Zionomicon, John De Goes talks about the history and journey of ZIO, as it went through various interesting stages…

• Starting out as an effect type, but embracing an emphasis on concurrency problems (late 2017)
• Committing to statically-typed errors by adding an error type, much like the Either[L,R] does, but also embracing subtyping and covariance (2018 with contribution by Weim Zine Elabidine)
• Developing an asynchronous queue (contributions by Artem Pyanykh and Pierre Ricadat) and adding an entire ZManaged subsystem for reliably handling resource allocation and cleanup, plus introducing a ZIO Stream library (contributions by Itamar Ravid for both in late 2018)
• Introduction of a third “environment” type, thus completing the ZIO[R, E, A] signature that we know today (late 2018, unveiled by De Goes with his “Death of Tagless Final” talk with contributions by Weim Zine Elabidine)
• Addition of STM (Software Transactional Memory) for handling concurrent changes across two or more concurrent structures (2019 with contributions by Weim Zine Elabidine and Dejan Mijic)
• ZIO Test testing framework, both enabling proper testing of effects and concurrency, but also solving very tough problems that other testing frameworks struggled at (2018, project led by Adam Frasier)
• Adding a sort of “dependency injection” system for the environment types (the R in ZIO[R,E,A]) that handles the interdependency of services with ZLayer (2019)
• ZIO 1.0 released in August 3, 2020 after a lot of work solving and completing the structured concurrency problem.

Since it’s 1.0 release in 2020, there’s been a lot of work on ZIO 2.0. Among the work there:

• Lots of performance enhancement in the ZIO Runtime System (see John De Goes presentation in April 2021)
• Improvement and abstraction of ZQueue with the addition of ZHub for a pub/sub model (see this great ZIO World presentation by Adam Fraser)
• An elegant internal reworking of ZStreams, building a unified model for constructing, transducing (pipelining), and consuming (sinking) a stream.
• Extreme simplification of the ZLayers system, moving away from the confusing Module Pattern to the sleeker Service Pattern 2.0 (see this excellent presentation by Kit Langton from ZIO World 2021)
• Even better simplification of the new ZLayer Service Pattern by completely eliminating the need for the Has keyword (December 2021, unveiled in ZIO 2.0-RC1)
• Complete removal of the entire ZManaged resource management system, greatly simplifying the overall API and reducing the size and complexity of the core codebase! (March 2022, unveiling in ZIO 2.0-RC3)

Don’t worry about the details of all this history. In my mind, this boils down to two significant takeaways:

1. ZIO started with being a fairly simple effect type, and has evolved over several years and many people's’ great ideas and contributions to become a comprehensive system for effects, concurrency, queues topics and streams, testing, resource management and dependency injection, scheduling, etc.
2. This evolution has included an aggressive and almost merciless refactoring of things, resulting in a ZIO 2.0 that both simpler, faster and more powerful.

Impressive Multiplatform Support

It’s worth mentioning that ZIO supports Scala on the JVM, ScalaJS, and Scala Native, and it works in both Scala 2 and 3. There’s been also a lot of “interop” work so you can use it with Cats Effect or a variety of other libraries like http4s, fs2, doobie, etc.

The Inner ZIO Ecosystem

When I was first mulling over writing this article, I was considering comparing ZIO with Enterprise Javabeans or J2EE. I’ve worked extensively with J2EE in the past, and I remember thinking it was the coolest thing in the world because of all the libraries and integrated functionality and plug-and-play modularity that it had. And like Java SE, it just came with a comprehensive library that allowed you to do a whole lot without worrying about managing a bunch of 3rd-party libraries.

But I think the comparison would be a little unfair. Yes, embracing ZIO is going to involve you with a core ecosystem with types like ZStream and ZQueue and ZLayer and ZRef, along with STM and ZIO Test, which tackles a bigger problem domain than other effect libraries do, but as I just mentioned earlier, there’s been a huge focus on tightening and streamlining and simplifying, like the removal of the entire ZManaged type from the ecosystem.

The Outer ZIO Ecosystem

Outside of these core types and modules, there’s an incredible list of external ZIO projects. Some are developed and maintained by the core ZIO developers, like zio-json, zio-sqs, zio-kafka, zio-config, and zio-nio—but there are also a lot of mature projects by outside organizations, two of the most exciting ones being zio-http by Dream11 (Read this compelling article from March 2021) and Quill.

Should I use ZIO in my current/upcoming project?

Lately, when I have a new project I want to knock off—whether its pulling and transforming and assembling data from a network API (I’m doing some work based on 2020 Census data) or simply looking through Wordle results to see what viable words are left—I have been starting with a clean ZIO project and using it as an excuse to tackle the resource management and streaming and related tasks.

I’ll admit that all of these small projects would have been completed much faster if I had taken a more straightforward “ZIO-less” approach, but there’s frankly a bit of hitting-my-head-against-the-wall that I’ve had to do before all the patterns started making sense.

You don’t want to build every bit of data processing into tiny ZIO effects, else you find yourself “flat-mapping” all over the place and suffering the death of a thousand for-comprehensions.

But if you are assembling an application that will involve lots of independent services, depending on interaction between databases and cloud APIs and data streams, needing to do all sorts of “mocking” for testing, etc., then something like ZIO may very well start to pay off.

Is ZIO better than X?

I’ve always tried to be fair and platform agnostic. I’ve been through way too many platform wars (and OS wars) to go down that path. I mentioned my appreciation of ZIO’s attempt at supporting a lot of “interop” capabilities, most notably with Cats Effect.

Last year I first played with FS2 to tackle a parallel streaming problem, and I really found it easy and powerful to use. I then tried tackling the same problem with ZStream, and I was able to get things to work, but there was more of a learning curve.

And although I’ve heard more than a few FP purists pooh-pooh Akka and Akka Streams, years ago I found it much easier to visualize a problem with the Actor Model, and I wrote some effective streaming prototypes easily with Akka Streams. Things like ZIO and Cats Effect (and tagless final) are going to be very hard to train Scala-newbies to understand, whereas the Lightbend ecosystem is well supported and well documented.

On that last point, the ZIO community is a very supportive one, and they are working hard to write documentation, but the effort is sporadic. There are lots of articles and videos, and people on their Discord board are very helpful and supportive, but if you want to learn and embrace ZIO, it won’t necessarily be a short and straightforward journey. Not quite yet, at least.

So I’ll let you decide for yourself. Hopefully based on this writer’s experience, you’ll have a better feel for whether you want to jump into that world.

Topical Starting-points for Beginners

If you are very new to Scala, I think you’ll get lost trying to learn ZIO without a little bit of practice dealing with type signatures and understanding things like Either[L,R] and flatMap and for-comprehensions.

I’ve got two series of articles that I would encourage you to read (or I also have videos if you prefer that format and want to see live code examples) to get you more familiar with these things.

1. How to really use Option (and Try and Either), teaching idiomatic use, understanding flatMap and for-comprehensions and ultimately monads. Video link here.
2. Scala and the three implicits, teaching implicit classes and parameters, typeclasses and the “pimp my library” pattern. Video link here.