Gerd Zellweger recently wrote a blog post about various shortcomings of GitHub Actions. He points out that starting from some basic requirements, it's relatively involved to get to a working solution for CI on GitHub Actions, and just as difficult to maintain it. Some of the problems he mentions are the annoyance of required checks, especially when they interact with autofixes and merge queues?; the security issues introduced by unpinned Actions being the default, and a rather obscure system of tokens; the difficulty of testing GitHub actions locally and in isolated steps; and speed.

He concludes:

I just wish the process to get there would be less time-consuming and that it would be easier to debug when things go wrong. I guess, I'm hoping for some innovation here.

How would that innovation look?

mikepurvis wrote an insightful comment on HN:

Strongly isolated systems like Nix and Bazel are amazing for giving no-fuss local reproducibility.

Every CI "platform" is trying to seduce you into breaking things out into steps so that you can see their little visualizations of what's running in parallel or write special logic in groovy or JS to talk to an API and generate notifications or badges or whatever on the build page. All of that is cute, but it's ultimately the tail wagging the dog— the underlying build tool should be what is managing and ordering the build, not the GUI.

What I'd really like for next gen CI is a system that can get deep hooks into local-first tools. Don't make me define a bunch of "steps" for you to run, instead talk to my build tool and just display for me what the build tool is doing. Show me the order of things it built, show me the individual logs of everything it did.

Same thing with test runners. How are we still stuck in a world where the test runner has its own totally opaque parallelism regime and our only insight is whatever it chooses to dump into XML at the end, which will be probably be nothing if the test executable crashes? Why can't the test runner tell the CI system what all the processes are that it forked off and where each one's respective log file and exit status is expected to be?

Because local reproducibility is a requirement, and vendor lock-in should be avoided, presumably the innovation we are looking for should be a standalone tool, which I can run on my own machine, or on GitHub Actions, or on my own CI service, with minimal fuss in switching between those. The tool should "break up" the run into the appropriate components, and any CI service should simply do a good job of displaying that. Even parallelism and sequencing should be controlled by you via that tool, and the CI service should just follow that lead.

Do we have a good tool that fits the bill? mikepurvis already suggested one:

Nix

In response to the article, a lot of people on Hacker News pointed out that Nix was part of their answer. We agree. (In fact, we agree so much that we built a CI platform on Nix!)

Nix runs builds in sandboxed environments, with tightly regulated access to the network and to the rest of the filesystem.

Here's how the CI picture changes when you switch to Nix:

1. There's no vendor lock-in, because it's built on an open source project. We happen to think using garnix makes many things nicer — faster to set up, better cache infrastructure, nicer logs, and faster runs. But if you don't like us or we go under, it's easy enough to switch. To GitHub Actions itself if you wish!
2. Relatedly, you can easily reproduce the run locally. It's just a nix build. No manual steps of installing dependencies, or worrying about CI and local builds diverging.
3. CI is not just CI, but a lot like having your own build farm. What you build locally and on CI is so much the same (see above), that you can substitute one for the other: when running anything locally, check whether CI has already done it (and uploaded to a cache); if so just download the artifacts. Nix does the checking transparently for you, and garnix automatically does the caching.
4. It's fast. Nix gives you caching from the get-go, so many parts of your build get skipped when possible: building dependencies, building unchanged parts of your project. garnix actually makes the cache global, so if anyone built it, you won't have to. (garnix also happens to use much better hardware, but that's not a Nix-specific thing.)
5. Running just a part of the CI pipeline is easy. Your pipeline will be separated into packages, and you can build each separately (as long the dependencies are built).
6. Security is a lot better. Gerd mentions the tj-actions/changed-files compromise; with Nix everything is pinned — that's in fact the point of Nix! And that's not even the main security improvement. Nix additionally sandboxes builds, removing most network access. The cases where such network access is allowed are made very explicit; and while a dependency of yours can request network access without your knowledge, it is built without access to your code, making it irrelevant that it has that network access. Insofar as you use that dependency in your own build, it won't have network access without your awareness.
7. Security is a lot better (pt. 2: tokens). In GitHub Actions tokens are needed for a lot of things, because logic such as creating a check or reading from/writing to the cache happen within the particular Action step. When you use Nix, there's a carefully considered interface between the build step (which is only semi-trusted, and can essentially only create files with specific names and return an exit code), and everything else.

(Of course there are still security concerns that should concern you when using Nix, and malicious dependencies can still do damage — by embedding themselves in the final build artifact, for instance.)

The main downside of using Nix for CI (or for anything else) has been that you have to learn Nix. We recently release garnix modules to help with that (and to make Nix invisible). If your stack is supported by us already, you can use this tool to generate the Nix code for you. You can use this with or without garnix.

Conclusion

Nix is the right way of doing CI. It's faster, more reliable, more debuggable, and safer. It certainly has problems of its own, such as being hard to learn, but increasingly we're seeing solutions to that too. If you're finding GitHub Actions frustrating or slow, consider trying it!