Today, we're releasing garnix modules.

On the surface, garnix modules is a CI/CD Platform-as-a-Service such as you might be familiar with. You tell us a little about your project, like the stack and compiler version, and we build, test and deploy it. There's been a renaissance of such platforms — witness Fly.io, Render, Railway. Still, a common refrain is that we peaked with Heroku almost 15 years ago, and there hasn't been substantial progress since.

garnix decided to go quite a different route from these alternatives, and we think this brings something new to the table. Some of the advantages it has, as pertains mostly to the CI side of CI/CD:

• Smarter build caching. Not only do you not have to worry about setting up a cache, about invalidating it too often or not often enough, but we make it possible to safely share the cache for external PRs, and even for external repositories! If someone built a package with garnix, you won't have to.
• Build coordination. If you push two different commits in quick succession, and a component of yours hasn't changed between them, only one of the check runs will build the package, while the other will reuse the logs.
• Your CI and infrastructure are easy to reproduce locally. With a lot of CIs, there's this excruciatingly slow dance, where you try to get the configuration right, push, notice a missing semicolon after the 5 minutes it takes to run, fix that, notice a period missing after another 5 minutes... Whereas testing locally would have been much easier and faster. And there is always the fact of vendor lock-in. garnix allows you to build locally (or anywhere else) exactly what garnix itself does. Not only that, but it makes it incredibly easy to run VMs for your entire infrastructure locally (or, again, anywhere else).
• You (and your coworkers/users) benefit from the cache. Usually CI systems have a cache only for the benefit of future CI runs. garnix makes its cache available to everyone (who has access), which means that if you're running things locally (for example, the VMs mentioned above), you won't need to build whatever garnix already has. In other words, the CI doubles up as your own build farm.
• Builds are safer. We know from various exploits of NPM, Python packaging, etc., that supply-chain attacks from dependencies can compromise not just the executables that use those dependencies, but any system involved in the building of that executable. This is because with a lot of toolchains, dependencies can run code at build time. garnix sandboxes each component of the build, making it safe for anyone to build untrusted code (even if not to run it).
• Builds are verifiable. The artifacts produced by garnix are often (though not always) reproducible. This means that you, and your users, can rebuild the artifacts to verify that they are in fact built from the given source code, with nothing sneaky added or removed.
• Builds are easy to test across different architectures. Usually if you want to to build and test a project on both Macs and Linux, you'll need to write a lot of the set up code twice. After all, things like apt don't work on Macs, nor brew on Linux; other differences also crop up. With garnix, it's usually just a question of enabling it.

There are in fact a number of other advantages, but this blog is about the CI side of things. (An upcoming blog post will discuss how garnix modules re-imagines hosting.) A founding belief for garnix is that a great PaaS has to be a great CI service. It should be so good that you would want to use it even if you're not deploying servers. Otherwise, you'll always be stuck writing glue code, and manually shuttling information that your CI system knows into your hosting provider. Moreover, somewhere between full VM network testing, preview apps, canary deployments and staging environments, the boundary between CI and hosting anyhow blurs.

garnix started its life as CI only, and that's still its most popular use, so there's reason to take us seriously as CI.

What are garnix modules

I've said that garnix modules are being released now, but also that garnix has been around for a while. What then exactly is garnix, and what are garnix modules?

garnix is a platform that builds and deploys Nix-capable projects. Nix is a remarkably powerful piece of software, undergirding many of the advantages mentioned above. But it's also a difficult technology to learn, with poor documentation, and lots of sharp corners.

That's where garnix modules come in.

Making Nix Invisible

garnix modules, the thing we're releasing today, is a way of abstracting the Nix away. A module is, in essence, a function along with a nice schema definition (and name, description, etc.) for the parameters of that function. We write (in Nix) modules for particular stacks and technologies, such as Rust or PostgreSQL, and each one describes precisely how to build, test, and deploy projects using those stacks. From these modules, we can generate web forms you can fill on our website, since we have the schema. The values filled out become parameters that the function receives.

Once you fill that out, we can apply the function. You can preview the build, checking that everything works. You then have the option of generating a pull-request to your repo with the entire (quite short) Nix code, so that your choices are captured in version control. The parameters are kept separate, so it's relatively easy to know what to update when something needs to change.

There's some similarity to buildpacks and Ansible modules. Each of these allow you to build and deploy software with minimal configuration. Each of them is also based on encapsulating the knowledge of domain experts (in both the stack they're building, and in the tool itself), and making usage simple. Compare, for instance, writing an Ansible module to using it. Similarly with garnix modules, in order to write a new module, you need to know Nix; to use it, you don't need to have special knowledge.

Nix in Brief

But, of course, garnix modules are also fundamentally different in that they do use Nix. ?

I'll repeat: you don't need to know any Nix to use garnix modules. But for the curious, and for an understanding what makes certain features of garnix possible, having a rough idea helps.

You're probably familiar with package lock files package-lock.json with NPM, cargo.lock with Cargo and Rust, and poetry.lock with Poetry and Python. Without these lock files, your package manager does a very unreproducible thing: it asks the internet what's the latest version of a package (potentially matching certain constraints), and downloads that. It's unreproducible because what the latest version of a package is might change: new packages are regularly added, and in some cases, packages are also modified or removed altogether.

This leads to lots of problems that you're likely familiar with. To avoid that, most language package managers can work with (and in fact generate) lock files, which lock down the specific versions they're downloading very precisely, storing this information for future runs.

Nix is similar, but with two significant changes:

• First, it applies this to everything, not just your language libraries. It applies it to all languages, to the compiler or interpreter itself, to system libraries, to extra files you might need during the build process, to environment variables, to the system architecture... they all get locked down.?

• With everything locked down, you can do a neat trick. When you build your project — your Rust binary, for example — you can give it a name that is the hash of your project's source code, plus of that lock file. If you store everything you ever built with that name in some directory, then before building, you can check whether the file with the name you know your end result will have already exists, and if so, skip the build altogether and reuse that. You know that substitution is safe because everything was locked down and is reflected in the name/hash. So the second thing Nix does is keep a huge directory with those builds: an input-addressed cache. ("Input-addressed" because the name or address is based on the hash of the inputs.)

A cache

This gives you a very powerful cache system already. It's in many ways better than what incremental build tools give you, since you're not constantly overwriting previous builds and only keeping the latest version. Thus, if you build, switch branches, build, switch branches back, and build again, that last build will be cached.

If you share that cache (the directory we mentioned) with others, they can also benefit from your build, and skip doing it themselves. As long, that is, as they trust you. garnix provides a cache with the artifacts of all builds, and it only puts things in there that it itself built, so that it's safe to trust. If someone out there built a Rust library as what you need (with the same version, compiler version etc. as what you need), you won't have to. You don't even need to know that person exists. That way, things only get built once.

Because, unlike the vast diversity of language-specific build tools, Nix uses a unified interface for this cache, garnix can handle all of the logic uniformly. You don't have to set up anything at all.

A build farm

Since this cache works so well, and garnix offers CI to populate it, your CI service ends up doubling up as a remote builder. Instead of building locally, you can commit and push; if you try to build after garnix is done, it'll transparently be substituted with the cached result. You can work on complex projects with long compilation times in the flimsiest of mobile devices, as long as you have a good internet connection!

And because garnix machines have different architectures, you can use our system as an alternative to cross-compilation. Have something you want in your Raspberry PI, but don't have the patience (or the memory) to build it on the device? Just push a commit, and garnix will handle it.

Finally, since we have a very good name that uniquely identifies every build (the hash of the source and the lock file), we also can deduplicate concurrent builds. If a build comes in that has the same name as one that is in flight, just use that one instead!

Verification

Because we locked down everything about a build, if we build twice, we should get exactly the same thing. So if you're working on a security-critical piece of software — if, say, you're developing a widely-used executable such as a browser, or a financially significant one such as a blockchain client — you can have independent entities build it, and compare the binary bit-for-bit. If they're the same, and you trust that not all of those entities are malicious or compromised, you should trust the binary.

(There are sources of non-determinism, such as the fact that parallelism is still allowed inside a build. Sometimes this can result in non-deterministic builds, but for the majority of cases it does not, or the fix is simple.)

Local development

Another cool thing about having these hash-based names that uniquely identify a build is that multiple versions of software can coexist peacefully. In other systems, updating Python (say) is a global operation, which has effects on a lot of things. With Nix, you tend to refer to software by its hash-based name; you don't so much "update" Python as download another version.

This forms the basis for Nix's extraordinarily good support for development environments (also known as devshells).

garnix modules creates these for you, with all of the software you need to build and develop. After you create a pull request with garnix modules, you can install Nix locally and run nix develop to enter a shell containing all the tools you need to work on your project.

Conclusion

We have a few modules that you can explore already, and intend to add more soon. Existing modules are all open source, so if something is missing it's easy to create an issue or submit a PR with the fix. But we've also designed with the aim to soon allow user-contributed modules. If you're interested in writing modules, get in touch via our Discord channel!

It's of course still the first public release of garnix modules, so expect rough edges. But we hope it's already a solid case for reinventing CI. In the next blog post, we'll hear more about CD and hosting. In the meantime, the modules page should be intuitive enough so that you can get started. Our free tier is quite generous with both CI minutes, and hosting servers.