Nix flakes, and how to convert to them

Nix's (relatively) new "flakes" feature has a lot going for it.? Software can now be easily distributed outside of the nixpkgs monorepo. Updating the versions of such software in your own project is as easy as with npm or poetry. And more importantly, nix finally achieves the hermeticity it needs to keep the works-here-works-anywhere promise.

There are lots of resources on how this works, and how to get started with flakes, among which:

• a series of blog posts by Eelco Dolstra, the creator of nix.
• another, still on-going series by Xe Iaso.
• a post by zimbatm, with most of the information you might need organized in a more reference-like format.
• a similar page on the nix wiki

Most resources on nix flakes, however, assume you're starting a project from scratch. They don't give you much guidance about how to convert an existing nix project to flakes.

In this blog post, we'll do just that. With a further constraint: we'll keep the project working as a legacy nix project, and with relatively minimal changes. That way, people who prefer flakes can use flakes, people who don't can use legacy nix.?

We'll see that there's an additional advantage to approaching flakes in this way: it makes the differences between the two different systems very clear. We'll be looking for a way to go, with the minimal changes, from one system to another, which in a way gives us the 'diff' between these systems.

Prerequisites

Since nix version 2.4, flakes have been supported by the default nix binary. Before that, there was a separate nix binary for flakes.

Check what version you have with nix --version. If it's 2.4 or higher, you are good to go. Otherwise, you'll need a newer version.

Then, enable flakes by adding

experimental-features = nix-command flakes

To nix.extraOptions if using NixOS, and to /etc/nix/nix.conf or ~/.config/nix/nix.conf otherwise.

Nix flake format; or, giving Nix a shape

With traditional nix, there aren't very many rules as to how projects are packaged, only loose conventions. Projects often have a top-level default.nix file, that often looks like this:

{ pkgs
, compiler ? "ghc883"
}:
{
  pkg1 = ...;
  pkg2 = ...;
}

That is, a function, which takes the version of nixpkgs (and potentially other options) as an argument, and which returns a list of derivation-valued attributes (key-value maps).

But often none of this is true. Maybe pkgs is pinned rather than being an option; or default.nix provides an overlayed package set, and a different file (e.g., release.nix) provides only the project-specific packages. Maybe the packages are in nested attributes.

Everything is more or less convention, and there are multiple conventions. The only file nix treats somewhat specially is default.nix. But even then, only in the quite minor way that you can import it via a short form, keeping only the directory (e.g. import ./nix instead of import ./nix/default.nix).

Flakes are quite different. If we try the flake build command:

$ nix build .
> error: path '<snip>' is not a flake (because it doesn't contain a 'flake.nix' file)

Compare that to legacy nix build:

$ nix-build .
> error: opening file '<snip>/default.nix': No such file or directory

In the latter case, maybe this is still a nix project - I just can't find the default file. A nix flake, on the other hand, must have a flake.nix, which must be a the project top-level. The project must be a git repo. Parameters must be provided with it's input format.

But given the default.nix above, we can leave it unchaged, and write a flake.nix that provides flake support. (It's hard to argue against adding flake support at so small a cost.)

{
  inputs.nixpkgs.url = github:NixOS/nixpkgs/nixos-21.11;
  inputs.flake-utils.url = github:numtide/flake-utils;

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      {
        packages = (import ./default.nix)
            { pkgs = nixpkgs.legacyPackages.${system}; };
      }
    );
}

There's a lot going on here, so let's break it down.

Analyzing that flake.nix

• inputs: this containts the dependencies of the flake. In this case, the dependencies are nixpkgs itself, and flake-utils. We'll talk a bit more about both.
• outputs: this is what the flake provides. As you can see, it's a function from the inputs. (More accurately, it's a function from the attrsets of the ouputs of the input flake.?

The format of the output is not obvious above, since we are using flake-utils' eachDefaultSystem function to generate it. It happens to be, roughly:

{
  packages.<system>.<pkgName> = ...
  checks.<system>.<checkName> = ...
  nixosConfigurations.<configName> = ...
  nixosModules.<moduleName> = ...
  ...
}

(Where <system> is one of x86_64-linux, aarch64_darwin, etc.) It turns out that writing the attribute set again and again for each system is tedious, and it's more convenient to have a single function from the system to the attribute set. We therefore use eachDefaultSystem, which allows us to phrase things that way, and then applies our provided function to the default list of systems. (This inconvenience is one of the current warts of flakes, in my opinion.)

Equipped with this knowledge, it should be easier to understand what we are doing. We are taking the existing default.nix, which contains a function returning an attrset of derivations:

default.nix :: { Nixpkgs, ... }->
    { pkg1 :: Derivation, pkg2 :: Derivation }

And we want:

outputs :: { Nixpkgs, ... } ->
  { packages = {
      x86_64-linux = { pkg1 :: Derivation, pkg2 :: Derivation };
      aarch64-darwin = { pkg1 :: Derivation, pkg2 :: Derivation };
      ...
  y}

Where eachDefaultSystem takes a function from the system to an attrset, and returns an attrset with the system names as keys in the right place, and the result of applying the function to that system as the value. For example:

eachDefaultSystem ( sys : { packages = { pkg1 = ..., pkg2 = ...} } )
  == { packages.x86_64-linux = { pkg1 = ..., pkg2 = ...};
       packages.aarc64-darwin = { pkg1 = ..., pkg2 = ...};
     }

nixpkgs.legacyPackages.${system} happens to be the equivalent of import nixpkgs { inherit system; }, with the advantage that it does not evaluate it twice.?

Ok, now that we understand what that initial flake.nix is doing, let's try building it:

$ nix build .
> error: source tree referenced by 'git+file:///<dir>?dir=<dir>&ref=<branch>&rev=<rev>' does not contain a 'flake.nix' file

Flakes and git

That's a bit mysterious. We just wrote a flake.nix file - how could it not be there?

The answer is that the flakes system only looks for files git knows about. This is quite nice: you don't have to use things like cleanSource of gitignoreSource; and a build with a clean working tree is completely described by the commit it's against. But the annoyance is that when you add a new file, you need to git add it too.

$ git add flake.nix
$ nix build .
> warning: Git tree '<snip>' is dirty
> warning: creating lock file '<snip>/flake.lock'
> warning: Git tree '<snip>' is dirty
> error: flake 'git+file:///<snip>' does not provide attribute 'packages.x86_64-linux.defaultPackage.x86_64-linux', 'legacyPackages.x86_64-linux.defaultPackage.x86_64-linux' or 'defaultPackage.x86_64-linux'

The first and third warning tell us that we haven't commited our changes yet (which as we've seen is relevant insofar as the current commit does not represent the version we're building).

The flake lock file

The second warning indicates that the lock file has been created. This lock file contains the state of all inputs. Just the git URL isn't enough to ensure source-binary determinism, since the commit we're on could change (or there could even be a SHA1 collision).

You might be familiar with patterns such as:

fetchFromGitHub {
  owner = "foo";
  repo = "bar";
  rev  = <rev>;
  sha256 = <sha256>;
}

or the similar fetchGit and fetchTarball. Because the sha256 is in the source code, this is "morally" pure, and deterministic - if the content at the other end changes, so will the SHA. But it's quite annoying to work with these things in nix sources - you have to put a "wrong" SHA first, get an error, replace the SHA with the one you see in the error message, and then try again. If you need to update the source, you do this again (and disturbingly, if you change the revision or repo but not the SHA, nix will silently accept that).?

Flakes substantially improve this situation. The inputs are a way of fetching that handles the SHA for you, by keeping it in a lock file. If you've ever used Nicolas Mattia's niv, you'll be familiar with this approach.

(Take a look at nix flake update --help and nix flake lock --help for more information on updating all or individual inputs.)

The defaultPackage attribute

What about the last error?

error: flake 'git+file:///<snip>' does not provide attribute 'packages.x86_64-linux.defaultPackage.x86_64-linux', 'legacyPackages.x86_64-linux.defaultPackage.x86_64-linux' or 'defaultPackage.x86_64-linux'

The error message here is a bit confusing. What is really happening is that the X in nix build X is an installable/attribute specification. This indicates what flake, and what specific attribute of the flake, you want to build. nix build github:nixos/nixpkgs#legacyPackages.x86_64-linux.hello for example, says that I want to build the attribute legacyPackages.x86_64-linux.hello in the flake that's in the github repo nixos/nixpkgs. nix build .#packages.x86_64-linux.foo specifies that I want to build packages.x86_64-linux.foo in the flake in the current directory.

But there are a lot of shorthands that apply in this syntax. #pkg becomes #packages.<this-system>.pkg, for example. And src# or src (such as .# or .) tries to build defaultPackage.<this-system>.?

Hence, the problem turns out to be that nix is expecting a defaultPackage.<system>, but we've only provided packages.<system>.<pkgName>. We can for now just be more explicit about what we want to build:

$ nix build .#pkg1
> ...

And it works!?

Some thoughts

If you're lucky, you now have a flake working alongside your legacy nix system. You might have run into issues, however. We discuss some of those below (and if there are others that you come across that you think should be listed here, get in touch!).

We also took some shortcuts, and there's room for improvements. We also talk a bit about that below.

Issues

import <nixpkgs>

It may happen that your existing project doesn't have pkgs as an argument, but instead imports <nixpkgs> directly:

let pkgs = import <nixpkgs>
in ...

In this case, you'll see an error such as:

error: cannot look up '<nixpkgs>' in pure evaluation mode (use '--impure' to override)

To fix this, refactor your code so the pkgs is an argument, and provide the argument from your flake as above. You can have the default value of the pkgs argument be the same:

{ pkgs ? import <nixpkgs> }:
...

But uses of the NIX_PATH are impurities that you probably shouldn't be using anyhow. You should instead likely pin your nixpkgs version.

currentSystem

You may get an error like this:

error: attribute 'currentSystem' missing

This happens because in the context of flakes, builtins.currentSystem does not exist (it is, after all, an impurity). If you come across this, try to refactor your legacy-nix portion so the system is always an argument, and provide that argument from your flake, as above.

Nested attrset

Flakes assume all derivations are flat: i.e, packages.x86_64-linux.foo is the derivation, not packages.x86_64-linux.foo.bar. If you encounter issues converting because of that, take a look at flake-utils' flattenTree.

nixosConfiguration

Improvements

Per-project cache

Flakes may declare their own, project-specific nixConfig, overriding global ones. The overrides are limited, but one that is allowed is substituters.

This allows you to specify a cache. Any other user of the repo will then automatically also get that cache. If you are using garnix, having:

{
  nixConfig.substituters = [ "https://cache.garnix.io" ];
}

Will do it.

Not everything belongs in packages

We took the blunt approach of putting everything in the packages attribute. But flakes have finer-grained structure than that. Some things may be better off in checks, or in nixosConfigurations. An improvement would be to split things off appropriately.

The defaultPackage again

We saw that nix makes building defaultPackage particularly convenient. (This has changed in version 2.7, to be packages.<system>.default.) An improvement is having this map to a sensible package:

{
  ...
  outputs = { self, ... }:
    ...
     {
       ...
       defaultPackage = self.packages.pkg2;
     }
}