TIL: how to optimise substitutions in Nix

This week, I ran into a problem.

I was working on a client project, and as I was building a derivation, it took a loooong time to download substitutions from the configured binary caches.

Now, the derivation in question has many input derivations for reasons that may become clearer in a future blog post. That being said, according to my betters at Numtide, the Nix daemon should pull them down from the binary caches with a reasonable degree of concurrency.

That didn’t happen based on what I saw in my terminal.

Here’s a clue as to why:

“it took a loooong time to download substitutions from the configured binary caches.”

Until now, I have taken quite a liberal approach to configuring binary caches. My system config used to include https://numtide.cachix.org, https://nix-community.cachix.org and my own binary cache by default.

On top of that, in any given client project, I might also throw in their company cache using the nixConfig.extra section in flake.nix.

I’ve never really noticed a problem until now.

But resolving that derivation with lots of little inputs required my Nix daemon to hit several caches in succession for each and every store path. And it just so happened, given the binary cache names, that the one I needed it to resolve against first was the one that it resolved against last!

I should have known better 🤦.

After all, I’ve been playing around with implementing binary caches enough to understand that each store provides a /nix-cache-info path, which contains a priority field.

Here’s the one from https://cache.nixos.org, for example:

StoreDir: /nix/store
WantMassQuery: 1
Priority: 40

Taking Control

That’s all fine and well, but you might ask yourself how this helps if the caches define their priority. What can we do if we want to control the priority instead?

nixConfig = {
  substituters = [
    "https://brianmcgee.cachix.org?priority=1"    
    "https://nix-community.cachix.org?priority=2"
    "https://numtide.cachix.org?priority=3"
    
  ];

  trusted-public-keys = [
    "brianmcgee.cachix.org-1:A2YXJUIW1spHGr2Q2sRuiL0FOcGVNYIdNjM1x+grIGg="    
    "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
    "numtide.cachix.org-1:2ps1kLBUWjxIneOy1Ik6cQjb41X0iXVXeHigGmycPPE="    
  ];
};

By appending ?priority=xxx to the binary cache URL, we can control the order in which caches are queried.

NOTE: The lower the number, the higher the priority.

In the example above the Nix Daemon will query the binary caches in the following order:

Extra performance

As I flailed around trying to make things faster, I came across the following settings in Nix config:

http-connections, which controls the maximum number of parallel TCP connections when fetching from binary caches
max-substitution-jobs, which controls the maximum number of substitution jobs that Nix will try to run in parallel.

By playing around with their values, I could significantly improve the speed with which paths were substituted. As a result, I now have both of them set to 128 in my system configuration.

Summary

Remember kids, a binary cache isn’t just for Xmas… 🎄 Treat them with respect and use them sparingly.

And if you do need to use a few of them, remember to set their priority!