Running NixOS like Gentoo
I used to daily drive Gentoo before NixOS. Nix's value add was that it had the nice parts of Gentoo (the useflags, customizability, and from-source packages) while retaining the usability of something like Ubuntu. Since I've come to Nix, I've wanted to run it like Gentoo: all the useflags, mega optimization, custom linker, etc. Here's a writeup of how I've made rebuilding the world every week with all the https://www.shlomifish.org/humour/by-others/funroll-loops/Gentoo-is-Rice.html of Gentoo ferrari practical.
1 -march=native (Arch specific optimizations)
Caveat: while you can do this with
-march=native
-march=native
, I think that's against the idea of Nix. It's definitionally impure, and I love Nix's purity. Instead, I will choose the architecture of my 5950x processor's
znver3
znver3
.
To do this, I had to take three steps. First: enable the ability to build for znver3:
nix.settings.system-features = [
// ..
"gccarch-znver3"
];
nix.settings.system-features = [
// ..
"gccarch-znver3"
];
This enables nix to build programs for znver3. Since I'm already on a znver3 machine, this enabled me to target the machine I'm running on.
Next, I enabled the relevant flags by setting
mtune
mtune
and
march
march
config options when importing nixpkgs:
import nixpkgs {
// "my current system is x86_64-linux"
localSystem = "x86_64-linux";
// I want to build for znver3 microarch
hostPlatform = {
system = "x86_64-linux";
gcc.arch = "znver3";
gcc.tune = "znver3";
gcc.abi = "64";
};
// I want any compilers I build to target znver3 microarch
targetPlatform = {
system = "x86_64-linux";
gcc.arch = "znver3";
gcc.tune = "znver3";
gcc.abi = "64";
};
}
import nixpkgs {
// "my current system is x86_64-linux"
localSystem = "x86_64-linux";
// I want to build for znver3 microarch
hostPlatform = {
system = "x86_64-linux";
gcc.arch = "znver3";
gcc.tune = "znver3";
gcc.abi = "64";
};
// I want any compilers I build to target znver3 microarch
targetPlatform = {
system = "x86_64-linux";
gcc.arch = "znver3";
gcc.tune = "znver3";
gcc.abi = "64";
};
}
We can take a similar action on the GPU. Nvidia has a config option to say which cudacompute capability to use:
import nixpkgs {
// ..
cudaSupport = true;
cudaCapabilities = [ "8.9" ];
}
import nixpkgs {
// ..
cudaSupport = true;
cudaCapabilities = [ "8.9" ];
}
This allows me to emit optimized cuda code for my specific card (4090).
2 But compiling takes a really long time
Yes, this is terrible! I like to ride loose and free, so one glaring optimization is checkPhase and the installCheckPhase to run. Waiting for these (especially python install check phase) takes a really. Long. Time.
My first instinct was to set the check option:
config.doCheckByDefault = false;
config.doCheckByDefault = false;
But, disabling tests is not so trivial (e.g. this didn't work). waffle8946 had the novel idea to unset
doCheck
doCheck
and
doInstallCheck
doInstallCheck
. I ended up doing this in a custom phase to force that phase to always be run.
TODO explain
It turns out this remains difficult in the particular builders like buildPythonApplication, because they introduce their own custom "check" boolean like TODO I forget why this was hard
3
Adding
-pipe
-pipe
and
-Wl,-z,pack-relative-relocs
-Wl,-z,pack-relative-relocs
TODO talk about this
4
More
march=native
march=native
TODO zig, rust
5 A custom linker (moldStdenv)
6 lto
7 Practicality
TODO niceness, ioniceness, systemd service changes, tc