Writing a "hello world" Riscv Kernel with Nix in Rust

First, we'll grab the gnu toolchain. In order to do so, we need to specify that this toolchain is cross compiled. The nixpkgs repo defines a function in its default.nix default.nix file. I'm informally abusing notation to use {} {} to denote an attribute set containing some type of metadata.

{system information} -> {package definitions}

{system information} -> {package definitions}

We run import import which tells nix to execute the nixpkgs nixpkgs function in its default.nix default.nix and return the result. In this case, we must provide some system information in an argument to this function: specifically that our host system (denoted localSystem localSystem ) is x86_64 x86_64 linux and our target system (denoted crossSystem crossSystem ) is riscv linux. We include the triples and ABI information:

riscvPkgs = import nixpkgs {
  localSystem = "${system}";
  crossSystem = {
    config = "riscv64-unknown-linux-gnu";
    abi = "lp64";
  };
};

riscvPkgs = import nixpkgs {
  localSystem = "${system}";
  crossSystem = {
    config = "riscv64-unknown-linux-gnu";
    abi = "lp64";
  };
};

This will return a package set targeting riscv64-unknown-linux-gnu riscv64-unknown-linux-gnu with the lp64 lp64 ABI assigned to the riscvPkgs riscvPkgs variable. riscvPkgs.gcc riscvPkgs.gcc will give us a gcc version compiled to run on a riscv linux host and compile to a riscv linux target. This is not quite what we want. Instead, we'll use riscvPkgs.buildPackages.gcc riscvPkgs.buildPackages.gcc which returns a cross compiler from our host, x8664 linux, to our target, riscv64 linux. The reason this is denoted buildPackages buildPackages is because these packages are used to build the target packages. These packages run on x8664 linux.

Nixpkgs contains a qemu package definition. So first, we'll need to get a version of nixpkgs targeting x86-64-linux x86-64-linux . So we just import nixpkgs default.nix default.nix again, this time without specifying a crossSystem. Nixpkgs assumes the target is the same as the host by default:

pkgs = import nixpkgs {
  localSystem = "${system}";
};

pkgs = import nixpkgs {
  localSystem = "${system}";
};

Unfortunately, the qemu version in nixpkgs is slightly out of date. So, we'll need to override the source to get the latest version. We do this by providing an "overlay" which is a way to modify package definitions. When we import nixpkgs, we can provide a list of overlays/package definition overrides that nixpkgs will apply.

More technically, an overlay is a function that takes in the original nixpkgs package set and the final "result" package set. This overlay function overwrites package attributes in the original package set in its returned attribute set. That returned attribute set is then magically merged with all other overlays to form the final package set.

For example:

(final: prev:
  { qemu = null;}
)

(final: prev:
  { qemu = null;}
)

This overwrites the qemu package with null. So our resulting package set with this overlay will simply be null. Now all that remains is to override the src src attribute of the qemu qemu package. We know that this is the right attribute to override since nix edit nixpkgs#qemu nix edit nixpkgs#qemu shows this attribute as the source location. We use the overrideAttrs overrideAttrs package attribute: that attribute takes in a function with signature: {oldAttributes} -> {newAttributes} {oldAttributes} -> {newAttributes} . The new attributes are then unioned with the old attributes and any duplicates are replaced with the values in newAttributes newAttributes .

So, our final expression ends up as:

pkgs = import nixpkgs {
  localSystem = "x86_64-linux";
  overlays = [
    (final: prev:
      {
        qemu = prev.qemu.overrideAttrs (oldAttrs: {
          src = builtins.fetchurl {
            url = "https://download.qemu.org/qemu-6.0.0.tar.xz";
            sha256 = "1f9hz8rf12jm8baa7kda34yl4hyl0xh0c4ap03krfjx23i3img47";
          };
        });
      }
    )
  ];
};

pkgs = import nixpkgs {
  localSystem = "x86_64-linux";
  overlays = [
    (final: prev:
      {
        qemu = prev.qemu.overrideAttrs (oldAttrs: {
          src = builtins.fetchurl {
            url = "https://download.qemu.org/qemu-6.0.0.tar.xz";
            sha256 = "1f9hz8rf12jm8baa7kda34yl4hyl0xh0c4ap03krfjx23i3img47";
          };
        });
      }
    )
  ];
};

builtins.fetchurl builtins.fetchurl wgets the url prior to build time and ensures the source sha256 hash matches what we expect (e.g. the provider did not change the input source). This is a step nix requires for reproducibility. Now, from this we can just reach in and grab our modified qemu package via pkgs.qemu pkgs.qemu .

Next, let's grab the rust cross compiler toolchain. To do this, we use the rust-overlay rust-overlay input flake. This flake provides a bunch of outputs. One for each toolchain we might want. We add this overlay to our overlay list in our pkgs pkgs definition to access the rust binaries rust-overlay rust-overlay provides:

...
  overlays = [rustoverlay.overlay ... ];
...

...
  overlays = [rustoverlay.overlay ... ];
...

We're going to use a fairly recent version of nightly, so we grab the May 10th nightly release: pkgs.rust-bin.nightly."2021-05-10.default" pkgs.rust-bin.nightly."2021-05-10.default" . This is almost good enough, except it's not a cross compiler. The package definition is a function that takes input arguments. We can override those input arguments to say we would like a cross compiler by setting the override attribute:

rust_build = pkgs.rust-bin.nightly."2021-05-10".default.override {
  targets = [ "riscv64imac-unknown-none-elf" ];
  extensions = [ "rust-src" "clippy" "cargo" "rustfmt-preview" ];
};

rust_build = pkgs.rust-bin.nightly."2021-05-10".default.override {
  targets = [ "riscv64imac-unknown-none-elf" ];
  extensions = [ "rust-src" "clippy" "cargo" "rustfmt-preview" ];
};

Note I also overwrote the extensions attribute to provide standard rust tooling. Since nixpkgs doesn't have rustup, this is the declarative equivalent to

rustup toolchain add riscv64imac-unknown-none-elf
rustup component add rust-src clippy cargo rustfmt-preview

rustup toolchain add riscv64imac-unknown-none-elf
rustup component add rust-src clippy cargo rustfmt-preview

To access this tooling, we just include the resulting rust_build rust_build derivation wherever we need it.

Using our imports, we can now add a development shell with all of these dependencies installed by defining a devshell.x86_64-linux devshell.x86_64-linux attribute in our output attribute set:

devShell.x86_64-linux = pkgs.mkShell {
  nativeBuildInputs = [ pkgs.qemu rust_build riscvPkgs.buildPackages.gcc riscvPkgs.buildPackages.gdb ];
};

devShell.x86_64-linux = pkgs.mkShell {
  nativeBuildInputs = [ pkgs.qemu rust_build riscvPkgs.buildPackages.gcc riscvPkgs.buildPackages.gdb ];
};

When we run nix develop nix develop , the packages listed in nativeBuildInputs nativeBuildInputs will be built (or pulled from a local/remote cache) and included on our path. This is very useful for per-project per-project tooling. Now that we have the relevant toolchains, we can write the rust kernel.

We first create a Cargo.toml Cargo.toml file. Pretty standard so far. We don't bother to specify any targets as the defaults are good enough. Then we create a src/main.rs src/main.rs file. We include a _start _start symbol and a panic handler. This is the bare minimum Rust requires to compile properly. We enable no_std no_std , as we will only be using the core core rust library on bare metal, no_main no_main as initially we will not have a main main function, and naked_functions naked_functions as we do not want Rust to start pushing registers to the stack in _start _start before we have initialized the stack pointer.

#![no_std]
#![no_main]
#![feature(naked_functions)]

use core::panic::PanicInfo;

#[panic_handler]
fn panic(_info: &PanicInfo) -> ! {
  loop {}
}

#[naked]
#[no_mangle]
pub extern "C" fn _start() -> ! {
  loop {}
}

#![no_std]
#![no_main]
#![feature(naked_functions)]

use core::panic::PanicInfo;

#[panic_handler]
fn panic(_info: &PanicInfo) -> ! {
  loop {}
}

#[naked]
#[no_mangle]
pub extern "C" fn _start() -> ! {
  loop {}
}

The exclamation mark return type means that the functions do not return. Another compile time check by Rustc.

We need to compile our kernel. It turns out that

cargo rustc --release --target="riscv64imac-unknown-none-elf"

cargo rustc --release --target="riscv64imac-unknown-none-elf"

will cross compile the kernel to a riscv64imac target. We invoke cargo in this unorthodox way because we'll need to pass flags to the linker later on. This command will also generate a Cargo.lock Cargo.lock file that we must git add git add so nix flakes may track it.

In order to build our derivation with nix, we'll use Naersk Naersk . Naersk Naersk provides a lib.x86_64-linux.buildPackage lib.x86_64-linux.buildPackage function that will use cargo (and Cargo.lock Cargo.lock ) to build rust packages with nix. We tell Naersk Naersk to use our cross compiler by overriding its input rust toolchain (in the same way as the rust-overlay override):

naersk_lib = naersk.lib."${system}".override {
  rustc = rust_build;
  cargo = rust_build;
};

naersk_lib = naersk.lib."${system}".override {
  rustc = rust_build;
  cargo = rust_build;
};

We use this naersk_lib naersk_lib to build our package:

sample_package = naersk_lib.buildPackage {
  pname = "example_kernel";
  root = ./.;
  cargoBuild = _orig: "CARGO_BUILD_TARGET_DIR=$out cargo rustc --release --target=\"riscv64imac-unknown-none-elf\"";
};

sample_package = naersk_lib.buildPackage {
  pname = "example_kernel";
  root = ./.;
  cargoBuild = _orig: "CARGO_BUILD_TARGET_DIR=$out cargo rustc --release --target=\"riscv64imac-unknown-none-elf\"";
};

The pname pname is the "package name". root root is the root is the directory in which naersk naersk will invoke the cargoBuild cargoBuild command. cargoBuild cargoBuild is a function that takes in the default cargo build command (which we subsequently drop), and returns a new cargo build command to be used. The only difference here is that CARGO_BUILD_TARGET CARGO_BUILD_TARGET cannot be our source directory. We need Cargo to build in the derivation's output directory, so we set it to $out $out (which points there).

We'd also like a script that runs this in qemu. We can create one:

sample_usage = pkgs.writeScript "run_toy_kernel" ''
  #!/usr/bin/env bash
  ${pkgs.qemu}/bin/qemu-system-riscv64 -kernel ${sample_package}/riscv64imac-unknown-none-elf/release/nix_example_kernel -machine sifive_u
'';

sample_usage = pkgs.writeScript "run_toy_kernel" ''
  #!/usr/bin/env bash
  ${pkgs.qemu}/bin/qemu-system-riscv64 -kernel ${sample_package}/riscv64imac-unknown-none-elf/release/nix_example_kernel -machine sifive_u
'';

This creates a sample script that runs the kernel nix builds (with openSBI as the bios) on the sifive_u machine. We will use this for testing.

In order to make these outputs accessible, we add them to the output attribute set:

packages.riscv64-linux.kernel = sample_package;
packages.riscv64-linux.defaultPackage = sample_package;
apps.x86_64-linux.toy_kernel = {
  type = "app";
  program = "${sample_usage}";
};
defaultApp.x86_64-linux = self.apps.x86_64-linux.toy_kernel;

packages.riscv64-linux.kernel = sample_package;
packages.riscv64-linux.defaultPackage = sample_package;
apps.x86_64-linux.toy_kernel = {
  type = "app";
  program = "${sample_usage}";
};
defaultApp.x86_64-linux = self.apps.x86_64-linux.toy_kernel;

nix run . nix run . executes the defaultApp defaultApp . We can call this from any linux x8664 box by calling nix run github:DieracDelta/NixKernelTutorial nix run github:DieracDelta/NixKernelTutorial . The same goes for defaultPackage defaultPackage . We may build the kernel by running nix build . nix build . or nix build github:DieracDelta/NixKernelTutorial nix build github:DieracDelta/NixKernelTutorial .

We want our kernel to do something: print hello world. In order to do this, we'll have to make sure our ELF sections get placed in the correct spot to match the memory map of the sifive_u machine. We'll also need a stack. When we nix run nix run our kernel and look at serial output of qemu with what we have so far, we'll see openSBI printed: Domain0 Next Address : 0x0000000080200000 Domain0 Next Address : 0x0000000080200000 . This tells us that OpenSBI expects our start symbol to be 0x80200000 0x80200000 for the sifive_u machine target.

Here's the example linker script I'm using:

OUTPUT_ARCH( "riscv" )

ENTRY( _start )

MEMORY
{
  ram (rwx) : ORIGIN = 0x80200000, LENGTH = 0x80000
}

SECTIONS
{
  .kernel : {
    *(.text .text.*)
    *(.rodata .rodata.*)
    *(.data .data.*)
  } > ram

  .stack (NOLOAD) : {
    . = . + 0x10000;
    PROVIDE(_end_stack = .);
  } > ram

}

OUTPUT_ARCH( "riscv" )

ENTRY( _start )

MEMORY
{
  ram (rwx) : ORIGIN = 0x80200000, LENGTH = 0x80000
}

SECTIONS
{
  .kernel : {
    *(.text .text.*)
    *(.rodata .rodata.*)
    *(.data .data.*)
  } > ram

  .stack (NOLOAD) : {
    . = . + 0x10000;
    PROVIDE(_end_stack = .);
  } > ram

}

I've chosen the RAM length attribute arbitrarily, but this allocates a kernel kernel section to load in the elf sections into and a stack. We can see this in the objdump ( objdump -h riscv64imac-unknown-none-elf/release/nix_example_kernel objdump -h riscv64imac-unknown-none-elf/release/nix_example_kernel ):

Sections:
Idx Name              Size      VMA               LMA               File off  Algn  Flags
  0 .text             000000d8  0000000080200000  0000000080200000  00001000  2**1  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .rodata           0000002b  00000000802000d8  00000000802000d8  000010d8  2**0  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .data             00000efd  0000000080200103  0000000080200103  00001103  2**0  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .stack            00010000  0000000080201000  0000000080201000  00002000  2**0  ALLOC
  4 .riscv.attributes 0000002b  0000000000000000  0000000000000000  00002000  2**0  CONTENTS, READONLY
  5 .comment          00000013  0000000000000000  0000000000000000  0000202b  2**0  CONTENTS, READONLY

Sections:
Idx Name              Size      VMA               LMA               File off  Algn  Flags
  0 .text             000000d8  0000000080200000  0000000080200000  00001000  2**1  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .rodata           0000002b  00000000802000d8  00000000802000d8  000010d8  2**0  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .data             00000efd  0000000080200103  0000000080200103  00001103  2**0  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .stack            00010000  0000000080201000  0000000080201000  00002000  2**0  ALLOC
  4 .riscv.attributes 0000002b  0000000000000000  0000000000000000  00002000  2**0  CONTENTS, READONLY
  5 .comment          00000013  0000000000000000  0000000000000000  0000202b  2**0  CONTENTS, READONLY

To build this with cargo, we need to add an argument through llvm to the linker:

cargo rustc --release --target=\"riscv64imac-unknown-none-elf\" -- -Clink-arg=-Tlinker.ld

cargo rustc --release --target=\"riscv64imac-unknown-none-elf\" -- -Clink-arg=-Tlinker.ld

In our _start _start function we must set up the stack. We can do this with some inline assembly:

extern "C" {
    static _end_stack: usize;
}
...
... fn _start( ... {
  asm!(
      "
          la sp, {end_stack}
          j main
      ",
      end_stack = sym _end_stack,
      options(noreturn)
  );
}

extern "C" {
    static _end_stack: usize;
}
...
... fn _start( ... {
  asm!(
      "
          la sp, {end_stack}
          j main
      ",
      end_stack = sym _end_stack,
      options(noreturn)
  );
}

The _end_stack _end_stack extern C definition tells the rust compiler to look for this symbol in the linker script. Then all we do is move the symbol into sp sp , and jump to main. We have to specify that the function does not return in order for the rust compiler's checks to pass. We'll also need to define a main main function to actually jump to, which we'll do later on.

We're using OpenSBI as a SEE or Supervisor Execution Environment. It runs in M mode (equivalent to Ring 0 on x86) and ensures that the kernel (running in S-mode/Ring 1) doesn't have as much power. The kernel can make "syscalls" to the SEE in much the same way that a userspace application makes syscalls to the kernel.

We would like to print "hello world" to one of the UARTs. We could do this by implementing a UART driver, but it's easier to just let openSBI do it for us. Not to mention this post is getting rather long. According to the SBI spec, if we do a syscall (ecall instruction) from S mode (which our kernel is running in) to SBI running in M mode with the SBI Extension ID (EID) of 1 stored in a7 a7 , and the address of the character we wish to print stored in a0 a0 , openSBI will print will print that character for us using its UART implementation. So we write a function to do this:

unsafe fn write_char(ch: u8) {
    asm!(
    "
    li a7, 0x1
    lw a0, 0({0})
    ecall
    " , in(reg) (&ch), out("a0") _, out("a7") _
    );
}

unsafe fn write_char(ch: u8) {
    asm!(
    "
    li a7, 0x1
    lw a0, 0({0})
    ecall
    " , in(reg) (&ch), out("a0") _, out("a7") _
    );
}

This is more or less the same as C's inline assembly, though more readable (to me at least). Note that a0 a0 and a7 a7 are clobbered, and the address of ch ch is used as an input. &ch &ch is now accessible via {0} {0} . We can then call this function in main main to print hello world:

#[no_mangle]
pub extern "C" fn main() -> ! {
    unsafe {
        "Hello World from a nixified rust kernel :D\n"
            .chars()
            .for_each(|c| write_char(c as u8));
    }
    loop {}
}

#[no_mangle]
pub extern "C" fn main() -> ! {
    unsafe {
        "Hello World from a nixified rust kernel :D\n"
            .chars()
            .for_each(|c| write_char(c as u8));
    }
    loop {}
}

And now we have a Rust kernel that prints "hello world" built by and runnable with nix. nix run github:DieracDelta/NixKernelTutorial nix run github:DieracDelta/NixKernelTutorial

It's trivial to add a github action that builds the kernel using nix nix . I've talked about this in my other posts so I won't repeat myself. See the cachix.yml cachix.yml workflow file.