NixOS on Clients #9

New issue

Open

opened 2022-01-04 10:41:51 +00:00 by sbruder · 2 comments

sbruder commented

2022-01-04 10:41:51 +00:00

(Migrated from gitlab.jalr.de)

Goal

This Issue should document the steps to achieve network booting on the client machines with NixOS.

Overview

Unlike labsync, the proposed implementation for the NixOS based client system is not using a squashfs filesystem for booting, but is using a more granular and hopefully more efficient (both space and time at both build and boot time) way of having all required resources available on boot.

Example boot process

Client boots and starts PXE
Client receives DHCP response from raven with file to boot via PXE
Client downloads pxelinux and a generated configuration via TFTP from raven
Client displays the boot menu with its first entry pointing to the latest generation
Client downloads initrd and kernel via HTTP from raven
Client starts initrd
Client begins NixOS phase 1 boot
Client automatically checks whether the partition layout on the disk is already as required, otherwise it will proceed to create that partition layout as part of phase 1 (declared in boot.initrd.preLVMCommands)
Client continues with regular phase 1 boot, which mounts the devices
After the file systems are mounted (boot.initrd.postMountCommands), the client uses Nix (which is included in the initrd) to download the system closure onto the local disk
Client continues with phase 1 and prepares for phase 2
Client runs phase 2, which executes the activation script of the generation selected at the boot menu
Client successfully boots into the generation

Implementation

Downloading the system closure can be achieved with a configuration like the following snippet:

{
  boot.initrd = {
    network.enable = true;
    postMountCommands = let
      # FIXME: replace with system closure
      # !!! for this to work in a VM (nixos-rebuild build-vm), the store path MUST NOT exist on the host
      remoteStorePath = "/nix/store/jdcv29qawqxlb3a33l3wksxzm39digk4-hello-2.10";
    in
    ''
      set -x
      mkdir -p /etc/ssl/certs /mnt-root/nix/store
      ln -s ${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt
      ls /mnt-root${remoteStorePath}/bin/hello
      ${pkgs.bashInteractive}/bin/bash -i
      USER=root NIX_REMOTE=local?root=/mnt-root ${pkgs.nix}/bin/nix-store -r ${remoteStorePath}
      ls /mnt-root${remoteStorePath}/bin/hello
      ${pkgs.bashInteractive}/bin/bash -i
      set +x
    '';
  };

  # this is required to use NixOS’ DHCP/DNS setup for initrd (system/boot/initrd-network.nix)
  # TODO: it might conflict with NetworkManager
  networking.interfaces.eth0.useDHCP = true;
}

## Goal This Issue should document the steps to achieve network booting on the client machines with NixOS. ## Overview Unlike labsync, the proposed implementation for the NixOS based client system is not using a squashfs filesystem for booting, but is using a more granular and hopefully more efficient (both space and time at both build and boot time) way of having all required resources available on boot. ### Example boot process 1. Client boots and starts PXE 2. Client receives DHCP response from raven with file to boot via PXE 3. Client downloads pxelinux and a generated configuration via TFTP from raven 4. Client displays the boot menu with its first entry pointing to the latest generation 5. Client downloads initrd and kernel via HTTP from raven 6. Client starts initrd 7. Client begins NixOS phase 1 boot 8. Client automatically checks whether the partition layout on the disk is already as required, otherwise it will proceed to create that partition layout as part of phase 1 (declared in `boot.initrd.preLVMCommands`) 9. Client continues with regular phase 1 boot, which mounts the devices 10. After the file systems are mounted (`boot.initrd.postMountCommands`), the client uses Nix (which is included in the initrd) to download the system closure onto the local disk 11. Client continues with phase 1 and prepares for phase 2 12. Client runs phase 2, which executes the activation script of the generation selected at the boot menu 13. Client successfully boots into the generation ## Implementation Downloading the system closure can be achieved with a configuration like the following snippet: ```nix { boot.initrd = { network.enable = true; postMountCommands = let # FIXME: replace with system closure # !!! for this to work in a VM (nixos-rebuild build-vm), the store path MUST NOT exist on the host remoteStorePath = "/nix/store/jdcv29qawqxlb3a33l3wksxzm39digk4-hello-2.10"; in '' set -x mkdir -p /etc/ssl/certs /mnt-root/nix/store ln -s ${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt ls /mnt-root${remoteStorePath}/bin/hello ${pkgs.bashInteractive}/bin/bash -i USER=root NIX_REMOTE=local?root=/mnt-root ${pkgs.nix}/bin/nix-store -r ${remoteStorePath} ls /mnt-root${remoteStorePath}/bin/hello ${pkgs.bashInteractive}/bin/bash -i set +x ''; }; # this is required to use NixOS’ DHCP/DNS setup for initrd (system/boot/initrd-network.nix) # TODO: it might conflict with NetworkManager networking.interfaces.eth0.useDHCP = true; } ```

sbruder commented

2022-01-04 10:41:51 +00:00

(Migrated from gitlab.jalr.de)

assigned to @sbruder

sbruder commented

2022-01-04 11:59:01 +00:00

(Migrated from gitlab.jalr.de)

changed the description