fablab-nea/labsync
1
0
Fork 0
Code Issues 11 Pull requests 2 Projects Releases Packages Wiki Activity Actions

NixOS #26

New issue
Open
opened 2020-08-23 12:51:35 +00:00 by sbruder · 7 comments
sbruder commented 2020-08-23 12:51:35 +00:00 (Migrated from gitlab.jalr.de)
Copy link

After using NixOS for some days, I think it is the perfect fit for this project. The following is a WIP list of benefits and possible problems along with a TODO list of what has to be done if we want to switch to NixOS.

Advantages

Built for declarative configuration

This is quite general and is always an advantage when using NixOS. We make use of ansible and other automation technologies (Packer, GitLab CI) extensively so it makes sense to use a distribution that was meant to be used like that.

Built-in support for older versions

We currently use our handmade sbruder/tftpgen> to generate a nice looking list of all previous versions and branches. NixOS also comes with this feature built in (“generations”). It should not be too hard to modify this to include a list of other branches too (TODO @sbruder: actually figure out if and how this can be done).

More software directly available from the repository

Since nixpkgs have a lower hurdle for “normal” users to add new packages or change/update an existing package, more – especially specialised – software can be installed without downloading a tarball and placing it somewhere hoping it just works or installing a .deb meant for Ubuntu 14.04 on the latest Debian. When software is not available in the stable channel, NixOS makes it extremely easy to use the unstable channel for just one package (without it influencing the stability of the other packages).

Packages where this is the case:

  • Arduino (currently installed by unpacking a tarball to /opt/ and adding a custom .desktop file to the system)
  • Google Fonts (currently installed by unpacking the git master tarball to /usr/local/share/fonts/google/)

Easier to add custom packages

When the required package is not currently part of nixpkgs, nix makes it quite easy to create a derivation. Even though it requires you to learn an entirely new functional programming language, once you understand it, it should be easier to add custom packages.

This could also benefit the entire nix ecosystem since – for packages that are not strictly only relevant to us – we could add them to upstream nixpkgs.

Packages where this would make sense:

  • VisiCut (currently .deb is downloaded from GitHub releases and installed manually)
  • Firefox distribution configuration/extensions (we could either still only copy the json files or could create a reusable NixOS configuration option)

Faster and more efficient builds due to safe build cache

Currently, we always rebuild the whole system image due to packer and debian limitations (when caching, the build might become less deterministic). NixOS can safely reuse derivations it built once from /nix/store/ without it influencing if the build is deterministic.

First-class support for testing things in a VM

nixos-rebuild build-vm makes it extremely easy to test changes in a VM. The only problem is the lack of state which luckily is no problem for us since the affected systems have to be function without any state. This would clean up the complicated Makefile (see #23).

An opportunity to clean up cruft

Some things in this repository are not needed any more and switching to another system entirely forces us to clean that up which could result in an overall faster system/build process and more order.

Disadvantages

Migrating takes a lot of time

The current configuration consists (according to tokei) of 840 non-empty and non-comment lines of yaml (which is the ansible and packer configuration). To successfully migrate, we would need to adapt all of this (and everything that is not part of ansible/packer, especially the “core” labsync part that runs in the initramfs). While NixOS makes this easier, it still is a lot of work.

How it could be done

Like it is done currently

We could just take the current approach of building an image in packer and, creating a squashfs of the root and downloading it in the initramfs. This, however, means we could not use much of the added flexibility NixOS has to offer.

Using some of the nix tools

With nixos-generators we could use a tool optimised for NixOS instead of our manual approach to create a system image. This allows us to specify the filesystem configuration directly in the NixOS configuration. However, I am not sure if generations can be handled with this tool.

Dynamically activating the configuration

This is by far the most elaborate option. It is also the one that needs the most work to set up, but in the end it will be the one that is least “hacked together” and mostly just applied NixOS tools.

(Warning, I do not fully understand how the boot process works, so please correct me if I am wrong)
The core idea is that NixOS activates the current (or if selected, than an older) configuration on every boot. If the configuration was built once (which might need some time even when only using packages avilable from binary caches, e.g. because of fontconfig triggers etc.) and still is present including all dependencies it can be activated almost instantly. So if the CI (maybe even on raven, since we have a efficient build cache and network speed is more limiting) built the configuration once it will boot on every system like it built that configuration itself.

This however requires that every system has access to the nix store. This could be achieved by placing the nix store on a central sever (in the best case on an SSD, because otherwise this and the network’s speed could be a bottleneck) and setting up a read only NFS (or similar) share. The CI would have write access to the nix store and could add the new derivations to the store.

TODO

  • Do we want NixOS?
  • How do we want it?
    • Dynamically activating the configuration
      • Nix store distribution (NFS, Rsync, torrent?)
  • TBD
After using NixOS for some days, I think it is the perfect fit for this project. The following is a WIP list of benefits and possible problems along with a TODO list of what has to be done if we want to switch to NixOS. ## Advantages ### Built for declarative configuration This is quite general and is always an advantage when using NixOS. We make use of ansible and other automation technologies (Packer, GitLab CI) extensively so it makes sense to use a distribution that was meant to be used like that. ### Built-in support for older versions We currently use our handmade sbruder/tftpgen> to generate a nice looking list of all previous versions and branches. NixOS also comes with this feature built in (“generations”). It should not be too hard to modify this to include a list of other branches too (TODO @sbruder: actually figure out if and how this can be done). ### More software directly available from the repository Since nixpkgs have a lower hurdle for “normal” users to add new packages or change/update an existing package, more – especially specialised – software can be installed without downloading a tarball and placing it somewhere hoping it just works or installing a `.deb` meant for Ubuntu 14.04 on the latest Debian. When software is not available in the stable channel, NixOS makes it extremely easy to use the unstable channel for just one package (without it influencing the stability of the other packages). Packages where this is the case: * Arduino (currently installed by unpacking a tarball to /opt/ and adding a custom `.desktop` file to the system) * Google Fonts (currently installed by unpacking the git master tarball to `/usr/local/share/fonts/google/`) ### Easier to add custom packages When the required package is not currently part of nixpkgs, nix makes it quite easy to create a derivation. Even though it requires you to learn an entirely new functional programming language, once you understand it, it should be easier to add custom packages. This could also benefit the entire nix ecosystem since – for packages that are not strictly only relevant to us – we could add them to upstream nixpkgs. Packages where this would make sense: * VisiCut (currently `.deb` is downloaded from GitHub releases and installed manually) * Firefox distribution configuration/extensions (we could either still only copy the json files or could create a reusable NixOS configuration option) ### Faster and more efficient builds due to safe build cache Currently, we always rebuild the whole system image due to packer and debian limitations (when caching, the build might become less deterministic). NixOS can safely reuse derivations it built once from `/nix/store/` without it influencing if the build is deterministic. ### First-class support for testing things in a VM `nixos-rebuild build-vm` makes it *extremely* easy to test changes in a VM. The only problem is the lack of state which luckily is no problem for us since the affected systems have to be function without any state. This would clean up the complicated Makefile (see #23). ### An opportunity to clean up cruft Some things in this repository are not needed any more and switching to another system entirely forces us to clean that up which could result in an overall faster system/build process and more order. ## Disadvantages ### Migrating takes a lot of time The current configuration consists (according to [tokei](https://github.com/XAMPPRocky/tokei)) of 840 non-empty and non-comment lines of yaml (which is the ansible and packer configuration). To successfully migrate, we would need to adapt all of this (and everything that is not part of ansible/packer, especially the “core” labsync part that runs in the initramfs). While NixOS makes this easier, it still is a lot of work. ## How it could be done ### Like it is done currently We could just take the current approach of building an image in packer and, creating a squashfs of the root and downloading it in the initramfs. This, however, means we could not use much of the added flexibility NixOS has to offer. ### Using some of the nix tools With [nixos-generators](https://github.com/nix-community/nixos-generators) we could use a tool optimised for NixOS instead of our manual approach to create a system image. This allows us to specify the filesystem configuration directly in the NixOS configuration. However, I am not sure if generations can be handled with this tool. ### Dynamically activating the configuration This is by far the most elaborate option. It is also the one that needs the most work to set up, but in the end it will be the one that is least “hacked together” and mostly just applied NixOS tools. (Warning, I do not fully understand how the boot process works, so please correct me if I am wrong) The core idea is that NixOS activates the current (or if selected, than an older) configuration on every boot. If the configuration was built once (which might need some time even when only using packages avilable from binary caches, e.g. because of fontconfig triggers etc.) and still is present including all dependencies it can be activated almost instantly. So if the CI (maybe even on `raven`, since we have a efficient build cache and network speed is more limiting) built the configuration once it will boot on every system like it built that configuration itself. This however requires that every system has access to the nix store. This could be achieved by placing the nix store on a central sever (in the best case on an SSD, because otherwise this and the network’s speed could be a bottleneck) and setting up a read only NFS (or similar) share. The CI would have write access to the nix store and could add the new derivations to the store. ## TODO - [ ] Do we want NixOS? - [ ] How do we want it? * Dynamically activating the configuration + [ ] Nix store distribution (NFS, Rsync, torrent?) - [ ] TBD
sbruder commented 2020-08-23 15:40:46 +00:00 (Migrated from gitlab.jalr.de)
Copy link

changed the description

changed the description
sbruder commented 2020-08-23 15:42:28 +00:00 (Migrated from gitlab.jalr.de)
Copy link

According to The NixOS Wiki it is not possible to mount the nix store over NFS on NixOS. However, I’m not sure if this also applies when all but one client mount the store strictly read only.

The VM created by nixos-rebuild build-vm mounts the host’s Nix store read only. So it should be possible to share a nix store as long as only one machine writes to it.

According to [The NixOS Wiki](https://nixos.wiki/wiki/NFS) it is not possible to mount the nix store over NFS on NixOS. However, I’m not sure if this also applies when all but one client mount the store strictly read only. The VM created by `nixos-rebuild build-vm` mounts the host’s Nix store read only. So it should be possible to share a nix store as long as only one machine writes to it.
sbruder commented 2020-08-23 15:43:51 +00:00 (Migrated from gitlab.jalr.de)
Copy link

changed the description

changed the description
sbruder commented 2020-08-23 15:49:59 +00:00 (Migrated from gitlab.jalr.de)
Copy link

A boot entry in GRUB looks like this

menuentry "NixOS - Default" {
search --set=drive1 --fs-uuid f064339c-63c3-4393-acb3-10f895196894
  linux ($drive1)//kernels/s8vzfncncggfnndz1w5xk6xym240v5vj-linux-5.4.58-bzImage systemConfig=/nix/store/q3w2br2sgdf1xclav7gl0qswd3nq589a-nixos-system-nunotaba-20.03.2796.3506bce1d40 init=/nix/store/q3w2br2sgdf1xclav7gl0qswd3nq589a-nixos-system-nunotaba-20.03.2796.3506bce1d40/init loglevel=4
  initrd ($drive1)//kernels/yiivf5519py899x2zsxd7z88jpk3mv88-initrd-linux-5.4.58-initrd
}

This means it is possible to distribute the kernel and initrd over http (like we currently do), however I’m not sure if pxelinux supports bzip. Also, since the kernel and initrd (obviously) aren’t in the nix store it is easy to serve them over http.

The systemConfig parameter (and the init parameter, which basically is ${systemConfig}/init) is what identifies the generation to be booted.

A boot entry in GRUB looks like this menuentry "NixOS - Default" { search --set=drive1 --fs-uuid f064339c-63c3-4393-acb3-10f895196894 linux ($drive1)//kernels/s8vzfncncggfnndz1w5xk6xym240v5vj-linux-5.4.58-bzImage systemConfig=/nix/store/q3w2br2sgdf1xclav7gl0qswd3nq589a-nixos-system-nunotaba-20.03.2796.3506bce1d40 init=/nix/store/q3w2br2sgdf1xclav7gl0qswd3nq589a-nixos-system-nunotaba-20.03.2796.3506bce1d40/init loglevel=4 initrd ($drive1)//kernels/yiivf5519py899x2zsxd7z88jpk3mv88-initrd-linux-5.4.58-initrd } This means it is possible to distribute the kernel and initrd over http (like we currently do), however I’m not sure if pxelinux supports bzip. Also, since the kernel and initrd (obviously) aren’t in the nix store it is easy to serve them over http. The `systemConfig` parameter (and the `init` parameter, which basically is `${systemConfig}/init`) is what identifies the generation to be booted.
sbruder commented 2020-08-23 16:00:57 +00:00 (Migrated from gitlab.jalr.de)
Copy link

Getting network and mounting NFS (or doing what else is needed to make the nix store available) can be done by setting boot.initrd.network.enable = true, setting the ip parameter of the kernel (like we currently do) and then add the commands for making the store available to boot.initrd.network.postCommands.

Getting network and mounting NFS (or doing what else is needed to make the nix store available) can be done by setting `boot.initrd.network.enable = true`, setting the `ip` parameter of the kernel (like we currently do) and then add the commands for making the store available to `boot.initrd.network.postCommands`.
sbruder commented 2020-08-25 07:47:45 +00:00 (Migrated from gitlab.jalr.de)
Copy link

From nixos-rebuild(8)

--profile-name, -p
Instead of using the Nix profile /nix/var/nix/profiles/system to keep
track of the current and previous system configurations, use
/nix/var/nix/profiles/system-profiles/name. When you use GRUB 2,
for every system profile created with this flag, NixOS will create a
submenu named “NixOS - Profile 'name'” in GRUB’s boot menu, containing
the current and previous configurations of this profile.

For instance, if you want to test a configuration file named test.nix
without affecting the default system profile, you would do:

        $ nixos-rebuild switch -p test -I nixos-config=./test.nix

The new configuration will appear in the GRUB 2 submenu “NixOS -
Profile 'test'”.

This could be used for feature branches.

From `nixos-rebuild(8)` --profile-name, -p Instead of using the Nix profile /nix/var/nix/profiles/system to keep track of the current and previous system configurations, use /nix/var/nix/profiles/system-profiles/name. When you use GRUB 2, for every system profile created with this flag, NixOS will create a submenu named “NixOS - Profile 'name'” in GRUB’s boot menu, containing the current and previous configurations of this profile. For instance, if you want to test a configuration file named test.nix without affecting the default system profile, you would do: $ nixos-rebuild switch -p test -I nixos-config=./test.nix The new configuration will appear in the GRUB 2 submenu “NixOS - Profile 'test'”. This could be used for feature branches.
sbruder commented 2020-10-09 13:58:36 +00:00 (Migrated from gitlab.jalr.de)
Copy link

https://grahamc.com/blog/erase-your-darlings

https://grahamc.com/blog/erase-your-darlings
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
use-quay.io-registry-test3
use-quay.io-registry-test1
use-quay.io-registry
debug
remove-cura
bump-arduino
fix-ansible-provisioner
inkstitch
license
fix-tty-keyboard-layout
cura-3.5.1
ldap
No results found.
Labels
Clear labels   
bug

   
feature

   
help-wanted

   
question

   
software

   
style
No labels
bug
feature
help-wanted
question
software
style
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: fablab-nea/labsync#26
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?