I don’t like the idea of persuading people to use something else, but lately I’ve been telling people to choose whatever, but to stay away from anything RHEL-based (unless absolutely necessary) and for a long time, to absolutely stay away from manjaro.
I don’t just change my defaults either. I’ve had bad experiences with arch over the years myself, some were arch problems, some were hardware, but that other distros would’ve very likely handled better, but I haven’t given it another chance. My opinion could get changed, but I never made use of the AUR (most of the installs I ran were “pure” and kept up to date and still borked themselves - although all were desktops config, never tried it as a server), so why even use arch then? That’s just me, if it works for you, keep at it. Changing your config just for sake of change is silly, unless you have a goal in mind (like learning or getting a highly paid job that requires nixos knowledge - although by that point it won’t be "just for the sake of change).
If you have many machines to manage and want to make sure everything is consistent (changing SSH keys and passwords often, maybe once every blue moon add a new software that you want deployed on all systems etc.), then nixos is the way to go. If you don’t care about that, maybe nix isn’t for you and that’s fine. Nix does have a lot more things up its sleeves and flakes are just one of them. I haven’t delved too deep into it, but lately I’ve been dealing with microvm.nix. This allows you to run a minimal nixos VM using technologies like firecracker, which saves a ton of resources, gives you the advantage of containers, but has a full kernel (so it’s a micro VM in the literal sense). You can define your VMs in a config file and even if the full system somehow gets screwed (all by itself I mean) and you have your user data (say a DB dump and a few documents), then all you need to do is reinstantiate the nix VM and everything will be rebuilt exactly the same, no need to recover from backups (so you don’t even need backups as large to get the whole OS recovered).
This is similar to how OCI containers work, but much more refined and applies to an entire OS, not just a container OS. But nix comes with yet another ace. Every installed program version is definable. Unlike OCI containers, which will not always recreate the same system, nix will. Nix goes way further for reproducible builds. When you run a docker build based on some program versions you define, like mysql 8 and nginx 1.25.1 today, your system (particularly the OS you choose, say alpine) will not build the same today as it will in 5 weeks from now. NixOS will. The best part about it? You can run nixos in an OCI or linux container (say, replacing alpine with nixos) and get the benefit of nix without dropping your currently preferred stack. You just need to add the stack definitions.
The above is a highly sought for enterprise feature. For example, I think the library of congress in the US were running older RHEL and trying to relaunch a broken mysql db, but compiling the precise mysql version and whatever else was difficult. So they just switched to nixos and recovered, managed to launch the db and get the data out of it. Probably there are more example like these.
What does it mean for the home user? Nothing really, since most home usage will have the systems running close-to or actual bleeding edge. Again, I’m not trying to convince you to switch.
As for ansible, I find it to be a worse version of nixos. Ansible doesn’t have generations like nix does. Let’s say you deploy ytop to 3 systems using ansible. And then after a while, you find out it’s deprecated, so you want to remove it. Let’s say 1 of these 3 systems is down.
How would you remove it in ansible? Just deleting the entry to install ytop from ansible is not enough. You have to specify “uninstall ytop” in your playbook. It goes to the 2 systems and uninstalls it. Then you remove the line “uninstall ytop” to keep your playbook clean. Then later on, you start your offline system, but didn’t realize it also had ytop installed, so now whenever ansible runs, it will not uninstall ytop, because you removed the definition from the playbook. You have to add it again. And you want to keep your playbook minimal to avoid redundant things like install ytop, then uninstall ytop on a fresh system.
How this can be achieved in nix? You just remove the package entry “nixos” and your whole system gets regenerated, without ytop. Think of a generation as a system snapshot, that’s always recreatable. You pushed your nixos config to the 2 systems, they generated a new generation without ytop and are running it. You power on your last system and you notice ytop is still there, meaning the configuration.nix (or whatever config you keep for just programs, there are good reasons for compartmentalizing) has not been changed. just push your latest configuration.nix to the last system and you’ve got the last system running the exact config as the other 2.
With ansible, you manage the state of the system. With nix, you define the state and nix recreates it entirely.
Some more examples about ansible vs nix
- 2023-01-01 add “install ytop neovim tmux”
- 2023-01-07 add “install nginx”
- 2023-01-08 add “install mysql”
- 2023-02-12 add “install docker”
- 2023-02-13 add “configure docker”
- 2023-02-14 add “uninstall nginx mysql”
- 2023-04-01 add “install zabbix-agent”
- 2023-09-11 replace “install neovim tmux” add “uninstall ytop”
In the above, you got changes that you add to the playbook. It’s messy and you want to clean it up, because you believe all the the systems have the same(-ish) state. So:
- 2023-10-01 remove “install nginx” “install mysql” “uninstall nginx mysql” “uninstall ytop”
Like in the previous example, now you realize that a system failed to run the playbook, for whatever reason. Maybe the ssh key got changed and you didn’t realize ansible was giving an error. Or maybe it was down for maintenance. Doesn’t matter. Now one of your system is out of sync and is running the previous state from 2023-04-01.
With nixos, you define the state, everything gets nuked at it runs the final state definition “install docker,” “configure docker,” “install zabbix-agent,” “install neovim tmux.”
With nixos, you can get even more insane and make the system immutable.
https://grahamc.com/blog/erase-your-darlings
https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/
The first guide is what made me want to switch to nix. The second one is the one I’m actually running on the few systems I have that run nixos. I’ve got a /nix/persist folder in which I keep all my user data and persistent config files (like configuration.nix, hardware.nix and custom.nix) and I define the pointers to them in the nix config.
I had a similar config with diskless alpine install. You can just back up the lbu ovl file and slap it on another system, but it’s not as refined as nixos and you’re also backing up the things like binaries, which is just cruft.
I was thinking, later in the future if I keep digging into nix, to have a single “main system” with the nix-store sharing it via NFS to systems that are in read-only mode with the nix-store (well, nix-store by default is read-only, but what I mean is that the slave systems would not be able to install software in the store themselves).
This is inspired by the gentoo NFS wiki, but I’d have to manage multiple generations, one for each system I have and make sure I don’t nuke one of the running generations. I already have delved into tftp pxe booting and root-on-nfs systems, but adapting it to nix would certainly be difficult.
https://wiki.gentoo.org/wiki/Diskless_nodes
This was just to showcase what cool (and useful) things you could do with nixos.
Back to nix flakes. I mentioned the microvm.nix, but there are other flakes, like dns.nix (running bind using a defined nix config, a bit gimmicky, but I suppose I could see uses for it - and it does solve the insane DNS zones configs, with the nix language structure, which looks like json). There’s NixOps (which is a nixos deployment tool, similar to terraform, but from what I’m reading, not as good as terraform), containers.nix, qemu-vm.nix, virtualization.nix (this one is part of nixops I think). The list goes on. It’s basically like a way more advanced AUR, people come up with all kinds of new ways of running and deploying software.
I’ve been running nixos on 2x raspberry pi 3s. Initially I was testing pi-hole in lxc on another distro (which failed, because I couldn’t bridge a wifi connection), then I tried docker (because it was easier to NAT, it’s just the default - I failed to NAT the wifi to the lxc network). But at some point the SD on one of them got corrupted when I plugged something into the same multi-port USB brick and cut the power to one of the pis.
Then when I switched to nixos (as I already had it on other SBCs, like my HC4) and wanted to have the same easily replicable system, because I couldn’t be bothered to back up docker containers for something that doesn’t even keep traffic logs - I was running in anonymous mode and the /var/log was tmpfs). I found that adguard home and blocky were present in the nix.pkgs. How cool is that? I just switched to blocky running straight on the system and couldn’t be happier. I could’ve slapped armbian on the pis and run the pi-hole install script, but then I’d have to again configure the user, hostname, wifi connection and so on. A royal PITA for me. NixOS does things much better.
As I’m expanding my home lab again (I was forced to start from scratch a few years ago), I’m looking for ways to save time, as I can’t seem to find the time to work on my homelab as often (neither the energy).
(cutting off the irrelevant part)
Probably part of the reason was the availability of ready-made meals all around me (a bunch of canteens and bakeries), but since I moved, I cook most of my meals (since I’m in almost the middle of nowhere, although there are restaurants, but I don’t like sitting at a table, I preferred grabbing my meal for later and getting 3 or 4 meals in one go - I also don’t like waiting for the food to be cooked, if that’s the case, I can just do it myself).
NixOS saves me time. I don’t have to think about how I install my next project, I just edit my custom.nix config and send it to the live nix iso along with the generic configuration.nix (unless it’s a VM, I just generate a new hardware.nix file on the system, it’s just one command).
(getting into more irrelevant stuff, this time tech related)
The thing I am struggling in my homelab with is keeping everything low power consuming. I’m running mostly ARM SBCs around here, so I can’t just slap a VM and call it a day. This got me into LXD and I really liked it, but I was having trouble with how to deploy the containers. I wanted to put them on either a block-device via iscsi, or one container image in its own ZFS dataset shared via NFS, to be able to easily run snapshots on the containers. I wanted to make something that is easily live migratable between other SBCs by just shutting down the container, moving the NFS / iscsi and starting it back up (or even better if this could be live migratable with CRIU).
I couldn’t figure out how to implement this and then canonical took over lxd back, so I’m currently in a stand-still. I’m waiting for Incus to pop-up and I’ll see if that use it instead. I’m not running nixos on everything in my homelab, in fact it’s quite a minor part of it. It might change if I can figure more about it and how to save resources. Firecracker is promising, but I’m also messing with OpenNebula to make any distro firecracker VMs, so I can more easily manage them (and probably be able to slap them on iscsi).
Ceph seems like the ideal system, but I don’t want to invest in 5 SBCs to run it in the recommended configuration (testing ceph in VMs ATM, which I have to shutdown, because it requires quite the resources).
Besides my blocky Pis, I’m running absolutely nothing in my homelab at the moment (except if you count my diy arm router and just my bare basic diy arm NAS serving NFS and iSCSI). I want to run some form of git site (and I’m struggling between choosing gitea or stagit), one static site generator (jekyll, hugo or rgzee ssg6), some monitoring software (zabbix or again prometheus + graphana, but this time add alert-manager if I do implement it), NUT, some form of static down-detector web page (uptime kuma or homer, but static and without JS, I can probably do it myself using zabbix or prometheus stats) and maybe I’ll run a mail server (connected to a VPS, so I don’t randomly change my IP address, although I’m not fond of the mail server management aspect, it’s a PITA).
Just picking one technology and sticking to it (like just using proxmox or docker) doesn’t cut it for me right now, because I’m aiming for both resume score points and for eventually running everything off of a 12v system. I got some of my homelab on USB C power delivery and I’m sure I can do more, but right now I don’t want to shut off things like my NAS or router.
This got too ranty, I’m sorry.