Need Procedure for Host Server Booting from ZFS and Hosting with KVM

I just got my new book “Mastering Ubuntu Server, 3rd Edition”, and now I’m getting ready to set up a new server. My background is with Microsoft Servers. My old host server is based on Hyper-V and, of course, NTFS.

I love everything I’m seeing about the ZFS file system. It seems clear that ZFS has a bright future, and rather than convert to it in the future, I hope to start out with it, including booting from it. I most want it’s redundancy, reliabiltiy, and recoverability.

After much research, I haven’t found a procedure that makes sense to me for installing Ubuntu Server to boot from ZFS 2.x. It seems such a procedure would require building a bootable USB drive with ZFS drivers. The disk interfaces are hosted on the motherboard directly, and not via a RAID controller.

My goal is to build a host server, that boots from ZFS raidz 3, and runs KVM to support my virtual machines. Here is my hardware:

CPU: AMD EPYC Rome 7262, 8 core
Motherboard: SuperMicro MBD-H12SSL-NTO
RAM: Nemix 8x16GB (128GB) DDR4-3200 ECC
HD: 6 x 4TB WD40EFZX Red Plus
NVMe: Corsair MP600 Pro 1TB M.2 Gen4 PCIe x4

Your advice and directions are most welcome.

My recommendation would be to use Proxmox. I’m not overly versed in XCP-ng, so I can’t comment there but @Jay and Tom Lawerence talk about it a good bit.

@Jay has a series on installing Proxmox already, and I’ve heard there is an updated version coming at some point - Setting up your own virtualization server with Proxmox Virtual Environment

If you’re going to virtualize something like TrueNAS, you’ll want to pass through your HBA (if you have one, e.g. amd_iommu=on in the grub config). If you don’t have an HBA, I’d recommend you get one rather than using the onboard SATA ports. ZFS likes to work at the device level (directly at the disk), and that’s where HBA’s shine.

Even if you’re not going to virtualize a NAS, you would still benefit from an HBA, especially with a system of that caliber. I have an LSI 8x HBA on my 2700X host, and it works really well with Proxmox, no complaints at all.

I didn’t see an affiliate link in Jay’s Store for an HBA, so, I’ll just stick an Amazon link up for it. This is the card I have.

You’ll need 2x of the cable harness.

As for booting ubuntu server from zfs. There is a good write-up at Ubuntu 20.04 Root on ZFS — OpenZFS documentation It seems really complicated.

On the other hand, ubuntu-desktop 21.04 has an option for booting from zfs in the installer as described at https://ubuntu.com/blog/zfs-focus-on-ubuntu-20-04-lts-whats-new . I could only get it to set up a single disk zpool when I tried it. Maybe I missed something. I am doing my testing on a virtual machine to which I assigned multiple disks.

1 Like

I’ve gone both routes, ZFS and BTRFS on bare metal installs. In both cases, I had marginal success with it. However, running VM’s on / through a hypervisor has been rather painless. That’s why I recommended putting Proxmox on it, then test what works and what doesn’t for various use cases. Either, way, it still has the underlying benefit of ZFS, just without all the hassle.

Edit: unless of course, one wants to become a ZFS Guru (of which I do not), then, all bets are off :slight_smile:

1 Like

dfarning,
thanks for taking the time to reply. I wanted the LTS version, but in this case, your suggestion for v21.04 may be the right thing to do.
The procedure I expected to find is to first boot to USB install media, and make my first install of Ubuntu Server to an External USB harddrive. That install will be used to create a ZFS array with 6 internal drives. Then, the final install of Ubuntu Server might be successfully made to the ZFS array. However, I’ve not seen any kind of procedure write-up that describes this process, and so I had hoped someone might know of one.
I intend to use Ubuntu Server. I do not intend to use Proxmox or other solutions.

@WynnSmith
I believe the write-up at Moving Ubuntu to Root-on-ZFS | Ivan Tomica describes what you are trying to do.

  1. Install ubuntu on a standard drive.
  2. Create zpool with remaining drives.
  3. Copy system from installed drive to zpool
  4. Fix things that broke

Sorry,
When working with ZFS on Linux you are pretty much at the bleeding edge.

I searched on this one for a couple of hours in between other activities. While I found a few “native methods” of doing what’s being asked here, none I would consider production worthy.

I did find a number of “somewhat reasonable” migration how-to’s that describe moving an existing ext3/4 install to Root on ZFS and some of the better documentation comes from OpenZFS Docs themselves. Even Ubuntu points in their direction with last years update.

It begs the question, if neither OpenZFS nor Ubuntu are pushing / promoting ZFS on Ubuntu server natively yet, why would they not?

KI7MT, thanks for your reply. ZFS is properly considered experimental. I would not yet recommend root-on-zfs for commercial installations. Yet, both it’s current status and future promises seem to place it far beyond any other file system. While this is my personal mission-critical system, it’s for my home where I don’t want to run multiple systems. I want this box to become my one and only system. For now, I will continue to use the Microsoft Server while I experiment with the new setup and ZFS’s native redundancy, replication, snapshot, and backup features.
BTW, there is no need to add a HBA disk controller. The Supermicro motherboard supports 16 drives via it’s two SFF-8654 interfaces. I will begin with 6 drives, hopefully configured with raidz3. I expect this will hold me until a future version of ZFS enables the ability to add drives one at a time.

1 Like

WynnSmith, I would think ZFS will make it’s way to prime-time at some point, but, it doesn’t seem to be there just yet. Regarding the SFF interface / HBA, obviously device count is not an issue. If I looked at the right board (the NT-O version), there seems to be plenty of PCI express lanes, both 8x and 16x. More than likely, unless you really loading it up with PCI devices (Vid Cards, Sound Processors, etc), should be ok and have expansion capability as well.

Yes, holding off on running zfs on ubuntu server is probably a good idea unless you are:

  1. A file system developer.
  2. Setting up a test system to check on the progress of zfs on Linux.
  3. A crazy person.

If you are a file systems developer, please seek help from a mental health professional. If you are testing or a crazy person, you are in the right place.

1 Like

ZFS definitely works, and works very well. It all depends on the work loads and how you want them configured. Here’s one of my Proxmox nodes (the one that has the HBA in it).

  • /dev/sd{ a … h } are on a RAIDZ2
  • /dev/sd{ i … j } is a ZFS Mirror
  • /nvme{ 1n1… 0n1 } is a ZFS Mirror

However, on the NVME’s, there’s two partitions that are non-ZFS (BIOS Boot, and EFI). The root is on ZFS.

root@2700X:~# lsblk -e7
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0   1.8T  0 disk 
├─sda1        8:1    0   1.8T  0 part 
└─sda9        8:9    0     8M  0 part 
sdb           8:16   0   1.8T  0 disk 
├─sdb1        8:17   0   1.8T  0 part 
└─sdb9        8:25   0     8M  0 part 
sdc           8:32   0   1.8T  0 disk 
├─sdc1        8:33   0   1.8T  0 part 
└─sdc9        8:41   0     8M  0 part 
sdd           8:48   0   1.8T  0 disk 
├─sdd1        8:49   0   1.8T  0 part 
└─sdd9        8:57   0     8M  0 part 
sde           8:64   0   1.8T  0 disk 
├─sde1        8:65   0   1.8T  0 part 
└─sde9        8:73   0     8M  0 part 
sdf           8:80   0   1.8T  0 disk 
├─sdf1        8:81   0   1.8T  0 part 
└─sdf9        8:89   0     8M  0 part 
sdg           8:96   0   1.8T  0 disk 
├─sdg1        8:97   0   1.8T  0 part 
└─sdg9        8:105  0     8M  0 part 
sdh           8:112  0   1.8T  0 disk 
├─sdh1        8:113  0   1.8T  0 part 
└─sdh9        8:121  0     8M  0 part 
sdi           8:128  0 223.6G  0 disk 
├─sdi1        8:129  0 223.6G  0 part 
└─sdi9        8:137  0     8M  0 part 
sdj           8:144  0 223.6G  0 disk 
├─sdj1        8:145  0 223.6G  0 part 
└─sdj9        8:153  0     8M  0 part 
nvme1n1     259:0    0 931.5G  0 disk 
├─nvme1n1p1 259:1    0  1007K  0 part 
├─nvme1n1p2 259:2    0   512M  0 part 
└─nvme1n1p3 259:3    0   931G  0 part 
nvme0n1     259:4    0 931.5G  0 disk 
├─nvme0n1p1 259:5    0  1007K  0 part 
├─nvme0n1p2 259:6    0   512M  0 part 
└─nvme0n1p3 259:7    0   931G  0 part 
root@2700X:~# 

Datacenter View of Storage

Here’s the NVME Partitions

Here’s the node info

Just for perspective:

Proxmox sits ontop of a Debian Server. OpenZFS has a Root on ZFS guide for Buster & Stretch, but not Bullseye (the Latest used by Proxmox 7).

If Ubuntu Server ZFS support is not ready for prime-time, maybe using Debian Server could suffice in the interim.

In terms of LTS, Debian is definitely LTS oriented. In fact, many get frustrated at how slow the release cycles are, but that is a discussion for another time.

Also note, as most know, Ubuntu is a direct descendant of Debian, and Ubuntu server is no exception.

Release Info from my 2700X Host Node above

root@2700X:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"