LVM RAID 1 - recover from drive failure?

1

Hi,

I recently got a new hard drive and I was planning on using it to setup a simple RAID 1. I don’t have a home lab or anything, so I wanted to keep things as simple as possible and figured that using LVM for this would be a good idea. Setting this up was pretty straight forward, and it worked fine. But before committing to use this long term, I wanted to test how would I recover my files should something happen to one of the drives.

When I unplug one of the drives things continue to work normally, I can access the files in the volume, etc. But when I unplug both drives, and then plug only one of them, things start to get messy very quickly. I only managed once to regain access to my files by running pvscan and then expanding the volume group to include the new drive using vgexpand. The rest of attempts went poorly as I tried using things like lvconvert --repair and similar.

Running lsblk shows that something is not right. Here, only one of the drives that I used to setup the RAID 1 is plugged in, /dev/sda, but it shows up as a regular drive. At the same time, it shows the logical volume mirrors there but I can’t mount it.

NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                            8:0    1  57.6G  0 disk
└─sda1                         8:1    1  57.6G  0 part
vg_backup-lv_backup_rmeta_1  254:2    0     4M  0 lvm
└─vg_backup-lv_backup        254:4    0     1G  0 lvm
vg_backup-lv_backup_rimage_1 254:3    0     1G  0 lvm
└─vg_backup-lv_backup        254:4    0     1G  0 lvm
nvme0n1                      259:0    0 232.9G  0 disk
├─nvme0n1p1                  259:1    0   512M  0 part /boot/efi
├─nvme0n1p2                  259:2    0 231.4G  0 part /
└─nvme0n1p3                  259:3    0   977M  0 part [SWAP]

From what I’ve found out it seems that the expected course of action would be to replace the drive as soon as possible. This makes perfect sense but of course I’d also like to be able to recover my files quickly. Does anyone have any recommendations or resources on how to do this? I know that things like BTRFS and ZFS exists but I’m a slow learner and I really want to keep things simple. Eventually, I might want to try a bigger setup where this may make more sense.

Thanks!

2

Repeat after me: RAID is not a backup!

To be expected. If both drives fail, you’re SOL. In a highly available file system, you’re not supposed to ever have everything unplugged at once. I’m surprised your system didn’t kernel panic, or that it kept working.

If you unmount the rootfs, unless you’re using a diskless distro (like alpine, that loads everything in RAM and / is a tmpfs), then your system should stop responding, because it wouldn’t be able to find bare basic commands anymore (because /bin and /sbin are unmounted).

IMO btrfs is simpler and easier than LVM. I prefer ZFS and I won’t run my systems on anything else, but getting it set up is a bit more complicated, unless you run a distro that supports it OOTB.

I’d say next action for you, if you don’t want a homelab and get an Odroid HC4 and use it as a NFS server to backup your files, at the very least buy an external USB drive and connect it to your PC from time to time to run backups (don’t leave it plugged-in at all times). You can use something like restic (that’s what I use over NFS), timeshift, duplicati, deja-dup, urbackup or other simple backup tools.

It’s better to have backups than a redundant file system (although having both is nice). But if this is your desktop and not a server, redundant FS is overkill, unless you care about data integrity (which only ZFS offers, because of its checksums and scrubs).

3

RAID is not a backup! :smiley:

I probably should’ve mention that I’m not using this for my root file system, just as a external drive for extra storage for my laptop while I’m on the move. That’s where me unplugging things comes from: I need to be able to unplug these drives, go somewhere else, plug them back in and carry on.

But in case that one of the drives gets damaged on the way or I lose it, I’d like to be able to plug the other one and upload its contents first and foremost, before worrying about replacing the failing/missing drive.

I’m currently running on Debian 12 on EXT4. Would BTRFS really be a better fit for this? I haven’t checked in a while but I was under the impression that RAID was problematic with BTRFS. Also, quick question, if I chose another filesystem would I also need to use it on my laptop or only on the drives?

Thanks!

4

Yes, should work fine on debian.

Parity RAID, yes. If you go for mirrors, it’s fine.

You can use any combinations of FS on any number of drives. You can have root on xfs, var/log on ext4, a zfs pool on /tank, a btrfs mirror in /mnt/backups and so on.

1 Like
5

That makes wonder though, what are some practical use cases for LVM? It seems to have many features, but also dedicated alternatives for just about any of them.

6

When you are not dealing with ZFS or BTRFS, LVM is great actually. It has a bunch of features (which the others have too, but LVM allows for just blk devices, although zfs does too with zvols).

LVM can be combined with any simple file systems, like ext4, xfs, jfs and so on + it can be combined with md for RAID (if you aren’t using LVM itself for RAID - I think md has better performance AFAIK, there was something about LVM “RAID” that is not parallelized on multiple devices, kind of like pooling instead of proper balancing) and with LUKS (or any other disk encryption tool).

LVM is used for easy partitioning and has support for snapshots (although janky). For example, instead of doing 10 partitions on a single disk (and keep in mind old DOS partition tables only support up to 4 primary partitions, after which you need to make logical partitions), you can do just 2 or 3 partitions and then manage volumes via LVM (which literally stands for logical volume manager).

So instead of sda1 for boot, sda2 for rootfs, sda3 for home and sda4 for swap, you can have sda1 for boot and sda2 for lvm, then on lvm, have everything else (rootfs, home, swap, var, var/log, opt etc.).

LVM has support for thin-pools (which basically means you can create logical volumes, but it won’t automatically use all the disk space). IDK how well thin-pools work with snapshots, but if they do, they’d work wonderful, because LVM snapshots need free space in-reserve on the “volume group” (an LVM term).

You have “physical volumes,” which means basically physical devices, no matter if they are a SATA disk or an iSCSI LUN, then you have “volume groups,” which is the place where all the physical volumes congregate to make a single large “pool of storage” and finally logical volumes, where you partition the big pool.

That means if you want snapshots, you can’t allocate all the VG space to LVs, unlike ZFS and BTRFS, which will by default make everything thin-provisioned and the whole pool storage is used for snapshots.

I find LVM particularly useful in VMs that need to be always available. In VMs, ZFS doesn’t make sense (IMO), because ZFS should be the one accessing the physical devices, in order to function properly (which is something you’ll hear often when you look online for ZFS and HBA cards, as opposed to using RAID cards with ZFS).

What I do is, when I configure a VM, I set up LVM for everything except /boot/efi and /boot. Then, if I ever need to increase the disk size, I can enlarge the disk in the hypervisor (allocate more space). Then I increase the size of the VG and then add more space to any LV, then use resize2fs (for ext4) or xfs_grow (for xfs) to increase the partition size live.

If you were to do just do normal partitions on a GPT partition table vdisk (which supports many partitions, unlike old DOS partition table), you would need to shutdown the VM, boot a live CD and then use something like gparted to resize the partition - basically deleting a partition in the partition table, recreating it with a larger size, then running resize2fs or xfs_grow, but all in an easy to use GUI. And if your partition is not near the free space, you’d first need to move a partition and its data, then increase the size of the partition you need.

No need to go through that hassle with LVM. I would not dismiss LVM for certain tasks, it’s a useful tool. But for physical hardware, ZFS is where it’s at (unless you do clustered file systems, then ceph).

1 Like
7

Thank you very much for the explanation. I’m very new to this side of Linux as I’ve never used LVM or anything related to file systems or similar.

So, now I’m curious about BTRFS and XFS. But I still need to know if they suit my use case. As I mentioned, this isn’t for a permanent setup but for my laptop while I travel. A single external drive works perfectly, but I’d like to add a second one that acts as a mirror, for a little extra safety.

LVM I tested with two USB drives. The setup was easy and the mirror worked fine, until I unplugged both drives and then plug only one of them. This would simulate a scenario where I lost one of the drives for whatever reason. In that case, it was really difficult for me to gain access to the files inside the “remaining” drive. I tried so many things that I don’t remember what worked, actually…
Maybe I was just using LVM in a way that it was not intended for. A lot of articles that I found online, including Red Hat’s documentation on LVM, address failing drives by immediately replacing them.

My question now is: what is the easiest way to gain access to the files in the remaining drive, without restoring the mirror? Will formatting the drives using BTRFS or XFS help me with this? I would also appreciate a good beginner-friendly resource to learn about these file systems. As I said, I’m still quite new to all of this.

And thanks again for the help! :pray:

8

BTRFS is fine. I wouldn’t use XFS as a general FS. It does well even generally, but by default, it’s great for very large files (like for databases). Of course, all file systems are tunable, but xfs shines with big files. I wouldn’t run it for a laptop.

For LVM? No clue, I never used LVM like that. You can combine multiple physical volumes in a large volume group, but that is not a mirror (it would count as RAID0 at best, but it actually functions more like extended data than splitting data equally between the two PVs).

If you do btrfs mirror, you should always have access to at least one drive. I’m not sure how btrfs handles plugging one disk at a time though. To me, resilvering a pool every time you want to read the data on another device sounds horrible. I hope btrfs doesn’t do that, but you’d have to test it.

I don’t have knowledge on portable setups. Except for usb sticks and external hdds, all I plug into a PC is always done offline (powered off).

1 Like
9

I’ll have to try BTRFS and see if I can get it to work for me. Although I’m starting to suspect that running rsync when both drives are connected will be a lot simpler :smiley:

Thank you for the help!

10

For certain. It does count as a backup and doesn’t have the same drawbacks as working with both drives live (if you delete a file on one, when running mirrors, the file will be deleted on both drives - but with rsync, you’ll be able to grab a delete file from the other).

1 Like
11

So, I thought I would spend some time over this weekend looking into LVM (I just want to make it work! :smiley: ) but I’m more and more convinced that a good old rsync is a better choice no doubt. I’m probably going to look at BTRFS soon though, I’ve read a bit about it and it seems to check all the boxes that I need.