Am new to Proxmox and i have a question regarding the optimal storage configuration for what i have.
Am building a home server to use as a media server (Plex, Nzbget, Sonarr etc), run PiHole, HA and a few other things.
How should i configure the storage? I currently have:
1x500GB NVME SSD
2x4TB HDDs
1x16TB HDD - planning to use this for the media share, don’t require redundancy
Should i install Proxmox on SATA with ZFS1 and host the VM’s on the NVME? Or should i install Proxmox and host the VM’s on the NVME and use the other hdds as storage? Do i need to be aware of anything when installing Proxmox on an SSD so it doesn’t wareout quickly?
How would you configure the storage with the drives i have?
I don’t think you will get a really definitive answer. OpenZFS, which used to b called ZFS on Linux, is evolving rapidly. So, a lot of the information that you find online will likely already be out of date.
To make things more complicated, it seems the loudest voices in the ZFS world are ‘systems lawyers’ who like to argue about obscure corner cases and ‘optimizers’ who are trying to tweak the last little bit of performance out of their systems. As a result, online discussions seem to peter out before coming to a conclusion.
A second issue is that ZFS is very new to proxmox so the GUI interface is very limited.
For the last couple of months, I have been running a similar situation to you. I had the best luck installing proxmox on the NVME. I run the VMs and containers from that drive. I have tried both ext4 and zfs with similar results.
Then, I attach my spinning disks as ‘storage directories’ within the proxmox gui.
For my home lab, a 500GB NVME is more than enough space for the host os and my ~ 20 VMs. I haven’t seen any problem with excessive wear out on the SSD with my pretty standard home lab type of services.
All that being said, I am in the process of building an experimental server that will have 2X4TB SATA HDDs in a ZFS mirror with a 256GB NVME SSD as a cache. I’ll report back with the results of my testing as I figure things out.
Am going down the same route as yourself - installing Proxmox and hosting the VM’s on the NVME and using the other disks as ‘storage directories’. Hopefully this will suffice for now, which by the sounds of it, it will.
Hope your experimental server goes as planned, and please do let me know how it goes. I was reading about using an SSD as a cache but i think thats a bit further down the line for me
That has been working really well for me in my little home lab. It has the additional advantage that it is really easy to back up the VMs and containers to an NFS dir or external drive in order to reinstall Proxmox if you are into experimentation.
My experience with ZFS on Proxmox has not been very good. It can be very useful, but I don’t feel that Proxmox manages it well. Sometimes, it’s hard to know how much memory is truly free, and VMs can lock up even though it appears you have a lot of resources free.
Of course, rough edges like that can be fixed if I put a little bit more work into it, but I removed ZFS on my end and decided I would try it again later on when they implement new features.
I agree the Linux implementation of OpenZFS is very new and evolving rapidly. As are the interfaces programs such as Proxmox use to interface with the zfs file system. Lots of rough edges.
I am pretty excited about TrueNas Scale. Hopefully, it will introduce a lot of new users and developers to the Linux implementation of ZFS.
One of the main reasons, I chose the new server over other options is the ability to run a virtual version or proxmox for testing.
I am not sure how much you have interacted with the Proxmox forums. It is pretty interesting.
It is not unusual for someone to ask a good question.
A developer answers.
The person who asked the question wonders if they should file a bug
The deveoper replies with, “no thanks I have already filed a patch.”
What are you using to store your bulk data? I have spinning disks that I’d like to set up in a NAS but after trying out virtualized truenas with pass-through scsi drives, it left a bad taste in my mouth (lost 30TB+ of data / plex set-up so I can just re-hydrate but will take a long time as re-hydration needs to happen over the WAN). It seems that the best practice is to pass through the storage controller when virtualizing like this but I just didn’t have the extra storage controller to do it with as my primary storage controller was also observing my VM SSD’s and you can’t pass through a non-dedicate storage controller and share it between proxmox and the vm from my experience.
I am also in a similar boat. I tried a truenas vm hosted by proxmox and it did not fit my needs. I am looking to try using an LXC container with smb server and then have proxmox pass ZFS array to the container. I heard this is possible but I don’t know what I am getting into and looking to learn more before I commit, as there will be a large dataset 8x12TB drives 7x8TB drives, 5x6TB drives media library | vm/physical drive backups | RTSP recordings from home security cameras.
