I don’t have experience with hyperconverged infrastructure, so I’ll be talking out of what I heard from others.
People seem to be praising Ceph and Ceph integration in Proxmox. I have been a Proxmox user for a while, both at home and at a really small data room. We have only used NFS NAS attached to our Proxmox hosts and aside from not having highly available storage (besides the local raid, but I’m talking complete NAS failure), the host HA did work wonders, being able to either live migrate, or move the activity on another host without noticing any downtime when a HA node fails.
I’m not in the business of doing a sales pitch for Proxmox though, on the low-end, it does have issues, like if your cluster ever enters a failed state, it’s going to be hard to recover the cluster without resurrecting the dead node(s), due to how Proxmox operates. But if you won’t be faced with such a situation and if you will have more than 5-7 hosts, it should be fine.
As for storage, I’m not sure how Ceph itself works when ran directly on the hosts. When you have a ceph cluster, you get the benefit of basically a highly available SAN. But I don’t know how it runs on the local hosts though. Common sense would dictate that running on the nodes doesn’t necessarily imply that a disk image would run on the same host, meaning you are likely to be reading data from another host, which besides breaking the point of running local storage, it also puts an additional strain on the hosts that could have been used for other things.
So my advice would be having a Proxmox cluster and a separate Ceph cluster. The ceph cluster can also run Proxmox, because I heard Ceph is easy to setup on Proxmox.
I just hope I am not wrong about my assumptions about ceph, or if there is something missing from my current knowledge, like having a way to run VMs on local storage and have them replicated on another server, because that would make it quite impressive.
Some additional points to make. Proxmox LXC integration is terrible. You are limited to what Proxmox provides to you. If you are ok with only running Alpine, Arch, CentOS, Debian, Fedora, Gentoo, OpenSUSE, Rocky, Turnkey and Ubuntu, you should be fine, but if you want something else, even if you have perfectly working LXC rootfs images, good luck on that, because Proxmox can only detect those ones. Also, Proxmox can’t do live migration on LXC, just on VMs.
As for K8s cluster, I suggest that, if the OCI containers (docker) themselves can be disposable, you should run them in LXC containers (nested containers). Make a few containers on each Proxmox node, like say 4 LXC containers for each Proxmox host, out of which 1 will be a master k8s node on each proxmox host, with the rest of them being worker nodes. That would give you 3 master nodes and 9 worker nodes to work with. If any of the worker nodes go down, the load can be spread on the other ones, or if one of the proxmox hosts go down, the rest of the K8s cluster running on the other 2 Proxmox hosts can take up the slack.
However, if the K8s workers storage is not to be considered disposable, I suggest you switch from LXC to VMs and make them HA. This is not the way Kubernetes was intended to be run, but nowadays so many applications are made into OCI containers (usually docker) that it’s hard not to run them as containers. So you would still have the same number of k8s nodes, but when a container fails, if you need the information from that specific one, you have to revive that node.
Still, if you can follow the perishable k8s nodes route, that would be better, as the containers can be relaunched by the master nodes on demand, or even when you need more resources (auto-scaling). But for monitoring software (NMS), I don’t think that’s how it works. At least I can’t imagine having my Prometheus and Grafana servers be considered perishable, I need the information contained in those servers and launching new instances when the old ones die wouldn’t help me without the data inside them.
