First, welcome aboard @tayray
I’d recommend searching for open-source applications within the area of applications you need.
For storage, I started with OMV and then later migrated to TrueNAS. I think OMV was easier to learn and get started with. TrueNAS, IMHO, is enterprise-grade.
I don’t have any security cameras, so I can’t be of much help there, but there open-source apps out there. I hear Frigate mentioned from time to time.
For recording, I use Ardour with a Scarlett 2i2. I have a pair of KRK v8’s connected directly to the Scarlett. This works well for me, but my recording needs are very basic. I’m only recording myself playing along with songs as a method to improve my playing technique.
You can use VMs to host your apps. I used to do that, but I have moved all of my apps into Docker (compose) and I don’t have any VMs running anymore (and I couldn’t be happier). Containers have a definite learning curve, but IMHO it is definitely worth it. For me, managing containers (backups, updates, and moving containers between servers) is soooo much easier than VMs.
I did run libvirt/KVM for several years, though, and managed the VMs with cockpit & VirtualMachineManager (remote access) and it worked for me.
I build a server from spare parts back around 2017. It was a 6-core AMD with 32GBs for RAM. I thought that would be enough to run a couple of VMs. I quickly found out that it was over-kill. At one point, I think I had 12 VMs running and the server spent almost all of it’s time idling, meaning that it was using very few resources (CPU/memory).
I’m currently hosting 18 containers on a rpi-4 with 8GB and it is using 12-25% CPU and less than 3 of the 8GBs of memory. I am slowly migrating to minipc’s with 2.5gb NICs as I slowly upgrade my network.
Good luck on your search for applications, and, welcome to the open-source community.