Qubes OS for security geeks

Linux Distributions
1

Hi everybody

I have been enjoying Jay’s videos very much. It helped me a lot. I love Linux and I appreciate Jay’s efforts to make Linux accessible and understandable.

While looking for a security-oriented Distro, I came across Qubes OS.
I managed to install it and try it. However, I am not very good at networking. It would be great to have some tutorials on how to technically make various qubes with various functions (for example, only for emails, for a cloud service or for other isolated functions).
The Qubes OS documentation explains most of the available functions. But setting up various qubes is pretty complicated for an average Linux user.

This blogpost explains a possible workflow. However, it is missing the technical steps to get it working on Qubes os.

If @jay and others are interested in the topic, it might be on one side a “security” and on the other side “Linux Distro” topic at the same time.

And thank you for the great community. I am glad to be part of it :slight_smile:

1 Like
2

A very interesting idea. I read the intro and was curious about why it is implemented as VM’s instead of containers. Personally, I think containers would be a better, more modern approach, and much more efficient use of resources.

3

Qubes is more like a hypervisor, than a desktop OS, just that all the VMs are using desktop windows to control the VNC session, unlike something like Proxmox, where you use a web browser / noVNC. So you have to think in terms of, basically, VMs. You have 1 VM for banking, one for this forum and a few chat programs, one for random internet browsing that you don’t want associated with your normal logged in accounts etc. I’m not sure if you really need networking skills for Qubes, but I never used it myself, so I can’t say, but you shouldn’t need much, if any IMO.

VMs are presumably harder to escape from. Containers require quite a bit of configuration to make them secure, but even so, if one container breaks loose, it may get access to the main kernel, which means it can affect other containers too. However, VMs are harder to crack open. At worst, a malicious actor can only gain access to one VM’s OS and potentially to the network, but still, the rest of the VMs should be safe. There are some situations where a malicious actor could gain access from the VM to the hypervisor (basically Qubes itsefl), but this scenario is hard to do.

Also, I believe the work on Qubes started way before containers really caught on. And with containers, you need to mess with SELinux / AppArmor and other security tools to keep it secure, with a VM, you don’t need much finagling.

1 Like
4

All of this for a single-user desktop system? Sounds like too much work. Is it more secure, perhaps, but is the juice worth the squeeze?

I understand your statement about VMs being more secure than containers, but I don’t agree with it. If one could break out of a container, one might gain access to the kernel. However, the same holds true for VM Escape (the security term in popular use), where one would have access to the hypervisor, and the kernel. With containers being chrooted, I’d say they are just as secure.

There is no requirement to use appArmor and/or SELinux with containers. Doing so would provide for additional security, but I have run docker and KVM without either.

Also, I’d add that container networking is not that much different than hypervisor networking. Either you are configured for a bridged network or are doing some host-level networking including routing or VLANs. The concepts are the same, only the configuration/implementation are different.

5

I’m not saying that Qubes doesn’t require work. IMO, the juice is not worth the squeeze, as I’m not using it, obviously.

VMs are inherently more secure by design. That doesn’t mean I prefer VMs over containers, they have their places, but I run LXD mainly, so I don’t make a lot of use of VMs, unless I need to run other OS than Linux, which rarely happens.

The reason we have seen things like FreeBSD Jails and Solaris Zones is because chroot is insecure. The only thing chroot does is change the path for the root folder and that’s it. No uid / gid change, nothing.

I don’t believe I said or even implied there is. Just that if you want security, running one of them along with other software is recommended. And I currently run LXD containers without either SELinux, nor AppArmor, only that my containers are unprivileged and that’s it.

As for the network side of thing, it depends if you set your network to act as a NAT for the containers / VMs, or if you bridge it to your LAN(s), and if you do, if you have other firewall rules in place (like you can do in Proxmox to prevent some traffic from going in or out of the container).

Escaping containers isn’t an easy feat either, but in theory it is easier than escaping a VM. Obviously, ask me how to do it, and I can’t give you an answer, because I don’t know. If I did, I’d probably be hardening my containers. All I’m doing at this point in time is run FOSS from the main repo of my distro of choice in the containers and trust that the programs I run will behave. For example, if I run postgres or haproxy from the official repo, it would be irrational to think that running it in a VM or bare metal would be any safer or insecure than running them in a container. I’m content with the level of security containers give. Additional security may be an exercise for when I expose things on the internet, but for now, I don’t have anything exposed, so it doesn’t really make sense to start hardening my infrastructure at this point.

6

I agree that escaping either one isn’t easy. I’m not aware of any way to do it (with containers or VM’s) unless you take advantage of an un-patched vulnerability, assuming one exists on the system.

If anyone knows how, I’ve love to hear about it.

I run containers and VM’s in my home lab. The VM’s are only to simulate a larger environment for me to develop and test automation (both network and server automation). I run Ansible for some of the automation and I script with Python for the rest.

7

I don’t know much about containers. Quebes OS is delivered with already configured VMs out of the box. As far as I understand, the vulnerable parts of the system do not have access to the Internet. There is also the possibility to use disposable VMs which will be deleted after you’re done using them. Everything that connects to the Internet goes through the Firewall first then through the Network → https://i.stack.imgur.com/WcuCr.png. It also has a color system for the VM Window where you can set color for each VM so you know for example that the red one is untrusted while the blue one is safe.
What I also find nice is that you can choose between using Debian or Fedora for each VM; or you can install another Distro.
On the other hand, it is pretty slow and requires a lot of RAM.

I believe it could be used by an average user out of the box. What was difficult for me was setting up special VMs for specific functions.

Right now I have it installed on an external SSD which seems to be working fine. I still however don’t feel that I can use it as my everyday OS (yet :slight_smile: ).

8

Sounds like the same configuration I use with containers and VM’s. In my network, everything is behind a network firewall, and a firewall on each node. Beyond that, I use VLANs to segment my network so that things like IoT devices and mobile phones can talk to each other, but the IoT devices can’t see anything else on my network. I do something very similar for my Wife’s and my work computers. They only get access to the internet and nothing else (not even my home printer).

Is this overkill for a home network? Maybe, but building and managing projects like this is how I like to learn and some of this is directly related to my profession.

9

@Mr_McBride
What do I need to learn / read / watch in order to be able to make my own network like you described?
I would rather have something like your Network (if I am able to set it up) than using Qubes OS. As mentioned Qubes OS is very slow and needs a lot of resources.

10

Grab a drink and join me for long post. Give me a bit to type this out.

1 Like
11

I do the same. Well, I had a restart in my life, so I’m still in the process of redoing a new homelab. I don’t have a lot of learning resources to recommend, I took the Cisco CCNA certification. CCNA 1 and 2 should teach you everything you need to know about layer 2 security, so maybe you can find some youtube videos about those. And they should give you a “getting started” for layer 3 automatic routing (OSPF), but the last part is a bit more than what you need to know for managing a home network.

I would say, if you are curious and you don’t have the budget to buy a managed switch, try learning how to do VLANs on Proxmox, create a pfSense / OPNsense VM and make it your main router on the Proxmox box and start creating either VMs or containers and connect them to each network you place on each subnet. After you learn the basics of bridging interfaces, it should be easier when you want to jump over physical infrastructure.

Not a good security practice telling people all over the world your setup, but do as I say, not as I do. My current setup is comprised of a Zyxel XGS1210-12 managed switch (unfortunately only managed through a web GUI, I couldn’t break into the SSH server that is clearly running on it), a Raspberry Pi 3 router (WiFi WAN, Ethernet port in trunk mode with 4 VLANs at the moment, will probably do more) and I bought a RockPro64 that will replace the Pi 3 (hopefully with OpenBSD) as the router, an Odroid N2+ for running my LXD containers and an Odroid HC4 as a NAS. The ON2+ will run LXD with most of my services, depending on the service, each sitting in its own VLAN. All devices will have a management VLAN dedicated to management and inter-device communications that will not have access to the internet (will have a separate subnet for that).

You can replicate this setup in a virtual environment, which is why I recommended you start playing with Proxmox. But the total for the physical hardware I own is currently around $600, not counting the microSD cards which I already owned, and I need to buy 2 SATA drives for the HC4. If you are going cheap, you can get a mini-lab like mine for about $700, but the reason why I went with ARM SBCs is because I’m testing something for ultra-low power consumption. It’d probably be cheaper to buy a 2nd hand managed switch and an old Dell Optiplex PC or Lenovo Thinkstation thinclient, as long as you don’t care that much for power consumption. My devices are going to run 24/7, so I care about power consumption (plus that I want to make this solar powered at some point in the future). But for a learning laboratory, you don’t need to have these running all the time, so you can save your money and still be able to learn.

1 Like
12

For me, everything is all about the journey. If I’m not learning then I need to change directions.

While I’m not a network engineer, I work very closely and support network engineers where I work. My skills are with application and network monitoring, and network configuration mgmt.

My journey with network segmentation started as a home lab project back in 2019 when I began studying for the Security+ certification. Like many others, I have IoT devices in my network and these are typically devices that rely on the vendor to provide for security updates. However, most vendors produce a device and support it for a limited time until they decide to produce a new device. Many times at that point, support for the older device stops. So, the importance of segmenting these devices becomes important as you don’t want a vulnerability in a device like a wifi capable camera to be used to attack other devices on your network.

So, I began looking at all of the devices on my home network and started to document what each device really needed access to. This allowed me to quickly group devices that only need access to the internet and nothing else. A Chromecast or Roku video streaming devices fall into this category. Once you group all of your devices and then consider their use case for network access, you have the blueprints, so to speak, of how many VLANs you need and which devices need to be a member of each VLAN.

Since I work with network engineers (and since I now work from home permanently), I thought it would be good to upgrade my home network from consumer level gear to commercial level gear. I started down the Unifi path, but didn’t really care for the network controller requirement and I wanted gear that was more like enterprise level gear. My requirements were this:

  1. Layer-3 switch with ssh access.
  2. A router that could do deep packet inspection and still route at gigabit speeds.

What I was looking for was devices that I could use to segment my network (security learning and practice), ssh access for automation (config mgmt - backup/restore), customization, etc.

Remeber, everything here was to promote an environment where I could learn and grown while building skills I can use at work.

A couple of use cases:

  • Use Ansible for network automation.
  • Use Python for network automation tasks (pull metric data, make a configuration change, etc).
  • Secure my network (segmentation, client isolation, etc).
  • Monitor network trends (sFlow).

In the end, I opted to stay with Ubiquiti but I chose to use their EdgeMax gear. However, I did stick with a Unifi AP for wifi. I ended with the following gear:

  • EdgeRouter-4
  • EdgeSwitch 24-lite
  • Unifi FlexHD

I have no functional need for the EdgeRouter in my home network as my firewall can handle all my routing needs and can manage VLANs (as can the EdgeSwitch), but I added it for learning purposes. Both Edge devices behave somewhat like a Cisco device. Both have console ports that work with roll-over cables, both have SFP ports for uplinks, etc. Both also have a usable webUI, but both can also be completely configured and managed via ssh.

Everything I’ve listed has been about network management. I also run KVM with a host of VM’s so simulate a server environment. All of my security patching is done via Ansible and I also am starting to do some provision/configuration mgmt with Ansible. I also run docker as I think learning docker is an easy way to learn about containers.

Other technologies in my home lab:

  • OPNsense.
  • Grafana/Influx monitoring with Telegraf.
  • Hubitat - home automation. I chose this device because it allows local mgmt. I want to move to Home Assistant at some point, though, because open-source reasons…
  • OpenVAS (similar to Nessus - security vulnerability scanning tool).
  • PiHole - local DNS
  • Syncthing - this is a project that I am just starting. I’ve got too many computers (if that is such a thing) and I want my home directory to follow me around.
  • Unifi controller (locally hosted and managed - no cloudkey here).

I also have a kubernetes cluster running on 4 RPI-4’s, but it is a work in progress and is not yet hosting any containers/pod’s. This is another learning project. All of the RPI-4’s are using PoE hats and are connected to an L3 PoE switch. This provides for fewer cables to manage.

Ok, time to end this wall of text.

1 Like
13

Best advice for starting a home lab is to start slow and grow slowly over time. Pick the projects you want to learn and grow from there.

I currently run Grafana on a VM, but it can also be run from a container.

RPi-4’s are great for small projects. I have run the following on RPi-4’s:

  • K8S - Kubernetes
  • Nagios-EMS
  • Nessus
  • PiHole
  • Zabbix
1 Like
14

Many thanks @ThatGuyB and @Mr_McBride for the detailed description of your Network and for the recommendations.
It seems that I have a long learning path.
I just checked the Cisco Academy and will see if I can get basic knowledge of Networking.
I don’t have a lot of IoT devices. For me, it’s about creating isolated spaces to protect the data / the system etc.
I do use RPi for testing some projects. The only thing that worked for me so far was a cloud server - I think because the instructions were straight forward.

I will keep your posts as guidance and will read about whatever I don’t know / understand.
I appreciate your help very much. Thank you :pray:

2 Likes
15

If you’re learning about networking, here’s my “Easy to Follow” guide for small secure network. Basically what Mr_McBride describes with isolated IoT VLAN. It’s still a work in progress though, I need to continue working on the dual-stack aspect of it (adding IPv6).

I call it “Untrusted” VLAN / subnet. I don’t have any IoT devices, because I don’t trust them, but the network was meant to serve that kind of purpose - basically isolate everything inside and restrict all traffic except what you need. You could only allow access from other VLANs to “Untrusted” on port 443 and maybe 22 and that would be plenty enough, while not allowing any access from “Untrusted,” not even to the internet (well, I thought about it more like a solution to stop devices calling their motherships).

My guide is more geared a bit towards privacy. I restrict all connections to DNS and NTP from my local subnets to the internet, I should have restricted ping too, but I don’t remember what I did there. I only allow them to access my router’s NTP and DNS server. It was more like an exercise if I were to go on darkwebs, I wouldn’t want to visit a malicious website that would do a ping or DNS request, or an NTP connection to a threat actor’s server and revealing my location. But I ended up never going on darkwebs in the first place. Oh well…

I need to convert this post (and my other “easy to follow” guides) to a git or wiki post. I’m thinking I should host stagit, or maybe Gitea if I’ll be too noob to use stagit, but I am undecided if I should be hosting a wikimedia server, or if I should just host static http articles. The advantage of wikimedia is that it would be easy to find stuff through its built-in search engine, but then I’d have to host a MariaDB server and back it up often (not that that’s hard to do though). However, I don’t write that often to make a wiki server justified, IMO. But it would be a somewhat nice to have.

+1 for Grafana, but I use it with Prometheus on the same VM, and run node-exporter on the servers / VMs I monitor. I haven’t gotten around to setting up a mail server, and because of that, alert-manager either. And other stuff (like, I want to host my own Matrix server).

I want to migrate the Prometheus+Grafana VM into 2 separate LXD containers, just for the lols.

Run LXD on the Pi and you can do a virtual k8s cluster (nested virtualization). It’s a very cheap way to get into OCI containers (Open Cloud Initiative containers, e.g. Docker, Podman, K8s, k3s, microk8s, k0s etc.).

2 Likes
16

Our approach’s sound identical. My VLANs are all designed for privacy and DNS and NTP are also local on my network.

I even have a VLAN to segment my work computer from the rest of my home network. I have 4 separate VLANs off of a single AP.

I have run Prometheus in the past, but I don’t have it running right now. I should probably add that to my project list.

1 Like
17

You already have InfluxDB, it’s probably not worth adding another monitoring software on your network, unless you have a specific need. But if it’s for learning or just for the lolz, go for it.

18

Yeah, it’s mainly for learning the differences between the different monitoring solutions. For example, I like the network discovery and networking mapping feature of Zabbix, but I prefer the configurability of Telegraf.

There are a lot of other tools that integrate with Prometheus, so I may end up replacing Influx with Prometheus as a database source.

1 Like
19

Hope your experience with Qubes is going well!

I just wanted to chime in to say that from what I’ve heard, distros like Qubes and Tails are meant for more advanced threat models. Not to say that folks with less urgent threat models can’t use them, but you might not have to deal with potentially poor performance (at least based on your comments) if you can still be served well by other distros.

This video by Techlore goes through hardening Linux. I’m sure there are more and less technical guides out there, but if you’re interested in other things you can do that are based on your threat model and are outside of the actual distro you choose, it could be helpful.

Just putting this out there, but in the end of course use what works for you. :slight_smile: