Send the Homelab to the Attic

Homelab
1

image

Hey guys,

I am in the process of rebuilding my network. The current setup is a Fresh Tomato router/access point running all traffic through a vpn. Then, I have an unmanaged tplink 24 port switch.

For the new setup I am dusting off some old stuff I have been wanting to put into service for years and I am finally going to get it all in the rack.

You guys in the homelab community have gotten me excited about tech for the first time in years. I really have no idea what I am doing, so any direction is appreciated. Here is the plan for my rack.

unify 24 port non poe switch.

Dell R230, now running pfsense and running all traffic through a vpn. I have a lot more setup I want to do, like jumping into vlans. I still don’t understand subnets and vlans and who can talk to who.

r730xd, this is my new toy. I got proxmox and a few vms spun up and working. Jellyfin and unify are both up and running. This is really just a temp setup to see if this would work, and it does. I have couple of intel ssds I am going to install in the rear, mirrored zfs. I picked up a refurb 4tb nas drive to add to my other 2 for a 3 drive raidz pool.

I have 2 11 year old Lenovo t440s laptops. One has a broken touch and eraser and bad catch on the external battery. I would love to put both in the rack with a job so I can justify picking up something about 5 years newer (t490s). I have proxmox on one with a ubuntu 18vm and jellyfin in docker. This is going be the entertainment server while I move the r730 over to the new network. I think I would like to use these 2 machines as a low power backup/fail over from the 730. All that is way over my head right now.

Next up I have a crashed qnap nas (a future project) and a working 4 bay Synology nas sharing a shelf.

A Belkin f6c1000-tw-rk 1500 14ah and a APC Smart UPS 1500 36ah. I fell in love with the NUT and the stuff Techno Tim went over.

My goals are to learn and practice good backup skills. Learn how to use vlans and implement some cool rules.

1 Like
2

image
image317Ă—612 51.1 KB

3

Welcome to the forum!

Think of VLANs as a virtual switch. You connect 3 PCs on a switch and give them an IP in the same subnet. A switch operates at OSI layer 2, the TCP/IP stack at layer 3.

To make it easy to imagine VLANs and subnets, think of this: you got 2 switches, with 3 PCs, 2 connected to a switch and 1 connected to the other switch. The switches are not connected to each other. You have the IPs 192.168.1.1, 1.2 and 1.3 on the PCs. The first 2 PCs on switch 1 can communicate with each other, but not with the 3rd one. Analogous, PC 3 can’t talk with PC 1 and 2.

That’s the purpose of VLANs. They separate hosts at layer 2, so even if someone tries to be smart and changes their IP to try and reach another network, they can’t. It’s a security measure.

Subnets, as mentioned above, are purely a layer 3 thing. Image you have 3 PCs on a single switch, each with the IP 192.168.1.1, 192.168.1.2 and 192.168.2.1. The first 2 can communicate with each other, but the 3rd cannot with either. For that, you need a router that has an IP in both subnets 1.0/24 and 2.0/24. Notice that above I said 1 switch. That means that the clients can communicate at layer 2 (in the broadcast domain), but because there’s no implementation of stuff that makes use of that, they can’t (well, there’s some stuff like Wake-on-LAN, FibreChannel-over-Ethernet and just basic ARP requests, but assume there isn’t anything).

A managed switch has the ability to split itself into virtual switches. Easiest to think of is port access-mode (no VLAN tagging) and splitting ports into multiple “virtual switches” (like ports 1-4 in “vSwitch” 1, ports 5-7 in “vswitch” 2 etc. - even with IP addresses the same subnet, being on different switches, these clients can’t communicate). Of course, someone can make a loop and (physically) connect port 4 with port 5 and expand the broadcast domain to the 2 “vswitches” and basically break security, but that’s another story.

I hope this explains it a bit and doesn’t add too much confusion. Again, easiest way to think of VLANs is unconnected physical switches. And for subnets, it’s an IP network divided into chunks (most commonly a /24 chunk, which has 256 IP addresses, of which only 254 are usable - expand to /23 and you get 512 IP addresses, then /22 you get 1024 and so on).

As for the hardware, I’m a low-power tech guy, I don’t like big servers because of how much power they use, but if power is cheap for you, then you can get some really cheap enterprise gear. If you’re a fan of proxmox and you want to use the laptops, I’d suggest you look what services you can get away with running in LXC.

4

Thanks for the explanation. I think it is making a bit more since. I had not thought about someone just changing their ip to get onto another subnet.

Is there a way for devices to communicate across a vlan with rules? I assume with multiple network ports a machine can be in multiple vlans? I would like to be able to share some read only files across vlans. Or maybe backup things across vlans.

5

VLANs are explicitly configured to not cross communicate. The only way possible is if you wire up two vlans by plugging a port from vlan A to a port from vlan B.

To communicate cross-vlan, you need to hop to layer 3, with either a router that has an IP address in both subnets, or if your host is connected to a trunk port (using VLAN tagging to map traffic to certain VLANs). Another L2 security stuff is preventing unknown hosts to just set their port in mode trunk (PC side, on PCs you don’t control) and trying to hop onto other VLANs, by setting the “untrusted” ports to mode access.

You could also have a host that is connected to 2 ports in mode access, or to a port in mode trunk and configure a bridge between vlans (basically a virtual ethernet cable connecting the two virtual switches together).

Hopping cross-vlan is usually not desirable. You want to use either layer 3 ip addressing (use routers and configure firewall rules to allow / deny traffic), or have a host in multiple vlans (with a trunk port) and just have IP addresses in multiple subnets, for each vlan (you can also have multiple IP addresses in a single VLAN, that’s generally not considered secure, but it does work).

Just have to remember that a router-firewall will not prevent hosts from talking directly to each other (if they’re in the same vlan & subnet), for that you’ll need a local firewall on each machine. I personally prefer separating hosts by subnets and giving a whole range of rules to the entire subnet (management subnet, wifi clients subnet, “public” / guest wifi subnet, servers subnet etc.).

6

With the previous post, I just went to theoreticals.

Set your router to permit SMB port (445) to be accessed from other subnets to the SMB share.

Same thing, depending on the protocol used. Allow SMB / NFS / SSH traffic to where you need, in your router and just threat each vlan as a subnet. Makes things relatively easy to understand (for example VLAN 100 can be mapped to subnet 192.168.100.0/24, VLAN 105 to 192.168.105.0/24 etc.). There’s VLAN description too, which you can put in your switch and save in a separate network description file on your PC (can be a txt file), like VLAN 100 = management, VLAN 105 = wifi, etc.

7

Well I still have a lot to think about when it comes to segmenting my network.

My rack showed up a couple days early and just in time for the weekend. I also eneded up with about half a day to work on things. I spent a fair amoutn of time on site prep and I even found a scrap of carpet to throw down to keep the noise down and give me a good place to wrestle the rack. The rack came with some picture only instructions. It was comical how many parts and bots there were. It was fun though. I had to take it apart several times to figure out my length. I made a 2x6 platform to give the rack a solid base and make it a little easier to work with.

I think I have the layout all figured out. So now it is just finding the time to implement. I fought racking the r730xd but I had just missed a center rivet. It is all racked up now. I think the rear drive bay I picked up on ebay came with the wrong cable. I found 2 other dell options for this setup that I should have soon. So far power use on this has stayed under 100w for testing. Idle is 84w.

I also got the r230 running pfsense racked up. The rails showed up today. So far, this is showing 38w while running.

The old belkin ups is in place setting on the bottom shelf. I replaced tthe batteries. 14ah in total.

The little NAS shelf is above that and installed.

I ran across this and I am excited to give it a try. I am going to use jellyfin, but I am not sure how I will run it. So far, I have only used it in a docker container.

image
image1920Ă—1883 177 KB

8

I just about have everything situated in the rack. My old belkin ups is a great fit for a rack. I have had it for years when a server room replaced it. I put new batteries in and plugged it in. I didn’t have anything else plugged into it but it still woke me up in the middle of the night beeping. I think am I am going to cut my loses on this one. If anyone one in this community wants if they can pick it up. I am in south west MO.

image
image1280Ă—960 43.2 KB

image
image890Ă—820 59.3 KB

I also have my other ups, a big heavy apc. I love it. and it fits in the back of my rack better than I thought. The batteries in this one are quite large. I have enough room to fit a second apc beside this one. I hope to be able to to get some useable run time out of this, as in power goes out and I can still turn a movie on with my wife. That kind of functionality get’s me brownie points so I can keep playing.

The next problem I ran into was my nas setup. I picked up a cheap 12" shelf for the 2 of them. I measured everything assuming I had 19 inches to work with. turn out that is more like 18", but the shelf I got cut into that even more. So I ditched the shelf and with the belkin out of the rack the bottom of the rack will nicely hold both nas devices side by side in the front and 2 apc tower ups’s in the back.

I am getting a cable unplugged error on the 730 but I was still able to install and boot to the 2 ebay intel ssds. To get past the error on boot I have to hit f1. Does anyone know what I need to use the idrac jnlp file on ubuntu or mint?

MY biggest issue right now is my 3, 4t drives I have in the dell 730. I was just testing on the old install and I setup a 3 drive zfs pool. I tried wiping the drives and moving them around. No go. I want to pass these 3 drives through to a truenas scale vm running in proxmox. I have read through a few articles but the drive passthrough with the existing pool has me stuck. More research on that this weekend.

and a photo for everyone I just bored to death.

image
image433Ă—790 108 KB

1 Like
9

Java / OpenJDK, with WebStart (javaws) IIRC.

Make sure that the dell perc card is in non-raid (HBA) mode. Then pass the drives, it should be easy. The problem is that the disks need to be formatted if you switch from raid to HBA, but given you already tried wiping them, that’s not an issue here.

10

I had this big sob story written up with all these details about how I could not find the serial number for my drives, and then all the sudden there they were. The card I have is the h330 or hba330 I can’t remember.

For anyone else, this guide was all I needed.
https://dannyda.com/2020/08/26/how-to-passthrough-hdd-ssd-physical-disks-to-vm-on-proxmox-vepve/

qm set 101 -scsi1 /dev/disk/by-id/ata-ST4000xxxxxx-xxxxx_xxx
qm set 101 -scsi2 /dev/disk/by-id/ata-ST4000xxxxxx-xxxxx_xxx
qm set 101 -scsi3 /dev/disk/by-id/ata-ST4000xxxxxx-xxxxx_xxx

So now I have truenas scale setup in a vm and the drives passed through. I created my first pool, but I had no idea what the other setup vdev is for raidz1 so I think I will have to read up a bit and redo it. I assume truenas can kill this pool easy enough.

image
image698Ă—350 26.6 KB

I also failed at installing the unifi controller on a debian vm and then a mint vm. Tom Hear’s video to the rescue.

Now I have that up and running and I can get to the switch. I also just got the unify a6+ access point. That synced up fine too.

Maybe you guys can give me some direction with storage. I have a 4 bay synology with 3 drives populated, that I am using right now for personal files and media. I want to setup a similiar sized 4 drive truenas vm that will become the primary and the synology will just be a backup to that.
I am unclear if I should you iscsii, nfs or smb for either or both. Right now I am using samba and mounting that manually in each vm, a big pain and I am bad at it. I also have plenty of free drive slots so easy future expansion is something I want to consider. I am going to go rewatch Jay’s video on proxmox storage.

image
image1197Ă—826 119 KB

Oh and one more thing, the fans on the r730 are spinning at a 44% and that is right on the verge of wife complaining territory.

image
image994Ă—774 63.4 KB

image
image1122Ă—641 69.7 KB

image
image1198Ă—936 81.8 KB

11

I am connecting a device/vm to the network then booting it up. Then I am going int to pfsense and clicking add static mapping and then setting the IP I want and nothing else, saving and applying changes. Then I reboot the device.

Some of the static IP addresses I am setting are after the start of the pool start. ex:

image
Should I change that? I think I would like to have about 30 that I I can set the address manually. IF so, how?

12

I prefer to have a DHCP pool of about 100 to 200 IPs and have a few for static mapping. I grab the MAC address of the VM and assign it in DHCP mapping, but I also set the IP statically in the VM’s OS. That’s so I get both DNS resolution and a quick “centralized source of truth” when it comes to IP allocation. Nothing should be configured without also adding stuff to the DHCP server.

1 Like
13

Just a little update on the fan situation. While running at 44% I decided I would just shut the machine down and power it back up. When it did come back up it was at 75% along and accompanied by spousal disapproval. I had ordered two potential correct cables for my 2 rear drives in the 730xd. The cable I got with the used ebay kit would only connect to over by the raid controller card. The correct cable came in and I connected it up to the front backplane like the cloud ninja videos show. Now my fans are running at 10% and the r230/pfsense box is again the loudest thing in the rack.

Before all that, I did this, now I am just keeping it in my backpocket in case I need it.

before that I tried modifying some of the fan controls through the command line. I followed this guide.

I tried this
racadm (connection info) get System.ThermalSettings.ThirdPartyPCIFanResponse

and then reverted with
racadm (connection info) set System.ThermalSettings.ThirdPartyPCIFanResponse disabled

14

I have been struggling with vdevs a bit. This has some good info from a truenas dev, but I need more. I am finding a disconnect in articles an videos setting up truenas with real world homelab type scenarios. I have been through several videos/articles that just end up setting up a single drive and then the other end of 60 drives or a petabyte of storage. What about 3 or 4 drives?

12 drives +4 is well within reach of us commoners though.

image
image711Ă—1112 156 KB

15

Vdev means virtual device and it’s just a zfs nomenclature. A vdev can be a physical disk, or a pool.

  • vdev1: /dev/sda1
  • vdev2: /dev/sdb1
  • mirror: vdev1 = vdev2

Another example.

  • vdev1: mirror: /dev/sda1 = /dev/sdb1
  • vdev2: mirror: /dev/sdc1 = /dev/sdd1
  • stripe: vdev1 + vdev2

I’m not going to watch the video, I don’t have time for that.

I think 3 drives is pretty useless, unless you are looking for the smallest power consumption. Go 4 drives in stripped mirrors, or 5 drives in raid-z (best bang-for-buck on capacity vs reliability) or 6 drives raid-z2 (a little less capacity for more reliability).

16

No worries on the watching the video. I am just throwing stuff out there I find useful in case another green person might find it useful. I am just trying to have good future expansion in my plan. A 5 drive Raidz1 would also work well. I could get the first 5 up and running and still have an easy upgrade path for 5 more, leaving only one free bay.

Your comment made me laugh. virtual dev makes since, I don’t think I had thought that one through.

I guess I will give you some more details on my storage plans. I have 3 drives collected so far. I can budget about 1 a month. My nas is also 4 bays, with the same size drives. I was planning to use that as the backup. My server has 12 bays + an optional 4. Having a real usable backup plan is a big goal of this project.

I managed to break my pfsense box, I was following a guide that talked about making some hardware acceleration changes. I am going to do a fresh install and try again. Another top priority is the ability to use and switch between multiple privacy VPNs. The deeper I get into this the taller that order seems to be. Over the past week I have heard Tom from Lawrence systems comment that it is difficult and I even found some unresolved attempts on his forums. I am using VPNs so I have the most generic web experience possible. I am hiding from as much personalized influence from big tech as possible.

pfSense baseline guide with VPN, Guest and VLAN support

https://nguvu.org/pfsense/pfsense-multi-vpn-wan/

If someone has a better guide about hopping between multiple privacy VPNs on pfsense, please share.

17

I have had some wins this week. I have pfsense reinstalled and a vpn setup. The guide above broke my system each time I tried it. I don’t have any vlans setup. I do have a good backup though.

I tried mounting my nfs test share on a few vms with no luck. I am getting pretty good at samba shares though.

I had some time to play with docker. I have 2 containers deployed with yaml files. One of those is tubearcivist. For some reason I cannot login. I have changed the user and password out but no change.

version: '3.5'

services:
  tubearchivist:
    container_name: tubearchivist
    restart: unless-stopped
    image: bbilly1/tubearchivist
    ports:
      - 8000:8000
    volumes:
      - media:/youtube
      - cache:/cache
    environment:
      - ES_URL=http://archivist-es:9200     # needs protocol e.g. http and port
      - REDIS_HOST=archivist-redis          # don't add protocol
      - HOST_UID=1000
      - HOST_GID=1000
      - TA_HOST=192.168.55.15         # set your host name
      - TA_USERNAME=tubearchivist           # your initial TA credentials
      - TA_PASSWORD=password              # your initial TA credentials
      - ELASTIC_PASSWORD=password         # set password for Elasticsearch
      - TZ=America/Chicago                 # set your time zone
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 2m
      timeout: 10s
      retries: 3
      start_period: 30s
    depends_on:
      - archivist-es
      - archivist-redis
  archivist-redis:
    image: redis/redis-stack-server
    container_name: archivist-redis
    restart: unless-stopped
    expose:
      - "6379"
    volumes:
      - redis:/data
    depends_on:
      - archivist-es
  archivist-es:
    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.12.0
    container_name: archivist-es
    restart: unless-stopped
    environment:
      - "ELASTIC_PASSWORD=password"       # matching Elasticsearch password
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "xpack.security.enabled=true"
      - "discovery.type=single-node"
      - "path.repo=/usr/share/elasticsearch/data/snapshot"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es:/usr/share/elasticsearch/data    # check for permission error when using bind mount, see readme
    expose:
      - "9200"

volumes:
  media:
  cache:
  redis:
  es:

I always end up with this:
image

I found this, but I have no idea if that answer is relevant to my situation.

I am going to start working on syncthing next.

18

Are you running Proxmox? Do the VMs have the firewall enabled at the “port configuration” page in the VM options? Are ports 111 and 2049 (both TCP and UDP) allowed in your firewall? Is the NFS export configured on the whole subnet of the VMs?

19

Yes I am running this in proxmox. Here is the firewall options for this VM.
image
This is default. I have not changed anything here.

and here they are for the datacenter level

image
image736Ă—220 25.3 KB

I got this message on my truenas vm server after trying to get syncthing going. I think I just need to change the folder.

Critical

SMB shares have path-related configuration issues that may impact service stability: pool4tb: SMB shares containing the apps dataset are not permitted

2024-03-30 01:00:06 (America/Los_Angeles)

image
image1143Ă—540 39.8 KB

I have spent about 2 days trying to get syncthing to sync from my laptop, to a share on truenas or my synology. Right now I am trying to use a mint vm to sync some of the files from my home folder on my desktop and laptop to the samba share on this vm. Maybe there is a better way to do this.

This is my fstab as it stands now. This does seem to work but something in syncthing does not like it.

image
image831Ă—35 10.2 KB

image

image

" Is the NFS export configured on the whole subnet of the VMs?"
I am not sure what this means so I will do some research. I have made some feeble attempts to get nfs to work but I have never made any headway.

20

You are mounting a CIFS share, but SyncThing is expecting Unix-style permissions. If you have the same share for both linux and windows, just use the NFS share instead for syncthing under /media/whatever.

The reason it is failing to set chmod, is because the backend file system is windows-style (at least seen as so, because SMB), i.e. unix permissions can’t be applied here.