Proxmox Shell lockup after adding vmbr1 Linux Bridge

aiwa · June 7, 2023, 5:38pm

Greetings,

I am using an HP Z440 workstation with proxomox. I have run memory test with no errors and smart tested both SSD and spinning hard drive in system, both have no errors, BIOS is on the latest version. I have installed an HP 361T 2-Port GbE NIC into the system. I have two Linux bridges enabled, vmbr0 on port enp8s0f0 IP address/CIDR 192.168.50.1/24, vmbr1 on port enp8s0f1 IP address/CIDR 192.168.25.0/24. Once I enable the second Linux bridge the proxmox shell and vm shell lock up after a few seconds. However if I remove the vmbr1 bridge then the proxmox shell and vm shell are ok. I have created a Linux bridge using the onboard NIC and the same results! I have assigned a static IP address to the vmbr1 bridge too. Can anyone point me in the right direction to resolve this issue? I really like using proxmox and would like to continue, but if I can’t figure this issue out then I will have to leave.

root@heartofgold:~# dmidecode | grep Product\ Name
Product Name: HP Z440 Workstation
Product Name: 212B

root@heartofgold:~# dmidecode | grep -A3 BIOS\ Inf
BIOS Information
Vendor: Hewlett-Packard
Version: M60 v02.61
Release Date: 03/23/2023

root@heartofgold:~# lspci | grep Ethernet
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-LM (rev 05)
08:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

root@heartofgold:~# dmesg | grep intel
[ 1.071550] intel_pstate: Intel P-state driver initializing
[ 4.106891] Btrfs loaded, crc32c=crc32c-intel, zoned=yes, fsverity=yes
[ 4.815929] snd_hda_intel 0000:00:1b.0: enabling device (0100 → 0102)
[ 4.816425] snd_hda_intel 0000:02:00.1: enabling device (0100 → 0102)
[ 4.816514] snd_hda_intel 0000:02:00.1: Force to non-snoop mode
[ 5.027242] intel_rapl_common: Found RAPL domain package
[ 5.027249] intel_rapl_common: Found RAPL domain dram
[ 5.027257] intel_rapl_common: DRAM domain energy unit 15300p

root@heartofgold:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp8s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether e4:11:5b:95:fa:b0 brd ff:ff:ff:ff:ff:ff
3: enp8s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether e4:11:5b:95:fa:b1 brd ff:ff:ff:ff:ff:ff
4: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether dc:4a:3e:99:bd:dc brd ff:ff:ff:ff:ff:ff
altname enp0s25
inet6 fe80::de4a:3eff:fe99:bddc/64 scope link
valid_lft forever preferred_lft forever
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e4:11:5b:95:fa:b0 brd ff:ff:ff:ff:ff:ff
inet 192.168.50.1/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::e611:5bff:fe95:fab0/64 scope link
valid_lft forever preferred_lft forever

root@heartofgold:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp8s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether e4:11:5b:95:fa:b0 brd ff:ff:ff:ff:ff:ff
3: enp8s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether e4:11:5b:95:fa:b1 brd ff:ff:ff:ff:ff:ff
4: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether dc:4a:3e:99:bd:dc brd ff:ff:ff:ff:ff:ff
altname enp0s25
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether e4:11:5b:95:fa:b0 brd ff:ff:ff:ff:ff:ff

ThatGuyB · June 8, 2023, 2:11am

What IP address were you accessing the web GUI when you were bridging vmbr1?
Are you creating the bridge inside Proxmox web GUI, or manually via /etc/network/interfaces?
What’s up with this?

Why are you using IP address “0” for the bridge 1? That one is reserved for determining the network. Try it with either 1, or try something higher if you already have a gateway using .1 as the IP address, like .2 or .5 or 104.

If that doesn’t work, try via CLI:

ip link add name vmbr1 type bridge
ip link set enp8s0f1 master vmbr1
ip link set enp8s0f1 up
ip link set vmbr1 up
ip addr add 192.168.25.104/24 brd + dev vmbr1

But I suspect that when using IP addr 0, the network service would crash. Expected behavior when wrong input is used. Try this and if it doesn’t work, show me the output of these commands.

aiwa · June 8, 2023, 4:54am

To setup the vmbr1 Linux bridge I followed Jay’s Class 13 - Networking YouTube Video. I created the vmbr1 inside Proxmox, vmbr0 was already created by Proxmox.

Datacenter → heartofgold → Network
Name: vmbr0 (Using as management interface)
Type: Linux Bridge
Active: Yes
Autostart: Yes
VLAN:
Bridge ports: enp8s0f0
IPv4/CIDR: 192.168.50.1/24
Gateway: 192.168.50.254
Comment: first Linux bridge

Create: Linux Bridge
Name: vmbr1
IPv4/CIDR: 192.168.25.0/24
Gateway (IPv4): (blank)
Bridge ports: enp8s0f1
Comment: second Linux bridge
Create
Apply Configuration

I am using 192.168.50.1 to access the Proxmox web GUI/management network.
I am using 192.168.25.0/24 for the VM/Container IP addresses network.
I changed the vmbr1 Linux bridge IPv4/CIDR address from 192.168.25.0/24 to 192.168.25.35/24 and shell inside proxmox locked up!

ip link add name vmbr1 type bridge → RTNETLINK answers: File exists

ip link set enp8s0f1 master vmbr1 ->  (Nothing returned)

ip link set enp8s0f1 up  ->  (Nothing returned)

ip link set vmbr1 up  ->  (Nothing returned)

ip addr add 192.168.25.35/24 brd + dev vmbr1 → RTNETLINK answers: File exists

When I do a ping -I vmbr0 8.8.8.8, I get multiple responses
When I do a ping -I vmbr1 8.8.8.8, I get one response then nothing, I use cntrl-c to break out.

Thank you for your help!

ThatGuyB · June 8, 2023, 11:16am

It seems like the bridge already existed when the commands were ran, which is why you got the File exists error.

Yes, because your default gateway is set to 192.168.50.254.

This will not work, because you do not have a route. You can only have one default gateway, but traffic can only go through one interface.

But I realized that you don’t need to assign an IP to bridge 1 necessarily, unless proxmox will be used to route traffic from that subnet.

Starting up my proxmox VM is a pain, I would like to look that up and see what’s wrong. Maybe try to delete bridged 1, then create it in proxmox again in the meantime?

hulxmash · June 8, 2023, 6:42pm

As ThatGuyB said, you don’t need to set the IP or subnet on the network bridge at all. On my system I created the bridge with no network setting. I simply created a new bridge right in the Proxmox GUI and added the physical card to it. No other settings. I have had no issue.

aiwa · June 8, 2023, 8:14pm

I just want to thank you for taking the time and helping me with my issue. Here is more datils about my home lab.

My Home Lab PC HP Z440 1x onboard NIC, 1x 2-port pCIe NIC

Internet Router - 192.168.40.0/24 → Connected to my physical pfSence PC (igb0) with 4-port NIC’s

Port 1 - Internet Modem 192.168.40.0/24 (igb0)
Port 2 - 192.168.50.0/24 VLAN 50 (igb1)
Port 3 - 192.168.30.0/24 VLAN 30 (igb2)
Port 4 - 192.168.25.0/24 VLAN 25 (igb3)

VMBR0 - 192.168.50.1 - web gui/management
Gateway -Blank
Bridge port: enps0f0

VMBR1 - 192.168.25.35/24
Gateway 192.168.25.254
Bridge port: enps0f1

Did a ping 8.8.8.8 at sequence 33 pings stop shell is locked I may be moving onto another hypervisor or maybe it could be the 2-Port PCIe NIC.
Thank you again for your help!

ThatGuyB · June 10, 2023, 3:24pm

I was away for a few days. Here is what my settings are in my Proxmox VM.

I have only one interface, enp1s0, but I set up vlans with a pfsense VM inside the proxmox VM.

Treat vmbr10.2 and 10.3 as a physical interface, like you have enp8s0f0 and enp8s0f1. You can see I went with a bridge vmbr22 connected to the virtual interface vmbr10.2, on which I have an IP assigned and another bridge, vmbr33 which is bridging to vmbr10.3.

Mine are vlans, but you probably don’t need to set up vlans at this point in time.

Your setup would look something like:

enp8s0f0 = no setup
enp8s0f1 = no setup
vmbr0 = bridge with enp8s0f0 with the IP 51.1, default gateway 254 (assumed)
vmbr1 = bridge with enp8s0f1 without any IP, you don’t need any, like I don’t have on vmbr33

I’m using bridges on top of bridges (think of bridges like switches), because it’s a VM. I could make another virualized NIC, but the way I had proxmox set up was to help someone else with VLAN configs.

Proxmox is an ok hypervisor. You do not need an IP address on the bridge if all you need is that the VMs connect to the network behind the physical interfaces.

My suggestion is that you use the default gateway on the management interface and keep the 25 subnet interface without any IP. Makes it more secure that way.

My VM network is set up this way: physical network → Proxmox enp1s0 → bridge vmbr0 on it. Then the pfSense VM has 2 interfaces, one connected to vmbr0 (serving as pfsense WAN), and another one connected to the floating vmbr10 (serving as vlan’ed LAN). Basically, for the sake of understanding the setup, ignore everything on the WAN side and pretend it’s not there (enp1s0 and vmbr0 can be thought of as the ISP connection to pfSense VM).

pfSense has internal vlans 2 and 3 on the 2nd interface. vmbr10 on proxmox, being a dumb switch (with vlan support) is receiving all the ethernet frames. vmbr10.2 and vmbr10.3 read the frames for each VLAN and send it to all the connected bridges to them.

vmbr22 is connected to vlan2 and is used for VMs that I want in vlan2. Analogous for vmbr33.

The reason I have it like this is because I need to access proxmox VM somehow (so I do it on its “wan”), but I made it so that vmbr22 is the “management interface” (using a VM inside vmbr22) and vmbr33 is the normal VMs network, that should not have access to the hypervisor. Not having an IP on vmbr33 is the whole point. And I’m using pfsense to filter traffic between the two interfaces and the WAN.

So your vmbr0 should be analogous to my vmbr22 and vmbr1 to my vmbr33. You should have the management IP and default gateway on proxmox on subnet 50 and have just a bridge with no IP connected to subnet 25, just for VMs only.

Let me know if you want me to delete this setup and configure it like you have, with 3 interfaces.

I don’t think it’s proxmox fault. I think it’s just the settings applied to it.

It’s possible that when you apply the bridge, the reason the proxmox web GUI freezes may be because the network config is wrong, so proxmox looses its networking capabilities until you kill the wrong configuration. I assume you are deleting the interfaces via CLI logged in to a local display, not via SSH.

aiwa · June 10, 2023, 5:58pm

Thank you so much for taking the time to help me with this issue! I was able figure the issue out, here is what my setting on my Proxmox server network. I removed my dual port NIC and place a single port NIC into my Proxmox server. I thought the NIC was bad, but I was wrong. I must be greeting old, I have no idea why I thought I needed to give the vmbr1 Linux bridge an IP address, layer 1 switches do not need IP addresses!

I have a physical pfSense running on a Dell PC

Proxmox Server - onboard eno1 - plugged into my Layer 1 switch then pfsense port igb1 - Subnet 192.168.50.0/24 - Management Interface

Proxmox Server - PCIe enp5s0 - plugged into my Layer 2 switch then pfsense port igb3 - Subnet 192.168.25.0/24 - VM/Container Interfaace

vmbr0 has an IP of 192.169.50.1, GW 192.168.50.254, interface eno1

vmbr1 assigned only interface enp5s0

using default GW 192.168.50.254

Everything is working great, sticking with Proxmox. Thank you again for taking the time to help me!