Hi, Everyone.
Hope you can help identify why 2 Ubuntu base workstation open any website behind double NAT. Please help me identify what did I miss that cause them not to be able to browse any website. Since this is lab network, if you have any suggestion that I can try please let me know I can do the change immediately.
Setup:
ISP->Pfsense->XCP-NG->Pfsense lab-> Ubuntu & PopOS, Windows 10 Workstation
Use case: I made lab network 192.168.70.x and in this lab network I setup another pfsense and gave it LAN IP of 10.0.0.1. The LAN interface of pfsense is using tunnel7 of XCP-NG to make a separated network. In this network I have 3 machines 2 Ubuntu based workstation and a Windows 10 workstation. Both Ubuntu workstation can ping google.com but doing curl or opening google.com in a browser fails. If I use the Windows 10 workstation I can browse any website.
I do not have any proxy configured either in my main network or in my lab network. All machines are using DHCP and using pfsense as a DNS resolver. I have also tried using an external DNS and still the same problem. I have also tried disabling IPv6 on both Linux workstation but still same problem.
Here is ping response from my PopOS:
Firewall Rules for my main pfsense:
Firewall rules in the pfsense lab:
If the pfsense VM and windows 10 can curl / navigate google, then the issue is more local to the ubuntu boxes. Being able to ping, but not to see websites, it can mean that their time and date may not be properly set up. Make sure you allow NTP traffic on your network.
If that doesn’t work, try to view the ISP’s router management interface by its IP from the Ubuntu boxes. If that works, but not google, it’s something local on ubuntu. But if that doesn’t work either, one of the pfsense boxes may be blocking web traffic.
Thank you for getting back to me, I was able to configure NTP sync and perform a sync. Unfortunately it still failed to connect:
In regards to your procedure to access ISP router interface. I don’t have ISP router but using the lab Windows lab VM I can access my main pfsense interface but failing on the Linux machine. I have also tried bypassing the pfsense lab and connected the linux lab machine directly to my main pfsense and it was able to browse the Internet. I connected the linux lab behind the pfsense lab then try adding firewall rule allowing anything from LAN to access the anything passing WAN Gateway but still the same issue
How are you connecting the workstations to the lab pfsense? You got a switch behind it? Or are you plugging straight to your NICs? Did you try plugging the ubuntu workstation in the port that Windows uses?
Pfsense lab, 2 ubuntu workstation and windows are all VMGuest of XCP-NG. In XCP-NG it has an option create PrivateNetwork in which I configure to allow the LAN traffic.
So on Pfsense lab it has 2 interface WAN is configured to use the labnetwork (192.168.70.x). the LAN of my pfsense lab is using PrivateNetwork (Tunnel 0) which is also used by 2 Ubuntu workstation and the Windows Workstation.
The private network should not be a Tunnel interface, but a Bridge interface. You need 2 bridges in xcp-ng. Bridge 1 connects to the LabNet and xcp-ng can have an IP on it, then the eth0 on the VM config page in xcp-ng gets bridged to Bridge 1. Eth0 will be the pfsense WAN.
Then you create Bridge 2 that does not bridge anything (basically a bridge floating in the sky). In the VM config page, you create eth1 and bridge it to Bridge 1, which will be the pfsense lab VM NIC1, on which you configure the LAN. On the Ubuntu, Pop!_OS and Windows VMs config pages, their eth0 should be all bridged to Bridge 2. This way, you get a private network inside xcp-ng.
For such a simple setup, you should not need NAT, you only need to add a static route to pfSense phy router towards the Lab’s LAN via the IP address on the pfsense Lab VM’s WAN IP. Also make sure, if you haven’t already, that you don’t block bogon traffic on pfsense Lab VM on WAN, as you are behind a private subnet.
And that should be the setup. I’m not familiar with xcp-ng’s nomenclature in the VM configurations, I only know what general linux distros use. I am sure that behind the scenes, they are still bridges and macvtaps, just that the GUI may have different naming conventions.
Btw, connection reset by peer can be also shown when things in the middle, like firewalls, send the TCP RST flag to the client, but even with all its handholding, pfsense should not be doing that by default. So the message is a bit confusing.
Not sure how to create this, I need to read through XCP-NG documentation. I will update you once I have configure this and if it works or same problem. Again thanks for providing great suggestion.
