Proxmox dual NIC issue

Networking & Firewalls
1

Alright, I have been working on an issue and would love to get some ideas as to what I may be overlooking/missing that is causing this issue.

Hardware Specs

============
Unit - Dell Precision T7510, 2- Intel Xeon E5-2680, 512GB RAM, 2- 1GB NIC builtin (Intel brand), 2 - 2.5GB NIC PCIe (Intel brand, cant remember for sure, but 82800 i believe). 36TB in open storage. M.2 drives (primarily used for templates and boot)

ISP - Cable Modem 1GB down, 75MB up, using an Arrys Surfboard 8200 (DOCSIS 3.1, supplied by ISP). Already verified ISP does not have MAC sticky enabled.

Proxmox - Currently running 9.2.11 (have tried this on other versions with no success) Everything updated.

I have a Dell Precision T7920 with similar specs running Unraid available to test if that is a better option, with a 10GB PCIe NIC installed, but same Xeon concept and same RAM amount.

Setup for PFSense - VM (not an LXC) using PF Sense ISO, allocating 4 cores, 16GB RAM (yes I know its overkill but future plans here), using LinuxBridge for network, static IP set, /22 network for size.

============
Physical Layout

============

ISP Modem ===> 8 port Non managed switch ====> Dell T7510 ====> NIC 1 (2.5GB)

===============
What I would like to get done

===============

ISP Modem ====> Dell T7510 NIC 1 (2.5GB WAN)=====> Dell T7510 NIC 1 (2.5GB LAN) =====> 8 Port Non managed switch ======> all other devices

============

Issue

When I install PF Sense (or OPNSense), the installation goes fine, and I can see settings and everything looks fine. But when I perform the restart, I lose all connectivity and cannot get to the outside world. I am attempting to connect using the 2 - 2.5GB connections (in/out idea)

===========

Attempted solutions / troubleshooting

I have tried quite a few things to help isolate the issue and figure out why I am losing my network. I believe its something pretty damn simple that i am overlooking and at this point, mental fatigue of trying everything has set in.

  • Changed internal settings to be 2 core CPU, 4GB Ram
  • Changed from 2.5GB NIC to 1GB NIC connections

- Used older version of Proxmox

  • Changed internal settings to be 2 core CPU, 4-8GB Ram, 100GB hdd
  • Changed from 2.5GB NIC to builtin 1GB NIC
  • Tested/checked wires
  • Verified no MAC sticky going on ( was certain it was this)
  • Rolled Proxmox back to slightly older version (up to 2 older versions)
  • Tried with OPNSense
  • Tried 1 older version of both PF Sense / OPNSense
  • Ensured no other devices were connected when setting up (disconnected Proxmox)
  • Tried rebooting after installing OS but before setting password
  • installed on a HDD 7200 instead of SSD
  • Installed on m.2 instead of SSD
  • Tried manually setting LAN (identifying) instead of auto finding NIC
  • changed ports after configuration and reboot (this has been the least successful)
  • RAID on/off to see if it made any difference

From what I can tell, this seems to have something to do with running a dual nic system (in this case having 4 ports available) and all the guides and “how-to” skip a setup like this. I cannot find any information that would help point to “AH HA HERE IS THE PROBLEM!!!” and would greatly appreciate some help from anyone who has ran into this issue. Pretty sure its something to do with either how my bridge is setup, or something pretty simple that i am just not recognizing. Again, any help is greatly appreciated as the mind is mush but the flesh is not.

2

Post up your /etc/network/interfaces file. FWIW, I am not sure you can do this with an unmanaged switch. Are you putting everything on the same VLAN/network segment? When I was running pfSense on Proxmox, I used four NICs, and I assigned each of them to a separate bridge. VMBR0 was assigned to the LAN port on pfSense, VMBR1 was assigned to WAN1 (I had redundant WAN connections), VMBR2 was assigned to WAN2, VMBR4 was assigned to my dedicated/non-routed storage network, and VMBR3 was assigned as a second port on the Proxmox interface, this way I could always plug in and access Proxmox if the pfSense VM died.

You are likely running into issues with network conflicts/network design issues rather than Proxmox issues.

I used to do it like this:

auto lo
iface lo inet loopback

#===== LAN Trunk (Mellanox Port 0 - top port) =====

auto enp3s0
iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet manual
bridge-ports enp3s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-100

#Primary Proxmox management on VLAN 100 (tagged on vmbr0)

auto vmbr0.100
iface vmbr0.100 inet static
address 192.168.10.6/24
gateway 192.168.10.1

#===== Storage VLAN / secondary Mellanox port (Port 1 - bottom port) =====

#Dedicated 10GbE port for storage traffic (VLAN 3, non-routed). Physical switch port should be:

#- VLAN 1 untagged/native

#- VLAN 3 tagged

auto enp3s0d1
iface enp3s0d1 inet manual

auto vmbr4
iface vmbr4 inet manual
bridge-ports enp3s0d1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-100

#Host IP for storage network on tagged VLAN 3

auto vmbr4.3
iface vmbr4.3 inet static
address 192.168.3.10/24

#===== Emergency / Rescue Management =====

#Dedicated bridge with a private subnet and NO gateway

allow-hotplug enx00e04c68021a
iface enx00e04c68021a inet manual

allow-hotplug vmbr3
iface vmbr3 inet static
address 192.168.99.6/24
bridge-ports enx00e04c68021a
bridge-stp off
bridge-fd 0
# no gateway here by design

#===== WAN1 (Realtek 2.5g) =====

auto enp1s0
iface enp1s0 inet manual

auto vmbr1
iface vmbr1 inet manual
bridge-ports enp1s0
bridge-stp off
bridge-fd 0

#===== WAN2 (Realtek 1g) =====

auto enp2s0f0
iface enp2s0f0 inet manual

auto vmbr2
iface vmbr2 inet manual
bridge-ports enp2s0f0
bridge-stp off
bridge-fd 0

source /etc/network/interfaces.d/*
3

Thanks for the thought Louie1961. What you had in the past is pretty damn close to what I am working on now. And basically that is what I have for the layout. I am not using any VLANs right now, and everything is on the network. I can see all devices, but when setting up PFSense/OPNSense, it never finds the WAN. It can find the LAN but the WAN is never discovered. I will get a copy of that file listed shortly, but I have finally found someone who created a YouTube about setting up Proxmox with multiple RJ45 connections and I am going to try that and see what the results show. If that fails, a log file will be posted here for some thoughts.

4

Maybe it is lack of experience on my part, but I don’t see how you could even do this without VLANs. If you are doing this with a stand alone box, your WAN would be on a different network segment than your LAN and the two could never reach each other except through pfSense. The only way to do that in a VM is with VLANs.