Hello forum,
I’m facing some issues in my new virtual infrastructure in my small business.
Beware, it is a long post, and I’m sorry.
The problems I’m facing may be interconnected in some way so I feel like they should be mentioned in the same post…
The infrastructure comprises the following items:
Firewalls
-2x HPE DL320e Gen8V2 each with the following components:
-1x Intel Xeon E3-1270V3
-2x 8GB DDR3 ECC 1600 RAM (Total 16GB)
-1x 128GB SSD for boot (for now)
-1x Intel I350-T4 quad port 1gbps nic
-1x Intel X520-DA1 single port SFP+ 10gbps nic
Switches
-2x Dell PowerConnect 8024F with TOR (TopOfRack) duties
-Mixture of old switches for access to the various branches of my business
Hosts
-3x HPE DL380p Gen8 each with the following components:
-2x Intel Xeon E5-2690V2
-16x 8GB DDR3 ECC 1600 RAM (Total 128GB)
-1x HPE P420i integrated raid controller with cache and battery
-2x 146GB 15k rpm HDD for ESXi boot in RAID1
-1x Intel X520-DA2 dual port SFP+ 10gbps nic
Storage
-1x HPE StoreVirtual 4530 with the following components:
-1x Dual port SFP+ 10gbps nic
-12x 600GB 15k rpm HDD in RAID6
-1x TrueNas Core appliance with the following components:
-Head Server:
-1x Supermicro X10DRU-i+ motherboard
-2x Intel Xeon E5-2698V3
-16x 16GB DDR4 ECC RAM (Total 256GB)
-1x Intel X540-T4 quad port RJ45 10gbps nic (AOC addon from Supermicro)
-1x Broadcom/LSI 9305-16i HBA with break-out cables for external expansion JBOD
-10x 4TB 7.2k rpm HDD SAS 12gbps drives
-2x 1.92TB SSD SAS 12gbps for L2ARC
-2x 256GB Samsung PM9A1 NVMe SSDs on a bifurcation capable expansion card for SLOG
-2x 128GB Kingston SSDs SATA for boot of TrueNas
-JBOD Expansion:
-24x 4TB 7.2k rpm HDD SAS 12gbps drives
Firewalls are running pfSense CE 2.7.0.
They are set up in High Availability using CARP and VIPs.
Hosts are currently running an evaluation of ESXi 8.0.2 while I decide if I’m going to stick with it of downgrade to 7.0.2.
The TrueNas storage pool is made up of 8 vdevs of 4 disks each in raid-z1, for a total of 32 disks, plus 2 disks in hot spare mode, for a total of 34 disks.
I set up in pfSense the out of the box LAN network as an OOB network (OutOfBand) on a range completely different from production.
I also set up 5 VLANs:
-vlan100 → xxx.10.0.1/16 → Management Network
-vlan200 → xxx.20.0.1/16 → Office Network
-vlan300 → xxx.30.0.1/16 → Workshop Network
-vlan400 → xxx.40.0.1/16 → Utility Network (Printers and other dumb stuff)
-vlan500 → xxx.50.0.1/16 → Phones Network
These vlans are also setup in the 2 PowerConnect switches as they are now a stack with a master and 1 member.
All the cabling from the TOR switches is done with DAC SFP+ cables for ease of maintenance.
The workers thereare not delicate enough to be trusted with fragile fibers and transceivers.
The connections to all the devices immediatelly downstram the TOR switches are done with link aggregation and LACP over the 2 switches.
This means for example host 1 is connected to the TOR switches on port 5 unit 1 and port 5 unit 2.
FIRST PROBLEM:
From the OOB interface of pfSense I’m not able to ping anything on any of the vlans.
I have followed a loooot of tutorials and I’m fairly sure I set everything up correctly but something doesn’t add up.
From OOB I can ping the VIP of pfSense in another vlan but nothing downstream pfSense in that particular vlan.
BUT from any other vlan I can ping everything correctly.
Rules in the firewall are all the same, open everything to anything since I’m not connected to internet for now.
This makes no sense to me…
SECOND PROBLEM:
I want to have the management network port group of ESXi on the management Network so vlan100.
But as soon as I apply the vlan settings from the host console directly i cannot ping nor go to the webui.
The hosts are connected to LAG ports on the TOR and those LAG interfaces are set to trunk mode with AllowAll policy.
I would expect to reach the management interface of the host with it set like this.
And I expect this because the TrueNas Core head server is connected to the infrastructure in a very similar way.
And I can reach all the vlan interfaces I set on the TrueNas from anywhere in the network as I expected.
My TrueNas box uses the .15 address on all vlans, so I can reach it from all 5 networks indipendently from which vlan I’m trying to reach.
I will change it later to be accessible only via routing through the pfSense box but I’m still in the goofing around stage of things.
If any of you wants to lend me a hand it would be hugely apreciated 
Thanks in advance and sorry for the really long and wordy post…
Fabio