I am in a bit of a pickle with no internet access in my home while I am supposed to be working remotely as a software developer. I’ll lay out my situation as best I can:
BACKGROUND
I have followed the Lawrence videos on setting up VLANS across pfSense, Unifi, and TrueNAS and it has been working well for years. I made a post about it in the Unifi forums and they helped me with the Unifi part, but now I’m struggling with connection to my pfSense box. The summary of where I am now:
I have an ADMIN VLAN that I use as the management VLAN for Unifi devices and the pfsense box allows ADMIN to talk to all other subnets/VLANS. All devices described are on this subnet, but there may be a problem with tagging that I’m not thinking of…
CURRENT PROBLEM
I can’t get any connectivity from the Unifi 16 Lite switch itself or devices attached to it to the pfsense box. Even direct connection from my PC to the pfsense doesn’t work.
If I connect directly or via unmanaged switch with no vlans like the TL-SG105PE that I had laying around, I can’t get dhcp from my pfsense router. The ONLY way that works is to connect my PC and pfsense to the flex mini switch that I described in the image in the following configuration:
Port 1
Port Profile: ALL
Connected to: pfsense
Port 5
Port Profile: ADMIN
Connected to: PC
Please help! I suspect something to do with the VLANS but it makes no sense to me why configuring the Unifi 16 Lite switch in the same way as the flex mini doesn’t work!!
I’ve been banging my head against this all day and could use a starting point on what to check/verify as far as my configuration goes. I haven’t changed anything on the pfSense side since it was all working yesterday and I have access to it if I move some wires around. Any help very much appreciated!
Ok so I was a little hasty in switching things around earlier - I am able to reach pfsense when I set the port my PC is plugged into as ADMIN VLAN. However, now I’m having a hard time managing other devices - I’ll leave this here for context provide an update if/when I need more help. I don’t want to waste anyone’s time with the original question
I got it figured out by carefully paying attention to the management vlans in unifi and setting a static ip on my PC for troubleshooting when I didn’t have a switch setting my outgoing traffic VLAN.
I have a broader question for the community:
It seems like the TrueNAS/Unifi/pfSense setup that I have (based on the Lawrence videos from a couple years ago) has a circular dependency that makes things very painful to debug when VLAN settings are incorrect. What are some tips and tricks to take away the pain? Take this scenario as an example:
A VLAN (or other) config change causes an issue with a Unifi switch not forwarding traffic properly - no DHCP and no access to even the switch from my PC
To fix this, I need to troubleshoot and/or modify VLAN tags on my switch via the Unifi management interface running on a TrueNAS jail
Because of incorrect VLAN tags, my PC can no longer access the Unifi management interface which is on a management VLAN
The TrueNAS machine has to be accessed directly, but it gets its GUI IP via DHCP, which it’s not getting because the switch connecting it to the pfsense router is misconfigured - this means I need to physically change my ethernet connections to route PC ↔ TrueNAS box after setting a static IP on my PC
Simultaneously, my PC doesn’t have the right VLAN tag to access the TrueNAS machine, forcing me to configure a new GUI interface address in TrueNAS via in-person console
Not a great experience. I just spent 10 hours on this exact scenario. I suspect there is a better way? It just feels weird that I can’t troubleshoot the switch until the switch is working AND TrueNAS is properly connected.
So how can I prepare things to be less painful if something similar happens again? I can think of one preventative measure:
Set static IP’s instead of relying on DHCP on all critical clients: unifi switch, TrueNAS interface, and Unifi Controller Jail. That way I can always at least connect to everything even if the switch isn’t communicating properly with the router.
DHCP is great but setting static IP addresses for critical infrastructure has always been a sound practice. As you have noticed, when some things don’t exactly work, being able to rely on critical servers always having some sort of connectivity is important.
For reference in my network all servers have static IP addresses and most clients (PCs, smart phones, IoT devices, etc.) have DHCP.
absolutely! for everything that is the backbone of you network: switches, routers, possibly APs, hypervisors, hypervisor storage backends, jump hosts (like Teleport), lights out management like IPMI, UPS UIs, DNS servers
no network shares for critical infrastructure like vdisks for VMs / containers that run DNS servers, hypervisor management, home automation, jump hosts, infrastructure management like Unifi Network app
redundancy for the core network infrastucture, at minimal for internal DNS servers and piholes, jump hosts, hypervisor management (like Xen Orchestra), backup appliance (for desaster recovery)
have a switch port configured providing the admin/management VLAN as untagged/native so you can access your important infrastructure even if routing/jump hosts fail
This is a good exercise. I have implemented these measures and I would highly appreciate additional input / ides from others.
Thanks for the detailed reply! Some follow-up questions:
no network shares for critical infrastructure like vdisks for VMs / containers that run DNS servers, hypervisor management, home automation, jump hosts, infrastructure management like Unifi Network app
Q2: What do you mean by “no network shares” for the Unifi Network app? As-in don’t make it only accessible via network share?
redundancy for the core network infrastucture, at minimal for internal DNS servers and piholes, jump hosts, hypervisor management (like Xen Orchestra), backup appliance (for desaster recovery)
Q3: When you say “backup appliance”, do you mean something like TrueNAS that has build-in local backups via ZFS partnered with periodic cloud backups?
have a switch port configured providing the admin/management VLAN as untagged/native so you can access your important infrastructure even if routing/jump hosts fail
Q4: In unifi controller, does this look like a port with the following settings?
I mean the Unifi network app and the data it uses should not use any remote storage at all. That means the app (e.g. the container) is stored on local disk and the app reads and writes app data only from/to local disk. Any storage dependency will render the app unusuable. In catastrophic situations or when making big changes your filers and/or other hypervisors may not be available, so if you need to make changes to the Unifi devices the Unifi app must run completely independent of other machines.
With backup appliance I mean the application you are using in case of data loss to restore data from your historical backup data - as well as the machine on which the application is running. You want this machine/application (hence appliance) to be self-contained as in independent from networked services. And you want to have a cold twin that you can use in case the appliance is rendered unusable. This appliance is the most important thing you have to bootstrap from catastrophic situations.
