Yet another vlan topic

Hi all,
First post here. I heard about this forum from a podcast @LTS_Tom participated in (uncast with ed).
Briefly about me:
I’m a long term “homelaber” :grinning:
I was not professional in IT/Security field but an enthusiast.
I’ve been setting up home networks for me and my family, building pc, setting up smart home,… for a long time.
I even managed my small company it for 10 years.
For various reasons i’m making a move to career change in cybersecurity. And as you need to start somewhere, i’m completing google cybersecurity certificate.
Logically i want to enhance my home network security; obviously for security concerns but also to learn.
I decided to start with network segmentation. I watched videos and read articles on the subject. I think i understand what it is, the purpose and the way to define vlan.
But what i miss is how to assign vlans (particularly to dockers).
To be clear i’ll first summarize the gearr and services i have in my network.
For the network i have an unifi dream router, usw-flex, uap plus 4 non unifi vlan capable switches. I also have a usw48 poe but decommissioned for now.
As i moved from a big house to an apartment i had to reduce the number of servers (from 6 to 1, actually 2 but the second is a backup server only powered up when needed).
My server runs unraid. It has 2x10G and 2x2,5G interfaces. 1 2,5G is dedicated to vpro.
It runs a few vms (home assistant, jeedom, pihole, windows) and many docker containers (around 70 some of them published through nginx proxy manager like nextcloud, homeassistant or plex).
I also have a bunch of devices like laptops, iphones, tablets, smart tv, apple tv, cctv cam, poe zigbee controllers, logitech squeezebox, nuki and other smart devices.

For now i have a flat network.
My first step would be to leverage vlans.
I plan to keep it simple so i guess default, guest, iot and cctv vlans are a good option.
For the devices it’s not difficult. But for containers and vm what vlan do i need to choose?

let’s talk about home assistant vm. It needs to communicate with every smarthome device so logically i have to put it into iot vlan.
It also need to communicate with my zigbee2mqtt, mqtt, scrypted, homebridge dockers. So i would create a docker network on this vlan and put these containers in it.
Can i assume unraid doesn’t have to communicate with containers trough networks ? Control is made trough docker.sock and storage need through path mappings.
But my laptops and phones need to communicate with homeassistant and the containers.
I read somewhere tom put smartphones in iot vlan.
Where do i put laptops ? in the default vlan and then define a rule to authorize communication from default to iot ?

For NPM i would put it in he same vlan as the published service (iot for plex and home assistant). But for nextcloud ?
All these questions is why i have been waiting to tomorrow for so long to set up vlan :sweat_smile:
And this just the first step i also want to implement sso, install siem tools, …

I hope that i’ve not been too long :innocent: :sweat_smile:

I assume the Unifi Dream will continue to be the router.

From your idea I am missing the following VLANs:

  • admin: where all the admin interfaces of equipment (unifi equipment, unraid IPMI, etc) are and critical VMs
  • server: default and iot and cctv will need to store/access data on you unraid, so this will by default be one of the most exposed machines. You dont want this in your default network. A word of caution: when you put unraid containers on the same machine into several VLANs, if your unraid gets breached it will be game over for those networks. A more secure approach would be having VMs on the unraid, one for each VLAN where you need containers and then run the containers in those VMs. unraid is not a greatly secure hypervisor, I’d consider an additinal physical machine with XCP-ng to run the VMs.
  • dmz: home assistant is a difficult machine to place, specifically if you want to access it from the Internet on the go, and if you have other mqtt devices like Tasmota/ESPhome, which connect TO your home assistant MQTT broker. You want it in a DMZ then.
  • nextcloud also belongs into a dmz
  • entertainment devices like appleTV, gaming boxes are very different form smart home devices (you can keep them from access internal networks, but they will need internet - whitelisting this can be whackamole), I would suggest to segregate those into entertainment and iot (you can keep thos e from accessing the internet and the internal networks)
  • regarding phones i don’t see much difference between smarphones and laptops. if you keep iot from accessing the internet I’d put the phones in default / guest (depending on the owner).

for the communication between the networks: you will need controlled filter rules to allow some traffic, it is unavoidable. This can be a lot of work whitelisting the stuff. A slightly less taxing approach is thinking about the vlans numbers as priorities and allowing higher priority VLANs to freely access lower priority VLANs, but not vice versa.You will still find that somethings are more complicated than you so far knew, like relationships between NFS client and server, mDNS and SSDP for entertainment equipment etc.

This is not a simple task and I’d recommend same detailed planning before you implement it. Otherwise it will be kind of chaotic and you need to open to many ports between networks or have machines with interfaces in too many networks - degrading the achievable security significantly.

A last word of caution: if you start with the Unifi Dream router now, migrating to a more versatile / powerful firewall can be a nightmare (I did a warm migration coming from OpenWRT and going to pfSense - took me several days). I’d think twice about the choice for the router and start implementing the rest when you have the router you feel confident with will be your platform for years to come. Hint: you can export pfSense config and import it on a new hardware running also pfSense.

Indeed it’s very complicated. :upside_down_face:
And i’ve not talked about geographical locations.
Basically i have 3 locations: basement, first floor, third floor.
ISP fiber arrive on first floor so that’s were the router is. On this floor there are apple tv, smart tv, iot devices.
To interconnect the 3 floors i use the spare black fiber from isp :blush:
So i have 10G between basement and first floor, 10G between basement and third floor.
And 10G between third floor and mezannine where my desk is. The backup server is here too.
My main and only server is in the basement and host pretty much everything.
And you pointed it, with this setup it’s quite difficult to choose where to put the different interfaces. Where would you putt the reverse proxy interface ?
I can’t dedicate a port to dmz, i have only one physical link between router and the basement switch. I can’t neither put the server on first floor (no room for it and the das shelf is noisy)

you can have all or a selection of your VLANs per interface and switch port (use a port profile in Unifi lingo)

as an example: put all internal VLANs on the switch port connected to Dream router, same on the connected router port. This is called a trunk.

On Unifi device the “native” traffic (can be a VLAN) goes to the connectd parent interface (without any VLAN extension) and the tagged traffic (can be several VLANs) goes to the connected VLAN sub/child interfaces with the corresponding VLAN extension.

that’s understandable. I think for that i need a bunch of rules as by default unifi admin interface listen to all gateway ip of the vlans and route inter vlan traffic.

If i put unraid admin interface in admin vlan the risk should be limited.
I don’t think i’ll go for another physical server because last time i did this i ended up with 6 servers (3 nas for storage and 3 to have a fully operational proxmox cluster with live migration. When i start this way i always want more :grin:
But i agree unraid is not the best solution particularly for docker management. I think i may use portainer to manage docker networks, stacks. The big advantage of unraid is that it’s make it simple to backup all the persistent data of containers and start a simple container (thanks to templates). For stacks there is a plugin but options are limited.
There should be a change in next version so i’ll wait till unraid 7 is out.
In the past i used hyper-v and esxi. I also used proxmox and still have a small node at my son’s with site2site vpn link. But i don’t know xcp-ng.

Thank you i’m aware of that. :grinning:
I was talking about dmz. Generally dmz has a dedicated port.
For now I have 2 devices connecting to mqtt: 2 poe zigbee controllers connecting through zigbee2mqtt unraid containers. And the mqtt server is an addon on hassos vm so i think it’s running as docker inside hassos vm. So no need for mqtt to talk to outside. But zigbee controllers need it for updates.
Also i use a reverse proxy for nextcloud and haos, do i need these services in dmz ?

I can’t see clearly where to put these devices.
For example take audio devices:
Squeezebox needs to communicate with logitech server (docker hosted), haos, phones, laptop and internet for radios
And the same for sonos. And for squeezebox the caveat is the wifi protocol. For now i have a dedicated hidden ssid as it can’t connect to my wpa2 dualband wifi network.

you need to just let one jump host vm access the admin VLAN on port 22. you then have a admin VM in the admin VLAN that you can use via SSH and/or KASM Workspaces.

nope, each container can have vulnerabilities. then the attacker needs to break out of the container and it is game over.
container isolation is not the same as VM isolation on y type 1 hypervisor.

if you dont want a decicated hypervisor, i’d do the following on unraid:

  • give unraid a dedicated admin interface if you have one for that
  • give unraid a trunk interface with all VLANs that the VMs running on unraid need (if you have no dedicated admin interface, also include the admin VLAN)
  • do not use containers for services (I already know you won’t do that)
  • place KVM VMs in the VLANs where you need services and run docker in these VMs (if you want a UI, look at Portainer and Yacht. I’d recommend to use docker compose on CLI though, you can mix that so you can inspect/start/stop/pause containers using the UI)

yes, the dmz is the right touch point for connections from the internet.

this is what I mean with that things are more complex than you think. I don’t use squeezebox (I like owntone using airplay). Many of these audio servers use multicast e.g. mDNS and/or SSDP for device discovery and opossibly other things. This can be solved using the package UDP Broadcast Relay.

Still doing some research to figure out the right path for me to upgrade my network security and ease of maintenance.
i’ve done a small diag of the actual network:

I plan to create vlans in this way (and it can change as i don’t fully understand how complex it can be):

DMZ: for reverse proxy with firewall rules to allow only traffic from reverse proxy ip to ip of the services published with protocol accordingly
firewall will permit only inbound traffic from outside to reverse proxy ip only.

SERVER: for published docker apps and vm. Traffic to DMZ and from HOME

MGMT: for hosted apps not intended to be published (like monitoring, backup, unraid interface,…) traffic to internet, from home vlan

IOT: for iot devices and smarthome apps: traffic from home vlan, to internet

GUEST: obviously for guests isolated from other vlans

HOME: for laptops and smartphones of family members. Traffic to other vlans (except GUEST) and internet.

I also plan, after vlan implementation, to setup :

  • docker management with portainer as unraid dockerman, despite being very easy to start, can be quite frustrating when you start to have more complex setup.
  • a sso like authentik. Logging will be more convenient for me and add security for apps with basic auth. I have to find out if you can force auth trough authentik for apps not compatible with oauth.
  • dashboard like homepage
  • siem tool ( i have to choose which one)

Do you have concrete questions about this?

Things I can say off the bat are:

You can take a look at the linuxserver.io SWAG container for reverse proxy. Uses Lets Encrypt with auto-renewal and support Authentik, based on Nginx. Lots of reverse proxy templates for LSIO containerized apps.

dont allow all HOME IPs to connect to MGT. ideally you want a jump host VM that you connect from HOME using ssh with pubkey auth, and then tunnel the desktop UI traffic (KASM, RDP, VNC, whatever you use) through the ssh connection.

your UDR only has 1G connection to the switches. any traffic that you are routing will have to go through that, this is a major bottle neck in your setup when you work with VLANs.

On the other hand this 1gps link allows for cheap traffic monitoring using a RPI5 with Corelight athome (=CE).

if you have a router with a faster uplink, you need to work with a mirror port on the connected switch (mirrors the router uplink port) and connect ae more potent machine with the same speed interface as the router is connected. (mirror port is less expensive than physical taps that go beyond 1Gbps).

regarding SIEM you probably do not even know yourself yet, what you want to do with it? Depending on what you are planning to do you can go in different directions.

Personally I like Security Onion for the convinient pivoting. If you dont plan to investigate a lot and need quick pivoting, many other solutions can be your friend (Gravwell, Graylog, Grafana Loki,…).

Grafana Loki was very unintuitive to configure, BTW.

What ever you choose, you can pull in the traffic sensor data (see above), usually Zeek logs and Suricata alerts and also direct all you syslogging to the log repo. The ultimate kicker is to have the logs host-based agents that let you connect the dots between network connections (ports) and host processes (pid, process name).

I found Gravwell to be very easy to set up, but pivoting around is by far not as comfortable as in Security Onion. SO uses Elastic, which is more resource intensive than Gravwell. Loki is the lightest but I wouldn’t recommend to use it as a SIEM.

SO, Gravwell and Graylog have mechanisms to pull in cyber threat intelligence indicators that you can use to enrich the logs or compare against. You need a CTI repo, like MISP. Gravwell has no good integration for MISP, kinda sucks. SO/Elastic supports MISP, but setup is also some work.

Another option is to run Zeek and/or Suricata on a pfSense router (not the UDR). I have not experience with this, but then you need to connect it manually to SO as you are not using their own sensor setup.

An additional thing to look at is network monitoring. This is usually more useful to early on detect things going south. I am very fond of Zabbix.

Thank you for your answer. :grinning:

Yes. Does this make sense ?
I don’t know, given my constraints, if i can achieve that with my main server.

I already use nginx proxy manager which i find easier to configure. In fact in the past i used swag and swap for npm as i struggled to proxy nextcloud.
But i love lsio i generally use their containerized apps.

i can do that but where do i put the VM?

The switch in the basement with 8xSFP+ is a L3 switch. I certainly can do something here. i have no need for high speed on first floor.
Perhaps i will replace UDR but i’d like to stick with unifi i have a bunch of wifi ap.

I will finish with a few more hosts in my basement :grin:

I tried splunk a few years ago but at the time i found the learning curve to steep and i didn’t have time. I also installed graylog but as with splunk i didn’t find plugins for my environnement. I struggled a lot with regex. :pensive:
First thing i want to do is to learn. And regarding siem the more obvious is to ingest, parse and analyze logs from router, reverse proxy and proxied apps.

I will explore security onion. As i understand if i continue using unifi gateway i’ll struggle to use correctly a siem. I find somehow reassuring for now to rely on unifi ids/ips.

i used zabbix 10 years ago in a windows network. If i remember correctly it’s far better when you can use agents. Right now i use snmp only for monitoring my ups.

I think it it realistic.

I am actually using LSIO SWAG with LSIO Nextcloud AIO because I was struggling with the original AIO container together with SWAG.

There are many options. One with many upsides is the following:

you have a central admin machine within the MGMT VLAN that you will use to access all other machines in that VLAN. On this machine you have at least sshd, KASM (or VNC) and your central cron jobs, possibly Ansible playbooks.

Your MGMT VLAN can be NATed on the router, so that any connections originating ftom MGMT look like the originate from the router, in order to hide any information about the machines located on the MGMT VLAN. In that case you need a port forwarding on the router which allows your jump machines outside of MGMT to connect to the admin machine within MGTM.

The jump machines (VMs) are in some high priority VLANs, e.g. HOME. Plural for failure resistance, each hosted on a different machine. Only the jump machines can access only the ssh port of the admin machine through a port forward and need to authenticate using public key.

If you do L3 switching, ensure that you do not switchroute traffic that shoudl go through the router for firewalling.

You can use the Unifi products independently for each other, you can use the Unifi switches and APs without having to use any Unifi router. You can use any other router (e.g. running pfSense) and combine it with the Unifi swicthes and APs.

If you found Splunk hard to use, you will find Graylog to be very similar, and Elastic Search isn’t easier to use out of the box. However, Security Onion provides a lot of useful dashboards on top of Elastic to allow comfy pivoting through the data. I would also think this is the most prospective choice for you.

Here is a simulator that helps you building regexps:

less router, more firewall. Dont forget DNS logs, very very helpful.

The Unifi isnt a problem here, you just may not get the logs out of it that you would like to have, but it is not a showblocker.

WHat you need to implement SO is a server where you put everything except for the network sensor. For the network sensor you ideally use a physical machine that matches the processing capacity for the router uplink and has a NIC of the same speed. Then you mirror the router uplink on the switch where the router is connected and connect the mirror port to the network sensor machine.

You definitely want to have such a network sensor, having the Zeek logs is gold for troubleshooting and security investigations in your SIEM. It also comes with the Suricata IDS and it also does full traffic capture for alerts and file transfers.

Yes, you can get the agents for the most operating systems and you can use snmp for all network equipment / UPSes. Under Linux you always have the agent in the pertinent package manager, albeit a possibly older version. If nothing else goes, you can still have Zabbix use ICMP ping to tell you when something turns unresponsive.