Network layout and configuration - need advice

Networking & Firewalls
1

Hi alltogether and first of all a happy new year and a better start to 2021 than 2020 has been all year long :wink:

i am new to this forum and stumbled across it via watching the good yt videos!
Thanks for alle the good tips in these videos. greatly appreciated.

as for my question, i am configuring my home network and homelab completly new and need some advice for my layout and how to configure it.

i am working as a backup administrator and have a diploma in business informatics and a degree in software engeneering, so no complete newbee :smiley: but network, vlan and other things are not my favorite topic.

my hardware:

dell r720 as proxmox host with 128gb ram and 8x 2tb sas drives for zfs datapool, 2x ssd for zfs boot drive.
onboard 2x 1gb and 2x 10gb sfp+ nic
add on 4x 1gb nic

dell r720 with 64gb ram and 2x ssd zfs boot drive for truenas core
onboard 2x 1gb and 2x 10gb sfp+ nic

1sfp+ port on each dell is connected to the other to form a separate network only between proxmox and truenas for max performance. 10.0.0.x

aruba s2500 switch 48 port POE with 4x sfp+ ports
one 10gb sfp+ port is connected to each R720
the other 2x sfp+ 10gb ports are connected to 2 other 10gb sfp+ switches (zyxel xgs)
one of these in the same network, the other comes from a different office.

8x IP cameras for security, cabled, no wifi, poe

avm fritzbox (german brand) isp router for internet connection. not exchangeable.

3x ruckus r610 wifi APs via POE

1 ddwrt router (siemens se505, old one, only 100mbit) for connectivity between my network and the other network from office coming via the other zyxel xgs switch.
i want to get rid of that router as my aruba is capable of routing between networks and subnet itself.

fire tv sticks, smart Tvs, sat receivers (enigma2 devices) for entertainment.
2 kids with tablet and iphone
2 adults with tablets and smartphones
laptops via wifi
computer via cable
gaming consoles (ps3, ps2, xbox, xbox one, psp, psvita, switch) via cable or wifi

proxmox:
pihole
plex mediaserver with data on truenas
syncthing coming on truenas as container (thanks to your youtube guide)
nextcloud, data on truenas
opnsense (and thats the point)
iobroker for smarthome
development for webpages
apache guacamole
gaming windows system with dedficated graphic card
windows system with dedicated graphic card for video and gpraphic editing
several testing enviroments
other things coming soon

truenas:
synthing as container, nothing else planned
datastore for VM backups
datastore for data :smiley:

syncing to remote truenas in other network (office)

my problem is… where to start and how to set up vlan, routing, networks etc

my goal should be to have everything separated from each other, so firetv stick and TV from kids wifi (vlan).
kids from parents.
cameras from everything else.
but nonetheless i need acces from every corner to the other.
so the kids should use plex to see holiday pictures and tv series i recorded.
fire tv sticks should use plex as well, plex has datrasotre on truenas core and so on.

my plan:
vlan for
kids
entertainment
smarthome
gaming
camera
network coming from the other office
guest
regular , everything else or our laptops and tablets etc… parents so to say.

wifi is separated already but no vlans configured
kids
gaming
entertainment
guests
normal wifi
smarthome (IoT)
motion (linux software for camera)

i need access from the office to my truenas for replication, my truenas to the office and vice versa.
office (my parents) should have access to my plex
plex access to truenas
entertainement access to truenas
kids to syncthing for syncing their fotos as any other mobile device in out house (all android onyl the kids iphone).
my wife and me access to nextcloud, nextcloud access to truenas for datastore.

avm fritzbox (ISP Router)
----------------------------------------> aruba switch
----------------------> proxmox
----------------------> opnsense

i want everything through opnsense and then through pihole, so i need to configure opnsense as gateway and DNS for all devices.
in opnsense i need to configure pihole as gateway and dns, right?
and in pihole i need to configure my avm router as gateway and dns, correct?

how will i handle the different access controls as i dont want any network to be aware of the other.

do i need to configure every vlan for truenas that needs access to it?

is it better to have a second nic in the second network, for example the cameras.
the cam itself is in net a, the motion linux server in net b. is routing from b to a (not the other way) better than having a second nic in net a?

i hope you can understand what i mean :smiley: as it is quite a bit to explain.

so feel free to ask.

i hope you can help me out.
i have no experiance with opnsense or vlan in promox, but that shouldnt be a big problem.

thanks very much in advance!

Stefan

2

Nope, you do not need Pihole for GW job or doing DHCP. If your opensense is powerfull enough just let it do the job. In the DHCP Service Settings in OPNS hand out the IP of your pihole as only DNS to all clients. Make some FW rules to forbid all other udp/53 traffic not going to pihole. (Need to have a look to block DNS TLS traffic too)

No - if you talking VLAN you are talking Routing. This is what I ment with a very powerfull OPNSense. You set OPNSense as default GW for each VLAN in your network.
Else you could use your Aruba (Which is a very powerfull L3 Switch) to be the central GW but if you have no clue about it its hard. In your current “concept” you are basically using a Ferrari for grocery shopping. Nothing wrong with that but any cheapo switch would have done the job.

In case you want to go the aruba route read and google about:
ip routing
and
ip helper-address

Proxmox is not the Problem you can just add a vswitch and give all required machines native IPs in each VLAN they need to be present in. Only thing you will get a problem is pihole and vlan based host overrides. It’s not going to work. So better have some internal domains for each VLAN like: device.wifi.home-local device.kids.home-local …

Second NIC and static VLAN can be done but just setting the VLANs in Proxmox is more flexible and comfortable.

OpnSense as def gw and DHCP or Aruba as def GW with OPNSense as DHCP.

Each VLAN have a own Subnet e.g. /23

VLAN 10; 10.10.0.0/23; kids
VLAN 20; 10.20.0.0/23; wifi
VLAN 30; 10.30.0.0/23; lan

All routing is setup in opensense or the Aruba. In OPNSense you use the firewall rules to specify which device can access which VLAN or maybe only one device from this VLAN. This can be done in Aruba too but not as easy as in the OPNSense GUI.

In General when you start configuring more complex network setups, even if it is lab or home net, start small and continue to grow the config complexity or else you will be totally lost.

1 Like
3

Hi,

thanks for the advices… i will use opnsense as it is more usefull and easier than the aruba funciotns.
and yes i know, i am using a ferrari :smiley:

i now configured my NICs on proxmox as followed:

eno1 10gbe to aruba
eno2 10gbe to truenas (direct connection)
eno3 1gbe
eno4 1gbe

enp68s0f0 1gbe
enp68s0f1 1gbe
enp68s0f2 1gbe
enp68s0f3 1gbe

bond0 eno3 + enp68s0f0 for administration of proxmox, active-backup config. so ein port from internal card, one from plugin 4port card

bond1
enpX1 + 2 + 3 - bond for the 3 ports of the plugin card LACP (activated for these 3 ports on the aruba as well)

bond2
bond1 + eno4 active-backup failover for 3ports of plugin and 1 onboard port

vmbr0
eno1 - just 10gbe

vmbr1
eno2 - just to truenas

vmbr2
bond2

vmbr0 + 2 are vlan aware

but here comes my problem…

if i use wmbr0 or 2 as nic in a vm (or container) and tag the vlan with … lets say 70…
how do i have to configure the ports on the switch?

i am not familiar with these vlan settings… so sorry for my question.
if i set up 2 ports on vlan 70 on the switch, they see each other, fine…
but not the one coming from proxmox.
do i need to set this port as trunk port? not gateway?
for now i only have the default vlan 1, switch mode access, access vlan 1, native vlan 1, allowed 1-4096, association all ports.
and one vlan 70, switch mode access, access vlan 70, native vlan 70, allowed vlan 70, switch port x +y from 2 different computer to test.

now i want the 3 ports with LCAP bonding to be able to use vlan 70 and other vlans i configure in proxmox for the guests…
what do i have to change, set, whatever?

hope you will help me out here.

also if someone could explain the different headlines like switch mode (access or trunk), access vlan, native vlan (if no vlan is configured, this is what it is set to, right?), allowed vlan
association should be clear, the ports using this vlan…

oh man many questions… sorry for that! but i really could need some help …

thanks!!

greetings

4

you need to define the vlan on the switch and tag that port or trunk. On Aruba if you have a trunk (bond / lacp) you need to tag the trk not the interface.

e.g.

vlan 1
     untagged 1-10,trk1,trk2,trk3
     exit
vlan 70
     tagged 1-10,trk1, trk2, trk3
     exit

in this case the Trunk 1-3 and Ports 1-10 are tagged VLAN 70. Important is also the PVID on your OPNsense LAN interface you need to work in untagged nativ VLAN 1 so the switch does. OPNSense VLAN 70 need to be tagged on the interface connecting to the switch.

The switch modes are mostly a UI “user friendly” definition. I have not much expericence with Aruba Web UI but on the CLI its pretty simple.

Native VLAN: Vlan all switches and devices use as default PVID for connecting to each other. You do not have to allow any traffic on that in the FW.
access: Access or Edge ports is a port role for end device connection, computers, other devices…
trunk: a connection between two switches transportiing all the VLANs (carefull in aruba unlike cisco you need to manually add the VLAN tagged to the trunks)
access vlan: the untagged VLAN on the access / edge ports

1 Like
5

Yes, just add a switch and give the IPs in each VLAN. If you are facing any issues related to the network layout, you can get it replaced by visiting siemens repair center. Hope that will help you.
Thanks.