ye… thats the plan… also why the aggregation is planned… for vLan30 primarily.
As it gives me “cheap” 10GbE SFP+ interfaces within the Unifi stack I use.
G
smart. I sourced a used US XG 16 for this. More expensive though.
1 Like
now t check/figure out if those Dell SFP+'s would be happy in the Unifi land.
sure they’d be happy in Topton land/hosts.
G
in case it wasn’t crystal clear regarding the hypervisors:
it seems that they have
- 4x 1Gbps or 4x 2.5 Gbps - In the following I assume that are 2.5Gbps NICs - if they are not, read 1Gbps instead of 2.5Gbps
- 2x SFP+ with 10Gbps modules
what i’d do would be:
- 1x SFP+ for storage VLAN (max speed for access to Truenas) to your aggregation switch
- 1x SFP+ for xMotion VLAN (hot VM mirgration) (max speed to migrate VMs between the hypervisors) - since your aggregation switch only has 8 SFP+ ports, use 1x 2.5 Gbps port instead to your Max 24
- 1x 2.5Gbps as trunk including admin VLAN (allow access to the admin VLAN where you should have the XCP CLI and XO VM for UI, and all other VLANs you have in your network - except for storage and xmotion)
that are 5 fiber optic cables to your aggregation switch and 10 RJ45 cables (2.5Gbps) to your Max24 switch.
Since the Max 24 only has 8 2.5Gbps ports you will have a 1Gbps trunk or a 1Gbps xMotion interface on 2 of your hypervisor.
regarding your truenas:
if it has 1x 1Gbps NIC, use this for trunk including admin network - except for storage and xmotion
if you have 1x 10 Gbps NIC: use this for storage VLAN, plug into aggregation switch
if you have 2x (or more) 10Gbps NIC, use LACP bonding of 2x 10Gbps and plug both into aggregation switch - this ensures that 2 hypervisors simultaneously stressing the TrueNAS can staurate both the 10Gbps links
1 Like
ordered with 4 x 2.5GbE and 2 x 10GbE SFP+
this going to so beautiful, I envy you
2 Likes
hehehe…
always need more ports…
like always need more TB’s
G
suggestion on STP’s settings for option 3A.
aggregation at 0, and MAX at 4096.
I have another 8port PoE Lite in the lounge which runs the entire media station/TV/AppleTV/Amp/xBox etc. will make that 8192
then have some Flex mini’s/5 ports, but well they just hubs so they dont get STP’s assigned.
RSTP, globally for all switches
Agg: 4096
Max24: 8192
8port: 36864
1 Like
and 8port PoE Lite, it’s patched into max PoE port
G
for access I have the US 8 60W PoE switches, but they are expensive. I was considering the Flex minis but I was unsure what capabilities they are missing.
can you deploy different VLANs to each port of these?
I originally had the flex mini 5, but ran out of ports so just upgraded to this. the flex mini was simply a hub, this is a layer 2 switch 
G
those are same price as the us 8 60w, but they have crappy plastic case instead of metal. is there anything those are better at than the us8 60w poe ?
this is behind the tv cabinet, mounted with their hang bracket, with devices just plugged into… so for location/purpose they perfect.
even the new 2.5Gbe PoE 8 port is big… and plastic, but it’s heavy solid, originally bought one and then took it back and swopped for the 24Port Max.
G
exactly, I was looking at the 2.5g version too and then found that the Max is not really more expensive…
I have a US48 and was thinking of replacing it with the Max24 to get 2.5G ports in my rack and I really dont need 48 ports in the rack. Dont need PoE there, it uncessecarily heats up the rack.
the 8port 1GbE PoE’s are pretty much the same, think the PoE Lite replaced the unit you listed. spec wise they look just about the same except for PoE output, and thats like 4-5w dif.
Ok, I’m logging off, sleep time for this hamster. thanks for the chat…
travelling atm, will be back home next week, then it’s build time. will post pictures…
G
Have a safe trip and enjoy the build, I hope all goes well with that.
1 Like
Little… 
diagram detailing last nights discussion,
Colour matching:
Blue => Fiber/10GbE SFP
Red => 2.5GbE Cat 6
you put proxmox 2.5G cables for xMotion and trunk? why not use another interface per proxmox to have xMotion and trunk on different cables, i.e. use 1G switch port for trunk cables
you could connect the router to the Max SFP+, so you can hafe 2 SFP+ on the aggregation swith for a LACP bond on the TrueNAS.
regarding your storage VLAN:
-
this would ideally ONLY carry the NFS or iSCSI traffic for your hypervisors, nothing else!
-
how do the VMs then store and access data (e.g. with minio)? you don’t run the data access layer (minio, nfsd, samba) on the truenas at for these. You run these in VMs on your hypervisors and they store data in VDIs. this has some overhead, but you have a completely isolated storage network that is not on the router and cannot be breached from any hacked VM, only if a hypervisor gets hacked on proxmox level.
-
so you run virual minio and virtual truenas on hypervisors where you need data for some VLAN (so you can have several TN instances one for each VLAN that needs a filer - best separation you can get).
-
of course you do NOT want to run any of the applications on the truenas itself due to security reasons! come on, qbt? this is a crazy security risk. Dont do this, please! All of these belong into containers in some VMs, where you can put each VM in a VLAN where it REALLY belongs security-wise.
-you want your truenas to be as vanialla as possible, only speaking NFS or iSCSI to your proxmoxes (I’d use NFS) - and nothing else! also, VLAN 30 only belongs on the SFP+ interfaces, nowhere else. if you really wanted to access storage through the trunk (VLAN 10) it will work as the access layer can listen on all interfaces.
for the matter of not confusing yourself: ensure that the host part of ip addresses is the same for each host in all subnets / VLANS (you dont do this for the proxmoxes).
I can run a 3rd cable per proxmox onto the 1GbE ports for trunk, leaving the 2.5GbE port for vMotion only, but then also I’m not running a prod environment and even if a vm needs to move, i’m not worried about the speed. the VM’s will primary be hosting a K3S cluster.
yes at the moment allot of my lab is as docker images on the TrueNas which I’m moving onto the proxmox cluster.
you could connect the router to the Max SFP+, so you can hafe 2 SFP+ on the aggregation swith for a LACP bond on the TrueNAS.
Re above… that was the difference between 3a and 3b, where we said either both switches go into the router or the Max hangs off the Aggregation switch.
VM’s will primarily be immutable and storage will primarily be MinIO for my use case.
TrueNAS is physical on the dedicated hw, it’s not virtual.