Lab - Proxmox/Ceph network setup

Networking & Firewalls
1

Hey,

I got myself 3 PCs to “play” with and I’d like to set up Proxmox/Ceph.

I have pfSense as the main router and a Mikrotik CSS326 inter-connecting the 3 PCs.

Each PC has multiple NICs.

What would be the best way to set the network, so that eventually the Proxmox/Ceph (and specifically public/cluster) networks are isolated from the rest of the house?

Thanks,

M.

P.B:

Just to make it clear: this is not necessarily about getting the maximum performance, but just for practice.

2

Make a VLAN on microtik 10.0.0.*/24 and assign pc1, 2, 3

pc1 10.0.0.11 is proxmox cluster
pc2 10.0.0.12 is proxmox attached to pc1 cluster
pc3 10.0.0.13 is proxmox attached to pc1 cluster

home (192.168.5.*/24) is not affected

WAN --pfsense -- microtik -- vlan 
                 home        pc1 pc2 pc3
3

Thanks @pavlos !

This is a Mikrotik switch (it doesn’t have a DHCP server), I would have to manually assign IP addresses right?

No changes needed on pfSense for thus setup?

What about isolating the ‘other’ proxmox/ceph networks?

M.

4

The VLAN is another interface, just like WAN and LAN.

pfsense offers DHCP to LAN, similarly it can offer DHCP to the VLAN.

Not sure what you mean by other proxmox networks.

5

Let’s focua on a single network for now, like you suggested.

I should add a VLAN and configure DHCP Server in pfSense and then let the Mikrotik tag the port, right?

6

yes, I dont have a microtik to play with but I have a managed switch.

plug in pc1 in the tagged port, verify it can access Internet.

then build proxmox cluster with pc1, 2, 3

test that your home network cannot see the vlan or vice versa.

VLANs are just logical segmentation of a physical network.

7

Inside of pfSense you can create VLANs to segment your network. Is the Mikrotik CSS326 your only switch? If it is, I would create 3 or more VLANs on pfSense: One for your family traffic, one for the management interface on Proxmox, one for Proxmox to talk with Ceph. Each VLAN in pfSense can have its own IP address range, and its own DHCP server. You have to configure the tagging on the ports in the Mikrotik to accomodate all the VLANs. Family traffic will be excluded from the ports you use for Proxmox, Ceph etc. And visa versa if you want.

Within the Mikrotik I would create an additional VLAN tag that the pfSense doesn’t have. I would use that for your ceph storage network. you want to avoid routing a storage network and just keep the traffic on the switch.

In each proxmox machine, you would use one NIC for all the VLAN traffic and connect it to a trunked port on the switch. You would use the other NIC (assuming you only have two nics on each machine) to connect to a untagged/access port for your non-routed storage VLAN. I do this for my storage even without Ceph

Inside of pfSense, my “home” VLAN has VLAN tag 10, my server VLAN has 20, my Proxmox management interface has 30. In my switch I have VLAN tags for 10, 20, 30, AND 40. All my NAS devices, Proxmox nodes and any VMs that need access to the storage all have a NIC on VLAN 40, and I assign static IP addresses to everything on VLAN 40. The switch will move all the traffic on VLAN 40 without needing to communicate with pfSense as all. This makes the network set up faster. Intervlan routing is handled by pfSense for all the other VLANs. But this adds a speed penalty.

8

Thanks @Louie1961

The Proxmox nodes have multiple NICs (I can even go 2x4, which I might do to play with bonds later on).

This was the missing piece I was looking for.

In case of Hyper-Converged Ceph Cluster, would it make sense to have another VLAN/NIC available for internal traffic?

M.

9

I can’t imagine that the extra NIC would hurt.

10

Is there a need for an extra VLAN for isolating an extra Proxmox/Ceph network?

How would you use it…?

11

I would think so, but I am not a ceph expert. Think of VLANs as a way to create many networks out of one physical network. The more stuff that resides on a VLAN, the more congestion of traffic there will be. Giving Ceph its own network (i.e, its own VLAN) reduces traffic contention. The best scenario is if you can give the Ceph back end its own physical network. Absent that, a VLAN that is resident on the switch only keeps the Ceph traffic on the switch (no need for routing) and limits the contention.

12

I can easily give it a phisical NIC or even a bond of NICs.

Shall I also use a switch only vlan for those?

13

I believe so. But again, I am not a ceph expert so proceed with caution. Don’t accidently disconnect ceph from your Proxmox instance. There are normally two ceph networks as I understand it, one for the OSDs to all talk and one for Proxmox to access ceph.