MTU advice please

Networking & Firewalls
1

I have 4 machines all connected to a private network. The bare metal machine provider has confirmed that the switches for this network are set for a MTU of 9216. The related interfaces for this network are all set for 1500. All machines are running 1/10gbe network cards. Currently they are running 1Gbe.

1 - pfsense server
1 - ubuntu / nfs server
2 - xcp-ng servers

I would like to make the changes necessary to move all machines to this value. I would like to confirm the right procedure for each env.

pfsense: add value to the lan interface parameters and save changes via the web ui.
ubuntu: add mtu value to the netplan file for permeance and issue the following command:
ip link set mtu 9216 eth1 to make immediate changes.

xcp-ng: issue the following command on each host: xe network-param-set uuid=[network-uuid] MTU=9216 Will this persist reboots?

Questions:

  1. for each host type will there be an interruption in service when I issue these commands.? This is a production env.
  2. if one host is changed at a time what happens to the network traffic while there is a mis match in mtu values?
  3. the ubuntu vm’s running on xcp my understanding is that because the networks are virtual that the only need is to change the hardware setting on the host. The vms will act accordingly. Is this true?
  4. if I make the same mtu change on the WAN, knowing the internet routinely fragments packets, would this be of any performance value?

Why am I doing this?

My use case is that my customers have a mix of large and small files. The load profile is that the application loads a shit tonn of small files and a bunch of large files… 1-25 meg range. I am hoping that with a bigger mtu that I can get some performance gains by being efficient with packets moving about the internal lan and possibly to my isp’s equipment over the WAN interface.

Hope this makes sense. thanks for your consideration

2

Point 1: Yes, it is almost a guarantee that this will impact production since you can’t change everything at once. Plan for downtime.
Point 2: If one side sends greater than 1500 and the other side is set to 1500 traffic will be dropped.
Point 3: I would expect the VMs will need to be updated as well.
Point 4: Don’t do it on the WAN because upstream the ISP is probably only set for 1500. Traffic will get dropped.

If you are going to setup jumbo frames, first this will only function in your internal network and not beyond your internet connection. If you move forward, start with your network infrastructure such as physical and virtual switches (ex XCP-NG or ESXi). Then do your hosts. If you are running NAS to a virtual host I recommend you shutdown all VMs first.

Personally I don’t think you’ll see a performance increase if you are running at 1Gb or faster.