Large file SMB Download pfSense issue

I’ve recently reconfigured my network and I’m struggling with transferring large files (over about 700Mb) when doing so across vLans, which get routed by pfSense (Layer 2 unifi switch).

It works fine on the same vlan / subnet and I get 1GB speeds.

When I do it across the vlan I get a sudden burst of speed which windows often says is higher than what is possible on a 1Gb connection (which is odd), then it quickly falls off a cliff to zero and I get unexpected network error.

Both NFS and SMB with the same result.

I see the same issues on all 3 of my computers regardless of whether wifi or hard wired although on wireless it doesn’t get that sudden burst of speed a fails a bit more slowly.

My linux server is OMV (Debian) running inside XCP-NG.
I have 2 VLANs within XCP-NG (VLAN 11 and 66) which I pass to OMV.
The switch port is set to “All” (Unifi)

VLAN 11 is my servers VLAN
VLAN 66 is my main computer network.

I have VLAN 99 for IoT and 77 for guests, so they are isolated but certain client need to access SMB within VLAN 11.

What I’ve tried so far …

I’ve simplified my network down to remove any complicating factors such as LACP.
I disabled all the offloading in pfSense incase it was a driver issue.
Tried setting the MTU to 1400.
Clean install of a windows client just in case.
Forced my linux file server to SMB 3.
Set the server MTU to 1400 (just in case).
Ensured there is an ICMP rule between subnets.
I’m using an Intel quad port Intel card the chips are embossed so its not a fake.
At one point I even swapped out my SATA controller and network card.
I have an SSD and an HDD and both on the server and both behave the same as to all their respective shares.

Is there anything else I should check?

ok, so this is like a self help group … so tell me about your problems … and what is it you think is causing them? …

Having written my post it occurred to me that maybe there were somehow multiple routes to the server.
I removed one of the VLANS so everything has to go via pfSense and hey presto, 1Gb line speed and no drop out.

that’s hours of my life I’ll never get back and I’m still not 100% sure how I have multiple routes.

I guess now I can put all my settings back.