Systemd mount and NFS v4 not working via direct link

Computer Hardware & Server Infrastructure Builds
1

This will be a long post, so find a summary of what works/doesn’t work at the bottom.

The setup:

  • TrueNAS server (24.10)

    • Interface 1:
      • 1gbps
      • default gateway
      • connection to LAN/WAN
    • Interface 2:
      • 10gpbs and jumbo frames
      • dedicated direct link with xcp-ng server
      • No routing
      • IP: 10.0.0.1/25

  • XCP-NG server (8.2.1)

    • Interface 1:
      • 10gbps
      • default gateway
      • connection to LAN/WAN
    • Interface 2:
      • 10gpbs and jumbo frames
      • dedicated direct link with TrueNAS
      • No routing
      • IP: 10.0.0.2/25
    • Has TrueNAS NFS share mounted as SR for VM storage

  • Ubuntu 24.04 server

    • Running as VM on XCP-NG server
    • Interface 1 - enX0 (via XCP-NG):
      • 10gpbs
      • default gateway
      • connection to LAN/WAN
    • Interface 2 - enX1 (via XCP-NG):
      • 10gpbs and jumbo frames
      • dedicated direct link with TrueNAS
      • No routing
      • IP: 10.0.0.5/25

I’m encountering a very strange issue when mounting a nfs v4 share through systemd mount AND through the dedicated/direct link (enX1). Mounting it through enX0 works fine.

With NFS v3, it works flawless no matters the interface, but the moment I want to use NFS v4 through the direct uplink (enX1), I can’t seem to make it work.
On TrueNAS I have disabled nfs version 3, and only enabled version 4 (when v3 is enabled, it works only when I specify to not use v4).

Doing it directly through the CLI on Ubuntu server works perfectly fine (no matter the interface or version):

mount -t nfs4 10.0.0.1:/mnt/data-dock/storage/minio /mnt/data -o defaults,_netdev

My systemd mount file looks as following:

[Unit]
Description=Mount the NFS share for data storage
After=network.target

[Mount]
What=10.0.0.1:/mnt/data-dock/storage/minio
Where=/mnt/data
Type=nfs4*
Options=defaults,_netdev

[Install]
WantedBy=multi-user.target

* NFS when v3 support is enabled

The logs on the Ubuntu server give me a bit more information:

mount.nfs: access denied by server while mounting 10.0.0.1:/mnt/data-dock/storage/minio

From this I assume that systemd mount for some reason falls back to version 3 (even when v4 is specified) on the enX1 interface and thus is getting the access denied. Even when v3 support is enabled on TrueNAS, but the systemd mount specifies v4, it will not work.

I have tried it with Ubuntu, Rocky linux 9, Debian bookworm and all have the same issue.
I have set the permissions on the NFS share as wide as possible and all clients can connect to it to narrow down possible causes.

TL;DR:

What Works:

  • NFS v3 (if v3 support is enabled on TrueNAS):
    • Systemd mount (on both interfaces)
    • Mounting NFS via CLI works (on both interfaces)
  • NFS v4 Mount:
    • Mounting NFS v4 manually via the CLI (on both interfaces)
    • Systemd mount (on both interfaces)

What Does Not Work:

  • NFS v4 (through systemd):
    • Systemd mount via direct uplink (enX1). Fails with the error:
      mount.nfs: access denied by server while mounting 10.0.0.1:/mnt/data-dock/storage/minio

Thank you for reading through everything

2

I would turn off jumbo frames and see if it works. They are often the cause of quirky issues due to how some switches handle them and they offer VERY little benefit.

3

Hey Tom, thank you for your reply.

I was able to fix the issue, to come back to your answer, there is no switch involved in the direct uplink.

The solution was as following:
I made another data set (with open permissions) directly under the root data/pool root directory exposed it as NFS share and it worked flawlessly with systemd mount.

Once I had a confirmation that it should work, I removed both the data set that was causing me the headache and its parent data set (also a NFS share).
After deleting them, I recreated both data sets, without exposed the parent dataset as a NFS share and it worked. I’m still not sure what exactly was causing me the issue (the fact that I originally had the parent data set exposed as NFS or something wrong with the permissions…)

Anyway, thanks everyone!

1 Like