TrueNAS Scale and Proxmox

I am having the devil of time trying to get Proxmox to mount and access TrueNAS Scale NFS shares.

I am certain it is permissions but I cannot get it to work after many attempts.

I have the NFS sevice configed for NFSv4 and the default, neither worked.

I have a dataset configured with root:root as the owner. I have also tried other users as well. 777 permissions.

I can see the see and create the share from Proxmox using NFSv4. Tried the default as well.

When I upload an ISO It loads t the tmp directory but hangs when it tries to copy it to the share.

Not sure what to do here and any advice would be welcomed. Proxmox needs to connect as root to the NFS share but nothing I have done on the TrueNAS side allows that. I can mount SMB shares on my windows clients but not NFS between linux.

I don’t really use Proxmox but I never have had any issues with XCP-NG or VMWare and TrueNAS. Since Proxmox is based on Debian try to SSH in to it and manually mount the TrueNAS and see if there is an error.

So this is giving me headaches for a few days. I have TrueNAS Scale running in a VM with a dedicated nic and drives. My Proxmox host has a separate dedicated nic and drives. I am up to date on everything. I have set up datasets and shares in TrueNAS and can access the SMB shares with out issue from my Windows clients. The NFS shares are another issue.

When in a vm on the same host or another host, I can mount the nfs shares and have RW capabilities to the share. These are tested with vanilla Debian v11.3 vm’s with nothing but the nfs client added. Simple “mount -vvv xxx.xxx.xxx.xxx:/mnt/data-01/ds-proxmox /opt/truenas” and no issues coping, editing, or deleting files/directories.

When I am on the Proxmox host or any other Proxmox host and try to the same thing, I can mount the share and list the files but cannot edit them and attempting to edit anything locks up the system where I have to reboot the host to clear it. Probably a NFS busy state and the session is dead.

I have tried many different permissions and NFS3/4 on the TrueNAS scale side but nothing works from any Proxmox host. I even have a share set with POSIX OPEN and running into the exact same issue from the Proxmox host. This prevents me from created NFS storage, shared or not, on any Proxmox host where the NFS share is running on TrueNAS scale.

At this point I have to believe there is something going on with Proxmox related to accessing TrueNAS Scale NFS shares, as it is the only common denominator that does not work properly. I posted this on the Proxmox forum as well as any help would be appreciated. This sounds like a video opportunity to me and may post it on the Learn Linux forum as well to see if anyone can tackle it and figure out what the disconnect can be. If it is a Proxmox issue then I cannot and probably will not be the only one running into this.

Proxmox videos on not really on my to do list because I don’t use it. Jay has a full course on it and Craft Computing has some videos on it as well. Hope the Proxmox forums are helpful.

Thanks Tom! I think this might be a no_root_squash issue? When I look at the exports file on TrueNAS I see the following:

I believe that Proxmox has to have no_root_squash, I have not found any place that talks about how to do that with TrueNAS scale. Any ideas?

Nope, might want to ask in the TrueNAS forums.

Tom,
Thanks again for looking at my post. I have cross posted over at the Proxmox forums and the Learn Linux forums as well. After working with everything I think I am having some kind of networking or recursion issue. Which is why I am back here as well as I have always regarded you as a networking guru.

Basically, I am trying to mount sn NFS share on a Proxmox host from a VM running an NFS server on the host. So recursion right, NFS is lost, maybe. Can you mount a share from within a NFS server VM running on the same Xen host as the VM with the NFS server? I can see how this recursion could be an issue but I thought I set it up to avoid that.

Since my goal is simply to set up a TrueNAS server on each of my (3) servers, share almost all the drives from each server to each server host and VM’s running on those host. The idea is to abstract the storage from each host it self and use all the shared storage across the (3) servers. Three big data pools for each server to use.

My setup:

I have a single host v7.2-4. On it I have a single VM running TrueNAS Scale v22.02.2.

The VM has a dedicated NIC passed in via PCI Passthrough and all drives passed via iscsi plus no config on the host for the nic or ports. The host has a seperate NIC config’d with a bridge. The TrueNAS VM does not use the host NIC or drives.

TrueNAS is set up properly, pools, datasets, users, and shares all good. I have a single share to a dataset and the export file on TrueNAS shows RW and no_root_squash.

I created a vanilla Debian 11 VM and installed the NFS common package on the same host. As root, I was able to mount the share to the ds_proxmox dataset/share. In that VM running on the same host using the only bridge on the host, I have full access to the share and can cp, write, delete to my hearts content.

On the Proxmox host, from the commandline or GUI, I can mount the same ds_proxmox dataset/share from the TrueNAS VM. When using the GUI, it creates all the directories and sub-directories. At the commandline, I can create a file on the share by using touch. If I try to VI the file, it hangs up the entire mount point. In the GUI, if I try to upload an ISO, it uploads to the tmp directory and hangs when attempting to cp the file to the directory on the share. It does create a 0-byte file but that is it.

Why can I have full access to the share between VM’s on the same host as the TrueNAS VM but I cannot fully access the share from the host it self? It shouldn’t be a routing issue as the NIC in the TrueNAS VM is removed from the host OS via iommu and pci passthrough. The client VM’s on the host, except for the TrueNAS VM, share the same bridge as the host.

I get if I was all using the same bridge that NFS could get lost but not with PCI passthrough of the TrueNAS nic. So how does someone mount an NFS share inside a VM back to the proxmox host file system. I cannot mount it locally then add it to Proxmox as a Directory and I cannot mount it as a NFS share.

The only thing I know that works is if I map a host directory into an LXC container running an NFS server. Then I can mount that share from the container back to the host as shared storage. I have that working but that is no substitute for something like TrueNAS.

Again, thanks for even looking at this and pondering a response.

@quartzeye, won’t you describe your network topology for us in more detail please? Including the ports/bonds/vlans/bridges on your Proxmox host.

A peek at Proxmox’s /etc/network/interfaces config file will also be informative.

I know this might sound left-of-field, but take a closer look at your MTU settings: gui-vs-config-vs-applied.

Solved. I posted this over on the Proxmox forums as well ad the Learn Linux community.

Since I have a full 10G network, I took my MTU up to 9000 in order to utilize jumbo frames. This worked for pretty much everthing but NFS. And it only hangs up NFS when it actually is trying to affect data by editing a file with VI or copying a file across the mount point. Why. I don’t know but I suspect someone with deep networking knowledge could figure it out. For me, I guess I will live with the performance hit and throttle mtu back to 1500 in order to at least have NFS work.

I certainly would welcome an explanation as to why NFS can’t move the jumbo packets or how to figure out where in the network the chock point actually resides. I certainly would welcome a video from Tom, Jay, or both on how to diagnose and remediate this bottleneck.

1 Like