“If you should avoid routing storage,” where should apps (such as immich, plex) be located?
Should the apps be running on the storage VLAN or some other place?
How do the devices (phones, computers, etc.) get access to the storage?
Still trying to get an understanding …
TrueNAS CE 25.10.0
There is more to unpack with this and not a straight forward answer.
Senario 1:
Senario 2:
The question about how do phones and computers access the storage?
Answer: You shouldn’t and should only be accessing the data though the app. But if you are talking about a senario where you need to add plex movies though SMB, for example, then routing SMB would be fine.
I tend to run my apps in a Debian 13 VM and not TrueNAS, so take this with a grain of salt. My Proxmox nodes, and my Synology NAS all have a separate 10gbe NIC and all are tied to a VLAN/subnet that lives only on my managed switch. It is not present inside of pfSense. Everything on my storage VLAN has a hard coded IP address (there’s really not more than a handful of IP addresses, easy to remember). All my data lives on NFS shares on the Synology, and I either mount the share in the VM, or more commonly, use the docker NFS driver so Docker can access the NAS drirectly.
If I was going to run my apps in TrueNAS, I would bind mount all of the persistent volumes, and there would be no need for a separate storage VLAN. Your apps in TrueNAS can proceed as per normal.
The point of avoiding routing of storage has more to do with speed than anything. And not much is going to be faster than a bind mount directly in TrueNAS to a TrueNAS dataset.
Another aspect in addition to performance (and performance is not just throughput, but also latency) is security by isolation.
I have 2 coarse categories of data:
reproducible media
irreproducible or personal data
Media are stored on an unRAID storage array and apps are running on the unRAID server. The apps access the media data locally, the users access the media data only through the apps remotely. The access to app on the unRAID server is routed. The storage has low performance and is less expensive as the drives may spin down when idle.
Irreproducible or personal data are inherently more valuable and are stored on a TrueNAS RAIDZ2 array. The storage has high performance and is more expensive as there are many parallel drives that may never spin down. This data has stronger security requirement than the media data. Therefore data is not stored in files on the array and then exported via SMB/NFS. Instead, data is stored in virtual disks used by VMs running on a hypervisor (cluster), the array functions as shard storage for the hypervisors. The hypervisors and the TrueNAS server and the hypervisors are located on a non-routed storage VLAN, which is not visible in the central router. So the hypervisors have non-routed 10Gbps access to the virtual disks using NFS. The data is accessed by apps on the VMs, the access is hypervisor-local regarding the filesystem and non-routed remote regarding the virtual disks. The users access the data through the apps on the VMs, where the VMs are located on the VLANs where the data is needed, i.e. the access to apps also is not routed. For this to work the hypervisors have a trunk network interface to be able to put the VMs on the VLANs.
Certainly, this is a trade-off between performance and security by isolation. Data on virtual disks is isolated by using VMs in different VLANs, where the VMs cannot access the storage VLAN, but only the TrueNAS server and the hypervisors. As the access is indirect through the remote virtual disks the performance is lower than when the data were stored on the TrueNAS filesystem and exported using NFS.
I guess I should add a few more details and then you can tell me that “I’ve lost my mind!”
pfSense on Netgate appliance with 4 unswitched LAN NICs (2.5Gb)
Unifi switch with two SFP+, four 2.5Gb, twelve 1Gb ports
Two (2) TrueNAS CE appliances: Main (better hardware) & Backup (repurposed desktop)
Backup is not configured (yet) for Replication. For now it’s just consuming electricity; storage pool setup; no datasets created; single 1Gb NIC
Main is intended be storage, shares with remote family members, some nice apps (plex, immich, cloud replacement [eventually]
Main has four (4) physical NICs available. Motherboard has two: 2.5Gb &10Gb, both RJ-45. Intel 550 add-in card with two (2) 10Gb SFP+. Main is NOT memory or storage constrained.
Here’s the path I started down …
On pfsense, use one interface for uplink to management network, use another uplink for all other uses (computers, phones, printers, TVs, etc). No VLANs defined in pfsense
On Unifi, define a VLAN to correspond to the non-management uplink. Configure the ports as necessary (I hope I can get that correct)
Main TrueNAS, has 2.5Gb NIC connected to management network & one 10Gb SFP+ connected to the non-management network.
On Main TrueNAS, use the Apps (custom docker if necessary). I’m using the Mgmt network to configure things.
Where should I point the storage mounts for plex, immich to take advantage of the 10Gb SFP+ speeds and “avoid routing storage?”
Does any of this make sense? I’m kinda noob at this; I had an old Core machine that failed and was able to put two TrueNAS machines into service.
Thanks for offering your insights.
You don’t seem to use dedicated hypervisors. So you can safely ignore my longish explanation further up in the thread.
Your reply was actually helpful. Thanks