I’ve started learning about and experimenting with self-hosting for like a year now. However, I haven’t yet made use of the hardware I bought as I feel I still lack sufficient info to commit to a certain system architecture. I appreciate all the feedback you can share with me.
Available servers:
Terramaster F4-424 Pro + 4x8TB WD Red Pro + 1x256 Lexar NM 620 SSD
Install PVE and provision a VM with most/all disk size on F8 SSD Plus as main server? or install TrueNAS as pure NAS with a separate app server (MS-01)? or install TrueNAS with apps/docker all consolidated on F8 SSD Plus? (Note: 10Gbps link is available between MS-01 and F8 SSD Plus)
other recommended architecture?
Backup philosophy for recommended architecture?
Personal thoughts so far:
The build I am leaning towards:
Install TrueNAS on F8 SSD Plus, acting as main NAS.
Install TrueNAS on F4-424 Pro, acting as backup NAS via ZFS replication.
Install PVE on MS-01, acting as main app server, with VMs and docker. Mount main NAS as NFS share inside docker containers as needed (e.g. NextCloud and Immich).
Reasoning: by separating server roles, I can offer easily scalable servers (NAS and app are separate). I can also switch app servers easily in case of fault without having to worry about data, as it’s stored on NAS.
Worries: I have doubts about maintaining data integrity in case of communication faults between apps and NAS shares (e.g. NextCloud AIO docker running on PVE VM with datadir on NAS). In general, I worry about overcomplicating the setup.
From an overview standpoint having separate APP and NAS makes sense. Also having one TrueNAS backup to the other TrueNAS via ZFS replication or having data synced via Syncthing while taking advantage of snapshots is good.
Nextcloud: I know it’s not a popular opinion but Nextcloud is buggy which is why I don’t use it for documents. I use the less popular solution of using Libre Office and local documents on my system synced in real time via Syncthing.
Password manager: Keypass is fine but for home use I would not self host, I use Bitwarden and they do support local export and backup even when not self hosting.
SSO I don’t bother but there are only a few people besides me using things and the added complexity of managing and maintaining an SSO system is more than setting a few peoples passwords.
It is very hard to damage those devices by experimenting with software. Presumably there is no data on these devices yet? If not, then just work with a copy of your data and leave a copy on Google or where ever it is today. If there is data on these devices, your first step is to make a safe copy and set it aside.
Looking at the hardware specs for the Terramaster machines, I would not use them as virtualization hosts. That falls to your MS-01 exclusively in my opinion. I am really torn about using TrueNAS on the Terramasters at first. I think it is the right call, but my issue is sequencing. I don’t want you to struggle with three partly configured/incorrectly configured devices and then trying to troubleshoot when something breaks. There’s two approaches and I am not sure which would be best for you: start with the F4-424 only, install TrueNAS, get it running and stable, then move on the F8, and work backwards to the MS-01, etc. The other approach is to leave TOS installed on the Terramaster devices and don’t mess with TrueNAS until you are comfortable with Proxmox running on MS-01. I think a big bang approach for you would be more trouble than its worth.
That being said, I mostly agree with your architecture that you are leaning towards. the F8 becomes your main NAS, the F4 becomes your backup and the MS-01 running Proxmox. What I would probably change is the following:
Your worry about overcomplicating the data setup is valid. That to me seems to be level 200 homelab stuff. To solve that, I would invest in bigger drives for your MS-01, and relegate the NAS to mainly backup duties. A lot depends on how much data you have.
I would run Proxmox Backup Server (PBS) on the F8. You could run that in a container on TrueNAS, but honestly running it as the sole operating system simplifies your setup considerably.
At some point you may want to run that F4 device off site at a friend or relatives house. Having all of your backup copies in one building is not a sound backup strategy.
If you want to fully install TrueNAS on the F8, and use it for things like storing VM disks over NFS or as your data store for Nextcloud, your NVME drives will not have the greatest write performance over NFS or SMB, because they lack DRAM cache and they lack power loss protection. Without power loss protection, the drive only acknowledges the write after it is complete, adding a delay in the process. With power loss protection the drive acknowledges the write immediately, creating a real performance advantage. I would add some Micron 7300 Pro drives to the empty slots for that type of workload. They are enterprise class drives with DRAM cache, power loss protection and they are 2280 length so they will fit.
Whatever your setup you want to make sure you have access to your data, when things fail. I’ve not used TrueNAS, but if the physical Terramaster failed could you access the drives in another machine or has it formatted the drive in way making it tricky.
If you have the money filling up the PVE with more RAM is never a bad idea. Proxmox with VMs and containers is a good staring point.
My setup is a QNAP NAS which acts as a datastore, in Proxmox I’ve setup a headless Debian server that is connected via USB to 8 disks, over SAMBA I can connect to these disks. It’s not super fast but dirt cheap
Terramaster and TOS use BTRFS if memory serves me correctly. Its been a few years since I owned one. But yeah, the data should be accessible from any modern linux system. Same for ZFS volumes from TrueNAS.
Have you considered tailscale? Minimal setup, easy to deploy, and from what I can see pretty secure. Only thing I can think of is the number of users, if it’s > 3 there’s a small monthly charge.
Synology NAS have a new(ish) feature, immutable shares, ideal for photos, books as you can add but not change or delete files.
First of all, thank you for all the great content you make, Tom! It’s been of great help! I’ve actually used several of your guides to setup the test environment I am using at the moment.
I thought things got better with Nextcloud since their adoption of the AIO deployment. In any case, I will definitely look into using syncthing, instead. The “untrusted devices” seem like a very interesting concept, might allow me to keep a copy of the critical data on the public cloud. The main aspect that would worry me is the configuration required on the phones of my remote family members, and the change of mentality from what they’re used to (Google Drive).
Yeah, I see your point there, makes sense.
***
Hey Louie, thank you for your detailed feedback!
You are right, there is no data on the devices yet. However, I really want to start moving data from my 4-5 different phones and PCs to the NAS already ahaha. I’ve been running a docker test environment on a Raspberry Pi 5 for like a year now, as well as a PVE on a Zimaboard 2 (should’ve just bought a TinyMiniMicro like I’ve later seen in the community). However, I do hope I am not biting more than I can chew :'D
About the phased deployment, I was thinking about PVE on MS-01 with a VM for docker apps, and TrueNAS on F8 (chosen mainly for the bit rot protection and block-level replication). The backup on F4-424 might be pushed to a later date, even if it makes more sense to deploy it, test the backup on a fictitious dataset, before actually putting real data there. I will have to evaluate that. The final setup should be a 3-2-1 backup, with a USB HDD for offline copy.
Unfortunately, I am not in a position to spend any more money, already broke the bank on this one. Also, the MS-01 has only 3 NVMe slots, it was never meant to host a lot of disk space. That’s why I was contemplating deploying PVE on F8 and consolidating the server roles there.
Definitely need to integrate PBS in the pipeline. That’s something I haven’t read about enough yet, though.
I understand the drawback from splitting NextCloud’s installation on 2 servers. That’s why I was contemplating installing NextCloud (& Immich?) on the NAS directly and not on the app server MS-01. However, it feels like it would introduce complexity in scalability and backups, having apps on NAS. Anyway, the explanation you provided afterwards is all new to me, thank you.
Thank you for the feedback, neogrid!
Maintaining free access to data without a hard technology/vendor-lock is one of the main goals. With TrueNAS, you should be able to read the pool on another TrueNAS setup if the physical server dies. However, I am still not sure I understand the ZFS replication and accessibility of snapshots on the backup device. Do I need to provide a new NAS when the main NAS dies just to recover the backup saved on the Backup NAS? Or can I access the data on the backup NAS directly? That’s on my “to understand” list.
Yeah, I would have loved to buy more RAM, but not feasible in this day and age :'D
That’s actually my go-to solution for remote access. I’ve been testing it for several months now, also from abroad, and it seems very solid. They upped the free plan to 6 users. I hope its security holds up as well :'D I’ve thought about NetBird and Pangolin, but as far as I understand, to completely self-host the solution you need a VPS. I’m not leaning towards that at this moment.
One thing that just came to mind is that you need to know how to decrypt your disks, if you encrypt them on their usual hosts on a NAS. Best to test that now when you have time and no urgency.
Oh yeah the other thing is to keep notes, it’s a bloody nightmare but perhaps setting up a wiki might aid with that.
