I’ve watched a lot of videos on this channel and currently use freenas, and Unifi as a result, I also have a windows box with my blue Iris setup on it.
My freenas hosts my plex, unif controller, adgaurd, resolio sync, etc as jails/ VMs.
I have a number of VNETs for core services, wifi, CCTV, IoT, etc. and a 4-way network card in my freenas box.
I’m now considering upgrading to a single more powerful / more power efficient box. I’m thinking of a 2u rack server. 2 x SPF+. I would have 3 raid disk pools … one for my VMs, One for CCTV, and one for general storage /NAS. this would also include upgrading to a 10Gb core network for server and key workstations.
I like the idea of XCP-ng to manage my hardware. I could have a hardware raid 10 for my VMs and another for my CCTV storage then a dedicated HBA for Freenas (for all the reasons discussed elsewhere when hosting FreeNAS as a VM). I would create VMs for my various services and perhaps combine some of them on a docker host. This will keep things nicely isolated and I feel it would be easier to manage my hardware allocation, in particular networks.
I also like the idea of using freeNAS as my base OS and my effective “hypervisor” and hosting everything within this including my blueIris server as a VM. It does feel like I’m not separating my concerns but then I get the benefit of being able to take advantage of ZFS and snapshots for everything. In this case I would just use one HBA and let freenas take care of the 3 arrays.
Some of my other considerations are power consumption it’s not cheap in the UK, and also when a power failure hits, I keep my essential CCTV etc, going and shut non-essential services down including FreeNAS. (CCTV has its own storage and offloads to FreeNAS).
I’m also toying with the idea of moving to pfSense mainly because I don’t like the uniFi firewall but also pfsense would be a cheaper way to route and firewall at 10Gb (instead of upgrading to a dream machine). If I did choose this route , XCP-ng seems like the best option I think?
Is there a good similar discussion (perhaps I missed a Lawrence tech video) that discusses this and why you might go one way or the other, pros/cons, etc.?
For what it’s worth, if you need the power it will no doubt result in more noise and higher energy bills ! Don’t know your budget but I think in the UK 10Gb is still pretty expensive those SPF+ modules are still pricey for my liking.
I would say you might want to consider having two lower powered boxes for the same price and creating a cluster for redundancy / backup, that’s definitely on my list.
Others seem to have bad experiences with USB so I’d stick with an SSD for the FreeNAS.
As for pfSense personally I favour it running on a physical device, there are various boxes you can buy off amazon that will do the job. Perhaps it’s better to invest in a 10GB switch rather than 10GB on the router unless you get 1GB internet.
I don’t know that I need the power … I just need to cut down by idle/tick over consumption which means more modern hardware and fewer devices.
My question is more about how to virtualise than the pfSense decision. I was just assuming that if I had pfSense virtualising in FreeNas was not a good idea, but if I didn’t go with pfSense I’m not sure if my virtualization strategy makes a huge difference. just wondering if I need the overhead of FreeNas constantly running and if there are pro/cons to virtualising everything within it.
I’ve been buying used SFP+ modules with LC optical connectors for around $7usd on ebay. OM4 cables are cheap, and you can get a Cisco 48 port gigabit with 2 10gb SFP+ ports for around $150 to $200 (again all prices USD). 10gb is no longer a scary or expensive proposition.
Heat and electric cost are indeed an issue. And I still don’t think I would virtualize my pfsense or my Freenas. It just doesn’t seem like the right thing to do. I also don’t like having pfsense run as single connection and vlan, too many things can jump vlans that may not be able to jump through the router if using separate cables. WAN comes in on one cable, LAN goes out on the other. If you are just doing routing for your internal networks, then router on a stick might be OK. Personal preference based on old school situations.
I’m planning multiple interfaces anyway. one for NAS traffic and one for services. if I had pfSense I’d probably have a dedicated SPF+ for it.
I could switch away from freeNAS. frankly I think I could achieve the same with OMV and BTRFS using hardware raid which would simplify my hardware if I went with hardware raid solution.
I quickly put together a diagram. pfSense aside, Is there any real difference between these?
I’d say xcp will give you more flexibility down the line, it’s handy to have the option to spin up full vms as required. I use OMV but it’s really a glorified samba server for me, you might encounter limitations in using it beyond a file server.
I use Proxmox which has the option to create containers, perhaps xcp has something similar with dockers to optimise resources.
Hmm I don’t know if you are asking the right questions. I don’t think RAID and ZFS are equivalent so you’re going to have to answer that question yourself. And your hypervisor – quite a bit of difference in virtualization between byhve and Linux KVM.
@daninmanchester
If you go with XCP-NG make sure you get a raid card in IT mode. XCP-NG has some quirks with their storage driver in BSD. It will always take the first available device starting from ada0 circumventing cam. If you have a sata HBA it will have conflicts, and even if it doesnt there might be still a driver realted issue with these cards. (I had both sadly, ordered a used hp h220 and cam assigns drives connected to it as daX so all of my problems were solved.)
Thanks. Yeah,I thought as much. I even thought about just using linux as a file server, as really all I need is a SAMBA server too. OMV just makes life a bit easier with a few bells and whistles and if I have a hardware raid cart OMV/BTRFS will be happy with that whereas with FreeNas I’ll need to do something different.
I didn’t know that thanks … If I went the freeNAS route I’d probably use it as my host OS and need direct access anyway.
I was thinking that if I went with XCP-NG I would just ditch freeNAS and move to OMV / BTRFS as it will run on the hardware raid no issue and I will no longer need the software raid. I just need a file server really.
@kevdog true, but if I go FreeNAS which I currently use I need direct access and then XCP seems a bit redundant. If I go the hardware raid route then all I need is a file server and OMV will be happy with the hardware raid so I can ditch freenas.
I guess unless there is any major advantage in terms of backuping , snapshotting, etc. either way that would be the thing that might sway me at this point. Otherwise as @neogrid said “Looks like six of one half a dozen of the other !”
I just wanted to make sure neither was a rabbit hole!
Bingo I started with creating a Debian CLI file server, however, I ran into several issues around permissions / protocols or encryption (I don’t recall precisely), I installed Webmin to give me a light GUI but I couldn’t get it to work as I wanted it to. It works fine for completely open file shares however.
OMV was kinda easier, plus I have 10 hdds in USB drive bays which OMV liked.
The other downside is OMV doesn’t like SyncThing it was previously supported but not any longer, RSync is supported but I’ve not set this up.
Thanks, I use USB for backups although I’m wanting to move away from this.
I use resilo sync for syncing my desktop/laptops now but wanted to re-visit syncthing (I had issues before) but it doesn’t have to be within OMV.
I’m getting beyond USB backups so my plan is to keep some of the old hardware, and create a backup box probably using unraid as I have a mix of 1TB, 4TB and 8Tb drives and could easily expand this with new drives. but not really explored this in any great detail yet.
I’ve been testing this approach and I’ve got everything up and running on a test machine. The issue I have is that XCP-NG does not support larger volumes. At present I’ve solved this by doing a direct pass through of the 8TB drive I’m testing with.
I don’t fully understand how raid controllers work. Does the raid controller present each raid as a single physical disk and so I would be able to do a direct pass through in the same way?
I don’t want to pass the entire controller through to a VM if I can avoid it otherwise I’ll need a second controller for my CCTV.
