Best way to set up a secure general purpose server for 'red and green zone' applications?

I intend to combine multiple applications on one computer. Which should include NAS functionality (green zone) and a f virtual machines. Among them websites (redzone)

The NAS functionality should be TrueNAS alike and the security more like XCP-ng. The server should connect to my pfSense router via multiple VLAN’s as related to multiple security domains.
(green zone (data storage / nas), redzones for websites, IOT-devices, etc, pc-lan, mngt lan etc.

When I started with that idea a year ago, my idea was to take TrueNas core as a base, and running the applications in jails. However, that does not work. My main problems: -a) terrible vlan support / vlan separation b) bad ipv6 support. Later on I did a trail with TrueNas SCALE however I stopped that, since I did not manage to setup separate vlans for the intended applications.

Then I looked tat XCP-ng and my impression is that XCP offers the option for clearly separated vlans there, … however not with ease … (I am somewhere halfway a script to setup the network part, not sure if I should really take that lane …)

Other issue, a bit less important issue is, that all those solutions are OK, as long as it relates to CPU and network sharing, but GPU sharing … no not a realistic possible at this moment in time …

The reason for this post is that, I am not sure which path to take:

  • TrueNas as host: I have lots of doubts for reasons described above (mainly vlan support)
  • XCP-NG as host: with true nas running in one of the VM’s
  • Proxmos??? No idea … ( I am not at all an expert on virtual machines)

So the question is what is the best way to implement this ‘general purpose’ server and why?


Not super clear why you couldn’t get your vlans working.

I don’t use XCP or truenas, however I’m running PROXMOX.

It’s a bit involving to get vlans running on PROXMOX but doable, it helps to use the on-board NIC for accessing PROXMOX UI, I’ve then got a LAGG over a 4 port NIC serving my vlans to my PROXMOX box.

When I create a vm I just select the network to use a particular address range for the vlan I want.

This setup works for me, not overly complicated, the vlans are defined in pfSense.

To explain my problem a bit further,

Proper vlan support should fulfill the following requirements:

  • a ‘VM’ should listen to one specific vlan

  • traffic arriving via that vlan, should also be answered via that vlan

  • outgoing traffic should be send via that vlan

  • the gateway is of course the pfSence vlan GW

  • this should work for both IPV4 and IPV6

  • each vlan should have its own routing table, default route etc.

A situation where traffic arriving via vlan A is answered via vlan B is completely unacceptable. A situation where multiple vlans share the same default route is completely ridiculous and unacceptable.

My experiences with TrueNas are that those requirements are absolutely not met. I think I can principle get it working that way, with XCP-NG, but not so easy / it is not the default behavoir.

It depends on how your firewall rules are setup. If you allow cross VLAN traffic in your rules then your criteria won’t be met. It’s entirely possible to completely separate VLAN’s with the proper rule set in pfsense. I personally wouldn’t use truenas for VMs unless to tinker with it. For my home lab and for stability reasons I have xcpng with any VLAN’s for IoT, DMZ, LAB and so on and none of them can communicate with each other.

Note that pfSense is not the problem. I have set up a significant number of vlans, which can only talk to each other as long as I have defined rules for that.

I do not have problems as long as each ^application^ is running on its own physical computer. But since that is not practical, my idea was to create one physical server with a couple of VM’s.

My original idea was to use my TrueNAS core system for that. But IMHO, the vlan implementation and the IPV6 implementation is absolutely not OK . After a few months, not having found a way to make it work the way it IMHO should, I gave up and decided to try TrueNAS scale. Partly other problems, but again not what I tried to archive.

Then I tried XCP-ng which comes at least a lot more in the direction I like …