TrueNAS Scale - Minio - NGINX Proxy

Business IT/MSP Talk
1

Currently we run TrueNAS Scale with the built in S3 Server.
And as many of you noticed if you updated to the latest version, this is going away in future updates.
The S3 Server built into TrueNAS Scale is old, and needs to be replaced with a Minio App / Docker.

Right so deploy the app and you’re done right well no.
You need to either deploy an older version and upgrade your S3 dataset and hope it goes well.
Or build it again.

So we want to build it again and migrate all the 160 Buckets :smiley: Yea that’s fun…

Right, we are already fighting this for 2-3 days now… And do have some issues.

Be Aware this is all internal!

  1. The Build in TrueNAS Charts version of MinIo
    Works, without HTTP no issues…
    But all things that use S3 like our Veeam Server need a Valid Certificate for S3
    1.1 So you want to load the Certs into Minio you can do that by linking some Dirs in the docker container. Tried this many different ways. With a WildCard Cert that we already had and a new requested cert from a CA. And both times the App will not start and spit errors.
  2. The Build in TrueNAS Enterprise version of Minio.
    Right problem solved then we define the MINIO_SERVER_URL and fixed right.
    No Same issue with the certiticates.
    Okay…
  3. Someone on the Forums of TrueNAS suggested running NGINX as a Reverse Proxy.
    Nice… Got that working, but now of course all traffic is flowing from through the NGINX Proxy server.
    Using NGINX Proxy Manager but can also be Plain NGINX.
    This is of course not what we want since it will cost performance.

Hereby a Flow Diagram to Visualize.

Data-Flow-Veeam-S3
Data-Flow-Veeam-S3554×505 37.7 KB

How can we;

  1. Make the real S3 data not flow trough NGINX Proxy Manger but Direct from VEEAM to Minio
  2. Even better do not use NGINX Proxy Manager at all and get the certificates working in MinIo.
    Really cannot seem to find a good writeup on this, and now tried so many times on a test system its not even funny anymore.

Hoping that someone here will have the answer.

For reference the load on NGINX Proxy Manager when running a job;

image
image373×704 15.2 KB

2

During the installation of minio there is a drop-down to add a certificate for HTTPS. Are you needing help setting up the certificate?

3

Where do you find that? Its not in the Charts version and not in the Enterprise Version?
Think what you mean is the built in S3 Service?

4

I am talking about the enterprise version and reading the documentation in figure 4.

If you are not wanting to run the enterprise version then you can take a look at the environment variables listed in the minio documentation to add when you are building minio.
https://min.io/docs/minio/linux/reference/minio-server/minio-server.html

I would argue that if you plan on using letsencrypt it would be annoying to update the certificates and the nginx proxy is the best route for this. Not to mention MinIO expects you to run it this way. Maybe you need to tweak your nginx setting for better performance. I would encourage you to read this documentation.
https://min.io/docs/minio/linux/integrations/setup-nginx-proxy-with-minio.html

5

Yes, i’ve checked the MinIO NGINX page, and used the exact parameters in a plain NGINX install. Same results. Traffic is going trough the NGINX Proxy… And cause quite a load on the server that is running NGINX… For internal that is not a wanted situation. Hoping someone has the right solution here.