I have a pfSense firewall (3200) and a homelab. Using acme and haproxy as a reverse proxy sounded like a great idea. I have been running Traefik for a couple of years now, and people say do fix what is not broken. I found the vid with Tom explaining how to setting it all up.
It is all working but the most important service, Nextcloud.
All services are running as docker containers, the ports for Nextcloud is actually 80 and 443.
I have added the pfsense to the config.php as a trusted proxy.
I can use Nextcloud using the server ip but not with its domain. Which is a Cloudflare domain.
I have an active ACME cert (I have 12 services), a backend for Nextcloud:
Those errors will give you a good place to start looking. HTTP errors in the 500 range are server errors and a quick search for 526 and 527 says they’re used primary by Cloudflare. So it seems like you might have a misconfiguration on that end.
Not when I use the Test Port toll inside pfSense I get an green ok for the domain with port 80 and 443 and all subdomains. All I changed was the the NAT rule from destination to This Firewall.
No matter what I change, I get the 522 error.
Arm I have one service - a simple one, this is what the HAProxy Stats say about it:
Now I am thinking of redoing the HAProxy setup from 0.
There are a few things that could account for your problems, but here’s what we’ll need to know to be helpful in diagnosing this:
Do you have a real, public IP address on your pfSense WAN port (one that’s not between 100.64.0.0 and 100.127.255.255)?
What is the LAN IP address of your pfSense?
What is the IP address of your HAProxy server?
What is the IP address of your Nextcloud server?
What do you mean in your initial post by Cloudflare domain? (Is the domain name registered with Cloudflare and/or are you using Cloudflare tunnels?)
Here are a few things that jump out at me:
Looking at the NAT table, I notice that the “Dest. Address” is set to the local address of your HAProxy instance (or so I assume). For that to work, the domain name (let’s say nextcloud.example.com) would have to resolve to that local address (which is unlikely if you’re hoping to reach it from the public internet). More likely, you’d want the “Dest. Address” to be your public IP (unless you’ve done something unique that I’m not understanding) and the “NAT IP” to be the address of HAProxy.
Looking at your HAProxy config file, it seems like all of these services are at 10.1.1.10. That would contradict your settings in pfSense, but we’ll know more once you fill us in with the info above.
On the Cloudflare front, it would be very useful to know more about that configuration. Is this a simple DNS record with Cloudflare or are you using some of their more intricate services? If it’s just a DNS record, does the entry match your public IP address? If you using something fancy like CF tunnels, you don’t need to use HAProxy at all.
Sure thing! So here are the things to look out for:
If you’re using the same IP for both pfSense and HAProxy, you’ll need to change which port the web UI listens on (only one service can listen on a port at a time—both can’t have port 443, for example). The way I actually like to handle this is instead to create a Virtual IP (say 10.1.1.2) and bind the HAProxy service to that. That way you can have pfSense’s web UI listen on 10.1.1.1:443 and all HAProxy traffic will have it’s own place (10.1.1.2:443)
The hostname on Cloudflare should resolve your public IP. If you’re doing separate DNS inside your network (like pfSense’s unbound DNS resolver), you can have that resolve to the HAProxy address.
The NAT rule should pass traffic destined for your public IP (or “This Firewall”) on ports 80 and 443 to HAProxy
The backend of HAProxy should point at the nextcloud IP, just as you have it. But I’d add the setting to check the service’s health (in bare HAProxy you just add check to the line with the backend, but I think there’s a drop-down in pfSense’s HAProxy). That way you’ll have it listed in the HAProxy stats page
I think that should get you going. For bonus points, since you’re using the Cloudflare DNS proxy, you can create a firewall alias with all the Cloudflare IP addresses and then in your NAT rule you can limit the source address to just those Cloudflare IPs.
If everything is working except Nextcloud, I’d focus on the backend and the Nextcloud server.
Based on the config you shared, visiting http://10.1.1.10 should load your Nextcloud server. Does HAProxy see that? You’ll have to enable health checks to find out.
You may also be able to get some info about what HAProxy is doing by visiting the site via curl. Something like: curl --resolve nextcloud.example.com:443:10.1.1.1 ``https://nextcloud.example.com. That will visit HAProxy directly and pass the appropriate SNI info to get to the Nextcloud backend.
That first bit suggests to me that the problem lies with Nextcloud and not with HAProxy. Something inside the Nextcloud stack is obviously doing that rewriting which would force the problem.
If I’m interpreting this one right, here’s what’s going on in the big picture:
That hits your router and is port-forwarded to HAProxy
HAProxy sees the SNI header and passes the traffic to the nextcloud backend at 10.1.1.10:80
Nextcloud is re-writing the IP address to cloud.domain.help, which resolves to your public IP and suddenly you’re back at the beginning of a loop, hence the timeout.
So check out that Nextcloud documentation about reverse proxies I linked up above. If you can’t get to 10.1.1.10:80, then neither can HAProxy. I’m not sure how you’re running your Nextcloud stack, but often docker setups include Nginx or some other reverse proxy that could be responsible for rewriting the request.
Here’s one other option that could solve this problem without messing with the Nextcloud stack: assuming you use pfSense’s built-in DNS resolver, you could set a host override that points cloud.domain.help to 10.1.1.10. That way when HAProxy gets the redirect from Nextcloud, instead of using a public nameserver, it’ll immediately resolve to the local 10.1.1.10 address. That means the loop would be broken and you should be able to get through.
Lets leave the troublesome Nextcloud out of this for now.
Now I have to very simple web apps active, same issue there. Time out.
Should I add them to the DNS resolver, but those run on specific ports?
Id like to reach speed.domain.help now, it is a one page web app.
Same troubleshooting steps apply. Try to visit the IP address and port directly. Check the HAProxy stats page. If everything is getting rewritten to the domain name when you try to visit it directly, then the problem’s on the host.
If you can load it directly without problem, (i.e. go to 10.1.1.10:5216 in your browser), then we need to look at whatever haproxy’s stat page and logs are telling you.
Thanks that is what Ill do. I discovered that my internal server was acting up, lost network connection at what not. So I have fixed that one and restarted this adventure.
First up is to open up port forwarding, without NAT this time,
I will also use ntopng to really watch the traffic happening in real time.
I have done this before, this exact setup, never had these kind of issues.
So I am back, after resetting everything, from server to Acme and HAProxy.
The only thing where I said, aha!, was on the frontend where you specify what cert to use.
Other than that it was smooth sailing.
Also I do think that the port forwarding was the key here.
For this I extend my heartfelt thanks to @tvcvt you help has helped me greatly!
So now back to the original issue… Nextcloud.
But I will create a new instance so no problems I pressume.
I found a bug in the pfSense haproxy plugin with the backend Server list section. I wonder if it was partially affecting you.
Basically the jist of the big is you cannot update that section. For example changing the port or turning encryption on or off. If you update a backend it’s hit or miss whether that change actually gets applied.
To check if it’s affecting you, look at the stats page and hover your cursor over the title of the backend. You’ll notice that your change did not follow through (the old port number is still being used).
For me I had to delete the backend Server list entry and add it again correctly. From my experience restarting the service does not help.
I was reading this thread with great interest since I, too, was having a lot of problems with HAProxy on pfSense. While it seems to work most of the time, I found this integration very finicky and challenging to maintain. Part of my complexity was that I also wanted to use Authelia for a few sites. When I was dabbling with Nextcloud, I recall a lot of complexity issues as well but, as it seems this thread is validating, is that the majority of issues with reverse proxy and Nextcloud are with Nextcloud. Aside from this reason, there were others that caused to abandon Nextcloud.
Recently I tried adding new apps and was again facing a lot issues with HAProxy. Even with AI help, which was helpful, I still could not get basic things working reliably; i.e.: changes made to backend servers were failing despite there being no good reason. This prompted be to explore other reverse proxies. I had also previously tried Traefik and struggled. Ultimately I chose Caddy2 which is really straight forward and simple for my use cases. I originally wired Caddy2 with Authentik but the CE version of Authentik has known issues with different types of authentication; i.e.: 1FA and 2FA for different services. So for me, Caddy2 with Authelia seems to hit the sweet spot.
I really wanted pfSense and HAProxy to work long term since it reduces my tech footprint with reverse proxy running directly on the router but alas, Caddy2 and Authelia were relatively simple to setup and configure in Docker.
Interesting - but as you said and what I did in the end where to redo everything, from the ground up.
And after that everything started to work. And I thought like KISS.
Yes, Nextcloud is something else, one thinks it is a simple webbapp but noooo, so much more and many moving parts and on top of that, a reverse proxy. Its a mouth full.
But when I redid everything even the reverse proxy part for nextcloud is working.
I have also used Traefik for years and really like the labels route. Caddy is something I have never managed to master, I have no idea why, it is just made in the wrong way for me.
Yesterday night I did go to battle with Nextcloud and lost. The reverse proxy part works every time without any issues. So during day it is time to get deep into Nextcloud to see what I can do.
Hmmm. Maybe I’ll explore Nextcloud again at some point since I changed reverse proxies. Everything with HAProxy always felt very finicky. Traefik also seemed like overkill for my home lab environment.
I do host services with Docker but also as Proxmox VM’s. I’ve never fully understood Docker labels. Maybe I should learn them. I tend use IPA:Port for most of my apps.
Good luck with your environment. I hope you get it sorted out.
I just remembered your post! Trying to get Nextcloud to work again behind HAProxy.
The issue have med me switch ports more than once with and without encryption.
Nothing worked, then it hit me. Got an error message stating that Nextcloud backend was down.
So now I have erased all the front/back-end even mentioning Nextcloud. And changed the port for Nextcloud in the docker compose. And now start from the beginning.
