PFsense HAProxy Wont Forward To Backend

Deacon587 · April 20, 2022, 1:16pm

Hello, im setting up HAProxy on my PFSense box and having some troubles. I followed a few different guides, and i feel like i have a decent understanding of what needs to be set and where.

But i think im missing something.

Setup:
Google domain name
Cloudflare account, created a “A” record for subdomain sub.domain.com to point to my public IP
Used ACME to successfully create a SSL certificate.
set up HAProxy’s back end, with the local IP of my end device
Set up HAProxys front end to listen on my WAN address port 8085 and to send data to the pre configured back end

Created a firewall rule to allow access on the WAN at port 8085.

When i look at my firewall log, i can see the entries for access on port 8085 of WAN, but i get nothing through the proxy, and the web browser errors with a “timeout”.

To test the forwarding of the domain to my public IP, i briefly exposed the port for the PFsense login and i was able to access it, so i know that’s working. (and i disabled that rule quickly)

Not sure what information to post that would help, but i can post whatever is needed.

Any help would be much appreciated.

Thanks!

LTS_Tom · April 20, 2022, 8:56pm

You need to bind the HAProxy front end to your WAN public IP and port and then create a WAN rule to allow the IP and port. Then the backend needs to attach to the server.
I have the entire process covered in this video

And I have a troubleshooting video as well

Deacon587 · April 20, 2022, 11:43pm

Tom! Thanks for the reply, I followed those videos to get to where I am now. And still having trouble.

I’ve verified packets coming through the firewall by logging my rule to pass them.

Heres pictures of my front/backend setup, and the text from the config file in the settings pages.

Ive redacted my public IP and domains, ive verified theyre accurate.

HAProxyConfig:

# Automaticaly generated, dont edit manually.
# Generated on: 2022-04-20 19:35
global
	maxconn			1000
	log			/var/run/log	local0	debug
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend vault
	bind			67.xxx.xxx.xx3:8085 name 6x.xxx.xxx.xx3:8085   ssl crt-list /var/etc/haproxy/vault.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	acl			vault	var(txn.txnhost) -m str -i vault.axxxxxxxxxxxo.com
	acl			aclcrt_vault	var(txn.txnhost) -m reg -i ^vault\.axxxxxxxxxxxxo\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	http-request  deny if { req.hdr_cnt(content-length) gt 1 }
	http-response deny if { res.hdr_cnt(content-length) gt 1 }
	use_backend vault_ipvANY  if  vault aclcrt_vault
	use_backend vault_ipvANY  if   aclcrt_vault

backend vault_ipvANY
	mode			http
	id			100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			Vault 192.168.1.3:80 id 101 ssl check inter 1000  verify none crt /var/etc/haproxy/server_clientcert_625efbcfc4f2a.pem

Deacon587 · April 20, 2022, 11:44pm

Deacon587 · April 20, 2022, 11:44pm

LTS_Tom · April 21, 2022, 12:47am

Your back end says “Encrypt SSL” but I will assume port 80 is running HTTP

Deacon587 · April 21, 2022, 1:47pm

Yes, ive been trying several different options to try and get it working. I have some success.

I Followed your video on wildcard certs and created one to use with all of my subdomains. Im using the same cert on both of my front ends, one is public and one is private.

I now have the following setup:
Frontend “PublicServers” which is listening on my public IP:443, and has ONE backend, “vault” which works, and can be accessed from the outside internet. Although, it seems to work, then not work, then work, randomly.

Second front end “PrivateServers” which is listening on my LAN IP:443, IP of the PFsense. There are 3 backends attached to this frontend. “vault”, “backup”, and “web”. These are to be accessed within the network.

I can access “vault” from both the public AND private frontends. However, i cannot access “backup” or “web” from the private frontend.

Ive verified the other two backends are setup identically to the vault backend, with the correct IP addresses and ports for the respective devices. I also have identical host overrides in the DNS resolver for all three vault, backup, and web subdomains.

It seems the connection is being made, it keeps going into timeout.

Do i need to turn off caching, or the cloudflare proxy on DNS?

I have wireguard setup, and it works flawlessly, so i know that i am able to make connections.

Here’s the config file, sensitive info redacted.

# Automaticaly generated, dont edit manually.
# Generated on: 2022-04-21 09:07
global
	maxconn			5000
	log			/var/run/log	local0	debug
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend PublicServers
	bind			67.xxx.xxx.xxx:443 name 67.xxx.xxx.xxx:443   ssl crt-list /var/etc/haproxy/PublicServers.crt_list  
	mode			http
	log			global
	option			httplog
	option			http-keep-alive
	timeout client		30000
	acl			vault	var(txn.txnhost) -m str -i vault.axo.com
	acl			aclcrt_PublicServers	var(txn.txnhost) -m reg -i ^axo\.com(:([0-9]){1,5})?$
	acl			aclcrt_PublicServers	var(txn.txnhost) -m reg -i ^([^\.]*)\.axo\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	http-request  deny if { req.hdr_cnt(content-length) gt 1 }
	http-response deny if { res.hdr_cnt(content-length) gt 1 }
	use_backend vault_ipvANY  if  vault aclcrt_PublicServers

frontend PrivateServers
	bind			192.168.1.210:443 name 192.168.1.210:443   ssl crt-list /var/etc/haproxy/PrivateServers.crt_list  
	mode			http
	log			global
	option			httplog
	option			http-keep-alive
	timeout client		30000
	acl			backup	var(txn.txnhost) -m str -i backup.axo.com
	acl			vault	var(txn.txnhost) -m str -i vault.axo.com
	acl			web	var(txn.txnhost) -m str -i web.axo.com
	acl			aclcrt_PrivateServers	var(txn.txnhost) -m reg -i ^axo\.com(:([0-9]){1,5})?$
	acl			aclcrt_PrivateServers	var(txn.txnhost) -m reg -i ^([^\.]*)\.axo\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	http-request  deny if { req.hdr_cnt(content-length) gt 1 }
	http-response deny if { res.hdr_cnt(content-length) gt 1 }
	use_backend backup_ipvANY  if  backup aclcrt_PrivateServers
	use_backend vault_ipvANY  if  vault aclcrt_PrivateServers
	use_backend web_ipvANY  if  web aclcrt_PrivateServers
	use_backend web_ipvANY  if   aclcrt_PrivateServers

backend vault_ipvANY
	mode			http
	id			100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			vault 192.168.1.11:80 id 101  

backend backup_ipvANY
	mode			http
	id			102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			backup 192.168.1.3:80 id 103  

backend web_ipvANY
	mode			http
	id			104
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			web 192.168.1.3:8080 id 103

Deacon587 · April 22, 2022, 7:44am

Ok, more progress.

Public front end:
I’ve seemed to have worked out the access from the public side (no idea how), which only proxy’s to one of my backends. I’m going to be checking it for reliability, as it seems to work, then not work, then work, somewhat randomly.

Not sure if it matters, but I’m using cloud flare, do I need to disable the cloud flare proxy on the A record that points to my publicIP? I feel like that would then expose my public IP, but I’m not sure if the cloud flare proxy would interfere with the HAProxy somehow.

Private front end:

The internal private proxy, for assigning SSL certs to backends for internal network access only, will work with one backend, but the other two won’t. The connections time out, and never complete.

However, if I uncheck “ssl offloading” in the private front end, they work perfect (obviously without SSL certs).

So my problem seems to be something to do with the SSL offloading configuration I assume?

But the strange part is that it works with one backend perfectly fine.

Deacon587 · April 22, 2022, 7:51am

SSL Config in my private frontend:

LTS_Tom · April 22, 2022, 10:34am

I have never used the Cloudflare proxy, so not really sure how that will interfere.

Deacon587 · April 22, 2022, 11:14am

Any reason you could think that ssl offload would give connection timeout problems?

I ran nslookup on my domains and it returns the correct IP, and also ran the OpenSSL xxxxxx grep subject commands, and it shows the correct wildcard certificate that I had generated.

Tmi · April 22, 2022, 12:21pm

This article might help you debug your issue. I’m not sure if it’s the same problem you’re experiencing but worth a look. Cloudflare Error 522: Main Causes and Three Confirmed Methods to Fix It

According to the article connection timeout issues could be caused by:

Server Overload
A firewall is blocking the Cloudflare requests
IP Addresses on your DNS do not match the Cloudflare ones
KeepAlive is disabled.

And they provide information on how to fix them.

Deacon587 · August 4, 2022, 12:46pm

Hello, i know its been a while, but just wanted to post an update.

Not sure how, But i have everything working.

I believe the issue i was having was related to the SSL cert i had generated, or the configuration of that cert in HA proxy. With the wildcard cert properly set up, everything works.

However i did run into another issue, which i have also resolved.

Currently i only run HA Proxy internally, because i don’t have a need to have anything accessible outside of my LAN. its purpose is to provide SSL for PFsence and other local webGUI’s to avoid the “unsafe website” errors in the browser.

HA proxy is monitoring my PFsense IP on port 443. DNS resolver is set to forward custom URLs to the PFsense IP.

Every once in a while, when navigating to one of my custom URL’s, that is overridden to the PFsense IP, it would take me to the rebinding error page of the PFsense webGUI, even though the PFsense port wasn’t entered, and the backend should have forwarded the request to a different endpoint. When this happens, just entering the same web URL again, would then properly be directed to the correct endpoint.

I found this setting, under advanced settings in PFsense, “WebGUI redirect rule”, that forwards a request received on port 80, to the webGUI port, regardless of the port assigned to the webGUI.

I believe what was happening is, when i would navigate to one of the custom URL’s, it would first send the request through port 80, instead of 443, and that rule, would then connect to the PFSense WebGUI because HA Proxy wasn’t listening on port 80. So it was a flip of a coin, on whether the browser would request through 80 or 443.

Since disabling that forwarding rule, it hasn’t happened.

I sssume having HA proxy monitor port 80 could have also solved this issue, but i want the browser to use SSL to avoid the errors, so I’m not sure if i should have HAproxy listen on port 80.

Either way, it all works now as i wanted.

Not sure if this was something i changed or not, but it was kind of annoying randomly receiving the DNS rebinding error, and that fixed it.

vasquezmi · January 23, 2023, 8:00am

I too have had issues with CF and how I set up DNS for entries. I honestly don’t feel this is going through pfSense and then offloading to HAProxy.

Besides rules, which I have tried I want to show the CF (Cloudflare) side of this.

CF DNS:

Using CF Proxy for my public IP. ICMP works but no access to website.
Using CF DNS straight to my public IP, ICMP works but no access to website. Confirmed that CF changed the IP from the CF Proxy to my public IP resolving.
Using CF DNS straight to my private IP…I get in! Of course, no cert so I allow exception condition and I am connected.

I looked at FW logs for my rules set at the WAN and nothing is coming into pfSense for any 1-3 above.

Still treking this journey so more to follow as I continue to troubleshoot.

reymond070605 · January 28, 2023, 1:39am

Have you tried using pftop or the HAProxy widget to check if you can see the traffic reaching your website? When checking the widget it will appear in a few seconds and will disappear.

vasquezmi · January 31, 2023, 5:56am

@reymond070605

Thank you for the quick tip to add the widget. So… now I do see the hits but I actually believe I understand my issue as it is hitting the f/w.

In HAProxy I created a redirect to force HTTPS traffic. If a request comes in as HTTP / 80 it redirects to HTTPS/443. I have tested a few cases where I run traffic at 80 and 443 and see the hits for those front ends. Like you stated the session hits and then disappears.

I was also able to get a quick hit on the Frontend for the specific FQDN but it still stops at the frontend. Does not pass to the backend. So I wanted to see if the backend is even showing up/down. I added a Basic Check and it reports as DOWN, even after restarting HAProxy.

I believe I know understand my problem…but working to define the solution. My issue is what sits after my pfSense. Yes, a UDMP…but right now because I have ufi switches and aggregator.

I believe what I need to do is give HAProxy a way to send to the UDMP (through VLAN) and back? Just thinking it through.

vasquezmi · January 31, 2023, 6:54am

@Deacon587,

I know you got yourself now working but I am still on my journey.

Reading through your updates here you stated that sometimes it would work and then stop and then work again. I think it might have had something to do with the stateful elements of pfSense. I found a comment to this behavior on the page for static routes which I where I am researching for my specific issue. See the !Note comment here: Routing — Static Routes | pfSense Documentation.

rblom · April 22, 2024, 1:10pm

New here!
I was experiencing the same problem. Word for word.
In my case I had accidentally added another certificate in another field in the frontend under SSL offloading. By removing this redundant certificate everything worked. Was a headache trying to figure it all out though!

Best.