PFsense HAProxy Wont Forward To Backend

Hello, im setting up HAProxy on my PFSense box and having some troubles. I followed a few different guides, and i feel like i have a decent understanding of what needs to be set and where.

But i think im missing something.

Setup:
Google domain name
Cloudflare account, created a “A” record for subdomain sub.domain.com to point to my public IP
Used ACME to successfully create a SSL certificate.
set up HAProxy’s back end, with the local IP of my end device
Set up HAProxys front end to listen on my WAN address port 8085 and to send data to the pre configured back end

Created a firewall rule to allow access on the WAN at port 8085.

When i look at my firewall log, i can see the entries for access on port 8085 of WAN, but i get nothing through the proxy, and the web browser errors with a “timeout”.

To test the forwarding of the domain to my public IP, i briefly exposed the port for the PFsense login and i was able to access it, so i know that’s working. (and i disabled that rule quickly)

Not sure what information to post that would help, but i can post whatever is needed.

Any help would be much appreciated.

Thanks!

You need to bind the HAProxy front end to your WAN public IP and port and then create a WAN rule to allow the IP and port. Then the backend needs to attach to the server.
I have the entire process covered in this video

And I have a troubleshooting video as well

Tom! Thanks for the reply, I followed those videos to get to where I am now. And still having trouble.

I’ve verified packets coming through the firewall by logging my rule to pass them.

Heres pictures of my front/backend setup, and the text from the config file in the settings pages.

Ive redacted my public IP and domains, ive verified theyre accurate.

HAProxyConfig:

# Automaticaly generated, dont edit manually.
# Generated on: 2022-04-20 19:35
global
	maxconn			1000
	log			/var/run/log	local0	debug
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend vault
	bind			67.xxx.xxx.xx3:8085 name 6x.xxx.xxx.xx3:8085   ssl crt-list /var/etc/haproxy/vault.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	acl			vault	var(txn.txnhost) -m str -i vault.axxxxxxxxxxxo.com
	acl			aclcrt_vault	var(txn.txnhost) -m reg -i ^vault\.axxxxxxxxxxxxo\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	http-request  deny if { req.hdr_cnt(content-length) gt 1 }
	http-response deny if { res.hdr_cnt(content-length) gt 1 }
	use_backend vault_ipvANY  if  vault aclcrt_vault
	use_backend vault_ipvANY  if   aclcrt_vault

backend vault_ipvANY
	mode			http
	id			100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			Vault 192.168.1.3:80 id 101 ssl check inter 1000  verify none crt /var/etc/haproxy/server_clientcert_625efbcfc4f2a.pem

Your back end says “Encrypt SSL” but I will assume port 80 is running HTTP

Yes, ive been trying several different options to try and get it working. I have some success.

I Followed your video on wildcard certs and created one to use with all of my subdomains. Im using the same cert on both of my front ends, one is public and one is private.

I now have the following setup:
Frontend “PublicServers” which is listening on my public IP:443, and has ONE backend, “vault” which works, and can be accessed from the outside internet. Although, it seems to work, then not work, then work, randomly.

Second front end “PrivateServers” which is listening on my LAN IP:443, IP of the PFsense. There are 3 backends attached to this frontend. “vault”, “backup”, and “web”. These are to be accessed within the network.

I can access “vault” from both the public AND private frontends. However, i cannot access “backup” or “web” from the private frontend.

Ive verified the other two backends are setup identically to the vault backend, with the correct IP addresses and ports for the respective devices. I also have identical host overrides in the DNS resolver for all three vault, backup, and web subdomains.

It seems the connection is being made, it keeps going into timeout.

Do i need to turn off caching, or the cloudflare proxy on DNS?

I have wireguard setup, and it works flawlessly, so i know that i am able to make connections.

Here’s the config file, sensitive info redacted.

# Automaticaly generated, dont edit manually.
# Generated on: 2022-04-21 09:07
global
	maxconn			5000
	log			/var/run/log	local0	debug
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend PublicServers
	bind			67.xxx.xxx.xxx:443 name 67.xxx.xxx.xxx:443   ssl crt-list /var/etc/haproxy/PublicServers.crt_list  
	mode			http
	log			global
	option			httplog
	option			http-keep-alive
	timeout client		30000
	acl			vault	var(txn.txnhost) -m str -i vault.axo.com
	acl			aclcrt_PublicServers	var(txn.txnhost) -m reg -i ^axo\.com(:([0-9]){1,5})?$
	acl			aclcrt_PublicServers	var(txn.txnhost) -m reg -i ^([^\.]*)\.axo\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	http-request  deny if { req.hdr_cnt(content-length) gt 1 }
	http-response deny if { res.hdr_cnt(content-length) gt 1 }
	use_backend vault_ipvANY  if  vault aclcrt_PublicServers

frontend PrivateServers
	bind			192.168.1.210:443 name 192.168.1.210:443   ssl crt-list /var/etc/haproxy/PrivateServers.crt_list  
	mode			http
	log			global
	option			httplog
	option			http-keep-alive
	timeout client		30000
	acl			backup	var(txn.txnhost) -m str -i backup.axo.com
	acl			vault	var(txn.txnhost) -m str -i vault.axo.com
	acl			web	var(txn.txnhost) -m str -i web.axo.com
	acl			aclcrt_PrivateServers	var(txn.txnhost) -m reg -i ^axo\.com(:([0-9]){1,5})?$
	acl			aclcrt_PrivateServers	var(txn.txnhost) -m reg -i ^([^\.]*)\.axo\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	http-request  deny if { req.hdr_cnt(content-length) gt 1 }
	http-response deny if { res.hdr_cnt(content-length) gt 1 }
	use_backend backup_ipvANY  if  backup aclcrt_PrivateServers
	use_backend vault_ipvANY  if  vault aclcrt_PrivateServers
	use_backend web_ipvANY  if  web aclcrt_PrivateServers
	use_backend web_ipvANY  if   aclcrt_PrivateServers

backend vault_ipvANY
	mode			http
	id			100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			vault 192.168.1.11:80 id 101  

backend backup_ipvANY
	mode			http
	id			102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			backup 192.168.1.3:80 id 103  

backend web_ipvANY
	mode			http
	id			104
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			web 192.168.1.3:8080 id 103

Ok, more progress.

Public front end:
I’ve seemed to have worked out the access from the public side (no idea how), which only proxy’s to one of my backends. I’m going to be checking it for reliability, as it seems to work, then not work, then work, somewhat randomly.

Not sure if it matters, but I’m using cloud flare, do I need to disable the cloud flare proxy on the A record that points to my publicIP? I feel like that would then expose my public IP, but I’m not sure if the cloud flare proxy would interfere with the HAProxy somehow.

Private front end:

The internal private proxy, for assigning SSL certs to backends for internal network access only, will work with one backend, but the other two won’t. The connections time out, and never complete.

However, if I uncheck “ssl offloading” in the private front end, they work perfect (obviously without SSL certs).

So my problem seems to be something to do with the SSL offloading configuration I assume?

But the strange part is that it works with one backend perfectly fine.

SSL Config in my private frontend:

I have never used the Cloudflare proxy, so not really sure how that will interfere.

Any reason you could think that ssl offload would give connection timeout problems?

I ran nslookup on my domains and it returns the correct IP, and also ran the OpenSSL xxxxxx grep subject commands, and it shows the correct wildcard certificate that I had generated.

This article might help you debug your issue. I’m not sure if it’s the same problem you’re experiencing but worth a look. Cloudflare Error 522: Main Causes and Three Confirmed Methods to Fix It

According to the article connection timeout issues could be caused by:

  1. Server Overload
  2. A firewall is blocking the Cloudflare requests
  3. IP Addresses on your DNS do not match the Cloudflare ones
  4. KeepAlive is disabled.

And they provide information on how to fix them.

Hello, i know its been a while, but just wanted to post an update.

Not sure how, But i have everything working.

I believe the issue i was having was related to the SSL cert i had generated, or the configuration of that cert in HA proxy. With the wildcard cert properly set up, everything works.

However i did run into another issue, which i have also resolved.

Currently i only run HA Proxy internally, because i don’t have a need to have anything accessible outside of my LAN. its purpose is to provide SSL for PFsence and other local webGUI’s to avoid the “unsafe website” errors in the browser.

HA proxy is monitoring my PFsense IP on port 443. DNS resolver is set to forward custom URLs to the PFsense IP.

Every once in a while, when navigating to one of my custom URL’s, that is overridden to the PFsense IP, it would take me to the rebinding error page of the PFsense webGUI, even though the PFsense port wasn’t entered, and the backend should have forwarded the request to a different endpoint. When this happens, just entering the same web URL again, would then properly be directed to the correct endpoint.

I found this setting, under advanced settings in PFsense, “WebGUI redirect rule”, that forwards a request received on port 80, to the webGUI port, regardless of the port assigned to the webGUI.

I believe what was happening is, when i would navigate to one of the custom URL’s, it would first send the request through port 80, instead of 443, and that rule, would then connect to the PFSense WebGUI because HA Proxy wasn’t listening on port 80. So it was a flip of a coin, on whether the browser would request through 80 or 443.

Since disabling that forwarding rule, it hasn’t happened.

I sssume having HA proxy monitor port 80 could have also solved this issue, but i want the browser to use SSL to avoid the errors, so I’m not sure if i should have HAproxy listen on port 80.

Either way, it all works now as i wanted.

Not sure if this was something i changed or not, but it was kind of annoying randomly receiving the DNS rebinding error, and that fixed it.