HAProxy delivers stats page instead of proxied server

Networking & Firewalls
1

Hi,

I have this simple setup

Internet <-> pfsense + HAProxy <-> Docker Apache

All appears to be good, except that the clients receive the HAproxy stats page instead of the proxied pages.

The HAProxy frontend (thre is only 1 defined) is configured to support HTTPS on port 443. The certificate is from Lets’s encrypt and this works as it should. The frontend ACL is configured with 1 backend

There is only 1 backend with 1 server configured in HAProxy. There is no SSL here. And I am using port 83

The backend server is Up (as reported by the Health check and seen in the Stat page and by verifying with a packet capture). Access from the LAN to the Apache server also works as it should; I am seeing the whoami dummy load that I have at the moment.

The firewall is properly configured too. Using a client on the Internet, I see the correct Let’s Encrypt certificate so this confirms that 443 is open.

But, on the client, what I see is the HAProxy stat page.

What I tried:

1 - Try to configured HAProxy to use the local syslog (/var/run/log). But I only see the transactions without detail on where the data came from
2 - I toggled the Not check box in the ACL in the frontend. When I do this, I get HTTP Error code 503 Service Unavailable. I have no default backend defined. But if I set a default backend, then the client again sees the stat page.

Looking at the data on the stats page I see the frontend counters moving each time I refresh the client. I also see the total counters in the Backend section to move at the same time (and the values matches what I see in the Frontend section). But, the counters on the line corresponding the backend I defined stay at 0. All of this means:

  • that haproxy sees the traffic coming in the frontend
  • it delivers it to the overall backend logic
  • select the correct backend (as per the test 2 above, when the Not box is not checked, or with a default backend that selects the one and only backend defined).

But somehow, the specific backend does not deliver to the external backend server despite the Health check being positive.

At this stage, I am running out of option on how to diagnose this. Any suggestions?

2

You’ll need to post your config here for anyone top help you.

Go to Services > HAProxy > Settings and scroll all the way to the bottom and click on the Show button next to “automatically generated configuration.”

3

I have a few setup video on HAProxy and a troubleshooting one here:

4

Here it is (and thanks for telling me where to get it!!)

global
maxconn 100
log /var/run/log local1 debug
stats socket /tmp/haproxy.socket level admin expose-fd listeners
uid 80
gid 80
nbproc 1
nbthread 1
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
bind 127.0.0.1:2200 name localstats
mode http
stats enable
stats refresh 10
stats admin if TRUE
stats show-legends
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000

frontend whoamiFrontEnd
bind x.x.x.x:443 name x.x.x.x:443 ssl crt-list /var/etc/haproxy/whoamiFrontEnd.crt_list
mode http
log global
option httplog
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
acl whoami var(txn.txnhost) -m str -i test.aaaa.com
acl aclcrt_whoamiFrontEnd var(txn.txnhost) -m reg -i ^test.aaaa.com(:([0-9]){1,5})?$
http-request set-var(txn.txnhost) hdr(host)
use_backend whoamiBackend_ipvANY if whoami aclcrt_whoamiFrontEnd

backend whoamiBackend_ipvANY
mode http
id 102
log global
stats enable
stats uri /
stats realm .
stats show-node NODE1
stats refresh 10
stats scope .
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server whoami 192.168.70.32:83 id 101 check inter 1000

image

5

That is a pretty interesting config file. Are you trying to expose a service running on 192.168.70.32:83?

If so here is what I would do

Backend

  • name = whoami
  • server list
    • Mode = Active
    • Name = whoami
    • forwardto = Address+port
    • port = 83
    • leave the rest default

frontend

  • Access Control list
    • name = whoami
    • Expression = host match
    • CS = no
    • not = no
    • value = test.aaaa.com
  • actions
    • action = use backend
    • condition acl names = whoami
    • backend = whoami

I wouldn’t add anything else special to your config. Only what I posted above and leave everything else default. Here is a sample of my frontend and backend.

frontend frontend-80
	bind			xxx.xxx.xxx.xxx:80 name xxx.xxx.xxx.xxx:80   
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	http-request redirect scheme https code 301 if !{ ssl_fc } 

frontend web
	bind			xxx.xxx.xxx.xxx:443 name xxx.xxx.xxx.xxx:443   ssl crt-list /var/etc/haproxy/web.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			server1	var(txn.txnhost) -m str -i server1.example.com
	acl			server2 var(txn.txnhost) -m str -i server2.example.com
	acl			server3	var(txn.txnhost) -m str -i server3.example.com
	acl			server4	var(txn.txnhost) -m beg -i server4.example.com
	acl			server5	var(txn.txnhost) -m str -i server5.example.com
	acl			aclcrt_web	var(txn.txnhost) -m reg -i ^([^\.]*)\.example\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	use_backend server1_ipvANY  if  server1 aclcrt_web
	use_backend server2_ipvANY  if  server2 aclcrt_web
	use_backend server3_ipvANY  if  server3 aclcrt_web
	use_backend server4_ipvANY  if  server4 aclcrt_web
	use_backend server5_ipvANY  if  server5 aclcrt_web

Single backend config

backend server1_ipvANY
	mode			http
	id			110
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			server1 192.168.13.101:80 id 106 check inter 1000
6

It is always easier when you can compare…

It is the stats section in the backend config. I did not understood that it was for serving a stats page in addition to the backend pages. In the context of pfsense I wrongly assumed that it related to some page in the pfsense’s GUI. When thinking about it with an haproxy only perspective then it makes sense.

So, by simply disabling or moving the stats to another URI (different than / that is!!) then all is good.

Thanks.