HAProxy + SSL within LAN: Layer6 Handshake failure – confused on this one. Help, please?!

Hello Lawrence Systems folk!

First of all — Tom’s YouTube guides have been INVALUABLE. What a phenomenal body of work for the internet. Thank you.

Of course however I’m here because I’m stuck!

I’ve spent quite a bit of time between a rock and a hard place trying to be able to connect via SSL to one particular server running on a proxmox VM on my home LAN using HAProxy on Pfsense.

Current setup:

  • I have a a few servers running in a couple of different VLANs for different purposes: pihole, proxmox, etc.
  • Created wildcard certificate for mydomain.com / *.mydomain.com in ACME
  • Set up HAProxy backends for these servers + frontend with SSL offloading
  • I can access https://pihole.mydomain.com, https://proxmox.mydomain.com, etc. all with zero issues — happy me… except for this new problem:

On Proxmox (which I’m recently converted to), there is 1 VM (Ubuntu Server) that’s running a database index with no webUI front-end (you connect with apps).

I can use my desktop software to connect to the index today by typing the IPv4 ssl://192.168.10.3 on port 50002 and the connection is successful + traffic is encrypted using a self-signed certificate on the VM.

However, I want to connect to ssl://index.mydomain.com:50002via a reverse proxy for SSL termination, as the mobile apps I use to connect require a valid SSL certificate.

I’ve set up HAProxy backend to point index to 192.168.10.3 on port 50002 with Encrypt (SSL) checked and SSL checks unchecked.

On the frontend, I already have the Listen Address as the Proxmox VLAN interface for port 443 (for the services with webUIs), so I added a new row, same interface, with port 50002 and SSL offloading checked.

Access control list and backend are just as the webUI services, and DNS resolver is configured also.

Using this, I could never get the service to connect and 50 youtube videos later was giving up. In the end, I added a NAT rule to port forward 50002 between HAProxy on the VLAN interface (192.168.10.1) and the IP of the VM itself (192.168.10.3) — amazing! I can connect.

EXCEPT — I get man-in-the-middle warnings, handshake failure warnings… always warnings. And the mobile apps won’t connect.

Using openssl I think I have identified the problem, but I’m not sure what I’ve done wrong:

When I run openssl s_client -servername index.mydomain.com -host 192.168.10.1 -port 50002 with the port forwarding enabled, I receive confirmation of the self-signed certificate. I can connect to the server with SSL warnings that I can ignore on desktop, but not on mobile.

With port forwarding disabled & running the same query, I receive confirmation of the wildcard certificate (what I expected), but I cannot connect to the server.

It’s been several days/weeks that I’m fiddling around here with this. Any help would be much appreciated!

I have never tried putting a reverse proxy in front of a database connection but my guess would be that is probably needs some extra parameters either in the database setup or HAProxy.

Thank you for the reply, Tom!

I’ve dug into this more and it looks like, were I using nginx, this would be a stream and you’d place the conf file in streams-available, etc.

I’ve been able to verify this works from a VPS with a wireguard tunnel into the router.

Any idea on how I achieve the equivalent with HAProxy for my LAN? I suspect it’s most likely I’m missing something obvious vs. something difficult.