Never Ending 522 Error...Losing My Mind | pfSense + HAProxy + CloudFlare + LetsEncrypt

Networking & Firewalls
1

Hi, I been trying to trouble shoot for a couple of days now and cant think anything else. Any comments/advice would be greatly appreciated!

Basic Network Setup

  • I’m running pfsense as my router and utilizing it’s HAProxy package to handle the ssl certs for my VM’s inside the network.

  • I am also using a managed port switch that connects downstream to a proxmox server. I configured it to be a trunk connection since I created vlan 100 to be for the proxmox web UI and vlan 150 for the VM’s

  • domain names are proxyed through cloudflare

The problem

I am able to obtain an ssl cert for whichever sub.domain.com, however, it takes a while to load and always results in error 522. The followings are what I confirmed:

  • can reach servers through the direct ip address & port number combination, however, this is unencrypted
  • HAProxy backend contains the correct ip address & port number
  • HAProxy frontend is set with following settings:
    • ssl-offloading enabled
    • port = 443
    • listening interface = wlan
  • A record on cloudflare’s dns is pointing to correct public ip that router is facing

High Level Steps:

The following were my high level steps:

  • created a new HAProxy backend for vm

    • used wildcard cert
    • enabled “Encrypted (SSL)”

  • when creating new HAProxy frontend

    • listening address = WAN
    • ssl-offloading is enabled
    • port = 443
    • type = http/https
    • ACL & Actions are set correctly
    • certificate = wildcard

Firewall rules

wan

  • pass tcp type from any source to any destination through 443
    • changed the default pfsense https port to not conflict with HAProxy

vm lan

  • block any type from reaching out to any other network
  • block tcp from reaching to router’s new web ui port number
  • allow any type from any source to any destination

Attemp for Local DNS Resolver

I then attempted to get a domain name to my proxmox server. I figured I can possible get it to work locally and not listen to the wan interface. This still resulted in a 522 error code for my desktop but resolves correcctly for my laptop. Still cant get it to work properly even when restarting the desktop.

High Level Steps:

The following were my high level steps:

  • created a new HAProxy backend for proxmox

    • used wildcard cert
    • enabled “Encrypted (SSL)”

  • when creating new HAProxy frontend

    • listening address = LAN
    • ssl-offloading is enabled
    • port = 443
    • type = http/https
    • ACL & Actions are set correctly
    • certificate = wildcard

  • finally, configured DNS Resolver and added Host Override.

    • ip to return for host = gateway

Trouble shooting

  • attempted to us nslookup within pfSense by entering the IP address of the desired server and was able to resolve. Looks like it returned normally
  • used the following command and returns with the corrent domain,
    openssl s_client -servername notes.example.com -host [PUBLIC-IP] -port 443 < /dev/null | grep subject=CN
  • used the command, dig notes.example.com, it returns 2 A records that fall within cloudflares network so it appears to working correctly
2

The HAProxy setup sounds connect but I have never tried to domain setup proxied through Cloudflare. Using something like Cloudflare tunnels might be an easier solution.

1 Like
3

Thank you for your response! Never heard of Cloudflare tunnels, I did see a video from you guy about it this morning but that might work. I’ll have to try it out later this evening.

Are there any diagnostic procedure or commands/tools that you recommend using to get a better understanding of what could be the issue? Even if Cloudlare tunnels works, I still want to get a better idea as to why I am seeing this error.

What bugs me is that I did the very same setup but using a VM pfSense. In my older home network setup, I had port forward 443 to my proxmox server, then my VM pfSense using HAProxy was able to deal with the ssl certs for my other VM’s. It worked great, but with the increased devices/demand on the network I had to get ride of the router that was also acting as an access point and switch and get dedicated devices.

4

522 is on their end & I have never spent anytime trying to debug a Cloudfare error.