pfSense+HAProxy doesn't talk to TrueNAS

Networking & Firewalls
1

I’m trying to setup HAProxy to talk to multiple backends. Before getting comments about it, the final goal isn’t to have TrueNAS connected to the internet, I’m just trying to get this part of the puzzle sorted out.

No matter what I do, I can’t get HAProxy to talk to the TrueNAS (scale) webUI. All installed apps work fine, as well as another webserver on the network, but the webUI cannot be reached. Health check times out, and disabling it altogether just times out into a 503 error message.

The logs say “backend TrueNAS_ipvANY has no server available!” and TrueNAS_ipvANY/<NOSRV>. The HAProxy stats page says Layer6 timeout.

I’ve also run tcpdump on pfsense, but I’m not sure how to interpret it or if it even helps at all. There’s a difference though between a working backend and the webUI.

Here’s a tcpdump of a (basic) health check of working backend: Jellyfin running as an app in TrueNAS.

23:54:10.609542 00:02:c9:05:6a:a9 > 7e:9b:61:e8:18:5e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.87.1.38566 > 192.168.87.33.30013: Flags [S], cksum 0x9be3 (correct), seq 1963072212, win 65228, options [mss 8960,nop,wscale 14,sackOK,TS val 640065192 ecr 0], length 0

23:54:10.609961 7e:9b:61:e8:18:5e > 00:02:c9:05:6a:a9, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.87.33.30013 > 192.168.87.1.38566: Flags [S.], cksum 0x6cfc (correct), seq 273964780, ack 1963072213, win 65160, options [mss 1460,sackOK,TS val 2603434495 ecr 640065192,nop,wscale 7], length 0

23:54:10.609987 00:02:c9:05:6a:a9 > 7e:9b:61:e8:18:5e, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.87.1.38566 > 192.168.87.33.30013: Flags [.], cksum 0x9a4c (correct), seq 1, ack 1, win 5, options [nop,nop,TS val 640065192 ecr 2603434495], length 0

23:54:10.610007 00:02:c9:05:6a:a9 > 7e:9b:61:e8:18:5e, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.87.1.38566 > 192.168.87.33.30013: Flags [R.], cksum 0x9a4d (correct), seq 1, ack 1, win 0, options [nop,nop,TS val 640065192 ecr 2603434495], length 0

Here’s a tcpdump of a failed (basic) health check on TrueNAS webUI:

23:54:11.680583 00:02:c9:05:6a:a9 > 7e:9b:61:e8:18:5e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.87.1.60451 > 192.168.87.33.443: Flags [S], cksum 0x6fe3 (correct), seq 3954313836, win 65228, options [mss 8960,nop,wscale 14,sackOK,TS val 362517232 ecr 0], length 0

23:54:11.680844 7e:9b:61:e8:18:5e > 00:02:c9:05:6a:a9, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.87.33.443 > 192.168.87.1.60451: Flags [S.], cksum 0x0218 (correct), seq 3076402932, ack 3954313837, win 62636, options [mss 8960,sackOK,TS val 982210307 ecr 362517232,nop,wscale 7], length 0

23:54:11.680878 00:02:c9:05:6a:a9 > 7e:9b:61:e8:18:5e, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.87.1.60451 > 192.168.87.33.443: Flags [.], cksum 0x42d8 (correct), seq 1, ack 1, win 5, options [nop,nop,TS val 362517232 ecr 982210307], length 0

23:54:11.681056 00:02:c9:05:6a:a9 > 7e:9b:61:e8:18:5e, ethertype IPv4 (0x0800), length 583: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 569)
    192.168.87.1.60451 > 192.168.87.33.443: Flags [P.], cksum 0x3ee0 (correct), seq 1:518, ack 1, win 5, options [nop,nop,TS val 362517232 ecr 982210307], length 517

23:54:11.681210 7e:9b:61:e8:18:5e > 00:02:c9:05:6a:a9, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 36475, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.87.33.443 > 192.168.87.1.60451: Flags [.], cksum 0x3ef2 (correct), seq 1, ack 518, win 486, options [nop,nop,TS val 982210307 ecr 362517232], length 0

23:54:13.682431 00:02:c9:05:6a:a9 > 7e:9b:61:e8:18:5e, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.87.1.60451 > 192.168.87.33.443: Flags [R.], cksum 0x3902 (correct), seq 518, ack 1, win 0, options [nop,nop,TS val 362519234 ecr 982210307], length 0
2

Quick fix would probably be to delete the back end, set it up again but make sure health check is disable during the setup.

3

Tried re-creating the backend, exactly as it was before, and exactly as all the other working backends are set up, but now making sure health check is “none” from the beginning. No change, after loading a long time the browser displays a 503.

4

Here is the back end for mine that works:

Mode Name Forwardto Address Port Encrypt(SSL) SSL Checks
active TrueNAS Address+Port: 172.16.16.5 443 yes no
5

Mine looks exactly the same, except I have the hostname as the address, but I’ve also tried directly with the IP address and it makes no difference (and tcpdump shows communication does happen).

I’m unsure about the client certificate option there. Default is none, and all my working backends have it, but I’ve also tried with the server cert issued by Acme/LE. Both options work with all other backends, neither works with TrueNAS.

1 Like