pfSense return traffic from route to another LAN not arriving

jogo · August 6, 2024, 7:43pm

Hi everyone,
This will be my first post in this forum, so if I’m not in the right section, please let me know. Also, sorry for the long post, but I want to make sure that I’m explaining the situation thoroughly.
For the past few weeks I’ve been having a problem that I’m unable to solve.
I’ve got a custom pfSense box currently in production environment that is working great. I’ve got a dual WAN setup with failover, multiple VLANs and an IPsec tunnel to our main coorporate building where the main site resides.

The issue is that some of our collaborators will be moving to this site and they will be connecting to our PBX server. Now, I could use the IPsec tunnel that is already working but I have a dedicated link that goes from this site to the main coorporate building. This dedicated link works like a VPN. The ISP gave me two network segments for each end of the link: 172.16.200.252/30 for the main coorporate building and 172.16.201.252/30 for this site.

On the main building, I’ve connected the end of the link directly to one of the PBX’s interfaces and configured it accordingly with the IP address 172.16.200.253. Routes were added to the PBX’s routing table so that it knows how to get to the other end of the link.

On the site that has the pfSense box, I connected the end of the link to the pfSense box and configured it as a WAN interface with a static IP 172.16.201.253. I then configured a new gateway for that WAN interface with IP 172.16.201.254. Then, I created a static route so that the destination network 172.16.200.252/30 is reachable through the created gateway. The last step was to configure a DNS resolver so that clients on the pfSense LAN can access the PBX server with a FQDN but routed through the dedicated link making it like this: pbx.example.com resolves to 172.16.200.253

Now the part that I haven’t been able to get right:
From the pfSense LAN and VLANs, I’m able to ping both the main building dedicated link gateway and the PBX server and vice versa. Also, I’ve checked that the FQDM for my PBX resolves to the correct IP in the pfSense LAN and VLANs. Traceroute from a client PC on the pfSense LAN to the PBX server is also succesful. However, when I try to use a web browser to enter the PBX system form the pfSense LAN it gives me a timeout error.

I’ve made packet captures on the client PC, pfSense box and PBX server and what I can see is that requests are getting out of the pfSense box, arriving at the PBX server, but when the PBX server replies, the packets never make it to the client PC.

The other interesting thing that I found was that if I create an allow all incoming traffic through the dedicated link interface on the pfSense box, if I try to SSH into the pfSense box, it accepts the traffic, but hangs until it says Connection closed by 172.16.201.253.

Another thing worth mentioning is that I’ve tested with an old Cisco RV320 router this same setup and it worked without any issues. The only things I did on that RV320 were configure the WAN with the same parameters as the pfSense, a static route and a resolver for the FQDN of the PBX server.

I’m really at my wits end. I hope someone can help me crack this thing.

Thank you and again sorry for the log post.

LTS_Tom · August 6, 2024, 8:44pm

If I read this correctly you might have an issue with the way it’s handling the static route rules. That can be changed by checking the “Bypass firewall rules for traffic on the same interface” under System → Advanced → Firewall & NAT
https://docs.netgate.com/pfsense/en/latest/routing/static.html

jogo · August 6, 2024, 9:32pm

Hi Tom,
Frist of all, thank you for your response. I’ve been following your Youtube channel for a long time and I love your content and the way you explain things. I’ll give it a try right now and get back to you.

Thank you!

jogo · August 7, 2024, 12:04am

Hi Tom,
I’ve tried the “Bypass firewall rules for traffic on the same interface ”, but no joy.
Digging a little deeper on the pcaps on both the pfSense interface and the client PC I found that there seems to be a problem with the TLS handshake. The part where the server sends the Certificate, Server Key Exchange and Server Hello Done never gets to the client PC. I’m by no means a Wireshark expert, but that seems to me wrong. Could pfSense be blocking those packets? I don’t see any relevant entries in the logs. Could this be related to the Outbound NAT configuration?
The packets captured on the client PC go from its IP address to the PBX address, but PBX only sees traffic from the pfSense interface. I feel like pfSense is not being able to forward the response to the client correctly. Am I going in the right direction here?

LTS_Tom · August 7, 2024, 10:59am

Flor clarity, can you make a diagram of your setup? You can use something like https://app.diagrams.net/ to make it simple.

jogo · August 7, 2024, 5:00pm

Sure! Here it is.

LTS_Tom · August 7, 2024, 8:42pm

If you have private IP’s on WAN do you also have the “Reserved Networks” boxes unchecked for each pfsense WAN interface?

jogo · August 7, 2024, 9:21pm

Hi Tom,
Yes. Block private networks and loopback addresses and Block bogon networks checkboxes are unchecked.
Last night a did some tests connecting another server instead of my PBX server. It worked both on http and https, This, and the previous test I did with an old cisco router instead of the pfSense leads me to believe that there is some misconfiguration or something is incompatible between my PBX server and the pfSense box but I just can’t figure out what it is.

pavlos · August 7, 2024, 9:58pm

I think the /30 is too restrictive. Can you open it and retest?

jogo · August 8, 2024, 10:11pm

Hi pavlos,
I tested with your /30 suggestion, still not working.
I connected the old cisco again and a test webserver and did some packet captures.

This firs image is the entire TCP conversation captured at the client PC.

This second image is the same conversation captured on a test web server.
Now, I’m no expert but I think that this is a good example of what a HTTPS conversation should look like.
Moving back to using pfSense and the test webserver, I’ve managed to configure Outbound NAT rules so that ports are static and now I’m recieving the whole TLS handshake. The problem now seems to be packets aren’t arriving. I’m seeing some retransmits on the server side and on the client some previous segment not captured warnings.
I’ll keep trying different combinations but I really don’t think I’m getting anywhere.

liquidjoe · August 9, 2024, 12:59pm

You may have tried the following, but this is what I would do:

make sure NAT is off on pfsense interface for direct link. Should not matter, but not needed.
check routing tables for both ends. make sure pbx knows about other side’s LAN.
open up FW rules on both sides of the direct link. make sure fw rules on other pfsense LAN interfaces allow out direct link.

pavlos · August 9, 2024, 1:45pm

In your DNS Resolver, do you have

jogo · August 13, 2024, 7:48pm

Hey guys,
Let me thank all of you who posted on this thread. I’d like to let you know that I tried all of your suggestions, but it turns out that it didn’t work. In my desperation and frustration, I thought that the only thing to do was to starto over from scratch. So, I decided to roll back all configurations about this and open a maintenance window so I could turn off the pfSense box and replace the network card.
Turns out that this did the trick. After the replacement NIC was installed, I configured it just as any other WAN interface with a static address and created the static route and the corresponding firewall rules and DNS resolver entries…AND IT WORKED!
Thanks again for taking the time! I’m really grateful to you all.
Cheers!