Pfsense and starlink wan failover issues

Networking & Firewalls
1

Hi,
I have a Netgate 6100, where WAN 1 is my local ISP, and WAN 2 is set to Starlink. Starlink is setup in bypass mode.

I have setup WAN failover so when my local ISP goes out it will use Starlink, but as of recently i’ve been having some issues with this setup. All gateway kill states are set to the default values and have setup a monitor ip for both WAN (local ISP) and WAN2 (Starlink). For each VLAN, under firewall rules, I have explicitly selected the WAN failover group.

When Starlink goes down, which it seems to do frequently, but my local ISP is up and running, i lose all access to the firewall and all local devices. The only thing that brings everything back up is to pull the power plug on the 6100 so that it reboots.

Is anyone else having this issue or anyone able to point me in the right direciton.

Thank you!!!

2

What do you have your tiers setup as on your group?

Screenshots of your configs would be helpful.

3

Screenshot 2024-08-12 at 19.02.11
Screenshot 2024-08-12 at 19.02.111138×541 64.8 KB

Local ISP Lan1 is Tier 1 and Starlink is Tier 2.

To add more info - when WAN 1 fails, then everything does shift to WAN 2, and when WAN 1 is back, everything will start using WAN 1 again - so it all works.

The following are the errors i get in system logs when this happens:

Screenshot 2024-08-12 at 19.05.25
Screenshot 2024-08-12 at 19.05.251050×781 85.2 KB

Thank you again!!

4

I don’t use the Member Down trigger level. I use the High Packet Loss or High Latency. I think when you set it to member down it is expecting the physical interface to be down. I could be wrong.

5

I used to use High packet Loss or High Latency, but my Local Internet would fluctuate sometimes and would keep trying to switch when this happens and was causing issues. So I had to put it to Member down.

6

Looks like you can adjust the threshold on the packet loss or latency based off the documentation.

image
image1242×2118 261 KB

7

thank you for finding that. I’ll try it out and see if it makes a difference.

8

This did not work. I still lose access to the firewall, and local devices lose ip addresses when WAN2 fails or gets removed from the WAN_Failover gateway group.

9

On the gateways, what are the monitor ip settings for the starlink ? It should be an external dns server and different to the other wan setting

When you say I lose access to the firewall , please expand. Losing connection on the wan , should not affect local machines just internet access - machines should still be able to access internal lan devices

Are the devices getting the correct ip address from the correct network ?

10

Hi,
Each WAN has different monitor ips, 1.1.1.1 for WAN1 and 8.8.8.8 for WAN2 (Starlink).

So all devices do get proper IP addresses and are working, but when Starlink goes down (WAN2), something happens where i can’t reach the firewall nor any computers, and appletv home hub becomes unavailable from inside the network (on LAN). At this point i have to restart the firewall to get everything back up.

Interestingly, I can console into the 6100, and i see the normal menu’s and no errors. So it looks like the webui stops working.

As i mentioned previously, my appletv home hub becomes unavailable, and from my computer, i can’t access the pfsense webUI.

Also, when WAN goes down, it comes back up and no issues. I can access everything from inside my network, home hubs stay up and i am able to access the firewall webUI.

Regarding Starlink, i’ve read that Starlink renews the WAN ips every min or so, although they try to give out the same WAN ip. I have also added WAN2 to ignore DHCP leases from 192.168.100.1 which is Dishy’s IP address.

Thank you for helping with this!!

11

What happens if you disable pass thru on the starlink and setup wan as double nat.

On the wan connection to starlink change to dhcp so it gets an ip address from the starlink router

12

I’ll try that, although I would prefer passthrough mode.

13

So before resetting Starlink and setting it up via a double nat. I use to have Service Watchdog monitor unbound, but I deleted it form Watchdog - i haven’t had the issue yet.

Is it bad to have Service Watchdog monitor unbound?

14

You only need the service watch dog monitoring if the service keeps on disabling

There was a issue with unbound, but this has been resolved in the latest versions of pfsense

15

I didn’t know that … thank you.

16

So this happened again, but WAN2 went down first and then WAN1 went down. All my devices lost their IP addresses. I know this because i looked at my computer and it had no ip address, and my unifi switches showed no ip address either.

I had to hard reboot the firewall to get things back up.

The following are logs from the event:

Screenshot 2024-08-15 at 08.12.29
Screenshot 2024-08-15 at 08.12.291092×850 100 KB

17

Have you changed any of your firewall rules for the gateway?

18

I have not changed any firewall rules for a while, this all just started randomly a month ago.

19

So I had the issue again, and was able to console into the system. I tried rebooting, and halting the system and it would not do either. Trying to look at logs via console didn’t show anything out of order.

Something weird, when consoled in WAN 2 or ix3, didn’t have an IP address, and since the system would not reboot or halt, it only came back up after i hard restarted it.

I had to physically unplug and replug in the 6100.

I’ve reset starlink and connected the 6100 to the starlink router so will see if this resolves the issue.

20

Wanted to provide an update. I reset Starlink and have pfsense connect to the router. So now the 6100 is double NAT’d and has a Private IP. It’s been stable and no issues.

I would prefer to use bypass mode, since the starlink router also broadcasts wifi, but its all stable now so will keep it this way for a while.