Pfsense in Lab, no internet?

Networking & Firewalls
1

I’m sure I am just missing a really simple setting but I can’t seem to figure what is creating this wierd issue.

So I have a dedicated pfSense running at the head of my network which runs great but I’ve been trying to setup pfsense in a xcp-ng vm for testing and other things.

I’ve installed pfSense running with fresh default settings. Using the following networks

  • WAN - 10.0.0.173 (DHCP, from head end using a /24 network)
  • LAN - 192.168.1.1 (XCP-NG private network interface, Running DHCP for clients)

When I put my test desktop on the LAN interface it grabs a ip and can talk to the vm pfsense but can not connect to the outside world expect for ping/dig. However the lab pfsense has full internet and can ssh/ping/dig out.

lab.pfSense

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=54 time=13.170 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=14.113 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=13.538 ms

— 8.8.8.8 ping statistics —
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 13.170/13.607/14.113/0.388 ms

PING cnn.com (151.101.193.67): 56 data bytes
64 bytes from 151.101.193.67: icmp_seq=0 ttl=55 time=9.866 ms
64 bytes from 151.101.193.67: icmp_seq=1 ttl=55 time=7.468 ms
64 bytes from 151.101.193.67: icmp_seq=2 ttl=55 time=6.995 ms

— cnn.com ping statistics —
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 6.995/8.110/9.866/1.257 ms

[2.4.5-RELEASE][admin@pflab.localdomain]/root: ssh r***.ca
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Desktop behind lab.pfSense:

┌─[barr@pop-os]─[~]
└──╼ $dig google.ca
; <<>> DiG 9.11.5-P4-5.1ubuntu2-Ubuntu <<>> google.ca
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52867
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;google.ca. IN A

;; ANSWER SECTION:
google.ca. 300 IN A 172.217.165.3

;; Query time: 142 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Tue Mar 31 15:30:55 EDT 2020
;; MSG SIZE rcvd: 54

┌─[barr@pop-os]─[~]
└──╼ $ping cnn.com
PING cnn.com (151.101.65.67) 56(84) bytes of data.
64 bytes from 151.101.65.67 (151.101.65.67): icmp_seq=1 ttl=54 time=16.4 ms
64 bytes from 151.101.65.67 (151.101.65.67): icmp_seq=2 ttl=54 time=10.5 ms
64 bytes from 151.101.65.67 (151.101.65.67): icmp_seq=3 ttl=54 time=9.83 ms
64 bytes from 151.101.65.67 (151.101.65.67): icmp_seq=4 ttl=54 time=10.1 ms
64 bytes from 151.101.65.67 (151.101.65.67): icmp_seq=5 ttl=54 time=15.8 ms
64 bytes from 151.101.65.67 (151.101.65.67): icmp_seq=6 ttl=54 time=16.3 ms
^C
— cnn.com ping statistics —
6 packets transmitted, 6 received, 0% packet loss, time 5007ms
rtt min/avg/max/mdev = 9.834/13.140/16.399/3.035 ms

┌─[barr@pop-os]─[~]
└──╼ $ssh r***.ca
.

clog -f /var/log/filter.log while trying to make outbound connections

[2.4.5-RELEASE][admin@pflab.localdomain]/root: clog -f /var/log/filter.log
Mar 31 19:17:37 pflab filterlog: 7,1000000105,xn1,match,block,in,6,0x00,0x00000,1,Options,0,56,fe80::981f:fc58:f7ba:7da1,ff02::16,HBH,RTALERT,0x0000,PADN,

So in TL;DR;

  • pflab.localdomain can access the world just fine
  • anything behind pflab.localdomain can ping/dns/dig out but no tcp/udp packets but nothing in the firewall?

Is this a double nat issue?

2

I’ve literally been doing the same thing this evening with pfSense on a vm in xcp-ng. I even had similar issues. I had no internet access from my lab pc, couldn’t ping or do anything, but all tests worked fine from the pfSense vm. My issue turned out to be the rules on my physical pfSense box blocking dns traffic (I have it locked down to only allow pfSense to act as a dns server). I had to set my vm pfSense to forward to my physical box. I couldn’t see the issue in the logs but if I created an allow all rule things worked.

TL;DR;
Put a temporary allow all rule on your physical box to rule that out, and if it works try narrowing it down one step at a time.

3

Thanks for the reply.

I too have a dns rule on the main pfSense, however I have it disabled for testing and dns is working on the lab pfSense box.

4

I’ve been having a play with my setup again and I can confirm that I have full Internet access from my lab even with the double NAT on two pfSense firewalls. I know that doesn’t really help you much, but at least you know it should work. I have however had random periods of the network not working, can’t ping etc. then it just starts working again, not sure if that is pfSense or XCP-NG

5

That’s good to hear its working for you :smiley: No difference from my end.

@LTS_Tom Would it be possible to get a backup xml of your pfsense that you used for the two public wan ip’s that you used in your video to compare to what I have?

6

I already purged those, but if you are using pfsense in a double NAT lab setup you may have to turn off the options “Block Private Netowrks” option on the WAN of the lab system. If you don’t it will block traffic from IP addresses that are reserved for private networks per RFC 1918.

https://docs.netgate.com/pfsense/en/latest/firewall/show-bogons.html

1 Like
7

Already off, and I would assume if it wasn’t it’d show up in the /var/log/filter.log?

I’m kinda lost to what is causing this issue.

8

Sloved! Turned out I needed Disable hardware checksum offload to be checked.

1 Like
9

Well done! I was actually just coming on here to post I’ve had to do exactly that on my test system to get it to be more stable. Strange as I’m using Intel NICs which are normal fine, but it’s only a test system so I don’t mind. So it looks like our problems were the same thing after all!