I am not a network engineer by any means.
But i was able to setup a oracle cloud account and spin up a VM.
Installed wireguard and using wgquick get few clients setup.
wg-server @ oracle cloud (10.66.66.1)
client1 @ pfsense at home (10.66.66.3)
client2 @ iphone (10.66.66.2)
client3 @ laptop
i am able to ping all clients from the server side and vice versa, but client to client ping is not working and also unable to access any services on my home network.
i am able to use the oracle wg as a VPN and access the internet with oracle IP as public address.
My issues -
want to access local services on the home network using wg on oracle server
Please let me know how to fix it
I require this setup because one of the ISP i have does not give public IP (behind NAT) and thus have to keep a outbound open connection so that other devices can connect to home network on the go.
iptables @ server
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:51821
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT udp -- anywhere anywhere udp spt:ntp
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
InstanceServices all -- anywhere link-local/16
May very well be a routing issue. If you want to route traffic from one peer (client3) through another peer (wg-server) into a network behind a third peer (client1) and back, there is a bit of config needed.
For a start, the Wireguard configs need the correct AllowedIPs entry. Then, you need to configure the host “client1” to allow incoming connections on the Wireguard interface from the tunnel subnet to the local networks you want to access. You also need to make sure that the routes for the responses are in place.
Could you provide the Wireguard configs (redact the keys) for wg-server, client1 and client3, please?
Paolo is right, this is most likely a routing issue that can be fixed by allowIP on both ends. Make sure your LAN subnets are in the allowIP on the wg-server and the server LAN subnets are in the pfsense allowIP list.
I think in pfsense I may have had to add the remote subnets in as static routes, but that doesn’t seem right now that I think about it. Maybe somebody could correct me if I am wrong?
Also, before you are done with this I would look to tighten up your forwarding rules.
Ok, this is what I’m getting from this. Let’s say you want to connect from the phone via the sever to the network behind pfSense, which is 192.168.1.0/24. The AllowedIPs = 0.0.0.0/0,::/0 on the phone config does two things: It configures the tunnel to allow traffic to anywhere to be sent to the server peer and it configures the OS to actually route the traffic there. Now that the server has received the traffic, it must forward it to the pfSense peer. However, from the config, the server does not seem to have a route for 192.168.1.0/24. At the very least, you need to add this network to the AllowedIPs of the pfSense peer section on the server config. I don’t know if that also automatically creates the appropriate route on the server, if not you have to add that too.
When that traffic reaches a host in the local network of the pfSense, it will have a source address of 10.66.66.2. Assuming the pfSense is the default gateway for this host, verify that there is a route to the source via the server.
Another issue might be that when the server eventually tries to forward the traffic to pfSense, there isn’t a connection established because there is no endpoint to initiate a connection from the server and there is no keepalive on the pfSense.
On the pfsesnse side I had to add static routes for my server’s LAN subnets. That was odd and I could be wrong on that one, but I don’t want to mess around with it right now. Just make sure your subnets don’t overlap.
One thing I would suggest is to separate your iptables from your wg config. Before I switched to nftables I liked using iptables-persistent, and then saving my config in a text file. Working with one text file was a lot cleaner than messing around with scripts or configs that added/changed rules in other places. Text file reads like a book, is easy to comment and backup. At least it was easier for me.
@paolo@liquidjoe
thanks a lot for the detailed analysis…
but as i said i am not a network engineer…but i can follow instructions regarding setting this up…
so from the above post what i understand is
have to add my home network ip subnet (192.168.1.0/24) to my peer allowedIP list along with 0.0.0.0
add static route in the wg server config on oracle VM
add static route in the pfsense firewall to direct traffic to internal network
seperate iptables from wg config
let me know if my understanding is in right direction.
last 3 points i actually dont know how to get it done…a step by step instructions will be life saving.
I’ve been where you are at - we all have. I spent a lot of time figuring this stuff out. Good knowledge takes time, don’t skip past the learning process just to get a project done. If you value the end result over the learning process, then hire Tom to get it done. That’s my $0.02 sermon for the morning.
Your oracle vm doesn’t need static routes. I imagine that is ubuntu or something. The AllowIPs you configure are the static routes. By doing that you just need to tell that router (your oracle vm basically) how to route to x.y.z subnets behind your pfsense router. The flip-side for your pfsense box. But with pfsense you also need to setup static routes. I tested that today to confirm it again. This is odd but it does look like it is required in pfsense.
I only forward all my traffic with the 0.0.0.0 config for my phone and laptop. You could do this with a point to point setup, but I don’t.
Separating the iptables config is just my personal preference. There are just as many people who don’t do that. I would encourage you to spend a good amount of time playing with the iptables config (or better yet migrating to nftables). This stuff isn’t rocket science, but it does take some time learning the syntax. And if you don’t know basic networking you got to learn that too. If you don’t enjoy learning this stuff then please hire Tom. You can study what he did and get the gist of it in 20 minutes instead of 10 hours.
so i added 192.168.1.0/24 to both iphone and pfsense config in allowedIPs… now i am able to ping my phone from my desktop behind the pfsense… but with few timeouts… dont know why…
but now how to access services behind the pfsense on iphone…
i am running few services like http, synology NAS, freepbx and others…
can you guys please point me in the right direction…
for your information:-
i have a wg server running on pfsense and another wg server is running on oracle VM. pfsense is acting as a client for the oracle wg server.
i require these two as when one ISP with direct public IP is down i have to rely on the other ISP which does not give me a public IP to connect to my home network.
This is indeed the intended behavior. The Wireguard service and other tunnels are, in and of themselves, not related to the routing table of the system they are running on. Some clients may automatically add routes if they have the necessary priviliges, either for convenience (e.g. Windows) or because the user wouldn’t have any other way of doing it (e.g. Android). I would imagine that for the pfSense client, maximum control and flexibility were the goals, so it makes sense to leave it up to the admin to set the routes - or not set them.
Another thing to be aware of is that when Wireguard encapsulation adds overhead to packets. Therefore you should lower the MTU of the Wireguard interfaces on all peers to 1280.
Adding the 192.168.1.0/24 is superfluous as it is contained in 0.0.0.0/0 already.
The dropped pings sound like a routing issue. Can you ping from a PC through the oracle VM, through pfsesnse, and the freepbx server? I’d set this up on a laptop first.
On your oracle vm I would not set 0.0.0.0 in the AllowIP. Just include the pfsense subntes and the wireguard subnet.
On the oracle vm & pfsense, can you see the all the routes? Run “ip route”. Should see something like,
10.0.30.0/24 dev wg0 scope link
10.0.38.0/24 dev wg0 scope link
10.0.39.0/24 dev wg0 scope link
10.0.40.0/24 dev wg0 scope link
10.0.222.0/24 dev wg0 proto kernel scope link src 10.0.222.1
oracle VM------internet-------myISP-------pfsense firewall--------desktopPC/freepbx/synology NAS (192.168.1.0/24)
even i agree, 0.0.0.0 should also include all other ip ranges including 192.168.1.0/24 but in real life if i do not include LAN subnet i am not able to do client to client pings
oracle VM ip route
ubuntu@webhost:~$ ip route
default via 10.0.0.1 dev ens3
default via 10.0.0.1 dev ens3 proto dhcp src 10.0.0.224 metric 100
10.0.0.0/24 dev ens3 proto kernel scope link src 10.0.0.224 metric 100
10.0.0.1 dev ens3 proto dhcp scope link src 10.0.0.224 metric 100
10.66.66.0/24 dev wg0 proto kernel scope link src 10.66.66.1
169.254.0.0/16 dev ens3 scope link
169.254.0.0/16 dev ens3 proto dhcp scope link src 10.0.0.224 metric 100
169.254.169.254 via 10.0.0.1 dev ens3 proto dhcp src 10.0.0.224 metric 100
That is what I thought your route would look like. With 0.0.0 you have replaced your default route on your oracle vm to that of the wg interface. That is probably breaking a number of things, especially if you want the oracle vm to be your bridge connecting your remote stuff to your pfsense LAN.
Your ubuntu wg routes should look like this. One big exception, my wg server is not running on my router (for security), so the default route is to a LAN gateway (192.168.4.1). Yours should list your oracle WAN IP.
default via 192.168.4.1 dev host0 proto dhcp src 192.168.4.2 metric 1024
10.0.30.0/24 dev wg0 scope link
10.0.38.0/24 dev wg0 scope link
10.0.39.0/24 dev wg0 scope link
10.0.40.0/24 dev wg0 scope link
10.0.222.0/24 dev wg0 proto kernel scope link src 10.0.222.1
To achieve this you probably only need this on your wg server. Routing from pfsense probably only needs 10.0.0.0/24 given that all remote endpoints are basically in the wg subnet (no remote subnet routes to think about).
AllowedIPs=10.0.0.0/24, 192.168.1.0/24
Once you have done all this test a ping from: laptop-----oracle VM------internet-------myISP-------pfsense firewall--------desktopPC/freepbx/
