Here is the eco system.
I have a kubernetes cluster provisioned in xcp-ng. I have a tailscale pod running that advertises the kubernetes subnet and also accepts routes. The subnet is 192.168.0.0/16 I have tailscale setup to dns cluster.local to 192.168.1.10 (internal kubernetes dns server)
I have pfsense on a totally separate machine and have installed tailscale package in pfsense and also advertises the private network of the machines and vms on 10.1.1.0/24 and accepts routes. I have tailscale setup to dns mydomain.local to 10.1.1.1 (ip of pfsense and dns resolver)
The physical machines pfsense, xcp, nfs AND vms are all on the subnet 10.1.1.0/24.
I have tailscale on my home pc out side of the physical network described above
So what is going on?
-
CAN
dig loki.monitor.svc.cluster.local @100.100.100.100(tailscale resolver) I get the correct ip returned. -
CAN
dig loki.monitor.svc.cluster.localfrom home (outside of pfsense firewall) with no problems. -
CANNOT dig from a vm using
dig loki.monitor.svc.cluster.localthis fails to resolve an ip. I find a server and the resolving server 127.0.0.53 returns an empty record. -
CANNOT
dig loki.monitor.svc.cluster.local @10.1.1.1(pfsene) I get an empty record, no ip returned. -
CANNOT
dig loki.monitor.svc.cluster.local @192.168.1.10' (kubernetes resolver) I getreply from unexpected source: 10.1.1.1#35667, expected 192.168.1.10#53`
What have I done to try to make this work?
- allow udp 41641 on wan interface to any and all.
- Enable the UPnP service and Allow NAT-PMP Port Mapping
- allow all and any on the tailscale interface
- setup a domain over ride, cluster.local → 100.100.100.100
I have exhausted my ideas. Would love to make this work so any ideas no matter how “out there” are greatly welcomed and hoped for. Thanks for taking the time to read.
Brad
