Ipsec Site to Site VPN breaks lan to vlan communication

I have a new install of Pfsense 2.52. Added vlan to Lan interface. Configured ipsec vpn to Headquarters Firewall. Added Phase 2 for Lan and Vlan network. Both network communicate with Main office just fine. But when the VPN connects I can no longer ping the Vlan from the Lan and vice versa. The logs show the pings go through the rules that have been setup. I see no logs that show anything being blocked. I do see logs that show the pings going through the rule. Not sure its a Rule issue as it seems its a Routing issue. Disable the VPN and they can now talk. Been stuck on this for days

Do the local subnets match the remote ones? If so then they are being redirected.

Yes, the Phase 2 connection is fine. Talking back and forth from the test site and the main network is fine.
I also noticed that once the VPN is connected I can no longer ping their interfaces from within their own networks. Disable the VPN and now I can.
This seems to be pretty basic of a setup so I am confused as to why its not working.

Here is more to the puzzle.
I created a Rule on the Ipsec to allow the Main network to ping the Vlan interface. That is working.
But the computer on the Vlan is unable to ping the Vlan interface. I am so confused.
The Vlan computer can ping the Main network just fine. GRRRRRRRRRRRRRRRRRRR

Loaded up version 2.4.5 and the same thing. Lan devices can ping its interface but anything on the VLAN cannot ping the VLAN interface. For fun I configured a third network on its own interface. Same as the Vlan. Again, this is only when the Ipsec is connected. The interfaces seem to be protected as I need to add a rule on the Vlan interface to access it when there is no Ipsec connected. About ready to give up on Pfsense again. Seems to always have some sort of issue that other paid options do not have. Would love to see someone try and reproduce this. Not sure what I am doing wrong since I do not see anything showing a block. Sounds like a route issue. This is very basic setup that I would think would work.