Modem > asus-wrt > xcp-ng host > pfSense vm firewall

You need port 4 to allow VLANs 100 and 500 based on your drawing.