Got a question for the pfsense admins who manage remote boxes. I setup a long distance friend with a pfsense box for his very small business. I have admin access on it and can reach it via vpn or from my ip. My question/concern is how risky is it to update this box from 22.5 to 23.x remotely?
I have heard there may be some issues with this update (php and package res-installation stuff), and I am having second thoughts about plowing ahead. I was going to rely on my “back door” access with the fw rule for my IP address, but I am less confident about that if the vpn client doesn’t dial home and at least show me his potentially new IP address. That and I am concerned about this update going so wrong the web interface doesn’t come up due to some catastrophic error.
How do you guys admin long-distance remote boxes? Just cross your fingers? I am thinking about taking the conservative approach and buying a new proteclli box (or another netgate box) and just having him swap it out. This would also give him a cold backup too.
Also, if you guys do box swaps like this for long distance updates, do you wait a little longer between big updates? Maybe a year or two? I know the risks, just weighing how long I should drag my feet on this.
OK, thanks. So the best course of action is to remove the wireguard package, run the update, say a little prayer, reboot, and hope he gets the same IP. If not, then I will need to call him. And then I hope the GUI comes up without any snags. Then reinstall wireguard.
Before any remote upgrade of any vendor, I make sure to take a config backup, and then reboot the appliance. Rebooting first lets you separate any recent hardware issues (failed drive might be silent in operation) from effects of the upgrade itself, and also clears out temporary files and cached memory.
Thanks for the feedback. I watched the video and have been mulling this over since my post.
A lot of the points in the video do assume physical access to the machine. Restoring from backup would be a nightmare with a pissed off friend over the phone. I knew admin’ing a machine from afar was risky, but now I feel it. Damn, the stove is hot!
So when the low probability event happens to you guys, is your plan to hop in a plan/train/automobile and hurry you or your tech’s ass over there? Do you mail them a replacement device? I am thinking the replacement device is the best route from the get go. They are using the cheap netgate box, so having a spare isn’t insurmountable. But I am still curious how others do this?
If I were to do this again, I would set that expectation from the beginning. You got to buy two devices, for physical backup and the annual or every x year update. Or they can always pay the local guys a pretty penny to run whatever meraki device they are hustling. Feel free to correct me if I am wrong.
You can download the config, load it on an up to date box that is the same as the one they have, ship it to them, and tell them to plug it in same way. It’s a simple process that works.
Yep, that is what I was thinking. Basically swap the two boxes back and forth, and always have the spare with the latest config in-case of hardware failure. There are pros and cons to this setup, but it is the safest way to do this AFAIK.
Got to admit, this update process is a little clunky. I imagine meraki and the others expensive vendors have update processes that are less labor intensive.?.
You will find the most about updates from pfsense because it’s used by so many, has forums, and community support. When updates from big vendors go wrong (and they many times do) it rarely ends up in a public forum, mostly people yelling at their support people and once and a while ranting on Reddit about it.
Depending on your level of sanity I created a python script to update pfsense boxes remotely via SSH.
The script will reach out in parallel about 5 pfsense boxes to do its upgrade. Once its done it will spit out a file with the logs of the upgrade based on the name.
Prerequisite
Enable SSH on you pfSense box
Supply the Name, IP or Hostname, Port Number, Admin account and Admin password in the Endpoints section of the script. Or if you don’t want the usernames and passwords in the script, you can use a file and load the endpoint list that way.
pip3 install paramiko tqdm
import paramiko
import concurrent.futures
import time
from tqdm import tqdm
def run_command(ssh, command):
stdin, stdout, stderr = ssh.exec_command(command)
exit_status = stdout.channel.recv_exit_status()
output = stdout.read().decode().strip()
error = stderr.read().decode().strip()
return exit_status, output, error
def connect_ssh(host, port, username, password):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, port=port, username=username, password=password)
return ssh
def process_endpoint(endpoint, pbar):
try:
ssh = connect_ssh(endpoint['host'], endpoint['port'], endpoint['username'], endpoint['password'])
pbar.set_description(f"Connected to {endpoint['name']}")
pbar.update(25)
time.sleep(2)
command = 'pfSense upgrade -y' # The command you want to run on each endpoint
pbar.set_description(f"Executing upgrade on {endpoint['name']}")
exit_status, output, error = run_command(ssh, command)
if exit_status == 0:
pbar.set_description(f"Upgrade execution on {endpoint['name']} successful")
with open(f"{endpoint['name']}.log", 'w') as file:
file.write(output)
pbar.update(50)
pbar.write(f"{endpoint['name']} update complete.")
time.sleep(2)
else:
pbar.set_description(f"Upgrade execution on {endpoint['name']} failed")
pbar.write(f"Error message for {endpoint['name']}:")
pbar.write(error)
except paramiko.AuthenticationException:
pbar.write(f"Authentication failed for {endpoint['name']}")
except paramiko.SSHException as ssh_exception:
pbar.write(f"Unable to establish SSH connection to {endpoint['name']}: {str(ssh_exception)}")
except Exception as e:
pbar.write(f"An error occurred while connecting to {endpoint['name']}: {str(e)}")
finally:
if ssh:
ssh.close()
pbar.update(25)
def main():
endpoints = [
{'name': '<Insert Name>', 'host': '<Insert IP or hostname>', 'port': <insert port number>, 'username': '<Insert admin>', 'password': '<Insert admin password>'},
# Add more endpoints as needed
]
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = []
progress_bars = [tqdm(total=100, position=pos, leave=False, miniters=1) for pos in range(len(endpoints))]
for endpoint, pbar in zip(endpoints, progress_bars):
future = executor.submit(process_endpoint, endpoint, pbar)
futures.append(future)
# Monitor the progress of tasks
for future in concurrent.futures.as_completed(futures):
pass
# Close progress bars
for pbar in progress_bars:
pbar.close()
if __name__ == '__main__':
main()
Ah, very cool. I imagine something like this is how most MSPs update a fleet of pfsense boxes. Clicking through the GUI doesn’t scale well. The only problem I can see is the cli update (pfsense upgrade -y) does not appear to follow the “best practice” of uninstalling packages before running the update, and then reinstalling them. Correct? Not doing this would bump up the risk by some unknown amount (which is arguably still small, but less small). But then again, adding all the code to do that in your script (if possible) wouldn’t be risk free either. Trade offs I guess.
Even the expensive and low tech approach with meraki (or some other equivalent) wouldn’t be risk free. They advertise automatic updates while you sleep! Any admin worth his salt would hate that idea. And Tom is right, they absolutely have their failed updates too.
I think I am going to do the unthinkable and just let my long-distance pfsense box sit. I may get the courage to update it around Christmas time, but I could see myself just closing my eyes and leaving it alone. I am weighing the cost of two low probability events (update failure vs future bugs), and I think I am less scared of some internal device attacking some future pfsense vulnerability. I wish I had gone with a different setup for him, but he wanted “easy” just in case.
If you disagree with any of this let me know. I value your opinion.
I think its all about the risk tolerance if you want to do something like this. I have preformed upgrades with this python script twice now with 18 pfsense firewalls in my script and all has went well.
That is true, the docs don’t say uninstalling packages is necessary. However, I still trust Tom when he suggests it is a good idea. The worst case is it’s unnecessary.
I guess my problem is two fold. 1) admin’ing machines two states away is a bad idea. I should know better. 2) I setup a system that I don’t have a lot of experience with & I don’t run myself. If I had a pfsense box here and was into it that would probably alleviate most of my concern.
I think turning a blind eye to updates is going to be my play (I can’t believe I am saying this). The biggest risk I see is with DHCP & DNS. Every user on the LAN interacts with it, and it is deployed dangerously in pfsense. The web login is on a small VLAN that should only have a few devices. All told, as long as the LAN devices are reasonably safe and there isn’t a remote code vulnerability with those must have services then this threat surface should be small. I do have to think about the access their stuff has to my network. I host their unifi controller and allow them access to my jellyfin & searx servers. As time goes by I should probably close those off. Unifi communication can be turned off for months at a time if need be.
If the client doesn’t have a public IP reservation, you can also set up dynamic DNS using something like afraid.org which gives you a free option. Once you’ve got your dynamic DNS entry set up and verified, you won’t have to worry about the IP changing.
That can work too. But I am concerned about the box not coming back up. That being, not coming back up at all or in some impaired state where it doesn’t launch said service.
There is always a small chance you need to be physically at the device. Which goes for anything. I should have used a setup that is a little more “appliance like”. Or better yet not taken on this in the first place.
