Im posting our disaster recovery process for a client of ours here using Solarwinds MSP Backup since Tom ultimately helped us choose the product in the first place. *thanks TOM!
I work with an MSP in Houston TX. watching Toms review last year on Solarwinds MSP Backup we decided to implement it for several of our customers.
Since we setup the product we have only had to perform individual file restores which has been quick and easy. Just 2 days ago our client they an erratic power failure i was told. It took down the Citrix XenServer on Dell hardware with a raid failure, taking down 2 virtual machines on it… 1 being their only domain controller/file server and the other their ERP with MS SQL on it. My team is in Houston, this client is in Tupelo Mississippi and on a slow DSL line so support can be challenging at times.
Both servers had full backups in MSP Backup with System State… sighhhhh of relief
But then we were tasks with putting the backups into action for a disaster recovery.
Luckly our client has a newer Dell server with Server 2012 R2 on it that they were prepping to be their new ERP server, but had not been put into production yet. its slim on memory and disk space but we decided it will fill the job for the restore.
Our first call to Solarwinds MSP support got us to a knowledgeable tech. he immediately pointed us to installing the “Recovery Console” for MSP backup on the server. He showed us the virtual restore option which lead us to having the ability to restore the complete system to a hyper-v or vmware hypervisor. We installed Hyper-V on the server and was able to get the restore process underway. The key piece here is the “Local Speed Vault” if you enabled it on your backups and have it available, it will pull the restore locally instead of down from the cloud. We had a copy of the Local Speed Vault on a Synology NAS share for both of these XenServer VMs. The Recovery console lets you set the location of the local speed vault for the restore.
We just finished the restore of the domain controller and have it booted up! Even with the local speed vault the restore took about 5 hours for 100GB of data, support said this is due to the decryption that happens during the restore. This process does require an internet connection which can be a major drawback for some.
A good take away, is that restore seems possible regardless of the system being physical/xen/hyper-v/vmware… it will restore to hyper-v/vmware/baremetal or your common virtual disk types.
I’m interested to hear of anyone’s experience going through a DR scenario with the product.
We have done a few DR restores from completely failed hardware and our experience was much the same. We did have one incident where the local speedvault & server was destroyed but their support was great at helping us make a few tweaks to use the full speed of the clients connection to get us up and running as fast as possible. We have been really happy with the product.
I’m all to familiar with that area and DSL connections. Couple of weeks ago I was at a client with 1.5Mbs down. Does Solarwinds support having a local copy as well on an external or something? We do that a fair amount since it’s faster than restoring from a cloud backup.
Yes, the local copy is kept in a folder termed “localspeedvault”. The data is kept in this folder in the same encrypted format that is kept in their cloud. Both locations stay in sync.
The downside here is the recovery time… In my recovery time, even with the data all being local… the data has to be unencrypted using the private key… and your recovery system has to be connected to the internet during this process. We were working with 101GB restore for a single server, even with E5-2640 CPU’s it took right at 5 hours to complete.
This morning we are working on the SQL server restoration which is closer to 300GB, about 2 hours into it now.
Missed that in the original post. Ouch. That’s still pretty bad but it could be worse. Sucks it happened on Easter weekend. The weather in that area has been pretty rough this past week.
yea… could have been worse. We have a tech at the office that verifies backup logs daily for the customers… and performs quarterly test restores on individual files. We are going to work in virtual restore tests after this issue!
If this was any of our bigger clients, this restore time would not fly.
Your bigger clients hopefully don’t have DSL as their primary. Poor internet connection is whats stopping us from moving to more cloud based solutions, unfortunately.
Thanks for sharing your experience. I’m on NCentral with the same MSP Backup. I started using MSP Backup a couple months ago for a few servers and have had a couple issues. I had no idea the restore times were so bad. I’m going to have to do a couple tests and see if it’s going to fly with my clients.
Both servers were restored using the virtual restore option to a Hyper-V server. Total with both servers together it was roughly 360GB of data within the virtual harddisks. Using the localmediavault, restore time was still close to 12 hours for both systems… that time is not counting the tech calls and us trying to clear disks space on the server, enabling hyper-v etc.
The MS SQL server booted up and had the database mounted. The customer went back onsite last night and verified all data was there, only losing 1 day of entries due the the backup not working that day due to the crash happening 1 hour before the daily backup at 11pm on Wednesday…
All in all, im happy with the product. it really came through for us here… i would highly recommend that you discuss the restore time in a DR scenarios with your customers… a 1 or 2 hour restore does not seem possible but test,test,test. Make sure you use the localspeedvault, we would have not been able to restore without it due to the client being on a 7/1 DSL line.
Not too bad. Thanks for the update. Restoring a backup is always a little scary and those restore times don’t seem too unreasonable but if it were double to triple the data/time I’d be concerned.
May I make a suggestion, check out the Backup and Replication software from Veeam. They have a price tier for the SMB market. Supports VMware and Hyper-V plus bare metal, Very low RTOs. Full disclosure I am a Veeam Pro Partner.
What RMM are you using and is Veeam integrated? I went with the MSP backup because it’s integrated with NCentral. That said, the integration is kind of weak. I mean the backup dashboard isn’t even under the dashboard heading - it’s under configuration. Is it really that hard to fix a hyper link???
Veean works with the Connect wise. I use machine to login to a management server. Veean end point can integrate with Veean suite. Get the docs from Veeam.com