the voip.ms website and SIP pops are under a DDOS attack:
https://twitter.com/voipms/status/1438963491994546178
I have customers being affected and I assume Lawrence Systems uses voip.ms for SIP trunking as when I call their main number I get a very garbled voice attendant message.
this is NOT GOOD for a critical service like SIP 
Yup, its sucks that someone is attacking them so persistently.
Do the owners of the infected machines perpetrating this attack will be informed of their situation ?
For exemple, a few years ago, I downloaded an in-theater movie from a torrent site. My ISP sent me an email in the following days stating that the movieās producer might sue me and that I should avoid doing that.
I mean, all those machines canāt all be behind VPNsā¦
Is there a way that their owners will be noticed and correct the situation in order to not perpetrate other attacks ?
There could be hundreds or even thousands of botnets used in a ddos attack so in the short term it would be like herding cats to mitigate an attack at the indivual botnet level becasue the attacker could just recruit or enable more bots. In the long run, yeah it would help if everyone took some preventive measures to prevent their machines from being compromised and used in a botnet, but good luck with that.
I found this article which has a good overview of the problem and prevention measures:
Looks like voip.ms is in the process of implementing Cloudflare to mitigate the attack, but that could take a while. What troubles me is that why they didnāt have ddos mitigation in place before since it appears their have been several high-visibility ddos attacks on sip providers over the past few months:
.
Been weathering the voip.ms storm the last several days. Luckily our managed VoIP customers are spread across a few different ITSPs - partly specifically for this reason. Regardless, it seems that the world just got a lot more complicated for ITSPs - Iām not sure how you defend a SIP POP against 100GB/s firehose and Iām not sure anyone else knows either right now. Implementing Cloudflare will keep the website/portal up - but I donāt believe Cloudflare can help with defending a SIP Proxy/POP.
Thanks for that link, much appreciated.
One of the two voip.ms pops we are using for clients is now back online so it seems they are gradually restoring servicesā¦ at least appears that way.
As to the question about how to mitigate ddos on a bunch of SIP POPs vs a single website, Iām not sure either but Iām sure its a more complex problem.
One thing for sure is that Iām going to vet any SIP providers I use going forward to make sure they have detection and mitigation measures in place.
DDoS might be a good topic to cover in a future episode @LTS_Tom maybe with someone from a vendor in the mitigation business?
I had been talking with bandwidth.com for a while about switching providers, but voip.ms never gave me a reason to make the move. I am going to revisit that conversation next week.
I would be very interested to hear from someone smarter than me about DDoS mitigation for a service like SIP - it seems like a much bigger technical challenge than mitigating a non-realtime service such as http(s) - which is clearly a huge challenge in and of itself.
I contacted them via DM on Twitter and they told me to switch to a specific server. My backup plan is to have all calls forwarded to my cell. Hopefully thatās going to keep working tomorrow. Iām a small client, with limited calls, so this option works for me. Iām just waiting to see if the newly recommended server will keep working tomorrow. I did a few inbound tests today and it worked and forwarded to my cell like I set it up to do. Iām not holding my breath, but if I can at least get the caller ID from the inbound call, I can call them back.
I first noticed an outage Thursday afternoon when an appointment came in and told me they called to let me know they would be early, but the phone said something like I was not taking calls. I called from my cell, and it was busy. No service for the most part since then.
I donāt need the phone right now, but letās be honestā¦ Every business needs their phone system working! Luckily I mainly do emails, but Iām sure this has cost me business.
Still happy with their service and understand their situation.
If tomorrow goes smooth, Iāll be happy. 1.5 business days out is not a deal breaker.
Hang in there everyone!
Reason for cell and or good old POTS lines for backup.
They are being extorted:
https://twitter.com/TomLawrenceTech/status/1439964636183465989
OMFGā¦Fortunately my clients have been understanding about their up and down phone serviceā¦I know last week a couple European ITSPs were under persistent attack like this also, I imaging others are going to find themselves under attack also.
Like others have pointed out, itās hard to beat 1000s of Mbps coming at you, itās really going to have to come down to individual ISPs cutting off customers that seem to be part of the botnets. Many SIP applications use a Session Border Controller (SBC) essentially as a SIP firewall, but even if you are stopping traffic from reaching the endpoints, if you have a full pipe, nothing is moving.
As much as Iāve liked voip.ms service and their support (I had DMs on Saturday night helping me find pops that were running to move some of my clients to), if they are unable to get the ddos under control, I may have to move some clients elsewhere for more reliable service.
What other ITSPs are people here using. Iām using Vitelity at a couple of sites, but theyāve become less admin friendly after the Inteliquent bought them. Iāve always eyed didforsale.com, but have never tried them. Flowroute looks interesting also, but have yet to try themā¦
I have used Flowroute as my backup/secondary provider for years. They are much more carrier oriented - no sub-account type of functionality or PBX features - but very reliable. I am going to be looking at bandwidth.com most likely moving forward.
That being said - the honeymoon may be over for a lot of ITSPs - I really donāt know how anyone can effectively stop an attack like this. Iām not pretending to be an expert - but on the surface, there doesnāt look like a lot of effective options once you are targetted.
Your firewall needs to be able to handle the RTP flows correctly if you try Flowroute - they do not anchor the RTP streams, so your RTP and SIP will be coming from different IP addresses.
Yeah Iāve been trying out Flowroute and had to open the RTP range of my PBX to the world because of what you just described. Iām a little concerned about it because the PBX will receive any UDP packet sent within that range even if itās not related to an active call. I was thinking about setting up a machine on my network and send arbitrary packets to the PBX to see how it handles them.
I good SIP aware firewall should handle that. I use Mikrotik routers a lot and they handle it just fine. You should not just open the RTP range to your server.
What you really need is a session border controller (SBC).
I have some SIP trunks from a local CLEC, and they require a SBC for their service
For direct competitors to voip.ms you have Call Centric or Anveo. These providers have comparable levels of features where you can basically create a cloud based phone system with 95% of the features you might expect on a full PBX just minus a few little things (Paging, BLF button support, and phone provisioning are the ones I can think of). Anveo (not anveo direct) has impressive features to build customized call rules but it can be a bit of a learning curve at first. Call Centric is also popular, I have no personal experience here.
On the full service side you have your Ring Central, or 8x8 type of providers. These tend to be pretty expensive options, I donāt find them very interesting personally but I can see the appeal for some. If you just want a business phone system with full features but donāt want to run any of it yourself these are good.
On the Trunking side you have your Flowroute, Telnyx, Twilio, and dozens of others. A mixture of capabilities often exist where they claim to be more than simple dumb trunking providers which is fine. Twilio and to lesser extent Telnyx for example is heavy into exposing API control so they tend to integrate with a million third parties. I personally really like Telnyx here. Usually do not have IVR, or very limited IVR features and few if any more enhancements that let you build a PBX type functionality. Maybe only voicemail for these kinds of providers.
Next you have the wholesale aggregators. Bare bones SIP trunking services that focus on least cost routing at the cheapest rates possible. This would be Anveo Direct (Not anveo retail) and ThinQ. Maybe not always the best choice and they can be hard to setup as you pretty much need a PBX and likely static IP with fairly permissive RTP ranges on the firewall. I love using these as the outbound carrier for calls from internal PBX. The prices are so crazy low itās amazing to me how a average SMB company can basically use 10k outgoing minutes a month for what is lunch money. Price is around $0.002/min (2/10th of a penny a minute)
BulkVS is one that seems to be somewhere between wholesale aggregator and trunking provider. They were more focused on providing e911 service early on, but have moved quickly to add more full slate of services. Now they are the cheapest option for inbound trunking in many areas by far, the 911 service they provide is still very good and valuable, outbound they are a weak option (fill in the weak spot with anveo direct if you need more than basic north american calling). Iām finding these guys a very interesting niche player that fills in some important holes for me.
Last is the true wholesale providers like Bandwidth, Iristel, and other CLEC providers and if you have the scale for it direct connections to some big tier 1 carriers. Can be harder to work with, might require minimums or longer time to sign up and be approved.
So for the purposes of what to do while you wait for voip.ms to mitigate the ddos? Likely depends on how you are setup. If you have lots of individual phones trying to register to voip.ms server it can be hard to go back and reconfigure everything. Anything you change temporarily probably needs to change back eventually. Service is working a bit better though still intermittent. Might want to setup forwards to mobile numbers while you wait.
If you have a single PBX and you can add a new SIP trunk and change a few routing rules it will be easier to put in work arounds. With another carrier your outbound calling should already be solved. Inbound calls, maybe forward to temporary number? Forwarding seems to have worked ok for me today so that seems like something they worked to fix. Will be harder for them to fix SIP traffic to customer phones while ddos still ongoing, at the end of the day when they hammer you with traffic there isnāt much you can do with it except try to get the upstream flows blocked but itās whack a mole.
As things go longer, those ddos nodes can start to burn out too (isp abuse reports, customer āfixesā their broken routers, etc)
Not sure if anyone is following here - but thought I would share some details of what I am observing for VoIP.ms POPs -