While I understand that they are under attack it makes me wonder why they didn’t have a better plan for this in place. It’s not like DDoS is new. I am considering porting out to Telnyx. My internal phone system has been spotty for days. For days I have been able to at least get inbound calls so I was ok with that. But today it’s a steady busy signal on inbounds. My business is suffering and I can’t hold out for even another day.
What is the “Plan” you propose for dealing with a large scale DDoS attack? How many SIP Proxies you think are needed to fend off an attack? How would you capacity plan for the scale of the attack?
@LTS_Tom Thanks for the YT video!
I think they have stumbled a bit, but no one ever sees the how the dominos are lined up until they start to fall.
I am really hopeful that they either standup or convert an existing proxy as a dedicated forwarding proxy with only backend access to their upstreams. That would be massively helpful and doesn’t seem like it would be a huge technical challenge - but I am talking over my paygrade.
My second hope would be to see certain proxies which are dedicated for whitelisted traffic for subaccounts configured for IP Auth - I would think that would push the problem back down to layer 3 and make it easier to keep those proxies up. I could switch all of my sub-accounts to IP Auth tomorrow if it meant having some proxies that they can effectively mitigate.
I guess it doesn’t help to arm-chair quarterback the situation - but I’d really like to see some definitive solutions start to take shape now that they have the basics under control with DNS and portal access.
The creator of FreePBX has a reddit discussion going on the topic here
https://www.reddit.com/r/VOIP/comments/pqwl14/what_we_know_so_far_about_the_voipms_outage_keep/
That’s odd. What is the reason for that? What kind of Trunk can your SBC do that any Asterisk/freeSWITCH based PBX cannot?
I see a lot of people are looking to port their numbers from voipms to a competitor of theirs, which IMO is a huge mistake.
These attackers have already attacked 3 providers and will continue attacking others once voipms recovers from this. If you are looking to switch provider, consider a larger provider that their infrastructure can handle large DDoS attacks.
I personally think, that once this is over, the voipms team will continue looking for DDoS prevention solutions as they have been hit hard, VS any competitor that did not experience this fire, will not be so eager to have something enterprise in place.
Honestly, I don’t think it’s bad at all to look at other options. If anything, this whole event points to the bottleneck and risk areas of the current VOIP infrastructure. We’ve ported a few of our numbers to another provider to keep our businesses alive. It’s nothing personal, but it’s what we’ve had to do.
That being said, we’re not going away from VOIP.ms, because we understand what’s going on, and we’re aware of the risk being exactly as described here.
What we ARE doing is having a second public number in a few cases, and we’re intentionally keeping the numbers diverse in terms of the provider we’re using. We’re also adding failover logic for our outbound streams with a webhook that our customers can use to manually force a throw over. This is mainly to help with any outbound calling limits.
We’re still figuring it out, and it’s not ideal, but it’s better than being without phones for six days – and that’s just the current count. This is probably far from over.
egftchman try Telnyx good rates and support https://telnyx.com/
about Telnyx’s network The CPaaS Network for Real-Time Carrier-Grade Communications
Remember this axiom, Two is one, one is none.
I don’t see anything there that looks like it would address a similar DDoS in any meaningful way. Honestly, it looks like marketing-speak for “we have a bunch of servers on Google/Azure/AWS”. I’m not saying that Telnyx isn’t a great provider - just saying that if you have public facing SIP POPs you are going to be vulnerable to this type of attack. The only real defense is going to be a deep bench of tech know-how, a strong incident response plan and team and deep pockets to throw money at the problem when it happens - IMHO
My voip.ms customers are on west coast pops sanjose 1 and 2 and as of 9/21 they are “up” but failing intermittently. I’m not going to swap to another VSP at this point as there are risks with that too, and my customers are working with me.
I used to work for an ISP/hosting company so I know how hard it is to mitigate a DDoS attack, but as the r/VOIP post and others point out, it would have been nice if at LEAST their website and back-end services had a proxy in front of them (e.g Cloudflare or other) so that customers would be notified and get up-to-date information without resorting to twitter. I’m hopeful that voip.ms will emerge stronger.
Thanks @LTS_Tom for your pragmatic and level-headed video update and posts.
1 Like
Everything I have is on Denver2 and WS1 - when those go, I don’t have a B-plan. I’ve been testing POPs all morning and have found none that are reliable at all - most are completely non-functional. The Toronto1-4 POPs looked promising, but haven’t held up for more than 30 min at a time.
It is really, really difficult to wrap my head around why they have been unable to get a dedicated and reliable forwarding POP online. That would solve all issue for me and I could ride it out for as long as it took to restore full functionality.
Have a second provider.
I really have no idea what that means. I have multiple providers. Only one provider can hold a DID. Being able to receive calls to that DID is the whole game. I’m honestly completely baffled by your response.
I am working on an updated video where I will be bringing on a VoIP expert explaining better how DDoS mitigation works for VoIP systems and what does not work because CLEARLY there is a LOT of confusion.
4 Likes
Great idea, Tom!
I suggest you should bring Fred Posner on the show. He’s an expert in Kamailio and has many videos online about VoIP fraud and security.
Would be a true service to the community.
And all the pops are down as of 1100 CDT on 9/24/21…I’ve been trying to ride out the storm with them and found a couple pops that were pretty stable all week, but over a week of this too much for our clients. I may have to cover their balances sitting in VoIP.ms out of pocket moving them to something else.
I’m getting new account set up for the last couple of days. Will hopefully be getting some number ports moving by tomorrow. The service issues are one thing - the communication is what really pushed me too far. The letter that they released from the CEO/COO was totally tone-deaf and gave me zero assurance that they are up to the task of handling this crisis at several levels.