Head Guru Posted June 2, 2009 Posted June 2, 2009 Hello TCH Family, I wanted to give a status update and inform the family of a network event that has been on going for the past 36 hours. Starting on May 31st at around 10:30am a dedicated server client that we have hosted for several years has been the target of a Distrubuted Denial of Service attack. The client is a simple mom and pop online store selling jewlery and jewelry accessories and has never presented any issues to TCH. This attack started off on a small scale and we were able to filter the attack without issue. However during the afternoon hours of May 31 the attack grew to such a size that we were unable to filter the attack and the attack spilled over into our general network causing a 25 minute widespread outage on our entire network last from 3:40pm until 4pm EST on 5/31. At that time we took the drastic measure of null routing all the traffic to this specific server. The null route worked and our traffic levels returned to normal and have been at a normal level until this morning at 5:20am. On June 1st at 10:30am we removed the null route and turned the clients server back online. Traffic remained normal and we had no issues until this morning at 5:20am. The attack resumed and once again effected all the servers/clients on our Distribution Switch #4. At 5:20am we made the call to simply null route the traffic to this server again so as not to effect the overall performance of the network. Our network is very robust. We have multiple carriers entering the data center over diverse paths. However, this attack was so large, at one point measuring over 1gbps of traffic, that the inbound attack was able to congest our entire network for those twenty minutes. For the most part we handled the entire attack without any major interruptions to our overall network. This is simply a update to explain the 20 minute outage to the overall network and to help ease the minds of any clients that have been more effected on the switch that the has been the target of the attack. Whilst this attack is still in progess, we have once again null routed the server that is the target of this attack. I will update this post with any more information as it becomes available to me. Thank you for your patience and understanding during the past few hours. Quote
Head Guru Posted June 2, 2009 Author Posted June 2, 2009 Update - The server owner that has been under attack has decided to cancel service with TotalChoice. Good Morning, Good Afternoon - GOOD NIGHT! We provided her many years of faithful service and .... whatever. I have powered down the server and this event is behind us. The attack is still on going, however we are filtering the inbound traffic. Just wanted to updated the family. Quote
OJB Posted June 2, 2009 Posted June 2, 2009 Thanks for the update, Bill. This has got me thinking though, is there any way to avoid this sort of attack? What would happen if a shared or reseller account was targetted? Quote
TCH-Ryan Posted June 3, 2009 Posted June 3, 2009 The nature of shared and reseller accounts is such that the onus is on us to remedy any inbound attack, for the sake of every client housed on that particular server. We can take any number of actions to mitigate attacks, all with varying degrees of success dependent on the type of attack that is occurring. Generally however if an attacker is persistent enough the ultimate outcome may be that we are required to shut off the site in question being attacked or renumerate it as so attackers are no longer hitting a live address. The bottom line is, our goal is to ensure the integrity of our network, servers and as a whole all clients we host on those assets, this sometimes means that we must make hard decisions against those being attacked in the interest of preserving uptime, performance and stability for our clients. Quote
OJB Posted June 3, 2009 Posted June 3, 2009 Thanks for the additional info. Good to know we are in safe hands and you guys are ready to take the hard decisions when necessary for the greater good of the rest of your customers. Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.