Head Guru Posted March 18, 2007 Posted March 18, 2007 In mid January TotalChoice Hosting started on its move to a data center in Michigan. We signed a deal with a wholesale bandwidth provider and our transit was coming in via two diverse paths from AT&T and level 3. Our preference was AT&T. Shortly after that we received our IP allocations from the upstream provider and we were ready to go! Starting in February we started moving some servers into our racks in the new space and things were going great. We started several server migrations and clients were praising us on the speed of the new servers and how fast things were going. Everything was moving along. Several days into the first round of migrations we were notified by clients about mail bouncing from several different ISP's. AOL and Earthlink were the two biggest ones. We immediately stopped migrations and started to look into this issue. We knew right away that the IP's provided to us were contaminated from a prior user and we were in for a rough ride. What seems like 4,000 emails later to AOL and Earthlink we thought the blocks were cleared off our new IP's? We checked over and over again and the clients noted the same things looked good. We started to go back into migration mode, and once again two days after migrating more servers we were hit with more blacklisting complaints. We stopped all migrations again at that point. A couple weeks roll on, we are still fighting with the upstream over the IP issue and then the worse happens. We start to notice issues over our DS3 lines. We started noting latency and packet loss and it has really not gotten any better. We have had several meetings with the wholesale bandwidth provider and looked at several routes to resolve the issues. We did not appear to be getting to a point of resolution in the time we needed. I am not one to allow my business to be slowed down by the lack luster performance of others so we started to take action. Due to the lack of performance with the current suppliers, we will be canceling the existing contract with our preferred upstream, and moving to a point where we will have excellent connections. We have already turned up a OC-3 to XO Communications, a gigE to Cogent Communications and a gigE to Level 3 will be up in the next 72 hours. So as you can see we will have plenty of backbone strength. We are looking to change during March. This will involve some downtime as we get everything changed over. Due to all the ongoing blacklisting issues, we will also be changing IP's so that should clear many of the issues associated with emailing to certain domains.
austin Posted March 18, 2007 Posted March 18, 2007 Thanks for the update, so are the migrations going to continue while this is worked out?
TCH-Andy Posted March 19, 2007 Posted March 19, 2007 There will be a short pause in the migrations, then they will be to the new connections
Madmanmcp Posted March 19, 2007 Posted March 19, 2007 Change is never easy, is it. There were a couple of blips trying to use the forums this morning, hope it was just blips and not a sign of more problems. Hoping for smooth sailing here on...
Head Guru Posted March 19, 2007 Author Posted March 19, 2007 Those blips would be fat #$! me breaking the mySQL server. Sorry guys. Things should be awesome now!
Deverill Posted March 19, 2007 Posted March 19, 2007 (edited) Bill, thanks for the updates and even more for not taking the easy way out by saying "It's not our fault..." like so many others would have. Again, this is why TCH rules. Unfortunately, that kind of thing can happen and the providers probably don't keep track of blacklist status on addresses returned to them. At least we can be assured that you're "on it" and doing everything you can to resolve these kind of issues. Edited March 19, 2007 by Deverill
Head Guru Posted March 19, 2007 Author Posted March 19, 2007 Thanks! It has been a pain thats for sure. However, the core TCH servers are now up on the new bandwidth providers and we are hoping to see great improvements with load times and speed. Time will tell. Do let us know what results you are seeing when browsing the forums, submitting a ticket or ordering your new hosting package Bill
Rednarb Posted March 19, 2007 Posted March 19, 2007 Thank you for the update. I was suspicious of some issues at the new DC (one of my domains is hosted on Naboo) but I didn't have enough evidence to submit a ticket. So I posted a question here on the forums which was promptly deleted. I then got a PM saying if I was having trouble to open a ticket. I wish he would have just told me that there were known issues. I then attempted to submit a ticket via email (which is how I've submitted low-priority issues in the past) and got a reject message of some sort. I blew it off, thinking you'd get it all fixed up eventually. Luckily for me I don't run this site to make money. I've had to endure complaints from my friends though, but I can handle that.
Head Guru Posted March 20, 2007 Author Posted March 20, 2007 We have been running our core servers on the new providers fiber for two days now. Anyone noticing any improvments? Looking forward to hearing any feedback.
Rednarb Posted March 20, 2007 Posted March 20, 2007 We have been running our core servers on the new providers fiber for two days now. Anyone noticing any improvments? Looking forward to hearing any feedback. Very good performance once again!
Mish Posted March 22, 2007 Posted March 22, 2007 Could this be why my SMF forum is down (was working last night, now I can't even access the cpanel)? As far as I know it's on Naboo as well, though the rest of my domains are working jus fine. Like Rednarb, I don't want to submit a ticket if there's a perfectly logical explanation.
TCH-Thomas Posted March 22, 2007 Posted March 22, 2007 I think Naboo have been fully migrated so please submit a ticket and the techs will look in to it.
Head Guru Posted March 23, 2007 Author Posted March 23, 2007 We are lighting up a gigE to Level(3) at the moment. Currently some outbound routes are routing over Level(3) and we should soon have inbound routes in our router BGP. :) GO TCH!
The Mighty Billy Posted March 25, 2007 Posted March 25, 2007 I spoke to someone last night about my website ( http://planetlive.net ) experiencing downtime of some sort. It's been unaccessible for almost a full day now. He told me it would be back up within a couple of hours. I still can't access the site. Is that being caused by this update? He claimed it was.
TCH-Bruce Posted March 25, 2007 Posted March 25, 2007 Welcome to the forums Billy. I can see your site just fine. Try clearing your browsers cache.
heyguy Posted March 25, 2007 Posted March 25, 2007 They changed all the ips on the servers last night, so there would have been a little downtime during propagation. The site is loading fine for me now. Try clearing your cache using ipconfig /flushdns from the command line.
The Mighty Billy Posted March 25, 2007 Posted March 25, 2007 Welcome to the forums Billy. I can see your site just fine. Try clearing your browsers cache. Yikes! You're right. I've been refreshing all morning for nothing. I opened it in Firefox and it showed right up! Thank you.
TCH-Thomas Posted March 26, 2007 Posted March 26, 2007 Welcome to the forum, Billy. I am glad it is working for you now.
clayartisan Posted March 27, 2007 Posted March 27, 2007 I just wanted to say a VERY BIG THANK YOU, to TCH and staff for these improvements. When I was rebuilding my store during February, I was experiencing frustrating and frequent time-outs, where I could not connect at all to my website. By running a route tracing program, I could see that a bottleneck was occuring at certain points, close to the TCH arrival. Since these improvements, the time outs have all but disappeared and the route tracing supports this. The improvements are definitely appreciated by this user! Patti Stoll http://www.pattistoll.com
Recommended Posts