Head Guru Posted March 18, 2007 Share Posted March 18, 2007 In mid January TotalChoice Hosting started on its move to a data center in Michigan. We signed a deal with a wholesale bandwidth provider and our transit was coming in via two diverse paths from AT&T and level 3. Our preference was AT&T. Shortly after that we received our IP allocations from the upstream provider and we were ready to go! Starting in February we started moving some servers into our racks in the new space and things were going great. We started several server migrations and clients were praising us on the speed of the new servers and how fast things were going. Everything was moving along. Several days into the first round of migrations we were notified by clients about mail bouncing from several different ISP's. AOL and Earthlink were the two biggest ones. We immediately stopped migrations and started to look into this issue. We knew right away that the IP's provided to us were contaminated from a prior user and we were in for a rough ride. What seems like 4,000 emails later to AOL and Earthlink we thought the blocks were cleared off our new IP's? We checked over and over again and the clients noted the same things looked good. We started to go back into migration mode, and once again two days after migrating more servers we were hit with more blacklisting complaints. We stopped all migrations again at that point. A couple weeks roll on, we are still fighting with the upstream over the IP issue and then the worse happens. We start to notice issues over our DS3 lines. We started noting latency and packet loss and it has really not gotten any better. We have had several meetings with the wholesale bandwidth provider and looked at several routes to resolve the issues. We did not appear to be getting to a point of resolution in the time we needed. I am not one to allow my business to be slowed down by the lack luster performance of others so we started to take action. Due to the lack of performance with the current suppliers, we will be canceling the existing contract with our preferred upstream, and moving to a point where we will have excellent connections. We have already turned up a OC-3 to XO Communications, a gigE to Cogent Communications and a gigE to Level 3 will be up in the next 72 hours. So as you can see we will have plenty of backbone strength. We are looking to change during March. This will involve some downtime as we get everything changed over. Due to all the ongoing blacklisting issues, we will also be changing IP's so that should clear many of the issues associated with emailing to certain domains. Link to comment Share on other sites More sharing options...
austin Posted March 18, 2007 Share Posted March 18, 2007 Thanks for the update, so are the migrations going to continue while this is worked out? Link to comment Share on other sites More sharing options...
TCH-Andy Posted March 19, 2007 Share Posted March 19, 2007 There will be a short pause in the migrations, then they will be to the new connections Link to comment Share on other sites More sharing options...
Madmanmcp Posted March 19, 2007 Share Posted March 19, 2007 Change is never easy, is it. There were a couple of blips trying to use the forums this morning, hope it was just blips and not a sign of more problems. Hoping for smooth sailing here on... Link to comment Share on other sites More sharing options...
Head Guru Posted March 19, 2007 Author Share Posted March 19, 2007 Those blips would be fat #$! me breaking the mySQL server. Sorry guys. Things should be awesome now! Link to comment Share on other sites More sharing options...
Deverill Posted March 19, 2007 Share Posted March 19, 2007 (edited) Bill, thanks for the updates and even more for not taking the easy way out by saying "It's not our fault..." like so many others would have. Again, this is why TCH rules. Unfortunately, that kind of thing can happen and the providers probably don't keep track of blacklist status on addresses returned to them. At least we can be assured that you're "on it" and doing everything you can to resolve these kind of issues. Edited March 19, 2007 by Deverill Link to comment Share on other sites More sharing options...
Head Guru Posted March 19, 2007 Author Share Posted March 19, 2007 Thanks! It has been a pain thats for sure. However, the core TCH servers are now up on the new bandwidth providers and we are hoping to see great improvements with load times and speed. Time will tell. Do let us know what results you are seeing when browsing the forums, submitting a ticket or ordering your new hosting package Bill Link to comment Share on other sites More sharing options...
Rednarb Posted March 19, 2007 Share Posted March 19, 2007 Thank you for the update. I was suspicious of some issues at the new DC (one of my domains is hosted on Naboo) but I didn't have enough evidence to submit a ticket. So I posted a question here on the forums which was promptly deleted. I then got a PM saying if I was having trouble to open a ticket. I wish he would have just told me that there were known issues. I then attempted to submit a ticket via email (which is how I've submitted low-priority issues in the past) and got a reject message of some sort. I blew it off, thinking you'd get it all fixed up eventually. Luckily for me I don't run this site to make money. I've had to endure complaints from my friends though, but I can handle that. Link to comment Share on other sites More sharing options...
Head Guru Posted March 19, 2007 Author Share Posted March 19, 2007 Link to comment Share on other sites More sharing options...
stevevan Posted March 19, 2007 Share Posted March 19, 2007 Thanks for the updates Bill! Link to comment Share on other sites More sharing options...
Head Guru Posted March 20, 2007 Author Share Posted March 20, 2007 We have been running our core servers on the new providers fiber for two days now. Anyone noticing any improvments? Looking forward to hearing any feedback. Link to comment Share on other sites More sharing options...
Rednarb Posted March 20, 2007 Share Posted March 20, 2007 We have been running our core servers on the new providers fiber for two days now. Anyone noticing any improvments? Looking forward to hearing any feedback. Very good performance once again! Link to comment Share on other sites More sharing options...
Mish Posted March 22, 2007 Share Posted March 22, 2007 Could this be why my SMF forum is down (was working last night, now I can't even access the cpanel)? As far as I know it's on Naboo as well, though the rest of my domains are working jus fine. Like Rednarb, I don't want to submit a ticket if there's a perfectly logical explanation. Link to comment Share on other sites More sharing options...
TCH-Thomas Posted March 22, 2007 Share Posted March 22, 2007 I think Naboo have been fully migrated so please submit a ticket and the techs will look in to it. Link to comment Share on other sites More sharing options...
Head Guru Posted March 23, 2007 Author Share Posted March 23, 2007 We are lighting up a gigE to Level(3) at the moment. Currently some outbound routes are routing over Level(3) and we should soon have inbound routes in our router BGP. :) GO TCH! Link to comment Share on other sites More sharing options...
deejay3615 Posted March 25, 2007 Share Posted March 25, 2007 It is both ZIPPY and ZOOMY ! Link to comment Share on other sites More sharing options...
The Mighty Billy Posted March 25, 2007 Share Posted March 25, 2007 I spoke to someone last night about my website ( http://planetlive.net ) experiencing downtime of some sort. It's been unaccessible for almost a full day now. He told me it would be back up within a couple of hours. I still can't access the site. Is that being caused by this update? He claimed it was. Link to comment Share on other sites More sharing options...
TCH-Bruce Posted March 25, 2007 Share Posted March 25, 2007 Welcome to the forums Billy. I can see your site just fine. Try clearing your browsers cache. Link to comment Share on other sites More sharing options...
heyguy Posted March 25, 2007 Share Posted March 25, 2007 They changed all the ips on the servers last night, so there would have been a little downtime during propagation. The site is loading fine for me now. Try clearing your cache using ipconfig /flushdns from the command line. Link to comment Share on other sites More sharing options...
The Mighty Billy Posted March 25, 2007 Share Posted March 25, 2007 Welcome to the forums Billy. I can see your site just fine. Try clearing your browsers cache. Yikes! You're right. I've been refreshing all morning for nothing. I opened it in Firefox and it showed right up! Thank you. Link to comment Share on other sites More sharing options...
TCH-Andy Posted March 25, 2007 Share Posted March 25, 2007 Welcome to the forums Billy Link to comment Share on other sites More sharing options...
The Mighty Billy Posted March 25, 2007 Share Posted March 25, 2007 Thanks, Andy. Link to comment Share on other sites More sharing options...
TCH-Thomas Posted March 26, 2007 Share Posted March 26, 2007 Welcome to the forum, Billy. I am glad it is working for you now. Link to comment Share on other sites More sharing options...
TCH-Don Posted March 26, 2007 Share Posted March 26, 2007 Welcome to the forum, Billy Link to comment Share on other sites More sharing options...
clayartisan Posted March 27, 2007 Share Posted March 27, 2007 I just wanted to say a VERY BIG THANK YOU, to TCH and staff for these improvements. When I was rebuilding my store during February, I was experiencing frustrating and frequent time-outs, where I could not connect at all to my website. By running a route tracing program, I could see that a bottleneck was occuring at certain points, close to the TCH arrival. Since these improvements, the time outs have all but disappeared and the route tracing supports this. The improvements are definitely appreciated by this user! Patti Stoll http://www.pattistoll.com Link to comment Share on other sites More sharing options...
Recommended Posts