Jump to content
Zach

What Is Going On With Newyork?

Recommended Posts

What is up with this whole Server 319 -> NewYork transition?

 

Never before have I been the slightest bit nervous about the quality of service at TCH but I'm really starting to worry.

 

The transition broke my custom MX records. Nobody told me before hand this would happen and I know me and some customers lost email because of it. Numerous accounts I had closed a long time ago re-appeared in WHM. Whatever downtime occurred yesterday made several of my webpages inaccessible for most of the day for many of my clients and I had to spend the whole day answering phone calls about it. And now a clients' very active blog has lost all new activity since October 3rd.

 

Can someone please be honest and let us know what's going on, what didn't work right, and what you're doing about it? Maybe I'm overreacting, maybe this transition has been perfectly smooth for everyone else. I just know it seems like lots of things have been really flaky lately and I have some clients who will just go find hosting elsewhere if this keeps up.

 

 

Respectfully, (really, I hope I don't come across as rude--I'm just scared and frustrated!)

 

Zach

Share this post


Link to post
Share on other sites

Zach,

 

I can tell you heads are rolling right now at TCH.

 

This is 100% not acceptable and I am very upset about this.

 

We have migrated tens of thousands of domains into our new data center and have had very few issues. It seems when an issue occurs its ugly and I roll heads over it.

 

I have spoken to our Data Center Manager about this and expressed my concerns over this horrible act of customer service. He is working on a reply to you now and should be posted here shortly. I have also spoken to our top admin and he will be working with your personally on this issue, so please do expect contact from Carl Noonan.

 

You have been with us a long time and understand that this is not how we run our business. I am VERY sorry for this issue and promise you a full detailed explanation is being worked on by my team right now. These issues are hereby stomped on by my big foot.

 

Please allow me to correct this issue and you can go back to years of faithful, reliable service from TotalChoice.

Share this post


Link to post
Share on other sites

Zach, Let me first say that you did not come off as rude by any means and the concerns you have expressed are very valid. I will not beat around the bush on this matter, the bottom line is cut and dry simple - we screwed up, yup that what I said - we dropped the ball on this one. I wish I had more comforting words to offer at hand but I am not going to make excuses on this situation as you and the community expect - deserve - better. So, on behalf of TCH, I apologize for the inconveniences we have caused you and your business during the migration process to newyork and hope we can restore you're faith in the standards you hold TCH too and that we strive to maintain.

 

The newyork migration had a few elements to it that caused it to stand out from our previous migrations in that we parted ways from some successful habits and also implemented some new "features" that together caused numerous issues in the overall migration process. Further, there was also some run-of-the-mill human error involved that only compounded the situation, in addition to a long standing issue of not importing custom MX records on migrations that only added fuel to an already blazing fire.

 

One of the first and foremost issues had begun from the onset, before the migration was even underway. During the preparation stages for a migration we typically run "fresh" backups on the older system, in this case server319 and then run a script that purges the backups of old accounts that are no longer active on the system. This has over time been a routine that we have been mindful of but never really acted upon other than manually running said script when required and in this situation having forgotten to run it lead to a number of inactive/previous deleted accounts being restored onto newyork. In order to see that this issue is corrected going forward and does not reoccur as a matter of simply absented mindedness, we have added the script to run on a regular basis on servers (twice a month) in addition to it now being integrated with the manual backup command we use in the pre-migration process. Those two changes combined provide more consistent assurance over the state of the local backups we maintain along with those that get transfered for migrations to new servers.

 

Second to the above was a decision made to enable by default a feature called MySQL InnoDB support, this is a feature that in the past has caused us some issues, namely it has filled up the database partitions on servers with binary innodb files causing the SQL service to go offline till manual intervention by a system administrator. This decision was one that really separated us from previous migrations and it took a big bite out of us when it hit newyork causing the same age old issue of filling the space on the database partition. This issue was rather easy to correct in hindsight as we have done in the past, simply disable the option in our MySQL server configuration file and leave well enough alone. That however is not the only thing we did, we also went out and researched the issue further and uncovered a number of finer grained SQL configuration options that are part of MySQL 5.x server that allow us to control the binary data files created by innodb that did not exist in our previous SQL versions we used earlier in the year. The short of it is we will be in the future leveraging the new configuration options along with more discretion on changing configuration standards of new builds "on the fly", which was on our part just a really bad decision.

 

Finally and perhaps one of the more consistent issues we encounter through most all migrations is that we do not traditionally migrate custom MX records or zone file entries, this is something we have always in the past just bit the bullet on as it is a very cumbersome process to handle. In addition to that, we also like to start with clean DNS records on the new servers as it provides a more reliable foundation for syncing into our DNS cluster. I do admit though that this is one of those issues that we have long "accepted" the status quo of doing nothing about and really those days are long gone where we can sit idle to the inconvenience this causes those that host with TCH. With that said, I have personally undertaken the task of developing a special set of scripts that will during the migration process identify accounts with custom DNS entries and import those onto the new server.

 

I hope that this post addresses the concerns you have in addition to providing you some insight into the problems encountered with this particular migration and how we are working to see that they do not happen in the future. On behalf of everyone at TCH, thank you for hosting with us and we express our sincere apologies for the inconveniences you have encountered during this migration process.

Share this post


Link to post
Share on other sites

I greatly appreciate the honesty here and the explanations for what has gone on. Carl Noonan was very helpful in getting my issue w/ the missing blog entries taken care of and thankfully that issue seems to be resolved. At the moment there are no other outstanding problems I'm aware of.

 

Thanks again and yes, I hope its smooth sailing from this point on.

Share this post


Link to post
Share on other sites

Hi Zach,

 

Thank you for confirming everything is in order. Ryan's post above would have cleared all your doubts. If you have further questions, or need more assistance, please do not hesitate to ask. We are grateful for your understanding and patience.

Share this post


Link to post
Share on other sites

Bill--I've emailed you and PM'ed you and can't seem to get a response. I have another issue that I can't seem to get any real help on and I'd appreciate it if you'd contact me.

 

 

 

Thanks

 

Zach

Share this post


Link to post
Share on other sites

Hi Zac,

 

I am out of town till Thursday. I will have Dick call you to follow up.

 

Thanks

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...