Head Guru Posted December 3, 2013 Posted December 3, 2013 TotalChoice Hosting Maintenance Notice :: Subject : Network Maintenance and Upgrades Dates : 12/5/2013, 12/6/2013 & 12/7/2013 Times : 2200 - 0400 EST Creator : Bill Kish Description :: On the above dates and times TotalChoice Hosting, Inc. will be conducting service affecting network maintenance in our Troy, MI Data Center. This network maintenance is scheduled to commence at 11:00 pm EST on each day listed above. TotalChoice customers can expect periods of network instability and network downtime during the maintenance window. This upgrade will increase the capacity, security and overall reliability of our network.
Head Guru Posted December 3, 2013 Author Posted December 3, 2013 Hello TCH Family! I wanted to start a discussion with the TCH family about what we are doing during this maintenance window. The first two days are planned for Distribution Switch Upgrades/Replacements. Many people have asked me why do you need to replace and or upgrade working switches. The answer is two fold. Firstly the in production switches have reached their end of life cycle. This means simply that they have given us a nice long run of uptime but its now time to replace them before we are faced with a 3am switch failure and everyone is in a total panic. Secondly, we are adding in another layer of redundancy to the network. Each of our new switches feature redundant uplinks / handoffs from the assigned aggregation switches. This means that we can take feeds from two network ports and feed the switch. This allows us to continue moving data even if we loose a line card, port or a cat6 run fails. During the first two days we will be dealing solely with Distribution switch replacements. During the maintenance window clients with servers attached to the switches we are working on will see periods of network instability and short period of downtime. These first two days will not effect the entire network, the only effected servers will be servers on the switches we are replacing. On Saturday we will phase into new Core Routers. This has been in planning phases for over 6 months and is a very large step for TCH. I am very excited about this upgrade. I can not go into full details as we would never want to talk publicly about our infrastructure but this upgrade will give us more capacity, stability, security and reliability. We will also be releasing a few new security policies on our network that should only effect people abusing the network. However with any new security polices we can expect to see a few bumps. I would be more than happy to answer any questions here on this thread about the upgrade. I will be personally overseeing this entire operation and will do my very best to make this a smooth and clean transition.
shoes Posted December 3, 2013 Posted December 3, 2013 (edited) As always TCH is a top notch player in the hosting business. I love it. Thank you for taking the time to tell me about the network updates Bill. Your company is the greatest. Edited December 3, 2013 by shoes
shoes Posted December 3, 2013 Posted December 3, 2013 Hey Bill what do you do with all the old switches. I am in need a a couple switches for my home network. I have 2 24 port 10meg switches that I would like to replace with gigabit speed switches.
Head Guru Posted December 3, 2013 Author Posted December 3, 2013 Shoes, Send me an email with your shipping information and I will send you out two used TCH switches at no cost. These should do a good job for you. Thank you for your business.
Guest John G. Posted December 4, 2013 Posted December 4, 2013 Bill, Thanks for information, We are sending information to our users. John G.
Head Guru Posted December 4, 2013 Author Posted December 4, 2013 Hi John, You are welcome and thank you for your continued support. You have been a loyal customer for a very long time and we are thankful to have you here.
Guest Carl S Posted December 4, 2013 Posted December 4, 2013 Hi Bill We are using email a great deal so during the maintaince downtime will we expect email to bounce or will they be queued for when the network is back up? Cheers Carl S
Head Guru Posted December 4, 2013 Author Posted December 4, 2013 Hey Carl, Email is a wonderful thing until it stops working. If someone send you an email and your server is offline the sending server will typically queue that failed sent email and attempt to send it again several times over a few day period. So you should get your emails when the network is back online. We do not expect a long period of downtime so I would not be to concerned about it. Plus this is why we have scheduled the windows of maintenance to be late in the evening to keep things as calm as possible. I hope this answers your question.
Head Guru Posted December 4, 2013 Author Posted December 4, 2013 Just wanted to give everyone a quick update. In preparation for Saturdays Maintenance Window, tonight we successfully lit some of our new core routing gear. It will be in standby mode until Saturday! It is a truly great day to be a TCH Guru!
Guest Dan Hauser Posted December 4, 2013 Posted December 4, 2013 Have you considered that you host e-commerce sites and this is the busiest time of the year for on-line shopping. If you must do maintenance 12/5, 6 and 7, please push it back until after midnight Pacific time.
Head Guru Posted December 4, 2013 Author Posted December 4, 2013 Hello, We run our operations based on Eastern Standard Time. So therefore our maintenance windows are always based on EST. In regards to hosting e-commerce sites. I am aware that we host tens of thousands of web sites and they range from large e-commerce sites down to local churches and schools. I do understand that any maintenance window is undesirable, however we can not change that this sort of maintenance is something that is required to maintain a reliable infrastructure and network. We will do our best to limit the impact on our clients as always.
Head Guru Posted December 4, 2013 Author Posted December 4, 2013 I have gotten a few call's from our colocation clients and Data Center 2 clients. This maintenance window will not effect our DC2 or Colo clients at all.
Head Guru Posted December 5, 2013 Author Posted December 5, 2013 We actually do have two data centers. However, DC 2 is not going to be effected by this maintenance window. If you would like more information about our new Data Center we will be posting some photos and updates next week concerning the new DC. Thank you for your question.
Head Guru Posted December 5, 2013 Author Posted December 5, 2013 We will be starting at 11pm tonight sharp.
Head Guru Posted December 6, 2013 Author Posted December 6, 2013 We are starting the first switch replacements. I will update when this first round is done.
Head Guru Posted December 6, 2013 Author Posted December 6, 2013 the first bit of downtime is starting now for a few servers.
Head Guru Posted December 6, 2013 Author Posted December 6, 2013 First set of servers that were effected are coming back online.
Head Guru Posted December 6, 2013 Author Posted December 6, 2013 Everything from the first round is back online and responding to services. The servers only lost network connection and were not rebooted, just a fyi for everyone.
Head Guru Posted December 6, 2013 Author Posted December 6, 2013 We are moving on to the next round. A few more servers will begin downtime now.
Head Guru Posted December 6, 2013 Author Posted December 6, 2013 The last round of servers are all back online and responding to services.
TCH-Bala Posted December 6, 2013 Posted December 6, 2013 Maintenance scheduled for tonight is complete. Thank you for your understanding and patience.
Head Guru Posted December 6, 2013 Author Posted December 6, 2013 We have completed our first round of network upgrades. Everything went without a hitch and as planned. We will be starting again tomorrow night at 11pm EST. I am headed home and Dick's feet are killing him.
Head Guru Posted December 6, 2013 Author Posted December 6, 2013 Hello TCH Family, We are on schedule to start our Network Maintenance at 11:00PM sharp. We will update further as we get closer to starting later tonight.
Head Guru Posted December 7, 2013 Author Posted December 7, 2013 We will be starting our maintenance window in the next 10 minutes. I will post updates here as we progress with things.
Head Guru Posted December 7, 2013 Author Posted December 7, 2013 We are starting with the maintenance.
Head Guru Posted December 7, 2013 Author Posted December 7, 2013 the first bit of downtime is starting now for a few servers.
Head Guru Posted December 7, 2013 Author Posted December 7, 2013 Servers that were effected are now starting to come back online.
Head Guru Posted December 7, 2013 Author Posted December 7, 2013 We have completed the switch swaps. Everything is progressing very smoothly. There will be one more task that we need to complete. This will effect a very limited number of servers. I will post when we are starting this last item on our list for tonight.
Head Guru Posted December 7, 2013 Author Posted December 7, 2013 We are starting the final switch swap. A few servers will be affected and I will post when we are done.
Head Guru Posted December 7, 2013 Author Posted December 7, 2013 We have completed our maintenance for tonight. All servers are back online and responding to services. We have one remaining night of maintenance, which will begin tomorrow night at 11pm. Thank you very much for your support. It is really awesome to be a TCH Guru!
Head Guru Posted December 8, 2013 Author Posted December 8, 2013 We are starting our final network maintenance window. Our network will go totally dark during this time. We will post updates as we move along.
Head Guru Posted December 8, 2013 Author Posted December 8, 2013 We are making excellent progress, expect things to be coming back up soon.
TCH-Bala Posted December 8, 2013 Posted December 8, 2013 99% of the servers are up, we are working as fast as we can to sort out the few servers that are not up on the network.
TCH-Bala Posted December 8, 2013 Posted December 8, 2013 Almost there, the last couple of servers remaining dick an bill are working to get them back into network.
TCH-Bala Posted December 8, 2013 Posted December 8, 2013 Maintenance scheduled for tonight is complete. We are still verifying the network to make sure everything is set
TCH-Bala Posted December 8, 2013 Posted December 8, 2013 If you are still facing issues, please drop in a ticket to our support department (link is on my signature) so that we can check it out. Thank you for your understanding and patience. This was a huge upgrade for TCH.
Recommended Posts