Head Guru

Ok guys, we are going to be using a canned reply to all our tech support tickets. Here it is: (just a fyi) Hello, Thank you for contacting us concerning your web site. Please head over to our forums for an up to the minute status on this issue. We have our entire staff working on this issue and rest assured that we are working non-stop to correct this issue. You can view the update here: http://www.totalchoicehosting.com/forums/index.php?showtopic=42941&pid=250121&st=0entry250121 Thank you for your support and understanding. The TotalChoice Hosting Gurus....

We are working as fast as possible to get sites restored from backups. I don't know what is going on just yet, but trust me when we know we will disclose everything to the family.

Dear TCH Family, At around 9:30AM EST today, we identified a website defacement attack effecting a large number of our customers. We are still investigating, but it appears the attack was targeted at index.php files. We are currently looking at this to determine the extent of the defacement. I will update you shortly. Thank you for your patience and understanding during this very serious matter.

I know guys.

Dear TCH Family, At around 9:30AM EST today, we identified a website defacement attack effecting a large number of our customers. We are still investigating, but it appears the attack was targeted at index.php files. We are currently looking at this to determine the extent of the defacement. I will update you shortly. Thank you for your patience and understanding during this very serious matter.

SERVICE IMPACTING maintenance on Saturday 5/26/2012 from 01:00 until 04:00 EST. We will be replacing a Distribution Switch in the Troy, MI Data Center facilities. We expect total downtime not to exceed 15 minutes but have opened a 3 hour window to prepare for the work. The following servers will be effected by this maintenance. quanton vortex zeltros cathar darthvader saijo nubia fresia elom felucia jandoon widek xerxes ophideran mandalor utapau kashyk ithor gryphon ukatis inysh almania If you have any questions, please do let us know. Thank you for your business and continued support.

Just a follow up note. To those clients that had hardware issues on their dedicated servers, we ended up giving both of them significant free upgrades in their hardware. Utility power was restored to our data center as promised. The faulty transfer switch has been replaced and we are back on Utility power as of around 12am last night. Thank you

Dear TCH Family, As some of you may be aware, today we suffered an outage at our Troy, Michigan Data Center. I wanted to take a few moments to explain what happened today and what actions we took to correct the issue. At approximately 1:35PM EST today we suffered a power outage from Detroit Edison. Typically when we lose power, all of our power is then bypassed over through a transfer switch to our UPS Battery Farm. Once the Emergency generator is started we then automatically transfer to the Genset for power. When the utility power failed our systems did attempt to transfer to UPS power. However, the Transfer Switch failed and we never ended up getting backup power. This meant all our equipment in the Data Center went dark with the exception of a few isolated racks that were on a secondary UPS and transfer switch. However, even the racks that did not lose power lost network connectivity due to the network gear being dark. The main power outage only lasted for approximately 45 minutes and at 2:10PM we were able to restore power to all of the Data Center. At 2:00PM we were able to bypass the transfer switch and move to Generator Power. Our core network was off line for a total of 35 minutes. Once power was fully restored our team went into action and started a row by row, rack by rack inspection of all our equipment. This included power cycling servers that did not properly come up on their own. We were faced with several switches that did not properly power up and those were also manually rebooted. All our routers and core networking came up without the need of any intervention. We ended up replacing several network components that were damaged during the power issue. Once we had all the servers powered up we then started looking at servers that were powered up but were not responding to network ping. We had approximately 60 servers that needed to be un-racked and placed on the work bench for manual inspection. We quickly narrowed the number of downed servers to about 20 and by 5pm we were down to just a handful of servers that still needed inspection. These servers needed power supply replacements and or manual File System Checks. We ended up with two shared servers that took some time to get back online and two dedicated servers that were also giving us some issues. At around 530pm we rang the all clear bell and everything in the data center was online and working as intended. The faulty transfer switch is being replaced and we will stay on Generator Power until then. Once it is ready we will only then switch back to utility power. Most of the TCH family only saw a short 45 minute outage, on the other hand other family members were down for a longer period of time. I want to personally thank everyone that reached out to me during this outage. Your kind words and support mean so much and it makes our job that much easier. I also want to reach out to the couple of very upset clients that called me during the outage and offer my sincere apologies. As most of the TCH family knows, we do not have many outages and when they do occur we work our tails off to get things back to normal. As always if you would like to discuss things please feel free to respond here or call me personally.

Rob needs to buy me lunch!

Hello, The cert has been renewed and you should be good to go. If you have any more issues, please do let us know. Thank you for your business!

After a very long period of talking, dealing and waiting I am proud to annouce that TCH has added Comcast to our mix of bandwidth. Comcast will allow TCH to better reach out to clients on local peering exchanges and also gives us another 1gb of network overhead. The link to Comcast went live today at 4pm. Enjoy!

Hello TCH Family, Just a brief notice about continued infrastructure upgrades here at TCH: We will be upgrading our Intrusion Detection System (IDS) hardware on Friday 1/27/2012. This hardware is dedicated to behind the scenes work that reviews all inbound traffic to our entire server farm and helps us sanitize bad or unwanted traffic to our servers. As our network traffic increases, so has the requirements of the IDS system, this is why we are upgrading the hardware. We do not anticipate any network downtime during this upgradation to our IDS hardware. This is simply a notice of our continued improvements to the services we provide to our clients. In addition, on 1/27/2012 we will be performing upgrades to our private network switching hardware. This will be a multi-step process and will take us several maintenance windows to complete. If your servers are relying on our private network, you should receive an email from us with the maintenance window for your servers. In this process we will be replacing each distribution switch with new units. These new switches will feature the same 1Gbps server to switch speed but will feature a 10Gbps switch to switch speed. We will also be upgrading our backup server’s network interfaces to 10gbps. There are many more improvements coming, but the above are the issues we have on the planning board for this week. Today is a great day to be a TCH GURU!

I have replied to your help desk ticket with the resolution that i think you were looking for. Please let me know if you have any questions or comments and I will be more than happy to assist.

Carbonize, I am personally reviewing the ticket and will respond to you via that ticket. I will get this fixed for you. You are a valued long term client and I thank you for your business,

Hello TCH Family, Over the past 6 months TCH has experienced some really amazing growth. In the last two months alone we have shattered month to month sales records! November 2011 marked our best all time month in the company’s history. We ended 2011 with our best December in over ten years. Even more amazing is the fact that we just broke our best 15 day period for new sales records. Thousands of new TCH family members have joined us in the past few months and we are very proud to have them in our family. We owe this all to our clients and staff. I wanted to personally thank everyone that has made TCH such an amazing company. With this explosion in growth we are making some changes to our internal infrastructure. All of these changes are happening behind the scenes and should be totally transparent to our clients. This morning around 6am EST, we brought on a new cluster of servers to handle our support desk. We have never had the need for such dedicated hardware to handle our support system, however with all the growth we started to notice a decline in the performance of our help desk and decided to take action. This new boost in speed also translates into allowing our desk to handle a larger volume of clients help requests per day. Over the past few days we have also brought on new cluster of DNS servers to handle our new clients Domain Name Resolution. This allows us to not overload our existing DNS Servers and most importantly allows us to continue to grow and not affect the performance of our current hosted domains. We are also bringing on new servers at an amazing rate. Just this past week we added several more servers to our growing server farm. Unlike other small hosts, we actually own and operate all our own hardware. We have our own IP allocations and maintain our very own Data Center Presence. We do not rent servers from other providers and resell other providers servers. We are truly independent and rely on only ourselves to reboot your servers or upgrade your servers ram. If our tech team sees a server that needs more ram, we just take action and do not have to sit around and wait for our vendor to send a technician to work on our rented server. We have much more to announce and will be doing so right here in our family forums. Many of you have been asking for a new web site with improved services and we are very close to doing so. Please stay tuned for more announcements and in the meantime enjoy the improved infrastructure here at TCH. Thank you again for all your support. It's a great day to be a TCH GURU!

I am always open for people to mail me computers.

http://www.webhostingtalk.com/showthread.php?t=390949&highlight=totalchoice LOL

Man the good ol' days.

Rob that is awesome! Ping me on IM one day, we have so much to catch up on! :) :thumbup1:

Chip, Glad to see that your happy with your new server. I know you want to order more servers, give me a call.

I hate when that happens.

I love when problems "go away".

Thank you for your business. Please remember to tell your friends about TCH!

Just for you we increased this limit to 7000 on your server. TCH.... rocks?

I agree Zac! Respect to Steve Jobs for sure