Head Guru

Thank you for the business. All our plans include the PHP Select Feature.

I wanted to personally thank all the clients that have called me and or texted me personally with words of support. TCH has been in this business for over 15 years and these sort of attacks are the worst part of our job at TCH. I want to also thank our network admin team for the hard work they have been doing to mitigate this attack. As Bala said, we are working on getting things under control and doing the very best that we can. Thank you again for your support and continued business.

This issue has been resolved for the past 30 minutes and things are looking good. We will continue to monitor the situation and will post any updates here for everyone. Thank you for your understanding.

Hello Everyone, I am sorry to have to report such things, however this is the ugly part of this hosting business. We have been the target of a denial of service attack. We are working with our upstream providers to isolate and block the attack. Progess is being made and I will keep the forums updated as we move along. Network is up just slow at this moment.

At this time all effected servers are online and all services are reporting normal once again. Thank you for your understanding during this issue. If you have any comments or concerns, please let us know.

This issue has been corrected and all affected servers should start coming online now. I will post a follow up shortly.

We have isoloated the issue with this network segment and are working to correct the problem. Updates to come very shortly.

We are working on this issue as quickly as possible. I will post further updates as they become available.

Toe When I was growing up and even way past my teenager years I would never admit to being wrong. In fact I think my maturity came at a much later age than most. I can now admit when I am wrong even thought it still hurts. Thank you for allowing my company the oppurtunity to serve your webhosting needs. If you need anything my contact information is in my signature.

Like Bruce stated. Looking forward to talking to you.

All our dedicated servers are back in stock and ready to deploy. Thank you for your patience while we struggled with some growing pains.

We are working to get more rack room, the demand for our services has out paced our ability to keep up with rack space in our datacenter The new racks, network gear and servers hav all arrived. We have been busy getting it all ready and we expect the new servers to be ready by end of this week. Thanks!

Here is our official report concerning the outage that occurred on UPS1. On Friday, October 3, one of four UPS systems in our DC1 facility experienced a fault causing it to drop customer load. We have concluded our analysis of this incident and will be presenting the results here: At approximately 2:43pm on Friday, October 3, there was a brief utility power interruption. This caused all UPS systems to operate on battery power for a brief period while the generator came online. All systems operated properly and customer load was supported by generator power for a period of approximately 30 minutes. On re-transfer to utility power, there was an unusually “hard transfer”. This means that the automatic transfer switch (ATS), which normally attempts to transfer when the sine waves of the utility and generator power are at approximately the same levels, transferred with a phase misalignment resulting in a more disruptive transfer event than normal. This is not normally an issue for customers since all critical customer loads are UPS protected and UPS systems will filter this as they would any other power anomaly. In this past event, UPS-1 did not filter the transfer event in the usual fashion. UPS-1 instead triggered an automatic internal bypass of the inverter. The unit will do this in the case of downstream overload conditions as well as in the case of internal equipment faults. There was an event in the fall of 2013 during which one of the two internal circuit breaker motor-operators that comprise the load transfer system failed. That unit was replaced last year and the new breakers (both units were replaced as a precautionary measure last year) operated properly in this recent event. The combination of the hard transfer and the automatic bypass triggering resulted in one of the isolation breakers in the wraparound bypass cabinet tripping. This caused customer load to be dropped from this UPS. The UPS was manually bypassed to restore customer load while emergency service was being arranged. Subsequent tests of the UPS inverted showed proper waveform and voltage levels. No internal problems were found with the inverter or bypass system. We believe the cause of the tripped circuit breaker was due to transformer inrush (magnetizing) current due to the combination of the automatic bypass event and the hard transfer of the ATS. Upon the successful completion of testing, and the corrective actions taken (described below), customer load was retransferred to UPS the Saturday immediately following the UPS bypass event. We believe the bypass event was caused by the UPS triggering the automatic bypass system at the time of the hard transfer due to the action of a TVSS system (which is essentially a very large surge protector). There are multiple TVSS systems in the building to protect against power transients such as those caused by switching large, inductive loads such as motors and transformers. We have taken the following corrective actions to ensure this problem does not recur: 1- We have installed a new TVSS on the 120/208 side of UPS-1 distribution transformer #2. 2- We have inspected the trip levels for all critical-bus circuit breakers to ensure the instantaneous trip levels are set properly. 3- We have, as a precautionary measure, restrung one of two battery strings on UPS-1 that was approaching end of life. 4- We have checked all other TVSS systems in the building for proper operation. UPS-1 recently, in the fall of 2013, underwent a full scheduled maintenance which included the replacement of all AC and DC bus capacitors as well as several fans that were not performing satisfactorily. This work was performed simultaneously with the replacement of the faulty circuit breaker motor operator mentioned previously. There are no wear items within the unit that are near end of life at this time so the UPS is not at increased risk of failure. We do not anticipate any further issues with our UPS-1 system.

Update concerning the outage that occurred :: The incident was due to a tripped 250A circuit breaker in the wrap around bypass cabinet on the output side of our UPS-1 system. Our three other UPS Units, UPS-2, UPS-3, and UPS-4 were completely unaffected. This issue caused a power disruption to circuits fed from UPS-1 only. The UPS Vendor has investigated the cause of the breaker tripping and has not identified any faulty equipment downstream of the affected breaker at this time. The breaker was replaced as of October 10th, 2014 and we continue to monitor the situation. Our vendor has performed testing of UPS-1 and verified it is producing proper voltages and waveforms and do believe the problem with this system was directly sourced to the breaker. We are still awaiting a final invoice and report from our UPS vendor, and once this is in my hands I will post it directly to the forums. Thank you

We are waiting for a report from the UPS manufacturer on what went wrong with the UPS unit. I have some prelimary data, but until I am confident I am holding off.

Thank you so much for your support, it means the world to us.

You are the reason I love this job so much. Thank you for your kind words and more importantly thank you for your support and business.

Hello, All services have been restored a few hours back. We have one pending issue which is an emergency restoration of the server unni. All other services, shared, reseller, dedicated, vps and colocation have all been restored. I will be releasing a full disclosure once all the facts of this incident are compiled. As Alex stated, we are all very sorry for this issue and we will continue to strive to do our best to handle any issues that arrise. Thank you for your business.

Yes of course. That would be no problem at all. Thank you for considering TCH.

Hello TCH Family, The server and all its VPS units have been restored to fully operational status at this time. The RAID array that contains all the data for the VPS servers has gone into a very unusual degraded state and I am not going to pull any punches here with anyone, it is not good. I am not going to make any further attempts to rebuild this array and have decided to migrate all the VPS units to new hardware. This will take us a bit of time, the new hardware is already completed and ready to start taking on the VPS units. We will be conducting fresh backups of the current VPS units over to our backup cloud. Once this is complete, we will then start a restore process of each VPS onto the new server. Everyone will keep their IP allocations and there will be no loss of any data. I do respectfully request that any site updates be put on hold until we complete this migration, if you make site changes during the backup process there is a chance that those changes will not be copied over. There should be no downtime during this migration and we intend to start working on immediatley. The VPS units are also being migrated to our new (unreleased) VPS control panel so clients will have much more control over their VPS units. Thank you for your understanding and we all greatly appreicate your business and loyal support. As stated, all VPS units that were effected are now back online and we are starting migrations to new hardware.

It appears the issue with the outbound email has been fixed, we have gotten replies out to all our clients and things are looking good. If your still having issues, please do let us know and we will get them solved for you.

Hello, We have been tracking an issue with outbound email from our server farm for the past couple hours and we believe that we have tracked down the issue. It appears an cPanel update has made some changes to our custom email configurations and has interrupted the outbound email. Our server engineers have made changes to the servers that should correct this issue. Any email that you sent should be in the servers email queue and we are pushing out the queues from each server now. Please accept our sincere apologizes for this issue and know that our entire staff has been working with this issue since it first was presented to us. If you have a help desk ticket open, and we are currently flooded with tickets, we are doing our best to reply to each ticket. Please be patient with us as we work on the replies. We will post further updates to this issue and please keep tuned here on this forum thread for more updates. Also, please note this has not effected any inbound email.

Ian, It only effects people that have SPF settings, which is well under 1% of our user base. I replied to your help desk ticket with the information you inquired about.

Since this issue was corrected, we have seen no further issues and things are looking good. If you need any help, as always, please just let us know. Thank you for your business and support.