Jump to content


Photo

Outage


  • Please log in to reply
23 replies to this topic

#1 TCH-Dick

TCH-Dick

    General Manager

  • Admins
  • PipPipPipPip
  • 5,889 posts

Posted 03 October 2014 - 02:30 PM

We are currently investigating a power issue. This issue is affecting several servers and part of our network. Our data center techs are onsite and reviewing the issue now.

 

Thanks for your patience and we will provide updates shortly.

 

 


Dick DeVance
General Manager
TotalChoice Hosting, Inc
dick@totalchoicehosting.com


Posted Image


#2 kweilbacher

kweilbacher

    New To The Neighborhood

  • Members
  • Pip
  • 24 posts

Posted 03 October 2014 - 04:29 PM

any update?


-- kw
"The days pass by so quickly now, the nights are seldom long"

#3 LeeGoldsmith

LeeGoldsmith

    Distant Family

  • Members
  • PipPipPip
  • 187 posts

Posted 03 October 2014 - 04:33 PM

What's up with the unni server, I don't even see it in the server status list??

 

Thanks

Lee


Lee Goldsmith
Lee's Fishing Page
Acton, ME 04001

#4 TCH-Alex

TCH-Alex

    Technical Support

  • Members
  • PipPipPipPip
  • 770 posts

Posted 03 October 2014 - 05:06 PM

We are sorry for the delay in updating this thread, but all the staff were busy on working on the servers.

 

We are now restored 95% of the servers and working hardly on the remaining server as of now.


Alex Spaford
Technical Support
TotalChoice Hosting, Inc.
Total Choice Hosting - Helpdesk


#5 kweilbacher

kweilbacher

    New To The Neighborhood

  • Members
  • Pip
  • 24 posts

Posted 03 October 2014 - 05:34 PM

Alex, why is coruscant not showing up in the TCH real-time status list page???


-- kw
"The days pass by so quickly now, the nights are seldom long"

#6 gmml

gmml
  • Members
  • 2 posts

Posted 03 October 2014 - 06:26 PM

I guess I'm in the 5% then? 

 

Any update when everything will be back up? 


Edited by gmml, 03 October 2014 - 06:26 PM.


#7 TCH-Alex

TCH-Alex

    Technical Support

  • Members
  • PipPipPipPip
  • 770 posts

Posted 03 October 2014 - 06:44 PM

We have done 98% of the servers as of now. Just a few more servers remaining and the entire team is working on it.


Alex Spaford
Technical Support
TotalChoice Hosting, Inc.
Total Choice Hosting - Helpdesk


#8 Sub_John

Sub_John

    New To The Neighborhood

  • Members
  • Pip
  • 7 posts

Posted 03 October 2014 - 06:49 PM

Fingers crossed, I must be on that last one :sick:



#9 rauhs

rauhs
  • Members
  • 1 posts

Posted 03 October 2014 - 07:23 PM

UGH, me too!



#10 Jhaacker

Jhaacker
  • Members
  • 1 posts

Posted 03 October 2014 - 08:00 PM

Please provide an update. Our site is on boblo. It has been down for over 6 hours.

#11 Sub_John

Sub_John

    New To The Neighborhood

  • Members
  • Pip
  • 7 posts

Posted 03 October 2014 - 08:07 PM

I'm getting worried and missed some deadlines. Hope the backup was there :sick:



#12 sabathedog

sabathedog
  • Members
  • 4 posts

Posted 03 October 2014 - 08:20 PM

ANy chance of a status? It's been 1 1/2 hours since 98% and still down



#13 gmml

gmml
  • Members
  • 2 posts

Posted 03 October 2014 - 09:08 PM

Approaching 8 hours down now.  4 hours since 95% and 2.5 since 98%.


Edited by gmml, 03 October 2014 - 09:08 PM.


#14 StuartBridge

StuartBridge
  • Members
  • 1 posts

Posted 03 October 2014 - 09:53 PM

My sever is up but my website is missing for my reseller site http://www.npmahome.com/ my other sites are working off the reseller but not the main doamin. when will these be fixed?


Edited by StuartBridge, 03 October 2014 - 09:53 PM.


#15 TCH-Alex

TCH-Alex

    Technical Support

  • Members
  • PipPipPipPip
  • 770 posts

Posted 03 October 2014 - 10:39 PM

We are sorry for the inconvenience. But we are still working on the last set of servers, that is not up yet. We understand the downtime is painful, but kindly allow us some time to work on the remaining servers.

 

We appreciate your patience and cooperation regarding this.


Alex Spaford
Technical Support
TotalChoice Hosting, Inc.
Total Choice Hosting - Helpdesk


#16 Head Guru

Head Guru

    Bill Kish Head Guru

  • Admins
  • PipPipPipPip
  • 6,878 posts

Posted 04 October 2014 - 04:45 AM

Hello,

 

All services have been restored a few hours back.  We have one pending issue which is an emergency restoration of the server unni.

 

All other services, shared, reseller, dedicated, vps and colocation have all been restored.

 

I will be releasing a full disclosure once all the facts of this incident are compiled.

 

As Alex stated, we are all very sorry for this issue and we will continue to strive to do our best to handle any issues that arrise.

 

Thank you for your business.


Bill Kish

Head Cook and Bottle Washer

If you need help with your account or have any questions, please feel free to contact me using any of the contact methods below.  I can be reached 24 hours a day seven days per week.

Office :: 800-930-0485 x211
Mobile :: 248-632-3243

email: bill(at)totalchoicehosting.com

Instant Messenger -
AOL Instant Messenger: tchgurubill
Yahoo Messenger : tchgurubill
MSN Messenger : tchgurubill@hotmail.com

Thank you for your support and continued business


#17 jbsquires

jbsquires
  • Members
  • 1 posts

Posted 04 October 2014 - 02:49 PM

I appreciate the dedication you guys have, I know when things go south it's a battle to get them turned around. 



#18 Head Guru

Head Guru

    Bill Kish Head Guru

  • Admins
  • PipPipPipPip
  • 6,878 posts

Posted 04 October 2014 - 03:49 PM

I appreciate the dedication you guys have, I know when things go south it's a battle to get them turned around. 

 

You are the reason I love this job so much.  Thank you for your kind words and more importantly thank you for your support and business.


Bill Kish

Head Cook and Bottle Washer

If you need help with your account or have any questions, please feel free to contact me using any of the contact methods below.  I can be reached 24 hours a day seven days per week.

Office :: 800-930-0485 x211
Mobile :: 248-632-3243

email: bill(at)totalchoicehosting.com

Instant Messenger -
AOL Instant Messenger: tchgurubill
Yahoo Messenger : tchgurubill
MSN Messenger : tchgurubill@hotmail.com

Thank you for your support and continued business


#19 Blackcat

Blackcat

    Distant Family

  • Members
  • PipPipPip
  • 126 posts

Posted 05 October 2014 - 03:31 PM

Years and years proud customer from oversea :) Thank you guys for all the hard work. Simply the best :) 



#20 kweilbacher

kweilbacher

    New To The Neighborhood

  • Members
  • Pip
  • 24 posts

Posted 06 October 2014 - 07:39 AM

When can we expect some type of post-mortem report? I have customers that I need to provide a response to this outage. Thanks.


-- kw
"The days pass by so quickly now, the nights are seldom long"

#21 Head Guru

Head Guru

    Bill Kish Head Guru

  • Admins
  • PipPipPipPip
  • 6,878 posts

Posted 06 October 2014 - 04:44 PM

Years and years proud customer from oversea :) Thank you guys for all the hard work. Simply the best :)

Thank you so much for your support, it means the world to us.


Bill Kish

Head Cook and Bottle Washer

If you need help with your account or have any questions, please feel free to contact me using any of the contact methods below.  I can be reached 24 hours a day seven days per week.

Office :: 800-930-0485 x211
Mobile :: 248-632-3243

email: bill(at)totalchoicehosting.com

Instant Messenger -
AOL Instant Messenger: tchgurubill
Yahoo Messenger : tchgurubill
MSN Messenger : tchgurubill@hotmail.com

Thank you for your support and continued business


#22 Head Guru

Head Guru

    Bill Kish Head Guru

  • Admins
  • PipPipPipPip
  • 6,878 posts

Posted 06 October 2014 - 04:46 PM

When can we expect some type of post-mortem report? I have customers that I need to provide a response to this outage. Thanks.

 

We are waiting for a report from the UPS manufacturer on what went wrong with the UPS unit.  I have some prelimary data, but until I am confident I am holding off.


Bill Kish

Head Cook and Bottle Washer

If you need help with your account or have any questions, please feel free to contact me using any of the contact methods below.  I can be reached 24 hours a day seven days per week.

Office :: 800-930-0485 x211
Mobile :: 248-632-3243

email: bill(at)totalchoicehosting.com

Instant Messenger -
AOL Instant Messenger: tchgurubill
Yahoo Messenger : tchgurubill
MSN Messenger : tchgurubill@hotmail.com

Thank you for your support and continued business


#23 Head Guru

Head Guru

    Bill Kish Head Guru

  • Admins
  • PipPipPipPip
  • 6,878 posts

Posted 14 October 2014 - 07:51 AM

Update concerning the outage that occurred ::

 

The incident was due to a tripped 250A circuit breaker in the wrap around bypass cabinet on the output side of our UPS-1 system. Our three other UPS Units, UPS-2, UPS-3, and UPS-4 were completely unaffected. This issue caused a power disruption to circuits fed from UPS-1 only.

 

The UPS Vendor has investigated the cause of the breaker tripping and has not identified any faulty equipment downstream of the affected breaker at this time. The breaker was replaced as of October 10th, 2014 and we continue to monitor the situation.
 

Our vendor has performed testing of UPS-1 and verified it is producing proper voltages and waveforms and do believe the problem with this system was directly sourced to the breaker.

 

We are still awaiting a final invoice and report from our UPS vendor, and once this is in my hands I will post it directly to the forums.

 

Thank you


Bill Kish

Head Cook and Bottle Washer

If you need help with your account or have any questions, please feel free to contact me using any of the contact methods below.  I can be reached 24 hours a day seven days per week.

Office :: 800-930-0485 x211
Mobile :: 248-632-3243

email: bill(at)totalchoicehosting.com

Instant Messenger -
AOL Instant Messenger: tchgurubill
Yahoo Messenger : tchgurubill
MSN Messenger : tchgurubill@hotmail.com

Thank you for your support and continued business


#24 Head Guru

Head Guru

    Bill Kish Head Guru

  • Admins
  • PipPipPipPip
  • 6,878 posts

Posted 22 October 2014 - 04:20 AM

Here is our official report concerning the outage that occurred on UPS1.
 

On Friday, October 3, one of four UPS systems in our DC1 facility experienced a fault causing it to drop
customer load. We have concluded our analysis of this incident and will be presenting the results here:

At approximately 2:43pm on Friday, October 3, there was a brief utility power interruption. This caused all UPS
systems to operate on battery power for a brief period while the generator came online. All systems operated
properly and customer load was supported by generator power for a period of approximately 30 minutes.
On re-transfer to utility power, there was an unusually “hard transfer”. This means that the automatic transfer
switch (ATS), which normally attempts to transfer when the sine waves of the utility and generator power are at
approximately the same levels, transferred with a phase misalignment resulting in a more disruptive transfer
event than normal. This is not normally an issue for customers since all critical customer loads are UPS
protected and UPS systems will filter this as they would any other power anomaly.
In this past event, UPS-1 did not filter the transfer event in the usual fashion. UPS-1 instead triggered an
automatic internal bypass of the inverter. The unit will do this in the case of downstream overload conditions as
well as in the case of internal equipment faults.
 

There was an event in the fall of 2013 during which one of the
two internal circuit breaker motor-operators that comprise the load transfer system failed. That unit was
replaced last year and the new breakers (both units were replaced as a precautionary measure last year) operated
properly in this recent event. The combination of the hard transfer and the automatic bypass triggering resulted
in one of the isolation breakers in the wraparound bypass cabinet tripping. This caused customer load to be
dropped from this UPS. The UPS was manually bypassed to restore customer load while emergency service was
being arranged.

Subsequent tests of the UPS inverted showed proper waveform and voltage levels. No internal problems were
found with the inverter or bypass system. We believe the cause of the tripped circuit breaker was due to
transformer inrush (magnetizing) current due to the combination of the automatic bypass event and the hard
transfer of the ATS. Upon the successful completion of testing, and the corrective actions taken (described
below), customer load was retransferred to UPS the Saturday immediately following the UPS bypass event.
We believe the bypass event was caused by the UPS triggering the automatic bypass system at the time of the
hard transfer due to the action of a TVSS system (which is essentially a very large surge protector). There are
multiple TVSS systems in the building to protect against power transients such as those caused by switching
large, inductive loads such as motors and transformers.

We have taken the following corrective actions to ensure this problem does not recur:
1- We have installed a new TVSS on the 120/208 side of UPS-1 distribution transformer #2.
2- We have inspected the trip levels for all critical-bus circuit breakers to ensure the instantaneous trip
levels are set properly.
3- We have, as a precautionary measure, restrung one of two battery strings on UPS-1 that was
approaching end of life.
4- We have checked all other TVSS systems in the building for proper operation.

UPS-1 recently, in the fall of 2013, underwent a full scheduled maintenance which included the replacement of
all AC and DC bus capacitors as well as several fans that were not performing satisfactorily. This work was
performed simultaneously with the replacement of the faulty circuit breaker motor operator mentioned
previously. There are no wear items within the unit that are near end of life at this time so the UPS is not at
increased risk of failure.

We do not anticipate any further issues with our UPS-1 system.


Bill Kish

Head Cook and Bottle Washer

If you need help with your account or have any questions, please feel free to contact me using any of the contact methods below.  I can be reached 24 hours a day seven days per week.

Office :: 800-930-0485 x211
Mobile :: 248-632-3243

email: bill(at)totalchoicehosting.com

Instant Messenger -
AOL Instant Messenger: tchgurubill
Yahoo Messenger : tchgurubill
MSN Messenger : tchgurubill@hotmail.com

Thank you for your support and continued business





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users