Jump to content

Recommended Posts

We are currently investigating a power issue. This issue is affecting several servers and part of our network. Our data center techs are onsite and reviewing the issue now.

 

Thanks for your patience and we will provide updates shortly.

 

 

Link to comment
Share on other sites

What's up with the unni server, I don't even see it in the server status list??

 

Thanks

Lee

Link to comment
Share on other sites

We are sorry for the inconvenience. But we are still working on the last set of servers, that is not up yet. We understand the downtime is painful, but kindly allow us some time to work on the remaining servers.

 

We appreciate your patience and cooperation regarding this.

Link to comment
Share on other sites

Hello,

 

All services have been restored a few hours back. We have one pending issue which is an emergency restoration of the server unni.

 

All other services, shared, reseller, dedicated, vps and colocation have all been restored.

 

I will be releasing a full disclosure once all the facts of this incident are compiled.

 

As Alex stated, we are all very sorry for this issue and we will continue to strive to do our best to handle any issues that arrise.

 

Thank you for your business.

  • Like 1
Link to comment
Share on other sites

I appreciate the dedication you guys have, I know when things go south it's a battle to get them turned around.

 

You are the reason I love this job so much. Thank you for your kind words and more importantly thank you for your support and business.

Link to comment
Share on other sites

When can we expect some type of post-mortem report? I have customers that I need to provide a response to this outage. Thanks.

 

We are waiting for a report from the UPS manufacturer on what went wrong with the UPS unit. I have some prelimary data, but until I am confident I am holding off.

Link to comment
Share on other sites

Update concerning the outage that occurred ::

 

The incident was due to a tripped 250A circuit breaker in the wrap around bypass cabinet on the output side of our UPS-1 system. Our three other UPS Units, UPS-2, UPS-3, and UPS-4 were completely unaffected. This issue caused a power disruption to circuits fed from UPS-1 only.

 

The UPS Vendor has investigated the cause of the breaker tripping and has not identified any faulty equipment downstream of the affected breaker at this time. The breaker was replaced as of October 10th, 2014 and we continue to monitor the situation.

Our vendor has performed testing of UPS-1 and verified it is producing proper voltages and waveforms and do believe the problem with this system was directly sourced to the breaker.

 

We are still awaiting a final invoice and report from our UPS vendor, and once this is in my hands I will post it directly to the forums.

 

Thank you

Link to comment
Share on other sites

Here is our official report concerning the outage that occurred on UPS1.

On Friday, October 3, one of four UPS systems in our DC1 facility experienced a fault causing it to drop
customer load. We have concluded our analysis of this incident and will be presenting the results here:

At approximately 2:43pm on Friday, October 3, there was a brief utility power interruption. This caused all UPS
systems to operate on battery power for a brief period while the generator came online. All systems operated
properly and customer load was supported by generator power for a period of approximately 30 minutes.
On re-transfer to utility power, there was an unusually “hard transfer”. This means that the automatic transfer
switch (ATS), which normally attempts to transfer when the sine waves of the utility and generator power are at
approximately the same levels, transferred with a phase misalignment resulting in a more disruptive transfer
event than normal. This is not normally an issue for customers since all critical customer loads are UPS
protected and UPS systems will filter this as they would any other power anomaly.
In this past event, UPS-1 did not filter the transfer event in the usual fashion. UPS-1 instead triggered an
automatic internal bypass of the inverter. The unit will do this in the case of downstream overload conditions as
well as in the case of internal equipment faults.

There was an event in the fall of 2013 during which one of the
two internal circuit breaker motor-operators that comprise the load transfer system failed. That unit was
replaced last year and the new breakers (both units were replaced as a precautionary measure last year) operated
properly in this recent event. The combination of the hard transfer and the automatic bypass triggering resulted
in one of the isolation breakers in the wraparound bypass cabinet tripping. This caused customer load to be
dropped from this UPS. The UPS was manually bypassed to restore customer load while emergency service was
being arranged.

Subsequent tests of the UPS inverted showed proper waveform and voltage levels. No internal problems were
found with the inverter or bypass system. We believe the cause of the tripped circuit breaker was due to
transformer inrush (magnetizing) current due to the combination of the automatic bypass event and the hard
transfer of the ATS. Upon the successful completion of testing, and the corrective actions taken (described
below), customer load was retransferred to UPS the Saturday immediately following the UPS bypass event.
We believe the bypass event was caused by the UPS triggering the automatic bypass system at the time of the
hard transfer due to the action of a TVSS system (which is essentially a very large surge protector). There are
multiple TVSS systems in the building to protect against power transients such as those caused by switching
large, inductive loads such as motors and transformers.

We have taken the following corrective actions to ensure this problem does not recur:
1- We have installed a new TVSS on the 120/208 side of UPS-1 distribution transformer #2.
2- We have inspected the trip levels for all critical-bus circuit breakers to ensure the instantaneous trip
levels are set properly.
3- We have, as a precautionary measure, restrung one of two battery strings on UPS-1 that was
approaching end of life.
4- We have checked all other TVSS systems in the building for proper operation.

UPS-1 recently, in the fall of 2013, underwent a full scheduled maintenance which included the replacement of
all AC and DC bus capacitors as well as several fans that were not performing satisfactorily. This work was
performed simultaneously with the replacement of the faulty circuit breaker motor operator mentioned
previously. There are no wear items within the unit that are near end of life at this time so the UPS is not at
increased risk of failure.

We do not anticipate any further issues with our UPS-1 system.

Link to comment
Share on other sites

×
×
  • Create New...