TCH-Bala Posted July 20, 2013 Share Posted July 20, 2013 We are experiencing downtime on our NOC, it is not just TCH but the entire facility which hosts our NOC. We are in the process of tracking this down We have all our team working on this. We will update you as soon as we get it from the Data Centre guys. Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 Looks like it is a major power fault at the building. They are working on it, am sorry to say there is no ETA yet. Link to comment Share on other sites More sharing options...
zethon Posted July 20, 2013 Share Posted July 20, 2013 Out of curiosity, what city and state is the NOC located? Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 Our facility is at Troy MI. At the moment we are waiting for electricians update Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 Got word that the building got hit by lightning. All power and fiber lines are down. Two portable generators are in route as is the utility company. The fiber to the building is down in multiple places due to bad storms, we have fiber crews out working it. Link to comment Share on other sites More sharing options...
digitex Posted July 20, 2013 Share Posted July 20, 2013 Glad you guys are working on this! I was planning on making some changes to our business website this morning but guess I can't until things are back up. Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 Fiber crew is on site. Link to comment Share on other sites More sharing options...
David Botts Posted July 20, 2013 Share Posted July 20, 2013 Post Pics Thanks for getting on this so fast. I noticed your own site is down as well, yikes. Link to comment Share on other sites More sharing options...
Tina Kendall Posted July 20, 2013 Share Posted July 20, 2013 Thanks for keeping us updated Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 We are coming up....................... Link to comment Share on other sites More sharing options...
Tina Kendall Posted July 20, 2013 Share Posted July 20, 2013 I'm up Link to comment Share on other sites More sharing options...
scotlandview Posted July 20, 2013 Share Posted July 20, 2013 Back Online! Thanks for the swift response and for keeping us well informed, much appreciated :-) Link to comment Share on other sites More sharing options...
gtman55 Posted July 20, 2013 Share Posted July 20, 2013 I'm on Jendar and my site is still down. Link to comment Share on other sites More sharing options...
rfrassetti Posted July 20, 2013 Share Posted July 20, 2013 I'm still down too... Link to comment Share on other sites More sharing options...
rfrassetti Posted July 20, 2013 Share Posted July 20, 2013 Wait change that, just came back up. Seems to be running slow though Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 We are going through the servers that are still down, some of which did not power up or on disk check. Link to comment Share on other sites More sharing options...
Tina Kendall Posted July 20, 2013 Share Posted July 20, 2013 Was up temperolrly .. now getting internal server error page  Link to comment Share on other sites More sharing options...
wdv Posted July 20, 2013 Share Posted July 20, 2013 Site came up quickly on Felucia, but some services (smtp, webmail, cpanel) not working yet. Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 endor.tchmachines.com :: network card failed, replacing Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 ossus is up. Link to comment Share on other sites More sharing options...
hatala Posted July 20, 2013 Share Posted July 20, 2013 sanantonio is back up and all seems well. Â Thank you for your quick response and continuous updates. Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 triton is up. Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 endor is up. Link to comment Share on other sites More sharing options...
Kevan Posted July 20, 2013 Share Posted July 20, 2013 Site on matra is still down.  Site on endor is working fine now  Thank You for keeping us informed during this busy time Link to comment Share on other sites More sharing options...
Gabi ~ ClaytonWorks Posted July 20, 2013 Share Posted July 20, 2013 coruscant is still down Thanks Gabi Link to comment Share on other sites More sharing options...
soned Posted July 20, 2013 Share Posted July 20, 2013 can edson be up soon as well Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 matra up Link to comment Share on other sites More sharing options...
Kevan Posted July 20, 2013 Share Posted July 20, 2013 matra is running but site is not working yet  get a 404, NOT FOUND Apache error when attempting to access the site. Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 matra taken down for inspection. Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 cathar is up Link to comment Share on other sites More sharing options...
nortk Posted July 20, 2013 Share Posted July 20, 2013 unni is giving "500 Internal Server Error". Link to comment Share on other sites More sharing options...
debdwright Posted July 20, 2013 Share Posted July 20, 2013 It appears that both naboo and skywalker mail servers still down. Â Thanks. Link to comment Share on other sites More sharing options...
debdwright Posted July 20, 2013 Share Posted July 20, 2013 Correction, the entire servers, as both of my sites are down as well. Â (Naboo and skywalker) Link to comment Share on other sites More sharing options...
Gabi ~ ClaytonWorks Posted July 20, 2013 Share Posted July 20, 2013 coruscant is up! Thanks, TCH! Link to comment Share on other sites More sharing options...
debdwright Posted July 20, 2013 Share Posted July 20, 2013 Matra must still be down for inspection? My other site is also 404 which resides on Matra. Link to comment Share on other sites More sharing options...
jnull Posted July 20, 2013 Share Posted July 20, 2013 My site is on Matra and just received this message from a site I manage there, a global membership site: Â "Got a message just now from a member that they sent a prospect to our website and they were not able to access. What is linked to our url now is WORSE than nothing. Please let me know you have received this message. Sorry to interrupt your Saturday. " Â Otherwise, someone else's site is now showing up rather than chenetwork.org ... Link to comment Share on other sites More sharing options...
TCH-Thomas Posted July 20, 2013 Share Posted July 20, 2013 Chenetwork.org is currently giving me a 404, so I guess the techs are working on it. 1 Link to comment Share on other sites More sharing options...
carbonize Posted July 20, 2013 Share Posted July 20, 2013 It's already been stated that Matra is being inspected. Pain in the rear since I was hoping to push out a new version of Lazarus :| Link to comment Share on other sites More sharing options...
Sicarius Posted July 20, 2013 Share Posted July 20, 2013 Any updates? I dont know which server we are on but I do know we are down. What is the current list of servers are up, down (inspection), down (other)? Link to comment Share on other sites More sharing options...
TCH-Mike Posted July 20, 2013 Share Posted July 20, 2013 We have our data center team and technicians working on this as we speak. This is our #1 priority and we are working diligently on restoring all services. We apologize for the inconvenience but assure you that we are doing what we can! Â Thanks for your patience and understanding. Link to comment Share on other sites More sharing options...
Head Guru Posted July 20, 2013 Share Posted July 20, 2013 Matra server is going thru a long File System Check. Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 Am afraid obiwan is going to need to be restored. Updates about that server will be posted HERE Link to comment Share on other sites More sharing options...
Kevan Posted July 20, 2013 Share Posted July 20, 2013 endor is now showing a 500 Internal error when attempting to access the site that has been working since 10am. Link to comment Share on other sites More sharing options...
Mdub702 Posted July 20, 2013 Share Posted July 20, 2013 Can anyone please explain how I would go about finding which server my site is on? Â obviously-marvelous.com Link to comment Share on other sites More sharing options...
Mdub702 Posted July 20, 2013 Share Posted July 20, 2013 Can anyone please explain how I would go about finding which server my site is on?  obviously-marvelous.com  Figured it out, Matra Link to comment Share on other sites More sharing options...
TCH-Bala Posted July 20, 2013 Author Share Posted July 20, 2013 Mdub702, I believe you are on matra. Link to comment Share on other sites More sharing options...
Tina Kendall Posted July 20, 2013 Share Posted July 20, 2013 Please could you tell me what server I'm on please.. Link to comment Share on other sites More sharing options...
TCH-Mike Posted July 20, 2013 Share Posted July 20, 2013 Tina what is your domain name? Link to comment Share on other sites More sharing options...
Tina Kendall Posted July 20, 2013 Share Posted July 20, 2013 Tina what is your domain name? Â http://agiftfromwithin.co.uk/ Â Thanks for all your hard work Link to comment Share on other sites More sharing options...
TCH-Mike Posted July 20, 2013 Share Posted July 20, 2013  http://agiftfromwithin.co.uk/  Thanks for all your hard work  You are on the server 'unni'. 1 Link to comment Share on other sites More sharing options...
Tina Kendall Posted July 20, 2013 Share Posted July 20, 2013 'unni' is back up. Thanks Link to comment Share on other sites More sharing options...
TCH-Mike Posted July 20, 2013 Share Posted July 20, 2013 We are making great progress on repairing all issues caused by the lightening strike. Thanks again for your patience. 1 Link to comment Share on other sites More sharing options...
carbonize Posted July 20, 2013 Share Posted July 20, 2013 Any danger of seeing Matra back up and working anytime soon? Link to comment Share on other sites More sharing options...
TCH-Mike Posted July 20, 2013 Share Posted July 20, 2013 Any danger of seeing Matra back up and working anytime soon? Â Matra is now back up and running Link to comment Share on other sites More sharing options...
j-f Posted July 20, 2013 Share Posted July 20, 2013 www.reefconcept.com we are on the ossus server and we are still down, the online store and forum still don't work since this morning Link to comment Share on other sites More sharing options...
carbonize Posted July 20, 2013 Share Posted July 20, 2013 (edited) Well Matra may be up and running but it's got issues. http://carbonize.co.uk/ you get a 404 and if you try to FTP in you get 421 Home directory not available. Â Can login to cpanel but File Manager says there are no files in my home directory. Edited July 20, 2013 by carbonize Link to comment Share on other sites More sharing options...
j-f Posted July 20, 2013 Share Posted July 20, 2013 on my cpanel : The MySQL® server is currently offline. we couldn't make any invoice today, we lost a lot of money... please make sure it's working asap http://www.reefconcept.com/forum/ Link to comment Share on other sites More sharing options...
Kevan Posted July 20, 2013 Share Posted July 20, 2013 The smf forum on our site is not working either. Instead of the correct folders and files in the forum folder /smf/, there are a number of core.???? type files in there. core.10001 for example  matra server Link to comment Share on other sites More sharing options...
carbonize Posted July 20, 2013 Share Posted July 20, 2013 (edited) God I'm always getting core files from SMF. They are usually created whenever something causes PHP or MySQL to crash. Edited July 20, 2013 by carbonize Link to comment Share on other sites More sharing options...
vdemente Posted July 21, 2013 Share Posted July 21, 2013 We are still having problems with mail, website and hosting. When you think this will be solved? This starts to be a big problem for us Link to comment Share on other sites More sharing options...
TCH-Mike Posted July 21, 2013 Share Posted July 21, 2013 Most sites are back up. If you are still experiencing issues please open a support ticket via https://support.totalchoicehosting.com  Thanks. Link to comment Share on other sites More sharing options...
Head Guru Posted July 21, 2013 Share Posted July 21, 2013 I will be posting a full disclosure of the events as soon as I can get the time.  We are still working some server issues, but this only accounts for  As Mike stated drop us a ticket if your having issues still, we will see the tickets faster than we will see anything you post here.  Thanks for your support during this serious issue at TCH. 1 Link to comment Share on other sites More sharing options...
Kevan Posted July 21, 2013 Share Posted July 21, 2013 Thank You all for your hard work and long hours today. Link to comment Share on other sites More sharing options...
Head Guru Posted July 21, 2013 Share Posted July 21, 2013 Kevan it is clients like you and the many thousands at TCH that make our job such a pleasure. Â I am still working with the team , many of us have been working non stop since 3am this morning, but we are slowly getting things back in order. Â I will update soon with full story. Link to comment Share on other sites More sharing options...
soned Posted July 21, 2013 Share Posted July 21, 2013 i have been with TCH for 10 years. TCH is the best. what happened was unexpected and only brings out the best in TCH. Â and these things do happen in other NOCs as well. I have a couple of servers at theplanet(now softlayer) and my server there was also affected by their power outage (short circuit) which i think is more preventable than a lightning strike. Here is the article on the >1 day power outage at theplanet which my server was a part of: Â http://www.datacenterknowledge.com/archives/2008/06/01/explosion-at-the-planet-causes-major-outage/ 1 Link to comment Share on other sites More sharing options...
debdwright Posted July 21, 2013 Share Posted July 21, 2013 Matra still having issues. Sent ticket in. My domain and email are still not working. Â www. orangeshoalsservicedirectory.org Link to comment Share on other sites More sharing options...
carbonize Posted July 21, 2013 Share Posted July 21, 2013 Matra still having issues. Sent ticket in. My domain and email are still not working.  www. orangeshoalsservicedirectory.org  I had a similar issue with my forum not working. I checked the error log via cpanel and it turns out that they have changed something on the server so access to my forum was being blocked because it's permissions were set to 777. Link to comment Share on other sites More sharing options...
ItsJustMe Posted July 21, 2013 Share Posted July 21, 2013 I still haven't received email that was sent during the outage (Felucia server). Please post here when all the issues are resolved so I'll know whether or not to submit another support ticket for this. (I've already submitted one and your tech said it should be restored within 24 hours.) Â Link to comment Share on other sites More sharing options...
TCH-Thomas Posted July 21, 2013 Share Posted July 21, 2013 There was no email sent to anyone abotu this (as far as I know) since communication regarding this has mostly been through the forum. Â Announcements about the outage has also been posted on Facebook & Twitter but not by email. Link to comment Share on other sites More sharing options...
ItsJustMe Posted July 21, 2013 Share Posted July 21, 2013 TCH-Thomas, I wasn't referring to email about the outage, I'm talking about emails sent to me by visitors and members of my website that haven't gone through due to the outage. I'm missing email from about a 12-hour period (estimated). I want to know when everything should be back to normal so if I don't get that email, I'll know to submit another support ticket. Link to comment Share on other sites More sharing options...
TCH-Thomas Posted July 21, 2013 Share Posted July 21, 2013 Ok. Then I missunderstood you. My apologize. Link to comment Share on other sites More sharing options...
Head Guru Posted July 22, 2013 Share Posted July 22, 2013 Â I had a similar issue with my forum not working. I checked the error log via cpanel and it turns out that they have changed something on the server so access to my forum was being blocked because it's permissions were set to 777. Most likley this occured during the FSCK, but I am not sure. Link to comment Share on other sites More sharing options...
ItsJustMe Posted July 22, 2013 Share Posted July 22, 2013 Head Guru, can you answer my question about email? I'm still missing email from the outage. Â Â Link to comment Share on other sites More sharing options...
Head Guru Posted July 22, 2013 Share Posted July 22, 2013 i have been with TCH for 10 years. TCH is the best. what happened was unexpected and only brings out the best in TCH. Â and these things do happen in other NOCs as well. I have a couple of servers at theplanet(now softlayer) and my server there was also affected by their power outage (short circuit) which i think is more preventable than a lightning strike. Here is the article on the >1 day power outage at theplanet which my server was a part of: Â http://www.datacenterknowledge.com/archives/2008/06/01/explosion-at-the-planet-causes-major-outage/ Â I remember the day that incident occured. It was very big explosion and they had thousands of clients off line for days and days. Link to comment Share on other sites More sharing options...
Head Guru Posted July 22, 2013 Share Posted July 22, 2013 Site on matra is still down.  Site on endor is working fine now  Thank You for keeping us informed during this busy time I am looking at upgrading the Matra Server into all new hardware, but this appears not to be an emergency so something that we will plan out. Link to comment Share on other sites More sharing options...
Head Guru Posted July 22, 2013 Share Posted July 22, 2013 Matra still having issues. Sent ticket in. My domain and email are still not working. Â www. orangeshoalsservicedirectory.org Your site appears to up and online. If your having issues please submit a help desk ticket, its really the fastest way to get assistance. Link to comment Share on other sites More sharing options...
Head Guru Posted July 22, 2013 Share Posted July 22, 2013 TCH-Thomas, I wasn't referring to email about the outage, I'm talking about emails sent to me by visitors and members of my website that haven't gone through due to the outage. I'm missing email from about a 12-hour period (estimated). I want to know when everything should be back to normal so if I don't get that email, I'll know to submit another support ticket. Â Me - Â Mail delivery is governed by the MX records in the receiving domain's DNS zone. You can have multiple MX records, and the records can be weighted with "preference" so that certain a certain server can be the primary receiver and other servers can be lower priority machines. Â So if multiple MX records exist and a mail server is down, then mail delivery goes to the next preferenced server. Â Typically, mail servers are configured with retry times, so if the mail servers for the recipient domain are unreachable, the mail is flagged for RETRY. The sending server waits a preset amount of time and then attempts delivery again. If the recipient mail exchangers are still not reachable, then the sending server may return the mail to the RETRY queue, again for a configured amount of time subsequent retries may be a longer wait, say 4 hours for example. The number of retries and the time increments between the first and subsequent retries are all configurable within the sending server by the servers adminisrator. Â Typically, after 12-24 hours of retry if the recipient servers are still not reachable, the sending server will give up and will likely send an undeliverable report to the sender indicating that it could not deliver the email. Â Bottom line is you should get any mail sent to you whenever you have an email outage. Link to comment Share on other sites More sharing options...
Head Guru Posted July 22, 2013 Share Posted July 22, 2013 I have lived in this state my entire life and I have never seen storms like this. Â Appears at one point there were 150K DTE customers without power. Â http://www.theoaklandpress.com/articles/2013/07/21/news/local_news/doc51ebdcb48105b081693297.txt Link to comment Share on other sites More sharing options...
ItsJustMe Posted July 22, 2013 Share Posted July 22, 2013 <Bottom line is you should get any mail sent to you whenever you have an email outage.> Â Great. So it's been 12-24 hours, so where is it? Link to comment Share on other sites More sharing options...
Head Guru Posted July 22, 2013 Share Posted July 22, 2013 Â Great. So it's been 12-24 hours, so where is it? Since this email did not orginate from our server, I can not tell you. I can only tell you how email should work. I know thats not the answer, but it is the the only one I can give you. Link to comment Share on other sites More sharing options...
ItsJustMe Posted July 22, 2013 Share Posted July 22, 2013 Okay, thanks. Link to comment Share on other sites More sharing options...
hatala Posted July 22, 2013 Share Posted July 22, 2013 I have lived in this state my entire life and I have never seen storms like this.  Appears at one point there were 150K DTE customers without power.  http://www.theoaklandpress.com/articles/2013/07/21/news/local_news/doc51ebdcb48105b081693297.txt   Bill,  Thank you for your tireless support and supreme customer support that you have provided and continue to provide for 10+ years now. I know we we not spoken as on late and TCH will always be my first and total choice for hosting. I am not as involved as much as of late but will always promote TCH as the hosting service of choice. Let's chat sometime soon. I hope you still find time for your bikes.  Gary Hatala ex owner of TSSB Link to comment Share on other sites More sharing options...
Recommended Posts