Jump to content


Photo

Applecore - Emergency Restore


  • This topic is locked This topic is locked
54 replies to this topic

#1 TCH-Dick

TCH-Dick

    General Manager

  • Admins
  • PipPipPipPip
  • 5,786 posts

Posted 11 February 2010 - 03:39 PM

One of the disks in the raid array on Applecore has failed, however, due to the way the array degraded, it took the whole raid array down with it. To correct this we must perform a full restore on Applecore and use the most recent backups available which are roughly 10 hours old.

We apologize for any inconvenience this issue will clearly caused and updates will be posted as the restore progresses.

Dick DeVance
General Manager
TotalChoice Hosting, Inc
dick@totalchoicehosting.com


Posted Image


#2 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 11 February 2010 - 04:25 PM

Thanks for this info and please keep us updated.

#3 TCWriter

TCWriter

    Family Friend

  • Members
  • PipPip
  • 41 posts

Posted 11 February 2010 - 04:39 PM

Already getting client calls; any ETA for completion?

#4 TCH-Alex

TCH-Alex

    Technical Support

  • Staff
  • PipPipPipPip
  • 663 posts

Posted 11 February 2010 - 05:54 PM

Operating System and cPanel restore completed.

Now, we are restoring the accounts on the server. We shall update this thread as we go up.

Alex Spaford
Technical Support
TotalChoice Hosting, Inc.
Total Choice Hosting - Helpdesk


#5 TCH-Alex

TCH-Alex

    Technical Support

  • Staff
  • PipPipPipPip
  • 663 posts

Posted 11 February 2010 - 08:35 PM

30% completed

Alex Spaford
Technical Support
TotalChoice Hosting, Inc.
Total Choice Hosting - Helpdesk


#6 Donnell

Donnell

    New To The Neighborhood

  • Members
  • Pip
  • 6 posts

Posted 11 February 2010 - 08:41 PM

Thank you for the attention to this, and for the updates. It really does help! I know it's a headache, and I appreciate the way you're handling it.

#7 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 11 February 2010 - 09:15 PM

For anyone, what does it mean if I can get into WHM OK but all mys sites have http://domainname.co...aultwebpage.cgi

" If you are the web site owner, it is possible you have reached this page because:
* The IP address has changed.
* There has been a server misconfiguration.
* The site may have been moved to a different server.
If you are the owner of this website and were not expecting to see this page, please contact your hosting provider."

...when trying to access them?

Also I cannot access any of my sites via CPanel or FTP - username and/or password doesn't work

Is that just the accounts not fully restored yet?

Edited by rick02840, 11 February 2010 - 09:30 PM.


#8 TCH-Alex

TCH-Alex

    Technical Support

  • Staff
  • PipPipPipPip
  • 663 posts

Posted 11 February 2010 - 09:46 PM

51% completed

For anyone, what does it mean if I can get into WHM OK but all mys sites have http://domainname.co...aultwebpage.cgi

" If you are the web site owner, it is possible you have reached this page because:

* The IP address has changed.
* There has been a server misconfiguration.
* The site may have been moved to a different server.

If you are the owner of this website and were not expecting to see this page, please contact your hosting provider."

...when trying to access them?

Is that just the accounts not fully restored yet?


This is because the accounts are being restored now. We really appreciate your kind cooperation and patience while we completing the restore process.

Alex Spaford
Technical Support
TotalChoice Hosting, Inc.
Total Choice Hosting - Helpdesk


#9 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 11 February 2010 - 10:02 PM

OK thanks for the info.

#10 DPK

DPK

    Family Friend

  • Members
  • PipPip
  • 59 posts

Posted 11 February 2010 - 10:13 PM

51% completed



This is because the accounts are being restored now. We really appreciate your kind cooperation and patience while we completing the restore process.



Can someone tell me exactly how things are impacted on our end as far as data is concerned? What time were backups, what happens to any data that has been in our MySql Databases. If the backup was from 8am, I take it anything added after that will be lost? Please confirm

Also, seems like applecore has been down so many times in the last few months. What do we tell our clients about our service who has grown weary of the quality of our web site and e-mail service.

Obviously businesses now days rely on their email extensively. Is there some other type of email service we can use that does not rely on this server?

Why is there a mirror not in place in case issues like this occur.

These are several of the questions from our clients that I have received and I would like to give them a quality response. I've been with TC for 5 yrs now and the issues of recent are making our service look extremely bad.

Concerned extensively,

Dan

#11 superfly

superfly
  • Members
  • 1 posts

Posted 11 February 2010 - 11:03 PM

My customers are starting to get upset. Can someone give me some information on this 'unable to connect to MySQL' error? I just need some data to give them.

#12 TCWriter

TCWriter

    Family Friend

  • Members
  • PipPip
  • 41 posts

Posted 11 February 2010 - 11:41 PM

One of my sites is back (blog), but none of the posts or comments put up since February 5th are extant. Another site is in the same boat, suggesting the backups used to restore these two (most of my other customer sites are still not operating) are more than ten hours old. More like 5-6 days.

I realize this is no fun for anyone, but the increasingly frequent outages my customers have experienced the last month have taken their toll, and one is pulling their site and moving to another host.

It's not a lot of revenue, but... [sigh]

#13 imaginejoey1

imaginejoey1

    New To The Neighborhood

  • Members
  • Pip
  • 5 posts

Posted 11 February 2010 - 11:41 PM

What is the status? I have been told within the hour to just a few more minutes and as we can tell -not true. I have clients really upset and I can't tell them anything because you wont inform us. STATUS!

#14 TCH-Alex

TCH-Alex

    Technical Support

  • Staff
  • PipPipPipPip
  • 663 posts

Posted 12 February 2010 - 12:24 AM

75% completed

Please note that service like Apache, Mysql, exim will not work till the restore completes eventhough cPanel/WHM is up, that is why the sites are not working now. We are working hardly to restore the services asap.

Please accept our apologies for the delay.

Alex Spaford
Technical Support
TotalChoice Hosting, Inc.
Total Choice Hosting - Helpdesk


#15 DPK

DPK

    Family Friend

  • Members
  • PipPip
  • 59 posts

Posted 12 February 2010 - 12:30 AM

75% completed

Please note that service like Apache, Mysql, exim will not work till the restore completes eventhough cPanel/WHM is up, that is why the sites are not working now. We are working hardly to restore the services asap.

Please accept our apologies for the delay.


One of my sites is up and running.... I take backups in a directory DAILY and the last file there was from Feb, 3rd. Am I going to get the files from FEB 4-10th ?

My MySQL databases have been restored from Feb 3rd. Are they going to get RESET to the 10 time period originally mentioned !!!

Come on... Need some good answers here regarding our data.

#16 imaginejoey1

imaginejoey1

    New To The Neighborhood

  • Members
  • Pip
  • 5 posts

Posted 12 February 2010 - 01:08 AM

My sites are still down. I can't access email. And I will probably lose clients over this. So any answers to this statement and questions?

"Can someone tell me exactly how things are impacted on our end as far as data is concerned? What time were backups, what happens to any data that has been in our MySql Databases. If the backup was from 8am, I take it anything added after that will be lost? Please confirm

Also, seems like applecore has been down so many times in the last few months. What do we tell our clients about our service who has grown weary of the quality of our web site and e-mail service.

Obviously businesses now days rely on their email extensively. Is there some other type of email service we can use that does not rely on this server?

Why is there a mirror not in place in case issues like this occur.

These are several of the questions from our clients that I have received and I would like to give them a quality response. I've been with TC for 5 yrs now and the issues of recent are making our service look extremely bad.

Concerned extensively,

Dan"

#17 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 12 February 2010 - 03:08 AM

Please update when you can. At this time (3:05am, east coast) looks like you may have had a problem:

"Thank You for installing cPanel / WHM.
In order to access the interface, you will need to make sure that the software license on this server is active and installed."

If you don't think this will be back up by 7:00am, please let me know.

----- update -----

Ah WHM is back up. I guess you are still restoring. If you have any estimate of timeframe, please let me know. Appreciate it.

Edited by rick02840, 12 February 2010 - 03:11 AM.


#18 folhapvm

folhapvm
  • Members
  • 2 posts

Posted 12 February 2010 - 04:33 AM

Please keep on keeping on!

#19 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 12 February 2010 - 04:39 AM

At present the data restore is completed, we had to ultimately fall back to a backup snapshot from "Feb. 6th 02:30 EST". Though we do have viable data in place in our 12h incremental backups, the speed at which they were restoring was putting us in an untenable position on a realistic or more importantly a sane restore time line. Having the snapshot from the 6th restored at very least gets everyone back online and the server on an even footing so we can begin work on getting data in from the incremental backups.

There is still a number of tasks we are wrapping up on the server but as of this moment most sites should be operating as intended, we do apologize for the clear inconvenience this whole situation has caused and thank everyone for there continued patience.

I will provide a full recap of what happened with applecore and where we stand on the restore shortly.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#20 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 12 February 2010 - 04:49 AM

Good! Many of my sites are back up and some later than Feb 6. Thanks!

#21 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 12 February 2010 - 06:17 AM

Alright let me recap what happened with applecore and where we are now...

Over the last couple of weeks we have had a number of issues with applecore, they were thought to initially be hardware induced so we replaced the servers power supply, swapped out memory and bumped it up to 6GB, swapped out disks in the raid array for new ones and replaced the raid card. Things were looking good for a couple of days, there was no notable outages - things were snappy on the server and we thought the worst of it was behind us.

On Thursday afternoon around 2PM EST the server started to throw file system errors and then hard crashed, upon further inspection the server would not come back online and I unracked it for review. It was quickly apparent that the file system on the main hard disk had catastrophically failed, a run through of fsck (disk checker for Linux) was performed but it was unable to recover anything meaningful. The decision was made to restore the server from backups as opposed to fussing around with disk utilities that would be time consuming and likely not yield any productive results given the state of the servers file system.

Now, some context about the hard disks in our servers -- although we do have raid mirroring (using Areca enterprise level raid) on all our servers, this only protects from (physical) hardware failures of the disks. The raid array has no knowledge of the (logical) file system or its current state, it will mirror data across the array no matter the file system operation that takes place, be it bad or good. So, once the file system had failed on the server by then it had already been mirrored across the array in real-time.

Once we had the basics out of the way with the restore - OS reinstall and server prep kits provisioned we immediately started to restore data from backups. Initially we decided to try restore data from our 12h R1 incremental backups however this proved to be an immensely time consuming task which we recognized after a few hours and made the decision to restore data instead from our weekly local snapshots stored on a physical hard drive instead of in a disk safe across our network.

After the decision was made to restore data from the weekly snapshot (dated Feb 6th 0230EST), the rest of the restore was very straight forward as we have done these type of restores countless times - it was more of a tedious waiting time sink than anything else. That brings us to the here and now, all data has been restored and all services are online and operating as intended.

At the moment we are able to restore data from the 12h incremental backups taken the morning of Feb. 11th 0145EST upon request and for /home data only, we acknowledge that this is not ideal but that is the situation we are in and we express our sincere apologies.

We understand we have inconvenienced you and your customers through this incident, we can promise that a full review of all TCH procedures will be undertaken to ensure this situation never happens again - we expect better from ourselves and you deserve better.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#22 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 12 February 2010 - 08:08 AM

"all data has been restored"

Many of my accounts are back, but some are not. In two cases the sites seems to be up, but MySQL not up yet (cannot find database) and also email is not up. I submitted ticket for those but checking here to see if you are 100% done with restore or not?

#23 imaginejoey1

imaginejoey1

    New To The Neighborhood

  • Members
  • Pip
  • 5 posts

Posted 12 February 2010 - 08:27 AM

I have 3 sites still down-what is the status

#24 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 12 February 2010 - 08:48 AM

We have restored the accounts the best backups that we have at the moment. If you are still facing issues, please raise a support ticket to our technical support team. We will gladly check it out.
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#25 imaginejoey1

imaginejoey1

    New To The Neighborhood

  • Members
  • Pip
  • 5 posts

Posted 12 February 2010 - 08:50 AM

Here's a support ticket: www.imagine-photo.com www.imagine360.com and www.jubileelivingresales.com none of these are working

#26 magicfly

magicfly
  • Members
  • 2 posts

Posted 12 February 2010 - 09:29 AM

Looks like my disk quotas are all off on the whm, making some accounts over quota that shouldn't be.

#27 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 12 February 2010 - 09:37 AM

Looks like my disk quotas are all off on the whm, making some accounts over quota that shouldn't be.


Same here. I am upping mine back to nromal now. Perhaps this is why email not working on some accounts yet. Will report back.

#28 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 12 February 2010 - 09:49 AM

Same here. I am upping mine back to nromal now. Perhaps this is why email not working on some accounts yet. Will report back.



Yes this was why email was still down - all working now for me. If you are having trouble with email try checking disk quotas.

#29 imaginejoey1

imaginejoey1

    New To The Neighborhood

  • Members
  • Pip
  • 5 posts

Posted 12 February 2010 - 11:26 AM

Question? Are all incoming emails from yesterday gone into LaLa land?

#30 TCWriter

TCWriter

    Family Friend

  • Members
  • PipPip
  • 41 posts

Posted 12 February 2010 - 11:48 AM

My disk quotas fubar too, but righting them seems to be solving email issues (I hope).

The email thing pushed another client over the edge. Probably me too.

#31 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 12 February 2010 - 12:10 PM

QUESTION: do you know for sure (or close to it) that what happened yesterday (what you describe above) was the cause of all the problems this server has had starting January and that the fix done overnight will take care of all of that?

Your other servers do not see to have had such problems so hopefully Applecore is now as good as any of them?

#32 Ironic

Ironic
  • Members
  • 3 posts

Posted 12 February 2010 - 12:26 PM

Thank you support for getting the server back up and restored. Hopefully this will be the last of the issues we've had with this server and it will get back to being the rock solid performer it has been since I started hosting my sites here.

As for you whiners you really should cut the support staff some slack. Hardware fails, it's a fact of life. But I am confident that the support staff will get everything back to the way it was.

And if you haven't realized by now the support staff does not monitor this forum for a reason, they are busy working the help desk. Where you should be directing questions about your accounts.

Thanks again support for working through the night getting the server back up and restored. As much as I hate downtime I also know that computers do fail eventually. My hats off to you for your hard work.

#33 TCH-Bruce

TCH-Bruce

    Volunteer Moderator

  • Members
  • PipPipPipPip
  • 19,960 posts

Posted 12 February 2010 - 12:30 PM

Keep it civil guys. We understand some of you are frustrated but we will not tolerate members abusing other members.

Bruce Richards
Forum Moderator
TotalChoice Hosting, Inc.
Webhosting by Total Choice Web Hosting - General Support Forum

I am a Forum Moderator. While I can assist in answering most of your hosting related questions, I am unable to answer questions about specifics relating to your account such as billing and server related issues. Should you need assistance in these areas, please contact our Help Desk or our many other options. Another good place to find answers is with our help pages, tutorials and movie tutorials.


#34 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 12 February 2010 - 12:30 PM

Here's a support ticket: www.imagine-photo.com www.imagine360.com and www.jubileelivingresales.com none of these are working

Support tickets are done here: https://ssl.totalcho...om/supportdesk/
I have done a bunch around this issue and they have been very responsive.

#35 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 12 February 2010 - 12:32 PM

Keep it civil guys. We understand some of you are frustrated but we will not tolerate members abusing other members.

Just want to say that support desk has been very responsive for me during this - solved a bunch of post restore issues. When you get a chance though, please answer my "QUESTION" above. Thanks!

#36 TCH-Bruce

TCH-Bruce

    Volunteer Moderator

  • Members
  • PipPipPipPip
  • 19,960 posts

Posted 12 February 2010 - 12:32 PM

Exactly. If you are still having problems with your accounts please open a ticket with the help desk.

Bruce Richards
Forum Moderator
TotalChoice Hosting, Inc.
Webhosting by Total Choice Web Hosting - General Support Forum

I am a Forum Moderator. While I can assist in answering most of your hosting related questions, I am unable to answer questions about specifics relating to your account such as billing and server related issues. Should you need assistance in these areas, please contact our Help Desk or our many other options. Another good place to find answers is with our help pages, tutorials and movie tutorials.


#37 TCH-Bruce

TCH-Bruce

    Volunteer Moderator

  • Members
  • PipPipPipPip
  • 19,960 posts

Posted 12 February 2010 - 12:35 PM

QUESTION: do you know for sure (or close to it) that what happened yesterday (what you describe above) was the cause of all the problems this server has had starting January and that the fix done overnight will take care of all of that?

Your other servers do not see to have had such problems so hopefully Applecore is now as good as any of them?


Since I do not have access to the servers I can't answer this. But if you ask the help desk maybe someone there can give a definitive answer.

I will say this, if it wasn't the cause the support staff will continue to work on this server until it is.

Bruce Richards
Forum Moderator
TotalChoice Hosting, Inc.
Webhosting by Total Choice Web Hosting - General Support Forum

I am a Forum Moderator. While I can assist in answering most of your hosting related questions, I am unable to answer questions about specifics relating to your account such as billing and server related issues. Should you need assistance in these areas, please contact our Help Desk or our many other options. Another good place to find answers is with our help pages, tutorials and movie tutorials.


#38 TCH-Bruce

TCH-Bruce

    Volunteer Moderator

  • Members
  • PipPipPipPip
  • 19,960 posts

Posted 12 February 2010 - 12:55 PM

A word about the quotas when scripts are installed files that are created by those scripts are assigned to user "nobody". When accounts are restored all the files in the account are set to the correct user accounts. This is why some accounts may be over quota. While these files were not being counted as part of the quota they should be and now are.

Bruce Richards
Forum Moderator
TotalChoice Hosting, Inc.
Webhosting by Total Choice Web Hosting - General Support Forum

I am a Forum Moderator. While I can assist in answering most of your hosting related questions, I am unable to answer questions about specifics relating to your account such as billing and server related issues. Should you need assistance in these areas, please contact our Help Desk or our many other options. Another good place to find answers is with our help pages, tutorials and movie tutorials.


#39 BarFly

BarFly
  • Members
  • 4 posts

Posted 12 February 2010 - 01:40 PM

Hardware failures are 100% understandable and a simple fact that we all have to deal with. That alone doesn't bother me and I appreciate that TCH worked hard to get everything back up, but that still leaves me with this concern: This hurt all of us on applecore. Support communication about the ongoing issues with applecore have not addressed anything in my opinion. Over the past month I have asked many questions of the help desk "is there anything I can do? Is it my sites causing this? Are there any upgrades I should make? Should I move to another machine?". The boiler plate "We are sorry, it is being worked on' response did nothing to either answer my questions or make any assurances.

I can't rightly remember how many years I have been hosting here, and one thing I appreciated is that you guys did such a great job that, up until the past month or so, I rarely had to contact support, and generally when I did, it ended up being because of my own screw ups/ignorance more than anything else.

This applecore situation, when we needed some real solutions and some assurances the most, left me feeling hung out to dry. This is the TCH Family Forum. In lieu of a making a hasty decision to divorce, maybe it's time for a little family counseling here. How about it TCH? What can we do to help your service run smoothly, and how can you help us feel confident that it won't be such a travesty should something like this happen again?

Can TCH offer some sort of redundant backup server to resellers? Having an "applecore2" set up, that could be flipped on in an emergency like this, would be awesome. I don't know about the rest of you guys who were affected, but I think it would be worth every penny to pay an extra monthly fee to cover the added costs of such a service.

#40 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 12 February 2010 - 01:47 PM

The e-mail protocol is a resilient one, typically when mail fails to deliver due to outages the remote server will queue the message for up to 72h and attempt to resend the mail at 6 or 12h intervals. In other cases, the remote server might be configured to fail immediately when messages do no deliver, in this situation the sender of the message will receive an error indicating there was a problem and that they should try again later.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#41 folhapvm

folhapvm
  • Members
  • 2 posts

Posted 12 February 2010 - 02:05 PM

Helpdesk Rocks! Thanks for the quick responses and fixes!

#42 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 12 February 2010 - 02:16 PM

I can't rightly remember how many years I have been hosting here, and one thing I appreciated is that you guys did such a great job that, up until the past month or so, I rarely had to contact support, and generally when I did, it ended up being because of my own screw ups/ignorance more than anything else.

This applecore situation, when we needed some real solutions and some assurances the most, left me feeling hung out to dry. This is the TCH Family Forum. In lieu of a making a hasty decision to divorce, maybe it's time for a little family counseling here. How about it TCH? What can we do to help your service run smoothly, and how can you help us feel confident that it won't be such a travesty should something like this happen again?

Can TCH offer some sort of redundant backup server to resellers? Having an "applecore2" set up, that could be flipped on in an emergency like this, would be awesome. I don't know about the rest of you guys who were affected, but I think it would be worth every penny to pay an extra monthly fee to cover the added costs of such a service.


Here is the straight deal, with applecore we dropped the ball - there is no excuses - we screwed up. Normally what happens when we have an issue with a server as consistently as applecore is that we would schedule a migration of the server, it would have involved us restoring fresh backups of applecore to a new system then once completed do a final sync of mysql/account data and "flipping" the IP routing from applecore to the new system. This would have and usually does provide a very seamless migration with no downtime and little in the way of support issues for anyone (do not confuse this with the migrations of when we moved into the TCH DC, that was a different kind of migration). This is something we have done in the past on multiple servers when they exhibit signs of failure but in this case with applecore it was simply not done for whatever reason and we take responsibility for that.

We are undertaking a full review of our restore/migration and backup procedures with changes effective very soon. In addition we will be implementing a stand-alone MySQL backup system that will perform full MySQL data dumps of all our shared, reseller and dedicated servers every X hours (we are thinking every 4 or 6h), this would provide a much more reliable and consistent platform for MySQL data to be retrieved from.

Additionally, we are working on a clustered MySQL project which might also be expanded into a high-availability hosting project where should you desire you can sign up to a hosting solution where everything is redundant from the web server, account data, mysql data, e-mail service down to the hardware side of having raided hard disks, redundant power supplies, redundant memory and so on.

We recognize that hosting is evolving and we intend to work towards a next generation hosting platform for all our clients - not an exclusive service - where 0 downtime & total redundancy is the standard. Please keep an eye out for forum updates as we will be posting some blogs in the coming days/weeks with exact details on how we have improved our backup system and what our high-availability hosting project will look like.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#43 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 12 February 2010 - 03:53 PM

Thanks for up front explanation. It sounds like you are confident that Applecore is all set now. Any idea what caused the file system errors to begin with and whether that might occur again? Thanks.

#44 BarFly

BarFly
  • Members
  • 4 posts

Posted 12 February 2010 - 10:25 PM

Yes, thank you for the explanation Ryan.

#45 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 13 February 2010 - 07:43 AM

We need to recompile apache to correct some of the settings and add a couple of php modules. Since this involves apache downtime of about 30 minutes, we are scheduling this to the service interval of 12am - 6am EST when the loads and traffic on the server is at its minimum. We will update this thread as we move along.
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#46 youneverknow

youneverknow

    Immediate Family

  • Members
  • PipPipPipPip
  • 702 posts

Posted 13 February 2010 - 06:42 PM

Just curious, why does Applecore not show up in the uptime statistics log as being down for all those hours?
youneverknow

Edit...Just saw the down time on the 2/11/10 Logs. But still it only shows 1 1/2 hours down...is that right?

Edited by youneverknow, 13 February 2010 - 06:47 PM.

In a world without walls or fences who needs windows or gates?

#47 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 14 February 2010 - 12:13 AM

We need to recompile apache to correct some of the settings and add a couple of php modules. Since this involves apache downtime of about 30 minutes, we are scheduling this to the service interval of 12am - 6am EST when the loads and traffic on the server is at its minimum. We will update this thread as we move along.


Using a new method to add php extensions, we were able to complete the required tasks on applecore with no downtime.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#48 TCH-Dick

TCH-Dick

    General Manager

  • Admins
  • PipPipPipPip
  • 5,786 posts

Posted 14 February 2010 - 01:41 AM

Just curious, why does Applecore not show up in the uptime statistics log as being down for all those hours?
youneverknow

Edit...Just saw the down time on the 2/11/10 Logs. But still it only shows 1 1/2 hours down...is that right?


Unlike our internal system that monitors all services, the external system is only used monitor the availability of the web server(Apache). While all services were not fully available to all sites, once the OS and cPanel were restored then Hyperspin seen the server as up. This is not normally an issue with the expection of rare case such as a restore.

We do our best to insure the statistics we provide are as accurate as possible, however, in this case that's just not possible.

As for the entire time frame from failure to repair to restore, the issues with this server begin at 11Feb2010 01:57 PM EST and the primary restore of accounts was completed at 12Feb2010 04:30 AM EST.

If you have any other questions please let us know.

Dick DeVance
General Manager
TotalChoice Hosting, Inc
dick@totalchoicehosting.com


Posted Image


#49 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 14 February 2010 - 11:00 AM

Using a new method to add php extensions, we were able to complete the required tasks on applecore with no downtime.

Good Job! Thanks. Everything good on my end so far.

#50 magicfly

magicfly
  • Members
  • 2 posts

Posted 17 February 2010 - 12:11 PM

Did we used to have Fantastico on this server? Or was that removed before the crash?

#51 TCH-Thomas

TCH-Thomas

    Volunteer Moderator

  • Members
  • PipPipPipPip
  • 14,908 posts

Posted 17 February 2010 - 01:10 PM

It should be on all servers. If it´s missing, please submit a ticket with the helpdesk and the techs will look into it. Link on top of page and in my signature.

Thomas Jikrantz
Forum Moderator
TotalChoice Hosting, Inc.

Any links or suggestions for third party software/sites should be used at your own risk. My opinions and recommendations are not necessary those of TCH and TCH is not responsible.

As a Forum Moderator I can assist in answering many of your hosting related questions. However, I am unable to answer questions about specifics relating to your account such as billing and server related issues. Should you need assistance in these areas, please contact our Help Desk or our many other options. Another good place to find answers is with our help pages, tutorials and movie tutorials.
Web Hosting by Total Choice Web Hosting - 24/7 Help Desk


#52 TCH-Alex

TCH-Alex

    Technical Support

  • Staff
  • PipPipPipPip
  • 663 posts

Posted 17 February 2010 - 06:55 PM

Did we used to have Fantastico on this server? Or was that removed before the crash?


Fantastico has been enabled on the server.

Alex Spaford
Technical Support
TotalChoice Hosting, Inc.
Total Choice Hosting - Helpdesk


#53 sheryltoo

sheryltoo

    Distant Family

  • Members
  • PipPipPip
  • 139 posts

Posted 19 February 2010 - 12:41 PM

I just tried to open all my forums on my data bases, and then tried to open my domain site and there's nothing there which is why I'm here looking for some answers.
It looks as if there was a problem a few days ago but it didn't effect me until now.
Is the fact that I can't access my sites today, related to what's been talked about in this thread?

#54 TCH-Thomas

TCH-Thomas

    Volunteer Moderator

  • Members
  • PipPipPipPip
  • 14,908 posts

Posted 19 February 2010 - 12:43 PM

No, the applecore server should be up and running again, so please open a ticket with the helpdesk and the techs will look into it.

Thomas Jikrantz
Forum Moderator
TotalChoice Hosting, Inc.

Any links or suggestions for third party software/sites should be used at your own risk. My opinions and recommendations are not necessary those of TCH and TCH is not responsible.

As a Forum Moderator I can assist in answering many of your hosting related questions. However, I am unable to answer questions about specifics relating to your account such as billing and server related issues. Should you need assistance in these areas, please contact our Help Desk or our many other options. Another good place to find answers is with our help pages, tutorials and movie tutorials.
Web Hosting by Total Choice Web Hosting - 24/7 Help Desk


#55 sheryltoo

sheryltoo

    Distant Family

  • Members
  • PipPipPip
  • 139 posts

Posted 19 February 2010 - 12:54 PM

No, the applecore server should be up and running again, so please open a ticket with the helpdesk and the techs will look into it.

Alright. Thanks.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users