Jump to content


Photo

Applecore


  • Please log in to reply
26 replies to this topic

#1 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 01 July 2010 - 07:45 PM

Ryan is at NOC checking this server. Please standby for updates.
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#2 Donnell

Donnell

    New To The Neighborhood

  • Members
  • Pip
  • 6 posts

Posted 01 July 2010 - 09:43 PM

Does anyone know what's happening with this? Our site is still down. Thanks!

Ryan is at NOC checking this server. Please standby for updates.



#3 DPK

DPK

    Family Friend

  • Members
  • PipPip
  • 59 posts

Posted 01 July 2010 - 10:29 PM

Ryan is at NOC checking this server. Please standby for updates.


Status update please. http://www.hyperspin.../18681/3/146080 Says it is up but I cannot get to any sites. Are we going to have the same issue as we had in February?

#4 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 01 July 2010 - 10:48 PM

Despite our best efforts, the server has crashed and we have decided to restore. A completely new server has already been set up. Cpanel install is currently underway which will be followed by server tweaks. Accounts will then be restored. We are sorry for the inconveniences caused. We will update you as things progress.
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#5 t0cS

t0cS

    New To The Neighborhood

  • Members
  • Pip
  • 12 posts

Posted 01 July 2010 - 10:49 PM

Does anyone know what's happening with this? Our site is still down. Thanks!


So are all our Reseller accounts.

#6 DPK

DPK

    Family Friend

  • Members
  • PipPip
  • 59 posts

Posted 01 July 2010 - 10:50 PM

Despite our best efforts, the server has crashed and we have decided to restore. A completely new server has already been set up. Cpanel install is currently underway which will be followed by server tweaks. Accounts will then be restored. We are sorry for the inconveniences caused. We will update you as things progress.



Thanks for the update, but if we have databases, when will the information be restored from. We need to know what to expect when things are back up.

#7 BarFly

BarFly
  • Members
  • 4 posts

Posted 01 July 2010 - 11:13 PM

Ryan is at NOC checking this server. Please standby for updates.


wonderful. I was quite upset with the handling of february's applecore issue, which hurt my revenues and reputation quite badly, but I remained loyal, even after though I didn't even get a courtesy hug. I gave it a second chance. Any reason why I shouldn't cut my my heavy losses?

#8 DPK

DPK

    Family Friend

  • Members
  • PipPip
  • 59 posts

Posted 01 July 2010 - 11:22 PM

wonderful. I was quite upset with the handling of february's applecore issue, which hurt my revenues and reputation quite badly, but I remained loyal, even after though I didn't even get a courtesy hug. I gave it a second chance. Any reason why I shouldn't cut my my heavy losses?


Agree with BarFly, same situation here. The impact I have felt because of February was great. 4 months later, what are we supposed to say. I have lost 20% of my hosting business.

I need to know what to expect for tonight or tomorrow. When your new server is up and running, what am I to expect workload wise as far as databases and email accounts. Also, how could a server crash after 4 months... I just dont understand what is going on over @ TCH.

#9 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 01 July 2010 - 11:24 PM

Cpanel installed, restore just kick started, currently at 1%. I will get you more details very shortly.
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#10 Head Guru

Head Guru

    Bill Kish Head Guru

  • Admins
  • PipPipPipPip
  • 6,798 posts

Posted 01 July 2010 - 11:38 PM

I wanted to give everyone a personal update on the status of this server.

Myself and Ryan headed into the Datacenter after receiving this server down alerts.

The issue appeared to be with the Power Supply. After replacing the power supply we attempted to reboot the server and were faced with a server that would not boot. After replacing the motherboard, ram and raid card we were still unable to get the server to boot. At this point, and taking into consideration the issues that occurred in February, I made the choice to migrate the users on this server to a NEW dell rack unit. This choice was made for several reasons. Those reasons broke down as follows:

1. The amount of time we were spending on attempting to restore the server was approaching critical mass.
2. The backups we had were fresh backups from last night, June 30th. We were running daily backups on this server.
3. The amount of issues we have had with this server were starting to add up
4. Moving to a new Dell unit would give us more reliability and stability.

The new server is a very nice setup. Featuring SCSI 15K RPM Hard Drives, Dual Xeons and all the good stuff that makes up a dell.

My staff is going to make this the best possible migration to a new server. Please understand that this choice was made in an effort to do the best thing for our clients.

Thank you for your continued business and support.

Bill Kish

Head Cook and Bottle Washer

If you need help with your account or have any questions, please feel free to contact me using any of the contact methods below.  I can be reached 24 hours a day seven days per week.

Office :: 800-930-0485 x211
Mobile :: 248-632-3243

email: bill(at)totalchoicehosting.com

Instant Messenger -
AOL Instant Messenger: tchgurubill
Yahoo Messenger : tchgurubill
MSN Messenger : tchgurubill@hotmail.com

Thank you for your support and continued business


#11 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 02 July 2010 - 12:11 AM

Restore at 10%
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#12 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 02 July 2010 - 01:11 AM

15%
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#13 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 02 July 2010 - 01:36 AM

20%
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#14 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 02 July 2010 - 02:38 AM

30%
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#15 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 02 July 2010 - 03:05 AM

35%
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#16 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 02 July 2010 - 04:01 AM

40%
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#17 BarFly

BarFly
  • Members
  • 4 posts

Posted 02 July 2010 - 04:29 AM

I wanted to give everyone a personal update on the status of this server.

Myself and Ryan headed into the Datacenter after receiving this server down alerts.

The issue appeared to be with the Power Supply. After replacing the power supply we attempted to reboot the server and were faced with a server that would not boot. After replacing the motherboard, ram and raid card we were still unable to get the server to boot. At this point, and taking into consideration the issues that occurred in February, I made the choice to migrate the users on this server to a NEW dell rack unit. This choice was made for several reasons. Those reasons broke down as follows:

1. The amount of time we were spending on attempting to restore the server was approaching critical mass.
2. The backups we had were fresh backups from last night, June 30th. We were running daily backups on this server.
3. The amount of issues we have had with this server were starting to add up
4. Moving to a new Dell unit would give us more reliability and stability.

The new server is a very nice setup. Featuring SCSI 15K RPM Hard Drives, Dual Xeons and all the good stuff that makes up a dell.

My staff is going to make this the best possible migration to a new server. Please understand that this choice was made in an effort to do the best thing for our clients.

Thank you for your continued business and support.


That is great an all Bill, but when the Ford we rent from you breaks down and strands us in the desert not just once, but twice, don't talk about the Ferrari that is being polished up before it comes to town, just send us a damn cab.

Let me refer you to the previous failure, and some of the statements made by myself, other customers, and TCH staff.
http://www.totalchoi...showtopic=39245

Prior to February's catastrophic failure, I had, via Support Tickets, expressed concern about the stability of Applecore, asked about moving to another box, offered to pay more to be on another. I've inquired if any scripts I had running were hogging up too many resources. I think I'm a fairly proactive customer in making sure I'm not taking advantage of the services you offer.

That said: It was a warm fuzzy feeling to receive a bill weeks after TCH admitted it's February screw ups that cost me exponentially more than the hosting fees.

It's the principle of the thing man. How can you salvage this relationship with those of us who have trusted our sites with RCH for nearly 6 years?

#18 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 02 July 2010 - 04:36 AM

45%
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#19 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 02 July 2010 - 04:58 AM

55%
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#20 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 02 July 2010 - 05:22 AM

60%
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#21 TCH-Carl

TCH-Carl

    Technical Support

  • Staff
  • PipPipPipPip
  • 1,180 posts

Posted 02 July 2010 - 06:06 AM

account restore is over, we are working on the final configuration after which dns will be re-synced. Please wait till the entire process is reported as complete and if you are still facing issues, please contact support.
Carl Noonan
Techical Support Manager
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#22 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 02 July 2010 - 07:03 AM

Can you let us know what period email was down for, and whether during that period incoming email will be lost? I just need to tell my clients this so they can contact anyone they may have been expecting email from during that period.

#23 TCH-Bruce

TCH-Bruce

    Volunteer Moderator

  • Members
  • PipPipPipPip
  • 19,960 posts

Posted 02 July 2010 - 07:06 AM

Email was down the entire time the server was down. Any email coming to your accounts should still arrive as most email transport systems will try for 5 days to deliver a message before giving up.

Bruce Richards
Forum Moderator
TotalChoice Hosting, Inc.
Webhosting by Total Choice Web Hosting - General Support Forum

I am a Forum Moderator. While I can assist in answering most of your hosting related questions, I am unable to answer questions about specifics relating to your account such as billing and server related issues. Should you need assistance in these areas, please contact our Help Desk or our many other options. Another good place to find answers is with our help pages, tutorials and movie tutorials.


#24 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 02 July 2010 - 07:25 AM

Thanks for info Bruce.

Some of my sites are back up but all with MySQL databases are getting errors like this:

user warning: Can't create/write to file '/tmp/#sql_37d9_0.MYI' (Errcode: 13) query: SELECT t.* FROM term_node r INNER JOIN term_data t ON r.tid = t.tid INNER JOIN vocabulary v ON t.vid = v.vid WHERE r.nid = 1 ORDER BY.....

Is that because MySQL backups not fully restored yet? Just want to make sure if I should contact support or wait a bit.

#25 rick02840

rick02840

    Family Friend

  • Members
  • PipPip
  • 49 posts

Posted 02 July 2010 - 07:49 AM

Thanks for info Bruce.

Some of my sites are back up but all with MySQL databases are getting errors like this:

user warning: Can't create/write to file '/tmp/#sql_37d9_0.MYI' (Errcode: 13) query: SELECT t.* FROM term_node r INNER JOIN term_data t ON r.tid = t.tid INNER JOIN vocabulary v ON t.vid = v.vid WHERE r.nid = 1 ORDER BY.....

Is that because MySQL backups not fully restored yet? Just want to make sure if I should contact support or wait a bit.


Suddenly all is good now; errors gone. Thanks.

#26 TCH-Alex

TCH-Alex

    Technical Support

  • Staff
  • PipPipPipPip
  • 663 posts

Posted 02 July 2010 - 08:18 AM

Server restore has been completed.

We verified random accounts and found they are working fine.

If you have any issues, please post a ticket to the help desk.

We thank all of you for your patience and kind cooperation.

Alex Spaford
Technical Support
TotalChoice Hosting, Inc.
Total Choice Hosting - Helpdesk


#27 Head Guru

Head Guru

    Bill Kish Head Guru

  • Admins
  • PipPipPipPip
  • 6,798 posts

Posted 02 July 2010 - 09:38 AM

That is great an all Bill, but when the Ford we rent from you breaks down and strands us in the desert not just once, but twice, don't talk about the Ferrari that is being polished up before it comes to town, just send us a damn cab.

Let me refer you to the previous failure, and some of the statements made by myself, other customers, and TCH staff.
http://www.totalchoi...showtopic=39245

Prior to February's catastrophic failure, I had, via Support Tickets, expressed concern about the stability of Applecore, asked about moving to another box, offered to pay more to be on another. I've inquired if any scripts I had running were hogging up too many resources. I think I'm a fairly proactive customer in making sure I'm not taking advantage of the services you offer.

That said: It was a warm fuzzy feeling to receive a bill weeks after TCH admitted it's February screw ups that cost me exponentially more than the hosting fees.

It's the principle of the thing man. How can you salvage this relationship with those of us who have trusted our sites with RCH for nearly 6 years?


I understand your frustration, and I am very sorry about applecore server giving us problems again.

I think this really boils down to a few key points that I talked about in my first post. I would like to clarify that the new server is not a better server, it is just a different server. Our "dead server policy" is to rebuild in box and re-rack. In lay terms this means if we have a server failure we replace the failed parts on the server and return the server to operational status. This policy has always served us very well as we use only enterprise level hardware. However, in the case of this server, I thought it prudent to simply throw away the server and start new. The new dell server is by no means a "Ferrari". It is simply a different server. The main difference is the Hard Drives used in the dell servers. they are SCSI drives versus the SATA drives in the old server.

On to your point about its a matter of principle. You are 100% correct, and this is why TCH spent large amounts of cash to build a very robust and reliable backup system within our Data Center. Down time is one thing that we try to avoid, and for a budget hosting company we do a fantastic job. The thing that I am most proud of is when we do loose a server, that we have always had very reliable backups to help our clients get back on line and in pretty timely manner. Do i feel awful this event occurred, of course I do. I am confident that the hardware issues on this server are forever behind us. I am also very proud of my staff and how they handled this issue. We always are truthful with our clients and never beat around the bush so to speak.

As far as leaving you stranded in the desert, not only did we send you a cab, but I was driving it and had cool Budweiser in the trunk. :helpsmilie: :helpsmilie:

I am really sorry for this and I ask that everyone allow us to show you why you have stayed with us for so long. I give you my professional word that we are standing on greener grass right now.

Here is to a good run of uptime on the new server.

Bill Kish

Head Cook and Bottle Washer

If you need help with your account or have any questions, please feel free to contact me using any of the contact methods below.  I can be reached 24 hours a day seven days per week.

Office :: 800-930-0485 x211
Mobile :: 248-632-3243

email: bill(at)totalchoicehosting.com

Instant Messenger -
AOL Instant Messenger: tchgurubill
Yahoo Messenger : tchgurubill
MSN Messenger : tchgurubill@hotmail.com

Thank you for your support and continued business





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users