Jump to content


Photo

Elom Restore


  • This topic is locked This topic is locked
29 replies to this topic

#1 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 07:27 AM

One of the disks in the raid array on elom has gone bad but in the process, due to the way the array degraded, it took the whole raid array down with it. As a result, we are forced to do a full restore on elom and use the most recent backups available which are roughly 12H old.

We apologize for any inconvenience this issue will clearly cause, updates will be posted as the situation progresses further.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#2 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 09:20 AM

The restore process is currently under way and at 9%, we will post updates as it progresses further.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#3 Samrc

Samrc

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,336 posts

Posted 26 June 2008 - 09:56 AM

THANK YOU so much for posting this information on the forum!
Saw my site was down and was relieved to find that you folks were already working on it.
Samantha
SamiSite.com CSB and WYSIWYG program tutorials
Feel free to right click the silly little forum images I use and save them to your own website space.
Then link to your own site. Please don't hotlink to my website to use the images. Thanks!

#4 phatfunkjazz

phatfunkjazz

    Distant Family

  • Members
  • PipPipPip
  • 115 posts

Posted 26 June 2008 - 10:38 AM

Ahh...computers, you have to love 'em. I'm always grateful when mine successfully boots up in the morning.
Thanks for the update; good luck with the repair.

Stefan

#5 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 10:47 AM

We just went past 21%, it going a bit slow but typically does at first, further updates as the restore progresses.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#6 debalex

debalex

    New To The Neighborhood

  • Members
  • Pip
  • 12 posts

Posted 26 June 2008 - 11:31 AM

If I was on former Server 26, would this explain why the site is down and there is no email access?

#7 TCH-Bruce

TCH-Bruce

    Volunteer Moderator

  • Members
  • PipPipPipPip
  • 19,960 posts

Posted 26 June 2008 - 11:36 AM

Yes, server 26 no longer exists and this is the server your account was migrated too.

Bruce Richards
Forum Moderator
TotalChoice Hosting, Inc.
Webhosting by Total Choice Web Hosting - General Support Forum

I am a Forum Moderator. While I can assist in answering most of your hosting related questions, I am unable to answer questions about specifics relating to your account such as billing and server related issues. Should you need assistance in these areas, please contact our Help Desk or our many other options. Another good place to find answers is with our help pages, tutorials and movie tutorials.


#8 debalex

debalex

    New To The Neighborhood

  • Members
  • Pip
  • 12 posts

Posted 26 June 2008 - 11:40 AM

Thanks.

I will check back in a few hours and hopefully all will be back to normal.

Have a great day.

#9 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 12:47 PM

35% and still kicking.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#10 metalfork

metalfork

    New To The Neighborhood

  • Members
  • Pip
  • 8 posts

Posted 26 June 2008 - 12:57 PM

thanks for the updates guys. i love your service!

#11 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 01:44 PM

41% or on account names beginning with gi*, accounts should be returning online/web accessible as they are restored.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#12 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 02:33 PM

52% or on accounts beginning with ia*
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#13 kaseytraeger

kaseytraeger

    Immediate Family

  • Members
  • PipPipPipPip
  • 520 posts

Posted 26 June 2008 - 03:02 PM

My site is hosted on elom but is much further down the alphabetical list. Based on the time you've said the last backup occurred and estimating roughly 11-15% of sites being restored each hour, I could conceivably be without email for roughly 24 hours. So here are my questions...

(1) What will happen to email that is sent to any email address associated with my account during the time that my account is "down?" Will they be saved and somehow properly routed to my email accounts when my main hosting account is restored? Or will they be bounced back to the sender(s) as undeliverable? Or, worse yet, will they just somehow disappear into the ethos we call cyberspace?

(2) Am I correct in my assumption that all email received before the server crashed (whether or not it was actually "read" before the server crashed), but after the last backup occurred, is gone for good?

I can recreate files I was working on yesterday with a relatively small amount of effort, but it's the email that I need to track down.

Thank you.
.: Kasey :.

#14 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 03:10 PM

kaseytraeger;
The way the SMTP (email) protocol works is that remote mail servers do one of two things when the destination server is down; 1) they typically queue the message for later delivery storing the message for up too 72hours then/or 2) reject the mail back to the sender with a destination can not be reached error in which case the sender is aware you did not receive the message.

Any data that was saved to the server between the last backup snapshot and the server failure is lost, this include emails and any data uploaded or modified. We currently run local network backups on 12 hour increments in addition to full weekly snapshots of the whole server, this is really the best balance of server load and data integrity we can provide - not to mention we also run raid arrays on all servers, which in this situation, was not able to avert a catastrophic server failure.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#15 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 03:11 PM

66% or on accounts beginning with L*
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#16 kaseytraeger

kaseytraeger

    Immediate Family

  • Members
  • PipPipPipPip
  • 520 posts

Posted 26 June 2008 - 03:31 PM

Hi TCH-Ryan,

Thank you for the explanation. Although I'm fairly tech-savvy from an end-user perspective, I don't know the backend workings of email protocol, server backups and restoration, etc.

Just so you know, I'm not angry at the situation. If I came across that way, I apologize. I completely understand that sometimes, in spite of our best efforts, #%^&* happens. I just needed to know how email was handled so I can properly address the situation on my end. Fortunately for me, I don't hand out my email address willy-nilly, so I can pretty much determine who might have sent email to each of my TCH-hosted addresses and can write to all those folks to let them know that if they sent email with the last day or so, it may not have been received.

Thanks again for your dedicated efforts to restore elom.

My best regards,
.: Kasey :.

#17 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 04:24 PM

73%
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#18 TruckRDJ

TruckRDJ
  • Members
  • 1 posts

Posted 26 June 2008 - 04:51 PM

Thanks for the quick response and keeping us posted.

#19 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 05:27 PM

I am unable to provide an accurate progress percentage as the restore process had to be interrupted and restarted due to a browser session issue in the restore manager. However, we can guesstimate our progress alphabetically, we are currently restoring accounts beginning with the letter P, so we are roughly around 80% give or take a bit.

Thank you for your continued patience while we work through this issue.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#20 metalfork

metalfork

    New To The Neighborhood

  • Members
  • Pip
  • 8 posts

Posted 26 June 2008 - 06:03 PM

Come on "T" :D

#21 kaseytraeger

kaseytraeger

    Immediate Family

  • Members
  • PipPipPipPip
  • 520 posts

Posted 26 June 2008 - 06:09 PM

Come on "T" :D


It may be a while still, metalfork. My domain is at the tail end of "P", and it's still awaiting restoration. Like yourself, I've been keeping an eye out :D all day!
.: Kasey :.

#22 Samrc

Samrc

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,336 posts

Posted 26 June 2008 - 06:25 PM

Been watching all day too.
S is a long way down the alphabet.....
How about next time you load based on owner's last name? That would boot me up by several hours! :D :D
Samantha
SamiSite.com CSB and WYSIWYG program tutorials
Feel free to right click the silly little forum images I use and save them to your own website space.
Then link to your own site. Please don't hotlink to my website to use the images. Thanks!

#23 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 06:27 PM

Accounts beginning with ro are currently being restored.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#24 phatfunkjazz

phatfunkjazz

    Distant Family

  • Members
  • PipPipPip
  • 115 posts

Posted 26 June 2008 - 07:02 PM

Accounts beginning with ro are currently being restored.


My account beginning with "ph" is still not available (as of 5:00 PM Pacific Time); should I submit a Help Desk ticket?

Stefan

#25 Animus

Animus
  • Members
  • 1 posts

Posted 26 June 2008 - 07:18 PM

Accounts beginning with ro are currently being restored.



I have the same question. My account starts with a ci and is still not up yet. Do we have to wait until everything is restored?

Thanks TCH Ryan!!
Gerry

#26 deejay3615

deejay3615
  • Members
  • 3 posts

Posted 26 June 2008 - 07:19 PM

Accounts beginning with ro are currently being restored.



My account beginning with "ki is still not available (as of 5:00 PM Pacific Time); should I submit a Help Desk ticket?

David (KINGSCHOICE.COM)

#27 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 07:20 PM

Tickets should only be submitted once the restore is announced completed. Although the restore process typically returns a site online after it has been individually restored, there are a number of factors that effect the situation based on the type of account, the resources it uses and content there-in. Quite simply put, sometimes it is only possible for the account to come back online fully once the restore as a whole has completed and post-restore operations go through such accounts and bring them into a full working state.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#28 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 07:51 PM

We are at accounts beginning with sp
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#29 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 08:47 PM

Currently at accounts beginning with to
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#30 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 26 June 2008 - 09:42 PM

The restore has completed to 100%, if you are experiencing any further issues then please open a support ticket and we will promptly investigate. We express our apologies again for any inconvenience this issue has caused and thank everyone for there patience.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users