Jump to content


Photo

Tarvis Maintenance @ 12:00am 10/07/07


  • This topic is locked This topic is locked
15 replies to this topic

#1 TCH-Ryan

TCH-Ryan

    Immediate Family

  • Members
  • PipPipPipPip
  • 274 posts

Posted 06 October 2007 - 09:41 AM

We will be conducting maintenance on tarvis this evening at 12:00am (10/07/07) to swap out hardware on the server in an effort to correct stability issues. The expected down time will be less than 30 minutes but can be up to 1 hour if there are any unpexected issues while the server is returning online with new hardware.

Our apologies in advance for any inconvenience this may cause and we will keep you updated in this thread as the maintenance progresses.
Kind Regards;
Ryan MacDonald
Data Center Manager | TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com | ryan@totalchoicehosting.com

#2 TCH-James

TCH-James

    Immediate Family

  • Members
  • PipPipPipPip
  • 404 posts

Posted 06 October 2007 - 10:56 PM

We are bringing this server down now. We will be replacing the drive and the raid card. When it comes back online, the raid array will be rebuilt. Once this completes, we will take it down momentarily to replace the second drive.

#3 TCH-James

TCH-James

    Immediate Family

  • Members
  • PipPipPipPip
  • 404 posts

Posted 06 October 2007 - 11:28 PM

Tarvis is currently running a filesystem check. I will post back when the check is completed.

#4 TCH-James

TCH-James

    Immediate Family

  • Members
  • PipPipPipPip
  • 404 posts

Posted 06 October 2007 - 11:51 PM

The server found some issues during the filesystem check which required a manual run of the filesystem check. We are currently working our way through this process and will post updates as we go.

#5 TCH-James

TCH-James

    Immediate Family

  • Members
  • PipPipPipPip
  • 404 posts

Posted 07 October 2007 - 12:07 AM

Tarvis is back online and we are currently rebuilding the raid array with the new disk. I will post updates as we continue.

#6 TCH-James

TCH-James

    Immediate Family

  • Members
  • PipPipPipPip
  • 404 posts

Posted 07 October 2007 - 01:16 AM

During the rebuild of the raid array, we lost the filesystem journal causing the server to crash. After rebooting the server, the raid array came up in a bad state. Due to these issues, we have placed a new machine online and will begin working on a restore from backups. We will keep updating this post as the restore continues.

#7 TCH-James

TCH-James

    Immediate Family

  • Members
  • PipPipPipPip
  • 404 posts

Posted 07 October 2007 - 01:49 AM

We were able to bring the old machine back online so that all users will be back online. We are currently prepping a new machine and will then restore from backups. Once this is done, we will put the new machine online in place of the old one.

Updates will be posted here as we progress.

Thank you for your patience and understanding during this time.

#8 TCH-Vivek

TCH-Vivek

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,814 posts

Posted 07 October 2007 - 03:01 AM

The old box is online as Jesse said. We have prepared the new box for production and accounts are being restored from Saturday's backup run. Once this is over we will be syncing databases and mails with the old box and bring the new box in place of the old one. 12% done so far.
Warmest regards,
Vivek Bhatia
Lead Systems Administrator
TotalChoice Hosting, LLC.
http://www.totalchoicehosting.com

#9 TCH-Vivek

TCH-Vivek

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,814 posts

Posted 07 October 2007 - 03:19 AM

23%
Warmest regards,
Vivek Bhatia
Lead Systems Administrator
TotalChoice Hosting, LLC.
http://www.totalchoicehosting.com

#10 TCH-Vivek

TCH-Vivek

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,814 posts

Posted 07 October 2007 - 03:40 AM

37%
Warmest regards,
Vivek Bhatia
Lead Systems Administrator
TotalChoice Hosting, LLC.
http://www.totalchoicehosting.com

#11 TCH-Vivek

TCH-Vivek

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,814 posts

Posted 07 October 2007 - 03:58 AM

47%
Warmest regards,
Vivek Bhatia
Lead Systems Administrator
TotalChoice Hosting, LLC.
http://www.totalchoicehosting.com

#12 TCH-Vivek

TCH-Vivek

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,814 posts

Posted 07 October 2007 - 04:26 AM

63%
Warmest regards,
Vivek Bhatia
Lead Systems Administrator
TotalChoice Hosting, LLC.
http://www.totalchoicehosting.com

#13 TCH-Vivek

TCH-Vivek

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,814 posts

Posted 07 October 2007 - 04:42 AM

72%
Warmest regards,
Vivek Bhatia
Lead Systems Administrator
TotalChoice Hosting, LLC.
http://www.totalchoicehosting.com

#14 TCH-Vivek

TCH-Vivek

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,814 posts

Posted 07 October 2007 - 04:57 AM

80% done
Warmest regards,
Vivek Bhatia
Lead Systems Administrator
TotalChoice Hosting, LLC.
http://www.totalchoicehosting.com

#15 TCH-Vivek

TCH-Vivek

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,814 posts

Posted 07 October 2007 - 05:25 AM

90% done.
Warmest regards,
Vivek Bhatia
Lead Systems Administrator
TotalChoice Hosting, LLC.
http://www.totalchoicehosting.com

#16 TCH-Vivek

TCH-Vivek

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,814 posts

Posted 07 October 2007 - 06:26 AM

Restore is over. We have also synced the mysql databases and mails with the old box to make sure things are up to date. The new box is now up as tarvis and everything looks fine. We are still monitoring the server. Should you have any issues please raise a helpdesk ticket. Thank you all for your co operation with us while we were working on it.
Warmest regards,
Vivek Bhatia
Lead Systems Administrator
TotalChoice Hosting, LLC.
http://www.totalchoicehosting.com




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users