Head Guru Posted July 1, 2005 Posted July 1, 2005 I wanted to bring this issue to everyone's attention. This only affected clients using NS1 and NS2.totalchoicehosting.com Tonight at approximately 8:25PM EST our master DNS server stopped responding to DNS queries. Upon initial investigation we determined that the server had crashed and required a hard reset. We were not concerned about this outage as our secondary DNS server was online and resolving DNS, so there should not have been any issues. However, this was a incorrect assumption on our part. A very small percentage of clients' domains stopped resolving when our master DNS server was taken off line for a reboot and disk check. We were unaware of this until clients contacted us about their downed sites. Of course we were shocked to hear this news, as the secondary DNS server should have been resolving the domains. However, this was not the case. The master DNS outage lasted approximately 40 minutes. This was due to the server running a FSCK upon reboot. We estimate that around 2,000 domains did not resolve during this outage. They were not resolving from around 8:25PM EST to 9:10PM EST. All DNS servers are up and the issue has been fully corrected. Our initial investigation has revealed a simple mistake on our part. In early January our DNS clustering system was down for several days due to maintenance. Also again in March and again just last week. During this time any DNS updates, such as, new accounts, moved accounts, sub domains etc, were only written to our master DNS. We failed to realize this until just tonight. Honesty is the best policy and there it is. We have corrected this issue and all DNS is up and working on both the master and secondary. Thank you for your time. Quote
TCH-RobertM Posted July 1, 2005 Posted July 1, 2005 As a follow up with Bill's Post We are setting up a process that will prevent this type of issue in the future. We will be periodically syncing up the DNS Servers and validating the contents on a more frequent basis. I would also like to thank everyone for their support and patience during this issue. Quote
Dumplin Posted July 2, 2005 Posted July 2, 2005 I wanted to bring this issue to everyone's attention. This only affected clients using NS1 and NS2.totalchoicehosting.com Tonight at approximately 8:25PM EST our master DNS server stopped responding to DNS queries. Upon initial investigation we determined that the server had crashed and required a hard reset. We were not concerned about this outage as our secondary DNS server was online and resolving DNS, so there should not have been any issues. However, this was a incorrect assumption on our part. A very small percentage of clients' domains stopped resolving when our master DNS server was taken off line for a reboot and disk check. We were unaware of this until clients contacted us about their downed sites. Of course we were shocked to hear this news, as the secondary DNS server should have been resolving the domains. However, this was not the case. The master DNS outage lasted approximately 40 minutes. This was due to the server running a FSCK upon reboot. We estimate that around 2,000 domains did not resolve during this outage. They were not resolving from around 8:25PM EST to 9:10PM EST. All DNS servers are up and the issue has been fully corrected. Our initial investigation has revealed a simple mistake on our part. In early January our DNS clustering system was down for several days due to maintenance. Also again in March and again just last week. During this time any DNS updates, such as, new accounts, moved accounts, sub domains etc, were only written to our master DNS. We failed to realize this until just tonight. Honesty is the best policy and there it is. We have corrected this issue and all DNS is up and working on both the master and secondary. Thank you for your time. <{POST_SNAPBACK}> Well I didn't notice a thing......we had really nasty storms come thru with over 2000 hits of lightning per hour so I was shut down and unpluged<G> but I appreciate your honesty and I will continue to be a happy customer for as long as your providing Webhosting Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.