Jump to content

Recommended Posts

At around 5:12AM EST today we experienced a routing issue between our main cisco router and our upstream provider, causing all our traffic between our core router and our upstreams to fall into a unstable status. After initial investigation we were able to determine the issue was not internal and we were required to contact our upstream providers.


After contacting our upstream we learned the issue was due to a misconfiguration on the route handoffs for our IP blocks. This was corrected and our traffic started to resume to normal levels.


This event is highly unusual and should not have happened. However as we have been growing in our tchnoc we have been using some very advanced routing tables to route our traffic. This was due to the fact that we were not fully routing all of our own layer 3 traffic. A majority of the traffic was being routed by the upstream and they were passing the traffic to us. Back in the middle of september TCH made a major investment and purchased our own Cisco layer 3 routing equipment and we had been in the testing phases for some time. We had planned on taking the hand off of all our layer 3 traffic on Decemeber 1. It was my view that we simply wanted to test the equipement fully and then implement it. I wanted to be fully sure that things were setup correctly.


Over the past month we have been segmenting our IP block into smaller segments. We have been working hard to complete this transition, with the end result being better perfomance and quality for our network. Now that this transition is complete, we are already seeing improved performance overall.


Long story short our upstreams handed off our direct route to us for our IP block. So that huge Cisco router we bought is now in full operation.


I hope that this clears up our short outage from this morning and I assure you that we are working hard to provide the best possible service that we can provide.

Link to comment
  • Create New...