EXTERNAL NETWORK CREATION OUTAGE – Root Cause Analysis 5/11/12

ROOT CAUSE ANALYSIS – External Network Creation Outage

Service Disruption:
On Thursday May 10th, 2012, from 3:20PM PST to 7:50PM PST, network creation and user sign-up were not working. Users received a “We’re sorry” page, and were unable to complete sign-up or External Network creation.

Root Cause:
At 7:00PM PST it was determined that the primary key sequence on the networks table was incorrect due to a hotfix that was applied the prior evening to resolve billing issues. The sequence was resolved by 7:50PM PST and customers were again able to perform network creation and user sign-up. No customer data was impacted during this period.

Mitigation Steps:
We have put in place additional checks after performing hot-fixes to minimize impacts to production.

This entry was posted in Uncategorized. Bookmark the permalink.