ROOT CAUSE ANALYSIS – External Network Creation Outage
On Thursday May 10th, 2012, from 3:20PM PST to 7:50PM PST, network creation and user sign-up were not working. Users received a “We’re sorry” page, and were unable to complete sign-up or External Network creation.
At 7:00PM PST it was determined that the primary key sequence on the networks table was incorrect due to a hotfix that was applied the prior evening to resolve billing issues. The sequence was resolved by 7:50PM PST and customers were again able to perform network creation and user sign-up. No customer data was impacted during this period.
We have put in place additional checks after performing hot-fixes to minimize impacts to production.