Yammer Site Status

ROOT CAUSE ANALYSIS – External Network Creation Outage

Service Disruption:
On Thursday May 10th, 2012, from 3:20PM PST to 7:50PM PST, network creation and user sign-up were not working. Users received a “We’re sorry” page, and were unable to complete sign-up or External Network creation.

Root Cause:
At 7:00PM PST it was determined that the primary key sequence on the networks table was incorrect due to a hotfix that was applied the prior evening to resolve billing issues. The sequence was resolved by 7:50PM PST and customers were again able to perform network creation and user sign-up. No customer data was impacted during this period.

Mitigation Steps:
We have put in place additional checks after performing hot-fixes to minimize impacts to production.

Dear Customer,

At Yammer we are committed to making our infrastructure more resilient and scaleable. To this end, on Saturday May 19th from 11AM-1PM PST, Yammer will be performing a major database upgrade which will require approximately 2 hours of downtime. All precautions have been taken to ensure downtime will be as minimal as possible and no customer data will be lost.

Sincerely,
Yammer

All is well.

The third party Yammer is using for Documents is currently experiencing some issues. We are in close communication with them, so I will be with you all.

This will not impact all of the other parts of Yammer in the least bit.

YAMMER OUTAGE 04/02 – Root Cause Analysis

Updated code was released on Friday night, which while tested, created some performance problems which only manifested during our peak traffic hours. The code in question dramatically increased requests to our primary databases.

Starting at ~5:30am PST, this increased database load began creating slow response times for all of Yammer.com, sometimes timing out completely making yammer.com unavailable. All key personnel were notified and mobilized shortly after our monitoring caught these performance issues.

At ~7:30am PST the aberrant code was identified and reverted, which immediately yielded improved response times. Performance did not return to fully nominal until 8:29am PST after the backlog of requests had been satisfied.

Increased monitoring and changes to how code is deployed are being planned in order to mitigate these types of widespread performance depredations in the future.