ROOT CAUSE ANALYSIS; Yammer Loading Issues; August 18 & 20, 2014

At 6:00AM, PDT, on August 18, some Yammer users experienced issues loading their networks and feeds. This impacted users beginning fresh Yammer sessions, especially. During this time, the Yammer on-call team observed that requests being made to our assets servers had unexpected pauses of a second or more while data were being returned. As a result, our content delivery network (CDN) was returning incomplete assets and Yammer content could not be consistently loaded. This issue cleared up without direct intervention from the on-call team, but the situation was monitored from this point on. The on-call team posted the all-clear at 8:20AM, PDT.

On August 20 at 6:39AM, PDT, an elevated server load was again detected, resulting in slow loading times for Yammer feeds and pages. The platform recovered again at 8:00AM, PDT.

Following further investigation into these two incidents, it was determined that this was due to some asset compression at the load balancer, which was causing inconsistent asset caching behavior at the CDN. This resulted in the CDN making multiple requests for the same content, which then built up and could not be returned.

To avoid a reoccurrence of this in the future, the Yammer Engineering team has disabled the asset compression that was identified as the root cause of this behavior. Additionally, the routing rules for some Yammer services were rerouted from the main load balancer to reduce the load on it. New alerting mechanisms have also been implemented to catch load balance risks sooner.

There will be no further updates on this incident.

Posted in Uncategorized

UPDATE: FILE UPLOAD ISSUES [22/08/2014][3:43 PDT]

THIS ISSUE HAS NOW BEEN FULLY RESOLVED BY OUR ON-CALL ENGINEERING TEAM

Thank you for your patience

Posted in Uncategorized

FILE UPLOAD ISSUES [22/08/2014] [3:02 PDT]

YAMMER IS CURRENTLY FACING ISSUES WITH FILE UPLOADS OF A CERTAIN TYPE.

CURRENTLY AFFECTED FILE TYPES INCLUDE, BUT NOT LIMITED TO WORD DOCUMENTS (.docx) AND PDFs

[ANY FILE WHICH PREVIEWS IN YAMMER IS AFFECTED]

Users receive a HTTP ERROR on upload.

Our On-call engineers are currently investigating thisĀ and are working to bring about a speedy resolution.

Thank you for your patience while we deal with this.

Posted in Uncategorized

[RESOLVED] Slow Yammer Load Times; 08/20/2014; 8:00AM, PDT

The Yammer on-call team has observed a large enough period of platform stability and are calling this incident resolved. We continue to investigate the root cause, but believe it was a networking issue triggered by a routine daily deployment. Further deployments are on hold until the root cause is confirmed. No further updates.

Posted in Uncategorized

[UPDATE] Slow Yammer Load Times; 08/20/2014; 7:39AM, PDT

The Yammer on-call team continues to works to resolve the errors we’re seeing that are causing loading issues across Yammer networks. More updates to follow.

Posted in Uncategorized

SLOW YAMMER LOAD TIMES; 08/20/2014; 6:39AM, PDT

Following a deployment this morning, Yammer networks are experiencing some unusually slow load times. Our on-call team is working to get this resolved as quickly as possible. More updates to follow.

Posted in Uncategorized

[RESOLVED] Feed Loading Issues; 08/18/2014; 8:36AM, PDT

This morning’s issue with loading Yammer feeds is now resolved. Affected users may need to perform a hard refresh (cache flush) in their browsers to get their feeds to load correctly again. A Root Cause Analysis for this incident will be posted here within seven days.

Posted in Uncategorized