Degraded Performance
Incident Report for Close
Postmortem

On October 13th we rolled out some backend changes as part of some soon-to-launch features.

These changes added unexpected strain on the database and the search indexer, and impacted Close.io performance. While performance was fine during much of the past 8 days, we also had times with significantly degraded performance that resulted in a poor user experience for many of our users.

We did a few things during this time in an attempt to mitigate the problem, such as temporarily disabling syncing of email accounts during some peak hours, and running some other migrations only during off-peak hours.

Ultimately we identified/fixed some performance bottlenecks, made significant hardware upgrades to our database server, and put a plan in place to ensure faster rollback of performance-hurting deployments in the future.

We believe that having an incredibly fast application is one of the most important features Close.io can have. We're sorry the last 8 days haven't reflected that priority.

Finally, in an attempt to have increased transparency around uptime and performance, and to better communicate during any future incidents, we have launched a new status page at http://status.close.io.

Posted Oct 21, 2013 - 16:16 PDT

Resolved
Recent performance issues of data loading (API), search, and email syncing are all resolved. See postmortem for more details.
Posted Oct 21, 2013 - 12:00 PDT