Issues accessing app.close.io
Incident Report for Close
Postmortem

On 12/29/16 we started to observe workloads that hadn’t been seen before. A large set of anomalous requests caused our caching and queue processing system to run out of memory. Alarms alerted us of the situation but we were unable to stop the memory growth before all of the resources were exhausted. We then proceeded to safely clear the cache data, freeing memory and restoring access to the Close.io application. Unfortunately, this was a manual process and the application was unavailable from 10:32 AM PST to 11:44 AM PST while we worked to restore service. API access remained unaffected during this time period.

We are extremely sorry for this downtime and know that you count on Close.io being reliable for your sales needs. Here's what steps we've taken and will take to help mitigate this issue:

  • Implement an automated failsafe system to spool backend tasks when the caching system is low on memory.
  • Isolate foreground caching from background queue processing.
  • Improve the scalability of our backend email syncing subsystem.
Posted Jan 04, 2017 - 14:33 PST

Resolved
We have resolved the issues with accessing the close.io application. Anyone who encountered issues with the desktop client should close and reopen the application and users of the web UI will need to reload their browser.
Posted Dec 29, 2016 - 17:08 PST
Monitoring
The close.io application is available again, however email syncing and webhooks are still delayed for some customers.
Posted Dec 29, 2016 - 11:51 PST
Identified
We are currently clearing out a backend queue to recover access to the application. Some webhooks and synced emails may be delayed.
Posted Dec 29, 2016 - 11:41 PST
Investigating
We are currently investigating an issue with accessing the close.io application
Posted Dec 29, 2016 - 10:50 PST