Issues accessing
Incident Report for

On 12/29/16 we started to observe workloads that hadn’t been seen before. A large set of anomalous requests caused our caching and queue processing system to run out of memory. Alarms alerted us of the situation but we were unable to stop the memory growth before all of the resources were exhausted. We then proceeded to safely clear the cache data, freeing memory and restoring access to the application. Unfortunately, this was a manual process and the application was unavailable from 10:32 AM PST to 11:44 AM PST while we worked to restore service. API access remained unaffected during this time period.

We are extremely sorry for this downtime and know that you count on being reliable for your sales needs. Here's what steps we've taken and will take to help mitigate this issue:

  • Implement an automated failsafe system to spool backend tasks when the caching system is low on memory.
  • Isolate foreground caching from background queue processing.
  • Improve the scalability of our backend email syncing subsystem.
Posted over 1 year ago. Jan 04, 2017 - 14:33 PST

We have resolved the issues with accessing the application. Anyone who encountered issues with the desktop client should close and reopen the application and users of the web UI will need to reload their browser.
Posted over 1 year ago. Dec 29, 2016 - 17:08 PST
The application is available again, however email syncing and webhooks are still delayed for some customers.
Posted over 1 year ago. Dec 29, 2016 - 11:51 PST
We are currently clearing out a backend queue to recover access to the application. Some webhooks and synced emails may be delayed.
Posted over 1 year ago. Dec 29, 2016 - 11:41 PST
We are currently investigating an issue with accessing the application
Posted over 1 year ago. Dec 29, 2016 - 10:50 PST