This prior deployment was tested in both a development and staging environment and had been deployed to production for 3 hours without issue before the incident began. However, due to a flaw in the new webpack static asset caching between builds, the next (unrelated) deployment led to deploying a broken static asset manifest.
Once deployed, we immediately spotted the issue and started working to rollback the problematic code. Within about 15 minutes some new loads of Close.io began working properly. Within about 30 minutes, all servers had been fully reverted to non-broken code and the incident was resolved.
We're extremely sorry for this outage and are doing everything we can to help make sure this type of problem doesn't occur again. These action items include: