On Monday, January 27th, from 16:03 ET to 16:37 ET, some clients experienced increased 404 responses that resulted in blank spaces or content that could not be loaded in their websites.
On Monday, January 27th as 16:03 ET, a scheduled release to Sandbox impacted a Production configuration. The misconfiguration resulted in traffic being routed to the incorrect environment resulting in the content not to be found. Once the correct configuration was restored, traffic responses and system returned to its normal state.
All times ET + 24 hour clock
| Time | Event |
|---|---|
| 16:03 | Sandbox deployment is started |
| 16:24 | Issue is reported by clients |
| 16:34 | Engineering team is alerted about the incident |
| 16:34 | Engineering team restores the correct configuration |
| 16:38 | System fully restored |
Automation will be created to avoid manual intervention when updating production configurations for Content API.
Extra monitoring will be added to alert on increases in failures for rendering.