On Wednesday, January 15th, from 2:02AM to 3:36AM, a component failure resulted in story changes failing to propagate to Content API. The affected documents were not available for rendering in Composer or WebSked. Once the propagation method was fixed, all queued updates were successfully processed.
The root cause was identified as a misconfiguration in the link between Draft API and Content API. This was the result of an infrastructure and configuration change. It was not previously identified as it required additional conditions to be triggered, which occurred today. Once the root cause was found, the misconfiguration was corrected and service restored.
All times ET + 24 hour clock
| Time | Event |
|---|---|
| 02:02 | Link between Draft API and Content API stops propagating updates in 1 region |
| 02:37 | First alert triggers |
| 03:20 | Root cause is identified |
| 03:33 | Hotfix issued |
| 03:37 | Service restored |
Arc will evaluate the deployment process for improvements related to configuration changes and review the affected alerts to increase alerting speed.