On Friday, February 14th, at 08:22 ET, clients reported that Circulations and the Primary Website field disappeared after publishing an article in Composer V2. This prevented users from saving stories, leading to disruptions in their publishing workflows.
On Friday, February 14th, at 08:22 ET, multiple users reported issues with stories failing to load properly. Key information, including the Primary Website and selected Circulations, was missing. Some users only discovered the issue when attempting to save their stories, which were blocked due to the missing required fields.
At 10:48 ET, the engineering team identified the root cause: a slow network response caused delays in retrieving origin. This delay triggered a race condition, preventing the Primary Website and circulations data from being properly loaded.
To resolve the issue, the Arc XP engineering team updated the front to properly wait for an initial API response before rendering story data. This ensured that all necessary metadata was available, even in cases of slower network requests. By 16:56 ET, users confirmed that the issue was fully resolved.
All times ET + 24 hour clock
| Time | Event |
|---|---|
| 08:22 | Initial customer report |
| 10:48 | The root cause is identified and hotfix development begins. |
| 15:21 | The deployment of the hotfix commences. |
| 16:27 | The hotfix deployment is successfully completed. |
| 16:56 | Users have reported that the issue has been successfully resolved. |
The team will conduct a post-mortem analysis to identify opportunities for improving the resilience of the Settings API. Additionally, the engineering team will review the frontend infrastructure to address potential race conditions and implement safeguards to prevent similar issues in the future.