On Wednesday, January 15th, 03:17 EST, Composer users encountered errors when attempting to save or publish content. The error message, "Missing code formatting," was triggered in scenarios involving the addition of underline colors to text or hyperlinks in stories. This issue prevented all users from progressing with their content workflows, significantly impacting their ability to publish articles.
On January 15th, 03:17 EST, a recently deployed update to the content sanitization logic in Composer 2.0 caused valid content, such as underline color formatting and hyperlinks, to be flagged erroneously as requiring code blocks. This resulted in a P0 incident where users were unable to save or publish their work due to these false positives.
To immediately resolve the issue, our engineers disabled the sanitization logic altogether, restoring users' ability to save and publish content while we investigate and refine the sanitization process to prevent such occurrences in the future.
All times ET + 24 hour clock
| Time | Event |
|---|---|
| 04:44 | P0 incident reported across all users, with errors blocking saving/publishing content. |
| 05:43 | Arc XP Engineering team identified the root cause was with a change to the code sanitation logic in Composer 2.0. |
| 06:12 | Arc XP Engineering began deploying a hotfix to remove the code sanitation logic. |
| 06:44 | Hotfix successfully deployed to Sandbox. |
| 07:14 | Hotfix successfully deployed to Prod. |
We are conducting a thorough review and re-testing of the sanitization logic to identify and address the issue that caused valid content to be flagged erroneously. Once the updated sanitization logic has been refined and validated, it will be reintroduced into production. Additionally, we are evaluating our deployment and testing processes to prevent similar issues from recurring in the future. Our goal is to ensure content accuracy, security, and seamless functionality for our users.