Readers of sites of Arc XP clients provisioned in the AWS us-east-1 received 403 errors on about 19% of requests when browsing pages and related assets. Services such as API delivery (including Subscriptions API and View API), and video delivery were not affected.
Arc XP relies on a centralized cache service within its CDN provider to ensure consistent content delivery across cache nodes serving end users. This cache is a shared (multi-tenant by our CDN provider) service and a surge in traffic from an attack against an Arc XP customer exceeded a service limit from the CDN provider.
When limit was hit, the centralized cache returned 403 (forbidden) status codes to requests issued to parent and edge servers that did not already have the requested content stored locally. This led to failed requests for end users until the surge subsided.
Based on a previous incident the resource limit had been increased by about 5x in order to prevent reoccurrence in normal situations while Arc XP worked to migrate to a dedicated tenant with 100x capacity within our CDN provider. The request rate was significantly larger than the previous incident and even though WAF policies mitigated the majority of the traffic the remaining volume was larger than the previous increase permitted.
All times ET + 24 hour clock
| Time | Event |
|---|---|
| 22:00 | Attack begins triggering rate limits in central cache |
| 22:50 | Attack ends, return to normal state |
Arc XP is working with its CDN provider to utilize a different method for shifting to a dedicated tenant (100x capacity). We have also shifted some rate policies to make them more effective in these sort of attack situations.