On Thursday, February 6th, from 13:46 ET to 14:09 ET, Site Service experienced elevated latency. This disrupted client site navigation from successfully loading and failures in Composer and WebSked when loading content.
The elevated latency in the service was caused by a sudden increase in traffic and the service's inability to scale up sufficiently to handle the load. Capacity was restored, and the service returned to normal operation. The cause of the traffic spike is still under investigation.
All times ET + 24 hour clock
Time Event 13:42 Traffic Increase 13:46 Automated alerts notify engineering team of the elevated latency 13:46 Engineering team begins investigating 13:56 Engineering team increases the service capacity 14:02 Engineering team restricts traffic so service can recover 14:02 Clients notify Arc that there are issues with the platform 14:05 Service recovers 14:09 Engineering team restores traffic and latency returns to normal levelsAutoscaling configurations will be updated for Site Service to ensure that its capacity is not limited when handling traffic spikes.