On June 17, 2025, clients using Photo Center in the us-east-1 region experienced a service disruption due to a planned database upgrade. During this period, Photo Center was unavailable, preventing users from accessing or managing photos.
The outage stemmed from unexpected technical issues during a planned database upgrade in us-east-1, which caused temporary read and write failures on the Photo Center API. Service was restored after we rolled back the upgrade, and the system returned to a healthy state.
This upgrade was a follow-up to the upgrade that triggered the previous week’s Photo Center outage. Although this type of upgrade typically incurs no downtime, Arc XP encountered recurring issues in us-east-1(our largest region) where the connection pool to our primary database became unexpectedly exhausted. While the same upgrade was deployed successfully in other regions, instability persisted in us-east-1, ultimately requiring a second rollback to restore service.
All times ET + 24 hour clock
| Time | Event |
|---|---|
| 22:10 | Database upgrades in us-east-1 began. |
| 22:25 | Photo Center became unavailable in us-east-1. |
| 22:30 | Arc XP identified similar symptoms to the prior outage and began mitigation efforts. |
| 23:15 | Photo Center service was restored following a rollback. |
Because the rollback consistently resolved the issue, we’ve updated our operational procedures to prioritize immediate rollback in response to similar instability, ensuring faster recovery times in the future.
Additionally, we identified under-scaling as the root cause for the recent upgrade failures in us-east-1. To mitigate this risk in future upgrades, we're implementing a blue/green deployment strategy to validate the upgrade in a Production-like environment without affecting any live services.
We're also deployed a hotfix during non-business hours on June 25, 2025 to make the service more resilient for our next database upgrade on June 27, 2025 at 1:00 AM ET.
Due to this outage, the Production release of Photo Center 1.83.2 has been delayed until July 2, 2025.