Photo Center UI

Incident Report for Arc XP

Postmortem

Customer Impact

On Wednesday, October 22, 2025, between ~12:50–13:30 ET, some customers in the US and EU regions experienced elevated errors and timeouts from the Photo Center API following a production upgrade. This appeared in the UI as broken upload modals and images not loading in Photo Center. Image delivery to end users was not affected. Service returned to normal after the upgrade was rolled back.

Root Cause

A platform upgrade changed application behavior and led to database connection pool exhaustion in the Photo Center API, causing requests to queue and time out. Rolling back the deployment in affected regions—and then across remaining regions as a precaution—restored normal performance by ~13:30 ET.

Timeline

All times ET + 24 hour clock

Time	Event
12:45	Upgrade deployment started in US and EU regions.
12:50	Alerts and customer reports of API errors/timeouts begin.
13:00	Investigation underway
13:18	Rollback to the prior version initiated in affected regions.
13:28	Rollback completed; service stabilized and alerts cleared.
13:30	Proactive rollback initiated in remaining regions; incident resolved

Arc XP Next Steps

Stability & Capacity: Right-size database connection pool limits and validate database headroom; add safeguards to prevent saturation.
Deployment Safety: Adopt a safer blue-green strategy for next rollout
Observability & Runbooks: Enhance pre/post-deploy checks, saturation alerts, and regional synthetic tests for key Photo Center workflows.

Posted Oct 29, 2025 - 08:50 EDT

Resolved

The issues affecting the Photo Center UI have been fully resolved. Service has been restored across all regions, and the application is operating normally.

We will continue to monitor performance to ensure stability. Thank you for your patience while we worked to resolve this incident.

Posted Oct 22, 2025 - 13:35 EDT

Update

We’ve identified the root cause of the Photo Center UI issues and have implemented a fix. Service restoration is in progress, and availability should return to normal across various regions shortly.

We continue to monitor performance and will confirm once the issue is fully resolved.

Posted Oct 22, 2025 - 13:28 EDT

Identified

We are currently investigating issues affecting the Photo Center user interface. Some users may experience errors accessing certain features.

Our engineering teams are actively working to identify the root cause and restore normal functionality as quickly as possible.

We will provide updates as more information becomes available.

Posted Oct 22, 2025 - 13:22 EDT