Global outage

Incident Report for Arc XP

Postmortem

Customer Impact

On Monday, October 20th, 03:11 AM EST, ARC’s AWS reported a global outage, which affected several of ARC API’s functionality, resulting in impaired availability to customers in the us-east-1 region.

Root Cause

AWS reported elevated error rates in DNS resolution of the DynamoDB API endpoint, network connectivity issues, launch failures in virtual servers, and intermittent function errors while making network requests to other services.

Timeline

All times ET + 24 hour clock

Time	Event
03:11	Increased error rates and latencies for multiple services
03:47	Confirmation of Global Outage in the main Infrastructure Provider
05:04	Outage continues with the provider identifying the potential root cause
05:55	ARC Status Page update with systems performance
07:17	Systems are functional momentarily
09:44	The systems were partially recovered
10:28	The platform report elevated errors again. Confirmation on the provider’s system status.
11:40	ARC Status Page update advising outage is still affecting our APIs.
13:17	Availability continues to be spotty
14:39	Any deployment or code change is frozen to avoid any further issues for ARC’s customers. ARC’s Infrastructure Provider advises 2+ hours for full recovery
16:22	Systems are still performing under stress and availability is limited.
16:40	Services start to recover
18:15	Confirmation that all systems have been fully recovered.

Action Items

We are reviewing and updating our Business Continuity and Disaster Recovery procedures in light of this outage.

Posted Oct 29, 2025 - 08:48 EDT

Resolved

All services have been fully recovered.
Video Center services are back to normal state, but all source systems should be double-checked to confirm successful reconnection.

Posted Oct 20, 2025 - 18:25 EDT

Update

Most services have been fully recovered. Video is still experiencing impaired functionally in some of its features, and it is continuing to process backlogs.

Posted Oct 20, 2025 - 18:15 EDT

Update

We are observing recovery in some of our applications. Page Builder deployment and IFX functionality are restored. Changes sent to those applications will possibly need to be retried.

Videos are still being processed in the backlog. And other APIs might still experience service interruptions.

We will continue providing feedback and updates.

Posted Oct 20, 2025 - 17:23 EDT

Update

In addition to previous reported services, we can confirm that IFX was severely impacted. We are observing partial recovery, with some integrations now restarting.

We are also seeing slow recovery for multiples other services: videos are being processed, but the backlog is significant, and will take time to process.

While access to Arc XP apps and APIs is improving, there are still errors, in particular at integration points, which continues to cause issues with updates not always propagating from one place to another. Our guidance remains to limit changes as much as possible for the time being.

We will continue providing updates as the situation evolves.

Posted Oct 20, 2025 - 16:40 EDT

Update

In addition to previous reported services, we can confirm that IFX was severely impacted. We are observing partial recovery, with some integrations now restarting.

We are also seeing slow recovery for multiple other services: videos are being processed, but the backlog is significant, and will take time to process.

While access to Arc XP apps and APIs is improving, there are still errors, in particular at integration points, which continues to cause issues with updates not always propagating from one place to another. Our guidance remains to limit changes as much as possible for the time being.

We will continue providing updates as the situation evolves.

Posted Oct 20, 2025 - 16:28 EDT

Update

We are continuing to observe significant service degradation across multiple Arc services in the US-East-1 region:

Access to several Arc UI is impacted and may be non-functional for some customers, with PageBuilder editor being the most affected.

There may be issues saving changes in PageBuilder, and saved changed may not reflect on websites, as propagation between the different serving layers is also impacted.

All video encoding services are down.

In addition, access to underlying APIs through Trident is also affected, which results in occasional errors for both APIs and UIs.

We are advising customers in this region to not deploy any new bundle.

Due to the extent of the outage and the number of services affected, we do not have an ETA at this time for full recovery of all services. We will continue to provide regular updates as information becomes available.

Posted Oct 20, 2025 - 14:56 EDT

Update

Our cloud provider outage continues and some services are degraded. We continue to monitor the situation and take corrective actions when possible.

There is no direct workaround, and availability may occasionally be spotty depending on the service and time.

We will continue providing updates as the situation evolves.

Posted Oct 20, 2025 - 13:25 EDT

Update

We are continuing to monitor for any further issues.

Posted Oct 20, 2025 - 13:25 EDT

Update

We are currently experiencing intermittent issues affecting multiple services. These disruptions appear to be related to an ongoing outage with our cloud provider. Our teams are actively working to restore full functionality and are closely monitoring the situation.

If you encounter any issues, please wait a few minutes and try again. We will continue to share updates as the situation evolves.

Posted Oct 20, 2025 - 12:16 EDT

Update

We are continuing to monitor for any further issues.

Posted Oct 20, 2025 - 12:16 EDT

Update

We are observing sporadic issues on several APIs, most notably WebSked, though others may be affected as well. Those tend to resolve within a few minutes, and appear to remain linked to our cloud provider outage. We are continuing to actively monitor the situation. If you experience issues within any of our apps, wait a few minutes and try again.

We will continue to provide updates as the situation evolves.

Posted Oct 20, 2025 - 10:35 EDT

Update

We are continuing to monitor for any further issues.

Posted Oct 20, 2025 - 10:34 EDT

Update

We are now seeing partial recovery for Jira was well, and many customers should be able to post tickets.

As a reminder, the recommended course of action for any failed operation during the outage is simply to retry. Saving/republishing stories will fix most editorial issues. Similarly, some stories that were scheduled for publishing may have missed their publishing deadlines. Re-publishing them manually should work as well.

If you continue to experience any other issue at this time, please feel free to contact our support via our regular channels.

Posted Oct 20, 2025 - 09:46 EDT

Update

At this time, Arc XP systems are considered functional, and we've seen recovery across the board. It appears our ticketing platform is still experiencing issues however. If you are in need of urgent assistance, we recommend reaching out to your TAM through the usual channels, which can then escalate internally.

We will continue to provide updates on a regular basis.

Posted Oct 20, 2025 - 07:18 EDT

Monitoring

We are observing recovery across multiple services and are continuing to actively monitor the situation. At this stage, we recommend retrying any failed action. It may take some time still for all services to be operating normally. We will continue to provide updates as they become available.

Posted Oct 20, 2025 - 05:59 EDT

Update

There are no updates at the moment; the system is still down.

Posted Oct 20, 2025 - 05:07 EDT

Investigating

We are observing a large global outage in our main infrastructure provider (AWS).
Several services (UI and API) may be affected.
Our ticketing system (JIRA) may also be affected by the same outage, which limits the ability to post support tickets.

Posted Oct 20, 2025 - 03:58 EDT

This incident affected: Creator Apps (Video Center).