Customer Impact
On Tuesday, January 14th, Pagebuilder Deployer 2.1 was released to customer sandbox environments in us-east-1. Affected customers were unable to deploy bundles in sandbox environments. The specifics of the impact as well as the root cause varied depending on the circumstances for each affected customer.
Root Cause
- Secondary Orgs: Some configuration updates to route customer secondary orgs were missed and resulted in the PageBuilder Deployer incorrectly loading the bundles and deployments for the primary org.
- Non-standard Environments: Some configuration updates to include non-standard environments including
outboundfeeds and pbeditor were missed and resulted in the PageBuilder Deployer not being able to access Engine releases.
- Missing bundles from those Deployed: PageBuilder Deployer 2.1 changed the sorting for deployments and enforced the 20 bundle limit so environments with more than 20 concurrent deployments experienced an issue where new deployments were not visible in the list.
- Themes Bundles Not Deploying: PageBuilder Deployer 2.1 enforces the two Engine Version Availability policy, but Themes still allows deployment using older versions of Engine, which caused deployment errors in Theme Settings. The recommendation for those customers is to upgrade to Themes Version 2.4.1 or newer.
- Showing Only Old Uploaded Bundles: The stated retention policy for uploaded bundles is to keep the last 90 days for unused bundles. However, this retention policy was not enforced and allowed environments to store over 1K bundles. In this scenario, the bundles were loading too slowly and caused newer bundles to not load. The recommendation was to delete non-deployed bundles older than 90 days. We offered to delete these bundles for the customers.
Timeline
All times ET + 24 hour clock
| Time |
Event |
| ~9:30 |
Deploy Deployer 2.0 |
| 12:02 |
ACS begins receiving reports relating to PageBuilder Deployer in Sandbox |
| 13:29 |
Arc Engineering team notified and begins investigation |
| ~14:00 |
Update trident config to fix secondary orgs |
| 15:55 |
All tickets resolved |
Arc Next Steps
1./2. Configurations for secondary orgs and non-standard environments have been implemented.
- Missing deployments in the UI: We are updating the UI to sort from latest to oldest so if a customer has over 20 deployments then the oldest non-live deployments drop from the list
- Themes customers should update to a newer version of Themes.
- Showing Only Old Uploaded Bundles: Creating a script to delete non-deployed bundles older than 90 days.