Learners not able to interact with H5P.

History

At 2:30 pm UTC, the Xen team investigated the issue and tried to reproduce it.

At 3:30 PM UTC, the Xen team reverted a change then re-deployed to production in order to restore functionality to the learners.

At 4:30 PM UTC the deployment was completed.

At 5:55 PM UTC, the customer was informed that the issue had been resolved.

Why did this happen?

The incident arose due to a misstep in the tagging process intended for production, which allowed a development branch item to be deployed in production.

What was the incident’s duration?

The incident lasted 3 hours, from discovery to remediation.

Were there any consequences or damages?

Some learners were in the North America region not able to interact with H5P content.

What will prevent further similar incidents?

To prevent similar incidents in the future, we have introduced additional status checks for all production deployments.

Specifically:

• Developers check that the custom app tags are created and merged from the main branch.
• Developers must cherry-pick the ready-for-release card PRs for the production deployment and merge them into the main branch.
• Additional status checks are now performed before any production deployment to confirm all changes are correctly tagged and merged.