Ordering Issues

Incident Report for Redbox Systems

Postmortem

What Exactly Caused This Issue?

At 19:05 on Tuesday 3 September, customer ordering services experienced disruption. As load increased during peak trading, reduced memory availability led to service degradation and failure to load ordering clients for some users.

The issue was caused by a configuration change introduced earlier in the day, which unintentionally set a memory limit below the required threshold for the ORM layer. This change was part of an effort to improve observability of resource metrics and transition from manual resource definitions to centrally managed configuration.

Impact

  • Affected Service(s): Customer Ordering
  • User Impact: Ordering clients failed to load during the affected window.
  • Duration: 19:05 – 19:50

Resolution

  • The misconfigured memory limit was identified and corrected.
  • Memory allocations were adjusted manually across affected containers.
  • Service availability returned to normal shortly after.
Posted Sep 04, 2025 - 08:36 BST

Resolved

This incident has been resolved.
Posted Sep 03, 2025 - 20:18 BST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Sep 03, 2025 - 19:57 BST

Investigating

We are currently investigating this issue.
Posted Sep 03, 2025 - 19:19 BST
This incident affected: Redbox Platform (Customer Services).