How We Diagnosed a Hidden Scheduler Failure in a Docker Swarm Cluster Serving 2 Million Users

Context: 120 Nodes, Strict SLAs, and Legacy Infrastructure

Our team is responsible for the mobile backend infrastructure serving over 2 million registered users. The , and Swarm is currently sufficient for the current scale. Services are distributed across nodes in groups and bound by labels: up to 4 worker nodes are allocated to heavier services, 2 to less loaded ones, and 1 to non-critical services. Nodes can host replicas of multiple services.