Alertmanager Routing Fixes to Cut Prometheus Alert Fatigue

Originally published on kuryzhev.cloud A pager goes off at 3 a.m. with dozens of notifications for the same disk-pressure event on one node, spread across three teams who all have the node exporter alert wired to their phones. By the time someone actually reads one of them, the disk has already recovered and the incident channel is full of "ack,...