Context
We had two Alertmanagers in different regions — one in Virginia, one in Ohio — sitting behind a single global Prometheus and a single Slack receiver. Whenever one of the peers degraded, deduplication broke: around 400 duplicate alerts a day, mostly resolved notifications, driving up operational noise and eroding trust in the channel.
Architecture diagram
Healthy cluster
Both peers reconcile state inside the deduplication window.
Two Alertmanagers, one Prometheus
A single Prometheus scrapes and sends alerts to both Alertmanagers, one per region (Virginia and Ohio).
Gossip reconciles in time
With push/pull every 60s and no lost broadcast, both peers converge on the same state before the dedup window closes.
Slack gets one notification
Virginia's Alertmanager owns the notification; Ohio recognizes the same alert via gossip and suppresses its own send.
Investigation
I led the investigation from the first duplicate alert, debugging both nodes in parallel and reproducing the problem in an isolated environment. The first hypothesis was network latency between the regions: if Gossip took too long to cross the VPN between Virginia and Ohio, both nodes would decide to notify before recognizing each other.
I tested that hypothesis first — measuring round-trip time between the peers and comparing it against the configured deduplication window. It didn’t explain the impact: the measured latency was too small on its own to account for 400 duplicates a day.
Hypotheses tested
| Hypothesis | Test | Result |
|---|---|---|
| Latency between regions | Round-trip over the VPN measured and compared against the dedup window | Too small to explain the volume — ruled out |
| State reconciliation via Gossip | Lost incremental broadcast vs. push/pull window vs. peer timeout | State didn't reconcile in time — root cause |
With the latency hypothesis ruled out, I dug deeper into the Gossip communication between the two Alertmanagers.
Architecture diagram
Degraded peer
An incremental Gossip broadcast is lost and state doesn't reconcile in time.
Same topology, degraded peer
The topology hasn't changed. What changes is network reliability between the two regions.
60s push/pull, 15s peer timeout
When an incremental Gossip broadcast is lost, the 15s peer timeout expires before the next full push/pull cycle (60s) can reconcile state.
Slack gets two notices
Each Alertmanager decides to resend the resolved notification without knowing the other peer's updated state — duplication, around 400 alerts a day.
The default configuration used a push/pull interval of 60s and a peer timeout of 15s. Push/pull is the full state sync over TCP; peer timeout is the wait window deduplication uses before deciding a peer hasn’t responded. With push/pull larger than the peer timeout, a lost incremental Gossip broadcast had no chance of being corrected by the next full cycle before the dedup window ran out — so the node would resend the resolved notification without knowing the other peer’s updated state.
Action
Cluster interval adjustment
Decision
Reduce the push/pull interval from 60s to 10s, keeping the peer timeout at 15s.
Rationale
Push/pull needs to be smaller than the peer timeout to guarantee state reconciles inside the dedup window, even when an incremental broadcast is lost.
Consequence
More sync traffic between peers, in exchange for state convergence inside the expected window.
- --cluster.pushpull-interval=60s
+ --cluster.pushpull-interval=10s
--cluster.peer-timeout=15sThe fix went to production as a direct patch to the cluster configuration.
Architecture diagram
Post-fix
push/pull drops to 10s; peer timeout stays at 15s.
Same topology, new interval
No node, region, or receiver changed. Only the cluster's state-sync interval.
10s push/pull, 15s peer timeout
With push/pull more frequent than the peer timeout, even a lost incremental broadcast gets covered by the next full cycle before the dedup window closes.
Slack goes back to one notification
Duplicates drop from ~400/day to 150–200/day. What's left isn't dedup anymore: it's legitimate noise from client-side network variation.
Result
Duplicate alerts dropped from roughly 400 a day to 150–200 a day. Before crediting the entire drop to the fix, I separated out what remained: the residual is the standard flow of alerts opened by real client-side network variation — not duplication from a dedup failure. Attributing that noise to the fix would have inflated the result and hidden a different problem.
Lessons learned
I now validate HA under failure too, not just the happy path:
How I validate cluster HA now
A healthy cluster only proves the happy path. Real HA needs to converge when peers are degraded — and measuring the result means separating the signal the fix actually solved from noise that was never a dedup problem.
Glossary
- Gossip protocol
- The protocol Alertmanager peers use to exchange state with each other, without depending on a central coordinator.
- Push/pull interval
- The interval at which each peer syncs its full state with the others over TCP, complementing Gossip's incremental broadcasts.
- Peer timeout
- The wait window deduplication uses before deciding a peer hasn't responded or reconciled state in time.
- Deduplication window
- The interval within which peers need to agree on an alert's state to avoid more than one of them notifying the same event.