← Lessons

quiz vs the machine

Gold1410

System Design

Alerting Rules Evaluation

How periodic queries become firing alerts with for durations.

5 min read · core · beat Gold to climb

Rules as periodic queries

An alerting rule is a query that the metrics system evaluates on a fixed evaluation interval. When the query returns matching series, those become pending alerts; the rule defines a condition over your metrics.

The for duration

A rule usually carries a for clause. An alert must stay true continuously for that duration before it moves from pending to firing. This suppresses brief blips and reduces noisy pages.

What happens on fire

  • Each firing series carries labels for routing and annotations for human context.
  • Firing alerts are sent to an alert manager that deduplicates, groups, and silences.
  • When the query stops matching, the alert resolves.

Good rule design

  • Alert on symptoms users feel, not every internal cause.
  • Use for to avoid flapping.
  • Keep labels meaningful so routing and grouping work.

Key idea

Alerting rules are queries evaluated on an interval; a for duration must hold before pending becomes firing, then an alert manager routes and dedupes.

Check yourself

Answer to earn rating on the learn ladder.

1. What does the for duration in an alerting rule do?

2. What component deduplicates and groups firing alerts?

3. Best practice is to alert primarily on