← Lessons

quiz vs the machine

Gold1450

System Design

The High Cardinality Problem

Why an explosion of unique series can sink a metrics system.

5 min read · core · beat Gold to climb

What cardinality means

Cardinality is the number of unique time series, which equals the number of distinct label combinations. A metric with a few low cardinality labels is cheap; one with an unbounded label is dangerous.

The explosion

Cardinality multiplies. If a metric has labels for region with five values and instance with one thousand values, that is already five thousand series. Add a user id label and cardinality can reach millions.

Why it hurts

  • The index that maps labels to series grows huge and slows queries.
  • Memory for active series headers balloons.
  • Ingestion and compaction get slower.

Offending labels

  • User ids, request ids, email addresses, full URLs, and timestamps as labels are classic mistakes since their values are effectively unbounded.

Defenses

  • Keep labels bounded and meaningful.
  • Drop or aggregate away high cardinality labels at ingestion.
  • Set per metric series limits and alert when approached.

Key idea

Cardinality is the count of unique series and multiplies with each label, so keep labels bounded to avoid index and memory blowups.

Check yourself

Answer to earn rating on the learn ladder.

1. What is cardinality in a metrics system?

2. Which label is most likely to cause a cardinality explosion?

3. A good defense against high cardinality is to