Fast local state
A stream processor keeps its working state in a local state store, often an embedded key value database on the same machine. Local reads and writes are fast because they avoid a network hop.
The durability problem
Local disk is not durable: if the machine dies, its state is gone. The processor must be able to rebuild state somewhere else.
The changelog topic
Every update to the state store is also written to a changelog topic, a compacted Kafka topic. The changelog is the source of truth for the state. Because it is compacted, it keeps the latest value per key.
Recovery by replay
When a task moves to a new machine after failure or a rebalance, it replays the changelog to rebuild the local store before processing resumes. Compaction keeps that replay bounded to one value per key rather than the full history.
Key idea
A local state store gives fast access while a compacted changelog topic durably backs it, so a task can replay the changelog to rebuild state after failure or a move.