What a broker does
A message broker sits between producers and consumers so they never talk directly. Producers publish messages; the broker stores them durably; consumers read them later. This decoupling means either side can scale, restart, or slow down without breaking the other.
The main parts
- Ingress: validates an incoming message and assigns it to a topic.
- Log or queue store: appends the message to durable storage, often an append only log on disk.
- Index: tracks where each message sits so reads stay fast.
- Egress: serves messages to consumers and tracks what has been delivered.
Why append only helps
Writing sequentially to disk is far faster than random writes. Brokers like Kafka exploit this by treating each partition as an ordered log. Reads are also sequential, which lets the operating system cache hot data efficiently.
Flow
Key idea
A broker is a durable buffer with an append only log at its heart, decoupling producers from consumers so each can fail or scale on its own.