Fixed cost per operation
Many operations carry a fixed overhead no matter the size, such as a network round trip, a syscall, or an index lookup. Doing them one item at a time pays that cost again and again.
Batching
Batching groups many items into one operation so the fixed cost is shared.
- Database writes as one multi row insert beat many single inserts.
- Network calls sending one request of 100 items beat 100 requests.
- Disk and log flushes write many records per fsync.
The latency tradeoff
Batching raises throughput but can add latency, because an item waits for the batch to fill.
- Size trigger flushes when enough items gather.
- Time trigger flushes after a small delay so a slow stream does not stall.
- Use both so batches form fast under load and never wait too long when idle.
Pitfalls
- A giant batch can blow memory or time out a single call.
- A failed batch may need careful partial retry so good items are not lost.
Key idea
Batching amortizes fixed per operation cost to lift throughput, traded against a little added latency tuned by size and time triggers.