Two shapes of a server
As a server scales, the cost of its concurrency model dominates. There are two classic shapes: thread per request and an event loop.
Thread per request
In this model each connection gets its own OS thread. It is simple to write because each thread blocks on its own work. The trouble is cost:
- Every thread reserves a stack, often one or two megabytes.
- The scheduler must context switch between thousands of threads, and that overhead grows with the count.
- Ten thousand idle connections can mean ten thousand parked threads burning memory.
Event loop
An event loop runs on a small number of threads and multiplexes many connections using non blocking IO. When a socket is not ready, the loop moves on instead of parking a thread.
- Memory per connection is just a small state object, not a full stack.
- Far fewer threads means far fewer context switches.
- The cost is that blocking calls in the loop stall everyone, so all work must be async.
Key idea
Threads are easy but cost a stack and a switch per connection, while an event loop scales to many idle connections cheaply as long as nothing blocks the loop.