What the buffers do
Every TCP socket has a kernel send buffer and receive buffer. The send buffer holds data the application has handed off but the network has not yet acknowledged. The receive buffer holds incoming bytes until the application reads them. Their sizes quietly bound how fast a single connection can go.
The bandwidth delay product
TCP can only keep as much data in flight as the receive window allows, and the window is limited by the receive buffer. To fill a link you need a buffer at least as large as the bandwidth delay product, which is the link bandwidth multiplied by the round trip time. A fast link with high latency, a long fat network, needs large buffers or throughput collapses far below capacity.
- Too small a buffer caps in flight data and starves a high latency link
- Too large a buffer wastes memory and can add latency through bufferbloat
- Modern kernels autotune buffer sizes within configured limits
Practical knobs
You set sizes with socket options like SO SNDBUF and SO RCVBUF, or let the kernel autotune between system wide minimum and maximum values. For most servers autotuning is best, with manual sizing reserved for known extreme paths.
Key idea
Socket send and receive buffers bound the data in flight, so a buffer at least the bandwidth delay product is needed to fill a long fat network link.