Waiting costs nothing
Each worker asks the kernel which of its connections are ready, handles those and asks again. A quiet connection is a few hundred bytes in a list, not a thread blocked on it.
How does one server hold ten thousand connections without breaking a sweat?
Most connections are idle most of the time. Nginx keeps them all in a few event loops, sends the real work to app servers behind it, and quietly steps around the ones that have crashed.
Requests run one at a time, and memory and timings are round numbers. The idle-connection cost is the one nginx.org quotes.
A browser asks for /orders. Nginx doesn't build pages: it accepts the connection, picks an app server, forwards the request and relays the answer. To the browser, nginx is the server.
ReadyNginx is listening on port 80, in front of three app servers. Send it a request.
Ready. Nginx is listening on port 80, in front of three app servers. Send it a request.
Send a request
Server
App servers
Requests, newest first✕ refused
| # | request | served by | status | ms |
|---|
Counters
0Requests
0Errors to clients
0Retried upstream
0Open connections
10Memory, MB
0App time, ms
Sheet 11 · One proxy, many connectionsRequests run one at a timeMemory and timings illustrative
Three ideas explain most of it
One front door
Clients only ever talk to nginx. It answers static files itself and forwards everything else to app servers it chooses, so they can be added, removed or restarted unseen.
Retry, then route around
A refused connection is retried on the next server, and the broken one is marked down for a while. Visitors only see an error when nobody is left.
A config file that describes the traffic
Live: the directive at work
worker_processes 4; # one per CPU coreevents { worker_connections 16384; } # each, not in totalhttp { upstream app { # round robin by default server app1:3000 max_fails=1 fail_timeout=10s; server app2:3000 max_fails=1 fail_timeout=10s; server app3:3000 max_fails=1 fail_timeout=10s; } server { listen 80; location /static/ { root /var/www; } location / { proxy_pass http://app; proxy_next_upstream error timeout; proxy_buffering on; } }}Live: connections right now
$ curl -s localhost/nginx_status
Active connections: 0
server accepts handled requests
0 0 0
Reading: 0 Writing: 0 Waiting: 0Older web servers gave each connection a process or a thread. That's simple to program, but every thread needs its own stack, and a thread waiting on a slow or silent client does nothing while holding it. Ten thousand connections meant ten thousand threads: the C10k problem.
Nginx starts one worker per CPU core instead, and each one handles thousands of connections without blocking: it reads whatever has arrived, writes whatever the socket can take, and moves on. With worker_connections 16384 and four workers, that's 65,536 connections from four processes.
The upstream block is the load balancer. Round robin is the default; least_conn and ip_hash are one line away. proxy_buffering decides who waits for a slow client: nginx, cheaply, or your app server, expensively. Press “A slow phone” with it on and off, and watch app time.
Where the limits really are
Idle clients, full server
A thread-per-connection server can be “full” while doing no work at all. Putting nginx in front of one fixes it: nginx holds the idle crowd and passes on only real requests.
All upstreams down
Retries hide one dead server. When every server is marked down, the answer is 502 Bad Gateway, which means nginx is fine and everything behind it isn't.
File descriptors and timeouts
Every connection is a file descriptor, so the operating system's limit has to be raised to match. And a slow app shows up as 504 Gateway Timeout, which no amount of nginx tuning fixes.
Two predictions
10,000 clients hold idle keep-alive connections. A server that gives each connection its own thread has a pool of 400. What happens to the next real request?
app2 has crashed, and nginx doesn't know yet. Round robin sends the next request to app2. What does the visitor get?
What this model leaves out
Concurrency
Here requests run one at a time so each step can be read. Real workers interleave thousands of requests, each a few steps along, which is the whole point of the event loop.
TLS and HTTP/2
Nginx usually terminates TLS too, doing the handshake and decryption so app servers speak plain HTTP on a private network. HTTP/2 and HTTP/3 also let one connection carry many requests at once.
Active health checks
Open-source nginx notices failures passively, from real requests, as shown. NGINX Plus and other proxies can also probe servers on a schedule and remove them before a visitor hits one.
Caching
With proxy_cache, nginx can keep app responses on disk and serve repeat requests without asking an app server at all, the same idea the Redis topic explores.