Waiting Without Wasting

This post is about waiting — which sounds like the least interesting thing a program can do, and turns out to be where the last link of the chain from Post 05 finally connects. By the end of it, the line from a hardware interrupt to a callback in your runtime will be unbroken.

The obvious design

One thread per connection.

It deserves more respect than it usually gets. The code is linear and reads like the protocol it implements: read the request, do the work, write the response, loop. There is no callback inversion, no state machine, no coloured functions. Each connection’s state lives on its own stack, exactly where you would put it if you were designing for clarity.

And the waiting is genuinely free of CPU cost. A thread blocked in a read is not spinning. It has been marked not-runnable and set aside — Post 02’s fourth context using its one distinguishing capability. The scheduler will not give it a core until something makes it runnable again.

For a great many servers, this is still the right design. It only breaks in one direction.

Where it breaks

Not CPU. Memory and scheduling.

Every thread needs a stack, and the default reservation is generous — commonly eight megabytes of address space each. Only the touched pages are real memory, so the true cost per idle thread is more like tens of kilobytes of stack plus a few kilobytes of kernel bookkeeping. Call it 30 kilobytes. At fifty thousand connections that is roughly a gigabyte and a half of memory doing nothing but existing.

The scheduler is the worse half. Every thread is an entity it must track. When data arrives for a hundred of them at once, a hundred threads become runnable and must be dispatched, each dispatch costing a context switch of a couple of microseconds plus the cache damage of loading a new working set onto the core. The scheduler’s own data structures grow. Cores spend a rising fraction of their time deciding what to run rather than running it.

But the deepest problem is a mismatch of scale. At any given instant, out of fifty thousand connections, perhaps two hundred have something to say. You are maintaining fifty thousand kernel-managed entities in order to express two hundred events.

Which suggests the real question. How do I wait for fifty thousand things using one thread?

The first answer, and why it is not enough

The original interfaces are straightforward: hand the kernel the list of descriptors you care about, and it tells you which ones are ready. There are two of them, and the older — select — cannot even be used at this scale, since it works on fixed-size bitmaps capped at 1024 descriptors. So assume its successor, poll, which takes an array and has no such limit.

It works. And its cost is the problem. On every call, your array is copied into the kernel, the kernel walks all fifty thousand entries checking each one, and the results are copied back. You do that, in a loop, forever — fifty thousand inspections per iteration to discover two hundred events.

The work is proportional to what you are watching, not to what happened. And since you are watching a great deal and very little happens, almost all of that work is wasted. Worse, you pay it again on every single iteration, because the kernel keeps no memory of what you asked last time.

What actually happens

The fix has two halves, and the first is almost embarrassingly simple: separate registering interest from waiting for events.

You create a kernel object — an epoll instance — and register each socket with it once, when the connection is accepted. The kernel remembers. From then on your loop only ever asks what has happened, and never re-describes what it cares about.

The second half is where it gets good.

When you register a socket, the kernel attaches an entry to a list hanging off that socket. It is the same list from Post 07 where a blocked thread would have parked itself — the socket’s collection of things to notify when its state changes.

Now follow what happens when a packet arrives.

A softirq, per Post 05, walks the packet up the stack. TCP appends the payload to the socket’s receive queue, per Post 07. And then, still in softirq context, it calls the socket’s “something changed” callback, which walks that list of interested parties. For a blocked thread, that means marking it runnable. For an epoll registration, it means moving this socket onto a ready list belonging to the epoll instance — an operation with a fixed, tiny cost that does not depend on how many sockets are registered.

By the time your event loop next asks, the answer has already been computed. It is sitting in a list. Your call takes items off the front of it.

The chain, complete

This is the moment to put the whole receive path in one sentence, because every link is now in place and not one of them involved your code until the last.

A voltage becomes a frame in the card. The card DMAs it into a buffer the driver provided in advance. The card raises an interrupt; the handler disables that interrupt and schedules a poll. A softirq drains the ring, walks the packet up through Ethernet, IP and TCP, looks up the five-tuple, finds your socket, appends the bytes to its receive queue, and calls the socket’s wakeup callback. That callback moves an entry onto an epoll ready list. Your event loop, blocked in a system call, becomes runnable. The scheduler gives it a core. It returns you a file descriptor number.

Nine handoffs, four execution contexts, and the first thing your program learns about any of it is an integer.

Two ways of being told

There is a choice in how the ready list reports, and it is the source of one specific, vicious bug.

Level-triggered is the default and answers the question is there data right now? If a socket has unread bytes, it will be reported every time you ask, until you have read them all. Forget to fully drain a socket and you will simply be told again. It is forgiving.

Edge-triggered answers has anything arrived since I last told you? You are notified once per arrival. If you read only some of the available bytes, you will not be told again — because nothing new arrived; the data was already there.

The bug writes itself. Under edge-triggered mode you must read from a socket repeatedly until it tells you there is nothing left. Anything less and the remaining bytes sit in the receive queue forever, your application waits for a notification that will never come, and the client waits for a response that will never be written. The connection hangs with data in it, which is a confusing thing to debug because every layer looks healthy.

The reward for the extra care is fewer system calls under load, which is why high-performance servers use it and most application frameworks do not.

Readiness and completion

Everything so far is a readiness interface. The kernel tells you you may now read without blocking. Then you call read, and the kernel copies the bytes into your buffer, for the reasons Post 03 gave.

That is two system calls per event, minimum, plus one to write the response. At a million events per second, the transitions in and out of the kernel are themselves a significant cost — and they got more expensive in 2018, when the mitigations for speculative-execution vulnerabilities made crossing that boundary measurably slower.

The newer interface inverts it. Instead of asking to be told when you can read, you submit the read itself — put data from this socket into this buffer — and later collect a completion saying it has been done.

The mechanism should feel familiar. You and the kernel share two ring buffers in memory: one where you place submissions, one where the kernel places completions. You write an entry and advance an index. The kernel reads it and advances its own. Nobody copies the ring; ownership of each slot passes back and forth by a published index.

That is Post 03’s descriptor ring, exactly — a queue in memory shared by two parties that run at different times, with ownership transferred by convention. The first time you met it, it was between a device and the kernel. This time it is between the kernel and you. The pattern is the same because the problem is the same.

That already collapses many operations into one crossing rather than two crossings per operation. And if you ask for it, a kernel thread will poll the submission ring on your behalf, at which point a busy server can submit and reap thousands of operations without entering the kernel at all — paying a core for the privilege, which is a trade rather than a free win.

So which is it: threads or events?

The honest answer is that the question is malformed, because both are solving the same problem and the difference is where the state lives.

A connection being serviced has state: how far through the request you are, what you are waiting for, what to do next. A thread keeps that state on a stack, implicitly, in the program counter and local variables, and the operating system schedules it. An event loop keeps it in a heap object, explicitly, and you schedule it.

Stacks are more pleasant to program against and more expensive to have fifty thousand of. Heap objects are cheap and force you to write your control flow inside out.

Which is precisely what modern runtimes resolve. A goroutine, an async task, a coroutine — each is a stack that is small, growable, and scheduled in user space rather than by the kernel. You get code that reads linearly, and state that costs a few kilobytes instead of a kernel thread.

And underneath every one of them, without exception, is the mechanism in this post. Go’s runtime has a network poller built on epoll. Node’s event loop is libuv, built on epoll. Python’s asyncio, Rust’s tokio, nginx, HAProxy, Envoy — all epoll, or its equivalent on other kernels.

See it for yourself

An idle event-loop server is doing exactly one thing, and you can watch it.

epoll_wait(5, [], 1024, 3218)           = 0
epoll_wait(5, [{EPOLLIN, {u32=14}}], 1024, 2465) = 1
recvfrom(14, "GET /users/42 HTTP/1.1
Host: "..., 65536, 0, NULL, NULL) = 187
sendto(14, "HTTP/1.1 200 OK
Content-Typ"..., 214, 0, NULL, 0) = 214
epoll_wait(5, [], 1024, 4000)           = 0

Two more worth a minute. ls -l /proc/PID/fd shows the sockets your process holds, and among them an entry of type eventpoll — the instance itself, a file descriptor like any other. And if your runtime uses io_uring, you will see almost nothing in strace, because there is almost nothing crossing the boundary to trace. That absence is the point.

What to carry forward

Your handler is about to run. Everything from the wire to here has been someone else’s code, running in contexts you cannot see, moving data through queues you did not create.

The next post is the shortest in the series, and deliberately so. Turning the bytes into a function call is the part you already understand — and seeing how little is left to do is the best evidence that the interesting work happened below.