The Interrupt, and Why the Kernel Refuses to Work Inside It

This failure has a name — receive livelock — and understanding why it happens tells you most of what you need to know about the receive path. The design that prevents it is, to my taste, the single most elegant thing in the network stack, and it is also the reason your interrupt counters behave in a way that looks backwards the first time you plot them.

The obvious design

A packet arrives. The card puts it in memory. The card raises an interrupt. The CPU stops what it is doing, handles the packet, and goes back to what it was doing.

This is not a strawman. It is how network cards worked for years, and it has real virtues. It is simple. It is correct. And it has excellent latency: the packet is attended to the instant it lands, with no waiting around for anyone to notice it.

Where it breaks

First, the arithmetic

An interrupt is not free. Taking one means the core stops mid-stream, saves enough state to return, switches to a different stack, jumps somewhere it was not going, and on the way out restores everything. It also wrecks caches: the handler’s code and data evict whatever the interrupted work had carefully warmed up, and that damage is charged not to the handler but to the innocent code that resumes afterwards and now misses on everything.

What that adds up to depends enormously on the hardware, and anyone quoting a single figure is selling something. The bare entry and exit can be a few hundred nanoseconds on a modern core; once the evicted cache lines are paid for by the code that resumes, low single-digit microseconds is a fair working number.

Take the low end of that — one microsecond — and recall Post 04’s arithmetic. A saturated ten-gigabit link carrying full-size frames delivers 812,000 packets per second. Servicing those interrupts alone then costs 0.8 seconds of CPU per second of wall clock: most of an entire core, spent purely on being notified, before a single byte is examined. Take the high end and it is several cores.

The argument does not depend on which end you believe.

With small packets the same link delivers 14.9 million packets per second, and the design is not merely expensive but arithmetically impossible by more than an order of magnitude.

Second, and much worse, the inversion

The arithmetic only explains why the naive design is slow. It does not explain why the server got worse as load increased. For that you have to remember what an interrupt is allowed to do.

An interrupt preempts everything. That is its defining property. It preempts your application, and it preempts kernel code, and — this is the fatal part — it preempts the softirq that was in the middle of processing the previous packet.

So picture the machine at overload. A packet arrives; the interrupt fires; it schedules the real work; the work begins. Another packet arrives before that work finishes. The interrupt fires again, preempting the work. It schedules more work. Another packet arrives.

The queue of scheduled work grows and is never drained, because every attempt to drain it is interrupted by the arrival of more. The machine spends one hundred percent of its time accepting notifications about work it will never perform. Throughput does not degrade gracefully toward a limit; it collapses toward zero, precisely when the system is most needed.

This is not a thought experiment. It was measured, named and solved by Mogul and Ramakrishnan in a 1996 paper called Eliminating Receive Livelock in an Interrupt-Driven Kernel, and the fix below is a direct descendant of it.

What actually happens

The fix begins with a reframing so small it sounds like wordplay, and is not.

The interrupt’s job is not to deliver a packet. Its job is to say that polling is now worth doing.

Once the kernel knows packets are arriving, it does not need to be told again. It can simply go and look. Interrupts are for the transition from nothing is happening to something is happening, and that transition occurs once per burst, not once per packet.

Here is the mechanism, which is called NAPI.

A packet arrives and the card raises the interrupt for its queue.

The hard interrupt handler does two things and then leaves. It disables further interrupts from that queue — the card will keep filling the ring silently, and will not raise another interrupt no matter how many packets arrive. And it adds the queue to a per-core list of devices that need polling, raising the receive softirq. Total time: under a microsecond, no packet touched.

The softirq runs the poll loop. For each queue on the list, it calls into the driver with a budget — a maximum number of packets it is allowed to take this time round. The driver walks the ring, takes up to that many packets, and sends each one up through the stack.

Then the interesting branch. If the driver drained the ring completely and stopped before its budget ran out, the burst is over: it removes the queue from the poll list and re-enables that queue’s interrupts. The system goes back to sleep and waits to be told again.

But if it used the whole budget with packets still waiting, the burst is ongoing. The queue stays on the list, interrupts stay off, the softirq yields so that other work on the core gets a turn, and polling resumes shortly.

Why this is the right answer

Read the two branches again and notice what the system just did without being configured to do it.

At low load, every packet gets its own interrupt and is handled immediately. The latency is the latency of the naive design, because at low load this is the naive design.

At high load, one interrupt starts a polling session that drains hundreds of packets, and the cost of being notified is amortised across all of them. The per-packet cost falls exactly when it needs to.

There is no mode switch, no threshold, no tuning parameter selecting between them. The system slides continuously between interrupt-driven and polled according to how much traffic there actually is, and the thing that drives the slide is simply whether the ring was empty when the driver last looked.

And livelock is now structurally impossible: while polling, interrupts from that queue are off, so arrival cannot preempt processing. Each poll does a bounded amount of work and then voluntarily gives up the core.

The budgets are visible, and so is running out of them

The budget is not a metaphor. There are two of them at the top of the poll loop: a number of packets the core may process in one go across all its queues, and a wall-clock limit on how long it may spend. Defaults are typically 300 packets and two milliseconds, whichever comes first.

When either runs out with work still pending, the kernel records the fact and defers. And “defers” here means the softirq is rescheduled, and if this keeps happening, the work migrates out to the ksoftirqd thread from Post 02 — at which point network processing stops preempting your application and starts competing with it under the ordinary scheduler.

That transition is one of the most useful things you can observe on a loaded server, and it has a counter.

0000e3f1 00000000 00000012 00000000 00000000 00000000 00000000 00000000
0004a8c2 00000000 00001f4c 00000000 00000000 00000000 00000000 00000000
00001b03 00000000 00000000 00000000 00000000 00000000 00000000 00000000

In that sample, the second core has run out of budget 8,012 times. That is not an error — a healthy busy server squeezes sometimes — but if it climbs steadily while that core also shows heavy ksoftirqd time, the story is coherent and complete: traffic is arriving faster than one core can process it inline, the work has spilled into a thread, and that thread is now taking CPU away from your application.

Column 2, by contrast, is a queue overflowing and should be zero.

The other dial: making the card wait

NAPI amortises everything after the first interrupt of a burst. Interrupt coalescing attacks the first one.

The card is told: when a packet arrives, do not raise the interrupt immediately. Wait up to so many microseconds, or until so many packets have accumulated, and then raise one interrupt for all of them.

Coalesce parameters for enp3s0:
Adaptive RX: on  TX: on
rx-usecs: 8
rx-frames: 0
tx-usecs: 8
tx-frames: 0

This is the clearest latency-versus-throughput dial in the whole stack, and it is worth being blunt about which way it goes. Setting rx-usecs to 100 means that on an otherwise idle machine, every single packet is now delayed by up to 100 microseconds before software sees it. If you are serving requests whose whole service time is 500 microseconds, you have just added twenty percent to your latency to save interrupts you were not short of.

The default of a few microseconds, with adaptive coalescing on, is right for almost everyone. The reason to know it exists is that a tuning guide will one day tell you to raise it, and you should know what you are paying.

Where the model now stands

The receive path, stated properly:

The card DMAs frames into the ring from Post 03. The first frame of a burst raises an interrupt. That interrupt disables itself, notes the queue, and leaves. A softirq then polls the ring in batches, sending packets up the stack, until either the ring runs dry — at which point interrupts come back on — or the budget runs out, at which point the work is deferred and may end up competing with your application as an ordinary thread.

Every number in that paragraph is observable, and the failure mode at each step has a counter.

The packets are now moving up through the stack in batches, in softirq context, on a core the card chose. Each one is an Ethernet frame that needs to find its way to exactly one of the several thousand sockets your server may have open.

How it does that — and what a socket turns out to be underneath the file descriptor — is next.