How the Kernel Finds Your Socket

The answer is the thing that also explains how the kernel delivers a packet to the right socket at all, and it is worth getting exactly right, because the folk version — the port identifies the connection — is wrong in a way that makes both facts above inexplicable.

The obvious design

A packet arrives carrying a destination port. Somewhere there is a table indexed by port number. Look up 443, find the socket, put the bytes in it.

This is clean, it is fast, and it is what nearly everyone believes.

Where it breaks

That table has one entry for port 443. Your server has a hundred thousand connections on port 443. One entry cannot be a hundred thousand sockets, and the packets arriving for them are not interchangeable — each belongs to exactly one conversation with its own sequence numbers, its own buffers, and its own recipient.

So the port cannot be the key. It is not even close to being the key.

What actually happens

A TCP connection is identified by four values, and the protocol makes five:

• the protocol (TCP)
• the local address
• the local port
• the remote address
• the remote port

That five-tuple, and nothing smaller, names a connection. The kernel hashes it and looks the result up in a table of established connections. A hundred thousand clients connecting to your port 443 produce a hundred thousand distinct tuples, because each client contributes a different remote address, a different remote port, or both.

And now both facts fall out

Inbound. Your server’s side of the tuple is fixed: its address, port 443. What varies is the remote half — a 32-bit address and a 16-bit port. The space of distinct connections you can accept is enormous, and the real limits are memory, file descriptors, and your ability to do useful work. Nothing about the tuple constrains you.

Outbound. Now invert it. Your machine connects repeatedly to one destination — a database at one address on one port. Three of the five values are pinned: the protocol, the remote address, the remote port. Your local address is generally pinned too, by routing. Exactly one value is free: your local port, drawn from the ephemeral range.

net.ipv4.ip_local_port_range = 32768	60999

Twenty-eight thousand. Not a tuning failure, not a leak: the arithmetic of the tuple.

And in practice you will hit it well before twenty-eight thousand, for a reason this post has already set up. A closed connection does not release its tuple immediately — the side that closed first holds it in a waiting state for about a minute. A client opening and closing five hundred connections a second to one endpoint therefore accumulates around thirty thousand held tuples before the oldest begin to expire, and starts failing to connect while having almost no connections actually open. The symptom is a machine that cannot connect and shows nothing in its connection count, which is bewildering until you count the waiting ones too.

Stated this way, the fixes stop being folklore and become obvious, because each one frees up a different field. Widen the port range. Give the machine more source addresses. Spread across more destination addresses or ports. Or — far better than any of those — stop opening so many connections and reuse them, which is what connection pooling has always really been for.

Getting there: the same move, four times

Before the lookup can happen, the packet has to reach TCP at all. That journey is one motion repeated, and it is worth seeing as a single pattern rather than four separate layers.

At the link layer, the packet is an Ethernet frame. Its header ends with a field naming the protocol inside. The value says IPv4, so the kernel advances its pointer past the fourteen-byte header and hands the rest to the IP code.

At the network layer, IP checks that the version and length are sane and the header checksum holds — unless the card already did it, in which case it trusts the card’s verdict from Post 04. Then it asks the routing question: is this destination address one of mine? If not, this packet would be forwarded, or dropped. It is, so this is local delivery. The IP header ends with a field naming the protocol inside. It says TCP, so the pointer advances past that header too.

At the transport layer, TCP reads the ports, builds the tuple, and performs the lookup above.

And at the application layer, some minutes of wall-clock time later, your web framework takes the first line of the request, reads the path, and decides which of your functions handles it.

Along the way the packet passes several points where firewall rules may inspect, rewrite, or drop it. Those are hooks on this path, not a parallel one — which is why a firewall rule can be described entirely in terms of where in this walk it runs.

What a socket actually is

The lookup returns a pointer. It is worth being concrete about what it points at, because “socket” is one of those words that stays abstract for years longer than it needs to.

It is a struct in kernel memory. Your file descriptor is a small integer indexing a per-process table; that entry points at a file object; that object points at this struct. Nothing more mystical than that, and it explains why a descriptor is meaningless in another process, and why fork produces two descriptors pointing at one socket.

The struct holds, among other things:

• the five-tuple, which is how the lookup found it
• the connection state — established, closing, and so on
• a receive queue of data that has arrived in order and not yet been read
• a separate out-of-order queue for data that arrived early
• a send queue of data you have written that has not yet been acknowledged
• accounting of how much memory those queues are allowed to consume
• a list of who to notify when something changes
• the congestion and timing state that governs sending

Two of those are the Recv-Q and Send-Q columns you met in Post 01. They are not statistics about the socket. They are the socket.

Delivery, and the second question TCP asks

Having found the socket, TCP asks where in the stream this segment belongs.

Every segment carries a sequence number. If it is the next one expected, its payload is appended to the receive queue and the expected number advances — possibly by more than one segment, if earlier arrivals have been sitting in the out-of-order queue waiting for this gap to close. If it arrived early, it goes into that holding area. If it duplicates data already received, it is dropped, and an acknowledgement is sent anyway so the sender stops retransmitting.

Then the socket’s receive queue has grown, and whoever is waiting on it needs to be told. That is Post 08.

When the lookup finds nothing at all, the kernel answers on your behalf: a reset for TCP, telling the other end there is nobody here. That is the difference between a connection refused immediately and one that hangs — one means a machine answered and had no socket, the other means nothing answered.

The listening socket is a different animal

There is one socket in your server that has no remote address, because it has no remote end. The one you created with listen.

It cannot be found by the established lookup, because there is no established tuple to match. So the kernel keeps a second table: listening sockets, keyed on local port and address, with wildcards. The order is what you would hope — established first, listeners second — so that a packet belonging to an existing conversation never gets mistaken for the start of a new one.

Its queues are also different, and this is where Post 01 left something unpaid.

A listening socket has two queues. When a connection request arrives, the kernel creates minimal state and replies, and that half-finished connection sits in a first queue awaiting the client’s final acknowledgement. When that acknowledgement comes, the connection is complete — the handshake is done, the kernel will accept data on it, the client believes it is connected — and it moves to a second queue to wait for your application to call accept.

That second queue is the accept queue, and its size is the backlog you passed to listen, capped by a system limit.

Now the important part. The connection is established before your application knows it exists. If your application is busy, the queue fills. When it is full, further completed connections are dropped, and the client — which believes it has a working connection — sends its request into a void and waits for a timeout.

This is one of the nastiest failure modes in server operations, because it is invisible from inside the application. No error is raised. No callback fires. The connection simply never arrives, and your logs show nothing at all.

See it for yourself

The tuple is not a concept. It is four columns, printed by default.

Recv-Q Send-Q   Local Address:Port    Peer Address:Port
0      0            10.0.0.7:8080        10.0.0.31:52418
0      0            10.0.0.7:8080        10.0.0.31:52420
0      0            10.0.0.7:8080        10.0.0.31:52421
1448   0            10.0.0.7:8080        10.0.2.14:41003
0      0            10.0.0.7:8080        10.0.2.14:41118

Two things to try. Run ss -s for a summary of how many sockets exist in each state. And on a server you can safely stress, stop calling accept — pause the process, or put a sleep in the accept loop — then watch ss -tln and see the Recv-Q on the listening row climb toward its limit while clients sit there waiting for a reply that will never come.

What to carry forward

The packet has arrived, been verified, been assigned a core, been walked up through three layers of header-stripping, matched against a five-tuple, and had its payload appended to a queue belonging to a specific socket.

Everything so far has been about transport. From here the story changes character, because the next three posts are about the illusion that transport sells to your application — a reliable, ordered, endless stream of bytes — and about what that illusion costs, what it hides, and what it hands back to you to deal with yourself.

It starts with the fact that the thing in the receive queue is no longer a packet, and never will be again.