The Front Door Does More Than Route

Courtyard has a dozen services now, from Chapter 13, and each owns its own data, from Chapter 14. And every one of them, to serve a request, has to do the same handful of things that have nothing to do with what the service is for. Every service has to check that the request comes from a logged-in user, by verifying the token from Chapter 7 or looking up the session. Every service has to terminate the security from Part I Chapter 21, decrypting the request and holding a certificate. Every service has to protect itself from abuse, from a client hammering it with requests, which is Part I Chapter 16’s problem carried out as a deliberate attack. Every service has to log what it did, for debugging and for the record. Every service has to check that the request is not malformed or oversized.

None of that is chat, or feed, or accounts. It is the same wrapper around all of them. And because each team implemented its own, there are a dozen implementations of checking the login token, and they are subtly different, and three of them have bugs. One does not check expiry. One is slower than the rest. One logs the token itself, which is a security hole, Part I Chapter 21’s leaked credential. The same work has been done a dozen times, a dozen ways, and wrong in three places. What has run out is engineer-hours, the team’s time, spent a dozen times over on identical plumbing that nobody’s service is about, and spent again every time a bug in one copy has to be fixed in all twelve.

There is already a layer that every request passes through before it reaches any service, and that is the load balancer from Chapter 6, the front door. Every request crosses it. So you do the common work there, once, for everyone.

The front door grows up into what the field calls an API gateway, the single entry point that does all the cross-cutting work before routing a request to whichever service handles it, which was already its job from Chapter 6. It terminates TLS, from Part I Chapter 21, so that one place holds the certificate and one place decrypts. It authenticates, so that one correct implementation checks the token and passes the verified identity along to the service, which can trust that identity and need not check it again. It rate-limits, so that one place counts requests per client and rejects abuse before it reaches any service. It logs, so that one place records every request uniformly. It rejects malformed and oversized requests at the door. Then it routes what is left to the right service, which receives a clean, authenticated, rate-limited request and can give its whole attention to being a chat service or a feed service.

The services get thinner and better. They shed the plumbing and keep the logic. The plumbing gets one correct implementation instead of a dozen approximate ones, so that a bug in token-checking is fixed once and every service is fixed with it. And protection from abuse, which each service was doing badly or not at all, is now uniform and comes nearly for nothing, because the gateway is the natural place to see that one client is sending ten thousand requests a second and to start rejecting them. That is Part I Chapter 16’s backoff, imposed on a client rather than chosen by a sender, and Part I Chapter 20’s 429, sent from one place.

The gateway is seductive, and the failure mode is to put too much in it. The line to hold is that the gateway does work that is the same for every request regardless of what the request is for. Authentication is the same whether you are posting a message or fetching a feed, so it belongs at the gateway. Rate limiting, TLS, logging, and request validation are the same for everyone, so they belong there too.

What does not belong there is anything specific to a service’s purpose. Whether Mira is allowed to post in this particular group is the chat service’s business logic. It needs to know about groups and membership, and the gateway must not know about groups, or it becomes coupled to chat’s domain. The moment the gateway contains logic about what services do, it stops being a shared entry point and becomes a place where every team has to make changes, which is the problem of the next few paragraphs made worse. Keep the gateway concerned with the envelope, in Part I Chapter 6’s sense of the word, meaning who the request is from, whether it is well formed, and whether it is allowed through the door at all, and keep it out of the letter, meaning what the request means, which is the owning service’s job. Authentication, the question of whether you are a valid user, goes at the gateway. Authorisation, the question of whether you are allowed to do this specific thing, goes mostly in the service, because it depends on what the thing is.

This chapter’s resource is engineer-hours on both sides of the ledger, and the gateway is not a free win. It is now a thing every request depends on and every team’s requests flow through, and that makes it two dangerous things at once.

The first is a shared dependency. If the gateway is down, everything is down, every service, all at once, because nothing can be reached except through it. Chapter 6 already noted that the front door is a single point of failure. The gateway concentrates far more into that one point, so it must be made highly redundant, with Part II Chapter 7’s anycast and multiple instances, and kept ruthlessly simple, because complexity in the one thing everyone depends on is risk multiplied by everyone.

The second is a shared chokepoint for changes. When the feed team needs a new kind of authentication, or the chat team needs a different rate-limit rule, they need a change to the gateway, which is owned by some platform team, which has its own queue, in which every other team is also waiting. The gateway that freed the services from duplicating plumbing has made every team dependent on one team for anything that touches the plumbing. The bottleneck has moved, from every team implementing its own authentication to every team waiting for the platform team to change the shared one. That is Chapter 13’s coordination cost, reintroduced at the front door. It is usually the better trade, since one team maintaining one correct implementation beats twelve teams maintaining twelve wrong ones, but it is a trade, and a gateway run by an overloaded or slow platform team becomes the thing every other team complains about.

The resolution most large systems reach is to make the gateway configurable rather than coded for the common variations. Teams declare their rate limits and authentication requirements in configuration that the gateway reads, so they can change their own policies without a change to the gateway’s code and without queuing behind the platform team. The gateway provides the mechanism and the teams control the policy. It is the same move as making the shard router a function rather than a table in Chapter 4, pushing the per-team decisions out to the teams and keeping only the shared mechanism in the shared thing, and it is how you get the gateway’s benefits without its chokepoint closing entirely.

The gateway is now the largest single point of failure Courtyard has, because everything is behind it, so its reliability is the ceiling on Courtyard’s reliability. Its performance is a tax on every request, since every request pays the gateway’s latency, and so it must be fast, which is Part I Chapter 15’s discipline of never blocking, applied on the busiest machine you have. Its correctness matters in a new way, because a bug in the gateway’s authentication is a bug in everyone’s authentication, and the blast radius of a mistake there is the whole system. And it is a coordination bottleneck for any change to shared behaviour, softened by configuration but not removed.

That is the fifteenth revision of Courtyard. Engineer-hours ran out, and you consolidated the cross-cutting concerns, authentication, TLS, rate limiting, and logging, into an API gateway that every request crosses. The gateway is now the largest single point of failure and a per-request tax, it must be made redundant and fast, and it is a coordination chokepoint softened by letting teams configure their own policies. Courtyard is now a dozen services behind a gateway across two regions, and with this many moving parts something, somewhere, is always broken.