The edge of a self-hosted cluster

Index

• What the edge is made of
• Should you put a CDN in front
• Websockets: the upgrade works and nothing else does
• The client address you no longer have
• What a CDN costs you
• WAF: theirs, yours, or neither
• What the kit does by default
• What I have not proven
• Three failures that filed no complaint
• Lesson one: the load balancer that never existed
• Lesson two: an email address became a routing problem
• Lesson three: two permissions that sound identical
• Where I land

What the edge is made of

Four things, in order, and it is worth being able to name all four before debugging any of them.

• A provider load balancer, created because a Service of type LoadBalancer exists. On Hetzner it terminates the TCP connection and opens a new one to a node. That detail costs you the client address; see below.
• Traefik, two replicas, spread across nodes by a required anti-affinity rule and held there by a disruption budget of minAvailable: 1. It serves Gateway API and nothing else — no Ingress, no IngressRoute, no experimental channel.
• One Gateway, owned by the platform, in the traefik namespace. It owns the addresses, the ports, the certificates, and the decision about which namespaces may attach anything at all.
• cert-manager, the only thing in the cluster that obtains a certificate.

Measured on a live cluster at rest, that whole edge is 2 pods, 4m of CPU and 150 Mi of memory. It is not the expensive part of the platform; the observability stack is. But it is the part that is public, and the part where a mistake is visible to strangers.

Should you put a CDN in front

First, what I mean by a CDN: a proxying one — Cloudflare’s orange cloud, Fastly, CloudFront with a custom origin — that terminates TLS and forwards to your origin. If yours only serves static assets from a separate hostname, almost none of what follows applies to you.

Probably not, if your traffic is mostly long-lived connections, or you handle health, legal, financial or credential data. In the first case you are adding a party with timers you cannot read to a protocol whose whole problem is timers. In the second, a proxying CDN sees the plaintext of every request and response, which puts it inside your compliance scope as a processor. That is not a risk to mitigate, it is the operating model.

Maybe, if you want caching in front of a single-region origin, or bot management, which is genuinely hard to build. Both are real. Neither is urgent enough to accept the rest of the trade blind.

Yes, if volumetric DDoS is a live threat to you. On a small cluster this is the single strongest argument, and it is not close. The alternative to absorbing an attack is not degraded service — it is your provider null-routing the address, and you have no move at that point.

Note that the split does not have to be all-or-nothing. Proxy the web hostname, leave the websocket hostname unproxied. That is not a compromise; it is the configuration Cloudflare itself recommends for work that exceeds its HTTP ceiling.

Websockets: the upgrade works and nothing else does

A websocket starts as an ordinary HTTP/1.1 request carrying Upgrade: websocket, the server answers 101, and the same TCP connection carries frames from then on. To a reverse proxy that is a request which never ends. There is no separate listener, no separate port, no separate route kind. HTTPRoute is correct and sufficient, and ingress-nginx states the same thing in one line: “Support for websockets is provided by NGINX out of the box. No special configuration required.”

So if your websocket never connects, the cause is TLS, routing, or your application. If it connects and then dies, it is one of the following.

Timeouts, which are set for a different kind of traffic. A proxy cannot tell an idle websocket from a hung backend, so every “this is taking too long” timer applies to a connection that is supposed to take long. From the primary docs:

ingress-nginx is explicit about the fix: “A more adequate value to support websockets is a value higher than one hour (3600).” Raise those on the entrypoint carrying long-lived traffic, and not globally — a raised read timeout everywhere is also a raised tolerance for a hung backend everywhere.

But send heartbeats anyway, and treat that as the primary defence. RFC 6455 has Ping and Pong frames for exactly this. A ping every 30 seconds resets every idle timer in the path, including the ones you do not know about — a corporate proxy, a mobile carrier’s NAT, a CDN. A raised timeout protects you from the proxies you configured. A heartbeat protects you from the ones you did not know were there. If you only get one, take the heartbeat.

Sticky routing is usually the wrong instinct. Separate two things: which pod serves this connection is not a problem — one connection is one TCP stream to one pod for its whole life, and there is nothing to be sticky about. Which pod serves the next connection is the real question, and it only matters if a server holds session state in its own memory. If state is in Redis, Postgres or a pub/sub bus, any replica can serve any reconnect, and affinity is dead weight that also ruins your load distribution.

If you do reach for it, know what each mechanism actually keys on. A Service with sessionAffinity: ClientIP keys on the client IP — which, behind a proxy, is the same address for everyone, so all users collapse onto one backend. Gateway API’s sessionPersistence is GEP-1619 and still in the Experimental state. Cookie stickiness works in a browser and silently does not work for a native mobile client or a Go service client, because those usually send no cookies unless you wrote the cookie jar yourself.

And the one that actually bites: a reverse proxy is a shared restart domain. It terminates every connection through it, so restarting it destroys them all at once. Graceful shutdown does not save you here, because a grace period is a deadline rather than a reprieve: Kubernetes gives a pod a termination period that “defaults to 30 seconds”, and Traefik’s graceTimeOut defaults to 10s, during which “no new requests are accepted”. A 200 ms request finishes inside that. A six-hour session never will.

A disruption budget and node anti-affinity are what stop the whole edge going at once — they turn “two replicas” into “two replicas are still running an hour from now”, and they are why the ingress in this kit runs more than one replica at all. What they do not do is save an individual connection. During a rolling update a client can reconnect onto a replica that has not been replaced yet and be dropped again when its turn comes, so worst case is one disconnect per replica. Then every client reconnects in the same second. Exponential backoff with jitter is not a nicety; without it the recovery outage is longer than the restart that caused it.

The rule I ended up writing down: do not put long-lived connection protocols behind the same proxy as short-lived ones. The short-lived traffic has effectively infinite tolerance for restarts, so it sets the schedule, and nobody is choosing to disrupt the websockets — nobody is thinking about them at all. If sessions are measured in hours and a reconnect is user-visible, run a second gateway deployment. Same chart, different Deployment, separate restart domain, one more pod per replica. It is the cheapest correct answer.

The client address you no longer have

Behind a Hetzner load balancer, every request arrives from the load balancer. Not because of externalTrafficPolicy — the provider’s balancer terminates the connection and opens a new one, so no policy setting recovers the address here. Recovering it needs PROXY protocol, which is two changes that must be made together: the annotation on the Service, and proxyProtocol.trustedIPs on the entrypoint. Enable the annotation alone and every request breaks. Enable the entrypoint alone, or trust a range wider than the balancer, and a client can forge its own address.

That asymmetry is the point. Rate limiting, IP allowlists and abuse investigation are not degraded when the address is wrong — they are inverted. A rate limiter keyed on one uniform address either never fires or blocks everyone. An allowlist for an admin path compares against the proxy’s address and therefore allows the entire internet. The audit log is full of a single value that looks authoritative and means nothing.

Add a CDN and the same problem reappears one layer out, as headers. Cloudflare sets CF-Connecting-IP and True-Client-IP and appends to X-Forwarded-For, and recommends the first two because they “have a consistent format containing only one IP address”, where X-Forwarded-For is a list every hop appends to.

But a header is a string the client chose. If your gateway trusts CF-Connecting-IP from any source, anyone who can reach your origin directly sets it to whatever they like — Cloudflare says so plainly for stacked setups: “its value can be spoofed to any value”. Trust has to be conditioned on the source address of the TCP connection, restricted to the CDN’s published ranges. Traefik has forwardedHeaders.trustedIPs for exactly that, and also forwardedHeaders.insecure, of which its own documentation says: “We recommend to use this option only for tests purposes, not in production.”

Trusting from the wrong source is strictly worse than not trusting at all, because it converts a missing feature into a working bypass. Which is why header trust and the origin firewall are one change, not two.

What a CDN costs you

Five rows change the moment you turn it on, and three of them fail with no error at the time and no alert afterwards.

Certificates move to DNS-01. An HTTP-01 challenge is a fetch — Let’s Encrypt only allows it on port 80, follows up to 10 redirects, and “cannot be used to issue wildcard certificates”. Once the hostname resolves to the CDN, that fetch lands on the CDN, and whether it reaches cert-manager’s solver pod is decided in someone else’s dashboard. It is not impossible; plenty of people run it. It is that a bot rule, a cache rule or an “Always Use HTTPS” toggle added months later breaks it, and the breakage is delayed until renewal and silent. DNS-01 never touches the request path. Mitigate its credential risk with CNAME delegation rather than careful handling: point _acme-challenge at a throwaway zone and scope the credential to that zone alone, so a leak buys an attacker certificates for nothing you own.

Origin concealment is not a control. The address leaks through historical DNS, through any unproxied record in the same zone, through outbound connections your origin makes, and through Certificate Transparency — every hostname you get a certificate for is a public record. Treat the origin address as public and defend it: a firewall restricted to the CDN’s ranges, refreshed automatically. Hetzner’s Cloud Firewalls default the right way for this — “If you do not set any rule, all inbound traffic will automatically be blocked” — so the change is an explicit allow rather than a deny bolted onto an open default.

And the firewall alone is not enough, which is the non-obvious part. An allowlist of a shared CDN’s ranges permits every other customer of that CDN. Anyone can sign up, point their zone at your address and arrive from an allowed range. Cloudflare’s answer is Authenticated Origin Pulls, and the caveat is the whole point: global AOP proves Cloudflare, not your account. Only per-zone AOP with your own certificate distinguishes you. A CDN-range firewall plus global AOP looks like two independent controls and is one.

Two more costs that are easy to skip past. Their outage is your outage, for every hostname you proxied, and you cannot engineer around it from inside the cluster. And failover is ordered: you must open the origin firewall first, then unproxy the DNS record, or you are down between the two steps — bounded by TTL, not by how fast you type. Write that runbook before you need it.

What you get is real: DDoS absorbed before it reaches a link you pay for, TLS terminated near the user with HTTP/3 you maintain none of, a global cache in front of a single-region origin, and bot management. The free tier is genuinely useful. The managed rulesets people mean when they say “we have a WAF” are on the paid plans.

WAF: theirs, yours, or neither

Theirs runs at the edge, so blocked traffic never costs you bandwidth, CPU or a pod, and it sees real client addresses natively with no forwarded-header configuration to get wrong. Against: you cannot read the rules, test them offline, reproduce a block locally, or version them next to what they protect. Every “the WAF is blocking our API” incident becomes an out-of-band investigation in a dashboard rather than a diff.

Yours puts the rules in git, identical in staging and production, with no vendor dependency and a detection mode you can read with full request context. The credible open-source engine is OWASP Coraza, and you should read its own maturity table before planning around it: the Caddy plugin and proxy-Wasm are stable, and the Traefik plugin is documented as preview. For a Traefik-fronted cluster that means the WAF would be less mature than the proxy it runs inside. That is a reason to decline, and it is not a criticism of Coraza — it is reading the label. Do not route around it via ingress-nginx and ModSecurity either: the maintainers have announced maintenance mode, with “1.13 in all likelihood … the last minor release” and a migration expected to take about two years.

Neither is the option I think is usually the best value. Most of what a Core Rule Set install catches in practice is caught by request size limits, per-route rate limits, blocking obviously hostile paths and agents, and a short list of targeted rules for the framework you actually run. That is a few dozen lines, it produces almost no false positives, and you understand every one of them. CRS in blocking mode without a tuning period will break legitimate traffic — uploads, rich-text bodies, anything with SQL-like strings in a legitimate field — and the standard outcome is that somebody disables it during an incident and it never comes back on. You are then left with the cost, the config and none of the protection.

One ordering constraint if you do self-host: a WAF sees the CDN’s address for every request until forwarded-header trust is correct. Client addresses first, WAF second, or its rate limiting and IP rules are wrong from the day it ships.

What the kit does by default

cert-manager is the sole ACME client, and that is what buys the second replica. An ingress controller with a built-in ACME resolver keeps certificate state in a local file, so two replicas race: both notice a missing certificate, both order it, and the account’s issuance limits pay for it. The universal workaround is to pin the ingress to one replica — which makes the most traffic-critical component in the cluster a single point of failure in exchange for a problem cert-manager does not have. cert-manager issues into a Secret and any number of replicas read it. The constraint disappears rather than being managed.

Then go and hunt the effects, because that is where these things hide. The single replica is why the disruption budget was disabled in every guide that copied it. Remove the cause and the budget can come back, along with a required anti-affinity rule so the two replicas are not both on the node that is about to reboot. The kit fails CI if a certificate resolver reappears in the Traefik values, because a comment saying “no ACME here” is not evidence of it.

There is no WAF, and a proxying CDN is opt-in. Not because either is a bad idea, but because turning the CDN on is four changes made together — DNS-01 issuance, forwarded-header trust, an origin firewall with per-zone AOP, and an explicit WAF decision with a named owner. Items one to three are not defence in depth if only some are done; each is what makes the next meaningful. Shipping that as a default would ship the appearance of it.

The experimental Gateway API channel is off, which means no TCPRoute or TLSRoute through this proxy. That is partly the API’s own warning about breaking changes, and partly the restart domain: routing raw TCP through the shared gateway looks like it solves the long-lived-connection problem because the protocol is no longer HTTP, and it does not. It solves multiplexing, not lifetime.

What I have not proven

The list I would rather publish than have you find.

• I have not watched a 101 come back through this gateway. Traefik has no websocket configuration surface, which is consistent with “nothing to configure”, and that is how the kit is written — but I could not find a current Traefik documentation page stating it either way, and I have not measured it.
• Whether Traefik’s readTimeout, documented as covering “reading the entire request”, applies to a connection that has been upgraded is not stated anywhere I could find. Do not assume either answer. Open a connection, send nothing, and time how long it survives.
• Cloudflare’s websocket idle timeout is unpublished. Their docs say only that they close a connection when no data is transmitted for a period, and that Enterprise customers can have a custom value. The widely repeated “100 seconds” figure is not in their documentation and I could not verify it from a primary source. The HTTP path is documented: a 125-second proxy read timeout and a 30-second write timeout, which kills long-polling fallbacks and slow API calls, and which a heartbeat does not help with.
• The PROXY protocol path on this provider is not what the kit ships. It is two commented-out blocks with a warning above them, and I have not run it.
• No CDN has been in front of this cluster. Everything in the CDN sections is read from primary documentation and reasoned about, not measured. I would treat it as a checklist to verify, not as a report.

Three failures that filed no complaint

Three edge failures, in the order I met them. None crashed. All three reported success on the way past, and each surfaced several layers from its cause.

Lesson one: the load balancer that never existed

The Service that creates the load balancer carries two provider annotations: a location and a type. I left one at CHANGE-ME.

The Helm release installed. The reconciliation reported success. The pods were ready. The cloud controller then rejected the value with “location or network zone not found”, in a Service event nobody reads, and retried silently for fifty minutes before I looked.

The result is an ingress with no address, on a cluster where every object you would think to check is green. Worth knowing too: the location must match where your nodes actually are, and the type must be one currently available there — “supported” and “available” are different things in a provider API, and a type that is merely supported applies cleanly and fails at creation.

A placeholder that fails loudly is a good placeholder. This one applied.

Lesson two: an email address became a routing problem

The certificate issuers shipped with a placeholder contact address. Let’s Encrypt rejects that outright:

Failed to register ACME account: 400 urn:ietf:params:acme:error:invalidContact

Follow the chain. No ACME account means no certificate. No certificate means the Secret the listener’s certificateRef names never exists. A listener referencing a Secret that does not exist makes the whole Gateway Accepted=False — “All Listeners must be valid” — and a Gateway in that state accepts no routes at all. The platform reconciliation waits for health, so it never finished, and everything behind it never reconciled either.

An unusable email address presented as a routing problem, four layers from its cause. On a fresh install, with nothing to indicate which file to open.

The fix was not a better error message, which I do not control. It was moving the issuers out of the default apply path entirely, so the cost of leaving a placeholder unedited is the feature that needs it — rather than everything downstream of it.

Lesson three: two permissions that sound identical

Gateway API has two cross-namespace mechanisms, both described as “cross-namespace permission”, and they do different jobs.

allowedRoutes lives on the listener, the platform sets it, and it decides which namespaces may attach a route to that listener. ReferenceGrant lives in the namespace being referenced, its owner writes it, and it decides whether an object may reference an object in another namespace — a route’s backendRef to a Service elsewhere, a listener’s certificateRef to a Secret elsewhere.

A ReferenceGrant does not let anyone attach a route. allowedRoutes does not let a route reach a Service in another namespace. Assuming either covers the other leaves one of the two open, and the failure is quiet in both directions: a route that may attach to no listener reports the failure on itself and nothing at all on the Gateway, so the Gateway looks perfectly healthy — from its side, nothing happened.

The direction of the grant is the part worth keeping. The permission is granted by the party giving something up, not claimed by the party taking it.

Where I land

For a small self-hosted cluster serving ordinary short-lived HTTP to a regional audience: no CDN, no WAF, and spend the effort on the two things that are free. Get client addresses right, because everything you would build on top of them is inverted rather than merely absent until you do. Then write the few dozen lines of size limits, rate limits and targeted rules that catch most of what a generic ruleset catches, with none of the tuning debt.

Put a proxying CDN in front the day volumetric DDoS becomes a real threat to you, and accept that it is four changes rather than one toggle. Do not put one in front of long-lived connections at all — give those a hostname that stays unproxied, or a gateway of their own.

And whichever you pick, measure it rather than reading about it. Open a websocket, send nothing, and write down how long it survives. Do that again after every ingress change. That number is worth more than any table on this page, including mine.