From ingress-nginx to Envoy Gateway
Three years ago, I explained how to protect services behind your ingress with OAuth2 Proxy.
Three annotations on an nginx ingress, a proxy installed once in the cluster, and you were done.
Today, we removed ingress-nginx from every one of our clusters. And oauth2-proxy with it.
On paper, migrating off an ingress controller is "just" rewriting manifests. In practice, it mostly means discovering everything nginx was doing for us without being asked.
This article tells that story, and above all what broke.
Why migrate?
The most obvious reason: ingress-nginx isn't maintained anymore. The project moved to minimal maintenance and then stopped in March 2026. Keeping an internet-facing component without security fixes isn't an option.
But it's also a chance to fix an older problem.
Kubernetes' Ingress object is quite poor. It can route a host and a path to a service, and that's about it. Everything else (request size limits, timeouts, authentication, websockets, sticky sessions…) goes through annotations specific to each controller:
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/auth-url: https://url-du-proxy/oauth2/auth
These annotations became a de facto API: untyped, unvalidated strings that mean nothing to any other controller.
Gateway API fixes this with a model split across several objects, each owned by a different role:
GatewayClass: which implementation (here, Envoy Gateway), owned by the platform teamGateway: an entry point, its listeners and portsHTTPRoute: routing for an application, owned by the team that deploys it
And for anything the standard doesn't cover yet, Envoy Gateway offers typed policies that attach to a Gateway or an HTTPRoute. We'll run into several of them.
Why Envoy Gateway and not Cilium?
Our clusters already run Cilium as the CNI, and Cilium implements the Gateway API. The choice seemed obvious.
It wasn't, for a very concrete reason: sticky sessions.
One of our applications is stateful on the server side. A user must always land on the same pod, otherwise they lose their session. With nginx, that was the affinity: cookie annotation.
In Gateway API, this feature exists (sessionPersistence, GEP-1619), but only in the experimental CRD channel. At the time we made the choice, Cilium didn't implement it, unlike Envoy Gateway.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: my-app
spec:
parentRefs:
- name: envoy-gateway
namespace: edge
hostnames:
- my-app.example.com
rules:
- backendRefs:
- name: my-app
port: 8080
sessionPersistence:
sessionName: my_app_sticky
type: Cookie
absoluteTimeout: 48h
cookieConfig:
lifetimeType: Permanent
So you need to install the experimental channel CRDs, and accept that these fields may still evolve. That's a trade-off we made knowingly.
Installing without shooting ourselves in the foot
Everything is deployed as GitOps with Argo CD. We split the install into three separate applications:
- the CRDs (Gateway API and Envoy Gateway)
- the Envoy Gateway controller
- the cluster-specific objects:
GatewayClass,Gateway,EnvoyProxy
The reason is simple: if the controller and the objects that depend on the CRDs land in the same sync, Argo CD can try to create a Gateway before its type exists. Splitting the applications (and adding sync waves) guarantees the order.
This split has another benefit, more important to me: it follows Gateway API's role model. Where a single Ingress resource mixed everything together (controller choice, TLS, routing, and nginx-specific annotations), Gateway API separates concerns:
GatewayClassandEnvoyProxy: how traffic enters the cluster, the platform's jobGateway: entry points, listeners, TLS, still on the platform sideHTTPRouteand the policies attached to a route: an application's routing, owned by the team deploying it
Within a platform mindset, that's exactly what we were after. The infra team owns the three Argo CD applications above. Product teams keep control of the upper layers: their HTTPRoute, their timeouts, their authentication, versioned alongside their application. They don't need to touch the Gateway to publish a new path, and a mistake in one route doesn't affect others.
The boundary is set on the Gateway itself, with allowedRoutes:
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
from: All lets any namespace attach. To be stricter, from: Selector restricts attachment to namespaces carrying a label chosen by the platform.
Exposing Envoy: a NodePort behind Traefik
Our clusters already had an entry point: a load balancer, then Traefik instances on VMs, outside the clusters, terminating TLS and spreading traffic across nodes.
We kept it as is. Envoy is simply exposed as NodePort: the same port is opened on every node, and Traefik treats all nodes as backends.
This is the standard model for Kubernetes outside the cloud. It's actually what cloud load balancers do too: an AWS NLB in instance mode does nothing more than target a NodePort on each node.
With Envoy Gateway, the service is driven by an EnvoyProxy object, referenced by the GatewayClass:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: envoy-gateway-nodeport
namespace: edge
spec:
provider:
type: Kubernetes
kubernetes:
envoyService:
patch:
type: StrategicMerge
value:
spec:
type: NodePort
externalTrafficPolicy: Cluster
ports:
- name: http
port: 80
targetPort: 10080
nodePort: 30080
externalTrafficPolicy: Cluster is a deliberate choice. With this value, any node accepts the connection and forwards it to an Envoy pod, possibly on another node. You pay one extra network hop, but every node is a valid backend for Traefik, even if it hosts no Envoy pod. A simple health check is enough.
With Local, you'd save that hop, but a node without an Envoy pod would refuse traffic. Traefik would then need to know, at all times, which nodes host an Envoy pod.
Recovering the client IP
Two proxies in a row have a direct consequence: Envoy no longer sees the client's IP, only Traefik's. To recover the real IP, you need to tell it to trust one hop in X-Forwarded-For:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name: envoy-gateway-trusted-proxy-headers
namespace: edge
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: envoy-gateway
clientIPDetection:
xForwardedFor:
numTrustedHops: 1
Without this, all your access logs, rate limits and IP allowlists see the same address. And with a value set too high, any client can forge its IP by adding an X-Forwarded-For header.
But this setting relies on a strong assumption: only Traefik talks to Envoy. If a client can reach Envoy directly, it writes the last X-Forwarded-For, and Envoy will take its word for it.
Letting only Traefik in
A NodePort listens on every interface of every node. Without filtering, any machine on the network can reach Envoy on port 30080, bypassing everything Traefik does (TLS, rate limiting, IP allowlists), and choosing whatever IP it wants to claim.
The first instinct would be a NetworkPolicy on the Envoy pods, accepting only Traefik's IPs. With externalTrafficPolicy: Cluster, that doesn't work: when a node forwards the connection to an Envoy pod on another node, it performs SNAT. The pod only sees the node's IP, not Traefik's.
So filtering has to happen before load balancing, on the host itself. Cilium can do this with its host firewall (hostFirewall.enabled=true) and a policy that selects nodes instead of pods:
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: host-nodeport-envoy
spec:
nodeSelector: {}
ingress:
- fromCIDR:
- 10.0.0.15/32
- 10.0.1.15/32
toPorts:
- ports:
- port: "30080"
protocol: TCP
Watch out for the trap: as soon as a host policy selects a node, that node switches to default-deny for all incoming traffic. SSH, the API server, kubelet, etcd, the VXLAN tunnel, Cilium's own health checks: anything not explicitly allowed gets cut off.
So we wrote the full policy (intra-cluster traffic via fromEntities: [cluster, remote-node, health], then the admin ports), and first deployed it with policyAuditMode enabled.
In that mode, Cilium lets everything through but logs what it would have blocked. A few days of Hubble later, nothing unexpected showed up, and we switched to real enforcement.
Along the way, this also fixes the X-Forwarded-For weakness seen above: only Traefik can now reach Envoy, so the last hop is always a trusted proxy.
Switching without a big bang
For each cluster, we ran nginx and Envoy side by side. Applications were first exposed through Envoy on a different URL, tested, then we switched the real hostname over and removed the ingress.
It takes longer, but it's exactly what let us discover the following issues during QA, before they reached users.
Uploads
This is the first thing we hit during QA: on the test URL served by Envoy, uploading large files failed, while it went through fine via nginx.
On nginx's side, we had proxy-body-size: 0, meaning no limit. And nginx has a particularity: when a request body exceeds its memory buffer, it writes it to a temporary file on disk. So it can absorb a multi-gigabyte file without blowing up memory.
Envoy has a different philosophy. It's built to stream: bytes flow through as they arrive, and it applies a per-connection buffer limit (32 KB by default) for flow control. As long as nothing needs the full body, it works great.
Concretely, when the backend reads slower than the client sends, Envoy buffers up to that limit, then stops reading the client's socket until it drains. The client slows down, memory stays stable. A 5 GB file passes through Envoy without ever occupying more than a few tens of kilobytes.
The problem shows up as soon as something in the chain needs the full body before letting it through. At that point, the limit stops being a flow-control threshold and becomes a hard cap. Beyond it, the request gets rejected with a 413 Payload Too Large.
We had to find out what, in our chain, was requiring the full body.
The culprit: our WAF
We had used the migration as an opportunity to add a WAF, Coraza, loaded into Envoy as a Wasm module, with the OWASP CRS rules. It was in DetectionOnly mode: it logs what it would have blocked, but blocks nothing. Or so we thought.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
name: coraza-waf
namespace: edge
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: envoy-gateway
wasm:
- name: coraza-waf
rootID: waf
failOpen: true
code:
type: Image
image:
url: ghcr.io/corazawaf/coraza-proxy-wasm:0.6.0
config:
default_directives: default
directives_map:
default:
- Include @recommended-conf
- Include @crs-setup-conf
- Include @owasp_crs/*.conf
- SecRuleEngine DetectionOnly
- SecRequestBodyAccess On
To inspect a request body (looking for a SQL injection in a form, for example), a WAF needs it in full. That's what SecRequestBodyAccess On does: Coraza asks Envoy to buffer the body before passing it on to the backend. That's exactly the case that turns the buffer limit into a hard cap.
And there's a second layer. Coraza's recommended configuration sets SecRequestBodyInMemoryLimit to 128 KiB, with SecRequestBodyLimitAction Reject. Beyond 128 KiB, the request is rejected.
The killer detail: this rejection doesn't go through the rule engine. So it applies even in DetectionOnly. Our "observation only" WAF was blocking every upload over 128 KiB.
The wrong answer
The instinctive move would be to raise Envoy's buffer limit, in the ClientTrafficPolicy:
spec:
connection:
bufferLimit: 1Gi
It works, partially: Envoy accepts keeping large bodies in memory. But it's a very bad idea.
This limit applies per connection, across every route in the cluster. A few concurrent large uploads, or a fast client facing a slow backend, and an Envoy pod can consume several gigabytes before getting OOM-killed. And all of the cluster's traffic flows through these pods. For comparison, Envoy's own guidance for an edge-facing proxy recommends staying around 32 KB.
And why not do like nginx and write to disk? Envoy does have a filter for that (file_system_buffer), but its documentation explicitly says: work in progress, not meant for production. Envoy Gateway doesn't even expose it.
The right answer
The problem wasn't the buffer size, it was that something demanded the full body. The fix is to tell Coraza to inspect the start of the body and let the rest stream through:
- SecRequestBodyAccess On
- SecRequestBodyLimitAction ProcessPartial
With ProcessPartial, Coraza analyzes the first 128 KiB and lets the rest flow without holding it back. Regular requests (forms, API JSON) still get inspected in full, and uploads go back to streaming.
This trade-off has a cost, and it's worth naming: an attacker can fill the first 128 KiB with harmless content, then place their payload after it. Coraza inspects the start, sees nothing, and lets the rest through without ever reading it. This is a well-known WAF bypass.
In DetectionOnly, it's acceptable: the WAF blocks nothing anyway, it's there for observation. But it's not the configuration to keep once you switch to blocking mode.
The goal is to not treat every route the same way. SQL, XSS or command injections go through form fields or JSON, which are a few kilobytes at most. A body of several hundred kilobytes on a route that isn't expecting a file is already suspicious on its own. Hence:
- strict inspection by default (
Reject), via anEnvoyExtensionPolicyattached to theGateway - a dedicated
EnvoyExtensionPolicy, attached only to uploadHTTPRoutes, usingProcessPartial - on those upload routes, protection carried by the application itself: file type, maximum size, antivirus scanning
Envoy's buffer limit went back down to a sane value, and Envoy pod memory no longer depends on uploaded file sizes.
The lesson goes beyond the WAF. With Envoy, when a large request body causes trouble, the right question isn't "which limit should I raise?" but "which filter needs the full body, and does it really need it?".
An ext_authz filter forwarding the body to an authorization service, a Lua filter, or another Wasm module can all produce the exact same effect.
Timeouts that don't mean the same thing
Second surprise, on long downloads and server-side processing that takes several minutes.
With nginx, proxy_read_timeout is a delay between two reads: as long as the backend keeps sending bytes regularly, the connection stays alive.
With Envoy, the route timeout is a total duration: the time given to the backend to send the complete response, starting from when the request was fully received. And its default value is 15 seconds.
An export that takes two minutes to generate? Cut off at 15 seconds, with a 504 and an upstream request timeout.
Gateway API lets you configure this per route rule:
rules:
- backendRefs:
- name: my-app
port: 8080
timeouts:
request: 5m
backendRequest: 5m
Rather than raising the value globally, we did it route by route, only where the need was real. A long timeout everywhere also means connections piling up when a backend is slow.
Websockets and custom protocols
With nginx, websockets worked with two timeout annotations. With Envoy, a connection upgrade is only accepted if it's explicitly allowed.
For an HTML5 remote desktop application that keeps a websocket connection open for the whole session:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: remote-desktop-session
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: remote-desktop
httpUpgrade:
- type: websocket
timeout:
http:
requestTimeout: 0s
connectionIdleTimeout: 3600s
requestTimeout: 0s disables the total timeout (a session can last hours), and connectionIdleTimeout cuts off genuinely idle sessions.
The most interesting case was Headscale, the open source implementation of the Tailscale control server. The Tailscale client doesn't perform a websocket upgrade: it requests an upgrade to its own protocol, tailscale-control-protocol, to establish an encrypted channel using the Noise protocol.
Nginx let through any Upgrade header. Envoy refuses it until it's been declared:
spec:
httpUpgrade:
- type: websocket
- type: tailscale-control-protocol
The symptom itself wasn't obvious at all: clients unable to register, and nothing informative on the application side. This is exactly the kind of implicit behavior you don't suspect until you switch proxies.
Goodbye oauth2-proxy
This is the part I enjoyed the most.
Envoy handles OIDC natively. Authentication is declared in a SecurityPolicy attached to a route:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
name: my-app-oidc
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: my-app
oidc:
provider:
issuer: "https://login.microsoftonline.com/${tenant_id}/v2.0"
clientID: "${client_id}"
clientSecret:
name: my-app-oidc
scopes:
- openid
- email
- profile
cookieConfig:
sameSite: Lax
Compared to the approach in my 2023 article, several things change:
- no more third-party service to deploy, monitor and keep updated
- authentication is declared per route, instead of a chain of annotations pointing to another service
- each application has its own OIDC client, so its own permissions. With oauth2-proxy, the cookie's scope (
*.staging.example.comin the old example) granted access to every service on the domain at once
There is a trap though, and not a small one.
By default, Envoy encrypts the tokens it stores in cookies. But browsers cap a cookie at roughly 4096 bytes (name and value combined).
For some users, whose access token carries a lot of claims (group memberships, for instance), the encrypted cookie exceeds that size. The browser silently drops it, and the user loops back to the login screen forever.
Only certain accounts were affected, which made the diagnosis fairly painful.
This is a structural limitation of Envoy Gateway's OIDC implementation: it only knows cookies as a session mechanism. oauth2-proxy, on the other hand, supports several storage backends, including Redis: the cookie then only holds a session identifier, and the full token stays server-side. Token size becomes a non-issue, at the cost of running a stateful service, the very thing we were trying to remove.
For internal services, we disabled token encryption (cookies stay HttpOnly):
oidc:
refreshToken: false
disableTokenEncryption: true
That's a trade-off. On a publicly exposed service, I'd rather reduce the token size on the identity provider side (fewer claims) than disable encryption.
EnvoyPatchPolicy: an escape hatch to use carefully
When a policy doesn't cover a need, Envoy Gateway offers an escape hatch: EnvoyPatchPolicy. It lets you apply a JSON Patch directly to the xDS configuration generated for Envoy.
It's powerful, and that's exactly the problem.
xDS resource names are generated by the controller, and can change between versions. A hand-written patch relies on a structure nobody guarantees. And crucially, it applies after every other policy.
We learned this the hard way: a replace-type patch on a listener's routes overwrote the configuration generated by an OIDC SecurityPolicy. Nothing in the Kubernetes objects showed it: the SecurityPolicy was accepted, its status was correct, but the configuration actually sent to Envoy no longer contained it.
My rule today:
EnvoyPatchPolicyonly as a last resort, with a comment explaining why no typed policy fits- as few
replaceoperations as possible, prefer targetedadd - verify the actual running configuration with
egctl config envoy-proxyafter every Envoy Gateway version bump
Conclusion
Technically, replacing ingress-nginx with Envoy Gateway isn't complicated. The manifests are more verbose, but more readable and schema-validated.
The real work is taking inventory of everything nginx was doing implicitly: buffering to disk, letting through any upgrade header, counting timeouts between reads rather than over total duration. None of this behavior showed up in our annotations, and yet our applications depended on it.
If you're planning the same migration, my advice: run both side by side, and prioritize testing large uploads, long downloads, websockets, and authentication with your accounts carrying the heaviest tokens.
