javascript —

How Google Traced GKE API Timeouts to a Linux Kernel TCP Bug

In January, a customer running multiple GKE clusters with Private Service Connect (PSC) for private control-plane access experienced intermittent timeouts when pods attempted to reach the Kubernetes API server.

How Google Traced GKE API Timeouts to a Linux Kernel TCP Bug

Packet captures disclosed that the control plane was responding to new TCP connection requests with stale ACK packets belonging to previously closed connections. The root cause involved three separate bugs across different networking layers.

And the final fix landed not as a config change or a feature flag, but as a patch to the Linux kernel itself. The investigation identified three interacting causes: packet reordering in PSC, Envoy's half-close behavior preventing TCP connections from being properly closed, and a Linux kernel bug that discarded certain reordered FIN packets. Thakur describes a complex troubleshooting case involving I/O timeouts across Private Service Connect, virtual network interfaces, and the Linux TCP state machine, which ultimately disclosed a kernel issue leaving sockets stuck indefinitely in the FIN_WAIT_2 state.

Earlier coverage

Reported by infoq.com.

javascriptcloud

Related coverage

Elsewhere in this edition