The availability of Kubernetes control planes, despite their design to isolate individual failures, remains contingent upon the behavior of components operating along the critical path under stress.
At fleet scale, an outage does not necessitate a component crash. A functional mechanism can amplify a localized defect through shared metadata, synchronized initialization tasks, or excessively aggressive health checking. For instance, an informer provides a controller with a local cluster state view, enabling worker progress without constant API server querying, which is efficient during independent controller starts.
However, when numerous controllers restart simultaneously across a fleet, reconstructing these views can impose substantial strain on the control plane. The informer mechanism itself becomes the point of amplification. As stated, "At fleet scale, a component does not have to crash to contribute to an outage. A valid mechanism can amplify a local defect through shared metadata, synchronized startup work, or an overly aggressive health check."
This analysis details three specific scenarios. The first involved an extension API server publishing an invalid schema following a dependency upgrade, blocking clients that had never interacted with that API. The second involved DaemonSet agents restarting across a large deployment, transforming routine startup work into an overwhelming request storm for the API server. The third case saw a deep liveness check on an API request path component treating a dependency outage as a process failure, triggering repeated restarts while the dependency remained unavailable.
These findings are relevant for teams managing extension APIs, controllers monitoring extensive object collections, and components within the API request path that utilize deep dependency validation.
In the first example, Helm deployments began failing for charts containing only standard Kubernetes resources, even though they were unrelated to the failing extension API. Users encountering the issue saw validation errors prior to requests reaching the API server. This failure path traversed metadata within a schema document aggregating definitions from APIs registered in the cluster.
Kubernetes can be extended via aggregated API servers, which introduce new APIs without modifying the core server. To users, this aggregated API appears native, possessing group, version, kind, and resource paths accessible via the cluster's standard endpoint. An extension API server generates an OpenAPI schema describing its objects, which clients use for pre-request validation.
The difference between OpenAPI v2 and v3 impacts the scope of failure. OpenAPI v2 uses a single document at /openapi/v2 containing core and aggregated extension API definitions. Conversely, OpenAPI v3 separates schemas by API group and version, which limits the impact on core API clients. As noted, "The v2 format creates a shared failure domain: an invalid definition from one extension can prevent clients from parsing the document used to validate unrelated resources."
A modification in Kubernetes PR #131755 made OpenAPI model names explicit. Previously, names were derived from Go package paths, linking them to implementation details. The new approach requires API authors to declare a stable model package, which code generation then uses for schema definitions and references.
One extension API server upgraded its Kubernetes API machinery dependency but omitted the required +k8s:openapi-model-package annotation. Consequently, its generated code supplied raw Go package paths when building schema definitions. Since the newer machinery expects a declared model package, these paths were published as identifiers, rendering the resulting schema invalid due to the presence of '/'.
Process health checks do not validate the published OpenAPI document. The issue manifested in clients reading OpenAPI v2 because the combined document allowed an invalid definition from one extension to cause an error before a request for a different resource was sent.
The second incident demonstrated how a rolling restart can induce a request storm. DaemonSet agents, which run one Pod per node, maintain cluster views via informers. When recovery synchronizes the work, many agents start almost simultaneously, each initiating an informer that begins by listing the same Pod collection.
Informers typically use resourceVersion=0 for their initial request, allowing the API server to respond from its in-memory watch cache rather than performing a full etcd read, which is usually fast. However, a limitation existed: the watch cache could not paginate like etcd lists. When the list was served from the watch cache with resourceVersion=0, the API server could ignore the client's limit parameter.
On a large cluster, a full Pod collection response could be massive, and when thousands of replacement agents request this simultaneously, the server risks exhausting memory and CPU constructing numerous identical responses, thereby stalling normal control plane traffic.
The Kubernetes community has addressed this with fixes such as correcting API Priority and Fairness list-cost estimation in Kubernetes 1.31 and introducing WatchList. Nevertheless, the fundamental design calculation remains: a cluster-wide informer in a DaemonSet scales its startup cost by the node count, and retry policies can synchronize this cost during failures.
These documented failure modes highlight systemic vulnerabilities in how distributed components interact with shared metadata and API contracts, emphasizing the need for rigorous validation beyond basic endpoint health checks for platform operators.
