When an SRE is asked about the optimal configuration for checkout-api, the answers derived from various cluster components can contradict one another. Helm values initially specified replicas at 3, CPU at 1000m, and memory at 2Gi. However, the live cluster presented conflicting data: HPA scaled the application to 11 replicas based on CPU load; VPA suggested it was over-provisioned, recommending CPU at 350m and memory at 3.1Gi; KEDA required 24 replicas due to an upstream Kafka lag spike; and Karpenter added two new nodes, sizing physical capacity based strictly on declared requests, not actual usage.
"Five sources yield five contradictory answers. Each controller acts logically in isolation, but none sees the complete picture." This disparity between isolated observations and the actual workload requirement constitutes a fundamental architectural shortcoming in contemporary Kubernetes deployments.
The architecture of Kubernetes and its components was founded upon the Unix philosophy: creating small, modular utilities that excel at a single task. This concept seemed logical for container orchestration, with HPA managing horizontal elasticity via runtime resource usage, VPA handling vertical sizing based on historical consumption, KEDA connecting external queue lag to replica counts, and Karpenter/Cluster Autoscaler managing node capacity for pending pods.
Unlike standard Unix environments where tools like awk and grep process immutable text streams sequentially, autoscaling controllers operate concurrently. They all act upon the identical target workload and modify the same API fields, such as .spec.replicas and .resources.requests. Each controller runs an independent reconciliation loop, assuming sole authority over workload sizing without awareness of neighboring tools.
"Determining workload capacity is a single, coupled mathematical equation. Splitting it across decoupled controllers creates thrashing instead of modular efficiency." This independence leads to four common destabilizing feedback loop modes when independent control loops manipulate interdependent variables.
One failure mode is the denominator collapse, occurring between HPA and VPA. HPA uses declared requests as its denominator when calculating target replicas based on pod CPU utilization. When VPA concurrently lowers CPU requests to reduce waste, it shrinks this denominator. A slight traffic uptick then pushes utilization past HPA's threshold, triggering an aggressive scale-out, which subsequently causes VPA to shrink requests further, creating a spiral of microscopic containers.
A second issue is the multi-writer race condition between native HPA and KEDA. If an event queue spikes, KEDA scales the deployment to handle the backlog. Yet, if those workers consume little CPU despite being I/O-heavy, the standalone HPA sees low utilization and scales the application back down, resulting in alternating API server events while throughput stalls.
The third collision involves GitOps drift ping-pong. When HPA scales a deployment during peak traffic, Argo CD registers the application as OutOfSync. If auto-sync is active, the GitOps system reverts the live deployment to its baseline, denying the application capacity precisely when it is needed.
Finally, the node layer suffers from the multiplier effect. Node autoscalers like Karpenter base placement decisions strictly on declared container requests, not actual consumption. If a container requests 1000m CPU but uses only 150m, the scheduler reserves the full core, causing Karpenter to provision physical instances for phantom demand and inflating cloud expenditures.
The operational necessity transcends narrow questions like "What should my individual setting be?" A true autoscaler must address the broader query: "What combination of replicas, resource requests, limits, and scaling thresholds satisfies latency SLOs at the lowest cost, and when should that configuration change?"
A genuine autoscaler requires five core capabilities: unified signal ingestion covering metrics, queue depth, latency SLOs, and IaC manifests; joint decision-making to calculate a single action vector for horizontal and vertical settings; continuous GitOps synchronization through reasoned pull requests; predictive capacity planning to pre-warm resources before demand arrives; and explainable auditing for every change.
Current autoscalers are merely automated, functioning like reflexes that react to isolated telemetry signals. However, true autonomy demands a closed loop that comprehends the entire operational milieu, balancing performance against cost, and assuming complete accountability for the outcome.
The Unix philosophy should guide the execution layer, allowing HPA, VPA, KEDA, and Karpenter to perform their specific API mutations. What is absent in Kubernetes is the central intelligence layer responsible for directing these actuators. Platform teams must shift their focus from debating individual controller implementation to determining who possesses the full context for workload sizing.
