Setting Up Horizontal Pod Autoscaler on Kubernetes: The Settings That Actually Matter

Setting Up Horizontal Pod Autoscaler on Kubernetes: The Settings That Actually Matter

by

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering

The default Horizontal Pod Autoscaler config scales on raw CPU with no stabilization window, which is exactly why it’s common to see replica counts bounce up and down every few minutes under normal, slightly-uneven traffic. The two settings that actually decide whether HPA is useful or annoying are the target utilization threshold and the behavior stabilization window — not the metric you pick.

This covers a basic CPU-based HPA, why the defaults thrash, tuning behavior to stop it, and scaling on a custom metric when CPU isn’t the right signal.

Prerequisite: metrics-server Has to Be Running

HPA reads CPU and memory from metrics-server, not from the kubelet directly. Confirm it’s installed and returning data before writing any HPA manifest:

kubectl top pods -n default

If that returns “error: Metrics API not available,” fix that first — an HPA pointed at a metrics source that isn’t there just sits idle with no scaling decisions at all, and gives no error to say why.

A Basic CPU-Based HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

This only works if the target Deployment’s containers have resources.requests.cpu set. HPA computes utilization as a percentage of the request, not an absolute value — with no request defined, there’s nothing for 70% to be 70% of, and the HPA will report <unknown> for current utilization instead of scaling.

Why Default HPAs Thrash

Scale-up reacts fast by design — a spike in traffic should get more pods quickly. Scale-down is deliberately more conservative, but the default stabilization window (5 minutes) still isn’t long enough for CPU that oscillates around the target threshold. A workload sitting near 70% utilization can cross the line every couple of minutes in either direction, and every crossing is a scaling event.

The fix isn’t a better metric — it’s telling the HPA how much noise to tolerate before it acts.

Tuning behavior to Stop the Flapping

spec:
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30

stabilizationWindowSeconds on scaleDown makes the HPA look at the highest recommended replica count over that whole window before scaling down, instead of reacting to the most recent data point. The policies block caps how much it can scale in either direction per period, so a single noisy sample can’t swing replica count by more than a controlled amount.

Scaling on a Custom Metric Instead of CPU

CPU is a poor proxy for anything I/O-bound or queue-driven — a worker pulling from a queue can be CPU-idle while badly behind on backlog. With the Prometheus Adapter exposing a custom metric, the HPA can scale on that instead:

spec:
  metrics:
    - type: External
      external:
        metric:
          name: queue_messages_ready
          selector:
            matchLabels:
              queue: orders
        target:
          type: AverageValue
          averageValue: "30"

This scales the Deployment to keep roughly 30 ready messages per pod, which is a much more direct signal for a queue worker than any CPU threshold would be.

Mistakes That Show Up Later, Not Immediately

  • No resource requests on the target Deployment. CPU-based HPA silently reports unknown utilization and never scales, with nothing in the HPA’s own status pointing at the missing request as the cause.
  • minReplicas set to 1. The first pod under a cold start absorbs a traffic spike alone until the HPA reacts, which is often exactly when latency matters most.
  • HPA and Cluster Autoscaler fighting over capacity. HPA can decide to add pods that Cluster Autoscaler then can’t schedule because there’s no node capacity yet, producing a period where the desired replica count and the running replica count disagree and everything looks stuck.
  • Scaling on a metric that’s a side effect, not a cause. CPU can spike for reasons unrelated to load (a GC pause, a background job). Confirm the metric actually correlates with the thing that needs more capacity.

Key Lessons

  1. HPA needs resources.requests set on the target, or CPU-based scaling never triggers.
    Utilization is a percentage of the request, not an absolute number.
  2. The default stabilization window is too short for CPU that oscillates near the threshold.
    That’s what causes replica counts to flap under normal, uneven traffic.
  3. behavior.scaleDown.stabilizationWindowSeconds fixes flapping by looking at a window of history, not the latest sample.
  4. CPU isn’t the right signal for I/O-bound or queue-driven workloads.
    A custom or external metric via an adapter scales on the thing that actually matters for that workload.
  5. HPA and Cluster Autoscaler need to agree on capacity, or scaling decisions stall waiting on nodes that haven’t provisioned yet.

Summary

Decision Do Avoid
Resource requests Set requests.cpu on every container the HPA targets Leaving requests unset and expecting utilization to compute
Flapping Tune behavior.scaleDown.stabilizationWindowSeconds Leaving defaults on a metric that oscillates near the threshold
Cold starts Set minReplicas above 1 for latency-sensitive services minReplicas: 1 on anything user-facing
Metric choice Scale on the metric that causes the bottleneck (queue depth, latency) Defaulting to CPU for I/O-bound or queue-driven workloads

Read Next

If you’re running Kubernetes in production, follow along on LinkedIn for more guides like this one as they’re published.

Tags:
#Kubernetes   #HPA   #Autoscaling   #AWS  
#DevOps   #SRE

Leave a Comment