We Burned Two Weeks of Cloud Budget Because Our Autoscaler Was Stuck in a Scale-Up, Scale-Down Loop

By KP | TZoneLabs | DevOps & Cloud Engineering

For nine days, our cluster added and removed the same three worker nodes every six minutes. Every dashboard we watched stayed green the whole time. Pods were running, nodes were Ready, CPU never crossed 60%. Nobody noticed until the AWS bill landed about 40% higher than the month before, with no matching jump in traffic.

This post covers how we traced a cost spike back to two autoscalers quietly fighting each other, and what we changed so a pattern like this shows up as an alert instead of an invoice.

What Happened

A few weeks earlier, we’d shipped a background worker that polls a queue every five minutes and does a short burst of CPU-heavy processing, then goes idle. On its own, that’s a normal workload. Combined with our existing autoscaling setup, it wasn’t.

The Horizontal Pod Autoscaler (HPA) watched CPU on that deployment and scaled pods up during the burst, then scaled them back down a few minutes later once the burst passed. Each scale-down freed up just enough room that the Cluster Autoscaler decided a node was unneeded and removed it. Then the next burst came, the HPA scaled pods back up, there was nowhere to put them, and the Cluster Autoscaler added a node right back.

Add, remove, add, remove, every six minutes, for over a week.

Where We Started Looking

Nothing in Kubernetes or our APM tool flagged this. Pods were always eventually scheduled. Nodes were always eventually Ready. There was no error to alert on, because nothing was actually failing.

Step 1 — Notice the anomaly in Cost Explorer

The first signal came from a monthly AWS Cost Explorer review, not monitoring:

aws ce get-cost-and-usage \
  --time-period Start=2026-06-15,End=2026-06-30 \
  --granularity DAILY \
  --metrics "UnblendedCost" \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Elastic Compute Cloud - Compute"]}}'

EC2 spend was climbing day over day, but our request volume from Google Site Kit and the load balancer logs was flat. Something was consuming compute that wasn’t tied to traffic.

Step 2 — Compare instance-hours to actual usage

aws ce get-cost-and-usage \
  --time-period Start=2026-06-15,End=2026-06-30 \
  --granularity DAILY \
  --metrics "UsageQuantity" \
  --filter '{"Dimensions":{"Key":"USAGE_TYPE_GROUP","Values":["EC2: Running Hours"]}}'

Instance-hours were higher than a stable 3-node cluster should ever produce. That meant instances were being launched and terminated repeatedly, not just running longer.

Confirming the Pattern

Step 3 — Pull Auto Scaling Group activity history

aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name <your-asg-name> \
  --max-items 50 \
  --output table

The output was the clue. Dozens of “Launching a new EC2 instance” and “Terminating EC2 instance” events, spaced almost exactly six minutes apart, going back over a week.

Step 4 — Graph node count over time

kubectl get nodes -o json | jq -r '.items[] | .metadata.creationTimestamp'

Plotted against time, node count wasn’t flat at 3. It was a sawtooth: 3, 4, 3, 4, 3, 4, on a loop. A point-in-time kubectl get nodes never caught it, because whoever ran the command usually landed on a “3” moment.

Step 5 — Cross-reference with HPA events

kubectl describe hpa worker-queue-hpa -n processing

Events:
  Type    Reason             Age                From                       Message
  ----    ------             ----                ----                       -------
  Normal  SuccessfulRescale  6m (x312 over 9d)   horizontal-pod-autoscaler  New size: 6; reason: cpu resource utilization above target
  Normal  SuccessfulRescale  6m (x309 over 9d)   horizontal-pod-autoscaler  New size: 2; reason: All metrics below target

312 rescale events in nine days. The HPA and the Cluster Autoscaler were both doing exactly what they were configured to do. That was the problem.

Why It Kept Happening

🔴 Root cause: three settings resonating with each other.

The HPA’s downscale stabilization window was set to 60 seconds, so it scaled pods down almost as soon as CPU dropped.

The Cluster Autoscaler’s scale-down-unneeded-time was set to 2 minutes, so a node became eligible for removal almost as soon as it emptied out.

The queue worker’s five-minute polling interval meant a new CPU burst arrived right after the node was gone, forcing a fresh scale-up.

Each setting was reasonable on its own. Together, they turned a five-minute periodic job into a permanent scaling loop.

The Fix

Slow down both autoscalers

# Cluster Autoscaler: wait longer before removing an "unneeded" node
--scale-down-unneeded-time=10m
--scale-down-delay-after-add=10m

# HPA: widen the downscale stabilization window
behavior:
  scaleDown:
    stabilizationWindowSeconds: 300

Give the periodic workload fixed capacity instead of elastic capacity

The bigger fix was recognizing that a job with a predictable five-minute cycle shouldn’t be riding an autoscaler at all. We moved the queue worker onto a small node pool sized for its peak load and set minReplicas to cover the burst, so it no longer needed new nodes to show up on a schedule.

nodeSelector:
  workload-type: steady-burst
resources:
  requests:
    cpu: "1"
  limits:
    cpu: "1"

Within a day of the change, the ASG activity log went quiet and stayed there.

What We Put in Place After

1. Cost Anomaly Detection

aws ce create-anomaly-monitor \
  --anomaly-monitor '{"MonitorName":"EC2-Compute-Monitor","MonitorType":"DIMENSIONAL","MonitorDimension":"SERVICE"}'

Set to alert on unexplained day-over-day EC2 cost increases, not just monthly totals.

2. Alert on scaling activity frequency, not just scaling failures

aws cloudwatch put-metric-alarm \
  --alarm-name "ASG-Excessive-Scaling-Activity" \
  --metric-name "GroupTotalInstances" \
  --namespace "AWS/AutoScaling" \
  --statistic "SampleCount" \
  --period 3600 \
  --threshold 10 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --dimensions Name=AutoScalingGroupName,Value=<your-asg-name> \
  --alarm-actions <your-sns-topic-arn>

More than 10 scaling activities in an hour on a stable-traffic cluster is worth a page, even if nothing is broken.

3. A node-count-over-time panel, not a node-count-right-now number

We added a Grafana panel graphing node count as a time series instead of a single-value stat. A sawtooth pattern is invisible in a snapshot and obvious in a graph.

4. A rule of thumb for new periodic workloads

Anything with a predictable cycle under 10 minutes now gets reviewed for fixed capacity before it’s allowed to depend on the Cluster Autoscaler.

Key Lessons

Two autoscalers reacting to each other can produce a stable-looking oscillation.
Nothing crashed, so nothing paged. The system was “working” the entire time, just expensively.
A point-in-time check can miss a pattern that only shows up over time.
kubectl get nodes run once will show a normal number. The same command graphed over nine days shows a loop.
Cost is a monitoring signal, not just a monthly report.
By the time a bill is unusual enough to notice by eye, the underlying issue has usually been running for weeks.
Predictable, periodic workloads don’t belong on elastic capacity.
An autoscaler reacting to a fixed five-minute cycle will overreact to it every time.
Autoscaler settings interact with each other, not just with your workload.
Tuning the HPA and the Cluster Autoscaler in isolation missed the resonance between them.

Summary

Layer	What Happened	Tool to Check
Billing	EC2 spend rose with no matching traffic increase	`aws ce get-cost-and-usage`
Auto Scaling Group	Nodes launched and terminated every six minutes	`aws autoscaling describe-scaling-activities`
Horizontal Pod Autoscaler	Scaled pods up and down 312 times in nine days	`kubectl describe hpa`
Workload	A five-minute polling cycle matched the scale-down window	Deployment spec + queue worker logs
Kubernetes	Every layer reported healthy the whole time	`kubectl get nodes`, dashboards

Nothing here ever crossed an error threshold. The only threshold it crossed was the invoice.

What Do You Think?

Have you caught an autoscaler fighting itself, or another autoscaler, before it showed up on a bill?

Drop your experience in the comments. What finally surfaced it for you?