By KP | TZoneLabs | DevOps & Cloud Engineering
Installing Cluster Autoscaler is one Helm command. Getting it to scale up fast enough and scale down without evicting something it shouldn’t is the part that takes actual configuration. The defaults are conservative on purpose, and left untouched they either leave pending pods waiting longer than necessary or leave nodes running long after their workloads are gone.
This walks through tagging node groups so it can find them, the scale-down settings worth changing from default, and the mistakes that turn an autoscaler into something fighting your workloads instead of matching capacity to them.
Tagging Node Groups So It Can Find Them
Cluster Autoscaler discovers Auto Scaling Groups by tag, not by name. Every node group you want it managing needs both of these on the underlying ASG:
aws autoscaling create-or-update-tags --tags \
ResourceId=my-node-group-asg,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/enabled,Value=true,PropagateAtLaunch=true \
ResourceId=my-node-group-asg,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/my-cluster-name,Value=owned,PropagateAtLaunch=true
If you’re managing node groups through eksctl or Terraform’s EKS module, these tags are usually applied automatically. Worth confirming directly on the ASG the first time, since a missing tag here doesn’t throw an error, it just means that node group silently never scales.
Installing via Helm
helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
--namespace kube-system \
--set autoDiscovery.clusterName=my-cluster-name \
--set awsRegion=us-east-1 \
--set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="arn:aws:iam::123456789012:role/cluster-autoscaler-role"
The service account needs an IAM role (via IRSA) with permission to describe and modify Auto Scaling Groups. Skip this and the pod runs, connects to nothing useful, and logs permission errors that are easy to miss if nobody’s looking at that namespace yet.
Sizing min and max Per Node Group
aws autoscaling update-auto-scaling-group \
--auto-scaling-group-name my-node-group-asg \
--min-size 2 \
--max-size 20
Setting min equal to max on any group defeats the entire point, that group will never scale in either direction regardless of what Cluster Autoscaler decides. Also avoid one giant node group spanning several instance types with different CPU and memory ratios. The autoscaler has to guess which shape a pending pod needs, and a group with mismatched instance types makes that guess worse, not more flexible.
Scale-Down Settings Worth Changing From Default
extraArgs:
scale-down-enabled: "true"
scale-down-unneeded-time: "10m"
scale-down-utilization-threshold: "0.5"
skip-nodes-with-local-storage: "false"
skip-nodes-with-system-pods: "true"
scale-down-unneeded-time is how long a node has to sit underutilized before it’s a scale-down candidate, 10 minutes by default. Shortening it scales down faster but risks removing a node right before a burst of traffic needs it back. scale-down-utilization-threshold is the ceiling below which a node counts as underutilized, 0.5 by default, meaning a node using less than half its allocatable capacity is a candidate. Raise it and you scale down more aggressively; lower it and nodes stick around longer than they’re doing useful work.
Protecting Pods From Being Evicted at the Wrong Time
Any pod without a PodDisruptionBudget and without an explicit opt-out is fair game for eviction during scale-down. For anything that shouldn’t move on short notice, whether a stateful workload or a job mid-run, be explicit:
apiVersion: v1
kind: Pod
metadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: api
The annotation stops that specific pod from being evicted at all. The PodDisruptionBudget is the broader guardrail, it caps how many replicas of a deployment can go down at once from voluntary disruptions, autoscaler scale-downs included.
Picking an Expander Strategy
When more than one node group could satisfy a pending pod, the expander setting decides which one gets picked:
extraArgs:
expander: least-waste
least-waste picks the node group that leaves the least unused capacity after the pod is scheduled, a reasonable default for most clusters. priority is worth switching to once you have node groups you actively want preferred, spot instances before on-demand, for instance, using an explicit priority ConfigMap rather than leaving the choice to whichever group wastes least.
Verifying It’s Actually Working
kubectl logs -n kube-system deployment/cluster-autoscaler --tail=100
kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml
The status ConfigMap shows the last scan time and the current state of every node group it’s tracking. If a node group you expect it to manage isn’t listed there, that’s the tagging step to revisit, not the scaling logic.
Common Mistakes
- One node group with mixed instance types. Bin-packing decisions get worse, not more flexible, when the autoscaler can’t predict what capacity a new node actually provides.
- No PodDisruptionBudgets on anything that matters. Without one, a scale-down event can take down every replica of a deployment simultaneously if they all happen to land on the same node.
minequal tomaxleft over from initial setup. A group that can never grow or shrink isn’t being autoscaled, it’s just a fixed-size group with extra steps.- Missing IRSA permissions caught only when a pending pod never gets a node. Check the pod’s logs at install time, not after the first real capacity crunch.
Key Lessons
-
ASG tags, not names, are how Cluster Autoscaler finds node groups.
A missing tag fails silently, the group just never scales, with no error pointing at the cause. -
Mixed instance types in one node group make bin-packing worse.
Separate node groups by instance shape so the autoscaler’s capacity guess is actually accurate. -
Scale-down defaults are conservative, not wrong.
Tunescale-down-unneeded-timeand the utilization threshold to your traffic pattern instead of assuming the defaults fit. -
Anything that shouldn’t move needs an explicit annotation or a PDB.
Without one, a pod is eligible for eviction during scale-down whether or not that’s actually safe. -
The status ConfigMap tells you what it’s actually tracking.
Check it before assuming a scaling problem is about thresholds when it might be about discovery.
Summary
| Setting | Purpose | Watch For |
|---|---|---|
| ASG tags | Node group discovery | Missing tag = silent no-op, not an error |
scale-down-unneeded-time |
How long before a node is scale-down eligible | Too short risks removing capacity before a burst |
safe-to-evict / PDB |
Protects specific pods during scale-down | Unset by default, not opt-out |
| Expander | Which node group wins when several qualify | Switch to priority once you have a group you actually prefer |
None of these settings are exotic. Most clusters just never revisit the defaults after the first Helm install, and the defaults were never tuned to any specific traffic pattern, including yours.
Read Next
- We Burned Two Weeks of Cloud Budget Because Our Autoscaler Was Stuck in a Scale-Up, Scale-Down Loop
- Your EKS Nodes Run Out of IPs, Not CPU: A Practical Guide to VPC CNI Warming and Prefix Delegation
If you’re running Kubernetes in production, follow along on LinkedIn for more guides like this one as they’re published.
Tags:
#Kubernetes #ClusterAutoscaler #AWS #EKS
#DevOps #SRE