A Missing VPC Endpoint Routed Every Image Pull Through NAT Gateway, and Our AWS Bill Jumped 9x

By KP | TZoneLabs | DevOps & Cloud Engineering

Our AWS bill for a new EKS environment came in nine times higher than the month before, and nothing about our traffic had changed. The entire difference was NAT Gateway data processing — a private-subnet VPC with no Gateway Endpoint for S3 was routing every ECR image pull and every application write to S3 through a NAT Gateway at $0.045/GB, instead of the free path one Terraform resource would have given it.

This post covers how we traced a cost spike back to a single missing endpoint, why nothing alerted on it for weeks, and what we changed so this shows up on day one instead of on next month’s invoice.

What Happened

We stood up a new EKS cluster for a services team in a private-subnet-only VPC — no public IPs on nodes, NAT Gateways for outbound egress, standard setup. Nodes pulled container images from ECR. The application wrote objects to S3 on every request. Everything worked. No errors, no failed deploys, no degraded requests.

Three weeks later, finance flagged the AWS invoice: NAT Gateway costs for that account were 9x the prior month. Nobody on the team had seen an alert, because nothing had actually broken.

Where We Started Looking

Step 1 — Confirm what’s actually driving the cost

aws ce get-cost-and-usage \
  --time-period Start=2026-08-01,End=2026-09-01 \
  --granularity MONTHLY \
  --metrics "UnblendedCost" \
  --group-by Type=DIMENSION,Key=USAGE_TYPE

One line item dwarfed everything else: USE1-NatGateway-Bytes. Not EC2, not EBS, not data transfer out to the internet — data processed by the NAT Gateway itself.

Step 2 — Confirm which VPC and which traffic

aws ec2 describe-flow-logs \
  --filter Name=resource-id,Values=vpc-0a1b2c3d

Flow logs were already enabled from a prior audit. Querying them in Athena for the busiest destinations through the NAT Gateway’s ENI showed almost all of the bytes going to IP ranges that belonged to Amazon S3, not to any external service we called.

Step 3 — Check the route tables

aws ec2 describe-route-tables \
  --route-table-ids rtb-0f1e2d3c \
  --query 'RouteTables[*].Routes'

[
  { "DestinationCidrBlock": "10.0.0.0/16", "GatewayId": "local" },
  { "DestinationCidrBlock": "0.0.0.0/0", "NatGatewayId": "nat-0abc123" }
]

Every private subnet had exactly one route besides local: everything, including S3 traffic, went to the NAT Gateway. There was no route to an S3 prefix list at all.

Step 4 — Check for a VPC endpoint

aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=vpc-0a1b2c3d

{
  "VpcEndpoints": []
}

No S3 Gateway Endpoint. Not disabled, not misconfigured — never created.

🔴 Root cause: the VPC’s Terraform module never provisioned an S3 Gateway Endpoint, so all S3-bound traffic went through the NAT Gateway instead of AWS’s internal network.

The module was copied from an older environment that predated the team’s S3-endpoint convention, and the resource was never carried over.

ECR image layers are stored in and served from S3, so every docker pull on every node — at cluster startup, on every deploy, on every autoscale event — counted as NAT Gateway data processing.

The application’s own S3 writes added to the same bill, at the same per-GB rate, for traffic that should never have left AWS’s internal network in the first place.

None of this produced an error. NAT Gateways don’t reject traffic because a cheaper path existed — they just process it and meter it.

The Fix

Immediate: add the S3 Gateway Endpoint

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private[*].id
}

Gateway Endpoints for S3 are free — no hourly charge, no per-GB charge. Traffic to S3 started routing internally the moment this applied, and the NAT Gateway line item dropped back to normal on the next day’s cost report.

Longer-term: interface endpoints for ECR

resource "aws_vpc_endpoint" "ecr_api" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ecr.api"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "ecr_dkr" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

These carry a small hourly and per-GB charge of their own, but at cluster scale they’re a fraction of what NAT Gateway processing cost for the same pulls — and they keep image pulls off the public path entirely, not just off the metered one.

What We Put in Place After

1. S3 and ECR endpoints made mandatory in the shared VPC module

Every new private-subnet VPC now provisions these endpoints by default, using the same module structure we standardized in structuring Terraform modules and remote state. A new environment can’t skip them without an explicit override, which itself requires review.

2. A Cost Anomaly Detection monitor scoped to NAT Gateway usage

aws ce create-anomaly-monitor \
  --anomaly-monitor '{
    "MonitorName": "nat-gateway-usage",
    "MonitorType": "DIMENSIONAL",
    "MonitorDimension": "SERVICE"
  }'

Scoped narrowly enough that a NAT Gateway cost jump generates its own alert instead of blending into total AWS spend, where a 9x increase on one line item barely moves the overall number.

3. A pre-launch VPC checklist

Before any new private-subnet VPC goes live: confirm an S3 Gateway Endpoint exists, confirm ECR interface endpoints exist if the VPC will run containers, and confirm route tables actually point at them. Fifteen minutes now instead of a surprise invoice later.

4. Cost review segmented by usage type, not just by service

Our monthly cost review previously grouped by service (EC2, S3, VPC). We added a usage-type breakdown specifically for VPC-related costs, since “VPC” as a service line hides NAT Gateway hourly charges, data processing, and endpoint costs inside one number.

Key Lessons

NAT Gateway costs scale invisibly with normal usage.
Nothing errors, nothing pages, no request fails. The infrastructure just gets more expensive as traffic grows through the wrong path.
ECR pulls are S3 traffic under the hood.
Any private cluster pulling images needs a non-NAT path to S3, or every deploy and every autoscale event pays the NAT tax.
A VPC module that “works” can still be missing a cost-critical resource.
Functional correctness and cost correctness are different bars. This module passed every test that checked whether pods could reach the internet.
Cost anomalies reach finance before they reach engineering.
The team that could actually fix the problem heard about it three weeks late, from a department that doesn’t read Terraform.
Gateway Endpoints for S3 and DynamoDB are free.
There’s no cost tradeoff that justifies leaving them out of a private VPC template — only the cost of forgetting.

Summary

Layer	What Happened	Tool to Check
Cost Explorer	NAT Gateway data processing was 9x the prior month	`aws ce get-cost-and-usage`
VPC Flow Logs	Nearly all NAT traffic was destined for S3 IP ranges	Flow Logs + Athena query
Route Tables	Only a default route to the NAT Gateway existed	`aws ec2 describe-route-tables`
VPC Endpoints	S3 Gateway Endpoint was never provisioned	`aws ec2 describe-vpc-endpoints`
Terraform Module	Endpoint resource was missing from the copied module	Module review / plan diff

The cluster worked perfectly the entire time. The bill was the only thing that was broken, and it took a human reading an invoice to notice.