A Traffic Spike Exhausted Our Database Connection Pool, and Generic Timeout Errors Sent Us Debugging the Wrong Layer for an Hour

By KP | TZoneLabs | DevOps & Cloud Engineering

During a traffic spike, every app server started throwing the same generic message: ETIMEDOUT. Nothing named the database. Nothing named the connection pool. So for the first hour, we debugged the network, the load balancer, and the application code, in that order, before anyone checked how many database connections were actually in use.

This post covers why the real bottleneck hid behind a vague error, and what we changed so the next timeout points at its actual cause.

What Happened

A marketing campaign drove a traffic spike roughly four times normal peak load. The application layer itself scaled fine, horizontal pod autoscaling added replicas within the expected window, and CPU and memory on the app pods stayed well within limits.

What didn’t scale was the database connection pool. Each app pod maintained a fixed-size pool of 20 connections to the database. As pod count went from 6 to 24 under the autoscaler, total connections requested went from 120 to 480, well past the database’s max_connections limit of 200.

Where We Started Looking

The error message was the problem. ETIMEDOUT from a Node.js app can mean a dropped network connection, a slow downstream service, a full thread pool, or a database that’s refusing new connections. It doesn’t say which.

Step 1 — Check the network path first (the wrong first move)

kubectl exec -it app-pod-xyz -n production -- curl -v --max-time 3 https://internal-db-proxy:5432

The network path was fine. This ruled out the layer we checked first, but cost 15 minutes we didn’t need to spend.

Step 2 — Check application-level metrics

kubectl top pods -n production | sort -k3 -rn | head -10

CPU and memory were normal across every pod. Nothing here explained a timeout, so the assumption shifted to “maybe it’s a slow query,” which was also wrong, and cost another 20 minutes writing and running EXPLAIN ANALYZE on the most frequent queries.

Step 3 — Finally check the database’s own connection count

SELECT count(*) FROM pg_stat_activity;

count
-------
  200

Exactly at max_connections. This is where the investigation should have started.

Step 4 — Confirm the source of the connection demand

SELECT client_addr, count(*)
FROM pg_stat_activity
GROUP BY client_addr
ORDER BY count(*) DESC;

Connections were spread evenly across all 24 app pods, roughly 8 each, not concentrated on one misbehaving instance. This wasn’t a leak. It was every pod correctly requesting its configured pool size, at a scale the database was never sized for.

Why the Error Pointed Nowhere

🔴 Root cause: pool size scaled with pod count, but the database ceiling didn’t move.

Each pod’s connection pool size (20) was set based on single-pod load testing, without accounting for what happens when the autoscaler adds more pods.

The database’s max_connections was set once, years earlier, and never revisited as the fleet grew.

A generic ETIMEDOUT is what a client sees when a connection request queues indefinitely waiting for a database slot that never frees up. It looks identical to a network-level timeout from the application’s side.

The autoscaler did exactly what it was supposed to do. It just multiplied a per-pod setting that was never meant to be multiplied.

The Fix

Immediate: reduce per-pod pool size and restart

const pool = new Pool({
  max: 8, // was 20
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 5000,
});

Real fix: put a connection pooler in front of the database

# PgBouncer in transaction pooling mode
[databases]
appdb = host=db-primary port=5432 dbname=appdb

[pgbouncer]
pool_mode = transaction
max_client_conn = 2000
default_pool_size = 25

PgBouncer sits between the app pods and the database, accepting up to 2,000 client connections while only maintaining a small, fixed number of real connections to Postgres. Pod count can now scale independently of the database’s actual connection ceiling.

What We Put in Place After

1. A CloudWatch/Postgres alert on connection count, not just CPU

-- Exported as a custom metric via postgres_exporter
SELECT count(*) * 100.0 / current_setting('max_connections')::int AS pct_used
FROM pg_stat_activity;

Alert at 70% of max_connections, well before the database starts refusing new connections outright.

2. Load testing that scales pod count, not just request rate

Previous load tests held pod count fixed and only increased request volume. We added a test scenario that scales pods to their configured HPA maximum and holds them there, to catch connection math that only breaks at full fleet size.

3. Error messages that name the actual resource

pool.on('error', (err) => {
  logger.error('Database pool exhausted', {
    poolTotal: pool.totalCount,
    poolIdle: pool.idleCount,
    poolWaiting: pool.waitingCount,
    error: err.message,
  });
});

The next timeout logs pool state alongside the error, so “waiting: 12, idle: 0” is visible immediately instead of inferred an hour later.

Key Lessons

A generic timeout error and a resource exhaustion error look identical from the client side.
ETIMEDOUT doesn’t distinguish a dropped connection from a full connection pool.
Per-pod settings that get multiplied by autoscaling need their own capacity math.
A pool size validated at 6 pods is a different number entirely at 24.
Debug from the resource most likely to have a hard ceiling first.
Databases have fixed connection limits. Application servers usually don’t. Check the fixed limit early, not last.
A connection pooler decouples application scaling from database scaling.
Without one, every autoscaling event is also, silently, a database capacity event.
Load test at your autoscaler’s maximum, not just at current steady-state.
Problems that only appear at full fleet size don’t show up in a test that never reaches full fleet size.

Summary

Layer	What Happened	Tool to Check
Network	Checked first, found healthy, cost 15 minutes	`curl` from inside the pod
Application	CPU and memory normal, queries not actually slow	`kubectl top`, `EXPLAIN ANALYZE`
Database	Connection count pinned at `max_connections`	`pg_stat_activity`
Autoscaler	Multiplied a per-pod setting nobody had recalculated	HPA config vs. pool size

The bottleneck was never hidden. It just wasn’t where the error message pointed.