A Killed CI Job Left a Terraform State Lock Behind, and It Blocked Every Pipeline for 40 Minutes

By KP | TZoneLabs | DevOps & Cloud Engineering

Every terraform plan across three separate pipelines started failing with the same message: Error acquiring the state lock. Nothing was running. The lock had been held by a CI job that got killed by a pipeline timeout forty minutes earlier, and it never got the chance to release it. We spent the first stretch of that outage looking for a rogue apply that didn’t exist.

This post covers how a killed job left a lock nobody was holding, why the error looked like concurrent access when it wasn’t, and what we changed so it can’t quietly block every pipeline again.

What Happened

We use an S3 backend for Terraform state with a DynamoDB table for locking, the standard setup. A scheduled infrastructure pipeline kicked off a terraform apply against a moderately large module, a provider API call hung, and the job ran past its configured CI timeout. The runner killed the process outright.

Terraform releases its state lock as part of a normal exit. A process killed mid-run by the CI scheduler doesn’t get that chance. The lock record sat in DynamoDB, pointing at a job that no longer existed, and every subsequent plan or apply against that state file, from any pipeline, failed immediately.

Where We Started Looking

Step 1 — Read what the error actually said

Error: Error acquiring the state lock

Lock Info:
  ID:        7f3a2e91-4b6c-4d8a-9c1e-2a5f8e7d3b90
  Path:      infra-prod/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner-847291@ci-shared-runner-03
  Version:   1.6.6
  Created:   2026-07-19 14:22:07 UTC
  Info:

The first read of this looks exactly like what it’s designed to warn about: someone else is running apply right now, wait for them. So the first move was to check for a running job.

Step 2 — Check for any pipeline actually in progress

curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.com/api/v4/projects/$PROJECT_ID/pipelines?status=running"

Nothing. No running pipeline, no running job, nothing holding a terminal open with an apply mid-flight. That ruled out the obvious explanation and cost fifteen minutes confirming a negative.

Step 3 — Check the timestamp on the lock, not just its existence

The lock’s Created field was 40 minutes old. Cross-referencing that timestamp against the CI job history for runner-847291 showed the job it belonged to had already finished, marked as failed: job execution timeout, thirty-nine minutes earlier.

Step 4 — Read the lock directly out of DynamoDB

aws dynamodb get-item \
  --table-name terraform-locks \
  --key '{"LockID": {"S": "infra-prod/terraform.tfstate"}}'

The item was still there, untouched, exactly matching the ID in the error. Terraform’s lock table has no TTL by default. A lock that’s never explicitly released sits there indefinitely, whether the process that created it is still alive or not.

🔴 Root cause: a hard process kill skips Terraform’s lock release, and DynamoDB doesn’t expire stale locks on its own.

The CI job’s timeout killed the terraform process with a hard stop, not a signal it could catch and clean up after.

Terraform only releases the DynamoDB lock item as part of a graceful exit. A killed process never reaches that code path.

The lock table has no expiration on lock items, so a lock from a dead process looks identical to a lock from a live one.

Every pipeline touching that state file was blocked by a job that no longer existed, and the error message gave no indication of that.

The Fix

Immediate: force-unlock, after confirming nothing is actually running

# Only after Step 2 confirms no pipeline is currently running against this state
terraform force-unlock 7f3a2e91-4b6c-4d8a-9c1e-2a5f8e7d3b90

Running this while a real apply is in progress corrupts state. It’s a manual, deliberate action, not something to script into an automatic recovery path.

Real fix: give the job time to clean up before it’s killed

terraform-apply:
  timeout: 25m
  script:
    - trap 'terraform force-unlock -force "$(terraform show -json | jq -r .lock_id 2>/dev/null)" || true' TERM
    - terraform apply -auto-approve -lock-timeout=5m

Two separate changes here. The job timeout was increased to give a normal apply enough headroom to finish, so timeouts stop being the routine outcome for anything slightly slower than average. And the trap on TERM gives the job a chance to attempt its own unlock in the seconds between the CI scheduler asking it to stop and killing it outright, when GitLab’s timeout mechanism sends SIGTERM before SIGKILL.

What We Put in Place After

1. An alert on lock age, not just lock existence

LOCK_AGE=$(aws dynamodb get-item \
  --table-name terraform-locks \
  --key '{"LockID": {"S": "infra-prod/terraform.tfstate"}}' \
  --query 'Item.Info.S' --output text | jq -r '.Created')

# Page if a lock has existed longer than the longest expected apply

A lock that’s normal for the first ten minutes of an apply is a problem at forty. The alert is on duration, since the lock’s mere presence is expected behavior most of the time.

2. A documented, safe force-unlock procedure in the runbook

Specifically: check running pipelines first, cross-reference the lock’s Who and Created fields against CI job history, and only then run force-unlock. Skipping the verification step to save two minutes is exactly how someone unlocks a state file mid-apply.

3. Job timeouts sized against real apply duration, not a round number

We pulled the p95 apply duration per module from CI job history and set each job’s timeout to roughly double that, instead of one timeout value copy-pasted across every pipeline regardless of what it actually runs.

Key Lessons

A stale lock and an active lock produce the identical error message.
The Lock Info block doesn’t tell you whether the process that created it is still alive.
A hard process kill skips cleanup code, including Terraform’s own unlock.
Anything that relies on a graceful exit needs a plan for what happens when the exit isn’t graceful.
DynamoDB lock tables don’t expire entries on their own.
A lock from a job that died an hour ago looks exactly like one from a job still running.
Check the lock’s age and owner before assuming concurrent access.
Cross-referencing the timestamp against CI job history took minutes and ruled out the wrong theory early.
Uniform CI timeouts across modules of different sizes guarantee this happens again.
Size the timeout to the job, not to whatever default the pipeline template shipped with.

Summary

Component	What Happened	Tool to Check
CI Job	Hit its timeout mid-apply and was hard-killed	CI job history / pipeline logs
Terraform	Never reached its own unlock code path	`terraform force-unlock`
DynamoDB Lock Table	Held a stale lock indefinitely, no TTL	`aws dynamodb get-item`
Every Downstream Pipeline	Blocked by a lock from a job that no longer existed	`terraform plan` / `apply` error output

Nothing was actually contending for the state file. The lock outlived the job that created it, and the error had no way of saying so.