Catching Terraform Drift in CI Before a Manual Change Reaches Production

By KP | TZoneLabs | DevOps & Cloud Engineering

Someone changed a security group in the console to unblock themselves during an incident, and it never made it back into the .tf file. Terraform doesn’t know that happened until the next apply either quietly reverts it or, worse, doesn’t touch it because nothing in the config changed. terraform plan -detailed-exitcode run on a schedule catches that gap before it turns into either outcome — exit code 2 means the real infrastructure no longer matches what’s in state, and CI can fail the build on that alone.

This covers how detailed exit codes work, wiring a drift-check job into GitLab CI on a schedule, and what to do when drift shows up.

Why Drift Happens Even When Nobody Touches the Repo

Console changes during an incident are the obvious source, but they’re not the only one. Auto-scaling can modify a launch template’s live values. A teammate can run apply from a stale branch. AWS itself can rotate defaults on certain resources during maintenance windows. None of this shows up as a diff in the repo, because the repo was never the thing that changed — the real infrastructure was.

The normal workflow doesn’t catch this. terraform plan only runs when someone remembers to run it, usually right before an intentional change, and by then the drift has been sitting in production for however long since it happened.

Reading Detailed Exit Codes Instead of Just the Plan Output

Plain terraform plan always exits 0 unless there’s an error, whether or not anything changed. -detailed-exitcode makes the exit code carry information:

terraform plan -detailed-exitcode -input=false

Exit code	Meaning
`0`	No changes — real infrastructure matches state
`1`	An error occurred
`2`	There are changes — this is drift if nobody intended to run `apply`

That single distinction is what makes automated drift detection possible: a CI job can check for exit code 2 specifically and fail the pipeline on it, without treating every plan as an error.

A Scheduled Drift-Check Job in GitLab CI

terraform-drift-check:
  stage: verify
  image: hashicorp/terraform:1.7
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
  script:
    - terraform init -input=false
    - |
      terraform plan -detailed-exitcode -input=false -out=drift.tfplan
      exit_code=$?
      if [ "$exit_code" -eq 2 ]; then
        echo "Drift detected — real infrastructure no longer matches state"
        terraform show drift.tfplan
        exit 1
      elif [ "$exit_code" -eq 1 ]; then
        echo "Plan failed"
        exit 1
      fi
      echo "No drift"

Set this on a GitLab CI schedule (hourly or daily, depending on how fast drift is expensive in your environment) rather than on every push — a drift check on every commit just re-answers a question that hasn’t had time to change.

Alerting on Drift Instead of Letting the Pipeline Fail Quietly

A red pipeline that nobody’s watching is the same as no detection at all. Pipe the failure into wherever incidents already get noticed:

if [ "$exit_code" -eq 2 ]; then
  curl -X POST -H 'Content-Type: application/json' \
    -d '{"text":"Terraform drift detected in '"$CI_PROJECT_NAME"' — check the drift-check pipeline"}' \
    "$SLACK_WEBHOOK_URL"
  exit 1
fi

The message only needs to say where to look. The terraform show drift.tfplan output in the job log has the actual detail on what changed.

What This Doesn’t Catch

Resources not in state at all. If something was created outside Terraform entirely, drift detection has nothing to compare it against. That’s an import problem, not a drift problem.
Drift inside a data source. Data sources are read fresh on every plan, so a change there won’t show as drift — it just silently changes what downstream resources compute from.
Concurrent applies racing the drift check. A drift-check job and a real apply running at the same time can both read a consistent-looking state and still step on each other. State locking prevents the actual collision, but expect an occasional false-positive drift result if the schedule overlaps a deploy window.

Key Lessons

-detailed-exitcode is what turns plan into something a script can act on.
Plain plan output is for humans; the exit code is for CI.
Drift checks belong on a schedule, not on every commit.
A push-triggered check just re-runs a comparison that hasn’t had time to change.
A failed drift-check pipeline needs to alert somewhere people look.
A red pipeline nobody’s watching catches nothing.
Drift detection can’t see resources that were never imported into state.
It only compares what Terraform already knows about.
A drift check racing a real apply can produce a false positive.
State locking prevents the actual conflict; expect occasional noise if schedules overlap deploys.

Summary

Decision	Do	Avoid
Detecting drift	`terraform plan -detailed-exitcode` on a schedule	Relying on someone noticing during a manual `plan`
Trigger	Hourly/daily CI schedule	Running the check on every push
On exit code 2	Alert to Slack/incident channel + fail the job	Letting the pipeline go red silently
Untracked resources	Import into state first	Assuming drift detection covers everything live

None of this needs a new pipeline to adopt — add the drift-check job to the existing Terraform pipeline with a schedule rule, point the failure at a channel someone actually watches, and expand from there.