We Leaked a Production Database Password Into Build Logs for Three Weeks Because One CI Variable Wasn't Marked Masked - TZoneLabs We Leaked a Production Database Password Into Build Logs for Three Weeks Because One CI Variable Wasn't Marked Masked

By KP | TZoneLabs | DevOps & Cloud Engineering

For three weeks, our staging database password was sitting in plain text in every CI job log a specific pipeline produced. Anyone with read access to the project, which included a few external contractors, could open a job log and copy it out. Nobody stole it, as far as we could tell. But the only reason we found it was a routine access review, not a security tool.

This post covers how a single unmasked CI/CD variable turned into a three-week exposure window, and what we changed so a future mistake like this gets caught in minutes instead of weeks.

What Happened

A new integration test suite needed direct database access, so someone added a CI/CD variable for the staging database connection string. It was correctly scoped as protected and only available on the right branches. But when it was created, the “Mask variable” checkbox was left unchecked, most likely because the value contained characters GitLab’s masking rules don’t allow (masking requires the value to meet certain length and character constraints, and this connection string had special characters in the password).

The test suite printed its configuration at the start of every run for debugging purposes, including the full connection string, straight into the job log.

Where We Started Looking

This wasn’t found through an alert. During a quarterly access review, someone checking which contractors still had repo access decided to spot-check what those contractors could actually see, not just what permissions they were granted.

Step 1 — Check who can view CI/CD job logs

# GitLab job logs are visible to anyone with at least Reporter access
# on the project, by default
curl --header "PRIVATE-TOKEN: <token>" \
  "https://gitlab.example.com/api/v4/projects/<id>/members/all"

Three contractors with Reporter access, brought on for a short-term integration project two months earlier, still had it. Their access alone wasn’t the finding. What they could see with it was.

Step 2 — Search recent job logs for anything that looks like a secret

for job_id in $(curl -s --header "PRIVATE-TOKEN: <token>" \
  "https://gitlab.example.com/api/v4/projects/<id>/jobs?per_page=100" | jq -r '.[].id'); do
  curl -s --header "PRIVATE-TOKEN: <token>" \
    "https://gitlab.example.com/api/v4/projects/<id>/jobs/$job_id/trace" \
    | grep -iE "password|postgres://|mysql://" && echo "Found in job $job_id"
done

This is what surfaced it. One specific job, run on every commit to a feature branch, was printing the full staging database connection string, password included, in plain text.

Step 3 — Confirm how long it had been happening

curl -s --header "PRIVATE-TOKEN: <token>" \
  "https://gitlab.example.com/api/v4/projects/<id>/pipelines?ref=feature/db-integration-tests&per_page=100" \
  | jq '.[].created_at'

The branch had been active for 21 days. Every single run on it had the same exposure.

Why the Masking Didn’t Catch It

🔴 Root cause: masking was silently unavailable, and nobody was told.

The password contained a @ character as part of a URL-encoded connection string, which GitLab’s masking regex does not support.

When a variable’s value can’t be masked, GitLab lets you save it anyway, unmasked, without a hard blocker.

Whoever created the variable didn’t notice the mask setting had reverted to off, because there was no follow-up check on protected variables after creation.

Nothing alerted on this. An unmasked protected variable looks identical to a masked one in the UI unless you check the box directly.

The Fix

Rotate first, investigate second

The staging database password was rotated immediately, before finishing the investigation. A password that might have been exposed gets treated as exposed.

ALTER USER staging_app WITH PASSWORD '<new-generated-password>';

Stop printing configuration values in test output

# Before
echo "Connecting with: $DATABASE_URL"

# After
echo "Connecting to: $(echo $DATABASE_URL | sed -E 's#://[^:]+:[^@]+@#://[user]:[redacted]@#')"

Use a secrets manager reference instead of a raw CI variable for anything with special characters

integration-test:
  script:
    - export DATABASE_URL=$(vault kv get -field=connection_string secret/staging/db)
    - npm run test:integration

Pulling the value at runtime from a secrets manager avoids the masking limitation entirely. The value never lives in a CI/CD variable field in the first place.

What We Put in Place After

1. A CI job that scans its own recent job logs for secret patterns

secret-log-scan:
  stage: .post
  script:
    - trufflehog filesystem --json $CI_PROJECT_DIR/job-logs/ | tee findings.json
    - test ! -s findings.json || (echo "Potential secret found in logs" && exit 1)
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'

2. A required check when adding any protected CI/CD variable

Any new protected variable now requires a reviewer to confirm the mask checkbox is checked, or that the value comes from the secrets manager instead. This is a two-minute manual step, but it’s the step that was missing.

3. Quarterly access reviews now include a log spot-check

Checking who has access is only half the review. Checking what that access actually exposes, by sampling real job logs, is what caught this the first time and now happens on a schedule instead of by chance.

4. Contractor access set to expire automatically

gitlab-cli project-members add --project-id <id> \
  --user-id <contractor-id> \
  --access-level 20 \
  --expires-at "2026-08-15"

Key Lessons

An unmasked variable looks the same as a masked one unless you check.
There’s no visual warning in the UI once a value fails to mask silently.
Special characters in secrets can defeat masking rules without failing the save.
A raw connection string with URL-encoded characters is a common trigger for this.
Debug logging is a common secret-leak vector, not just misconfigured variables.
Anything that prints its own configuration for troubleshooting needs to redact before it prints.
Access reviews should check exposure, not just permissions.
Knowing who has Reporter access tells you nothing about what they can actually see.
Treat “might have been exposed” the same as “was exposed.”
Rotating first and investigating second costs a few minutes. Waiting until certainty costs a lot more if it turns out to matter.

Summary

Layer	What Happened	Tool to Check
CI/CD Variable	Password saved unmasked due to unsupported characters	GitLab CI/CD settings, variable audit
Job Logs	Full connection string printed on every run for 21 days	GitLab Jobs API, log grep
Access Control	Contractors retained Reporter access after their project ended	Project members API
Detection	Found via manual access review, not automated scanning	Quarterly review checklist

The password itself wasn’t the failure. Trusting a checkbox without verifying it took effect was.

We Leaked a Production Database Password Into Build Logs for Three Weeks Because One CI Variable Wasn’t Marked Masked