Most “works on my machine” outages are not mysterious. Someone changed a key in staging, forgot it in production, rotated a secret in the vault but not in the deploy pipeline, or left a debug flag only on their laptop. Configuration drift is the quiet killer: the app starts, then fails halfway through a payment call or silently uses the wrong database.
This guide is a pre-deploy checklist you can run by hand or wire into CI. It is deliberately stack-agnostic: Docker, bare metal, Kubernetes, and serverless all have the same underlying problem — more than one place claims to own “the environment.”
Three failure modes show up over and over:
STRIPE_SECRET_KEY. The process boots; first charge throws..env around in chat or email, you already have drift —
you just have no audit trail for when it happened.
Before you can diff anything, write down where config lives. A realistic service often has five or more:
.env / .env.local (never committed).env.example or schema file (committed, keys only)
Pick a baseline for each stage (dev, staging, prod). For many small teams the baseline is a key-only
.env.example plus a short doc that says “prod secrets live in X.” Mark each key required or optional.
Optional keys without a default are how silent feature-off switches get deployed by accident.
Assign one human (or on-call rotation) as owner of “prod config.” When two people can edit vault paths without a PR trail, drift is a process bug, not a tooling bug.
Compare key sets, not full values in Slack. A useful report answers three questions:
Prefer tools that redact values in shared output. Secret managers’ audit UIs, editor plugins, and dedicated env-diff CLIs all work; pick something the team will actually run. Pasting a full prod dump into a ticket is a security incident waiting for a forward button.
# Conceptual: key-set only compare (illustrative) # left = .env.example keys, right = exported key names from staging comm -3 <(sort example.keys) <(sort staging.keys)
Treat these differently in process and in automation:
A common trap: the app accepts empty string as “present.” Your health check is green; the third-party API rejects empty auth. Validate that required secrets are non-empty at boot, not only that the key exists.
On any PR that changes env examples, Helm values, Compose files, or Terraform that injects env, run a key-set check against the baseline. Fail closed for required keys. Optional niceties:
/When drift does bite, write a one-page postmortem with the actual key name (not the value) and the stage that was wrong. Then add one automatic check that would have caught it. Teams that only “remind everyone to be careful” get the same incident quarterly.
Related: Dotenv tooling landscape · Browser JSON tools · Tactical Data Concepts