The Plan Was Green Because Two of Three Views Agreed. The Third One Was the Truth.

A terraform plan came back zero for a managed observability stack. The on-call had hand-fixed it during an incident two weeks earlier. The fix was live. The fix was in the file. State alone was behind. An apply would have reverted every incident-mitigating line in a single shot. The plan’s “nothing to do” was, in this case, indistinguishable from “no drift.” This is not a story about a green plan that caught a real problem; it is a story about a green plan that hid one.

A plan is a diff. The system has three views

A typical plan is a diff between two things: state and file. “State” is the last successful apply; “file” is what you would write if you were starting now. The plan renders unchanged whenever state and file agree, which is the green every operator learns to want.

The system has three sources of truth, and only two of them are in the diff. Live is the running thing, the cluster or the workload or the storage bucket. State is what Terraform or Helm recorded last. File is what the repo currently says. Each is authored separately. Each can drift. A plan ignores the first one.

The three sources can agree for very different reasons. They can agree because all three are derived from the same file (the canonical case, which is what a green plan correctly reports). They can agree because state is behind and file and live have already caught up (the case that hides the most dangerous drift, because state-behind looks identical to no-drift from a terraform plan perspective). They can agree because file is behind and live and state agree (the “I never wrote the change down” case, which the plan reads as drift in the wrong direction). They can agree because live is behind and state and file agree (the “the platform reverted my change because it crashed” case, which the plan reads as drift toward file again).

Two views agreeing with each other while disagreeing with the third is the case a green plan silently papers over. A plan is a diff over a comparison between two things, and the third thing does not get a vote.

The first exhibit: the managed observability stack

It was the middle of an incident. The on-call had run a hand helm upgrade against the live release; values had landed in-cluster but the file in the repo had not been updated. The incident closed, the team caught up later that week and updated the file to match, and on the surface everything reconciled. Then the planning phase of the next change came around, and terraform plan came back clean.

The case was the worst-case shape for the silent agreement: live matched the file (both reflected the fix), state had been written before the hand upgrade and never updated, and the plan rendered state-vs-file as no diff because state and the file agreed about everything except the post-incident fix; on the new file the fix was the only diff, and state still held the pre-incident shape. None of that was the plan’s job. The plan’s job is to compute the diff between state and file, and the only diff there was the new fix that the file had incorporated and the state never had. So the plan rightly reported: this file wants to apply this fix.

That is the moment the green becomes dangerous. “Nothing to do” would have been the right read if state had been the truth and file had matched it. “Nothing to do” would have been the right read if live, state, and file all held the pre-incident shape. “Nothing to do” was the wrong read the moment live held the post-incident shape and only state disagreed, because an apply would have reverted every line of the incident fix in one shot.

The window during which live and file agreed against a state-behind backend is the bug. It is easy to miss because live and file genuinely do agree, and a green plan is the dream. The postmortem would have read “apply reverted the production fix during the next change,” and would have been right, and would have been the most expensive possible framing of a misunderstanding the plan itself could not have caught.

The second exhibit: the managed Kubernetes cluster, a day later

Same shape, different surface. The repo’s terraform plan for a managed Kubernetes cluster came back with a single concrete change: a node group resizing general_arm from 6 to 4, a real capacity drain deliberately requested. That was the only diff in the output. That was also not all of the truth.

Three of the other node groups on the same cluster were state-behind. A code sync had landed a week earlier; live had been updated by hand-out-of-band edits during an upstream change, and live already matched the file; only state had not seen the new config. The plan rendered state-vs-file as no diff because state and the file agreed, and the next apply would have reverted the live shape back to whatever state thought the cluster looked like a week ago.

The single loud diff dominated operator attention, which is exactly the point. Operators tend to read the first diff they see, mentally check that diff for safety, and file the rest under “no changes.” The silent ones never show up in the diff because they are not in the diff. The cluster would have applied a real capacity change and quietly reverted three other groups to a shape nobody was looking at. The postmortem would have read “the apply dragged along unintended changes,” and would have been right, and would have misattributed the cause to the loud diff instead of to the three quiet ones.

Both exhibits walk the same mechanism. What changes is which two views agreed. The first exhibit was live-and-file-agree, state-behind. The second exhibit was live-and-state-agree, file-behind. The mechanism is the same: two-agree-one-differ silent agreement, green plan, apply reverts the thing the live system had that the state did not know about.

Why a plan can’t see it

terraform plan consults state and file. It does not consult live at apply time. helm template does not consult live at plan time. Neither tool was designed to verify the live system; both were designed to verify what would change given the last known state. The three-way merge the platform does not do is the one you have to do yourself, before the plan, not as part of it.

The shape of the limitation is consistent across tools. Whatever is recorded somewhere outside the platform’s model is the thing the plan cannot see. For Terraform, that is the live API. For Helm, that is the live release that helm template does not query. For kubectl diff against a server-side apply, that is whichever admission controller last modified what, which the local dry-run never sees. The plan tool and the platform are paying attention to two different sources; the third source is whichever one the user forgets to mention.

A plan that comes back clean is not a statement that live matches file. It is a statement that state matches file, which is information the operator needs to convert into a different question, the question of what live is doing that neither knows about.

The way to actually read it

Three cheap reads against the three sources, in order: state from the backend, file from disk, live from the API. Lay them out side by side. The case where any two agree and one differs is the case a green plan silently papers over. The case where all three agree is the case a green plan correctly reports. Distinguishing them costs a state pull and a read, no apply, no review latency.

For Terraform:

# Read 1: state
terraform state pull
# Read 2: file
terraform plan -no-color
# Read 3: live
# Whatever the platform surfaces (e.g., aws eks describe-cluster ...)

The point is not the commands. The point is the three reads together answer “which view is behind,” which is the question a single green plan cannot.

For Helm:

# Read 1: state
helm get values <release> -n <ns>
# Read 2: file
helm template <release> <chart> -f values.yaml
# Read 3: live (after dry-run)
helm diff upgrade <release> <chart> -f values.yaml

Each pair of reads can be cross-checked by eye. Any pair that disagrees against a third read that agrees with neither is the case worth pausing on. The reads take longer to set up the first time than to run; after the first time they are a script, and the script is a check that runs before every plan, and the check is what turns the green diff from a guess into a decision.

The most valuable moment to run this is the moment you are about to call a plan “nothing to do” after recent hand-fixes to the same system. That is the only moment a green diff is more likely to be wrong than right, and it is the only moment where reading the third view is cheap. Cost is fixed; timing is everything.

The rule and the twin

Read all three sources before you plan; the plan is a diff over the gap, not a statement of truth.

The first exhibit’s contribution is that the greenest plan read as “nothing to do,” and the cost of being wrong was reverting the fix that made the cluster survivable. The second exhibit’s contribution is that the loudest diff in the plan drowned the silent ones, and operators tend to read the first diff they see. They are the same shape because the mechanism is the same; what differs is which view was behind.

The next time a plan comes back unremarkable after a hand-fix or an out-of-band edit on the same system, the question is not “is this right.” The question is which of the three views is the one that’s behind. The answer is whatever the third source says, and reading it takes less time than writing this post did.