The Bug I Blamed on the Provider

For weeks, one stack had a terraform plan I had learned to ignore. Every run, on every environment, it wanted to strip the server-side-encryption rule off an S3 bucket and replace it with an empty one. The live bucket was encrypted correctly. Nothing was broken in AWS. The plan just would not converge, and no matter what values I declared, the next plan asked for the same change again.

I filed it mentally under “AWS provider quirk, not worth fixing” and moved on. That was the mistake. The provider was doing exactly what it was told. The lie was upstream of Terraform entirely, in my own code, and it had been sitting in plain sight in a file I had never bothered to open.

The reflex, and why it’s wrong

The stack is CDKTF in Python. You write resources as typed classes or plain dicts, cdktf synth renders them to a cdk.tf.json file, and Terraform plans against that. When a plan misbehaves, the reflex is to blame the layer you can’t see into: the provider’s diff logic, a known-flaky resource, a state quirk. S3 encryption resources have a reputation for drift noise, so this fit the story I wanted to believe.

The reflex is wrong because CDKTF has an artifact you can read. cdk.tf.json is the exact JSON Terraform plans against. It is not a black box; it is the compiled output, sitting on disk after every synth. I had spent weeks reasoning about the provider’s behavior and zero seconds looking at what I was actually handing it.

When I finally opened the file, the rule was there. It was just empty:

"rule": [
  {}
]

Terraform was not drifting. It was faithfully trying to reconcile a bucket that had a real encryption rule against a declared config that said the rule should have no contents. An empty block is a positive instruction, not an absence. The provider was innocent.

The root cause

Here is the code that produced that empty block. It looks completely ordinary:

S3BucketServerSideEncryptionConfiguration(
    self, "encryption",
    bucket=bucket.id,
    rule=[{
        "apply_server_side_encryption_by_default": {
            "sse_algorithm": "aws:kms",
            "kms_master_key_id": key.arn,
        },
        "bucket_key_enabled": True,
    }],
)

The rule argument takes a list. Each item is a nested dict: a block containing another block. That nesting is the whole problem.

CDKTF’s Python bindings sit on top of jsii, which auto-converts Python dicts into the typed structs the underlying TypeScript expects. That conversion does not recurse into list items that are themselves nested dicts. When jsii walks a Sequence[Union[SomeTypedClass, Dict]] and finds a dict whose values are more dicts, it fails to map the inner structure and quietly emits an empty object. No exception. No warning. Synth exits zero.

What makes this genuinely nasty is that the same pattern works everywhere else. A list of dicts with flat scalar values converts fine:

# This renders correctly. The list items are flat.
taint=[{
    "key": "dedicated",
    "value": "gpu",
    "effect": "NO_SCHEDULE",
}]

So the failing case and the working case are visually identical: a list of dicts passed to a resource argument. The only difference is whether the dict’s values are scalars or more dicts, and nothing in the code, the synth output, or your assertions distinguishes them. You cannot spot it by reading the Python. You can only spot it by reading the JSON.

The trap inside the trap

The empty block is worse than a no-op. It is a positive declaration of “this rule has no configuration,” which means the plan can never converge. Terraform sees a live rule, sees a declared-empty rule, and proposes the delta. You edit the Python to add more values, and jsii drops them the same way, so the JSON is still [{}] and the next plan is identical. The config edit that should fix it cannot fix it, because your values never survive synth to reach Terraform in the first place. That is why it read as unfixable drift. In a sense it was, as long as I kept trying to fix it in the wrong layer.

The fix

Stop handing jsii a nested dict to guess at. Construct the typed class explicitly, and the conversion has nothing to infer:

S3BucketServerSideEncryptionConfiguration(
    self, "encryption",
    bucket=bucket.id,
    rule=[S3BucketServerSideEncryptionConfigurationRuleA(
        apply_server_side_encryption_by_default=(
            S3BucketServerSideEncryptionConfigurationRuleApplyServerSideEncryptionByDefaultA(
                sse_algorithm="aws:kms",
                kms_master_key_id=key.arn,
            )
        ),
        bucket_key_enabled=True,
    )],
)

The class names are verbose and the trailing A is an artifact of how the bindings are generated, but the point is that jsii now receives a fully-typed object instead of a dict it has to introspect. The nested structure survives, the JSON renders every key, and the plan converges on the first try. The rule of thumb: any resource whose list items themselves contain a nested block wants the typed class, not a raw dict.

The lesson worth keeping

I had actually seen this bug before and not connected it. On the Kubernetes provider, building resources from dicts had silently dropped volumeMounts, left a StatefulSet with a blank password env, and stripped readiness probes, all with no synth error. I had treated that as a Kubernetes-provider oddity. It is not. It is a property of the CDKTF Python bindings, and it will surface on any provider the moment your resource has a nested block inside a list.

The durable lesson is not about S3 or Kubernetes or any specific resource. It is this: a successful synth means your code ran, not that it rendered what you meant. The two are not the same, and the gap between them is exactly where this bug lives. Grep-based assertions pass, because the fields that get dropped are the deeply-nested ones nobody writes a grep for. Synth passing tells you nothing, because dropping the fields is what synth does.

So when a plan will not converge and the resource looks correct in the console, resist the urge to blame the provider. Open the compiled artifact. Read the cdk.tf.json Terraform is actually planning against, not the Python you wish it were. The tool is almost always doing exactly what you told it. The interesting question is what you actually told it, and that answer is written down, one synth away, in a file most people never open.