The EKS 401 That Wasn't About Credentials

The first prod rollout of our new deploy pipeline went green through every step except the last. Identity check, chart pull, update-kubeconfig; all of it succeeded. Then helm upgrade failed with a single line:

Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials

The first thing everyone reaches for is “rotate the AWS access key.” That instinct is wrong. The error is not from AWS. The token was issued, signed, and presented to the kube-apiserver; the apiserver rejected it. The cluster had no RBAC for the principal that was asking.

This is what the failure actually looks like, end-to-end, on a healthy pipeline where nothing was rotated, nothing expired, and nothing was misconfigured at the credential layer.

Why the instinct is wrong

The word “credentials” in that error is misleading. The message comes from client-go, the Kubernetes client library, not from AWS STS. AWS credential failures surface much earlier with a different message, typically something like exec: ... could not get token, and they fail at the aws eks get-token step, not at helm upgrade.

In our trace, everything upstream of the helm step worked:

• The runner’s IRSA role assumed the cross-account deploy role via AssumeRoleWithWebIdentity.
• The deploy role pulled the chart from the operations ECR.
• update-kubeconfig wrote a valid kubeconfig pointed at the prod cluster.
• aws eks get-token returned a token.

The token was mintable. The token was presentable. The token was then rejected by the cluster. That is a 401, and a 401 is an authorization problem, not an authentication problem in the AWS sense.

The three-layer mental model

“Credentials” in an EKS deploy pipeline can fail at three different layers, and only one of them produces the wording above. A quick way to split them:

Layer	What it does	Failure message	First check
AWS STS	Issues the session for the runner or deploy role	could not get token, ExpiredToken, AccessDenied	aws sts get-caller-identity --profile <profile>
EKS access control	Maps the IAM principal to a Kubernetes identity	server has asked for the client to provide credentials (with valid token)	aws eks list-access-entries --cluster-name <c>
Kubernetes RBAC	Allows the mapped identity to do something	Forbidden, cannot ...	kubectl auth can-i --list

A 401 from the apiserver sits in the middle. A 403 sits at the bottom. A connection refused or a timeout is a network or kubeconfig problem, not in this table at all. Once you know which layer you’re on, the rest of the diagnostic is mechanical.

Two confirmations that take thirty seconds

Before chasing any fix, confirm the AWS side is healthy so you don’t end up rotating a key that was never the problem:

# Side A: the AWS identity your deploy role assumes
aws sts get-caller-identity --profile <spoke-account>

# Side B: the cluster itself can issue a token for that identity
aws eks get-token --cluster-name <cluster> --region <region> --profile <spoke-account>

If both of those succeed, you are not looking at a credential problem. You are looking at an access entry problem. The token in Side B is exactly the token the apiserver will see; if Side B returns a token, the kube-apiserver will see one too. The token is not the issue.

Then check which principals the cluster actually knows about:

aws eks list-access-entries \
  --cluster-name <cluster> \
  --region <region> \
  --profile <admin-profile>

If the deploy role’s principalArn is not in that list, the cluster has no record of it. Every request that role makes will be rejected at the access-control layer, regardless of how valid the token is.

What EKS access entries actually are

EKS access entries are the API-level replacement for the aws-auth ConfigMap. They were introduced as AWS deprecated the aws-auth pattern of mapping IAM roles to Kubernetes users by hand-editing a YAML file in kube-system. The migration has been underway for years; many clusters still run with authenticationMode: CONFIG_MAP and rely on aws-auth, some run with API_AND_CONFIG_MAP during the transition, and newer ones are API only.

Under any mode that includes the API, the chain is:

1. The IAM principal (your role) presents a presigned EKS token.
2. The apiserver validates the token against STS.
3. The apiserver looks up the principal in the access entry table.
4. If the principal is found, the associated access policies are mapped to Kubernetes groups or users.
5. Those groups and users are then subject to normal Kubernetes RBAC.

A 401 at step 3 means the lookup returned nothing. The principal is valid AWS-side and the token is valid STS-side; the cluster simply has no row for it. The token is presented, the lookup fails, the apiserver asks for credentials (because it has none to map to), and client-go reports that to the user as “server has asked for the client to provide credentials.”

The exact same wording shows up when a human role tries to use a freshly recreated cluster. The cluster creator identity gets implicit admin and works; everyone else needs an entry.

Where the gap came from in our case

Our EKS module takes an access_entries map. Each entry is { principal_arn = "..." }, and the module creates the aws_eks_access_entry plus a defaulting AmazonEKSClusterAdminPolicy association at cluster scope. Adding a key to that map is the only thing one needs to do in code to grant a deploy role full access to a cluster.

The CI deploy role’s entry was added to the map during the prod EKS rollout. The sibling dashboard-admin entry in the same block was already on the cluster, proving the block had been applied at least once. The deploy entry was added after that last apply. Edit-without-apply is one of the easier mistakes to make with Terragrunt, because the plan output for a single new entry is small and clean and the cost of skipping the apply “just this once” is invisible until the first deploy needs it.

Sandbox and pre both worked because their entries had been applied long before this gap. Prod was the first deploy through the new pipeline, so the gap only surfaced there. The earlier envs were unaffected.

The fix

terragrunt apply --all --target <env>/eks

The apply is only strictly additive if no other declared principals already exist click-ops. In our case the sibling dashboard-admin and viewer-role entries were already live on the cluster, so the apply would have failed with ResourceInUseException if we had run it cold. The first thing to do is verify the live cluster state:

aws eks list-access-entries \
  --cluster-name <cluster> \
  --region <region> \
  --profile <admin-profile>

If the entries the module wants to manage are already on the cluster, import them before applying:

terragrunt import 'aws_eks_access_entry.this["dashboard-admin"]' \
  '<cluster>:<dashboard-admin-role-arn>'

Once the import is done, an apply against the same module is a clean create for the new entry and a no-op for the existing ones. End-to-end proof is the next deploy run. If the apply succeeded and the deploy still fails, the entry is not actually mapped to a usable policy association, and list-associated-access-policies is the next stop.

A diagram of the path

The whole pipeline, with the failure point highlighted:

  ┌───────────────┐     ┌────────────────────┐     ┌───────────────┐
  │  runner IRSA  │ ──▶ │  cross-account     │ ──▶ │  aws eks      │
  │  role         │     │  deploy role       │     │  get-token    │
  └───────────────┘     └────────────────────┘     └───────────────┘
                                                          │
                                                          ▼
  ┌───────────────┐     ┌────────────────────┐     ┌───────────────┐
  │  Kubernetes   │ ◀── │  access entry      │ ◀── │  kube-        │
  │  RBAC         │     │  lookup  ❌  no row │     │  apiserver    │
  └───────────────┘     └────────────────────┘     └───────────────┘

The runner IRSA role assumes the cross-account role, the cross-account role issues a token, the apiserver receives the token, the apiserver looks up the principal in the access entry table, and the lookup returns nothing. Everything before the lookup is fine. Everything after it is unreachable.

The broader pattern

“Credentials” is a word that covers three different layers in an EKS deploy pipeline: AWS credentials (do you have a valid principal?), EKS access control (does the cluster know about that principal?), and Kubernetes RBAC (is that principal allowed to do the thing it’s asking for?). Only the second one produces the message above. Rotating AWS keys does not help with the second one. Updating kubeconfig does not help with the second one. The only thing that helps with the second one is an aws_eks_access_entry for the principal that is asking.

When the message says “credentials,” check which credentials it is actually talking about before rotating any of them. The error message is the worst part of the diagnostic; it survives across all three layers and points at the wrong one in two of the three cases. The chain aws sts get-caller-identity → aws eks get-token → aws eks list-access-entries resolves the ambiguity in under a minute and tells you which layer to fix.

The same pattern shows up in adjacent failure modes. A human admin role that works on the original cluster and stops working after a destroy/recreate is the same gap, with a different trigger. A CI role that worked on the previous env and stops working on the new one is the same gap, with a different apply history. The token is fine. The cluster is fine. The entry is missing.