Dev Platform: Teaching AWS to Trust My Cluster | Arcade Lab

Go back<br>🔑 Dev Platform: Teaching AWS to Trust My Cluster

Milestone 2, part one: IRSA without the magic — OIDC from first principles, and hosting my own OpenID provider on an S3 bucket

🎯 What I was trying to do

Part 2 ended with a quiet, correct foundation: encrypted secrets, CI gates, and Cilium doing the networking. Nothing on the cluster yet that needs to reach out of it.

That changes now. The platform services I'm about to install — the EBS CSI driver, the AWS Load Balancer Controller, Velero — all need to call the AWS API . The EBS driver creates volumes, the LB controller spins up NLBs, Velero writes backups to S3. And AWS only authenticates IAM identities ; it has no idea what a "Kubernetes pod" is.

The lazy answer is to create an IAM user, generate an access key, and drop it into the cluster as a Secret. Long-lived static credentials that never rotate and leak the moment anyone gets read access to a namespace. The right answer is IRSA — IAM Roles for Service Accounts — where a pod proves who it is with a short-lived token and gets temporary credentials, no static key anywhere.

I knew the words. I did not actually understand the mechanism. So before writing a line of Terraform, I made myself learn it properly — and that turned out to be the real content of this milestone.

🧩 The thing I had to understand first: OIDC

Every IRSA explanation I found assumed you already understood OIDC. What made it click for me was ignoring the provider details and thinking of it as three roles passing a signed token: the client (app), the OpenID Provider (issues and signs tokens), and the resource server (the API).

Setup — discovery, before any login

On startup, the client reads the provider's config from a well-known URL (cached afterwards, so this isn't part of the per-login flow):

GET https:///.well-known/openid-configuration<br>→ { "issuer": "...", "authorization_endpoint": "...",<br>"token_endpoint": "...", "jwks_uri": "..." }

Four fields carry the whole flow:

issuer — the provider's identity; must exactly match the iss claim inside the tokens it hands out.

authorization_endpoint — where the client sends the user to log in.

token_endpoint — where the client swaps the authorization code for tokens.

jwks_uri — where the provider publishes its public signing keys; this is the one that matters later, and the client fetches and caches it too.

(For IRSA, only issuer and jwks_uri end up mattering — there's no interactive login, so the two endpoints go unused.)

The auth flow — Authorization Code + PKCE

Now the per-login flow. The modern, correct version carries three pieces of anti-abuse machinery, and skipping any one of them is a real vulnerability.

1. The client starts the login. Before redirecting, it generates three short-lived values and stores them (a server-side session behind an HttpOnly cookie for a confidential client; sessionStorage for a SPA with no backend):

state — random, returned unchanged at the end. Ties the callback to this browser session, defeating CSRF / forged callbacks.

nonce — random, must reappear inside the returned ID token, defeating replay (reusing a genuine token issued during another login).

a PKCE pair — a secret code_verifier and its SHA-256 hash code_challenge. Only the hash is sent now.

It redirects the browser to the authorization endpoint :

GET<br>?response_type=code&client_id=...&redirect_uri=...<br>&scope=openid&state=&nonce=<br>&code_challenge=&code_challenge_method=S256

2. The user authenticates on the provider's page (and may grant consent). The provider redirects back to redirect_uri with a one-time authorization code and the original state. The client first checks state matches what it stored — if not, drop it.

3. The client exchanges the code for tokens on the back channel (server-to-server), revealing the PKCE secret now:

POST<br>grant_type=authorization_code&code=<br>&code_verifier=&redirect_uri=...<br>&client_id=... # + client_secret for a confidential client<br>→ { "id_token": "...", "access_token": "...", "refresh_token": "..." }

PKCE is what makes a stolen code worthless: whoever intercepts it can't redeem it without the code_verifier, which never travelled over the wire. state protects the callback ; PKCE protects the code .

4. Two tokens come back, and they are not the same thing — the distinction that took me longest:

The ID token proves who the user is , and it's for the client . The client validates it (signature against the cached JWKS, plus iss, aud, exp, and the nonce it sent), reads the sub claim, and establishes its own session.

The access token authorizes API calls , and it's intended for the resource server . Clients typically treat it as a credential and attach it when calling the API rather than relying on its contents.

5. The client calls the API with the access token, and the resource server validates it independently — signature against the same JWKS, then iss, aud, exp, nbf, algorithm,...