Rotate a client secret without overlap, then signing keys with overlap

Setup

1. Open two terminals, A and B. Each is one of the printer's servers. In both, set ISSUER, CLIENT_ID and the current CLIENT_SECRET for lab-print-orders, and API_ID and API_SECRET for lab-photo-api. Define one helper in both.

get_token() { curl -s -u "$CLIENT_ID:$CLIENT_SECRET" -d grant_type=client_credentials -d scope=prints.create "$ISSUER/oauth/token"; }

2. Press Start on this page.

Walkthrough

1. Both servers get a token. In terminal A, keep it: A_TOKEN=$(get_token | jq -r .access_token). In terminal B, get_token | jq .scope.

2. Replace the secret. Open Access Token Management > Client authentication secret, choose lab-print-orders and Rotate client secret. Load the new value into terminal B only: read -rs CLIENT_SECRET.

3. Terminal A, not yet updated, asks for a token: get_token | jq answers invalid_client. Terminal B: a token.

Why it matters: this tenant allows one secret per client, so the old one stopped at once. Rotation here is a coordinated switch with a short failure window, the lesson's outage on a small scale.

4. Find the servers still using the old secret. In Audit, filter for oauth.token, outcome rejected, client lab-print-orders, reason invalid_client. Each row has a request ID and a time. Audit cannot tell you which secret a request presented, or when each secret was last used (G10).

5. Tokens issued before the rotation still work: in terminal A, btl-lab introspect "$A_TOKEN" answers active: true.

6. Respond to a leak, in the lesson's order.

   1. Revoke first: rotate the secret again and do not deploy the new value yet. The leaked secret is now useless, and so is every server's copy. Both terminals now get invalid_client.

   2. End what was already issued: open OAuth > Clients > lab-print-orders, untick Enabled and save, which revokes all its tokens. btl-lab introspect "$A_TOKEN" now answers {"active": false}.

   3. Tick Enabled again and save, then load the newest secret into both terminals with read -rs CLIENT_SECRET. Both get tokens.

Why it matters: when someone else may hold the credential, a short, honest outage is better than leaving it valid while the deployment proceeds. Tokens issued before the revocation do not disappear on their own, so they are ended separately.

Restore: confirm lab-print-orders is Enabled and both terminals hold the newest secret.

7. Overlap, on the tenant's own access token signing keys. Client key pairs need G8 and G56; the tenant's keys show the same pattern for real.

   1. In Key Management, choose Generate key with ES256. btl-lab discover "$ISSUER" now lists two keys. Get a token in terminal A as OLD_KEY_TOKEN; btl-lab decode shows the old kid.

   2. Try to retire the old key now. It is refused: a key that a token manager still signs with cannot be retired.

   3. In Access Token Management, edit Default access tokens to sign with the new key, and do the same for any other manager still using the old one. New tokens carry the new kid, while btl-lab verify "$OLD_KEY_TOKEN" --issuer "$ISSUER" --audience "$ISSUER/resource" --type at+jwt still passes, because the old key is still published.

   4. In Key Management, retire the old key. It stays in the JWKS while tokens it signed may still be valid.

   5. Disable the old key. It leaves the JWKS, btl-lab verify on OLD_KEY_TOKEN now fails to find its kid, and btl-lab introspect "$OLD_KEY_TOKEN" answers {"active": false}.

Why it matters: publishing the new key before signing with it, and removing the old one only after its tokens are no longer needed, means there is never a moment when the only valid credential is one a relying party cannot check.

Planned walkthrough

These steps need two concurrent client secrets (G10), and client key sets with private_key_jwt (G8, G56).

1. Overlapping secret rotation with zero failures. Add a second secret to lab-print-orders; both terminals keep working with the old one. Deploy the new secret to terminal B, then A. Confirm in the client's view and in Audit that the old secret's last use stopped. Remove the old secret. No request fails.

2. Client key rotation through a jwks_uri: publish a second public key in your own key set, switch signing to it, let the last old assertion expire, and remove the old key, without changing the registration.

Break it

Steps 3, 6.1 and 7.2 are the deliberate failures.

Check your work

Press Check my progress. In tenant Audit you can also find a second tenant.oauth.credentials.rotate, the tenant.oauth.clients.update that re-enabled the client, tenant.oauth.managers.update for the signing key change, and tenant.oauth.keys.retire.

Cleanup

1. Make sure both terminals hold the newest secret.

2. Leave the new signing key active.

Missing infrastructure

• G10, client secret rotation overlap. Two concurrent secrets per client with Add, Deploy, Confirm and Remove, and a last-used time per secret shown in the client and recorded in Audit. With it, the planned rotation runs with zero failed requests.

• G8 and G56, client key pairs. private_key_jwt with a registered jwks_uri, so the client rotates its own keys by publishing and retiring them without touching the registration.