Rotate your tenant's signing key without breaking anyone

Setup

In the lesson, the clinic's sign-in service holds a key that can make any patient's token. Your tenant holds exactly such keys. This lab changes the real signing key for access tokens in Lab Photos; ID tokens keep their own key throughout.

1. On this lab page choose Lab Photos and press Start.

2. Open Key Management and note the keys: an ES256 key (call it K1) and an RS256 key. Open Access Token Management: the default access token manager signs with K1 and shows the access token lifetime. Open ID Token Management: the default ID token manager signs with the RS256 key.

3. Set the variables in a working folder:

mkdir -p ~/btl-keys-lab && cd ~/btl-keys-lab
ISSUER=https://tenant-<id>.beyondthelogin.dev

Walkthrough

1. Limit each key. Write down which manager uses which kid. Access tokens and ID tokens are signed with different keys and different algorithms. If you set up Lab Mail, compare its key set with this one: no kid is shared.

Why it matters: a leak of one key affects one purpose in one tenant, and a message signed for one purpose cannot pass as another.

2. Where keys live. Key Management shows public key fields and status only; there is no export or download of a private key. Compare that with clinic.key from the Secrets and key pairs lab, which any process on your laptop could copy.

Why it matters: the tenant signs on the server with keys it stores encrypted. Management access lets you use, retire and disable keys, not take them away, which is the lesson's non-exportable model.

3. Capture an old token and the current key set. Sign in as Ava through $ISSUER/token-decoder and copy the access token. Its header kid is K1.

read -rs T_OLD
btl-lab decode "$T_OLD"
curl -s "$ISSUER/oauth/jwks" > jwks-before.json

4. Publish first. In Key Management choose Generate key, ES256, and note its kid as K2. Then read the published set and how long verifiers may cache it:

GET $ISSUER/oauth/jwks HTTP/1.1
Accept: application/json

Expected: cache-control: public, max-age=300, and the body lists K1 and K2. Wait five minutes before the next step that signs with K2.

Why it matters: verifiers may hold the set for five minutes. Publishing before signing means no verifier ever sees a kid it cannot find.

5. Try to retire K1 now, while the default manager still signs with it. Expected: refused with 409, because the key is in use. Audit shows tenant.oauth.keys.retire rejected.

Why it matters: the tenant will not let you strand its own tokens. Switch signing first, then retire.

6. Switch signing. In Access Token Management, set the default manager's signing key to K2 and save. Get a new access token from the Token Decoder and verify both:

read -rs T_NEW
btl-lab decode "$T_NEW"
for t in "$T_OLD" "$T_NEW"; do btl-lab verify "$t" --issuer "$ISSUER" --audience "$ISSUER/resource" --type at+jwt; done

Expected: T_NEW carries kid K2, and both tokens verify.

7. Retire K1. Its status becomes retiring and it stays in the key set, so T_OLD still verifies.

Why it matters: keep the old public key published until every token it signed has expired. No valid token is rejected during the switch.

8. Once T_OLD has expired (the manager's access token lifetime after you got it), Disable K1. It leaves the key set, and Audit shows tenant.oauth.keys.disable.

9. Emergency drill: pretend K2 leaked. First save the set a cached verifier would still hold, then replace the key without waiting:

curl -s "$ISSUER/oauth/jwks" > jwks-cached.json

Generate K3, point the default manager at K3, then immediately retire and disable K2. Now check T_NEW three ways:

btl-lab verify "$T_NEW" --issuer "$ISSUER" --audience "$ISSUER/resource" --type at+jwt
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T_NEW" "$ISSUER/oidc/userinfo"
K2_KID=<K2's kid from Key Management>
jq --arg kid "$K2_KID" '[.keys[] | select(.kid == $kid)] | length' jwks-cached.json

Expected: the fresh verification cannot find the key, UserInfo returns 401 because the tenant revoked the tokens K2 signed, and the cached set still contains K2 (1).

Why it matters: an emergency means accepting that tokens stop working, because some of them may be forgeries. A verifier holding a stale cache keeps trusting the removed key until it refreshes, which is why caches must be short and refreshed on an unknown kid.

Break it

1. Try to Disable an active key directly, such as K3. It is refused: a key must be retired first, and only a key no manager uses can be retired.

Check your work

Press Check my progress. It looks for:

• tenant.oauth.keys.generate succeeded.

• tenant.oauth.keys.retire rejected (the key still in use).

• tenant.oauth.managers.update succeeded.

• tenant.oauth.keys.retire succeeded.

• tenant.oauth.keys.disable succeeded.

• oidc.userinfo rejected with reason invalid_token (the token signed by the disabled key).

Cleanup

1. Leave K3 active for access tokens.

2. Delete the saved jwks-*.json files and run unset T_OLD T_NEW. Delete ~/btl-keys-lab if you no longer need clinic.key.

3. Optional: repeat the planned rotation for the ID token key through ID Token Management.