Run a production and a staging API and keep their facts correct

Setup

1. Choose Lab Photos as the lab tenant and press Start. Lab Mail plays the staging authorization server.

2. Load ISSUER, CLIENT_ID and CLIENT_SECRET for lab-printer with the helpers from Present an access token correctly; API_ID and API_SECRET for lab-photo-api, exported; and ISSUER2, MAIL_COLLAGE_ID and MAIL_COLLAGE_SECRET for lab-mail-collage in Lab Mail.

3. Start the two deployments in their own terminals, each logging to a file. Each trusts exactly one issuer from its own configuration and reads the key set address from that issuer's metadata:

   • production: btl-lab resource --mode jwt --issuer "$ISSUER" | tee ~/lab-prod-api.log

   • staging: btl-lab resource --mode jwt --issuer "$ISSUER2" --port 8770 | tee ~/lab-staging-api.log

Walkthrough

1. Run the deployment check. Get a production token from Lab Photos (authorize "photos.read", sign in as Ava, exchange, keep PROD=$TOKEN) and a staging token from Lab Mail with lab-mail-collage (keep STAGING). Then:

curl -s -o /dev/null -w 'staging token at production: %{http_code}\n' http://127.0.0.1:8766/photos -H "Authorization: Bearer $STAGING"
curl -s -o /dev/null -w 'production token at staging: %{http_code}\n' http://127.0.0.1:8770/photos -H "Authorization: Bearer $PROD"

Both return 401. The production log shows unknown_kid.

Why it matters: staging servers have test accounts, relaxed policies and more administrators. A production API that accepted their correctly signed tokens would open real photo libraries to all of them.

2. Follow a key rotation without a restart. In Lab Photos, Key Management, Generate key (ES256), then edit Default access tokens to sign with it. Get a new production token and call production: 200. The log shows one key set fetch for the unfamiliar kid.

Why it matters: the key set address came from metadata, not a pasted key, so the new key is found by the ordinary mechanism. A wave of unknown_kid just after a rotation would mean an API is not seeing the new key.

3. Count rejections for an alert threshold:

jq -r 'select(.outcome == "rejected") | "\(.reason) \(.client_id)"' ~/lab-prod-api.log | sort | uniq -c

Decide how many unknown_kid refusals in five minutes should page someone.

4. Rehearse clock drift. Restart production with its clock an hour and two minutes ahead, past the default one-hour token lifetime: btl-lab resource --mode jwt --issuer "$ISSUER" --clock-offset 3720. A fresh token is refused as expired.

Why it matters: one drifting server rejects fresh tokens while its neighbours accept them, so the fault looks random. Monitoring each server's clock offset turns it into an ordinary alert.

Restore: restart production without --clock-offset.

5. Migrate a scope in stages. The rehearsal uses a temporary scope as the "old" broad scope.

   • Add. In Scopes, create lab-tmp-photos (Common). Restart production so it still accepts the old scope wherever the new one is needed: btl-lab resource --mode jwt --issuer "$ISSUER" --alias-scope lab-tmp-photos=photos.read.

   • Measure. Get a token for lab-tmp-photos only and keep it as OLD. Call GET /photos with it: 200, and the log records the alias use with the client_id.

   • Stop new grants. Change lab-tmp-photos to Exclusive, assigned to no client. A new authorize "lab-tmp-photos" request now fails with invalid_scope, while OLD still works at the API.

   • Remove. After the announced date, restart production without --alias-scope. OLD now gets 403 insufficient_scope.

Why it matters: tokens and refresh tokens outlive deployments. Adding, measuring, stopping new grants and announcing come before removal, so no client breaks without warning.

6. Rehearse an authorization server outage. Start the introspecting deployment too: btl-lab resource --mode introspect --port 8768. In Metadata Management, change the Introspection endpoint path to /oauth/introspect-moved and save. With a fresh production token:

   • the introspecting deployment answers 503, logged as introspection_unavailable

   • the JWT deployment keeps answering 200 with its cached keys

Why it matters: decide in advance which routes depend on the authorization server, keep timeouts short, and fail closed only where you must.

Restore: set the Introspection endpoint path back to /oauth/introspect and save.

Break it

Step 4 is the clock failure and step 6 the outage, each restored in place. Step 5's removal deliberately breaks a client that kept using the old scope, which is why it comes last.

Check your work

Press Check my progress. The checks look for, in order: the new signing key, the creation of lab-tmp-photos, a token carrying it, the change that stopped new grants, the refused request for it, and the moved introspection endpoint.

The API logs should show unknown_kid, expired, the alias use, insufficient_scope after removal, and introspection_unavailable.

Cleanup

1. Confirm the Introspection endpoint path is /oauth/introspect.

2. Delete the scope lab-tmp-photos. Deleting a scope also revokes tokens that carry it.

3. Keep the new key active on Default access tokens, and disable the old key after an hour.

4. Stop all three deployments and delete their logs: rm ~/lab-prod-api.log ~/lab-staging-api.log. Run unset PROD STAGING OLD TOKEN.

Missing infrastructure

• G3 Hosted protected resource with RFC 9728 metadata. The deployments run locally from the toolkit. With a hosted photo API publishing /.well-known/oauth-protected-resource, step 5 would also update its scopes_supported in the same release as the scope change, generated from the configuration the API enforces, and a client could discover the API's authorization server from it.

• G66 Second lab tenant: this lab uses Lab Mail, a second tenant. Additional tenants currently need a paid subscription or a BTL grant, so an ordinary learner can do only the Lab Photos steps until every learner can have a second lab tenant.