Rotate a client certificate without an outage

Setup

1. Keep the CA and certificate files, the b64url helper and the lab-print-orders variables from the previous two labs.

2. Planned (G8, G27, G57): lab-tmp-mtls-orders uses tls_client_auth with certificate-bound access tokens on, and lab-ca.crt is its tenant's trust anchor.

Planned walkthrough

1. Renew with the same request and SAN, with overlapping validity. This step runs today.

openssl x509 -req -in orders.csr -CA lab-ca.crt -CAkey lab-ca.key -CAcreateserial -out orders-renewed.crt -days 90 -extfile client-ext.cnf
for c in orders.crt orders-renewed.crt; do openssl x509 -in $c -noout -serial -enddate; openssl x509 -in $c -outform DER | openssl dgst -sha256 -binary | b64url; echo; done

The serials and thumbprints differ, even though the key is the same.

Why it matters: "Tokens bound to the old certificate". The thumbprint covers the whole certificate, so every renewal changes it.

2. Get a token over the alias with orders-renewed.crt. It succeeds with no registration change, and cnf holds the new thumbprint.

Why it matters: "Renewing without changing the registration". The PKI method registers a subject, not a certificate.

3. Use a token bound to the old certificate over a connection made with the renewed one. Expect 401 invalid_token. Discard tokens bound to the old certificate and request new ones.

Why it matters: keep cached tokens alongside the certificate they were obtained with, and drop them when that certificate is replaced.

4. Switch the client to self_signed_tls_client_auth. Create a second self-signed certificate, add it to the client's key set, wait until the tenant shows both, switch, then remove the old one.

Why it matters: register first, switch second. The reverse order produces invalid_client until the new certificate is added.

5. Change the trust anchor: create lab-ca-2, upload it beside lab-ca.crt, issue from it, and remove lab-ca.crt only after the last certificate it issued has expired.

Why it matters: the authorization server trusts a new CA before any partner receives a certificate from it.

Do today

1. Run Planned walkthrough step 1 and record both thumbprints.

2. Write the certificate inventory the lesson asks for, with one alert date for each.

for c in lab-ca.crt orders.crt orders-renewed.crt; do echo "$c"; openssl x509 -in $c -noout -subject -issuer -enddate; done

Include the partner CA's own certificate: it is one you depend on but would not control in real life.

3. See what "only configured trust anchors" means. Create a second CA you will never upload and issue a certificate with the same SAN from it.

openssl req -x509 -new -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -keyout lab-ca-3.key -out lab-ca-3.crt -days 30 -subj "/CN=Lab Other CA 3" \
  -addext "basicConstraints=critical,CA:TRUE" -addext "keyUsage=critical,keyCertSign,cRLSign"
openssl x509 -req -in orders.csr -CA lab-ca-3.crt -CAkey lab-ca-3.key -CAcreateserial -out orders-other.crt -days 30 -extfile client-ext.cnf
openssl verify -CAfile lab-ca.crt -purpose sslclient orders-other.crt

Verification against your configured anchor fails, though the name inside is identical. A server that trusted every CA for client authentication would accept it.

4. Compare with the tenant's signing key rotation, which supports overlap today. In OAuth > Signing keys, generate a new key and point the access token manager at it. The old key moves to retiring and stays in the JWKS.

GET $ISSUER/oauth/jwks

Both keys are listed, so tokens signed before the switch still validate.

5. Compare with client secret rotation, which has no overlap. Rotate lab-print-orders' secret in OAuth > Clients, then request a token with the old one.

curl -s -u "$ORDERS_ID:$ORDERS_SECRET" "$ISSUER/oauth/token" -d grant_type=client_credentials -d scope=prints.create | jq .
read -rs ORDERS_SECRET; export ORDERS_SECRET     # store the new secret

The old secret fails at once with 401 invalid_client, and Audit shows oauth.token rejected invalid_client. That is the outage a careful certificate rotation avoids.

Break it

These run once the gaps close.

1. In step 4, switch to the new self-signed certificate before registering it. Expect 401 invalid_client until you add it.

2. Use orders-other.crt from Do today step 3 at the alias. Expect 401 invalid_client: only configured anchors count.

Check your work

Today, Audit shows tenant.oauth.keys.generate and the token manager update, tenant.oauth.credentials.rotate for lab-print-orders, and oauth.token rejected invalid_client for the old secret. Once the gaps close, Audit also shows client key set and trust anchor changes, oauth.token successes with each certificate, and the Break it refusals.

Cleanup

1. Leave the new signing key active; retire the old one only after the tokens it signed have expired.

2. Make sure your stored ORDERS_SECRET is the new secret.

3. Delete the local files from the three mutual TLS labs: lab-ca*, orders*, self.crt and client-ext.cnf. Once the gaps close, also delete lab-tmp-mtls-orders and the uploaded CAs.

Missing infrastructure

• G8, G27 and G57: mutual TLS client authentication, tenant trust anchors and the alias host, as in the previous two labs.

• The tenant portal should also show expiry for uploaded CAs and registered certificates, so an administrator can watch them as the lesson recommends.

• Once these exist, the Planned walkthrough runs as written.