Key selection and validation failures

Opening both layers

The printer registered UserInfo responses that are signed with RS256 and then encrypted with RSA-OAEP-256 and A256GCM. It opens each response in this order:

1. Expect the registered form. The response must be application/jwt with five parts. Plain JSON, or a signed JWT with three parts, is rejected even if everything in it would check out. The specification gives the same rule for ID tokens when encryption was registered.
2. Check the outer header against the registration. alg must be exactly RSA-OAEP-256 and enc exactly A256GCM. Anything else is rejected before any decryption is attempted.
3. Select the decryption key. kid must name one of the printer's own private keys for encryption, and that key's one algorithm must be the algorithm in the header.
4. Decrypt. If the authentication tag does not verify, stop.
5. Verify the inner JWT. Its alg must be the registered RS256, and its key comes from the photo service's key set, found through configuration and selected by kid, here photos-rs-2026-09.
6. Check the claims. iss must be exactly https://auth.photos.example, aud must include photo-printer, and sub must equal the ID token's sub, as The UserInfo endpoint required.

The two layers use keys from opposite sides: the printer's own private key to decrypt, because the message was encrypted to the printer, and the photo service's public key to verify, because the photo service signed it. A library configured with one mixed list of keys for both jobs can fail in confusing ways or accept more than it should. Keeping the two sources apart makes the order easy to follow:

outer = parse_jwe(body)                                     # five parts, or reject
if outer.header.alg != "RSA-OAEP-256" or outer.header.enc != "A256GCM":
    return reject("unexpected_encryption_algorithm")

key = printer_private_keys.find(outer.header.kid, use="enc")  # the printer's own keys
if key is None or key.alg != outer.header.alg:
    return reject("decryption_failed")
inner = decrypt(key, outer)                                 # None if the tag does not verify
if inner is None or outer.header.cty != "JWT":
    return reject("decryption_failed")

signing_key = photo_service_keys.find(inner.header.kid)     # from the photo service's jwks_uri
if signing_key is None or inner.header.alg != "RS256":
    return reject("unexpected_signing_key_or_algorithm")
if not verify(signing_key, inner):
    return reject("bad_signature")
check_userinfo_claims(inner.claims, id_token)               # iss, aud, sub

The helper names are illustrative. An unknown decryption kid gets the same rejection as a failed decryption, for reasons the last section explains.

When the algorithms do not match

The algorithm check in step 2 can look pedantic. It is the step that matters most when someone is probing on purpose.

Suppose a message arrives whose header names RSA1_5 with the printer's key printer-enc-2026-10. RSA1_5 is an older RSA encryption scheme with a long history of attacks. A recipient that reveals whether a decrypted key was correctly formatted, through a different error or a different response time, lets an attacker who can submit many crafted messages work out an encrypted key piece by piece. Changing a captured message's header from an OAEP algorithm to RSA1_5 is a known way to set up that attack, even though the message was encrypted with OAEP. A key restricted to one algorithm refuses at once, and current guidance says to avoid RSA1_5 altogether.

Other mismatches have duller causes but get the same answer:

If the photo service has a good reason to change algorithms, the change belongs in the registration, made deliberately by both sides, not in a list the printer widens until errors stop.

Rotating encryption keys

Signing keys and encryption keys rotate in opposite directions. When the photo service replaces its signing key, it is the sender: it starts signing with the new key when it chooses, and the new kid tells the printer to fetch the key set again, as Signing keys and rotation described.

With encryption, the printer is the recipient, and the sender picks the key. The photo service chooses from its cached copy of the printer's key set, which it may keep for the hour that the set's Cache-Control: max-age=3600 allows. The printer cannot signal anything through kid, because the photo service writes the kid. So the printer rotates like this:

1. Generate printer-enc-2027-01, keeping its private key with the sign-in service.
2. Publish it, and at the same time remove printer-enc-2026-10 from the published set, so that any fresh copy of the set offers only the new key.
3. Keep the old private key, unpublished, for at least the cache lifetime plus a margin. Until every cached copy has expired, some messages will still be encrypted to the old key, and the printer must be able to open them.
4. Once responses name only the new kid, retire the old private key.

Step 3 is the one the printer skipped. It deleted the old private key at the moment it published the new one. The photo service had fetched the printer's set forty minutes earlier, so for the remaining twenty minutes of its cache it went on encrypting to a key the printer no longer had.

When the printer encrypts request objects to the photo service, the roles reverse. The printer picks a key marked use enc from the photo service's set and refreshes its copy when that set's cache lifetime runs out.

A leaked encryption key is worse than it first looks. Removing it from the published set stops new messages being encrypted to it, but anyone who recorded earlier responses can now decrypt them. The printer replaces the key at once and treats everything encrypted to it as exposed.

Failing without giving clues

When a message fails to decrypt, the error has two audiences: the person in front of the screen, and anyone trying to learn something from the failure.

The second audience matters wherever an attacker can submit encrypted messages and watch the result, as anyone can with request objects sent to the photo service through a browser, or with encrypted ID tokens sent to a relying party that accepts them through the browser. A recipient that reports a padding error for one message and a length error for another, or answers some faster than others, gives an attacker the signal the RSA1_5 attacks depend on. The JWE specification therefore requires recipients not to distinguish between format, padding, and length errors in the encrypted key, and recommends carrying on with a random key when that key is malformed, so that every failure surfaces at the same later step. A JWE library should handle this internally, and the code around it must not undo that work by reporting more detail.

The printer's UserInfo responses arrive only from the photo service, over a connection the printer opened, so an attacker has no such opportunity there. The same discipline costs nothing, and it carries over to every place the printer does accept encrypted content:

• One outcome for every decryption failure, both in what the caller sees and in the error that leaves the decryption code.
• A log entry with the stage (decryption, signature, or claims), the kid and algorithms from the header, the issuer, and a correlation ID. Never the message, the decrypted claims, or any key material.
• For you, a plain message that sign-in could not be completed, and a way to try again.

The tempting fixes are the ones When validation fails warned against, in a new form: accepting unencrypted responses until the provider sorts things out, adding algorithms until the errors stop, or skipping the inner signature because the outer layer decrypted. Any of them would have hidden the printer's twenty-minute outage instead of explaining it. The real fix was to keep the old private key for the cache lifetime.