Conformance testing

Tests written from the specification

To test a provider, the tester gives the suite the provider's discovery address and registers clients whose redirect URIs point back to the suite, which then sends authentication requests and examines every response. To test a relying party, the tester configures it to use the suite as its provider and signs in through it, and the suite issues the codes, tokens, and key sets.

Tests are grouped into test plans, one for each profile and configuration, such as the code flow with a client registered in advance. Many tests are ordinary sign-ins examined closely. Others probe behavior the specification defines for unusual requests. A provider is sent prompt=none when no one is signed in and should answer with an error such as login_required. It is sent the same code twice and should refuse the second exchange. It is asked for max_age=1 and should make the person sign in again.

Some of these need a person. When the right behavior is a page the suite cannot see, such as a second sign-in page or an error page shown instead of a redirect, the tester uploads a screenshot of it, and the result waits for review.

Profiles and certification

A profile is a named set of features with its own test plan. A provider is tested against the profiles it claims to support:

Relying parties have matching profiles: the basic, implicit, and hybrid flows, configuration through discovery, dynamic registration, form post, and logout. Further plans cover features such as third-party initiated login.

Certification follows the self-certification process the FAPI lesson described: the implementer runs the plan for a profile against its own deployment and submits the logs, any screenshots, and a signed declaration. To qualify, no test in the plan may have failed or been interrupted, and the results behind OpenID Connect certifications are published, so anyone can see which tests were run and what happened.

A certification describes the deployment that was tested. The Foundation treats a different protocol endpoint as a different deployment that needs its own certification, and a certification for one profile says nothing about the others. A provider certified for the basic profile has not shown that its form post responses are right.

Testing a relying party

A provider's tests mostly check that it produces correct messages. A relying party's tests are more revealing, because the most important thing a relying party does is refuse. When the suite plays the provider, many of its tests send something wrong on purpose, and each of these should end without anyone being signed in:

• an ID token whose signature does not verify
• an ID token whose iss is not the provider's issuer
• an ID token whose aud is missing or does not include the client
• an ID token whose nonce differs from the one in the request
• an ID token with no sub or no iat
• an implicit or hybrid response whose at_hash or c_hash is missing or wrong

The suite cannot see the error page the relying party shows. It watches what the relying party does next. A relying party that accepted an ID token with the wrong nonce would carry on, perhaps calling the UserInfo endpoint with the access token it received, or in the hybrid flow redeeming the code. A relying party that refused makes no further request, and the test passes when nothing arrives within a waiting period.

Other tests check handling rather than refusal. The suite answers UserInfo with a different sub, and the relying party must discard that answer. It signs one ID token, then rotates its keys and signs the next with a key published only after the first sign-in, and the relying party must fetch the key set again and accept it. These are the same behaviors The UserInfo endpoint and Signing keys and rotation described, now checked by someone else.

The suite also follows the specification exactly where this series chose to be stricter. For an ID token received directly from the token endpoint in the code flow, the specification lets a relying party rely on its TLS connection instead of the signature, so a relying party that skips that signature check is not failed for it. The printer verifies the signature anyway, as Receiving the ID token explained, and its own tests are where that choice is enforced.

What passing proves

A passing result shows that, in the tested configuration, the implementation produced the messages the specification requires and refused the faults the tests present. The earlier lesson set out its general limits: one configuration, one point in time, and the protocol rather than the application around it. For a relying party built on a certified library, those limits take particular shapes:

• The configuration is the printer's. A library certified with RS256 and the code flow says nothing about an application that configures it to accept other algorithms, widens its clock tolerance, or turns off the nonce check. The application inherits the certification's meaning only if it keeps the configuration that earned it.
• Accounts and sessions are untested. No conformance test checks that the printer finds accounts by issuer and subject rather than email, replaces its session identifier after sign-in, or ends its own session before redirecting to the provider at sign-out.
• The faults are known ones. The tests cover the failure cases their authors chose. Passing them is not a security review of the printer.

For the printer, conformance tests and its own tests do different jobs. The suite checks the printer's reading of the specification. The printer's tests, built on the seams from Implementing a relying party, check the decisions the specification leaves to the printer. Running the relying party test plans again before upgrading the library or changing its configuration catches regressions before customers meet them. When choosing a provider or a library, certification is a useful filter, as long as you check that it covers the profiles you will actually use.