Interoperability and conformance testing

Why a specification is not enough

Developers read the same text differently, and two kinds of mistake follow.

Some mistakes reject what should be accepted. A server might refuse a client assertion whose iat is half a second in the future, although the profile requires a tolerance of up to 10 seconds. A client might fail on a DPoP-Nonce challenge it never expected. These are interoperability failures. They are annoying, but they announce themselves quickly, because honest partners hit them.

Other mistakes accept what should be rejected. A server might accept a code a second time, forget to check that a token request carries a proof from the key the code was bound to, or accept authorization requests that did not come through PAR. A client might never compare the iss parameter at all. These are security failures, and they stay silent. Honest partners never send the bad request, so everything works, and a server that rejects nothing at all looks perfect to every honest client.

The FAPI 2.0 profile makes the point itself: its protections depend on complete and correct implementations, and it says deployments should use certified implementations. Finding the silent failures needs someone who deliberately sends the requests an honest partner never would.

Conformance testing

The OpenID Foundation maintains a conformance suite for that purpose. It is open source and free to use, either as a hosted service or run locally, for example as part of an automated build. It plays the other side of the exchange. To test an authorization server, the suite acts as its clients, including calls to a protected API that the tester points it to. To test a client, it acts as the authorization server and the API.

A tester chooses a test plan and configures it to match the implementation: which client authentication method, which way of binding tokens, whether OpenID Connect is involved. The plan then runs a series of test modules, each one a scenario. Some follow the normal flow. Many are negative: they make a deliberate mistake and check that the implementation refuses it.

Test plan: FAPI 2.0 Security Profile, authorization server
Client authentication: private_key_jwt
Sender constraining: DPoP

PASSED   complete flow succeeds
PASSED   authorization request without PAR is rejected
PASSED   second use of the same code is rejected
PASSED   client assertion with the wrong audience is rejected
FAILED   token request with a proof from a different key is accepted
REVIEW   error page shown for a request_uri the server never issued

This summary is illustrative. The real suite names its modules differently and runs many more of them. Its shape is accurate, though. Setting up a server test involves registering two separate clients with different keys or certificates, so the suite can try mixing them, for example by presenting one client's access token with the other client's certificate or key. Some tests also involve a person: when the correct behavior is for the authorization server to show an error page instead of redirecting anywhere, the tester records that page as evidence.

The failed line above is exactly the silent kind of failure. Lakeside's server would work with every honest client, and its tokens would still not be properly bound.

Certification

Once an implementation passes a test plan, its developer can apply to the OpenID Foundation for certification. The developer submits the test results with a signed declaration of conformance, a formal statement to the foundation and to the public that the tested deployment conforms, and pays a fee. The foundation publishes the certification, naming the implementation and the profile and options it was certified for, and the implementation may then carry the OpenID Certified mark.

This process is self-certification. The developer runs the tests and makes the declaration, rather than an independent auditor. Its value comes from the shared test suite and from the public, formal nature of the declaration. Some ecosystems require certification before a bank or a third party may join, which gives every participant the same baseline evidence about the others.

For the printer, certification is a reason to choose a library. A certified client library has been through the negative tests the printer would otherwise have to write itself.

What testing proves

A passing test plan shows that a particular implementation, in a particular configuration, behaved as the profile requires in the scenarios the suite covers, at the time it was tested. That is valuable evidence, and it is narrower than it sounds.

It does not cover other configurations, or the next release. A server certified with DPoP has not been tested with mutual TLS, and a change made after certification has not been tested at all. Teams that rely on the suite run it again whenever their implementation or configuration changes, ideally automatically.

It does not cover how the deployment is operated: where private keys are kept, how certificates are renewed, who can change client registrations, whether anyone notices unusual traffic. The FAPI 2.0 Attacker Model left those out on purpose, and so does the suite.

And it does not cover the application around the protocol. The suite can confirm that the printer's token is bound and that Lakeside's API checks the binding. It cannot confirm that the consent screen showed 42.50 EUR and the right payee, or that the payment API took the money from the account you chose. The printer and Lakeside still need their own tests for their own business rules.

Conformance testing is one part of building and running OAuth well over time. Implementation and operations takes up the rest: building clients and servers, testing integrations, and keeping them working once they are live. For sign-in, Conformance testing returns to the same suite and its OpenID Connect test plans.