Follow your tenant's certificate chain to a trust anchor

Setup

You need tenant.pem and the ~/btl-pki-lab folder from the previous lab, and openssl 3.x. The private PKI in step 7 lives only on your machine and is deleted at the end of the certificate labs. Nothing in the tenant changes.

cd ~/btl-pki-lab
ISSUER=https://tenant-<id>.beyondthelogin.dev
HOST=${ISSUER#https://}

Note: if openssl verify on your system has no default trust store, add -CAfile with your operating system's CA bundle to the commands in steps 3 and 4.

Walkthrough

1. See what the server sends:

openssl s_client -connect "$HOST:443" -servername "$HOST" -showcerts </dev/null 2>/dev/null \
  | awk '/BEGIN CERT/{n++; f="chain-" n ".pem"} f{print > f} /END CERT/{f=""}'
for f in chain-*.pem; do echo "== $f"; openssl x509 -in "$f" -noout -subject -issuer; done

Expected: the leaf (chain-1.pem) and one or more intermediates. The leaf's issuer equals the next certificate's subject. The root is not sent.

Why it matters: this is the lesson's chain. The server supplies its certificate and the intermediate, and the client completes the chain with a root it already trusts.

2. Keep authorities in check. Look at the intermediate:

openssl x509 -in chain-2.pem -noout -ext basicConstraints,keyUsage

Expected: CA:TRUE, often with a path length, and certificate signing in the key usage.

Why it matters: only certificates marked as CAs may sign other certificates, and a path length limits how many levels may appear below them.

3. Follow the chain with your trust store:

openssl verify -show_chain -untrusted chain-2.pem chain-1.pem

Expected: OK and a chain that ends at a root from your trust store.

4. The missing intermediate:

openssl verify chain-1.pem

Expected: error 20 ... unable to get local issuer certificate.

Why it matters: this is the error some clients see when a server sends only its own certificate. The fix belongs on the server, never in asking users to trust the server certificate directly.

5. Trust anchors are a decision. Connect with an empty trust store, then fetch the root named by the intermediate's issuer address and trust only that:

openssl s_client -connect "$HOST:443" -servername "$HOST" -no-CAfile -no-CApath -no-CAstore </dev/null 2>/dev/null | grep 'Verify return code'
openssl x509 -in chain-2.pem -noout -ext authorityInfoAccess
curl -s -o anchor.der <CA Issuers URL from the output>
openssl x509 -inform DER -in anchor.der -out anchor.pem 2>/dev/null || cp anchor.der anchor.pem
openssl x509 -in anchor.pem -noout -subject -issuer
openssl s_client -connect "$HOST:443" -servername "$HOST" -CAfile anchor.pem -no-CApath -no-CAstore </dev/null 2>/dev/null | grep 'Verify return code'

Expected: a failure with the empty store, a root whose subject equals its issuer (self-signed), and Verify return code: 0 (ok) with only that root. If the URL returns another intermediate rather than the root, repeat with its issuer address.

Why it matters: the root is trusted because a trust store contains it, not because of its self-signature. Anyone can create a self-signed certificate claiming any name.

6. Certificate Transparency. Open https://crt.sh/?q=<your tenant host>, or the parent domain if your host is covered by a wildcard. Compare the logged certificates with the SCTs listed in tenant.pem.

Why it matters: public, append-only logs let the domain owner notice a certificate it never requested.

7. A private PKI on your machine: a root, an issuing CA, and a server certificate for localhost.

openssl req -x509 -new -noenc -newkey ec -pkeyopt ec_paramgen_curve:P-256 -keyout root.key -out myroot.pem -days 30 \
  -subj "/CN=Lab Root CA" -addext "basicConstraints=critical,CA:TRUE" -addext "keyUsage=critical,keyCertSign,cRLSign"
openssl req -new -noenc -newkey ec -pkeyopt ec_paramgen_curve:P-256 -keyout int.key -out int.csr -subj "/CN=Lab Issuing CA"
printf 'basicConstraints=critical,CA:TRUE,pathlen:0\nkeyUsage=critical,keyCertSign,cRLSign\n' > int.ext
openssl x509 -req -in int.csr -CA myroot.pem -CAkey root.key -out int.pem -days 30 -extfile int.ext
openssl req -new -noenc -newkey ec -pkeyopt ec_paramgen_curve:P-256 -keyout leaf.key -out leaf.csr -subj "/CN=localhost"
printf 'basicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature\nextendedKeyUsage=serverAuth\nsubjectAltName=DNS:localhost\n' > leaf.ext
openssl x509 -req -in leaf.csr -CA int.pem -CAkey int.key -out leaf.pem -days 7 -extfile leaf.ext

Run a server that sends the full chain, and connect trusting only your root:

openssl s_server -accept 8443 -cert leaf.pem -key leaf.key -cert_chain int.pem -www &
openssl s_client -connect localhost:8443 -CAfile myroot.pem -no-CApath -no-CAstore -verify_hostname localhost </dev/null 2>/dev/null | grep 'Verify return code'
openssl s_client -connect localhost:8443 -verify_hostname localhost </dev/null 2>/dev/null | grep 'Verify return code'

Expected: 0 (ok) with your root, and a failure with your normal trust store.

Why it matters: a private PKI is trusted only by systems told to trust its root, which is exactly right for services the public never visits.

Break it

1. Stop the server with kill %1, start it again without -cert_chain int.pem, and connect with -CAfile myroot.pem as before. Expected: Verify return code: 20 (unable to get local issuer certificate). This is the lesson's misconfigured server, reproduced.

2. Try to make the leaf act as a CA. Sign another certificate with leaf.pem and leaf.key, then verify it:

openssl req -new -noenc -newkey ec -pkeyopt ec_paramgen_curve:P-256 -keyout bad.key -out bad.csr -subj "/CN=bad.localhost"
openssl x509 -req -in bad.csr -CA leaf.pem -CAkey leaf.key -out bad.pem -days 1
cat int.pem leaf.pem > untrusted.pem
openssl verify -CAfile myroot.pem -untrusted untrusted.pem bad.pem

Expected: verification fails, because the leaf says CA:FALSE.

Check your work

This lab does not touch the tenant's services, so there are no tenant events to check. You should have:

• openssl verify -show_chain output for your tenant's chain, ending at a root

• error 20 for the missing intermediate, from both the tenant leaf and your local server

• Verify return code: 0 (ok) against your tenant using only anchor.pem, and against your local server using only myroot.pem

• a failed verification for the certificate signed by the leaf

Cleanup

1. Stop s_server with kill %1. Delete bad.* and untrusted.pem.

2. Keep myroot.pem, root.key, int.*, leaf.* and the tenant chain files for the next two labs.