Rotating credentials and keys after exposure

A secret in the wrong place

On Day 1 at Cedar Inc., an attacker who had taken over the account of Riley, in finance, read an email from Cedar's payments vendor. Months earlier, the vendor had sent Cedar a new API key for its payment integration, pasted into the body of that message, and the message was still in Riley's mailbox. At 11:30 the attacker read it through Invoice Sync, an app they had approved on Riley's account, using the access that approval granted. Quinn, on Cedar's security team, found that read on Day 2 while investigating, as Investigating an account compromise describes.

That key was never meant to be in a mailbox. It belongs in the payment service's secrets store, readable only by the systems that use it. Once a copy sat in Riley's mail, anyone who could read that mailbox could use the key, and on Day 1 that included the attacker. A secret is exposed when a copy reaches anywhere outside the people and systems meant to hold it. Exposure does not prove misuse, but the holder rarely knows who else saw the copy, so the safe assumption is that someone did.

Mailboxes are only one wrong place. Secrets land in code repositories when a configuration file is committed, in build logs when a script prints its settings, in tickets and chat when someone pastes an error message, in screenshots and shared documents, and in operational logs that recorded a whole request, which Recording identity activity warns against. Deleting the copy is necessary, but it is never enough. Whoever saw it may already have taken it, as Issuing, renewing, and revoking certificates showed with a private key committed to a public repository.

How leaks are found

Cedar learned of its exposure from an investigation, which is common but late. Other leaks are found deliberately. Secret scanning searches code repositories, build logs, tickets and chat for strings shaped like credentials. It works best when secrets have a recognizable format. Many providers now give their keys a fixed prefix, so a scanner can tell a payments key from random text, and some providers watch public code for their own keys and revoke or report any they find.

How a secret is used gives the next clue. A key that suddenly makes requests from an unfamiliar network, at unusual hours, or for operations its service never performs is behaving like a key in someone else's hands. Recording which credential each request used, by its identifier rather than its value, makes that visible, a point Rotating client credentials makes about client secrets. Some teams also plant decoy credentials: realistic-looking keys that no legitimate system ever uses, left where an intruder would look, so that any use at all is an alert.

And sometimes someone simply tells you: a colleague notices a key in a screenshot, a researcher reports one in a public repository, or a vendor calls because a key is being used from somewhere new. Those reports only help if there is a clear place to send them and someone ready to act the same day.

Knowing what depends on it

Before replacing a secret, you need to know what will break. An inventory of secrets answers that. For each secret it records what the secret unlocks, which systems use it, where it is stored, who owns it, when it was last replaced and how to replace it. Without an inventory, every rotation starts with a search for copies, carried out under pressure.

Cedar's inventory listed the payments key with one consumer: the service that submits payment runs. When Cedar replaced the key, a reconciliation script on a finance colleague's laptop failed the next morning. Someone had copied the key into that script months earlier. The failure was the first anyone knew of that copy, which also meant the key had been sitting on a laptop, outside the secrets store, all along.

What a secret can do matters as much as where it lives. Cedar's payments key could read payment status and also create payments, which made its exposure urgent. A key that can only read a public price list could wait for an ordinary rotation.

Rotating without an outage

Most replacements are planned: a schedule comes due, someone with access leaves, or a stronger algorithm becomes available. A planned replacement, or rotation, can run without an outage by overlapping the old and new secrets, as Rotating client credentials describes. Add the new secret while the old one still works, deploy it everywhere, confirm that nothing uses the old one any more, and only then revoke it. Signing keys follow the same pattern through their published public keys, as Storing and using keys explains, and certificates are renewed well before they expire.

The confirmation step is where an inventory pays off. A provider that records when each secret was last used shows whether anything still presents the old one. Cedar's forgotten script would have shown up there, still using the old key after the payment service had switched. Planned rotation is also rehearsal. A team that replaces its secrets routinely, ideally automatically, already has the steps written down and tested when one of them leaks.

When there is no time to overlap

Overlap assumes nobody else holds the old secret. Once a secret has reached someone hostile, every minute it keeps working is a minute they can use it, so the order flips. Revocation makes a credential stop working immediately, whatever its expiry date. In an emergency, the exposed secret is revoked first, and the new one is issued and deployed afterward, accepting an outage in between.

Cedar faced exactly this choice on Day 2. The key could create payments, and an attacker had read it. Cedar's payments team asked the vendor to revoke the key straight away, paused the day's payment run, and collected a new key from the vendor's portal rather than by email. The payment run went out a couple of hours late.

Not every exposure calls for this. The decision rests on a few questions. Is it likely that someone hostile has the secret? What can it do? How public was the exposure, and for how long? Does the provider allow two secrets at once, and how quickly can a new one be deployed? A key pasted into a private ticket that three colleagues saw might be rotated with an overlap the same afternoon. A key in a public repository, or in an attacker's hands, is revoked now. Some providers offer steps in between, such as restricting the old key to known network addresses, which buy time for a fast overlap. And when the issuer is another organization, as Cedar's vendor was, revocation may need their help, so the contact and procedure belong in the inventory too.

What else the secret touched

Revocation stops future use. It does nothing about what the secret did while it was exposed. Cedar asked the vendor for every request made with the old key since 11:30 on Day 1. The records showed requests only from Cedar's own network addresses, and no payments Cedar had not made itself. That answer is only as good as the vendor's records. Had the vendor not recorded where each request came from, Cedar could not have told its own requests from an attacker's.

Some secrets leave consequences that outlive them. A client secret could have been used to obtain tokens, and those tokens stay valid until they expire unless they are revoked as well. An exposed token-signing key is worse: anyone holding it could have minted tokens that nobody issued, so replacing it is not enough. Every verifier has to stop trusting the old key, and everything it signed has to be treated as suspect, as Forged tokens and stolen signing keys explains.

For a signing key, the inventory is the list of everything that trusts it: each application that verifies its tokens, how often each one fetches the published keys again, and each partner that installed a SAML signing certificate by hand. Overlap is ruled out, so the exposed key leaves the published set at once, and the outage falls on everyone holding a token it signed, who has to sign in or refresh again. The review then compares the tokens that verifiers accepted under the old key with the issuer's own records of what it issued, because a token with no matching issuance was forged. Signing keys and rotation shows how verifiers pick up a replaced key.

An exposed private key behind a certificate means revoking that certificate. And a key that encrypted data may have exposed the data itself, which no rotation can take back.

Finally, fix the place the secret leaked from. Cedar asked its vendor to stop sending keys by email, and now collects them from the vendor's portal straight into its secrets store. A build that printed a secret has to stop printing it, and the other logs that build wrote need checking for the same mistake.

Here is a different leak to work through, at the photo application.

[build 4471] 14:02:31 Loading configuration for export-job
[build 4471] 14:02:31 EXPORT_CLIENT_ID=photos-export
[build 4471] 14:02:31 EXPORT_CLIENT_SECRET=example-only-7Hq2vX9mRk4Tz
[build 4471] 14:02:33 Configuration loaded

Time      Client         Grant               Source        Result
14:30:05  photos-export  client_credentials  192.0.2.15    token tok_a1, expires 15:30
15:12:47  photos-export  client_credentials  203.0.113.9   token tok_b7, expires 16:12
16:30:04  photos-export  client_credentials  192.0.2.15    token tok_c3, expires 17:30
Export servers: 192.0.2.15 and 192.0.2.16

Fewer secrets to leak

The surest way not to leak a secret is not to have one. Many long-lived shared secrets can be replaced with something harder to copy, or worthless once copied. A confidential client can authenticate with a key pair instead of a client secret, signing short-lived assertions with a private key that never leaves its server, as Secrets and signed assertions describes. Kept in hardware that refuses to export it, that key cannot be pasted into a ticket at all.

Software running on a cloud platform or in a build system can often get short-lived credentials from that platform, proving what it is through the platform's own identity instead of a stored secret. Short lifetimes help even where a secret must exist: a credential that expires in an hour gives a leak an hour of value, not a year. Tokens bound to a key, using DPoP or mutual TLS, are useless to someone who copies only the token.

Narrow permissions limit what any leak can do. Had Cedar's key been able to read payment status but not create payments, the attacker would have held a view of Cedar's payments rather than a way to make them, and the hours between 11:30 on Day 1 and the revocation on Day 2 would have mattered far less.