Recover from lost answers, late writes, drift and a backlog

Setup

You play the identity system's provisioning service. Its intended state for one group is a file it owns.

1. Open a bash shell and set the variables and helpers from the directory lab.

2. Write the intended state for Photo moderation:

cat > intended.json <<'EOF'
{"group": "Photo moderation", "members": ["[email protected]", "[email protected]"]}
EOF
export MOD=$(scim -G "$SCIM/Groups" --data-urlencode 'filter=displayName eq "Photo moderation"' | jq -r '.Resources[0].id')

3. In Groups, make Photo moderation match the file: Ava and Cora only.

4. Press Start on the lab page.

Walkthrough

1. A request with no answer. Send a create and throw the response away, as a timeout would.

newuser() { jq -n --arg u "$1" --arg x "$2" --arg g "$3" --arg f "$4" '{schemas:["urn:ietf:params:scim:schemas:core:2.0:User"], externalId:$x, userName:$u,
  name:{givenName:$g,familyName:$f}, emails:[{value:($u+"@example.com"),type:"work",primary:true}], active:true}'; }
newuser lab-tmp-dana E10900 Dana Okafor | scim -o /dev/null -X POST "$SCIM/Users" --data-binary @-

You do not know whether it worked, and you have no id.

2. Look for your own key before anything else.

GET $ISSUER/scim/v2/Users?filter=externalId%20eq%20%22E10900%22 HTTP/1.1
Authorization: Bearer $TOKEN
Accept: application/scim+json

totalResults is 1: the create worked and only the answer was lost. Store the id with export DANA=<id>.

Why it matters: this is "A request with no answer". The outcome was unresolved until the search settled it. Only a search that finds nothing makes a second create safe.

3. Make a create idempotent. SCIM defines no idempotency key, but this tenant accepts X-Correlation-ID on a create and returns the original result for a repeat.

export CID=$(uuidgen | tr A-Z a-z)
newuser lab-tmp-jo E10901 Jo Brandt | scim -X POST "$SCIM/Users" -H "X-Correlation-ID: $CID" --data-binary @- | jq -r .id
newuser lab-tmp-jo E10901 Jo Brandt | scim -X POST "$SCIM/Users" -H "X-Correlation-ID: $CID" --data-binary @- | jq -r .id
newuser lab-tmp-jo2 E10902 Jo Brandt | scim -X POST "$SCIM/Users" -H "X-Correlation-ID: $CID" --data-binary @- | jq .

The first two return 201 with the same id. Search Audit for $CID: the original scim.user.create and a second record whose detail says it was replayed, with no new user. The third, the same key with a different body, returns 409 with command_conflict.

Why it matters: this is "Safe retries". An idempotent request can simply be sent again after a timeout. A key reused for a different change is refused rather than guessed at.

4. A late write. Build a full replacement from Dana's current copy, but hold it back.

export E1=$(scim -i "$SCIM/Users/$DANA" | awk 'tolower($1)=="etag:" {print $2}' | tr -d '\r')
scim "$SCIM/Users/$DANA" | jq 'del(.meta, .groups, .id) | .title = "Print technician"' > dana.json

Now the deactivation arrives first: replace Dana's active with false. Then the held request finally arrives, with the version it was built from.

scim -X PUT "$SCIM/Users/$DANA" -H "If-Match: $E1" --data-binary @dana.json | jq .

The answer is 412. Read Dana again: she should stay disabled, so you rebuild the change from the current intended state, and there is nothing to send.

Why it matters: this is "Order and timing". The stale copy still said active: true. Without the version check, a late request would have enabled a leaver again.

5. Create drift. In the portal, add Ben to Photo moderation, a manual edit outside your process. Then create [email protected] (Locum Three) in the portal, an account your provisioning service never heard of.

6. Reconcile. Compare intended with actual, and look for accounts no source has claimed.

ACTUAL=$(scim "$SCIM/Groups/$MOD" | jq -r '.members[].value' | while read id; do scim "$SCIM/Users/$id" | jq -r .userName; done | sort)
INTENDED=$(jq -r '.members[]' intended.json | sort)
comm -13 <(echo "$INTENDED") <(echo "$ACTUAL") | sed 's/^/extra member: /'
comm -23 <(echo "$INTENDED") <(echo "$ACTUAL") | sed 's/^/missing member: /'
scim -G "$SCIM/Users" --data-urlencode 'filter=active eq true and not (externalId pr)' | jq -r '.Resources[].userName' | sed 's/^/no externalId: /'

The report shows extra member: [email protected] and lists [email protected] among the unclaimed accounts. Decide each one. Ben's membership is access added outside the process: remove it, and find who added it by searching Audit for the group ID (tenant.groups.members by you). The locum account has no identity behind it, so it is an orphan to investigate, not to delete blindly.

Why it matters: this is "Reconciliation". It sees changes the provisioning service did not make, and each difference needs its own response.

7. A limit and a backlog. In User Management > Provisioning, set SCIM requests per minute to 10 and save. Fire 20 reads.

for i in $(seq 1 20); do scim -o /dev/null -D - "$SCIM/Users?count=1" | grep -i -E '^(HTTP|retry-after)' | tr -d '\r' | tr '\n' ' '; echo; done

After about ten, the answers are 429 with Retry-After. Audit records scim.users.search rejected with rate_limited and the limit policy.

8. Drain in priority order. Queue three changes for Dana and the group and send them in this order, waiting as long as each Retry-After says: Dana's deactivation (already done, so confirm it), the removal of Ben from Photo moderation, then a title update for Dana.

Why it matters: this is "Limits and backlogs". A backlog waits as told and sends leavers and removals first. A delayed title is an inconvenience; a delayed removal is access that should be gone.

Restore: in Provisioning, clear SCIM requests per minute and save. A burst of 20 reads now succeeds.

9. Let the HR Simulator do it. Start a Real-world edge cases run with stale_if_match and duplicate_username, and burst on. Read each event: 412 followed by a re-read, 409 reported as expected, 429 with the simulator honoring Retry-After, and any event marked unresolved reconciled before another attempt.

Break it

1. Send step 4's held PUT again, this time without If-Match: scim -X PUT "$SCIM/Users/$DANA" --data-binary @dana.json | jq .active. It succeeds, and Dana is active again. A stale request just enabled a leaver.

Restore: replace Dana's active with false again.

Check your work

Press Check my progress. The checks look for, in order:

• scim.user.create succeeded by lab-provisioning (step 1 or 3)

• scim.user.create rejected with command_conflict (step 3)

• scim.user.replace rejected with version_mismatch (step 4)

• tenant.groups.members succeeded (step 5)

• tenant.provisioning.update succeeded (step 7)

• scim.users.search rejected with rate_limited (step 7)

Cleanup

1. Make sure Ben is out of Photo moderation and the request limit is cleared.

2. Delete lab-tmp-dana, lab-tmp-jo and lab-tmp-locum3, and delete intended.json and dana.json.