Lab: the key hierarchy under compromise
Build a KEK and DEK tree, take a shell on the application server, then rotate the KEK and watch it change nothing.
The key hierarchy lesson claimed that an HSM protects against key theft rather than key misuse, and that the audit log is therefore your detection surface rather than a byproduct. This lab puts an attacker inside the boundary that claim is about.
Each step appends to the same script, so every run reprints the sections before it. The expected output shown under each step is the new part; scroll to the bottom of your terminal to find it.
- 1
Build the hierarchy. The HSM object holds the KEK and never returns it, exactly as real non-exportability works; the application server holds ciphertext and wrapped DEKs.
bashmkdir -p /tmp/hsmlab && cd /tmp/hsmlab cat > hsm.py <<'EOF' import os xor = lambda a, b: bytes(x ^ y for x, y in zip(a, b * (len(a)//len(b) + 1))) class HSM: """Holds the KEK. Performs operations, never hands the key back.""" def __init__(self): self._kek = os.urandom(32) self.log = [] def unwrap(self, wrapped, caller): self.log.append({"caller": caller, "op": "unwrap"}) return xor(wrapped, self._kek) def rotate(self): old, self._kek = self._kek, os.urandom(32) return old hsm = HSM() STORE = {} for name, value in [("db/password", b"prod-db-pw-9f2a"), ("stripe/key", b"sk-live-9f2a")]: dek = os.urandom(32) STORE[name] = {"ciphertext": xor(value, dek), "wrapped_dek": xor(dek, hsm._kek)} def read_secret(name, caller): e = STORE[name] return xor(e["ciphertext"], hsm.unwrap(e["wrapped_dek"], caller)) print("normal operation:", read_secret("db/password", "app-server").decode()) EOF python3 hsm.py - 2
Now put an attacker on the application server. They cannot take the KEK, because it is not there to take.
bashcat >> hsm.py <<'EOF' print("\n--- attacker has a shell on the app server ---") print("steal the KEK? ", "no, it never leaves the HSM") print("read a secret? ", read_secret("stripe/key", "app-server(attacker)").decode()) EOF python3 hsm.py - 3
The key is safe and the secret is not.
text--- attacker has a shell on the app server --- steal the KEK? no, it never leaves the HSM read a secret? sk-live-9f2a - 4
Do the thing everyone reaches for in an incident: rotate the KEK and re-wrap every DEK.
bashcat >> hsm.py <<'EOF' print("\n--- incident response: rotate the KEK ---") old = hsm.rotate() for e in STORE.values(): dek = xor(e["wrapped_dek"], old) e["wrapped_dek"] = xor(dek, hsm._kek) print("rotated, all DEKs re-wrapped.") print("attacker still on the box:", read_secret("stripe/key", "app-server(attacker)").decode()) EOF python3 hsm.py - 5
Rotation completed successfully and achieved nothing, because the attacker was never using the key material. They were using the ability to ask.
text--- incident response: rotate the KEK --- rotated, all DEKs re-wrapped. attacker still on the box: sk-live-9f2a - 6
Look at what does tell you something.
bashcat >> hsm.py <<'EOF' print("\n--- the HSM audit log ---") for entry in hsm.log: print(" ", entry) EOF python3 hsm.py - 7
Clean up.
bashcd /tmp && rm -rf hsmlab
Three conclusions, and the second is the one that changes how you run an incident.
- Non-exportability worked exactly as advertised. The attacker could not take the key with them, so their access ended when their access ended, and nothing was compromised permanently.
- Rotation was the wrong response. It is the reflex for key compromise, and this was credential compromise. The effective action is revoking whatever let the attacker call the HSM, which in production is the token or certificate the application authenticates with.
- The log is the only record of what actually happened. Every unwrap is there, attributed to a caller. Without it you would know an attacker had access and nothing about what they did with it.
This is why the credential that calls the HSM deserves the same care as key material. The key file is protected by hardware; the token authorising calls to it usually sits in an environment variable.
In an incident, ask what the attacker was actually using. If they held key material, rotate. If they held the ability to invoke, revoke the invoker and read the log to scope the damage. Rotating in the second case burns hours and changes nothing.
Check your understanding
3 questionsWhy did rotating the KEK not stop the attacker?
What did non-exportability actually achieve here?
What is the practical consequence for the credential the application uses to call the HSM?