Lab: cause an outage by rotating correctly

The rotation lesson gave you a sequence and said that skipping the verification step is how the last consumer finds out at 3am. Here is the same rotation run both ways, with the consumers visible.

1. 1

   Model a secret with versions and three services pointing at one of them.

   bash Copy

   cat > /tmp/rot.py <<'EOF'
   versions  = {1: {"value": "old-pw", "enabled": True}}
   consumers = {"api": 1, "worker": 1, "cron": 1}
   
   def read(who):
       v = versions[consumers[who]]
       return v["value"] if v["enabled"] else "ERROR 410 Gone"
   
   def show(label):
       print(f"  {label:32} " + ", ".join(f"{k}={read(k)}" for k in consumers))
   
   show("start")
   versions[2] = {"value": "new-pw", "enabled": True}
   show("new version written")
   
   print("\nWRONG ORDER: disable the old version first")
   versions[1]["enabled"] = False
   show("old disabled")
   
   versions[1]["enabled"] = True          # undo for the second run
   print("\nRIGHT ORDER: migrate, verify, then disable")
   for c in consumers:
       consumers[c] = 2
   show("consumers migrated")
   versions[1]["enabled"] = False
   show("old disabled")
   EOF
   python3 /tmp/rot.py

2. 2

   Same four operations. One order is a rotation and the other is an incident.

   text Copy

     start                            api=old-pw, worker=old-pw, cron=old-pw
     new version written              api=old-pw, worker=old-pw, cron=old-pw
   
   WRONG ORDER: disable the old version first
     old disabled                     api=ERROR 410 Gone, worker=ERROR 410 Gone, cron=ERROR 410 Gone
   
   RIGHT ORDER: migrate, verify, then disable
     consumers migrated               api=new-pw, worker=new-pw, cron=new-pw
     old disabled                     api=new-pw, worker=new-pw, cron=new-pw

3. 3

   Clean up.

   bash Copy

   rm /tmp/rot.py

Writing the new version broke nothing, which is the part that makes the wrong order tempting. Versions coexist deliberately, so the new value is available immediately and the old one keeps working, and nothing signals that a step has been missed.

• The failure is simultaneous rather than gradual. Every consumer on the old version stops at the same instant, which is why this shows up as a total outage rather than a degraded service.
• It is also silent until then. Disabling a version that nothing reads and disabling one that everything reads look identical at the moment you run the command.
• The verification step is the whole safeguard: read the audit log for the old version and confirm nothing has touched it since you migrated.