PROCESS/INCIDENT-RESPONSE
sdk-v2/lab/process/INCIDENT-RESPONSE.md · commit 96368332 · GLI CSR §2.2: §2.11 incident management · ← package index · raw on GitHub ↗INCIDENT RESPONSE — RAIN RNG node (GLI CSR §2.11(h); UKGC RTS security 5.24–5.28; NJ 13:69O-1.5(d) "immediately cease… immediately notify")
1. Detection (what exists in the node)
/healthz503 with reason fields:store,chainHeadroom,clockSkew,attribution,kat: fail,selfTest.fingerprints: fail,outputInhibited.- Prometheus
/metrics:rain_rng_store_up,duplicate_commits_total(mutated retries — possible attack),reveals_replayed_total,round_latency_p95_ms,chain_remaining_min,errors_total. Alert thresholds in HOUSE-NODE §7. - Structured JSON logs to stderr (
lvl,msg,extra): KAT failure, fingerprint mismatch, persisted-commit-without-reveal at startup, attribution failure.
2. Classification and immediate action
| Class | Examples | Automatic behaviour | Human action |
|---|---|---|---|
| S1 — RNG integrity | KAT failure; artifact fingerprint mismatch (strict); evidence of a second reveal for one k; store rollback detected (order/bad_prev storms) |
Node inhibits RNG output (503 on terms/open/reveal) on KAT failure / strict fingerprint mismatch | Stop game intake at the operator layer; preserve WAL/DB and container image (§4); notify regulator per jurisdiction (NJ: immediately; UKGC: key event within 5 working days under LCCP 15.2.1 if it affects fairness; MGA: as an essential-component incident) and the lab; do not restart on the same image |
| S2 — availability | store down, node crash loop, LB failure | /healthz 503; clients hold abort() path; escrow protocol handles stalls |
Restore per HOSTING-HA-DR.md; no fairness impact (outcomes for committed k are fixed) |
| S3 — suspicious client behaviour | duplicate_commits_total rising; many 409s |
node refuses second reveals | Investigate the client/LB; block abusive sources |
| S4 — key/secret exposure | operator key or store exposure | none | Rotate operator key (HOUSE-NODE §7); for store exposure: end affected sessions (force rotation), deploy anchoring, disclose per KEY-MANAGEMENT.md impact column |
3. Player communication
Outcomes already revealed are unaffected by any later incident (each r_k is fixed by public commitments); publish the /verify links and, for S1, a notice describing scope, the time window, and how players can recompute their rounds (docs/RNG.md §5).
4. Evidence collection (ISO 27001 5.28)
Snapshot, read-only, with SHA-256: wal.jsonl / Postgres dump, container image digest, lab/FINGERPRINTS.json in use, /selftest output, logs for the window, /metrics scrape. Record who collected, when, and where stored.
5. Post-mortem
Within 10 working days: timeline, root cause, whether any round's outcome could have been influenced (answer with the Appendix A argument and the WAL: every revealed k has a persisted commit that precedes it), corrective actions, and — if code changed — the change-control record and re-certification decision (CHANGE-CONTROL.md §5).
← Back to the package index · Rendered 2026-09-14 09:34 UTC from the repository copy; the markdown in the zip / repo is the document of record.