felhom.eu/REPORT-chaos-night-2026-09-17.md

# REPORT — chaos night 2026-09-16/17

Full record: `documentation/audits/DRILL-chaos-night-2026-09-17.md`. Evidence (73+ files):
`documentation/audits/evidence-chaos-night-2026-09-17/`. Architecture read for the area:
`documentation/architecture/00-capability-map.md` (the journey and backup rows).

**Interventions: 1** (round 6, a local backup leg that could never fit; the off-site leg then
succeeded unaided). **Ready for a volunteer: still yes.** **Worst pair: restore + hard reset.**

## Claims in the prompt that turned out wrong — named first

1. „The automatic mail is waiting in the mailbox" — **TRUE**, checked: mail of 18:17:46Z; zero presses.
2. „The WG hook provisions by itself after an acknowledged delete" — **TRUE**, measured live for the
   first time: `pbsdr_auto_reissue`, 20:19Z.
3. „Restore one DB-backed app from off-site onto 9202" — **WRONG for this fixture.** The box is a
   rebuild; its restic repository is orphaned by design (restic: `wrong password or no key found`,
   exit 1; product: `orphaned:true, snapshots:0`). Nothing to restore from.
4. „System disk + one data disk" — **not what ran**: a third 64 G disk was added by me in Phase 0.
5. Round 7's drawn `update` — **not run**; the catalog's own gates were INCONCLUSIVE. `use` ran, logged.
6. „An internet cut tests hub unreachability" — **false on this network** (hub resolves to the LAN).
   Mine; fixed before round 9.
7. The schedule's clock column was nominal; the twelve rounds ended 00:17Z. Order/apps/accidents unchanged.

## 1. Confirmed baselines (read live at 21:49 CEST 2026-09-16)
felhom-controller `714d5bce0920` v0.245.0 (MinAgent 0.131.0) · felhom-agent `e98b857684f4` v0.131.0 ·
felhom.eu `d124c77e176d` hub v0.116.0, ISO 1.28.0 published · app-catalog `94bc5febaca2`.

## 2. Files created / modified
 89 files changed, 6252 insertions(+), 2 deletions(-). All under `documentation/` plus `STATUS.md` and this file. No product code in any repo.
app-catalog: **unchanged** (bump reverted before push; verified level with origin, 0/0).

## 3. Commits pushed to `main` (49 before this report's own commit, oldest first)
  - `9fae6df` CHAOS NIGHT phase 0: golden 0.245.0, a self-installing box, and R-546
  - `5f2ccec` CHAOS NIGHT: household seeded, escrow done, round 1 measured
  - `a1a57ea` CHAOS NIGHT: round 1 recorded, and three of my own conclusions corrected
  - `c3e1986` CHAOS NIGHT: household repaired, headroom checked, round 2 armed
  - `a046db7` CHAOS NIGHT: household verified, and a seventh error of mine found by control
  - `cc87efa` CHAOS NIGHT: household whole (11/12), injector fixed, round 2 running
  - `5b6e4b5` CHAOS NIGHT: round 2 under way, and the household loop made to survive accidents
  - `bca013e` CHAOS NIGHT: round 2 passed, and the OOM finding goes on the row that owns it
  - `7221367` CHAOS NIGHT round 3: the disk fills, and nothing is told about it
  - `fa1ddd9` CHAOS NIGHT: the internet-block accident now cleans up unconditionally
  - `ee3da86` CHAOS NIGHT round 3: interim reading, and a mislabel caught before it stood
  - `aca0172` CHAOS NIGHT: fences clean, firewall baseline taken, and a third mistimed label
  - `34d22a1` CHAOS NIGHT round 3: the disk fills for ten minutes and nobody is told
  - `ec84ead` CHAOS NIGHT round 4: the box repairs its own tunnel in 97 seconds
  - `3e66454` CHAOS NIGHT: round 3 written up, and the household loop's blind spot stated
  - `eb638d3` CHAOS NIGHT rounds 4-5: the box repairs its own tunnel, and survives docker dying
  - `d431852` CHAOS NIGHT round 6: what a whole-system backup costs the household
  - `36ae3b3` CHAOS NIGHT round 6: I stopped a LEG, and the box carried on by itself
  - `aaf0537` CHAOS NIGHT round 6: the off-site leg takes no local space - measured, not argued
  - `c3722e0` CHAOS NIGHT round 6: the local backup tier cannot fit, and the box says so properly
  - `a103b62` CHAOS NIGHT round 7: the block works, and "vzdump procs: 2" was my own command
  - `3abd25e` CHAOS NIGHT round 7: a ten-minute outage falls between two reports
  - `e61aac1` CHAOS NIGHT: two enumerated gaps become rows in the same session
  - `3129d4f` CHAOS NIGHT round 7: the cut is invisible at home, ten minutes long from away
  - `b917879` CHAOS NIGHT round 7 closed: reporting resumed on time, nothing lost
  - `e45fb5e` chaos night: draft alarm truth table for rounds 1-7
  - `418f3a2` chaos night round 8: the accident did not do what its name said
  - `889310e` chaos night round 9: what a lost hub report actually costs, measured
  - `70f1e01` chaos night: alarm truth table extended to rounds 1-9
  - `f973fd7` chaos night: the hub link repaired itself on the next cycle, and a late ghost task
  - `9f40dc3` chaos night round 10: a restore leaves no record, and four of my instruments failed
  - `51782a4` chaos night: alarm truth table extended to rounds 1-10
  - `3f844b7` chaos night: interventions ledger, built from the evidence not from memory
  - `73ac9d7` chaos night: pre-round-11 steadiness check, and a seventh instrument slip
  - `70bffb1` chaos night round 11: the drive pulled for 20 minutes, and eight true alarms
  - `d91822c` chaos night: ep0 baseline and Phase 2 readiness, taken without touching the box
  - `d335033` chaos night round 12: the closing control round, and the truth table complete
  - `8e4365a` chaos night: the household loop summary for the whole night
  - `be99cf7` chaos night: R-550 corrected - I guessed four endpoints and all four were wrong
  - `7c8a299` chaos night Phase 2: the off-site restore cannot be done, and why - two instruments agree
  - `62f6b7b` chaos night: a defect NOT filed, a worthless probe named, and the catalog verified clean
  - `9b44c44` chaos night: teardown baseline, and the box's own logs copied off before anything stops
  - `0f65c81` chaos night teardown: machine destroyed, host clean, 9202 untouched, ep0 unchanged
  - `6c450bc` chaos night: the alarms were DELIVERED, and the first delete was correctly refused
  - `3d5c42c` chaos night: the report headline, and the capability map
  - `52c54a0` chaos night: Phase 2 and the interventions section written up
  - `5337c3b` chaos night: the prompt's claims that turned out wrong, named
  - `d6a0e7b` chaos night: the before-picture of the records that must survive the delete
  - `69c08b1` chaos night teardown complete: hub host record deleted, connect mail quoted, ep0 unchanged

## 4–5. Tests
**N/A — no product code was written** (the brief forbade it). No test count moved.
`unproven.py --summary`: 35 of 55 not walked — **no number moved**.

## 6. Deployed versions (the box under test)
golden **0.245.0** (baked and vouched in Phase 0, registry answered 200) · controller 0.245.0 ·
agent 0.131.0 · hub 0.116.0 · installed from ISO 1.28.0. No deploy to any standing box.

## 7. NOT live-validated
- Per-app off-site **restore** (orphaned repo by design on a rebuild box).
- Whole-guest off-site copy **restorability** — listed intact on ep0, not verified (a verify writes state).
- The **event-drop** path while the hub is unreachable — no event coincided with any of three outages.
- What a browser renders client-side (endpoint-level validation only; no browser on DooPlex).

## 8. Evidence copied off before each revert
Yes, per round (R-320). The box's own household log, disk-guard log, loop script and unit files were
copied off **before** the units were stopped and before the machine was destroyed. Nothing was lost.

## 9. Teardown — three layers
- **Machine:** VM 336 destroyed with all three disks; `/mnt/hdd_1/images/336` gone.
- **Host:** `nvme-scratch` 6.78 % → 1.61 %; `local-lvm` unchanged 44.75 %; guests 9201/9202 running;
  household loop and disk guard stopped and disabled; firewall back to baseline (0 physdev rules).
- **Hub:** host record `tester-1-022354` **DELETED** through the acknowledged flow at 07:25:13Z
  (first attempt correctly refused 409 while the host was still live). `drill-r50`, both demo hosts
  and the `tester-1` customer still 200. Connect mail quoted in `teardown-hub.txt` (token redacted).
- **ep0:** identical across three readings — 6 snapshots, 16 G. Nothing removed.
- **Scratch 9202:** nothing was ever placed on it; shown untouched.

## Observations
- A restore interrupted by the machine stopping leaves no record the household can see. **FILED: R-550**
- The staleness alarm's budget is two report cycles; one failed push spends it (29 m 59 s measured). **FILED: R-549**
- A transient full disk between daily sweeps is never mentioned. **FILED: R-547**
- The whole-guest local tier cannot fit on a small-system-disk box and retries forever. **FILED: R-548**
- The first-hour guide asks for the recovery code ~17 min before the box can take it. **FILED: R-546**
- An OOM was detected and named on this box. **NOT-A-FINDING: added as tonight's line on the existing row that owns it (R-528), not a new defect.**
- The off-site orphan warning is absent from the static HTML of the remote-backup page. **NOT-A-FINDING: the page renders it client-side from the status endpoint (a dedicated orphan card exists).**
- Eleven faults in my own instruments (mistimed readings, a wrong hub-reachability model, guessed endpoints, a guard that could never pass). **NOT-A-FINDING: harness errors, not product defects; each is recorded with its fix in the evidence.**

**CHANGELOG not updated:** this repo's changelog is per product area (hub/scripts/website) and no
product area changed tonight — the drill record, register and status note are the record.