Files

T

gates / gates (push) Successful in 21s

Details

chaos night: the morning note and the report

STATUS.md gets the morning note in the rules' order - decisions (none under the
unattended rule), what was exercised, what broke (nothing in the product; three
fixes worth making, all filed), rows (five opened, none closed), what could not
be tested, cleanup, and what needs the operator with the cost of doing nothing.

REPORT-chaos-night-2026-09-17.md follows template section 15 and names the
prompt's wrong claims first. It is a separate file because REPORT.md holds the
earlier session's write-up and this repo's rule forbids clobbering it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

2026-09-17 09:28:22 +02:00

9.3 KiB

Raw Blame History

REPORT — chaos night 2026-09-16/17

Full record: documentation/audits/DRILL-chaos-night-2026-09-17.md. Evidence (73+ files): documentation/audits/evidence-chaos-night-2026-09-17/. Architecture read for the area: documentation/architecture/00-capability-map.md (the journey and backup rows).

Interventions: 1 (round 6, a local backup leg that could never fit; the off-site leg then succeeded unaided). Ready for a volunteer: still yes. Worst pair: restore + hard reset.

Claims in the prompt that turned out wrong — named first

„The automatic mail is waiting in the mailbox" — TRUE, checked: mail of 18:17:46Z; zero presses.
„The WG hook provisions by itself after an acknowledged delete" — TRUE, measured live for the first time: pbsdr_auto_reissue, 20:19Z.
„Restore one DB-backed app from off-site onto 9202" — WRONG for this fixture. The box is a rebuild; its restic repository is orphaned by design (restic: wrong password or no key found, exit 1; product: orphaned:true, snapshots:0). Nothing to restore from.
„System disk + one data disk" — not what ran: a third 64 G disk was added by me in Phase 0.
Round 7's drawn update — not run; the catalog's own gates were INCONCLUSIVE. use ran, logged.
„An internet cut tests hub unreachability" — false on this network (hub resolves to the LAN). Mine; fixed before round 9.
The schedule's clock column was nominal; the twelve rounds ended 00:17Z. Order/apps/accidents unchanged.

1. Confirmed baselines (read live at 21:49 CEST 2026-09-16)

felhom-controller 714d5bce0920 v0.245.0 (MinAgent 0.131.0) · felhom-agent e98b857684f4 v0.131.0 · felhom.eu d124c77e176d hub v0.116.0, ISO 1.28.0 published · app-catalog 94bc5febaca2.

2. Files created / modified

89 files changed, 6252 insertions(+), 2 deletions(-). All under documentation/ plus STATUS.md and this file. No product code in any repo. app-catalog: unchanged (bump reverted before push; verified level with origin, 0/0).

3. Commits pushed to `main` (49 before this report's own commit, oldest first)

9fae6df CHAOS NIGHT phase 0: golden 0.245.0, a self-installing box, and R-546
5f2ccec CHAOS NIGHT: household seeded, escrow done, round 1 measured
a1a57ea CHAOS NIGHT: round 1 recorded, and three of my own conclusions corrected
c3e1986 CHAOS NIGHT: household repaired, headroom checked, round 2 armed
a046db7 CHAOS NIGHT: household verified, and a seventh error of mine found by control
cc87efa CHAOS NIGHT: household whole (11/12), injector fixed, round 2 running
5b6e4b5 CHAOS NIGHT: round 2 under way, and the household loop made to survive accidents
bca013e CHAOS NIGHT: round 2 passed, and the OOM finding goes on the row that owns it
7221367 CHAOS NIGHT round 3: the disk fills, and nothing is told about it
fa1ddd9 CHAOS NIGHT: the internet-block accident now cleans up unconditionally
ee3da86 CHAOS NIGHT round 3: interim reading, and a mislabel caught before it stood
aca0172 CHAOS NIGHT: fences clean, firewall baseline taken, and a third mistimed label
34d22a1 CHAOS NIGHT round 3: the disk fills for ten minutes and nobody is told
ec84ead CHAOS NIGHT round 4: the box repairs its own tunnel in 97 seconds
3e66454 CHAOS NIGHT: round 3 written up, and the household loop's blind spot stated
eb638d3 CHAOS NIGHT rounds 4-5: the box repairs its own tunnel, and survives docker dying
d431852 CHAOS NIGHT round 6: what a whole-system backup costs the household
36ae3b3 CHAOS NIGHT round 6: I stopped a LEG, and the box carried on by itself
aaf0537 CHAOS NIGHT round 6: the off-site leg takes no local space - measured, not argued
c3722e0 CHAOS NIGHT round 6: the local backup tier cannot fit, and the box says so properly
a103b62 CHAOS NIGHT round 7: the block works, and "vzdump procs: 2" was my own command
3abd25e CHAOS NIGHT round 7: a ten-minute outage falls between two reports
e61aac1 CHAOS NIGHT: two enumerated gaps become rows in the same session
3129d4f CHAOS NIGHT round 7: the cut is invisible at home, ten minutes long from away
b917879 CHAOS NIGHT round 7 closed: reporting resumed on time, nothing lost
e45fb5e chaos night: draft alarm truth table for rounds 1-7
418f3a2 chaos night round 8: the accident did not do what its name said
889310e chaos night round 9: what a lost hub report actually costs, measured
70f1e01 chaos night: alarm truth table extended to rounds 1-9
f973fd7 chaos night: the hub link repaired itself on the next cycle, and a late ghost task
9f40dc3 chaos night round 10: a restore leaves no record, and four of my instruments failed
51782a4 chaos night: alarm truth table extended to rounds 1-10
3f844b7 chaos night: interventions ledger, built from the evidence not from memory
73ac9d7 chaos night: pre-round-11 steadiness check, and a seventh instrument slip
70bffb1 chaos night round 11: the drive pulled for 20 minutes, and eight true alarms
d91822c chaos night: ep0 baseline and Phase 2 readiness, taken without touching the box
d335033 chaos night round 12: the closing control round, and the truth table complete
8e4365a chaos night: the household loop summary for the whole night
be99cf7 chaos night: R-550 corrected - I guessed four endpoints and all four were wrong
7c8a299 chaos night Phase 2: the off-site restore cannot be done, and why - two instruments agree
62f6b7b chaos night: a defect NOT filed, a worthless probe named, and the catalog verified clean
9b44c44 chaos night: teardown baseline, and the box's own logs copied off before anything stops
0f65c81 chaos night teardown: machine destroyed, host clean, 9202 untouched, ep0 unchanged
6c450bc chaos night: the alarms were DELIVERED, and the first delete was correctly refused
3d5c42c chaos night: the report headline, and the capability map
52c54a0 chaos night: Phase 2 and the interventions section written up
5337c3b chaos night: the prompt's claims that turned out wrong, named
d6a0e7b chaos night: the before-picture of the records that must survive the delete
69c08b1 chaos night teardown complete: hub host record deleted, connect mail quoted, ep0 unchanged

4–5. Tests

N/A — no product code was written (the brief forbade it). No test count moved. unproven.py --summary: 35 of 55 not walked — no number moved.

6. Deployed versions (the box under test)

golden 0.245.0 (baked and vouched in Phase 0, registry answered 200) · controller 0.245.0 · agent 0.131.0 · hub 0.116.0 · installed from ISO 1.28.0. No deploy to any standing box.

7. NOT live-validated

Per-app off-site restore (orphaned repo by design on a rebuild box).
Whole-guest off-site copy restorability — listed intact on ep0, not verified (a verify writes state).
The event-drop path while the hub is unreachable — no event coincided with any of three outages.
What a browser renders client-side (endpoint-level validation only; no browser on DooPlex).

8. Evidence copied off before each revert

Yes, per round (R-320). The box's own household log, disk-guard log, loop script and unit files were copied off before the units were stopped and before the machine was destroyed. Nothing was lost.

9. Teardown — three layers

Machine: VM 336 destroyed with all three disks; /mnt/hdd_1/images/336 gone.
Host: nvme-scratch 6.78 % → 1.61 %; local-lvm unchanged 44.75 %; guests 9201/9202 running; household loop and disk guard stopped and disabled; firewall back to baseline (0 physdev rules).
Hub: host record tester-1-022354 DELETED through the acknowledged flow at 07:25:13Z (first attempt correctly refused 409 while the host was still live). drill-r50, both demo hosts and the tester-1 customer still 200. Connect mail quoted in teardown-hub.txt (token redacted).
ep0: identical across three readings — 6 snapshots, 16 G. Nothing removed.
Scratch 9202: nothing was ever placed on it; shown untouched.

Observations

A restore interrupted by the machine stopping leaves no record the household can see. FILED: R-550
The staleness alarm's budget is two report cycles; one failed push spends it (29 m 59 s measured). FILED: R-549
A transient full disk between daily sweeps is never mentioned. FILED: R-547
The whole-guest local tier cannot fit on a small-system-disk box and retries forever. FILED: R-548
The first-hour guide asks for the recovery code ~17 min before the box can take it. FILED: R-546
An OOM was detected and named on this box. NOT-A-FINDING: added as tonight's line on the existing row that owns it (R-528), not a new defect.
The off-site orphan warning is absent from the static HTML of the remote-backup page. NOT-A-FINDING: the page renders it client-side from the status endpoint (a dedicated orphan card exists).
Eleven faults in my own instruments (mistimed readings, a wrong hub-reachability model, guessed endpoints, a guard that could never pass). NOT-A-FINDING: harness errors, not product defects; each is recorded with its fix in the evidence.

CHANGELOG not updated: this repo's changelog is per product area (hub/scripts/website) and no product area changed tonight — the drill record, register and status note are the record.

9.3 KiB Raw Blame History Unescape Escape