felhom.eu/documentation/audits/DRILL-chaos-night-2026-09-17.md

# DRILL — CHAOS NIGHT: random actions on random apps while random things go wrong (2026-09-16/17)

**Interventions: 1.** One, at 21:59:45Z in round 6: I killed the **local leg** of a whole-guest
backup that could never have fit (a ~29 GB source into a 14 GB root filesystem, falling at ~16 MB/s).
The **off-site leg then ran by itself from the same snapshot and succeeded**, so the data still left
the house. Both pre-declared presses went **unused**: the automatic self-bind mail was already
waiting, and the acknowledged-delete path re-issued the PBS credentials on its own. Phase 0's
seeding repairs are listed separately in `evidence-chaos-night-2026-09-17/interventions.txt` — that
damage was mine, not the product's, and every repair went through the product's own endpoints.

**Ready for a volunteer: still yes.** Across twelve rounds — a power cut mid-restore, a hard reset
four seconds into another, a full disk, a killed tunnel, a restarted Docker, three severed networks
and the data drive pulled out of a running machine for twenty minutes — **nothing cost a byte of
customer data, and the box healed itself every single time with no human involved.** Seventeen
alarms fired, **all seventeen were true, none were missing**, and the mailbox proves every one was
**delivered** rather than merely stored. The honest qualifications: two P2 legibility gaps are filed
(an interrupted restore leaves no record; one failed report spends the whole 30-minute staleness
budget, measured at 29 m 59 s), and one thing this night could **not** test — per-app off-site
restore, because this box is a rebuild whose restic repository is orphaned **by design**, which the
product surfaced honestly within seconds.

**The accident-plus-action pair that hurt most: `restore` + hard reset (round 10).** Not because the
box suffered — it was back with 26 of 26 containers in **150 s**, boot reconciliation naming the app
it recovered, every front door serving. It hurt most because it is the **only** pair of the night
where the household is left not knowing what happened: they pressed restore, were told it had
started, the machine went dark four seconds later, and afterwards **nothing anywhere tells them
whether it finished.** The status surface exists and answers with the zero value; the record is
in-memory only and does not survive the machine stopping.

> **Baselines, verified live against Gitea at 21:49 CEST 2026-09-16 (not copied from the brief):**
> felhom-controller `714d5bce0920` v0.245.0 (MinAgent 0.131.0) · felhom-agent `e98b857684f4` v0.131.0 ·
> felhom.eu `d124c77e176d` hub v0.116.0, ISO **1.28.0 published** · app-catalog `94bc5febaca2`.
> All four trees clean and in sync. Highest register row **R-545**, 212 open. Golden waiver valid to
> 2026-09-27. Customer **`tester-1`** (`enkicsifelhom.hu`, `tester1@felhom.eu`, no host).
> Venue: `demo-hp` (Tier 0), a fresh nested VM, disk on the NVMe at its root. Evidence:
> `evidence-chaos-night-2026-09-17/`.

## The schedule — drawn ONCE, before round 1, and written here first

The point of this section's position in the document is that the night could not be chosen after the
fact. `chaos_schedule.py` is committed beside the evidence; re-running it reproduces this table.

- **seed:** `20260917` (the date)
- **script sha256:** `4b98afe65d042df7e7dc417553b33565cfbb4afd451c68456a7cabec0858d2a1`
- **generator:** `evidence-chaos-night-2026-09-17/chaos_schedule.py`, stdlib `random` seeded with the seed

| # | time | X — the action | Y — the app | Z — the accident |
|---|---|---|---|---|
| 1 | 23:30 | offsite-run | adventurelog | nothing |
| 2 | 23:55 | restore | gokapi | power cut |
| 3 | 00:20 | use | bookstack | disk 95% full |
| 4 | 00:45 | offsite-run | mealie | tunnel down 10min |
| 5 | 01:10 | use | privatebin | docker restarted |
| 6 | 01:35 | backup-system | adventurelog | nothing |
| 7 | 02:00 | update | nextcloud | internet gone 10min |
| 8 | 02:25 | backup-app | nextcloud | internet gone 10min |
| 9 | 02:50 | use | uptime-kuma | internet gone 10min |
| 10 | 03:15 | restore | uptime-kuma | hard reset |
| 11 | 03:40 | use | paperless-ngx | drive pulled 20min |
| 12 | 04:05 | use | paperless-ngx | nothing |

**Re-draw log** — a silent re-draw is a schedule chosen by the person running it, so every one is here:

- r02  X=reinstall re-drawn (nothing has been removed yet)
- r06  Z=disk 95% full re-drawn (constraint 4: at most once)
- r07  Z=nothing re-drawn (constraint 6: never two in a row after r2)
- r08  X=reinstall re-drawn (nothing has been removed yet)

**What the draw happened to give, said plainly before the night judges it:** no `remove` round was
ever drawn, so `reinstall` had nothing to reinstall and was re-drawn twice (rounds 2 and 8). Three
`internet gone` rounds land consecutively (7, 8, 9) — that is the seed's doing, and it makes rounds
7–9 a de-facto endurance test of the same accident against three different actions rather than three
independent samples. `controller killed`, `drive pulled 90s`, `memory pressure`, `agent restarted`
and `hub unreachable` were never drawn at all; **this night does not test them**, and the morning
verdict must not claim it did.

## Phase 0 — the golden, the box, the household

**0.1 Golden 0.245.0, baked and published.** Launched 19:52:47Z as a transient unit in the drill VM,
finished 19:58:15Z. Markers: `overlay2`=1, `including mount point`=2, `upload OK (HTTP 201)`=1,
FATAL=0, publish-skipped=0. `GOLDEN_SHA256=7a08aa1ad0bdd622247e1901e422ed2f72df22ef531a144135a44b66fc455626`.
**The teardown was gated on the REGISTRY answering 200**, not on an exit code — and that mattered:
the wrapper exited **144** while every measured outcome was good. Vouched in the hub and read back
from the page (`golden currently vouched: 0.245.0`). The three bake failures of 2026-09-16 (`scp -P`,
`chmod 0700`, `GITEA_USER=admin`) were each guarded and none recurred.

*Decision, recorded because silence reads as agreement:* the global controller floor was left at
0.244.0. The new box installs golden 0.245.0, which already carries controller 0.245.0, so no floor
was needed to deliver anything tonight; raising it would have pushed an update onto demo-felhom, a
box not in this drill.

**0.2 The box.** VM 336 on demo-hp: 8 GiB, 4 cores, 32 G system + 100 G data disk on the NVMe at its
root, booted from the **published** ISO 1.28.0. Boot order set in its own `qm set` (combining it
silently yields `order=net0;ide2`). Install completion was judged **from the disk** — blocks used
grew 3233 → 6942 MiB then held across three checks — because „Automatically reboot" is ticked and a
finished install looks exactly like a stuck one on screen. The summary page was read before pressing
Install, and the line that made it safe was **„Disk(s): /dev/sda"** — the 32 G system disk alone.

**The walk, as a volunteer, cost ZERO operator presses.** The box registered itself as an unclaimed
appliance and polled, visibly, until bound. The bind link came from the **waiting mail** (minted
18:17:46Z by yesterday's acknowledged host delete), the pairing code off the box's own console
(`4SY-4TX`), the „Tulajdonosi jelmondat" from the hub's customer record:
  POST /bind/<token> -> 200, „Sikeres összekötés."
Then day-0 ran on its own and the hub recorded, without anyone pressing anything:
  `appliance_bound` (customer_selfbind) · `appliance_credential_delivered` · `claim_reissued_reenroll`
  · `offsite_reissued` · **`pbsdr_auto_reissue` — „Previous key destroyed (acknowledged deletion) —
  credentials re-issued automatically."**

**That last event is a first.** The brief named „the WG hook provisions by itself after an
acknowledged delete" as a claim never measured live. It ran tonight, unprompted. **Both pre-declared
presses (O1 self-bind, O2 re-issue) were therefore unnecessary.**

The dashboard was claimed with the mailed code and **proven by logging in with the new password** —
a claim page that re-renders looks identical to success from the status code alone. The 100 GB data
drive was initialised through the wizard's own endpoint (`POST /api/storage/init`, polled to
`phase: done`), and `df` shows it mounted at `/mnt/felhom-drives/hdd_1` with 93 G free.

**The box landed on tonight's golden with no hand upgrade:** controller **0.245.0** (healthy), agent
**0.131.0**, host `tester-1-022354` ONLINE. And the **R-543 escrow reminder bar shipped hours earlier
was live on it**, on a box nobody had touched.

**0.3 The household — and the first real trouble.** Twelve deploys were fired; **ten were accepted,
two refused** for memory with both numbers quoted. Then nine of the ten failed: the guest's disks are
thin-provisioned over an ~11.8 GB pool carved from a 32 GB system disk, ten simultaneous image pulls
filled it, and the hub recorded `storage_fill_critical` (100 %) plus **nine `app_deploy_failed`
warnings**, one per app, each naming the failing pull. Only PrivateBin installed.

**The product behaved; the harness did not.** R-536's failure event — shipped that same morning so an
interrupted install is not silence — fired for all nine within two minutes. The memory guard refused
rather than over-committing. The per-stack record stayed honest (`deployed: false`). The two faults
were mine: firing twelve deploys in two seconds is not household behaviour, and a 32 GB system disk
was copied from an earlier drill without checking what that drill had installed.

**0.4 The escrow ceremony could NOT be completed — and this one is about tonight's own release.**
See „Finding: the recovery-code step cannot be done when the guide says to do it" below.


## Finding: the recovery-code step cannot be done when the guide says to do it (R-546)

The box was at exactly the point of the guide this release added hours earlier — installed, bound
with no press, claimed, drive initialised, **no apps yet** — and the escrow reminder bar was on every
page telling the household to create their recovery code. It could not be done.

    POST /api/escrow/start   -> 200   {"job_id":"escrow-1789590499667361664","phase":"running"}
    GET  /api/escrow/status  -> claimable:false —
         detail: "exit 2: … selftest=escrow-create requires -storage <pbs-storage-id> (or escrow.pbs_storage…"
    POST /api/escrow/claim   -> **409** „A folyamat jelenlegi állapotában a kód nem kérhető le."

**Both sides agreed on the cause.** The hub's own Backup & DR panel read „host enrolled **done** · WG
tunnel peer registered **done** · descriptor provisioned (namespace tester-1, token felhom@pbs!tester-1)
**waiting** · ceremony possible once the descriptor is applied on the box". The box had no PBS storage
(`pvesm status`: `local`, `local-lvm` only) and no `escrow` section in `agent.json` at all.

**It self-heals, and that was measured rather than assumed.** The box was left alone and polled:

    20:30:12Z … 20:34:15Z   pbs_storage=none         escrow.pbs_storage_id=none
    **20:35:16Z              pbs_storage=felhom-pbs   escrow.pbs_storage_id=felhom-pbs**

~17 minutes after the bind. The retried ceremony passed every preflight item, the claim returned
**200** (83-character code, 129.2 bits of entropy, revealed once), and `escrow_state` became
**escrowed**. The bar then vanished from all four pages checked — the R-543 fix working through its
whole lifecycle on a box nobody had set up for the test.

**So the defect is timing and wording, not mechanism.** For ~17 minutes a volunteer following
tonight's guide meets a stderr fragment about a `-storage` flag, while every page urges them on.
Filed **R-546** (P2). No product code was changed — this is a validation run.

## Phase 1 — the rounds

Rounds run at ~25-minute spacing. The schedule above is fixed; only the wall-clock start moved,
because Phase 0 ran long (the storage wizard submits by JavaScript and the endpoint was worked out
rather than guessed). Round 1 began 23:07 CEST.

### Round 1 — `offsite-run` / adventurelog / accident: **nothing** (control round)

| the five things | |
|---|---|
| what the customer saw | „A távoli mentés elindult — az állapot itt frissül." and, at the end, „A távoli mentési tároló elárvult: a benne lévő mentések egy korábbi, már nem elérhető kulccsal készültek (újratelepítés)." |
| what the box did by itself | walked all twelve apps — stop, dump each volume with real byte counts, restart — captured eleven, could not capture the one that was crash-looping, and finished |
| time to steady | **1m45s** (`last_duration`), `last_run` 21:09:00Z, `progress.active` false, `last_error` empty. A control round: the box never left steady |
| alarm fired / true? | **three, all true** — `app_start_failed` named Nextcloud · `backup_run_failures` „1 of 12 apps failed to back up in this nightly run: nextcloud" · `offbox_repo_orphaned`, matching the status endpoint's own `"orphaned": true` |
| should have fired, did not | **none** |

**Household loop in the window:** 3 lines marked FAILED, **all three mine** — the loop counted the
dashboard's 301 redirect as a failure while accepting the same 301 for app reads. Fixed at 21:11:25Z
and marked in the log; only lines after that marker are scored.

**What round 1 actually establishes.** The off-site tier is armed (escrowed) and the run works
end-to-end, but on THIS box — a rebuild for an existing customer — the remote repository was written
under a key the box no longer holds, so **no snapshot was written**. That is the documented rebuild
behaviour, surfaced honestly with the route out named in the message rather than reported as success.

**Two things that looked like defects in this round and are not**, both established with controls
rather than inference — five front doors answering 404 (traefik has no route to an unhealthy
container; identical byte-for-byte to a no-such-host control) and a crash-looping Nextcloud (image
layers corrupted while my thin pool stood at 100 %; „invalid ELF header", repaired by a re-pull).
Detail in `evidence-chaos-night-2026-09-17/round-1-notes.txt`.


### Phase 0 postscript — repairing my own damage, and three conclusions I had to retract

Nine of the first ten deploys failed because I fired twelve at once onto a thin pool carved from a
32 GB disk, and the pool hit 100 %. Repairing that took the rest of Phase 0 and produced **two
distinct faults of mine, with different cures**, which only separating them made fixable:

| fault | symptom | cure |
|---|---|---|
| image layers written while the pool was full | `php: … libxml2.so.2: **invalid ELF header**`, exit 127 crash loop | drop the image, let compose pull it again |
| my re-seed generated **fresh database passwords** over volumes already initialised with the first set | Postgres `auth_failed`, MariaDB „Access denied for user … (using password: YES)" | remove the app **with its data**, deploy once with one consistent secret set |

A fresh image did not fix gokapi and a fresh database did — that is the evidence the two faults are
different things rather than one.

**Three conclusions I wrote and then had to retract, each corrected where it stood:**
1. „the five 404s were my mistimed sweep" — wrong for four of them. A **negative control** (a
   no-such-host request) returned the identical 404 of 19 bytes, and a positive control returned
   200/1200 bytes: traefik simply has **no route to an unhealthy container**.
2. „nextcloud is repaired" — wrong. The re-pull fixed the crash, and the app still could not reach its
   database. The container reported **`healthy`** throughout, because the image's own healthcheck asks
   whether Apache answers, not whether the application works.
3. „all the broken apps are corrupt layers" — wrong. bookstack logged a clean startup, gokapi logged
   nothing at all, immich showed a Postgres auth failure.

I also nearly filed a defect against the drive gate, which was working and logging at DEBUG while I
read a settings snapshot inside its 30-second tick. **An absent log line is not evidence.**

None of this is a product defect and none of it is filed as one. What the product did throughout was
correct and legible: it refused to route to unhealthy containers, `app_start_failed` named the app
that was down, `backup_run_failures` said „1 of 12 … nextcloud", the memory guard refused the
eleventh and twelfth installs with both numbers quoted, and `storage_fill_critical` fired at 100 %.

### Round 2 — `restore` gokapi / accident: **power cut**, 20 s into the restore

| the five things | |
|---|---|
| what the customer saw | „Visszaállítás elindult — az állapot itt frissül." then the box went dark mid-restore; on return the dashboard and every app were back |
| what the box did by itself | everything — containers 0 → **25 at t+131s** → **26 at t+148s**, nothing stuck, no shell used. **gokapi, the app being restored when the plug came out, returned `Up 30 seconds (healthy)`** |
| time to steady | **148 s**, measured against the container count this round took itself before the accident |
| alarm fired / true? | `controller_started` (info) — true and correct. **No false alarm.** |
| should have fired, did not | **none** — per the ladder a 60-second outage yields no `node_stale` (30 min threshold) and no `app_start_failed` (90 s boot grace), and neither appeared |

**Household loop: NOT COLLECTED.** The loop was a transient unit on the VM and died with the power
cut — the first accident that could have produced household failures instead produced no lines at
all. Zero lines is not zero failures, so it is recorded as not collected, and the loop is now a
persistent systemd unit that returns with the box.

**A finding this round handed over:** `app_oom` (warning) — „Alkalmazás memóriája elfogyott: immich
(immich-postgres) — egy folyamatát a memóriakorlát leállította". That is immich's whole mystery
solved: its Postgres was OOM-killed during the reverse-geocoding import, which is why the server saw
`CONNECTION_CLOSED` and crash-looped twelve times. **The controller caught an OOM inside an LXC guest
and named the exact container** — worth recording against this project's standing finding that those
signals are usually invisible there. The diagnosis I spent twenty minutes reaching from logs was in
the alarm feed, correctly labelled, the whole time.

### Round 3 — `use` bookstack / accident: **system disk filled to 96 % for ten minutes**

| the five things | |
|---|---|
| what the customer saw | **nothing.** wiki, status and paste all answered before, during and after. No banner, no warning, no mail — the household was never told the disk was full |
| what the box did by itself | kept all twelve apps running on a 96 %-full root filesystem and released the space cleanly when the fill was removed (29 G used → 944 M used). The shared thin pool never moved (**39.69 %**) and the filesystem stayed writable |
| time to steady | the box never left steady — **26 containers before, 26 after**, none restarted |
| alarm fired / true? | **none fired**, checked twice independently after the fill was released |
| should have fired, did not | `disk_critical` is defined at ≥95 % used and the disk sat at **96 % for ten minutes**. **But this is the ladder working as designed, not a miss:** the fill-watch is a daily sweep (03:30) plus one check ~90 s after a controller start. Predicted before the round, confirmed after |

**Household loop: 12 operations, 0 failures.** The household kept using its apps normally throughout.

**The finding is the silence.** The honest answer to „would the household be told their disk is
full?" is **no** — unless the controller happens to restart while it is full. Here the timing was
almost comic: the controller restarted at 21:28 after round 2's power cut, so its one opportunistic
check ran about twenty seconds *before* the disk filled, and the next is not due until 03:30.

**A correction, recorded where it happened:** I twice labelled a mid-window reading „end of window",
estimating the clock instead of reading it. The readings were unchanged, but „nothing yet, five
minutes in" and „nothing in the whole window" are different findings. From here the end-of-window
check is taken when the round's own runner reports completion.

### Round 4 — `offsite-run` (mealie) / accident: **tunnel killed for ten minutes**

**The drawn ACTION never ran.** The runner aborted with „no session — mine, not the product's":
round 2's power cut had rebooted the guest, `/tmp` is cleared on boot, and the dashboard password
file lived there. Round 3 was a `use` round and never needed it, so round 4 was the first to find it
gone. **The accident was measured; the off-site run was not.** Recorded as half-measured rather than
re-run and presented as whole — re-running a round after seeing it fail is how a drill starts
choosing its own results. The file now lives in `/root`, which survives a reboot.

| the five things | |
|---|---|
| what the customer saw | from outside, the apps vanished for ~90 s (public route **530**) and came back on their own (**200**); from inside the house, nothing — traefik answered **301** throughout |
| what the box did by itself | **repaired its own tunnel.** cloudflared killed 21:41:33Z, running again **21:43:07.478Z (~97 s)**, with `RestartCount=0` — so Docker's `unless-stopped` policy did *not* do it; the controller's protected-infra recovery redeployed it („[infra] deploying cloudflared →…") |
| time to steady | the apps never stopped; the way IN was restored in **~97 s** |
| alarm fired / true? | **two, correctly paired** — `health_critical` (error) 21:43, `health_recovered` (info) 21:48. Exactly what the ladder predicts for a missing protected container, and **the alarm was not a dead end** |
| should have fired, did not | none for the accident |

**Household loop: 10 operations, 0 failures — and that number is narrower than it looks.** The loop
does not follow redirects, so it measures „is the app serving on the box", never „can the household
reach it from outside". It saw nothing while the public route was returning 530.

### Round 5 — `use` privatebin / accident: **docker restarted inside the guest**

| the five things | |
|---|---|
| what the customer saw | a gap well under a minute: 200 before, **404** two seconds after docker returned (traefik had not re-registered routes), serving again by 21:54:19Z — about **40 s** of shut doors |
| what the box did by itself | everything. Restart ran 21:53:27Z→21:53:41Z; **all 26 containers back at t+16s**; the controller returned with them, waited 51 s for the fleet to settle and found **nothing boot-orphaned to repair** — correct, since every container had already come back on its own policy |
| time to steady | **16 s** to 26 of 26 containers; **~40 s** until the doors served. The slower number is the one a household feels |
| alarm fired / true? | **`controller_started` (info) — true and correct**, and the only line the ladder expects: no `app_start_failed` (90 s boot grace), no liveness alarm |
| should have fired, did not | **none** |

**Household loop: NOT SAMPLED** — 0 lines, because the round lasted ~18 s and the loop samples every
2 minutes. Recorded as not sampled, never as a pass.

**A trap avoided.** The round's own alarm snapshot was taken **six seconds** after the controller
started, and from it `controller_started` looked missing. A later reading shows it present at 21:53.
An alarm cannot be called missing by a measurement taken before it could have fired.

### Round 6 — `backup-system` (whole-guest backup) / accident: **none** (control round)

| the five things | |
|---|---|
| what the customer saw | their apps went away and came back: during the backup **4 of 26** containers were up and every app answered **404** publicly; all 26 were serving again by 22:01:13Z. No banner, no mail — correctly |
| what the box did by itself | quiesced the apps, snapshotted, ran the local tier, **failed** it, announced the failure with a retry schedule, then ran the **off-site tier from the snapshot with the apps already back up** — 21:59:54Z → **22:08:27Z (~8½ min)**, encrypted to ep0, taking **no local disk at all** |
| time to steady | apps down ~21:55:30Z → 22:01:13Z ≈ **5m43s** — **contaminated by my own intervention** (I killed the local leg at 21:59:45), so it is an upper bound on the quiesce window, not a clean measurement |
| alarm fired / true? | **`whole_guest_backup_failed` (error) — true, and better than true:** „Whole-guest backup FAILED on the **local tier** — retrying with backoff (next attempt in 15m0s)". It names the tier, not „the backup", and says what it will do next. The status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`). **No alarm for the 22 apps it stopped** — correct, those stops are suppressed |
| should have fired, did not | **none** |

**The finding to carry forward:** a whole-guest backup on this box **cannot use its local tier** — a
~29 GB source into a 14 GB root filesystem — and the product handles that honestly: it fails the tier,
says which tier, schedules a retry, and still gets the data out of the house on the off-site tier.
The local tier will keep retrying and keep failing on a box shaped like this one.

**My intervention, and the correction it needed.** I killed the local leg when two samples showed `/`
falling at ~16 MB/s with 3.6 GB left — under four minutes from wedging the nested PVE. The arithmetic
stands, but I first wrote „I stopped the backup", which was wrong: I stopped **one leg**, and the
off-site leg started by itself seconds later and succeeded. Counted as an intervention either way.

### Round 7 — `use` nextcloud / accident: **internet cut for ten minutes**

*Drawn as `update`; ran as `use`, because the catalog bump was never pushed — the catalog gates
returned INCONCLUSIVE and undetermined is never a pass. Decided and recorded before the round, not
after it.*

| the five things | |
|---|---|
| what the customer saw | **depends where they stood.** At home: nothing — traefik answered `301` throughout and every app kept serving. Away from home: ten minutes of nothing — the public route gave **502**, then **200** about a minute after the block lifted |
| what the box did by itself | kept all **26** containers running, needed no repair, and re-established the way in unaided: 530 three seconds after unblocking, **all four apps 200 by 22:21:52Z — ~64 s** |
| time to steady | the apps never left steady; only the path in broke and healed, in **~64 s** |
| alarm fired / true? | **none, and none should have** — the ladder puts `node_stale` at 30 minutes and this was ten. Nothing false was raised either |
| should have fired, did not | **none** |

**Household loop: 10 operations, 0 failures — narrower than it looks.** It does not follow redirects,
so it measured the box (up throughout) and was blind to the public outage.

**The question this round was meant to answer, and honestly did not.** Are alarms raised while the hub
is unreachable retried and then silently dropped? **Not exercised.** The box reports every **15m0s**
(measured: 21:53:49Z, 22:08:43Z) and the cut fell entirely between two reports — the next was due
~22:23:43Z, after it lifted. Nothing was attempted, so nothing could be lost. Recorded as *not
exercised*, never as *passed*.

**The fence held, and it was checked against a baseline taken beforehand.** The accident flips a
host-wide sysctl on demo-hp and inserts two `physdev` rules. Afterwards: `-P FORWARD ACCEPT`, **0**
physdev rules, sysctl **0** — identical to the pre-round reading. demo-hp also carries guests 9201
and 9202, so an abandoned rule would have been a fence breach, not an untidy drill.

### Round 8 — `backup-app` nextcloud / accident: **internet cut for ten minutes**

**22:35:48Z–22:47:59Z.** The app-data backup ran first and finished in 1 m 55 s
(`db_dump` count 5, `success:true`, 22:37:48Z). The cut began six seconds later, so the two barely
overlapped — and would not have interacted in any case: `backup-app` is the **local** app-data tier
and needs no internet. The off-site tier is a different action.

| the five things | |
|---|---|
| what the customer saw | **Nothing at home.** Every front door kept serving on the LAN for the whole ten minutes. From outside the house the sites were unreachable — the public path was down. 26 apps up before, 26 after, never fewer. |
| what the box did by itself | Kept every container running, kept backing up, kept reporting to the hub, and rebuilt the public path unaided when the link returned. No restart, no intervention, noaction from me. |
| time to steady | **≤43 s** after the link returned (public doors 200 again at 22:48:38Z). Containers never left steady at all. |
| alarm fired / true? | **none fired, and none should have** — `node_stale` is a 30-minute threshold and this was ten. Nothing false was raised. |
| should have fired, did not | **none** |

**And the finding of the round is against my own instrument, not the box.**

The hub report due at **22:38:43Z fell inside the cut** — and it **succeeded**:
„Hub report pushed successfully (15526 bytes)". It succeeded because `hub.felhom.eu` resolves to
**192.168.0.192**, a LAN address (measured from both the guest and the host), and my injector blocks
everything **except** the LAN. So the accident named „internet gone" only ever removed the **public**
path. The box never lost the hub, in round 7 or in round 8.

Two consequences, both stated plainly:

1. **The dropped-event question is still unmeasured** after two rounds that appeared to measure it.
   Events pushed while the hub is unreachable are retried three times and then dropped permanently,
   with no queue — that behaviour has still never been seen live.
2. **My own memory file carries this exact warning** („hub.felhom.eu resolves to the LAN here; an
   internet-cut drill must block it too") and I did not apply it. A warning that is written down and
   not read is worth nothing, which is the same class of failure as an unread alarm.

The injector is corrected for round 9's drawn internet cut so that the hub address is blocked too —
**from the VM's side, at the host's tap rule.** The hub itself is never touched; the fence is kept.
This is a repair to a broken instrument, not a re-draw: the drawn action, app and accident for every
remaining round are unchanged.

**A sixth mistimed reading, and the fix is in the runner now.** The round's own AFTER step read the
public doors **three seconds** after the unblock and reported 530 on all four. That reading could
never have been fair. The re-measurement 43 s later read 200 on all four. The runner measured the
recovery before the recovery could begin — so the runner is the thing that gets fixed, not the note.

### Round 9 — `use` uptime-kuma / accident: **internet cut for ten minutes, hub included**

**23:00:50Z–23:12:00Z.** The first cut of the night that really removed the hub. The injector was
corrected between rounds 8 and 9; the drawn action, app and accident were **not** changed.

| the five things | |
|---|---|
| what the customer saw | **Nothing at home.** uptime-kuma answered 200 on all three reads during the action, and all four front doors read 200 at both post-accident readings. 26 apps before, 26 after, never fewer. |
| what the box did by itself | Kept every container running while it was cut off from the hub **and** from its own host agent. Built its report on schedule, tried to push it **three times over 1 m 40.8 s**, gave up, and carried on serving. Both links repaired themselves the instant the block lifted — no restart, no repair action, nothing from me. |
| time to steady | The apps never left steady. The doors read 200 at the first reading, **2 s** after the unblock, and again 63 s later. |
| alarm fired / true? | **none fired, and none should have** — `node_stale` is a 30-minute threshold. Nothing false was raised. |
| should have fired, did not | **none** — but see the near-miss below, which is a finding in its own right. |

**What a lost report costs: measured, not assumed.**

```
23:08:42 [INFO]  [scheduler] Running job: hub-report
23:08:42 [INFO]  [report] Building system report
23:10:23 [WARN]  [report] Push failed: … context deadline exceeded
23:10:23 [ERROR] [scheduler] Job hub-report failed: hub push failed after 3 attempts (took 1m40.813s)
```

Three attempts, then it stops. Nothing is queued. It gave up **31 seconds before** the link returned.
And that is correct: a report is a **snapshot**, so a lost one costs nothing — the next snapshot
carries the same truth, and the controller says so itself („backing off (the 15-min cycle still
reconciles)"). **This is not the event path.** A dropped event is a lost *fact*, not a stale copy of
a picture that will be redrawn. No event happened to be raised during the cut, so the event-drop
behaviour is **still unmeasured** after three rounds of internet cuts.

**And it came back by itself, on schedule.** The very next scheduled report went through —
**23:23:43Z, „Hub report pushed successfully (15354 bytes)”** — exactly 15 minutes after the
cycle that failed, and nothing was done to the box to achieve it. The reading was taken at
23:24:21Z, deliberately *after* the report was due, so it could not be premature. So the whole
shape of a hub outage is now measured end to end: build → three attempts → give up →
keep serving → next cycle succeeds → no alarm, no loss.

**The near-miss, which is luck and not design.** Last good report 22:53:43Z; next scheduled
23:23:42Z; `node_stale` trips at 30 minutes. The gap is **29 m 59 s**. The staleness threshold is
exactly twice the report cadence, so **a single failed push spends the entire budget** — one second
of ordinary jitter and the operator is paged about a box that was healthy throughout and had already
repaired itself. Filed as a register row.

**A second fidelity fault in my accident, named like the first.** The cut also severed the controller
from its host agent (`GET /backup/tiers` to the link-local `169.254.253.1` timed out twice). In a
real house an ISP outage does not do that — controller and agent share one machine. So „internet
gone" as injected is **broader than its name**: internet, hub *and* local agent. No further internet
cuts are drawn, so the injector stays as it is and this caveat travels with rounds 7, 8 and 9.

### Round 10 — `restore` uptime-kuma / accident: **hard reset, four seconds into the restore**

**23:26:06Z–23:29:46Z.** The roughest pair drawn. The restore was accepted at 23:26:08Z
(302, „Visszaállítás elindítva"); the reset button was pressed at 23:26:12Z, mid-write.

| the five things | |
|---|---|
| what the customer saw | They pressed restore, were told it had started, and **four seconds later the whole machine went dark.** About two minutes of nothing. Then every app was back and every front door answered. **Nothing ever told them what became of the restore.** |
| what the box did by itself | Booted, and brought **26 of 26 containers** back with no help. Boot reconciliation named the one app it had to recover („1 app(s) recovered in 1 attempt(s): [paperless-ngx]"), sent a startup hub report at 23:28:26Z, and settled its health probes. No intervention. |
| time to steady | **150 s** — 0 containers at t+12 s, 25 at t+133 s, 26 at t+150 s. Doors 200 at both readings (23:28:42Z and 23:29:44Z). |
| alarm fired / true? | **one, true** — `controller_started` (info). Exactly what the ladder expects after a reboot. No false alarm. |
| should have fired, did not | **none from the alarm ladder** — but the restore silence below is a legibility gap, filed as a row. |

**The restore left no trace anywhere, and the product has no place to leave one.** Four candidate
status endpoints all 404 (`/api/restore/status`, `/api/backup/restore/status`,
`/backup/restore/status`, `/api/restore`). `/api/backup/status` carries no restore field at all.
On the pages, the only restore text is a **button label** and a JavaScript label expression. On disk,
in the real data directory, there is no restore, lock or state file anywhere — and **no file at all
was modified in the reset window**. An interrupted restore and a restore that never happened are
indistinguishable, to the customer and to me.

**CORRECTION, 00:24Z — the paragraph above is wrong and stays visible so the correction is too.**
The four endpoints I called were four I **guessed**, and all four were wrong. The real route, read
out of the restore page's own JavaScript, is **`/api/backup/restore-status`**, and it exists:
`{"ok":true,"data":{"running":false,"started_at":"0001-01-01T00:00:00Z"}}`. So a restore status
surface **does** exist. What is true — and is the better finding — is that after the reboot it is
**blank**: `started_at` is the Go zero value, and the payload carries no `last` field at all, while
the page's own script renders „<operation> sikertelen." from `st.last.message`. **The restore record
is in-memory only and does not survive the machine stopping** — precisely the case a hard reset
creates, and precisely when a household would want to be told. The register row is corrected to say
that instead. I found the real routes by asking the controller for its own rendered links, which is
what I should have done before filing anything.

**The limit of that measurement, stated rather than glossed.** Only four seconds elapsed, so the
restore may have finished or may never have written a byte — and I cannot tell, because the
controller's log stream holds **zero lines before 23:28:00Z** (a reset starts it fresh) and the debug
ring died with the machine. What is independently verifiable is the **absence of any restore record**,
and that is what is filed; it holds however far the restore got.

**Four of my own instruments failed in this round, and all four are recorded in the evidence:** a
claim that was unfalsifiable when written; an on-disk check against a directory that does not exist;
a household count that reported 0 lines and 0 failures when the truth was one line and it *was* a
failure; and a disk guard that reported „active" all night while being a **transient** unit that
vanished at the reset. It is now file-backed and enabled, and its script has been copied off the box.

### Round 11 — `use` paperless-ngx / accident: **the data drive pulled out for twenty minutes**

**23:50:49Z–00:13:09Z.** The drive was detached from the **running** box at 23:50:54Z and put back at
00:10:58Z. This is the round that produced the most alarms of the night, and every one of them was
true.

| the five things | |
|---|---|
| what the customer saw | **The four apps whose files live on that drive stopped** — Paperless, Jellyfin, Nextcloud, Immich — and Paperless's front door went 404. The other eleven apps kept serving normally throughout. About twenty minutes later everything was back, roughly a minute after the drive was plugged in again. |
| what the box did by itself | Noticed the drive had gone and **named it by the label the household sees** („Adatlemez"), named **each** broken app individually, degraded its own health, waited, noticed the drive return, restarted the apps and recovered its health. No restart, no repair, nothing from me. |
| time to steady | **67 s after the drive returned** (26 containers again at 00:12:05Z). The door followed at 00:13:06Z. |
| alarm fired / true? | **eight, all true, correctly paired at both ends** — `storage_disconnected` (error) → four `app_start_failed` (warning) → `health_degraded` (warning) → `storage_reconnected` (info) → `health_recovered` (info). The four apps named are **exactly** the four with data on the pulled drive. Nothing false was raised. |
| should have fired, did not | **none** |

**The alarm that looked missing, and was not.** The round's own snapshot at 00:13:08Z showed
`health_degraded` with no recovery — which would have been the first missing alarm of the night. The
recovery fired at **00:13**, and the snapshot missed it by **seconds**. A re-read at 00:14:13Z, taken
after the apps were back, found it. This is the discipline from the earlier mistimed readings earning
its keep: when a measurement could have been early, it is re-taken rather than turned into a verdict.

**The drive came back clean** — `/dev/sdd`, 98 G, 2 % used, mounted at `/mnt/felhom-drives/hdd_1`,
with the guest's mountpoint config unchanged, 26 containers up and every real front door serving.

### Round 12 — `use` paperless-ngx / accident: **nothing** (closing control round)

**00:15:48Z–00:16:55Z.** The night's last round, drawn as a control.

| the five things | |
|---|---|
| what the customer saw | Nothing at all. The app answered 200 on all three reads, and every front door answered 200 at both readings. |
| what the box did by itself | Nothing needed doing. 26 containers before and after. |
| time to steady | **1 s** — it never left steady. |
| alarm fired / true? | **none, and none should have.** The newest entry in the feed is still round 11's `health_recovered` at 00:13. |
| should have fired, did not | **none** |

**Checked rather than assumed:** `inject.sh` has no „nothing" case — its default branch exits 2 on an
unknown accident. The control rounds never reach it, because the runner handles the no-accident case
itself and says so („accident: none — control round, deliberately"). This matters because *a broken
injector produces exactly the same result as a control round*, and the only way to tell them apart is
to look at which code path ran.

**What the closing control round is worth.** It shows the quiet is real: after eleven rounds of power
cuts, resets, full disks, severed networks and a drive pulled out of a running machine, a round in
which nothing was done produced nothing — no alarm, no restart, no drift. The alarm feed is not
simply noisy.

## Phase 2 — the morning after

**Every app answered through its own front door.** All twelve real names, measured at 00:19:16Z —
`cloud`, `inventory`, `media`, `paperless`, `paste`, `photos`, `recipes`, `share`, `status`,
`travel`, `vault`, `wiki` — **200 on the public path, every one**. And healthy is not inferred from
a door: all **26 containers** report `Up … (healthy)`, except `traefik` and `cloudflared`, which
carry no healthcheck and show a bare `Up`. The four showing seven minutes are the drive-backed apps
restarted after round 11.

**The off-site restore could not be done, and two independent instruments agree why.** The brief asked
for one DB-backed app restored from off-site onto scratch 9202. It has nothing to restore from:

| instrument | answer |
|---|---|
| `restic`, with the box's own key, password file and `known_hosts` | `Fatal: wrong password or no key found` — exit status **1**, read from restic itself rather than from the end of a pipeline |
| the product's own status surface, which is what a customer sees | `{"orphaned":true, … ,"snapshots":0,"status":"error"}` |

The repository is orphaned **because this box is a rebuild for an existing customer**: on a rebuild
the restic password is minted fresh, so snapshots written under the old one can never be opened
again. That is a known, documented shape, and **the product surfaced it honestly** — the true
`offbox_repo_orphaned` alarm of round 1, mailed to the operator within two seconds of the run.

**What did leave the house.** The *whole-guest* off-site copy is a different store and it worked: ep0
holds two intact snapshots for this box, including tonight's at **21:59:54Z** — round 6's off-site
leg, the one that started by itself after I killed the local leg. Its file index is roughly four
times the afternoon copy's, consistent with a guest by then carrying twelve apps. **Stated as a
limit:** that is a *listing*, not a verification. A PBS verify job would prove restorability and it
**writes** verify state, so it was not run — ep0 is read-only for evidence tonight.

**The household loop.** 204 probes from 21:06:30Z to 00:18:10Z. Seven lines flagged as failures, of
which **only two are real events** — `wiki` during round 2's power cut and `cloud` during round 10's
hard reset, each a single sample, each healed before the next probe. The other three were **my own
classifier** counting a 301 redirect as a dashboard failure; the log carries that correction in its
own words at 21:11:25Z and the wrong lines were left in place so the correction stays visible. Ten of
the twelve rounds left **no mark at all** in this log, including the twenty minutes with the drive
pulled — the loop samples each name every two minutes, so that silence is the instrument's sampling
rate and **not** evidence the household saw nothing.

**The catalog bump: verified reverted, not remembered.** Clean tree, local exactly level with
`origin/main` (0 ahead, 0 behind), the nextcloud template still on its original `redis:7-alpine` pin
and `catalog_since: "2026-07-18"`, newest commit 2026-09-15. The bump was prepared, **refused by the
catalog's own gates** (image-resolvable and volume-persistence both INCONCLUSIVE — its own canary
failed, so the verdict was UNDETERMINED, never a pass) and reverted before any push.

**And one delivery proof the truth table could not give.** The mailbox shows every alarm **arrived**
at the operator, not merely that it was stored — round 11's whole set, round 6's tier-naming backup
failure, round 1's orphan warning, the OOM, the deploy failure and my own thin-pool `storage_fill_critical`.
In a project where 91 events once sat in a database having e-mailed nobody, *raised* and *delivered*
are two different claims, and only one of them had evidence before tonight.

## Interventions — counted, with the reason for each verdict

**One.** At 21:59:45Z in round 6 I killed the **local leg** of the whole-guest backup. The arithmetic
that forced it: a ~29 GB source being written into a 14 GB root filesystem at ~16 MB/s, i.e. under
four minutes to a full `/` on the nested host, mid-round. What it cost: a leg that could never have
succeeded. What happened next without me: **the off-site leg started by itself from the same snapshot
and succeeded in about eight and a half minutes.** Filed as **R-548**.

My first note on it said „I stopped the backup". That was wrong and is corrected in the evidence: I
stopped **a leg** of it, and the box completed the other one unaided.

**Both pre-declared presses went unused.** O1, the „Send self-bind link" button, was not needed —
the automatic mail was already waiting (18:17:46Z) and the box bound with **zero** operator presses.
O2, „Re-issue PBS credentials", was not needed either — the acknowledged-delete path re-issued them
by itself (`pbsdr_auto_reissue`, 20:19Z). **Both prompt claims they were insurance against turned out
to be true**, and the F-14 path was measured live for the first time.

**Counted separately, because it is not a round result:** Phase 0's seeding repairs. I filled the LVM
thin pool to 100 % by firing twelve deploys at once, then repaired the damage — a guest restart to
clear an `emergency_ro` remount, dropping corrupt image layers, a remove-with-data and one consistent
re-deploy after a re-seed minted fresh database passwords over initialised volumes, and a rewritten
`APP_KEY`. **The damage was mine, not the product's**, the product's behaviour throughout was correct,
and every repair went through the product's own endpoints rather than by hand-running compose. Listed
in full in `evidence-chaos-night-2026-09-17/interventions.txt` so the distinction is visible rather
than convenient.

**Not counted, and why:** acts on my **own instruments** — moving the dashboard password after a
power cut cleared `/tmp`, rewriting the injector and the runner, re-creating the disk guard as a real
unit. Counting those would flatter the night in one direction and pad the stop-rule count in the
other. **Round 4 is the clearest case of declining to intervene:** the tunnel was left dead on
purpose — „NOT restarting it by hand — whether it returns by itself IS the measurement". It
returned by itself in 97 s.

**Standing against the stop rule: 1 of 4.** The night ran its full twelve rounds.

## Teardown — three layers, stated

**Machine — gone.** VM 336 stopped and destroyed with all three disks purged (00:35:25Z), gated on
its **name** rather than its number because two standing guests share the host. `qm list` shows no
VMs; `/mnt/hdd_1/images/336` no longer exists. The storage was verified to *be* `/mnt/hdd_1` from its
own definition (`nvme-scratch`, `path /mnt/hdd_1`, `is_mountpoint yes`) rather than assumed. The
machine had **three** disks, not the two the brief asked for — the third was mine, added in Phase 0
after I filled the thin pool — and that is recorded rather than quietly removed.

**Host — clean, measured before and after.** `nvme-scratch` 6.78 % → **1.61 %** (~48.5 GB returned);
`local-lvm` **unchanged at 44.75 %**, so the fence that said *never local-lvm* held; free space on
`/mnt/hdd_1` 827 G → 875 G. Guests 9201 and 9202 still running. The household loop and the disk guard
were stopped and disabled **after** their logs were copied off (the guard's log was 0 bytes — it
never fired). Firewall back to `-P FORWARD ACCEPT` with **0** physdev rules, so none of the three
network accidents left a rule behind. Scratch 9202: **nothing to remove**, shown rather than said —
three infrastructure containers, no app deployed, no stack touched since 21:00, no off-site config.

**Hub — host record deleted through the acknowledged flow.** The first attempt at 00:38:37Z was
**correctly refused** (409, „Host is ONLINE") — the box had died inside the hub's liveness window. The
acknowledged delete went through at **07:25:13Z** (`confirm_host_id` + `delete_escrow=1` → 303). Every
line of the after-state written down *before* the act matched: the host answers 404; `drill-r50`,
both demo hosts and the `tester-1` **customer** still answer 200; the customer now lists zero hosts.
**The automatic connect mail arrived two seconds later** (07:25:15Z, „Kösd össze a Felhom dobozodat"),
quoted in full with its token redacted in `teardown-hub.txt` — and it is provably tonight's, because
the mailbox held no such mail newer than 18:17:46Z when checked at 00:38Z.

**Why the hub layer finished six hours late — my fault, not the product's.** The retry was guarded
by „don't post while the host page contains ONLINE". That word lives in a JavaScript string that is
**always** on the page, so the guard could never pass. It refused six times and gave up at 01:20Z,
while the hub's structured answer would have said `"status":"down"` from about 00:54Z. Nothing ran
again until 07:24Z.

**ep0 — backups stayed, nothing removed.** Read three times: 00:17:15Z, 00:36:52Z (just before the
delete) and 07:25:35Z (after). Identical every time — three namespaces, two snapshots each, six in
total, 16 G used. No prune, no verify, no write.

## Claims in the prompt that turned out wrong — named first, as asked

**The two the brief itself flagged both turned out TRUE, and both were checked tonight rather than
assumed.**

1. **„The automatic mail is waiting in the mailbox."** The brief warned this had been read from
   *yesterday's* host delete and not verified. **It was true.** The self-bind mail of **18:17:46Z**
   was in the mailbox, and the box bound with **zero operator presses** — so the pre-declared press
   O1 was never needed.
2. **„The WG hook provisions by itself after an acknowledged delete" (the F-14 path).** The brief
   noted this had never been measured live. **It was true, and it was measured live for the first
   time:** `pbsdr_auto_reissue` at **20:19Z** — „Previous key destroyed (acknowledged deletion) —
   credentials re-issued automatically." The second pre-declared press, O2, was never needed either.

**Now the ones that were wrong.**

3. **WRONG: „restore one DB-backed app from off-site onto scratch 9202."** It could not be done at
   all on this box, and not because anything broke. This box is a **rebuild for an existing
   customer**, so its restic password was minted fresh and the snapshots already in the remote store
   can never be opened by it again. Two independent instruments agree: `restic` itself
   (`Fatal: wrong password or no key found`, exit 1) and the product's own status
   (`orphaned:true, snapshots:0, status:"error"`). The brief assumed an off-site app repository this
   box could open; on a rebuild fixture there is none.
4. **WRONG in effect: „a system disk + one data disk."** The machine ended the night with **three**
   disks. The third, 64 G, was added by me in Phase 0 to extend the LVM thin pool after I filled it
   to 100 % by firing twelve deploys at once. **The deviation is mine, not the brief's**, but the
   fixture was not the one the brief described and saying so is the point.
5. **WRONG: round 7's drawn action `update` was not performed as drawn.** The catalog's own gates
   returned `image-resolvable INCONCLUSIVE` and `volume-persistence INCONCLUSIVE` — its **own canary
   failed**, so the verdict was UNDETERMINED, which is never a pass. The round ran `use` instead. A
   deviation from the drawn schedule, logged rather than quietly substituted.
6. **WRONG, and mine rather than the brief's: „an internet cut tests what happens when the hub is
   unreachable."** The accident's *name* implies it; on this network it was false. `hub.felhom.eu`
   resolves to a **LAN** address here, and my injector allowed the whole LAN — so rounds 7 and 8 cut
   the public path only, and the box never lost the hub. My own memory file carries that exact
   warning and I did not apply it. Fixed between rounds 8 and 9 by blocking the hub address **from
   the VM's side**, which is what finally made round 9 the measurement it was supposed to be.
7. **WRONG as a description of the night's clock: the schedule table's times.** The table drawn from
   the seed lists rounds at 23:30 through 04:05. Those were **nominal**. The real spacing was 25
   minutes from each round's actual start, and the night's twelve rounds finished at **00:17Z**,
   roughly four hours earlier than the table's own column suggests. Each round's real timestamps are
   recorded in its own section; **the drawn order, apps and accidents were never changed** — only
   the wall-clock the table guessed at.

**And one the brief did not make, which the night could not answer.** Events pushed while the hub is
unreachable are retried three times and then dropped permanently, with no queue. Three ten-minute
hub outages happened and **no event was raised during any of them**, so that path is still
unmeasured. What *was* measured is the **report** path: built, three attempts over 1 m 40.8 s, given
up, and the next scheduled report succeeded — and a report is a snapshot, so nothing was lost.