felhom.eu/documentation/audits/DRILL-prove-fixes-0243-2026-09-16.md

# DRILL — prove the P1 fixes on a fresh box (2026-09-16)

**Interventions: 0** (O1 the operator's self-bind press and O2 the PBS re-issue press were pre-declared and are counted apart).
**Ready for a volunteer: NO — and the reason is new, not one of the old ones.** Every P1 fix this drill set out to prove did hold on a fresh box. But a one-drive box with no off-site tier — the state every fresh install starts in — keeps **none of the household's own files in any backup**, while the backup page says it does, and a restore then reports success and leaves the app listing photos it cannot open (R-537, R-538).
**The automatic connect e-mail: PASSED.** The host record was deleted at 12:22:59Z and the mail „Kösd össze a Felhom dobozodat” reached `tester1@felhom.eu` at **12:23:00Z — one second later**, with `selfbind_link_sent … (host delete)` on the customer timeline. The requirement was two minutes.

> Baselines at start (re-verified against live Gitea): controller `383a30b3c07b` v0.243.0 (`Unreleased`: the
> „0 B" tile fix), agent `e98b857684f4` v0.131.0, felhom.eu `351296114c4d` hub v0.114.0, catalog `94bc5febaca2`.
> Golden before this run: 0.242.0 (2026-09-14). Customer `tester-1`, domain `enkicsifelhom.hu`, e-mail
> `tester1@felhom.eu`, **no host**, DR tier ticked, ep0 token held with no descriptor, namespace EMPTY
> (`phase3-ep0-before.txt`).

## Phase 0 — rulings, the golden, the N100

**Ruling 1 (2026-09-16, operator).** The signing keys stay on DooPlex, owner-only
(`/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519`, mode 0600);
CC may sign `agent_update` jobs with them until the first PAYING customer — testers excluded. Recorded in
`CONTEXT.md` and `04-control-plane-authorization.md` §3.1. **R-533 closes** (path + mode);
**R-530 narrows** to "a fleet rollout step is still undesigned".

**N100 signed and updated (R-530).** `felhom-opsign -op agent_update -host demo-felhom-8363b5 -key-id
felhom-op-1 … -agent-version 0.131.0 -sha256 1118b552…c9c`, queued 08:51:34Z, **the box took it at
09:04:43Z** (its own poll, ~13 min), dwelled 60 s and committed; `controller-supervisor: started`
(interval 30s, confirm 2, crash-loop 3/15m) in its journal. Peti's box untouched. The N100 keeps
controller 0.242.0; its floor was not moved by this run.

**Ruling 2 — hub v0.115.0 (R-529).** `host_stale` / `host_down` / `host_recovered` join the `node_*`
cooldown bypass with the same 5-minute dedupe. Red-proof: with the three host types removed the test fails
at "host_stale 39 min after the previous one was suppressed (sent=1)". Recorded in `08-alarm-ladder.md` §6.2
beside the 2026-09-15 node ruling. Deployed by GitOps: ArgoCD Synced, `deploy/hub` image 0.115.0.

**Golden 0.243.0 baked and vouched.** First attempt FAILED and is recorded: the template picker took the
LAST `debian-13` line, which is **arm64**, and the container would not start ("Detected container
architecture: arm64"); no golden was produced, the scratch guest was destroyed and the disk reverted.
Re-run with the amd64 template: `GOLDEN_VERSION=0.243.0`,
`GOLDEN_SHA256=e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a`. Markers from the saved log
(`phase0-bake-full.log`, 326 lines): `docker OK (overlay2` ×1, `including mount point` ×2 (rootfs + mp0),
`upload OK (HTTP 201)` ×1, `FATAL` 0, `excluding` 0. Token leak checks: control copy 1, committed log 0.
Vouched as a three-field change — golden 0.243.0 + agent 0.131.0 + min agent 0.131.0 — hub logged
`Artifact manifest set: agent=0.131.0 golden=0.243.0 min_agent="0.131.0" wrapper_sha=true`.

## Phase 1 — the walk, as a volunteer

| # | step | what happened | time (UTC) |
|---|---|---|---|
| 1 | download from `felhom.eu/letoltes` | the page names `felhom-installer-1.27.1-pve9.2-1.iso`, 1 705 322 496 B, sha256 `25637007…c053`; the downloaded file matched **byte-for-byte** | 08:55–08:56 |
| 2 | install | boot menu default is the **graphical** entry (15 s). It IS drivable by keyboard (Tab moves focus, Enter presses), but each step needs a screenshot to see where focus is, so the walk switched to the **text-mode** entry — the other entry a volunteer is offered — and says so. Screens: EULA (English Proxmox), disk target **`/dev/sda 32 GiB` only** (the 100 GB data disk is never offered as the target), Hungary/Europe-Budapest/Hungarian prefilled (the keyboard was set to U.S. English **for typing only**), password + `mail@example.invalid` prefilled, host name prefilled `pve.example.invalid` (replaced with `tester1.enkicsifelhom.hu`), DHCP `192.168.0.128/24`, summary with **[X] Automatically reboot after successful installation** | 09:00–09:12 start |
| 2b | the reboot trap, measured | the automatic reboot **re-entered the installer**: a guest reboot reuses the running process's boot order, so a disk-first change made during the install does not apply. A volunteer with the stick still in sees the installer again. Cold stop + remove the stick + start → boots the installed system | 09:57 |
| 3 | first console screen | **Felhom-only, Hungarian, no admin URL** (`8006` 0 occurrences): „Felhom otthoni szerver … Ezen a gépen most nincs dolgod … Párosító kód: 37S-NFE … Nyisd meg az e-mailben kapott linket". The hub's Hosts page lists the same box as an unclaimed appliance with the same code | 09:58 |
| 4 | the connect mail + bind | **O1 (pre-declared, not counted):** the operator's „Send self-bind link" pressed once — the customer was already waiting when the automatic trigger shipped. Mail arrived at **09:59:56Z** to `tester1@felhom.eu`, subject „Kösd össze a Felhom dobozodat", naming the two things to enter and the 7-day / 5-attempt limits. The bind page took the pairing code + owner passphrase and answered **„Sikeres összekötés."** | 09:59:55 → 10:01:14 |

| 5 | „ready" | the hub served **agent 0.131.0 / golden 0.243.0** (this run's own bake) at 09:01:59Z; the guest came up and the controller announced itself: `controller_started (0.243.0)` at 10:03:30Z. The box's first host-report carrying a guest: 10:17:16Z (1 guest, 2 storage targets, 1 backup) | 10:01–10:17 |
| 6 | **the tunnel from outside** (R-510) | three GETs from DooPlex to `https://felhom.enkicsifelhom.hu` → **302 ×3 (cloudflare)**, following → **200** on „A szerver beállítása" — no 502, no 530. **R-510 CLOSED** | 10:04:19–25 |
| 6b | claim | the setup-code mail („Új beállító kód — újratelepült a szervered", 10:01:57Z, 72 h) + a new dashboard password → **302 → /**. First try refused with „Érvénytelen űrlap": the field names are `code` / `new_password` / `confirm_password` | 10:06:18 |
| 7 | **the off-site tier** (R-511 → **R-534**) | the WG hook refused exactly as R-511 describes; **O2 pressed** → hub v0.114.0's ADOPT path ran and the ENDPOINT refused: `status 255 … missing Datastore.Modify on /datastore/felhom-offsite` → 502, nothing written (fail-closed). So the adopt fix is sound and **inert until the ep0 grant is fixed** — new row **R-534 (P1)**. The box therefore has **no** whole-guest off-site tier | 10:02:15 / 10:03:29 |
| 8 | **the file manager** (R-513 on a FRESH box) | the app page shows „Kezdeti belépési adatok — Felhasználónév admin … A jelszót a Felhom állította be ezen a gépen"; reveal → 200, 16 chars; through the tunnel `files.enkicsifelhom.hu`: revealed **200**, `admin`/`admin` **401**, wrong **401** — and admin/admin was already 401 on a probe taken BEFORE any dashboard action | 10:06:43 |
| 8b | the data drive | „Nincs regisztrált adattároló" on arrival; the wizard formatted and registered `/dev/sdb` as „Adatlemez" (`/mnt/felhom-drives/adatlemez`, default, 97.9 GB) in ~2 s. **Two refusals worth recording:** the mount name rejects accented letters („érvénytelen csatlakoztatási név"), which is the first thing a Hungarian volunteer types, and the API needs `mount_name`, not `label` | 10:20 |
| 11 | **the backup page** (R-517 on a FRESH box) | per tier: „Helyi tároló (local) ✓ Utolsó sikeres mentés: 2026-09-16 12:08 · 624.3 MB · Naprakész"; „Biztonsági szerver – külön hardver (PBS) **nincs beállítva**"; the remote tile „Távoli rendszermentés **nincs beállítva**" (not ticked); the button says „A mentés alatt az alkalmazások leállnak — általában néhány perc, nagyobb adatnál több." | 10:22 |

| 9 | four apps | deployed through the dashboard API with the new drive as their data path: privatebin and vaultwarden **running by 10:23:29Z**, paperless-ngx and nextcloud **by 10:24:29Z** (2–3 min each); the hub logged four `app_deployed` events. Cards: **no `admin / admin` anywhere**; Vaultwarden's „Első lépések" is invite-first („Users" → „Invite User", „a regisztráció alapból le van zárva"); Paperless points at „Beállítások → Automatikusan generált értékek" and at the file manager's own login page | 10:21–10:24 |
| 9b | **Vaultwarden, R-512 on a fresh box** | over the public address, a stranger's `send-verification-email` → **400 „Registration not allowed or user already exists"**; the app answers otherwise (its `/api/config` 200) | 10:25:05 |
| 9c | **Paperless, R-514 on a fresh box** | 20 three-page PDFs posted at once through the public address → **20 SUCCESS, 20 documents**, no OOM; the container carries the new catalog limits (1280M, 1 worker × 1 thread) | 10:25:26–10:27 |
| 10–11 | **„Mentés most" and the page** | the button's promise measured: privatebin answered 200, went unreachable **10:27:17→10:27:43Z = 26 s**, then answered again while the dump continued — inside „általában néhány perc". The page kept „Helyi tároló (local) ✓ … 624.3 MB · Naprakész" and „PBS — nincs beállítva" throughout, and showed „Mentés folyamatban… (fázis: snapshotted)" while running. **The box's own first scheduled backup had already proven R-518 live**: `backup_tier_skipped (warning)` at 12:08:25 CEST — „tier felhom-pbs skipped: its storage does not exist … No app was stopped for it" + operator mail | 10:27 |
| 12 | which golden it landed on | **MEASURED, not inferred:** the day-0 install left the golden archive on the box (`vzdump-lxc-9100-…tar.zst`, 653 997 919 B) whose sha256 is **e2d1843c…c10a** — byte-identical to this drill's own bake; the guest runs controller **0.243.0**, so no self-update was needed | 12:02 |

**New finding on the walk: R-535 (P2)** — 25 minutes after a successful bind AND claim, with apps deploying, the box's console still showed „a doboz készen áll, és a párosításra vár" with the pairing code, and still promised „Ez a képernyő magától frissül".


## Phase 2 — the faults

Five faults, each recorded the same way: what the customer saw · what the box did · time to steady ·
did an alarm fire and was it true · did an alarm that should have fired stay silent. Evidence per
fault in `evidence-drill-0243-2026-09-16/phase2-*.txt`.

| fault | customer saw | box did | steady | alarm |
|---|---|---|---|---|
| **F9'** — controller killed 5 s into a deploy, empty budget | dashboard gone ~37 s; the interrupted app simply not installed | agent saw it on sweep 1, confirmed and restarted on sweep 2 (12:31:45 → 12:32:15/16 CEST) | **37 s** | `controller_restarted_by_agent` — true. **But** `app_deployed` for Mealie had already been sent at accept time → **R-536** |
| **F9''** — three more kills, 20 min apart, at idle | dashboard gone 30–90 s each time; the apps themselves never stopped | restarted every time: **61 s / 41 s / 61 s** back to 200 | 61/41/61 s | none — and that is the finding: the restarts NEVER accumulated (window 15 min), so the 30-minute brake was never armed. A controller dying every 20 minutes is restarted forever, traced only by an `info` event that mails nobody → **R-531**, an operator ruling, measured not changed |
| **F10** — a child deletes the photo folder | folder and 5 photos gone; after the restore the folder is **back and lists all five**, and **none opens** (`Sabre\DAV\Exception\NotFound`) | tier-1 restore replayed 3 volumes + the database in **35 s** and reported plain success | 35 s | none fired, and **none exists** for "restored database points at files that are not there" → **R-537, R-538** |
| **F11** — forgotten passwords ×5 | Nextcloud: 401 five times, no lock-out, correct password still works. The box's setup code: wrong twice, then **"Túl sok próbálkozás — próbáld újra 15 perc múlva"** from the third try; "Új beállító kód kérése" still works | claim counter locks the page for 15 min | immediate | `claim_lockout` (warning) at 13:11:42 CEST, operator mail **1 s later** — fired and **true** |
| **F12** — two reboots inside two minutes | dashboard gone ~2 min, then everything back | boot reconciler restarted all seven stacks; no double start; nothing stuck "telepítés folyamatban" | **124 s** after the second reset | none fired; correct — a reboot inside the liveness window is not an alarm |
| **M1** — memory pressure (R-528) | app restarted 9–10 times | all three OOM signals **silent**: `OOMKilled=false`, `docker events oom` empty, the cgroup invisible inside the guest, `dmesg` unreadable | — | no OOM alarm is possible on this box, same as on scratch 9202 |

**F10 is the finding of this drill.** On a one-drive box with no off-site tier — the state every fresh
install starts in — the household's own files are in **no backup at all**: the whole-guest tiers exclude
the data drive by design (`07-backup-architecture.md`, "[FACT] What the whole-guest tiers do NOT carry",
confirmed live: `excluding bind mount point mp8 … (not a volume)`), and the app's file leg lives at
tier 2 / tier 3, both unset. The design is not the defect. The defects are that the page calls tier 1
„DB + Konfig + Adatok" and prints the drive size beside it (**R-537**), and that a restore reports
success while leaving the app listing files it cannot open — and wipes the app's own trash, which still
held every byte (**R-538**).

## Phase 3 — the morning after

**Every app healthy, labels true.** Front doors at 11:15Z: PrivateBin 200, Vaultwarden 200, Paperless
302 (its login redirect), Nextcloud `status.php` 200, file manager 200 at `files.enkicsifelhom.hu`.
All seven stacks read `running`. Controller **0.243.0** on the page, agent **0.131.0** on the hub, the
customer row reads Version 0.243.0 / Floor v0.242.0 — a floor is a minimum, so that is correct, and the
box runs the golden this drill baked.

**Off-site restore onto 9202 — NOT WALKED, and not for time.** There is nothing to restore from. The
off-site tier was never provisioned on this box: the re-issue fails on the endpoint token's missing
`Datastore.Modify` grant (**R-534**), so no descriptor and no upload path were ever created. Read-only
listing of ep0 confirms it: `ns/tester-1/ct` is **empty** both before (09:12Z) and after (11:14Z) the
drill, while the neighbouring `ns/demo-hp/ct` lists group 9201 — the positive control proving the
listing method shows groups when they exist. **Nothing on ep0 was written, removed or pruned by this
session.**

## Interventions — counted, with the reason for each verdict

**Pre-declared and counted apart:** **O1** the operator's „Send self-bind link" press (the customer was
already waiting when the automatic trigger shipped) and **O2** the „Re-issue PBS credentials" press.

**Counted interventions: 0.** Nothing on this walk needed a shell, an operator, or knowledge a household
does not have. The four moments that could be mistaken for one, and why each is not:

| moment | why it is not an intervention |
|---|---|
| the automatic reboot re-entered the installer | a volunteer removes the USB stick and powers the box off and on — the same act. It is a real trap and is recorded as one, not as help from outside |
| the text-mode installer entry was used | the graphical entry IS keyboard-drivable; the walk switched because *I* cannot see the screen without a screenshot. A volunteer looking at a monitor has no such problem. A deviation of method, declared |
| the claim form and the drive form each refused once | both refusals were **mine**: I drove the API and guessed field names (`code`/`new_password`/`confirm_password`, and `mount_name` not `label`). A volunteer uses the browser form and never meets them |
| Paperless had to be repaired mid-drill | **my own damage** — I re-ran `docker compose up -d` by hand during the memory measurement and lost the controller-injected environment. Repaired through the controller's own API. Not a product fault |

**What a volunteer WOULD have hit, as defects rather than interventions:** the console keeps showing the
pairing banner long after the box is bound and claimed (**R-535**), the drive wizard rejects an accented
mount name — the first thing a Hungarian types (recorded on the walk), and the backup story in Phase 2
(**R-537**, **R-538**).

## Teardown — three layers, stated

**Evidence first (R-320).** The agent journal (2000 lines), the controller log (91 lines) and the box's
own state (`pct config`, `qm list`, `pvesm status`, `docker ps`) were copied to DooPlex **before** anything
was destroyed. Token-leak control: 0 hits in all three files.

**Layer 1 — the machine.** Nested VM **334** (`tester1-drill-0243`) stopped and destroyed with
`--purge --destroy-unreferenced-disks 1`. After: `qm list` empty, `/mnt/hdd_1/images/334` absent, no file
matching `*334*` left on the drive. **Fence check: demo-hp's own containers 9201 and 9202 are both still
running and were not touched.**

**Layer 2 — the host.** Nothing to remove: the drill box WAS the nested VM. demo-hp keeps its standing apps
and `bentopdf`.

**Layer 3 — the hub.** The **host record** `tester-1-652049` is deleted; the **customer** `tester-1` stays,
with its e-mail, domain and tunnel. **`RESET` was never used** — the customer-delete form on the hub is the
one that resets everything, and it was deliberately not touched. The host delete is **refused while a host
is ONLINE** (409, with no override by design, because a live agent would be permanently 401'd), so the
record had to fall stale first — which it does only after the machine is really gone. That wait is part of
the proof, not an obstacle.