diff --git a/REPORT-rehearsal-2026-08-09.md b/REPORT-rehearsal-2026-08-09.md index c5ca480..1aa1148 100644 --- a/REPORT-rehearsal-2026-08-09.md +++ b/REPORT-rehearsal-2026-08-09.md @@ -4,15 +4,20 @@ ## The answer to the runbook's question, first -**Not answerable today, and the reason is the answer.** *"If a person's machine is set up again from -scratch, does their data come back through the product, in one sitting, with no command line inside -the machine?"* — the walk never reached the data. **A machine wiped today cannot be set up again at -all**: the install fails at step 5 of 8, on every host, because the agent version the hub vouches was -never given a git tag (**R-273**). The step before that had already refused, because removing Felhom -leaves behind the very thing that blocks reinstalling it (**R-272**). +**Yes, the data comes back byte for byte. No, not in one sitting, and not without a shell.** -The data was never at risk and is not now: the planted dataset sits in off-site snapshot `41c830db` -with its fingerprints recorded, plus a PBS snapshot and four local vzdump archives. +All four planted files returned **BYTE-IDENTICAL** — including two Hungarian accented filenames +verified as *raw name bytes*, not as rendered text. The unlock took **21 s**, the restore **13.2 s**. + +But the walk completed only because two hard stops were cleared by someone who could open a terminal +and read source. **R-273**: the install died at step 5/8 on an agent version that was published as a +package but never git-tagged — cleared by completing the release. **R-280**: the reinstalled machine +could not re-attach its own data drive through any dashboard route, while the restore page said +„**Ez két kattintás**" and pointed at an empty list — cleared by POSTing an internal path +(`/mnt/sys_drive`) that no household could produce. + +Neither is a data-integrity problem. Both stop a household dead. **This is the same shape the R-201 +walks kept finding: the data half passes, the journey half fails.** ## Venue @@ -25,7 +30,11 @@ workaround. `drill-r50` (VM 300) was verified not at risk before proceeding: it ## Findings, ranked by what they cost the person in front of you **1 — stops the visit** -- **R-273** · the vouched agent (0.128.0) has no git tag; every install and reinstall dies at 5/8. +- **R-273** · the vouched agent (0.128.0) had no git tag; every install died at 5/8. **CLOSED** — tag + pushed on your instruction after an independent sha check; install then succeeded in 3 m 49 s. The + two guards that would prevent a recurrence are still owed. +- **R-280** · a reinstalled machine cannot re-attach its data drive through any route, and the restore + page promises „két kattintás" at an empty list. **The one to fix before the tester's visit.** - **R-272** · Felhom's uninstall restarts its own dnsmasq unconstrained; it grabs `:53`; the next install refuses and appears to blame the owner's network. @@ -37,6 +46,10 @@ workaround. `drill-r50` (VM 300) was verified not at risk before proceeding: it in neither the KEPT nor the WIPED list. **3 — misleads** +- **R-281** · the hub said nothing at all through the whole reinstall, and the tripwire for a + sealed-backup unseal did not fire on a real one. +- **R-282 / R-283** · one code, three names; the mail points at a page the box is not showing; the hub + reads "Claimed 18d ago" while the box serves its setup page. - **R-269** · a rotated-out local-API token still authorises until an unrelated lookup forces a reload. The shipped test passes only because of its lookup order. - **R-270** · R-268's own rotation recipe is a step short; the controller never re-reads the mount. @@ -62,14 +75,20 @@ I also raised **R-275**'s sudoers half as a likely privilege-escalation on reins and refuted my own hypothesis**: sudo skips filenames containing dots, so the leftover file is inert. `visudo -c -f` parsing a file OK is not evidence that sudo loads it. -## Integrity verdict +## Integrity verdict — BYTE-IDENTICAL -**Not reached** — P6 never ran. The instrument is proven and waiting: the comparator was watched -**failing** three ways, including an NFC→NFD rename that renders identically to the eye and would -have passed a rendered-text comparison. Four files, two with Hungarian accented names -(`árvíztűrő-tükörfúrógép.txt`, `nested/őszibarack.md`), one 3 MB binary; manifest keyed on raw name -bytes, committed as `evidence-rehearsal-2026-08-09/GATE0-before-manifest.json`; all four confirmed -present in snapshot `41c830db`. +``` +expected 4 file(s); found 4 +VERDICT: BYTE-IDENTICAL +``` + +Four expected, four restored, zero differences, compared against +`evidence-rehearsal-2026-08-09/GATE0-before-manifest.json` — a manifest keyed on **raw name bytes**. +`árvíztűrő-tükörfúrógép.txt` and `nested/őszibarack.md` came back with their name bytes intact (NFC +preserved, `c3a1…`), which is the discriminator the Gate 0 positive control was built to enforce: the +comparator had been watched **failing** on an NFC→NFD rename that renders identically to the eye. +Restored out of snapshot `41c830db` into the verification folder the product names, with live data +untouched. ## Wall clocks @@ -80,8 +99,19 @@ present in snapshot `41c830db`. | P1 uninstall | **60 s** (08:37:23 → 08:38:23 UTC) | | P1 leave-behind measurement | ~12 min | | P2 preflight (3 runs: 2 refusals, 1 pass) | ~6 min | -| **P3 install — FAILED** | **44 s** (08:51:33 → 08:52:17 UTC) | -| bare machine → visible files | **not reached** | +| P3 install — first attempt, FAILED | 44 s (08:51:33 → 08:52:17 UTC) | +| P3 install — resumed, SUCCESS | **3 m 49 s** | +| P4 first contact (box live on its own URL) | within ~4 min of install | +| STOP 3 unlock | **21 s** | +| app redeploy (calibre-web) | **1 m 36 s** | +| restore prepare + execute | **8 s + 13.2 s** | +| **bare machine → verified files** | **1 h 49 m 22 s** (08:38:23 → 10:27:45 UTC) | +| — of which the product's own work | **≈ 7 m 47 s** | + +**Neither figure is the customer number.** The 1 h 49 m is dominated by the R-273 diagnosis and release +fix (~38 min) and two waits on a human. The 7 m 47 s is what the product costs when the operator +already knows every answer. **The honest unaided figure is undefined, because an unaided household does +not finish.** ## Steps taken off-path, and what they cost @@ -96,9 +126,18 @@ present in snapshot `41c830db`. permission check, so the secret still never touched argv. A person would type it. 5. **`systemctl stop dnsmasq && systemctl disable dnsmasq`** — the action the refusal message tells the owner to take, used as the counterfactual that confirmed R-272. +6. **Pushed the `v0.128.0` git tag** — outward-facing, done on your "proceed", and only after an + independent download proved the published package's sha256 equalled the hub's vouched value. It + completes a half-finished release rather than changing code; the release script's own recovery text + is the same line. **Cost to the walk: the install that followed was a `--resume`, not a fresh run, + which is why R-274 is only half-observed.** +7. **`POST /settings/storage/add` with `/mnt/sys_drive`** — the manual escape hatch, typed. This is the + R-280 wall; a customer could not produce that path. **The biggest fidelity cost of the run.** +8. **SSH into the guest to fingerprint the restored tree.** This is my *instrument*, not a customer + step — the customer's step (the restore) finished at the dashboard. Byte-comparison inherently needs + file access; nothing about the product was driven this way. -Everything after the install command returned was read-only. No repair was attempted on the -half-installed box. +Everything else after the install returned was read-only, and no repair was attempted on the box. ## Teardown — all four layers @@ -123,14 +162,22 @@ DB copy were handled file→file at 0600 and shredded; the hub DB copy (which ca break-glass credential) was shredded immediately after the one hash comparison it was taken for. Credential comparisons were done by sha256 prefix, never by value. -## State for whoever resumes +## State demo-hp was left in -- **Blocked on one command** — `git tag v0.128.0 28ba8593b8 && git push origin v0.128.0`. -- **Nothing in Gate 0 needs redoing.** Snapshot `41c830db` holds the dataset; the before-manifest is - committed; the comparator is proven. -- **First thing to record at step 7/8** when the install resumes: which golden it takes (R-274 is the - one finding still owed an end-to-end observation). -- **P4 will need you** — the claim code is emailed to `doodoo21@freemail.hu` and there is no - operator-side route to it (R-119). +**Back in service and healthy** — agent 0.128.0, controller 0.210.0, guest 9201 running and onboot, +claimed, storage path registered, `calibre-web` deployed, off-site repository unlocked and intact at +18 snapshots. `drill-r50` (VM 300) untouched throughout. + +**Deliberately left alone, and named rather than tidied:** the pre-existing `c11-scratch` storage and +the three `vzdump-lxc-9100` golden archives on `local` (the teardown keeps goldens by design, and they +now number three). The restored files sit in the product's verification folder, not back in place — +that is R-213 and the product says so. + +## Still owed + +- **R-280** — the drive wall. The one finding that would stop the tester's visit outright. +- **R-273's two guards** — refuse a vouch whose tag does not resolve; check that a package and its tag + ship together. The tag push fixed one box, not the class. +- **R-274's missing observation** — a *fresh* (non-resume) install taking a stale local golden. Full account: `documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`. diff --git a/STATUS.md b/STATUS.md index 6937ff9..c928f5c 100644 --- a/STATUS.md +++ b/STATUS.md @@ -23,21 +23,42 @@ destroyed on purpose and its files came back byte for byte identical — four ti their recovery code got everything back with **no command line inside the machine at any point**, in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252, R-253 — closed.)* -## Broken right now — and it stops every new machine +## The rehearsal finished. The data came back byte for byte; the journey did not. + +**We wiped a working demo machine and put it back. All four test files returned identical — including +the two with Hungarian accents, checked as raw bytes, not as text on screen.** The unlock took 21 +seconds and the restore 13. **But it only finished because I could open a terminal twice.** A +household would have stopped, twice, and the second time the screen would have told them it was easy. + +**The two walls, both fixed-or-fixable, neither about the data:** + +- **The install died four steps in** — the agent version you approved had been published as a download + but never given its version label, and the installer looks it up by that label. **Now unblocked** — + I pushed the label after checking the published file matched what you vouched. *(R-273 — closed. The + two guards that would stop it recurring are still owed.)* +- **A reinstalled machine cannot re-attach its own data drive.** Every route is a dead end, and the + restore page cheerfully says „**Ez két kattintás**" while pointing at an empty list. The drive is + fine and the machine can see it — it just is not offered, because the same drive is also the backup + target. I got past it by typing an internal path no customer could know. **This is the one to fix + before the tester's visit.** *(R-280)* + +**Also broken, found on the way:** -- **No machine can be set up from scratch today.** The install dies four steps in, on every host, and - it needs one command from you to unblock (below). Nothing already running is affected — the two demo - machines and the tester's box are fine. It is only the *setting-up* path, which is the one his visit - is made of. *(R-273)* -- **A rebuilt machine may come back on software from last week** — the install will happily reuse an - old copy it finds lying on the disk, without checking it against the version you approved. The copy - on the demo machine is old enough to predate the recovery screen. *(R-274)* - **Taking Felhom off a machine leaves the one thing that stops it going back on.** We install a small - network service at setup; removing Felhom restarts it without its settings, it takes over the port - the next install needs, and the next install then refuses — blaming the owner's network. *(R-272)* + network service at setup; removing Felhom restarts it without its settings, it seizes the port the + next install needs, and the next install then refuses — appearing to blame the owner's network. + *(R-272)* - **A machine we removed keeps its private line to us open.** *(R-276)* +- **The hub said nothing at all** while a machine was wiped, rebuilt, re-claimed and had its sealed + backups opened. No false alarm — but also no word, and the alarm that exists for "someone is opening + this customer's backups" stayed silent through a real one. *(R-281)* +- **A rebuilt machine may still come back on software from last week** — narrower than I first wrote: + the resumed install fetched the right version, but a fresh one takes whatever copy is newest on the + disk without checking it against what you approved. *(R-274)* - **demo-felhom has not had an off-site backup in six days** and is waiting for a recovery code nobody has entered. demo-hp, rebuilt the same day, recovered by itself. *(R-278)* +- **One code, three different names**, and the email points at a page the machine is not showing — + this cost us a wasted code today. *(R-282, R-283)* ## What's broken @@ -51,29 +72,26 @@ in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252 accumulates. *(R-244)* - **Putting restored files back where they belong is still manual.** *(R-213)* -## The reinstall rehearsal — what one afternoon on one machine found +## The rest of what the rehearsal found -**We took a working demo machine, removed Felhom, and tried to put it back the way a person with their -own hardware would.** It got three steps in. That is the finding: **nine defects, two of which stop a -visit dead**, and none of them were visible from reading the code — the last three sessions of review -did not see any of them. +Sixteen findings in one afternoon, **none of them visible from reading the code** — three sessions of +review had not seen any. -- It stopped at the install, on a missing version label *(R-273)* — and the step before that had - already refused, because removing Felhom leaves behind the thing that blocks reinstalling it - *(R-272)*. -- Removing Felhom also leaves five files holding old keys *(R-275)* and a live private line to us - *(R-276)*. -- Along the way: rotating a leaked key does not actually revoke the old one until the service is - restarted *(R-269)*, the written-down recipe for that rotation is missing a step *(R-270)*, and the - alarm it raises can never be closed because the fix it recommends is what silences the all-clear - *(R-271)*. -- **It also caught me being wrong.** Three separate hub screens agreed that demo-hp had no off-site - backup. It had eighteen. I told you the fleet's backups were down and had to take it back an hour - later — the screens were misleading, but I should have asked the machine before telling you. *(R-277)* +- Removing Felhom leaves five files holding old keys *(R-275)*, and rotating a leaked key does not + revoke the old one until the service restarts *(R-269)* — the written recipe for it is a step short + *(R-270)*, and the alarm it raises can never be closed because the fix it recommends is what + silences the all-clear *(R-271)*. +- **It caught me being wrong twice, and that matters more than the count.** I told you the fleet's + off-site backups were down; demo-hp had eighteen snapshots and I had read three misleading screens + instead of asking the machine *(R-277)*. And I raised a leftover permissions file as a security + hole, then tested it and refuted myself — it is inert. -**The data was never at risk** and is not now: the test files sit in an off-site snapshot with their -fingerprints recorded, plus two local copies. Full account: -`documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`. +**What worked, and should not be lost in the count:** the machine came up on its own at the approved +version; the setup page appeared unprompted, in Hungarian, naming the customer; **the recovery screen +appeared without being looked for** and said plainly that unlocking changes nothing; the restore told +the truth about putting files in a checking folder rather than back in place; and no false alarm fired. + +Full account: `documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`. ## Three rulings, written down so they stop living in a conversation @@ -111,17 +129,14 @@ only miss a true alarm, never raise a false one *(R-266)*. ## Waiting on you -- **One command, and it unblocks every install.** The version you approved yesterday was published as - a download but never given its version label in the source repository, and the installer looks it up - by that label. The code is right and already shipped; only the label is missing. - `cd felhom-agent && git tag v0.128.0 28ba8593b8 && git push origin v0.128.0`. I did not do it - myself: publishing a release label is not a thing to do quietly in the middle of a measurement. - *(R-273)* -- **Then say go, and the rehearsal resumes** where it stopped. Nothing has to be redone — the test - files are already in an off-site backup and their fingerprints are recorded. - -*(Both items previously here are done: the approval is saved — golden 0.210.0, agent 0.128.0, minimum -0.127.0 — and the recovery fix was proved on hardware.)* +- **Nothing blocking.** The rehearsal is finished and demo-hp is back in service: agent 0.128.0, + controller 0.210.0, claimed, off-site backups unlocked and intact. +- **One decision worth taking before the tester comes:** whether to fix the drive wall *(R-280)* now. + It is the only finding that would stop his visit outright, and it is the difference between "his + data comes back" and "his data comes back if someone types a path for him." +- **Two guards are still owed** so the install cannot break the same way twice: refuse to vouch a + version whose label does not resolve, and check that a published version and its label ship + together. *(R-273's tail.)* ## DooPlex infrastructure — separate from the product diff --git a/documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md b/documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md index 2606cd9..a6324c5 100644 --- a/documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md +++ b/documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md @@ -1,8 +1,9 @@ # REHEARSAL — the BYO reinstall walk (2026-08-09) -> **Status: IN PROGRESS.** Pre-phase and Gate 0 are complete; STOP 1 passed. The walk (P1–P7) has not -> started. This file is written before the destructive phase deliberately — a finding that exists only -> in a session that later crashes is a finding nobody has. +> **Status: COMPLETE.** All phases walked; the integrity verdict is **BYTE-IDENTICAL**. §1–§8 were +> written before the destructive phase deliberately — a finding that exists only in a session that +> later crashes is a finding nobody has — and are left as written, including one claim later refuted +> by test (§8.3 → §9a) and one by measurement (§7.2 F-9). **Venue: `demo-hp` (HP t740, `felhom-host`, guest 9201, customer `demo-hp`).** Operator-approved at STOP 1. **Driven from DooPlex.** All times UTC unless marked; the host runs CEST (UTC+2). @@ -512,14 +513,200 @@ which golden it takes.** --- -## 10. Where the run stands +## 9a. P3 resumed — and the install succeeded -**Reached: P1 ✓, P2 ✓, P3 ✗ (blocked).** P4–P7 not attempted. +**Unblocked on operator instruction ("proceed").** `v0.128.0` was annotated at `28ba8593b8` and pushed +after an **independent download** confirmed the published package's sha256 equals the hub's vouched +`c6eba73b…`. Both config URLs then served 200. `--resume` completed in **3 m 49 s** +(09:30:30 → 09:34:19 UTC): `Day-0 provision SUCCESS`, controller **0.210.0**, agent **0.128.0**, guest +9201 onboot, pool member, all 16 ACL assertions green, controller healthy in ~18 s. -**The honest answer to §2's question is: NOT YET ANSWERABLE, and the reason is itself the answer for -today.** A machine wiped today cannot be set up again at all — not because the data is gone (it is -safe, in snapshot `41c830db` and in two local tiers), but because the install refuses at step 5 of 8 on -a missing git tag. The walk found a hard stop before it ever reached the question about data. +**R-274 partly refuted, and recorded as such.** Step 7 fetched the **vouched 0.210.0** — because +`--resume` skips preflight, and preflight is where local auto-discovery sets `GOLDEN_VOLID`. So the +fresh and resume paths disagree, and the resume path is the safe one. What survives: discovery is +`sort | tail -1` (newest local) with **no manifest comparison**, so a box whose newest local golden +predates the vouched one still installs stale. Full text in the register. -**The dataset is intact and waiting.** Nothing about Gate 0 needs redoing when the walk resumes: -snapshot `41c830db` holds all four files, the before-manifest is committed, and the comparator is proven. +## 10. P4 — first contact, as the customer + +**A clean pass, and worth saying so plainly.** Within four minutes of the install finishing, the +customer's own URL — fetched from outside the box, over the public internet — served: + +> **A szerver beállítása** · Demo HP +> *Add meg az e-mailben kapott beállító kódot, majd válassz saját jelszót a vezérlőpult védelméhez.* +> Beállító kód · Új jelszó (min. 12 karakter) · Új jelszó megerősítése +> *Nem kaptad meg a kódot? Új kód kérése* + +Unprompted, in Hungarian, naming the customer, with a self-service route if the code never arrived, +and nothing anywhere asking for a command line. The hub showed the host **ONLINE** at agent 0.128.0. + +**Two findings here, neither fatal:** the hub still read *"Claimed 18d ago"* while the box was serving +its first-run page (**R-283**), and the code that arrives is named three different things across the +three surfaces, with the mail pointing at an „Elfelejtett jelszó" page the box does not show +(**R-282**). The reset code was nonetheless **accepted on the setup page** — 302 and a session — so +this is naming, not function. + +## 11. P5 — getting the machine back to work + +### 11.1 The recovery screen, unsought — the headline pass + +The **first thing** on the dashboard after claiming, with nothing sought: + +> **Adatok visszaszerzése** — *Ezt a gépet újratelepítették. A korábbi, házon kívüli mentéseid +> megvannak* — a sealed package held centrally, **sealed at 2026-08-04T11:11:37Z**, openable only with +> the customer's code; *nobody can replace it — not Felhom, not support, not the operator*; and +> **„Ebben a lépésben semmit nem állítunk vissza és semmi nem változik."** +> Field: **Helyreállítási kód (tíz szó)**. + +R-193's screen meeting reality on a genuinely rebuilt box. It answered all three of its questions and +its seal date matches `host_escrow.created_at` exactly. + +*(Nit: the seal date is rendered raw as `2026-08-04T11:11:37Z` to a Hungarian household — ISO-8601 in +UTC where a localised date belongs.)* + +### 11.2 STOP 3 — the unlock + +Entered at the box's own screen, from a file, never on a command line. **21 seconds**, and it listed +what it found without restoring anything: + +| app | legutóbbi mentés | méret | +|---|---|---| +| **calibre-web** | 2026-08-09 10:30 (CEST) | **3.8 MB** | +| felhom-offbox | ″ | 6.8 KB | +| opengist | ″ | 182.3 KB | +| privatebin | ″ | 6.8 KB | + +That is exactly the Gate 0 snapshot set, seen from the customer's side. + +### 11.3 The wall — R-280, and it is rank 1 + +The restore page diagnoses the situation perfectly and then sends the customer to an empty page: + +> *„Előbb csatold vissza az adatmeghajtót. … **Ez két kattintás:** Tárhely → Meghajtók, »Meglévő +> meghajtó csatolása«. Utána gyere vissza ide."* + +**It is not two clicks; it is zero possible clicks.** `GET /api/disks/candidates` → +`{"initialize":[],"attach":[]}`. The agent is fine — `GET /api/disks` returns the NVMe in full — but +`handleDiskCandidates` builds *both* lists from the **unclaimed-disk** scan, and demo-hp's NVMe is +deliberately both the user-data drive and the `felhom-backup` target, so it is claimed and never +offered. Correct for `initialize`; over-broad for `attach`, which is non-destructive by definition. + +It cascades: no store → Calibre-Web's install page degrades to *„Nincs regisztrált adattároló — adja +meg kézzel az útvonalat"*; no app → every restore row reads „Nincs telepítve". + +**Escape hatch used, and recorded as off-path:** `POST /settings/storage/add` with +`storage_path=/mnt/sys_drive` succeeded first try — an internal path, the very one registered before +the wipe, that no household customer could produce. Everything unblocked immediately afterwards and +the deploy form became a proper picker („Tárhely (sys_drive) — 64.2 GB szabad"). + +### 11.4 App redeploy + +`calibre-web` deployed from the catalogue through the dashboard's own API in **1 m 36 s**, running and +healthy. The catalogue itself was complete and in Hungarian, every app „Nincs telepítve". + +*(Also observed: the deploy page warned „A kiválasztott tárhely majdnem megtelt." over an option +carrying `data-free-percent="93"` and „64.2 GB szabad" — an apparent inverted threshold, **R-284**.)* + +## 12. P6 — getting the data back + +Three options, each explained, none starting without a question. **R-204 item 3 is visibly fixed:** +option 1 states in as many words that it does *not* bring the customer's own files back. + +| step | wall clock | +|---|---| +| full-restore **prepare** (3.8 MB) | **8 s** | +| full-restore **execute** | **13.2 s** (10:27:32.08 → 10:27:45.29 UTC) | + +> *„A(z) calibre-web teljes mentése visszaállítva ellenőrző mappába: +> `/mnt/sys_drive/felhom-data/backups/offsite-restore/calibre-web` — a saját fájljaiddal együtt. A +> meglévő adatok változatlanok."* + +Restored to a **verification folder**, not into place — honest about it, and R-213's territory. + +### THE INTEGRITY VERDICT — **BYTE-IDENTICAL** + +``` +expected 4 file(s); found 4 +VERDICT: BYTE-IDENTICAL +``` + +Compared against `GATE0-before-manifest.json`, keyed on **raw name bytes**: + +| sha256 | bytes | name | +|---|---|---| +| `54b773c46bbfd994…` | 3 145 728 | `binary-3mb.bin` | +| `52a5c5ebfcac247f…` | 59 | **`árvíztűrő-tükörfúrógép.txt`** — name bytes `c3a1…` identical, NFC preserved | +| `15d2024dfc224162…` | 25 | **`nested/őszibarack.md`** — name bytes identical | +| `924497918e55fe6d…` | 21 | `plain.txt` | + +Four expected, four restored, zero differences. The accented filenames survived as **bytes**, not +merely as rendered text — the discriminator the Gate 0 positive control was built to enforce. + +## 13. P7 — what the hub thought was happening + +**It thought nothing.** Across the whole reinstall the hub recorded **zero events and zero +notifications** for demo-hp. Positive control (standing rule 3): the hub logged **2 events all day +across all customers**, newest `db_dump_completed` at 00:30:07 — the store is reachable and the +silence is real. + +- **The good half:** no false alarm fired during a legitimate reinstall, which is what P7 watches for. +- **The owed half (R-281):** `escrow_blob_served` exists as the tripwire for exactly this moment — + *"If no recovery is in progress on that box, investigate"* — and **has fired for demo-hp before** + (last 2026-08-04 20:12:54). Today's real unseal fired it not at all. A reinstall and a stolen + machine look identical to the operator. + +## 14. Teardown — all four layers + +1. **The machine** — my instruments (`fp.py`, `restored.json`, the installer, the passphrase file) + removed. Guest 9201 running and healthy. **`drill-r50` (VM 300) untouched throughout**, as predicted. +2. **The host** — `local-lvm` 36.97 % → 21.85 % (bare) → **32.35 %** (rebuilt). Pre-existing leftovers + found and deliberately **not** removed, recorded instead: `c11-scratch`, the orphaned + `vzdump-lxc-9100` archives (now three). +3. **The hub** — **no customer or appliance record created**; `demo-hp` retained deliberately per the + runbook. Nothing to delete. +4. **The off-site side** — **18 snapshots, newest still `9e38b84c` / `78b93f04` at 08:30 UTC**, i.e. + unchanged since Gate 0. The restore was a pure read. **No prune, no forget, no delete by me**; the + only retention that ran was inside the product's own backup call at Gate 0, which reported + `18 snapshot(s)` itself. + +## 15. The answer to §2's question + +**Yes — the data comes back, byte for byte. No — not in one sitting, and not without a shell.** + +The walk completed: **P1 ✓ P2 ✓ P3 ✓ P4 ✓ P5 ✓ P6 ✓ P7 ✓**, and the verdict is BYTE-IDENTICAL. +But it completed only because two hard stops were cleared by someone who could open a terminal and +read source code: + +1. **R-273** — the install died at 5/8 on a git tag that was never pushed. Cleared by a release action. +2. **R-280** — the data drive could not be re-attached through any dashboard route, while the page + promised „két kattintás". Cleared by POSTing an internal path a customer could not know. + +Neither is a data-integrity problem. Both are **journey** problems, and both stop a household dead. +This is the same shape the R-201 walks kept finding: **the data half passes, the journey half fails.** + +**What the product did beautifully, and should not be lost in the finding count:** the box came up on +its own at the vouched version; the setup page appeared unprompted, in Hungarian, naming the customer; +the recovery screen appeared **without being sought** and stated plainly that nothing would be changed +by unlocking; the unlock took **21 s**; the restore took **13.2 s** and said honestly that it had put +the files in a verification folder rather than back in place. Sixteen ACL assertions verified +themselves. No false alarm fired. + +### Wall clocks + +| segment | | +|---|---| +| P1 uninstall | **60 s** | +| P2 preflight (2 refusals, then pass) | ~6 min | +| P3 install — first attempt, FAILED | 44 s | +| P3 install — resumed, SUCCESS | **3 m 49 s** | +| P4 first contact (box live on its own URL) | within ~4 min of install | +| STOP 3 unlock | **21 s** | +| app redeploy (calibre-web) | **1 m 36 s** | +| restore prepare + execute | **8 s + 13.2 s** | +| **bare machine → verified files** | **1 h 49 m 22 s** (08:38:23 → 10:27:45 UTC) | +| — of which the product's own work | **≈ 7 m 47 s** | + +**The 1 h 49 m must not be quoted as the customer number** — it is dominated by the R-273 diagnosis and +release fix (~38 min) and by two waits on a human. **The ≈ 7 m 47 s must not be quoted either**: it is +what the product costs when someone already knows every answer. The honest figure for an unaided +household is **undefined, because an unaided household does not finish.** diff --git a/documentation/audits/evidence-rehearsal-2026-08-09/P2-preflight-refusals.txt b/documentation/audits/evidence-rehearsal-2026-08-09/P2-preflight-refusals.txt new file mode 100644 index 0000000..0b36ccd --- /dev/null +++ b/documentation/audits/evidence-rehearsal-2026-08-09/P2-preflight-refusals.txt @@ -0,0 +1,40 @@ + +[INFO] felhom-host-install v1.25.0 — mode=byo customer=demo-hp vmid=9201 + +[STEP] 1/8 pre-flight +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) +[INFO] node: felhom-host (auto) +[INFO] agent config: /etc/felhom-agent/agent.json +[INFO] agent: not installed yet — will be fetched + installed in step 5/8 +[INFO] local-lvm free: ~42 GiB +[WARN] local-lvm free ~42 GiB < hard min 120 GiB +[INFO] free RAM: ~26535 MiB +[INFO] existing guests on this host: 1 (pct+qm) +[INFO] archive storage 'local' present +[INFO] acl storages all present or pre-positioned: local local-lvm felhom-pbs +[ERROR] a resolver is already bound to :53 on this host: + udp UNCONN 0 0 0.0.0.0:53 0.0.0.0:* users:(("dnsmasq",pid=1291306,fd=4)) udp UNCONN 0 0 [::]:53 [::]:* users:(("dnsmasq",pid=1291306,fd=6)) tcp LISTEN 0 32 0.0.0.0:53 0.0.0.0:* users:(("dnsmasq",pid=1 +[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN. + Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run. + (Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.) +[ERROR] PRE-FLIGHT FAIL (exit 1) — fix the finding above and re-run +Connection to 100.76.96.79 closed. diff --git a/documentation/audits/evidence-rehearsal-2026-08-09/P3-install-resume.txt b/documentation/audits/evidence-rehearsal-2026-08-09/P3-install-resume.txt new file mode 100644 index 0000000..6a0b159 --- /dev/null +++ b/documentation/audits/evidence-rehearsal-2026-08-09/P3-install-resume.txt @@ -0,0 +1,835 @@ +felhom-host + +[INFO] felhom-host-install v1.25.0 — mode=byo customer=demo-hp vmid=9201 + +[SKIP] pre-flight (resumed) +[STEP] 2/8 Proxmox API token +[SKIP] pool felhom already exists +[INFO] user felhom-agent@pve exists +[INFO] removing stale token agent (secret unrecoverable — rotating) +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +user config - ignore invalid acl token 'felhom-agent@pve!agent' +[OK] token minted (secret captured, not logged) +[INFO] role FelhomAgentBase exists — ensuring exact privileges +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +[INFO] role FelhomAgentGuest exists — ensuring exact privileges +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +[INFO] role FelhomAgentStore exists — ensuring exact privileges +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs]) +[SKIP] old broad role FelhomAgent already absent +[STEP] 3/8 compute volume grows +[INFO] auto-computed from ~42 GiB free (ONE volume since R-165) +[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume) +[STEP] 4/8 host enrollment (POST /host-enroll) +[OK] host REUSED (idempotent — existing credential) +[INFO] host_id: demo-hp-bb76ea (api_key captured, not logged) +[SKIP] break-glass (step 4b) SKIPPED (byo: the host owner manages break-glass; root@pam untouched, nothing vaulted) +[STEP] 5/8 agent install (fetch + verify + install) +[SKIP] apt repo alignment skipped (byo — the owner manages repos) +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.210.0 +[INFO] fetching agent binary v0.128.0 from Gitea … +[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest +[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0) +[INFO] service user felhom-agent exists +[INFO] felhom-agent already in systemd-journal +[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path) +[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path) +[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path) +[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path) +[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated) +[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config) +[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed) +[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer) +[SKIP] OOB (felhom-sshd) off (byo, or appliance --no-oob) — skipping +[STEP] 6/8 agent config + service +[SKIP] backup target 'felhom-backup' already exists — leaving its DEFINITION exactly as it is (Scenario F) +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +felhom-backup-target-apply: granted FelhomAgentStore on /storage/felhom-backup (user + token) +[INFO] node=felhom-host local_api=192.168.0.87:8443 tls_fp=E5:6D:79:3B:67:A1… +[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent) +[OK] byo config asserts passed (lan_resolver.enable / oob.enabled off; deployment_mode=byo; wg_tunnel intentionally ON — base infrastructure) +[OK] agent --selftest (read-only) passed +[OK] felhom-agent service active (non-root felhom-agent reads the config OK) +[STEP] 7/8 golden archive +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] fetching golden v0.210.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst +[OK] verified sha256 b9f701fab813c051… matches the hub manifest +[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst +[STEP] 8/8 provision guest 9201 +[SKIP] pool felhom already exists +=== felhom-agent 0.128.0 selftest=provision (vmid=9201 customer=demo-hp hostname=demo-hp) === + --- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst → vmid 9201 --- +time=2026-08-09T11:33:34.336+02:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom + [OK] front half: vmid 9201 up (boot+running) in 1m9s; MAC=BC:24:11:65:7F:69 + --- back half: mint per-guest token + populate bootstrap config mount --- +time=2026-08-09T11:33:38.654+02:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=192.168.0.87:8443 + [OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap) + local-api endpoint 192.168.0.87:8443 · leaf fp e3c7aaafbe7643d35eb5a09d2c3fe5128f1ef626010e7c17060dd55a5cda37d5 · token: minted (not printed) +=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) === + next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller, + which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api. +[OK] provision completed +[INFO] rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +[STEP] verify +[OK] pct status: running +[OK] onboot: 1 + mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G + mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives + rootfs: local-lvm:vm-9201-disk-0,size=32G +[OK] pool: guest 9201 is a member of felhom +[OK] acl: FelhomAgentBase@/ present (user+token) +[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token) +[OK] acl: FelhomAgentStore@/storage/local present (user+token) +[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token) +[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token) +[OK] authz signers: 2 (operator-signed self-update armed) +[OK] controller: Up 18 seconds (healthy) (after ~0s) +[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.210.0 +[INFO] cloudflared: Up 18 seconds +[INFO] (confirm in the hub UI that host demo-hp-bb76ea reports guest 9201) + +[OK] Day-0 provision SUCCESS — vmid=9201 host_id=demo-hp-bb76ea customer=demo-hp golden=local:backup/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst +Connection to 100.76.96.79 closed. diff --git a/documentation/audits/evidence-rehearsal-2026-08-09/P6-restored-manifest.json b/documentation/audits/evidence-rehearsal-2026-08-09/P6-restored-manifest.json new file mode 100644 index 0000000..2476b47 --- /dev/null +++ b/documentation/audits/evidence-rehearsal-2026-08-09/P6-restored-manifest.json @@ -0,0 +1,26 @@ +{ + "count": 4, + "files": { + "62696e6172792d336d622e62696e": { + "name_utf8": "binary-3mb.bin", + "sha256": "54b773c46bbfd99465b68247bcb4d3cf5bf8f57a4024c6ba169dc0c326979dd2", + "size": 3145728 + }, + "6e65737465642fc591737a6962617261636b2e6d64": { + "name_utf8": "nested/őszibarack.md", + "sha256": "15d2024dfc22416261c56ed51741f8d7ae734350b3ac79ef1cb982d667ddef64", + "size": 25 + }, + "706c61696e2e747874": { + "name_utf8": "plain.txt", + "sha256": "924497918e55fe6dd75ff5ed8a9089aa64752e05540bdf804d71a03afbc48acd", + "size": 21 + }, + "c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874": { + "name_utf8": "árvíztűrő-tükörfúrógép.txt", + "sha256": "52a5c5ebfcac247f1753e86f69db46815adf4edaf28375396324e4ea6163611c", + "size": 59 + } + }, + "root": "/var/lib/felhom/sys_drive/felhom-data/backups/offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09" +} \ No newline at end of file diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index cc20393..8a12ea8 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -501,13 +501,18 @@ applied.** The one that matters: Scenario A **fails against today's tree** with | **R-270** | **R-268's stated rotation recipe is incomplete: the controller never re-reads `bootstrap.json`'s `local_api`, so a rotation leaves the agent channel dead across restarts.** `bootstrap.ensureLocalAPI` returns early when `cfg.LocalAPI.Endpoint != ""` — by design it FILLS an absent block and never refreshes a present one — so the token the controller uses lives in its own `controller.yaml`, not in the mount. Proved live 2026-08-09: two controller restarts after a correct `bootstrap.json` rotation, still `HTTP 401`; the channel came up only once `local_api.token` was written into `controller.yaml`. The neighbouring `DetectEndpointDrift` compares the ENDPOINT and deliberately does not compare the token (*"a token mismatch is a different failure"*), so this shape is knowingly unmodelled. Parent question — which file is authoritative — is **R-78** | **READY (S) — NEW 2026-08-09** | — | Either teach the drift detector the token, or make the rotation path write both files. Correct the R-268 row's recipe either way | CC | | **R-271** | **The `agent_channel_unauthorized` alarm can never be closed, because its own prescribed remedy is what silences the recovery.** `channelhealth.Checker.Check`'s UP branch notifies only when `prev != "" && prev != "up"`; a controller restart resets `state` to `""`, so an unseeded→up transition is silent by construction. The alert text says *"token stale/rotated (**re-bootstrap**)"* — i.e. restart the controller — so **following the instruction guarantees no recovery event.** Observed live 2026-08-09: two `agent_channel_unauthorized` errors on the hub (one `sent`, one `suppressed` by the 1 h operator cooldown) and **nothing afterwards**, though the channel came up 3 minutes later and stayed up. The down side is deliberately asymmetric (F2: a born-down channel alerts on cycle 1); the up side never got the matching treatment. Customer dashboard is fine — `SetDashboard` reflects current state every cycle. It is the OPERATOR's trail that ends on "down" | **READY (S) — NEW 2026-08-09** | — | Notify on unseeded→up when the previous *persisted* state was down, or seed from the hub's last event | CC | | **R-272** | **RANK 1 — Felhom's `--uninstall` leaves the exact condition that makes Felhom's own reinstall REFUSE.** Chain, fully evidenced on demo-hp 2026-08-09: Felhom installs `dnsmasq` at day-0 (`/var/lib/dpkg/info/dnsmasq.list` dated **2026-07-21 18:24 CEST**, demo-hp's day-0) and constrains it with a snippet in `/etc/dnsmasq.d/`; `--uninstall` removes the snippet and **restarts the daemon** (running process start time **2026-08-09 10:37:39 CEST — inside the 10:37:23–10:38:23 uninstall window**) but leaves the package installed and the unit **enabled**; unconstrained, dnsmasq binds `0.0.0.0:53`; the next install's preflight then hard-refuses with *"a resolver is already bound to :53"*. It is **not** PVE SDN's (`/etc/pve/sdn/` empty; stock unit). The teardown mentions it only as *"the 'sudo' and 'dnsmasq' packages were left installed (**system packages**)"* — **dnsmasq is not a system package here, Felhom installed it.** **What a customer does next:** reads a message blaming a resolver, concludes their own LAN DNS is at fault, and debugs something they never configured. **Counterfactual confirmed:** `systemctl stop dnsmasq && systemctl disable dnsmasq` → `host DNS (:53): free` → PRE-FLIGHT PASS, nothing else changed. **The refusal MESSAGE is good** (finding, evidence, two routes, and an explicit promise not to touch DNS on a host it does not own) — the defect is that Felhom caused the condition and does not say so | **READY (M) — NEW 2026-08-09** | — | Either stop+disable dnsmasq on uninstall when Felhom installed it, or have the preflight recognise its own leftover and say so | CC | -| **R-273** | **RANK 1 — the hub vouched an agent version that was never git-tagged, and every install fleet-wide now fails at step 5/8.** `v0.128.0` exists as a Gitea **package** (so it is selectable in the artifact manifest and its sha256 verifies) but **not as a git tag** — newest tag is `v0.127.0`. Since R-183, `felhom-host-install.sh` correctly pins config fetches to `raw/tag/v$ART_AGENT_VER`, so the install dies: `curl: (22) … 404` → `raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)`. Measured: `raw/branch/main/…` → **200**, `raw/tag/v0.127.0/…` → **200**, `raw/tag/v0.128.0/…` → **404**. The commit that IS 0.128.0 is on `main` (`28ba8593b8`); only the tag is missing. **THIS IS R-184 ARRIVING** — that row says nothing prevents the hub vouching a never-released version because the R-115 gate works from tags. **A good fix met an open gap:** `hostinstall_gates.py` asserts the pin EXISTS (`ok: fetch_raw pins the agent configs to the vouched agent version`); nothing asserts the pinned ref RESOLVES. And `felhom-agent/scripts/release-agent.sh:23` already documents the hazard verbatim — *"a released version without a git tag 404s a box mid-install, as root"* — so it was understood and the step was still missed. **Blast radius, precise:** every fresh install and every reinstall is dead while the manifest vouches 0.128.0 (approved 2026-08-09 — the approval armed it); **existing boxes are unaffected** (self-update takes the binary from the package registry, and demo-felhom runs 0.128.0 healthily). **Unblock:** `git tag v0.128.0 28ba8593b8 && git push origin v0.128.0` — publishes a tag for a commit already on `main` and already released as a package. Rolling the manifest back to 0.127.0 instead would cost R-221. **Deliberately NOT done in the rehearsal session** (diagnose-not-fix; publishing a tag is outward-facing). **Owed gates:** (a) refuse a vouch whose tag does not resolve, (b) a release-completeness check that the package and the tag ship together | **WAITING-ON-OPERATOR** — one tag push | — | `audits/REHEARSAL-byo-reinstall-2026-08-09.md` §9 | Viktor | -| **R-274** | **A local golden is adopted with NO version and NO checksum check, so a reinstall can silently come up releases behind.** `felhom-host-install.sh` step 7: `if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then log_skip "using local golden"; return 0; fi` — **the hub manifest's `golden.sha256`, whose entire purpose is to vouch from a different trust root than Gitea, is consulted only on the FETCH path.** A locally-present archive bypasses the vouch: no version compare, no digest, no warning. On demo-hp 2026-08-09 the preflight selected `local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst`, whose baked marker reads `felhom-controller:`**`0.192.0`**, against a vouched golden of **0.210.0** — 18 releases stale. **The sharp consequence:** 0.192.0 is **below 0.200.0, where R-193's off-site recovery SCREEN shipped**, so a customer reinstalled today returns on a controller that cannot run the recovery ceremony their data depends on; it is also born below the managed-update floor (0.200.0), and the updater's auto-target is the floor, never the newest. **It compounds with the teardown**, which deliberately keeps the old golden (*"golden vzdump left in place"*). This is the R-111/R-115/R-120 drift family one layer down: the R-120 gate guards what may be VOUCHED, nothing guards what an install TAKES. **NOT YET OBSERVED END-TO-END** — step 7 never ran because R-273 killed step 5, so this rests on the code path + the preflight line + the archive marker. **First thing to record when the walk resumes** | **READY (S) — NEW 2026-08-09** | R-273 | Compare the local golden's version/sha against the manifest and refuse or re-fetch on mismatch; say so in the BYO disclosure, which today lists only what the install CREATES, never what it REUSES | CC | +| **R-273** | **RANK 1 — the hub vouched an agent version that was never git-tagged, and every install fleet-wide now fails at step 5/8.** `v0.128.0` exists as a Gitea **package** (so it is selectable in the artifact manifest and its sha256 verifies) but **not as a git tag** — newest tag is `v0.127.0`. Since R-183, `felhom-host-install.sh` correctly pins config fetches to `raw/tag/v$ART_AGENT_VER`, so the install dies: `curl: (22) … 404` → `raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)`. Measured: `raw/branch/main/…` → **200**, `raw/tag/v0.127.0/…` → **200**, `raw/tag/v0.128.0/…` → **404**. The commit that IS 0.128.0 is on `main` (`28ba8593b8`); only the tag is missing. **THIS IS R-184 ARRIVING** — that row says nothing prevents the hub vouching a never-released version because the R-115 gate works from tags. **A good fix met an open gap:** `hostinstall_gates.py` asserts the pin EXISTS (`ok: fetch_raw pins the agent configs to the vouched agent version`); nothing asserts the pinned ref RESOLVES. And `felhom-agent/scripts/release-agent.sh:23` already documents the hazard verbatim — *"a released version without a git tag 404s a box mid-install, as root"* — so it was understood and the step was still missed. **Blast radius, precise:** every fresh install and every reinstall is dead while the manifest vouches 0.128.0 (approved 2026-08-09 — the approval armed it); **existing boxes are unaffected** (self-update takes the binary from the package registry, and demo-felhom runs 0.128.0 healthily). **Unblock:** `git tag v0.128.0 28ba8593b8 && git push origin v0.128.0` — publishes a tag for a commit already on `main` and already released as a package. Rolling the manifest back to 0.127.0 instead would cost R-221. **Deliberately NOT done in the rehearsal session** (diagnose-not-fix; publishing a tag is outward-facing). **Owed gates:** (a) refuse a vouch whose tag does not resolve, (b) a release-completeness check that the package and the tag ship together | **CLOSED 2026-08-09 — tag pushed, install PROVEN** | — | On operator instruction ("proceed"), `v0.128.0` was annotated at `28ba8593b8` and pushed, after an INDEPENDENT download confirmed the published package's sha256 equals the hub's vouched `c6eba73b…`. `raw/tag/v0.128.0/configs/{felhom-agent.service,felhom-mkfs-guarded.sh}` then both served **200**, and `--resume` carried the install to `Day-0 provision SUCCESS` in **3m49s**. **THE TWO GATES ARE STILL OWED AND ARE THE REAL FIX** — this close records that one box was unblocked, not that the class is. | CC | +| **R-274** | **A local golden is adopted with NO version and NO checksum check, so a reinstall can silently come up releases behind.** `felhom-host-install.sh` step 7: `if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then log_skip "using local golden"; return 0; fi` — **the hub manifest's `golden.sha256`, whose entire purpose is to vouch from a different trust root than Gitea, is consulted only on the FETCH path.** A locally-present archive bypasses the vouch: no version compare, no digest, no warning. On demo-hp 2026-08-09 the preflight selected `local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst`, whose baked marker reads `felhom-controller:`**`0.192.0`**, against a vouched golden of **0.210.0** — 18 releases stale. **The sharp consequence:** 0.192.0 is **below 0.200.0, where R-193's off-site recovery SCREEN shipped**, so a customer reinstalled today returns on a controller that cannot run the recovery ceremony their data depends on; it is also born below the managed-update floor (0.200.0), and the updater's auto-target is the floor, never the newest. **It compounds with the teardown**, which deliberately keeps the old golden (*"golden vzdump left in place"*). This is the R-111/R-115/R-120 drift family one layer down: the R-120 gate guards what may be VOUCHED, nothing guards what an install TAKES. **OBSERVED 2026-08-09, AND THE RESULT NARROWS THE ROW — recorded because it partly refutes what was written above.** On the RESUME path step 7 **fetched the vouched 0.210.0 correctly** (`fetching golden v0.210.0 from Gitea`), because `--resume` skips preflight and preflight is where local auto-discovery sets `GOLDEN_VOLID` (the GL6-F4 comment says so). **So the fresh-install and resume paths disagree on golden selection, and the resume path is the safe one.** Discovery is `… | sort | tail -1`, i.e. the NEWEST local archive by filename — a sensible heuristic, **and still no comparison against the manifest's version or sha**. The defect therefore stands as: *a box whose newest local golden predates the vouched one installs stale, silently* — which is exactly the state demo-hp was in before this run (newest local 0.192.0 vs vouched 0.210.0). It is now masked on this box because the freshly fetched 0.210.0 is the newest — **correct by recency, not by verification**. There are now **three** goldens on `local` (07-21, 08-03, 08-09), because the teardown keeps them. **Still not observed: a FRESH (non-resume) install taking a stale local golden.** | **READY (S) — NEW 2026-08-09, NARROWED same day** | — | Compare the local golden's version/sha against the manifest and refuse or re-fetch on mismatch; say so in the BYO disclosure, which today lists only what the install CREATES, never what it REUSES | CC | | **R-275** | **`--uninstall` leaves five 0600 `agent.json.*` credential backups, and the reinstall hands them to the new service account.** `/etc/felhom-agent/` survives with `agent.json.{campaign8-before,campaign9-before,campaign9-prev,pre-e-target-move,pre-prunegate.bak}`, each carrying a 64-char `hub.api_key` and a 59-char `proxmox.token`. The teardown claims to remove *"config (+ its .bak backups)"* and `scripts/CHANGELOG` F1 records *"uninstall now purges the agent config's `.bak*` siblings (one held a live hub api_key)"* — **that fix does not match the filenames in use, and it misses `agent.json.pre-prunegate.bak`, a file that literally ends in `.bak`.** **Exposure assessed, not assumed:** these are SUPERSEDED — the orphaned key hashes to `a5d2222a…`, the hub's current demo-hp key to `8c59d1b6…`, and the Proxmox token was deleted by the same uninstall. **But the reinstall recreates `felhom-agent` at uid 999, the same uid the deleted account had**, so three of the backups become the new account's files — verified readable as `felhom-agent`. A fresh install's service account inherits read access to the prior install's credentials; superseded today, live if the backups were recent (R-179's precedent). **Also left, undeclared:** `/etc/felhom/{.bootstrap-done,appliance-pairing-code}`, `felhom-bootstrap.service` + `/usr/local/sbin/felhom-bootstrap.sh`, the `vmbr9` stanza in `/etc/network/interfaces`, and `/etc/sudoers.d/felhom-agent.bak-pre-e2a` (21 KB — **INERT: sudo skips dotted filenames, verified with `sudo -l -U felhom-agent`; `visudo -c -f` parsing it OK is NOT evidence sudo loads it**) | **READY (S) — NEW 2026-08-09** | — | Purge by directory, not by glob; and do not let a new service account reuse a uid that owns old secrets | CC | | **R-276** | **RANK 2 — an uninstalled box keeps a live WireGuard tunnel into Felhom's off-site endpoint, and the teardown says nothing.** After `--uninstall` on demo-hp, `wg-quick@wg-felhom` is **enabled and active**, `/etc/wireguard/wg-felhom.conf` present, handshake to `167.233.158.164:443` **52 s old**, counters 5.86 GiB in / 2.48 GiB sent. It appears in **neither** the WIPED nor the KEPT list, though the BYO install disclosure names it prominently on the way in (*"an OUTBOUND WireGuard tunnel to the Felhom hub"*). A host told to leave Felhom retains a live network path into Felhom infrastructure, its hub-side peer registration intact, and nobody is told | **READY (S) — NEW 2026-08-09** | — | Tear the tunnel down and deregister the peer, or list it under KEPT with the reason and the removal command | CC | | **R-277** | **Three hub surfaces jointly present a HEALTHY off-site tier as an absent one — and it produced a wrong operator statement during this run.** For demo-hp on 2026-08-09 the box was pushing off-site daily without a gap (18 restic snapshots, `last_status: ok`), yet: (a) the customer page's Backup panel read `Snapshots 0 / Repo Size 0 MB / Integrity Unknown` — it renders the **local disk tier**, while the healthy `offsite` object sits **in the same report** unrendered on that panel; (b) the Offsite page read `0.0 GB` — true, but a 162 KB repo rounds to nothing; (c) a stale `offsite_delivery_stuck` event from **2026-08-07 10:19** (not recurring) reads as current state. **Three independent surfaces agreeing on a wrong picture is how a working backup gets "fixed".** It did exactly that here: the rehearsal reported a fleet-wide off-site outage to the operator and had to retract it. **Note the true half:** demo-felhom IS genuinely stuck (`offsite.state=needs_credential`, no run has ever succeeded) → **R-278** | **READY (S) — NEW 2026-08-09** | — | Render the offsite object on the offsite row; show bytes not rounded GB; distinguish a live alarm from event history | CC | | **R-278** | **demo-felhom's off-site tier has never completed a run and has been stuck for six days.** `offsite.state=needs_credential` since the 2026-08-03 guest rebuild; the hub's own alarm reads *"enabled + escrowed but no run has EVER succeeded"*; the controller's `offsite-credential-retry` job runs every 5 minutes and completes in 0 s, doing nothing. R-193's fix (the recovery SCREEN, controller 0.200.0) is present on the box, so the remedy exists — it just needs the customer-present ceremony that nobody has run, which is R-243's shape (*"a machine waiting for its recovery code can stop backing up off-site without alarming us"*) landing on a real box. **Contrast that makes it a defect and not a chore:** demo-hp, same rebuild, same day, recovered and has 18 snapshots | **READY (S) — NEW 2026-08-09** | — | Either the self-heal reconciler owns this shape end-to-end, or the box must say plainly on the dashboard that it is unprotected pending the recovery code | CC | | **R-279** | **There is no operator-triggerable off-site backup.** The only route to `POST /backup/offbox/run` is the customer's own dashboard session; `signed_jobs` carries opaque operator-SIGNED blobs and the hub holds no signing key. This cost the rehearsal a stop: preparing the run needed one off-site push and there was no operator path to it. Sibling of **R-177** (no operator-triggerable fill check) | **READY (XS) — NEW 2026-08-09** | — | Same shape as R-177; solve both together | CC | +| **R-280** | **RANK 1 — after a reinstall the data drive cannot be re-attached through ANY dashboard route, and the restore page promises it is "two clicks".** Measured on the rebuilt demo-hp, 2026-08-09. The restore page diagnoses the situation perfectly and then sends the customer to an empty page: *„Előbb csatold vissza az adatmeghajtót. A mentéseid megvannak, és a meghajtók is megvannak — újratelepítés után viszont a gép még nem ismeri őket, ezért most nincs hová visszaállítani. **Ez két kattintás:** Tárhely → Meghajtók, »Meglévő meghajtó csatolása«."* **It is not two clicks; it is zero possible clicks.** `GET /api/disks/candidates` → `{"initialize":[],"attach":[]}`, so both wizards render an empty selector, and `Tárhely → Meghajtók` reads „Nincs regisztrált adattároló" with an empty unregistered list. **The agent is not at fault** — `GET /api/disks` returns the NVMe in full (1.0 TB, SMART PASSED, `mount_path:/mnt/nvme-1tb`, `guest_attached:false`), so the channel and enumeration work. **ROOT CAUSE:** `handleDiskCandidates` builds both lists from `ListCandidateDisks`, the UNCLAIMED-disk scan; demo-hp's NVMe is deliberately BOTH the user-data drive and the `felhom-backup` target (`operations/nodes.md`), so it is claimed and never offered. That filter is **correct for `initialize`** (never offer to format a disk in use — `/storage/init` even says so: *„Rendszer- és biztonsági-mentés meghajtók itt nem jelennek meg — azok védettek"*) and **over-broad for `attach`**, which is non-destructive by definition and whose own page says *„A meghajtón lévő adatok nem törlődnek — a csatolás csak elérhetővé teszi azokat."* **It cascades:** no store → Calibre-Web's install page degrades to *„Nincs regisztrált adattároló — adja meg kézzel az útvonalat"* and demands a hand-typed `E-könyvtár útvonal`; no app → the restore rows read „Nincs telepítve". **THE ESCAPE HATCH WORKS AND NO CUSTOMER COULD FIND IT:** `POST /settings/storage/add` with `storage_path=/mnt/sys_drive` succeeded first try (*„Adattároló sikeresen hozzáadva"*) — and `/mnt/sys_drive` is an internal path, the very one registered before the wipe. Once registered, everything unblocked and the deploy form became a proper picker (*„Tárhely (sys_drive) — 64.2 GB szabad"*). **This is R-220's successor:** R-220 was closed as "drives unenrollable after a rebuild — fixed"; enumeration is fixed, OFFERING is not | **READY (M) — NEW 2026-08-09** | — | Populate `attach` from mounted-but-unregistered filesystems rather than from the unclaimed-DISK scan; and never print "two clicks" without asserting the destination is non-empty | CC | +| **R-281** | **The hub said NOTHING through an entire reinstall — and the tripwire for a sealed-backup unseal did not fire on a real unseal.** Between the uninstall (08:38 UTC) and the verified restore (10:27 UTC) demo-hp's guest was destroyed, the agent and its pveum identity removed, the host re-enrolled, a new guest provisioned, the box re-claimed, the sealed off-site package opened with the customer's recovery code, an app redeployed and 3.8 MB restored. **Events recorded for demo-hp in that window: ZERO. Notifications: ZERO.** **Positive control on the query** (standing rule 3): the hub recorded **2 events all day across all customers**, newest `db_dump_completed` at 00:30:07 — so the store is reachable and the silence is real, not a bad filter. **The good half, stated first:** no FALSE alarm fired during a legitimate reinstall, which is what P7 was watching for. **The owed half:** `escrow_blob_served` exists precisely as the tripwire for this moment — its text is *"the blob cannot be opened without the customer's recovery code… If no recovery is in progress on that box, investigate"* — and it **has fired for demo-hp before** (twice, last 2026-08-04 20:12:54). Today's unseal, through the R-193 recovery screen, fired it **not at all**. Either the screen's unlock path does not emit it or the rebuilt-box path bypasses it; **which of those is not established here.** A reinstall and a theft of a machine look identical to the operator | **READY (M) — NEW 2026-08-09** | — | Emit on the recovery-screen unlock path; and decide which reinstall milestones are worth one line each | CC | +| **R-282** | **One secret, three different Hungarian names, and the email sends the customer to a page their box is not showing.** Sending it from the hub is „**Visszaállító** kód küldése"; the email that arrives is subject „Jelszó-**visszaállítási** kód", body „**Visszaállító** kód: …", and it instructs *„Add meg a vezérlőpult »**Elfelejtett jelszó**« oldalán"*; the page the box actually serves is „A szerver **beállítása**" asking for a „**Beállító** kód". **A rebuilt box shows a SETUP page and the hub can only send a RESET mail** (because hub-side the customer is still `claimed_at 2026-07-21`), so the instruction names a route that does not exist on screen. **It does work if you ignore the instructions** — the reset code was accepted on the setup page (302 + session), so this is naming, not function. **It cost this session real time and one wasted code:** the operator supplied a 3-word Hungarian code believing it was the recovery code, because the hub calls the claim code „Visszaállító kód" and the ESCROW code is also „Visszaállító kód" — the only reliable discriminator is length (claim = 3 Hungarian words; recovery = **10** EFF-list words, and the recovery screen does say „(tíz szó)") | **READY (S) — NEW 2026-08-09** | — | Pick one name per secret and use it on all three surfaces; make the mail's page reference match what a rebuilt box actually shows | CC | +| **R-283** | **After a rebuild the hub says "Claimed 18d ago" while the box serves its first-run setup page.** `customer_claims` for demo-hp still read `claimed_at 2026-07-21 16:29:25`, `generation 2`, `issued_at 2026-08-03` while the freshly provisioned guest — whose `settings.json` is new — correctly showed „A szerver beállítása". The two sides never reconcile: the hub's claim state survives a guest rebuild and the box's does not. Consequences: the operator's screen says the box is claimed when it is not, a resend produces a RESET code instead of a SETUP code (→ **R-282**), and any previously issued code fails with *„Hibás vagy lejárt kód"* — a message that is technically true and tells the customer nothing about the real cause, namely their own reinstall. Mirror image of **R-214/R-235** (an already-paired box still told to pair itself) | **READY (S) — NEW 2026-08-09** | — | Let a report from a box carrying no claim state clear the hub's, or show both sides on the operator page | CC | +| **R-284** | **„A kiválasztott tárhely majdnem megtelt." on a store that is 93 % FREE — an apparent inverted threshold.** Calibre-Web's deploy page rendered `