Rehearsal 2026-08-09 COMPLETE: data BYTE-IDENTICAL, journey needs a shell twice
gates / gates (push) Successful in 23s

The walk finished. All four planted files came back byte-identical out of
snapshot 41c830db, including two Hungarian accented filenames verified as RAW
NAME BYTES (NFC preserved) — the discriminator the Gate 0 positive control was
built for, having been watched failing on an NFC->NFD rename that renders the
same. Unlock 21s, restore 13.2s.

It finished only because a terminal was available twice:

- R-273 CLOSED. v0.128.0 was published as a package and never git-tagged, so
  every install died at 5/8. Tag pushed on operator instruction after an
  INDEPENDENT download proved the package sha equalled the vouched value;
  --resume then reached Day-0 SUCCESS in 3m49s on controller 0.210.0. The two
  guards that would stop the class recurring are still owed.
- R-280 NEW, rank 1. A reinstalled box cannot re-attach its own data drive by
  any dashboard route: /api/disks/candidates returns empty because both lists
  are built from the UNCLAIMED-disk scan, and the drive is claimed precisely
  because it is also the backup target. Correct for "initialise", over-broad for
  "attach", which is non-destructive by definition. The restore page meanwhile
  says "Ez ket kattintas" and points at that empty list. Cleared by POSTing
  /mnt/sys_drive — an internal path no household could produce.

Also new: R-281 the hub said NOTHING through the entire reinstall and the
sealed-backup tripwire did not fire on a real unseal (positive control: 2 events
all day fleet-wide); R-282 one code with three names and a mail pointing at a
page the box does not show; R-283 hub reads "Claimed 18d ago" while the box
serves its setup page; R-284 "almost full" over a 93%-free store.

R-274 NARROWED by measurement rather than left as written: the resume path
fetched the vouched golden correctly, because --resume skips the preflight that
does local discovery. What survives is real — discovery is sort|tail -1 with no
manifest comparison — but a FRESH install taking a stale golden is still not
observed, and the row says so.

Two of my own claims were refuted by test and are recorded as refuted, not
quietly dropped: the leftover sudoers file is inert (sudo skips dotted names),
and demo-hp's off-site tier was healthy all along.
This commit is contained in:
2026-08-09 12:35:26 +02:00
parent b1afbb8a4d
commit 1d6f1c522d
7 changed files with 1237 additions and 82 deletions
+75 -28
View File
@@ -4,15 +4,20 @@
## The answer to the runbook's question, first
**Not answerable today, and the reason is the answer.** *"If a person's machine is set up again from
scratch, does their data come back through the product, in one sitting, with no command line inside
the machine?"* — the walk never reached the data. **A machine wiped today cannot be set up again at
all**: the install fails at step 5 of 8, on every host, because the agent version the hub vouches was
never given a git tag (**R-273**). The step before that had already refused, because removing Felhom
leaves behind the very thing that blocks reinstalling it (**R-272**).
**Yes, the data comes back byte for byte. No, not in one sitting, and not without a shell.**
The data was never at risk and is not now: the planted dataset sits in off-site snapshot `41c830db`
with its fingerprints recorded, plus a PBS snapshot and four local vzdump archives.
All four planted files returned **BYTE-IDENTICAL** — including two Hungarian accented filenames
verified as *raw name bytes*, not as rendered text. The unlock took **21 s**, the restore **13.2 s**.
But the walk completed only because two hard stops were cleared by someone who could open a terminal
and read source. **R-273**: the install died at step 5/8 on an agent version that was published as a
package but never git-tagged — cleared by completing the release. **R-280**: the reinstalled machine
could not re-attach its own data drive through any dashboard route, while the restore page said
„**Ez két kattintás**" and pointed at an empty list — cleared by POSTing an internal path
(`/mnt/sys_drive`) that no household could produce.
Neither is a data-integrity problem. Both stop a household dead. **This is the same shape the R-201
walks kept finding: the data half passes, the journey half fails.**
## Venue
@@ -25,7 +30,11 @@ workaround. `drill-r50` (VM 300) was verified not at risk before proceeding: it
## Findings, ranked by what they cost the person in front of you
**1 — stops the visit**
- **R-273** · the vouched agent (0.128.0) has no git tag; every install and reinstall dies at 5/8.
- **R-273** · the vouched agent (0.128.0) had no git tag; every install died at 5/8. **CLOSED** — tag
pushed on your instruction after an independent sha check; install then succeeded in 3 m 49 s. The
two guards that would prevent a recurrence are still owed.
- **R-280** · a reinstalled machine cannot re-attach its data drive through any route, and the restore
page promises „két kattintás" at an empty list. **The one to fix before the tester's visit.**
- **R-272** · Felhom's uninstall restarts its own dnsmasq unconstrained; it grabs `:53`; the next
install refuses and appears to blame the owner's network.
@@ -37,6 +46,10 @@ workaround. `drill-r50` (VM 300) was verified not at risk before proceeding: it
in neither the KEPT nor the WIPED list.
**3 — misleads**
- **R-281** · the hub said nothing at all through the whole reinstall, and the tripwire for a
sealed-backup unseal did not fire on a real one.
- **R-282 / R-283** · one code, three names; the mail points at a page the box is not showing; the hub
reads "Claimed 18d ago" while the box serves its setup page.
- **R-269** · a rotated-out local-API token still authorises until an unrelated lookup forces a
reload. The shipped test passes only because of its lookup order.
- **R-270** · R-268's own rotation recipe is a step short; the controller never re-reads the mount.
@@ -62,14 +75,20 @@ I also raised **R-275**'s sudoers half as a likely privilege-escalation on reins
and refuted my own hypothesis**: sudo skips filenames containing dots, so the leftover file is inert.
`visudo -c -f` parsing a file OK is not evidence that sudo loads it.
## Integrity verdict
## Integrity verdict — BYTE-IDENTICAL
**Not reached** — P6 never ran. The instrument is proven and waiting: the comparator was watched
**failing** three ways, including an NFC→NFD rename that renders identically to the eye and would
have passed a rendered-text comparison. Four files, two with Hungarian accented names
(`árvíztűrő-tükörfúrógép.txt`, `nested/őszibarack.md`), one 3 MB binary; manifest keyed on raw name
bytes, committed as `evidence-rehearsal-2026-08-09/GATE0-before-manifest.json`; all four confirmed
present in snapshot `41c830db`.
```
expected 4 file(s); found 4
VERDICT: BYTE-IDENTICAL
```
Four expected, four restored, zero differences, compared against
`evidence-rehearsal-2026-08-09/GATE0-before-manifest.json` — a manifest keyed on **raw name bytes**.
`árvíztűrő-tükörfúrógép.txt` and `nested/őszibarack.md` came back with their name bytes intact (NFC
preserved, `c3a1…`), which is the discriminator the Gate 0 positive control was built to enforce: the
comparator had been watched **failing** on an NFC→NFD rename that renders identically to the eye.
Restored out of snapshot `41c830db` into the verification folder the product names, with live data
untouched.
## Wall clocks
@@ -80,8 +99,19 @@ present in snapshot `41c830db`.
| P1 uninstall | **60 s** (08:37:23 → 08:38:23 UTC) |
| P1 leave-behind measurement | ~12 min |
| P2 preflight (3 runs: 2 refusals, 1 pass) | ~6 min |
| **P3 install — FAILED** | **44 s** (08:51:33 → 08:52:17 UTC) |
| bare machine → visible files | **not reached** |
| P3 install — first attempt, FAILED | 44 s (08:51:33 → 08:52:17 UTC) |
| P3 install — resumed, SUCCESS | **3 m 49 s** |
| P4 first contact (box live on its own URL) | within ~4 min of install |
| STOP 3 unlock | **21 s** |
| app redeploy (calibre-web) | **1 m 36 s** |
| restore prepare + execute | **8 s + 13.2 s** |
| **bare machine → verified files** | **1 h 49 m 22 s** (08:38:23 → 10:27:45 UTC) |
| — of which the product's own work | **≈ 7 m 47 s** |
**Neither figure is the customer number.** The 1 h 49 m is dominated by the R-273 diagnosis and release
fix (~38 min) and two waits on a human. The 7 m 47 s is what the product costs when the operator
already knows every answer. **The honest unaided figure is undefined, because an unaided household does
not finish.**
## Steps taken off-path, and what they cost
@@ -96,9 +126,18 @@ present in snapshot `41c830db`.
permission check, so the secret still never touched argv. A person would type it.
5. **`systemctl stop dnsmasq && systemctl disable dnsmasq`** — the action the refusal message tells
the owner to take, used as the counterfactual that confirmed R-272.
6. **Pushed the `v0.128.0` git tag** — outward-facing, done on your "proceed", and only after an
independent download proved the published package's sha256 equalled the hub's vouched value. It
completes a half-finished release rather than changing code; the release script's own recovery text
is the same line. **Cost to the walk: the install that followed was a `--resume`, not a fresh run,
which is why R-274 is only half-observed.**
7. **`POST /settings/storage/add` with `/mnt/sys_drive`** — the manual escape hatch, typed. This is the
R-280 wall; a customer could not produce that path. **The biggest fidelity cost of the run.**
8. **SSH into the guest to fingerprint the restored tree.** This is my *instrument*, not a customer
step — the customer's step (the restore) finished at the dashboard. Byte-comparison inherently needs
file access; nothing about the product was driven this way.
Everything after the install command returned was read-only. No repair was attempted on the
half-installed box.
Everything else after the install returned was read-only, and no repair was attempted on the box.
## Teardown — all four layers
@@ -123,14 +162,22 @@ DB copy were handled file→file at 0600 and shredded; the hub DB copy (which ca
break-glass credential) was shredded immediately after the one hash comparison it was taken for.
Credential comparisons were done by sha256 prefix, never by value.
## State for whoever resumes
## State demo-hp was left in
- **Blocked on one command** — `git tag v0.128.0 28ba8593b8 && git push origin v0.128.0`.
- **Nothing in Gate 0 needs redoing.** Snapshot `41c830db` holds the dataset; the before-manifest is
committed; the comparator is proven.
- **First thing to record at step 7/8** when the install resumes: which golden it takes (R-274 is the
one finding still owed an end-to-end observation).
- **P4 will need you** — the claim code is emailed to `doodoo21@freemail.hu` and there is no
operator-side route to it (R-119).
**Back in service and healthy**agent 0.128.0, controller 0.210.0, guest 9201 running and onboot,
claimed, storage path registered, `calibre-web` deployed, off-site repository unlocked and intact at
18 snapshots. `drill-r50` (VM 300) untouched throughout.
**Deliberately left alone, and named rather than tidied:** the pre-existing `c11-scratch` storage and
the three `vzdump-lxc-9100` golden archives on `local` (the teardown keeps goldens by design, and they
now number three). The restored files sit in the product's verification folder, not back in place —
that is R-213 and the product says so.
## Still owed
- **R-280** — the drive wall. The one finding that would stop the tester's visit outright.
- **R-273's two guards** — refuse a vouch whose tag does not resolve; check that a package and its tag
ship together. The tag push fixed one box, not the class.
- **R-274's missing observation** — a *fresh* (non-resume) install taking a stale local golden.
Full account: `documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`.
+56 -41
View File
@@ -23,21 +23,42 @@ destroyed on purpose and its files came back byte for byte identical — four ti
their recovery code got everything back with **no command line inside the machine at any point**,
in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252, R-253 — closed.)*
## Broken right now — and it stops every new machine
## The rehearsal finished. The data came back byte for byte; the journey did not.
**We wiped a working demo machine and put it back. All four test files returned identical — including
the two with Hungarian accents, checked as raw bytes, not as text on screen.** The unlock took 21
seconds and the restore 13. **But it only finished because I could open a terminal twice.** A
household would have stopped, twice, and the second time the screen would have told them it was easy.
**The two walls, both fixed-or-fixable, neither about the data:**
- **The install died four steps in** — the agent version you approved had been published as a download
but never given its version label, and the installer looks it up by that label. **Now unblocked**
I pushed the label after checking the published file matched what you vouched. *(R-273 — closed. The
two guards that would stop it recurring are still owed.)*
- **A reinstalled machine cannot re-attach its own data drive.** Every route is a dead end, and the
restore page cheerfully says „**Ez két kattintás**" while pointing at an empty list. The drive is
fine and the machine can see it — it just is not offered, because the same drive is also the backup
target. I got past it by typing an internal path no customer could know. **This is the one to fix
before the tester's visit.** *(R-280)*
**Also broken, found on the way:**
- **No machine can be set up from scratch today.** The install dies four steps in, on every host, and
it needs one command from you to unblock (below). Nothing already running is affected — the two demo
machines and the tester's box are fine. It is only the *setting-up* path, which is the one his visit
is made of. *(R-273)*
- **A rebuilt machine may come back on software from last week** — the install will happily reuse an
old copy it finds lying on the disk, without checking it against the version you approved. The copy
on the demo machine is old enough to predate the recovery screen. *(R-274)*
- **Taking Felhom off a machine leaves the one thing that stops it going back on.** We install a small
network service at setup; removing Felhom restarts it without its settings, it takes over the port
the next install needs, and the next install then refuses — blaming the owner's network. *(R-272)*
network service at setup; removing Felhom restarts it without its settings, it seizes the port the
next install needs, and the next install then refuses — appearing to blame the owner's network.
*(R-272)*
- **A machine we removed keeps its private line to us open.** *(R-276)*
- **The hub said nothing at all** while a machine was wiped, rebuilt, re-claimed and had its sealed
backups opened. No false alarm — but also no word, and the alarm that exists for "someone is opening
this customer's backups" stayed silent through a real one. *(R-281)*
- **A rebuilt machine may still come back on software from last week** — narrower than I first wrote:
the resumed install fetched the right version, but a fresh one takes whatever copy is newest on the
disk without checking it against what you approved. *(R-274)*
- **demo-felhom has not had an off-site backup in six days** and is waiting for a recovery code nobody
has entered. demo-hp, rebuilt the same day, recovered by itself. *(R-278)*
- **One code, three different names**, and the email points at a page the machine is not showing —
this cost us a wasted code today. *(R-282, R-283)*
## What's broken
@@ -51,29 +72,26 @@ in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252
accumulates. *(R-244)*
- **Putting restored files back where they belong is still manual.** *(R-213)*
## The reinstall rehearsal — what one afternoon on one machine found
## The rest of what the rehearsal found
**We took a working demo machine, removed Felhom, and tried to put it back the way a person with their
own hardware would.** It got three steps in. That is the finding: **nine defects, two of which stop a
visit dead**, and none of them were visible from reading the code — the last three sessions of review
did not see any of them.
Sixteen findings in one afternoon, **none of them visible from reading the code** — three sessions of
review had not seen any.
- It stopped at the install, on a missing version label *(R-273)* and the step before that had
already refused, because removing Felhom leaves behind the thing that blocks reinstalling it
*(R-272)*.
- Removing Felhom also leaves five files holding old keys *(R-275)* and a live private line to us
*(R-276)*.
- Along the way: rotating a leaked key does not actually revoke the old one until the service is
restarted *(R-269)*, the written-down recipe for that rotation is missing a step *(R-270)*, and the
alarm it raises can never be closed because the fix it recommends is what silences the all-clear
*(R-271)*.
- **It also caught me being wrong.** Three separate hub screens agreed that demo-hp had no off-site
backup. It had eighteen. I told you the fleet's backups were down and had to take it back an hour
later — the screens were misleading, but I should have asked the machine before telling you. *(R-277)*
- Removing Felhom leaves five files holding old keys *(R-275)*, and rotating a leaked key does not
revoke the old one until the service restarts *(R-269)* — the written recipe for it is a step short
*(R-270)*, and the alarm it raises can never be closed because the fix it recommends is what
silences the all-clear *(R-271)*.
- **It caught me being wrong twice, and that matters more than the count.** I told you the fleet's
off-site backups were down; demo-hp had eighteen snapshots and I had read three misleading screens
instead of asking the machine *(R-277)*. And I raised a leftover permissions file as a security
hole, then tested it and refuted myself — it is inert.
**The data was never at risk** and is not now: the test files sit in an off-site snapshot with their
fingerprints recorded, plus two local copies. Full account:
`documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`.
**What worked, and should not be lost in the count:** the machine came up on its own at the approved
version; the setup page appeared unprompted, in Hungarian, naming the customer; **the recovery screen
appeared without being looked for** and said plainly that unlocking changes nothing; the restore told
the truth about putting files in a checking folder rather than back in place; and no false alarm fired.
Full account: `documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`.
## Three rulings, written down so they stop living in a conversation
@@ -111,17 +129,14 @@ only miss a true alarm, never raise a false one *(R-266)*.
## Waiting on you
- **One command, and it unblocks every install.** The version you approved yesterday was published as
a download but never given its version label in the source repository, and the installer looks it up
by that label. The code is right and already shipped; only the label is missing.
`cd felhom-agent && git tag v0.128.0 28ba8593b8 && git push origin v0.128.0`. I did not do it
myself: publishing a release label is not a thing to do quietly in the middle of a measurement.
*(R-273)*
- **Then say go, and the rehearsal resumes** where it stopped. Nothing has to be redone — the test
files are already in an off-site backup and their fingerprints are recorded.
*(Both items previously here are done: the approval is saved — golden 0.210.0, agent 0.128.0, minimum
0.127.0 — and the recovery fix was proved on hardware.)*
- **Nothing blocking.** The rehearsal is finished and demo-hp is back in service: agent 0.128.0,
controller 0.210.0, claimed, off-site backups unlocked and intact.
- **One decision worth taking before the tester comes:** whether to fix the drive wall *(R-280)* now.
It is the only finding that would stop his visit outright, and it is the difference between "his
data comes back" and "his data comes back if someone types a path for him."
- **Two guards are still owed** so the install cannot break the same way twice: refuse to vouch a
version whose label does not resolve, and check that a published version and its label ship
together. *(R-273's tail.)*
## DooPlex infrastructure — separate from the product
@@ -1,8 +1,9 @@
# REHEARSAL — the BYO reinstall walk (2026-08-09)
> **Status: IN PROGRESS.** Pre-phase and Gate 0 are complete; STOP 1 passed. The walk (P1P7) has not
> started. This file is written before the destructive phase deliberately — a finding that exists only
> in a session that later crashes is a finding nobody has.
> **Status: COMPLETE.** All phases walked; the integrity verdict is **BYTE-IDENTICAL**. §1–§8 were
> written before the destructive phase deliberately — a finding that exists only in a session that
> later crashes is a finding nobody has — and are left as written, including one claim later refuted
> by test (§8.3 → §9a) and one by measurement (§7.2 F-9).
**Venue: `demo-hp` (HP t740, `felhom-host`, guest 9201, customer `demo-hp`).** Operator-approved at
STOP 1. **Driven from DooPlex.** All times UTC unless marked; the host runs CEST (UTC+2).
@@ -512,14 +513,200 @@ which golden it takes.**
---
## 10. Where the run stands
## 9a. P3 resumed — and the install succeeded
**Reached: P1 ✓, P2 ✓, P3 ✗ (blocked).** P4P7 not attempted.
**Unblocked on operator instruction ("proceed").** `v0.128.0` was annotated at `28ba8593b8` and pushed
after an **independent download** confirmed the published package's sha256 equals the hub's vouched
`c6eba73b…`. Both config URLs then served 200. `--resume` completed in **3 m 49 s**
(09:30:30 → 09:34:19 UTC): `Day-0 provision SUCCESS`, controller **0.210.0**, agent **0.128.0**, guest
9201 onboot, pool member, all 16 ACL assertions green, controller healthy in ~18 s.
**The honest answer to §2's question is: NOT YET ANSWERABLE, and the reason is itself the answer for
today.** A machine wiped today cannot be set up again at all — not because the data is gone (it is
safe, in snapshot `41c830db` and in two local tiers), but because the install refuses at step 5 of 8 on
a missing git tag. The walk found a hard stop before it ever reached the question about data.
**R-274 partly refuted, and recorded as such.** Step 7 fetched the **vouched 0.210.0** — because
`--resume` skips preflight, and preflight is where local auto-discovery sets `GOLDEN_VOLID`. So the
fresh and resume paths disagree, and the resume path is the safe one. What survives: discovery is
`sort | tail -1` (newest local) with **no manifest comparison**, so a box whose newest local golden
predates the vouched one still installs stale. Full text in the register.
**The dataset is intact and waiting.** Nothing about Gate 0 needs redoing when the walk resumes:
snapshot `41c830db` holds all four files, the before-manifest is committed, and the comparator is proven.
## 10. P4 — first contact, as the customer
**A clean pass, and worth saying so plainly.** Within four minutes of the install finishing, the
customer's own URL — fetched from outside the box, over the public internet — served:
> **A szerver beállítása** · Demo HP
> *Add meg az e-mailben kapott beállító kódot, majd válassz saját jelszót a vezérlőpult védelméhez.*
> Beállító kód · Új jelszó (min. 12 karakter) · Új jelszó megerősítése
> *Nem kaptad meg a kódot? Új kód kérése*
Unprompted, in Hungarian, naming the customer, with a self-service route if the code never arrived,
and nothing anywhere asking for a command line. The hub showed the host **ONLINE** at agent 0.128.0.
**Two findings here, neither fatal:** the hub still read *"Claimed 18d ago"* while the box was serving
its first-run page (**R-283**), and the code that arrives is named three different things across the
three surfaces, with the mail pointing at an „Elfelejtett jelszó" page the box does not show
(**R-282**). The reset code was nonetheless **accepted on the setup page** — 302 and a session — so
this is naming, not function.
## 11. P5 — getting the machine back to work
### 11.1 The recovery screen, unsought — the headline pass
The **first thing** on the dashboard after claiming, with nothing sought:
> **Adatok visszaszerzése** — *Ezt a gépet újratelepítették. A korábbi, házon kívüli mentéseid
> megvannak* — a sealed package held centrally, **sealed at 2026-08-04T11:11:37Z**, openable only with
> the customer's code; *nobody can replace it — not Felhom, not support, not the operator*; and
> **„Ebben a lépésben semmit nem állítunk vissza és semmi nem változik."**
> Field: **Helyreállítási kód (tíz szó)**.
R-193's screen meeting reality on a genuinely rebuilt box. It answered all three of its questions and
its seal date matches `host_escrow.created_at` exactly.
*(Nit: the seal date is rendered raw as `2026-08-04T11:11:37Z` to a Hungarian household — ISO-8601 in
UTC where a localised date belongs.)*
### 11.2 STOP 3 — the unlock
Entered at the box's own screen, from a file, never on a command line. **21 seconds**, and it listed
what it found without restoring anything:
| app | legutóbbi mentés | méret |
|---|---|---|
| **calibre-web** | 2026-08-09 10:30 (CEST) | **3.8 MB** |
| felhom-offbox | ″ | 6.8 KB |
| opengist | ″ | 182.3 KB |
| privatebin | ″ | 6.8 KB |
That is exactly the Gate 0 snapshot set, seen from the customer's side.
### 11.3 The wall — R-280, and it is rank 1
The restore page diagnoses the situation perfectly and then sends the customer to an empty page:
> *„Előbb csatold vissza az adatmeghajtót. … **Ez két kattintás:** Tárhely → Meghajtók, »Meglévő
> meghajtó csatolása«. Utána gyere vissza ide."*
**It is not two clicks; it is zero possible clicks.** `GET /api/disks/candidates` →
`{"initialize":[],"attach":[]}`. The agent is fine — `GET /api/disks` returns the NVMe in full — but
`handleDiskCandidates` builds *both* lists from the **unclaimed-disk** scan, and demo-hp's NVMe is
deliberately both the user-data drive and the `felhom-backup` target, so it is claimed and never
offered. Correct for `initialize`; over-broad for `attach`, which is non-destructive by definition.
It cascades: no store → Calibre-Web's install page degrades to *„Nincs regisztrált adattároló — adja
meg kézzel az útvonalat"*; no app → every restore row reads „Nincs telepítve".
**Escape hatch used, and recorded as off-path:** `POST /settings/storage/add` with
`storage_path=/mnt/sys_drive` succeeded first try — an internal path, the very one registered before
the wipe, that no household customer could produce. Everything unblocked immediately afterwards and
the deploy form became a proper picker („Tárhely (sys_drive) — 64.2 GB szabad").
### 11.4 App redeploy
`calibre-web` deployed from the catalogue through the dashboard's own API in **1 m 36 s**, running and
healthy. The catalogue itself was complete and in Hungarian, every app „Nincs telepítve".
*(Also observed: the deploy page warned „A kiválasztott tárhely majdnem megtelt." over an option
carrying `data-free-percent="93"` and „64.2 GB szabad" — an apparent inverted threshold, **R-284**.)*
## 12. P6 — getting the data back
Three options, each explained, none starting without a question. **R-204 item 3 is visibly fixed:**
option 1 states in as many words that it does *not* bring the customer's own files back.
| step | wall clock |
|---|---|
| full-restore **prepare** (3.8 MB) | **8 s** |
| full-restore **execute** | **13.2 s** (10:27:32.08 → 10:27:45.29 UTC) |
> *„A(z) calibre-web teljes mentése visszaállítva ellenőrző mappába:
> `/mnt/sys_drive/felhom-data/backups/offsite-restore/calibre-web` — a saját fájljaiddal együtt. A
> meglévő adatok változatlanok."*
Restored to a **verification folder**, not into place — honest about it, and R-213's territory.
### THE INTEGRITY VERDICT — **BYTE-IDENTICAL**
```
expected 4 file(s); found 4
VERDICT: BYTE-IDENTICAL
```
Compared against `GATE0-before-manifest.json`, keyed on **raw name bytes**:
| sha256 | bytes | name |
|---|---|---|
| `54b773c46bbfd994…` | 3 145 728 | `binary-3mb.bin` |
| `52a5c5ebfcac247f…` | 59 | **`árvíztűrő-tükörfúrógép.txt`** — name bytes `c3a1…` identical, NFC preserved |
| `15d2024dfc224162…` | 25 | **`nested/őszibarack.md`** — name bytes identical |
| `924497918e55fe6d…` | 21 | `plain.txt` |
Four expected, four restored, zero differences. The accented filenames survived as **bytes**, not
merely as rendered text — the discriminator the Gate 0 positive control was built to enforce.
## 13. P7 — what the hub thought was happening
**It thought nothing.** Across the whole reinstall the hub recorded **zero events and zero
notifications** for demo-hp. Positive control (standing rule 3): the hub logged **2 events all day
across all customers**, newest `db_dump_completed` at 00:30:07 — the store is reachable and the
silence is real.
- **The good half:** no false alarm fired during a legitimate reinstall, which is what P7 watches for.
- **The owed half (R-281):** `escrow_blob_served` exists as the tripwire for exactly this moment —
*"If no recovery is in progress on that box, investigate"* — and **has fired for demo-hp before**
(last 2026-08-04 20:12:54). Today's real unseal fired it not at all. A reinstall and a stolen
machine look identical to the operator.
## 14. Teardown — all four layers
1. **The machine** — my instruments (`fp.py`, `restored.json`, the installer, the passphrase file)
removed. Guest 9201 running and healthy. **`drill-r50` (VM 300) untouched throughout**, as predicted.
2. **The host** — `local-lvm` 36.97 % → 21.85 % (bare) → **32.35 %** (rebuilt). Pre-existing leftovers
found and deliberately **not** removed, recorded instead: `c11-scratch`, the orphaned
`vzdump-lxc-9100` archives (now three).
3. **The hub** — **no customer or appliance record created**; `demo-hp` retained deliberately per the
runbook. Nothing to delete.
4. **The off-site side** — **18 snapshots, newest still `9e38b84c` / `78b93f04` at 08:30 UTC**, i.e.
unchanged since Gate 0. The restore was a pure read. **No prune, no forget, no delete by me**; the
only retention that ran was inside the product's own backup call at Gate 0, which reported
`18 snapshot(s)` itself.
## 15. The answer to §2's question
**Yes — the data comes back, byte for byte. No — not in one sitting, and not without a shell.**
The walk completed: **P1 ✓ P2 ✓ P3 ✓ P4 ✓ P5 ✓ P6 ✓ P7 ✓**, and the verdict is BYTE-IDENTICAL.
But it completed only because two hard stops were cleared by someone who could open a terminal and
read source code:
1. **R-273** — the install died at 5/8 on a git tag that was never pushed. Cleared by a release action.
2. **R-280** — the data drive could not be re-attached through any dashboard route, while the page
promised „két kattintás". Cleared by POSTing an internal path a customer could not know.
Neither is a data-integrity problem. Both are **journey** problems, and both stop a household dead.
This is the same shape the R-201 walks kept finding: **the data half passes, the journey half fails.**
**What the product did beautifully, and should not be lost in the finding count:** the box came up on
its own at the vouched version; the setup page appeared unprompted, in Hungarian, naming the customer;
the recovery screen appeared **without being sought** and stated plainly that nothing would be changed
by unlocking; the unlock took **21 s**; the restore took **13.2 s** and said honestly that it had put
the files in a verification folder rather than back in place. Sixteen ACL assertions verified
themselves. No false alarm fired.
### Wall clocks
| segment | |
|---|---|
| P1 uninstall | **60 s** |
| P2 preflight (2 refusals, then pass) | ~6 min |
| P3 install — first attempt, FAILED | 44 s |
| P3 install — resumed, SUCCESS | **3 m 49 s** |
| P4 first contact (box live on its own URL) | within ~4 min of install |
| STOP 3 unlock | **21 s** |
| app redeploy (calibre-web) | **1 m 36 s** |
| restore prepare + execute | **8 s + 13.2 s** |
| **bare machine → verified files** | **1 h 49 m 22 s** (08:38:23 → 10:27:45 UTC) |
| — of which the product's own work | **≈ 7 m 47 s** |
**The 1 h 49 m must not be quoted as the customer number** — it is dominated by the R-273 diagnosis and
release fix (~38 min) and by two waits on a human. **The ≈ 7 m 47 s must not be quoted either**: it is
what the product costs when someone already knows every answer. The honest figure for an unaided
household is **undefined, because an unaided household does not finish.**
@@ -0,0 +1,40 @@
[INFO] felhom-host-install v1.25.0 — mode=byo customer=demo-hp vmid=9201
[STEP] 1/8 pre-flight
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: felhom-host (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~42 GiB
[WARN] local-lvm free ~42 GiB < hard min 120 GiB
[INFO] free RAM: ~26535 MiB
[INFO] existing guests on this host: 1 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storages all present or pre-positioned: local local-lvm felhom-pbs
[ERROR] a resolver is already bound to :53 on this host:
udp UNCONN 0 0 0.0.0.0:53 0.0.0.0:* users:(("dnsmasq",pid=1291306,fd=4)) udp UNCONN 0 0 [::]:53 [::]:* users:(("dnsmasq",pid=1291306,fd=6)) tcp LISTEN 0 32 0.0.0.0:53 0.0.0.0:* users:(("dnsmasq",pid=1
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
(Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
[ERROR] PRE-FLIGHT FAIL (exit 1) — fix the finding above and re-run
Connection to 100.76.96.79 closed.
@@ -0,0 +1,835 @@
felhom-host
[INFO] felhom-host-install v1.25.0 — mode=byo customer=demo-hp vmid=9201
[SKIP] pre-flight (resumed)
[STEP] 2/8 Proxmox API token
[SKIP] pool felhom already exists
[INFO] user felhom-agent@pve exists
[INFO] removing stale token agent (secret unrecoverable — rotating)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
[OK] token minted (secret captured, not logged)
[INFO] role FelhomAgentBase exists — ensuring exact privileges
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] role FelhomAgentGuest exists — ensuring exact privileges
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] role FelhomAgentStore exists — ensuring exact privileges
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from ~42 GiB free (ONE volume since R-165)
[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host REUSED (idempotent — existing credential)
[INFO] host_id: demo-hp-bb76ea (api_key captured, not logged)
[SKIP] break-glass (step 4b) SKIPPED (byo: the host owner manages break-glass; root@pam untouched, nothing vaulted)
[STEP] 5/8 agent install (fetch + verify + install)
[SKIP] apt repo alignment skipped (byo — the owner manages repos)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.210.0
[INFO] fetching agent binary v0.128.0 from Gitea …
[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[INFO] service user felhom-agent exists
[INFO] felhom-agent already in systemd-journal
[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path)
[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path)
[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path)
[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path)
[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated)
[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config)
[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed)
[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer)
[SKIP] OOB (felhom-sshd) off (byo, or appliance --no-oob) — skipping
[STEP] 6/8 agent config + service
[SKIP] backup target 'felhom-backup' already exists — leaving its DEFINITION exactly as it is (Scenario F)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
felhom-backup-target-apply: granted FelhomAgentStore on /storage/felhom-backup (user + token)
[INFO] node=felhom-host local_api=192.168.0.87:8443 tls_fp=E5:6D:79:3B:67:A1…
[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
[OK] byo config asserts passed (lan_resolver.enable / oob.enabled off; deployment_mode=byo; wg_tunnel intentionally ON — base infrastructure)
[OK] agent --selftest (read-only) passed
[OK] felhom-agent service active (non-root felhom-agent reads the config OK)
[STEP] 7/8 golden archive
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] fetching golden v0.210.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst
[OK] verified sha256 b9f701fab813c051… matches the hub manifest
[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst
[STEP] 8/8 provision guest 9201
[SKIP] pool felhom already exists
=== felhom-agent 0.128.0 selftest=provision (vmid=9201 customer=demo-hp hostname=demo-hp) ===
--- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst → vmid 9201 ---
time=2026-08-09T11:33:34.336+02:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom
[OK] front half: vmid 9201 up (boot+running) in 1m9s; MAC=BC:24:11:65:7F:69
--- back half: mint per-guest token + populate bootstrap config mount ---
time=2026-08-09T11:33:38.654+02:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=192.168.0.87:8443
[OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap)
local-api endpoint 192.168.0.87:8443 · leaf fp e3c7aaafbe7643d35eb5a09d2c3fe5128f1ef626010e7c17060dd55a5cda37d5 · token: minted (not printed)
=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) ===
next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller,
which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api.
[OK] provision completed
[INFO] rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[STEP] verify
[OK] pct status: running
[OK] onboot: 1
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
rootfs: local-lvm:vm-9201-disk-0,size=32G
[OK] pool: guest 9201 is a member of felhom
[OK] acl: FelhomAgentBase@/ present (user+token)
[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token)
[OK] acl: FelhomAgentStore@/storage/local present (user+token)
[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token)
[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token)
[OK] authz signers: 2 (operator-signed self-update armed)
[OK] controller: Up 18 seconds (healthy) (after ~0s)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.210.0
[INFO] cloudflared: Up 18 seconds
[INFO] (confirm in the hub UI that host demo-hp-bb76ea reports guest 9201)
[OK] Day-0 provision SUCCESS — vmid=9201 host_id=demo-hp-bb76ea customer=demo-hp golden=local:backup/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst
Connection to 100.76.96.79 closed.
@@ -0,0 +1,26 @@
{
"count": 4,
"files": {
"62696e6172792d336d622e62696e": {
"name_utf8": "binary-3mb.bin",
"sha256": "54b773c46bbfd99465b68247bcb4d3cf5bf8f57a4024c6ba169dc0c326979dd2",
"size": 3145728
},
"6e65737465642fc591737a6962617261636b2e6d64": {
"name_utf8": "nested/őszibarack.md",
"sha256": "15d2024dfc22416261c56ed51741f8d7ae734350b3ac79ef1cb982d667ddef64",
"size": 25
},
"706c61696e2e747874": {
"name_utf8": "plain.txt",
"sha256": "924497918e55fe6dd75ff5ed8a9089aa64752e05540bdf804d71a03afbc48acd",
"size": 21
},
"c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874": {
"name_utf8": "árvíztűrő-tükörfúrógép.txt",
"sha256": "52a5c5ebfcac247f1753e86f69db46815adf4edaf28375396324e4ea6163611c",
"size": 59
}
},
"root": "/var/lib/felhom/sys_drive/felhom-data/backups/offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09"
}
+7 -2
View File
@@ -501,13 +501,18 @@ applied.** The one that matters: Scenario A **fails against today's tree** with
| **R-270** | **R-268's stated rotation recipe is incomplete: the controller never re-reads `bootstrap.json`'s `local_api`, so a rotation leaves the agent channel dead across restarts.** `bootstrap.ensureLocalAPI` returns early when `cfg.LocalAPI.Endpoint != ""` — by design it FILLS an absent block and never refreshes a present one — so the token the controller uses lives in its own `controller.yaml`, not in the mount. Proved live 2026-08-09: two controller restarts after a correct `bootstrap.json` rotation, still `HTTP 401`; the channel came up only once `local_api.token` was written into `controller.yaml`. The neighbouring `DetectEndpointDrift` compares the ENDPOINT and deliberately does not compare the token (*"a token mismatch is a different failure"*), so this shape is knowingly unmodelled. Parent question — which file is authoritative — is **R-78** | **READY (S) — NEW 2026-08-09** | — | Either teach the drift detector the token, or make the rotation path write both files. Correct the R-268 row's recipe either way | CC |
| **R-271** | **The `agent_channel_unauthorized` alarm can never be closed, because its own prescribed remedy is what silences the recovery.** `channelhealth.Checker.Check`'s UP branch notifies only when `prev != "" && prev != "up"`; a controller restart resets `state` to `""`, so an unseeded→up transition is silent by construction. The alert text says *"token stale/rotated (**re-bootstrap**)"* — i.e. restart the controller — so **following the instruction guarantees no recovery event.** Observed live 2026-08-09: two `agent_channel_unauthorized` errors on the hub (one `sent`, one `suppressed` by the 1 h operator cooldown) and **nothing afterwards**, though the channel came up 3 minutes later and stayed up. The down side is deliberately asymmetric (F2: a born-down channel alerts on cycle 1); the up side never got the matching treatment. Customer dashboard is fine — `SetDashboard` reflects current state every cycle. It is the OPERATOR's trail that ends on "down" | **READY (S) — NEW 2026-08-09** | — | Notify on unseeded→up when the previous *persisted* state was down, or seed from the hub's last event | CC |
| **R-272** | **RANK 1 — Felhom's `--uninstall` leaves the exact condition that makes Felhom's own reinstall REFUSE.** Chain, fully evidenced on demo-hp 2026-08-09: Felhom installs `dnsmasq` at day-0 (`/var/lib/dpkg/info/dnsmasq.list` dated **2026-07-21 18:24 CEST**, demo-hp's day-0) and constrains it with a snippet in `/etc/dnsmasq.d/`; `--uninstall` removes the snippet and **restarts the daemon** (running process start time **2026-08-09 10:37:39 CEST — inside the 10:37:2310:38:23 uninstall window**) but leaves the package installed and the unit **enabled**; unconstrained, dnsmasq binds `0.0.0.0:53`; the next install's preflight then hard-refuses with *"a resolver is already bound to :53"*. It is **not** PVE SDN's (`/etc/pve/sdn/` empty; stock unit). The teardown mentions it only as *"the 'sudo' and 'dnsmasq' packages were left installed (**system packages**)"* — **dnsmasq is not a system package here, Felhom installed it.** **What a customer does next:** reads a message blaming a resolver, concludes their own LAN DNS is at fault, and debugs something they never configured. **Counterfactual confirmed:** `systemctl stop dnsmasq && systemctl disable dnsmasq``host DNS (:53): free` → PRE-FLIGHT PASS, nothing else changed. **The refusal MESSAGE is good** (finding, evidence, two routes, and an explicit promise not to touch DNS on a host it does not own) — the defect is that Felhom caused the condition and does not say so | **READY (M) — NEW 2026-08-09** | — | Either stop+disable dnsmasq on uninstall when Felhom installed it, or have the preflight recognise its own leftover and say so | CC |
| **R-273** | **RANK 1 — the hub vouched an agent version that was never git-tagged, and every install fleet-wide now fails at step 5/8.** `v0.128.0` exists as a Gitea **package** (so it is selectable in the artifact manifest and its sha256 verifies) but **not as a git tag** — newest tag is `v0.127.0`. Since R-183, `felhom-host-install.sh` correctly pins config fetches to `raw/tag/v$ART_AGENT_VER`, so the install dies: `curl: (22) … 404``raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)`. Measured: `raw/branch/main/…`**200**, `raw/tag/v0.127.0/…`**200**, `raw/tag/v0.128.0/…`**404**. The commit that IS 0.128.0 is on `main` (`28ba8593b8`); only the tag is missing. **THIS IS R-184 ARRIVING** — that row says nothing prevents the hub vouching a never-released version because the R-115 gate works from tags. **A good fix met an open gap:** `hostinstall_gates.py` asserts the pin EXISTS (`ok: fetch_raw pins the agent configs to the vouched agent version`); nothing asserts the pinned ref RESOLVES. And `felhom-agent/scripts/release-agent.sh:23` already documents the hazard verbatim — *"a released version without a git tag 404s a box mid-install, as root"* — so it was understood and the step was still missed. **Blast radius, precise:** every fresh install and every reinstall is dead while the manifest vouches 0.128.0 (approved 2026-08-09 — the approval armed it); **existing boxes are unaffected** (self-update takes the binary from the package registry, and demo-felhom runs 0.128.0 healthily). **Unblock:** `git tag v0.128.0 28ba8593b8 && git push origin v0.128.0` — publishes a tag for a commit already on `main` and already released as a package. Rolling the manifest back to 0.127.0 instead would cost R-221. **Deliberately NOT done in the rehearsal session** (diagnose-not-fix; publishing a tag is outward-facing). **Owed gates:** (a) refuse a vouch whose tag does not resolve, (b) a release-completeness check that the package and the tag ship together | **WAITING-ON-OPERATOR** — one tag push | — | `audits/REHEARSAL-byo-reinstall-2026-08-09.md` §9 | Viktor |
| **R-274** | **A local golden is adopted with NO version and NO checksum check, so a reinstall can silently come up releases behind.** `felhom-host-install.sh` step 7: `if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then log_skip "using local golden"; return 0; fi`**the hub manifest's `golden.sha256`, whose entire purpose is to vouch from a different trust root than Gitea, is consulted only on the FETCH path.** A locally-present archive bypasses the vouch: no version compare, no digest, no warning. On demo-hp 2026-08-09 the preflight selected `local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst`, whose baked marker reads `felhom-controller:`**`0.192.0`**, against a vouched golden of **0.210.0** — 18 releases stale. **The sharp consequence:** 0.192.0 is **below 0.200.0, where R-193's off-site recovery SCREEN shipped**, so a customer reinstalled today returns on a controller that cannot run the recovery ceremony their data depends on; it is also born below the managed-update floor (0.200.0), and the updater's auto-target is the floor, never the newest. **It compounds with the teardown**, which deliberately keeps the old golden (*"golden vzdump left in place"*). This is the R-111/R-115/R-120 drift family one layer down: the R-120 gate guards what may be VOUCHED, nothing guards what an install TAKES. **NOT YET OBSERVED END-TO-END** — step 7 never ran because R-273 killed step 5, so this rests on the code path + the preflight line + the archive marker. **First thing to record when the walk resumes** | **READY (S) — NEW 2026-08-09** | R-273 | Compare the local golden's version/sha against the manifest and refuse or re-fetch on mismatch; say so in the BYO disclosure, which today lists only what the install CREATES, never what it REUSES | CC |
| **R-273** | **RANK 1 — the hub vouched an agent version that was never git-tagged, and every install fleet-wide now fails at step 5/8.** `v0.128.0` exists as a Gitea **package** (so it is selectable in the artifact manifest and its sha256 verifies) but **not as a git tag** — newest tag is `v0.127.0`. Since R-183, `felhom-host-install.sh` correctly pins config fetches to `raw/tag/v$ART_AGENT_VER`, so the install dies: `curl: (22) … 404``raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)`. Measured: `raw/branch/main/…`**200**, `raw/tag/v0.127.0/…`**200**, `raw/tag/v0.128.0/…`**404**. The commit that IS 0.128.0 is on `main` (`28ba8593b8`); only the tag is missing. **THIS IS R-184 ARRIVING** — that row says nothing prevents the hub vouching a never-released version because the R-115 gate works from tags. **A good fix met an open gap:** `hostinstall_gates.py` asserts the pin EXISTS (`ok: fetch_raw pins the agent configs to the vouched agent version`); nothing asserts the pinned ref RESOLVES. And `felhom-agent/scripts/release-agent.sh:23` already documents the hazard verbatim — *"a released version without a git tag 404s a box mid-install, as root"* — so it was understood and the step was still missed. **Blast radius, precise:** every fresh install and every reinstall is dead while the manifest vouches 0.128.0 (approved 2026-08-09 — the approval armed it); **existing boxes are unaffected** (self-update takes the binary from the package registry, and demo-felhom runs 0.128.0 healthily). **Unblock:** `git tag v0.128.0 28ba8593b8 && git push origin v0.128.0` — publishes a tag for a commit already on `main` and already released as a package. Rolling the manifest back to 0.127.0 instead would cost R-221. **Deliberately NOT done in the rehearsal session** (diagnose-not-fix; publishing a tag is outward-facing). **Owed gates:** (a) refuse a vouch whose tag does not resolve, (b) a release-completeness check that the package and the tag ship together | **CLOSED 2026-08-09 — tag pushed, install PROVEN** | — | On operator instruction ("proceed"), `v0.128.0` was annotated at `28ba8593b8` and pushed, after an INDEPENDENT download confirmed the published package's sha256 equals the hub's vouched `c6eba73b…`. `raw/tag/v0.128.0/configs/{felhom-agent.service,felhom-mkfs-guarded.sh}` then both served **200**, and `--resume` carried the install to `Day-0 provision SUCCESS` in **3m49s**. **THE TWO GATES ARE STILL OWED AND ARE THE REAL FIX** — this close records that one box was unblocked, not that the class is. | CC |
| **R-274** | **A local golden is adopted with NO version and NO checksum check, so a reinstall can silently come up releases behind.** `felhom-host-install.sh` step 7: `if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then log_skip "using local golden"; return 0; fi`**the hub manifest's `golden.sha256`, whose entire purpose is to vouch from a different trust root than Gitea, is consulted only on the FETCH path.** A locally-present archive bypasses the vouch: no version compare, no digest, no warning. On demo-hp 2026-08-09 the preflight selected `local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst`, whose baked marker reads `felhom-controller:`**`0.192.0`**, against a vouched golden of **0.210.0** — 18 releases stale. **The sharp consequence:** 0.192.0 is **below 0.200.0, where R-193's off-site recovery SCREEN shipped**, so a customer reinstalled today returns on a controller that cannot run the recovery ceremony their data depends on; it is also born below the managed-update floor (0.200.0), and the updater's auto-target is the floor, never the newest. **It compounds with the teardown**, which deliberately keeps the old golden (*"golden vzdump left in place"*). This is the R-111/R-115/R-120 drift family one layer down: the R-120 gate guards what may be VOUCHED, nothing guards what an install TAKES. **OBSERVED 2026-08-09, AND THE RESULT NARROWS THE ROW — recorded because it partly refutes what was written above.** On the RESUME path step 7 **fetched the vouched 0.210.0 correctly** (`fetching golden v0.210.0 from Gitea`), because `--resume` skips preflight and preflight is where local auto-discovery sets `GOLDEN_VOLID` (the GL6-F4 comment says so). **So the fresh-install and resume paths disagree on golden selection, and the resume path is the safe one.** Discovery is `… | sort | tail -1`, i.e. the NEWEST local archive by filename — a sensible heuristic, **and still no comparison against the manifest's version or sha**. The defect therefore stands as: *a box whose newest local golden predates the vouched one installs stale, silently* — which is exactly the state demo-hp was in before this run (newest local 0.192.0 vs vouched 0.210.0). It is now masked on this box because the freshly fetched 0.210.0 is the newest — **correct by recency, not by verification**. There are now **three** goldens on `local` (07-21, 08-03, 08-09), because the teardown keeps them. **Still not observed: a FRESH (non-resume) install taking a stale local golden.** | **READY (S) — NEW 2026-08-09, NARROWED same day** | | Compare the local golden's version/sha against the manifest and refuse or re-fetch on mismatch; say so in the BYO disclosure, which today lists only what the install CREATES, never what it REUSES | CC |
| **R-275** | **`--uninstall` leaves five 0600 `agent.json.*` credential backups, and the reinstall hands them to the new service account.** `/etc/felhom-agent/` survives with `agent.json.{campaign8-before,campaign9-before,campaign9-prev,pre-e-target-move,pre-prunegate.bak}`, each carrying a 64-char `hub.api_key` and a 59-char `proxmox.token`. The teardown claims to remove *"config (+ its .bak backups)"* and `scripts/CHANGELOG` F1 records *"uninstall now purges the agent config's `.bak*` siblings (one held a live hub api_key)"* — **that fix does not match the filenames in use, and it misses `agent.json.pre-prunegate.bak`, a file that literally ends in `.bak`.** **Exposure assessed, not assumed:** these are SUPERSEDED — the orphaned key hashes to `a5d2222a…`, the hub's current demo-hp key to `8c59d1b6…`, and the Proxmox token was deleted by the same uninstall. **But the reinstall recreates `felhom-agent` at uid 999, the same uid the deleted account had**, so three of the backups become the new account's files — verified readable as `felhom-agent`. A fresh install's service account inherits read access to the prior install's credentials; superseded today, live if the backups were recent (R-179's precedent). **Also left, undeclared:** `/etc/felhom/{.bootstrap-done,appliance-pairing-code}`, `felhom-bootstrap.service` + `/usr/local/sbin/felhom-bootstrap.sh`, the `vmbr9` stanza in `/etc/network/interfaces`, and `/etc/sudoers.d/felhom-agent.bak-pre-e2a` (21 KB — **INERT: sudo skips dotted filenames, verified with `sudo -l -U felhom-agent`; `visudo -c -f` parsing it OK is NOT evidence sudo loads it**) | **READY (S) — NEW 2026-08-09** | — | Purge by directory, not by glob; and do not let a new service account reuse a uid that owns old secrets | CC |
| **R-276** | **RANK 2 — an uninstalled box keeps a live WireGuard tunnel into Felhom's off-site endpoint, and the teardown says nothing.** After `--uninstall` on demo-hp, `wg-quick@wg-felhom` is **enabled and active**, `/etc/wireguard/wg-felhom.conf` present, handshake to `167.233.158.164:443` **52 s old**, counters 5.86 GiB in / 2.48 GiB sent. It appears in **neither** the WIPED nor the KEPT list, though the BYO install disclosure names it prominently on the way in (*"an OUTBOUND WireGuard tunnel to the Felhom hub"*). A host told to leave Felhom retains a live network path into Felhom infrastructure, its hub-side peer registration intact, and nobody is told | **READY (S) — NEW 2026-08-09** | — | Tear the tunnel down and deregister the peer, or list it under KEPT with the reason and the removal command | CC |
| **R-277** | **Three hub surfaces jointly present a HEALTHY off-site tier as an absent one — and it produced a wrong operator statement during this run.** For demo-hp on 2026-08-09 the box was pushing off-site daily without a gap (18 restic snapshots, `last_status: ok`), yet: (a) the customer page's Backup panel read `Snapshots 0 / Repo Size 0 MB / Integrity Unknown` — it renders the **local disk tier**, while the healthy `offsite` object sits **in the same report** unrendered on that panel; (b) the Offsite page read `0.0 GB` — true, but a 162 KB repo rounds to nothing; (c) a stale `offsite_delivery_stuck` event from **2026-08-07 10:19** (not recurring) reads as current state. **Three independent surfaces agreeing on a wrong picture is how a working backup gets "fixed".** It did exactly that here: the rehearsal reported a fleet-wide off-site outage to the operator and had to retract it. **Note the true half:** demo-felhom IS genuinely stuck (`offsite.state=needs_credential`, no run has ever succeeded) → **R-278** | **READY (S) — NEW 2026-08-09** | — | Render the offsite object on the offsite row; show bytes not rounded GB; distinguish a live alarm from event history | CC |
| **R-278** | **demo-felhom's off-site tier has never completed a run and has been stuck for six days.** `offsite.state=needs_credential` since the 2026-08-03 guest rebuild; the hub's own alarm reads *"enabled + escrowed but no run has EVER succeeded"*; the controller's `offsite-credential-retry` job runs every 5 minutes and completes in 0 s, doing nothing. R-193's fix (the recovery SCREEN, controller 0.200.0) is present on the box, so the remedy exists — it just needs the customer-present ceremony that nobody has run, which is R-243's shape (*"a machine waiting for its recovery code can stop backing up off-site without alarming us"*) landing on a real box. **Contrast that makes it a defect and not a chore:** demo-hp, same rebuild, same day, recovered and has 18 snapshots | **READY (S) — NEW 2026-08-09** | — | Either the self-heal reconciler owns this shape end-to-end, or the box must say plainly on the dashboard that it is unprotected pending the recovery code | CC |
| **R-279** | **There is no operator-triggerable off-site backup.** The only route to `POST /backup/offbox/run` is the customer's own dashboard session; `signed_jobs` carries opaque operator-SIGNED blobs and the hub holds no signing key. This cost the rehearsal a stop: preparing the run needed one off-site push and there was no operator path to it. Sibling of **R-177** (no operator-triggerable fill check) | **READY (XS) — NEW 2026-08-09** | — | Same shape as R-177; solve both together | CC |
| **R-280** | **RANK 1 — after a reinstall the data drive cannot be re-attached through ANY dashboard route, and the restore page promises it is "two clicks".** Measured on the rebuilt demo-hp, 2026-08-09. The restore page diagnoses the situation perfectly and then sends the customer to an empty page: *„Előbb csatold vissza az adatmeghajtót. A mentéseid megvannak, és a meghajtók is megvannak — újratelepítés után viszont a gép még nem ismeri őket, ezért most nincs hová visszaállítani. **Ez két kattintás:** Tárhely → Meghajtók, »Meglévő meghajtó csatolása«."* **It is not two clicks; it is zero possible clicks.** `GET /api/disks/candidates``{"initialize":[],"attach":[]}`, so both wizards render an empty selector, and `Tárhely → Meghajtók` reads „Nincs regisztrált adattároló" with an empty unregistered list. **The agent is not at fault**`GET /api/disks` returns the NVMe in full (1.0 TB, SMART PASSED, `mount_path:/mnt/nvme-1tb`, `guest_attached:false`), so the channel and enumeration work. **ROOT CAUSE:** `handleDiskCandidates` builds both lists from `ListCandidateDisks`, the UNCLAIMED-disk scan; demo-hp's NVMe is deliberately BOTH the user-data drive and the `felhom-backup` target (`operations/nodes.md`), so it is claimed and never offered. That filter is **correct for `initialize`** (never offer to format a disk in use — `/storage/init` even says so: *„Rendszer- és biztonsági-mentés meghajtók itt nem jelennek meg — azok védettek"*) and **over-broad for `attach`**, which is non-destructive by definition and whose own page says *„A meghajtón lévő adatok nem törlődnek — a csatolás csak elérhetővé teszi azokat."* **It cascades:** no store → Calibre-Web's install page degrades to *„Nincs regisztrált adattároló — adja meg kézzel az útvonalat"* and demands a hand-typed `E-könyvtár útvonal`; no app → the restore rows read „Nincs telepítve". **THE ESCAPE HATCH WORKS AND NO CUSTOMER COULD FIND IT:** `POST /settings/storage/add` with `storage_path=/mnt/sys_drive` succeeded first try (*„Adattároló sikeresen hozzáadva"*) — and `/mnt/sys_drive` is an internal path, the very one registered before the wipe. Once registered, everything unblocked and the deploy form became a proper picker (*„Tárhely (sys_drive) — 64.2 GB szabad"*). **This is R-220's successor:** R-220 was closed as "drives unenrollable after a rebuild — fixed"; enumeration is fixed, OFFERING is not | **READY (M) — NEW 2026-08-09** | — | Populate `attach` from mounted-but-unregistered filesystems rather than from the unclaimed-DISK scan; and never print "two clicks" without asserting the destination is non-empty | CC |
| **R-281** | **The hub said NOTHING through an entire reinstall — and the tripwire for a sealed-backup unseal did not fire on a real unseal.** Between the uninstall (08:38 UTC) and the verified restore (10:27 UTC) demo-hp's guest was destroyed, the agent and its pveum identity removed, the host re-enrolled, a new guest provisioned, the box re-claimed, the sealed off-site package opened with the customer's recovery code, an app redeployed and 3.8 MB restored. **Events recorded for demo-hp in that window: ZERO. Notifications: ZERO.** **Positive control on the query** (standing rule 3): the hub recorded **2 events all day across all customers**, newest `db_dump_completed` at 00:30:07 — so the store is reachable and the silence is real, not a bad filter. **The good half, stated first:** no FALSE alarm fired during a legitimate reinstall, which is what P7 was watching for. **The owed half:** `escrow_blob_served` exists precisely as the tripwire for this moment — its text is *"the blob cannot be opened without the customer's recovery code… If no recovery is in progress on that box, investigate"* — and it **has fired for demo-hp before** (twice, last 2026-08-04 20:12:54). Today's unseal, through the R-193 recovery screen, fired it **not at all**. Either the screen's unlock path does not emit it or the rebuilt-box path bypasses it; **which of those is not established here.** A reinstall and a theft of a machine look identical to the operator | **READY (M) — NEW 2026-08-09** | — | Emit on the recovery-screen unlock path; and decide which reinstall milestones are worth one line each | CC |
| **R-282** | **One secret, three different Hungarian names, and the email sends the customer to a page their box is not showing.** Sending it from the hub is „**Visszaállító** kód küldése"; the email that arrives is subject „Jelszó-**visszaállítási** kód", body „**Visszaállító** kód: …", and it instructs *„Add meg a vezérlőpult »**Elfelejtett jelszó**« oldalán"*; the page the box actually serves is „A szerver **beállítása**" asking for a „**Beállító** kód". **A rebuilt box shows a SETUP page and the hub can only send a RESET mail** (because hub-side the customer is still `claimed_at 2026-07-21`), so the instruction names a route that does not exist on screen. **It does work if you ignore the instructions** — the reset code was accepted on the setup page (302 + session), so this is naming, not function. **It cost this session real time and one wasted code:** the operator supplied a 3-word Hungarian code believing it was the recovery code, because the hub calls the claim code „Visszaállító kód" and the ESCROW code is also „Visszaállító kód" — the only reliable discriminator is length (claim = 3 Hungarian words; recovery = **10** EFF-list words, and the recovery screen does say „(tíz szó)") | **READY (S) — NEW 2026-08-09** | — | Pick one name per secret and use it on all three surfaces; make the mail's page reference match what a rebuilt box actually shows | CC |
| **R-283** | **After a rebuild the hub says "Claimed 18d ago" while the box serves its first-run setup page.** `customer_claims` for demo-hp still read `claimed_at 2026-07-21 16:29:25`, `generation 2`, `issued_at 2026-08-03` while the freshly provisioned guest — whose `settings.json` is new — correctly showed „A szerver beállítása". The two sides never reconcile: the hub's claim state survives a guest rebuild and the box's does not. Consequences: the operator's screen says the box is claimed when it is not, a resend produces a RESET code instead of a SETUP code (→ **R-282**), and any previously issued code fails with *„Hibás vagy lejárt kód"* — a message that is technically true and tells the customer nothing about the real cause, namely their own reinstall. Mirror image of **R-214/R-235** (an already-paired box still told to pair itself) | **READY (S) — NEW 2026-08-09** | — | Let a report from a box carrying no claim state clear the hub's, or show both sides on the operator page | CC |
| **R-284** | **„A kiválasztott tárhely majdnem megtelt." on a store that is 93 % FREE — an apparent inverted threshold.** Calibre-Web's deploy page rendered `<option value="/mnt/sys_drive" data-free-percent="93">` alongside „Tárhely (sys_drive) — **64.2 GB szabad**" and the warning „A kiválasztott tárhely majdnem megtelt." 93 % free read as 93 % used is the obvious candidate, and `checkStorageSpace(this)` is the function to look at. **Not confirmed by reading the code** — reported as measured output only. A capacity warning that cries wolf on an empty disk is one a customer learns to click past | **READY (XS) — NEW 2026-08-09** | — | Check `checkStorageSpace`'s comparison against `data-free-percent`; add a render test per branch | CC |
**Explicitly still open, untouched by this session:** R-246, R-255, R-256, R-257, R-261, R-262,
R-263, **R-264** (the twenty-one undecided facts — a design session of its own), R-240, R-243,