admin
34646295dc
gates / gates (push) Successful in 29s
Venue demo-hp, operator-approved at STOP 1. Records the state that P1 destroys,
plus seven pre-walk findings, while they can still be checked against a live box.
R-268 CLOSED — the leaked per-guest local-API token is rotated and the rotation
is PROVEN in both directions (old refused, new accepted, channel up with a
positive observable). Rotating it surfaced three defects:
- an out-of-process rotation does NOT revoke the old token. The daemon serves
Lookup from a stale index and re-reads only on a MISS, so a superseded token
is a direct hit. Red-proved in a unit probe AND live on hardware; the shipped
RemintCoherence test passes only because it looks up the NEW token first.
- R-268's own recipe is incomplete: ensureLocalAPI returns early on a present
local_api block, so writing bootstrap.json is not enough — the controller
serves the old token from controller.yaml across restarts.
- the agent-channel alarm never closes: the UP branch does not notify from an
unseeded state, and the alarm's own remedy ("re-bootstrap") resets it.
Gate 0 complete: dataset planted in the Calibre library (coverage verified, not
assumed) with two Hungarian accented filenames; the comparator watched FAILING
three ways including an NFC->NFD rename that renders identically; off-site run
driven through the product's own button; restore point recorded by identity as
snapshot 41c830db, confirmed to carry all four files.
Also corrects the record: demo-hp's off-site tier is HEALTHY. Three hub surfaces
agreed it was absent and all three mislead — the panel showing 0 snapshots renders
the LOCAL tier, 162 KB rounds to 0.0 GB, and a two-day-old stuck event reads as
current. And the managed-update floor is live at 0.200.0, not 0.156.0.
257 lines
15 KiB
Markdown
257 lines
15 KiB
Markdown
# REHEARSAL — the BYO reinstall walk (2026-08-09)
|
||
|
||
> **Status: IN PROGRESS.** Pre-phase and Gate 0 are complete; STOP 1 passed. The walk (P1–P7) has not
|
||
> started. This file is written before the destructive phase deliberately — a finding that exists only
|
||
> in a session that later crashes is a finding nobody has.
|
||
|
||
**Venue: `demo-hp` (HP t740, `felhom-host`, guest 9201, customer `demo-hp`).** Operator-approved at
|
||
STOP 1. **Driven from DooPlex.** All times UTC unless marked; the host runs CEST (UTC+2).
|
||
|
||
---
|
||
|
||
## 1. Baselines — re-confirmed live on arrival, not taken from the spec
|
||
|
||
| | spec said | live reading | source |
|
||
|---|---|---|---|
|
||
| `felhom-agent` | v0.128.0 @ `28ba8593b8` | **0.128.0** on demo-felhom, **0.127.0** on demo-hp; HEAD == `origin/main` == `28ba8593b8` | `felhom-agent --version` on both nodes; `git rev-parse` |
|
||
| `felhom-controller` | v0.210.0 @ `c732fe1283` | **0.210.0** demo-felhom, **0.208.0** demo-hp; HEAD == `origin/main` == `c732fe1283` | hub `/configs`; `git rev-parse` |
|
||
| hub | v0.101.0 @ `56f8aa611c` | **0.101.0** (deployed image tag matches) | `kubectl get deploy hub`; page footer |
|
||
| Day-0 manifest | golden 0.210.0 / agent 0.128.0 / min 0.127.0 | **all three already saved** — golden `0.210.0` (`b9f701fa…`), agent `0.128.0` (`c6eba73b…`), min agent `0.127.0`, wrapper `104db0a4…` | hub `/configuration`, selected `<option>` values |
|
||
|
||
All three repos were clean and at `origin/main` before anything ran.
|
||
|
||
**The fleet is not uniform, and the venue is the box that is behind** — demo-hp runs agent 0.127.0 and
|
||
controller 0.208.0 against a manifest vouching 0.128.0/0.210.0. That is an argument *for* the venue: the
|
||
reinstall genuinely exercises the manifest rather than re-installing what is already there.
|
||
|
||
---
|
||
|
||
## 2. Venue comparison — the fidelity table that decided it
|
||
|
||
The runbook's criterion is *"the box whose current state most resembles what the tester will have"*.
|
||
|
||
| criterion | `demo-felhom` (N100) | `demo-hp` (t740) |
|
||
|---|---|---|
|
||
| customer apps deployed | 1 — opengist | **3 — privatebin, opengist, calibre-web** |
|
||
| enrolled user-data drive | **none registered.** `/mnt/felhom-drives` empty; controller logs `[WARN] Storage paths: no storage paths registered` | `/mnt/felhom-drives/Felhom-Share` + NVMe enrolled 2026-07-22; one registered storage path `/mnt/sys_drive` |
|
||
| off-site tier has actually run | **NO** — `offsite.state=needs_credential`, no run has ever succeeded | **YES** — 18 restic snapshots, daily, unbroken |
|
||
| escrow / recovery ceremony | present, generation 2 | present, generation 2 |
|
||
| install shape | mkimage loader, **Secure Boot OFF** (firmware workaround) | shim loader, **Secure Boot ENABLED** — the customer shape |
|
||
| collateral on the box | none | `drill-r50` (VM 300); leftover `c11-scratch` storage |
|
||
|
||
**Chosen: demo-hp.** It wins every fidelity criterion that distinguishes them.
|
||
|
||
**`drill-r50` is not at risk, and this was checked rather than assumed:** VM 300 is **not** in the
|
||
`felhom` pool (`pvesh get /pools/felhom` → members `[9201]`), its disks are on `local-lvm`, and
|
||
`--uninstall` removes no storage definitions and no non-pool guest — its own end-of-teardown statement
|
||
says the enrolled drives are *"unmounted only, NEVER wiped"* and that it never deletes PBS backups, hub
|
||
records or escrow blobs.
|
||
|
||
**Correction recorded, because it was part of the comparison put in front of the operator:** the
|
||
enrolled-drive row was first reported as *"real files on an enrolled drive"*. `Felhom-Share` is
|
||
**empty** — the share exists and is enrolled, but holds no files. demo-hp still wins the row (demo-hp
|
||
has a registered storage path; demo-felhom has none), on a narrower margin than first stated.
|
||
|
||
---
|
||
|
||
## 3. Pre-phase — R-268, the leaked per-guest local-API token
|
||
|
||
**Done first and independently, on `demo-felhom` guest 9201, as the runbook required.** No token value
|
||
reached stdout at any point: old value captured file→file from `bootstrap.json`, new value minted into
|
||
a file, both shredded at the end.
|
||
|
||
### 3.1 The rotation, and its proof
|
||
|
||
| step | old token | new token |
|
||
|---|---|---|
|
||
| before rotation | **HTTP 200** | — |
|
||
| bogus token (negative control) | **HTTP 401** | — |
|
||
| after the on-disk rotation, before any reload | **HTTP 200** ← *see 3.2* | 200 |
|
||
| after that lookup forced a reload | **HTTP 401** | — |
|
||
| after `systemctl restart felhom-agent` | **HTTP 401** | **HTTP 200** |
|
||
|
||
**Positive observable for the controller half:** `local-api: channel up (agent 169.254.253.1:8443) —
|
||
guest 9201, 3 mount(s) visible`, enumerating mp8/mp9/mp0 — data only the agent can supply. An absent
|
||
error would not have been evidence.
|
||
|
||
**R-268 is CLOSED.** The old token is refused; the new one works; the controller's channel is up.
|
||
|
||
### 3.2 FINDING — an out-of-process token rotation does not revoke the old token
|
||
|
||
`localapi.TokenStore.Mint`'s doc comment states *"last-write wins — any previous token for this guest is
|
||
revoked"*. **Across processes that is false until something else forces a reload.** The long-lived agent
|
||
serves `Lookup` from an in-memory index and re-reads the store **only on a miss** (the B3
|
||
reload-on-miss optimisation). A superseded token is still a direct map **hit**, so it authorises.
|
||
|
||
**Red-proved twice.** A temporary unit probe (Scenario C with the lookup order reversed — present the
|
||
rotated-out token *first*) fails on the claim; the shipped
|
||
`TestTokenStore_ReloadOnMiss_RemintCoherence` passes **only because it looks up the NEW token first**,
|
||
which is what triggers the reload that then evicts the old hash. Then proved on real hardware: the row
|
||
marked ← above is a live 200 from a token that the store on disk had already superseded.
|
||
|
||
This is the `CLAUDE.md` case exactly — a comment asserting an invariant with a test that appears to pin
|
||
it and does not. The honest fix is either a test that pins the reversed order, or an eviction that does
|
||
not depend on an unrelated lookup missing first.
|
||
|
||
### 3.3 FINDING — R-268's own stated rotation recipe is incomplete
|
||
|
||
The register row says the new plaintext *"must also be written into the guest's
|
||
`/etc/felhom-bootstrap/bootstrap.json` or the in-guest controller loses its agent access"*. **Writing
|
||
bootstrap.json is not sufficient.** `bootstrap.ensureLocalAPI` returns early when
|
||
`cfg.LocalAPI.Endpoint != ""` — it fills an *absent* block and never refreshes a present one — so after
|
||
a rotation the controller keeps serving the old token from its own `controller.yaml` **across a
|
||
restart**. Proved live: two controller restarts, still HTTP 401. The rotation only completed once
|
||
`local_api.token` was written into `controller.yaml` itself.
|
||
|
||
The neighbouring `DetectEndpointDrift` compares the **endpoint** and deliberately does not compare the
|
||
token (*"a token mismatch is a different failure"*). So this shape is known to be unmodelled, and R-78
|
||
("which file is authoritative") is its parent.
|
||
|
||
### 3.4 FINDING — the agent-channel alarm never closes, and its own prescribed remedy is why
|
||
|
||
`channelhealth` alerted correctly and promptly: `agent_channel_unauthorized`, no debounce, pushed to
|
||
the hub — *"agent rejected the controller token (HTTP 401) — token stale/rotated (**re-bootstrap**)"*.
|
||
|
||
The channel then recovered, and **no recovery event was ever sent**. In `Checker.Check`'s UP branch the
|
||
notify is guarded by `prev != "" && prev != "up"`, and a controller restart resets `state` to `""`. The
|
||
alert's own instruction is to re-bootstrap — i.e. restart the controller — so **following the alarm's
|
||
advice structurally guarantees the recovery is silent.** The down side is deliberately asymmetric
|
||
(F2: a born-down channel alerts on cycle 1); the up side is not.
|
||
|
||
Operator-visible result on the hub: two `agent_channel_unauthorized` errors, one `sent` and one
|
||
`suppressed` by cooldown, and nothing afterwards. The customer dashboard is fine — `SetDashboard`
|
||
reflects current state every cycle. It is the operator's trail that ends on "down".
|
||
|
||
---
|
||
|
||
## 4. Gate 0
|
||
|
||
### 4.1 The recovery code — CONFIRMED IN THE OPERATOR'S HANDS (STOP 1)
|
||
Not printed, not logged, not requested. It will be supplied at the screen at STOP 3.
|
||
|
||
### 4.2 The dataset — planted and fingerprinted
|
||
|
||
Planted at `/var/lib/felhom/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/` — inside
|
||
the **Calibre library**, which is real app-owned customer data and is **demonstrably** carried off-site
|
||
(the path appears in every `felhom-offbox,calibre-web` snapshot). Chosen after checking coverage rather
|
||
than assuming it: the per-app tier-1 unit carries only the named docker volume, so a file dropped in an
|
||
arbitrary place would not have travelled.
|
||
|
||
| sha256 (16) | bytes | name | name bytes (hex) |
|
||
|---|---|---|---|
|
||
| `54b773c46bbfd994…` | 3 145 728 | `binary-3mb.bin` | `62696e6172792d336d622e62696e` |
|
||
| `52a5c5ebfcac247f…` | 59 | **`árvíztűrő-tükörfúrógép.txt`** | `c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874` |
|
||
| `15d2024dfc224162…` | 25 | **`nested/őszibarack.md`** | `6e65737465642fc591737a6962617261636b2e6d64` |
|
||
| `924497918e55fe6d…` | 21 | `plain.txt` | `706c61696e2e747874` |
|
||
|
||
Manifest: `documentation/audits/evidence/` → recorded in the session scratchpad as
|
||
`GATE0-before-manifest.json`. The manifest keys on **raw name bytes**, not rendered text.
|
||
|
||
### 4.3 The positive control — the comparator was watched failing, three ways
|
||
|
||
| control | expected | result |
|
||
|---|---|---|
|
||
| identical copy | PASS | **PASS**, exit 0 |
|
||
| one bit flipped at byte 1 500 000 of the 3 MB binary | FAIL | **FAIL** — named the file and both sha256s |
|
||
| **NFC→NFD rename of the accented name** (renders *identically*) | FAIL | **FAIL** — caught as MISSING+EXTRA on the name bytes |
|
||
| one file deleted | FAIL | **FAIL** — named the missing file |
|
||
|
||
The NFD case is the one that matters: a comparison of *rendered* filenames would have passed it. Every
|
||
altered copy was discarded after the control ran.
|
||
|
||
### 4.4 The off-site tier — has run since the dataset was planted
|
||
|
||
Triggered through the product's own dashboard button (`POST /backup/offbox/run` → flash *„A távoli
|
||
mentés elindult"*), not by invoking restic by hand.
|
||
|
||
```
|
||
[offbox] backup run started (3 app(s) toggled)
|
||
[offbox] pre-push dump leg completed in 7.329s — snapshot pair is coherent
|
||
[offbox] backed up calibre-web (…/backups/primary/calibre-web, 1 mandatory path(s))
|
||
[offbox] backup OK: 3 app(s) backed up, 18 snapshot(s), 1m22s
|
||
```
|
||
|
||
**The restore point, by identity — never by size:**
|
||
|
||
| snapshot | time (UTC) | tag | paths |
|
||
|---|---|---|---|
|
||
| **`41c830db`** | 2026-08-09 08:30:38 | `felhom-offbox,calibre-web` | `…/backups/primary/calibre-web` + `…/userdata/media/books` |
|
||
| `9e38b84c` | 2026-08-09 08:30:49 | `felhom-offbox,opengist` | `…/backups/primary/opengist` |
|
||
| `78b93f04` | 2026-08-09 08:30:53 | `felhom-offbox,privatebin` | `…/backups/primary/privatebin` |
|
||
|
||
`restic ls 41c830db` confirms all four planted files are present, **both accented filenames included**.
|
||
|
||
### 4.5 The box, captured
|
||
`GATE0-demo-hp-before.txt` (282 lines): agent version, `pveversion`, `pct`/`qm` lists, pool membership,
|
||
`pvesm status`, `storage.cfg`, `df`, `lsblk` with serials, the agent unit + service user, sudoers, the
|
||
pveum users/tokens/roles/ACL, the guest hook, the WireGuard tunnel, the OOB belt, the agent state dir,
|
||
the install state file, PBS snapshots, local vzdump archives, and the enrolled-drive tree.
|
||
|
||
---
|
||
|
||
## 5. Findings so far, ranked by what they cost the person in front of you
|
||
|
||
Nothing yet is rank 1 or 2 — the walk has not started.
|
||
|
||
### Rank 3 — misleads
|
||
|
||
- **F-1 · A rotated-out local-API token still authorises** (§3.2). *A customer would do nothing — they
|
||
cannot see it. An operator rotating a leaked token would believe it revoked when it is not.* The
|
||
shipped test passes for the wrong reason.
|
||
- **F-2 · R-268's rotation recipe is incomplete** (§3.3). *Whoever follows it gets a box whose agent
|
||
channel is dead across restarts, with a correct-sounding alarm and no hint that `controller.yaml` is
|
||
the file that matters.*
|
||
- **F-3 · The agent-channel alarm never closes** (§3.4). *The operator fixes it as instructed and is
|
||
never told it worked; the hub's last word stays "down".*
|
||
- **F-4 · Three hub surfaces jointly misrepresent a healthy off-site tier as absent.** The customer page
|
||
reads `Snapshots 0 / Repo Size 0 MB / Integrity Unknown` (that panel renders the **local** tier, while
|
||
the healthy `offsite` object — `snapshot_count: 18`, `last_status: ok` — sits unrendered in the same
|
||
report); the Offsite page shows `0.0 GB` (true, but a 162 KB repo rounds to nothing); and a stale
|
||
`offsite_delivery_stuck` event from 2026-08-07 reads as current. *Three independent surfaces agreeing
|
||
on a wrong picture is how a working backup gets "fixed".* **This one caught me** — it produced a wrong
|
||
statement to the operator at STOP 1, corrected before it changed anything.
|
||
- **F-5 · There is no operator-triggerable off-site backup.** The only route is the customer's dashboard
|
||
button; `signed_jobs` needs a signing key the hub does not hold. Sibling of R-177.
|
||
|
||
### Rank 4 — cosmetic / hygiene
|
||
|
||
- **F-6 · Leave-behinds from earlier sessions**, found while capturing state:
|
||
- demo-hp host: storage `c11-scratch` (Campaign 11), and `felhom-backup:…vzdump-lxc-9100-2026_08_03…`
|
||
for a guest that no longer exists;
|
||
- demo-hp guest 9201 `/root`: `.dpw` (13 bytes), `.h`, `.sec.html`, all 2026-08-07 16:08;
|
||
- demo-felhom guest 9201: `/root/r204-backup/offbox.orig`.
|
||
- **F-7 · The managed-update floor is `0.200.0`, not `0.156.0`** — the runbook's §8.3 premise is stale.
|
||
Confirmed twice: hub `/configuration` reads *"Effective floor v0.200.0 — source: DB (hub_settings);
|
||
env fallback would be v0.120.0"*, and both boxes log `settle-gate: GO — at/above floor 0.200.0`. The
|
||
ruling's **substance** still needs recording; its **number** must not be written down as 0.156.0.
|
||
|
||
### Not a finding, checked and cleared
|
||
- The two `userdata` trees under `sys_drive` are the **same directory** surfaced twice, not a
|
||
backed-up copy beside an unprotected one. Checked against calibre-web's actual bind mounts.
|
||
|
||
---
|
||
|
||
## 6. Steps taken off-path, and what they cost
|
||
|
||
Rule 2 (no shell after the install command returns) does not bind yet — the walk has not begun. These
|
||
are Gate 0 preparation acts by the operator, recorded because they are mutations:
|
||
|
||
1. **R-268 rotation on demo-felhom** — token store appended, `bootstrap.json` and `controller.yaml`
|
||
rewritten, agent and controller restarted. Required by the runbook's pre-phase. *Not the venue.*
|
||
2. **demo-hp's dashboard password was re-set to the value in the operator's credentials file.** The
|
||
customer-owned password was not known to this session, and no operator-side route to the off-site
|
||
button exists (F-5). The prior `password_hash` is preserved at `/root/.orighash` and
|
||
`/root/settings.json.rehearsal-orig` in guest 9201; `claimed` was left `true`. **Done on operator
|
||
instruction.** Cost to fidelity: none for the walk — P1 destroys this guest, and P4 re-claims with a
|
||
password the operator chooses.
|
||
3. **The off-site run was started by a script pressing the dashboard's own endpoint**, with a real
|
||
session and a real CSRF token, rather than by a person clicking. The server path is identical; only
|
||
the click is synthetic.
|
||
|
||
---
|
||
|
||
## 7. What happens next
|
||
|
||
P1 — remove Felhom from demo-hp with `--uninstall`, then measure what it leaves behind. Nothing in
|
||
§1–§4 is repeated after that point; the "before" is fixed as of this file.
|