Rehearsal 2026-08-09: the walk stops at P3 — R-273 blocks every install fleet-wide
gates / gates (push) Successful in 24s
gates / gates (push) Successful in 24s
P1 uninstall, P2 preflight, P3 install. The install FAILED at step 5/8 in 44s, and the two rank-1 findings are both on the setting-up path a tester's visit is made of. Eleven register rows minted (R-269..R-279); ceiling moves 268 -> 279. R-273 (RANK 1) — the hub vouches agent 0.128.0; that version was published as a Gitea PACKAGE but never git-tagged. Since R-183 the installer correctly pins its config fetches to raw/tag/v<vouched>, so every fresh install and every reinstall now 404s as root, mid-install. Measured: main 200, v0.127.0 200, v0.128.0 404. This is R-184 arriving; release-agent.sh:23 already documents the exact hazard. Existing boxes are fine (self-update takes the binary from the registry). NOT fixed here — publishing a release tag is outward-facing and the runbook says stop and report. One command unblocks it; it is in STATUS.md. R-272 (RANK 1) — Felhom's own uninstall leaves the condition that makes Felhom's own reinstall refuse. It installs dnsmasq at day-0, then on teardown removes the snippet and RESTARTS the daemon unconstrained (process start time lands inside the uninstall window), which grabs 0.0.0.0:53; the next preflight then refuses, and the message reads as though the owner's LAN DNS is at fault. R-274 — a local golden is adopted with no version and no sha check; the manifest vouch is consulted only on the fetch path. demo-hp's local copy is controller 0.192.0 against a vouched 0.210.0, and below the 0.200.0 where the recovery screen shipped. Not yet observed end-to-end (R-273 killed step 5 first). Also: R-275 orphaned credential backups + uid reuse, R-276 the wg tunnel outlives the uninstall, R-269/270/271 from the token rotation, R-277 three hub surfaces misreport a healthy off-site tier, R-278 demo-felhom six days unprotected, R-279 no operator-triggerable off-site run. Two hypotheses of mine were tested and REFUTED rather than shipped as findings: the leftover sudoers file is inert (sudo skips dotted filenames), and demo-hp's off-site tier was healthy all along - I had misread the hub and said so. STATUS.md records the three rulings §8.3 asked for, with the floor CORRECTED to its live value 0.200.0 and the count corrected to twenty.
This commit is contained in:
@@ -250,7 +250,276 @@ are Gate 0 preparation acts by the operator, recorded because they are mutations
|
||||
|
||||
---
|
||||
|
||||
## 7. What happens next
|
||||
## 7. P1 — removing Felhom from the host
|
||||
|
||||
P1 — remove Felhom from demo-hp with `--uninstall`, then measure what it leaves behind. Nothing in
|
||||
§1–§4 is repeated after that point; the "before" is fixed as of this file.
|
||||
`--uninstall`, the product's own path, dry-run first. **60 seconds** (08:37:23 → 08:38:23 UTC), exit 0.
|
||||
The typed-vmid confirmation was satisfied, not bypassed.
|
||||
|
||||
### 7.1 The reversal, proved rather than trusted
|
||||
|
||||
| component | before | after |
|
||||
|---|---|---|
|
||||
| agent binary | `/usr/local/bin/felhom-agent` 0.127.0 | **ABSENT** |
|
||||
| `felhom-agent` service user | present | **ABSENT** |
|
||||
| `felhom-op` (OOB) user | present | **ABSENT** |
|
||||
| `/etc/felhom-sshd` (OOB belt) | present | **ABSENT** |
|
||||
| systemd units | agent, mgmt-watchdog, felhom-sshd, oob-nft, shared-parent | **all removed** |
|
||||
| pveum roles | `FelhomAgentBase/Guest/Store` | **none** |
|
||||
| pveum user + token | `felhom-agent@pve!agent` | **gone**; user list is `root@pam` only |
|
||||
| pveum ACL | pool + 9 scratch-vmid + 3 storage grants | **empty** |
|
||||
| `felhom` pool | members `[9201]` | **`[]` — pool removed** |
|
||||
| guest 9201 | running | **destroyed** |
|
||||
| guest-hook snippet | present | **removed** |
|
||||
| dnsmasq snippets | present | **removed** |
|
||||
| `/var/lib/felhom-agent`, `/var/lib/felhom-install` | present | **removed** |
|
||||
| **KEPT as declared** | | PBS snapshots (2), local vzdump (4), `/mnt/felhom-drives/Felhom-Share`, storage defs |
|
||||
|
||||
Space returned: `local-lvm` 20 904 790 → 12 355 143 KiB (**≈8.5 GiB**); `local` ≈64 MiB; the NVMe
|
||||
unchanged (backups deliberately kept). `drill-r50` (VM 300) untouched, as predicted.
|
||||
|
||||
### 7.2 What it left behind that it did not declare
|
||||
|
||||
**F-8 · Five orphaned `agent.json.*` backups survive, each carrying a hub API key and a Proxmox
|
||||
token.** `/etc/felhom-agent/` remains with `agent.json.campaign8-before`, `.campaign9-before`,
|
||||
`.campaign9-prev`, `.pre-e-target-move`, `.pre-prunegate.bak` — all 0600, three owned by the **now-deleted
|
||||
service user's uid 999**, each holding a 64-char `hub.api_key` and a 59-char `proxmox.token`.
|
||||
|
||||
The uninstall's own WIPED statement claims it removes *"config (+ its .bak backups)"*, and
|
||||
`scripts/CHANGELOG` records F1 as *"uninstall now purges the agent config's `.bak*` siblings (one held a
|
||||
live hub api_key)"*. **That fix does not match the filenames in use** — and note it misses
|
||||
`agent.json.pre-prunegate.bak`, a file that literally ends in `.bak`.
|
||||
|
||||
**Assessed rather than assumed: these particular secrets are SUPERSEDED, not live.** The orphaned
|
||||
`hub.api_key` hashes to `a5d2222a…`; the hub's current key for demo-hp is `8c59d1b6…`. The Proxmox token
|
||||
was deleted by this same uninstall. **The exposure is timing-dependent, not absent** — a backup taken
|
||||
shortly before an uninstall would hold a current key, which is exactly what R-179's precedent records.
|
||||
The broken thing is the stated guarantee.
|
||||
|
||||
**And the reinstall hands them to the new service account.** The rebuilt box recreates `felhom-agent` at
|
||||
**uid 999 — the same uid the deleted one had** — so the three backups owned by uid 999 become the new
|
||||
account's files. Verified: `su -s /bin/sh felhom-agent -c 'test -r …'` succeeds on
|
||||
`agent.json.campaign8-before` and `agent.json.pre-prunegate.bak`. A fresh install's service account
|
||||
inherits read access to the previous install's credentials. Superseded here; live if the backups are recent.
|
||||
|
||||
**F-9 · A stale sudoers file survives — and it is INERT. Hypothesis raised, then refuted by test.**
|
||||
`/etc/sudoers.d/felhom-agent.bak-pre-e2a` (21 833 bytes, 2026-07-28, 0440 root:root) survives the
|
||||
uninstall and grants `felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, …`,
|
||||
and `visudo -c -f` on it says **`parsed OK`**.
|
||||
|
||||
The obvious worry — that the reinstall recreates the `felhom-agent` user and the box silently regains a
|
||||
pre-E-2a privilege set — **was tested against the rebuilt box and is FALSE.** With the user recreated
|
||||
(uid 999) and this as the *only* felhom file in `/etc/sudoers.d`, `sudo -l -U felhom-agent` returns
|
||||
**"User felhom-agent is not allowed to run sudo on felhom-host."** sudo ignores files in `sudoers.d`
|
||||
whose names contain a dot, so `….bak-pre-e2a` is never loaded. That is a robust reason, not luck —
|
||||
`visudo -c -f <file>` checks a file explicitly and therefore does **not** tell you whether sudo loads it.
|
||||
|
||||
**What is left is rank 4:** 21 KB of the privilege grammar lingering on a host Felhom was told to leave,
|
||||
inside a teardown that claims to have removed the sudoers file.
|
||||
|
||||
**F-10 · The WireGuard tunnel to Felhom's off-site endpoint stays up.** After an uninstall,
|
||||
`wg-quick@wg-felhom` is **enabled and active**, `/etc/wireguard/wg-felhom.conf` present, with a live
|
||||
handshake to `167.233.158.164:443` 52 seconds old and 5.86 GiB received / 2.48 GiB sent on the counter.
|
||||
It appears in **neither** the WIPED nor the KEPT list. A box told to leave Felhom keeps a live inbound
|
||||
route into Felhom's infrastructure, and the operator is not told.
|
||||
|
||||
**F-11 · Smaller undeclared residue:** `/etc/felhom/` (`.bootstrap-done`, `appliance-pairing-code` —
|
||||
the console pairing code); `/etc/systemd/system/felhom-bootstrap.service` + `/usr/local/sbin/felhom-bootstrap.sh`;
|
||||
the `vmbr9` island-bridge stanza (169.254.253.1/30) still in `/etc/network/interfaces`.
|
||||
|
||||
**F-12 · A teardown-ordering warning:** `user config - ignore invalid acl role 'FelhomAgentStore'` —
|
||||
the role is deleted before an ACL that still references it. Harmless here; noise that would mask a real one.
|
||||
|
||||
---
|
||||
|
||||
## 8. P2 — the preflight, as the owner runs it
|
||||
|
||||
### 8.1 The refusals, judged on their messages
|
||||
|
||||
**First run, no caps:** `byo mode requires explicit --cores and --memory (noisy-neighbor caps on a host
|
||||
you do not own)`. **Good** — names the missing thing and the reason.
|
||||
|
||||
**Second run:** it refused again —
|
||||
|
||||
```
|
||||
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
|
||||
Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
|
||||
(Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
|
||||
```
|
||||
|
||||
As a *message* this is excellent: the finding, the evidence, two routes out, and an explicit promise
|
||||
about what it will not do.
|
||||
|
||||
### 8.2 F-13 (RANK 1 — STOPS THE VISIT) · Felhom's uninstall leaves the exact condition that makes Felhom's reinstall refuse
|
||||
|
||||
The resolver it is refusing over **is Felhom's own**, and the chain is fully evidenced:
|
||||
|
||||
| fact | evidence |
|
||||
|---|---|
|
||||
| Felhom installed `dnsmasq` | `/var/lib/dpkg/info/dnsmasq.list` dated **2026-07-21 18:24 CEST** — demo-hp's Felhom day-0 |
|
||||
| it was constrained by a Felhom snippet | `/etc/dnsmasq.d/` held Felhom snippets before; only `README` after |
|
||||
| **the uninstall restarted it unconstrained** | the running process started **2026-08-09 10:37:39 CEST — inside the uninstall window (10:37:23–10:38:23)** |
|
||||
| unconstrained, it grabs the wildcard | `0.0.0.0:53` and `[::]:53`, `--local-service`, empty `dnsmasq.conf` |
|
||||
| it is not PVE's | `/etc/pve/sdn/` empty; unit is stock `/usr/lib/systemd/system/dnsmasq.service`, **enabled** |
|
||||
|
||||
The uninstall mentions it only as *"the 'sudo' and 'dnsmasq' packages were left installed (**system
|
||||
packages**)"* — but dnsmasq is not a system package here; Felhom installed it, and it left it **enabled
|
||||
and running as a wildcard resolver**.
|
||||
|
||||
**What a customer would do next:** read a message that says a resolver is in the way, conclude their own
|
||||
network setup is at fault, and start debugging DNS they never configured. On a BYO box with no shell
|
||||
guidance, this is where the visit stops.
|
||||
|
||||
**Confirmed by counterfactual:** `systemctl stop dnsmasq && systemctl disable dnsmasq` → `host DNS (:53):
|
||||
free` → **PRE-FLIGHT PASS**. Nothing else changed. *(Recorded as an intervention — it is the action the
|
||||
message tells the owner to take.)*
|
||||
|
||||
### 8.3 F-14 (RANK 2) · A local golden is adopted with NO version and NO checksum check
|
||||
|
||||
Preflight reported `golden (local): local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst`. Step 7 is:
|
||||
|
||||
```sh
|
||||
if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then
|
||||
log_skip " using local golden: $GOLDEN_VOLID"
|
||||
_state_mark golden; return 0
|
||||
fi
|
||||
```
|
||||
|
||||
**The hub manifest's `golden.sha256` — the whole point of the hub vouching a different trust root than
|
||||
Gitea — is consulted only on the fetch path.** A locally-present archive bypasses the vouch entirely: no
|
||||
version comparison, no digest, no warning.
|
||||
|
||||
**What that archive actually contains, read out of the tar:**
|
||||
`gitea.dooplex.hu/admin/felhom-controller:`**`0.192.0`**, against a vouched golden of **0.210.0**.
|
||||
|
||||
Two consequences, and the second is the serious one:
|
||||
|
||||
1. The rebuilt box comes up **18 controller releases stale**, reporting success.
|
||||
2. **0.192.0 is below 0.200.0 — the release where R-193's off-site recovery SCREEN shipped.** So a
|
||||
customer reinstalled today returns on a controller that cannot run the recovery ceremony their data
|
||||
depends on. It is also born below the managed-update floor (0.200.0), and the updater's auto-target
|
||||
is the floor, never the newest — so it converges to 0.200.0, still not the vouched build.
|
||||
|
||||
**This is the R-111/R-115/R-120 drift family, one layer down:** the R-120 gate guards what the operator
|
||||
may *vouch*; nothing guards what an install actually *takes*. It also compounds with P1 — the uninstall
|
||||
deliberately keeps the old golden (*"golden vzdump left in place"*), and the reinstall then adopts it.
|
||||
|
||||
**The walk continues unflagged** (`--force-gitea-golden` NOT passed), because measuring the path a real
|
||||
reinstall takes is the point. §9 records which controller version actually came up.
|
||||
|
||||
### 8.4 The BYO mutation disclosure — read as a customer
|
||||
|
||||
Comprehensive and honest: every pveum artifact, the sudoers file, the state and config paths, the
|
||||
packages it will install, the guest's capped size, the self-update authority, and — creditably — the
|
||||
**outbound WireGuard tunnel**, in plain words, with the note that the operator can disable it per
|
||||
customer. It closes with `NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd.`
|
||||
|
||||
**One material omission, given §8.3:** it does not say that a pre-existing local golden may be used
|
||||
instead of the vouched one. The disclosure lists everything the install will *create* and nothing about
|
||||
what it may *reuse*.
|
||||
|
||||
---
|
||||
|
||||
## 9. P3 — the install. IT FAILED, AND THE WALK STOPS HERE
|
||||
|
||||
Started 08:51:33 UTC, **failed 08:52:17 UTC (44 s), exit 1, at step 5/8.** Run unflagged, as a real
|
||||
owner would. The typed hostname acknowledgement was given, not bypassed.
|
||||
|
||||
```
|
||||
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.210.0
|
||||
[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest
|
||||
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
|
||||
[OK] created service user felhom-agent
|
||||
curl: (22) The requested URL returned error: 404
|
||||
[ERROR] raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)
|
||||
```
|
||||
|
||||
### 9.1 F-15 (RANK 1 — STOPS EVERY INSTALL, FLEET-WIDE) · The vouched agent version was never tagged
|
||||
|
||||
**`v0.128.0` does not exist as a git tag** — not locally, not on Gitea. The newest tag is `v0.127.0`.
|
||||
The *binary* was published to the Gitea package registry (which is why it is selectable in the hub's
|
||||
artifact dropdown and why its sha256 verified), but the release's tag step never happened.
|
||||
|
||||
| ref | HTTP |
|
||||
|---|---|
|
||||
| `raw/branch/main/configs/felhom-mkfs-guarded.sh` | **200** |
|
||||
| `raw/tag/v0.127.0/configs/felhom-mkfs-guarded.sh` | **200** |
|
||||
| **`raw/tag/v0.128.0/configs/felhom-mkfs-guarded.sh`** | **404** |
|
||||
|
||||
The commit that *is* 0.128.0 exists and is on `main`: `28ba8593b8 — v0.128.0 — R-221: the escrow seed is
|
||||
asserted every tick, not remembered once`. Only the tag is missing.
|
||||
|
||||
**This is R-184 happening.** R-184 reads: *"Nothing prevents the hub from vouching an agent version that
|
||||
was never released. The R-115 gate proves every RELEASED version is installable, but it works from git
|
||||
tags."* It was filed as a gap. It is now a live outage.
|
||||
|
||||
**It is the collision of a good fix with an open gap.** R-183 correctly hardened the installer to pin
|
||||
config fetches to the vouched tag (`hostinstall_gates.py` even asserts the pin exists —
|
||||
`ok: fetch_raw pins the agent configs to the vouched agent version`). Nothing asserts that the pinned ref
|
||||
**resolves**. So the first vouch naming an untagged version fails every install closed.
|
||||
|
||||
**The release script already knows.** `felhom-agent/scripts/release-agent.sh:23` says in as many words:
|
||||
*"`raw/tag/v<version>`, a released version without a git tag 404s a box mid-install, as root."* The
|
||||
hazard was understood, documented, and the tag step was still missed.
|
||||
|
||||
**Blast radius — stated precisely, not inflated:**
|
||||
- **Every fresh install and every reinstall, on every host, fails at step 5/8** for as long as the Day-0
|
||||
manifest vouches agent 0.128.0. That manifest was approved on 2026-08-09; the approval is what armed it.
|
||||
- **Existing boxes are unaffected.** demo-felhom runs 0.128.0 happily — the self-update path takes the
|
||||
binary from the package registry, not from a tag.
|
||||
- So the damage is confined to the install path. That is also the only path a new customer, or a
|
||||
reinstalling tester, ever walks.
|
||||
|
||||
**What a customer would do next:** nothing. `curl: (22) … 404` and a question about a git tag is not a
|
||||
message a household can act on. The visit ends here and a developer is required.
|
||||
|
||||
### 9.2 The box's state, left as found for inspection
|
||||
|
||||
`--uninstall` is not re-run and no manual repair is attempted (rule 1). demo-hp is mid-install:
|
||||
|
||||
```
|
||||
state.json completed: [preflight, token, grows, enroll] mode: byo
|
||||
agent 0.128.0 installed, unit NOT created (not-found), inactive
|
||||
user felhom-agent uid=999 created
|
||||
pveum 2 users, 3 roles, 63 ACL lines, pool 'felhom' recreated (empty)
|
||||
guest none
|
||||
```
|
||||
|
||||
The documented `--resume` path exists and **was deliberately not used: it cannot help.** Resume replays
|
||||
from `enroll`, hits the same 404, and fails identically — the missing tag is not a transient.
|
||||
|
||||
### 9.3 The one-line unblock, for the operator to decide
|
||||
|
||||
```sh
|
||||
cd /mnt/5_hdd/felhom.eu/git/felhom-agent
|
||||
git tag v0.128.0 28ba8593b8 && git push origin v0.128.0
|
||||
```
|
||||
|
||||
This publishes the tag for a commit that is already on `main` and already released as a package — it
|
||||
completes a half-finished release rather than changing any code. The alternative, rolling the manifest
|
||||
back to agent 0.127.0, costs R-221 — the fix `STATUS.md` calls *"the last item blocking a customer from
|
||||
something we promise them"* — so it is the worse option.
|
||||
|
||||
**Not done in this session.** Publishing a release tag is an outward-facing act and the runbook is
|
||||
explicit: *"If a defect is so severe that the run cannot continue, stop and report; do not fix it here."*
|
||||
|
||||
### 9.4 What F-14 still owes
|
||||
|
||||
Step 7 never ran, so **whether the install would actually have adopted the stale 0.192.0 golden is not
|
||||
yet observed** — F-14 rests on the code path plus the preflight's `golden (local): …9100…2026_08_03…`
|
||||
line and the archive's own `felhom-controller:0.192.0` marker. That is strong, but it is not the
|
||||
observation. **When the tag is published and the walk resumes, the first thing to record at step 7/8 is
|
||||
which golden it takes.**
|
||||
|
||||
---
|
||||
|
||||
## 10. Where the run stands
|
||||
|
||||
**Reached: P1 ✓, P2 ✓, P3 ✗ (blocked).** P4–P7 not attempted.
|
||||
|
||||
**The honest answer to §2's question is: NOT YET ANSWERABLE, and the reason is itself the answer for
|
||||
today.** A machine wiped today cannot be set up again at all — not because the data is gone (it is
|
||||
safe, in snapshot `41c830db` and in two local tiers), but because the install refuses at step 5 of 8 on
|
||||
a missing git tag. The walk found a hard stop before it ever reached the question about data.
|
||||
|
||||
**The dataset is intact and waiting.** Nothing about Gate 0 needs redoing when the walk resumes:
|
||||
snapshot `41c830db` holds all four files, the before-manifest is committed, and the comparator is proven.
|
||||
|
||||
Reference in New Issue
Block a user