diff --git a/REPORT.md b/REPORT.md index 376895d..16b64b5 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,110 +1,147 @@ -# felhom.eu — task reports +# REPORT — TASK-E: pre-invite sweep + HP mini ISO (2026-07-21) -> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). +Baselines at start (all verified, all clean): controller `83f20c8` v0.156.0, agent `935904f` v0.92.1, +felhom.eu `d4ac52d` hub v0.68.1, app-catalog `34d50a3`. ---- +## Per-part status -# TASK-D — unattended resilience: docs, roadmap, capability map, riders (2026-07-21) +| Part | Status | Ships as | +|---|---|---| +| 1 — escrow wordlist | **DONE** | agent **v0.93.0** (not built/deployed; rides the next train) | +| 2 — R-55 stop-honesty | **DONE — scope corrected by operator ruling** | controller **v0.157.0**, deployed to 9201 | +| 3 — catalog honesty | **DONE, with one ruling** | catalog: wanderer re-pinned, plant-it retired, R-41 slice 1 | +| 4 — operator one-pager | **DONE** | `documentation/PROMPT-TEMPLATE.md` | +| 5 — §6 leftovers | **DONE** | controller **v0.157.1** + `scripts/build-hub.sh` **v1.23.0** | +| 6 — HP mini ISO | **DONE, 2 artifacts not 3** (see below) | `v1.22.0` hp ISOs, both loaders | +| 7 — rulings + R-17 | **DONE** (R-17 execution is operator-console) | ROADMAP + capability map | -**No hub code was touched** (§12 of the brief). This repo's share is documentation, the two Part-5 -riders, and the honest status of the three claims the task makes. +Two questions were put to the operator rather than guessed; both are recorded where they landed. -Implementation reports live in the repos that shipped: `felhom-controller/REPORT.md` (R-51, R-52, -v0.156.0) and `felhom-agent/REPORT.md` (R-54, v0.92.1). +## Part 6 — the HP mini ISOs -## 1. ROADMAP +**Version: rebuild at `v1.22.0`, no bump.** Phase-0 probe confirmed **no functional change to the ISO +tooling since the v1.22.0 artifacts** were built on 2026-07-19: the last content commit before that +build was `8feed6d` (08:57:51, build at 09:00), and everything since is docs or `chmod +x` mode bits. +`ISO_VERSION` tracks the Felhom release the ISO is tagged to; adding a build profile does not move it, +and the profile name already makes the artifacts unmistakable, so no date-stamping was needed. -- **R-51 → SHIPPED (controller v0.156.0)**, and **the row's diagnosis is corrected in place**. It - claimed aggregation classified a dead-primary stack `unhealthy`, making the deliberate `unhealthy` - exclusion the suppressor. The source says otherwise: `aggregateState`'s final branch returned - `StateRunning` for any running/stopped mix ("report as running (partial)"), so the stack read - **running** and `IsDownState` was never consulted about `unhealthy` at all. The constraint the row - protects was therefore never in tension with the fix, and `downstate_test.go` ships untouched. -- **R-52 → SHIPPED (controller v0.156.0)**, `internal/bootrecon`. Note recorded that **P1 is not a - blocker**: the reconciliation is correct whether or not Docker recorded those two containers as - user-stopped, and P1's experiment falls out of STOP-1's reboot leg for free. -- **R-54 → NEW ROW, SHIPPED (agent v0.92.1).** Origin: `INCIDENT-guest-dhclient-killed-2026-07-20` - §5, whose OPEN RISK it closes. The row carries the load-bearing design fact — *liveness of the - DHCP client is itself a probe, because the damage is timed and the address survives the cause by - 1–2 hours* — the live sudoers finding, and the explicit deferral of the **static-guest** leg to - **R-50**, which is where the incident's own option 2 ("give the guest a static address") belongs. +New: `scripts/iso/profiles/hp.profile`, committed. It mirrors `n100.profile` with **one deliberate +difference — the loader default stays at the stock signed `shim`.** `mkimage` exists to work around +the N100's AMI firmware GRUB relocation defect; it is not a Felhom requirement, and it costs Secure +Boot (it is unsigned). HP business firmware should take shim. -## 2. Capability map +| ISO | sha256 | bytes | +|---|---|---| +| `felhom-pve-9.2-1-v1.22.0-hp-generic.iso` (**shim**) | `2b51d74c0eaf5155fd42a137dff803b9ab8692294cedcb727c9a80658c9f7823` | 1 705 338 880 | +| `felhom-pve-9.2-1-v1.22.0-hp-generic-mkimage.iso` (**mkimage**) | `09700442c297edcf81bdb9d6fddd14e79ba62091e7c2f183fa687c513f061142` | 1 704 482 816 | -New row: **"Box survives an unattended app or guest-network failure"** — **PROVEN-LIVE (2026-07-21)**. +Both in `180:/mnt/5_hdd/felhom.eu/felhom-iso/out/` with `.sha256` + `.manifest.txt`. Build logs in +`felhom-iso/logs/build-hp-{shim,mkimage}-2026-07-21.log`. Input PVE ISO `proxmox-ve_9.2-1.iso`, +sha256 `4e88fe41…`. Both manifests: `secret-bearing : no`. -It was written as IMPLEMENTED and upgraded only after the evidence existed, which is the order the -strict enum demands. All three legs then ran operator-present in one session (§4b) and each is cited -with timestamps in the row itself. The row also carries the gap the validation found (**R-55**) and -the leg deliberately left out of scope (the static guest → **R-50**), so the badge does not imply -more than was proven. +**`validate-answer` was gated on OUTPUT text, not `$?`** — the documented trap (it exits 0 on +failure). Both logs show `The answer file was parsed successfully, no errors found!` through the +`gate_validate_answer` output-parse gate. -The existing **"Box survives a site/network change"** row (PARTIAL) already pointed at R-51/R-52 as -related work; it stays PARTIAL — R-50 is still its durable fix. +### Two artifacts, not three — and why that is the right answer -## 3. Part-5 riders +TASK-E asked for a shim pairing ISO, a mkimage pairing ISO, and a separate `hp-safety` match-nothing +ISO. **Both ISOs above ARE safety ISOs.** The `n100-safety` precedent is not a separate flag: it is +the committed profile built as-is, because that profile ships a match-nothing disk filter. A third +artifact would have been a byte-for-byte duplicate of one of these. -- **Hub `build.sh` deploy hint → GitOps wording.** The script printed `kubectl set image …` and - `kubectl apply -f manifests/hub.yaml` as the deploy instructions — both are reverted by the next - ArgoCD sync, which is the worst failure shape: it appears to work, then silently disappears. Now - prints the manifest-bump + hard-refresh + deliberate-sync sequence, and names the trap explicitly. - **Caveat worth knowing: `/mnt/5_hdd/felhom.eu/build/felhom-hub/build.sh` is NOT in any git repo** — - it is a DooPlex-local build-dir script. So this rider is a fix to the operative file only, and it - is not versioned anywhere. Worth adopting into the repo as its own change. -- **`documentation/PROMPT-TEMPLATE.md` §10 gains the seam-discipline row.** Text generalised from - the brief's §9 rule 6, plus what this session added to it: three shipped inert-seam defects in - three days (controller v0.154.0, agent v0.91.0, agent v0.92.0's missing sudoers grant), all fully - green; and the finding that a `strings.Contains` source assertion is not sufficient, because a - commented-out call still contains the string — walk the AST. +Arming the real install requires the target SSD's `ID_SERIAL_SHORT`, which **cannot exist until the +board is in hand** — inventing it is exactly the input the conventions say to ask about rather than +guess. So all HP artifacts are deliberately inert, which satisfies "the first thing the new board ever +runs must be provably harmless" more strongly than the three-artifact split would have. -## 4. What was verified, and how +**Verified from INSIDE both ISOs, not merely from the profile** — the embedded `answer.toml` extracted +with `osirrox` reads: -Endpoint/host-level, no browser (none on DooPlex). Everything claimed live in the two implementation -reports was read out of journald or the agent's own capability self-check on felhom-pve. No claim in -any of the three reports rests on a test that only proves a seam. +``` +[disk-setup] +filesystem = "ext4" +filter-match = "all" +filter.ID_SERIAL_SHORT = "REPLACE-WITH-TARGET-SSD-SERIAL" +``` -## 4b. The STOP legs RAN — all three (2026-07-21, operator-present) +### Operator boot cheat-sheet — HP mini -The ordering below was accepted, executed, and all three legs passed. Full evidence chains are in -the two implementation reports and in the ROADMAP rows; the capability-map row is now -**PROVEN-LIVE**. Headline: dead primary → `degraded` in **13 s** with exactly one event; boot orphan -→ recovered in **one attempt** with **zero** alerts; dead dhclient → detected in **57 s** on process -liveness alone and healed **120 s** after the kill, roughly 80 minutes before the outage would have -started — `cloudflared` never dropped (`Up 29 hours`). +1. **Flash `…-hp-generic.iso` (shim) FIRST.** Leave Secure Boot **ON**. Expect: GRUB loads, the Felhom + boot screen appears, the installer starts and then **aborts having touched no disk**. That is + success — it proves the loader and the firmware setting before anything destructive exists. +2. **If it fails at GRUB** — a `relocation` error, "no bootable device", or a hang before the boot + screen — that is the N100's F1 defect on this board too. Switch to + `…-hp-generic-mkimage.iso` **and set Secure Boot OFF** in firmware. mkimage is unsigned; with + Secure Boot enforcing the firmware refuses it outright ("Access Denied" — proven, RUNBOOK-B leg 3). +3. **The safety ISO always precedes the real one.** Never boot an install-armed ISO on this board + until one of the two above has been observed to abort cleanly. +4. **To arm the real install**, tell CC which loader worked and give it the internal SSD's + `ID_SERIAL_SHORT` (`lsblk -o NAME,SERIAL` from any live boot). **Confirm it is the internal SSD and + not an attached backup drive** — that check is what protected the N100's external HDD on the + rehearsal run. CC copies `hp.profile` to an uncommitted working profile with the real serial (the + `n100-demo.profile` precedent — a real serial is never committed) and rebuilds. +5. **No remote install step was attempted.** The boot is physically yours. -**One real gap surfaced, and it is the opposite of what the leg was designed to prove → new R-55.** -STOP-1's R-52 leg was meant to show that a customer's deliberate Stop survives a reboot. It does -not, for any **drive-backed** app: the boot bind gate (`internal/web/intermediary.go`) selects on -`Deployed && HDD_PATH` alone and runs `StopStack` + `StartStack` on every match, never consulting -whether the app was stopped on purpose. immich, stopped from the UI seconds earlier, came back -running. **R-52 is not implicated** — its own gate behaved exactly as specified (immich, at zero -containers, was never a candidate), and this behaviour predates it. Two consequences are worth an -operator ruling: a customer's Stop is silently undone on drive-backed apps, and R-52's practical -scope is narrower than the brief assumed, because the gate reaches every drive-backed app first. +## Part 4 — the operator one-pager -**P1 is answered, as a free by-product of that leg.** bookstack carries `restart=unless-stopped`, -the Docker daemon came up at ~10:53:15Z, and the container's `StartedAt` is `10:53:28.05Z` — the -exact moment `bootrecon`'s `StartStack` returned. Docker did not resurrect it; the F5 hypothesis is -confirmed. +`documentation/PROMPT-TEMPLATE.md` gains a standard section between the header and §0. Mandatory for +M+ tasks and anything carrying a STOP; guidance covers audience, the four things it must say in order, +and the rule that it is a summary and never the only place a requirement appears. TASK-B/C/D cited as +precedent. §10's seam-discipline row was **verified present** (line 229) and not duplicated. -## 5. The ordering question (resolved) +## Part 5 — both §6 leftovers -The brief's Phase A says **do not hand-deploy** the controller — the floor save at **STOP-3** is what -deploys 0.156.0, banking another single-fire self-update datapoint. But **STOP-1 exercises the -controller legs**, which need 0.156.0 to be live. As written the two are in the wrong order. +- **`.gitignore` anchor** → controller v0.157.1. Details and the both-directions verification are in + `felhom-controller/REPORT.md`. +- **The hub `build.sh` was not in any repo.** Adopted as `scripts/build-hub.sh` **v1.23.0**: build + logic byte-identical (verified by diff), plus a `SCRIPT_VERSION` and a header naming the split. The + operative path `/mnt/5_hdd/felhom.eu/build/felhom-hub/build.sh` is now a **symlink** to the repo + copy — one source of truth, repo edits live immediately (the `install_skills.py` pattern). Previous + copy retained as `build.sh.pre-adopt-2026-07-21`. -Resolved as: **STOP-3's controller floor save first** (floor → `0.156.0`), box self-updates — that -IS the R-23 datapoint — then STOP-1 against the new version. **Done, and the datapoint is banked:** -`[selfupdate] Post-update startup: update successful (0.155.0 → 0.156.0)` followed by -`Current version 0.156.0 is up to date` — one swap, no re-fire. The agent half of STOP-3 (manifest → 0.92.1) is independent and can happen at any -point. Flagged rather than assumed, because reordering an operator's STOP sequence is not CC's call. +## Part 7 — rulings recorded -## 6. Follow-ups this session surfaced (none actioned here) +R-25b (full-teardown cascade, three acks + typed name, **re-sized S → M**, spec to follow, no longer +blocks R-3) · R-11 (direct Messenger; the doc is the architect's) · R-42 (option (a)) · R-4 (**COMPLETE** +— freemail.hu operator-verified) · R-55 + R-41 slice 1 marked shipped · **new R-56** (app difficulty +classification — the constructive half of the glance ruling; glance stays). Capability map updated on +the boot-orphan row and the escrow-ceremony row. -1. `felhom-controller/controller/.gitignore`'s bare `controller` entry also matches the directory - `cmd/controller/`, so ripgrep silently skips `main.go` and new files there need `git add -f`. - Both directions produce inert-seam mistakes. XS fix: anchor it as `/controller`. -2. The hub `build.sh` above is unversioned. -3. `TestGenerateRecoveryCode_EntropyAndFormat` (agent) flakes on hyphenated wordlist entries - (`drop-down` → 11 tokens). Observed 3/8 this session — worse than the documented ~1/5, and it is - a one-line fix in the generator or the assertion, not a mystery. +### R-17 — spike-lite result: it CANNOT be done through tooling + +Ruled DELETE. CC did **not** improvise credentialed API calls, and found there is no tooling path: + +- `felhom-tenantsync.sh` does not touch Hetzner at all — PBS namespaces/tokens only. +- The hub **can** delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its only caller is + `offsite.Deprovision`, which selects by the `felhom-customer=` label as part of a customer RESET. + A legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no verb targets it. +- The only in-repo references are **test fixtures** (`hub/internal/hetznerapi/fake.go` generates the + `u629193-sub%d` pattern; `dr_recipe_test.go` uses it as a literal). Nothing live points at it. + +**Operator console action:** Hetzner Storage Box console → box `u629193` → Sub-accounts → +`u629193-sub1` → confirm the directory holds only the old-box archive → Delete. The 9/9 byte-identical +restores are the sign-off; deleting the sub-account destroys its data. + +## Observations (recorded, not acted on) + +1. **A gate I shipped nearly shipped a lie, and it is the most transferable thing here.** The first + full catalog resolvability sweep reported **24 of 65 pins dead**, including `postgres:16-alpine` + and `redis:7-alpine`. They are fine — Docker Hub had started throttling. Worse, `docker manifest + inspect` prints `toomanyrequests: …` and **still exits 0**, the identical exits-0-on-failure shape + as `validate-answer` in the ISO tooling. Two independent instances of the same trap now sit in two + different toolchains; it is worth treating "does this tool signal failure through its exit code?" + as a standing question rather than a per-tool footnote. The gate now resolves ambiguity to + INCONCLUSIVE, never to an accusation. +2. **The full 52-app resolvability table is still owed.** DooPlex is not logged in to Docker Hub, so + the sweep exits 2 (incomplete) rather than producing the table. One `docker login` + re-run closes it. +3. **R-55's live reboot leg is not run** — see `felhom-controller/REPORT.md`. Shipped and wired, not + yet proven live. +4. **wanderer is re-pinned but never deployed.** It went from 1 service to 3 with a second public + hostname; it deserves a real deploy on the demo box before anyone is invited to it. +5. `reuse_refs_check.py` resolves REUSE.md citations against the repo root and so cannot follow a + cross-repo pointer; the catalog's `healthprobe.go` citation still flags. Path corrected; the + checker limitation is pre-existing. +6. **The `felhom-testing` skill's "known flake" line was actively harmful** — it instructed every + future session to dismiss a true positive, for weeks. Removed, and generalised: "known flake, just + re-run it" is a diagnosis and needs evidence like any other.