docs(report): TASK-E wrap-up - per-part status, HP ISO hashes + boot cheat-sheet, R-17 spike-lite
This commit is contained in:
@@ -1,110 +1,147 @@
|
|||||||
# felhom.eu — task reports
|
# REPORT — TASK-E: pre-invite sweep + HP mini ISO (2026-07-21)
|
||||||
|
|
||||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
Baselines at start (all verified, all clean): controller `83f20c8` v0.156.0, agent `935904f` v0.92.1,
|
||||||
|
felhom.eu `d4ac52d` hub v0.68.1, app-catalog `34d50a3`.
|
||||||
|
|
||||||
---
|
## Per-part status
|
||||||
|
|
||||||
# TASK-D — unattended resilience: docs, roadmap, capability map, riders (2026-07-21)
|
| Part | Status | Ships as |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 — escrow wordlist | **DONE** | agent **v0.93.0** (not built/deployed; rides the next train) |
|
||||||
|
| 2 — R-55 stop-honesty | **DONE — scope corrected by operator ruling** | controller **v0.157.0**, deployed to 9201 |
|
||||||
|
| 3 — catalog honesty | **DONE, with one ruling** | catalog: wanderer re-pinned, plant-it retired, R-41 slice 1 |
|
||||||
|
| 4 — operator one-pager | **DONE** | `documentation/PROMPT-TEMPLATE.md` |
|
||||||
|
| 5 — §6 leftovers | **DONE** | controller **v0.157.1** + `scripts/build-hub.sh` **v1.23.0** |
|
||||||
|
| 6 — HP mini ISO | **DONE, 2 artifacts not 3** (see below) | `v1.22.0` hp ISOs, both loaders |
|
||||||
|
| 7 — rulings + R-17 | **DONE** (R-17 execution is operator-console) | ROADMAP + capability map |
|
||||||
|
|
||||||
**No hub code was touched** (§12 of the brief). This repo's share is documentation, the two Part-5
|
Two questions were put to the operator rather than guessed; both are recorded where they landed.
|
||||||
riders, and the honest status of the three claims the task makes.
|
|
||||||
|
|
||||||
Implementation reports live in the repos that shipped: `felhom-controller/REPORT.md` (R-51, R-52,
|
## Part 6 — the HP mini ISOs
|
||||||
v0.156.0) and `felhom-agent/REPORT.md` (R-54, v0.92.1).
|
|
||||||
|
|
||||||
## 1. ROADMAP
|
**Version: rebuild at `v1.22.0`, no bump.** Phase-0 probe confirmed **no functional change to the ISO
|
||||||
|
tooling since the v1.22.0 artifacts** were built on 2026-07-19: the last content commit before that
|
||||||
|
build was `8feed6d` (08:57:51, build at 09:00), and everything since is docs or `chmod +x` mode bits.
|
||||||
|
`ISO_VERSION` tracks the Felhom release the ISO is tagged to; adding a build profile does not move it,
|
||||||
|
and the profile name already makes the artifacts unmistakable, so no date-stamping was needed.
|
||||||
|
|
||||||
- **R-51 → SHIPPED (controller v0.156.0)**, and **the row's diagnosis is corrected in place**. It
|
New: `scripts/iso/profiles/hp.profile`, committed. It mirrors `n100.profile` with **one deliberate
|
||||||
claimed aggregation classified a dead-primary stack `unhealthy`, making the deliberate `unhealthy`
|
difference — the loader default stays at the stock signed `shim`.** `mkimage` exists to work around
|
||||||
exclusion the suppressor. The source says otherwise: `aggregateState`'s final branch returned
|
the N100's AMI firmware GRUB relocation defect; it is not a Felhom requirement, and it costs Secure
|
||||||
`StateRunning` for any running/stopped mix ("report as running (partial)"), so the stack read
|
Boot (it is unsigned). HP business firmware should take shim.
|
||||||
**running** and `IsDownState` was never consulted about `unhealthy` at all. The constraint the row
|
|
||||||
protects was therefore never in tension with the fix, and `downstate_test.go` ships untouched.
|
|
||||||
- **R-52 → SHIPPED (controller v0.156.0)**, `internal/bootrecon`. Note recorded that **P1 is not a
|
|
||||||
blocker**: the reconciliation is correct whether or not Docker recorded those two containers as
|
|
||||||
user-stopped, and P1's experiment falls out of STOP-1's reboot leg for free.
|
|
||||||
- **R-54 → NEW ROW, SHIPPED (agent v0.92.1).** Origin: `INCIDENT-guest-dhclient-killed-2026-07-20`
|
|
||||||
§5, whose OPEN RISK it closes. The row carries the load-bearing design fact — *liveness of the
|
|
||||||
DHCP client is itself a probe, because the damage is timed and the address survives the cause by
|
|
||||||
1–2 hours* — the live sudoers finding, and the explicit deferral of the **static-guest** leg to
|
|
||||||
**R-50**, which is where the incident's own option 2 ("give the guest a static address") belongs.
|
|
||||||
|
|
||||||
## 2. Capability map
|
| ISO | sha256 | bytes |
|
||||||
|
|---|---|---|
|
||||||
|
| `felhom-pve-9.2-1-v1.22.0-hp-generic.iso` (**shim**) | `2b51d74c0eaf5155fd42a137dff803b9ab8692294cedcb727c9a80658c9f7823` | 1 705 338 880 |
|
||||||
|
| `felhom-pve-9.2-1-v1.22.0-hp-generic-mkimage.iso` (**mkimage**) | `09700442c297edcf81bdb9d6fddd14e79ba62091e7c2f183fa687c513f061142` | 1 704 482 816 |
|
||||||
|
|
||||||
New row: **"Box survives an unattended app or guest-network failure"** — **PROVEN-LIVE (2026-07-21)**.
|
Both in `180:/mnt/5_hdd/felhom.eu/felhom-iso/out/` with `.sha256` + `.manifest.txt`. Build logs in
|
||||||
|
`felhom-iso/logs/build-hp-{shim,mkimage}-2026-07-21.log`. Input PVE ISO `proxmox-ve_9.2-1.iso`,
|
||||||
|
sha256 `4e88fe41…`. Both manifests: `secret-bearing : no`.
|
||||||
|
|
||||||
It was written as IMPLEMENTED and upgraded only after the evidence existed, which is the order the
|
**`validate-answer` was gated on OUTPUT text, not `$?`** — the documented trap (it exits 0 on
|
||||||
strict enum demands. All three legs then ran operator-present in one session (§4b) and each is cited
|
failure). Both logs show `The answer file was parsed successfully, no errors found!` through the
|
||||||
with timestamps in the row itself. The row also carries the gap the validation found (**R-55**) and
|
`gate_validate_answer` output-parse gate.
|
||||||
the leg deliberately left out of scope (the static guest → **R-50**), so the badge does not imply
|
|
||||||
more than was proven.
|
|
||||||
|
|
||||||
The existing **"Box survives a site/network change"** row (PARTIAL) already pointed at R-51/R-52 as
|
### Two artifacts, not three — and why that is the right answer
|
||||||
related work; it stays PARTIAL — R-50 is still its durable fix.
|
|
||||||
|
|
||||||
## 3. Part-5 riders
|
TASK-E asked for a shim pairing ISO, a mkimage pairing ISO, and a separate `hp-safety` match-nothing
|
||||||
|
ISO. **Both ISOs above ARE safety ISOs.** The `n100-safety` precedent is not a separate flag: it is
|
||||||
|
the committed profile built as-is, because that profile ships a match-nothing disk filter. A third
|
||||||
|
artifact would have been a byte-for-byte duplicate of one of these.
|
||||||
|
|
||||||
- **Hub `build.sh` deploy hint → GitOps wording.** The script printed `kubectl set image …` and
|
Arming the real install requires the target SSD's `ID_SERIAL_SHORT`, which **cannot exist until the
|
||||||
`kubectl apply -f manifests/hub.yaml` as the deploy instructions — both are reverted by the next
|
board is in hand** — inventing it is exactly the input the conventions say to ask about rather than
|
||||||
ArgoCD sync, which is the worst failure shape: it appears to work, then silently disappears. Now
|
guess. So all HP artifacts are deliberately inert, which satisfies "the first thing the new board ever
|
||||||
prints the manifest-bump + hard-refresh + deliberate-sync sequence, and names the trap explicitly.
|
runs must be provably harmless" more strongly than the three-artifact split would have.
|
||||||
**Caveat worth knowing: `/mnt/5_hdd/felhom.eu/build/felhom-hub/build.sh` is NOT in any git repo** —
|
|
||||||
it is a DooPlex-local build-dir script. So this rider is a fix to the operative file only, and it
|
|
||||||
is not versioned anywhere. Worth adopting into the repo as its own change.
|
|
||||||
- **`documentation/PROMPT-TEMPLATE.md` §10 gains the seam-discipline row.** Text generalised from
|
|
||||||
the brief's §9 rule 6, plus what this session added to it: three shipped inert-seam defects in
|
|
||||||
three days (controller v0.154.0, agent v0.91.0, agent v0.92.0's missing sudoers grant), all fully
|
|
||||||
green; and the finding that a `strings.Contains` source assertion is not sufficient, because a
|
|
||||||
commented-out call still contains the string — walk the AST.
|
|
||||||
|
|
||||||
## 4. What was verified, and how
|
**Verified from INSIDE both ISOs, not merely from the profile** — the embedded `answer.toml` extracted
|
||||||
|
with `osirrox` reads:
|
||||||
|
|
||||||
Endpoint/host-level, no browser (none on DooPlex). Everything claimed live in the two implementation
|
```
|
||||||
reports was read out of journald or the agent's own capability self-check on felhom-pve. No claim in
|
[disk-setup]
|
||||||
any of the three reports rests on a test that only proves a seam.
|
filesystem = "ext4"
|
||||||
|
filter-match = "all"
|
||||||
|
filter.ID_SERIAL_SHORT = "REPLACE-WITH-TARGET-SSD-SERIAL"
|
||||||
|
```
|
||||||
|
|
||||||
## 4b. The STOP legs RAN — all three (2026-07-21, operator-present)
|
### Operator boot cheat-sheet — HP mini
|
||||||
|
|
||||||
The ordering below was accepted, executed, and all three legs passed. Full evidence chains are in
|
1. **Flash `…-hp-generic.iso` (shim) FIRST.** Leave Secure Boot **ON**. Expect: GRUB loads, the Felhom
|
||||||
the two implementation reports and in the ROADMAP rows; the capability-map row is now
|
boot screen appears, the installer starts and then **aborts having touched no disk**. That is
|
||||||
**PROVEN-LIVE**. Headline: dead primary → `degraded` in **13 s** with exactly one event; boot orphan
|
success — it proves the loader and the firmware setting before anything destructive exists.
|
||||||
→ recovered in **one attempt** with **zero** alerts; dead dhclient → detected in **57 s** on process
|
2. **If it fails at GRUB** — a `relocation` error, "no bootable device", or a hang before the boot
|
||||||
liveness alone and healed **120 s** after the kill, roughly 80 minutes before the outage would have
|
screen — that is the N100's F1 defect on this board too. Switch to
|
||||||
started — `cloudflared` never dropped (`Up 29 hours`).
|
`…-hp-generic-mkimage.iso` **and set Secure Boot OFF** in firmware. mkimage is unsigned; with
|
||||||
|
Secure Boot enforcing the firmware refuses it outright ("Access Denied" — proven, RUNBOOK-B leg 3).
|
||||||
|
3. **The safety ISO always precedes the real one.** Never boot an install-armed ISO on this board
|
||||||
|
until one of the two above has been observed to abort cleanly.
|
||||||
|
4. **To arm the real install**, tell CC which loader worked and give it the internal SSD's
|
||||||
|
`ID_SERIAL_SHORT` (`lsblk -o NAME,SERIAL` from any live boot). **Confirm it is the internal SSD and
|
||||||
|
not an attached backup drive** — that check is what protected the N100's external HDD on the
|
||||||
|
rehearsal run. CC copies `hp.profile` to an uncommitted working profile with the real serial (the
|
||||||
|
`n100-demo.profile` precedent — a real serial is never committed) and rebuilds.
|
||||||
|
5. **No remote install step was attempted.** The boot is physically yours.
|
||||||
|
|
||||||
**One real gap surfaced, and it is the opposite of what the leg was designed to prove → new R-55.**
|
## Part 4 — the operator one-pager
|
||||||
STOP-1's R-52 leg was meant to show that a customer's deliberate Stop survives a reboot. It does
|
|
||||||
not, for any **drive-backed** app: the boot bind gate (`internal/web/intermediary.go`) selects on
|
|
||||||
`Deployed && HDD_PATH` alone and runs `StopStack` + `StartStack` on every match, never consulting
|
|
||||||
whether the app was stopped on purpose. immich, stopped from the UI seconds earlier, came back
|
|
||||||
running. **R-52 is not implicated** — its own gate behaved exactly as specified (immich, at zero
|
|
||||||
containers, was never a candidate), and this behaviour predates it. Two consequences are worth an
|
|
||||||
operator ruling: a customer's Stop is silently undone on drive-backed apps, and R-52's practical
|
|
||||||
scope is narrower than the brief assumed, because the gate reaches every drive-backed app first.
|
|
||||||
|
|
||||||
**P1 is answered, as a free by-product of that leg.** bookstack carries `restart=unless-stopped`,
|
`documentation/PROMPT-TEMPLATE.md` gains a standard section between the header and §0. Mandatory for
|
||||||
the Docker daemon came up at ~10:53:15Z, and the container's `StartedAt` is `10:53:28.05Z` — the
|
M+ tasks and anything carrying a STOP; guidance covers audience, the four things it must say in order,
|
||||||
exact moment `bootrecon`'s `StartStack` returned. Docker did not resurrect it; the F5 hypothesis is
|
and the rule that it is a summary and never the only place a requirement appears. TASK-B/C/D cited as
|
||||||
confirmed.
|
precedent. §10's seam-discipline row was **verified present** (line 229) and not duplicated.
|
||||||
|
|
||||||
## 5. The ordering question (resolved)
|
## Part 5 — both §6 leftovers
|
||||||
|
|
||||||
The brief's Phase A says **do not hand-deploy** the controller — the floor save at **STOP-3** is what
|
- **`.gitignore` anchor** → controller v0.157.1. Details and the both-directions verification are in
|
||||||
deploys 0.156.0, banking another single-fire self-update datapoint. But **STOP-1 exercises the
|
`felhom-controller/REPORT.md`.
|
||||||
controller legs**, which need 0.156.0 to be live. As written the two are in the wrong order.
|
- **The hub `build.sh` was not in any repo.** Adopted as `scripts/build-hub.sh` **v1.23.0**: build
|
||||||
|
logic byte-identical (verified by diff), plus a `SCRIPT_VERSION` and a header naming the split. The
|
||||||
|
operative path `/mnt/5_hdd/felhom.eu/build/felhom-hub/build.sh` is now a **symlink** to the repo
|
||||||
|
copy — one source of truth, repo edits live immediately (the `install_skills.py` pattern). Previous
|
||||||
|
copy retained as `build.sh.pre-adopt-2026-07-21`.
|
||||||
|
|
||||||
Resolved as: **STOP-3's controller floor save first** (floor → `0.156.0`), box self-updates — that
|
## Part 7 — rulings recorded
|
||||||
IS the R-23 datapoint — then STOP-1 against the new version. **Done, and the datapoint is banked:**
|
|
||||||
`[selfupdate] Post-update startup: update successful (0.155.0 → 0.156.0)` followed by
|
|
||||||
`Current version 0.156.0 is up to date` — one swap, no re-fire. The agent half of STOP-3 (manifest → 0.92.1) is independent and can happen at any
|
|
||||||
point. Flagged rather than assumed, because reordering an operator's STOP sequence is not CC's call.
|
|
||||||
|
|
||||||
## 6. Follow-ups this session surfaced (none actioned here)
|
R-25b (full-teardown cascade, three acks + typed name, **re-sized S → M**, spec to follow, no longer
|
||||||
|
blocks R-3) · R-11 (direct Messenger; the doc is the architect's) · R-42 (option (a)) · R-4 (**COMPLETE**
|
||||||
|
— freemail.hu operator-verified) · R-55 + R-41 slice 1 marked shipped · **new R-56** (app difficulty
|
||||||
|
classification — the constructive half of the glance ruling; glance stays). Capability map updated on
|
||||||
|
the boot-orphan row and the escrow-ceremony row.
|
||||||
|
|
||||||
1. `felhom-controller/controller/.gitignore`'s bare `controller` entry also matches the directory
|
### R-17 — spike-lite result: it CANNOT be done through tooling
|
||||||
`cmd/controller/`, so ripgrep silently skips `main.go` and new files there need `git add -f`.
|
|
||||||
Both directions produce inert-seam mistakes. XS fix: anchor it as `/controller`.
|
Ruled DELETE. CC did **not** improvise credentialed API calls, and found there is no tooling path:
|
||||||
2. The hub `build.sh` above is unversioned.
|
|
||||||
3. `TestGenerateRecoveryCode_EntropyAndFormat` (agent) flakes on hyphenated wordlist entries
|
- `felhom-tenantsync.sh` does not touch Hetzner at all — PBS namespaces/tokens only.
|
||||||
(`drop-down` → 11 tokens). Observed 3/8 this session — worse than the documented ~1/5, and it is
|
- The hub **can** delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its only caller is
|
||||||
a one-line fix in the generator or the assertion, not a mystery.
|
`offsite.Deprovision`, which selects by the `felhom-customer=<id>` label as part of a customer RESET.
|
||||||
|
A legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no verb targets it.
|
||||||
|
- The only in-repo references are **test fixtures** (`hub/internal/hetznerapi/fake.go` generates the
|
||||||
|
`u629193-sub%d` pattern; `dr_recipe_test.go` uses it as a literal). Nothing live points at it.
|
||||||
|
|
||||||
|
**Operator console action:** Hetzner Storage Box console → box `u629193` → Sub-accounts →
|
||||||
|
`u629193-sub1` → confirm the directory holds only the old-box archive → Delete. The 9/9 byte-identical
|
||||||
|
restores are the sign-off; deleting the sub-account destroys its data.
|
||||||
|
|
||||||
|
## Observations (recorded, not acted on)
|
||||||
|
|
||||||
|
1. **A gate I shipped nearly shipped a lie, and it is the most transferable thing here.** The first
|
||||||
|
full catalog resolvability sweep reported **24 of 65 pins dead**, including `postgres:16-alpine`
|
||||||
|
and `redis:7-alpine`. They are fine — Docker Hub had started throttling. Worse, `docker manifest
|
||||||
|
inspect` prints `toomanyrequests: …` and **still exits 0**, the identical exits-0-on-failure shape
|
||||||
|
as `validate-answer` in the ISO tooling. Two independent instances of the same trap now sit in two
|
||||||
|
different toolchains; it is worth treating "does this tool signal failure through its exit code?"
|
||||||
|
as a standing question rather than a per-tool footnote. The gate now resolves ambiguity to
|
||||||
|
INCONCLUSIVE, never to an accusation.
|
||||||
|
2. **The full 52-app resolvability table is still owed.** DooPlex is not logged in to Docker Hub, so
|
||||||
|
the sweep exits 2 (incomplete) rather than producing the table. One `docker login` + re-run closes it.
|
||||||
|
3. **R-55's live reboot leg is not run** — see `felhom-controller/REPORT.md`. Shipped and wired, not
|
||||||
|
yet proven live.
|
||||||
|
4. **wanderer is re-pinned but never deployed.** It went from 1 service to 3 with a second public
|
||||||
|
hostname; it deserves a real deploy on the demo box before anyone is invited to it.
|
||||||
|
5. `reuse_refs_check.py` resolves REUSE.md citations against the repo root and so cannot follow a
|
||||||
|
cross-repo pointer; the catalog's `healthprobe.go` citation still flags. Path corrected; the
|
||||||
|
checker limitation is pre-existing.
|
||||||
|
6. **The `felhom-testing` skill's "known flake" line was actively harmful** — it instructed every
|
||||||
|
future session to dismiss a true positive, for weeks. Removed, and generalised: "known flake, just
|
||||||
|
re-run it" is a diagnosis and needs evidence like any other.
|
||||||
|
|||||||
Reference in New Issue
Block a user