feat(iso): R-21 slice A — bare-metal Felhom ISO pipeline + first-boot bootstrap

scripts/iso/: a DooPlex pipeline (build-felhom-iso.sh + Dockerfile.assistant) that
turns the official PVE ISO into a Felhom auto-install ISO whose first-boot stub
installs a retry-forever felhom-bootstrap unit which unattended-fetches
felhom-host-install.sh from the public felhom.eu/scripts channel and runs it until the
host is enrolled + a guest provisioned. host-install is UNMODIFIED (invoked only).

- build gates the answer on validate-answer OUTPUT text, never $? (spike S1 exit-0 trap)
- stub is from-iso, fully-up, exactly-once; retry unit owns all network work (S8a)
- retry-vs-resume encoded once: plain first, --resume when install state exists (v1.11.3)
- secret-bearing (embeds the retrieval passphrase): supervised/single-use; env shredded on success

Validated on VM 310: build gate + red-proof, disk-filter fail-safe, chain + retry,
resume-decision, exactly-once, no-net retry+recovery. Terminal host-install rc-0 success
operator-gated (drill customer needs the password-gated create-UI). scripts v1.16.0;
ROADMAP R-21 -> in-progress. Detail in REPORT.md.
This commit is contained in:
2026-07-16 13:47:08 +02:00
parent 914d51aa5f
commit 1fa3250aa3
13 changed files with 743 additions and 109 deletions
+107 -108
View File
@@ -2,126 +2,125 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## Capability-map audit — verify every status claim against evidence — 2026-07-16
## R-21 slice A: bare-metal Felhom ISO (build pipeline + first-boot bootstrap) — 2026-07-16
**Task:** docs-only audit of `documentation/architecture/00-capability-map.md`. Open every citation,
confirm/correct/demote each of the 55 rows against the actual evidence doc (scenario-specific PASS,
not topic adjacency), prove the MISSING rows absent by code sweep, close ROADMAP R-20, and swap the
draft caveat in the map header. **Zero code changes; read-only against three sibling repos.**
**Repo:** felhom.eu · **Baseline:** `main` @ `914d51a` → scripts **v1.16.0** (ISO pipeline tag;
`felhom-host-install` was *already* v1.16.0 from 2026-07-13 — see Versioning note).
**Validation:** nested VM **310** on felhom-pve; ISO builds on DooPlex. `felhom-host-install.sh`
**not modified** (read-only, invoked only). Hub code untouched.
### 1. Confirmed baselines (re-confirmed at task start)
### Files created (`scripts/iso/`)
| Repo | `main` @ commit | Role |
|------|-----------------|------|
| felhom.eu | `4b18cc5` (local == origin) | audit target — WRITE |
| felhom-controller | `e99c675` (v0.137.0) | read-only reference |
| felhom-agent | `c040c18` (v0.88.0) | read-only reference |
| app-catalog-felhom.eu | `21e8df1` | read-only reference |
`Dockerfile.assistant`, `build-felhom-iso.sh`, `answer.toml.tmpl`, `stub-first-boot.sh`,
`felhom-bootstrap.sh`, `felhom-bootstrap.service`, `profiles/nested-vm.profile`,
`profiles/nested-canary.profile`, `README.md`. Plus `scripts/CHANGELOG.md`,
`documentation/backlog/ROADMAP.md` (R-21 → in-progress), this `REPORT.md`, `CONTEXT.md`.
The map's evidence corpus is read at felhom.eu tip `4b18cc5`; the header caveat now records that.
### Green gates
### 2. Counts
`bash -n` clean on all three scripts; `shellcheck -x` **exit 0, no findings** (run in the assistant
container). Retry-unit behavior de-risked on systemd 257 before wiring (transient unit: `Type=oneshot`
+ `Restart=on-failure` **does** retry — NRestarts climbed).
**55 rows audited = 23 confirmed (18 confirmed + 5 confirmed-absent) + 26 corrected + 6 demoted.**
No row was upgraded (candidate upgrades are Observations). Status distribution after the audit:
**32 PROVEN-LIVE / 15 IMPLEMENTED / 3 PARTIAL / 5 MISSING** (was 38 PROVEN-LIVE → 6 demoted).
### §4.4 finding — does the distribution channel embed a secret? (decides slice C)
### 3. Per-row verdict table
**The channel: no. The unattended install: yes (the passphrase).** The hub install-command generator
(`hub/internal/web/templates/customer_unified.html`, Option 1) emits
`curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o … && sudo bash … --customer-id <id>
--mode <appliance|byo>` — a **public URL, no token/secret in the command**. But
`felhom-host-install.sh` **requires the customer retrieval passphrase** (`read_passphrase`
`die "empty passphrase"`; used for config fetch + `POST /host-enroll`). Unattended ⇒ it must be
supplied via `--passphrase-file`, so **the slice-A ISO is secret-bearing** (embeds the passphrase).
**Slice-C implication:** the *distributed* ISO must NOT bake the passphrase — the claim/pairing model
must deliver it (or the config) to the box *after* the customer claims it. Slice-A ISOs are
supervised/single-use; the bootstrap `shred`s `bootstrap.env` on success.
Each verdict names the doc opened and the leg/section carrying (or failing to carry) the scenario.
### 2.3 retry-vs-resume ruling (source-verified, encoded once)
| Row | Scenario | Verdict | Evidence opened → leg / reason |
|-----|----------|---------|--------------------------------|
| A1 | Appliance day-0 zero-click auto-confirm | **CONFIRMED** | `DRILL-day0-take2` §2/§3 (auto-confirm 6m39s, zero clicks; floor self-update 0.120→0.122; claim gate closed) + `DRILL-day0-vm` §5.6 |
| A2 | BYO install (PARTIAL) | **CONFIRMED** | `DRILL-GL6` Phase 2 byo happy-path PASS; coexistence guards (GL6-F3/F6) only fixed v1.12.0, never live-revalidated — PARTIAL right |
| A3 | Customer claim | **CORRECTED** | `DRILL-day0-vm` §10/F-4 carries it; dropped mis-cited `CAMPAIGN-4` F-C (that is the escrow-claim 502, not the password claim) |
| A4 | Escrow ceremony | **CORRECTED** | agent v0.88 REPORT (claim 200→410, R absent). Qualifier fixed demo→**drill VM**; browser-wizard leg flagged not-yet-validated |
| A5 | DR tier by default (IMPLEMENTED) | **CORRECTED** | cited `SPIKE-pbs-tier-provisioning` = slice-0, shipped nothing → `DRILL-day0-take2` §2 (WG both modes, PBS-DR descriptor auto-provisioned live). Upgrade candidate |
| A6 | Uninstall (PARTIAL) | **CORRECTED** | `DRILL-GL6` Phase 1/5 (KEPT-vs-WIPED printed, data intact ×3); secret-purge fixed v1.12.0, `mnt-*.mount` residue open — PARTIAL right |
| B1 | Deploy app from catalog | **CORRECTED** | `CAMPAIGN-2` T-DEPLOY-SET + `RERUN-p1p3` PASS; dropped T-RES-MEMGUARD (guard never *fired* — ample RAM / auth-walled) |
| B2 | App lifecycle | **CORRECTED** | T-LIFECYCLE PASS + remove live in `CAMPAIGN-3`; T-REMOVE-REDEPLOY never cleanly passed (stale dryrun) — noted open edge |
| B3 | Protected infra un-stoppable | **CORRECTED** | cited `CAMPAIGN-2` T-SEC-PROTECTED was a stale-dryrun FAIL → `CAMPAIGN-nomercy` + `RERUN-p1p3` T-SEC-PROTECTED PASS |
| B4 | Catalog sync + validation choke | **CONFIRMED** | `CAMPAIGN-2` T-SYNC-IDEMPOTENT PASS; choke-point is unit-red-proofed (v0.132) — status held |
| B5 | App crash → customer notified (IMPLEMENTED) | **CORRECTED** | citation `CAMPAIGN-3 fix-3 arc` (surfaced the gap) → controller v0.120.0 (impl+red-proof); "delivery proven" softened (6B deferred/6C inconclusive) |
| B6 | Post-deploy config + restart | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | T-PAGE-ALL is a GET-only page-render smoke test; config-save+restart exercised in no campaign (CAMPAIGN-3 skipped it) |
| B7 | Backup classification (13 apps) | **CONFIRMED** | `SPIKE-backup-classification` SQ2 (13 bind apps) + 6D/6E live legs |
| C1 | Nightly DB dumps, atomic writes | **CORRECTED** | cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes + T-RST-DB auth-hollow → `CAMPAIGN-6B` P4 + `6E` B1/B2 (SIGKILL→last-good byte-unchanged) + `6D` P-FAB |
| C2 | Tier-2 copy | **CONFIRMED** | `CAMPAIGN-6E` P-TIER2 deep-4 4/4 PASS + `6C` |
| C3 | Offsite restic | **CORRECTED** | enlargement gate live (`6D` P3-DELIVERY); noted raw-data-quota/retention are `SPIKE-restic-snapshot-shape` **dry-run** only |
| C4 | Offsite restore | **CONFIRMED** | `CAMPAIGN-6D` P-IMMICH/P-PLACE (functional from offsite alone, byte-identical) |
| C5 | Manual `.fab` | **CORRECTED** | `6D` P-FAB (1.7GB circle) + `6B` P2 (chunking at real CF edge via `curl --resolve`); rendered-browser upload leg still deferred; C6B-F1 was the 6B *finding* |
| C6 | Guest-loss PBS DR | **CORRECTED** | cited `VALIDATION-newbox-restore` = offbox **restic** (wrong tier) → `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore`→healthy) + T-PBS-VERIFY (13 snapshots) |
| C7 | PBS-DR self-heal (IMPLEMENTED) | **CORRECTED** | cited spike = findings-only → hub v0.56.0 reconciler + §10 red-proofs. Facts fixed: **scoped to one host** (not fleet-wide); **already fired live** on drill qm300 |
| C8 | Crash mid-backup/migration self-heal | **CORRECTED** | cited `CAMPAIGN-2` T-RBT-* legs empty/auth-hollow → `6D` P5-REST + `6E` B1-B3; mid-*migration* crash still weakest sub-claim |
| C9 | Soft-quota block + notify | **CONFIRMED** | `6D` P3-DELIVERY (block + op+customer email live); usage-bar UI only weakly corroborated |
| C10 | Customer performs a restore (MISSING) | **CONFIRMED-ABSENT** | sweep: only self-reference at map L34 "never executed by a non-Viktor human"; all restores operator-driven |
| D1 | Drive wizard incl. legacy LVM-root | **CORRECTED** | `DISPOSITION-ia` (legacy EFI+LVM, root not offered) + `storage-lifecycle-acceptance` (enroll live) + agent fence self-test; cited CAMPAIGN-2 legs were auth-hollow; UI-wizard leg supervised |
| D2 | Data migration (all/per-app) | **CORRECTED** | `6C` 4P-5 (per-app) + `storage-lifecycle-acceptance` (2× migrate-all via UI, byte-identical); cited CAMPAIGN-2 legs auth-hollow; crash-safe design-level only |
| D3 | NAS client verify-before-commit | **CONFIRMED** | `SPIKE-nas-verify` Q2/Q4 + `SPIKE-nas-dsm` (real DSM 7.2 end-to-end) + `CAMPAIGN-3` |
| D4 | USB enroll/unplug/recommission | **CORRECTED** | `storage-lifecycle-acceptance` E4 (yank→auto-rebind) + E10 (re-enroll) + `4/6A` (3-USB reshuffle); cited RUNBOOK could not complete enroll |
| D5 | Decommission / eject | **CORRECTED** | `storage-lifecycle-acceptance` E9 (decommission-anyway, reboot-safe) + E12; cited CAMPAIGN-2 legs auth-hollow, SPIKE was report-only |
| D6 | Boot ordering survives reboot | **CORRECTED** | **cited `CAMPAIGN-3` F10/F11/F12 are the CRITICAL/HIGH failures** → fixes shipped agent v0.85 + re-validated live `CAMPAIGN-4` (0 ordering cycles) + `6A` 1D |
| E1 | Remote CF Tunnel | **CONFIRMED** | `CAMPAIGN-2` T-FLT-CF PASS (not in the H1-hollow set; recovered 20s) |
| E2 | LAN access when net down (IMPLEMENTED) | **CONFIRMED** | empty citation consistent with IMPLEMENTED; never drilled as customer experience → R-19 |
| E3 | Phone photo backup | **CONFIRMED** | `6D` Accept #1 + SQ3 (immich functional from offsite, byte-identical) |
| E4 | Documents/OCR | **CORRECTED** | tightened vague "6C/6E legs" → `6C` 4P-1 (paperless consume+OCR PASS) |
| E5 | SMB **server** (MISSING) | **CONFIRMED-ABSENT** | sweep `smbd\|samba\|wsdd\|net usershare\|smb.conf`: 10 hits, all client/offsite/docs — no server. Hetzner `SambaEnabled` = a box we write to, not a share we expose |
| E6 | DLNA to TV (MISSING) | **CONFIRMED-ABSENT** | sweep `minidlna\|dlna\|ssdp\|upnp`: 3 hits, all ROADMAP/map docs; zero implementation |
| E7 | File access via browser | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | T-PAGE-ALL renders only controller dashboard pages, not FileBrowser; no browse/download proof anywhere |
| E8 | Forgot-password reset code | **CORRECTED** | tightened "F-15 arc" → `DRILL-day0-take2` F-15 (live re-run, hash applied 1s, code accepted first try) |
| E9 | Multi-user accounts (MISSING) | **CONFIRMED-ABSENT** | sweep of controller `auth.go`: single `effectivePasswordHash()`, no user table / role / permission model |
| E10 | WireGuard OOB (IMPLEMENTED) | **CONFIRMED** | `SPIKE-oob-wg-operator-peer` GO + `SPIKE-felhom-sshd`; unproven on true 100.64/10 — IMPLEMENTED right |
| E11 | Break-glass recovery (IMPLEMENTED) | **CONFIRMED** | `runbooks/break-glass.md`; live incident reproduced in SPIKE-felhom-sshd; no end-to-end recovery drill — IMPLEMENTED right |
| F1 | Health-degradation email | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | cited `6D` P3-DELIVERY is the **enlarge-block** trigger, not health-degradation; `NotifyHealthChange` never fired an email live in any doc |
| F2 | Event catalog | **CORRECTED** | vague "per-arc" → 3 live-delivered families: enlarge-block (`6D`), claim code (`DRILL-day0-vm` F-4), reset code (`take2` F-15); `app_start_failed`/dead-app unit-only |
| F3 | Empty-email wipe guard (IMPLEMENTED) | **CONFIRMED** | controller v0.137.0, red-proofed 07-15 |
| F4 | System/container metrics | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | cited T-RES-CPU/T-SOAK-LOOP are H1/H2 harness artifacts (auth-302); SQLite/Chart.js/30-day downsampling validated nowhere |
| F5 | Debug rings + log-bundle pulls | **CORRECTED** | rings live-exercised `CAMPAIGN-3` fix-6; noted log-bundle-pull TTL/custody half is changelog-only |
| F6 | Operator alerting (Healthchecks) | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | no corpus doc; healthchecks-config app not deployed; per enum "no citation → not PROVEN-LIVE". Re-upgrade candidate if operator cites a live alert |
| G1 | Customer/host management UI | **CORRECTED** | cited "daily operator use" no-doc + `AUDIT-hub-gui` predates features (v0.25) → hub v0.53 (Peti-cluster live) + `CAMPAIGN-4` + `take2` F-16; 8-tab render = render-test-validated (UI password-gated) |
| G2 | Publish train (PARTIAL) | **CONFIRMED** | `runbooks/publish-train-rules.md` (floor-LAST, MinAgent gate); never proven on a real remote customer — PARTIAL right |
| G3 | Agent self-update | **CORRECTED** | cited `SPIKE-agent-selfupdate` is dummy-only, no operator-signed leg → `DRILL-GL6` Phase 3 (armed-key round-trip with the REAL key) |
| G4 | Controller self-update | **CORRECTED** | cited "07-10 arc" CHANGELOG → `DRILL-day0-take2` §2 (credential-empty box self-updated 0.120→0.122 live) |
| G5 | Offsite provisioning | **CONFIRMED** | `VALIDATION-offsite-provisioning-e2e` PASS (real box 611714, end-to-end) + `SPIKE-hetzner-api` + re-issue in `CAMPAIGN-offsite-overnight` |
| G6 | Per-customer offsite fill/staleness/freeze | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | `OffsiteChecker` in code but no live-fired leg; `CAMPAIGN-offsite-overnight` = no fill/staleness emails, freeze write-block inconclusive |
| G7 | Box-level Storage Box aggregate (MISSING) | **CONFIRMED-ABSENT** | hub sweep `storagebox\|aggregate\|oversubscription\|sum.*quota`: only per-customer OffsiteChecker + app-telemetry; no box-level roll-up |
| G8 | Secrets hygiene (IMPLEMENTED) | **CONFIRMED** | hub v0.53.0 closing bundle (bearer → k8s Secret) |
| G9 | Operator login password from UI (IMPLEMENTED) | **CONFIRMED** | hub v0.54.0 (`handleChangePassword`, `effectivePasswordHash()`) |
**Plain first attempt; add `--resume` on any later attempt that finds
`/var/lib/felhom-install/state.json`.** Source landmark: `felhom-host-install.sh` header **v1.11.3**
("--resume correctness — the producer steps (token/enroll/grows) run every pass … a resumed install no
longer writes a config missing hub.host_id/proxmox.token") — so `--resume` is safe, whereas a plain
re-invoke over an existing install state would re-hit the populated-host leaf guard / existing-vmid
refusal. `--mode` is required in both forms. Encoded in `felhom-bootstrap.sh` with the landmark cited
inline; **empirically confirmed** (below).
### 4. R-20 verdict — **CLOSED**
### Per-scenario results (evidence under `180:~/felhom-iso/evidence/`)
Read from current `scripts/felhom-host-install.sh` source (not the changelog). Operator-key pinning
is **fully automatic in the day-0 flow, no manual step**: keys resolve at L11811219 (script
constants `OPERATOR_KEY_OPERATIONAL_*`/`_RECOVERY_*`, already populated, optional
`--operator-pubkey-file` override); pinned by `step_agent_config()` — "STEP 6/8 agent config +
service" (L2044; embedded python builds `authz.signers` L21462156, and a re-install with no keys
preserves an existing box's signers); verified at the step-8 check L23322337 ("authz signers: N
(operator-signed self-update armed)"). No interactive prompt or post-install hand-edit gates it →
ROADMAP R-20 set to **closed**. One loose end (doc-drift, not a reopen): the L193197 comment still
says the constants are "EMPTY by default" though L198201 are populated — folded into R-16 hygiene.
- **C — build gate blocks a broken answer (+ red-proof) — PASS.** Answer with both `root-password`
and `root-password-hashed`: the output-parse gate surfaced `` `global.root-password` and
`global.root-password-hashed` cannot be set at the same time / Error: Found issues `` and aborted
(`BUILD-EXIT=1`, **no ISO** — count unchanged), *even though `validate-answer` exits 0*.
**Red-proof:** gutting the gate to trust `$?` made it wrongly report `[OK] answer validated`; the
broken answer was then stopped only by `prepare-iso`'s own error **plus the build's `[[ -f out.iso ]]`
guard** — notably `prepare-iso` *also* exited 0 on the error, so the file-existence check was the
real backstop. Reverted (mutation done on the DooPlex copy; committed tree never touched).
- **D — disk-filter fail-safe through the pipeline — PASS.** Pipeline-built canary ISO
(`filter.ID_SERIAL_SHORT="no-such-disk-zzz"`) → `ERROR: Installation failed: filter did not match any
device`, abort exit 1, no interactive fallback. 8 GB canary disk **byte-identical** after
(`sha256 -c: OK`, host-side; `scenarioD-filter-abort.png`).
- **A — full unattended chain — MECHANISM PASS; terminal success operator-gated.** Zero keypress;
stub ran once (uid 0) and enabled+started `felhom-bootstrap`; the unit **fetched host-install from the
REAL channel** `https://felhom.eu/scripts/felhom-host-install.sh` and invoked it
`--customer-id … --mode appliance --hub-url …`; host-install ran unattended through preflight
(hub reachable) — with the drill customer absent it correctly failed at config preflight (`404`) and
the unit **retried** (`bootstrap-journal.log`; NRestarts ≥3; 12 host-install invocations).
**Not live-proven:** host-install **rc-0** → hub-enrolled + guest-provisioned → done-flag +
unit-disable + no-re-run. Reason: that needs a properly created drill customer, and CC cannot use the
password-gated create-UI while a direct write to the *live, locked* hub `hub.db` (verified open by the
running pod) risks corruption. The done-flag/disable/env-scrub success path is code-reviewed +
shellcheck-clean; empirical proof requires an operator-created drill customer (supervised).
- **B — no network at first boot — PASS.** During a host-side `link_down` window
(13:37:56→13:39:37) the chain failed repeatedly — `host-install FAILED rc=6` (host-install's own hub
fetch, no net) and `ERROR: host-install fetch failed (no network yet?)` — and **the unit kept
retrying (never died)**; on link-up it **resumed automatically** (`13:40:04` fetch+run). No
"single failure → dead unit". The stub-runs-without-network property is spike-proven (S8a);
`bootstrap-journal.log`.
- **2.3 resume decision — PASS.** After creating `/var/lib/felhom-install/state.json`, the next attempt
logged `prior install state present (…) -> adding --resume`; earlier attempts ran plain.
- **Exactly-once stub — PASS.** Host-side reboot → `/var/log/felhom-first-boot.log` hash **unchanged**
(`13cf3fe4…`), pending-flag consumed; the retry unit correctly persisted (still `activating`, not done).
### 5. Commits pushed to `main`
### ISO artifacts (build ≈ 12 s each, source already local)
- (recorded on push) — `docs(map): capability-map evidence audit` (00-capability-map.md header + 26 corrected / 6 demoted rows; ROADMAP R-20 closed; this REPORT).
| ISO | sha256 | size |
|---|---|---|
| `felhom-pve-9.2-1-v1.16.0-nested-vm.iso` | `8cbe5eaeaa0d…82bb` | 1,707,737,088 B |
| `felhom-pve-9.2-1-v1.16.0-nested-canary.iso` | `6b4568eaeac0…0301` | 1,707,737,088 B |
| (validation build, key baked) `…-nested-vm-val.iso` | `0042acfdb70a…98e7` | — |
### 6. Observations (out of scope — NOT acted on)
Source ISO `proxmox-ve_9.2-1.iso` sha256 `4e88fe4…f2c6c` (verified vs official SHA256SUMS). Assistant
`proxmox-installer-common 9.2.7` (trixie no-sub). Artifacts kept under `180:~/felhom-iso/out/`, never
committed (secret-bearing).
- **Candidate upgrades (audit cannot upgrade):** **A5** DR-tier-by-default is shown working *live and
hands-free* in `DRILL-day0-take2` §2 — arguably PROVEN-LIVE(drill), left IMPLEMENTED per the
no-upgrade rule. **C7** self-heal has code + red-proofs + a hands-free live convergence on qm300 —
stronger than a bare IMPLEMENTED, but scoped-to-one-host with no real-customer firing.
- **F6 demotion is a judgment call:** operator Healthchecks→Resend alerting may genuinely run in
production; it simply has no corpus doc, and the strict enum forbids PROVEN-LIVE without a citation.
Re-upgrade the moment a live alert is cited.
- **Recurring "cite-the-finding" inversion:** several rows cited the doc where a defect was *discovered*
as if it were the proof — D6 (F10/F11/F12 failures), C1 (F7 non-atomic finding), C6
(VALIDATION-newbox = wrong tier), C5/B5 (6B/CAMPAIGN-3 findings). The fixes live in later legs. Worth
a standing authoring rule: cite the fix-validation leg, never the finding ID.
- **`storage-lifecycle-acceptance-2026-06-15` is an uncited, high-value corpus doc** — it live-proves
D2/D4/D5 (and local-drive boot survival). Now cited; consider it first for storage rows.
- **CAMPAIGN-2 is fragile as a citation source for restore/reboot/storage rows:** harness bug H1
(auth-session expiry → CSRF-403) hollowed every `T-RST-*`, `T-STG-*`, `T-RBT-*`; several cited
evidence dirs are literally empty or a 302. Re-anchor such rows on 6D P5-REST / 6E Phase B /
storage-lifecycle-acceptance, which actually exercised these live.
- **MEMORY.md drift:** the note that `CAMPAIGN-6D` "DEFERRED P3-BROWSER/P3-DELIVERY" is stale — the
committed 6D marks both **PASS**.
### VM 310 final state
Stopped, kept; SeaBIOS, scsi0 32 GB + scsi1 8 GB (canary intact), ISO detached. Boundaries honored:
VM 300 (drill) running untouched, 9000 stopped, LXC 9201 (demo) running untouched, hub untouched.
Secret-bearing pipeline ISOs removed from felhom-pve (their embedded passphrase was a placeholder, not a
real credential).
### NOT-live-validated
- **Physical N100 run (RUNBOOK B):** real DMI/firmware, USB-boot media, vendor Secure Boot key set,
real-NIC naming, prior-LVM `blkdiscard` on real hardware. Not started (out of scope by design).
- **host-install rc-0 terminal success** (hub-enrolled + guest-provisioned + done-flag + disable) and
the **2.3 `--resume` executed against a real partial install** — both need an operator-created drill
customer (password-gated create-UI; live-DB insert unsafe). The *decisions/plumbing* are proven; only
the terminal success is deferred.
### Versioning note
`felhom-host-install` reached v1.16.0 on 2026-07-13 (before this task's stated 1.15.x baseline). The
new ISO pipeline is tagged **v1.16.0** to stay coherent with that release (the ISO's baked invocation
targets host-install's flag surface). The `scripts/CHANGELOG.md` entry is a new component under the
same release tag, not a bump of host-install.
### Observations (recorded, not acted on)
- `prepare-iso` (not just `validate-answer`) was observed to **exit 0 on a bad answer** — the build's
`[[ -f out.iso ]]` output-existence guard is therefore load-bearing, not just belt-and-suspenders.
- host-install prints `unexpected hub status 404404` (doubled) for an absent customer — cosmetic
host-install quirk; left untouched (read-only this task).
- host-install exits **rc=6** when it cannot reach the hub (no net) — the bootstrap correctly treats
any non-zero rc as retry.