From af8a7a58a4ba372a0f8a05f0ba3723ab9b3edee8 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 21 Jul 2026 15:32:56 +0200 Subject: [PATCH] docs+scripts: TASK-E parts 4-7 - operator one-pager, rulings, hub build.sh adoption PROMPT-TEMPLATE: standard 'For the operator' plain-language section, mandatory for M+ tasks and anything with a STOP. ROADMAP rulings (operator, 2026-07-21): R-25b full-teardown cascade with three acks + typed name (M-sized, spec to follow, no longer blocks R-3); R-11 channel = direct Messenger, doc is the architect's; R-42 option (a), sidecars follow the app; R-17 delete the archive - spike-lite found NO tooling verb targets it, so it is an operator console action; R-4 complete (freemail.hu verified). R-55 + R-41 slice 1 marked shipped; new R-56 (app difficulty classification - the constructive half of the glance ruling). scripts/build-hub.sh v1.23.0: the hub build script was outside any repo. Adopted verbatim + versioned; the build-dir path is now a symlink to it. felhom-testing skill: the ~1/5 recovery-code 'known flake' is retired - it was a real defect the test was correctly detecting. --- documentation/PROMPT-TEMPLATE.md | 32 +++ .../architecture/00-capability-map.md | 4 +- documentation/backlog/ROADMAP.md | 15 +- scripts/CHANGELOG.md | 16 ++ scripts/build-hub.sh | 218 ++++++++++++++++++ skills/felhom-testing/SKILL.md | 9 +- 6 files changed, 283 insertions(+), 11 deletions(-) create mode 100755 scripts/build-hub.sh diff --git a/documentation/PROMPT-TEMPLATE.md b/documentation/PROMPT-TEMPLATE.md index 8508e8b..7e1945e 100644 --- a/documentation/PROMPT-TEMPLATE.md +++ b/documentation/PROMPT-TEMPLATE.md @@ -24,6 +24,38 @@ --- +## For the operator — what this fixes, in one page (plain language) + + + +--- + ## 0. Task class & scope diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index aec0b97..664495b 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -34,7 +34,7 @@ | Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; **universal secret-free / operator-bind** mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE** (physical N100, one pass, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — the full chain on real metal in a single pass:** the generic reusable pairing ISO (v1.20.0, `--loader mkimage`, SB off) booted the cheap AMI board that F1 had blocked, installed unattended, and the box **self-registered as an unclaimed appliance at 16:17:14 — the same second it first booted** (`appliance_registrations` id=3), then bound → credential-delivered → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware.** Prior nested legs: slice A `SPIKE-baremetal-iso-2026-07-16` (build gate, disk-filter fail-safe, stub→host-install fetch); slice B RUNBOOK-B (shim boots+installs OVMF SB-enforcing + SeaBIOS; `--loader mkimage` boots+installs SB-off; mkimage SB-enforcing **FAILS** `Access Denied`; surgery byte-identical); **slice C (2026-07-17): the GENERIC secret-free ISO** — box self-registers as an unclaimed appliance (`POST /api/v1/appliance/register`, one-shot poll delivery, 404-no-oracle — all live-verified through the public ingress), operator binds on the Hosts page, hub delivers credentials once; bootstrap harness proves direct(zero-appliance-calls)/pairing/delivery; artifact proven secret-free (baked env = hub URL only) | **F1 loader caveat:** `--loader mkimage` fixes cheap AMI firmware that can't USB-boot the stock GRUB — UNSIGNED → **Secure Boot must be OFF**; default `shim` keeps SB. **Slice C bind is operator-password-gated** (CC stages, Viktor binds) → the live boot→register→bind→day-0 composition + physical N100 boot fold into the supervised rehearsal (R-1). Customer-facing **self-bind page = R-27 slice 1 SHIPPED (hub v0.66.0, 2026-07-17)** — see the dedicated self-bind row | | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. **Deliverability (R-4), gmail half DONE 2026-07-18:** the rehearsal's claim email was the first sent under the tightened DMARC `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu remains Viktor's open half.** (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) | | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends | -| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody` | +| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test | | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). | | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **Consistency gap → R-25b:** the Danger-zone DELETE leaves host rows and doesn't run this teardown | | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 | @@ -62,7 +62,7 @@ | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D | | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill | | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending | -| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **A real gap this validation surfaced → R-55:** for a *drive-backed* app a customer's deliberate Stop does NOT survive a reboot — the boot bind gate recreates and starts every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct), but it means "a stopped app stays stopped" is currently true only for non-drive-backed apps. The **static-guest** half of the network leg stays deliberately out of scope → **R-50** | +| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too** — with one honest caveat on the evidence: the fix is deployed and verified present in the running binary, but the reboot leg proving it end to end has NOT been run yet (it needs an operator-present `pct reboot 9201`), so this half is **shipped-not-yet-proven-live**. The **static-guest** half of the network leg stays deliberately out of scope → **R-50** | | Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) | | Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent, controller, bootstrap | **PARTIAL** | `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — a real relocation of the demo box: guest + hub telemetry + WG/PBS + Cloudflare tunnel all survived untouched, but the **controller↔agent control plane did not** (agent binds a LAN literal → `bind: cannot assign requested address` → storage/PBS-backup/quiesce/restore-test/DR down until fixed). Mitigated for the window by pinning `vmbr0` static | **R-50** (island-bridge control plane, spike-first) is the durable fix. Related: **R-51** (dead-primary alerting) and **R-52** (boot desired-state reconciliation) — the same event left two apps `Exited` with no alarm and no recovery | | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 853bbba..0ac85a5 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -23,7 +23,7 @@ | R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path | | R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) | | R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b | -| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu / citromail.hu / t-online.hu; tighten DMARC `p=none` → `p=quarantine` (pending since email.md 02-04) | S | **DNS half DONE + gmail half DONE (2026-07-18); freemail.hu open** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **Remaining: the freemail.hu test-send** (Viktor, remote-doable, `dmarc=pass` check) | +| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none` → `p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** | ## P2 — during alpha @@ -47,10 +47,10 @@ | R-8 | DLNA (**gate input now exists — R-6 spiked 2026-07-18: SSDP reaches LAN clients from host-net**): validate Jellyfin's built-in DLNA server first; only add minidlna to the catalog if Jellyfin-DLNA fails | S | idea (unblocked) | Don't add catalog weight before proving the cheap path. **R-6 confirmed the cheap path is physically viable — Jellyfin DLNA must run host-network (same multicast constraint as R-7)** | | R-9 | Uninstaller trio (from 07-15 Peti session): cluster-aware `felhom_guests` guard (node-local `pct list` deletes cluster-wide pveum objects); saferemove detection + time estimate + opt-in `--quick-remove` (never mutate `storage.cfg`); smarter `restore_storage` default for BYO clusters (shared storage, not local-lvm) | M | idea | Second item's rejected alternative (temp-disable-and-restore) stays rejected — crash window silently downgrades cluster wipe policy | | R-10 | T-6E-1: DB-dump dir-fsync asymmetry (LOW, confirmed in 6E) | XS | idea | One-line hardening; batch with the next controller task | -| R-11 | Tester-facing one-pager: what the box does, known limitations, how to report (channel decision: Messenger group?) | S | idea | Pairs with R-3 | +| R-11 | Tester-facing one-pager: what the box does, known limitations, how to report | S | **RULED 2026-07-21 (channel); doc is the architect's** | Pairs with R-3. ****operator ruling 2026-07-21**: the reporting channel is DIRECT MESSENGER to the operator** — no group, no tracker, no form. Rationale: the expected tester count at first invite is ~0–few, and a channel nobody is on is worse than a channel everyone already uses. **The one-pager and the tester agreement are the ARCHITECT's deliverable, not CC's** — CC does not write them; it may be asked to fact-check claims in them against the code. Revisit the channel only if tester count makes direct messages unmanageable | | R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. **Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17):** the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above | | R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. | -| R-17 | Old-box archive (u629193-sub1) retirement decision — 9/9 byte-identical restores verified | XS | awaiting-decision | Viktor ruling | +| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **RULED 2026-07-21 — DELETE; execution is an operator console action** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off | | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE | | R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) | | R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f | @@ -63,7 +63,8 @@ | R-51 | **Dead-primary alerting — a multi-container app whose MAIN container is dead must alert.** Aggregation currently classifies such a stack `unhealthy`, and `IsDownState` deliberately excludes `unhealthy`, so nothing fires. | S | **SHIPPED 2026-07-21 — controller v0.156.0** | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F4). Observed live: `immich-server` was `Exited` for **18 h** with the app 100 % unreachable, and the box produced **no** dead-app banner and **no** `app_start_failed` hub event — while single-container Calibre-Web, down for the same reason, alerted correctly within 90 s. **Constraint (load-bearing): do NOT simply fold `unhealthy` into down.** That exclusion is deliberate (`stacks/manager.go` fix-3, `downstate_test.go`) and reverting it reintroduces the flapping it was added to stop. Direction: distinguish *member-container-exited* from *healthcheck-failing* in the aggregation, and treat a dead primary as down | **SHIPPED 2026-07-21 (controller v0.156.0).** **The diagnosis in this row was WRONG at the source and is corrected here:** aggregation did NOT classify the stack `unhealthy`. `aggregateState`'s final branch returned `StateRunning` for any mix of running and stopped members — the comment said "report as running (partial)" — so the stack read as RUNNING and `IsDownState` had nothing to fire on. The `unhealthy` exclusion was never involved, and the constraint it protects was therefore never in tension with the fix. New `StateDegraded`: a DOWN member whose docker restart policy is `always`/`unless-stopped` means docker was supposed to be keeping it up, so the stack is degraded (a down state, alerting through the EXISTING banner + `app_start_failed` path, unchanged); `no`/`on-failure` is a finished one-shot init/migrate container and stays benign. An UNREADABLE policy counts as supervised — fail-CLOSED, deliberately the opposite of `IsDownState`'s fail-open, because there the *state* is ambiguous while here a member is known dead and only the excuse is missing (P2 census 2026-07-21: all 53 catalog templates / 78 services are `unless-stopped`, zero one-shot containers exist today). `IsDownState` gained `degraded` and NOTHING else — the `unhealthy`/`restarting`/`paused`/`unknown` exclusions are byte-identical and `downstate_test.go` is untouched and green. Red-proof: the mix branch reverted to `return StateRunning` makes the immich fixture and both production-path tests fail with `"running"`. Live leg (STOP-1) pending. Evidence: `felhom-controller/REPORT.md` (2026-07-21) **PROVEN LIVE 2026-07-21 (STOP-1, operator-present).** On guest 9201, `docker stop immich-server` at **12:50:40 CEST** (policy `unless-stopped`, three helpers left running — the exact F4 shape). **12:50:53 — 13 seconds later — the stack read `degraded`** where it read `running` for 18 hours on 2026-07-20. `10:51:11Z` **exactly ONE** `app_start_failed (warn) — Telepített alkalmazás nem fut: Immich` (single-fire verified by count, not by eye). Dashboard rendered the banner *Telepített alkalmazás nem fut: Immich (degraded)* plus the new „Részlegesen leállt" state label. `docker start` at 12:51:37 → `running` by 12:51:52 and the banner **self-cleared** (state-based, as designed). Note the banner appends the raw state in English — `(degraded)` — which is pre-existing behaviour, not introduced here, but now more visible | | R-52 | **Boot desired-state reconciliation — a `deployed: true` app should be running after boot.** The controller *reports* deployed-but-stopped apps (30 s `deadapp-check`) but never starts them, so an app that misses its boot start stays down until a human notices. | M | **SHIPPED 2026-07-21 — controller v0.156.0** | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F5). Observed live: the pre-transport shutdown left `immich-server` and `calibre-web` `Exited`; **10 sibling containers came back and those two did not**, and they were still down ~18 h later. **Includes root-causing why `restart: unless-stopped` did not resurrect them** — both were stopped ~25 s before power-off, so Docker most likely recorded them as user-stopped; that hypothesis is untested because the guest journal is volatile and the controller's own logs were rotated by the container recreate. Direction: a bounded start-once reconciliation (N attempts, reusing the existing boot grace), never a restart loop. Pairs with R-51 — that one is the *alarm*, this one is the *recovery* | **SHIPPED 2026-07-21 (controller v0.156.0), `internal/bootrecon`.** One bounded start-once sweep at controller startup: every deployed, non-protected, not-mid-deploy stack that still HAS containers and is down gets `StartStack`, at most **2 attempts 30 s apart**, then it stops and the alarm owns the problem. Never a restart loop. The whole sweep (5 s settle + one 30 s gap) fits inside the existing 90 s `deadAppBootGrace`, so a successful recovery never alerts and a failed one alerts honestly — asserted by a test rather than left to a comment. **The safety argument is the container gate:** the UI's Stop is `compose down`, which REMOVES the containers, while an interrupted boot leaves them behind as `Exited` — so "deployed, has containers, and they are down" is exactly the boot-orphan signature, and a zero-container stack is never touched. Red-proof: dropping that gate makes the user-stopped app get started, which is the one thing this must never do. **P1 (the `unless-stopped` root-cause probe) is deliberately NOT what this shipped on** — the reconciliation is correct whether or not Docker recorded those two containers as user-stopped, and the probe is recorded as an open question rather than a blocker. Live leg (STOP-1) pending. Evidence: `felhom-controller/REPORT.md` (2026-07-21) **PROVEN LIVE 2026-07-21 (STOP-1, operator-present), and it answered P1 for free.** Fixture on 9201: `docker stop` on bookstack (2 containers, left in place) + calibre-web, and a UI Stop on immich (`compose down` → 0 containers); then `pct reboot 9201`. Result: `10:53:22Z [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [bookstack] — up to 2 attempt(s)` → `10:53:28Z attempt 1/2: started "bookstack" (took 6.0s)` → `complete: 1 app(s) recovered in 1 attempt(s)`, and **ZERO `app_start_failed`** — a successful recovery inside the boot grace is silent, exactly as designed. **P1 IS NOW ANSWERED, and the F5 hypothesis is CONFIRMED:** bookstack carries `restart=unless-stopped`, the Docker daemon came up at ~10:53:15Z, and the container's `StartedAt` is **`10:53:28.05Z` — the exact moment `bootrecon`'s `StartStack` returned**. Docker's own restart policy did NOT resurrect it; a container stopped before shutdown is recorded user-stopped and stays down. Only R-52 brought it back. **BUT the end-to-end "a deliberate Stop survives a reboot" property is NOT true today, for a reason outside R-52 — see R-55.** R-52's own gate is correct and was observed to be: immich (0 containers) was never a candidate | | R-54 | **[P2-HIGH] The guest's DHCP client is unsupervised — its death takes the box off the internet 1-2 hours later, invisibly.** ifupdown starts `dhclient` once at guest boot and nothing restarts it. | S-M | **SHIPPED 2026-07-21 — agent v0.92.1** | Origin: `audits/INCIDENT-guest-dhclient-killed-2026-07-20.md` §5 "OPEN RISK" — this row closes it. On 2026-07-20 a cleanup step killed guest 9201's dhclient (visible in the HOST's pid namespace; §4's `/proc//cgroup` rule exists because of it). **The guest then kept working for another ~80 minutes on its unexpired lease**; only at expiry did the address and default route vanish, taking the Cloudflare tunnel, hub reports, catalog sync and the controller→agent channel with them — 1h15m outage, and every observable signal said healthy for the first 80 minutes. **The design consequence: liveness of the DHCP client is itself a probe.** `internal/guestnet` flags a DHCP guest unhealthy on `pgrep -x dhclient` alone, while the lease is still live — waiting for the IP to disappear is waiting out precisely that silent window (red-proof: reverting to IP-presence-only makes the July-20 fixture report **healthy** with zero heals). Four fixed-shape `pct exec` probes (address / default route / `/etc/network/interfaces` mode / client liveness, parsers pinned to output captured live from 9201), the incident's restored invocation as the heal, verbatim, and dampers throughout: two CONSECUTIVE bad probes, ≥10 min between heals per guest, ≤3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on a static guest (dhclient must never fight a static config — reported loudly and left to **R-50**, which is where option 2 of the incident's three choices belongs), on an unknown mode, on an unprobeable guest, or on an ownership-unproven guest list (source is `ListLXC` ∩ the felhom pool, audit A1). Host-tier by necessity: a guest with no default route cannot repair its own default route. **A live finding during deployment:** the first sweep on felhom-pve logged `dhclient liveness probe failed: sudo: a password is required` and reported `state=unknown` — fail-safe, but blind. TASK-D assumed no sudoers change was needed; three of the four probes had no grant. `FELHOM_GUESTNET` + four `guestnet-*` capability rows shipped in v0.92.1 (v0.92.0 superseded, do not vouch). **Healthy cycle PROVEN LIVE 2026-07-21** on felhom-pve: caps `68/68 ok, degraded=0` and `level=DEBUG guestnet: guest network healthy vmid=9201 mode=dhcp has_route=true dhclient_alive=true`. The HEAL leg (STOP-2, a deliberate replay of the incident) is operator-present and pending. Note the guest and the host still differ (host static since the F1 mitigation, guest DHCP) — choosing one for both remains **R-50**'s call, not this row's. Evidence: `felhom-agent/REPORT.md` (2026-07-21) **PROVEN LIVE 2026-07-21 (STOP-2, operator-present) — a deliberate replay of the incident.** `kill -9` on guest 9201's dhclient at **12:43:18 CEST** (pid 336708, `cgroup=0::/lxc/9201/ns/.lxc` verified BEFORE the kill — the incident's own §4 rule, applied). **12:44:15 — 57 seconds later — detected on process liveness alone**, with the address and default route still perfectly present and 83 minutes of lease remaining: `unhealthy (first bad probe — not acting yet) bad_probes=1 required=2`. **12:45:15** second consecutive bad probe → `healing`, `cmd="pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0"` (the incident's invocation, logged verbatim before it ran) → **12:45:18 `guest network healed` ip=192.168.0.104 has_route=true dhclient_alive=true heals_last_hour=1**. Healed **120 s** after the kill — ~80 minutes before the outage would have begun. **The strongest evidence is what did not happen:** `cloudflared` stayed `Up 29 hours`, DNS resolved, the hub answered 302 in 0.16 s and the public URL 302 in 0.25 s. The 2026-07-20 outage was prevented, not merely detected. Exactly ONE heal, zero re-heals. **The damper then proved itself against a REAL transient, unplanned:** the STOP-1 guest reboot at 12:53:16 produced `unhealthy … no IPv4 address on eth0 bad_probes=1` — **no action** — then `guest network recovered` on the next cycle. A single blip really is not a diagnosis, and a booting guest is not injected with a dhclient. **Negative leg run in full:** 12:45:18 → 13:16:15 = **30 healthy Debug cycles, 0 heals, 0 WARN, 0 ERROR**. **Confirmed HUB-SIDE (read-only query of the hub store, after STOP-3):** the `guest_net` stanza arrives complete, and the report history traces the whole session — `10:30:27Z` agent 0.92.0 `state=unknown` (the sudoers-blind window, reported honestly rather than as a false healthy — independent proof that superseding 0.92.0 was right), then `10:48:12Z` 0.92.1 `healthy` carrying `last_heal_at=10:45:12Z heals_last_hour=1`. **Known limitation:** the damping state is IN-MEMORY, so an agent restart resets the heal counters — the ≥10 min / ≤3 per hour ceilings hold within one agent lifetime, not across restarts (a restart does re-arm the 3-minute settle window, so the practical exposure is small). Do not quote the ceiling as a hard guarantee | -| R-55 | **[P2] A customer's deliberate Stop does NOT survive a guest reboot for any drive-backed app** — the boot bind gate recreates and starts it unconditionally. | S | **idea (found live 2026-07-21)** | Surfaced by STOP-1's R-52 leg, which was designed to prove the opposite property and instead proved it is not true end to end. `internal/web/intermediary.go`'s boot gate (`[gate] boot : live bind confirmed — recreating drive-backed app `) selects on `cfg.Deployed && HDD_PATH` alone and then runs `StopStack` + **`StartStack`** on every match; the stack's `state` is passed in for the log line and is never consulted. So after `pct reboot 9201`, calibre-web and immich came back **running** at `10:53:18Z` even though immich had been stopped from the UI seconds earlier (`compose down`, 0 containers) — while bookstack, which is NOT drive-backed, correctly fell through to R-52. **R-52 is not implicated:** its own gate behaved exactly as specified (immich, with zero containers, was never a candidate), and this behaviour predates it. Two consequences worth ruling on: (a) a customer who stops a drive-backed app finds it running again after any reboot, which is the same class of dishonesty as offering a control that cannot work; (b) **R-52's practical scope on a real box is narrower than the brief assumed** — the gate reaches every drive-backed app first, so the boot-orphan sweep only ever sees the rest. Likely shape: the gate should recreate only apps that were RUNNING (or `deploying`) at shutdown, which is the same running/stopped distinction R-52 already draws, rather than every deployed app. Needs a ruling, not a drive-by change — the gate exists to fix a real bind-ordering bug and must keep doing that | +| R-55 | **[P2] A customer's deliberate Stop does NOT survive a guest reboot for any drive-backed app** — the boot bind gate recreated and started it unconditionally. | S | **SHIPPED 2026-07-21 — controller v0.157.0** | Surfaced by STOP-1's R-52 leg, which was designed to prove the opposite property and instead proved it is not true end to end. `internal/web/intermediary.go`'s boot gate (`[gate] boot : live bind confirmed — recreating drive-backed app `) selects on `cfg.Deployed && HDD_PATH` alone and then runs `StopStack` + **`StartStack`** on every match; the stack's `state` is passed in for the log line and is never consulted. So after `pct reboot 9201`, calibre-web and immich came back **running** at `10:53:18Z` even though immich had been stopped from the UI seconds earlier (`compose down`, 0 containers) — while bookstack, which is NOT drive-backed, correctly fell through to R-52. **R-52 is not implicated:** its own gate behaved exactly as specified (immich, with zero containers, was never a candidate), and this behaviour predates it. Two consequences worth ruling on: (a) a customer who stops a drive-backed app finds it running again after any reboot, which is the same class of dishonesty as offering a control that cannot work; (b) **R-52's practical scope on a real box is narrower than the brief assumed** — the gate reaches every drive-backed app first, so the boot-orphan sweep only ever sees the rest. Likely shape: the gate should recreate only apps that were RUNNING (or `deploying`) at shutdown, which is the same running/stopped distinction R-52 already draws, rather than every deployed app. Needs a ruling, not a drive-by change — the gate exists to fix a real bind-ordering bug and must keep doing that | **SHIPPED 2026-07-21 — controller v0.157.0, deployed to 9201.** The ruling that shaped it: an unconditional recreate could drown the node when apps were deliberately stopped to free resources for others — so honouring Stop is not only about honesty, it is about not fighting the customer's own capacity management. `shouldRecreateOnBoot` now additionally requires `len(Stack.Containers) > 0` — **R-52's `existing-Exited vs absent` predicate (`bootrecon.isBootOrphan`), translated to this gate**: a UI Stop is `compose down` and REMOVES the containers, while a guest that went down under a running app leaves them behind, and docker's records survive the reboot. **Spike-lite outcome: a truthful running-at-shutdown source already existed and nothing was invented** — `AppConfig` carries no run-state (`Deployed` is a deploy-lifecycle flag that stays true across a Stop), but container presence is durable, already collected via `docker ps -a`, and already trusted by R-52. **Container STATE is still deliberately NOT a filter** — that half of the original design is load-bearing (a `State != stopped` filter misses an app docker has not restarted yet, or one stuck `Exited` on a create-time bind failure with `RestartCount=0`, which is the bug the boot-id path exists to fix). `TestShouldRecreateOnBoot` now pins both axes at once; they pull in opposite directions, which is the difficulty of this gate. **Ordering trap handled:** the evidence is sampled into the snapshot BEFORE any recreate, because `recreate`'s own `StopStack` destroys it. **No regression to the drive-absent gate:** apps it stopped are also at zero containers and are now skipped here, correctly — they are restored by `ReconcileDriveGates`' `Return` branch from `StoragePath.StoppedStacks` on the same loop tick. Honoured Stops are counted and logged separately (INFO `left stopped …`) so an intended outcome never fires a WARN. **Red-proof:** dropping `hasContainers` makes `TestRecreateDriveBackedApps_HonoursCustomerStop` fail with `recreated=[romm immich]` — the live defect by name. **NO new privileged verb** — `StopStack`/`StartStack` are plain `docker compose` execs in the controller's own container; the boot path's only agent calls are read-only. **Live status: deployed and WIRED but the reboot leg is NOT yet run** — 0.157.0 is up and healthy on 9201 and the new log string is present in the running binary (checked, so this is not an inert seam), but proving it end to end needs a `pct reboot 9201` with a drive-backed app stopped first, and CC did not reboot a box at a remote site unprompted. That leg is one operator-present STOP away | +| R-56 | **[P3] Apps do not say how technical they are, so a beginner can be ambushed by a config-heavy one.** The catalog presents every app as equally approachable — one Telepítés button, the same Hungarian copy — but they are not. Glance needs a hand-written `glance.yml` before it does anything; some apps need a reverse-proxy or API concept to configure; others genuinely are install-and-use. A tester who picks the wrong first app concludes the PRODUCT is broken, not that they picked an advanced app. | S | **idea (filed 2026-07-21)** | Origin: TASK-E Part 3 — **filed, deliberately not implemented**. Shape: a `difficulty:` field in `.felhom.yml` (`kezdő` / `haladó` / `technikás`) surfaced as a catalog-card badge and repeated on the deploy screen. Cheap and incremental: one optional metadata field plus a badge, classifiable app-by-app with no migration — an app with no `difficulty:` simply shows no badge. **This is the constructive half of the glance ruling**: glance STAYS in the catalog (operator ruling 2026-07-21 — it is a legitimate app, not a broken one; its missing seeded `glance.yml` is a known pre-existing finding), and the honest fix is to LABEL it rather than hide it. Pairs with R-41: that gate proves an app CAN still deploy; this field tells a customer whether THEY should be the one deploying it | | R-53 | **`app_export.html` substituted the CSRF token where the customer domain belongs** - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | **SHIPPED (controller v0.150.0, 2026-07-20)** | One template token (`{{$.CSRFToken}}` -> `{{$.Domain}}`) plus the `Domain` key in `exportPageHandler`'s data map - that handler does not go through `baseData`, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined `.` and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F7) | ## P3 — post-alpha @@ -73,7 +74,7 @@ | R-26 | **Guided old-history recovery via a retained superseded escrow + the recovery code.** Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (`host_escrow_superseded`, `ListSupersededEscrow`). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside `.orphaned-` repo for restore. | M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin `DIAGNOSE-offbox-repo-orphaned-2026-07-17` | | R-27 | **Customer-facing self-bind page (R-21 slice C follow-on).** Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | **SHIPPED (slice 1, hub v0.66.0, 2026-07-17)** | **Slice 1 = the FIRST-box flow, DONE:** operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor `/bind/` (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same `BindAppliance`, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP `/bind/` exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). **Slice 1 PROVEN-LIVE 2026-07-18:** in the rehearsal the operator minted + emailed the link at 16:28:55 and **the customer bound their own box at 16:29:55 with zero failed attempts**, `appliance_bound` carrying source `customer_selfbind`, credential delivered 26 s later with no operator action (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). The new-ISO console banner rendered on the real box. **Slice 2 = console-passphrase bind → R-27c** (below). Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. **Multi-box per link is out of scope by design** (single-active, one-shot → binding several boxes = repeated operator sends) | | R-27b | **Customer self-bind, second-box flow (controller side).** For a customer who ALREADY has a bound box and installs another, the controller shows a dismissable "bind another box" prompt (and a bind-later entry under settings) that walks to the hub `/bind/` page — so a returning customer isn't emailed a fresh operator-sent link for every box. Mechanism sketched in the hub v0.66.0 REPORT; NOT built (R-27 slice 1 deliberately did not touch the controller). | M | idea (minted by hub v0.66.0) | Origin: hub v0.66.0 slice-1 ship (first-box only). Reuses the same `/bind/` public page + tokenized-link machinery; adds a controller-side entry point + the operator "mint a link for an existing customer" affordance | -| R-25b | **Customer DELETE ↔ RESET consistency.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown — it purges escrow custody + drops the config only. Decide the model: DELETE requires a prior RESET, or DELETE subsumes RESET's teardown, or they stay orthogonal (RESET = recycle-in-place, DELETE = escrow-purge). | S | idea | Origin: hub v0.61.0 RESET ship. Flips a future "customer fully offboarded (external resources released)" map row. Cheap once the model is chosen | +| R-25b | **RULED: customer DELETE becomes a guided full-teardown cascade.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown. | M (was S) | **RULED 2026-07-21 — spec to follow; NOT implemented** | **operator ruling 2026-07-21**: **DELETE subsumes the whole cascade, behind explicit consent.** Three separate acknowledgements, each its own checkbox — (1) the host(s) will be deleted, (2) the customer will be RESET including external teardown and offsite data destruction, (3) the customer record and escrow will be purged — plus a **typed customer-name confirmation** before the button arms. Internal order is **host-delete → RESET → delete**, which preserves every existing invariant rather than relaxing any: RESET keeps its no-hosts precondition (hosts are already gone by then), and escrow keeps its demote-then-purge custody rule (host delete DEMOTES to retained custody, the final delete PURGES — the one true purge point). **Re-sized S → M: this is a multi-step destructive wizard with three acks and a typed confirmation, not a checkbox.** Implementation is explicitly NOT part of TASK-E; the row carries the ruling and awaits its own spec. **It no longer blocks R-3** — the model is decided, so the friend-alpha runbook can be written against it | | R-25 | **Device-node TOCTOU hardening (drive init).** Graduate the controller v0.141.0 Observation: the `format → resolveEnrollUUID(path) → AssignDisk(uuid)` sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. | S | idea | From the v0.141.0 F6 commit's security-review finding (`felhom-controller` REPORT). Low real risk (single-operator, agent-guarded), but cheap to close | | R-24 | **Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17).** The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE `SetConfig` (live cgroup, no reboot, Phase-0 proven). **Framing note:** the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). **COMPLETE — the apply is proven in BOTH directions on metal (2026-07-18):** customer zero shrank the guest **11675 → 8192 MB (16:50:22)** and grew it **8192 → 12288 MB (17:02:17)**, each a live cgroup apply with no reboot, the new total rippling into the deploy page's memory math. The nested demo had proven the refusals; the N100 proved the applies. `tests/VALIDATION-n100-rehearsal-2026-07-18.md` ledger 9. **Cores/live-resize as hub desired-state is NOT built** (deferred, low demand). | M | **SHIPPED + PROVEN-LIVE** | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". | | R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster | @@ -87,8 +88,8 @@ | R-37 | **Post-RESET health card shows stale pre-RESET warnings.** After a RESET the card should read **„RESET óta nincs adat"** instead of carrying warnings about a lifecycle that no longer exists. | XS | **SHIPPED (hub v0.67.0, 2026-07-18)** | The customer page raises a banner when a RESET **completed** after the newest report, quoting „RESET óta nincs adat" and the reset timestamp, because until the box reports again every health figure describes a lifecycle that no longer exists. Deliberately narrow: an **in-flight** reset does not trigger it (only a completed one), and it **clears itself** on the first post-RESET report. Ties resolve to STALE — SQLite timestamps are second-resolution and a same-second report almost certainly arrived just before the reset destroyed what it describes; erring the other way would hide the banner exactly when it matters most. Red-proofed (neutering the predicate fails the assertion). — Origin: 2026-07-18 rehearsal. Same family as R-36 — the hub knows the state changed and the UI has not caught up | | R-38 | **Installer GRUB slice.** A single default „Felhom telepítés" entry; the **interactive installers REMOVED** (safety: an interactive entry is how a wrong-disk manual install happens); felhom background. | S | **SHIPPED (scripts v1.22.0, 2026-07-19)** | Origin: 2026-07-18 rehearsal, alongside R-21's physical closure. Every ISO is repacked after `prepare-iso`: exactly **one** entry („Felhom telepítés", default, 5 s), with Graphical / Terminal UI / serial and the whole **Advanced Options** submenu (nomodeset ×2, three debug variants, Rescue Boot, memtest, UEFI settings) **not emitted** — not hidden, not password-gated. Boot behavior unchanged: the `linux`/`initrd` lines are lifted **verbatim at repack time** from the ISO's own automated entry, so a PVE bump tracks automatically; the build fails if they are missing, if the append line has lost `proxmox-start-auto-installer`, or if `auto-installer-mode.toml` is absent. Gated for 1 entry / 0 submenus / no live banned references, then **re-verified by reading the menu back out of the finished ISO**. Background generated at repack time from `website/assets/og-image_2.png` (one source, no second copy to drift), with the card's own grid phase-locked across the letterbox fill. Live-validated on the nested canary (UEFI/OVMF): renders, auto-fires, reaches the same match-nothing abort as v1.21.0. **Squashfs/theme rebranding still DEFERRED** — post-GRUB screens remain Proxmox-branded. GRUB's `e`/`c` keys also remain reachable (suppressing them needs a superuser password — a separate decision); the menu no longer advertises them | | R-40 | **[P2-HIGH] The update path cannot express a MULTI-HOP major upgrade.** A template pin is a single value; the customer's update button pulls whatever the catalog now says. For apps whose upstream forbids version skipping this produces a broken upgrade. Nextcloud is explicit: *"You cannot skip major releases. Please re-run the upgrade until you have reached the highest available release."* Campaign 7 moved its template **31 → 34** (a fresh deploy validates fine — 302, 3/3 healthy), so an existing 31 customer pressing update would attempt a jump Nextcloud refuses. | M | idea | Origin: CAMPAIGN 7 (`audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` §7 F7). Not nextcloud-only — any app with sequential-major rules (gitea, tandoor, outline…) has the same shape. Directions: a per-app `upgrade_path:`/`max_hop:` in `.felhom.yml` that the update button walks in stages; or refuse-and-explain when the installed major is >1 behind; or pin an intermediate "stepping-stone" tag. **Until this exists, a >1-major catalog bump is safe for NEW deploys and unsafe for the update button** — which is exactly the asymmetry the campaign's MAJOR flag was meant to record but cannot enforce | -| R-41 | **[P2-HIGH] The catalog has no standing "does every template still deploy?" check.** Campaign 7 was the first thing that ever tried to deploy all 53 apps, and found **5 that had NEVER been deployable**: papra (missing required `AUTH_SECRET`), zipline (v4 renamed `CORE_DATABASE_URL` → `DATABASE_URL`), wishlist (Docker Hub image gone; upstream moved to ghcr.io), homebox (upstream dropped the `v` tag prefix + new required env), glance (needs a seeded `glance.yml` the template never provides — PROVEN pre-existing: the pre-campaign v0.7.4 pin fails identically). Plus **7 broken healthchecks** and 2 apps whose images no longer resolve at all (plant-it, wanderer). | M | idea | Origin: CAMPAIGN 7 (§7 F5/F6). The repo already has the right pattern in `scripts/check-image-pins.py` — a mechanical gate run on every change. Cheap first slice: a **resolvability gate** (`docker manifest inspect` every pin) would alone have caught plant-it, wanderer, wishlist and homebox, and needs no box. Full slice: a periodic deploy-all sweep on the demo box reusing the campaign's engine. **Silent rot is the real risk** — an app can die upstream and nobody learns until a customer clicks Telepítés | -| R-42 | **Ruling needed: do DB/cache sidecar majors follow the app, or the newest tag?** Campaign 7 deliberately did NOT bump sidecar majors (postgres 16→18, redis 7→8, mariadb 11.6→12.3, postgis 16→17) while bumping ~40 app images to current. | S | **decision pending (Viktor)** | Origin: CAMPAIGN 7 §4. The case for not bumping: a DB major is a **data-plane decision the application owns** — immich proves it, upstream pins one specific tested `postgres:14-vectorchord…` build — and `postgres:16-alpine`/`redis:7-alpine` already track the newest patch inside their major, so they are not stale. The case for bumping: EOL majors eventually stop getting security patches, and "we never bump" silently becomes "we ship EOL databases". Suggested shape: per-app sidecar pin follows **upstream's own compose** where upstream publishes one, else stay within the current major and revisit at that major's EOL date | +| R-41 | **[SLICE 1 SHIPPED 2026-07-21] The catalog has no standing "does every template still deploy?" check.** Campaign 7 was the first thing that ever tried to deploy all 53 apps, and found **5 that had NEVER been deployable**: papra (missing required `AUTH_SECRET`), zipline (v4 renamed `CORE_DATABASE_URL` → `DATABASE_URL`), wishlist (Docker Hub image gone; upstream moved to ghcr.io), homebox (upstream dropped the `v` tag prefix + new required env), glance (needs a seeded `glance.yml` the template never provides — PROVEN pre-existing: the pre-campaign v0.7.4 pin fails identically). Plus **7 broken healthchecks** and 2 apps whose images no longer resolve at all (plant-it, wanderer). | M | idea | Origin: CAMPAIGN 7 (§7 F5/F6). The repo already has the right pattern in `scripts/check-image-pins.py` — a mechanical gate run on every change. Cheap first slice: a **resolvability gate** (`docker manifest inspect` every pin) would alone have caught plant-it, wanderer, wishlist and homebox, and needs no box. Full slice: a periodic deploy-all sweep on the demo box reusing the campaign's engine. **Silent rot is the real risk** — an app can die upstream and nobody learns until a customer clicks Telepítés | **SLICE 1 SHIPPED 2026-07-21 — `app-catalog-felhom.eu/scripts/check-image-resolvable.py`** (+ 14 fixture tests, no network, resolver injected). Resolves every unique pin with `docker manifest inspect`, ONE image at a time; exit 0 / 1 (the registry says GONE) / 2 (inconclusive). **Two traps encoded, both hit live while building it:** (a) `docker manifest inspect` prints `toomanyrequests: …` and **still exits 0** — the same exits-0-on-failure shape as the ISO tooling's `validate-answer`, so stderr is inspected even on rc=0; (b) the inverse and more dangerous one — the first full sweep called **24 of 65 pins dead, including `postgres:16-alpine` and `redis:7-alpine`**, purely because Docker Hub throttled it partway through. Ambiguity therefore resolves to INCONCLUSIVE and never to an accusation: a gate that cries wolf gets ignored, and then it protects nothing. **The full sweep is still OWED** — DooPlex is not logged in to Docker Hub, so the 52-app table needs one re-run after `docker login`. Wired into `CLAUDE.md` + `REUSE.md` as a start-of-campaign / pre-publish-train step. It immediately paid for itself: it is what turned plant-it and wanderer from 'images do not resolve' into two DIFFERENT diagnoses (see the 2026-07-21 catalog entry). Full slice — the periodic deploy-all sweep on the demo box — remains open | +| R-42 | **RULED: sidecar majors follow the APP, never the newest tag.** Campaign 7 deliberately did NOT bump sidecar majors (postgres 16→18, redis 7→8, mariadb 11.6→12.3, postgis 16→17) while bumping ~40 app images to current. | S | **RULED 2026-07-21 — option (a)** | Origin: CAMPAIGN 7 §4. The case for not bumping: a DB major is a **data-plane decision the application owns** — immich proves it, upstream pins one specific tested `postgres:14-vectorchord…` build — and `postgres:16-alpine`/`redis:7-alpine` already track the newest patch inside their major, so they are not stale. The case for bumping: EOL majors eventually stop getting security patches, and "we never bump" silently becomes "we ship EOL databases". Suggested shape: per-app sidecar pin follows **upstream's own compose** where upstream publishes one, else stay within the current major and revisit at that major's EOL date | | R-43 | **[P2-HIGH] No offsite restore path can restore a database — offsite restore cannot reconstitute a DB-indexed app.** All three offsite buttons are file-only. „Ellenőrző visszaállítás" and „Teljes visszaállítás" restore into a **scratch/verify folder** and never load a dump into postgres (`controller/internal/backup/offbox_restore.go:171-232`, `:130-164`); „Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" is `rsync -a --ignore-existing` over **mandatory userdata paths only** — it never touches a DB and never stops/starts the stack (`offbox_restore.go:346-421`). The DB dump is faithfully carried *into* the snapshot and can never be replayed *out of* it by any offsite action; the only code that replays into a live DB is `RestoreFromRecoveryUnit` (`backup/restore_unit.go:74`), reachable solely from the **local** restore panel (`web/handlers.go:1108`). Compounding: the `live recovery unit present — not overwriting` guard (`offbox_restore.go:400-405`) skips the **entire** unit — compose, manifest **and db-dumps** — whenever a local `backups/primary/` exists, which is true on any box whose local backup has run, so on a normal box the offsite dump is never even staged to disk. | M | **SHIPPED controller v0.148.0 (2026-07-19) — live acceptance PENDING** | **SHIPPED:** new „Teljes visszaállítás (fájlok + adatbázis)" (`ReconstituteFromOffsite`, `/backup/offbox/reconstitute`): safety dump → stop → files overwritten to the snapshot version → start → the snapshot's own dump replayed (`reimportDBDumpsFrom`, reading the SCRATCH unit — the live unit is still never overwritten) → health wait. Two invariants, both red-proofed: **nothing is ever deleted** (`rsync -a`, no `--ignore-existing`, no `--delete`; post-snapshot files survive as extras) and **the undo exists before the act** (the `pre-restore-` dump is verified ON DISK before anything is stopped/overwritten/replayed; if it cannot be taken the operation refuses with zero changes). Outcome-shaped flash; the old missing-only button now states what it does NOT do. Deployed to demo guest 9201, healthy; page renders, button correctly gated behind a prepared full scratch. **STILL OPEN — the §9 live acceptance has NOT run** (upload → push → empty the trash for real → one button → photos back in the timeline). Until it does, no capability-map flip, the customer-restore row stays MISSING and R-3 stays DRAFT. — **Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (live on demo-felhom).** Viktor deleted 11 immich photos to test offsite restore; both place runs merged **0 files** and flashed success. Root cause is two-layered — the UI delete only *trashed* the assets so nothing was missing to restore, **but a valid test would also have failed**: files return, the library stays empty, because immich's truth lives in postgres. **The customer-visible failure mode: correct bytes on disk and an app that cannot see them.** Severity case: this is the offsite tier — the one that survives losing the box — and most of the catalog is DB-indexed. Direction (needs a spec, do not improvise): either offsite gains a real DB-replay leg reusing `RestoreFromRecoveryUnit` behind a stop→restore→start sequence, or the offsite UI states plainly that it restores files only and routes the customer to the local recovery-unit flow for the DB. Pairs with R-44 (a replay leg is worthless while the dump is stale). Flips: the **customer-restore map row stays MISSING** and gains this as a note — the C6 evidence attempt found a product gap, which is the rehearsal system working | | R-44 | **[P2-HIGH] A manual offsite push ships an unrefreshed DB dump — "backed up now" is false for the DB half.** `offboxRunHandler` → `RunOffboxBackup` goes straight to the restic push and never calls `RunDBDumps` / `captureAllRecoveryUnits` (`controller/internal/web/offbox_handlers.go:203-227`, `backup/offbox.go:574-759`); the recovery unit merely **enumerates** existing dump filenames via `listFileNames`, never creates them (`backup/recovery_unit.go:105-106`). Dumps come only from the separate local `db-dump` daily at **02:30** (`cmd/controller/main.go:542`), with the scheduled offsite at 04:15 — so a *manual* run at any other hour ships a dump up to ~24 h old. **There is no freshness check and no RPO surface anywhere:** zero `RPO` hits across `controller/`; `offboxUnitTime` is only a two-drive tiebreak (`offbox.go:827-837`); the `DBValidationCache` exists (`backup.go:364-370`) but no offsite or restore path reads it. | S–M | **SHIPPED controller v0.148.0 (2026-07-19)** | **SHIPPED:** every offsite run — **manual AND nightly** — now refreshes the DB/volume dumps and recovery units (`offsitePreDump` → `runDBDumpsInternal`) BEFORE the restic capture, so each snapshot is an internally coherent `{DB@T, files@T}` bundle and retention becomes a history of restorable points. Order is the mechanism and is red-proofed (moving the capture first yields `[capture dump]`): the gap can only ADD files the DB does not reference yet, never remove one it does. This also makes the nightly ordering **structural** rather than a coincidence of two scheduler entries at 02:30 and 04:15. Each unit manifest carries `offsite_run_id` + `dumps_at`, so a pair's coherence is verifiable at restore time instead of assumed; the periodic refresh carries a prior stamp forward and never invents one. A dump-leg failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none). Honesty surfaces, all warn-level and none a gate: an unstamped pre-v0.148 pair reports its skew in the confirm, and `ValidateDump` gained an **exact-match** accounts-table sniff for customer-empty dumps (a substring match on "user" would flag `user_metadata`/`album_user`/`user_audit` on every healthy single-user box — red-proofed). — **Evidence: `audits/DIAG-immich-restore-2026-07-19.md`.** Today's unit dump `immich-postgres.sql` (51 954 452 B, mtime **02:30 CEST**) probed to **`asset: 0 rows`, `user: 0 rows`, `album: 0 rows`** — the 52 MB is entirely immich's shipped `geodata_places`/`naturalearth_countries` reference data. It predates both the admin user (created 07:56:25) and the photos (07:57). Same for the unit's `immich_immich_postgres_data.tar` (323 MB, also 02:30). **A dump that looks substantial by size can contain zero customer content** — size is not a health signal, and nothing in the product says otherwise. **Latent hazard:** had a full restore actually loaded that dump it would have written an empty DB over the live one, destroying the trashed rows that were the only surviving recovery path. Direction: dump-before-push on manual runs (the honest fix), **or** an explicit RPO line in the UI („adatbázis-állapot: ") so the operator/customer can see what they are actually shipping. Cheap interim: surface dump mtime + row-count sanity from the existing `DBValidationCache` on `/backups/restore` | | R-45 | **[P2] Unified async-job feedback.** Every long operation invents its own progress surface, or none. Tonight produced three more one-off cards (v0.147.x: samba bring-up, offsite progress, restore result) on top of two existing patterns (deploy 3-step panel; storage-init/netstorage status poll). They agree on nothing: some use `{ok,data}` envelopes and some raw JSON, some poll 1 s / 1.5 s / 3 s, some are in-memory-only and lie after a restart, and each re-implements single-flight + snapshot + phase→Hungarian mapping. | M | idea | Origin: 2026-07-19 feedback slice 1 (controller v0.147.0). The cases to generalise from are all in-tree: `web/storage_init_job.go` (the best shape — acquire/release/set/snapshot), `web/netstorage_job.go`, `web/samba_ensure_job.go`, `backup/opstatus.go`, `backup/offbox_progress.go`. Shape: one job registry + one poll endpoint + one client-side renderer, phases declared per job. **Two lessons tonight that any framework must encode:** (1) a terminal state must be **probed, not inferred** — `compose up -d` exits 0 on a crash-loop; (2) a progress source that reports nothing is normal, not broken — restic reports 0 bytes for a whole incremental run, and a bar that sits at 0% is worse than no bar. Also fixes the restart hole: in-memory job state currently vanishes and the card silently disagrees with reality **2026-07-20 — the first bill for NOT having this arrived, and it was customer-facing.** The samba card's poll (`web/samba_ensure_job.go` + `sharing.html`) mixed a job EDGE and a service LEVEL on one JSON field, and `/sharing` reload-looped at ~1.2 s for every customer with sharing enabled until controller v0.151.0 (`audits/DIAG-sharing-2026-07-20.md`, S-1/S-4). v0.151.0 fixed THAT card's contract only — the framework is still this item. **Third lesson for it to encode, beside the two already listed:** a phase a client answers with a one-shot action must be an EDGE the registry SERVES ONCE, and must never be synthesised from a level; if it can be re-read, it will be re-acted on. | diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 8d24c0a..9c1c92e 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,5 +1,21 @@ # Felhom scripts — Changelog +## build-hub.sh v1.23.0 — the hub build script enters the repo (2026-07-21) + +It never was in one. `/mnt/5_hdd/felhom.eu/build/felhom-hub/build.sh` lived only in a DooPlex build +directory — unversioned, unreviewed, un-diffable and gone with the box, while being the only way the +hub image gets built. Recorded as a follow-up in `felhom.eu/REPORT.md` §6; adopted here. + +Adopted verbatim — the build logic is byte-identical (verified by diff), and the only additions are +a `SCRIPT_VERSION` constant echoed in the banner and a header naming the canonical/operative split. +The build-dir path is now a **symlink** to this file, so there is one source of truth and a repo edit +is live immediately (the `install_skills.py` pattern). The previous copy is retained beside it as +`build.sh.pre-adopt-2026-07-21`. + +It already carried the GitOps deploy hint fixed earlier in the week (manifest bump → hard-refresh → +deliberate sync, and the named trap that `kubectl set image` / `kubectl apply` are silently reverted +by the next ArgoCD sync) — that wording now lives in git along with the rest of it. + ## install_skills.py — cross-platform (POSIX symlink / Windows junction) (2026-07-19) Claude Code now runs on DooPlex (Debian 13), where `mklink /J` does not exist — the script would diff --git a/scripts/build-hub.sh b/scripts/build-hub.sh new file mode 100755 index 0000000..4f57272 --- /dev/null +++ b/scripts/build-hub.sh @@ -0,0 +1,218 @@ +#!/usr/bin/env bash +# ============================================================================= +# felhom-hub — Docker image build script +# ============================================================================= +# CANONICAL LOCATION: felhom.eu/scripts/build-hub.sh (this file, in git). +# OPERATIVE LOCATION: /mnt/5_hdd/felhom.eu/build/felhom-hub/build.sh — a SYMLINK to this file, so +# there is exactly one source of truth and a repo edit is live immediately (the install_skills.py +# pattern). It lived ONLY in that build dir until v1.23.0, outside any repo and therefore +# unversioned, unreviewed and unrecoverable — noted as a follow-up in felhom.eu/REPORT.md §6. +# +# Copies hub/ source from the felhom.eu git repo, builds the Docker image. Build artifacts stay in +# the build dir's workspace/ — the git repo stays clean. +# +# Usage: +# ./build.sh # Build for current platform, tag as :dev +# ./build.sh 0.1.0 # Build with version tag +# ./build.sh 0.1.0 --push # Build + push to Gitea registry +# ./build.sh 0.1.0 --multiarch # Build amd64+arm64 + push +# ============================================================================= +set -euo pipefail + +SCRIPT_VERSION="1.23.0" + +# --- Configuration (edit these if your paths differ) --- +REPO_DIR="/mnt/5_hdd/felhom.eu/git/felhom.eu" +HUB_SRC="${REPO_DIR}/hub" +REGISTRY="gitea.dooplex.hu/admin" +IMAGE="${REGISTRY}/felhom-hub" + +# Build workspace — a temp directory next to this script, NOT in the git repo +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +BUILD_DIR="${SCRIPT_DIR}/workspace" + +# --- Parse arguments --- +VERSION="${1:-dev}" +ACTION="${2:-}" # --push or --multiarch + +# --- Colors --- +RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m' +info() { echo -e "${GREEN}[INFO]${NC} $*"; } +warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } +error() { echo -e "${RED}[ERROR]${NC} $*"; } +step() { echo -e "${CYAN}[STEP]${NC} $*"; } + +# --- Pre-flight checks --- +if [[ ! -d "${HUB_SRC}" ]]; then + error "Hub source not found: ${HUB_SRC}" + error "Clone the repo first: git clone https://gitea.dooplex.hu/admin/felhom.eu.git ${REPO_DIR}" + exit 1 +fi + +if ! command -v docker &>/dev/null; then + error "Docker not found." + exit 1 +fi + +# Get git commit for build metadata +GIT_COMMIT=$(cd "${REPO_DIR}" && git rev-parse --short HEAD 2>/dev/null || echo "unknown") + +echo "" +info "╔══════════════════════════════════════╗" +info "║ felhom-hub build ║" +info "╚══════════════════════════════════════╝" +info "Version: ${VERSION} (build-hub.sh v${SCRIPT_VERSION})" +info "Commit: ${GIT_COMMIT}" +info "Source: ${HUB_SRC}" +info "Build: ${BUILD_DIR}" +info "Image: ${IMAGE}:${VERSION}" +echo "" + +# ========================================================================= +# Step 1: Pull latest code +# ========================================================================= +step "1/3 — Pulling latest code..." + +cd "${REPO_DIR}" +git pull --ff-only 2>/dev/null || warn "git pull failed (offline or dirty tree?)" + +# ========================================================================= +# Step 2: Sync source to build workspace +# ========================================================================= +step "2/3 — Syncing source to workspace..." + +mkdir -p "${BUILD_DIR}" + +rsync -a --delete \ + --exclude '.git' \ + --exclude 'bin/' \ + --exclude '*.exe' \ + "${HUB_SRC}/" "${BUILD_DIR}/" + +info "Source synced to ${BUILD_DIR}" + +# Sync app assets from website directory into build workspace +WEBSITE_ASSETS_DIR="${REPO_DIR}/website/assets" +if [[ -d "${WEBSITE_ASSETS_DIR}" ]]; then + mkdir -p "${BUILD_DIR}/assets" + cp "${WEBSITE_ASSETS_DIR}"/*-logo.svg "${BUILD_DIR}/assets/" 2>/dev/null || true + cp "${WEBSITE_ASSETS_DIR}"/*-logo.png "${BUILD_DIR}/assets/" 2>/dev/null || true + cp "${WEBSITE_ASSETS_DIR}"/*-screenshot-*.webp "${BUILD_DIR}/assets/" 2>/dev/null || true + ASSET_COUNT=$(ls "${BUILD_DIR}/assets/" 2>/dev/null | wc -l) + info "App assets synced (${ASSET_COUNT} files)" +else + warn "Website assets dir not found: ${WEBSITE_ASSETS_DIR}" + mkdir -p "${BUILD_DIR}/assets" +fi + +# Verify structure +cd "${BUILD_DIR}" + +for dir in cmd/hub internal/api internal/store internal/web; do + if [[ ! -d "${dir}" ]]; then + error "Missing expected directory: ${dir}" + error "Hub source structure may be incomplete." + exit 1 + fi +done +info "Package structure verified ✓" + +if [[ ! -f "go.mod" ]]; then + error "go.mod not found in build workspace!" + exit 1 +fi + +# Ensure go.sum exists (Dockerfile COPY will fail without it) +if command -v go &>/dev/null; then + info "Running go mod tidy..." + go mod tidy 2>&1 || warn "go mod tidy had issues (Docker build may still work)" +elif [[ ! -f "go.sum" ]]; then + warn "go.sum missing and Go not installed locally — creating empty go.sum" + warn "(Docker build stage will resolve dependencies via 'go mod download')" + touch go.sum +fi + +# ========================================================================= +# Step 3: Docker build +# ========================================================================= +step "3/3 — Building Docker image..." + +BUILD_ARGS=( + --build-arg "VERSION=${VERSION}" + --build-arg "BUILD_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")" +) + +case "${ACTION}" in + --push) + info "Building for current platform + pushing..." + docker build "${BUILD_ARGS[@]}" \ + -t "${IMAGE}:${VERSION}" \ + -t "${IMAGE}:latest" \ + . + + info "Pushing..." + docker push "${IMAGE}:${VERSION}" + docker push "${IMAGE}:latest" + ;; + + --multiarch) + info "Building multi-arch (amd64 + arm64) + pushing..." + + # Ensure buildx builder exists + if ! docker buildx inspect felhom-builder &>/dev/null; then + info "Creating buildx builder (one-time setup)..." + docker buildx create --name felhom-builder --use --bootstrap + else + docker buildx use felhom-builder + fi + + docker buildx build "${BUILD_ARGS[@]}" \ + --platform linux/amd64,linux/arm64 \ + -t "${IMAGE}:${VERSION}" \ + -t "${IMAGE}:latest" \ + --push \ + . + ;; + + *) + info "Building for current platform (local only)..." + docker build "${BUILD_ARGS[@]}" \ + -t "${IMAGE}:${VERSION}" \ + -t "${IMAGE}:latest" \ + . + ;; +esac + +# ========================================================================= +# Summary +# ========================================================================= +echo "" +info "╔══════════════════════════════════════╗" +info "║ Build complete ✓ ║" +info "╚══════════════════════════════════════╝" +info "Image: ${IMAGE}:${VERSION}" + +# Show image size if available locally +SIZE=$(docker image inspect "${IMAGE}:${VERSION}" --format='{{.Size}}' 2>/dev/null || echo "") +if [[ -n "${SIZE}" ]]; then + SIZE_HUMAN=$(numfmt --to=iec "${SIZE}" 2>/dev/null || echo "${SIZE} bytes") + info "Size: ${SIZE_HUMAN}" +fi + +echo "" +if [[ "${ACTION}" == "" ]]; then + info "Image is local only. To push: ./build.sh ${VERSION} --push" +else + info "Image pushed to registry." + echo "" + info "Deploy to k3s — GitOps ONLY. The manifest in git is the source of truth:" + info " 1. bump the image tag in ${REPO_DIR}/manifests/hub.yaml to ${VERSION}, commit, push" + info " 2. hard-refresh + sync the ArgoCD app (auto-sync is OFF — the sync is deliberate):" + info " sudo kubectl -n argocd annotate application felhom argocd.argoproj.io/refresh=hard --overwrite" + info " sudo kubectl -n argocd patch application felhom --type merge \\" + info " -p '"'"'{\"operation\":{\"initiatedBy\":{\"username\":\"cc\"},\"sync\":{\"syncStrategy\":{\"apply\":{}}}}}'"'"'" + info "" + info " NEVER \`kubectl set image\` or \`kubectl apply\` by hand — ArgoCD reverts it on the next" + info " sync, so the change looks applied, works for a while, and then silently disappears." +fi +echo "" diff --git a/skills/felhom-testing/SKILL.md b/skills/felhom-testing/SKILL.md index 9069ffd..1e77f38 100644 --- a/skills/felhom-testing/SKILL.md +++ b/skills/felhom-testing/SKILL.md @@ -57,8 +57,13 @@ classifier functions get fixture tables (agent `classifyClaim` style). go build ./... && go vet ./... && go test ./... ``` Run it in the module dir: `felhom.eu/hub/`, `felhom-controller/controller/`, `felhom-agent/` root. -Known flake: agent `TestGenerateRecoveryCode_EntropyAndFormat` fails ~1/5 (hyphenated wordlist word) — -re-run before diagnosing; it is not a regression. +**No known flakes.** The long-standing agent `TestGenerateRecoveryCode_EntropyAndFormat` ~1/5 failure +was **fixed in agent v0.93.0 (2026-07-21)** — and it was never a flaky test. It was a real defect the +test was correctly detecting and everyone had been told to re-run past: the EFF wordlist contains four +hyphenated entries, so a recovery code could come out reading as 11 words instead of 10. If it fails +now, it is a regression. **The lesson generalises: "known flake, just re-run it" is a diagnosis, and +it needs the same evidence as any other one.** A test that fails at a stable, explainable rate is +usually telling the truth about a rare input, not misbehaving. ## Live validation doctrine (after unit-land)