# ROADMAP — future features & open work

> **What this is:** the prioritized decision log of planned/open work. Items are *intentions*, not
> claims about live behavior — the capability map (`architecture/00-capability-map.md`) is the only
> place that states what the platform does today.
>
> **Lifecycle:** idea → spiked → spec'd → in-progress → **shipped** (item collapses to a one-liner
> with the version, and the corresponding capability-map row changes status with evidence). Items
> can also be **killed** (keep the one-liner + why — decisions are worth remembering).
>
> **Coupling rule:** every item names the capability-map row(s) it flips. Every map gap row points
> back here by ID. Neither file duplicates the other's content.
>
> **Priorities:** P1 = closed-alpha blocker · P2 = close during alpha · P3 = post-alpha.
> Existing loose notes in this folder (`FOLLOWUP-*`, `FIX-M*`) are absorbed as references below.

---

## P1 — closed-alpha blockers

| ID | Item | Size | Status | Notes / map rows flipped |
|----|------|------|--------|--------------------------|
| R-115 | **Publishing is a remembered step — forgotten within eight hours of being documented as forgettable** | M | idea — **WAITING-ON-OPERATOR**, 2026-07-29 | A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published, so **"deployed" and "installable" are independent states that drift silently**. **Instance 1 — R-111** (morning): 17 agent releases v0.97.0–v0.113.0 stranded; a new customer would have installed without the whole R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT. Found only because the E-2d Phase 0 gate happened to look. **Instance 2 — agent 0.114.0** (same afternoon): the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, never published — which blocked Session C, since a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix. **The finding is the RECURRENCE, not either instance** — both are fixed. R-111 named this leg in its own text (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; it recurred the same day, which is the evidence that **a note is not a mechanism**. **Class: → R-29, one layer up** (a control that exists and is never walked) — deliberately NOT given a second ID. **Filed as its own item rather than reopening R-111** because R-111's finding (the channel *was* stale) is closed and verified end-to-end by the E-2d install, while the process defect that caused it is a distinct problem with a distinct fix and a distinct owner. **Operator's decision, mechanisms first:** (a) publish as a step in the build/release path so deployed and installable cannot diverge; (b) a gate that refuses to deploy an unpublished+unvouched version — strongest, fails closed; (c) a session-end checklist entry; (d) accept manual + a pre-Session-C verification. **(a)/(b) are mechanisms, (c)/(d) are reminders — and R-29's whole finding is that reminders do not hold.** No code written when filed, by design |
| R-113 | **The drive-absent gate cannot fire on device loss — E-2b's alarm is wired to an unreachable condition** | M | idea — **PROVEN LIVE 2026-07-29** | `planDriveGates` (`felhom-controller/internal/web/intermediary.go:216-262`) treats a path as present by OR-ing in `d.BoundUnderParent`, which the agent derives from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc/<pid>/mountinfo`"* (`internal/localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is not device-bound and its mountinfo entry outlives the device**, so the gate reads it as present and `notifyDriveAbsent` is never called. Live on a fresh box: target drive hot-detached, agent said `enrolled drive absent by UUID` every 20 s for 4½ min, controller logged **0** `[gate]` lines, hub received **zero** events — neither `backup_target_absent` nor the generic `storage_disconnected`. Not a virtualisation artefact (device-bound-mount vs manual-bind is the same on metal); caveat: SCSI hot-detach, physical unplug not staged. **Sixth instance of seam-built-but-never-wired — E-2b wired the seam to a condition that cannot occur.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 |
| R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 |
| R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 |
| R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/<customer>`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 |
| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path |
| R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
| R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none` → `p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** |

## P2 — during alpha

> **Sub-rank `P2-HIGH` = close before the first REMOTE tester.** These are the 2026-07-18 N100
> rehearsal's findings (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). They are not P1 — the
> rehearsal proved the product flow works — but each one either misleads the operator, misleads the
> customer, or hides a failure, and all of that gets materially worse the moment the box is somewhere
> you cannot walk over to.

| ID | Item | Size | Status | Notes |
|----|------|------|--------|-------|
| R-39 | **[P2-HIGH] The PBS DR tier can be `applied` and dead at the same time — and nothing notices.** On the reborn N100 the descriptor auto-provisioned and the agent converged `state=applied`, yet `pvesm status` reports `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive` and a direct probe with the stored credential 401s on **every** endpoint including `/version` (WG transport healthy: handshake 9 s, 27.9 ms RTT — so authentication, not ACL scope). Three compounding defects: **(a)** a **mint/consume race** — the hub minted a SECOND token secret at 16:47:52, two minutes *after* the agent applied the first, and `consumed_at` is still NULL; **(b)** the converged state machine will not re-apply, so the box is pinned to a stale secret; **(c)** the agent's 15-minute PBS verify loop **cannot read the credential to detect any of it** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — the non-root agent writes that file through a root sudo wrapper, then reads it directly). | M | **CLOSED 2026-07-21 — PROVEN LIVE (hub 0.68.1 + agent 0.91.2)** | **DIAGNOSIS (2026-07-18, live on the N100 — supersedes the initial hypothesis).** The brief guessed "the re-mint fails to bump the generation". **That is FALSE and no hub fix was shipped:** `store.SetHostDesired` bumps `desired_generation` unconditionally (it went 2→3 on the re-issue), and `web/configs.go`'s `applyPBSDR` is likewise exonerated — its "no re-key, no second secret, no spurious generation bump" comment is accurate, guarded by the `cur != nil && cur.Namespace != ""` early return, and the hub log shows mint #2 came from the **re-issue** path, not from an Edit-tab Save. **The real mechanism is a signal mismatch between the two tiers.** The hub's re-consume signal is *a generation bump + a poke*; the agent's re-apply trigger is *a change in the DESCRIPTOR CONTENT HASH* (`felhom-agent internal/pbsdr/manager.go` ~L235: `if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { return }`). An ep0 credential re-issue re-keys the **secret of an existing token**, so `token_id` and `fingerprint` are unchanged and the descriptor is **byte-identical** — only the side-table `host_pbs_secrets` row rotates. Same hash → the converged agent short-circuits → the fresh secret is never consumed → the box keeps presenting a revoked credential → **401 forever**. Proof in one line: the agent's `consumed-failed.json` carries hash `a4e5424…`, **identical** to the `marker.json` written 2 min before the re-issue. The comment at `hub/internal/web/pbsdr.go:320` asserts the reissue refreshes the descriptor "with the NEW token_id/fingerprint" — that assumption is simply false for this op. **A SECOND, independent defect was found while healing** and is fixed: `configs/felhom-pbs-apply`'s `reconcile` passed `--server` to `pvesm set`, which PVE rejects wholesale as a create-only parameter, so *every* re-apply exited 255 — and because the agent consumes the one-time secret BEFORE calling the wrapper, each re-issue **burned a credential**. Fixed in **agent v0.90.1** (one argv line + red-proof `TestReconcileNeverPassesServerToPvesmSet`); proven live (`pvesm set --server <same>` rejected, without it rc 0). **BOX HEALED 2026-07-18:** after the wrapper hotfix, Viktor's Re-issue click converged in 9 s and the tier went `401/inactive` → **`active`** (token probe 401 → 200) with a real backup landing PBS-side — `felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z`, 9 744 319 312 B, encrypted under the escrowed key. **2026-07-21 — the "cheap half" is ALREADY CLOSED in the field, and the planned artifact publish was CANCELLED as a false signal (operator ruling, same day).** v0.90.1 was to be built, published and deployed as "the PBS wrapper argv fix". Inspection of `9596d5a` shows it changes **zero non-test Go files** — `CHANGELOG.md`, `REPORT.md`, `configs/felhom-pbs-apply` (the fix) and `internal/pbsdr/manager_test.go` (the red-proof); its own commit message states *"the Go binary is unchanged"*. So the 0.90.1 binary is functionally identical to the 0.90.0 in the field. The fix is live anyway by two independent paths: felhom-pve carries the hotfixed wrapper since 2026-07-18 (`args=(--fingerprint "$fp")` at L107, `.bak-20260718-preR39` retained), and **every new install fetches the wrapper unversioned** — `felhom-host-install.sh:1914` `fetch_raw` pulls `configs/felhom-pbs-apply` from `raw/branch/main`, and `9596d5a` is an ancestor of `main`. Publishing 0.90.1 would therefore have delivered no behaviour change, restarted the agent on a production host at a remote site for nothing, and — once the Day-0 manifest was saved to 0.90.1 — advertised that the fix shipped as a versioned artifact when the artifact channel never carried it. **Ruled: leave 0.90.0 published; record the closure here instead.** Evidence: `felhom-controller/REPORT.md` §5 (2026-07-21). **This surfaced a NEW item — see R-50b: a root-owned privileged host artifact is delivered from `main` with no version, so "which wrapper is on this host" is not answerable from any manifest.**  **FLEET FIX SHIPPED 2026-07-21 — hub v0.68.0 + agent v0.91.2.** All three legs closed. **(a) the re-key is finally VISIBLE:** `host_pbs_secrets` gains a monotonic per-host `generation`, advanced by every fresh MINT and by nothing else, stamped into the descriptor as `secret_generation` — the only field a re-key moves, and because `descriptorHash` marshals the parsed struct, the thing that finally re-arms a converged agent. A re-STAGE deliberately does not advance it (same secret, same descriptor content). `omitempty` is load-bearing: emitting a zero would move every pre-existing descriptor's hash at once. *Spec deviation, deliberate:* the brief said to reuse "the new row's id, no schema change" — there is no row id (the table is `host_id PRIMARY KEY`, UPSERTed last-write-wins) and `created_at` collides within a second, so an additive counter column is the only monotonic source. **(b) the agent can finally READ its own credential:** a narrow wrapper `read` verb + exactly one sudoers line + a `pbsdr-read` capability row. Proven live on felhom-pve: `sudo -u felhom-agent sudo -n felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage` → `rc=0`, 37 bytes, empty stderr. **(c) authentication is PROBED:** `pbs.ProbeAuth` (`GET /version` + an `ErrUnauthorized` sentinel) runs on the 15-minute collect path and becomes a loud `auth_failed` that `pbsdrheal` escalates to a fresh mint through the EXISTING damper — closing the loop end to end. Proven live: `level=DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite`. 403 is deliberately NOT unauthorized (a narrow ACL must not be re-keyed forever); a transport error is UNKNOWN, never a rejection (a blip must not burn a credential). Plus a **consumed_at honesty gauge**: an unconsumed secret past a 15-min grace under a box reporting `applied` — the exact July-18 fingerprint, a disagreement no single tier can see — is surfaced with its own event, deliberately as a SURFACE not a heal (minting on top of an unconsumed secret is the R-39(a) race). **A LOAD-BEARING FACT the spec did not flag, checked rather than trusted:** `Apply` bails out if the storage status probe ERRORS, and `adopt` converges WITHOUT consuming when the storage reads active — so the whole fix depended on PVE's 401 behaviour. PVE's `storage_info` wraps `activate_storage`/`$plugin->status` in `eval{}` and leaves the pre-initialised `active => 0`, so a 401 returns **HTTP 200 with `active: 0`, never an API error** — `Apply` correctly falls through to verify → consume → reconcile. **A defect I shipped and caught:** v0.91.0 built the probe seam and `main.go` never called `SetAuthSink`, so the whole leg was INERT and every test still passed (the seam was injected directly) — same class as controller v0.154.0 the day before; fixed in v0.91.1, artifact superseded not overwritten, and v0.91.2 made a healthy probe observable so "no auth_failed" can never again be confused with "never probed". Four red-proofs, all at the assertion level. **CLOSED 2026-07-21 — STOP-2 and STOP-3 both done, chain proven end to end in 13 seconds.** STOP-3 saved (agent 0.91.2 / sha `34d309be…` / **wrapper sha `104db0a4…`** / MinAgent 0.91.2; the agent reports the same wrapper hash so the R-50b drift gauge reads ok). STOP-2: hub `08:39:31Z` fresh mint **generation 0 → 1** + descriptor `"secret_generation": 1` (token_id/fingerprint byte-identical — the re-key shape that used to be invisible) → agent `10:39:34` reads the credential via the wrapper → `10:39:38` **`REJECTED … applied and DEAD` `previous_state=applied`** (the R-39 state, loud for the first time) → `10:39:45` **`one-time token secret consumed`** (NO short-circuit — the line that never appeared on 2026-07-18) → `10:39:45` set-only `reconcile` → `10:39:47` **`converged state=applied`**. Marker hash moved to `afbb3b41…`; `consumed_at` stamped; secret mtime `2026-07-18` → `2026-07-21 10:39:45`; new credential probes **200**; reports trace `applied → auth_failed → applied`; **zero** self-heal escalations, ONE mint, ONE consume, no `consumed-failed.json`. **A NOTE ON THE FIRST ATTEMPT:** the operator initially pressed the **offsite** re-issue (two different Re-issue actions exist) — harmless to PBS-DR, but it rotated the restic password and correctly marked the escrow STALE, so the recovery-code ceremony had to be re-run. Worth naming the surface explicitly in any future runbook step. Evidence: `felhom-agent/REPORT.md`, `felhom.eu/REPORT.md`. *(Superseded — the spec was written and shipped:)* **REMAINING (fleet, needs its own spec — deliberately NOT improvised):** (a) make a fresh unconsumed secret actually un-converge the agent — either `consumed_at` becomes authoritative or the descriptor carries a secret generation/nonce so the hash moves; (b) fix the verify loop's read path — it reads `/etc/pve/priv/storage/<id>.pw` **directly as non-root**, a file it can only ever *write* through the root wrapper (`/etc/pve/priv` is `0700 root:www-data`; sudoers exposes `create|reconcile|grant` and **no read verb**), so the one loop that could catch this is permanently blind; (c) an auth probe in the reconciler/gauge/ceremony-precheck so `applied` can never mean `401`. — **Original finding note:** discovered by CC while collecting Phase-A evidence; not on the brief's finding list, so its P2-HIGH rank is provisional pending Viktor. The severity case: this is the DR tier, the failure is silent, and it would surface first at a real restore. Suggested shape: make `consumed_at` authoritative (a fresh unconsumed secret must un-converge the reconciler), fix the verify loop's read path (read via the same root wrapper that writes it), and make a failing `pvesm status` a LOUD state rather than a skipped datastore. **Blocks the DR-tier map row's candidate upgrade to PROVEN-LIVE — that upgrade is now explicitly WITHDRAWN.** Evidence `pbs-dr-state.txt`, `hub-state.txt` |
| R-30 | **[P2-HIGH] Liveness presence should come from the wait channel, not the report clock.** The box was powered off at the start of the rehearsal, yet the hub carried it as healthy until the staleness threshold expired ~30 min later (`host_stale` 16:05:24 "no report for 30m"; cleared 16:33:24 "was stale for 27m"). The host-delete guard compounds it: RESET refuses while any host row exists, so a stale-but-"Online" host stalls a forced teardown. | M | idea | Direction: derive presence from **Dir-2 long-poll connectedness (~90 s grace)**, decoupled from notification hysteresis (the hysteresis is right for *alerting*, wrong for *presence*); an agent/ep0 analog can follow. Pairs with R-13/R-23 — the transport already exists, this is about believing it. *(Discussed in-session as "R-29"; that number was already taken by the gate-rot item earlier the same day, so it is R-30.)* |
| R-31 | **[P2-HIGH] Offsite provisioning is synchronous with no status affordance.** Save runs the Hetzner sync in-request, so the request can hit the nginx 504 **while succeeding server-side**: the operator cannot tell failed from slow, and a retry races the first attempt. | M | idea | Direction: make it async + a status card, reusing the proven **awaiting-card/poll idiom** (v0.138.0 escrow card). **Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click.** |
| R-32 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. | M | idea | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` |
| R-33 | **[P2-HIGH] Bootstrap pairing-poll spams the customer-visible console.** Each 30 s pairing poll is a separate `felhom-bootstrap.service` run that exits 1, so systemd prints `Failed to start …` on the screen the customer is watching — **52 FAILED lines in ~11 minutes** while nothing was wrong (the box was correctly waiting to be bound). | S | **SHIPPED (scripts v1.21.0, 2026-07-18)** | `run_pairing()` is now a `while true` loop sleeping `POLL_INTERVAL` (30s — the hub-side rate is unchanged) instead of exiting non-zero per poll, so the unit sits in `activating` and systemd prints nothing. Registration split into `register_appliance()`, whose transient failures the loop retries rather than taking the unit down. The journal is quiet without going dark: the wait is logged once on entry (worded so nobody reads it as an error), then on a 10-min heartbeat with elapsed minutes; `410` still exits non-zero on purpose (delivery consumed but no local env is a real crash window). Console banner every 5 min instead of every cycle, single accented spelling, plus the reassurance the rehearsal showed was missing („Ez a képernyő magától frissül"). **The load-bearing half is `TimeoutStartSec=infinity` in the unit** — a `Type=oneshot` ExecStart is killed at `DefaultTimeoutStartSec` (90s), so without it systemd would kill the new wait and `Restart=on-failure` would silently reinstate the exact spam this removes, *after appearing to work for the first three polls*. Verified behaviourally in a container against a stub hub answering 204 five times then delivering: **one log line plus one heartbeat, zero exits between polls**, then a clean fall-through to the direct install and `exit 0` — where the old design produced 5 unit invocations and 5 `Failed to start` console lines. ISOs rebuilt at v1.21.0 and the shipped payload verified byte-identical to the committed source. Evidence `bootstrap-journal.txt`; ship commit `bcdb042` |
| R-5 | Hub: offsite storage visibility — RESTIC box aggregate (v0.64.0) + PBS DR datastore (v0.65.0), each with fill + operator alert; Offsite tabs + dual dashboard gauges | M | **SHIPPED (hub v0.64.0 + v0.65.0 + tenantsync v1.2.0, 2026-07-17)** | Restic: `monitor.OffsiteBoxChecker` (throttled Hetzner GET; fill 80/90% + oversub 2.0×; scope `"pool-box"`). PBS DR (v0.65.0, Option A): read-only `usage` op on the ep0 `felhom-tenantsync` forced command (`df`, no mutation) → `monitor.PBSDRBoxChecker` (fill 80/90%, states ok/unavailable/degraded, scope `"pbsdr-box"`); `/offsite` Restic/PBS-DR tabs + two gauges; hub deploy ⟂ ep0 update (graceful "n/a"). **Both data pipelines PROVEN-LIVE** (restic `0.2% full`; PBS `19.1% full, 7.1 GB of 37.2 GB` after ep0 → v1.2.0). Thresholds pending Viktor (named keys; separate restic/PBS pairs). OPEN: UI render unit-only (password-gated), alert emails unit + red-proof only (both nominal). See hub v0.64.0 + v0.65.0 REPORTs |
| R-6 | **Spike: LAN service discovery from the guest** — SSDP multicast (UDP 1900, DLNA), WSD (Windows discovery), mDNS; host-network vs macvlan; is the customer LXC LAN-bridged in appliance deployments? | M | **spiked (2026-07-18)** | **VERDICT: appliance guest IS LAN-bridged (own DHCP lease on the household /24); multicast discovery works ONLY in the guest netns — guest-direct or Docker `--network host` (SSDP/mDNS/WSD all PASS both ways); the default docker bridge is categorically DEAF to LAN multicast (WSD/mDNS RX FAIL, unicast-publish PASS). Real samba+wsdd on host-net → Windows 11 ProbeMatch + FELHOM-SPIKE renders in Explorer + 445 + authenticated SMB round-trip all PASS; real SSDP `MediaServer:1` advert reaches both LAN clients. → R-7 SMB stack MUST be host-network LAN-bound; R-8 Jellyfin-DLNA plausible if host-network. Caveat: `vmbr0 multicast_snooping=1` worked only because the household router is a live querier — customer LANs w/ snooping+no-querier, and Peti's BYO bridge, are UNTESTED gaps.** **S4b (human leg, the sharpest finding): wsdd makes the box VISIBLE but the Explorer double-click FAILS `0x80070035` — WSD gives no name resolution; the flat `\\FELHOM-SPIKE` resolved by no path. Adding `nmbd` (NetBIOS) fixed it live (flat name resolves + mounts). → R-7 needs smbd+wsdd+nmbd (+avahi/.local for modern clients), not wsdd alone.** Doc: `audits/SPIKE-lan-discovery-2026-07-18.md`. |
| R-7 | SMB **server** share — LAN file sharing as an embedded controller feature („Megosztás") | L | **SHIPPED slice 1 (controller v0.144.0 + `felhom-samba:1.0.0`, 2026-07-18)** | Flips map row *Files from Windows Explorer / Mac Finder (SMB server)* MISSING → **IMPLEMENTED**. Delivered as the FOURTH protected infra stack (host-net, smbd+**nmbd**+wsdd, `bind interfaces only = lo eth0`), NOT a catalog app. „Megosztás" → „Hálózati megosztás": enable + ONE household password (STDIN→smbpasswd, never persisted — only `user_set`), shares table, create flow (new folder under `<storage>/shares/` or an existing folder via a guarded picker). Picker guard = absolute → EvalSymlinks → containment in a registered LIVE root → deny-listed system subtree (derived from `ProtectedHDDPaths`, provably a subset) → is-a-dir, with UNIFORM refusals (no filesystem oracle). Per-share read-only enforced in smb.conf AND as a `:ro` bind. `force user = felhom` → SMB writes land as uid 1000. Live-validated on demo through the real endpoints + a Win11 workstation (445 True, NetBIOS `FELHOM` resolves, write/read byte-compare PASS, **write to a read-only share REFUSED**, WSD ProbeMatch). **Explorer leg PASSED 2026-07-18 (Viktor):** both shares open from the Network view; an Explorer save into the writable share landed as uid 1000 and a write into the read-only share was refused, folder untouched. Slice 1 is fully PROVEN-LIVE. Docs: `controller/sharing.md`; ship report `felhom-controller/REPORT.md`. **Slice 2 remainder:** avahi/`.local` for modern+Apple clients, curated app-folder presets (one-click paperless `consume/`), per-share users + guest access, recycle bin. **Backup gap split out → R-7b.** |
| R-7b | **Share backup EXECUTION** — put share data into the live tier-2 + offsite runs (the design fork reported by R-7 slice 1) | M | **SHIPPED (controller v0.145.0, 2026-07-18)** | **Viktor's ruling: Model B′ — a SIBLING shares source.** New, additive job/leg code reusing the proven primitives (tier-2 mirror seam, restic wrappers, soft-quota/enlargement gate, status recorders) while leaving **every per-app engine path byte-identical** — NOT a synthetic recovery unit (breaks on multi-drive shares, wraps 1 KB of JSON in dump machinery) and NOT engine-loop surgery. The B′ invariant is enforced by test in both tiers, red-proofed. Tier 2 → `RunSharesTier2` (legs grouped by SOURCE drive → `backups/secondary/_shares/<driveKey>/<share>`, payload at `_payload/`, layout marker LAST). Tier 3 → `runOffboxSharesLeg`: ONE extra `restic backup --tag felhom-offbox --tag _shares` placed after the app loop and BEFORE retention, so `forget --group-by host,tags` covers the new group with no flag change; a quota-blocked push degrades to the **manifest only, never to nothing**. Restore → „Megosztások" on `/backups/restore`: scratch, then a missing-only merge whose every destination is PREFIX-ASSERTED against live storage roots, definitions merged existing-wins, then `ReconcileSamba`, then the credential. The **payload** (`_shares-manifest.json` + a best-effort secret-bearing `passdb.tar`) is what makes DR return files + configuration + password rather than loose bytes. **Fold-in: samba joins the liveness set** — `EffectiveProtected` adds the CONTAINER `felhom-samba` exactly while sharing is on. **FULLY PROVEN-LIVE on demo (2026-07-18), all four legs.** (1) tier-2: real `/api/backup/tier2` trigger → `_shares` tree + marker + payload on the cross-drive target, mirrored file md5-identical, payload 0600 preserved. (2) offsite: Viktor's manual run 12:18:16Z → snapshot **`e0b9d723`** (tags `felhom-offbox,_shares`) with the payload dir + both share folders; a second run via the „Távoli mentés" button → **`4e2b15ec`**, containing `_shares-manifest.json` (418 B) AND `passdb.tar` (855 040 B), both 0600, share files with uid 1000 preserved. (3) restore round-trip: probe file + the `dokumentumok` DEFINITION deleted via the real endpoints, then „Megosztások" restore + place → `1 file(s), 1 definition(s) re-added, 1 kept, 0 refused, credential=true`; probe back md5-identical, the two pre-existing files NOT overwritten (missing-only proven on live data), definition back with its ORIGINAL flags and created_at, `smb.conf` re-rendered, `filmek` untouched. (4) liveness: samba stopped → `health_critical` pushed and hub-accepted (200) → self-healed. Remaining human leg: SMB positive auth with the real household password (never persisted by design). **Correction:** an earlier revision of this row and of the ship REPORT wrongly claimed the demo box had no offsite target — the verification read a guessed settings key (`offbox_target`) instead of the real one (`offbox`); root cause dissected in REPORT §7b. Findings: the reserved-name assumption was FALSE (`nbNameRe` accepted „_shares" as a share name — now refused); the alert/e-mail pipeline needed NO change and adds no new event type. Docs: `controller/sharing.md`; ship report `felhom-controller/REPORT.md`. |
| R-8 | DLNA (**gate input now exists — R-6 spiked 2026-07-18: SSDP reaches LAN clients from host-net**): validate Jellyfin's built-in DLNA server first; only add minidlna to the catalog if Jellyfin-DLNA fails | S | idea (unblocked) | Don't add catalog weight before proving the cheap path. **R-6 confirmed the cheap path is physically viable — Jellyfin DLNA must run host-network (same multicast constraint as R-7)** |
| R-9 | Uninstaller trio (from 07-15 Peti session): cluster-aware `felhom_guests` guard (node-local `pct list` deletes cluster-wide pveum objects); saferemove detection + time estimate + opt-in `--quick-remove` (never mutate `storage.cfg`); smarter `restore_storage` default for BYO clusters (shared storage, not local-lvm) | M | idea | Second item's rejected alternative (temp-disable-and-restore) stays rejected — crash window silently downgrades cluster wipe policy |
| R-10 | T-6E-1: DB-dump dir-fsync asymmetry (LOW, confirmed in 6E) | XS | idea | One-line hardening; batch with the next controller task |
| R-11 | Tester-facing one-pager: what the box does, known limitations, how to report | S | **RULED 2026-07-21 (channel); doc is the architect's** | Pairs with R-3. ****operator ruling 2026-07-21**: the reporting channel is DIRECT MESSENGER to the operator** — no group, no tracker, no form. Rationale: the expected tester count at first invite is ~0–few, and a channel nobody is on is worse than a channel everyone already uses. **The one-pager and the tester agreement are the ARCHITECT's deliverable, not CC's** — CC does not write them; it may be asked to fact-check claims in them against the code. Revisit the channel only if tester count makes direct messages unmanageable |
| R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. **Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17):** the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above |
| R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/<id>` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. |
| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |
| R-34 | **Backup data lifecycle management.** An "inactive backups" section on „Távoli mentés": apps that have snapshots but no active backup — **disabled OR uninstalled** — listed with name / size / last snapshot / restorable, plus an explicit **double-confirmed per-app delete** via `restic forget --tag` + nightly prune. | M | idea | **RULING: the offsite toggle NEVER offers deletion — policy and destruction stay decoupled.** Turning backups off must never be a data-destroying act, and deletion must never hide behind a toggle. Origin: 2026-07-18 rehearsal. Pairs with R-32 (that one is the operator's view of dead bytes; this one is the customer's) |
| R-35 | **Config-apply should not end the customer's session.** The offsite config push bumped `config_version` 10→11 at 16:54:58 and the controller self-restarted (container `StartedAt` 16:54:59Z, back up 16:55:02); in-memory sessions died with it and **customer zero was force-logged-out mid-flow**. | S | idea | Direction: **hot-apply the offbox target** (no restart for a config the running process can adopt), or **persist sessions** across restart. The restart itself is by design — the collateral is not. Evidence `controller-log-full.txt` |
| R-36 | **Post-RESET re-enroll leaves offsite "enabled but unprovisioned" — silently.** The hub knows the state and says nothing on the customer page. | S | **SHIPPED (hub v0.67.0, 2026-07-18)** | Both halves delivered. **(1) The warning:** the customer page now names the state and the fix — enabled-but-unprovisioned raises an amber banner saying provisioning is *Save*-triggered (press Save once, then verify), reusing the exact `enabled && type == ""` predicate the offsite re-issue handler already refuses on. **(2) The related sub-item, also done:** the self-bind link is now **auto-minted at customer creation AND at RESET completion**, so the console banner's „e-mailben kapott link" is already true instead of true-once-the-operator-remembers. Extracting the shared `mintAndSendSelfBindLink` core keeps the button and the auto-mint callers on the same F1/F2 honesty rules, and the auto-mint never fails the operation it rides on. **Gap found and closed while wiring it:** `PurgeCustomerResetDBState` does NOT clear `selfbind_tokens`, so a link minted BEFORE a reset would have stayed live across it — the skip paths now clear stale tokens, giving the invariant "after auto-mint, the only live link is one we just issued, or none". Tests assert the banner is ABSENT in all three nominal cases too; red-proofed. — Original analysis: Source-cited behaviour, confirmed live in the rehearsal: **provisioning is Save-triggered** (`configs.go` `applyOffsite`) — which also answers S6's open question — and the re-enroll auto-re-issue **correctly** skips unprovisioned targets (`handler.go`). So nothing is broken; the gap is that nobody is told. Direction: flash it on the customer page. **Interim: an R-3 step.** **Related sub-item:** auto-mint the **self-bind link on customer create/RESET**, so the console banner's „e-mailben kapott link" is always already true instead of true-once-the-operator-remembers |
| R-27c | **Customer self-bind, slice 2 — console-passphrase bind.** Viktor's direction: bind using a passphrase shown on the box console, alongside (not instead of) the emailed capability link. | M | idea | **Security constraints from the session ruling, all load-bearing:** passphrase **issued at customer creation**; the global-lookup endpoint must be **spray-hardened** — per-appliance **and** per-IP caps, constant-time comparison, a **single generic failure** (no oracle), alerting on abuse; an **accent-free wordlist** (console keymaps are not Hungarian); the **web capability-link path is RETAINED**; **claim-by-email is RETAINED** as the delivery-channel proof. **Also under this item:** the self-bind email gains the **public universal-ISO download link + two-line instructions** (the DIY case). **Secret-bearing per-customer ISOs are ruled OUT.** Sibling of R-27b (second-box flow) — different axis, both build on the same `/bind/` page |
| R-50 | **[P2-HIGH] Island-bridge control plane — make controller↔agent independent of the LAN.** The agent's `localapi` binds a **LAN literal** (`listen_addr`) and the guest dials that same literal from `bootstrap.json`. Move both onto a **host-internal bridge with a fixed, private address** that no router, DHCP lease, or site move can invalidate, so the control plane is location-independent and golden-image-stable. | M | **SHIPPED 2026-07-25 — fleet-migrated (agent v0.96.0 + host-install v1.19.0)** | **DONE (2026-07-25). Both demo boxes migrated to the island, apps served throughout (0 container restarts); island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), hub reports 0.96.0.** Fresh appliance installs are born on the island (host-install v1.19.0 default; agent auto-attaches `net1` — A4 proven live). Migration procedure: `runbooks/RUNBOOK-island-migration.md`. **One-host fleet only; the 2-node cluster (Peti) is R-74 (SDN/bridge parity), a separate supervised runbook.** ── Spike/impl detail (superseded, kept for provenance): **GO (2026-07-25 PM, `audits/SPIKE-island-bridge-2026-07-25.md`): validated end-to-end on a real nested-PVE drill (`drill-r50`, qm300 on the t740, installed via the v1.25.0 ISO through the actual day-0).** All probes PASS: `vmbr9` portless island bridge (vmbr0 untouched); guest island NIC `eth1 169.254.253.2/30` hot-added (LAN leg undisturbed); **F1 replay = the money shot** — LAN moved `.176→.200` with `listen_addr` on the island → agent stays `active`, control plane HTTP 200; the LAN-literal contrast reproduced the 2026-07-20 bug verbatim (`localapi: bind 192.168.0.176:8443` → daemon exit 1); **dnsmasq trap CONFIRMED LIVE and its `lan_resolver.host_ip=LAN IP` fix PROVEN**; **pin address-independent** (served leaf SHA-256 unchanged, HTTP 200 over the island — no cert re-issue); **survival matrix** (agent restart / guest reboot / host COLD reboot) all return the control plane on the island with zero intervention. **Method caveat:** probes drove the runtime chain via manual config edits — the *provisioning* path (host-install writing the island config + golden-bake bootstrap template) is the impl task, now de-risked. **Next: write the Phase A/B/C production spec (drill-proven → demo → Peti SDN).** — Prior (superseded): **2026-07-25 spike: the drill environment (qm300 / `demo-vm-felhom-2f4b00`) is GONE** (`qm list` empty; only guest 9201 remains), so per the hard drill-only rule the empirical probes (bridge create / NIC hot-add / island bind / **F1 replay** / survival) were NOT run — production was left untouched, **no live GO/NO-GO**. The **source-grounded** half IS done: recommend **link-local `169.254.253.1/30`↔`.2/30`** (structurally uncollidable vs LAN); the F1 literal is baked in TWO places to move atomically (`config.go:229` + `felhom-host-install.sh:2226` for the bind, `provision/backhalf.go:129`/`bootstrap.json` for the guest dial); **NEW dnsmasq trap CONFIRMED in source** — `LANResolverConfig.WithDefaults` (`config.go:208–210`) derives the DNS listen-addr from `listen_addr`, so moving the bind to the island silently kills LAN DNS → the spec MUST set `lan_resolver.host_ip = LAN IP` explicitly; pin is leaf-DER-based (address-independent → no cert re-issue expected); provisioning inventory + cluster-parity (SDN on Peti's 2 nodes) + an implementation skeleton recorded. **Empirical attempt 2 (2026-07-25 PM, on the t740 per the operator's drill-host ruling): STILL BLOCKED — there is NO drill VM on the t740** (`qm list` empty; it is a real demo node running only its own guest 9201). The only drill artifact is a stale golden-bake `drill.qcow2` on the production DooPlex node (off, nested guest purged). The ruling ("drill+build VMs on the HP from now on") is forward-looking and not yet realized. **Operator decision: they will provision a nested-PVE drill VM on the t740 (agent + nested guest); this spike re-runs then.** GO/NO-GO PENDING. **Remaining: a provisioned t740 drill VM to validate probes P2–P7 before the production spec.** Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F1), where this failed for real. The demo box moved to a remote site, DHCP handed it `.147` instead of `.162`, and the agent then **could not start at all** — `bind: cannot assign requested address`, systemd gave up after 4 retries — taking storage, PBS backup, quiesce, restore-test and DR down for as long as nobody noticed. Mitigated for that window by pinning `vmbr0` static back to `.162`; that is a **window mitigation, not the fix** — it still depends on the site's subnet being `192.168.0.0/24` and free at that address. **Spike-first is mandatory:** validate end-to-end on the drill environment (agent bind + guest dial + TLS SAN/pin + reinstall/golden survival + the bootstrap-config migration for already-deployed boxes) BEFORE any production spec. **Pin fact (verified 2026-07-20, `agentapi/client.go` L105-129 — supersedes the earlier "the SAN set must cover the new address" note in this entry, which was wrong):** the controller-to-agent leg sets `InsecureSkipVerify: true` and replaces chain verification with a custom `VerifyPeerCertificate` that does a raw **SHA-256 match on the leaf DER** against the bootstrap fingerprint. Hostname/SAN therefore never enters verification on this leg, so moving the agent listen address most likely needs **no cert re-issuance** - only the endpoint the guest dials. The spike must still confirm this empirically rather than trust the read. Flips a future "box survives a site/network change" map row |
| R-50b | **[P2] A root-owned privileged host artifact is delivered unversioned from `main` — "which wrapper is on this host?" is unanswerable.** `configs/felhom-pbs-apply` installs to `/usr/local/sbin/felhom-pbs-apply` (0755 root:root) and is the pinned sudoers vector for `create\|reconcile\|grant` against `/etc/pve/priv/storage`. It is fetched by `felhom-host-install.sh:1914` via `fetch_raw`, which hits `raw/branch/main/<path>` — **no tag, no pin, no checksum, and no record in the Day-0 artifact manifest**, unlike the agent binary (sha256-vouched) and the golden image. Three consequences: (1) two hosts installed a week apart can carry different privileged wrapper code while both reporting the same agent version; (2) a host hotfixed in place (felhom-pve, 2026-07-18) is indistinguishable from one that fetched the same content — the fleet has no inventory of it; (3) an accidental push to `main` reaches the next install of every host with no review gate between commit and root-owned deployment. | S–M | **(a) SHIPPED 2026-07-21; (b)/(c) open** | **Surfaced 2026-07-21 while stopping the R-39 v0.90.1 publish** (`felhom-controller/REPORT.md` §5): the publish was cancelled precisely because the version number would have claimed to carry a fix that in fact rides this unversioned channel. Candidate shapes, in increasing cost: (a) record the wrapper's sha256 in the Day-0 artifact manifest beside the agent binary and have the agent report the installed file's hash, so drift is at least *visible*; (b) `fetch_raw` takes a pinned ref (tag or commit) supplied by the manifest rather than `main`; (c) the wrapper becomes a published generic-registry artifact with the same gate ladder as the agent binary. **(a) is the cheap honest first step and would have caught this class already.** Pairs with R-39 (whose remaining fleet half is specced separately)  **(a) SHIPPED 2026-07-21 — hub v0.68.0 + agent v0.91.2.** `ArtifactManifest.WrapperSHA256` + an operator field; agents report the installed wrapper's sha256 each cycle and the host page surfaces a mismatch. **An unknown on EITHER side reads as quiet, never as drift** — lighting every host amber on rollout day is how a warning becomes background noise. Live confirmation of exactly the problem: felhom-pve's July-18 in-place hotfix hashed `2888f2ea…`, matching **no commit anyone could name**; it now reports `104db0a4…` against a vouchable manifest value. **(b)/(c) REMAIN OPEN:** the wrapper is still fetched unversioned from `raw/branch/main` — this makes drift *visible*, it does not fix the channel. Also recorded: the 0440 sudoers file is not agent-readable, so its drift stays invisible. |
| R-51 | **Dead-primary alerting — a multi-container app whose MAIN container is dead must alert.** Aggregation currently classifies such a stack `unhealthy`, and `IsDownState` deliberately excludes `unhealthy`, so nothing fires. | S | **SHIPPED 2026-07-21 — controller v0.156.0** | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F4). Observed live: `immich-server` was `Exited` for **18 h** with the app 100 % unreachable, and the box produced **no** dead-app banner and **no** `app_start_failed` hub event — while single-container Calibre-Web, down for the same reason, alerted correctly within 90 s. **Constraint (load-bearing): do NOT simply fold `unhealthy` into down.** That exclusion is deliberate (`stacks/manager.go` fix-3, `downstate_test.go`) and reverting it reintroduces the flapping it was added to stop. Direction: distinguish *member-container-exited* from *healthcheck-failing* in the aggregation, and treat a dead primary as down | **SHIPPED 2026-07-21 (controller v0.156.0).** **The diagnosis in this row was WRONG at the source and is corrected here:** aggregation did NOT classify the stack `unhealthy`. `aggregateState`'s final branch returned `StateRunning` for any mix of running and stopped members — the comment said "report as running (partial)" — so the stack read as RUNNING and `IsDownState` had nothing to fire on. The `unhealthy` exclusion was never involved, and the constraint it protects was therefore never in tension with the fix. New `StateDegraded`: a DOWN member whose docker restart policy is `always`/`unless-stopped` means docker was supposed to be keeping it up, so the stack is degraded (a down state, alerting through the EXISTING banner + `app_start_failed` path, unchanged); `no`/`on-failure` is a finished one-shot init/migrate container and stays benign. An UNREADABLE policy counts as supervised — fail-CLOSED, deliberately the opposite of `IsDownState`'s fail-open, because there the *state* is ambiguous while here a member is known dead and only the excuse is missing (P2 census 2026-07-21: all 53 catalog templates / 78 services are `unless-stopped`, zero one-shot containers exist today). `IsDownState` gained `degraded` and NOTHING else — the `unhealthy`/`restarting`/`paused`/`unknown` exclusions are byte-identical and `downstate_test.go` is untouched and green. Red-proof: the mix branch reverted to `return StateRunning` makes the immich fixture and both production-path tests fail with `"running"`. Live leg (STOP-1) pending. Evidence: `felhom-controller/REPORT.md` (2026-07-21)  **PROVEN LIVE 2026-07-21 (STOP-1, operator-present).** On guest 9201, `docker stop immich-server` at **12:50:40 CEST** (policy `unless-stopped`, three helpers left running — the exact F4 shape). **12:50:53 — 13 seconds later — the stack read `degraded`** where it read `running` for 18 hours on 2026-07-20. `10:51:11Z` **exactly ONE** `app_start_failed (warn) — Telepített alkalmazás nem fut: Immich` (single-fire verified by count, not by eye). Dashboard rendered the banner *Telepített alkalmazás nem fut: Immich (degraded)* plus the new „Részlegesen leállt" state label. `docker start` at 12:51:37 → `running` by 12:51:52 and the banner **self-cleared** (state-based, as designed). Note the banner appends the raw state in English — `(degraded)` — which is pre-existing behaviour, not introduced here, but now more visible |
| R-52 | **Boot desired-state reconciliation — a `deployed: true` app should be running after boot.** The controller *reports* deployed-but-stopped apps (30 s `deadapp-check`) but never starts them, so an app that misses its boot start stays down until a human notices. | M | **SHIPPED 2026-07-21 — controller v0.156.0** | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F5). Observed live: the pre-transport shutdown left `immich-server` and `calibre-web` `Exited`; **10 sibling containers came back and those two did not**, and they were still down ~18 h later. **Includes root-causing why `restart: unless-stopped` did not resurrect them** — both were stopped ~25 s before power-off, so Docker most likely recorded them as user-stopped; that hypothesis is untested because the guest journal is volatile and the controller's own logs were rotated by the container recreate. Direction: a bounded start-once reconciliation (N attempts, reusing the existing boot grace), never a restart loop. Pairs with R-51 — that one is the *alarm*, this one is the *recovery* | **SHIPPED 2026-07-21 (controller v0.156.0), `internal/bootrecon`.** One bounded start-once sweep at controller startup: every deployed, non-protected, not-mid-deploy stack that still HAS containers and is down gets `StartStack`, at most **2 attempts 30 s apart**, then it stops and the alarm owns the problem. Never a restart loop. The whole sweep (5 s settle + one 30 s gap) fits inside the existing 90 s `deadAppBootGrace`, so a successful recovery never alerts and a failed one alerts honestly — asserted by a test rather than left to a comment. **The safety argument is the container gate:** the UI's Stop is `compose down`, which REMOVES the containers, while an interrupted boot leaves them behind as `Exited` — so "deployed, has containers, and they are down" is exactly the boot-orphan signature, and a zero-container stack is never touched. Red-proof: dropping that gate makes the user-stopped app get started, which is the one thing this must never do. **P1 (the `unless-stopped` root-cause probe) is deliberately NOT what this shipped on** — the reconciliation is correct whether or not Docker recorded those two containers as user-stopped, and the probe is recorded as an open question rather than a blocker. Live leg (STOP-1) pending. Evidence: `felhom-controller/REPORT.md` (2026-07-21)  **PROVEN LIVE 2026-07-21 (STOP-1, operator-present), and it answered P1 for free.** Fixture on 9201: `docker stop` on bookstack (2 containers, left in place) + calibre-web, and a UI Stop on immich (`compose down` → 0 containers); then `pct reboot 9201`. Result: `10:53:22Z [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [bookstack] — up to 2 attempt(s)` → `10:53:28Z attempt 1/2: started "bookstack" (took 6.0s)` → `complete: 1 app(s) recovered in 1 attempt(s)`, and **ZERO `app_start_failed`** — a successful recovery inside the boot grace is silent, exactly as designed. **P1 IS NOW ANSWERED, and the F5 hypothesis is CONFIRMED:** bookstack carries `restart=unless-stopped`, the Docker daemon came up at ~10:53:15Z, and the container's `StartedAt` is **`10:53:28.05Z` — the exact moment `bootrecon`'s `StartStack` returned**. Docker's own restart policy did NOT resurrect it; a container stopped before shutdown is recorded user-stopped and stays down. Only R-52 brought it back. **BUT the end-to-end "a deliberate Stop survives a reboot" property is NOT true today, for a reason outside R-52 — see R-55.** R-52's own gate is correct and was observed to be: immich (0 containers) was never a candidate |
| R-54 | **[P2-HIGH] The guest's DHCP client is unsupervised — its death takes the box off the internet 1-2 hours later, invisibly.** ifupdown starts `dhclient` once at guest boot and nothing restarts it. | S-M | **SHIPPED 2026-07-21 — agent v0.92.1** | Origin: `audits/INCIDENT-guest-dhclient-killed-2026-07-20.md` §5 "OPEN RISK" — this row closes it. On 2026-07-20 a cleanup step killed guest 9201's dhclient (visible in the HOST's pid namespace; §4's `/proc/<pid>/cgroup` rule exists because of it). **The guest then kept working for another ~80 minutes on its unexpired lease**; only at expiry did the address and default route vanish, taking the Cloudflare tunnel, hub reports, catalog sync and the controller→agent channel with them — 1h15m outage, and every observable signal said healthy for the first 80 minutes. **The design consequence: liveness of the DHCP client is itself a probe.** `internal/guestnet` flags a DHCP guest unhealthy on `pgrep -x dhclient` alone, while the lease is still live — waiting for the IP to disappear is waiting out precisely that silent window (red-proof: reverting to IP-presence-only makes the July-20 fixture report **healthy** with zero heals). Four fixed-shape `pct exec` probes (address / default route / `/etc/network/interfaces` mode / client liveness, parsers pinned to output captured live from 9201), the incident's restored invocation as the heal, verbatim, and dampers throughout: two CONSECUTIVE bad probes, ≥10 min between heals per guest, ≤3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on a static guest (dhclient must never fight a static config — reported loudly and left to **R-50**, which is where option 2 of the incident's three choices belongs), on an unknown mode, on an unprobeable guest, or on an ownership-unproven guest list (source is `ListLXC` ∩ the felhom pool, audit A1). Host-tier by necessity: a guest with no default route cannot repair its own default route. **A live finding during deployment:** the first sweep on felhom-pve logged `dhclient liveness probe failed: sudo: a password is required` and reported `state=unknown` — fail-safe, but blind. TASK-D assumed no sudoers change was needed; three of the four probes had no grant. `FELHOM_GUESTNET` + four `guestnet-*` capability rows shipped in v0.92.1 (v0.92.0 superseded, do not vouch). **Healthy cycle PROVEN LIVE 2026-07-21** on felhom-pve: caps `68/68 ok, degraded=0` and `level=DEBUG guestnet: guest network healthy vmid=9201 mode=dhcp has_route=true dhclient_alive=true`. The HEAL leg (STOP-2, a deliberate replay of the incident) is operator-present and pending. Note the guest and the host still differ (host static since the F1 mitigation, guest DHCP) — choosing one for both remains **R-50**'s call, not this row's. Evidence: `felhom-agent/REPORT.md` (2026-07-21)  **PROVEN LIVE 2026-07-21 (STOP-2, operator-present) — a deliberate replay of the incident.** `kill -9` on guest 9201's dhclient at **12:43:18 CEST** (pid 336708, `cgroup=0::/lxc/9201/ns/.lxc` verified BEFORE the kill — the incident's own §4 rule, applied). **12:44:15 — 57 seconds later — detected on process liveness alone**, with the address and default route still perfectly present and 83 minutes of lease remaining: `unhealthy (first bad probe — not acting yet) bad_probes=1 required=2`. **12:45:15** second consecutive bad probe → `healing`, `cmd="pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0"` (the incident's invocation, logged verbatim before it ran) → **12:45:18 `guest network healed` ip=192.168.0.104 has_route=true dhclient_alive=true heals_last_hour=1**. Healed **120 s** after the kill — ~80 minutes before the outage would have begun. **The strongest evidence is what did not happen:** `cloudflared` stayed `Up 29 hours`, DNS resolved, the hub answered 302 in 0.16 s and the public URL 302 in 0.25 s. The 2026-07-20 outage was prevented, not merely detected. Exactly ONE heal, zero re-heals. **The damper then proved itself against a REAL transient, unplanned:** the STOP-1 guest reboot at 12:53:16 produced `unhealthy … no IPv4 address on eth0 bad_probes=1` — **no action** — then `guest network recovered` on the next cycle. A single blip really is not a diagnosis, and a booting guest is not injected with a dhclient. **Negative leg run in full:** 12:45:18 → 13:16:15 = **30 healthy Debug cycles, 0 heals, 0 WARN, 0 ERROR**. **Confirmed HUB-SIDE (read-only query of the hub store, after STOP-3):** the `guest_net` stanza arrives complete, and the report history traces the whole session — `10:30:27Z` agent 0.92.0 `state=unknown` (the sudoers-blind window, reported honestly rather than as a false healthy — independent proof that superseding 0.92.0 was right), then `10:48:12Z` 0.92.1 `healthy` carrying `last_heal_at=10:45:12Z heals_last_hour=1`. **Known limitation:** the damping state is IN-MEMORY, so an agent restart resets the heal counters — the ≥10 min / ≤3 per hour ceilings hold within one agent lifetime, not across restarts (a restart does re-arm the 3-minute settle window, so the practical exposure is small). Do not quote the ceiling as a hard guarantee |
| R-55 | **[P2] A customer's deliberate Stop does NOT survive a guest reboot for any drive-backed app** — the boot bind gate recreated and started it unconditionally. | S | **SHIPPED 2026-07-21 — controller v0.157.0** | Surfaced by STOP-1's R-52 leg, which was designed to prove the opposite property and instead proved it is not true end to end. `internal/web/intermediary.go`'s boot gate (`[gate] boot <id>: live bind confirmed — recreating drive-backed app <x>`) selects on `cfg.Deployed && HDD_PATH` alone and then runs `StopStack` + **`StartStack`** on every match; the stack's `state` is passed in for the log line and is never consulted. So after `pct reboot 9201`, calibre-web and immich came back **running** at `10:53:18Z` even though immich had been stopped from the UI seconds earlier (`compose down`, 0 containers) — while bookstack, which is NOT drive-backed, correctly fell through to R-52. **R-52 is not implicated:** its own gate behaved exactly as specified (immich, with zero containers, was never a candidate), and this behaviour predates it. Two consequences worth ruling on: (a) a customer who stops a drive-backed app finds it running again after any reboot, which is the same class of dishonesty as offering a control that cannot work; (b) **R-52's practical scope on a real box is narrower than the brief assumed** — the gate reaches every drive-backed app first, so the boot-orphan sweep only ever sees the rest. Likely shape: the gate should recreate only apps that were RUNNING (or `deploying`) at shutdown, which is the same running/stopped distinction R-52 already draws, rather than every deployed app. Needs a ruling, not a drive-by change — the gate exists to fix a real bind-ordering bug and must keep doing that | **SHIPPED 2026-07-21 — controller v0.157.0, deployed to 9201.** The ruling that shaped it: an unconditional recreate could drown the node when apps were deliberately stopped to free resources for others — so honouring Stop is not only about honesty, it is about not fighting the customer's own capacity management. `shouldRecreateOnBoot` now additionally requires `len(Stack.Containers) > 0` — **R-52's `existing-Exited vs absent` predicate (`bootrecon.isBootOrphan`), translated to this gate**: a UI Stop is `compose down` and REMOVES the containers, while a guest that went down under a running app leaves them behind, and docker's records survive the reboot. **Spike-lite outcome: a truthful running-at-shutdown source already existed and nothing was invented** — `AppConfig` carries no run-state (`Deployed` is a deploy-lifecycle flag that stays true across a Stop), but container presence is durable, already collected via `docker ps -a`, and already trusted by R-52. **Container STATE is still deliberately NOT a filter** — that half of the original design is load-bearing (a `State != stopped` filter misses an app docker has not restarted yet, or one stuck `Exited` on a create-time bind failure with `RestartCount=0`, which is the bug the boot-id path exists to fix). `TestShouldRecreateOnBoot` now pins both axes at once; they pull in opposite directions, which is the difficulty of this gate. **Ordering trap handled:** the evidence is sampled into the snapshot BEFORE any recreate, because `recreate`'s own `StopStack` destroys it. **No regression to the drive-absent gate:** apps it stopped are also at zero containers and are now skipped here, correctly — they are restored by `ReconcileDriveGates`' `Return` branch from `StoragePath.StoppedStacks` on the same loop tick. Honoured Stops are counted and logged separately (INFO `left stopped …`) so an intended outcome never fires a WARN. **Red-proof:** dropping `hasContainers` makes `TestRecreateDriveBackedApps_HonoursCustomerStop` fail with `recreated=[romm immich]` — the live defect by name. **NO new privileged verb** — `StopStack`/`StartStack` are plain `docker compose` execs in the controller's own container; the boot path's only agent calls are read-only. **PROVEN LIVE 2026-07-21 (operator-present, TASK-F Part 3).** Fixture on 9201: immich (drive-backed, 4 containers) STOPPED through the real UI endpoint `POST /api/stacks/immich/stop` → `compose down` → **0 containers**; calibre-web (drive-backed) and bookstack (NOT drive-backed) left running. Then `pct reboot 9201` at 14:29:14Z. Result, from the gate's own log:

> `14:29:28 [gate] boot 1784525102-11906045: live bind confirmed — recreating drive-backed app calibre-web (state=stopped) onto /mnt/felhom-drives/hdd_1`
> `14:29:29 [gate] boot 1784525102-11906045: 1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose`

**immich is absent from the recreate list and came back STOPPED (0 containers)** — on 2026-07-21 before this fix, the identical fixture brought it back RUNNING. calibre-web recreated, bookstack back, `[bootrecon] no boot-orphaned apps` (consistent — nothing was left orphaned for it to adopt), **ZERO alerts**, whole convergence ~15 s from reboot to steady state. The `left stopped` INFO line fired in production for the first time, so the honoured path is observable rather than silent |
| R-56 | **[P3] Apps do not say how technical they are, so a beginner can be ambushed by a config-heavy one.** The catalog presents every app as equally approachable — one Telepítés button, the same Hungarian copy — but they are not. Glance needs a hand-written `glance.yml` before it does anything; some apps need a reverse-proxy or API concept to configure; others genuinely are install-and-use. A tester who picks the wrong first app concludes the PRODUCT is broken, not that they picked an advanced app. | S | **idea (filed 2026-07-21)** | Origin: TASK-E Part 3 — **filed, deliberately not implemented**. Shape: a `difficulty:` field in `.felhom.yml` (`kezdő` / `haladó` / `technikás`) surfaced as a catalog-card badge and repeated on the deploy screen. Cheap and incremental: one optional metadata field plus a badge, classifiable app-by-app with no migration — an app with no `difficulty:` simply shows no badge. **This is the constructive half of the glance ruling**: glance STAYS in the catalog (operator ruling 2026-07-21 — it is a legitimate app, not a broken one; its missing seeded `glance.yml` is a known pre-existing finding), and the honest fix is to LABEL it rather than hide it. Pairs with R-41: that gate proves an app CAN still deploy; this field tells a customer whether THEY should be the one deploying it. **Badge plumbing is ALREADY BUILT (controller v0.158.0)** — `web.MetaBadge` + the `meta_badge` template partial + the `lifecycleBadge` funcmap entry were written generic for exactly this: a `difficultyBadge` funcmap function returning the same `*MetaBadge`, plus a `difficulty:` field on `stacks.Metadata`, is the whole remaining job. No new markup, no new CSS. Re-sized accordingly |
| R-57 | **An app can be withdrawn from the catalog without orphaning the customers already running it** — `.felhom.yml` `lifecycle: available / hidden / abandoned`. | S | **SHIPPED 2026-07-21 — controller v0.158.0 (+ v0.158.1 fix), LIVE-PROVEN** | **Motivating case: plant-it.** Earlier the same day it was withdrawn by moving its directory to `retired/` — which un-offers the app but ALSO makes the controller's orphan detector see the template as GONE for anyone running it, flagging their working install `Elavult` and offering a Törlés button. Withdrawing an app must never take a working app away from a customer, so the directory move was replaced by metadata. **Operator requirements, verbatim (ruling 2026-07-21):** states `available` / `hidden` / `abandoned`; abandoned apps are NOT offered to new installs (no badge-but-installable middle state); deployed instances of hidden/abandoned apps keep full function; an abandoned app shows a permanent notice that *„Az alkalmazás fejlesztője felhagyott a fejlesztéssel. A telepített verzió továbbra is használható, de frissítések és biztonsági javítások már nem érkeznek hozzá."* **Design points that matter beyond this feature:** (a) the deploy gate is server-side and fail-CLOSED before any mutation — hiding a button is not a gate, and a stale link or direct POST must be refused; (b) an unknown lifecycle value fails OPEN (→ available + one WARN), deliberately opposite, because a typo or a state from a newer catalog must never pull a working app out of every customer's list — both read the same `EffectiveLifecycle`, so they cannot disagree; (c) lifecycle NEVER reaches orphan detection, red-proofed. **LIVE-PROVEN 2026-07-21 on 9201** through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders **0 times** on the Alkalmazások page while the control app renders 10; a direct `POST /api/stacks/plant-it/deploy` returns **HTTP 409 `{"ok":false,"error":"Ez az alkalmazás jelenleg nem telepíthető."}`**; the app page carries the notice and no Telepítés button. **v0.158.1 is a shipped-and-caught defect worth remembering:** the three predicates were declared with POINTER receivers, and html/template cannot call those on the non-addressable value the handler passes — every `/apps/<slug>` returned 500, for every app, while compiling cleanly with a fully green suite, because no test rendered `app_info`. A template method call is only checked when the template runs. Follow-on: R-56's difficulty badge reuses this plumbing |
| R-58 | **[P2] Assisted disk-picker install mode — the installer should let the operator CHOOSE the target disk instead of requiring the serial up front.** Today an install is either unattended (the answer file pins one `ID_SERIAL_SHORT`, which you can only know by first booting the machine) or match-nothing safety (aborts by design). That forces a two-boot dance for every new box: boot the safety ISO to read the serial, rebuild the ISO armed, boot again. | S–M | **idea — operator ruling 2026-07-21** | **Operator's argument, verbatim:** *"the installer should list the available storage devices (excluding the installation media) and let us select one, and continue."* **Shape:** a THIRD ISO mode alongside the two that exist — unattended-serial and match-nothing-safety. It enumerates candidate disks with **size / model / serial**, excludes the installation media itself, takes a selection plus a confirm, and proceeds. **Unattended+serial REMAINS the appliance/factory mode** — it is the right shape when the machine is provisioned in bulk and nobody is standing there; the picker is for the case where somebody is. **Slice 1 (cheap, same code surface, do this first):** improve the abort screen. On filter-no-match the installer currently just fails safe and says nothing useful — it should print the candidate table (size/model/serial) plus the one-line hint naming which serial to put in the profile. That alone collapses the two-boot dance from "boot, guess, go read docs, rebuild" to "boot, copy the serial off the screen, rebuild", and it is the same enumeration code the full picker needs. **Why it matters beyond convenience:** it is the BYO / reinstall flow — a customer's existing hardware, or a rebuild of a box whose disk layout nobody recorded, is exactly where the serial is unknown and a wrong guess is destructive. The current fail-safe is correct but mute. Origin: TASK-G, arming the HP install ISO — the serial had to be read off the board by hand between two boots |
| R-59 | **[P1] A no-DHCP install must HARD-ABORT — instead it bakes the installer's fallback address as a STATIC config and completes, producing a box that can never call home.** | S | **SHIPPED v1.24.0 (2026-07-22) — as a FIRST-BOOT refuse-loudly gate, with a RECORDED DEVIATION: the install-time abort is out of scope (the 192.168.100.2 fallback is baked inside the Proxmox auto-installer itself, unreachable without an installer-initrd hook; operator-acked, not silently dropped). The screen + gate proven on the nested drill (SPIKE-firstboot-nic-sweep-2026-07-22.md); nested ≠ metal — metal proof rides the next real install** | **This is the worst silent onboarding failure shape there is:** the install *succeeds*, the box looks finished, and it is permanently unreachable — no hub check-in, no pairing, no way in except a keyboard and monitor. Found on the HP t740's first install: the 4-port NIC got no DHCP lease at this site (see the t740 gotcha in `scripts/iso/README.md`), and rather than refusing, the installer wrote its **192.168.100.2 fallback as a static `vmbr0` address** into `/etc/network/interfaces` and carried on. **The philosophy is already established one layer over — the disk filter refuses loudly and touches nothing when it cannot identify its target** (spike S5c, proven twice on real boards). Networking deserves the identical treatment: no lease on any carrier-bearing NIC ⇒ **abort with a legible screen**, never invent an address. Slice: detect "DHCP produced no lease" in the answer/first-boot path and fail with the candidate NIC table (name / MAC / carrier / link speed) plus the one-line remedy, exactly as R-58 slice 1 does for disks — same refuse-loudly grammar, same screen shape. Pairs with **R-60**, which is the self-heal for the case where the cable simply moved |
| R-60 | **[P2] First-boot NIC sweep self-heal: if the hub is unreachable, try DHCP across every carrier-bearing NIC before settling.** | S | **SHIPPED v1.24.0 (2026-07-22) — spike + nested drill proven (SPIKE-firstboot-nic-sweep-2026-07-22.md): cable move → sweep → heal + hub registration unaided in <1 min; sweep is structurally first-boot-only (state.json gate + the unit's done-flag condition); drill also surfaced and fixed the baked-fallback-default-route trap (flush before the bounded dhclient)** | `felhom-bootstrap` currently accepts whatever addressing the installer left behind and, if the hub cannot be reached, simply stays broken. On demo-hp the fix was a human moving one cable from the 4-port card to the onboard port — **a sweep would have healed it unaided**: enumerate NICs with `carrier=1`, DHCP each in turn, and keep the first that reaches the hub. Cheap because the box has nothing to lose at first boot (no customer data, no running guests) and the failure it repairs is total. Deliberately scoped to FIRST BOOT and to the hub-unreachable condition only — a running box must never re-shuffle its own networking. Complements **R-59**: that one refuses to produce an unreachable box, this one repairs the case where the truth changed after the install (cable moved, switch port died, the installer guessed the wrong port) |
| R-61 | **[P1] The baked root password must be knowable by the operator — the recurring console lockout.** | S | **slice 1 SHIPPED v1.24.0 (2026-07-22): the build emits the plaintext into a 0600 sibling `<iso>.rootpw.txt` (single record of truth — never logged/manifested/committed); drill-verified against the installed box's shadow hash. Follow-up (appliance-grade record-keeping) stays open** | The ISO mints a **fresh throwaway crypt hash per build** and the plaintext is discarded, so nobody — including the person holding the machine — can log into the console of a box they just installed. Today that meant reaching demo-hp only through the G1 break-glass credential vaulted in the hub, which is the right mechanism for a *lost* password and the wrong one for a *never-known* password: it requires a working hub, a working network, and operator tooling, at exactly the moment the likely reason you need the console is that one of those is broken. **Slice 1 (do this):** the ISO build emits the baked root password into the build REPORT and the operator cheat-sheet alongside the sha256 — it is already a per-build value, so surfacing it costs nothing and closes the lockout. **Follow-up (appliance-grade):** keep it per-build random and treat the build output as the record of truth. **A fixed well-known password is explicitly REJECTED (operator ruling 2026-07-21)** — a pre-pairing box sits on a stranger's LAN with a predictable root credential, which is a far worse exposure than the lockout it would fix. Relates to G1 break-glass (the vault stays; this is about the window before/without it) |
| R-62 | **[P3] Hub delete dialog: show the customer-id the operator must type, and reword the three acks for the ghost shape.** | XS | **idea (operator, 2026-07-22)** | Cosmetic, hub-only, docs-only in the v1.24.0 train. The delete confirmation asks the operator to type the customer-id, but the id appears NOWHERE on the Edit page the dialog opens from — the operator has to fish it out of the URL or another tab. Also: for a GHOST customer (host already gone) the three acknowledgement checkboxes describe teardown steps that cannot happen; **wording only** — the server MUST keep requiring all three (the render-gate lesson of v0.70.1 stands: reachability and requirements are separate concerns). |
| R-63 | **The install console learns ő/ű** — the kernel default console font lacks the Hungarian double-acute glyphs, so the R-59 network screen and pairing banner rendered ő as blanks. | XS | **SHIPPED (scripts v1.25.0, 2026-07-23)** | `felhom-bootstrap.sh` loads a Latin-2 console font (`Lat2-Terminus16` → `Lat2-Fixed16` → `Lat2-Terminus14`) ONCE before the first paint — idempotent, best-effort (a missing font/ioctl never blocks boot). Lat2 ships in the trixie/PVE base (console-setup), so no copy rewording needed. Font names verified against the package. Nested-console capture proof rides the v1.25.0 drill. Also in the same train: **F9 belt-everywhere RESOLVED** (host-install default appliance leg + live on both boxes + login proven) and the **R-71 build-gate** (`build-felhom-iso` asserts golden ≥ managed floor, `publish-train-rules.md` rule 5). **Live finding: golden 0.153.0 < floor 0.156.0 in production NOW** — the gate catches it; the fix is the golden republish at 0.161.0 (Part 4, pending; the managed floor stays 0.156.0). See `REPORT.md` (2026-07-23). |
| R-64 | **„Felhom↔Felhom media pairing blessed" — the two-box SMB pairing (one box shares, the other mounts it as NAS storage) becomes a supported, documented flow.** | XS–S | idea (2026-07-22) | Origin: the operator ran the pairing drill on the live demo pair and it WORKS — the drill itself is the pending evidence leg (a written run-through with the R-66 surfaces in play). R-66 shipped the enabling visibility: the serving box's address is now on its own Beállítások → Rendszer „Hálózat" card, and the add form names the NetBIOS trap. Blessing = a short customer-facing recipe (`documentation/controller/network-storage-nas.md` naming-caveat paragraph is the seed) + one supported-path sentence in the capability map. Flips: would add a "Felhom↔Felhom media pairing" capability row (currently unlisted). Pairs with R-65 (same two-box topology, entirely different transport + guarantees) |
| R-66 | **The box's own address becomes visible — „Hálózat" card, Debug network dump, NetBIOS hint.** | XS | **SHIPPED (controller v0.159.0, 2026-07-22)** | Origin: the pairing drill — the serving box's IP was findable only as a hint buried on the OTHER box's Megosztás page, and the add form's failure for „FELHOM" taught nothing. Three legs: (A) „Hálózat" card on Beállítások → Rendszer (Helyi cím / Hálózati név only-while-sharing / Átjáró; live per render, stored nowhere — S-5; „—" when unavailable); (B) `network` section in the Debug dump (interfaces/route/DNS/lan_address, best-effort per item); (C) the NetBIOS trap named (Szerver helper text + a purely lexical hint on `unreachable` for single-label non-IP names). **Design decision recorded:** the controller is bridge-netns'd, so ALL guest-net reads go through the one netns door (docker exec into host-networked felhom-samba, `stacks/guestnet.go`) — with Megosztás off the card honestly shows „—" rather than the plausible-wrong 172.x answer. Deployed demo-felhom + demo-hp 2026-07-22; demo-hp live-shows the closed-door path (sharing off → dashes + in-place dump errors), demo-felhom the open one (real .104/.1/\\FELHOM values). Flips no capability-map row (diagnosability/UX polish); enables R-64 |
| R-67 | **The NAS share appears in FileBrowser — browse what you mounted.** | S | **SHIPPED (controller v0.160.0, 2026-07-22)** | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). **Couples to R-64: browsing was its missing UX half.** A registered network storage now binds its share ROOT into FileBrowser (`/mnt/felhom-drives/<name>:/srv/<name>:rslave`) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (`Host is down` in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: `felhom-controller/REPORT.md` (2026-07-22) |
| R-68 | **Notification train: paired recovery mails + prefs seeding at claim + priority headers (power-outage audit F11+F12+F14-light).** The dead-man's-switch fired perfectly on 07-22 and the customer who got „A szerver nem elérhető!" was never told it recovered (F11); a customer without a `customer_notifications` row is silently unnotifiable (F12, demo-hp live); delivered ≠ noticed (F14). | M | **SHIPPED (hub v0.71.0, 2026-07-22)** | Origin: `AUDIT-power-outage-recovery-2026-07-22.md`. Recovery = explicit eventType branch (severity semantics frozen; `severityNotifies` untouched): operator always hears both edges, customer iff PAIRED (customer-channel `sent` stale/down row newer than the last sent recovery — `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for recovery; ties → no mail, flap-safe). Seed-at-claim: `MarkClaimed` → `SeedNotificationPrefs` (INSERT-if-absent, never upsert — red-proofed; empty email no-op; never fails the claim; default critical-only set). Hub-side empty-email no-clobber belt in `handleSavePreferences` (controller 0.160.0 already guards its own two push legs — latent, not live). `X-Priority: 1` + `Importance: high` on error/critical via Resend `headers` (live-probed HTTP 200 before implementation); the `test` event now also mails the operator with those headers (one click proves both channels + rendering); latent `sendTestEmail` nil-prefs panic fixed. 17 tests + 4 red-proofs. **Live legs pending:** natural `*_recovered` mail on the next real staleness cycle (or the reboot-drill arc — NEVER fabricated by blocking reports, that is F9-bypass-shaped) and seed-at-claim on a real claim (Peti Friday reinstall is the natural candidate) |
| R-69 | **F14-full: an operator push channel that actually interrupts (ntfy / Telegram / similar), beyond mail-client priority flags.** F14-light (v0.71.0 headers + Gmail filter) nudges a mail client; a 15:29 node_down should reach the operator's pocket in seconds regardless of inbox hygiene. Needs: channel choice (self-hosted ntfy on k3s vs Telegram bot), dispatcher fan-out seam, per-severity routing, quiet hours. | M | idea | Origin: `AUDIT-power-outage-recovery-2026-07-22.md` F14. Deliberately NOT built in the v0.71.0 train (scope-forked per the task spec) |
| R-70 | **[P2-HIGH] The offsite last mile is invisible on BOTH surfaces — the hub cannot tell "staged" from "delivered" from "applied".** demo-hp sat 2 days with the hub customer page saying "Provisioned: … the transient password is delivered to the controller once" while the box said „Még nincs beállítva távoli mentési cél" — and a real customer would sit unprotected indefinitely believing otherwise. The hub HAS the signal (`one_time_secrets.consumed_at` + 153 consecutive reports carrying no offbox object) and reads none of it: the "Provisioned" line is static copy gated only on `offsite.host` in ConfigJSON (`config_form_body.html:119–120`). | S–M | **SHIPPED (hub v0.72.0 + controller v0.161.0, 2026-07-23)** — detector `offsite.DeliveryStateFor` (one impl, all consumers), customer-card state line with age (static "delivered once" copy GONE), `offsite_delivery_stuck` warning event (24h durable cooldown), controller truthful empty-state banner. Live validation on the two fixtures (demo-hp `applied`; demo-felhom `applied` + stale-staged info) recorded in `felhom.eu/REPORT.md`; the banner leg is unit-proven/live-pending (no box occupies the enabled+no-offbox window). | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. Two legs: **hub customer card** shows the real delivery state ("provisioned, awaiting box consumption" / "consumed, awaiting apply" / "applied" — consumed_at × report-offbox-presence is enough for all three), and **controller banner** when the descriptor is enabled but no target is configured („Felhom offsite készen áll — a beállítás automatikus, folyamatban"). Couple to **R-31**'s async/status-card idiom (same surface likely serves both) and to the **R-39 `consumed_at` honesty gauge** precedent on the PBS side — a consumed secret + N report cycles with no offbox status is the same "disagreement no single tier can see" shape and deserves the same loud event. Supporting live datum: demo-felhom's 07-21 staged secret is still unconsumed today (key-auth-first path never consumes) — invisible for the same reason. |
| R-71 | **[P1] Day-0 race: the managed floor-update kills the offsite apply-bridge between password-consume and persist — the one-shot credential is burned and the box lands in the silent consume-404 dead-end forever.** Proven on demo-hp (07-21): consume 16:27:42 → managed update 0.153.0→0.156.0 replaces the container 16:28:17, ~35 s later, mid `ssh-copy-id` window; nothing persisted, no installed key ⇒ the key-auth-first recovery path can never engage, and every subsequent start logs the WARN and gives up until an operator Re-issue. **This recurs structurally on every fresh onboarding whose ISO floor lags the managed floor** — the update fires minutes after first boot, exactly when the bridge first runs. demo-felhom escaped by timing only. | M | **SHIPPED — (a)+(c); (b) rejected-by-design.** **(a) SHIPPED (controller v0.162.0, 2026-07-24): the apply-bridge settle-gate.** `offsiteapply.SettleProvider.SettleState()` + `SettleFunc` adapter over the updater's OWN `GetFloor()`/`IsUpdateRunning()` (no second floor path); `Bridge.AwaitSettle` defers the consume while `updateRunning` OR below the known floor (the auto-floor update is imminent), releasing at/above floor on the first poll with zero added latency (B′). Bounds 10 s poll / 90 s floor sub-bound / 5 min overall — both bounds GO+WARN (a hub that can't serve a floor can't serve a consume → no burn; R-71c is the belt). Ordering-only — the consume/persist order and the 404-no-oracle contract are UNTOUCHED. Finding: the floor is in-memory (report-ACK ~5–10 s), not persisted → unknown until the first ACK on any restart; the 90 s sub-bound is sized to that. 5 scenarios (A–E) + 4 red-proofs all observed FAIL then restored. Deferral paths NOT live-fired — their precondition is now structurally prevented by the v1.25.0 golden≥floor build gate (the point); live leg = the B′ first-poll GO line on both above-floor boxes. **(c) SHIPPED (hub v0.72.0, 2026-07-23): self-heal restage via the EXISTING Re-issue path (`monitor.OffsiteDeliveryChecker`), consumed ≥1h + ≥4 offbox-less reports + zero offbox evidence, one restage/customer/24h, `offsite_credential_restaged` warning; R-39(a) guard red-proofed.** **(b) two-phase consume REJECTED-BY-DESIGN** — it would touch the 404-no-oracle contract; (a)+build-gate+(c) cover the race without it. **Layering: v1.25.0 build gate PREVENTS, (a) DEFERS, (c) HEALS.** | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` (mechanism cites: bridge order `offsiteapply.go:106–187` consume-then-persist, retry only on process start; the dead-end is even documented in source l.168–173 "the password is spent; reset it on the hub to retry"). Candidate directions, spec-first: (a) **order** — first-boot path lets the managed update settle before the bridge's consume step (cheapest; the race window is the update, not the reboot); (b) **two-phase consume** — hub marks consumed only on a controller ack-after-persist (touches the 404-no-oracle contract, design carefully); (c) **hub-side self-heal** — auto-restage a fresh secret when `consumed_at` is set but K consecutive reports show offsite enabled with no offbox status (the R-39(a) mint-race lesson applies: never restage on top of an UNCONSUMED secret). Pairs with R-70 (visibility) — but visibility alone is not the fix; the burned credential needs an unattended recovery path. |
| R-53 | **`app_export.html` substituted the CSRF token where the customer domain belongs** - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | **SHIPPED (controller v0.150.0, 2026-07-20)** | One template token (`{{$.CSRFToken}}` -> `{{$.Domain}}`) plus the `Domain` key in `exportPageHandler`'s data map - that handler does not go through `baseData`, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined `<sub>.<domain>` and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F7) |

### Recovery-model gaps (2026-07-28, `07-backup-architecture.md` §10.2)

> Minted when `07-backup-architecture.md` was rewritten as the recovery model. Every one of these is
> a divergence between that model and the system as it is, and each is cited there. **They are filed
> at P2 as the neutral default, not ranked** — ranking them needs the per-scenario RTO/RPO targets
> that `07` §11-C records as never having been stated. Flips: `00-capability-map.md` §C rows, which
> now cite the matrix rather than restating the route.

| ID | Item | Size | Status | Notes / map rows flipped |
|----|------|------|--------|--------------------------|
| R-102 | **Tier-2 writes a full `recovery-unit/` mirror on every run and no code path reads it.** Written at `internal/backup/tier2.go:368-369` ("Unit leg (always)"); `RecoveryUnitPath` resolves to `backups/**primary**/` (`internal/appbackup/paths.go:46-48`) and the only reader of the secondary tree is `internal/backup/tier2_restore.go`, which reads `hdd/`+`userdata/` only (`:101-104`) | M | READY — 2026-07-28 | **Was C9-F4** (`OPEN-ITEMS.md`). The sharp edge is *when* it bites: Tier-2 exists for primary-drive loss, and in exactly that failure the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action — leaving offsite as the only route. LIVE: demo-felhom's `backups/secondary/{bookstack,docmost}/` hold `recovery-unit` and nothing else, at 156 MB and 86 MB. Flips: the Tier-2 row in map §C; `07` §6.3, §7.2 |
| R-103 | **The Tier-2 no-coverage refusal names the working action but does not route to it.** v0.183.0 refuses up front without stopping the app and tells the customer to use „Visszaállítás indítása" on the other page; it does not take them there | S | READY — 2026-07-28 | **Was C9-F1b.** Deliberately its own item: it puts a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference. Flips: nothing until shipped; `07` §10.2 |
| R-104 | **An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach.** `resticStep` has `unlock --remove-all` (`internal/backup/offbox.go:634-648`) but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`:77-93`) has no lock case → `"other"` → fail-fast; `ClassifyOffsiteFailure` likewise, so the operator is told *„A távoli mentés ismeretlen okból nem sikerült"* for a precisely-known, self-healable condition | S | READY — 2026-07-28 | **Was C9-F3.** Reachable by any interruption — container restart, OOM, network drop, host reboot mid-backup. The tier stays dead until a human runs `restic unlock --remove-all`. Flips: the offsite row in map §C; `07` §8 row 15 |
| R-105 | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) | M | READY — 2026-07-28 | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. **UPDATE 2026-07-28 (vzdump-target move): the `drives` third is TRACED and now POPULATED on both demo boxes.** Cause: the enrolled data drives were never PVE storages at all — only agent-generated systemd mounts — so they never entered `report.StorageTargets` and `isUserDataDrive` never saw them. Giving each drive a `dir` storage at its own mountpoint supplied all three required fields at once (type `local-dir`, fs-UUID durable id, mount path), and the recipe now emits `uuid:91d2dc2d-…`/`/mnt/nvme-1tb` on demo-hp and `uuid:47a3361a-…`/`/mnt/hdd_1` on demo-felhom. **The other two fields (`hosts.dr_record_json`, `host_escrow.directive_json`) are untouched and still `{}`** — this narrows R-105, it does not close it. See `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §5.4. Flips: `07` §4 |
| R-106 | **The DR recipe records the PBS namespace as `"root"` on every box** while the real namespaces are `demo-felhom` / `demo-hp` | XS | READY — 2026-07-28 | Traced to source: `Snapshot.Namespace` decodes a `ns` field (`felhom-agent/internal/pbs/client.go:97`) that PBS does not echo per item when the request is already namespace-scoped via `?ns=` (`:117-120`), so it is empty and `ToHub` normalises empty → `"root"` (`internal/pbs/report.go:23-27`), which `latestPBSCoord` writes into the recipe (`internal/hub/dr_recipe.go:149`). `latest_snapshot_id: "9201"` is **not** a defect — it is `BackupID`, documented as a coordinate (`:56`). Verified against ep0's real namespaces and a live `pct restore` command line carrying `--ns demo-felhom`. Flips: `07` §4 |
| R-107 | **No offsite action unpacks the named-volume tars Tier-3 captures on every run.** `ReconstituteFromOffsite` skips the unit outright (`internal/backup/offbox_reconstitute.go:284-289`); `PlaceOffsiteRestore` places it only when the live unit is ABSENT (`internal/backup/offbox_restore.go:352-356`). The DB **is** replayed, from the scratch unit's `db-dumps/` | M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but **no single action does it and no UI routes it** — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; `07` §6.3, §7.2 |
| R-108 | **Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5.** For local drives FileBrowser binds `<drive>/userdata` only (`internal/web/handlers.go:2450-2460`); for network paths it binds the share **root** with `:rslave` (`:2432`) and serves it with `download: true` (`internal/infra/infra.go:326`). Nothing stops an app's namespace landing there: `GetSchedulableStoragePaths()` has no `IsNetwork()` filter (`internal/settings/settings.go:904-914`, feeds the deploy dropdown at `handlers.go:462-473`), the per-app migrate target list has none (`handlers.go:674-679`), and `handleStorageMigrateApp` does not call `refuseNetworkLifecycle` though its whole-namespace sibling does (`internal/web/storage_handlers.go:397` vs `:410-424`) | M | **READY — blocks an architectural target** — 2026-07-28 | **Today this is not a secret leak** — the recovery unit's `app.yaml` is secret-stripped (`internal/backup/recovery_unit.go:73`). It becomes one under **D5**, which would put app secrets in the local unit precisely so Tier-1/Tier-2 restore stop needing the guest. Verified LIVE in demo-hp's generated compose, where the asymmetry is visible rather than inferred: `- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb` beside `- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave`. **Second effect, independent of D5:** `.fab` bundles already carry **plaintext** secrets by design with an OPTIONAL password (`internal/appexport/export.go:484,506-511,307`) and `storageDriveList()` does not filter network paths (`internal/web/handler_export.go:377-387`), so one can be exported onto a NAS today. Full sweep of every read surface: `07-backup-architecture.md` §10.1. Flips: `07` §7.3 (D5 target → adopted) once closed |
| R-109 | **The DR recipe records no backup target.** `BuildDRRecipeHostHalf` (`felhom-agent/internal/hub/dr_recipe.go:86`) emits `guests`, `drives`, `pve_storage` and the PBS coordinate, but nothing that says WHICH storage holds the local whole-guest archives. Harmless while the target was the well-known `local`; not harmless now — after the 2026-07-28 vzdump-target move each demo box carries **two** `content=backup` dir storages, `felhom-backup` (live) and `local` (frozen 2026-07-28 archives, never refreshed again) | XS | READY — 2026-07-28 | Found by `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` Phase 0 §5 and confirmed live in the recipe both boxes emit. A restorer reading the recipe cannot distinguish the live target from the stale one, and picking wrong yields a silently outdated guest. Add the resolved `BackupTarget()` to the host-half. Sits with **R-105**/**R-106** as the third recipe-completeness defect. Flips: `07` §4 |
| E-2 | **Drive-role machinery around the moved vzdump target.** The 2026-07-28 runbook proved the architecture change by hand on both demo boxes; this is the machinery: a **backup-target role** on `StoragePath` beside `Schedulable`/`IsDefault`/`Kind`; **assignment in the storage wizard** (suggest by attribute, refuse the absurd, never decide by transport or `removable` — on the reference hardware demo-felhom's target IS a USB HDD and BOTH drives report `removable=0`); **unassigned drives do nothing automatically**; **stickiness** (never silently retarget); `felhom-host-install.sh` creating the target with `--is_mountpoint 1` **and** issuing the `FelhomAgentStore` ACL; **absent-target policy**; **retention/space accounting** on a drive the customer shares; the honest **single-drive label**; remaining fleet migration | M | READY — 2026-07-28 | Full scope + rationale in `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §7. Two traps already paid for live: the storage `path` must BE the mountpoint or the agent reports the target `disconnected` forever (`internal/storage/observe.go:321`), and the per-storage `FelhomAgentStore` grant is mandatory or every backup 403s. Absent-drive behaviour today is **fail-loudly, no silent retarget** (`is_mountpoint 1` proven live) — which is NOT the intended fall-back-and-alarm design. Flips: matrix row 4 |

## P3 — post-alpha

| ID | Item | Size | Status | Notes |
|----|------|------|--------|-------|
| R-26 | **Guided old-history recovery via a retained superseded escrow + the recovery code.** Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (`host_escrow_superseded`, `ListSupersededEscrow`). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside `.orphaned-<date>` repo for restore. | M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin `DIAGNOSE-offbox-repo-orphaned-2026-07-17` |
| R-27 | **Customer-facing self-bind page (R-21 slice C follow-on).** Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | **SHIPPED (slice 1, hub v0.66.0, 2026-07-17)** | **Slice 1 = the FIRST-box flow, DONE:** operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor `/bind/<token>` (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same `BindAppliance`, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP `/bind/` exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). **Slice 1 PROVEN-LIVE 2026-07-18:** in the rehearsal the operator minted + emailed the link at 16:28:55 and **the customer bound their own box at 16:29:55 with zero failed attempts**, `appliance_bound` carrying source `customer_selfbind`, credential delivered 26 s later with no operator action (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). The new-ISO console banner rendered on the real box. **Slice 2 = console-passphrase bind → R-27c** (below). Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. **Multi-box per link is out of scope by design** (single-active, one-shot → binding several boxes = repeated operator sends) |
| R-27b | **Customer self-bind, second-box flow (controller side).** For a customer who ALREADY has a bound box and installs another, the controller shows a dismissable "bind another box" prompt (and a bind-later entry under settings) that walks to the hub `/bind/` page — so a returning customer isn't emailed a fresh operator-sent link for every box. Mechanism sketched in the hub v0.66.0 REPORT; NOT built (R-27 slice 1 deliberately did not touch the controller). | M | idea (minted by hub v0.66.0) | Origin: hub v0.66.0 slice-1 ship (first-box only). Reuses the same `/bind/` public page + tokenized-link machinery; adds a controller-side entry point + the operator "mint a link for an existing customer" affordance |
| R-25b | **RULED: customer DELETE becomes a guided full-teardown cascade.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown. | M (was S) | **SHIPPED hub v0.69.0 (2026-07-21)** | **operator ruling 2026-07-21**: **DELETE subsumes the whole cascade, behind explicit consent.** Three separate acknowledgements, each its own checkbox — (1) the host(s) will be deleted, (2) the customer will be RESET including external teardown and offsite data destruction, (3) the customer record and escrow will be purged — plus a **typed customer-name confirmation** before the button arms. Internal order is **host-delete → RESET → delete**, which preserves every existing invariant rather than relaxing any: RESET keeps its no-hosts precondition (hosts are already gone by then), and escrow keeps its demote-then-purge custody rule (host delete DEMOTES to retained custody, the final delete PURGES — the one true purge point). **Re-sized S → M: this is a multi-step destructive wizard with three acks and a typed confirmation, not a checkbox.** Implementation is explicitly NOT part of TASK-E; the row carries the ruling and awaits its own spec. **It no longer blocks R-3** — the model is decided, so the friend-alpha runbook can be written against it. **IMPLEMENTED per the ruling (TASK-I, hub v0.69.0):** `POST /configs/{id}/delete` now runs `hosts → RESET → purge`; three acks + typed customer-id + a stale-preview check + the ONLINE-host refusal, all gates before any write (zero side effects on refusal); custody purged exactly ONCE in leg 3 (leg 2 runs with `purgeEscrow=false`); ruling-3 preserved BY CONSTRUCTION and asserted from inside leg 2; failed legs retain the journal and the dialog offers Resume. Standalone RESET byte-identical. 5 red-proofs. Offboarding guidance: `runbooks/RUNBOOK-onboarding-draft-v4.md` §G. **v0.70.0 follow-up (same day, found validating against the live hub):** a completed delete still left the customer on the Customers list and still ALERTING, because `GetCustomers()` is report-derived and no tier ever deleted a report — new **residue** leg (reports/telemetry/log-tails/notif-prefs + the credential-bearing `appliance_registrations`/`selfbind_tokens`), and **ghost customers are now deletable** (404 = nothing here, not no-config-row). **v0.70.1 (2026-07-22): the ghost delete was implemented but UNREACHABLE** — the Danger-zone card (and the `customerDeleteOpen` script) sat inside `{{if .HasConfig}}`, so a ghost rendered no Delete button at all (the fourth inert-seam defect; handler tests POST directly and proved nothing about reachability). Render gate split: RESET stays HasConfig-gated, Danger zone gates on `Deletable` (the exact negation of the preview's 404 predicate), Block/Unblock stay config-only; render tests per branch + 2 red-proofs. **Operator live leg: the demo-vm-felhom ghost delete click — PENDING** (doubles as the v0.70.0+v0.70.1 live validation; expect `residue=ok customer_delete=ok` with `skipped_no_config` Hetzner/descriptor legs, staleness emails stop) |
| R-25 | **Device-node TOCTOU hardening (drive init).** Graduate the controller v0.141.0 Observation: the `format → resolveEnrollUUID(path) → AssignDisk(uuid)` sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. | S | idea | From the v0.141.0 F6 commit's security-review finding (`felhom-controller` REPORT). Low real risk (single-operator, agent-guarded), but cheap to close |
| R-24 | **Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17).** The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE `SetConfig` (live cgroup, no reboot, Phase-0 proven). **Framing note:** the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). **COMPLETE — the apply is proven in BOTH directions on metal (2026-07-18):** customer zero shrank the guest **11675 → 8192 MB (16:50:22)** and grew it **8192 → 12288 MB (17:02:17)**, each a live cgroup apply with no reboot, the new total rippling into the deploy page's memory math. The nested demo had proven the refusals; the N100 proved the applies. `tests/VALIDATION-n100-rehearsal-2026-07-18.md` ledger 9. **Cores/live-resize as hub desired-state is NOT built** (deferred, low demand). | M | **SHIPPED + PROVEN-LIVE** | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". |
| R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster |
| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | **first slice PROVEN-LIVE (poke channel)** | **FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17):** the ep0-relayed contentless poke (hub→ep0 `felhom-poke` forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the **agent** and fires an immediate desired-state cycle. Full path live: real operator manifest save → `sync-poke delivered to 10.77.0.2`; box → `poke received → immediate desired-state cycle` (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per `SPIKE-immediate-sync-transport-2026-07-16` P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately |
| R-28 | **Agent fast-tick-until-first-convergence — SHIPPED (agent v0.90.0, 2026-07-17).** Closes the immediacy leg a hub poke can't reach (the pre-tunnel WG-registration window): while ANY desired-state item is unapplied the agent pulses the out-of-band trigger every 30 s and **self-disarms** on convergence (state-based; four cached sources — desired-gen==0, reconcile Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational; LOUD states + pending_signature excluded). LIVE on both demo agents (startup line verified). **Guests-0/0 passenger:** diagnosed live and REFUTED as a bug — the 0/0 was the legitimate pre-provision reporting window; the fast-tick is exactly its mitigation (≤15 min → ~30 s). **Unfired live:** the real-onboarding proof (a fresh box converging in seconds without a poke) → the rehearsal S5. | M | **SHIPPED** | See felhom-agent REPORT (v0.90.0). Third immediacy leg alongside R-13 (poke) + R-23 (UI). |
| R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | |
| R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11). **Launcher coupling — REVISED (controller v0.165.0):** the "share the launcher outside the household" need is now met WITHOUT member accounts — the **Indítópult megosztása** capability-URL guest link (`/s/<token>`, information-only, no account) shipped in v0.165.0. What remains for this arc is member-specific: **per-member tile visibility** (each member sees only their apps) and the launcher-as-member-landing-page — both live inside this SSO/members arc; the guest-link ruling explicitly SUPERSEDES the earlier "members are how you share the launcher" framing |
| R-72 | Curate `brand_color` for the top catalog apps | XS | idea | Parked follow-up to the v0.163.0 launcher. `.felhom.yml` `brand_color` (`#rgb`/`#rrggbb`) overrides the deterministic slug-hash tile color; no catalog app sets it yet. Pick brand-accurate colors for the most-installed apps so their launcher tiles match their real brand. Catalog-only change (`app-catalog-felhom.eu`), `brand_color` is already `omitempty` and consumed by the controller |
| R-73 | **SMART history / trending (hub-side)** | M | idea (parked) | Parked follow-up to the v0.169.0 disk-health card + degradation alert (agent v0.94.0 serializes SMART into `/disks`; controller renders „Lemezek állapota" + fires `disk_health_degraded` on a degradation). Today the verdict is point-in-time and the baseline is in-memory (a controller restart re-baselines silently). Hub-side would let the operator SEE a disk's SMART counters trend over time (reallocated/pending sectors climbing, NVMe wear %), catch slow degradation the 6h transition check can miss between reboots, and persist the baseline across restarts. Needs a hub ingest surface for periodic SMART snapshots + a small time-series store + an operator trend view. **No demo hardware exposes real SMART** (virtualized), so this only earns evidence on a real box |
| R-74 | **Island control plane on a CLUSTER (Peti's 2 nodes)** — bring R-50's island bridge to a multi-node PVE cluster. | M | idea (Phase C of R-50, parked) | R-50 shipped the island for the ONE-host fleet (demo-hp, demo-felhom). A cluster needs **bridge parity on every node**: either per-node identical `/etc/network/interfaces` `vmbr9` stanzas (simplest, drift-prone) or — preferred at ≥2 nodes — a Proxmox **SDN zone/vnet** defined cluster-wide (one definition, auto-applied per node). The guest island IP is per-guest + node-independent; the **agent-follows-guest** rule holds (each node's agent binds its own `vmbr9` `169.254.253.1`). Migration order per the spike: drill-proven → demo (done) → **Peti (this row)**. Its own supervised runbook, coordinated with Peti (a live customer). Completes the capability-map "site/network change" row for clustered installs. Source: `audits/SPIKE-island-bridge-2026-07-25.md` (cluster-parity finding) + `RUNBOOK-island-migration.md` (single-host procedure to generalise) |
| R-75 | **Catalog-derived userdata skeleton + import surfaces** | M | **SHIPPED (controller v0.172.0 + catalog, 2026-07-26)** | `${IMPORT_PATH}` = `<system namespace root>/userdata/import` — ONE canonical drop-zone per box on the system drive, with **no per-drive fallback** (unresolvable leaves the var unset so compose fails loudly). *Operator ruling overriding the spike's Fork-1 per-drive recommendation:* each drop-zone app has exactly one ingest bind, so a per-drive `import/` would put a dead lookalike on every other drive — unbacked too, since `import/*` is `class: excluded`. Third `BindRoot` (`RootImport`) threaded through the classifier AND every resolver (`resolveAbs` would otherwise have named a directory on the WRONG DRIVE); catalog-derived skeleton, **sorted** (the naive map-order derivation measured 20 distinct outputs from 20 identical runs — a fleet-wide FileBrowser restart loop via `fbNeedsRecreate`); one authoritative compose parser; FileBrowser „Beolvasás" source; app-page „Hova tegyem a fájlokat?" with `PathEscape` deep links and **class-driven** copy; `data_paths:` annotation (malformed path ⇒ whole-block reject, unknown role ⇒ fails OPEN); system-owned `beolvasas` SMB share refused server-side at handler AND store. **Live-proven on demo-hp:** consume bind resolves to the system drive, a file dropped through FileBrowser's mount was ingested and deleted in ~30 s, the share auto-appeared on enable and survived a direct delete POST, and the before/after directory diff on BOTH boxes shows **zero removals** (demo-hp gained `media/podcasts`, the one genuinely derived entry). Two defects found and fixed during the live legs: `EnsureImportRoot` left the parent `userdata` at 755, and the carry-list initially re-created the per-drive drop-zone forever. Capability-map row *File access via browser* **deliberately NOT upgraded** — nothing drove the FileBrowser HTTP UI. Docs: `controller/import-and-data-paths.md`, spike `audits/SPIKE-catalog-data-paths-2026-07-26.md` |
| R-77 | **Endpoint-drift detection, samba protected-set gate, channel log honesty** | S | **SHIPPED (controller v0.173.0 + hub v0.74.0, 2026-07-26)** | Fixes the two defects in `audits/DIAG-agent-channel-2026-07-26.md`. `bootstrap.DetectEndpointDrift` names a `controller.yaml` vs `bootstrap.json` `local_api.endpoint` divergence with its own event type `local_api_endpoint_drift` and its own banner — and **writes nothing** (see R-78). `EffectiveProtected` now gates samba on `Enabled && UserSet`, mirroring both of `reconcileSambaAt`'s early returns, plus the doc-comment correction. Channel log distinguishes born-down (`unseeded->down`) from a real transition, logging-only. **Part 0 repaired both production boxes** (endpoint → `169.254.253.1:8443`; fingerprint+token agreed). Tests 951→959; red-proofs A/E/F. |
| R-78 | **`local_api` authority ruling — auto-reconcile vs detect-only** | M | idea (deferred OUT of R-77 on purpose) | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. |
| R-79 | **`report.Issues` / `report.Warnings` are English on customer-facing surfaces** | M | idea | **Whole-surface, not a one-off** (DIAG §6): every producer is English — `"SSD/HDD disk usage critical"`, `"Docker: %v"`, `"Protected container not running: %s"`, and all six `Warnings` strings. They render on the customer's Hungarian dashboard, and the `health_critical` path has reached the **customer** email channel three times historically. Deliberately NOT bundled into R-77: a copy sweep across every producer would have buried two safety fixes in string churn, and the seam is not obvious — translate at the producer, or at the render/notification boundary where operator-English and customer-Hungarian already diverge? Pick the seam in a spike; the strings are mechanical after. |
| R-80 | **`expected_backup_missed` false alarm — diagnosed + class-fixed** | M | **SHIPPED (hub v0.75.0, 2026-07-26)** | Diagnosed in `audits/DIAG-backup-missed-2026-07-26.md`, fixed as R-81. **The premise was wrong on both counts:** it fired ONCE (not nightly), and no real external customer was notified — the one customer-channel mail went to the operator's own mailbox; `peti-felhom` (the only real external customer) did not fire. **The 7.3 days did NOT predate the channel outage** — it is the age of the *PBS* snapshot, reached only as a fallback once the vzdump array went empty. Local vzdump was never stale (archives on disk 07-24/25/26). Cause: the agent's backup store is in-memory, so the R-50 fleet restart at 12:44 UTC emptied `backups` until the next backup at 07:03, and the hub read empty as "no backup exists". Fixed by anchoring (hub v0.75.0). **Left standing as a REAL finding → R-82:** the PBS/offsite-DR tier has no schedule at all. |
| R-81 | **"No signal" is not "bad signal" — anchor the backup deadline check** | S | **SHIPPED (hub v0.75.0, 2026-07-26)** | Third instance of one class (hub v0.12.0, v0.73.0, this). `assessBackupFreshness` now returns OK/UNKNOWN/MISSED; absence is UNKNOWN until it outlives an anchored window. The hub reads its own retained host-report history (`GetHostReportsSince` + `newestBackupEvidence`, 7-day bounded lookback) to answer "when did I last SEE evidence of a backup?", and anchors absence at first contact (`GetFirstHostReportAt`), reusing the existing 26 h threshold as the grace — no new knob, the v0.73.0 shape. Deferrals are logged; reason strings kept distinct. Tests 493→508; red-proofs A/B/C. **Not silence** — a genuinely dead box still alarms, which is the half the naive fix breaks. |
| R-82 | **The backup target split — local daily + offsite weekly** | M | **SHIPPED (agent v0.102.0 + controller v0.175.0 + hub v0.76.0 + host-install 1.20.0, 2026-07-26)** | Phase 0: `audits/SPIKE-r82-phase0-2026-07-26.md` (weekly CONFIRMED; the `pvesm status` 0/0/0 resolved as a namespace-scoped-token reporting artifact; capacity measured). `backup_targets[]` gives each tier its OWN cadence and retention; the controller runs both due tiers in ONE quiesce window; the hub judges each against its own threshold (host 26h / offsite 8d); host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly, and an unprovisioned tier DEFERS rather than firing at a missing storage. Operator rulings 2026-07-26: 2-week offsite retention, first backup runs as long as needed, one backup at a time per guest, drill box dropped from the rollout. **Live:** demo-felhom + demo-hp both on local-24h + felhom-pbs-168h; demo-felhom's first real PBS backup landed (`TASK OK`, 41 min, 14.46 GB) and restored cleanly. **Four defects found by RUNNING it, not reviewing it:** 30-min wait bound vs a 41-min backup (v0.98.0), restore tier read from the configured target not the archive (v0.100.0 — a silent regression of the S4.1 fix), a leaked scratch keeping `onboot:1` (v0.101.0), and a tier firing at a not-yet-provisioned storage (v0.102.0). **REMAINING GATE:** the installer default is written but the fleet-wide flip waits on a full weekly cycle holding; and the offsite tier is never AUTOMATICALLY restore-tested (the scheduled test only selects the primary tier) — that gap, plus the hub inferring cadence from storage TYPE, are the two follow-ups. |
| R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. |
| R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. |
| R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). |
| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — operator ruling 2026-07-27** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
| R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. |
| R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | **Part 1 SHIPPED (controller v0.176.0, 2026-07-27); Part 2 OPEN (agent wire change)** — **Part 1** added the failure breaker: consecutive failures tracked per TARGET, backoff `15m→30m→1h→2h→4h` capped, reset on success, gating the QUIESCE rather than the backup so no stack is stopped for a doomed attempt. Never permanent (the cap bounds the retry interval — a latched breaker is a silent backup outage), never global (a broken offsite tier cannot suppress a healthy local one), never applied to `TriggerNow`, and `stillRunning` is not a failure. In-memory on purpose: forgetting a backoff costs one attempt, persisting one could outlive the fix. **Part 2 is agent-side and needs a wire change** — see the P0 finding below. Live on both boxes; the loop reproduction is UNIT-TEST ONLY (reproducing it live would need PBS taken down on ep0, which has no swap). | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **NOT bounded, and it also DISABLES the business-hours protection — an availability fault, not a logging nuisance.** The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves, and a second reading ("it stopped after three tries, so something limits it") was wrong too — the live run stopped after three only because the storage came back. **Verified in code: there is no limiter.** `internal/quiesce` holds no failure counter, no backoff, no breaker, no attempt budget; `runOnce` keeps no state between ticks, and the driver is a plain 5-minute `time.Ticker` (`quiesce.go:149`, default `quiesce.go:107-109`). Had the target stayed down, the loop would have continued indefinitely.

Each attempt runs the **full quiesce cycle**, so every customer app stack is STOPPED and RESTARTED for a backup that cannot succeed. Measured on demo-felhom: `07:07:58 quiescing 4 stack(s): [bookstack calibre-web docmost immich]` → `07:08:17 unquiescing (backup failed)` → `07:08:45 failed` — **~19 s of app downtime per cycle (~50 s per full cycle), every 5 minutes.**

**The amplifier, and the part worth designing against:** the agent answers `Due: true, Reason: "no successful backup recorded yet", AgeSecs: nil`, and that **nil age does double duty**. `scheduledRunAllowed` (`quiesce.go:466-480`) returns `true` whenever `lastAgeSecs == nil` — *"no recorded backup yet — never withhold the first one"* — so the same nil that makes every poll due **also bypasses the time-of-day gate** `[W+2h, W+6h)`. On the live box the gate was `[04:30, 08:30)` and the cycles ran at 09:02–09:12 Budapest, i.e. **outside the backup window entirely**. So the fault stops customer apps every 5 minutes *at any hour, including business hours* — the one protection specifically built to prevent that is switched off by the same missing value. A safety valve written for a genuine first-ever backup is being triggered by an unreachable storage read, which is not the same thing at all.

Self-resolves the moment the target answers (the storage read succeeds, sees the archive, tier stops being due) — which is why it can hide indefinitely: it needs an offsite outage to appear at all. **PHASE-0 ROOT CAUSE, established at source 2026-07-27 — it is AGENT-side, case (a).** The storage read **errored** (`could not read the backup storage for the due-check … err=…` at 09:02:57/09:07:58/09:12:57 CEST), so this was never an empty-success. The failure is a **type boundary**: `newestArchiveOn` (`localapi/server.go:1095-1111`) documents *"Errors and unsupported services degrade to unknown, never to 'no backup'"* — but its `(time.Time, bool)` signature **cannot represent unknown**, so an error and a genuinely-empty storage both collapse to `(zero, false)`, and `handleBackupDue` (`server.go:934-941`) then emits a POSITIVE claim: `Due: true, Reason: "no successful backup recorded yet", AgeSecs: nil`. The fail-safe that *does* exist — `targetStoragePresent`'s "a storage-view error must never be read as 'not there'" (`server.go:1131-1151`) — answers a different question (does the storage exist) and behaved correctly. **Decisive for scoping: the errored path and the genuine-never path are BYTE-IDENTICAL on the wire** — same `Due`, same `Reason` string, same nil `AgeSecs` — so the controller has nothing to discriminate on and Part 2 CANNOT be done controller-side. **Two further P0 findings:** the agent restarted **4× on 2026-07-27** (07:36:39, 07:54:06, 08:50:16, 11:31:52 CEST) — all deliberate (`NRestarts=0`, `Restart=on-failure`, `Result=success`), zero self-update — so the trigger is armed by ordinary operator/config work far more often than "only when ep0 is down"; and **the loop alerted NOBODY** — zero `backup_failed` events despite the hub allowlist carrying that type, because **`internal/quiesce` does not import `internal/notify` at all**. Its only trace was `07:13:27 info app_start_failed "Telepített alkalmazás nem fut: BookStack"` — a customer-tier, Hungarian, info-severity SYMPTOM of the third cycle catching BookStack mid-restart. **The whole-guest backup tier R-82 built has no failure signal to the hub → its own item.** **Shape:** distinguish *storage unreachable* from *storage readable and empty*. Unreachable is UNKNOWN — defer the due-verdict rather than resolving it either way, exactly as R-81 made the hub do with a missing report. Only a target that is reachable AND has no archive is genuinely due. **Fix the window bypass in the same slice:** `AgeSecs == nil` must stop meaning "run now regardless of the hour". Either the agent distinguishes *never backed up* from *cannot tell* in what it reports, or `scheduledRunAllowed` gates on the former only — otherwise any future nil-age path re-opens the same hole. Note this does NOT weaken R-84's fail-safe intent: a tier whose storage is merely slow or briefly unreadable should still err toward backing up — it is specifically the **cold-store + unreachable** pair that must defer, because there the fallback has no information at all, only an empty default that looks like a fact. |
| R-89 | **Retention is a COMMERCIAL attribute — it belongs to the hub, not to ep0 or a box** | M | idea — operator ruling 2026-07-27, first increment SHIPPED same day | **Ruling (2026-07-27):** retention is a per-customer *commercial* attribute (a paid tier may buy longer retention), so the **hub owns the policy** and ep0 merely executes it. Execution stays **server-side**: a reconciler writes a **PBS prune job** and PBS's own scheduler runs it, so hub downtime leaves the last-known policy running rather than silently stopping retention. **Increment 1 SHIPPED 2026-07-27** (`runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`): boxes no longer attempt prune (`keep_last: 0` → `allowPBSPrune=false`, config only — no code, no grant), and per-namespace prune jobs run on ep0 daily 03:30 UTC (`keep-last 2`), dry-run gated and verified `TASK OK`. This also closed a live defect — **every** demo-hp PBS backup since the tier was created on 07-26 had reported `job errors` while the data landed correctly, because `DatastoreBackup` grants `Datastore.Backup` but not `Datastore.Prune`; a tier that cries wolf on every success makes a genuine failure invisible, which is exactly what happened during the 07-27 migration window. **Remaining work:** retention becomes a per-customer attribute on the hub — a **policy** (keep-last/daily/weekly/monthly), NOT a bare number, since a paid tier will want "12 weekly + 6 monthly"; a reconciler converges it into the ep0 prune job in the same descriptor-and-converge shape as `pbs_dr`. **SECURITY PROPERTY TO PRESERVE — do not "fix" a future prune error by widening the grant:** box tokens stay **write-only** (`DatastoreBackup`), never `DatastorePowerUser`/`DatastoreAdmin`. A compromised box must not be able to delete its own offsite backups — that is the scenario offsite DR exists to survive. **PARALLEL QUESTION, unanswered and arguably the more urgent half: does the restic key on `storage-box-pool-1` (`u629488`) have DELETE rights?** If so the daily app-data offsite tier carries the identical exposure, and restic's **append-only** mode is the equivalent answer. **Rule once for both tiers.** Flips the capability-map row for offsite retention (currently: retention configured but unenforceable) |
| R-97 | **The whole-guest backup tier has NO failure signal to the hub — `internal/quiesce` never notifies** | S | **SHIPPED (controller v0.177.0 + hub v0.78.0, 2026-07-27)** — **R-97a:** `quiesce.TierNotifier`, a seam (not an import) wired by an init-only setter, edge-triggered on the R-88 breaker ARMING so a failing tier is reported once per run rather than once per retry; recovery rides `recordSuccess`'s existing bool. **NEW operator-only event types** `whole_guest_backup_failed`/`_recovered` — deliberately NOT `backup_failed`, which carries a customer Hungarian template AND sits in demo-felhom's live `enabled_events`, so reusing it would have emailed the CUSTOMER about a backup they cannot act on while it was still retrying. The recovery joins `recoveredPairedDownTypes` because its `info` severity would otherwise be dropped by `severityNotifies` — the operator would hear it break and never hear it heal. **The hub's operator cooldown was keyed `customerID:eventType` alone**, so one tier would have masked the other for an hour; now narrowly extended with a `tier` suffix taken from the event details, leaving every other event type unchanged. **R-97b:** a suppression window keyed to the quiesce CYCLE (not a state test — v0.164.0's `!= StateStopped` filter cannot see an app caught MID-RESTART, which is exactly how BookStack alarmed), consumed at the same single derivation point `classifyRunStates`. Grace = **180 s**, derived from the deploy flow's 120 s health timeout and Mealie's 60 s `start_period`; it **expires**, so an app that genuinely fails to come back still alarms. **PROVEN LIVE end-to-end with a control:** the new type POSTs 200 from inside guest 9201 while a bogus type 400s, and `notification_log` shows **1 operator row, 0 customer rows**. The quiesce→notify link itself is unit-proven only. | On 2026-07-27 three whole-guest backups failed and three quiesce cycles stopped and restarted every customer app stack, and **not one `backup_failed` event reached the hub.** It is not the allowlist — the hub already carries `backup_failed` and `backup_completed` (they are emitted by the controller's *app-data* backup path). The cause is that **`internal/quiesce` does not import `internal/notify` at all**: the tier R-82 built has no route to the hub, so a whole-guest backup can fail indefinitely in silence. The loop's only trace was `app_start_failed` — **info** severity, **Hungarian**, on the **customer** channel — telling the customer BookStack was down (it had been caught mid-restart by the third cycle) without saying why, during an outage the system itself caused. So the one signal that did fire was both the wrong tier and the wrong story. **Shape:** emit `backup_failed`/`backup_completed` from `quiesceAndPollTiers` naming the TIER, operator-tier; and decide whether a quiesce-induced restart should suppress `app_start_failed` the way controller **v0.164.0**'s deliberate-stop filter does — an app the backup stopped on purpose is not a fault. R-88's breaker bounds the repetition but changes nothing about the silence |
| R-95 | **The restic offsite tier's credential CAN DELETE — R-89's "parallel question", now ANSWERED** | M | idea — established read-only 2026-07-27 | **The exposure closed on the weekly PBS tier is fully open on the daily restic tier**, which holds the customer's actual documents and photos and is the only tier that survives losing the box. Established without mutating anything: **(1) Identity** — a per-customer *subaccount* on `storage-box-pool-1` (box 611714, bx11, `u629488`): `u629488-sub1` home `felhom-demo-felhom`, `sub2` peti-felhom, `sub3` demo-hp, each labelled `felhom-customer`. Auth is an **SSH key stored ON THE BOX** (`…/felhom-controller-data/_data/data/offbox/ssh_key`, 0600, beside `repo_password` + a pinned `known_hosts`) — customer-side, not hub-side, so a compromised guest holds it. **(2) Read-write: YES** — the API reports **`readonly=False` on all three subaccounts**, and it is not merely latent: the controller runs `restic forget --group-by host,tags --keep-daily 7 --keep-weekly … --prune` **from the box** (`backup/offbox.go:984`, also `:1070`). Delete rights are exercised on every run. **(3) Append-only: NO, and not expressible** — the repo is built as `sftp:` (`offbox.go:482`); restic's append-only mode requires the **REST server** backend, which plain SFTP cannot provide. **(4) A zero-code mitigation exists and is unused:** the box type carries `snapshot_limit=10` and the API reports `snapshot_plan=null` with **0 snapshots** and `size_snapshots=0`. Hetzner Storage Box snapshots are taken **server-side, outside the SFTP namespace** — an SFTP subaccount cannot delete them — so they are a genuine immutability layer at no extra cost and with no code change. **Rule once for both tiers, per R-89.** Options, cheapest first: enable a snapshot plan (operator click, immediate); split backup-write from prune so pruning runs somewhere the box cannot reach; or move the repo to restic's REST server with `--append-only`. Flips the capability-map row for offsite immutability |
| R-94 | **A hand-synced version constant drifts, and the gate that would catch it is never run** | XS | idea — found 2026-07-27, **corrected 2026-07-29** | **RETRACTION.** The prior detail's conclusion — *"a new install driven from the hub still gets the pre-R-82 default"* — **was false and is retracted.** It assumed `hostInstallVersion` selects a script. It selects nothing: its only two consumers are `configs.go:487` (`ScriptVersion`) and `render_test.go:219`, and it is rendered as a **text label** at `customer_unified.html:494` ("Day-0 host bootstrap for host-install {{.ScriptVersion}}"). The install command printed **beneath that label** fetches `https://felhom.eu/scripts/felhom-host-install.sh` (static block `customer_unified.html:563`; JS generator `:1262`) — as does the ISO's `felhom-bootstrap.sh:96`, which fetches from the **website, not the hub**. That URL is the website's `git-sync` working tree tracking `--branch=main --period=30s` with `/scripts/` in the sparse-checkout, served by nginx's `location /scripts/` (`manifests/webpage.yaml`) — no image build, no ArgoCD step. **So 1.22.0 is what every install already gets**, confirmed by live fetch 2026-07-29 (`curl https://felhom.eu/scripts/felhom-host-install.sh` → `SCRIPT_VERSION="1.22.0"`). Every flag the generator emits (`customer_unified.html`~`:1210`–`:1238`) is parsed by 1.22.0 (`felhom-host-install.sh:1177`–`:1210`): **no functional gap, only a wrong number on the operator's screen.** **What survives from the original entry is its diagnosis, which was right all along: a hand-synced version constant in a second repo drifts every time the first one ships.** `configs.go:28` is still `"1.19.0"` against `felhom-host-install.sh:187`'s `SCRIPT_VERSION="1.22.0"`. Three legs, all XS: **(a)** derive the label from `SCRIPT_VERSION` rather than hand-syncing it, or delete the label; **(b)** `scripts/hostinstall_gates.py` exists precisely to catch this and **fails today** (`FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1) — but it is invoked by no Makefile, git hook, CI config or `CLAUDE.md` (only `site_gates.py` is mandated, `CLAUDE.md:153`), so wire it or delete it: a gate nobody runs reads as coverage it is not providing; **(c)** `render_test.go:219` asserts `strings.Contains(html, hostInstallVersion)` — it compares the constant to itself, passes at any value, and cannot detect drift; replace it with the cross-file assertion. **v0.73.2 is NOT an undeployed artifact** — its content is inside the live hub. See also **R-110**, found by the same investigation: the fetch URL has no staging |
| R-110 | **`main` is the installer's publish channel — there is no staging** | S | idea — found 2026-07-29, **WAITING-ON-OPERATOR (a ruling, not a defect)** | `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a `--period=30s`, and nginx serves that working tree directly (`location /scripts/`, `root /usr/share/nginx/html/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`, `:1262`; `runbooks/day0-install.md` C.1) receives. **There is no tag, no pinned-version path, no staging copy and no rollback other than another push** — for the one artifact that runs as **root on a virgin box**, the most privileged thing Felhom ships. **Two consequences worth stating plainly:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29, so the proof run confirms what customers already receive rather than clearing it for release; and the precaution the old R-94 row recorded ("do not point every new box at an installer that has never run") **was never available to take**, because nothing points boxes at a version. **Open question for the operator, not a defect to fix blind:** should `/scripts/` serve a pinned release — a tag-tracked git-sync ref, or a versioned directory (`/scripts/1.22.0/…`) with the hub's generated command naming a version — or is `main`-tracking the accepted shape for a one-operator product where the alternative is a release ritual nobody performs? **Exposure today is zero** (no boxes are installing), which is exactly why it is cheap to decide now. Whichever way it goes, it decides whether R-94 leg (a) makes the label a *fact* (derived from the served script) or keeps it a *claim*. Flips no capability-map row — the map states what the platform does, and this changes nothing about that |
| R-90 | **ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86** | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: `Swap: 0B`, nothing in `/etc/fstab`, no swap unit, no `/swapfile` — **swap was never added.** On 2026-07-27 06:58:12 a global OOM fired while `proxmox-backup-proxy` held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration `rsync` held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed **R-88** downstream. Still exposed today: 3.8 GB, no swap, one kill on record. **This gates R-86** — backup-aligned restore-testing puts a multi-GB read on this box *on a schedule*, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console |
| R-91 | **The old 13 GB datastore copy is still on ep0's root disk** | XS | WATCHING — gated on demo-felhom's first post-migration PBS backup | The datastore moved to a Hetzner Cloud Volume on 2026-07-27 (`/dev/sdb`, 100 GiB, attached 06:29:40 UTC, now `/mnt/pbs-datastore`, 13 G used of 98 G). The pre-migration copy survives at **`/srv/pbs-felhom`, 13 G**, on `/` (38 G total, 16 G used, 21 G free). **Do not delete yet:** demo-hp has landed two post-migration snapshots (07-27 08:25:47Z, 09:37:29Z) but **demo-felhom's newest is 2026-07-26T12:21:48Z — before the migration**, so the new volume has not yet proven a write for that namespace. Delete once it has. **Doc drift to fix in the same commit:** `CONTEXT.md:1018` still records the datastore at `/srv/pbs-felhom` |
| R-92 | **The hub's PBS-DR gauge is 0.1 GB-granular, so small deltas are unverifiable** | XS | idea — 2026-07-27 | The PBS-DR box card rounds to 0.1 GB, which is coarser than the changes an operator wants to confirm after a prune or a GC — a successful prune of a small namespace moves the number by less than one displayed digit, so the UI cannot distinguish "it worked" from "nothing happened". Cosmetic today; it becomes load-bearing the moment retention (R-89) is customer-visible and someone needs to see that a policy change took effect |
| R-93 | **`drill-r50` is both a blocked customer and the only drift fixture** | XS | idea — 2026-07-27 | The drill customer is blocked in the hub (so it stops alarming) yet it is also the only record exercising the endpoint-drift path R-77 added. Blocking hides it from `GetActiveCustomerIDs`, so the fixture it provides is silently inert — a monitor with no live subject reads exactly like a monitor that passes. Decide: retire it and build a synthetic fixture, or unblock it and silence per-customer instead (the operator has a per-alert silencing feature planned). Related to the R-50 drill VM, now shut down |
| R-96 | **Two standing rules were agreed in chat and never committed** | XS | idea — found 2026-07-27 | Verified by grep across every `CLAUDE.md`, `CONTEXT.md`, `REUSE.md` and `skills/`: **(a) "never combine a test run and a commit in one command"** — zero hits, despite three separate occurrences (the worst pushed a red suite because `packages ok: 28` was read while `rc=1` was not); **(b) "a 'no access' claim must list what was tried"** — zero hits, despite a wrong "no access exists" verdict on ep0 on 2026-07-27 where the working route was documented in memory and only one path had been attempted. **The third rule IS committed** — `CONTEXT.md:8` S-1, N.5's third leg (architecture docs same-session coupled). A rule that lives only in a chat log binds nobody; both belong in `CONTEXT.md` beside S-1/S-2 |
| R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog |
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**)
| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v <volumeName>:/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** **2026-07-29 — audit list extended, and a THIRD independent re-raise absorbed under the same rule (again no new ID):** add `scripts/hostinstall_gates.py`, which **postdates this item** (it comes from drill F-1, 2026-07-12) and is therefore not a design-v2 gate — but it is the identical failure shape and is tracked as **R-94 leg (b)**. It is **RED as of 2026-07-29**: `hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1, with its nine other assertions green. `scripts/hub_confirm_gate.py`, already on the list above, was **verified orphan on the same date**. Both confirmed by repo-wide grep across all file types plus sibling repos, `~/.claude` settings/skills/hooks, `.git/hooks` (no non-sample hooks exist), a Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) and a CI-directory find (**`felhom.eu` has no CI configuration at all**) — all 19 hits are docstrings, code comments or prose; **zero are invocations.** Only `site_gates.py` is mandated (`CLAUDE.md:153`); `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **Now also filed in `OPEN-ITEMS.md`** — this item predates the 2026-07-27 register rebuild and was never carried across, so an open item about work not getting done was itself missing from the page that decides what gets done |

| R-37 | **Post-RESET health card shows stale pre-RESET warnings.** After a RESET the card should read **„RESET óta nincs adat"** instead of carrying warnings about a lifecycle that no longer exists. | XS | **SHIPPED (hub v0.67.0, 2026-07-18)** | The customer page raises a banner when a RESET **completed** after the newest report, quoting „RESET óta nincs adat" and the reset timestamp, because until the box reports again every health figure describes a lifecycle that no longer exists. Deliberately narrow: an **in-flight** reset does not trigger it (only a completed one), and it **clears itself** on the first post-RESET report. Ties resolve to STALE — SQLite timestamps are second-resolution and a same-second report almost certainly arrived just before the reset destroyed what it describes; erring the other way would hide the banner exactly when it matters most. Red-proofed (neutering the predicate fails the assertion). — Origin: 2026-07-18 rehearsal. Same family as R-36 — the hub knows the state changed and the UI has not caught up |
| R-38 | **Installer GRUB slice.** A single default „Felhom telepítés" entry; the **interactive installers REMOVED** (safety: an interactive entry is how a wrong-disk manual install happens); felhom background. | S | **SHIPPED (scripts v1.22.0, 2026-07-19)** | Origin: 2026-07-18 rehearsal, alongside R-21's physical closure. Every ISO is repacked after `prepare-iso`: exactly **one** entry („Felhom telepítés", default, 5 s), with Graphical / Terminal UI / serial and the whole **Advanced Options** submenu (nomodeset ×2, three debug variants, Rescue Boot, memtest, UEFI settings) **not emitted** — not hidden, not password-gated. Boot behavior unchanged: the `linux`/`initrd` lines are lifted **verbatim at repack time** from the ISO's own automated entry, so a PVE bump tracks automatically; the build fails if they are missing, if the append line has lost `proxmox-start-auto-installer`, or if `auto-installer-mode.toml` is absent. Gated for 1 entry / 0 submenus / no live banned references, then **re-verified by reading the menu back out of the finished ISO**. Background generated at repack time from `website/assets/og-image_2.png` (one source, no second copy to drift), with the card's own grid phase-locked across the letterbox fill. Live-validated on the nested canary (UEFI/OVMF): renders, auto-fires, reaches the same match-nothing abort as v1.21.0. **Squashfs/theme rebranding still DEFERRED** — post-GRUB screens remain Proxmox-branded. GRUB's `e`/`c` keys also remain reachable (suppressing them needs a superuser password — a separate decision); the menu no longer advertises them |
| R-40 | **[P2-HIGH] The update path cannot express a MULTI-HOP major upgrade.** A template pin is a single value; the customer's update button pulls whatever the catalog now says. For apps whose upstream forbids version skipping this produces a broken upgrade. Nextcloud is explicit: *"You cannot skip major releases. Please re-run the upgrade until you have reached the highest available release."* Campaign 7 moved its template **31 → 34** (a fresh deploy validates fine — 302, 3/3 healthy), so an existing 31 customer pressing update would attempt a jump Nextcloud refuses. | M | idea | Origin: CAMPAIGN 7 (`audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` §7 F7). Not nextcloud-only — any app with sequential-major rules (gitea, tandoor, outline…) has the same shape. Directions: a per-app `upgrade_path:`/`max_hop:` in `.felhom.yml` that the update button walks in stages; or refuse-and-explain when the installed major is >1 behind; or pin an intermediate "stepping-stone" tag. **Until this exists, a >1-major catalog bump is safe for NEW deploys and unsafe for the update button** — which is exactly the asymmetry the campaign's MAJOR flag was meant to record but cannot enforce |
| R-41 | **[SLICE 1 SHIPPED 2026-07-21] The catalog has no standing "does every template still deploy?" check.** Campaign 7 was the first thing that ever tried to deploy all 53 apps, and found **5 that had NEVER been deployable**: papra (missing required `AUTH_SECRET`), zipline (v4 renamed `CORE_DATABASE_URL` → `DATABASE_URL`), wishlist (Docker Hub image gone; upstream moved to ghcr.io), homebox (upstream dropped the `v` tag prefix + new required env), glance (needs a seeded `glance.yml` the template never provides — PROVEN pre-existing: the pre-campaign v0.7.4 pin fails identically). Plus **7 broken healthchecks** and 2 apps whose images no longer resolve at all (plant-it, wanderer). | M | idea | Origin: CAMPAIGN 7 (§7 F5/F6). The repo already has the right pattern in `scripts/check-image-pins.py` — a mechanical gate run on every change. Cheap first slice: a **resolvability gate** (`docker manifest inspect` every pin) would alone have caught plant-it, wanderer, wishlist and homebox, and needs no box. Full slice: a periodic deploy-all sweep on the demo box reusing the campaign's engine. **Silent rot is the real risk** — an app can die upstream and nobody learns until a customer clicks Telepítés | **SLICE 1 SHIPPED 2026-07-21 — `app-catalog-felhom.eu/scripts/check-image-resolvable.py`** (+ 14 fixture tests, no network, resolver injected). Resolves every unique pin with `docker manifest inspect`, ONE image at a time; exit 0 / 1 (the registry says GONE) / 2 (inconclusive). **Two traps encoded, both hit live while building it:** (a) `docker manifest inspect` prints `toomanyrequests: …` and **still exits 0** — the same exits-0-on-failure shape as the ISO tooling's `validate-answer`, so stderr is inspected even on rc=0; (b) the inverse and more dangerous one — the first full sweep called **24 of 65 pins dead, including `postgres:16-alpine` and `redis:7-alpine`**, purely because Docker Hub throttled it partway through. Ambiguity therefore resolves to INCONCLUSIVE and never to an accusation: a gate that cries wolf gets ignored, and then it protects nothing. **The full sweep is still OWED** — DooPlex is not logged in to Docker Hub, so the 52-app table needs one re-run after `docker login`. Wired into `CLAUDE.md` + `REUSE.md` as a start-of-campaign / pre-publish-train step. It immediately paid for itself: it is what turned plant-it and wanderer from 'images do not resolve' into two DIFFERENT diagnoses (see the 2026-07-21 catalog entry). Full slice — the periodic deploy-all sweep on the demo box — remains open |
| R-42 | **RULED: sidecar majors follow the APP, never the newest tag.** Campaign 7 deliberately did NOT bump sidecar majors (postgres 16→18, redis 7→8, mariadb 11.6→12.3, postgis 16→17) while bumping ~40 app images to current. | S | **RULED 2026-07-21 — option (a)** | Origin: CAMPAIGN 7 §4. The case for not bumping: a DB major is a **data-plane decision the application owns** — immich proves it, upstream pins one specific tested `postgres:14-vectorchord…` build — and `postgres:16-alpine`/`redis:7-alpine` already track the newest patch inside their major, so they are not stale. The case for bumping: EOL majors eventually stop getting security patches, and "we never bump" silently becomes "we ship EOL databases". Suggested shape: per-app sidecar pin follows **upstream's own compose** where upstream publishes one, else stay within the current major and revisit at that major's EOL date |
| R-43 | **[P2-HIGH] No offsite restore path can restore a database — offsite restore cannot reconstitute a DB-indexed app.** All three offsite buttons are file-only. „Ellenőrző visszaállítás" and „Teljes visszaállítás" restore into a **scratch/verify folder** and never load a dump into postgres (`controller/internal/backup/offbox_restore.go:171-232`, `:130-164`); „Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" is `rsync -a --ignore-existing` over **mandatory userdata paths only** — it never touches a DB and never stops/starts the stack (`offbox_restore.go:346-421`). The DB dump is faithfully carried *into* the snapshot and can never be replayed *out of* it by any offsite action; the only code that replays into a live DB is `RestoreFromRecoveryUnit` (`backup/restore_unit.go:74`), reachable solely from the **local** restore panel (`web/handlers.go:1108`). Compounding: the `live recovery unit present — not overwriting` guard (`offbox_restore.go:400-405`) skips the **entire** unit — compose, manifest **and db-dumps** — whenever a local `backups/primary/<app>` exists, which is true on any box whose local backup has run, so on a normal box the offsite dump is never even staged to disk. | M | **SHIPPED controller v0.148.0 (2026-07-19) — live acceptance PENDING** | **SHIPPED:** new „Teljes visszaállítás (fájlok + adatbázis)" (`ReconstituteFromOffsite`, `/backup/offbox/reconstitute`): safety dump → stop → files overwritten to the snapshot version → start → the snapshot's own dump replayed (`reimportDBDumpsFrom`, reading the SCRATCH unit — the live unit is still never overwritten) → health wait. Two invariants, both red-proofed: **nothing is ever deleted** (`rsync -a`, no `--ignore-existing`, no `--delete`; post-snapshot files survive as extras) and **the undo exists before the act** (the `pre-restore-` dump is verified ON DISK before anything is stopped/overwritten/replayed; if it cannot be taken the operation refuses with zero changes). Outcome-shaped flash; the old missing-only button now states what it does NOT do. Deployed to demo guest 9201, healthy; page renders, button correctly gated behind a prepared full scratch. **STILL OPEN — the §9 live acceptance has NOT run** (upload → push → empty the trash for real → one button → photos back in the timeline). Until it does, no capability-map flip, the customer-restore row stays MISSING and R-3 stays DRAFT. — **Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (live on demo-felhom).** Viktor deleted 11 immich photos to test offsite restore; both place runs merged **0 files** and flashed success. Root cause is two-layered — the UI delete only *trashed* the assets so nothing was missing to restore, **but a valid test would also have failed**: files return, the library stays empty, because immich's truth lives in postgres. **The customer-visible failure mode: correct bytes on disk and an app that cannot see them.** Severity case: this is the offsite tier — the one that survives losing the box — and most of the catalog is DB-indexed. Direction (needs a spec, do not improvise): either offsite gains a real DB-replay leg reusing `RestoreFromRecoveryUnit` behind a stop→restore→start sequence, or the offsite UI states plainly that it restores files only and routes the customer to the local recovery-unit flow for the DB. Pairs with R-44 (a replay leg is worthless while the dump is stale). Flips: the **customer-restore map row stays MISSING** and gains this as a note — the C6 evidence attempt found a product gap, which is the rehearsal system working |
| R-44 | **[P2-HIGH] A manual offsite push ships an unrefreshed DB dump — "backed up now" is false for the DB half.** `offboxRunHandler` → `RunOffboxBackup` goes straight to the restic push and never calls `RunDBDumps` / `captureAllRecoveryUnits` (`controller/internal/web/offbox_handlers.go:203-227`, `backup/offbox.go:574-759`); the recovery unit merely **enumerates** existing dump filenames via `listFileNames`, never creates them (`backup/recovery_unit.go:105-106`). Dumps come only from the separate local `db-dump` daily at **02:30** (`cmd/controller/main.go:542`), with the scheduled offsite at 04:15 — so a *manual* run at any other hour ships a dump up to ~24 h old. **There is no freshness check and no RPO surface anywhere:** zero `RPO` hits across `controller/`; `offboxUnitTime` is only a two-drive tiebreak (`offbox.go:827-837`); the `DBValidationCache` exists (`backup.go:364-370`) but no offsite or restore path reads it. | S–M | **SHIPPED controller v0.148.0 (2026-07-19)** | **SHIPPED:** every offsite run — **manual AND nightly** — now refreshes the DB/volume dumps and recovery units (`offsitePreDump` → `runDBDumpsInternal`) BEFORE the restic capture, so each snapshot is an internally coherent `{DB@T, files@T}` bundle and retention becomes a history of restorable points. Order is the mechanism and is red-proofed (moving the capture first yields `[capture dump]`): the gap can only ADD files the DB does not reference yet, never remove one it does. This also makes the nightly ordering **structural** rather than a coincidence of two scheduler entries at 02:30 and 04:15. Each unit manifest carries `offsite_run_id` + `dumps_at`, so a pair's coherence is verifiable at restore time instead of assumed; the periodic refresh carries a prior stamp forward and never invents one. A dump-leg failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none). Honesty surfaces, all warn-level and none a gate: an unstamped pre-v0.148 pair reports its skew in the confirm, and `ValidateDump` gained an **exact-match** accounts-table sniff for customer-empty dumps (a substring match on "user" would flag `user_metadata`/`album_user`/`user_audit` on every healthy single-user box — red-proofed). — **Evidence: `audits/DIAG-immich-restore-2026-07-19.md`.** Today's unit dump `immich-postgres.sql` (51 954 452 B, mtime **02:30 CEST**) probed to **`asset: 0 rows`, `user: 0 rows`, `album: 0 rows`** — the 52 MB is entirely immich's shipped `geodata_places`/`naturalearth_countries` reference data. It predates both the admin user (created 07:56:25) and the photos (07:57). Same for the unit's `immich_immich_postgres_data.tar` (323 MB, also 02:30). **A dump that looks substantial by size can contain zero customer content** — size is not a health signal, and nothing in the product says otherwise. **Latent hazard:** had a full restore actually loaded that dump it would have written an empty DB over the live one, destroying the trashed rows that were the only surviving recovery path. Direction: dump-before-push on manual runs (the honest fix), **or** an explicit RPO line in the UI („adatbázis-állapot: <dump time>") so the operator/customer can see what they are actually shipping. Cheap interim: surface dump mtime + row-count sanity from the existing `DBValidationCache` on `/backups/restore` |
| R-45 | **[P2] Unified async-job feedback.** Every long operation invents its own progress surface, or none. Tonight produced three more one-off cards (v0.147.x: samba bring-up, offsite progress, restore result) on top of two existing patterns (deploy 3-step panel; storage-init/netstorage status poll). They agree on nothing: some use `{ok,data}` envelopes and some raw JSON, some poll 1 s / 1.5 s / 3 s, some are in-memory-only and lie after a restart, and each re-implements single-flight + snapshot + phase→Hungarian mapping. | M | idea | Origin: 2026-07-19 feedback slice 1 (controller v0.147.0). The cases to generalise from are all in-tree: `web/storage_init_job.go` (the best shape — acquire/release/set/snapshot), `web/netstorage_job.go`, `web/samba_ensure_job.go`, `backup/opstatus.go`, `backup/offbox_progress.go`. Shape: one job registry + one poll endpoint + one client-side renderer, phases declared per job. **Two lessons tonight that any framework must encode:** (1) a terminal state must be **probed, not inferred** — `compose up -d` exits 0 on a crash-loop; (2) a progress source that reports nothing is normal, not broken — restic reports 0 bytes for a whole incremental run, and a bar that sits at 0% is worse than no bar. Also fixes the restart hole: in-memory job state currently vanishes and the card silently disagrees with reality  **2026-07-20 — the first bill for NOT having this arrived, and it was customer-facing.** The samba card's poll (`web/samba_ensure_job.go` + `sharing.html`) mixed a job EDGE and a service LEVEL on one JSON field, and `/sharing` reload-looped at ~1.2 s for every customer with sharing enabled until controller v0.151.0 (`audits/DIAG-sharing-2026-07-20.md`, S-1/S-4). v0.151.0 fixed THAT card's contract only — the framework is still this item. **Third lesson for it to encode, beside the two already listed:** a phase a client answers with a one-shot action must be an EDGE the registry SERVES ONCE, and must never be synthesised from a level; if it can be re-read, it will be re-acted on. |
| R-46 | **[P2] Verification copies need a customer-visible browse surface and an expiry.** v0.147.0 made them *visible* (listed with path/size/date, individually deletable) — but the customer still cannot LOOK INSIDE a verification restore to confirm the file they wanted is really there, which is the entire point of a verification restore, and nothing ever removes them. | S–M | idea | Origin: 2026-07-19 feedback slice 4a, registered as the explicit follow-up to it. Two gaps, deliberately designed together because they are the same object: (a) **the invisible-result gap** — a read-only browse of `backups/offsite-restore/<app>` (the FileBrowser infra stack already exists and already serves scoped roots, so this may be a mount rather than new code); (b) **the disk-lifecycle gap** — auto-expiry after N days with the count/size surfaced before it fires, so a drive is never quietly filled by verification restores nobody remembers taking. Pairs with R-43: a browse surface is also how a customer would discover that a DB-indexed app's files came back but the app still cannot see them |
| R-47 | **[P2-HIGH] The DB replay races the application's own schema repair.** | M | **SHIPPED — controller v0.153.0, 2026-07-20** | Fixed on **BOTH** paths (it was a class defect, not an offsite-only one): offsite `ReconstituteFromOffsite` AND local `RestoreFromRecoveryUnit` now open a **DB-only window** — `appbackup.DBServiceNames` names the database SERVICE from the compose `services:` map, `stacks.Manager.StartStackServices` brings up only that service, the dump is replayed with the application still stopped, and the full start runs only after the replay exits 0. **Fail-closed:** a `.sql` dump with no identifiable DB service refuses BEFORE the first mutation. Every exit from the window (replay failure, DB-only start failure) still does a best-effort full start, so a failed restore never leaves a box with a database and no application. `--clean --if-exists` + `ON_ERROR_STOP=1` untouched — the bug was the window, not the flags. Enablers: `RedeployFromEnv` split into `PersistUnitRedeployConfig` + its unchanged tail; `StackDataProvider.RecreateStackFromUnit` → `RecreateStackDefinitionFromUnit` (the hidden `up -d` inside the old name is what carried the defect locally). **No agent coupling — MinAgent stays 0.90.0.** 19 tests + 3 red-proofs, 23/23 green. Unblocks the capability-map **offsite-restore (PARTIAL)** and **customer-restore (MISSING)** rows, both of which now await only one clean acceptance run. **LIVE-VALIDATED 2026-07-20** against the SAME snapshot that aborted in round 2 (`49e7cb46`): log shows `Starting stack immich services only: [immich-postgres]` → replay rc-0 in 20 s → full start; no `already exists`; immich's own DatabaseService logged **`No schema drift detected`** twice (round 2 left it reporting drift); 11 assets `active`, 4/4 containers healthy. **Golden 0.153.0 baked + published the same day** (sha256 `15fdd191f3c660a6…`) — the first golden carrying all FOUR infra images. Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` (H4) + `felhom-controller/REPORT.md` §4b/§4c |
| R-48 | **[P2-HIGH] Restore controls are separable only by layout — and the difference between them is whether the data comes back.** The offsite restore row renders four buttons plus hint text into an overlapping, unreadable line, and the decisive second step („Teljes visszaállítás indítása") appears ONLY after „…előkészítése" was pressed, with no signposting that a second step exists or that the first one did nothing to live data. | M | idea | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` (finding 1) — this is not theoretical: it is the CAUSE of the round-2 incident.** An operator who had read the code pressed the missing-only button instead of the full restore; the controller log shows `/backup/offbox/reconstitute` was never hit at all. The rule this establishes, worth stating once and applying beyond this page: **two adjacent controls whose difference is "your data comes back" vs "your data cannot come back" must not be distinguishable only by layout.** Direction (ruled in principle, spec rides v0.149): collapse to a single „Visszaállítás…" guided dialog — one intent, visible phases, the escrow-wizard precedent. Pairs with R-45 (the phases are exactly the async-feedback surface) and R-46  **SHIPPED 2026-07-21 — controller v0.154.0** (`3a9d744`). Each app row on `/backups/restore` now carries ONE „Visszaállítás…" entry linking to a per-app wizard at `GET /backups/restore/app?name=<app>`: three intent CARDS each with a consequence sentence (ellenőrzés külön mappába / hiányzó fájlok visszahozása / teljes visszaállítás), a visible phase strip so the sequence is legible *before* the first click, danger styling on the destructive card, and the R-43 double-confirm carried over verbatim with its pair-honesty facts. `deriveWizardStep` is a PURE function of (op running, size-gate flash, scratch ready) — the step is never taken from the request, and a running op outranks a stale `?full_prep=` so no commit button survives into a restore. While ANY op runs every mutation form is suppressed server-side rather than offered and then refused. **No new mutation endpoint** (one GET route; every card posts to the pre-existing `/backup/offbox/*` with unchanged field names and gates) and **no R-45 graft** — the wizard polls the two existing status surfaces as-is. Works with JavaScript disabled. Latent bug fixed on the way: `offboxRedirectTo` hardcoded `"?"` when appending its flash, which against the wizard's `?name=<app>` target would have buried the flash inside the app name. 9 new tests + the Group-B red-proof (trivial always-INTENT impl → all 7 rows red). **Live click-through + one non-destructive Ellenőrzés still PENDING** (rides the operator's floor save). Evidence: `felhom-controller/REPORT.md` §3 (2026-07-21). |
| R-49 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. | S–M | idea | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app |
| R-65 | **Buddy-box backup replication, cross-household — two Felhom boxes in different homes replicate backups to each other.** | L | idea (post-alpha, spike-first, 2026-07-22) | The natural big sibling of R-64: two households each hosting the other's encrypted backup tier. Explicitly **spike-first** — the transport is NOT SMB (R-64's live-share protocol is wrong for backup replication across the internet: no auth story between households, no resumability, cleartext LAN assumptions); candidates to spike: restic rest-server / rclone / syncthing over the existing WG/tailnet plumbing, encryption keyed so the buddy can never read the payload. Sits on top of the offsite tier's FILL/OVERSUB thresholds thinking (R-5 aggregate). Flips: would add a "cross-household buddy replication" capability row (currently unlisted). Pairs with R-64 (same topology, different transport + guarantees) |

## Pre-invite checklist — what stands between here and the first remote tester

Not roadmap items in their own right; the short list the 2026-07-18 rehearsal leaves behind.
Everything here is **remote-doable** — the N100 is packed, and none of it needs hands on the box.

| Action | Owner | Note |
|---|---|---|
| ~~Rebuild the golden → 0.146.0~~ **BAKED + PUBLISHED 2026-07-18; awaiting the operator's two saves** | Viktor (saves) | Golden **0.146.0** baked on the drill VM and published to gitea — `felhom-golden/0.146.0/golden.tar.zst`, **sha256 `4834c703162c5437467a329144b1a523019bf5693ab9d439558be7323587e955`**, 612 696 588 B (584 MB archive). All pass markers green: `Result=success`/`ExecMainStatus=0`, **0** FATAL/exclusions, `docker OK (overlay2)`, **all three mounts included** (rootfs + mp0 `/var/lib/docker` + mp1 `/mnt/sys_drive`), pre-delete **404**, upload **HTTP 201**; controller **0.146.0** confirmed baked in. Integrity round-trip independent of the build host: anonymous `GET | sha256sum` **matches byte-for-byte**, ranged GET **206**, `content-length` matches. Teardown per GL-1: guest 9100 purged, token/script/log shredded in-VM, VM powered off, drill disk reverted to `virgin` exactly-as-found, **token-leak grep = 0**. Log retained `180:/mnt/5_hdd/felhom.eu/drill/bake-0.146.0.log`. Now selectable in the hub dropdown (versions: 0.136.0, 0.143.0, **0.146.0**). **REMAINING = operator, password-gated:** Day-0 manifest Golden → 0.146.0 (Agent stays 0.90.0, MinAgent stays 0.90.0 — the v0.146.0 CHANGELOG declares no new agent coupling) → save; **then** floor → v0.146.0 saved **LAST** |
| **Golden ≥ 0.147.x carries ALL FOUR infra images** | — (next bake) | `build-golden.sh` v2.1.0 (2026-07-19) now derives the pre-pull list from the controller binary it is about to bake (`--print-infra-images`) instead of a hand-maintained copy that had already drifted: `felhom-samba` was never added to it, so every golden so far baked **3 of 4** — which is why enabling Megosztás on a fresh box pulled from the registry with zero feedback. **No golden rebuild for this alone**; it takes effect at the next bake. Until then a fresh box still pulls felhom-samba at enable time, which controller v0.147.0's progress card now at least explains |
| **freemail.hu test-send** | Viktor | The open half of R-4; the gmail half closed on 2026-07-18 under `p=quarantine` |
| **C6 — customer performs a restore, unassisted** | Viktor as customer zero | The one open script step in R-3 and still MISSING as capability evidence. Remote-doable on the reborn box — the dashboard is remote |
| **R-11 rulings** | Viktor | Contact channel, tester agreement, alert thresholds (the R-5 gauge thresholds are still pending a ruling) |

## Absorbed / superseded notes in this folder

- `FOLLOWUP-nas-automount-guest-reboot-reassert.md` — **shipped** (agent v0.84/v0.85, CAMPAIGN-3); keep for history
- `FOLLOWUP-golden-default-controller-tag.md` — verify against current golden flow; close or promote to an item
- `FIX-M18-NOTES.md`, `FIX-M19-NOTES.md`, `DIAGNOSIS-f9-storage-registration-gap-2026-06-14.md` — historical diagnoses; superseded by shipped fixes