felhom.eu/documentation/backlog/ROADMAP.md

# ROADMAP — future features & open work

> **What this is:** the prioritized decision log of planned/open work. Items are *intentions*, not
> claims about live behavior — the capability map (`architecture/00-capability-map.md`) is the only
> place that states what the platform does today.
>
> **Lifecycle:** idea → spiked → spec'd → in-progress → **shipped** (item collapses to a one-liner
> with the version, and the corresponding capability-map row changes status with evidence). Items
> can also be **killed** (keep the one-liner + why — decisions are worth remembering).
>
> **Coupling rule:** every item names the capability-map row(s) it flips. Every map gap row points
> back here by ID. Neither file duplicates the other's content.
>
> **Priorities:** P1 = closed-alpha blocker · P2 = close during alpha · P3 = post-alpha.
> Existing loose notes in this folder (`FOLLOWUP-*`, `FIX-M*`) are absorbed as references below.

---

## P1 — closed-alpha blockers

| ID | Item | Size | Status | Notes / map rows flipped |
|----|------|------|--------|--------------------------|
| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path |
| R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
| R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none` → `p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** |

## P2 — during alpha

> **Sub-rank `P2-HIGH` = close before the first REMOTE tester.** These are the 2026-07-18 N100
> rehearsal's findings (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). They are not P1 — the
> rehearsal proved the product flow works — but each one either misleads the operator, misleads the
> customer, or hides a failure, and all of that gets materially worse the moment the box is somewhere
> you cannot walk over to.

| ID | Item | Size | Status | Notes |
|----|------|------|--------|-------|
| R-39 | **[P2-HIGH] The PBS DR tier can be `applied` and dead at the same time — and nothing notices.** On the reborn N100 the descriptor auto-provisioned and the agent converged `state=applied`, yet `pvesm status` reports `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive` and a direct probe with the stored credential 401s on **every** endpoint including `/version` (WG transport healthy: handshake 9 s, 27.9 ms RTT — so authentication, not ACL scope). Three compounding defects: **(a)** a **mint/consume race** — the hub minted a SECOND token secret at 16:47:52, two minutes *after* the agent applied the first, and `consumed_at` is still NULL; **(b)** the converged state machine will not re-apply, so the box is pinned to a stale secret; **(c)** the agent's 15-minute PBS verify loop **cannot read the credential to detect any of it** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — the non-root agent writes that file through a root sudo wrapper, then reads it directly). | M | **CLOSED 2026-07-21 — PROVEN LIVE (hub 0.68.1 + agent 0.91.2)** | **DIAGNOSIS (2026-07-18, live on the N100 — supersedes the initial hypothesis).** The brief guessed "the re-mint fails to bump the generation". **That is FALSE and no hub fix was shipped:** `store.SetHostDesired` bumps `desired_generation` unconditionally (it went 2→3 on the re-issue), and `web/configs.go`'s `applyPBSDR` is likewise exonerated — its "no re-key, no second secret, no spurious generation bump" comment is accurate, guarded by the `cur != nil && cur.Namespace != ""` early return, and the hub log shows mint #2 came from the **re-issue** path, not from an Edit-tab Save. **The real mechanism is a signal mismatch between the two tiers.** The hub's re-consume signal is *a generation bump + a poke*; the agent's re-apply trigger is *a change in the DESCRIPTOR CONTENT HASH* (`felhom-agent internal/pbsdr/manager.go` ~L235: `if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { return }`). An ep0 credential re-issue re-keys the **secret of an existing token**, so `token_id` and `fingerprint` are unchanged and the descriptor is **byte-identical** — only the side-table `host_pbs_secrets` row rotates. Same hash → the converged agent short-circuits → the fresh secret is never consumed → the box keeps presenting a revoked credential → **401 forever**. Proof in one line: the agent's `consumed-failed.json` carries hash `a4e5424…`, **identical** to the `marker.json` written 2 min before the re-issue. The comment at `hub/internal/web/pbsdr.go:320` asserts the reissue refreshes the descriptor "with the NEW token_id/fingerprint" — that assumption is simply false for this op. **A SECOND, independent defect was found while healing** and is fixed: `configs/felhom-pbs-apply`'s `reconcile` passed `--server` to `pvesm set`, which PVE rejects wholesale as a create-only parameter, so *every* re-apply exited 255 — and because the agent consumes the one-time secret BEFORE calling the wrapper, each re-issue **burned a credential**. Fixed in **agent v0.90.1** (one argv line + red-proof `TestReconcileNeverPassesServerToPvesmSet`); proven live (`pvesm set --server <same>` rejected, without it rc 0). **BOX HEALED 2026-07-18:** after the wrapper hotfix, Viktor's Re-issue click converged in 9 s and the tier went `401/inactive` → **`active`** (token probe 401 → 200) with a real backup landing PBS-side — `felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z`, 9 744 319 312 B, encrypted under the escrowed key. **2026-07-21 — the "cheap half" is ALREADY CLOSED in the field, and the planned artifact publish was CANCELLED as a false signal (operator ruling, same day).** v0.90.1 was to be built, published and deployed as "the PBS wrapper argv fix". Inspection of `9596d5a` shows it changes **zero non-test Go files** — `CHANGELOG.md`, `REPORT.md`, `configs/felhom-pbs-apply` (the fix) and `internal/pbsdr/manager_test.go` (the red-proof); its own commit message states *"the Go binary is unchanged"*. So the 0.90.1 binary is functionally identical to the 0.90.0 in the field. The fix is live anyway by two independent paths: felhom-pve carries the hotfixed wrapper since 2026-07-18 (`args=(--fingerprint "$fp")` at L107, `.bak-20260718-preR39` retained), and **every new install fetches the wrapper unversioned** — `felhom-host-install.sh:1914` `fetch_raw` pulls `configs/felhom-pbs-apply` from `raw/branch/main`, and `9596d5a` is an ancestor of `main`. Publishing 0.90.1 would therefore have delivered no behaviour change, restarted the agent on a production host at a remote site for nothing, and — once the Day-0 manifest was saved to 0.90.1 — advertised that the fix shipped as a versioned artifact when the artifact channel never carried it. **Ruled: leave 0.90.0 published; record the closure here instead.** Evidence: `felhom-controller/REPORT.md` §5 (2026-07-21). **This surfaced a NEW item — see R-50b: a root-owned privileged host artifact is delivered from `main` with no version, so "which wrapper is on this host" is not answerable from any manifest.**  **FLEET FIX SHIPPED 2026-07-21 — hub v0.68.0 + agent v0.91.2.** All three legs closed. **(a) the re-key is finally VISIBLE:** `host_pbs_secrets` gains a monotonic per-host `generation`, advanced by every fresh MINT and by nothing else, stamped into the descriptor as `secret_generation` — the only field a re-key moves, and because `descriptorHash` marshals the parsed struct, the thing that finally re-arms a converged agent. A re-STAGE deliberately does not advance it (same secret, same descriptor content). `omitempty` is load-bearing: emitting a zero would move every pre-existing descriptor's hash at once. *Spec deviation, deliberate:* the brief said to reuse "the new row's id, no schema change" — there is no row id (the table is `host_id PRIMARY KEY`, UPSERTed last-write-wins) and `created_at` collides within a second, so an additive counter column is the only monotonic source. **(b) the agent can finally READ its own credential:** a narrow wrapper `read` verb + exactly one sudoers line + a `pbsdr-read` capability row. Proven live on felhom-pve: `sudo -u felhom-agent sudo -n felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage` → `rc=0`, 37 bytes, empty stderr. **(c) authentication is PROBED:** `pbs.ProbeAuth` (`GET /version` + an `ErrUnauthorized` sentinel) runs on the 15-minute collect path and becomes a loud `auth_failed` that `pbsdrheal` escalates to a fresh mint through the EXISTING damper — closing the loop end to end. Proven live: `level=DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite`. 403 is deliberately NOT unauthorized (a narrow ACL must not be re-keyed forever); a transport error is UNKNOWN, never a rejection (a blip must not burn a credential). Plus a **consumed_at honesty gauge**: an unconsumed secret past a 15-min grace under a box reporting `applied` — the exact July-18 fingerprint, a disagreement no single tier can see — is surfaced with its own event, deliberately as a SURFACE not a heal (minting on top of an unconsumed secret is the R-39(a) race). **A LOAD-BEARING FACT the spec did not flag, checked rather than trusted:** `Apply` bails out if the storage status probe ERRORS, and `adopt` converges WITHOUT consuming when the storage reads active — so the whole fix depended on PVE's 401 behaviour. PVE's `storage_info` wraps `activate_storage`/`$plugin->status` in `eval{}` and leaves the pre-initialised `active => 0`, so a 401 returns **HTTP 200 with `active: 0`, never an API error** — `Apply` correctly falls through to verify → consume → reconcile. **A defect I shipped and caught:** v0.91.0 built the probe seam and `main.go` never called `SetAuthSink`, so the whole leg was INERT and every test still passed (the seam was injected directly) — same class as controller v0.154.0 the day before; fixed in v0.91.1, artifact superseded not overwritten, and v0.91.2 made a healthy probe observable so "no auth_failed" can never again be confused with "never probed". Four red-proofs, all at the assertion level. **CLOSED 2026-07-21 — STOP-2 and STOP-3 both done, chain proven end to end in 13 seconds.** STOP-3 saved (agent 0.91.2 / sha `34d309be…` / **wrapper sha `104db0a4…`** / MinAgent 0.91.2; the agent reports the same wrapper hash so the R-50b drift gauge reads ok). STOP-2: hub `08:39:31Z` fresh mint **generation 0 → 1** + descriptor `"secret_generation": 1` (token_id/fingerprint byte-identical — the re-key shape that used to be invisible) → agent `10:39:34` reads the credential via the wrapper → `10:39:38` **`REJECTED … applied and DEAD` `previous_state=applied`** (the R-39 state, loud for the first time) → `10:39:45` **`one-time token secret consumed`** (NO short-circuit — the line that never appeared on 2026-07-18) → `10:39:45` set-only `reconcile` → `10:39:47` **`converged state=applied`**. Marker hash moved to `afbb3b41…`; `consumed_at` stamped; secret mtime `2026-07-18` → `2026-07-21 10:39:45`; new credential probes **200**; reports trace `applied → auth_failed → applied`; **zero** self-heal escalations, ONE mint, ONE consume, no `consumed-failed.json`. **A NOTE ON THE FIRST ATTEMPT:** the operator initially pressed the **offsite** re-issue (two different Re-issue actions exist) — harmless to PBS-DR, but it rotated the restic password and correctly marked the escrow STALE, so the recovery-code ceremony had to be re-run. Worth naming the surface explicitly in any future runbook step. Evidence: `felhom-agent/REPORT.md`, `felhom.eu/REPORT.md`. *(Superseded — the spec was written and shipped:)* **REMAINING (fleet, needs its own spec — deliberately NOT improvised):** (a) make a fresh unconsumed secret actually un-converge the agent — either `consumed_at` becomes authoritative or the descriptor carries a secret generation/nonce so the hash moves; (b) fix the verify loop's read path — it reads `/etc/pve/priv/storage/<id>.pw` **directly as non-root**, a file it can only ever *write* through the root wrapper (`/etc/pve/priv` is `0700 root:www-data`; sudoers exposes `create|reconcile|grant` and **no read verb**), so the one loop that could catch this is permanently blind; (c) an auth probe in the reconciler/gauge/ceremony-precheck so `applied` can never mean `401`. — **Original finding note:** discovered by CC while collecting Phase-A evidence; not on the brief's finding list, so its P2-HIGH rank is provisional pending Viktor. The severity case: this is the DR tier, the failure is silent, and it would surface first at a real restore. Suggested shape: make `consumed_at` authoritative (a fresh unconsumed secret must un-converge the reconciler), fix the verify loop's read path (read via the same root wrapper that writes it), and make a failing `pvesm status` a LOUD state rather than a skipped datastore. **Blocks the DR-tier map row's candidate upgrade to PROVEN-LIVE — that upgrade is now explicitly WITHDRAWN.** Evidence `pbs-dr-state.txt`, `hub-state.txt` |
| R-30 | **[P2-HIGH] Liveness presence should come from the wait channel, not the report clock.** The box was powered off at the start of the rehearsal, yet the hub carried it as healthy until the staleness threshold expired ~30 min later (`host_stale` 16:05:24 "no report for 30m"; cleared 16:33:24 "was stale for 27m"). The host-delete guard compounds it: RESET refuses while any host row exists, so a stale-but-"Online" host stalls a forced teardown. | M | idea | Direction: derive presence from **Dir-2 long-poll connectedness (~90 s grace)**, decoupled from notification hysteresis (the hysteresis is right for *alerting*, wrong for *presence*); an agent/ep0 analog can follow. Pairs with R-13/R-23 — the transport already exists, this is about believing it. *(Discussed in-session as "R-29"; that number was already taken by the gate-rot item earlier the same day, so it is R-30.)* |
| R-31 | **[P2-HIGH] Offsite provisioning is synchronous with no status affordance.** Save runs the Hetzner sync in-request, so the request can hit the nginx 504 **while succeeding server-side**: the operator cannot tell failed from slow, and a retry races the first attempt. | M | idea | Direction: make it async + a status card, reusing the proven **awaiting-card/poll idiom** (v0.138.0 escrow card). **Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click.** |
| R-32 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. | M | idea | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` |
| R-33 | **[P2-HIGH] Bootstrap pairing-poll spams the customer-visible console.** Each 30 s pairing poll is a separate `felhom-bootstrap.service` run that exits 1, so systemd prints `Failed to start …` on the screen the customer is watching — **52 FAILED lines in ~11 minutes** while nothing was wrong (the box was correctly waiting to be bound). | S | **SHIPPED (scripts v1.21.0, 2026-07-18)** | `run_pairing()` is now a `while true` loop sleeping `POLL_INTERVAL` (30s — the hub-side rate is unchanged) instead of exiting non-zero per poll, so the unit sits in `activating` and systemd prints nothing. Registration split into `register_appliance()`, whose transient failures the loop retries rather than taking the unit down. The journal is quiet without going dark: the wait is logged once on entry (worded so nobody reads it as an error), then on a 10-min heartbeat with elapsed minutes; `410` still exits non-zero on purpose (delivery consumed but no local env is a real crash window). Console banner every 5 min instead of every cycle, single accented spelling, plus the reassurance the rehearsal showed was missing („Ez a képernyő magától frissül"). **The load-bearing half is `TimeoutStartSec=infinity` in the unit** — a `Type=oneshot` ExecStart is killed at `DefaultTimeoutStartSec` (90s), so without it systemd would kill the new wait and `Restart=on-failure` would silently reinstate the exact spam this removes, *after appearing to work for the first three polls*. Verified behaviourally in a container against a stub hub answering 204 five times then delivering: **one log line plus one heartbeat, zero exits between polls**, then a clean fall-through to the direct install and `exit 0` — where the old design produced 5 unit invocations and 5 `Failed to start` console lines. ISOs rebuilt at v1.21.0 and the shipped payload verified byte-identical to the committed source. Evidence `bootstrap-journal.txt`; ship commit `bcdb042` |
| R-5 | Hub: offsite storage visibility — RESTIC box aggregate (v0.64.0) + PBS DR datastore (v0.65.0), each with fill + operator alert; Offsite tabs + dual dashboard gauges | M | **SHIPPED (hub v0.64.0 + v0.65.0 + tenantsync v1.2.0, 2026-07-17)** | Restic: `monitor.OffsiteBoxChecker` (throttled Hetzner GET; fill 80/90% + oversub 2.0×; scope `"pool-box"`). PBS DR (v0.65.0, Option A): read-only `usage` op on the ep0 `felhom-tenantsync` forced command (`df`, no mutation) → `monitor.PBSDRBoxChecker` (fill 80/90%, states ok/unavailable/degraded, scope `"pbsdr-box"`); `/offsite` Restic/PBS-DR tabs + two gauges; hub deploy ⟂ ep0 update (graceful "n/a"). **Both data pipelines PROVEN-LIVE** (restic `0.2% full`; PBS `19.1% full, 7.1 GB of 37.2 GB` after ep0 → v1.2.0). Thresholds pending Viktor (named keys; separate restic/PBS pairs). OPEN: UI render unit-only (password-gated), alert emails unit + red-proof only (both nominal). See hub v0.64.0 + v0.65.0 REPORTs |
| R-6 | **Spike: LAN service discovery from the guest** — SSDP multicast (UDP 1900, DLNA), WSD (Windows discovery), mDNS; host-network vs macvlan; is the customer LXC LAN-bridged in appliance deployments? | M | **spiked (2026-07-18)** | **VERDICT: appliance guest IS LAN-bridged (own DHCP lease on the household /24); multicast discovery works ONLY in the guest netns — guest-direct or Docker `--network host` (SSDP/mDNS/WSD all PASS both ways); the default docker bridge is categorically DEAF to LAN multicast (WSD/mDNS RX FAIL, unicast-publish PASS). Real samba+wsdd on host-net → Windows 11 ProbeMatch + FELHOM-SPIKE renders in Explorer + 445 + authenticated SMB round-trip all PASS; real SSDP `MediaServer:1` advert reaches both LAN clients. → R-7 SMB stack MUST be host-network LAN-bound; R-8 Jellyfin-DLNA plausible if host-network. Caveat: `vmbr0 multicast_snooping=1` worked only because the household router is a live querier — customer LANs w/ snooping+no-querier, and Peti's BYO bridge, are UNTESTED gaps.** **S4b (human leg, the sharpest finding): wsdd makes the box VISIBLE but the Explorer double-click FAILS `0x80070035` — WSD gives no name resolution; the flat `\\FELHOM-SPIKE` resolved by no path. Adding `nmbd` (NetBIOS) fixed it live (flat name resolves + mounts). → R-7 needs smbd+wsdd+nmbd (+avahi/.local for modern clients), not wsdd alone.** Doc: `audits/SPIKE-lan-discovery-2026-07-18.md`. |
| R-7 | SMB **server** share — LAN file sharing as an embedded controller feature („Megosztás") | L | **SHIPPED slice 1 (controller v0.144.0 + `felhom-samba:1.0.0`, 2026-07-18)** | Flips map row *Files from Windows Explorer / Mac Finder (SMB server)* MISSING → **IMPLEMENTED**. Delivered as the FOURTH protected infra stack (host-net, smbd+**nmbd**+wsdd, `bind interfaces only = lo eth0`), NOT a catalog app. „Megosztás" → „Hálózati megosztás": enable + ONE household password (STDIN→smbpasswd, never persisted — only `user_set`), shares table, create flow (new folder under `<storage>/shares/` or an existing folder via a guarded picker). Picker guard = absolute → EvalSymlinks → containment in a registered LIVE root → deny-listed system subtree (derived from `ProtectedHDDPaths`, provably a subset) → is-a-dir, with UNIFORM refusals (no filesystem oracle). Per-share read-only enforced in smb.conf AND as a `:ro` bind. `force user = felhom` → SMB writes land as uid 1000. Live-validated on demo through the real endpoints + a Win11 workstation (445 True, NetBIOS `FELHOM` resolves, write/read byte-compare PASS, **write to a read-only share REFUSED**, WSD ProbeMatch). **Explorer leg PASSED 2026-07-18 (Viktor):** both shares open from the Network view; an Explorer save into the writable share landed as uid 1000 and a write into the read-only share was refused, folder untouched. Slice 1 is fully PROVEN-LIVE. Docs: `controller/sharing.md`; ship report `felhom-controller/REPORT.md`. **Slice 2 remainder:** avahi/`.local` for modern+Apple clients, curated app-folder presets (one-click paperless `consume/`), per-share users + guest access, recycle bin. **Backup gap split out → R-7b.** |
| R-7b | **Share backup EXECUTION** — put share data into the live tier-2 + offsite runs (the design fork reported by R-7 slice 1) | M | **SHIPPED (controller v0.145.0, 2026-07-18)** | **Viktor's ruling: Model B′ — a SIBLING shares source.** New, additive job/leg code reusing the proven primitives (tier-2 mirror seam, restic wrappers, soft-quota/enlargement gate, status recorders) while leaving **every per-app engine path byte-identical** — NOT a synthetic recovery unit (breaks on multi-drive shares, wraps 1 KB of JSON in dump machinery) and NOT engine-loop surgery. The B′ invariant is enforced by test in both tiers, red-proofed. Tier 2 → `RunSharesTier2` (legs grouped by SOURCE drive → `backups/secondary/_shares/<driveKey>/<share>`, payload at `_payload/`, layout marker LAST). Tier 3 → `runOffboxSharesLeg`: ONE extra `restic backup --tag felhom-offbox --tag _shares` placed after the app loop and BEFORE retention, so `forget --group-by host,tags` covers the new group with no flag change; a quota-blocked push degrades to the **manifest only, never to nothing**. Restore → „Megosztások" on `/backups/restore`: scratch, then a missing-only merge whose every destination is PREFIX-ASSERTED against live storage roots, definitions merged existing-wins, then `ReconcileSamba`, then the credential. The **payload** (`_shares-manifest.json` + a best-effort secret-bearing `passdb.tar`) is what makes DR return files + configuration + password rather than loose bytes. **Fold-in: samba joins the liveness set** — `EffectiveProtected` adds the CONTAINER `felhom-samba` exactly while sharing is on. **FULLY PROVEN-LIVE on demo (2026-07-18), all four legs.** (1) tier-2: real `/api/backup/tier2` trigger → `_shares` tree + marker + payload on the cross-drive target, mirrored file md5-identical, payload 0600 preserved. (2) offsite: Viktor's manual run 12:18:16Z → snapshot **`e0b9d723`** (tags `felhom-offbox,_shares`) with the payload dir + both share folders; a second run via the „Távoli mentés" button → **`4e2b15ec`**, containing `_shares-manifest.json` (418 B) AND `passdb.tar` (855 040 B), both 0600, share files with uid 1000 preserved. (3) restore round-trip: probe file + the `dokumentumok` DEFINITION deleted via the real endpoints, then „Megosztások" restore + place → `1 file(s), 1 definition(s) re-added, 1 kept, 0 refused, credential=true`; probe back md5-identical, the two pre-existing files NOT overwritten (missing-only proven on live data), definition back with its ORIGINAL flags and created_at, `smb.conf` re-rendered, `filmek` untouched. (4) liveness: samba stopped → `health_critical` pushed and hub-accepted (200) → self-healed. Remaining human leg: SMB positive auth with the real household password (never persisted by design). **Correction:** an earlier revision of this row and of the ship REPORT wrongly claimed the demo box had no offsite target — the verification read a guessed settings key (`offbox_target`) instead of the real one (`offbox`); root cause dissected in REPORT §7b. Findings: the reserved-name assumption was FALSE (`nbNameRe` accepted „_shares" as a share name — now refused); the alert/e-mail pipeline needed NO change and adds no new event type. Docs: `controller/sharing.md`; ship report `felhom-controller/REPORT.md`. |
| R-8 | DLNA (**gate input now exists — R-6 spiked 2026-07-18: SSDP reaches LAN clients from host-net**): validate Jellyfin's built-in DLNA server first; only add minidlna to the catalog if Jellyfin-DLNA fails | S | idea (unblocked) | Don't add catalog weight before proving the cheap path. **R-6 confirmed the cheap path is physically viable — Jellyfin DLNA must run host-network (same multicast constraint as R-7)** |
| R-9 | Uninstaller trio (from 07-15 Peti session): cluster-aware `felhom_guests` guard (node-local `pct list` deletes cluster-wide pveum objects); saferemove detection + time estimate + opt-in `--quick-remove` (never mutate `storage.cfg`); smarter `restore_storage` default for BYO clusters (shared storage, not local-lvm) | M | idea | Second item's rejected alternative (temp-disable-and-restore) stays rejected — crash window silently downgrades cluster wipe policy |
| R-10 | T-6E-1: DB-dump dir-fsync asymmetry (LOW, confirmed in 6E) | XS | idea | One-line hardening; batch with the next controller task |
| R-11 | Tester-facing one-pager: what the box does, known limitations, how to report | S | **RULED 2026-07-21 (channel); doc is the architect's** | Pairs with R-3. ****operator ruling 2026-07-21**: the reporting channel is DIRECT MESSENGER to the operator** — no group, no tracker, no form. Rationale: the expected tester count at first invite is ~0–few, and a channel nobody is on is worse than a channel everyone already uses. **The one-pager and the tester agreement are the ARCHITECT's deliverable, not CC's** — CC does not write them; it may be asked to fact-check claims in them against the code. Revisit the channel only if tester count makes direct messages unmanageable |
| R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. **Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17):** the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above |
| R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/<id>` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. |
| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |
| R-34 | **Backup data lifecycle management.** An "inactive backups" section on „Távoli mentés": apps that have snapshots but no active backup — **disabled OR uninstalled** — listed with name / size / last snapshot / restorable, plus an explicit **double-confirmed per-app delete** via `restic forget --tag` + nightly prune. | M | idea | **RULING: the offsite toggle NEVER offers deletion — policy and destruction stay decoupled.** Turning backups off must never be a data-destroying act, and deletion must never hide behind a toggle. Origin: 2026-07-18 rehearsal. Pairs with R-32 (that one is the operator's view of dead bytes; this one is the customer's) |
| R-35 | **Config-apply should not end the customer's session.** The offsite config push bumped `config_version` 10→11 at 16:54:58 and the controller self-restarted (container `StartedAt` 16:54:59Z, back up 16:55:02); in-memory sessions died with it and **customer zero was force-logged-out mid-flow**. | S | idea | Direction: **hot-apply the offbox target** (no restart for a config the running process can adopt), or **persist sessions** across restart. The restart itself is by design — the collateral is not. Evidence `controller-log-full.txt` |
| R-36 | **Post-RESET re-enroll leaves offsite "enabled but unprovisioned" — silently.** The hub knows the state and says nothing on the customer page. | S | **SHIPPED (hub v0.67.0, 2026-07-18)** | Both halves delivered. **(1) The warning:** the customer page now names the state and the fix — enabled-but-unprovisioned raises an amber banner saying provisioning is *Save*-triggered (press Save once, then verify), reusing the exact `enabled && type == ""` predicate the offsite re-issue handler already refuses on. **(2) The related sub-item, also done:** the self-bind link is now **auto-minted at customer creation AND at RESET completion**, so the console banner's „e-mailben kapott link" is already true instead of true-once-the-operator-remembers. Extracting the shared `mintAndSendSelfBindLink` core keeps the button and the auto-mint callers on the same F1/F2 honesty rules, and the auto-mint never fails the operation it rides on. **Gap found and closed while wiring it:** `PurgeCustomerResetDBState` does NOT clear `selfbind_tokens`, so a link minted BEFORE a reset would have stayed live across it — the skip paths now clear stale tokens, giving the invariant "after auto-mint, the only live link is one we just issued, or none". Tests assert the banner is ABSENT in all three nominal cases too; red-proofed. — Original analysis: Source-cited behaviour, confirmed live in the rehearsal: **provisioning is Save-triggered** (`configs.go` `applyOffsite`) — which also answers S6's open question — and the re-enroll auto-re-issue **correctly** skips unprovisioned targets (`handler.go`). So nothing is broken; the gap is that nobody is told. Direction: flash it on the customer page. **Interim: an R-3 step.** **Related sub-item:** auto-mint the **self-bind link on customer create/RESET**, so the console banner's „e-mailben kapott link" is always already true instead of true-once-the-operator-remembers |
| R-27c | **Customer self-bind, slice 2 — console-passphrase bind.** Viktor's direction: bind using a passphrase shown on the box console, alongside (not instead of) the emailed capability link. | M | idea | **Security constraints from the session ruling, all load-bearing:** passphrase **issued at customer creation**; the global-lookup endpoint must be **spray-hardened** — per-appliance **and** per-IP caps, constant-time comparison, a **single generic failure** (no oracle), alerting on abuse; an **accent-free wordlist** (console keymaps are not Hungarian); the **web capability-link path is RETAINED**; **claim-by-email is RETAINED** as the delivery-channel proof. **Also under this item:** the self-bind email gains the **public universal-ISO download link + two-line instructions** (the DIY case). **Secret-bearing per-customer ISOs are ruled OUT.** Sibling of R-27b (second-box flow) — different axis, both build on the same `/bind/` page |
| R-50 | **[P2-HIGH] Island-bridge control plane — make controller↔agent independent of the LAN.** The agent's `localapi` binds a **LAN literal** (`listen_addr`) and the guest dials that same literal from `bootstrap.json`. Move both onto a **host-internal bridge with a fixed, private address** that no router, DHCP lease, or site move can invalidate, so the control plane is location-independent and golden-image-stable. | M | **SHIPPED 2026-07-25 — fleet-migrated (agent v0.96.0 + host-install v1.19.0)** | **DONE (2026-07-25). Both demo boxes migrated to the island, apps served throughout (0 container restarts); island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), hub reports 0.96.0.** Fresh appliance installs are born on the island (host-install v1.19.0 default; agent auto-attaches `net1` — A4 proven live). Migration procedure: `runbooks/RUNBOOK-island-migration.md`. **One-host fleet only; the 2-node cluster (Peti) is R-74 (SDN/bridge parity), a separate supervised runbook.** ── Spike/impl detail (superseded, kept for provenance): **GO (2026-07-25 PM, `audits/SPIKE-island-bridge-2026-07-25.md`): validated end-to-end on a real nested-PVE drill (`drill-r50`, qm300 on the t740, installed via the v1.25.0 ISO through the actual day-0).** All probes PASS: `vmbr9` portless island bridge (vmbr0 untouched); guest island NIC `eth1 169.254.253.2/30` hot-added (LAN leg undisturbed); **F1 replay = the money shot** — LAN moved `.176→.200` with `listen_addr` on the island → agent stays `active`, control plane HTTP 200; the LAN-literal contrast reproduced the 2026-07-20 bug verbatim (`localapi: bind 192.168.0.176:8443` → daemon exit 1); **dnsmasq trap CONFIRMED LIVE and its `lan_resolver.host_ip=LAN IP` fix PROVEN**; **pin address-independent** (served leaf SHA-256 unchanged, HTTP 200 over the island — no cert re-issue); **survival matrix** (agent restart / guest reboot / host COLD reboot) all return the control plane on the island with zero intervention. **Method caveat:** probes drove the runtime chain via manual config edits — the *provisioning* path (host-install writing the island config + golden-bake bootstrap template) is the impl task, now de-risked. **Next: write the Phase A/B/C production spec (drill-proven → demo → Peti SDN).** — Prior (superseded): **2026-07-25 spike: the drill environment (qm300 / `demo-vm-felhom-2f4b00`) is GONE** (`qm list` empty; only guest 9201 remains), so per the hard drill-only rule the empirical probes (bridge create / NIC hot-add / island bind / **F1 replay** / survival) were NOT run — production was left untouched, **no live GO/NO-GO**. The **source-grounded** half IS done: recommend **link-local `169.254.253.1/30`↔`.2/30`** (structurally uncollidable vs LAN); the F1 literal is baked in TWO places to move atomically (`config.go:229` + `felhom-host-install.sh:2226` for the bind, `provision/backhalf.go:129`/`bootstrap.json` for the guest dial); **NEW dnsmasq trap CONFIRMED in source** — `LANResolverConfig.WithDefaults` (`config.go:208–210`) derives the DNS listen-addr from `listen_addr`, so moving the bind to the island silently kills LAN DNS → the spec MUST set `lan_resolver.host_ip = LAN IP` explicitly; pin is leaf-DER-based (address-independent → no cert re-issue expected); provisioning inventory + cluster-parity (SDN on Peti's 2 nodes) + an implementation skeleton recorded. **Empirical attempt 2 (2026-07-25 PM, on the t740 per the operator's drill-host ruling): STILL BLOCKED — there is NO drill VM on the t740** (`qm list` empty; it is a real demo node running only its own guest 9201). The only drill artifact is a stale golden-bake `drill.qcow2` on the production DooPlex node (off, nested guest purged). The ruling ("drill+build VMs on the HP from now on") is forward-looking and not yet realized. **Operator decision: they will provision a nested-PVE drill VM on the t740 (agent + nested guest); this spike re-runs then.** GO/NO-GO PENDING. **Remaining: a provisioned t740 drill VM to validate probes P2–P7 before the production spec.** Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F1), where this failed for real. The demo box moved to a remote site, DHCP handed it `.147` instead of `.162`, and the agent then **could not start at all** — `bind: cannot assign requested address`, systemd gave up after 4 retries — taking storage, PBS backup, quiesce, restore-test and DR down for as long as nobody noticed. Mitigated for that window by pinning `vmbr0` static back to `.162`; that is a **window mitigation, not the fix** — it still depends on the site's subnet being `192.168.0.0/24` and free at that address. **Spike-first is mandatory:** validate end-to-end on the drill environment (agent bind + guest dial + TLS SAN/pin + reinstall/golden survival + the bootstrap-config migration for already-deployed boxes) BEFORE any production spec. **Pin fact (verified 2026-07-20, `agentapi/client.go` L105-129 — supersedes the earlier "the SAN set must cover the new address" note in this entry, which was wrong):** the controller-to-agent leg sets `InsecureSkipVerify: true` and replaces chain verification with a custom `VerifyPeerCertificate` that does a raw **SHA-256 match on the leaf DER** against the bootstrap fingerprint. Hostname/SAN therefore never enters verification on this leg, so moving the agent listen address most likely needs **no cert re-issuance** - only the endpoint the guest dials. The spike must still confirm this empirically rather than trust the read. Flips a future "box survives a site/network change" map row |
| R-50b | **[P2] A root-owned privileged host artifact is delivered unversioned from `main` — "which wrapper is on this host?" is unanswerable.** `configs/felhom-pbs-apply` installs to `/usr/local/sbin/felhom-pbs-apply` (0755 root:root) and is the pinned sudoers vector for `create\|reconcile\|grant` against `/etc/pve/priv/storage`. It is fetched by `felhom-host-install.sh:1914` via `fetch_raw`, which hits `raw/branch/main/<path>` — **no tag, no pin, no checksum, and no record in the Day-0 artifact manifest**, unlike the agent binary (sha256-vouched) and the golden image. Three consequences: (1) two hosts installed a week apart can carry different privileged wrapper code while both reporting the same agent version; (2) a host hotfixed in place (felhom-pve, 2026-07-18) is indistinguishable from one that fetched the same content — the fleet has no inventory of it; (3) an accidental push to `main` reaches the next install of every host with no review gate between commit and root-owned deployment. | S–M | **(a) SHIPPED 2026-07-21; (b)/(c) open** | **Surfaced 2026-07-21 while stopping the R-39 v0.90.1 publish** (`felhom-controller/REPORT.md` §5): the publish was cancelled precisely because the version number would have claimed to carry a fix that in fact rides this unversioned channel. Candidate shapes, in increasing cost: (a) record the wrapper's sha256 in the Day-0 artifact manifest beside the agent binary and have the agent report the installed file's hash, so drift is at least *visible*; (b) `fetch_raw` takes a pinned ref (tag or commit) supplied by the manifest rather than `main`; (c) the wrapper becomes a published generic-registry artifact with the same gate ladder as the agent binary. **(a) is the cheap honest first step and would have caught this class already.** Pairs with R-39 (whose remaining fleet half is specced separately)  **(a) SHIPPED 2026-07-21 — hub v0.68.0 + agent v0.91.2.** `ArtifactManifest.WrapperSHA256` + an operator field; agents report the installed wrapper's sha256 each cycle and the host page surfaces a mismatch. **An unknown on EITHER side reads as quiet, never as drift** — lighting every host amber on rollout day is how a warning becomes background noise. Live confirmation of exactly the problem: felhom-pve's July-18 in-place hotfix hashed `2888f2ea…`, matching **no commit anyone could name**; it now reports `104db0a4…` against a vouchable manifest value. **(b)/(c) REMAIN OPEN:** the wrapper is still fetched unversioned from `raw/branch/main` — this makes drift *visible*, it does not fix the channel. Also recorded: the 0440 sudoers file is not agent-readable, so its drift stays invisible. |
| R-51 | **Dead-primary alerting — a multi-container app whose MAIN container is dead must alert.** Aggregation currently classifies such a stack `unhealthy`, and `IsDownState` deliberately excludes `unhealthy`, so nothing fires. | S | **SHIPPED 2026-07-21 — controller v0.156.0** | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F4). Observed live: `immich-server` was `Exited` for **18 h** with the app 100 % unreachable, and the box produced **no** dead-app banner and **no** `app_start_failed` hub event — while single-container Calibre-Web, down for the same reason, alerted correctly within 90 s. **Constraint (load-bearing): do NOT simply fold `unhealthy` into down.** That exclusion is deliberate (`stacks/manager.go` fix-3, `downstate_test.go`) and reverting it reintroduces the flapping it was added to stop. Direction: distinguish *member-container-exited* from *healthcheck-failing* in the aggregation, and treat a dead primary as down | **SHIPPED 2026-07-21 (controller v0.156.0).** **The diagnosis in this row was WRONG at the source and is corrected here:** aggregation did NOT classify the stack `unhealthy`. `aggregateState`'s final branch returned `StateRunning` for any mix of running and stopped members — the comment said "report as running (partial)" — so the stack read as RUNNING and `IsDownState` had nothing to fire on. The `unhealthy` exclusion was never involved, and the constraint it protects was therefore never in tension with the fix. New `StateDegraded`: a DOWN member whose docker restart policy is `always`/`unless-stopped` means docker was supposed to be keeping it up, so the stack is degraded (a down state, alerting through the EXISTING banner + `app_start_failed` path, unchanged); `no`/`on-failure` is a finished one-shot init/migrate container and stays benign. An UNREADABLE policy counts as supervised — fail-CLOSED, deliberately the opposite of `IsDownState`'s fail-open, because there the *state* is ambiguous while here a member is known dead and only the excuse is missing (P2 census 2026-07-21: all 53 catalog templates / 78 services are `unless-stopped`, zero one-shot containers exist today). `IsDownState` gained `degraded` and NOTHING else — the `unhealthy`/`restarting`/`paused`/`unknown` exclusions are byte-identical and `downstate_test.go` is untouched and green. Red-proof: the mix branch reverted to `return StateRunning` makes the immich fixture and both production-path tests fail with `"running"`. Live leg (STOP-1) pending. Evidence: `felhom-controller/REPORT.md` (2026-07-21)  **PROVEN LIVE 2026-07-21 (STOP-1, operator-present).** On guest 9201, `docker stop immich-server` at **12:50:40 CEST** (policy `unless-stopped`, three helpers left running — the exact F4 shape). **12:50:53 — 13 seconds later — the stack read `degraded`** where it read `running` for 18 hours on 2026-07-20. `10:51:11Z` **exactly ONE** `app_start_failed (warn) — Telepített alkalmazás nem fut: Immich` (single-fire verified by count, not by eye). Dashboard rendered the banner *Telepített alkalmazás nem fut: Immich (degraded)* plus the new „Részlegesen leállt" state label. `docker start` at 12:51:37 → `running` by 12:51:52 and the banner **self-cleared** (state-based, as designed). Note the banner appends the raw state in English — `(degraded)` — which is pre-existing behaviour, not introduced here, but now more visible |
| R-52 | **Boot desired-state reconciliation — a `deployed: true` app should be running after boot.** The controller *reports* deployed-but-stopped apps (30 s `deadapp-check`) but never starts them, so an app that misses its boot start stays down until a human notices. | M | **SHIPPED 2026-07-21 — controller v0.156.0** | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F5). Observed live: the pre-transport shutdown left `immich-server` and `calibre-web` `Exited`; **10 sibling containers came back and those two did not**, and they were still down ~18 h later. **Includes root-causing why `restart: unless-stopped` did not resurrect them** — both were stopped ~25 s before power-off, so Docker most likely recorded them as user-stopped; that hypothesis is untested because the guest journal is volatile and the controller's own logs were rotated by the container recreate. Direction: a bounded start-once reconciliation (N attempts, reusing the existing boot grace), never a restart loop. Pairs with R-51 — that one is the *alarm*, this one is the *recovery* | **SHIPPED 2026-07-21 (controller v0.156.0), `internal/bootrecon`.** One bounded start-once sweep at controller startup: every deployed, non-protected, not-mid-deploy stack that still HAS containers and is down gets `StartStack`, at most **2 attempts 30 s apart**, then it stops and the alarm owns the problem. Never a restart loop. The whole sweep (5 s settle + one 30 s gap) fits inside the existing 90 s `deadAppBootGrace`, so a successful recovery never alerts and a failed one alerts honestly — asserted by a test rather than left to a comment. **The safety argument is the container gate:** the UI's Stop is `compose down`, which REMOVES the containers, while an interrupted boot leaves them behind as `Exited` — so "deployed, has containers, and they are down" is exactly the boot-orphan signature, and a zero-container stack is never touched. Red-proof: dropping that gate makes the user-stopped app get started, which is the one thing this must never do. **P1 (the `unless-stopped` root-cause probe) is deliberately NOT what this shipped on** — the reconciliation is correct whether or not Docker recorded those two containers as user-stopped, and the probe is recorded as an open question rather than a blocker. Live leg (STOP-1) pending. Evidence: `felhom-controller/REPORT.md` (2026-07-21)  **PROVEN LIVE 2026-07-21 (STOP-1, operator-present), and it answered P1 for free.** Fixture on 9201: `docker stop` on bookstack (2 containers, left in place) + calibre-web, and a UI Stop on immich (`compose down` → 0 containers); then `pct reboot 9201`. Result: `10:53:22Z [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [bookstack] — up to 2 attempt(s)` → `10:53:28Z attempt 1/2: started "bookstack" (took 6.0s)` → `complete: 1 app(s) recovered in 1 attempt(s)`, and **ZERO `app_start_failed`** — a successful recovery inside the boot grace is silent, exactly as designed. **P1 IS NOW ANSWERED, and the F5 hypothesis is CONFIRMED:** bookstack carries `restart=unless-stopped`, the Docker daemon came up at ~10:53:15Z, and the container's `StartedAt` is **`10:53:28.05Z` — the exact moment `bootrecon`'s `StartStack` returned**. Docker's own restart policy did NOT resurrect it; a container stopped before shutdown is recorded user-stopped and stays down. Only R-52 brought it back. **BUT the end-to-end "a deliberate Stop survives a reboot" property is NOT true today, for a reason outside R-52 — see R-55.** R-52's own gate is correct and was observed to be: immich (0 containers) was never a candidate |
| R-54 | **[P2-HIGH] The guest's DHCP client is unsupervised — its death takes the box off the internet 1-2 hours later, invisibly.** ifupdown starts `dhclient` once at guest boot and nothing restarts it. | S-M | **SHIPPED 2026-07-21 — agent v0.92.1** | Origin: `audits/INCIDENT-guest-dhclient-killed-2026-07-20.md` §5 "OPEN RISK" — this row closes it. On 2026-07-20 a cleanup step killed guest 9201's dhclient (visible in the HOST's pid namespace; §4's `/proc/<pid>/cgroup` rule exists because of it). **The guest then kept working for another ~80 minutes on its unexpired lease**; only at expiry did the address and default route vanish, taking the Cloudflare tunnel, hub reports, catalog sync and the controller→agent channel with them — 1h15m outage, and every observable signal said healthy for the first 80 minutes. **The design consequence: liveness of the DHCP client is itself a probe.** `internal/guestnet` flags a DHCP guest unhealthy on `pgrep -x dhclient` alone, while the lease is still live — waiting for the IP to disappear is waiting out precisely that silent window (red-proof: reverting to IP-presence-only makes the July-20 fixture report **healthy** with zero heals). Four fixed-shape `pct exec` probes (address / default route / `/etc/network/interfaces` mode / client liveness, parsers pinned to output captured live from 9201), the incident's restored invocation as the heal, verbatim, and dampers throughout: two CONSECUTIVE bad probes, ≥10 min between heals per guest, ≤3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on a static guest (dhclient must never fight a static config — reported loudly and left to **R-50**, which is where option 2 of the incident's three choices belongs), on an unknown mode, on an unprobeable guest, or on an ownership-unproven guest list (source is `ListLXC` ∩ the felhom pool, audit A1). Host-tier by necessity: a guest with no default route cannot repair its own default route. **A live finding during deployment:** the first sweep on felhom-pve logged `dhclient liveness probe failed: sudo: a password is required` and reported `state=unknown` — fail-safe, but blind. TASK-D assumed no sudoers change was needed; three of the four probes had no grant. `FELHOM_GUESTNET` + four `guestnet-*` capability rows shipped in v0.92.1 (v0.92.0 superseded, do not vouch). **Healthy cycle PROVEN LIVE 2026-07-21** on felhom-pve: caps `68/68 ok, degraded=0` and `level=DEBUG guestnet: guest network healthy vmid=9201 mode=dhcp has_route=true dhclient_alive=true`. The HEAL leg (STOP-2, a deliberate replay of the incident) is operator-present and pending. Note the guest and the host still differ (host static since the F1 mitigation, guest DHCP) — choosing one for both remains **R-50**'s call, not this row's. Evidence: `felhom-agent/REPORT.md` (2026-07-21)  **PROVEN LIVE 2026-07-21 (STOP-2, operator-present) — a deliberate replay of the incident.** `kill -9` on guest 9201's dhclient at **12:43:18 CEST** (pid 336708, `cgroup=0::/lxc/9201/ns/.lxc` verified BEFORE the kill — the incident's own §4 rule, applied). **12:44:15 — 57 seconds later — detected on process liveness alone**, with the address and default route still perfectly present and 83 minutes of lease remaining: `unhealthy (first bad probe — not acting yet) bad_probes=1 required=2`. **12:45:15** second consecutive bad probe → `healing`, `cmd="pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0"` (the incident's invocation, logged verbatim before it ran) → **12:45:18 `guest network healed` ip=192.168.0.104 has_route=true dhclient_alive=true heals_last_hour=1**. Healed **120 s** after the kill — ~80 minutes before the outage would have begun. **The strongest evidence is what did not happen:** `cloudflared` stayed `Up 29 hours`, DNS resolved, the hub answered 302 in 0.16 s and the public URL 302 in 0.25 s. The 2026-07-20 outage was prevented, not merely detected. Exactly ONE heal, zero re-heals. **The damper then proved itself against a REAL transient, unplanned:** the STOP-1 guest reboot at 12:53:16 produced `unhealthy … no IPv4 address on eth0 bad_probes=1` — **no action** — then `guest network recovered` on the next cycle. A single blip really is not a diagnosis, and a booting guest is not injected with a dhclient. **Negative leg run in full:** 12:45:18 → 13:16:15 = **30 healthy Debug cycles, 0 heals, 0 WARN, 0 ERROR**. **Confirmed HUB-SIDE (read-only query of the hub store, after STOP-3):** the `guest_net` stanza arrives complete, and the report history traces the whole session — `10:30:27Z` agent 0.92.0 `state=unknown` (the sudoers-blind window, reported honestly rather than as a false healthy — independent proof that superseding 0.92.0 was right), then `10:48:12Z` 0.92.1 `healthy` carrying `last_heal_at=10:45:12Z heals_last_hour=1`. **Known limitation:** the damping state is IN-MEMORY, so an agent restart resets the heal counters — the ≥10 min / ≤3 per hour ceilings hold within one agent lifetime, not across restarts (a restart does re-arm the 3-minute settle window, so the practical exposure is small). Do not quote the ceiling as a hard guarantee |
| R-55 | **[P2] A customer's deliberate Stop does NOT survive a guest reboot for any drive-backed app** — the boot bind gate recreated and started it unconditionally. | S | **SHIPPED 2026-07-21 — controller v0.157.0** | Surfaced by STOP-1's R-52 leg, which was designed to prove the opposite property and instead proved it is not true end to end. `internal/web/intermediary.go`'s boot gate (`[gate] boot <id>: live bind confirmed — recreating drive-backed app <x>`) selects on `cfg.Deployed && HDD_PATH` alone and then runs `StopStack` + **`StartStack`** on every match; the stack's `state` is passed in for the log line and is never consulted. So after `pct reboot 9201`, calibre-web and immich came back **running** at `10:53:18Z` even though immich had been stopped from the UI seconds earlier (`compose down`, 0 containers) — while bookstack, which is NOT drive-backed, correctly fell through to R-52. **R-52 is not implicated:** its own gate behaved exactly as specified (immich, with zero containers, was never a candidate), and this behaviour predates it. Two consequences worth ruling on: (a) a customer who stops a drive-backed app finds it running again after any reboot, which is the same class of dishonesty as offering a control that cannot work; (b) **R-52's practical scope on a real box is narrower than the brief assumed** — the gate reaches every drive-backed app first, so the boot-orphan sweep only ever sees the rest. Likely shape: the gate should recreate only apps that were RUNNING (or `deploying`) at shutdown, which is the same running/stopped distinction R-52 already draws, rather than every deployed app. Needs a ruling, not a drive-by change — the gate exists to fix a real bind-ordering bug and must keep doing that | **SHIPPED 2026-07-21 — controller v0.157.0, deployed to 9201.** The ruling that shaped it: an unconditional recreate could drown the node when apps were deliberately stopped to free resources for others — so honouring Stop is not only about honesty, it is about not fighting the customer's own capacity management. `shouldRecreateOnBoot` now additionally requires `len(Stack.Containers) > 0` — **R-52's `existing-Exited vs absent` predicate (`bootrecon.isBootOrphan`), translated to this gate**: a UI Stop is `compose down` and REMOVES the containers, while a guest that went down under a running app leaves them behind, and docker's records survive the reboot. **Spike-lite outcome: a truthful running-at-shutdown source already existed and nothing was invented** — `AppConfig` carries no run-state (`Deployed` is a deploy-lifecycle flag that stays true across a Stop), but container presence is durable, already collected via `docker ps -a`, and already trusted by R-52. **Container STATE is still deliberately NOT a filter** — that half of the original design is load-bearing (a `State != stopped` filter misses an app docker has not restarted yet, or one stuck `Exited` on a create-time bind failure with `RestartCount=0`, which is the bug the boot-id path exists to fix). `TestShouldRecreateOnBoot` now pins both axes at once; they pull in opposite directions, which is the difficulty of this gate. **Ordering trap handled:** the evidence is sampled into the snapshot BEFORE any recreate, because `recreate`'s own `StopStack` destroys it. **No regression to the drive-absent gate:** apps it stopped are also at zero containers and are now skipped here, correctly — they are restored by `ReconcileDriveGates`' `Return` branch from `StoragePath.StoppedStacks` on the same loop tick. Honoured Stops are counted and logged separately (INFO `left stopped …`) so an intended outcome never fires a WARN. **Red-proof:** dropping `hasContainers` makes `TestRecreateDriveBackedApps_HonoursCustomerStop` fail with `recreated=[romm immich]` — the live defect by name. **NO new privileged verb** — `StopStack`/`StartStack` are plain `docker compose` execs in the controller's own container; the boot path's only agent calls are read-only. **PROVEN LIVE 2026-07-21 (operator-present, TASK-F Part 3).** Fixture on 9201: immich (drive-backed, 4 containers) STOPPED through the real UI endpoint `POST /api/stacks/immich/stop` → `compose down` → **0 containers**; calibre-web (drive-backed) and bookstack (NOT drive-backed) left running. Then `pct reboot 9201` at 14:29:14Z. Result, from the gate's own log:

> `14:29:28 [gate] boot 1784525102-11906045: live bind confirmed — recreating drive-backed app calibre-web (state=stopped) onto /mnt/felhom-drives/hdd_1`
> `14:29:29 [gate] boot 1784525102-11906045: 1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose`

**immich is absent from the recreate list and came back STOPPED (0 containers)** — on 2026-07-21 before this fix, the identical fixture brought it back RUNNING. calibre-web recreated, bookstack back, `[bootrecon] no boot-orphaned apps` (consistent — nothing was left orphaned for it to adopt), **ZERO alerts**, whole convergence ~15 s from reboot to steady state. The `left stopped` INFO line fired in production for the first time, so the honoured path is observable rather than silent |
| R-56 | **[P3] Apps do not say how technical they are, so a beginner can be ambushed by a config-heavy one.** The catalog presents every app as equally approachable — one Telepítés button, the same Hungarian copy — but they are not. Glance needs a hand-written `glance.yml` before it does anything; some apps need a reverse-proxy or API concept to configure; others genuinely are install-and-use. A tester who picks the wrong first app concludes the PRODUCT is broken, not that they picked an advanced app. | S | **idea (filed 2026-07-21)** | Origin: TASK-E Part 3 — **filed, deliberately not implemented**. Shape: a `difficulty:` field in `.felhom.yml` (`kezdő` / `haladó` / `technikás`) surfaced as a catalog-card badge and repeated on the deploy screen. Cheap and incremental: one optional metadata field plus a badge, classifiable app-by-app with no migration — an app with no `difficulty:` simply shows no badge. **This is the constructive half of the glance ruling**: glance STAYS in the catalog (operator ruling 2026-07-21 — it is a legitimate app, not a broken one; its missing seeded `glance.yml` is a known pre-existing finding), and the honest fix is to LABEL it rather than hide it. Pairs with R-41: that gate proves an app CAN still deploy; this field tells a customer whether THEY should be the one deploying it. **Badge plumbing is ALREADY BUILT (controller v0.158.0)** — `web.MetaBadge` + the `meta_badge` template partial + the `lifecycleBadge` funcmap entry were written generic for exactly this: a `difficultyBadge` funcmap function returning the same `*MetaBadge`, plus a `difficulty:` field on `stacks.Metadata`, is the whole remaining job. No new markup, no new CSS. Re-sized accordingly |
| R-57 | **An app can be withdrawn from the catalog without orphaning the customers already running it** — `.felhom.yml` `lifecycle: available / hidden / abandoned`. | S | **SHIPPED 2026-07-21 — controller v0.158.0 (+ v0.158.1 fix), LIVE-PROVEN** | **Motivating case: plant-it.** Earlier the same day it was withdrawn by moving its directory to `retired/` — which un-offers the app but ALSO makes the controller's orphan detector see the template as GONE for anyone running it, flagging their working install `Elavult` and offering a Törlés button. Withdrawing an app must never take a working app away from a customer, so the directory move was replaced by metadata. **Operator requirements, verbatim (ruling 2026-07-21):** states `available` / `hidden` / `abandoned`; abandoned apps are NOT offered to new installs (no badge-but-installable middle state); deployed instances of hidden/abandoned apps keep full function; an abandoned app shows a permanent notice that *„Az alkalmazás fejlesztője felhagyott a fejlesztéssel. A telepített verzió továbbra is használható, de frissítések és biztonsági javítások már nem érkeznek hozzá."* **Design points that matter beyond this feature:** (a) the deploy gate is server-side and fail-CLOSED before any mutation — hiding a button is not a gate, and a stale link or direct POST must be refused; (b) an unknown lifecycle value fails OPEN (→ available + one WARN), deliberately opposite, because a typo or a state from a newer catalog must never pull a working app out of every customer's list — both read the same `EffectiveLifecycle`, so they cannot disagree; (c) lifecycle NEVER reaches orphan detection, red-proofed. **LIVE-PROVEN 2026-07-21 on 9201** through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders **0 times** on the Alkalmazások page while the control app renders 10; a direct `POST /api/stacks/plant-it/deploy` returns **HTTP 409 `{"ok":false,"error":"Ez az alkalmazás jelenleg nem telepíthető."}`**; the app page carries the notice and no Telepítés button. **v0.158.1 is a shipped-and-caught defect worth remembering:** the three predicates were declared with POINTER receivers, and html/template cannot call those on the non-addressable value the handler passes — every `/apps/<slug>` returned 500, for every app, while compiling cleanly with a fully green suite, because no test rendered `app_info`. A template method call is only checked when the template runs. Follow-on: R-56's difficulty badge reuses this plumbing |
| R-58 | **[P2] Assisted disk-picker install mode — the installer should let the operator CHOOSE the target disk instead of requiring the serial up front.** Today an install is either unattended (the answer file pins one `ID_SERIAL_SHORT`, which you can only know by first booting the machine) or match-nothing safety (aborts by design). That forces a two-boot dance for every new box: boot the safety ISO to read the serial, rebuild the ISO armed, boot again. | S–M | **idea — operator ruling 2026-07-21** | **Operator's argument, verbatim:** *"the installer should list the available storage devices (excluding the installation media) and let us select one, and continue."* **Shape:** a THIRD ISO mode alongside the two that exist — unattended-serial and match-nothing-safety. It enumerates candidate disks with **size / model / serial**, excludes the installation media itself, takes a selection plus a confirm, and proceeds. **Unattended+serial REMAINS the appliance/factory mode** — it is the right shape when the machine is provisioned in bulk and nobody is standing there; the picker is for the case where somebody is. **Slice 1 (cheap, same code surface, do this first):** improve the abort screen. On filter-no-match the installer currently just fails safe and says nothing useful — it should print the candidate table (size/model/serial) plus the one-line hint naming which serial to put in the profile. That alone collapses the two-boot dance from "boot, guess, go read docs, rebuild" to "boot, copy the serial off the screen, rebuild", and it is the same enumeration code the full picker needs. **Why it matters beyond convenience:** it is the BYO / reinstall flow — a customer's existing hardware, or a rebuild of a box whose disk layout nobody recorded, is exactly where the serial is unknown and a wrong guess is destructive. The current fail-safe is correct but mute. Origin: TASK-G, arming the HP install ISO — the serial had to be read off the board by hand between two boots |
| R-59 | **[P1] A no-DHCP install must HARD-ABORT — instead it bakes the installer's fallback address as a STATIC config and completes, producing a box that can never call home.** | S | **SHIPPED v1.24.0 (2026-07-22) — as a FIRST-BOOT refuse-loudly gate, with a RECORDED DEVIATION: the install-time abort is out of scope (the 192.168.100.2 fallback is baked inside the Proxmox auto-installer itself, unreachable without an installer-initrd hook; operator-acked, not silently dropped). The screen + gate proven on the nested drill (SPIKE-firstboot-nic-sweep-2026-07-22.md); nested ≠ metal — metal proof rides the next real install** | **This is the worst silent onboarding failure shape there is:** the install *succeeds*, the box looks finished, and it is permanently unreachable — no hub check-in, no pairing, no way in except a keyboard and monitor. Found on the HP t740's first install: the 4-port NIC got no DHCP lease at this site (see the t740 gotcha in `scripts/iso/README.md`), and rather than refusing, the installer wrote its **192.168.100.2 fallback as a static `vmbr0` address** into `/etc/network/interfaces` and carried on. **The philosophy is already established one layer over — the disk filter refuses loudly and touches nothing when it cannot identify its target** (spike S5c, proven twice on real boards). Networking deserves the identical treatment: no lease on any carrier-bearing NIC ⇒ **abort with a legible screen**, never invent an address. Slice: detect "DHCP produced no lease" in the answer/first-boot path and fail with the candidate NIC table (name / MAC / carrier / link speed) plus the one-line remedy, exactly as R-58 slice 1 does for disks — same refuse-loudly grammar, same screen shape. Pairs with **R-60**, which is the self-heal for the case where the cable simply moved |
| R-60 | **[P2] First-boot NIC sweep self-heal: if the hub is unreachable, try DHCP across every carrier-bearing NIC before settling.** | S | **SHIPPED v1.24.0 (2026-07-22) — spike + nested drill proven (SPIKE-firstboot-nic-sweep-2026-07-22.md): cable move → sweep → heal + hub registration unaided in <1 min; sweep is structurally first-boot-only (state.json gate + the unit's done-flag condition); drill also surfaced and fixed the baked-fallback-default-route trap (flush before the bounded dhclient)** | `felhom-bootstrap` currently accepts whatever addressing the installer left behind and, if the hub cannot be reached, simply stays broken. On demo-hp the fix was a human moving one cable from the 4-port card to the onboard port — **a sweep would have healed it unaided**: enumerate NICs with `carrier=1`, DHCP each in turn, and keep the first that reaches the hub. Cheap because the box has nothing to lose at first boot (no customer data, no running guests) and the failure it repairs is total. Deliberately scoped to FIRST BOOT and to the hub-unreachable condition only — a running box must never re-shuffle its own networking. Complements **R-59**: that one refuses to produce an unreachable box, this one repairs the case where the truth changed after the install (cable moved, switch port died, the installer guessed the wrong port) |
| R-61 | **[P1] The baked root password must be knowable by the operator — the recurring console lockout.** | S | **slice 1 SHIPPED v1.24.0 (2026-07-22): the build emits the plaintext into a 0600 sibling `<iso>.rootpw.txt` (single record of truth — never logged/manifested/committed); drill-verified against the installed box's shadow hash. Follow-up (appliance-grade record-keeping) stays open** | The ISO mints a **fresh throwaway crypt hash per build** and the plaintext is discarded, so nobody — including the person holding the machine — can log into the console of a box they just installed. Today that meant reaching demo-hp only through the G1 break-glass credential vaulted in the hub, which is the right mechanism for a *lost* password and the wrong one for a *never-known* password: it requires a working hub, a working network, and operator tooling, at exactly the moment the likely reason you need the console is that one of those is broken. **Slice 1 (do this):** the ISO build emits the baked root password into the build REPORT and the operator cheat-sheet alongside the sha256 — it is already a per-build value, so surfacing it costs nothing and closes the lockout. **Follow-up (appliance-grade):** keep it per-build random and treat the build output as the record of truth. **A fixed well-known password is explicitly REJECTED (operator ruling 2026-07-21)** — a pre-pairing box sits on a stranger's LAN with a predictable root credential, which is a far worse exposure than the lockout it would fix. Relates to G1 break-glass (the vault stays; this is about the window before/without it) |
| R-62 | **[P3] Hub delete dialog: show the customer-id the operator must type, and reword the three acks for the ghost shape.** | XS | **idea (operator, 2026-07-22)** | Cosmetic, hub-only, docs-only in the v1.24.0 train. The delete confirmation asks the operator to type the customer-id, but the id appears NOWHERE on the Edit page the dialog opens from — the operator has to fish it out of the URL or another tab. Also: for a GHOST customer (host already gone) the three acknowledgement checkboxes describe teardown steps that cannot happen; **wording only** — the server MUST keep requiring all three (the render-gate lesson of v0.70.1 stands: reachability and requirements are separate concerns). |
| R-63 | **The install console learns ő/ű** — the kernel default console font lacks the Hungarian double-acute glyphs, so the R-59 network screen and pairing banner rendered ő as blanks. | XS | **SHIPPED (scripts v1.25.0, 2026-07-23)** | `felhom-bootstrap.sh` loads a Latin-2 console font (`Lat2-Terminus16` → `Lat2-Fixed16` → `Lat2-Terminus14`) ONCE before the first paint — idempotent, best-effort (a missing font/ioctl never blocks boot). Lat2 ships in the trixie/PVE base (console-setup), so no copy rewording needed. Font names verified against the package. Nested-console capture proof rides the v1.25.0 drill. Also in the same train: **F9 belt-everywhere RESOLVED** (host-install default appliance leg + live on both boxes + login proven) and the **R-71 build-gate** (`build-felhom-iso` asserts golden ≥ managed floor, `publish-train-rules.md` rule 5). **Live finding: golden 0.153.0 < floor 0.156.0 in production NOW** — the gate catches it; the fix is the golden republish at 0.161.0 (Part 4, pending; the managed floor stays 0.156.0). See `REPORT.md` (2026-07-23). |
| R-64 | **„Felhom↔Felhom media pairing blessed" — the two-box SMB pairing (one box shares, the other mounts it as NAS storage) becomes a supported, documented flow.** | XS–S | idea (2026-07-22) | Origin: the operator ran the pairing drill on the live demo pair and it WORKS — the drill itself is the pending evidence leg (a written run-through with the R-66 surfaces in play). R-66 shipped the enabling visibility: the serving box's address is now on its own Beállítások → Rendszer „Hálózat" card, and the add form names the NetBIOS trap. Blessing = a short customer-facing recipe (`documentation/controller/network-storage-nas.md` naming-caveat paragraph is the seed) + one supported-path sentence in the capability map. Flips: would add a "Felhom↔Felhom media pairing" capability row (currently unlisted). Pairs with R-65 (same two-box topology, entirely different transport + guarantees) |
| R-66 | **The box's own address becomes visible — „Hálózat" card, Debug network dump, NetBIOS hint.** | XS | **SHIPPED (controller v0.159.0, 2026-07-22)** | Origin: the pairing drill — the serving box's IP was findable only as a hint buried on the OTHER box's Megosztás page, and the add form's failure for „FELHOM" taught nothing. Three legs: (A) „Hálózat" card on Beállítások → Rendszer (Helyi cím / Hálózati név only-while-sharing / Átjáró; live per render, stored nowhere — S-5; „—" when unavailable); (B) `network` section in the Debug dump (interfaces/route/DNS/lan_address, best-effort per item); (C) the NetBIOS trap named (Szerver helper text + a purely lexical hint on `unreachable` for single-label non-IP names). **Design decision recorded:** the controller is bridge-netns'd, so ALL guest-net reads go through the one netns door (docker exec into host-networked felhom-samba, `stacks/guestnet.go`) — with Megosztás off the card honestly shows „—" rather than the plausible-wrong 172.x answer. Deployed demo-felhom + demo-hp 2026-07-22; demo-hp live-shows the closed-door path (sharing off → dashes + in-place dump errors), demo-felhom the open one (real .104/.1/\\FELHOM values). Flips no capability-map row (diagnosability/UX polish); enables R-64 |
| R-67 | **The NAS share appears in FileBrowser — browse what you mounted.** | S | **SHIPPED (controller v0.160.0, 2026-07-22)** | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). **Couples to R-64: browsing was its missing UX half.** A registered network storage now binds its share ROOT into FileBrowser (`/mnt/felhom-drives/<name>:/srv/<name>:rslave`) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (`Host is down` in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: `felhom-controller/REPORT.md` (2026-07-22) |
| R-68 | **Notification train: paired recovery mails + prefs seeding at claim + priority headers (power-outage audit F11+F12+F14-light).** The dead-man's-switch fired perfectly on 07-22 and the customer who got „A szerver nem elérhető!" was never told it recovered (F11); a customer without a `customer_notifications` row is silently unnotifiable (F12, demo-hp live); delivered ≠ noticed (F14). | M | **SHIPPED (hub v0.71.0, 2026-07-22)** | Origin: `AUDIT-power-outage-recovery-2026-07-22.md`. Recovery = explicit eventType branch (severity semantics frozen; `severityNotifies` untouched): operator always hears both edges, customer iff PAIRED (customer-channel `sent` stale/down row newer than the last sent recovery — `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for recovery; ties → no mail, flap-safe). Seed-at-claim: `MarkClaimed` → `SeedNotificationPrefs` (INSERT-if-absent, never upsert — red-proofed; empty email no-op; never fails the claim; default critical-only set). Hub-side empty-email no-clobber belt in `handleSavePreferences` (controller 0.160.0 already guards its own two push legs — latent, not live). `X-Priority: 1` + `Importance: high` on error/critical via Resend `headers` (live-probed HTTP 200 before implementation); the `test` event now also mails the operator with those headers (one click proves both channels + rendering); latent `sendTestEmail` nil-prefs panic fixed. 17 tests + 4 red-proofs. **Live legs pending:** natural `*_recovered` mail on the next real staleness cycle (or the reboot-drill arc — NEVER fabricated by blocking reports, that is F9-bypass-shaped) and seed-at-claim on a real claim (Peti Friday reinstall is the natural candidate) |
| R-69 | **F14-full: an operator push channel that actually interrupts (ntfy / Telegram / similar), beyond mail-client priority flags.** F14-light (v0.71.0 headers + Gmail filter) nudges a mail client; a 15:29 node_down should reach the operator's pocket in seconds regardless of inbox hygiene. Needs: channel choice (self-hosted ntfy on k3s vs Telegram bot), dispatcher fan-out seam, per-severity routing, quiet hours. | M | idea | Origin: `AUDIT-power-outage-recovery-2026-07-22.md` F14. Deliberately NOT built in the v0.71.0 train (scope-forked per the task spec) |
| R-70 | **[P2-HIGH] The offsite last mile is invisible on BOTH surfaces — the hub cannot tell "staged" from "delivered" from "applied".** demo-hp sat 2 days with the hub customer page saying "Provisioned: … the transient password is delivered to the controller once" while the box said „Még nincs beállítva távoli mentési cél" — and a real customer would sit unprotected indefinitely believing otherwise. The hub HAS the signal (`one_time_secrets.consumed_at` + 153 consecutive reports carrying no offbox object) and reads none of it: the "Provisioned" line is static copy gated only on `offsite.host` in ConfigJSON (`config_form_body.html:119–120`). | S–M | **SHIPPED (hub v0.72.0 + controller v0.161.0, 2026-07-23)** — detector `offsite.DeliveryStateFor` (one impl, all consumers), customer-card state line with age (static "delivered once" copy GONE), `offsite_delivery_stuck` warning event (24h durable cooldown), controller truthful empty-state banner. Live validation on the two fixtures (demo-hp `applied`; demo-felhom `applied` + stale-staged info) recorded in `felhom.eu/REPORT.md`; the banner leg is unit-proven/live-pending (no box occupies the enabled+no-offbox window). | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. Two legs: **hub customer card** shows the real delivery state ("provisioned, awaiting box consumption" / "consumed, awaiting apply" / "applied" — consumed_at × report-offbox-presence is enough for all three), and **controller banner** when the descriptor is enabled but no target is configured („Felhom offsite készen áll — a beállítás automatikus, folyamatban"). Couple to **R-31**'s async/status-card idiom (same surface likely serves both) and to the **R-39 `consumed_at` honesty gauge** precedent on the PBS side — a consumed secret + N report cycles with no offbox status is the same "disagreement no single tier can see" shape and deserves the same loud event. Supporting live datum: demo-felhom's 07-21 staged secret is still unconsumed today (key-auth-first path never consumes) — invisible for the same reason. |
| R-71 | **[P1] Day-0 race: the managed floor-update kills the offsite apply-bridge between password-consume and persist — the one-shot credential is burned and the box lands in the silent consume-404 dead-end forever.** Proven on demo-hp (07-21): consume 16:27:42 → managed update 0.153.0→0.156.0 replaces the container 16:28:17, ~35 s later, mid `ssh-copy-id` window; nothing persisted, no installed key ⇒ the key-auth-first recovery path can never engage, and every subsequent start logs the WARN and gives up until an operator Re-issue. **This recurs structurally on every fresh onboarding whose ISO floor lags the managed floor** — the update fires minutes after first boot, exactly when the bridge first runs. demo-felhom escaped by timing only. | M | **SHIPPED — (a)+(c); (b) rejected-by-design.** **(a) SHIPPED (controller v0.162.0, 2026-07-24): the apply-bridge settle-gate.** `offsiteapply.SettleProvider.SettleState()` + `SettleFunc` adapter over the updater's OWN `GetFloor()`/`IsUpdateRunning()` (no second floor path); `Bridge.AwaitSettle` defers the consume while `updateRunning` OR below the known floor (the auto-floor update is imminent), releasing at/above floor on the first poll with zero added latency (B′). Bounds 10 s poll / 90 s floor sub-bound / 5 min overall — both bounds GO+WARN (a hub that can't serve a floor can't serve a consume → no burn; R-71c is the belt). Ordering-only — the consume/persist order and the 404-no-oracle contract are UNTOUCHED. Finding: the floor is in-memory (report-ACK ~5–10 s), not persisted → unknown until the first ACK on any restart; the 90 s sub-bound is sized to that. 5 scenarios (A–E) + 4 red-proofs all observed FAIL then restored. Deferral paths NOT live-fired — their precondition is now structurally prevented by the v1.25.0 golden≥floor build gate (the point); live leg = the B′ first-poll GO line on both above-floor boxes. **(c) SHIPPED (hub v0.72.0, 2026-07-23): self-heal restage via the EXISTING Re-issue path (`monitor.OffsiteDeliveryChecker`), consumed ≥1h + ≥4 offbox-less reports + zero offbox evidence, one restage/customer/24h, `offsite_credential_restaged` warning; R-39(a) guard red-proofed.** **(b) two-phase consume REJECTED-BY-DESIGN** — it would touch the 404-no-oracle contract; (a)+build-gate+(c) cover the race without it. **Layering: v1.25.0 build gate PREVENTS, (a) DEFERS, (c) HEALS.** | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` (mechanism cites: bridge order `offsiteapply.go:106–187` consume-then-persist, retry only on process start; the dead-end is even documented in source l.168–173 "the password is spent; reset it on the hub to retry"). Candidate directions, spec-first: (a) **order** — first-boot path lets the managed update settle before the bridge's consume step (cheapest; the race window is the update, not the reboot); (b) **two-phase consume** — hub marks consumed only on a controller ack-after-persist (touches the 404-no-oracle contract, design carefully); (c) **hub-side self-heal** — auto-restage a fresh secret when `consumed_at` is set but K consecutive reports show offsite enabled with no offbox status (the R-39(a) mint-race lesson applies: never restage on top of an UNCONSUMED secret). Pairs with R-70 (visibility) — but visibility alone is not the fix; the burned credential needs an unattended recovery path. |
| R-53 | **`app_export.html` substituted the CSRF token where the customer domain belongs** - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | **SHIPPED (controller v0.150.0, 2026-07-20)** | One template token (`{{$.CSRFToken}}` -> `{{$.Domain}}`) plus the `Domain` key in `exportPageHandler`'s data map - that handler does not go through `baseData`, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined `<sub>.<domain>` and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F7) |

## P3 — post-alpha

| ID | Item | Size | Status | Notes |
|----|------|------|--------|-------|
| R-26 | **Guided old-history recovery via a retained superseded escrow + the recovery code.** Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (`host_escrow_superseded`, `ListSupersededEscrow`). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside `.orphaned-<date>` repo for restore. | M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin `DIAGNOSE-offbox-repo-orphaned-2026-07-17` |
| R-27 | **Customer-facing self-bind page (R-21 slice C follow-on).** Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | **SHIPPED (slice 1, hub v0.66.0, 2026-07-17)** | **Slice 1 = the FIRST-box flow, DONE:** operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor `/bind/<token>` (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same `BindAppliance`, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP `/bind/` exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). **Slice 1 PROVEN-LIVE 2026-07-18:** in the rehearsal the operator minted + emailed the link at 16:28:55 and **the customer bound their own box at 16:29:55 with zero failed attempts**, `appliance_bound` carrying source `customer_selfbind`, credential delivered 26 s later with no operator action (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). The new-ISO console banner rendered on the real box. **Slice 2 = console-passphrase bind → R-27c** (below). Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. **Multi-box per link is out of scope by design** (single-active, one-shot → binding several boxes = repeated operator sends) |
| R-27b | **Customer self-bind, second-box flow (controller side).** For a customer who ALREADY has a bound box and installs another, the controller shows a dismissable "bind another box" prompt (and a bind-later entry under settings) that walks to the hub `/bind/` page — so a returning customer isn't emailed a fresh operator-sent link for every box. Mechanism sketched in the hub v0.66.0 REPORT; NOT built (R-27 slice 1 deliberately did not touch the controller). | M | idea (minted by hub v0.66.0) | Origin: hub v0.66.0 slice-1 ship (first-box only). Reuses the same `/bind/` public page + tokenized-link machinery; adds a controller-side entry point + the operator "mint a link for an existing customer" affordance |
| R-25b | **RULED: customer DELETE becomes a guided full-teardown cascade.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown. | M (was S) | **SHIPPED hub v0.69.0 (2026-07-21)** | **operator ruling 2026-07-21**: **DELETE subsumes the whole cascade, behind explicit consent.** Three separate acknowledgements, each its own checkbox — (1) the host(s) will be deleted, (2) the customer will be RESET including external teardown and offsite data destruction, (3) the customer record and escrow will be purged — plus a **typed customer-name confirmation** before the button arms. Internal order is **host-delete → RESET → delete**, which preserves every existing invariant rather than relaxing any: RESET keeps its no-hosts precondition (hosts are already gone by then), and escrow keeps its demote-then-purge custody rule (host delete DEMOTES to retained custody, the final delete PURGES — the one true purge point). **Re-sized S → M: this is a multi-step destructive wizard with three acks and a typed confirmation, not a checkbox.** Implementation is explicitly NOT part of TASK-E; the row carries the ruling and awaits its own spec. **It no longer blocks R-3** — the model is decided, so the friend-alpha runbook can be written against it. **IMPLEMENTED per the ruling (TASK-I, hub v0.69.0):** `POST /configs/{id}/delete` now runs `hosts → RESET → purge`; three acks + typed customer-id + a stale-preview check + the ONLINE-host refusal, all gates before any write (zero side effects on refusal); custody purged exactly ONCE in leg 3 (leg 2 runs with `purgeEscrow=false`); ruling-3 preserved BY CONSTRUCTION and asserted from inside leg 2; failed legs retain the journal and the dialog offers Resume. Standalone RESET byte-identical. 5 red-proofs. Offboarding guidance: `runbooks/RUNBOOK-onboarding-draft-v4.md` §G. **v0.70.0 follow-up (same day, found validating against the live hub):** a completed delete still left the customer on the Customers list and still ALERTING, because `GetCustomers()` is report-derived and no tier ever deleted a report — new **residue** leg (reports/telemetry/log-tails/notif-prefs + the credential-bearing `appliance_registrations`/`selfbind_tokens`), and **ghost customers are now deletable** (404 = nothing here, not no-config-row). **v0.70.1 (2026-07-22): the ghost delete was implemented but UNREACHABLE** — the Danger-zone card (and the `customerDeleteOpen` script) sat inside `{{if .HasConfig}}`, so a ghost rendered no Delete button at all (the fourth inert-seam defect; handler tests POST directly and proved nothing about reachability). Render gate split: RESET stays HasConfig-gated, Danger zone gates on `Deletable` (the exact negation of the preview's 404 predicate), Block/Unblock stay config-only; render tests per branch + 2 red-proofs. **Operator live leg: the demo-vm-felhom ghost delete click — PENDING** (doubles as the v0.70.0+v0.70.1 live validation; expect `residue=ok customer_delete=ok` with `skipped_no_config` Hetzner/descriptor legs, staleness emails stop) |
| R-25 | **Device-node TOCTOU hardening (drive init).** Graduate the controller v0.141.0 Observation: the `format → resolveEnrollUUID(path) → AssignDisk(uuid)` sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. | S | idea | From the v0.141.0 F6 commit's security-review finding (`felhom-controller` REPORT). Low real risk (single-operator, agent-guarded), but cheap to close |
| R-24 | **Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17).** The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE `SetConfig` (live cgroup, no reboot, Phase-0 proven). **Framing note:** the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). **COMPLETE — the apply is proven in BOTH directions on metal (2026-07-18):** customer zero shrank the guest **11675 → 8192 MB (16:50:22)** and grew it **8192 → 12288 MB (17:02:17)**, each a live cgroup apply with no reboot, the new total rippling into the deploy page's memory math. The nested demo had proven the refusals; the N100 proved the applies. `tests/VALIDATION-n100-rehearsal-2026-07-18.md` ledger 9. **Cores/live-resize as hub desired-state is NOT built** (deferred, low demand). | M | **SHIPPED + PROVEN-LIVE** | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". |
| R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster |
| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | **first slice PROVEN-LIVE (poke channel)** | **FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17):** the ep0-relayed contentless poke (hub→ep0 `felhom-poke` forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the **agent** and fires an immediate desired-state cycle. Full path live: real operator manifest save → `sync-poke delivered to 10.77.0.2`; box → `poke received → immediate desired-state cycle` (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per `SPIKE-immediate-sync-transport-2026-07-16` P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately |
| R-28 | **Agent fast-tick-until-first-convergence — SHIPPED (agent v0.90.0, 2026-07-17).** Closes the immediacy leg a hub poke can't reach (the pre-tunnel WG-registration window): while ANY desired-state item is unapplied the agent pulses the out-of-band trigger every 30 s and **self-disarms** on convergence (state-based; four cached sources — desired-gen==0, reconcile Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational; LOUD states + pending_signature excluded). LIVE on both demo agents (startup line verified). **Guests-0/0 passenger:** diagnosed live and REFUTED as a bug — the 0/0 was the legitimate pre-provision reporting window; the fast-tick is exactly its mitigation (≤15 min → ~30 s). **Unfired live:** the real-onboarding proof (a fresh box converging in seconds without a poke) → the rehearsal S5. | M | **SHIPPED** | See felhom-agent REPORT (v0.90.0). Third immediacy leg alongside R-13 (poke) + R-23 (UI). |
| R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | |
| R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11). **Launcher coupling — REVISED (controller v0.165.0):** the "share the launcher outside the household" need is now met WITHOUT member accounts — the **Indítópult megosztása** capability-URL guest link (`/s/<token>`, information-only, no account) shipped in v0.165.0. What remains for this arc is member-specific: **per-member tile visibility** (each member sees only their apps) and the launcher-as-member-landing-page — both live inside this SSO/members arc; the guest-link ruling explicitly SUPERSEDES the earlier "members are how you share the launcher" framing |
| R-72 | Curate `brand_color` for the top catalog apps | XS | idea | Parked follow-up to the v0.163.0 launcher. `.felhom.yml` `brand_color` (`#rgb`/`#rrggbb`) overrides the deterministic slug-hash tile color; no catalog app sets it yet. Pick brand-accurate colors for the most-installed apps so their launcher tiles match their real brand. Catalog-only change (`app-catalog-felhom.eu`), `brand_color` is already `omitempty` and consumed by the controller |
| R-73 | **SMART history / trending (hub-side)** | M | idea (parked) | Parked follow-up to the v0.169.0 disk-health card + degradation alert (agent v0.94.0 serializes SMART into `/disks`; controller renders „Lemezek állapota" + fires `disk_health_degraded` on a degradation). Today the verdict is point-in-time and the baseline is in-memory (a controller restart re-baselines silently). Hub-side would let the operator SEE a disk's SMART counters trend over time (reallocated/pending sectors climbing, NVMe wear %), catch slow degradation the 6h transition check can miss between reboots, and persist the baseline across restarts. Needs a hub ingest surface for periodic SMART snapshots + a small time-series store + an operator trend view. **No demo hardware exposes real SMART** (virtualized), so this only earns evidence on a real box |
| R-74 | **Island control plane on a CLUSTER (Peti's 2 nodes)** — bring R-50's island bridge to a multi-node PVE cluster. | M | idea (Phase C of R-50, parked) | R-50 shipped the island for the ONE-host fleet (demo-hp, demo-felhom). A cluster needs **bridge parity on every node**: either per-node identical `/etc/network/interfaces` `vmbr9` stanzas (simplest, drift-prone) or — preferred at ≥2 nodes — a Proxmox **SDN zone/vnet** defined cluster-wide (one definition, auto-applied per node). The guest island IP is per-guest + node-independent; the **agent-follows-guest** rule holds (each node's agent binds its own `vmbr9` `169.254.253.1`). Migration order per the spike: drill-proven → demo (done) → **Peti (this row)**. Its own supervised runbook, coordinated with Peti (a live customer). Completes the capability-map "site/network change" row for clustered installs. Source: `audits/SPIKE-island-bridge-2026-07-25.md` (cluster-parity finding) + `RUNBOOK-island-migration.md` (single-host procedure to generalise) |
| R-75 | **Catalog-derived userdata skeleton + import surfaces** | M | **SHIPPED (controller v0.172.0 + catalog, 2026-07-26)** | `${IMPORT_PATH}` = `<system namespace root>/userdata/import` — ONE canonical drop-zone per box on the system drive, with **no per-drive fallback** (unresolvable leaves the var unset so compose fails loudly). *Operator ruling overriding the spike's Fork-1 per-drive recommendation:* each drop-zone app has exactly one ingest bind, so a per-drive `import/` would put a dead lookalike on every other drive — unbacked too, since `import/*` is `class: excluded`. Third `BindRoot` (`RootImport`) threaded through the classifier AND every resolver (`resolveAbs` would otherwise have named a directory on the WRONG DRIVE); catalog-derived skeleton, **sorted** (the naive map-order derivation measured 20 distinct outputs from 20 identical runs — a fleet-wide FileBrowser restart loop via `fbNeedsRecreate`); one authoritative compose parser; FileBrowser „Beolvasás" source; app-page „Hova tegyem a fájlokat?" with `PathEscape` deep links and **class-driven** copy; `data_paths:` annotation (malformed path ⇒ whole-block reject, unknown role ⇒ fails OPEN); system-owned `beolvasas` SMB share refused server-side at handler AND store. **Live-proven on demo-hp:** consume bind resolves to the system drive, a file dropped through FileBrowser's mount was ingested and deleted in ~30 s, the share auto-appeared on enable and survived a direct delete POST, and the before/after directory diff on BOTH boxes shows **zero removals** (demo-hp gained `media/podcasts`, the one genuinely derived entry). Two defects found and fixed during the live legs: `EnsureImportRoot` left the parent `userdata` at 755, and the carry-list initially re-created the per-drive drop-zone forever. Capability-map row *File access via browser* **deliberately NOT upgraded** — nothing drove the FileBrowser HTTP UI. Docs: `controller/import-and-data-paths.md`, spike `audits/SPIKE-catalog-data-paths-2026-07-26.md` |
| R-77 | **Endpoint-drift detection, samba protected-set gate, channel log honesty** | S | **SHIPPED (controller v0.173.0 + hub v0.74.0, 2026-07-26)** | Fixes the two defects in `audits/DIAG-agent-channel-2026-07-26.md`. `bootstrap.DetectEndpointDrift` names a `controller.yaml` vs `bootstrap.json` `local_api.endpoint` divergence with its own event type `local_api_endpoint_drift` and its own banner — and **writes nothing** (see R-78). `EffectiveProtected` now gates samba on `Enabled && UserSet`, mirroring both of `reconcileSambaAt`'s early returns, plus the doc-comment correction. Channel log distinguishes born-down (`unseeded->down`) from a real transition, logging-only. **Part 0 repaired both production boxes** (endpoint → `169.254.253.1:8443`; fingerprint+token agreed). Tests 951→959; red-proofs A/E/F. |
| R-78 | **`local_api` authority ruling — auto-reconcile vs detect-only** | M | idea (deferred OUT of R-77 on purpose) | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. |
| R-79 | **`report.Issues` / `report.Warnings` are English on customer-facing surfaces** | M | idea | **Whole-surface, not a one-off** (DIAG §6): every producer is English — `"SSD/HDD disk usage critical"`, `"Docker: %v"`, `"Protected container not running: %s"`, and all six `Warnings` strings. They render on the customer's Hungarian dashboard, and the `health_critical` path has reached the **customer** email channel three times historically. Deliberately NOT bundled into R-77: a copy sweep across every producer would have buried two safety fixes in string churn, and the seam is not obvious — translate at the producer, or at the render/notification boundary where operator-English and customer-Hungarian already diverge? Pick the seam in a spike; the strings are mechanical after. |
| R-80 | **`expected_backup_missed` false alarm — diagnosed + class-fixed** | M | **SHIPPED (hub v0.75.0, 2026-07-26)** | Diagnosed in `audits/DIAG-backup-missed-2026-07-26.md`, fixed as R-81. **The premise was wrong on both counts:** it fired ONCE (not nightly), and no real external customer was notified — the one customer-channel mail went to the operator's own mailbox; `peti-felhom` (the only real external customer) did not fire. **The 7.3 days did NOT predate the channel outage** — it is the age of the *PBS* snapshot, reached only as a fallback once the vzdump array went empty. Local vzdump was never stale (archives on disk 07-24/25/26). Cause: the agent's backup store is in-memory, so the R-50 fleet restart at 12:44 UTC emptied `backups` until the next backup at 07:03, and the hub read empty as "no backup exists". Fixed by anchoring (hub v0.75.0). **Left standing as a REAL finding → R-82:** the PBS/offsite-DR tier has no schedule at all. |
| R-81 | **"No signal" is not "bad signal" — anchor the backup deadline check** | S | **SHIPPED (hub v0.75.0, 2026-07-26)** | Third instance of one class (hub v0.12.0, v0.73.0, this). `assessBackupFreshness` now returns OK/UNKNOWN/MISSED; absence is UNKNOWN until it outlives an anchored window. The hub reads its own retained host-report history (`GetHostReportsSince` + `newestBackupEvidence`, 7-day bounded lookback) to answer "when did I last SEE evidence of a backup?", and anchors absence at first contact (`GetFirstHostReportAt`), reusing the existing 26 h threshold as the grace — no new knob, the v0.73.0 shape. Deferrals are logged; reason strings kept distinct. Tests 493→508; red-proofs A/B/C. **Not silence** — a genuinely dead box still alarms, which is the half the naive fix breaks. |
| R-82 | **The backup target split — local daily + offsite weekly** | M | **SHIPPED (agent v0.102.0 + controller v0.175.0 + hub v0.76.0 + host-install 1.20.0, 2026-07-26)** | Phase 0: `audits/SPIKE-r82-phase0-2026-07-26.md` (weekly CONFIRMED; the `pvesm status` 0/0/0 resolved as a namespace-scoped-token reporting artifact; capacity measured). `backup_targets[]` gives each tier its OWN cadence and retention; the controller runs both due tiers in ONE quiesce window; the hub judges each against its own threshold (host 26h / offsite 8d); host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly, and an unprovisioned tier DEFERS rather than firing at a missing storage. Operator rulings 2026-07-26: 2-week offsite retention, first backup runs as long as needed, one backup at a time per guest, drill box dropped from the rollout. **Live:** demo-felhom + demo-hp both on local-24h + felhom-pbs-168h; demo-felhom's first real PBS backup landed (`TASK OK`, 41 min, 14.46 GB) and restored cleanly. **Four defects found by RUNNING it, not reviewing it:** 30-min wait bound vs a 41-min backup (v0.98.0), restore tier read from the configured target not the archive (v0.100.0 — a silent regression of the S4.1 fix), a leaked scratch keeping `onboot:1` (v0.101.0), and a tier firing at a not-yet-provisioned storage (v0.102.0). **REMAINING GATE:** the installer default is written but the fleet-wide flip waits on a full weekly cycle holding; and the offsite tier is never AUTOMATICALLY restore-tested (the scheduled test only selects the primary tier) — that gap, plus the hub inferring cadence from storage TYPE, are the two follow-ups. |
| R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. |
| R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. |
| R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). |
| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — operator ruling 2026-07-27** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
| R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. |
| R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | idea — observed live 2026-07-27 | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **Bounded, not severe:** the doomed backup fails in ~0 s at storage activation, so it is not the wasted multi-hour WAN upload R-84 was written to stop; the cost is a spurious `backup failed` event per restart, and potentially a stream of them if the link flaps. **Shape:** distinguish *storage unreachable* from *storage readable and empty*. Unreachable is UNKNOWN — defer the due-verdict rather than resolving it either way, exactly as R-81 made the hub do with a missing report. Only a target that is reachable AND has no archive is genuinely due. Note this does NOT weaken R-84's fail-safe intent: a tier whose storage is merely slow or briefly unreadable should still err toward backing up — it is specifically the **cold-store + unreachable** pair that must defer, because there the fallback has no information at all, only an empty default that looks like a fact. |
| R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog |
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**)
| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v <volumeName>:/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** |

| R-37 | **Post-RESET health card shows stale pre-RESET warnings.** After a RESET the card should read **„RESET óta nincs adat"** instead of carrying warnings about a lifecycle that no longer exists. | XS | **SHIPPED (hub v0.67.0, 2026-07-18)** | The customer page raises a banner when a RESET **completed** after the newest report, quoting „RESET óta nincs adat" and the reset timestamp, because until the box reports again every health figure describes a lifecycle that no longer exists. Deliberately narrow: an **in-flight** reset does not trigger it (only a completed one), and it **clears itself** on the first post-RESET report. Ties resolve to STALE — SQLite timestamps are second-resolution and a same-second report almost certainly arrived just before the reset destroyed what it describes; erring the other way would hide the banner exactly when it matters most. Red-proofed (neutering the predicate fails the assertion). — Origin: 2026-07-18 rehearsal. Same family as R-36 — the hub knows the state changed and the UI has not caught up |
| R-38 | **Installer GRUB slice.** A single default „Felhom telepítés" entry; the **interactive installers REMOVED** (safety: an interactive entry is how a wrong-disk manual install happens); felhom background. | S | **SHIPPED (scripts v1.22.0, 2026-07-19)** | Origin: 2026-07-18 rehearsal, alongside R-21's physical closure. Every ISO is repacked after `prepare-iso`: exactly **one** entry („Felhom telepítés", default, 5 s), with Graphical / Terminal UI / serial and the whole **Advanced Options** submenu (nomodeset ×2, three debug variants, Rescue Boot, memtest, UEFI settings) **not emitted** — not hidden, not password-gated. Boot behavior unchanged: the `linux`/`initrd` lines are lifted **verbatim at repack time** from the ISO's own automated entry, so a PVE bump tracks automatically; the build fails if they are missing, if the append line has lost `proxmox-start-auto-installer`, or if `auto-installer-mode.toml` is absent. Gated for 1 entry / 0 submenus / no live banned references, then **re-verified by reading the menu back out of the finished ISO**. Background generated at repack time from `website/assets/og-image_2.png` (one source, no second copy to drift), with the card's own grid phase-locked across the letterbox fill. Live-validated on the nested canary (UEFI/OVMF): renders, auto-fires, reaches the same match-nothing abort as v1.21.0. **Squashfs/theme rebranding still DEFERRED** — post-GRUB screens remain Proxmox-branded. GRUB's `e`/`c` keys also remain reachable (suppressing them needs a superuser password — a separate decision); the menu no longer advertises them |
| R-40 | **[P2-HIGH] The update path cannot express a MULTI-HOP major upgrade.** A template pin is a single value; the customer's update button pulls whatever the catalog now says. For apps whose upstream forbids version skipping this produces a broken upgrade. Nextcloud is explicit: *"You cannot skip major releases. Please re-run the upgrade until you have reached the highest available release."* Campaign 7 moved its template **31 → 34** (a fresh deploy validates fine — 302, 3/3 healthy), so an existing 31 customer pressing update would attempt a jump Nextcloud refuses. | M | idea | Origin: CAMPAIGN 7 (`audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` §7 F7). Not nextcloud-only — any app with sequential-major rules (gitea, tandoor, outline…) has the same shape. Directions: a per-app `upgrade_path:`/`max_hop:` in `.felhom.yml` that the update button walks in stages; or refuse-and-explain when the installed major is >1 behind; or pin an intermediate "stepping-stone" tag. **Until this exists, a >1-major catalog bump is safe for NEW deploys and unsafe for the update button** — which is exactly the asymmetry the campaign's MAJOR flag was meant to record but cannot enforce |
| R-41 | **[SLICE 1 SHIPPED 2026-07-21] The catalog has no standing "does every template still deploy?" check.** Campaign 7 was the first thing that ever tried to deploy all 53 apps, and found **5 that had NEVER been deployable**: papra (missing required `AUTH_SECRET`), zipline (v4 renamed `CORE_DATABASE_URL` → `DATABASE_URL`), wishlist (Docker Hub image gone; upstream moved to ghcr.io), homebox (upstream dropped the `v` tag prefix + new required env), glance (needs a seeded `glance.yml` the template never provides — PROVEN pre-existing: the pre-campaign v0.7.4 pin fails identically). Plus **7 broken healthchecks** and 2 apps whose images no longer resolve at all (plant-it, wanderer). | M | idea | Origin: CAMPAIGN 7 (§7 F5/F6). The repo already has the right pattern in `scripts/check-image-pins.py` — a mechanical gate run on every change. Cheap first slice: a **resolvability gate** (`docker manifest inspect` every pin) would alone have caught plant-it, wanderer, wishlist and homebox, and needs no box. Full slice: a periodic deploy-all sweep on the demo box reusing the campaign's engine. **Silent rot is the real risk** — an app can die upstream and nobody learns until a customer clicks Telepítés | **SLICE 1 SHIPPED 2026-07-21 — `app-catalog-felhom.eu/scripts/check-image-resolvable.py`** (+ 14 fixture tests, no network, resolver injected). Resolves every unique pin with `docker manifest inspect`, ONE image at a time; exit 0 / 1 (the registry says GONE) / 2 (inconclusive). **Two traps encoded, both hit live while building it:** (a) `docker manifest inspect` prints `toomanyrequests: …` and **still exits 0** — the same exits-0-on-failure shape as the ISO tooling's `validate-answer`, so stderr is inspected even on rc=0; (b) the inverse and more dangerous one — the first full sweep called **24 of 65 pins dead, including `postgres:16-alpine` and `redis:7-alpine`**, purely because Docker Hub throttled it partway through. Ambiguity therefore resolves to INCONCLUSIVE and never to an accusation: a gate that cries wolf gets ignored, and then it protects nothing. **The full sweep is still OWED** — DooPlex is not logged in to Docker Hub, so the 52-app table needs one re-run after `docker login`. Wired into `CLAUDE.md` + `REUSE.md` as a start-of-campaign / pre-publish-train step. It immediately paid for itself: it is what turned plant-it and wanderer from 'images do not resolve' into two DIFFERENT diagnoses (see the 2026-07-21 catalog entry). Full slice — the periodic deploy-all sweep on the demo box — remains open |
| R-42 | **RULED: sidecar majors follow the APP, never the newest tag.** Campaign 7 deliberately did NOT bump sidecar majors (postgres 16→18, redis 7→8, mariadb 11.6→12.3, postgis 16→17) while bumping ~40 app images to current. | S | **RULED 2026-07-21 — option (a)** | Origin: CAMPAIGN 7 §4. The case for not bumping: a DB major is a **data-plane decision the application owns** — immich proves it, upstream pins one specific tested `postgres:14-vectorchord…` build — and `postgres:16-alpine`/`redis:7-alpine` already track the newest patch inside their major, so they are not stale. The case for bumping: EOL majors eventually stop getting security patches, and "we never bump" silently becomes "we ship EOL databases". Suggested shape: per-app sidecar pin follows **upstream's own compose** where upstream publishes one, else stay within the current major and revisit at that major's EOL date |
| R-43 | **[P2-HIGH] No offsite restore path can restore a database — offsite restore cannot reconstitute a DB-indexed app.** All three offsite buttons are file-only. „Ellenőrző visszaállítás" and „Teljes visszaállítás" restore into a **scratch/verify folder** and never load a dump into postgres (`controller/internal/backup/offbox_restore.go:171-232`, `:130-164`); „Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" is `rsync -a --ignore-existing` over **mandatory userdata paths only** — it never touches a DB and never stops/starts the stack (`offbox_restore.go:346-421`). The DB dump is faithfully carried *into* the snapshot and can never be replayed *out of* it by any offsite action; the only code that replays into a live DB is `RestoreFromRecoveryUnit` (`backup/restore_unit.go:74`), reachable solely from the **local** restore panel (`web/handlers.go:1108`). Compounding: the `live recovery unit present — not overwriting` guard (`offbox_restore.go:400-405`) skips the **entire** unit — compose, manifest **and db-dumps** — whenever a local `backups/primary/<app>` exists, which is true on any box whose local backup has run, so on a normal box the offsite dump is never even staged to disk. | M | **SHIPPED controller v0.148.0 (2026-07-19) — live acceptance PENDING** | **SHIPPED:** new „Teljes visszaállítás (fájlok + adatbázis)" (`ReconstituteFromOffsite`, `/backup/offbox/reconstitute`): safety dump → stop → files overwritten to the snapshot version → start → the snapshot's own dump replayed (`reimportDBDumpsFrom`, reading the SCRATCH unit — the live unit is still never overwritten) → health wait. Two invariants, both red-proofed: **nothing is ever deleted** (`rsync -a`, no `--ignore-existing`, no `--delete`; post-snapshot files survive as extras) and **the undo exists before the act** (the `pre-restore-` dump is verified ON DISK before anything is stopped/overwritten/replayed; if it cannot be taken the operation refuses with zero changes). Outcome-shaped flash; the old missing-only button now states what it does NOT do. Deployed to demo guest 9201, healthy; page renders, button correctly gated behind a prepared full scratch. **STILL OPEN — the §9 live acceptance has NOT run** (upload → push → empty the trash for real → one button → photos back in the timeline). Until it does, no capability-map flip, the customer-restore row stays MISSING and R-3 stays DRAFT. — **Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (live on demo-felhom).** Viktor deleted 11 immich photos to test offsite restore; both place runs merged **0 files** and flashed success. Root cause is two-layered — the UI delete only *trashed* the assets so nothing was missing to restore, **but a valid test would also have failed**: files return, the library stays empty, because immich's truth lives in postgres. **The customer-visible failure mode: correct bytes on disk and an app that cannot see them.** Severity case: this is the offsite tier — the one that survives losing the box — and most of the catalog is DB-indexed. Direction (needs a spec, do not improvise): either offsite gains a real DB-replay leg reusing `RestoreFromRecoveryUnit` behind a stop→restore→start sequence, or the offsite UI states plainly that it restores files only and routes the customer to the local recovery-unit flow for the DB. Pairs with R-44 (a replay leg is worthless while the dump is stale). Flips: the **customer-restore map row stays MISSING** and gains this as a note — the C6 evidence attempt found a product gap, which is the rehearsal system working |
| R-44 | **[P2-HIGH] A manual offsite push ships an unrefreshed DB dump — "backed up now" is false for the DB half.** `offboxRunHandler` → `RunOffboxBackup` goes straight to the restic push and never calls `RunDBDumps` / `captureAllRecoveryUnits` (`controller/internal/web/offbox_handlers.go:203-227`, `backup/offbox.go:574-759`); the recovery unit merely **enumerates** existing dump filenames via `listFileNames`, never creates them (`backup/recovery_unit.go:105-106`). Dumps come only from the separate local `db-dump` daily at **02:30** (`cmd/controller/main.go:542`), with the scheduled offsite at 04:15 — so a *manual* run at any other hour ships a dump up to ~24 h old. **There is no freshness check and no RPO surface anywhere:** zero `RPO` hits across `controller/`; `offboxUnitTime` is only a two-drive tiebreak (`offbox.go:827-837`); the `DBValidationCache` exists (`backup.go:364-370`) but no offsite or restore path reads it. | S–M | **SHIPPED controller v0.148.0 (2026-07-19)** | **SHIPPED:** every offsite run — **manual AND nightly** — now refreshes the DB/volume dumps and recovery units (`offsitePreDump` → `runDBDumpsInternal`) BEFORE the restic capture, so each snapshot is an internally coherent `{DB@T, files@T}` bundle and retention becomes a history of restorable points. Order is the mechanism and is red-proofed (moving the capture first yields `[capture dump]`): the gap can only ADD files the DB does not reference yet, never remove one it does. This also makes the nightly ordering **structural** rather than a coincidence of two scheduler entries at 02:30 and 04:15. Each unit manifest carries `offsite_run_id` + `dumps_at`, so a pair's coherence is verifiable at restore time instead of assumed; the periodic refresh carries a prior stamp forward and never invents one. A dump-leg failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none). Honesty surfaces, all warn-level and none a gate: an unstamped pre-v0.148 pair reports its skew in the confirm, and `ValidateDump` gained an **exact-match** accounts-table sniff for customer-empty dumps (a substring match on "user" would flag `user_metadata`/`album_user`/`user_audit` on every healthy single-user box — red-proofed). — **Evidence: `audits/DIAG-immich-restore-2026-07-19.md`.** Today's unit dump `immich-postgres.sql` (51 954 452 B, mtime **02:30 CEST**) probed to **`asset: 0 rows`, `user: 0 rows`, `album: 0 rows`** — the 52 MB is entirely immich's shipped `geodata_places`/`naturalearth_countries` reference data. It predates both the admin user (created 07:56:25) and the photos (07:57). Same for the unit's `immich_immich_postgres_data.tar` (323 MB, also 02:30). **A dump that looks substantial by size can contain zero customer content** — size is not a health signal, and nothing in the product says otherwise. **Latent hazard:** had a full restore actually loaded that dump it would have written an empty DB over the live one, destroying the trashed rows that were the only surviving recovery path. Direction: dump-before-push on manual runs (the honest fix), **or** an explicit RPO line in the UI („adatbázis-állapot: <dump time>") so the operator/customer can see what they are actually shipping. Cheap interim: surface dump mtime + row-count sanity from the existing `DBValidationCache` on `/backups/restore` |
| R-45 | **[P2] Unified async-job feedback.** Every long operation invents its own progress surface, or none. Tonight produced three more one-off cards (v0.147.x: samba bring-up, offsite progress, restore result) on top of two existing patterns (deploy 3-step panel; storage-init/netstorage status poll). They agree on nothing: some use `{ok,data}` envelopes and some raw JSON, some poll 1 s / 1.5 s / 3 s, some are in-memory-only and lie after a restart, and each re-implements single-flight + snapshot + phase→Hungarian mapping. | M | idea | Origin: 2026-07-19 feedback slice 1 (controller v0.147.0). The cases to generalise from are all in-tree: `web/storage_init_job.go` (the best shape — acquire/release/set/snapshot), `web/netstorage_job.go`, `web/samba_ensure_job.go`, `backup/opstatus.go`, `backup/offbox_progress.go`. Shape: one job registry + one poll endpoint + one client-side renderer, phases declared per job. **Two lessons tonight that any framework must encode:** (1) a terminal state must be **probed, not inferred** — `compose up -d` exits 0 on a crash-loop; (2) a progress source that reports nothing is normal, not broken — restic reports 0 bytes for a whole incremental run, and a bar that sits at 0% is worse than no bar. Also fixes the restart hole: in-memory job state currently vanishes and the card silently disagrees with reality  **2026-07-20 — the first bill for NOT having this arrived, and it was customer-facing.** The samba card's poll (`web/samba_ensure_job.go` + `sharing.html`) mixed a job EDGE and a service LEVEL on one JSON field, and `/sharing` reload-looped at ~1.2 s for every customer with sharing enabled until controller v0.151.0 (`audits/DIAG-sharing-2026-07-20.md`, S-1/S-4). v0.151.0 fixed THAT card's contract only — the framework is still this item. **Third lesson for it to encode, beside the two already listed:** a phase a client answers with a one-shot action must be an EDGE the registry SERVES ONCE, and must never be synthesised from a level; if it can be re-read, it will be re-acted on. |
| R-46 | **[P2] Verification copies need a customer-visible browse surface and an expiry.** v0.147.0 made them *visible* (listed with path/size/date, individually deletable) — but the customer still cannot LOOK INSIDE a verification restore to confirm the file they wanted is really there, which is the entire point of a verification restore, and nothing ever removes them. | S–M | idea | Origin: 2026-07-19 feedback slice 4a, registered as the explicit follow-up to it. Two gaps, deliberately designed together because they are the same object: (a) **the invisible-result gap** — a read-only browse of `backups/offsite-restore/<app>` (the FileBrowser infra stack already exists and already serves scoped roots, so this may be a mount rather than new code); (b) **the disk-lifecycle gap** — auto-expiry after N days with the count/size surfaced before it fires, so a drive is never quietly filled by verification restores nobody remembers taking. Pairs with R-43: a browse surface is also how a customer would discover that a DB-indexed app's files came back but the app still cannot see them |
| R-47 | **[P2-HIGH] The DB replay races the application's own schema repair.** | M | **SHIPPED — controller v0.153.0, 2026-07-20** | Fixed on **BOTH** paths (it was a class defect, not an offsite-only one): offsite `ReconstituteFromOffsite` AND local `RestoreFromRecoveryUnit` now open a **DB-only window** — `appbackup.DBServiceNames` names the database SERVICE from the compose `services:` map, `stacks.Manager.StartStackServices` brings up only that service, the dump is replayed with the application still stopped, and the full start runs only after the replay exits 0. **Fail-closed:** a `.sql` dump with no identifiable DB service refuses BEFORE the first mutation. Every exit from the window (replay failure, DB-only start failure) still does a best-effort full start, so a failed restore never leaves a box with a database and no application. `--clean --if-exists` + `ON_ERROR_STOP=1` untouched — the bug was the window, not the flags. Enablers: `RedeployFromEnv` split into `PersistUnitRedeployConfig` + its unchanged tail; `StackDataProvider.RecreateStackFromUnit` → `RecreateStackDefinitionFromUnit` (the hidden `up -d` inside the old name is what carried the defect locally). **No agent coupling — MinAgent stays 0.90.0.** 19 tests + 3 red-proofs, 23/23 green. Unblocks the capability-map **offsite-restore (PARTIAL)** and **customer-restore (MISSING)** rows, both of which now await only one clean acceptance run. **LIVE-VALIDATED 2026-07-20** against the SAME snapshot that aborted in round 2 (`49e7cb46`): log shows `Starting stack immich services only: [immich-postgres]` → replay rc-0 in 20 s → full start; no `already exists`; immich's own DatabaseService logged **`No schema drift detected`** twice (round 2 left it reporting drift); 11 assets `active`, 4/4 containers healthy. **Golden 0.153.0 baked + published the same day** (sha256 `15fdd191f3c660a6…`) — the first golden carrying all FOUR infra images. Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` (H4) + `felhom-controller/REPORT.md` §4b/§4c |
| R-48 | **[P2-HIGH] Restore controls are separable only by layout — and the difference between them is whether the data comes back.** The offsite restore row renders four buttons plus hint text into an overlapping, unreadable line, and the decisive second step („Teljes visszaállítás indítása") appears ONLY after „…előkészítése" was pressed, with no signposting that a second step exists or that the first one did nothing to live data. | M | idea | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` (finding 1) — this is not theoretical: it is the CAUSE of the round-2 incident.** An operator who had read the code pressed the missing-only button instead of the full restore; the controller log shows `/backup/offbox/reconstitute` was never hit at all. The rule this establishes, worth stating once and applying beyond this page: **two adjacent controls whose difference is "your data comes back" vs "your data cannot come back" must not be distinguishable only by layout.** Direction (ruled in principle, spec rides v0.149): collapse to a single „Visszaállítás…" guided dialog — one intent, visible phases, the escrow-wizard precedent. Pairs with R-45 (the phases are exactly the async-feedback surface) and R-46  **SHIPPED 2026-07-21 — controller v0.154.0** (`3a9d744`). Each app row on `/backups/restore` now carries ONE „Visszaállítás…" entry linking to a per-app wizard at `GET /backups/restore/app?name=<app>`: three intent CARDS each with a consequence sentence (ellenőrzés külön mappába / hiányzó fájlok visszahozása / teljes visszaállítás), a visible phase strip so the sequence is legible *before* the first click, danger styling on the destructive card, and the R-43 double-confirm carried over verbatim with its pair-honesty facts. `deriveWizardStep` is a PURE function of (op running, size-gate flash, scratch ready) — the step is never taken from the request, and a running op outranks a stale `?full_prep=` so no commit button survives into a restore. While ANY op runs every mutation form is suppressed server-side rather than offered and then refused. **No new mutation endpoint** (one GET route; every card posts to the pre-existing `/backup/offbox/*` with unchanged field names and gates) and **no R-45 graft** — the wizard polls the two existing status surfaces as-is. Works with JavaScript disabled. Latent bug fixed on the way: `offboxRedirectTo` hardcoded `"?"` when appending its flash, which against the wizard's `?name=<app>` target would have buried the flash inside the app name. 9 new tests + the Group-B red-proof (trivial always-INTENT impl → all 7 rows red). **Live click-through + one non-destructive Ellenőrzés still PENDING** (rides the operator's floor save). Evidence: `felhom-controller/REPORT.md` §3 (2026-07-21). |
| R-49 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. | S–M | idea | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app |
| R-65 | **Buddy-box backup replication, cross-household — two Felhom boxes in different homes replicate backups to each other.** | L | idea (post-alpha, spike-first, 2026-07-22) | The natural big sibling of R-64: two households each hosting the other's encrypted backup tier. Explicitly **spike-first** — the transport is NOT SMB (R-64's live-share protocol is wrong for backup replication across the internet: no auth story between households, no resumability, cleartext LAN assumptions); candidates to spike: restic rest-server / rclone / syncthing over the existing WG/tailnet plumbing, encryption keyed so the buddy can never read the payload. Sits on top of the offsite tier's FILL/OVERSUB thresholds thinking (R-5 aggregate). Flips: would add a "cross-household buddy replication" capability row (currently unlisted). Pairs with R-64 (same topology, different transport + guarantees) |

## Pre-invite checklist — what stands between here and the first remote tester

Not roadmap items in their own right; the short list the 2026-07-18 rehearsal leaves behind.
Everything here is **remote-doable** — the N100 is packed, and none of it needs hands on the box.

| Action | Owner | Note |
|---|---|---|
| ~~Rebuild the golden → 0.146.0~~ **BAKED + PUBLISHED 2026-07-18; awaiting the operator's two saves** | Viktor (saves) | Golden **0.146.0** baked on the drill VM and published to gitea — `felhom-golden/0.146.0/golden.tar.zst`, **sha256 `4834c703162c5437467a329144b1a523019bf5693ab9d439558be7323587e955`**, 612 696 588 B (584 MB archive). All pass markers green: `Result=success`/`ExecMainStatus=0`, **0** FATAL/exclusions, `docker OK (overlay2)`, **all three mounts included** (rootfs + mp0 `/var/lib/docker` + mp1 `/mnt/sys_drive`), pre-delete **404**, upload **HTTP 201**; controller **0.146.0** confirmed baked in. Integrity round-trip independent of the build host: anonymous `GET | sha256sum` **matches byte-for-byte**, ranged GET **206**, `content-length` matches. Teardown per GL-1: guest 9100 purged, token/script/log shredded in-VM, VM powered off, drill disk reverted to `virgin` exactly-as-found, **token-leak grep = 0**. Log retained `180:/mnt/5_hdd/felhom.eu/drill/bake-0.146.0.log`. Now selectable in the hub dropdown (versions: 0.136.0, 0.143.0, **0.146.0**). **REMAINING = operator, password-gated:** Day-0 manifest Golden → 0.146.0 (Agent stays 0.90.0, MinAgent stays 0.90.0 — the v0.146.0 CHANGELOG declares no new agent coupling) → save; **then** floor → v0.146.0 saved **LAST** |
| **Golden ≥ 0.147.x carries ALL FOUR infra images** | — (next bake) | `build-golden.sh` v2.1.0 (2026-07-19) now derives the pre-pull list from the controller binary it is about to bake (`--print-infra-images`) instead of a hand-maintained copy that had already drifted: `felhom-samba` was never added to it, so every golden so far baked **3 of 4** — which is why enabling Megosztás on a fresh box pulled from the registry with zero feedback. **No golden rebuild for this alone**; it takes effect at the next bake. Until then a fresh box still pulls felhom-samba at enable time, which controller v0.147.0's progress card now at least explains |
| **freemail.hu test-send** | Viktor | The open half of R-4; the gmail half closed on 2026-07-18 under `p=quarantine` |
| **C6 — customer performs a restore, unassisted** | Viktor as customer zero | The one open script step in R-3 and still MISSING as capability evidence. Remote-doable on the reborn box — the dashboard is remote |
| **R-11 rulings** | Viktor | Contact channel, tester agreement, alert thresholds (the R-5 gauge thresholds are still pending a ruling) |

## Absorbed / superseded notes in this folder

- `FOLLOWUP-nas-automount-guest-reboot-reassert.md` — **shipped** (agent v0.84/v0.85, CAMPAIGN-3); keep for history
- `FOLLOWUP-golden-default-controller-tag.md` — verify against current golden flow; close or promote to an item
- `FIX-M18-NOTES.md`, `FIX-M19-NOTES.md`, `DIAGNOSIS-f9-storage-registration-gap-2026-06-14.md` — historical diagnoses; superseded by shipped fixes