Replaces the 2026-07-14 DRAFT entirely - it was verified against controller v0.132.0, 51 versions stale, while being cited as authoritative. Now verified against controller 0.183.0 / agent 0.110.0 / hub 0.80.0 / catalog 4252121. NOT ratified. Every statement is labelled [DESIGN] (a decision from the architecture discussion) or [FACT] (file:line, live output, or a citation to _recovery-inventory-2026-07-28.md). Phase 0 verified D5's premise and it FAILED. Ten read surfaces were checked; nine are clean (SMB under both namespace shapes, FileBrowser for local drives, .fab import and download, /api/debug/*, all three ServeFile sites, registering the backup dir as a drive). One is not: FileBrowser binds a NETWORK share at its ROOT while local drives are userdata-scoped, and no IsNetwork() filter guards the deploy dropdown, the per-app migrate targets, or handleStorageMigrateApp. So an app namespace - and under D5 its plaintext secrets - can land somewhere a browsing surface serves with download:true. Not a leak today (the unit's app.yaml is secret-stripped); it becomes one under D5. D5 is therefore written as a BLOCKED target, not as settled. -> R-108 Core artifact is the failure -> recovery matrix (15 rows). RTO cells carry only measured durations; 12 are blank and each blank is listed explicitly as a finding. RPO cells carry the configured cadence that bounds RPO, labelled as such - no RPO has ever been measured from an incident and none is invented. The 7/53 vs 9/43/1 Tier-2 coverage counts are both recorded, unresolved; neither adopted. Coupling (S-1): capability map section C now cites the matrix instead of restating routes; ROADMAP gains R-102..R-108; OPEN-ITEMS gains only the in-flight R-108 plus C9-F1b/F3/F4 -> R-n cross-references; CONTEXT records D1-D6 as standing ruling S-3. No code, no version bump, no deploy, no box touched.
182 KiB
ROADMAP — future features & open work
What this is: the prioritized decision log of planned/open work. Items are intentions, not claims about live behavior — the capability map (
architecture/00-capability-map.md) is the only place that states what the platform does today.Lifecycle: idea → spiked → spec'd → in-progress → shipped (item collapses to a one-liner with the version, and the corresponding capability-map row changes status with evidence). Items can also be killed (keep the one-liner + why — decisions are worth remembering).
Coupling rule: every item names the capability-map row(s) it flips. Every map gap row points back here by ID. Neither file duplicates the other's content.
Priorities: P1 = closed-alpha blocker · P2 = close during alpha · P3 = post-alpha. Existing loose notes in this folder (
FOLLOWUP-*,FIX-M*) are absorbed as references below.
P1 — closed-alpha blockers
| ID | Item | Size | Status | Notes / map rows flipped |
|---|---|---|---|---|
| R-1 | Peti convergence — the appliance half is DONE; this item is now Peti-only. Rehearsal EXECUTED 2026-07-18 (tests/VALIDATION-n100-rehearsal-2026-07-18.md): the full final-product flow ran on real metal in one pass (RESET → generic ISO → customer self-bind → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer — the one thing a demo box on the operator's own LAN can never prove. |
L | rehearsal DONE; Peti half open | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. Reinstall arc SHIPPED hub v0.57.0 (2026-07-16): the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path |
| R-2 | ~hub/internal/notify/, store.go, hub/internal/claim/) |
S | killed (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — notify/+claim/engine.go in 6b40eb8 (v0.50.0), store.go in a1d0450 (v0.54.0), plus follow-up e205a2d; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
| R-3 | Friend-alpha onboarding runbook (generalized from pilot/RUNBOOK-peti-return-2026-07-13): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step |
M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of PETI-tester-agreement.md. Next from-scratch rehearsal to include customer DELETE + re-create — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). S6b (manual stale-host delete before re-enroll) is OBSOLETE — re-enrollment upserts the existing host row cleanly (store.UpsertHost ON CONFLICT DO UPDATE; handleAdminCreateHost no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE): one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). Remaining consistency gap: the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC p=none → p=quarantine |
S | DONE 2026-07-21 (all three halves) | A claim code in spam bricks onboarding at step 1. DMARC now p=quarantine (RUNBOOK-publish-0.90-0.143-2026-07-18 Phase 4): _dmarc.felhom.eu TXT v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu set by Viktor (CF, Path B); dig gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM d=felhom.eu + relaxed send. SPF). gmail half CLOSED by the rehearsal (2026-07-18): the run's claim email was the first sent under p=quarantine and landed in the gmail Inbox, not spam (tests/VALIDATION-n100-rehearsal-2026-07-18.md). freemail.hu half CLOSED — operator-verified 2026-07-21: a [Felhom] Teszt értesítés was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by p=quarantine, R-4 is COMPLETE |
P2 — during alpha
Sub-rank
P2-HIGH= close before the first REMOTE tester. These are the 2026-07-18 N100 rehearsal's findings (tests/VALIDATION-n100-rehearsal-2026-07-18.md). They are not P1 — the rehearsal proved the product flow works — but each one either misleads the operator, misleads the customer, or hides a failure, and all of that gets materially worse the moment the box is somewhere you cannot walk over to.
| ID | Item | Size | Status | Notes |
|---|---|---|---|---|
| R-39 | [P2-HIGH] The PBS DR tier can be applied and dead at the same time — and nothing notices. On the reborn N100 the descriptor auto-provisioned and the agent converged state=applied, yet pvesm status reports felhom-pbs: error fetching datastores - 401 Unauthorized / inactive and a direct probe with the stored credential 401s on every endpoint including /version (WG transport healthy: handshake 9 s, 27.9 ms RTT — so authentication, not ACL scope). Three compounding defects: (a) a mint/consume race — the hub minted a SECOND token secret at 16:47:52, two minutes after the agent applied the first, and consumed_at is still NULL; (b) the converged state machine will not re-apply, so the box is pinned to a stale secret; (c) the agent's 15-minute PBS verify loop cannot read the credential to detect any of it (open /etc/pve/priv/storage/felhom-pbs.pw: permission denied — the non-root agent writes that file through a root sudo wrapper, then reads it directly). |
M | CLOSED 2026-07-21 — PROVEN LIVE (hub 0.68.1 + agent 0.91.2) | DIAGNOSIS (2026-07-18, live on the N100 — supersedes the initial hypothesis). The brief guessed "the re-mint fails to bump the generation". That is FALSE and no hub fix was shipped: store.SetHostDesired bumps desired_generation unconditionally (it went 2→3 on the re-issue), and web/configs.go's applyPBSDR is likewise exonerated — its "no re-key, no second secret, no spurious generation bump" comment is accurate, guarded by the cur != nil && cur.Namespace != "" early return, and the hub log shows mint #2 came from the re-issue path, not from an Edit-tab Save. The real mechanism is a signal mismatch between the two tiers. The hub's re-consume signal is a generation bump + a poke; the agent's re-apply trigger is a change in the DESCRIPTOR CONTENT HASH (felhom-agent internal/pbsdr/manager.go ~L235: `if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil |
| R-30 | [P2-HIGH] Liveness presence should come from the wait channel, not the report clock. The box was powered off at the start of the rehearsal, yet the hub carried it as healthy until the staleness threshold expired ~30 min later (host_stale 16:05:24 "no report for 30m"; cleared 16:33:24 "was stale for 27m"). The host-delete guard compounds it: RESET refuses while any host row exists, so a stale-but-"Online" host stalls a forced teardown. |
M | idea | Direction: derive presence from Dir-2 long-poll connectedness (~90 s grace), decoupled from notification hysteresis (the hysteresis is right for alerting, wrong for presence); an agent/ep0 analog can follow. Pairs with R-13/R-23 — the transport already exists, this is about believing it. (Discussed in-session as "R-29"; that number was already taken by the gate-rot item earlier the same day, so it is R-30.) |
| R-31 | [P2-HIGH] Offsite provisioning is synchronous with no status affordance. Save runs the Hetzner sync in-request, so the request can hit the nginx 504 while succeeding server-side: the operator cannot tell failed from slow, and a retry races the first attempt. | M | idea | Direction: make it async + a status card, reusing the proven awaiting-card/poll idiom (v0.138.0 escrow card). Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click. |
| R-32 | [P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible. The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's "hetzner":"ok" leg destroys the sub-account, but a Hetzner sub-account is an access-control object, not a data object — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. |
M | idea | Ruling from the run (three parts, deliberately separate): (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable BY DESIGN → RESET gains a main-account purge of the customer base dir (the existing operator ack already covers it); (2) the move-aside guard STAYS for reinstall-without-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator Restic tab shows per-customer directory bytes vs attributed snapshot bytes, so dead data cannot hide. Measured on the pool box that night: 49 M attributed (2 snapshots, 48.717 MiB) against 1.4 G + 3.0 M unattributed across TWO .orphaned-* dirs. Evidence restic-and-pool.txt |
| R-33 | [P2-HIGH] Bootstrap pairing-poll spams the customer-visible console. Each 30 s pairing poll is a separate felhom-bootstrap.service run that exits 1, so systemd prints Failed to start … on the screen the customer is watching — 52 FAILED lines in ~11 minutes while nothing was wrong (the box was correctly waiting to be bound). |
S | SHIPPED (scripts v1.21.0, 2026-07-18) | run_pairing() is now a while true loop sleeping POLL_INTERVAL (30s — the hub-side rate is unchanged) instead of exiting non-zero per poll, so the unit sits in activating and systemd prints nothing. Registration split into register_appliance(), whose transient failures the loop retries rather than taking the unit down. The journal is quiet without going dark: the wait is logged once on entry (worded so nobody reads it as an error), then on a 10-min heartbeat with elapsed minutes; 410 still exits non-zero on purpose (delivery consumed but no local env is a real crash window). Console banner every 5 min instead of every cycle, single accented spelling, plus the reassurance the rehearsal showed was missing („Ez a képernyő magától frissül"). The load-bearing half is TimeoutStartSec=infinity in the unit — a Type=oneshot ExecStart is killed at DefaultTimeoutStartSec (90s), so without it systemd would kill the new wait and Restart=on-failure would silently reinstate the exact spam this removes, after appearing to work for the first three polls. Verified behaviourally in a container against a stub hub answering 204 five times then delivering: one log line plus one heartbeat, zero exits between polls, then a clean fall-through to the direct install and exit 0 — where the old design produced 5 unit invocations and 5 Failed to start console lines. ISOs rebuilt at v1.21.0 and the shipped payload verified byte-identical to the committed source. Evidence bootstrap-journal.txt; ship commit bcdb042 |
| R-5 | Hub: offsite storage visibility — RESTIC box aggregate (v0.64.0) + PBS DR datastore (v0.65.0), each with fill + operator alert; Offsite tabs + dual dashboard gauges | M | SHIPPED (hub v0.64.0 + v0.65.0 + tenantsync v1.2.0, 2026-07-17) | Restic: monitor.OffsiteBoxChecker (throttled Hetzner GET; fill 80/90% + oversub 2.0×; scope "pool-box"). PBS DR (v0.65.0, Option A): read-only usage op on the ep0 felhom-tenantsync forced command (df, no mutation) → monitor.PBSDRBoxChecker (fill 80/90%, states ok/unavailable/degraded, scope "pbsdr-box"); /offsite Restic/PBS-DR tabs + two gauges; hub deploy ⟂ ep0 update (graceful "n/a"). Both data pipelines PROVEN-LIVE (restic 0.2% full; PBS 19.1% full, 7.1 GB of 37.2 GB after ep0 → v1.2.0). Thresholds pending Viktor (named keys; separate restic/PBS pairs). OPEN: UI render unit-only (password-gated), alert emails unit + red-proof only (both nominal). See hub v0.64.0 + v0.65.0 REPORTs |
| R-6 | Spike: LAN service discovery from the guest — SSDP multicast (UDP 1900, DLNA), WSD (Windows discovery), mDNS; host-network vs macvlan; is the customer LXC LAN-bridged in appliance deployments? | M | spiked (2026-07-18) | VERDICT: appliance guest IS LAN-bridged (own DHCP lease on the household /24); multicast discovery works ONLY in the guest netns — guest-direct or Docker --network host (SSDP/mDNS/WSD all PASS both ways); the default docker bridge is categorically DEAF to LAN multicast (WSD/mDNS RX FAIL, unicast-publish PASS). Real samba+wsdd on host-net → Windows 11 ProbeMatch + FELHOM-SPIKE renders in Explorer + 445 + authenticated SMB round-trip all PASS; real SSDP MediaServer:1 advert reaches both LAN clients. → R-7 SMB stack MUST be host-network LAN-bound; R-8 Jellyfin-DLNA plausible if host-network. Caveat: vmbr0 multicast_snooping=1 worked only because the household router is a live querier — customer LANs w/ snooping+no-querier, and Peti's BYO bridge, are UNTESTED gaps. S4b (human leg, the sharpest finding): wsdd makes the box VISIBLE but the Explorer double-click FAILS 0x80070035 — WSD gives no name resolution; the flat \\FELHOM-SPIKE resolved by no path. Adding nmbd (NetBIOS) fixed it live (flat name resolves + mounts). → R-7 needs smbd+wsdd+nmbd (+avahi/.local for modern clients), not wsdd alone. Doc: audits/SPIKE-lan-discovery-2026-07-18.md. |
| R-7 | SMB server share — LAN file sharing as an embedded controller feature („Megosztás") | L | SHIPPED slice 1 (controller v0.144.0 + felhom-samba:1.0.0, 2026-07-18) |
Flips map row Files from Windows Explorer / Mac Finder (SMB server) MISSING → IMPLEMENTED. Delivered as the FOURTH protected infra stack (host-net, smbd+nmbd+wsdd, bind interfaces only = lo eth0), NOT a catalog app. „Megosztás" → „Hálózati megosztás": enable + ONE household password (STDIN→smbpasswd, never persisted — only user_set), shares table, create flow (new folder under <storage>/shares/ or an existing folder via a guarded picker). Picker guard = absolute → EvalSymlinks → containment in a registered LIVE root → deny-listed system subtree (derived from ProtectedHDDPaths, provably a subset) → is-a-dir, with UNIFORM refusals (no filesystem oracle). Per-share read-only enforced in smb.conf AND as a :ro bind. force user = felhom → SMB writes land as uid 1000. Live-validated on demo through the real endpoints + a Win11 workstation (445 True, NetBIOS FELHOM resolves, write/read byte-compare PASS, write to a read-only share REFUSED, WSD ProbeMatch). Explorer leg PASSED 2026-07-18 (Viktor): both shares open from the Network view; an Explorer save into the writable share landed as uid 1000 and a write into the read-only share was refused, folder untouched. Slice 1 is fully PROVEN-LIVE. Docs: controller/sharing.md; ship report felhom-controller/REPORT.md. Slice 2 remainder: avahi/.local for modern+Apple clients, curated app-folder presets (one-click paperless consume/), per-share users + guest access, recycle bin. Backup gap split out → R-7b. |
| R-7b | Share backup EXECUTION — put share data into the live tier-2 + offsite runs (the design fork reported by R-7 slice 1) | M | SHIPPED (controller v0.145.0, 2026-07-18) | Viktor's ruling: Model B′ — a SIBLING shares source. New, additive job/leg code reusing the proven primitives (tier-2 mirror seam, restic wrappers, soft-quota/enlargement gate, status recorders) while leaving every per-app engine path byte-identical — NOT a synthetic recovery unit (breaks on multi-drive shares, wraps 1 KB of JSON in dump machinery) and NOT engine-loop surgery. The B′ invariant is enforced by test in both tiers, red-proofed. Tier 2 → RunSharesTier2 (legs grouped by SOURCE drive → backups/secondary/_shares/<driveKey>/<share>, payload at _payload/, layout marker LAST). Tier 3 → runOffboxSharesLeg: ONE extra restic backup --tag felhom-offbox --tag _shares placed after the app loop and BEFORE retention, so forget --group-by host,tags covers the new group with no flag change; a quota-blocked push degrades to the manifest only, never to nothing. Restore → „Megosztások" on /backups/restore: scratch, then a missing-only merge whose every destination is PREFIX-ASSERTED against live storage roots, definitions merged existing-wins, then ReconcileSamba, then the credential. The payload (_shares-manifest.json + a best-effort secret-bearing passdb.tar) is what makes DR return files + configuration + password rather than loose bytes. Fold-in: samba joins the liveness set — EffectiveProtected adds the CONTAINER felhom-samba exactly while sharing is on. FULLY PROVEN-LIVE on demo (2026-07-18), all four legs. (1) tier-2: real /api/backup/tier2 trigger → _shares tree + marker + payload on the cross-drive target, mirrored file md5-identical, payload 0600 preserved. (2) offsite: Viktor's manual run 12:18:16Z → snapshot e0b9d723 (tags felhom-offbox,_shares) with the payload dir + both share folders; a second run via the „Távoli mentés" button → 4e2b15ec, containing _shares-manifest.json (418 B) AND passdb.tar (855 040 B), both 0600, share files with uid 1000 preserved. (3) restore round-trip: probe file + the dokumentumok DEFINITION deleted via the real endpoints, then „Megosztások" restore + place → 1 file(s), 1 definition(s) re-added, 1 kept, 0 refused, credential=true; probe back md5-identical, the two pre-existing files NOT overwritten (missing-only proven on live data), definition back with its ORIGINAL flags and created_at, smb.conf re-rendered, filmek untouched. (4) liveness: samba stopped → health_critical pushed and hub-accepted (200) → self-healed. Remaining human leg: SMB positive auth with the real household password (never persisted by design). Correction: an earlier revision of this row and of the ship REPORT wrongly claimed the demo box had no offsite target — the verification read a guessed settings key (offbox_target) instead of the real one (offbox); root cause dissected in REPORT §7b. Findings: the reserved-name assumption was FALSE (nbNameRe accepted „_shares" as a share name — now refused); the alert/e-mail pipeline needed NO change and adds no new event type. Docs: controller/sharing.md; ship report felhom-controller/REPORT.md. |
| R-8 | DLNA (gate input now exists — R-6 spiked 2026-07-18: SSDP reaches LAN clients from host-net): validate Jellyfin's built-in DLNA server first; only add minidlna to the catalog if Jellyfin-DLNA fails | S | idea (unblocked) | Don't add catalog weight before proving the cheap path. R-6 confirmed the cheap path is physically viable — Jellyfin DLNA must run host-network (same multicast constraint as R-7) |
| R-9 | Uninstaller trio (from 07-15 Peti session): cluster-aware felhom_guests guard (node-local pct list deletes cluster-wide pveum objects); saferemove detection + time estimate + opt-in --quick-remove (never mutate storage.cfg); smarter restore_storage default for BYO clusters (shared storage, not local-lvm) |
M | idea | Second item's rejected alternative (temp-disable-and-restore) stays rejected — crash window silently downgrades cluster wipe policy |
| R-10 | T-6E-1: DB-dump dir-fsync asymmetry (LOW, confirmed in 6E) | XS | idea | One-line hardening; batch with the next controller task |
| R-11 | Tester-facing one-pager: what the box does, known limitations, how to report | S | RULED 2026-07-21 (channel); doc is the architect's | Pairs with R-3. operator ruling 2026-07-21: the reporting channel is DIRECT MESSENGER to the operator — no group, no tracker, no form. Rationale: the expected tester count at first invite is ~0–few, and a channel nobody is on is worse than a channel everyone already uses. The one-pager and the tester agreement are the ARCHITECT's deliverable, not CC's — CC does not write them; it may be asked to fact-check claims in them against the code. Revisit the channel only if tester count makes direct messages unmanageable |
| R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17): the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above |
| R-22 | PBS-DR pre-check self-grant (F4). On a non-default storage id the token-auth GET /storage/<id> pre-check 403s (no ACL yet) and used to abort before the root-run grant that creates it. |
S | SHIPPED + PROVEN-LIVE agent v0.89.0 (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → converged state=adopted in ~3 s, ACLs restored, offsite active). Origin tests/VALIDATION-n100-baremetal-2026-07-16.md F4. |
| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | CLOSED 2026-07-22 — archive deleted (operator console) | operator ruling 2026-07-21: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls: (a) felhom-tenantsync.sh does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (hetznerapi.DeleteSubaccount), but its ONLY caller is offsite.Deprovision, which selects by the felhom-customer=<id> label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (hub/internal/hetznerapi/fake.go generates the u629193-sub%d pattern; felhom-controller/.../dr_recipe_test.go uses it as a literal) — nothing live points at it. Operator console steps: Hetzner Storage Box console → box u629193 → Sub-accounts → u629193-sub1 → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. CLOSED 2026-07-22: the operator executed the console action — u629193-sub1 deleted, confirmed no sub-accounts remain on PBS-storage-1 |
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
| R-20 | XS | closed (2026-07-16) | Confirmed against scripts/felhom-host-install.sh source (not changelog): keys resolve at L1181–1219 (script constants OPERATOR_KEY_*, populated, --operator-pubkey-file override), pinned automatically by step_agent_config() "STEP 6/8" (L2044; python builds authz.signers L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
|
| R-23 | Immediate-sync Direction-2 follow-ups (hub v0.58 / controller v0.140, 2026-07-16): (a) — BANKED 2026-07-21 (both legs). The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub 18:56:27 CEST) released the controller's held wait in the same second (16:56:27Z wait woke: generation=1 - firing out-of-cycle report), with the report built 2 s later; the ring also shows wait baseline generation=0 at startup (baseline recorded WITHOUT firing, as designed) then generation=1, so the generation advanced past 0. RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE. Operator saved global floor 0.153.0 → v0.154.0 (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): 06:57:13 UpdateState pending written initiated_by=auto-floor → 06:57:17 agent controller swap requested 0.153.0 -> 0.154.0 → 06:57:19 image file written, restarting bootstrap → 06:57:21 container StartedAt + UpdateState completed_at → 06:57:29 agent new controller healthy. 16 s end to end. Assertions over the whole window (06:50 → 07:29, 39 min): controller swap requested = 1, agent-driven bootstrap restarts = 1, new controller healthy = 1, rollback/swap-failed/unhealthy = 0, container RestartCount = 0. VerifyStartup banked it on the next boot (Post-update startup: update successful (0.153.0 → 0.154.0)) and the 06:57:52 periodic check logged Current version 0.154.0 is up to date — the at/above-floor branch correctly doing nothing. No storm, no rollback, no second attempt. Caveat, disclosed: a hand-deploy of v0.155.0 at 07:17:10 sits inside the observation window and is what StartedAt shows after that point; it never touches SwapController, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged Current version 0.155.0 is up to date and did nothing. Evidence: felhom-controller/REPORT.md §6 (2026-07-21). (Superseded note:) the self-restart single-fire leg was - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. Trap found while banking this: the wake is logx.Debugf, so it is INVISIBLE in docker logs at INFO and lives only in the debug ring (GET /api/debug/logs?level=DEBUG) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (pollOnce blocks ~240 s), not at reconnect |
S | (a) BANKED in full; only (b) cosmetic remains | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: felhom-controller/REPORT.md 4f |
| R-34 | Backup data lifecycle management. An "inactive backups" section on „Távoli mentés": apps that have snapshots but no active backup — disabled OR uninstalled — listed with name / size / last snapshot / restorable, plus an explicit double-confirmed per-app delete via restic forget --tag + nightly prune. |
M | idea | RULING: the offsite toggle NEVER offers deletion — policy and destruction stay decoupled. Turning backups off must never be a data-destroying act, and deletion must never hide behind a toggle. Origin: 2026-07-18 rehearsal. Pairs with R-32 (that one is the operator's view of dead bytes; this one is the customer's) |
| R-35 | Config-apply should not end the customer's session. The offsite config push bumped config_version 10→11 at 16:54:58 and the controller self-restarted (container StartedAt 16:54:59Z, back up 16:55:02); in-memory sessions died with it and customer zero was force-logged-out mid-flow. |
S | idea | Direction: hot-apply the offbox target (no restart for a config the running process can adopt), or persist sessions across restart. The restart itself is by design — the collateral is not. Evidence controller-log-full.txt |
| R-36 | Post-RESET re-enroll leaves offsite "enabled but unprovisioned" — silently. The hub knows the state and says nothing on the customer page. | S | SHIPPED (hub v0.67.0, 2026-07-18) | Both halves delivered. (1) The warning: the customer page now names the state and the fix — enabled-but-unprovisioned raises an amber banner saying provisioning is Save-triggered (press Save once, then verify), reusing the exact enabled && type == "" predicate the offsite re-issue handler already refuses on. (2) The related sub-item, also done: the self-bind link is now auto-minted at customer creation AND at RESET completion, so the console banner's „e-mailben kapott link" is already true instead of true-once-the-operator-remembers. Extracting the shared mintAndSendSelfBindLink core keeps the button and the auto-mint callers on the same F1/F2 honesty rules, and the auto-mint never fails the operation it rides on. Gap found and closed while wiring it: PurgeCustomerResetDBState does NOT clear selfbind_tokens, so a link minted BEFORE a reset would have stayed live across it — the skip paths now clear stale tokens, giving the invariant "after auto-mint, the only live link is one we just issued, or none". Tests assert the banner is ABSENT in all three nominal cases too; red-proofed. — Original analysis: Source-cited behaviour, confirmed live in the rehearsal: provisioning is Save-triggered (configs.go applyOffsite) — which also answers S6's open question — and the re-enroll auto-re-issue correctly skips unprovisioned targets (handler.go). So nothing is broken; the gap is that nobody is told. Direction: flash it on the customer page. Interim: an R-3 step. Related sub-item: auto-mint the self-bind link on customer create/RESET, so the console banner's „e-mailben kapott link" is always already true instead of true-once-the-operator-remembers |
| R-27c | Customer self-bind, slice 2 — console-passphrase bind. Viktor's direction: bind using a passphrase shown on the box console, alongside (not instead of) the emailed capability link. | M | idea | Security constraints from the session ruling, all load-bearing: passphrase issued at customer creation; the global-lookup endpoint must be spray-hardened — per-appliance and per-IP caps, constant-time comparison, a single generic failure (no oracle), alerting on abuse; an accent-free wordlist (console keymaps are not Hungarian); the web capability-link path is RETAINED; claim-by-email is RETAINED as the delivery-channel proof. Also under this item: the self-bind email gains the public universal-ISO download link + two-line instructions (the DIY case). Secret-bearing per-customer ISOs are ruled OUT. Sibling of R-27b (second-box flow) — different axis, both build on the same /bind/ page |
| R-50 | [P2-HIGH] Island-bridge control plane — make controller↔agent independent of the LAN. The agent's localapi binds a LAN literal (listen_addr) and the guest dials that same literal from bootstrap.json. Move both onto a host-internal bridge with a fixed, private address that no router, DHCP lease, or site move can invalidate, so the control plane is location-independent and golden-image-stable. |
M | SHIPPED 2026-07-25 — fleet-migrated (agent v0.96.0 + host-install v1.19.0) | DONE (2026-07-25). Both demo boxes migrated to the island, apps served throughout (0 container restarts); island /storage HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), hub reports 0.96.0. Fresh appliance installs are born on the island (host-install v1.19.0 default; agent auto-attaches net1 — A4 proven live). Migration procedure: runbooks/RUNBOOK-island-migration.md. One-host fleet only; the 2-node cluster (Peti) is R-74 (SDN/bridge parity), a separate supervised runbook. ── Spike/impl detail (superseded, kept for provenance): GO (2026-07-25 PM, audits/SPIKE-island-bridge-2026-07-25.md): validated end-to-end on a real nested-PVE drill (drill-r50, qm300 on the t740, installed via the v1.25.0 ISO through the actual day-0). All probes PASS: vmbr9 portless island bridge (vmbr0 untouched); guest island NIC eth1 169.254.253.2/30 hot-added (LAN leg undisturbed); F1 replay = the money shot — LAN moved .176→.200 with listen_addr on the island → agent stays active, control plane HTTP 200; the LAN-literal contrast reproduced the 2026-07-20 bug verbatim (localapi: bind 192.168.0.176:8443 → daemon exit 1); dnsmasq trap CONFIRMED LIVE and its lan_resolver.host_ip=LAN IP fix PROVEN; pin address-independent (served leaf SHA-256 unchanged, HTTP 200 over the island — no cert re-issue); survival matrix (agent restart / guest reboot / host COLD reboot) all return the control plane on the island with zero intervention. Method caveat: probes drove the runtime chain via manual config edits — the provisioning path (host-install writing the island config + golden-bake bootstrap template) is the impl task, now de-risked. Next: write the Phase A/B/C production spec (drill-proven → demo → Peti SDN). — Prior (superseded): 2026-07-25 spike: the drill environment (qm300 / demo-vm-felhom-2f4b00) is GONE (qm list empty; only guest 9201 remains), so per the hard drill-only rule the empirical probes (bridge create / NIC hot-add / island bind / F1 replay / survival) were NOT run — production was left untouched, no live GO/NO-GO. The source-grounded half IS done: recommend link-local 169.254.253.1/30↔.2/30 (structurally uncollidable vs LAN); the F1 literal is baked in TWO places to move atomically (config.go:229 + felhom-host-install.sh:2226 for the bind, provision/backhalf.go:129/bootstrap.json for the guest dial); NEW dnsmasq trap CONFIRMED in source — LANResolverConfig.WithDefaults (config.go:208–210) derives the DNS listen-addr from listen_addr, so moving the bind to the island silently kills LAN DNS → the spec MUST set lan_resolver.host_ip = LAN IP explicitly; pin is leaf-DER-based (address-independent → no cert re-issue expected); provisioning inventory + cluster-parity (SDN on Peti's 2 nodes) + an implementation skeleton recorded. Empirical attempt 2 (2026-07-25 PM, on the t740 per the operator's drill-host ruling): STILL BLOCKED — there is NO drill VM on the t740 (qm list empty; it is a real demo node running only its own guest 9201). The only drill artifact is a stale golden-bake drill.qcow2 on the production DooPlex node (off, nested guest purged). The ruling ("drill+build VMs on the HP from now on") is forward-looking and not yet realized. Operator decision: they will provision a nested-PVE drill VM on the t740 (agent + nested guest); this spike re-runs then. GO/NO-GO PENDING. Remaining: a provisioned t740 drill VM to validate probes P2–P7 before the production spec. Origin: audits/AUDIT-vacation-remote-ops-2026-07-20.md (F1), where this failed for real. The demo box moved to a remote site, DHCP handed it .147 instead of .162, and the agent then could not start at all — bind: cannot assign requested address, systemd gave up after 4 retries — taking storage, PBS backup, quiesce, restore-test and DR down for as long as nobody noticed. Mitigated for that window by pinning vmbr0 static back to .162; that is a window mitigation, not the fix — it still depends on the site's subnet being 192.168.0.0/24 and free at that address. Spike-first is mandatory: validate end-to-end on the drill environment (agent bind + guest dial + TLS SAN/pin + reinstall/golden survival + the bootstrap-config migration for already-deployed boxes) BEFORE any production spec. Pin fact (verified 2026-07-20, agentapi/client.go L105-129 — supersedes the earlier "the SAN set must cover the new address" note in this entry, which was wrong): the controller-to-agent leg sets InsecureSkipVerify: true and replaces chain verification with a custom VerifyPeerCertificate that does a raw SHA-256 match on the leaf DER against the bootstrap fingerprint. Hostname/SAN therefore never enters verification on this leg, so moving the agent listen address most likely needs no cert re-issuance - only the endpoint the guest dials. The spike must still confirm this empirically rather than trust the read. Flips a future "box survives a site/network change" map row |
| R-50b | [P2] A root-owned privileged host artifact is delivered unversioned from main — "which wrapper is on this host?" is unanswerable. configs/felhom-pbs-apply installs to /usr/local/sbin/felhom-pbs-apply (0755 root:root) and is the pinned sudoers vector for create|reconcile|grant against /etc/pve/priv/storage. It is fetched by felhom-host-install.sh:1914 via fetch_raw, which hits raw/branch/main/<path> — no tag, no pin, no checksum, and no record in the Day-0 artifact manifest, unlike the agent binary (sha256-vouched) and the golden image. Three consequences: (1) two hosts installed a week apart can carry different privileged wrapper code while both reporting the same agent version; (2) a host hotfixed in place (felhom-pve, 2026-07-18) is indistinguishable from one that fetched the same content — the fleet has no inventory of it; (3) an accidental push to main reaches the next install of every host with no review gate between commit and root-owned deployment. |
S–M | (a) SHIPPED 2026-07-21; (b)/(c) open | Surfaced 2026-07-21 while stopping the R-39 v0.90.1 publish (felhom-controller/REPORT.md §5): the publish was cancelled precisely because the version number would have claimed to carry a fix that in fact rides this unversioned channel. Candidate shapes, in increasing cost: (a) record the wrapper's sha256 in the Day-0 artifact manifest beside the agent binary and have the agent report the installed file's hash, so drift is at least visible; (b) fetch_raw takes a pinned ref (tag or commit) supplied by the manifest rather than main; (c) the wrapper becomes a published generic-registry artifact with the same gate ladder as the agent binary. (a) is the cheap honest first step and would have caught this class already. Pairs with R-39 (whose remaining fleet half is specced separately) (a) SHIPPED 2026-07-21 — hub v0.68.0 + agent v0.91.2. ArtifactManifest.WrapperSHA256 + an operator field; agents report the installed wrapper's sha256 each cycle and the host page surfaces a mismatch. An unknown on EITHER side reads as quiet, never as drift — lighting every host amber on rollout day is how a warning becomes background noise. Live confirmation of exactly the problem: felhom-pve's July-18 in-place hotfix hashed 2888f2ea…, matching no commit anyone could name; it now reports 104db0a4… against a vouchable manifest value. (b)/(c) REMAIN OPEN: the wrapper is still fetched unversioned from raw/branch/main — this makes drift visible, it does not fix the channel. Also recorded: the 0440 sudoers file is not agent-readable, so its drift stays invisible. |
| R-51 | Dead-primary alerting — a multi-container app whose MAIN container is dead must alert. Aggregation currently classifies such a stack unhealthy, and IsDownState deliberately excludes unhealthy, so nothing fires. |
S | SHIPPED 2026-07-21 — controller v0.156.0 | Origin: AUDIT-vacation-remote-ops-2026-07-20.md (F4). Observed live: immich-server was Exited for 18 h with the app 100 % unreachable, and the box produced no dead-app banner and no app_start_failed hub event — while single-container Calibre-Web, down for the same reason, alerted correctly within 90 s. Constraint (load-bearing): do NOT simply fold unhealthy into down. That exclusion is deliberate (stacks/manager.go fix-3, downstate_test.go) and reverting it reintroduces the flapping it was added to stop. Direction: distinguish member-container-exited from healthcheck-failing in the aggregation, and treat a dead primary as down |
| R-52 | Boot desired-state reconciliation — a deployed: true app should be running after boot. The controller reports deployed-but-stopped apps (30 s deadapp-check) but never starts them, so an app that misses its boot start stays down until a human notices. |
M | SHIPPED 2026-07-21 — controller v0.156.0 | Origin: AUDIT-vacation-remote-ops-2026-07-20.md (F5). Observed live: the pre-transport shutdown left immich-server and calibre-web Exited; 10 sibling containers came back and those two did not, and they were still down ~18 h later. Includes root-causing why restart: unless-stopped did not resurrect them — both were stopped ~25 s before power-off, so Docker most likely recorded them as user-stopped; that hypothesis is untested because the guest journal is volatile and the controller's own logs were rotated by the container recreate. Direction: a bounded start-once reconciliation (N attempts, reusing the existing boot grace), never a restart loop. Pairs with R-51 — that one is the alarm, this one is the recovery |
| R-54 | [P2-HIGH] The guest's DHCP client is unsupervised — its death takes the box off the internet 1-2 hours later, invisibly. ifupdown starts dhclient once at guest boot and nothing restarts it. |
S-M | SHIPPED 2026-07-21 — agent v0.92.1 | Origin: audits/INCIDENT-guest-dhclient-killed-2026-07-20.md §5 "OPEN RISK" — this row closes it. On 2026-07-20 a cleanup step killed guest 9201's dhclient (visible in the HOST's pid namespace; §4's /proc/<pid>/cgroup rule exists because of it). The guest then kept working for another ~80 minutes on its unexpired lease; only at expiry did the address and default route vanish, taking the Cloudflare tunnel, hub reports, catalog sync and the controller→agent channel with them — 1h15m outage, and every observable signal said healthy for the first 80 minutes. The design consequence: liveness of the DHCP client is itself a probe. internal/guestnet flags a DHCP guest unhealthy on pgrep -x dhclient alone, while the lease is still live — waiting for the IP to disappear is waiting out precisely that silent window (red-proof: reverting to IP-presence-only makes the July-20 fixture report healthy with zero heals). Four fixed-shape pct exec probes (address / default route / /etc/network/interfaces mode / client liveness, parsers pinned to output captured live from 9201), the incident's restored invocation as the heal, verbatim, and dampers throughout: two CONSECUTIVE bad probes, ≥10 min between heals per guest, ≤3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on a static guest (dhclient must never fight a static config — reported loudly and left to R-50, which is where option 2 of the incident's three choices belongs), on an unknown mode, on an unprobeable guest, or on an ownership-unproven guest list (source is ListLXC ∩ the felhom pool, audit A1). Host-tier by necessity: a guest with no default route cannot repair its own default route. A live finding during deployment: the first sweep on felhom-pve logged dhclient liveness probe failed: sudo: a password is required and reported state=unknown — fail-safe, but blind. TASK-D assumed no sudoers change was needed; three of the four probes had no grant. FELHOM_GUESTNET + four guestnet-* capability rows shipped in v0.92.1 (v0.92.0 superseded, do not vouch). Healthy cycle PROVEN LIVE 2026-07-21 on felhom-pve: caps 68/68 ok, degraded=0 and level=DEBUG guestnet: guest network healthy vmid=9201 mode=dhcp has_route=true dhclient_alive=true. The HEAL leg (STOP-2, a deliberate replay of the incident) is operator-present and pending. Note the guest and the host still differ (host static since the F1 mitigation, guest DHCP) — choosing one for both remains R-50's call, not this row's. Evidence: felhom-agent/REPORT.md (2026-07-21) PROVEN LIVE 2026-07-21 (STOP-2, operator-present) — a deliberate replay of the incident. kill -9 on guest 9201's dhclient at 12:43:18 CEST (pid 336708, cgroup=0::/lxc/9201/ns/.lxc verified BEFORE the kill — the incident's own §4 rule, applied). 12:44:15 — 57 seconds later — detected on process liveness alone, with the address and default route still perfectly present and 83 minutes of lease remaining: unhealthy (first bad probe — not acting yet) bad_probes=1 required=2. 12:45:15 second consecutive bad probe → healing, cmd="pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0" (the incident's invocation, logged verbatim before it ran) → 12:45:18 guest network healed ip=192.168.0.104 has_route=true dhclient_alive=true heals_last_hour=1. Healed 120 s after the kill — ~80 minutes before the outage would have begun. The strongest evidence is what did not happen: cloudflared stayed Up 29 hours, DNS resolved, the hub answered 302 in 0.16 s and the public URL 302 in 0.25 s. The 2026-07-20 outage was prevented, not merely detected. Exactly ONE heal, zero re-heals. The damper then proved itself against a REAL transient, unplanned: the STOP-1 guest reboot at 12:53:16 produced unhealthy … no IPv4 address on eth0 bad_probes=1 — no action — then guest network recovered on the next cycle. A single blip really is not a diagnosis, and a booting guest is not injected with a dhclient. Negative leg run in full: 12:45:18 → 13:16:15 = 30 healthy Debug cycles, 0 heals, 0 WARN, 0 ERROR. Confirmed HUB-SIDE (read-only query of the hub store, after STOP-3): the guest_net stanza arrives complete, and the report history traces the whole session — 10:30:27Z agent 0.92.0 state=unknown (the sudoers-blind window, reported honestly rather than as a false healthy — independent proof that superseding 0.92.0 was right), then 10:48:12Z 0.92.1 healthy carrying last_heal_at=10:45:12Z heals_last_hour=1. Known limitation: the damping state is IN-MEMORY, so an agent restart resets the heal counters — the ≥10 min / ≤3 per hour ceilings hold within one agent lifetime, not across restarts (a restart does re-arm the 3-minute settle window, so the practical exposure is small). Do not quote the ceiling as a hard guarantee |
| R-55 | [P2] A customer's deliberate Stop does NOT survive a guest reboot for any drive-backed app — the boot bind gate recreated and started it unconditionally. | S | SHIPPED 2026-07-21 — controller v0.157.0 | Surfaced by STOP-1's R-52 leg, which was designed to prove the opposite property and instead proved it is not true end to end. internal/web/intermediary.go's boot gate ([gate] boot <id>: live bind confirmed — recreating drive-backed app <x>) selects on cfg.Deployed && HDD_PATH alone and then runs StopStack + StartStack on every match; the stack's state is passed in for the log line and is never consulted. So after pct reboot 9201, calibre-web and immich came back running at 10:53:18Z even though immich had been stopped from the UI seconds earlier (compose down, 0 containers) — while bookstack, which is NOT drive-backed, correctly fell through to R-52. R-52 is not implicated: its own gate behaved exactly as specified (immich, with zero containers, was never a candidate), and this behaviour predates it. Two consequences worth ruling on: (a) a customer who stops a drive-backed app finds it running again after any reboot, which is the same class of dishonesty as offering a control that cannot work; (b) R-52's practical scope on a real box is narrower than the brief assumed — the gate reaches every drive-backed app first, so the boot-orphan sweep only ever sees the rest. Likely shape: the gate should recreate only apps that were RUNNING (or deploying) at shutdown, which is the same running/stopped distinction R-52 already draws, rather than every deployed app. Needs a ruling, not a drive-by change — the gate exists to fix a real bind-ordering bug and must keep doing that |
14:29:28 [gate] boot 1784525102-11906045: live bind confirmed — recreating drive-backed app calibre-web (state=stopped) onto /mnt/felhom-drives/hdd_114:29:29 [gate] boot 1784525102-11906045: 1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose
immich is absent from the recreate list and came back STOPPED (0 containers) — on 2026-07-21 before this fix, the identical fixture brought it back RUNNING. calibre-web recreated, bookstack back, [bootrecon] no boot-orphaned apps (consistent — nothing was left orphaned for it to adopt), ZERO alerts, whole convergence ~15 s from reboot to steady state. The left stopped INFO line fired in production for the first time, so the honoured path is observable rather than silent |
| R-56 | [P3] Apps do not say how technical they are, so a beginner can be ambushed by a config-heavy one. The catalog presents every app as equally approachable — one Telepítés button, the same Hungarian copy — but they are not. Glance needs a hand-written glance.yml before it does anything; some apps need a reverse-proxy or API concept to configure; others genuinely are install-and-use. A tester who picks the wrong first app concludes the PRODUCT is broken, not that they picked an advanced app. | S | idea (filed 2026-07-21) | Origin: TASK-E Part 3 — filed, deliberately not implemented. Shape: a difficulty: field in .felhom.yml (kezdő / haladó / technikás) surfaced as a catalog-card badge and repeated on the deploy screen. Cheap and incremental: one optional metadata field plus a badge, classifiable app-by-app with no migration — an app with no difficulty: simply shows no badge. This is the constructive half of the glance ruling: glance STAYS in the catalog (operator ruling 2026-07-21 — it is a legitimate app, not a broken one; its missing seeded glance.yml is a known pre-existing finding), and the honest fix is to LABEL it rather than hide it. Pairs with R-41: that gate proves an app CAN still deploy; this field tells a customer whether THEY should be the one deploying it. Badge plumbing is ALREADY BUILT (controller v0.158.0) — web.MetaBadge + the meta_badge template partial + the lifecycleBadge funcmap entry were written generic for exactly this: a difficultyBadge funcmap function returning the same *MetaBadge, plus a difficulty: field on stacks.Metadata, is the whole remaining job. No new markup, no new CSS. Re-sized accordingly |
| R-57 | An app can be withdrawn from the catalog without orphaning the customers already running it — .felhom.yml lifecycle: available / hidden / abandoned. | S | SHIPPED 2026-07-21 — controller v0.158.0 (+ v0.158.1 fix), LIVE-PROVEN | Motivating case: plant-it. Earlier the same day it was withdrawn by moving its directory to retired/ — which un-offers the app but ALSO makes the controller's orphan detector see the template as GONE for anyone running it, flagging their working install Elavult and offering a Törlés button. Withdrawing an app must never take a working app away from a customer, so the directory move was replaced by metadata. Operator requirements, verbatim (ruling 2026-07-21): states available / hidden / abandoned; abandoned apps are NOT offered to new installs (no badge-but-installable middle state); deployed instances of hidden/abandoned apps keep full function; an abandoned app shows a permanent notice that „Az alkalmazás fejlesztője felhagyott a fejlesztéssel. A telepített verzió továbbra is használható, de frissítések és biztonsági javítások már nem érkeznek hozzá." Design points that matter beyond this feature: (a) the deploy gate is server-side and fail-CLOSED before any mutation — hiding a button is not a gate, and a stale link or direct POST must be refused; (b) an unknown lifecycle value fails OPEN (→ available + one WARN), deliberately opposite, because a typo or a state from a newer catalog must never pull a working app out of every customer's list — both read the same EffectiveLifecycle, so they cannot disagree; (c) lifecycle NEVER reaches orphan detection, red-proofed. LIVE-PROVEN 2026-07-21 on 9201 through the real endpoints: lifecycle: abandoned arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page while the control app renders 10; a direct POST /api/stacks/plant-it/deploy returns HTTP 409 {"ok":false,"error":"Ez az alkalmazás jelenleg nem telepíthető."}; the app page carries the notice and no Telepítés button. v0.158.1 is a shipped-and-caught defect worth remembering: the three predicates were declared with POINTER receivers, and html/template cannot call those on the non-addressable value the handler passes — every /apps/<slug> returned 500, for every app, while compiling cleanly with a fully green suite, because no test rendered app_info. A template method call is only checked when the template runs. Follow-on: R-56's difficulty badge reuses this plumbing |
| R-58 | [P2] Assisted disk-picker install mode — the installer should let the operator CHOOSE the target disk instead of requiring the serial up front. Today an install is either unattended (the answer file pins one ID_SERIAL_SHORT, which you can only know by first booting the machine) or match-nothing safety (aborts by design). That forces a two-boot dance for every new box: boot the safety ISO to read the serial, rebuild the ISO armed, boot again. | S–M | idea — operator ruling 2026-07-21 | Operator's argument, verbatim: "the installer should list the available storage devices (excluding the installation media) and let us select one, and continue." Shape: a THIRD ISO mode alongside the two that exist — unattended-serial and match-nothing-safety. It enumerates candidate disks with size / model / serial, excludes the installation media itself, takes a selection plus a confirm, and proceeds. Unattended+serial REMAINS the appliance/factory mode — it is the right shape when the machine is provisioned in bulk and nobody is standing there; the picker is for the case where somebody is. Slice 1 (cheap, same code surface, do this first): improve the abort screen. On filter-no-match the installer currently just fails safe and says nothing useful — it should print the candidate table (size/model/serial) plus the one-line hint naming which serial to put in the profile. That alone collapses the two-boot dance from "boot, guess, go read docs, rebuild" to "boot, copy the serial off the screen, rebuild", and it is the same enumeration code the full picker needs. Why it matters beyond convenience: it is the BYO / reinstall flow — a customer's existing hardware, or a rebuild of a box whose disk layout nobody recorded, is exactly where the serial is unknown and a wrong guess is destructive. The current fail-safe is correct but mute. Origin: TASK-G, arming the HP install ISO — the serial had to be read off the board by hand between two boots |
| R-59 | [P1] A no-DHCP install must HARD-ABORT — instead it bakes the installer's fallback address as a STATIC config and completes, producing a box that can never call home. | S | SHIPPED v1.24.0 (2026-07-22) — as a FIRST-BOOT refuse-loudly gate, with a RECORDED DEVIATION: the install-time abort is out of scope (the 192.168.100.2 fallback is baked inside the Proxmox auto-installer itself, unreachable without an installer-initrd hook; operator-acked, not silently dropped). The screen + gate proven on the nested drill (SPIKE-firstboot-nic-sweep-2026-07-22.md); nested ≠ metal — metal proof rides the next real install | This is the worst silent onboarding failure shape there is: the install succeeds, the box looks finished, and it is permanently unreachable — no hub check-in, no pairing, no way in except a keyboard and monitor. Found on the HP t740's first install: the 4-port NIC got no DHCP lease at this site (see the t740 gotcha in scripts/iso/README.md), and rather than refusing, the installer wrote its 192.168.100.2 fallback as a static vmbr0 address into /etc/network/interfaces and carried on. The philosophy is already established one layer over — the disk filter refuses loudly and touches nothing when it cannot identify its target (spike S5c, proven twice on real boards). Networking deserves the identical treatment: no lease on any carrier-bearing NIC ⇒ abort with a legible screen, never invent an address. Slice: detect "DHCP produced no lease" in the answer/first-boot path and fail with the candidate NIC table (name / MAC / carrier / link speed) plus the one-line remedy, exactly as R-58 slice 1 does for disks — same refuse-loudly grammar, same screen shape. Pairs with R-60, which is the self-heal for the case where the cable simply moved |
| R-60 | [P2] First-boot NIC sweep self-heal: if the hub is unreachable, try DHCP across every carrier-bearing NIC before settling. | S | SHIPPED v1.24.0 (2026-07-22) — spike + nested drill proven (SPIKE-firstboot-nic-sweep-2026-07-22.md): cable move → sweep → heal + hub registration unaided in <1 min; sweep is structurally first-boot-only (state.json gate + the unit's done-flag condition); drill also surfaced and fixed the baked-fallback-default-route trap (flush before the bounded dhclient) | felhom-bootstrap currently accepts whatever addressing the installer left behind and, if the hub cannot be reached, simply stays broken. On demo-hp the fix was a human moving one cable from the 4-port card to the onboard port — a sweep would have healed it unaided: enumerate NICs with carrier=1, DHCP each in turn, and keep the first that reaches the hub. Cheap because the box has nothing to lose at first boot (no customer data, no running guests) and the failure it repairs is total. Deliberately scoped to FIRST BOOT and to the hub-unreachable condition only — a running box must never re-shuffle its own networking. Complements R-59: that one refuses to produce an unreachable box, this one repairs the case where the truth changed after the install (cable moved, switch port died, the installer guessed the wrong port) |
| R-61 | [P1] The baked root password must be knowable by the operator — the recurring console lockout. | S | slice 1 SHIPPED v1.24.0 (2026-07-22): the build emits the plaintext into a 0600 sibling <iso>.rootpw.txt (single record of truth — never logged/manifested/committed); drill-verified against the installed box's shadow hash. Follow-up (appliance-grade record-keeping) stays open | The ISO mints a fresh throwaway crypt hash per build and the plaintext is discarded, so nobody — including the person holding the machine — can log into the console of a box they just installed. Today that meant reaching demo-hp only through the G1 break-glass credential vaulted in the hub, which is the right mechanism for a lost password and the wrong one for a never-known password: it requires a working hub, a working network, and operator tooling, at exactly the moment the likely reason you need the console is that one of those is broken. Slice 1 (do this): the ISO build emits the baked root password into the build REPORT and the operator cheat-sheet alongside the sha256 — it is already a per-build value, so surfacing it costs nothing and closes the lockout. Follow-up (appliance-grade): keep it per-build random and treat the build output as the record of truth. A fixed well-known password is explicitly REJECTED (operator ruling 2026-07-21) — a pre-pairing box sits on a stranger's LAN with a predictable root credential, which is a far worse exposure than the lockout it would fix. Relates to G1 break-glass (the vault stays; this is about the window before/without it) |
| R-62 | [P3] Hub delete dialog: show the customer-id the operator must type, and reword the three acks for the ghost shape. | XS | idea (operator, 2026-07-22) | Cosmetic, hub-only, docs-only in the v1.24.0 train. The delete confirmation asks the operator to type the customer-id, but the id appears NOWHERE on the Edit page the dialog opens from — the operator has to fish it out of the URL or another tab. Also: for a GHOST customer (host already gone) the three acknowledgement checkboxes describe teardown steps that cannot happen; wording only — the server MUST keep requiring all three (the render-gate lesson of v0.70.1 stands: reachability and requirements are separate concerns). |
| R-63 | The install console learns ő/ű — the kernel default console font lacks the Hungarian double-acute glyphs, so the R-59 network screen and pairing banner rendered ő as blanks. | XS | SHIPPED (scripts v1.25.0, 2026-07-23) | felhom-bootstrap.sh loads a Latin-2 console font (Lat2-Terminus16 → Lat2-Fixed16 → Lat2-Terminus14) ONCE before the first paint — idempotent, best-effort (a missing font/ioctl never blocks boot). Lat2 ships in the trixie/PVE base (console-setup), so no copy rewording needed. Font names verified against the package. Nested-console capture proof rides the v1.25.0 drill. Also in the same train: F9 belt-everywhere RESOLVED (host-install default appliance leg + live on both boxes + login proven) and the R-71 build-gate (build-felhom-iso asserts golden ≥ managed floor, publish-train-rules.md rule 5). Live finding: golden 0.153.0 < floor 0.156.0 in production NOW — the gate catches it; the fix is the golden republish at 0.161.0 (Part 4, pending; the managed floor stays 0.156.0). See REPORT.md (2026-07-23). |
| R-64 | „Felhom↔Felhom media pairing blessed" — the two-box SMB pairing (one box shares, the other mounts it as NAS storage) becomes a supported, documented flow. | XS–S | idea (2026-07-22) | Origin: the operator ran the pairing drill on the live demo pair and it WORKS — the drill itself is the pending evidence leg (a written run-through with the R-66 surfaces in play). R-66 shipped the enabling visibility: the serving box's address is now on its own Beállítások → Rendszer „Hálózat" card, and the add form names the NetBIOS trap. Blessing = a short customer-facing recipe (documentation/controller/network-storage-nas.md naming-caveat paragraph is the seed) + one supported-path sentence in the capability map. Flips: would add a "Felhom↔Felhom media pairing" capability row (currently unlisted). Pairs with R-65 (same two-box topology, entirely different transport + guarantees) |
| R-66 | The box's own address becomes visible — „Hálózat" card, Debug network dump, NetBIOS hint. | XS | SHIPPED (controller v0.159.0, 2026-07-22) | Origin: the pairing drill — the serving box's IP was findable only as a hint buried on the OTHER box's Megosztás page, and the add form's failure for „FELHOM" taught nothing. Three legs: (A) „Hálózat" card on Beállítások → Rendszer (Helyi cím / Hálózati név only-while-sharing / Átjáró; live per render, stored nowhere — S-5; „—" when unavailable); (B) network section in the Debug dump (interfaces/route/DNS/lan_address, best-effort per item); (C) the NetBIOS trap named (Szerver helper text + a purely lexical hint on unreachable for single-label non-IP names). Design decision recorded: the controller is bridge-netns'd, so ALL guest-net reads go through the one netns door (docker exec into host-networked felhom-samba, stacks/guestnet.go) — with Megosztás off the card honestly shows „—" rather than the plausible-wrong 172.x answer. Deployed demo-felhom + demo-hp 2026-07-22; demo-hp live-shows the closed-door path (sharing off → dashes + in-place dump errors), demo-felhom the open one (real .104/.1/\FELHOM values). Flips no capability-map row (diagnosability/UX polish); enables R-64 |
| R-67 | The NAS share appears in FileBrowser — browse what you mounted. | S | SHIPPED (controller v0.160.0, 2026-07-22) | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). Couples to R-64: browsing was its missing UX half. A registered network storage now binds its share ROOT into FileBrowser (/mnt/felhom-drives/<name>:/srv/<name>:rslave) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (Host is down in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: felhom-controller/REPORT.md (2026-07-22) |
| R-68 | Notification train: paired recovery mails + prefs seeding at claim + priority headers (power-outage audit F11+F12+F14-light). The dead-man's-switch fired perfectly on 07-22 and the customer who got „A szerver nem elérhető!" was never told it recovered (F11); a customer without a customer_notifications row is silently unnotifiable (F12, demo-hp live); delivered ≠ noticed (F14). | M | SHIPPED (hub v0.71.0, 2026-07-22) | Origin: AUDIT-power-outage-recovery-2026-07-22.md. Recovery = explicit eventType branch (severity semantics frozen; severityNotifies untouched): operator always hears both edges, customer iff PAIRED (customer-channel sent stale/down row newer than the last sent recovery — store.LastCustomerSentAt; enabled_events deliberately ignored for recovery; ties → no mail, flap-safe). Seed-at-claim: MarkClaimed → SeedNotificationPrefs (INSERT-if-absent, never upsert — red-proofed; empty email no-op; never fails the claim; default critical-only set). Hub-side empty-email no-clobber belt in handleSavePreferences (controller 0.160.0 already guards its own two push legs — latent, not live). X-Priority: 1 + Importance: high on error/critical via Resend headers (live-probed HTTP 200 before implementation); the test event now also mails the operator with those headers (one click proves both channels + rendering); latent sendTestEmail nil-prefs panic fixed. 17 tests + 4 red-proofs. Live legs pending: natural *_recovered mail on the next real staleness cycle (or the reboot-drill arc — NEVER fabricated by blocking reports, that is F9-bypass-shaped) and seed-at-claim on a real claim (Peti Friday reinstall is the natural candidate) |
| R-69 | F14-full: an operator push channel that actually interrupts (ntfy / Telegram / similar), beyond mail-client priority flags. F14-light (v0.71.0 headers + Gmail filter) nudges a mail client; a 15:29 node_down should reach the operator's pocket in seconds regardless of inbox hygiene. Needs: channel choice (self-hosted ntfy on k3s vs Telegram bot), dispatcher fan-out seam, per-severity routing, quiet hours. | M | idea | Origin: AUDIT-power-outage-recovery-2026-07-22.md F14. Deliberately NOT built in the v0.71.0 train (scope-forked per the task spec) |
| R-70 | [P2-HIGH] The offsite last mile is invisible on BOTH surfaces — the hub cannot tell "staged" from "delivered" from "applied". demo-hp sat 2 days with the hub customer page saying "Provisioned: … the transient password is delivered to the controller once" while the box said „Még nincs beállítva távoli mentési cél" — and a real customer would sit unprotected indefinitely believing otherwise. The hub HAS the signal (one_time_secrets.consumed_at + 153 consecutive reports carrying no offbox object) and reads none of it: the "Provisioned" line is static copy gated only on offsite.host in ConfigJSON (config_form_body.html:119–120). | S–M | SHIPPED (hub v0.72.0 + controller v0.161.0, 2026-07-23) — detector offsite.DeliveryStateFor (one impl, all consumers), customer-card state line with age (static "delivered once" copy GONE), offsite_delivery_stuck warning event (24h durable cooldown), controller truthful empty-state banner. Live validation on the two fixtures (demo-hp applied; demo-felhom applied + stale-staged info) recorded in felhom.eu/REPORT.md; the banner leg is unit-proven/live-pending (no box occupies the enabled+no-offbox window). | Origin: audits/DIAG-f10-demo-hp-offsite-2026-07-23.md. Two legs: hub customer card shows the real delivery state ("provisioned, awaiting box consumption" / "consumed, awaiting apply" / "applied" — consumed_at × report-offbox-presence is enough for all three), and controller banner when the descriptor is enabled but no target is configured („Felhom offsite készen áll — a beállítás automatikus, folyamatban"). Couple to R-31's async/status-card idiom (same surface likely serves both) and to the R-39 consumed_at honesty gauge precedent on the PBS side — a consumed secret + N report cycles with no offbox status is the same "disagreement no single tier can see" shape and deserves the same loud event. Supporting live datum: demo-felhom's 07-21 staged secret is still unconsumed today (key-auth-first path never consumes) — invisible for the same reason. |
| R-71 | [P1] Day-0 race: the managed floor-update kills the offsite apply-bridge between password-consume and persist — the one-shot credential is burned and the box lands in the silent consume-404 dead-end forever. Proven on demo-hp (07-21): consume 16:27:42 → managed update 0.153.0→0.156.0 replaces the container 16:28:17, ~35 s later, mid ssh-copy-id window; nothing persisted, no installed key ⇒ the key-auth-first recovery path can never engage, and every subsequent start logs the WARN and gives up until an operator Re-issue. This recurs structurally on every fresh onboarding whose ISO floor lags the managed floor — the update fires minutes after first boot, exactly when the bridge first runs. demo-felhom escaped by timing only. | M | SHIPPED — (a)+(c); (b) rejected-by-design. (a) SHIPPED (controller v0.162.0, 2026-07-24): the apply-bridge settle-gate. offsiteapply.SettleProvider.SettleState() + SettleFunc adapter over the updater's OWN GetFloor()/IsUpdateRunning() (no second floor path); Bridge.AwaitSettle defers the consume while updateRunning OR below the known floor (the auto-floor update is imminent), releasing at/above floor on the first poll with zero added latency (B′). Bounds 10 s poll / 90 s floor sub-bound / 5 min overall — both bounds GO+WARN (a hub that can't serve a floor can't serve a consume → no burn; R-71c is the belt). Ordering-only — the consume/persist order and the 404-no-oracle contract are UNTOUCHED. Finding: the floor is in-memory (report-ACK ~5–10 s), not persisted → unknown until the first ACK on any restart; the 90 s sub-bound is sized to that. 5 scenarios (A–E) + 4 red-proofs all observed FAIL then restored. Deferral paths NOT live-fired — their precondition is now structurally prevented by the v1.25.0 golden≥floor build gate (the point); live leg = the B′ first-poll GO line on both above-floor boxes. (c) SHIPPED (hub v0.72.0, 2026-07-23): self-heal restage via the EXISTING Re-issue path (monitor.OffsiteDeliveryChecker), consumed ≥1h + ≥4 offbox-less reports + zero offbox evidence, one restage/customer/24h, offsite_credential_restaged warning; R-39(a) guard red-proofed. (b) two-phase consume REJECTED-BY-DESIGN — it would touch the 404-no-oracle contract; (a)+build-gate+(c) cover the race without it. Layering: v1.25.0 build gate PREVENTS, (a) DEFERS, (c) HEALS. | Origin: audits/DIAG-f10-demo-hp-offsite-2026-07-23.md (mechanism cites: bridge order offsiteapply.go:106–187 consume-then-persist, retry only on process start; the dead-end is even documented in source l.168–173 "the password is spent; reset it on the hub to retry"). Candidate directions, spec-first: (a) order — first-boot path lets the managed update settle before the bridge's consume step (cheapest; the race window is the update, not the reboot); (b) two-phase consume — hub marks consumed only on a controller ack-after-persist (touches the 404-no-oracle contract, design carefully); (c) hub-side self-heal — auto-restage a fresh secret when consumed_at is set but K consecutive reports show offsite enabled with no offbox status (the R-39(a) mint-race lesson applies: never restage on top of an UNCONSUMED secret). Pairs with R-70 (visibility) — but visibility alone is not the fix; the burned credential needs an unattended recovery path. |
| R-53 | app_export.html substituted the CSRF token where the customer domain belongs - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | SHIPPED (controller v0.150.0, 2026-07-20) | One template token ({{$.CSRFToken}} -> {{$.Domain}}) plus the Domain key in exportPageHandler's data map - that handler does not go through baseData, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined <sub>.<domain> and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: audits/AUDIT-vacation-remote-ops-2026-07-20.md (F7) |
Recovery-model gaps (2026-07-28, 07-backup-architecture.md §10.2)
Minted when
07-backup-architecture.mdwas rewritten as the recovery model. Every one of these is a divergence between that model and the system as it is, and each is cited there. They are filed at P2 as the neutral default, not ranked — ranking them needs the per-scenario RTO/RPO targets that07§11-C records as never having been stated. Flips:00-capability-map.md§C rows, which now cite the matrix rather than restating the route.
| ID | Item | Size | Status | Notes / map rows flipped |
|---|---|---|---|---|
| R-102 | Tier-2 writes a full recovery-unit/ mirror on every run and no code path reads it. Written at internal/backup/tier2.go:368-369 ("Unit leg (always)"); RecoveryUnitPath resolves to backups/**primary**/ (internal/appbackup/paths.go:46-48) and the only reader of the secondary tree is internal/backup/tier2_restore.go, which reads hdd/+userdata/ only (:101-104) |
M | READY — 2026-07-28 | Was C9-F4 (OPEN-ITEMS.md). The sharp edge is when it bites: Tier-2 exists for primary-drive loss, and in exactly that failure the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action — leaving offsite as the only route. LIVE: demo-felhom's backups/secondary/{bookstack,docmost}/ hold recovery-unit and nothing else, at 156 MB and 86 MB. Flips: the Tier-2 row in map §C; 07 §6.3, §7.2 |
| R-103 | The Tier-2 no-coverage refusal names the working action but does not route to it. v0.183.0 refuses up front without stopping the app and tells the customer to use „Visszaállítás indítása" on the other page; it does not take them there | S | READY — 2026-07-28 | Was C9-F1b. Deliberately its own item: it puts a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference. Flips: nothing until shipped; 07 §10.2 |
| R-104 | An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach. resticStep has unlock --remove-all (internal/backup/offbox.go:634-648) but ensureOffboxRepo's probe fails first, classifyResticProbe (:77-93) has no lock case → "other" → fail-fast; ClassifyOffsiteFailure likewise, so the operator is told „A távoli mentés ismeretlen okból nem sikerült" for a precisely-known, self-healable condition |
S | READY — 2026-07-28 | Was C9-F3. Reachable by any interruption — container restart, OOM, network drop, host reboot mid-backup. The tier stays dead until a human runs restic unlock --remove-all. Flips: the offsite row in map §C; 07 §8 row 15 |
| R-105 | Three hub-held DR records are empty on the entire live fleet. hosts.dr_record_json = {} on all 3 hosts; host_escrow.directive_json = {} on both escrowed hosts; dr_recipe.host_half.drives = [] on every customer including two with enrolled data drives (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) |
M | READY — 2026-07-28 | These are exactly the fields a host-loss recovery reads: 05-hub-architecture.md:175-176,186 names the slim DR record as one of four durable sources; 06-offsite-connectivity.md:148-150 says the escrow upload carried the DR directive; felhom-agent/internal/dr/plan.go:34-35 makes PlannedDrive the re-attach-by-durable_id wrong-disk guard. The three may have different causes — isUserDataDrive (internal/hub/dr_recipe.go:129-136) requires type usb/local-dir and a non-empty DurableID and MountPath, and which of the three fails was not traced. Evidence: architecture/_recovery-inventory-2026-07-28.md Part D2.3. Flips: 07 §4 |
| R-106 | The DR recipe records the PBS namespace as "root" on every box while the real namespaces are demo-felhom / demo-hp |
XS | READY — 2026-07-28 | Traced to source: Snapshot.Namespace decodes a ns field (felhom-agent/internal/pbs/client.go:97) that PBS does not echo per item when the request is already namespace-scoped via ?ns= (:117-120), so it is empty and ToHub normalises empty → "root" (internal/pbs/report.go:23-27), which latestPBSCoord writes into the recipe (internal/hub/dr_recipe.go:149). latest_snapshot_id: "9201" is not a defect — it is BackupID, documented as a coordinate (:56). Verified against ep0's real namespaces and a live pct restore command line carrying --ns demo-felhom. Flips: 07 §4 |
| R-107 | No offsite action unpacks the named-volume tars Tier-3 captures on every run. ReconstituteFromOffsite skips the unit outright (internal/backup/offbox_reconstitute.go:284-289); PlaceOffsiteRestore places it only when the live unit is ABSENT (internal/backup/offbox_restore.go:352-356). The DB is replayed, from the scratch unit's db-dumps/ |
M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but no single action does it and no UI routes it — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; 07 §6.3, §7.2 |
| R-108 | Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5. For local drives FileBrowser binds <drive>/userdata only (internal/web/handlers.go:2450-2460); for network paths it binds the share root with :rslave (:2432) and serves it with download: true (internal/infra/infra.go:326). Nothing stops an app's namespace landing there: GetSchedulableStoragePaths() has no IsNetwork() filter (internal/settings/settings.go:904-914, feeds the deploy dropdown at handlers.go:462-473), the per-app migrate target list has none (handlers.go:674-679), and handleStorageMigrateApp does not call refuseNetworkLifecycle though its whole-namespace sibling does (internal/web/storage_handlers.go:397 vs :410-424) |
M | READY — blocks an architectural target — 2026-07-28 | Today this is not a secret leak — the recovery unit's app.yaml is secret-stripped (internal/backup/recovery_unit.go:73). It becomes one under D5, which would put app secrets in the local unit precisely so Tier-1/Tier-2 restore stop needing the guest. Verified LIVE in demo-hp's generated compose, where the asymmetry is visible rather than inferred: - /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb beside - /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave. Second effect, independent of D5: .fab bundles already carry plaintext secrets by design with an OPTIONAL password (internal/appexport/export.go:484,506-511,307) and storageDriveList() does not filter network paths (internal/web/handler_export.go:377-387), so one can be exported onto a NAS today. Full sweep of every read surface: 07-backup-architecture.md §10.1. Flips: 07 §7.3 (D5 target → adopted) once closed |
P3 — post-alpha
| ID | Item | Size | Status | Notes |
|---|---|---|---|---|
| R-26 | Guided old-history recovery via a retained superseded escrow + the recovery code. Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (host_escrow_superseded, ListSupersededEscrow). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside .orphaned-<date> repo for restore. |
M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin DIAGNOSE-offbox-repo-orphaned-2026-07-17 |
| R-27 | Customer-facing self-bind page (R-21 slice C follow-on). Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | SHIPPED (slice 1, hub v0.66.0, 2026-07-17) | Slice 1 = the FIRST-box flow, DONE: operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor /bind/<token> (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same BindAppliance, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP /bind/ exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). Slice 1 PROVEN-LIVE 2026-07-18: in the rehearsal the operator minted + emailed the link at 16:28:55 and the customer bound their own box at 16:29:55 with zero failed attempts, appliance_bound carrying source customer_selfbind, credential delivered 26 s later with no operator action (tests/VALIDATION-n100-rehearsal-2026-07-18.md). The new-ISO console banner rendered on the real box. Slice 2 = console-passphrase bind → R-27c (below). Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. Multi-box per link is out of scope by design (single-active, one-shot → binding several boxes = repeated operator sends) |
| R-27b | Customer self-bind, second-box flow (controller side). For a customer who ALREADY has a bound box and installs another, the controller shows a dismissable "bind another box" prompt (and a bind-later entry under settings) that walks to the hub /bind/ page — so a returning customer isn't emailed a fresh operator-sent link for every box. Mechanism sketched in the hub v0.66.0 REPORT; NOT built (R-27 slice 1 deliberately did not touch the controller). |
M | idea (minted by hub v0.66.0) | Origin: hub v0.66.0 slice-1 ship (first-box only). Reuses the same /bind/ public page + tokenized-link machinery; adds a controller-side entry point + the operator "mint a link for an existing customer" affordance |
| R-25b | RULED: customer DELETE becomes a guided full-teardown cascade. The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown. | M (was S) | SHIPPED hub v0.69.0 (2026-07-21) | operator ruling 2026-07-21: DELETE subsumes the whole cascade, behind explicit consent. Three separate acknowledgements, each its own checkbox — (1) the host(s) will be deleted, (2) the customer will be RESET including external teardown and offsite data destruction, (3) the customer record and escrow will be purged — plus a typed customer-name confirmation before the button arms. Internal order is host-delete → RESET → delete, which preserves every existing invariant rather than relaxing any: RESET keeps its no-hosts precondition (hosts are already gone by then), and escrow keeps its demote-then-purge custody rule (host delete DEMOTES to retained custody, the final delete PURGES — the one true purge point). Re-sized S → M: this is a multi-step destructive wizard with three acks and a typed confirmation, not a checkbox. Implementation is explicitly NOT part of TASK-E; the row carries the ruling and awaits its own spec. It no longer blocks R-3 — the model is decided, so the friend-alpha runbook can be written against it. IMPLEMENTED per the ruling (TASK-I, hub v0.69.0): POST /configs/{id}/delete now runs hosts → RESET → purge; three acks + typed customer-id + a stale-preview check + the ONLINE-host refusal, all gates before any write (zero side effects on refusal); custody purged exactly ONCE in leg 3 (leg 2 runs with purgeEscrow=false); ruling-3 preserved BY CONSTRUCTION and asserted from inside leg 2; failed legs retain the journal and the dialog offers Resume. Standalone RESET byte-identical. 5 red-proofs. Offboarding guidance: runbooks/RUNBOOK-onboarding-draft-v4.md §G. v0.70.0 follow-up (same day, found validating against the live hub): a completed delete still left the customer on the Customers list and still ALERTING, because GetCustomers() is report-derived and no tier ever deleted a report — new residue leg (reports/telemetry/log-tails/notif-prefs + the credential-bearing appliance_registrations/selfbind_tokens), and ghost customers are now deletable (404 = nothing here, not no-config-row). v0.70.1 (2026-07-22): the ghost delete was implemented but UNREACHABLE — the Danger-zone card (and the customerDeleteOpen script) sat inside {{if .HasConfig}}, so a ghost rendered no Delete button at all (the fourth inert-seam defect; handler tests POST directly and proved nothing about reachability). Render gate split: RESET stays HasConfig-gated, Danger zone gates on Deletable (the exact negation of the preview's 404 predicate), Block/Unblock stay config-only; render tests per branch + 2 red-proofs. Operator live leg: the demo-vm-felhom ghost delete click — PENDING (doubles as the v0.70.0+v0.70.1 live validation; expect residue=ok customer_delete=ok with skipped_no_config Hetzner/descriptor legs, staleness emails stop) |
| R-25 | Device-node TOCTOU hardening (drive init). Graduate the controller v0.141.0 Observation: the format → resolveEnrollUUID(path) → AssignDisk(uuid) sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. |
S | idea | From the v0.141.0 F6 commit's security-review finding (felhom-controller REPORT). Low real risk (single-operator, agent-guarded), but cheap to close |
| R-24 | Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17). The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE SetConfig (live cgroup, no reboot, Phase-0 proven). Framing note: the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). COMPLETE — the apply is proven in BOTH directions on metal (2026-07-18): customer zero shrank the guest 11675 → 8192 MB (16:50:22) and grew it 8192 → 12288 MB (17:02:17), each a live cgroup apply with no reboot, the new total rippling into the deploy page's memory math. The nested demo had proven the refusals; the N100 proved the applies. tests/VALIDATION-n100-rehearsal-2026-07-18.md ledger 9. Cores/live-resize as hub desired-state is NOT built (deferred, low demand). |
M | SHIPPED + PROVEN-LIVE | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". |
| R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster |
| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | first slice PROVEN-LIVE (poke channel) | FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17): the ep0-relayed contentless poke (hub→ep0 felhom-poke forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the agent and fires an immediate desired-state cycle. Full path live: real operator manifest save → sync-poke delivered to 10.77.0.2; box → poke received → immediate desired-state cycle (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per SPIKE-immediate-sync-transport-2026-07-16 P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately |
| R-28 | Agent fast-tick-until-first-convergence — SHIPPED (agent v0.90.0, 2026-07-17). Closes the immediacy leg a hub poke can't reach (the pre-tunnel WG-registration window): while ANY desired-state item is unapplied the agent pulses the out-of-band trigger every 30 s and self-disarms on convergence (state-based; four cached sources — desired-gen==0, reconcile Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational; LOUD states + pending_signature excluded). LIVE on both demo agents (startup line verified). Guests-0/0 passenger: diagnosed live and REFUTED as a bug — the 0/0 was the legitimate pre-provision reporting window; the fast-tick is exactly its mitigation (≤15 min → ~30 s). Unfired live: the real-onboarding proof (a fresh box converging in seconds without a poke) → the rehearsal S5. | M | SHIPPED | See felhom-agent REPORT (v0.90.0). Third immediacy leg alongside R-13 (poke) + R-23 (UI). |
| R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | |
| R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11). Launcher coupling — REVISED (controller v0.165.0): the "share the launcher outside the household" need is now met WITHOUT member accounts — the Indítópult megosztása capability-URL guest link (/s/<token>, information-only, no account) shipped in v0.165.0. What remains for this arc is member-specific: per-member tile visibility (each member sees only their apps) and the launcher-as-member-landing-page — both live inside this SSO/members arc; the guest-link ruling explicitly SUPERSEDES the earlier "members are how you share the launcher" framing |
| R-72 | Curate brand_color for the top catalog apps |
XS | idea | Parked follow-up to the v0.163.0 launcher. .felhom.yml brand_color (#rgb/#rrggbb) overrides the deterministic slug-hash tile color; no catalog app sets it yet. Pick brand-accurate colors for the most-installed apps so their launcher tiles match their real brand. Catalog-only change (app-catalog-felhom.eu), brand_color is already omitempty and consumed by the controller |
| R-73 | SMART history / trending (hub-side) | M | idea (parked) | Parked follow-up to the v0.169.0 disk-health card + degradation alert (agent v0.94.0 serializes SMART into /disks; controller renders „Lemezek állapota" + fires disk_health_degraded on a degradation). Today the verdict is point-in-time and the baseline is in-memory (a controller restart re-baselines silently). Hub-side would let the operator SEE a disk's SMART counters trend over time (reallocated/pending sectors climbing, NVMe wear %), catch slow degradation the 6h transition check can miss between reboots, and persist the baseline across restarts. Needs a hub ingest surface for periodic SMART snapshots + a small time-series store + an operator trend view. No demo hardware exposes real SMART (virtualized), so this only earns evidence on a real box |
| R-74 | Island control plane on a CLUSTER (Peti's 2 nodes) — bring R-50's island bridge to a multi-node PVE cluster. | M | idea (Phase C of R-50, parked) | R-50 shipped the island for the ONE-host fleet (demo-hp, demo-felhom). A cluster needs bridge parity on every node: either per-node identical /etc/network/interfaces vmbr9 stanzas (simplest, drift-prone) or — preferred at ≥2 nodes — a Proxmox SDN zone/vnet defined cluster-wide (one definition, auto-applied per node). The guest island IP is per-guest + node-independent; the agent-follows-guest rule holds (each node's agent binds its own vmbr9 169.254.253.1). Migration order per the spike: drill-proven → demo (done) → Peti (this row). Its own supervised runbook, coordinated with Peti (a live customer). Completes the capability-map "site/network change" row for clustered installs. Source: audits/SPIKE-island-bridge-2026-07-25.md (cluster-parity finding) + RUNBOOK-island-migration.md (single-host procedure to generalise) |
| R-75 | Catalog-derived userdata skeleton + import surfaces | M | SHIPPED (controller v0.172.0 + catalog, 2026-07-26) | ${IMPORT_PATH} = <system namespace root>/userdata/import — ONE canonical drop-zone per box on the system drive, with no per-drive fallback (unresolvable leaves the var unset so compose fails loudly). Operator ruling overriding the spike's Fork-1 per-drive recommendation: each drop-zone app has exactly one ingest bind, so a per-drive import/ would put a dead lookalike on every other drive — unbacked too, since import/* is class: excluded. Third BindRoot (RootImport) threaded through the classifier AND every resolver (resolveAbs would otherwise have named a directory on the WRONG DRIVE); catalog-derived skeleton, sorted (the naive map-order derivation measured 20 distinct outputs from 20 identical runs — a fleet-wide FileBrowser restart loop via fbNeedsRecreate); one authoritative compose parser; FileBrowser „Beolvasás" source; app-page „Hova tegyem a fájlokat?" with PathEscape deep links and class-driven copy; data_paths: annotation (malformed path ⇒ whole-block reject, unknown role ⇒ fails OPEN); system-owned beolvasas SMB share refused server-side at handler AND store. Live-proven on demo-hp: consume bind resolves to the system drive, a file dropped through FileBrowser's mount was ingested and deleted in ~30 s, the share auto-appeared on enable and survived a direct delete POST, and the before/after directory diff on BOTH boxes shows zero removals (demo-hp gained media/podcasts, the one genuinely derived entry). Two defects found and fixed during the live legs: EnsureImportRoot left the parent userdata at 755, and the carry-list initially re-created the per-drive drop-zone forever. Capability-map row File access via browser deliberately NOT upgraded — nothing drove the FileBrowser HTTP UI. Docs: controller/import-and-data-paths.md, spike audits/SPIKE-catalog-data-paths-2026-07-26.md |
| R-77 | Endpoint-drift detection, samba protected-set gate, channel log honesty | S | SHIPPED (controller v0.173.0 + hub v0.74.0, 2026-07-26) | Fixes the two defects in audits/DIAG-agent-channel-2026-07-26.md. bootstrap.DetectEndpointDrift names a controller.yaml vs bootstrap.json local_api.endpoint divergence with its own event type local_api_endpoint_drift and its own banner — and writes nothing (see R-78). EffectiveProtected now gates samba on Enabled && UserSet, mirroring both of reconcileSambaAt's early returns, plus the doc-comment correction. Channel log distinguishes born-down (unseeded->down) from a real transition, logging-only. Part 0 repaired both production boxes (endpoint → 169.254.253.1:8443; fingerprint+token agreed). Tests 951→959; red-proofs A/E/F. |
| R-78 | local_api authority ruling — auto-reconcile vs detect-only |
M | idea (deferred OUT of R-77 on purpose) | R-77 ships detection because the fix is genuinely undecided, and both directions can lose customer-visible function. Direction 1 (today): controller.yaml wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. Direction 2 (bootstrap.json wins, auto-reconcile on boot): a guest whose controller.yaml is CORRECT and whose bootstrap.json is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a working channel clobbered on the next restart, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — mergeLocalAPI replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. |
| R-79 | report.Issues / report.Warnings are English on customer-facing surfaces |
M | idea | Whole-surface, not a one-off (DIAG §6): every producer is English — "SSD/HDD disk usage critical", "Docker: %v", "Protected container not running: %s", and all six Warnings strings. They render on the customer's Hungarian dashboard, and the health_critical path has reached the customer email channel three times historically. Deliberately NOT bundled into R-77: a copy sweep across every producer would have buried two safety fixes in string churn, and the seam is not obvious — translate at the producer, or at the render/notification boundary where operator-English and customer-Hungarian already diverge? Pick the seam in a spike; the strings are mechanical after. |
| R-80 | expected_backup_missed false alarm — diagnosed + class-fixed |
M | SHIPPED (hub v0.75.0, 2026-07-26) | Diagnosed in audits/DIAG-backup-missed-2026-07-26.md, fixed as R-81. The premise was wrong on both counts: it fired ONCE (not nightly), and no real external customer was notified — the one customer-channel mail went to the operator's own mailbox; peti-felhom (the only real external customer) did not fire. The 7.3 days did NOT predate the channel outage — it is the age of the PBS snapshot, reached only as a fallback once the vzdump array went empty. Local vzdump was never stale (archives on disk 07-24/25/26). Cause: the agent's backup store is in-memory, so the R-50 fleet restart at 12:44 UTC emptied backups until the next backup at 07:03, and the hub read empty as "no backup exists". Fixed by anchoring (hub v0.75.0). Left standing as a REAL finding → R-82: the PBS/offsite-DR tier has no schedule at all. |
| R-81 | "No signal" is not "bad signal" — anchor the backup deadline check | S | SHIPPED (hub v0.75.0, 2026-07-26) | Third instance of one class (hub v0.12.0, v0.73.0, this). assessBackupFreshness now returns OK/UNKNOWN/MISSED; absence is UNKNOWN until it outlives an anchored window. The hub reads its own retained host-report history (GetHostReportsSince + newestBackupEvidence, 7-day bounded lookback) to answer "when did I last SEE evidence of a backup?", and anchors absence at first contact (GetFirstHostReportAt), reusing the existing 26 h threshold as the grace — no new knob, the v0.73.0 shape. Deferrals are logged; reason strings kept distinct. Tests 493→508; red-proofs A/B/C. Not silence — a genuinely dead box still alarms, which is the half the naive fix breaks. |
| R-82 | The backup target split — local daily + offsite weekly | M | SHIPPED (agent v0.102.0 + controller v0.175.0 + hub v0.76.0 + host-install 1.20.0, 2026-07-26) | Phase 0: audits/SPIKE-r82-phase0-2026-07-26.md (weekly CONFIRMED; the pvesm status 0/0/0 resolved as a namespace-scoped-token reporting artifact; capacity measured). backup_targets[] gives each tier its OWN cadence and retention; the controller runs both due tiers in ONE quiesce window; the hub judges each against its own threshold (host 26h / offsite 8d); host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly, and an unprovisioned tier DEFERS rather than firing at a missing storage. Operator rulings 2026-07-26: 2-week offsite retention, first backup runs as long as needed, one backup at a time per guest, drill box dropped from the rollout. Live: demo-felhom + demo-hp both on local-24h + felhom-pbs-168h; demo-felhom's first real PBS backup landed (TASK OK, 41 min, 14.46 GB) and restored cleanly. Four defects found by RUNNING it, not reviewing it: 30-min wait bound vs a 41-min backup (v0.98.0), restore tier read from the configured target not the archive (v0.100.0 — a silent regression of the S4.1 fix), a leaked scratch keeping onboot:1 (v0.101.0), and a tier firing at a not-yet-provisioned storage (v0.102.0). REMAINING GATE: the installer default is written but the fleet-wide flip waits on a full weekly cycle holding; and the offsite tier is never AUTOMATICALLY restore-tested (the scheduled test only selects the primary tier) — that gap, plus the hub inferring cadence from storage TYPE, are the two follow-ups. |
| R-83 | Ratify or retire 07-backup-architecture.md |
S | DISCHARGED (2026-07-26) — brought current, NOT ratified | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ STALE outside the PBS tier verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names felhom-pbs → felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2 — and §9.1 records the measured capacity. Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's. R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. |
| R-84 | The agent's cold backup Store no longer causes a redundant backup |
S | SHIPPED (agent v0.103.0, 2026-07-26) | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The Store is in-memory ("lost on restart; the cadence re-populates"), so after every restart /backup/due said no successful backup recorded yet and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. Resolved by asking the STORAGE rather than persisting the store (BackupRunner.NewestArchiveTime + the optional localapi.BackupArchiveLister): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only when did a backup last land — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. Proven live on both boxes immediately after a restart with the store cold: demo-felhom local age=5298s / felhom-pbs age=14373s, demo-hp local age=498s / felhom-pbs age=2346s, all due:false. |
| R-85 | The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD | M | Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE | R-82 proved a PBS restore round-trip by hand. Three defects kept the unattended path from covering it, each verified at source: (a) buildRestoreTestScheduler took cfg.Backup.BackupTarget() and built ONE runner on it, so the offsite tier's archives were never candidates; (b) the Spec was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; (c) a failed restore-test was a [WARN] log line with no event, no notification, no gauge — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. Selection ruling (operator, 2026-07-26): oldest-first (Option 1) — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported so the achieved interval can be MEASURED before anyone decides whether a knob is needed — measure, then configure, not the reverse. Shipped: per-run SpecBuilder; PickRestoreCandidateOn (a tier with no archive NO-OPS, never fails); persisted RestoreTestState (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); backup.InFlight, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub restore_test_failed / restore_test_stale, kept distinct and anchored on R-81's verdict structure, operator-tier only. NOT yet proven: rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). |
| R-86 | Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup | S | NEXT — operator ruling 2026-07-27 | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express "the day after the PBS backup": any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. The ruling: weekly per tier (daily is overkill), timed ~1 day after that tier produced a new backup — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). Shape: trigger a tier from its own last successful backup rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: BackupRunner.NewestArchiveTime, R-84) alongside RestoreTestState (R-85). Interim in force: restore_test_cadence_seconds = 302400 (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
| R-87 | The restic (app-data offsite) tier is NEVER restore-tested | M | idea — surfaced 2026-07-27 while closing R-85 | R-85 covers whole-guest vzdump tiers only (local, felhom-pbs); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box and carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (/mnt/felhom-drives). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped. Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. |
| R-88 | An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it | S | Part 1 SHIPPED (controller v0.176.0, 2026-07-27); Part 2 OPEN (agent wire change) — Part 1 added the failure breaker: consecutive failures tracked per TARGET, backoff 15m→30m→1h→2h→4h capped, reset on success, gating the QUIESCE rather than the backup so no stack is stopped for a doomed attempt. Never permanent (the cap bounds the retry interval — a latched breaker is a silent backup outage), never global (a broken offsite tier cannot suppress a healthy local one), never applied to TriggerNow, and stillRunning is not a failure. In-memory on purpose: forgetting a backoff costs one attempt, persisting one could outlive the fix. Part 2 is agent-side and needs a wire change — see the P0 finding below. Live on both boxes; the loop reproduction is UNIT-TEST ONLY (reproducing it live would need PBS taken down on ep0, which has no swap). |
The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check. R-84 replaced the in-memory Store with storage ground truth (NewestArchiveTime), and fails safe on a read error by falling back to the in-memory record — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: a cold store plus an unreachable target. After a restart the in-memory record is empty, so the fallback answers no backup has ever been recorded → tier due → the agent fires a backup at the very storage it just failed to read. Observed: felhom-agent restarted on demo-felhom at 07:02:57 UTC; local-api: could not read the backup storage for the due-check — falling back to the in-memory record (vmid=9201 target=felhom-pbs); 11 s later vzdump 9201 → felhom-pbs failed with could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused). The offsite PBS service was down at the time — so the trigger was real, not synthetic. NOT bounded, and it also DISABLES the business-hours protection — an availability fault, not a logging nuisance. The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves, and a second reading ("it stopped after three tries, so something limits it") was wrong too — the live run stopped after three only because the storage came back. Verified in code: there is no limiter. internal/quiesce holds no failure counter, no backoff, no breaker, no attempt budget; runOnce keeps no state between ticks, and the driver is a plain 5-minute time.Ticker (quiesce.go:149, default quiesce.go:107-109). Had the target stayed down, the loop would have continued indefinitely. |
Each attempt runs the full quiesce cycle, so every customer app stack is STOPPED and RESTARTED for a backup that cannot succeed. Measured on demo-felhom: 07:07:58 quiescing 4 stack(s): [bookstack calibre-web docmost immich] → 07:08:17 unquiescing (backup failed) → 07:08:45 failed — ~19 s of app downtime per cycle (~50 s per full cycle), every 5 minutes.
The amplifier, and the part worth designing against: the agent answers Due: true, Reason: "no successful backup recorded yet", AgeSecs: nil, and that nil age does double duty. scheduledRunAllowed (quiesce.go:466-480) returns true whenever lastAgeSecs == nil — "no recorded backup yet — never withhold the first one" — so the same nil that makes every poll due also bypasses the time-of-day gate [W+2h, W+6h). On the live box the gate was [04:30, 08:30) and the cycles ran at 09:02–09:12 Budapest, i.e. outside the backup window entirely. So the fault stops customer apps every 5 minutes at any hour, including business hours — the one protection specifically built to prevent that is switched off by the same missing value. A safety valve written for a genuine first-ever backup is being triggered by an unreachable storage read, which is not the same thing at all.
Self-resolves the moment the target answers (the storage read succeeds, sees the archive, tier stops being due) — which is why it can hide indefinitely: it needs an offsite outage to appear at all. PHASE-0 ROOT CAUSE, established at source 2026-07-27 — it is AGENT-side, case (a). The storage read errored (could not read the backup storage for the due-check … err=… at 09:02:57/09:07:58/09:12:57 CEST), so this was never an empty-success. The failure is a type boundary: newestArchiveOn (localapi/server.go:1095-1111) documents "Errors and unsupported services degrade to unknown, never to 'no backup'" — but its (time.Time, bool) signature cannot represent unknown, so an error and a genuinely-empty storage both collapse to (zero, false), and handleBackupDue (server.go:934-941) then emits a POSITIVE claim: Due: true, Reason: "no successful backup recorded yet", AgeSecs: nil. The fail-safe that does exist — targetStoragePresent's "a storage-view error must never be read as 'not there'" (server.go:1131-1151) — answers a different question (does the storage exist) and behaved correctly. Decisive for scoping: the errored path and the genuine-never path are BYTE-IDENTICAL on the wire — same Due, same Reason string, same nil AgeSecs — so the controller has nothing to discriminate on and Part 2 CANNOT be done controller-side. Two further P0 findings: the agent restarted 4× on 2026-07-27 (07:36:39, 07:54:06, 08:50:16, 11:31:52 CEST) — all deliberate (NRestarts=0, Restart=on-failure, Result=success), zero self-update — so the trigger is armed by ordinary operator/config work far more often than "only when ep0 is down"; and the loop alerted NOBODY — zero backup_failed events despite the hub allowlist carrying that type, because internal/quiesce does not import internal/notify at all. Its only trace was 07:13:27 info app_start_failed "Telepített alkalmazás nem fut: BookStack" — a customer-tier, Hungarian, info-severity SYMPTOM of the third cycle catching BookStack mid-restart. The whole-guest backup tier R-82 built has no failure signal to the hub → its own item. Shape: distinguish storage unreachable from storage readable and empty. Unreachable is UNKNOWN — defer the due-verdict rather than resolving it either way, exactly as R-81 made the hub do with a missing report. Only a target that is reachable AND has no archive is genuinely due. Fix the window bypass in the same slice: AgeSecs == nil must stop meaning "run now regardless of the hour". Either the agent distinguishes never backed up from cannot tell in what it reports, or scheduledRunAllowed gates on the former only — otherwise any future nil-age path re-opens the same hole. Note this does NOT weaken R-84's fail-safe intent: a tier whose storage is merely slow or briefly unreadable should still err toward backing up — it is specifically the cold-store + unreachable pair that must defer, because there the fallback has no information at all, only an empty default that looks like a fact. |
| R-89 | Retention is a COMMERCIAL attribute — it belongs to the hub, not to ep0 or a box | M | idea — operator ruling 2026-07-27, first increment SHIPPED same day | Ruling (2026-07-27): retention is a per-customer commercial attribute (a paid tier may buy longer retention), so the hub owns the policy and ep0 merely executes it. Execution stays server-side: a reconciler writes a PBS prune job and PBS's own scheduler runs it, so hub downtime leaves the last-known policy running rather than silently stopping retention. Increment 1 SHIPPED 2026-07-27 (runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md): boxes no longer attempt prune (keep_last: 0 → allowPBSPrune=false, config only — no code, no grant), and per-namespace prune jobs run on ep0 daily 03:30 UTC (keep-last 2), dry-run gated and verified TASK OK. This also closed a live defect — every demo-hp PBS backup since the tier was created on 07-26 had reported job errors while the data landed correctly, because DatastoreBackup grants Datastore.Backup but not Datastore.Prune; a tier that cries wolf on every success makes a genuine failure invisible, which is exactly what happened during the 07-27 migration window. Remaining work: retention becomes a per-customer attribute on the hub — a policy (keep-last/daily/weekly/monthly), NOT a bare number, since a paid tier will want "12 weekly + 6 monthly"; a reconciler converges it into the ep0 prune job in the same descriptor-and-converge shape as pbs_dr. SECURITY PROPERTY TO PRESERVE — do not "fix" a future prune error by widening the grant: box tokens stay write-only (DatastoreBackup), never DatastorePowerUser/DatastoreAdmin. A compromised box must not be able to delete its own offsite backups — that is the scenario offsite DR exists to survive. PARALLEL QUESTION, unanswered and arguably the more urgent half: does the restic key on storage-box-pool-1 (u629488) have DELETE rights? If so the daily app-data offsite tier carries the identical exposure, and restic's append-only mode is the equivalent answer. Rule once for both tiers. Flips the capability-map row for offsite retention (currently: retention configured but unenforceable) |
| R-97 | The whole-guest backup tier has NO failure signal to the hub — internal/quiesce never notifies | S | SHIPPED (controller v0.177.0 + hub v0.78.0, 2026-07-27) — R-97a: quiesce.TierNotifier, a seam (not an import) wired by an init-only setter, edge-triggered on the R-88 breaker ARMING so a failing tier is reported once per run rather than once per retry; recovery rides recordSuccess's existing bool. NEW operator-only event types whole_guest_backup_failed/_recovered — deliberately NOT backup_failed, which carries a customer Hungarian template AND sits in demo-felhom's live enabled_events, so reusing it would have emailed the CUSTOMER about a backup they cannot act on while it was still retrying. The recovery joins recoveredPairedDownTypes because its info severity would otherwise be dropped by severityNotifies — the operator would hear it break and never hear it heal. The hub's operator cooldown was keyed customerID:eventType alone, so one tier would have masked the other for an hour; now narrowly extended with a tier suffix taken from the event details, leaving every other event type unchanged. R-97b: a suppression window keyed to the quiesce CYCLE (not a state test — v0.164.0's != StateStopped filter cannot see an app caught MID-RESTART, which is exactly how BookStack alarmed), consumed at the same single derivation point classifyRunStates. Grace = 180 s, derived from the deploy flow's 120 s health timeout and Mealie's 60 s start_period; it expires, so an app that genuinely fails to come back still alarms. PROVEN LIVE end-to-end with a control: the new type POSTs 200 from inside guest 9201 while a bogus type 400s, and notification_log shows 1 operator row, 0 customer rows. The quiesce→notify link itself is unit-proven only. | On 2026-07-27 three whole-guest backups failed and three quiesce cycles stopped and restarted every customer app stack, and not one backup_failed event reached the hub. It is not the allowlist — the hub already carries backup_failed and backup_completed (they are emitted by the controller's app-data backup path). The cause is that internal/quiesce does not import internal/notify at all: the tier R-82 built has no route to the hub, so a whole-guest backup can fail indefinitely in silence. The loop's only trace was app_start_failed — info severity, Hungarian, on the customer channel — telling the customer BookStack was down (it had been caught mid-restart by the third cycle) without saying why, during an outage the system itself caused. So the one signal that did fire was both the wrong tier and the wrong story. Shape: emit backup_failed/backup_completed from quiesceAndPollTiers naming the TIER, operator-tier; and decide whether a quiesce-induced restart should suppress app_start_failed the way R-164's deliberate-stop filter does — an app the backup stopped on purpose is not a fault. R-88's breaker bounds the repetition but changes nothing about the silence |
| R-95 | The restic offsite tier's credential CAN DELETE — R-89's "parallel question", now ANSWERED | M | idea — established read-only 2026-07-27 | The exposure closed on the weekly PBS tier is fully open on the daily restic tier, which holds the customer's actual documents and photos and is the only tier that survives losing the box. Established without mutating anything: (1) Identity — a per-customer subaccount on storage-box-pool-1 (box 611714, bx11, u629488): u629488-sub1 home felhom-demo-felhom, sub2 peti-felhom, sub3 demo-hp, each labelled felhom-customer. Auth is an SSH key stored ON THE BOX (…/felhom-controller-data/_data/data/offbox/ssh_key, 0600, beside repo_password + a pinned known_hosts) — customer-side, not hub-side, so a compromised guest holds it. (2) Read-write: YES — the API reports readonly=False on all three subaccounts, and it is not merely latent: the controller runs restic forget --group-by host,tags --keep-daily 7 --keep-weekly … --prune from the box (backup/offbox.go:984, also :1070). Delete rights are exercised on every run. (3) Append-only: NO, and not expressible — the repo is built as sftp: (offbox.go:482); restic's append-only mode requires the REST server backend, which plain SFTP cannot provide. (4) A zero-code mitigation exists and is unused: the box type carries snapshot_limit=10 and the API reports snapshot_plan=null with 0 snapshots and size_snapshots=0. Hetzner Storage Box snapshots are taken server-side, outside the SFTP namespace — an SFTP subaccount cannot delete them — so they are a genuine immutability layer at no extra cost and with no code change. Rule once for both tiers, per R-89. Options, cheapest first: enable a snapshot plan (operator click, immediate); split backup-write from prune so pruning runs somewhere the box cannot reach; or move the repo to restic's REST server with --append-only. Flips the capability-map row for offsite immutability |
| R-94 | The hub hands out host-install 1.19.0 while 1.20.0 is what ships R-82's backup default | XS | idea — found 2026-07-27 | hub/internal/web/configs.go:28 pins const hostInstallVersion = "1.19.0", and that constant renders the customer page's install command (configs.go:487 ScriptVersion). scripts/felhom-host-install.sh:187 is SCRIPT_VERSION="1.20.0" — the version R-82 shipped to default a fresh box to local-daily + offsite-weekly. So a new install driven from the hub still gets the pre-R-82 default. This is the same constant hub v0.73.2 existed to sync (→1.19.0); it went stale again one installer release later, which is the actual finding: a hand-synced version constant in a second repo drifts every time the first one ships. v0.73.2 itself is NOT an undeployed artifact — its content is inside the live 0.77.0. Related to R-82's "REMAINING GATE" note, but that gate is about the fleet flip; this is the hub simply pointing at the older script |
| R-90 | ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86 | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: Swap: 0B, nothing in /etc/fstab, no swap unit, no /swapfile — swap was never added. On 2026-07-27 06:58:12 a global OOM fired while proxmox-backup-proxy held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration rsync held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed R-88 downstream. Still exposed today: 3.8 GB, no swap, one kill on record. This gates R-86 — backup-aligned restore-testing puts a multi-GB read on this box on a schedule, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console |
| R-91 | The old 13 GB datastore copy is still on ep0's root disk | XS | WATCHING — gated on demo-felhom's first post-migration PBS backup | The datastore moved to a Hetzner Cloud Volume on 2026-07-27 (/dev/sdb, 100 GiB, attached 06:29:40 UTC, now /mnt/pbs-datastore, 13 G used of 98 G). The pre-migration copy survives at /srv/pbs-felhom, 13 G, on / (38 G total, 16 G used, 21 G free). Do not delete yet: demo-hp has landed two post-migration snapshots (07-27 08:25:47Z, 09:37:29Z) but demo-felhom's newest is 2026-07-26T12:21:48Z — before the migration, so the new volume has not yet proven a write for that namespace. Delete once it has. Doc drift to fix in the same commit: CONTEXT.md:1018 still records the datastore at /srv/pbs-felhom |
| R-92 | The hub's PBS-DR gauge is 0.1 GB-granular, so small deltas are unverifiable | XS | idea — 2026-07-27 | The PBS-DR box card rounds to 0.1 GB, which is coarser than the changes an operator wants to confirm after a prune or a GC — a successful prune of a small namespace moves the number by less than one displayed digit, so the UI cannot distinguish "it worked" from "nothing happened". Cosmetic today; it becomes load-bearing the moment retention (R-89) is customer-visible and someone needs to see that a policy change took effect |
| R-93 | drill-r50 is both a blocked customer and the only drift fixture | XS | idea — 2026-07-27 | The drill customer is blocked in the hub (so it stops alarming) yet it is also the only record exercising the endpoint-drift path R-77 added. Blocking hides it from GetActiveCustomerIDs, so the fixture it provides is silently inert — a monitor with no live subject reads exactly like a monitor that passes. Decide: retire it and build a synthetic fixture, or unblock it and silence per-customer instead (the operator has a per-alert silencing feature planned). Related to the R-50 drill VM, now shut down |
| R-96 | Two standing rules were agreed in chat and never committed | XS | idea — found 2026-07-27 | Verified by grep across every CLAUDE.md, CONTEXT.md, REUSE.md and skills/: (a) "never combine a test run and a commit in one command" — zero hits, despite three separate occurrences (the worst pushed a red suite because packages ok: 28 was read while rc=1 was not); (b) "a 'no access' claim must list what was tried" — zero hits, despite a wrong "no access exists" verdict on ep0 on 2026-07-27 where the working route was documented in memory and only one path had been attempted. The third rule IS committed — CONTEXT.md:8 S-1, N.5's third leg (architecture docs same-session coupled). A rule that lives only in a chat log binds nobody; both belong in CONTEXT.md beside S-1/S-2 |
| R-76 | FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from audits/SPIKE-catalog-data-paths-2026-07-26.md P3/P5, both pre-existing and deliberately left alone by that spike. (a) FileBrowser Quantum 1.3.3 creates files 0644 and folders 0755 and does not propagate the setgid bit — even though the entrypoint wrapper's umask 002 really is in effect (/proc/1/status Umask: 0002). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's group half holds and only its mode half is lost. The consequence is proven with a control: inside a UI-created 0755 folder a gid-1000 process's file landed group 1000, while the identical write into the 2775 parent landed group 100. So any folder a customer creates through FileBrowser breaks the shared-group chain one level down. Latent today — every userdata-touching catalog app that declares an identity declares uid/gid 1000, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at infra/infra.go:156 is right that the image ignores -e UMASK but does not say the wrapper fails to achieve the intended mode either. (b) import/calibre is live on demo-felhom at 755 1000:1000 where every sibling is 2775 root:1000 — with media/books on the same box, same app, same deploy, at 2775 as the control, and no parser asymmetry (checked: ParseComposeUserdataMounts picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; not confirmed causally. Together they mean a drop-zone directory's mode is not stable against either the customer or the consuming app — verify before building anything that assumes import/* stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog |
| R-21 | Bare-metal Felhom ISO — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes felhom-host-install.sh) → universal secret-free / operator-bind (option C) | XL | SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18) | PHYSICAL CLOSURE 2026-07-18 (tests/VALIDATION-n100-rehearsal-2026-07-18.md): the generic pairing ISO v1.20.0 (--loader mkimage, SB off) booted the very AMI board that F1 blocked, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. F1 is closed on physical hardware and the rehearsal dependency on this item is discharged. Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → SB must be OFF. The installer's GRUB menu still offers interactive installers → new item R-38. — Prior: PHYSICAL RUN 2026-07-16 (tests/VALIDATION-n100-baremetal-2026-07-16.md): demo N100 reinstalled clean-slate from a pipeline ISO → chain reached rc-0 first try on real hardware (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. F1 (HIGH, slice-B input): this cheap AMI AN3PLUS 0.01 firmware won't UEFI-boot the ISO's GRUB from USB (relocation 0x0) — SB-off/shim-bypass don't help; worked around live with a grub-mkimage loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310: build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). Slice B — SHIPPED (scripts v1.18.0, 2026-07-17): the F1 firmware fix is now a first-class pipeline mode build-felhom-iso.sh --loader shim|mkimage (default shim; mkimage = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS Access Denied (unsigned → SB must be OFF, documented); surgery byte-identical payload. Physical N100 boot on the real board still pending → folds into the supervised rehearsal (R-1; an n100-safety match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). PXE/network-boot (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17): the GENERIC secret-free universal ISO (build-felhom-iso.sh --pairing). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (/api/v1/appliance/register + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. One unit, two modes (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest secret-bearing: no). The bind is operator-password-gated → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). Customer-facing self-bind page = R-27 (future). Origin spike audits/SPIKE-baremetal-iso-2026-07-16.md: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS and UEFI incl. Secure Boot enforcing (no MOK/keypress); first-boot hook fully-up = root + working pvesh/pct, exactly-once via pending-first-boot-setup flag; post-install webhook carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and fails-safe on match-nothing / bad disk; from-url + cert-fingerprint fails CLOSED. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, gate on validate-answer output not $? (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. (brief called this R-22) | SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result. The universal secret-free ISO was booted on a completely different machine — an HP t740 (Ryzen V1756B, AMI M42 firmware) versus the N100 it was proven on — and the whole chain ran on virgin hardware in one pass: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host demo-hp-bb76ea. Two things generalise from the second board specifically: the shim loader booted with Secure Boot ENABLED (mokutil --sb-state → SecureBoot enabled), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's 1TB NVMe untouched and unenrolled (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. Not clean, and the failures are filed: the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (R-59), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (R-61)
| R-29 | The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases. controller/scripts/docker_run_volume_path_gate.py has failed continuously since 2026-07-14 (v0.129.0) and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. (a) The finding itself is benign and the fix is 3 lines. The flagged call is internal/appexport/estimate.go:179 docker run --rm -v <volumeName>:/vol:ro alpine du — a NAMED-VOLUME mount, i.e. daemon-side with no host path, which is the safe shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (export.go volName+":/vol", backup.go volName+":/vol:ro", restore.go volName+":/vol"). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, not a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. (b) The systemic half is the real item: the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note 'Windows green gate silently red (read-only fsync)'. Decide where they run (pre-push hook, build.sh step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, felhom-controller REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller template_id_gate / emoji_gate / native_confirm_gate / offbox_rename_gate / mojibake_gate / app_row_dedup_gate / docker_run_volume_path_gate, hub hub_confirm_gate, manifests manifest_bearer_gate, website site_gates. Do not bundle (a) into an unrelated feature commit — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. 2026-07-18 rehearsal note: the run's finding list independently re-raised "assign the pre-existing docker_run_volume_path_gate failure its ID so red stops normalizing" — that is this item; no second ID was minted. |
| R-37 | Post-RESET health card shows stale pre-RESET warnings. After a RESET the card should read „RESET óta nincs adat" instead of carrying warnings about a lifecycle that no longer exists. | XS | SHIPPED (hub v0.67.0, 2026-07-18) | The customer page raises a banner when a RESET completed after the newest report, quoting „RESET óta nincs adat" and the reset timestamp, because until the box reports again every health figure describes a lifecycle that no longer exists. Deliberately narrow: an in-flight reset does not trigger it (only a completed one), and it clears itself on the first post-RESET report. Ties resolve to STALE — SQLite timestamps are second-resolution and a same-second report almost certainly arrived just before the reset destroyed what it describes; erring the other way would hide the banner exactly when it matters most. Red-proofed (neutering the predicate fails the assertion). — Origin: 2026-07-18 rehearsal. Same family as R-36 — the hub knows the state changed and the UI has not caught up |
| R-38 | Installer GRUB slice. A single default „Felhom telepítés" entry; the interactive installers REMOVED (safety: an interactive entry is how a wrong-disk manual install happens); felhom background. | S | SHIPPED (scripts v1.22.0, 2026-07-19) | Origin: 2026-07-18 rehearsal, alongside R-21's physical closure. Every ISO is repacked after prepare-iso: exactly one entry („Felhom telepítés", default, 5 s), with Graphical / Terminal UI / serial and the whole Advanced Options submenu (nomodeset ×2, three debug variants, Rescue Boot, memtest, UEFI settings) not emitted — not hidden, not password-gated. Boot behavior unchanged: the linux/initrd lines are lifted verbatim at repack time from the ISO's own automated entry, so a PVE bump tracks automatically; the build fails if they are missing, if the append line has lost proxmox-start-auto-installer, or if auto-installer-mode.toml is absent. Gated for 1 entry / 0 submenus / no live banned references, then re-verified by reading the menu back out of the finished ISO. Background generated at repack time from website/assets/og-image_2.png (one source, no second copy to drift), with the card's own grid phase-locked across the letterbox fill. Live-validated on the nested canary (UEFI/OVMF): renders, auto-fires, reaches the same match-nothing abort as v1.21.0. Squashfs/theme rebranding still DEFERRED — post-GRUB screens remain Proxmox-branded. GRUB's e/c keys also remain reachable (suppressing them needs a superuser password — a separate decision); the menu no longer advertises them |
| R-40 | [P2-HIGH] The update path cannot express a MULTI-HOP major upgrade. A template pin is a single value; the customer's update button pulls whatever the catalog now says. For apps whose upstream forbids version skipping this produces a broken upgrade. Nextcloud is explicit: "You cannot skip major releases. Please re-run the upgrade until you have reached the highest available release." Campaign 7 moved its template 31 → 34 (a fresh deploy validates fine — 302, 3/3 healthy), so an existing 31 customer pressing update would attempt a jump Nextcloud refuses. | M | idea | Origin: CAMPAIGN 7 (audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md §7 F7). Not nextcloud-only — any app with sequential-major rules (gitea, tandoor, outline…) has the same shape. Directions: a per-app upgrade_path:/max_hop: in .felhom.yml that the update button walks in stages; or refuse-and-explain when the installed major is >1 behind; or pin an intermediate "stepping-stone" tag. Until this exists, a >1-major catalog bump is safe for NEW deploys and unsafe for the update button — which is exactly the asymmetry the campaign's MAJOR flag was meant to record but cannot enforce |
| R-41 | [SLICE 1 SHIPPED 2026-07-21] The catalog has no standing "does every template still deploy?" check. Campaign 7 was the first thing that ever tried to deploy all 53 apps, and found 5 that had NEVER been deployable: papra (missing required AUTH_SECRET), zipline (v4 renamed CORE_DATABASE_URL → DATABASE_URL), wishlist (Docker Hub image gone; upstream moved to ghcr.io), homebox (upstream dropped the v tag prefix + new required env), glance (needs a seeded glance.yml the template never provides — PROVEN pre-existing: the pre-campaign v0.7.4 pin fails identically). Plus 7 broken healthchecks and 2 apps whose images no longer resolve at all (plant-it, wanderer). | M | idea | Origin: CAMPAIGN 7 (§7 F5/F6). The repo already has the right pattern in scripts/check-image-pins.py — a mechanical gate run on every change. Cheap first slice: a resolvability gate (docker manifest inspect every pin) would alone have caught plant-it, wanderer, wishlist and homebox, and needs no box. Full slice: a periodic deploy-all sweep on the demo box reusing the campaign's engine. Silent rot is the real risk — an app can die upstream and nobody learns until a customer clicks Telepítés | SLICE 1 SHIPPED 2026-07-21 — app-catalog-felhom.eu/scripts/check-image-resolvable.py (+ 14 fixture tests, no network, resolver injected). Resolves every unique pin with docker manifest inspect, ONE image at a time; exit 0 / 1 (the registry says GONE) / 2 (inconclusive). Two traps encoded, both hit live while building it: (a) docker manifest inspect prints toomanyrequests: … and still exits 0 — the same exits-0-on-failure shape as the ISO tooling's validate-answer, so stderr is inspected even on rc=0; (b) the inverse and more dangerous one — the first full sweep called 24 of 65 pins dead, including postgres:16-alpine and redis:7-alpine, purely because Docker Hub throttled it partway through. Ambiguity therefore resolves to INCONCLUSIVE and never to an accusation: a gate that cries wolf gets ignored, and then it protects nothing. The full sweep is still OWED — DooPlex is not logged in to Docker Hub, so the 52-app table needs one re-run after docker login. Wired into CLAUDE.md + REUSE.md as a start-of-campaign / pre-publish-train step. It immediately paid for itself: it is what turned plant-it and wanderer from 'images do not resolve' into two DIFFERENT diagnoses (see the 2026-07-21 catalog entry). Full slice — the periodic deploy-all sweep on the demo box — remains open |
| R-42 | RULED: sidecar majors follow the APP, never the newest tag. Campaign 7 deliberately did NOT bump sidecar majors (postgres 16→18, redis 7→8, mariadb 11.6→12.3, postgis 16→17) while bumping ~40 app images to current. | S | RULED 2026-07-21 — option (a) | Origin: CAMPAIGN 7 §4. The case for not bumping: a DB major is a data-plane decision the application owns — immich proves it, upstream pins one specific tested postgres:14-vectorchord… build — and postgres:16-alpine/redis:7-alpine already track the newest patch inside their major, so they are not stale. The case for bumping: EOL majors eventually stop getting security patches, and "we never bump" silently becomes "we ship EOL databases". Suggested shape: per-app sidecar pin follows upstream's own compose where upstream publishes one, else stay within the current major and revisit at that major's EOL date |
| R-43 | [P2-HIGH] No offsite restore path can restore a database — offsite restore cannot reconstitute a DB-indexed app. All three offsite buttons are file-only. „Ellenőrző visszaállítás" and „Teljes visszaállítás" restore into a scratch/verify folder and never load a dump into postgres (controller/internal/backup/offbox_restore.go:171-232, :130-164); „Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" is rsync -a --ignore-existing over mandatory userdata paths only — it never touches a DB and never stops/starts the stack (offbox_restore.go:346-421). The DB dump is faithfully carried into the snapshot and can never be replayed out of it by any offsite action; the only code that replays into a live DB is RestoreFromRecoveryUnit (backup/restore_unit.go:74), reachable solely from the local restore panel (web/handlers.go:1108). Compounding: the live recovery unit present — not overwriting guard (offbox_restore.go:400-405) skips the entire unit — compose, manifest and db-dumps — whenever a local backups/primary/<app> exists, which is true on any box whose local backup has run, so on a normal box the offsite dump is never even staged to disk. | M | SHIPPED controller v0.148.0 (2026-07-19) — live acceptance PENDING | SHIPPED: new „Teljes visszaállítás (fájlok + adatbázis)" (ReconstituteFromOffsite, /backup/offbox/reconstitute): safety dump → stop → files overwritten to the snapshot version → start → the snapshot's own dump replayed (reimportDBDumpsFrom, reading the SCRATCH unit — the live unit is still never overwritten) → health wait. Two invariants, both red-proofed: nothing is ever deleted (rsync -a, no --ignore-existing, no --delete; post-snapshot files survive as extras) and the undo exists before the act (the pre-restore- dump is verified ON DISK before anything is stopped/overwritten/replayed; if it cannot be taken the operation refuses with zero changes). Outcome-shaped flash; the old missing-only button now states what it does NOT do. Deployed to demo guest 9201, healthy; page renders, button correctly gated behind a prepared full scratch. STILL OPEN — the §9 live acceptance has NOT run (upload → push → empty the trash for real → one button → photos back in the timeline). Until it does, no capability-map flip, the customer-restore row stays MISSING and R-3 stays DRAFT. — Evidence: audits/DIAG-immich-restore-2026-07-19.md (live on demo-felhom). Viktor deleted 11 immich photos to test offsite restore; both place runs merged 0 files and flashed success. Root cause is two-layered — the UI delete only trashed the assets so nothing was missing to restore, but a valid test would also have failed: files return, the library stays empty, because immich's truth lives in postgres. The customer-visible failure mode: correct bytes on disk and an app that cannot see them. Severity case: this is the offsite tier — the one that survives losing the box — and most of the catalog is DB-indexed. Direction (needs a spec, do not improvise): either offsite gains a real DB-replay leg reusing RestoreFromRecoveryUnit behind a stop→restore→start sequence, or the offsite UI states plainly that it restores files only and routes the customer to the local recovery-unit flow for the DB. Pairs with R-44 (a replay leg is worthless while the dump is stale). Flips: the customer-restore map row stays MISSING and gains this as a note — the C6 evidence attempt found a product gap, which is the rehearsal system working |
| R-44 | [P2-HIGH] A manual offsite push ships an unrefreshed DB dump — "backed up now" is false for the DB half. offboxRunHandler → RunOffboxBackup goes straight to the restic push and never calls RunDBDumps / captureAllRecoveryUnits (controller/internal/web/offbox_handlers.go:203-227, backup/offbox.go:574-759); the recovery unit merely enumerates existing dump filenames via listFileNames, never creates them (backup/recovery_unit.go:105-106). Dumps come only from the separate local db-dump daily at 02:30 (cmd/controller/main.go:542), with the scheduled offsite at 04:15 — so a manual run at any other hour ships a dump up to ~24 h old. There is no freshness check and no RPO surface anywhere: zero RPO hits across controller/; offboxUnitTime is only a two-drive tiebreak (offbox.go:827-837); the DBValidationCache exists (backup.go:364-370) but no offsite or restore path reads it. | S–M | SHIPPED controller v0.148.0 (2026-07-19) | SHIPPED: every offsite run — manual AND nightly — now refreshes the DB/volume dumps and recovery units (offsitePreDump → runDBDumpsInternal) BEFORE the restic capture, so each snapshot is an internally coherent {DB@T, files@T} bundle and retention becomes a history of restorable points. Order is the mechanism and is red-proofed (moving the capture first yields [capture dump]): the gap can only ADD files the DB does not reference yet, never remove one it does. This also makes the nightly ordering structural rather than a coincidence of two scheduler entries at 02:30 and 04:15. Each unit manifest carries offsite_run_id + dumps_at, so a pair's coherence is verifiable at restore time instead of assumed; the periodic refresh carries a prior stamp forward and never invents one. A dump-leg failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none). Honesty surfaces, all warn-level and none a gate: an unstamped pre-v0.148 pair reports its skew in the confirm, and ValidateDump gained an exact-match accounts-table sniff for customer-empty dumps (a substring match on "user" would flag user_metadata/album_user/user_audit on every healthy single-user box — red-proofed). — Evidence: audits/DIAG-immich-restore-2026-07-19.md. Today's unit dump immich-postgres.sql (51 954 452 B, mtime 02:30 CEST) probed to asset: 0 rows, user: 0 rows, album: 0 rows — the 52 MB is entirely immich's shipped geodata_places/naturalearth_countries reference data. It predates both the admin user (created 07:56:25) and the photos (07:57). Same for the unit's immich_immich_postgres_data.tar (323 MB, also 02:30). A dump that looks substantial by size can contain zero customer content — size is not a health signal, and nothing in the product says otherwise. Latent hazard: had a full restore actually loaded that dump it would have written an empty DB over the live one, destroying the trashed rows that were the only surviving recovery path. Direction: dump-before-push on manual runs (the honest fix), or an explicit RPO line in the UI („adatbázis-állapot: ") so the operator/customer can see what they are actually shipping. Cheap interim: surface dump mtime + row-count sanity from the existing DBValidationCache on /backups/restore |
| R-45 | [P2] Unified async-job feedback. Every long operation invents its own progress surface, or none. Tonight produced three more one-off cards (v0.147.x: samba bring-up, offsite progress, restore result) on top of two existing patterns (deploy 3-step panel; storage-init/netstorage status poll). They agree on nothing: some use {ok,data} envelopes and some raw JSON, some poll 1 s / 1.5 s / 3 s, some are in-memory-only and lie after a restart, and each re-implements single-flight + snapshot + phase→Hungarian mapping. | M | idea | Origin: 2026-07-19 feedback slice 1 (controller v0.147.0). The cases to generalise from are all in-tree: web/storage_init_job.go (the best shape — acquire/release/set/snapshot), web/netstorage_job.go, web/samba_ensure_job.go, backup/opstatus.go, backup/offbox_progress.go. Shape: one job registry + one poll endpoint + one client-side renderer, phases declared per job. Two lessons tonight that any framework must encode: (1) a terminal state must be probed, not inferred — compose up -d exits 0 on a crash-loop; (2) a progress source that reports nothing is normal, not broken — restic reports 0 bytes for a whole incremental run, and a bar that sits at 0% is worse than no bar. Also fixes the restart hole: in-memory job state currently vanishes and the card silently disagrees with reality 2026-07-20 — the first bill for NOT having this arrived, and it was customer-facing. The samba card's poll (web/samba_ensure_job.go + sharing.html) mixed a job EDGE and a service LEVEL on one JSON field, and /sharing reload-looped at ~1.2 s for every customer with sharing enabled until controller v0.151.0 (audits/DIAG-sharing-2026-07-20.md, S-1/S-4). v0.151.0 fixed THAT card's contract only — the framework is still this item. Third lesson for it to encode, beside the two already listed: a phase a client answers with a one-shot action must be an EDGE the registry SERVES ONCE, and must never be synthesised from a level; if it can be re-read, it will be re-acted on. |
| R-46 | [P2] Verification copies need a customer-visible browse surface and an expiry. v0.147.0 made them visible (listed with path/size/date, individually deletable) — but the customer still cannot LOOK INSIDE a verification restore to confirm the file they wanted is really there, which is the entire point of a verification restore, and nothing ever removes them. | S–M | idea | Origin: 2026-07-19 feedback slice 4a, registered as the explicit follow-up to it. Two gaps, deliberately designed together because they are the same object: (a) the invisible-result gap — a read-only browse of backups/offsite-restore/<app> (the FileBrowser infra stack already exists and already serves scoped roots, so this may be a mount rather than new code); (b) the disk-lifecycle gap — auto-expiry after N days with the count/size surfaced before it fires, so a drive is never quietly filled by verification restores nobody remembers taking. Pairs with R-43: a browse surface is also how a customer would discover that a DB-indexed app's files came back but the app still cannot see them |
| R-47 | [P2-HIGH] The DB replay races the application's own schema repair. | M | SHIPPED — controller v0.153.0, 2026-07-20 | Fixed on BOTH paths (it was a class defect, not an offsite-only one): offsite ReconstituteFromOffsite AND local RestoreFromRecoveryUnit now open a DB-only window — appbackup.DBServiceNames names the database SERVICE from the compose services: map, stacks.Manager.StartStackServices brings up only that service, the dump is replayed with the application still stopped, and the full start runs only after the replay exits 0. Fail-closed: a .sql dump with no identifiable DB service refuses BEFORE the first mutation. Every exit from the window (replay failure, DB-only start failure) still does a best-effort full start, so a failed restore never leaves a box with a database and no application. --clean --if-exists + ON_ERROR_STOP=1 untouched — the bug was the window, not the flags. Enablers: RedeployFromEnv split into PersistUnitRedeployConfig + its unchanged tail; StackDataProvider.RecreateStackFromUnit → RecreateStackDefinitionFromUnit (the hidden up -d inside the old name is what carried the defect locally). No agent coupling — MinAgent stays 0.90.0. 19 tests + 3 red-proofs, 23/23 green. Unblocks the capability-map offsite-restore (PARTIAL) and customer-restore (MISSING) rows, both of which now await only one clean acceptance run. LIVE-VALIDATED 2026-07-20 against the SAME snapshot that aborted in round 2 (49e7cb46): log shows Starting stack immich services only: [immich-postgres] → replay rc-0 in 20 s → full start; no already exists; immich's own DatabaseService logged No schema drift detected twice (round 2 left it reporting drift); 11 assets active, 4/4 containers healthy. Golden 0.153.0 baked + published the same day (sha256 15fdd191f3c660a6…) — the first golden carrying all FOUR infra images. Evidence: audits/DIAG-immich-restore-round2-2026-07-19.md (H4) + felhom-controller/REPORT.md §4b/§4c |
| R-48 | [P2-HIGH] Restore controls are separable only by layout — and the difference between them is whether the data comes back. The offsite restore row renders four buttons plus hint text into an overlapping, unreadable line, and the decisive second step („Teljes visszaállítás indítása") appears ONLY after „…előkészítése" was pressed, with no signposting that a second step exists or that the first one did nothing to live data. | M | idea | Evidence: audits/DIAG-immich-restore-round2-2026-07-19.md (finding 1) — this is not theoretical: it is the CAUSE of the round-2 incident. An operator who had read the code pressed the missing-only button instead of the full restore; the controller log shows /backup/offbox/reconstitute was never hit at all. The rule this establishes, worth stating once and applying beyond this page: two adjacent controls whose difference is "your data comes back" vs "your data cannot come back" must not be distinguishable only by layout. Direction (ruled in principle, spec rides v0.149): collapse to a single „Visszaállítás…" guided dialog — one intent, visible phases, the escrow-wizard precedent. Pairs with R-45 (the phases are exactly the async-feedback surface) and R-46 SHIPPED 2026-07-21 — controller v0.154.0 (3a9d744). Each app row on /backups/restore now carries ONE „Visszaállítás…" entry linking to a per-app wizard at GET /backups/restore/app?name=<app>: three intent CARDS each with a consequence sentence (ellenőrzés külön mappába / hiányzó fájlok visszahozása / teljes visszaállítás), a visible phase strip so the sequence is legible before the first click, danger styling on the destructive card, and the R-43 double-confirm carried over verbatim with its pair-honesty facts. deriveWizardStep is a PURE function of (op running, size-gate flash, scratch ready) — the step is never taken from the request, and a running op outranks a stale ?full_prep= so no commit button survives into a restore. While ANY op runs every mutation form is suppressed server-side rather than offered and then refused. No new mutation endpoint (one GET route; every card posts to the pre-existing /backup/offbox/* with unchanged field names and gates) and no R-45 graft — the wizard polls the two existing status surfaces as-is. Works with JavaScript disabled. Latent bug fixed on the way: offboxRedirectTo hardcoded "?" when appending its flash, which against the wizard's ?name=<app> target would have buried the flash inside the app name. 9 new tests + the Group-B red-proof (trivial always-INTENT impl → all 7 rows red). Live click-through + one non-destructive Ellenőrzés still PENDING (rides the operator's floor save). Evidence: felhom-controller/REPORT.md §3 (2026-07-21). |
| R-49 | [P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup". Measured 2026-07-19: immich_ml_cache.tar 823 660 032 B (~60%) — re-downloadable ML model weights; immich_postgres_data.tar 308 251 136 B (~23%) — a raw tar of the postgres data dir that DUPLICATES the logical .sql dump captured beside it; upload/backups/ 18 MB — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 dccc13fe… tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: 72 MB of originals. | S–M | idea | Evidence: audits/DIAG-immich-restore-round2-2026-07-19.md §4 (full byte breakdown). This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. Recorded, deliberately not changed — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) immich_ml_cache — pure cache, strongest case; (b) the postgres_data volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) upload/backups/. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app |
| R-65 | Buddy-box backup replication, cross-household — two Felhom boxes in different homes replicate backups to each other. | L | idea (post-alpha, spike-first, 2026-07-22) | The natural big sibling of R-64: two households each hosting the other's encrypted backup tier. Explicitly spike-first — the transport is NOT SMB (R-64's live-share protocol is wrong for backup replication across the internet: no auth story between households, no resumability, cleartext LAN assumptions); candidates to spike: restic rest-server / rclone / syncthing over the existing WG/tailnet plumbing, encryption keyed so the buddy can never read the payload. Sits on top of the offsite tier's FILL/OVERSUB thresholds thinking (R-5 aggregate). Flips: would add a "cross-household buddy replication" capability row (currently unlisted). Pairs with R-64 (same topology, different transport + guarantees) |
Pre-invite checklist — what stands between here and the first remote tester
Not roadmap items in their own right; the short list the 2026-07-18 rehearsal leaves behind. Everything here is remote-doable — the N100 is packed, and none of it needs hands on the box.
| Action | Owner | Note |
|---|---|---|
| Viktor (saves) | Golden 0.146.0 baked on the drill VM and published to gitea — felhom-golden/0.146.0/golden.tar.zst, sha256 4834c703162c5437467a329144b1a523019bf5693ab9d439558be7323587e955, 612 696 588 B (584 MB archive). All pass markers green: Result=success/ExecMainStatus=0, 0 FATAL/exclusions, docker OK (overlay2), all three mounts included (rootfs + mp0 /var/lib/docker + mp1 /mnt/sys_drive), pre-delete 404, upload HTTP 201; controller 0.146.0 confirmed baked in. Integrity round-trip independent of the build host: anonymous `GET |
|
| Golden ≥ 0.147.x carries ALL FOUR infra images | — (next bake) | build-golden.sh v2.1.0 (2026-07-19) now derives the pre-pull list from the controller binary it is about to bake (--print-infra-images) instead of a hand-maintained copy that had already drifted: felhom-samba was never added to it, so every golden so far baked 3 of 4 — which is why enabling Megosztás on a fresh box pulled from the registry with zero feedback. No golden rebuild for this alone; it takes effect at the next bake. Until then a fresh box still pulls felhom-samba at enable time, which controller v0.147.0's progress card now at least explains |
| freemail.hu test-send | Viktor | The open half of R-4; the gmail half closed on 2026-07-18 under p=quarantine |
| C6 — customer performs a restore, unassisted | Viktor as customer zero | The one open script step in R-3 and still MISSING as capability evidence. Remote-doable on the reborn box — the dashboard is remote |
| R-11 rulings | Viktor | Contact channel, tester agreement, alert thresholds (the R-5 gauge thresholds are still pending a ruling) |
Absorbed / superseded notes in this folder
FOLLOWUP-nas-automount-guest-reboot-reassert.md— shipped (agent v0.84/v0.85, CAMPAIGN-3); keep for historyFOLLOWUP-golden-default-controller-tag.md— verify against current golden flow; close or promote to an itemFIX-M18-NOTES.md,FIX-M19-NOTES.md,DIAGNOSIS-f9-storage-registration-gap-2026-06-14.md— historical diagnoses; superseded by shipped fixes