admin
97249099d6
VERDICT. H1 confirmed: the reconstitution never ran. /backup/offbox/reconstitute
was never hit (reconstituted/safety-dump/replay line counts all 0); the operator
clicked the old missing-only button, which merged 34 files and left the DB
untouched. H2 rejected (the full path never executed; staging used the correct
snapshot 49e7cb46). H3 rejected (no replay happened; DB was genuinely asset:0).
R-44 is exonerated and working: 49e7cb46 carries offsite_run_id 20260719T102851Z
and its dump probes to asset:11 / user:1 — a properly coherent pair, against
round 1's asset:0 / user:0.
H4 (NEW, the more important result). Phase-3 recovery ran the real sequence and
the v0.148.0 path executed correctly — safety dump, stop, start, replay — then
the replay ABORTED: relation "clip_index" already exists. Proven to the second:
10:58:25 controller begins replay
10:58:33 immich-server "Reindexing clip_index" -> "Reindexed clip_index"
10:58:35 dump's CREATE INDEX clip_index fails, exit 3
The replay races the app's own schema repair, because reconstitution starts the
stack before replaying (ImportDump needs a live container). The photos survived
only because pg_dump emits COPY data before CREATE INDEX — luck, not design. The
same start-then-replay shape is on the LOCAL path, so it is a class defect.
Recovery: PARTIAL and STOPPED as instructed. All 11 assets are back and visible
by every DB criterion (status=active, deletedAt null, all files resolve), but the
operation reported failure and immich reports schema drift. Not clean acceptance
evidence; no second attempt made; safety dump read only.
Findings -> R-47 (the replay race), R-48 (restore controls separable only by
layout — the proven cause of this incident), R-49 (1.1 GB of a 1.2 GB immich
backup is ML model cache + a postgres data-dir tar duplicating the .sql dump +
immich's own nightly dump; recorded, not changed).
RULINGS LANDED.
- 00-capability-map.md:61 -> PARTIAL, scope-corrected: 6D's destruction hit the
FILE TREE only; the DB survived in its named volume, so "end-to-end from
offsite alone" overclaimed scope.
- The 704.6 MiB "discrepancy" -> CLOSED, not a defect: it was immich's own
Tarhely widget, never a controller page. Same for round 2's 650MiB->1.4GiB.
- customer-restore row: partial evidence recorded, NOT flipped.
Diagnosis only — no code, label or layout changes; no forget/prune.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
68 KiB
ROADMAP — future features & open work
What this is: the prioritized decision log of planned/open work. Items are intentions, not claims about live behavior — the capability map (
architecture/00-capability-map.md) is the only place that states what the platform does today.Lifecycle: idea → spiked → spec'd → in-progress → shipped (item collapses to a one-liner with the version, and the corresponding capability-map row changes status with evidence). Items can also be killed (keep the one-liner + why — decisions are worth remembering).
Coupling rule: every item names the capability-map row(s) it flips. Every map gap row points back here by ID. Neither file duplicates the other's content.
Priorities: P1 = closed-alpha blocker · P2 = close during alpha · P3 = post-alpha. Existing loose notes in this folder (
FOLLOWUP-*,FIX-M*) are absorbed as references below.
P1 — closed-alpha blockers
| ID | Item | Size | Status | Notes / map rows flipped |
|---|---|---|---|---|
| R-1 | Peti convergence — the appliance half is DONE; this item is now Peti-only. Rehearsal EXECUTED 2026-07-18 (tests/VALIDATION-n100-rehearsal-2026-07-18.md): the full final-product flow ran on real metal in one pass (RESET → generic ISO → customer self-bind → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer — the one thing a demo box on the operator's own LAN can never prove. |
L | rehearsal DONE; Peti half open | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. Reinstall arc SHIPPED hub v0.57.0 (2026-07-16): the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path |
| R-2 | ~hub/internal/notify/, store.go, hub/internal/claim/) |
S | killed (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — notify/+claim/engine.go in 6b40eb8 (v0.50.0), store.go in a1d0450 (v0.54.0), plus follow-up e205a2d; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
| R-3 | Friend-alpha onboarding runbook (generalized from pilot/RUNBOOK-peti-return-2026-07-13): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step |
M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of PETI-tester-agreement.md. Next from-scratch rehearsal to include customer DELETE + re-create — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). S6b (manual stale-host delete before re-enroll) is OBSOLETE — re-enrollment upserts the existing host row cleanly (store.UpsertHost ON CONFLICT DO UPDATE; handleAdminCreateHost no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE): one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). Remaining consistency gap: the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu / citromail.hu / t-online.hu; tighten DMARC p=none → p=quarantine (pending since email.md 02-04) |
S | DNS half DONE + gmail half DONE (2026-07-18); freemail.hu open | A claim code in spam bricks onboarding at step 1. DMARC now p=quarantine (RUNBOOK-publish-0.90-0.143-2026-07-18 Phase 4): _dmarc.felhom.eu TXT v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu set by Viktor (CF, Path B); dig gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM d=felhom.eu + relaxed send. SPF). gmail half CLOSED by the rehearsal (2026-07-18): the run's claim email was the first sent under p=quarantine and landed in the gmail Inbox, not spam (tests/VALIDATION-n100-rehearsal-2026-07-18.md). Remaining: the freemail.hu test-send (Viktor, remote-doable, dmarc=pass check) |
P2 — during alpha
Sub-rank
P2-HIGH= close before the first REMOTE tester. These are the 2026-07-18 N100 rehearsal's findings (tests/VALIDATION-n100-rehearsal-2026-07-18.md). They are not P1 — the rehearsal proved the product flow works — but each one either misleads the operator, misleads the customer, or hides a failure, and all of that gets materially worse the moment the box is somewhere you cannot walk over to.
| ID | Item | Size | Status | Notes |
|---|---|---|---|---|
| R-39 | [P2-HIGH] The PBS DR tier can be applied and dead at the same time — and nothing notices. On the reborn N100 the descriptor auto-provisioned and the agent converged state=applied, yet pvesm status reports felhom-pbs: error fetching datastores - 401 Unauthorized / inactive and a direct probe with the stored credential 401s on every endpoint including /version (WG transport healthy: handshake 9 s, 27.9 ms RTT — so authentication, not ACL scope). Three compounding defects: (a) a mint/consume race — the hub minted a SECOND token secret at 16:47:52, two minutes after the agent applied the first, and consumed_at is still NULL; (b) the converged state machine will not re-apply, so the box is pinned to a stale secret; (c) the agent's 15-minute PBS verify loop cannot read the credential to detect any of it (open /etc/pve/priv/storage/felhom-pbs.pw: permission denied — the non-root agent writes that file through a root sudo wrapper, then reads it directly). |
M | diagnosed + box healed 2026-07-18; FLEET FIX PENDING SPEC | DIAGNOSIS (2026-07-18, live on the N100 — supersedes the initial hypothesis). The brief guessed "the re-mint fails to bump the generation". That is FALSE and no hub fix was shipped: store.SetHostDesired bumps desired_generation unconditionally (it went 2→3 on the re-issue), and web/configs.go's applyPBSDR is likewise exonerated — its "no re-key, no second secret, no spurious generation bump" comment is accurate, guarded by the cur != nil && cur.Namespace != "" early return, and the hub log shows mint #2 came from the re-issue path, not from an Edit-tab Save. The real mechanism is a signal mismatch between the two tiers. The hub's re-consume signal is a generation bump + a poke; the agent's re-apply trigger is a change in the DESCRIPTOR CONTENT HASH (felhom-agent internal/pbsdr/manager.go ~L235: `if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil |
| R-30 | [P2-HIGH] Liveness presence should come from the wait channel, not the report clock. The box was powered off at the start of the rehearsal, yet the hub carried it as healthy until the staleness threshold expired ~30 min later (host_stale 16:05:24 "no report for 30m"; cleared 16:33:24 "was stale for 27m"). The host-delete guard compounds it: RESET refuses while any host row exists, so a stale-but-"Online" host stalls a forced teardown. |
M | idea | Direction: derive presence from Dir-2 long-poll connectedness (~90 s grace), decoupled from notification hysteresis (the hysteresis is right for alerting, wrong for presence); an agent/ep0 analog can follow. Pairs with R-13/R-23 — the transport already exists, this is about believing it. (Discussed in-session as "R-29"; that number was already taken by the gate-rot item earlier the same day, so it is R-30.) |
| R-31 | [P2-HIGH] Offsite provisioning is synchronous with no status affordance. Save runs the Hetzner sync in-request, so the request can hit the nginx 504 while succeeding server-side: the operator cannot tell failed from slow, and a retry races the first attempt. | M | idea | Direction: make it async + a status card, reusing the proven awaiting-card/poll idiom (v0.138.0 escrow card). Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click. |
| R-32 | [P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible. The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's "hetzner":"ok" leg destroys the sub-account, but a Hetzner sub-account is an access-control object, not a data object — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. |
M | idea | Ruling from the run (three parts, deliberately separate): (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable BY DESIGN → RESET gains a main-account purge of the customer base dir (the existing operator ack already covers it); (2) the move-aside guard STAYS for reinstall-without-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator Restic tab shows per-customer directory bytes vs attributed snapshot bytes, so dead data cannot hide. Measured on the pool box that night: 49 M attributed (2 snapshots, 48.717 MiB) against 1.4 G + 3.0 M unattributed across TWO .orphaned-* dirs. Evidence restic-and-pool.txt |
| R-33 | [P2-HIGH] Bootstrap pairing-poll spams the customer-visible console. Each 30 s pairing poll is a separate felhom-bootstrap.service run that exits 1, so systemd prints Failed to start … on the screen the customer is watching — 52 FAILED lines in ~11 minutes while nothing was wrong (the box was correctly waiting to be bound). |
S | SHIPPED (scripts v1.21.0, 2026-07-18) | run_pairing() is now a while true loop sleeping POLL_INTERVAL (30s — the hub-side rate is unchanged) instead of exiting non-zero per poll, so the unit sits in activating and systemd prints nothing. Registration split into register_appliance(), whose transient failures the loop retries rather than taking the unit down. The journal is quiet without going dark: the wait is logged once on entry (worded so nobody reads it as an error), then on a 10-min heartbeat with elapsed minutes; 410 still exits non-zero on purpose (delivery consumed but no local env is a real crash window). Console banner every 5 min instead of every cycle, single accented spelling, plus the reassurance the rehearsal showed was missing („Ez a képernyő magától frissül"). The load-bearing half is TimeoutStartSec=infinity in the unit — a Type=oneshot ExecStart is killed at DefaultTimeoutStartSec (90s), so without it systemd would kill the new wait and Restart=on-failure would silently reinstate the exact spam this removes, after appearing to work for the first three polls. Verified behaviourally in a container against a stub hub answering 204 five times then delivering: one log line plus one heartbeat, zero exits between polls, then a clean fall-through to the direct install and exit 0 — where the old design produced 5 unit invocations and 5 Failed to start console lines. ISOs rebuilt at v1.21.0 and the shipped payload verified byte-identical to the committed source. Evidence bootstrap-journal.txt; ship commit bcdb042 |
| R-5 | Hub: offsite storage visibility — RESTIC box aggregate (v0.64.0) + PBS DR datastore (v0.65.0), each with fill + operator alert; Offsite tabs + dual dashboard gauges | M | SHIPPED (hub v0.64.0 + v0.65.0 + tenantsync v1.2.0, 2026-07-17) | Restic: monitor.OffsiteBoxChecker (throttled Hetzner GET; fill 80/90% + oversub 2.0×; scope "pool-box"). PBS DR (v0.65.0, Option A): read-only usage op on the ep0 felhom-tenantsync forced command (df, no mutation) → monitor.PBSDRBoxChecker (fill 80/90%, states ok/unavailable/degraded, scope "pbsdr-box"); /offsite Restic/PBS-DR tabs + two gauges; hub deploy ⟂ ep0 update (graceful "n/a"). Both data pipelines PROVEN-LIVE (restic 0.2% full; PBS 19.1% full, 7.1 GB of 37.2 GB after ep0 → v1.2.0). Thresholds pending Viktor (named keys; separate restic/PBS pairs). OPEN: UI render unit-only (password-gated), alert emails unit + red-proof only (both nominal). See hub v0.64.0 + v0.65.0 REPORTs |
| R-6 | Spike: LAN service discovery from the guest — SSDP multicast (UDP 1900, DLNA), WSD (Windows discovery), mDNS; host-network vs macvlan; is the customer LXC LAN-bridged in appliance deployments? | M | spiked (2026-07-18) | VERDICT: appliance guest IS LAN-bridged (own DHCP lease on the household /24); multicast discovery works ONLY in the guest netns — guest-direct or Docker --network host (SSDP/mDNS/WSD all PASS both ways); the default docker bridge is categorically DEAF to LAN multicast (WSD/mDNS RX FAIL, unicast-publish PASS). Real samba+wsdd on host-net → Windows 11 ProbeMatch + FELHOM-SPIKE renders in Explorer + 445 + authenticated SMB round-trip all PASS; real SSDP MediaServer:1 advert reaches both LAN clients. → R-7 SMB stack MUST be host-network LAN-bound; R-8 Jellyfin-DLNA plausible if host-network. Caveat: vmbr0 multicast_snooping=1 worked only because the household router is a live querier — customer LANs w/ snooping+no-querier, and Peti's BYO bridge, are UNTESTED gaps. S4b (human leg, the sharpest finding): wsdd makes the box VISIBLE but the Explorer double-click FAILS 0x80070035 — WSD gives no name resolution; the flat \\FELHOM-SPIKE resolved by no path. Adding nmbd (NetBIOS) fixed it live (flat name resolves + mounts). → R-7 needs smbd+wsdd+nmbd (+avahi/.local for modern clients), not wsdd alone. Doc: audits/SPIKE-lan-discovery-2026-07-18.md. |
| R-7 | SMB server share — LAN file sharing as an embedded controller feature („Megosztás") | L | SHIPPED slice 1 (controller v0.144.0 + felhom-samba:1.0.0, 2026-07-18) |
Flips map row Files from Windows Explorer / Mac Finder (SMB server) MISSING → IMPLEMENTED. Delivered as the FOURTH protected infra stack (host-net, smbd+nmbd+wsdd, bind interfaces only = lo eth0), NOT a catalog app. „Megosztás" → „Hálózati megosztás": enable + ONE household password (STDIN→smbpasswd, never persisted — only user_set), shares table, create flow (new folder under <storage>/shares/ or an existing folder via a guarded picker). Picker guard = absolute → EvalSymlinks → containment in a registered LIVE root → deny-listed system subtree (derived from ProtectedHDDPaths, provably a subset) → is-a-dir, with UNIFORM refusals (no filesystem oracle). Per-share read-only enforced in smb.conf AND as a :ro bind. force user = felhom → SMB writes land as uid 1000. Live-validated on demo through the real endpoints + a Win11 workstation (445 True, NetBIOS FELHOM resolves, write/read byte-compare PASS, write to a read-only share REFUSED, WSD ProbeMatch). Explorer leg PASSED 2026-07-18 (Viktor): both shares open from the Network view; an Explorer save into the writable share landed as uid 1000 and a write into the read-only share was refused, folder untouched. Slice 1 is fully PROVEN-LIVE. Docs: controller/sharing.md; ship report felhom-controller/REPORT.md. Slice 2 remainder: avahi/.local for modern+Apple clients, curated app-folder presets (one-click paperless consume/), per-share users + guest access, recycle bin. Backup gap split out → R-7b. |
| R-7b | Share backup EXECUTION — put share data into the live tier-2 + offsite runs (the design fork reported by R-7 slice 1) | M | SHIPPED (controller v0.145.0, 2026-07-18) | Viktor's ruling: Model B′ — a SIBLING shares source. New, additive job/leg code reusing the proven primitives (tier-2 mirror seam, restic wrappers, soft-quota/enlargement gate, status recorders) while leaving every per-app engine path byte-identical — NOT a synthetic recovery unit (breaks on multi-drive shares, wraps 1 KB of JSON in dump machinery) and NOT engine-loop surgery. The B′ invariant is enforced by test in both tiers, red-proofed. Tier 2 → RunSharesTier2 (legs grouped by SOURCE drive → backups/secondary/_shares/<driveKey>/<share>, payload at _payload/, layout marker LAST). Tier 3 → runOffboxSharesLeg: ONE extra restic backup --tag felhom-offbox --tag _shares placed after the app loop and BEFORE retention, so forget --group-by host,tags covers the new group with no flag change; a quota-blocked push degrades to the manifest only, never to nothing. Restore → „Megosztások" on /backups/restore: scratch, then a missing-only merge whose every destination is PREFIX-ASSERTED against live storage roots, definitions merged existing-wins, then ReconcileSamba, then the credential. The payload (_shares-manifest.json + a best-effort secret-bearing passdb.tar) is what makes DR return files + configuration + password rather than loose bytes. Fold-in: samba joins the liveness set — EffectiveProtected adds the CONTAINER felhom-samba exactly while sharing is on. FULLY PROVEN-LIVE on demo (2026-07-18), all four legs. (1) tier-2: real /api/backup/tier2 trigger → _shares tree + marker + payload on the cross-drive target, mirrored file md5-identical, payload 0600 preserved. (2) offsite: Viktor's manual run 12:18:16Z → snapshot e0b9d723 (tags felhom-offbox,_shares) with the payload dir + both share folders; a second run via the „Távoli mentés" button → 4e2b15ec, containing _shares-manifest.json (418 B) AND passdb.tar (855 040 B), both 0600, share files with uid 1000 preserved. (3) restore round-trip: probe file + the dokumentumok DEFINITION deleted via the real endpoints, then „Megosztások" restore + place → 1 file(s), 1 definition(s) re-added, 1 kept, 0 refused, credential=true; probe back md5-identical, the two pre-existing files NOT overwritten (missing-only proven on live data), definition back with its ORIGINAL flags and created_at, smb.conf re-rendered, filmek untouched. (4) liveness: samba stopped → health_critical pushed and hub-accepted (200) → self-healed. Remaining human leg: SMB positive auth with the real household password (never persisted by design). Correction: an earlier revision of this row and of the ship REPORT wrongly claimed the demo box had no offsite target — the verification read a guessed settings key (offbox_target) instead of the real one (offbox); root cause dissected in REPORT §7b. Findings: the reserved-name assumption was FALSE (nbNameRe accepted „_shares" as a share name — now refused); the alert/e-mail pipeline needed NO change and adds no new event type. Docs: controller/sharing.md; ship report felhom-controller/REPORT.md. |
| R-8 | DLNA (gate input now exists — R-6 spiked 2026-07-18: SSDP reaches LAN clients from host-net): validate Jellyfin's built-in DLNA server first; only add minidlna to the catalog if Jellyfin-DLNA fails | S | idea (unblocked) | Don't add catalog weight before proving the cheap path. R-6 confirmed the cheap path is physically viable — Jellyfin DLNA must run host-network (same multicast constraint as R-7) |
| R-9 | Uninstaller trio (from 07-15 Peti session): cluster-aware felhom_guests guard (node-local pct list deletes cluster-wide pveum objects); saferemove detection + time estimate + opt-in --quick-remove (never mutate storage.cfg); smarter restore_storage default for BYO clusters (shared storage, not local-lvm) |
M | idea | Second item's rejected alternative (temp-disable-and-restore) stays rejected — crash window silently downgrades cluster wipe policy |
| R-10 | T-6E-1: DB-dump dir-fsync asymmetry (LOW, confirmed in 6E) | XS | idea | One-line hardening; batch with the next controller task |
| R-11 | Tester-facing one-pager: what the box does, known limitations, how to report (channel decision: Messenger group?) | S | idea | Pairs with R-3 |
| R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17): the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above |
| R-22 | PBS-DR pre-check self-grant (F4). On a non-default storage id the token-auth GET /storage/<id> pre-check 403s (no ACL yet) and used to abort before the root-run grant that creates it. |
S | SHIPPED + PROVEN-LIVE agent v0.89.0 (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → converged state=adopted in ~3 s, ACLs restored, offsite active). Origin tests/VALIDATION-n100-baremetal-2026-07-16.md F4. |
| R-17 | Old-box archive (u629193-sub1) retirement decision — 9/9 byte-identical restores verified | XS | awaiting-decision | Viktor ruling |
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
| R-20 | XS | closed (2026-07-16) | Confirmed against scripts/felhom-host-install.sh source (not changelog): keys resolve at L1181–1219 (script constants OPERATOR_KEY_*, populated, --operator-pubkey-file override), pinned automatically by step_agent_config() "STEP 6/8" (L2044; python builds authz.signers L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
|
| R-23 | Immediate-sync Direction-2 follow-ups (hub v0.58 / controller v0.140, shipped 2026-07-16): (a) live-validate the operator-UI save→apply round-trip end-to-end — needs an operator login, so fold into a Peti/alpha supervised session (one config save → box wakes in seconds → self-restart → startup report; also demonstrates the restart single-fire once a bump has advanced the generation past 0); (b) cosmetic: the controller Waiter's "recovered" INFO logs on the next hold completion (pollOnce blocks ~240 s), not at reconnect — surface recovery at connect time |
S | idea | Flips the new map row "config/state change round-trips in seconds" PARTIAL→PROVEN-LIVE. Transport + mechanism already proven live (SPIKE-immediate-sync-transport-2026-07-16, v0.58/v0.140 REPORTs: 240 s no-annotation hold, 0.047 s wake, restart 1-WARN/0-storm); only the login-gated UI-triggered bump leg is unexercised |
| R-34 | Backup data lifecycle management. An "inactive backups" section on „Távoli mentés": apps that have snapshots but no active backup — disabled OR uninstalled — listed with name / size / last snapshot / restorable, plus an explicit double-confirmed per-app delete via restic forget --tag + nightly prune. |
M | idea | RULING: the offsite toggle NEVER offers deletion — policy and destruction stay decoupled. Turning backups off must never be a data-destroying act, and deletion must never hide behind a toggle. Origin: 2026-07-18 rehearsal. Pairs with R-32 (that one is the operator's view of dead bytes; this one is the customer's) |
| R-35 | Config-apply should not end the customer's session. The offsite config push bumped config_version 10→11 at 16:54:58 and the controller self-restarted (container StartedAt 16:54:59Z, back up 16:55:02); in-memory sessions died with it and customer zero was force-logged-out mid-flow. |
S | idea | Direction: hot-apply the offbox target (no restart for a config the running process can adopt), or persist sessions across restart. The restart itself is by design — the collateral is not. Evidence controller-log-full.txt |
| R-36 | Post-RESET re-enroll leaves offsite "enabled but unprovisioned" — silently. The hub knows the state and says nothing on the customer page. | S | SHIPPED (hub v0.67.0, 2026-07-18) | Both halves delivered. (1) The warning: the customer page now names the state and the fix — enabled-but-unprovisioned raises an amber banner saying provisioning is Save-triggered (press Save once, then verify), reusing the exact enabled && type == "" predicate the offsite re-issue handler already refuses on. (2) The related sub-item, also done: the self-bind link is now auto-minted at customer creation AND at RESET completion, so the console banner's „e-mailben kapott link" is already true instead of true-once-the-operator-remembers. Extracting the shared mintAndSendSelfBindLink core keeps the button and the auto-mint callers on the same F1/F2 honesty rules, and the auto-mint never fails the operation it rides on. Gap found and closed while wiring it: PurgeCustomerResetDBState does NOT clear selfbind_tokens, so a link minted BEFORE a reset would have stayed live across it — the skip paths now clear stale tokens, giving the invariant "after auto-mint, the only live link is one we just issued, or none". Tests assert the banner is ABSENT in all three nominal cases too; red-proofed. — Original analysis: Source-cited behaviour, confirmed live in the rehearsal: provisioning is Save-triggered (configs.go applyOffsite) — which also answers S6's open question — and the re-enroll auto-re-issue correctly skips unprovisioned targets (handler.go). So nothing is broken; the gap is that nobody is told. Direction: flash it on the customer page. Interim: an R-3 step. Related sub-item: auto-mint the self-bind link on customer create/RESET, so the console banner's „e-mailben kapott link" is always already true instead of true-once-the-operator-remembers |
| R-27c | Customer self-bind, slice 2 — console-passphrase bind. Viktor's direction: bind using a passphrase shown on the box console, alongside (not instead of) the emailed capability link. | M | idea | Security constraints from the session ruling, all load-bearing: passphrase issued at customer creation; the global-lookup endpoint must be spray-hardened — per-appliance and per-IP caps, constant-time comparison, a single generic failure (no oracle), alerting on abuse; an accent-free wordlist (console keymaps are not Hungarian); the web capability-link path is RETAINED; claim-by-email is RETAINED as the delivery-channel proof. Also under this item: the self-bind email gains the public universal-ISO download link + two-line instructions (the DIY case). Secret-bearing per-customer ISOs are ruled OUT. Sibling of R-27b (second-box flow) — different axis, both build on the same /bind/ page |
P3 — post-alpha
| ID | Item | Size | Status | Notes |
|---|---|---|---|---|
| R-26 | Guided old-history recovery via a retained superseded escrow + the recovery code. Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (host_escrow_superseded, ListSupersededEscrow). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside .orphaned-<date> repo for restore. |
M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin DIAGNOSE-offbox-repo-orphaned-2026-07-17 |
| R-27 | Customer-facing self-bind page (R-21 slice C follow-on). Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | SHIPPED (slice 1, hub v0.66.0, 2026-07-17) | Slice 1 = the FIRST-box flow, DONE: operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor /bind/<token> (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same BindAppliance, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP /bind/ exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). Slice 1 PROVEN-LIVE 2026-07-18: in the rehearsal the operator minted + emailed the link at 16:28:55 and the customer bound their own box at 16:29:55 with zero failed attempts, appliance_bound carrying source customer_selfbind, credential delivered 26 s later with no operator action (tests/VALIDATION-n100-rehearsal-2026-07-18.md). The new-ISO console banner rendered on the real box. Slice 2 = console-passphrase bind → R-27c (below). Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. Multi-box per link is out of scope by design (single-active, one-shot → binding several boxes = repeated operator sends) |
| R-27b | Customer self-bind, second-box flow (controller side). For a customer who ALREADY has a bound box and installs another, the controller shows a dismissable "bind another box" prompt (and a bind-later entry under settings) that walks to the hub /bind/ page — so a returning customer isn't emailed a fresh operator-sent link for every box. Mechanism sketched in the hub v0.66.0 REPORT; NOT built (R-27 slice 1 deliberately did not touch the controller). |
M | idea (minted by hub v0.66.0) | Origin: hub v0.66.0 slice-1 ship (first-box only). Reuses the same /bind/ public page + tokenized-link machinery; adds a controller-side entry point + the operator "mint a link for an existing customer" affordance |
| R-25b | Customer DELETE ↔ RESET consistency. The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown — it purges escrow custody + drops the config only. Decide the model: DELETE requires a prior RESET, or DELETE subsumes RESET's teardown, or they stay orthogonal (RESET = recycle-in-place, DELETE = escrow-purge). | S | idea | Origin: hub v0.61.0 RESET ship. Flips a future "customer fully offboarded (external resources released)" map row. Cheap once the model is chosen |
| R-25 | Device-node TOCTOU hardening (drive init). Graduate the controller v0.141.0 Observation: the format → resolveEnrollUUID(path) → AssignDisk(uuid) sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. |
S | idea | From the v0.141.0 F6 commit's security-review finding (felhom-controller REPORT). Low real risk (single-operator, agent-guarded), but cheap to close |
| R-24 | Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17). The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE SetConfig (live cgroup, no reboot, Phase-0 proven). Framing note: the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). COMPLETE — the apply is proven in BOTH directions on metal (2026-07-18): customer zero shrank the guest 11675 → 8192 MB (16:50:22) and grew it 8192 → 12288 MB (17:02:17), each a live cgroup apply with no reboot, the new total rippling into the deploy page's memory math. The nested demo had proven the refusals; the N100 proved the applies. tests/VALIDATION-n100-rehearsal-2026-07-18.md ledger 9. Cores/live-resize as hub desired-state is NOT built (deferred, low demand). |
M | SHIPPED + PROVEN-LIVE | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". |
| R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster |
| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | first slice PROVEN-LIVE (poke channel) | FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17): the ep0-relayed contentless poke (hub→ep0 felhom-poke forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the agent and fires an immediate desired-state cycle. Full path live: real operator manifest save → sync-poke delivered to 10.77.0.2; box → poke received → immediate desired-state cycle (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per SPIKE-immediate-sync-transport-2026-07-16 P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately |
| R-28 | Agent fast-tick-until-first-convergence — SHIPPED (agent v0.90.0, 2026-07-17). Closes the immediacy leg a hub poke can't reach (the pre-tunnel WG-registration window): while ANY desired-state item is unapplied the agent pulses the out-of-band trigger every 30 s and self-disarms on convergence (state-based; four cached sources — desired-gen==0, reconcile Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational; LOUD states + pending_signature excluded). LIVE on both demo agents (startup line verified). Guests-0/0 passenger: diagnosed live and REFUTED as a bug — the 0/0 was the legitimate pre-provision reporting window; the fast-tick is exactly its mitigation (≤15 min → ~30 s). Unfired live: the real-onboarding proof (a fresh box converging in seconds without a poke) → the rehearsal S5. | M | SHIPPED | See felhom-agent REPORT (v0.90.0). Third immediacy leg alongside R-13 (poke) + R-23 (UI). |
| R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | |
| R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11) |
| R-21 | Bare-metal Felhom ISO — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes felhom-host-install.sh) → universal secret-free / operator-bind (option C) |
XL | SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18) | PHYSICAL CLOSURE 2026-07-18 (tests/VALIDATION-n100-rehearsal-2026-07-18.md): the generic pairing ISO v1.20.0 (--loader mkimage, SB off) booted the very AMI board that F1 blocked, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. F1 is closed on physical hardware and the rehearsal dependency on this item is discharged. Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → SB must be OFF. The installer's GRUB menu still offers interactive installers → new item R-38. — Prior: PHYSICAL RUN 2026-07-16 (tests/VALIDATION-n100-baremetal-2026-07-16.md): demo N100 reinstalled clean-slate from a pipeline ISO → chain reached rc-0 first try on real hardware (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. F1 (HIGH, slice-B input): this cheap AMI AN3PLUS 0.01 firmware won't UEFI-boot the ISO's GRUB from USB (relocation 0x0) — SB-off/shim-bypass don't help; worked around live with a grub-mkimage loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310: build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). Slice B — SHIPPED (scripts v1.18.0, 2026-07-17): the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim |
| R-29 | The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases. controller/scripts/docker_run_volume_path_gate.py has failed continuously since 2026-07-14 (v0.129.0) and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. (a) The finding itself is benign and the fix is 3 lines. The flagged call is internal/appexport/estimate.go:179 docker run --rm -v <volumeName>:/vol:ro alpine du — a NAMED-VOLUME mount, i.e. daemon-side with no host path, which is the safe shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (export.go volName+":/vol", backup.go volName+":/vol:ro", restore.go volName+":/vol"). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, not a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. (b) The systemic half is the real item: the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note 'Windows green gate silently red (read-only fsync)'. Decide where they run (pre-push hook, build.sh step, or a CI job) and make a red gate block the train the way the Go green gate does. |
S (a) / M (b) | idea | Origin: R-7b close-out, felhom-controller REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller template_id_gate / emoji_gate / native_confirm_gate / offbox_rename_gate / mojibake_gate / app_row_dedup_gate / docker_run_volume_path_gate, hub hub_confirm_gate, manifests manifest_bearer_gate, website site_gates. Do not bundle (a) into an unrelated feature commit — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. 2026-07-18 rehearsal note: the run's finding list independently re-raised "assign the pre-existing docker_run_volume_path_gate failure its ID so red stops normalizing" — that is this item; no second ID was minted. |
| R-37 | Post-RESET health card shows stale pre-RESET warnings. After a RESET the card should read „RESET óta nincs adat" instead of carrying warnings about a lifecycle that no longer exists. | XS | SHIPPED (hub v0.67.0, 2026-07-18) | The customer page raises a banner when a RESET completed after the newest report, quoting „RESET óta nincs adat" and the reset timestamp, because until the box reports again every health figure describes a lifecycle that no longer exists. Deliberately narrow: an in-flight reset does not trigger it (only a completed one), and it clears itself on the first post-RESET report. Ties resolve to STALE — SQLite timestamps are second-resolution and a same-second report almost certainly arrived just before the reset destroyed what it describes; erring the other way would hide the banner exactly when it matters most. Red-proofed (neutering the predicate fails the assertion). — Origin: 2026-07-18 rehearsal. Same family as R-36 — the hub knows the state changed and the UI has not caught up |
| R-38 | Installer GRUB slice. A single default „Felhom telepítés" entry; the interactive installers REMOVED (safety: an interactive entry is how a wrong-disk manual install happens); felhom background. | S | SHIPPED (scripts v1.22.0, 2026-07-19) | Origin: 2026-07-18 rehearsal, alongside R-21's physical closure. Every ISO is repacked after prepare-iso: exactly one entry („Felhom telepítés", default, 5 s), with Graphical / Terminal UI / serial and the whole Advanced Options submenu (nomodeset ×2, three debug variants, Rescue Boot, memtest, UEFI settings) not emitted — not hidden, not password-gated. Boot behavior unchanged: the linux/initrd lines are lifted verbatim at repack time from the ISO's own automated entry, so a PVE bump tracks automatically; the build fails if they are missing, if the append line has lost proxmox-start-auto-installer, or if auto-installer-mode.toml is absent. Gated for 1 entry / 0 submenus / no live banned references, then re-verified by reading the menu back out of the finished ISO. Background generated at repack time from website/assets/og-image_2.png (one source, no second copy to drift), with the card's own grid phase-locked across the letterbox fill. Live-validated on the nested canary (UEFI/OVMF): renders, auto-fires, reaches the same match-nothing abort as v1.21.0. Squashfs/theme rebranding still DEFERRED — post-GRUB screens remain Proxmox-branded. GRUB's e/c keys also remain reachable (suppressing them needs a superuser password — a separate decision); the menu no longer advertises them |
| R-40 | [P2-HIGH] The update path cannot express a MULTI-HOP major upgrade. A template pin is a single value; the customer's update button pulls whatever the catalog now says. For apps whose upstream forbids version skipping this produces a broken upgrade. Nextcloud is explicit: "You cannot skip major releases. Please re-run the upgrade until you have reached the highest available release." Campaign 7 moved its template 31 → 34 (a fresh deploy validates fine — 302, 3/3 healthy), so an existing 31 customer pressing update would attempt a jump Nextcloud refuses. | M | idea | Origin: CAMPAIGN 7 (audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md §7 F7). Not nextcloud-only — any app with sequential-major rules (gitea, tandoor, outline…) has the same shape. Directions: a per-app upgrade_path:/max_hop: in .felhom.yml that the update button walks in stages; or refuse-and-explain when the installed major is >1 behind; or pin an intermediate "stepping-stone" tag. Until this exists, a >1-major catalog bump is safe for NEW deploys and unsafe for the update button — which is exactly the asymmetry the campaign's MAJOR flag was meant to record but cannot enforce |
| R-41 | [P2-HIGH] The catalog has no standing "does every template still deploy?" check. Campaign 7 was the first thing that ever tried to deploy all 53 apps, and found 5 that had NEVER been deployable: papra (missing required AUTH_SECRET), zipline (v4 renamed CORE_DATABASE_URL → DATABASE_URL), wishlist (Docker Hub image gone; upstream moved to ghcr.io), homebox (upstream dropped the v tag prefix + new required env), glance (needs a seeded glance.yml the template never provides — PROVEN pre-existing: the pre-campaign v0.7.4 pin fails identically). Plus 7 broken healthchecks and 2 apps whose images no longer resolve at all (plant-it, wanderer). | M | idea | Origin: CAMPAIGN 7 (§7 F5/F6). The repo already has the right pattern in scripts/check-image-pins.py — a mechanical gate run on every change. Cheap first slice: a resolvability gate (docker manifest inspect every pin) would alone have caught plant-it, wanderer, wishlist and homebox, and needs no box. Full slice: a periodic deploy-all sweep on the demo box reusing the campaign's engine. Silent rot is the real risk — an app can die upstream and nobody learns until a customer clicks Telepítés |
| R-42 | Ruling needed: do DB/cache sidecar majors follow the app, or the newest tag? Campaign 7 deliberately did NOT bump sidecar majors (postgres 16→18, redis 7→8, mariadb 11.6→12.3, postgis 16→17) while bumping ~40 app images to current. | S | decision pending (Viktor) | Origin: CAMPAIGN 7 §4. The case for not bumping: a DB major is a data-plane decision the application owns — immich proves it, upstream pins one specific tested postgres:14-vectorchord… build — and postgres:16-alpine/redis:7-alpine already track the newest patch inside their major, so they are not stale. The case for bumping: EOL majors eventually stop getting security patches, and "we never bump" silently becomes "we ship EOL databases". Suggested shape: per-app sidecar pin follows upstream's own compose where upstream publishes one, else stay within the current major and revisit at that major's EOL date |
| R-43 | [P2-HIGH] No offsite restore path can restore a database — offsite restore cannot reconstitute a DB-indexed app. All three offsite buttons are file-only. „Ellenőrző visszaállítás" and „Teljes visszaállítás" restore into a scratch/verify folder and never load a dump into postgres (controller/internal/backup/offbox_restore.go:171-232, :130-164); „Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" is rsync -a --ignore-existing over mandatory userdata paths only — it never touches a DB and never stops/starts the stack (offbox_restore.go:346-421). The DB dump is faithfully carried into the snapshot and can never be replayed out of it by any offsite action; the only code that replays into a live DB is RestoreFromRecoveryUnit (backup/restore_unit.go:74), reachable solely from the local restore panel (web/handlers.go:1108). Compounding: the live recovery unit present — not overwriting guard (offbox_restore.go:400-405) skips the entire unit — compose, manifest and db-dumps — whenever a local backups/primary/<app> exists, which is true on any box whose local backup has run, so on a normal box the offsite dump is never even staged to disk. | M | SHIPPED controller v0.148.0 (2026-07-19) — live acceptance PENDING | SHIPPED: new „Teljes visszaállítás (fájlok + adatbázis)" (ReconstituteFromOffsite, /backup/offbox/reconstitute): safety dump → stop → files overwritten to the snapshot version → start → the snapshot's own dump replayed (reimportDBDumpsFrom, reading the SCRATCH unit — the live unit is still never overwritten) → health wait. Two invariants, both red-proofed: nothing is ever deleted (rsync -a, no --ignore-existing, no --delete; post-snapshot files survive as extras) and the undo exists before the act (the pre-restore- dump is verified ON DISK before anything is stopped/overwritten/replayed; if it cannot be taken the operation refuses with zero changes). Outcome-shaped flash; the old missing-only button now states what it does NOT do. Deployed to demo guest 9201, healthy; page renders, button correctly gated behind a prepared full scratch. STILL OPEN — the §9 live acceptance has NOT run (upload → push → empty the trash for real → one button → photos back in the timeline). Until it does, no capability-map flip, the customer-restore row stays MISSING and R-3 stays DRAFT. — Evidence: audits/DIAG-immich-restore-2026-07-19.md (live on demo-felhom). Viktor deleted 11 immich photos to test offsite restore; both place runs merged 0 files and flashed success. Root cause is two-layered — the UI delete only trashed the assets so nothing was missing to restore, but a valid test would also have failed: files return, the library stays empty, because immich's truth lives in postgres. The customer-visible failure mode: correct bytes on disk and an app that cannot see them. Severity case: this is the offsite tier — the one that survives losing the box — and most of the catalog is DB-indexed. Direction (needs a spec, do not improvise): either offsite gains a real DB-replay leg reusing RestoreFromRecoveryUnit behind a stop→restore→start sequence, or the offsite UI states plainly that it restores files only and routes the customer to the local recovery-unit flow for the DB. Pairs with R-44 (a replay leg is worthless while the dump is stale). Flips: the customer-restore map row stays MISSING and gains this as a note — the C6 evidence attempt found a product gap, which is the rehearsal system working |
| R-44 | [P2-HIGH] A manual offsite push ships an unrefreshed DB dump — "backed up now" is false for the DB half. offboxRunHandler → RunOffboxBackup goes straight to the restic push and never calls RunDBDumps / captureAllRecoveryUnits (controller/internal/web/offbox_handlers.go:203-227, backup/offbox.go:574-759); the recovery unit merely enumerates existing dump filenames via listFileNames, never creates them (backup/recovery_unit.go:105-106). Dumps come only from the separate local db-dump daily at 02:30 (cmd/controller/main.go:542), with the scheduled offsite at 04:15 — so a manual run at any other hour ships a dump up to ~24 h old. There is no freshness check and no RPO surface anywhere: zero RPO hits across controller/; offboxUnitTime is only a two-drive tiebreak (offbox.go:827-837); the DBValidationCache exists (backup.go:364-370) but no offsite or restore path reads it. | S–M | SHIPPED controller v0.148.0 (2026-07-19) | SHIPPED: every offsite run — manual AND nightly — now refreshes the DB/volume dumps and recovery units (offsitePreDump → runDBDumpsInternal) BEFORE the restic capture, so each snapshot is an internally coherent {DB@T, files@T} bundle and retention becomes a history of restorable points. Order is the mechanism and is red-proofed (moving the capture first yields [capture dump]): the gap can only ADD files the DB does not reference yet, never remove one it does. This also makes the nightly ordering structural rather than a coincidence of two scheduler entries at 02:30 and 04:15. Each unit manifest carries offsite_run_id + dumps_at, so a pair's coherence is verifiable at restore time instead of assumed; the periodic refresh carries a prior stamp forward and never invents one. A dump-leg failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none). Honesty surfaces, all warn-level and none a gate: an unstamped pre-v0.148 pair reports its skew in the confirm, and ValidateDump gained an exact-match accounts-table sniff for customer-empty dumps (a substring match on "user" would flag user_metadata/album_user/user_audit on every healthy single-user box — red-proofed). — Evidence: audits/DIAG-immich-restore-2026-07-19.md. Today's unit dump immich-postgres.sql (51 954 452 B, mtime 02:30 CEST) probed to asset: 0 rows, user: 0 rows, album: 0 rows — the 52 MB is entirely immich's shipped geodata_places/naturalearth_countries reference data. It predates both the admin user (created 07:56:25) and the photos (07:57). Same for the unit's immich_immich_postgres_data.tar (323 MB, also 02:30). A dump that looks substantial by size can contain zero customer content — size is not a health signal, and nothing in the product says otherwise. Latent hazard: had a full restore actually loaded that dump it would have written an empty DB over the live one, destroying the trashed rows that were the only surviving recovery path. Direction: dump-before-push on manual runs (the honest fix), or an explicit RPO line in the UI („adatbázis-állapot: ") so the operator/customer can see what they are actually shipping. Cheap interim: surface dump mtime + row-count sanity from the existing DBValidationCache on /backups/restore |
| R-45 | [P2] Unified async-job feedback. Every long operation invents its own progress surface, or none. Tonight produced three more one-off cards (v0.147.x: samba bring-up, offsite progress, restore result) on top of two existing patterns (deploy 3-step panel; storage-init/netstorage status poll). They agree on nothing: some use {ok,data} envelopes and some raw JSON, some poll 1 s / 1.5 s / 3 s, some are in-memory-only and lie after a restart, and each re-implements single-flight + snapshot + phase→Hungarian mapping. | M | idea | Origin: 2026-07-19 feedback slice 1 (controller v0.147.0). The cases to generalise from are all in-tree: web/storage_init_job.go (the best shape — acquire/release/set/snapshot), web/netstorage_job.go, web/samba_ensure_job.go, backup/opstatus.go, backup/offbox_progress.go. Shape: one job registry + one poll endpoint + one client-side renderer, phases declared per job. Two lessons tonight that any framework must encode: (1) a terminal state must be probed, not inferred — compose up -d exits 0 on a crash-loop; (2) a progress source that reports nothing is normal, not broken — restic reports 0 bytes for a whole incremental run, and a bar that sits at 0% is worse than no bar. Also fixes the restart hole: in-memory job state currently vanishes and the card silently disagrees with reality |
| R-46 | [P2] Verification copies need a customer-visible browse surface and an expiry. v0.147.0 made them visible (listed with path/size/date, individually deletable) — but the customer still cannot LOOK INSIDE a verification restore to confirm the file they wanted is really there, which is the entire point of a verification restore, and nothing ever removes them. | S–M | idea | Origin: 2026-07-19 feedback slice 4a, registered as the explicit follow-up to it. Two gaps, deliberately designed together because they are the same object: (a) the invisible-result gap — a read-only browse of backups/offsite-restore/<app> (the FileBrowser infra stack already exists and already serves scoped roots, so this may be a mount rather than new code); (b) the disk-lifecycle gap — auto-expiry after N days with the count/size surfaced before it fires, so a drive is never quietly filled by verification restores nobody remembers taking. Pairs with R-43: a browse surface is also how a customer would discover that a DB-indexed app's files came back but the app still cannot see them |
| R-47 | [P2-HIGH] The DB replay races the application's own schema repair — a restore can abort half-applied. The reconstitution starts the stack BEFORE replaying (ImportDump needs a live container), which hands the app a window to recreate schema objects the dump is about to create. Proven to the second on 2026-07-19: controller began the replay 10:58:25, immich-server logged Reindexing clip_index → Reindexed clip_index at 10:58:33, and the dump's own CREATE INDEX clip_index failed at 10:58:35 with already exists (exit 3, ON_ERROR_STOP=1). immich then reported schema drift — the indexes the aborted script never reached. | M | idea | Evidence: audits/DIAG-immich-restore-round2-2026-07-19.md (H4, live on demo-felhom). The photos survived only because pg_dump emits COPY data BEFORE CREATE INDEX, so the abort landed after the rows — that ordering is luck, not design: a collision earlier in the script aborts before the data and leaves a genuinely half-restored database, reported identically. Class defect, not offsite-only: the same start-then-replay shape is on the LOCAL path (RestoreFromRecoveryUnit → RecreateStackFromUnit → reimportDBDumpsCtx), so the local restore carries the same race. Direction (NOT decided — needs a spec): bring up the DB container alone for the replay and start the app only afterwards, or quiesce the app's schema management for the duration. Note --clean --if-exists + ON_ERROR_STOP=1 are both CORRECT and should stay — the bug is the window, not the flags. Blocks the §9 acceptance and therefore the customer-restore map row |
| R-48 | [P2-HIGH] Restore controls are separable only by layout — and the difference between them is whether the data comes back. The offsite restore row renders four buttons plus hint text into an overlapping, unreadable line, and the decisive second step („Teljes visszaállítás indítása") appears ONLY after „…előkészítése" was pressed, with no signposting that a second step exists or that the first one did nothing to live data. | M | idea | Evidence: audits/DIAG-immich-restore-round2-2026-07-19.md (finding 1) — this is not theoretical: it is the CAUSE of the round-2 incident. An operator who had read the code pressed the missing-only button instead of the full restore; the controller log shows /backup/offbox/reconstitute was never hit at all. The rule this establishes, worth stating once and applying beyond this page: two adjacent controls whose difference is "your data comes back" vs "your data cannot come back" must not be distinguishable only by layout. Direction (ruled in principle, spec rides v0.149): collapse to a single „Visszaállítás…" guided dialog — one intent, visible phases, the escrow-wizard precedent. Pairs with R-45 (the phases are exactly the async-feedback surface) and R-46 |
| R-49 | [P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup". Measured 2026-07-19: immich_ml_cache.tar 823 660 032 B (~60%) — re-downloadable ML model weights; immich_postgres_data.tar 308 251 136 B (~23%) — a raw tar of the postgres data dir that DUPLICATES the logical .sql dump captured beside it; upload/backups/ 18 MB — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 dccc13fe… tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: 72 MB of originals. | S–M | idea | Evidence: audits/DIAG-immich-restore-round2-2026-07-19.md §4 (full byte breakdown). This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. Recorded, deliberately not changed — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) immich_ml_cache — pure cache, strongest case; (b) the postgres_data volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) upload/backups/. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app |
Pre-invite checklist — what stands between here and the first remote tester
Not roadmap items in their own right; the short list the 2026-07-18 rehearsal leaves behind. Everything here is remote-doable — the N100 is packed, and none of it needs hands on the box.
| Action | Owner | Note |
|---|---|---|
| Viktor (saves) | Golden 0.146.0 baked on the drill VM and published to gitea — felhom-golden/0.146.0/golden.tar.zst, sha256 4834c703162c5437467a329144b1a523019bf5693ab9d439558be7323587e955, 612 696 588 B (584 MB archive). All pass markers green: Result=success/ExecMainStatus=0, 0 FATAL/exclusions, docker OK (overlay2), all three mounts included (rootfs + mp0 /var/lib/docker + mp1 /mnt/sys_drive), pre-delete 404, upload HTTP 201; controller 0.146.0 confirmed baked in. Integrity round-trip independent of the build host: anonymous `GET |
|
| Golden ≥ 0.147.x carries ALL FOUR infra images | — (next bake) | build-golden.sh v2.1.0 (2026-07-19) now derives the pre-pull list from the controller binary it is about to bake (--print-infra-images) instead of a hand-maintained copy that had already drifted: felhom-samba was never added to it, so every golden so far baked 3 of 4 — which is why enabling Megosztás on a fresh box pulled from the registry with zero feedback. No golden rebuild for this alone; it takes effect at the next bake. Until then a fresh box still pulls felhom-samba at enable time, which controller v0.147.0's progress card now at least explains |
| freemail.hu test-send | Viktor | The open half of R-4; the gmail half closed on 2026-07-18 under p=quarantine |
| C6 — customer performs a restore, unassisted | Viktor as customer zero | The one open script step in R-3 and still MISSING as capability evidence. Remote-doable on the reborn box — the dashboard is remote |
| R-11 rulings | Viktor | Contact channel, tester agreement, alert thresholds (the R-5 gauge thresholds are still pending a ruling) |
Absorbed / superseded notes in this folder
FOLLOWUP-nas-automount-guest-reboot-reassert.md— shipped (agent v0.84/v0.85, CAMPAIGN-3); keep for historyFOLLOWUP-golden-default-controller-tag.md— verify against current golden flow; close or promote to an itemFIX-M18-NOTES.md,FIX-M19-NOTES.md,DIAGNOSIS-f9-storage-registration-gap-2026-06-14.md— historical diagnoses; superseded by shipped fixes