d839ddcb60
The delete was correctly refused at four successive gates while the host still read ONLINE (acknowledgements -> typed confirm_id -> expect_hosts stale-preview -> "host is ONLINE"). Rather than force it, the run waited for the destroyed host to age to DOWN; delete-impact then reported deletable:true and the documented cascade ran: host deleted (escrow demoted to retained custody), tenantsync deprovisioned, PBS tenancy deprovisioned, claim reset to unclaimed, residue purged (reports=5 app_telemetry=5 notif_prefs=1 appliance_registrations=1) Verified after: 0 occurrences of "e2d" anywhere on the hosts page; demo-felhom and demo-hp ONLINE on agent 0.113.0; drill-r50 and peti-felhom unchanged; demo-hp carries only guest 9201 and VM 300. Scoping checked rather than assumed: the single purged appliance_registration was this run's own appliance (810d10c5, bound to e2d-fresh). The unrelated stale 2026-07-25 appliance (206c8838 / QWA-WJE) was NOT touched by the cascade — the operator removed it separately. - OPEN-ITEMS.md: the drill-cleanup WATCHING row is removed (done, not open). - audits/E2D-fresh-vm-2026-07-29.md §8 + REPORT-e2d.md: teardown recorded as complete, with the cascade output and the appliance-scoping note.
39 KiB
39 KiB
OPEN-ITEMS — the single source of truth for open work
Rebuilt 2026-07-27 by read-only triage. ROADMAP.md keeps the full history and reasoning; this
page keeps only what is open, and it is the file to read first. Root REPORT.md is per-session
and overwritten — nothing durable may live only there; a session that must not clobber it writes
a non-overwritten REPORT-<topic>.md sibling instead (CLAUDE.md:82-87), of which 14 now exist.
State: BLOCKED · READY · WAITING-ON-OPERATOR · WATCHING. Every row has an owner.
| ID | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|
| R-88a | SHIPPED (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — | |
| R-88b | /backup/due cannot say unknown |
SHIPPED + PROVEN-LIVE (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | age_state=unknown captured on real hardware during a deliberate ep0 outage; controller deferred, zero app stacks stopped |
— |
| E-2d | Prove E-2 on a fresh VM on the t740 — the only remaining route to four unproven items: a real felhom-host-install.sh 1.22.0 run (never done), Case B naturally (single-drive install renders the degraded banner without degrading a live box), a claimable customer so the three claim-gated items stop being gated, then add a drive (the offer) and unplug it (backup_target_absent end-to-end) |
READY (M) | — | Space checked 2026-07-29 — NOT a blocker, with one constraint: the VM disk must NOT go on local-lvm. That thin pool is over-subscribed (144 GB allocated against 54 GB, 38.8% used) on a box running a live customer guest, and a full thin pool corrupts every guest on it. local has only 23.7 GB and sits on pve-root. Use /mnt/nvme-1tb (888 GB free). Caveat found: a dir storage at a SUBDIRECTORY there will fail the agent's exactMount check and report disconnected in the host report — cosmetic, but decide the placement deliberately. Do NOT unblock drill-r50 (deliberately blocked; unblocking it means the fixture stops representing anything real). CORRECTED 2026-07-29 — the ISO is not an obstacle to the 1.22.0 proof, it is the best route to it. felhom-bootstrap.sh:96 fetches from https://felhom.eu/scripts/felhom-host-install.sh, not the hub, and that URL serves 1.22.0 (git-sync from main, ≤30 s — live fetch confirmed). A fresh ISO install therefore exercises 1.22.0 automatically — which makes the ISO leg the stronger proof, because it is the real customer path rather than a proxy for it, and it retires the "run 1.22.0 manually" step. The Phase 0 question that decides the shape is ANSWERED — it was a source read, and the answer is yes. A fresh VM with no baked customer-id lands in PAIRING mode (felhom-bootstrap.sh:537-541), not DIRECT (:312), and only DIRECT passes --customer-id / --mode / --passphrase-file. But on a 200 from /api/v1/appliance/poll the pairing loop writes the hub-delivered FELHOM_CUSTOMER_ID + FELHOM_RETRIEVAL_PASSPHRASE into the 0600 env, re-sources it and calls run_direct in the same invocation (:495-499) — so pairing reaches the identical installer invocation (:322-343 — the single $INSTALL_URL fetch at :322-330, the --customer-id/--mode/--hub-url/--passphrase-file args array at :334, and the bash "$SCRIPT_TMP" "${args[@]}" call itself at :343) and the customer it yields is the one the operator bound, i.e. claimable. So the ISO leg is the spine; a manual 1.22.0 run is not needed as a separate scenario. RUN ATTEMPTED 2026-07-29 — STOPPED AT PHASE 0, no VM created, nothing touched (audits/E2D-fresh-vm-2026-07-29.md). The blocker is R-111: a fresh box installs agent 0.96.0 + controller 0.161.0, not main's 0.113.0/0.185.1, so C3/C4/C5 test surfaces that do not exist on it — the degraded banner + GET /api/storage/backup-target landed in controller v0.185.1 (cdaeb36) with the copy in v0.185.0 (3f7cf2a); backup_target_absent in v0.184.0 (c1a63de); the offer's apply needs agent v0.113.0 (58b598b). C1 (a real rc=0 1.22.0 install) and C2 (Case B natural, configure_backup_target() felhom-host-install.sh:627, warnings :653-655) remain ACHIEVABLE TODAY — both are installer-side and host-install is served at 1.22.0. C3 unblocks cheaply by raising the hub global floor 0.156.0 → ≥0.185.0 (controller 0.185.1 IS in the registry): measured fleet impact is nil — both demo boxes already run 0.185.1, and Peti is DOWN 14 d and already below the current floor. C4/C5 need R-111 first. Phase 0 answers are all recorded in the audit, so a resumed run does not re-derive them: drive-gate cadence 30 s (intermediary.go:337, registered server.go:217) ⇒ a 60 s two-cycle budget; hot-detach available (virtio-scsi-single + default hotplug, VM 300 is the working reference); ISO present (…v1.25.0-nested-vm-generic-mkimage.iso); /mnt/nvme-1tb 888 G free and the local-lvm fence re-measured (38.77 %, unchanged). The §5.1a operator STOP is retired — HUB_PW is in ~/.config/credentials and hub auth was verified, so CC can bind. One open decision carried forward: where the VM disk's dir storage goes, since local-lvm is forbidden and felhom-backup is the live backup target — see the audit §4.1. RUN EXECUTED 2026-07-29 after R-111 was fixed — audits/E2D-fresh-vm-2026-07-29.md. Full ISO/PAIRING route on a nested VM on demo-hp; bind → running controller in 3 m 35 s; teardown clean (pvesm status after == before, local-lvm 38.77 %, guest 9201 untouched). C1 PROVEN (felhom-host-install v1.22.0, Day-0 provision SUCCESS, guest 9201 running, golden = the one baked 20 min earlier). C2 PROVEN (both DEGRADED lines verbatim, local_backup_target=local, install did not abort). C3/C4 PARTIAL — API exact and complete, but customer-invisible → R-112. C5 FAILED → R-113. C4's decline path PROVEN (registration confers no role), restart_required:true PROVEN, agent did not self-restart, E-2a wrapper created the storage at the drive's own mountpoint, healthy renders nothing. E-2d's own premise needed amending: a fresh install does NOT yield a CC-drivable claimable customer — the claim code is bcrypt-hashed and email-only, so one operator relay was required (and the claim flow is now proven end to end). This row stays OPEN only for the residue: C5 re-test after R-113, and the C3/C4 UI legs after R-112 |
CC |
| R-94 | A hand-synced version constant drifts, and the gate that would catch it is never run — hub/internal/web/configs.go:28 pins hostInstallVersion = "1.19.0" while scripts/felhom-host-install.sh:187 is SCRIPT_VERSION="1.22.0" |
READY (XS) | — | CORRECTED 2026-07-29 — the earlier framing of this row was false and is retracted. The constant selects no script: its only consumers are configs.go:487 (ScriptVersion) and render_test.go:219, and it renders as a label at customer_unified.html:494. The install command beneath that label fetches https://felhom.eu/scripts/felhom-host-install.sh (customer_unified.html:563, :1262), which the website git-syncs from main on a 30 s period (manifests/webpage.yaml) — so 1.22.0 is what every install already gets (live fetch, 2026-07-29). Every flag the generator emits is parsed by 1.22.0 (customer_unified.html~:1210–:1238 vs felhom-host-install.sh:1177–:1210): no functional gap, only a wrong number on the operator's screen. Three legs, all XS: (a) derive the label from SCRIPT_VERSION rather than hand-syncing it, or delete it; (b) scripts/hostinstall_gates.py fails today and is invoked by no Makefile, hook or CLAUDE.md — wire it next to site_gates.py or delete it, because a gate nobody runs reads as coverage it is not providing (this leg is one instance of → R-29, which is the class: gates are enforced nowhere, and the enforcement decision belongs there, not here); (c) render_test.go:219 compares the constant to itself and passes at any value — replace it with the cross-file assertion. No longer blocked on E-2d — it never gated anything. 2026-07-29: a real 1.22.0 install has now happened (audits/E2D-fresh-vm-2026-07-29.md), so even the original (retracted) precaution is discharged — nothing stands in front of this row |
CC |
| R-110 | main is the installer's publish channel — there is no staging. manifests/webpage.yaml git-syncs /scripts/ from --branch=main on a 30 s period and nginx serves that working tree directly (location /scripts/, root …/current). So pushing scripts/felhom-host-install.sh is publishing it: within thirty seconds it is what every subsequent felhom-bootstrap.sh fetch (scripts/iso/felhom-bootstrap.sh:96) and every operator-run day-0 command (customer_unified.html:563) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as root on a virgin box, the single most privileged thing Felhom ships |
WAITING-ON-OPERATOR (S) | operator ruling | Two consequences worth stating: E-2d is not a gate before exposure — 1.22.0 has been the live installer since it hit main on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" was never available to take. Open question for the operator, not a defect to fix blind: whether /scripts/ should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether main-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID: felhom-host-install.sh fetches nine files from raw/branch/main (:2072–:2206) and the hub manifest vouches a sha for exactly one (wrapper_sha256 → felhom-pbs-apply; re-checked this run, no drift). E-2a's felhom-backup-target-apply (:2116) is installed 0755 to /usr/local/sbin and root-fenced in sudoers, validated only by bash -n — a root-executed artifact taken from main with no pinned integrity, which is this row's class exactly |
CC |
| R-111 | 0.96.0, not 0.113.0.felhom-host-install.sh does not use main: it reads the hub-vouched manifest (:423-436) and fetches Gitea generic packages (agent :1945, golden :2573). Gitea's newest are agent 0.96.0 and golden 0.161.0, and the hub's manifest selects exactly those — so a fresh box lands on agent 0.96.0 + controller 0.161.0 (global floor v0.156.0 < the golden's 0.161.0, so no self-update) against main's 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by direct deploy and was never published |
SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1 | — | FIXED the same day it was found. Agent 0.113.0 built from the clean tree @ 58b598b and published (scripts/publish-agent.sh), sha 5f3247f756cb658e…, round-trip GET verified. Golden 0.185.1 baked on the nested drill VM embedding controller 0.185.1, published, sha dba00f3e845c415e… — bake clean: Result=success, overlay2, all 3 mounts included (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log drill/bake-0.185.1.log; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to virgin). Hub Day-0 manifest moved both together in one POST so it never vouched a new agent against an old golden; min_agent 0.93.0 → 0.113.0, which is what controller v0.185.0 declares (felhom-controller/CHANGELOG.md:15) — zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held. wrapper_sha256 preserved verbatim (re-checked against configs/felhom-pbs-apply — no drift). The global controller floor was deliberately NOT raised: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created. 17 unpublished releases (v0.97.0–v0.113.0) strand the entire R-82 tiered-backup arc plus F-CRIT-2 (a failed backup looking fresh — 7 days silent) and F-REBOOT (a guest rebooted mid-backup never returns): a new customer's box would install without them. Blocks E-2d's C3/C4/C5 — those test endpoints and events that do not exist in 0.96.0/0.161.0. The controller is fine (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. Mirror of R-110, not a duplicate: R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: audits/E2D-fresh-vm-2026-07-29.md |
CC |
| R-113 | The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition. planDriveGates (intermediary.go:216-262) computes presence by OR-ing d.BoundUnderParent into present[GuestPath], and the agent derives BoundUnderParent from GuestSeesMount() — "is this path a mount target in the guest's /proc/<pid>/mountinfo" (localapi/disks.go:210). The raw drive mount is a device-bound systemd unit and dies with the device; the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device. The gate reads the surviving bind as "present" ⇒ no Stop action ⇒ notifyDriveAbsent never called |
READY (M) — P1 | — | PROVEN LIVE 2026-07-29 on a fresh box. Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the agent said enrolled drive absent by UUID every 20 s, the controller logged 0 [gate] lines, and the hub received zero events — no backup_target_absent and no generic storage_disconnected. Measured with the device gone: /mnt/mentes2 NOT mounted, /mnt/felhom-drives/mentes2 still /dev/sdb[/felhom-data]. Not a virtualisation artefact — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). Mirror scenario reasoned, not observed: both the specific and generic events come from the same a.Stop branch, so the generic one is equally unreachable. Sixth instance of seam-built-but-never-wired — E-2b wired the seam, to a condition that cannot occur. Evidence: audits/E2D-fresh-vm-2026-07-29.md §5.2 |
CC |
| R-112 | E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it. GET /api/storage/backup-target returns byte-exact copy (proven live), and nothing fetches it: grep 'backup-target' across all *.html/*.js/*.css → 0 hits; no template references OfferPath/Degraded/the copy; resolveBackupTargetState + degradedMessageFor are consumed only by the JSON handler — no page handler injects the state |
READY (S) — P1 | — | The decisive contrast: templates fetch 18 distinct /api/storage/* endpoints; backup-target and backup-target/assign are the only two with zero references. The handler's own comment calls itself "the dashboard's source for the degraded banner and the offer" — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as "the offer endpoints were mounted where nothing routed to them" — it fixed the router mount and stopped one layer short of the render; its test TestBackupTargetRoutesLiveUnderTheStorageAPIMount pins dispatch, not reachability, which is exactly what CLAUDE.md's seam rule warns about. Fifth instance of seam-built-but-never-wired. Fix R-114 FIRST — wiring this alone would start showing customers the wrong message. Evidence: audits/E2D-fresh-vm-2026-07-29.md §5.1 |
CC |
| R-114 | On target-drive loss the customer is told the wrong story and offered the drive that just vanished. With the assigned target absent, the endpoint returned degraded:true, target:"felhom-backup" plus the "a rendszermentés ugyanazon a lemezen van, mint a rendszer" message — false, the target is a drive that has disappeared, not the system disk — and offer_path pointing at the missing drive as the remedy |
READY (S) | — | resolveBackupTargetState falls through to the generic degraded branch whenever no disk satisfies d.BackupTarget && d.MountPath != "", never distinguishing never configured from configured and now missing. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. Currently invisible ONLY because of R-112; fix this before wiring that. Also seen: after reattach the drive returned as /dev/sdc while the stable bind still recorded /dev/sdb, and the state read healthy. Evidence: audits/E2D-fresh-vm-2026-07-29.md §5.3 |
CC |
| R-29 | The green gates are not enforced anywhere — one was RED for 16 releases before anyone ran it. This is the class, not an instance: a gate that exists, asserts something true, is red, and is invoked by nothing reads as coverage it is not providing. controller/scripts/docker_run_volume_path_gate.py failed continuously from 2026-07-14 (v0.129.0) until R-7b's close-out ran it by hand at v0.145.0 — sixteen releases in which every REPORT said "green" |
READY (S for (a) / M for (b)) | — | This item has existed at ROADMAP.md:158 since before the register was rebuilt (2026-07-27) and was never carried across — that omission is itself part of the finding, because it is an open item about work not getting done that then went missing from the page that decides what gets done. Two separable parts, per R-29's own analysis: (a) the docker_run_volume_path_gate finding is benign and the fix is a 3-line ALLOWLIST addition with its why — not a rewrite of the flagged call — and it gets its own reviewed diff, never bundled into a feature commit; (b) the systemic half, the real item: decide where gates run (pre-push hook, build.sh step, or CI) and make a red gate block the train the way the Go green gate does. Two further orphans confirmed 2026-07-29 by repo-wide grep across all file types + sibling repos + ~/.claude settings/skills/hooks + .git/hooks (none non-sample) + Makefile/justfile/Taskfile find (only hub/Makefile, zero gate occurrences) + CI-directory find (this repo has no CI at all) — every one of the 19 hits is a docstring, a code comment or prose, and not one is an invocation: scripts/hostinstall_gates.py — RED today (hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0, exit 1), the same finding as R-94 leg (b) — and scripts/hub_confirm_gate.py. Of the four gates in scripts/, only site_gates.py is mandated anywhere (CLAUDE.md:153) and manifest_bearer_gate.py is named in runbooks/secrets.md:76. In R-29's own words, carried forward deliberately: do not mint a new ID for a new instance — the 2026-07-18 rehearsal independently re-raised this item and no second ID was minted then either |
CC |
| R-95 | restic offsite credential can delete (readonly=False, forget --prune runs from the box); SFTP cannot express append-only |
READY | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST --append-only |
CC |
| R-86 | Restore-tests are interval-scheduled, not backup-aligned | READY | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after its own newest archive | CC |
| R-87 | The restic tier is never restore-tested | READY | — | Design a controller-side test (no scratch-guest analogue transfers) | CC |
| — | Storage Box snapshots on storage-box-pool-1 — plan SET (daily 00:00, keep 7) but 0 taken yet |
WATCHING | first run tonight 00:00 | Confirm size_snapshots > 0 tomorrow; until then the mitigation is armed, not proven |
CC |
| — | PBS-storage-1 (u629193, box 611421) still status=active, 19.9 MB |
WAITING-ON-OPERATOR | operator console | Delete the box | operator |
| R-90 | ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged | BLOCKED (interim proven) | Hetzner CX33 availability — confirmed unavailable even powered OFF, so it is the Cost-Optimized "Limited availability", not the power state | Re-check CX33; escape hatch if urgent = CPX/CCX lines (no availability warning, higher cost) | operator |
| R-91 | Old 13 GB datastore copy at /srv/pbs-felhom on ep0's root disk |
WATCHING | demo-felhom's first post-migration PBS backup | Delete once it lands; fix CONTEXT.md:1018 same commit |
CC |
| — | First-ever GC on felhom-offsite (armed today 13:11 UTC, never run) |
WATCHING | schedule | Sun 2026-08-02 04:30 UTC — confirm it completes | CC |
| — | demo-felhom's next weekly PBS backup (newest is 2026-07-26) | WATCHING | schedule | ~2026-08-02; also releases R-91 | CC |
| — | demo-felhom's next restore-test (84 h cadence, last 2026-07-27 06:38 UTC) | WATCHING | schedule | ~2026-07-30 18:38 UTC | CC |
| R-97 | SHIPPED (controller v0.177.0 + hub v0.78.0/v0.79.0, 2026-07-27) | — | v0.79.0 (R-97c) replaced a FALSE operator-only comment with a real operatorOnlyEvents register |
— | |
| F-CRIT-2 | SHIPPED + PROVEN-LIVE (agent v0.106.0, 2026-07-28) | — | NewestArchiveTime now counts only plausibly-complete entries (measured 1 MiB floor; undecidable ⇒ not counted). Campaign fault 2 replayed on demo-hp: phantom rejected + logged once, tier correctly DUE and backed up, and no thrash on the inverse |
— | |
| R-99 | Server-side prune never removes a phantom snapshot. Confirmed it does NOT count them toward keep-last (dry-run kept 2 real + the phantom) so there is no retention/data-loss bug — but one accumulates per aborted upload, forever |
READY (S) | — | Decide a cleanup path. Deletion on a customer datastore is a separate ruling — detection shipped, removal deliberately not automated | CC |
| F-CRIT-1 | restartAll discarded the error AND StateStopped was whitelisted on invariant I1, which the quiesce path had made false |
SHIPPED + PROVEN-LIVE (controller v0.179.0, 2026-07-28) | — | Both causes fixed. Live on demo-hp: alarmed 9s after grace expiry, banner shows (stopped); a deliberate user stop stayed silent through 9 dead-app scans |
— |
| F-A1 | SHIPPED + PROVEN-LIVE (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → 0 operator emails, real failure → 1 | — | |
| R-100 | isStale counted from LastRun, which the controller writes unconditionally on failure |
SHIPPED + PROVEN-LIVE (controller v0.181.0 + hub v0.80.0, 2026-07-28) | — | Anchored on a new last_success. Severity corrected during Phase 0: this was NOT a silence — backup_failed does fire nightly and reaches the operator (live DB: 5 sends). The real defect is defeated defence in depth: the hub-side pull net was anchored on a field the failing controller keeps refreshing, so it could not compensate for a lost push (cf. F-HUB). Live on demo-hp: induced failure → last_run advanced 11:25:48Z, last_success held 11:24:20Z; demo-felhom healthy → anchor advanced. Legacy degrade logged once per customer, live |
— |
| R-101 | LastRun is written on failure and rendered to the customer as „Legutóbbi másolat" — including in the restore confirm dialog |
SHIPPED + PROVEN-LIVE (controller v0.182.0, 2026-07-28) | — | CrossDriveBackup.LastSuccess + SuccessTracked; the dialog names the last successful copy and discloses a failed newest attempt. Legacy rows migrate truthfully on first touch (an ok row adopts its time; an error row seeds nothing) — without the marker all 7 fleet rows would have flipped to „Még nincs sikeres másolat" on deploy. Part 2: the three record* sites rebuilt the whole struct; replaced by tier2Update (copy-and-overlay, safe by construction) — the naive fix would have had recordTier2Failure CLEAR the anchor. Live on demo-hp, rendered dialog read in both states |
— |
| C9-F1 | SHIPPED + PROVEN-LIVE (controller v0.183.0, 2026-07-28) | — | Phase 0 sized it: 43 of 53 catalog apps read NOTHING, 9 read file legs but never their DB/volumes, 1 stateless. Honesty half shipped: Tier2RestoreCoverage refuses UP FRONT without stopping the app and NAMES the working action; a run that proceeds claims only what it examined and discloses that the database and volumes are not covered. Live on demo-felhom: bookstack refused, uptime stayed „Up About an hour" (was „Up 25 seconds"); paperless A1 re-run still byte-identical, 16/16 docs clean |
— | |
| C9-F2 | StateRestarting is in no down-set |
SHIPPED + PROVEN-LIVE (controller v0.183.0, 2026-07-28) | — | StateRestarting deliberately NOT added to IsDownState (that alarms on every deploy fleet-wide); a SUSTAINED run becomes down after crashLoopAfter=5m, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace. Dashboard counter uses the same predicate so it no longer contradicts the alarm. Red-proof that matters: the naive IsDownState change fails the brief-restart test |
— |
| C9-F3 → R-104 | An interrupted offsite run leaves an exclusive restic lock the self-heal cannot reach: resticStep (offbox.go:634-648) has unlock --remove-all, but ensureOffboxRepo's probe fails first, classifyResticProbe (offbox.go:77-93) has no lock case → "other" → fail-fast. Tier dead until a human unlocks; ClassifyOffsiteFailure likewise has no lock case so the operator is told „A távoli mentés ismeretlen okból nem sikerült" for a precisely-known, self-healable condition |
READY (MEDIUM) | — | Add a lock case to both classifiers and let the probe path escalate to unlock --remove-all. Answers Phase C item 8: the repo is NOT usable after a killed run. Cleared manually this run; tier proven working again (ok, 1m35s). Reachable by any interruption — container restart, OOM, host reboot mid-backup |
CC |
| C9-F1b → R-103 | Tier-2's restore cannot cover 43 of 53 apps; the action that CAN is the keep-side unit restore (POST /backup/restore → RestoreFromRecoveryUnit, replays volume tars + DB dumps). v0.183.0 NAMES it in the refusal text but does not route to it |
READY | — | Put the working action in the card the customer already opened. Deliberately its own task: it places a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference — the reason it was not folded into v0.183.0 | CC |
| C9-F4 → R-102 | Nothing reads the Tier-2 copy's recovery-unit/ mirror. It is written by EVERY Tier-2 run (tier2.go:369, „Unit leg (always)") and read by no code path: RecoveryUnitPath resolves to backups/**primary**/ (appbackup/paths.go:46-48), and the only reader of the secondary tree is tier2_restore.go:79, which reads hdd/+userdata/ only |
READY (potentially > C9-F1) | — | Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action, leaving offsite as the only route. Verified by enumeration: 6 references to "secondary" in the tree, one writer, one reader, one wipe-warning lister |
CC |
| R-108 | Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT. Local drives are userdata-scoped (web/handlers.go:2450-2460); network paths are bound at the share root (:2432) and served with download: true. No IsNetwork() filter guards the deploy dropdown (settings.go:904-914), the per-app migrate targets (handlers.go:674-679), or handleStorageMigrateApp (storage_handlers.go:410-424 — its whole-namespace sibling DOES refuse, :397) |
READY — BLOCKS an architectural target | — | This is why D5 was not adopted in the 2026-07-28 07-backup-architecture.md rewrite: D5 moves app secrets into the local recovery unit so Tier-1/Tier-2 restore stop needing the guest, and that is safe only if no browsing surface can reach the backup tree. Every other surface was verified clean (SMB both namespace shapes, FileBrowser for drives, .fab import + download, /api/debug/*, all three ServeFile sites, storage-path add) — 07 §10.1 has the full sweep. Not a leak today (the unit's app.yaml is secret-stripped). Verified LIVE in demo-hp's generated FileBrowser compose |
CC |
| F-DIAG | SHIPPED (controller v0.182.0, 2026-07-28) | — | ClassifyOffsiteFailure → quota / orphaned / no_repo / no_units / transport / unknown, each with its own Hungarian message. Unclassifiable says so rather than being folded into a neighbour. Secrets: the old message was a raw err.Error() passthrough carrying sftp:<user>@<host>:<path>; redaction is now by the target's actual host/user/path (a first regex-only attempt leaked on a bare hostname and its own test caught it). Unit-proven; not yet exercised by a live offsite failure of each class |
— | |
| F-OPS | pct restore inherits the source guest's bind mounts — during a real DR, on a different host, under pressure |
DOCUMENTED (2026-07-28) | — | documentation/runbooks/RUNBOOK-manual-guest-restore.md: which mpN are volumes vs host binds, the mp9 source-VMID trap (it can bind another guest's bootstrap credentials), strip-and-re-add before first boot, and a positive pre-start verification. Docs only by design — the agent already neutralises binds on its own restore paths, and a second implementation would drift |
— |
| F-REBOOT | SHIPPED + PROVEN-LIVE (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; onboot is the deliberate-stop discriminator (already the stale-lock path's, and what pve-guests consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: 120 s unattended vs the incident's 587 s with a human; Scenario B proven (an onboot:0 guest left stopped) |
— | |
| F-LEAK | VM.Allocate); the 10-slot VMID band shrinks silently |
SHIPPED + PROVEN-LIVE (agent v0.110.0 + host-install v1.21.0, 2026-07-28) | — | Three attempts, two refuted live. (1) Pool adoption: PUT /pools/{pool} also needs VM.Allocate on the VM — membership cannot bootstrap its own authority. (2) Per-path /vms/990000..990009 ACLs: work, but PVE's destroy calls remove_vm_access (LXC.pm:906) which deletes every ACL at /vms/<vmid> — consumed by the op it authorises, one use per slot. (3) SHIPPED: 4th root-fenced exception, band enforced in sudoers literally (pct destroy 99000[0-9] --purge) + in code + at the caller; API destroy still tried first. Live: band PERMITTED, 9201/9100/9999/990010/1 REFUSED, and pct start 990000 REFUSED too |
— |
| F-OBS | deadapp-check leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran" |
SHIPPED + PROVEN-LIVE (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0 — it logged only at startup and when it acted, so its health could be read only from absence | — |
| E-2 | SHIPPED (hub 0.81.0, agent 0.113.0, controller 0.185.1, host-install 1.22.0 — 2026-07-29) | — | Parts 1–5 complete. Role model + offer-only assignment + Hungarian degraded banner + absent-target signal + installer Case A/B. NOT yet live-proven: the DEGRADED banner and the offer acceptance (both demo boxes are healthy, so neither state occurs naturally) and backup_target_absent end-to-end. Installer is audits/E2D-fresh-vm-2026-07-29.md §3). Of the "NOT yet live-proven" list: Case B + the degraded state are now PROVEN at the installer and API level; the OFFER ACCEPTANCE is PROVEN at the API level (decline path, restart_required:true, E-2a wrapper, healthy-renders-nothing). Still NOT proven, and now known to be BROKEN rather than merely untested: the banner/offer never reach a customer (R-112) and backup_target_absent cannot fire on device loss (R-113), with the absent-state message itself wrong (R-114) |
CC | |
| E-2a | SHIPPED + PROVEN-LIVE (agent v0.113.0 + host-install v1.22.0, 2026-07-29) | — | felhom-backup-target-apply behind a literal FELHOM_BACKUPTARGET sudoers alias; the agent's PVE role was NOT widened. Enforces F-1 (mountpoint -q) and F-2 (is_mountpoint 1 hardcoded), refuses a root-device target, has NO storage-removal path (grep-assertable), is idempotent and refuses to repoint. All five laws proven live as root on demo-hp with 0 stray storages |
— | |
| E-2b | NotifyStorageDisconnected/Reconnected defined and called NOWHERE — a drive going absent emitted no event on any channel |
SHIPPED + PROVEN-LIVE (controller v0.184.1 + agent v0.112.0 + hub v0.81.0, 2026-07-29) | — | Seam wired in ReconcileDriveGates; a target drive raises the specific backup_target_absent instead. A keying bug was caught before deploy: a.Path is the registered GUEST path, not the agent's host MountPath, so the target branch was unreachable — every absent drive, target included, fell through to the generic event (v0.184.1). Tests observe the WIRE (httptest hub), not a mock |
— |
| E-2c | POST /disks/eject would eject |
SHIPPED + PROVEN-LIVE (agent v0.112.0, 2026-07-29) | — | Eject + decommission refuse 409 on the backup-target mount, naming the storage and the remedy. Live on BOTH boxes: demo-hp /mnt/nvme-1tb and demo-felhom /mnt/hdd_1 both refused, drives unmoved. NOT a role reclassification — RoleForStorage untouched, because on both boxes that drive is ALSO the enrolled user-data drive; TestEjectStillAllowedOnANonTargetDrive pins the non-over-correction and /var/lib/vz is still refused by the PRE-EXISTING role gate, not this one |
— |
| PETI | peti-felhom deliberately NOT migrated. Its whole-guest backup still shares a device with its guest, so a drive failure there is offsite-only recovery |
ACCEPTED RISK — parked | operator's next visit (tester reinstalling from scratch) | Accepted until the reinstall; re-evaluate if that slips past ~2026-09-01. Do not migrate, do not touch | operator |
| R-109 | The DR recipe records no backup target. It lists every storage's name/type/content but never which one holds the local archives — and each demo box now carries TWO content=backup dir storages, felhom-backup (live) and local (frozen 2026-07-28 archives) |
READY (XS) | — | Add the resolved BackupTarget() to the host-half. Third recipe-completeness defect beside R-105/R-106 |
CC |
| R-89 | Retention as a per-customer commercial policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
| R-92 | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
| R-93 | drill-r50 is both a blocked customer and the only drift fixture |
READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
Why the TOP READY rows rank this way
This covers the next few only — it is deliberately not a full ordering of the table above, so that there is one ranking to maintain rather than two.
- R-95 — the largest data exposure: the tier holding the customer's documents and photos is the
one whose credential can delete. The snapshot mitigation is now armed (daily 00:00, keep 7),
but it has taken zero snapshots so far and it does not touch the root cause — the box can still
forget --pruneits own repo. - R-94 — de-ranked 2026-07-29. The prior rationale ("until it moves every hub-driven install gets the pre-R-82 default") was false: the constant selects no script and every install already fetches 1.22.0. What remains is a wrong label plus two pieces of dead safety equipment — a drift gate nobody runs and a test that compares a constant to itself. Cheap and worth doing; not high-consequence, and it blocks nothing.
- R-86 — an operator ruling already exists; it only waits on knowing what load ep0 can take.
- R-87 — real and unbuilt, but needs its own design, so it should not jump work that is specified.
- R-110 — last because it is not a READY row: the ruling is the operator's, not CC's, and there is nothing for CC to build until it lands. Ranked here rather than omitted because it is the only item on this page about the publish channel of the most privileged artifact Felhom ships, and today's exposure is zero — which makes now the cheapest moment it will ever be to decide.