From d202b43b9523466a1cdf82dde69f4ad677fda548 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 16 Jul 2026 09:19:51 +0200 Subject: [PATCH] =?UTF-8?q?docs(map):=20capability-map=20evidence=20audit?= =?UTF-8?q?=20=E2=80=94=20verify=20every=20status=20claim=20against=20sour?= =?UTF-8?q?ce?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opened every citation in 00-capability-map.md and judged each of the 55 rows against the specific scenario (not topic adjacency). 23 confirmed (incl. 5 MISSING proven-absent by code sweep), 26 citation-corrected, 6 demoted; no upgrades. Status: 32 PROVEN-LIVE / 15 IMPLEMENTED / 3 PARTIAL / 5 MISSING (was 38 PROVEN-LIVE). Demotions PROVEN-LIVE->IMPLEMENTED: B6 (post-deploy config — T-PAGE-ALL is render-only), E7 (FileBrowser access — no browse/download proof), F1 (health email — cited P3-DELIVERY is the enlarge-block trigger), F4 (metrics — cited legs are H1/H2 harness artifacts), F6 (operator alerting — no corpus doc), G6 (offsite fill/staleness/freeze — no live-fired leg, freeze inconclusive). Key corrections: D6 cited CAMPAIGN-3 F10/F11/F12 which are the CRITICAL failures — fixes shipped agent v0.85 and were re-validated live in CAMPAIGN-4 /6A; C6 cited offbox-restic doc for a PBS-restore claim -> CAMPAIGN-2 T-P9; C1/C5/B5 cited defect-discovery docs instead of the fix-validation legs. Closed ROADMAP R-20: operator-key pinning is fully automatic in day-0 (felhom-host-install.sh step_agent_config, L2044/2146-2156/2332-2337). Header draft caveat swapped for the verified stamp. Full per-row verdict table in REPORT.md. --- REPORT.md | 155 +++++++++++++----- .../architecture/00-capability-map.md | 64 ++++---- documentation/backlog/ROADMAP.md | 2 +- 3 files changed, 150 insertions(+), 71 deletions(-) diff --git a/REPORT.md b/REPORT.md index cf3aed3..58f531e 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,47 +2,126 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## PBS-DR self-heal reconciler (hub v0.56.0) — 2026-07-15 +## Capability-map audit — verify every status claim against evidence — 2026-07-16 -**Task:** implement `SPIKE-pbsdr-selfheal-2026-07-15` (`e8f8c44`) — a hub periodic reconciler that -re-stages a consumable PBS-DR secret for boxes stuck in `waiting_secret`/`consumed_failed` after -losing their agent-side converged marker (re-install / restore / snapshot rollback onto a stable -`host_id`). Class: risky/supervised. Repo: `felhom.eu` (`hub/`) only — **no agent change** (the spike -proved the agent already self-heals once a secret is consumable). +**Task:** docs-only audit of `documentation/architecture/00-capability-map.md`. Open every citation, +confirm/correct/demote each of the 55 rows against the actual evidence doc (scenario-specific PASS, +not topic adjacency), prove the MISSING rows absent by code sweep, close ROADMAP R-20, and swap the +draft caveat in the map header. **Zero code changes; read-only against three sibling repos.** -**Baselines:** hub main `e8f8c44` → this work; hub live image v0.55.0 → **v0.56.0**. Green gate -(`go build/vet/test ./...` in `hub/`) passing across the module. +### 1. Confirmed baselines (re-confirmed at task start) -**What shipped** -- `internal/store/pbsdr.go` — `RestageHostPBSSecret(hostID) (bool, error)`: clears `consumed_at` IFF - a secret row exists (no INSERT, no value change, **no generation bump**); `restaged=false` → no row - → the caller escalates. `PBSDRHealStates()`: one query joining each host's descriptor - enable/provision flags to its latest report `pbs_dr.state` + id (mirrors `GetHostOOBStates`). -- `internal/pbsdrheal/reconciler.go` (new) — 5-min reconciler (wgsync shape). Work set = descriptor - **enabled + provisioned**. Per host, on the latest report `pbs_dr.state` held across a **debounce - (≥2 distinct reports)**: `waiting_secret` → re-stage the stored secret (no ep0 call, no gen bump), - or escalate to Re-issue if none stored; `consumed_failed` → escalate to Re-issue only. - Converged/`disabled`/`verify_failed`/unprovisioned/DR-OFF = pure no-op. Distinct audit events per - heal. `PBSDRHEAL_ONLY_HOST` scopes a supervised rollout. -- `internal/web/pbsdr.go` — `ReissuePBSDR(ctx, customerID)`: the non-HTTP core of the operator - Re-issue button, now the reconciler's escalation seam. The operator handler is unchanged (303/400 - codes stay test-pinned). -- `cmd/hub/main.go` — reconciler started unconditionally; `PBSDRHEAL_ONLY_HOST` env read. +| Repo | `main` @ commit | Role | +|------|-----------------|------| +| felhom.eu | `4b18cc5` (local == origin) | audit target — WRITE | +| felhom-controller | `e99c675` (v0.137.0) | read-only reference | +| felhom-agent | `c040c18` (v0.88.0) | read-only reference | +| app-catalog-felhom.eu | `21e8df1` | read-only reference | -**Tests (green) + red-proofs (verified):** `internal/pbsdrheal/reconciler_test.go` (Scenarios A–F + -scope + no-re-heal, real store + fake action seam, audit events asserted); `internal/store/pbsdr_test.go` -(re-stage semantics + no-generation-bump guard + `PBSDRHealStates` parsing). All six §10 red-proofs -(A, A-gen, B, C, D, E) failed exactly their guard test then reverted. +The map's evidence corpus is read at felhom.eu tip `4b18cc5`; the header caveat now records that. -**Deploy + live validation:** built `felhom-hub:0.56.0`, manifest bump + ArgoCD sync (Synced/Healthy); -first rollout scoped via `PBSDRHEAL_ONLY_HOST=demo-vm-felhom-2f4b00` (STOP discipline). Drill live-leg -(qm300 rolled to `post_day0_golden136` → stuck `waiting_secret`): the reconciler observed -`waiting_secret` across two reports and at **17:10:00 UTC re-staged** the stored secret (event -`pbsdr_selfheal_restaged`; no ep0 token, no gen bump) → agent **re-consumed 17:10:26** → **converged -`applied` 17:10:28**, hands-free. The demo host `demo-felhom-01` was never touched (out of scope). -**Fleet-wide widening (remove `PBSDRHEAL_ONLY_HOST`) is a deliberate operator follow-up**, not done -this session. +### 2. Counts -**⚠️ ARCHITECTURE IMPACT:** before this, no automatic recovery existed for a box re-installed/restored -onto its stable `host_id` — PBS-DR (hence escrow + offsite) stayed silently unconverged until a manual -operator Re-issue. This reconciler closes that gap. +**55 rows audited = 23 confirmed (18 confirmed + 5 confirmed-absent) + 26 corrected + 6 demoted.** +No row was upgraded (candidate upgrades are Observations). Status distribution after the audit: +**32 PROVEN-LIVE / 15 IMPLEMENTED / 3 PARTIAL / 5 MISSING** (was 38 PROVEN-LIVE → 6 demoted). + +### 3. Per-row verdict table + +Each verdict names the doc opened and the leg/section carrying (or failing to carry) the scenario. + +| Row | Scenario | Verdict | Evidence opened → leg / reason | +|-----|----------|---------|--------------------------------| +| A1 | Appliance day-0 zero-click auto-confirm | **CONFIRMED** | `DRILL-day0-take2` §2/§3 (auto-confirm 6m39s, zero clicks; floor self-update 0.120→0.122; claim gate closed) + `DRILL-day0-vm` §5.6 | +| A2 | BYO install (PARTIAL) | **CONFIRMED** | `DRILL-GL6` Phase 2 byo happy-path PASS; coexistence guards (GL6-F3/F6) only fixed v1.12.0, never live-revalidated — PARTIAL right | +| A3 | Customer claim | **CORRECTED** | `DRILL-day0-vm` §10/F-4 carries it; dropped mis-cited `CAMPAIGN-4` F-C (that is the escrow-claim 502, not the password claim) | +| A4 | Escrow ceremony | **CORRECTED** | agent v0.88 REPORT (claim 200→410, R absent). Qualifier fixed demo→**drill VM**; browser-wizard leg flagged not-yet-validated | +| A5 | DR tier by default (IMPLEMENTED) | **CORRECTED** | cited `SPIKE-pbs-tier-provisioning` = slice-0, shipped nothing → `DRILL-day0-take2` §2 (WG both modes, PBS-DR descriptor auto-provisioned live). Upgrade candidate | +| A6 | Uninstall (PARTIAL) | **CORRECTED** | `DRILL-GL6` Phase 1/5 (KEPT-vs-WIPED printed, data intact ×3); secret-purge fixed v1.12.0, `mnt-*.mount` residue open — PARTIAL right | +| B1 | Deploy app from catalog | **CORRECTED** | `CAMPAIGN-2` T-DEPLOY-SET + `RERUN-p1p3` PASS; dropped T-RES-MEMGUARD (guard never *fired* — ample RAM / auth-walled) | +| B2 | App lifecycle | **CORRECTED** | T-LIFECYCLE PASS + remove live in `CAMPAIGN-3`; T-REMOVE-REDEPLOY never cleanly passed (stale dryrun) — noted open edge | +| B3 | Protected infra un-stoppable | **CORRECTED** | cited `CAMPAIGN-2` T-SEC-PROTECTED was a stale-dryrun FAIL → `CAMPAIGN-nomercy` + `RERUN-p1p3` T-SEC-PROTECTED PASS | +| B4 | Catalog sync + validation choke | **CONFIRMED** | `CAMPAIGN-2` T-SYNC-IDEMPOTENT PASS; choke-point is unit-red-proofed (v0.132) — status held | +| B5 | App crash → customer notified (IMPLEMENTED) | **CORRECTED** | citation `CAMPAIGN-3 fix-3 arc` (surfaced the gap) → controller v0.120.0 (impl+red-proof); "delivery proven" softened (6B deferred/6C inconclusive) | +| B6 | Post-deploy config + restart | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | T-PAGE-ALL is a GET-only page-render smoke test; config-save+restart exercised in no campaign (CAMPAIGN-3 skipped it) | +| B7 | Backup classification (13 apps) | **CONFIRMED** | `SPIKE-backup-classification` SQ2 (13 bind apps) + 6D/6E live legs | +| C1 | Nightly DB dumps, atomic writes | **CORRECTED** | cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes + T-RST-DB auth-hollow → `CAMPAIGN-6B` P4 + `6E` B1/B2 (SIGKILL→last-good byte-unchanged) + `6D` P-FAB | +| C2 | Tier-2 copy | **CONFIRMED** | `CAMPAIGN-6E` P-TIER2 deep-4 4/4 PASS + `6C` | +| C3 | Offsite restic | **CORRECTED** | enlargement gate live (`6D` P3-DELIVERY); noted raw-data-quota/retention are `SPIKE-restic-snapshot-shape` **dry-run** only | +| C4 | Offsite restore | **CONFIRMED** | `CAMPAIGN-6D` P-IMMICH/P-PLACE (functional from offsite alone, byte-identical) | +| C5 | Manual `.fab` | **CORRECTED** | `6D` P-FAB (1.7GB circle) + `6B` P2 (chunking at real CF edge via `curl --resolve`); rendered-browser upload leg still deferred; C6B-F1 was the 6B *finding* | +| C6 | Guest-loss PBS DR | **CORRECTED** | cited `VALIDATION-newbox-restore` = offbox **restic** (wrong tier) → `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore`→healthy) + T-PBS-VERIFY (13 snapshots) | +| C7 | PBS-DR self-heal (IMPLEMENTED) | **CORRECTED** | cited spike = findings-only → hub v0.56.0 reconciler + §10 red-proofs. Facts fixed: **scoped to one host** (not fleet-wide); **already fired live** on drill qm300 | +| C8 | Crash mid-backup/migration self-heal | **CORRECTED** | cited `CAMPAIGN-2` T-RBT-* legs empty/auth-hollow → `6D` P5-REST + `6E` B1-B3; mid-*migration* crash still weakest sub-claim | +| C9 | Soft-quota block + notify | **CONFIRMED** | `6D` P3-DELIVERY (block + op+customer email live); usage-bar UI only weakly corroborated | +| C10 | Customer performs a restore (MISSING) | **CONFIRMED-ABSENT** | sweep: only self-reference at map L34 "never executed by a non-Viktor human"; all restores operator-driven | +| D1 | Drive wizard incl. legacy LVM-root | **CORRECTED** | `DISPOSITION-ia` (legacy EFI+LVM, root not offered) + `storage-lifecycle-acceptance` (enroll live) + agent fence self-test; cited CAMPAIGN-2 legs were auth-hollow; UI-wizard leg supervised | +| D2 | Data migration (all/per-app) | **CORRECTED** | `6C` 4P-5 (per-app) + `storage-lifecycle-acceptance` (2× migrate-all via UI, byte-identical); cited CAMPAIGN-2 legs auth-hollow; crash-safe design-level only | +| D3 | NAS client verify-before-commit | **CONFIRMED** | `SPIKE-nas-verify` Q2/Q4 + `SPIKE-nas-dsm` (real DSM 7.2 end-to-end) + `CAMPAIGN-3` | +| D4 | USB enroll/unplug/recommission | **CORRECTED** | `storage-lifecycle-acceptance` E4 (yank→auto-rebind) + E10 (re-enroll) + `4/6A` (3-USB reshuffle); cited RUNBOOK could not complete enroll | +| D5 | Decommission / eject | **CORRECTED** | `storage-lifecycle-acceptance` E9 (decommission-anyway, reboot-safe) + E12; cited CAMPAIGN-2 legs auth-hollow, SPIKE was report-only | +| D6 | Boot ordering survives reboot | **CORRECTED** | **cited `CAMPAIGN-3` F10/F11/F12 are the CRITICAL/HIGH failures** → fixes shipped agent v0.85 + re-validated live `CAMPAIGN-4` (0 ordering cycles) + `6A` 1D | +| E1 | Remote CF Tunnel | **CONFIRMED** | `CAMPAIGN-2` T-FLT-CF PASS (not in the H1-hollow set; recovered 20s) | +| E2 | LAN access when net down (IMPLEMENTED) | **CONFIRMED** | empty citation consistent with IMPLEMENTED; never drilled as customer experience → R-19 | +| E3 | Phone photo backup | **CONFIRMED** | `6D` Accept #1 + SQ3 (immich functional from offsite, byte-identical) | +| E4 | Documents/OCR | **CORRECTED** | tightened vague "6C/6E legs" → `6C` 4P-1 (paperless consume+OCR PASS) | +| E5 | SMB **server** (MISSING) | **CONFIRMED-ABSENT** | sweep `smbd\|samba\|wsdd\|net usershare\|smb.conf`: 10 hits, all client/offsite/docs — no server. Hetzner `SambaEnabled` = a box we write to, not a share we expose | +| E6 | DLNA to TV (MISSING) | **CONFIRMED-ABSENT** | sweep `minidlna\|dlna\|ssdp\|upnp`: 3 hits, all ROADMAP/map docs; zero implementation | +| E7 | File access via browser | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | T-PAGE-ALL renders only controller dashboard pages, not FileBrowser; no browse/download proof anywhere | +| E8 | Forgot-password reset code | **CORRECTED** | tightened "F-15 arc" → `DRILL-day0-take2` F-15 (live re-run, hash applied 1s, code accepted first try) | +| E9 | Multi-user accounts (MISSING) | **CONFIRMED-ABSENT** | sweep of controller `auth.go`: single `effectivePasswordHash()`, no user table / role / permission model | +| E10 | WireGuard OOB (IMPLEMENTED) | **CONFIRMED** | `SPIKE-oob-wg-operator-peer` GO + `SPIKE-felhom-sshd`; unproven on true 100.64/10 — IMPLEMENTED right | +| E11 | Break-glass recovery (IMPLEMENTED) | **CONFIRMED** | `runbooks/break-glass.md`; live incident reproduced in SPIKE-felhom-sshd; no end-to-end recovery drill — IMPLEMENTED right | +| F1 | Health-degradation email | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | cited `6D` P3-DELIVERY is the **enlarge-block** trigger, not health-degradation; `NotifyHealthChange` never fired an email live in any doc | +| F2 | Event catalog | **CORRECTED** | vague "per-arc" → 3 live-delivered families: enlarge-block (`6D`), claim code (`DRILL-day0-vm` F-4), reset code (`take2` F-15); `app_start_failed`/dead-app unit-only | +| F3 | Empty-email wipe guard (IMPLEMENTED) | **CONFIRMED** | controller v0.137.0, red-proofed 07-15 | +| F4 | System/container metrics | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | cited T-RES-CPU/T-SOAK-LOOP are H1/H2 harness artifacts (auth-302); SQLite/Chart.js/30-day downsampling validated nowhere | +| F5 | Debug rings + log-bundle pulls | **CORRECTED** | rings live-exercised `CAMPAIGN-3` fix-6; noted log-bundle-pull TTL/custody half is changelog-only | +| F6 | Operator alerting (Healthchecks) | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | no corpus doc; healthchecks-config app not deployed; per enum "no citation → not PROVEN-LIVE". Re-upgrade candidate if operator cites a live alert | +| G1 | Customer/host management UI | **CORRECTED** | cited "daily operator use" no-doc + `AUDIT-hub-gui` predates features (v0.25) → hub v0.53 (Peti-cluster live) + `CAMPAIGN-4` + `take2` F-16; 8-tab render = render-test-validated (UI password-gated) | +| G2 | Publish train (PARTIAL) | **CONFIRMED** | `runbooks/publish-train-rules.md` (floor-LAST, MinAgent gate); never proven on a real remote customer — PARTIAL right | +| G3 | Agent self-update | **CORRECTED** | cited `SPIKE-agent-selfupdate` is dummy-only, no operator-signed leg → `DRILL-GL6` Phase 3 (armed-key round-trip with the REAL key) | +| G4 | Controller self-update | **CORRECTED** | cited "07-10 arc" CHANGELOG → `DRILL-day0-take2` §2 (credential-empty box self-updated 0.120→0.122 live) | +| G5 | Offsite provisioning | **CONFIRMED** | `VALIDATION-offsite-provisioning-e2e` PASS (real box 611714, end-to-end) + `SPIKE-hetzner-api` + re-issue in `CAMPAIGN-offsite-overnight` | +| G6 | Per-customer offsite fill/staleness/freeze | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | `OffsiteChecker` in code but no live-fired leg; `CAMPAIGN-offsite-overnight` = no fill/staleness emails, freeze write-block inconclusive | +| G7 | Box-level Storage Box aggregate (MISSING) | **CONFIRMED-ABSENT** | hub sweep `storagebox\|aggregate\|oversubscription\|sum.*quota`: only per-customer OffsiteChecker + app-telemetry; no box-level roll-up | +| G8 | Secrets hygiene (IMPLEMENTED) | **CONFIRMED** | hub v0.53.0 closing bundle (bearer → k8s Secret) | +| G9 | Operator login password from UI (IMPLEMENTED) | **CONFIRMED** | hub v0.54.0 (`handleChangePassword`, `effectivePasswordHash()`) | + +### 4. R-20 verdict — **CLOSED** + +Read from current `scripts/felhom-host-install.sh` source (not the changelog). Operator-key pinning +is **fully automatic in the day-0 flow, no manual step**: keys resolve at L1181–1219 (script +constants `OPERATOR_KEY_OPERATIONAL_*`/`_RECOVERY_*`, already populated, optional +`--operator-pubkey-file` override); pinned by `step_agent_config()` — "STEP 6/8 agent config + +service" (L2044; embedded python builds `authz.signers` L2146–2156, and a re-install with no keys +preserves an existing box's signers); verified at the step-8 check L2332–2337 ("authz signers: N +(operator-signed self-update armed)"). No interactive prompt or post-install hand-edit gates it → +ROADMAP R-20 set to **closed**. One loose end (doc-drift, not a reopen): the L193–197 comment still +says the constants are "EMPTY by default" though L198–201 are populated — folded into R-16 hygiene. + +### 5. Commits pushed to `main` + +- (recorded on push) — `docs(map): capability-map evidence audit` (00-capability-map.md header + 26 corrected / 6 demoted rows; ROADMAP R-20 closed; this REPORT). + +### 6. Observations (out of scope — NOT acted on) + +- **Candidate upgrades (audit cannot upgrade):** **A5** DR-tier-by-default is shown working *live and + hands-free* in `DRILL-day0-take2` §2 — arguably PROVEN-LIVE(drill), left IMPLEMENTED per the + no-upgrade rule. **C7** self-heal has code + red-proofs + a hands-free live convergence on qm300 — + stronger than a bare IMPLEMENTED, but scoped-to-one-host with no real-customer firing. +- **F6 demotion is a judgment call:** operator Healthchecks→Resend alerting may genuinely run in + production; it simply has no corpus doc, and the strict enum forbids PROVEN-LIVE without a citation. + Re-upgrade the moment a live alert is cited. +- **Recurring "cite-the-finding" inversion:** several rows cited the doc where a defect was *discovered* + as if it were the proof — D6 (F10/F11/F12 failures), C1 (F7 non-atomic finding), C6 + (VALIDATION-newbox = wrong tier), C5/B5 (6B/CAMPAIGN-3 findings). The fixes live in later legs. Worth + a standing authoring rule: cite the fix-validation leg, never the finding ID. +- **`storage-lifecycle-acceptance-2026-06-15` is an uncited, high-value corpus doc** — it live-proves + D2/D4/D5 (and local-drive boot survival). Now cited; consider it first for storage rows. +- **CAMPAIGN-2 is fragile as a citation source for restore/reboot/storage rows:** harness bug H1 + (auth-session expiry → CSRF-403) hollowed every `T-RST-*`, `T-STG-*`, `T-RBT-*`; several cited + evidence dirs are literally empty or a 302. Re-anchor such rows on 6D P5-REST / 6E Phase B / + storage-lifecycle-acceptance, which actually exercised these live. +- **MEMORY.md drift:** the note that `CAMPAIGN-6D` "DEFERRED P3-BROWSER/P3-DELIVERY" is stale — the + committed 6D marks both **PASS**. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 05195c8..88f6813 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -20,8 +20,8 @@ > update the row in the same session — with the new evidence citation. A PROVEN-LIVE claim is > subject to the cardinal rule like any other claim. > -> *Drafted 2026-07-16 by project-Claude from live CHANGELOGs (controller v0.137.0, agent v0.88.0, -> hub v0.56.0) + audit index. Verify statuses once against source before first commit.* +> *Verified 2026-07-16 against evidence corpus @ felhom.eu tip `4b18cc5` by CC (capability-map audit); +> see REPORT.md for the per-row verdict table.* --- @@ -31,35 +31,35 @@ |---|---|---|---|---| | Appliance day-0 install: golden image → first boot → auto-confirm (zero clicks) → claimable box | installer, agent, hub, golden | **PROVEN-LIVE** (nested VM) | `DRILL-day0-vm-2026-07-12`, `DRILL-day0-take2-2026-07-12` | First firing on real customer hardware pending → R-1 | | BYO install: `--mode byo`, mandatory caps, host-mutation disclosure, coexistence guards | installer v1.15+, agent | **PARTIAL** | `DRILL-GL6-2026-07-08` (demo box); GL-8 coexistence fixes | Peti clean-slate reinstall on proxmox2 is the first real BYO run of the current path → R-1 | -| Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` (F-4 arc), `CAMPAIGN-4-2026-07-13` (F-C) | Never executed by a non-Viktor human → R-3 | -| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88 | **PROVEN-LIVE** (demo, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim + 410); `SPIKE-controller-escrow-2026-07-13` | First supervised ceremony with a real customer pending → R-1 | -| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `SPIKE-pbs-tier-provisioning-2026-07-10` | Live only on demo/drill fleet | -| Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | GL-4 (uninstaller gaps closed) | Cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 | +| Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) | +| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | Endpoints driven on the drill VM; customer-facing **browser wizard** leg not yet live-validated. First supervised ceremony with a real customer pending → R-1 | +| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected. Candidate upgrade to PROVEN-LIVE — see REPORT) | +| Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 | ## B. Apps & catalog | Scenario | Components | Status | Evidence | Gap / roadmap | |---|---|---|---|---| -| Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | `CAMPAIGN-2` T-DEPLOY-SET (7 apps), T-RES-MEMGUARD | | -| App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | `CAMPAIGN-2` T-LIFECYCLE, T-REMOVE-REDEPLOY | | -| Protected infra stacks can't be stopped/removed from UI | controller | **PROVEN-LIVE** | `CAMPAIGN-2` T-SEC-PROTECTED | | +| Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | `CAMPAIGN-2` T-DEPLOY-SET (7 apps, env config, health-aware); `RERUN-p1p3` (×4 PASS) | Memory-guard FIRING is not live-shown (T-RES-MEMGUARD never fired: ample RAM / auth-walled) — implemented + unit-level only | +| App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | `CAMPAIGN-2` T-LIFECYCLE (stop/start/restart/update/logs); remove live in `CAMPAIGN-3` | Redeploy-after-remove edge remains open (T-REMOVE-REDEPLOY never cleanly passed — stale dryrun journal); non-pilot-critical | +| Protected infra stacks can't be stopped/removed from UI | controller | **PROVEN-LIVE** | `CAMPAIGN-nomercy` + `RERUN-p1p3` T-SEC-PROTECTED (refuse stop/remove, stay Up) | (Cited `CAMPAIGN-2` T-SEC-PROTECTED was a stale-dryrun FAIL — corrected to the runs with a real server-side refusal) | | Catalog sync (git, 15 min) + orphan lifecycle + validation choke point (bad `backup:` block degrades to legacy, loudly) | controller v0.132, catalog | **PROVEN-LIVE** | `CAMPAIGN-2` T-SYNC-IDEMPOTENT; v0.132 LoadMetadata red-proofs | | -| App crashes → customer notified (one event per transition, no flapping spam) | controller v0.120, hub v0.48 | **IMPLEMENTED** | CAMPAIGN-3 fix-3 arc | Delivery proven; long-run anti-spam behavior only unit-proven | -| Post-deploy optional config (API keys etc.) with restart | controller, catalog `.felhom.yml` | **PROVEN-LIVE** | long-standing; page sweep in `CAMPAIGN-2` T-PAGE-ALL | | +| App crashes → customer notified (one event per transition, no flapping spam) | controller v0.120, hub v0.48 | **IMPLEMENTED** | controller v0.120.0 (dead-app alerting, `app_start_failed`, one-event-per-transition red-proofs); `CAMPAIGN-3` F11 surfaced the gap | End-to-end crash→customer-email delivery never live-confirmed (6B deferred / 6C inconclusive: clean stop ≠ crash); anti-spam unit-proven | +| Post-deploy optional config (API keys etc.) with restart | controller, catalog `.felhom.yml` | **IMPLEMENTED** | feature long-standing; config page renders (`CAMPAIGN-2` T-PAGE-ALL is GET-only) | The config-save+restart flow is exercised in no campaign (CAMPAIGN-3 explicitly skipped interactive app config). Demoted: T-PAGE-ALL is a page-render smoke test, not this flow | | Backup classification: 13 bind-bearing apps carry mandatory/optional/excluded classes | catalog, controller v0.132–133 | **PROVEN-LIVE** | `SPIKE-backup-classification-2026-07-14`, `CAMPAIGN-6D/6E` | Remaining ~39 apps are legacy-class by design (unit-only offsite) | ## C. Protection & recovery (the product promise) | Scenario | Components | Status | Evidence | Gap / roadmap | |---|---|---|---|---| -| Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL, T-RST-DB; CAMPAIGN-3 F7 fix | T-6E-1 dir-fsync asymmetry (LOW) → R-10 | +| Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 | | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | | -| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15`, `SPIKE-restic-snapshot-shape-2026-07-14`, `VALIDATION-offbox-storagebox-2026-07-09` | | +| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired | | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | | -| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` Accept #1 (≥1 GiB full circle); `CAMPAIGN-6B` (C6B-F1 hollow-export fix) | | -| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `VALIDATION-newbox-restore-2026-07-10`, `DRILL-GL6-2026-07-08` | | -| PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | `SPIKE-pbsdr-selfheal-2026-07-15` (root cause proven) | Reconciler fleet-wide since 07-15; first real-world firing pending | -| Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-2` T-FLT-KILL9, T-RBT-MIDBACKUP, T-RBT-MIDMIGRATION | | +| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D | +| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill | +| PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending | +| Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) | | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | | | **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3 | @@ -67,12 +67,12 @@ | Scenario | Components | Status | Evidence | Gap / roadmap | |---|---|---|---|---| -| Drive wizard: scan/format/mount/enroll, incl. legacy-boot LVM-root hosts | controller, agent v0.87 | **PROVEN-LIVE** | `CAMPAIGN-2` T-STG-ENROLL/SEC-FORMAT; `DISPOSITION-ia-finding2-systemdisks-2026-07-13` | | -| Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | `CAMPAIGN-2` T-STG-MIGRATE-*, `CAMPAIGN-6C` (scope=app round-trip) | | +| Drive wizard: scan/format/mount/enroll, incl. legacy-boot LVM-root hosts | controller, agent v0.87 | **PROVEN-LIVE** | `DISPOSITION-ia-finding2-systemdisks-2026-07-13` (legacy EFI+LVM host, root not offered, byte-identical); enroll/format live in `storage-lifecycle-acceptance-2026-06-15` (E10 re-enroll, data intact); agent fence self-test refuses `/dev/sda` | (Cited `CAMPAIGN-2` T-STG-ENROLL/SEC-FORMAT were auth-hollow CSRF-403.) Fresh-USB **wizard enroll+format through the customer UI** remains a supervised leg | +| Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | `CAMPAIGN-6C` 4P-5 (scope=app round-trip, byte-identical); `storage-lifecycle-acceptance-2026-06-15` (two migrate-all runs via dashboard UI, sha256 byte-identical) | (Cited `CAMPAIGN-2` T-STG-MIGRATE-* were auth-hollow.) "crash-safe" is design-level (copy→verify→remove) — no clean live crash-during-migration PASS | | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | `SPIKE-nas-verify-2026-07-11`, `SPIKE-nas-dsm-2026-07-11`, `CAMPAIGN-3-2026-07-11` (boot/reassert fixes) | | -| USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | `RUNBOOK-usb-enrollment-3bfix-2026-07-01`, `CAMPAIGN-2` T-STG-UNPLUG/RECOMMISSION | | -| Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | `CAMPAIGN-2` T-STG-DECOM-*; `SPIKE-decommission-migration-2026-06-14` | | -| Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | `CAMPAIGN-3-2026-07-11` (F10/F11/F12) | | +| USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E4 (yanked-while-running → agent auto-rebind) + E10 (re-enroll, data intact); `CAMPAIGN-4`/`6A` (3 USB re-establish across device-letter reshuffle) | (Cited `RUNBOOK-usb` could NOT complete a wizard enrollment; `CAMPAIGN-2` legs were auth-hollow.) Fresh-USB **wizard enrollment** specifically still unproven | +| Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E9 (decommission-anyway → bind detached, parent mp untouched, reboot-safe) + E12 (eject drive holding all apps) | (Cited `CAMPAIGN-2` T-STG-DECOM-* were auth-hollow; `SPIKE-decommission` was report-only, button still vestigial.) | +| Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | `CAMPAIGN-4-2026-07-13` (F12 fix HOLDS: demo-host reboot + 5-boot storm, 0 ordering cycles, caps 63/63, WG re-handshake) + `CAMPAIGN-6A-2026-07-14` 1D (re-arm reboot-survival across 9 guest + 1 host reboots) | (`CAMPAIGN-3` F10/F11/F12 were the CRITICAL/HIGH *failures*; fixes shipped in agent v0.85 and were re-validated live in 4/6A — cite the validation, not the finding.) Residual: `skip-active` on `pct reboot` carried by the heal path; a NAS outage spanning a guest reboot can strand the share until agent restart (6A) | ## E. Access, networking & household use @@ -81,11 +81,11 @@ | Remote access via Cloudflare Tunnel + Traefik (per-app subdomains) | cloudflared, traefik | **PROVEN-LIVE** | `CAMPAIGN-2` T-FLT-CF | Per-customer zone-scoped CF tokens (blast-radius ruling) | | LAN access when internet is down (lan_resolver) | agent | **IMPLEMENTED** | — | Never drilled as a customer experience ("net down — can I reach my photos?") → R-19 | | Phone photo backup | immich (classified) | **PROVEN-LIVE** | 6D end-to-end restore proof | | -| Documents/OCR | paperless-ngx (classified) | **PROVEN-LIVE** | first deployed app; 6C/6E legs | Consume-folder ingestion awkward without SMB → R-7 | +| Documents/OCR | paperless-ngx (classified) | **PROVEN-LIVE** | `CAMPAIGN-6C` 4P-1 (deploy paperless-ngx, ingest 3 docs via consume flow, OCR + PDF/A ~90s) + 4P-2/3/5 | Consume-folder ingestion awkward without SMB → R-7 | | Files from Windows Explorer / Mac Finder (SMB **server**) | — | **MISSING** | — | → R-6 (spike), R-7 (feature) | | Media to TV via DLNA | — | **MISSING** | — | Jellyfin app exists; DLNA/SSDP unvalidated → R-6, R-8 | -| File access via browser | FileBrowser (infra app, auto-mount sync) | **PROVEN-LIVE** | long-standing; `CAMPAIGN-2` T-PAGE-ALL | | -| Forgot dashboard password → instant reset code | controller v0.123, hub | **PROVEN-LIVE** | F-15 arc, drill validation | | +| File access via browser | FileBrowser (infra app, auto-mount sync) | **IMPLEMENTED** | FileBrowser runs healthy + userdata-bound (`storage-lifecycle-acceptance-2026-06-15`, `CAMPAIGN-3`) | Actual browse/download **through FileBrowser** is exercised in no doc. (Cited `CAMPAIGN-2` T-PAGE-ALL renders only the controller dashboard pages, not FileBrowser.) Demoted | +| Forgot dashboard password → instant reset code | controller v0.123, hub | **PROVEN-LIVE** | `DRILL-day0-take2-2026-07-12` F-15 (live re-run of the exact failure path: hash applied 1s after request, code accepted first try) | | | Multiple household users / per-person accounts | — | **MISSING** | — | Single dashboard password; acceptable for alpha → R-15 | | WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | `SPIKE-oob-wg-operator-peer-2026-07-05`, `SPIKE-felhom-sshd-2026-07-05` | Mutual-repair desired-state arc not built → R-13 | | Break-glass management-plane recovery | agent v0.71, hub v0.34 | **IMPLEMENTED** | `runbooks/break-glass.md` | | @@ -94,23 +94,23 @@ | Scenario | Components | Status | Evidence | Gap / roadmap | |---|---|---|---|---| -| Health-degradation email (edge-triggered, cooldowns, Hungarian) via hub → Resend | controller, hub | **PROVEN-LIVE** | 6D P3-DELIVERY | Deliverability to HU freemail providers untested → R-4 | -| Event catalog: app_start_failed, dead-app, offbox_enlarge_blocked, claim/reset codes, critical severity | controller, hub v0.31/48/50/55 | **PROVEN-LIVE** | per-arc validations | | +| Health-degradation email (edge-triggered, cooldowns, Hungarian) via hub → Resend | controller, hub | **IMPLEMENTED** | delivery pipeline live-proven for the **enlarge-block** trigger (`CAMPAIGN-6D` P3-DELIVERY, op+customer "Kedves Ügyfél!"); `NotifyHealthChange` ok→warn/fail edge-trigger implemented | The **health-degradation** trigger specifically has never fired an email live in any doc. Demoted (pipeline proven for a different event). Deliverability to HU freemail → R-4 | +| Event catalog: app_start_failed, dead-app, offbox_enlarge_blocked, claim/reset codes, critical severity | controller, hub v0.31/48/50/55 | **PROVEN-LIVE** | live-delivered: `CAMPAIGN-6D` P3-DELIVERY (enlarge-block, op+customer); `DRILL-day0-vm` F-4 (claim code); `DRILL-day0-take2` F-15 (reset code) | `app_start_failed`/`dead-app` delivery is unit-only (6C inconclusive) — the pipeline + 3 event families are live, those two are not | | Prefs safety: empty-email wipe guard | controller v0.137 | **IMPLEMENTED** | red-proofed 07-15 | Born from a live incident; guard itself unit-proven | -| System + container metrics (SQLite, Chart.js, 30-day downsampling) | controller | **PROVEN-LIVE** | `CAMPAIGN-2` T-RES-CPU/T-SOAK-LOOP | | -| Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | observability arc 07-11 | | -| Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **PROVEN-LIVE** | in production since 02-04 | | +| System + container metrics (SQLite, Chart.js, 30-day downsampling) | controller | **IMPLEMENTED** | metrics collection + `/monitoring` render present (page 200) | The cited `CAMPAIGN-2` T-RES-CPU/T-SOAK-LOOP are H1/H2 harness artifacts (auth-302), not metrics tests; SQLite/Chart.js/30-day downsampling validated in no campaign. Demoted | +| Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | debug rings live-exercised `CAMPAIGN-3` fix-6 (1000-cap ring, ~55min horizon under load) | The **log-bundle-pull TTL/custody** half is changelog-only (no dedicated observability audit doc); ring persistence across restart is a known gap | +| Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | operator infra, stated in production since 02-04; no corpus validation doc | Per the status enum, no citation → not PROVEN-LIVE. Demoted pending an operator-cited live alert (candidate re-upgrade — see REPORT) | ## G. Fleet & operator (hub) | Scenario | Components | Status | Evidence | Gap / roadmap | |---|---|---|---|---| -| Customer/host management: 8-tab detail, scoped auto-refresh, safe stale-host deletion, capability chips | hub v0.47–0.53 | **PROVEN-LIVE** | in daily operator use | | +| Customer/host management: 8-tab detail, scoped auto-refresh, safe stale-host deletion, capability chips | hub v0.47–0.53 | **PROVEN-LIVE** | hub v0.53.0 dead-host roll-up live on the Peti cluster (proxmox1 down 23h); `CAMPAIGN-4-2026-07-13` (operator UI driven live); `DRILL-day0-take2` F-16 (offsite/freeze buttons live) | 8-tab render + capability chips are **render-test-validated** (hub UI is password-gated; CC cannot log in). (Cited "daily operator use" was a no-doc citation; `AUDIT-hub-gui-2026-06-30` predates these features at hub v0.25) | | Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | `runbooks/publish-train-rules.md`; demo-fleet updates proven | **Never proven on a real remote customer** — parked trains `RUNBOOK-publish-0.79/0.81/0.85-*` await Peti → R-1 | | Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed | agent v0.70+ | **PROVEN-LIVE** (demo) | `SPIKE-agent-selfupdate-2026-07-05` | Remote-customer proof pending → R-1 | | Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** (demo) | 07-10 arc | | | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | `VALIDATION-offsite-provisioning-e2e-2026-07-09`, `SPIKE-hetzner-api-provisioning-2026-07-09` | | -| Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **PROVEN-LIVE** | OffsiteChecker in production | | +| Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | `OffsiteChecker` (`hub/internal/monitor/offsite.go`): fill 90/95% vs soft quota, staleness >48h | No live-fired leg: `CAMPAIGN-offsite-overnight-2026-07-10` recorded no quota/fill/staleness emails, and the freeze write-block was **inconclusive** (only the Hetzner `readonly:true` API op succeeded). Demoted | | **Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert)** | — | **MISSING** | — | → R-5 | | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | 07-13 closing bundle | | | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | 07-13 | | \ No newline at end of file diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 8503534..21a83ee 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -39,7 +39,7 @@ | R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open | Viktor's own action items from 6D/6E | | R-17 | Old-box archive (u629193-sub1) retirement decision — 9/9 byte-identical restores verified | XS | awaiting-decision | Viktor ruling | | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE | -| R-20 | Verify operator-key pinning is fully in the day-0 install flow (GL-4 folded plumbing in; confirm no manual step remains) | XS | verify | If a manual step remains, promote to a task | +| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) | ## P3 — post-alpha