Files
felhom.eu/documentation/architecture/capability-map-detail/G.md
T
admin 0d463a859e
gates / gates (push) Successful in 6m12s
answers 2026-10-12 Parts D, F, G: controller 0.308.0 delivered, the capability map restructured, the notification design; 128 -> 97
- closed: R-922 R-255 R-266 R-333 R-577 R-554 R-562 (controller 0.308.0 on demo-hp, demo-felhom, Tester 1), R-288 (map 212 KB -> 48 KB, detail per section)
- R-521 narrowed to F7; R-388 design in 08 §8.1 (two questions for the operator)
- privacy draft: why the registered address stays (S1); 02 module map: setup wizard deleted
- STATUS + REPORT-answers-2026-10-12.md; docs.md routing line names capability-map-detail/

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-12 15:54:44 +02:00

350 lines
27 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Capability map — detail, section G. Fleet & operator (hub)
> **What this file is.** The long text of every row of section G of
> [`00-capability-map.md`](../00-capability-map.md), moved here **verbatim** on 2026-10-12 (R-288,
> R-290 folded in; `09` §3 decisions 214 and 229) from the map at commit `1b45bb4d`. Each entry is one
> former table row, its cells laid out one per paragraph; the cell text itself was not edited. The
> map row carries the capability's ONE status word and its dated evidence pointer; this file keeps
> the argument, the corrections and the history. **Where this text and the map row disagree on the
> status, the map row is current** (see the map header for the words that were normalised).
>
> A new correction goes **under the entry it corrects, dated** — never into the map row.
<a id="g1"></a>
## G1 — Customer/host management: 8-tab detail, scoped auto-refresh, safe stale-host deletion, capability chips
*Map row: [G1](../00-capability-map.md#g1) · former map line 225.*
**Scenario (as written):** Customer/host management: 8-tab detail, scoped auto-refresh, safe stale-host deletion, capability chips
**Components (as written):** hub v0.47–0.53
**Status (as written):** **PROVEN-LIVE**
**Evidence (as written):** hub v0.53.0 dead-host roll-up live on the Peti cluster (proxmox1 down 23h); `CAMPAIGN-4-2026-07-13` (operator UI driven live); `DRILL-day0-take2` F-16 (offsite/freeze buttons live)
**Gap / roadmap (as written):** 8-tab render + capability chips are **render-test-validated** (hub UI is password-gated; CC cannot log in). (Cited "daily operator use" was a no-doc citation; `AUDIT-hub-gui-2026-06-30` predates these features at hub v0.25)
---
<a id="g2"></a>
## G2 — Config/state change round-trips in seconds (hub↔box immediacy; 15-min cycle stays the backbone): box→hub out-of-cycle report (Dir 1) + hub→box `GET /api/v1/wait` long-poll wake (Dir 2)
*Map row: [G2](../00-capability-map.md#g2) · former map line 226.*
**Scenario (as written):** Config/state change round-trips in **seconds** (hub↔box immediacy; 15-min cycle stays the backbone): box→hub out-of-cycle report (Dir 1) + hub→box `GET /api/v1/wait` long-poll wake (Dir 2)
**Components (as written):** controller v0.139/140, hub v0.58/0.63
**Status (as written):** **PROVEN-LIVE** (2026-07-21)
**Evidence (as written):** Transport proven live through the real DNS-only ingress: `SPIKE-immediate-sync-transport-2026-07-16` + hub v0.58.0 / controller v0.140.0 REPORTs — 240 s no-annotation hold (25 s heartbeat defeats nginx's 60 s `proxy_read_timeout`, no ingress change), 0.047 s wake-on-change, hub `rollout restart` = 1 WARN + 0-storm reconnect; Dir-1 2 s box→hub round-trip live in controller v0.139.0
**Gap / roadmap (as written):** The operator-UI **save→apply** round-trip is not fired end-to-end live (hub UI password-gated; CC can't log in) → R-23; the wake transport and the ACK→config_version→`ConfigRefresher` delivery chain are each proven, only the UI-triggered bump leg is unexercised. Agent-plane (host-domain desired-state) poke **first slice PROVEN-LIVE** (Direction-2a, agent v0.89.0 + hub v0.59.0, 2026-07-17): contentless ep0-relayed UDP poke → agent immediate desired-state cycle, per `SPIKE-immediate-sync-transport-2026-07-16` P4. Full path live-proven: a real operator manifest save fired `poke: sync-poke delivered to 10.77.0.2`; the box (0.89.0) received it and logged `poke received → triggering an immediate desired-state cycle` → `out-of-band report triggered` — **~31 ms ep0→box, sub-ms to the report cycle** (WG-confined, from 10.77.0.1 to the 10.77.0.2-bound socket); save→tick ≈ ~0.45 s (SSH-dominated), well under ≤2–3 s. R-13 first slice (listener+sender only; the rest of the mutual-repair arc stays open). **System-initiated immediacy wired (hub v0.63.0, this REPORT):** the mutation sites that only OPERATOR actions used to notify now fire the correct plane's notifier when the hub itself mints state — agent-plane pokes at `PBSDRAutoProvision` (the observed slice-C lag), `ReissuePBSDR` (also the pbsdrheal escalation), `handlePBSDRReissue`, and the two admin desired-state api writers; controller-plane bump at `reissueOnReenroll`. Unit-tested + red-proofed, not yet fired on a real system event (folds into the rehearsal bind sequence). Still PARTIAL: the R-23 operator-UI save→apply leg and the agent **fast-tick-until-first-convergence** SECONDARY (the WG-registration leg a poke can't reach pre-tunnel) remain unfired live. **Fast-tick SHIPPED (agent v0.90.0, R-28):** while any desired-state item is unapplied — incl. the pre-tunnel window a poke can't reach — the agent pulses the out-of-band trigger every 30 s and self-disarms on convergence (state-based; four cached sources; LOUD states excluded). LIVE on both demo agents (the `fast-tick armed: 30s …` startup line verified). **REAL-ONBOARDING PROOF DONE — `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (ledger 8, S5):** on a genuine first onboarding on metal, **every post-bind leg landed seconds apart with no ~15-minute stall anywhere** — bind 16:29:55 → credential delivered 16:30:21 (26 s) → agent 0.90.0 up 16:30:49 → **WG registered + tunnel applied 16:30:51 (~2 s)** → poke listener 16:30:54 → controller 16:32:28 → floor-lifted and running current 16:32:39. **Bind → running-current = 2 min 44 s.** The PBS-DR descriptor auto-provisioned on the same cadence (agent `converged state=applied` 16:45:53) — though see the DR-tier row: the descriptor converged while the credential behind it was already stale (R-39). The pre-tunnel fast-tick window is therefore proven in its real setting; the remaining PARTIAL is the R-23 operator-UI save→apply leg alone **2026-07-21 — THE LAST PARTIAL LEG IS CLOSED (R-23(a) restart leg).** The operator saved the global floor to a version the box did NOT run (0.153.0 → **v0.154.0**) and the managed self-update fired **exactly once**: `06:57:13Z` UpdateState pending (`initiated_by=auto-floor`) → `06:57:17Z` agent `controller swap requested` → `06:57:21Z` container restarted → `06:57:29Z` `new controller healthy`. **Save → healthy on the new version = 16 s.** Over a 39-minute window: swap requests **1**, agent-driven bootstrap restarts **1**, rollbacks **0**, container `RestartCount` **0**; `VerifyStartup` confirmed on the next boot and the following periodic check logged `Current version 0.154.0 is up to date` (the at/above-floor branch doing nothing, as designed). The 2026-07-20 attempt could not prove this because it targeted an already-running version. Evidence: `felhom-controller/REPORT.md` §6.
---
<a id="g3"></a>
## G3 — Customer right-sizes guest RAM from the controller (agent-enforced bounds, live cgroup apply, no reboot)
*Map row: [G3](../00-capability-map.md#g3) · former map line 227.*
**Scenario (as written):** Customer right-sizes guest RAM from the controller (agent-enforced bounds, live cgroup apply, no reboot)
**Components (as written):** agent v0.90.0 + controller v0.143.0 (R-24)
**Status (as written):** **PROVEN-LIVE** (grow **and** shrink on metal, 2026-07-18)
**Evidence (as written):** **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` (ledger 9) — the apply is now proven in both directions on a normal-sized box:** customer zero **shrank 11675 → 8192 MB at 16:50:22** and **grew 8192 → 12288 MB at 17:02:17**, each a live cgroup apply with **no reboot** (`local-api: guest-memory resized` in the agent journal, `[web] memory resized` in the controller log), and the new total rippled into the deploy page's memory math at 17:05:15 (`total=12288MB`). F5 auto-sizing had landed the guest at 11675 MB. Controller-direct (R-24's hub-desired-state framing SUPERSEDED, Viktor 2026-07-17). Agent `GET`/`POST /guest/memory` enforces every bound FRESH (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)) + verify-after-apply; PVE `SetConfig` hot-applies (**Phase-0 PROVEN** on the nested box: maxmem moves with the guest running, /proc/meminfo ripples via lxcfs, no reboot). Controller "Szerver memória (RAM)" card + code→Hungarian map, gated on `FeatureGuestMemoryResize` (MinAgent 0.90.0). LIVE-validated end-to-end through the real endpoint on the demo (above_max + below_min refusals render the Hungarian, agent English never leaks; SupportYes via the version header)
**Gap / roadmap (as written):** Row complete as of the 2026-07-18 rehearsal — the refusals were proven on the nested demo, the **applies** on the N100. Cores stay observation
---
<a id="g4"></a>
## G4 — Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules
*Map row: [G4](../00-capability-map.md#g4) · former map line 228.*
**Scenario (as written):** Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules
**Components (as written):** hub v0.45/0.53, agent
**Status (as written):** **PARTIAL**
**Evidence and gap (as written — this row's text carries a `|` outside code, so its last cells are kept together as they stood):** `runbooks/publish-train-rules.md`; demo-fleet updates proven | **Box-side floor lift PROVEN-LIVE on a fresh install** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the day-0 golden deployed controller **0.143.0** at 16:32:28 and the managed floor lifted it to **0.145.0 by 16:32:34 — a 5-second, fully unattended update inside the first minute of controller life**, `update-state.json` recording `initiated_by: auto-floor` with `controller_updated` pushed to the hub. So the *mechanism* is no longer nested-only. **Still never proven on a real REMOTE customer** — parked trains `RUNBOOK-publish-0.79/0.81/0.85-*` await Peti → R-1. **Action before first invite: rebuild the golden to 0.145.x** now that this evidence is banked, so fresh boxes don't sit two versions stale | **Fresh-box floor lift proven 2026-07-21 (demo-hp):** a brand-new box came up on golden **0.153.0** and self-updated to the fleet floor **0.156.0** during day-0, unattended — the floor mechanism works on first contact, not only on boxes that have been in the fleet a while. That is the R-23(a) single-swap behaviour observed on a machine with no history at all.
---
<a id="g5"></a>
## G5 — Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed
*Map row: [G5](../00-capability-map.md#g5) · former map line 229.*
**Scenario (as written):** Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed
**Components (as written):** agent v0.70+
**Status (as written):** **PROVEN-LIVE** (demo)
**Evidence (as written):** `SPIKE-agent-selfupdate-2026-07-05`
**Gap / roadmap (as written):** Remote-customer proof pending → R-1
---
<a id="g6"></a>
## G6 — Controller self-update: anonymous registry, no credentials in guest
*Map row: [G6](../00-capability-map.md#g6) · former map line 230.*
**Scenario (as written):** Controller self-update: anonymous registry, no credentials in guest
**Components (as written):** controller v0.112
**Status (as written):** **PROVEN-LIVE** (demo)
**Evidence (as written):** 07-10 arc
**Gap / roadmap (as written):**
---
<a id="g7"></a>
## G7 — Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue
*Map row: [G7](../00-capability-map.md#g7) · former map line 231.*
**Scenario (as written):** Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue
**Components (as written):** hub v0.37–0.39
**Status (as written):** **PROVEN-LIVE**
**Evidence (as written):** `VALIDATION-offsite-provisioning-e2e-2026-07-09`, `SPIKE-hetzner-api-provisioning-2026-07-09`
**Gap / roadmap (as written):**
---
<a id="g8"></a>
## G8 — Per-customer offsite fill + staleness + freeze lever
*Map row: [G8](../00-capability-map.md#g8) · former map line 232.*
**Scenario (as written):** Per-customer offsite fill + staleness + freeze lever
**Components (as written):** hub v0.41
**Status (as written):** **IMPLEMENTED**
**Evidence (as written):** `OffsiteChecker` (`hub/internal/monitor/offsite.go`): fill 90/95% vs soft quota, staleness >48h
**Gap / roadmap (as written):** No live-fired leg: `CAMPAIGN-offsite-overnight-2026-07-10` recorded no quota/fill/staleness emails, and the freeze write-block was **inconclusive** (only the Hetzner `readonly:true` API op succeeded). Demoted
---
<a id="g9"></a>
## G9 — Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert)
*Map row: [G9](../00-capability-map.md#g9) · former map line 233.*
**Scenario (as written):** **Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert)**
**Components (as written):** hub v0.64.0 (R-5)
**Status (as written):** **IMPLEMENTED** (data pipeline PROVEN-LIVE)
**Evidence (as written):** `monitor.OffsiteBoxChecker` — fetch-throttled Hetzner GET (1/15 min), fill (used/`storage_box_type.size`, 80/90%) + oversubscription (Σ shared+enabled ConfigJSON quotas / capacity, 2.0×), escalation-only operator alert on the customer-less `"pool-box"` scope; Offsite-tab panel + dashboard tile. **Phase-0-pinned** live shape (box 611714) + **live-computed in-cluster:** `0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x`. Tests + 4 red-proofs; hub v0.64.0 REPORT
**Gap / roadmap (as written):** Two open legs: the **UI render** is unit-verified only (hub UI password-gated → no screenshot); the **alert emails** are unit + red-proof verified but NOT fired live (real pool nominal — a live-fire emails Viktor). Thresholds pending Viktor's ruling (named config keys). READ-ONLY (GET)
---
<a id="g10"></a>
## G10 — Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge)
*Map row: [G10](../00-capability-map.md#g10) · former map line 234.*
**Scenario (as written):** **Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge)**
**Components (as written):** hub v0.65.0 + tenantsync v1.2.0 (R-5)
**Status (as written):** **IMPLEMENTED** (data pipeline PROVEN-LIVE)
**Evidence (as written):** The PBS DR datastore (`felhom-offsite` on ep0) fill — NOT a Hetzner box. **Option A:** a read-only `usage` op on the `felhom-tenantsync` ep0 forced command (twin of `fingerprint`; `df` on the datastore path — no customer_id, no admin token, NO mutation), polled by `monitor.PBSDRBoxChecker` (OffsiteBoxChecker clone; 15-min throttle; states ok/unavailable/degraded; fill 80/90% on the `"pbsdr-box"` operator scope). `/offsite` split into Restic + PBS DR tabs; two dashboard gauges. **Graceful: hub deploy ⟂ ep0 update** (ep0 ≤ v1.1.0 → gauge "n/a" until updated). **Phase-0-pinned** (`df` on ep0 PBS 4.2.3) + **live-computed in-cluster** (ep0 updated to v1.2.0 this session): `19.1% full (7.1 GB of 37.2 GB)`. 10 Go tests + a bash harness + 3 red-proofs; hub v0.65.0 REPORT
**Gap / roadmap (as written):** Open legs: **UI render** unit-verified only (hub UI password-gated); the **fill alert email** is unit + red-proof verified, NOT fired live (datastore nominal at 19%). Separate PBS threshold keys (default 80/90); no oversubscription (namespaces, not quotas). READ-ONLY
---
<a id="g11"></a>
## G11 — The hub reports LOSS OF VISIBILITY into either off-site store (not just how full it is)
*Map row: [G11](../00-capability-map.md#g11) · former map line 235.*
**Scenario (as written):** **The hub reports LOSS OF VISIBILITY into either off-site store (not just how full it is)**
**Components (as written):** hub **v0.106.0** (R-339)
**Status (as written):** **IMPLEMENTED — deliberately NOT proven-live**
**Evidence (as written):** Both box checkers count consecutive failed fetch windows and emit `pbsdr_box_unreachable` / `offsite_box_unreachable` (severity `warning`) past a default 3 windows (≈30–45 min), each with a paired `*_recovered` all-clear routed via `recoveredPairedDownTypes` — required because the recoveries are severity `info`, which `severityNotifies` drops. Scopes stay customer-less (`pbsdr-box` / `pool-box`) → operator channel only. Fill logic untouched: a degraded read still drives no band transition. Evidence: `internal/monitor/box_reachability_test.go` + the cross-package wiring test in `internal/notify/`, which asserts an actual operator mail rather than a map entry. **Filed BECAUSE of a measured gap**, not a hypothesis: the 2026-08-18 ep0 outage ran 9 h 37 m with the hub silent
**Gap / roadmap (as written):** **The gap that remains is R-340**, and it is not small: the ep0 read is the `usage` op, which rides the LOCAL API daemon — the daemon that incident explicitly cleared — so this check would have shown GREEN for that entire outage. It closes "ep0 is unreachable as a host"; it does not close what actually happened. **No live or constructed outage has exercised the emit path**, and one cannot be manufactured against ep0 (Tier 2, protected)
---
<a id="g12"></a>
## G12 — Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store
*Map row: [G12](../00-capability-map.md#g12) · former map line 236.*
**Scenario (as written):** Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store
**Components (as written):** hub v0.53, conventions
**Status (as written):** **IMPLEMENTED**
**Evidence (as written):** 07-13 closing bundle
**Gap / roadmap (as written):**
---
<a id="g13"></a>
## G13 — Operator login password changeable from UI
*Map row: [G13](../00-capability-map.md#g13) · former map line 237.*
**Scenario (as written):** Operator login password changeable from UI
**Components (as written):** hub v0.54
**Status (as written):** **IMPLEMENTED**
**Evidence (as written):** 07-13
**Gap / roadmap (as written):**
---
<a id="g14"></a>
## G14 — The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail
*Map row: [G14](../00-capability-map.md#g14) · former map line 238.*
**Scenario (as written):** **The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail**
**Components (as written):** hub **v0.135.0** (R-604, R-530)
**Status (as written):** **IMPLEMENTED — the page PROVEN-LIVE, the alarm and the mail unit-proven**
**Evidence (as written):** `audits/hub-safety-2026-10-05/partD/` (live System page: three per-customer floors, Tester 2 `0.142.0 → 0.145.0`); `osupdates/r530_agent_alarm_test.go`, `web/r604_floor_held_back_test.go`
**Gap / roadmap (as written):** the mail fires only on a GLOBAL raise below an override — not exercised live
---
<a id="g15"></a>
## G15 — The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops
*Map row: [G15](../00-capability-map.md#g15) · former map line 239.*
**Scenario (as written):** **The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops**
**Components (as written):** hub **v0.136.0** (R-173), `scripts/hub-db-backup/`, homelab-manifests rules
**Status (as written):** **PROVEN-LIVE (2026-10-05)** — first push, ep0 listing, restore test (4 hosts, 4 sealed, 0 readable), token limits, a key rebuilt from the paper copy decrypts, the saved seal key opens 4/4 console passwords in the restored copy; the alarm by `promtool` rule test + red-proofs ; **2026-10-09: the whole recovery (§3 steps 1–5) PROVEN on a throwaway k3s** — the live hub image started on the restored copy, customers 4/4 and hosts 4/4 equal live, 4/4 console passwords revealed
**Evidence (as written):** `audits/hub-db-offsite-2026-10-05/`; `audits/dooplex-survival-2026-10-09/partD-*.txt`; `runbooks/RUNBOOK-hub-db-offsite-backup.md`
**Gap / roadmap (as written):** A restored hub mails households pending notices at once — a test restore has no network (runbook §3)
---
<a id="g16"></a>
## G16 — Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail
*Map row: [G16](../00-capability-map.md#g16) · former map line 240.*
**Scenario (as written):** **Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail**
**Components (as written):** `scripts/dooplex-offsite/` (R-232), homelab-manifests rules
**Status (as written):** **PROVEN-LIVE (2026-10-09)** — first push 57 s, read back with the read-only token (27 805 files, 10 repos pass `git fsck`); Gitea restored into a throwaway and started: 10/10 repos, product `main` = live, a file byte for byte, a login; alarm by `promtool` + red-proofs; failure mail reached the inbox
**Evidence (as written):** `audits/dooplex-survival-2026-10-09/`; `runbooks/gitea-restore.md`
**Gap / roadmap (as written):** The container registry is NOT copied (images rebuild from the code); the on-box backup tree is still one writable path (R-232 c); the copy opens only with keys that must be on paper (`runbooks/break-glass-sheet.md`; the signing keys are in no copy, R-924)
---
<a id="g17"></a>
## G17 — The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`)
*Map row: [G17](../00-capability-map.md#g17) · former map line 241.*
**Scenario (as written):** **The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`)**
**Components (as written):** hub **v0.135.0** (R-135)
**Status (as written):** **PROVEN-LIVE (2026-10-05)**
**Evidence (as written):** `audits/hub-safety-2026-10-05/partA/live.txt` (Basic, no header → 403 even on an unknown path; with the header → passes; header without credentials → 401); `web/r135_csrf_test.go` (39 paths)
**Gap / roadmap (as written):**
---
<a id="g18"></a>
## G18 — Box operating-system security updates (Proxmox host, guest Debian, Docker engine)
*Map row: [G18](../00-capability-map.md#g18) · former map line 242.*
**Scenario (as written):** Box operating-system security updates (Proxmox host, guest Debian, Docker engine)
**Components (as written):** agent v0.143.0, hub v0.133.0
**Status (as written):** **PARTIAL — the GUEST and HOST Debian fast lanes and the DOCKER engine slow lane are PROVEN-LIVE (2026-10-04), with the System page, the fleet view and the alarms; the KERNEL lane is MISSING**
**Evidence (as written):** Guest: `audits/os-guest-lane-2026-10-04/`. Host + fleet + alarms: `audits/os-host-lane-2026-10-04/`. Docker + System page + crash guard: `audits/os-docker-crash-2026-10-04/` — live-restore on with the same container ids on every box; Docker 29.8.2 on both demo boxes; operator-approved Docker release; a signed undo and a signed ring-1 step; a replay refused; the crash guard restarted demo-hp twice and kept it off the third time. Design `architecture/11-os-updates.md` §5.8, §5.9, §8
**Gap / roadmap (as written):** **No automatic undo** (guest: last night's backup; host: by-hand runbook; Docker: a signed undo job); existing boxes get root-owned files by the signed config bundle since 2026-10-04 (R-840 CLOSED; a box from before agent 0.143.0 needs one by-hand bootstrap — Tester 2: R-862; `audits/r840-config-bundle-2026-10-04/`); test approvals now end with the test (R-859); the agent's sudoers is root-equivalent (R-861); the kernel lane (R-836); facts reach the hub late after a boot (R-853). **2026-10-05 (agent v0.144.1):** R8 measures the real download (R-865); a killed pass still reports (R-868, live); the debug pass runs with the hub away (R-866, live); **a power cut mid-update was proven by day on demo-hp — the box came back by itself in 37 s, but the next pass fails until `dpkg --configure -a` is run by hand (R-876, P2, open)** — `audits/night-fixes-2026-10-05/`. **2026-10-05 afternoon (agent v0.145.0): R-876 FIXED and proven live — after a second crash mid-unpack the next pass repaired dpkg by itself (`REPAIR … journal=1`) and finished** — `audits/catchup-2026-10-05/partD/`
---
<a id="g19"></a>
## G19 — An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps
*Map row: [G19](../00-capability-map.md#g19) · former map line 243.*
**Scenario (as written):** **An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps**
**Components (as written):** controller **v0.259.0** + hub **v0.119.0** + ISO 1.29.0 + the whole catalog
**Status (as written):** **PROVEN-LIVE on 0.258.0 with one blocker; THE BLOCKER IS FIXED AND PROVEN, THE WALK IS NOT REPEATED**
**Evidence (as written):** `audits/DRILL-first-hour-en-0258-2026-09-20.md` — a fresh install 2026-09-20, one intervention (R-494), stop rule not reached. Then `audits/i18n-closing-2026-09-21/live/` — the three blockers fixed and each proven on a live box or in the operator's inbox: the claim page answers English through the real cookie path; the Backup page's tier names follow the language; and the setup mail carries **four plain-ASCII English words** where the drill's carried `képző-szkítia-ásatás`, one day apart in the same inbox.
**Gap / roadmap (as written):** **R-596, R-597 and R-598 are CLOSED.** What this row still does NOT claim: **the fixed journey has not been walked end to end by a stranger on a fresh install.** Three fixes proven at the endpoint are not an hour proven by a person, and this project's own rule is that fixes are not a journey (see the recovery-journey row). **Also not walked:** the recovery code (needs ep0), backup/restore/remove/power-cut (proven 2026-09-14), and the two Backup-page *warnings* themselves — guest 9201 is healthy and a healthy box renders none, so they are covered by handler render tests, not live. **Verdict: nothing known now stands between an English-speaking tester and their box — and that is a different sentence from "the walk passed".**
---
<a id="g20"></a>
## G20 — A deletion of a customer's off-site history is NOTICED within a day
*Map row: [G20](../00-capability-map.md#g20) · former map line 244.*
**Scenario (as written):** **A deletion of a customer's off-site history is NOTICED within a day**
**Components (as written):** hub **v0.111.0** (R-431)
**Status (as written):** **IMPLEMENTED — not yet PROVEN-LIVE**
**Evidence (as written):** 09-01
**Gap / roadmap (as written):** `hub/internal/monitor/offsite.go` — third signal beside FILL and STALENESS. **On the hub deliberately:** a detector on the box is one the deletion can silence. Alarms when the reported count falls by more than HALF and by at least 5, guarded by `StatsKnown` (R-331), the declared `State` (R-204) and run success (R-100). **Threshold reasoned, not invented:** over 12 898 reports every decrease lands on ZERO and predates `stats_known`; in the 380-report `stats_known` window there are none. **ACCEPTANCE: 9 009 real points replayed → ZERO alarms** (`offsite_r431_test.go`, fixture committed). **What PROVEN-LIVE would need and this does NOT have:** a real drop observed on a live box producing a real mail — the live firing done at ship time was driven through the hub's own path with synthetic counts, which is an end-to-end delivery proof, not a proof that a genuine deletion is caught.
---