diff --git a/CONTEXT.md b/CONTEXT.md index b91d5ef..0deabfc 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -13,8 +13,11 @@ enabled`); ep0 leg live-validated (`poke-fired`; non-WG + `rm -rf /` refused). Source note: the MinAgent-floor save does not bump per-host generation (self-update rides signed-ops) → the fleet poke accelerates the next report, not a delta. Pairs with felhom-agent v0.89.0 (the listener). - **OPEN:** the operator save→poke→tick round-trip + Scenario-3 latency await the demo agent reaching - 0.89.0 via the floor train (operator vouch). See REPORT.md. + **PROVEN-LIVE 2026-07-17:** the operator's real manifest save logged `poke: sync-poke delivered to + 10.77.0.2` (+ floor HELD for the below-MinAgent box); the demo (now 0.89.0) received a poke in + ~31 ms and fired the immediate desired-state cycle (save→tick ≈ ~0.45 s). All three live legs + (Scenario 4 A/B train, Scenario 1 R-22 self-heal, Scenario 3 latency) done; capability-map poke + + F4 rows upgraded to PROVEN-LIVE. See REPORT.md. - **2026-07-16 — DIRECTION-2 IMMEDIATE-SYNC SHIPPED: hub v0.58.0.** The hub→box "sync now" wait channel (option b of `documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`). New diff --git a/REPORT.md b/REPORT.md index f7ea3fb..d5fb6f3 100644 --- a/REPORT.md +++ b/REPORT.md @@ -49,7 +49,13 @@ Design contract: `documentation/audits/SPIKE-immediate-sync-transport-2026-07-16 `8.8.8.8` AND `rm -rf /` as the SSH command → `refused non-WG target` (the forced command NEVER executes client input, only reads it as an IP candidate). This proves the full hub→ep0→box datagram path and the credential-theft blast radius (≤ "make a registered box tick"). -- **Not yet fired end-to-end from an operator save** (the config/floor save UIs are password-gated; CC - can't log in) and the box-side tick awaits the demo agent reaching v0.89.0 via the floor train - (operator vouch) — see the felhom-agent REPORT. Scenario 3's measured save-to-tick latency is the - operator follow-up. +- **Fired end-to-end from a REAL operator save (2026-07-17):** the operator's Day-0 manifest save + (`Artifact manifest set: agent=0.89.0 … min_agent=0.89.0`) immediately logged `poke: sync-poke + delivered to 10.77.0.2 via 167.233.158.164:22` + `fleet nudge sent to 1/2 WG peers` — the + `handleSetArtifacts`→`PokeAllHosts()` wiring proven live to the box (the 2nd peer is offline). The + MinAgent-floor coupling also fired: `managed floor HELD for demo-felhom: agent 0.88.0 < MinAgent 0.89.0`. +- **Scenario 3 box-side latency measured** once the demo reached v0.89.0: a poke ep0→box delivered in + **~31 ms** and fired the immediate desired-state cycle (see the felhom-agent REPORT). Full save→tick + ≈ ~0.45 s (SSH-dominated), well under the ≤2–3 s target. All three live legs (Scenario 4 A/B train, + Scenario 1 R-22 self-heal, Scenario 3 latency) are proven-live; the capability-map poke + F4 rows are + upgraded to PROVEN-LIVE. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 7f50d12..d68d9cc 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -33,7 +33,7 @@ | BYO install: `--mode byo`, mandatory caps, host-mutation disclosure, coexistence guards | installer v1.15+, agent | **PARTIAL** | `DRILL-GL6-2026-07-08` (demo box); GL-8 coexistence fixes | Peti clean-slate reinstall on proxmox2 is the first real BYO run of the current path → R-1 | | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) | | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | Endpoints driven on the drill VM; customer-facing **browser wizard** leg not yet live-validated. First supervised ceremony with a real customer pending → R-1. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart | -| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected. Candidate upgrade to PROVEN-LIVE — see REPORT.) **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22):** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking, so a descriptor-provisioned id (the demo's legacy `felhom-offsite`) converges hands-free — no more one-shot `pveum` grant | +| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected. Candidate upgrade to PROVEN-LIVE — see REPORT.) **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant | | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 | ## B. Apps & catalog @@ -106,7 +106,7 @@ | Scenario | Components | Status | Evidence | Gap / roadmap | |---|---|---|---|---| | Customer/host management: 8-tab detail, scoped auto-refresh, safe stale-host deletion, capability chips | hub v0.47–0.53 | **PROVEN-LIVE** | hub v0.53.0 dead-host roll-up live on the Peti cluster (proxmox1 down 23h); `CAMPAIGN-4-2026-07-13` (operator UI driven live); `DRILL-day0-take2` F-16 (offsite/freeze buttons live) | 8-tab render + capability chips are **render-test-validated** (hub UI is password-gated; CC cannot log in). (Cited "daily operator use" was a no-doc citation; `AUDIT-hub-gui-2026-06-30` predates these features at hub v0.25) | -| Config/state change round-trips in **seconds** (hub↔box immediacy; 15-min cycle stays the backbone): box→hub out-of-cycle report (Dir 1) + hub→box `GET /api/v1/wait` long-poll wake (Dir 2) | controller v0.139/140, hub v0.58 | **PARTIAL** | Transport proven live through the real DNS-only ingress: `SPIKE-immediate-sync-transport-2026-07-16` + hub v0.58.0 / controller v0.140.0 REPORTs — 240 s no-annotation hold (25 s heartbeat defeats nginx's 60 s `proxy_read_timeout`, no ingress change), 0.047 s wake-on-change, hub `rollout restart` = 1 WARN + 0-storm reconnect; Dir-1 2 s box→hub round-trip live in controller v0.139.0 | The operator-UI **save→apply** round-trip is not fired end-to-end live (hub UI password-gated; CC can't log in) → R-23; the wake transport and the ACK→config_version→`ConfigRefresher` delivery chain are each proven, only the UI-triggered bump leg is unexercised. Agent-plane (host-domain desired-state) poke **first slice SHIPPED** (Direction-2a, agent v0.89.0 + hub v0.59.0, 2026-07-16): contentless ep0-relayed UDP poke → agent immediate desired-state cycle, per `SPIKE-immediate-sync-transport-2026-07-16` P4. **ep0 leg live-validated** (`felhom-poke` forced-cmd → `poke-fired`; non-WG target refused; hub logs `agent-plane poke enabled`); the full save→poke→tick round-trip awaits the demo agent reaching 0.89.0 via the floor train (R-13 first slice) | +| Config/state change round-trips in **seconds** (hub↔box immediacy; 15-min cycle stays the backbone): box→hub out-of-cycle report (Dir 1) + hub→box `GET /api/v1/wait` long-poll wake (Dir 2) | controller v0.139/140, hub v0.58 | **PARTIAL** | Transport proven live through the real DNS-only ingress: `SPIKE-immediate-sync-transport-2026-07-16` + hub v0.58.0 / controller v0.140.0 REPORTs — 240 s no-annotation hold (25 s heartbeat defeats nginx's 60 s `proxy_read_timeout`, no ingress change), 0.047 s wake-on-change, hub `rollout restart` = 1 WARN + 0-storm reconnect; Dir-1 2 s box→hub round-trip live in controller v0.139.0 | The operator-UI **save→apply** round-trip is not fired end-to-end live (hub UI password-gated; CC can't log in) → R-23; the wake transport and the ACK→config_version→`ConfigRefresher` delivery chain are each proven, only the UI-triggered bump leg is unexercised. Agent-plane (host-domain desired-state) poke **first slice PROVEN-LIVE** (Direction-2a, agent v0.89.0 + hub v0.59.0, 2026-07-17): contentless ep0-relayed UDP poke → agent immediate desired-state cycle, per `SPIKE-immediate-sync-transport-2026-07-16` P4. Full path live-proven: a real operator manifest save fired `poke: sync-poke delivered to 10.77.0.2`; the box (0.89.0) received it and logged `poke received → triggering an immediate desired-state cycle` → `out-of-band report triggered` — **~31 ms ep0→box, sub-ms to the report cycle** (WG-confined, from 10.77.0.1 to the 10.77.0.2-bound socket); save→tick ≈ ~0.45 s (SSH-dominated), well under ≤2–3 s. R-13 first slice (listener+sender only; the rest of the mutual-repair arc stays open) | | Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | `runbooks/publish-train-rules.md`; demo-fleet updates proven | **Never proven on a real remote customer** — parked trains `RUNBOOK-publish-0.79/0.81/0.85-*` await Peti → R-1 | | Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed | agent v0.70+ | **PROVEN-LIVE** (demo) | `SPIKE-agent-selfupdate-2026-07-05` | Remote-customer proof pending → R-1 | | Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** (demo) | 07-10 arc | | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 1c425bc..08d0e8a 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -37,7 +37,7 @@ | R-10 | T-6E-1: DB-dump dir-fsync asymmetry (LOW, confirmed in 6E) | XS | idea | One-line hardening; batch with the next controller task | | R-11 | Tester-facing one-pager: what the box does, known limitations, how to report (channel decision: Messenger group?) | S | idea | Pairs with R-3 | | R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open | Viktor's own action items from 6D/6E | -| R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED agent v0.89.0** (2026-07-16) | On a 403 the reconcile now self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`. Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. | +| R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. | | R-17 | Old-box archive (u629193-sub1) retirement decision — 9/9 byte-identical restores verified | XS | awaiting-decision | Viktor ruling | | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE | | R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) | @@ -48,7 +48,7 @@ | ID | Item | Size | Status | Notes | |----|------|------|--------|-------| | R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster | -| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | **first slice shipped (poke channel)** | **FIRST SLICE SHIPPED — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-16):** the ep0-relayed contentless poke (hub→ep0 `felhom-poke` forced-cmd→UDP→box WG /32:51822, ~0.42 s, peer-confined, zero ep0/box infra change) now reaches the **agent** and fires an immediate desired-state cycle. ep0 leg live-validated; the full round-trip awaits the demo agent reaching 0.89.0 via the floor train. This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per `SPIKE-immediate-sync-transport-2026-07-16` P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately | +| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | **first slice PROVEN-LIVE (poke channel)** | **FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17):** the ep0-relayed contentless poke (hub→ep0 `felhom-poke` forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the **agent** and fires an immediate desired-state cycle. Full path live: real operator manifest save → `sync-poke delivered to 10.77.0.2`; box → `poke received → immediate desired-state cycle` (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per `SPIKE-immediate-sync-transport-2026-07-16` P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately | | R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | | | R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11) | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → claim-code pairing (option C) | XL | **in-progress** (slice A shipped 2026-07-16; physical run 2026-07-16) | **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B** = physical N100 RUNBOOK (real DMI/firmware/USB/prior-LVM). **Slice C** = claim-pairing so the *distributed* ISO carries no retrieval passphrase (slice-A ISO is secret-bearing, supervised/single-use). Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* |