From fa0ecb2cd7fbcb2f6081072d9ab316aea2c95c74 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 13 Jul 2026 16:24:33 +0200 Subject: [PATCH] =?UTF-8?q?RUNBOOK-peti-return:=20Phase=200a/1=20execution?= =?UTF-8?q?=20record=20=E2=80=94=20STOPPED=20at=20Gate=20P1=20(controller?= =?UTF-8?q?=20not=20converging;=20cluster=20split:=20guest=20not=20on=20th?= =?UTF-8?q?e=20agent's=20node)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GzammAMzsJTgpQHqxwM2bC --- REPORT.md | 189 +++---------- .../pilot/RUNBOOK-peti-return-2026-07-13.md | 261 ++++++++++++++++++ 2 files changed, 293 insertions(+), 157 deletions(-) create mode 100644 documentation/pilot/RUNBOOK-peti-return-2026-07-13.md diff --git a/REPORT.md b/REPORT.md index f43ed1c..b396376 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,167 +2,42 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## Hub closing bundle — v0.53.0 (F-14 gated auto-Reissue + dead-host roll-up + bearer out of git) — 2026-07-13 +## RUNBOOK-peti-return — Phase 0a/1 executed, **STOPPED at Gate P1 for a ruling** — 2026-07-13 -The last engineering items on the pre-tester board. Commits `2321077` (part 1), `04861a7` -(part 2), `36c7213` (part 3), `80aca3a` + `3682303` (part 4 + gate re-land), `3efb1b1` (docs), -`3616dcb` (manifest 0.53.0); felhom-controller `51c871a` (example-config scrub). -Deployed: **felhom-hub:0.53.0**, ArgoCD Synced/Healthy, rollout clean, startup log clean. +Operational run, read-only throughout (hub API with the bearer + a `hub.db` snapshot queried on +180 and deleted after). Nothing signed, pushed, or changed on any box or in the hub. Full +evidence: `documentation/pilot/RUNBOOK-peti-return-2026-07-13.md` §EXECUTION RECORD. -### Baselines (re-confirmed live before work) +### Gate P1 FAILS — controller 0.115.0 is NOT converging to the 0.122.0 floor -- Hub v0.52.0 at head (`74fa61c`, past the 0.126.x doc commits); manifest image `0.52.0`. -- The operator bearer literal committed at `manifests/hub.yaml:77` (the runbook-flagged item). +- The hub **is serving** the floor (DB floor 0.122.0; agent 0.81.0 ≥ MinAgent 0.81.0; no HELD + lines in live hub logs). The stall is box-side. +- Self-update worked until 07-11 13:38 (0.110→0.112→0.113→0.115), then the **controller→agent + :8443 channel died at 07-11 14:01** ("no route to host 192.168.1.170:8443") and is **still + refusing today** (fresh `agent_channel_unreachable` 16:15 local, hours after host-reports + resumed). The floor auto-update delegates the container swap to the agent over that channel; + a failed attempt is persisted **once-per-floor** (anti-flapping) and never retried. +- **Root cause below it: the cluster split the runbook itself declares a STOP.** The agent's node + ("proxmox", 128 GB RAM) rebooted ~3.5 h ago and reports **0 guests**; the guest's controller + reports 4.2 days uptime on an i5-2500/8 GB — **the felhom guest is not on the node the agent is + enrolled on** (proxmox1/proxmox2 shape; single host row verified). -### Step-0 citation — where the tenancy↔host linkage lives +### Phase 0a record (the notification-pipeline first-real-customer answer) -The ep0 tenancy is **customer-keyed** (`tenantsync.Provision(ctx, customerID)`; namespace + -`felhom@pbs!` token per customer — `hub/internal/tenantsync/client.go`). Its only -host-side anchors are (a) the `pbs_dr` descriptor inside `hosts.desired_json` -(`hub/internal/web/pbsdr.go:63-92` read/merge) and (b) the consume-once secret in -`host_pbs_secrets` keyed by host_id (`hub/internal/store/pbsdr.go:10-17`). **Both die inside the -DeleteHost cascade** (`hub/internal/store/store.go` — `host_pbs_secrets` and `hosts` deletes in -the one-tx statement list), so no surviving row could carry a provenance column → a dedicated -`host_deletions` table is the justified shape. The F-14 gate matches on `customer_id` because -the tenancy is customer-scoped; it reads the MOST RECENT record so an old acked deletion never -whitelists a newer un-acked one. +- Claim code: issued gen 1 + **emailed 2026-07-12 16:49:09Z**, NOT yet claimed. +- **FINDING P1-F1: the motioneye 100 % warning NEVER emailed Peti** — all 16 `storage_fill_critical` + notifications went operator-channel only; no `customer_notifications` row exists pre-claim. + Follow-up task: define intended pre-claim customer-notification behavior. +- Collateral: P1-F2 agent local vzdump of 9201 fails every cycle (`not a UPID: "OK"` — cross-node + vmid); P1-F3 `wg-handshake-read` DEGRADED (wg tools missing on his node); P1-F4 the MinAgent + conditional floor is served from last-known agent_version with no freshness check (floor was + served while his agent was 40 h dead, guaranteeing the failed swap). motioneye VG: 1.2 TB at + 99.66 % (4.09 GB free). -### Part 1 — deletion provenance (`host_deletions`) +### Ruling needed (all queued in the runbook doc) -- Schema: `id, host_id, customer_id, deleted_at, escrow_acked` + customer index - (store.go, additive v0.53.0 block). **No backfill** — pre-record deletions (incl. the - take-two drill's own ep0 residue, already manually cleared) legitimately fall to the manual - path. -- Written **inside** the DeleteHost transaction, before the cascade deletes read the host row - away. `escrow_acked = deleteEscrow AND an escrow row was actually present` — "removed through - the escrow-ack flow" means an acknowledged destruction happened; ticking the box over nothing - records `false` (justification: the ruling's operative fact is destroyed-with-acknowledgement, - not checkbox state; pinned by test). -- `LatestHostDeletion(customerID)` — newest record only. - -### Part 2 — F-14 gated auto-Reissue - -`pbsdrProvisionAtom` (pbsdr.go), on `tenantsync.ErrTokenExists`: reads the customer's latest -deletion record. `escrow_acked=true` → invokes the **existing** `tenantsync.Reissue` op (no new -endpoint interaction), stores the `pbsdr_auto_reissue` hub-source audit event with the operator -note **"Previous key destroyed (acknowledged deletion) — credentials re-issued automatically."** -(+ deleted_host/deleted_at/new_host/token_id details), logs INFO, and proceeds into the normal -secret+descriptor+bump tail. No record / un-acked → the pre-existing refusal string, -**byte-unchanged**, and the manual Re-issue button path untouched. Both rulings recorded -verbatim in CONTEXT.md; take-two F-14 row dispositioned SHIPPED (live validation = the next -real host-reset cycle, per spec option 4 — fixtures + red-proofs carry it until then). - -### Part 3 — dead-host roll-up honesty - -- `hub/internal/web/rollup.go`: `controllerStatus()` (the ONE copy of the previously - triple-inlined controller-derived chain — behavior-preserving) + - `(s *Server) foldHostStatus(customerID, base, customerHasReported)` = - `worst(controllerDerived, hostStatusOf(each expected host))` via **`Server.hostStatus` - only** (hosts.go — the single staleness definition shared with HostStalenessChecker; no - second threshold introduced anywhere). -- Mapping: host `down`/`stale` → customer capped at **warn** + cause chip - `host down|stale: `; `pending` hosts worsen only once the customer has ever - reported (customer-summary row exists — the durable proxy for "never had an ok report"; - reports prune at 90d so a per-report query can't be the definition). Statuses already worse - (down) or administrative (disabled/blocked) keep their token; the chip still names the host. -- Wired at all three derivation sites: dashboard (server.go), /configs list (configs.go), - customer detail header + summary strip (configs.go + customer_unified.html). Chip renders as - an existing `status-badge status-badge-warn` token (no new CSS). Display + derivation only — - checker alerting untouched. `hub_confirm_gate.py` green. - -### Part 4 — bearer out of git - -- `manifests/hub.yaml`: ConfigMap `report_api_key` → empty placeholder; Deployment env - `REPORT_API_KEY` ← `secretKeyRef: report-api/REPORT_API_KEY`, **deliberately not - `optional:`** (missing Secret must fail Ready). `cmd/hub/main.go`: `REPORT_API_KEY` env - override (the RESEND_API_KEY twin) — the hub read the key from the YAML config, so this - minimal adaptation was required (cited: main.go env-override block, Config.API.ReportAPIKey). -- Gate: `scripts/manifest_bearer_gate.py` — 64-hex bearer shapes anywhere in `manifests/`, - exit 1; `felhom.secret.yaml`'s pre-existing umami APP_SECRET prints as a **visible non-fatal - KNOWN-BACKLOG line** (secrets.md backlog, out of scope — no history rewrite, no umami touch). - GOTCHA found: the repo `.gitignore` `*secret*` pattern silently un-tracked the gate's first - name (`manifest_secret_gate.py` never landed in commit `80aca3a`) — re-landed as - `manifest_bearer_gate.py` (`3682303`). -- **Consumer list (the GLOBAL key only — per-customer/per-host keys are hub-generated and - unaffected):** (1) the hub itself (now via Secret/report-api); (2) operator-typed tooling — - `felhom-ops … -hub-key` (GL-6/PBS-DR runbooks) and the runbook curl probes - (break-glass.md §, offsite-endpoint.md §§) — all reference it "out-of-band", nothing - machine-persisted; (3) `felhom-controller/controller/configs/controller.yaml.example` carried - the real literal as example text (never a live consumer) — **scrubbed** (`51c871a`). -- Both publish runbooks' ROTATION notes now carry the 2026-07-13 disposition; full - create/rotate/verify procedure in `documentation/runbooks/secrets.md` §"Operator/global - bearer key". - -### Deploy + live validation (§13) - -- Image `felhom-hub:0.53.0` built+pushed on 180; **Secret/report-api created FIRST** - (value extracted from git history on 180 straight into a 0600 file — never echoed — - `kubectl create secret`, file shredded); manifest bump `3616dcb`; ArgoCD hard-refresh + - deliberate sync → **Synced/Healthy**, rollout OK, image verified, startup log clean. -- **Scenario E live:** authed probe with the Secret-provided value → **200**; wrong key → - **401**; /healthz 200. Fail-Ready property verified non-disruptively: a throwaway pod with - the same non-optional `secretKeyRef` shape against an absent Secret name → - `CreateContainerConfigError` (container never starts); the live manifest's `report-api` ref - carries zero `optional:` flags. -- **§13.2 roll-up live leg (drill host `demo-vm-felhom-2f4b00`, hostname drill-day0) — - the exact masking shape, disproven live, then recovered (all via the logged-in operator - browser session; screenshots recorded):** - - BEFORE (13:0x Z): dashboard "Demo in VM = **OK**, no chip"; /hosts host **ONLINE**. - - `systemctl stop felhom-agent` at **13:09:28Z**; the guest controller kept reporting. - - AFTER (~13:45Z, past the 30m stale threshold): dashboard row - **"Demo in VM = WARN + chip `HOST STALE: DEMO-VM-FELHOM-2F4B00`"** with the controller's - last-seen at **1 min ago** — pre-fix this row was green (the drill-1/Peti masking bug). - Customer detail header + summary strip both carry the same chip → the header says WHICH - host; the controller Health card honestly still reads STATUS: OK (the warn comes from the - host fold, correctly attributed). The HostStalenessChecker independently emitted its - warning event (events badge 2→4) — alerting untouched, as specified. - - RECOVERY: `systemctl start felhom-agent` at **13:46:22Z** → first heartbeat → row back to - **OK, chip gone** by 13:47Z. Both states + recovery recorded. - -### Red-proofs (all run → FAIL → restored; `git diff` clean after each) - -| # | Mutation | Failing evidence | -|---|---|---| -| Part 1 | provenance INSERT dropped from the DeleteHost tx | `TestDeleteHost_ProvenanceRecord`: "no deletion record written by the escrow-ack delete"; scenario A: 502 refusal instead of 303 (gate found nothing) | -| B | gate bypassed (auto-reissue unconditionally, nil-safe) | both `TestPBSDR_F14NoRecordNeverRekeys` subtests: save = **303** (silent re-key proceeded) instead of the 502 refusal | -| C | `foldHostStatus` short-circuited to controller-only | `TestRollup_DeadHostMasking`: "dashboard row is GREEN over a 23h-dead host" + missing chip + detail header silent; 2 boundary subtests also red | -| E | 64-hex literal reintroduced into manifests/hub.yaml | `manifest_bearer_gate.py` exit **1**, literal reported masked (`00000000...0000`) | - -### Tests (scenarios A–E) - -- A: `TestPBSDR_F14AutoReissueOnAckedDeletion` — real escrow-ack DeleteHost, real handler path; - asserts reissue call count, descriptor, generation, consume-once secret, the stored audit - event + note text, and zero secret leakage into logs. -- B: `TestPBSDR_F14NoRecordNeverRekeys` (no record / latest un-acked) — the byte-unchanged 502 - + **zero** Reissue calls + zero state writes (the exact non-effect). -- C: `TestRollup_DeadHostMasking` — the Peti fixture (host backdated 23h, controller report - fresh) rendered through the REAL dashboard + customer handlers. -- D: `TestRollup_Boundaries` — all-ok pass-through, stale-host warn chip, onboarding pending - exclusion, post-onboarding pending worsening. -- E: gate (committed) + the live auth/fail-Ready legs above. -- Full hub green gate: `go build ./... && go vet ./... && go test ./...` — all packages ok. - -### Observations - -- **Peti's box is BACK ONLINE** (dashboard: Peti Proxmox OK, controller **0.115.0**, agent - **0.81.0**, fresh reports; 5/5/5 event badges). The parked publish-train items (Peti D/E/G: - sign agent_update 0.79→0.85 — his agent already shows 0.81 —, journal group, G1 temp-creds - deletion) look actionable; his `motioneye` storage target reads **100% full** (WORST STORAGE - on /hosts). Operator follow-up. -- The spec's "F-13 reference resolved as typo-or-assigned": the take-two record's findings run - F-14/F-15/F-16; **F-13 is the CAMPAIGN-3 nfs4-under-mp8 finding** (separate, still open) — - the numbering is continuous across records, no typo, nothing to reassign. -- The "remove from system" decommission finisher stays parked (untouched by this bundle). -- The `.gitignore *secret*` un-tracking gotcha (above) is recorded in CONTEXT.md — any future - "secret"-named script would silently never land. - -### Operator steps remaining (STOP points — not executed) - -1. **Bearer value ROTATION** (Viktor GO + his out-of-band consumers): the supervised procedure - in documentation/runbooks/secrets.md §"Operator/global bearer key" — mint → Secret update → - hub rollout → verify new-key 200 / old-key 401 / customer boxes unaffected. Until it runs, - the git-history copy of the old value is still live (it currently IS the Secret's value — - deployed same-value first so nothing broke). -2. **F-14 full live leg** (optional per spec §13.4): a complete host-reset cycle on the drill - box reproduces the dead-end end-to-end; the scenario A/B fixtures + red-proofs carry it - until the next real reset validates it in production terms. +1. Cluster split: migrate guest back / move the agent / accelerate agent-follows-guest. +2. Post-heal convergence path: floor bump (0.125.0 earmarked), manual trigger, or state clear. +3. Optional: hub controller-log pull to confirm the stall variant (CC did not trigger — write op). +4. Phase 0b/0c (message Peti, call slot) — Viktor, unaffected. Phase 4 rotations independent — + can proceed on GO. diff --git a/documentation/pilot/RUNBOOK-peti-return-2026-07-13.md b/documentation/pilot/RUNBOOK-peti-return-2026-07-13.md new file mode 100644 index 0000000..4df70ed --- /dev/null +++ b/documentation/pilot/RUNBOOK-peti-return-2026-07-13.md @@ -0,0 +1,261 @@ +# RUNBOOK — Peti's return: convergence, the parked train, the first real-customer onboarding, and the credential rotations + + + +--- + +## 0. Scope & standing rules + +- **In scope:** convergence verification, the publish train's parked D/E/G at CURRENT versions, + claim, WG consent, DR-tier migration, ceremony + auto-confirm, offsite opt-in, motioneye + remediation, CF token rotation (BOTH boxes — see the trap in Phase 4), hub bearer rotation. +- **Out of scope:** cluster/agent-follows-guest work (his proxmox1/proxmox2 HA shape is a known + roadmap item — if the guest migrates mid-runbook, STOP and reassess); the old-box archive + (u629193-sub1) retirement decision — separate. +- **Custody rule for the whole runbook:** Peti's password and R are PETI'S. They never appear on + Viktor's or CC's screen, in transcripts, or in the report. Where a step displays them, Peti + drives. +- No gate-lifts anywhere (standing rule). Diagnose-before-fix on every anomaly. + +--- + +## Phase 0 — TODAY, before anything technical (Viktor, ~15 min) + +0a. **Hub → Peti's customer page, record:** current controller version (self-update may be +mid-flight or done), claim state (code issued? issued-at? claimed?), the notification/event log — +**specifically whether the motioneye 100% warning ever emailed him.** That answer is the +notification pipeline's first real-customer test; record it either way, and if it never fired, +that is a FINDING with its own follow-up task. + +0b. **Message Peti** (before the machine talks to him): fan congratulations; "your box is +updating itself over the next hours — you'll get (or already have) an email with a setup code, +it's real, don't delete it"; camera drive is full, we'll fix it together; propose the onboarding +call (~90 min, needs him at a keyboard with root on his own box). + +0c. Sign nothing yet. If the claim email was never issued or landed in spam, note it — the +resend button is Phase 3 ammunition, not a Phase 0 action. + +**Gate P0:** Peti has acknowledged and a call slot exists. + +--- + +## Phase 1 — passive convergence check (CC, strictly read-only) + +Confirm from hub data only: controller reached **0.122.0** (floor); the claim-code hash was +ACK-delivered; the box transitioned legacy-banner → gate armed; reports healthy throughout; the +roll-up row honest (it will read WARN "storage" territory only if the storage health warns — +motioneye's full drive may legitimately color it). If the controller is NOT converging: diagnose +first (report errors? MinAgent mismatch? update loop?), report, and STOP for a ruling — do not +push anything. + +**Gate P1:** controller ≥ 0.122.0, gate armed, box healthy. Record the self-update elapsed time +(fleet's second floor-proof, first on customer hardware). + +--- + +## Phase 2 — the parked train phases, at current versions (CC executes, Viktor signs) + +Execute RUNBOOK-publish-0.85-0.120 **Phases D/E/G + gates 0b/0g** against `peti-felhom-86d37d`, +adapted: agent target is **0.87.0** (already published + manifest-vouched — re-run the +immutability and live-bytes gates for 0.87.0 rather than 0.85.0; cite the shas in the report). + +- **STOP — Viktor signs the agent op** (0.81.0 → 0.87.0) only after the gates pass. Never sign + for a host that has gone offline again (gate 0a re-checked at signing time). +- Post-update: clean restart, self-check; **expected capability state:** `pbsdr-*` will read + DEGRADED (binary missing) until Phase 3c's migration — record it, don't chase it. Everything + else green. +- Re-run the 0b/0g creds+key gates per the train runbook. + +**Gate P2:** agent 0.87.0 live on his host, self-check clean modulo the expected pbsdr gap. + +--- + +## Phase 3 — the onboarding call (Viktor + Peti; CC assists; Peti-paced throughout) + +3a. **Claim.** Peti opens `felhom.sajatfelhom.hu`, enters his code (resend from the hub if +needed — registered address only), sets HIS password. Verify: login works, code reuse refused. +Record the **G10 closure line for the first real customer** (timestamped). + +3b. **WG consent.** Walk the tester agreement's WireGuard disclosure (base-infrastructure +tunnel, outbound, hub-toggleable). Peti acknowledges explicitly; record the ack in the pilot doc. +If he declines: DR-tier and offsite are off the table today — the runbook continues at 3f; no +pressure mechanics. + +3c. **DR-tier migration** (supervised SSH on his host — the documented pre-v1.15 one-liner): +felhom-pbs-apply into /usr/local/sbin, `age` installed, `wg_tunnel.enabled: true`, ACL restored +to the default set (incl. felhom-pbs + /storage grant). Then hub: DR flag ON → watch the cascade +(WG peer → descriptor → applied) exactly as drilled. Capabilities go clean (pbsdr inactive→active +states per the 0.86 semantics). + +3d. **The ceremony — operator-blind R.** PETI runs +`felhom-agent --selftest=escrow-create --upload` in HIS terminal on HIS box; Viktor coaches from +the ceremony runbook without screen-sharing the output. Peti writes R on paper, confirms the +command's self-verify line appeared. This is the escrow model's promise performed for real: the +operator never sees R. CC verifies only the OUTCOMES on the hub (blob present, hash staged). + +3e. **Auto-confirm, hands off.** Drilled twice at 6-8 minutes; watch pending → escrowed on the +hub; nobody clicks anything. A hash-mismatch → do NOT manual-confirm; finding + re-ceremony. + +3f. **Offsite opt-in** — Peti's decision, with the honest pitch (encrypted recovery units to the +Storage Box; media stays local; quota; what R recovers). If yes: hub provisioning → he toggles +his chosen apps → first run; the restore-to-verify can run same-call or that evening. + +3g. **motioneye triage.** Options in order of preference: retention settings inside motioneye +(cap days/size), move its HDD_PATH to a larger drive via the migration feature, or manual +cleanup. His call; verify recordings resume and the storage warning clears (and that the +CLEARED state also behaves correctly in notifications). + +**Gate P3:** claimed + (if consented) DR applied + escrowed + offsite per his choice + camera +recording again. Capture every moment of confusion VERBATIM — these notes are the alpha +one-pager's raw material. + +--- + +## Phase 4 — credential rotations (Viktor; CC assists) + +4a. **CF token — mind the shared-token trap:** the old "Edit zone DNS" token covers BOTH +remaining zones and lives on BOTH boxes — revoking it before re-tokening the demo kills the +demo's ACME too. Order is mandatory: + 1. Mint `sajatfelhom.hu-token` AND `demo-felhom.eu-token` (each: Zone DNS:Edit + Zone WAF:Edit + + Zone:Read, single zone). + 2. Deliver each to its box (controller.yaml `infrastructure.cf_api_token` + traefik env + refresh — the manual path; the absence of a hub-pushed rotation flow is a KNOWN gap, note it + again in the report). + 3. Verify cert issuance on both (force one renewal or confirm a clean DNS-01 in traefik logs). + 4. Only then revoke the old multi-zone token; confirm "Last used" stops moving. + +4b. **Hub bearer VALUE rotation** — the pending STOP from the closing bundle: secrets.md +§"Operator/global bearer key": mint → Secret update → rollout → new-key 200 / old-key 401. The +git-history copy is dead only after this. + +**Gate P4:** three single-zone tokens live (enkisfelhom already done), zero multi-zone customer- +resident credentials, bearer rotated. + +--- + +## Phase 5 — wrap + +Hub screenshots (Peti's row: ONLINE, claimed, DR state per his consent, escrowed, storage +recovering); close the parked rows in the train runbook; pilot doc updated with the onboarding +record (consent acks, G10 line, ceremony outcome); REPORT overwrite + CONTEXT; findings table +(expected candidates: the notification-pipeline answer from 0a, any claim-flow friction, the +capability display during the DEGRADED window, anything Peti says that a stranger would also +say). + +--- + +## §15 report must include + +Per-phase gate evidence; the P1 floor-proof timing; the signed-op record (sha, signature, +timestamps); the G10 + ceremony + auto-confirm lines for the first real customer; the 4a +before/after token inventory ("Last used" evidence on the revoked token); the bearer rotation +transcript (values redacted); the verbatim friction notes; explicit list of what was deliberately +NOT done (cluster work, archive retirement) with their queue positions. + +--- +--- + +# EXECUTION RECORD — 2026-07-13 (CC) — **STOPPED AT GATE P1 for a ruling** + +Everything below is hub-side evidence only (API reads with the bearer + a read-only `hub.db` +snapshot queried on 180 and deleted afterwards). **Nothing was pushed, signed, requested, or +changed on any box or in the hub.** Per Phase 1's own rule ("if the controller is NOT +converging: diagnose first, report, and STOP for a ruling"), the runbook is halted at P1. + +## Phase 0a — the hub-side record (CC-gathered; Viktor's 0b/0c messaging still open) + +| Item | Value (evidence: hub API + hub.db, 2026-07-13 ~14:10Z) | +|---|---| +| Controller version | **0.115.0** (NOT converged; floor is 0.122.0) | +| Claim state | code **issued gen 1 2026-07-12 16:49:08Z, emailed 16:49:09Z** (`claim_claim`, channel=customer, status=sent), **NOT claimed**, 0 resets | +| Reports | controller reporting healthy every 15 min, unbroken through the whole window; `health_status=ok`, last seen 14:04:08Z | +| Agent | 0.81.0, host-reports resumed **2026-07-13 10:29:12Z** after a ~40h gap (07-11 ~14:00 → 07-13 10:29) | +| motioneye emailed him? | **NO — FINDING P1-F1.** Every `storage_fill_critical` notification (16 firings since 07-12) went to channel **operator only**. The ONLY customer-channel email peti-felhom has ever received is the claim email. There is **no `customer_notifications` row** for peti-felhom (no prefs exist pre-claim) — the customer tier of the pipeline never engaged. Follow-up task needed: decide the intended pre-claim customer-notification behavior. | + +## Phase 1 — convergence check: **GATE P1 FAILS — controller is NOT converging** + +**Facts (each with source):** + +1. **The floor IS being served.** DB `hub_settings.min_controller_version = 0.122.0` (set + 2026-07-12 17:06:08Z); manifest MinAgent 0.81.0; hosts row agent_version 0.81.0 → + `ResolveManagedFloor` serves the floor (confirmed by absence of any "managed floor HELD for + peti-felhom" line in live hub logs while reports flow every 15 min). Manifest also confirmed + vouching agent **0.87.0** sha `2447d4a3…dc7a` + golden 0.120.0 (runbook ground state matches). +2. **The self-update mechanism itself worked while the channel was up:** `controller_updated` + events 0.110→0.112 (07-10 16:01), 0.112→0.113 (07-11 10:41), 0.113→0.115 (07-11 13:38). +3. **The controller→agent channel died 23 minutes after the last successful update and is STILL + dead:** `agent_channel_unreachable` 07-11 14:01:53 (connection refused), then + `agent_channel_unknown` "dial tcp **192.168.1.170:8443**: no route to host" (07-11 14:02–14:06), + and a FRESH `agent_channel_unreachable` (connection refused) at **2026-07-13 16:15:08 local** + — after the agent's host-reports had already resumed. The controller cannot reach the agent's + local :8443 API even now. +4. **Why that blocks convergence (code-verified,** `controller/internal/selfupdate/updater.go`**):** + the floor auto-update pulls the image in-guest, then **delegates the container swap to the + host agent** over that same channel. A failed swap persists `UpdateState{TargetVersion: + , Status: failed}` and `MaybeAutoUpdate` **permanently skips a floor that has a + persisted failed attempt** (anti-flapping, by design). Floors 0.120.0 (07-12 11:19) and + 0.122.0 (07-12 17:06) were both set while the channel was down. So either (a) an attempt at + 0.122.0 failed at the swap (or pull) step and is now persistently skipped, or (b) the registry + pre-check fails each cycle and it defers forever — both stall exactly as observed, and both + are unresolvable from the hub side. The controller's own log (hub log-pull button, or the call) + will say which. +5. **`controller_started` count unchanged since 07-11 14:04** — no restart/swap ever happened + after the channel died. Consistent with all of the above. + +**Root cause underneath it all — the out-of-scope trip-wire is ALREADY TRIPPED:** + +- The agent's node (`host.node = "proxmox"`) reports **uptime 12,663 s (~3.5 h — rebooted ~12:35Z + today), 128 GB RAM, 0 guests**. +- The guest's controller reports **uptime 365,289 s (~4.2 days), i5-2500, 8 GB** — the guest has + been up longer than the node the agent runs on, on visibly different hardware. +- ⇒ **The felhom guest is NOT running on the node the agent is enrolled on.** Peti's + proxmox1/proxmox2 cluster shape has the guest on one node and `peti-felhom-86d37d`'s agent on + the other. Only ONE host row is enrolled for peti-felhom (verified). The runbook's own scope + rule — "if the guest migrates mid-runbook, STOP and reassess" — describes the CURRENT state, + before the runbook even starts. + +**Collateral findings (recorded, not chased):** + +- **P1-F2:** agent local backup fails every cycle: `backup FAILED: target=local vmid=9201 err= + "proxmox: not a UPID: \"OK\""` → `expected_backup_missed` (newest backup >40 h). Almost + certainly the cross-node vzdump of a vmid that is not on this node (and/or agent 0.81 vs his + newer PVE — kernel 6.17.2-1-pve). Re-evaluate after the cluster ruling; don't debug in place. +- **P1-F3:** `wg-handshake-read` capability DEGRADED ("binary not found", critical=true) on his + node — WireGuard tools missing; matters for Phase 3b/3c. `pbsdr-*` degraded is the EXPECTED + pre-3c state (per Phase 2 note). Host `cloudflared: inactive` also noted. +- **P1-F4 (design observation):** while the agent was down 40 h, the MinAgent conditional floor + kept being served because `ResolveManagedFloor` uses the LAST-KNOWN `hosts.agent_version` with + no freshness check — the floor was served to a box whose agent was dead, guaranteeing the + failed-swap attempt in (4). Worth a ruling on whether the floor should be held when the host is + stale/down. +- motioneye storage (his own LVM VG, 1.2 TB): 99.66 % full, 4.09 GB free — Phase 3g stands. +- Storage roll-up honesty: `storage_fill_critical` fires from the HOST report; the customer-row + coloring behaved as designed. + +## The ruling needed before anything continues + +1. **Cluster split** (the blocker): migrate guest 9201 back to the agent's node, or move/reinstall + the agent where the guest lives, or accelerate agent-follows-guest. Viktor + Peti decision — + this reshapes Phases 2 and 3c–3e (all assume agent and guest share a host). +2. **After the channel heals**, convergence still needs one of: floor bump (0.125.0 is the + earmarked candidate and carries the .fab strand fix; NEVER halt above 0.124.0 without it), a + manual update trigger, or clearing the persisted update state during the call — because of the + once-per-floor guard in (4). +3. Optional pre-call diagnostic: hub controller-log pull (operator UI button) to confirm which + stall variant (persisted-failed vs registry-defer) — CC did NOT trigger it (write action; + Phase 1 is read-only). +4. Phase 0b/0c (message Peti, call slot) are unaffected and remain open — Viktor. +5. Phase 4 (CF token + bearer rotation) is independent of the cluster question and could proceed + on Viktor's GO at any time. + +**State: STOPPED at Gate P1. Phases 2–5 not started. Nothing signed, nothing pushed.**