felhom-controller/REPORT.md

REPORT — felhom-controller v0.138.0: escrow "awaiting hub confirmation" waiting state

Task: Diagnose the stuck "Helyreállítási kód szükséges" banner after a completed escrow ceremony on the demo box; add the post-ceremony waiting-state UX; ship the hub stale-clear fix only if Phase-0 diagnosed Hypothesis B.

Shipped: controller v0.138.0 (Part 1, unconditional). Hub: NO change (Phase-0 = A, and Hypothesis B verified false). felhom-agent untouched.

---

Phase 0 — DIAGNOSIS (read-only). Verdict: A (report-cycle lag) — already resolved on the box.

The banner renders while Offbox.EscrowState != "escrowed". The pending→escrowed flip happens only in report.EscrowAutoConfirmer.Reconcile on a report ACK, on hash-match between the ACK's restic_pw_sha256 and the local repo password. The observed stuck banner was the ~15-min gap between a completed ceremony and the next ACK.

Evidence — demo box felhom-controller logs (guest 9201, running 0.136.0):

2026/07/16 16:13:31 [INFO] [web] escrow ceremony started via wizard (job escrow-1784218411576550074)
2026/07/16 16:13:39 [INFO] [web] escrow recovery code claimed (one-shot; not logged)
2026/07/16 16:27:58 [INFO] [report] Hub report pushed successfully (2992 bytes)
2026/07/16 16:27:58 [INFO] [escrow-confirm] hub-verified: the escrow covers the current repo password (hash d517ce7f9982…) — EscrowState auto-confirmed escrowed; offsite runs enabled

• Ceremony claimed 16:13:39; the very next hub-report ACK at 16:27:58 (~14 min later) flipped pending→escrowed via hash-match (d517ce7f9982…). One INFO confirm line, no mismatch warning, no withheld-hash gap.
• Current box state: settings.json → "escrow_state": "escrowed" (verdict A resolved itself; the box is confirmed; OffboxRunnable() true).

Hypothesis B (hub never clears stale on upload) — verified FALSE, so NO hub change:

• store.SaveHostEscrow (the escrow upload sink, handler.go:1083) already clears stale on a new blob: INSERT … ON CONFLICT(host_id) DO UPDATE SET … stale_at = NULL (store.go:2084).
• The ACK only withholds the hash while stale_at is set (store.go:2150: if staleAt != "").
• So a re-issue→re-ceremony cycle recovers on its own; the only thing not emitted is a cosmetic escrow_refreshed event — out of scope (Part 2 gated on B, which does not hold).

Hypothesis C (upload silently failed): ruled out — the box reached escrowed via a hash-bearing ACK, which requires the fresh blob + hash to be present on the hub.

Operational note: the demo Proxmox host key changed since the last session (box reprovisioned for the N100 customer-zero run). Refreshed known_hosts (accept-new) before any read — expected for the demo host, not a security event.

---

Part 1 — controller v0.138.0 (the waiting state)

1. Persisted stamp. New settings.OffboxTarget.CeremonyCompletedAt (ceremony_completed_at, RFC3339). SET on a successful recovery-code claim (web/escrow_handlers.go, only while still pending; best-effort — a stamp failure never fails the claim, the code is already revealed and the blob already uploaded). ZEROED on the pending→escrowed flip — the auto-confirmer Flip closure (cmd/controller/main.go) and the deprecated manual confirm (web/offbox_handlers.go). Persisted ⇒ survives a controller restart mid-wait.
2. The pick. web/handlers.go: offboxCeremonyWaitState(t) → (awaiting, timedOut); escrowCeremonyGraceWindow = 35m (2 report cycles + slack, named). Falls back to the plain pending CTA when escrowed, unstamped, or the stamp is unparseable.
3. Card branch. backups_remote.html: one new escrow-card branch ahead of the existing chain — info (blue --blue) "Helyreállítási kód létrehozva … megerősítésre vár, legfeljebb 15 perc", degrading to warn "A megerősítés nem érkezett meg a várt időn belül …" + re-ceremony CTA past the window. Existing pending / stale (Scenario F) / escrowed branches untouched.
4. Wizard final screen. backups_escrow.html "7. Befejezés" step gains the "Mi történik ezután?" info note (generated / sent to the hub / bounded ≤15-min wait) — the wizard finishes to /backups/remote, so the customer lands directly on the awaiting card.

NOT built (as instructed): no scheduler rework; no manual report-trigger. Observation: no pre-existing manual/API report-trigger seam was found (report push is the scheduler hub-report job), so the one-liner "fire a report right after the ceremony" option was correctly skipped.

Part 2 — hub v0.57.1: NOT shipped (Phase-0 = A, Hypothesis B false; see diagnosis above).

---

Green gates & tests

• go build ./... OK · go vet ./... OK · go test ./... 0 failures.
• Design gates OK: template_id_gate, emoji_gate, native_confirm_gate, offbox_rename_gate.
• New test web/escrow_wait_state_test.go — 8-case truth table + mutual-exclusion invariant. Red-proof (run→fail→revert): dropped t.EscrowState == "escrowed" from the early-return guard → escrowed clears the wait case FAILED (awaiting=true, want false — the interim card would resurface after the healthy state) → reverted → green.

Scenarios

• S-A (waiting state): ceremony completes → info card shows immediately (stamp set on claim); confirming ACK zeroes the stamp AND sets escrowed → escrowed branch wins → healthy card. ✔ (logic + live: box is escrowed).
• S-B (timeout): stamp aged past escrowCeremonyGraceWindow with state still pending → warn + re-ceremony CTA. ✔ (unit: stamped past window → timed out, at boundary → timed out).
• S-C (restart mid-wait): stamp is persisted in settings.json → the info card survives a restart. ✔ (persisted field; unit covers the stamped-pending path).
• S-D (hub, if B): N/A — Phase-0 = A, no hub change.

Deploy & live verification

• Committed to main (1203321); image gitea.dooplex.hu/admin/felhom-controller:0.138.0 built+pushed on 180 (digest sha256:49f9474c…).
• Update train 0.136.0 → 0.138.0 delivered cleanly to guest 9201 (0.137.0 was built but never deployed to the box; the bootstrap pull jumped straight to 0.138.0). docker ps: …felhom-controller:0.138.0 Up (healthy); startup clean (Web UI listening on :8080, Starting scheduler with 13 jobs).
• Live end-state (matches the task target):
  • settings.json → escrow_state: "escrowed" (Phase-0 A already flipped it; no ceremony_completed_at present, correct for an already-escrowed box) → renders the healthy "A helyreállítási kód letétbe helyezve" branch (unchanged).
  • offbox enabled: true + escrowed + Hetzner storagebox target ⇒ OffboxRunnable() true.
  • New template strings confirmed in the deployed binary (embedded templates): "megerősítésre vár" ×2 (remote card + wizard note), "A megerősítés nem érkezett meg" ×1, "Mi történik ezután" ×1.
• Not live-rendered: the awaiting/timeout cards themselves — they require an in-flight pending ceremony, and re-running the ceremony is Viktor's supervised flow (would un-escrow → re-escrow the live box). S-A/S-B/S-C are covered by the unit truth table + the persisted-field design; the card markup is byte-identical to what the unit-verified booleans gate.

NOT done — Viktor's click: the first "Távoli mentés most" (first offsite run) is the customer's, per the task. CC stops at escrowed-verified.