felhom.eu/CONTEXT.md

CONTEXT — felhom.eu working state

Created with the REUSE.md rollout (2026-07-03). Authoritative history: hub/CHANGELOG.md (hub), website/CHANGELOG.md, scripts/CHANGELOG.md; end-of-task detail in REPORT.md.

• 2026-07-23 — hub v0.72.0 + controller v0.161.0: R-70 SHIPPED + R-71(c) PARTIAL (the offsite last mile is visible; burned credentials self-heal). One detector (offsite.DeliveryStateFor — secret-row timestamps × report offsite-presence), four consumers: operator card (static "delivered once" copy DELETED; state+age, amber past 30 min, stale-staged info line), offsite_delivery_stuck warning (≥1 h, 24 h durable cooldown via LastEventAt on the events table — restart-proof), R-71c self-heal through the EXISTING Re-issue path (monitor.OffsiteReissuer ← webServer, armed only with a provisioner; consumed ≥1 h + ≥4 offbox-less reports + zero offbox evidence; one restage/24 h; every firing = a warning event), and the controller's truthful empty-state banner (v0.161.0, both boxes). Rulings: applied wins (stale-staged is a flag, never a downgrade); cooldowns durable via events table; both new events operator-only (no customerMessages, not in allowedEventTypes — pbsdr_* precedent); R-39(a) guard in the CALLER — the heal re-reads the secret row at act time and refuses over an unconsumed row; the store's clobber semantics stay (Re-issue depends on supersede). Clobber red-proof observed (reissue calls = 1 + row clobbered with the guard removed). Self-heal + banner ship unit-proven, NOT live-fired (no broken box; F9 rule) — arms on the next natural occurrence. Live: checker silent on the healthy fleet; demo-felhom's stale-staged specimen intact (its cleanup is the operator's, post-validation). R-71(a) day-0 ordering OPEN → next spec. Evidence: REPORT.md (2026-07-23).

• 2026-07-23 — ISO train v1.25.0 + hub v0.73.0 (four rulings + rider). host-install 1.17.0→1.18.0, build-felhom-iso 1.24.0→1.25.0, felhom-bootstrap R-63, hub v0.73.0. Done + live-verified: (1) F9 belt everywhere — host-install installs the H1 OOB belt by default in appliance mode (--no-oob opts out; byo still refuses), oob.enabled written so the agent activates it; belt LIVE on both boxes, login PROVEN on felhom-pve (felhom-op@demo-felhom); the orphaned operator identity was re-anchored to the operator's real machine (WG peer + SSH key re-registered, ep0 forward rules persisted). (2) apt — align_apt_repos (appliance-only) switches enterprise→no-subscription; LIVE-aligned both boxes (apt-get update clean). (3) R-63 — Lat2 console font before first paint. (4) R-71 build-gate — assert_golden_ge_floor in build-felhom-iso + rule 5; red-proofed. (6) root vault — felhom-pve root rotated+vaulted+ verified over :22 (G1 parity). (7) hub v0.73.0 — offsite_stale never-ran branch ANCHORED (applied-only + consumed_at/escrow anchor; one-state-one-owner with offsite_delivery_stuck), shipped+deployed, red-proofed. CRITICAL live finding: golden 0.153.0 < floor 0.156.0 in production now — the R-71 gap is real; the gate correctly blocks a v1.25.0 build. Remaining (enforced-sequenced): golden republish at 0.161.0 + operator vouch (password-gated) → then the v1.25.0 ISO rebuild + nested drill. Evidence: REPORT.md, scripts/CHANGELOG.md, operations/nodes.md.

• 2026-07-23 — F10 offsite leg RESOLVED on demo-hp (diagnostic + designed-path repair + tier proof; no code changed). The hub said "Provisioned" while the box said „Még nincs beállítva" — root cause: the day-0 managed update (0.153.0→0.156.0, 07-21 16:28) killed the offsite apply-bridge ~35 s after it consumed the one-time password, before key-install/persist; the box then hit the consume-404 dead-end at every start, silently, for 2 days. Repair = operator Re-issue click (converged in 8 s), escrow ceremony via the real wizard (auto-confirm 3 s), then the full proof: paperless-ngx toggled into scope, first offsite run (repo initialized on sub3, 79.8 MB, snapshots=1), probe deleted → mode=full restore (snapshot 2bf7f2e1) → place merged 1 file missing-only → md5-identical. Minted R-70 (last-mile invisible on both surfaces; hub never reads one_time_secrets.consumed_at) and R-71 [P1] (the race recurs on every fresh onboarding whose ISO floor lags the managed floor). Record: documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md. F10's PBS-DR half stays open (F13 + ceremony R-moment).

• 2026-07-22 — hub v0.71.0: the notification train (audit F11+F12+F14-light). Four rulings now standing: (1) recovery pairing — *_recovered notifies the operator always and the customer iff the customer was mailed the paired stale/down (evidence = notification_log customer- channel sent rows via store.LastCustomerSentAt; enabled_events deliberately ignored for recovery; ties → no mail). Severity semantics FROZEN — recovery stays info, routed by an explicit eventType branch before the severity gate. (2) seed-at-claim — MarkClaimed seeds customer_notifications from the registered email, INSERT-if-absent only (never touches an edited row; empty email = no-op; seed failure never fails the claim); default set is critical-only: node_down, backup_failed, disk_critical, host_disk_critical, storage_fill_critical, offbox_repo_orphaned. (3) empty-email no-clobber — a prefs push with empty email preserves a stored address (hub-side belt; controller 0.160.0 already guards its two push legs). (4) priority headers — error/critical mails carry X-Priority: 1 + Importance: high (Resend headers, live-probed); the test event now also mails the operator with those headers (one click proves both channels + rendering). Latent nil-deref fixed: sendTestEmail panicked on a customer with no prefs row. NOT yet live: a natural *_recovered mail (next real staleness cycle / reboot drill — never fabricate one by blocking reports) and seed-at-claim on a real claim (Peti's Friday reinstall is the natural candidate).

• 2026-07-22 — hub v0.70.1: the ghost customer's Delete button exists now. The v0.70.0 ghost-delete path was fully implemented and fully unreachable — the fourth inert-seam defect, this time a TEMPLATE GATE: the Danger-zone card (and the customerDeleteOpen script) sat inside {{if .HasConfig}}, so a ghost rendered no Delete button (found live on demo-vm-felhom). Fix: Deletable page flag = the exact negation of the delete preview's 404 predicate; RESET stays HasConfig-gated, Block/Unblock config-only. Render tests per branch + 2 red-proofs. The seam-wiring rule now covers template gates (see CLAUDE.md) — handler tests that POST directly prove nothing about reachability. Operator live leg PENDING: the demo-vm-felhom ghost delete click (doubles as v0.70.0+v0.70.1 live validation). Same day: **agent 0.93.0 published

  • deployed fleet-wide** (pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md), R-17 CLOSED (u629193-sub1 deleted at the console).

• 2026-07-21 — THE FLEET IS TWO HOSTS. A second Proxmox node exists: demo-hp-bb76ea, an HP t740 Thin Client (Ryzen V1756B, 30 GiB, PVE node felhom-host, customer demo-hp), installed from the armed universal ISO the same day. Both hosts run agent 0.92.1; the N100 is demo-felhom-8363b5. Full inventory: documentation/operations/nodes.md — read it before touching the HP, because two things there are load-bearing: the box's 1TB Toshiba NVMe is present and UNENROLLED (still carrying its old NTFS partition; a future Tárhely candidate that must join through the storage flow, never the installer), and there is no operator SSH key baked on it, so access is the hub-vaulted G1 break-glass password (R-61 is the fix for that lockout). Both nodes are at the VACATION site and travel home ~2026-08-02. demo-hp joined the tailnet as 100.76.96.79 — an operator-lab exception, not product shape; real customer boxes never get tailscale. Second-hardware proof, worth keeping: the pairing/day-0 chain ran end to end on virgin hardware it had never seen, the shim loader booted with Secure Boot ENABLED (so SB-off was an N100-firmware workaround, not a Felhom requirement), and the box self-lifted the floor 0.153.0 → 0.156.0 during day-0. The install was not clean, and the failures are filed: R-59 (no DHCP on the 4-port NIC → the installer baked a static fallback instead of aborting), R-60 (first-boot NIC sweep self-heal), R-61 (baked root password unknowable).

• 2026-07-19 — N100 MADE LOCATION-INDEPENDENT via Tailscale. felhom-pve (the N100) joined the tailnet as a host package (100.70.170.35, expiry disabled); DooPlex already advertised 192.168.0.0/24 via its GitOps k3s admin-system/tailscale pod (100.107.87.53). ssh felhom-pve now targets the tailnet IP (direct-over-LAN at home, tunnel when away); felhom-pve-lan = 192.168.0.162 fallback. --accept-dns=false on the host (pi-hole split-horizon preserved). The host was converted static→DHCP (reservation MAC 68:1d:ef:5d:a6:64→.162 keeps home behaviour). Measured spike: --accept-routes on the N100 while on 192.168.0.0/24 hairpins the local subnet (table 52 overrides main) → kept OFF, it's a travel-only opt-in. PBS is offsite (its own WireGuard tunnel to Hetzner, not the home LAN) so it needs none of this. Recorded, NOT fixed: the controller↔agent plane is hard-pinned to 192.168.0.162 (felhom-agent listen_addr + guest bootstrap.json), so the box is reachable + backs up off-LAN but its local operator plane is not yet portable. Full doc: documentation/operations/tailscale.md. No code changed.

• 2026-07-19 — CC WORKFLOW MIGRATED TO DOOPLEX. Claude Code now runs natively on DooPlex (192.168.0.180, Debian 13, kisfenyo) in a tmux session, working directly in /mnt/5_hdd/felhom.eu/git/<repo>. DooPlex is now the canonical CC environment; the Windows workstation trees are stale as of felhom.eu ee48a48288, felhom-agent d81fddc7b1, felhom-controller ac7323dc9a, app-catalog-felhom.eu a14c97db65 (unchanged — it had no environment-specific content). Builds are local commands; felhom-pve is one ssh hop; kubectl is local. claude-in-chrome is NOT available here — endpoint-level validation is the standard method. New clean-tree gate before any build (git status --porcelain empty AND HEAD == origin/main), because the CC working tree is now the tree build.sh builds from. Windows knowledge is preserved in per-file "Legacy: Windows workstation" sections and the versioned workspace file documentation/runbooks/workspace-CLAUDE.md (live copy: /mnt/5_hdd/felhom.eu/git/CLAUDE.md).

• 2026-07-17 — CUSTOMER SELF-BIND shipped (hub v0.66.0 + ISO scripts v1.20.0, R-27 slice 1). A customer binds their OWN freshly-installed appliance without the operator. Operator clicks "Send self-bind link" on the customer Setup tab → hub mints a 7-day tokenized capability link → emails it (Hungarian, sibling sender, NOT via the claim engine) → customer opens the public, Hungarian /bind/<token> page (no login — the URL token IS the auth) → enters the console pairing code (shown on the box screen) + their retrieval passphrase → hub stages the bind via the same BindAppliance (provenance customer_selfbind) → the box's ~30 s appliance poll delivers. Viktor's three rulings, verbatim: (a) "only their own visible" → console pairing code, no appliance list ever rendered; (b) first-box entry → operator-sent tokenized capability link over Hungarian email; (c) lockout after 5 failed attempts → token locks, "call support". Wrong code and wrong passphrase = one identical generic failure (no oracle; both factors compared unconditionally); expiry falls back to operator-bind unchanged. THE TRAP (§9.2): one public prefix /bind/, exempted from auth+CSRF at both /login gate sites via a single isPublicBindPath predicate (tight trailing-slash match; ServeMux ..-cleans; handler rejects / in the token) — 4 red-proofs verified red-then-green (lockout→C1, oracle→B, widened-prefix→E, single-active→C4). GC verdict: no appliance-staleness GC exists (applianceStaleAfter is a display badge only) → the 7-day token TTL stands alone, single-active-per-customer, no reaper needed. Controller/agent untouched; R-27b (controller second-box dismissable prompt) DEFERRED, mechanism sketched. Green: full hub build/vet/test (17 ok, +9 self-bind tests) + bash -n. DEPLOYED + LIVE (hub 0.66.0 on k3s, Synced/Healthy). Live through the real ingress: GET /bind/<garbage> → 200 Hungarian expired page (public, logged-out, no-oracle — the 200-not-500 proves selfbind_tokens migrated live); / + /hosts → 302 /login (gate intact); POST /bind/<garbage> no-CSRF → 200 (exempt) while POST /customers/acme/block no-CSRF → 302 /login (exemption is /bind/-only). PENDING (operator-gated, NOT-live): operator "Send self-bind link" + the customer's real two-factor success (hub UI password-gated → CC can't log in) + the new-ISO console banner on a re-imaged box → fold into the R-1 rehearsal. See REPORT.md.

• 2026-07-17 — PBS DR STORAGE VISIBILITY + Offsite tab split + dual gauges shipped (hub v0.65.0 + tenantsync v1.2.0, R-5), LIVE. Scoping correction (Viktor): "restic box" and "PBS box" are NOT two Hetzner Storage Boxes — restic = subaccounts on the shared Hetzner box (Hetzner API, v0.64.0); PBS DR = the felhom-offsite PBS datastore on ep0 (NO Hetzner API; the hub had no read of its fill). Option A (ruled): a read-only usage op on the felhom-tenantsync ep0 forced command (twin of fingerprint, df on the datastore path — no customer_id, no admin token, NO mutation), polled by monitor.PBSDRBoxChecker (OffsiteBoxChecker clone over a usageReader seam; 15-min throttle; three states ok/unavailable/degraded; escalation-only pbsdr_box_fill on the "pbsdr-box" operator scope). /offsite now has Restic + PBS DR tabs (endpoint cards under PBS DR — the endpoint IS the PBS host); the dashboard has two gauges (restic %·ratio, PBS %). Graceful degradation: the hub deploy is INDEPENDENT of the ep0 update — an ep0 ≤ v1.1.0 → ErrUsageUnsupported → the gauge shows "n/a" until updated (no hub redeploy). PROVEN-LIVE (data pipeline, both gauges): ep0 updated to v1.2.0 this session; the hub polled it — PBS-DR box refreshed: 19.1% full (7.1 GB of 37.2 GB). DECISION: PBS thresholds are separate keys (PBSDRBoxFill*) defaulting to restic's 80/90 (independently tunable; Viktor's call for a different pair); no oversubscription for PBS (namespaces, not quotas). OPEN: UI render unit-verified only (hub UI password-gated); the PBS fill alert email is unit + red-proof verified, NOT fired live (datastore nominal at 19%). READ-ONLY against ep0 + Hetzner. See REPORT.md.

• 2026-07-17 — OFFSITE POOL-BOX AGGREGATE shipped (hub v0.64.0, R-5). The operator now sees the shared pool box's real state on the hub: total fill vs capacity, Σ(shared soft quotas)/capacity (the oversubscription ratio), per-customer usage/quota bars, and a box-level operator alert (fill 80/90%, oversub 2.0×). New monitor.OffsiteBoxChecker (OffsiteChecker-sibling): fetch-throttled to one Hetzner GET / 15 min, cached BoxSnapshot, escalation-only + recovery re-arm; two INDEPENDENT signals (fill + oversub). Capacity = storage_box_type.size, usage = stats.size (Phase-0-pinned, bytes); Σ(quota) from the ConfigJSON Descriptor (offsite.ReadDescriptor, new — NEVER the report echo; dedicated + disabled excluded); per-customer USAGE from the report echo. Events scope "pool-box" → operator channel only, no SaveEvent. Web: Offsite-tab panel + compact dashboard tile, reading the cached snapshot only (never fetches). PROVEN-LIVE (data pipeline): the in-cluster checker computed the real box — 0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x. DECISION: thresholds (80/90/2.0×) are Claude's encoding — Viktor's ruling pending; named Alerting.OffsiteBox* keys = the one-line flip. OPEN: the UI render is unit-verified only (hub UI password-gated → CC can't screenshot); the alert emails are unit + red-proof verified but NOT fired live (real pool nominal — a live-fire would email Viktor; offered on demand). READ-ONLY against Hetzner throughout. See REPORT.md.

• 2026-07-17 — SYSTEM-INITIATED IMMEDIACY wired per-plane (hub v0.63.0). The immediate-sync arc covered only OPERATOR-initiated changes; system-initiated desired-state mutations bumped the generation silently, so a freshly onboarded box waited a full agent tick for state the hub had already minted (observed live at slice-C onboarding). Decision: wire the existing, live-proven notifiers into every system site on the CORRECT plane — agent-plane poke.Notifier at PBSDRAutoProvision (the exact lag), ReissuePBSDR (which also lifts the pbsdrheal reconciler's escalation with zero reconciler changes), handlePBSDRReissue, and the two admin api desired-state writers (new nil-safe Poker seam); controller-plane intent.Hub.Bump at reissueOnReenroll. All fire AFTER the store write only. The pbsdrheal Restage path is deliberately poke-free (no generation bump → the agent's 60 s pbsdr ticker is its pickup path — a poke there is a verified no-op); the WG register/delete bumps stay poke-free (undeliverable pre-tunnel / transport removed — the agent fast-tick SECONDARY owns the register leg). The first-hold-baseline race is still OPEN by design (a bump landing during a fresh controller's FIRST wait is recorded as baseline without firing — the ruled fix is carrying intent_gen in the report ACK, its own future task). Unit-tested + red-proofed; live legs fire only on real system events (fold into the rehearsal bind sequence). See REPORT.md + hub/CHANGELOG.md v0.63.0.

• 2026-07-17 — R-21 SLICE C SHIPPED: the universal secret-free ISO (hub v0.62.0 + scripts v1.19.0, LIVE). The distributed ISO carries NO customer secret. A box booted from build-felhom-iso.sh --pairing self-registers as an UNCLAIMED APPLIANCE (POST /api/v1/appliance/register, keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker; token = sha256-stored, 256-bit), the operator BINDS it to a customer on the Hosts page (new "Unclaimed appliances" section, SSH-key fingerprints, host-count display-only), and the hub delivers customer-id + retrieval passphrase ONCE (one-shot poll: 404 no-oracle / 204 unbound / 200 deliver / 410 gone). The bootstrap is one unit, two modes — direct (env-baked, byte-identical, regression-proven zero-appliance-calls) vs pairing (register→poll→write env→fall through to direct). Red-proofs: one-shot delivery + register idempotency (both proven red). Live-verified through the public ingress (register→token, 204, bogus→404). Artifact proven secret-free (baked env = hub URL only; manifest secret-bearing: no). Bind is operator-password-gated → the live boot→bind→day-0 composition (+ physical N100) folds into the supervised rehearsal, which now runs the COMPLETE final flow in one pass (RESET demo → generic ISO → bind → day-0). SSH-key pinning = stored + fingerprints displayed (host-attach on bind deferred, no clean surface). Self-bind page = R-27. R-21 now SHIPPED (A+B+C). See REPORT.md.

• 2026-07-17 — R-21 SLICE B SHIPPED: firmware loader option (scripts v1.18.0). Closes N100 F1 (HIGH) — cheap AMI (AN3PLUS 0.01-class) firmware can't USB-boot the ISO's stock GRUB (relocation 0x0). build-felhom-iso.sh --loader shim|mkimage (default shim, unchanged; profile-settable via FELHOM_LOADER). mkimage = a monolithic grub-mkimage loader built from the ISO's own GRUB (module set from its grub.cfg, embedded search --fs-uuid→configfile), swapped into the ISO9660 tree + efi.img ESP by mkimage-surgery.sh after prepare-iso (payload provably untouched), re-mastered with xorriso preserving BIOS-hybrid + UEFI + GPT-ESP (drops only the irrelevant Apple HFS+/APM). Unsigned → target Secure Boot MUST be OFF. Loud: banner + manifest loader: + -mkimage filename. New profiles/n100.profile; Dockerfile gains grub/mtools. RUNBOOK-B legs (nested VM 311, OVMF): shim boots+installs under SB-enforcing + SeaBIOS; mkimage boots+installs under SB-off; mkimage under SB-enforcing FAILS Access Denied (red-proof); surgery byte-identical payload. Physical N100 boot still pending → supervised rehearsal (an n100-safety match-nothing ISO built + sha-recorded, unbooted). PXE stays a deferred R-21 note. See REPORT.md.

• 2026-07-17 — CUSTOMER RESET SHIPPED: hub v0.61.0 + felhom-tenantsync v1.1.0 (LIVE). The middle lifecycle tier (host delete < RESET < customer Delete). One operator action → pre-first-install: all OPERATIONAL state dies (offsite repo, PBS namespace+backups, DR recipe, one-time secret, claim state, retained escrow custody); IDENTITY + basic config (incl. the offsite tier CHOICE) + provenance + events SURVIVE. Rulings: separate escrow-custody ack; clears claim (fresh code next onboarding); REFUSES while any host row exists; live-counted confirm inventory. Discipline: external teardown FIRST, DB purge LAST, every leg idempotent → partial run re-runs from the top (purge withheld until externals ok). New: store/customer_reset.go (journal+inventory+ack-gated purge), claim.ResetToUnclaimed, offsite.{Deprovision,OffsiteIdentifier,ClearProvisionedDescriptor}, tenantsync.Deprovision + felhom-tenantsync.sh deprovision op (v1.1.0, destroys ns+groups+token, shared user untouched), web/customer_reset.go (GET inventory JSON / POST orchestration) + an amber RESET card distinct from the red Danger-zone Delete. Red-proofs: ack-gate + partial-failure resumability (both proven red). Live-drilled on ep0 (throwaway drill-reset-01 with a real backup): deprovision deleted:true, idempotent re-run deleted:false, all 3 real tenants survived. Hetzner-delete + the password-gated web POST covered by tests (offsite Deprovision mirrors live-proven ReissueCredentials). See REPORT.md.

• 2026-07-17 — HOST-DELETE DEMOTES ESCROW: hub v0.60.1 (LIVE). Closes the v0.60.0 review gap: DeleteHost(deleteEscrow=true) now DEMOTES the current escrow blob into host_escrow_superseded (copy-before-delete, same tx) + spares existing superseded rows — never destroys custody. The customer Danger-zone Delete (DeleteCustomerConfig, which before touched ONLY customer_configs) is now the one true purge point: cascades both escrow tables for all the customer's hosts incl. already-deleted (via F-14 host_deletions). One shared copy routine demoteCurrentEscrowTx (reused by SaveHostEscrow). Checkbox/refusal/Danger-zone wording → demotion. S6b VERDICT: OBSOLETE — re-enroll upserts cleanly (UpsertHost ON CONFLICT DO UPDATE; handleAdminCreateHost no dup refusal) + v0.57.0 arc auto-fires re-issues (ROADMAP R-3 refined). Red-proofs TestDeleteHost_DemotesEscrowNeverDestroys + TestDeleteCustomer_PurgesEscrowCustody. Observation: customer-delete still doesn't touch host rows / the broader offboarding (rehearsal's question). REPORT.md.

• 2026-07-17 — OFFSITE CONTINUITY Part B SHIPPED: hub v0.60.0 (LIVE). Superseded-escrow retention (data-first, Viktor's ruling). SaveHostEscrow's destructive ON CONFLICT overwrite destroyed the old passphrase's only copy (the reinstall-orphan incident); now a new different-sha blob RETAINS the old into the new host_escrow_superseded history table (tx, before overwrite); same-sha = idempotent. ACK/restore read the current row unchanged. escrow_superseded event + operator retained-count on host detail; registered offbox_repo_orphaned/offbox_repo_reset. Red-proof TestSaveHostEscrow_RetainsSuperseded. Pairs with controller v0.142.0 (Part A orphaned-repo guard). Guided old-history recovery = R-26 (not built). See REPORT.md.

• 2026-07-17 — N100 polish (F5 scripts + F6/F7 controller). F5 (host-install v1.17.0): appliance mode auto-sizes the guest (RAM=clamp(host-4096,min 4096,max host-2048,ceil host-1024) + cores=host-1 min 2) when no explicit cap; explicit --memory/--cores win. Harness red-proof (8/16/32 GB). Same commit: the stale "EMPTY by default" operator-key comment fixed (R-16 doc-drift, keys are PINNED). F6/F7 landed in controller v0.141.0 (LIVE on demo; fresh-USB wizard enroll+format now PROVEN-LIVE — see the felhom-controller REPORT). New ROADMAP R-24 (guest resources as desired-state / live resize = F5 follow-on). Operator sidebar #1 (BIOS State-After-G3→Power-On) added to scripts/iso/README.md. See REPORT.md.

• 2026-07-16 — DIRECTION-2a AGENT-PLANE POKE SHIPPED: hub v0.59.0 (LIVE). The agent-plane sibling of the v0.58.0 wait channel: internal/poke — a pinned-host-key SSH sender (wgsync sibling) + fire-and-forget Notifier (PokeHost/PokeAllHosts, nil-safe, detached) that nudges a box in seconds via a CONTENTLESS UDP poke relayed hub → ep0 felhom-poke forced-cmd → wg0-origin → the box WG /32:51822. Wired at applyPBSDR (per-host) + handleSetArtifacts (fleet). ep0 surface (scripts/felhom-poke.sh, non-root, no sudoers; runbook §11) + Secret/agent-poke + manifests/hub.yaml (tag 0.59.0). Deployed live (ArgoCD Synced/Healthy, agent-plane poke enabled); ep0 leg live-validated (poke-fired; non-WG + rm -rf / refused). Source note: the MinAgent-floor save does not bump per-host generation (self-update rides signed-ops) → the fleet poke accelerates the next report, not a delta. Pairs with felhom-agent v0.89.0 (the listener). PROVEN-LIVE 2026-07-17: the operator's real manifest save logged poke: sync-poke delivered to 10.77.0.2 (+ floor HELD for the below-MinAgent box); the demo (now 0.89.0) received a poke in ~31 ms and fired the immediate desired-state cycle (save→tick ≈ ~0.45 s). All three live legs (Scenario 4 A/B train, Scenario 1 R-22 self-heal, Scenario 3 latency) done; capability-map poke + F4 rows upgraded to PROVEN-LIVE. See REPORT.md.

• 2026-07-16 — DIRECTION-2 IMMEDIATE-SYNC SHIPPED: hub v0.58.0. The hub→box "sync now" wait channel (option b of documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md). New internal/intent in-memory per-customer generation notifier (Bump/Wait/Close; coalesce-to-latest; race-closer; in-memory by design → a restart costs one harmless full-state report, never a storm). GET /api/v1/wait long-poll: per-customer only (global key 400; A can't see B), 240 s hold with a 25 s heartbeat newline that defeats nginx's 60 s proxy_read_timeout without any ingress annotation (WriteTimeout lifted per-connection via ResponseController; global timeouts untouched). Every operator-intent web handler bumps AFTER its store write (config CRUD, claim resend, offsite re-issue/freeze, password regen, block/unblock, floors [global bumps all managed], controller log-tail + log-bundle). intentHub.Close() before server.Shutdown. Pairs with controller v0.140.0 (the long-poll client). The immediate-sync arc is COMPLETE for the controller plane (Direction 1 = ctrl v0.139.0, Direction 2 = hub v0.58.0 / ctrl v0.140.0); the agent-plane ep0-poke stays PARKED in the OOB arc. Detail: hub/CHANGELOG.md v0.58.0, REPORT.md.

• 2026-07-16 — REINSTALL ARC SHIPPED: hub v0.57.0 (F2/F3/2.3). The clean-slate reinstall-of-existing-customer path is now first-class (Peti R-1 prerequisite). On host-enroll's mint path (single-bump-per-re-enroll): claim re-issue for a claimed customer (ReissueForReenroll, reset code — the hub never stores the password, so fork B not A), offsite re-issue to the fresh box (ReissueOffsiteForCustomer), and escrow-honesty — re-issuing offsite marks the escrow STALE (MarkEscrowStale), withholds the mismatched restic-hash from auto-confirm, and shows "stale" not "ceremony done" (2.3, red-proofed). Part 0 unblocked the demo live (ACL grant on /storage/felhom-offsite → PBS-DR converged, escrow.pbs_storage_id seeded). Controller + scripts unchanged — both spec premises were contradicted by source (the controller reads escrow prereqs live from the agent; the installer can't know the descriptor-provisioned storage id). F4 root fix is agent-side → ROADMAP R-22 (agent train: the reconcile pre-check 403 must not abort before the root-run apply grants the ACL). Detail: hub/CHANGELOG.md v0.57.0, REPORT.md.

• 2026-07-16 — R-21 PHYSICAL RUN on the demo N100 (tests/VALIDATION-n100-baremetal-2026-07-16.md). demo-felhom host reinstalled clean-slate from a pipeline-built ISO; first-boot chain reached rc-0 on the first attempt on real hardware (closes slice A's operator-gated boundary), serial-filter safety proven on metal (external-HDD canary byte-identical), PBS-DR reconciler self-healed on the reused peer. The obstacle was firmware, not us: the cheap AMI AN3PLUS 0.01 board can't UEFI-boot the ISO's GRUB from USB (relocation 0x0; SB-off + shim-bypass don't help) — worked around live by rebuilding the stick's loader with grub-mkimage from the box's own working GRUB. Findings (to tackle): F1 firmware boot loader (slice-B), F2 claim-re-issue on reinstall, F3 offsite-re-issue, F4 PBS-DR ACL on non-default storage id, F5 guest-RAM configurability, F6 drive-init doesn't mount+attach, F7 back-route. Box healthy, BookStack deployed, demo live.

• 2026-07-16 — R-21 SLICE A SHIPPED: bare-metal Felhom ISO pipeline (scripts/iso/, scripts v1.16.0). A DooPlex pipeline (build-felhom-iso.sh + Dockerfile.assistant) turns the official PVE ISO into a Felhom auto-install ISO whose first-boot stub installs a retry-forever felhom-bootstrap unit that unattended-fetches felhom-host-install.sh from the public felhom.eu/scripts/ channel and runs it (host-install unmodified). Validated on VM 310: build gate + red-proof, disk-filter fail-safe, stub→unit→real-fetch→invoke→retry, resume-decision, exactly-once, no-net retry+recovery — all GREEN. Rulings settled: stub is from-iso (no cert-rotation risk); ordering fully-up; the retry unit owns all network work. §4.4 finding (shapes slice C): the distribution URL carries no secret, but the unattended install needs the customer retrieval passphrase → the slice-A ISO is secret-bearing (supervised/single-use; the bootstrap shreds the env on success). OPEN: host-install rc-0 terminal success (hub-enrolled + guest-provisioned) is operator-gated — a drill customer needs the password-gated create-UI (a live-hub.db insert is unsafe). Slice B = physical N100 RUNBOOK; slice C = claim-pairing to de-secret the distributed ISO. Detail: REPORT.md, documentation/audits/SPIKE-baremetal-iso-2026-07-16.md.

• 2026-07-13 eve — ESCROW CEREMONY WIZARD SHIPPED (agent v0.88.0 + controller v0.127.0; this repo: host-install v1.16.0 + RUNBOOK rewrite). The recovery-code ceremony is now customer-driveable from the controller (/backup/escrow): one-shot R reveal, re-stage-first, Scenario-F stale-blob re-check (fired LIVE on both boxes' hash-less blobs at first ACK; drill's repaired via a real ceremony — hub hash now covers). Ruling F1 (2026-07-13), settled: R transiting the Cloudflare tunnel once at reveal is an accepted risk — same trust class as the claim code/login password; LAN-direct delivery is PARKED. (threat model: runbooks/RUNBOOK-escrow-ceremony.md). host-install v1.16.0 = FELHOM_ESCROW rides the canonical sudoers fetch; hub hostInstallVersion synced (gate green), hub NOT redeployed (const rides the next train). OPEN: supervised full-browser wizard pass (Viktor's drill login; also demo, to clear its legacy stale warning); agent 0.88.0 publish + Day-0 vouch at the next train.

• 2026-07-13 — CLOSING BUNDLE SHIPPED: hub v0.53.0 — the last engineering items on the pre-tester board: F-14 gated auto-Reissue (deletion provenance host_deletions in-tx + token_exists gate → the EXISTING Reissue op + pbsdr_auto_reissue audit event; no record / un-acked → refusal byte-unchanged), dead-host roll-up honesty (customer status = worst of controller-derived + per-host hostStatus on dashboard//configs/detail, cause chip names the host; the Peti 23h-masking shape is the pinned fixture), operator bearer OUT OF GIT (Secret/report-api secretKeyRef non-optional + REPORT_API_KEY env override + scripts/manifest_bearer_gate.py; VALUE ROTATION = supervised step in documentation/runbooks/secrets.md — the git-history copy is dead only after it). Two operator rulings (2026-07-13), recorded verbatim — future sessions must treat these as settled:

  1. F-14: auto-Reissue of offsite credentials is permitted ONLY when the hub's own deletion record shows the tenancy's owning host was removed through the escrow-ack flow — acknowledged destruction is not silent re-keying. No record → manual Re-issue stays the only path.
  2. Roll-up: a customer's status may never look better than its worst expected host. Single staleness definition (hostStatusOf) — no second threshold anywhere. Gate additions: python scripts/manifest_bearer_gate.py after ANY manifests/ change (64-hex bearer shapes; felhom.secret.yaml carve-out = visible KNOWN-BACKLOG line). GOTCHA: the repo .gitignore *secret* pattern silently un-tracks any "secret"-named script — hence the "bearer" name. The controller repo's controller.yaml.example carried the REAL bearer literal — scrubbed (felhom-controller 51c871a).

• 2026-07-13 — POLISH BATCH SHIPPED: hub v0.52.0 (+ controller v0.123.0) — take-two F-15 CLOSED (reset-request response carries the rotated code hash — ACK shape/guarantee — emailed codes work immediately; live 1 s apply + first-try accept on Viktor's exact failure path) and F-16 CLOSED (zero native confirm() in the hub UI — inline_confirm.html partial, Igen/Mégse inline two-step, scripts/hub_confirm_gate.py enforces; danger-zone typed-confirm untouched; live: offsite re-issue completed under browser automation without freezing). Operator resend keeps the ACK lag — flash says so. Take-two §4 dispositions updated (F-15/F-16/zero-toggle obs → SHIPPED). Remaining from take-two: F-14 (needs operator ruling: auto-Reissue re-adopt vs never-silently-re-key) + S5 DR-restore drill; hub bearer key rotation still DUE.

• 2026-07-12 — DR-TIER-BY-DEFAULT BATCH SHIPPED: installer v1.15.0 + hub v0.51.0 (+ agent v0.86.0 in felhom-agent) — closes DRILL-day0-vm F-1/F-2/F-3/F-7/F-8/F-9/F-10 and F-6 (by policy). The five operator decisions (2026-07-12), recorded verbatim — future sessions must treat these as settled:

  1. Every install deploys the full DR plumbing uniformly (pbs-apply wrapper, age, WG enabled, full ACL). Capability is baked; activation is a hub flag.
  2. DR tier defaults ON for new customers. Disabling is the per-customer exception — and it removes the Felhom-side cost entirely (no PBS namespace, no endpoint dependency).
  3. Offsite app backup requires the DR tier ON (fork-4 needs the ceremony; the ceremony needs PBS). Enforced hub-side by extending the drill-proven dependency guard.
  4. Identity-only escrow is formally PARKED — closed by policy, not implemented. Revisit only if a customer class demands lean-tier offsite. Record in the ceremony runbook + CONTEXT so future sessions know it was a decision, not an oversight.
  5. WG is base infrastructure, not a DR component — always-on like cloudflared; it is also the future OOB-management transport. BYO consent: a disclosure line lands in the tester-agreement doc; per-customer hub disable remains possible. Mechanics: installer ships felhom-pbs-apply + age + wg_tunnel.enabled: true (byo wg-off assert RETIRED — consent = typed disclosure ack + tester-agreement line), single VERSION source gated by scripts/hostinstall_gates.py (cross-checks the hub Setup-tab const), default ACL keeps felhom-pbs (doc advice REVERSED — narrowing caused the drill 403). Hub: per-customer dr_tier flag (new default ON; legacy backfilled from descriptor reality, one-time), cascade-not-error semantics + per-stage status on the edit form, WG-registration auto-provision hook (scenario A hands-free), offsite-requires-DR guard (exact F-6 message), host-page capability chips (agent 0.86.0 inactive = neutral; pre-1.15 "binary not found" → migration hint). felhom-pve's latent F-3 state fixed live; Peti's box unreachable (his F-3 check + D/E/G deferred).

• 2026-07-12 — HUB v0.50.0 SHIPPED + LIVE: customer-claim password arc (closes DRILL-day0-vm F-4/F-5). The customer OWNS the dashboard password via a one-time claim code the hub emails to the registered address (bcrypt(code) stored, plaintext only in the send). internal/claim engine (EnsureIssued idempotent Day-0/live-box issue+email, Resend, RequestReset 3/day, MarkClaimed set-only) + store.customer_claims (RotateClaimCode bumps generation + preserves claimed_at) + delivery (config bake web.claim_code_*, report-ACK claim object, POST /api/v1/claim/reset-request) + Setup-tab chip/resend + 3 Hungarian emails. Pairs with controller v0.122.0 (the gate). LIVE-proven on drill box demo-vm-felhom: gate ON via the real edge (/ → 302 claim page, /api → 401); code emailed to the registered address. Deployed felhom-hub:0.50.0 (ArgoCD synced/healthy). Floor raise 0.120→0.122 = operator's supervised LAST step. Observed: the hub issues a claim code for EVERY reporting customer on first report (benign for pre-0.122 boxes — no gate, ACK ignored). Report: documentation/audits/DRILL-day0-vm-2026-07-12.md (F-4/F-5 RESOLVED).

• 2026-07-12 — HUB v0.49.0 SHIPPED: Edit tab merge (edit-a) + scoped auto-refresh + riders (CHANGELOG v0.49.0; commits e740147 → 2e03de1 → 1d94b1a + docs/manifest). The standalone customer edit page merged into the customer page's Edit tab (renamed from Settings): shared config_form_body sub-template + configFormData builder (REUSE.md rows added), Block/Delete relocated to a danger-zone card, header cleaned, GET /configs/{id}/edit → 302 #tab=edit, post-action redirects anchored to their tab, settings→edit legacy-hash alias. Update handler gained server-side required-field validation (error path preserves SUBMITTED values — red-proofed). The 60s auto-refresh is now scoped: fires only on live tabs (data-live-tabs="overview,applications,events,host") and never while a form is dirty; muted (paused) hint. Riders: /style.css?v={{hubVersion}} cache-bust on every template (closes the v0.47.0 max-age=3600 gotcha) + the CLAUDE.md git add -A ban codified. NOTE: the task spec said v0.48.0, but v0.48.0 (app_start_failed) had already shipped — published tags are never re-pointed, so this train is v0.49.0. Out of scope (observed, not done): type-to-confirm for customer delete (still confirm(), relocated as-is); the workspace-root E:\git\CLAUDE.md and other repos' CLAUDE.md files still carry git add -A snippets — flag for their own sessions.

• 2026-07-12 — Day-0 VM DRILL COMPLETE (auto-confirm FIRST LIVE FIRING): full arc proven on a fresh nested-PVE box — appliance Day-0 → floor-at-first-report → ceremony → auto-confirm pending→escrowed in ~7.5 min, zero clicks → offsite backup + restore round-trip. Two HIGH gaps: F-4 no operator password-set path exists (G10 unclosable, dashboards born OPEN — blocks tester gate) and F-6 identity-only ceremony never implemented (no-PBS appliance can't escrow — drill forked to PBS DR tier = full Peti-sequence rehearsal, all green). Installer fresh-box gaps: felhom-pbs-apply not shipped (F-7), age missing (F-10), root-owned guests/ parents (F-3 — check demo for the latent copy), silent root@pam rotation UX (F-8). Runbook fixes committed (day0 A.2 anonymous-fetch; escrow-ceremony identity-only claim CORRECTED + age prereq). Report: documentation/audits/DRILL-day0-vm-2026-07-12.md. Drill VM qm 300 kept (3 snapshots) for re-drills; teardown list in report §9.

• 2026-07-12 — CAMPAIGN-3 Task A SHIPPED: agent v0.85.0 boot/recovery plane + appliance self-heal. Fixes F12 (CRITICAL boot ordering cycle — templates drop network-online, MigrateNetworkUnits repairs installed units), F11/F10/F9 (reassert: fstype-driven classify, reset-failed+rearm, per-unit verdict, + re-arm-on-guest-blind matrix-correction), F2/F1 (zero residue: reset-failed + rmdir), the guest-hook rc255 belt (recover+timeout, wrapper no longer execs), and the appliance node self-heal (internal/selfheal: host networking recovery, gated on deployment_mode — host-install --mode templates it; byo remedy unreachable). 3 sudoers grants added (reset-failed, rmdir, start networking). Live matrix on demo felhom-pve: host reboot ×2 = 0 ordering-cycle lines both boots (F12 dead); appliance self-heal recovered networking on attempt 1; F10 reset-failed+rearm sweep loud; zero residue; guest-reboot heals. NEW finding F13 (HIGH, deferred): active nfs4 under mp8 can fail PVE rbind (rc255); NAS-down-at-boot → apps Created (fix-3). Peti's box (agent 0.81) unaffected until his rollout — carries the F12 coin flip if it has a NAS share (interim drop-in in the agent REPORT). host-install v1.14.0; scripts CHANGELOG. Details: documentation/controller/network-storage-nas.md.

• 2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; host recovered, RCA closed. Unattended chaos campaign 22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/ restore byte-identical/EIO instant/stub badge + deploy-409 live-validated). The NAS automount plane produced 2 CRITICAL + 3 HIGH: F12 CRITICAL (= the overnight host loss, RCA'd next morning): the agent automount template's After/Wants=network-online.target creates a boot ordering cycle — systemd deletes an arbitrary job to break it; boot -1 sacrificed networking (host dark 7h, hardware exonerated), boot 0 sacrificed the automount (NAS dead). Every host boot with an enrolled share = coin flip until the template drops that ordering (_netdev on the .mount suffices). F10 CRITICAL (start-limit never re-armed, once blocked guest start), F11/F9 HIGH (post-start reassert "skip-active" skips the healing automount restart; agent-restart sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots; F7 HIGH (in-place dump write → 0-byte tar replaced last good backup during outage window). Fix-6 numbers: ring horizon 55min idle / 6.5min under load, wiped on every restart. Box left in defined state (6 wave apps healthy, temps removed, NAS baseline-identical). Full ledger/evidence: documentation/audits/CAMPAIGN-3-2026-07-11.md + 180:~/campaign3/.

• 2026-07-11 — HUB v0.47.0 UI REORGANIZATION SHIPPED (CHANGELOG v0.47.0; commits 9f29bf3 → 0daddcd + docs). Five deliverables: (1) .data-table td a:not(.btn) button-contrast fix; (2) customer page = 8 hash tabs (#tab=…, sticky summary strip, Events error badge, no-JS graceful degradation — panels hide only via a JS-added body class); (3) shared host_detail_body sub-template rendered by /hosts/{id} AND the new per-customer Host tab (a list by design; store.ListHostsByCustomer + hostDetailData builder); (4) stale host removal — GET /hosts/{id}/delete-impact (counts/booleans only) + POST /hosts/{id}/delete behind type-to-confirm; ONLINE → 409 always (no override), escrow needs an explicit checkbox (ErrHostEscrowPresent, tx never starts), one-tx cascade incl. the bound wg peer (wgsync's 5-min declarative push converges the endpoint; log bundles die by scope_id == host_id only); (5) /offsite multi-endpoint management UI — all wg_endpoints rows as cards + add/edit/delete with 409 guards (peers-in-subnet pins subnet + delete). Deferral (explicit): peer allocation, the wgsync reconciler push and the desired-state merge stay lowest-endpoint-id (GetWGEndpoint untouched); the future arc is a wg_peers.endpoint_id migration + per-endpoint allocation/reconciler/desired-state. Five red-proofs ran (online gate, escrow ack, bundle scope, endpoint-delete guard, subnet-change guard). Resolved: the earlier ⚠️ about 146d165 sweeping the Part-4 WIP — the sweep caught hosts.go mid-red-proof (escrow ack bypassed); 068427a restored the gate; the arc is now complete and author-reviewed. Stale-doc note: the workspace-root CLAUDE.md's hub deploy wording predates GitOps — deploys are manifest-tag bumps + deliberate ArgoCD sync, never kubectl set image.

• 2026-07-11 — RCA FIXES 1+2+4 SHIPPED + Q1c GREEN (agent v0.84.0 + controller v0.117.0 LIVE on demo). Reboot survival: automatic since agent 0.84.0, live-proven 2026-07-11 21:20 (pct reboot 9201 under operator GO — the hook re-armed + verified nas-media DURING the start task, CWA auto-started healthy, health ok first cycle, no badge). Agent: ReassertNetworkMounts (hook post-start + startup; stop+enable-now on existing sudoers verbs — NO sudoers widening needed; never in periodic paths). Controller: statfs consuming-namespace classifier → probe not_network_fs (stub can never verify), deploy-time stub refusal (409; idle autofs deploys fine), distinct stub badge (wins over unreachable; unreachable line byte-identical), deploy view selects stored HDD_PATH (RCA fix 4). MinAgent unchanged 0.81.0. All red-proofs recorded in repo REPORTs. Feature doc: documentation/controller/network-storage-nas.md §Reboot survival; audit §7 follow-up added. Queued: RCA fixes 3/5/6; Peti publish train now carries 0.84 + 0.117.

• 2026-07-11 — NAS/CWA RCA (diagnostic, no fixes): documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md. The nas-media re-add verified legitimately (new host automounts DO propagate into running guests); a second pct reboot 9201 at 15:39 severed the trigger → guest-side stub; CWA deployed correctly to the NAS (app.yaml + binds + data all nas-media) but is DOWN since 15:39 (docker can't recreate the bind source; perms luckily blocked any system-drive writes; zero data loss). UI "felhom-usb" = display bug (deploy.html selects by IsDefault, never stored HDD_PATH); hub device-ID WARN was CORRECT and the only guard that fired; NAS badge masks because it asks the agent (host ns). 6 prioritized fix tasks in the audit §5. Remediation §6 PERFORMED on operator GO (~20:14–20:20 CEST): automount-unit restart re-planted the trigger in the running guest instantly (fix-1 mechanism live-proven), CWA restarted healthy on the NAS library, uid-1000 write OK, controller health back to status=ok at 20:18. Demo fully healed.

• 2026-07-11 — OBSERVABILITY PASS (hub v0.46.0 LIVE Synced/Healthy; agent v0.83.0 + controller v0.116.1 on demo). Remote pull-only log access to both box components' always-DEBUG capture rings. Hub: log_bundle_requests/log_bundles store (gzip, newest-3, 72 h TTL on the 60 s sweep), the secret gate fail-closed before storing (blocked: possible secret; [REDACTED]/ checksums pass), report-ACK controller_log_requested + heartbeat-envelope log_tail_requested channels (consume-once; pre-0.83 agents stay visibly pending), host-detail Diagnostics section (Request controller/agent logs + View/Download; the hosts read-only invariant amended: these two forms are the only actions). Conventions codified: documentation/runbooks/logging-conventions.md + CLAUDE.md one-liners in all three repos. OPEN: operator clicks both request buttons on hub.felhom.eu/hosts/ to close the live round-trip (UI password-gated; the flow is fully API-test-proven). Hub bearer key rotation still DUE (pre-existing).

• 2026-07-11 — PUBLISH-TRAIN RULES CODIFIED (documentation/runbooks/publish-train-rules.md):

  1. manifest before floor; 2. the manifest screen carries the LIVE DB floor (hub_settings.min_controller_version overrides env, acts immediately) — save the floor field LAST; 3. MinAgent gate — now HUB-ENFORCED PER-BOX (hub v0.45.0, store.ResolveManagedFloor), the manual fleet check retired; 4. box-level backstop = the controller's agent-capability gate (v0.114.0 route probe → v0.115.0 version compare). (Floor-UI separation + per-box gating SHIPPED in hub v0.45.0 — see the backlog-clear entry above; this rules-doc note supersedes the original "hub untouched / option-3 deferred" wording.)

• 2026-07-11 — NAS/coupling backlog CLEARED (hub v0.45.0 LIVE + agent v0.82.0/ctrl v0.115.0 on demo). Composite A–F. Hub v0.45.0 (Synced/Healthy): floor-UI separation + effective-floor source line (DB hub_settings vs env, store.ResolveGlobalFloor) + type-to-confirm w/ live below-floor count; per-box MinAgent conditional floor (store.ResolveManagedFloor HOLDS the controller floor for a box whose hosts.agent_version < the manifest's MinAgent, flags it on the Hosts dashboard) — manual fleet check retired (publish-train-rules.md rule 3). THE one comparator extracted to internal/semver. Agent v0.82.0 = X-Felhom-Agent-Version header (demo-only, Peti stays 0.81 = probe fallback); ctrl v0.115.0 = version-aware Supports + DSM-validated NFS guidance. DSM spike (real DSM 7.2 via virtual-dsm): consumer recipes validated E2E, hardlink caveat closed for Synology (record: documentation/audits/SPIKE-nas-dsm-2026-07-11.md); QNAP caveat stands. Q1c (supervised reboot) FAILED: a NAS automount trigger does NOT survive a guest reboot (empty-dir; agent lacks a network-mount reassert) — fix spec'd at documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md. NOT published (agent 0.82 demo-only). Record: felhom.eu/REPORT.md + each repo's REPORT.

• 2026-07-11 — PUBLISH TRAIN 0.81/0.113 COMPLETE — PETI LIVE on agent 0.81.0 + controller 0.113.0. Record: documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md. Agent 0.81.0 published (sha 914bb3b5…30df0b, live-bytes, anon-verified) + golden 0.113.0 baked/published (sha 66de5e64…82054c, drill VM, virgin-reverted); manifest + signed agent_update (key-id felhom-op-1) by Viktor → Peti 0.79.0→0.81.0 clean (no rollback); floor auto-confirm proof CAPTURED (10:41 0.112.0 → 10:42 0.113.0, anonymous pull proven on HIS box) — KEY LEARNING: the hub UI manifest screen ALSO sets the GLOBAL floor as a DB hub_settings.min_controller_version row that OVERRIDES the env and acts immediately (this is how Peti reached 0.112 and now 0.113 before the GitOps env bump 16390d0); a transient forbidden skew (ctrl 0.113 + agent 0.79, ~10 min) was benign only because zero NAS shares existed — next train: save the floor field LAST. OPEN: Peti journal-group one-liner (E), temp Git Sync creds deletion (G1), hub bearer key rotation now DUE (committed in manifests/hub.yaml + screenshot-exposed this session).

• 2026-07-11 — NAS verify-before-commit SHIPPED (agent v0.81.0 + controller v0.113.0 LIVE on demo; host-install v1.13.0). The spike's production task landed same-day: add = agent verify (retry=0 unit, journal-classified, auto-rollback) → controller uid-1000 write probe (--netprobe re-exec) → register LAST; full rollback on any failure; §3.2 Hungarian error map (nfs_export merged); storage_network.html redesigned (SMB-first, Route-A guidance); orphan rows; systemd-journal group (NO sudoers). Live A–E validated vs an isolated sim NAS, Route A proven in production (alien-uid 1060 server-side). Feature doc: documentation/controller/network-storage-nas.md. NOT published (no Gitea/Day-0 bump; Peti pending with the usermod one-liner). Record: felhom-controller/REPORT.md.

• 2026-07-11 — SPIKE: NAS verify mechanisms + NFS squash matrix — ALL FIVE QUESTIONS ANSWERED, no blocker. Record: documentation/audits/SPIKE-nas-verify-2026-07-11.md. Q1 in-guest access DOES wake the host automount (make-or-break PASS; container bind-mount too); Q2 controller drops to uid 1000 and writes (setpriv proof; SETUID/SETGID caps present); Q3 Route A ACCEPTED-with-caveats (all_squash to ANY server uid = full rw+hardlink for the guest app; guest sees nobody; chown EPERM-clean, chmod persists; June's anonuid=1000 WRONG case pinned = anonuid without all_squash is a no-op for non-root); Q4 error taxonomy captured (NFSv4 makes no-export vs not-permitted INDISTINGUISHABLE; systemd 90 s start-timeout is the real ceiling; retry=0 → production unit: dead-NAS access 91 s→3.8 s; verify must NOT ride the 15 s agentapi timeout → own path/status-poll, ~120 s budget); Q5 plain-user SMB PASS incl. hardlink. Teardown verified to pre-counts. Production TASK is the operator side's next move.

• 2026-07-11 — PBS DR tier SLICE 2 SHIPPED (agent v0.80.0): the apply-bridge — demo ADOPTION-PROVEN live + PUBLISHED; the supervised Peti phase PENDING. Full chain fired on the demo: hub enable (storage-id felhom-offsite) → tenantsync provisioned ns demo-felhom on ep0 → agent ADOPTED the manual entry (tenancy entry-owned, "entry wins"; staged secret asserted UNCONSUMED; DR green; escrow.pbs_storage_id seeded). Laws red-proof-verified: set-only (K destruction guard), secret-on-stdin, verify-pin-before-consume, loud consumed-failed. Part 0 evidence: write path needs the §4b dual-grant exactly (403 Datastore.AllocateSpace → re-assert → token vzdump OK); demo grants restored. Live finding fixed: escrow seed = in-place write (/etc/felhom-agent dir is root-owned; rename impossible for the non-root agent). 0.80.0 published (f2ba62ca…f06d2, == deployed bytes). NEXT = RUNBOOK-peti-pbsdr-2026-07-11.md (Peti box prep: sudoers+wrapper+wg_tunnel enable — no tunnel/OOB on his box today; Viktor opsign 0.80.0; hub enable → FRESH path, K born, degradation clears; ceremony → auto-confirm → first offsite run = the arc closes). Detail: felhom-agent/REPORT.md + CHANGELOG v0.80.0.

• 2026-07-10 — PBS DR tier SLICE 1 SHIPPED+LIVE (hub v0.44.0): ep0 tenantsync surface + hub provisioning. felhom-tenantsync v1.0.0 installed on ep0 (second forced-command key, peersync untouched — hashes asserted; smoke caught + fixed the client {"data":[…]} json wrap; orphan root@pam!spike cleaned). Hub: internal/tenantsync (pinned-SSH twin of wgsync; errors never embed stdout), config-form "PBS DR tier (ep0)" section → WG-peer check (fail-closed) → provision → host_pbs_secrets consume-once (agent fetches POST /api/v1/hosts/{id}/pbs/consume-token — note: /hosts/, not the task's /host/) → pbs_dr descriptor into HOST desired_json + gen bump; idempotent re-save; Re-issue button (F4 twin). Red-proofs: consume-once / fail-closed / idempotency all mutation-proven. NOT live-provisioned — slice 2 (agent bridge) + supervised Peti enable pending. Runbook §10 addendum in offsite-endpoint.md. Commits ce6a566 + 32ca1f6.

• 2026-07-10 — SPIKE: PBS DR tier auto-provisioning SLICE 0 — all three mechanisms SETTLED. Record: documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md. (1) PVE token storage-create impossible (create/modify/delete all check /storage root) → sudoers vector proven (pinned-id alias, cycle green as the agent user, unpinned denied; secret must ride a stdin-wrapper, never sudo argv). (2) ep0 tenancy op-set + one-time-secret-over-SSH-stdout proven live on ep0 (🛑 Viktor-approved; read/write cross-tenant 403s); current peersync channel can't carry it → slice 1 ships a second forced-command surface (felhom-tenantsync). (3) pvesm add pbs --encryption-key autogen births K at PBSEncKeyPath + vzdump lands ciphertext (doc-06 property) + escrow-create finds K. ⚠ pvesm remove DELETES K → re-apply must be set-only. All spike artifacts torn down; flags: orphan root@pam!spike on ep0 scratch; demo's §4b step-3 storage grants absent. Epic slice map in the record §5.

• 2026-07-10 — new-box restore round-trip VALIDATED + old-box archive RETIRED. Record: documentation/audits/VALIDATION-newbox-restore-2026-07-10.md. Both toggled apps restored from the production pool box via the real endpoint — 9/9 sha256-identical (immich 44MB db-dump exact, d78466fcbf…4526b, agreeing with the campaign's independent round). ~10s/app. Viktor's decision: RETIRE the old archive — he deletes u629193-sub1 in the old project's console (the PBS-storage-1 box itself stays for dev). C2 fixed same day as controller v0.110.0 (crash-lock self-heal, live-proven). Offsite arc: build-complete + restore-proven both directions; remaining = the Peti onboarding runbook.

• 2026-07-10 — offsite arc overnight soak (unattended) — PASS + 1 HIGH finding. Record: documentation/audits/CAMPAIGN-offsite-overnight-2026-07-10.md. F4 re-issue proved LIVE (reset invalidates the old password); restore round-trip 9/9 byte-identical new-box↔USB (immich db-dump exact → old-box archive u629193* safe to retire, Viktor's decision); single-flight + crash-restart (zero consume, escrow untouched) PASS. FINDING C2 (HIGH, OPEN): a controller crash MID-PRUNE leaves a stale restic exclusive lock the controller can't self-heal → offbox runs fail until manual restic unlock (fix candidate: ensureOffboxRepo auto-unlocks a self-owned stale lock). Track B (live quota-crossing) + hub-orchestrated Track A blocked — the hub operator config surface is session/CSRF-gated (unattended can't reach it) → Peti runbook. Demo restored to start-state. One expected backup_failed email (the crash test).

• 2026-07-10 — SLICE 4 soft-quota + offsite health SHIPPED+LIVE (hub v0.41.0 + controller v0.109.1) — the offsite arc is BUILD-COMPLETE. Controller: quota_gb rides the descriptor into the target (hash includes it → quota changes re-apply via key-auth-first, no consume); repo_size_bytes from restic stats; pre-run gate (≥100% refuses NEW backups Hungarian-loudly, prune still runs — red-proofed; restore never gated; ≥80% warns); /backups usage bar. Hub: monitor.OffsiteChecker (fill 90/95 of quota + offsite_stale >48h for enabled+escrowed only — red-proofed; nil-safe on old reports) + manual-only Freeze/Unfreeze lever (readonly on the exactly-1 labelled sub-account, SSH preserved). Live: key-auth-first proven against the real box (the deploy's own re-apply); that re-apply exposed

  • fixed v0.109.1 (re-apply must preserve EscrowState + runtime status — it demoted the escrowed demo); demo repaired (manual confirm + run: escrowed, 45 MB / 50 GB, checker healthy 0 events, usage bar renders). Freeze + 100%-refusal NOT live-exercised (STOP; red-proofs cover). NEXT: Peti onboarding runbook, NEW-box restore round-trip → old-box retirement.

• 2026-07-09 — SLICE 3 hub-verified escrow auto-confirm SHIPPED+LIVE (hub v0.40.0 + controller v0.108.0 + agent v0.79.0). Replaces the operator-trust confirm with a verified fact: ceremony hashes the sealed repo password (agent, restic_pw_sha256,omitempty on the escrow upload — contract test moved in lockstep) → hub stores it (additive host_escrow migration, NULL on legacy rows) + serves escrow:{identity_blob_present,restic_pw_sha256,created_at} in the report ACK → controller (report.EscrowAutoConfirmer) flips pending→escrowed ONLY on hash match with its local repo password. Red-proofed core: blob-presence alone never confirms (stale blob = false custody claim); mismatch warns once per hash naming the ceremony; never un-confirms; flip wipes the agent-staged secret. Hashers pinned by the same cross-repo vector. Manual confirm = deprecated fallback (legacy blobs — demo included, hash-NULL verified live; the escrowed demo produced zero confirm activity = Scenario E live). Full happy chain proves out at the next real enable (Peti). NEXT: SLICE 4 soft-quota, Peti runbook.

• 2026-07-09 — offsite hardening bundle SHIPPED+LIVE (hub v0.39.0 + controller v0.107.0 + agent v0.78.0). The e2e's sharp edges closed before slices 3–4: F4 "Re-issue offsite credentials" (explicit operator recovery for consumed-password dead-ends — exactly-1 label scope, fresh one-time secret, ConfigVersion bump; hetznerapi.ResetBoxPassword added for dedicated); F2 host-key scan retry (~60s ladder — first save survives fresh-subaccount DNS lag); F5 submit-disable + in-flight notice; key-auth-first bridge (working installed key → re-pin+reconfigure WITHOUT consuming; fingerprint verify never bypassed; kills the stale-descriptor consume-404 loop); staged-secret wipe (agent DELETE /escrow/stage-secret; controller calls it on every flip to escrowed; the e2e's lingering staged file wiped live through the new endpoint, idempotency proven). All red-proofed. F4 reset NOT exercised live (demo credential untouchable — throwaway customer under supervision when needed). NEXT: SLICE 3 auto-confirm, SLICE 4 soft-quota, Peti runbook.

• 2026-07-09 — offsite provisioning VALIDATED LIVE end-to-end (hub v0.38.1 + controller v0.106.1) — PASS. Full loop through the real pipeline on demo-felhom against the new dedicated-project pool box (611714 / u629488, project 15282031): UI save → sub-account 269008 provisioned → descriptor v6 + one-time password → controller config-refresh (≤15 min) → apply-bridge (verify-pin → consume-once → pinned ssh-copy-id → EscrowState=pending) → run-gate refusal proven → escrow confirm (Option A: the repo password was already escrowed — WriteOffboxSecrets preserves it) → first run: restic repo + 2 snapshots on the new box, key-only. Two live bugs found+fixed mid-run: F1 hub v0.38.1 (applyOffsite on context.WithoutCancel — a re-click canceled the provision between create and password-store, stranding the sub-account) and F3 controller v0.106.1 (ssh-copy-id -s needs ~/.ssh to exist in the container — died AFTER consuming the password). OPEN: F2 (fresh sub-account DNS lags → first-save scan 502s; retry-with-backoff), F4 (hub has no password-reset path — recovery today is delete-subaccount + re-save), F5 (no save spinner → re-click bait). Phase-0 wiring: manifests/hub.yaml now injects HETZNER_TOKEN/HETZNER_POOL_BOX_ID from Secret/storagebox; scoped-token check = list boxes with it (must see ONLY 611714). Old-box offsite kept as frozen archive until a NEW-box restore round-trip. Record: documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md. NEXT: SLICE 3 (auto-confirm), SLICE 4 (soft-quota), F2/F4/F5, Peti onboarding runbook.

• 2026-07-09 — offsite provisioning SLICE 2 (hub v0.38.0 + controller v0.106.0). The controller apply-bridge: on startup it reconciles the hub-served offsite: descriptor into a key-only offbox target (controller/internal/offsiteapply.Bridge) — verify-pin the box host key against the hub-captured host_fingerprint (no blind TOFU) → consume the one-time password (single-use) → sshpass ssh-copy-id -s -f install → configure offbox → EscrowState="pending" → persist a descriptor-hash marker. Idempotent + fail-safe; both red-proofs green. Hub v0.38.0 adds Descriptor.HostFingerprint captured via an x/crypto/ssh keyscan (fail-closed). NOT yet live-applied — supervised end-to-end (hub provisions on the new pool box → controller apply) is the next runbook, gated on the hub's new scoped HETZNER_TOKEN. NEXT: SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota).

• 2026-07-09 — hub offsite provisioning SLICE 1 (hub v0.37.0). The hub can now provision the offsite tier on operator enable: internal/hetznerapi (typed client, base api.hetzner.com/v1, CloudAPI interface + exported Fake, WaitAction), internal/offsite (Provisioner.ProvisionOffsite — idempotent by label felhom-customer=<id>, shared sub-account / dedicated box, generates the transient password, builds the NON-SECRET descriptor, fail-closed), one_time_secrets store (single-use Save/ConsumeOneTimeSecret), POST /offsite/consume-password/{id} (customer-API-key auth, once), config-form Offsite section → applyOffsite (502 + no-save on error) → descriptor in ConfigJSON → ConfigVersion bump. Token from HETZNER_TOKEN (out-of-band); password NEVER in ConfigJSON/logs. Unit-tested vs a faked Cloud API (+ a fail-closed red-proof). NOT yet live-provisioned — awaiting the dedicated-project scoped token (the current token can delete ep0). NEXT: SLICE 2 (controller apply-bridge consuming the descriptor + one-time password), SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota). Deployed via GitOps.

• 2026-07-09 — Hetzner API provisioning spike (offsite tier). Both offsite models are API-provisionable (FIT). #1 correction: the Storage Box API is https://api.hetzner.com/v1, NOT api.hetzner.cloud/v1 (Cloud API 404s for storage boxes); same Bearer token. Proven live on spike-* (torn down): sub-account create/reset/readonly/delete + a billable bx11 box create/rescale(change_type)/delete. Create takes a password (no ssh-key param) → transient-password→ssh-copy-id -s -f→restic-reach→reset_subaccount_password (transient pw invalidated, key survives) = key-only steady state. RepoPath /home/<repo> (chroot). Quota: dedicated=box type/change_type (hard); shared=no native quota → Felhom soft-quota + update_access_settings{readonly} freeze. Idempotency = label_selector (box names not unique). Rate 3600/h. SECURITY: the token is project-wide unscopeable Read+Write — it can also touch ep0 (the prod offsite server) + 3 SSH keys → move customer boxes to a DEDICATED Hetzner project before the hub holds a write token. Record: documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md. NEXT: the hub-provisioning impl.

• 2026-07-09 — fork-4 custody VALIDATED end-to-end live (supervised). The full custody cycle proven on the demo (agent v0.77.0 + controller v0.105.0): enable → controller pushes the repo password to the agent → real escrow ceremony (--selftest=escrow-create, K + restic-pw under a fresh R, uploaded to the hub, staged secret wiped, K byte-unchanged) → atomicity proof: a run skipped before confirm-escrow and proceeded after (2 apps, 2 snapshots) — the gate opens exactly on confirm. DR money shot: the restic password recovered from the production hub blob (host_escrow.identity_blob for demo-felhom-01, via identity-consume with R) is byte-identical to the live password (sha 24c44f49…) and decrypted the real box repo; the real inject-password→restore endpoints reproduced a 45 MB db-dump byte-perfect. Demo offbox left escrowed (un-paused); Viktor holds R out-of-band; no secret in any log; live repo_password unchanged. Record: documentation/audits/VALIDATION-fork4-custody-2026-07-09.md. NEXT: hub-provisioning + soft-quota spec (hub-verified auto-confirm replacing operator-trust; both storage models); per-customer onboarding runbook; the identity-consume OK-message fix (omits the recovered restic field).

• 2026-07-09 — offbox v0.104.0 VALIDATED live against the Storage Box (supervised). Closed the offsite arc: controller :0.104.0@908e4b9 ran the real /backup/offbox/* endpoints against a fresh box sub-account (u629193-sub1, RepoPath /home/felhom-demo-repo). The capstone proof — immich (undeployed, recovery unit only on the USB drive, the exact case v0.103.0 mis-resolved to systemDataPath) → discovery found the USB unit → ok, 1 snapshot; 45 MB db-dump restore sha256-matched. No-silent-success confirmed on the live box: 0/N → error + backup_failed operator alert; partial → ok + Hungarian LastWarning on /backups. Persists across restart. Forks: (1) direct topology + (2) sub-account-per-customer (RepoPath /home/<name>, ssh-copy-id -p 23 -s -f — console SSH-key field still broken, F3) confirmed; (3) PBS/WG tier deferred; (4) custody surface RECORDED, escrow design OPEN — the per-customer restic repo password + SSH key are 0600 in the controller data dir, NOT independently recoverable (ride DR only via the PBS whole-CT) → the load-bearing follow-up before real reliance. KEPT as a standing demo-offsite (€4.06/mo). Record: documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md.

• 2026-07-09 — TASK GL-7 SHIPPED: customer-page passphrase hardening + install-command generator (hub v0.36.0, 844fbfa7; deploy 02c748eb; LIVE, ArgoCD Synced/Healthy) — go-live G7/G9/G10/G11. Two coupled, security-first changes to the operator customer page (customer_unified.html + configs.go). (1) Passphrase hardening: the per-customer retrieval passphrase was cleartext twice (visible #retrieval-pw + baked into the Option-3 debug curl X-Retrieval-Password: header); now masked-by-default with reveal (toggleSecret) + copy (copySecret), value in data-secret (existing model), and the Option-3 command carries a <YOUR-RETRIEVAL-PASSWORD> placeholder — no secret in any copyable command. (Zero-secret-in-DOM fetch = noted future follow-up, NOT scoped.) (2) Install-command generator: the three hard-coded <code> blocks became a client-side builder (vanilla JS, no framework/CDN/network) emitting ONLY real host-install v1.12.0 flags, download-then- run, byo-requires-caps enforced client-side, CustomerID prefilled from the server, JS-off static fallback; the seven dangerous/operator-only flags (--force/--rotate-recovery/--enable-oob/ --remove-golden/--uninstall/--adopt-pool/--rescope-acl) are never offered as controls. Tests TestTemplates_PassphraseHardened + TestTemplates_InstallGenerator (green; passphrase test red-proofs). Tester agreement written (documentation/pilot/PETI-tester-agreement.md): BYO trust boundary, honest limitations (no offsite backup, F2 removable-drive, F7 pool-reassert, F6 :53-free), exit rights, Peti box facts (80c/128 GB → caps 12/32768, sajatfelhom.hu re-point), byo install command, onboarding sequence. GO-LIVE-PACKAGE: GL-7 ✔; G9/G10/G11 → done. One open pilot question (owned by the agreement §6): Peti's local backup TARGET — does his box have a second disk/pool? → sets --acl-storages at onboarding. Deploy gotcha: first build was stale (local commits not pushed before build.sh pulls origin) — pushed then rebuilt. NEXT: Peti day-0 onboarding (operator-run).

• 2026-07-08 — TASK GL-8 SHIPPED: BYO coexistence hardening (host-install v1.12.0, 18a556a5) — the GL-6 findings F3/F6/F1. F3: the populated-host leaf guard fires only when a felhom_guests guest exists (not pct list) → a fresh byo install on a host running only the OWNER's guests needs no --allow-new-leaf; the pin protection for real Felhom guests is unchanged. F6: byo preflight REFUSES a foreign process on :53 with remediation and mutates NOTHING (Viktor's ruling: refuse+ instruct only — the dnsmasq analog of the break-glass no-touch rule; appliance untouched). F1: uninstall purges the config .bak* siblings (a GL-6 residue held a live hub api_key). F6 live-proven on felhom-pve (byo preflight refuses at :53 — the exact GL-6 scenario, read-only die). Harness 36/36 (+GL8-F3/F6/F1 static, GL8-F1b behavioural, GL8-F6 live) + red-proofs RP-F3/F6/F1; GL-2/GL-4 regression green; shellcheck clean. Shipped BEFORE GL-7 so Peti's command needs no workaround. F2/F7 remain open (non-blocking). NEXT: GL-7.

• 2026-07-08 — TASK GL-5 SHIPPED: DR bring-up bind overrides (agent v0.75.0, LIVE + scratch-DR validated) — go-live G8; docs-only here (GO-LIVE-PACKAGE G8/G6/G1 flips + decision log; the code + full report live in felhom-agent b3446213). Guest-loss DR restore WORKS now: the two structural binds (mp8/mp9) are overridden with throwaways at restore and swapped for the REAL binds pre-start (root pct via the new Engine HostRunner seam), displaced unusedN deleted. TWO live-discovered PVE rules beyond the spike: explicit-params restore needs an explicit rootfs AND silently DROPS unlisted mountpoints — the full param set now derives from the archive's embedded config (ExtractArchiveConfig, 200 under the scoped token). Live proof: scratch 9310 from a real 9201 archive — mp0 200G + mp1 50G restored WITH content (7m23s), real binds, zero residue, clean teardown; the same op failed outright on v0.74.0. OPERATOR: publish agent 0.75.0 + bump the manifest to it (vouching 0.74.0 ships broken DR). Follow-ups: restore-test boot-verifies without mp0/mp1 (same PVE rule, reverse direction — candidate reuse of drRestoreOverrides); DR-selftest KeepMAC duplicates a live source's MAC briefly; no host-loss DR runbook exists yet (GL-6/S5 writes it).

• 2026-07-08 — TASK GL-4 SHIPPED: uninstaller gap-closure + operator-key day-0 fold (host-install v1.11.0) — go-live G6 + the G1 key-pin follow-up; awaiting GL-6 for the real (non-dry) teardown + armed-pin install. Uninstall now removes the self-update artifacts (4b4: guarded wrapper, .prev/.new.* slots, rollback unit, limits drop-in — derived from configs/felhom-selfupdate-guarded), unmounts enrolled drives under /mnt/felhom-drives (plain umount ONLY, busy = warn+guidance, root-bind guarded), and ends with a KEPT-vs-WIPED statement (drives/PBS/hub record/escrow/vaulted recovery credential live on; guest-only mode prints the vmid's bind-store drives). Key-pin: OPERATOR_KEY_* constants (EMPTY until the operator ceremony)

  • --operator-pubkey-file (validated at argv, comment=key_id required) → authz.signers written at step 6 per the agent SignerKey schema; no-keys-resolved KEEPS preserved signers (never un-pin a manually-pinned box); verify reports armed/dormant (dormant = WARN). Harness 28/28 on felhom-pve incl. the NEW GL4 H-U full-uninstall DRY transcript vs live 9201; red-proofs RP-1..3 green. GO-LIVE-PACKAGE.md was ABSENT AGAIN (spec said the operator attaches it; not present in the repo or workspace) — G6/G1 status recorded here, doc still pending its operator commit. Key CEREMONY (real keypairs + felhom-pve pin) = operator step.

• 2026-07-07 — RUNBOOK GL-1 EXECUTED: agent 0.74.0 + golden 0.103.0 PUBLISHED — go-live G1 (partial): the published-artifact chain is current. Agent = the LIVE felhom-pve bytes (sha 1ec3f588…76af05, provenance preserved, publish-agent.sh round-trip verified); golden baked clean-room in the drill VM (build-golden.sh v2.0.0 @ ced60ddf, controller 0.103.0 baked, all integrity gates green, sha 8481e8a1…6026e, 583MB) and published HTTP 201. Both artifacts fetch-verified ANONYMOUSLY (Gate 3c finding: generic packages are world-readable → G3 git.token read-only rotation CANNOT break fetches; but the INSTALLER dies on an empty token — script-side requirement, not Gitea's). Drill VM restored to virgin; evidence /mnt/5_hdd/felhom.eu/drill/bake-0.103.0.log on 180 + documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md. OPERATOR NEXT: bump the hub Day-0 manifest to agent 0.74.0 / golden 0.103.0 (the four values are in the execution record) — until then fresh installs land 0.63.0/0.98.3. Also recorded: operator-key pin follow-up (fold into GL-4); GO-LIVE-PACKAGE.md still not in the repo (G1 status recorded here instead).

• 2026-07-07 — TASK GL-2 SHIPPED: --mode appliance|byo install profile (host-install v1.10.0 + harness) — go-live items G2/G4/G5 implemented, awaiting the supervised GL-6 drill for live provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old --mode provision value retired with a specific die, dr stub kept); break-glass gated at its CALL SITE (byo never touches root@pam); byo requires --cores+--memory, refuses --enable-oob/--rotate-recovery/non-9.x-PVE/missing --acl-storages; disclosure + typed-hostname ack; the config write's lan_resolver default flips to OFF in byo (the spec assumed default-off — it was default-ON; appliance unchanged) + post-write asserts refuse lan_resolver/wg_tunnel/oob=true; pool+ACL verify asserts in BOTH modes (R2 lesson); --preflight-only (no state, PASS/FAIL verdict, proven live on felhom-pve). NEW scripts/hostinstall-mode-harness.sh (static C1–C4 + invariants everywhere; PVE tier: C5 + A/B dry transcripts) — 16/16 PASS on felhom-pve incl. C5 live; red-proofs RP-1..3 run→fail→revert. FINDING: demo-felhom's controller.yaml has EMPTY git.username/git.token → any step-5 (re)install for that customer dies until the operator sets real Gitea read creds (relates to the standing scope-down+rotate follow-up). documentation/pilot/GO-LIVE-PACKAGE.md is NOT in the repo and no operator copy was provided — the G2/G4/G5 status flip is recorded here instead; add the doc when the operator supplies it. Next: GL-6 supervised drill (appliance + byo end-to-end, uninstall/re-provision, C7, the ack prompt interactively).

• 2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent v0.72.0) — merged E1+H1. Hub: operator OOB peer (store/wg_operator.go, PUT/GET /admin/wg/operator-peer global key) + oob_peer_ip/oob_operator_ssh_key in the desired-state wireguard block + monitor/host_oob (oob_degraded/recovered). Installer: --enable-oob/install_oob (static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key.

• 2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent v0.71.0) — prereq for felhom-sshd (H1). Hub: store.host_recovery vault (per-host root@pam console password; PUT /hosts/{id}/recovery-credential self-scoped + GET /admin/hosts/{id}/ recovery-credential global-only, secret never logged) + host_mgmtplane monitor raising mgmt_plane_healed. Installer: step_break_glass (generate+set+vault root@pam, --rotate-recovery)

  • install_mgmt_watchdog (the 3 host artifacts, RuntimeDirectory-guarded). Live-validated: agent-independent /run/sshd auto-heal in 30.0s; warning fired end-to-end; break-glass vault→retrieve→PVE-ticket(200). felhom-pve root@pam is now the vaulted value.

• 2026-07-04 — S3 SHIPPED: agent WG tunnel (felhom-agent v0.64.0; docs-only here) — the doc-06 §3.3 handshake is now END-TO-END LIVE on demo: agent keygen → POST /hosts/{id}/wg → desired-state wireguard block → agent-managed wg-quick@wg-felhom → PBS page over 10.77.0.1:8007 in 3 s. wg_tunnel.enabled defaults FALSE agent-side (rollouts to other boxes are no-ops until the production endpoint exists). Revocation semantics completed (§3.5): removed peer → agent disables + keeps its marker + NEVER re-registers; operator re-adds via the pubkey the heartbeat report now carries. Escrow: IdentityBundle.WGPrivateKey auto-injected on new escrow-create runs (pre-S3 blobs can't be retrofitted — honest fallback = fresh-key re-register, /32 kept). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix, deferred non-blocking); S4 = PBS over the tunnel; hub-side S6 = tunnel-health alerting on the new report stanza. Detail: felhom-agent CHANGELOG v0.64.0 + REPORT.

• 2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0) — doc-06 row S2. DECISIONS: the WG assignment is hub-owned state, merged into served desired-state at READ time — stored desired_json stays a pure operator blob; admin PUT with a top-level wireguard key → 400 (no copy-paste clobber); re-key-in-place keeps the /32 (stable tunnel addressing across rotation/DR); one bound peer per host (partial unique index); generation bumps ONLY on real change (register/re-key/adopt/bound-delete — idempotent paths move nothing). POST /hosts/{id}/wg self-scoped per-host key; NEW golden desired-state-wireguard.golden.json = the S3 cross-repo contract (byte-identical agent copy rule). peersync v1.0.1 (strip out of process substitution — the S1 exit-swallow finding; sandbox red-proofed) redeployed to the dev endpoint. Live-proven on the REAL demo-felhom-01 record via the real per-host key: gen 2→3, served block = golden shape, endpoint wg show carried the peer, v0.63.0 agent tolerated the unknown key (fetch clean, guests untouched), then removed → gen 4, all baselines restored. /offsite page unit-proven (live view is operator-password-gated — eyeball after next login). Next: S3 (agent keygen + registration + wg-felhom service + escrow join + CGNAT smoke — risky/supervised).

• 2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1) — the doc-06 roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: hub pushes over SSH to a forced-command felhom-peersync script (pull rejected: weakens immediate revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev felhom-hetzner box, provisioned by the NEW runbook documentation/runbooks/offsite-endpoint.md (WG on 443/udp, subnet 10.77.0.0/24, PBS datastore felhom-offsite at /srv/pbs-felhom, nftables drop-all + wg0-only 8007, ep0.felhom.eu grey-cloud). Hub: wg_endpoints/wg_peers + /32 allocator + /admin/wg/* (global-key, pubkey-in-body) + internal/wgsync (FixedHostKey pin

  • pinned-type HostKeyAlgorithms — live-caught bug: multi-hostkey sshd presented ECDSA vs the ed25519 pin; fixed v0.32.1 + regression test) + full-list declarative reconciler (5-min drift repair). Live: add/remove→wg show, malformed→state-untouched, reboot→persist+converge, client tunnel via ep0:443→PBS login page, public 8007 closed. INCIDENT handled: first WG server key leaked via wg show dump into the session log → rotated immediately (warning now in the runbook). OPERATOR follow-up: fix the AAAA record (currently …7aa1::, must be …7aa1::1). Next: S2 (hub schema host-join + wireguard block on WireDesiredState + box-facing registration + UI).

• 2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06) — documentation/architecture/06-offsite-connectivity.md: records the six settled decisions (plain WG / host-side agent-managed wg-felhom as the agent-managed-unit PILOT / one shared hub-driven endpoint VM running WG+PBS, NO agent on it / hub = source of truth, wireguard block rides WireDesiredState on the existing DesiredGeneration channel / one datastore + per-customer namespaces / PBS ON the VM, relay-through-DooPlex REJECTED as non-scaling), the Day-0 join handshake, robustness set (WG-roaming-not-DynDNS, DNS re-resolve watchdog, MTU 1420, per-/32 isolation, tunnel-health via storage-target model), trust conformance (outbound only; operator-inbound + game-ingress out of scope), the honest ledger (CGNAT still unmeasured — mobile-hotspot smoke closure; peer-sync push-vs-pull = slice-1 design point), and the S1–S6 slice roadmap (MVP = S1 endpoint + S2 hub schema/desired-state + S3 agent WG service, then S4 PBS-over-tunnel; S5 merges with DR-completeness). day0-install backlog line now cites spike + design doc. Docs-only; slices are separate future tasks.

• 2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung) — plain WireGuard, host-side, box-dials-out to a small public endpoint; no Headscale/DERP needed for backup (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway Hetzner: 11.4-min idle hold, PBS API through the tunnel with the exact-cert pin intact (positive + negative), real 2 GiB worst-case backup at 4.26 MiB/s (= the home uplink; tunnel overhead ~5%), UDP 51820 and 443 both pass. P0 honesty: the operator's line is plain-NAT (fixed public IP, One Hungary), NOT CGNAT, and has zero IPv6 — CGNAT confirmation deferred to Peti's VM 110 (second ISP). P7 surprise: kernel WG works inside the unprivileged guest (no TUN needed) — host placement stands on architecture (backups originate host-side; cloudflared precedent), not infeasibility. Evidence + architecture recommendation (endpoint VM shapes, key custody via IdentityBundle escrow, per-peer /32 isolation): documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md. This unblocks the DR-completeness work: next task = production connectivity spec (endpoint VM, agent-managed WG service, hub peer registry, escrow join). Throwaway Hetzner box parked powered-off (secrets shredded); demo-felhom verified residue-free.

• 2026-07-03 — 4th skill: felhom-app-catalog (catalog authoring workflow; points at app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by install_skills.py, 4/4 found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected healthchecks, probe-container naming rule discovered + REUSE row added, UI-applied, controller probe healthy). App-count drift recorded: website says "45+", per-category spans sum 49, real catalog 53 — website fix is a separate task.

• 2026-07-03 — GOLDEN 0.98.3 LIVE + VOUCHED; guide D.1b RETIRED (drill B1+B5 FIXED) — build-golden.sh v2.0.0 (mandatory controller tag + baked bootstrap .path unit); golden baked with controller 0.98.3, clean-room validated (drill VM) through all four scenarios incl. the published-artifact --force-gitea-golden path, published + operator-vouched (Day-0 manifest now: agent 0.63.0 + golden 0.98.3). Fresh installs land CURRENT and self-manage — D.1b is a one-line check; the prior "vouch agent 0.63.0" operator follow-up is DONE. Evidence: documentation/audits/DRILL-golden-098-2026-07-03.md. NEW operator follow-ups: (1) SECURITY — the customer-config git.token is a Gitea admin token with package-WRITE (proven during the bake); scope down + rotate. (2) optionally raise the global controller floor to 0.98.3.

• 2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published) — token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir. Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. OPERATOR: vouch agent 0.63.0 in the Day-0 manifest UI (until then fresh installs fetch 0.62.0).

• 2026-07-03 — GO-LIVE PACKAGE SHIPPED (host-install v1.9.1 + day0-install runbook + clean-room drill) — documentation/runbooks/day0-install.md is the operator Day-0 guide, every command executed verbatim on a throwaway nested PVE 9.2.2 (build server): install → verify → app deploy → uninstall → zero-residue diff → reinstall. v1.9.1 = residue-free uninstall (agent config w/ hub api_key, shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq snippets) + post-provision guest reboot (without it the controller never deploys on slower hardware) + header sync. Evidence + go/no-go for Peti (GO, 3 install-day items): documentation/audits/DRILL-day0-cleanroom-2026-07-03.md. Key OQ-2 finding: golden bakes a pre-floor controller 0.85.1 → fresh installs need the one-time D.1b update (agent restart → update button → 0.98.3); follow-ups recorded: rebuild golden ≥0.86.0 + re-vouch (operator), agent mkdir -p for the guesthook snippet dir, fresh-install local-API 401 (pre-existing), hub has no host-delete path. Drill VM parked virgin-snapshotted on 180 (/mnt/5_hdd/felhom.eu/drill/ — moved off the SSD 2026-07-18).

• 2026-07-03 — A1 CLOSED (host-install v1.9.0 + agent v0.62.0) — Pool.Audit added to FelhomAgentGuest; the agent's stale-lock reaper now intersects its scan with GET /pools/felhom membership (fail-safe skip on read failure). Spike: documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md. Deploy order on any box: rescope ACL first, then the agent. STILL PENDING (operator): rotate + de-git the 3 committed secrets in manifests/felhom.secret.yaml (spike appendix; REUSE §2 errata added).

• 2026-07-03 — hub v0.31.0 LIVE (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy; live probe: critical event stored + operator email delivered).

• 2026-07-03 — Felhom skills exist: skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}, installed via scripts/install_skills.py — junction mode (repo edits live immediately; no re-install needed after editing). Fresh-session discovery verified.

• 2026-07-03 — CLAUDE.md refreshed (all four repos): no version-pinned state in CLAUDE.md — current state lives here / CHANGELOG / REUSE.md.

• 2026-07-03 — REUSE.md exists at the repo root (hub helpers / website conventions / gate scripts / manifest patterns, code-verified); maintenance rule active: update it in the same commit that changes a shared helper. scripts/reuse_refs_check.py machine-checks its citations across all four felhom repos.