felhom.eu/CONTEXT.md

CONTEXT — felhom.eu working state

Created with the REUSE.md rollout (2026-07-03). Authoritative history: hub/CHANGELOG.md (hub), website/CHANGELOG.md, scripts/CHANGELOG.md; end-of-task detail in REPORT.md.

documentation/backlog/OPEN-ITEMS.md is the single source of truth for open work (2026-07-27). Read it first; ROADMAP.md holds the full history and reasoning behind each item. REPORT.md is per-session and overwritten — nothing durable may live only there.

This file is NOT STATUS.md, and the two must not be consolidated (2026-08-02). CONTEXT.md is technical working state written for Claude Code: rulings with their mechanisms, file paths, version numbers, the reasoning a session needs before it touches code. Root STATUS.md is written for the operator — technical, but not in the code and not a reader of the audits — in plain language, one screen, no identifiers in the prose. Same subjects, different readers; merging them would make one of the two audiences stop reading. STATUS.md is also a view of OPEN-ITEMS.md and holds nothing of its own; this file does hold its own content, namely the standing rulings below.

Standing rulings

S-1 — N.5 gains a third leg: architecture docs are same-session coupled (2026-07-26, R-81). Any task that changes an architectural contract — tiers, targets, cadences, trust boundaries — updates the owning documentation/architecture/*.md in the same session, under exactly the same coupling rule that already binds the capability map and the ROADMAP. Origin: R-80/R-81 found 07-backup-architecture.md still describing a backup model that the shipped code does not implement (single target, single cadence), while being cited as authoritative. A stale architecture doc is worse than a missing one, because it is trusted.

S-2 — architecture docs carry an honest status header (2026-07-26, R-81). Every documentation/architecture/*.md opens with the version it was verified against and the date. A doc more than a few trains behind its subject is marked STALE in that header, so a reader meets the warning before the content, not after acting on it. Origin: 07-backup-architecture.md — DRAFT since 2026-07-14, verified against controller v0.132.0, now 41 versions stale (live v0.173.0), and cited as authoritative throughout the R-80 diagnostic. Ratifying or retiring it is → R-83.

S-3 — the recovery model: six decisions, 2026-07-28. Taken in an architecture discussion and expressed in the 07-backup-architecture.md full rewrite (which replaces the 2026-07-14 DRAFT entirely — that doc was verified against controller v0.132.0, 51 versions stale, while being cited as authoritative). They are decisions, not observations; the rewrite labels every statement [DESIGN] or [FACT] so a reader can tell which is which.

• D1 — two lanes, different owners. Customer, unassisted: files and app data, through the „Visszaállítás" surfaces. Operator: LXC guest and host recovery. This is a deliberate product decision, not a limitation to apologise for.
• D2 — encryption follows the boundary, not the tier. Data on the customer's own drives is plaintext, deliberately: someone who can take the second drive can take the first, local encryption adds a key-loss path that turns a working backup into a brick, and it would break FileBrowser and SMB browsing, which is a feature. Data leaving the premises is encrypted (restic, PBS) — the provider must not be able to read it.
• D3 — recovery is a chain, not a set of tiers. Every app-tier restore is conditioned on the whole-guest tier: Tier-1 needs the guest's secrets (restore_unit.go:17-22, "Regenerate NOTHING"), and Tier-2/Tier-3 need the app still deployed. 3-2-1 describes copies; it does not describe recovery. This is the reason the rewrite exists.
• D4 — three-part model. Recipe (scaffolding, hub) · Escrow (identity key + restic repo password, hub, R-wrapped) · Bytes (the tiers). Losing one is a different problem from losing another, and the failure→recovery matrix is organised around that.
• D5 — R's role narrows. CONDITIONAL, and currently BLOCKED. The target is app secrets travelling with the LOCAL recovery unit, so Tier-1/Tier-2 restore work without the guest and without R, leaving R needed for offsite recovery and host identity only. Not adopted: the premise (the backup tree unreachable from every browsing/download surface) was tested and failed — see R-108. Every other surface verified clean; the exposure is FileBrowser's root-bind of network shares combined with the absence of an IsNetwork() filter on the deploy and per-app-migrate paths.
• D6 — the trust model, stated honestly. The operator holds root SSH on every box; "the operator cannot read customer data" was never the actual security property and no document may imply it. What R does provide, and must keep providing: the hub alone is not enough — a compromised hub yields blobs nobody can open, provided the operator's key is never stored in the hub. That proviso is why escrow custody is an open decision (07 §11-A).

S-4 — the hub session password alone now unlocks console root on every managed box (2026-07-31, hub v0.84.0). Retrieving a host's vaulted break-glass root@pam credential previously required the global operator API key, a secret distinct from the hub login and kept out-of-band. The Console access card on the host page puts that retrieval behind the hub session (POST /hosts/{id}/reveal-recovery-credential, CSRF-gated, audited), so the hub login password is now sufficient. Accepted deliberately, and the reasoning is the ruling: the hub is single-operator and geo-fenced to Hungary, and it already stores these passwords in plaintext at rest — so the DB and the login were already jointly sufficient, and this is a convenience trade rather than a new exposure. It is a real trade, not a free one, and it is recorded here rather than only in a changelog. Two things follow. (a) The global-key path (api/handler.go handleAdminGetRecoveryCredential) stays untouched and must acquire no dependency on the session layer — it is the route for when the hub UI itself is what is broken, and coupling the two would delete the independence that makes it a fallback. (b) The plaintext-at-rest half is now the named open item R-133: every hub DB backup is a fleet-wide console-credential dump, and the fix shape is envelope encryption under a KEK held outside the DB, the way escrow blobs are already opaque.

Five decisions were deliberately left open for the operator and are recorded, unanswered, in 07 §11: escrow custody · lost-R policy · per-scenario RTO/RPO targets (none have ever been stated) · Hetzner as a single failure domain · and local vzdump sharing a physical device with the guest it backs up. Gaps minted the same session: R-102 … R-108.

S-5 — four operator decisions taken in discussion on 2026-08-02, recorded before anything is built. They existed only in conversation, which is the condition the standing rules were written against. Labels are the ones used in the discussion (D-a … D-d) and are deliberately kept distinct from S-3's D1…D6. None of a, b or c is implemented by the session that recorded them; their work is carried as backlog rows (R-165, R-166, R-167), not as prose here.

• D-a — the dedicated backup partition is MERGED AWAY, not resized. The appliance's system drive carries the OS, a 50 G docker area (mp0) and a separate 20 G area holding local recovery units (mp1; demo-hp's guest 9201 ships exactly that). The decision is to merge the backup area into the large data area so local backups share space with app data and the ceiling stops existing — deliberately not to pick a larger number, because a larger number is the same wall further away. Two conditions, recorded with the decision, not as commentary: (1) it changes the disk layout, so it must land before any external install — after that, every deployed box is a migration; (2) it removes a wall that currently fails safely, so the storage monitoring of D-c must ship with it, never after. Closes the sizing question in R-163, which is re-framed rather than answered with a ratio and is not closed until the merge lands. Work: R-165.

• D-b — app state gets a desired/observed model with its own store. Three kinds of information, deliberately separated: desired (running, or stopped because the customer said so) stays in app.yaml with that app's other settings; in-flight operations ("a backup started and has not reported finishing") are persisted, written before the operation starts and after it ends, so an interruption is visible after a reboot; observed (unhealthy, restarting, drive gone) is not persisted and is rebuilt by looking. The last two live in a state store separate from app.yaml, so frequent writes never touch a file holding deploy settings and secrets. Binding safety rule, verbatim: losing the state store must never cause an app to be deleted, restarted wrongly, or reported healthy when it is not — the worst acceptable outcome is re-running a backup that already ran. The controller must also check every container of an app, so a partly-dead app cannot read as healthy (F-CRIT-1's shape). Two things to establish before this is specced, not now: whether the crash-safe journal already in the backup code covers the in-flight case — it may already exist and merely be unwired, which has been the answer four times — and whether the existing SQLite store is reachable from this code path. Work: R-166.

• D-c — storage monitoring and backup alerts. Two pieces: a warning to the customer as a drive fills, before anything fails; and a failure alert to the hub/operator when any backup fails for any reason, lack of space included. Prerequisite for D-a — see that decision's condition (2). Subsumes and widens R-158, which is the same alert one tier down; R-158 stays as the named seam and is not duplicated. Work: R-167.

• D-d — which machines are protected: DooPlex and Peti's box, and nothing else. DooPlex because it holds Gitea, the hub, the backups and the registry — everything else rebuilds from it. Peti's box because there is a real person behind it. Every other box, both demo boxes included, is disposable and may be broken or reinstalled freely. The earlier caution was costing sessions and pushing work onto DooPlex — the one machine that should never host it — so the ruling is a correction, not a relaxation. documentation/runbooks/target-selection.md updated the same session (2026-08-02); no backlog row, the decision is the change.

• 2026-07-27 — R-85: the DR tier is restore-tested UNATTENDED, and its failure is HEARD. agent v0.104.0 + hub v0.77.0. R-82 proved a PBS restore round-trip by hand; three defects kept the unattended path from covering it. (a) buildRestoreTestScheduler built ONE runner on cfg.Backup.BackupTarget(), so the offsite tier's archives were never candidates — the DR tier could not be restore-tested unattended AT ALL. (b) the Spec was an immediately-invoked function evaluated once at daemon start, so nothing tier-varying was expressible and any config change stayed stale until restart. (c) a failed restore-test was a [WARN] line with no event, no notification, no gauge — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) was the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. Ruling (operator, 2026-07-26): oldest-first (Option 1) — never-proven sorts first, no cadence knob; the per-tier last-proven timestamp is reported so the achieved interval can be measured before anyone decides a knob is needed. Shipped: per-run SpecBuilder; PickRestoreCandidateOn (a tier with no archive NO-OPS); persisted RestoreTestState (survives restart; credit only on SUCCESS); backup.InFlight, the host-wide one-heavy-op gate shared with the backup path (a LINK concern — an offsite restore pulls multi-GB over the tunnel a backup pushes one); hub restore_test_failed / restore_test_stale, kept DISTINCT and anchored on R-81, operator-tier only. Also fixed a time bomb I planted in Slice C: a deadline test hard-coded the incident timestamp 2026-07-18T18:31:06Z while comparing against the REAL clock — harmless under one 26h threshold, but once the offsite tier got an 8-day limit it passed all day and began failing at exactly 18:31 UTC, 8 days later. A test that passes at commit time and fails hours later is worse than one that fails immediately. NOT yet proven: rotation has not been observed selecting both tiers across consecutive UNATTENDED cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). Spec: documentation/backlog/SPEC-r85-phase4-5-2026-07-26.md.

• 2026-07-26 — R-82 SHIPPED: the backup target split (local daily + offsite weekly). agent v0.97→v0.102.0, controller v0.174→v0.175.0, hub v0.76.0, host-install 1.20.0. BackupTarget() returned ONE string and BackupCadence() ONE 24h window, so "local daily AND PBS weekly" was not expressible — which is why the DR tier was applied since 07-21 with one snapshot on demo-felhom and zero, ever on demo-hp. Now: backup_targets[] per-tier cadence+retention; ONE quiesce window for both due tiers (never two app outages for one night); per-tier hub thresholds (host 26h / offsite 8d); fresh-install default; an unprovisioned tier DEFERS. Operator rulings: 2-week offsite retention, first backup runs as long as it needs, one backup at a time per guest, drill box dropped from the rollout. RETENTION IS A COMMERCIAL ATTRIBUTE — the hub decides, ep0 executes (operator ruling 2026-07-27, R-89). A paid tier may buy longer retention, so the policy belongs with customer config on the hub, never in ep0's PBS config and never in a box's config. Execution stays server-side: a reconciler writes a PBS prune job and PBS's own scheduler runs it, so hub downtime leaves the last-known policy running rather than silently stopping retention. Box tokens stay write-only (DatastoreBackup) — never widen a grant to fix a prune error: a compromised box must not be able to delete its own offsite backups, which is the scenario offsite DR exists to survive. Increment 1 shipped 2026-07-27 (boxes stop attempting prune via keep_last: 0; per-namespace prune jobs on ep0, daily 03:30 UTC) — runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md. This closed a live false-negative: every demo-hp PBS backup since 07-26 reported job errors while the data landed correctly, because DatastoreBackup carries no Datastore.Prune. Four defects found by RUNNING it, not reviewing it — a 30-min wait bound against a 41-min backup (the agent recorded success:false while the backup was still going); the restore tier read from the configured target instead of the archive (a silent regression of the S4.1 fix — the mechanism was never removed, its INPUT changed when local_backup_target was retargeted to local); a leaked scratch keeping onboot:1; and a tier firing at a not-yet-provisioned storage. A correction I had to make mid-arc: I reported that the restore-test would boot a network-conflicting clone and break the control plane. It does not — step 2 link-downs every NIC before boot, and it is unit-tested. I read a config artifact and inferred behaviour without reading the code path that consumes it, then escalated before finishing the check. NOT done: the offsite tier is never AUTOMATICALLY restore-tested (the scheduled test only ever selects the primary tier); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence. The installer-default fleet flip waits on a full weekly cycle. R-84 SHIPPED (agent v0.103.0) — the cold in-memory store no longer causes a redundant backup: the due-check asks the STORAGE (ground truth) instead of persisting state, so a pruned archive correctly stops counting. Proven live on both boxes with the store cold. demo-hp's FIRST EVER offsite backup landed (4.25 GB) — the R-82 finding closed on the box where it was worst. Controller v0.175.0 deployed to both boxes. Evidence: audits/SPIKE-r82-phase0-2026-07-26.md, REPORT.md.

• 2026-07-26 — hub v0.75.0: R-81 SHIPPED — "no signal" is not "bad signal". Third instance of one class (hub v0.12.0 expected_backup_missed daily-for-everyone; hub v0.73.0 offsite_stale minutes after a healthy repair; now this), fixed as a class, with the invariant written at the head of assessBackupFreshness naming all three. assessBackupFreshness returns OK/UNKNOWN/MISSED instead of missed bool; absence is UNKNOWN until it outlives an anchored window. The anchor was free — Phase-0 probe found the hub retains 90 days of host-reports, so GetHostReportsSince + newestBackupEvidence (7-day bounded lookback, early-exit on fresh evidence) answer "when did I last SEE evidence of a backup?" rather than "what does the latest report say?"; absence is graded from first contact (GetFirstHostReportAt) reusing the existing 26 h threshold as the grace — no new knob, no agent change, no new persisted state. Rulings: absence is never silent (a genuinely dead box must still alarm — that is the half the naive fix breaks, and TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms is what makes the suppression safe); reason strings stay distinct (the whole 07-26 diagnosis turned on reading the exact string); deferrals are logged so quiet ≠ not-checked. Tests 493→508; red-proofs A/B/C observed — A reproduces newest backup is 176h0m0s old verbatim the mail demo-felhom sent. Replayed against the real 03:00 reports (600/417/77 rows): all three now silent. backupStaleAfter deliberately untouched — ⚠️ it will alarm on a healthy WEEKLY PBS snapshot, landmine recorded, owned by R-82. The real finding behind R-80 stands: the PBS/offsite-DR tier has no schedule at all → R-82. Cause-fix (persist the agent store) → R-84. Evidence: documentation/audits/DIAG-backup-missed-2026-07-26.md, hub/CHANGELOG.md v0.75.0.

• 2026-07-23 — hub v0.72.0 + controller v0.161.0: R-70 SHIPPED + R-71(c) PARTIAL (the offsite last mile is visible; burned credentials self-heal). One detector (offsite.DeliveryStateFor — secret-row timestamps × report offsite-presence), four consumers: operator card (static "delivered once" copy DELETED; state+age, amber past 30 min, stale-staged info line), offsite_delivery_stuck warning (≥1 h, 24 h durable cooldown via LastEventAt on the events table — restart-proof), R-71c self-heal through the EXISTING Re-issue path (monitor.OffsiteReissuer ← webServer, armed only with a provisioner; consumed ≥1 h + ≥4 offbox-less reports + zero offbox evidence; one restage/24 h; every firing = a warning event), and the controller's truthful empty-state banner (v0.161.0, both boxes). Rulings: applied wins (stale-staged is a flag, never a downgrade); cooldowns durable via events table; both new events operator-only (no customerMessages, not in allowedEventTypes — pbsdr_* precedent); R-39(a) guard in the CALLER — the heal re-reads the secret row at act time and refuses over an unconsumed row; the store's clobber semantics stay (Re-issue depends on supersede). Clobber red-proof observed (reissue calls = 1 + row clobbered with the guard removed). Self-heal + banner ship unit-proven, NOT live-fired (no broken box; F9 rule) — arms on the next natural occurrence. Live: checker silent on the healthy fleet; demo-felhom's stale-staged specimen intact (its cleanup is the operator's, post-validation). R-71(a) day-0 ordering OPEN → next spec. Evidence: REPORT.md (2026-07-23).

• 2026-07-23 — ISO train v1.25.0 + hub v0.73.0 (four rulings + rider). host-install 1.17.0→1.18.0, build-felhom-iso 1.24.0→1.25.0, felhom-bootstrap R-63, hub v0.73.0. Done + live-verified: (1) F9 belt everywhere — host-install installs the H1 OOB belt by default in appliance mode (--no-oob opts out; byo still refuses), oob.enabled written so the agent activates it; belt LIVE on both boxes, login PROVEN on felhom-pve (felhom-op@demo-felhom); the orphaned operator identity was re-anchored to the operator's real machine (WG peer + SSH key re-registered, ep0 forward rules persisted). (2) apt — align_apt_repos (appliance-only) switches enterprise→no-subscription; LIVE-aligned both boxes (apt-get update clean). (3) R-63 — Lat2 console font before first paint. (4) R-71 build-gate — assert_golden_ge_floor in build-felhom-iso + rule 5; red-proofed. (6) root vault — felhom-pve root rotated+vaulted+ verified over :22 (G1 parity). (7) hub v0.73.0 — offsite_stale never-ran branch ANCHORED (applied-only + consumed_at/escrow anchor; one-state-one-owner with offsite_delivery_stuck), shipped+deployed, red-proofed. CRITICAL live finding: golden 0.153.0 < floor 0.156.0 in production now — the R-71 gap is real; the gate correctly blocks a v1.25.0 build. golden republished at 0.161.0 + vouched (gap CLOSED), v1.25.0 ISO built (gate live-passed). Nested drill aborted on the N100 (install crawled — the N100 was contending with live guest 9201; KVM-confirmed, not memory) → cleaned up (9201 untouched); operator steer: future drills on the HP. R-63 font code-verified; the live ő/ű console capture rides the next HP drill (belt/apt already proven live on both boxes; no-floor-update is structurally guaranteed post-vouch). Evidence: REPORT.md, scripts/CHANGELOG.md, operations/nodes.md.

• 2026-07-23 — F10 offsite leg RESOLVED on demo-hp (diagnostic + designed-path repair + tier proof; no code changed). The hub said "Provisioned" while the box said „Még nincs beállítva" — root cause: the day-0 managed update (0.153.0→0.156.0, 07-21 16:28) killed the offsite apply-bridge ~35 s after it consumed the one-time password, before key-install/persist; the box then hit the consume-404 dead-end at every start, silently, for 2 days. Repair = operator Re-issue click (converged in 8 s), escrow ceremony via the real wizard (auto-confirm 3 s), then the full proof: paperless-ngx toggled into scope, first offsite run (repo initialized on sub3, 79.8 MB, snapshots=1), probe deleted → mode=full restore (snapshot 2bf7f2e1) → place merged 1 file missing-only → md5-identical. Minted R-70 (last-mile invisible on both surfaces; hub never reads one_time_secrets.consumed_at) and R-71 [P1] (the race recurs on every fresh onboarding whose ISO floor lags the managed floor). Record: documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md. F10's PBS-DR half stays open (F13 + ceremony R-moment).

• 2026-07-22 — hub v0.71.0: the notification train (audit F11+F12+F14-light). Four rulings now standing: (1) recovery pairing — *_recovered notifies the operator always and the customer iff the customer was mailed the paired stale/down (evidence = notification_log customer- channel sent rows via store.LastCustomerSentAt; enabled_events deliberately ignored for recovery; ties → no mail). Severity semantics FROZEN — recovery stays info, routed by an explicit eventType branch before the severity gate. (2) seed-at-claim — MarkClaimed seeds customer_notifications from the registered email, INSERT-if-absent only (never touches an edited row; empty email = no-op; seed failure never fails the claim); default set is critical-only: node_down, backup_failed, disk_critical, host_disk_critical, storage_fill_critical, offbox_repo_orphaned. (3) empty-email no-clobber — a prefs push with empty email preserves a stored address (hub-side belt; controller 0.160.0 already guards its two push legs). (4) priority headers — error/critical mails carry X-Priority: 1 + Importance: high (Resend headers, live-probed); the test event now also mails the operator with those headers (one click proves both channels + rendering). Latent nil-deref fixed: sendTestEmail panicked on a customer with no prefs row. NOT yet live: a natural *_recovered mail (next real staleness cycle / reboot drill — never fabricate one by blocking reports) and seed-at-claim on a real claim (Peti's Friday reinstall is the natural candidate).

• 2026-07-22 — hub v0.70.1: the ghost customer's Delete button exists now. The v0.70.0 ghost-delete path was fully implemented and fully unreachable — the fourth inert-seam defect, this time a TEMPLATE GATE: the Danger-zone card (and the customerDeleteOpen script) sat inside {{if .HasConfig}}, so a ghost rendered no Delete button (found live on demo-vm-felhom). Fix: Deletable page flag = the exact negation of the delete preview's 404 predicate; RESET stays HasConfig-gated, Block/Unblock config-only. Render tests per branch + 2 red-proofs. The seam-wiring rule now covers template gates (see CLAUDE.md) — handler tests that POST directly prove nothing about reachability. Operator live leg PENDING: the demo-vm-felhom ghost delete click (doubles as v0.70.0+v0.70.1 live validation). Same day: **agent 0.93.0 published

  • deployed fleet-wide** (pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md), R-17 CLOSED (u629193-sub1 deleted at the console).

• 2026-07-21 — THE FLEET IS TWO HOSTS. A second Proxmox node exists: demo-hp-bb76ea, an HP t740 Thin Client (Ryzen V1756B, 30 GiB, PVE node felhom-host, customer demo-hp), installed from the armed universal ISO the same day. Both hosts run agent 0.92.1; the N100 is demo-felhom-8363b5. Full inventory: documentation/operations/nodes.md — read it before touching the HP, because two things there are load-bearing: the box's 1TB Toshiba NVMe is present and UNENROLLED (still carrying its old NTFS partition; a future Tárhely candidate that must join through the storage flow, never the installer), and there is no operator SSH key baked on it, so access is the hub-vaulted G1 break-glass password (R-61 is the fix for that lockout). Both nodes are at the VACATION site and travel home ~2026-08-02. demo-hp joined the tailnet as 100.76.96.79 — an operator-lab exception, not product shape; real customer boxes never get tailscale. Second-hardware proof, worth keeping: the pairing/day-0 chain ran end to end on virgin hardware it had never seen, the shim loader booted with Secure Boot ENABLED (so SB-off was an N100-firmware workaround, not a Felhom requirement), and the box self-lifted the floor 0.153.0 → 0.156.0 during day-0. The install was not clean, and the failures are filed: R-59 (no DHCP on the 4-port NIC → the installer baked a static fallback instead of aborting), R-60 (first-boot NIC sweep self-heal), R-61 (baked root password unknowable).

• 2026-07-19 — N100 MADE LOCATION-INDEPENDENT via Tailscale. felhom-pve (the N100) joined the tailnet as a host package (100.70.170.35, expiry disabled); DooPlex already advertised 192.168.0.0/24 via its GitOps k3s admin-system/tailscale pod (100.107.87.53). ssh felhom-pve now targets the tailnet IP (direct-over-LAN at home, tunnel when away); felhom-pve-lan = 192.168.0.162 fallback. --accept-dns=false on the host (pi-hole split-horizon preserved). The host was converted static→DHCP (reservation MAC 68:1d:ef:5d:a6:64→.162 keeps home behaviour). Measured spike: --accept-routes on the N100 while on 192.168.0.0/24 hairpins the local subnet (table 52 overrides main) → kept OFF, it's a travel-only opt-in. PBS is offsite (its own WireGuard tunnel to Hetzner, not the home LAN) so it needs none of this. Recorded, NOT fixed: the controller↔agent plane is hard-pinned to 192.168.0.162 (felhom-agent listen_addr + guest bootstrap.json), so the box is reachable + backs up off-LAN but its local operator plane is not yet portable. Full doc: documentation/operations/tailscale.md. No code changed.

• 2026-07-19 — CC WORKFLOW MIGRATED TO DOOPLEX. Claude Code now runs natively on DooPlex (192.168.0.180, Debian 13, kisfenyo) in a tmux session, working directly in /mnt/5_hdd/felhom.eu/git/<repo>. DooPlex is now the canonical CC environment; the Windows workstation trees are stale as of felhom.eu ee48a48288, felhom-agent d81fddc7b1, felhom-controller ac7323dc9a, app-catalog-felhom.eu a14c97db65 (unchanged — it had no environment-specific content). Builds are local commands; felhom-pve is one ssh hop; kubectl is local. claude-in-chrome is NOT available here — endpoint-level validation is the standard method. New clean-tree gate before any build (git status --porcelain empty AND HEAD == origin/main), because the CC working tree is now the tree build.sh builds from. Windows knowledge is preserved in per-file "Legacy: Windows workstation" sections and the versioned workspace file documentation/runbooks/workspace-CLAUDE.md (live copy: /mnt/5_hdd/felhom.eu/git/CLAUDE.md).

• 2026-07-17 — CUSTOMER SELF-BIND shipped (hub v0.66.0 + ISO scripts v1.20.0, R-27 slice 1). A customer binds their OWN freshly-installed appliance without the operator. Operator clicks "Send self-bind link" on the customer Setup tab → hub mints a 7-day tokenized capability link → emails it (Hungarian, sibling sender, NOT via the claim engine) → customer opens the public, Hungarian /bind/<token> page (no login — the URL token IS the auth) → enters the console pairing code (shown on the box screen) + their retrieval passphrase → hub stages the bind via the same BindAppliance (provenance customer_selfbind) → the box's ~30 s appliance poll delivers. Viktor's three rulings, verbatim: (a) "only their own visible" → console pairing code, no appliance list ever rendered; (b) first-box entry → operator-sent tokenized capability link over Hungarian email; (c) lockout after 5 failed attempts → token locks, "call support". Wrong code and wrong passphrase = one identical generic failure (no oracle; both factors compared unconditionally); expiry falls back to operator-bind unchanged. THE TRAP (§9.2): one public prefix /bind/, exempted from auth+CSRF at both /login gate sites via a single isPublicBindPath predicate (tight trailing-slash match; ServeMux ..-cleans; handler rejects / in the token) — 4 red-proofs verified red-then-green (lockout→C1, oracle→B, widened-prefix→E, single-active→C4). GC verdict: no appliance-staleness GC exists (applianceStaleAfter is a display badge only) → the 7-day token TTL stands alone, single-active-per-customer, no reaper needed. Controller/agent untouched; R-27b (controller second-box dismissable prompt) DEFERRED, mechanism sketched. Green: full hub build/vet/test (17 ok, +9 self-bind tests) + bash -n. DEPLOYED + LIVE (hub 0.66.0 on k3s, Synced/Healthy). Live through the real ingress: GET /bind/<garbage> → 200 Hungarian expired page (public, logged-out, no-oracle — the 200-not-500 proves selfbind_tokens migrated live); / + /hosts → 302 /login (gate intact); POST /bind/<garbage> no-CSRF → 200 (exempt) while POST /customers/acme/block no-CSRF → 302 /login (exemption is /bind/-only). PENDING (operator-gated, NOT-live): operator "Send self-bind link" + the customer's real two-factor success (hub UI password-gated → CC can't log in) + the new-ISO console banner on a re-imaged box → fold into the R-1 rehearsal. See REPORT.md.

• 2026-07-17 — PBS DR STORAGE VISIBILITY + Offsite tab split + dual gauges shipped (hub v0.65.0 + tenantsync v1.2.0, R-5), LIVE. Scoping correction (Viktor): "restic box" and "PBS box" are NOT two Hetzner Storage Boxes — restic = subaccounts on the shared Hetzner box (Hetzner API, v0.64.0); PBS DR = the felhom-offsite PBS datastore on ep0 (NO Hetzner API; the hub had no read of its fill). Option A (ruled): a read-only usage op on the felhom-tenantsync ep0 forced command (twin of fingerprint, df on the datastore path — no customer_id, no admin token, NO mutation), polled by monitor.PBSDRBoxChecker (OffsiteBoxChecker clone over a usageReader seam; 15-min throttle; three states ok/unavailable/degraded; escalation-only pbsdr_box_fill on the "pbsdr-box" operator scope). /offsite now has Restic + PBS DR tabs (endpoint cards under PBS DR — the endpoint IS the PBS host); the dashboard has two gauges (restic %·ratio, PBS %). Graceful degradation: the hub deploy is INDEPENDENT of the ep0 update — an ep0 ≤ v1.1.0 → ErrUsageUnsupported → the gauge shows "n/a" until updated (no hub redeploy). PROVEN-LIVE (data pipeline, both gauges): ep0 updated to v1.2.0 this session; the hub polled it — PBS-DR box refreshed: 19.1% full (7.1 GB of 37.2 GB). DECISION: PBS thresholds are separate keys (PBSDRBoxFill*) defaulting to restic's 80/90 (independently tunable; Viktor's call for a different pair); no oversubscription for PBS (namespaces, not quotas). OPEN: UI render unit-verified only (hub UI password-gated); the PBS fill alert email is unit + red-proof verified, NOT fired live (datastore nominal at 19%). READ-ONLY against ep0 + Hetzner. See REPORT.md.

• 2026-07-17 — OFFSITE POOL-BOX AGGREGATE shipped (hub v0.64.0, R-5). The operator now sees the shared pool box's real state on the hub: total fill vs capacity, Σ(shared soft quotas)/capacity (the oversubscription ratio), per-customer usage/quota bars, and a box-level operator alert (fill 80/90%, oversub 2.0×). New monitor.OffsiteBoxChecker (OffsiteChecker-sibling): fetch-throttled to one Hetzner GET / 15 min, cached BoxSnapshot, escalation-only + recovery re-arm; two INDEPENDENT signals (fill + oversub). Capacity = storage_box_type.size, usage = stats.size (Phase-0-pinned, bytes); Σ(quota) from the ConfigJSON Descriptor (offsite.ReadDescriptor, new — NEVER the report echo; dedicated + disabled excluded); per-customer USAGE from the report echo. Events scope "pool-box" → operator channel only, no SaveEvent. Web: Offsite-tab panel + compact dashboard tile, reading the cached snapshot only (never fetches). PROVEN-LIVE (data pipeline): the in-cluster checker computed the real box — 0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x. DECISION: thresholds (80/90/2.0×) are Claude's encoding — Viktor's ruling pending; named Alerting.OffsiteBox* keys = the one-line flip. OPEN: the UI render is unit-verified only (hub UI password-gated → CC can't screenshot); the alert emails are unit + red-proof verified but NOT fired live (real pool nominal — a live-fire would email Viktor; offered on demand). READ-ONLY against Hetzner throughout. See REPORT.md.

• 2026-07-17 — SYSTEM-INITIATED IMMEDIACY wired per-plane (hub v0.63.0). The immediate-sync arc covered only OPERATOR-initiated changes; system-initiated desired-state mutations bumped the generation silently, so a freshly onboarded box waited a full agent tick for state the hub had already minted (observed live at slice-C onboarding). Decision: wire the existing, live-proven notifiers into every system site on the CORRECT plane — agent-plane poke.Notifier at PBSDRAutoProvision (the exact lag), ReissuePBSDR (which also lifts the pbsdrheal reconciler's escalation with zero reconciler changes), handlePBSDRReissue, and the two admin api desired-state writers (new nil-safe Poker seam); controller-plane intent.Hub.Bump at reissueOnReenroll. All fire AFTER the store write only. The pbsdrheal Restage path is deliberately poke-free (no generation bump → the agent's 60 s pbsdr ticker is its pickup path — a poke there is a verified no-op); the WG register/delete bumps stay poke-free (undeliverable pre-tunnel / transport removed — the agent fast-tick SECONDARY owns the register leg). The first-hold-baseline race is still OPEN by design (a bump landing during a fresh controller's FIRST wait is recorded as baseline without firing — the ruled fix is carrying intent_gen in the report ACK, its own future task). Unit-tested + red-proofed; live legs fire only on real system events (fold into the rehearsal bind sequence). See REPORT.md + hub/CHANGELOG.md v0.63.0.

• 2026-07-17 — R-21 SLICE C SHIPPED: the universal secret-free ISO (hub v0.62.0 + scripts v1.19.0, LIVE). The distributed ISO carries NO customer secret. A box booted from build-felhom-iso.sh --pairing self-registers as an UNCLAIMED APPLIANCE (POST /api/v1/appliance/register, keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker; token = sha256-stored, 256-bit), the operator BINDS it to a customer on the Hosts page (new "Unclaimed appliances" section, SSH-key fingerprints, host-count display-only), and the hub delivers customer-id + retrieval passphrase ONCE (one-shot poll: 404 no-oracle / 204 unbound / 200 deliver / 410 gone). The bootstrap is one unit, two modes — direct (env-baked, byte-identical, regression-proven zero-appliance-calls) vs pairing (register→poll→write env→fall through to direct). Red-proofs: one-shot delivery + register idempotency (both proven red). Live-verified through the public ingress (register→token, 204, bogus→404). Artifact proven secret-free (baked env = hub URL only; manifest secret-bearing: no). Bind is operator-password-gated → the live boot→bind→day-0 composition (+ physical N100) folds into the supervised rehearsal, which now runs the COMPLETE final flow in one pass (RESET demo → generic ISO → bind → day-0). SSH-key pinning = stored + fingerprints displayed (host-attach on bind deferred, no clean surface). Self-bind page = R-27. R-21 now SHIPPED (A+B+C). See REPORT.md.

• 2026-07-17 — R-21 SLICE B SHIPPED: firmware loader option (scripts v1.18.0). Closes N100 F1 (HIGH) — cheap AMI (AN3PLUS 0.01-class) firmware can't USB-boot the ISO's stock GRUB (relocation 0x0). build-felhom-iso.sh --loader shim|mkimage (default shim, unchanged; profile-settable via FELHOM_LOADER). mkimage = a monolithic grub-mkimage loader built from the ISO's own GRUB (module set from its grub.cfg, embedded search --fs-uuid→configfile), swapped into the ISO9660 tree + efi.img ESP by mkimage-surgery.sh after prepare-iso (payload provably untouched), re-mastered with xorriso preserving BIOS-hybrid + UEFI + GPT-ESP (drops only the irrelevant Apple HFS+/APM). Unsigned → target Secure Boot MUST be OFF. Loud: banner + manifest loader: + -mkimage filename. New profiles/n100.profile; Dockerfile gains grub/mtools. RUNBOOK-B legs (nested VM 311, OVMF): shim boots+installs under SB-enforcing + SeaBIOS; mkimage boots+installs under SB-off; mkimage under SB-enforcing FAILS Access Denied (red-proof); surgery byte-identical payload. Physical N100 boot still pending → supervised rehearsal (an n100-safety match-nothing ISO built + sha-recorded, unbooted). PXE stays a deferred R-21 note. See REPORT.md.

• 2026-07-17 — CUSTOMER RESET SHIPPED: hub v0.61.0 + felhom-tenantsync v1.1.0 (LIVE). The middle lifecycle tier (host delete < RESET < customer Delete). One operator action → pre-first-install: all OPERATIONAL state dies (offsite repo, PBS namespace+backups, DR recipe, one-time secret, claim state, retained escrow custody); IDENTITY + basic config (incl. the offsite tier CHOICE) + provenance + events SURVIVE. Rulings: separate escrow-custody ack; clears claim (fresh code next onboarding); REFUSES while any host row exists; live-counted confirm inventory. Discipline: external teardown FIRST, DB purge LAST, every leg idempotent → partial run re-runs from the top (purge withheld until externals ok). New: store/customer_reset.go (journal+inventory+ack-gated purge), claim.ResetToUnclaimed, offsite.{Deprovision,OffsiteIdentifier,ClearProvisionedDescriptor}, tenantsync.Deprovision + felhom-tenantsync.sh deprovision op (v1.1.0, destroys ns+groups+token, shared user untouched), web/customer_reset.go (GET inventory JSON / POST orchestration) + an amber RESET card distinct from the red Danger-zone Delete. Red-proofs: ack-gate + partial-failure resumability (both proven red). Live-drilled on ep0 (throwaway drill-reset-01 with a real backup): deprovision deleted:true, idempotent re-run deleted:false, all 3 real tenants survived. Hetzner-delete + the password-gated web POST covered by tests (offsite Deprovision mirrors live-proven ReissueCredentials). See REPORT.md.

• 2026-07-17 — HOST-DELETE DEMOTES ESCROW: hub v0.60.1 (LIVE). Closes the v0.60.0 review gap: DeleteHost(deleteEscrow=true) now DEMOTES the current escrow blob into host_escrow_superseded (copy-before-delete, same tx) + spares existing superseded rows — never destroys custody. The customer Danger-zone Delete (DeleteCustomerConfig, which before touched ONLY customer_configs) is now the one true purge point: cascades both escrow tables for all the customer's hosts incl. already-deleted (via F-14 host_deletions). One shared copy routine demoteCurrentEscrowTx (reused by SaveHostEscrow). Checkbox/refusal/Danger-zone wording → demotion. S6b VERDICT: OBSOLETE — re-enroll upserts cleanly (UpsertHost ON CONFLICT DO UPDATE; handleAdminCreateHost no dup refusal) + v0.57.0 arc auto-fires re-issues (ROADMAP R-3 refined). Red-proofs TestDeleteHost_DemotesEscrowNeverDestroys + TestDeleteCustomer_PurgesEscrowCustody. Observation: customer-delete still doesn't touch host rows / the broader offboarding (rehearsal's question). REPORT.md.

• 2026-07-17 — OFFSITE CONTINUITY Part B SHIPPED: hub v0.60.0 (LIVE). Superseded-escrow retention (data-first, Viktor's ruling). SaveHostEscrow's destructive ON CONFLICT overwrite destroyed the old passphrase's only copy (the reinstall-orphan incident); now a new different-sha blob RETAINS the old into the new host_escrow_superseded history table (tx, before overwrite); same-sha = idempotent. ACK/restore read the current row unchanged. escrow_superseded event + operator retained-count on host detail; registered offbox_repo_orphaned/offbox_repo_reset. Red-proof TestSaveHostEscrow_RetainsSuperseded. Pairs with controller v0.142.0 (Part A orphaned-repo guard). Guided old-history recovery = R-26 (not built). See REPORT.md.

• 2026-07-17 — N100 polish (F5 scripts + F6/F7 controller). F5 (host-install v1.17.0): appliance mode auto-sizes the guest (RAM=clamp(host-4096,min 4096,max host-2048,ceil host-1024) + cores=host-1 min 2) when no explicit cap; explicit --memory/--cores win. Harness red-proof (8/16/32 GB). Same commit: the stale "EMPTY by default" operator-key comment fixed (R-16 doc-drift, keys are PINNED). F6/F7 landed in controller v0.141.0 (LIVE on demo; fresh-USB wizard enroll+format now PROVEN-LIVE — see the felhom-controller REPORT). New ROADMAP R-24 (guest resources as desired-state / live resize = F5 follow-on). Operator sidebar #1 (BIOS State-After-G3→Power-On) added to scripts/iso/README.md. See REPORT.md.

• 2026-07-16 — DIRECTION-2a AGENT-PLANE POKE SHIPPED: hub v0.59.0 (LIVE). The agent-plane sibling of the v0.58.0 wait channel: internal/poke — a pinned-host-key SSH sender (wgsync sibling) + fire-and-forget Notifier (PokeHost/PokeAllHosts, nil-safe, detached) that nudges a box in seconds via a CONTENTLESS UDP poke relayed hub → ep0 felhom-poke forced-cmd → wg0-origin → the box WG /32:51822. Wired at applyPBSDR (per-host) + handleSetArtifacts (fleet). ep0 surface (scripts/felhom-poke.sh, non-root, no sudoers; runbook §11) + Secret/agent-poke + manifests/hub.yaml (tag 0.59.0). Deployed live (ArgoCD Synced/Healthy, agent-plane poke enabled); ep0 leg live-validated (poke-fired; non-WG + rm -rf / refused). Source note: the MinAgent-floor save does not bump per-host generation (self-update rides signed-ops) → the fleet poke accelerates the next report, not a delta. Pairs with felhom-agent v0.89.0 (the listener). PROVEN-LIVE 2026-07-17: the operator's real manifest save logged poke: sync-poke delivered to 10.77.0.2 (+ floor HELD for the below-MinAgent box); the demo (now 0.89.0) received a poke in ~31 ms and fired the immediate desired-state cycle (save→tick ≈ ~0.45 s). All three live legs (Scenario 4 A/B train, Scenario 1 R-22 self-heal, Scenario 3 latency) done; capability-map poke + F4 rows upgraded to PROVEN-LIVE. See REPORT.md.

• 2026-07-16 — DIRECTION-2 IMMEDIATE-SYNC SHIPPED: hub v0.58.0. The hub→box "sync now" wait channel (option b of documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md). New internal/intent in-memory per-customer generation notifier (Bump/Wait/Close; coalesce-to-latest; race-closer; in-memory by design → a restart costs one harmless full-state report, never a storm). GET /api/v1/wait long-poll: per-customer only (global key 400; A can't see B), 240 s hold with a 25 s heartbeat newline that defeats nginx's 60 s proxy_read_timeout without any ingress annotation (WriteTimeout lifted per-connection via ResponseController; global timeouts untouched). Every operator-intent web handler bumps AFTER its store write (config CRUD, claim resend, offsite re-issue/freeze, password regen, block/unblock, floors [global bumps all managed], controller log-tail + log-bundle). intentHub.Close() before server.Shutdown. Pairs with controller v0.140.0 (the long-poll client). The immediate-sync arc is COMPLETE for the controller plane (Direction 1 = ctrl v0.139.0, Direction 2 = hub v0.58.0 / ctrl v0.140.0); the agent-plane ep0-poke stays PARKED in the OOB arc. Detail: hub/CHANGELOG.md v0.58.0, REPORT.md.

• 2026-07-16 — REINSTALL ARC SHIPPED: hub v0.57.0 (F2/F3/2.3). The clean-slate reinstall-of-existing-customer path is now first-class (Peti R-1 prerequisite). On host-enroll's mint path (single-bump-per-re-enroll): claim re-issue for a claimed customer (ReissueForReenroll, reset code — the hub never stores the password, so fork B not A), offsite re-issue to the fresh box (ReissueOffsiteForCustomer), and escrow-honesty — re-issuing offsite marks the escrow STALE (MarkEscrowStale), withholds the mismatched restic-hash from auto-confirm, and shows "stale" not "ceremony done" (2.3, red-proofed). Part 0 unblocked the demo live (ACL grant on /storage/felhom-offsite → PBS-DR converged, escrow.pbs_storage_id seeded). Controller + scripts unchanged — both spec premises were contradicted by source (the controller reads escrow prereqs live from the agent; the installer can't know the descriptor-provisioned storage id). F4 root fix is agent-side → ROADMAP R-22 (agent train: the reconcile pre-check 403 must not abort before the root-run apply grants the ACL). Detail: hub/CHANGELOG.md v0.57.0, REPORT.md.

• 2026-07-16 — R-21 PHYSICAL RUN on the demo N100 (tests/VALIDATION-n100-baremetal-2026-07-16.md). demo-felhom host reinstalled clean-slate from a pipeline-built ISO; first-boot chain reached rc-0 on the first attempt on real hardware (closes slice A's operator-gated boundary), serial-filter safety proven on metal (external-HDD canary byte-identical), PBS-DR reconciler self-healed on the reused peer. The obstacle was firmware, not us: the cheap AMI AN3PLUS 0.01 board can't UEFI-boot the ISO's GRUB from USB (relocation 0x0; SB-off + shim-bypass don't help) — worked around live by rebuilding the stick's loader with grub-mkimage from the box's own working GRUB. Findings (to tackle): F1 firmware boot loader (slice-B), F2 claim-re-issue on reinstall, F3 offsite-re-issue, F4 PBS-DR ACL on non-default storage id, F5 guest-RAM configurability, F6 drive-init doesn't mount+attach, F7 back-route. Box healthy, BookStack deployed, demo live.

• 2026-07-16 — R-21 SLICE A SHIPPED: bare-metal Felhom ISO pipeline (scripts/iso/, scripts v1.16.0). A DooPlex pipeline (build-felhom-iso.sh + Dockerfile.assistant) turns the official PVE ISO into a Felhom auto-install ISO whose first-boot stub installs a retry-forever felhom-bootstrap unit that unattended-fetches felhom-host-install.sh from the public felhom.eu/scripts/ channel and runs it (host-install unmodified). Validated on VM 310: build gate + red-proof, disk-filter fail-safe, stub→unit→real-fetch→invoke→retry, resume-decision, exactly-once, no-net retry+recovery — all GREEN. Rulings settled: stub is from-iso (no cert-rotation risk); ordering fully-up; the retry unit owns all network work. §4.4 finding (shapes slice C): the distribution URL carries no secret, but the unattended install needs the customer retrieval passphrase → the slice-A ISO is secret-bearing (supervised/single-use; the bootstrap shreds the env on success). OPEN: host-install rc-0 terminal success (hub-enrolled + guest-provisioned) is operator-gated — a drill customer needs the password-gated create-UI (a live-hub.db insert is unsafe). Slice B = physical N100 RUNBOOK; slice C = claim-pairing to de-secret the distributed ISO. Detail: REPORT.md, documentation/audits/SPIKE-baremetal-iso-2026-07-16.md.

• 2026-07-13 eve — ESCROW CEREMONY WIZARD SHIPPED (agent v0.88.0 + controller v0.127.0; this repo: host-install v1.16.0 + RUNBOOK rewrite). The recovery-code ceremony is now customer-driveable from the controller (/backup/escrow): one-shot R reveal, re-stage-first, Scenario-F stale-blob re-check (fired LIVE on both boxes' hash-less blobs at first ACK; drill's repaired via a real ceremony — hub hash now covers). Ruling F1 (2026-07-13), settled: R transiting the Cloudflare tunnel once at reveal is an accepted risk — same trust class as the claim code/login password; LAN-direct delivery is PARKED. (threat model: runbooks/RUNBOOK-escrow-ceremony.md). host-install v1.16.0 = FELHOM_ESCROW rides the canonical sudoers fetch; hub hostInstallVersion synced (gate green), hub NOT redeployed (const rides the next train). OPEN: supervised full-browser wizard pass (Viktor's drill login; also demo, to clear its legacy stale warning); agent 0.88.0 publish + Day-0 vouch at the next train.

• 2026-07-13 — CLOSING BUNDLE SHIPPED: hub v0.53.0 — the last engineering items on the pre-tester board: F-14 gated auto-Reissue (deletion provenance host_deletions in-tx + token_exists gate → the EXISTING Reissue op + pbsdr_auto_reissue audit event; no record / un-acked → refusal byte-unchanged), dead-host roll-up honesty (customer status = worst of controller-derived + per-host hostStatus on dashboard//configs/detail, cause chip names the host; the Peti 23h-masking shape is the pinned fixture), operator bearer OUT OF GIT (Secret/report-api secretKeyRef non-optional + REPORT_API_KEY env override + scripts/manifest_bearer_gate.py; VALUE ROTATION = supervised step in documentation/runbooks/secrets.md — the git-history copy is dead only after it). Two operator rulings (2026-07-13), recorded verbatim — future sessions must treat these as settled:

  1. F-14: auto-Reissue of offsite credentials is permitted ONLY when the hub's own deletion record shows the tenancy's owning host was removed through the escrow-ack flow — acknowledged destruction is not silent re-keying. No record → manual Re-issue stays the only path.
  2. Roll-up: a customer's status may never look better than its worst expected host. Single staleness definition (hostStatusOf) — no second threshold anywhere. Gate additions: python scripts/manifest_bearer_gate.py after ANY manifests/ change (64-hex bearer shapes; felhom.secret.yaml carve-out = visible KNOWN-BACKLOG line). GOTCHA: the repo .gitignore *secret* pattern silently un-tracks any "secret"-named script — hence the "bearer" name. The controller repo's controller.yaml.example carried the REAL bearer literal — scrubbed (felhom-controller 51c871a).

• 2026-07-13 — POLISH BATCH SHIPPED: hub v0.52.0 (+ controller v0.123.0) — take-two F-15 CLOSED (reset-request response carries the rotated code hash — ACK shape/guarantee — emailed codes work immediately; live 1 s apply + first-try accept on Viktor's exact failure path) and F-16 CLOSED (zero native confirm() in the hub UI — inline_confirm.html partial, Igen/Mégse inline two-step, asserted by scripts/hub_confirm_gate.py — which exists but is NOT enforced: it is invoked by nothing (R-29); danger-zone typed-confirm untouched; live: offsite re-issue completed under browser automation without freezing). Operator resend keeps the ACK lag — flash says so. Take-two §4 dispositions updated (F-15/F-16/zero-toggle obs → SHIPPED). Remaining from take-two: F-14 (needs operator ruling: auto-Reissue re-adopt vs never-silently-re-key) + S5 DR-restore drill; hub bearer key rotation still DUE.

• 2026-07-12 — DR-TIER-BY-DEFAULT BATCH SHIPPED: installer v1.15.0 + hub v0.51.0 (+ agent v0.86.0 in felhom-agent) — closes DRILL-day0-vm F-1/F-2/F-3/F-7/F-8/F-9/F-10 and F-6 (by policy). The five operator decisions (2026-07-12), recorded verbatim — future sessions must treat these as settled:

  1. Every install deploys the full DR plumbing uniformly (pbs-apply wrapper, age, WG enabled, full ACL). Capability is baked; activation is a hub flag.
  2. DR tier defaults ON for new customers. Disabling is the per-customer exception — and it removes the Felhom-side cost entirely (no PBS namespace, no endpoint dependency).
  3. Offsite app backup requires the DR tier ON (fork-4 needs the ceremony; the ceremony needs PBS). Enforced hub-side by extending the drill-proven dependency guard.
  4. Identity-only escrow is formally PARKED — closed by policy, not implemented. Revisit only if a customer class demands lean-tier offsite. Record in the ceremony runbook + CONTEXT so future sessions know it was a decision, not an oversight.
  5. WG is base infrastructure, not a DR component — always-on like cloudflared; it is also the future OOB-management transport. BYO consent: a disclosure line lands in the tester-agreement doc; per-customer hub disable remains possible. Mechanics: installer ships felhom-pbs-apply + age + wg_tunnel.enabled: true (byo wg-off assert RETIRED — consent = typed disclosure ack + tester-agreement line), single VERSION source with a cross-check against the hub Setup-tab const written as scripts/hostinstall_gates.py — but NOT enforced: the script exists, is RED today (1.19.0 != 1.22.0) and is invoked by nothing (tracked as R-94 leg (b), class R-29) — default ACL keeps felhom-pbs (doc advice REVERSED — narrowing caused the drill 403). Hub: per-customer dr_tier flag (new default ON; legacy backfilled from descriptor reality, one-time), cascade-not-error semantics + per-stage status on the edit form, WG-registration auto-provision hook (scenario A hands-free), offsite-requires-DR guard (exact F-6 message), host-page capability chips (agent 0.86.0 inactive = neutral; pre-1.15 "binary not found" → migration hint). felhom-pve's latent F-3 state fixed live; Peti's box unreachable (his F-3 check + D/E/G deferred).

• 2026-07-12 — HUB v0.50.0 SHIPPED + LIVE: customer-claim password arc (closes DRILL-day0-vm F-4/F-5). The customer OWNS the dashboard password via a one-time claim code the hub emails to the registered address (bcrypt(code) stored, plaintext only in the send). internal/claim engine (EnsureIssued idempotent Day-0/live-box issue+email, Resend, RequestReset 3/day, MarkClaimed set-only) + store.customer_claims (RotateClaimCode bumps generation + preserves claimed_at) + delivery (config bake web.claim_code_*, report-ACK claim object, POST /api/v1/claim/reset-request) + Setup-tab chip/resend + 3 Hungarian emails. Pairs with controller v0.122.0 (the gate). LIVE-proven on drill box demo-vm-felhom: gate ON via the real edge (/ → 302 claim page, /api → 401); code emailed to the registered address. Deployed felhom-hub:0.50.0 (ArgoCD synced/healthy). Floor raise 0.120→0.122 = operator's supervised LAST step. Observed: the hub issues a claim code for EVERY reporting customer on first report (benign for pre-0.122 boxes — no gate, ACK ignored). Report: documentation/audits/DRILL-day0-vm-2026-07-12.md (F-4/F-5 RESOLVED).

• 2026-07-12 — HUB v0.49.0 SHIPPED: Edit tab merge (edit-a) + scoped auto-refresh + riders (CHANGELOG v0.49.0; commits e740147 → 2e03de1 → 1d94b1a + docs/manifest). The standalone customer edit page merged into the customer page's Edit tab (renamed from Settings): shared config_form_body sub-template + configFormData builder (REUSE.md rows added), Block/Delete relocated to a danger-zone card, header cleaned, GET /configs/{id}/edit → 302 #tab=edit, post-action redirects anchored to their tab, settings→edit legacy-hash alias. Update handler gained server-side required-field validation (error path preserves SUBMITTED values — red-proofed). The 60s auto-refresh is now scoped: fires only on live tabs (data-live-tabs="overview,applications,events,host") and never while a form is dirty; muted (paused) hint. Riders: /style.css?v={{hubVersion}} cache-bust on every template (closes the v0.47.0 max-age=3600 gotcha) + the CLAUDE.md git add -A ban codified. NOTE: the task spec said v0.48.0, but v0.48.0 (app_start_failed) had already shipped — published tags are never re-pointed, so this train is v0.49.0. Out of scope (observed, not done): type-to-confirm for customer delete (still confirm(), relocated as-is); the workspace-root E:\git\CLAUDE.md and other repos' CLAUDE.md files still carry git add -A snippets — flag for their own sessions.

• 2026-07-12 — Day-0 VM DRILL COMPLETE (auto-confirm FIRST LIVE FIRING): full arc proven on a fresh nested-PVE box — appliance Day-0 → floor-at-first-report → ceremony → auto-confirm pending→escrowed in ~7.5 min, zero clicks → offsite backup + restore round-trip. Two HIGH gaps: F-4 no operator password-set path exists (G10 unclosable, dashboards born OPEN — blocks tester gate) and F-6 identity-only ceremony never implemented (no-PBS appliance can't escrow — drill forked to PBS DR tier = full Peti-sequence rehearsal, all green). Installer fresh-box gaps: felhom-pbs-apply not shipped (F-7), age missing (F-10), root-owned guests/ parents (F-3 — check demo for the latent copy), silent root@pam rotation UX (F-8). Runbook fixes committed (day0 A.2 anonymous-fetch; escrow-ceremony identity-only claim CORRECTED + age prereq). Report: documentation/audits/DRILL-day0-vm-2026-07-12.md. Drill VM qm 300 kept (3 snapshots) for re-drills; teardown list in report §9.

• 2026-07-12 — CAMPAIGN-3 Task A SHIPPED: agent v0.85.0 boot/recovery plane + appliance self-heal. Fixes F12 (CRITICAL boot ordering cycle — templates drop network-online, MigrateNetworkUnits repairs installed units), F11/F10/F9 (reassert: fstype-driven classify, reset-failed+rearm, per-unit verdict, + re-arm-on-guest-blind matrix-correction), F2/F1 (zero residue: reset-failed + rmdir), the guest-hook rc255 belt (recover+timeout, wrapper no longer execs), and the appliance node self-heal (internal/selfheal: host networking recovery, gated on deployment_mode — host-install --mode templates it; byo remedy unreachable). 3 sudoers grants added (reset-failed, rmdir, start networking). Live matrix on demo felhom-pve: host reboot ×2 = 0 ordering-cycle lines both boots (F12 dead); appliance self-heal recovered networking on attempt 1; F10 reset-failed+rearm sweep loud; zero residue; guest-reboot heals. NEW finding F13 (HIGH, deferred): active nfs4 under mp8 can fail PVE rbind (rc255); NAS-down-at-boot → apps Created (fix-3). Peti's box (agent 0.81) unaffected until his rollout — carries the F12 coin flip if it has a NAS share (interim drop-in in the agent REPORT). host-install v1.14.0; scripts CHANGELOG. Details: documentation/controller/network-storage-nas.md.

• 2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; host recovered, RCA closed. Unattended chaos campaign 22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/ restore byte-identical/EIO instant/stub badge + deploy-409 live-validated). The NAS automount plane produced 2 CRITICAL + 3 HIGH: F12 CRITICAL (= the overnight host loss, RCA'd next morning): the agent automount template's After/Wants=network-online.target creates a boot ordering cycle — systemd deletes an arbitrary job to break it; boot -1 sacrificed networking (host dark 7h, hardware exonerated), boot 0 sacrificed the automount (NAS dead). Every host boot with an enrolled share = coin flip until the template drops that ordering (_netdev on the .mount suffices). F10 CRITICAL (start-limit never re-armed, once blocked guest start), F11/F9 HIGH (post-start reassert "skip-active" skips the healing automount restart; agent-restart sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots; F7 HIGH (in-place dump write → 0-byte tar replaced last good backup during outage window). Fix-6 numbers: ring horizon 55min idle / 6.5min under load, wiped on every restart. Box left in defined state (6 wave apps healthy, temps removed, NAS baseline-identical). Full ledger/evidence: documentation/audits/CAMPAIGN-3-2026-07-11.md + 180:~/campaign3/.

• 2026-07-11 — HUB v0.47.0 UI REORGANIZATION SHIPPED (CHANGELOG v0.47.0; commits 9f29bf3 → 0daddcd + docs). Five deliverables: (1) .data-table td a:not(.btn) button-contrast fix; (2) customer page = 8 hash tabs (#tab=…, sticky summary strip, Events error badge, no-JS graceful degradation — panels hide only via a JS-added body class); (3) shared host_detail_body sub-template rendered by /hosts/{id} AND the new per-customer Host tab (a list by design; store.ListHostsByCustomer + hostDetailData builder); (4) stale host removal — GET /hosts/{id}/delete-impact (counts/booleans only) + POST /hosts/{id}/delete behind type-to-confirm; ONLINE → 409 always (no override), escrow needs an explicit checkbox (ErrHostEscrowPresent, tx never starts), one-tx cascade incl. the bound wg peer (wgsync's 5-min declarative push converges the endpoint; log bundles die by scope_id == host_id only); (5) /offsite multi-endpoint management UI — all wg_endpoints rows as cards + add/edit/delete with 409 guards (peers-in-subnet pins subnet + delete). Deferral (explicit): peer allocation, the wgsync reconciler push and the desired-state merge stay lowest-endpoint-id (GetWGEndpoint untouched); the future arc is a wg_peers.endpoint_id migration + per-endpoint allocation/reconciler/desired-state. Five red-proofs ran (online gate, escrow ack, bundle scope, endpoint-delete guard, subnet-change guard). Resolved: the earlier ⚠️ about 146d165 sweeping the Part-4 WIP — the sweep caught hosts.go mid-red-proof (escrow ack bypassed); 068427a restored the gate; the arc is now complete and author-reviewed. Stale-doc note: the workspace-root CLAUDE.md's hub deploy wording predates GitOps — deploys are manifest-tag bumps + deliberate ArgoCD sync, never kubectl set image.

• 2026-07-11 — RCA FIXES 1+2+4 SHIPPED + Q1c GREEN (agent v0.84.0 + controller v0.117.0 LIVE on demo). Reboot survival: automatic since agent 0.84.0, live-proven 2026-07-11 21:20 (pct reboot 9201 under operator GO — the hook re-armed + verified nas-media DURING the start task, CWA auto-started healthy, health ok first cycle, no badge). Agent: ReassertNetworkMounts (hook post-start + startup; stop+enable-now on existing sudoers verbs — NO sudoers widening needed; never in periodic paths). Controller: statfs consuming-namespace classifier → probe not_network_fs (stub can never verify), deploy-time stub refusal (409; idle autofs deploys fine), distinct stub badge (wins over unreachable; unreachable line byte-identical), deploy view selects stored HDD_PATH (RCA fix 4). MinAgent unchanged 0.81.0. All red-proofs recorded in repo REPORTs. Feature doc: documentation/controller/network-storage-nas.md §Reboot survival; audit §7 follow-up added. Queued: RCA fixes 3/5/6; Peti publish train now carries 0.84 + 0.117.

• 2026-07-11 — NAS/CWA RCA (diagnostic, no fixes): documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md. The nas-media re-add verified legitimately (new host automounts DO propagate into running guests); a second pct reboot 9201 at 15:39 severed the trigger → guest-side stub; CWA deployed correctly to the NAS (app.yaml + binds + data all nas-media) but is DOWN since 15:39 (docker can't recreate the bind source; perms luckily blocked any system-drive writes; zero data loss). UI "felhom-usb" = display bug (deploy.html selects by IsDefault, never stored HDD_PATH); hub device-ID WARN was CORRECT and the only guard that fired; NAS badge masks because it asks the agent (host ns). 6 prioritized fix tasks in the audit §5. Remediation §6 PERFORMED on operator GO (~20:14–20:20 CEST): automount-unit restart re-planted the trigger in the running guest instantly (fix-1 mechanism live-proven), CWA restarted healthy on the NAS library, uid-1000 write OK, controller health back to status=ok at 20:18. Demo fully healed.

• 2026-07-11 — OBSERVABILITY PASS (hub v0.46.0 LIVE Synced/Healthy; agent v0.83.0 + controller v0.116.1 on demo). Remote pull-only log access to both box components' always-DEBUG capture rings. Hub: log_bundle_requests/log_bundles store (gzip, newest-3, 72 h TTL on the 60 s sweep), the secret gate fail-closed before storing (blocked: possible secret; [REDACTED]/ checksums pass), report-ACK controller_log_requested + heartbeat-envelope log_tail_requested channels (consume-once; pre-0.83 agents stay visibly pending), host-detail Diagnostics section (Request controller/agent logs + View/Download; the hosts read-only invariant amended: these two forms are the only actions). Conventions codified: documentation/runbooks/logging-conventions.md + CLAUDE.md one-liners in all three repos. OPEN: operator clicks both request buttons on hub.felhom.eu/hosts/ to close the live round-trip (UI password-gated; the flow is fully API-test-proven). Hub bearer key rotation still DUE (pre-existing).

• 2026-07-11 — PUBLISH-TRAIN RULES CODIFIED (documentation/runbooks/publish-train-rules.md):

  1. manifest before floor; 2. the manifest screen carries the LIVE DB floor (hub_settings.min_controller_version overrides env, acts immediately) — save the floor field LAST; 3. MinAgent gate — now HUB-ENFORCED PER-BOX (hub v0.45.0, store.ResolveManagedFloor), the manual fleet check retired; 4. box-level backstop = the controller's agent-capability gate (v0.114.0 route probe → v0.115.0 version compare). (Floor-UI separation + per-box gating SHIPPED in hub v0.45.0 — see the backlog-clear entry above; this rules-doc note supersedes the original "hub untouched / option-3 deferred" wording.)

• 2026-07-11 — NAS/coupling backlog CLEARED (hub v0.45.0 LIVE + agent v0.82.0/ctrl v0.115.0 on demo). Composite A–F. Hub v0.45.0 (Synced/Healthy): floor-UI separation + effective-floor source line (DB hub_settings vs env, store.ResolveGlobalFloor) + type-to-confirm w/ live below-floor count; per-box MinAgent conditional floor (store.ResolveManagedFloor HOLDS the controller floor for a box whose hosts.agent_version < the manifest's MinAgent, flags it on the Hosts dashboard) — manual fleet check retired (publish-train-rules.md rule 3). THE one comparator extracted to internal/semver. Agent v0.82.0 = X-Felhom-Agent-Version header (demo-only, Peti stays 0.81 = probe fallback); ctrl v0.115.0 = version-aware Supports + DSM-validated NFS guidance. DSM spike (real DSM 7.2 via virtual-dsm): consumer recipes validated E2E, hardlink caveat closed for Synology (record: documentation/audits/SPIKE-nas-dsm-2026-07-11.md); QNAP caveat stands. Q1c (supervised reboot) FAILED: a NAS automount trigger does NOT survive a guest reboot (empty-dir; agent lacks a network-mount reassert) — fix spec'd at documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md. NOT published (agent 0.82 demo-only). Record: felhom.eu/REPORT.md + each repo's REPORT.

• 2026-07-11 — PUBLISH TRAIN 0.81/0.113 COMPLETE — PETI LIVE on agent 0.81.0 + controller 0.113.0. Record: documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md. Agent 0.81.0 published (sha 914bb3b5…30df0b, live-bytes, anon-verified) + golden 0.113.0 baked/published (sha 66de5e64…82054c, drill VM, virgin-reverted); manifest + signed agent_update (key-id felhom-op-1) by Viktor → Peti 0.79.0→0.81.0 clean (no rollback); floor auto-confirm proof CAPTURED (10:41 0.112.0 → 10:42 0.113.0, anonymous pull proven on HIS box) — KEY LEARNING: the hub UI manifest screen ALSO sets the GLOBAL floor as a DB hub_settings.min_controller_version row that OVERRIDES the env and acts immediately (this is how Peti reached 0.112 and now 0.113 before the GitOps env bump 16390d0); a transient forbidden skew (ctrl 0.113 + agent 0.79, ~10 min) was benign only because zero NAS shares existed — next train: save the floor field LAST. OPEN: Peti journal-group one-liner (E), temp Git Sync creds deletion (G1), hub bearer key rotation now DUE (committed in manifests/hub.yaml + screenshot-exposed this session).

• 2026-07-11 — NAS verify-before-commit SHIPPED (agent v0.81.0 + controller v0.113.0 LIVE on demo; host-install v1.13.0). The spike's production task landed same-day: add = agent verify (retry=0 unit, journal-classified, auto-rollback) → controller uid-1000 write probe (--netprobe re-exec) → register LAST; full rollback on any failure; §3.2 Hungarian error map (nfs_export merged); storage_network.html redesigned (SMB-first, Route-A guidance); orphan rows; systemd-journal group (NO sudoers). Live A–E validated vs an isolated sim NAS, Route A proven in production (alien-uid 1060 server-side). Feature doc: documentation/controller/network-storage-nas.md. NOT published (no Gitea/Day-0 bump; Peti pending with the usermod one-liner). Record: felhom-controller/REPORT.md.

• 2026-07-11 — SPIKE: NAS verify mechanisms + NFS squash matrix — ALL FIVE QUESTIONS ANSWERED, no blocker. Record: documentation/audits/SPIKE-nas-verify-2026-07-11.md. Q1 in-guest access DOES wake the host automount (make-or-break PASS; container bind-mount too); Q2 controller drops to uid 1000 and writes (setpriv proof; SETUID/SETGID caps present); Q3 Route A ACCEPTED-with-caveats (all_squash to ANY server uid = full rw+hardlink for the guest app; guest sees nobody; chown EPERM-clean, chmod persists; June's anonuid=1000 WRONG case pinned = anonuid without all_squash is a no-op for non-root); Q4 error taxonomy captured (NFSv4 makes no-export vs not-permitted INDISTINGUISHABLE; systemd 90 s start-timeout is the real ceiling; retry=0 → production unit: dead-NAS access 91 s→3.8 s; verify must NOT ride the 15 s agentapi timeout → own path/status-poll, ~120 s budget); Q5 plain-user SMB PASS incl. hardlink. Teardown verified to pre-counts. Production TASK is the operator side's next move.

• 2026-07-11 — PBS DR tier SLICE 2 SHIPPED (agent v0.80.0): the apply-bridge — demo ADOPTION-PROVEN live + PUBLISHED; the supervised Peti phase PENDING. Full chain fired on the demo: hub enable (storage-id felhom-offsite) → tenantsync provisioned ns demo-felhom on ep0 → agent ADOPTED the manual entry (tenancy entry-owned, "entry wins"; staged secret asserted UNCONSUMED; DR green; escrow.pbs_storage_id seeded). Laws red-proof-verified: set-only (K destruction guard), secret-on-stdin, verify-pin-before-consume, loud consumed-failed. Part 0 evidence: write path needs the §4b dual-grant exactly (403 Datastore.AllocateSpace → re-assert → token vzdump OK); demo grants restored. Live finding fixed: escrow seed = in-place write (/etc/felhom-agent dir is root-owned; rename impossible for the non-root agent). 0.80.0 published (f2ba62ca…f06d2, == deployed bytes). NEXT = RUNBOOK-peti-pbsdr-2026-07-11.md (Peti box prep: sudoers+wrapper+wg_tunnel enable — no tunnel/OOB on his box today; Viktor opsign 0.80.0; hub enable → FRESH path, K born, degradation clears; ceremony → auto-confirm → first offsite run = the arc closes). Detail: felhom-agent/REPORT.md + CHANGELOG v0.80.0.

• 2026-07-10 — PBS DR tier SLICE 1 SHIPPED+LIVE (hub v0.44.0): ep0 tenantsync surface + hub provisioning. felhom-tenantsync v1.0.0 installed on ep0 (second forced-command key, peersync untouched — hashes asserted; smoke caught + fixed the client {"data":[…]} json wrap; orphan root@pam!spike cleaned). Hub: internal/tenantsync (pinned-SSH twin of wgsync; errors never embed stdout), config-form "PBS DR tier (ep0)" section → WG-peer check (fail-closed) → provision → host_pbs_secrets consume-once (agent fetches POST /api/v1/hosts/{id}/pbs/consume-token — note: /hosts/, not the task's /host/) → pbs_dr descriptor into HOST desired_json + gen bump; idempotent re-save; Re-issue button (F4 twin). Red-proofs: consume-once / fail-closed / idempotency all mutation-proven. NOT live-provisioned — slice 2 (agent bridge) + supervised Peti enable pending. Runbook §10 addendum in offsite-endpoint.md. Commits ce6a566 + 32ca1f6.

• 2026-07-10 — SPIKE: PBS DR tier auto-provisioning SLICE 0 — all three mechanisms SETTLED. Record: documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md. (1) PVE token storage-create impossible (create/modify/delete all check /storage root) → sudoers vector proven (pinned-id alias, cycle green as the agent user, unpinned denied; secret must ride a stdin-wrapper, never sudo argv). (2) ep0 tenancy op-set + one-time-secret-over-SSH-stdout proven live on ep0 (🛑 Viktor-approved; read/write cross-tenant 403s); current peersync channel can't carry it → slice 1 ships a second forced-command surface (felhom-tenantsync). (3) pvesm add pbs --encryption-key autogen births K at PBSEncKeyPath + vzdump lands ciphertext (doc-06 property) + escrow-create finds K. ⚠ pvesm remove DELETES K → re-apply must be set-only. All spike artifacts torn down; flags: orphan root@pam!spike on ep0 scratch; demo's §4b step-3 storage grants absent. Epic slice map in the record §5.

• 2026-07-10 — new-box restore round-trip VALIDATED + old-box archive RETIRED. Record: documentation/audits/VALIDATION-newbox-restore-2026-07-10.md. Both toggled apps restored from the production pool box via the real endpoint — 9/9 sha256-identical (immich 44MB db-dump exact, d78466fcbf…4526b, agreeing with the campaign's independent round). ~10s/app. Viktor's decision: RETIRE the old archive — he deletes u629193-sub1 in the old project's console (the PBS-storage-1 box itself stays for dev). C2 fixed same day as controller v0.110.0 (crash-lock self-heal, live-proven). Offsite arc: build-complete + restore-proven both directions; remaining = the Peti onboarding runbook.

• 2026-07-10 — offsite arc overnight soak (unattended) — PASS + 1 HIGH finding. Record: documentation/audits/CAMPAIGN-offsite-overnight-2026-07-10.md. F4 re-issue proved LIVE (reset invalidates the old password); restore round-trip 9/9 byte-identical new-box↔USB (immich db-dump exact → old-box archive u629193* safe to retire, Viktor's decision); single-flight + crash-restart (zero consume, escrow untouched) PASS. FINDING C2 (HIGH, OPEN): a controller crash MID-PRUNE leaves a stale restic exclusive lock the controller can't self-heal → offbox runs fail until manual restic unlock (fix candidate: ensureOffboxRepo auto-unlocks a self-owned stale lock). Track B (live quota-crossing) + hub-orchestrated Track A blocked — the hub operator config surface is session/CSRF-gated (unattended can't reach it) → Peti runbook. Demo restored to start-state. One expected backup_failed email (the crash test).

• 2026-07-10 — SLICE 4 soft-quota + offsite health SHIPPED+LIVE (hub v0.41.0 + controller v0.109.1) — the offsite arc is BUILD-COMPLETE. Controller: quota_gb rides the descriptor into the target (hash includes it → quota changes re-apply via key-auth-first, no consume); repo_size_bytes from restic stats; pre-run gate (≥100% refuses NEW backups Hungarian-loudly, prune still runs — red-proofed; restore never gated; ≥80% warns); /backups usage bar. Hub: monitor.OffsiteChecker (fill 90/95 of quota + offsite_stale >48h for enabled+escrowed only — red-proofed; nil-safe on old reports) + manual-only Freeze/Unfreeze lever (readonly on the exactly-1 labelled sub-account, SSH preserved). Live: key-auth-first proven against the real box (the deploy's own re-apply); that re-apply exposed

  • fixed v0.109.1 (re-apply must preserve EscrowState + runtime status — it demoted the escrowed demo); demo repaired (manual confirm + run: escrowed, 45 MB / 50 GB, checker healthy 0 events, usage bar renders). Freeze + 100%-refusal NOT live-exercised (STOP; red-proofs cover). NEXT: Peti onboarding runbook, NEW-box restore round-trip → old-box retirement.

• 2026-07-09 — SLICE 3 hub-verified escrow auto-confirm SHIPPED+LIVE (hub v0.40.0 + controller v0.108.0 + agent v0.79.0). Replaces the operator-trust confirm with a verified fact: ceremony hashes the sealed repo password (agent, restic_pw_sha256,omitempty on the escrow upload — contract test moved in lockstep) → hub stores it (additive host_escrow migration, NULL on legacy rows) + serves escrow:{identity_blob_present,restic_pw_sha256,created_at} in the report ACK → controller (report.EscrowAutoConfirmer) flips pending→escrowed ONLY on hash match with its local repo password. Red-proofed core: blob-presence alone never confirms (stale blob = false custody claim); mismatch warns once per hash naming the ceremony; never un-confirms; flip wipes the agent-staged secret. Hashers pinned by the same cross-repo vector. Manual confirm = deprecated fallback (legacy blobs — demo included, hash-NULL verified live; the escrowed demo produced zero confirm activity = Scenario E live). Full happy chain proves out at the next real enable (Peti). NEXT: SLICE 4 soft-quota, Peti runbook.

• 2026-07-09 — offsite hardening bundle SHIPPED+LIVE (hub v0.39.0 + controller v0.107.0 + agent v0.78.0). The e2e's sharp edges closed before slices 3–4: F4 "Re-issue offsite credentials" (explicit operator recovery for consumed-password dead-ends — exactly-1 label scope, fresh one-time secret, ConfigVersion bump; hetznerapi.ResetBoxPassword added for dedicated); F2 host-key scan retry (~60s ladder — first save survives fresh-subaccount DNS lag); F5 submit-disable + in-flight notice; key-auth-first bridge (working installed key → re-pin+reconfigure WITHOUT consuming; fingerprint verify never bypassed; kills the stale-descriptor consume-404 loop); staged-secret wipe (agent DELETE /escrow/stage-secret; controller calls it on every flip to escrowed; the e2e's lingering staged file wiped live through the new endpoint, idempotency proven). All red-proofed. F4 reset NOT exercised live (demo credential untouchable — throwaway customer under supervision when needed). NEXT: SLICE 3 auto-confirm, SLICE 4 soft-quota, Peti runbook.

• 2026-07-09 — offsite provisioning VALIDATED LIVE end-to-end (hub v0.38.1 + controller v0.106.1) — PASS. Full loop through the real pipeline on demo-felhom against the new dedicated-project pool box (611714 / u629488, project 15282031): UI save → sub-account 269008 provisioned → descriptor v6 + one-time password → controller config-refresh (≤15 min) → apply-bridge (verify-pin → consume-once → pinned ssh-copy-id → EscrowState=pending) → run-gate refusal proven → escrow confirm (Option A: the repo password was already escrowed — WriteOffboxSecrets preserves it) → first run: restic repo + 2 snapshots on the new box, key-only. Two live bugs found+fixed mid-run: F1 hub v0.38.1 (applyOffsite on context.WithoutCancel — a re-click canceled the provision between create and password-store, stranding the sub-account) and F3 controller v0.106.1 (ssh-copy-id -s needs ~/.ssh to exist in the container — died AFTER consuming the password). OPEN: F2 (fresh sub-account DNS lags → first-save scan 502s; retry-with-backoff), F4 (hub has no password-reset path — recovery today is delete-subaccount + re-save), F5 (no save spinner → re-click bait). Phase-0 wiring: manifests/hub.yaml now injects HETZNER_TOKEN/HETZNER_POOL_BOX_ID from Secret/storagebox; scoped-token check = list boxes with it (must see ONLY 611714). Old-box offsite kept as frozen archive until a NEW-box restore round-trip. Record: documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md. NEXT: SLICE 3 (auto-confirm), SLICE 4 (soft-quota), F2/F4/F5, Peti onboarding runbook.

• 2026-07-09 — offsite provisioning SLICE 2 (hub v0.38.0 + controller v0.106.0). The controller apply-bridge: on startup it reconciles the hub-served offsite: descriptor into a key-only offbox target (controller/internal/offsiteapply.Bridge) — verify-pin the box host key against the hub-captured host_fingerprint (no blind TOFU) → consume the one-time password (single-use) → sshpass ssh-copy-id -s -f install → configure offbox → EscrowState="pending" → persist a descriptor-hash marker. Idempotent + fail-safe; both red-proofs green. Hub v0.38.0 adds Descriptor.HostFingerprint captured via an x/crypto/ssh keyscan (fail-closed). NOT yet live-applied — supervised end-to-end (hub provisions on the new pool box → controller apply) is the next runbook, gated on the hub's new scoped HETZNER_TOKEN. NEXT: SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota).

• 2026-07-09 — hub offsite provisioning SLICE 1 (hub v0.37.0). The hub can now provision the offsite tier on operator enable: internal/hetznerapi (typed client, base api.hetzner.com/v1, CloudAPI interface + exported Fake, WaitAction), internal/offsite (Provisioner.ProvisionOffsite — idempotent by label felhom-customer=<id>, shared sub-account / dedicated box, generates the transient password, builds the NON-SECRET descriptor, fail-closed), one_time_secrets store (single-use Save/ConsumeOneTimeSecret), POST /offsite/consume-password/{id} (customer-API-key auth, once), config-form Offsite section → applyOffsite (502 + no-save on error) → descriptor in ConfigJSON → ConfigVersion bump. Token from HETZNER_TOKEN (out-of-band); password NEVER in ConfigJSON/logs. Unit-tested vs a faked Cloud API (+ a fail-closed red-proof). NOT yet live-provisioned — awaiting the dedicated-project scoped token (the current token can delete ep0). NEXT: SLICE 2 (controller apply-bridge consuming the descriptor + one-time password), SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota). Deployed via GitOps.

• 2026-07-09 — Hetzner API provisioning spike (offsite tier). Both offsite models are API-provisionable (FIT). #1 correction: the Storage Box API is https://api.hetzner.com/v1, NOT api.hetzner.cloud/v1 (Cloud API 404s for storage boxes); same Bearer token. Proven live on spike-* (torn down): sub-account create/reset/readonly/delete + a billable bx11 box create/rescale(change_type)/delete. Create takes a password (no ssh-key param) → transient-password→ssh-copy-id -s -f→restic-reach→reset_subaccount_password (transient pw invalidated, key survives) = key-only steady state. RepoPath /home/<repo> (chroot). Quota: dedicated=box type/change_type (hard); shared=no native quota → Felhom soft-quota + update_access_settings{readonly} freeze. Idempotency = label_selector (box names not unique). Rate 3600/h. SECURITY: the token is project-wide unscopeable Read+Write — it can also touch ep0 (the prod offsite server) + 3 SSH keys → move customer boxes to a DEDICATED Hetzner project before the hub holds a write token. Record: documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md. NEXT: the hub-provisioning impl.

• 2026-07-09 — fork-4 custody VALIDATED end-to-end live (supervised). The full custody cycle proven on the demo (agent v0.77.0 + controller v0.105.0): enable → controller pushes the repo password to the agent → real escrow ceremony (--selftest=escrow-create, K + restic-pw under a fresh R, uploaded to the hub, staged secret wiped, K byte-unchanged) → atomicity proof: a run skipped before confirm-escrow and proceeded after (2 apps, 2 snapshots) — the gate opens exactly on confirm. DR money shot: the restic password recovered from the production hub blob (host_escrow.identity_blob for demo-felhom-01, via identity-consume with R) is byte-identical to the live password (sha 24c44f49…) and decrypted the real box repo; the real inject-password→restore endpoints reproduced a 45 MB db-dump byte-perfect. Demo offbox left escrowed (un-paused); Viktor holds R out-of-band; no secret in any log; live repo_password unchanged. Record: documentation/audits/VALIDATION-fork4-custody-2026-07-09.md. NEXT: hub-provisioning + soft-quota spec (hub-verified auto-confirm replacing operator-trust; both storage models); per-customer onboarding runbook; the identity-consume OK-message fix (omits the recovered restic field).

• 2026-07-09 — offbox v0.104.0 VALIDATED live against the Storage Box (supervised). Closed the offsite arc: controller :0.104.0@908e4b9 ran the real /backup/offbox/* endpoints against a fresh box sub-account (u629193-sub1, RepoPath /home/felhom-demo-repo). The capstone proof — immich (undeployed, recovery unit only on the USB drive, the exact case v0.103.0 mis-resolved to systemDataPath) → discovery found the USB unit → ok, 1 snapshot; 45 MB db-dump restore sha256-matched. No-silent-success confirmed on the live box: 0/N → error + backup_failed operator alert; partial → ok + Hungarian LastWarning on /backups. Persists across restart. Forks: (1) direct topology + (2) sub-account-per-customer (RepoPath /home/<name>, ssh-copy-id -p 23 -s -f — console SSH-key field still broken, F3) confirmed; (3) PBS/WG tier deferred; (4) custody surface RECORDED, escrow design OPEN — the per-customer restic repo password + SSH key are 0600 in the controller data dir, NOT independently recoverable (ride DR only via the PBS whole-CT) → the load-bearing follow-up before real reliance. KEPT as a standing demo-offsite (€4.06/mo). Record: documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md.

• 2026-07-09 — TASK GL-7 SHIPPED: customer-page passphrase hardening + install-command generator (hub v0.36.0, 844fbfa7; deploy 02c748eb; LIVE, ArgoCD Synced/Healthy) — go-live G7/G9/G10/G11. Two coupled, security-first changes to the operator customer page (customer_unified.html + configs.go). (1) Passphrase hardening: the per-customer retrieval passphrase was cleartext twice (visible #retrieval-pw + baked into the Option-3 debug curl X-Retrieval-Password: header); now masked-by-default with reveal (toggleSecret) + copy (copySecret), value in data-secret (existing model), and the Option-3 command carries a <YOUR-RETRIEVAL-PASSWORD> placeholder — no secret in any copyable command. (Zero-secret-in-DOM fetch = noted future follow-up, NOT scoped.) (2) Install-command generator: the three hard-coded <code> blocks became a client-side builder (vanilla JS, no framework/CDN/network) emitting ONLY real host-install v1.12.0 flags, download-then- run, byo-requires-caps enforced client-side, CustomerID prefilled from the server, JS-off static fallback; the seven dangerous/operator-only flags (--force/--rotate-recovery/--enable-oob/ --remove-golden/--uninstall/--adopt-pool/--rescope-acl) are never offered as controls. Tests TestTemplates_PassphraseHardened + TestTemplates_InstallGenerator (green; passphrase test red-proofs). Tester agreement written (documentation/pilot/PETI-tester-agreement.md): BYO trust boundary, honest limitations (no offsite backup, F2 removable-drive, F7 pool-reassert, F6 :53-free), exit rights, Peti box facts (80c/128 GB → caps 12/32768, sajatfelhom.hu re-point), byo install command, onboarding sequence. GO-LIVE-PACKAGE: GL-7 ✔; G9/G10/G11 → done. One open pilot question (owned by the agreement §6): Peti's local backup TARGET — does his box have a second disk/pool? → sets --acl-storages at onboarding. Deploy gotcha: first build was stale (local commits not pushed before build.sh pulls origin) — pushed then rebuilt. NEXT: Peti day-0 onboarding (operator-run).

• 2026-07-08 — TASK GL-8 SHIPPED: BYO coexistence hardening (host-install v1.12.0, 18a556a5) — the GL-6 findings F3/F6/F1. F3: the populated-host leaf guard fires only when a felhom_guests guest exists (not pct list) → a fresh byo install on a host running only the OWNER's guests needs no --allow-new-leaf; the pin protection for real Felhom guests is unchanged. F6: byo preflight REFUSES a foreign process on :53 with remediation and mutates NOTHING (Viktor's ruling: refuse+ instruct only — the dnsmasq analog of the break-glass no-touch rule; appliance untouched). F1: uninstall purges the config .bak* siblings (a GL-6 residue held a live hub api_key). F6 live-proven on felhom-pve (byo preflight refuses at :53 — the exact GL-6 scenario, read-only die). Harness 36/36 (+GL8-F3/F6/F1 static, GL8-F1b behavioural, GL8-F6 live) + red-proofs RP-F3/F6/F1; GL-2/GL-4 regression green; shellcheck clean. Shipped BEFORE GL-7 so Peti's command needs no workaround. F2/F7 remain open (non-blocking). NEXT: GL-7.

• 2026-07-08 — TASK GL-5 SHIPPED: DR bring-up bind overrides (agent v0.75.0, LIVE + scratch-DR validated) — go-live G8; docs-only here (GO-LIVE-PACKAGE G8/G6/G1 flips + decision log; the code + full report live in felhom-agent b3446213). Guest-loss DR restore WORKS now: the two structural binds (mp8/mp9) are overridden with throwaways at restore and swapped for the REAL binds pre-start (root pct via the new Engine HostRunner seam), displaced unusedN deleted. TWO live-discovered PVE rules beyond the spike: explicit-params restore needs an explicit rootfs AND silently DROPS unlisted mountpoints — the full param set now derives from the archive's embedded config (ExtractArchiveConfig, 200 under the scoped token). Live proof: scratch 9310 from a real 9201 archive — mp0 200G + mp1 50G restored WITH content (7m23s), real binds, zero residue, clean teardown; the same op failed outright on v0.74.0. OPERATOR: publish agent 0.75.0 + bump the manifest to it (vouching 0.74.0 ships broken DR). Follow-ups: restore-test boot-verifies without mp0/mp1 (same PVE rule, reverse direction — candidate reuse of drRestoreOverrides); DR-selftest KeepMAC duplicates a live source's MAC briefly; no host-loss DR runbook exists yet (GL-6/S5 writes it).

• 2026-07-08 — TASK GL-4 SHIPPED: uninstaller gap-closure + operator-key day-0 fold (host-install v1.11.0) — go-live G6 + the G1 key-pin follow-up; awaiting GL-6 for the real (non-dry) teardown + armed-pin install. Uninstall now removes the self-update artifacts (4b4: guarded wrapper, .prev/.new.* slots, rollback unit, limits drop-in — derived from configs/felhom-selfupdate-guarded), unmounts enrolled drives under /mnt/felhom-drives (plain umount ONLY, busy = warn+guidance, root-bind guarded), and ends with a KEPT-vs-WIPED statement (drives/PBS/hub record/escrow/vaulted recovery credential live on; guest-only mode prints the vmid's bind-store drives). Key-pin: OPERATOR_KEY_* constants (EMPTY until the operator ceremony)

  • --operator-pubkey-file (validated at argv, comment=key_id required) → authz.signers written at step 6 per the agent SignerKey schema; no-keys-resolved KEEPS preserved signers (never un-pin a manually-pinned box); verify reports armed/dormant (dormant = WARN). Harness 28/28 on felhom-pve incl. the NEW GL4 H-U full-uninstall DRY transcript vs live 9201; red-proofs RP-1..3 green. GO-LIVE-PACKAGE.md was ABSENT AGAIN (spec said the operator attaches it; not present in the repo or workspace) — G6/G1 status recorded here, doc still pending its operator commit. Key CEREMONY (real keypairs + felhom-pve pin) = operator step.

• 2026-07-07 — RUNBOOK GL-1 EXECUTED: agent 0.74.0 + golden 0.103.0 PUBLISHED — go-live G1 (partial): the published-artifact chain is current. Agent = the LIVE felhom-pve bytes (sha 1ec3f588…76af05, provenance preserved, publish-agent.sh round-trip verified); golden baked clean-room in the drill VM (build-golden.sh v2.0.0 @ ced60ddf, controller 0.103.0 baked, all integrity gates green, sha 8481e8a1…6026e, 583MB) and published HTTP 201. Both artifacts fetch-verified ANONYMOUSLY (Gate 3c finding: generic packages are world-readable → G3 git.token read-only rotation CANNOT break fetches; but the INSTALLER dies on an empty token — script-side requirement, not Gitea's). Drill VM restored to virgin; evidence /mnt/5_hdd/felhom.eu/drill/bake-0.103.0.log on 180 + documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md. OPERATOR NEXT: bump the hub Day-0 manifest to agent 0.74.0 / golden 0.103.0 (the four values are in the execution record) — until then fresh installs land 0.63.0/0.98.3. Also recorded: operator-key pin follow-up (fold into GL-4); GO-LIVE-PACKAGE.md still not in the repo (G1 status recorded here instead).

• 2026-07-07 — TASK GL-2 SHIPPED: --mode appliance|byo install profile (host-install v1.10.0 + harness) — go-live items G2/G4/G5 implemented, awaiting the supervised GL-6 drill for live provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old --mode provision value retired with a specific die, dr stub kept); break-glass gated at its CALL SITE (byo never touches root@pam); byo requires --cores+--memory, refuses --enable-oob/--rotate-recovery/non-9.x-PVE/missing --acl-storages; disclosure + typed-hostname ack; the config write's lan_resolver default flips to OFF in byo (the spec assumed default-off — it was default-ON; appliance unchanged) + post-write asserts refuse lan_resolver/wg_tunnel/oob=true; pool+ACL verify asserts in BOTH modes (R2 lesson); --preflight-only (no state, PASS/FAIL verdict, proven live on felhom-pve). NEW scripts/hostinstall-mode-harness.sh (static C1–C4 + invariants everywhere; PVE tier: C5 + A/B dry transcripts) — 16/16 PASS on felhom-pve incl. C5 live; red-proofs RP-1..3 run→fail→revert. FINDING: demo-felhom's controller.yaml has EMPTY git.username/git.token → any step-5 (re)install for that customer dies until the operator sets real Gitea read creds (relates to the standing scope-down+rotate follow-up). documentation/pilot/GO-LIVE-PACKAGE.md is NOT in the repo and no operator copy was provided — the G2/G4/G5 status flip is recorded here instead; add the doc when the operator supplies it. Next: GL-6 supervised drill (appliance + byo end-to-end, uninstall/re-provision, C7, the ack prompt interactively).

• 2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent v0.72.0) — merged E1+H1. Hub: operator OOB peer (store/wg_operator.go, PUT/GET /admin/wg/operator-peer global key) + oob_peer_ip/oob_operator_ssh_key in the desired-state wireguard block + monitor/host_oob (oob_degraded/recovered). Installer: --enable-oob/install_oob (static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key.

• 2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent v0.71.0) — prereq for felhom-sshd (H1). Hub: store.host_recovery vault (per-host root@pam console password; PUT /hosts/{id}/recovery-credential self-scoped + GET /admin/hosts/{id}/ recovery-credential global-only, secret never logged) + host_mgmtplane monitor raising mgmt_plane_healed. Installer: step_break_glass (generate+set+vault root@pam, --rotate-recovery)

  • install_mgmt_watchdog (the 3 host artifacts, RuntimeDirectory-guarded). Live-validated: agent-independent /run/sshd auto-heal in 30.0s; warning fired end-to-end; break-glass vault→retrieve→PVE-ticket(200). felhom-pve root@pam is now the vaulted value.

• 2026-07-04 — S3 SHIPPED: agent WG tunnel (felhom-agent v0.64.0; docs-only here) — the doc-06 §3.3 handshake is now END-TO-END LIVE on demo: agent keygen → POST /hosts/{id}/wg → desired-state wireguard block → agent-managed wg-quick@wg-felhom → PBS page over 10.77.0.1:8007 in 3 s. wg_tunnel.enabled defaults FALSE agent-side (rollouts to other boxes are no-ops until the production endpoint exists). Revocation semantics completed (§3.5): removed peer → agent disables + keeps its marker + NEVER re-registers; operator re-adds via the pubkey the heartbeat report now carries. Escrow: IdentityBundle.WGPrivateKey auto-injected on new escrow-create runs (pre-S3 blobs can't be retrofitted — honest fallback = fresh-key re-register, /32 kept). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix, deferred non-blocking); S4 = PBS over the tunnel; hub-side S6 = tunnel-health alerting on the new report stanza. Detail: felhom-agent CHANGELOG v0.64.0 + REPORT.

• 2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0) — doc-06 row S2. DECISIONS: the WG assignment is hub-owned state, merged into served desired-state at READ time — stored desired_json stays a pure operator blob; admin PUT with a top-level wireguard key → 400 (no copy-paste clobber); re-key-in-place keeps the /32 (stable tunnel addressing across rotation/DR); one bound peer per host (partial unique index); generation bumps ONLY on real change (register/re-key/adopt/bound-delete — idempotent paths move nothing). POST /hosts/{id}/wg self-scoped per-host key; NEW golden desired-state-wireguard.golden.json = the S3 cross-repo contract (byte-identical agent copy rule). peersync v1.0.1 (strip out of process substitution — the S1 exit-swallow finding; sandbox red-proofed) redeployed to the dev endpoint. Live-proven on the REAL demo-felhom-01 record via the real per-host key: gen 2→3, served block = golden shape, endpoint wg show carried the peer, v0.63.0 agent tolerated the unknown key (fetch clean, guests untouched), then removed → gen 4, all baselines restored. /offsite page unit-proven (live view is operator-password-gated — eyeball after next login). Next: S3 (agent keygen + registration + wg-felhom service + escrow join + CGNAT smoke — risky/supervised).

• 2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1) — the doc-06 roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: hub pushes over SSH to a forced-command felhom-peersync script (pull rejected: weakens immediate revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev felhom-hetzner box, provisioned by the NEW runbook documentation/runbooks/offsite-endpoint.md (WG on 443/udp, subnet 10.77.0.0/24, PBS datastore felhom-offsite at /mnt/pbs-datastore — a 100 GiB Hetzner Cloud Volume attached 2026-07-27 06:29:40 UTC; the pre-migration copy at /srv/pbs-felhom is 13 G of dead weight on / awaiting R-91's go-ahead — nftables drop-all + wg0-only 8007, ep0.felhom.eu grey-cloud). Hub: wg_endpoints/wg_peers + /32 allocator + /admin/wg/* (global-key, pubkey-in-body) + internal/wgsync (FixedHostKey pin

  • pinned-type HostKeyAlgorithms — live-caught bug: multi-hostkey sshd presented ECDSA vs the ed25519 pin; fixed v0.32.1 + regression test) + full-list declarative reconciler (5-min drift repair). Live: add/remove→wg show, malformed→state-untouched, reboot→persist+converge, client tunnel via ep0:443→PBS login page, public 8007 closed. INCIDENT handled: first WG server key leaked via wg show dump into the session log → rotated immediately (warning now in the runbook). OPERATOR follow-up: fix the AAAA record (currently …7aa1::, must be …7aa1::1). Next: S2 (hub schema host-join + wireguard block on WireDesiredState + box-facing registration + UI).

• 2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06) — documentation/architecture/06-offsite-connectivity.md: records the six settled decisions (plain WG / host-side agent-managed wg-felhom as the agent-managed-unit PILOT / one shared hub-driven endpoint VM running WG+PBS, NO agent on it / hub = source of truth, wireguard block rides WireDesiredState on the existing DesiredGeneration channel / one datastore + per-customer namespaces / PBS ON the VM, relay-through-DooPlex REJECTED as non-scaling), the Day-0 join handshake, robustness set (WG-roaming-not-DynDNS, DNS re-resolve watchdog, MTU 1420, per-/32 isolation, tunnel-health via storage-target model), trust conformance (outbound only; operator-inbound + game-ingress out of scope), the honest ledger (CGNAT still unmeasured — mobile-hotspot smoke closure; peer-sync push-vs-pull = slice-1 design point), and the S1–S6 slice roadmap (MVP = S1 endpoint + S2 hub schema/desired-state + S3 agent WG service, then S4 PBS-over-tunnel; S5 merges with DR-completeness). day0-install backlog line now cites spike + design doc. Docs-only; slices are separate future tasks.

• 2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung) — plain WireGuard, host-side, box-dials-out to a small public endpoint; no Headscale/DERP needed for backup (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway Hetzner: 11.4-min idle hold, PBS API through the tunnel with the exact-cert pin intact (positive + negative), real 2 GiB worst-case backup at 4.26 MiB/s (= the home uplink; tunnel overhead ~5%), UDP 51820 and 443 both pass. P0 honesty: the operator's line is plain-NAT (fixed public IP, One Hungary), NOT CGNAT, and has zero IPv6 — CGNAT confirmation deferred to Peti's VM 110 (second ISP). P7 surprise: kernel WG works inside the unprivileged guest (no TUN needed) — host placement stands on architecture (backups originate host-side; cloudflared precedent), not infeasibility. Evidence + architecture recommendation (endpoint VM shapes, key custody via IdentityBundle escrow, per-peer /32 isolation): documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md. This unblocks the DR-completeness work: next task = production connectivity spec (endpoint VM, agent-managed WG service, hub peer registry, escrow join). Throwaway Hetzner box parked powered-off (secrets shredded); demo-felhom verified residue-free.

• 2026-07-03 — 4th skill: felhom-app-catalog (catalog authoring workflow; points at app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by install_skills.py, 4/4 found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected healthchecks, probe-container naming rule discovered + REUSE row added, UI-applied, controller probe healthy). App-count drift recorded: website says "45+", per-category spans sum 49, real catalog 53 — website fix is a separate task.

• 2026-07-03 — GOLDEN 0.98.3 LIVE + VOUCHED; guide D.1b RETIRED (drill B1+B5 FIXED) — build-golden.sh v2.0.0 (mandatory controller tag + baked bootstrap .path unit); golden baked with controller 0.98.3, clean-room validated (drill VM) through all four scenarios incl. the published-artifact --force-gitea-golden path, published + operator-vouched (Day-0 manifest now: agent 0.63.0 + golden 0.98.3). Fresh installs land CURRENT and self-manage — D.1b is a one-line check; the prior "vouch agent 0.63.0" operator follow-up is DONE. Evidence: documentation/audits/DRILL-golden-098-2026-07-03.md. NEW operator follow-ups: (1) SECURITY — the customer-config git.token is a Gitea admin token with package-WRITE (proven during the bake); scope down + rotate. (2) optionally raise the global controller floor to 0.98.3.

• 2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published) — token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir. Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. OPERATOR: vouch agent 0.63.0 in the Day-0 manifest UI (until then fresh installs fetch 0.62.0).

• 2026-07-03 — GO-LIVE PACKAGE SHIPPED (host-install v1.9.1 + day0-install runbook + clean-room drill) — documentation/runbooks/day0-install.md is the operator Day-0 guide, every command executed verbatim on a throwaway nested PVE 9.2.2 (build server): install → verify → app deploy → uninstall → zero-residue diff → reinstall. v1.9.1 = residue-free uninstall (agent config w/ hub api_key, shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq snippets) + post-provision guest reboot (without it the controller never deploys on slower hardware) + header sync. Evidence + go/no-go for Peti (GO, 3 install-day items): documentation/audits/DRILL-day0-cleanroom-2026-07-03.md. Key OQ-2 finding: golden bakes a pre-floor controller 0.85.1 → fresh installs need the one-time D.1b update (agent restart → update button → 0.98.3); follow-ups recorded: rebuild golden ≥0.86.0 + re-vouch (operator), agent mkdir -p for the guesthook snippet dir, fresh-install local-API 401 (pre-existing), hub has no host-delete path. Drill VM parked virgin-snapshotted on 180 (/mnt/5_hdd/felhom.eu/drill/ — moved off the SSD 2026-07-18).

• 2026-07-03 — A1 CLOSED (host-install v1.9.0 + agent v0.62.0) — Pool.Audit added to FelhomAgentGuest; the agent's stale-lock reaper now intersects its scan with GET /pools/felhom membership (fail-safe skip on read failure). Spike: documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md. Deploy order on any box: rescope ACL first, then the agent. STILL PENDING (operator): rotate + de-git the 3 committed secrets in manifests/felhom.secret.yaml (spike appendix; REUSE §2 errata added).

• 2026-07-03 — hub v0.31.0 LIVE (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy; live probe: critical event stored + operator email delivered).

• 2026-07-03 — Felhom skills exist: skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}, installed via scripts/install_skills.py — junction mode (repo edits live immediately; no re-install needed after editing). Fresh-session discovery verified.

• 2026-07-03 — CLAUDE.md refreshed (all four repos): no version-pinned state in CLAUDE.md — current state lives here / CHANGELOG / REUSE.md.

• 2026-07-03 — REUSE.md exists at the repo root (hub helpers / website conventions / gate scripts / manifest patterns, code-verified); maintenance rule active: update it in the same commit that changes a shared helper. scripts/reuse_refs_check.py machine-checks its citations across all four felhom repos.