admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
790ec845f7988971a5da3c6f2a802aaaf322bc9d
felhom.eu/CONTEXT.md
T

650 lines
58 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CONTEXT — felhom.eu working state
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
- **2026-07-13 eve — ESCROW CEREMONY WIZARD SHIPPED (agent v0.88.0 + controller v0.127.0; this
repo: host-install v1.16.0 + RUNBOOK rewrite).** The recovery-code ceremony is now
customer-driveable from the controller (`/backup/escrow`): one-shot R reveal, re-stage-first,
Scenario-F stale-blob re-check (fired LIVE on both boxes' hash-less blobs at first ACK; drill's
repaired via a real ceremony — hub hash now covers). **Ruling F1 (2026-07-13), settled:** *R
transiting the Cloudflare tunnel once at reveal is an accepted risk — same trust class as the
claim code/login password; LAN-direct delivery is PARKED.* (threat model:
runbooks/RUNBOOK-escrow-ceremony.md). host-install v1.16.0 = FELHOM_ESCROW rides the canonical
sudoers fetch; hub `hostInstallVersion` synced (gate green), **hub NOT redeployed** (const rides
the next train). OPEN: supervised full-browser wizard pass (Viktor's drill login; also demo, to
clear its legacy stale warning); agent 0.88.0 publish + Day-0 vouch at the next train.
- **2026-07-13 — CLOSING BUNDLE SHIPPED: hub v0.53.0** — the last engineering items on the
pre-tester board: F-14 gated auto-Reissue (deletion provenance `host_deletions` in-tx +
`token_exists` gate → the EXISTING Reissue op + `pbsdr_auto_reissue` audit event; no record /
un-acked → refusal byte-unchanged), dead-host roll-up honesty (customer status = worst of
controller-derived + per-host `hostStatus` on dashboard//configs/detail, cause chip names the
host; the Peti 23h-masking shape is the pinned fixture), operator bearer OUT OF GIT
(`Secret/report-api` secretKeyRef non-optional + `REPORT_API_KEY` env override +
`scripts/manifest_bearer_gate.py`; VALUE ROTATION = supervised step in
documentation/runbooks/secrets.md — the git-history copy is dead only after it).
**Two operator rulings (2026-07-13), recorded verbatim — future sessions must treat these as
settled:**
1. ***F-14:** auto-Reissue of offsite credentials is permitted ONLY when the hub's own deletion
record shows the tenancy's owning host was removed through the escrow-ack flow — acknowledged
destruction is not silent re-keying. No record → manual Re-issue stays the only path.*
2. ***Roll-up:** a customer's status may never look better than its worst expected host. Single
staleness definition (`hostStatusOf`) — no second threshold anywhere.*
Gate additions: `python scripts/manifest_bearer_gate.py` after ANY manifests/ change (64-hex
bearer shapes; felhom.secret.yaml carve-out = visible KNOWN-BACKLOG line). GOTCHA: the repo
.gitignore `*secret*` pattern silently un-tracks any "secret"-named script — hence the
"bearer" name. The controller repo's controller.yaml.example carried the REAL bearer literal —
scrubbed (felhom-controller 51c871a).
- **2026-07-13 — POLISH BATCH SHIPPED: hub v0.52.0 (+ controller v0.123.0)** — take-two F-15
CLOSED (reset-request response carries the rotated code hash — ACK shape/guarantee — emailed
codes work immediately; live 1 s apply + first-try accept on Viktor's exact failure path) and
F-16 CLOSED (zero native `confirm()` in the hub UI — `inline_confirm.html` partial, Igen/Mégse
inline two-step, `scripts/hub_confirm_gate.py` enforces; danger-zone typed-confirm untouched;
live: offsite re-issue completed under browser automation without freezing). Operator resend
keeps the ACK lag — flash says so. Take-two §4 dispositions updated (F-15/F-16/zero-toggle obs
→ SHIPPED). Remaining from take-two: **F-14** (needs operator ruling: auto-Reissue re-adopt vs
never-silently-re-key) + S5 DR-restore drill; hub bearer key rotation still DUE.
- **2026-07-12 — DR-TIER-BY-DEFAULT BATCH SHIPPED: installer v1.15.0 + hub v0.51.0 (+ agent
v0.86.0 in felhom-agent)** — closes DRILL-day0-vm F-1/F-2/F-3/F-7/F-8/F-9/F-10 and F-6 (by
policy). **The five operator decisions (2026-07-12), recorded verbatim — future sessions must
treat these as settled:**
1. *Every install deploys the full DR plumbing uniformly (pbs-apply wrapper, `age`, WG enabled,
full ACL). **Capability is baked; activation is a hub flag.***
2. ***DR tier defaults ON for new customers.** Disabling is the per-customer exception — and it
removes the Felhom-side cost entirely (no PBS namespace, no endpoint dependency).*
3. ***Offsite app backup requires the DR tier ON** (fork-4 needs the ceremony; the ceremony
needs PBS). Enforced hub-side by extending the drill-proven dependency guard.*
4. ***Identity-only escrow is formally PARKED** — closed by policy, not implemented. Revisit
only if a customer class demands lean-tier offsite. Record in the ceremony runbook + CONTEXT
so future sessions know it was a decision, not an oversight.*
5. ***WG is base infrastructure, not a DR component** — always-on like cloudflared; it is also
the future OOB-management transport. BYO consent: a disclosure line lands in the
tester-agreement doc; per-customer hub disable remains possible.*
Mechanics: installer ships `felhom-pbs-apply` + `age` + `wg_tunnel.enabled: true` (byo wg-off
assert RETIRED — consent = typed disclosure ack + tester-agreement line), single VERSION source
gated by `scripts/hostinstall_gates.py` (cross-checks the hub Setup-tab const), default ACL
keeps felhom-pbs (doc advice REVERSED — narrowing caused the drill 403). Hub: per-customer
`dr_tier` flag (new default ON; legacy backfilled from descriptor reality, one-time),
cascade-not-error semantics + per-stage status on the edit form, WG-registration
auto-provision hook (scenario A hands-free), offsite-requires-DR guard (exact F-6 message),
host-page capability chips (agent 0.86.0 `inactive` = neutral; pre-1.15 "binary not found" →
migration hint). felhom-pve's latent F-3 state fixed live; Peti's box unreachable (his F-3
check + D/E/G deferred).
- **2026-07-12 — HUB v0.50.0 SHIPPED + LIVE: customer-claim password arc (closes DRILL-day0-vm
F-4/F-5).** The customer OWNS the dashboard password via a one-time claim code the hub emails to
the registered address (`bcrypt(code)` stored, plaintext only in the send). `internal/claim`
engine (EnsureIssued idempotent Day-0/live-box issue+email, Resend, RequestReset 3/day,
MarkClaimed set-only) + `store.customer_claims` (RotateClaimCode bumps generation + preserves
claimed_at) + delivery (config bake `web.claim_code_*`, report-ACK `claim` object, `POST
/api/v1/claim/reset-request`) + Setup-tab chip/resend + 3 Hungarian emails. Pairs with controller
**v0.122.0** (the gate). **LIVE-proven on drill box demo-vm-felhom: gate ON via the real edge (/ →
302 claim page, /api → 401); code emailed to the registered address.** Deployed felhom-hub:0.50.0
(ArgoCD synced/healthy). Floor raise 0.120→0.122 = operator's supervised LAST step. Observed: the
hub issues a claim code for EVERY reporting customer on first report (benign for pre-0.122 boxes —
no gate, ACK ignored). Report: documentation/audits/DRILL-day0-vm-2026-07-12.md (F-4/F-5 RESOLVED).
- **2026-07-12 — HUB v0.49.0 SHIPPED: Edit tab merge (edit-a) + scoped auto-refresh + riders**
(CHANGELOG v0.49.0; commits `e740147` → `2e03de1` → `1d94b1a` + docs/manifest). The standalone
customer edit page merged into the customer page's **Edit** tab (renamed from Settings): shared
`config_form_body` sub-template + `configFormData` builder (REUSE.md rows added), Block/Delete
relocated to a danger-zone card, header cleaned, `GET /configs/{id}/edit` → 302 `#tab=edit`,
post-action redirects anchored to their tab, `settings→edit` legacy-hash alias. Update handler
gained server-side required-field validation (error path preserves SUBMITTED values —
red-proofed). The 60s auto-refresh is now scoped: fires only on live tabs
(`data-live-tabs="overview,applications,events,host"`) and never while a form is dirty; muted
`(paused)` hint. Riders: `/style.css?v={{hubVersion}}` cache-bust on every template (closes the
v0.47.0 max-age=3600 gotcha) + the CLAUDE.md `git add -A` ban codified. **NOTE: the task spec
said v0.48.0, but v0.48.0 (app_start_failed) had already shipped — published tags are never
re-pointed, so this train is v0.49.0.** Out of scope (observed, not done): type-to-confirm for
customer delete (still `confirm()`, relocated as-is); the workspace-root `E:\git\CLAUDE.md` and
other repos' CLAUDE.md files still carry `git add -A` snippets — flag for their own sessions.
- **2026-07-12 — Day-0 VM DRILL COMPLETE (auto-confirm FIRST LIVE FIRING): full arc proven on a
fresh nested-PVE box** — appliance Day-0 → floor-at-first-report → ceremony → **auto-confirm
pending→escrowed in ~7.5 min, zero clicks** → offsite backup + restore round-trip. Two HIGH gaps:
**F-4 no operator password-set path exists (G10 unclosable, dashboards born OPEN — blocks tester
gate)** and **F-6 identity-only ceremony never implemented (no-PBS appliance can't escrow — drill
forked to PBS DR tier = full Peti-sequence rehearsal, all green)**. Installer fresh-box gaps:
felhom-pbs-apply not shipped (F-7), `age` missing (F-10), root-owned guests/ parents (F-3 —
check demo for the latent copy), silent root@pam rotation UX (F-8). Runbook fixes committed
(day0 A.2 anonymous-fetch; escrow-ceremony identity-only claim CORRECTED + age prereq). Report:
`documentation/audits/DRILL-day0-vm-2026-07-12.md`. Drill VM qm 300 kept (3 snapshots) for
re-drills; teardown list in report §9.
- **2026-07-12 — CAMPAIGN-3 Task A SHIPPED: agent v0.85.0 boot/recovery plane + appliance self-heal.**
Fixes F12 (CRITICAL boot ordering cycle — templates drop network-online, `MigrateNetworkUnits`
repairs installed units), F11/F10/F9 (reassert: fstype-driven classify, reset-failed+rearm, per-unit
verdict, + re-arm-on-guest-blind matrix-correction), F2/F1 (zero residue: reset-failed + rmdir), the
guest-hook rc255 belt (recover+timeout, wrapper no longer execs), and the appliance node self-heal
(`internal/selfheal`: host networking recovery, gated on `deployment_mode` — host-install `--mode`
templates it; byo remedy unreachable). 3 sudoers grants added (reset-failed, rmdir, start networking).
**Live matrix on demo felhom-pve: host reboot ×2 = 0 ordering-cycle lines both boots (F12 dead);
appliance self-heal recovered networking on attempt 1; F10 reset-failed+rearm sweep loud; zero
residue; guest-reboot heals.** **NEW finding F13 (HIGH, deferred):** active nfs4 under mp8 can fail
PVE rbind (rc255); NAS-down-at-boot → apps `Created` (fix-3). Peti's box (agent 0.81) unaffected
until his rollout — carries the F12 coin flip if it has a NAS share (interim drop-in in the agent
REPORT). host-install v1.14.0; scripts CHANGELOG. Details: `documentation/controller/network-storage-nas.md`.
- **2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; host recovered, RCA closed.** Unattended chaos campaign
22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/
restore byte-identical/EIO instant/stub badge + deploy-409 live-validated). The NAS automount
plane produced 2 CRITICAL + 3 HIGH: **F12 CRITICAL (= the overnight host loss, RCA'd next
morning): the agent automount template's `After/Wants=network-online.target` creates a boot
ordering cycle — systemd deletes an arbitrary job to break it; boot -1 sacrificed networking
(host dark 7h, hardware exonerated), boot 0 sacrificed the automount (NAS dead). Every host
boot with an enrolled share = coin flip until the template drops that ordering (`_netdev` on
the .mount suffices).** F10 CRITICAL (start-limit never re-armed, once blocked guest start),
F11/F9 HIGH (post-start reassert "skip-active" skips the healing automount restart; agent-restart
sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots; F7 HIGH (in-place dump write →
0-byte tar replaced last good backup during outage window). Fix-6 numbers: ring horizon 55min
idle / **6.5min under load**, wiped on every restart. Box left in defined state (6 wave apps
healthy, temps removed, NAS baseline-identical). Full ledger/evidence:
`documentation/audits/CAMPAIGN-3-2026-07-11.md` + `180:~/campaign3/`.
- **2026-07-11 — HUB v0.47.0 UI REORGANIZATION SHIPPED** (CHANGELOG v0.47.0; commits `9f29bf3` →
`0daddcd` + docs). Five deliverables: **(1)** `.data-table td a:not(.btn)` button-contrast fix;
**(2)** customer page = **8 hash tabs** (`#tab=…`, sticky summary strip, Events error badge,
no-JS graceful degradation — panels hide only via a JS-added body class); **(3)** shared
`host_detail_body` sub-template rendered by `/hosts/{id}` AND the new per-customer **Host tab**
(a list by design; `store.ListHostsByCustomer` + `hostDetailData` builder); **(4)** **stale host
removal** — `GET /hosts/{id}/delete-impact` (counts/booleans only) + `POST /hosts/{id}/delete`
behind type-to-confirm; ONLINE → 409 always (no override), escrow needs an explicit checkbox
(`ErrHostEscrowPresent`, tx never starts), one-tx cascade incl. the bound wg peer (wgsync's 5-min
declarative push converges the endpoint; log bundles die by `scope_id == host_id` only);
**(5)** **/offsite multi-endpoint management UI** — all `wg_endpoints` rows as cards +
add/edit/delete with 409 guards (peers-in-subnet pins subnet + delete). **Deferral (explicit):
peer allocation, the wgsync reconciler push and the desired-state merge stay lowest-endpoint-id
(`GetWGEndpoint` untouched); the future arc is a `wg_peers.endpoint_id` migration + per-endpoint
allocation/reconciler/desired-state.** Five red-proofs ran (online gate, escrow ack, bundle
scope, endpoint-delete guard, subnet-change guard). *Resolved:* the earlier ⚠️ about `146d165`
sweeping the Part-4 WIP — the sweep caught `hosts.go` mid-red-proof (escrow ack bypassed);
`068427a` restored the gate; the arc is now complete and author-reviewed. **Stale-doc note:** the
workspace-root CLAUDE.md's hub deploy wording predates GitOps — deploys are manifest-tag bumps +
deliberate ArgoCD sync, never `kubectl set image`.
- **2026-07-11 — RCA FIXES 1+2+4 SHIPPED + Q1c GREEN (agent v0.84.0 + controller v0.117.0 LIVE on
demo).** Reboot survival: automatic since agent 0.84.0, live-proven 2026-07-11 21:20 (`pct reboot
9201` under operator GO — the hook re-armed + verified nas-media DURING the start task, CWA
auto-started healthy, health ok first cycle, no badge). Agent: `ReassertNetworkMounts` (hook
post-start + startup; stop+enable-now on existing sudoers verbs — NO sudoers widening needed;
never in periodic paths). Controller: statfs consuming-namespace classifier → probe
`not_network_fs` (stub can never verify), deploy-time stub refusal (409; idle autofs deploys
fine), distinct stub badge (wins over unreachable; unreachable line byte-identical), deploy view
selects stored HDD_PATH (RCA fix 4). MinAgent unchanged 0.81.0. All red-proofs recorded in repo
REPORTs. Feature doc: `documentation/controller/network-storage-nas.md` §Reboot survival; audit §7
follow-up added. Queued: RCA fixes 3/5/6; Peti publish train now carries 0.84 + 0.117.
- **2026-07-11 — NAS/CWA RCA (diagnostic, no fixes):** `documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md`.
The nas-media re-add verified legitimately (new host automounts DO propagate into running
guests); a second `pct reboot 9201` at 15:39 severed the trigger → guest-side stub; CWA
deployed correctly to the NAS (app.yaml + binds + data all nas-media) but is DOWN since 15:39
(docker can't recreate the bind source; perms luckily blocked any system-drive writes; zero data
loss). UI "felhom-usb" = display bug (deploy.html selects by IsDefault, never stored HDD_PATH);
hub device-ID WARN was CORRECT and the only guard that fired; NAS badge masks because it asks
the agent (host ns). 6 prioritized fix tasks in the audit §5. **Remediation §6 PERFORMED on
operator GO (~20:14–20:20 CEST): automount-unit restart re-planted the trigger in the running
guest instantly (fix-1 mechanism live-proven), CWA restarted healthy on the NAS library,
uid-1000 write OK, controller health back to status=ok at 20:18. Demo fully healed.**
- **2026-07-11 — OBSERVABILITY PASS (hub v0.46.0 LIVE Synced/Healthy; agent v0.83.0 + controller
v0.116.1 on demo).** Remote pull-only log access to both box components' always-DEBUG capture
rings. Hub: `log_bundle_requests`/`log_bundles` store (gzip, newest-3, **72 h TTL** on the 60 s
sweep), the **secret gate fail-closed** before storing (`blocked: possible secret`; `[REDACTED]`/
checksums pass), report-ACK `controller_log_requested` + heartbeat-envelope `log_tail_requested`
channels (consume-once; pre-0.83 agents stay visibly pending), host-detail Diagnostics section
(Request controller/agent logs + View/Download; the hosts read-only invariant amended: these two
forms are the only actions). Conventions codified:
`documentation/runbooks/logging-conventions.md` + CLAUDE.md one-liners in all three repos.
**OPEN: operator clicks both request buttons on hub.felhom.eu/hosts/<demo> to close the live
round-trip** (UI password-gated; the flow is fully API-test-proven). Hub bearer key rotation
still DUE (pre-existing).
- **2026-07-11 — PUBLISH-TRAIN RULES CODIFIED** (`documentation/runbooks/publish-train-rules.md`):
1. manifest before floor; 2. the manifest screen carries the LIVE DB floor
(`hub_settings.min_controller_version` overrides env, acts immediately) — save the floor field
LAST; 3. MinAgent gate — **now HUB-ENFORCED PER-BOX (hub v0.45.0, `store.ResolveManagedFloor`),
the manual fleet check retired**; 4. box-level backstop = the controller's agent-capability gate
(v0.114.0 route probe → v0.115.0 version compare). (Floor-UI separation + per-box gating SHIPPED
in hub v0.45.0 — see the backlog-clear entry above; this rules-doc note supersedes the original
"hub untouched / option-3 deferred" wording.)
- **2026-07-11 — NAS/coupling backlog CLEARED (hub v0.45.0 LIVE + agent v0.82.0/ctrl v0.115.0 on demo).**
Composite A–F. Hub v0.45.0 (Synced/Healthy): floor-UI separation + effective-floor source line
(DB `hub_settings` vs env, `store.ResolveGlobalFloor`) + type-to-confirm w/ live below-floor count;
per-box MinAgent conditional floor (`store.ResolveManagedFloor` HOLDS the controller floor for a box
whose `hosts.agent_version` < the manifest's MinAgent, flags it on the Hosts dashboard) — manual
fleet check retired (publish-train-rules.md rule 3). THE one comparator extracted to
`internal/semver`. Agent v0.82.0 = `X-Felhom-Agent-Version` header (demo-only, Peti stays 0.81 =
probe fallback); ctrl v0.115.0 = version-aware Supports + DSM-validated NFS guidance. **DSM spike**
(real DSM 7.2 via virtual-dsm): consumer recipes validated E2E, hardlink caveat closed for Synology
(record: documentation/audits/SPIKE-nas-dsm-2026-07-11.md); QNAP caveat stands. **Q1c (supervised
reboot) FAILED**: a NAS automount trigger does NOT survive a guest reboot (empty-dir; agent lacks a
network-mount reassert) — fix spec'd at documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md.
NOT published (agent 0.82 demo-only). Record: felhom.eu/REPORT.md + each repo's REPORT.
- **2026-07-11 — PUBLISH TRAIN 0.81/0.113 COMPLETE — PETI LIVE on agent 0.81.0 + controller 0.113.0.**
Record: `documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md`. Agent 0.81.0 published
(sha `914bb3b5…30df0b`, live-bytes, anon-verified) + golden 0.113.0 baked/published (sha
`66de5e64…82054c`, drill VM, virgin-reverted); manifest + signed `agent_update`
(key-id `felhom-op-1`) by Viktor → Peti 0.79.0→0.81.0 clean (no rollback); **floor auto-confirm
proof CAPTURED** (10:41 0.112.0 → 10:42 0.113.0, anonymous pull proven on HIS box) — KEY LEARNING:
the hub UI manifest screen ALSO sets the GLOBAL floor as a DB `hub_settings.min_controller_version`
row that OVERRIDES the env and acts immediately (this is how Peti reached 0.112 and now 0.113
before the GitOps env bump `16390d0`); a transient forbidden skew (ctrl 0.113 + agent 0.79,
~10 min) was benign only because zero NAS shares existed — next train: save the floor field LAST.
OPEN: Peti journal-group one-liner (E), temp Git Sync creds deletion (G1), **hub bearer key
rotation now DUE** (committed in manifests/hub.yaml + screenshot-exposed this session).
- **2026-07-11 — NAS verify-before-commit SHIPPED (agent v0.81.0 + controller v0.113.0 LIVE on demo;
host-install v1.13.0).** The spike's production task landed same-day: add = agent verify (retry=0
unit, journal-classified, auto-rollback) → controller uid-1000 write probe (--netprobe re-exec) →
register LAST; full rollback on any failure; §3.2 Hungarian error map (nfs_export merged);
storage_network.html redesigned (SMB-first, Route-A guidance); orphan rows; systemd-journal group
(NO sudoers). Live A–E validated vs an isolated sim NAS, Route A proven in production (alien-uid
1060 server-side). Feature doc: `documentation/controller/network-storage-nas.md`. NOT published
(no Gitea/Day-0 bump; Peti pending with the usermod one-liner). Record:
`felhom-controller/REPORT.md`.
- **2026-07-11 — SPIKE: NAS verify mechanisms + NFS squash matrix — ALL FIVE QUESTIONS ANSWERED, no
blocker.** Record: `documentation/audits/SPIKE-nas-verify-2026-07-11.md`. Q1 in-guest access DOES
wake the host automount (make-or-break PASS; container bind-mount too); Q2 controller drops to uid
1000 and writes (setpriv proof; SETUID/SETGID caps present); Q3 **Route A ACCEPTED-with-caveats**
(`all_squash` to ANY server uid = full rw+hardlink for the guest app; guest sees nobody; chown
EPERM-clean, chmod persists; June's `anonuid=1000` WRONG case pinned = anonuid without all_squash
is a no-op for non-root); Q4 error taxonomy captured (NFSv4 makes no-export vs not-permitted
INDISTINGUISHABLE; systemd 90 s start-timeout is the real ceiling; **`retry=0` → production unit**:
dead-NAS access 91 s→3.8 s; verify must NOT ride the 15 s agentapi timeout → own path/status-poll,
~120 s budget); Q5 plain-user SMB PASS incl. hardlink. Teardown verified to pre-counts. Production
TASK is the operator side's next move.
- **2026-07-11 — PBS DR tier SLICE 2 SHIPPED (agent v0.80.0): the apply-bridge — demo
ADOPTION-PROVEN live + PUBLISHED; the supervised Peti phase PENDING.** Full chain fired on the
demo: hub enable (storage-id `felhom-offsite`) → tenantsync provisioned ns `demo-felhom` on ep0
→ agent ADOPTED the manual entry (tenancy entry-owned, "entry wins"; staged secret asserted
UNCONSUMED; DR green; `escrow.pbs_storage_id` seeded). Laws red-proof-verified: set-only (K
destruction guard), secret-on-stdin, verify-pin-before-consume, loud consumed-failed. Part 0
evidence: write path needs the §4b dual-grant exactly (403 `Datastore.AllocateSpace` → re-assert
→ token vzdump OK); demo grants restored. Live finding fixed: escrow seed = in-place write
(/etc/felhom-agent dir is root-owned; rename impossible for the non-root agent). 0.80.0
published (`f2ba62ca…f06d2`, == deployed bytes). **NEXT = RUNBOOK-peti-pbsdr-2026-07-11.md**
(Peti box prep: sudoers+wrapper+wg_tunnel enable — no tunnel/OOB on his box today; Viktor opsign
0.80.0; hub enable → FRESH path, K born, degradation clears; ceremony → auto-confirm → first
offsite run = the arc closes). Detail: felhom-agent/REPORT.md + CHANGELOG v0.80.0.
- **2026-07-10 — PBS DR tier SLICE 1 SHIPPED+LIVE (hub v0.44.0): ep0 tenantsync surface + hub
provisioning.** `felhom-tenantsync` v1.0.0 installed on ep0 (second forced-command key, peersync
untouched — hashes asserted; smoke caught + fixed the client `{"data":[…]}` json wrap; orphan
`root@pam!spike` cleaned). Hub: `internal/tenantsync` (pinned-SSH twin of wgsync; errors never
embed stdout), config-form "PBS DR tier (ep0)" section → WG-peer check (fail-closed) → provision →
`host_pbs_secrets` consume-once (agent fetches `POST /api/v1/hosts/{id}/pbs/consume-token` —
note: /hosts/, not the task's /host/) → `pbs_dr` descriptor into HOST desired_json + gen bump;
idempotent re-save; Re-issue button (F4 twin). Red-proofs: consume-once / fail-closed /
idempotency all mutation-proven. **NOT live-provisioned — slice 2 (agent bridge) + supervised
Peti enable pending.** Runbook §10 addendum in offsite-endpoint.md. Commits ce6a566 + 32ca1f6.
- **2026-07-10 — SPIKE: PBS DR tier auto-provisioning SLICE 0 — all three mechanisms SETTLED.** Record:
`documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`. (1) PVE token storage-create **impossible**
(create/modify/delete all check `/storage` root) → **sudoers vector proven** (pinned-id alias, cycle green as
the agent user, unpinned denied; secret must ride a stdin-wrapper, never sudo argv). (2) ep0 tenancy op-set +
one-time-secret-over-SSH-stdout **proven live on ep0** (🛑 Viktor-approved; read/write cross-tenant 403s);
current peersync channel can't carry it → slice 1 ships a second forced-command surface (`felhom-tenantsync`).
(3) `pvesm add pbs --encryption-key autogen` births K at `PBSEncKeyPath` + vzdump lands ciphertext (doc-06
property) + escrow-create finds K. **⚠ `pvesm remove` DELETES K** → re-apply must be set-only. All spike
artifacts torn down; flags: orphan `root@pam!spike` on ep0 scratch; demo's §4b step-3 storage grants absent.
Epic slice map in the record §5.
- **2026-07-10 — new-box restore round-trip VALIDATED + old-box archive RETIRED.** Record:
`documentation/audits/VALIDATION-newbox-restore-2026-07-10.md`. Both toggled apps restored from the
production pool box via the real endpoint — **9/9 sha256-identical** (immich 44MB db-dump exact,
`d78466fcbf…4526b`, agreeing with the campaign's independent round). ~10s/app. **Viktor's decision:
RETIRE the old archive** — he deletes `u629193-sub1` in the old project's console (the PBS-storage-1 box
itself stays for dev). C2 fixed same day as controller v0.110.0 (crash-lock self-heal, live-proven).
Offsite arc: build-complete + restore-proven both directions; remaining = the Peti onboarding runbook.
- **2026-07-10 — offsite arc overnight soak (unattended) — PASS + 1 HIGH finding.** Record:
`documentation/audits/CAMPAIGN-offsite-overnight-2026-07-10.md`. F4 re-issue proved LIVE (reset invalidates
the old password); restore round-trip 9/9 byte-identical new-box↔USB (immich db-dump exact → old-box
archive `u629193*` safe to retire, Viktor's decision); single-flight + crash-restart (zero consume, escrow
untouched) PASS. **FINDING C2 (HIGH, OPEN):** a controller crash MID-PRUNE leaves a stale restic exclusive
lock the controller can't self-heal → offbox runs fail until manual `restic unlock` (fix candidate:
`ensureOffboxRepo` auto-unlocks a self-owned stale lock). Track B (live quota-crossing) + hub-orchestrated
Track A blocked — the hub operator config surface is session/CSRF-gated (unattended can't reach it) → Peti
runbook. Demo restored to start-state. One expected `backup_failed` email (the crash test).
- **2026-07-10 — SLICE 4 soft-quota + offsite health SHIPPED+LIVE (hub v0.41.0 + controller v0.109.1) —
the offsite arc is BUILD-COMPLETE.** Controller: `quota_gb` rides the descriptor into the target (hash
includes it → quota changes re-apply via key-auth-first, no consume); `repo_size_bytes` from restic
stats; pre-run gate (≥100% refuses NEW backups Hungarian-loudly, **prune still runs** — red-proofed;
restore never gated; ≥80% warns); /backups usage bar. Hub: `monitor.OffsiteChecker` (fill 90/95 of
quota + `offsite_stale` >48h for enabled+escrowed only — red-proofed; nil-safe on old reports) +
**manual-only Freeze/Unfreeze lever** (readonly on the exactly-1 labelled sub-account, SSH preserved).
**Live:** key-auth-first proven against the real box (the deploy's own re-apply); that re-apply exposed
+ fixed v0.109.1 (re-apply must preserve EscrowState + runtime status — it demoted the escrowed demo);
demo repaired (manual confirm + run: escrowed, 45 MB / 50 GB, checker healthy 0 events, usage bar
renders). Freeze + 100%-refusal NOT live-exercised (STOP; red-proofs cover). NEXT: Peti onboarding
runbook, NEW-box restore round-trip → old-box retirement.
- **2026-07-09 — SLICE 3 hub-verified escrow auto-confirm SHIPPED+LIVE (hub v0.40.0 + controller v0.108.0 +
agent v0.79.0).** Replaces the operator-trust confirm with a verified fact: ceremony hashes the sealed
repo password (agent, `restic_pw_sha256,omitempty` on the escrow upload — contract test moved in lockstep)
→ hub stores it (additive `host_escrow` migration, NULL on legacy rows) + serves
`escrow:{identity_blob_present,restic_pw_sha256,created_at}` in the report ACK → controller
(`report.EscrowAutoConfirmer`) flips pending→escrowed ONLY on hash match with its local repo password.
**Red-proofed core: blob-presence alone never confirms** (stale blob = false custody claim); mismatch
warns once per hash naming the ceremony; never un-confirms; flip wipes the agent-staged secret. Hashers
pinned by the same cross-repo vector. Manual confirm = deprecated fallback (legacy blobs — demo included,
hash-NULL verified live; the escrowed demo produced zero confirm activity = Scenario E live). Full happy
chain proves out at the next real enable (Peti). NEXT: SLICE 4 soft-quota, Peti runbook.
- **2026-07-09 — offsite hardening bundle SHIPPED+LIVE (hub v0.39.0 + controller v0.107.0 + agent v0.78.0).**
The e2e's sharp edges closed before slices 3–4: **F4** "Re-issue offsite credentials" (explicit operator
recovery for consumed-password dead-ends — exactly-1 label scope, fresh one-time secret, ConfigVersion
bump; `hetznerapi.ResetBoxPassword` added for dedicated); **F2** host-key scan retry (~60s ladder — first
save survives fresh-subaccount DNS lag); **F5** submit-disable + in-flight notice; **key-auth-first bridge**
(working installed key → re-pin+reconfigure WITHOUT consuming; fingerprint verify never bypassed; kills the
stale-descriptor consume-404 loop); **staged-secret wipe** (agent `DELETE /escrow/stage-secret`; controller
calls it on every flip to `escrowed`; the e2e's lingering staged file wiped live through the new endpoint,
idempotency proven). All red-proofed. F4 reset NOT exercised live (demo credential untouchable — throwaway
customer under supervision when needed). NEXT: SLICE 3 auto-confirm, SLICE 4 soft-quota, Peti runbook.
- **2026-07-09 — offsite provisioning VALIDATED LIVE end-to-end (hub v0.38.1 + controller v0.106.1) — PASS.**
Full loop through the real pipeline on `demo-felhom` against the new dedicated-project pool box (611714 /
`u629488`, project 15282031): UI save → sub-account 269008 provisioned → descriptor v6 + one-time password →
controller config-refresh (≤15 min) → apply-bridge (verify-pin → consume-once → pinned `ssh-copy-id` →
`EscrowState=pending`) → run-gate refusal proven → escrow confirm (Option A: the repo password was already
escrowed — `WriteOffboxSecrets` preserves it) → first run: **restic repo + 2 snapshots on the new box,
key-only**. Two live bugs found+fixed mid-run: **F1** hub v0.38.1 (`applyOffsite` on `context.WithoutCancel` —
a re-click canceled the provision between create and password-store, stranding the sub-account) and **F3**
controller v0.106.1 (`ssh-copy-id -s` needs `~/.ssh` to exist in the container — died AFTER consuming the
password). OPEN: F2 (fresh sub-account DNS lags → first-save scan 502s; retry-with-backoff), F4 (hub has no
password-reset path — recovery today is delete-subaccount + re-save), F5 (no save spinner → re-click bait).
Phase-0 wiring: `manifests/hub.yaml` now injects `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID` from
Secret/storagebox; scoped-token check = list boxes with it (must see ONLY 611714). Old-box offsite kept as
frozen archive until a NEW-box restore round-trip. Record:
`documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`. NEXT: SLICE 3 (auto-confirm),
SLICE 4 (soft-quota), F2/F4/F5, Peti onboarding runbook.
- **2026-07-09 — offsite provisioning SLICE 2 (hub v0.38.0 + controller v0.106.0).** The controller apply-bridge:
on startup it reconciles the hub-served `offsite:` descriptor into a key-only offbox target
(`controller/internal/offsiteapply.Bridge`) — **verify-pin the box host key against the hub-captured
`host_fingerprint` (no blind TOFU)** → consume the one-time password (single-use) → `sshpass ssh-copy-id -s -f`
install → configure offbox → `EscrowState="pending"` → persist a descriptor-hash marker. Idempotent +
fail-safe; both red-proofs green. Hub v0.38.0 adds `Descriptor.HostFingerprint` captured via an
`x/crypto/ssh` keyscan (fail-closed). **NOT yet live-applied** — supervised end-to-end (hub provisions on
the new pool box → controller apply) is the next runbook, gated on the hub's new scoped `HETZNER_TOKEN`.
NEXT: SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota).
- **2026-07-09 — hub offsite provisioning SLICE 1 (hub v0.37.0).** The hub can now provision the offsite tier
on operator enable: `internal/hetznerapi` (typed client, base **api.hetzner.com/v1**, `CloudAPI` interface +
exported `Fake`, `WaitAction`), `internal/offsite` (`Provisioner.ProvisionOffsite` — idempotent by label
`felhom-customer=<id>`, shared sub-account / dedicated box, generates the transient password, builds the
NON-SECRET descriptor, **fail-closed**), `one_time_secrets` store (single-use `Save/ConsumeOneTimeSecret`),
`POST /offsite/consume-password/{id}` (customer-API-key auth, once), config-form Offsite section →
`applyOffsite` (502 + no-save on error) → descriptor in ConfigJSON → ConfigVersion bump. Token from
`HETZNER_TOKEN` (out-of-band); password NEVER in ConfigJSON/logs. Unit-tested vs a faked Cloud API (+ a
fail-closed red-proof). **NOT yet live-provisioned** — awaiting the **dedicated-project scoped token** (the
current token can delete ep0). NEXT: SLICE 2 (controller apply-bridge consuming the descriptor + one-time
password), SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota). Deployed via GitOps.
- **2026-07-09 — Hetzner API provisioning spike (offsite tier).** Both offsite models are API-provisionable
(FIT). **#1 correction: the Storage Box API is `https://api.hetzner.com/v1`, NOT `api.hetzner.cloud/v1`**
(Cloud API 404s for storage boxes); same Bearer token. Proven live on `spike-*` (torn down): sub-account
create/reset/readonly/delete + a billable bx11 box create/rescale(`change_type`)/delete. Create takes a
**password (no ssh-key param)** → transient-password→`ssh-copy-id -s -f`→restic-reach→`reset_subaccount_password`
(transient pw invalidated, key survives) = key-only steady state. RepoPath `/home/<repo>` (chroot). Quota:
dedicated=box type/`change_type` (hard); shared=no native quota → Felhom soft-quota + `update_access_settings{readonly}`
freeze. Idempotency = **label_selector** (box names not unique). Rate 3600/h. **SECURITY: the token is
project-wide unscopeable Read+Write — it can also touch ep0 (the prod offsite server) + 3 SSH keys →
move customer boxes to a DEDICATED Hetzner project before the hub holds a write token.** Record:
`documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md`. NEXT: the hub-provisioning impl.
- **2026-07-09 — fork-4 custody VALIDATED end-to-end live (supervised).** The full custody cycle proven on
the demo (agent v0.77.0 + controller v0.105.0): enable → controller pushes the repo password to the agent →
**real escrow ceremony** (`--selftest=escrow-create`, K + restic-pw under a fresh R, uploaded to the hub,
staged secret wiped, K byte-unchanged) → **atomicity proof:** a run **skipped** before `confirm-escrow` and
**proceeded** after (2 apps, 2 snapshots) — the gate opens exactly on confirm. **DR money shot:** the restic
password recovered from the **production hub blob** (`host_escrow.identity_blob` for `demo-felhom-01`, via
`identity-consume` with R) is byte-identical to the live password (`sha 24c44f49…`) and **decrypted the real
box repo**; the real `inject-password`→`restore` endpoints reproduced a 45 MB db-dump byte-perfect. Demo
offbox left **escrowed** (un-paused); Viktor holds R out-of-band; no secret in any log; live `repo_password`
unchanged. Record: `documentation/audits/VALIDATION-fork4-custody-2026-07-09.md`. **NEXT:** hub-provisioning +
soft-quota spec (hub-verified auto-confirm replacing operator-trust; both storage models); per-customer
onboarding runbook; the `identity-consume` OK-message fix (omits the recovered restic field).
- **2026-07-09 — offbox v0.104.0 VALIDATED live against the Storage Box (supervised).** Closed the offsite
arc: controller `:0.104.0`@`908e4b9` ran the real `/backup/offbox/*` endpoints against a fresh box
**sub-account** (`u629193-sub1`, RepoPath `/home/felhom-demo-repo`). The capstone proof — **immich**
(undeployed, recovery unit only on the USB drive, the exact case v0.103.0 mis-resolved to `systemDataPath`)
→ discovery found the USB unit → `ok`, 1 snapshot; 45 MB db-dump restore sha256-matched. No-silent-success
confirmed on the live box: 0/N → `error` + `backup_failed` operator alert; partial → `ok` +
Hungarian `LastWarning` on `/backups`. Persists across restart. **Forks:** (1) direct topology + (2)
sub-account-per-customer (RepoPath `/home/<name>`, `ssh-copy-id -p 23 -s -f` — console SSH-key field still
broken, F3) confirmed; (3) PBS/WG tier deferred; **(4) custody surface RECORDED, escrow design OPEN** —
the per-customer restic repo password + SSH key are 0600 in the controller data dir, **NOT independently
recoverable** (ride DR only via the PBS whole-CT) → the load-bearing follow-up before real reliance.
**KEPT** as a standing demo-offsite (€4.06/mo). Record: `documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md`.
- **2026-07-09 — TASK GL-7 SHIPPED: customer-page passphrase hardening + install-command generator
(hub v0.36.0, `844fbfa7`; deploy `02c748eb`; LIVE, ArgoCD Synced/Healthy)** — go-live G7/G9/G10/G11.
Two coupled, security-first changes to the operator customer page (`customer_unified.html` +
`configs.go`). **(1) Passphrase hardening:** the per-customer retrieval passphrase was cleartext
twice (visible `#retrieval-pw` + baked into the Option-3 debug curl `X-Retrieval-Password:` header);
now masked-by-default with reveal (`toggleSecret`) + copy (`copySecret`), value in `data-secret`
(existing model), and the Option-3 command carries a `<YOUR-RETRIEVAL-PASSWORD>` placeholder — no
secret in any copyable command. (Zero-secret-in-DOM fetch = noted future follow-up, NOT scoped.)
**(2) Install-command generator:** the three hard-coded `<code>` blocks became a client-side builder
(vanilla JS, no framework/CDN/network) emitting ONLY real host-install v1.12.0 flags, download-then-
run, byo-requires-caps enforced client-side, CustomerID prefilled from the server, JS-off static
fallback; the seven dangerous/operator-only flags (`--force`/`--rotate-recovery`/`--enable-oob`/
`--remove-golden`/`--uninstall`/`--adopt-pool`/`--rescope-acl`) are never offered as controls. Tests
`TestTemplates_PassphraseHardened` + `TestTemplates_InstallGenerator` (green; passphrase test
red-proofs). **Tester agreement written** (`documentation/pilot/PETI-tester-agreement.md`): BYO
trust boundary, honest limitations (no offsite backup, F2 removable-drive, F7 pool-reassert, F6
`:53`-free), exit rights, Peti box facts (80c/128 GB → caps 12/32768, `sajatfelhom.hu` re-point),
byo install command, onboarding sequence. GO-LIVE-PACKAGE: GL-7 ✔; G9/G10/G11 → done. **One open
pilot question (owned by the agreement §6):** Peti's local backup TARGET — does his box have a
second disk/pool? → sets `--acl-storages` at onboarding. Deploy gotcha: first build was stale
(local commits not pushed before `build.sh` pulls origin) — pushed then rebuilt. **NEXT: Peti day-0
onboarding (operator-run).**
- **2026-07-08 — TASK GL-8 SHIPPED: BYO coexistence hardening (host-install v1.12.0, `18a556a5`)** —
the GL-6 findings F3/F6/F1. **F3:** the populated-host leaf guard fires only when a `felhom_guests`
guest exists (not `pct list`) → a fresh byo install on a host running only the OWNER's guests needs
no `--allow-new-leaf`; the pin protection for real Felhom guests is unchanged. **F6:** byo preflight
REFUSES a foreign process on `:53` with remediation and mutates NOTHING (Viktor's ruling: refuse+
instruct only — the dnsmasq analog of the break-glass no-touch rule; appliance untouched). **F1:**
uninstall purges the config `.bak*` siblings (a GL-6 residue held a live hub api_key). F6
live-proven on felhom-pve (byo preflight refuses at :53 — the exact GL-6 scenario, read-only die).
Harness 36/36 (+GL8-F3/F6/F1 static, GL8-F1b behavioural, GL8-F6 live) + red-proofs RP-F3/F6/F1;
GL-2/GL-4 regression green; shellcheck clean. Shipped BEFORE GL-7 so Peti's command needs no
workaround. **F2/F7 remain open (non-blocking). NEXT: GL-7.**
- **2026-07-08 — TASK GL-5 SHIPPED: DR bring-up bind overrides (agent v0.75.0, LIVE + scratch-DR
validated)** — go-live G8; docs-only here (GO-LIVE-PACKAGE G8/G6/G1 flips + decision log; the
code + full report live in felhom-agent `b3446213`). Guest-loss DR restore WORKS now: the two
structural binds (mp8/mp9) are overridden with throwaways at restore and swapped for the REAL
binds pre-start (root pct via the new Engine HostRunner seam), displaced unusedN deleted. TWO
live-discovered PVE rules beyond the spike: explicit-params restore needs an explicit rootfs AND
silently DROPS unlisted mountpoints — the full param set now derives from the archive's embedded
config (`ExtractArchiveConfig`, 200 under the scoped token). Live proof: scratch 9310 from a real
9201 archive — mp0 200G + mp1 50G restored WITH content (7m23s), real binds, zero residue, clean
teardown; the same op failed outright on v0.74.0. OPERATOR: **publish agent 0.75.0 + bump the
manifest to it** (vouching 0.74.0 ships broken DR). Follow-ups: restore-test boot-verifies
without mp0/mp1 (same PVE rule, reverse direction — candidate reuse of drRestoreOverrides);
DR-selftest KeepMAC duplicates a live source's MAC briefly; no host-loss DR runbook exists yet
(GL-6/S5 writes it).
- **2026-07-08 — TASK GL-4 SHIPPED: uninstaller gap-closure + operator-key day-0 fold
(host-install v1.11.0)** — go-live G6 + the G1 key-pin follow-up; **awaiting GL-6** for the real
(non-dry) teardown + armed-pin install. Uninstall now removes the self-update artifacts (4b4:
guarded wrapper, .prev/.new.* slots, rollback unit, limits drop-in — derived from
configs/felhom-selfupdate-guarded), unmounts enrolled drives under /mnt/felhom-drives (plain
umount ONLY, busy = warn+guidance, root-bind guarded), and ends with a KEPT-vs-WIPED statement
(drives/PBS/hub record/escrow/vaulted recovery credential live on; guest-only mode prints the
vmid's bind-store drives). Key-pin: OPERATOR_KEY_* constants (EMPTY until the operator ceremony)
+ `--operator-pubkey-file` (validated at argv, comment=key_id required) → authz.signers written
at step 6 per the agent SignerKey schema; **no-keys-resolved KEEPS preserved signers** (never
un-pin a manually-pinned box); verify reports armed/dormant (dormant = WARN). Harness 28/28 on
felhom-pve incl. the NEW GL4 H-U full-uninstall DRY transcript vs live 9201; red-proofs RP-1..3
green. **GO-LIVE-PACKAGE.md was ABSENT AGAIN** (spec said the operator attaches it; not present
in the repo or workspace) — G6/G1 status recorded here, doc still pending its operator commit.
Key CEREMONY (real keypairs + felhom-pve pin) = operator step.
- **2026-07-07 — RUNBOOK GL-1 EXECUTED: agent 0.74.0 + golden 0.103.0 PUBLISHED** — go-live G1
(partial): the published-artifact chain is current. Agent = the LIVE felhom-pve bytes (sha
`1ec3f588…76af05`, provenance preserved, publish-agent.sh round-trip verified); golden baked
clean-room in the drill VM (build-golden.sh v2.0.0 @ ced60ddf, controller 0.103.0 baked, all
integrity gates green, sha `8481e8a1…6026e`, 583MB) and published HTTP 201. Both artifacts
fetch-verified **ANONYMOUSLY** (Gate 3c finding: generic packages are world-readable → G3
git.token read-only rotation CANNOT break fetches; but the INSTALLER dies on an empty token —
script-side requirement, not Gitea's). Drill VM restored to `virgin`; evidence
`~/drill/bake-0.103.0.log` on 180 + `documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`.
**OPERATOR NEXT: bump the hub Day-0 manifest** to agent 0.74.0 / golden 0.103.0 (the four values
are in the execution record) — until then fresh installs land 0.63.0/0.98.3. Also recorded:
operator-key pin follow-up (fold into GL-4); GO-LIVE-PACKAGE.md still not in the repo (G1 status
recorded here instead).
- **2026-07-07 — TASK GL-2 SHIPPED: `--mode appliance|byo` install profile (host-install v1.10.0 +
harness)** — go-live items G2/G4/G5 implemented, **awaiting the supervised GL-6 drill** for live
provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old
`--mode provision` value retired with a specific die, `dr` stub kept); break-glass gated at its
CALL SITE (byo never touches root@pam); byo requires `--cores`+`--memory`, refuses
`--enable-oob`/`--rotate-recovery`/non-9.x-PVE/missing `--acl-storages`; disclosure + typed-hostname
ack; **the config write's `lan_resolver` default flips to OFF in byo** (the spec assumed
default-off — it was default-ON; appliance unchanged) + post-write asserts refuse
lan_resolver/wg_tunnel/oob=true; pool+ACL verify asserts in BOTH modes (R2 lesson);
`--preflight-only` (no state, PASS/FAIL verdict, proven live on felhom-pve). NEW
`scripts/hostinstall-mode-harness.sh` (static C1–C4 + invariants everywhere; PVE tier: C5 +
A/B dry transcripts) — 16/16 PASS on felhom-pve incl. C5 live; red-proofs RP-1..3 run→fail→revert.
FINDING: demo-felhom's controller.yaml has EMPTY git.username/git.token → any step-5
(re)install for that customer dies until the operator sets real Gitea read creds (relates to the
standing scope-down+rotate follow-up). `documentation/pilot/GO-LIVE-PACKAGE.md` is NOT in the
repo and no operator copy was provided — the G2/G4/G5 status flip is recorded here instead;
add the doc when the operator supplies it. Next: **GL-6 supervised drill** (appliance + byo
end-to-end, uninstall/re-provision, C7, the ack prompt interactively).
- **2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent
v0.72.0)** — merged E1+H1. Hub: operator OOB peer (`store/wg_operator.go`, `PUT/GET
/admin/wg/operator-peer` global key) + `oob_peer_ip`/`oob_operator_ssh_key` in the desired-state
wireguard block + `monitor/host_oob` (oob_degraded/recovered). Installer: `--enable-oob`/`install_oob`
(static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box
drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as
felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt
drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key.
- **2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent
v0.71.0)** — prereq for felhom-sshd (H1). Hub: `store.host_recovery` vault (per-host root@pam
console password; `PUT /hosts/{id}/recovery-credential` self-scoped + `GET /admin/hosts/{id}/
recovery-credential` global-only, secret never logged) + `host_mgmtplane` monitor raising
`mgmt_plane_healed`. Installer: `step_break_glass` (generate+set+vault root@pam, `--rotate-recovery`)
+ `install_mgmt_watchdog` (the 3 host artifacts, RuntimeDirectory-guarded). Live-validated:
agent-independent `/run/sshd` auto-heal in 30.0s; warning fired end-to-end; break-glass
vault→retrieve→PVE-ticket(200). felhom-pve root@pam is now the vaulted value.
- **2026-07-04 — S3 SHIPPED: agent WG tunnel (felhom-agent v0.64.0; docs-only here)** — the
doc-06 §3.3 handshake is now END-TO-END LIVE on demo: agent keygen → `POST /hosts/{id}/wg` →
desired-state `wireguard` block → agent-managed `wg-quick@wg-felhom` → PBS page over
10.77.0.1:8007 in 3 s. **`wg_tunnel.enabled` defaults FALSE agent-side** (rollouts to other
boxes are no-ops until the production endpoint exists). Revocation semantics completed (§3.5):
removed peer → agent disables + keeps its marker + NEVER re-registers; operator re-adds via the
pubkey the heartbeat report now carries. Escrow: `IdentityBundle.WGPrivateKey` auto-injected on
new escrow-create runs (pre-S3 blobs can't be retrofitted — honest fallback = fresh-key
re-register, /32 kept). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix, deferred
non-blocking); S4 = PBS over the tunnel; hub-side S6 = tunnel-health alerting on the new report
stanza. Detail: felhom-agent CHANGELOG v0.64.0 + REPORT.
- **2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0)** —
doc-06 row S2. DECISIONS: the WG assignment is **hub-owned state, merged into served
desired-state at READ time** — stored `desired_json` stays a pure operator blob; admin PUT with
a top-level `wireguard` key → 400 (no copy-paste clobber); **re-key-in-place keeps the /32**
(stable tunnel addressing across rotation/DR); one bound peer per host (partial unique index);
generation bumps ONLY on real change (register/re-key/adopt/bound-delete — idempotent paths
move nothing). `POST /hosts/{id}/wg` self-scoped per-host key; NEW golden
`desired-state-wireguard.golden.json` = the S3 cross-repo contract (byte-identical agent copy
rule). peersync **v1.0.1** (strip out of process substitution — the S1 exit-swallow finding;
sandbox red-proofed) redeployed to the dev endpoint. Live-proven on the REAL demo-felhom-01
record via the real per-host key: gen 2→3, served block = golden shape, endpoint `wg show`
carried the peer, v0.63.0 agent tolerated the unknown key (fetch clean, guests untouched),
then removed → gen 4, all baselines restored. `/offsite` page unit-proven (live view is
operator-password-gated — eyeball after next login). Next: **S3** (agent keygen +
registration + `wg-felhom` service + escrow join + CGNAT smoke — risky/supervised).
- **2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1)** — the doc-06
roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: **hub pushes
over SSH** to a forced-command `felhom-peersync` script (pull rejected: weakens immediate
revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev
`felhom-hetzner` box, provisioned by the NEW runbook `documentation/runbooks/offsite-endpoint.md`
(WG on **443/udp**, subnet 10.77.0.0/24, PBS datastore `felhom-offsite` at `/srv/pbs-felhom`,
nftables drop-all + wg0-only 8007, `ep0.felhom.eu` grey-cloud). Hub: `wg_endpoints`/`wg_peers` +
/32 allocator + `/admin/wg/*` (global-key, pubkey-in-body) + `internal/wgsync` (FixedHostKey pin
+ **pinned-type HostKeyAlgorithms — live-caught bug**: multi-hostkey sshd presented ECDSA vs the
ed25519 pin; fixed v0.32.1 + regression test) + full-list declarative reconciler (5-min drift
repair). Live: add/remove→wg show, malformed→state-untouched, reboot→persist+converge, client
tunnel via ep0:443→PBS login page, public 8007 closed. INCIDENT handled: first WG server key
leaked via `wg show dump` into the session log → rotated immediately (warning now in the
runbook). OPERATOR follow-up: **fix the AAAA record** (currently `…7aa1::`, must be `…7aa1::1`).
Next: **S2** (hub schema host-join + `wireguard` block on WireDesiredState + box-facing
registration + UI).
- **2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06)** —
`documentation/architecture/06-offsite-connectivity.md`: records the six settled decisions
(plain WG / host-side agent-managed `wg-felhom` as the agent-managed-unit PILOT / one shared
hub-driven endpoint VM running WG+PBS, NO agent on it / hub = source of truth, `wireguard`
block rides `WireDesiredState` on the existing DesiredGeneration channel / one datastore +
per-customer namespaces / PBS ON the VM, relay-through-DooPlex REJECTED as non-scaling),
the Day-0 join handshake, robustness set (WG-roaming-not-DynDNS, DNS re-resolve watchdog, MTU
1420, per-/32 isolation, tunnel-health via storage-target model), trust conformance (outbound
only; operator-inbound + game-ingress out of scope), the honest ledger (CGNAT still unmeasured
— mobile-hotspot smoke closure; peer-sync push-vs-pull = slice-1 design point), and the S1–S6
slice roadmap (**MVP = S1 endpoint + S2 hub schema/desired-state + S3 agent WG service, then S4
PBS-over-tunnel**; S5 merges with DR-completeness). day0-install backlog line now cites spike +
design doc. Docs-only; slices are separate future tasks.
- **2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung)** —
plain WireGuard, host-side, box-dials-out to a small public endpoint; **no Headscale/DERP needed
for backup** (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway
Hetzner: 11.4-min idle hold, PBS API through the tunnel with the exact-cert pin intact
(positive + negative), real 2 GiB worst-case backup at 4.26 MiB/s (= the home uplink; tunnel
overhead ~5%), UDP 51820 *and* 443 both pass. P0 honesty: the operator's line is plain-NAT
(fixed public IP, One Hungary), NOT CGNAT, and has zero IPv6 — CGNAT confirmation deferred to
Peti's VM 110 (second ISP). P7 surprise: kernel WG *works* inside the unprivileged guest (no TUN
needed) — host placement stands on architecture (backups originate host-side; cloudflared
precedent), not infeasibility. Evidence + architecture recommendation (endpoint VM shapes, key
custody via IdentityBundle escrow, per-peer /32 isolation):
`documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`. **This unblocks the
DR-completeness work: next task = production connectivity spec** (endpoint VM, agent-managed WG
service, hub peer registry, escrow join). Throwaway Hetzner box parked powered-off (secrets
shredded); demo-felhom verified residue-free.
- **2026-07-03 — 4th skill: `felhom-app-catalog`** (catalog authoring workflow; points at
app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by `install_skills.py`, 4/4
found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected
healthchecks, probe-container naming rule discovered + REUSE row added, UI-applied, controller
probe healthy). App-count drift recorded: website says "45+", per-category spans sum 49, real
catalog 53 — website fix is a separate task.
- **2026-07-03 — GOLDEN 0.98.3 LIVE + VOUCHED; guide D.1b RETIRED (drill B1+B5 FIXED)** —
`build-golden.sh` v2.0.0 (mandatory controller tag + baked bootstrap `.path` unit); golden baked
with controller 0.98.3, clean-room validated (drill VM) through all four scenarios incl. the
published-artifact `--force-gitea-golden` path, published + operator-vouched (Day-0 manifest now:
agent 0.63.0 + golden 0.98.3). Fresh installs land CURRENT and self-manage — D.1b is a one-line
check; the prior "vouch agent 0.63.0" operator follow-up is DONE. Evidence:
`documentation/audits/DRILL-golden-098-2026-07-03.md`. NEW operator follow-ups: (1) SECURITY —
the customer-config `git.token` is a Gitea admin token with package-WRITE (proven during the
bake); scope down + rotate. (2) optionally raise the global controller floor to 0.98.3.
- **2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published)** —
token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir.
Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. **OPERATOR: vouch
agent 0.63.0 in the Day-0 manifest UI** (until then fresh installs fetch 0.62.0).
- **2026-07-03 — GO-LIVE PACKAGE SHIPPED (host-install v1.9.1 + day0-install runbook + clean-room drill)** —
`documentation/runbooks/day0-install.md` is the operator Day-0 guide, **every command executed
verbatim** on a throwaway nested PVE 9.2.2 (build server): install → verify → app deploy →
uninstall → **zero-residue diff** → reinstall. v1.9.1 = residue-free uninstall (agent config w/
hub api_key, shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq snippets) +
post-provision guest reboot (without it the controller never deploys on slower hardware) + header
sync. Evidence + go/no-go for Peti (**GO**, 3 install-day items):
`documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`. Key OQ-2 finding: golden bakes a
pre-floor controller 0.85.1 → **fresh installs need the one-time D.1b update** (agent restart →
update button → 0.98.3); follow-ups recorded: rebuild golden ≥0.86.0 + re-vouch (operator), agent
`mkdir -p` for the guesthook snippet dir, fresh-install local-API 401 (pre-existing), hub has no
host-delete path. Drill VM parked virgin-snapshotted on 180 (`~/drill/`).
- **2026-07-03 — A1 CLOSED (host-install v1.9.0 + agent v0.62.0)** — `Pool.Audit` added to
`FelhomAgentGuest`; the agent's stale-lock reaper now intersects its scan with
`GET /pools/felhom` membership (fail-safe skip on read failure). Spike:
`documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. Deploy order on any box:
rescope ACL first, then the agent. STILL PENDING (operator): rotate + de-git the 3 committed
secrets in `manifests/felhom.secret.yaml` (spike appendix; REUSE §2 errata added).
- **2026-07-03 — hub v0.31.0 LIVE** (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy;
live probe: critical event stored + operator email delivered).
- **2026-07-03 — Felhom skills exist**: `skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}`,
installed via `scripts/install_skills.py` — **junction mode** (repo edits live immediately; no
re-install needed after editing). Fresh-session discovery verified.
- **2026-07-03 — CLAUDE.md refreshed** (all four repos): no version-pinned state in CLAUDE.md —
current state lives here / CHANGELOG / REUSE.md.
- **2026-07-03 — `REUSE.md` exists at the repo root** (hub helpers / website conventions / gate scripts / manifest patterns, code-verified); maintenance rule active: update it in the same commit that changes a shared helper. `scripts/reuse_refs_check.py` machine-checks its citations across all four felhom repos.
Reference in New Issue View Git Blame Copy Permalink