felhom.eu/CONTEXT.md

CONTEXT — felhom.eu working state

Created with the REUSE.md rollout (2026-07-03). Authoritative history: hub/CHANGELOG.md (hub), website/CHANGELOG.md, scripts/CHANGELOG.md; end-of-task detail in REPORT.md.

documentation/backlog/OPEN-ITEMS.md is the single source of truth for open work (2026-07-27). Read it first; ROADMAP.md holds the full history and reasoning behind each item. REPORT.md is per-session and overwritten — nothing durable may live only there.

This file is NOT STATUS.md, and the two must not be consolidated (2026-08-02). CONTEXT.md is technical working state written for Claude Code: rulings with their mechanisms, file paths, version numbers, the reasoning a session needs before it touches code. Root STATUS.md is written for the operator — technical, but not in the code and not a reader of the audits — in plain language, one screen, no identifiers in the prose. Same subjects, different readers; merging them would make one of the two audiences stop reading. STATUS.md is also a view of OPEN-ITEMS.md and holds nothing of its own; this file does hold its own content, namely the standing rulings below.

Standing rulings

S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes (2026-08-04). Scope: those two boxes, not the fleet.

Four SCHEDULED runs overnight, none triggered by hand: demo-felhom host 83.8 s / offsite 540.4 s; demo-hp host 109.3 s / offsite 300.1 s. Every one restored into a scratch guest, booted, verified and destroyed itself.

What this closes. Until yesterday every live restore-proof this project held was on the OFFSITE tier. The on-box tier — the one an ordinary recovery uses — had never been proven on either box, and not because it failed: the agent could not read the storage it lives on (R-185), so it never saw an archive there to test.

What was observed for the first time. Both boxes had BOTH tiers due simultaneously. Never-proven sorts first, so each took its host tier, deferred the offsite one, and picked that up on the next evaluation six hours later — R-86's ordering and the one-heavy-operation gate, working together, unsupervised. The host-tier proof then reached the hub through R-189's merge, which is that path carrying a host-tier entry for the first time.

The asymmetry worth remembering: a host-tier restore is 83–109 s; an offsite one is 300–540 s. The tier that matters for an ordinary recovery is also the cheapest to prove.

S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).

The defect. On both demo boxes the agent's token had FelhomAgentStore on local, local-lvm and felhom-pbs and not on felhom-backup — the storage the same installer configured as local_backup_target. The content API answered {"data":[]} through the token while root listed three archives. pickForThisRun skipped the tier as "no settled archive yet", which is exactly what a brand-new tier reports, so the host tier was never restore-testable and nothing said so.

The rule. The permission question has a definite answer where the listing does not. Permissions reads /access/permissions?path=/storage/<target> as the agent's own token — asking as root answers a different question and always says yes — and one capability.Status per configured tier reports it. The probed set comes from BackupTiers(), never a fixed list: a hardcoded probe list is the defect reproduced inside the fix.

The measured trap, because the obvious reading is wrong. An ungranted path answers neither empty nor 403: it carries the privileges inherited from the box-wide / grant (Sys.Audit, SDN.Use, Datastore.Audit). Testing for path-presence, or for Datastore.Audit, reports a blinded storage HEALTHY. The probe tests Datastore.AllocateSpace, and re-measuring is required before that constant is ever changed.

Criticality, weighed once. Critical, because the hub alerts only on critical and a non-critical entry would ride the report and alert nobody. Except the local fallback target, which host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it does not page — turning an ordinary documented setup into an alert is how a signal becomes something an operator archives unread. It never consults content, so it cannot alarm on a newborn tier by construction, and it never reports ok when it could not ask.

S-22 — the installer's Scenario-F arm must finish the job, not just leave the definition alone (2026-08-03, R-185). configure_backup_target has two arms. Case A creates the storage and grants in the same breath. The reuse arm — "the target already exists" — returned without granting, and that, not PVE_STORAGES, is where the drift came from: a box whose target pre-dated the install (the vzdump-target-move runbook, or a reinstall) pointed local_backup_target at a storage its token could not read. The reuse arm now ensures the ACL through the same guarded wrapper. Scenario F is unviolated — the storage DEFINITION is untouched, and granting the role the agent is supposed to have on the target this script is about to write into agent.json is finishing the job, not retargeting the box. $BACKUP_TARGET_ID stays OUT of PVE_STORAGES: that list is granted a step before the target is resolved, and --acl-storages entries are preflight-checked for existence. A gate asserts every arm that resolves the target also grants on it.

S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is the design (2026-08-03, R-189; agent v0.122.0).

The rule. Only successful restore-tests are written to RestoreTestState, and that state is what the host report carries after a restart. Failures live only in the in-memory backup.Store.

Why, in one line each. A success suppresses future work — under R-86's per-archive due-check a proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it believes, for a whole archive generation (a week on the offsite tier). A failure causes future work — a failing tier stays due and is retried at the next evaluation, so a lost failure heals itself within one interval, while a persisted failure would outlive the fault it describes.

What the report does with the two. The collector merges them: one entry per tier, newest by TestedAt wins. A fresh failure therefore beats a stored success (the failure is the news and exists nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier can never appear twice — the hub would read that as two tests.

It refuses to lie. A persisted record missing the archive or the tier produces no entry: an unproven tier reading as proven would be worse than the defect this closes. Run mechanics (scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be. Migration consequence, seen live: a pre-R-189 record has no tier, so upgrading does not retroactively make an old proof visible to the hub; the tier's next real proof fills it in.

S-20 — the release order is build → tag LOCALLY → publish → push tag, and every step protects something (2026-08-03, R-188 + R-186).

The tag is created before the publish, so the build and the tag describe the same commit. It is pushed after, because the push is what wakes CI (on: [push]) and a tag visible before its package made check-published-versions.py correctly fail a correct release — measured on roughly every second release, and R-168 mails those failures to the operator.

The invariant the old order protected is asserted directly instead: the gate now also refuses a published version with no tag, as a bounded probe (frontier + patch gaps) that prints its own coverage, because the package listing api is 401 without a token and absence cannot be enumerated. A half-done release is loud: publish-then-failed-push dies naming the recovery command, and a failed publish deletes the local-only tag so a retry is clean.

A released binary is independently verifiable — -trimpath -buildvcs=false means the same source yields the same bytes with or without the tag; the verification command lives in felhom-agent/CLAUDE.md. Both build paths (release-agent.sh and publish-agent.sh's fallback) use identical flags: they differed by CGO_ENABLED=0 and produced binaries 74 KB apart for one version.

S-17 — restore-testing is PER ARCHIVE GENERATION, and the hub's staleness window follows each tier's own rhythm (2026-08-03, R-86; agent v0.121.0 + hub v0.91.0).

The rule. Let A be the newest archive on a tier that has settled for at least the settle lag (24 h). The tier is DUE when A exists and A has not already been proven. The daemon-start ticker survives only as the evaluation interval (6 h). A daily tier is proved daily on yesterday's archive; a weekly tier weekly on its own; a tier with no archive is UNKNOWN, never a fault.

The trap, written down so it is not reintroduced. The literal reading of R-86 — "due when the newest archive is ≥24 h old" — is never true on a daily tier, because a new archive resets the newest-archive age to zero long before it reaches the lag. It would have switched restore-testing off for the tier that matters most, silently. Red-proved at 0 runs over 5 simulated days (felhom-agent/internal/backup/restoretest_due_test.go).

What the state holds now. RestoreTestState records which archive was proven, not just when a tier passed — a timestamp cannot answer "have we proven THIS archive". A pre-R-86 file keeps its time (rotation ordering survives a deploy) and yields no proven archive, so each tier is due exactly once after the upgrade.

The old config key. backup.restore_test_cadence_seconds is DEPRECATED. Negative still disables verbatim; a positive value now seeds the settle lag only, and the daemon WARNs once at start-up naming restore_test_eval_interval_seconds (default 6 h) and restore_test_settle_seconds (default 24 h). It is deliberately NOT carried into the evaluation interval.

The hub half is not optional. restoreProvenStaleAfter was a flat 7 days derived from the cadence R-86 removes, and a healthy weekly tier's proof age reaches EXACTLY 168 h just before its next proof — it sat ON the line. restoreProvenWindow(tier, observed, ok) now takes the tier's own observed archive interval × 4 generations, floored at 7 days, capped at 12 days (strictly inside the 2-week offsite retention), falling back to the tier's declared rhythm (backupStaleAfter 26 h / offsiteBackupStaleAfter 8 d — the backup-freshness checker's own thresholds) when history is too short to observe one. Shipping Part 1 alone would have produced a nightly false alarm.

S-18 — ep0 is Tier 2, PROTECTED (operator ruling, 2026-08-03). D-d named two protected machines and did not name ep0 either way; runbooks/target-selection.md carried the question in writing for two days. The ruling extends D-d's protected list to three machines: DooPlex, Peti's cluster, ep0. It is a classification, not a new set of prohibitions — destroying datastores, prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and the ordinary off-site READ a restore-test performs remains permitted.

S-13 — the mp1 merge landed, and the variant was chosen on measurement (2026-08-03, R-165 / D-a). The appliance's two data volumes are one. Variant V-c: the volume mounts at the NEUTRAL path /var/lib/felhom, and both /var/lib/docker and /mnt/sys_drive are binds of subdirectories of it. Three shapes were built and rebooted before choosing (audits/SPIKE-r165-phase0-2026-08-03.md) — all three boot, reboot 3/3, give ONE df figure and keep a container's statfs("/") on the merged volume, so the ordering risk that motivated the probe was not what mattered. They differ only in which documented guarantee they break: volume-at-/var/lib/docker puts customer backups inside Docker's data-root, so the ordinary "clear /var/lib/docker" reflex destroys every local unit; volume-at-/mnt/sys_drive puts Docker's entire data-root under /mnt, which the controller container mounts wholesale — measured: it then sees /mnt/sys_drive/docker, falsifying the bootstrap's own comment that /mnt holds only Felhom's namespace mounts. V-c breaks neither, for one extra path.

B2 is the bulkhead replacement, and "or prune the oldest" is REJECTED with its reason, because the question will be asked again: nothing on that filesystem is generational — a unit is ONE fixed path per app (backups/primary/<app>) refreshed in place, and a DB dump is <stack>-<dbtype>.sql, also fixed — so pruning could only mean deleting a different app's only local recovery unit. pruneStalePrimaryDirs is an ORPHAN sweep with no notion of age and must never be repurposed.

No migration exists, and that is a ruling not an omission: every node is REINSTALLED. Both demo boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed shortly. So R-176's in-place migration rehearsal is withdrawn, not deferred.

S-14 — prove first, then vouch (2026-08-03) — SPENT, and the ordering did not survive contact. The rule was: golden 0.192.0 stays UNVOUCHED until a box has been proven from it, because vouching is what makes a fresh install pick a golden up. In the event the golden was vouched at 07:23:26 CEST on 2026-08-03, before any box was reinstalled (hub log Artifact manifest set: agent=0.119.0 golden=0.192.0), so the ordering was already spent when R-178's session opened; the operator elected to accept it rather than revert the manifest. Both boxes were then reinstalled and proven (R-178, REPORT.md), so the end state is the intended one and no unproven layout was ever in front of a real install — but the rule protected nothing, because nothing enforced it. The lesson is R-115's, one layer up: an ordering that lives only in a CONTEXT.md sentence and a runbook's §7 is a reminder, and reminders do not hold. If prove-then-vouch is to be a rule it needs the shape R-120's gate has — a refusal at handleSetArtifacts, the sole path to SetArtifactManifest, which runs without anyone choosing to run it.

S-15 — the merged layout is proven live, by two different supply paths (2026-08-03, R-178). Both demo boxes were wiped and reinstalled from golden 0.192.0 and taken through claim → deploy → back up → restore. demo-hp was installed with --golden <local volid> (the layout proof) and demo-felhom by the normal manifest route with --force-gitea-golden (the pipeline proof — verified sha256 54e2a4c431daf580… matches the hub manifest), deliberately different so the session proved the disk shape and the delivery route rather than one of them twice. Live shape on both: mp0 at /var/lib/felhom, backup=1, no mp1; /var/lib/docker and /mnt/sys_drive both real mounts of its subdirectories via /etc/fstab; ONE df figure and one device id on all three paths; 3/3 reboots each with the binds surviving every time. B2 was not proven on that pass → R-181: the floor guarded captureAllRecoveryUnits and not runVolumeDumps, the leg that fills the volume, and its refusal's "the previous unit is untouched" was measured false. R-181 CLOSED the same day (controller v0.193.0 + v0.193.1), so R-165 is now PROVEN-LIVE in both halves — see S-14.

S-14 — the reserve is a per-app, per-run ADMISSION decision, not a capture check (2026-08-03, R-181; controller v0.193.0 + v0.193.1). B2 as first shipped was consulted in exactly one place — captureAllRecoveryUnits, a few KB — while RunDBDumps' database leg and runVolumeDumps wrote the bulk into the same backups/primary/<app> tree, first and unguarded. The reserve was therefore consumed by the very write it exists to bound, and the refusal then claimed "the previous unit is untouched" about a tree the earlier leg had already rewritten (182,272 B → 2,147,666,432 B under a manifest that had not moved). Sixth entry in CLAUDE.md's table of shipped guarantees the code did not provide, and the fourth of those found on live hardware rather than by review.

• internal/backup/admission.go — admitApp is now THE gate, and every per-app write leg calls it. One verdict per app per run covers all three; they share one per-app root, which is what makes that honest.
• Decided lazily at the app's first write, never once at run start (app A's dump can put app B under the reserve), never re-decided between an app's own legs (that is the split it closes), and reset per run.
• Ahead of DumpAppVolumesSafe, which stops the stack as its first act — a refusal decided inside it has already bounced the app. After the volume-less check, which has no write to gate.
• Size term added: would THIS app's write cross the reserve?, estimated from the app's previous .sql + .tar. No history → headroom-only, deliberately — otherwise the first backup is the one that can never happen.
• A container-based du was MEASURED and rejected, not waved away: median ~355 ms/volume over 66 runs on demo-hp, on volumes holding tens of KB (container start-up, not the walk). Decisive on top: docker run needs the writable layer, so the instrument can fail under exactly the pressure the reserve handles.
• The wording was NOT weakened; the behaviour moved so it became true, and it is checked by sha256 tree fingerprint, never by reading the log line — the log line is what lied.
• v0.193.1, found by the proof run itself: a 178 KB estimate printed as 0.00 GiB, which reads as no estimate available. Rendering moved to humanizeBytes; arithmetic still in GiB.
• New finding, deliberately not fixed here → R-182: GetFullStatus's periodic capture sweep has no run scope, so a refused app re-alerts on every status refresh (measured: a second identical alert pair 13 s after the run's). Pre-existing in v0.192.0; R-181 changed neither caller.

S-15 — publishing is an act, not a side-effect of pushing (2026-08-03, R-110 + R-115 + R-183). Two rulings, one shape: something became live because someone pushed, not because anyone decided.

• The installer. /scripts/ now git-syncs the tag installer-v<SCRIPT_VERSION>; the website keeps tracking main in a second sync, because pinning both would make every copy edit a release. Publish = cut the next tag + bump the manifest --ref + sync. Roll back = move the tag back, which takes ~30 s and needs no ArgoCD sync at all — git-sync v4.4.0 follows a moved tag, and that half was measured before the manifest was touched because the whole model rests on it.
• The sixteen run-time fetches were NOT what the spec described — sixteen, not nine, and from felhom-agent, not this repo — so no tag here could cover them. They are pinned to raw/tag/v$ART_AGENT_VER instead, which is strictly better: the agent's configs now come from the same ref as the agent binary being installed. That closed a real skew (R-183), not just a channel.
• The URL needed no change, and that is worth knowing rather than re-deriving: it never carried a ref, so both producers follow the tag automatically — and no hub change means no hub bump.
• The agent. scripts/release-agent.sh is THE release path: build → tag → publish → verify by an independent download. It does not vouch. check-published-versions.py refuses a v<semver> tag with no downloadable package, and CI now runs the full gate set rather than --fast, without which that gate would have been registered and never run.
• The gate's invariant is not the one specified, and P-C is why: the hub manifest and Gitea's package listing are both 401 anonymously; the package download and the tags api are not. So CI can ask is this installable but not what is vouched. The residue is R-184.
• Neither gate asserts "the newest version is published." That would go red on the very push that bumps a version, before publishing — and a gate that fails on the normal path is one people learn to ignore.

S-16 — a backup run NOTIFIES ONCE and RECORDS ALWAYS, and those are different things (2026-08-03, R-182; controller v0.194.0 + hub v0.90.0/.1). Measured: nine per-app capture failures reached the hub, two were mailed, seven were dropped by a cooldown whose key carries no app identifier — before LogNotification, so they left no row anywhere.

• The record: recovery_unit_capture_failed, per app, unconditionally, now routed record-only by the hub (recordOnlyEvents) — stored and logged every time, never competing for an e-mail slot.
• The notification: backup_run_failures, ONE per run, listing every failed app with its leg and reason plus the counts and free space. A clean run emits nothing.
• A suppressed operator event now leaves a suppressed row carrying the key that suppressed it — for every operator type. "We chose not to e-mail you" and "nothing happened" must never look identical; that is the whole finding, stated as a rule.
• The periodic sweep gets a digest too, with NO run_id, so it stays under the ordinary hourly cooldown. Without it the sweep's failures would be recorded and never notified — a new silence created while closing one. A real run's digest carries a unique run_id precisely so the cooldown can never collapse a manual run into the nightly one.
• Why the silence is safe: the hub's deadline check raises expected_backup_missed from report freshness, independently of any mail the box sends (monitor/deadline.go:396,417). If that check is ever weakened, this design loses its footing.
• Not taken, and why: putting app in the cooldown key. It fixes the swallowing by producing one mail per failing app — a dozen on a full disk.

ep0 was rescaled by the operator to a CX33 (2026-08-03): 4 vCPU, 8 GB RAM, measured on the box, and the 4 GiB swapfile survived. The 40 GB local disk is UNCHANGED — a CPU/RAM resize only, so no disk figure in any runbook needed correcting. That closed R-90 and unblocked R-86.

S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED). Decision D-c splits two signals by AUDIENCE, and the split is the ruling: a fill warning is the CUSTOMER's (they can free space, delete files, add a drive) and a per-app backup capture failure is the OPERATOR's (they can do none of those things about it). R-158 proposed emitting the existing backup_failed for the capture failure. That was rejected and D-c wins, because backup_failed carries a customerMessages entry AND sits in settings.DefaultEnabledEvents — so reusing it emails the customer, in Hungarian, that their backup failed, about something they cannot act on. It is exactly the mistake R-97a avoided by minting whole_guest_backup_failed, and the reasoning is written into hub/internal/api/handler.go's allowlist. New type: recovery_unit_capture_failed, in allowedEventTypes and notify.operatorOnlyEvents — the second register is what makes it operator-only; the first does not, and v0.78.0 claimed otherwise and shipped the defect.

The customer half reused the pair that already existed rather than minting a seventh type. disk_warning/disk_critical were allowlisted, carried Hungarian copy, sat in DefaultEnabledEvents and had a UI checkbox — and nothing in any repo emitted them. A complete customer pipeline with no producer: the sixth built-but-never-wired instance in this project. internal/fillwatch is now that producer. Their generic customerMessages entries were deleted, because FormatCustomerEmail PREFERS the entry over the message and a static template would discard the drive label and the free-space figures — the same reason offbox_enlarge_blocked and disk_health_degraded have none. notify.IsOperatorOnly was added so ONE test pins both registers; checked separately, an allowlisted-but-not-operator-only type is invisible.

S-12 — the monitoring landed BEFORE the merge, not with it (2026-08-02). D-a's condition (2) says R-167 ships in the same step as the mp1→mp0 merge and never after, because the merge removes a wall that currently fails safely. This session landed it FIRST, which is strictly better and costs nothing: the warnings went in and were proven on real hardware while the wall is still standing, so the merge session inherits a proven signal instead of an untested one. No disk layout was touched. R-165's measurement is audits/SPIKE-r165-mp1-merge-2026-08-02.md, which STOPS at a question for the operator (which merge shape; what replaces the bulkhead). Its two load-bearing findings for anyone picking that up: "the layout" is not one thing (demo-felhom 200G/50G, demo-hp 50G/20G, golden 16G/8G — so §7.5's bound is one box's, → R-175), and mp1 is also a BULKHEAD, not only a ceiling — today an overflow cannot reach /var/lib/docker, and after the merge it can.

S-8 — CI detects; it does not block, and that is structural (2026-08-02, R-168). A Gitea Actions runner in gitea-system re-runs every repo's gate entry point on every push, independent of who pushed and of what they typed. It cannot refuse a push: every felhom repo pushes straight to main with no pull request, so there is no merge for a status check to stand at. That is not a gap in the runner — there is no gate in the road. The refusing half is .githooks/pre-push (per-clone, --no-verify-able); this half is what notices when that hook was skipped or was never armed. Making CI blocking needs branch protection plus a PR workflow, which changes how the operator works and is their call → R-169. Do not "fix" this by adding branch protection.

S-9 — a detector that tells no one is not finished (2026-08-02, R-168 probe P5). Probe P5 measured that a failed run produces no mail, no notification row and no log line from Gitea. So the workflow sends its own alarm on the project's existing Resend path and prints the provider's accepted id, which makes "a message left the machine" an observable rather than an assumption. The acceptance test for this work was never a green pipeline — it was a red run with a message in hand (RESEND-ACCEPTED id=…). Two traps found while building it, both worth keeping: the runner image has no curl (deliberately — python3 and git only, so use urllib), and api.resend.com sits behind Cloudflare, which 403s the default Python-urllib User-Agent with error 1010 — a failure that looks exactly like an auth failure and is not one.

S-10 — the runner is unprivileged, and the reason is the host (2026-08-02). The usual act_runner recipe pairs it with a docker:dind sidecar and privileged: true. Rejected: DooPlex is Tier 2 and is the recovery chain — Gitea, the hub, the registry, PBS and k3s + Longhorn all live on it and it cannot be rebuilt from anything else. Host execution mode costs nothing here because every CI job is one Python command. Consequence to remember: in host mode the job sees exactly the runner image's tools, which is why python3 had to be baked in (probe P2: stock act_runner carries git but not python3). If a future job genuinely needs Docker, that is a conversation, not a patch.

S-11 — CI reproduces the workspace's sibling layout, because two entry points depend on it (2026-08-02). controller_gates.py and agent_gates.py invoke the shared reuse_refs_check.py that lives in the felhom.eu clone next door and is deliberately never copied, and both repos' REUSE.md files cite a path that lives in the hub. Their workflows therefore clone felhom.eu as a sibling; without it the gate fails closed — correctly, but for the wrong reason. Verified that CI and the local hook then agree exactly (controller 126 exact / 6 suffix / 1 cross-repo).

S-6 — the hub renders no host-install version, and the gate pins its absence (2026-08-02, R-94). The Setup tab's "host-install 1.19.0" label is deleted, not derived. Deriving it is not achievable honestly: the Option-1 command downloads felhom-host-install.sh from the website at run time, and the website git-syncs main every 30 seconds (R-110) — so the hub cannot know which version a given box will run, at build time or at render time, and any literal there is a guess wearing a version number's authority. The real one drifted to 1.19.0-vs-1.22.0 and stayed wrong for nineteen days. hostInstallVersion, pageData.ScriptVersion and the rendered label are gone; a NOTE sits where the const was so it is not helpfully re-added, and scripts/hostinstall_gates.py gate 1 inverted — it now asserts the hub carries no host-install version literal in any of six code shapes across every .go/.html under hub/. Corollary that generalises past this row: the tautological render_test.go assertion (html contains hostInstallVersion, where the same constant put it there) passed at 9.9.9 — an assertion that compares a value to itself tests the plumbing, never the claim.

S-7 — gates run from ONE entry point per repo, and reuse_refs_check was fixed rather than the convention it polices (2026-08-02, R-29). Two rulings from the same census.

Where gates run. Thirteen gate scripts exist across the four repos. Measured 2026-08-02: every check a CLAUDE.md tells a person to run was passing, and two of the four nobody is told to run were failing — one since 14 July. The correlation was exact, so the fix is not more gates but one place to run them from: scripts/repo_gates.py, felhom-controller/controller/scripts/controller_gates.py, felhom-agent/scripts/agent_gates.py, app-catalog-felhom.eu/scripts/catalog_gates.py — each mandated in its CLAUDE.md, each wired to .githooks/pre-push with --fast. The canonical shape is catalog_gates.py (R-161), not site_gates.py, which is a gate (eight assertions in one file) and not a runner; copying it produces another monolith nobody invokes. A missing gate script is a FAILURE with the path printed, never a skip. The hook's limits are real and are written into the hook: per-clone (core.hooksPath is local config) and --no-verify-able on purpose. The unbypassable half is CI → R-168.

Why the checker moved and the docs did not. reuse_refs_check.py was RED on all four repos with 13 findings, of which a hand audit found zero genuine drift — twelve were package shorthand (appbackup/userdata.go → controller/internal/appbackup/userdata.go) and one, wgsync/reconciler.go, is cited by the controller and lives in the hub. REUSE.md cites by package shorthand and across repos deliberately; that convention is the useful one. Rejected, so they are not revisited: rewriting all four REUSE.md files to full paths (makes the docs worse to serve the tool), and deleting the checker (drift across four repos is a live risk). The checker now resolves exact → suffix → ambiguous → sibling repo → FAIL, prints every non-exact hit and a per-rule tally (because "0 failures" alone cannot tell a working checker from a blind one), and lists every resolution attempted on a failure. It stays in one place and is invoked across the workspace — never copied, which would recreate the drift it detects.

S-1 — N.5 gains a third leg: architecture docs are same-session coupled (2026-07-26, R-81). Any task that changes an architectural contract — tiers, targets, cadences, trust boundaries — updates the owning documentation/architecture/*.md in the same session, under exactly the same coupling rule that already binds the capability map and the ROADMAP. Origin: R-80/R-81 found 07-backup-architecture.md still describing a backup model that the shipped code does not implement (single target, single cadence), while being cited as authoritative. A stale architecture doc is worse than a missing one, because it is trusted.

S-2 — architecture docs carry an honest status header (2026-07-26, R-81). Every documentation/architecture/*.md opens with the version it was verified against and the date. A doc more than a few trains behind its subject is marked STALE in that header, so a reader meets the warning before the content, not after acting on it. Origin: 07-backup-architecture.md — DRAFT since 2026-07-14, verified against controller v0.132.0, now 41 versions stale (live v0.173.0), and cited as authoritative throughout the R-80 diagnostic. Ratifying or retiring it is → R-83.

S-3 — the recovery model: six decisions, 2026-07-28. Taken in an architecture discussion and expressed in the 07-backup-architecture.md full rewrite (which replaces the 2026-07-14 DRAFT entirely — that doc was verified against controller v0.132.0, 51 versions stale, while being cited as authoritative). They are decisions, not observations; the rewrite labels every statement [DESIGN] or [FACT] so a reader can tell which is which.

• D1 — two lanes, different owners. Customer, unassisted: files and app data, through the „Visszaállítás" surfaces. Operator: LXC guest and host recovery. This is a deliberate product decision, not a limitation to apologise for.
• D2 — encryption follows the boundary, not the tier. Data on the customer's own drives is plaintext, deliberately: someone who can take the second drive can take the first, local encryption adds a key-loss path that turns a working backup into a brick, and it would break FileBrowser and SMB browsing, which is a feature. Data leaving the premises is encrypted (restic, PBS) — the provider must not be able to read it.
• D3 — recovery is a chain, not a set of tiers. Every app-tier restore is conditioned on the whole-guest tier: Tier-1 needs the guest's secrets (restore_unit.go:17-22, "Regenerate NOTHING"), and Tier-2/Tier-3 need the app still deployed. 3-2-1 describes copies; it does not describe recovery. This is the reason the rewrite exists.
• D4 — three-part model. Recipe (scaffolding, hub) · Escrow (identity key + restic repo password, hub, R-wrapped) · Bytes (the tiers). Losing one is a different problem from losing another, and the failure→recovery matrix is organised around that.
• D5 — R's role narrows. CONDITIONAL, and currently BLOCKED. The target is app secrets travelling with the LOCAL recovery unit, so Tier-1/Tier-2 restore work without the guest and without R, leaving R needed for offsite recovery and host identity only. Not adopted: the premise (the backup tree unreachable from every browsing/download surface) was tested and failed — see R-108. Every other surface verified clean; the exposure is FileBrowser's root-bind of network shares combined with the absence of an IsNetwork() filter on the deploy and per-app-migrate paths.
• D6 — the trust model, stated honestly. The operator holds root SSH on every box; "the operator cannot read customer data" was never the actual security property and no document may imply it. What R does provide, and must keep providing: the hub alone is not enough — a compromised hub yields blobs nobody can open, provided the operator's key is never stored in the hub. That proviso is why escrow custody is an open decision (07 §11-A).

S-4 — the hub session password alone now unlocks console root on every managed box (2026-07-31, hub v0.84.0). Retrieving a host's vaulted break-glass root@pam credential previously required the global operator API key, a secret distinct from the hub login and kept out-of-band. The Console access card on the host page puts that retrieval behind the hub session (POST /hosts/{id}/reveal-recovery-credential, CSRF-gated, audited), so the hub login password is now sufficient. Accepted deliberately, and the reasoning is the ruling: the hub is single-operator and geo-fenced to Hungary, and it already stores these passwords in plaintext at rest — so the DB and the login were already jointly sufficient, and this is a convenience trade rather than a new exposure. It is a real trade, not a free one, and it is recorded here rather than only in a changelog. Two things follow. (a) The global-key path (api/handler.go handleAdminGetRecoveryCredential) stays untouched and must acquire no dependency on the session layer — it is the route for when the hub UI itself is what is broken, and coupling the two would delete the independence that makes it a fallback. (b) The plaintext-at-rest half is now the named open item R-133: every hub DB backup is a fleet-wide console-credential dump, and the fix shape is envelope encryption under a KEK held outside the DB, the way escrow blobs are already opaque.

Five decisions were deliberately left open for the operator and are recorded, unanswered, in 07 §11: escrow custody · lost-R policy · per-scenario RTO/RPO targets (none have ever been stated) · Hetzner as a single failure domain · and local vzdump sharing a physical device with the guest it backs up. Gaps minted the same session: R-102 … R-108.

S-13 — boot recovery finished, and the lesson is about the DIAGNOSIS ORDER (controller v0.190.0, 2026-08-02, R-157 A · R-170 · R-171).

The session's most valuable half hour was spent NOT writing code. A hole was reasoned out of the v0.189.0 diff — replacing the container-count term with recorded intent should make a drive-gate-stopped app read as a boot orphan — and the task's own rule was to CONFIRM it on hardware before writing a fix. The first attempt to confirm it produced a false negative, and reporting that as a disproof would have been wrong: unmounting only the parent bind is healed by the agent within ~60 s, so the drive gate's startup reconcile restarted the apps one second before the sweep looked. no boot-orphaned apps in that log is a race that went the safe way, not a mechanism. Holding the drive genuinely absent reproduced it immediately. "It didn't happen this time" is not a disproof — name the mechanism or run it again.

The confirmation also changed the severity, in both directions. The write hazard did NOT materialise: compose failed mkdir …/userdata: permission denied, because the unbound mountpoint is host-root-owned and the guest is unprivileged. That protection is accidental — no code chose it, no test pins it, it is one chown or one privileged guest away from gone, and its removal would be invisible until data landed on the wrong disk. Meanwhile the harm that DID occur was real on every box and was not in the hypothesis: two wasted attempts and a false dead-app alarm for an app the drive gate is deliberately holding. Diagnosing first is what produced both facts.

The fix was already in the codebase, on another path. The API's startGatedByMissingDrive refuses a customer's start on an absent drive with a Hungarian message. The sweep bypassed it by calling Manager.StartStack directly. StartStack has no gate of its own — that is the durable fact worth carrying: every caller that is not the customer must decide for itself whether the app may run, and there are now fourteen of them.

Widening a window makes previously-unreachable overlaps reachable, and that is a design input, not an afterthought. The old T+5 s sweep never met a quiesce or an in-flight app-data operation; a 50 s window can. All three holders answer one seam rather than three, because they differ only in the reason string.

A test rejected my first constant, and the comment now says so. settle + budget + one retry must fit inside deadAppBootGrace or a successful recovery stops being silent; 60 s gave 95 s against a 90 s grace. The budget is 50 s because a test said so, and the code records that rather than presenting the number as taste. Widening the grace to fit was rejected outright: it hides a late recovery instead of reporting one.

AND THE FIX HAD ITS OWN DEFECT, FOUND BY LIVE VALIDATION AND NOT BY REVIEW. The window sampled GetStacks() — the Manager's in-memory map, refreshed by the scheduler every 10 s — every 5 s. Two identical samples could therefore mean the cache did not update, not that the fleet had settled. It surfaced as a container removed ~5 s before the window closed still being in the sampled fleet, with the sweep logging no boot-orphaned apps for an app that had none. Generalise it: a settle detector is only as good as the freshness of what it samples — if the source is cached, refresh it or you are watching the cache settle, not the system.

Live: 6/6 hard resets on the shipped build (every app back; a customer-stopped app down in all six), window settle times 10/40/10/10/15/15 s — routinely 2–8× the old fixed 5 s. The sharpest evidence is a same-app before/after on one box: missed at 18:08:35, recovered at 18:18:50.

S-12 — D-b is BUILT (controller v0.189.0, 2026-08-02, R-166). The desired/in-flight/observed split now exists; the S-1 contract lives in architecture/02-controller-module-map.md §0a.

Both facts D-b said to establish first were established at source, and both changed the shape. (a) Does the crash-safe journal in the backup code already cover the in-flight case? The pattern DID already exist — twice (quiesce marker+Recover, migrate journal+RecoverMigration) — and covered none of the app-data path: DumpAppVolumesSafe stopped and restarted an app with no marker, no journal and not even a defer. So the answer was neither "it exists, wire it" nor "build it": copy the proven shape into its own file. It was the fifth time the question was worth asking and the first time the answer was "the pattern, not the coverage". (b) Is the SQLite store reachable? Reachable and deliberately not used — metrics.db is optional by design (the controller runs with it absent), and operational state must not live in a store built to be dropped.

The ruling that carried the design: absent means UNKNOWN, never "running". Every app.yaml on every box predates the field, so absent is what the whole fleet reads on upgrade; reading it as running would have started every deliberately-stopped app on the first boot after the upgrade — fleet-wide, silently. Where intent is unknown the box keeps the OLD inference rather than inventing an answer. That is also why the backfill is running-only: "zero containers ⇒ stopped" is the defect itself, so an ambiguous app stays ambiguous until a customer presses a button.

The other load-bearing ruling: StartStack/StopStack are NOT writers of intent. A census found 14 callers, of which exactly 2 are the customer. Recording intent in the primitive would make a nightly backup indistinguishable from the customer pressing Stop — the confusion being removed.

Found on the way, and it would have silently eaten the feature: SaveAppConfig rebuilt AppConfig field-by-field. That is the R-100 shape, which v0.181.0 shipped two live instances of. The literal named five fields, so the sixth would have been dropped on every save across nine call sites — a customer's Stop erased by the next unrelated app.yaml write. Copy-and-overlay is safe by construction; the failure mode is generic, so treat any field-by-field struct rebuild in a save path as a defect on sight. Measured and documented rather than assumed: app.yaml does NOT round-trip YAML keys the struct does not model.

Closes R-157 mechanism B; mechanism A (the sweep observes ~5 s after start and never re-checks) is untouched and is now the whole of R-157 — and B's fix makes A cost more, since the sweep now has more it could legitimately recover. New: R-170 — shouldRecreateOnBoot (internal/web/intermediary.go:131) still infers a Stop from hasContainers, i.e. the same defect one gate over for drive-backed apps. Left deliberately: the task scoped bootrecon, and two boot behaviour changes under one live validation is one too many.

Live on 9201, three flows, each with a positive observable. The interrupted-operation half is IMPLEMENTED, not PROVEN-LIVE — unit-proven and red-proofed, but nobody killed the controller mid-backup on real hardware; the capability map says so rather than rounding it up.

S-5 — four operator decisions taken in discussion on 2026-08-02, recorded before anything is built. They existed only in conversation, which is the condition the standing rules were written against. Labels are the ones used in the discussion (D-a … D-d) and are deliberately kept distinct from S-3's D1…D6. None of a, b or c is implemented by the session that recorded them; their work is carried as backlog rows (R-165, R-166, R-167), not as prose here.

• D-a — the dedicated backup partition is MERGED AWAY, not resized. The appliance's system drive carries the OS, a 50 G docker area (mp0) and a separate 20 G area holding local recovery units (mp1; demo-hp's guest 9201 ships exactly that). The decision is to merge the backup area into the large data area so local backups share space with app data and the ceiling stops existing — deliberately not to pick a larger number, because a larger number is the same wall further away. Two conditions, recorded with the decision, not as commentary: (1) it changes the disk layout, so it must land before any external install — after that, every deployed box is a migration; (2) it removes a wall that currently fails safely, so the storage monitoring of D-c must ship with it, never after. Closes the sizing question in R-163, which is re-framed rather than answered with a ratio and is not closed until the merge lands. Work: R-165.

• D-b — app state gets a desired/observed model with its own store. Three kinds of information, deliberately separated: desired (running, or stopped because the customer said so) stays in app.yaml with that app's other settings; in-flight operations ("a backup started and has not reported finishing") are persisted, written before the operation starts and after it ends, so an interruption is visible after a reboot; observed (unhealthy, restarting, drive gone) is not persisted and is rebuilt by looking. The last two live in a state store separate from app.yaml, so frequent writes never touch a file holding deploy settings and secrets. Binding safety rule, verbatim: losing the state store must never cause an app to be deleted, restarted wrongly, or reported healthy when it is not — the worst acceptable outcome is re-running a backup that already ran. The controller must also check every container of an app, so a partly-dead app cannot read as healthy (F-CRIT-1's shape). Two things to establish before this is specced, not now: whether the crash-safe journal already in the backup code covers the in-flight case — it may already exist and merely be unwired, which has been the answer four times — and whether the existing SQLite store is reachable from this code path. Work: R-166.

• D-c — storage monitoring and backup alerts. Two pieces: a warning to the customer as a drive fills, before anything fails; and a failure alert to the hub/operator when any backup fails for any reason, lack of space included. Prerequisite for D-a — see that decision's condition (2). Subsumes and widens R-158, which is the same alert one tier down; R-158 stays as the named seam and is not duplicated. Work: R-167.

• D-d — which machines are protected: DooPlex and Peti's box, and nothing else. DooPlex because it holds Gitea, the hub, the backups and the registry — everything else rebuilds from it. Peti's box because there is a real person behind it. Every other box, both demo boxes included, is disposable and may be broken or reinstalled freely. The earlier caution was costing sessions and pushing work onto DooPlex — the one machine that should never host it — so the ruling is a correction, not a relaxation. documentation/runbooks/target-selection.md updated the same session (2026-08-02); no backlog row, the decision is the change.

• 2026-07-27 — R-85: the DR tier is restore-tested UNATTENDED, and its failure is HEARD. agent v0.104.0 + hub v0.77.0. R-82 proved a PBS restore round-trip by hand; three defects kept the unattended path from covering it. (a) buildRestoreTestScheduler built ONE runner on cfg.Backup.BackupTarget(), so the offsite tier's archives were never candidates — the DR tier could not be restore-tested unattended AT ALL. (b) the Spec was an immediately-invoked function evaluated once at daemon start, so nothing tier-varying was expressible and any config change stayed stale until restart. (c) a failed restore-test was a [WARN] line with no event, no notification, no gauge — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) was the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. Ruling (operator, 2026-07-26): oldest-first (Option 1) — never-proven sorts first, no cadence knob; the per-tier last-proven timestamp is reported so the achieved interval can be measured before anyone decides a knob is needed. Shipped: per-run SpecBuilder; PickRestoreCandidateOn (a tier with no archive NO-OPS); persisted RestoreTestState (survives restart; credit only on SUCCESS); backup.InFlight, the host-wide one-heavy-op gate shared with the backup path (a LINK concern — an offsite restore pulls multi-GB over the tunnel a backup pushes one); hub restore_test_failed / restore_test_stale, kept DISTINCT and anchored on R-81, operator-tier only. Also fixed a time bomb I planted in Slice C: a deadline test hard-coded the incident timestamp 2026-07-18T18:31:06Z while comparing against the REAL clock — harmless under one 26h threshold, but once the offsite tier got an 8-day limit it passed all day and began failing at exactly 18:31 UTC, 8 days later. A test that passes at commit time and fails hours later is worse than one that fails immediately. NOT yet proven: rotation has not been observed selecting both tiers across consecutive UNATTENDED cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). Spec: documentation/backlog/SPEC-r85-phase4-5-2026-07-26.md.

• 2026-07-26 — R-82 SHIPPED: the backup target split (local daily + offsite weekly). agent v0.97→v0.102.0, controller v0.174→v0.175.0, hub v0.76.0, host-install 1.20.0. BackupTarget() returned ONE string and BackupCadence() ONE 24h window, so "local daily AND PBS weekly" was not expressible — which is why the DR tier was applied since 07-21 with one snapshot on demo-felhom and zero, ever on demo-hp. Now: backup_targets[] per-tier cadence+retention; ONE quiesce window for both due tiers (never two app outages for one night); per-tier hub thresholds (host 26h / offsite 8d); fresh-install default; an unprovisioned tier DEFERS. Operator rulings: 2-week offsite retention, first backup runs as long as it needs, one backup at a time per guest, drill box dropped from the rollout. RETENTION IS A COMMERCIAL ATTRIBUTE — the hub decides, ep0 executes (operator ruling 2026-07-27, R-89). A paid tier may buy longer retention, so the policy belongs with customer config on the hub, never in ep0's PBS config and never in a box's config. Execution stays server-side: a reconciler writes a PBS prune job and PBS's own scheduler runs it, so hub downtime leaves the last-known policy running rather than silently stopping retention. Box tokens stay write-only (DatastoreBackup) — never widen a grant to fix a prune error: a compromised box must not be able to delete its own offsite backups, which is the scenario offsite DR exists to survive. Increment 1 shipped 2026-07-27 (boxes stop attempting prune via keep_last: 0; per-namespace prune jobs on ep0, daily 03:30 UTC) — runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md. This closed a live false-negative: every demo-hp PBS backup since 07-26 reported job errors while the data landed correctly, because DatastoreBackup carries no Datastore.Prune. Four defects found by RUNNING it, not reviewing it — a 30-min wait bound against a 41-min backup (the agent recorded success:false while the backup was still going); the restore tier read from the configured target instead of the archive (a silent regression of the S4.1 fix — the mechanism was never removed, its INPUT changed when local_backup_target was retargeted to local); a leaked scratch keeping onboot:1; and a tier firing at a not-yet-provisioned storage. A correction I had to make mid-arc: I reported that the restore-test would boot a network-conflicting clone and break the control plane. It does not — step 2 link-downs every NIC before boot, and it is unit-tested. I read a config artifact and inferred behaviour without reading the code path that consumes it, then escalated before finishing the check. NOT done: the offsite tier is never AUTOMATICALLY restore-tested (the scheduled test only ever selects the primary tier); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence. The installer-default fleet flip waits on a full weekly cycle. R-84 SHIPPED (agent v0.103.0) — the cold in-memory store no longer causes a redundant backup: the due-check asks the STORAGE (ground truth) instead of persisting state, so a pruned archive correctly stops counting. Proven live on both boxes with the store cold. demo-hp's FIRST EVER offsite backup landed (4.25 GB) — the R-82 finding closed on the box where it was worst. Controller v0.175.0 deployed to both boxes. Evidence: audits/SPIKE-r82-phase0-2026-07-26.md, REPORT.md.

• 2026-07-26 — hub v0.75.0: R-81 SHIPPED — "no signal" is not "bad signal". Third instance of one class (hub v0.12.0 expected_backup_missed daily-for-everyone; hub v0.73.0 offsite_stale minutes after a healthy repair; now this), fixed as a class, with the invariant written at the head of assessBackupFreshness naming all three. assessBackupFreshness returns OK/UNKNOWN/MISSED instead of missed bool; absence is UNKNOWN until it outlives an anchored window. The anchor was free — Phase-0 probe found the hub retains 90 days of host-reports, so GetHostReportsSince + newestBackupEvidence (7-day bounded lookback, early-exit on fresh evidence) answer "when did I last SEE evidence of a backup?" rather than "what does the latest report say?"; absence is graded from first contact (GetFirstHostReportAt) reusing the existing 26 h threshold as the grace — no new knob, no agent change, no new persisted state. Rulings: absence is never silent (a genuinely dead box must still alarm — that is the half the naive fix breaks, and TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms is what makes the suppression safe); reason strings stay distinct (the whole 07-26 diagnosis turned on reading the exact string); deferrals are logged so quiet ≠ not-checked. Tests 493→508; red-proofs A/B/C observed — A reproduces newest backup is 176h0m0s old verbatim the mail demo-felhom sent. Replayed against the real 03:00 reports (600/417/77 rows): all three now silent. backupStaleAfter deliberately untouched — ⚠️ it will alarm on a healthy WEEKLY PBS snapshot, landmine recorded, owned by R-82. The real finding behind R-80 stands: the PBS/offsite-DR tier has no schedule at all → R-82. Cause-fix (persist the agent store) → R-84. Evidence: documentation/audits/DIAG-backup-missed-2026-07-26.md, hub/CHANGELOG.md v0.75.0.

• 2026-07-23 — hub v0.72.0 + controller v0.161.0: R-70 SHIPPED + R-71(c) PARTIAL (the offsite last mile is visible; burned credentials self-heal). One detector (offsite.DeliveryStateFor — secret-row timestamps × report offsite-presence), four consumers: operator card (static "delivered once" copy DELETED; state+age, amber past 30 min, stale-staged info line), offsite_delivery_stuck warning (≥1 h, 24 h durable cooldown via LastEventAt on the events table — restart-proof), R-71c self-heal through the EXISTING Re-issue path (monitor.OffsiteReissuer ← webServer, armed only with a provisioner; consumed ≥1 h + ≥4 offbox-less reports + zero offbox evidence; one restage/24 h; every firing = a warning event), and the controller's truthful empty-state banner (v0.161.0, both boxes). Rulings: applied wins (stale-staged is a flag, never a downgrade); cooldowns durable via events table; both new events operator-only (no customerMessages, not in allowedEventTypes — pbsdr_* precedent); R-39(a) guard in the CALLER — the heal re-reads the secret row at act time and refuses over an unconsumed row; the store's clobber semantics stay (Re-issue depends on supersede). Clobber red-proof observed (reissue calls = 1 + row clobbered with the guard removed). Self-heal + banner ship unit-proven, NOT live-fired (no broken box; F9 rule) — arms on the next natural occurrence. Live: checker silent on the healthy fleet; demo-felhom's stale-staged specimen intact (its cleanup is the operator's, post-validation). R-71(a) day-0 ordering OPEN → next spec. Evidence: REPORT.md (2026-07-23).

• 2026-07-23 — ISO train v1.25.0 + hub v0.73.0 (four rulings + rider). host-install 1.17.0→1.18.0, build-felhom-iso 1.24.0→1.25.0, felhom-bootstrap R-63, hub v0.73.0. Done + live-verified: (1) F9 belt everywhere — host-install installs the H1 OOB belt by default in appliance mode (--no-oob opts out; byo still refuses), oob.enabled written so the agent activates it; belt LIVE on both boxes, login PROVEN on felhom-pve (felhom-op@demo-felhom); the orphaned operator identity was re-anchored to the operator's real machine (WG peer + SSH key re-registered, ep0 forward rules persisted). (2) apt — align_apt_repos (appliance-only) switches enterprise→no-subscription; LIVE-aligned both boxes (apt-get update clean). (3) R-63 — Lat2 console font before first paint. (4) R-71 build-gate — assert_golden_ge_floor in build-felhom-iso + rule 5; red-proofed. (6) root vault — felhom-pve root rotated+vaulted+ verified over :22 (G1 parity). (7) hub v0.73.0 — offsite_stale never-ran branch ANCHORED (applied-only + consumed_at/escrow anchor; one-state-one-owner with offsite_delivery_stuck), shipped+deployed, red-proofed. CRITICAL live finding: golden 0.153.0 < floor 0.156.0 in production now — the R-71 gap is real; the gate correctly blocks a v1.25.0 build. golden republished at 0.161.0 + vouched (gap CLOSED), v1.25.0 ISO built (gate live-passed). Nested drill aborted on the N100 (install crawled — the N100 was contending with live guest 9201; KVM-confirmed, not memory) → cleaned up (9201 untouched); operator steer: future drills on the HP. R-63 font code-verified; the live ő/ű console capture rides the next HP drill (belt/apt already proven live on both boxes; no-floor-update is structurally guaranteed post-vouch). Evidence: REPORT.md, scripts/CHANGELOG.md, operations/nodes.md.

• 2026-07-23 — F10 offsite leg RESOLVED on demo-hp (diagnostic + designed-path repair + tier proof; no code changed). The hub said "Provisioned" while the box said „Még nincs beállítva" — root cause: the day-0 managed update (0.153.0→0.156.0, 07-21 16:28) killed the offsite apply-bridge ~35 s after it consumed the one-time password, before key-install/persist; the box then hit the consume-404 dead-end at every start, silently, for 2 days. Repair = operator Re-issue click (converged in 8 s), escrow ceremony via the real wizard (auto-confirm 3 s), then the full proof: paperless-ngx toggled into scope, first offsite run (repo initialized on sub3, 79.8 MB, snapshots=1), probe deleted → mode=full restore (snapshot 2bf7f2e1) → place merged 1 file missing-only → md5-identical. Minted R-70 (last-mile invisible on both surfaces; hub never reads one_time_secrets.consumed_at) and R-71 [P1] (the race recurs on every fresh onboarding whose ISO floor lags the managed floor). Record: documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md. F10's PBS-DR half stays open (F13 + ceremony R-moment).

• 2026-07-22 — hub v0.71.0: the notification train (audit F11+F12+F14-light). Four rulings now standing: (1) recovery pairing — *_recovered notifies the operator always and the customer iff the customer was mailed the paired stale/down (evidence = notification_log customer- channel sent rows via store.LastCustomerSentAt; enabled_events deliberately ignored for recovery; ties → no mail). Severity semantics FROZEN — recovery stays info, routed by an explicit eventType branch before the severity gate. (2) seed-at-claim — MarkClaimed seeds customer_notifications from the registered email, INSERT-if-absent only (never touches an edited row; empty email = no-op; seed failure never fails the claim); default set is critical-only: node_down, backup_failed, disk_critical, host_disk_critical, storage_fill_critical, offbox_repo_orphaned. (3) empty-email no-clobber — a prefs push with empty email preserves a stored address (hub-side belt; controller 0.160.0 already guards its two push legs). (4) priority headers — error/critical mails carry X-Priority: 1 + Importance: high (Resend headers, live-probed); the test event now also mails the operator with those headers (one click proves both channels + rendering). Latent nil-deref fixed: sendTestEmail panicked on a customer with no prefs row. NOT yet live: a natural *_recovered mail (next real staleness cycle / reboot drill — never fabricate one by blocking reports) and seed-at-claim on a real claim (Peti's Friday reinstall is the natural candidate).

• 2026-07-22 — hub v0.70.1: the ghost customer's Delete button exists now. The v0.70.0 ghost-delete path was fully implemented and fully unreachable — the fourth inert-seam defect, this time a TEMPLATE GATE: the Danger-zone card (and the customerDeleteOpen script) sat inside {{if .HasConfig}}, so a ghost rendered no Delete button (found live on demo-vm-felhom). Fix: Deletable page flag = the exact negation of the delete preview's 404 predicate; RESET stays HasConfig-gated, Block/Unblock config-only. Render tests per branch + 2 red-proofs. The seam-wiring rule now covers template gates (see CLAUDE.md) — handler tests that POST directly prove nothing about reachability. Operator live leg PENDING: the demo-vm-felhom ghost delete click (doubles as v0.70.0+v0.70.1 live validation). Same day: **agent 0.93.0 published

  • deployed fleet-wide** (pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md), R-17 CLOSED (u629193-sub1 deleted at the console).

• 2026-07-21 — THE FLEET IS TWO HOSTS. A second Proxmox node exists: demo-hp-bb76ea, an HP t740 Thin Client (Ryzen V1756B, 30 GiB, PVE node felhom-host, customer demo-hp), installed from the armed universal ISO the same day. Both hosts run agent 0.92.1; the N100 is demo-felhom-8363b5. Full inventory: documentation/operations/nodes.md — read it before touching the HP, because two things there are load-bearing: the box's 1TB Toshiba NVMe is present and UNENROLLED (still carrying its old NTFS partition; a future Tárhely candidate that must join through the storage flow, never the installer), and there is no operator SSH key baked on it, so access is the hub-vaulted G1 break-glass password (R-61 is the fix for that lockout). Both nodes are at the VACATION site and travel home ~2026-08-02. demo-hp joined the tailnet as 100.76.96.79 — an operator-lab exception, not product shape; real customer boxes never get tailscale. Second-hardware proof, worth keeping: the pairing/day-0 chain ran end to end on virgin hardware it had never seen, the shim loader booted with Secure Boot ENABLED (so SB-off was an N100-firmware workaround, not a Felhom requirement), and the box self-lifted the floor 0.153.0 → 0.156.0 during day-0. The install was not clean, and the failures are filed: R-59 (no DHCP on the 4-port NIC → the installer baked a static fallback instead of aborting), R-60 (first-boot NIC sweep self-heal), R-61 (baked root password unknowable).

• 2026-07-19 — N100 MADE LOCATION-INDEPENDENT via Tailscale. felhom-pve (the N100) joined the tailnet as a host package (100.70.170.35, expiry disabled); DooPlex already advertised 192.168.0.0/24 via its GitOps k3s admin-system/tailscale pod (100.107.87.53). ssh felhom-pve now targets the tailnet IP (direct-over-LAN at home, tunnel when away); felhom-pve-lan = 192.168.0.162 fallback. --accept-dns=false on the host (pi-hole split-horizon preserved). The host was converted static→DHCP (reservation MAC 68:1d:ef:5d:a6:64→.162 keeps home behaviour). Measured spike: --accept-routes on the N100 while on 192.168.0.0/24 hairpins the local subnet (table 52 overrides main) → kept OFF, it's a travel-only opt-in. PBS is offsite (its own WireGuard tunnel to Hetzner, not the home LAN) so it needs none of this. Recorded, NOT fixed: the controller↔agent plane is hard-pinned to 192.168.0.162 (felhom-agent listen_addr + guest bootstrap.json), so the box is reachable + backs up off-LAN but its local operator plane is not yet portable. Full doc: documentation/operations/tailscale.md. No code changed.

• 2026-07-19 — CC WORKFLOW MIGRATED TO DOOPLEX. Claude Code now runs natively on DooPlex (192.168.0.180, Debian 13, kisfenyo) in a tmux session, working directly in /mnt/5_hdd/felhom.eu/git/<repo>. DooPlex is now the canonical CC environment; the Windows workstation trees are stale as of felhom.eu ee48a48288, felhom-agent d81fddc7b1, felhom-controller ac7323dc9a, app-catalog-felhom.eu a14c97db65 (unchanged — it had no environment-specific content). Builds are local commands; felhom-pve is one ssh hop; kubectl is local. claude-in-chrome is NOT available here — endpoint-level validation is the standard method. New clean-tree gate before any build (git status --porcelain empty AND HEAD == origin/main), because the CC working tree is now the tree build.sh builds from. Windows knowledge is preserved in per-file "Legacy: Windows workstation" sections and the versioned workspace file documentation/runbooks/workspace-CLAUDE.md (live copy: /mnt/5_hdd/felhom.eu/git/CLAUDE.md).

• 2026-07-17 — CUSTOMER SELF-BIND shipped (hub v0.66.0 + ISO scripts v1.20.0, R-27 slice 1). A customer binds their OWN freshly-installed appliance without the operator. Operator clicks "Send self-bind link" on the customer Setup tab → hub mints a 7-day tokenized capability link → emails it (Hungarian, sibling sender, NOT via the claim engine) → customer opens the public, Hungarian /bind/<token> page (no login — the URL token IS the auth) → enters the console pairing code (shown on the box screen) + their retrieval passphrase → hub stages the bind via the same BindAppliance (provenance customer_selfbind) → the box's ~30 s appliance poll delivers. Viktor's three rulings, verbatim: (a) "only their own visible" → console pairing code, no appliance list ever rendered; (b) first-box entry → operator-sent tokenized capability link over Hungarian email; (c) lockout after 5 failed attempts → token locks, "call support". Wrong code and wrong passphrase = one identical generic failure (no oracle; both factors compared unconditionally); expiry falls back to operator-bind unchanged. THE TRAP (§9.2): one public prefix /bind/, exempted from auth+CSRF at both /login gate sites via a single isPublicBindPath predicate (tight trailing-slash match; ServeMux ..-cleans; handler rejects / in the token) — 4 red-proofs verified red-then-green (lockout→C1, oracle→B, widened-prefix→E, single-active→C4). GC verdict: no appliance-staleness GC exists (applianceStaleAfter is a display badge only) → the 7-day token TTL stands alone, single-active-per-customer, no reaper needed. Controller/agent untouched; R-27b (controller second-box dismissable prompt) DEFERRED, mechanism sketched. Green: full hub build/vet/test (17 ok, +9 self-bind tests) + bash -n. DEPLOYED + LIVE (hub 0.66.0 on k3s, Synced/Healthy). Live through the real ingress: GET /bind/<garbage> → 200 Hungarian expired page (public, logged-out, no-oracle — the 200-not-500 proves selfbind_tokens migrated live); / + /hosts → 302 /login (gate intact); POST /bind/<garbage> no-CSRF → 200 (exempt) while POST /customers/acme/block no-CSRF → 302 /login (exemption is /bind/-only). PENDING (operator-gated, NOT-live): operator "Send self-bind link" + the customer's real two-factor success (hub UI password-gated → CC can't log in) + the new-ISO console banner on a re-imaged box → fold into the R-1 rehearsal. See REPORT.md.

• 2026-07-17 — PBS DR STORAGE VISIBILITY + Offsite tab split + dual gauges shipped (hub v0.65.0 + tenantsync v1.2.0, R-5), LIVE. Scoping correction (Viktor): "restic box" and "PBS box" are NOT two Hetzner Storage Boxes — restic = subaccounts on the shared Hetzner box (Hetzner API, v0.64.0); PBS DR = the felhom-offsite PBS datastore on ep0 (NO Hetzner API; the hub had no read of its fill). Option A (ruled): a read-only usage op on the felhom-tenantsync ep0 forced command (twin of fingerprint, df on the datastore path — no customer_id, no admin token, NO mutation), polled by monitor.PBSDRBoxChecker (OffsiteBoxChecker clone over a usageReader seam; 15-min throttle; three states ok/unavailable/degraded; escalation-only pbsdr_box_fill on the "pbsdr-box" operator scope). /offsite now has Restic + PBS DR tabs (endpoint cards under PBS DR — the endpoint IS the PBS host); the dashboard has two gauges (restic %·ratio, PBS %). Graceful degradation: the hub deploy is INDEPENDENT of the ep0 update — an ep0 ≤ v1.1.0 → ErrUsageUnsupported → the gauge shows "n/a" until updated (no hub redeploy). PROVEN-LIVE (data pipeline, both gauges): ep0 updated to v1.2.0 this session; the hub polled it — PBS-DR box refreshed: 19.1% full (7.1 GB of 37.2 GB). DECISION: PBS thresholds are separate keys (PBSDRBoxFill*) defaulting to restic's 80/90 (independently tunable; Viktor's call for a different pair); no oversubscription for PBS (namespaces, not quotas). OPEN: UI render unit-verified only (hub UI password-gated); the PBS fill alert email is unit + red-proof verified, NOT fired live (datastore nominal at 19%). READ-ONLY against ep0 + Hetzner. See REPORT.md.

• 2026-07-17 — OFFSITE POOL-BOX AGGREGATE shipped (hub v0.64.0, R-5). The operator now sees the shared pool box's real state on the hub: total fill vs capacity, Σ(shared soft quotas)/capacity (the oversubscription ratio), per-customer usage/quota bars, and a box-level operator alert (fill 80/90%, oversub 2.0×). New monitor.OffsiteBoxChecker (OffsiteChecker-sibling): fetch-throttled to one Hetzner GET / 15 min, cached BoxSnapshot, escalation-only + recovery re-arm; two INDEPENDENT signals (fill + oversub). Capacity = storage_box_type.size, usage = stats.size (Phase-0-pinned, bytes); Σ(quota) from the ConfigJSON Descriptor (offsite.ReadDescriptor, new — NEVER the report echo; dedicated + disabled excluded); per-customer USAGE from the report echo. Events scope "pool-box" → operator channel only, no SaveEvent. Web: Offsite-tab panel + compact dashboard tile, reading the cached snapshot only (never fetches). PROVEN-LIVE (data pipeline): the in-cluster checker computed the real box — 0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x. DECISION: thresholds (80/90/2.0×) are Claude's encoding — Viktor's ruling pending; named Alerting.OffsiteBox* keys = the one-line flip. OPEN: the UI render is unit-verified only (hub UI password-gated → CC can't screenshot); the alert emails are unit + red-proof verified but NOT fired live (real pool nominal — a live-fire would email Viktor; offered on demand). READ-ONLY against Hetzner throughout. See REPORT.md.

• 2026-07-17 — SYSTEM-INITIATED IMMEDIACY wired per-plane (hub v0.63.0). The immediate-sync arc covered only OPERATOR-initiated changes; system-initiated desired-state mutations bumped the generation silently, so a freshly onboarded box waited a full agent tick for state the hub had already minted (observed live at slice-C onboarding). Decision: wire the existing, live-proven notifiers into every system site on the CORRECT plane — agent-plane poke.Notifier at PBSDRAutoProvision (the exact lag), ReissuePBSDR (which also lifts the pbsdrheal reconciler's escalation with zero reconciler changes), handlePBSDRReissue, and the two admin api desired-state writers (new nil-safe Poker seam); controller-plane intent.Hub.Bump at reissueOnReenroll. All fire AFTER the store write only. The pbsdrheal Restage path is deliberately poke-free (no generation bump → the agent's 60 s pbsdr ticker is its pickup path — a poke there is a verified no-op); the WG register/delete bumps stay poke-free (undeliverable pre-tunnel / transport removed — the agent fast-tick SECONDARY owns the register leg). The first-hold-baseline race is still OPEN by design (a bump landing during a fresh controller's FIRST wait is recorded as baseline without firing — the ruled fix is carrying intent_gen in the report ACK, its own future task). Unit-tested + red-proofed; live legs fire only on real system events (fold into the rehearsal bind sequence). See REPORT.md + hub/CHANGELOG.md v0.63.0.

• 2026-07-17 — R-21 SLICE C SHIPPED: the universal secret-free ISO (hub v0.62.0 + scripts v1.19.0, LIVE). The distributed ISO carries NO customer secret. A box booted from build-felhom-iso.sh --pairing self-registers as an UNCLAIMED APPLIANCE (POST /api/v1/appliance/register, keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker; token = sha256-stored, 256-bit), the operator BINDS it to a customer on the Hosts page (new "Unclaimed appliances" section, SSH-key fingerprints, host-count display-only), and the hub delivers customer-id + retrieval passphrase ONCE (one-shot poll: 404 no-oracle / 204 unbound / 200 deliver / 410 gone). The bootstrap is one unit, two modes — direct (env-baked, byte-identical, regression-proven zero-appliance-calls) vs pairing (register→poll→write env→fall through to direct). Red-proofs: one-shot delivery + register idempotency (both proven red). Live-verified through the public ingress (register→token, 204, bogus→404). Artifact proven secret-free (baked env = hub URL only; manifest secret-bearing: no). Bind is operator-password-gated → the live boot→bind→day-0 composition (+ physical N100) folds into the supervised rehearsal, which now runs the COMPLETE final flow in one pass (RESET demo → generic ISO → bind → day-0). SSH-key pinning = stored + fingerprints displayed (host-attach on bind deferred, no clean surface). Self-bind page = R-27. R-21 now SHIPPED (A+B+C). See REPORT.md.

• 2026-07-17 — R-21 SLICE B SHIPPED: firmware loader option (scripts v1.18.0). Closes N100 F1 (HIGH) — cheap AMI (AN3PLUS 0.01-class) firmware can't USB-boot the ISO's stock GRUB (relocation 0x0). build-felhom-iso.sh --loader shim|mkimage (default shim, unchanged; profile-settable via FELHOM_LOADER). mkimage = a monolithic grub-mkimage loader built from the ISO's own GRUB (module set from its grub.cfg, embedded search --fs-uuid→configfile), swapped into the ISO9660 tree + efi.img ESP by mkimage-surgery.sh after prepare-iso (payload provably untouched), re-mastered with xorriso preserving BIOS-hybrid + UEFI + GPT-ESP (drops only the irrelevant Apple HFS+/APM). Unsigned → target Secure Boot MUST be OFF. Loud: banner + manifest loader: + -mkimage filename. New profiles/n100.profile; Dockerfile gains grub/mtools. RUNBOOK-B legs (nested VM 311, OVMF): shim boots+installs under SB-enforcing + SeaBIOS; mkimage boots+installs under SB-off; mkimage under SB-enforcing FAILS Access Denied (red-proof); surgery byte-identical payload. Physical N100 boot still pending → supervised rehearsal (an n100-safety match-nothing ISO built + sha-recorded, unbooted). PXE stays a deferred R-21 note. See REPORT.md.

• 2026-07-17 — CUSTOMER RESET SHIPPED: hub v0.61.0 + felhom-tenantsync v1.1.0 (LIVE). The middle lifecycle tier (host delete < RESET < customer Delete). One operator action → pre-first-install: all OPERATIONAL state dies (offsite repo, PBS namespace+backups, DR recipe, one-time secret, claim state, retained escrow custody); IDENTITY + basic config (incl. the offsite tier CHOICE) + provenance + events SURVIVE. Rulings: separate escrow-custody ack; clears claim (fresh code next onboarding); REFUSES while any host row exists; live-counted confirm inventory. Discipline: external teardown FIRST, DB purge LAST, every leg idempotent → partial run re-runs from the top (purge withheld until externals ok). New: store/customer_reset.go (journal+inventory+ack-gated purge), claim.ResetToUnclaimed, offsite.{Deprovision,OffsiteIdentifier,ClearProvisionedDescriptor}, tenantsync.Deprovision + felhom-tenantsync.sh deprovision op (v1.1.0, destroys ns+groups+token, shared user untouched), web/customer_reset.go (GET inventory JSON / POST orchestration) + an amber RESET card distinct from the red Danger-zone Delete. Red-proofs: ack-gate + partial-failure resumability (both proven red). Live-drilled on ep0 (throwaway drill-reset-01 with a real backup): deprovision deleted:true, idempotent re-run deleted:false, all 3 real tenants survived. Hetzner-delete + the password-gated web POST covered by tests (offsite Deprovision mirrors live-proven ReissueCredentials). See REPORT.md.

• 2026-07-17 — HOST-DELETE DEMOTES ESCROW: hub v0.60.1 (LIVE). Closes the v0.60.0 review gap: DeleteHost(deleteEscrow=true) now DEMOTES the current escrow blob into host_escrow_superseded (copy-before-delete, same tx) + spares existing superseded rows — never destroys custody. The customer Danger-zone Delete (DeleteCustomerConfig, which before touched ONLY customer_configs) is now the one true purge point: cascades both escrow tables for all the customer's hosts incl. already-deleted (via F-14 host_deletions). One shared copy routine demoteCurrentEscrowTx (reused by SaveHostEscrow). Checkbox/refusal/Danger-zone wording → demotion. S6b VERDICT: OBSOLETE — re-enroll upserts cleanly (UpsertHost ON CONFLICT DO UPDATE; handleAdminCreateHost no dup refusal) + v0.57.0 arc auto-fires re-issues (ROADMAP R-3 refined). Red-proofs TestDeleteHost_DemotesEscrowNeverDestroys + TestDeleteCustomer_PurgesEscrowCustody. Observation: customer-delete still doesn't touch host rows / the broader offboarding (rehearsal's question). REPORT.md.

• 2026-07-17 — OFFSITE CONTINUITY Part B SHIPPED: hub v0.60.0 (LIVE). Superseded-escrow retention (data-first, Viktor's ruling). SaveHostEscrow's destructive ON CONFLICT overwrite destroyed the old passphrase's only copy (the reinstall-orphan incident); now a new different-sha blob RETAINS the old into the new host_escrow_superseded history table (tx, before overwrite); same-sha = idempotent. ACK/restore read the current row unchanged. escrow_superseded event + operator retained-count on host detail; registered offbox_repo_orphaned/offbox_repo_reset. Red-proof TestSaveHostEscrow_RetainsSuperseded. Pairs with controller v0.142.0 (Part A orphaned-repo guard). Guided old-history recovery = R-26 (not built). See REPORT.md.

• 2026-07-17 — N100 polish (F5 scripts + F6/F7 controller). F5 (host-install v1.17.0): appliance mode auto-sizes the guest (RAM=clamp(host-4096,min 4096,max host-2048,ceil host-1024) + cores=host-1 min 2) when no explicit cap; explicit --memory/--cores win. Harness red-proof (8/16/32 GB). Same commit: the stale "EMPTY by default" operator-key comment fixed (R-16 doc-drift, keys are PINNED). F6/F7 landed in controller v0.141.0 (LIVE on demo; fresh-USB wizard enroll+format now PROVEN-LIVE — see the felhom-controller REPORT). New ROADMAP R-24 (guest resources as desired-state / live resize = F5 follow-on). Operator sidebar #1 (BIOS State-After-G3→Power-On) added to scripts/iso/README.md. See REPORT.md.

• 2026-07-16 — DIRECTION-2a AGENT-PLANE POKE SHIPPED: hub v0.59.0 (LIVE). The agent-plane sibling of the v0.58.0 wait channel: internal/poke — a pinned-host-key SSH sender (wgsync sibling) + fire-and-forget Notifier (PokeHost/PokeAllHosts, nil-safe, detached) that nudges a box in seconds via a CONTENTLESS UDP poke relayed hub → ep0 felhom-poke forced-cmd → wg0-origin → the box WG /32:51822. Wired at applyPBSDR (per-host) + handleSetArtifacts (fleet). ep0 surface (scripts/felhom-poke.sh, non-root, no sudoers; runbook §11) + Secret/agent-poke + manifests/hub.yaml (tag 0.59.0). Deployed live (ArgoCD Synced/Healthy, agent-plane poke enabled); ep0 leg live-validated (poke-fired; non-WG + rm -rf / refused). Source note: the MinAgent-floor save does not bump per-host generation (self-update rides signed-ops) → the fleet poke accelerates the next report, not a delta. Pairs with felhom-agent v0.89.0 (the listener). PROVEN-LIVE 2026-07-17: the operator's real manifest save logged poke: sync-poke delivered to 10.77.0.2 (+ floor HELD for the below-MinAgent box); the demo (now 0.89.0) received a poke in ~31 ms and fired the immediate desired-state cycle (save→tick ≈ ~0.45 s). All three live legs (Scenario 4 A/B train, Scenario 1 R-22 self-heal, Scenario 3 latency) done; capability-map poke + F4 rows upgraded to PROVEN-LIVE. See REPORT.md.

• 2026-07-16 — DIRECTION-2 IMMEDIATE-SYNC SHIPPED: hub v0.58.0. The hub→box "sync now" wait channel (option b of documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md). New internal/intent in-memory per-customer generation notifier (Bump/Wait/Close; coalesce-to-latest; race-closer; in-memory by design → a restart costs one harmless full-state report, never a storm). GET /api/v1/wait long-poll: per-customer only (global key 400; A can't see B), 240 s hold with a 25 s heartbeat newline that defeats nginx's 60 s proxy_read_timeout without any ingress annotation (WriteTimeout lifted per-connection via ResponseController; global timeouts untouched). Every operator-intent web handler bumps AFTER its store write (config CRUD, claim resend, offsite re-issue/freeze, password regen, block/unblock, floors [global bumps all managed], controller log-tail + log-bundle). intentHub.Close() before server.Shutdown. Pairs with controller v0.140.0 (the long-poll client). The immediate-sync arc is COMPLETE for the controller plane (Direction 1 = ctrl v0.139.0, Direction 2 = hub v0.58.0 / ctrl v0.140.0); the agent-plane ep0-poke stays PARKED in the OOB arc. Detail: hub/CHANGELOG.md v0.58.0, REPORT.md.

• 2026-07-16 — REINSTALL ARC SHIPPED: hub v0.57.0 (F2/F3/2.3). The clean-slate reinstall-of-existing-customer path is now first-class (Peti R-1 prerequisite). On host-enroll's mint path (single-bump-per-re-enroll): claim re-issue for a claimed customer (ReissueForReenroll, reset code — the hub never stores the password, so fork B not A), offsite re-issue to the fresh box (ReissueOffsiteForCustomer), and escrow-honesty — re-issuing offsite marks the escrow STALE (MarkEscrowStale), withholds the mismatched restic-hash from auto-confirm, and shows "stale" not "ceremony done" (2.3, red-proofed). Part 0 unblocked the demo live (ACL grant on /storage/felhom-offsite → PBS-DR converged, escrow.pbs_storage_id seeded). Controller + scripts unchanged — both spec premises were contradicted by source (the controller reads escrow prereqs live from the agent; the installer can't know the descriptor-provisioned storage id). F4 root fix is agent-side → ROADMAP R-22 (agent train: the reconcile pre-check 403 must not abort before the root-run apply grants the ACL). Detail: hub/CHANGELOG.md v0.57.0, REPORT.md.

• 2026-07-16 — R-21 PHYSICAL RUN on the demo N100 (tests/VALIDATION-n100-baremetal-2026-07-16.md). demo-felhom host reinstalled clean-slate from a pipeline-built ISO; first-boot chain reached rc-0 on the first attempt on real hardware (closes slice A's operator-gated boundary), serial-filter safety proven on metal (external-HDD canary byte-identical), PBS-DR reconciler self-healed on the reused peer. The obstacle was firmware, not us: the cheap AMI AN3PLUS 0.01 board can't UEFI-boot the ISO's GRUB from USB (relocation 0x0; SB-off + shim-bypass don't help) — worked around live by rebuilding the stick's loader with grub-mkimage from the box's own working GRUB. Findings (to tackle): F1 firmware boot loader (slice-B), F2 claim-re-issue on reinstall, F3 offsite-re-issue, F4 PBS-DR ACL on non-default storage id, F5 guest-RAM configurability, F6 drive-init doesn't mount+attach, F7 back-route. Box healthy, BookStack deployed, demo live.

• 2026-07-16 — R-21 SLICE A SHIPPED: bare-metal Felhom ISO pipeline (scripts/iso/, scripts v1.16.0). A DooPlex pipeline (build-felhom-iso.sh + Dockerfile.assistant) turns the official PVE ISO into a Felhom auto-install ISO whose first-boot stub installs a retry-forever felhom-bootstrap unit that unattended-fetches felhom-host-install.sh from the public felhom.eu/scripts/ channel and runs it (host-install unmodified). Validated on VM 310: build gate + red-proof, disk-filter fail-safe, stub→unit→real-fetch→invoke→retry, resume-decision, exactly-once, no-net retry+recovery — all GREEN. Rulings settled: stub is from-iso (no cert-rotation risk); ordering fully-up; the retry unit owns all network work. §4.4 finding (shapes slice C): the distribution URL carries no secret, but the unattended install needs the customer retrieval passphrase → the slice-A ISO is secret-bearing (supervised/single-use; the bootstrap shreds the env on success). OPEN: host-install rc-0 terminal success (hub-enrolled + guest-provisioned) is operator-gated — a drill customer needs the password-gated create-UI (a live-hub.db insert is unsafe). Slice B = physical N100 RUNBOOK; slice C = claim-pairing to de-secret the distributed ISO. Detail: REPORT.md, documentation/audits/SPIKE-baremetal-iso-2026-07-16.md.

• 2026-07-13 eve — ESCROW CEREMONY WIZARD SHIPPED (agent v0.88.0 + controller v0.127.0; this repo: host-install v1.16.0 + RUNBOOK rewrite). The recovery-code ceremony is now customer-driveable from the controller (/backup/escrow): one-shot R reveal, re-stage-first, Scenario-F stale-blob re-check (fired LIVE on both boxes' hash-less blobs at first ACK; drill's repaired via a real ceremony — hub hash now covers). Ruling F1 (2026-07-13), settled: R transiting the Cloudflare tunnel once at reveal is an accepted risk — same trust class as the claim code/login password; LAN-direct delivery is PARKED. (threat model: runbooks/RUNBOOK-escrow-ceremony.md). host-install v1.16.0 = FELHOM_ESCROW rides the canonical sudoers fetch; hub hostInstallVersion synced (gate green), hub NOT redeployed (const rides the next train). OPEN: supervised full-browser wizard pass (Viktor's drill login; also demo, to clear its legacy stale warning); agent 0.88.0 publish + Day-0 vouch at the next train.

• 2026-07-13 — CLOSING BUNDLE SHIPPED: hub v0.53.0 — the last engineering items on the pre-tester board: F-14 gated auto-Reissue (deletion provenance host_deletions in-tx + token_exists gate → the EXISTING Reissue op + pbsdr_auto_reissue audit event; no record / un-acked → refusal byte-unchanged), dead-host roll-up honesty (customer status = worst of controller-derived + per-host hostStatus on dashboard//configs/detail, cause chip names the host; the Peti 23h-masking shape is the pinned fixture), operator bearer OUT OF GIT (Secret/report-api secretKeyRef non-optional + REPORT_API_KEY env override + scripts/manifest_bearer_gate.py; VALUE ROTATION = supervised step in documentation/runbooks/secrets.md — the git-history copy is dead only after it). Two operator rulings (2026-07-13), recorded verbatim — future sessions must treat these as settled:

  1. F-14: auto-Reissue of offsite credentials is permitted ONLY when the hub's own deletion record shows the tenancy's owning host was removed through the escrow-ack flow — acknowledged destruction is not silent re-keying. No record → manual Re-issue stays the only path.
  2. Roll-up: a customer's status may never look better than its worst expected host. Single staleness definition (hostStatusOf) — no second threshold anywhere. Gate additions: python scripts/manifest_bearer_gate.py after ANY manifests/ change (64-hex bearer shapes; felhom.secret.yaml carve-out = visible KNOWN-BACKLOG line). GOTCHA: the repo .gitignore *secret* pattern silently un-tracks any "secret"-named script — hence the "bearer" name. The controller repo's controller.yaml.example carried the REAL bearer literal — scrubbed (felhom-controller 51c871a).

• 2026-07-13 — POLISH BATCH SHIPPED: hub v0.52.0 (+ controller v0.123.0) — take-two F-15 CLOSED (reset-request response carries the rotated code hash — ACK shape/guarantee — emailed codes work immediately; live 1 s apply + first-try accept on Viktor's exact failure path) and F-16 CLOSED (zero native confirm() in the hub UI — inline_confirm.html partial, Igen/Mégse inline two-step, asserted by scripts/hub_confirm_gate.py — which exists but is NOT enforced: it is invoked by nothing (R-29); danger-zone typed-confirm untouched; live: offsite re-issue completed under browser automation without freezing). Operator resend keeps the ACK lag — flash says so. Take-two §4 dispositions updated (F-15/F-16/zero-toggle obs → SHIPPED). Remaining from take-two: F-14 (needs operator ruling: auto-Reissue re-adopt vs never-silently-re-key) + S5 DR-restore drill; hub bearer key rotation still DUE.

• 2026-07-12 — DR-TIER-BY-DEFAULT BATCH SHIPPED: installer v1.15.0 + hub v0.51.0 (+ agent v0.86.0 in felhom-agent) — closes DRILL-day0-vm F-1/F-2/F-3/F-7/F-8/F-9/F-10 and F-6 (by policy). The five operator decisions (2026-07-12), recorded verbatim — future sessions must treat these as settled:

  1. Every install deploys the full DR plumbing uniformly (pbs-apply wrapper, age, WG enabled, full ACL). Capability is baked; activation is a hub flag.
  2. DR tier defaults ON for new customers. Disabling is the per-customer exception — and it removes the Felhom-side cost entirely (no PBS namespace, no endpoint dependency).
  3. Offsite app backup requires the DR tier ON (fork-4 needs the ceremony; the ceremony needs PBS). Enforced hub-side by extending the drill-proven dependency guard.
  4. Identity-only escrow is formally PARKED — closed by policy, not implemented. Revisit only if a customer class demands lean-tier offsite. Record in the ceremony runbook + CONTEXT so future sessions know it was a decision, not an oversight.
  5. WG is base infrastructure, not a DR component — always-on like cloudflared; it is also the future OOB-management transport. BYO consent: a disclosure line lands in the tester-agreement doc; per-customer hub disable remains possible. Mechanics: installer ships felhom-pbs-apply + age + wg_tunnel.enabled: true (byo wg-off assert RETIRED — consent = typed disclosure ack + tester-agreement line), single VERSION source with a cross-check against the hub Setup-tab const written as scripts/hostinstall_gates.py — but NOT enforced: the script exists, is RED today (1.19.0 != 1.22.0) and is invoked by nothing (tracked as R-94 leg (b), class R-29) — default ACL keeps felhom-pbs (doc advice REVERSED — narrowing caused the drill 403). Hub: per-customer dr_tier flag (new default ON; legacy backfilled from descriptor reality, one-time), cascade-not-error semantics + per-stage status on the edit form, WG-registration auto-provision hook (scenario A hands-free), offsite-requires-DR guard (exact F-6 message), host-page capability chips (agent 0.86.0 inactive = neutral; pre-1.15 "binary not found" → migration hint). felhom-pve's latent F-3 state fixed live; Peti's box unreachable (his F-3 check + D/E/G deferred).

• 2026-07-12 — HUB v0.50.0 SHIPPED + LIVE: customer-claim password arc (closes DRILL-day0-vm F-4/F-5). The customer OWNS the dashboard password via a one-time claim code the hub emails to the registered address (bcrypt(code) stored, plaintext only in the send). internal/claim engine (EnsureIssued idempotent Day-0/live-box issue+email, Resend, RequestReset 3/day, MarkClaimed set-only) + store.customer_claims (RotateClaimCode bumps generation + preserves claimed_at) + delivery (config bake web.claim_code_*, report-ACK claim object, POST /api/v1/claim/reset-request) + Setup-tab chip/resend + 3 Hungarian emails. Pairs with controller v0.122.0 (the gate). LIVE-proven on drill box demo-vm-felhom: gate ON via the real edge (/ → 302 claim page, /api → 401); code emailed to the registered address. Deployed felhom-hub:0.50.0 (ArgoCD synced/healthy). Floor raise 0.120→0.122 = operator's supervised LAST step. Observed: the hub issues a claim code for EVERY reporting customer on first report (benign for pre-0.122 boxes — no gate, ACK ignored). Report: documentation/audits/DRILL-day0-vm-2026-07-12.md (F-4/F-5 RESOLVED).

• 2026-07-12 — HUB v0.49.0 SHIPPED: Edit tab merge (edit-a) + scoped auto-refresh + riders (CHANGELOG v0.49.0; commits e740147 → 2e03de1 → 1d94b1a + docs/manifest). The standalone customer edit page merged into the customer page's Edit tab (renamed from Settings): shared config_form_body sub-template + configFormData builder (REUSE.md rows added), Block/Delete relocated to a danger-zone card, header cleaned, GET /configs/{id}/edit → 302 #tab=edit, post-action redirects anchored to their tab, settings→edit legacy-hash alias. Update handler gained server-side required-field validation (error path preserves SUBMITTED values — red-proofed). The 60s auto-refresh is now scoped: fires only on live tabs (data-live-tabs="overview,applications,events,host") and never while a form is dirty; muted (paused) hint. Riders: /style.css?v={{hubVersion}} cache-bust on every template (closes the v0.47.0 max-age=3600 gotcha) + the CLAUDE.md git add -A ban codified. NOTE: the task spec said v0.48.0, but v0.48.0 (app_start_failed) had already shipped — published tags are never re-pointed, so this train is v0.49.0. Out of scope (observed, not done): type-to-confirm for customer delete (still confirm(), relocated as-is); the workspace-root E:\git\CLAUDE.md and other repos' CLAUDE.md files still carry git add -A snippets — flag for their own sessions.

• 2026-07-12 — Day-0 VM DRILL COMPLETE (auto-confirm FIRST LIVE FIRING): full arc proven on a fresh nested-PVE box — appliance Day-0 → floor-at-first-report → ceremony → auto-confirm pending→escrowed in ~7.5 min, zero clicks → offsite backup + restore round-trip. Two HIGH gaps: F-4 no operator password-set path exists (G10 unclosable, dashboards born OPEN — blocks tester gate) and F-6 identity-only ceremony never implemented (no-PBS appliance can't escrow — drill forked to PBS DR tier = full Peti-sequence rehearsal, all green). Installer fresh-box gaps: felhom-pbs-apply not shipped (F-7), age missing (F-10), root-owned guests/ parents (F-3 — check demo for the latent copy), silent root@pam rotation UX (F-8). Runbook fixes committed (day0 A.2 anonymous-fetch; escrow-ceremony identity-only claim CORRECTED + age prereq). Report: documentation/audits/DRILL-day0-vm-2026-07-12.md. Drill VM qm 300 kept (3 snapshots) for re-drills; teardown list in report §9.

• 2026-07-12 — CAMPAIGN-3 Task A SHIPPED: agent v0.85.0 boot/recovery plane + appliance self-heal. Fixes F12 (CRITICAL boot ordering cycle — templates drop network-online, MigrateNetworkUnits repairs installed units), F11/F10/F9 (reassert: fstype-driven classify, reset-failed+rearm, per-unit verdict, + re-arm-on-guest-blind matrix-correction), F2/F1 (zero residue: reset-failed + rmdir), the guest-hook rc255 belt (recover+timeout, wrapper no longer execs), and the appliance node self-heal (internal/selfheal: host networking recovery, gated on deployment_mode — host-install --mode templates it; byo remedy unreachable). 3 sudoers grants added (reset-failed, rmdir, start networking). Live matrix on demo felhom-pve: host reboot ×2 = 0 ordering-cycle lines both boots (F12 dead); appliance self-heal recovered networking on attempt 1; F10 reset-failed+rearm sweep loud; zero residue; guest-reboot heals. NEW finding F13 (HIGH, deferred): active nfs4 under mp8 can fail PVE rbind (rc255); NAS-down-at-boot → apps Created (fix-3). Peti's box (agent 0.81) unaffected until his rollout — carries the F12 coin flip if it has a NAS share (interim drop-in in the agent REPORT). host-install v1.14.0; scripts CHANGELOG. Details: documentation/controller/network-storage-nas.md.

• 2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; host recovered, RCA closed. Unattended chaos campaign 22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/ restore byte-identical/EIO instant/stub badge + deploy-409 live-validated). The NAS automount plane produced 2 CRITICAL + 3 HIGH: F12 CRITICAL (= the overnight host loss, RCA'd next morning): the agent automount template's After/Wants=network-online.target creates a boot ordering cycle — systemd deletes an arbitrary job to break it; boot -1 sacrificed networking (host dark 7h, hardware exonerated), boot 0 sacrificed the automount (NAS dead). Every host boot with an enrolled share = coin flip until the template drops that ordering (_netdev on the .mount suffices). F10 CRITICAL (start-limit never re-armed, once blocked guest start), F11/F9 HIGH (post-start reassert "skip-active" skips the healing automount restart; agent-restart sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots; F7 HIGH (in-place dump write → 0-byte tar replaced last good backup during outage window). Fix-6 numbers: ring horizon 55min idle / 6.5min under load, wiped on every restart. Box left in defined state (6 wave apps healthy, temps removed, NAS baseline-identical). Full ledger/evidence: documentation/audits/CAMPAIGN-3-2026-07-11.md + 180:~/campaign3/.

• 2026-07-11 — HUB v0.47.0 UI REORGANIZATION SHIPPED (CHANGELOG v0.47.0; commits 9f29bf3 → 0daddcd + docs). Five deliverables: (1) .data-table td a:not(.btn) button-contrast fix; (2) customer page = 8 hash tabs (#tab=…, sticky summary strip, Events error badge, no-JS graceful degradation — panels hide only via a JS-added body class); (3) shared host_detail_body sub-template rendered by /hosts/{id} AND the new per-customer Host tab (a list by design; store.ListHostsByCustomer + hostDetailData builder); (4) stale host removal — GET /hosts/{id}/delete-impact (counts/booleans only) + POST /hosts/{id}/delete behind type-to-confirm; ONLINE → 409 always (no override), escrow needs an explicit checkbox (ErrHostEscrowPresent, tx never starts), one-tx cascade incl. the bound wg peer (wgsync's 5-min declarative push converges the endpoint; log bundles die by scope_id == host_id only); (5) /offsite multi-endpoint management UI — all wg_endpoints rows as cards + add/edit/delete with 409 guards (peers-in-subnet pins subnet + delete). Deferral (explicit): peer allocation, the wgsync reconciler push and the desired-state merge stay lowest-endpoint-id (GetWGEndpoint untouched); the future arc is a wg_peers.endpoint_id migration + per-endpoint allocation/reconciler/desired-state. Five red-proofs ran (online gate, escrow ack, bundle scope, endpoint-delete guard, subnet-change guard). Resolved: the earlier ⚠️ about 146d165 sweeping the Part-4 WIP — the sweep caught hosts.go mid-red-proof (escrow ack bypassed); 068427a restored the gate; the arc is now complete and author-reviewed. Stale-doc note: the workspace-root CLAUDE.md's hub deploy wording predates GitOps — deploys are manifest-tag bumps + deliberate ArgoCD sync, never kubectl set image.

• 2026-07-11 — RCA FIXES 1+2+4 SHIPPED + Q1c GREEN (agent v0.84.0 + controller v0.117.0 LIVE on demo). Reboot survival: automatic since agent 0.84.0, live-proven 2026-07-11 21:20 (pct reboot 9201 under operator GO — the hook re-armed + verified nas-media DURING the start task, CWA auto-started healthy, health ok first cycle, no badge). Agent: ReassertNetworkMounts (hook post-start + startup; stop+enable-now on existing sudoers verbs — NO sudoers widening needed; never in periodic paths). Controller: statfs consuming-namespace classifier → probe not_network_fs (stub can never verify), deploy-time stub refusal (409; idle autofs deploys fine), distinct stub badge (wins over unreachable; unreachable line byte-identical), deploy view selects stored HDD_PATH (RCA fix 4). MinAgent unchanged 0.81.0. All red-proofs recorded in repo REPORTs. Feature doc: documentation/controller/network-storage-nas.md §Reboot survival; audit §7 follow-up added. Queued: RCA fixes 3/5/6; Peti publish train now carries 0.84 + 0.117.

• 2026-07-11 — NAS/CWA RCA (diagnostic, no fixes): documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md. The nas-media re-add verified legitimately (new host automounts DO propagate into running guests); a second pct reboot 9201 at 15:39 severed the trigger → guest-side stub; CWA deployed correctly to the NAS (app.yaml + binds + data all nas-media) but is DOWN since 15:39 (docker can't recreate the bind source; perms luckily blocked any system-drive writes; zero data loss). UI "felhom-usb" = display bug (deploy.html selects by IsDefault, never stored HDD_PATH); hub device-ID WARN was CORRECT and the only guard that fired; NAS badge masks because it asks the agent (host ns). 6 prioritized fix tasks in the audit §5. Remediation §6 PERFORMED on operator GO (~20:14–20:20 CEST): automount-unit restart re-planted the trigger in the running guest instantly (fix-1 mechanism live-proven), CWA restarted healthy on the NAS library, uid-1000 write OK, controller health back to status=ok at 20:18. Demo fully healed.

• 2026-07-11 — OBSERVABILITY PASS (hub v0.46.0 LIVE Synced/Healthy; agent v0.83.0 + controller v0.116.1 on demo). Remote pull-only log access to both box components' always-DEBUG capture rings. Hub: log_bundle_requests/log_bundles store (gzip, newest-3, 72 h TTL on the 60 s sweep), the secret gate fail-closed before storing (blocked: possible secret; [REDACTED]/ checksums pass), report-ACK controller_log_requested + heartbeat-envelope log_tail_requested channels (consume-once; pre-0.83 agents stay visibly pending), host-detail Diagnostics section (Request controller/agent logs + View/Download; the hosts read-only invariant amended: these two forms are the only actions). Conventions codified: documentation/runbooks/logging-conventions.md + CLAUDE.md one-liners in all three repos. OPEN: operator clicks both request buttons on hub.felhom.eu/hosts/ to close the live round-trip (UI password-gated; the flow is fully API-test-proven). Hub bearer key rotation still DUE (pre-existing).

• 2026-07-11 — PUBLISH-TRAIN RULES CODIFIED (documentation/runbooks/publish-train-rules.md):

  1. manifest before floor; 2. the manifest screen carries the LIVE DB floor (hub_settings.min_controller_version overrides env, acts immediately) — save the floor field LAST; 3. MinAgent gate — now HUB-ENFORCED PER-BOX (hub v0.45.0, store.ResolveManagedFloor), the manual fleet check retired; 4. box-level backstop = the controller's agent-capability gate (v0.114.0 route probe → v0.115.0 version compare). (Floor-UI separation + per-box gating SHIPPED in hub v0.45.0 — see the backlog-clear entry above; this rules-doc note supersedes the original "hub untouched / option-3 deferred" wording.)

• 2026-07-11 — NAS/coupling backlog CLEARED (hub v0.45.0 LIVE + agent v0.82.0/ctrl v0.115.0 on demo). Composite A–F. Hub v0.45.0 (Synced/Healthy): floor-UI separation + effective-floor source line (DB hub_settings vs env, store.ResolveGlobalFloor) + type-to-confirm w/ live below-floor count; per-box MinAgent conditional floor (store.ResolveManagedFloor HOLDS the controller floor for a box whose hosts.agent_version < the manifest's MinAgent, flags it on the Hosts dashboard) — manual fleet check retired (publish-train-rules.md rule 3). THE one comparator extracted to internal/semver. Agent v0.82.0 = X-Felhom-Agent-Version header (demo-only, Peti stays 0.81 = probe fallback); ctrl v0.115.0 = version-aware Supports + DSM-validated NFS guidance. DSM spike (real DSM 7.2 via virtual-dsm): consumer recipes validated E2E, hardlink caveat closed for Synology (record: documentation/audits/SPIKE-nas-dsm-2026-07-11.md); QNAP caveat stands. Q1c (supervised reboot) FAILED: a NAS automount trigger does NOT survive a guest reboot (empty-dir; agent lacks a network-mount reassert) — fix spec'd at documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md. NOT published (agent 0.82 demo-only). Record: felhom.eu/REPORT.md + each repo's REPORT.

• 2026-07-11 — PUBLISH TRAIN 0.81/0.113 COMPLETE — PETI LIVE on agent 0.81.0 + controller 0.113.0. Record: documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md. Agent 0.81.0 published (sha 914bb3b5…30df0b, live-bytes, anon-verified) + golden 0.113.0 baked/published (sha 66de5e64…82054c, drill VM, virgin-reverted); manifest + signed agent_update (key-id felhom-op-1) by Viktor → Peti 0.79.0→0.81.0 clean (no rollback); floor auto-confirm proof CAPTURED (10:41 0.112.0 → 10:42 0.113.0, anonymous pull proven on HIS box) — KEY LEARNING: the hub UI manifest screen ALSO sets the GLOBAL floor as a DB hub_settings.min_controller_version row that OVERRIDES the env and acts immediately (this is how Peti reached 0.112 and now 0.113 before the GitOps env bump 16390d0); a transient forbidden skew (ctrl 0.113 + agent 0.79, ~10 min) was benign only because zero NAS shares existed — next train: save the floor field LAST. OPEN: Peti journal-group one-liner (E), temp Git Sync creds deletion (G1), hub bearer key rotation now DUE (committed in manifests/hub.yaml + screenshot-exposed this session).

• 2026-07-11 — NAS verify-before-commit SHIPPED (agent v0.81.0 + controller v0.113.0 LIVE on demo; host-install v1.13.0). The spike's production task landed same-day: add = agent verify (retry=0 unit, journal-classified, auto-rollback) → controller uid-1000 write probe (--netprobe re-exec) → register LAST; full rollback on any failure; §3.2 Hungarian error map (nfs_export merged); storage_network.html redesigned (SMB-first, Route-A guidance); orphan rows; systemd-journal group (NO sudoers). Live A–E validated vs an isolated sim NAS, Route A proven in production (alien-uid 1060 server-side). Feature doc: documentation/controller/network-storage-nas.md. NOT published (no Gitea/Day-0 bump; Peti pending with the usermod one-liner). Record: felhom-controller/REPORT.md.

• 2026-07-11 — SPIKE: NAS verify mechanisms + NFS squash matrix — ALL FIVE QUESTIONS ANSWERED, no blocker. Record: documentation/audits/SPIKE-nas-verify-2026-07-11.md. Q1 in-guest access DOES wake the host automount (make-or-break PASS; container bind-mount too); Q2 controller drops to uid 1000 and writes (setpriv proof; SETUID/SETGID caps present); Q3 Route A ACCEPTED-with-caveats (all_squash to ANY server uid = full rw+hardlink for the guest app; guest sees nobody; chown EPERM-clean, chmod persists; June's anonuid=1000 WRONG case pinned = anonuid without all_squash is a no-op for non-root); Q4 error taxonomy captured (NFSv4 makes no-export vs not-permitted INDISTINGUISHABLE; systemd 90 s start-timeout is the real ceiling; retry=0 → production unit: dead-NAS access 91 s→3.8 s; verify must NOT ride the 15 s agentapi timeout → own path/status-poll, ~120 s budget); Q5 plain-user SMB PASS incl. hardlink. Teardown verified to pre-counts. Production TASK is the operator side's next move.

• 2026-07-11 — PBS DR tier SLICE 2 SHIPPED (agent v0.80.0): the apply-bridge — demo ADOPTION-PROVEN live + PUBLISHED; the supervised Peti phase PENDING. Full chain fired on the demo: hub enable (storage-id felhom-offsite) → tenantsync provisioned ns demo-felhom on ep0 → agent ADOPTED the manual entry (tenancy entry-owned, "entry wins"; staged secret asserted UNCONSUMED; DR green; escrow.pbs_storage_id seeded). Laws red-proof-verified: set-only (K destruction guard), secret-on-stdin, verify-pin-before-consume, loud consumed-failed. Part 0 evidence: write path needs the §4b dual-grant exactly (403 Datastore.AllocateSpace → re-assert → token vzdump OK); demo grants restored. Live finding fixed: escrow seed = in-place write (/etc/felhom-agent dir is root-owned; rename impossible for the non-root agent). 0.80.0 published (f2ba62ca…f06d2, == deployed bytes). NEXT = RUNBOOK-peti-pbsdr-2026-07-11.md (Peti box prep: sudoers+wrapper+wg_tunnel enable — no tunnel/OOB on his box today; Viktor opsign 0.80.0; hub enable → FRESH path, K born, degradation clears; ceremony → auto-confirm → first offsite run = the arc closes). Detail: felhom-agent/REPORT.md + CHANGELOG v0.80.0.

• 2026-07-10 — PBS DR tier SLICE 1 SHIPPED+LIVE (hub v0.44.0): ep0 tenantsync surface + hub provisioning. felhom-tenantsync v1.0.0 installed on ep0 (second forced-command key, peersync untouched — hashes asserted; smoke caught + fixed the client {"data":[…]} json wrap; orphan root@pam!spike cleaned). Hub: internal/tenantsync (pinned-SSH twin of wgsync; errors never embed stdout), config-form "PBS DR tier (ep0)" section → WG-peer check (fail-closed) → provision → host_pbs_secrets consume-once (agent fetches POST /api/v1/hosts/{id}/pbs/consume-token — note: /hosts/, not the task's /host/) → pbs_dr descriptor into HOST desired_json + gen bump; idempotent re-save; Re-issue button (F4 twin). Red-proofs: consume-once / fail-closed / idempotency all mutation-proven. NOT live-provisioned — slice 2 (agent bridge) + supervised Peti enable pending. Runbook §10 addendum in offsite-endpoint.md. Commits ce6a566 + 32ca1f6.

• 2026-07-10 — SPIKE: PBS DR tier auto-provisioning SLICE 0 — all three mechanisms SETTLED. Record: documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md. (1) PVE token storage-create impossible (create/modify/delete all check /storage root) → sudoers vector proven (pinned-id alias, cycle green as the agent user, unpinned denied; secret must ride a stdin-wrapper, never sudo argv). (2) ep0 tenancy op-set + one-time-secret-over-SSH-stdout proven live on ep0 (🛑 Viktor-approved; read/write cross-tenant 403s); current peersync channel can't carry it → slice 1 ships a second forced-command surface (felhom-tenantsync). (3) pvesm add pbs --encryption-key autogen births K at PBSEncKeyPath + vzdump lands ciphertext (doc-06 property) + escrow-create finds K. ⚠ pvesm remove DELETES K → re-apply must be set-only. All spike artifacts torn down; flags: orphan root@pam!spike on ep0 scratch; demo's §4b step-3 storage grants absent. Epic slice map in the record §5.

• 2026-07-10 — new-box restore round-trip VALIDATED + old-box archive RETIRED. Record: documentation/audits/VALIDATION-newbox-restore-2026-07-10.md. Both toggled apps restored from the production pool box via the real endpoint — 9/9 sha256-identical (immich 44MB db-dump exact, d78466fcbf…4526b, agreeing with the campaign's independent round). ~10s/app. Viktor's decision: RETIRE the old archive — he deletes u629193-sub1 in the old project's console (the PBS-storage-1 box itself stays for dev). C2 fixed same day as controller v0.110.0 (crash-lock self-heal, live-proven). Offsite arc: build-complete + restore-proven both directions; remaining = the Peti onboarding runbook.

• 2026-07-10 — offsite arc overnight soak (unattended) — PASS + 1 HIGH finding. Record: documentation/audits/CAMPAIGN-offsite-overnight-2026-07-10.md. F4 re-issue proved LIVE (reset invalidates the old password); restore round-trip 9/9 byte-identical new-box↔USB (immich db-dump exact → old-box archive u629193* safe to retire, Viktor's decision); single-flight + crash-restart (zero consume, escrow untouched) PASS. FINDING C2 (HIGH, OPEN): a controller crash MID-PRUNE leaves a stale restic exclusive lock the controller can't self-heal → offbox runs fail until manual restic unlock (fix candidate: ensureOffboxRepo auto-unlocks a self-owned stale lock). Track B (live quota-crossing) + hub-orchestrated Track A blocked — the hub operator config surface is session/CSRF-gated (unattended can't reach it) → Peti runbook. Demo restored to start-state. One expected backup_failed email (the crash test).

• 2026-07-10 — SLICE 4 soft-quota + offsite health SHIPPED+LIVE (hub v0.41.0 + controller v0.109.1) — the offsite arc is BUILD-COMPLETE. Controller: quota_gb rides the descriptor into the target (hash includes it → quota changes re-apply via key-auth-first, no consume); repo_size_bytes from restic stats; pre-run gate (≥100% refuses NEW backups Hungarian-loudly, prune still runs — red-proofed; restore never gated; ≥80% warns); /backups usage bar. Hub: monitor.OffsiteChecker (fill 90/95 of quota + offsite_stale >48h for enabled+escrowed only — red-proofed; nil-safe on old reports) + manual-only Freeze/Unfreeze lever (readonly on the exactly-1 labelled sub-account, SSH preserved). Live: key-auth-first proven against the real box (the deploy's own re-apply); that re-apply exposed

  • fixed v0.109.1 (re-apply must preserve EscrowState + runtime status — it demoted the escrowed demo); demo repaired (manual confirm + run: escrowed, 45 MB / 50 GB, checker healthy 0 events, usage bar renders). Freeze + 100%-refusal NOT live-exercised (STOP; red-proofs cover). NEXT: Peti onboarding runbook, NEW-box restore round-trip → old-box retirement.

• 2026-07-09 — SLICE 3 hub-verified escrow auto-confirm SHIPPED+LIVE (hub v0.40.0 + controller v0.108.0 + agent v0.79.0). Replaces the operator-trust confirm with a verified fact: ceremony hashes the sealed repo password (agent, restic_pw_sha256,omitempty on the escrow upload — contract test moved in lockstep) → hub stores it (additive host_escrow migration, NULL on legacy rows) + serves escrow:{identity_blob_present,restic_pw_sha256,created_at} in the report ACK → controller (report.EscrowAutoConfirmer) flips pending→escrowed ONLY on hash match with its local repo password. Red-proofed core: blob-presence alone never confirms (stale blob = false custody claim); mismatch warns once per hash naming the ceremony; never un-confirms; flip wipes the agent-staged secret. Hashers pinned by the same cross-repo vector. Manual confirm = deprecated fallback (legacy blobs — demo included, hash-NULL verified live; the escrowed demo produced zero confirm activity = Scenario E live). Full happy chain proves out at the next real enable (Peti). NEXT: SLICE 4 soft-quota, Peti runbook.

• 2026-07-09 — offsite hardening bundle SHIPPED+LIVE (hub v0.39.0 + controller v0.107.0 + agent v0.78.0). The e2e's sharp edges closed before slices 3–4: F4 "Re-issue offsite credentials" (explicit operator recovery for consumed-password dead-ends — exactly-1 label scope, fresh one-time secret, ConfigVersion bump; hetznerapi.ResetBoxPassword added for dedicated); F2 host-key scan retry (~60s ladder — first save survives fresh-subaccount DNS lag); F5 submit-disable + in-flight notice; key-auth-first bridge (working installed key → re-pin+reconfigure WITHOUT consuming; fingerprint verify never bypassed; kills the stale-descriptor consume-404 loop); staged-secret wipe (agent DELETE /escrow/stage-secret; controller calls it on every flip to escrowed; the e2e's lingering staged file wiped live through the new endpoint, idempotency proven). All red-proofed. F4 reset NOT exercised live (demo credential untouchable — throwaway customer under supervision when needed). NEXT: SLICE 3 auto-confirm, SLICE 4 soft-quota, Peti runbook.

• 2026-07-09 — offsite provisioning VALIDATED LIVE end-to-end (hub v0.38.1 + controller v0.106.1) — PASS. Full loop through the real pipeline on demo-felhom against the new dedicated-project pool box (611714 / u629488, project 15282031): UI save → sub-account 269008 provisioned → descriptor v6 + one-time password → controller config-refresh (≤15 min) → apply-bridge (verify-pin → consume-once → pinned ssh-copy-id → EscrowState=pending) → run-gate refusal proven → escrow confirm (Option A: the repo password was already escrowed — WriteOffboxSecrets preserves it) → first run: restic repo + 2 snapshots on the new box, key-only. Two live bugs found+fixed mid-run: F1 hub v0.38.1 (applyOffsite on context.WithoutCancel — a re-click canceled the provision between create and password-store, stranding the sub-account) and F3 controller v0.106.1 (ssh-copy-id -s needs ~/.ssh to exist in the container — died AFTER consuming the password). OPEN: F2 (fresh sub-account DNS lags → first-save scan 502s; retry-with-backoff), F4 (hub has no password-reset path — recovery today is delete-subaccount + re-save), F5 (no save spinner → re-click bait). Phase-0 wiring: manifests/hub.yaml now injects HETZNER_TOKEN/HETZNER_POOL_BOX_ID from Secret/storagebox; scoped-token check = list boxes with it (must see ONLY 611714). Old-box offsite kept as frozen archive until a NEW-box restore round-trip. Record: documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md. NEXT: SLICE 3 (auto-confirm), SLICE 4 (soft-quota), F2/F4/F5, Peti onboarding runbook.

• 2026-07-09 — offsite provisioning SLICE 2 (hub v0.38.0 + controller v0.106.0). The controller apply-bridge: on startup it reconciles the hub-served offsite: descriptor into a key-only offbox target (controller/internal/offsiteapply.Bridge) — verify-pin the box host key against the hub-captured host_fingerprint (no blind TOFU) → consume the one-time password (single-use) → sshpass ssh-copy-id -s -f install → configure offbox → EscrowState="pending" → persist a descriptor-hash marker. Idempotent + fail-safe; both red-proofs green. Hub v0.38.0 adds Descriptor.HostFingerprint captured via an x/crypto/ssh keyscan (fail-closed). NOT yet live-applied — supervised end-to-end (hub provisions on the new pool box → controller apply) is the next runbook, gated on the hub's new scoped HETZNER_TOKEN. NEXT: SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota).

• 2026-07-09 — hub offsite provisioning SLICE 1 (hub v0.37.0). The hub can now provision the offsite tier on operator enable: internal/hetznerapi (typed client, base api.hetzner.com/v1, CloudAPI interface + exported Fake, WaitAction), internal/offsite (Provisioner.ProvisionOffsite — idempotent by label felhom-customer=<id>, shared sub-account / dedicated box, generates the transient password, builds the NON-SECRET descriptor, fail-closed), one_time_secrets store (single-use Save/ConsumeOneTimeSecret), POST /offsite/consume-password/{id} (customer-API-key auth, once), config-form Offsite section → applyOffsite (502 + no-save on error) → descriptor in ConfigJSON → ConfigVersion bump. Token from HETZNER_TOKEN (out-of-band); password NEVER in ConfigJSON/logs. Unit-tested vs a faked Cloud API (+ a fail-closed red-proof). NOT yet live-provisioned — awaiting the dedicated-project scoped token (the current token can delete ep0). NEXT: SLICE 2 (controller apply-bridge consuming the descriptor + one-time password), SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota). Deployed via GitOps.

• 2026-07-09 — Hetzner API provisioning spike (offsite tier). Both offsite models are API-provisionable (FIT). #1 correction: the Storage Box API is https://api.hetzner.com/v1, NOT api.hetzner.cloud/v1 (Cloud API 404s for storage boxes); same Bearer token. Proven live on spike-* (torn down): sub-account create/reset/readonly/delete + a billable bx11 box create/rescale(change_type)/delete. Create takes a password (no ssh-key param) → transient-password→ssh-copy-id -s -f→restic-reach→reset_subaccount_password (transient pw invalidated, key survives) = key-only steady state. RepoPath /home/<repo> (chroot). Quota: dedicated=box type/change_type (hard); shared=no native quota → Felhom soft-quota + update_access_settings{readonly} freeze. Idempotency = label_selector (box names not unique). Rate 3600/h. SECURITY: the token is project-wide unscopeable Read+Write — it can also touch ep0 (the prod offsite server) + 3 SSH keys → move customer boxes to a DEDICATED Hetzner project before the hub holds a write token. Record: documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md. NEXT: the hub-provisioning impl.

• 2026-07-09 — fork-4 custody VALIDATED end-to-end live (supervised). The full custody cycle proven on the demo (agent v0.77.0 + controller v0.105.0): enable → controller pushes the repo password to the agent → real escrow ceremony (--selftest=escrow-create, K + restic-pw under a fresh R, uploaded to the hub, staged secret wiped, K byte-unchanged) → atomicity proof: a run skipped before confirm-escrow and proceeded after (2 apps, 2 snapshots) — the gate opens exactly on confirm. DR money shot: the restic password recovered from the production hub blob (host_escrow.identity_blob for demo-felhom-01, via identity-consume with R) is byte-identical to the live password (sha 24c44f49…) and decrypted the real box repo; the real inject-password→restore endpoints reproduced a 45 MB db-dump byte-perfect. Demo offbox left escrowed (un-paused); Viktor holds R out-of-band; no secret in any log; live repo_password unchanged. Record: documentation/audits/VALIDATION-fork4-custody-2026-07-09.md. NEXT: hub-provisioning + soft-quota spec (hub-verified auto-confirm replacing operator-trust; both storage models); per-customer onboarding runbook; the identity-consume OK-message fix (omits the recovered restic field).

• 2026-07-09 — offbox v0.104.0 VALIDATED live against the Storage Box (supervised). Closed the offsite arc: controller :0.104.0@908e4b9 ran the real /backup/offbox/* endpoints against a fresh box sub-account (u629193-sub1, RepoPath /home/felhom-demo-repo). The capstone proof — immich (undeployed, recovery unit only on the USB drive, the exact case v0.103.0 mis-resolved to systemDataPath) → discovery found the USB unit → ok, 1 snapshot; 45 MB db-dump restore sha256-matched. No-silent-success confirmed on the live box: 0/N → error + backup_failed operator alert; partial → ok + Hungarian LastWarning on /backups. Persists across restart. Forks: (1) direct topology + (2) sub-account-per-customer (RepoPath /home/<name>, ssh-copy-id -p 23 -s -f — console SSH-key field still broken, F3) confirmed; (3) PBS/WG tier deferred; (4) custody surface RECORDED, escrow design OPEN — the per-customer restic repo password + SSH key are 0600 in the controller data dir, NOT independently recoverable (ride DR only via the PBS whole-CT) → the load-bearing follow-up before real reliance. KEPT as a standing demo-offsite (€4.06/mo). Record: documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md.

• 2026-07-09 — TASK GL-7 SHIPPED: customer-page passphrase hardening + install-command generator (hub v0.36.0, 844fbfa7; deploy 02c748eb; LIVE, ArgoCD Synced/Healthy) — go-live G7/G9/G10/G11. Two coupled, security-first changes to the operator customer page (customer_unified.html + configs.go). (1) Passphrase hardening: the per-customer retrieval passphrase was cleartext twice (visible #retrieval-pw + baked into the Option-3 debug curl X-Retrieval-Password: header); now masked-by-default with reveal (toggleSecret) + copy (copySecret), value in data-secret (existing model), and the Option-3 command carries a <YOUR-RETRIEVAL-PASSWORD> placeholder — no secret in any copyable command. (Zero-secret-in-DOM fetch = noted future follow-up, NOT scoped.) (2) Install-command generator: the three hard-coded <code> blocks became a client-side builder (vanilla JS, no framework/CDN/network) emitting ONLY real host-install v1.12.0 flags, download-then- run, byo-requires-caps enforced client-side, CustomerID prefilled from the server, JS-off static fallback; the seven dangerous/operator-only flags (--force/--rotate-recovery/--enable-oob/ --remove-golden/--uninstall/--adopt-pool/--rescope-acl) are never offered as controls. Tests TestTemplates_PassphraseHardened + TestTemplates_InstallGenerator (green; passphrase test red-proofs). Tester agreement written (documentation/pilot/PETI-tester-agreement.md): BYO trust boundary, honest limitations (no offsite backup, F2 removable-drive, F7 pool-reassert, F6 :53-free), exit rights, Peti box facts (80c/128 GB → caps 12/32768, sajatfelhom.hu re-point), byo install command, onboarding sequence. GO-LIVE-PACKAGE: GL-7 ✔; G9/G10/G11 → done. One open pilot question (owned by the agreement §6): Peti's local backup TARGET — does his box have a second disk/pool? → sets --acl-storages at onboarding. Deploy gotcha: first build was stale (local commits not pushed before build.sh pulls origin) — pushed then rebuilt. NEXT: Peti day-0 onboarding (operator-run).

• 2026-07-08 — TASK GL-8 SHIPPED: BYO coexistence hardening (host-install v1.12.0, 18a556a5) — the GL-6 findings F3/F6/F1. F3: the populated-host leaf guard fires only when a felhom_guests guest exists (not pct list) → a fresh byo install on a host running only the OWNER's guests needs no --allow-new-leaf; the pin protection for real Felhom guests is unchanged. F6: byo preflight REFUSES a foreign process on :53 with remediation and mutates NOTHING (Viktor's ruling: refuse+ instruct only — the dnsmasq analog of the break-glass no-touch rule; appliance untouched). F1: uninstall purges the config .bak* siblings (a GL-6 residue held a live hub api_key). F6 live-proven on felhom-pve (byo preflight refuses at :53 — the exact GL-6 scenario, read-only die). Harness 36/36 (+GL8-F3/F6/F1 static, GL8-F1b behavioural, GL8-F6 live) + red-proofs RP-F3/F6/F1; GL-2/GL-4 regression green; shellcheck clean. Shipped BEFORE GL-7 so Peti's command needs no workaround. F2/F7 remain open (non-blocking). NEXT: GL-7.

• 2026-07-08 — TASK GL-5 SHIPPED: DR bring-up bind overrides (agent v0.75.0, LIVE + scratch-DR validated) — go-live G8; docs-only here (GO-LIVE-PACKAGE G8/G6/G1 flips + decision log; the code + full report live in felhom-agent b3446213). Guest-loss DR restore WORKS now: the two structural binds (mp8/mp9) are overridden with throwaways at restore and swapped for the REAL binds pre-start (root pct via the new Engine HostRunner seam), displaced unusedN deleted. TWO live-discovered PVE rules beyond the spike: explicit-params restore needs an explicit rootfs AND silently DROPS unlisted mountpoints — the full param set now derives from the archive's embedded config (ExtractArchiveConfig, 200 under the scoped token). Live proof: scratch 9310 from a real 9201 archive — mp0 200G + mp1 50G restored WITH content (7m23s), real binds, zero residue, clean teardown; the same op failed outright on v0.74.0. OPERATOR: publish agent 0.75.0 + bump the manifest to it (vouching 0.74.0 ships broken DR). Follow-ups: restore-test boot-verifies without mp0/mp1 (same PVE rule, reverse direction — candidate reuse of drRestoreOverrides); DR-selftest KeepMAC duplicates a live source's MAC briefly; no host-loss DR runbook exists yet (GL-6/S5 writes it).

• 2026-07-08 — TASK GL-4 SHIPPED: uninstaller gap-closure + operator-key day-0 fold (host-install v1.11.0) — go-live G6 + the G1 key-pin follow-up; awaiting GL-6 for the real (non-dry) teardown + armed-pin install. Uninstall now removes the self-update artifacts (4b4: guarded wrapper, .prev/.new.* slots, rollback unit, limits drop-in — derived from configs/felhom-selfupdate-guarded), unmounts enrolled drives under /mnt/felhom-drives (plain umount ONLY, busy = warn+guidance, root-bind guarded), and ends with a KEPT-vs-WIPED statement (drives/PBS/hub record/escrow/vaulted recovery credential live on; guest-only mode prints the vmid's bind-store drives). Key-pin: OPERATOR_KEY_* constants (EMPTY until the operator ceremony)

  • --operator-pubkey-file (validated at argv, comment=key_id required) → authz.signers written at step 6 per the agent SignerKey schema; no-keys-resolved KEEPS preserved signers (never un-pin a manually-pinned box); verify reports armed/dormant (dormant = WARN). Harness 28/28 on felhom-pve incl. the NEW GL4 H-U full-uninstall DRY transcript vs live 9201; red-proofs RP-1..3 green. GO-LIVE-PACKAGE.md was ABSENT AGAIN (spec said the operator attaches it; not present in the repo or workspace) — G6/G1 status recorded here, doc still pending its operator commit. Key CEREMONY (real keypairs + felhom-pve pin) = operator step.

• 2026-07-07 — RUNBOOK GL-1 EXECUTED: agent 0.74.0 + golden 0.103.0 PUBLISHED — go-live G1 (partial): the published-artifact chain is current. Agent = the LIVE felhom-pve bytes (sha 1ec3f588…76af05, provenance preserved, publish-agent.sh round-trip verified); golden baked clean-room in the drill VM (build-golden.sh v2.0.0 @ ced60ddf, controller 0.103.0 baked, all integrity gates green, sha 8481e8a1…6026e, 583MB) and published HTTP 201. Both artifacts fetch-verified ANONYMOUSLY (Gate 3c finding: generic packages are world-readable → G3 git.token read-only rotation CANNOT break fetches; but the INSTALLER dies on an empty token — script-side requirement, not Gitea's). Drill VM restored to virgin; evidence /mnt/5_hdd/felhom.eu/drill/bake-0.103.0.log on 180 + documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md. OPERATOR NEXT: bump the hub Day-0 manifest to agent 0.74.0 / golden 0.103.0 (the four values are in the execution record) — until then fresh installs land 0.63.0/0.98.3. Also recorded: operator-key pin follow-up (fold into GL-4); GO-LIVE-PACKAGE.md still not in the repo (G1 status recorded here instead).

• 2026-07-07 — TASK GL-2 SHIPPED: --mode appliance|byo install profile (host-install v1.10.0 + harness) — go-live items G2/G4/G5 implemented, awaiting the supervised GL-6 drill for live provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old --mode provision value retired with a specific die, dr stub kept); break-glass gated at its CALL SITE (byo never touches root@pam); byo requires --cores+--memory, refuses --enable-oob/--rotate-recovery/non-9.x-PVE/missing --acl-storages; disclosure + typed-hostname ack; the config write's lan_resolver default flips to OFF in byo (the spec assumed default-off — it was default-ON; appliance unchanged) + post-write asserts refuse lan_resolver/wg_tunnel/oob=true; pool+ACL verify asserts in BOTH modes (R2 lesson); --preflight-only (no state, PASS/FAIL verdict, proven live on felhom-pve). NEW scripts/hostinstall-mode-harness.sh (static C1–C4 + invariants everywhere; PVE tier: C5 + A/B dry transcripts) — 16/16 PASS on felhom-pve incl. C5 live; red-proofs RP-1..3 run→fail→revert. FINDING: demo-felhom's controller.yaml has EMPTY git.username/git.token → any step-5 (re)install for that customer dies until the operator sets real Gitea read creds (relates to the standing scope-down+rotate follow-up). documentation/pilot/GO-LIVE-PACKAGE.md is NOT in the repo and no operator copy was provided — the G2/G4/G5 status flip is recorded here instead; add the doc when the operator supplies it. Next: GL-6 supervised drill (appliance + byo end-to-end, uninstall/re-provision, C7, the ack prompt interactively).

• 2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent v0.72.0) — merged E1+H1. Hub: operator OOB peer (store/wg_operator.go, PUT/GET /admin/wg/operator-peer global key) + oob_peer_ip/oob_operator_ssh_key in the desired-state wireguard block + monitor/host_oob (oob_degraded/recovered). Installer: --enable-oob/install_oob (static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key.

• 2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent v0.71.0) — prereq for felhom-sshd (H1). Hub: store.host_recovery vault (per-host root@pam console password; PUT /hosts/{id}/recovery-credential self-scoped + GET /admin/hosts/{id}/ recovery-credential global-only, secret never logged) + host_mgmtplane monitor raising mgmt_plane_healed. Installer: step_break_glass (generate+set+vault root@pam, --rotate-recovery)

  • install_mgmt_watchdog (the 3 host artifacts, RuntimeDirectory-guarded). Live-validated: agent-independent /run/sshd auto-heal in 30.0s; warning fired end-to-end; break-glass vault→retrieve→PVE-ticket(200). felhom-pve root@pam is now the vaulted value.

• 2026-07-04 — S3 SHIPPED: agent WG tunnel (felhom-agent v0.64.0; docs-only here) — the doc-06 §3.3 handshake is now END-TO-END LIVE on demo: agent keygen → POST /hosts/{id}/wg → desired-state wireguard block → agent-managed wg-quick@wg-felhom → PBS page over 10.77.0.1:8007 in 3 s. wg_tunnel.enabled defaults FALSE agent-side (rollouts to other boxes are no-ops until the production endpoint exists). Revocation semantics completed (§3.5): removed peer → agent disables + keeps its marker + NEVER re-registers; operator re-adds via the pubkey the heartbeat report now carries. Escrow: IdentityBundle.WGPrivateKey auto-injected on new escrow-create runs (pre-S3 blobs can't be retrofitted — honest fallback = fresh-key re-register, /32 kept). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix, deferred non-blocking); S4 = PBS over the tunnel; hub-side S6 = tunnel-health alerting on the new report stanza. Detail: felhom-agent CHANGELOG v0.64.0 + REPORT.

• 2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0) — doc-06 row S2. DECISIONS: the WG assignment is hub-owned state, merged into served desired-state at READ time — stored desired_json stays a pure operator blob; admin PUT with a top-level wireguard key → 400 (no copy-paste clobber); re-key-in-place keeps the /32 (stable tunnel addressing across rotation/DR); one bound peer per host (partial unique index); generation bumps ONLY on real change (register/re-key/adopt/bound-delete — idempotent paths move nothing). POST /hosts/{id}/wg self-scoped per-host key; NEW golden desired-state-wireguard.golden.json = the S3 cross-repo contract (byte-identical agent copy rule). peersync v1.0.1 (strip out of process substitution — the S1 exit-swallow finding; sandbox red-proofed) redeployed to the dev endpoint. Live-proven on the REAL demo-felhom-01 record via the real per-host key: gen 2→3, served block = golden shape, endpoint wg show carried the peer, v0.63.0 agent tolerated the unknown key (fetch clean, guests untouched), then removed → gen 4, all baselines restored. /offsite page unit-proven (live view is operator-password-gated — eyeball after next login). Next: S3 (agent keygen + registration + wg-felhom service + escrow join + CGNAT smoke — risky/supervised).

• 2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1) — the doc-06 roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: hub pushes over SSH to a forced-command felhom-peersync script (pull rejected: weakens immediate revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev felhom-hetzner box, provisioned by the NEW runbook documentation/runbooks/offsite-endpoint.md (WG on 443/udp, subnet 10.77.0.0/24, PBS datastore felhom-offsite at /mnt/pbs-datastore — a 100 GiB Hetzner Cloud Volume attached 2026-07-27 06:29:40 UTC; the pre-migration copy at /srv/pbs-felhom is 13 G of dead weight on / awaiting R-91's go-ahead — nftables drop-all + wg0-only 8007, ep0.felhom.eu grey-cloud). Hub: wg_endpoints/wg_peers + /32 allocator + /admin/wg/* (global-key, pubkey-in-body) + internal/wgsync (FixedHostKey pin

  • pinned-type HostKeyAlgorithms — live-caught bug: multi-hostkey sshd presented ECDSA vs the ed25519 pin; fixed v0.32.1 + regression test) + full-list declarative reconciler (5-min drift repair). Live: add/remove→wg show, malformed→state-untouched, reboot→persist+converge, client tunnel via ep0:443→PBS login page, public 8007 closed. INCIDENT handled: first WG server key leaked via wg show dump into the session log → rotated immediately (warning now in the runbook). OPERATOR follow-up: fix the AAAA record (currently …7aa1::, must be …7aa1::1). Next: S2 (hub schema host-join + wireguard block on WireDesiredState + box-facing registration + UI).

• 2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06) — documentation/architecture/06-offsite-connectivity.md: records the six settled decisions (plain WG / host-side agent-managed wg-felhom as the agent-managed-unit PILOT / one shared hub-driven endpoint VM running WG+PBS, NO agent on it / hub = source of truth, wireguard block rides WireDesiredState on the existing DesiredGeneration channel / one datastore + per-customer namespaces / PBS ON the VM, relay-through-DooPlex REJECTED as non-scaling), the Day-0 join handshake, robustness set (WG-roaming-not-DynDNS, DNS re-resolve watchdog, MTU 1420, per-/32 isolation, tunnel-health via storage-target model), trust conformance (outbound only; operator-inbound + game-ingress out of scope), the honest ledger (CGNAT still unmeasured — mobile-hotspot smoke closure; peer-sync push-vs-pull = slice-1 design point), and the S1–S6 slice roadmap (MVP = S1 endpoint + S2 hub schema/desired-state + S3 agent WG service, then S4 PBS-over-tunnel; S5 merges with DR-completeness). day0-install backlog line now cites spike + design doc. Docs-only; slices are separate future tasks.

• 2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung) — plain WireGuard, host-side, box-dials-out to a small public endpoint; no Headscale/DERP needed for backup (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway Hetzner: 11.4-min idle hold, PBS API through the tunnel with the exact-cert pin intact (positive + negative), real 2 GiB worst-case backup at 4.26 MiB/s (= the home uplink; tunnel overhead ~5%), UDP 51820 and 443 both pass. P0 honesty: the operator's line is plain-NAT (fixed public IP, One Hungary), NOT CGNAT, and has zero IPv6 — CGNAT confirmation deferred to Peti's VM 110 (second ISP). P7 surprise: kernel WG works inside the unprivileged guest (no TUN needed) — host placement stands on architecture (backups originate host-side; cloudflared precedent), not infeasibility. Evidence + architecture recommendation (endpoint VM shapes, key custody via IdentityBundle escrow, per-peer /32 isolation): documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md. This unblocks the DR-completeness work: next task = production connectivity spec (endpoint VM, agent-managed WG service, hub peer registry, escrow join). Throwaway Hetzner box parked powered-off (secrets shredded); demo-felhom verified residue-free.

• 2026-07-03 — 4th skill: felhom-app-catalog (catalog authoring workflow; points at app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by install_skills.py, 4/4 found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected healthchecks, probe-container naming rule discovered + REUSE row added, UI-applied, controller probe healthy). App-count drift recorded: website says "45+", per-category spans sum 49, real catalog 53 — website fix is a separate task.

• 2026-07-03 — GOLDEN 0.98.3 LIVE + VOUCHED; guide D.1b RETIRED (drill B1+B5 FIXED) — build-golden.sh v2.0.0 (mandatory controller tag + baked bootstrap .path unit); golden baked with controller 0.98.3, clean-room validated (drill VM) through all four scenarios incl. the published-artifact --force-gitea-golden path, published + operator-vouched (Day-0 manifest now: agent 0.63.0 + golden 0.98.3). Fresh installs land CURRENT and self-manage — D.1b is a one-line check; the prior "vouch agent 0.63.0" operator follow-up is DONE. Evidence: documentation/audits/DRILL-golden-098-2026-07-03.md. NEW operator follow-ups: (1) SECURITY — the customer-config git.token is a Gitea admin token with package-WRITE (proven during the bake); scope down + rotate. (2) optionally raise the global controller floor to 0.98.3.

• 2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published) — token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir. Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. OPERATOR: vouch agent 0.63.0 in the Day-0 manifest UI (until then fresh installs fetch 0.62.0).

• 2026-07-03 — GO-LIVE PACKAGE SHIPPED (host-install v1.9.1 + day0-install runbook + clean-room drill) — documentation/runbooks/day0-install.md is the operator Day-0 guide, every command executed verbatim on a throwaway nested PVE 9.2.2 (build server): install → verify → app deploy → uninstall → zero-residue diff → reinstall. v1.9.1 = residue-free uninstall (agent config w/ hub api_key, shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq snippets) + post-provision guest reboot (without it the controller never deploys on slower hardware) + header sync. Evidence + go/no-go for Peti (GO, 3 install-day items): documentation/audits/DRILL-day0-cleanroom-2026-07-03.md. Key OQ-2 finding: golden bakes a pre-floor controller 0.85.1 → fresh installs need the one-time D.1b update (agent restart → update button → 0.98.3); follow-ups recorded: rebuild golden ≥0.86.0 + re-vouch (operator), agent mkdir -p for the guesthook snippet dir, fresh-install local-API 401 (pre-existing), hub has no host-delete path. Drill VM parked virgin-snapshotted on 180 (/mnt/5_hdd/felhom.eu/drill/ — moved off the SSD 2026-07-18).

• 2026-07-03 — A1 CLOSED (host-install v1.9.0 + agent v0.62.0) — Pool.Audit added to FelhomAgentGuest; the agent's stale-lock reaper now intersects its scan with GET /pools/felhom membership (fail-safe skip on read failure). Spike: documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md. Deploy order on any box: rescope ACL first, then the agent. STILL PENDING (operator): rotate + de-git the 3 committed secrets in manifests/felhom.secret.yaml (spike appendix; REUSE §2 errata added).

• 2026-07-03 — hub v0.31.0 LIVE (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy; live probe: critical event stored + operator email delivered).

• 2026-07-03 — Felhom skills exist: skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}, installed via scripts/install_skills.py — junction mode (repo edits live immediately; no re-install needed after editing). Fresh-session discovery verified.

• 2026-07-03 — CLAUDE.md refreshed (all four repos): no version-pinned state in CLAUDE.md — current state lives here / CHANGELOG / REUSE.md.

• 2026-07-03 — REUSE.md exists at the repo root (hub helpers / website conventions / gate scripts / manifest patterns, code-verified); maintenance rule active: update it in the same commit that changes a shared helper. scripts/reuse_refs_check.py machine-checks its citations across all four felhom repos.