felhom.eu/CONTEXT.md

# CONTEXT — felhom.eu working state

> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.

> **`documentation/backlog/OPEN-ITEMS.md` is the single source of truth for open work** (2026-07-27).
> Read it first; `ROADMAP.md` holds the full history and reasoning behind each item. **`REPORT.md` is
> per-session and overwritten — nothing durable may live only there.**

> **This file is NOT `STATUS.md`, and the two must not be consolidated** (2026-08-02). `CONTEXT.md`
> is technical working state written for Claude Code: rulings with their mechanisms, file paths,
> version numbers, the reasoning a session needs before it touches code. Root `STATUS.md` is written
> for the **operator** — technical, but not in the code and not a reader of the audits — in plain
> language, one screen, no identifiers in the prose. Same subjects, different readers; merging them
> would make one of the two audiences stop reading. `STATUS.md` is also a **view of `OPEN-ITEMS.md`**
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.

## Standing rulings

**S-8 — CI detects; it does not block, and that is structural (2026-08-02, R-168).**
A Gitea Actions runner in `gitea-system` re-runs every repo's gate entry point on every push,
independent of who pushed and of what they typed. It **cannot refuse a push**: every felhom repo
pushes straight to `main` with no pull request, so there is no merge for a status check to stand at.
That is not a gap in the runner — there is no gate in the road. The refusing half is
`.githooks/pre-push` (per-clone, `--no-verify`-able); this half is what notices when that hook was
skipped or was never armed. Making CI blocking needs branch protection plus a PR workflow, which
changes how the operator works and is **their** call → R-169. Do not "fix" this by adding branch
protection.

**S-9 — a detector that tells no one is not finished (2026-08-02, R-168 probe P5).**
Probe P5 measured that a failed run produces **no mail, no notification row and no log line** from
Gitea. So the workflow sends its own alarm on the project's existing Resend path and **prints the
provider's accepted id**, which makes "a message left the machine" an observable rather than an
assumption. The acceptance test for this work was never a green pipeline — it was a red run with a
message in hand (`RESEND-ACCEPTED id=…`). Two traps found while building it, both worth keeping:
the runner image has **no `curl`** (deliberately — python3 and git only, so use `urllib`), and
`api.resend.com` sits behind **Cloudflare, which 403s the default `Python-urllib` User-Agent with
error 1010** — a failure that looks exactly like an auth failure and is not one.

**S-10 — the runner is unprivileged, and the reason is the host (2026-08-02).**
The usual `act_runner` recipe pairs it with a `docker:dind` sidecar and `privileged: true`. Rejected:
DooPlex is **Tier 2** and *is* the recovery chain — Gitea, the hub, the registry, PBS and
k3s + Longhorn all live on it and it cannot be rebuilt from anything else. Host execution mode costs
nothing here because every CI job is one Python command. Consequence to remember: **in host mode the
job sees exactly the runner image's tools**, which is why `python3` had to be baked in (probe P2:
stock `act_runner` carries git but not python3). If a future job genuinely needs Docker, that is a
conversation, not a patch.

**S-11 — CI reproduces the workspace's sibling layout, because two entry points depend on it
(2026-08-02).** `controller_gates.py` and `agent_gates.py` invoke the shared `reuse_refs_check.py`
that lives in the `felhom.eu` clone next door and is deliberately never copied, and both repos'
`REUSE.md` files cite a path that lives in the hub. Their workflows therefore clone `felhom.eu` as a
sibling; without it the gate fails **closed** — correctly, but for the wrong reason. Verified that CI
and the local hook then agree exactly (controller 126 exact / 6 suffix / 1 cross-repo).

**S-6 — the hub renders no host-install version, and the gate pins its absence (2026-08-02, R-94).**
The Setup tab's *"host-install 1.19.0"* label is **deleted, not derived**. Deriving it is not
achievable honestly: the Option-1 command downloads `felhom-host-install.sh` from the website **at
run time**, and the website git-syncs `main` every 30 seconds (R-110) — so the hub cannot know which
version a given box will run, at build time or at render time, and any literal there is a guess
wearing a version number's authority. The real one drifted to 1.19.0-vs-1.22.0 and stayed wrong for
nineteen days. `hostInstallVersion`, `pageData.ScriptVersion` and the rendered label are gone; a NOTE
sits where the const was so it is not helpfully re-added, and `scripts/hostinstall_gates.py` gate 1
**inverted** — it now asserts the hub carries no host-install version literal in any of six code
shapes across every `.go`/`.html` under `hub/`. Corollary that generalises past this row: the
tautological `render_test.go` assertion (`html contains hostInstallVersion`, where the same constant
put it there) **passed at `9.9.9`** — an assertion that compares a value to itself tests the
plumbing, never the claim.

**S-7 — gates run from ONE entry point per repo, and `reuse_refs_check` was fixed rather than the
convention it polices (2026-08-02, R-29).** Two rulings from the same census.

*Where gates run.* Thirteen gate scripts exist across the four repos. Measured 2026-08-02: **every
check a `CLAUDE.md` tells a person to run was passing, and two of the four nobody is told to run were
failing** — one since 14 July. The correlation was exact, so the fix is not more gates but one place
to run them from: `scripts/repo_gates.py`, `felhom-controller/controller/scripts/controller_gates.py`,
`felhom-agent/scripts/agent_gates.py`, `app-catalog-felhom.eu/scripts/catalog_gates.py` — each
mandated in its `CLAUDE.md`, each wired to `.githooks/pre-push` with `--fast`. The canonical shape is
`catalog_gates.py` (R-161), **not** `site_gates.py`, which is a gate (eight assertions in one file)
and not a runner; copying it produces another monolith nobody invokes. **A missing gate script is a
FAILURE with the path printed, never a skip.** The hook's limits are real and are written into the
hook: per-clone (`core.hooksPath` is local config) and `--no-verify`-able on purpose. The
unbypassable half is CI → **R-168**.

*Why the checker moved and the docs did not.* `reuse_refs_check.py` was RED on all four repos with
13 findings, of which a hand audit found **zero** genuine drift — twelve were package shorthand
(`appbackup/userdata.go` → `controller/internal/appbackup/userdata.go`) and one, `wgsync/reconciler.go`,
is cited by the controller and lives in the hub. `REUSE.md` cites by package shorthand and across
repos deliberately; that convention is the useful one. **Rejected, so they are not revisited:**
rewriting all four `REUSE.md` files to full paths (makes the docs worse to serve the tool), and
deleting the checker (drift across four repos is a live risk). The checker now resolves
exact → suffix → ambiguous → sibling repo → FAIL, **prints every non-exact hit and a per-rule tally**
(because "0 failures" alone cannot tell a working checker from a blind one), and lists every
resolution attempted on a failure. It stays in **one** place and is invoked across the workspace —
never copied, which would recreate the drift it detects.

**S-1 — N.5 gains a third leg: architecture docs are same-session coupled (2026-07-26, R-81).**
Any task that changes an **architectural contract** — tiers, targets, cadences, trust boundaries —
updates the owning `documentation/architecture/*.md` in the **same session**, under exactly the same
coupling rule that already binds the capability map and the ROADMAP. Origin: R-80/R-81 found
`07-backup-architecture.md` still describing a backup model that the shipped code does not implement
(single target, single cadence), while being cited as authoritative. A stale architecture doc is
worse than a missing one, because it is trusted.

**S-2 — architecture docs carry an honest status header (2026-07-26, R-81).**
Every `documentation/architecture/*.md` opens with the version it was **verified against** and the
date. A doc more than a few trains behind its subject is marked **STALE** *in that header*, so a
reader meets the warning before the content, not after acting on it. Origin:
`07-backup-architecture.md` — DRAFT since 2026-07-14, verified against controller v0.132.0, now 41
versions stale (live v0.173.0), and cited as authoritative throughout the R-80 diagnostic. Ratifying
or retiring it is → **R-83**.

**S-3 — the recovery model: six decisions, 2026-07-28.** Taken in an architecture discussion and
expressed in the `07-backup-architecture.md` full rewrite (which replaces the 2026-07-14 DRAFT
entirely — that doc was verified against controller v0.132.0, **51 versions stale**, while being
cited as authoritative). They are **decisions, not observations**; the rewrite labels every statement
`[DESIGN]` or `[FACT]` so a reader can tell which is which.

- **D1 — two lanes, different owners.** *Customer, unassisted:* files and app data, through the
  „Visszaállítás" surfaces. *Operator:* LXC guest and host recovery. This is a deliberate product
  decision, not a limitation to apologise for.
- **D2 — encryption follows the boundary, not the tier.** Data on the customer's own drives is
  **plaintext**, deliberately: someone who can take the second drive can take the first, local
  encryption adds a key-loss path that turns a working backup into a brick, and it would break
  FileBrowser and SMB browsing, which is a feature. Data **leaving the premises** is encrypted
  (restic, PBS) — the provider must not be able to read it.
- **D3 — recovery is a chain, not a set of tiers.** Every app-tier restore is conditioned on the
  whole-guest tier: Tier-1 needs the guest's secrets (`restore_unit.go:17-22`, *"Regenerate
  NOTHING"*), and Tier-2/Tier-3 need the app still deployed. **3-2-1 describes copies; it does not
  describe recovery.** This is the reason the rewrite exists.
- **D4 — three-part model.** *Recipe* (scaffolding, hub) · *Escrow* (identity key + restic repo
  password, hub, R-wrapped) · *Bytes* (the tiers). Losing one is a different problem from losing
  another, and the failure→recovery matrix is organised around that.
- **D5 — R's role narrows. CONDITIONAL, and currently BLOCKED.** The target is app secrets travelling
  with the LOCAL recovery unit, so Tier-1/Tier-2 restore work without the guest and without R,
  leaving R needed for offsite recovery and host identity only. **Not adopted:** the premise
  (the backup tree unreachable from every browsing/download surface) was tested and failed — see
  **R-108**. Every other surface verified clean; the exposure is FileBrowser's root-bind of network
  shares combined with the absence of an `IsNetwork()` filter on the deploy and per-app-migrate
  paths.
- **D6 — the trust model, stated honestly.** The operator holds root SSH on every box; *"the
  operator cannot read customer data"* was never the actual security property and no document may
  imply it. What R **does** provide, and must keep providing: **the hub alone is not enough** — a
  compromised hub yields blobs nobody can open, *provided the operator's key is never stored in the
  hub*. That proviso is why escrow custody is an open decision (`07` §11-A).

**S-4 — the hub session password alone now unlocks console root on every managed box (2026-07-31, hub v0.84.0).**
Retrieving a host's vaulted break-glass `root@pam` credential previously required the **global operator
API key**, a secret distinct from the hub login and kept out-of-band. The `Console access` card on the
host page puts that retrieval behind the **hub session** (`POST /hosts/{id}/reveal-recovery-credential`,
CSRF-gated, audited), so the hub login password is now sufficient. **Accepted deliberately**, and the
reasoning is the ruling: the hub is single-operator and geo-fenced to Hungary, and it already stores
these passwords **in plaintext at rest** — so the DB and the login were already jointly sufficient, and
this is a convenience trade rather than a new exposure. It is a real trade, not a free one, and it is
recorded here rather than only in a changelog. Two things follow. **(a)** The global-key path
(`api/handler.go handleAdminGetRecoveryCredential`) stays untouched and must acquire **no** dependency
on the session layer — it is the route for when the hub UI itself is what is broken, and coupling the
two would delete the independence that makes it a fallback. **(b)** The plaintext-at-rest half is now
the named open item **R-133**: every hub DB backup is a fleet-wide console-credential dump, and the fix
shape is envelope encryption under a KEK held outside the DB, the way escrow blobs are already opaque.

Five decisions were deliberately **left open for the operator** and are recorded, unanswered, in
`07` §11: escrow custody · lost-R policy · per-scenario RTO/RPO targets (**none have ever been
stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with
the guest it backs up. Gaps minted the same session: **R-102 … R-108**.

**S-13 — boot recovery finished, and the lesson is about the DIAGNOSIS ORDER (controller v0.190.0,
2026-08-02, R-157 A · R-170 · R-171).**

**The session's most valuable half hour was spent NOT writing code.** A hole was reasoned out of the
v0.189.0 diff — replacing the container-count term with recorded intent should make a
drive-gate-stopped app read as a boot orphan — and the task's own rule was to CONFIRM it on hardware
before writing a fix. **The first attempt to confirm it produced a false negative**, and reporting
that as a disproof would have been wrong: unmounting only the parent bind is healed by the agent
within ~60 s, so the drive gate's startup reconcile restarted the apps **one second before** the
sweep looked. `no boot-orphaned apps` in that log is a race that went the safe way, not a mechanism.
Holding the drive genuinely absent reproduced it immediately. **"It didn't happen this time" is not a
disproof — name the mechanism or run it again.**

**The confirmation also changed the severity, in both directions.** The write hazard did NOT
materialise: compose failed `mkdir …/userdata: permission denied`, because the unbound mountpoint is
host-root-owned and the guest is unprivileged. **That protection is accidental** — no code chose it,
no test pins it, it is one `chown` or one privileged guest away from gone, and its removal would be
invisible until data landed on the wrong disk. Meanwhile the harm that DID occur was real on every
box and was not in the hypothesis: two wasted attempts and a **false dead-app alarm for an app the
drive gate is deliberately holding**. Diagnosing first is what produced both facts.

**The fix was already in the codebase, on another path.** The API's `startGatedByMissingDrive`
refuses a customer's start on an absent drive with a Hungarian message. The sweep bypassed it by
calling `Manager.StartStack` directly. **`StartStack` has no gate of its own** — that is the durable
fact worth carrying: every caller that is not the customer must decide for itself whether the app may
run, and there are now fourteen of them.

**Widening a window makes previously-unreachable overlaps reachable, and that is a design input, not
an afterthought.** The old T+5 s sweep never met a quiesce or an in-flight app-data operation; a
50 s window can. All three holders answer one seam rather than three, because they differ only in
the reason string.

**A test rejected my first constant, and the comment now says so.** `settle + budget + one retry`
must fit inside `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s
against a 90 s grace. The budget is 50 s **because a test said so**, and the code records that rather
than presenting the number as taste. Widening the grace to fit was rejected outright: it hides a late
recovery instead of reporting one.

**AND THE FIX HAD ITS OWN DEFECT, FOUND BY LIVE VALIDATION AND NOT BY REVIEW.** The window sampled
`GetStacks()` — the Manager's in-memory map, refreshed by the scheduler every 10 s — every 5 s. Two
identical samples could therefore mean *the cache did not update*, not that the fleet had settled. It
surfaced as a container removed ~5 s before the window closed still being in the sampled fleet, with
the sweep logging `no boot-orphaned apps` for an app that had none. **Generalise it: a settle
detector is only as good as the freshness of what it samples — if the source is cached, refresh it or
you are watching the cache settle, not the system.**

**Live: 6/6 hard resets on the shipped build** (every app back; a customer-stopped app down in all
six), window settle times 10/40/10/10/15/15 s — routinely 2–8× the old fixed 5 s. The sharpest
evidence is a same-app before/after on one box: missed at 18:08:35, recovered at 18:18:50.

**S-12 — D-b is BUILT (controller v0.189.0, 2026-08-02, R-166).** The desired/in-flight/observed
split now exists; the S-1 contract lives in `architecture/02-controller-module-map.md` §0a.

**Both facts D-b said to establish first were established at source, and both changed the shape.**
(a) *Does the crash-safe journal in the backup code already cover the in-flight case?* The pattern
DID already exist — twice (`quiesce` marker+`Recover`, `migrate` journal+`RecoverMigration`) — and
covered **none** of the app-data path: `DumpAppVolumesSafe` stopped and restarted an app with no
marker, no journal and **not even a `defer`**. So the answer was neither "it exists, wire it" nor
"build it": copy the proven shape into its own file. It was the fifth time the question was worth
asking and the first time the answer was "the pattern, not the coverage". (b) *Is the SQLite store
reachable?* Reachable and **deliberately not used** — `metrics.db` is optional by design (the
controller runs with it absent), and operational state must not live in a store built to be dropped.

**The ruling that carried the design: absent means UNKNOWN, never "running".** Every `app.yaml` on
every box predates the field, so absent is what the whole fleet reads on upgrade; reading it as
running would have started every deliberately-stopped app on the first boot after the upgrade —
fleet-wide, silently. Where intent is unknown the box keeps the OLD inference rather than inventing
an answer. That is also why the backfill is **running-only**: "zero containers ⇒ stopped" is the
defect itself, so an ambiguous app stays ambiguous until a customer presses a button.

**The other load-bearing ruling: `StartStack`/`StopStack` are NOT writers of intent.** A census found
14 callers, of which exactly 2 are the customer. Recording intent in the primitive would make a
nightly backup indistinguishable from the customer pressing Stop — the confusion being removed.

**Found on the way, and it would have silently eaten the feature: `SaveAppConfig` rebuilt `AppConfig`
field-by-field.** That is the R-100 shape, which v0.181.0 shipped two live instances of. The literal
named five fields, so the sixth would have been dropped on every save across nine call sites — a
customer's Stop erased by the next unrelated `app.yaml` write. Copy-and-overlay is safe by
construction; the failure mode is generic, so **treat any field-by-field struct rebuild in a save
path as a defect on sight.** Measured and documented rather than assumed: `app.yaml` does NOT
round-trip YAML keys the struct does not model.

**Closes R-157 mechanism B; mechanism A (the sweep observes ~5 s after start and never re-checks) is
untouched and is now the whole of R-157** — and B's fix makes A cost more, since the sweep now has
more it could legitimately recover. **New: R-170** — `shouldRecreateOnBoot`
(`internal/web/intermediary.go:131`) still infers a Stop from `hasContainers`, i.e. the same defect
one gate over for drive-backed apps. Left deliberately: the task scoped `bootrecon`, and two boot
behaviour changes under one live validation is one too many.

**Live on 9201, three flows, each with a positive observable.** The interrupted-operation half is
**IMPLEMENTED, not PROVEN-LIVE** — unit-proven and red-proofed, but nobody killed the controller
mid-backup on real hardware; the capability map says so rather than rounding it up.

**S-5 — four operator decisions taken in discussion on 2026-08-02, recorded before anything is
built.** They existed only in conversation, which is the condition the standing rules were written
against. Labels are the ones used in the discussion (**D-a … D-d**) and are deliberately kept
distinct from S-3's `D1…D6`. **None of a, b or c is implemented by the session that recorded them**;
their work is carried as backlog rows (**R-165, R-166, R-167**), not as prose here.

- **D-a — the dedicated backup partition is MERGED AWAY, not resized.** The appliance's system drive
  carries the OS, a **50 G** docker area (`mp0`) and a separate **20 G** area holding local recovery
  units (`mp1`; demo-hp's guest 9201 ships exactly that). The decision is to **merge the backup area
  into the large data area** so local backups share space with app data and the ceiling stops
  existing — deliberately *not* to pick a larger number, because a larger number is the same wall
  further away. **Two conditions, recorded with the decision, not as commentary:** (1) it changes the
  **disk layout**, so it must land **before any external install** — after that, every deployed box
  is a migration; (2) it **removes a wall that currently fails safely**, so the storage monitoring of
  **D-c** must ship **with** it, never after. Closes the sizing question in **R-163**, which is
  **re-framed rather than answered with a ratio** and is not closed until the merge lands. Work:
  **R-165**.
- **D-b — app state gets a desired/observed model with its own store.** Three kinds of information,
  deliberately separated: **desired** (running, or stopped because the customer said so) stays in
  `app.yaml` with that app's other settings; **in-flight operations** ("a backup started and has not
  reported finishing") are **persisted**, written before the operation starts and after it ends, so
  an interruption is visible after a reboot; **observed** (unhealthy, restarting, drive gone) is
  **not** persisted and is rebuilt by looking. The last two live in a **state store separate from
  `app.yaml`**, so frequent writes never touch a file holding deploy settings and secrets.
  **Binding safety rule, verbatim:** *losing the state store must never cause an app to be deleted,
  restarted wrongly, or reported healthy when it is not — the worst acceptable outcome is re-running
  a backup that already ran.* The controller must also check **every container of an app**, so a
  partly-dead app cannot read as healthy (F-CRIT-1's shape). **Two things to establish before this is
  specced, not now:** whether the crash-safe journal already in the backup code covers the in-flight
  case — *it may already exist and merely be unwired, which has been the answer four times* — and
  whether the existing SQLite store is reachable from this code path. Work: **R-166**.
- **D-c — storage monitoring and backup alerts.** Two pieces: a warning to the **customer** as a
  drive fills, **before** anything fails; and a failure alert to the **hub/operator** when any backup
  fails for any reason, lack of space included. **Prerequisite for D-a** — see that decision's
  condition (2). Subsumes and widens **R-158**, which is the same alert one tier down; R-158 stays as
  the named seam and is not duplicated. Work: **R-167**.
- **D-d — which machines are protected: DooPlex and Peti's box, and nothing else.** DooPlex because
  it holds Gitea, the hub, the backups and the registry — everything else rebuilds from it. Peti's
  box because there is a real person behind it. **Every other box, both demo boxes included, is
  disposable and may be broken or reinstalled freely.** The earlier caution was costing sessions and
  pushing work onto DooPlex — the one machine that should never host it — so the ruling is a
  correction, not a relaxation. `documentation/runbooks/target-selection.md` updated the same session
  (2026-08-02); no backlog row, the decision *is* the change.

- **2026-07-27 — R-85: the DR tier is restore-tested UNATTENDED, and its failure is HEARD.** agent
  v0.104.0 + hub v0.77.0. R-82 proved a PBS restore round-trip **by hand**; three defects kept the
  unattended path from covering it. (a) `buildRestoreTestScheduler` built ONE runner on
  `cfg.Backup.BackupTarget()`, so the offsite tier's archives were never candidates — the DR tier
  could not be restore-tested unattended AT ALL. (b) the `Spec` was an immediately-invoked function
  evaluated once at daemon start, so nothing tier-varying was expressible and any config change
  stayed stale until restart. (c) a failed restore-test was a `[WARN]` line with **no event, no
  notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR
  signal this system produces was inaudible. **(c) was the one that mattered:** rotating tiers
  without it would only mean two tiers can fail silently instead of one.
  **Ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, no cadence
  knob; the per-tier last-proven timestamp is reported so the achieved interval can be **measured**
  before anyone decides a knob is needed.
  **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS);
  persisted `RestoreTestState` (survives restart; credit only on SUCCESS); `backup.InFlight`, the
  host-wide one-heavy-op gate shared with the backup path (a LINK concern — an offsite restore pulls
  multi-GB over the tunnel a backup pushes one); hub `restore_test_failed` / `restore_test_stale`,
  kept DISTINCT and anchored on R-81, operator-tier only.
  **Also fixed a time bomb I planted in Slice C:** a deadline test hard-coded the incident timestamp
  `2026-07-18T18:31:06Z` while comparing against the REAL clock — harmless under one 26h threshold,
  but once the offsite tier got an 8-day limit it passed all day and began failing at exactly 18:31
  UTC, 8 days later. A test that passes at commit time and fails hours later is worse than one that
  fails immediately.
  **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive
  UNATTENDED cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82).
  Spec: `documentation/backlog/SPEC-r85-phase4-5-2026-07-26.md`.

- **2026-07-26 — R-82 SHIPPED: the backup target split (local daily + offsite weekly).** agent
  v0.97→v0.102.0, controller v0.174→v0.175.0, hub v0.76.0, host-install 1.20.0. `BackupTarget()`
  returned ONE string and `BackupCadence()` ONE 24h window, so "local daily AND PBS weekly" was not
  expressible — which is why the DR tier was `applied` since 07-21 with **one** snapshot on
  demo-felhom and **zero, ever** on demo-hp. Now: `backup_targets[]` per-tier cadence+retention;
  ONE quiesce window for both due tiers (never two app outages for one night); per-tier hub
  thresholds (host 26h / offsite 8d); fresh-install default; an unprovisioned tier DEFERS.
  **Operator rulings:** 2-week offsite retention, first backup runs as long as it needs, one backup
  at a time per guest, drill box dropped from the rollout.
  **RETENTION IS A COMMERCIAL ATTRIBUTE — the hub decides, ep0 executes (operator ruling 2026-07-27,
  R-89).** A paid tier may buy longer retention, so the policy belongs with customer config on the
  hub, never in ep0's PBS config and never in a box's config. Execution stays server-side: a
  reconciler writes a **PBS prune job** and PBS's own scheduler runs it, so hub downtime leaves the
  last-known policy running rather than silently stopping retention. **Box tokens stay write-only
  (`DatastoreBackup`) — never widen a grant to fix a prune error:** a compromised box must not be
  able to delete its own offsite backups, which is the scenario offsite DR exists to survive.
  Increment 1 shipped 2026-07-27 (boxes stop attempting prune via `keep_last: 0`; per-namespace prune
  jobs on ep0, daily 03:30 UTC) — `runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`. This closed a
  live false-negative: **every** demo-hp PBS backup since 07-26 reported `job errors` while the data
  landed correctly, because `DatastoreBackup` carries no `Datastore.Prune`.
  **Four defects found by RUNNING it, not reviewing it** — a 30-min wait bound against a 41-min
  backup (the agent recorded `success:false` while the backup was still going); the restore tier read
  from the configured target instead of the archive (**a silent regression of the S4.1 fix** — the
  mechanism was never removed, its INPUT changed when `local_backup_target` was retargeted to
  `local`); a leaked scratch keeping `onboot:1`; and a tier firing at a not-yet-provisioned storage.
  **A correction I had to make mid-arc:** I reported that the restore-test would boot a
  network-conflicting clone and break the control plane. It does not — step 2 link-downs every NIC
  before boot, and it is unit-tested. I read a config artifact and inferred behaviour without reading
  the code path that consumes it, then escalated before finishing the check.
  **NOT done:** the offsite tier is never AUTOMATICALLY restore-tested (the scheduled test only ever
  selects the primary tier); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported
  cadence. The installer-default fleet flip waits on a full weekly cycle.
  **R-84 SHIPPED (agent v0.103.0)** — the cold in-memory store no longer causes a redundant backup:
  the due-check asks the STORAGE (ground truth) instead of persisting state, so a pruned archive
  correctly stops counting. Proven live on both boxes with the store cold. **demo-hp's FIRST EVER
  offsite backup landed** (4.25 GB) — the R-82 finding closed on the box where it was worst.
  Controller v0.175.0 deployed to both boxes. Evidence: `audits/SPIKE-r82-phase0-2026-07-26.md`, `REPORT.md`.

- **2026-07-26 — hub v0.75.0: R-81 SHIPPED — "no signal" is not "bad signal".** Third instance of
  one class (hub v0.12.0 `expected_backup_missed` daily-for-everyone; hub v0.73.0 `offsite_stale`
  minutes after a healthy repair; now this), fixed **as a class**, with the invariant written at the
  head of `assessBackupFreshness` naming all three. `assessBackupFreshness` returns
  OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored
  window. **The anchor was free** — Phase-0 probe found the hub retains 90 days of host-reports, so
  `GetHostReportsSince` + `newestBackupEvidence` (7-day bounded lookback, early-exit on fresh
  evidence) answer *"when did I last SEE evidence of a backup?"* rather than *"what does the latest
  report say?"*; absence is graded from first contact (`GetFirstHostReportAt`) reusing the existing
  26 h threshold as the grace — **no new knob, no agent change, no new persisted state**.
  **Rulings:** absence is never silent (a genuinely dead box must still alarm — that is the half the
  naive fix breaks, and `TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms` is what makes the
  suppression safe); reason strings stay distinct (the whole 07-26 diagnosis turned on reading the
  exact string); deferrals are logged so quiet ≠ not-checked. Tests 493→508; red-proofs A/B/C
  observed — A reproduces `newest backup is 176h0m0s old` **verbatim the mail demo-felhom sent**.
  **Replayed against the real 03:00 reports** (600/417/77 rows): all three now silent. `backupStaleAfter`
  deliberately untouched — ⚠️ it will alarm on a healthy WEEKLY PBS snapshot, landmine recorded,
  owned by **R-82**. The real finding behind R-80 stands: the **PBS/offsite-DR tier has no schedule
  at all** → R-82. Cause-fix (persist the agent store) → R-84. Evidence:
  `documentation/audits/DIAG-backup-missed-2026-07-26.md`, `hub/CHANGELOG.md` v0.75.0.

- **2026-07-23 — hub v0.72.0 + controller v0.161.0: R-70 SHIPPED + R-71(c) PARTIAL (the offsite
  last mile is visible; burned credentials self-heal).** One detector
  (`offsite.DeliveryStateFor` — secret-row timestamps × report offsite-presence), four consumers:
  operator card (static "delivered once" copy DELETED; state+age, amber past 30 min, stale-staged
  info line), `offsite_delivery_stuck` warning (≥1 h, 24 h durable cooldown via `LastEventAt` on
  the events table — restart-proof), R-71c self-heal through the EXISTING Re-issue path
  (`monitor.OffsiteReissuer` ← webServer, armed only with a provisioner; consumed ≥1 h + ≥4
  offbox-less reports + zero offbox evidence; one restage/24 h; every firing = a warning event),
  and the controller's truthful empty-state banner (v0.161.0, both boxes). **Rulings:** applied
  wins (stale-staged is a flag, never a downgrade); cooldowns durable via events table; both new
  events operator-only (no customerMessages, not in allowedEventTypes — pbsdr_* precedent);
  **R-39(a) guard in the CALLER** — the heal re-reads the secret row at act time and refuses over
  an unconsumed row; the store's clobber semantics stay (Re-issue depends on supersede). Clobber
  red-proof observed (`reissue calls = 1` + row clobbered with the guard removed). Self-heal +
  banner ship unit-proven, NOT live-fired (no broken box; F9 rule) — arms on the next natural
  occurrence. Live: checker silent on the healthy fleet; demo-felhom's stale-staged specimen
  intact (its cleanup is the operator's, post-validation). R-71(a) day-0 ordering OPEN → next
  spec. Evidence: `REPORT.md` (2026-07-23).

- **2026-07-23 — ISO train v1.25.0 + hub v0.73.0 (four rulings + rider).** host-install
  1.17.0→1.18.0, build-felhom-iso 1.24.0→1.25.0, felhom-bootstrap R-63, hub v0.73.0. **Done +
  live-verified:** (1) **F9 belt everywhere** — host-install installs the H1 OOB belt by default in
  appliance mode (`--no-oob` opts out; byo still refuses), `oob.enabled` written so the agent
  activates it; belt LIVE on both boxes, **login PROVEN on felhom-pve** (`felhom-op@demo-felhom`);
  the orphaned operator identity was re-anchored to the operator's real machine (WG peer + SSH key
  re-registered, ep0 forward rules persisted). (2) **apt** — `align_apt_repos` (appliance-only)
  switches enterprise→no-subscription; LIVE-aligned both boxes (`apt-get update` clean). (3) **R-63**
  — Lat2 console font before first paint. (4) **R-71 build-gate** — `assert_golden_ge_floor` in
  build-felhom-iso + rule 5; red-proofed. (6) **root vault** — felhom-pve root rotated+vaulted+
  verified over :22 (G1 parity). (7) **hub v0.73.0** — `offsite_stale` never-ran branch ANCHORED
  (applied-only + consumed_at/escrow anchor; one-state-one-owner with `offsite_delivery_stuck`),
  shipped+deployed, red-proofed. **CRITICAL live finding:** golden **0.153.0 < floor 0.156.0** in
  production now — the R-71 gap is real; the gate correctly blocks a v1.25.0 build. golden republished at 0.161.0 + **vouched** (gap CLOSED), v1.25.0 ISO **built**
  (gate live-passed). **Nested drill aborted on the N100** (install crawled — the N100 was
  contending with live guest 9201; KVM-confirmed, not memory) → cleaned up (9201 untouched);
  **operator steer: future drills on the HP**. R-63 font code-verified; the live ő/ű console
  capture rides the next HP drill (belt/apt already proven live on both boxes; no-floor-update is
  structurally guaranteed post-vouch). Evidence: `REPORT.md`, `scripts/CHANGELOG.md`,
  `operations/nodes.md`.

- **2026-07-23 — F10 offsite leg RESOLVED on demo-hp (diagnostic + designed-path repair + tier
  proof; no code changed).** The hub said "Provisioned" while the box said „Még nincs beállítva" —
  root cause: the day-0 managed update (0.153.0→0.156.0, 07-21 16:28) killed the offsite
  apply-bridge ~35 s after it consumed the one-time password, before key-install/persist; the box
  then hit the consume-404 dead-end at every start, silently, for 2 days. Repair = operator
  **Re-issue** click (converged in 8 s), escrow ceremony via the real wizard (auto-confirm 3 s),
  then the full proof: paperless-ngx toggled into scope, first offsite run (repo initialized on
  sub3, 79.8 MB, snapshots=1), probe deleted → `mode=full` restore (snapshot `2bf7f2e1`) → place
  merged 1 file missing-only → **md5-identical**. Minted **R-70** (last-mile invisible on both
  surfaces; hub never reads `one_time_secrets.consumed_at`) and **R-71 [P1]** (the race recurs on
  every fresh onboarding whose ISO floor lags the managed floor). Record:
  `documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. F10's PBS-DR half stays open
  (F13 + ceremony R-moment).

- **2026-07-22 — hub v0.71.0: the notification train (audit F11+F12+F14-light).** Four rulings now
  standing: **(1) recovery pairing** — `*_recovered` notifies the operator always and the customer
  **iff the customer was mailed the paired stale/down** (evidence = `notification_log` customer-
  channel sent rows via `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for
  recovery; ties → no mail). Severity semantics FROZEN — recovery stays `info`, routed by an
  explicit eventType branch before the severity gate. **(2) seed-at-claim** — `MarkClaimed` seeds
  `customer_notifications` from the registered email, INSERT-if-absent only (never touches an
  edited row; empty email = no-op; seed failure never fails the claim); default set is
  critical-only: node_down, backup_failed, disk_critical, host_disk_critical,
  storage_fill_critical, offbox_repo_orphaned. **(3) empty-email no-clobber** — a prefs push with
  empty email preserves a stored address (hub-side belt; controller 0.160.0 already guards its two
  push legs). **(4) priority headers** — error/critical mails carry `X-Priority: 1` +
  `Importance: high` (Resend `headers`, live-probed); the `test` event now also mails the operator
  with those headers (one click proves both channels + rendering). Latent nil-deref fixed:
  `sendTestEmail` panicked on a customer with no prefs row. NOT yet live: a natural `*_recovered`
  mail (next real staleness cycle / reboot drill — never fabricate one by blocking reports) and
  seed-at-claim on a real claim (Peti's Friday reinstall is the natural candidate).

- **2026-07-22 — hub v0.70.1: the ghost customer's Delete button exists now.** The v0.70.0
  ghost-delete path was fully implemented and fully unreachable — the **fourth inert-seam defect**,
  this time a TEMPLATE GATE: the Danger-zone card (and the `customerDeleteOpen` script) sat inside
  `{{if .HasConfig}}`, so a ghost rendered no Delete button (found live on `demo-vm-felhom`).
  Fix: `Deletable` page flag = the exact negation of the delete preview's 404 predicate; RESET
  stays HasConfig-gated, Block/Unblock config-only. Render tests per branch + 2 red-proofs.
  **The seam-wiring rule now covers template gates** (see CLAUDE.md) — handler tests that POST
  directly prove nothing about reachability. **Operator live leg PENDING: the demo-vm-felhom ghost
  delete click** (doubles as v0.70.0+v0.70.1 live validation). Same day: **agent 0.93.0 published
  + deployed fleet-wide** (`pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`), **R-17 CLOSED**
  (u629193-sub1 deleted at the console).

- **2026-07-21 — THE FLEET IS TWO HOSTS.** A second Proxmox node exists: **`demo-hp-bb76ea`**, an
  **HP t740 Thin Client** (Ryzen V1756B, 30 GiB, PVE node `felhom-host`, customer `demo-hp`), installed
  from the armed universal ISO the same day. Both hosts run **agent 0.92.1**; the N100 is
  `demo-felhom-8363b5`. **Full inventory: `documentation/operations/nodes.md`** — read it before
  touching the HP, because two things there are load-bearing: the box's **1TB Toshiba NVMe is present
  and UNENROLLED** (still carrying its old NTFS partition; a future Tárhely candidate that must join
  through the storage flow, never the installer), and there is **no operator SSH key baked** on it, so
  access is the hub-vaulted G1 break-glass password (R-61 is the fix for that lockout).
  **Both nodes are at the VACATION site and travel home ~2026-08-02.** `demo-hp` joined the tailnet as
  `100.76.96.79` — an **operator-lab exception**, not product shape; real customer boxes never get
  tailscale. **Second-hardware proof, worth keeping:** the pairing/day-0 chain ran end to end on virgin
  hardware it had never seen, the **shim loader booted with Secure Boot ENABLED** (so SB-off was an
  N100-firmware workaround, not a Felhom requirement), and the box **self-lifted the floor
  0.153.0 → 0.156.0 during day-0**. The install was not clean, and the failures are filed: **R-59**
  (no DHCP on the 4-port NIC → the installer baked a static fallback instead of aborting), **R-60**
  (first-boot NIC sweep self-heal), **R-61** (baked root password unknowable).

- **2026-07-19 — N100 MADE LOCATION-INDEPENDENT via Tailscale.** `felhom-pve` (the N100) joined the
  tailnet as a host package (`100.70.170.35`, expiry disabled); DooPlex already advertised
  `192.168.0.0/24` via its GitOps k3s `admin-system/tailscale` pod (`100.107.87.53`). `ssh felhom-pve`
  now targets the tailnet IP (direct-over-LAN at home, tunnel when away); `felhom-pve-lan` =
  `192.168.0.162` fallback. `--accept-dns=false` on the host (pi-hole split-horizon preserved). The
  host was converted **static→DHCP** (reservation MAC `68:1d:ef:5d:a6:64`→`.162` keeps home behaviour).
  **Measured spike:** `--accept-routes` on the N100 *while on `192.168.0.0/24`* hairpins the local
  subnet (table 52 overrides `main`) → kept OFF, it's a travel-only opt-in. **PBS is offsite** (its own
  WireGuard tunnel to Hetzner, not the home LAN) so it needs none of this. **Recorded, NOT fixed:** the
  controller↔agent plane is hard-pinned to `192.168.0.162` (`felhom-agent` `listen_addr` +
  guest `bootstrap.json`), so the box is reachable + backs up off-LAN but its local operator plane is
  not yet portable. Full doc: `documentation/operations/tailscale.md`. No code changed.

- **2026-07-19 — CC WORKFLOW MIGRATED TO DOOPLEX.** Claude Code now runs natively on DooPlex
  (192.168.0.180, Debian 13, `kisfenyo`) in a tmux session, working directly in
  `/mnt/5_hdd/felhom.eu/git/<repo>`. **DooPlex is now the canonical CC environment; the Windows
  workstation trees are stale** as of felhom.eu `ee48a48288`, felhom-agent `d81fddc7b1`,
  felhom-controller `ac7323dc9a`, app-catalog-felhom.eu `a14c97db65` (unchanged — it had no
  environment-specific content). Builds are local commands; felhom-pve is one `ssh` hop; `kubectl`
  is local. **`claude-in-chrome` is NOT available here** — endpoint-level validation is the
  standard method. New **clean-tree gate** before any build (`git status --porcelain` empty AND
  `HEAD` == `origin/main`), because the CC working tree is now the tree `build.sh` builds from.
  Windows knowledge is preserved in per-file "Legacy: Windows workstation" sections and the
  versioned workspace file `documentation/runbooks/workspace-CLAUDE.md` (live copy:
  `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`).

- **2026-07-17 — CUSTOMER SELF-BIND shipped (hub v0.66.0 + ISO scripts v1.20.0, R-27 slice 1).** A
  customer binds their OWN freshly-installed appliance without the operator. Operator clicks **"Send
  self-bind link"** on the customer Setup tab → hub mints a **7-day tokenized capability link** →
  emails it (Hungarian, sibling sender, NOT via the claim engine) → customer opens the **public,
  Hungarian `/bind/<token>`** page (no login — the URL token IS the auth) → enters the **console
  pairing code** (shown on the box screen) + their **retrieval passphrase** → hub stages the bind via
  the same `BindAppliance` (provenance `customer_selfbind`) → the box's ~30 s appliance poll delivers.
  **Viktor's three rulings, verbatim:** (a) *"only their own visible"* → console pairing code, **no
  appliance list ever rendered**; (b) *first-box entry* → operator-sent tokenized capability link over
  Hungarian email; (c) *lockout after 5 failed attempts* → token locks, "call support". Wrong code and
  wrong passphrase = **one identical generic failure** (no oracle; both factors compared
  unconditionally); expiry falls back to operator-bind unchanged. **THE TRAP (§9.2):** one public
  prefix `/bind/`, exempted from auth+CSRF at both `/login` gate sites via a single `isPublicBindPath`
  predicate (tight trailing-slash match; ServeMux `..`-cleans; handler rejects `/` in the token) — 4
  red-proofs verified red-then-green (lockout→C1, oracle→B, widened-prefix→E, single-active→C4).
  **GC verdict:** no appliance-staleness GC exists (`applianceStaleAfter` is a display badge only) → the
  7-day token TTL stands alone, single-active-per-customer, no reaper needed. **Controller/agent
  untouched; R-27b (controller second-box dismissable prompt) DEFERRED, mechanism sketched.** Green:
  full hub `build/vet/test` (17 ok, +9 self-bind tests) + `bash -n`. **DEPLOYED + LIVE (hub 0.66.0 on
  k3s, Synced/Healthy).** Live through the real ingress: `GET /bind/<garbage>` → 200 Hungarian expired
  page (public, logged-out, no-oracle — the 200-not-500 proves `selfbind_tokens` migrated live); `/` +
  `/hosts` → 302 /login (gate intact); `POST /bind/<garbage>` no-CSRF → 200 (exempt) while
  `POST /customers/acme/block` no-CSRF → 302 /login (exemption is `/bind/`-only). **PENDING
  (operator-gated, NOT-live):** operator "Send self-bind link" + the customer's real two-factor success
  (hub UI password-gated → CC can't log in) + the new-ISO console banner on a re-imaged box → fold into
  the R-1 rehearsal. See REPORT.md.

- **2026-07-17 — PBS DR STORAGE VISIBILITY + Offsite tab split + dual gauges shipped (hub v0.65.0 +
  tenantsync v1.2.0, R-5), LIVE.** Scoping correction (Viktor): "restic box" and "PBS box" are NOT two
  Hetzner Storage Boxes — restic = subaccounts on the shared Hetzner box (Hetzner API, v0.64.0); **PBS DR =
  the `felhom-offsite` PBS datastore on ep0** (NO Hetzner API; the hub had no read of its fill). **Option A
  (ruled):** a read-only `usage` op on the `felhom-tenantsync` ep0 forced command (twin of `fingerprint`,
  `df` on the datastore path — no customer_id, no admin token, NO mutation), polled by `monitor.PBSDRBoxChecker`
  (OffsiteBoxChecker clone over a `usageReader` seam; 15-min throttle; three states ok/unavailable/degraded;
  escalation-only `pbsdr_box_fill` on the `"pbsdr-box"` operator scope). `/offsite` now has **Restic** +
  **PBS DR** tabs (endpoint cards under PBS DR — the endpoint IS the PBS host); the dashboard has **two
  gauges** (restic %·ratio, PBS %). **Graceful degradation: the hub deploy is INDEPENDENT of the ep0 update**
  — an ep0 ≤ v1.1.0 → `ErrUsageUnsupported` → the gauge shows "n/a" until updated (no hub redeploy).
  **PROVEN-LIVE (data pipeline, both gauges):** ep0 updated to v1.2.0 this session; the hub polled it —
  `PBS-DR box refreshed: 19.1% full (7.1 GB of 37.2 GB)`. **DECISION:** PBS thresholds are separate keys
  (`PBSDRBoxFill*`) defaulting to restic's 80/90 (independently tunable; Viktor's call for a different pair);
  no oversubscription for PBS (namespaces, not quotas). **OPEN:** UI render unit-verified only (hub UI
  password-gated); the PBS fill alert email is unit + red-proof verified, NOT fired live (datastore nominal
  at 19%). READ-ONLY against ep0 + Hetzner. See REPORT.md.

- **2026-07-17 — OFFSITE POOL-BOX AGGREGATE shipped (hub v0.64.0, R-5).** The operator now sees the
  shared pool box's real state on the hub: total fill vs capacity, Σ(shared soft quotas)/capacity (the
  oversubscription ratio), per-customer usage/quota bars, and a box-level operator alert (fill 80/90%,
  oversub 2.0×). New `monitor.OffsiteBoxChecker` (OffsiteChecker-sibling): fetch-throttled to one Hetzner
  GET / 15 min, cached `BoxSnapshot`, escalation-only + recovery re-arm; two INDEPENDENT signals (fill +
  oversub). Capacity = `storage_box_type.size`, usage = `stats.size` (Phase-0-pinned, bytes); Σ(quota)
  from the ConfigJSON `Descriptor` (`offsite.ReadDescriptor`, new — NEVER the report echo; dedicated +
  disabled excluded); per-customer USAGE from the report echo. Events scope `"pool-box"` → operator channel
  only, no SaveEvent. Web: Offsite-tab panel + compact dashboard tile, reading the cached snapshot only
  (never fetches). **PROVEN-LIVE (data pipeline):** the in-cluster checker computed the real box —
  `0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x`. **DECISION: thresholds
  (80/90/2.0×) are Claude's encoding — Viktor's ruling pending; named `Alerting.OffsiteBox*` keys = the
  one-line flip.** **OPEN:** the UI render is unit-verified only (hub UI password-gated → CC can't
  screenshot); the alert emails are unit + red-proof verified but NOT fired live (real pool nominal — a
  live-fire would email Viktor; offered on demand). READ-ONLY against Hetzner throughout. See REPORT.md.

- **2026-07-17 — SYSTEM-INITIATED IMMEDIACY wired per-plane (hub v0.63.0).** The immediate-sync arc
  covered only OPERATOR-initiated changes; **system-initiated** desired-state mutations bumped the
  generation silently, so a freshly onboarded box waited a full agent tick for state the hub had already
  minted (observed live at slice-C onboarding). Decision: wire the existing, live-proven notifiers into
  every system site on the CORRECT plane — agent-plane `poke.Notifier` at `PBSDRAutoProvision` (the exact
  lag), `ReissuePBSDR` (which also lifts the pbsdrheal reconciler's escalation with zero reconciler
  changes), `handlePBSDRReissue`, and the two admin api desired-state writers (new nil-safe `Poker` seam);
  controller-plane `intent.Hub.Bump` at `reissueOnReenroll`. All fire AFTER the store write only. The
  **pbsdrheal Restage path is deliberately poke-free** (no generation bump → the agent's 60 s pbsdr ticker
  is its pickup path — a poke there is a verified no-op); the WG register/delete bumps stay poke-free
  (undeliverable pre-tunnel / transport removed — the agent fast-tick SECONDARY owns the register leg).
  The **first-hold-baseline race** is still OPEN by design (a bump landing during a fresh controller's
  FIRST wait is recorded as baseline without firing — the ruled fix is carrying `intent_gen` in the report
  ACK, its own future task). Unit-tested + red-proofed; live legs fire only on real system events (fold
  into the rehearsal bind sequence). See REPORT.md + hub/CHANGELOG.md v0.63.0.

- **2026-07-17 — R-21 SLICE C SHIPPED: the universal secret-free ISO (hub v0.62.0 + scripts v1.19.0, LIVE).**
  The distributed ISO carries NO customer secret. A box booted from `build-felhom-iso.sh --pairing`
  self-registers as an UNCLAIMED APPLIANCE (`POST /api/v1/appliance/register`, keyed by SMBIOS-uuid +
  MAC set — the DMI-verdict tiebreaker; token = sha256-stored, 256-bit), the operator BINDS it to a
  customer on the Hosts page (new "Unclaimed appliances" section, SSH-key fingerprints, host-count
  display-only), and the hub delivers customer-id + retrieval passphrase ONCE (one-shot poll: 404
  no-oracle / 204 unbound / 200 deliver / 410 gone). The bootstrap is **one unit, two modes** — direct
  (env-baked, byte-identical, regression-proven zero-appliance-calls) vs pairing (register→poll→write
  env→fall through to direct). Red-proofs: one-shot delivery + register idempotency (both proven red).
  Live-verified through the public ingress (register→token, 204, bogus→404). Artifact proven secret-free
  (baked env = hub URL only; manifest `secret-bearing: no`). **Bind is operator-password-gated** → the
  live boot→bind→day-0 composition (+ physical N100) folds into the supervised rehearsal, which now runs
  the COMPLETE final flow in one pass (RESET demo → generic ISO → bind → day-0). SSH-key pinning =
  stored + fingerprints displayed (host-attach on bind deferred, no clean surface). Self-bind page = R-27.
  R-21 now SHIPPED (A+B+C). See REPORT.md.

- **2026-07-17 — R-21 SLICE B SHIPPED: firmware loader option (scripts v1.18.0).** Closes N100 **F1
  (HIGH)** — cheap AMI (`AN3PLUS 0.01`-class) firmware can't USB-boot the ISO's stock GRUB
  (`relocation 0x0`). `build-felhom-iso.sh --loader shim|mkimage` (default **shim**, unchanged;
  profile-settable via `FELHOM_LOADER`). **mkimage** = a monolithic `grub-mkimage` loader built from
  the ISO's own GRUB (module set from its `grub.cfg`, embedded `search --fs-uuid`→`configfile`), swapped
  into the ISO9660 tree + `efi.img` ESP by `mkimage-surgery.sh` **after** `prepare-iso` (payload
  provably untouched), re-mastered with xorriso preserving BIOS-hybrid + UEFI + GPT-ESP (drops only the
  irrelevant Apple HFS+/APM). **Unsigned → target Secure Boot MUST be OFF.** Loud: banner + manifest
  `loader:` + `-mkimage` filename. New `profiles/n100.profile`; Dockerfile gains grub/mtools.
  **RUNBOOK-B legs (nested VM 311, OVMF):** shim boots+installs under SB-enforcing + SeaBIOS; mkimage
  boots+installs under SB-off; **mkimage under SB-enforcing FAILS `Access Denied`** (red-proof); surgery
  byte-identical payload. Physical N100 boot still pending → supervised rehearsal (an `n100-safety`
  match-nothing ISO built + sha-recorded, unbooted). PXE stays a deferred R-21 note. See REPORT.md.

- **2026-07-17 — CUSTOMER RESET SHIPPED: hub v0.61.0 + felhom-tenantsync v1.1.0 (LIVE).** The middle
  lifecycle tier (host delete < RESET < customer Delete). One operator action → pre-first-install: all
  OPERATIONAL state dies (offsite repo, PBS namespace+backups, DR recipe, one-time secret, claim state,
  retained escrow custody); IDENTITY + basic config (incl. the offsite tier CHOICE) + provenance + events
  SURVIVE. Rulings: separate escrow-custody ack; clears claim (fresh code next onboarding); REFUSES while
  any host row exists; live-counted confirm inventory. Discipline: external teardown FIRST, DB purge LAST,
  every leg idempotent → partial run re-runs from the top (purge withheld until externals ok). New:
  `store/customer_reset.go` (journal+inventory+ack-gated purge), `claim.ResetToUnclaimed`,
  `offsite.{Deprovision,OffsiteIdentifier,ClearProvisionedDescriptor}`, `tenantsync.Deprovision` +
  `felhom-tenantsync.sh` **deprovision** op (v1.1.0, destroys ns+groups+token, shared user untouched),
  `web/customer_reset.go` (GET inventory JSON / POST orchestration) + an **amber** RESET card distinct
  from the red Danger-zone Delete. Red-proofs: ack-gate + partial-failure resumability (both proven red).
  **Live-drilled on ep0** (throwaway `drill-reset-01` with a real backup): deprovision `deleted:true`,
  idempotent re-run `deleted:false`, all 3 real tenants survived. Hetzner-delete + the password-gated web
  POST covered by tests (offsite Deprovision mirrors live-proven ReissueCredentials). See REPORT.md.

- **2026-07-17 — HOST-DELETE DEMOTES ESCROW: hub v0.60.1 (LIVE).** Closes the v0.60.0 review gap:
  `DeleteHost(deleteEscrow=true)` now DEMOTES the current escrow blob into `host_escrow_superseded`
  (copy-before-delete, same tx) + spares existing superseded rows — never destroys custody. The
  customer Danger-zone Delete (`DeleteCustomerConfig`, which before touched ONLY `customer_configs`)
  is now the one true purge point: cascades both escrow tables for all the customer's hosts incl.
  already-deleted (via F-14 `host_deletions`). One shared copy routine `demoteCurrentEscrowTx` (reused
  by SaveHostEscrow). Checkbox/refusal/Danger-zone wording → demotion. **S6b VERDICT: OBSOLETE** —
  re-enroll upserts cleanly (`UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no dup
  refusal) + v0.57.0 arc auto-fires re-issues (ROADMAP R-3 refined). Red-proofs
  `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. Observation:
  customer-delete still doesn't touch host rows / the broader offboarding (rehearsal's question). REPORT.md.

- **2026-07-17 — OFFSITE CONTINUITY Part B SHIPPED: hub v0.60.0 (LIVE).** Superseded-escrow
  retention (data-first, Viktor's ruling). `SaveHostEscrow`'s destructive `ON CONFLICT` overwrite
  destroyed the old passphrase's only copy (the reinstall-orphan incident); now a new
  different-sha blob RETAINS the old into the new `host_escrow_superseded` history table (tx, before
  overwrite); same-sha = idempotent. ACK/restore read the current row unchanged. `escrow_superseded`
  event + operator retained-count on host detail; registered `offbox_repo_orphaned`/`offbox_repo_reset`.
  Red-proof `TestSaveHostEscrow_RetainsSuperseded`. Pairs with controller v0.142.0 (Part A
  orphaned-repo guard). Guided old-history recovery = R-26 (not built). See REPORT.md.

- **2026-07-17 — N100 polish (F5 scripts + F6/F7 controller).** F5 (host-install v1.17.0):
  appliance mode auto-sizes the guest (RAM=clamp(host-4096,min 4096,max host-2048,ceil host-1024) +
  cores=host-1 min 2) when no explicit cap; explicit `--memory`/`--cores` win. Harness red-proof
  (8/16/32 GB). Same commit: the stale "EMPTY by default" operator-key comment fixed (R-16 doc-drift,
  keys are PINNED). F6/F7 landed in controller v0.141.0 (LIVE on demo; fresh-USB wizard enroll+format
  now PROVEN-LIVE — see the felhom-controller REPORT). New ROADMAP R-24 (guest resources as
  desired-state / live resize = F5 follow-on). Operator sidebar #1 (BIOS State-After-G3→Power-On)
  added to `scripts/iso/README.md`. See REPORT.md.

- **2026-07-16 — DIRECTION-2a AGENT-PLANE POKE SHIPPED: hub v0.59.0 (LIVE).** The agent-plane
  sibling of the v0.58.0 wait channel: `internal/poke` — a pinned-host-key SSH sender (wgsync
  sibling) + fire-and-forget `Notifier` (`PokeHost`/`PokeAllHosts`, nil-safe, detached) that nudges a
  box in seconds via a CONTENTLESS UDP poke relayed hub → ep0 `felhom-poke` forced-cmd → wg0-origin →
  the box WG /32:51822. Wired at `applyPBSDR` (per-host) + `handleSetArtifacts` (fleet). ep0 surface
  (`scripts/felhom-poke.sh`, non-root, no sudoers; runbook §11) + `Secret/agent-poke` +
  `manifests/hub.yaml` (tag 0.59.0). **Deployed live** (ArgoCD Synced/Healthy, `agent-plane poke
  enabled`); ep0 leg live-validated (`poke-fired`; non-WG + `rm -rf /` refused). Source note: the
  MinAgent-floor save does not bump per-host generation (self-update rides signed-ops) → the fleet
  poke accelerates the next report, not a delta. Pairs with felhom-agent v0.89.0 (the listener).
  **PROVEN-LIVE 2026-07-17:** the operator's real manifest save logged `poke: sync-poke delivered to
  10.77.0.2` (+ floor HELD for the below-MinAgent box); the demo (now 0.89.0) received a poke in
  ~31 ms and fired the immediate desired-state cycle (save→tick ≈ ~0.45 s). All three live legs
  (Scenario 4 A/B train, Scenario 1 R-22 self-heal, Scenario 3 latency) done; capability-map poke +
  F4 rows upgraded to PROVEN-LIVE. See REPORT.md.

- **2026-07-16 — DIRECTION-2 IMMEDIATE-SYNC SHIPPED: hub v0.58.0.** The hub→box "sync now" wait
  channel (option b of `documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`). New
  `internal/intent` in-memory per-customer generation notifier (Bump/Wait/Close; coalesce-to-latest;
  race-closer; in-memory by design → a restart costs one harmless full-state report, never a storm).
  `GET /api/v1/wait` long-poll: per-customer only (global key 400; A can't see B), 240 s hold with a
  25 s heartbeat newline that defeats nginx's 60 s `proxy_read_timeout` **without any ingress
  annotation** (WriteTimeout lifted per-connection via `ResponseController`; global timeouts
  untouched). Every operator-intent web handler bumps AFTER its store write (config CRUD, claim
  resend, offsite re-issue/freeze, password regen, block/unblock, floors [global bumps all managed],
  controller log-tail + log-bundle). `intentHub.Close()` before `server.Shutdown`. Pairs with
  controller v0.140.0 (the long-poll client). The immediate-sync arc is COMPLETE for the controller
  plane (Direction 1 = ctrl v0.139.0, Direction 2 = hub v0.58.0 / ctrl v0.140.0); the agent-plane
  ep0-poke stays PARKED in the OOB arc. Detail: `hub/CHANGELOG.md` v0.58.0, `REPORT.md`.

- **2026-07-16 — REINSTALL ARC SHIPPED: hub v0.57.0 (F2/F3/2.3).** The clean-slate
  reinstall-of-existing-customer path is now first-class (Peti R-1 prerequisite). On host-enroll's
  mint path (single-bump-per-re-enroll): claim re-issue for a claimed customer (`ReissueForReenroll`,
  reset code — the hub never stores the password, so fork B not A), offsite re-issue to the fresh box
  (`ReissueOffsiteForCustomer`), and escrow-honesty — re-issuing offsite marks the escrow STALE
  (`MarkEscrowStale`), withholds the mismatched restic-hash from auto-confirm, and shows "stale" not
  "ceremony done" (2.3, red-proofed). **Part 0** unblocked the demo live (ACL grant on
  `/storage/felhom-offsite` → PBS-DR converged, `escrow.pbs_storage_id` seeded). **Controller +
  scripts unchanged** — both spec premises were contradicted by source (the controller reads escrow
  prereqs live from the agent; the installer can't know the descriptor-provisioned storage id). **F4**
  root fix is agent-side → **ROADMAP R-22** (agent train: the reconcile pre-check 403 must not abort
  before the root-run apply grants the ACL). Detail: `hub/CHANGELOG.md` v0.57.0, `REPORT.md`.

- **2026-07-16 — R-21 PHYSICAL RUN on the demo N100 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`).**
  demo-felhom host reinstalled clean-slate from a pipeline-built ISO; first-boot chain reached
  **rc-0 on the first attempt on real hardware** (closes slice A's operator-gated boundary),
  serial-filter safety proven on metal (external-HDD canary byte-identical), PBS-DR reconciler
  self-healed on the reused peer. **The obstacle was firmware, not us:** the cheap AMI `AN3PLUS 0.01`
  board can't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`; SB-off + shim-bypass don't help) —
  worked around live by rebuilding the stick's loader with `grub-mkimage` from the box's own working
  GRUB. Findings (to tackle): F1 firmware boot loader (slice-B), F2 claim-re-issue on reinstall,
  F3 offsite-re-issue, F4 PBS-DR ACL on non-default storage id, F5 guest-RAM configurability,
  F6 drive-init doesn't mount+attach, F7 back-route. Box healthy, BookStack deployed, demo live.

- **2026-07-16 — R-21 SLICE A SHIPPED: bare-metal Felhom ISO pipeline (`scripts/iso/`, scripts
  v1.16.0).** A DooPlex pipeline (`build-felhom-iso.sh` + `Dockerfile.assistant`) turns the official
  PVE ISO into a Felhom auto-install ISO whose first-boot stub installs a retry-forever
  `felhom-bootstrap` unit that unattended-fetches `felhom-host-install.sh` from the public
  `felhom.eu/scripts/` channel and runs it (host-install **unmodified**). Validated on VM 310:
  build gate + red-proof, disk-filter fail-safe, stub→unit→real-fetch→invoke→retry, resume-decision,
  exactly-once, no-net retry+recovery — all GREEN. **Rulings settled:** stub is from-iso (no
  cert-rotation risk); ordering fully-up; the retry unit owns all network work. **§4.4 finding
  (shapes slice C):** the distribution URL carries no secret, but the unattended install needs the
  customer retrieval passphrase → **the slice-A ISO is secret-bearing** (supervised/single-use; the
  bootstrap shreds the env on success). OPEN: host-install rc-0 terminal success (hub-enrolled +
  guest-provisioned) is operator-gated — a drill customer needs the password-gated create-UI (a
  live-`hub.db` insert is unsafe). Slice B = physical N100 RUNBOOK; slice C = claim-pairing to
  de-secret the distributed ISO. Detail: `REPORT.md`, `documentation/audits/SPIKE-baremetal-iso-2026-07-16.md`.

- **2026-07-13 eve — ESCROW CEREMONY WIZARD SHIPPED (agent v0.88.0 + controller v0.127.0; this
  repo: host-install v1.16.0 + RUNBOOK rewrite).** The recovery-code ceremony is now
  customer-driveable from the controller (`/backup/escrow`): one-shot R reveal, re-stage-first,
  Scenario-F stale-blob re-check (fired LIVE on both boxes' hash-less blobs at first ACK; drill's
  repaired via a real ceremony — hub hash now covers). **Ruling F1 (2026-07-13), settled:** *R
  transiting the Cloudflare tunnel once at reveal is an accepted risk — same trust class as the
  claim code/login password; LAN-direct delivery is PARKED.* (threat model:
  runbooks/RUNBOOK-escrow-ceremony.md). host-install v1.16.0 = FELHOM_ESCROW rides the canonical
  sudoers fetch; hub `hostInstallVersion` synced (gate green), **hub NOT redeployed** (const rides
  the next train). OPEN: supervised full-browser wizard pass (Viktor's drill login; also demo, to
  clear its legacy stale warning); agent 0.88.0 publish + Day-0 vouch at the next train.

- **2026-07-13 — CLOSING BUNDLE SHIPPED: hub v0.53.0** — the last engineering items on the
  pre-tester board: F-14 gated auto-Reissue (deletion provenance `host_deletions` in-tx +
  `token_exists` gate → the EXISTING Reissue op + `pbsdr_auto_reissue` audit event; no record /
  un-acked → refusal byte-unchanged), dead-host roll-up honesty (customer status = worst of
  controller-derived + per-host `hostStatus` on dashboard//configs/detail, cause chip names the
  host; the Peti 23h-masking shape is the pinned fixture), operator bearer OUT OF GIT
  (`Secret/report-api` secretKeyRef non-optional + `REPORT_API_KEY` env override +
  `scripts/manifest_bearer_gate.py`; VALUE ROTATION = supervised step in
  documentation/runbooks/secrets.md — the git-history copy is dead only after it).
  **Two operator rulings (2026-07-13), recorded verbatim — future sessions must treat these as
  settled:**
  1. ***F-14:** auto-Reissue of offsite credentials is permitted ONLY when the hub's own deletion
     record shows the tenancy's owning host was removed through the escrow-ack flow — acknowledged
     destruction is not silent re-keying. No record → manual Re-issue stays the only path.*
  2. ***Roll-up:** a customer's status may never look better than its worst expected host. Single
     staleness definition (`hostStatusOf`) — no second threshold anywhere.*
  Gate additions: `python scripts/manifest_bearer_gate.py` after ANY manifests/ change (64-hex
  bearer shapes; felhom.secret.yaml carve-out = visible KNOWN-BACKLOG line). GOTCHA: the repo
  .gitignore `*secret*` pattern silently un-tracks any "secret"-named script — hence the
  "bearer" name. The controller repo's controller.yaml.example carried the REAL bearer literal —
  scrubbed (felhom-controller 51c871a).

- **2026-07-13 — POLISH BATCH SHIPPED: hub v0.52.0 (+ controller v0.123.0)** — take-two F-15
  CLOSED (reset-request response carries the rotated code hash — ACK shape/guarantee — emailed
  codes work immediately; live 1 s apply + first-try accept on Viktor's exact failure path) and
  F-16 CLOSED (zero native `confirm()` in the hub UI — `inline_confirm.html` partial, Igen/Mégse
  inline two-step, asserted by `scripts/hub_confirm_gate.py` — **which exists but is NOT enforced:
  it is invoked by nothing** (R-29); danger-zone typed-confirm untouched;
  live: offsite re-issue completed under browser automation without freezing). Operator resend
  keeps the ACK lag — flash says so. Take-two §4 dispositions updated (F-15/F-16/zero-toggle obs
  → SHIPPED). Remaining from take-two: **F-14** (needs operator ruling: auto-Reissue re-adopt vs
  never-silently-re-key) + S5 DR-restore drill; hub bearer key rotation still DUE.

- **2026-07-12 — DR-TIER-BY-DEFAULT BATCH SHIPPED: installer v1.15.0 + hub v0.51.0 (+ agent
  v0.86.0 in felhom-agent)** — closes DRILL-day0-vm F-1/F-2/F-3/F-7/F-8/F-9/F-10 and F-6 (by
  policy). **The five operator decisions (2026-07-12), recorded verbatim — future sessions must
  treat these as settled:**
  1. *Every install deploys the full DR plumbing uniformly (pbs-apply wrapper, `age`, WG enabled,
     full ACL). **Capability is baked; activation is a hub flag.***
  2. ***DR tier defaults ON for new customers.** Disabling is the per-customer exception — and it
     removes the Felhom-side cost entirely (no PBS namespace, no endpoint dependency).*
  3. ***Offsite app backup requires the DR tier ON** (fork-4 needs the ceremony; the ceremony
     needs PBS). Enforced hub-side by extending the drill-proven dependency guard.*
  4. ***Identity-only escrow is formally PARKED** — closed by policy, not implemented. Revisit
     only if a customer class demands lean-tier offsite. Record in the ceremony runbook + CONTEXT
     so future sessions know it was a decision, not an oversight.*
  5. ***WG is base infrastructure, not a DR component** — always-on like cloudflared; it is also
     the future OOB-management transport. BYO consent: a disclosure line lands in the
     tester-agreement doc; per-customer hub disable remains possible.*
  Mechanics: installer ships `felhom-pbs-apply` + `age` + `wg_tunnel.enabled: true` (byo wg-off
  assert RETIRED — consent = typed disclosure ack + tester-agreement line), single VERSION source
  with a cross-check against the hub Setup-tab const written as `scripts/hostinstall_gates.py` —
  **but NOT enforced: the script exists, is RED today (1.19.0 != 1.22.0) and is invoked by nothing**
  (tracked as R-94 leg (b), class R-29) — default ACL
  keeps felhom-pbs (doc advice REVERSED — narrowing caused the drill 403). Hub: per-customer
  `dr_tier` flag (new default ON; legacy backfilled from descriptor reality, one-time),
  cascade-not-error semantics + per-stage status on the edit form, WG-registration
  auto-provision hook (scenario A hands-free), offsite-requires-DR guard (exact F-6 message),
  host-page capability chips (agent 0.86.0 `inactive` = neutral; pre-1.15 "binary not found" →
  migration hint). felhom-pve's latent F-3 state fixed live; Peti's box unreachable (his F-3
  check + D/E/G deferred).
- **2026-07-12 — HUB v0.50.0 SHIPPED + LIVE: customer-claim password arc (closes DRILL-day0-vm
  F-4/F-5).** The customer OWNS the dashboard password via a one-time claim code the hub emails to
  the registered address (`bcrypt(code)` stored, plaintext only in the send). `internal/claim`
  engine (EnsureIssued idempotent Day-0/live-box issue+email, Resend, RequestReset 3/day,
  MarkClaimed set-only) + `store.customer_claims` (RotateClaimCode bumps generation + preserves
  claimed_at) + delivery (config bake `web.claim_code_*`, report-ACK `claim` object, `POST
  /api/v1/claim/reset-request`) + Setup-tab chip/resend + 3 Hungarian emails. Pairs with controller
  **v0.122.0** (the gate). **LIVE-proven on drill box demo-vm-felhom: gate ON via the real edge (/ →
  302 claim page, /api → 401); code emailed to the registered address.** Deployed felhom-hub:0.50.0
  (ArgoCD synced/healthy). Floor raise 0.120→0.122 = operator's supervised LAST step. Observed: the
  hub issues a claim code for EVERY reporting customer on first report (benign for pre-0.122 boxes —
  no gate, ACK ignored). Report: documentation/audits/DRILL-day0-vm-2026-07-12.md (F-4/F-5 RESOLVED).
- **2026-07-12 — HUB v0.49.0 SHIPPED: Edit tab merge (edit-a) + scoped auto-refresh + riders**
  (CHANGELOG v0.49.0; commits `e740147` → `2e03de1` → `1d94b1a` + docs/manifest). The standalone
  customer edit page merged into the customer page's **Edit** tab (renamed from Settings): shared
  `config_form_body` sub-template + `configFormData` builder (REUSE.md rows added), Block/Delete
  relocated to a danger-zone card, header cleaned, `GET /configs/{id}/edit` → 302 `#tab=edit`,
  post-action redirects anchored to their tab, `settings→edit` legacy-hash alias. Update handler
  gained server-side required-field validation (error path preserves SUBMITTED values —
  red-proofed). The 60s auto-refresh is now scoped: fires only on live tabs
  (`data-live-tabs="overview,applications,events,host"`) and never while a form is dirty; muted
  `(paused)` hint. Riders: `/style.css?v={{hubVersion}}` cache-bust on every template (closes the
  v0.47.0 max-age=3600 gotcha) + the CLAUDE.md `git add -A` ban codified. **NOTE: the task spec
  said v0.48.0, but v0.48.0 (app_start_failed) had already shipped — published tags are never
  re-pointed, so this train is v0.49.0.** Out of scope (observed, not done): type-to-confirm for
  customer delete (still `confirm()`, relocated as-is); the workspace-root `E:\git\CLAUDE.md` and
  other repos' CLAUDE.md files still carry `git add -A` snippets — flag for their own sessions.
- **2026-07-12 — Day-0 VM DRILL COMPLETE (auto-confirm FIRST LIVE FIRING): full arc proven on a
  fresh nested-PVE box** — appliance Day-0 → floor-at-first-report → ceremony → **auto-confirm
  pending→escrowed in ~7.5 min, zero clicks** → offsite backup + restore round-trip. Two HIGH gaps:
  **F-4 no operator password-set path exists (G10 unclosable, dashboards born OPEN — blocks tester
  gate)** and **F-6 identity-only ceremony never implemented (no-PBS appliance can't escrow — drill
  forked to PBS DR tier = full Peti-sequence rehearsal, all green)**. Installer fresh-box gaps:
  felhom-pbs-apply not shipped (F-7), `age` missing (F-10), root-owned guests/ parents (F-3 —
  check demo for the latent copy), silent root@pam rotation UX (F-8). Runbook fixes committed
  (day0 A.2 anonymous-fetch; escrow-ceremony identity-only claim CORRECTED + age prereq). Report:
  `documentation/audits/DRILL-day0-vm-2026-07-12.md`. Drill VM qm 300 kept (3 snapshots) for
  re-drills; teardown list in report §9.
- **2026-07-12 — CAMPAIGN-3 Task A SHIPPED: agent v0.85.0 boot/recovery plane + appliance self-heal.**
  Fixes F12 (CRITICAL boot ordering cycle — templates drop network-online, `MigrateNetworkUnits`
  repairs installed units), F11/F10/F9 (reassert: fstype-driven classify, reset-failed+rearm, per-unit
  verdict, + re-arm-on-guest-blind matrix-correction), F2/F1 (zero residue: reset-failed + rmdir), the
  guest-hook rc255 belt (recover+timeout, wrapper no longer execs), and the appliance node self-heal
  (`internal/selfheal`: host networking recovery, gated on `deployment_mode` — host-install `--mode`
  templates it; byo remedy unreachable). 3 sudoers grants added (reset-failed, rmdir, start networking).
  **Live matrix on demo felhom-pve: host reboot ×2 = 0 ordering-cycle lines both boots (F12 dead);
  appliance self-heal recovered networking on attempt 1; F10 reset-failed+rearm sweep loud; zero
  residue; guest-reboot heals.** **NEW finding F13 (HIGH, deferred):** active nfs4 under mp8 can fail
  PVE rbind (rc255); NAS-down-at-boot → apps `Created` (fix-3). Peti's box (agent 0.81) unaffected
  until his rollout — carries the F12 coin flip if it has a NAS share (interim drop-in in the agent
  REPORT). host-install v1.14.0; scripts CHANGELOG. Details: `documentation/controller/network-storage-nas.md`.
- **2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; host recovered, RCA closed.** Unattended chaos campaign
  22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/
  restore byte-identical/EIO instant/stub badge + deploy-409 live-validated). The NAS automount
  plane produced 2 CRITICAL + 3 HIGH: **F12 CRITICAL (= the overnight host loss, RCA'd next
  morning): the agent automount template's `After/Wants=network-online.target` creates a boot
  ordering cycle — systemd deletes an arbitrary job to break it; boot -1 sacrificed networking
  (host dark 7h, hardware exonerated), boot 0 sacrificed the automount (NAS dead). Every host
  boot with an enrolled share = coin flip until the template drops that ordering (`_netdev` on
  the .mount suffices).** F10 CRITICAL (start-limit never re-armed, once blocked guest start),
  F11/F9 HIGH (post-start reassert "skip-active" skips the healing automount restart; agent-restart
  sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots; F7 HIGH (in-place dump write →
  0-byte tar replaced last good backup during outage window). Fix-6 numbers: ring horizon 55min
  idle / **6.5min under load**, wiped on every restart. Box left in defined state (6 wave apps
  healthy, temps removed, NAS baseline-identical). Full ledger/evidence:
  `documentation/audits/CAMPAIGN-3-2026-07-11.md` + `180:~/campaign3/`.
- **2026-07-11 — HUB v0.47.0 UI REORGANIZATION SHIPPED** (CHANGELOG v0.47.0; commits `9f29bf3` →
  `0daddcd` + docs). Five deliverables: **(1)** `.data-table td a:not(.btn)` button-contrast fix;
  **(2)** customer page = **8 hash tabs** (`#tab=…`, sticky summary strip, Events error badge,
  no-JS graceful degradation — panels hide only via a JS-added body class); **(3)** shared
  `host_detail_body` sub-template rendered by `/hosts/{id}` AND the new per-customer **Host tab**
  (a list by design; `store.ListHostsByCustomer` + `hostDetailData` builder); **(4)** **stale host
  removal** — `GET /hosts/{id}/delete-impact` (counts/booleans only) + `POST /hosts/{id}/delete`
  behind type-to-confirm; ONLINE → 409 always (no override), escrow needs an explicit checkbox
  (`ErrHostEscrowPresent`, tx never starts), one-tx cascade incl. the bound wg peer (wgsync's 5-min
  declarative push converges the endpoint; log bundles die by `scope_id == host_id` only);
  **(5)** **/offsite multi-endpoint management UI** — all `wg_endpoints` rows as cards +
  add/edit/delete with 409 guards (peers-in-subnet pins subnet + delete). **Deferral (explicit):
  peer allocation, the wgsync reconciler push and the desired-state merge stay lowest-endpoint-id
  (`GetWGEndpoint` untouched); the future arc is a `wg_peers.endpoint_id` migration + per-endpoint
  allocation/reconciler/desired-state.** Five red-proofs ran (online gate, escrow ack, bundle
  scope, endpoint-delete guard, subnet-change guard). *Resolved:* the earlier ⚠️ about `146d165`
  sweeping the Part-4 WIP — the sweep caught `hosts.go` mid-red-proof (escrow ack bypassed);
  `068427a` restored the gate; the arc is now complete and author-reviewed. **Stale-doc note:** the
  workspace-root CLAUDE.md's hub deploy wording predates GitOps — deploys are manifest-tag bumps +
  deliberate ArgoCD sync, never `kubectl set image`.

- **2026-07-11 — RCA FIXES 1+2+4 SHIPPED + Q1c GREEN (agent v0.84.0 + controller v0.117.0 LIVE on
  demo).** Reboot survival: automatic since agent 0.84.0, live-proven 2026-07-11 21:20 (`pct reboot
  9201` under operator GO — the hook re-armed + verified nas-media DURING the start task, CWA
  auto-started healthy, health ok first cycle, no badge). Agent: `ReassertNetworkMounts` (hook
  post-start + startup; stop+enable-now on existing sudoers verbs — NO sudoers widening needed;
  never in periodic paths). Controller: statfs consuming-namespace classifier → probe
  `not_network_fs` (stub can never verify), deploy-time stub refusal (409; idle autofs deploys
  fine), distinct stub badge (wins over unreachable; unreachable line byte-identical), deploy view
  selects stored HDD_PATH (RCA fix 4). MinAgent unchanged 0.81.0. All red-proofs recorded in repo
  REPORTs. Feature doc: `documentation/controller/network-storage-nas.md` §Reboot survival; audit §7
  follow-up added. Queued: RCA fixes 3/5/6; Peti publish train now carries 0.84 + 0.117.

- **2026-07-11 — NAS/CWA RCA (diagnostic, no fixes):** `documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md`.
  The nas-media re-add verified legitimately (new host automounts DO propagate into running
  guests); a second `pct reboot 9201` at 15:39 severed the trigger → guest-side stub; CWA
  deployed correctly to the NAS (app.yaml + binds + data all nas-media) but is DOWN since 15:39
  (docker can't recreate the bind source; perms luckily blocked any system-drive writes; zero data
  loss). UI "felhom-usb" = display bug (deploy.html selects by IsDefault, never stored HDD_PATH);
  hub device-ID WARN was CORRECT and the only guard that fired; NAS badge masks because it asks
  the agent (host ns). 6 prioritized fix tasks in the audit §5. **Remediation §6 PERFORMED on
  operator GO (~20:14–20:20 CEST): automount-unit restart re-planted the trigger in the running
  guest instantly (fix-1 mechanism live-proven), CWA restarted healthy on the NAS library,
  uid-1000 write OK, controller health back to status=ok at 20:18. Demo fully healed.**

- **2026-07-11 — OBSERVABILITY PASS (hub v0.46.0 LIVE Synced/Healthy; agent v0.83.0 + controller
  v0.116.1 on demo).** Remote pull-only log access to both box components' always-DEBUG capture
  rings. Hub: `log_bundle_requests`/`log_bundles` store (gzip, newest-3, **72 h TTL** on the 60 s
  sweep), the **secret gate fail-closed** before storing (`blocked: possible secret`; `[REDACTED]`/
  checksums pass), report-ACK `controller_log_requested` + heartbeat-envelope `log_tail_requested`
  channels (consume-once; pre-0.83 agents stay visibly pending), host-detail Diagnostics section
  (Request controller/agent logs + View/Download; the hosts read-only invariant amended: these two
  forms are the only actions). Conventions codified:
  `documentation/runbooks/logging-conventions.md` + CLAUDE.md one-liners in all three repos.
  **OPEN: operator clicks both request buttons on hub.felhom.eu/hosts/<demo> to close the live
  round-trip** (UI password-gated; the flow is fully API-test-proven). Hub bearer key rotation
  still DUE (pre-existing).

- **2026-07-11 — PUBLISH-TRAIN RULES CODIFIED** (`documentation/runbooks/publish-train-rules.md`):
  1. manifest before floor; 2. the manifest screen carries the LIVE DB floor
  (`hub_settings.min_controller_version` overrides env, acts immediately) — save the floor field
  LAST; 3. MinAgent gate — **now HUB-ENFORCED PER-BOX (hub v0.45.0, `store.ResolveManagedFloor`),
  the manual fleet check retired**; 4. box-level backstop = the controller's agent-capability gate
  (v0.114.0 route probe → v0.115.0 version compare). (Floor-UI separation + per-box gating SHIPPED
  in hub v0.45.0 — see the backlog-clear entry above; this rules-doc note supersedes the original
  "hub untouched / option-3 deferred" wording.)

- **2026-07-11 — NAS/coupling backlog CLEARED (hub v0.45.0 LIVE + agent v0.82.0/ctrl v0.115.0 on demo).**
  Composite A–F. Hub v0.45.0 (Synced/Healthy): floor-UI separation + effective-floor source line
  (DB `hub_settings` vs env, `store.ResolveGlobalFloor`) + type-to-confirm w/ live below-floor count;
  per-box MinAgent conditional floor (`store.ResolveManagedFloor` HOLDS the controller floor for a box
  whose `hosts.agent_version` < the manifest's MinAgent, flags it on the Hosts dashboard) — manual
  fleet check retired (publish-train-rules.md rule 3). THE one comparator extracted to
  `internal/semver`. Agent v0.82.0 = `X-Felhom-Agent-Version` header (demo-only, Peti stays 0.81 =
  probe fallback); ctrl v0.115.0 = version-aware Supports + DSM-validated NFS guidance. **DSM spike**
  (real DSM 7.2 via virtual-dsm): consumer recipes validated E2E, hardlink caveat closed for Synology
  (record: documentation/audits/SPIKE-nas-dsm-2026-07-11.md); QNAP caveat stands. **Q1c (supervised
  reboot) FAILED**: a NAS automount trigger does NOT survive a guest reboot (empty-dir; agent lacks a
  network-mount reassert) — fix spec'd at documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md.
  NOT published (agent 0.82 demo-only). Record: felhom.eu/REPORT.md + each repo's REPORT.

- **2026-07-11 — PUBLISH TRAIN 0.81/0.113 COMPLETE — PETI LIVE on agent 0.81.0 + controller 0.113.0.**
  Record: `documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md`. Agent 0.81.0 published
  (sha `914bb3b5…30df0b`, live-bytes, anon-verified) + golden 0.113.0 baked/published (sha
  `66de5e64…82054c`, drill VM, virgin-reverted); manifest + signed `agent_update`
  (key-id `felhom-op-1`) by Viktor → Peti 0.79.0→0.81.0 clean (no rollback); **floor auto-confirm
  proof CAPTURED** (10:41 0.112.0 → 10:42 0.113.0, anonymous pull proven on HIS box) — KEY LEARNING:
  the hub UI manifest screen ALSO sets the GLOBAL floor as a DB `hub_settings.min_controller_version`
  row that OVERRIDES the env and acts immediately (this is how Peti reached 0.112 and now 0.113
  before the GitOps env bump `16390d0`); a transient forbidden skew (ctrl 0.113 + agent 0.79,
  ~10 min) was benign only because zero NAS shares existed — next train: save the floor field LAST.
  OPEN: Peti journal-group one-liner (E), temp Git Sync creds deletion (G1), **hub bearer key
  rotation now DUE** (committed in manifests/hub.yaml + screenshot-exposed this session).

- **2026-07-11 — NAS verify-before-commit SHIPPED (agent v0.81.0 + controller v0.113.0 LIVE on demo;
  host-install v1.13.0).** The spike's production task landed same-day: add = agent verify (retry=0
  unit, journal-classified, auto-rollback) → controller uid-1000 write probe (--netprobe re-exec) →
  register LAST; full rollback on any failure; §3.2 Hungarian error map (nfs_export merged);
  storage_network.html redesigned (SMB-first, Route-A guidance); orphan rows; systemd-journal group
  (NO sudoers). Live A–E validated vs an isolated sim NAS, Route A proven in production (alien-uid
  1060 server-side). Feature doc: `documentation/controller/network-storage-nas.md`. NOT published
  (no Gitea/Day-0 bump; Peti pending with the usermod one-liner). Record:
  `felhom-controller/REPORT.md`.

- **2026-07-11 — SPIKE: NAS verify mechanisms + NFS squash matrix — ALL FIVE QUESTIONS ANSWERED, no
  blocker.** Record: `documentation/audits/SPIKE-nas-verify-2026-07-11.md`. Q1 in-guest access DOES
  wake the host automount (make-or-break PASS; container bind-mount too); Q2 controller drops to uid
  1000 and writes (setpriv proof; SETUID/SETGID caps present); Q3 **Route A ACCEPTED-with-caveats**
  (`all_squash` to ANY server uid = full rw+hardlink for the guest app; guest sees nobody; chown
  EPERM-clean, chmod persists; June's `anonuid=1000` WRONG case pinned = anonuid without all_squash
  is a no-op for non-root); Q4 error taxonomy captured (NFSv4 makes no-export vs not-permitted
  INDISTINGUISHABLE; systemd 90 s start-timeout is the real ceiling; **`retry=0` → production unit**:
  dead-NAS access 91 s→3.8 s; verify must NOT ride the 15 s agentapi timeout → own path/status-poll,
  ~120 s budget); Q5 plain-user SMB PASS incl. hardlink. Teardown verified to pre-counts. Production
  TASK is the operator side's next move.

- **2026-07-11 — PBS DR tier SLICE 2 SHIPPED (agent v0.80.0): the apply-bridge — demo
  ADOPTION-PROVEN live + PUBLISHED; the supervised Peti phase PENDING.** Full chain fired on the
  demo: hub enable (storage-id `felhom-offsite`) → tenantsync provisioned ns `demo-felhom` on ep0
  → agent ADOPTED the manual entry (tenancy entry-owned, "entry wins"; staged secret asserted
  UNCONSUMED; DR green; `escrow.pbs_storage_id` seeded). Laws red-proof-verified: set-only (K
  destruction guard), secret-on-stdin, verify-pin-before-consume, loud consumed-failed. Part 0
  evidence: write path needs the §4b dual-grant exactly (403 `Datastore.AllocateSpace` → re-assert
  → token vzdump OK); demo grants restored. Live finding fixed: escrow seed = in-place write
  (/etc/felhom-agent dir is root-owned; rename impossible for the non-root agent). 0.80.0
  published (`f2ba62ca…f06d2`, == deployed bytes). **NEXT = RUNBOOK-peti-pbsdr-2026-07-11.md**
  (Peti box prep: sudoers+wrapper+wg_tunnel enable — no tunnel/OOB on his box today; Viktor opsign
  0.80.0; hub enable → FRESH path, K born, degradation clears; ceremony → auto-confirm → first
  offsite run = the arc closes). Detail: felhom-agent/REPORT.md + CHANGELOG v0.80.0.

- **2026-07-10 — PBS DR tier SLICE 1 SHIPPED+LIVE (hub v0.44.0): ep0 tenantsync surface + hub
  provisioning.** `felhom-tenantsync` v1.0.0 installed on ep0 (second forced-command key, peersync
  untouched — hashes asserted; smoke caught + fixed the client `{"data":[…]}` json wrap; orphan
  `root@pam!spike` cleaned). Hub: `internal/tenantsync` (pinned-SSH twin of wgsync; errors never
  embed stdout), config-form "PBS DR tier (ep0)" section → WG-peer check (fail-closed) → provision →
  `host_pbs_secrets` consume-once (agent fetches `POST /api/v1/hosts/{id}/pbs/consume-token` —
  note: /hosts/, not the task's /host/) → `pbs_dr` descriptor into HOST desired_json + gen bump;
  idempotent re-save; Re-issue button (F4 twin). Red-proofs: consume-once / fail-closed /
  idempotency all mutation-proven. **NOT live-provisioned — slice 2 (agent bridge) + supervised
  Peti enable pending.** Runbook §10 addendum in offsite-endpoint.md. Commits ce6a566 + 32ca1f6.

- **2026-07-10 — SPIKE: PBS DR tier auto-provisioning SLICE 0 — all three mechanisms SETTLED.** Record:
  `documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`. (1) PVE token storage-create **impossible**
  (create/modify/delete all check `/storage` root) → **sudoers vector proven** (pinned-id alias, cycle green as
  the agent user, unpinned denied; secret must ride a stdin-wrapper, never sudo argv). (2) ep0 tenancy op-set +
  one-time-secret-over-SSH-stdout **proven live on ep0** (🛑 Viktor-approved; read/write cross-tenant 403s);
  current peersync channel can't carry it → slice 1 ships a second forced-command surface (`felhom-tenantsync`).
  (3) `pvesm add pbs --encryption-key autogen` births K at `PBSEncKeyPath` + vzdump lands ciphertext (doc-06
  property) + escrow-create finds K. **⚠ `pvesm remove` DELETES K** → re-apply must be set-only. All spike
  artifacts torn down; flags: orphan `root@pam!spike` on ep0 scratch; demo's §4b step-3 storage grants absent.
  Epic slice map in the record §5.

- **2026-07-10 — new-box restore round-trip VALIDATED + old-box archive RETIRED.** Record:
  `documentation/audits/VALIDATION-newbox-restore-2026-07-10.md`. Both toggled apps restored from the
  production pool box via the real endpoint — **9/9 sha256-identical** (immich 44MB db-dump exact,
  `d78466fcbf…4526b`, agreeing with the campaign's independent round). ~10s/app. **Viktor's decision:
  RETIRE the old archive** — he deletes `u629193-sub1` in the old project's console (the PBS-storage-1 box
  itself stays for dev). C2 fixed same day as controller v0.110.0 (crash-lock self-heal, live-proven).
  Offsite arc: build-complete + restore-proven both directions; remaining = the Peti onboarding runbook.

- **2026-07-10 — offsite arc overnight soak (unattended) — PASS + 1 HIGH finding.** Record:
  `documentation/audits/CAMPAIGN-offsite-overnight-2026-07-10.md`. F4 re-issue proved LIVE (reset invalidates
  the old password); restore round-trip 9/9 byte-identical new-box↔USB (immich db-dump exact → old-box
  archive `u629193*` safe to retire, Viktor's decision); single-flight + crash-restart (zero consume, escrow
  untouched) PASS. **FINDING C2 (HIGH, OPEN):** a controller crash MID-PRUNE leaves a stale restic exclusive
  lock the controller can't self-heal → offbox runs fail until manual `restic unlock` (fix candidate:
  `ensureOffboxRepo` auto-unlocks a self-owned stale lock). Track B (live quota-crossing) + hub-orchestrated
  Track A blocked — the hub operator config surface is session/CSRF-gated (unattended can't reach it) → Peti
  runbook. Demo restored to start-state. One expected `backup_failed` email (the crash test).

- **2026-07-10 — SLICE 4 soft-quota + offsite health SHIPPED+LIVE (hub v0.41.0 + controller v0.109.1) —
  the offsite arc is BUILD-COMPLETE.** Controller: `quota_gb` rides the descriptor into the target (hash
  includes it → quota changes re-apply via key-auth-first, no consume); `repo_size_bytes` from restic
  stats; pre-run gate (≥100% refuses NEW backups Hungarian-loudly, **prune still runs** — red-proofed;
  restore never gated; ≥80% warns); /backups usage bar. Hub: `monitor.OffsiteChecker` (fill 90/95 of
  quota + `offsite_stale` >48h for enabled+escrowed only — red-proofed; nil-safe on old reports) +
  **manual-only Freeze/Unfreeze lever** (readonly on the exactly-1 labelled sub-account, SSH preserved).
  **Live:** key-auth-first proven against the real box (the deploy's own re-apply); that re-apply exposed
  + fixed v0.109.1 (re-apply must preserve EscrowState + runtime status — it demoted the escrowed demo);
  demo repaired (manual confirm + run: escrowed, 45 MB / 50 GB, checker healthy 0 events, usage bar
  renders). Freeze + 100%-refusal NOT live-exercised (STOP; red-proofs cover). NEXT: Peti onboarding
  runbook, NEW-box restore round-trip → old-box retirement.

- **2026-07-09 — SLICE 3 hub-verified escrow auto-confirm SHIPPED+LIVE (hub v0.40.0 + controller v0.108.0 +
  agent v0.79.0).** Replaces the operator-trust confirm with a verified fact: ceremony hashes the sealed
  repo password (agent, `restic_pw_sha256,omitempty` on the escrow upload — contract test moved in lockstep)
  → hub stores it (additive `host_escrow` migration, NULL on legacy rows) + serves
  `escrow:{identity_blob_present,restic_pw_sha256,created_at}` in the report ACK → controller
  (`report.EscrowAutoConfirmer`) flips pending→escrowed ONLY on hash match with its local repo password.
  **Red-proofed core: blob-presence alone never confirms** (stale blob = false custody claim); mismatch
  warns once per hash naming the ceremony; never un-confirms; flip wipes the agent-staged secret. Hashers
  pinned by the same cross-repo vector. Manual confirm = deprecated fallback (legacy blobs — demo included,
  hash-NULL verified live; the escrowed demo produced zero confirm activity = Scenario E live). Full happy
  chain proves out at the next real enable (Peti). NEXT: SLICE 4 soft-quota, Peti runbook.

- **2026-07-09 — offsite hardening bundle SHIPPED+LIVE (hub v0.39.0 + controller v0.107.0 + agent v0.78.0).**
  The e2e's sharp edges closed before slices 3–4: **F4** "Re-issue offsite credentials" (explicit operator
  recovery for consumed-password dead-ends — exactly-1 label scope, fresh one-time secret, ConfigVersion
  bump; `hetznerapi.ResetBoxPassword` added for dedicated); **F2** host-key scan retry (~60s ladder — first
  save survives fresh-subaccount DNS lag); **F5** submit-disable + in-flight notice; **key-auth-first bridge**
  (working installed key → re-pin+reconfigure WITHOUT consuming; fingerprint verify never bypassed; kills the
  stale-descriptor consume-404 loop); **staged-secret wipe** (agent `DELETE /escrow/stage-secret`; controller
  calls it on every flip to `escrowed`; the e2e's lingering staged file wiped live through the new endpoint,
  idempotency proven). All red-proofed. F4 reset NOT exercised live (demo credential untouchable — throwaway
  customer under supervision when needed). NEXT: SLICE 3 auto-confirm, SLICE 4 soft-quota, Peti runbook.

- **2026-07-09 — offsite provisioning VALIDATED LIVE end-to-end (hub v0.38.1 + controller v0.106.1) — PASS.**
  Full loop through the real pipeline on `demo-felhom` against the new dedicated-project pool box (611714 /
  `u629488`, project 15282031): UI save → sub-account 269008 provisioned → descriptor v6 + one-time password →
  controller config-refresh (≤15 min) → apply-bridge (verify-pin → consume-once → pinned `ssh-copy-id` →
  `EscrowState=pending`) → run-gate refusal proven → escrow confirm (Option A: the repo password was already
  escrowed — `WriteOffboxSecrets` preserves it) → first run: **restic repo + 2 snapshots on the new box,
  key-only**. Two live bugs found+fixed mid-run: **F1** hub v0.38.1 (`applyOffsite` on `context.WithoutCancel` —
  a re-click canceled the provision between create and password-store, stranding the sub-account) and **F3**
  controller v0.106.1 (`ssh-copy-id -s` needs `~/.ssh` to exist in the container — died AFTER consuming the
  password). OPEN: F2 (fresh sub-account DNS lags → first-save scan 502s; retry-with-backoff), F4 (hub has no
  password-reset path — recovery today is delete-subaccount + re-save), F5 (no save spinner → re-click bait).
  Phase-0 wiring: `manifests/hub.yaml` now injects `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID` from
  Secret/storagebox; scoped-token check = list boxes with it (must see ONLY 611714). Old-box offsite kept as
  frozen archive until a NEW-box restore round-trip. Record:
  `documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`. NEXT: SLICE 3 (auto-confirm),
  SLICE 4 (soft-quota), F2/F4/F5, Peti onboarding runbook.

- **2026-07-09 — offsite provisioning SLICE 2 (hub v0.38.0 + controller v0.106.0).** The controller apply-bridge:
  on startup it reconciles the hub-served `offsite:` descriptor into a key-only offbox target
  (`controller/internal/offsiteapply.Bridge`) — **verify-pin the box host key against the hub-captured
  `host_fingerprint` (no blind TOFU)** → consume the one-time password (single-use) → `sshpass ssh-copy-id -s -f`
  install → configure offbox → `EscrowState="pending"` → persist a descriptor-hash marker. Idempotent +
  fail-safe; both red-proofs green. Hub v0.38.0 adds `Descriptor.HostFingerprint` captured via an
  `x/crypto/ssh` keyscan (fail-closed). **NOT yet live-applied** — supervised end-to-end (hub provisions on
  the new pool box → controller apply) is the next runbook, gated on the hub's new scoped `HETZNER_TOKEN`.
  NEXT: SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota).

- **2026-07-09 — hub offsite provisioning SLICE 1 (hub v0.37.0).** The hub can now provision the offsite tier
  on operator enable: `internal/hetznerapi` (typed client, base **api.hetzner.com/v1**, `CloudAPI` interface +
  exported `Fake`, `WaitAction`), `internal/offsite` (`Provisioner.ProvisionOffsite` — idempotent by label
  `felhom-customer=<id>`, shared sub-account / dedicated box, generates the transient password, builds the
  NON-SECRET descriptor, **fail-closed**), `one_time_secrets` store (single-use `Save/ConsumeOneTimeSecret`),
  `POST /offsite/consume-password/{id}` (customer-API-key auth, once), config-form Offsite section →
  `applyOffsite` (502 + no-save on error) → descriptor in ConfigJSON → ConfigVersion bump. Token from
  `HETZNER_TOKEN` (out-of-band); password NEVER in ConfigJSON/logs. Unit-tested vs a faked Cloud API (+ a
  fail-closed red-proof). **NOT yet live-provisioned** — awaiting the **dedicated-project scoped token** (the
  current token can delete ep0). NEXT: SLICE 2 (controller apply-bridge consuming the descriptor + one-time
  password), SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota). Deployed via GitOps.

- **2026-07-09 — Hetzner API provisioning spike (offsite tier).** Both offsite models are API-provisionable
  (FIT). **#1 correction: the Storage Box API is `https://api.hetzner.com/v1`, NOT `api.hetzner.cloud/v1`**
  (Cloud API 404s for storage boxes); same Bearer token. Proven live on `spike-*` (torn down): sub-account
  create/reset/readonly/delete + a billable bx11 box create/rescale(`change_type`)/delete. Create takes a
  **password (no ssh-key param)** → transient-password→`ssh-copy-id -s -f`→restic-reach→`reset_subaccount_password`
  (transient pw invalidated, key survives) = key-only steady state. RepoPath `/home/<repo>` (chroot). Quota:
  dedicated=box type/`change_type` (hard); shared=no native quota → Felhom soft-quota + `update_access_settings{readonly}`
  freeze. Idempotency = **label_selector** (box names not unique). Rate 3600/h. **SECURITY: the token is
  project-wide unscopeable Read+Write — it can also touch ep0 (the prod offsite server) + 3 SSH keys →
  move customer boxes to a DEDICATED Hetzner project before the hub holds a write token.** Record:
  `documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md`. NEXT: the hub-provisioning impl.

- **2026-07-09 — fork-4 custody VALIDATED end-to-end live (supervised).** The full custody cycle proven on
  the demo (agent v0.77.0 + controller v0.105.0): enable → controller pushes the repo password to the agent →
  **real escrow ceremony** (`--selftest=escrow-create`, K + restic-pw under a fresh R, uploaded to the hub,
  staged secret wiped, K byte-unchanged) → **atomicity proof:** a run **skipped** before `confirm-escrow` and
  **proceeded** after (2 apps, 2 snapshots) — the gate opens exactly on confirm. **DR money shot:** the restic
  password recovered from the **production hub blob** (`host_escrow.identity_blob` for `demo-felhom-01`, via
  `identity-consume` with R) is byte-identical to the live password (`sha 24c44f49…`) and **decrypted the real
  box repo**; the real `inject-password`→`restore` endpoints reproduced a 45 MB db-dump byte-perfect. Demo
  offbox left **escrowed** (un-paused); Viktor holds R out-of-band; no secret in any log; live `repo_password`
  unchanged. Record: `documentation/audits/VALIDATION-fork4-custody-2026-07-09.md`. **NEXT:** hub-provisioning +
  soft-quota spec (hub-verified auto-confirm replacing operator-trust; both storage models); per-customer
  onboarding runbook; the `identity-consume` OK-message fix (omits the recovered restic field).

- **2026-07-09 — offbox v0.104.0 VALIDATED live against the Storage Box (supervised).** Closed the offsite
  arc: controller `:0.104.0`@`908e4b9` ran the real `/backup/offbox/*` endpoints against a fresh box
  **sub-account** (`u629193-sub1`, RepoPath `/home/felhom-demo-repo`). The capstone proof — **immich**
  (undeployed, recovery unit only on the USB drive, the exact case v0.103.0 mis-resolved to `systemDataPath`)
  → discovery found the USB unit → `ok`, 1 snapshot; 45 MB db-dump restore sha256-matched. No-silent-success
  confirmed on the live box: 0/N → `error` + `backup_failed` operator alert; partial → `ok` +
  Hungarian `LastWarning` on `/backups`. Persists across restart. **Forks:** (1) direct topology + (2)
  sub-account-per-customer (RepoPath `/home/<name>`, `ssh-copy-id -p 23 -s -f` — console SSH-key field still
  broken, F3) confirmed; (3) PBS/WG tier deferred; **(4) custody surface RECORDED, escrow design OPEN** —
  the per-customer restic repo password + SSH key are 0600 in the controller data dir, **NOT independently
  recoverable** (ride DR only via the PBS whole-CT) → the load-bearing follow-up before real reliance.
  **KEPT** as a standing demo-offsite (€4.06/mo). Record: `documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md`.

- **2026-07-09 — TASK GL-7 SHIPPED: customer-page passphrase hardening + install-command generator
  (hub v0.36.0, `844fbfa7`; deploy `02c748eb`; LIVE, ArgoCD Synced/Healthy)** — go-live G7/G9/G10/G11.
  Two coupled, security-first changes to the operator customer page (`customer_unified.html` +
  `configs.go`). **(1) Passphrase hardening:** the per-customer retrieval passphrase was cleartext
  twice (visible `#retrieval-pw` + baked into the Option-3 debug curl `X-Retrieval-Password:` header);
  now masked-by-default with reveal (`toggleSecret`) + copy (`copySecret`), value in `data-secret`
  (existing model), and the Option-3 command carries a `<YOUR-RETRIEVAL-PASSWORD>` placeholder — no
  secret in any copyable command. (Zero-secret-in-DOM fetch = noted future follow-up, NOT scoped.)
  **(2) Install-command generator:** the three hard-coded `<code>` blocks became a client-side builder
  (vanilla JS, no framework/CDN/network) emitting ONLY real host-install v1.12.0 flags, download-then-
  run, byo-requires-caps enforced client-side, CustomerID prefilled from the server, JS-off static
  fallback; the seven dangerous/operator-only flags (`--force`/`--rotate-recovery`/`--enable-oob`/
  `--remove-golden`/`--uninstall`/`--adopt-pool`/`--rescope-acl`) are never offered as controls. Tests
  `TestTemplates_PassphraseHardened` + `TestTemplates_InstallGenerator` (green; passphrase test
  red-proofs). **Tester agreement written** (`documentation/pilot/PETI-tester-agreement.md`): BYO
  trust boundary, honest limitations (no offsite backup, F2 removable-drive, F7 pool-reassert, F6
  `:53`-free), exit rights, Peti box facts (80c/128 GB → caps 12/32768, `sajatfelhom.hu` re-point),
  byo install command, onboarding sequence. GO-LIVE-PACKAGE: GL-7 ✔; G9/G10/G11 → done. **One open
  pilot question (owned by the agreement §6):** Peti's local backup TARGET — does his box have a
  second disk/pool? → sets `--acl-storages` at onboarding. Deploy gotcha: first build was stale
  (local commits not pushed before `build.sh` pulls origin) — pushed then rebuilt. **NEXT: Peti day-0
  onboarding (operator-run).**
- **2026-07-08 — TASK GL-8 SHIPPED: BYO coexistence hardening (host-install v1.12.0, `18a556a5`)** —
  the GL-6 findings F3/F6/F1. **F3:** the populated-host leaf guard fires only when a `felhom_guests`
  guest exists (not `pct list`) → a fresh byo install on a host running only the OWNER's guests needs
  no `--allow-new-leaf`; the pin protection for real Felhom guests is unchanged. **F6:** byo preflight
  REFUSES a foreign process on `:53` with remediation and mutates NOTHING (Viktor's ruling: refuse+
  instruct only — the dnsmasq analog of the break-glass no-touch rule; appliance untouched). **F1:**
  uninstall purges the config `.bak*` siblings (a GL-6 residue held a live hub api_key). F6
  live-proven on felhom-pve (byo preflight refuses at :53 — the exact GL-6 scenario, read-only die).
  Harness 36/36 (+GL8-F3/F6/F1 static, GL8-F1b behavioural, GL8-F6 live) + red-proofs RP-F3/F6/F1;
  GL-2/GL-4 regression green; shellcheck clean. Shipped BEFORE GL-7 so Peti's command needs no
  workaround. **F2/F7 remain open (non-blocking). NEXT: GL-7.**
- **2026-07-08 — TASK GL-5 SHIPPED: DR bring-up bind overrides (agent v0.75.0, LIVE + scratch-DR
  validated)** — go-live G8; docs-only here (GO-LIVE-PACKAGE G8/G6/G1 flips + decision log; the
  code + full report live in felhom-agent `b3446213`). Guest-loss DR restore WORKS now: the two
  structural binds (mp8/mp9) are overridden with throwaways at restore and swapped for the REAL
  binds pre-start (root pct via the new Engine HostRunner seam), displaced unusedN deleted. TWO
  live-discovered PVE rules beyond the spike: explicit-params restore needs an explicit rootfs AND
  silently DROPS unlisted mountpoints — the full param set now derives from the archive's embedded
  config (`ExtractArchiveConfig`, 200 under the scoped token). Live proof: scratch 9310 from a real
  9201 archive — mp0 200G + mp1 50G restored WITH content (7m23s), real binds, zero residue, clean
  teardown; the same op failed outright on v0.74.0. OPERATOR: **publish agent 0.75.0 + bump the
  manifest to it** (vouching 0.74.0 ships broken DR). Follow-ups: restore-test boot-verifies
  without mp0/mp1 (same PVE rule, reverse direction — candidate reuse of drRestoreOverrides);
  DR-selftest KeepMAC duplicates a live source's MAC briefly; no host-loss DR runbook exists yet
  (GL-6/S5 writes it).
- **2026-07-08 — TASK GL-4 SHIPPED: uninstaller gap-closure + operator-key day-0 fold
  (host-install v1.11.0)** — go-live G6 + the G1 key-pin follow-up; **awaiting GL-6** for the real
  (non-dry) teardown + armed-pin install. Uninstall now removes the self-update artifacts (4b4:
  guarded wrapper, .prev/.new.* slots, rollback unit, limits drop-in — derived from
  configs/felhom-selfupdate-guarded), unmounts enrolled drives under /mnt/felhom-drives (plain
  umount ONLY, busy = warn+guidance, root-bind guarded), and ends with a KEPT-vs-WIPED statement
  (drives/PBS/hub record/escrow/vaulted recovery credential live on; guest-only mode prints the
  vmid's bind-store drives). Key-pin: OPERATOR_KEY_* constants (EMPTY until the operator ceremony)
  + `--operator-pubkey-file` (validated at argv, comment=key_id required) → authz.signers written
  at step 6 per the agent SignerKey schema; **no-keys-resolved KEEPS preserved signers** (never
  un-pin a manually-pinned box); verify reports armed/dormant (dormant = WARN). Harness 28/28 on
  felhom-pve incl. the NEW GL4 H-U full-uninstall DRY transcript vs live 9201; red-proofs RP-1..3
  green. **GO-LIVE-PACKAGE.md was ABSENT AGAIN** (spec said the operator attaches it; not present
  in the repo or workspace) — G6/G1 status recorded here, doc still pending its operator commit.
  Key CEREMONY (real keypairs + felhom-pve pin) = operator step.
- **2026-07-07 — RUNBOOK GL-1 EXECUTED: agent 0.74.0 + golden 0.103.0 PUBLISHED** — go-live G1
  (partial): the published-artifact chain is current. Agent = the LIVE felhom-pve bytes (sha
  `1ec3f588…76af05`, provenance preserved, publish-agent.sh round-trip verified); golden baked
  clean-room in the drill VM (build-golden.sh v2.0.0 @ ced60ddf, controller 0.103.0 baked, all
  integrity gates green, sha `8481e8a1…6026e`, 583MB) and published HTTP 201. Both artifacts
  fetch-verified **ANONYMOUSLY** (Gate 3c finding: generic packages are world-readable → G3
  git.token read-only rotation CANNOT break fetches; but the INSTALLER dies on an empty token —
  script-side requirement, not Gitea's). Drill VM restored to `virgin`; evidence
  `/mnt/5_hdd/felhom.eu/drill/bake-0.103.0.log` on 180 + `documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`.
  **OPERATOR NEXT: bump the hub Day-0 manifest** to agent 0.74.0 / golden 0.103.0 (the four values
  are in the execution record) — until then fresh installs land 0.63.0/0.98.3. Also recorded:
  operator-key pin follow-up (fold into GL-4); GO-LIVE-PACKAGE.md still not in the repo (G1 status
  recorded here instead).
- **2026-07-07 — TASK GL-2 SHIPPED: `--mode appliance|byo` install profile (host-install v1.10.0 +
  harness)** — go-live items G2/G4/G5 implemented, **awaiting the supervised GL-6 drill** for live
  provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old
  `--mode provision` value retired with a specific die, `dr` stub kept); break-glass gated at its
  CALL SITE (byo never touches root@pam); byo requires `--cores`+`--memory`, refuses
  `--enable-oob`/`--rotate-recovery`/non-9.x-PVE/missing `--acl-storages`; disclosure + typed-hostname
  ack; **the config write's `lan_resolver` default flips to OFF in byo** (the spec assumed
  default-off — it was default-ON; appliance unchanged) + post-write asserts refuse
  lan_resolver/wg_tunnel/oob=true; pool+ACL verify asserts in BOTH modes (R2 lesson);
  `--preflight-only` (no state, PASS/FAIL verdict, proven live on felhom-pve). NEW
  `scripts/hostinstall-mode-harness.sh` (static C1–C4 + invariants everywhere; PVE tier: C5 +
  A/B dry transcripts) — 16/16 PASS on felhom-pve incl. C5 live; red-proofs RP-1..3 run→fail→revert.
  FINDING: demo-felhom's controller.yaml has EMPTY git.username/git.token → any step-5
  (re)install for that customer dies until the operator sets real Gitea read creds (relates to the
  standing scope-down+rotate follow-up). `documentation/pilot/GO-LIVE-PACKAGE.md` is NOT in the
  repo and no operator copy was provided — the G2/G4/G5 status flip is recorded here instead;
  add the doc when the operator supplies it. Next: **GL-6 supervised drill** (appliance + byo
  end-to-end, uninstall/re-provision, C7, the ack prompt interactively).
- **2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent
  v0.72.0)** — merged E1+H1. Hub: operator OOB peer (`store/wg_operator.go`, `PUT/GET
  /admin/wg/operator-peer` global key) + `oob_peer_ip`/`oob_operator_ssh_key` in the desired-state
  wireguard block + `monitor/host_oob` (oob_degraded/recovered). Installer: `--enable-oob`/`install_oob`
  (static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box
  drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as
  felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt
  drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key.
- **2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent
  v0.71.0)** — prereq for felhom-sshd (H1). Hub: `store.host_recovery` vault (per-host root@pam
  console password; `PUT /hosts/{id}/recovery-credential` self-scoped + `GET /admin/hosts/{id}/
  recovery-credential` global-only, secret never logged) + `host_mgmtplane` monitor raising
  `mgmt_plane_healed`. Installer: `step_break_glass` (generate+set+vault root@pam, `--rotate-recovery`)
  + `install_mgmt_watchdog` (the 3 host artifacts, RuntimeDirectory-guarded). Live-validated:
  agent-independent `/run/sshd` auto-heal in 30.0s; warning fired end-to-end; break-glass
  vault→retrieve→PVE-ticket(200). felhom-pve root@pam is now the vaulted value.
- **2026-07-04 — S3 SHIPPED: agent WG tunnel (felhom-agent v0.64.0; docs-only here)** — the
  doc-06 §3.3 handshake is now END-TO-END LIVE on demo: agent keygen → `POST /hosts/{id}/wg` →
  desired-state `wireguard` block → agent-managed `wg-quick@wg-felhom` → PBS page over
  10.77.0.1:8007 in 3 s. **`wg_tunnel.enabled` defaults FALSE agent-side** (rollouts to other
  boxes are no-ops until the production endpoint exists). Revocation semantics completed (§3.5):
  removed peer → agent disables + keeps its marker + NEVER re-registers; operator re-adds via the
  pubkey the heartbeat report now carries. Escrow: `IdentityBundle.WGPrivateKey` auto-injected on
  new escrow-create runs (pre-S3 blobs can't be retrofitted — honest fallback = fresh-key
  re-register, /32 kept). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix, deferred
  non-blocking); S4 = PBS over the tunnel; hub-side S6 = tunnel-health alerting on the new report
  stanza. Detail: felhom-agent CHANGELOG v0.64.0 + REPORT.
- **2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0)** —
  doc-06 row S2. DECISIONS: the WG assignment is **hub-owned state, merged into served
  desired-state at READ time** — stored `desired_json` stays a pure operator blob; admin PUT with
  a top-level `wireguard` key → 400 (no copy-paste clobber); **re-key-in-place keeps the /32**
  (stable tunnel addressing across rotation/DR); one bound peer per host (partial unique index);
  generation bumps ONLY on real change (register/re-key/adopt/bound-delete — idempotent paths
  move nothing). `POST /hosts/{id}/wg` self-scoped per-host key; NEW golden
  `desired-state-wireguard.golden.json` = the S3 cross-repo contract (byte-identical agent copy
  rule). peersync **v1.0.1** (strip out of process substitution — the S1 exit-swallow finding;
  sandbox red-proofed) redeployed to the dev endpoint. Live-proven on the REAL demo-felhom-01
  record via the real per-host key: gen 2→3, served block = golden shape, endpoint `wg show`
  carried the peer, v0.63.0 agent tolerated the unknown key (fetch clean, guests untouched),
  then removed → gen 4, all baselines restored. `/offsite` page unit-proven (live view is
  operator-password-gated — eyeball after next login). Next: **S3** (agent keygen +
  registration + `wg-felhom` service + escrow join + CGNAT smoke — risky/supervised).
- **2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1)** — the doc-06
  roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: **hub pushes
  over SSH** to a forced-command `felhom-peersync` script (pull rejected: weakens immediate
  revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev
  `felhom-hetzner` box, provisioned by the NEW runbook `documentation/runbooks/offsite-endpoint.md`
  (WG on **443/udp**, subnet 10.77.0.0/24, PBS datastore `felhom-offsite` at `/mnt/pbs-datastore`
  — a 100 GiB Hetzner Cloud Volume attached 2026-07-27 06:29:40 UTC; the pre-migration copy at
  `/srv/pbs-felhom` is 13 G of dead weight on `/` awaiting R-91's go-ahead —
  nftables drop-all + wg0-only 8007, `ep0.felhom.eu` grey-cloud). Hub: `wg_endpoints`/`wg_peers` +
  /32 allocator + `/admin/wg/*` (global-key, pubkey-in-body) + `internal/wgsync` (FixedHostKey pin
  + **pinned-type HostKeyAlgorithms — live-caught bug**: multi-hostkey sshd presented ECDSA vs the
  ed25519 pin; fixed v0.32.1 + regression test) + full-list declarative reconciler (5-min drift
  repair). Live: add/remove→wg show, malformed→state-untouched, reboot→persist+converge, client
  tunnel via ep0:443→PBS login page, public 8007 closed. INCIDENT handled: first WG server key
  leaked via `wg show dump` into the session log → rotated immediately (warning now in the
  runbook). OPERATOR follow-up: **fix the AAAA record** (currently `…7aa1::`, must be `…7aa1::1`).
  Next: **S2** (hub schema host-join + `wireguard` block on WireDesiredState + box-facing
  registration + UI).
- **2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06)** —
  `documentation/architecture/06-offsite-connectivity.md`: records the six settled decisions
  (plain WG / host-side agent-managed `wg-felhom` as the agent-managed-unit PILOT / one shared
  hub-driven endpoint VM running WG+PBS, NO agent on it / hub = source of truth, `wireguard`
  block rides `WireDesiredState` on the existing DesiredGeneration channel / one datastore +
  per-customer namespaces / PBS ON the VM, relay-through-DooPlex REJECTED as non-scaling),
  the Day-0 join handshake, robustness set (WG-roaming-not-DynDNS, DNS re-resolve watchdog, MTU
  1420, per-/32 isolation, tunnel-health via storage-target model), trust conformance (outbound
  only; operator-inbound + game-ingress out of scope), the honest ledger (CGNAT still unmeasured
  — mobile-hotspot smoke closure; peer-sync push-vs-pull = slice-1 design point), and the S1–S6
  slice roadmap (**MVP = S1 endpoint + S2 hub schema/desired-state + S3 agent WG service, then S4
  PBS-over-tunnel**; S5 merges with DR-completeness). day0-install backlog line now cites spike +
  design doc. Docs-only; slices are separate future tasks.
- **2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung)** —
  plain WireGuard, host-side, box-dials-out to a small public endpoint; **no Headscale/DERP needed
  for backup** (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway
  Hetzner: 11.4-min idle hold, PBS API through the tunnel with the exact-cert pin intact
  (positive + negative), real 2 GiB worst-case backup at 4.26 MiB/s (= the home uplink; tunnel
  overhead ~5%), UDP 51820 *and* 443 both pass. P0 honesty: the operator's line is plain-NAT
  (fixed public IP, One Hungary), NOT CGNAT, and has zero IPv6 — CGNAT confirmation deferred to
  Peti's VM 110 (second ISP). P7 surprise: kernel WG *works* inside the unprivileged guest (no TUN
  needed) — host placement stands on architecture (backups originate host-side; cloudflared
  precedent), not infeasibility. Evidence + architecture recommendation (endpoint VM shapes, key
  custody via IdentityBundle escrow, per-peer /32 isolation):
  `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`. **This unblocks the
  DR-completeness work: next task = production connectivity spec** (endpoint VM, agent-managed WG
  service, hub peer registry, escrow join). Throwaway Hetzner box parked powered-off (secrets
  shredded); demo-felhom verified residue-free.
- **2026-07-03 — 4th skill: `felhom-app-catalog`** (catalog authoring workflow; points at
  app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by `install_skills.py`, 4/4
  found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected
  healthchecks, probe-container naming rule discovered + REUSE row added, UI-applied, controller
  probe healthy). App-count drift recorded: website says "45+", per-category spans sum 49, real
  catalog 53 — website fix is a separate task.
- **2026-07-03 — GOLDEN 0.98.3 LIVE + VOUCHED; guide D.1b RETIRED (drill B1+B5 FIXED)** —
  `build-golden.sh` v2.0.0 (mandatory controller tag + baked bootstrap `.path` unit); golden baked
  with controller 0.98.3, clean-room validated (drill VM) through all four scenarios incl. the
  published-artifact `--force-gitea-golden` path, published + operator-vouched (Day-0 manifest now:
  agent 0.63.0 + golden 0.98.3). Fresh installs land CURRENT and self-manage — D.1b is a one-line
  check; the prior "vouch agent 0.63.0" operator follow-up is DONE. Evidence:
  `documentation/audits/DRILL-golden-098-2026-07-03.md`. NEW operator follow-ups: (1) SECURITY —
  the customer-config `git.token` is a Gitea admin token with package-WRITE (proven during the
  bake); scope down + rotate. (2) optionally raise the global controller floor to 0.98.3.
- **2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published)** —
  token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir.
  Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. **OPERATOR: vouch
  agent 0.63.0 in the Day-0 manifest UI** (until then fresh installs fetch 0.62.0).
- **2026-07-03 — GO-LIVE PACKAGE SHIPPED (host-install v1.9.1 + day0-install runbook + clean-room drill)** —
  `documentation/runbooks/day0-install.md` is the operator Day-0 guide, **every command executed
  verbatim** on a throwaway nested PVE 9.2.2 (build server): install → verify → app deploy →
  uninstall → **zero-residue diff** → reinstall. v1.9.1 = residue-free uninstall (agent config w/
  hub api_key, shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq snippets) +
  post-provision guest reboot (without it the controller never deploys on slower hardware) + header
  sync. Evidence + go/no-go for Peti (**GO**, 3 install-day items):
  `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`. Key OQ-2 finding: golden bakes a
  pre-floor controller 0.85.1 → **fresh installs need the one-time D.1b update** (agent restart →
  update button → 0.98.3); follow-ups recorded: rebuild golden ≥0.86.0 + re-vouch (operator), agent
  `mkdir -p` for the guesthook snippet dir, fresh-install local-API 401 (pre-existing), hub has no
  host-delete path. Drill VM parked virgin-snapshotted on 180 (`/mnt/5_hdd/felhom.eu/drill/` — moved off the SSD 2026-07-18).
- **2026-07-03 — A1 CLOSED (host-install v1.9.0 + agent v0.62.0)** — `Pool.Audit` added to
  `FelhomAgentGuest`; the agent's stale-lock reaper now intersects its scan with
  `GET /pools/felhom` membership (fail-safe skip on read failure). Spike:
  `documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. Deploy order on any box:
  rescope ACL first, then the agent. STILL PENDING (operator): rotate + de-git the 3 committed
  secrets in `manifests/felhom.secret.yaml` (spike appendix; REUSE §2 errata added).
- **2026-07-03 — hub v0.31.0 LIVE** (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy;
  live probe: critical event stored + operator email delivered).
- **2026-07-03 — Felhom skills exist**: `skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}`,
  installed via `scripts/install_skills.py` — **junction mode** (repo edits live immediately; no
  re-install needed after editing). Fresh-session discovery verified.
- **2026-07-03 — CLAUDE.md refreshed** (all four repos): no version-pinned state in CLAUDE.md —
  current state lives here / CHANGELOG / REUSE.md.

- **2026-07-03 — `REUSE.md` exists at the repo root** (hub helpers / website conventions / gate scripts / manifest patterns, code-verified); maintenance rule active: update it in the same commit that changes a shared helper. `scripts/reuse_refs_check.py` machine-checks its citations across all four felhom repos.