admin
1956e5d390
The credential existed and was not reachable when it was wanted. Every box has
had a strong random root@pam password since TASK G1, vaulted in the hub at day 0
and used for real during the sshd incident — but the only way to read it back was
a hand-written curl carrying the global operator key, a secret kept out-of-band.
In practice the PVE web console on a demo box felt locked.
The host page grows a Console access card: presence + username + set_at by
default, Reveal fetches the plaintext on demand for 60 s with a Copy button.
Masking clears the JS variable, and also fires on a second click and on
visibilitychange. A host with nothing vaulted says so, and says why.
The secret is NEVER rendered into the page, and that constraint shapes the
change. The render path uses a new store.GetHostRecoveryMeta whose struct and
SELECT both omit the secret column, so it is structurally incapable of carrying
one. The plaintext crosses the wire only in the response to POST
/hosts/{id}/reveal-recovery-credential (Cache-Control: no-store, CSRF-gated at
the ServeHTTP level; POST precisely so that gate applies and so no secret is
retrievable by URL alone). Deliberately NOT the customer page's data-secret
widget, which embeds the plaintext on every load.
A delivered reveal writes one recovery_credential_revealed event on the host's
customer timeline (info, source hub, Hungarian) via SaveEvent alone — no
dispatcher, nobody emailed, the log_tail_requested shape. Two reveals write two
events: the register records accesses, not states. A 404 is not an access. An
unbound host reveals fine and writes no event; the [INFO] hub line, carrying the
username and a length only, is then the record.
The global-key API path is untouched by design — it is the route for when the
hub UI itself is broken, and coupling it to the session layer would delete the
independence that makes it a fallback.
Recorded as a real trade: the hub session password alone now unlocks console root
fleet-wide, where retrieval previously also needed the global key. Accepted for a
single-operator, HU-geo-fenced hub that already stores these passwords in
plaintext at rest (CONTEXT.md ruling S-4). The plaintext-at-rest half is filed as
R-133 — every hub DB backup is a fleet-wide console-credential dump.
Tests 550 -> 559; four red-proofs (page leak, audit event, CSRF gate, route
order) each run, observed failing, and reverted. The route-order proof is a seam
test driving ServeHTTP: a handler-level test cannot see that defect, because the
handler is correct and simply never runs.
1166 lines
104 KiB
Markdown
1166 lines
104 KiB
Markdown
# CONTEXT — felhom.eu working state
|
||
|
||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||
|
||
> **`documentation/backlog/OPEN-ITEMS.md` is the single source of truth for open work** (2026-07-27).
|
||
> Read it first; `ROADMAP.md` holds the full history and reasoning behind each item. **`REPORT.md` is
|
||
> per-session and overwritten — nothing durable may live only there.**
|
||
|
||
## Standing rulings
|
||
|
||
**S-1 — N.5 gains a third leg: architecture docs are same-session coupled (2026-07-26, R-81).**
|
||
Any task that changes an **architectural contract** — tiers, targets, cadences, trust boundaries —
|
||
updates the owning `documentation/architecture/*.md` in the **same session**, under exactly the same
|
||
coupling rule that already binds the capability map and the ROADMAP. Origin: R-80/R-81 found
|
||
`07-backup-architecture.md` still describing a backup model that the shipped code does not implement
|
||
(single target, single cadence), while being cited as authoritative. A stale architecture doc is
|
||
worse than a missing one, because it is trusted.
|
||
|
||
**S-2 — architecture docs carry an honest status header (2026-07-26, R-81).**
|
||
Every `documentation/architecture/*.md` opens with the version it was **verified against** and the
|
||
date. A doc more than a few trains behind its subject is marked **STALE** *in that header*, so a
|
||
reader meets the warning before the content, not after acting on it. Origin:
|
||
`07-backup-architecture.md` — DRAFT since 2026-07-14, verified against controller v0.132.0, now 41
|
||
versions stale (live v0.173.0), and cited as authoritative throughout the R-80 diagnostic. Ratifying
|
||
or retiring it is → **R-83**.
|
||
|
||
**S-3 — the recovery model: six decisions, 2026-07-28.** Taken in an architecture discussion and
|
||
expressed in the `07-backup-architecture.md` full rewrite (which replaces the 2026-07-14 DRAFT
|
||
entirely — that doc was verified against controller v0.132.0, **51 versions stale**, while being
|
||
cited as authoritative). They are **decisions, not observations**; the rewrite labels every statement
|
||
`[DESIGN]` or `[FACT]` so a reader can tell which is which.
|
||
|
||
- **D1 — two lanes, different owners.** *Customer, unassisted:* files and app data, through the
|
||
„Visszaállítás" surfaces. *Operator:* LXC guest and host recovery. This is a deliberate product
|
||
decision, not a limitation to apologise for.
|
||
- **D2 — encryption follows the boundary, not the tier.** Data on the customer's own drives is
|
||
**plaintext**, deliberately: someone who can take the second drive can take the first, local
|
||
encryption adds a key-loss path that turns a working backup into a brick, and it would break
|
||
FileBrowser and SMB browsing, which is a feature. Data **leaving the premises** is encrypted
|
||
(restic, PBS) — the provider must not be able to read it.
|
||
- **D3 — recovery is a chain, not a set of tiers.** Every app-tier restore is conditioned on the
|
||
whole-guest tier: Tier-1 needs the guest's secrets (`restore_unit.go:17-22`, *"Regenerate
|
||
NOTHING"*), and Tier-2/Tier-3 need the app still deployed. **3-2-1 describes copies; it does not
|
||
describe recovery.** This is the reason the rewrite exists.
|
||
- **D4 — three-part model.** *Recipe* (scaffolding, hub) · *Escrow* (identity key + restic repo
|
||
password, hub, R-wrapped) · *Bytes* (the tiers). Losing one is a different problem from losing
|
||
another, and the failure→recovery matrix is organised around that.
|
||
- **D5 — R's role narrows. CONDITIONAL, and currently BLOCKED.** The target is app secrets travelling
|
||
with the LOCAL recovery unit, so Tier-1/Tier-2 restore work without the guest and without R,
|
||
leaving R needed for offsite recovery and host identity only. **Not adopted:** the premise
|
||
(the backup tree unreachable from every browsing/download surface) was tested and failed — see
|
||
**R-108**. Every other surface verified clean; the exposure is FileBrowser's root-bind of network
|
||
shares combined with the absence of an `IsNetwork()` filter on the deploy and per-app-migrate
|
||
paths.
|
||
- **D6 — the trust model, stated honestly.** The operator holds root SSH on every box; *"the
|
||
operator cannot read customer data"* was never the actual security property and no document may
|
||
imply it. What R **does** provide, and must keep providing: **the hub alone is not enough** — a
|
||
compromised hub yields blobs nobody can open, *provided the operator's key is never stored in the
|
||
hub*. That proviso is why escrow custody is an open decision (`07` §11-A).
|
||
|
||
**S-4 — the hub session password alone now unlocks console root on every managed box (2026-07-31, hub v0.84.0).**
|
||
Retrieving a host's vaulted break-glass `root@pam` credential previously required the **global operator
|
||
API key**, a secret distinct from the hub login and kept out-of-band. The `Console access` card on the
|
||
host page puts that retrieval behind the **hub session** (`POST /hosts/{id}/reveal-recovery-credential`,
|
||
CSRF-gated, audited), so the hub login password is now sufficient. **Accepted deliberately**, and the
|
||
reasoning is the ruling: the hub is single-operator and geo-fenced to Hungary, and it already stores
|
||
these passwords **in plaintext at rest** — so the DB and the login were already jointly sufficient, and
|
||
this is a convenience trade rather than a new exposure. It is a real trade, not a free one, and it is
|
||
recorded here rather than only in a changelog. Two things follow. **(a)** The global-key path
|
||
(`api/handler.go handleAdminGetRecoveryCredential`) stays untouched and must acquire **no** dependency
|
||
on the session layer — it is the route for when the hub UI itself is what is broken, and coupling the
|
||
two would delete the independence that makes it a fallback. **(b)** The plaintext-at-rest half is now
|
||
the named open item **R-133**: every hub DB backup is a fleet-wide console-credential dump, and the fix
|
||
shape is envelope encryption under a KEK held outside the DB, the way escrow blobs are already opaque.
|
||
|
||
Five decisions were deliberately **left open for the operator** and are recorded, unanswered, in
|
||
`07` §11: escrow custody · lost-R policy · per-scenario RTO/RPO targets (**none have ever been
|
||
stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with
|
||
the guest it backs up. Gaps minted the same session: **R-102 … R-108**.
|
||
|
||
- **2026-07-27 — R-85: the DR tier is restore-tested UNATTENDED, and its failure is HEARD.** agent
|
||
v0.104.0 + hub v0.77.0. R-82 proved a PBS restore round-trip **by hand**; three defects kept the
|
||
unattended path from covering it. (a) `buildRestoreTestScheduler` built ONE runner on
|
||
`cfg.Backup.BackupTarget()`, so the offsite tier's archives were never candidates — the DR tier
|
||
could not be restore-tested unattended AT ALL. (b) the `Spec` was an immediately-invoked function
|
||
evaluated once at daemon start, so nothing tier-varying was expressible and any config change
|
||
stayed stale until restart. (c) a failed restore-test was a `[WARN]` line with **no event, no
|
||
notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR
|
||
signal this system produces was inaudible. **(c) was the one that mattered:** rotating tiers
|
||
without it would only mean two tiers can fail silently instead of one.
|
||
**Ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, no cadence
|
||
knob; the per-tier last-proven timestamp is reported so the achieved interval can be **measured**
|
||
before anyone decides a knob is needed.
|
||
**Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS);
|
||
persisted `RestoreTestState` (survives restart; credit only on SUCCESS); `backup.InFlight`, the
|
||
host-wide one-heavy-op gate shared with the backup path (a LINK concern — an offsite restore pulls
|
||
multi-GB over the tunnel a backup pushes one); hub `restore_test_failed` / `restore_test_stale`,
|
||
kept DISTINCT and anchored on R-81, operator-tier only.
|
||
**Also fixed a time bomb I planted in Slice C:** a deadline test hard-coded the incident timestamp
|
||
`2026-07-18T18:31:06Z` while comparing against the REAL clock — harmless under one 26h threshold,
|
||
but once the offsite tier got an 8-day limit it passed all day and began failing at exactly 18:31
|
||
UTC, 8 days later. A test that passes at commit time and fails hours later is worse than one that
|
||
fails immediately.
|
||
**NOT yet proven:** rotation has not been observed selecting both tiers across consecutive
|
||
UNATTENDED cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82).
|
||
Spec: `documentation/backlog/SPEC-r85-phase4-5-2026-07-26.md`.
|
||
|
||
- **2026-07-26 — R-82 SHIPPED: the backup target split (local daily + offsite weekly).** agent
|
||
v0.97→v0.102.0, controller v0.174→v0.175.0, hub v0.76.0, host-install 1.20.0. `BackupTarget()`
|
||
returned ONE string and `BackupCadence()` ONE 24h window, so "local daily AND PBS weekly" was not
|
||
expressible — which is why the DR tier was `applied` since 07-21 with **one** snapshot on
|
||
demo-felhom and **zero, ever** on demo-hp. Now: `backup_targets[]` per-tier cadence+retention;
|
||
ONE quiesce window for both due tiers (never two app outages for one night); per-tier hub
|
||
thresholds (host 26h / offsite 8d); fresh-install default; an unprovisioned tier DEFERS.
|
||
**Operator rulings:** 2-week offsite retention, first backup runs as long as it needs, one backup
|
||
at a time per guest, drill box dropped from the rollout.
|
||
**RETENTION IS A COMMERCIAL ATTRIBUTE — the hub decides, ep0 executes (operator ruling 2026-07-27,
|
||
R-89).** A paid tier may buy longer retention, so the policy belongs with customer config on the
|
||
hub, never in ep0's PBS config and never in a box's config. Execution stays server-side: a
|
||
reconciler writes a **PBS prune job** and PBS's own scheduler runs it, so hub downtime leaves the
|
||
last-known policy running rather than silently stopping retention. **Box tokens stay write-only
|
||
(`DatastoreBackup`) — never widen a grant to fix a prune error:** a compromised box must not be
|
||
able to delete its own offsite backups, which is the scenario offsite DR exists to survive.
|
||
Increment 1 shipped 2026-07-27 (boxes stop attempting prune via `keep_last: 0`; per-namespace prune
|
||
jobs on ep0, daily 03:30 UTC) — `runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`. This closed a
|
||
live false-negative: **every** demo-hp PBS backup since 07-26 reported `job errors` while the data
|
||
landed correctly, because `DatastoreBackup` carries no `Datastore.Prune`.
|
||
**Four defects found by RUNNING it, not reviewing it** — a 30-min wait bound against a 41-min
|
||
backup (the agent recorded `success:false` while the backup was still going); the restore tier read
|
||
from the configured target instead of the archive (**a silent regression of the S4.1 fix** — the
|
||
mechanism was never removed, its INPUT changed when `local_backup_target` was retargeted to
|
||
`local`); a leaked scratch keeping `onboot:1`; and a tier firing at a not-yet-provisioned storage.
|
||
**A correction I had to make mid-arc:** I reported that the restore-test would boot a
|
||
network-conflicting clone and break the control plane. It does not — step 2 link-downs every NIC
|
||
before boot, and it is unit-tested. I read a config artifact and inferred behaviour without reading
|
||
the code path that consumes it, then escalated before finishing the check.
|
||
**NOT done:** the offsite tier is never AUTOMATICALLY restore-tested (the scheduled test only ever
|
||
selects the primary tier); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported
|
||
cadence. The installer-default fleet flip waits on a full weekly cycle.
|
||
**R-84 SHIPPED (agent v0.103.0)** — the cold in-memory store no longer causes a redundant backup:
|
||
the due-check asks the STORAGE (ground truth) instead of persisting state, so a pruned archive
|
||
correctly stops counting. Proven live on both boxes with the store cold. **demo-hp's FIRST EVER
|
||
offsite backup landed** (4.25 GB) — the R-82 finding closed on the box where it was worst.
|
||
Controller v0.175.0 deployed to both boxes. Evidence: `audits/SPIKE-r82-phase0-2026-07-26.md`, `REPORT.md`.
|
||
|
||
- **2026-07-26 — hub v0.75.0: R-81 SHIPPED — "no signal" is not "bad signal".** Third instance of
|
||
one class (hub v0.12.0 `expected_backup_missed` daily-for-everyone; hub v0.73.0 `offsite_stale`
|
||
minutes after a healthy repair; now this), fixed **as a class**, with the invariant written at the
|
||
head of `assessBackupFreshness` naming all three. `assessBackupFreshness` returns
|
||
OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored
|
||
window. **The anchor was free** — Phase-0 probe found the hub retains 90 days of host-reports, so
|
||
`GetHostReportsSince` + `newestBackupEvidence` (7-day bounded lookback, early-exit on fresh
|
||
evidence) answer *"when did I last SEE evidence of a backup?"* rather than *"what does the latest
|
||
report say?"*; absence is graded from first contact (`GetFirstHostReportAt`) reusing the existing
|
||
26 h threshold as the grace — **no new knob, no agent change, no new persisted state**.
|
||
**Rulings:** absence is never silent (a genuinely dead box must still alarm — that is the half the
|
||
naive fix breaks, and `TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms` is what makes the
|
||
suppression safe); reason strings stay distinct (the whole 07-26 diagnosis turned on reading the
|
||
exact string); deferrals are logged so quiet ≠ not-checked. Tests 493→508; red-proofs A/B/C
|
||
observed — A reproduces `newest backup is 176h0m0s old` **verbatim the mail demo-felhom sent**.
|
||
**Replayed against the real 03:00 reports** (600/417/77 rows): all three now silent. `backupStaleAfter`
|
||
deliberately untouched — ⚠️ it will alarm on a healthy WEEKLY PBS snapshot, landmine recorded,
|
||
owned by **R-82**. The real finding behind R-80 stands: the **PBS/offsite-DR tier has no schedule
|
||
at all** → R-82. Cause-fix (persist the agent store) → R-84. Evidence:
|
||
`documentation/audits/DIAG-backup-missed-2026-07-26.md`, `hub/CHANGELOG.md` v0.75.0.
|
||
|
||
- **2026-07-23 — hub v0.72.0 + controller v0.161.0: R-70 SHIPPED + R-71(c) PARTIAL (the offsite
|
||
last mile is visible; burned credentials self-heal).** One detector
|
||
(`offsite.DeliveryStateFor` — secret-row timestamps × report offsite-presence), four consumers:
|
||
operator card (static "delivered once" copy DELETED; state+age, amber past 30 min, stale-staged
|
||
info line), `offsite_delivery_stuck` warning (≥1 h, 24 h durable cooldown via `LastEventAt` on
|
||
the events table — restart-proof), R-71c self-heal through the EXISTING Re-issue path
|
||
(`monitor.OffsiteReissuer` ← webServer, armed only with a provisioner; consumed ≥1 h + ≥4
|
||
offbox-less reports + zero offbox evidence; one restage/24 h; every firing = a warning event),
|
||
and the controller's truthful empty-state banner (v0.161.0, both boxes). **Rulings:** applied
|
||
wins (stale-staged is a flag, never a downgrade); cooldowns durable via events table; both new
|
||
events operator-only (no customerMessages, not in allowedEventTypes — pbsdr_* precedent);
|
||
**R-39(a) guard in the CALLER** — the heal re-reads the secret row at act time and refuses over
|
||
an unconsumed row; the store's clobber semantics stay (Re-issue depends on supersede). Clobber
|
||
red-proof observed (`reissue calls = 1` + row clobbered with the guard removed). Self-heal +
|
||
banner ship unit-proven, NOT live-fired (no broken box; F9 rule) — arms on the next natural
|
||
occurrence. Live: checker silent on the healthy fleet; demo-felhom's stale-staged specimen
|
||
intact (its cleanup is the operator's, post-validation). R-71(a) day-0 ordering OPEN → next
|
||
spec. Evidence: `REPORT.md` (2026-07-23).
|
||
|
||
- **2026-07-23 — ISO train v1.25.0 + hub v0.73.0 (four rulings + rider).** host-install
|
||
1.17.0→1.18.0, build-felhom-iso 1.24.0→1.25.0, felhom-bootstrap R-63, hub v0.73.0. **Done +
|
||
live-verified:** (1) **F9 belt everywhere** — host-install installs the H1 OOB belt by default in
|
||
appliance mode (`--no-oob` opts out; byo still refuses), `oob.enabled` written so the agent
|
||
activates it; belt LIVE on both boxes, **login PROVEN on felhom-pve** (`felhom-op@demo-felhom`);
|
||
the orphaned operator identity was re-anchored to the operator's real machine (WG peer + SSH key
|
||
re-registered, ep0 forward rules persisted). (2) **apt** — `align_apt_repos` (appliance-only)
|
||
switches enterprise→no-subscription; LIVE-aligned both boxes (`apt-get update` clean). (3) **R-63**
|
||
— Lat2 console font before first paint. (4) **R-71 build-gate** — `assert_golden_ge_floor` in
|
||
build-felhom-iso + rule 5; red-proofed. (6) **root vault** — felhom-pve root rotated+vaulted+
|
||
verified over :22 (G1 parity). (7) **hub v0.73.0** — `offsite_stale` never-ran branch ANCHORED
|
||
(applied-only + consumed_at/escrow anchor; one-state-one-owner with `offsite_delivery_stuck`),
|
||
shipped+deployed, red-proofed. **CRITICAL live finding:** golden **0.153.0 < floor 0.156.0** in
|
||
production now — the R-71 gap is real; the gate correctly blocks a v1.25.0 build. golden republished at 0.161.0 + **vouched** (gap CLOSED), v1.25.0 ISO **built**
|
||
(gate live-passed). **Nested drill aborted on the N100** (install crawled — the N100 was
|
||
contending with live guest 9201; KVM-confirmed, not memory) → cleaned up (9201 untouched);
|
||
**operator steer: future drills on the HP**. R-63 font code-verified; the live ő/ű console
|
||
capture rides the next HP drill (belt/apt already proven live on both boxes; no-floor-update is
|
||
structurally guaranteed post-vouch). Evidence: `REPORT.md`, `scripts/CHANGELOG.md`,
|
||
`operations/nodes.md`.
|
||
|
||
- **2026-07-23 — F10 offsite leg RESOLVED on demo-hp (diagnostic + designed-path repair + tier
|
||
proof; no code changed).** The hub said "Provisioned" while the box said „Még nincs beállítva" —
|
||
root cause: the day-0 managed update (0.153.0→0.156.0, 07-21 16:28) killed the offsite
|
||
apply-bridge ~35 s after it consumed the one-time password, before key-install/persist; the box
|
||
then hit the consume-404 dead-end at every start, silently, for 2 days. Repair = operator
|
||
**Re-issue** click (converged in 8 s), escrow ceremony via the real wizard (auto-confirm 3 s),
|
||
then the full proof: paperless-ngx toggled into scope, first offsite run (repo initialized on
|
||
sub3, 79.8 MB, snapshots=1), probe deleted → `mode=full` restore (snapshot `2bf7f2e1`) → place
|
||
merged 1 file missing-only → **md5-identical**. Minted **R-70** (last-mile invisible on both
|
||
surfaces; hub never reads `one_time_secrets.consumed_at`) and **R-71 [P1]** (the race recurs on
|
||
every fresh onboarding whose ISO floor lags the managed floor). Record:
|
||
`documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. F10's PBS-DR half stays open
|
||
(F13 + ceremony R-moment).
|
||
|
||
- **2026-07-22 — hub v0.71.0: the notification train (audit F11+F12+F14-light).** Four rulings now
|
||
standing: **(1) recovery pairing** — `*_recovered` notifies the operator always and the customer
|
||
**iff the customer was mailed the paired stale/down** (evidence = `notification_log` customer-
|
||
channel sent rows via `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for
|
||
recovery; ties → no mail). Severity semantics FROZEN — recovery stays `info`, routed by an
|
||
explicit eventType branch before the severity gate. **(2) seed-at-claim** — `MarkClaimed` seeds
|
||
`customer_notifications` from the registered email, INSERT-if-absent only (never touches an
|
||
edited row; empty email = no-op; seed failure never fails the claim); default set is
|
||
critical-only: node_down, backup_failed, disk_critical, host_disk_critical,
|
||
storage_fill_critical, offbox_repo_orphaned. **(3) empty-email no-clobber** — a prefs push with
|
||
empty email preserves a stored address (hub-side belt; controller 0.160.0 already guards its two
|
||
push legs). **(4) priority headers** — error/critical mails carry `X-Priority: 1` +
|
||
`Importance: high` (Resend `headers`, live-probed); the `test` event now also mails the operator
|
||
with those headers (one click proves both channels + rendering). Latent nil-deref fixed:
|
||
`sendTestEmail` panicked on a customer with no prefs row. NOT yet live: a natural `*_recovered`
|
||
mail (next real staleness cycle / reboot drill — never fabricate one by blocking reports) and
|
||
seed-at-claim on a real claim (Peti's Friday reinstall is the natural candidate).
|
||
|
||
- **2026-07-22 — hub v0.70.1: the ghost customer's Delete button exists now.** The v0.70.0
|
||
ghost-delete path was fully implemented and fully unreachable — the **fourth inert-seam defect**,
|
||
this time a TEMPLATE GATE: the Danger-zone card (and the `customerDeleteOpen` script) sat inside
|
||
`{{if .HasConfig}}`, so a ghost rendered no Delete button (found live on `demo-vm-felhom`).
|
||
Fix: `Deletable` page flag = the exact negation of the delete preview's 404 predicate; RESET
|
||
stays HasConfig-gated, Block/Unblock config-only. Render tests per branch + 2 red-proofs.
|
||
**The seam-wiring rule now covers template gates** (see CLAUDE.md) — handler tests that POST
|
||
directly prove nothing about reachability. **Operator live leg PENDING: the demo-vm-felhom ghost
|
||
delete click** (doubles as v0.70.0+v0.70.1 live validation). Same day: **agent 0.93.0 published
|
||
+ deployed fleet-wide** (`pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`), **R-17 CLOSED**
|
||
(u629193-sub1 deleted at the console).
|
||
|
||
- **2026-07-21 — THE FLEET IS TWO HOSTS.** A second Proxmox node exists: **`demo-hp-bb76ea`**, an
|
||
**HP t740 Thin Client** (Ryzen V1756B, 30 GiB, PVE node `felhom-host`, customer `demo-hp`), installed
|
||
from the armed universal ISO the same day. Both hosts run **agent 0.92.1**; the N100 is
|
||
`demo-felhom-8363b5`. **Full inventory: `documentation/operations/nodes.md`** — read it before
|
||
touching the HP, because two things there are load-bearing: the box's **1TB Toshiba NVMe is present
|
||
and UNENROLLED** (still carrying its old NTFS partition; a future Tárhely candidate that must join
|
||
through the storage flow, never the installer), and there is **no operator SSH key baked** on it, so
|
||
access is the hub-vaulted G1 break-glass password (R-61 is the fix for that lockout).
|
||
**Both nodes are at the VACATION site and travel home ~2026-08-02.** `demo-hp` joined the tailnet as
|
||
`100.76.96.79` — an **operator-lab exception**, not product shape; real customer boxes never get
|
||
tailscale. **Second-hardware proof, worth keeping:** the pairing/day-0 chain ran end to end on virgin
|
||
hardware it had never seen, the **shim loader booted with Secure Boot ENABLED** (so SB-off was an
|
||
N100-firmware workaround, not a Felhom requirement), and the box **self-lifted the floor
|
||
0.153.0 → 0.156.0 during day-0**. The install was not clean, and the failures are filed: **R-59**
|
||
(no DHCP on the 4-port NIC → the installer baked a static fallback instead of aborting), **R-60**
|
||
(first-boot NIC sweep self-heal), **R-61** (baked root password unknowable).
|
||
|
||
- **2026-07-19 — N100 MADE LOCATION-INDEPENDENT via Tailscale.** `felhom-pve` (the N100) joined the
|
||
tailnet as a host package (`100.70.170.35`, expiry disabled); DooPlex already advertised
|
||
`192.168.0.0/24` via its GitOps k3s `admin-system/tailscale` pod (`100.107.87.53`). `ssh felhom-pve`
|
||
now targets the tailnet IP (direct-over-LAN at home, tunnel when away); `felhom-pve-lan` =
|
||
`192.168.0.162` fallback. `--accept-dns=false` on the host (pi-hole split-horizon preserved). The
|
||
host was converted **static→DHCP** (reservation MAC `68:1d:ef:5d:a6:64`→`.162` keeps home behaviour).
|
||
**Measured spike:** `--accept-routes` on the N100 *while on `192.168.0.0/24`* hairpins the local
|
||
subnet (table 52 overrides `main`) → kept OFF, it's a travel-only opt-in. **PBS is offsite** (its own
|
||
WireGuard tunnel to Hetzner, not the home LAN) so it needs none of this. **Recorded, NOT fixed:** the
|
||
controller↔agent plane is hard-pinned to `192.168.0.162` (`felhom-agent` `listen_addr` +
|
||
guest `bootstrap.json`), so the box is reachable + backs up off-LAN but its local operator plane is
|
||
not yet portable. Full doc: `documentation/operations/tailscale.md`. No code changed.
|
||
|
||
- **2026-07-19 — CC WORKFLOW MIGRATED TO DOOPLEX.** Claude Code now runs natively on DooPlex
|
||
(192.168.0.180, Debian 13, `kisfenyo`) in a tmux session, working directly in
|
||
`/mnt/5_hdd/felhom.eu/git/<repo>`. **DooPlex is now the canonical CC environment; the Windows
|
||
workstation trees are stale** as of felhom.eu `ee48a48288`, felhom-agent `d81fddc7b1`,
|
||
felhom-controller `ac7323dc9a`, app-catalog-felhom.eu `a14c97db65` (unchanged — it had no
|
||
environment-specific content). Builds are local commands; felhom-pve is one `ssh` hop; `kubectl`
|
||
is local. **`claude-in-chrome` is NOT available here** — endpoint-level validation is the
|
||
standard method. New **clean-tree gate** before any build (`git status --porcelain` empty AND
|
||
`HEAD` == `origin/main`), because the CC working tree is now the tree `build.sh` builds from.
|
||
Windows knowledge is preserved in per-file "Legacy: Windows workstation" sections and the
|
||
versioned workspace file `documentation/runbooks/workspace-CLAUDE.md` (live copy:
|
||
`/mnt/5_hdd/felhom.eu/git/CLAUDE.md`).
|
||
|
||
- **2026-07-17 — CUSTOMER SELF-BIND shipped (hub v0.66.0 + ISO scripts v1.20.0, R-27 slice 1).** A
|
||
customer binds their OWN freshly-installed appliance without the operator. Operator clicks **"Send
|
||
self-bind link"** on the customer Setup tab → hub mints a **7-day tokenized capability link** →
|
||
emails it (Hungarian, sibling sender, NOT via the claim engine) → customer opens the **public,
|
||
Hungarian `/bind/<token>`** page (no login — the URL token IS the auth) → enters the **console
|
||
pairing code** (shown on the box screen) + their **retrieval passphrase** → hub stages the bind via
|
||
the same `BindAppliance` (provenance `customer_selfbind`) → the box's ~30 s appliance poll delivers.
|
||
**Viktor's three rulings, verbatim:** (a) *"only their own visible"* → console pairing code, **no
|
||
appliance list ever rendered**; (b) *first-box entry* → operator-sent tokenized capability link over
|
||
Hungarian email; (c) *lockout after 5 failed attempts* → token locks, "call support". Wrong code and
|
||
wrong passphrase = **one identical generic failure** (no oracle; both factors compared
|
||
unconditionally); expiry falls back to operator-bind unchanged. **THE TRAP (§9.2):** one public
|
||
prefix `/bind/`, exempted from auth+CSRF at both `/login` gate sites via a single `isPublicBindPath`
|
||
predicate (tight trailing-slash match; ServeMux `..`-cleans; handler rejects `/` in the token) — 4
|
||
red-proofs verified red-then-green (lockout→C1, oracle→B, widened-prefix→E, single-active→C4).
|
||
**GC verdict:** no appliance-staleness GC exists (`applianceStaleAfter` is a display badge only) → the
|
||
7-day token TTL stands alone, single-active-per-customer, no reaper needed. **Controller/agent
|
||
untouched; R-27b (controller second-box dismissable prompt) DEFERRED, mechanism sketched.** Green:
|
||
full hub `build/vet/test` (17 ok, +9 self-bind tests) + `bash -n`. **DEPLOYED + LIVE (hub 0.66.0 on
|
||
k3s, Synced/Healthy).** Live through the real ingress: `GET /bind/<garbage>` → 200 Hungarian expired
|
||
page (public, logged-out, no-oracle — the 200-not-500 proves `selfbind_tokens` migrated live); `/` +
|
||
`/hosts` → 302 /login (gate intact); `POST /bind/<garbage>` no-CSRF → 200 (exempt) while
|
||
`POST /customers/acme/block` no-CSRF → 302 /login (exemption is `/bind/`-only). **PENDING
|
||
(operator-gated, NOT-live):** operator "Send self-bind link" + the customer's real two-factor success
|
||
(hub UI password-gated → CC can't log in) + the new-ISO console banner on a re-imaged box → fold into
|
||
the R-1 rehearsal. See REPORT.md.
|
||
|
||
- **2026-07-17 — PBS DR STORAGE VISIBILITY + Offsite tab split + dual gauges shipped (hub v0.65.0 +
|
||
tenantsync v1.2.0, R-5), LIVE.** Scoping correction (Viktor): "restic box" and "PBS box" are NOT two
|
||
Hetzner Storage Boxes — restic = subaccounts on the shared Hetzner box (Hetzner API, v0.64.0); **PBS DR =
|
||
the `felhom-offsite` PBS datastore on ep0** (NO Hetzner API; the hub had no read of its fill). **Option A
|
||
(ruled):** a read-only `usage` op on the `felhom-tenantsync` ep0 forced command (twin of `fingerprint`,
|
||
`df` on the datastore path — no customer_id, no admin token, NO mutation), polled by `monitor.PBSDRBoxChecker`
|
||
(OffsiteBoxChecker clone over a `usageReader` seam; 15-min throttle; three states ok/unavailable/degraded;
|
||
escalation-only `pbsdr_box_fill` on the `"pbsdr-box"` operator scope). `/offsite` now has **Restic** +
|
||
**PBS DR** tabs (endpoint cards under PBS DR — the endpoint IS the PBS host); the dashboard has **two
|
||
gauges** (restic %·ratio, PBS %). **Graceful degradation: the hub deploy is INDEPENDENT of the ep0 update**
|
||
— an ep0 ≤ v1.1.0 → `ErrUsageUnsupported` → the gauge shows "n/a" until updated (no hub redeploy).
|
||
**PROVEN-LIVE (data pipeline, both gauges):** ep0 updated to v1.2.0 this session; the hub polled it —
|
||
`PBS-DR box refreshed: 19.1% full (7.1 GB of 37.2 GB)`. **DECISION:** PBS thresholds are separate keys
|
||
(`PBSDRBoxFill*`) defaulting to restic's 80/90 (independently tunable; Viktor's call for a different pair);
|
||
no oversubscription for PBS (namespaces, not quotas). **OPEN:** UI render unit-verified only (hub UI
|
||
password-gated); the PBS fill alert email is unit + red-proof verified, NOT fired live (datastore nominal
|
||
at 19%). READ-ONLY against ep0 + Hetzner. See REPORT.md.
|
||
|
||
- **2026-07-17 — OFFSITE POOL-BOX AGGREGATE shipped (hub v0.64.0, R-5).** The operator now sees the
|
||
shared pool box's real state on the hub: total fill vs capacity, Σ(shared soft quotas)/capacity (the
|
||
oversubscription ratio), per-customer usage/quota bars, and a box-level operator alert (fill 80/90%,
|
||
oversub 2.0×). New `monitor.OffsiteBoxChecker` (OffsiteChecker-sibling): fetch-throttled to one Hetzner
|
||
GET / 15 min, cached `BoxSnapshot`, escalation-only + recovery re-arm; two INDEPENDENT signals (fill +
|
||
oversub). Capacity = `storage_box_type.size`, usage = `stats.size` (Phase-0-pinned, bytes); Σ(quota)
|
||
from the ConfigJSON `Descriptor` (`offsite.ReadDescriptor`, new — NEVER the report echo; dedicated +
|
||
disabled excluded); per-customer USAGE from the report echo. Events scope `"pool-box"` → operator channel
|
||
only, no SaveEvent. Web: Offsite-tab panel + compact dashboard tile, reading the cached snapshot only
|
||
(never fetches). **PROVEN-LIVE (data pipeline):** the in-cluster checker computed the real box —
|
||
`0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x`. **DECISION: thresholds
|
||
(80/90/2.0×) are Claude's encoding — Viktor's ruling pending; named `Alerting.OffsiteBox*` keys = the
|
||
one-line flip.** **OPEN:** the UI render is unit-verified only (hub UI password-gated → CC can't
|
||
screenshot); the alert emails are unit + red-proof verified but NOT fired live (real pool nominal — a
|
||
live-fire would email Viktor; offered on demand). READ-ONLY against Hetzner throughout. See REPORT.md.
|
||
|
||
- **2026-07-17 — SYSTEM-INITIATED IMMEDIACY wired per-plane (hub v0.63.0).** The immediate-sync arc
|
||
covered only OPERATOR-initiated changes; **system-initiated** desired-state mutations bumped the
|
||
generation silently, so a freshly onboarded box waited a full agent tick for state the hub had already
|
||
minted (observed live at slice-C onboarding). Decision: wire the existing, live-proven notifiers into
|
||
every system site on the CORRECT plane — agent-plane `poke.Notifier` at `PBSDRAutoProvision` (the exact
|
||
lag), `ReissuePBSDR` (which also lifts the pbsdrheal reconciler's escalation with zero reconciler
|
||
changes), `handlePBSDRReissue`, and the two admin api desired-state writers (new nil-safe `Poker` seam);
|
||
controller-plane `intent.Hub.Bump` at `reissueOnReenroll`. All fire AFTER the store write only. The
|
||
**pbsdrheal Restage path is deliberately poke-free** (no generation bump → the agent's 60 s pbsdr ticker
|
||
is its pickup path — a poke there is a verified no-op); the WG register/delete bumps stay poke-free
|
||
(undeliverable pre-tunnel / transport removed — the agent fast-tick SECONDARY owns the register leg).
|
||
The **first-hold-baseline race** is still OPEN by design (a bump landing during a fresh controller's
|
||
FIRST wait is recorded as baseline without firing — the ruled fix is carrying `intent_gen` in the report
|
||
ACK, its own future task). Unit-tested + red-proofed; live legs fire only on real system events (fold
|
||
into the rehearsal bind sequence). See REPORT.md + hub/CHANGELOG.md v0.63.0.
|
||
|
||
- **2026-07-17 — R-21 SLICE C SHIPPED: the universal secret-free ISO (hub v0.62.0 + scripts v1.19.0, LIVE).**
|
||
The distributed ISO carries NO customer secret. A box booted from `build-felhom-iso.sh --pairing`
|
||
self-registers as an UNCLAIMED APPLIANCE (`POST /api/v1/appliance/register`, keyed by SMBIOS-uuid +
|
||
MAC set — the DMI-verdict tiebreaker; token = sha256-stored, 256-bit), the operator BINDS it to a
|
||
customer on the Hosts page (new "Unclaimed appliances" section, SSH-key fingerprints, host-count
|
||
display-only), and the hub delivers customer-id + retrieval passphrase ONCE (one-shot poll: 404
|
||
no-oracle / 204 unbound / 200 deliver / 410 gone). The bootstrap is **one unit, two modes** — direct
|
||
(env-baked, byte-identical, regression-proven zero-appliance-calls) vs pairing (register→poll→write
|
||
env→fall through to direct). Red-proofs: one-shot delivery + register idempotency (both proven red).
|
||
Live-verified through the public ingress (register→token, 204, bogus→404). Artifact proven secret-free
|
||
(baked env = hub URL only; manifest `secret-bearing: no`). **Bind is operator-password-gated** → the
|
||
live boot→bind→day-0 composition (+ physical N100) folds into the supervised rehearsal, which now runs
|
||
the COMPLETE final flow in one pass (RESET demo → generic ISO → bind → day-0). SSH-key pinning =
|
||
stored + fingerprints displayed (host-attach on bind deferred, no clean surface). Self-bind page = R-27.
|
||
R-21 now SHIPPED (A+B+C). See REPORT.md.
|
||
|
||
- **2026-07-17 — R-21 SLICE B SHIPPED: firmware loader option (scripts v1.18.0).** Closes N100 **F1
|
||
(HIGH)** — cheap AMI (`AN3PLUS 0.01`-class) firmware can't USB-boot the ISO's stock GRUB
|
||
(`relocation 0x0`). `build-felhom-iso.sh --loader shim|mkimage` (default **shim**, unchanged;
|
||
profile-settable via `FELHOM_LOADER`). **mkimage** = a monolithic `grub-mkimage` loader built from
|
||
the ISO's own GRUB (module set from its `grub.cfg`, embedded `search --fs-uuid`→`configfile`), swapped
|
||
into the ISO9660 tree + `efi.img` ESP by `mkimage-surgery.sh` **after** `prepare-iso` (payload
|
||
provably untouched), re-mastered with xorriso preserving BIOS-hybrid + UEFI + GPT-ESP (drops only the
|
||
irrelevant Apple HFS+/APM). **Unsigned → target Secure Boot MUST be OFF.** Loud: banner + manifest
|
||
`loader:` + `-mkimage` filename. New `profiles/n100.profile`; Dockerfile gains grub/mtools.
|
||
**RUNBOOK-B legs (nested VM 311, OVMF):** shim boots+installs under SB-enforcing + SeaBIOS; mkimage
|
||
boots+installs under SB-off; **mkimage under SB-enforcing FAILS `Access Denied`** (red-proof); surgery
|
||
byte-identical payload. Physical N100 boot still pending → supervised rehearsal (an `n100-safety`
|
||
match-nothing ISO built + sha-recorded, unbooted). PXE stays a deferred R-21 note. See REPORT.md.
|
||
|
||
- **2026-07-17 — CUSTOMER RESET SHIPPED: hub v0.61.0 + felhom-tenantsync v1.1.0 (LIVE).** The middle
|
||
lifecycle tier (host delete < RESET < customer Delete). One operator action → pre-first-install: all
|
||
OPERATIONAL state dies (offsite repo, PBS namespace+backups, DR recipe, one-time secret, claim state,
|
||
retained escrow custody); IDENTITY + basic config (incl. the offsite tier CHOICE) + provenance + events
|
||
SURVIVE. Rulings: separate escrow-custody ack; clears claim (fresh code next onboarding); REFUSES while
|
||
any host row exists; live-counted confirm inventory. Discipline: external teardown FIRST, DB purge LAST,
|
||
every leg idempotent → partial run re-runs from the top (purge withheld until externals ok). New:
|
||
`store/customer_reset.go` (journal+inventory+ack-gated purge), `claim.ResetToUnclaimed`,
|
||
`offsite.{Deprovision,OffsiteIdentifier,ClearProvisionedDescriptor}`, `tenantsync.Deprovision` +
|
||
`felhom-tenantsync.sh` **deprovision** op (v1.1.0, destroys ns+groups+token, shared user untouched),
|
||
`web/customer_reset.go` (GET inventory JSON / POST orchestration) + an **amber** RESET card distinct
|
||
from the red Danger-zone Delete. Red-proofs: ack-gate + partial-failure resumability (both proven red).
|
||
**Live-drilled on ep0** (throwaway `drill-reset-01` with a real backup): deprovision `deleted:true`,
|
||
idempotent re-run `deleted:false`, all 3 real tenants survived. Hetzner-delete + the password-gated web
|
||
POST covered by tests (offsite Deprovision mirrors live-proven ReissueCredentials). See REPORT.md.
|
||
|
||
- **2026-07-17 — HOST-DELETE DEMOTES ESCROW: hub v0.60.1 (LIVE).** Closes the v0.60.0 review gap:
|
||
`DeleteHost(deleteEscrow=true)` now DEMOTES the current escrow blob into `host_escrow_superseded`
|
||
(copy-before-delete, same tx) + spares existing superseded rows — never destroys custody. The
|
||
customer Danger-zone Delete (`DeleteCustomerConfig`, which before touched ONLY `customer_configs`)
|
||
is now the one true purge point: cascades both escrow tables for all the customer's hosts incl.
|
||
already-deleted (via F-14 `host_deletions`). One shared copy routine `demoteCurrentEscrowTx` (reused
|
||
by SaveHostEscrow). Checkbox/refusal/Danger-zone wording → demotion. **S6b VERDICT: OBSOLETE** —
|
||
re-enroll upserts cleanly (`UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no dup
|
||
refusal) + v0.57.0 arc auto-fires re-issues (ROADMAP R-3 refined). Red-proofs
|
||
`TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. Observation:
|
||
customer-delete still doesn't touch host rows / the broader offboarding (rehearsal's question). REPORT.md.
|
||
|
||
- **2026-07-17 — OFFSITE CONTINUITY Part B SHIPPED: hub v0.60.0 (LIVE).** Superseded-escrow
|
||
retention (data-first, Viktor's ruling). `SaveHostEscrow`'s destructive `ON CONFLICT` overwrite
|
||
destroyed the old passphrase's only copy (the reinstall-orphan incident); now a new
|
||
different-sha blob RETAINS the old into the new `host_escrow_superseded` history table (tx, before
|
||
overwrite); same-sha = idempotent. ACK/restore read the current row unchanged. `escrow_superseded`
|
||
event + operator retained-count on host detail; registered `offbox_repo_orphaned`/`offbox_repo_reset`.
|
||
Red-proof `TestSaveHostEscrow_RetainsSuperseded`. Pairs with controller v0.142.0 (Part A
|
||
orphaned-repo guard). Guided old-history recovery = R-26 (not built). See REPORT.md.
|
||
|
||
- **2026-07-17 — N100 polish (F5 scripts + F6/F7 controller).** F5 (host-install v1.17.0):
|
||
appliance mode auto-sizes the guest (RAM=clamp(host-4096,min 4096,max host-2048,ceil host-1024) +
|
||
cores=host-1 min 2) when no explicit cap; explicit `--memory`/`--cores` win. Harness red-proof
|
||
(8/16/32 GB). Same commit: the stale "EMPTY by default" operator-key comment fixed (R-16 doc-drift,
|
||
keys are PINNED). F6/F7 landed in controller v0.141.0 (LIVE on demo; fresh-USB wizard enroll+format
|
||
now PROVEN-LIVE — see the felhom-controller REPORT). New ROADMAP R-24 (guest resources as
|
||
desired-state / live resize = F5 follow-on). Operator sidebar #1 (BIOS State-After-G3→Power-On)
|
||
added to `scripts/iso/README.md`. See REPORT.md.
|
||
|
||
- **2026-07-16 — DIRECTION-2a AGENT-PLANE POKE SHIPPED: hub v0.59.0 (LIVE).** The agent-plane
|
||
sibling of the v0.58.0 wait channel: `internal/poke` — a pinned-host-key SSH sender (wgsync
|
||
sibling) + fire-and-forget `Notifier` (`PokeHost`/`PokeAllHosts`, nil-safe, detached) that nudges a
|
||
box in seconds via a CONTENTLESS UDP poke relayed hub → ep0 `felhom-poke` forced-cmd → wg0-origin →
|
||
the box WG /32:51822. Wired at `applyPBSDR` (per-host) + `handleSetArtifacts` (fleet). ep0 surface
|
||
(`scripts/felhom-poke.sh`, non-root, no sudoers; runbook §11) + `Secret/agent-poke` +
|
||
`manifests/hub.yaml` (tag 0.59.0). **Deployed live** (ArgoCD Synced/Healthy, `agent-plane poke
|
||
enabled`); ep0 leg live-validated (`poke-fired`; non-WG + `rm -rf /` refused). Source note: the
|
||
MinAgent-floor save does not bump per-host generation (self-update rides signed-ops) → the fleet
|
||
poke accelerates the next report, not a delta. Pairs with felhom-agent v0.89.0 (the listener).
|
||
**PROVEN-LIVE 2026-07-17:** the operator's real manifest save logged `poke: sync-poke delivered to
|
||
10.77.0.2` (+ floor HELD for the below-MinAgent box); the demo (now 0.89.0) received a poke in
|
||
~31 ms and fired the immediate desired-state cycle (save→tick ≈ ~0.45 s). All three live legs
|
||
(Scenario 4 A/B train, Scenario 1 R-22 self-heal, Scenario 3 latency) done; capability-map poke +
|
||
F4 rows upgraded to PROVEN-LIVE. See REPORT.md.
|
||
|
||
- **2026-07-16 — DIRECTION-2 IMMEDIATE-SYNC SHIPPED: hub v0.58.0.** The hub→box "sync now" wait
|
||
channel (option b of `documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`). New
|
||
`internal/intent` in-memory per-customer generation notifier (Bump/Wait/Close; coalesce-to-latest;
|
||
race-closer; in-memory by design → a restart costs one harmless full-state report, never a storm).
|
||
`GET /api/v1/wait` long-poll: per-customer only (global key 400; A can't see B), 240 s hold with a
|
||
25 s heartbeat newline that defeats nginx's 60 s `proxy_read_timeout` **without any ingress
|
||
annotation** (WriteTimeout lifted per-connection via `ResponseController`; global timeouts
|
||
untouched). Every operator-intent web handler bumps AFTER its store write (config CRUD, claim
|
||
resend, offsite re-issue/freeze, password regen, block/unblock, floors [global bumps all managed],
|
||
controller log-tail + log-bundle). `intentHub.Close()` before `server.Shutdown`. Pairs with
|
||
controller v0.140.0 (the long-poll client). The immediate-sync arc is COMPLETE for the controller
|
||
plane (Direction 1 = ctrl v0.139.0, Direction 2 = hub v0.58.0 / ctrl v0.140.0); the agent-plane
|
||
ep0-poke stays PARKED in the OOB arc. Detail: `hub/CHANGELOG.md` v0.58.0, `REPORT.md`.
|
||
|
||
- **2026-07-16 — REINSTALL ARC SHIPPED: hub v0.57.0 (F2/F3/2.3).** The clean-slate
|
||
reinstall-of-existing-customer path is now first-class (Peti R-1 prerequisite). On host-enroll's
|
||
mint path (single-bump-per-re-enroll): claim re-issue for a claimed customer (`ReissueForReenroll`,
|
||
reset code — the hub never stores the password, so fork B not A), offsite re-issue to the fresh box
|
||
(`ReissueOffsiteForCustomer`), and escrow-honesty — re-issuing offsite marks the escrow STALE
|
||
(`MarkEscrowStale`), withholds the mismatched restic-hash from auto-confirm, and shows "stale" not
|
||
"ceremony done" (2.3, red-proofed). **Part 0** unblocked the demo live (ACL grant on
|
||
`/storage/felhom-offsite` → PBS-DR converged, `escrow.pbs_storage_id` seeded). **Controller +
|
||
scripts unchanged** — both spec premises were contradicted by source (the controller reads escrow
|
||
prereqs live from the agent; the installer can't know the descriptor-provisioned storage id). **F4**
|
||
root fix is agent-side → **ROADMAP R-22** (agent train: the reconcile pre-check 403 must not abort
|
||
before the root-run apply grants the ACL). Detail: `hub/CHANGELOG.md` v0.57.0, `REPORT.md`.
|
||
|
||
- **2026-07-16 — R-21 PHYSICAL RUN on the demo N100 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`).**
|
||
demo-felhom host reinstalled clean-slate from a pipeline-built ISO; first-boot chain reached
|
||
**rc-0 on the first attempt on real hardware** (closes slice A's operator-gated boundary),
|
||
serial-filter safety proven on metal (external-HDD canary byte-identical), PBS-DR reconciler
|
||
self-healed on the reused peer. **The obstacle was firmware, not us:** the cheap AMI `AN3PLUS 0.01`
|
||
board can't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`; SB-off + shim-bypass don't help) —
|
||
worked around live by rebuilding the stick's loader with `grub-mkimage` from the box's own working
|
||
GRUB. Findings (to tackle): F1 firmware boot loader (slice-B), F2 claim-re-issue on reinstall,
|
||
F3 offsite-re-issue, F4 PBS-DR ACL on non-default storage id, F5 guest-RAM configurability,
|
||
F6 drive-init doesn't mount+attach, F7 back-route. Box healthy, BookStack deployed, demo live.
|
||
|
||
- **2026-07-16 — R-21 SLICE A SHIPPED: bare-metal Felhom ISO pipeline (`scripts/iso/`, scripts
|
||
v1.16.0).** A DooPlex pipeline (`build-felhom-iso.sh` + `Dockerfile.assistant`) turns the official
|
||
PVE ISO into a Felhom auto-install ISO whose first-boot stub installs a retry-forever
|
||
`felhom-bootstrap` unit that unattended-fetches `felhom-host-install.sh` from the public
|
||
`felhom.eu/scripts/` channel and runs it (host-install **unmodified**). Validated on VM 310:
|
||
build gate + red-proof, disk-filter fail-safe, stub→unit→real-fetch→invoke→retry, resume-decision,
|
||
exactly-once, no-net retry+recovery — all GREEN. **Rulings settled:** stub is from-iso (no
|
||
cert-rotation risk); ordering fully-up; the retry unit owns all network work. **§4.4 finding
|
||
(shapes slice C):** the distribution URL carries no secret, but the unattended install needs the
|
||
customer retrieval passphrase → **the slice-A ISO is secret-bearing** (supervised/single-use; the
|
||
bootstrap shreds the env on success). OPEN: host-install rc-0 terminal success (hub-enrolled +
|
||
guest-provisioned) is operator-gated — a drill customer needs the password-gated create-UI (a
|
||
live-`hub.db` insert is unsafe). Slice B = physical N100 RUNBOOK; slice C = claim-pairing to
|
||
de-secret the distributed ISO. Detail: `REPORT.md`, `documentation/audits/SPIKE-baremetal-iso-2026-07-16.md`.
|
||
|
||
- **2026-07-13 eve — ESCROW CEREMONY WIZARD SHIPPED (agent v0.88.0 + controller v0.127.0; this
|
||
repo: host-install v1.16.0 + RUNBOOK rewrite).** The recovery-code ceremony is now
|
||
customer-driveable from the controller (`/backup/escrow`): one-shot R reveal, re-stage-first,
|
||
Scenario-F stale-blob re-check (fired LIVE on both boxes' hash-less blobs at first ACK; drill's
|
||
repaired via a real ceremony — hub hash now covers). **Ruling F1 (2026-07-13), settled:** *R
|
||
transiting the Cloudflare tunnel once at reveal is an accepted risk — same trust class as the
|
||
claim code/login password; LAN-direct delivery is PARKED.* (threat model:
|
||
runbooks/RUNBOOK-escrow-ceremony.md). host-install v1.16.0 = FELHOM_ESCROW rides the canonical
|
||
sudoers fetch; hub `hostInstallVersion` synced (gate green), **hub NOT redeployed** (const rides
|
||
the next train). OPEN: supervised full-browser wizard pass (Viktor's drill login; also demo, to
|
||
clear its legacy stale warning); agent 0.88.0 publish + Day-0 vouch at the next train.
|
||
|
||
- **2026-07-13 — CLOSING BUNDLE SHIPPED: hub v0.53.0** — the last engineering items on the
|
||
pre-tester board: F-14 gated auto-Reissue (deletion provenance `host_deletions` in-tx +
|
||
`token_exists` gate → the EXISTING Reissue op + `pbsdr_auto_reissue` audit event; no record /
|
||
un-acked → refusal byte-unchanged), dead-host roll-up honesty (customer status = worst of
|
||
controller-derived + per-host `hostStatus` on dashboard//configs/detail, cause chip names the
|
||
host; the Peti 23h-masking shape is the pinned fixture), operator bearer OUT OF GIT
|
||
(`Secret/report-api` secretKeyRef non-optional + `REPORT_API_KEY` env override +
|
||
`scripts/manifest_bearer_gate.py`; VALUE ROTATION = supervised step in
|
||
documentation/runbooks/secrets.md — the git-history copy is dead only after it).
|
||
**Two operator rulings (2026-07-13), recorded verbatim — future sessions must treat these as
|
||
settled:**
|
||
1. ***F-14:** auto-Reissue of offsite credentials is permitted ONLY when the hub's own deletion
|
||
record shows the tenancy's owning host was removed through the escrow-ack flow — acknowledged
|
||
destruction is not silent re-keying. No record → manual Re-issue stays the only path.*
|
||
2. ***Roll-up:** a customer's status may never look better than its worst expected host. Single
|
||
staleness definition (`hostStatusOf`) — no second threshold anywhere.*
|
||
Gate additions: `python scripts/manifest_bearer_gate.py` after ANY manifests/ change (64-hex
|
||
bearer shapes; felhom.secret.yaml carve-out = visible KNOWN-BACKLOG line). GOTCHA: the repo
|
||
.gitignore `*secret*` pattern silently un-tracks any "secret"-named script — hence the
|
||
"bearer" name. The controller repo's controller.yaml.example carried the REAL bearer literal —
|
||
scrubbed (felhom-controller 51c871a).
|
||
|
||
- **2026-07-13 — POLISH BATCH SHIPPED: hub v0.52.0 (+ controller v0.123.0)** — take-two F-15
|
||
CLOSED (reset-request response carries the rotated code hash — ACK shape/guarantee — emailed
|
||
codes work immediately; live 1 s apply + first-try accept on Viktor's exact failure path) and
|
||
F-16 CLOSED (zero native `confirm()` in the hub UI — `inline_confirm.html` partial, Igen/Mégse
|
||
inline two-step, asserted by `scripts/hub_confirm_gate.py` — **which exists but is NOT enforced:
|
||
it is invoked by nothing** (R-29); danger-zone typed-confirm untouched;
|
||
live: offsite re-issue completed under browser automation without freezing). Operator resend
|
||
keeps the ACK lag — flash says so. Take-two §4 dispositions updated (F-15/F-16/zero-toggle obs
|
||
→ SHIPPED). Remaining from take-two: **F-14** (needs operator ruling: auto-Reissue re-adopt vs
|
||
never-silently-re-key) + S5 DR-restore drill; hub bearer key rotation still DUE.
|
||
|
||
- **2026-07-12 — DR-TIER-BY-DEFAULT BATCH SHIPPED: installer v1.15.0 + hub v0.51.0 (+ agent
|
||
v0.86.0 in felhom-agent)** — closes DRILL-day0-vm F-1/F-2/F-3/F-7/F-8/F-9/F-10 and F-6 (by
|
||
policy). **The five operator decisions (2026-07-12), recorded verbatim — future sessions must
|
||
treat these as settled:**
|
||
1. *Every install deploys the full DR plumbing uniformly (pbs-apply wrapper, `age`, WG enabled,
|
||
full ACL). **Capability is baked; activation is a hub flag.***
|
||
2. ***DR tier defaults ON for new customers.** Disabling is the per-customer exception — and it
|
||
removes the Felhom-side cost entirely (no PBS namespace, no endpoint dependency).*
|
||
3. ***Offsite app backup requires the DR tier ON** (fork-4 needs the ceremony; the ceremony
|
||
needs PBS). Enforced hub-side by extending the drill-proven dependency guard.*
|
||
4. ***Identity-only escrow is formally PARKED** — closed by policy, not implemented. Revisit
|
||
only if a customer class demands lean-tier offsite. Record in the ceremony runbook + CONTEXT
|
||
so future sessions know it was a decision, not an oversight.*
|
||
5. ***WG is base infrastructure, not a DR component** — always-on like cloudflared; it is also
|
||
the future OOB-management transport. BYO consent: a disclosure line lands in the
|
||
tester-agreement doc; per-customer hub disable remains possible.*
|
||
Mechanics: installer ships `felhom-pbs-apply` + `age` + `wg_tunnel.enabled: true` (byo wg-off
|
||
assert RETIRED — consent = typed disclosure ack + tester-agreement line), single VERSION source
|
||
with a cross-check against the hub Setup-tab const written as `scripts/hostinstall_gates.py` —
|
||
**but NOT enforced: the script exists, is RED today (1.19.0 != 1.22.0) and is invoked by nothing**
|
||
(tracked as R-94 leg (b), class R-29) — default ACL
|
||
keeps felhom-pbs (doc advice REVERSED — narrowing caused the drill 403). Hub: per-customer
|
||
`dr_tier` flag (new default ON; legacy backfilled from descriptor reality, one-time),
|
||
cascade-not-error semantics + per-stage status on the edit form, WG-registration
|
||
auto-provision hook (scenario A hands-free), offsite-requires-DR guard (exact F-6 message),
|
||
host-page capability chips (agent 0.86.0 `inactive` = neutral; pre-1.15 "binary not found" →
|
||
migration hint). felhom-pve's latent F-3 state fixed live; Peti's box unreachable (his F-3
|
||
check + D/E/G deferred).
|
||
- **2026-07-12 — HUB v0.50.0 SHIPPED + LIVE: customer-claim password arc (closes DRILL-day0-vm
|
||
F-4/F-5).** The customer OWNS the dashboard password via a one-time claim code the hub emails to
|
||
the registered address (`bcrypt(code)` stored, plaintext only in the send). `internal/claim`
|
||
engine (EnsureIssued idempotent Day-0/live-box issue+email, Resend, RequestReset 3/day,
|
||
MarkClaimed set-only) + `store.customer_claims` (RotateClaimCode bumps generation + preserves
|
||
claimed_at) + delivery (config bake `web.claim_code_*`, report-ACK `claim` object, `POST
|
||
/api/v1/claim/reset-request`) + Setup-tab chip/resend + 3 Hungarian emails. Pairs with controller
|
||
**v0.122.0** (the gate). **LIVE-proven on drill box demo-vm-felhom: gate ON via the real edge (/ →
|
||
302 claim page, /api → 401); code emailed to the registered address.** Deployed felhom-hub:0.50.0
|
||
(ArgoCD synced/healthy). Floor raise 0.120→0.122 = operator's supervised LAST step. Observed: the
|
||
hub issues a claim code for EVERY reporting customer on first report (benign for pre-0.122 boxes —
|
||
no gate, ACK ignored). Report: documentation/audits/DRILL-day0-vm-2026-07-12.md (F-4/F-5 RESOLVED).
|
||
- **2026-07-12 — HUB v0.49.0 SHIPPED: Edit tab merge (edit-a) + scoped auto-refresh + riders**
|
||
(CHANGELOG v0.49.0; commits `e740147` → `2e03de1` → `1d94b1a` + docs/manifest). The standalone
|
||
customer edit page merged into the customer page's **Edit** tab (renamed from Settings): shared
|
||
`config_form_body` sub-template + `configFormData` builder (REUSE.md rows added), Block/Delete
|
||
relocated to a danger-zone card, header cleaned, `GET /configs/{id}/edit` → 302 `#tab=edit`,
|
||
post-action redirects anchored to their tab, `settings→edit` legacy-hash alias. Update handler
|
||
gained server-side required-field validation (error path preserves SUBMITTED values —
|
||
red-proofed). The 60s auto-refresh is now scoped: fires only on live tabs
|
||
(`data-live-tabs="overview,applications,events,host"`) and never while a form is dirty; muted
|
||
`(paused)` hint. Riders: `/style.css?v={{hubVersion}}` cache-bust on every template (closes the
|
||
v0.47.0 max-age=3600 gotcha) + the CLAUDE.md `git add -A` ban codified. **NOTE: the task spec
|
||
said v0.48.0, but v0.48.0 (app_start_failed) had already shipped — published tags are never
|
||
re-pointed, so this train is v0.49.0.** Out of scope (observed, not done): type-to-confirm for
|
||
customer delete (still `confirm()`, relocated as-is); the workspace-root `E:\git\CLAUDE.md` and
|
||
other repos' CLAUDE.md files still carry `git add -A` snippets — flag for their own sessions.
|
||
- **2026-07-12 — Day-0 VM DRILL COMPLETE (auto-confirm FIRST LIVE FIRING): full arc proven on a
|
||
fresh nested-PVE box** — appliance Day-0 → floor-at-first-report → ceremony → **auto-confirm
|
||
pending→escrowed in ~7.5 min, zero clicks** → offsite backup + restore round-trip. Two HIGH gaps:
|
||
**F-4 no operator password-set path exists (G10 unclosable, dashboards born OPEN — blocks tester
|
||
gate)** and **F-6 identity-only ceremony never implemented (no-PBS appliance can't escrow — drill
|
||
forked to PBS DR tier = full Peti-sequence rehearsal, all green)**. Installer fresh-box gaps:
|
||
felhom-pbs-apply not shipped (F-7), `age` missing (F-10), root-owned guests/ parents (F-3 —
|
||
check demo for the latent copy), silent root@pam rotation UX (F-8). Runbook fixes committed
|
||
(day0 A.2 anonymous-fetch; escrow-ceremony identity-only claim CORRECTED + age prereq). Report:
|
||
`documentation/audits/DRILL-day0-vm-2026-07-12.md`. Drill VM qm 300 kept (3 snapshots) for
|
||
re-drills; teardown list in report §9.
|
||
- **2026-07-12 — CAMPAIGN-3 Task A SHIPPED: agent v0.85.0 boot/recovery plane + appliance self-heal.**
|
||
Fixes F12 (CRITICAL boot ordering cycle — templates drop network-online, `MigrateNetworkUnits`
|
||
repairs installed units), F11/F10/F9 (reassert: fstype-driven classify, reset-failed+rearm, per-unit
|
||
verdict, + re-arm-on-guest-blind matrix-correction), F2/F1 (zero residue: reset-failed + rmdir), the
|
||
guest-hook rc255 belt (recover+timeout, wrapper no longer execs), and the appliance node self-heal
|
||
(`internal/selfheal`: host networking recovery, gated on `deployment_mode` — host-install `--mode`
|
||
templates it; byo remedy unreachable). 3 sudoers grants added (reset-failed, rmdir, start networking).
|
||
**Live matrix on demo felhom-pve: host reboot ×2 = 0 ordering-cycle lines both boots (F12 dead);
|
||
appliance self-heal recovered networking on attempt 1; F10 reset-failed+rearm sweep loud; zero
|
||
residue; guest-reboot heals.** **NEW finding F13 (HIGH, deferred):** active nfs4 under mp8 can fail
|
||
PVE rbind (rc255); NAS-down-at-boot → apps `Created` (fix-3). Peti's box (agent 0.81) unaffected
|
||
until his rollout — carries the F12 coin flip if it has a NAS share (interim drop-in in the agent
|
||
REPORT). host-install v1.14.0; scripts CHANGELOG. Details: `documentation/controller/network-storage-nas.md`.
|
||
- **2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; host recovered, RCA closed.** Unattended chaos campaign
|
||
22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/
|
||
restore byte-identical/EIO instant/stub badge + deploy-409 live-validated). The NAS automount
|
||
plane produced 2 CRITICAL + 3 HIGH: **F12 CRITICAL (= the overnight host loss, RCA'd next
|
||
morning): the agent automount template's `After/Wants=network-online.target` creates a boot
|
||
ordering cycle — systemd deletes an arbitrary job to break it; boot -1 sacrificed networking
|
||
(host dark 7h, hardware exonerated), boot 0 sacrificed the automount (NAS dead). Every host
|
||
boot with an enrolled share = coin flip until the template drops that ordering (`_netdev` on
|
||
the .mount suffices).** F10 CRITICAL (start-limit never re-armed, once blocked guest start),
|
||
F11/F9 HIGH (post-start reassert "skip-active" skips the healing automount restart; agent-restart
|
||
sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots; F7 HIGH (in-place dump write →
|
||
0-byte tar replaced last good backup during outage window). Fix-6 numbers: ring horizon 55min
|
||
idle / **6.5min under load**, wiped on every restart. Box left in defined state (6 wave apps
|
||
healthy, temps removed, NAS baseline-identical). Full ledger/evidence:
|
||
`documentation/audits/CAMPAIGN-3-2026-07-11.md` + `180:~/campaign3/`.
|
||
- **2026-07-11 — HUB v0.47.0 UI REORGANIZATION SHIPPED** (CHANGELOG v0.47.0; commits `9f29bf3` →
|
||
`0daddcd` + docs). Five deliverables: **(1)** `.data-table td a:not(.btn)` button-contrast fix;
|
||
**(2)** customer page = **8 hash tabs** (`#tab=…`, sticky summary strip, Events error badge,
|
||
no-JS graceful degradation — panels hide only via a JS-added body class); **(3)** shared
|
||
`host_detail_body` sub-template rendered by `/hosts/{id}` AND the new per-customer **Host tab**
|
||
(a list by design; `store.ListHostsByCustomer` + `hostDetailData` builder); **(4)** **stale host
|
||
removal** — `GET /hosts/{id}/delete-impact` (counts/booleans only) + `POST /hosts/{id}/delete`
|
||
behind type-to-confirm; ONLINE → 409 always (no override), escrow needs an explicit checkbox
|
||
(`ErrHostEscrowPresent`, tx never starts), one-tx cascade incl. the bound wg peer (wgsync's 5-min
|
||
declarative push converges the endpoint; log bundles die by `scope_id == host_id` only);
|
||
**(5)** **/offsite multi-endpoint management UI** — all `wg_endpoints` rows as cards +
|
||
add/edit/delete with 409 guards (peers-in-subnet pins subnet + delete). **Deferral (explicit):
|
||
peer allocation, the wgsync reconciler push and the desired-state merge stay lowest-endpoint-id
|
||
(`GetWGEndpoint` untouched); the future arc is a `wg_peers.endpoint_id` migration + per-endpoint
|
||
allocation/reconciler/desired-state.** Five red-proofs ran (online gate, escrow ack, bundle
|
||
scope, endpoint-delete guard, subnet-change guard). *Resolved:* the earlier ⚠️ about `146d165`
|
||
sweeping the Part-4 WIP — the sweep caught `hosts.go` mid-red-proof (escrow ack bypassed);
|
||
`068427a` restored the gate; the arc is now complete and author-reviewed. **Stale-doc note:** the
|
||
workspace-root CLAUDE.md's hub deploy wording predates GitOps — deploys are manifest-tag bumps +
|
||
deliberate ArgoCD sync, never `kubectl set image`.
|
||
|
||
- **2026-07-11 — RCA FIXES 1+2+4 SHIPPED + Q1c GREEN (agent v0.84.0 + controller v0.117.0 LIVE on
|
||
demo).** Reboot survival: automatic since agent 0.84.0, live-proven 2026-07-11 21:20 (`pct reboot
|
||
9201` under operator GO — the hook re-armed + verified nas-media DURING the start task, CWA
|
||
auto-started healthy, health ok first cycle, no badge). Agent: `ReassertNetworkMounts` (hook
|
||
post-start + startup; stop+enable-now on existing sudoers verbs — NO sudoers widening needed;
|
||
never in periodic paths). Controller: statfs consuming-namespace classifier → probe
|
||
`not_network_fs` (stub can never verify), deploy-time stub refusal (409; idle autofs deploys
|
||
fine), distinct stub badge (wins over unreachable; unreachable line byte-identical), deploy view
|
||
selects stored HDD_PATH (RCA fix 4). MinAgent unchanged 0.81.0. All red-proofs recorded in repo
|
||
REPORTs. Feature doc: `documentation/controller/network-storage-nas.md` §Reboot survival; audit §7
|
||
follow-up added. Queued: RCA fixes 3/5/6; Peti publish train now carries 0.84 + 0.117.
|
||
|
||
- **2026-07-11 — NAS/CWA RCA (diagnostic, no fixes):** `documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md`.
|
||
The nas-media re-add verified legitimately (new host automounts DO propagate into running
|
||
guests); a second `pct reboot 9201` at 15:39 severed the trigger → guest-side stub; CWA
|
||
deployed correctly to the NAS (app.yaml + binds + data all nas-media) but is DOWN since 15:39
|
||
(docker can't recreate the bind source; perms luckily blocked any system-drive writes; zero data
|
||
loss). UI "felhom-usb" = display bug (deploy.html selects by IsDefault, never stored HDD_PATH);
|
||
hub device-ID WARN was CORRECT and the only guard that fired; NAS badge masks because it asks
|
||
the agent (host ns). 6 prioritized fix tasks in the audit §5. **Remediation §6 PERFORMED on
|
||
operator GO (~20:14–20:20 CEST): automount-unit restart re-planted the trigger in the running
|
||
guest instantly (fix-1 mechanism live-proven), CWA restarted healthy on the NAS library,
|
||
uid-1000 write OK, controller health back to status=ok at 20:18. Demo fully healed.**
|
||
|
||
- **2026-07-11 — OBSERVABILITY PASS (hub v0.46.0 LIVE Synced/Healthy; agent v0.83.0 + controller
|
||
v0.116.1 on demo).** Remote pull-only log access to both box components' always-DEBUG capture
|
||
rings. Hub: `log_bundle_requests`/`log_bundles` store (gzip, newest-3, **72 h TTL** on the 60 s
|
||
sweep), the **secret gate fail-closed** before storing (`blocked: possible secret`; `[REDACTED]`/
|
||
checksums pass), report-ACK `controller_log_requested` + heartbeat-envelope `log_tail_requested`
|
||
channels (consume-once; pre-0.83 agents stay visibly pending), host-detail Diagnostics section
|
||
(Request controller/agent logs + View/Download; the hosts read-only invariant amended: these two
|
||
forms are the only actions). Conventions codified:
|
||
`documentation/runbooks/logging-conventions.md` + CLAUDE.md one-liners in all three repos.
|
||
**OPEN: operator clicks both request buttons on hub.felhom.eu/hosts/<demo> to close the live
|
||
round-trip** (UI password-gated; the flow is fully API-test-proven). Hub bearer key rotation
|
||
still DUE (pre-existing).
|
||
|
||
- **2026-07-11 — PUBLISH-TRAIN RULES CODIFIED** (`documentation/runbooks/publish-train-rules.md`):
|
||
1. manifest before floor; 2. the manifest screen carries the LIVE DB floor
|
||
(`hub_settings.min_controller_version` overrides env, acts immediately) — save the floor field
|
||
LAST; 3. MinAgent gate — **now HUB-ENFORCED PER-BOX (hub v0.45.0, `store.ResolveManagedFloor`),
|
||
the manual fleet check retired**; 4. box-level backstop = the controller's agent-capability gate
|
||
(v0.114.0 route probe → v0.115.0 version compare). (Floor-UI separation + per-box gating SHIPPED
|
||
in hub v0.45.0 — see the backlog-clear entry above; this rules-doc note supersedes the original
|
||
"hub untouched / option-3 deferred" wording.)
|
||
|
||
- **2026-07-11 — NAS/coupling backlog CLEARED (hub v0.45.0 LIVE + agent v0.82.0/ctrl v0.115.0 on demo).**
|
||
Composite A–F. Hub v0.45.0 (Synced/Healthy): floor-UI separation + effective-floor source line
|
||
(DB `hub_settings` vs env, `store.ResolveGlobalFloor`) + type-to-confirm w/ live below-floor count;
|
||
per-box MinAgent conditional floor (`store.ResolveManagedFloor` HOLDS the controller floor for a box
|
||
whose `hosts.agent_version` < the manifest's MinAgent, flags it on the Hosts dashboard) — manual
|
||
fleet check retired (publish-train-rules.md rule 3). THE one comparator extracted to
|
||
`internal/semver`. Agent v0.82.0 = `X-Felhom-Agent-Version` header (demo-only, Peti stays 0.81 =
|
||
probe fallback); ctrl v0.115.0 = version-aware Supports + DSM-validated NFS guidance. **DSM spike**
|
||
(real DSM 7.2 via virtual-dsm): consumer recipes validated E2E, hardlink caveat closed for Synology
|
||
(record: documentation/audits/SPIKE-nas-dsm-2026-07-11.md); QNAP caveat stands. **Q1c (supervised
|
||
reboot) FAILED**: a NAS automount trigger does NOT survive a guest reboot (empty-dir; agent lacks a
|
||
network-mount reassert) — fix spec'd at documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md.
|
||
NOT published (agent 0.82 demo-only). Record: felhom.eu/REPORT.md + each repo's REPORT.
|
||
|
||
- **2026-07-11 — PUBLISH TRAIN 0.81/0.113 COMPLETE — PETI LIVE on agent 0.81.0 + controller 0.113.0.**
|
||
Record: `documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md`. Agent 0.81.0 published
|
||
(sha `914bb3b5…30df0b`, live-bytes, anon-verified) + golden 0.113.0 baked/published (sha
|
||
`66de5e64…82054c`, drill VM, virgin-reverted); manifest + signed `agent_update`
|
||
(key-id `felhom-op-1`) by Viktor → Peti 0.79.0→0.81.0 clean (no rollback); **floor auto-confirm
|
||
proof CAPTURED** (10:41 0.112.0 → 10:42 0.113.0, anonymous pull proven on HIS box) — KEY LEARNING:
|
||
the hub UI manifest screen ALSO sets the GLOBAL floor as a DB `hub_settings.min_controller_version`
|
||
row that OVERRIDES the env and acts immediately (this is how Peti reached 0.112 and now 0.113
|
||
before the GitOps env bump `16390d0`); a transient forbidden skew (ctrl 0.113 + agent 0.79,
|
||
~10 min) was benign only because zero NAS shares existed — next train: save the floor field LAST.
|
||
OPEN: Peti journal-group one-liner (E), temp Git Sync creds deletion (G1), **hub bearer key
|
||
rotation now DUE** (committed in manifests/hub.yaml + screenshot-exposed this session).
|
||
|
||
- **2026-07-11 — NAS verify-before-commit SHIPPED (agent v0.81.0 + controller v0.113.0 LIVE on demo;
|
||
host-install v1.13.0).** The spike's production task landed same-day: add = agent verify (retry=0
|
||
unit, journal-classified, auto-rollback) → controller uid-1000 write probe (--netprobe re-exec) →
|
||
register LAST; full rollback on any failure; §3.2 Hungarian error map (nfs_export merged);
|
||
storage_network.html redesigned (SMB-first, Route-A guidance); orphan rows; systemd-journal group
|
||
(NO sudoers). Live A–E validated vs an isolated sim NAS, Route A proven in production (alien-uid
|
||
1060 server-side). Feature doc: `documentation/controller/network-storage-nas.md`. NOT published
|
||
(no Gitea/Day-0 bump; Peti pending with the usermod one-liner). Record:
|
||
`felhom-controller/REPORT.md`.
|
||
|
||
- **2026-07-11 — SPIKE: NAS verify mechanisms + NFS squash matrix — ALL FIVE QUESTIONS ANSWERED, no
|
||
blocker.** Record: `documentation/audits/SPIKE-nas-verify-2026-07-11.md`. Q1 in-guest access DOES
|
||
wake the host automount (make-or-break PASS; container bind-mount too); Q2 controller drops to uid
|
||
1000 and writes (setpriv proof; SETUID/SETGID caps present); Q3 **Route A ACCEPTED-with-caveats**
|
||
(`all_squash` to ANY server uid = full rw+hardlink for the guest app; guest sees nobody; chown
|
||
EPERM-clean, chmod persists; June's `anonuid=1000` WRONG case pinned = anonuid without all_squash
|
||
is a no-op for non-root); Q4 error taxonomy captured (NFSv4 makes no-export vs not-permitted
|
||
INDISTINGUISHABLE; systemd 90 s start-timeout is the real ceiling; **`retry=0` → production unit**:
|
||
dead-NAS access 91 s→3.8 s; verify must NOT ride the 15 s agentapi timeout → own path/status-poll,
|
||
~120 s budget); Q5 plain-user SMB PASS incl. hardlink. Teardown verified to pre-counts. Production
|
||
TASK is the operator side's next move.
|
||
|
||
- **2026-07-11 — PBS DR tier SLICE 2 SHIPPED (agent v0.80.0): the apply-bridge — demo
|
||
ADOPTION-PROVEN live + PUBLISHED; the supervised Peti phase PENDING.** Full chain fired on the
|
||
demo: hub enable (storage-id `felhom-offsite`) → tenantsync provisioned ns `demo-felhom` on ep0
|
||
→ agent ADOPTED the manual entry (tenancy entry-owned, "entry wins"; staged secret asserted
|
||
UNCONSUMED; DR green; `escrow.pbs_storage_id` seeded). Laws red-proof-verified: set-only (K
|
||
destruction guard), secret-on-stdin, verify-pin-before-consume, loud consumed-failed. Part 0
|
||
evidence: write path needs the §4b dual-grant exactly (403 `Datastore.AllocateSpace` → re-assert
|
||
→ token vzdump OK); demo grants restored. Live finding fixed: escrow seed = in-place write
|
||
(/etc/felhom-agent dir is root-owned; rename impossible for the non-root agent). 0.80.0
|
||
published (`f2ba62ca…f06d2`, == deployed bytes). **NEXT = RUNBOOK-peti-pbsdr-2026-07-11.md**
|
||
(Peti box prep: sudoers+wrapper+wg_tunnel enable — no tunnel/OOB on his box today; Viktor opsign
|
||
0.80.0; hub enable → FRESH path, K born, degradation clears; ceremony → auto-confirm → first
|
||
offsite run = the arc closes). Detail: felhom-agent/REPORT.md + CHANGELOG v0.80.0.
|
||
|
||
- **2026-07-10 — PBS DR tier SLICE 1 SHIPPED+LIVE (hub v0.44.0): ep0 tenantsync surface + hub
|
||
provisioning.** `felhom-tenantsync` v1.0.0 installed on ep0 (second forced-command key, peersync
|
||
untouched — hashes asserted; smoke caught + fixed the client `{"data":[…]}` json wrap; orphan
|
||
`root@pam!spike` cleaned). Hub: `internal/tenantsync` (pinned-SSH twin of wgsync; errors never
|
||
embed stdout), config-form "PBS DR tier (ep0)" section → WG-peer check (fail-closed) → provision →
|
||
`host_pbs_secrets` consume-once (agent fetches `POST /api/v1/hosts/{id}/pbs/consume-token` —
|
||
note: /hosts/, not the task's /host/) → `pbs_dr` descriptor into HOST desired_json + gen bump;
|
||
idempotent re-save; Re-issue button (F4 twin). Red-proofs: consume-once / fail-closed /
|
||
idempotency all mutation-proven. **NOT live-provisioned — slice 2 (agent bridge) + supervised
|
||
Peti enable pending.** Runbook §10 addendum in offsite-endpoint.md. Commits ce6a566 + 32ca1f6.
|
||
|
||
- **2026-07-10 — SPIKE: PBS DR tier auto-provisioning SLICE 0 — all three mechanisms SETTLED.** Record:
|
||
`documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`. (1) PVE token storage-create **impossible**
|
||
(create/modify/delete all check `/storage` root) → **sudoers vector proven** (pinned-id alias, cycle green as
|
||
the agent user, unpinned denied; secret must ride a stdin-wrapper, never sudo argv). (2) ep0 tenancy op-set +
|
||
one-time-secret-over-SSH-stdout **proven live on ep0** (🛑 Viktor-approved; read/write cross-tenant 403s);
|
||
current peersync channel can't carry it → slice 1 ships a second forced-command surface (`felhom-tenantsync`).
|
||
(3) `pvesm add pbs --encryption-key autogen` births K at `PBSEncKeyPath` + vzdump lands ciphertext (doc-06
|
||
property) + escrow-create finds K. **⚠ `pvesm remove` DELETES K** → re-apply must be set-only. All spike
|
||
artifacts torn down; flags: orphan `root@pam!spike` on ep0 scratch; demo's §4b step-3 storage grants absent.
|
||
Epic slice map in the record §5.
|
||
|
||
- **2026-07-10 — new-box restore round-trip VALIDATED + old-box archive RETIRED.** Record:
|
||
`documentation/audits/VALIDATION-newbox-restore-2026-07-10.md`. Both toggled apps restored from the
|
||
production pool box via the real endpoint — **9/9 sha256-identical** (immich 44MB db-dump exact,
|
||
`d78466fcbf…4526b`, agreeing with the campaign's independent round). ~10s/app. **Viktor's decision:
|
||
RETIRE the old archive** — he deletes `u629193-sub1` in the old project's console (the PBS-storage-1 box
|
||
itself stays for dev). C2 fixed same day as controller v0.110.0 (crash-lock self-heal, live-proven).
|
||
Offsite arc: build-complete + restore-proven both directions; remaining = the Peti onboarding runbook.
|
||
|
||
- **2026-07-10 — offsite arc overnight soak (unattended) — PASS + 1 HIGH finding.** Record:
|
||
`documentation/audits/CAMPAIGN-offsite-overnight-2026-07-10.md`. F4 re-issue proved LIVE (reset invalidates
|
||
the old password); restore round-trip 9/9 byte-identical new-box↔USB (immich db-dump exact → old-box
|
||
archive `u629193*` safe to retire, Viktor's decision); single-flight + crash-restart (zero consume, escrow
|
||
untouched) PASS. **FINDING C2 (HIGH, OPEN):** a controller crash MID-PRUNE leaves a stale restic exclusive
|
||
lock the controller can't self-heal → offbox runs fail until manual `restic unlock` (fix candidate:
|
||
`ensureOffboxRepo` auto-unlocks a self-owned stale lock). Track B (live quota-crossing) + hub-orchestrated
|
||
Track A blocked — the hub operator config surface is session/CSRF-gated (unattended can't reach it) → Peti
|
||
runbook. Demo restored to start-state. One expected `backup_failed` email (the crash test).
|
||
|
||
- **2026-07-10 — SLICE 4 soft-quota + offsite health SHIPPED+LIVE (hub v0.41.0 + controller v0.109.1) —
|
||
the offsite arc is BUILD-COMPLETE.** Controller: `quota_gb` rides the descriptor into the target (hash
|
||
includes it → quota changes re-apply via key-auth-first, no consume); `repo_size_bytes` from restic
|
||
stats; pre-run gate (≥100% refuses NEW backups Hungarian-loudly, **prune still runs** — red-proofed;
|
||
restore never gated; ≥80% warns); /backups usage bar. Hub: `monitor.OffsiteChecker` (fill 90/95 of
|
||
quota + `offsite_stale` >48h for enabled+escrowed only — red-proofed; nil-safe on old reports) +
|
||
**manual-only Freeze/Unfreeze lever** (readonly on the exactly-1 labelled sub-account, SSH preserved).
|
||
**Live:** key-auth-first proven against the real box (the deploy's own re-apply); that re-apply exposed
|
||
+ fixed v0.109.1 (re-apply must preserve EscrowState + runtime status — it demoted the escrowed demo);
|
||
demo repaired (manual confirm + run: escrowed, 45 MB / 50 GB, checker healthy 0 events, usage bar
|
||
renders). Freeze + 100%-refusal NOT live-exercised (STOP; red-proofs cover). NEXT: Peti onboarding
|
||
runbook, NEW-box restore round-trip → old-box retirement.
|
||
|
||
- **2026-07-09 — SLICE 3 hub-verified escrow auto-confirm SHIPPED+LIVE (hub v0.40.0 + controller v0.108.0 +
|
||
agent v0.79.0).** Replaces the operator-trust confirm with a verified fact: ceremony hashes the sealed
|
||
repo password (agent, `restic_pw_sha256,omitempty` on the escrow upload — contract test moved in lockstep)
|
||
→ hub stores it (additive `host_escrow` migration, NULL on legacy rows) + serves
|
||
`escrow:{identity_blob_present,restic_pw_sha256,created_at}` in the report ACK → controller
|
||
(`report.EscrowAutoConfirmer`) flips pending→escrowed ONLY on hash match with its local repo password.
|
||
**Red-proofed core: blob-presence alone never confirms** (stale blob = false custody claim); mismatch
|
||
warns once per hash naming the ceremony; never un-confirms; flip wipes the agent-staged secret. Hashers
|
||
pinned by the same cross-repo vector. Manual confirm = deprecated fallback (legacy blobs — demo included,
|
||
hash-NULL verified live; the escrowed demo produced zero confirm activity = Scenario E live). Full happy
|
||
chain proves out at the next real enable (Peti). NEXT: SLICE 4 soft-quota, Peti runbook.
|
||
|
||
- **2026-07-09 — offsite hardening bundle SHIPPED+LIVE (hub v0.39.0 + controller v0.107.0 + agent v0.78.0).**
|
||
The e2e's sharp edges closed before slices 3–4: **F4** "Re-issue offsite credentials" (explicit operator
|
||
recovery for consumed-password dead-ends — exactly-1 label scope, fresh one-time secret, ConfigVersion
|
||
bump; `hetznerapi.ResetBoxPassword` added for dedicated); **F2** host-key scan retry (~60s ladder — first
|
||
save survives fresh-subaccount DNS lag); **F5** submit-disable + in-flight notice; **key-auth-first bridge**
|
||
(working installed key → re-pin+reconfigure WITHOUT consuming; fingerprint verify never bypassed; kills the
|
||
stale-descriptor consume-404 loop); **staged-secret wipe** (agent `DELETE /escrow/stage-secret`; controller
|
||
calls it on every flip to `escrowed`; the e2e's lingering staged file wiped live through the new endpoint,
|
||
idempotency proven). All red-proofed. F4 reset NOT exercised live (demo credential untouchable — throwaway
|
||
customer under supervision when needed). NEXT: SLICE 3 auto-confirm, SLICE 4 soft-quota, Peti runbook.
|
||
|
||
- **2026-07-09 — offsite provisioning VALIDATED LIVE end-to-end (hub v0.38.1 + controller v0.106.1) — PASS.**
|
||
Full loop through the real pipeline on `demo-felhom` against the new dedicated-project pool box (611714 /
|
||
`u629488`, project 15282031): UI save → sub-account 269008 provisioned → descriptor v6 + one-time password →
|
||
controller config-refresh (≤15 min) → apply-bridge (verify-pin → consume-once → pinned `ssh-copy-id` →
|
||
`EscrowState=pending`) → run-gate refusal proven → escrow confirm (Option A: the repo password was already
|
||
escrowed — `WriteOffboxSecrets` preserves it) → first run: **restic repo + 2 snapshots on the new box,
|
||
key-only**. Two live bugs found+fixed mid-run: **F1** hub v0.38.1 (`applyOffsite` on `context.WithoutCancel` —
|
||
a re-click canceled the provision between create and password-store, stranding the sub-account) and **F3**
|
||
controller v0.106.1 (`ssh-copy-id -s` needs `~/.ssh` to exist in the container — died AFTER consuming the
|
||
password). OPEN: F2 (fresh sub-account DNS lags → first-save scan 502s; retry-with-backoff), F4 (hub has no
|
||
password-reset path — recovery today is delete-subaccount + re-save), F5 (no save spinner → re-click bait).
|
||
Phase-0 wiring: `manifests/hub.yaml` now injects `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID` from
|
||
Secret/storagebox; scoped-token check = list boxes with it (must see ONLY 611714). Old-box offsite kept as
|
||
frozen archive until a NEW-box restore round-trip. Record:
|
||
`documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`. NEXT: SLICE 3 (auto-confirm),
|
||
SLICE 4 (soft-quota), F2/F4/F5, Peti onboarding runbook.
|
||
|
||
- **2026-07-09 — offsite provisioning SLICE 2 (hub v0.38.0 + controller v0.106.0).** The controller apply-bridge:
|
||
on startup it reconciles the hub-served `offsite:` descriptor into a key-only offbox target
|
||
(`controller/internal/offsiteapply.Bridge`) — **verify-pin the box host key against the hub-captured
|
||
`host_fingerprint` (no blind TOFU)** → consume the one-time password (single-use) → `sshpass ssh-copy-id -s -f`
|
||
install → configure offbox → `EscrowState="pending"` → persist a descriptor-hash marker. Idempotent +
|
||
fail-safe; both red-proofs green. Hub v0.38.0 adds `Descriptor.HostFingerprint` captured via an
|
||
`x/crypto/ssh` keyscan (fail-closed). **NOT yet live-applied** — supervised end-to-end (hub provisions on
|
||
the new pool box → controller apply) is the next runbook, gated on the hub's new scoped `HETZNER_TOKEN`.
|
||
NEXT: SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota).
|
||
|
||
- **2026-07-09 — hub offsite provisioning SLICE 1 (hub v0.37.0).** The hub can now provision the offsite tier
|
||
on operator enable: `internal/hetznerapi` (typed client, base **api.hetzner.com/v1**, `CloudAPI` interface +
|
||
exported `Fake`, `WaitAction`), `internal/offsite` (`Provisioner.ProvisionOffsite` — idempotent by label
|
||
`felhom-customer=<id>`, shared sub-account / dedicated box, generates the transient password, builds the
|
||
NON-SECRET descriptor, **fail-closed**), `one_time_secrets` store (single-use `Save/ConsumeOneTimeSecret`),
|
||
`POST /offsite/consume-password/{id}` (customer-API-key auth, once), config-form Offsite section →
|
||
`applyOffsite` (502 + no-save on error) → descriptor in ConfigJSON → ConfigVersion bump. Token from
|
||
`HETZNER_TOKEN` (out-of-band); password NEVER in ConfigJSON/logs. Unit-tested vs a faked Cloud API (+ a
|
||
fail-closed red-proof). **NOT yet live-provisioned** — awaiting the **dedicated-project scoped token** (the
|
||
current token can delete ep0). NEXT: SLICE 2 (controller apply-bridge consuming the descriptor + one-time
|
||
password), SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota). Deployed via GitOps.
|
||
|
||
- **2026-07-09 — Hetzner API provisioning spike (offsite tier).** Both offsite models are API-provisionable
|
||
(FIT). **#1 correction: the Storage Box API is `https://api.hetzner.com/v1`, NOT `api.hetzner.cloud/v1`**
|
||
(Cloud API 404s for storage boxes); same Bearer token. Proven live on `spike-*` (torn down): sub-account
|
||
create/reset/readonly/delete + a billable bx11 box create/rescale(`change_type`)/delete. Create takes a
|
||
**password (no ssh-key param)** → transient-password→`ssh-copy-id -s -f`→restic-reach→`reset_subaccount_password`
|
||
(transient pw invalidated, key survives) = key-only steady state. RepoPath `/home/<repo>` (chroot). Quota:
|
||
dedicated=box type/`change_type` (hard); shared=no native quota → Felhom soft-quota + `update_access_settings{readonly}`
|
||
freeze. Idempotency = **label_selector** (box names not unique). Rate 3600/h. **SECURITY: the token is
|
||
project-wide unscopeable Read+Write — it can also touch ep0 (the prod offsite server) + 3 SSH keys →
|
||
move customer boxes to a DEDICATED Hetzner project before the hub holds a write token.** Record:
|
||
`documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md`. NEXT: the hub-provisioning impl.
|
||
|
||
- **2026-07-09 — fork-4 custody VALIDATED end-to-end live (supervised).** The full custody cycle proven on
|
||
the demo (agent v0.77.0 + controller v0.105.0): enable → controller pushes the repo password to the agent →
|
||
**real escrow ceremony** (`--selftest=escrow-create`, K + restic-pw under a fresh R, uploaded to the hub,
|
||
staged secret wiped, K byte-unchanged) → **atomicity proof:** a run **skipped** before `confirm-escrow` and
|
||
**proceeded** after (2 apps, 2 snapshots) — the gate opens exactly on confirm. **DR money shot:** the restic
|
||
password recovered from the **production hub blob** (`host_escrow.identity_blob` for `demo-felhom-01`, via
|
||
`identity-consume` with R) is byte-identical to the live password (`sha 24c44f49…`) and **decrypted the real
|
||
box repo**; the real `inject-password`→`restore` endpoints reproduced a 45 MB db-dump byte-perfect. Demo
|
||
offbox left **escrowed** (un-paused); Viktor holds R out-of-band; no secret in any log; live `repo_password`
|
||
unchanged. Record: `documentation/audits/VALIDATION-fork4-custody-2026-07-09.md`. **NEXT:** hub-provisioning +
|
||
soft-quota spec (hub-verified auto-confirm replacing operator-trust; both storage models); per-customer
|
||
onboarding runbook; the `identity-consume` OK-message fix (omits the recovered restic field).
|
||
|
||
- **2026-07-09 — offbox v0.104.0 VALIDATED live against the Storage Box (supervised).** Closed the offsite
|
||
arc: controller `:0.104.0`@`908e4b9` ran the real `/backup/offbox/*` endpoints against a fresh box
|
||
**sub-account** (`u629193-sub1`, RepoPath `/home/felhom-demo-repo`). The capstone proof — **immich**
|
||
(undeployed, recovery unit only on the USB drive, the exact case v0.103.0 mis-resolved to `systemDataPath`)
|
||
→ discovery found the USB unit → `ok`, 1 snapshot; 45 MB db-dump restore sha256-matched. No-silent-success
|
||
confirmed on the live box: 0/N → `error` + `backup_failed` operator alert; partial → `ok` +
|
||
Hungarian `LastWarning` on `/backups`. Persists across restart. **Forks:** (1) direct topology + (2)
|
||
sub-account-per-customer (RepoPath `/home/<name>`, `ssh-copy-id -p 23 -s -f` — console SSH-key field still
|
||
broken, F3) confirmed; (3) PBS/WG tier deferred; **(4) custody surface RECORDED, escrow design OPEN** —
|
||
the per-customer restic repo password + SSH key are 0600 in the controller data dir, **NOT independently
|
||
recoverable** (ride DR only via the PBS whole-CT) → the load-bearing follow-up before real reliance.
|
||
**KEPT** as a standing demo-offsite (€4.06/mo). Record: `documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md`.
|
||
|
||
- **2026-07-09 — TASK GL-7 SHIPPED: customer-page passphrase hardening + install-command generator
|
||
(hub v0.36.0, `844fbfa7`; deploy `02c748eb`; LIVE, ArgoCD Synced/Healthy)** — go-live G7/G9/G10/G11.
|
||
Two coupled, security-first changes to the operator customer page (`customer_unified.html` +
|
||
`configs.go`). **(1) Passphrase hardening:** the per-customer retrieval passphrase was cleartext
|
||
twice (visible `#retrieval-pw` + baked into the Option-3 debug curl `X-Retrieval-Password:` header);
|
||
now masked-by-default with reveal (`toggleSecret`) + copy (`copySecret`), value in `data-secret`
|
||
(existing model), and the Option-3 command carries a `<YOUR-RETRIEVAL-PASSWORD>` placeholder — no
|
||
secret in any copyable command. (Zero-secret-in-DOM fetch = noted future follow-up, NOT scoped.)
|
||
**(2) Install-command generator:** the three hard-coded `<code>` blocks became a client-side builder
|
||
(vanilla JS, no framework/CDN/network) emitting ONLY real host-install v1.12.0 flags, download-then-
|
||
run, byo-requires-caps enforced client-side, CustomerID prefilled from the server, JS-off static
|
||
fallback; the seven dangerous/operator-only flags (`--force`/`--rotate-recovery`/`--enable-oob`/
|
||
`--remove-golden`/`--uninstall`/`--adopt-pool`/`--rescope-acl`) are never offered as controls. Tests
|
||
`TestTemplates_PassphraseHardened` + `TestTemplates_InstallGenerator` (green; passphrase test
|
||
red-proofs). **Tester agreement written** (`documentation/pilot/PETI-tester-agreement.md`): BYO
|
||
trust boundary, honest limitations (no offsite backup, F2 removable-drive, F7 pool-reassert, F6
|
||
`:53`-free), exit rights, Peti box facts (80c/128 GB → caps 12/32768, `sajatfelhom.hu` re-point),
|
||
byo install command, onboarding sequence. GO-LIVE-PACKAGE: GL-7 ✔; G9/G10/G11 → done. **One open
|
||
pilot question (owned by the agreement §6):** Peti's local backup TARGET — does his box have a
|
||
second disk/pool? → sets `--acl-storages` at onboarding. Deploy gotcha: first build was stale
|
||
(local commits not pushed before `build.sh` pulls origin) — pushed then rebuilt. **NEXT: Peti day-0
|
||
onboarding (operator-run).**
|
||
- **2026-07-08 — TASK GL-8 SHIPPED: BYO coexistence hardening (host-install v1.12.0, `18a556a5`)** —
|
||
the GL-6 findings F3/F6/F1. **F3:** the populated-host leaf guard fires only when a `felhom_guests`
|
||
guest exists (not `pct list`) → a fresh byo install on a host running only the OWNER's guests needs
|
||
no `--allow-new-leaf`; the pin protection for real Felhom guests is unchanged. **F6:** byo preflight
|
||
REFUSES a foreign process on `:53` with remediation and mutates NOTHING (Viktor's ruling: refuse+
|
||
instruct only — the dnsmasq analog of the break-glass no-touch rule; appliance untouched). **F1:**
|
||
uninstall purges the config `.bak*` siblings (a GL-6 residue held a live hub api_key). F6
|
||
live-proven on felhom-pve (byo preflight refuses at :53 — the exact GL-6 scenario, read-only die).
|
||
Harness 36/36 (+GL8-F3/F6/F1 static, GL8-F1b behavioural, GL8-F6 live) + red-proofs RP-F3/F6/F1;
|
||
GL-2/GL-4 regression green; shellcheck clean. Shipped BEFORE GL-7 so Peti's command needs no
|
||
workaround. **F2/F7 remain open (non-blocking). NEXT: GL-7.**
|
||
- **2026-07-08 — TASK GL-5 SHIPPED: DR bring-up bind overrides (agent v0.75.0, LIVE + scratch-DR
|
||
validated)** — go-live G8; docs-only here (GO-LIVE-PACKAGE G8/G6/G1 flips + decision log; the
|
||
code + full report live in felhom-agent `b3446213`). Guest-loss DR restore WORKS now: the two
|
||
structural binds (mp8/mp9) are overridden with throwaways at restore and swapped for the REAL
|
||
binds pre-start (root pct via the new Engine HostRunner seam), displaced unusedN deleted. TWO
|
||
live-discovered PVE rules beyond the spike: explicit-params restore needs an explicit rootfs AND
|
||
silently DROPS unlisted mountpoints — the full param set now derives from the archive's embedded
|
||
config (`ExtractArchiveConfig`, 200 under the scoped token). Live proof: scratch 9310 from a real
|
||
9201 archive — mp0 200G + mp1 50G restored WITH content (7m23s), real binds, zero residue, clean
|
||
teardown; the same op failed outright on v0.74.0. OPERATOR: **publish agent 0.75.0 + bump the
|
||
manifest to it** (vouching 0.74.0 ships broken DR). Follow-ups: restore-test boot-verifies
|
||
without mp0/mp1 (same PVE rule, reverse direction — candidate reuse of drRestoreOverrides);
|
||
DR-selftest KeepMAC duplicates a live source's MAC briefly; no host-loss DR runbook exists yet
|
||
(GL-6/S5 writes it).
|
||
- **2026-07-08 — TASK GL-4 SHIPPED: uninstaller gap-closure + operator-key day-0 fold
|
||
(host-install v1.11.0)** — go-live G6 + the G1 key-pin follow-up; **awaiting GL-6** for the real
|
||
(non-dry) teardown + armed-pin install. Uninstall now removes the self-update artifacts (4b4:
|
||
guarded wrapper, .prev/.new.* slots, rollback unit, limits drop-in — derived from
|
||
configs/felhom-selfupdate-guarded), unmounts enrolled drives under /mnt/felhom-drives (plain
|
||
umount ONLY, busy = warn+guidance, root-bind guarded), and ends with a KEPT-vs-WIPED statement
|
||
(drives/PBS/hub record/escrow/vaulted recovery credential live on; guest-only mode prints the
|
||
vmid's bind-store drives). Key-pin: OPERATOR_KEY_* constants (EMPTY until the operator ceremony)
|
||
+ `--operator-pubkey-file` (validated at argv, comment=key_id required) → authz.signers written
|
||
at step 6 per the agent SignerKey schema; **no-keys-resolved KEEPS preserved signers** (never
|
||
un-pin a manually-pinned box); verify reports armed/dormant (dormant = WARN). Harness 28/28 on
|
||
felhom-pve incl. the NEW GL4 H-U full-uninstall DRY transcript vs live 9201; red-proofs RP-1..3
|
||
green. **GO-LIVE-PACKAGE.md was ABSENT AGAIN** (spec said the operator attaches it; not present
|
||
in the repo or workspace) — G6/G1 status recorded here, doc still pending its operator commit.
|
||
Key CEREMONY (real keypairs + felhom-pve pin) = operator step.
|
||
- **2026-07-07 — RUNBOOK GL-1 EXECUTED: agent 0.74.0 + golden 0.103.0 PUBLISHED** — go-live G1
|
||
(partial): the published-artifact chain is current. Agent = the LIVE felhom-pve bytes (sha
|
||
`1ec3f588…76af05`, provenance preserved, publish-agent.sh round-trip verified); golden baked
|
||
clean-room in the drill VM (build-golden.sh v2.0.0 @ ced60ddf, controller 0.103.0 baked, all
|
||
integrity gates green, sha `8481e8a1…6026e`, 583MB) and published HTTP 201. Both artifacts
|
||
fetch-verified **ANONYMOUSLY** (Gate 3c finding: generic packages are world-readable → G3
|
||
git.token read-only rotation CANNOT break fetches; but the INSTALLER dies on an empty token —
|
||
script-side requirement, not Gitea's). Drill VM restored to `virgin`; evidence
|
||
`/mnt/5_hdd/felhom.eu/drill/bake-0.103.0.log` on 180 + `documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`.
|
||
**OPERATOR NEXT: bump the hub Day-0 manifest** to agent 0.74.0 / golden 0.103.0 (the four values
|
||
are in the execution record) — until then fresh installs land 0.63.0/0.98.3. Also recorded:
|
||
operator-key pin follow-up (fold into GL-4); GO-LIVE-PACKAGE.md still not in the repo (G1 status
|
||
recorded here instead).
|
||
- **2026-07-07 — TASK GL-2 SHIPPED: `--mode appliance|byo` install profile (host-install v1.10.0 +
|
||
harness)** — go-live items G2/G4/G5 implemented, **awaiting the supervised GL-6 drill** for live
|
||
provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old
|
||
`--mode provision` value retired with a specific die, `dr` stub kept); break-glass gated at its
|
||
CALL SITE (byo never touches root@pam); byo requires `--cores`+`--memory`, refuses
|
||
`--enable-oob`/`--rotate-recovery`/non-9.x-PVE/missing `--acl-storages`; disclosure + typed-hostname
|
||
ack; **the config write's `lan_resolver` default flips to OFF in byo** (the spec assumed
|
||
default-off — it was default-ON; appliance unchanged) + post-write asserts refuse
|
||
lan_resolver/wg_tunnel/oob=true; pool+ACL verify asserts in BOTH modes (R2 lesson);
|
||
`--preflight-only` (no state, PASS/FAIL verdict, proven live on felhom-pve). NEW
|
||
`scripts/hostinstall-mode-harness.sh` (static C1–C4 + invariants everywhere; PVE tier: C5 +
|
||
A/B dry transcripts) — 16/16 PASS on felhom-pve incl. C5 live; red-proofs RP-1..3 run→fail→revert.
|
||
FINDING: demo-felhom's controller.yaml has EMPTY git.username/git.token → any step-5
|
||
(re)install for that customer dies until the operator sets real Gitea read creds (relates to the
|
||
standing scope-down+rotate follow-up). `documentation/pilot/GO-LIVE-PACKAGE.md` is NOT in the
|
||
repo and no operator copy was provided — the G2/G4/G5 status flip is recorded here instead;
|
||
add the doc when the operator supplies it. Next: **GL-6 supervised drill** (appliance + byo
|
||
end-to-end, uninstall/re-provision, C7, the ack prompt interactively).
|
||
- **2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent
|
||
v0.72.0)** — merged E1+H1. Hub: operator OOB peer (`store/wg_operator.go`, `PUT/GET
|
||
/admin/wg/operator-peer` global key) + `oob_peer_ip`/`oob_operator_ssh_key` in the desired-state
|
||
wireguard block + `monitor/host_oob` (oob_degraded/recovered). Installer: `--enable-oob`/`install_oob`
|
||
(static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box
|
||
drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as
|
||
felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt
|
||
drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key.
|
||
- **2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent
|
||
v0.71.0)** — prereq for felhom-sshd (H1). Hub: `store.host_recovery` vault (per-host root@pam
|
||
console password; `PUT /hosts/{id}/recovery-credential` self-scoped + `GET /admin/hosts/{id}/
|
||
recovery-credential` global-only, secret never logged) + `host_mgmtplane` monitor raising
|
||
`mgmt_plane_healed`. Installer: `step_break_glass` (generate+set+vault root@pam, `--rotate-recovery`)
|
||
+ `install_mgmt_watchdog` (the 3 host artifacts, RuntimeDirectory-guarded). Live-validated:
|
||
agent-independent `/run/sshd` auto-heal in 30.0s; warning fired end-to-end; break-glass
|
||
vault→retrieve→PVE-ticket(200). felhom-pve root@pam is now the vaulted value.
|
||
- **2026-07-04 — S3 SHIPPED: agent WG tunnel (felhom-agent v0.64.0; docs-only here)** — the
|
||
doc-06 §3.3 handshake is now END-TO-END LIVE on demo: agent keygen → `POST /hosts/{id}/wg` →
|
||
desired-state `wireguard` block → agent-managed `wg-quick@wg-felhom` → PBS page over
|
||
10.77.0.1:8007 in 3 s. **`wg_tunnel.enabled` defaults FALSE agent-side** (rollouts to other
|
||
boxes are no-ops until the production endpoint exists). Revocation semantics completed (§3.5):
|
||
removed peer → agent disables + keeps its marker + NEVER re-registers; operator re-adds via the
|
||
pubkey the heartbeat report now carries. Escrow: `IdentityBundle.WGPrivateKey` auto-injected on
|
||
new escrow-create runs (pre-S3 blobs can't be retrofitted — honest fallback = fresh-key
|
||
re-register, /32 kept). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix, deferred
|
||
non-blocking); S4 = PBS over the tunnel; hub-side S6 = tunnel-health alerting on the new report
|
||
stanza. Detail: felhom-agent CHANGELOG v0.64.0 + REPORT.
|
||
- **2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0)** —
|
||
doc-06 row S2. DECISIONS: the WG assignment is **hub-owned state, merged into served
|
||
desired-state at READ time** — stored `desired_json` stays a pure operator blob; admin PUT with
|
||
a top-level `wireguard` key → 400 (no copy-paste clobber); **re-key-in-place keeps the /32**
|
||
(stable tunnel addressing across rotation/DR); one bound peer per host (partial unique index);
|
||
generation bumps ONLY on real change (register/re-key/adopt/bound-delete — idempotent paths
|
||
move nothing). `POST /hosts/{id}/wg` self-scoped per-host key; NEW golden
|
||
`desired-state-wireguard.golden.json` = the S3 cross-repo contract (byte-identical agent copy
|
||
rule). peersync **v1.0.1** (strip out of process substitution — the S1 exit-swallow finding;
|
||
sandbox red-proofed) redeployed to the dev endpoint. Live-proven on the REAL demo-felhom-01
|
||
record via the real per-host key: gen 2→3, served block = golden shape, endpoint `wg show`
|
||
carried the peer, v0.63.0 agent tolerated the unknown key (fetch clean, guests untouched),
|
||
then removed → gen 4, all baselines restored. `/offsite` page unit-proven (live view is
|
||
operator-password-gated — eyeball after next login). Next: **S3** (agent keygen +
|
||
registration + `wg-felhom` service + escrow join + CGNAT smoke — risky/supervised).
|
||
- **2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1)** — the doc-06
|
||
roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: **hub pushes
|
||
over SSH** to a forced-command `felhom-peersync` script (pull rejected: weakens immediate
|
||
revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev
|
||
`felhom-hetzner` box, provisioned by the NEW runbook `documentation/runbooks/offsite-endpoint.md`
|
||
(WG on **443/udp**, subnet 10.77.0.0/24, PBS datastore `felhom-offsite` at `/mnt/pbs-datastore`
|
||
— a 100 GiB Hetzner Cloud Volume attached 2026-07-27 06:29:40 UTC; the pre-migration copy at
|
||
`/srv/pbs-felhom` is 13 G of dead weight on `/` awaiting R-91's go-ahead —
|
||
nftables drop-all + wg0-only 8007, `ep0.felhom.eu` grey-cloud). Hub: `wg_endpoints`/`wg_peers` +
|
||
/32 allocator + `/admin/wg/*` (global-key, pubkey-in-body) + `internal/wgsync` (FixedHostKey pin
|
||
+ **pinned-type HostKeyAlgorithms — live-caught bug**: multi-hostkey sshd presented ECDSA vs the
|
||
ed25519 pin; fixed v0.32.1 + regression test) + full-list declarative reconciler (5-min drift
|
||
repair). Live: add/remove→wg show, malformed→state-untouched, reboot→persist+converge, client
|
||
tunnel via ep0:443→PBS login page, public 8007 closed. INCIDENT handled: first WG server key
|
||
leaked via `wg show dump` into the session log → rotated immediately (warning now in the
|
||
runbook). OPERATOR follow-up: **fix the AAAA record** (currently `…7aa1::`, must be `…7aa1::1`).
|
||
Next: **S2** (hub schema host-join + `wireguard` block on WireDesiredState + box-facing
|
||
registration + UI).
|
||
- **2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06)** —
|
||
`documentation/architecture/06-offsite-connectivity.md`: records the six settled decisions
|
||
(plain WG / host-side agent-managed `wg-felhom` as the agent-managed-unit PILOT / one shared
|
||
hub-driven endpoint VM running WG+PBS, NO agent on it / hub = source of truth, `wireguard`
|
||
block rides `WireDesiredState` on the existing DesiredGeneration channel / one datastore +
|
||
per-customer namespaces / PBS ON the VM, relay-through-DooPlex REJECTED as non-scaling),
|
||
the Day-0 join handshake, robustness set (WG-roaming-not-DynDNS, DNS re-resolve watchdog, MTU
|
||
1420, per-/32 isolation, tunnel-health via storage-target model), trust conformance (outbound
|
||
only; operator-inbound + game-ingress out of scope), the honest ledger (CGNAT still unmeasured
|
||
— mobile-hotspot smoke closure; peer-sync push-vs-pull = slice-1 design point), and the S1–S6
|
||
slice roadmap (**MVP = S1 endpoint + S2 hub schema/desired-state + S3 agent WG service, then S4
|
||
PBS-over-tunnel**; S5 merges with DR-completeness). day0-install backlog line now cites spike +
|
||
design doc. Docs-only; slices are separate future tasks.
|
||
- **2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung)** —
|
||
plain WireGuard, host-side, box-dials-out to a small public endpoint; **no Headscale/DERP needed
|
||
for backup** (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway
|
||
Hetzner: 11.4-min idle hold, PBS API through the tunnel with the exact-cert pin intact
|
||
(positive + negative), real 2 GiB worst-case backup at 4.26 MiB/s (= the home uplink; tunnel
|
||
overhead ~5%), UDP 51820 *and* 443 both pass. P0 honesty: the operator's line is plain-NAT
|
||
(fixed public IP, One Hungary), NOT CGNAT, and has zero IPv6 — CGNAT confirmation deferred to
|
||
Peti's VM 110 (second ISP). P7 surprise: kernel WG *works* inside the unprivileged guest (no TUN
|
||
needed) — host placement stands on architecture (backups originate host-side; cloudflared
|
||
precedent), not infeasibility. Evidence + architecture recommendation (endpoint VM shapes, key
|
||
custody via IdentityBundle escrow, per-peer /32 isolation):
|
||
`documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`. **This unblocks the
|
||
DR-completeness work: next task = production connectivity spec** (endpoint VM, agent-managed WG
|
||
service, hub peer registry, escrow join). Throwaway Hetzner box parked powered-off (secrets
|
||
shredded); demo-felhom verified residue-free.
|
||
- **2026-07-03 — 4th skill: `felhom-app-catalog`** (catalog authoring workflow; points at
|
||
app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by `install_skills.py`, 4/4
|
||
found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected
|
||
healthchecks, probe-container naming rule discovered + REUSE row added, UI-applied, controller
|
||
probe healthy). App-count drift recorded: website says "45+", per-category spans sum 49, real
|
||
catalog 53 — website fix is a separate task.
|
||
- **2026-07-03 — GOLDEN 0.98.3 LIVE + VOUCHED; guide D.1b RETIRED (drill B1+B5 FIXED)** —
|
||
`build-golden.sh` v2.0.0 (mandatory controller tag + baked bootstrap `.path` unit); golden baked
|
||
with controller 0.98.3, clean-room validated (drill VM) through all four scenarios incl. the
|
||
published-artifact `--force-gitea-golden` path, published + operator-vouched (Day-0 manifest now:
|
||
agent 0.63.0 + golden 0.98.3). Fresh installs land CURRENT and self-manage — D.1b is a one-line
|
||
check; the prior "vouch agent 0.63.0" operator follow-up is DONE. Evidence:
|
||
`documentation/audits/DRILL-golden-098-2026-07-03.md`. NEW operator follow-ups: (1) SECURITY —
|
||
the customer-config `git.token` is a Gitea admin token with package-WRITE (proven during the
|
||
bake); scope down + rotate. (2) optionally raise the global controller floor to 0.98.3.
|
||
- **2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published)** —
|
||
token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir.
|
||
Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. **OPERATOR: vouch
|
||
agent 0.63.0 in the Day-0 manifest UI** (until then fresh installs fetch 0.62.0).
|
||
- **2026-07-03 — GO-LIVE PACKAGE SHIPPED (host-install v1.9.1 + day0-install runbook + clean-room drill)** —
|
||
`documentation/runbooks/day0-install.md` is the operator Day-0 guide, **every command executed
|
||
verbatim** on a throwaway nested PVE 9.2.2 (build server): install → verify → app deploy →
|
||
uninstall → **zero-residue diff** → reinstall. v1.9.1 = residue-free uninstall (agent config w/
|
||
hub api_key, shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq snippets) +
|
||
post-provision guest reboot (without it the controller never deploys on slower hardware) + header
|
||
sync. Evidence + go/no-go for Peti (**GO**, 3 install-day items):
|
||
`documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`. Key OQ-2 finding: golden bakes a
|
||
pre-floor controller 0.85.1 → **fresh installs need the one-time D.1b update** (agent restart →
|
||
update button → 0.98.3); follow-ups recorded: rebuild golden ≥0.86.0 + re-vouch (operator), agent
|
||
`mkdir -p` for the guesthook snippet dir, fresh-install local-API 401 (pre-existing), hub has no
|
||
host-delete path. Drill VM parked virgin-snapshotted on 180 (`/mnt/5_hdd/felhom.eu/drill/` — moved off the SSD 2026-07-18).
|
||
- **2026-07-03 — A1 CLOSED (host-install v1.9.0 + agent v0.62.0)** — `Pool.Audit` added to
|
||
`FelhomAgentGuest`; the agent's stale-lock reaper now intersects its scan with
|
||
`GET /pools/felhom` membership (fail-safe skip on read failure). Spike:
|
||
`documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. Deploy order on any box:
|
||
rescope ACL first, then the agent. STILL PENDING (operator): rotate + de-git the 3 committed
|
||
secrets in `manifests/felhom.secret.yaml` (spike appendix; REUSE §2 errata added).
|
||
- **2026-07-03 — hub v0.31.0 LIVE** (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy;
|
||
live probe: critical event stored + operator email delivered).
|
||
- **2026-07-03 — Felhom skills exist**: `skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}`,
|
||
installed via `scripts/install_skills.py` — **junction mode** (repo edits live immediately; no
|
||
re-install needed after editing). Fresh-session discovery verified.
|
||
- **2026-07-03 — CLAUDE.md refreshed** (all four repos): no version-pinned state in CLAUDE.md —
|
||
current state lives here / CHANGELOG / REUSE.md.
|
||
|
||
- **2026-07-03 — `REUSE.md` exists at the repo root** (hub helpers / website conventions / gate scripts / manifest patterns, code-verified); maintenance rule active: update it in the same commit that changes a shared helper. `scripts/reuse_refs_check.py` machine-checks its citations across all four felhom repos.
|