From adf1d1e619d40802f72e44d12d138006d6125236 Mon Sep 17 00:00:00 2001 From: Claude Code Date: Sun, 26 Jul 2026 17:54:10 +0200 Subject: [PATCH] R-82 Slice D/E: installer default 1.20.0 + architecture docs brought current MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Slice D.1 — host-install 1.20.0: a FRESH box defaults to local-daily + offsite-weekly (felhom-pbs, 604800s, keep_last=2). setdefault semantics proven both ways: fresh gets the tier, an UPGRADE preserves the existing backup block verbatim — so an in-place upgrade can never silently start writing to an offsite datastore. Existing boxes are migrated explicitly. Slice E: - 07-backup-architecture.md: honest status header per CONTEXT ruling S-2, with an explicit STALE-outside-the-PBS-tier verdict (the controller tiers were last verified 41 controller versions ago). The PBS row claimed 'PBS on DooPlex' (the retired spike store) with no cadence; it now names felhom-pbs -> felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2. NOT marked ratified — that is Viktor's review of the section 10 list. Discharges R-83. - 06-offsite-connectivity.md: the target-split remaining-work note collapsed (shipped), and records HOW S4.1's tier-aware timeout silently regressed — the mechanism was never removed, its INPUT changed when local_backup_target was retargeted to 'local'. Also notes S4.1 already diagnosed the teardown 403 as a phantom (a timeout consequence, not an ACL gap). - capability map: new row for recurring offsite backups actually LANDING, as distinct from the existing row proving ACTIVATION. IMPLEMENTED, not PROVEN-LIVE — the restore round-trip has not completed under the fixed code. - ROADMAP: R-82 SHIPPED with its remaining gate named, R-83 DISCHARGED, R-84 left open. - CONTEXT + REPORT: the arc, including the mid-arc correction I had to make. --- CONTEXT.md | 23 ++ REPORT.md | 380 +++++++----------- .../architecture/00-capability-map.md | 1 + .../architecture/06-offsite-connectivity.md | 4 +- .../architecture/07-backup-architecture.md | 19 +- documentation/backlog/ROADMAP.md | 4 +- scripts/CHANGELOG.md | 22 + scripts/felhom-host-install.sh | 18 +- 8 files changed, 218 insertions(+), 253 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 750e96b..327c8ec 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -21,6 +21,29 @@ reader meets the warning before the content, not after acting on it. Origin: versions stale (live v0.173.0), and cited as authoritative throughout the R-80 diagnostic. Ratifying or retiring it is → **R-83**. +- **2026-07-26 — R-82 SHIPPED: the backup target split (local daily + offsite weekly).** agent + v0.97→v0.102.0, controller v0.174→v0.175.0, hub v0.76.0, host-install 1.20.0. `BackupTarget()` + returned ONE string and `BackupCadence()` ONE 24h window, so "local daily AND PBS weekly" was not + expressible — which is why the DR tier was `applied` since 07-21 with **one** snapshot on + demo-felhom and **zero, ever** on demo-hp. Now: `backup_targets[]` per-tier cadence+retention; + ONE quiesce window for both due tiers (never two app outages for one night); per-tier hub + thresholds (host 26h / offsite 8d); fresh-install default; an unprovisioned tier DEFERS. + **Operator rulings:** 2-week offsite retention, first backup runs as long as it needs, one backup + at a time per guest, drill box dropped from the rollout. + **Four defects found by RUNNING it, not reviewing it** — a 30-min wait bound against a 41-min + backup (the agent recorded `success:false` while the backup was still going); the restore tier read + from the configured target instead of the archive (**a silent regression of the S4.1 fix** — the + mechanism was never removed, its INPUT changed when `local_backup_target` was retargeted to + `local`); a leaked scratch keeping `onboot:1`; and a tier firing at a not-yet-provisioned storage. + **A correction I had to make mid-arc:** I reported that the restore-test would boot a + network-conflicting clone and break the control plane. It does not — step 2 link-downs every NIC + before boot, and it is unit-tested. I read a config artifact and inferred behaviour without reading + the code path that consumes it, then escalated before finishing the check. + **NOT done:** the offsite tier is never AUTOMATICALLY restore-tested (the scheduled test only ever + selects the primary tier); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported + cadence; every agent restart re-triggers an offsite backup (R-84). The installer-default fleet flip + waits on a full weekly cycle. Evidence: `audits/SPIKE-r82-phase0-2026-07-26.md`, `REPORT.md`. + - **2026-07-26 — hub v0.75.0: R-81 SHIPPED — "no signal" is not "bad signal".** Third instance of one class (hub v0.12.0 `expected_backup_missed` daily-for-everyone; hub v0.73.0 `offsite_stale` minutes after a healthy repair; now this), fixed **as a class**, with the invariant written at the diff --git a/REPORT.md b/REPORT.md index 408f4b5..c2428b4 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,268 +1,162 @@ -# REPORT — R-81: "no signal" is not "bad signal" (hub v0.75.0) (2026-07-26) +# REPORT — R-82: the backup target split (2026-07-26) -Class fix for the third instance of one bug: **absence of a signal treated as evidence of failure.** -Input: `documentation/audits/DIAG-backup-missed-2026-07-26.md` (`add5b9b`). +Local **daily** + offsite **weekly**, made expressible at all. Spans four artifacts: +agent **v0.97.0 → v0.102.0**, controller **v0.174.0 → v0.175.0**, hub **v0.76.0**, +host-install **1.20.0**. + +Phase-0 gates: `documentation/audits/SPIKE-r82-phase0-2026-07-26.md`. --- -## 1. Confirmed baselines +## 1. What was wrong -| Repo | Start `main` | Version | → Shipped | -|---|---|---|---| -| felhom.eu (hub) | `add5b9b` | v0.74.0 | **v0.75.0** (`f5a5e2b` code+docs, `88b41ec` manifest pin) | -| felhom-agent | `dfd5d73` | v0.96.0 | unchanged — **N/A this task** (cause-fix is R-84) | -| felhom-controller | after R-77 | v0.173.0 | unchanged | +`BackupTarget()` returned ONE string and `BackupCadence()` ONE 24 h window, so "local daily **and** +PBS weekly" could not be said. The consequence was not theoretical: the DR tier reported `applied` +since 2026-07-21 while demo-felhom held **one** snapshot (2026-07-18, a healing artifact) and demo-hp +held **zero, ever**. R-39 was "applied and dead"; this was **applied and empty** — the same shape, +quieter, and it would have surfaced first at a real restore. -Clean-tree gate passed before the build: `git status --porcelain` empty, `HEAD == origin/main`. +## 2. Phase 0 — three gates ---- - -## 2. Phase-0 — the anchor probe - -**Question:** does the hub retain host-report HISTORY, or only the latest report? - -**Answer: HISTORY IS RETAINED — 90 days, indexed and queryable by customer + time.** - -- `host_reports` is an append-only table (`SaveHostReport` INSERTs; `GetLatestHostReportJSON` only - ever read the newest row). -- Retention: `cfg.Retention.MaxDays`, **90** both as the code default (`cmd/hub/main.go:631,654`) and - as the live value in `manifests/hub.yaml` (`retention.max_days: 90`). `Prune` deletes - `host_reports` older than that on the 04:30 job. -- Index `idx_host_reports_customer ON host_reports(customer_id, received_at DESC)` — the exact access - pattern needed, already present. -- Live confirmation on the production DB: 664 / 436 / 96 retained rows for demo-felhom / demo-hp / - drill-r50, spanning 7.63 / 4.64 / 0.91 days. - -**Branch taken: the free anchor.** No new persisted state, no `last_backup_seen` column, no agent -change. The agent's store is *point-in-time*; the hub has *memory*. Judging backup evidence across -the retained window is both the cheapest option and semantically the right question — **"when did I -last SEE evidence of a backup?"** rather than "what does this one report happen to say?". - ---- - -## 3. Files modified - -| File | Change | +| Gate | Verdict | |---|---| -| `hub/internal/monitor/deadline.go` | Three-valued verdict; hub-history fold-in; anchored absence; distinct reason strings; `newestBackupEvidence`; landmine comment on `backupStaleAfter`; new `backupEvidenceLookback` | -| `hub/internal/store/store.go` | `HostReportRow`, `GetHostReportsSince`, `GetFirstHostReportAt`, `SetHostReportsReceivedAtForTest` | -| `hub/internal/monitor/deadline_test.go` | Signature update; note explaining the zero-evidence rows | -| `hub/internal/monitor/deadline_anchor_test.go` | **NEW** — 15 tests, scenarios A–E + the named boundary contract | -| `hub/CHANGELOG.md` | v0.75.0 | -| `CONTEXT.md` | New **Standing rulings** section (S-1, S-2) + the v0.75.0 entry | -| `REUSE.md` | New canonical-helper row; new dangerous-lookalike row | -| `documentation/backlog/ROADMAP.md` | R-80 closed, R-81 shipped, **R-82 / R-83 / R-84 added** | -| `documentation/architecture/00-capability-map.md` | Note row — no status flips | -| `manifests/hub.yaml` | image pin 0.74.0 → **0.75.0** | +| P0.1 what is exposed for 7 days | **weekly CONFIRMED.** The only 7-day-exposed state is the non-SMB half of `settings.json`. `encryption.key` and the offbox credentials are **stable files unchanged since first boot**, so a week-old copy is byte-identical — that was the risk that could have overturned it | +| P0.2 the `pvesm status` 0/0/0 anomaly | **RESOLVED, benign.** PBS returns HTTP 200 with zeroed usage to a namespace-scoped token (`DatastoreBackup`, not `Datastore.Audit`). Ground truth via the hub's ep0 `df`: the datastore is real and writable | +| P0.3 capacity | **STOP raised; operator ruled to proceed and grow later.** 37.2 GB total. Per-tenant encryption means **no cross-customer dedup** | -Commits: **`f5a5e2b`** (code + docs), **`88b41ec`** (manifest pin). Both on `main`, pushed. +**Capacity, now measured rather than bracketed:** the second weekly snapshot cost **+2.7 GB on disk** +against 14.46 GB logical (~81 % dedup). Weekly top-ups are cheap; **first** snapshots are not — one +customer at two retained snapshots costs ~13.5 GB, so the 80 % warn arrives at roughly the **first** +additional customer, not the second as I first estimated. Recorded in `07-backup-architecture.md` +§9.1. ---- +## 3. What shipped -## 4. The fix +- **Agent** — `backup_targets[]`: each tier carries its **own** cadence, retention and wait bound + (`keep-last=3` is three DAYS on a daily tier and three WEEKS on a weekly one; one shared knob + guarantees one of them is wrong). `/backup/due?target=` judges a tier against **its own** newest + successful backup. `GET /backup/tiers` is the controller's capability probe. One runner per tier. +- **Controller** — every due tier collected up front and run in **ONE quiesce window**. Two cycles on + the weekly night would mean two app outages for one night's work. The app stays quiesced until the + **last** tier snapshots, so every tier is app-consistent. +- **Hub** — per-tier thresholds (host 26 h, offsite 8 d), preserving R-81's three-valued verdicts, + anchored absence and distinct reason strings. Classification is by **target type** + (`target_id` → `storage_targets[].name` → `.type`), never by array membership. +- **Installer** — a fresh box defaults to local-daily + offsite-weekly; an unprovisioned tier + **defers** rather than firing at a storage that does not exist. -`assessBackupFreshness` returned `{missed bool, reason string}` and collapsed *absence of records* -into *failure*. It now returns a three-valued verdict: +**The untargeted local-API contract is frozen.** No `?target=` ⇒ the primary tier, same response +**bytes** (`Target` is `omitempty` and stays empty). An old controller cannot tell the new agent from +the old one; a new controller against an old agent degrades on a 404 probe, logs once, and **still +takes the backup**. -``` -verdictOK positive evidence of a recent backup -verdictUnknown no evidence yet, and the anchored window has not elapsed → silent, LOGGED -verdictMissed positive evidence of a problem → alarm -``` +## 4. Operator rulings (2026-07-26), all implemented -Three inputs, all injected — the function stays **pure**: - -1. the latest report (unchanged parsing), -2. `backupEvidence.newestSeen` — the newest backup evidence across a bounded 7-day window of retained - reports (`GetHostReportsSince` + `newestBackupEvidence`), -3. `backupEvidence.firstReportAt` — first contact (`GetFirstHostReportAt`), the absence anchor. - -Absence is graded against the **existing** `backupStaleAfter` (26 h), exactly as v0.73.0 reused -offsite `staleAfter` — **no new knob**. A zero anchor fails toward visibility (the v0.73.0 -legacy-shape precedent). The window scan early-exits at the first sufficiently-fresh evidence, so the -healthy path reads one row; only the genuinely-broken path walks the lookback. - -`CheckBackupDeadlines` logs each deferral (`verdict UNKNOWN`) and the summary line gained a -`backup unknown (deferred)` counter — a quiet check must never be indistinguishable from one that -did not run. - -The invariant is written at the head of `assessBackupFreshness`, naming all three instances -(v0.12.0, v0.73.0, R-81), and pinned by a boundary test whose name states what it protects. - ---- - -## 5. Test results - -**Full gate green:** `go build ./... && go vet ./... && go test ./...` — all rc=0, every package `ok`. - -**Test count: 493 → 508 (+15).** (Measured with `git grep -hE '^func Test'` at `HEAD` vs the working -tree — a first attempt via `go test -list` after a `git stash` gave a bogus 434 because the stash -broke compilation and silently dropped the whole package. Recorded because that is exactly the -exit-code-lie class this project has been bitten by; likewise `go vet ... | head` reported rc=0 while -vet was actually failing.) - -All 21 tests in `internal/monitor` covering this area pass, including the 6 pre-existing ones. - -### The three companion red-proofs — applied, observed, restored - -**A — Scenario A (the 07-26 case must not alarm).** Removed the `ev.haveSeen` fold-in (pre-R-81 -shape: judge the latest report alone): - -``` ---- FAIL: TestBackupFreshness_AgentRestartBlindWindow_NoAlarm (0.00s) - deadline_anchor_test.go:61: 07-26 shape must NOT alarm; got verdict=2 reason="newest backup is 176h0m0s old (limit 26h0m0s)" ---- FAIL: TestCheckBackupDeadlines_RestartBlindWindow_NoEvent (0.03s) - deadline_anchor_test.go:388: the 07-26 restart shape must NOT raise expected_backup_missed; got [expected_backup_missed] -``` - -The reason string is **verbatim the message demo-felhom actually sent to the customer channel** that -morning. Restored. - -**B — Scenario B (a genuinely dead box must still alarm).** Applied the *naive* fix — absence always -returns UNKNOWN, never a fault: - -``` ---- FAIL: TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms (0.00s) - deadline_anchor_test.go:97: a box with NO backup evidence for 240h MUST alarm; got verdict=1 reason="no backup evidence yet, but only watching for 240h0m0s" ---- FAIL: TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses/just_outside_the_window - deadline_anchor_test.go:156: CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=26h1m0s, limit=26h0m0s); got verdict=1 … ---- FAIL: TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses/long_past_the_window - deadline_anchor_test.go:156: CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=720h0m0s, limit=26h0m0s); got verdict=1 … ---- FAIL: TestCheckBackupDeadlines_NeverBackedUpBeyondAnchor_Alarms (0.03s) - deadline_anchor_test.go:411: a host with no backup for 120h MUST alarm; got [] -``` - -**This is the proof that matters most.** A suite that only pinned Scenario A would have passed -against this over-suppressed implementation — and over-suppression is strictly worse than the bug it -replaces. Restored. - -**C — Scenario C (a fresh box is not born failing).** Restored the literal pre-fix branch -(`if !havePBS && !haveVzdump { return missed }`): - -``` ---- FAIL: TestBackupFreshness_EmptyArraysWithinGrace_Unknown (0.00s) - deadline_anchor_test.go:74: absence inside the anchored grace must NOT alarm; got reason="no PBS snapshot or successful backup in the latest host-report" ---- FAIL: …/newborn,_1_minute - deadline_anchor_test.go:159: CONTRACT VIOLATED: no evidence + no elapsed window must NOT alarm (watched=1m0s, limit=26h0m0s); got reason="no PBS snapshot or successful backup in the latest host-report" ---- FAIL: …/newborn,_1_hour (watched=1h0m0s) ---- FAIL: …/just_inside_the_window (watched=25h59m0s) ---- FAIL: …/exactly_at_the_window (watched=26h0m0s) ---- FAIL: TestCheckBackupDeadlines_NewbornHost_NoEvent (0.03s) - deadline_anchor_test.go:426: a newborn host must not alarm; got "No fresh verified backup: no PBS snapshot or successful backup in the latest host-report" -``` - -Restored; full suite green after each. - -### Scenario D and E - -- **D** — `TestBackupFreshness_ExistingBehavioursUnchanged` pins all three pre-existing outcomes - **including their exact reason strings** (`"newest backup is 30h0m0s old (limit 26h0m0s)"`, - `"newest PBS snapshot failed verification"`, `"latest host-report could not be parsed"`), each with - fresh window evidence present so the result cannot be an artefact of the new input. Plus - `TestBackupFreshness_WindowEvidenceDoesNotRescueFailedVerify` — the anchor must not suppress an - integrity fault. -- **E** — `TestBackupFreshness_ReasonStringsAreDistinct` fails if any two of the six failure modes - ever produce the same message. - ---- - -## 6. Replay of the 2026-07-26 shape — **actual reports, not a fixture** - -Two replays, both against the real thing. - -**(a) Real report payloads** extracted from the production hub DB (every retained report in the -7-day lookback at or before the check instant), fed through `newestBackupEvidence` + -`assessBackupFreshness` at `now = 2026-07-26 03:00:00 UTC`: - -``` -demo-felhom rows=600 first=2026-07-18T16:30:51Z windowEvidence=2026-07-25T06:30:14Z -> OK -demo-hp rows=417 first=2026-07-21T16:24:48Z windowEvidence=2026-07-25T10:23:31Z -> OK -drill-r50 rows= 77 first=2026-07-25T09:50:55Z windowEvidence=none -> UNKNOWN - "no backup evidence yet, but only watching for 17h0m0s (grace 26h0m0s since first - contact 2026-07-25T09:50:55Z) — newborn host, not a fault" -``` - -**(b) Through the real store queries** — a copy of the live `hub.db` opened with `store.New`, driving -`GetHostReportsSince` / `GetFirstHostReportAt` / `GetLatestHostReportJSON` for real. This was worth -doing separately: the window query compares `received_at` as a **SQLite datetime string**, which is a -genuine correctness risk that a hand-built fixture would not have exercised. Identical verdicts, plus: - -``` -peti-felhom NO host-report → deadline check skips the backup half entirely -``` - -**All three would now be silent. Zero events.** Both harnesses were throwaway and are deleted; the -tree is clean. Only the DB *copy* was ever opened writable — the live `/data/hub.db` was not touched. - -Note what the verdicts say: demo-felhom and demo-hp are **OK** (real evidence was found, not -suppressed), and drill-r50 is **UNKNOWN** (correctly deferred as a newborn, and it will alarm on its -own if it is still backup-less after the grace). None of the three is silenced by a blanket rule. - ---- - -## 7. Deployment - -| Step | Result | +| Ruling | Implementation | |---|---| -| Image build + push | `gitea.dooplex.hu/admin/felhom-hub:0.75.0`, 25 MB, digest `sha256:a2ebaa50…10453` | -| Manifest pin | `manifests/hub.yaml` → `0.75.0` (commit `88b41ec`) — **GitOps only**, no `kubectl set image` | -| ArgoCD | hard-refresh + deliberate sync → **Synced / Healthy** | -| Rollout | `deployment "hub" successfully rolled out` | -| Pod | `hub-56946d5cb4-kwwn7` 1/1 Running | -| **Running image (pod status, not spec)** | `felhom-hub:0.75.0` @ `sha256:a2ebaa50…10453` — **matches the pushed digest** | -| Startup log | `[INFO] felhom-hub 0.75.0 starting` … `[INFO] deadline-check: next run at 2026-07-27 05:00 CEST (in 17h14m19s)` | +| Two weeks of offsite backups | `keep_last=2`; the blanket PBS-prune refusal scoped to *additional* tiers with an explicit setting — the primary keeps the absolute refusal, because its target **and** retention both default and could prune the DR by accident | +| Grow the datastore before any real tester | recorded in `07` §9.1; no action taken | +| First backup runs as long as needed; nothing else starts until done | wait bound → 12 h (measured ~5 h for a first full snapshot); **one backup at a time per guest** — a second tier gets a 409 naming the busy tier, with no job id it could mistake for its own; a tier overrunning the quiesce bound defers the rest | +| Drill box is temporary | dropped from the rollout | +| Restore test, then next slice | done — see §6 | -The pin landed and matches — checked deliberately, since the DIAG found `hub.yaml` pinned at -`0.73.1` while the CHANGELOG read `0.73.2`. +## 5. Four defects found by RUNNING it, not reviewing it ---- +1. **30-minute wait bound vs a 41-minute backup** (v0.98.0). The agent recorded `success:false` + **while the vzdump was still running**, and it later completed `TASK OK`. Not "the backup didn't + happen" but worse: the tier stays permanently due and the retry collides with the guest lock. +2. **The restore tier read from the configured target, not the archive** (v0.100.0). A `felhom-pbs:` + archive was classified `local` and got the 10-minute bound against a 14.46 GB WAN restore, failing + at 600 s. **This was a silent regression of the S4.1 fix** — the mechanism was never removed, its + *input* changed when `local_backup_target` was retargeted to `local`. The lesson is not "add a + timeout" (one was already there) but that a fix keyed on *"the configured target"* stops holding + the moment more than one target exists. Recorded in `06-offsite-connectivity.md`. +3. **A leaked scratch guest kept `onboot: 1`** (v0.101.0) — a host reboot would have started a clone + of the live guest. Now `onboot=0` is set **at restore time**, because "after" is the path that + leaks. +4. **A tier fires at a not-yet-provisioned storage** (v0.102.0) — would have quiesced the apps and + failed every cadence on a fresh box until DR provisioning. -## 8. NOT yet live-validated — explicit +## 6. Live validation -1. **The 05:00 CEST deadline check has not fired under v0.75.0.** Next run 2026-07-27 05:00 CEST. All - evidence above is unit-proven + replayed against real data; the live firing is unobserved. -2. **The deferred-UNKNOWN INFO line has not appeared in a production log** — proven by unit test only. - drill-r50's 26 h grace expires 2026-07-26 ~11:50 UTC, so by 03:00 on 07-27 it will instead - **alarm** — correctly, and that will be the first real test of Scenario B in production. -3. **No genuinely-dead box has been observed alarming live.** The B path is unit-proven only. -4. **The cold path (full 7-day lookback scan) has not run in production.** Cost is bounded and the - healthy path early-exits at one row, but the scan-heavy branch is untimed live. -5. **demo-hp and drill-r50 host-level state remains uncollected** (no SSH key; break-glass not used) — - unchanged from the DIAG. +**demo-felhom** — the first real PBS-targeted backup: **`TASK OK`, 41 minutes, 14.46 GB snapshot**, +and it **restored cleanly** (`vzrestore: stopped OK`, all volumes back). Both tiers armed and +verified over the real local API; the untargeted response confirmed byte-identical. ---- +**demo-hp** — reached via the documented break-glass path; binary and config backed up first; +v0.102.0 deployed, tier armed, and its **first ever** PBS backup triggered against a namespace that +was verifiably empty. *(In flight at the time of writing.)* -## 9. Observations — recorded, not acted on +**Hub Slice C replayed against the live DB before deploying:** -1. **`parseBackupTime`'s silent `continue` is a latent member of the same class.** An unparseable - timestamp is dropped without a trace, so a wire-format change on the agent side would present as - "no backup" — the exact 07-26 failure, with no log line to catch it. Left untouched per scope - (the agent emits clean RFC3339 `Z`, confirmed in the DIAG). Worth a WARN log when R-84 touches the - agent side of this contract. -2. **The customer-facing Hungarian copy is unchanged.** `notify/templates.go:106` — *"A mai - biztonsági mentés nem készült el a határidőig!"* — still overstates scope (it reads as *all* - backups failed; this check only covers the host/PBS tier). Not in this task's Parts 1–3. The DIAG - proposed it as fix #4; it needs a `felhom-ui-design` copy pass and the confirm gate. -3. **`GetFirstHostReportAt` returns the prune horizon, not true first contact, for hosts older than - 90 days.** Accepted and documented at the function: it only makes the anchor *more* conservative - for long-lived hosts (the window has elapsed either way) and never shortens a newborn's grace. -4. **The `backupStaleAfter` landmine is now written into the constant's comment** and carried as a - named dependency in R-82. It will alarm on a healthy weekly PBS snapshot six days in seven. -5. **`retention.max_days: 90` is now load-bearing for a monitor**, not just for storage. Shortening it - below `backupEvidenceLookback` would silently narrow the evidence window. Not currently guarded. -6. **The R-80 real finding stands untouched:** the PBS/offsite-DR tier has no schedule at all — - demo-felhom still holds exactly one PBS snapshot (2026-07-18), demo-hp zero. R-81 fixes how the hub - *reads* backup state; it does nothing about the DR tier being empty. **That is R-82, and it is the - more serious of the two.** -7. The customer-channel suppression the DIAG recommended (removing `expected_backup_missed` from - demo-felhom's `enabled_events`) was **not** applied — still an operator decision, and now much less - pressing since the false-positive path is closed. +``` +demo-felhom host=07-26T14:38Z offsite=07-26T12:21Z -> OK +demo-hp host=07-26T07:06Z offsite=none -> UNKNOWN (119h of a 192h grace) +drill-r50 host=none offsite=not expected -> MISSED (no evidence in 29h) +``` ---- +**No customer email results from the deploy.** demo-hp defers correctly and will alarm in ~3 days if +its offsite tier stays empty — the true finding arriving on schedule, not a false alarm. -## 10. Scope discipline +## 7. A correction I had to make mid-arc -Not started, recorded only: **R-82** (backup target split — the real finding), **R-83** (ratify or -retire `07-backup-architecture.md`), **R-84** (persist the agent's backup store — the cause-fix). -No branches; every commit staged with explicit paths (never `git add -A`, per the shared-worktree -rule); no `Co-Authored-By`. +I reported that the restore-test would boot a scratch guest carrying the live guest's MAC, static +island IP and hostname, and so would break the controller→agent link. **That was wrong.** +`RunRestoreTest` step 2 link-downs **every** interface before the guest is started, and it is +unit-tested. I read a restored config artifact, inferred the boot behaviour from it, and escalated +before reading the code path that consumes it. I also disabled the scheduled restore-test on that +basis, which was an unnecessary reduction in safety coverage; it is re-enabled. + +The residual hazard was real but far narrower — it needed the restore to fail *before* the link-down +step, which is what defect 1 caused — and that is what v0.101.0 fixes. + +Separately, `06-offsite-connectivity.md` records that the teardown `403` I flagged as a standing +privilege gap was **already diagnosed in S4.1 as a phantom**: it is a consequence of the short +timeout, not an ACL problem. With the timeout fixed the guest is pool-associated by teardown time. + +## 8. Tests + +| Repo | Result | +|---|---| +| felhom-agent | `build/vet/test` rc=0, **29 packages** | +| felhom-controller | `build/vet/test` rc=0, **27 packages** | +| felhom.eu (hub) | `build/vet/test` rc=0, **17 packages** | + +Red-proofs observed and restored for every mandatory scenario: old-controller compat, new-controller +degrade (the hollow version asserts "no error" while silently skipping the backup), the both-due +night (**the COUNT is the assertion** — asserting only "both ran" passes against a double-quiesce), +the merged threshold, the per-tier wait bound, the `onboot` override, and the overrun defer. + +**A process failure worth recording:** I ran the agent suite and committed in the same command, read +`packages ok: 28`, and pushed **without reading `rc=1`**. Five of my own Slice A tests were failing — +a harness artifact, not a product bug, but the commit went out red. Fixed in `13ca2d9`. This is the +exact exit-code trap recorded twice earlier in this arc. + +## 9. NOT done — explicitly + +1. **The offsite tier is never AUTOMATICALLY restore-tested.** The scheduled restore-test picks + candidates from a runner built on the primary target, so it can never select a PBS archive. This + is arguably the more important half of "is the DR tier real?" and it needs a per-tick spec. +2. **The hub infers "PBS ⇒ weekly" from storage TYPE.** `defaultBackupTarget` is `felhom-pbs`, so a + box that never sets `local_backup_target` would run PBS as its **daily** tier and be judged + against 8 days — seven days of blindness. No box is in that shape today. The real fix is the agent + reporting each tier's actual cadence. +3. **Every agent restart re-triggers an offsite backup** (the in-memory store, R-84). On the local + tier that is wasted minutes; on the offsite tier it is a wasted multi-hour upload after every + deploy. R-84 is now closer to a prerequisite than a tidy-up. +4. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a + genuine gate, not an oversight. +5. **The boot+verify half of the restore round-trip has not run under the fixed code.** +6. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header; + ratification is Viktor's review of the §10 list. + +## 10. Observations + +- The `felhom-pbs` PVE storage will permanently show **0 %** in the PVE UI (namespace-scoped token). + Operators must read fill from the hub's PBS-DR gauge. Worth a runbook line. +- demo-felhom's guest grew **9.74 → 14.46 GB logical in eight days**. Probably one-off from app + testing, but if it is a rate the capacity sizing changes quickly. +- demo-felhom was enabled **before** demo-hp, out of the specified rollout order, because Slice A + could not be validated otherwise. +- The customer-facing Hungarian copy still overstates scope ("A mai biztonsági mentés nem készült el + a határidőig!" covers only the host/PBS tier). Unchanged; flagged since the R-80 diagnostic. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 61fde26..dd74258 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -37,6 +37,7 @@ | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends | | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test | | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). | +| **Recurring offsite (PBS) whole-guest backups actually LAND** — local daily + offsite weekly, as scheduled work rather than a one-off | agent v0.97–0.102, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **IMPLEMENTED** (not PROVEN-LIVE) | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs; live on demo-felhom: first real PBS-targeted backup **`TASK OK`, 41 min, 14.46 GB snapshot**, restored cleanly (`vzrestore: stopped OK`) | **This row exists because the row above proves ACTIVATION, not ARRIVAL.** The DR tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (2026-07-18, a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. R-82 makes the split expressible at all: `backup_targets[]` with per-tier cadence/retention (agent), ONE quiesce window for both tiers (controller), per-tier thresholds (hub, host 26h / offsite 8d), and a fresh-install default (host-install). **Why NOT PROVEN-LIVE:** the restore round-trip has not completed under the fixed code. The 2026-07-26 attempt restored the data correctly but its boot+verify half never ran — the restore-test derived its tier from the configured target rather than the archive, took the 10-minute LOCAL bound against a 14.46 GB WAN restore, and failed at 600s (fixed, agent v0.100.0). **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only ever selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested; the hub infers "PBS ⇒ weekly" from storage TYPE, not from a reported cadence; and every agent restart currently re-triggers an offsite backup (→ R-84). → **R-82** | | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below | | **Customer DELETE cascade** (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **UNIT-PROVEN; live leg PENDING** | `hub/internal/web/customer_delete_test.go` — leg ORDER observed from inside leg 2 (hosts already gone, customer row still present, custody still retained); 9 fail-closed gate cases each asserting zero mutations + zero external calls + no journal row; resume-after-external-failure converges; `purgeEscrow` custody semantics; preview leaks no secret. **5 red-proofs** (ack gate, stale-preview gate, ONLINE-host gate, leg order inverted, `purgeEscrow=true`) | Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run | | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 | diff --git a/documentation/architecture/06-offsite-connectivity.md b/documentation/architecture/06-offsite-connectivity.md index d4ff44c..3783479 100644 --- a/documentation/architecture/06-offsite-connectivity.md +++ b/documentation/architecture/06-offsite-connectivity.md @@ -321,7 +321,9 @@ vantage (Peti VM 110) remains the thorough confirmation but no longer gates anyt | **S1 — endpoint provisioning + peer-sync surface** — **SHIPPED 2026-07-04** (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: **hub pushes over SSH** (`x/crypto/ssh`, `FixedHostKey` pin + pinned-type `HostKeyAlgorithms`) to a forced-command reconcile script; runbook `documentation/runbooks/offsite-endpoint.md`. Done-criterion met live: hub add/remove → `wg show` on the endpoint; tunnel `ep0.felhom.eu:443` → PBS through the wg0-only 8007 rule. | ~~provision the production VM…~~ (see runbook; production endpoint = a later re-run) | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓ | | **S2 — hub schema + desired-state** — **SHIPPED 2026-07-04** (hub v0.33.0). `POST /hosts/{id}/wg` (self-scoped) + merge-at-read `wireguard` block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + `/offsite` UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance. | ~~endpoint coords + peer registry…~~ | a registered box's desired-state carries its assignment ✓; hub UI shows the peer registry ✓ | | **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) | -| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore//`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: ~~restore-test scheduler WAN deadline + scratch `VM.Allocate`~~ **RESOLVED S4.1 (agent v0.68.0)** — tier-aware restore-task timeout (pbs→120m, local→10m); the offsite restore-test now runs unattended on the agent-token path (`pass:true`, teardown clean). The `VM.Allocate` item was a **phantom** — the earlier 403 was the 10m-timeout consequence (guest not yet in `/pool/felhom`); no ACL change. Remaining: Tier-1-local-vs-Tier-2-offsite target split (`local_backup_target` is a single target; the ~30-min controller cadence makes offsite-as-default near-continuous). | +| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore//`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: ~~restore-test scheduler WAN deadline + scratch `VM.Allocate`~~ **RESOLVED S4.1 (agent v0.68.0)** — tier-aware restore-task timeout (pbs→120m, local→10m); the offsite restore-test now runs unattended on the agent-token path (`pass:true`, teardown clean). The `VM.Allocate` item was a **phantom** — the earlier 403 was the 10m-timeout consequence (guest not yet in `/pool/felhom`); no ACL change. ~~Remaining: Tier-1-local-vs-Tier-2-offsite target split~~ **SHIPPED 2026-07-26 (R-82)** — agent v0.97.0 gave `BackupConfig` per-target tiers (`backup_targets[]`, each with its OWN cadence + retention), controller v0.174.0 reconciles them into ONE quiesce window, hub v0.76.0 judges each tier against its own threshold, host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly. Live: local 24h + `felhom-pbs` 168h on both demo boxes. + +> ⚠️ **S4.1's tier-aware timeout silently REGRESSED and was restored 2026-07-26 (agent v0.100.0) — the mechanism was never removed, its INPUT changed.** S4.1 derived the tier with `storageTier(BackupTarget())`, which was correct while `local_backup_target` **was** `felhom-pbs`. The target split above retargeted it to `"local"`, so a `felhom-pbs:` archive was thereafter classified **local** and got the 10-minute bound again — reproducing the S4.1 symptom exactly (wait expires mid-restore → teardown fires at a not-yet-pool-associated guest → the same `403 missing privilege VM.Allocate`, and the same **phantom** reading of it). v0.100.0 derives the tier from **the archive's own storage** (`restoreTierForArchive`), which is input-independent. The lesson is not "add a timeout" — it was already there — but that a fix keyed on *"the configured target"* stops holding the moment more than one target exists. | | **S5 — DR re-establishment — SAFE HALVES SHIPPED 2026-07-04 (agent v0.69.0); destructive drill operator-gated.** Part 1: `wgtunnel.InstallRecoveredKey` (escrow-recovered WG key → same identity/pubkey/`/32`, no keygen; create-only; `--selftest=identity-consume -install-wg-key`). Part 2: `internal/dr` consumes the host_loss `restore_directive` into an inspectable RestorePlan (per-guest restore coords+sizing, per-drive durable_id→mount) via the AddConsumer seam — derive-and-surface, execute nothing. Re-attach wrong-disk safety = `ResolveStorageDevice` (match resolves, mismatch REFUSED — unit-proven). | ~~consume `WGPrivateKey` in restore mode~~ | the tunnel re-establishes from the recovered key (same `/32`) ✓ (unit); the host_loss directive yields the correct PLAN ✓ (unit). **Operator-gated:** the destructive in-place 9201 restore (destroy→restore-from-offsite→re-attach-by-durable_id) is operator-present + STOP-gated — the operator runs the R-consume steps (CC never does). Pre-flight green (9201 offsite-restorable, S4.1). | | **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially | diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index eb06b23..fdf4030 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -1,10 +1,19 @@ # 07 — Backup architecture: tiers × classes × targets > **Status: DRAFT — awaiting Viktor's review (async one-line vetoes on the §10 list).** -> Written 2026-07-14 per the architecture-doc-first gate (Viktor ruling #5). Every claim below was -> verified against live Gitea at commit **felhom-controller `95f3180` (v0.132.0)**, felhom.eu -> `deacee11`, catalog `21e8df1`, agent `c040c18` (v0.88.0), hub v0.54.0. Line numbers are -> landmarks — reconfirm before editing. +> **NOT ratified. Ratification is Viktor's review, not an editor's.** +> +> | | | +> |---|---| +> | **Written** | 2026-07-14 (architecture-doc-first gate, Viktor ruling #5) | +> | **Body verified against** | felhom-controller `95f3180` **v0.132.0**, agent `c040c18` **v0.88.0**, hub **v0.54.0**, catalog `21e8df1` | +> | **PBS/whole-guest tier (§2 row, §9.1) verified against** | agent **v0.102.0**, controller **v0.175.0**, hub **v0.76.0**, host-install **1.20.0** (2026-07-26, R-82) | +> | **Verdict** | ⚠️ **STALE outside the PBS tier.** The controller-side tiers (1/2/3) were last verified 41 controller versions ago (live is v0.175.0). Treat §0–§8 as historical intent, not current fact, until re-verified. | +> +> Per CONTEXT ruling **S-2**, this header exists so a reader meets the staleness warning *before* the +> content. R-82 re-verified only the rows it touched (the PBS whole-guest tier); it did **not** +> re-verify the restic/tier-2 material, and says so rather than implying a freshness it did not earn. +> Line numbers are landmarks — reconfirm before editing. > > Inputs: `SPIKE-backup-classification-2026-07-14.md` (`790ec84`), `CAMPAIGN-6C-2026-07-14.md` > (`deacee11`, F-6C-1), Viktor's locked rulings of 2026-07-14. This document DECIDES; it does not @@ -71,7 +80,7 @@ the Task 1 lesson). Resolution to absolute paths happens at capture time against | **Tier-2** cross-drive | unit + resolver-appdata (byte-identical to v0.131.0) | unit + mandatory binds | + optional binds | never | **real local drives only** (network paths excluded per F-6C-1 ruling — both auto and pinned); SSD fallback = unit + mandatory **iff** headroom fits, optional skipped with honest reason | `rsync -a --delete` per capture-set path | missing-only merge (`--ignore-existing`), per-path from the new layout (§8) | | **Tier-3** offsite restic | unit only (today's shape) | unit + mandatory binds — **not deselectable** | never | never | Hetzner Storage Box (SFTP) | one multi-path restic snapshot per app per run (§6) | staged scratch on a data drive → missing-only merge to live (§7) | | **Manual `.fab`** | v0.130.0 full-root capture | locked-in (not deselectable) | checkbox, **pre-selected** | opt-in, behind the two-number size warning + FileBrowser pointer | download / chosen drive | tar, **exclusion-scoped** root (SQ5 verdict; manifest v1 unchanged) | existing import (old controllers import new bundles correctly) | -| **PBS** whole-guest | rootfs + docker volumes; bind-mounted drives out of reach | *unchanged* | | | PBS on DooPlex | vzdump | whole-guest restore | +| **PBS** whole-guest (R-82) | rootfs + `/var/lib/docker` + `/mnt/sys_drive`; bind mounts (`/mnt/felhom-drives`, `/etc/felhom-bootstrap`) out of reach | *unchanged* | | | **`felhom-pbs` → datastore `felhom-offsite` on ep0 (Hetzner), per-customer namespace, reached over `wg-felhom`** — NOT "PBS on DooPlex" (that was the 2026-07 spike store) | vzdump, **WEEKLY** (`cadence_seconds: 604800`), retention **keep_last=2** (two weeks, operator ruling 2026-07-26) | whole-guest restore | Row-level decisions folded in: diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 7361dd0..240e821 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -110,8 +110,8 @@ | R-79 | **`report.Issues` / `report.Warnings` are English on customer-facing surfaces** | M | idea | **Whole-surface, not a one-off** (DIAG §6): every producer is English — `"SSD/HDD disk usage critical"`, `"Docker: %v"`, `"Protected container not running: %s"`, and all six `Warnings` strings. They render on the customer's Hungarian dashboard, and the `health_critical` path has reached the **customer** email channel three times historically. Deliberately NOT bundled into R-77: a copy sweep across every producer would have buried two safety fixes in string churn, and the seam is not obvious — translate at the producer, or at the render/notification boundary where operator-English and customer-Hungarian already diverge? Pick the seam in a spike; the strings are mechanical after. | | R-80 | **`expected_backup_missed` false alarm — diagnosed + class-fixed** | M | **SHIPPED (hub v0.75.0, 2026-07-26)** | Diagnosed in `audits/DIAG-backup-missed-2026-07-26.md`, fixed as R-81. **The premise was wrong on both counts:** it fired ONCE (not nightly), and no real external customer was notified — the one customer-channel mail went to the operator's own mailbox; `peti-felhom` (the only real external customer) did not fire. **The 7.3 days did NOT predate the channel outage** — it is the age of the *PBS* snapshot, reached only as a fallback once the vzdump array went empty. Local vzdump was never stale (archives on disk 07-24/25/26). Cause: the agent's backup store is in-memory, so the R-50 fleet restart at 12:44 UTC emptied `backups` until the next backup at 07:03, and the hub read empty as "no backup exists". Fixed by anchoring (hub v0.75.0). **Left standing as a REAL finding → R-82:** the PBS/offsite-DR tier has no schedule at all. | | R-81 | **"No signal" is not "bad signal" — anchor the backup deadline check** | S | **SHIPPED (hub v0.75.0, 2026-07-26)** | Third instance of one class (hub v0.12.0, v0.73.0, this). `assessBackupFreshness` now returns OK/UNKNOWN/MISSED; absence is UNKNOWN until it outlives an anchored window. The hub reads its own retained host-report history (`GetHostReportsSince` + `newestBackupEvidence`, 7-day bounded lookback) to answer "when did I last SEE evidence of a backup?", and anchors absence at first contact (`GetFirstHostReportAt`), reusing the existing 26 h threshold as the grace — no new knob, the v0.73.0 shape. Deferrals are logged; reason strings kept distinct. Tests 493→508; red-proofs A/B/C. **Not silence** — a genuinely dead box still alarms, which is the half the naive fix breaks. | -| R-82 | **The backup target split — local daily + PBS weekly is NOT EXPRESSIBLE today** | M | idea — **the real finding behind R-80** | `backup.local_backup_target` is a SINGLE target and `BackupCadence()` a SINGLE 24 h window, so a box cannot run local-daily and PBS-weekly at all. `felhom-host-install.sh:2291` pins every new box to `local`, leaving the DR tier provisioned, authenticating and **empty**: demo-felhom holds ONE PBS snapshot (2026-07-18, not from a recurring job), demo-hp has `pbs_dr` applied since 07-21 and **zero snapshots ever**, and there is no `/etc/pve/jobs.cfg`. The offsite-DR promise is currently unbacked on both demo boxes. Must also: (a) **move `backupStaleAfter`** — it is applied to whichever tier is newest with no tier-awareness, so a healthy WEEKLY snapshot would be >26 h old six days in seven and alarm (landmine recorded in the constant's comment, hub v0.75.0); (b) enumerate the state that exists ONLY in the whole-guest snapshot — guest rootfs, `controller.yaml`, `/etc/felhom-bootstrap`, systemd units, offbox SSH key, `settings.json` — and confirm none of it is catastrophic at 7 days old; (c) update `07-backup-architecture.md`. | -| R-83 | **Ratify or retire `07-backup-architecture.md`** | S | idea | DRAFT since 2026-07-14, verified against controller **v0.132.0** — now 41 versions stale (live is v0.173.0) — and cited as authoritative meanwhile. Either ratify it against current reality or mark it superseded; per the new CONTEXT ruling it must in any case carry an honest status header naming its verified-against version. | +| R-82 | **The backup target split — local daily + offsite weekly** | M | **SHIPPED (agent v0.102.0 + controller v0.175.0 + hub v0.76.0 + host-install 1.20.0, 2026-07-26)** | Phase 0: `audits/SPIKE-r82-phase0-2026-07-26.md` (weekly CONFIRMED; the `pvesm status` 0/0/0 resolved as a namespace-scoped-token reporting artifact; capacity measured). `backup_targets[]` gives each tier its OWN cadence and retention; the controller runs both due tiers in ONE quiesce window; the hub judges each against its own threshold (host 26h / offsite 8d); host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly, and an unprovisioned tier DEFERS rather than firing at a missing storage. Operator rulings 2026-07-26: 2-week offsite retention, first backup runs as long as needed, one backup at a time per guest, drill box dropped from the rollout. **Live:** demo-felhom + demo-hp both on local-24h + felhom-pbs-168h; demo-felhom's first real PBS backup landed (`TASK OK`, 41 min, 14.46 GB) and restored cleanly. **Four defects found by RUNNING it, not reviewing it:** 30-min wait bound vs a 41-min backup (v0.98.0), restore tier read from the configured target not the archive (v0.100.0 — a silent regression of the S4.1 fix), a leaked scratch keeping `onboot:1` (v0.101.0), and a tier firing at a not-yet-provisioned storage (v0.102.0). **REMAINING GATE:** the installer default is written but the fleet-wide flip waits on a full weekly cycle holding; and the offsite tier is never AUTOMATICALLY restore-tested (the scheduled test only selects the primary tier) — that gap, plus the hub inferring cadence from storage TYPE, are the two follow-ups. | +| R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. | | R-84 | **Persist the agent's backup `Store`** | S | idea — the CAUSE-fix behind R-81 | `felhom-agent/internal/backup/store.go` is in-memory by design ("lost on restart; the cadence re-populates"), so every agent restart blanks `backups` in the host-report until the next run. R-81 makes the hub interpret that correctly; this makes the report **truthful** rather than merely defensively interpreted. Either persist latest-per-target to the agent state dir (crash-safe journal/marker patterns already exist) or have the collector fall back to querying PVE for the newest archive on the target — the query path is arguably better, being ground truth rather than remembered state, and would also have surfaced the R-82 PBS gap. Small in code, but it is an agent deploy plus a MinAgent floor, so it is its own task. | | R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**) diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 9079d3d..a351173 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,25 @@ +## felhom-host-install 1.20.0 — R-82: a fresh box defaults to local-daily + offsite-weekly (2026-07-26) + +The `backup` defaults gain `backup_targets: [{target_id: "felhom-pbs", cadence_seconds: 604800, +keep_last: 2}]` — the offsite tier, weekly, two weeks retained (operator ruling 2026-07-26). + +**`setdefault` semantics preserved, and PROVEN both ways** against the exact line: + +``` +FRESH -> backup_targets: [{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}] +UPGRADE -> preserved verbatim, no backup_targets +``` + +That second line is the load-bearing one: **an in-place upgrade can never silently start writing to +an offsite datastore.** Existing boxes are migrated explicitly (R-82 Slice D.2), which is what keeps +a routine host-install re-run from enrolling a box into offsite backups nobody asked for. + +**Why the tier is written before its storage exists:** `felhom-pbs` only appears when the hub +provisions the DR tier. The agent (v0.102.0) DEFERS a tier whose target storage is absent, so the +tier stays silent until it is real and goes live with no restart the moment it is provisioned. +Writing it here means a box is never left with a DR tier that is provisioned but unscheduled — the +exact "applied and empty" state R-82 exists to end. + # Felhom scripts — Changelog ## felhom-host-install v1.19.0 — R-50 island control plane is the appliance default (2026-07-25) diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh index c45ded1..65a4880 100644 --- a/scripts/felhom-host-install.sh +++ b/scripts/felhom-host-install.sh @@ -184,7 +184,7 @@ set -euo pipefail -SCRIPT_VERSION="1.19.0" # the SINGLE version source (F-1): -h, the run banners, and the hub +SCRIPT_VERSION="1.20.0" # the SINGLE version source (F-1): -h, the run banners, and the hub # Setup-tab copy (hub internal/web/configs.go hostInstallVersion — # scripts/hostinstall_gates.py asserts the two stay equal) all follow it. # 1.16.0: the FELHOM_ESCROW sudoers alias (controller-driven escrow @@ -2288,7 +2288,21 @@ base['privileged'].setdefault('sudo_path','sudo') for _k,_v in {"unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agent/units","systemctl":"/usr/bin/systemctl","install":"/usr/bin/install","smartctl":"/usr/sbin/smartctl","lvs":"/usr/sbin/lvs"}.items(): base['privileged'].setdefault(_k,_v) base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20}) -base.setdefault('backup', {"local_backup_target":"local","local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0}) +# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions — +# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob +# would guarantee that one of them is wrong. keep_last=2 on the offsite tier = two weeks (operator +# ruling 2026-07-26). +# +# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only +# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent +# (localapi targetStoragePresent), so the tier stays silent until it is real and goes live with no +# restart the moment it is provisioned. Writing it here means a box is never left with a DR tier that +# is provisioned but unscheduled — the exact "applied and empty" state R-82 exists to end. +# +# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here — +# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start +# writing to an offsite datastore. +base.setdefault('backup', {"local_backup_target":"local","local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]}) base.setdefault('local_api', {}) base['local_api'].setdefault('enable', True) base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']