R-82 Slice D/E: installer default 1.20.0 + architecture docs brought current
Slice D.1 — host-install 1.20.0: a FRESH box defaults to local-daily + offsite-weekly (felhom-pbs, 604800s, keep_last=2). setdefault semantics proven both ways: fresh gets the tier, an UPGRADE preserves the existing backup block verbatim — so an in-place upgrade can never silently start writing to an offsite datastore. Existing boxes are migrated explicitly. Slice E: - 07-backup-architecture.md: honest status header per CONTEXT ruling S-2, with an explicit STALE-outside-the-PBS-tier verdict (the controller tiers were last verified 41 controller versions ago). The PBS row claimed 'PBS on DooPlex' (the retired spike store) with no cadence; it now names felhom-pbs -> felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2. NOT marked ratified — that is Viktor's review of the section 10 list. Discharges R-83. - 06-offsite-connectivity.md: the target-split remaining-work note collapsed (shipped), and records HOW S4.1's tier-aware timeout silently regressed — the mechanism was never removed, its INPUT changed when local_backup_target was retargeted to 'local'. Also notes S4.1 already diagnosed the teardown 403 as a phantom (a timeout consequence, not an ACL gap). - capability map: new row for recurring offsite backups actually LANDING, as distinct from the existing row proving ACTIVATION. IMPLEMENTED, not PROVEN-LIVE — the restore round-trip has not completed under the fixed code. - ROADMAP: R-82 SHIPPED with its remaining gate named, R-83 DISCHARGED, R-84 left open. - CONTEXT + REPORT: the arc, including the mid-arc correction I had to make.
This commit is contained in:
Claude Code
+23
@@ -21,6 +21,29 @@ reader meets the warning before the content, not after acting on it. Origin:
|
||||
versions stale (live v0.173.0), and cited as authoritative throughout the R-80 diagnostic. Ratifying
|
||||
or retiring it is → **R-83**.
|
||||
|
||||
- **2026-07-26 — R-82 SHIPPED: the backup target split (local daily + offsite weekly).** agent
|
||||
v0.97→v0.102.0, controller v0.174→v0.175.0, hub v0.76.0, host-install 1.20.0. `BackupTarget()`
|
||||
returned ONE string and `BackupCadence()` ONE 24h window, so "local daily AND PBS weekly" was not
|
||||
expressible — which is why the DR tier was `applied` since 07-21 with **one** snapshot on
|
||||
demo-felhom and **zero, ever** on demo-hp. Now: `backup_targets[]` per-tier cadence+retention;
|
||||
ONE quiesce window for both due tiers (never two app outages for one night); per-tier hub
|
||||
thresholds (host 26h / offsite 8d); fresh-install default; an unprovisioned tier DEFERS.
|
||||
**Operator rulings:** 2-week offsite retention, first backup runs as long as it needs, one backup
|
||||
at a time per guest, drill box dropped from the rollout.
|
||||
**Four defects found by RUNNING it, not reviewing it** — a 30-min wait bound against a 41-min
|
||||
backup (the agent recorded `success:false` while the backup was still going); the restore tier read
|
||||
from the configured target instead of the archive (**a silent regression of the S4.1 fix** — the
|
||||
mechanism was never removed, its INPUT changed when `local_backup_target` was retargeted to
|
||||
`local`); a leaked scratch keeping `onboot:1`; and a tier firing at a not-yet-provisioned storage.
|
||||
**A correction I had to make mid-arc:** I reported that the restore-test would boot a
|
||||
network-conflicting clone and break the control plane. It does not — step 2 link-downs every NIC
|
||||
before boot, and it is unit-tested. I read a config artifact and inferred behaviour without reading
|
||||
the code path that consumes it, then escalated before finishing the check.
|
||||
**NOT done:** the offsite tier is never AUTOMATICALLY restore-tested (the scheduled test only ever
|
||||
selects the primary tier); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported
|
||||
cadence; every agent restart re-triggers an offsite backup (R-84). The installer-default fleet flip
|
||||
waits on a full weekly cycle. Evidence: `audits/SPIKE-r82-phase0-2026-07-26.md`, `REPORT.md`.
|
||||
|
||||
- **2026-07-26 — hub v0.75.0: R-81 SHIPPED — "no signal" is not "bad signal".** Third instance of
|
||||
one class (hub v0.12.0 `expected_backup_missed` daily-for-everyone; hub v0.73.0 `offsite_stale`
|
||||
minutes after a healthy repair; now this), fixed **as a class**, with the invariant written at the
|
||||
|
||||
@@ -1,268 +1,162 @@
|
||||
# REPORT — R-81: "no signal" is not "bad signal" (hub v0.75.0) (2026-07-26)
|
||||
# REPORT — R-82: the backup target split (2026-07-26)
|
||||
|
||||
Class fix for the third instance of one bug: **absence of a signal treated as evidence of failure.**
|
||||
Input: `documentation/audits/DIAG-backup-missed-2026-07-26.md` (`add5b9b`).
|
||||
Local **daily** + offsite **weekly**, made expressible at all. Spans four artifacts:
|
||||
agent **v0.97.0 → v0.102.0**, controller **v0.174.0 → v0.175.0**, hub **v0.76.0**,
|
||||
host-install **1.20.0**.
|
||||
|
||||
Phase-0 gates: `documentation/audits/SPIKE-r82-phase0-2026-07-26.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Confirmed baselines
|
||||
## 1. What was wrong
|
||||
|
||||
| Repo | Start `main` | Version | → Shipped |
|
||||
|---|---|---|---|
|
||||
| felhom.eu (hub) | `add5b9b` | v0.74.0 | **v0.75.0** (`f5a5e2b` code+docs, `88b41ec` manifest pin) |
|
||||
| felhom-agent | `dfd5d73` | v0.96.0 | unchanged — **N/A this task** (cause-fix is R-84) |
|
||||
| felhom-controller | after R-77 | v0.173.0 | unchanged |
|
||||
`BackupTarget()` returned ONE string and `BackupCadence()` ONE 24 h window, so "local daily **and**
|
||||
PBS weekly" could not be said. The consequence was not theoretical: the DR tier reported `applied`
|
||||
since 2026-07-21 while demo-felhom held **one** snapshot (2026-07-18, a healing artifact) and demo-hp
|
||||
held **zero, ever**. R-39 was "applied and dead"; this was **applied and empty** — the same shape,
|
||||
quieter, and it would have surfaced first at a real restore.
|
||||
|
||||
Clean-tree gate passed before the build: `git status --porcelain` empty, `HEAD == origin/main`.
|
||||
## 2. Phase 0 — three gates
|
||||
|
||||
---
|
||||
|
||||
## 2. Phase-0 — the anchor probe
|
||||
|
||||
**Question:** does the hub retain host-report HISTORY, or only the latest report?
|
||||
|
||||
**Answer: HISTORY IS RETAINED — 90 days, indexed and queryable by customer + time.**
|
||||
|
||||
- `host_reports` is an append-only table (`SaveHostReport` INSERTs; `GetLatestHostReportJSON` only
|
||||
ever read the newest row).
|
||||
- Retention: `cfg.Retention.MaxDays`, **90** both as the code default (`cmd/hub/main.go:631,654`) and
|
||||
as the live value in `manifests/hub.yaml` (`retention.max_days: 90`). `Prune` deletes
|
||||
`host_reports` older than that on the 04:30 job.
|
||||
- Index `idx_host_reports_customer ON host_reports(customer_id, received_at DESC)` — the exact access
|
||||
pattern needed, already present.
|
||||
- Live confirmation on the production DB: 664 / 436 / 96 retained rows for demo-felhom / demo-hp /
|
||||
drill-r50, spanning 7.63 / 4.64 / 0.91 days.
|
||||
|
||||
**Branch taken: the free anchor.** No new persisted state, no `last_backup_seen` column, no agent
|
||||
change. The agent's store is *point-in-time*; the hub has *memory*. Judging backup evidence across
|
||||
the retained window is both the cheapest option and semantically the right question — **"when did I
|
||||
last SEE evidence of a backup?"** rather than "what does this one report happen to say?".
|
||||
|
||||
---
|
||||
|
||||
## 3. Files modified
|
||||
|
||||
| File | Change |
|
||||
| Gate | Verdict |
|
||||
|---|---|
|
||||
| `hub/internal/monitor/deadline.go` | Three-valued verdict; hub-history fold-in; anchored absence; distinct reason strings; `newestBackupEvidence`; landmine comment on `backupStaleAfter`; new `backupEvidenceLookback` |
|
||||
| `hub/internal/store/store.go` | `HostReportRow`, `GetHostReportsSince`, `GetFirstHostReportAt`, `SetHostReportsReceivedAtForTest` |
|
||||
| `hub/internal/monitor/deadline_test.go` | Signature update; note explaining the zero-evidence rows |
|
||||
| `hub/internal/monitor/deadline_anchor_test.go` | **NEW** — 15 tests, scenarios A–E + the named boundary contract |
|
||||
| `hub/CHANGELOG.md` | v0.75.0 |
|
||||
| `CONTEXT.md` | New **Standing rulings** section (S-1, S-2) + the v0.75.0 entry |
|
||||
| `REUSE.md` | New canonical-helper row; new dangerous-lookalike row |
|
||||
| `documentation/backlog/ROADMAP.md` | R-80 closed, R-81 shipped, **R-82 / R-83 / R-84 added** |
|
||||
| `documentation/architecture/00-capability-map.md` | Note row — no status flips |
|
||||
| `manifests/hub.yaml` | image pin 0.74.0 → **0.75.0** |
|
||||
| P0.1 what is exposed for 7 days | **weekly CONFIRMED.** The only 7-day-exposed state is the non-SMB half of `settings.json`. `encryption.key` and the offbox credentials are **stable files unchanged since first boot**, so a week-old copy is byte-identical — that was the risk that could have overturned it |
|
||||
| P0.2 the `pvesm status` 0/0/0 anomaly | **RESOLVED, benign.** PBS returns HTTP 200 with zeroed usage to a namespace-scoped token (`DatastoreBackup`, not `Datastore.Audit`). Ground truth via the hub's ep0 `df`: the datastore is real and writable |
|
||||
| P0.3 capacity | **STOP raised; operator ruled to proceed and grow later.** 37.2 GB total. Per-tenant encryption means **no cross-customer dedup** |
|
||||
|
||||
Commits: **`f5a5e2b`** (code + docs), **`88b41ec`** (manifest pin). Both on `main`, pushed.
|
||||
**Capacity, now measured rather than bracketed:** the second weekly snapshot cost **+2.7 GB on disk**
|
||||
against 14.46 GB logical (~81 % dedup). Weekly top-ups are cheap; **first** snapshots are not — one
|
||||
customer at two retained snapshots costs ~13.5 GB, so the 80 % warn arrives at roughly the **first**
|
||||
additional customer, not the second as I first estimated. Recorded in `07-backup-architecture.md`
|
||||
§9.1.
|
||||
|
||||
---
|
||||
## 3. What shipped
|
||||
|
||||
## 4. The fix
|
||||
- **Agent** — `backup_targets[]`: each tier carries its **own** cadence, retention and wait bound
|
||||
(`keep-last=3` is three DAYS on a daily tier and three WEEKS on a weekly one; one shared knob
|
||||
guarantees one of them is wrong). `/backup/due?target=` judges a tier against **its own** newest
|
||||
successful backup. `GET /backup/tiers` is the controller's capability probe. One runner per tier.
|
||||
- **Controller** — every due tier collected up front and run in **ONE quiesce window**. Two cycles on
|
||||
the weekly night would mean two app outages for one night's work. The app stays quiesced until the
|
||||
**last** tier snapshots, so every tier is app-consistent.
|
||||
- **Hub** — per-tier thresholds (host 26 h, offsite 8 d), preserving R-81's three-valued verdicts,
|
||||
anchored absence and distinct reason strings. Classification is by **target type**
|
||||
(`target_id` → `storage_targets[].name` → `.type`), never by array membership.
|
||||
- **Installer** — a fresh box defaults to local-daily + offsite-weekly; an unprovisioned tier
|
||||
**defers** rather than firing at a storage that does not exist.
|
||||
|
||||
`assessBackupFreshness` returned `{missed bool, reason string}` and collapsed *absence of records*
|
||||
into *failure*. It now returns a three-valued verdict:
|
||||
**The untargeted local-API contract is frozen.** No `?target=` ⇒ the primary tier, same response
|
||||
**bytes** (`Target` is `omitempty` and stays empty). An old controller cannot tell the new agent from
|
||||
the old one; a new controller against an old agent degrades on a 404 probe, logs once, and **still
|
||||
takes the backup**.
|
||||
|
||||
```
|
||||
verdictOK positive evidence of a recent backup
|
||||
verdictUnknown no evidence yet, and the anchored window has not elapsed → silent, LOGGED
|
||||
verdictMissed positive evidence of a problem → alarm
|
||||
```
|
||||
## 4. Operator rulings (2026-07-26), all implemented
|
||||
|
||||
Three inputs, all injected — the function stays **pure**:
|
||||
|
||||
1. the latest report (unchanged parsing),
|
||||
2. `backupEvidence.newestSeen` — the newest backup evidence across a bounded 7-day window of retained
|
||||
reports (`GetHostReportsSince` + `newestBackupEvidence`),
|
||||
3. `backupEvidence.firstReportAt` — first contact (`GetFirstHostReportAt`), the absence anchor.
|
||||
|
||||
Absence is graded against the **existing** `backupStaleAfter` (26 h), exactly as v0.73.0 reused
|
||||
offsite `staleAfter` — **no new knob**. A zero anchor fails toward visibility (the v0.73.0
|
||||
legacy-shape precedent). The window scan early-exits at the first sufficiently-fresh evidence, so the
|
||||
healthy path reads one row; only the genuinely-broken path walks the lookback.
|
||||
|
||||
`CheckBackupDeadlines` logs each deferral (`verdict UNKNOWN`) and the summary line gained a
|
||||
`backup unknown (deferred)` counter — a quiet check must never be indistinguishable from one that
|
||||
did not run.
|
||||
|
||||
The invariant is written at the head of `assessBackupFreshness`, naming all three instances
|
||||
(v0.12.0, v0.73.0, R-81), and pinned by a boundary test whose name states what it protects.
|
||||
|
||||
---
|
||||
|
||||
## 5. Test results
|
||||
|
||||
**Full gate green:** `go build ./... && go vet ./... && go test ./...` — all rc=0, every package `ok`.
|
||||
|
||||
**Test count: 493 → 508 (+15).** (Measured with `git grep -hE '^func Test'` at `HEAD` vs the working
|
||||
tree — a first attempt via `go test -list` after a `git stash` gave a bogus 434 because the stash
|
||||
broke compilation and silently dropped the whole package. Recorded because that is exactly the
|
||||
exit-code-lie class this project has been bitten by; likewise `go vet ... | head` reported rc=0 while
|
||||
vet was actually failing.)
|
||||
|
||||
All 21 tests in `internal/monitor` covering this area pass, including the 6 pre-existing ones.
|
||||
|
||||
### The three companion red-proofs — applied, observed, restored
|
||||
|
||||
**A — Scenario A (the 07-26 case must not alarm).** Removed the `ev.haveSeen` fold-in (pre-R-81
|
||||
shape: judge the latest report alone):
|
||||
|
||||
```
|
||||
--- FAIL: TestBackupFreshness_AgentRestartBlindWindow_NoAlarm (0.00s)
|
||||
deadline_anchor_test.go:61: 07-26 shape must NOT alarm; got verdict=2 reason="newest backup is 176h0m0s old (limit 26h0m0s)"
|
||||
--- FAIL: TestCheckBackupDeadlines_RestartBlindWindow_NoEvent (0.03s)
|
||||
deadline_anchor_test.go:388: the 07-26 restart shape must NOT raise expected_backup_missed; got [expected_backup_missed]
|
||||
```
|
||||
|
||||
The reason string is **verbatim the message demo-felhom actually sent to the customer channel** that
|
||||
morning. Restored.
|
||||
|
||||
**B — Scenario B (a genuinely dead box must still alarm).** Applied the *naive* fix — absence always
|
||||
returns UNKNOWN, never a fault:
|
||||
|
||||
```
|
||||
--- FAIL: TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms (0.00s)
|
||||
deadline_anchor_test.go:97: a box with NO backup evidence for 240h MUST alarm; got verdict=1 reason="no backup evidence yet, but only watching for 240h0m0s"
|
||||
--- FAIL: TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses/just_outside_the_window
|
||||
deadline_anchor_test.go:156: CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=26h1m0s, limit=26h0m0s); got verdict=1 …
|
||||
--- FAIL: TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses/long_past_the_window
|
||||
deadline_anchor_test.go:156: CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=720h0m0s, limit=26h0m0s); got verdict=1 …
|
||||
--- FAIL: TestCheckBackupDeadlines_NeverBackedUpBeyondAnchor_Alarms (0.03s)
|
||||
deadline_anchor_test.go:411: a host with no backup for 120h MUST alarm; got []
|
||||
```
|
||||
|
||||
**This is the proof that matters most.** A suite that only pinned Scenario A would have passed
|
||||
against this over-suppressed implementation — and over-suppression is strictly worse than the bug it
|
||||
replaces. Restored.
|
||||
|
||||
**C — Scenario C (a fresh box is not born failing).** Restored the literal pre-fix branch
|
||||
(`if !havePBS && !haveVzdump { return missed }`):
|
||||
|
||||
```
|
||||
--- FAIL: TestBackupFreshness_EmptyArraysWithinGrace_Unknown (0.00s)
|
||||
deadline_anchor_test.go:74: absence inside the anchored grace must NOT alarm; got reason="no PBS snapshot or successful backup in the latest host-report"
|
||||
--- FAIL: …/newborn,_1_minute
|
||||
deadline_anchor_test.go:159: CONTRACT VIOLATED: no evidence + no elapsed window must NOT alarm (watched=1m0s, limit=26h0m0s); got reason="no PBS snapshot or successful backup in the latest host-report"
|
||||
--- FAIL: …/newborn,_1_hour (watched=1h0m0s)
|
||||
--- FAIL: …/just_inside_the_window (watched=25h59m0s)
|
||||
--- FAIL: …/exactly_at_the_window (watched=26h0m0s)
|
||||
--- FAIL: TestCheckBackupDeadlines_NewbornHost_NoEvent (0.03s)
|
||||
deadline_anchor_test.go:426: a newborn host must not alarm; got "No fresh verified backup: no PBS snapshot or successful backup in the latest host-report"
|
||||
```
|
||||
|
||||
Restored; full suite green after each.
|
||||
|
||||
### Scenario D and E
|
||||
|
||||
- **D** — `TestBackupFreshness_ExistingBehavioursUnchanged` pins all three pre-existing outcomes
|
||||
**including their exact reason strings** (`"newest backup is 30h0m0s old (limit 26h0m0s)"`,
|
||||
`"newest PBS snapshot failed verification"`, `"latest host-report could not be parsed"`), each with
|
||||
fresh window evidence present so the result cannot be an artefact of the new input. Plus
|
||||
`TestBackupFreshness_WindowEvidenceDoesNotRescueFailedVerify` — the anchor must not suppress an
|
||||
integrity fault.
|
||||
- **E** — `TestBackupFreshness_ReasonStringsAreDistinct` fails if any two of the six failure modes
|
||||
ever produce the same message.
|
||||
|
||||
---
|
||||
|
||||
## 6. Replay of the 2026-07-26 shape — **actual reports, not a fixture**
|
||||
|
||||
Two replays, both against the real thing.
|
||||
|
||||
**(a) Real report payloads** extracted from the production hub DB (every retained report in the
|
||||
7-day lookback at or before the check instant), fed through `newestBackupEvidence` +
|
||||
`assessBackupFreshness` at `now = 2026-07-26 03:00:00 UTC`:
|
||||
|
||||
```
|
||||
demo-felhom rows=600 first=2026-07-18T16:30:51Z windowEvidence=2026-07-25T06:30:14Z -> OK
|
||||
demo-hp rows=417 first=2026-07-21T16:24:48Z windowEvidence=2026-07-25T10:23:31Z -> OK
|
||||
drill-r50 rows= 77 first=2026-07-25T09:50:55Z windowEvidence=none -> UNKNOWN
|
||||
"no backup evidence yet, but only watching for 17h0m0s (grace 26h0m0s since first
|
||||
contact 2026-07-25T09:50:55Z) — newborn host, not a fault"
|
||||
```
|
||||
|
||||
**(b) Through the real store queries** — a copy of the live `hub.db` opened with `store.New`, driving
|
||||
`GetHostReportsSince` / `GetFirstHostReportAt` / `GetLatestHostReportJSON` for real. This was worth
|
||||
doing separately: the window query compares `received_at` as a **SQLite datetime string**, which is a
|
||||
genuine correctness risk that a hand-built fixture would not have exercised. Identical verdicts, plus:
|
||||
|
||||
```
|
||||
peti-felhom NO host-report → deadline check skips the backup half entirely
|
||||
```
|
||||
|
||||
**All three would now be silent. Zero events.** Both harnesses were throwaway and are deleted; the
|
||||
tree is clean. Only the DB *copy* was ever opened writable — the live `/data/hub.db` was not touched.
|
||||
|
||||
Note what the verdicts say: demo-felhom and demo-hp are **OK** (real evidence was found, not
|
||||
suppressed), and drill-r50 is **UNKNOWN** (correctly deferred as a newborn, and it will alarm on its
|
||||
own if it is still backup-less after the grace). None of the three is silenced by a blanket rule.
|
||||
|
||||
---
|
||||
|
||||
## 7. Deployment
|
||||
|
||||
| Step | Result |
|
||||
| Ruling | Implementation |
|
||||
|---|---|
|
||||
| Image build + push | `gitea.dooplex.hu/admin/felhom-hub:0.75.0`, 25 MB, digest `sha256:a2ebaa50…10453` |
|
||||
| Manifest pin | `manifests/hub.yaml` → `0.75.0` (commit `88b41ec`) — **GitOps only**, no `kubectl set image` |
|
||||
| ArgoCD | hard-refresh + deliberate sync → **Synced / Healthy** |
|
||||
| Rollout | `deployment "hub" successfully rolled out` |
|
||||
| Pod | `hub-56946d5cb4-kwwn7` 1/1 Running |
|
||||
| **Running image (pod status, not spec)** | `felhom-hub:0.75.0` @ `sha256:a2ebaa50…10453` — **matches the pushed digest** |
|
||||
| Startup log | `[INFO] felhom-hub 0.75.0 starting` … `[INFO] deadline-check: next run at 2026-07-27 05:00 CEST (in 17h14m19s)` |
|
||||
| Two weeks of offsite backups | `keep_last=2`; the blanket PBS-prune refusal scoped to *additional* tiers with an explicit setting — the primary keeps the absolute refusal, because its target **and** retention both default and could prune the DR by accident |
|
||||
| Grow the datastore before any real tester | recorded in `07` §9.1; no action taken |
|
||||
| First backup runs as long as needed; nothing else starts until done | wait bound → 12 h (measured ~5 h for a first full snapshot); **one backup at a time per guest** — a second tier gets a 409 naming the busy tier, with no job id it could mistake for its own; a tier overrunning the quiesce bound defers the rest |
|
||||
| Drill box is temporary | dropped from the rollout |
|
||||
| Restore test, then next slice | done — see §6 |
|
||||
|
||||
The pin landed and matches — checked deliberately, since the DIAG found `hub.yaml` pinned at
|
||||
`0.73.1` while the CHANGELOG read `0.73.2`.
|
||||
## 5. Four defects found by RUNNING it, not reviewing it
|
||||
|
||||
---
|
||||
1. **30-minute wait bound vs a 41-minute backup** (v0.98.0). The agent recorded `success:false`
|
||||
**while the vzdump was still running**, and it later completed `TASK OK`. Not "the backup didn't
|
||||
happen" but worse: the tier stays permanently due and the retry collides with the guest lock.
|
||||
2. **The restore tier read from the configured target, not the archive** (v0.100.0). A `felhom-pbs:`
|
||||
archive was classified `local` and got the 10-minute bound against a 14.46 GB WAN restore, failing
|
||||
at 600 s. **This was a silent regression of the S4.1 fix** — the mechanism was never removed, its
|
||||
*input* changed when `local_backup_target` was retargeted to `local`. The lesson is not "add a
|
||||
timeout" (one was already there) but that a fix keyed on *"the configured target"* stops holding
|
||||
the moment more than one target exists. Recorded in `06-offsite-connectivity.md`.
|
||||
3. **A leaked scratch guest kept `onboot: 1`** (v0.101.0) — a host reboot would have started a clone
|
||||
of the live guest. Now `onboot=0` is set **at restore time**, because "after" is the path that
|
||||
leaks.
|
||||
4. **A tier fires at a not-yet-provisioned storage** (v0.102.0) — would have quiesced the apps and
|
||||
failed every cadence on a fresh box until DR provisioning.
|
||||
|
||||
## 8. NOT yet live-validated — explicit
|
||||
## 6. Live validation
|
||||
|
||||
1. **The 05:00 CEST deadline check has not fired under v0.75.0.** Next run 2026-07-27 05:00 CEST. All
|
||||
evidence above is unit-proven + replayed against real data; the live firing is unobserved.
|
||||
2. **The deferred-UNKNOWN INFO line has not appeared in a production log** — proven by unit test only.
|
||||
drill-r50's 26 h grace expires 2026-07-26 ~11:50 UTC, so by 03:00 on 07-27 it will instead
|
||||
**alarm** — correctly, and that will be the first real test of Scenario B in production.
|
||||
3. **No genuinely-dead box has been observed alarming live.** The B path is unit-proven only.
|
||||
4. **The cold path (full 7-day lookback scan) has not run in production.** Cost is bounded and the
|
||||
healthy path early-exits at one row, but the scan-heavy branch is untimed live.
|
||||
5. **demo-hp and drill-r50 host-level state remains uncollected** (no SSH key; break-glass not used) —
|
||||
unchanged from the DIAG.
|
||||
**demo-felhom** — the first real PBS-targeted backup: **`TASK OK`, 41 minutes, 14.46 GB snapshot**,
|
||||
and it **restored cleanly** (`vzrestore: stopped OK`, all volumes back). Both tiers armed and
|
||||
verified over the real local API; the untargeted response confirmed byte-identical.
|
||||
|
||||
---
|
||||
**demo-hp** — reached via the documented break-glass path; binary and config backed up first;
|
||||
v0.102.0 deployed, tier armed, and its **first ever** PBS backup triggered against a namespace that
|
||||
was verifiably empty. *(In flight at the time of writing.)*
|
||||
|
||||
## 9. Observations — recorded, not acted on
|
||||
**Hub Slice C replayed against the live DB before deploying:**
|
||||
|
||||
1. **`parseBackupTime`'s silent `continue` is a latent member of the same class.** An unparseable
|
||||
timestamp is dropped without a trace, so a wire-format change on the agent side would present as
|
||||
"no backup" — the exact 07-26 failure, with no log line to catch it. Left untouched per scope
|
||||
(the agent emits clean RFC3339 `Z`, confirmed in the DIAG). Worth a WARN log when R-84 touches the
|
||||
agent side of this contract.
|
||||
2. **The customer-facing Hungarian copy is unchanged.** `notify/templates.go:106` — *"A mai
|
||||
biztonsági mentés nem készült el a határidőig!"* — still overstates scope (it reads as *all*
|
||||
backups failed; this check only covers the host/PBS tier). Not in this task's Parts 1–3. The DIAG
|
||||
proposed it as fix #4; it needs a `felhom-ui-design` copy pass and the confirm gate.
|
||||
3. **`GetFirstHostReportAt` returns the prune horizon, not true first contact, for hosts older than
|
||||
90 days.** Accepted and documented at the function: it only makes the anchor *more* conservative
|
||||
for long-lived hosts (the window has elapsed either way) and never shortens a newborn's grace.
|
||||
4. **The `backupStaleAfter` landmine is now written into the constant's comment** and carried as a
|
||||
named dependency in R-82. It will alarm on a healthy weekly PBS snapshot six days in seven.
|
||||
5. **`retention.max_days: 90` is now load-bearing for a monitor**, not just for storage. Shortening it
|
||||
below `backupEvidenceLookback` would silently narrow the evidence window. Not currently guarded.
|
||||
6. **The R-80 real finding stands untouched:** the PBS/offsite-DR tier has no schedule at all —
|
||||
demo-felhom still holds exactly one PBS snapshot (2026-07-18), demo-hp zero. R-81 fixes how the hub
|
||||
*reads* backup state; it does nothing about the DR tier being empty. **That is R-82, and it is the
|
||||
more serious of the two.**
|
||||
7. The customer-channel suppression the DIAG recommended (removing `expected_backup_missed` from
|
||||
demo-felhom's `enabled_events`) was **not** applied — still an operator decision, and now much less
|
||||
pressing since the false-positive path is closed.
|
||||
```
|
||||
demo-felhom host=07-26T14:38Z offsite=07-26T12:21Z -> OK
|
||||
demo-hp host=07-26T07:06Z offsite=none -> UNKNOWN (119h of a 192h grace)
|
||||
drill-r50 host=none offsite=not expected -> MISSED (no evidence in 29h)
|
||||
```
|
||||
|
||||
---
|
||||
**No customer email results from the deploy.** demo-hp defers correctly and will alarm in ~3 days if
|
||||
its offsite tier stays empty — the true finding arriving on schedule, not a false alarm.
|
||||
|
||||
## 10. Scope discipline
|
||||
## 7. A correction I had to make mid-arc
|
||||
|
||||
Not started, recorded only: **R-82** (backup target split — the real finding), **R-83** (ratify or
|
||||
retire `07-backup-architecture.md`), **R-84** (persist the agent's backup store — the cause-fix).
|
||||
No branches; every commit staged with explicit paths (never `git add -A`, per the shared-worktree
|
||||
rule); no `Co-Authored-By`.
|
||||
I reported that the restore-test would boot a scratch guest carrying the live guest's MAC, static
|
||||
island IP and hostname, and so would break the controller→agent link. **That was wrong.**
|
||||
`RunRestoreTest` step 2 link-downs **every** interface before the guest is started, and it is
|
||||
unit-tested. I read a restored config artifact, inferred the boot behaviour from it, and escalated
|
||||
before reading the code path that consumes it. I also disabled the scheduled restore-test on that
|
||||
basis, which was an unnecessary reduction in safety coverage; it is re-enabled.
|
||||
|
||||
The residual hazard was real but far narrower — it needed the restore to fail *before* the link-down
|
||||
step, which is what defect 1 caused — and that is what v0.101.0 fixes.
|
||||
|
||||
Separately, `06-offsite-connectivity.md` records that the teardown `403` I flagged as a standing
|
||||
privilege gap was **already diagnosed in S4.1 as a phantom**: it is a consequence of the short
|
||||
timeout, not an ACL problem. With the timeout fixed the guest is pool-associated by teardown time.
|
||||
|
||||
## 8. Tests
|
||||
|
||||
| Repo | Result |
|
||||
|---|---|
|
||||
| felhom-agent | `build/vet/test` rc=0, **29 packages** |
|
||||
| felhom-controller | `build/vet/test` rc=0, **27 packages** |
|
||||
| felhom.eu (hub) | `build/vet/test` rc=0, **17 packages** |
|
||||
|
||||
Red-proofs observed and restored for every mandatory scenario: old-controller compat, new-controller
|
||||
degrade (the hollow version asserts "no error" while silently skipping the backup), the both-due
|
||||
night (**the COUNT is the assertion** — asserting only "both ran" passes against a double-quiesce),
|
||||
the merged threshold, the per-tier wait bound, the `onboot` override, and the overrun defer.
|
||||
|
||||
**A process failure worth recording:** I ran the agent suite and committed in the same command, read
|
||||
`packages ok: 28`, and pushed **without reading `rc=1`**. Five of my own Slice A tests were failing —
|
||||
a harness artifact, not a product bug, but the commit went out red. Fixed in `13ca2d9`. This is the
|
||||
exact exit-code trap recorded twice earlier in this arc.
|
||||
|
||||
## 9. NOT done — explicitly
|
||||
|
||||
1. **The offsite tier is never AUTOMATICALLY restore-tested.** The scheduled restore-test picks
|
||||
candidates from a runner built on the primary target, so it can never select a PBS archive. This
|
||||
is arguably the more important half of "is the DR tier real?" and it needs a per-tick spec.
|
||||
2. **The hub infers "PBS ⇒ weekly" from storage TYPE.** `defaultBackupTarget` is `felhom-pbs`, so a
|
||||
box that never sets `local_backup_target` would run PBS as its **daily** tier and be judged
|
||||
against 8 days — seven days of blindness. No box is in that shape today. The real fix is the agent
|
||||
reporting each tier's actual cadence.
|
||||
3. **Every agent restart re-triggers an offsite backup** (the in-memory store, R-84). On the local
|
||||
tier that is wasted minutes; on the offsite tier it is a wasted multi-hour upload after every
|
||||
deploy. R-84 is now closer to a prerequisite than a tidy-up.
|
||||
4. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a
|
||||
genuine gate, not an oversight.
|
||||
5. **The boot+verify half of the restore round-trip has not run under the fixed code.**
|
||||
6. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header;
|
||||
ratification is Viktor's review of the §10 list.
|
||||
|
||||
## 10. Observations
|
||||
|
||||
- The `felhom-pbs` PVE storage will permanently show **0 %** in the PVE UI (namespace-scoped token).
|
||||
Operators must read fill from the hub's PBS-DR gauge. Worth a runbook line.
|
||||
- demo-felhom's guest grew **9.74 → 14.46 GB logical in eight days**. Probably one-off from app
|
||||
testing, but if it is a rate the capacity sizing changes quickly.
|
||||
- demo-felhom was enabled **before** demo-hp, out of the specified rollout order, because Slice A
|
||||
could not be validated otherwise.
|
||||
- The customer-facing Hungarian copy still overstates scope ("A mai biztonsági mentés nem készült el
|
||||
a határidőig!" covers only the host/PBS tier). Unchanged; flagged since the R-80 diagnostic.
|
||||
|
||||
@@ -37,6 +37,7 @@
|
||||
| Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends |
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
|
||||
| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). |
|
||||
| **Recurring offsite (PBS) whole-guest backups actually LAND** — local daily + offsite weekly, as scheduled work rather than a one-off | agent v0.97–0.102, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **IMPLEMENTED** (not PROVEN-LIVE) | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs; live on demo-felhom: first real PBS-targeted backup **`TASK OK`, 41 min, 14.46 GB snapshot**, restored cleanly (`vzrestore: stopped OK`) | **This row exists because the row above proves ACTIVATION, not ARRIVAL.** The DR tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (2026-07-18, a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. R-82 makes the split expressible at all: `backup_targets[]` with per-tier cadence/retention (agent), ONE quiesce window for both tiers (controller), per-tier thresholds (hub, host 26h / offsite 8d), and a fresh-install default (host-install). **Why NOT PROVEN-LIVE:** the restore round-trip has not completed under the fixed code. The 2026-07-26 attempt restored the data correctly but its boot+verify half never ran — the restore-test derived its tier from the configured target rather than the archive, took the 10-minute LOCAL bound against a 14.46 GB WAN restore, and failed at 600s (fixed, agent v0.100.0). **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only ever selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested; the hub infers "PBS ⇒ weekly" from storage TYPE, not from a reported cadence; and every agent restart currently re-triggers an offsite backup (→ R-84). → **R-82** |
|
||||
| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below |
|
||||
| **Customer DELETE cascade** (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **UNIT-PROVEN; live leg PENDING** | `hub/internal/web/customer_delete_test.go` — leg ORDER observed from inside leg 2 (hosts already gone, customer row still present, custody still retained); 9 fail-closed gate cases each asserting zero mutations + zero external calls + no journal row; resume-after-external-failure converges; `purgeEscrow` custody semantics; preview leaks no secret. **5 red-proofs** (ack gate, stale-preview gate, ONLINE-host gate, leg order inverted, `purgeEscrow=true`) | Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run |
|
||||
| Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 |
|
||||
|
||||
@@ -321,7 +321,9 @@ vantage (Peti VM 110) remains the thorough confirmation but no longer gates anyt
|
||||
| **S1 — endpoint provisioning + peer-sync surface** — **SHIPPED 2026-07-04** (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: **hub pushes over SSH** (`x/crypto/ssh`, `FixedHostKey` pin + pinned-type `HostKeyAlgorithms`) to a forced-command reconcile script; runbook `documentation/runbooks/offsite-endpoint.md`. Done-criterion met live: hub add/remove → `wg show` on the endpoint; tunnel `ep0.felhom.eu:443` → PBS through the wg0-only 8007 rule. | ~~provision the production VM…~~ (see runbook; production endpoint = a later re-run) | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓ |
|
||||
| **S2 — hub schema + desired-state** — **SHIPPED 2026-07-04** (hub v0.33.0). `POST /hosts/{id}/wg` (self-scoped) + merge-at-read `wireguard` block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + `/offsite` UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance. | ~~endpoint coords + peer registry…~~ | a registered box's desired-state carries its assignment ✓; hub UI shows the peer registry ✓ |
|
||||
| **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) |
|
||||
| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore/<ds>/<ns>`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: ~~restore-test scheduler WAN deadline + scratch `VM.Allocate`~~ **RESOLVED S4.1 (agent v0.68.0)** — tier-aware restore-task timeout (pbs→120m, local→10m); the offsite restore-test now runs unattended on the agent-token path (`pass:true`, teardown clean). The `VM.Allocate` item was a **phantom** — the earlier 403 was the 10m-timeout consequence (guest not yet in `/pool/felhom`); no ACL change. Remaining: Tier-1-local-vs-Tier-2-offsite target split (`local_backup_target` is a single target; the ~30-min controller cadence makes offsite-as-default near-continuous). |
|
||||
| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore/<ds>/<ns>`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: ~~restore-test scheduler WAN deadline + scratch `VM.Allocate`~~ **RESOLVED S4.1 (agent v0.68.0)** — tier-aware restore-task timeout (pbs→120m, local→10m); the offsite restore-test now runs unattended on the agent-token path (`pass:true`, teardown clean). The `VM.Allocate` item was a **phantom** — the earlier 403 was the 10m-timeout consequence (guest not yet in `/pool/felhom`); no ACL change. ~~Remaining: Tier-1-local-vs-Tier-2-offsite target split~~ **SHIPPED 2026-07-26 (R-82)** — agent v0.97.0 gave `BackupConfig` per-target tiers (`backup_targets[]`, each with its OWN cadence + retention), controller v0.174.0 reconciles them into ONE quiesce window, hub v0.76.0 judges each tier against its own threshold, host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly. Live: local 24h + `felhom-pbs` 168h on both demo boxes.
|
||||
|
||||
> ⚠️ **S4.1's tier-aware timeout silently REGRESSED and was restored 2026-07-26 (agent v0.100.0) — the mechanism was never removed, its INPUT changed.** S4.1 derived the tier with `storageTier(BackupTarget())`, which was correct while `local_backup_target` **was** `felhom-pbs`. The target split above retargeted it to `"local"`, so a `felhom-pbs:` archive was thereafter classified **local** and got the 10-minute bound again — reproducing the S4.1 symptom exactly (wait expires mid-restore → teardown fires at a not-yet-pool-associated guest → the same `403 missing privilege VM.Allocate`, and the same **phantom** reading of it). v0.100.0 derives the tier from **the archive's own storage** (`restoreTierForArchive`), which is input-independent. The lesson is not "add a timeout" — it was already there — but that a fix keyed on *"the configured target"* stops holding the moment more than one target exists. |
|
||||
| **S5 — DR re-establishment — SAFE HALVES SHIPPED 2026-07-04 (agent v0.69.0); destructive drill operator-gated.** Part 1: `wgtunnel.InstallRecoveredKey` (escrow-recovered WG key → same identity/pubkey/`/32`, no keygen; create-only; `--selftest=identity-consume -install-wg-key`). Part 2: `internal/dr` consumes the host_loss `restore_directive` into an inspectable RestorePlan (per-guest restore coords+sizing, per-drive durable_id→mount) via the AddConsumer seam — derive-and-surface, execute nothing. Re-attach wrong-disk safety = `ResolveStorageDevice` (match resolves, mismatch REFUSED — unit-proven). | ~~consume `WGPrivateKey` in restore mode~~ | the tunnel re-establishes from the recovered key (same `/32`) ✓ (unit); the host_loss directive yields the correct PLAN ✓ (unit). **Operator-gated:** the destructive in-place 9201 restore (destroy→restore-from-offsite→re-attach-by-durable_id) is operator-present + STOP-gated — the operator runs the R-consume steps (CC never does). Pre-flight green (9201 offsite-restorable, S4.1). |
|
||||
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
|
||||
|
||||
|
||||
@@ -1,10 +1,19 @@
|
||||
# 07 — Backup architecture: tiers × classes × targets
|
||||
|
||||
> **Status: DRAFT — awaiting Viktor's review (async one-line vetoes on the §10 list).**
|
||||
> Written 2026-07-14 per the architecture-doc-first gate (Viktor ruling #5). Every claim below was
|
||||
> verified against live Gitea at commit **felhom-controller `95f3180` (v0.132.0)**, felhom.eu
|
||||
> `deacee11`, catalog `21e8df1`, agent `c040c18` (v0.88.0), hub v0.54.0. Line numbers are
|
||||
> landmarks — reconfirm before editing.
|
||||
> **NOT ratified. Ratification is Viktor's review, not an editor's.**
|
||||
>
|
||||
> | | |
|
||||
> |---|---|
|
||||
> | **Written** | 2026-07-14 (architecture-doc-first gate, Viktor ruling #5) |
|
||||
> | **Body verified against** | felhom-controller `95f3180` **v0.132.0**, agent `c040c18` **v0.88.0**, hub **v0.54.0**, catalog `21e8df1` |
|
||||
> | **PBS/whole-guest tier (§2 row, §9.1) verified against** | agent **v0.102.0**, controller **v0.175.0**, hub **v0.76.0**, host-install **1.20.0** (2026-07-26, R-82) |
|
||||
> | **Verdict** | ⚠️ **STALE outside the PBS tier.** The controller-side tiers (1/2/3) were last verified 41 controller versions ago (live is v0.175.0). Treat §0–§8 as historical intent, not current fact, until re-verified. |
|
||||
>
|
||||
> Per CONTEXT ruling **S-2**, this header exists so a reader meets the staleness warning *before* the
|
||||
> content. R-82 re-verified only the rows it touched (the PBS whole-guest tier); it did **not**
|
||||
> re-verify the restic/tier-2 material, and says so rather than implying a freshness it did not earn.
|
||||
> Line numbers are landmarks — reconfirm before editing.
|
||||
>
|
||||
> Inputs: `SPIKE-backup-classification-2026-07-14.md` (`790ec84`), `CAMPAIGN-6C-2026-07-14.md`
|
||||
> (`deacee11`, F-6C-1), Viktor's locked rulings of 2026-07-14. This document DECIDES; it does not
|
||||
@@ -71,7 +80,7 @@ the Task 1 lesson). Resolution to absolute paths happens at capture time against
|
||||
| **Tier-2** cross-drive | unit + resolver-appdata (byte-identical to v0.131.0) | unit + mandatory binds | + optional binds | never | **real local drives only** (network paths excluded per F-6C-1 ruling — both auto and pinned); SSD fallback = unit + mandatory **iff** headroom fits, optional skipped with honest reason | `rsync -a --delete` per capture-set path | missing-only merge (`--ignore-existing`), per-path from the new layout (§8) |
|
||||
| **Tier-3** offsite restic | unit only (today's shape) | unit + mandatory binds — **not deselectable** | never | never | Hetzner Storage Box (SFTP) | one multi-path restic snapshot per app per run (§6) | staged scratch on a data drive → missing-only merge to live (§7) |
|
||||
| **Manual `.fab`** | v0.130.0 full-root capture | locked-in (not deselectable) | checkbox, **pre-selected** | opt-in, behind the two-number size warning + FileBrowser pointer | download / chosen drive | tar, **exclusion-scoped** root (SQ5 verdict; manifest v1 unchanged) | existing import (old controllers import new bundles correctly) |
|
||||
| **PBS** whole-guest | rootfs + docker volumes; bind-mounted drives out of reach | *unchanged* | | | PBS on DooPlex | vzdump | whole-guest restore |
|
||||
| **PBS** whole-guest (R-82) | rootfs + `/var/lib/docker` + `/mnt/sys_drive`; bind mounts (`/mnt/felhom-drives`, `/etc/felhom-bootstrap`) out of reach | *unchanged* | | | **`felhom-pbs` → datastore `felhom-offsite` on ep0 (Hetzner), per-customer namespace, reached over `wg-felhom`** — NOT "PBS on DooPlex" (that was the 2026-07 spike store) | vzdump, **WEEKLY** (`cadence_seconds: 604800`), retention **keep_last=2** (two weeks, operator ruling 2026-07-26) | whole-guest restore |
|
||||
|
||||
Row-level decisions folded in:
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,3 +1,25 @@
|
||||
## felhom-host-install 1.20.0 — R-82: a fresh box defaults to local-daily + offsite-weekly (2026-07-26)
|
||||
|
||||
The `backup` defaults gain `backup_targets: [{target_id: "felhom-pbs", cadence_seconds: 604800,
|
||||
keep_last: 2}]` — the offsite tier, weekly, two weeks retained (operator ruling 2026-07-26).
|
||||
|
||||
**`setdefault` semantics preserved, and PROVEN both ways** against the exact line:
|
||||
|
||||
```
|
||||
FRESH -> backup_targets: [{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]
|
||||
UPGRADE -> preserved verbatim, no backup_targets
|
||||
```
|
||||
|
||||
That second line is the load-bearing one: **an in-place upgrade can never silently start writing to
|
||||
an offsite datastore.** Existing boxes are migrated explicitly (R-82 Slice D.2), which is what keeps
|
||||
a routine host-install re-run from enrolling a box into offsite backups nobody asked for.
|
||||
|
||||
**Why the tier is written before its storage exists:** `felhom-pbs` only appears when the hub
|
||||
provisions the DR tier. The agent (v0.102.0) DEFERS a tier whose target storage is absent, so the
|
||||
tier stays silent until it is real and goes live with no restart the moment it is provisioned.
|
||||
Writing it here means a box is never left with a DR tier that is provisioned but unscheduled — the
|
||||
exact "applied and empty" state R-82 exists to end.
|
||||
|
||||
# Felhom scripts — Changelog
|
||||
|
||||
## felhom-host-install v1.19.0 — R-50 island control plane is the appliance default (2026-07-25)
|
||||
|
||||
@@ -184,7 +184,7 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_VERSION="1.19.0" # the SINGLE version source (F-1): -h, the run banners, and the hub
|
||||
SCRIPT_VERSION="1.20.0" # the SINGLE version source (F-1): -h, the run banners, and the hub
|
||||
# Setup-tab copy (hub internal/web/configs.go hostInstallVersion —
|
||||
# scripts/hostinstall_gates.py asserts the two stay equal) all follow it.
|
||||
# 1.16.0: the FELHOM_ESCROW sudoers alias (controller-driven escrow
|
||||
@@ -2288,7 +2288,21 @@ base['privileged'].setdefault('sudo_path','sudo')
|
||||
for _k,_v in {"unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agent/units","systemctl":"/usr/bin/systemctl","install":"/usr/bin/install","smartctl":"/usr/sbin/smartctl","lvs":"/usr/sbin/lvs"}.items():
|
||||
base['privileged'].setdefault(_k,_v)
|
||||
base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20})
|
||||
base.setdefault('backup', {"local_backup_target":"local","local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0})
|
||||
# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions —
|
||||
# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob
|
||||
# would guarantee that one of them is wrong. keep_last=2 on the offsite tier = two weeks (operator
|
||||
# ruling 2026-07-26).
|
||||
#
|
||||
# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only
|
||||
# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent
|
||||
# (localapi targetStoragePresent), so the tier stays silent until it is real and goes live with no
|
||||
# restart the moment it is provisioned. Writing it here means a box is never left with a DR tier that
|
||||
# is provisioned but unscheduled — the exact "applied and empty" state R-82 exists to end.
|
||||
#
|
||||
# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here —
|
||||
# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start
|
||||
# writing to an offsite datastore.
|
||||
base.setdefault('backup', {"local_backup_target":"local","local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]})
|
||||
base.setdefault('local_api', {})
|
||||
base['local_api'].setdefault('enable', True)
|
||||
base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']
|
||||
|
||||
Reference in New Issue
Block a user