Compare commits
25 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 admin
|
51871a7ea6 | ||
|
admin
|
33f0ab2f33 | ||
|
admin
|
2584dfb938 | ||
|
admin
|
185661062a | ||
|
admin
|
967ddf2f44 | ||
|
admin
|
d692c46db0 | ||
|
admin
|
b93ee06abc | ||
|
admin
|
e3187c86d5 | ||
|
admin
|
311dc06c13 | ||
|
admin
|
688470c945 | ||
|
admin
|
7a5694341d | ||
|
admin
|
81d4438630 | ||
|
admin
|
c9a3e48b21 | ||
|
admin
|
0779c5db85 | ||
|
admin
|
0f8f084817 | ||
|
admin
|
c102832892 | ||
|
admin
|
ff2655cf19 | ||
|
admin
|
046df303b6 | ||
|
admin
|
687fedd8ee | ||
|
admin
|
323f45a5ef | ||
|
admin
|
e34b614e5b | ||
|
admin
|
f21e7caed1 | ||
|
admin
|
dd40f85bb8 | ||
|
admin
|
7dc1744eec | ||
|
admin
|
a5cd480280 |
+169
@@ -17,6 +17,148 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
|
||||
(2026-08-04). Scope: those two boxes, not the fleet.**
|
||||
|
||||
Four SCHEDULED runs overnight, none triggered by hand: demo-felhom host **83.8 s** / offsite
|
||||
**540.4 s**; demo-hp host **109.3 s** / offsite **300.1 s**. Every one restored into a scratch guest,
|
||||
booted, verified and destroyed itself.
|
||||
|
||||
*What this closes.* Until yesterday every live restore-proof this project held was on the OFFSITE
|
||||
tier. The on-box tier — the one an ordinary recovery uses — had never been proven on either box, and
|
||||
not because it failed: the agent could not read the storage it lives on (R-185), so it never saw an
|
||||
archive there to test.
|
||||
|
||||
*What was observed for the first time.* Both boxes had BOTH tiers due simultaneously. Never-proven
|
||||
sorts first, so each took its host tier, deferred the offsite one, and picked that up on the next
|
||||
evaluation six hours later — R-86's ordering and the one-heavy-operation gate, working together,
|
||||
unsupervised. The host-tier proof then reached the hub through R-189's merge, which is that path
|
||||
carrying a host-tier entry for the first time.
|
||||
|
||||
*The asymmetry worth remembering:* a host-tier restore is **83–109 s**; an offsite one is
|
||||
**300–540 s**. The tier that matters for an ordinary recovery is also the cheapest to prove.
|
||||
|
||||
**S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission
|
||||
question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).**
|
||||
|
||||
*The defect.* On both demo boxes the agent's token had `FelhomAgentStore` on `local`, `local-lvm` and
|
||||
`felhom-pbs` and **not** on `felhom-backup` — the storage the same installer configured as
|
||||
`local_backup_target`. The content API answered `{"data":[]}` through the token while root listed
|
||||
three archives. `pickForThisRun` skipped the tier as *"no settled archive yet"*, which is exactly
|
||||
what a brand-new tier reports, so the host tier was never restore-testable and nothing said so.
|
||||
|
||||
*The rule.* The permission question has a definite answer where the listing does not. `Permissions`
|
||||
reads `/access/permissions?path=/storage/<target>` **as the agent's own token** — asking as root
|
||||
answers a different question and always says yes — and one `capability.Status` per configured tier
|
||||
reports it. The probed set comes from `BackupTiers()`, never a fixed list: a hardcoded probe list is
|
||||
the defect reproduced inside the fix.
|
||||
|
||||
*The measured trap, because the obvious reading is wrong.* An ungranted path answers **neither empty
|
||||
nor 403**: it carries the privileges inherited from the box-wide `/` grant
|
||||
(`Sys.Audit, SDN.Use, Datastore.Audit`). Testing for path-presence, or for `Datastore.Audit`, reports
|
||||
a blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**, and re-measuring is required
|
||||
before that constant is ever changed.
|
||||
|
||||
*Criticality, weighed once.* Critical, because the hub alerts only on critical and a non-critical
|
||||
entry would ride the report and alert nobody. **Except** the `local` fallback target, which
|
||||
host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it
|
||||
does not page — turning an ordinary documented setup into an alert is how a signal becomes something
|
||||
an operator archives unread. It never consults content, so it cannot alarm on a newborn tier by
|
||||
construction, and it never reports ok when it could not ask.
|
||||
|
||||
**S-22 — the installer's Scenario-F arm must finish the job, not just leave the definition alone
|
||||
(2026-08-03, R-185).** `configure_backup_target` has two arms. Case A creates the storage and grants
|
||||
in the same breath. The reuse arm — *"the target already exists"* — returned **without granting**, and
|
||||
that, not `PVE_STORAGES`, is where the drift came from: a box whose target pre-dated the install
|
||||
(the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token
|
||||
could not read. The reuse arm now ensures the ACL through the same guarded wrapper. **Scenario F is
|
||||
unviolated** — the storage DEFINITION is untouched, and granting the role the agent is supposed to
|
||||
have on the target this script is about to write into `agent.json` is finishing the job, not
|
||||
retargeting the box. `$BACKUP_TARGET_ID` stays OUT of `PVE_STORAGES`: that list is granted a step
|
||||
before the target is resolved, and `--acl-storages` entries are preflight-checked for existence.
|
||||
A gate asserts every arm that resolves the target also grants on it.
|
||||
|
||||
**S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is
|
||||
the design (2026-08-03, R-189; agent v0.122.0).**
|
||||
|
||||
*The rule.* Only successful restore-tests are written to `RestoreTestState`, and that state is what
|
||||
the host report carries after a restart. Failures live only in the in-memory `backup.Store`.
|
||||
|
||||
*Why, in one line each.* A **success suppresses future work** — under R-86's per-archive due-check a
|
||||
proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it
|
||||
believes, for a whole archive generation (a week on the offsite tier). A **failure causes future
|
||||
work** — a failing tier stays due and is retried at the next evaluation, so a lost failure heals
|
||||
itself within one interval, while a *persisted* failure would outlive the fault it describes.
|
||||
|
||||
*What the report does with the two.* The collector merges them: **one entry per tier, newest by
|
||||
`TestedAt` wins**. A fresh failure therefore beats a stored success (the failure is the news and
|
||||
exists nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier can
|
||||
never appear twice — the hub would read that as two tests.
|
||||
|
||||
*It refuses to lie.* A persisted record missing the archive **or** the tier produces **no entry**: an
|
||||
unproven tier reading as proven would be worse than the defect this closes. Run mechanics (scratch
|
||||
VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be.
|
||||
**Migration consequence, seen live:** a pre-R-189 record has no tier, so upgrading does not
|
||||
retroactively make an old proof visible to the hub; the tier's next real proof fills it in.
|
||||
|
||||
**S-20 — the release order is build → tag LOCALLY → publish → push tag, and every step protects
|
||||
something (2026-08-03, R-188 + R-186).**
|
||||
|
||||
The tag is created before the publish, so the build and the tag describe the same commit. It is
|
||||
**pushed** after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
|
||||
package made `check-published-versions.py` correctly fail a *correct* release — measured on roughly
|
||||
every second release, and R-168 mails those failures to the operator.
|
||||
|
||||
The invariant the old order protected is **asserted directly instead**: the gate now also refuses a
|
||||
**published version with no tag**, as a bounded probe (frontier + patch gaps) that prints its own
|
||||
coverage, because the package listing api is 401 without a token and absence cannot be enumerated.
|
||||
A half-done release is loud: publish-then-failed-push dies naming the recovery command, and a failed
|
||||
publish deletes the local-only tag so a retry is clean.
|
||||
|
||||
**A released binary is independently verifiable** — `-trimpath -buildvcs=false` means the same source
|
||||
yields the same bytes with or without the tag; the verification command lives in
|
||||
`felhom-agent/CLAUDE.md`. Both build paths (`release-agent.sh` and `publish-agent.sh`'s fallback) use
|
||||
identical flags: they differed by `CGO_ENABLED=0` and produced binaries 74 KB apart for one version.
|
||||
|
||||
**S-17 — restore-testing is PER ARCHIVE GENERATION, and the hub's staleness window follows each
|
||||
tier's own rhythm (2026-08-03, R-86; agent v0.121.0 + hub v0.91.0).**
|
||||
|
||||
*The rule.* Let **A** be the newest archive on a tier that has settled for at least the settle lag
|
||||
(24 h). The tier is **DUE** when A exists and **A has not already been proven**. The daemon-start
|
||||
ticker survives only as the **evaluation interval** (6 h). A daily tier is proved daily on yesterday's
|
||||
archive; a weekly tier weekly on its own; a tier with no archive is UNKNOWN, never a fault.
|
||||
|
||||
*The trap, written down so it is not reintroduced.* The literal reading of R-86 — *"due when the
|
||||
newest archive is ≥24 h old"* — is **never true on a daily tier**, because a new archive resets the
|
||||
newest-archive age to zero long before it reaches the lag. It would have switched restore-testing off
|
||||
for the tier that matters most, silently. Red-proved at 0 runs over 5 simulated days
|
||||
(`felhom-agent/internal/backup/restoretest_due_test.go`).
|
||||
|
||||
*What the state holds now.* `RestoreTestState` records **which archive** was proven, not just when a
|
||||
tier passed — a timestamp cannot answer *"have we proven THIS archive"*. A pre-R-86 file keeps its
|
||||
time (rotation ordering survives a deploy) and yields no proven archive, so each tier is due exactly
|
||||
once after the upgrade.
|
||||
|
||||
*The old config key.* `backup.restore_test_cadence_seconds` is DEPRECATED. **Negative still disables**
|
||||
verbatim; a positive value now seeds the **settle lag** only, and the daemon WARNs once at start-up
|
||||
naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
|
||||
(default 24 h). It is deliberately NOT carried into the evaluation interval.
|
||||
|
||||
*The hub half is not optional.* `restoreProvenStaleAfter` was a flat 7 days **derived from the cadence
|
||||
R-86 removes**, and a healthy weekly tier's proof age reaches EXACTLY 168 h just before its next
|
||||
proof — it sat ON the line. `restoreProvenWindow(tier, observed, ok)` now takes the tier's own
|
||||
observed archive interval × 4 generations, floored at 7 days, capped at 12 days (strictly inside the
|
||||
2-week offsite retention), falling back to the tier's **declared** rhythm (`backupStaleAfter` 26 h /
|
||||
`offsiteBackupStaleAfter` 8 d — the backup-freshness checker's own thresholds) when history is too
|
||||
short to observe one. Shipping Part 1 alone would have produced a nightly false alarm.
|
||||
|
||||
**S-18 — `ep0` is Tier 2, PROTECTED (operator ruling, 2026-08-03).** D-d named two protected machines
|
||||
and did not name ep0 either way; `runbooks/target-selection.md` carried the question in writing for
|
||||
two days. The ruling **extends D-d's protected list to three machines**: DooPlex, Peti's cluster,
|
||||
**ep0**. It is a classification, not a new set of prohibitions — destroying datastores, prune jobs,
|
||||
tunnel config or nftables rules was already forbidden by what it would destroy, and the ordinary
|
||||
off-site READ a restore-test performs remains permitted.
|
||||
|
||||
**S-13 — the `mp1` merge landed, and the variant was chosen on measurement (2026-08-03, R-165 / D-a).**
|
||||
The appliance's two data volumes are one. **Variant V-c**: the volume mounts at the NEUTRAL path
|
||||
`/var/lib/felhom`, and both `/var/lib/docker` and `/mnt/sys_drive` are binds of subdirectories of it.
|
||||
@@ -123,6 +265,33 @@ Two rulings, one shape: something became live because someone pushed, not becaus
|
||||
that bumps a version, before publishing — and a gate that fails on the normal path is one people
|
||||
learn to ignore.
|
||||
|
||||
**S-16 — a backup run NOTIFIES ONCE and RECORDS ALWAYS, and those are different things
|
||||
(2026-08-03, R-182; controller v0.194.0 + hub v0.90.0/.1).** Measured: nine per-app capture failures
|
||||
reached the hub, two were mailed, seven were dropped by a cooldown whose key carries no app
|
||||
identifier — *before* `LogNotification`, so they left no row anywhere.
|
||||
|
||||
- **The record:** `recovery_unit_capture_failed`, per app, unconditionally, now routed
|
||||
**record-only** by the hub (`recordOnlyEvents`) — stored and logged every time, never competing
|
||||
for an e-mail slot.
|
||||
- **The notification:** `backup_run_failures`, ONE per run, listing every failed app with its leg
|
||||
and reason plus the counts and free space. **A clean run emits nothing.**
|
||||
- **A suppressed operator event now leaves a `suppressed` row** carrying the key that suppressed it —
|
||||
for every operator type. *"We chose not to e-mail you"* and *"nothing happened"* must never look
|
||||
identical; that is the whole finding, stated as a rule.
|
||||
- **The periodic sweep gets a digest too, with NO `run_id`**, so it stays under the ordinary hourly
|
||||
cooldown. Without it the sweep's failures would be recorded and never notified — a new silence
|
||||
created while closing one. A real run's digest carries a unique `run_id` precisely so the cooldown
|
||||
can never collapse a manual run into the nightly one.
|
||||
- **Why the silence is safe:** the hub's deadline check raises `expected_backup_missed` from report
|
||||
freshness, independently of any mail the box sends (`monitor/deadline.go:396,417`). **If that check
|
||||
is ever weakened, this design loses its footing.**
|
||||
- **Not taken, and why:** putting `app` in the cooldown key. It fixes the swallowing by producing one
|
||||
mail per failing app — a dozen on a full disk.
|
||||
|
||||
**ep0 was rescaled by the operator to a CX33 (2026-08-03): 4 vCPU, 8 GB RAM, measured on the box, and
|
||||
the 4 GiB swapfile survived. The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only, so no disk
|
||||
figure in any runbook needed correcting. That closed **R-90** and unblocked **R-86**.
|
||||
|
||||
**S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).**
|
||||
Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the
|
||||
CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure
|
||||
|
||||
@@ -1,113 +0,0 @@
|
||||
# REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)
|
||||
|
||||
Follows `REPORT-campaign7/8/9.md`. Root `REPORT.md` is another session's (hub v0.85.0) and was not
|
||||
clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
|
||||
|
||||
**Full audit + evidence:** `documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`,
|
||||
`documentation/tests/campaign10-evidence-2026-07-31/`.
|
||||
|
||||
## The sentence that matters
|
||||
|
||||
**Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the
|
||||
"drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No
|
||||
resource leak over 13.5 h. Two findings: R-156 and R-157.**
|
||||
|
||||
## What was established
|
||||
|
||||
- **Venue** — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the
|
||||
`/mnt/nvme-1tb` **mount root** (a subdirectory would have emitted `storage_disconnected` for demo-hp
|
||||
all night — the exact signal I1/I2 discriminate).
|
||||
- **Baselines, all read fresh.** controller `main` 0.188.0, **golden 0.188.0 (not behind)**, agent
|
||||
0.119.0 published+vouched, hub 0.86.0, ISO **1.26.1** (`f3cc86d5…`, round-trip verified live).
|
||||
The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
|
||||
- **Isolation gate — both denials captured, each with a positive control.** The PBS control **failed
|
||||
first**: four clean-looking 403s were worthless because the token was denied on its own datastore
|
||||
too (PBS token privilege separation). Fixed, re-run, denials stand.
|
||||
- **A1** fresh install from the **published** ISO. 1.26.1 is a public release image — verified against
|
||||
its bytes that it has no auto-install path — so it was driven blind via screendump + `sendkey`
|
||||
through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which
|
||||
would otherwise have been mangled and locked the box out.
|
||||
- **A2** claimed for real; discriminator flipped `dashboard not yet claimed` → `authentication required`.
|
||||
- **A3** both drives enrolled through the **real** endpoint; `mentes` accepted as backup target via
|
||||
the offer flow, ending `degraded:false / target:felhom-backup` — the I5/I6 healthy baseline.
|
||||
Four apps healthy spanning both sides of D5's split (4 × `type: secret`, 1 × `type: password`).
|
||||
- **A4** discriminators seed and read back across all four apps; rallly's over the path
|
||||
`DATABASE_URL` actually names, not the trusted socket that produced D5's false pass.
|
||||
|
||||
## Phase B — the soak
|
||||
|
||||
Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped
|
||||
deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**).
|
||||
1 461 invariant checks in total.
|
||||
|
||||
- **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale,
|
||||
never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as
|
||||
skipped rather than silently green — it fired once.
|
||||
- **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place
|
||||
variants and 330 secret-class assertions covering both sides of D5's split.
|
||||
- **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced
|
||||
`backup_target_absent` and a clean recovery.
|
||||
- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via
|
||||
`bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names
|
||||
it. That is the case R-117's spike called "the worse half".
|
||||
- **RTO, both bands measured.** S: 66 MB → **42.0 s** / **41.4 s** across two passes (66 restores).
|
||||
M: 21.1 GB → **608 s** mean over 2 reps, both returning the correct discriminator. 327× the data
|
||||
cost 14.5× the time, giving **RTO ≈ 40 s + 26.9 s/GB** (backup ≈ 29 s + 17.4 s/GB). The fixed ~40 s
|
||||
dominates below ~1.5 GB — that is the S band, and why its numbers clustered so tightly.
|
||||
- **Capacity ceiling, and the more consequential result:** a DB-backed app's recovery unit is **1.90×**
|
||||
its data (volume tar + SQL dump). The default `/mnt/sys_drive` is **20 GB**, so on a default box the
|
||||
largest locally-backupable app is **≈ 10 GB** — **the M band does not fit at all** without a
|
||||
per-customer `SysDataGrowGB`. **RPO still not measured.**
|
||||
- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds
|
||||
flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots
|
||||
and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** —
|
||||
bounded by journald, but a lot of logging.
|
||||
- Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that
|
||||
was not first observed live.
|
||||
- **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system
|
||||
disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the
|
||||
soak would have produced green cycles that tested nothing cross-drive.
|
||||
- **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that
|
||||
never landed became a fake "stale restore", and a real one would have looked identical. Fixed and
|
||||
red-proofed before restarting.
|
||||
|
||||
## Findings
|
||||
|
||||
- **R-156 (new)** — **papra's data is neither persisted nor backed up, and it reports healthy.** The
|
||||
template mounts `papra_data:/app/data`; the app writes `/app/app-data/db/db.sqlite`. The volume is
|
||||
empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
|
||||
sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
|
||||
Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
|
||||
- **R-157 (new)** — **bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two
|
||||
mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while
|
||||
docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences,
|
||||
intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is
|
||||
exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp
|
||||
check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on
|
||||
every channel. A settle-condition fix closes A and leaves B open. Not fixed.
|
||||
- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
|
||||
(`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
|
||||
`hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
|
||||
therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by
|
||||
not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.
|
||||
|
||||
## What did not run
|
||||
|
||||
**12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place,
|
||||
kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive).
|
||||
Previously reported as 6 of 12 — that was run 1 only.
|
||||
|
||||
**Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not
|
||||
automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed.
|
||||
`kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom
|
||||
covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed.
|
||||
|
||||
**Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34–c39 clean on
|
||||
every invariant. Beyond 39 is untested, not proven clean.
|
||||
|
||||
## Teardown — OWED, nothing removed
|
||||
|
||||
Still intact: the rig is reusable for the atoms that did not run. VM 311, `c10-scratch`, PBS datastore
|
||||
`felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition:
|
||||
DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four
|
||||
orphaned scratch customers left by exactly this omission.
|
||||
@@ -1,43 +0,0 @@
|
||||
# REPORT — CAMPAIGN 7 (felhom.eu side: docs only)
|
||||
|
||||
> Written as `REPORT-campaign7.md`, **not** the shared `REPORT.md`, per the convention this run
|
||||
> added to `CLAUDE.md`: `REPORT.md` is overwritten, so a second concurrent session in this repo
|
||||
> would clobber it. This session's implementation work was in `app-catalog-felhom.eu`; here it only
|
||||
> touched documentation.
|
||||
|
||||
**Run:** 2026-07-18 evening → 2026-07-19 morning. **Class:** campaign (record-and-rank + a defined
|
||||
allowed-fix set). **Implementation repo:** `app-catalog-felhom.eu` (see its `REPORT.md`).
|
||||
|
||||
## What changed in this repo
|
||||
|
||||
| file | change |
|
||||
|---|---|
|
||||
| `documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` | **new** — method, uninstall-semantics map, trio detail, full 53-app matrix, ranked findings, coverage |
|
||||
| `documentation/backlog/ROADMAP.md` | **+3 items** — R-40 (multi-hop major upgrade path), R-41 (no standing catalog deployability check), R-42 (sidecar-major ruling) |
|
||||
| `CLAUDE.md` | REPORT.md parallel-session rule: the second session writes `REPORT-<topic>.md` |
|
||||
|
||||
No hub/agent/scripts/website code was touched (campaign scope: catalog + docs).
|
||||
|
||||
## Headline for this repo's readers
|
||||
|
||||
1. **Uninstall semantics map row PARTIAL → PROVEN** (campaign doc §2), with live evidence from all
|
||||
three trio apps: remove requires stop first; named docker volumes are **always destroyed**
|
||||
(including the app's database); HDD bind-mount data and `backups/primary/<app>` survive unless
|
||||
explicitly ticked; images are kept; `app.yaml` goes, the template stays; the per-app **offsite
|
||||
toggle survives** the uninstall while tier-2 config is cleared. The confirmation modal does warn
|
||||
about the volumes, so there is **no consent gap**.
|
||||
2. **A lying healthcheck takes an app OFF-LINE, it does not merely mislead.** Traefik will not route
|
||||
to an `unhealthy` container, so a probe that cannot execute → permanent unhealthy → **404 to the
|
||||
customer while the app serves 200 on its own port**. 7 of 53 apps were in that state.
|
||||
3. **The pre-flight gate's own signal is missing:** the 0.145.0 → 0.146.0 floor-lift emitted no
|
||||
`controller_updated` event, though the identical bootstrap path emitted one for 0.143.0 → 0.145.0
|
||||
two hours earlier (§0, finding F1). The box did converge — golden, floor and runtime all agreed —
|
||||
but the event trail under-reports version transitions.
|
||||
|
||||
## Open items owned outside this repo
|
||||
|
||||
- **plant-it / wanderer** — images do not resolve at all (neither the new tag nor the one the
|
||||
catalog already ships). Upstream research needed; recorded as findings, not deletions.
|
||||
- **gokapi** — pinned back to v1.9.6; v2 needs the seeded `config.json` regenerated. Security-
|
||||
relevant, should not sit on a superseded line indefinitely.
|
||||
- **glance** — never had a seeded `glance.yml`; proven pre-existing.
|
||||
@@ -1,62 +0,0 @@
|
||||
# REPORT — CAMPAIGN 8: the backup & restore subsystem (2026-07-27/28)
|
||||
|
||||
Adversarial, destructive, unattended run against `demo-felhom`, `demo-hp` and `ep0`.
|
||||
Full report: `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
Evidence: `DooPlex:~/campaign8/evidence/` (103 files, 35 MB, written continuously by 11 collectors).
|
||||
|
||||
**No production code was changed.** Findings are recorded and ranked, never fixed inline, per the
|
||||
campaign's own rules.
|
||||
|
||||
## Scope safety
|
||||
`peti-felhom`, its namespace and `u629488-sub2` were never touched. Phase 0 established with five
|
||||
documented probes that peti has **no data at all** in `felhom-offsite`, which is what made the
|
||||
operator-approved 100% datastore-fill safe. The 13 GB rollback copy `/srv/pbs-felhom` on ep0 is
|
||||
intact.
|
||||
|
||||
## Findings
|
||||
|
||||
| # | Finding | Severity | Class |
|
||||
|---|---|---|---|
|
||||
| F-CRIT-1 | An app that fails to restart after a quiesce **never alarms**, on any channel | **HIGH** | DEFECT |
|
||||
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT |
|
||||
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
|
||||
| F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT |
|
||||
| F-LEAK | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed | MEDIUM | DEFECT (root-caused by fault 18) |
|
||||
| F-REBOOT | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
|
||||
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
|
||||
| F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT |
|
||||
| F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational |
|
||||
|
||||
Both HIGH findings are in the same place: **the system's ability to tell you a backup did not
|
||||
happen.** Both cite the code and the comment that asserts the property the code does not provide.
|
||||
|
||||
## What is now proven that was not before
|
||||
R-88 breaker arming **and its full ladder** (15m/30m/1h/2h/4h/4h-cap) · per-tier isolation under a
|
||||
real one-tier-fails case · `whole_guest_backup_failed` end-to-end with correct tier attribution ·
|
||||
R-97c operator-only routing verified against the hub DB (zero customer rows with `status='sent'`) ·
|
||||
`whole_guest_backup_recovered` + the R-68 pairing gate firing live · **`age_state=absent`** ·
|
||||
R-97b's suppression half · the crash-recovery unquiesce by an actual SIGKILL (1 s) ·
|
||||
**R-87 — the first restic restore round-trip ever performed**, byte-verified (6/7 sha256 identical,
|
||||
the 7th explained) · R-82 one-quiesce-two-tiers · single-flight on two independent paths.
|
||||
|
||||
## Restore round-trips
|
||||
restic (R-87) · local vzdump → fresh CT · PBS offsite → fresh CT · corrupted snapshot → fails
|
||||
cleanly. `mount_parity` exact on both whole-guest tiers, `unprivileged: 1` preserved.
|
||||
|
||||
## Fleet state
|
||||
**Healthy. Nothing left broken.** All four compression knobs reverted and verified; every fault
|
||||
unwound; no leaked scratch guests, nft rules, ballast files or clock skew; ep0 datastore clean with
|
||||
zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
|
||||
|
||||
## Not tested (with reasons)
|
||||
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the
|
||||
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), fault 12 (host
|
||||
reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for
|
||||
documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings.
|
||||
|
||||
**Campaign-caused outage, stated plainly:** fault 11 took demo-hp guest 9201 down for ~9m47s
|
||||
(the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after.
|
||||
|
||||
## Note on repo conventions
|
||||
This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an
|
||||
entry — there is nothing shipped to log. The deliverable is the audit document plus this report.
|
||||
@@ -1,85 +0,0 @@
|
||||
# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28)
|
||||
|
||||
**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign,
|
||||
and findings are recorded, never fixed inline. Full write-up:
|
||||
`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`.
|
||||
Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously).
|
||||
|
||||
Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes.
|
||||
**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact.
|
||||
|
||||
## The headline — two never-proven restore paths are now proven
|
||||
|
||||
Driven through the **real endpoints the UI posts to**, over https through traefik with a real session
|
||||
and CSRF token, on live hardware.
|
||||
|
||||
| proof | result |
|
||||
|---|---|
|
||||
| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
|
||||
| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
|
||||
| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
|
||||
| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
|
||||
| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
|
||||
| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
|
||||
|
||||
The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
|
||||
at their last-backup state.**
|
||||
|
||||
## Findings — 3 defects, ranked (none fixed)
|
||||
|
||||
| # | finding | severity |
|
||||
|---|---|---|
|
||||
| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
|
||||
| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
|
||||
| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
|
||||
|
||||
Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
|
||||
sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
|
||||
artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
|
||||
so live.
|
||||
|
||||
## Mechanisms confirmed working, live
|
||||
|
||||
R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target
|
||||
breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app
|
||||
thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of
|
||||
the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on
|
||||
demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's
|
||||
sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the
|
||||
hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable;
|
||||
F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests).
|
||||
|
||||
## Where it stopped, and what remains
|
||||
|
||||
Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host
|
||||
reboot mid-backup) was deliberately not started — a large new fault class against boxes that are
|
||||
remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown.
|
||||
|
||||
**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's
|
||||
threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time
|
||||
`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign
|
||||
forbids. Reported rather than worked around. The no-code-change alternative (age the controller's
|
||||
reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended
|
||||
method next time.
|
||||
|
||||
**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an
|
||||
intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way
|
||||
concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`.
|
||||
|
||||
## Recovery
|
||||
|
||||
Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive
|
||||
observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active),
|
||||
windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on
|
||||
either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and
|
||||
the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots).
|
||||
|
||||
One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box.
|
||||
Its `until` is long past so it blocks nothing; it clears on the next successful backup or any
|
||||
controller restart (by design, not persisted). Clearing it would have cost another app outage for no
|
||||
benefit.
|
||||
|
||||
**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in
|
||||
`/opt/docker/stacks/<app>` starts a Felhom app **without its secrets** — they are injected by the
|
||||
controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a
|
||||
crash loop during recovery. Manual recovery must go through `POST /api/stacks/<name>/restart`.
|
||||
@@ -1,71 +0,0 @@
|
||||
# REPORT — DIAGNOSE immich offsite restore (felhom.eu side: docs only)
|
||||
|
||||
> Written as `REPORT-diag-immich-restore.md`, **not** the shared `REPORT.md`, per the CLAUDE.md
|
||||
> convention — `REPORT.md` is overwritten and currently holds the 2026-07-18 website refresh.
|
||||
> No code shipped in this run; findings only.
|
||||
|
||||
**Date:** 2026-07-19 · **Box:** demo-felhom (felhom-pve guest 9201), controller 0.146.0, immich v3.0.3
|
||||
|
||||
> **CLOSED IN CODE 2026-07-19 — controller v0.148.0.** Findings 1 and 2 shipped as R-43 (offsite
|
||||
> reconstitution: safety dump → stop → overwrite files → start → replay the snapshot's dump) and
|
||||
> R-44 (every run dumps before it captures; manifests carry `offsite_run_id` + `dumps_at`). Deployed
|
||||
> to demo 9201, healthy. **The §9 live acceptance has NOT run**, so no capability-map flip: the
|
||||
> offsite row is PARTIAL, the customer-restore row stays MISSING, R-3 stays DRAFT. Implementation
|
||||
> detail lives in `felhom-controller/REPORT.md`.
|
||||
|
||||
## What ran
|
||||
|
||||
A diagnosis of "immich offsite restore succeeds but photos do not reappear". No product code was
|
||||
changed: no restore/backup logic, no labels, no flashes, no `restic prune`/`forget`, no snapshot or
|
||||
escrow changes.
|
||||
|
||||
## Outcome
|
||||
|
||||
The restore did not fail. **It was never invoked on missing data, and could not have worked
|
||||
if it had been.**
|
||||
|
||||
1. Viktor deleted the 11 photos in the immich UI to test offsite restore. A UI delete means
|
||||
**trash**, not deletion — no file left the disk. Both „csak a hiányzó fájlok" runs merged
|
||||
**0 files**, correctly, and flashed success. The test proved nothing.
|
||||
2. A *valid* test would also have failed: **no offsite path loads a database.** All three buttons
|
||||
are file-only. Files would return; the library would stay empty.
|
||||
3. The shipped dump is additionally stale by design — from the 02:30 local run, never refreshed
|
||||
before a manual push. Probed: **`asset: 0`, `user: 0`, `album: 0`**.
|
||||
|
||||
**Photos:** left in trash at Viktor's instruction (recovery not wanted). All 11 files verified
|
||||
present on disk and all 11 rows intact, so an ordinary un-trash recovers them until immich's
|
||||
30-day auto-purge.
|
||||
|
||||
**Answer to "can a customer trust same-day offsite?"** For a DB-indexed app: **no — files come
|
||||
back, content does not.** The backup half is honest; the restore half cannot reconstitute the app.
|
||||
|
||||
## Decisive evidence
|
||||
|
||||
- `updatedAt` == `deletedAt` on all 11 asset rows ⇒ **no restore operation touched the DB.**
|
||||
- Unit dump `immich-postgres.sql`, 51 954 452 B, mtime **02:30 CEST** ⇒ `asset: 0 / user: 0 /
|
||||
album: 0`. The 52 MB is entirely immich's shipped geodata reference tables. It predates the admin
|
||||
user (07:56:25) and the photos (07:57).
|
||||
- **Latent hazard:** had a full restore loaded that dump, it would have written an empty DB over the
|
||||
live one, destroying the trashed rows that were the only surviving recovery path.
|
||||
|
||||
## Files written
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `documentation/audits/DIAG-immich-restore-2026-07-19.md` | new — timeline, evidence, source mechanism table, 7 findings |
|
||||
| `documentation/backlog/ROADMAP.md` | **R-43** (P2-HIGH, offsite restore cannot restore a DB) + **R-44** (P2-HIGH, manual push ships unrefreshed dump) |
|
||||
| `documentation/architecture/00-capability-map.md` | customer-restore row **stays MISSING**, gains the finding + a method note for R-3; offsite-restore row flagged *scope contested* |
|
||||
|
||||
## Needs Viktor
|
||||
|
||||
1. **Capability-map ruling (line 61).** The offsite-restore row claims PROVEN-LIVE citing
|
||||
"immich end-to-end from offsite alone" (`CAMPAIGN-6D`). That phrase is contested by this
|
||||
diagnosis. I flagged it rather than downgrading a proven row — did 6D's accept leg exercise the
|
||||
DB half, or only the file half?
|
||||
2. **Optional real red-proof.** Emptying the trash would create genuine data loss and convert the
|
||||
dump-probe inference into a live proof. Offered, **not run** — irreversible, and the probe
|
||||
already settles it.
|
||||
3. **Unreconciled: the 704.6 MiB figure.** Measured 126 MB total on the library storage. If
|
||||
704.6 MiB came off a controller Storage page, that gap is its own defect.
|
||||
4. **Orphaned pre-v3 tree** `dccc13fe…` (~55 MB across upload/thumbs/encoded-video) stranded by the
|
||||
immich 2→3 redeploy — worth a sweep policy for major redeploys.
|
||||
@@ -1,53 +0,0 @@
|
||||
# REPORT — Megosztás diagnosis (2026-07-20)
|
||||
|
||||
Topic-scoped report (parallel-session rule: shared `REPORT.md` untouched).
|
||||
|
||||
**Run:** RUNBOOK "Megosztás diagnosis — SMB unreachable from Mac + sharing-page reload loop".
|
||||
Read-only diagnosis. **No code changes, no version bumps, no builds, no restarts.**
|
||||
|
||||
**Deliverable:** `documentation/audits/DIAG-sharing-2026-07-20.md`.
|
||||
|
||||
## Verdicts
|
||||
|
||||
* **Reload loop — ROOT-CAUSED (HIGH).** `sharingStatusHandler` (`sharing_handlers.go:246`, added in
|
||||
`b5d78d1`, controller v0.147.0, 2026-07-19) coerces `idle` → `running` whenever the samba
|
||||
container is alive. `sharing.html` L320–326 treats `running` as a one-shot job-success edge and
|
||||
calls `location.reload()` 1.2 s later. The first `tick()` fires synchronously on every page load,
|
||||
so the page reloads forever. Unconditional for any customer with sharing enabled — the Megosztás
|
||||
page is currently unusable. Proven live: 6 consecutive `/sharing/status` polls all returned
|
||||
`{"phase":"running","running":true}`, and the controller log shows **no ensure job ran at all**,
|
||||
so the phase is manufactured by that line rather than left over from a stuck job.
|
||||
* **`smb://192.168.0.162` — ROOT-CAUSED.** `.162` is the Proxmox host and never was an SMB endpoint.
|
||||
smbd runs in guest 9201 and binds `192.168.0.104:445`. `nc` from the host: `.104:445` **open**,
|
||||
`.162:445` **refused**. Wrong target, stale Finder favourite.
|
||||
* **`smb://FELHOM` — OPEN, narrowed.** NetBIOS resolution works on the wire
|
||||
(`nmblookup -B 192.168.0.255 FELHOM` → `192.168.0.104 FELHOM<00>`), but the stack advertises **no
|
||||
mDNS/Bonjour** (nothing on udp/5353; the R-6 spike selected `smbd + nmbd + wsdd` only) — the
|
||||
mechanism macOS Finder prefers. Closing this needs one probe from the Mac (`smbutil lookup FELHOM`
|
||||
/ `dns-sd -B _smb._tcp`), listed in the audit's Mac test matrix.
|
||||
* **H2 (container down/crash-looping) — RULED OUT** at the first probe: `felhom-samba` Up 3 h, clean
|
||||
logs, smbd/nmbd/wsdd all bound as `infra/samba.go` intends, live `smb.conf` matches the renderer
|
||||
with no baked address literal, no PVE firewall in the path.
|
||||
|
||||
## Findings (full table + evidence in the audit doc)
|
||||
|
||||
S-1 HIGH reload loop (XS fix: latch a `sawInFlight` flag in the JS; red-proof required) ·
|
||||
S-2 MED the UI never shows the connect address, only `\\FELHOM` — customers guess IPs ·
|
||||
S-3 MED no mDNS advertisement (image slice, needs republish) ·
|
||||
S-4 LOW ensure-job phase never resets — fold into the async-job-feedback roadmap item ·
|
||||
S-5 INFO the guest's LAN IP is DHCP, so any displayed address must be read live, never cached.
|
||||
|
||||
Recommended packaging: S-1 as an immediate patch task (it bricks a shipped page), S-2 alongside it
|
||||
if the guest IP is reachable from the sharing handler, S-3 as its own slice.
|
||||
|
||||
## Also noticed
|
||||
|
||||
The **remote site's LAN is `192.168.0.0/24` — the same prefix as the DooPlex home LAN** that the
|
||||
Tailscale subnet router advertises (`192.168.0.180` shows `FAILED` in felhom-pve's neighbour table).
|
||||
A successful `ping 192.168.0.162` therefore does not by itself prove the Mac is on the remote
|
||||
segment; the Mac matrix starts by confirming which network it is on.
|
||||
|
||||
## Actions taken
|
||||
|
||||
None. Every command was a read, except a `POST /login` to obtain a session for the status polls.
|
||||
No secrets are recorded in either document.
|
||||
@@ -1,85 +0,0 @@
|
||||
# REPORT — R-111 fixed, then E-2 proven on a fresh box (2026-07-29)
|
||||
|
||||
Two phases in one session. Full evidence: `documentation/audits/E2D-fresh-vm-2026-07-29.md`.
|
||||
Root `REPORT.md` untouched.
|
||||
|
||||
## Phase 1 — R-111: the Day-0 channel now serves the current software
|
||||
|
||||
A Phase 0 gate earlier the same day stopped the E-2d run before any VM existed: a fresh box would
|
||||
have installed **agent 0.96.0 + controller 0.161.0**, ~17 and ~24 releases behind `main`.
|
||||
|
||||
| | Before | Now |
|
||||
|---|---|---|
|
||||
| agent (Gitea generic) | 0.96.0 | **0.113.0**, sha `5f3247f7…`, round-trip verified |
|
||||
| golden (Gitea generic) | 0.161.0 | **0.185.1**, sha `dba00f3e…`, embeds controller 0.185.1 |
|
||||
| hub `min_agent` | 0.93.0 | **0.113.0** (what controller v0.185.0 declares) |
|
||||
|
||||
Bake clean on every marker: `Result=success`, overlay2, **all three mounts in the archive**, 0
|
||||
FATAL/exclusions, HTTP 201, token-leak grep 0. GL-1 teardown: guest 9100 purged, secrets shredded,
|
||||
drill disk restored to `virgin`. Agent + golden moved in **one** manifest POST so it never vouched a
|
||||
new agent against an old golden. `min_agent` verified zero-impact first (all three enrolled hosts
|
||||
already at 0.113.0). Global floor deliberately **not** raised — the golden now bakes 0.185.1.
|
||||
|
||||
Commit `3dff357`.
|
||||
|
||||
## Phase 2 — the E-2d run, full ISO/PAIRING route
|
||||
|
||||
Nested PVE VM on demo-hp, one disk, outside the `felhom` pool. Bind → running controller in
|
||||
**3 m 35 s**. The install fetched exactly the artifacts published an hour earlier and restored
|
||||
`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked 20 minutes before. The publish train is
|
||||
proven end to end on a real install.
|
||||
|
||||
| Claim | Verdict |
|
||||
|---|---|
|
||||
| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** |
|
||||
| **C2** Case B fires naturally | ✅ **PROVEN** — both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort |
|
||||
| **C3** degraded banner renders **to a customer** | ⚠️ **PARTIAL** — API byte-exact; **no UI consumer exists** → **R-112** |
|
||||
| **C4** offer appears and moves the target | ⚠️ **PARTIAL** — decline path, `restart_required:true`, no self-restart, E-2a wrapper, healthy-renders-nothing all PROVEN at API level; offer equally invisible → **R-112** |
|
||||
| **C5** `backup_target_absent` end to end | ❌ **FAILED** — zero events on any channel → **R-113** |
|
||||
|
||||
## The three findings
|
||||
|
||||
**R-112 (P1)** — E-2's banner and offer have **no UI consumer**. The endpoint returns byte-exact copy;
|
||||
`grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**, and no page handler injects
|
||||
the state. Decisive contrast: templates fetch **18** distinct `/api/storage/*` endpoints;
|
||||
`backup-target` and `backup-target/assign` are the only two with zero references. v0.185.1 fixed the
|
||||
router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired.
|
||||
|
||||
**R-113 (P1)** — the drive-absent gate **cannot fire on device loss**. `planDriveGates` reads presence
|
||||
from `BoundUnderParent` = "is this path in the guest's mountinfo". The raw mount is a device-bound
|
||||
systemd unit and dies; **the agent's own bind is not device-bound and outlives the device**, so the
|
||||
gate sees "present" forever. Live: agent said `enrolled drive absent by UUID` every 20 s for 4½
|
||||
minutes, controller logged **0** `[gate]` lines, hub got **zero** events — neither the specific nor the
|
||||
generic one. Sixth instance of the class, one layer deeper: E-2b wired the seam to a condition that
|
||||
cannot occur.
|
||||
|
||||
**R-114** — on target-drive loss the message says the backup is *"on the same disk as the system"*
|
||||
(false) and offers **the drive that just vanished**. Invisible today only because of R-112 — so
|
||||
**R-114 must be fixed before R-112 is wired.**
|
||||
|
||||
Also filed as a **second instance under R-110** (not a new ID): host-install fetches **nine** files
|
||||
from `raw/branch/main` and the hub vouches a sha for **one**; E-2a's wrapper is installed 0755 to
|
||||
`/usr/local/sbin`, root-fenced in sudoers, validated only by `bash -n`.
|
||||
|
||||
## Record
|
||||
|
||||
- `OPEN-ITEMS.md` — **R-112/R-113/R-114 opened**; E-2d re-stated with results and left open for the
|
||||
residue; E-2's "NOT yet live-proven" list resolved into proven / known-broken; R-94 fully unblocked;
|
||||
R-110 extended. The drill-cleanup row was opened and then **closed the same session** once the
|
||||
teardown completed, so it is not carried in the register.
|
||||
- `ROADMAP.md` — R-112/R-113/R-114 under P1; R-111 marked SHIPPED.
|
||||
- **`architecture/00-capability-map.md` not touched** — for two reasons: the customer-facing legs are
|
||||
broken rather than proven, and the map has **no E-2 / backup-target rows at all** (worth noting
|
||||
against the ROADMAP's coupling rule).
|
||||
|
||||
## Teardown
|
||||
|
||||
VM destroyed, scratch storage removed, **`pvesm status` after == before** (`local-lvm` 38.77 %,
|
||||
byte-identical), guest 9201 and drill-r50 untouched. **Hub records removed — teardown complete.** The delete was correctly refused at four gates while the host still read ONLINE; once the destroyed host aged to DOWN (`delete-impact` → `deletable:true`) the documented cascade ran and completed: host deleted, PBS tenancy deprovisioned, claim reset, residue purged. Verified after: **0** `e2d` occurrences on the hosts page, fleet unchanged. The one purged `appliance_registrations=1` was this run's own appliance; the unrelated stale 2026-07-25 appliance (`206c8838…`) was not touched by the cascade — the operator removed it separately.
|
||||
|
||||
## One human step, and a premise correction
|
||||
|
||||
The runbook's §5.1a operator STOP (the bind) is **retired** — CC did it. But E-2d's premise that a
|
||||
fresh install yields a CC-drivable claimable customer is **wrong**: the claim code is bcrypt-hashed and
|
||||
email-only, and the gate covers everything except `/claim`, `/api/health`, `/static/`. One operator
|
||||
relay of the emailed code was required — which also proved the claim flow end to end.
|
||||
@@ -1,92 +0,0 @@
|
||||
# REPORT — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run (RUNBOOK execution). **No code changed. No version bump.**
|
||||
Written as `REPORT-<topic>.md` per the parallel-session rule — the shared `REPORT.md` was not touched.
|
||||
|
||||
**Full record with all evidence:** `documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`
|
||||
|
||||
---
|
||||
|
||||
## Outcome: DONE and verified
|
||||
|
||||
`felhom-offsite` now lives on a dedicated 100 GB Hetzner Cloud Volume instead of ep0's 40 GB root disk.
|
||||
|
||||
| | Before | After |
|
||||
|---|---|---|
|
||||
| Path | `/srv/pbs-felhom` (root disk) | **`/mnt/pbs-datastore`** (volume) |
|
||||
| Datastore total | 37.2 GB | **98 GB** (hub gauge: 97.9 GB) |
|
||||
| Used | 28.9 % | **13 %** (hub gauge: 12.6 GB, 13 %) |
|
||||
| Headroom to the 80 % warn | 19 GB | **≈65 GB** |
|
||||
| Additional customers before warn | ≈2 | **≈7–13** |
|
||||
|
||||
Datastore **name unchanged** — the PBS-DR descriptors, per-box storage ids, ACLs and namespace
|
||||
layout that R-39/R-82 made self-healing are untouched.
|
||||
|
||||
**Window:** 06:58 → 07:19 UTC (PBS down 07:00 → 07:17). **Nothing was deleted.**
|
||||
|
||||
### Acceptance evidence
|
||||
|
||||
| Gate | Result |
|
||||
|---|---|
|
||||
| Copy integrity | 13,242,207,822 = 13,242,207,822 B · **9,748 = 9,748 chunks** · 7 = 7 snapshots · `backup:backup` · itemised dry-run **0 lines** |
|
||||
| Snapshot counts per ns | `demo-felhom` 2=2, `demo-felhom-01` 3=3, `demo-hp` 2=2 |
|
||||
| atime semantics | `rw,relatime,discard` — **`relatime` present, `noatime` absent** (GC correctness) |
|
||||
| Verify job | `TASK OK`, 3/3 groups, forced re-verification of every snapshot, **0 errors** |
|
||||
| §6 mount guard | **refusal observed** — `Job … failed with result 'dependency'`; mountpoint stayed empty |
|
||||
| §8 restore round-trip | `source_tier: pbs`, `pass: true`, `mount_parity: ok`, clean teardown, 12m1s |
|
||||
|
||||
---
|
||||
|
||||
## Three findings the operator should act on
|
||||
|
||||
1. **`scratch` datastore is configured at a path that does not exist** (`/srv/pbs-scratch`).
|
||||
Pre-existing, not caused here, but now logs `ENOENT` on every PBS start. This is the PRIME RISK
|
||||
shape ("reports fine, is not there") already live in the config. **Decision needed:** remove the
|
||||
stanza or create the directory.
|
||||
|
||||
2. **The runbook's §6 acceptance test proves the wrong proposition.** `RequiresMountsFor` is a
|
||||
mount-first ordering guarantee, not a refusal — systemd silently *re-mounts* an unmounted volume
|
||||
and PBS then starts safely. The test only bites when the device is genuinely unavailable, which
|
||||
is how it was re-run and passed. **Amendment recommended in the runbook record.**
|
||||
|
||||
3. **§11 — storage box `u629193` is NOT simply unused.** No live backup path references it (no
|
||||
datastore, no restic repo, no fstab, no `known_hosts` pin; R-17 already deleted `u629193-sub1`),
|
||||
**but ep0 carries an enabled, currently-mounted sshfs unit** `mnt-pbs\x2dstoragebox.mount` →
|
||||
`/mnt/pbs-storagebox`, holding spike leftovers. Disable and remove that unit before deleting the
|
||||
box, or ep0 logs a failed mount every boot. **The deletion is the operator's console click.**
|
||||
|
||||
## Deviations from the runbook as written
|
||||
|
||||
- **The volume arrived already formatted and mounted** by Hetzner at `/mnt/HC_Volume_106469259`
|
||||
(§2 assumed neither). Operator ruled: reformat + repath. The 5 % reserve was reclaimed (`-m 0`).
|
||||
- **§8 ran on demo-felhom, not demo-hp** — DooPlex holds no SSH key for demo-hp (the G1 gap). Same
|
||||
tier, same relocated datastore, larger archive.
|
||||
- **The window was contended** by a stale 10-minute restore-test cadence on demo-felhom: the config
|
||||
had already been reverted to 3.5 days on disk, but the cadence is read once at daemon start and
|
||||
`NRestarts=0`. Restarting the agent applied it (`cadence=84h0m0s`). The in-flight test was allowed
|
||||
to finish rather than aborted.
|
||||
|
||||
## Process errors made during this run (recorded deliberately)
|
||||
|
||||
- `rsync -aHAX` **OOM-killed** ep0 (3.7 GB RAM, no swap). Cause: a PBS `.chunks/` tree pre-creates
|
||||
all 65536 shard dirs → 75,341 inodes, and `-H` retains the whole inode map. `-H` was dropped only
|
||||
after **proving** no hardlinks exist (`-links +1` → 0; max link count → 1); PBS references chunks
|
||||
by digest, never by hardlink.
|
||||
- `/usr/bin/time -v` is not installed on ep0 → exit 127, rsync never ran, and a `| grep … || true`
|
||||
wrapper swallowed it while printing a success-looking line.
|
||||
- `rsync --version | head -1` reported a working rsync 3.4.1 as "missing" — **the §12 pipe-into-head
|
||||
trap, fourth recorded instance in this project.** Both fixed by capturing the command's own `$?`.
|
||||
|
||||
## Deferred
|
||||
|
||||
1. **Old copy retained** at `/srv/pbs-felhom` (13 GB, 9,748 chunks) as the rollback. Rollback is a
|
||||
two-line `datastore.cfg` revert. Reclaim only after a new weekly offsite backup lands on the
|
||||
volume, with explicit go-ahead.
|
||||
2. **GC not run** — now unblocked by the round-trip, but left for a separate deliberate run. No GC
|
||||
schedule is configured on this PBS at all.
|
||||
3. ~~Hub PBS-DR capacity gauge not re-read.~~ **CLOSED — verified correct.** The hub operator UI
|
||||
(Offsite → PBS DR) reports `felhom-offsite (ep0)` at **97.9 GB capacity, 12.6 GB used, 13 % full**,
|
||||
agreeing with the on-box `df`. The gauge follows the datastore's configured path, so the move
|
||||
needed no hub-side change and the suspected "wrong filesystem" bug does not exist.
|
||||
4. **ep0 has no swap** (temporary 4 GB file removed; box left as found). Worth a small permanent
|
||||
swapfile — outside this runbook's scope.
|
||||
@@ -1,63 +0,0 @@
|
||||
# REPORT — F-CRIT-1 + F-A1 fixed (controller v0.179.0, 2026-07-28)
|
||||
|
||||
Docs here. Implementation, all six red-proofs and the full live replay live in
|
||||
`felhom-controller/REPORT.md`. The campaign that found both:
|
||||
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
|
||||
## What changed
|
||||
**F-CRIT-1** — an app that failed to restart after a quiesce never alarmed, for two independent
|
||||
reasons, either of which alone kept it dead: `restartAll` returned nothing (the failure was logged
|
||||
and dropped), and `classifyRunStates` whitelisted `StateStopped` on invariant I1 ("the user stopped
|
||||
it") — which the quiesce loop had made false by stopping stacks the same `docker compose down` way.
|
||||
A failed restart and a user stop are the *same* Docker state; the only difference is that the loop
|
||||
tried and could not, now surfaced by `Loop.FailedRestarts()`.
|
||||
|
||||
**F-A1** — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a
|
||||
failure. It is now contention: the tier stays DUE, is dropped before anything stops, and unending
|
||||
contention raises its own **BLOCKED** signal rather than going silent.
|
||||
|
||||
## Bounds, justified against measured reality
|
||||
- `contentionRetryAfter` **15m** — longest restore-test observed on the fleet is 12m01s; the agent's
|
||||
local restore-test wait is 10m. Caps app-stop churn at 4/hour instead of 12/hour.
|
||||
- `contentionAlarmAfter` **3h** — the agent's own PBS restore-test task is capped at 120 minutes, so
|
||||
contention outliving that is a *stuck* gate, not a busy one. 3h adds margin and is 15× the longest
|
||||
contention actually observed.
|
||||
|
||||
## Verified live, with the hub DB as arbiter — not from logs
|
||||
Same box, same day, same event type; the only difference is 409 versus a genuine error:
|
||||
|
||||
| injection | operator emails (demo-hp) |
|
||||
|---|---|
|
||||
| **409 contention** | 8 → **8** (none) |
|
||||
| **real transport failure** | 8 → **9** |
|
||||
|
||||
And for F-CRIT-1: the failed restart alarmed **9 seconds** after grace expiry with the dashboard
|
||||
banner naming the `(stopped)` state, while a **deliberate** user stop on the same box stayed silent
|
||||
through **9** dead-app scans (the positive observable that the silence is suppression, not a dead
|
||||
detector).
|
||||
|
||||
## The rule this arc earned
|
||||
Added to **both** copies of `CLAUDE.md` (live + `documentation/runbooks/workspace-CLAUDE.md`):
|
||||
**a comment asserting an invariant needs a test pinning it, or it is a wish.** Six instances in this
|
||||
project have shipped guarantees the code did not provide — `EffectiveProtected`, `newestArchiveOn`,
|
||||
the R-97a operator-only claim, `classifyRunStates`' I1, `inflight.go`'s defer claim, and
|
||||
`quiesce.go`'s spurious-failure claim. Two were found only on live hardware, and one of those had a
|
||||
green, red-proofed test suite over a production path broken two independent ways.
|
||||
|
||||
Corollary recorded with it: prefer a test that asserts the **consequence** (does the alarm fire?)
|
||||
over one that asserts the **mechanism** (does suppression expire?). R-97b's Scenario F proved the
|
||||
mechanism; the consequence was still broken.
|
||||
|
||||
## Docs touched
|
||||
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-1 and F-A1 → SHIPPED + PROVEN-LIVE.
|
||||
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — both rows struck through, closing
|
||||
section added. **All three of the campaign's alarm findings are now closed** (F-CRIT-1, F-CRIT-2,
|
||||
F-A1).
|
||||
- `documentation/runbooks/workspace-CLAUDE.md` — the invariant rule.
|
||||
|
||||
## Still open, highest first
|
||||
**Fault 4** (restic transport interruption) — four injection approaches were defeated by
|
||||
guest-bridged networking, and it is now the most valuable follow-up: F-CRIT-2 answered the phantom
|
||||
question for PBS and left the identical question open for restic. Then **R-99** (prune never removes
|
||||
phantoms) and **F-LEAK** (a failed restore-test cannot destroy its own scratch guest — observed
|
||||
again during this work).
|
||||
@@ -1,40 +0,0 @@
|
||||
# REPORT — F-CRIT-2 fixed: a failed backup no longer looks like a fresh one (2026-07-28)
|
||||
|
||||
Scope: `felhom-agent` v0.105.0 → **v0.106.0**. Docs here. Implementation detail and the full live
|
||||
re-test live in `felhom-agent/REPORT.md`; the campaign that found it is
|
||||
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
|
||||
## What changed
|
||||
`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, and NEWEST) as a
|
||||
successful backup, so the tier read fresh, went **not due**, and was never retried — seven days of
|
||||
silence on the real 168h cadence, invisible to both the R-88 breaker (defers only *due* tiers) and
|
||||
the hub deadline monitor (reads the same freshness). It now counts only *plausibly complete*
|
||||
entries via a measured 1 MiB floor; undecidable ⇒ not counted.
|
||||
|
||||
**Size is the only tier-agnostic discriminator.** `verification` and `encrypted` are absent on every
|
||||
local (dir) archive AND on a good PBS snapshot until `verify-new` catches up — gating on either
|
||||
would have rejected 100% of local backups and produced fleet-wide backup thrash. That inverse risk
|
||||
is a first-class test, red-proofed by making the filter reject everything.
|
||||
|
||||
## Verified live, not just in unit tests
|
||||
Campaign fault 2 was replayed against the fixed agent on demo-hp — phantom created, rejected and
|
||||
announced once; the tier correctly reported DUE and backed up (4,359,968,099 B landed); and the
|
||||
inverse showed **no thrash**, with 91 scheduler ticks as the positive observable that the loop was
|
||||
alive rather than dead.
|
||||
|
||||
## Settled along the way — no retention bug
|
||||
Server-side prune does **not** count phantoms toward `keep-last`: a dry-run against three real
|
||||
snapshots plus a phantom retained two real ones plus the phantom. The feared "two phantoms ⇒ zero
|
||||
real backups" does not occur. Prune never removes them either, so they accumulate one per aborted
|
||||
upload — filed as **R-99** (LOW, hygiene), not as a retention bug.
|
||||
|
||||
## Docs touched
|
||||
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-2 → SHIPPED+PROVEN-LIVE; **R-99** filed;
|
||||
**F-CRIT-1** filed as READY-HIGHEST (Campaign 8's other HIGH finding, untouched here).
|
||||
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — F-CRIT-2 row struck through and
|
||||
a closing section added.
|
||||
|
||||
## Still open, highest first
|
||||
**F-CRIT-1** — an app that fails to restart after a quiesce never alarms, for two independent
|
||||
reasons. Then fault 4 (restic transport interruption), which this fix makes more pointed: the
|
||||
phantom question is now answered for PBS and still open for restic.
|
||||
@@ -1,140 +0,0 @@
|
||||
# REPORT — F-REBOOT + F-LEAK + F-OBS, and two investigations (2026-07-28)
|
||||
|
||||
Scope in this repo: **`scripts/felhom-host-install.sh` v1.20.0 → v1.21.0** (which is where F-LEAK's
|
||||
*actual* fix lives), plus the Campaign 8 audit doc and `OPEN-ITEMS.md`. Written as
|
||||
`REPORT-freboot-fleak-fobs.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
Code companions: `felhom-agent` v0.106.0 → **v0.110.0**, `felhom-controller` v0.179.0 → **v0.180.0**.
|
||||
|
||||
**Correction to this repo's part of the story:** v1.21.0's band-scoped ACL is *not* the final F-LEAK
|
||||
fix. It works, but only **once per slot** — PVE's destroy path calls
|
||||
`AccessControl::remove_vm_access($vmid)` (`API2/LXC.pm:906`), which deletes every ACL at `/vms/<vmid>`
|
||||
(`AccessControl.pm:1898`), so **the grant is consumed by the operation it authorises**. Found by counting
|
||||
ACL rows after the first successful teardown (`/vms/990000` → 0 grants), not by reasoning about it. The
|
||||
durable fix is agent **v0.110.0**'s band-scoped fenced destroy; v1.21.0 remains valuable because it makes
|
||||
the common case need no privileged call, and it is now the *first* of two layers rather than the only one.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom.eu d0cec9d`, `felhom-agent af1c21a`, `felhom-controller fb91c8d`, all clean. Agent `0.106.0`
|
||||
and controller `0.179.0` live on both demo boxes.
|
||||
|
||||
---
|
||||
|
||||
## host-install v1.21.0 — F-LEAK, and why the fix landed *here* rather than in the agent
|
||||
|
||||
**The finding.** A restore-test whose restore **fails** leaves a scratch guest the agent cannot destroy
|
||||
(`403 missing privilege VM.Allocate`), so a half-restored guest holds its disks until a human removes
|
||||
it and the 10-slot scratch band shrinks silently.
|
||||
|
||||
**The cause is structural, not a missing privilege in the role.** `FelhomAgentGuest` is granted at
|
||||
`/pool/felhom`, and **a guest joins that pool only when its restore completes**. A failed restore
|
||||
therefore produces a guest that exists, is in no pool, and is out of the token's reach entirely.
|
||||
|
||||
**The first fix was wrong, and its own live replay is what proved it.** Agent v0.107.0 shipped a
|
||||
teardown fallback that adopted the stranded guest into the pool and retried — reasoning from
|
||||
`Pool.Allocate` on `/pool/felhom`. It fired exactly as designed and PVE refused it:
|
||||
|
||||
```
|
||||
ERROR restore-test: pool adoption failed; left for Recover vmid=990000
|
||||
err="proxmox: PUT /pools/felhom -> HTTP 500: permission denied at /vms/990000 (missing privilege ...)"
|
||||
```
|
||||
|
||||
`PUT /pools/{pool}` **also** requires `VM.Allocate` on the VM being added. **Pool membership cannot
|
||||
bootstrap its own authority.** Removed in agent v0.108.0 rather than left in place — a path that
|
||||
provably cannot work is worse than none, because it reads as a fix.
|
||||
|
||||
**What shipped instead.** `apply_scoped_acl` now grants `FelhomAgentGuest` at each `/vms/<id>` in
|
||||
`PVE_SCRATCH_VMID_MIN..PVE_SCRATCH_VMID_MAX` (990000–990009 — the band the restore-test already picks
|
||||
from), to **both** the user and the token, because the privsep-intersection rule applies here as
|
||||
everywhere.
|
||||
|
||||
Two supporting changes, both load-bearing rather than tidy-up:
|
||||
- **`remove_scoped_acl` deletes the band grants before the role delete.** PVE refuses to delete a role
|
||||
still referenced by any ACL, so omitting this would have broken the uninstall — a failure that would
|
||||
only surface on a decommission.
|
||||
- **`step_verify` asserts the band grants.** A missing one is otherwise invisible until a restore-test
|
||||
*fails*, which is precisely the case that leaked a guest in the first place.
|
||||
|
||||
### Why the grant is still not a widening — proven live, at the seam the defect lives in
|
||||
|
||||
A real PBS restore to `990000` **without `--pool`** reproduced the exact stranded state
|
||||
(`990000 stopped`; `felhom pool members: [9201]`; `990000 in pool: False`). Then, with the agent's own
|
||||
token, same guest, minutes apart:
|
||||
|
||||
| | `DELETE /nodes/<node>/lxc/990000` |
|
||||
|---|---|
|
||||
| **grant removed** (the original defect) | `403 Permission check failed (/vms/990000, VM.Allocate)` — guest still present |
|
||||
| **grant restored** (the fix) | `200 UPID:...:vzdestroy:990000:felhom-agent@pve!agent` — guest gone |
|
||||
|
||||
And it still cannot reach anything else:
|
||||
|
||||
| target | result |
|
||||
|---|---|
|
||||
| `/vms/990010` (one past the band) | **403** `Permission check failed (/vms/990010, VM.Allocate)` |
|
||||
| `/vms/100` | **403** same |
|
||||
|
||||
`990010` does not exist and PVE **still** answered 403 rather than "does not exist" — so PVE evaluates
|
||||
**permission before existence**, which makes these genuine authorization refusals rather than artifacts
|
||||
of a missing guest. Granting at `/vms` was considered and rejected: it would authorise destroying every
|
||||
guest on the box, including a co-tenant's.
|
||||
|
||||
Applied on **both** demo boxes (demo-hp and demo-felhom) so the live fleet matches the installer.
|
||||
|
||||
**A careless step of mine, recorded rather than buried.** The probe loop also issued a live `DELETE`
|
||||
against running guest **9201**. It was refused with `500 container is running` — but the *permission
|
||||
check passed* (9201 is a pool member by design), so had the guest been stopped I would have destroyed
|
||||
the live demo guest. The scratch-band probes were the safe ones; 9201 had no business in that list.
|
||||
|
||||
---
|
||||
|
||||
## Documentation changes
|
||||
|
||||
- **`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`** — F-REBOOT, F-LEAK and F-OBS
|
||||
written up as FIXED with their live evidence, including F-LEAK's refuted first attempt (recorded
|
||||
precisely *because* it looked right), plus a new **§6b** for the follow-up investigation.
|
||||
- **`documentation/backlog/OPEN-ITEMS.md`** — three findings closed, **R-100** filed.
|
||||
|
||||
---
|
||||
|
||||
## R-100 — the investigation's finding, deliberately NOT fixed
|
||||
|
||||
**A restic offsite tier that fails every night never goes stale on the hub.** This is **F-CRIT-2's
|
||||
defect class one layer up and on the other tier** — a *failed* run resetting the freshness clock — and
|
||||
it was found by asking whether F-CRIT-2's shape existed anywhere else.
|
||||
|
||||
Both halves verified in the source, not inferred:
|
||||
- **Controller:** `o.LastRun = time.Now()` is set **unconditionally** at
|
||||
`controller/internal/backup/offbox.go:716`, *outside* the `runErr` branch. The failure is recorded
|
||||
faithfully, but into a different field — `o.LastStatus = "error"` at `:725`.
|
||||
- **Hub:** `isStale()` reads **only** `off.LastRun` (`hub/internal/monitor/offsite.go:120`, `:127`,
|
||||
`:131`) and never consults `LastStatus`.
|
||||
|
||||
So a nightly restic run that fails every night keeps `LastRun` fresh, `isStale` is permanently false,
|
||||
and the staleness alarm never fires — with no successful offsite backup having occurred at all.
|
||||
|
||||
**Scope of the silence, stated precisely rather than dramatically.** `LastStatus` *does* reach the hub —
|
||||
it is parsed into the report struct and **only logged** (`offsite.go:270`); it drives no checker and no
|
||||
notification. The controller's own guest UI surfaces `LastStatus="error"`, so the failure is visible to
|
||||
someone who looks. What is missing is the **push**: the operator's fleet-wide alarm plane is silent,
|
||||
which is the plane that matters for an unattended appliance.
|
||||
|
||||
Not fixed, per this task's investigation-only scope. Fix direction: gate staleness on the last
|
||||
*successful* run rather than the last attempt — exactly what F-CRIT-2's `NewestArchiveTime` fix did for
|
||||
the PBS tier.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
`bash -n scripts/felhom-host-install.sh` clean. The ACL behaviour was verified **live on real hardware**
|
||||
rather than by dry-run, since the whole finding is about what PVE's authorizer actually does — and the
|
||||
live run is what refuted my first design.
|
||||
|
||||
## Fleet state
|
||||
Agent **0.110.0** (with the updated sudoers) and controller **0.180.0** on both demo boxes, all healthy.
|
||||
Scratch-band ACLs at 20 rows on both — re-applied after the attempt-2 destroy consumed one. No leftover scratch guests. demo-hp's `restore_test_cadence_seconds` reverted **600 → 302400**
|
||||
(a bounded change made for the replay).
|
||||
|
||||
`felhom.eu`: this repo has a **foreign uncommitted WIP file** (`documentation/PROMPT-TEMPLATE.md`) from
|
||||
another session in the shared worktree. Left untouched; my commits staged explicit paths only, per the
|
||||
never-`git add -A` rule.
|
||||
@@ -1,119 +0,0 @@
|
||||
# REPORT — ISO boot branding + single-entry GRUB menu (R-38) · website grid restored (2026-07-19)
|
||||
|
||||
> `REPORT-<topic>.md` per this repo's parallel-session rule: another session was writing in this
|
||||
> clone tonight (CAMPAIGN 7 / `DIAG-immich-restore-2026-07-19.md`), so the shared `REPORT.md` is left
|
||||
> untouched.
|
||||
|
||||
Parts 1 and 2 of the polish train. Parts 3 and 4 landed in `felhom-agent` and `felhom-controller`;
|
||||
see their own `REPORT.md`.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — scripts v1.22.0: GRUB branding + single-entry safety (R-38)
|
||||
|
||||
### What shipped
|
||||
|
||||
Every ISO is now **repacked** after `prepare-iso`. `mkimage-surgery.sh` → **`iso-repack.sh`**:
|
||||
branding and the slice-B loader swap need the same extract → modify → re-master cycle, so they share
|
||||
one pass instead of re-mastering twice. **The mkimage recipe is untouched.**
|
||||
|
||||
**The safety half — the one that matters.** The stock PVE menu offers *Graphical*, *Terminal UI*, a
|
||||
serial variant, and an **Advanced Options** submenu holding two `nomodeset` entries, three debug
|
||||
entries, *Rescue Boot*, memtest and *UEFI Firmware Settings*. Every one of those reaches the
|
||||
**manual** installer, whose first question is which disk to wipe. They are **not emitted** — not
|
||||
hidden, not password-gated. What ships is one entry, „Felhom telepítés", default, 5 s.
|
||||
|
||||
**Boot behavior is unchanged.** The `linux`/`initrd` lines are lifted **verbatim at repack time**
|
||||
from the ISO's own *Install Proxmox VE (Automated)* entry rather than frozen into a copy in this
|
||||
repo, so a PVE bump that moves the kernel path or edits the append line tracks automatically. The
|
||||
build **fails** if they cannot be found, if the append line has lost `proxmox-start-auto-installer`,
|
||||
or if `auto-installer-mode.toml` is absent — that last one because without it the single
|
||||
Felhom-labelled entry would boot a *manual* installer, i.e. exactly what this change prevents.
|
||||
|
||||
**Gates, then a re-check against the shipped artifact.** The rendered menu is asserted to have
|
||||
exactly 1 `menuentry`, 0 `submenu`s and no *live* reference to
|
||||
`proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` (comments are stripped first —
|
||||
the template's header names the dropped entries deliberately). Then the menu and theme background
|
||||
are read back **out of `final.iso`**, not out of the extract tree.
|
||||
|
||||
**The boot card.** `grub/generate-grub-background.sh` letterboxes `website/assets/og-image_2.png`
|
||||
onto a 1024×768 gfxterm canvas at repack time (ImageMagick added to the assistant image), so the boot
|
||||
screen has **one source** and not a second pre-rendered PNG to drift. The card's own subtle grid
|
||||
(measured: 4px lines of `#0D131A` on `#0D1117`, pitch 131px) is continued across the letterbox fill
|
||||
**phase-locked** to where the card's grid lands, so the fill is seamless instead of a 500px square of
|
||||
grid floating in flat navy. The generator refuses a source whose geometry no longer matches the
|
||||
measured constants — a swapped asset would misplace every line, and that only shows up on a boot
|
||||
screen nobody re-checks. Menu positioning needs a gfxmenu theme (plain `background_image` cannot move
|
||||
the menu off the wordmark), so `grub/felhom-theme.txt` puts it in the lower third the layout leaves
|
||||
empty, optically centered (measured off a canary screenshot; the comment records the measurement).
|
||||
|
||||
### Live validation — nested canary, UEFI/OVMF, PVE 9.2-1
|
||||
|
||||
Booted the built canary ISO under QEMU with OVMF and captured the framebuffer.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| GRUB renders the Felhom card | **PASS** — background + grid visible at 1024×768 |
|
||||
| Exactly one entry, selected | **PASS** — „Felhom telepítés" only |
|
||||
| Hungarian accents under gfxterm | **PASS** — „telepítés", „Indítás … másodperc múlva" render correctly |
|
||||
| Countdown visible and counting | **PASS** — 5 → 0 |
|
||||
| Auto-fires at 0 | **PASS** — serial shows ``Booting `Felhom telepítés'`` |
|
||||
| Unattended install proceeds | **PASS** — „Fetching answers for automatic installation" → auto installer |
|
||||
| Same abort as v1.21.0 | **PASS** — `ERROR: Installation failed: filter did not match any device` → `Installation aborted`; no disk touched |
|
||||
|
||||
A first build **correctly failed closed**: the banned-entry gate matched the template's own
|
||||
explanatory header. Fixed to strip comments before matching (a comment naming a removed entry is the
|
||||
point; a directive using one is the bug), which is a gate behaving as designed.
|
||||
|
||||
### Artifacts (rebuilt on 180, `/mnt/5_hdd/felhom.eu/felhom-iso/out/`)
|
||||
|
||||
| ISO | sha256 | bytes |
|
||||
|---|---|---|
|
||||
| `felhom-pve-9.2-1-v1.22.0-n100-generic-mkimage.iso` (safety) | `ff6f06ba1dbfe10f27d703afc29516001000349147426b43c9a424a0ea28bdbf` | 1 704 482 816 |
|
||||
| `felhom-pve-9.2-1-v1.22.0-n100-demo-generic-mkimage.iso` (real) | `494db0ddf859b6b152cad4d0e0d9e9cefd27255cde07e2b41aba3ac12a217888` | 1 704 482 816 |
|
||||
| `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso` (validation) | `83c61c0413c84e27b26a37bb5dfaed2fcd44fd25e3e571c7310142bd305f2f9d` | 1 705 338 880 |
|
||||
|
||||
Both shipping ISOs: `embedding 60 modules`, `El Torito boot images=2`, fs-uuid preserved, and the
|
||||
post-re-master verification confirming 1 entry + theme background inside the finished image.
|
||||
|
||||
**Deliberately not done** (per the task): no squashfs/initrd rebranding — post-GRUB screens are still
|
||||
Proxmox-branded; no disk-setup or answer-generation change; the pairing banner is untouched.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — website: the index grid background, restored
|
||||
|
||||
**Archaeology.** Not a deliberate removal. The grid lived as a fixed `body::before` in
|
||||
`index.html`'s inline `<style>` block and was dropped in **`bed8675`** ("D3 Part 2: index + kapcsolat
|
||||
on design system v2"), the commit that migrated the page onto the shared `assets/site.css`.
|
||||
`dd54e4c`, which *created* `site.css`, has no `body::before` at all — it was a porting omission and
|
||||
nothing took its place. `ccbb13a` (the other five pages) never had it. No asset was lost: the
|
||||
mechanism was pure CSS (two stacked `linear-gradient`s), which is why nothing looked missing in the
|
||||
worktree.
|
||||
|
||||
**Restoration, not redesign.** Same 50px cells, same 1px lines, same 3% opacity, same
|
||||
`position:fixed` / `z-index:-1`. One deliberate difference: the accent is the v2 `--blue` `#0083D8`
|
||||
instead of the retired legacy `#0088cc`, which `site_gates.py` bans. Scoped to `body.page-index`,
|
||||
because index is the only page that ever had it. `site.css` cache-bust bumped `?v=1` → `?v=2` across
|
||||
all seven pages (nginx caches 7 days); BOM preserved on every file.
|
||||
|
||||
**Live verification** (felhom.eu, after git-sync deploy):
|
||||
|
||||
- Desktop: grid renders behind the hero, at its original subtlety.
|
||||
- **376px viewport** (via a same-origin iframe — the browser window would not resize in this
|
||||
environment, so the narrow case was exercised for real rather than asserted): grid renders, mobile
|
||||
layout unchanged, `scrollWidth === clientWidth` so **no horizontal overflow**.
|
||||
- Computed style confirmed live: `linear-gradient(rgba(0,131,216,0.03) 1px, …)`, `50px 50px`,
|
||||
`position: fixed`, `z-index: -1`, `pointer-events: none`.
|
||||
- `python scripts/site_gates.py` — **OK** (BOM, no legacy tokens, no `<style>` blocks, cache-busted).
|
||||
|
||||
---
|
||||
|
||||
## Docs
|
||||
|
||||
- `scripts/CHANGELOG.md` — v1.22.0 entry.
|
||||
- `website/CHANGELOG.md` — grid restoration entry.
|
||||
- `documentation/backlog/ROADMAP.md` — **R-38 flipped to SHIPPED**; **R-45** (unified async-job
|
||||
feedback) and **R-46** (verification-copy browse + expiry) added; pre-invite checklist gained the
|
||||
"golden ≥ 0.147.x carries all four infra images" line.
|
||||
- Capability map: **untouched** — no capability moved. These are UX and packaging.
|
||||
@@ -1,307 +0,0 @@
|
||||
# REPORT — the universal ISO: **PUBLISHED** (2026-07-31)
|
||||
|
||||
**Live:** `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso`
|
||||
**sha256:** `f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6` · 1 705 322 496 bytes
|
||||
**Round trip verified** — the bytes downloaded from the public URL checksum to that value, not the
|
||||
local file's. `.sha256` and manifest published beside it.
|
||||
|
||||
> Written as `REPORT-iso-release.md`, not root `REPORT.md`, per the task and the shared-clone rule.
|
||||
|
||||
## 0. Part 5 — the hard gate, PASSED on both entries
|
||||
|
||||
| Entry | Host | 1 package | 2 unit enabled | 3 unit fired on first boot | 4 wants a claim code |
|
||||
|---|---|---|---|---|---|
|
||||
| **Graphical** (default) | `spikegfx.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | `activating`; *"registering unclaimed appliance at the hub"* | **`J7N-2DA`**, token 64 B mode 600 |
|
||||
| **Terminal UI** | `spikesix.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | same | **`ZY5-YY4`**, token 64 B mode 600 |
|
||||
|
||||
Both: normal manual install, own disk chosen in the installer, own root password, real completion
|
||||
signal (installer wrote ~7 GB and rebooted; the installed system was then reached over SSH). Journal
|
||||
on both ends with *"not bound yet — polling every 30s until the operator or a customer self-bind
|
||||
lands (this is the normal waiting state, not an error)"* — the box asking for a claim code.
|
||||
|
||||
Spike 4 reasoned the graphical path would follow from shared `Install.pm`. **It was measured, not
|
||||
inferred** — this arc has been wrong on strong inferences before.
|
||||
|
||||
## 1. Venue and baselines
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Host** | `demo-hp` (t740), Tier 0 |
|
||||
| **VMs** | **500 `spike5-gfx`**, **501 `spike5-tui`** — both created with `qm` so the run is visible in the web console |
|
||||
| **Storage** | **`spike5`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images`. Root chosen deliberately: a storage at a *subdirectory* reads `disconnected` forever via the agent's `exactMount` check. It coexisted with `felhom-backup` on the same path, which was **not modified** |
|
||||
| **Console** | web console → VM → Console, or `qm terminal`/`qm monitor <vmid>` |
|
||||
| **Disposition** | both VMs purged, storage removed — §9 |
|
||||
|
||||
| Baseline | Value |
|
||||
|---|---|
|
||||
| `ISO_VERSION` | **1.26.0** (was 1.25.0) — `scripts/iso/build-felhom-iso.sh:51` |
|
||||
| `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` |
|
||||
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
|
||||
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
|
||||
| controller on `main` | `0.188.0` (`4115e88`) |
|
||||
| `felhom.eu` HEAD at build | clean, pushed, `== origin/main` |
|
||||
|
||||
## 2. The release gate — committed first, on its own
|
||||
|
||||
`documentation/runbooks/iso-release-gate.md`, commit **`e787391`**, written and pushed **before the
|
||||
first build** so it could not be rationalised afterwards. Twelve criteria, each checkable against the
|
||||
uploaded file rather than the build inputs, each carrying the spike measurement that justifies it.
|
||||
|
||||
**One criterion was amended before the build, with its reasoning recorded in the runbook.** G6 was
|
||||
first written with the six-token ban `iso-repack.sh:160-164` enforces, on the rationale *"no live route
|
||||
to a manual disk-picker"*. That rationale is obsolete for a public image — the ruling makes the manual
|
||||
installer **the product**. `proxtui` (the Terminal-UI installer we deliberately ship) and `nomodeset`
|
||||
(its graphics fallback) are dropped **for release images only**; `proxdebug`, `Rescue Boot`, `memtest`
|
||||
and `fwsetup` stay banned in both modes, and the six-token list is **unchanged** for appliance images.
|
||||
|
||||
## 3. The stub package
|
||||
|
||||
`scripts/iso/pkg/` — source committed, built by `build-deb.sh`.
|
||||
|
||||
**Contents: exactly two files, deliberately not three.**
|
||||
```
|
||||
-rwxr-xr-x ./usr/local/sbin/felhom-bootstrap.sh
|
||||
-rw-r--r-- ./lib/systemd/system/felhom-bootstrap.service
|
||||
```
|
||||
The old first-boot stub also wrote `/etc/felhom/bootstrap.env` (0600). This package does not:
|
||||
`felhom-bootstrap.sh:91` reads it only `if [[ -r ]]`, and its defaults at `:95-96`
|
||||
(`https://hub.felhom.eu`, `https://felhom.eu/scripts/felhom-host-install.sh`) are **exactly** what the
|
||||
generic pairing env set (`build-felhom-iso.sh:257-258`). Shipping it would add a 0600 file to a public
|
||||
package to express values the script already defaults to.
|
||||
|
||||
**Dependencies: none, and that is a finding.** `dpkg-deb -I` shows no `Depends` line. The payload is a
|
||||
shell script and a unit file; the binaries the script calls (`curl`, `ip`, `dhclient`, `python3`,
|
||||
`systemctl`) run at **first boot**, not at postinst time. **Spike 4's open `dpkg --configure -a`
|
||||
ordering question therefore does not arise** — confirmed, not carried.
|
||||
|
||||
**How the postinst is structurally incapable of failing** — no `set -e`, every statement individually
|
||||
guarded with `|| true` or an `if`, and an unconditional `exit 0`. `build-deb.sh` refuses to emit a
|
||||
package that violates any of it.
|
||||
|
||||
**The guarantee was tested, not asserted.** Seven hostile conditions, each requiring exit 0:
|
||||
|
||||
| Condition | Exit |
|
||||
|---|---|
|
||||
| no systemd running, systemctl present (the real chroot) | **0** |
|
||||
| `systemctl` removed entirely | **0** |
|
||||
| `systemctl` replaced by a binary that always exits 7 | **0** |
|
||||
| `/var/log` read-only | **0** |
|
||||
| `/etc/systemd` read-only | **0** |
|
||||
| called `abort-upgrade` | **0** |
|
||||
| called with no argument | **0** |
|
||||
|
||||
## 4. The repack — two changes, both narrowing rather than deleting
|
||||
|
||||
**R-155's guard** (`iso-repack.sh:100-106`) **protected the single-entry mode's promise**: that menu
|
||||
shows one item labelled "Felhom telepítés" which boots the *automated* installer, and without
|
||||
`auto-installer-mode.toml` the same label would drop the user into a manual disk-picker — a button
|
||||
promising an unattended install that silently does the opposite. That promise is real, so the guard is
|
||||
**kept unchanged for `FELHOM_MENU=single`** and simply does not apply to `release`, where the absence
|
||||
of that file is release-gate criterion G1 rather than a defect.
|
||||
|
||||
**The menu collapse** happens at `iso-repack.sh:144-148` (the stock `grub.cfg` is replaced by a
|
||||
rendered template). A `release` template now renders **two interactive entries**; entry-count and
|
||||
banned-token gates are per-mode; the post-remaster verification reads the count back out of
|
||||
`final.iso`.
|
||||
|
||||
**Ruling — default entry and timeout.** Default is **the graphical interactive entry**; timeout **15 s**.
|
||||
Reasoning: Spike 1 measured that no automated disk selection can be safe on unseen hardware (no
|
||||
property distinguishes an internal disk from a customer's backup drive; a two-disk match silently wipes
|
||||
one), so a public image whose default is unattended puts the unsafe path in front of anyone who boots
|
||||
and walks away. And Spike 2 lost a probe to a **1-second** menu — a person reading two options needs
|
||||
longer than a machine.
|
||||
|
||||
**The automated entry is absent, not broken.** Skipping `prepare-iso` means no
|
||||
`auto-installer-mode.toml`, and the stock `grub.cfg` emits the Automated entry only inside
|
||||
`if [ -f auto-installer-mode.toml ]`. There is no entry that could fail in front of a customer.
|
||||
|
||||
## 5. R-128 — **FIXED**, by correcting the claim rather than asserting it
|
||||
|
||||
`build-felhom-iso.sh:44` claimed `ISO_VERSION` "aligns with felhom-host-install `SCRIPT_VERSION`".
|
||||
Nothing evaluated it and the two had drifted. **I did not turn it into a real assertion, because the
|
||||
coupling it claimed does not exist:** the ISO is a frozen artifact, while `felhom-host-install.sh` is
|
||||
fetched at run time from the website's git-sync of `main` (R-94/R-110), so whatever version an ISO
|
||||
carries, the script a box runs is always current. An assertion would invent a constraint. The comment
|
||||
now states the independence, and `ISO_VERSION` is `1.26.0`.
|
||||
|
||||
## 6. Part 5 — the defect, the fix, and where it now stands
|
||||
|
||||
### Round 1 (`1.26.0`) — the Terminal UI install FAILED on observable 4
|
||||
|
||||
Three of four passed: the package installed, the unit was enabled from inside the installer chroot,
|
||||
and the unit **fired on first boot** and registered at the hub. The fourth failed:
|
||||
|
||||
```
|
||||
felhom-bootstrap.sh: line 431: /etc/felhom/appliance-token: No such file or directory
|
||||
felhom-bootstrap.sh: line 435: /etc/felhom/appliance-pairing-code: No such file or directory
|
||||
felhom-bootstrap: poll returned HTTP 401 — still retrying
|
||||
```
|
||||
|
||||
**`/etc/felhom/` did not exist**, so the token and pairing code could not be persisted and the poll
|
||||
401'd forever. No claim code would ever appear.
|
||||
|
||||
**Root cause, mine.** `stub-first-boot.sh` opened with
|
||||
`install -d -m 0755 /etc/felhom /usr/local/sbin`. §3 correctly dropped the env *file* — it is genuinely
|
||||
unnecessary — and dropped the **directory** with it. `felhom-bootstrap.sh` uses `/etc/felhom/` for its
|
||||
runtime state.
|
||||
|
||||
**Why the gate missed it.** G9 proves the packaged script is byte-identical to repo HEAD, and it was.
|
||||
**I verified the payload files and never the directory the payload writes into** — a check that proves
|
||||
the thing present and not the thing it depends on.
|
||||
|
||||
### The fix, and its red-proof
|
||||
|
||||
`build-deb.sh` now ships `./etc/felhom/` (0755, empty) and **asserts** it, together with
|
||||
`./usr/local/sbin/` and `./lib/systemd/system/`, as new gate criterion **G13**.
|
||||
|
||||
**Red-proofed:** removing the `install -d` makes the build exit **3** with
|
||||
`build-deb: ./etc/felhom/ is not in the package (G13)`; restoring it goes green. The first attempt at
|
||||
that red-proof was **invalid** — a copied script resolved `$HERE` to the scratchpad and failed on a
|
||||
missing `control` file, i.e. non-zero for the wrong reason — and was redone in place.
|
||||
|
||||
### Round 2 (`1.26.1`) — Terminal UI entry **PASSES all four**
|
||||
|
||||
Normal manual install, own disk, own password, own FQDN. Host `spikesix.felhom.eu`.
|
||||
|
||||
| # | Observable | Result |
|
||||
|---|---|---|
|
||||
| 1 | the `.deb` is installed | **PASS** — `ii felhom-bootstrap 1.26.1 all` |
|
||||
| 2 | the unit is enabled | **PASS** — `enabled` |
|
||||
| 3 | the unit fired on first boot | **PASS** — `activating`; journal shows *"PAIRING mode (generic ISO, no baked customer/passphrase)"* → *"registering unclaimed appliance at the hub"* → *"registered — appliance token stored (0600)"* |
|
||||
| 4 | **the box wants a claim code** | **PASS** — `/etc/felhom/appliance-pairing-code` = **`ZY5-YY4`**; `appliance-token` present, 64 B, mode `600`; journal: *"not bound yet — polling every 30s until the operator or a customer self-bind lands (this is the normal waiting state, not an error)"* |
|
||||
|
||||
That is the product working end-to-end from a public image on a manual install: own disk, own
|
||||
password, nothing baked, and the box asking for a claim code.
|
||||
|
||||
### The Graphical entry — **NOT COMPLETED**, and this is why nothing is published
|
||||
|
||||
It reached the installer from the same image (KVM dialog, EULA, and the **Target Harddisk** screen
|
||||
showing `/dev/sda (20.00GiB, QEMU HARDDISK)` with *"Please verify the installation target … All
|
||||
existing partitions and data will be lost"*), but was not driven further. `Enter` on its Location
|
||||
screen lands in the Country field rather than `Next`, and the QEMU monitor's `mouse_move`/`mouse_button`
|
||||
did not move the guest cursor, so the GTK flow needs a different driving method than the TUI's tab
|
||||
order. **Part 5 requires both entries. It is not fully passed, so Part 7 did not run.**
|
||||
|
||||
The `.deb` path lives in `Install.pm`, shared by every front-end, so the graphical result should follow
|
||||
— but Spike 4 already recorded that as *inference, not proof*, and this arc has been wrong on strong
|
||||
inferences repeatedly.
|
||||
|
||||
### A fixture bug of mine, recorded twice because it cost two diagnoses
|
||||
|
||||
`qm set <vmid> --scsi0 … --boot order="scsi0;ide2"` silently produced `boot: order=net0;ide2` — PVE
|
||||
processed `--boot` before `--scsi0` existed. Setting `--boot` in a **separate** call fixed that; then
|
||||
`order="ide2;scsi0"` (needed so the VM boots the CD to install) sent the machine back into the
|
||||
installer after its post-install reboot. **Detach the CD, or flip the order to `scsi0`, once the
|
||||
install completes.** Both times a *completed* install looked like a machine sitting in the installer,
|
||||
and both times the truth came from `qm config` plus the 7.0 GB disk rather than from the screen.
|
||||
|
||||
## 7. Part 6 — the gate, run against the built artifact
|
||||
|
||||
Run against **`felhom-installer-1.26.1-pve9.2-1.iso`**,
|
||||
sha256 **`f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6`** — the image the
|
||||
Terminal-UI install in §6 was performed from, and the one that would be uploaded.
|
||||
|
||||
| # | Criterion | Scanned for | Result |
|
||||
|---|---|---|---|
|
||||
| **G1** | no `answer.toml` / `auto-installer-mode.toml` | both names at ISO root | **PASS — 0** |
|
||||
| **G2** | no root password or hash | `.rootpw.txt` companion; the answer file that would carry a hash | **PASS** — no `.rootpw.txt` emitted; no answer file exists to hold one |
|
||||
| **G3** | no SSH key | `root-ssh-keys`, `ssh-rsa`, `ssh-ed25519` | **PASS** — no answer file; package carries only a script and a unit |
|
||||
| **G4** | no customer identity | `FELHOM_CUSTOMER_ID`/`RETRIEVAL_PASSPHRASE` with values, claim code, api key, Bearer | **PASS** — only the empty initialisers at `felhom-bootstrap.sh:89` |
|
||||
| **G5** | credential scan **by enumeration** vs the stock PVE ISO | full recursive file-list diff, both directions | **PASS** — exactly **four** added paths: the three `felhomtheme/` files and `/proxmox/packages/felhom-bootstrap_1.26.0_all.deb`; three removed (`pvetheme/`) |
|
||||
| **G6** | menu present, both paths, human timeout | entry count, `set default`/`timeout`/`timeout_style`, banned tokens | **PASS** — 2 entries, `default=0` (graphical), `timeout=15`, `timeout_style` underscore |
|
||||
| **G7** | one `felhom-*.deb`, version recorded | `/proxmox/packages/felhom-*` | **PASS** — exactly 1, `Package: felhom-bootstrap`, `Version: 1.26.0`, **no `Depends`** |
|
||||
| **G8** | postinst cannot fail | live (comment-stripped) `systemctl start\|daemon-reload\|restart`, network commands, `set -e`, last line | **PASS — 0, 0, 0**, ends `exit 0` |
|
||||
| **G9** | `felhom-bootstrap.sh` == repo HEAD | sha256 of the packaged file vs the repo file | **PASS** — both `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
|
||||
| **G10** | build inputs committed | `git status --porcelain`, HEAD vs origin | **PASS** — clean and pushed at build time |
|
||||
| **G11** | published checksum + round trip | — | **NOT RUN** — nothing was published |
|
||||
| **G12** | bucket stays private | — | **NOT RUN** — the bucket was never touched |
|
||||
| **G13** | *(new, from Part 5's failure)* every directory the payload writes into is in the package | `./etc/felhom/`, `./usr/local/sbin/`, `./lib/systemd/system/` in `dpkg-deb -c` | **PASS** — all three present in `felhom-bootstrap_1.26.1_all.deb`; asserted by `build-deb.sh` and red-proofed |
|
||||
|
||||
**A gate refinement found by running it.** G7 also asked that the ISO's copy of the `.deb` sha256-match
|
||||
the package built from source. It does not, and cannot: `dpkg-deb` embeds build timestamps, so two
|
||||
builds of identical source differ. **G9 — the payload's identity — is the meaningful check**, and it
|
||||
passes. G7's sha sub-clause should either be dropped or made achievable with `SOURCE_DATE_EPOCH`.
|
||||
|
||||
## 8. Publication — done, and verified by round trip
|
||||
|
||||
Uploaded with `rclone` **in a container, configured entirely by environment variables**, so no
|
||||
credential file was ever written to disk — the fence asks for config files to be kept out of repo
|
||||
paths and removed at teardown; none was created to remove. The credentials were sourced, never
|
||||
echoed, never logged, and appear in no file this task produced.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| objects in the bucket | the ISO (1 705 322 496 B), `.sha256` (103 B), `.manifest.txt` (2 492 B) |
|
||||
| **round trip** | `curl https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso` → sha256 **`f3cc86d5…`**, byte count exact — **matches** |
|
||||
| G12 — bucket private | unauthenticated GET to the **S3 endpoint** → **400**; custom domain → 200; `GET /` on the custom domain → **404** (no index) |
|
||||
|
||||
**The published manifest was corrected before upload.** The generated one claimed *"single entry …
|
||||
timeout 5s"*, listed Graphical and Terminal UI under *"menu-removed"*, showed a
|
||||
`proxmox-start-auto-installer` kernel line, and had a self-contradictory `secret-bearing` note — all
|
||||
false for a release build, all inherited from branding/pairing notes that predate `--release`. The
|
||||
generator is fixed and the sidecar regenerated. **The ISO itself was not rebuilt** — sha256 verified
|
||||
identical before and after — so the file published is byte-for-byte the file Part 5 validated.
|
||||
|
||||
## 9. Teardown
|
||||
|
||||
**demo-hp:** VMs 500/501 `qm destroy --purge`; **scratch storage `spike5` removed**
|
||||
(`storage.cfg` back to 4 entries, `grep -c spike5` = 0); `/mnt/nvme-1tb/images/` empty; usage
|
||||
**6.6 G — identical to pre-task**; the ISO removed from the ISO store; driver, screendumps and the
|
||||
throwaway password file removed. `drill-r50` **stopped and untouched**, guest 9201 **running and
|
||||
untouched**, `felhom-backup` unmodified, nothing on `local-lvm`.
|
||||
|
||||
**demo-felhom:** not contacted.
|
||||
|
||||
**DooPlex:** scratchpad 84 K; build logs and the package build tree removed. `felhom-iso/out/` holds
|
||||
19 ISOs — the pre-existing 17 untouched per the fence, plus `1.26.0` and `1.26.1`, both unpublished
|
||||
and **neither with a `.rootpw.txt`**, which is G2's own evidence. Repo tree clean and pushed.
|
||||
|
||||
### Hub-side — **cleared**
|
||||
|
||||
Observable 4 works *by* the box registering itself, so each proof install created an unclaimed
|
||||
appliance. All three were discarded: **16** and **17** (the 1.26.0 round), then **18** (the two
|
||||
1.26.1 proofs). `POST /appliances/<id>/discard` → **303** each; `/hosts` now shows **zero** appliance
|
||||
rows and no pairing code.
|
||||
|
||||
The endpoint is `/discard`, **not** `/delete` — `hub/internal/web/server.go:345`, POST only. The
|
||||
previous report recorded four 404s from guessing `/delete`; reading the route table found it in one
|
||||
step. **R-131 gains no row.**
|
||||
|
||||
## 10. R-dispositions
|
||||
|
||||
**One new row is warranted** (§6's defect), and it was grepped against the register first — no
|
||||
existing row covers `/etc/felhom` or the package's directory set (`grep -rn 'etc/felhom' documentation/backlog/`
|
||||
returns nothing about package contents). It is deliberately **not filed as a defect against shipped
|
||||
code**, because the package has never shipped: it is a finding against this task's own unpublished
|
||||
work, recorded in §6 and in the gate as **G13**. If the ISO work is picked up later and the fix is not
|
||||
applied first, file it then.
|
||||
|
||||
Otherwise, no new rows. Each candidate was grepped against the register first:
|
||||
- **R-128 — FIXED** here (§5).
|
||||
- **R-155 — RESOLVED** here (§4): the guard is narrowed, not deleted.
|
||||
- **R-154** (`[first-boot]` is automated-only and nothing in the tree says so) — **addressed in code
|
||||
rather than by a row**: `pkg/build-deb.sh`'s header and `grub-release.cfg.tmpl` both state it with
|
||||
the measurements. The register row can close when the docs land.
|
||||
- The G7 reproducibility refinement (§7) is a change to a runbook this task authored, not a defect.
|
||||
|
||||
## 11. What did not happen, and what is still open
|
||||
|
||||
- **Part 8 partially done.** The release-gate runbook (`e787391`), `day0-install.md` C.0 (ISO vs
|
||||
manual, and when to use which) and `scripts/CHANGELOG.md` are written. **`OPEN-ITEMS.md` /
|
||||
`ROADMAP.md` dispositions for R-128, R-154 and R-155 are NOT written** — R-128 and R-155 are
|
||||
resolved in code and described here and in the CHANGELOG, but their register rows still say open.
|
||||
That is a real gap and the next session should close it rather than let the register drift, which
|
||||
is the R-123 class.
|
||||
- **The `.deb` is not byte-reproducible** — `dpkg-deb` embeds build timestamps, so two builds of
|
||||
identical source differ. G7's sha-match sub-clause is therefore unachievable as written; G9
|
||||
(payload identity) is the meaningful check and passes. Either drop the sub-clause or set
|
||||
`SOURCE_DATE_EPOCH`.
|
||||
- **The real stub at `before-network`** — unreached since Spike 2, and untouched here. It is now
|
||||
narrower than it was: on the `.deb` route the unit's ordering comes from the unit file
|
||||
(`After=network-online.target …`), not from `[first-boot].ordering`, so it governs operator-built
|
||||
appliance images only.
|
||||
- **Secure Boot** was not exercised. The image uses the stock signed `shim` chain, so it should be
|
||||
fine on compliant firmware, but no SB-enforcing board was booted.
|
||||
- **Only virtual hardware** was tested. Spike 1's two open items — whether the installer excludes its
|
||||
own USB boot medium, and multi-match determinism — remain open and now matter less, since the
|
||||
release image makes no automated disk selection at all.
|
||||
@@ -1,71 +0,0 @@
|
||||
# REPORT — PBS prune moved server-side, write proof closed (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run. **No code, no version bump.** Topic-scoped per the
|
||||
parallel-session rule; shared `REPORT.md` untouched.
|
||||
|
||||
**Full record:** `documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`
|
||||
|
||||
---
|
||||
|
||||
## Outcome — all parts complete
|
||||
|
||||
| Part | Result |
|
||||
|---|---|
|
||||
| 1 — prune gate | **Config-gated.** `keep_last: 0` on the PBS tier, both boxes → `prune_pbs_allowed=false`. No code, **no grant** |
|
||||
| 2 — prune jobs | 2 jobs, per live namespace, `keep-last 2`, daily **03:30 UTC / 05:30 CEST** |
|
||||
| 3 — dry run → real | Gate passed; both `TASK OK`; demo-hp 3→2, demo-felhom untouched |
|
||||
| 4 — write proof | **CLOSED — `TASK OK`, no job errors** |
|
||||
| 5 — GC | Scheduled `sun 04:30 UTC / 06:30 CEST`. **NOT run** |
|
||||
| 6 — `verify-new` | **Enabled** (operator ruling) |
|
||||
| — legacy ns | `demo-felhom-01` deleted with its ACLs + token (operator ruling, confirmed twice) |
|
||||
| 7 — roadmap | **R-89** + CONTEXT.md note |
|
||||
|
||||
## The fix, in one line
|
||||
|
||||
`allowPBSPrune := !t.Primary && t.KeepLast > 0` — so setting the PBS tier's `keep_last` to `0`
|
||||
disables both the `--prune-backups` value and the gate, in one config edit, **while the tier stays
|
||||
armed**. Verified: `backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0
|
||||
prune_pbs_allowed=false`, no `tier REJECTED` line.
|
||||
|
||||
## The proof
|
||||
|
||||
```
|
||||
07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied
|
||||
07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the change
|
||||
```
|
||||
|
||||
New snapshot `ns/demo-hp/ct/9201/2026-07-27T09:37:29Z`, chunks 9,787 → **9,813**, 97.0 % reused,
|
||||
45.80 s, **prune step absent entirely**. Driven via `POST /api/guest-backup/trigger` → `TriggerNow()`
|
||||
— the UI's „Mentés most" path, not `--selftest`, not raw `vzdump`.
|
||||
|
||||
**Hub gauge evidence NOT satisfied** — a +32.8 MB delta is below its 0.1 GB display granularity, so it
|
||||
still reads 12.6 GB / 13 %. Stated plainly rather than dressed up.
|
||||
|
||||
## The demo-felhom prediction — CLOSED
|
||||
|
||||
The claim was that demo-felhom's next weekly backup would make 3 snapshots and reproduce the prune
|
||||
failure. Neutralised on both halves: the box no longer attempts prune, and `prune-demo-felhom` covers
|
||||
the namespace server-side (verified live, `TASK OK`). **It will not reproduce.**
|
||||
|
||||
## Why it mattered more than the unpruned snapshots
|
||||
|
||||
demo-hp's PBS tier had reported failure on **every** backup since the tier was created on 07-26, while
|
||||
the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure
|
||||
invisible — which is precisely what happened at 07:13 UTC, when a real outage produced an
|
||||
indistinguishable result.
|
||||
|
||||
## Security property preserved
|
||||
|
||||
**No prune right was granted to any box.** Final ACLs are four entries, write-only
|
||||
(`DatastoreBackup`), live namespaces only. A compromised box still cannot delete its own offsite
|
||||
backups. `felhom-tenantsync.sh` was **not** edited — the ruling makes its current grant correct.
|
||||
|
||||
## Open
|
||||
|
||||
1. **R-89** — hub-owned retention policy (today's jobs are increment 1, not a stopgap).
|
||||
2. **Does the restic key on `storage-box-pool-1` have DELETE rights?** Unanswered, carried in R-89,
|
||||
and the more urgent half — if so, the daily app-data tier has the identical exposure and
|
||||
append-only mode is the equivalent answer. Rule once for both tiers.
|
||||
3. **GC has still never run.** First execution Sunday 04:30 UTC; worth watching, as nothing has ever
|
||||
exercised it here.
|
||||
4. Old 13 GB datastore copy still at `/srv/pbs-felhom` — rollback intact.
|
||||
-194
@@ -1,194 +0,0 @@
|
||||
# REPORT — R-100: a failing offsite tier must go stale (2026-07-28)
|
||||
|
||||
Hub **v0.79.0 → v0.80.0**; companion `felhom-controller` **v0.180.0 → v0.181.0** (the producer, shipped
|
||||
first). Written as `REPORT-r100.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom.eu 6369570`, `felhom-controller 4056fec`, `felhom-agent d5c7691` — all = origin/main. The only
|
||||
dirt in `felhom.eu` was a **foreign** `documentation/PROMPT-TEMPLATE.md` (shared worktree, untouched).
|
||||
Hub manifest **and** running pod both `0.79.0`; `staleAfter` = 48h; controller 0.180.0 and agent 0.110.0
|
||||
live on both boxes.
|
||||
|
||||
---
|
||||
|
||||
## The premise was wrong, and it was mine
|
||||
|
||||
R-100 was filed yesterday claiming *"the operator's fleet-wide alarm plane is silent"*. Phase 0 refuted
|
||||
that, twice:
|
||||
|
||||
1. **A failing offsite run does alarm.** `main.go:655` wires `SetOffboxNotify` → `NotifyBackupFailed`;
|
||||
the notify cooldown is 6h against a 24h cadence, so a nightly failure alarms nightly. Live hub DB:
|
||||
`backup_failed | operator | sent | 5`, latest 2026-07-27 17:42. The `isStale` doc comment —
|
||||
*"a recent-but-failing run is NOT stale (backup_failed owns that signal)"* — was **accurate**.
|
||||
2. **The orphaned-repo path I expected to be an indefinite hole is already covered.** The scheduled run
|
||||
returns early at `offbox.go:606`, *before* the `LastRun` write at `:716`, so `LastRun` freezes and
|
||||
`offsite_stale` fires normally.
|
||||
|
||||
I could find no failure mode that both advances `LastRun` and produces no operator signal.
|
||||
|
||||
**The real defect — defeated defence in depth.** `offsite_stale` is the hub-side, *pull-based* net that
|
||||
exists to be independent of controller-*pushed* events. Anchoring it on `LastRun` made it depend on the
|
||||
very thing it backs up: when the push is lost, the net cannot compensate, because the failing controller
|
||||
keeps refreshing the field the net reads. **F-HUB — this campaign's own finding, the hub dropping an
|
||||
event under `SQLITE_BUSY` with no retry** — is exactly that loss.
|
||||
|
||||
**Honest severity: MEDIUM**, not the top-ranked item. The fix is unchanged; the justification is not.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 answers
|
||||
|
||||
**P0.1 — a last-success timestamp did not exist.** `OffboxTarget` carried `LastRun`/`LastStatus`/
|
||||
`LastError`/`LastDuration` only. Recording one is a new field, not a transmission of something known.
|
||||
|
||||
**P0.2 — `LastStatus` on the wire**, from 4000 live reports (not from source alone):
|
||||
|
||||
| value | count | paired with |
|
||||
|---|---|---|
|
||||
| `ok` | 2269 | `last_run` set |
|
||||
| absent/null | 541 | `last_run` **empty** — never-ran |
|
||||
| `error` | 27 | `last_run` set |
|
||||
| **`running`** | 7 | a report captured **mid-run** |
|
||||
|
||||
Plus 1156 reports with no `offsite` object at all. **The legacy trap — status absent *with* a real
|
||||
`last_run` — occurs 0 times**, because `LastStatus="running"` is written the moment a run starts. It is
|
||||
still handled explicitly, but it is not a live shape. `running` being real is why the verdict ignores
|
||||
status entirely.
|
||||
|
||||
**P0.3 — sweep**
|
||||
|
||||
| tier | `LastRun` written on failure? | read as success by a verdict? |
|
||||
|---|---|---|
|
||||
| **Offsite restic** | YES (`offbox.go:716`) | **YES — hub `isStale`.** The defect |
|
||||
| **Tier 2 cross-drive** | YES (`recordTier2Failure`) | No hub verdict; UI only → **R-101, filed** |
|
||||
| Tier 1 recovery units | **NO** — derived from an actual artifact | structurally immune |
|
||||
| Shares offsite leg | YES | `sharing.html:180` shows the time only when status=="ok" — honest |
|
||||
| DB dump | n/a — **event-based** (`db_dump_completed`/`db_dump_failed`) | immune by design |
|
||||
|
||||
`offsite.go` is the **only** hub verdict anchored on a `LastRun`-shaped field. The deadline checker
|
||||
already uses distinct success/failure *events* — the pattern this converges on.
|
||||
|
||||
**P0.4 — the customer is NOT shown a failed offsite run as successful.** `backups_remote.html:34-36`
|
||||
leads with the status (`✓ Rendben` / `✗ Hiba` / `Fut…`). Two narrower Tier-2 instances → **R-101**.
|
||||
|
||||
---
|
||||
|
||||
## The fix
|
||||
|
||||
**Controller v0.181.0 (producer, shipped first).** `OffboxTarget.LastSuccess`, carried on the report as
|
||||
`last_success`. The rule is a pure function called unconditionally beside the `LastRun` write:
|
||||
|
||||
```go
|
||||
func offboxAnchorAfterRun(prev, at string, runErr error) string {
|
||||
if runErr != nil { return prev } // failures neither advance nor clear
|
||||
return at
|
||||
}
|
||||
```
|
||||
|
||||
Both directions are separate bugs: a failure must not **advance** it (the original defect) and must not
|
||||
**clear** it (one bad night making an established tier read as never-succeeded).
|
||||
|
||||
**Two silent-wipe sites found and closed** — the "seam built but never wired" shape, where the field
|
||||
exists, the writer sets it, and an unrelated routine path zeroes it:
|
||||
- `offboxConfigHandler` rebuilds the target from the form and copies runtime status field by field, so
|
||||
an ordinary settings save would have erased the anchor;
|
||||
- `ApplyOffsiteTarget` does the same on a hub re-apply.
|
||||
|
||||
Neither would have surfaced until the verdict changed, days later. **The first was proven live** — see
|
||||
below.
|
||||
|
||||
**Hub v0.80.0.** Three deliberate branches:
|
||||
- **never ran** — unchanged v0.73.0 anchored behaviour, still keyed on `last_run` on purpose: that field
|
||||
answers "has anything ever happened here", and a box whose *first* run failed is a run, not a newborn.
|
||||
- **legacy** (`last_run` set, no `last_success`) — degrades **explicitly** to the old behaviour, logged
|
||||
**once** per customer. Absence-as-failure would alarm the whole un-upgraded fleet; absence-as-success
|
||||
keeps the bug. Same degrade direction as R-88 Part 2's `age_state`.
|
||||
- **anchored** — counts from `last_success`; `last_status` is deliberately not consulted, because
|
||||
"error ⇒ stale" pages on every blip (the F-A1 noise path).
|
||||
|
||||
**The alarm text had to move with the verdict.** `emitStale` still said `last run 8h ago` while firing on
|
||||
a six-day-old success — a true alarm that reads as false. `staleAge` now separates *"runs are happening
|
||||
and failing — check the error, not the schedule"* from *"the offsite leg is silently not running"*.
|
||||
|
||||
---
|
||||
|
||||
## Red-proofs — all observed failing
|
||||
|
||||
| # | red-proof | observed failure |
|
||||
|---|---|---|
|
||||
| A | restore the `LastRun` anchor | `a tier that has not succeeded in 6 days reads as FRESH — that is R-100` |
|
||||
| B | delete the never-ran branch | `a newborn box alarmed — this is the 2026-07-23 cry-wolf that v0.73.0 fixed` |
|
||||
| C | collapse to `LastStatus == "error"` | `a single transient failure alarmed — 20h ... well inside the 48h threshold` |
|
||||
| D | delete the legacy degrade | `a legacy controller alarmed — that is a fleet-wide alarm storm on an un-upgraded fleet` |
|
||||
| + | drop the `runErr` guard (controller) | `a FAILED run advanced LastSuccess ... that is the R-100 defect in mirror image` |
|
||||
| + | always return `prev` | `a successful run did not advance the anchor` |
|
||||
| + | drop the wire field | `OffboxReportStatus dropped LastSuccess — the hub would degrade forever` |
|
||||
| + | drop the handler preservation | `a settings save erased LastSuccess` |
|
||||
|
||||
**A hollow test of my own, caught by red-proofing it.** The first version of the controller test
|
||||
re-implemented the rule in a local closure — mutating production code left it **green**. That is why
|
||||
`offboxAnchorAfterRun` was extracted: the test now calls the real rule.
|
||||
|
||||
Fixtures are the **real** wire shapes from P0.2, not invented JSON.
|
||||
`go build`/`go vet`/`go test` green in both repos (hub 17 pkgs, controller 27 pkgs), run separately
|
||||
from every commit.
|
||||
|
||||
---
|
||||
|
||||
## §6 — LIVE, on demo-hp (disposable; `peti-felhom` never touched)
|
||||
|
||||
A genuine restic failure was induced by pointing the target at a **closed port** (23 → 2) — it creates
|
||||
nothing, touches no data, and is exactly reversible.
|
||||
|
||||
```
|
||||
success run → last_status=ok last_run=11:24:20Z last_success=11:24:20Z
|
||||
INJECT port 23 → 2 ... and the settings save PRESERVED last_success = 11:24:20Z ← the wipe-site fix, live
|
||||
failing run → last_status=error last_run=11:25:48Z last_success=11:24:20Z ← ANCHOR HELD
|
||||
```
|
||||
|
||||
**As the hub received it:**
|
||||
|
||||
| box | status | `last_run` | `last_success` | anchor |
|
||||
|---|---|---|---|---|
|
||||
| **demo-hp** (induced failure) | `error` | 11:25:48Z | **11:24:20Z** | **HELD** |
|
||||
| **demo-felhom** (healthy) | `ok` | 11:29:22Z | **11:29:22Z** | **advanced** |
|
||||
|
||||
Also observed live, unplanned: **Scenario E**. Both boxes were still on the old controller at hub
|
||||
startup, and the degrade logged **exactly once per customer** —
|
||||
`[WARN] [offsite] demo-hp: controller sends no last_success — staleness degraded to the last-ATTEMPT
|
||||
anchor`. Two lines, two customers, same second.
|
||||
|
||||
**No spurious alarms:** 0 `offsite_stale` events since deploy (correct — both tiers succeeded minutes
|
||||
ago). `backup_failed` fired for demo-hp at 11:25:48 from the induced failure, confirming the
|
||||
pre-existing channel is intact and re-confirming the Phase 0 correction.
|
||||
|
||||
**Config restored** and verified field by field: `host=u629488-sub3.your-storagebox.de port=23
|
||||
user=u629488-sub3 repo=/home/felhom-repo enabled=True escrow=escrowed`.
|
||||
|
||||
### Proven live vs. proven by injected clock — stated plainly
|
||||
- **Live:** the anchor does not advance on failure; it does on success; it survives a settings save;
|
||||
`last_success` reaches the hub; the legacy degrade fires once per customer; no spurious alarms.
|
||||
- **Unit, injected clock only:** the 48h **threshold** behaviour itself — Scenarios A/B/C/D turning on
|
||||
elapsed time. A live threshold test would take days. **The threshold was NOT proven live.**
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — the rule
|
||||
**"Presence is not success"** added to `CLAUDE.md` and its versioned copy, with both instances
|
||||
(F-CRIT-2's phantom ctime, R-100's `LastRun`) and the corollary R-100's fix produced: when a verdict
|
||||
changes which field it counts from, the **alarm text must change with it**. `// R-100` notes sit at
|
||||
`isStale` and at the controller write site, each naming the test that pins it.
|
||||
|
||||
## Filed, not fixed
|
||||
- **R-101** — Tier-2 `LastRun` is also written on failure, and three customer surfaces render it without
|
||||
a status (two degraded branches plus the restore-confirm dialog). No hub verdict reads it.
|
||||
|
||||
## NOT yet live-validated (carried forward)
|
||||
- **The 48h staleness threshold itself** (see above) — and with it Scenario A end-to-end: no
|
||||
`offsite_stale` event has yet been *observed firing* from a genuinely stale success anchor, because
|
||||
that needs 48h of failure.
|
||||
- **Fault 4** — restic transport interruption; four injection approaches defeated by guest-bridged
|
||||
networking. (This task's closed-port injection sidesteps it rather than solving it.)
|
||||
- **R-99** — prune never removes phantom snapshots.
|
||||
- **R-101** — filed today, unvalidated.
|
||||
- `contentionAlarmAfter` (3h) — injected clock only.
|
||||
-181
@@ -1,181 +0,0 @@
|
||||
# REPORT — R-101 + F-DIAG + F-OPS (2026-07-28)
|
||||
|
||||
Controller **v0.181.0 → v0.182.0**; `felhom.eu` gains the manual-restore runbook (F-OPS) and the
|
||||
OPEN-ITEMS rows. Written as `REPORT-r101.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom-controller 3db8bfb`, `felhom.eu 6b7d516`, `felhom-agent d5c7691` — all = origin/main; the only
|
||||
dirt in `felhom.eu` was a **foreign** `PROMPT-TEMPLATE.md`. Controller **0.181.0** live on both boxes,
|
||||
hub `felhom-hub:0.80.0` ready 1/1.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0
|
||||
|
||||
**The render sites — three dishonest, two already honest.** The spec listed `backups_apps.html:216`
|
||||
as a defect site; it is in fact the one branch that *already* pairs its timestamp with a status badge.
|
||||
The real third site is the `Tier2DestInactive` branch.
|
||||
|
||||
| site | rendered | honest? |
|
||||
|---|---|---|
|
||||
| `:231` **restore confirm dialog** | `Legutóbbi másolat: {{.Tier2LastRun}}` — raw RFC3339, no status | **NO** — the one that matters |
|
||||
| `:195` `Tier2DestDisconnected` | `Utolsó: …`, no status | **NO** |
|
||||
| `:206` `Tier2DestInactive` | `Utolsó: …`, no status | **NO** |
|
||||
| `:217` main configured branch | `Utolsó: …` **+ status badge** | already honest |
|
||||
| `sharing.html:181` | rendered **only** when status=="ok" | already honest |
|
||||
|
||||
`Tier2LastStatus` was already set unconditionally at `handlers.go:1182`, so this was a wording/anchor
|
||||
problem, not a plumbing one. The restore button was gated on `{{if .Tier2LastRun}}`, so **Scenario C
|
||||
was live-reachable**: a tier that had attempted and never succeeded offered a restore and a timestamp.
|
||||
|
||||
**`cd.LastRun` is written on failure** — `recordTier2Failure` (`tier2.go:573-574`) writes it alongside
|
||||
`LastStatus:"error"`. Identical shape to R-100.
|
||||
|
||||
**Legacy state is universal, not an edge case.** All 7 Tier-2 rows across both boxes had `last_run` and
|
||||
no anchor. Scenario E was therefore the *initial state of every customer*, which is what made the
|
||||
legacy marker non-optional.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — the strings shipped
|
||||
|
||||
| case | string |
|
||||
|---|---|
|
||||
| dialog, normal | `… Legutóbbi sikeres másolat: 2026-07-28 16:43.` |
|
||||
| dialog, newest attempt failed | `… Legutóbbi sikeres másolat: 2026-07-28 16:40. Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek lehetnek.` |
|
||||
| card | `Utolsó sikeres: 2 perce` |
|
||||
| never succeeded | `Még nincs sikeres másolat` + `Még nincs sikeres másolat, amiből vissza lehetne állítani.` (restore removed) |
|
||||
| **legacy row** | `Utolsó: …` / `Legutóbbi másolat: …` — **today's wording, unchanged**, logged once per stack |
|
||||
|
||||
**Timestamp made human-readable** (agreed): new `fmtTimeStr` renders Budapest-local `2026-07-28 16:40`
|
||||
instead of the raw UTC `2026-07-28T14:40:55Z` a customer was previously asked to reason about.
|
||||
|
||||
**`SuccessTracked` is what makes the legacy case possible at all.** Without it, "row predates the
|
||||
anchor" and "row has an anchor and it is empty" are indistinguishable — both are `LastSuccess==""` —
|
||||
and every existing row would have rendered as never-succeeded on deploy. Legacy rows migrate on first
|
||||
touch: a row whose last known state was `ok` adopts that time (truthful — under the old code that run
|
||||
did succeed); a row whose last state was `error` seeds **nothing**, because the old data evidences no
|
||||
success.
|
||||
|
||||
## Part 2 — the copy-site hazard, and it was in the path
|
||||
|
||||
The three `record*` helpers each built a **whole `CrossDriveBackup` literal**, with a helper re-applying
|
||||
exactly two fields; everything else was zeroed on every status write. Adding `LastSuccess` to that shape
|
||||
would have had `recordTier2Failure` **clear** it — the mirror image of the defect, firing on the *first*
|
||||
failure rather than lying dormant.
|
||||
|
||||
Replaced with **`tier2Update`**, which copies the existing row and overlays the outcome: **compile-safe
|
||||
by construction** — a new field carries over unless deliberately overwritten, so nothing is preserved by
|
||||
a list that can fall out of date. Callers now clear explicitly what a run invalidates, reproducing the
|
||||
old behaviour exactly.
|
||||
|
||||
**Sweep of other rebuild sites:** `SetTier2Preference` mutates in place (safe); `SetCrossDriveConfig(name, nil)`
|
||||
in `api/router.go:774` is a deliberate delete. No others.
|
||||
|
||||
## Part 3 — F-DIAG
|
||||
|
||||
| class | signal it maps to | message head |
|
||||
|---|---|---|
|
||||
| `quota` | the pre-run soft-quota gate | `A távoli mentés nem fért el a tárhelykereten belül` |
|
||||
| `orphaned` | `ErrOffboxOrphaned` sentinel | `A távoli tárhely egy korábbi, már nem elérhető kulccsal készült` |
|
||||
| `no_repo` | restic "unable to open config file" | `A távoli tárhelyen nincs mentési adattár` |
|
||||
| `no_units` | "produced no snapshots" | `Nem volt mit menteni: egyetlen kijelölt alkalmazásnak sem található mentése` |
|
||||
| `transport` | refused/reset/timeout/authn/host-key | `A távoli tárhely nem érhető el (hálózat vagy bejelentkezés)` |
|
||||
| **`unknown`** | anything else | `A távoli mentés ismeretlen okból nem sikerült` |
|
||||
|
||||
The `unknown` class is deliberate: a cause that cannot be told apart where the error is produced is
|
||||
reported as unknown rather than folded into a neighbour.
|
||||
|
||||
**Secrets — and this caught a bug in my own first attempt.** The old message was
|
||||
`"…: " + err.Error()`, carrying the repo reference `sftp:<user>@<host>:<path>` off the box. My first
|
||||
sanitiser regex-matched `sftp:…` and `user@host` and *looked* complete; its own test caught it leaking
|
||||
on `ssh: connect to host <host> port 23: Connection refused` — a bare hostname in neither shape. It now
|
||||
redacts the target's **actual** host/user/repo-path literally, with the regex kept only as a backstop.
|
||||
Guessing at what a secret looks like fails exactly where it matters.
|
||||
|
||||
## Part 4 — F-OPS
|
||||
|
||||
`documentation/runbooks/RUNBOOK-manual-guest-restore.md`. Grounded in the real bind shape read off live
|
||||
guest 9201, not written from memory. Covers: which `mpN` are storage volumes (restored) versus **host
|
||||
binds** (taken as-is on the target); the `mp9` trap — it embeds the **source** VMID, so restoring to a
|
||||
different VMID can bind **another guest's bootstrap credentials**; strip-and-re-add before first boot;
|
||||
the hookscript check; and a positive pre-start verification that asserts every bind path exists rather
|
||||
than accepting "no error". Docs only, by design.
|
||||
|
||||
---
|
||||
|
||||
## Red-proofs — all observed failing
|
||||
|
||||
| # | red-proof | observed failure |
|
||||
|---|---|---|
|
||||
| A | dialog back on the attempt clock | `the dialog does not name the last SUCCESSFUL copy` |
|
||||
| C | gate the restore on `LastRun` again | `a tier that has NEVER succeeded still offers a restore — the dialog would promise a copy that does not exist` |
|
||||
| D | make the caution unconditional | `a HEALTHY tier shows the failed-attempt caution ("nem sikerült")` |
|
||||
| F | clear the anchor on failure | `a FAILED run wiped the success anchor (round 1) — one bad night would read as 'no copy has ever succeeded'` |
|
||||
| + | raw sanitiser | `the repo reference reached the message ("sftp:" leaked)` |
|
||||
|
||||
**F exercises the real `recordTier2Success` → `recordTier2Failure` sequence**, not a modelled copy — the
|
||||
R-100 lesson. The Scenario A/C/D tests **render the production template tree** and assert on the string
|
||||
the customer reads; a test asserting a template variable would prove nothing about wording, which is
|
||||
the defect.
|
||||
|
||||
`go build`, `go vet ./...`, `go test ./...` — 27 packages, `rc=0`; `template_id_gate.py` and
|
||||
`emoji_gate.py` both OK. Run separately from every commit.
|
||||
|
||||
---
|
||||
|
||||
## LIVE on demo-hp — the rendered dialog, which is the deliverable
|
||||
|
||||
**Legacy state** (before any run under v0.182.0) — today's wording, no fright:
|
||||
```
|
||||
Legutóbbi másolat: 2026-07-28 03:30
|
||||
```
|
||||
|
||||
Failure induced genuinely: the Tier-2 destination directory was **moved aside** and replaced by a file,
|
||||
so `mkdir …/recovery-unit` fails. (`chmod` does not work — the controller runs as root, which bypasses
|
||||
permission bits; `chattr +i` is refused, the unprivileged container lacks `CAP_LINUX_IMMUTABLE`. Both
|
||||
were tried and reported rather than glossed.) The real data was only ever moved, never deleted.
|
||||
|
||||
```
|
||||
status = error
|
||||
last_run = 2026-07-28T14:42:18Z ← ADVANCED
|
||||
last_success = 2026-07-28T14:40:55Z ← HELD
|
||||
last_error = mkdir …/paperless-ngx/recovery-unit: …
|
||||
```
|
||||
|
||||
**The rendered dialog, failed state:**
|
||||
```
|
||||
Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
|
||||
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:40.
|
||||
Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek
|
||||
lehetnek.
|
||||
```
|
||||
|
||||
**The rendered dialog, healthy state** (after restoring the destination and a successful run) — no
|
||||
caution, no tonal change:
|
||||
```
|
||||
Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
|
||||
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:43.
|
||||
```
|
||||
|
||||
Card lines: `Utolsó sikeres: 2 perce` → `Utolsó sikeres: most`.
|
||||
|
||||
**Everything restored:** destination is a directory again, 86 MB intact, mode 755, `.r101-aside` gone,
|
||||
`status=ok`, `last_success=2026-07-28T14:43:23Z`.
|
||||
|
||||
**demo-felhom is the untouched control:** all 5 rows still `tracked=None` after the deploy, rendering
|
||||
today's way, 15/15 containers up. Scenario E holding across a whole box nobody ran.
|
||||
|
||||
---
|
||||
|
||||
## NOT yet live-validated (carried forward)
|
||||
- **F-DIAG's classes** — unit-proven only. No live offsite failure of each class was induced; the
|
||||
`transport` class is the only one this arc exercised indirectly.
|
||||
- **Scenario C live** — the never-succeeded rendering is unit-proven; no fleet row is in that state
|
||||
(every row either migrated or has a real success), and manufacturing one would mean breaking a
|
||||
customer app's only Tier-2 history.
|
||||
- **The Tier-2 restore itself** was not executed — this arc changed what the dialog *says*, not what the
|
||||
restore does.
|
||||
- **R-100's 48h staleness threshold** — injected clock only.
|
||||
- **Fault 4** (restic transport interruption), **R-99**, **F-HUB**, fault 12, the three-way concurrency
|
||||
overlap — next campaign's material, untouched here.
|
||||
@@ -1,136 +0,0 @@
|
||||
# REPORT — R-106 + R-109 (+ R-122): closing the recipe-completeness set (2026-07-30)
|
||||
|
||||
Non-overwritten sibling per `CLAUDE.md:82-87` — the shared `REPORT.md` holds R-117 and is not touched.
|
||||
|
||||
Shipped: **agent v0.118.0 → v0.118.1** (`felhom-agent` `1c8a67e`, `6b5dade`) + **hub v0.83.0**
|
||||
(`felhom.eu` `acfc2b7`). Neither half is useful alone.
|
||||
|
||||
**Read §3 first if you read nothing else:** v0.118.0's R-106 half shipped INERT and live validation is
|
||||
what caught it — the recipe still said `"root"`, now with `namespace_state: resolved` beside it. Full
|
||||
account in the audit §6, filed as **R-125**.
|
||||
|
||||
## Part 0 — the answers, before the fix
|
||||
|
||||
### 0.1 Which items are actually open, and R-105/R-106's registration
|
||||
|
||||
`OPEN-ITEMS.md` calls itself "the single source of truth for open work" (`:1`), with `ROADMAP.md` keeping
|
||||
"the full history and reasoning" (`:3-4`).
|
||||
|
||||
| item | `ROADMAP.md` | `OPEN-ITEMS.md` | verdict |
|
||||
|---|---|---|---|
|
||||
| R-105 | row, `READY — 2026-07-28` (`:108`) | **absent** | **open but UNREGISTERED** |
|
||||
| R-106 | row, `READY — 2026-07-28` (`:109`) | **absent** | **open but UNREGISTERED** |
|
||||
| R-108 | row (`:111`) | row (`:50`) | registered |
|
||||
| R-109 | row (`:112`) | row (`:61`) | registered |
|
||||
|
||||
So R-109's own cell — "third recipe-completeness defect beside R-105/R-106" — was the **only** place in the
|
||||
register naming two open items. That is exactly the thread-loss the register exists to prevent, and it is
|
||||
itself a finding (filed **R-123**). Both now have rows.
|
||||
|
||||
**The set this task closes is R-106 + R-109**, matching the arc's stated definition of done (`OPEN-ITEMS.md:14`).
|
||||
**R-105 is NOT in it** and was not worked: it is M-sized and is about three *hub-held DR records* being `{}`
|
||||
(`hosts.dr_record_json`, `host_escrow.directive_json`, and the `drives` third — already traced and populated
|
||||
by the 2026-07-28 target move). Different fields, different owner, different size.
|
||||
|
||||
### 0.2 Where the recipe is generated — three producers, not two
|
||||
|
||||
| half | repo | function |
|
||||
|---|---|---|
|
||||
| host (guests/pbs/drives/pve_storage) | `felhom-agent` | `BuildDRRecipeHostHalf`, `internal/hub/dr_recipe.go:86` |
|
||||
| app (customer/apps/offsite_restic) | `felhom-controller` | `controller/internal/report/dr_recipe.go` |
|
||||
| **assembly + delivery** | `felhom.eu/hub` | `AssembleDRRecipe`, `internal/store/dr_recipe.go:104`; served by `handleDRRecipeDownload`, `internal/web/dr_recipe.go:14`, route `internal/web/server.go:439` |
|
||||
|
||||
R-109's "host-half" is therefore the **agent**, and the field must also pass the **hub's** allow-list — see §2.
|
||||
|
||||
### 0.3 What the namespace field actually contained — verified, and the brief was RIGHT
|
||||
|
||||
The eleven-session-old brief held up. Live, pre-fix, from the hub for **both** boxes:
|
||||
|
||||
```json
|
||||
"pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }
|
||||
```
|
||||
|
||||
against `/etc/pve/storage.cfg` on the same boxes:
|
||||
|
||||
```
|
||||
pbs: felhom-pbs
|
||||
datastore felhom-offsite
|
||||
namespace demo-felhom # demo-hp reads: namespace demo-hp
|
||||
```
|
||||
|
||||
Traced to source: `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo
|
||||
per item once the list is namespace-scoped via `?ns=` (`:118-120`) → always empty → `ToHub` normalises empty
|
||||
to `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it in.
|
||||
|
||||
**The authority taken, and why:** storage.cfg's `namespace` on the pbs storage. It is the same field
|
||||
`vzdump --storage <pbs>` makes PVE read, and the agent's own verify client is built from it
|
||||
(`cmd/felhom-agent/main.go:1164`). Deriving the recipe from anything else is how it drifts again.
|
||||
|
||||
## 1. R-109's ambiguity is real, in the boxes' own pre-fix recipe
|
||||
|
||||
```json
|
||||
"pve_storage": [
|
||||
{ "name": "local-lvm", "type": "lvmthin", "content": "images,rootdir" },
|
||||
{ "name": "felhom-backup", "type": "local-dir", "content": "backup" },
|
||||
{ "name": "felhom-pbs", "type": "pbs", "content": "backup" },
|
||||
{ "name": "local", "type": "local", "content": "backup,import,vztmpl,iso" }
|
||||
]
|
||||
```
|
||||
|
||||
No `backup_target` key anywhere. `felhom-backup` (live, `/mnt/hdd_1`) and `local` (`/var/lib/vz`, archives
|
||||
frozen 2026-07-28) are both `content=backup` dir storages; `local` is also the *historically* correct answer,
|
||||
which is what makes guessing it so easy.
|
||||
|
||||
## 2. R-122 — a fourth defect, found here, and it had already shipped
|
||||
|
||||
`AssembleDRRecipe`'s `hostHalfShape`/`appHalfShape` are **allow-lists** dressed as forward-compat. The
|
||||
controller has emitted `offsite_restic` since fork-4 (`controller/internal/report/dr_recipe.go:39-41`, "so DR
|
||||
knows WHERE to recover from"); `appHalfShape` never listed the key. Verified both ways:
|
||||
|
||||
- **stored**: `dr_recipe.app_half_json` carries it for all three real customers —
|
||||
`peti-felhom`, `demo-felhom` (`u629488-sub1.your-storagebox.de:23/home/felhom-repo`), `demo-hp`.
|
||||
- **delivered**: the downloaded recipe's top-level keys were
|
||||
`recipe_version, customer, guests, pbs, drives, pve_storage, apps` — **no `offsite_restic`**.
|
||||
|
||||
So a restorer reading the recipe had **no offsite location at all**, for the whole life of the feature, with
|
||||
a green suite throughout — because the test fixture `drAppHalf` is hand-written and omits the field.
|
||||
|
||||
**Deviation from the task's §7.10 ("Findings — filed as R-n, none fixed"), stated rather than absorbed:**
|
||||
I fixed it. Reasons — (a) Part 0 authorises working the real set if it differs; (b) it is the same
|
||||
symptom the task is named for (the recipe is incomplete), and the worst instance, a whole section missing;
|
||||
(c) it is in the *same two structs* R-109 forced me to edit, and leaving one of three known keys off a
|
||||
drop-list I was already correcting would be indefensible. It is filed as R-122 with a SHIPPED disposition.
|
||||
|
||||
## 3. The before/after recipe — both boxes, quoted
|
||||
|
||||
```
|
||||
demo-felhom BEFORE "namespace":"root" backup_target absent offsite_restic absent
|
||||
AFTER "namespace":"demo-felhom" backup_target {resolved, felhom-backup, /mnt/hdd_1}
|
||||
offsite_restic {u629488-sub1…}
|
||||
demo-hp BEFORE "namespace":"root" backup_target absent offsite_restic absent
|
||||
AFTER "namespace":"demo-hp" backup_target {resolved, felhom-backup, /mnt/nvme-1tb}
|
||||
offsite_restic {u629488-sub3…}
|
||||
```
|
||||
|
||||
The two boxes DISAGREEING is the point — nothing is hardcoded. And the ambiguity was not theoretical:
|
||||
on both boxes `felhom-backup` holds an archive from **07-30 04:36** while `local` stops at
|
||||
**07-28 17:5x**, frozen at the target-move date. The recipe now names the live one.
|
||||
|
||||
Full evidence, all seven red-proofs and the publish observables:
|
||||
`documentation/audits/R106-R109-recipe-completeness-2026-07-30.md`.
|
||||
|
||||
## 4. Findings filed (none of them fixed except R-122, see §2)
|
||||
|
||||
| id | finding |
|
||||
|---|---|
|
||||
| **R-122** | `AssembleDRRecipe` allow-list dropped `offsite_restic` for the feature's whole life — **FIXED here**, hub v0.83.0 |
|
||||
| **R-123** | R-105 and R-106 were `READY` in `ROADMAP.md` with no `OPEN-ITEMS.md` row — referenced only inside R-109's prose. Registered here |
|
||||
| **R-125** | v0.118.0 shipped an inert R-106 because the "production path" test injected `fakeObserver` one layer below the break — **FIXED** in v0.118.1; filed for the doctrine point (name the seam you inject at) |
|
||||
| **R-124** | The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""` and there is no namespace literally named `root` — a restorer pasting it into `pct restore --ns root` would fail. Pre-existing wire convention, deliberately unchanged; documented at `PBSRootNamespace` |
|
||||
|
||||
## 5. Not done, and why
|
||||
|
||||
- **R-105, R-108, D5** — out of scope by the task's §6. R-108 blocks D5; starting either would leave both half-done.
|
||||
- **The backup machinery** — untouched. This corrects the record, not the doing.
|
||||
- **`sess-f` (0.116.0) and `drill-r50` (0.113.0) were not upgraded** — neither was named as a venue, and `drill-r50` is fenced by the task's §6.
|
||||
- **R-124 not fixed** — changing the wire's spelling of the root namespace mid-R-106 would shift the field's meaning during the fix meant to make it trustworthy.
|
||||
@@ -1,117 +0,0 @@
|
||||
# REPORT-r116-diag — the `/disks` payload captured, R-116's mechanism isolated (2026-07-30)
|
||||
|
||||
Read-only diagnosis run by CC on DooPlex. **No code written, nothing built, nothing published.**
|
||||
Full evidence: `documentation/audits/DIAG-r116-disks-payload-2026-07-30.md`.
|
||||
|
||||
A `REPORT-*.md` sibling, not the shared `REPORT.md` (`CLAUDE.md` parallel-session rule).
|
||||
|
||||
## Outcome
|
||||
|
||||
**Both goals met.** The `/disks` read path is solved and written down verbatim, proven by a
|
||||
present-drive control run *first*; and the absent-state payload was captured, which isolates the
|
||||
mechanism.
|
||||
|
||||
**R-116 is theory #1 — "the registry-union row writes `false`" — the theory that was raised, declared
|
||||
wrong, and retracted. The retraction was the error.**
|
||||
|
||||
In the absent state `/disks` returns **4 rows, not 3**. The drive appears twice and the two facts the
|
||||
controller needs sit on different rows:
|
||||
|
||||
| row | source | `mount_path` | `guest_path` | `backup_target` |
|
||||
|---|---|---|---|---|
|
||||
| `felhom-backup` | Observe (`disks.go:196-284`) | `""` | `""` | **`true`** |
|
||||
| `694034cc-…` (the UUID) | registry union (`disks.go:297-339`) | `/mnt/cel` | `/mnt/felhom-drives/cel` | **field ABSENT ⇒ `false`** |
|
||||
|
||||
So the row holding the flag contributes **no key** to `driveTargetByPath`, and the row that owns the key
|
||||
says `false` → `isTarget[a.Path]` is `false` → generic `storage_disconnected`. On return the rows
|
||||
re-merge into one carrying both facts → specific `backup_target_restored`. Applying
|
||||
`intermediary.go:602-618` to the captured payloads gives PRESENT `True` / ABSENT `False` /
|
||||
RETURNED `True` — **the live asymmetry reproduced from payload alone.**
|
||||
|
||||
The union row's `MountPath` survives the device because the union source is the systemd **`.mount` unit
|
||||
file** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never reads the mount table. The
|
||||
dedup at `:298` therefore does not fire, because `seen` is keyed on the one field the absent state
|
||||
empties (`:290-295`).
|
||||
|
||||
**Theory #2 (the basis of the shipped v0.115.0) is false on both halves**; **#3 is false too**
|
||||
(`isTarget["/mnt/cel"]` is `false` as well). **v0.115.0 is provably inert** — its fallback calls
|
||||
`StablePathForRaw("")`, which returns `""` (`intermediary.go:69-75`), so it assigns nothing.
|
||||
|
||||
## The read path (this cost two prior sessions — it should never cost again)
|
||||
|
||||
The token plaintext exists in exactly one place: `bootstrap.json` **on the Proxmox host**. The agent's
|
||||
own store keeps SHA-256 hashes only (`tokenstore.go:26-32`), which is what defeated the earlier attempts.
|
||||
|
||||
```bash
|
||||
ssh felhom-pve
|
||||
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
|
||||
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
|
||||
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])")
|
||||
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
|
||||
```
|
||||
|
||||
Control run, live felhom-pve, drive present: **HTTP 200, 2483 bytes, 4 plausible rows** — so Part 5's
|
||||
0-rows-on-a-present-drive failure mode is excluded.
|
||||
|
||||
## Where the absent state was staged
|
||||
|
||||
No new box. The existing DooPlex **nested-PVE drill fixture** (`drill/drill.qcow2`, snapshot `virgin`) —
|
||||
my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with
|
||||
the **byte-identical live agent binary** (`sha256 f48544ad…`, `--version` 0.115.0) and every
|
||||
state-producing step through the real endpoints (`format` → `assign` → `guest-attach` →
|
||||
`backup/target`). Its present-state row matched felhom-pve's control run field-for-field before it was
|
||||
trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token
|
||||
record, no controller) are enumerated in the audit §4.
|
||||
|
||||
## Two new findings, filed not chased
|
||||
|
||||
- **R-117 (READY M) — outranks R-116.** After a detach/reattach the guest's bind is a **dead mount**:
|
||||
host is healthy on the new device node, guest still names the old one, and `ls`/write through it
|
||||
return **`EIO`** — while `/disks` reports `attached` + `bound_under_parent:true` + `backup_target:true`.
|
||||
`planDriveGates` therefore takes the `Return` branch and **restarts the customer's apps onto a dead
|
||||
namespace, reporting healthy, with no alarm on any channel.** R-113's conjunction cannot catch it:
|
||||
one half is satisfied by the stale entry, the other by the new device, and neither compares them.
|
||||
This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs.
|
||||
- **R-118 (READY XS).** An absent drive's union row reports the **root filesystem's** capacity as its own
|
||||
(46 GiB / 9.2 % for a 4 GB drive) — `statfsCapacity` at `disks.go:335-338` statfs's a bare directory on
|
||||
root. `observe.go:176-183` guards the Observe path against exactly this; the union path does not.
|
||||
`durable_id` is still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id.
|
||||
|
||||
## Register
|
||||
|
||||
`documentation/backlog/OPEN-ITEMS.md` — R-116 updated with the mechanism and the fix constraints;
|
||||
R-117 and R-118 added. The single register edit this session makes.
|
||||
|
||||
## Record correction
|
||||
|
||||
The brief's baseline `controller 0.185.1` is the version the **golden bakes**. **0.186.0** (R-114 +
|
||||
R-112, 2026-07-29) is what **demo-felhom** runs — **demo-hp is still on 0.185.1**, so the fleet is
|
||||
split, and R-114's `TargetAbsent` branch exists only on demo-felhom. Confirmed: agent **0.115.0**
|
||||
(felhom-pve) / **0.113.0** (demo-hp), hub **0.81.0** (manifest pin and live pod image agree),
|
||||
host-install **1.22.0**, `felhom.eu` HEAD `c3ce4c7`.
|
||||
|
||||
> **Correction, 2026-07-30.** As first written this section said 0.186.0 was what *both* demo boxes run.
|
||||
> That was wrong — only felhom-pve's guest was sampled and the result generalised to the fleet. demo-hp
|
||||
> re-checked directly → `0.185.1`. Fixed here and in the audit's baseline table.
|
||||
|
||||
## Teardown and fences
|
||||
|
||||
Drill guest destroyed, scratch storage removed, mount unit deleted, secrets `shred -u`'d, VM powered
|
||||
off, **`drill.qcow2` restored to `virgin`** (the golden-bake fixture is exactly as found), scratch qcow2
|
||||
and console dumps deleted. DooPlex `/mnt/5_hdd` at 24 %, unchanged.
|
||||
|
||||
Both demo boxes **read-only throughout** and re-verified after teardown: demo-hp `local-lvm` **38.83 %**
|
||||
(identical before/after and to Part 5), `drill-r50` still stopped, felhom-pve `felhom-backup` still
|
||||
active on `/dev/sdb`, both guests running, **v0.115.0 untouched**.
|
||||
|
||||
`sess-d-0452c4` now reads **STALE**, and the delete gate refuses only on ONLINE
|
||||
(`hub/internal/web/customer_delete.go:220-228`) — so it **is** now deletable; the command is recorded in
|
||||
the audit rather than executed (customer delete runs external teardown plus a DB purge). **`sess-c` is
|
||||
also still present and was not recorded by the Session-C audit** — same terms.
|
||||
|
||||
## Not done, deliberately
|
||||
|
||||
No fresh controller gate-log / hub-event correlation: that observable was already captured live and
|
||||
identically twice, and only the payload was missing. Staging a controller meant a hub customer, a
|
||||
pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8
|
||||
correlates the payload to those existing measurements and labels that step as inference.
|
||||
@@ -1,96 +0,0 @@
|
||||
# REPORT — SPIKE R-117: a dead bind that reports healthy (2026-07-30)
|
||||
|
||||
Written as `REPORT-<topic>.md`, not `REPORT.md`, per this repo's parallel-session rule and the
|
||||
established local pattern (`REPORT-r116-diag.md`, `REPORT-session-c.md`, …). The shared `REPORT.md`
|
||||
was not touched.
|
||||
|
||||
**Class: Spike.** Deliverable is a findings document. **No production code was written; no `.go` file
|
||||
in either repo was modified; nothing was built for deployment, published, or version-bumped.**
|
||||
|
||||
## Deliverables
|
||||
|
||||
| File | Action |
|
||||
|---|---|
|
||||
| `documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md` | **new** — the findings doc (Q1–Q7, evidence, recipe, probe comparison, recommendation) |
|
||||
| `documentation/backlog/OPEN-ITEMS.md` | **R-117 row rewritten** with the mechanism, the reproduction recipe and the fix constraint — the one register edit, per the brief |
|
||||
|
||||
**No CHANGELOG entry.** This repo's changelogs are per-area (`hub/`, `scripts/`, `website/`); a
|
||||
documentation-only change belongs to none of them. Stated rather than silently skipped, per standing
|
||||
rule 4.
|
||||
|
||||
## Baselines
|
||||
|
||||
agent `main` **v0.116.0** @ `d4eb259` · controller `main` **v0.186.0** @ `b331f18` ·
|
||||
`felhom.eu` HEAD `29bcfeb` · hub **live 0.82.0** · golden **0.186.0** ·
|
||||
demo-hp PVE 9.2.2 / kernel 7.0.2-6-pve, **live agent 0.113.0** (= manifest `MinAgent`; never used as
|
||||
the source of a behavioural claim — every predicate result came from a probe built from `main`).
|
||||
|
||||
## Results
|
||||
|
||||
**All seven questions answered empirically.**
|
||||
|
||||
- **Q1 — reproduced 3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp.
|
||||
**The device-node change is a consequence of the defect, not a precondition** — control test: with
|
||||
the stale bind held the drive returns as `sdc` (8:32); released, the letter is reused (`sdb`, 8:16).
|
||||
- **Q2 — two death states**: device removed ⇒ superopts gain `shutdown`, `EIO`(5) on read and write,
|
||||
host and guest; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads served from cache.
|
||||
The raw host mount is genuinely healthy in both. **No cross-device mis-identification is possible**
|
||||
on this path — the unit is fs-UUID-keyed.
|
||||
- **Q3 — confirmed at source and live.** Both halves of the R-113 conjunction compare **field 5** of a
|
||||
mountinfo line and **never read field 3 (`major:minor`)**, so neither can see that the bind and the
|
||||
raw mount name different devices. Measured `BoundUnderParent = TRUE` over an `EIO` namespace.
|
||||
- **Q4 — a pure-`/proc` check costs 0.16–0.45 ms**, cannot hang, spins up no disk, writes nothing.
|
||||
**`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow. The
|
||||
hang case is below; it is the sharpest result in the run.
|
||||
- **Q5 — the agent**, and not on balance: the controller runs inside the guest and cannot see the host
|
||||
mount tables the check needs.
|
||||
- **Q6 — recovery works in place, guest never restarted** (init PID identical). **The repair code
|
||||
already exists and three call sites already invoke it**, including the controller's `Return` branch
|
||||
*before* it restarts apps — all defeated by one early return.
|
||||
- **Q7 — a bind can die in steady state, no cycle at all.** The gate produces no action and **nothing
|
||||
is emitted on any channel.** A `Return`-branch fix cannot reach this half.
|
||||
|
||||
## Q4's hang case — measured, and it is the sharpest result
|
||||
|
||||
Against a `dmsetup suspend`ed device (I/O queues instead of returning `EIO`):
|
||||
|
||||
- **P1 and P2 completed in 364 µs / 206 µs.** They read `/proc`, so no block device is involved.
|
||||
- **`statfs` and `getdents` completed and reported HEALTHY** — on a wedged device they do not even hang.
|
||||
- **Every probe that touches the device blocked, including a buffered write with no `fsync`** — the
|
||||
`O_CREAT` metadata path needs journal access (`wchan=do_get_write_access`). There is no cheap-and-safe
|
||||
write probe.
|
||||
- **The blocked process survived `SIGTERM` and `SIGKILL`** (`stat=D`, still alive 3m50s after `kill -9`)
|
||||
and died only when the device was resumed. So **`systemctl restart felhom-agent` would hang**, leaving
|
||||
the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal
|
||||
the leak (5→5, 5→6).
|
||||
|
||||
**A timeout protects the caller's control flow and nothing else.** This turns "prefer a cheap probe" into
|
||||
a fence: **the fix must issue no block I/O.**
|
||||
|
||||
## Teardown — done, all three layers
|
||||
|
||||
Guest 9301 destroyed; `r117scratch` removed; both dm devices and both loop devices gone; `scsi_debug`
|
||||
unloaded (`/dev/sd*` back to `sda1..3`); no `r117` mounts, `/mnt` and `/root` exactly as found; `local`
|
||||
**37.02 %** against a session-start **37.00 %**. Fences re-verified *after* teardown: 9201 `running`,
|
||||
`drill-r50` `stopped`, `local-lvm` **38.84 % byte-identical**, `felhom-backup` `content backup`
|
||||
unchanged, live `/mnt/felhom-drives` intact with both submounts, agent service `active`. **Layer 3 is
|
||||
genuinely empty** — 9301 had no network interface and ran no controller, so no hub-side record was ever
|
||||
created.
|
||||
|
||||
**Ordering trap worth keeping:** a suspended dm device must be `dmsetup resume`d *before* any `umount`,
|
||||
or the teardown itself blocks on the same uninterruptible sleep.
|
||||
|
||||
## Not measured
|
||||
|
||||
No load or duration testing of the recommended check — P1/P2 were single calls, not a sustained
|
||||
reconcile loop on a many-drive box. Nothing suggests a problem (they are two `/proc` reads the code
|
||||
already performs), but it was not measured.
|
||||
|
||||
## Findings filed, none fixed
|
||||
|
||||
R-117 (mechanism + recipe), **R-117a** steady-state death with no event (HIGH, larger than R-117 as
|
||||
filed), **R-117b** `statfs`/`getdents` are hollow liveness probes, **R-117c** three untested comments
|
||||
promising "live + usable in the guest", **R-117d** the self-heal that already exists is short-circuited
|
||||
(HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing internet,
|
||||
including the WireGuard OOB path, **R-117f** an I/O liveness probe turns a wedged drive into an
|
||||
unkillable agent (HIGH — disqualifies a whole probe class).
|
||||
@@ -1,160 +0,0 @@
|
||||
# REPORT — hub v0.88.0: the WAL that never was (R-172), plus R-173 found
|
||||
|
||||
**Session artefact naming:** written as `REPORT-r172-hub-wal.md`, not `REPORT.md`, per this repo's
|
||||
parallel-session rule — the shared `REPORT.md` belongs to the controller boot-recovery session that
|
||||
ran immediately before this one and must not be clobbered.
|
||||
|
||||
**Repo:** `felhom.eu` (hub `v0.87.0` → **`v0.88.0`**) · **Trigger:** a `HOST STALE` banner the
|
||||
operator spotted on `hub.felhom.eu` after the previous session finished.
|
||||
|
||||
---
|
||||
|
||||
## 1. What the alarm actually was
|
||||
|
||||
**Not the agent, not the guest.** The agent was up **2 days**, never restarted, and actively
|
||||
reconciling; the controller was reporting normally (the header read "Last report just now", 0.190.0,
|
||||
10/10 containers). The failure was the hub **writing** the host report:
|
||||
|
||||
```
|
||||
20:26:34 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY)
|
||||
20:41:32 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY)
|
||||
20:42:32 [INFO] Host staleness: demo-felhom-8363b5 ok → stale (host_stale)
|
||||
20:42:33 [INFO] Operator email sent for demo-felhom/host_stale
|
||||
```
|
||||
|
||||
The chain, and the margin is exactly one retry wide: reports are every **15 min**, staleness fires at
|
||||
**30 min**, the hub returns **500** on `SQLITE_BUSY` without retrying, and the agent logs
|
||||
`keeping current interval` and waits a full interval without retrying either. **Two consecutive
|
||||
collisions = a false alarm.** It had already fired once that day (19:12:32, recovered 19:20:32).
|
||||
|
||||
**Was it caused by the preceding session?** Partly amplified, not caused. 13 collisions in one pod
|
||||
lifetime; **the first at 15:56 CEST, ~3 h before that session's first deploy**. 7 of 13 fell inside
|
||||
its window of ~13 controller restarts, which raises write concurrency — so the burst made a
|
||||
pre-existing fault more likely, and the fault was not new.
|
||||
|
||||
## 2. Root cause — the pragmas were never applied
|
||||
|
||||
The DSN was `?_journal_mode=WAL&_busy_timeout=5000`. That is **mattn/go-sqlite3** syntax. The driver
|
||||
is **modernc.org/sqlite v1.45.0**, whose `applyQueryParams` (confirmed at source in the module cache)
|
||||
reads only `_pragma`, `_time_format`, `_time_integer_format`, `_txlock` and `_inttotime` — and
|
||||
**silently ignores everything else**. No error, no warning.
|
||||
|
||||
So the hub ran in the default **rollback-journal** mode with **`busy_timeout=0`** for its entire life
|
||||
while its own source said WAL. In rollback-journal mode a reader excludes a writer, so rendering an
|
||||
operator page can block a host report — which is precisely the observed 500.
|
||||
|
||||
**The observable that proved it before any code changed:** a 128 MB `/data/hub.db` with **no
|
||||
`-wal`/`-shm` file beside it while the database was open**. In WAL mode those must exist.
|
||||
|
||||
This is the project's recurring class — a configuration asserting an invariant the code does not
|
||||
provide — and it is the second one this week.
|
||||
|
||||
## 3. The fix
|
||||
|
||||
```
|
||||
?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate
|
||||
```
|
||||
|
||||
| Parameter | Why it is not optional |
|
||||
|---|---|
|
||||
| `journal_mode(WAL)` | readers and one writer proceed concurrently, so a page render can no longer block a report; it is a property of the database FILE and persists once set |
|
||||
| `busy_timeout(5000)` | writers still serialise; without a timeout SQLite returns `SQLITE_BUSY` *immediately* rather than waiting |
|
||||
| `_txlock=immediate` | **the one that is easy to miss.** `database/sql`'s `Begin()` is DEFERRED, so a read-then-write transaction must upgrade its lock, and a failed upgrade is `SQLITE_BUSY_SNAPSHOT` — which **`busy_timeout` does not retry**. This store has **10+ `db.Begin()` sites and they are all write paths** (customer delete/reset, wg, appliance, pbsdr, telemetry, log bundles). WAL + busy_timeout alone would have shipped half a fix with a known un-retryable path left open |
|
||||
|
||||
**Retry options (b) and (c) from R-172 were deliberately NOT taken.** With readers no longer blocking
|
||||
writers and the upgrade path covered, a `SQLITE_BUSY` reaching a handler should now be rare enough to
|
||||
be a real signal; a retry would hide it. Revisit only on evidence.
|
||||
|
||||
## 4. Tests and the red-proof
|
||||
|
||||
**Every assertion reads the value back from the DATABASE, never the DSN string** — a string assertion
|
||||
would have passed happily for the entire life of the bug. Six tests in `internal/store/pragma_test.go`:
|
||||
|
||||
| Test | Asserts |
|
||||
|---|---|
|
||||
| `TestStorePragmasAreActuallyApplied` | runtime `journal_mode` = wal, `busy_timeout` ≥ 5000 |
|
||||
| `TestStoreWALFilesExistWhileOpen` | `-wal`/`-shm` exist beside an open DB — **the production signature, pinned** |
|
||||
| `TestStoreReaderDoesNotBlockWriter` | the CONSEQUENCE: a write during a held read succeeds |
|
||||
| `TestStoreConcurrentWritersDoNotReturnBusy` | 8 concurrent writers all wait rather than error |
|
||||
| `TestStoreTransactionUpgradeDoesNotReturnBusySnapshot` | 6 racing read-then-write transactions all commit |
|
||||
| `TestSQLiteDriverIgnoresMattnStyleParams` | guards the ROOT CAUSE: fails if the pragmas are "tidied" back to mattn form; skips itself with instructions if a future driver starts honouring them |
|
||||
|
||||
**Red-proof — restore the DSN that shipped.** Observed FAIL, then reverted with a passing control:
|
||||
|
||||
```
|
||||
journal_mode = "delete", want "wal"
|
||||
hub.db-wal is missing beside an OPEN database
|
||||
a write FAILED while a read was open: database is locked (5) (SQLITE_BUSY) ← the live error, exactly
|
||||
```
|
||||
|
||||
`go build ./... && go vet ./... && go test ./...` in `hub/` → **rc=0**. `scripts/repo_gates.py --fast`
|
||||
→ all 5 gates OK.
|
||||
|
||||
## 5. Operational consequence — handled, not discovered later
|
||||
|
||||
**A WAL database cannot be copied by taking `hub.db` alone.** A committed transaction may still live
|
||||
in `hub.db-wal`, so a bare `cat` yields a copy that **opens cleanly and silently omits the newest
|
||||
writes** — the worst possible shape for a credential lookup. The break-glass root-password retrieval
|
||||
in `documentation/operations/nodes.md` used exactly that command, and `_recovery-inventory-2026-07-28.md`
|
||||
records it as a past action that reads like a recipe. Both are now WAL-aware: copy the `-wal`
|
||||
alongside, `|| true` because an absent `-wal` is legitimate, and **shred both** (the WAL holds the
|
||||
same secrets).
|
||||
|
||||
Not hypothetical: the live `-wal` measured **729,272 bytes** during verification, all of which a bare
|
||||
`cat` would have dropped.
|
||||
|
||||
## 6. Live verification
|
||||
|
||||
- Image built and pushed; **`manifests/hub.yaml` 0.87.0 → 0.88.0** (the only thing ArgoCD deploys
|
||||
from), hard-refresh + deliberate sync (auto-sync is OFF). ArgoCD **Synced / Healthy**.
|
||||
- `/data/` now shows **`hub.db-wal` and `hub.db-shm`** beside the open DB — the exact observable whose
|
||||
absence proved the bug.
|
||||
- **Zero `SQLITE_BUSY` since the rollout.**
|
||||
- Host report landed at 21:11:33; staleness checker reports `2 ok, 0 stale` — `demo-felhom-8363b5`
|
||||
is back to `ok`.
|
||||
- `PRAGMA integrity_check` → **`ok`**, `journal_mode` → **`wal`**, all tables intact
|
||||
(hosts 4, customer_configs 6, host_recovery 4, host_reports 2756, events 2293). Read via the new
|
||||
WAL-aware copy recipe and shredded afterwards.
|
||||
|
||||
## 7. A mistake I made, and it caused a real outage
|
||||
|
||||
**I OOM-killed the hub pod three times** with a 60-concurrent page-render load test intended to prove
|
||||
the fix under contention. The pod's limit is **256 Mi**; 60 simultaneous renders of a heavy customer
|
||||
page exceeded it (`OOMKilled`, exit 137, readiness probe timeouts). The hub was unavailable for parts
|
||||
of ~6 minutes and recovered on its own.
|
||||
|
||||
Three things worth stating plainly:
|
||||
|
||||
1. **It was not the WAL change.** WAL's extra footprint is the 32 KB `-shm` mapping; the OOM was
|
||||
template rendering under concurrency I created.
|
||||
2. **I should not have run it.** DooPlex is **Tier 2 — precious**, and the hub is part of the
|
||||
recovery chain. A synthetic load test at that concurrency against a memory-limited pod on that box
|
||||
was the wrong call; the unit tests already proved the property, and the live proof needed was the
|
||||
`-wal` file plus a clean report — both of which I already had.
|
||||
3. **It did produce one piece of genuine evidence**, which does not excuse it: after three hard kills
|
||||
the WAL replayed cleanly and `integrity_check` returned `ok`, which is a real (if unplanned)
|
||||
demonstration of WAL crash-safety on this volume.
|
||||
|
||||
No data was lost. The pod is `Ready`, restart count 3, serving normally.
|
||||
|
||||
## 8. Backlog
|
||||
|
||||
- **R-172 → CLOSED**, with the root cause recorded as *the pragmas were never applied*, not as tuning.
|
||||
- **R-173 → NEW.** While checking who copies `hub.db` for the WAL change, I found `pvc/hub-data`
|
||||
carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly`
|
||||
are the only recurring jobs and both target `default`. **The hub database has no volume-level
|
||||
backup** — and it holds `host_recovery` (every box's break-glass root password), `host_escrow` +
|
||||
`host_escrow_superseded`, `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg peers.
|
||||
Filed rather than fixed: whether the exclusion is deliberate is a question for the operator, and the
|
||||
manual hot copy recorded in `_recovery-inventory` is not a backup. `grep` established the ID free.
|
||||
|
||||
## 9. Observations — noticed, NOT acted on
|
||||
|
||||
1. **The hub returns HTTP 500 for a transient lock**, which is what turned a retryable condition into
|
||||
an alarm. Left as-is deliberately (§3) so a surviving `SQLITE_BUSY` stays visible.
|
||||
2. **The agent does not retry a failed report**, so one collision costs a full 15-minute interval —
|
||||
half the staleness budget. Same reasoning; if collisions recur, this is the cheaper of the two
|
||||
retry fixes.
|
||||
3. **`store.New` sets no `SetMaxOpenConns`**, so `database/sql` may open unbounded connections against
|
||||
a single-writer database. Not changed here — WAL plus the immediate-lock covers the observed
|
||||
failure, and bounding the pool changes latency characteristics that nothing currently measures.
|
||||
@@ -1,203 +0,0 @@
|
||||
# REPORT — installer-channel record correction + R-29 filing (2026-07-29)
|
||||
|
||||
Two commits, documentation only. No code, no version bump, no CHANGELOG entry, no build, no deploy,
|
||||
no box touched. Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so root `REPORT.md` (the E-2
|
||||
increment-1 report) is preserved.
|
||||
|
||||
| # | Commit | Baseline | Scope |
|
||||
|---|--------|----------|-------|
|
||||
| 1 | `d4c07873ca0c3d3e547373a9fafc0e472a6535e8` | `36d635a4cdc1`, unmoved | Retract a false R-94/E-2d finding; open R-110 |
|
||||
| 2 | (this commit) | `d4c07873`, unmoved | File R-29 to the register; three record defects; this report |
|
||||
|
||||
---
|
||||
|
||||
## Commit 1 — `d4c07873`
|
||||
|
||||
### What was false
|
||||
|
||||
`36d635a4` recorded that `felhom-bootstrap.sh` fetches the installer **from the hub**, that the hub
|
||||
therefore serves 1.19.0, and that a fresh ISO install runs the pre-E-2 installer. All three wrong.
|
||||
The claim had propagated into two `OPEN-ITEMS.md` rows, the ranking rationale, and `ROADMAP.md:149`.
|
||||
|
||||
### Confirmation table — all PASS
|
||||
|
||||
| # | Claim | Read at | Result |
|
||||
|---|-------|---------|--------|
|
||||
| F1 | bootstrap fetches from the **website** | `scripts/iso/felhom-bootstrap.sh:96` | PASS — `INSTALL_URL="${FELHOM_INSTALL_URL:-https://felhom.eu/scripts/felhom-host-install.sh}"` |
|
||||
| F2 | hub-rendered command points at the same URL | `customer_unified.html:563`, `:1262` | PASS — and **three** emission sites, not two: `:563` static, `:1262` JS error branch, **`:1267` JS success branch** |
|
||||
| F3 | website serves `/scripts/` from a git-sync tree tracking `main` | `manifests/webpage.yaml` — nginx `:74-77`, sparse-checkout CM `:211-218`, git-sync `:272-281`, init `:299-307` | PASS — `--branch=main --period=30s --link=current`; sparse-checkout `/website/` + `/scripts/`; `location /scripts/ { root …/current; }`. No image build, no ArgoCD step |
|
||||
| F4 | `hostInstallVersion` selects nothing | `configs.go:28`, `:487`; `render_test.go:219`; `customer_unified.html:494` | PASS — repo-wide grep returns exactly those 4 code sites; all other hits prose. Rendered as a text label |
|
||||
| F5 | every generated flag is parsed by 1.22.0 | generator `customer_unified.html:1206-1239` vs parser `felhom-host-install.sh:1175-1212` | PASS — `--mode --cores --memory --vmid --node --acl-storages --operator-pubkey-file --preserve-state-from --skip-provision --dry-run --preflight-only --allow-new-leaf` (+ `--customer-id`); every one a parser case. **No functional gap** |
|
||||
| F6 | installer is 1.22.0 | `scripts/felhom-host-install.sh:187` | PASS |
|
||||
| F7 | the drift test is hollow | `render_test.go:219-221` | PASS — `strings.Contains(html, hostInstallVersion)` compares the constant to itself; passes at any value |
|
||||
|
||||
### Live command 1 — what the URL actually serves
|
||||
|
||||
```
|
||||
$ curl -fsS https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 '^SCRIPT_VERSION='
|
||||
SCRIPT_VERSION="1.22.0" # the SINGLE version source (F-1): -h, the run banners, and the hub
|
||||
```
|
||||
|
||||
### Live command 2 — the drift gate's real state
|
||||
|
||||
```
|
||||
$ python3 scripts/hostinstall_gates.py; echo "exit=$?"
|
||||
ok: SCRIPT_VERSION=1.22.0
|
||||
ok: header has no version literal
|
||||
FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0 (F-1: bump both together)
|
||||
ok: age is in the installed package set
|
||||
… (six further ok lines) …
|
||||
hostinstall gates: 1 FAILURE(S)
|
||||
exit=1
|
||||
```
|
||||
|
||||
### Phase 0 source read — PAIRING reaches the same installer invocation
|
||||
|
||||
Mode selection `felhom-bootstrap.sh:537-541`: a fresh VM with no baked customer-id calls
|
||||
`run_pairing`. On HTTP 200 from `/api/v1/appliance/poll` the loop writes the hub-delivered
|
||||
`FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it, and calls
|
||||
`run_direct` **in the same invocation** (`:495-499`). `run_direct` is the single site that fetches
|
||||
`$INSTALL_URL` (`:322-330`), builds the args (`:334`) and invokes `bash "$SCRIPT_TMP" "${args[@]}"`
|
||||
(`:343`). The customer it yields is the one the operator bound — claimable. **So the ISO leg is the
|
||||
spine for E-2d**, not an obstacle to it.
|
||||
|
||||
### Rows changed
|
||||
|
||||
| ID | Before | After |
|
||||
|---|---|---|
|
||||
| R-94 (line 14) | `READY — deferred until E-2d`, blocked on E-2d, false ISO/hub framing | `READY (XS)`, blocked on nothing, retracted + re-scoped to three legs |
|
||||
| R-94 (line 16) | duplicate row, `READY #2`, 1.19.0 vs 1.20.0 | **deleted** — merged |
|
||||
| R-110 | did not exist | opened, `WAITING-ON-OPERATOR (S)` |
|
||||
| E-2d | ISO implied as obstacle | Next-action appended; ISO is the spine |
|
||||
| ranked list | 1 R-95 · 2 R-94 (high-consequence) · 3 R-86 · 4 R-87 | 1 R-95 · 2 R-94 **de-ranked** · 3 R-86 · 4 R-87 · 5 R-110 |
|
||||
|
||||
Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
|
||||
`documentation/runbooks/day0-install.md`.
|
||||
|
||||
---
|
||||
|
||||
## Commit 2 — R-29 filing + record hygiene
|
||||
|
||||
### The §1.1 ruling — is R-29 the right home for a non-design-v2 gate? **Yes. Proceeded.**
|
||||
|
||||
R-29's title says *"the design-v2 green gates"*, and `scripts/hostinstall_gates.py` is not one — it
|
||||
comes from drill F-1 (2026-07-12) and postdates the item. Four things decide it anyway:
|
||||
|
||||
1. **R-29's own audit list already spans well beyond design-v2 subject matter.** It names
|
||||
`docker_run_volume_path_gate` (docker mount safety), `offbox_rename_gate`, `app_row_dedup_gate`
|
||||
and `manifest_bearer_gate` (secrets — `runbooks/secrets.md:76`). The title is a misnomer relative
|
||||
to the item's own body.
|
||||
2. **Part (b) — "the systemic half is the real item" in R-29's words — is stated with no
|
||||
design-v2 restriction.** It is about the *enforcement mechanism*: "the gates run only when a human
|
||||
remembers to run them… decide where they run (pre-push hook, `build.sh` step, or a CI job) and make
|
||||
a red gate block the train." That is gate-agnostic and repo-wide.
|
||||
3. **`hub_confirm_gate.py` is already on R-29's list and lives in the same `scripts/` directory** as
|
||||
`hostinstall_gates.py`. Wiring one and not the other would be arbitrary.
|
||||
4. **Identical failure shape, identical genre.** Both self-describe as mechanical grep-assertions
|
||||
(`hostinstall_gates.py:2`, `hub_confirm_gate.py:1-8`); both exist, assert something true, and are
|
||||
invoked by nothing.
|
||||
|
||||
R-29 has already absorbed one independent re-raise without minting an ID (2026-07-18 rehearsal note)
|
||||
and says so explicitly. This is the third. No new ID minted.
|
||||
|
||||
### Orphan-search evidence — `hostinstall_gates.py` and `hub_confirm_gate.py`
|
||||
|
||||
Re-established at `d4c07873`. Pattern `hostinstall_gates\|hub_confirm_gate`.
|
||||
|
||||
| # | Scope | Command | Result |
|
||||
|---|-------|---------|--------|
|
||||
| S1 | `felhom.eu`, all file types | `grep -rn "$PAT" . --exclude-dir=.git` | **19 hits, zero invocations.** All are docstrings (`scripts/hostinstall_gates.py:5`, `scripts/hub_confirm_gate.py:7`), code comments (`hub/internal/web/configs.go:27`, `scripts/felhom-host-install.sh:189`) or prose (`REUSE.md:62`, `CONTEXT.md:540,564`, `hub/CHANGELOG.md:371,1292,1351,1385`, `scripts/CHANGELOG.md:483,524`, 3 files under `documentation/audits/`, `ROADMAP.md:149,158`, `OPEN-ITEMS.md:14`) |
|
||||
| S2 | sibling repos | `grep -rln "$PAT" /mnt/5_hdd/felhom.eu/git --exclude-dir=.git` minus this repo | 3 files, all in `.claude-memory/` (`MEMORY.md`, `drtier-by-default-2026-07-12.md`, `polish-batch-2026-07-13.md`). Notes, not invokers |
|
||||
| S3 | `~/.claude` | `grep -rln "$PAT" /home/kisfenyo/.claude`, and targeted on `settings*.json`, `skills/`, `hooks/` | **Zero hits in settings, skills or hooks** — where an invoker would live. Remaining hits are `file-history/` (Claude Code's own backups of files edited in past sessions) and `paste-cache/` (pasted task specs). Neither is an invocation site |
|
||||
| S4 | git hooks | `ls -1 .git/hooks/ \| grep -v '\.sample$'` | **Empty — every hook is a `.sample` stub** |
|
||||
| S5 | build files | `find . -type f \( -iname Makefile -o -iname '*.mk' -o -iname justfile -o -iname 'Taskfile*' \)` | Only `hub/Makefile`; `grep -n gate hub/Makefile` → **zero occurrences** |
|
||||
| S6 | CI | `find . -type d \( -name .github -o -name .gitea -o -name .woodpecker* -o -name .drone* -o -name .circleci \)` | **Empty — `felhom.eu` has no CI configuration at all** |
|
||||
|
||||
Of the four gates in `scripts/`, only `site_gates.py` is mandated (`CLAUDE.md:153`);
|
||||
`manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`.
|
||||
|
||||
### Rows changed
|
||||
|
||||
| ID | Before | After |
|
||||
|---|---|---|
|
||||
| **R-29** | **absent from `OPEN-ITEMS.md`** (`grep -c` → 0) while present at `ROADMAP.md:158` since before the 2026-07-27 rebuild | **opened, `READY (S for (a) / M for (b))`**, owner CC |
|
||||
| R-94 | leg (b) stood alone | leg (b) carries `→ R-29` as its class. Row otherwise untouched |
|
||||
| E-2d | cited `:322-341` for an invocation at `:343` | cited `:322-343`, with the fetch / args / call each pinned separately |
|
||||
| R-95, R-86, R-87 | `READY #1`, `READY #3`, `READY #4` | `READY` — markers dropped (see below) |
|
||||
|
||||
Not added to the ranked list under *"Why the READY rows rank this way"*: that list is a top-N
|
||||
rationale, not a complete ordering (R-99, R-102, R-103, R-104, R-108, R-109, R-89, R-92, R-93 and
|
||||
E-2d are all READY and absent from it). Ranking R-29 was not asked for and would be a priority claim
|
||||
this task has no basis to make.
|
||||
|
||||
### §2.2 — markers dropped, not renumbered
|
||||
|
||||
The `#1/#3/#4` markers duplicated ranked-list positions 1/3/4 for exactly those three rows; two
|
||||
orderings of one set is what produced the hole when the `#2` row was merged in `d4c07873`. Removing
|
||||
the duplicate leaves the ranked list as the single maintained ordering.
|
||||
|
||||
### The other two record defects
|
||||
|
||||
- `ROADMAP.md:147` — cited a non-existent **R-164**; it means controller **v0.164.0**'s
|
||||
deliberate-stop filter. Corrected. (It had already cost one max-ID scan a false positive.)
|
||||
- `CONTEXT.md:564` — asserted in the present tense that the single VERSION source is *"gated by
|
||||
`scripts/hostinstall_gates.py`"*. The gate exists, asserts exactly that, is red, and runs nowhere.
|
||||
Corrected to say the cross-check exists but is not enforced, tracked as R-94 leg (b) / R-29.
|
||||
`hub/internal/web/configs.go:27` carries the same false claim in a code comment and was left alone
|
||||
— R-94 leg (b) territory, needs a hub build.
|
||||
|
||||
Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
|
||||
`documentation/backlog/README.md`, `CONTEXT.md`, this file.
|
||||
|
||||
---
|
||||
|
||||
## Commit 3 — `de5a3e5` — the record-hygiene rider
|
||||
|
||||
Baseline `7383400a`, unmoved. Four XS items from `7383400a` §9; the three deferred observations stay
|
||||
deferred.
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `CONTEXT.md:540` | *"`scripts/hub_confirm_gate.py` enforces"* → asserts, but is not enforced (R-29). Third instance of the class after `:564` and `configs.go:27` |
|
||||
| `REUSE.md:62` | Same claim, *"enforces zero"*. The **rule stays** — never native `confirm()`/`prompt()` is correct guidance and this is a reuse-reference row — only the enforcement claim changes |
|
||||
| `OPEN-ITEMS.md:4` | Root `REPORT.md` = overwritten per-session; `REPORT-<topic>.md` = non-clobbering sibling (`CLAUDE.md:82-87`), 14 of them. Prohibition unchanged |
|
||||
| `OPEN-ITEMS.md:55` | Heading scoped to *"the **TOP** READY rows"* + a half-sentence that it is deliberately not a full ordering. **No row added to the list** |
|
||||
|
||||
`hub/internal/web/configs.go:27` — the fourth instance — left alone (R-94 leg (b), needs a hub build).
|
||||
|
||||
### Part 3 NOT done — its stated evidence is false
|
||||
|
||||
The spec proposed recording a convention: *"`ROADMAP`'s P3 `Status` column only ever takes `idea` or a
|
||||
`SHIPPED`/`PROVEN-LIVE` variant, and never takes `READY`, `BLOCKED` or `WAITING-ON-OPERATOR`"*, and
|
||||
concluded from that vocabulary split that `ROADMAP` `Status` is **disposition** while `OPEN-ITEMS`
|
||||
`State` is **live actionability**.
|
||||
|
||||
Refuted inside P3 itself (`ROADMAP.md`, lines 113–172):
|
||||
|
||||
| Line | ID | P3 `Status` cell |
|
||||
|---|---|---|
|
||||
| 151 | R-90 | `BLOCKED on Hetzner CX33 availability (operator, 2026-07-27)` |
|
||||
| 152 | R-91 | `WATCHING — gated on demo-felhom's first post-migration PBS backup` |
|
||||
| 150 | R-110 | `idea — found 2026-07-29, **WAITING-ON-OPERATOR (a ruling, not a defect)**` |
|
||||
| 137 | R-86 | `**NEXT — operator ruling 2026-07-27**` |
|
||||
|
||||
And P2's recovery-gaps sub-table (lines 99–107) uses a bare `READY` for nine rows, including `E-2`.
|
||||
|
||||
Both halves fail: the column **does** take the `OPEN-ITEMS` vocabulary, and `BLOCKED`/`WATCHING`/
|
||||
`NEXT` are live-actionability words, not dispositions. The real shape is a free-text status field
|
||||
that has absorbed both vocabularies over time — usually disposition, sometimes not. Writing the
|
||||
proposed convention down would install a **new false invariant** in the commit chain whose whole
|
||||
purpose was removing them, so per the spec's own instruction (*"If you think this ruling is wrong, say
|
||||
so and make no edit"*) `documentation/backlog/README.md` was not touched.
|
||||
|
||||
The narrow question that prompted it still resolves benignly: R-29 and R-94 reading `idea` in
|
||||
`ROADMAP` and `READY` in `OPEN-ITEMS` is **not** a defect and needs no sync. What is unsupported is
|
||||
generalising that into a rule.
|
||||
|
||||
---
|
||||
|
||||
## Standing note
|
||||
|
||||
**No gate was wired, fixed, run as a hook, or deleted by any of the three commits.** Filing R-29 is
|
||||
not doing R-29; the enforcement decision (pre-push hook / `build.sh` step / CI) and the audit of the
|
||||
remaining eight gates are R-29 part (b), M-sized, and its own task.
|
||||
@@ -1,94 +0,0 @@
|
||||
# REPORT — Session C: R-113, R-114, R-112 proven on a real box; C5 fails on a new defect (2026-07-29)
|
||||
|
||||
`RUNBOOK-session-c-2026-07-29.md`. Full evidence: `documentation/audits/SESSION-C-2026-07-29.md`.
|
||||
Root `REPORT.md` untouched.
|
||||
|
||||
## Verdicts
|
||||
|
||||
| Claim | Fix | Verdict |
|
||||
|---|---|---|
|
||||
| **C4** — offer appears and moves the target | R-112 | ✅ **PASS** |
|
||||
| **C3** — absent target tells the truth, offers nothing | R-114 | ✅ **PASS** |
|
||||
| **C5** — `backup_target_absent` + `backup_target_restored` | R-113 | ❌ **FAIL** — generic alarm, specific recovery → **R-116** |
|
||||
|
||||
**All three shipped fixes work.** R-113's gate fires in **4 seconds** (E-2d measured zero over 4½
|
||||
minutes). R-114's message is correct. R-112's banner reaches the customer. C5 fails on a **fourth,
|
||||
separate defect that was unreachable until R-113 made the gate fire at all.**
|
||||
|
||||
## What the drill box ran
|
||||
|
||||
Agent **0.114.0 from the Day-0 manifest** — the shipped binary, so C5 tested the real artifact and
|
||||
closed R-115's observation 1 for R-113. Controller **0.186.0 hand-deployed** after install (§3.1
|
||||
ruling (a)); the vouched golden bakes 0.185.1, so **C3/C4 prove the code, not the shipped golden** —
|
||||
that lag is filed against R-115, not a new ID.
|
||||
|
||||
## The three headline observables
|
||||
|
||||
**R-113** — detach at 18:43:50, gate at **18:43:54**, on exactly the shape that defeated it before:
|
||||
```
|
||||
raw /mnt/mentes : NOT mounted
|
||||
bind /mnt/felhom-drives/mentes : /dev/sdb[/felhom-data] <- the stale bind SURVIVED
|
||||
```
|
||||
|
||||
**R-114** — with the target absent: absent copy 1, **system-disk copy 0**, **offer block 0**. Both of
|
||||
E-2d's falsehoods gone.
|
||||
|
||||
**R-112** — banner element 1 and the never-configured copy in the HTML; after the wizard, the offer
|
||||
with `data-path="/mnt/felhom-drives/mentes"`. Healthy renders nothing, **proven positively**: idle
|
||||
delta 0 `/backup/tiers` calls, page-load delta **+1**, single caller ⇒ the seam ran and chose silence.
|
||||
|
||||
**Decline path proven** (registration confers no role), `restart_required:true`, agent did **not**
|
||||
self-restart, in-flight check recorded before I restarted it, wrapper created the storage at the
|
||||
drive's own mountpoint.
|
||||
|
||||
## C5's failure
|
||||
|
||||
```
|
||||
absent : Event pushed: storage_disconnected (error) <- GENERIC
|
||||
return : Event pushed: backup_target_restored (info) <- SPECIFIC
|
||||
```
|
||||
|
||||
`backup_target_absent` count **0** across the whole run. The alarm and its recovery cannot be matched
|
||||
— precisely what `notifyDriveReturned`'s own comment forbids.
|
||||
|
||||
**Root cause (R-116):** `driveTargetByPath` builds `out[GuestPath] = d.BackupTarget`, but the drive is
|
||||
**two `/disks` rows** and the flag and the guest path sit on different ones — the `felhom-backup`
|
||||
storage row carries `BackupTarget: true` and gets a guest path only while classified user-data; the
|
||||
registry union row carries the guest path and **never assigns `BackupTarget`**. Absent ⇒ they separate
|
||||
⇒ generic. Return ⇒ they rejoin ⇒ specific. v0.184.1 fixed the *keying*, not this.
|
||||
|
||||
## Mirror + over-correction guard — PASS, with a caveat
|
||||
|
||||
Non-target drive detached ⇒ `storage_disconnected`, `backup_target_absent` count 0. **Over-correction
|
||||
guard passes**: both drives present ⇒ 0 ABSENT lines, target stayed healthy — R-113's stricter presence
|
||||
did not make a healthy drive read absent. **Caveat: the mirror passes trivially**, because the target
|
||||
also produced the generic event; it confirms no over-correction but cannot confirm discrimination.
|
||||
|
||||
## Record
|
||||
|
||||
- `OPEN-ITEMS.md` — **R-113, R-114, R-112 → SHIPPED + PROVEN-LIVE**; **R-116 opened** (READY (S), P1);
|
||||
**E-2 and E-2d CLOSED as partially proven** with R-116 as the one named open leg, per the runbook's
|
||||
§9 decided-in-advance rule.
|
||||
- `ROADMAP.md` — R-116 under P1.
|
||||
- **Capability map NOT touched** — it still has **no E-2 / backup-target rows at all**, so no row could
|
||||
be moved to PROVEN-LIVE. Creating them is a design act, not a validation act. Third session running
|
||||
that this has been noted.
|
||||
|
||||
## Teardown
|
||||
|
||||
VM destroyed, storage removed, **`pvesm status` after == before** (`local-lvm` 38.78 %), guest 9201 and
|
||||
`drill-r50` untouched. **Customer ruling: DELETE**; attempted and correctly refused (`host … is
|
||||
ONLINE`) — deletable once the destroyed host ages to DOWN (>1 h), command recorded in `OPEN-ITEMS.md`.
|
||||
|
||||
## What did not happen
|
||||
|
||||
`backup_target_absent` never fired, so its severity, Hungarian copy and hub routing remain unexercised
|
||||
— R-116 blocks them. The offer was accepted via the endpoint the button POSTs, not a browser click (no
|
||||
browser automation on DooPlex); the rendered control and its non-auto-submission were verified in HTML.
|
||||
The stale bind still naming a dead device node after return was observed, not investigated.
|
||||
|
||||
## The arc
|
||||
|
||||
E-2 ends here. Its stated definition of done is **R-106 + R-109, R-108 and D5** — none of which this
|
||||
detour touched. The detour was worth taking: it found six real defects (R-111 through R-116), four of
|
||||
them customer-affecting, none of which any unit suite had caught.
|
||||
@@ -1,98 +0,0 @@
|
||||
# REPORT — tester gate: golden re-baked to 0.188.0, fresh-install proof PASSED (2026-07-31)
|
||||
|
||||
Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so the shared `REPORT.md` (E-2 increment 1) is
|
||||
not clobbered. Full record with every observable: `documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md`.
|
||||
|
||||
## Outcome
|
||||
|
||||
**§7.2 — YES: a fresh install is safe to hand to an external tester.** ISO boot → claimable,
|
||||
app-serving box in ~10 minutes unattended, and an app's data restored **from the drive with the
|
||||
guest's `app.yaml` gone**, proven readable by the application over its own TCP path.
|
||||
|
||||
**Golden 0.186.0 → 0.188.0** baked, published, vouched. **No ISO rebuilt** — Part 0 proved none was
|
||||
needed. No existing box changed; floor still v0.156.0, MinAgent still 0.113.0.
|
||||
|
||||
## Part 0 — the ISO does not need rebuilding
|
||||
|
||||
Verified against the ISO **on disk**, not from source. It bakes exactly three Felhom payloads
|
||||
(`felhom-bootstrap.sh`, its unit, the secret-free pairing env) — full-base64 match, 1 hit each — and
|
||||
**0** hits for `SCRIPT_VERSION="1.2`, `felhom-controller`, `vzdump-lxc-9100`. The installer is fetched
|
||||
at run time (`felhom-bootstrap.sh:96`) and the live URL is byte-identical to repo HEAD
|
||||
(sha `ab8b283e…`, v1.22.0, committed six days *after* the ISO). The golden arrives via the hub-vouched
|
||||
artifact manifest (`felhom-host-install.sh:423-433`). The one genuinely baked, drift-capable thing is
|
||||
`felhom-bootstrap.sh` itself — currently at repo HEAD.
|
||||
|
||||
**Proven live**, not just argued: the fresh box ran `felhom-host-install v1.22.0` and fetched golden
|
||||
**v0.188.0**, sha-verified.
|
||||
|
||||
## Part 1 — bake / publish / vouch
|
||||
|
||||
Baked **0.188.0**, not the brief's 0.187.0: 0.187.0 lacks D5, and Part 2 step 6 *is* the D5 claim, so
|
||||
that golden could not have passed the proof this task exists for. 0.188.0 satisfies R-120 anyway.
|
||||
Stated rather than absorbed, per standing rule 4.
|
||||
|
||||
GOLDEN_VERSION=0.188.0
|
||||
GOLDEN_SHA256=7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299
|
||||
|
||||
Three observables: 404 pre-gate (with a 200 control on 0.186.0 so it is not vacuous), then an
|
||||
**anonymous** download returning `http=200 bytes=649310288` and a matching sha; the manifest read back
|
||||
showing `0.188.0` selected; and the consumer call `GET https://hub.felhom.eu/api/v1/artifacts/sess-g`
|
||||
returning the pair. Plus a fourth: a real fresh box fetched and sha-verified it.
|
||||
|
||||
**R-120's gate evaluated and allowed.** Exercised both ways rather than inferred from silence —
|
||||
vouching 0.185.1 first produced `flash=golden_behind_fleet`, the logged `artifact vouch REFUSED`, and
|
||||
**no write** (the manifest still read 0.186.0); then 0.188.0 produced
|
||||
`Artifact manifest set: agent=0.118.1 golden=0.188.0`.
|
||||
|
||||
## Part 2 — the clean-install proof, on demo-hp
|
||||
|
||||
All seven steps PASS. Venue was demo-hp (Tier 0, the designated drill host) using the scratch dir
|
||||
storage at `/mnt/nvme-1tb` that `target-selection.md:38-40` names; `local-lvm`, `drill-r50` and both
|
||||
9201s untouched.
|
||||
|
||||
Highlights: real day-0 pairing → bind → install; a **real** claim (the code is emailed-only, R-119 —
|
||||
the operator relayed it), with the gate flipping `dashboard not yet claimed` → `authentication
|
||||
required`; controller **0.188.0** confirmed *from the box*; **rallly** (postgres) + **homebox**
|
||||
deployed through the real endpoints.
|
||||
|
||||
The D5 leg: recovery unit `portable-carried=2/2, withheld=0`; the carried `DB_PASSWORD` matched the
|
||||
live one **by fingerprint** (`14c8f515…`, never printed); guest `app.yaml` moved aside; restore
|
||||
returned `secrets recovered=2/2`. Step 7 read the data from **rallly's own network namespace** over
|
||||
TCP to `rallly-postgres` — not the localhost trust socket that produced D5's false pass — and the same
|
||||
path with a wrong password returned `FATAL: password authentication failed`, proving the credential
|
||||
does real work. **The discriminator held: PRE-BACKUP row = 1, POST-BACKUP row = 0.**
|
||||
|
||||
## Part 3 — runbook integrity
|
||||
|
||||
`RUNBOOK-manual-build.md` told the reader to use a "RECORDED" qemu line that is itself labelled
|
||||
*reconstructed*, and whose source says it *"was never saved"*. The real invocation is now captured
|
||||
from this bake and recorded as canonical in **§4.0**, alongside the bake/publish/teardown steps, the
|
||||
template-rot warning and where the R-120 gate actually lives. The old runbook's deviation entry is
|
||||
marked SUPERSEDED with a forward pointer.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
1. VM 310 destroyed with `--purge --destroy-unreferenced-disks 1`; `/mnt/nvme-1tb/images/` empty.
|
||||
2. `cc-scratch` removed, `storage.cfg` back to its original four entries; `felhom-backup` available
|
||||
**926 492 284 KiB before and after** — space returned exactly.
|
||||
3. Hub: **`sess-g` and its host record DELETED, full cascade** — `customer DELETE cascade COMPLETE
|
||||
for sess-g (journal #8) — full teardown`, residue purged including `appliance_registrations=1`.
|
||||
Verified positively: `/configs` and `/hosts` both loaded (10160 / 9880 bytes) with **0** hits for
|
||||
`sess-g` and 0 for the appliance UUID. The gate refused twice first (409 host ONLINE, then 400
|
||||
missing acknowledgements) — the record cannot be deleted until the destroyed box ages out of
|
||||
ONLINE, ~30 min. **`sess-f` deliberately NOT deleted** (R-131); its command is in the audit §7.1.
|
||||
Secrets shredded in the guest and on the box.
|
||||
|
||||
## Findings — filed, none fixed
|
||||
|
||||
`R-128` ISO_VERSION/SCRIPT_VERSION comment is false · `R-129` demo-hp's "no baked SSH key" is stale
|
||||
(key auth works) · `R-130` `HARD_MIN_LVM_GIB` warns and proceeds — a hard min that is not hard ·
|
||||
`R-131` `sess-f` is a fourth orphaned scratch customer · **`R-132` — `curl -w '%{redirect_url}'`
|
||||
printed the hub operator password into a session transcript; `HUB_PW` needs rotating.**
|
||||
|
||||
## Not done, deliberately
|
||||
|
||||
No ISO built; no defect fixed; no golden deployed to an existing box; no floor or MinAgent change;
|
||||
offsite/PBS-DR legs not exercised (the task forbids pointing anything at production PBS or the real
|
||||
restic offsite, so `sess-g` ran DR-tier off); Campaign 10, the demo-hp repartition and subdomain
|
||||
onboarding untouched.
|
||||
@@ -1,97 +0,0 @@
|
||||
# REPORT — SPIKE 4: can a `.deb` in the ISO deliver the stub on an interactive install? (2026-07-31)
|
||||
|
||||
> Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub
|
||||
> v0.85.0 session and the second session in a shared clone never touches it. Supersedes this file's
|
||||
> Spike 1–3 contents.
|
||||
|
||||
**Class: Spike.** Findings only — no production file changed, no release ISO built, nothing published.
|
||||
Evidence: `documentation/audits/SPIKE-universal-iso-4-2026-07-31.md`.
|
||||
|
||||
## The answer is yes, and it was measured with the negative control in the same box
|
||||
|
||||
One ISO, 15 GRUB entries, a trivial probe `.deb` injected into `/proxmox/packages/`. Two VMs on
|
||||
demo-hp built with `qm` so the run was visible in the web console: **400 interactive**, **401
|
||||
automated control**.
|
||||
|
||||
On the **interactive** install (`spikefour.felhom.eu`):
|
||||
|
||||
- the package is installed — `ii felhom-spike4-probe 0.0.1`
|
||||
- its **postinst ran** — marker file present, content intact
|
||||
- it **enabled a systemd unit**, and **that unit fired on first boot** (uptime 7.98 s, `pid1: systemd`)
|
||||
- and on **that same machine**, `proxmox-first-boot` is not installed and `/var/lib/proxmox-first-boot`
|
||||
does not exist — Spike 3's negative reproduced, not assumed
|
||||
|
||||
So the two delivery mechanisms are independent, and the one that survives the path we are actually
|
||||
shipping is the `.deb`. **The product — insert the stick, install Proxmox normally choosing your own
|
||||
disk and password, box sets itself up and waits for a claim code — is now measured rather than hoped
|
||||
for.** With one honest caveat: what was measured is a trivial probe package, not Felhom's real stub.
|
||||
Packaging the real stub and confirming pairing end-to-end is the last step before a spec (~60 min).
|
||||
|
||||
## What a postinst may and may not do
|
||||
|
||||
Identical on both paths: `pid1 = unconfigured.sh`, **no running systemd**, `/proc` and `/sys` mounted,
|
||||
and **`systemctl enable` succeeds** (it wrote the symlink). Network and DNS *happened* to be up —
|
||||
inherited from the installer's own DHCP.
|
||||
|
||||
Four constraints for the real postinst, so they get written against rather than discovered:
|
||||
|
||||
1. Never `systemctl start` or `daemon-reload` — there is no systemd running. `enable` is the only verb.
|
||||
2. **Never require the network**, despite it being present here. A box installed with the cable out
|
||||
gives a postinst no route, and a failing postinst breaks the customer's install.
|
||||
3. Never fail — guard everything, `exit 0`.
|
||||
4. Do the real work in the unit at first boot, where systemd, network and a booted kernel exist.
|
||||
|
||||
## Two smaller results
|
||||
|
||||
**The repack preserves the `.deb`, but not naively.** `xorriso … -boot_image any replay` fails with
|
||||
*"Overlapping MBR partition entries"* — and `iso-repack.sh:270-292` already documents that exact
|
||||
failure and its fix. Mirroring it produced a working image (19 El Torito entries; the `.deb` extracted
|
||||
back out is byte-identical). So this is an insertion into an extract→modify→re-master cycle our repack
|
||||
already performs, not a new build stage.
|
||||
|
||||
**Q3:** `iso-repack.sh:100-106` refuses an ISO without `auto-installer-mode.toml`. It is a guard, not
|
||||
a structural requirement, and its reasoning is sound for the shape it was written for — already R-155,
|
||||
cited exactly here. With no mode file the stock grub.cfg does not emit the Automated entry at all; with
|
||||
a mode file but no answer, that entry aborts safely and loudly.
|
||||
|
||||
## A correction I owe you from last session
|
||||
|
||||
**R-153 is retracted.** The register grep this task mandated shows R-94 already carries it verbatim at
|
||||
`OPEN-ITEMS.md:15`, status `READY (XS)`, with leg (b) being precisely "the gate fails today and is
|
||||
invoked by nothing" — and R-29, the class, says in terms *"do not mint a new ID for a new instance."*
|
||||
Spike 3 filed a duplicate.
|
||||
|
||||
**And the substantive half of that Spike 3 claim was wrong.** I wrote that the drift left the customer
|
||||
page's install-command generator "targeting a flag surface three minor versions stale." R-94 explicitly
|
||||
retracts exactly that reading: the constant selects no script — it renders as a text label, and the
|
||||
command beneath it fetches the script the website git-syncs from `main`, so **1.22.0 is what every
|
||||
install already gets**. It is a wrong number on your screen and nothing more. I overstated it.
|
||||
|
||||
## Still unknown
|
||||
|
||||
**The real stub has not been packaged** — that is the one thing between here and a build spec.
|
||||
Also unproven: `dpkg --configure -a` ordering for a package with dependencies; an ISO that never went
|
||||
through `prepare-iso` (blocked by R-155, which this spike was fenced from changing); and the Graphical
|
||||
installer, where the result should hold *a fortiori* since the `.deb` path is in `Install.pm`, shared
|
||||
by all front-ends — but that is inference, not measurement.
|
||||
|
||||
**Spike 3's Q3** — the real stub at `before-network` — **this session did not touch it.** Note it is
|
||||
now partly superseded: on the `.deb` route the unit's ordering comes from the unit file, not from
|
||||
`[first-boot].ordering`.
|
||||
|
||||
## R-rows
|
||||
|
||||
**None opened.** Each candidate was grepped against the register first: the delivery result is a
|
||||
positive finding, the postinst constraints belong in the build spec, and the repack guard is already
|
||||
R-155. **R-153 retracted** into R-94 leg (b) / R-29.
|
||||
|
||||
## Teardown
|
||||
|
||||
All three layers plus the scratch storage, verified positively. demo-hp: VMs 400/401 purged, **storage
|
||||
`spike4` removed** (`storage.cfg` back to 4, `grep -c spike4` = 0), `/mnt/nvme-1tb/images/` empty,
|
||||
**disk usage 6.6 G — identical to pre-spike**, probe ISO and driver removed, 0 loop devices,
|
||||
`drill-r50` stopped and untouched, 9201 running, `felhom-backup` unmodified, nothing on `local-lvm`.
|
||||
DooPlex: workspace scratch **4.8 GB removed**, scratchpad **3.3 GB → 88 K**, both throwaway passwords
|
||||
destroyed, 17 ISOs in `out/` untouched, no production file modified. **Hub-side: nothing created** —
|
||||
the VMs took LAN DHCP leases but never ran `felhom-host-install.sh` or contacted the hub; verified by
|
||||
fetching and searching the customer list. Nothing published.
|
||||
@@ -1,200 +1,173 @@
|
||||
# REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed
|
||||
# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes
|
||||
|
||||
**Date:** 2026-08-03 · **Repos:** `felhom.eu` (installer **v1.22.0 → v1.23.0**), `felhom-agent` (**no bump**)
|
||||
**Nothing was built** — no image, no binary, no golden. **Hub stays v0.89.0.**
|
||||
**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built
|
||||
and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the
|
||||
box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` /
|
||||
`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at
|
||||
source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**.
|
||||
|
||||
## 1. Baselines — re-read on arrival, both matched §1
|
||||
**Outcome: four scheduled runs, all passed, nothing triggered by hand.**
|
||||
|
||||
| Repo | @ arrival | Version | Result |
|
||||
|---|---|---|---|
|
||||
| `felhom.eu` | `8360f940bfb2` | hub v0.89.0, `SCRIPT_VERSION="1.22.0"`, **0 tags** (confirmed) | installer **v1.23.0**, first tag `installer-v1.23.0` |
|
||||
| `felhom-agent` | `9dfd89cb947e` | v0.120.0 | **unchanged** — scripts and gates only |
|
||||
---
|
||||
|
||||
## 2. Part 0 — the R-182 measurement, and it REVERSED the row
|
||||
## 1. Preconditions
|
||||
|
||||
Filed yesterday as *"the reserve re-alerts on every status refresh"* — **too many** alerts, observed
|
||||
at the sending end. Measured at the **receiving end**, it is the opposite.
|
||||
|
||||
Method: the hub's SQLite copied **with its `-wal`** (4 MB and newer than the db — copying `hub.db`
|
||||
alone would have read stale data, the exact trap this project recorded before), freshness confirmed by
|
||||
the newest `notification_log` row post-dating the session.
|
||||
|
||||
**9 `recovery_unit_capture_failed` events received today → 2 operator emails sent.**
|
||||
|
||||
| time | apps refused (events in) | operator emails out |
|
||||
| # | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| 06:40:03 | privatebin, opengist | **opengist only** |
|
||||
| 08:59:46/47 | opengist, privatebin | **privatebin only** |
|
||||
| 08:59:59 | privatebin, opengist | **none** |
|
||||
| 09:03:00 | opengist | **none** |
|
||||
| 09:07:06 | privatebin, opengist | **none** |
|
||||
| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) |
|
||||
| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots |
|
||||
| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) |
|
||||
| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** |
|
||||
| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one |
|
||||
| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's |
|
||||
| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window |
|
||||
|
||||
**Cause, confirmed at source:** the operator cooldown key is
|
||||
`customerID + ":" + eventType + cooldownTierSuffix(details)` (`dispatcher.go:268`, 1 hour hardcoded).
|
||||
`RecoveryUnitFailureDetails` carries **`app`** and **no `tier`**, so the suffix is empty and the key
|
||||
holds **no app identifier**. The first refused app takes the slot; every other app's refusal for the
|
||||
next hour is dropped — and dropped **before `LogNotification`**, so it leaves **no row on any
|
||||
channel** and cannot be audited afterwards.
|
||||
|
||||
This is **R-97a's failure mode in a second event type**; that row's own comment states it
|
||||
(*"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20"*). `cooldownTierSuffix` was
|
||||
written narrow on purpose; `recovery_unit_capture_failed` simply never opted in.
|
||||
|
||||
**A correction I owe on yesterday's report.** It said *"one `recovery_unit_capture_failed` per app,
|
||||
HTTP 200"*. That was true of what the **controller pushed**, and a reader would take it as *the
|
||||
operator was told about each app* — which is false. The gap between an accepted event and a sent
|
||||
email is the whole of this row.
|
||||
|
||||
**Nothing was changed** (§8.5). R-182 is re-scoped with the evidence and the fix shape.
|
||||
|
||||
## 3. Probes
|
||||
|
||||
| | Question | Method | Verdict |
|
||||
|---|---|---|---|
|
||||
| **P-A** | does git-sync v4.4.0 follow a tag, and notice a **moved** one? | throwaway `docker run` git-sync against this repo, tag moved under it | **PASS both halves** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period (~20 s) |
|
||||
| **P-B** | does Gitea serve `raw/tag/<tag>/<path>`? | one fetch on a throwaway tag | **PASS** — HTTP 200, byte-identical to `raw/branch/main` |
|
||||
| **P-C** | can CI read the package registry? | anonymous fetches | **PARTIAL, and it changed the gate's design** — package **download** 200 (and **404** for a fake version, so it discriminates), **tags** api 200; package **listing** api **401**, hub artifact manifest **401** |
|
||||
|
||||
**Publish model P-A implies:** publishing is **moving the tag**; rollback is **moving it back**, in
|
||||
~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone
|
||||
(`git ls-remote --tags` → 0 at the time).
|
||||
|
||||
## 4. §8.2's three channels — enumerated
|
||||
|
||||
| Channel | Before | After | |
|
||||
|---|---|---|---|
|
||||
| 1. the served script | `main`, 30 s | **`installer-v1.23.0`** | **MOVED** — `webpage.yaml` split into two syncs |
|
||||
| 2. the run-time fetches | `raw/branch/main` | **`raw/tag/v$ART_AGENT_VER`** | **MOVED** — but see below |
|
||||
| 3. the URL producers | `main` | unchanged | **NO CHANGE NEEDED** — and that is a finding, not an omission |
|
||||
|
||||
**Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable.**
|
||||
There are **sixteen** fetches, not nine, and they come from **`felhom-agent`**, not `felhom.eu` — so
|
||||
no tag on this repo could ever have covered them, and §8.1's *"derive the tag from `SCRIPT_VERSION`"*
|
||||
was impossible for them. Raised before building; operator ruled to pin them to **the agent version
|
||||
being installed**, which the installer already resolves from the hub manifest and already sha-verifies.
|
||||
That is strictly better than any installer-derived tag: binary and configs now come from one ref.
|
||||
|
||||
**Channel 3 needed no change because the URL never carried a ref** —
|
||||
`https://felhom.eu/scripts/felhom-host-install.sh` is path-based; the ref lives in the manifest. So
|
||||
`felhom-bootstrap.sh` and the hub's day-0 command follow the tag automatically. **No hub template
|
||||
change ⇒ no hub bump**, so §1's rule was never in tension and the STOP it anticipated never arose.
|
||||
|
||||
## 5. The tag convention
|
||||
|
||||
- **Shape:** `installer-v<SCRIPT_VERSION>` in `felhom.eu` (prefixed so it cannot be read as a hub,
|
||||
agent, controller or golden version); `v<semver>` in `felhom-agent` (that repo versions one thing).
|
||||
**No new constant in the installer** — channel 2 derives its ref from `$ART_AGENT_VER` at run time,
|
||||
and channel 1's ref lives only in the manifest.
|
||||
- **Publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml` (sidecar *and*
|
||||
init container), commit, sync.
|
||||
- **Roll back:** move the tag back to the previous commit — takes ~30 s, **no ArgoCD sync, no deploy**.
|
||||
|
||||
## 6. Scenario A — proven by HTTP
|
||||
|
||||
A real commit was pushed to `main` (a marker comment in the installer) **without moving the tag**, and
|
||||
three sync periods were allowed to pass so "unchanged" means "had every chance to change":
|
||||
## 2. The due verdicts before the run, quoted
|
||||
|
||||
```
|
||||
website tree (main): .worktrees/6a82719… <- ADVANCED to the new commit
|
||||
scripts tree (tag): .worktrees/bee6848… <- STAYED
|
||||
sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
|
||||
sha256 after push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
|
||||
marker present at the served URL? 0
|
||||
https://felhom.eu/ -> HTTP 200
|
||||
demo-felhom tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst"
|
||||
reason: newest settled archive … has not been proven; nothing proven on this tier yet
|
||||
tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z"
|
||||
reason: … has not been proven (last proven archive was a different one)
|
||||
|
||||
demo-hp tier=felhom-backup due=true archive="…2026_08_02-04_49_29.tar.zst"
|
||||
tier=felhom-pbs due=true archive="…2026-07-28T19:19:45Z"
|
||||
```
|
||||
|
||||
Both halves of the split in one observation: the site still tracks `main`, the installer does not.
|
||||
**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical.
|
||||
|
||||
## 7. Scenario B — publish and rollback, both directions
|
||||
## 3. The runs — all four SCHEDULED, none triggered
|
||||
|
||||
| act | result |
|
||||
|---|---|
|
||||
| tag moved `bee6848 → 6a82719` | scripts tree moved in **~40 s**; served `sha256 ea2b4aa9…`; **marker present** |
|
||||
| tag moved back `→ bee6848` | scripts tree back in **~40 s**; served `sha256 2f859555…` — **exactly** the pre-publish sha; **marker gone** |
|
||||
| box | tier | due at | archive | result |
|
||||
|---|---|---|---|---|
|
||||
| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 |
|
||||
| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 |
|
||||
| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 |
|
||||
| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 |
|
||||
|
||||
`https://felhom.eu/` returned 200 throughout. The marker commit was then reverted, and the tag moved
|
||||
to `main`'s head — a **byte no-op**, verified by the served sha not changing.
|
||||
**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The
|
||||
only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2,
|
||||
which start nothing.
|
||||
|
||||
## 8. Files, commits, tags
|
||||
**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of
|
||||
proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following
|
||||
evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That
|
||||
is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.
|
||||
|
||||
**`felhom.eu`** — `bee6848` (installer + gate + manifest), `6a82719` (Scenario A marker), `e79a20b`
|
||||
(marker removed), plus the docs commit below.
|
||||
`scripts/felhom-host-install.sh` · `scripts/hostinstall_gates.py` · `scripts/CHANGELOG.md` ·
|
||||
`manifests/webpage.yaml` · `CLAUDE.md` · `CONTEXT.md` · `STATUS.md` · `REPORT.md` ·
|
||||
`documentation/backlog/{OPEN-ITEMS,ROADMAP}.md` · `documentation/architecture/00-capability-map.md`
|
||||
**The asymmetry worth keeping:** a host-tier restore is **83–109 s**; an offsite one **300–540 s**. The
|
||||
tier an ordinary recovery uses is also the cheapest to prove.
|
||||
|
||||
**`felhom-agent`** — `dd2d1fe` (release path + gate + CI), `0db7766` (REPORT).
|
||||
`scripts/release-agent.sh` **(new)** · `scripts/check-published-versions.py` **(new)** ·
|
||||
`scripts/agent_gates.py` · `.gitea/workflows/gates.yml` · `CLAUDE.md` · `CHANGELOG.md` · `REPORT.md`
|
||||
## 4. What the runs left behind
|
||||
|
||||
**Tags created:** `felhom.eu/installer-v1.23.0` (the first tag this repo has ever had) and
|
||||
`felhom-agent/v0.120.0` (retroactive, at `cd6e267` — the commit the published binary was built from;
|
||||
`configs/` is byte-identical there and at `main`, so nothing depended on the choice).
|
||||
**Persisted state — v3, naming the archive, the tier and what was verified:**
|
||||
|
||||
## 9. Tests and red-proofs
|
||||
```json
|
||||
demo-felhom felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
|
||||
proven_at 2026-08-03T22:56:45Z}
|
||||
felhom-pbs → {archive …2026-07-28T04:49:43Z, tier pbs, proven_at 2026-08-04T05:04:21Z}
|
||||
demo-hp felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
|
||||
felhom-pbs → {archive …2026-07-28T19:19:45Z, tier pbs, proven_at 2026-08-04T06:10:39Z}
|
||||
```
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `felhom.eu` `repo_gates.py --fast` | all 5 gates OK |
|
||||
| `felhom-agent` `go build ./... && go vet ./...` | OK |
|
||||
| `felhom-agent` `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
|
||||
| `agent_gates.py --fast` | `published` correctly **SKIPPED** (hook must not fail on a network blip) |
|
||||
| `agent_gates.py` (full) | both OK |
|
||||
**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's
|
||||
latest report holds **two** entries, one per tier; the `local` one can only have come from the
|
||||
persisted state, because the in-memory store held only that morning's offsite run:
|
||||
|
||||
**Red-proofs, each demonstrated failing then restored:**
|
||||
```
|
||||
demo-felhom tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
|
||||
tier=pbs pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z tested_at=2026-08-04T05:04:21Z
|
||||
demo-hp tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z
|
||||
```
|
||||
|
||||
| # | Mutation | Result |
|
||||
**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now:
|
||||
|
||||
```
|
||||
tier=felhom-pbs due=false proven="…2026-07-28T04:49:43Z"
|
||||
reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
|
||||
```
|
||||
|
||||
…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h.
|
||||
That is not a defect: it is "proved daily, on its own archive", visible one day later.
|
||||
|
||||
## 5. Teardown — three layers, per box
|
||||
|
||||
| layer | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| C | one of the sixteen fetches reverted to `raw/branch/main` | **RED** — gate 6a *and* 6b both fired |
|
||||
| D | assertions 6a **and** 6b removed (every guard the test covers), same bad installer | **zero** mentions of the regression — the guards are what catch it |
|
||||
| 6c | the manifest before the split | **RED** on its own, before I fixed it — the gate was demonstrated red by the real pre-change state |
|
||||
| F | `v9.9.9` tagged and not published | **RED**, `binary NOT downloadable (HTTP 404 …)`, rc=1 |
|
||||
| F′ | the gate **deregistered** from `agent_gates.py`, same bad state | **rc=0, "all agent gates OK"** — restored → `CONVICTED: published`, rc=1 |
|
||||
| the machine | `pct list` → **0** entries for 990000 | **0** |
|
||||
| the host | `lvs` → **0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after |
|
||||
| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same |
|
||||
|
||||
**Scenario F measured on real CI, not inferred.** Runs **69** and **70** are on the *same commit*
|
||||
`0db7766`: **success** before `v9.9.9` existed, **failure** after pushing it. One variable. This also
|
||||
retrospectively explains runs 67/68. **One deliberate CI failure email reached the operator — that was
|
||||
this proof, not an incident.** I could not read CI's own step log: the jobs endpoint needs a Gitea API
|
||||
token, and the only credential available (`~/.docker/config.json`) is a registry password that the API
|
||||
rejects — so the controlled before/after replaced the log rather than an assumption standing in for it.
|
||||
Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.
|
||||
|
||||
## 10. No version bumps, nothing built
|
||||
## 6. What the run surfaced — three findings, two of them corrections to my own record
|
||||
|
||||
`felhom-agent` **v0.120.0** unchanged (no Go code changed). Hub **v0.89.0** unchanged (no hub file
|
||||
touched). The installer's `SCRIPT_VERSION` **did** go 1.22.0 → 1.23.0 — the installer is not in §12's
|
||||
no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or
|
||||
golden was built.
|
||||
### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning
|
||||
|
||||
## 11. Register
|
||||
A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on
|
||||
2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by
|
||||
~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall
|
||||
(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry.
|
||||
Correlated but not established: guest 9201 was reprovisioned at 09:15–09:19, nine minutes before.
|
||||
**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**,
|
||||
and v0.123.0's probe detects the *state* but says nothing about the *transition*.
|
||||
|
||||
| ID | Outcome |
|
||||
|---|---|
|
||||
| **R-110** | **CLOSED — SHIPPED** (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed |
|
||||
| **R-115** | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, no bump) |
|
||||
| **R-182** | **RE-SCOPED — the direction reversed** by Part 0's measurement; still open, now correctly described |
|
||||
| **R-183** | **NEW, and CLOSED the same session** — binary and configs came from two different refs |
|
||||
| **R-184** | **NEW, open** — nothing stops the hub vouching a version that was never released |
|
||||
### R-191 (new) — every weekly offsite backup reports FAILED although it worked
|
||||
|
||||
**IDs established free:** `^| \*\*R-183\*\*` / `^| \*\*R-184\*\*` in `OPEN-ITEMS.md` → **0 rows** each;
|
||||
all other hits are this session's own code and changelogs (forward references I wrote). `R-185` → 0
|
||||
hits anywhere and remains free.
|
||||
demo-felhom, 06:49–06:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused
|
||||
incrementally) and the job then failed on the prune —
|
||||
`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job
|
||||
errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed**
|
||||
(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm
|
||||
the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that
|
||||
must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary
|
||||
— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here**
|
||||
(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two
|
||||
namespaces.
|
||||
|
||||
## 12. Observations — noticed, documented, NOT acted on
|
||||
### Two corrections to yesterday's record
|
||||
|
||||
1. **The gate cannot see what is vouched** — filed as R-184 rather than papered over. Closing it needs
|
||||
either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails
|
||||
closed where the mistake is made, needs no new credential).
|
||||
2. **A suppressed operator alert leaves no row at all.** The cooldown returns before `LogNotification`,
|
||||
so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside
|
||||
R-182 because it is what made that row take a day to get the right way round.
|
||||
3. **`on: [push]` fires CI for tag pushes too.** Useful (it is how Scenario F was measured), but it
|
||||
means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate.
|
||||
4. **`felhom.eu` CI still runs `--fast`.** Correct today, since all its gates are network-free; if a
|
||||
network gate is ever added there, that workflow needs the same change the agent's just got.
|
||||
1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd
|
||||
**six times** (09:24 → 17:34) on that storage and privilege, and the hub raised
|
||||
`whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation
|
||||
said backups kept landing because writes go through a root path — wrong, and now corrected in the
|
||||
runbook and on the row.
|
||||
2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route;
|
||||
I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none
|
||||
since the R-86 work — and one **more** since, below.
|
||||
|
||||
## 13. Teardown
|
||||
**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at
|
||||
2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the
|
||||
host tier had never been proven, and the hub said so on its own, without being asked. It has not
|
||||
re-fired; the signal is edge-triggered, so the return to healthy is silent by design.
|
||||
|
||||
Probe container, probe sync tree and probe tag (`probe-r110-delete-me`) removed; the red-proof tag
|
||||
`v9.9.9` deleted (`git ls-remote --tags` → only `v0.120.0`); the Scenario A marker reverted from
|
||||
`main` and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl
|
||||
pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file.
|
||||
## 7. The capability map
|
||||
|
||||
The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo
|
||||
boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the
|
||||
teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box
|
||||
is untested and untouched, and one box proving something does not make it a fleet property.
|
||||
|
||||
## 8. Registers
|
||||
|
||||
- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the
|
||||
correction in §6.
|
||||
- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was
|
||||
the highest in use).
|
||||
- `ROADMAP.md` holds none of these rows, so nothing to collapse.
|
||||
- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under
|
||||
"What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.
|
||||
|
||||
## 9. Observations — noticed, NOT acted on
|
||||
|
||||
- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %,
|
||||
returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes
|
||||
warn about, and the offsite archive is the larger of the two. A materially bigger guest would want
|
||||
the restore pointed at `/mnt/nvme-1tb` instead.
|
||||
- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has
|
||||
none". Nothing depends on that assumption now, but the operations notes still carry it.
|
||||
- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest
|
||||
role — a 2.7× difference worth understanding before either is used to size anything.
|
||||
- **Both waiters this session produced no output** despite the runs completing; the evidence was
|
||||
gathered by direct query afterwards. A watcher that silently produces nothing is exactly the
|
||||
instrument class this project distrusts — the conclusions here rest on the boxes' own journals and
|
||||
the hub's database, not on the waiters.
|
||||
|
||||
@@ -104,7 +104,7 @@
|
||||
| `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
|
||||
| `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). |
|
||||
| `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
|
||||
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. |
|
||||
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). |
|
||||
| `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
|
||||
| `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
|
||||
| `cfg.DRTier` + offsite coupling | hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field `dr_tier` (formBool helper). |
|
||||
|
||||
@@ -14,114 +14,76 @@ A blank machine boots the Felhom disc, installs itself unattended, and is claime
|
||||
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
||||
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
||||
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
||||
restore files and app data from the drive alone. Proven end to end on real hardware.
|
||||
|
||||
**Apps come back after a power cut.** The machine tells an app the customer switched off from one
|
||||
that simply did not come back, and waits for the system to finish starting before deciding instead of
|
||||
glancing once, five seconds in. Hard-reset the demo box six times in a row: everything came back every
|
||||
time, and an app switched off deliberately stayed off every time.
|
||||
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
|
||||
box six times, everything returned every time, and an app switched off deliberately stayed off.
|
||||
Proven end to end on real hardware.
|
||||
|
||||
## What's broken
|
||||
|
||||
**The off-site copy can be erased by the machine that made it** — the credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
||||
*(R-95, R-87)*
|
||||
|
||||
**A full disk tells you about ONE app and silently swallows the rest.** Yesterday this was written
|
||||
down the wrong way round — as *too many* emails. Measuring the receiving end reversed it: of nine
|
||||
refusals the machine reported today, **two emails were sent**. When two apps are refused in the same
|
||||
second you are told about one of them, and the other leaves no trace anywhere — not an email, not
|
||||
even a line in the log saying it was held back. So a second app can be going unbacked-up while you
|
||||
have already been told the problem is handled. It is the same fault we fixed once before for
|
||||
whole-machine backups, in a second place that never opted into the fix. *(R-182)*
|
||||
- **Rebuilding a machine silently takes away its off-site app-data backup.** `demo-hp` was rebuilt on
|
||||
3 August and came up without one, and stayed that way for a day. **Fixed on 4 August** — re-issued,
|
||||
the machine picked the new password up in 15 seconds and reattached to the same repository, and you
|
||||
escrowed the key. **The underlying fault is not fixed:** the off-site password is delivered exactly
|
||||
once and a rebuilt machine cannot ask for another, so this will happen again on the next rebuild.
|
||||
The other machine survived the same rebuild only because an unused password happened to be waiting
|
||||
for it. *(R-193)*
|
||||
- **The daily email about it tells you the wrong story**, and the automatic repair that exists for
|
||||
this declines without saying why. The message says the password was never applied; it was, on
|
||||
23 July, and worked for eleven days. *(R-192)*
|
||||
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
|
||||
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
|
||||
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
|
||||
setting per machine fixes it. *(R-191)*
|
||||
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap.
|
||||
*(R-95, R-87)*
|
||||
|
||||
## What shipped recently
|
||||
|
||||
**Pushing the installer no longer publishes it.** The script that runs as root on a brand-new
|
||||
machine was copied from the main branch and served within thirty seconds, so pushing it *was*
|
||||
publishing it, with no staging and no way back but another push. It now comes from a **labelled**
|
||||
version: publishing is moving the label, and undoing it is moving the label back — about half a
|
||||
minute, no deploy. The website is untouched by this and still updates in thirty seconds, because a
|
||||
typo fix must never need a release. Proven by actually doing it: a real push changed nothing that
|
||||
anyone downloads, moving the label published it, moving it back restored the previous bytes exactly.
|
||||
- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
|
||||
demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
|
||||
checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
|
||||
had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
|
||||
never been tested on either machine. Both also proved their off-site copy on the same night, one
|
||||
after the other rather than at once, which is the machine deciding for itself what to do first.
|
||||
*(closes the last open half of R-86/R-185)*
|
||||
- **A backup copy the machine was never allowed to read — and could not tell you about**, on both
|
||||
demo machines. The permission was one command; the silence was the real fault, and the machine now
|
||||
checks whether it may read each copy it depends on and says so when it may not. *(R-185)*
|
||||
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
||||
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
||||
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
||||
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
|
||||
written to disk with the result and survives. This was caught happening, not predicted: a real
|
||||
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
|
||||
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
|
||||
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
|
||||
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
|
||||
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
|
||||
|
||||
**The catch that would have made it cosmetic was found and covered.** While it runs, the installer
|
||||
fetches sixteen more files — not nine, and from the *agent's* repository, not the website's. They now
|
||||
come from the same version of the agent the machine is installing. That closed a real fault nobody
|
||||
had noticed: a new machine was getting the agent's tested program and its untested settings files, in
|
||||
one install, from two different places. *(R-110, R-183)*
|
||||
|
||||
**Releasing the agent now publishes it, in one command.** Putting a built agent where a new machine
|
||||
can download it was a step someone had to remember, and it was forgotten three times in five days —
|
||||
the last time leaving both demo machines running a version nobody could download, so a rebuild would
|
||||
have quietly installed the *older* one and reported success. There is now one command that builds,
|
||||
labels, publishes and then **downloads it back to check** — and a check that refuses to stay quiet if
|
||||
a released version cannot actually be fetched. Proven by making CI fail on purpose and then go green
|
||||
again on the same code. *(R-115)*
|
||||
|
||||
**The backup partition is gone and both demo machines run on the new shape** — wiped, rebuilt and
|
||||
taken through the whole customer journey on 3 August, by two deliberately different routes so the disk
|
||||
shape and the delivery route are both proven. The space a backup can use went from 19 GB to 65 GB on
|
||||
the small machine and 45 GB to 233 GB on the big one. Their previous demo apps and data are gone; that
|
||||
was the point of a wipe, and you approved it. *(R-165, R-178)*
|
||||
|
||||
**What replaced the wall now watches the right moment.** The wall was quietly keeping a runaway
|
||||
backup from eating the space the machine needs to run. As first built, that replacement was checked
|
||||
too late — the big write happened first, unchecked — while still promising your last good copy was
|
||||
untouched. Fixed and proven on 3 August: the machine decides once, per app, **before it writes
|
||||
anything**, and that one answer covers all three steps, so a refused app writes nothing, is not
|
||||
restarted, and the promise is now literally true. It also stopped being blind to size. Nothing is ever
|
||||
deleted to make room. *(R-181)*
|
||||
|
||||
**The last of the three apps that never saved their data is fixed.** Installed nowhere, so nothing was
|
||||
stranded — checked on both demo machines and in the fleet list rather than assumed. Proven by the check
|
||||
that caught it, run in both directions: it clears the fixed version and still convicts the old one.
|
||||
*(R-156)*
|
||||
|
||||
**A filling disk warns the customer before anything breaks, and a failed backup reaches you** — the
|
||||
customer while there is still room to act, naming the drive and the space left; you when one app's
|
||||
backup fails, with the disk figures. The customer is deliberately not told about the second: they can
|
||||
free space, but they can do nothing about a failed backup. Both proven by filling a real disk. There
|
||||
are two rules and not one because the serious warning fired on free space while the disk was only 91%
|
||||
full — a percentage alone would have missed it. *(R-167, R-158)*
|
||||
|
||||
**The checks have two nets and the second emails you.** Every repository has one command that runs all
|
||||
its checks, before every push. That one can be skipped, so the build server runs them again and emails
|
||||
you on failure. It cannot *stop* a change — everything goes straight to the main copy with no review
|
||||
step — but it notices quickly and tells you. *(R-29, R-161, R-168, R-169)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** nothing outstanding from today — the reserve, the last unsaved app, and both of your
|
||||
decisions are all built and proven.
|
||||
- **Next:** the alert that tells you about one app and swallows the second *(R-182)*.
|
||||
- **After:** the off-site copy that the machine making it can still erase *(R-95, R-87)*.
|
||||
- **Now:** nothing outstanding.
|
||||
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
||||
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
||||
*(R-87)*
|
||||
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **Nothing else.** Both decisions you took on 3 August are now built and proven. One small question
|
||||
will come back later: the automatic check cannot see which version you have told machines to
|
||||
install, only which ones exist — closing that either needs a password given to the build server or
|
||||
a check inside the hub itself. Filed, not urgent. *(R-184)*
|
||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
||||
machines to install, only which ones exist. Closing that needs either a password given to the build
|
||||
server or a check inside the hub itself. *(R-184)*
|
||||
- **Nothing else.**
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-03** — Publishing became something you do rather than something that happens: the
|
||||
installer and the agent both moved onto labelled versions with a way back, and a check now refuses
|
||||
a release nobody can download. Earlier the same day: the reserve now guards the step that fills the
|
||||
disk and its promise is true, and the last app whose data was never saved is fixed. All proven on
|
||||
real machines, not just in tests.
|
||||
Earlier the same day: both demo machines wiped and rebuilt from the new base image and taken through
|
||||
set-up → install an app → back it up → restore it, with the backup space ceiling gone and measured.
|
||||
|
||||
- **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in
|
||||
a mode where reading a page cannot block a machine's status update; a one-word difference meant that
|
||||
setting had **never taken effect**, for the hub's whole life. Fixed and verified live. **Also found:
|
||||
the hub's own database is in no automatic backup** — it holds every machine's emergency password.
|
||||
Filed, not yet fixed.
|
||||
|
||||
- **2026-08-02** — Thirteen mechanical checks had built up and nothing ran most of them; two were
|
||||
failing quietly. Fixed. Decided the same day: the 20 GB backup partition goes away; and only this
|
||||
machine and the tester's box are protected, every other box may be broken or reinstalled freely.
|
||||
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
||||
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
||||
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
||||
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
|
||||
that could vanish on a restart (seen happening), a release that emailed a failure for a release
|
||||
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -132,6 +132,41 @@ executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by const
|
||||
(`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self**
|
||||
(INV Part D1).
|
||||
|
||||
### Lane 2's restore-test is scheduled PER ARCHIVE GENERATION (R-86, 2026-08-03)
|
||||
|
||||
**[CONTRACT, changed 2026-08-03 — agent v0.121.0 + hub v0.91.0.]** The scheduled restore-test used to
|
||||
fire on an interval started at daemon start. It no longer does. The rule is:
|
||||
|
||||
> Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h).
|
||||
> The tier is **DUE** when **A** exists and **A has not already been proven**.
|
||||
|
||||
So a tier is proved **once per archive**, on its own archive, and the proof follows the backup rather
|
||||
than the process's uptime:
|
||||
|
||||
| tier rhythm | what is proved, and when |
|
||||
|---|---|
|
||||
| daily (host tier) | yesterday's archive, once a day |
|
||||
| weekly (offsite tier) | last week's archive, once a week |
|
||||
| newborn (no archive yet) | nothing — **UNKNOWN, never a fault** |
|
||||
|
||||
**The trap in the obvious formulation, recorded so it is not reintroduced:** *"due when the newest
|
||||
archive is ≥ 24 h old"* is never true on a **daily** tier — a new archive resets the newest-archive
|
||||
age to zero long before it reaches the lag — so the literal reading silently switches restore-testing
|
||||
off for the tier that matters most.
|
||||
|
||||
What survives unchanged: the restore-test itself (restore → boot → verify → destroy the scratch), its
|
||||
journal and crash recovery, the scratch VMID band, the one-heavy-operation gate, proof credit only on
|
||||
success, and oldest-proven ordering, which is now the tie-break **between due tiers**. A ticker
|
||||
remains, but only as the **evaluation interval** (6 h by default, chosen from a measured cost: one
|
||||
due-check is 18 ms on a local dir storage and 392 ms on the PBS tier over the WAN).
|
||||
|
||||
**The hub's half is not optional.** `restoreProvenStaleAfter` was a flat 7 days derived from the very
|
||||
cadence this replaced, and a weekly tier proved weekly reaches a proof age of **exactly** one interval
|
||||
just before its next proof — 168 h against a 168 h window. It sat ON the line, so any ordinary delay
|
||||
tipped a healthy tier into a nightly alarm. The window is now per tier, from that tier's observed
|
||||
archive interval, floored at the old 7 days, capped at 12 days (strictly inside the two-week offsite
|
||||
retention), and falling back to the tier's declared rhythm when history is too short to observe one.
|
||||
|
||||
### Why the split is right, stated once
|
||||
|
||||
**[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about
|
||||
@@ -574,6 +609,30 @@ because that was a different filesystem. On a merged box it can. Decision **B2**
|
||||
It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
|
||||
only destroy a different app's only local copy.
|
||||
|
||||
**WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).**
|
||||
The two are deliberately different mechanisms, because conflating them is how seven failures went
|
||||
missing on 2026-08-03 without leaving a trace.
|
||||
|
||||
| | Record | Notification |
|
||||
|---|---|---|
|
||||
| what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN |
|
||||
| when | every time, unconditionally | at the end of a run, **only if something failed** |
|
||||
| gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown |
|
||||
| where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox |
|
||||
|
||||
- **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that
|
||||
is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's
|
||||
REPORT freshness, independent of any mail the box sends. That check is load-bearing for this
|
||||
design; weakening it re-opens a silent-failure path.
|
||||
- **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it.
|
||||
Deciding not to tell someone is itself an event worth recording.
|
||||
- **Deliberate skips are not failures** and never appear in the digest — a disconnected or
|
||||
decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the
|
||||
operator stops reading.
|
||||
- **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within
|
||||
the hour, because someone pressing the button is actively trying to get a backup. The periodic
|
||||
capture sweep is capped by the ordinary hourly cooldown.
|
||||
|
||||
**THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a
|
||||
**per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately
|
||||
before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the
|
||||
@@ -796,7 +855,7 @@ does **not** hold as written. → **R-108**
|
||||
| ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
|
||||
| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
|
||||
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
|
||||
| R-86 (open) | Restore-tests are interval-scheduled, not backup-aligned | a tier's proof cadence is unrelated to when its archives are written |
|
||||
| ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) |
|
||||
| R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof |
|
||||
|
||||
### 10.3 Divergences that are documented elsewhere and are not re-opened here
|
||||
@@ -862,7 +921,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
|
||||
| whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C |
|
||||
| corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 |
|
||||
| the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 |
|
||||
| unattended restore-test across tiers | **IMPLEMENTED** (rotation not observed across consecutive cadences) | `00-capability-map.md:41`; LIVE per-tier timestamps this session |
|
||||
| unattended restore-test across tiers | **IMPLEMENTED**; **per-archive due-ness PROVEN-LIVE 2026-08-03** (agent v0.121.0) | `00-capability-map.md:41`; the due verdict + a real offsite run on demo-felhom (§3, Lane 2's per-archive rule) |
|
||||
| guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s |
|
||||
| quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL |
|
||||
| break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` |
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
**Class:** SPIKE (empirical validation; no product code). **Repos:** felhom.eu (this doc only);
|
||||
felhom-agent read-only for grounding (`internal/pbs/{client,pin}.go`, `configs/build-golden.sh`,
|
||||
`internal/hub/cloudflared.go`, `internal/escrow/identity.go`).
|
||||
**Probe ends:** `felhom-hetzner` = Hetzner CX23, Debian 13.4, public IPv4 `167.233.158.164`,
|
||||
**Probe ends:** `felhom-hetzner` = Hetzner CX23 (**note added 2026-08-03: rescaled to a CX33, 8 GB RAM — this spike records the machine as probed and its body is deliberately unchanged**), Debian 13.4, public IPv4 `167.233.158.164`,
|
||||
global IPv6 `2a01:4f8:...::/64` (throwaway — NOT the live jarrs.eu box) ⟷ **demo-felhom** =
|
||||
the real PVE 9.2.2 host on the operator's home line (One Hungary fixed cable, Budapest), driven
|
||||
over the existing LAN SSH path; the tunnel itself always dialed **out**.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -144,7 +144,7 @@
|
||||
| R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. |
|
||||
| R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. |
|
||||
| R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). |
|
||||
| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — operator ruling 2026-07-27** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
|
||||
| R-86 | ~~**Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup**~~ | S | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03 (agent v0.121.0 + hub v0.91.0)** | **The reasoning that produced the row still holds and is worth keeping:** R-85 scheduled on a free-running INTERVAL with oldest-proven-first rotation, which cannot express *"the day after the PBS backup"* — any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. The operator's ruling was weekly per tier, ~1 day after that tier produced a new backup, so the gap between an archive being written and being proven stays small and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **What shipped is that ruling, with one correction the row's own wording would have got wrong:** the shape written here — *"test tier X once its newest archive is ~24 h old and not yet proven"* — is NEVER true on a DAILY tier, because a new archive resets the newest-archive age to zero before it reaches 24 h. The rule that works is *the newest archive that HAS settled 24 h, and has not been proven*: `A = newest archive older than the settle lag`, due iff A exists and A ≠ the proven archive. **The interim is retired:** `restore_test_cadence_seconds = 302400` (3.5 d) is no longer the mechanism; that key now only disables (negative) or seeds the settle lag. Alignment is no longer approximated by a cadence — it is the schedule. Full detail: `OPEN-ITEMS.md` R-86, `07-backup-architecture.md` §3 |
|
||||
| R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. |
|
||||
| R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | **Part 1 SHIPPED (controller v0.176.0, 2026-07-27); Part 2 OPEN (agent wire change)** — **Part 1** added the failure breaker: consecutive failures tracked per TARGET, backoff `15m→30m→1h→2h→4h` capped, reset on success, gating the QUIESCE rather than the backup so no stack is stopped for a doomed attempt. Never permanent (the cap bounds the retry interval — a latched breaker is a silent backup outage), never global (a broken offsite tier cannot suppress a healthy local one), never applied to `TriggerNow`, and `stillRunning` is not a failure. In-memory on purpose: forgetting a backoff costs one attempt, persisting one could outlive the fix. **Part 2 is agent-side and needs a wire change** — see the P0 finding below. Live on both boxes; the loop reproduction is UNIT-TEST ONLY (reproducing it live would need PBS taken down on ep0, which has no swap). | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **NOT bounded, and it also DISABLES the business-hours protection — an availability fault, not a logging nuisance.** The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves, and a second reading ("it stopped after three tries, so something limits it") was wrong too — the live run stopped after three only because the storage came back. **Verified in code: there is no limiter.** `internal/quiesce` holds no failure counter, no backoff, no breaker, no attempt budget; `runOnce` keeps no state between ticks, and the driver is a plain 5-minute `time.Ticker` (`quiesce.go:149`, default `quiesce.go:107-109`). Had the target stayed down, the loop would have continued indefinitely.
|
||||
|
||||
@@ -175,7 +175,7 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
|
||||
| R-162 | **`docker diff` is the gate's only witness and its failure mode is quiet** | XS | WATCHING — 2026-08-02 | A limitation, not a defect. The gate's power is `docker diff` excluding mounted paths; on a driver where it is unsupported or lies, the gate degrades to mount-occupancy + writability **and would not say so**. It fails closed (the canary self-test stops reporting BROKEN and the gate then refuses to report), but the message blames the prober rather than the driver. Revisit only if a non-overlay driver ships |
|
||||
| R-163 | **`mp1` is RETENTION, not staging — and it is sized as if it were neither** | M | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | Settled by reading: a unit is the KEPT copy on the app's **own** drive (`backup.go:245-255`), falling back to the system SSD for driveless apps — *"the SSD-only system-data fallback"* (`appbackup/paths.go:26-27`); **no post-copy deletion**, the only prune is F5 residue-on-old-drives (`backup.go:1053-1112`). So `mp1` **20 G** retains every driveless app's unit while `mp0` permits **50 G** of volumes, and a DB app's unit is up to **~2×** its data (measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to 0 (`felhom-agent/.../main.go:178`), not derived from the drive; demo-hp's real guest ships 50G/20G. **`mp1` gates the whole chain** — Tier-2 mirrors the unit *"(always)"* (`tier2.go:302,368`) and Tier-3 carries it. Bounded: a unit is volume tars + DB dumps only, **never `mp8` userdata** (`recovery_unit.go:20-25`). **No number proposed.** → `07-backup-architecture.md` §7.5 |
|
||||
| R-164 | **C2's chain — the DB volume tar cannot be dropped until a sound dump predicate exists** | S | BLOCKED — on the predicate (2026-08-02) | The unit holds a volume tar **and** a SQL dump and the restore uses both: the dump is authoritative and replayed after the tar so it WINS (F17), with only the DB service up (R-47) — `restore_unit.go:262-266`. Dropping the DB tar would halve DB-app units and close R-127(b)'s initdb-skip trap. **The obvious gate is dead, measured:** `ValidateDump`'s empty-`accounts` warning was **correct** (the DB truly had 0 rows; seeding one stopped the warning and put the row in the dump) — but **a fresh appliance legitimately has zero accounts**, so gating on it blocks every new customer's first backup. Order: sound predicate (dump vs **live** per-table counts) → warn→gate → tar-drop. Pairs with **R-127** |
|
||||
| R-90 | **ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86** | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: `Swap: 0B`, nothing in `/etc/fstab`, no swap unit, no `/swapfile` — **swap was never added.** On 2026-07-27 06:58:12 a global OOM fired while `proxmox-backup-proxy` held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration `rsync` held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed **R-88** downstream. Still exposed today: 3.8 GB, no swap, one kill on record. **This gates R-86** — backup-aligned restore-testing puts a multi-GB read on this box *on a schedule*, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console |
|
||||
| R-90 | ~~**ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86**~~ | S | **CLOSED 2026-08-03 — the operator rescaled ep0 to a CX33.** MEASURED on the box, not read from an invoice: `Mem: 7757` MB (**8 GB**, was 3.8), `nproc` **4**, and the 4 GiB swapfile added 2026-07-27 **survived the resize** and is active (checked because a resize is a stop/start, so "still there" was an assumption until measured). **The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only — so no disk figure in any runbook needed correcting. Unblocks R-86 | Runbooks corrected; SPIKE/evidence documents ANNOTATED not revised. `target-selection.md`'s "D-d did not name ep0 either way" stays open — it is the operator's |
|
||||
| R-91 | **The old 13 GB datastore copy is still on ep0's root disk** | XS | WATCHING — gated on demo-felhom's first post-migration PBS backup | The datastore moved to a Hetzner Cloud Volume on 2026-07-27 (`/dev/sdb`, 100 GiB, attached 06:29:40 UTC, now `/mnt/pbs-datastore`, 13 G used of 98 G). The pre-migration copy survives at **`/srv/pbs-felhom`, 13 G**, on `/` (38 G total, 16 G used, 21 G free). **Do not delete yet:** demo-hp has landed two post-migration snapshots (07-27 08:25:47Z, 09:37:29Z) but **demo-felhom's newest is 2026-07-26T12:21:48Z — before the migration**, so the new volume has not yet proven a write for that namespace. Delete once it has. **Doc drift to fix in the same commit:** `CONTEXT.md:1018` still records the datastore at `/srv/pbs-felhom` |
|
||||
| R-92 | **The hub's PBS-DR gauge is 0.1 GB-granular, so small deltas are unverifiable** | XS | idea — 2026-07-27 | The PBS-DR box card rounds to 0.1 GB, which is coarser than the changes an operator wants to confirm after a prune or a GC — a successful prune of a small namespace moves the number by less than one displayed digit, so the UI cannot distinguish "it worked" from "nothing happened". Cosmetic today; it becomes load-bearing the moment retention (R-89) is customer-visible and someone needs to see that a policy change took effect |
|
||||
| R-93 | **`drill-r50` is both a blocked customer and the only drift fixture** | XS | idea — 2026-07-27 | The drill customer is blocked in the hub (so it stops alarming) yet it is also the only record exercising the endpoint-drift path R-77 added. Blocking hides it from `GetActiveCustomerIDs`, so the fixture it provides is silently inert — a monitor with no live subject reads exactly like a monitor that passes. Decide: retire it and build a synthetic fixture, or unblock it and silence per-customer instead (the operator has a per-alert silencing feature planned). Related to the R-50 drill VM, now shut down |
|
||||
|
||||
@@ -3,7 +3,8 @@
|
||||
**Class:** supervised operational run. **No repo version bump** — the only commits are this record
|
||||
and the capacity note. **Nothing was deleted.**
|
||||
|
||||
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner CX23, Nuremberg.
|
||||
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner **CX33 (4 vCPU / 8 GB RAM)**, Nuremberg.
|
||||
> **Rescaled 2026-08-03** from the CX23 (2 vCPU / 3.8 GB) this runbook was written against. **The 40 GB local disk did NOT change** — this was a CPU/RAM resize — so every disk figure below still stands. The 4 GiB swapfile added on 2026-07-27 survived the resize.
|
||||
**Datastore moved:** `felhom-offsite`, `/srv/pbs-felhom` → **`/mnt/pbs-datastore`** (name unchanged).
|
||||
**Window:** 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC.
|
||||
|
||||
|
||||
@@ -229,7 +229,9 @@ as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as
|
||||
`keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
|
||||
minutes of ep0 CPU per weekly backup.
|
||||
|
||||
> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
|
||||
> Watch item (**superseded 2026-08-03**: ep0 is now a **CX33, 8 GB RAM**, and it HAS a 4 GiB swapfile
|
||||
> which survived the resize — so the pressure below is much reduced, though the shape of the concern
|
||||
> stands). As written: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
|
||||
> window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
|
||||
> `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.
|
||||
|
||||
@@ -307,7 +309,7 @@ Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13
|
||||
watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
|
||||
4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
|
||||
it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
|
||||
5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.
|
||||
5. ~~**ep0 has no swap** (3.7 GB CX23)~~ — **corrected 2026-08-03: ep0 is a CX33 with 8 GB RAM and an active 4 GiB swapfile.** See the volume runbook's OOM for the original incident.
|
||||
|
||||
## 11. Observations
|
||||
|
||||
|
||||
@@ -431,7 +431,7 @@ label. Filed under E-2.
|
||||
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
|
||||
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
|
||||
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
|
||||
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. |
|
||||
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. **CORRECTION 2026-08-03, same day, measured on the box: it DID surface as a 403, exactly as this item predicted — the earlier annotation here said otherwise and was wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST: `POST /nodes/demo-felhom/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)`. The hub raised `whole_guest_backup_failed` at the first one (*"retrying with backoff"*) and edge-triggering correctly suppressed the rest, so the operator was told once. It ALSO surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). |
|
||||
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
|
||||
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
|
||||
| 8 | The honest **single-drive label**. |
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
> firewall, and the hub-driven `felhom-peersync` reconcile surface. Re-running it on a fresh VM
|
||||
> re-creates the endpoint from nothing (that is the DR story, step 8).
|
||||
>
|
||||
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23, Debian 13,
|
||||
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23 **at the time — rescaled
|
||||
> to a CX33, 4 vCPU / 8 GB RAM, on 2026-08-03; the 40 GB local disk is unchanged**, Debian 13,
|
||||
> `167.233.158.164` / `2a01:4f8:1c16:7aa1::1`) with hub v0.32.0. The production endpoint is a
|
||||
> later re-run of this runbook on a production VM.
|
||||
>
|
||||
@@ -31,7 +32,7 @@ Parameters used throughout (adjust for a new endpoint):
|
||||
points at nothing (live-run finding). Home-resolver propagation can lag public DNS by
|
||||
minutes — a client-side `wg-quick up` that fails to resolve right after record creation
|
||||
just needs a retry.
|
||||
- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway CX23), not
|
||||
- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway box, **CX33 since 2026-08-03**), not
|
||||
any production box.
|
||||
|
||||
## 1. Base (on the box, as root)
|
||||
|
||||
@@ -29,20 +29,23 @@ prohibition as covering the act it names and nothing more.
|
||||
|---|---|---|
|
||||
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
|
||||
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
|
||||
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
|
||||
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **`ep0`** (operator ruling 2026-08-03) — and nothing else |
|
||||
|
||||
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
|
||||
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
|
||||
thing under test, the source of truth for it, and the backups, at once.
|
||||
|
||||
**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
|
||||
list names two machines and ep0 is not one of them. **That does not make them scratch, and the
|
||||
difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
|
||||
the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
|
||||
*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
|
||||
destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
|
||||
**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
|
||||
read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
|
||||
**`ep0` is Tier 2 — PROTECTED. Operator ruling, 2026-08-03.** D-d named two protected machines and did
|
||||
not name ep0 either way, so this page carried the question in writing for two days and read it the
|
||||
narrow way meanwhile (not protected, but not wipeable). The ruling settles it and **extends D-d's
|
||||
protected list to three machines**: DooPlex, Peti's cluster, ep0.
|
||||
|
||||
The reason it was never really in doubt: ep0 holds the **PBS-DR datastore and the restic copy of a
|
||||
real customer's data**, which is the only off-premises copy that exists. So *deleting datastores,
|
||||
prune jobs, tunnel config or nftables rules* was already forbidden by what it would destroy; the
|
||||
ruling makes the classification say so plainly instead of leaving each session to re-derive it.
|
||||
**Reads are fine** — including the ordinary off-site read a restore-test performs (R-86) — and it is
|
||||
never a drill target. The Hetzner Storage Boxes ride the same reasoning.
|
||||
|
||||
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
|
||||
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
|
||||
@@ -95,11 +98,15 @@ still shares a device with its guest, so a drive failure is **offsite-only recov
|
||||
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
|
||||
enrolled host. No access route from DooPlex, and nothing here needs one.
|
||||
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2, PROTECTED** (operator ruling 2026-08-03)
|
||||
|
||||
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
|
||||
path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune
|
||||
jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
|
||||
path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
|
||||
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching — the 8 GB
|
||||
is comfortable, not unbounded, and the OOM that started R-90 was a 14.46 GB restore read against
|
||||
3.8 GB. Do not delete datastores, prune jobs, tunnel config or nftables rules; never a drill target.
|
||||
**The ordinary off-site READ a restore-test performs is permitted and unchanged by the ruling**
|
||||
(R-86): the classification forbids destruction, not use. The Storage Boxes hold the restic copy —
|
||||
customer documents and photos, on a credential that can still delete (R-95).
|
||||
**Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
|
||||
produced a false "unreachable" verdict (standing rule 2).
|
||||
|
||||
@@ -139,7 +139,7 @@ is not reachable, for two reasons that are each **already-recorded deliberate po
|
||||
`endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
|
||||
|
||||
So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is
|
||||
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90) and fenced by §3 — or *DR tier off* →
|
||||
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90 — **note added 2026-08-03: ep0 has since been rescaled to a CX33 with 8 GB RAM; this journal records what was true when it was written and is deliberately not revised**) and fenced by §3 — or *DR tier off* →
|
||||
no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits.
|
||||
|
||||
Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the
|
||||
|
||||
@@ -1,3 +1,132 @@
|
||||
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
|
||||
|
||||
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
|
||||
offsite tier holds two retained snapshots — `2026-07-27T19:55:41Z` and `2026-07-28T04:49:43Z` —
|
||||
**8 h 54 m apart**, because one is a healing artefact and the other a real weekly run. The mean-gap
|
||||
estimator therefore reads a **weekly** tier as nine-hourly: ×4 gives 36 h, the 7-day floor lifts it to
|
||||
168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm this whole
|
||||
task exists to prevent would have returned within a week, on the box it had just shipped to.**
|
||||
|
||||
`restoreProvenWindow` now takes `max(observed, declared)`. Observation refines a tier's rhythm
|
||||
**upward** and is ignored downward, which is right on its own terms and not merely cautious: a gap
|
||||
SHORTER than the declared rhythm is routine and means nothing — a retry, a manual run, a heal, a
|
||||
catch-up after an outage — while a gap LONGER than it is real information, saying this tier genuinely
|
||||
receives archives less often than the model assumes and its window must widen or it alarms.
|
||||
|
||||
**The cost, stated rather than hidden:** a tier that truly runs faster than its declared rhythm gets a
|
||||
wider window than it strictly needs, i.e. a slower `restore_test_stale` signal. That is the right
|
||||
direction for a signal whose message is *"unverified"*. *"Broken now"* is `restore_test_failed`, which
|
||||
is immediate and untouched.
|
||||
|
||||
Three live-derived cases added to `TestRestoreProvenWindow_Contract`, including the exact 8 h 54 m
|
||||
gap measured on the box; red-proved by restoring the tighten-too branch
|
||||
(`window(pbs, observed=8h54m) = 168h, want 288h`).
|
||||
|
||||
## v0.91.0 — a tier's staleness window learns the tier's own rhythm (2026-08-03, R-86 Part 2)
|
||||
|
||||
**This ships WITH the agent's v0.121.0, not after it.** The agent now proves a tier once per ARCHIVE
|
||||
GENERATION rather than on a 24h timer, so a tier backed up weekly is proved weekly — correctly, and
|
||||
in perfect health. `restoreProvenStaleAfter` was a flat 7 days, and its own comment derived that
|
||||
number from the cadence R-86 removes:
|
||||
|
||||
> *"the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
|
||||
> proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."*
|
||||
|
||||
A weekly tier proved weekly reaches a proof age of **exactly one interval** just before its next
|
||||
proof — 168h against a 168h window, false by a hair — so it did not merely sit near that line, it sat
|
||||
ON it, and any ordinary delay (a late backup, one deferral behind a running backup) tipped it into a
|
||||
nightly alarm about a system that is working. Shipping the agent alone would have converted the
|
||||
improvement into a false alarm.
|
||||
|
||||
**The window is now per tier**, `restoreProvenWindow(tier, observed, ok)`:
|
||||
|
||||
- **the tier's own archive interval**, observed from the host-reports the hub already holds —
|
||||
`pbs_snapshots[]` plus successful `backups[]` attributed by TARGET TYPE (the slice-A.4 rule: a
|
||||
PBS-targeted vzdump appears in both arrays, and classifying by array membership would attribute an
|
||||
offsite archive to the host tier);
|
||||
- **× 4 generations** — the settle generation plus ~3 missed opportunities, deliberately the same
|
||||
tolerance the flat constant expressed. The change is to the RHYTHM, not to the patience;
|
||||
- **floored at 7 days** — the old constant kept as a floor, so no tier is judged more tightly than
|
||||
before;
|
||||
- **capped at 12 days** — strictly inside the 2-week offsite retention with two days to spare, so a
|
||||
tier is never called stale against an archive PBS has already pruned;
|
||||
- **falling back to the DECLARED rhythm** when a box's history is too short to observe one:
|
||||
`backupStaleAfter` (26h) for the host tier and `offsiteBackupStaleAfter` (8d) for the offsite tier
|
||||
— the very thresholds the backup-freshness checker already judges those tiers against. A fresh box
|
||||
with one snapshot has no observable interval, and falling back to the FLOOR there would recreate
|
||||
the false alarm on exactly the tier this task is about.
|
||||
|
||||
**Kept, because both were earned:** absence is UNKNOWN until an anchored window has passed (R-81's
|
||||
structure, untouched), and the stale signal stays edge-triggered. `restore_test_failed` and
|
||||
`restore_test_stale` remain DISTINCT — one says your recovery is broken, the other says it is
|
||||
unverified, and the second is the one that quietly becomes the first.
|
||||
|
||||
**Every reason string now states the window it was judged against** — R-100's corollary: when a
|
||||
verdict changes what it counts from, the alarm text has to change with it, or an operator reads
|
||||
"limit 168h" under a tier actually judged at 288h and dismisses a true alarm.
|
||||
|
||||
**The window READ is unchanged in cost** (14 days), which is both enough to find proof inside the
|
||||
widest window and enough to see two generations of a weekly tier.
|
||||
|
||||
## v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (2026-08-03, R-182)
|
||||
|
||||
**Backfilled 2026-08-03 (R-86 session).** This version was built, deployed and recorded in `REPORT.md`
|
||||
and the R-182 row, but never given a CHANGELOG entry — and `REPORT.md` is overwritten every session,
|
||||
so the per-repo history under-reported what was actually running. The deployed image has been
|
||||
`felhom-hub:0.90.1` since `f21e7ca`.
|
||||
|
||||
Found by reading the first REAL digest, not by design: every app row ended with the same usage clause
|
||||
the mail already prints once on its own Filesystem line. On a two-app box that is untidy; down a list
|
||||
of a dozen it is the same forty characters twelve times, pushing the part that DIFFERS off a phone
|
||||
screen at 07:00 — the only moment that mail has to work.
|
||||
|
||||
The reserve's refusal message is authored for a single-app alert where naming the filesystem is
|
||||
right, so the message is unchanged and the DIGEST trims the duplicate when rendering.
|
||||
`trimRepeatedUsage` removes ONLY an exact "— <target path>:" suffix, so an unrelated reason is
|
||||
untouched and a reason that is nothing but the usage clause is left alone rather than emptied.
|
||||
|
||||
Also inverted the operator half of `TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer`: it
|
||||
required the operator to be e-mailed a per-app capture failure, which was correct when that event was
|
||||
the only signal and is wrong now that it is the RECORD and the digest is the notification. The
|
||||
customer-safety claim is unchanged — R-158's guarantee MOVED, it did not weaken.
|
||||
|
||||
## v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (2026-08-03, R-182)
|
||||
|
||||
**The smallest change on the board with the largest effect on trust: `processOperator`'s cooldown no
|
||||
longer returns bare.** It used to drop the event *before* `LogNotification`, so a suppressed operator
|
||||
alert and an event that never happened were indistinguishable — from the operator's side **and from
|
||||
the hub's own records**. Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two
|
||||
were mailed, and **seven left no row anywhere**. That is why the defect took a day to get the right
|
||||
way round: there was nothing to read.
|
||||
|
||||
A suppressed operator event now writes a `suppressed` row carrying the message and **the key that
|
||||
suppressed it**, so the collision is readable without reading code. This applies to **every** operator
|
||||
event, not only the one that exposed it. It deliberately does **not** change the cooldown's duration
|
||||
or semantics — it makes the drop visible, not absent.
|
||||
|
||||
**`backup_run_failures` — the per-run digest.** One operator mail at the end of a backup run listing
|
||||
every app that failed, its leg and its reason, with the counts and the target filesystem's free
|
||||
space. Added to `allowedEventTypes` **and** to `operatorOnlyEvents` — allowlisting alone does not make
|
||||
an event operator-only, and `FormatCustomerEmail` falls back to the raw English message rather than
|
||||
blocking. A test demonstrates a customer with the type in their enabled list receiving nothing.
|
||||
|
||||
**`recordOnlyEvents` — a third routing class.** Types that are STORED and RECORDED but never mailed.
|
||||
`recovery_unit_capture_failed` moves here: it is the durable per-failure record, and the digest is the
|
||||
notification. Deliberately a register rather than downgrading the severity to `info`, which would have
|
||||
had the same routing effect while relabelling a genuine failure as informational in the events table,
|
||||
the operator UI and every historical query.
|
||||
|
||||
**`cooldownRunSuffix` — the run discriminator.** A sibling of `cooldownTierSuffix` rather than a
|
||||
branch inside it, so `tier` keeps byte-identical semantics and R-97a's tests are untouched. It makes
|
||||
the cooldown effectively inert for the digest, **which is the intent**: a digest is already
|
||||
rate-limited by construction, one per run and only when something failed, so there is nothing for a
|
||||
timer to collapse — while the periodic refresh sweep sends **no** `run_id` and therefore stays under
|
||||
the ordinary hourly cooldown.
|
||||
|
||||
**The e-mail is rendered as a list, not a JSON blob** — the one operator mail with a variable-length
|
||||
payload, and a dozen apps on one line is unreadable on a phone at 07:00. An absent space reading
|
||||
renders as *unavailable*, never as zeros.
|
||||
|
||||
## v0.89.0 — the two halves of decision D-c (2026-08-02, R-167 · R-158)
|
||||
|
||||
**Decision D-c routes two new signals to two different audiences, and the hub is where that routing
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
package api
|
||||
|
||||
import "testing"
|
||||
|
||||
// R-182 — the backup run digest needs the allowlist half, and NOT the customerMessages half.
|
||||
//
|
||||
// A new event type is a pair of register entries, and each half fails differently. For this type the
|
||||
// pair is unusual and that is the point:
|
||||
//
|
||||
// - missing from `allowedEventTypes` → POST /event returns 400 and the digest VANISHES,
|
||||
// which would rebuild the exact silence R-182 exists to end;
|
||||
// - PRESENT in `customerMessages` → the customer would be e-mailed, in Hungarian, a
|
||||
// list of which apps' backups failed and why — operator detail they can take no action on. So
|
||||
// this type must be allowlisted and must NOT have a customer message.
|
||||
//
|
||||
// The customer-facing half of decision D-c is the FILL WARNING, which fires before this and is
|
||||
// actionable (free space, delete files, add a drive). This is the operator's half.
|
||||
//
|
||||
// Operator-only routing itself is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a
|
||||
// customerMessages entry — that assumption shipped in v0.78.0 and was wrong, because
|
||||
// FormatCustomerEmail falls back to the raw message. It is pinned in
|
||||
// `internal/notify/backup_run_digest_test.go`, which demonstrates a customer with the type in their
|
||||
// enabled list receiving nothing.
|
||||
func TestBackupRunDigestIsAllowlisted(t *testing.T) {
|
||||
if !allowedEventTypes["backup_run_failures"] {
|
||||
t.Fatal("backup_run_failures must be in allowedEventTypes, or POST /event 400s and the " +
|
||||
"whole run digest is dropped at the door — the silence R-182 was filed against")
|
||||
}
|
||||
}
|
||||
|
||||
// The per-app event is the RECORD and must not be removed while the digest is the notification.
|
||||
// Deleting it would make the digest the only trace, and a digest that fails to send would then take
|
||||
// the record with it — the coupling R-182's fix exists to break.
|
||||
func TestPerAppCaptureEventStaysAllowlisted(t *testing.T) {
|
||||
if !allowedEventTypes["recovery_unit_capture_failed"] {
|
||||
t.Fatal("recovery_unit_capture_failed was removed from allowedEventTypes — it is the " +
|
||||
"durable per-failure RECORD, and the digest is only the notification; the operator " +
|
||||
"register and every historical query depend on it")
|
||||
}
|
||||
}
|
||||
@@ -1586,6 +1586,11 @@ var allowedEventTypes = map[string]bool{
|
||||
// OPERATOR-ONLY IS ENFORCED BY `notify.operatorOnlyEvents` — see the paragraph above. This entry
|
||||
// alone does NOT make it operator-only.
|
||||
"recovery_unit_capture_failed": true,
|
||||
// R-182. The per-run backup digest: one event at the end of a run, listing every app whose
|
||||
// backup failed or was refused. Allowlisting it is NOT what keeps it away from customers —
|
||||
// `notify.operatorOnlyEvents` is (see the comment there); both entries ship together and
|
||||
// `backup_run_digest_event_test.go` pins the pair.
|
||||
"backup_run_failures": true,
|
||||
|
||||
// Controller-pushed events
|
||||
"controller_started": true,
|
||||
|
||||
@@ -32,15 +32,104 @@ import (
|
||||
// is a NEW monitor written straight after the third, so it copies R-81's verdict structure rather
|
||||
// than re-deriving it. A tier never proven on a newborn box is UNKNOWN, never FAILED.
|
||||
|
||||
// restoreProvenStaleAfter is how long a tier may go unproven before it is called stale.
|
||||
// ── HOW LONG MAY A TIER GO UNPROVEN? (R-86 Part 2) ───────────────────────────────────────────
|
||||
//
|
||||
// Derivation, not a guess: the restore-test cadence is 24h and rotation is oldest-first across two
|
||||
// tiers, so each tier is proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive
|
||||
// missed opportunities before alarming — loud enough to matter, quiet enough not to fire on one
|
||||
// skipped cycle (a deferral behind a long backup is normal, not a fault). It is also comfortably
|
||||
// inside the 2-week offsite retention (operator ruling 2026-07-26), so a tier is never reported
|
||||
// stale against an archive that is about to be pruned anyway.
|
||||
const restoreProvenStaleAfter = 7 * 24 * time.Hour
|
||||
// This was one flat constant, 7 days, and its comment derived that number like this:
|
||||
//
|
||||
// "the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
|
||||
// proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."
|
||||
//
|
||||
// **That premise is exactly what R-86 removed.** The agent no longer tests on an interval at all: a
|
||||
// tier is tested once per ARCHIVE GENERATION — when it holds a settled archive that has not been
|
||||
// proven. A tier backed up weekly is therefore proved weekly, by design and in perfect health, and
|
||||
// against a flat 7-day window it would sit on the line and alarm every night about a system that is
|
||||
// working. Shipping the agent's half alone would have converted the improvement into a false alarm,
|
||||
// which is why the two ship together.
|
||||
//
|
||||
// The window is now derived from **the tier's own backup rhythm**, which the hub can observe from
|
||||
// the reports it already receives, and it keeps everything the constant had earned:
|
||||
//
|
||||
// - absence is UNKNOWN until an anchored window has passed (R-81's structure, untouched);
|
||||
// - the signal stays edge-triggered;
|
||||
// - it never exceeds the offsite retention, so a tier is never called stale against an archive
|
||||
// that is about to be pruned;
|
||||
// - and it is never TIGHTER than the 7 days that were already tolerated.
|
||||
const (
|
||||
// restoreProvenGenerations is how many archive generations may pass unproven before alarming.
|
||||
// 4 = the settle lag's own generation plus ~3 missed opportunities — deliberately the same
|
||||
// tolerance the flat constant expressed, so the change is to the RHYTHM, not to the patience.
|
||||
restoreProvenGenerations = 4
|
||||
|
||||
// restoreProvenWindowFloor is the shortest window that may be applied to any tier. It is the
|
||||
// old constant, kept as a FLOOR rather than deleted: a daily tier computes 4 days from its own
|
||||
// rhythm, and tightening a live threshold is not what this task is for. A deferral behind a
|
||||
// long backup is normal, not a fault.
|
||||
restoreProvenWindowFloor = 7 * 24 * time.Hour
|
||||
|
||||
// restoreProvenWindowCap keeps the window strictly inside the 2-week offsite retention
|
||||
// (operator ruling 2026-07-26) with two days to spare. Beyond it the hub would be judging a
|
||||
// tier against an archive PBS has already pruned — an alarm nobody can act on, and the bound
|
||||
// the old constant respected in its own way.
|
||||
restoreProvenWindowCap = 12 * 24 * time.Hour
|
||||
|
||||
// restoreWindowRead is how far back the hub reads host-reports for this check: far enough to
|
||||
// find proof anywhere inside the widest window, and to see at least two archive generations of
|
||||
// a WEEKLY tier so its rhythm is observable at all.
|
||||
restoreWindowRead = 2 * restoreProvenWindowFloor
|
||||
)
|
||||
|
||||
// declaredArchiveInterval is the rhythm the hub ALREADY attributes to a tier — the same thresholds
|
||||
// the backup-freshness checker judges it against (deadline.go / deadline_tiers.go). It is the
|
||||
// fallback when a box's history is too short to observe a rhythm, and it is the right fallback
|
||||
// precisely because it is not a second opinion: if these two checkers disagreed about how often a
|
||||
// tier is expected to receive an archive, one of them would be alarming on the other's model.
|
||||
//
|
||||
// It is stated per RESTORE tier name ("local"/"pbs" — what the agent reports as source_tier), which
|
||||
// is the same split the backup tiers use under different names ("host"/"offsite").
|
||||
func declaredArchiveInterval(tier string) time.Duration {
|
||||
if tier == "pbs" {
|
||||
return offsiteBackupStaleAfter // 8 days: the weekly cadence plus a day of headroom
|
||||
}
|
||||
return backupStaleAfter // 26 hours: the daily cadence plus headroom
|
||||
}
|
||||
|
||||
// restoreProvenWindow is how long THIS tier may go unproven, given its observed archive interval.
|
||||
//
|
||||
// observedOK=false means the box's retained history did not contain two archive generations for
|
||||
// this tier, so the declared rhythm is used. That fallback matters most for exactly the tier this
|
||||
// task is about: a fresh box with a weekly offsite tier has one snapshot and no observable
|
||||
// interval, and falling back to the FLOOR there would recreate the false alarm.
|
||||
//
|
||||
// OBSERVATION MAY ONLY WIDEN, NEVER TIGHTEN — and this is not caution, it is a live measurement.
|
||||
// On demo-felhom (2026-08-03) the offsite tier's two retained snapshots are `2026-07-27T19:55:41Z`
|
||||
// and `2026-07-28T04:49:43Z`: **8 h 54 m apart**, because one is a healing artefact and the other a
|
||||
// real weekly run. A mean-gap estimate therefore reads a WEEKLY tier as nine-hourly, ×4 gives 36 h,
|
||||
// the floor lifts it to 7 days — and a weekly tier proved weekly reaches ~8.25 days of proof age, so
|
||||
// the false alarm this whole task exists to prevent would have returned within a week, on the very
|
||||
// box it shipped to.
|
||||
//
|
||||
// The asymmetry is right on its own terms too. A gap SHORTER than the declared rhythm is routine and
|
||||
// means nothing — a retry, a manual run, a heal, a catch-up after an outage. A gap LONGER than the
|
||||
// declared rhythm is real information: this tier genuinely receives archives less often than the
|
||||
// model says, and its window must widen or it alarms. So observation refines the rhythm upward and
|
||||
// is ignored downward. The cost is stated plainly: a tier that truly runs FASTER than its declared
|
||||
// rhythm gets a wider window than it strictly needs, i.e. a slower stale signal. That is the right
|
||||
// direction for a signal whose message is "unverified" — "broken NOW" is `restore_test_failed`, and
|
||||
// that one is immediate and unaffected.
|
||||
func restoreProvenWindow(tier string, observed time.Duration, observedOK bool) time.Duration {
|
||||
interval := declaredArchiveInterval(tier)
|
||||
if observedOK && observed > interval {
|
||||
interval = observed
|
||||
}
|
||||
w := time.Duration(restoreProvenGenerations) * interval
|
||||
if w < restoreProvenWindowFloor {
|
||||
w = restoreProvenWindowFloor
|
||||
}
|
||||
if w > restoreProvenWindowCap {
|
||||
w = restoreProvenWindowCap
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// Event types. Operator-tier only — see the dispatcher note in RestoreTestChecker.
|
||||
const (
|
||||
@@ -156,12 +245,13 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
return
|
||||
}
|
||||
|
||||
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-2*restoreProvenStaleAfter))
|
||||
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-restoreWindowRead))
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] restore-test check: window read failed for %s: %v", customerID, err)
|
||||
return
|
||||
}
|
||||
proven := lastProvenPerTier(rows)
|
||||
intervals := observedArchiveIntervals(rows)
|
||||
|
||||
first, ferr := c.store.GetFirstHostReportAt(customerID)
|
||||
if ferr != nil {
|
||||
@@ -170,7 +260,9 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
}
|
||||
|
||||
for _, tier := range tiers {
|
||||
v := assessRestoreProven(tier, proven[tier], first, now)
|
||||
observed, observedOK := intervals[tier]
|
||||
window := restoreProvenWindow(tier, observed, observedOK)
|
||||
v := assessRestoreProven(tier, proven[tier], first, now, window)
|
||||
key := customerID + "|" + tier
|
||||
c.mu.Lock()
|
||||
prev := c.staleStates[key]
|
||||
@@ -192,37 +284,120 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
}
|
||||
}
|
||||
|
||||
// assessRestoreProven is the per-tier verdict. PURE (now injected) so the policy is unit-tested —
|
||||
// the property that made R-81 provable, kept deliberately.
|
||||
// assessRestoreProven is the per-tier verdict. PURE (now and the window injected) so the policy is
|
||||
// unit-tested — the property that made R-81 provable, kept deliberately.
|
||||
//
|
||||
// no proof, anchor NOT elapsed → UNKNOWN (newborn box; never an alarm)
|
||||
// no proof, anchor elapsed → MISSED
|
||||
// proof older than the window → MISSED
|
||||
// otherwise → OK
|
||||
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time) backupAssessment {
|
||||
//
|
||||
// `window` is now the TIER'S OWN (R-86 Part 2) rather than one constant for every tier, and every
|
||||
// reason string states the window it was judged against. That is R-100's corollary applied here:
|
||||
// when a verdict changes what it counts from, the alarm text has to change with it, or an operator
|
||||
// reads "limit 168h" under a tier that was actually judged at 288h and dismisses a true alarm.
|
||||
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time, window time.Duration) backupAssessment {
|
||||
if provenAt.IsZero() {
|
||||
if firstReportAt.IsZero() {
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: never restore-proven, and no first-contact anchor to defer against", tier)}
|
||||
}
|
||||
watched := now.Sub(firstReportAt)
|
||||
if watched <= restoreProvenStaleAfter {
|
||||
if watched <= window {
|
||||
return backupAssessment{verdict: verdictUnknown,
|
||||
reason: fmt.Sprintf("%s tier: not restore-proven yet, but only watching for %s (grace %s since first contact %s) — newborn, not a fault",
|
||||
tier, watched.Round(time.Hour), restoreProvenStaleAfter, firstReportAt.Format(time.RFC3339))}
|
||||
tier, watched.Round(time.Hour), window, firstReportAt.Format(time.RFC3339))}
|
||||
}
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s) — the tier is unverified, not known-broken",
|
||||
tier, watched.Round(time.Hour), restoreProvenStaleAfter)}
|
||||
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
|
||||
tier, watched.Round(time.Hour), window)}
|
||||
}
|
||||
if age := now.Sub(provenAt); age > restoreProvenStaleAfter {
|
||||
if age := now.Sub(provenAt); age > window {
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s) — the tier is unverified, not known-broken",
|
||||
tier, age.Round(time.Hour), restoreProvenStaleAfter)}
|
||||
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
|
||||
tier, age.Round(time.Hour), window)}
|
||||
}
|
||||
return backupAssessment{verdict: verdictOK}
|
||||
}
|
||||
|
||||
// observedArchiveIntervals estimates how often each RESTORE tier actually receives an archive, from
|
||||
// the host-reports the hub already holds. Keyed by restore-tier name ("local"/"pbs").
|
||||
//
|
||||
// Evidence is every distinct archive timestamp in the window: `pbs_snapshots[]` for the offsite
|
||||
// tier (PBS enumerates its whole retention in each report, so one report usually settles the
|
||||
// question) and successful `backups[]` records attributed by TARGET TYPE for both tiers — the
|
||||
// slice-A.4 rule, because a PBS-targeted vzdump appears in BOTH arrays and classifying by array
|
||||
// membership would attribute an offsite archive to the host tier.
|
||||
//
|
||||
// The estimate is the MEAN gap: (newest − oldest) / (generations − 1). It needs two generations;
|
||||
// with fewer, ok=false and the caller falls back to the declared rhythm. It is deliberately crude,
|
||||
// and can afford to be: restoreProvenWindow clamps the result between a 7-day floor and a 12-day
|
||||
// cap, so the only discrimination this has to get right is "roughly daily" versus "several days or
|
||||
// slower" — which is exactly the distinction that turns a healthy weekly tier into a false alarm.
|
||||
func observedArchiveIntervals(rows []store.HostReportRow) map[string]time.Duration {
|
||||
seen := map[string]map[int64]struct{}{ // tier → set of archive unix times
|
||||
"local": {},
|
||||
"pbs": {},
|
||||
}
|
||||
add := func(tier string, t time.Time) {
|
||||
if t.IsZero() {
|
||||
return
|
||||
}
|
||||
seen[tier][t.UTC().Unix()] = struct{}{}
|
||||
}
|
||||
|
||||
for _, r := range rows {
|
||||
var hr hostReportBackups
|
||||
if json.Unmarshal([]byte(r.ReportJSON), &hr) != nil {
|
||||
continue // one malformed retained report must not blind the scan
|
||||
}
|
||||
pbs := pbsTargetSet(hr)
|
||||
for _, ps := range hr.PBSSnapshots {
|
||||
if t, ok := parseBackupTime(ps.BackupTime); ok {
|
||||
add("pbs", t)
|
||||
}
|
||||
}
|
||||
for _, b := range hr.Backups {
|
||||
if !b.Success {
|
||||
continue
|
||||
}
|
||||
t, ok := parseBackupTime(b.StartedAt)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if pbs[b.TargetID] {
|
||||
add("pbs", t)
|
||||
} else {
|
||||
add("local", t)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
out := map[string]time.Duration{}
|
||||
for tier, set := range seen {
|
||||
if len(set) < 2 {
|
||||
continue // not observable — the caller uses the declared rhythm
|
||||
}
|
||||
var oldest, newest int64
|
||||
first := true
|
||||
for ts := range set {
|
||||
if first || ts < oldest {
|
||||
oldest = ts
|
||||
}
|
||||
if first || ts > newest {
|
||||
newest = ts
|
||||
}
|
||||
first = false
|
||||
}
|
||||
span := time.Duration(newest-oldest) * time.Second
|
||||
if span <= 0 {
|
||||
continue
|
||||
}
|
||||
out[tier] = span / time.Duration(len(set)-1)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// expectedRestoreTiers names the tiers this box actually HAS, so a box without an offsite tier is
|
||||
// never reported stale for one. Same gate as Slice C's `expected`, and for the same reason: without
|
||||
// it every box lacking a tier would alarm once the anchor elapsed — absence-is-not-failure,
|
||||
|
||||
@@ -182,26 +182,33 @@ func TestRestoreTest_NewbornDoesNotAlarm(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The boundary, pinned by name so a refactor has to delete an obviously-named contract.
|
||||
// The boundary, pinned by name so a refactor has to delete an obviously-named contract. R-86 made
|
||||
// the limit per-tier, so the anchor is now measured against THE TIER'S OWN window — here the local
|
||||
// tier's, which clamps to the 7-day floor and so keeps this contract numerically identical to the
|
||||
// one the flat constant expressed.
|
||||
func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
window := restoreProvenWindow("local", 24*time.Hour, true)
|
||||
if window != restoreProvenWindowFloor {
|
||||
t.Fatalf("precondition: a daily local tier must clamp to the floor; got %s", window)
|
||||
}
|
||||
cases := []struct {
|
||||
name string
|
||||
watched time.Duration
|
||||
wantMissed bool
|
||||
}{
|
||||
{"newborn, 1h", time.Hour, false},
|
||||
{"just inside", restoreProvenStaleAfter - time.Minute, false},
|
||||
{"exactly at the limit", restoreProvenStaleAfter, false},
|
||||
{"just outside", restoreProvenStaleAfter + time.Minute, true},
|
||||
{"just inside", window - time.Minute, false},
|
||||
{"exactly at the limit", window, false},
|
||||
{"just outside", window + time.Minute, true},
|
||||
{"long past", 30 * 24 * time.Hour, true},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := assessRestoreProven("pbs", time.Time{}, now.Add(-c.watched), now)
|
||||
got := assessRestoreProven("local", time.Time{}, now.Add(-c.watched), now, window)
|
||||
if got.missed() != c.wantMissed {
|
||||
t.Fatalf("CONTRACT VIOLATED: unproven for %s (limit %s) → missed=%v, want %v (reason %q)",
|
||||
c.watched, restoreProvenStaleAfter, got.missed(), c.wantMissed, got.reason)
|
||||
c.watched, window, got.missed(), c.wantMissed, got.reason)
|
||||
}
|
||||
if !c.wantMissed && got.verdict != verdictUnknown {
|
||||
t.Fatalf("a deferred tier must be UNKNOWN (visible), not OK; got verdict=%d", got.verdict)
|
||||
@@ -215,15 +222,17 @@ func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.
|
||||
func TestRestoreTest_StaleIsSeparateFromFailure(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
stale := assessRestoreProven("pbs", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now)
|
||||
// A DAILY tier judged on its own rhythm: the window clamps to the 7-day floor.
|
||||
daily := restoreProvenWindow("local", 24*time.Hour, true)
|
||||
stale := assessRestoreProven("local", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
|
||||
if !stale.missed() {
|
||||
t.Fatalf("a tier last proven 9 days ago (limit %s) must be stale; got %q", restoreProvenStaleAfter, stale.reason)
|
||||
t.Fatalf("a daily tier last proven 9 days ago (limit %s) must be stale; got %q", daily, stale.reason)
|
||||
}
|
||||
// The wording must not read as "broken" — that is the other signal.
|
||||
if !strings.Contains(stale.reason, "unverified, not known-broken") {
|
||||
t.Fatalf("staleness must say UNVERIFIED, not broken — merging the two is the thing this avoids; got %q", stale.reason)
|
||||
}
|
||||
fresh := assessRestoreProven("pbs", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now)
|
||||
fresh := assessRestoreProven("local", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
|
||||
if fresh.verdict != verdictOK {
|
||||
t.Fatalf("a tier proven 2 days ago is fine; got verdict=%d reason=%q", fresh.verdict, fresh.reason)
|
||||
}
|
||||
@@ -283,3 +292,208 @@ func boolStr(b bool) string {
|
||||
}
|
||||
return "false"
|
||||
}
|
||||
|
||||
// ── SCENARIO G — a healthy WEEKLY tier is never reported stale (R-86 Part 2) ─────────────────
|
||||
//
|
||||
// This is the test that pins the false alarm this change would otherwise have CREATED. The agent
|
||||
// now proves a tier once per archive generation, so a weekly offsite tier is proved weekly — in
|
||||
// perfect health. Against the old flat 7-day window it would sit on the line and alarm every night.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): pin the window flat, as it was —
|
||||
//
|
||||
// - window := restoreProvenWindow(tier, observed, observedOK)
|
||||
// - window := restoreProvenWindowFloor // the pre-R-86 flat 7 days
|
||||
//
|
||||
// → --- FAIL: TestRestoreTest_HealthyWeeklyTierIsNeverStale
|
||||
//
|
||||
// week 0: a weekly tier proved on its own archive must never be stale (proof age 172h0m0s,
|
||||
// window 168h0m0s); verdict=2 reason="pbs tier: last successful restore-test was 172h0m0s ago
|
||||
// (limit 168h0m0s, this tier's own backup rhythm) — the tier is unverified, not known-broken"
|
||||
//
|
||||
// Restored. The mutation is one line because the whole of Part 2 is one decision: whose rhythm.
|
||||
//
|
||||
// NOTE, because it is the finding this test nearly hid: the FIRST version of this fixture had NO
|
||||
// jitter, and it PASSED under the mutation. A perfectly regular weekly tier's proof age reaches
|
||||
// EXACTLY 168h just before the next proof, and `age > window` is false by a hair — a hollow test
|
||||
// that would have shipped Part 1 and its false alarm together. The jitter below is what makes this
|
||||
// a test, and it is also the truth about the old constant: a healthy weekly tier did not merely sit
|
||||
// near the line, it sat ON it, so any ordinary delay tipped it over.
|
||||
func TestRestoreTest_HealthyWeeklyTierIsNeverStale(t *testing.T) {
|
||||
start := time.Date(2026, 6, 1, 3, 0, 0, 0, time.UTC)
|
||||
firstContact := start.Add(-24 * time.Hour)
|
||||
|
||||
// The observable rhythm of a weekly tier, as the hub would compute it from the reports. No
|
||||
// assertion about the window ITSELF here on purpose: that is the mechanism, and it is pinned in
|
||||
// TestRestoreProvenWindow_Contract. What this test asserts is the CONSEQUENCE — does the alarm
|
||||
// fire? — because R-97b proved a mechanism and shipped a broken consequence anyway.
|
||||
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
|
||||
|
||||
// Walk several weeks of a HEALTHY tier, with the jitter a real one has: the backup does not land
|
||||
// to the second, and a restore-test can be deferred one evaluation behind a running backup.
|
||||
//
|
||||
// The jitter is the point. A perfectly regular weekly tier's proof reaches an age of EXACTLY one
|
||||
// interval (168h) just before the next proof, and against a flat 168h window `age > window` is
|
||||
// false by a hair — so a regular fixture would pass against the very constant this task must
|
||||
// change, and prove nothing. That is the brief's "sits exactly on that line": every real-world
|
||||
// delay pushes it over, and the alarm is about a system that is working.
|
||||
settle, evalLatency := 24*time.Hour, 6*time.Hour
|
||||
archiveLate := []time.Duration{0, 4 * time.Hour, 2 * time.Hour, 6 * time.Hour, 0, 3 * time.Hour}
|
||||
deferred := []time.Duration{0, 0, 6 * time.Hour, 0, 0, 6 * time.Hour} // one evaluation behind a backup
|
||||
|
||||
archiveAt := func(week int) time.Time {
|
||||
return start.AddDate(0, 0, 7*week).Add(archiveLate[week])
|
||||
}
|
||||
provenAt := func(week int) time.Time {
|
||||
return archiveAt(week).Add(settle + evalLatency + deferred[week])
|
||||
}
|
||||
|
||||
var worst time.Duration
|
||||
for week := 0; week+1 < len(archiveLate); week++ {
|
||||
// The widest the proof's age ever gets: the instant before the NEXT week's proof lands.
|
||||
now := provenAt(week + 1).Add(-time.Second)
|
||||
age := now.Sub(provenAt(week))
|
||||
if age > worst {
|
||||
worst = age
|
||||
}
|
||||
v := assessRestoreProven("pbs", provenAt(week), firstContact, now, weekly)
|
||||
if v.verdict != verdictOK {
|
||||
t.Fatalf("week %d: a weekly tier proved on its own archive must never be stale (proof age %s, window %s); verdict=%d reason=%q",
|
||||
week, age.Round(time.Hour), weekly, v.verdict, v.reason)
|
||||
}
|
||||
}
|
||||
// The fixture must actually EXERCISE the boundary — a jitter-free walk would sit at exactly one
|
||||
// interval and pass against a flat 7-day window, which is the hollow version of this test.
|
||||
if worst <= restoreProvenWindowFloor {
|
||||
t.Fatalf("this fixture never exceeds the old flat window (worst proof age %s) — it cannot detect the defect it exists for", worst)
|
||||
}
|
||||
}
|
||||
|
||||
// ...and a weekly tier that genuinely STOPS being proved must still alarm. A window that never
|
||||
// fires is not a fix, it is a deletion.
|
||||
func TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
|
||||
|
||||
v := assessRestoreProven("pbs", now.Add(-weekly-time.Hour), now.Add(-90*24*time.Hour), now, weekly)
|
||||
if !v.missed() {
|
||||
t.Fatalf("a weekly tier unproven for longer than its own window MUST alarm; got verdict=%d reason=%q", v.verdict, v.reason)
|
||||
}
|
||||
if !strings.Contains(v.reason, weekly.String()) {
|
||||
t.Fatalf("the alarm must state the window it was judged against (R-100's corollary); got %q", v.reason)
|
||||
}
|
||||
}
|
||||
|
||||
// The window's own contract: derived from the tier's rhythm, floored, capped, and never dependent
|
||||
// on an unobservable history for the tier that would suffer most from a wrong answer.
|
||||
func TestRestoreProvenWindow_Contract(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
tier string
|
||||
observed time.Duration
|
||||
observedOK bool
|
||||
want time.Duration
|
||||
}{
|
||||
{"daily local clamps to the floor", "local", 24 * time.Hour, true, restoreProvenWindowFloor},
|
||||
{"weekly pbs widens", "pbs", 7 * 24 * time.Hour, true, restoreProvenWindowCap},
|
||||
{"3-day tier sits between", "pbs", 72 * time.Hour, true, 12 * 24 * time.Hour},
|
||||
{"unobservable local falls back to its declared rhythm", "local", 0, false, restoreProvenWindowFloor},
|
||||
{"unobservable pbs falls back WIDE, not to the floor", "pbs", 0, false, restoreProvenWindowCap},
|
||||
{"a nonsense zero interval is ignored", "pbs", 0, true, restoreProvenWindowCap},
|
||||
// MEASURED ON THE LIVE BOX, and the reason observation may only WIDEN. demo-felhom's two
|
||||
// retained PBS snapshots sit 8h54m apart (one is a healing artefact), so a mean-gap estimate
|
||||
// reads a WEEKLY tier as nine-hourly. Taking that at face value gives 4x9h = 36h → the 7-day
|
||||
// floor → and a weekly tier proved weekly (~8.25d of proof age) alarms within a week of this
|
||||
// shipping, on the box it shipped to.
|
||||
{"a short observed gap must NOT tighten a weekly tier", "pbs", 8*time.Hour + 54*time.Minute, true, restoreProvenWindowCap},
|
||||
{"a short observed gap must not tighten the host tier either", "local", 30 * time.Minute, true, restoreProvenWindowFloor},
|
||||
// ...but a tier that genuinely runs SLOWER than its declared rhythm still widens.
|
||||
{"a genuinely slower tier widens", "local", 4 * 24 * time.Hour, true, restoreProvenWindowCap},
|
||||
}
|
||||
// The relationship Part 1 depends on: a weekly tier's window must be WIDER than a daily tier's,
|
||||
// or proving weekly (which is now correct behaviour) alarms on itself.
|
||||
if restoreProvenWindow("pbs", 7*24*time.Hour, true) <= restoreProvenWindow("local", 24*time.Hour, true) {
|
||||
t.Fatal("a weekly tier must earn a wider window than a daily one — otherwise R-86's agent half alarms about itself")
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := restoreProvenWindow(c.tier, c.observed, c.observedOK)
|
||||
if got != c.want {
|
||||
t.Fatalf("window(%s, observed=%s ok=%v) = %s, want %s", c.tier, c.observed, c.observedOK, got, c.want)
|
||||
}
|
||||
if got < restoreProvenWindowFloor || got > restoreProvenWindowCap {
|
||||
t.Fatalf("every window must stay inside [%s, %s]; got %s", restoreProvenWindowFloor, restoreProvenWindowCap, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The rhythm must be OBSERVED from the reports, not assumed — including the slice-A.4 rule that a
|
||||
// PBS-targeted vzdump appears in both arrays and must be attributed by TARGET TYPE.
|
||||
func TestObservedArchiveIntervals_FromReports(t *testing.T) {
|
||||
base := time.Date(2026, 7, 1, 2, 0, 0, 0, time.UTC)
|
||||
mk := func(localAt []time.Time, pbsAt []time.Time) string {
|
||||
type stg struct{ Name, Type, Content string }
|
||||
type bk struct {
|
||||
TargetID string `json:"target_id"`
|
||||
Success bool `json:"success"`
|
||||
StartedAt string `json:"started_at"`
|
||||
}
|
||||
type snap struct {
|
||||
BackupTime string `json:"backup_time"`
|
||||
}
|
||||
payload := struct {
|
||||
StorageTargets []struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
} `json:"storage_targets"`
|
||||
Backups []bk `json:"backups"`
|
||||
PBSSnapshots []snap `json:"pbs_snapshots"`
|
||||
}{}
|
||||
payload.StorageTargets = append(payload.StorageTargets, struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
}{"felhom-backup", "dir", "backup"}, struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
}{"felhom-pbs", "pbs", "backup"})
|
||||
for _, at := range localAt {
|
||||
payload.Backups = append(payload.Backups, bk{"felhom-backup", true, at.Format(time.RFC3339)})
|
||||
}
|
||||
for _, at := range pbsAt {
|
||||
// The SAME archive appears as a vzdump record AND as a snapshot — slice A.4.
|
||||
payload.Backups = append(payload.Backups, bk{"felhom-pbs", true, at.Format(time.RFC3339)})
|
||||
payload.PBSSnapshots = append(payload.PBSSnapshots, snap{at.Format(time.RFC3339)})
|
||||
}
|
||||
b, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
rows := []store.HostReportRow{
|
||||
{ReportJSON: mk(
|
||||
[]time.Time{base, base.AddDate(0, 0, 1), base.AddDate(0, 0, 2)},
|
||||
[]time.Time{base, base.AddDate(0, 0, 7)},
|
||||
)},
|
||||
{ReportJSON: `{{{malformed`}, // must not blind the scan
|
||||
}
|
||||
|
||||
got := observedArchiveIntervals(rows)
|
||||
if d, ok := got["local"]; !ok || d != 24*time.Hour {
|
||||
t.Fatalf("a daily host tier must be observed as ~24h; got %s ok=%v", d, ok)
|
||||
}
|
||||
if d, ok := got["pbs"]; !ok || d != 7*24*time.Hour {
|
||||
t.Fatalf("a weekly offsite tier must be observed as ~7d — and its vzdump record must NOT be "+
|
||||
"counted into the host tier (slice A.4); got %s ok=%v", d, ok)
|
||||
}
|
||||
|
||||
// One generation is not a rhythm: unobservable, so the caller falls back to the declared one.
|
||||
single := []store.HostReportRow{{ReportJSON: mk(nil, []time.Time{base})}}
|
||||
if d, ok := observedArchiveIntervals(single)["pbs"]; ok {
|
||||
t.Fatalf("one archive cannot yield an interval; got %s", d)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,301 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-182 — one e-mail per backup run, and nothing dropped without a trace.
|
||||
//
|
||||
// MEASURED, NOT SUPPOSED. On 2026-08-03 nine `recovery_unit_capture_failed` events reached the hub
|
||||
// and TWO operator e-mails went out. The operator cooldown key is
|
||||
// `customerID + ":" + eventType + cooldownTierSuffix(details)`, that event carries `app` but no
|
||||
// `tier`, so the key held no app identifier: the first refused app took the hour's slot and every
|
||||
// other app's failure was discarded — **before `LogNotification`**, so it left no row on any channel
|
||||
// and could not be found afterwards.
|
||||
//
|
||||
// The operator ruled against the obvious fix (putting `app` in the key), because on a full disk that
|
||||
// is one e-mail per app. These pin the shape that replaced it.
|
||||
|
||||
// ── Scenario D — a suppressed operator event leaves a trace ───────────────────────────────────────
|
||||
|
||||
// The bare `return` at the cooldown is the whole reason this defect took a day to get the right way
|
||||
// round: there was nothing to read. A drop must be as visible in the record as a send.
|
||||
//
|
||||
// DELIBERATELY EXERCISED ON A DIFFERENT EVENT TYPE than the one that exposed the defect.
|
||||
// `recovery_unit_capture_failed` is now record-only and never reaches the cooldown at all, so using
|
||||
// it here would prove nothing. `whole_guest_backup_failed` is an ordinary operator event, and using
|
||||
// it pins §2.1's actual claim: the suppression row applies to EVERY operator event, not only the one
|
||||
// that happened to be measured.
|
||||
func TestSuppressedOperatorEvent_LeavesARow(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
// Two events of the SAME type with no discriminator — the second must be suppressed.
|
||||
d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "opengist failed", `{"app":"opengist"}`, "controller")
|
||||
d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "privatebin failed", `{"app":"privatebin"}`, "controller")
|
||||
|
||||
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
|
||||
t.Fatalf("operator mails = %d, want 1 — the premise of this test is that the second IS suppressed", got)
|
||||
}
|
||||
|
||||
rows, err := st.GetRecentNotifications("c1", 50)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var sentRows, suppressed []store2Row
|
||||
for _, r := range rows {
|
||||
if r.Channel != "operator" || r.EventType != "whole_guest_backup_failed" {
|
||||
continue
|
||||
}
|
||||
switch r.Status {
|
||||
case "sent":
|
||||
sentRows = append(sentRows, store2Row{r.Status, r.Message, r.ErrorMessage})
|
||||
case "suppressed":
|
||||
suppressed = append(suppressed, store2Row{r.Status, r.Message, r.ErrorMessage})
|
||||
}
|
||||
}
|
||||
if len(sentRows) != 1 {
|
||||
t.Fatalf("want 1 'sent' operator row, got %d", len(sentRows))
|
||||
}
|
||||
if len(suppressed) != 1 {
|
||||
t.Fatalf("want 1 'suppressed' operator row, got %d — a cooldown drop that writes NOTHING is "+
|
||||
"indistinguishable from an event that never happened, which is exactly how seven "+
|
||||
"failures went missing on 2026-08-03", len(suppressed))
|
||||
}
|
||||
// The row must name the app that was dropped, or it records that something was suppressed
|
||||
// without recording WHAT — half a fix.
|
||||
if !strings.Contains(suppressed[0].message, "privatebin") {
|
||||
t.Fatalf("the suppressed row does not name the dropped event: %q", suppressed[0].message)
|
||||
}
|
||||
// And it must carry the key, so the reason it collided is readable without reading code.
|
||||
if !strings.Contains(suppressed[0].errMsg, "key=") {
|
||||
t.Fatalf("the suppressed row does not carry the cooldown key: %q", suppressed[0].errMsg)
|
||||
}
|
||||
}
|
||||
|
||||
type store2Row struct{ status, message, errMsg string }
|
||||
|
||||
// ── Scenario E — two runs in a day each report ───────────────────────────────────────────────────
|
||||
|
||||
// The operator ruled explicitly on this: someone pressing the backup button is actively trying to
|
||||
// get a backup, and finding out tomorrow would be worse than an extra mail in a rare case.
|
||||
func TestTwoRunsInAnHour_BothReport(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
nightly := `{"run_id":"run-a","run_kind":"nightly","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
|
||||
manual := `{"run_id":"run-b","run_kind":"manual","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
|
||||
|
||||
d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", nightly, "controller")
|
||||
d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", manual, "controller")
|
||||
|
||||
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 2 {
|
||||
t.Fatalf("operator mails = %d, want 2 — the 1-hour cooldown swallowed the manual run's "+
|
||||
"digest, which is the fix reappearing one level up: press the button, the run fails, "+
|
||||
"and you are told nothing because the machine already wrote this hour", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The run discriminator must be NARROW, exactly like its `tier` sibling — empty unless the producer
|
||||
// opts in — or every existing event type's cooldown silently stops collapsing anything.
|
||||
func TestCooldownRunSuffix_EmptyForEverythingElse(t *testing.T) {
|
||||
cases := []struct{ name, details string }{
|
||||
{"no details", ""},
|
||||
{"details without run_id", `{"app":"opengist","error":"boom"}`},
|
||||
{"empty run_id value", `{"run_id":""}`},
|
||||
{"malformed json", `{{{nope`},
|
||||
{"run_id mentioned in a STRING, not as a key", `{"error":"the run_id: abc failed"}`},
|
||||
{"null details", `null`},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := cooldownRunSuffix(c.details); got != "" {
|
||||
t.Errorf("%s: suffix must be EMPTY so every other type's cooldown is unchanged, got %q", c.name, got)
|
||||
}
|
||||
}
|
||||
if got := cooldownRunSuffix(`{"run_id":"run-a"}`); got != ":run-a" {
|
||||
t.Fatalf("suffix should be the run id, got %q", got)
|
||||
}
|
||||
// The two suffixes must not interfere: a tier event still keys on its tier and nothing else.
|
||||
if got := cooldownTierSuffix(`{"tier":"felhom-pbs"}`) + cooldownRunSuffix(`{"tier":"felhom-pbs"}`); got != ":felhom-pbs" {
|
||||
t.Fatalf("a tier-only event's key changed to %q — R-97a's behaviour must be byte-identical", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario G — the customer never receives the digest ──────────────────────────────────────────
|
||||
|
||||
// v0.78.0 asserted in a COMMENT that a type with no `customerMessages` entry structurally cannot
|
||||
// reach a customer. It can: FormatCustomerEmail falls back to the raw English message and the only
|
||||
// customer gate is configuration. So this is demonstrated, not argued.
|
||||
func TestDigest_IsOperatorOnly_EvenWithAWideEnabledList(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
// A customer who has enabled EVERYTHING, including this type by name.
|
||||
if err := st.SaveNotificationPrefs("c1", "cust@example.com",
|
||||
[]string{"backup_run_failures", "node_down", "disk_warning"}, 6); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
details := `{"run_id":"run-a","run_kind":"nightly","failed":1,"attempted":3,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"}]}`
|
||||
d.ProcessEvent("c1", "backup_run_failures", "error", "1 of 3 apps failed", details, "controller")
|
||||
|
||||
if got := mailsFor(*sent, "cust@example.com"); len(got) != 0 {
|
||||
t.Fatalf("the CUSTOMER received an operator digest (%d mails) — a list of which apps' "+
|
||||
"backups failed is not something they can act on, and the raw body is English", len(got))
|
||||
}
|
||||
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
|
||||
t.Fatalf("operator mails = %d, want 1", got)
|
||||
}
|
||||
if !operatorOnlyEvents["backup_run_failures"] {
|
||||
t.Fatal("backup_run_failures is not in operatorOnlyEvents — allowlisting alone does NOT " +
|
||||
"keep it from a customer; that assumption shipped once and was wrong (v0.78.0)")
|
||||
}
|
||||
}
|
||||
|
||||
// ── Part 3 — the e-mail a person actually reads ──────────────────────────────────────────────────
|
||||
|
||||
func TestDigestEmail_ListsAppsLegsAndReasons(t *testing.T) {
|
||||
details := `{"run_id":"run-a","run_kind":"nightly","failed":3,"attempted":40,` +
|
||||
`"target_path":"/mnt/sys_drive","used_gb":64.3,"avail_gb":0.9,"total_gb":68.7,` +
|
||||
`"used_percent":94,"space_known":true,"apps":[` +
|
||||
`{"app":"opengist","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
|
||||
`{"app":"privatebin","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
|
||||
`{"app":"immich","leg":"database dump","reason":"pg_dump: connection refused"}]}`
|
||||
|
||||
subject, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error",
|
||||
"3 of 40 apps failed to back up", details)
|
||||
|
||||
// The subject must carry the counts: the operator's first decision is made from it alone.
|
||||
for _, want := range []string{"demo-hp", "3 of 40", "nightly"} {
|
||||
if !strings.Contains(subject, want) {
|
||||
t.Errorf("subject %q missing %q", subject, want)
|
||||
}
|
||||
}
|
||||
// Every app, its leg and its reason.
|
||||
for _, want := range []string{
|
||||
"opengist", "privatebin", "immich",
|
||||
"volume dump", "database dump",
|
||||
"below the reserve", "pg_dump: connection refused",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("body missing %q:\n%s", want, body)
|
||||
}
|
||||
}
|
||||
// The counts and the free space, so "one broken app" and "a full disk" read differently.
|
||||
if !strings.Contains(body, "3 of 40") {
|
||||
t.Errorf("body does not carry the failed-of-attempted count:\n%s", body)
|
||||
}
|
||||
if !strings.Contains(body, "0.9 GB free") {
|
||||
t.Errorf("body does not carry the free space:\n%s", body)
|
||||
}
|
||||
// It must NOT be a JSON blob.
|
||||
if strings.Contains(body, `"apps":[`) {
|
||||
t.Errorf("the digest rendered as raw JSON — unreadable on a phone at 07:00:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// An absent space reading must render as unavailable, never as zeros: "0 GB free" and "we could not
|
||||
// look" are opposite diagnoses, and the operator acts differently on each.
|
||||
func TestDigestEmail_UnknownSpaceIsNotZero(t *testing.T) {
|
||||
details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/x",` +
|
||||
`"space_known":false,"apps":[{"app":"a","leg":"capture","reason":"boom"}]}`
|
||||
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
|
||||
if strings.Contains(body, "0.0 GB free") {
|
||||
t.Fatalf("an unreadable filesystem rendered as zeros:\n%s", body)
|
||||
}
|
||||
if !strings.Contains(body, "unavailable") {
|
||||
t.Fatalf("an unreadable filesystem must say so:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// A payload that cannot be parsed must still produce a mail — degraded, never swallowed.
|
||||
func TestDigestEmail_UnparseableDetailsStillMails(t *testing.T) {
|
||||
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "something failed", `{{{`)
|
||||
if body == "" || !strings.Contains(body, "something failed") {
|
||||
t.Fatalf("an unparseable digest lost the mail entirely:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario C — every failure is RECORDED, e-mailed or not ──────────────────────────────────────
|
||||
|
||||
// The per-app event is the record; the digest is the notification. The record must not inherit the
|
||||
// notification's conditions — no cooldown, no preferences, no dependence on a mail going out.
|
||||
func TestPerAppFailure_IsRecordedButNotMailed(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
apps := []string{"opengist", "privatebin", "immich", "homebox", "nextcloud"}
|
||||
for _, a := range apps {
|
||||
d.ProcessEvent("c1", "recovery_unit_capture_failed", "error",
|
||||
"Recovery unit capture FAILED for \""+a+"\"", `{"app":"`+a+`"}`, "controller")
|
||||
}
|
||||
|
||||
// NOT mailed — the digest is the notification.
|
||||
if got := len(*sent); got != 0 {
|
||||
t.Fatalf("%d mail(s) sent for per-app failures — they are the RECORD; one mail per app on a "+
|
||||
"full disk is the volume problem wearing the correctness problem's clothes, which is "+
|
||||
"exactly what the operator ruled against", got)
|
||||
}
|
||||
|
||||
// But ALL FIVE recorded — this is the assertion yesterday's defect would have failed: nine
|
||||
// arrived, two were mailed, seven left no row anywhere.
|
||||
rows, err := st.GetRecentNotifications("c1", 50)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, r := range rows {
|
||||
if r.EventType == "recovery_unit_capture_failed" && r.Status == "recorded" {
|
||||
for _, a := range apps {
|
||||
if strings.Contains(r.Message, a) {
|
||||
seen[a] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(seen) != len(apps) {
|
||||
t.Fatalf("only %d of %d per-app failures were recorded (%v) — a failure that produced no row "+
|
||||
"anywhere is the measured defect of 2026-08-03", len(seen), len(apps), seen)
|
||||
}
|
||||
}
|
||||
|
||||
// The per-app reason must not repeat the filesystem figures the digest already prints once. Reviewed
|
||||
// as copy against the first real digest, not designed in the abstract.
|
||||
func TestDigestEmail_ReasonDoesNotRepeatTheUsageLine(t *testing.T) {
|
||||
reason := "refused: below the reserve (reserve: 97% used or 1.0 GiB free) — /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free"
|
||||
details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/sys_drive",` +
|
||||
`"used_gb":65,"avail_gb":0.2,"total_gb":68.7,"used_percent":95,"space_known":true,` +
|
||||
`"apps":[{"app":"opengist","leg":"whole app","reason":"` + reason + `"}]}`
|
||||
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
|
||||
|
||||
// The figures appear ONCE, on the Filesystem line — not again on every app row.
|
||||
if strings.Count(body, "65.0/68.7 GB used") != 1 {
|
||||
t.Fatalf("the usage clause appears %d times; it must appear once, on its own line — repeated "+
|
||||
"down a list of a dozen apps it pushes the part that DIFFERS off a phone screen:\n%s",
|
||||
strings.Count(body, "65.0/68.7 GB used"), body)
|
||||
}
|
||||
// But the reason itself survives — trimming must not eat the diagnosis.
|
||||
if !strings.Contains(body, "below the reserve") {
|
||||
t.Fatalf("the reason was trimmed away entirely:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// A reason naming a DIFFERENT path, or none, must be left completely alone.
|
||||
func TestTrimRepeatedUsage_LeavesUnrelatedReasonsAlone(t *testing.T) {
|
||||
for _, c := range []struct{ reason, target string }{
|
||||
{"pg_dump: connection refused", "/mnt/sys_drive"},
|
||||
{"tar failed — /mnt/other: 1/2 GB used (50%), 1 GB free", "/mnt/sys_drive"},
|
||||
{"boom", ""},
|
||||
{"", "/mnt/sys_drive"},
|
||||
} {
|
||||
if got := trimRepeatedUsage(c.reason, c.target); got != c.reason {
|
||||
t.Errorf("reason %q (target %q) was altered to %q", c.reason, c.target, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -116,6 +116,18 @@ func (d *Dispatcher) ProcessEvent(customerID, eventType, severity, message, deta
|
||||
return
|
||||
}
|
||||
|
||||
// R-182: record-only types are written down and never mailed. Placed BEFORE the severity gate
|
||||
// so the row is written whatever the severity — the record must not inherit the notification's
|
||||
// conditions, which is the coupling this whole finding is about.
|
||||
if recordOnlyEvents[eventType] {
|
||||
if err := d.store.LogNotification(customerID, eventType, severity, message, "recorded",
|
||||
"record-only: the per-run digest (backup_run_failures) carries the notification", "operator"); err != nil {
|
||||
d.logger.Printf("[WARN] Failed to record %s for %s: %v", eventType, customerID, err)
|
||||
}
|
||||
d.logger.Printf("[INFO] Recorded (not mailed) %s for %s — the run digest is the notification", eventType, customerID)
|
||||
return
|
||||
}
|
||||
|
||||
// warning / error / critical trigger notifications. "info" is an intentional non-notify (status/
|
||||
// recovery events). Anything else is UNRECOGNIZED — log it (don't silently drop), so a bad severity
|
||||
// surfaces instead of vanishing (the felhom-pve-class lesson: a critical event must never be lost).
|
||||
@@ -260,15 +272,67 @@ func cooldownTierSuffix(detailsJSON string) string {
|
||||
return ":" + d.Tier
|
||||
}
|
||||
|
||||
// cooldownRunSuffix returns ":"+run_id when the event's details carry a non-empty `run_id`, else "".
|
||||
//
|
||||
// R-182. `cooldownTierSuffix`'s sibling, and deliberately a SEPARATE function rather than an extra
|
||||
// branch inside it: `tier` keeps byte-identical semantics for every type that uses it, so R-97a's
|
||||
// behaviour and its tests are untouched by this.
|
||||
//
|
||||
// WHY A BACKUP RUN NEEDS ONE. The run digest describes ONE RUN, and a box can have two in a day —
|
||||
// the nightly one and a manual one the operator triggered *because* something looked wrong. With no
|
||||
// run-scoped discriminator the 1-hour cooldown would swallow the second, which is the failure this
|
||||
// row exists to fix, reappearing one level up: the operator presses the button, the run fails, and
|
||||
// they are told nothing because the machine already wrote that hour.
|
||||
//
|
||||
// IT MAKES THE COOLDOWN EFFECTIVELY INERT FOR THIS TYPE, AND THAT IS THE INTENT, NOT AN OVERSIGHT.
|
||||
// A digest is already rate-limited by construction — one per run, emitted only when something
|
||||
// failed — so there is nothing for a timer to collapse. The cooldown protects against a repeating
|
||||
// identical alert; a digest cannot repeat, because each run is a different run.
|
||||
//
|
||||
// NARROW, LIKE ITS SIBLING: empty unless the producer opts in by sending a `run_id`, so no existing
|
||||
// event type's cooldown behaviour changes.
|
||||
func cooldownRunSuffix(detailsJSON string) string {
|
||||
if detailsJSON == "" || !strings.Contains(detailsJSON, "\"run_id\"") {
|
||||
return ""
|
||||
}
|
||||
var d struct {
|
||||
RunID string `json:"run_id"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || d.RunID == "" {
|
||||
return ""
|
||||
}
|
||||
return ":" + d.RunID
|
||||
}
|
||||
|
||||
func (d *Dispatcher) processOperator(customerID, eventType, severity, message, detailsJSON, source string) {
|
||||
if !d.operatorOn || d.operatorEmail == "" {
|
||||
return
|
||||
}
|
||||
|
||||
cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON)
|
||||
cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON) + cooldownRunSuffix(detailsJSON)
|
||||
d.mu.Lock()
|
||||
if last, ok := d.opCooldowns[cooldownKey]; ok && time.Since(last) < 1*time.Hour {
|
||||
d.mu.Unlock()
|
||||
// R-182: RECORD THE SUPPRESSION. This used to be a bare `return` — the event was dropped
|
||||
// before any LogNotification, so a cooldown drop and an event that never happened were
|
||||
// indistinguishable from the operator's side AND from the hub's own records.
|
||||
//
|
||||
// Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two emails were
|
||||
// sent, and the other seven left NO ROW ON ANY CHANNEL. The defect that hid was serious —
|
||||
// the cooldown key carries no app identifier, so the first refused app took the slot and
|
||||
// every other app's failure that hour was discarded — but the reason it took a day to find
|
||||
// the right way round is this line: there was nothing to read.
|
||||
//
|
||||
// "We chose not to e-mail you" and "nothing happened" must never look identical. This
|
||||
// applies to EVERY operator event, not only the one that exposed it. It makes the drop
|
||||
// visible; it deliberately does NOT change the cooldown's duration or semantics.
|
||||
if err := d.store.LogNotification(customerID, eventType, severity, message,
|
||||
"suppressed", "operator cooldown 1h, key="+cooldownKey, "operator"); err != nil {
|
||||
d.logger.Printf("[WARN] Failed to record suppressed operator notification for %s/%s: %v",
|
||||
customerID, eventType, err)
|
||||
}
|
||||
d.logger.Printf("[INFO] Operator email suppressed for %s/%s — cooldown (key=%s)",
|
||||
customerID, eventType, cooldownKey)
|
||||
return
|
||||
}
|
||||
d.opCooldowns[cooldownKey] = time.Now()
|
||||
@@ -285,6 +349,33 @@ func (d *Dispatcher) processOperator(customerID, eventType, severity, message, d
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "operator")
|
||||
}
|
||||
|
||||
// recordOnlyEvents are STORED and RECORDED but never e-mailed, on either channel.
|
||||
//
|
||||
// R-182. The distinction this register exists to make is the whole of that finding: **the record and
|
||||
// the notification are different things.** A per-app backup failure must always be written down —
|
||||
// every time, unconditionally, regardless of cooldowns, preferences or whether any mail went out —
|
||||
// and it must NOT compete for an e-mail slot, because the per-run digest
|
||||
// (`backup_run_failures`) is what a person is meant to read.
|
||||
//
|
||||
// Before this, `recovery_unit_capture_failed` was both at once, and it did neither well: on
|
||||
// 2026-08-03 nine of them arrived, two were e-mailed, and the other seven were dropped by the
|
||||
// 1-hour cooldown BEFORE anything was written down. So the operator was told about one app, the
|
||||
// other apps' failures were discarded, and nothing anywhere recorded that a choice had been made.
|
||||
//
|
||||
// WHY A REGISTER AND NOT severity "info". Downgrading the severity would have the same routing
|
||||
// effect — `severityNotifies` drops info — but it would also relabel a genuine failure as
|
||||
// informational in the events table, the operator UI and every historical query, and it would
|
||||
// silently drop the X-Priority handling if the type were ever promoted back. This says what it
|
||||
// means: not silent, not urgent, RECORDED.
|
||||
//
|
||||
// IT IS NOT A WAY TO MUTE THINGS. A type belongs here only when something else carries its
|
||||
// notification. Adding one with no digest behind it rebuilds the silence R-182 was filed against.
|
||||
var recordOnlyEvents = map[string]bool{
|
||||
// The per-app Tier-1 capture/refusal failure. Its notification is the run digest, which lists
|
||||
// every failed app in one mail; this row is the durable per-failure record behind it.
|
||||
"recovery_unit_capture_failed": true,
|
||||
}
|
||||
|
||||
// operatorOnlyEvents are event types that must NEVER reach a customer, whatever their preferences say.
|
||||
//
|
||||
// R-97c. This register exists because the guarantee it provides was previously ASSERTED IN A COMMENT
|
||||
@@ -321,6 +412,15 @@ var operatorOnlyEvents = map[string]bool{
|
||||
// figures, the raw error). The customer's half of D-c is the FILL WARNING, which fires BEFORE
|
||||
// this and is actionable: free space, delete files, add a drive.
|
||||
"recovery_unit_capture_failed": true,
|
||||
// R-182. The per-run backup digest. It is the same class as the line above and for the same
|
||||
// reason — a customer can act on a full disk (that is the fill warning, which fires first and
|
||||
// IS customer-facing) but not on a list of which apps' backups failed and why. It also carries
|
||||
// operator-grade detail: per-app leg names, raw refusal reasons and byte figures.
|
||||
//
|
||||
// Listed here rather than relying on the absence of a `customerMessages` entry, which is NOT a
|
||||
// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
|
||||
// once (v0.78.0) and the comment above records it.
|
||||
"backup_run_failures": true,
|
||||
}
|
||||
|
||||
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
|
||||
|
||||
@@ -39,33 +39,45 @@ func TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The operator must still get it: the register mutes the customer channel, not the signal.
|
||||
gotOperator := false
|
||||
// R-182 CHANGED WHAT THIS ASSERTS, DELIBERATELY, AND THE OLD ASSERTION IS WORTH KEEPING IN VIEW.
|
||||
//
|
||||
// Until 2026-08-03 this test required the OPERATOR to be e-mailed here, on the grounds that "the
|
||||
// alert is the whole point of R-158". That was right when this event was the only signal, and it
|
||||
// is wrong now: measured, nine of these arrived at the hub and two were mailed, because the
|
||||
// operator cooldown key carries no app identifier — so as an alarm it told the operator about one
|
||||
// app and threw the rest away.
|
||||
//
|
||||
// The type is now RECORD-ONLY: written down every time, never mailed. R-158's guarantee — the
|
||||
// operator learns WHICH app failed and WHY — is not weakened, it MOVED: the per-run digest
|
||||
// `backup_run_failures` carries every failed app in one mail, and is pinned by
|
||||
// backup_run_digest_test.go. The customer safety claim below is untouched and is the reason this
|
||||
// test still exists.
|
||||
for _, to := range rec.to {
|
||||
if to == "operator@felhom.eu" {
|
||||
gotOperator = true
|
||||
t.Fatal("the operator was e-mailed a PER-APP capture failure — this type is the record " +
|
||||
"now, not the alarm. One mail per failing app on a full disk is a dozen mails, which " +
|
||||
"is the volume problem the operator ruled against; the digest is the notification")
|
||||
}
|
||||
}
|
||||
if !gotOperator {
|
||||
t.Fatal("the operator was not notified of a recovery-unit capture failure — the alert is the " +
|
||||
"whole point of R-158 and it went nowhere")
|
||||
}
|
||||
|
||||
// The skip must be VISIBLE. An absent log row is equally consistent with "correctly skipped" and
|
||||
// "the dispatcher never ran" — the positive observable is the row itself (standing rule 3).
|
||||
// The RECORD must exist, always. It is what makes the digest trustworthy: if the digest is ever
|
||||
// lost, delayed or suppressed, the failures are still individually written down. An absent row is
|
||||
// equally consistent with "correctly not mailed" and "the dispatcher never ran" — the positive
|
||||
// observable is the row itself (standing rule 3).
|
||||
logs, err := st.GetRecentNotifications("c1", 20)
|
||||
if err != nil {
|
||||
t.Fatalf("GetRecentNotifications: %v", err)
|
||||
}
|
||||
found := false
|
||||
for _, l := range logs {
|
||||
if l.Channel == "customer" && l.Status == "skipped" && strings.Contains(l.ErrorMessage, "operator_only") {
|
||||
if l.EventType == "recovery_unit_capture_failed" && l.Status == "recorded" &&
|
||||
strings.Contains(l.ErrorMessage, "record-only") {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("the customer skip is not logged as skipped/operator_only — it is indistinguishable "+
|
||||
"from a delivery that never happened; got %d row(s)", len(logs))
|
||||
t.Fatalf("the per-app failure left no 'recorded' row — a failure that is neither mailed nor "+
|
||||
"written down is exactly the 2026-08-03 defect, rebuilt; got %d row(s)", len(logs))
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -41,6 +42,17 @@ Severity: %s
|
||||
Time: %s
|
||||
Message: %s`, customerID, eventType, severity, now, message)
|
||||
|
||||
// R-182: the backup run digest gets a rendered list instead of a raw JSON blob. It is the one
|
||||
// operator mail that carries a VARIABLE-LENGTH payload, and a dozen apps as one line of JSON is
|
||||
// unreadable on a phone at 07:00, which is the only time it matters.
|
||||
if eventType == "backup_run_failures" {
|
||||
if rendered, sub, ok := renderBackupRunFailures(customerID, detailsJSON); ok {
|
||||
return sub, body + rendered + fmt.Sprintf("\n\nDashboard: https://hub.felhom.eu/customers/%s", customerID)
|
||||
}
|
||||
// Unparseable details fall through to the raw form below rather than losing the mail. A
|
||||
// digest that renders badly still tells the operator something; a swallowed one does not.
|
||||
}
|
||||
|
||||
if detailsJSON != "" && detailsJSON != "{}" {
|
||||
body += fmt.Sprintf("\nDetails: %s", detailsJSON)
|
||||
}
|
||||
@@ -282,3 +294,115 @@ Ha nem te kérted ezt, hagyd figyelmen kívül ezt az e-mailt.
|
||||
Felhom.eu`, link)
|
||||
return subject, body
|
||||
}
|
||||
|
||||
// ──────────────────────────────────────────────────────────────────────
|
||||
// R-182 — the backup run digest
|
||||
// ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
// backupRunFailure is one app's failed leg within a run.
|
||||
type backupRunFailure struct {
|
||||
App string `json:"app"`
|
||||
Leg string `json:"leg"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
// backupRunDetails is the digest payload the controller sends.
|
||||
type backupRunDetails struct {
|
||||
RunID string `json:"run_id"`
|
||||
RunKind string `json:"run_kind"`
|
||||
Failed int `json:"failed"`
|
||||
Attempted int `json:"attempted"`
|
||||
TargetPath string `json:"target_path"`
|
||||
UsedGB float64 `json:"used_gb"`
|
||||
AvailGB float64 `json:"avail_gb"`
|
||||
TotalGB float64 `json:"total_gb"`
|
||||
UsedPercent float64 `json:"used_percent"`
|
||||
SpaceKnown bool `json:"space_known"`
|
||||
Apps []backupRunFailure `json:"apps"`
|
||||
}
|
||||
|
||||
// renderBackupRunFailures turns the digest details into an operator-readable block and a subject
|
||||
// that says the count without being opened. Returns ok=false when the payload cannot be parsed or
|
||||
// names no apps, so the caller can fall back to the raw rendering rather than mail an empty list.
|
||||
//
|
||||
// THE SUCCESS COUNT IS NOT DECORATION. "3 of 4 apps failed" is a catastrophe and "3 of 40" is a bad
|
||||
// night; the list alone cannot tell them apart, and the operator's first decision — get up now, or
|
||||
// look after coffee — depends entirely on which it is.
|
||||
func renderBackupRunFailures(customerID, detailsJSON string) (string, string, bool) {
|
||||
if detailsJSON == "" {
|
||||
return "", "", false
|
||||
}
|
||||
var d backupRunDetails
|
||||
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || len(d.Apps) == 0 {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
kind := d.RunKind
|
||||
if kind == "" {
|
||||
kind = "backup"
|
||||
}
|
||||
subject := fmt.Sprintf("[Felhom] 🔴 %s: %d of %d apps failed to back up (%s run)",
|
||||
customerID, d.Failed, d.Attempted, kind)
|
||||
|
||||
// Column-align the app names so the leg and reason line up and the block scans vertically.
|
||||
width := 0
|
||||
for _, a := range d.Apps {
|
||||
if len(a.App) > width {
|
||||
width = len(a.App)
|
||||
}
|
||||
}
|
||||
legWidth := 0
|
||||
for _, a := range d.Apps {
|
||||
if len(a.Leg) > legWidth {
|
||||
legWidth = len(a.Leg)
|
||||
}
|
||||
}
|
||||
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "\n\nFAILED: %d of %d apps attempted in this %s run.\n\n", d.Failed, d.Attempted, kind)
|
||||
for _, a := range d.Apps {
|
||||
reason := trimRepeatedUsage(a.Reason, d.TargetPath)
|
||||
if reason == "" {
|
||||
reason = "(no reason recorded)"
|
||||
}
|
||||
fmt.Fprintf(&b, " %-*s %-*s %s\n", width, a.App, legWidth, a.Leg, reason)
|
||||
}
|
||||
|
||||
// The space figures answer "is this one broken app or a full disk" before the reasons are read.
|
||||
// An absent reading renders as unavailable, never as zeros — "0 GB free" and "we could not look"
|
||||
// are opposite diagnoses (the UnitSpace rule, same reasoning, other side of the wire).
|
||||
if d.SpaceKnown {
|
||||
fmt.Fprintf(&b, "\nFilesystem: %s — %.1f/%.1f GB used (%.0f%%), %.1f GB free\n",
|
||||
d.TargetPath, d.UsedGB, d.TotalGB, d.UsedPercent, d.AvailGB)
|
||||
} else {
|
||||
fmt.Fprintf(&b, "\nFilesystem: %s — usage unavailable (the filesystem could not be read)\n", d.TargetPath)
|
||||
}
|
||||
|
||||
b.WriteString("\nEvery failure above is also recorded individually in the notification log,\n")
|
||||
b.WriteString("whether or not this mail was sent.")
|
||||
return b.String(), subject, true
|
||||
}
|
||||
|
||||
// trimRepeatedUsage strips the trailing "— /path: X/Y GB used (Z%), W GB free" clause from a per-app
|
||||
// reason, because the digest prints those figures ONCE for the whole run on its own line.
|
||||
//
|
||||
// This is a copy fix, and it was made after reading the first real digest rather than from the
|
||||
// design. The reserve's refusal message is authored for a single-app alert, where naming the
|
||||
// filesystem is exactly right; repeated down a list of a dozen apps it is the same forty characters
|
||||
// twelve times, and it pushes the part that differs off the right-hand edge of a phone screen at
|
||||
// 07:00 — which is the only moment this mail has to work.
|
||||
//
|
||||
// It trims ONLY an exact "— <target path>:" suffix, so a reason that mentions a different path, or
|
||||
// none, is left completely alone. A reason that is nothing but the usage clause is left alone too:
|
||||
// removing everything would turn a bad line into an empty one.
|
||||
func trimRepeatedUsage(reason, targetPath string) string {
|
||||
if reason == "" || targetPath == "" {
|
||||
return reason
|
||||
}
|
||||
marker := " — " + targetPath + ":"
|
||||
i := strings.LastIndex(reason, marker)
|
||||
if i <= 0 {
|
||||
return reason
|
||||
}
|
||||
return strings.TrimSpace(reason[:i])
|
||||
}
|
||||
|
||||
@@ -815,7 +815,7 @@ type NotificationLogEntry struct {
|
||||
EventType string
|
||||
Severity string
|
||||
Message string
|
||||
Status string // "sent", "skipped", "failed"
|
||||
Status string // "sent", "skipped", "failed", "suppressed" (R-182: a cooldown drop, recorded rather than silent)
|
||||
ErrorMessage string
|
||||
Channel string // "operator" or "customer"
|
||||
CreatedAt time.Time
|
||||
|
||||
+1
-1
@@ -125,7 +125,7 @@ spec:
|
||||
spec:
|
||||
containers:
|
||||
- name: hub
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.89.0
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.91.1
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
|
||||
@@ -324,7 +324,7 @@ spec:
|
||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||
args:
|
||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||
- --ref=installer-v1.23.0
|
||||
- --ref=installer-v1.24.0
|
||||
- --root=/git-scripts
|
||||
- --link=current
|
||||
- --period=30s
|
||||
@@ -369,7 +369,7 @@ spec:
|
||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||
args:
|
||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||
- --ref=installer-v1.23.0
|
||||
- --ref=installer-v1.24.0
|
||||
- --root=/git-scripts
|
||||
- --link=current
|
||||
- --one-time
|
||||
|
||||
@@ -1,3 +1,55 @@
|
||||
## 1.25.0 — the off-site tier stops asking to prune (2026-08-04, R-191)
|
||||
|
||||
**A backup that worked must not report failure.** The off-site tier was written with `keep_last: 2`,
|
||||
so every weekly run uploaded its snapshot successfully and then failed the whole job on a prune the
|
||||
box's token is deliberately refused: `prune 'ct/9201': permission check failed - missing
|
||||
Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the
|
||||
operator's inbox. Every week, on both boxes, about a backup that had already succeeded.
|
||||
|
||||
**R-89 moved off-site pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay
|
||||
write-only, so a box can never delete its own off-site history. The 2026-07-26 "two weeks" ruling was
|
||||
not reversed; where it is ENFORCED moved, and this value did not follow. The tier now writes
|
||||
`keep_last: 0`, which the agent's existing guard (`allowPBSPrune = !primary && keep_last > 0`) already
|
||||
reads as "never prune from the box" — no agent change needed.
|
||||
|
||||
**VERIFIED BEFORE CHANGING IT** (read-only on ep0, 2026-08-04): prune jobs `prune-demo-felhom` and
|
||||
`prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
|
||||
and have run **every day since 2026-07-27 — 18 tasks, all `status=OK`**, the newest showing
|
||||
`retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Disabling
|
||||
the client-side prune without that check would have traded a weekly false alarm for unbounded growth.
|
||||
|
||||
A gate now asserts the off-site tier carries no client-side prune, so the value cannot drift back
|
||||
quietly. The local tier's retention is untouched.
|
||||
|
||||
## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
|
||||
|
||||
**`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
|
||||
storage and calls `felhom-backup-target-apply grant` in the same breath — a box that builds its own
|
||||
target has always been fine. The **Scenario-F arm** — *"the target already exists, leave it exactly as
|
||||
it is"* — returned without ever granting.
|
||||
|
||||
So a box whose `felhom-backup` pre-dated the install (created by the vzdump-target-move runbook, or
|
||||
surviving a reinstall) ended up with `local_backup_target: felhom-backup` while its token held
|
||||
`FelhomAgentStore` on only `local`, `local-lvm` and `felhom-pbs`. Measured on **both** demo boxes
|
||||
2026-08-03: the content API answers `{"data":[]}` through the agent's token while root lists three
|
||||
archives. That tier was invisible to the agent and never restore-tested — and nothing said so,
|
||||
because an empty listing is also what a brand-new tier returns.
|
||||
|
||||
The reuse arm now ensures the ACL through the same guarded wrapper, so both arms leave the box in the
|
||||
same state. **Scenario F is unviolated:** the storage DEFINITION is still untouched — granting the
|
||||
role the agent is supposed to have on the target this script is about to write into `agent.json` is
|
||||
finishing the job, not retargeting the box. `pveum acl modify` is idempotent, so a box that already
|
||||
has the grant is unchanged and a box whose token was rotated gets it back.
|
||||
|
||||
`$BACKUP_TARGET_ID` is deliberately **not** added to `PVE_STORAGES`, and the comment now says why: that
|
||||
list is granted in step 4/5, before the target has been resolved in step 6, and `--acl-storages`
|
||||
entries are preflight-checked for existence. The grant belongs with the resolution, which is where it
|
||||
already was for a newly created target.
|
||||
|
||||
**A gate now asserts it** (`hostinstall_gates.py`): every arm of `configure_backup_target` that
|
||||
resolves the target must also grant on it. Red-proved by reverting the reuse arm — `resolves the
|
||||
backup target in 2 place(s) but grants in only 1`.
|
||||
|
||||
## v1.23.0 — the installer is published, not pushed (2026-08-03, R-110 + R-183)
|
||||
|
||||
**Two channels moved off `main` in the same change, because either one left behind makes the other
|
||||
|
||||
@@ -184,7 +184,7 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_VERSION="1.23.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
||||
SCRIPT_VERSION="1.25.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
||||
# The hub used to carry a copy for its Setup tab; R-94 DELETED it
|
||||
# (2026-08-02) because the hub cannot know which version a box runs —
|
||||
# the Setup command fetches this script at run time. scripts/
|
||||
@@ -307,6 +307,14 @@ PVE_PRIVS_STORE="Datastore.Allocate Datastore.AllocateSpace"
|
||||
PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit"
|
||||
# Storages the agent reads/writes (archive+dump=local, restore=local-lvm, offsite DR=felhom-pbs). The
|
||||
# offsite felhom-pbs MUST be included or the agent's DR backup 403s (SPIKE residual #1). --acl-storages overrides.
|
||||
#
|
||||
# `$BACKUP_TARGET_ID` is deliberately NOT in this list, and the reason matters because the obvious
|
||||
# tidy-up is to add it (R-185, 2026-08-03). This list is granted in STEP 4/5, before
|
||||
# configure_backup_target has run in STEP 6 — so at this point the target may not exist yet, and
|
||||
# --acl-storages entries are preflight-checked for existence. The target's grant therefore belongs
|
||||
# with the target's RESOLUTION, where it already is for a freshly created one, and now also for a
|
||||
# pre-existing one. Adding it here would grant on a storage that may not exist and would still leave
|
||||
# the resolution path as the single owner of that decision, split across two places.
|
||||
PVE_STORAGES=(local local-lvm felhom-pbs)
|
||||
# E-2: the whole-guest backup target storage id, and what configure_backup_target resolved to.
|
||||
# BACKUP_TARGET_RESOLVED feeds agent.json's local_backup_target; "local" means DEGRADED (Case B).
|
||||
@@ -652,8 +660,31 @@ configure_backup_target() {
|
||||
# existing id is an error, and repointing a live target is exactly the silent retarget this
|
||||
# whole arc closes.
|
||||
if pvesm status --storage "$BACKUP_TARGET_ID" >/dev/null 2>&1; then
|
||||
log_skip " backup target '$BACKUP_TARGET_ID' already exists — leaving it exactly as it is (Scenario F)"
|
||||
log_skip " backup target '$BACKUP_TARGET_ID' already exists — leaving its DEFINITION exactly as it is (Scenario F)"
|
||||
BACKUP_TARGET_RESOLVED="$BACKUP_TARGET_ID"
|
||||
# R-185: …but STILL ensure the ACL. "The storage already exists" says nothing about whether
|
||||
# the agent may READ it, and this early return is where the two came apart.
|
||||
#
|
||||
# THE DEFECT THIS CLOSES, measured on both demo boxes 2026-08-03. The CASE A path below
|
||||
# creates the storage and grants in the same breath, so a box that built its own target is
|
||||
# fine. A box whose target ALREADY existed — created by the vzdump-target-move runbook, or
|
||||
# surviving a reinstall — returned here and never granted. The result: `local_backup_target`
|
||||
# pointed at `felhom-backup` while the token held FelhomAgentStore only on local, local-lvm
|
||||
# and felhom-pbs, so the API answered `{"data":[]}` for that storage while root saw three
|
||||
# archives. The tier was invisible to the agent and never restore-tested, and nothing said so
|
||||
# — because an empty listing is also what a brand-new tier returns.
|
||||
#
|
||||
# Scenario F is UNVIOLATED: the storage definition is still untouched. Granting a role the
|
||||
# agent is supposed to have on the target this same script is about to write into
|
||||
# agent.json is not "touching the box's target", it is finishing the job. `pveum acl modify`
|
||||
# is idempotent, so a box that already has the grant is unchanged, and a box that had its
|
||||
# token rotated gets it back.
|
||||
if $DRY_RUN; then
|
||||
log_dry "felhom-backup-target-apply grant $BACKUP_TARGET_ID # R-185: ACL on a pre-existing target"
|
||||
else
|
||||
/usr/local/sbin/felhom-backup-target-apply grant "$BACKUP_TARGET_ID" \
|
||||
|| die "backup target grant failed on the pre-existing target — the agent could not read its own backup tier (R-185)"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
local mp
|
||||
@@ -2448,8 +2479,30 @@ for _k,_v in {"unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agen
|
||||
base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20})
|
||||
# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions —
|
||||
# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob
|
||||
# would guarantee that one of them is wrong. keep_last=2 on the offsite tier = two weeks (operator
|
||||
# ruling 2026-07-26).
|
||||
# would guarantee that one of them is wrong.
|
||||
#
|
||||
# THE OFFSITE TIER CARRIES NO CLIENT-SIDE RETENTION, AND THAT IS THE POINT (R-191, 2026-08-04).
|
||||
# It used to be written `keep_last: 2` ("two weeks", operator ruling 2026-07-26). **R-89 then moved
|
||||
# offsite pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay write-only,
|
||||
# deliberately, so that a box can never delete its own offsite history. The 2026-07-26 ruling was not
|
||||
# reversed; where it is ENFORCED moved, and this value did not follow.
|
||||
#
|
||||
# The consequence was weekly and wrong in the worst direction: vzdump UPLOADED the snapshot fine and
|
||||
# then failed the whole job on the prune the token is refused —
|
||||
# `prune 'ct/9201': permission check failed - missing Datastore.Modify|Datastore.Prune`
|
||||
# → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the operator's inbox. Every week, on
|
||||
# both boxes, about a backup that had already succeeded. A tier that cries wolf weekly is a tier
|
||||
# whose real failure nobody will see.
|
||||
#
|
||||
# VERIFIED BEFORE CHANGING IT (2026-08-04, read-only on ep0): prune jobs `prune-demo-felhom` and
|
||||
# `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
|
||||
# and have run EVERY DAY since 2026-07-27 — 18 tasks, all `status=OK`, the newest showing
|
||||
# `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Retention
|
||||
# happens; it happens THERE. **If that ever stops being true, this zero is unbounded growth** — check
|
||||
# ep0's prune jobs before assuming the offsite tier is retained.
|
||||
#
|
||||
# `keep_last: 0` means "never prune from the box" and is the value the agent's own guard reads
|
||||
# (allowPBSPrune = !primary && keep_last > 0), so no agent change is needed to honour it.
|
||||
#
|
||||
# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only
|
||||
# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent
|
||||
@@ -2460,7 +2513,7 @@ base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_sec
|
||||
# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here —
|
||||
# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start
|
||||
# writing to an offsite datastore.
|
||||
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]})
|
||||
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":0}]})
|
||||
base.setdefault('local_api', {})
|
||||
base['local_api'].setdefault('enable', True)
|
||||
base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']
|
||||
|
||||
@@ -202,6 +202,63 @@ if wp is not None:
|
||||
fail("manifests/webpage.yaml no longer tracks main for the website — pinning the SITE to "
|
||||
"the installer tag turns every copy edit into a release")
|
||||
|
||||
# ── R-185: every path that RESOLVES the backup target must also grant on it ──────────────────
|
||||
#
|
||||
# THE DEFECT THIS WOULD HAVE CAUGHT, measured on both demo boxes 2026-08-03. `configure_backup_target`
|
||||
# has two arms. The CASE A arm creates the storage and grants in the same breath. The Scenario-F arm —
|
||||
# "the target already exists, leave it alone" — returned WITHOUT granting, so a box whose target
|
||||
# pre-dated the install pointed `local_backup_target` at a storage its own token could not read. The
|
||||
# API answered `{"data":[]}` while root saw three archives, and nothing said so, because an empty
|
||||
# listing is also what a brand-new tier returns.
|
||||
#
|
||||
# The assertion is deliberately about the FUNCTION, not about PVE_STORAGES: the target's grant belongs
|
||||
# with the target's resolution (PVE_STORAGES is granted a step earlier, before the target exists), so
|
||||
# what must hold is that no arm of that function can resolve a target and skip the grant.
|
||||
fn = re.search(r'^configure_backup_target\(\)\s*\{(.*?)^\}', src, re.S | re.M)
|
||||
if not fn:
|
||||
fail("cannot find configure_backup_target() — the backup-target ACL assertion cannot run, and a "
|
||||
"check that cannot run must never report OK (R-185)")
|
||||
else:
|
||||
body = fn.group(1)
|
||||
resolutions = len(re.findall(r'BACKUP_TARGET_RESOLVED="\$BACKUP_TARGET_ID"', body))
|
||||
grants = len(re.findall(r'felhom-backup-target-apply grant', body))
|
||||
if resolutions == 0:
|
||||
fail("configure_backup_target no longer resolves BACKUP_TARGET_ID anywhere — re-read it")
|
||||
elif grants >= resolutions:
|
||||
ok("every arm that resolves the backup target also grants on it (%d resolution(s), %d grant(s))"
|
||||
% (resolutions, grants))
|
||||
else:
|
||||
fail("configure_backup_target resolves the backup target in %d place(s) but grants in only %d "
|
||||
"— an arm resolves a target the agent may not READ. That is R-185: the tier's archives are "
|
||||
"invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
|
||||
"like a brand-new tier." % (resolutions, grants))
|
||||
|
||||
# ── R-191: the OFFSITE tier must not arm a client-side prune ─────────────────────────────────
|
||||
#
|
||||
# R-89 moved offsite pruning SERVER-SIDE — ep0 runs a per-namespace prune job and box tokens stay
|
||||
# write-only, so the box is REFUSED if it asks. When this default was `keep_last: 2` the effect was a
|
||||
# weekly lie: vzdump uploaded the snapshot, then failed the whole job on the prune, and the operator
|
||||
# was told the offsite backup had failed when it had succeeded.
|
||||
#
|
||||
# The assertion is on the OFFSITE entry only. The local tier's `local_backup_retention` is untouched
|
||||
# and must stay untouched — it prunes correctly and is allowed to.
|
||||
m = re.search(r'"backup_targets":\s*\[(.*?)\]', src, re.S)
|
||||
if not m:
|
||||
fail("cannot find backup_targets in the rendered agent.json defaults — the offsite-retention "
|
||||
"assertion cannot run, and a check that cannot run must never report OK (R-191)")
|
||||
else:
|
||||
targets = m.group(1)
|
||||
kl = re.search(r'"keep_last"\s*:\s*(\d+)', targets)
|
||||
if not kl:
|
||||
fail("the offsite backup_target carries no keep_last at all — expected an explicit 0 "
|
||||
"(R-191: 0 means 'never prune from the box'; absent is not the same statement)")
|
||||
elif kl.group(1) != "0":
|
||||
fail("the offsite backup_target arms a CLIENT-SIDE prune (keep_last=%s). R-89 moved offsite "
|
||||
"pruning server-side to ep0 and box tokens are write-only, so every weekly run will "
|
||||
"upload successfully and then FAIL the job on a refused prune (R-191)." % kl.group(1))
|
||||
else:
|
||||
ok("the offsite tier arms no client-side prune (keep_last=0; retention is ep0's prune jobs)")
|
||||
|
||||
print()
|
||||
if fails:
|
||||
print("hostinstall gates: %d FAILURE(S)" % len(fails))
|
||||
|
||||
Reference in New Issue
Block a user