Compare commits
64 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 51871a7ea6 | |||
| 33f0ab2f33 | |||
| 2584dfb938 | |||
| 185661062a | |||
| 967ddf2f44 | |||
| d692c46db0 | |||
| b93ee06abc | |||
| e3187c86d5 | |||
| 311dc06c13 | |||
| 688470c945 | |||
| 7a5694341d | |||
| 81d4438630 | |||
| c9a3e48b21 | |||
| 0779c5db85 | |||
| 0f8f084817 | |||
| c102832892 | |||
| ff2655cf19 | |||
| 046df303b6 | |||
| 687fedd8ee | |||
| 323f45a5ef | |||
| e34b614e5b | |||
| f21e7caed1 | |||
| dd40f85bb8 | |||
| 7dc1744eec | |||
| a5cd480280 | |||
| b0b269b28d | |||
| e79a20bbed | |||
| 6a82719426 | |||
| bee6848458 | |||
| 8360f940bf | |||
| fb652024ea | |||
| 06cbf8df29 | |||
| 6b5d64c1fa | |||
| aa62449694 | |||
| bdd1a9d130 | |||
| 14d8c00781 | |||
| e3525e62ac | |||
| 7406ac7bbf | |||
| 1806dfa8e9 | |||
| 41dbecb264 | |||
| 6d359a5360 | |||
| 179dd79882 | |||
| 8ef92a3fa7 | |||
| d5774d3189 | |||
| 0fc54e0122 | |||
| 2c35c4204a | |||
| ad28699761 | |||
| 5c97fbc397 | |||
| c04ea4f6c2 | |||
| ca3c8f6784 | |||
| c718aad1bc | |||
| 4cc123809c | |||
| 9530de722d | |||
| f7dbc335ac | |||
| dd13f632c8 | |||
| 3252d51104 | |||
| 666a34da88 | |||
| bbd6231909 | |||
| af2d103880 | |||
| 8d9b78c153 | |||
| 4707be755c | |||
| 9bd1a54d71 | |||
| 2137094799 | |||
| d319ae573e |
@@ -0,0 +1,97 @@
|
||||
# gates — re-run this repo's gate entry point on every push, on a machine that does not care who
|
||||
# pushed or what they typed.
|
||||
#
|
||||
# *** THIS REPORTS. IT CANNOT REFUSE. ***
|
||||
#
|
||||
# felhom repos push straight to `main` with no pull request, so there is no merge for a status
|
||||
# check to stand at. The refusing half is `.githooks/pre-push`, which is local to a clone and which
|
||||
# `git push --no-verify` skips; this half is what notices when that happened. Neither half is the
|
||||
# whole thing, and both are named in documentation/backlog/OPEN-ITEMS.md R-168.
|
||||
#
|
||||
# NO `uses:` STEP ANYWHERE, deliberately: JavaScript actions need a node runtime in the runner, and
|
||||
# the runner is a host-mode container with python3 and git and nothing else (see
|
||||
# homelab-manifests/gitea-system/act-runner.yaml for why it is not privileged). Probe P3 measured
|
||||
# that a plain `git fetch` of the pushed SHA from the in-cluster Gitea service is enough.
|
||||
#
|
||||
# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
|
||||
# layer up. That is the last step, and it runs ONLY on failure.
|
||||
name: gates
|
||||
on: [push]
|
||||
|
||||
jobs:
|
||||
gates:
|
||||
runs-on: felhom-gates
|
||||
steps:
|
||||
- name: Fetch the pushed commit
|
||||
run: |
|
||||
# Shallow, and pinned to the exact SHA that was pushed — not to the branch tip, which can
|
||||
# move under us if two pushes race. Probe P3 proved the two are equal when done this way.
|
||||
git init -q .
|
||||
git remote add origin http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git
|
||||
git fetch -q --depth 1 origin "$GITHUB_SHA"
|
||||
git checkout -q FETCH_HEAD
|
||||
echo "checked out $(git rev-parse HEAD)"
|
||||
|
||||
- name: Run the gate entry point
|
||||
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy — those are either
|
||||
# already reliably run by a person or none of CI's business. The exit code IS the result:
|
||||
# no `|| true`, no pipe that could swallow it.
|
||||
run: python3 scripts/repo_gates.py --fast
|
||||
|
||||
- name: Alarm on failure
|
||||
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
|
||||
# notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
|
||||
# is exactly the shape R-29 filed against. So the run sends its own alarm, on the project's
|
||||
# existing transactional path (Resend, the same one the hub uses), and prints the provider's
|
||||
# accepted id so "a message left the machine" is an observable, not an assumption.
|
||||
#
|
||||
# Pure python3 and urllib, NOT curl: the runner image carries python3 and git and nothing
|
||||
# else on purpose, and the first version of this step died on `curl: command not found`.
|
||||
# Reaching for a bigger image to send one HTTP request would have been the wrong trade.
|
||||
if: failure()
|
||||
env:
|
||||
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import json, os, sys, urllib.request, urllib.error
|
||||
|
||||
key = os.environ.get("RESEND_API_KEY", "")
|
||||
if not key:
|
||||
sys.exit("ALARM FAILED: RESEND_API_KEY is empty — the alarm cannot be sent, and a "
|
||||
"silent alarm is worse than none. Set the user-level Actions secret.")
|
||||
|
||||
repo = os.environ.get("GITHUB_REPOSITORY", "?")
|
||||
sha = os.environ.get("GITHUB_SHA", "?")
|
||||
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
|
||||
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
|
||||
|
||||
body = json.dumps({
|
||||
"from": "Felhom CI <monitoring@felhom.eu>",
|
||||
"to": ["admin@felhom.eu"],
|
||||
"subject": "[felhom CI] gates FAILED in %s" % repo,
|
||||
"text": (
|
||||
"The gate entry point exited non-zero.\n\n"
|
||||
"Repository : %s\n"
|
||||
"Commit : %s\n"
|
||||
"Run : %s/%s/actions/runs/%s\n\n"
|
||||
"The failing gate names itself in the run log.\n\n"
|
||||
"If the local pre-push hook was GREEN for this commit, then CI and the hook\n"
|
||||
"disagree - that is a finding about the gates themselves, not about CI, and it\n"
|
||||
"outranks whatever the push was for.\n"
|
||||
) % (repo, sha, srv, repo, run),
|
||||
}).encode()
|
||||
|
||||
req = urllib.request.Request(
|
||||
"https://api.resend.com/emails", data=body, method="POST",
|
||||
headers={"Authorization": "Bearer %s" % key,
|
||||
"Content-Type": "application/json",
|
||||
# Cloudflare fronts api.resend.com and BLOCKS the default
|
||||
# "Python-urllib/3.x" agent with its own 403 (error 1010) — which looks
|
||||
# exactly like an auth failure and is not one. Measured 2026-08-02.
|
||||
"User-Agent": "felhom-ci/1.0"})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
print("RESEND-ACCEPTED id=%s" % json.load(r)["id"])
|
||||
except urllib.error.HTTPError as e:
|
||||
sys.exit("ALARM FAILED: Resend returned HTTP %s: %s" % (e.code, e.read().decode()[:300]))
|
||||
PY
|
||||
Executable
+47
@@ -0,0 +1,47 @@
|
||||
#!/bin/sh
|
||||
# pre-push — refuse a push that carries a broken gate. (2026-08-02, R-29 leg (b) first half.)
|
||||
#
|
||||
# Runs this repo's ONE gate entry point in --fast mode: only checks that touch no network and no
|
||||
# container runtime, so a push stays a push and never pulls images or starts containers. The slow
|
||||
# gates stay deliberate periodic runs; a hook that takes minutes gets bypassed within a week and
|
||||
# the bypass becomes the habit.
|
||||
#
|
||||
# BOTH LINES BELOW ARE DELIBERATE. An absent log line is not evidence a hook ran — a silent pass is
|
||||
# equally consistent with "gates green" and "hook never fired", so a passing push says so out loud.
|
||||
#
|
||||
# HONEST LIMITS, stated so this is not mistaken for enforcement it cannot provide:
|
||||
# * per-clone — core.hooksPath is local config and a clone does not carry it. Arm a clone once:
|
||||
# git config core.hooksPath .githooks
|
||||
# Any manual entry-point run WARNS when the clone is unarmed.
|
||||
# * skippable — `git push --no-verify` bypasses this entirely. That is on purpose: an escape
|
||||
# hatch that cannot be reached is one that gets removed the first time it is
|
||||
# inconvenient. USING IT MUST BE STATED IN THE SESSION REPORT.
|
||||
# The half that is neither per-clone nor skippable is CI — felhom.eu OPEN-ITEMS.md R-168.
|
||||
#
|
||||
# Measured 2026-08-02 (git 2.47.3): a relative core.hooksPath resolves correctly and the hook's cwd
|
||||
# is the repo root whether `git push` is issued from the root or from any subdirectory. The
|
||||
# explicit rev-parse below does not depend on that.
|
||||
set -u
|
||||
|
||||
root=$(git rev-parse --show-toplevel 2>/dev/null) || {
|
||||
echo "pre-push: FAIL - cannot resolve the repo root (git rev-parse --show-toplevel)." >&2
|
||||
exit 1
|
||||
}
|
||||
cd "$root" || exit 1
|
||||
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
|
||||
echo " pass by default. Install python3, or push with --no-verify and say so." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "pre-push [felhom.eu]: running scripts/repo_gates.py --fast ..."
|
||||
python3 "scripts/repo_gates.py" --fast
|
||||
rc=$?
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo "pre-push [felhom.eu]: PUSH REFUSED - gates exited $rc. Fix the finding above, or bypass with" >&2
|
||||
echo " 'git push --no-verify' and state that you did in the session report." >&2
|
||||
else
|
||||
echo "pre-push [felhom.eu]: gates OK - push proceeding."
|
||||
fi
|
||||
exit $rc
|
||||
@@ -32,3 +32,7 @@ Thumbs.db
|
||||
# Temporary files
|
||||
*.tmp
|
||||
*.bak
|
||||
|
||||
# Python bytecode from the gate scripts + their fixture tests
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
@@ -56,6 +56,13 @@ UI. Package map, helpers, seams, extension points: **`REUSE.md`** (e.g. new even
|
||||
(`{{if .Flag}}` around a button/form/script) ships with a render test per branch of the gate —
|
||||
handler tests that POST directly prove nothing about reachability. The v0.70.0 ghost-delete was
|
||||
fully implemented server-side and fully dead UI because the button sat inside the wrong gate.
|
||||
- **A `go test -run` pattern that matches no test prints `ok` and exits 0.** Found 2026-08-02 while
|
||||
red-proofing: `-run TestCustomerUnified` matched nothing in the target file and reported
|
||||
`ok … 0.062s`, which was read as a passing red-proof. **A red-proof that uses `-run` must first
|
||||
prove the filter matched something** (`-v` and look for `=== RUN`). This is the "an absent line is
|
||||
not evidence" rule aimed at the one place a false green costs most — the proof itself. The same
|
||||
class bit twice that day: a `| tail -5` inside a census query silently dropped rows and looked
|
||||
exactly like a real finding. **An instrument that can drop results silently is not a measurement.**
|
||||
- **A health check issues no block I/O.** A probe that touches a wedged device enters uninterruptible
|
||||
sleep, survives `SIGKILL`, and cannot be recovered until the device returns or the host reboots — so
|
||||
`systemctl restart` hangs too. A timeout protects the caller's control flow and nothing else: the
|
||||
@@ -115,6 +122,11 @@ pushes; **you (Claude Code) implement**. A file being open in the editor is NOT
|
||||
plain language, and is deliberately **not** `CONTEXT.md` — do not consolidate the two.
|
||||
- **A finding goes in `OPEN-ITEMS.md` first**, never only in a report, an audit or `STATUS.md`. Four
|
||||
items in this project were minted in a spike doc and lost (R-153/154/155, R-156/157).
|
||||
- **Confirm your own last push's CI run went green, by run ID.** CI emails on failure, which is a
|
||||
PUSH signal — this is the PULL check that catches a lost, filtered or unread mail. Quote the run
|
||||
id and its conclusion in the session report, e.g.
|
||||
`curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/<repo>/actions/tasks?limit=3"` → match the
|
||||
`head_sha` to your commit. An unchecked green is an assumption, not an observation.
|
||||
|
||||
## Tech stack (Hub)
|
||||
|
||||
@@ -174,11 +186,49 @@ Steps: commit+push code → `cd /mnt/5_hdd/felhom.eu/build/felhom-hub && ./build
|
||||
(local) → bump `manifests/hub.yaml` tag + push → ArgoCD hard-refresh + sync (kubectl-patch method in
|
||||
the skill, now local `sudo kubectl`) → verify Synced/Healthy + rollout + image + startup log.
|
||||
|
||||
## Gates — ONE entry point
|
||||
|
||||
**Run `python3 scripts/repo_gates.py` after ANY change in this repo.** It is the one entry point
|
||||
and runs every gate — `site_gates.py`, `hostinstall_gates.py`, `hub_confirm_gate.py`,
|
||||
`manifest_bearer_gate.py` and `reuse_refs_check.py` on this root — streaming each gate's own output
|
||||
and exiting non-zero if any fails. `--fast` selects only the gates that touch no network and no
|
||||
container runtime; today that is all of them. A missing gate script is a FAILURE, never a skip.
|
||||
|
||||
**Why a runner and not five invocations** (2026-08-02, R-29): a census of all thirteen gates across
|
||||
the four repos found that every check a `CLAUDE.md` names was passing, and two of the four nobody
|
||||
is told to run were failing — one since 14 July. The single-entry-point shape is the only one that
|
||||
demonstrably gets run here; `app-catalog-felhom.eu/scripts/catalog_gates.py` is the canonical
|
||||
version of it (R-161) and `repo_gates.py` copies it. `site_gates.py` is a *gate*, not a runner —
|
||||
do not model new work on it.
|
||||
|
||||
**The pre-push hook.** `.githooks/pre-push` runs `repo_gates.py --fast` and refuses the push if it
|
||||
fails. It is **per-clone** and switched on once with `git config core.hooksPath .githooks` — a
|
||||
clone does not carry it, and any manual `repo_gates.py` run WARNS when this clone is unarmed.
|
||||
`git push --no-verify` bypasses it deliberately; **say so in the session report when you use it**.
|
||||
Both facts are why continuous integration is still owed (`OPEN-ITEMS.md` R-168) — this hook is
|
||||
local and skippable, and only CI is neither.
|
||||
|
||||
## Build & deploy — Website / Manifests
|
||||
|
||||
- **Website** auto-deploys via git-sync; just push to `main` (live in 1–2 min). **Run
|
||||
`python3 scripts/site_gates.py` after ANY website change**; new pages go into its `PAGES` list.
|
||||
Emergency edits: https://files.felhom.eu. All `website/` HTML is **UTF-8 with BOM** — preserve it.
|
||||
- **Website** auto-deploys via git-sync; just push to `main` (live in 1–2 min). Website changes go
|
||||
through `repo_gates.py` above (it runs `site_gates.py`); new pages go into that gate's `PAGES`
|
||||
list. Emergency edits: https://files.felhom.eu. All `website/` HTML is **UTF-8 with BOM** — preserve it.
|
||||
- **THE INSTALLER DOES NOT (R-110, 2026-08-03).** `manifests/webpage.yaml` runs **two** git-syncs:
|
||||
the website from `main` as above, and `/scripts/` from the tag **`installer-v<SCRIPT_VERSION>`**.
|
||||
Pushing `scripts/felhom-host-install.sh` therefore changes nothing that any machine downloads —
|
||||
which it used to, within thirty seconds, for the one artifact that runs as **root on a virgin box**.
|
||||
- **To publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml`
|
||||
(both the sidecar and the init container), commit, and sync. `hostinstall_gates.py` gate 6
|
||||
fails if the manifest stops naming an `installer-v…` tag or if the website stops tracking `main`.
|
||||
- **To roll back:** move the tag back to the previous commit and wait ~30 s. **No ArgoCD sync and
|
||||
no deploy** — git-sync picks up a moved tag on its next period, measured live on 2026-08-03 in
|
||||
both directions. That is the emergency lever; fix forward with a new version afterwards.
|
||||
- **Do NOT pin the website to the tag.** The sparse-checkout used to cover `/website/` and
|
||||
`/scripts/` in one sync, and pinning that would turn every copy edit into a release.
|
||||
- The **URL never carries a ref** (`https://felhom.eu/scripts/felhom-host-install.sh`), so
|
||||
`felhom-bootstrap.sh` and the hub's day-0 command follow the tag with no edit — do not add one.
|
||||
- The installer's own sixteen run-time fetches are pinned separately, to `raw/tag/v$ART_AGENT_VER`
|
||||
in the **agent** repo (R-183) — they are the agent's configs, not this repo's.
|
||||
- **Manifests** are GitOps via the `felhom` app — commit to `main`, then deliberate sync.
|
||||
|
||||
## Key patterns
|
||||
|
||||
+479
@@ -17,6 +17,392 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
|
||||
(2026-08-04). Scope: those two boxes, not the fleet.**
|
||||
|
||||
Four SCHEDULED runs overnight, none triggered by hand: demo-felhom host **83.8 s** / offsite
|
||||
**540.4 s**; demo-hp host **109.3 s** / offsite **300.1 s**. Every one restored into a scratch guest,
|
||||
booted, verified and destroyed itself.
|
||||
|
||||
*What this closes.* Until yesterday every live restore-proof this project held was on the OFFSITE
|
||||
tier. The on-box tier — the one an ordinary recovery uses — had never been proven on either box, and
|
||||
not because it failed: the agent could not read the storage it lives on (R-185), so it never saw an
|
||||
archive there to test.
|
||||
|
||||
*What was observed for the first time.* Both boxes had BOTH tiers due simultaneously. Never-proven
|
||||
sorts first, so each took its host tier, deferred the offsite one, and picked that up on the next
|
||||
evaluation six hours later — R-86's ordering and the one-heavy-operation gate, working together,
|
||||
unsupervised. The host-tier proof then reached the hub through R-189's merge, which is that path
|
||||
carrying a host-tier entry for the first time.
|
||||
|
||||
*The asymmetry worth remembering:* a host-tier restore is **83–109 s**; an offsite one is
|
||||
**300–540 s**. The tier that matters for an ordinary recovery is also the cheapest to prove.
|
||||
|
||||
**S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission
|
||||
question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).**
|
||||
|
||||
*The defect.* On both demo boxes the agent's token had `FelhomAgentStore` on `local`, `local-lvm` and
|
||||
`felhom-pbs` and **not** on `felhom-backup` — the storage the same installer configured as
|
||||
`local_backup_target`. The content API answered `{"data":[]}` through the token while root listed
|
||||
three archives. `pickForThisRun` skipped the tier as *"no settled archive yet"*, which is exactly
|
||||
what a brand-new tier reports, so the host tier was never restore-testable and nothing said so.
|
||||
|
||||
*The rule.* The permission question has a definite answer where the listing does not. `Permissions`
|
||||
reads `/access/permissions?path=/storage/<target>` **as the agent's own token** — asking as root
|
||||
answers a different question and always says yes — and one `capability.Status` per configured tier
|
||||
reports it. The probed set comes from `BackupTiers()`, never a fixed list: a hardcoded probe list is
|
||||
the defect reproduced inside the fix.
|
||||
|
||||
*The measured trap, because the obvious reading is wrong.* An ungranted path answers **neither empty
|
||||
nor 403**: it carries the privileges inherited from the box-wide `/` grant
|
||||
(`Sys.Audit, SDN.Use, Datastore.Audit`). Testing for path-presence, or for `Datastore.Audit`, reports
|
||||
a blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**, and re-measuring is required
|
||||
before that constant is ever changed.
|
||||
|
||||
*Criticality, weighed once.* Critical, because the hub alerts only on critical and a non-critical
|
||||
entry would ride the report and alert nobody. **Except** the `local` fallback target, which
|
||||
host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it
|
||||
does not page — turning an ordinary documented setup into an alert is how a signal becomes something
|
||||
an operator archives unread. It never consults content, so it cannot alarm on a newborn tier by
|
||||
construction, and it never reports ok when it could not ask.
|
||||
|
||||
**S-22 — the installer's Scenario-F arm must finish the job, not just leave the definition alone
|
||||
(2026-08-03, R-185).** `configure_backup_target` has two arms. Case A creates the storage and grants
|
||||
in the same breath. The reuse arm — *"the target already exists"* — returned **without granting**, and
|
||||
that, not `PVE_STORAGES`, is where the drift came from: a box whose target pre-dated the install
|
||||
(the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token
|
||||
could not read. The reuse arm now ensures the ACL through the same guarded wrapper. **Scenario F is
|
||||
unviolated** — the storage DEFINITION is untouched, and granting the role the agent is supposed to
|
||||
have on the target this script is about to write into `agent.json` is finishing the job, not
|
||||
retargeting the box. `$BACKUP_TARGET_ID` stays OUT of `PVE_STORAGES`: that list is granted a step
|
||||
before the target is resolved, and `--acl-storages` entries are preflight-checked for existence.
|
||||
A gate asserts every arm that resolves the target also grants on it.
|
||||
|
||||
**S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is
|
||||
the design (2026-08-03, R-189; agent v0.122.0).**
|
||||
|
||||
*The rule.* Only successful restore-tests are written to `RestoreTestState`, and that state is what
|
||||
the host report carries after a restart. Failures live only in the in-memory `backup.Store`.
|
||||
|
||||
*Why, in one line each.* A **success suppresses future work** — under R-86's per-archive due-check a
|
||||
proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it
|
||||
believes, for a whole archive generation (a week on the offsite tier). A **failure causes future
|
||||
work** — a failing tier stays due and is retried at the next evaluation, so a lost failure heals
|
||||
itself within one interval, while a *persisted* failure would outlive the fault it describes.
|
||||
|
||||
*What the report does with the two.* The collector merges them: **one entry per tier, newest by
|
||||
`TestedAt` wins**. A fresh failure therefore beats a stored success (the failure is the news and
|
||||
exists nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier can
|
||||
never appear twice — the hub would read that as two tests.
|
||||
|
||||
*It refuses to lie.* A persisted record missing the archive **or** the tier produces **no entry**: an
|
||||
unproven tier reading as proven would be worse than the defect this closes. Run mechanics (scratch
|
||||
VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be.
|
||||
**Migration consequence, seen live:** a pre-R-189 record has no tier, so upgrading does not
|
||||
retroactively make an old proof visible to the hub; the tier's next real proof fills it in.
|
||||
|
||||
**S-20 — the release order is build → tag LOCALLY → publish → push tag, and every step protects
|
||||
something (2026-08-03, R-188 + R-186).**
|
||||
|
||||
The tag is created before the publish, so the build and the tag describe the same commit. It is
|
||||
**pushed** after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
|
||||
package made `check-published-versions.py` correctly fail a *correct* release — measured on roughly
|
||||
every second release, and R-168 mails those failures to the operator.
|
||||
|
||||
The invariant the old order protected is **asserted directly instead**: the gate now also refuses a
|
||||
**published version with no tag**, as a bounded probe (frontier + patch gaps) that prints its own
|
||||
coverage, because the package listing api is 401 without a token and absence cannot be enumerated.
|
||||
A half-done release is loud: publish-then-failed-push dies naming the recovery command, and a failed
|
||||
publish deletes the local-only tag so a retry is clean.
|
||||
|
||||
**A released binary is independently verifiable** — `-trimpath -buildvcs=false` means the same source
|
||||
yields the same bytes with or without the tag; the verification command lives in
|
||||
`felhom-agent/CLAUDE.md`. Both build paths (`release-agent.sh` and `publish-agent.sh`'s fallback) use
|
||||
identical flags: they differed by `CGO_ENABLED=0` and produced binaries 74 KB apart for one version.
|
||||
|
||||
**S-17 — restore-testing is PER ARCHIVE GENERATION, and the hub's staleness window follows each
|
||||
tier's own rhythm (2026-08-03, R-86; agent v0.121.0 + hub v0.91.0).**
|
||||
|
||||
*The rule.* Let **A** be the newest archive on a tier that has settled for at least the settle lag
|
||||
(24 h). The tier is **DUE** when A exists and **A has not already been proven**. The daemon-start
|
||||
ticker survives only as the **evaluation interval** (6 h). A daily tier is proved daily on yesterday's
|
||||
archive; a weekly tier weekly on its own; a tier with no archive is UNKNOWN, never a fault.
|
||||
|
||||
*The trap, written down so it is not reintroduced.* The literal reading of R-86 — *"due when the
|
||||
newest archive is ≥24 h old"* — is **never true on a daily tier**, because a new archive resets the
|
||||
newest-archive age to zero long before it reaches the lag. It would have switched restore-testing off
|
||||
for the tier that matters most, silently. Red-proved at 0 runs over 5 simulated days
|
||||
(`felhom-agent/internal/backup/restoretest_due_test.go`).
|
||||
|
||||
*What the state holds now.* `RestoreTestState` records **which archive** was proven, not just when a
|
||||
tier passed — a timestamp cannot answer *"have we proven THIS archive"*. A pre-R-86 file keeps its
|
||||
time (rotation ordering survives a deploy) and yields no proven archive, so each tier is due exactly
|
||||
once after the upgrade.
|
||||
|
||||
*The old config key.* `backup.restore_test_cadence_seconds` is DEPRECATED. **Negative still disables**
|
||||
verbatim; a positive value now seeds the **settle lag** only, and the daemon WARNs once at start-up
|
||||
naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
|
||||
(default 24 h). It is deliberately NOT carried into the evaluation interval.
|
||||
|
||||
*The hub half is not optional.* `restoreProvenStaleAfter` was a flat 7 days **derived from the cadence
|
||||
R-86 removes**, and a healthy weekly tier's proof age reaches EXACTLY 168 h just before its next
|
||||
proof — it sat ON the line. `restoreProvenWindow(tier, observed, ok)` now takes the tier's own
|
||||
observed archive interval × 4 generations, floored at 7 days, capped at 12 days (strictly inside the
|
||||
2-week offsite retention), falling back to the tier's **declared** rhythm (`backupStaleAfter` 26 h /
|
||||
`offsiteBackupStaleAfter` 8 d — the backup-freshness checker's own thresholds) when history is too
|
||||
short to observe one. Shipping Part 1 alone would have produced a nightly false alarm.
|
||||
|
||||
**S-18 — `ep0` is Tier 2, PROTECTED (operator ruling, 2026-08-03).** D-d named two protected machines
|
||||
and did not name ep0 either way; `runbooks/target-selection.md` carried the question in writing for
|
||||
two days. The ruling **extends D-d's protected list to three machines**: DooPlex, Peti's cluster,
|
||||
**ep0**. It is a classification, not a new set of prohibitions — destroying datastores, prune jobs,
|
||||
tunnel config or nftables rules was already forbidden by what it would destroy, and the ordinary
|
||||
off-site READ a restore-test performs remains permitted.
|
||||
|
||||
**S-13 — the `mp1` merge landed, and the variant was chosen on measurement (2026-08-03, R-165 / D-a).**
|
||||
The appliance's two data volumes are one. **Variant V-c**: the volume mounts at the NEUTRAL path
|
||||
`/var/lib/felhom`, and both `/var/lib/docker` and `/mnt/sys_drive` are binds of subdirectories of it.
|
||||
**Three shapes were built and rebooted before choosing** (`audits/SPIKE-r165-phase0-2026-08-03.md`) —
|
||||
all three boot, reboot 3/3, give ONE `df` figure and keep a container's `statfs("/")` on the merged
|
||||
volume, so **the ordering risk that motivated the probe was not what mattered.** They differ only in
|
||||
which documented guarantee they break: volume-at-`/var/lib/docker` puts customer backups inside
|
||||
Docker's data-root, so the ordinary "clear `/var/lib/docker`" reflex destroys every local unit;
|
||||
volume-at-`/mnt/sys_drive` puts Docker's entire data-root under `/mnt`, which the controller container
|
||||
mounts wholesale — **measured: it then sees `/mnt/sys_drive/docker`**, falsifying the bootstrap's own
|
||||
comment that `/mnt` holds only Felhom's namespace mounts. **V-c breaks neither**, for one extra path.
|
||||
|
||||
**B2 is the bulkhead replacement, and "or prune the oldest" is REJECTED with its reason**, because the
|
||||
question will be asked again: nothing on that filesystem is generational — a unit is ONE fixed path per
|
||||
app (`backups/primary/<app>`) refreshed in place, and a DB dump is `<stack>-<dbtype>.sql`, also fixed —
|
||||
so pruning could only mean deleting a **different** app's only local recovery unit.
|
||||
`pruneStalePrimaryDirs` is an ORPHAN sweep with no notion of age and must never be repurposed.
|
||||
|
||||
**No migration exists, and that is a ruling not an omission:** every node is REINSTALLED. Both demo
|
||||
boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed shortly.
|
||||
So R-176's in-place migration rehearsal is **withdrawn**, not deferred.
|
||||
|
||||
**S-14 — prove first, then vouch (2026-08-03) — SPENT, and the ordering did not survive contact.** The
|
||||
rule was: golden **0.192.0** stays UNVOUCHED until a box has been proven from it, because vouching is
|
||||
what makes a fresh install pick a golden up. **In the event the golden was vouched at 07:23:26 CEST on
|
||||
2026-08-03, before any box was reinstalled** (hub log `Artifact manifest set: agent=0.119.0
|
||||
golden=0.192.0`), so the ordering was already spent when R-178's session opened; the operator elected
|
||||
to accept it rather than revert the manifest. **Both boxes were then reinstalled and proven** (R-178,
|
||||
`REPORT.md`), so the end state is the intended one and no unproven layout was ever in front of a real
|
||||
install — but the rule protected nothing, because nothing enforced it. **The lesson is R-115's, one
|
||||
layer up:** an ordering that lives only in a `CONTEXT.md` sentence and a runbook's §7 is a reminder,
|
||||
and reminders do not hold. If prove-then-vouch is to be a rule it needs the shape R-120's gate has —
|
||||
a refusal at `handleSetArtifacts`, the sole path to `SetArtifactManifest`, which runs without anyone
|
||||
choosing to run it.
|
||||
|
||||
**S-15 — the merged layout is proven live, by two different supply paths (2026-08-03, R-178).** Both
|
||||
demo boxes were wiped and reinstalled from golden 0.192.0 and taken through claim → deploy → back up →
|
||||
**restore**. **demo-hp** was installed with `--golden <local volid>` (the layout proof) and
|
||||
**demo-felhom** by the normal manifest route with `--force-gitea-golden` (the pipeline proof —
|
||||
`verified sha256 54e2a4c431daf580… matches the hub manifest`), deliberately different so the session
|
||||
proved the disk shape *and* the delivery route rather than one of them twice. Live shape on both:
|
||||
`mp0` at `/var/lib/felhom`, `backup=1`, **no `mp1`**; `/var/lib/docker` and `/mnt/sys_drive` both real
|
||||
mounts of its subdirectories via `/etc/fstab`; ONE `df` figure and one device id on all three paths;
|
||||
3/3 reboots each with the binds surviving every time. B2 was **not** proven on that pass → **R-181**:
|
||||
the floor guarded `captureAllRecoveryUnits` and not `runVolumeDumps`, the leg that fills the volume,
|
||||
and its refusal's "the previous unit is untouched" was measured false. **R-181 CLOSED the same day
|
||||
(controller v0.193.0 + v0.193.1), so R-165 is now PROVEN-LIVE in both halves** — see S-14.
|
||||
|
||||
**S-14 — the reserve is a per-app, per-run ADMISSION decision, not a capture check (2026-08-03, R-181;
|
||||
controller v0.193.0 + v0.193.1).** B2 as first shipped was consulted in exactly one place —
|
||||
`captureAllRecoveryUnits`, a few KB — while `RunDBDumps`' database leg and `runVolumeDumps` wrote the
|
||||
bulk into the same `backups/primary/<app>` tree, first and unguarded. The reserve was therefore
|
||||
consumed by the very write it exists to bound, and the refusal then claimed *"the previous unit is
|
||||
untouched"* about a tree the earlier leg had already rewritten (182,272 B → 2,147,666,432 B under a
|
||||
manifest that had not moved). **Sixth entry in `CLAUDE.md`'s table of shipped guarantees the code did
|
||||
not provide, and the fourth of those found on live hardware rather than by review.**
|
||||
|
||||
- **`internal/backup/admission.go` — `admitApp` is now THE gate**, and every per-app write leg calls
|
||||
it. One verdict per app per run covers all three; they share one per-app root, which is what makes
|
||||
that honest.
|
||||
- **Decided lazily at the app's first write, never once at run start** (app A's dump can put app B
|
||||
under the reserve), **never re-decided between an app's own legs** (that is the split it closes),
|
||||
and **reset per run**.
|
||||
- **Ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided
|
||||
inside it has already bounced the app. **After** the volume-less check, which has no write to gate.
|
||||
- **Size term added:** *would THIS app's write cross the reserve?*, estimated from the app's previous
|
||||
`.sql` + `.tar`. **No history → headroom-only**, deliberately — otherwise the first backup is the
|
||||
one that can never happen.
|
||||
- **A container-based `du` was MEASURED and rejected**, not waved away: median **~355 ms/volume** over
|
||||
66 runs on demo-hp, on volumes holding tens of KB (container start-up, not the walk). Decisive on
|
||||
top: `docker run` needs the writable layer, so the instrument can fail under exactly the pressure
|
||||
the reserve handles.
|
||||
- **The wording was NOT weakened; the behaviour moved so it became true**, and it is checked by
|
||||
sha256 tree fingerprint, never by reading the log line — the log line is what lied.
|
||||
- **v0.193.1**, found by the proof run itself: a 178 KB estimate printed as `0.00 GiB`, which reads as
|
||||
*no estimate available*. Rendering moved to `humanizeBytes`; arithmetic still in GiB.
|
||||
- **New finding, deliberately not fixed here → R-182**: `GetFullStatus`'s periodic capture sweep has
|
||||
no run scope, so a refused app re-alerts on every status refresh (measured: a second identical alert
|
||||
pair 13 s after the run's). Pre-existing in v0.192.0; R-181 changed neither caller.
|
||||
|
||||
**S-15 — publishing is an act, not a side-effect of pushing (2026-08-03, R-110 + R-115 + R-183).**
|
||||
Two rulings, one shape: something became live because someone pushed, not because anyone decided.
|
||||
|
||||
- **The installer.** `/scripts/` now git-syncs the tag `installer-v<SCRIPT_VERSION>`; the **website
|
||||
keeps tracking `main`** in a second sync, because pinning both would make every copy edit a
|
||||
release. Publish = cut the next tag + bump the manifest `--ref` + sync. **Roll back = move the tag
|
||||
back**, which takes ~30 s and needs no ArgoCD sync at all — git-sync v4.4.0 follows a moved tag,
|
||||
and that half was measured before the manifest was touched because the whole model rests on it.
|
||||
- **The sixteen run-time fetches were NOT what the spec described** — sixteen, not nine, and from
|
||||
`felhom-agent`, not this repo — so no tag here could cover them. They are pinned to
|
||||
`raw/tag/v$ART_AGENT_VER` instead, which is strictly better: the agent's configs now come from the
|
||||
same ref as the agent binary being installed. That closed a real skew (**R-183**), not just a
|
||||
channel.
|
||||
- **The URL needed no change**, and that is worth knowing rather than re-deriving: it never carried
|
||||
a ref, so both producers follow the tag automatically — and no hub change means no hub bump.
|
||||
- **The agent.** `scripts/release-agent.sh` is THE release path: build → tag → publish → **verify by
|
||||
an independent download**. It does not vouch. `check-published-versions.py` refuses a `v<semver>`
|
||||
tag with no downloadable package, and **CI now runs the full gate set** rather than `--fast`,
|
||||
without which that gate would have been registered and never run.
|
||||
- **The gate's invariant is not the one specified, and P-C is why:** the hub manifest and Gitea's
|
||||
package listing are both **401** anonymously; the package download and the tags api are not. So CI
|
||||
can ask *is this installable* but not *what is vouched*. The residue is **R-184**.
|
||||
- **Neither gate asserts "the newest version is published."** That would go red on the very push
|
||||
that bumps a version, before publishing — and a gate that fails on the normal path is one people
|
||||
learn to ignore.
|
||||
|
||||
**S-16 — a backup run NOTIFIES ONCE and RECORDS ALWAYS, and those are different things
|
||||
(2026-08-03, R-182; controller v0.194.0 + hub v0.90.0/.1).** Measured: nine per-app capture failures
|
||||
reached the hub, two were mailed, seven were dropped by a cooldown whose key carries no app
|
||||
identifier — *before* `LogNotification`, so they left no row anywhere.
|
||||
|
||||
- **The record:** `recovery_unit_capture_failed`, per app, unconditionally, now routed
|
||||
**record-only** by the hub (`recordOnlyEvents`) — stored and logged every time, never competing
|
||||
for an e-mail slot.
|
||||
- **The notification:** `backup_run_failures`, ONE per run, listing every failed app with its leg
|
||||
and reason plus the counts and free space. **A clean run emits nothing.**
|
||||
- **A suppressed operator event now leaves a `suppressed` row** carrying the key that suppressed it —
|
||||
for every operator type. *"We chose not to e-mail you"* and *"nothing happened"* must never look
|
||||
identical; that is the whole finding, stated as a rule.
|
||||
- **The periodic sweep gets a digest too, with NO `run_id`**, so it stays under the ordinary hourly
|
||||
cooldown. Without it the sweep's failures would be recorded and never notified — a new silence
|
||||
created while closing one. A real run's digest carries a unique `run_id` precisely so the cooldown
|
||||
can never collapse a manual run into the nightly one.
|
||||
- **Why the silence is safe:** the hub's deadline check raises `expected_backup_missed` from report
|
||||
freshness, independently of any mail the box sends (`monitor/deadline.go:396,417`). **If that check
|
||||
is ever weakened, this design loses its footing.**
|
||||
- **Not taken, and why:** putting `app` in the cooldown key. It fixes the swallowing by producing one
|
||||
mail per failing app — a dozen on a full disk.
|
||||
|
||||
**ep0 was rescaled by the operator to a CX33 (2026-08-03): 4 vCPU, 8 GB RAM, measured on the box, and
|
||||
the 4 GiB swapfile survived. The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only, so no disk
|
||||
figure in any runbook needed correcting. That closed **R-90** and unblocked **R-86**.
|
||||
|
||||
**S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).**
|
||||
Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the
|
||||
CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure
|
||||
is the OPERATOR's** (they can do none of those things about it). R-158 proposed emitting the existing
|
||||
`backup_failed` for the capture failure. **That was rejected and D-c wins**, because `backup_failed`
|
||||
carries a `customerMessages` entry AND sits in `settings.DefaultEnabledEvents` — so reusing it emails
|
||||
the customer, in Hungarian, that their backup failed, about something they cannot act on. It is
|
||||
exactly the mistake R-97a avoided by minting `whole_guest_backup_failed`, and the reasoning is written
|
||||
into `hub/internal/api/handler.go`'s allowlist. New type: `recovery_unit_capture_failed`, in
|
||||
`allowedEventTypes` **and** `notify.operatorOnlyEvents` — **the second register is what makes it
|
||||
operator-only; the first does not**, and v0.78.0 claimed otherwise and shipped the defect.
|
||||
|
||||
**The customer half reused the pair that already existed rather than minting a seventh type.**
|
||||
`disk_warning`/`disk_critical` were allowlisted, carried Hungarian copy, sat in `DefaultEnabledEvents`
|
||||
and had a UI checkbox — and **nothing in any repo emitted them**. A complete customer pipeline with no
|
||||
producer: the **sixth** *built-but-never-wired* instance in this project. `internal/fillwatch` is now
|
||||
that producer. Their generic `customerMessages` entries were **deleted**, because
|
||||
`FormatCustomerEmail` PREFERS the entry over the message and a static template would discard the drive
|
||||
label and the free-space figures — the same reason `offbox_enlarge_blocked` and `disk_health_degraded`
|
||||
have none. `notify.IsOperatorOnly` was added so ONE test pins both registers; checked separately, an
|
||||
allowlisted-but-not-operator-only type is invisible.
|
||||
|
||||
**S-12 — the monitoring landed BEFORE the merge, not with it (2026-08-02).**
|
||||
D-a's condition (2) says R-167 ships in the same step as the `mp1`→`mp0` merge and never after,
|
||||
because the merge removes a wall that currently fails safely. **This session landed it FIRST**, which
|
||||
is strictly better and costs nothing: the warnings went in and were proven on real hardware while the
|
||||
wall is still standing, so the merge session inherits a proven signal instead of an untested one.
|
||||
**No disk layout was touched.** R-165's measurement is `audits/SPIKE-r165-mp1-merge-2026-08-02.md`,
|
||||
which STOPS at a question for the operator (which merge shape; what replaces the bulkhead). Its two
|
||||
load-bearing findings for anyone picking that up: **"the layout" is not one thing** (demo-felhom
|
||||
`200G/50G`, demo-hp `50G/20G`, golden `16G/8G` — so §7.5's bound is one box's, → R-175), and **`mp1`
|
||||
is also a BULKHEAD**, not only a ceiling — today an overflow cannot reach `/var/lib/docker`, and after
|
||||
the merge it can.
|
||||
|
||||
**S-8 — CI detects; it does not block, and that is structural (2026-08-02, R-168).**
|
||||
A Gitea Actions runner in `gitea-system` re-runs every repo's gate entry point on every push,
|
||||
independent of who pushed and of what they typed. It **cannot refuse a push**: every felhom repo
|
||||
pushes straight to `main` with no pull request, so there is no merge for a status check to stand at.
|
||||
That is not a gap in the runner — there is no gate in the road. The refusing half is
|
||||
`.githooks/pre-push` (per-clone, `--no-verify`-able); this half is what notices when that hook was
|
||||
skipped or was never armed. Making CI blocking needs branch protection plus a PR workflow, which
|
||||
changes how the operator works and is **their** call → R-169. Do not "fix" this by adding branch
|
||||
protection.
|
||||
|
||||
**S-9 — a detector that tells no one is not finished (2026-08-02, R-168 probe P5).**
|
||||
Probe P5 measured that a failed run produces **no mail, no notification row and no log line** from
|
||||
Gitea. So the workflow sends its own alarm on the project's existing Resend path and **prints the
|
||||
provider's accepted id**, which makes "a message left the machine" an observable rather than an
|
||||
assumption. The acceptance test for this work was never a green pipeline — it was a red run with a
|
||||
message in hand (`RESEND-ACCEPTED id=…`). Two traps found while building it, both worth keeping:
|
||||
the runner image has **no `curl`** (deliberately — python3 and git only, so use `urllib`), and
|
||||
`api.resend.com` sits behind **Cloudflare, which 403s the default `Python-urllib` User-Agent with
|
||||
error 1010** — a failure that looks exactly like an auth failure and is not one.
|
||||
|
||||
**S-10 — the runner is unprivileged, and the reason is the host (2026-08-02).**
|
||||
The usual `act_runner` recipe pairs it with a `docker:dind` sidecar and `privileged: true`. Rejected:
|
||||
DooPlex is **Tier 2** and *is* the recovery chain — Gitea, the hub, the registry, PBS and
|
||||
k3s + Longhorn all live on it and it cannot be rebuilt from anything else. Host execution mode costs
|
||||
nothing here because every CI job is one Python command. Consequence to remember: **in host mode the
|
||||
job sees exactly the runner image's tools**, which is why `python3` had to be baked in (probe P2:
|
||||
stock `act_runner` carries git but not python3). If a future job genuinely needs Docker, that is a
|
||||
conversation, not a patch.
|
||||
|
||||
**S-11 — CI reproduces the workspace's sibling layout, because two entry points depend on it
|
||||
(2026-08-02).** `controller_gates.py` and `agent_gates.py` invoke the shared `reuse_refs_check.py`
|
||||
that lives in the `felhom.eu` clone next door and is deliberately never copied, and both repos'
|
||||
`REUSE.md` files cite a path that lives in the hub. Their workflows therefore clone `felhom.eu` as a
|
||||
sibling; without it the gate fails **closed** — correctly, but for the wrong reason. Verified that CI
|
||||
and the local hook then agree exactly (controller 126 exact / 6 suffix / 1 cross-repo).
|
||||
|
||||
**S-6 — the hub renders no host-install version, and the gate pins its absence (2026-08-02, R-94).**
|
||||
The Setup tab's *"host-install 1.19.0"* label is **deleted, not derived**. Deriving it is not
|
||||
achievable honestly: the Option-1 command downloads `felhom-host-install.sh` from the website **at
|
||||
run time**, and the website git-syncs `main` every 30 seconds (R-110) — so the hub cannot know which
|
||||
version a given box will run, at build time or at render time, and any literal there is a guess
|
||||
wearing a version number's authority. The real one drifted to 1.19.0-vs-1.22.0 and stayed wrong for
|
||||
nineteen days. `hostInstallVersion`, `pageData.ScriptVersion` and the rendered label are gone; a NOTE
|
||||
sits where the const was so it is not helpfully re-added, and `scripts/hostinstall_gates.py` gate 1
|
||||
**inverted** — it now asserts the hub carries no host-install version literal in any of six code
|
||||
shapes across every `.go`/`.html` under `hub/`. Corollary that generalises past this row: the
|
||||
tautological `render_test.go` assertion (`html contains hostInstallVersion`, where the same constant
|
||||
put it there) **passed at `9.9.9`** — an assertion that compares a value to itself tests the
|
||||
plumbing, never the claim.
|
||||
|
||||
**S-7 — gates run from ONE entry point per repo, and `reuse_refs_check` was fixed rather than the
|
||||
convention it polices (2026-08-02, R-29).** Two rulings from the same census.
|
||||
|
||||
*Where gates run.* Thirteen gate scripts exist across the four repos. Measured 2026-08-02: **every
|
||||
check a `CLAUDE.md` tells a person to run was passing, and two of the four nobody is told to run were
|
||||
failing** — one since 14 July. The correlation was exact, so the fix is not more gates but one place
|
||||
to run them from: `scripts/repo_gates.py`, `felhom-controller/controller/scripts/controller_gates.py`,
|
||||
`felhom-agent/scripts/agent_gates.py`, `app-catalog-felhom.eu/scripts/catalog_gates.py` — each
|
||||
mandated in its `CLAUDE.md`, each wired to `.githooks/pre-push` with `--fast`. The canonical shape is
|
||||
`catalog_gates.py` (R-161), **not** `site_gates.py`, which is a gate (eight assertions in one file)
|
||||
and not a runner; copying it produces another monolith nobody invokes. **A missing gate script is a
|
||||
FAILURE with the path printed, never a skip.** The hook's limits are real and are written into the
|
||||
hook: per-clone (`core.hooksPath` is local config) and `--no-verify`-able on purpose. The
|
||||
unbypassable half is CI → **R-168**.
|
||||
|
||||
*Why the checker moved and the docs did not.* `reuse_refs_check.py` was RED on all four repos with
|
||||
13 findings, of which a hand audit found **zero** genuine drift — twelve were package shorthand
|
||||
(`appbackup/userdata.go` → `controller/internal/appbackup/userdata.go`) and one, `wgsync/reconciler.go`,
|
||||
is cited by the controller and lives in the hub. `REUSE.md` cites by package shorthand and across
|
||||
repos deliberately; that convention is the useful one. **Rejected, so they are not revisited:**
|
||||
rewriting all four `REUSE.md` files to full paths (makes the docs worse to serve the tool), and
|
||||
deleting the checker (drift across four repos is a live risk). The checker now resolves
|
||||
exact → suffix → ambiguous → sibling repo → FAIL, **prints every non-exact hit and a per-rule tally**
|
||||
(because "0 failures" alone cannot tell a working checker from a blind one), and lists every
|
||||
resolution attempted on a failure. It stays in **one** place and is invoked across the workspace —
|
||||
never copied, which would recreate the drift it detects.
|
||||
|
||||
**S-1 — N.5 gains a third leg: architecture docs are same-session coupled (2026-07-26, R-81).**
|
||||
Any task that changes an **architectural contract** — tiers, targets, cadences, trust boundaries —
|
||||
updates the owning `documentation/architecture/*.md` in the **same session**, under exactly the same
|
||||
@@ -87,6 +473,99 @@ Five decisions were deliberately **left open for the operator** and are recorded
|
||||
stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with
|
||||
the guest it backs up. Gaps minted the same session: **R-102 … R-108**.
|
||||
|
||||
**S-13 — boot recovery finished, and the lesson is about the DIAGNOSIS ORDER (controller v0.190.0,
|
||||
2026-08-02, R-157 A · R-170 · R-171).**
|
||||
|
||||
**The session's most valuable half hour was spent NOT writing code.** A hole was reasoned out of the
|
||||
v0.189.0 diff — replacing the container-count term with recorded intent should make a
|
||||
drive-gate-stopped app read as a boot orphan — and the task's own rule was to CONFIRM it on hardware
|
||||
before writing a fix. **The first attempt to confirm it produced a false negative**, and reporting
|
||||
that as a disproof would have been wrong: unmounting only the parent bind is healed by the agent
|
||||
within ~60 s, so the drive gate's startup reconcile restarted the apps **one second before** the
|
||||
sweep looked. `no boot-orphaned apps` in that log is a race that went the safe way, not a mechanism.
|
||||
Holding the drive genuinely absent reproduced it immediately. **"It didn't happen this time" is not a
|
||||
disproof — name the mechanism or run it again.**
|
||||
|
||||
**The confirmation also changed the severity, in both directions.** The write hazard did NOT
|
||||
materialise: compose failed `mkdir …/userdata: permission denied`, because the unbound mountpoint is
|
||||
host-root-owned and the guest is unprivileged. **That protection is accidental** — no code chose it,
|
||||
no test pins it, it is one `chown` or one privileged guest away from gone, and its removal would be
|
||||
invisible until data landed on the wrong disk. Meanwhile the harm that DID occur was real on every
|
||||
box and was not in the hypothesis: two wasted attempts and a **false dead-app alarm for an app the
|
||||
drive gate is deliberately holding**. Diagnosing first is what produced both facts.
|
||||
|
||||
**The fix was already in the codebase, on another path.** The API's `startGatedByMissingDrive`
|
||||
refuses a customer's start on an absent drive with a Hungarian message. The sweep bypassed it by
|
||||
calling `Manager.StartStack` directly. **`StartStack` has no gate of its own** — that is the durable
|
||||
fact worth carrying: every caller that is not the customer must decide for itself whether the app may
|
||||
run, and there are now fourteen of them.
|
||||
|
||||
**Widening a window makes previously-unreachable overlaps reachable, and that is a design input, not
|
||||
an afterthought.** The old T+5 s sweep never met a quiesce or an in-flight app-data operation; a
|
||||
50 s window can. All three holders answer one seam rather than three, because they differ only in
|
||||
the reason string.
|
||||
|
||||
**A test rejected my first constant, and the comment now says so.** `settle + budget + one retry`
|
||||
must fit inside `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s
|
||||
against a 90 s grace. The budget is 50 s **because a test said so**, and the code records that rather
|
||||
than presenting the number as taste. Widening the grace to fit was rejected outright: it hides a late
|
||||
recovery instead of reporting one.
|
||||
|
||||
**AND THE FIX HAD ITS OWN DEFECT, FOUND BY LIVE VALIDATION AND NOT BY REVIEW.** The window sampled
|
||||
`GetStacks()` — the Manager's in-memory map, refreshed by the scheduler every 10 s — every 5 s. Two
|
||||
identical samples could therefore mean *the cache did not update*, not that the fleet had settled. It
|
||||
surfaced as a container removed ~5 s before the window closed still being in the sampled fleet, with
|
||||
the sweep logging `no boot-orphaned apps` for an app that had none. **Generalise it: a settle
|
||||
detector is only as good as the freshness of what it samples — if the source is cached, refresh it or
|
||||
you are watching the cache settle, not the system.**
|
||||
|
||||
**Live: 6/6 hard resets on the shipped build** (every app back; a customer-stopped app down in all
|
||||
six), window settle times 10/40/10/10/15/15 s — routinely 2–8× the old fixed 5 s. The sharpest
|
||||
evidence is a same-app before/after on one box: missed at 18:08:35, recovered at 18:18:50.
|
||||
|
||||
**S-12 — D-b is BUILT (controller v0.189.0, 2026-08-02, R-166).** The desired/in-flight/observed
|
||||
split now exists; the S-1 contract lives in `architecture/02-controller-module-map.md` §0a.
|
||||
|
||||
**Both facts D-b said to establish first were established at source, and both changed the shape.**
|
||||
(a) *Does the crash-safe journal in the backup code already cover the in-flight case?* The pattern
|
||||
DID already exist — twice (`quiesce` marker+`Recover`, `migrate` journal+`RecoverMigration`) — and
|
||||
covered **none** of the app-data path: `DumpAppVolumesSafe` stopped and restarted an app with no
|
||||
marker, no journal and **not even a `defer`**. So the answer was neither "it exists, wire it" nor
|
||||
"build it": copy the proven shape into its own file. It was the fifth time the question was worth
|
||||
asking and the first time the answer was "the pattern, not the coverage". (b) *Is the SQLite store
|
||||
reachable?* Reachable and **deliberately not used** — `metrics.db` is optional by design (the
|
||||
controller runs with it absent), and operational state must not live in a store built to be dropped.
|
||||
|
||||
**The ruling that carried the design: absent means UNKNOWN, never "running".** Every `app.yaml` on
|
||||
every box predates the field, so absent is what the whole fleet reads on upgrade; reading it as
|
||||
running would have started every deliberately-stopped app on the first boot after the upgrade —
|
||||
fleet-wide, silently. Where intent is unknown the box keeps the OLD inference rather than inventing
|
||||
an answer. That is also why the backfill is **running-only**: "zero containers ⇒ stopped" is the
|
||||
defect itself, so an ambiguous app stays ambiguous until a customer presses a button.
|
||||
|
||||
**The other load-bearing ruling: `StartStack`/`StopStack` are NOT writers of intent.** A census found
|
||||
14 callers, of which exactly 2 are the customer. Recording intent in the primitive would make a
|
||||
nightly backup indistinguishable from the customer pressing Stop — the confusion being removed.
|
||||
|
||||
**Found on the way, and it would have silently eaten the feature: `SaveAppConfig` rebuilt `AppConfig`
|
||||
field-by-field.** That is the R-100 shape, which v0.181.0 shipped two live instances of. The literal
|
||||
named five fields, so the sixth would have been dropped on every save across nine call sites — a
|
||||
customer's Stop erased by the next unrelated `app.yaml` write. Copy-and-overlay is safe by
|
||||
construction; the failure mode is generic, so **treat any field-by-field struct rebuild in a save
|
||||
path as a defect on sight.** Measured and documented rather than assumed: `app.yaml` does NOT
|
||||
round-trip YAML keys the struct does not model.
|
||||
|
||||
**Closes R-157 mechanism B; mechanism A (the sweep observes ~5 s after start and never re-checks) is
|
||||
untouched and is now the whole of R-157** — and B's fix makes A cost more, since the sweep now has
|
||||
more it could legitimately recover. **New: R-170** — `shouldRecreateOnBoot`
|
||||
(`internal/web/intermediary.go:131`) still infers a Stop from `hasContainers`, i.e. the same defect
|
||||
one gate over for drive-backed apps. Left deliberately: the task scoped `bootrecon`, and two boot
|
||||
behaviour changes under one live validation is one too many.
|
||||
|
||||
**Live on 9201, three flows, each with a positive observable.** The interrupted-operation half is
|
||||
**IMPLEMENTED, not PROVEN-LIVE** — unit-proven and red-proofed, but nobody killed the controller
|
||||
mid-backup on real hardware; the capability map says so rather than rounding it up.
|
||||
|
||||
**S-5 — four operator decisions taken in discussion on 2026-08-02, recorded before anything is
|
||||
built.** They existed only in conversation, which is the condition the standing rules were written
|
||||
against. Labels are the ones used in the discussion (**D-a … D-d**) and are deliberately kept
|
||||
|
||||
@@ -1,113 +0,0 @@
|
||||
# REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)
|
||||
|
||||
Follows `REPORT-campaign7/8/9.md`. Root `REPORT.md` is another session's (hub v0.85.0) and was not
|
||||
clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
|
||||
|
||||
**Full audit + evidence:** `documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`,
|
||||
`documentation/tests/campaign10-evidence-2026-07-31/`.
|
||||
|
||||
## The sentence that matters
|
||||
|
||||
**Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the
|
||||
"drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No
|
||||
resource leak over 13.5 h. Two findings: R-156 and R-157.**
|
||||
|
||||
## What was established
|
||||
|
||||
- **Venue** — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the
|
||||
`/mnt/nvme-1tb` **mount root** (a subdirectory would have emitted `storage_disconnected` for demo-hp
|
||||
all night — the exact signal I1/I2 discriminate).
|
||||
- **Baselines, all read fresh.** controller `main` 0.188.0, **golden 0.188.0 (not behind)**, agent
|
||||
0.119.0 published+vouched, hub 0.86.0, ISO **1.26.1** (`f3cc86d5…`, round-trip verified live).
|
||||
The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
|
||||
- **Isolation gate — both denials captured, each with a positive control.** The PBS control **failed
|
||||
first**: four clean-looking 403s were worthless because the token was denied on its own datastore
|
||||
too (PBS token privilege separation). Fixed, re-run, denials stand.
|
||||
- **A1** fresh install from the **published** ISO. 1.26.1 is a public release image — verified against
|
||||
its bytes that it has no auto-install path — so it was driven blind via screendump + `sendkey`
|
||||
through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which
|
||||
would otherwise have been mangled and locked the box out.
|
||||
- **A2** claimed for real; discriminator flipped `dashboard not yet claimed` → `authentication required`.
|
||||
- **A3** both drives enrolled through the **real** endpoint; `mentes` accepted as backup target via
|
||||
the offer flow, ending `degraded:false / target:felhom-backup` — the I5/I6 healthy baseline.
|
||||
Four apps healthy spanning both sides of D5's split (4 × `type: secret`, 1 × `type: password`).
|
||||
- **A4** discriminators seed and read back across all four apps; rallly's over the path
|
||||
`DATABASE_URL` actually names, not the trusted socket that produced D5's false pass.
|
||||
|
||||
## Phase B — the soak
|
||||
|
||||
Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped
|
||||
deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**).
|
||||
1 461 invariant checks in total.
|
||||
|
||||
- **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale,
|
||||
never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as
|
||||
skipped rather than silently green — it fired once.
|
||||
- **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place
|
||||
variants and 330 secret-class assertions covering both sides of D5's split.
|
||||
- **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced
|
||||
`backup_target_absent` and a clean recovery.
|
||||
- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via
|
||||
`bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names
|
||||
it. That is the case R-117's spike called "the worse half".
|
||||
- **RTO, both bands measured.** S: 66 MB → **42.0 s** / **41.4 s** across two passes (66 restores).
|
||||
M: 21.1 GB → **608 s** mean over 2 reps, both returning the correct discriminator. 327× the data
|
||||
cost 14.5× the time, giving **RTO ≈ 40 s + 26.9 s/GB** (backup ≈ 29 s + 17.4 s/GB). The fixed ~40 s
|
||||
dominates below ~1.5 GB — that is the S band, and why its numbers clustered so tightly.
|
||||
- **Capacity ceiling, and the more consequential result:** a DB-backed app's recovery unit is **1.90×**
|
||||
its data (volume tar + SQL dump). The default `/mnt/sys_drive` is **20 GB**, so on a default box the
|
||||
largest locally-backupable app is **≈ 10 GB** — **the M band does not fit at all** without a
|
||||
per-customer `SysDataGrowGB`. **RPO still not measured.**
|
||||
- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds
|
||||
flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots
|
||||
and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** —
|
||||
bounded by journald, but a lot of logging.
|
||||
- Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that
|
||||
was not first observed live.
|
||||
- **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system
|
||||
disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the
|
||||
soak would have produced green cycles that tested nothing cross-drive.
|
||||
- **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that
|
||||
never landed became a fake "stale restore", and a real one would have looked identical. Fixed and
|
||||
red-proofed before restarting.
|
||||
|
||||
## Findings
|
||||
|
||||
- **R-156 (new)** — **papra's data is neither persisted nor backed up, and it reports healthy.** The
|
||||
template mounts `papra_data:/app/data`; the app writes `/app/app-data/db/db.sqlite`. The volume is
|
||||
empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
|
||||
sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
|
||||
Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
|
||||
- **R-157 (new)** — **bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two
|
||||
mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while
|
||||
docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences,
|
||||
intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is
|
||||
exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp
|
||||
check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on
|
||||
every channel. A settle-condition fix closes A and leaves B open. Not fixed.
|
||||
- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
|
||||
(`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
|
||||
`hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
|
||||
therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by
|
||||
not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.
|
||||
|
||||
## What did not run
|
||||
|
||||
**12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place,
|
||||
kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive).
|
||||
Previously reported as 6 of 12 — that was run 1 only.
|
||||
|
||||
**Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not
|
||||
automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed.
|
||||
`kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom
|
||||
covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed.
|
||||
|
||||
**Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34–c39 clean on
|
||||
every invariant. Beyond 39 is untested, not proven clean.
|
||||
|
||||
## Teardown — OWED, nothing removed
|
||||
|
||||
Still intact: the rig is reusable for the atoms that did not run. VM 311, `c10-scratch`, PBS datastore
|
||||
`felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition:
|
||||
DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four
|
||||
orphaned scratch customers left by exactly this omission.
|
||||
@@ -1,43 +0,0 @@
|
||||
# REPORT — CAMPAIGN 7 (felhom.eu side: docs only)
|
||||
|
||||
> Written as `REPORT-campaign7.md`, **not** the shared `REPORT.md`, per the convention this run
|
||||
> added to `CLAUDE.md`: `REPORT.md` is overwritten, so a second concurrent session in this repo
|
||||
> would clobber it. This session's implementation work was in `app-catalog-felhom.eu`; here it only
|
||||
> touched documentation.
|
||||
|
||||
**Run:** 2026-07-18 evening → 2026-07-19 morning. **Class:** campaign (record-and-rank + a defined
|
||||
allowed-fix set). **Implementation repo:** `app-catalog-felhom.eu` (see its `REPORT.md`).
|
||||
|
||||
## What changed in this repo
|
||||
|
||||
| file | change |
|
||||
|---|---|
|
||||
| `documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` | **new** — method, uninstall-semantics map, trio detail, full 53-app matrix, ranked findings, coverage |
|
||||
| `documentation/backlog/ROADMAP.md` | **+3 items** — R-40 (multi-hop major upgrade path), R-41 (no standing catalog deployability check), R-42 (sidecar-major ruling) |
|
||||
| `CLAUDE.md` | REPORT.md parallel-session rule: the second session writes `REPORT-<topic>.md` |
|
||||
|
||||
No hub/agent/scripts/website code was touched (campaign scope: catalog + docs).
|
||||
|
||||
## Headline for this repo's readers
|
||||
|
||||
1. **Uninstall semantics map row PARTIAL → PROVEN** (campaign doc §2), with live evidence from all
|
||||
three trio apps: remove requires stop first; named docker volumes are **always destroyed**
|
||||
(including the app's database); HDD bind-mount data and `backups/primary/<app>` survive unless
|
||||
explicitly ticked; images are kept; `app.yaml` goes, the template stays; the per-app **offsite
|
||||
toggle survives** the uninstall while tier-2 config is cleared. The confirmation modal does warn
|
||||
about the volumes, so there is **no consent gap**.
|
||||
2. **A lying healthcheck takes an app OFF-LINE, it does not merely mislead.** Traefik will not route
|
||||
to an `unhealthy` container, so a probe that cannot execute → permanent unhealthy → **404 to the
|
||||
customer while the app serves 200 on its own port**. 7 of 53 apps were in that state.
|
||||
3. **The pre-flight gate's own signal is missing:** the 0.145.0 → 0.146.0 floor-lift emitted no
|
||||
`controller_updated` event, though the identical bootstrap path emitted one for 0.143.0 → 0.145.0
|
||||
two hours earlier (§0, finding F1). The box did converge — golden, floor and runtime all agreed —
|
||||
but the event trail under-reports version transitions.
|
||||
|
||||
## Open items owned outside this repo
|
||||
|
||||
- **plant-it / wanderer** — images do not resolve at all (neither the new tag nor the one the
|
||||
catalog already ships). Upstream research needed; recorded as findings, not deletions.
|
||||
- **gokapi** — pinned back to v1.9.6; v2 needs the seeded `config.json` regenerated. Security-
|
||||
relevant, should not sit on a superseded line indefinitely.
|
||||
- **glance** — never had a seeded `glance.yml`; proven pre-existing.
|
||||
@@ -1,62 +0,0 @@
|
||||
# REPORT — CAMPAIGN 8: the backup & restore subsystem (2026-07-27/28)
|
||||
|
||||
Adversarial, destructive, unattended run against `demo-felhom`, `demo-hp` and `ep0`.
|
||||
Full report: `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
Evidence: `DooPlex:~/campaign8/evidence/` (103 files, 35 MB, written continuously by 11 collectors).
|
||||
|
||||
**No production code was changed.** Findings are recorded and ranked, never fixed inline, per the
|
||||
campaign's own rules.
|
||||
|
||||
## Scope safety
|
||||
`peti-felhom`, its namespace and `u629488-sub2` were never touched. Phase 0 established with five
|
||||
documented probes that peti has **no data at all** in `felhom-offsite`, which is what made the
|
||||
operator-approved 100% datastore-fill safe. The 13 GB rollback copy `/srv/pbs-felhom` on ep0 is
|
||||
intact.
|
||||
|
||||
## Findings
|
||||
|
||||
| # | Finding | Severity | Class |
|
||||
|---|---|---|---|
|
||||
| F-CRIT-1 | An app that fails to restart after a quiesce **never alarms**, on any channel | **HIGH** | DEFECT |
|
||||
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT |
|
||||
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
|
||||
| F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT |
|
||||
| F-LEAK | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed | MEDIUM | DEFECT (root-caused by fault 18) |
|
||||
| F-REBOOT | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
|
||||
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
|
||||
| F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT |
|
||||
| F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational |
|
||||
|
||||
Both HIGH findings are in the same place: **the system's ability to tell you a backup did not
|
||||
happen.** Both cite the code and the comment that asserts the property the code does not provide.
|
||||
|
||||
## What is now proven that was not before
|
||||
R-88 breaker arming **and its full ladder** (15m/30m/1h/2h/4h/4h-cap) · per-tier isolation under a
|
||||
real one-tier-fails case · `whole_guest_backup_failed` end-to-end with correct tier attribution ·
|
||||
R-97c operator-only routing verified against the hub DB (zero customer rows with `status='sent'`) ·
|
||||
`whole_guest_backup_recovered` + the R-68 pairing gate firing live · **`age_state=absent`** ·
|
||||
R-97b's suppression half · the crash-recovery unquiesce by an actual SIGKILL (1 s) ·
|
||||
**R-87 — the first restic restore round-trip ever performed**, byte-verified (6/7 sha256 identical,
|
||||
the 7th explained) · R-82 one-quiesce-two-tiers · single-flight on two independent paths.
|
||||
|
||||
## Restore round-trips
|
||||
restic (R-87) · local vzdump → fresh CT · PBS offsite → fresh CT · corrupted snapshot → fails
|
||||
cleanly. `mount_parity` exact on both whole-guest tiers, `unprivileged: 1` preserved.
|
||||
|
||||
## Fleet state
|
||||
**Healthy. Nothing left broken.** All four compression knobs reverted and verified; every fault
|
||||
unwound; no leaked scratch guests, nft rules, ballast files or clock skew; ep0 datastore clean with
|
||||
zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
|
||||
|
||||
## Not tested (with reasons)
|
||||
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the
|
||||
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), fault 12 (host
|
||||
reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for
|
||||
documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings.
|
||||
|
||||
**Campaign-caused outage, stated plainly:** fault 11 took demo-hp guest 9201 down for ~9m47s
|
||||
(the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after.
|
||||
|
||||
## Note on repo conventions
|
||||
This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an
|
||||
entry — there is nothing shipped to log. The deliverable is the audit document plus this report.
|
||||
@@ -1,85 +0,0 @@
|
||||
# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28)
|
||||
|
||||
**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign,
|
||||
and findings are recorded, never fixed inline. Full write-up:
|
||||
`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`.
|
||||
Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously).
|
||||
|
||||
Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes.
|
||||
**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact.
|
||||
|
||||
## The headline — two never-proven restore paths are now proven
|
||||
|
||||
Driven through the **real endpoints the UI posts to**, over https through traefik with a real session
|
||||
and CSRF token, on live hardware.
|
||||
|
||||
| proof | result |
|
||||
|---|---|
|
||||
| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
|
||||
| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
|
||||
| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
|
||||
| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
|
||||
| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
|
||||
| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
|
||||
|
||||
The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
|
||||
at their last-backup state.**
|
||||
|
||||
## Findings — 3 defects, ranked (none fixed)
|
||||
|
||||
| # | finding | severity |
|
||||
|---|---|---|
|
||||
| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
|
||||
| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
|
||||
| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
|
||||
|
||||
Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
|
||||
sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
|
||||
artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
|
||||
so live.
|
||||
|
||||
## Mechanisms confirmed working, live
|
||||
|
||||
R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target
|
||||
breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app
|
||||
thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of
|
||||
the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on
|
||||
demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's
|
||||
sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the
|
||||
hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable;
|
||||
F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests).
|
||||
|
||||
## Where it stopped, and what remains
|
||||
|
||||
Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host
|
||||
reboot mid-backup) was deliberately not started — a large new fault class against boxes that are
|
||||
remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown.
|
||||
|
||||
**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's
|
||||
threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time
|
||||
`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign
|
||||
forbids. Reported rather than worked around. The no-code-change alternative (age the controller's
|
||||
reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended
|
||||
method next time.
|
||||
|
||||
**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an
|
||||
intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way
|
||||
concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`.
|
||||
|
||||
## Recovery
|
||||
|
||||
Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive
|
||||
observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active),
|
||||
windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on
|
||||
either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and
|
||||
the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots).
|
||||
|
||||
One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box.
|
||||
Its `until` is long past so it blocks nothing; it clears on the next successful backup or any
|
||||
controller restart (by design, not persisted). Clearing it would have cost another app outage for no
|
||||
benefit.
|
||||
|
||||
**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in
|
||||
`/opt/docker/stacks/<app>` starts a Felhom app **without its secrets** — they are injected by the
|
||||
controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a
|
||||
crash loop during recovery. Manual recovery must go through `POST /api/stacks/<name>/restart`.
|
||||
@@ -1,71 +0,0 @@
|
||||
# REPORT — DIAGNOSE immich offsite restore (felhom.eu side: docs only)
|
||||
|
||||
> Written as `REPORT-diag-immich-restore.md`, **not** the shared `REPORT.md`, per the CLAUDE.md
|
||||
> convention — `REPORT.md` is overwritten and currently holds the 2026-07-18 website refresh.
|
||||
> No code shipped in this run; findings only.
|
||||
|
||||
**Date:** 2026-07-19 · **Box:** demo-felhom (felhom-pve guest 9201), controller 0.146.0, immich v3.0.3
|
||||
|
||||
> **CLOSED IN CODE 2026-07-19 — controller v0.148.0.** Findings 1 and 2 shipped as R-43 (offsite
|
||||
> reconstitution: safety dump → stop → overwrite files → start → replay the snapshot's dump) and
|
||||
> R-44 (every run dumps before it captures; manifests carry `offsite_run_id` + `dumps_at`). Deployed
|
||||
> to demo 9201, healthy. **The §9 live acceptance has NOT run**, so no capability-map flip: the
|
||||
> offsite row is PARTIAL, the customer-restore row stays MISSING, R-3 stays DRAFT. Implementation
|
||||
> detail lives in `felhom-controller/REPORT.md`.
|
||||
|
||||
## What ran
|
||||
|
||||
A diagnosis of "immich offsite restore succeeds but photos do not reappear". No product code was
|
||||
changed: no restore/backup logic, no labels, no flashes, no `restic prune`/`forget`, no snapshot or
|
||||
escrow changes.
|
||||
|
||||
## Outcome
|
||||
|
||||
The restore did not fail. **It was never invoked on missing data, and could not have worked
|
||||
if it had been.**
|
||||
|
||||
1. Viktor deleted the 11 photos in the immich UI to test offsite restore. A UI delete means
|
||||
**trash**, not deletion — no file left the disk. Both „csak a hiányzó fájlok" runs merged
|
||||
**0 files**, correctly, and flashed success. The test proved nothing.
|
||||
2. A *valid* test would also have failed: **no offsite path loads a database.** All three buttons
|
||||
are file-only. Files would return; the library would stay empty.
|
||||
3. The shipped dump is additionally stale by design — from the 02:30 local run, never refreshed
|
||||
before a manual push. Probed: **`asset: 0`, `user: 0`, `album: 0`**.
|
||||
|
||||
**Photos:** left in trash at Viktor's instruction (recovery not wanted). All 11 files verified
|
||||
present on disk and all 11 rows intact, so an ordinary un-trash recovers them until immich's
|
||||
30-day auto-purge.
|
||||
|
||||
**Answer to "can a customer trust same-day offsite?"** For a DB-indexed app: **no — files come
|
||||
back, content does not.** The backup half is honest; the restore half cannot reconstitute the app.
|
||||
|
||||
## Decisive evidence
|
||||
|
||||
- `updatedAt` == `deletedAt` on all 11 asset rows ⇒ **no restore operation touched the DB.**
|
||||
- Unit dump `immich-postgres.sql`, 51 954 452 B, mtime **02:30 CEST** ⇒ `asset: 0 / user: 0 /
|
||||
album: 0`. The 52 MB is entirely immich's shipped geodata reference tables. It predates the admin
|
||||
user (07:56:25) and the photos (07:57).
|
||||
- **Latent hazard:** had a full restore loaded that dump, it would have written an empty DB over the
|
||||
live one, destroying the trashed rows that were the only surviving recovery path.
|
||||
|
||||
## Files written
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `documentation/audits/DIAG-immich-restore-2026-07-19.md` | new — timeline, evidence, source mechanism table, 7 findings |
|
||||
| `documentation/backlog/ROADMAP.md` | **R-43** (P2-HIGH, offsite restore cannot restore a DB) + **R-44** (P2-HIGH, manual push ships unrefreshed dump) |
|
||||
| `documentation/architecture/00-capability-map.md` | customer-restore row **stays MISSING**, gains the finding + a method note for R-3; offsite-restore row flagged *scope contested* |
|
||||
|
||||
## Needs Viktor
|
||||
|
||||
1. **Capability-map ruling (line 61).** The offsite-restore row claims PROVEN-LIVE citing
|
||||
"immich end-to-end from offsite alone" (`CAMPAIGN-6D`). That phrase is contested by this
|
||||
diagnosis. I flagged it rather than downgrading a proven row — did 6D's accept leg exercise the
|
||||
DB half, or only the file half?
|
||||
2. **Optional real red-proof.** Emptying the trash would create genuine data loss and convert the
|
||||
dump-probe inference into a live proof. Offered, **not run** — irreversible, and the probe
|
||||
already settles it.
|
||||
3. **Unreconciled: the 704.6 MiB figure.** Measured 126 MB total on the library storage. If
|
||||
704.6 MiB came off a controller Storage page, that gap is its own defect.
|
||||
4. **Orphaned pre-v3 tree** `dccc13fe…` (~55 MB across upload/thumbs/encoded-video) stranded by the
|
||||
immich 2→3 redeploy — worth a sweep policy for major redeploys.
|
||||
@@ -1,53 +0,0 @@
|
||||
# REPORT — Megosztás diagnosis (2026-07-20)
|
||||
|
||||
Topic-scoped report (parallel-session rule: shared `REPORT.md` untouched).
|
||||
|
||||
**Run:** RUNBOOK "Megosztás diagnosis — SMB unreachable from Mac + sharing-page reload loop".
|
||||
Read-only diagnosis. **No code changes, no version bumps, no builds, no restarts.**
|
||||
|
||||
**Deliverable:** `documentation/audits/DIAG-sharing-2026-07-20.md`.
|
||||
|
||||
## Verdicts
|
||||
|
||||
* **Reload loop — ROOT-CAUSED (HIGH).** `sharingStatusHandler` (`sharing_handlers.go:246`, added in
|
||||
`b5d78d1`, controller v0.147.0, 2026-07-19) coerces `idle` → `running` whenever the samba
|
||||
container is alive. `sharing.html` L320–326 treats `running` as a one-shot job-success edge and
|
||||
calls `location.reload()` 1.2 s later. The first `tick()` fires synchronously on every page load,
|
||||
so the page reloads forever. Unconditional for any customer with sharing enabled — the Megosztás
|
||||
page is currently unusable. Proven live: 6 consecutive `/sharing/status` polls all returned
|
||||
`{"phase":"running","running":true}`, and the controller log shows **no ensure job ran at all**,
|
||||
so the phase is manufactured by that line rather than left over from a stuck job.
|
||||
* **`smb://192.168.0.162` — ROOT-CAUSED.** `.162` is the Proxmox host and never was an SMB endpoint.
|
||||
smbd runs in guest 9201 and binds `192.168.0.104:445`. `nc` from the host: `.104:445` **open**,
|
||||
`.162:445` **refused**. Wrong target, stale Finder favourite.
|
||||
* **`smb://FELHOM` — OPEN, narrowed.** NetBIOS resolution works on the wire
|
||||
(`nmblookup -B 192.168.0.255 FELHOM` → `192.168.0.104 FELHOM<00>`), but the stack advertises **no
|
||||
mDNS/Bonjour** (nothing on udp/5353; the R-6 spike selected `smbd + nmbd + wsdd` only) — the
|
||||
mechanism macOS Finder prefers. Closing this needs one probe from the Mac (`smbutil lookup FELHOM`
|
||||
/ `dns-sd -B _smb._tcp`), listed in the audit's Mac test matrix.
|
||||
* **H2 (container down/crash-looping) — RULED OUT** at the first probe: `felhom-samba` Up 3 h, clean
|
||||
logs, smbd/nmbd/wsdd all bound as `infra/samba.go` intends, live `smb.conf` matches the renderer
|
||||
with no baked address literal, no PVE firewall in the path.
|
||||
|
||||
## Findings (full table + evidence in the audit doc)
|
||||
|
||||
S-1 HIGH reload loop (XS fix: latch a `sawInFlight` flag in the JS; red-proof required) ·
|
||||
S-2 MED the UI never shows the connect address, only `\\FELHOM` — customers guess IPs ·
|
||||
S-3 MED no mDNS advertisement (image slice, needs republish) ·
|
||||
S-4 LOW ensure-job phase never resets — fold into the async-job-feedback roadmap item ·
|
||||
S-5 INFO the guest's LAN IP is DHCP, so any displayed address must be read live, never cached.
|
||||
|
||||
Recommended packaging: S-1 as an immediate patch task (it bricks a shipped page), S-2 alongside it
|
||||
if the guest IP is reachable from the sharing handler, S-3 as its own slice.
|
||||
|
||||
## Also noticed
|
||||
|
||||
The **remote site's LAN is `192.168.0.0/24` — the same prefix as the DooPlex home LAN** that the
|
||||
Tailscale subnet router advertises (`192.168.0.180` shows `FAILED` in felhom-pve's neighbour table).
|
||||
A successful `ping 192.168.0.162` therefore does not by itself prove the Mac is on the remote
|
||||
segment; the Mac matrix starts by confirming which network it is on.
|
||||
|
||||
## Actions taken
|
||||
|
||||
None. Every command was a read, except a `POST /login` to obtain a session for the status polls.
|
||||
No secrets are recorded in either document.
|
||||
@@ -1,85 +0,0 @@
|
||||
# REPORT — R-111 fixed, then E-2 proven on a fresh box (2026-07-29)
|
||||
|
||||
Two phases in one session. Full evidence: `documentation/audits/E2D-fresh-vm-2026-07-29.md`.
|
||||
Root `REPORT.md` untouched.
|
||||
|
||||
## Phase 1 — R-111: the Day-0 channel now serves the current software
|
||||
|
||||
A Phase 0 gate earlier the same day stopped the E-2d run before any VM existed: a fresh box would
|
||||
have installed **agent 0.96.0 + controller 0.161.0**, ~17 and ~24 releases behind `main`.
|
||||
|
||||
| | Before | Now |
|
||||
|---|---|---|
|
||||
| agent (Gitea generic) | 0.96.0 | **0.113.0**, sha `5f3247f7…`, round-trip verified |
|
||||
| golden (Gitea generic) | 0.161.0 | **0.185.1**, sha `dba00f3e…`, embeds controller 0.185.1 |
|
||||
| hub `min_agent` | 0.93.0 | **0.113.0** (what controller v0.185.0 declares) |
|
||||
|
||||
Bake clean on every marker: `Result=success`, overlay2, **all three mounts in the archive**, 0
|
||||
FATAL/exclusions, HTTP 201, token-leak grep 0. GL-1 teardown: guest 9100 purged, secrets shredded,
|
||||
drill disk restored to `virgin`. Agent + golden moved in **one** manifest POST so it never vouched a
|
||||
new agent against an old golden. `min_agent` verified zero-impact first (all three enrolled hosts
|
||||
already at 0.113.0). Global floor deliberately **not** raised — the golden now bakes 0.185.1.
|
||||
|
||||
Commit `3dff357`.
|
||||
|
||||
## Phase 2 — the E-2d run, full ISO/PAIRING route
|
||||
|
||||
Nested PVE VM on demo-hp, one disk, outside the `felhom` pool. Bind → running controller in
|
||||
**3 m 35 s**. The install fetched exactly the artifacts published an hour earlier and restored
|
||||
`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked 20 minutes before. The publish train is
|
||||
proven end to end on a real install.
|
||||
|
||||
| Claim | Verdict |
|
||||
|---|---|
|
||||
| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** |
|
||||
| **C2** Case B fires naturally | ✅ **PROVEN** — both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort |
|
||||
| **C3** degraded banner renders **to a customer** | ⚠️ **PARTIAL** — API byte-exact; **no UI consumer exists** → **R-112** |
|
||||
| **C4** offer appears and moves the target | ⚠️ **PARTIAL** — decline path, `restart_required:true`, no self-restart, E-2a wrapper, healthy-renders-nothing all PROVEN at API level; offer equally invisible → **R-112** |
|
||||
| **C5** `backup_target_absent` end to end | ❌ **FAILED** — zero events on any channel → **R-113** |
|
||||
|
||||
## The three findings
|
||||
|
||||
**R-112 (P1)** — E-2's banner and offer have **no UI consumer**. The endpoint returns byte-exact copy;
|
||||
`grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**, and no page handler injects
|
||||
the state. Decisive contrast: templates fetch **18** distinct `/api/storage/*` endpoints;
|
||||
`backup-target` and `backup-target/assign` are the only two with zero references. v0.185.1 fixed the
|
||||
router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired.
|
||||
|
||||
**R-113 (P1)** — the drive-absent gate **cannot fire on device loss**. `planDriveGates` reads presence
|
||||
from `BoundUnderParent` = "is this path in the guest's mountinfo". The raw mount is a device-bound
|
||||
systemd unit and dies; **the agent's own bind is not device-bound and outlives the device**, so the
|
||||
gate sees "present" forever. Live: agent said `enrolled drive absent by UUID` every 20 s for 4½
|
||||
minutes, controller logged **0** `[gate]` lines, hub got **zero** events — neither the specific nor the
|
||||
generic one. Sixth instance of the class, one layer deeper: E-2b wired the seam to a condition that
|
||||
cannot occur.
|
||||
|
||||
**R-114** — on target-drive loss the message says the backup is *"on the same disk as the system"*
|
||||
(false) and offers **the drive that just vanished**. Invisible today only because of R-112 — so
|
||||
**R-114 must be fixed before R-112 is wired.**
|
||||
|
||||
Also filed as a **second instance under R-110** (not a new ID): host-install fetches **nine** files
|
||||
from `raw/branch/main` and the hub vouches a sha for **one**; E-2a's wrapper is installed 0755 to
|
||||
`/usr/local/sbin`, root-fenced in sudoers, validated only by `bash -n`.
|
||||
|
||||
## Record
|
||||
|
||||
- `OPEN-ITEMS.md` — **R-112/R-113/R-114 opened**; E-2d re-stated with results and left open for the
|
||||
residue; E-2's "NOT yet live-proven" list resolved into proven / known-broken; R-94 fully unblocked;
|
||||
R-110 extended. The drill-cleanup row was opened and then **closed the same session** once the
|
||||
teardown completed, so it is not carried in the register.
|
||||
- `ROADMAP.md` — R-112/R-113/R-114 under P1; R-111 marked SHIPPED.
|
||||
- **`architecture/00-capability-map.md` not touched** — for two reasons: the customer-facing legs are
|
||||
broken rather than proven, and the map has **no E-2 / backup-target rows at all** (worth noting
|
||||
against the ROADMAP's coupling rule).
|
||||
|
||||
## Teardown
|
||||
|
||||
VM destroyed, scratch storage removed, **`pvesm status` after == before** (`local-lvm` 38.77 %,
|
||||
byte-identical), guest 9201 and drill-r50 untouched. **Hub records removed — teardown complete.** The delete was correctly refused at four gates while the host still read ONLINE; once the destroyed host aged to DOWN (`delete-impact` → `deletable:true`) the documented cascade ran and completed: host deleted, PBS tenancy deprovisioned, claim reset, residue purged. Verified after: **0** `e2d` occurrences on the hosts page, fleet unchanged. The one purged `appliance_registrations=1` was this run's own appliance; the unrelated stale 2026-07-25 appliance (`206c8838…`) was not touched by the cascade — the operator removed it separately.
|
||||
|
||||
## One human step, and a premise correction
|
||||
|
||||
The runbook's §5.1a operator STOP (the bind) is **retired** — CC did it. But E-2d's premise that a
|
||||
fresh install yields a CC-drivable claimable customer is **wrong**: the claim code is bcrypt-hashed and
|
||||
email-only, and the gate covers everything except `/claim`, `/api/health`, `/static/`. One operator
|
||||
relay of the emailed code was required — which also proved the claim flow end to end.
|
||||
@@ -1,92 +0,0 @@
|
||||
# REPORT — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run (RUNBOOK execution). **No code changed. No version bump.**
|
||||
Written as `REPORT-<topic>.md` per the parallel-session rule — the shared `REPORT.md` was not touched.
|
||||
|
||||
**Full record with all evidence:** `documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`
|
||||
|
||||
---
|
||||
|
||||
## Outcome: DONE and verified
|
||||
|
||||
`felhom-offsite` now lives on a dedicated 100 GB Hetzner Cloud Volume instead of ep0's 40 GB root disk.
|
||||
|
||||
| | Before | After |
|
||||
|---|---|---|
|
||||
| Path | `/srv/pbs-felhom` (root disk) | **`/mnt/pbs-datastore`** (volume) |
|
||||
| Datastore total | 37.2 GB | **98 GB** (hub gauge: 97.9 GB) |
|
||||
| Used | 28.9 % | **13 %** (hub gauge: 12.6 GB, 13 %) |
|
||||
| Headroom to the 80 % warn | 19 GB | **≈65 GB** |
|
||||
| Additional customers before warn | ≈2 | **≈7–13** |
|
||||
|
||||
Datastore **name unchanged** — the PBS-DR descriptors, per-box storage ids, ACLs and namespace
|
||||
layout that R-39/R-82 made self-healing are untouched.
|
||||
|
||||
**Window:** 06:58 → 07:19 UTC (PBS down 07:00 → 07:17). **Nothing was deleted.**
|
||||
|
||||
### Acceptance evidence
|
||||
|
||||
| Gate | Result |
|
||||
|---|---|
|
||||
| Copy integrity | 13,242,207,822 = 13,242,207,822 B · **9,748 = 9,748 chunks** · 7 = 7 snapshots · `backup:backup` · itemised dry-run **0 lines** |
|
||||
| Snapshot counts per ns | `demo-felhom` 2=2, `demo-felhom-01` 3=3, `demo-hp` 2=2 |
|
||||
| atime semantics | `rw,relatime,discard` — **`relatime` present, `noatime` absent** (GC correctness) |
|
||||
| Verify job | `TASK OK`, 3/3 groups, forced re-verification of every snapshot, **0 errors** |
|
||||
| §6 mount guard | **refusal observed** — `Job … failed with result 'dependency'`; mountpoint stayed empty |
|
||||
| §8 restore round-trip | `source_tier: pbs`, `pass: true`, `mount_parity: ok`, clean teardown, 12m1s |
|
||||
|
||||
---
|
||||
|
||||
## Three findings the operator should act on
|
||||
|
||||
1. **`scratch` datastore is configured at a path that does not exist** (`/srv/pbs-scratch`).
|
||||
Pre-existing, not caused here, but now logs `ENOENT` on every PBS start. This is the PRIME RISK
|
||||
shape ("reports fine, is not there") already live in the config. **Decision needed:** remove the
|
||||
stanza or create the directory.
|
||||
|
||||
2. **The runbook's §6 acceptance test proves the wrong proposition.** `RequiresMountsFor` is a
|
||||
mount-first ordering guarantee, not a refusal — systemd silently *re-mounts* an unmounted volume
|
||||
and PBS then starts safely. The test only bites when the device is genuinely unavailable, which
|
||||
is how it was re-run and passed. **Amendment recommended in the runbook record.**
|
||||
|
||||
3. **§11 — storage box `u629193` is NOT simply unused.** No live backup path references it (no
|
||||
datastore, no restic repo, no fstab, no `known_hosts` pin; R-17 already deleted `u629193-sub1`),
|
||||
**but ep0 carries an enabled, currently-mounted sshfs unit** `mnt-pbs\x2dstoragebox.mount` →
|
||||
`/mnt/pbs-storagebox`, holding spike leftovers. Disable and remove that unit before deleting the
|
||||
box, or ep0 logs a failed mount every boot. **The deletion is the operator's console click.**
|
||||
|
||||
## Deviations from the runbook as written
|
||||
|
||||
- **The volume arrived already formatted and mounted** by Hetzner at `/mnt/HC_Volume_106469259`
|
||||
(§2 assumed neither). Operator ruled: reformat + repath. The 5 % reserve was reclaimed (`-m 0`).
|
||||
- **§8 ran on demo-felhom, not demo-hp** — DooPlex holds no SSH key for demo-hp (the G1 gap). Same
|
||||
tier, same relocated datastore, larger archive.
|
||||
- **The window was contended** by a stale 10-minute restore-test cadence on demo-felhom: the config
|
||||
had already been reverted to 3.5 days on disk, but the cadence is read once at daemon start and
|
||||
`NRestarts=0`. Restarting the agent applied it (`cadence=84h0m0s`). The in-flight test was allowed
|
||||
to finish rather than aborted.
|
||||
|
||||
## Process errors made during this run (recorded deliberately)
|
||||
|
||||
- `rsync -aHAX` **OOM-killed** ep0 (3.7 GB RAM, no swap). Cause: a PBS `.chunks/` tree pre-creates
|
||||
all 65536 shard dirs → 75,341 inodes, and `-H` retains the whole inode map. `-H` was dropped only
|
||||
after **proving** no hardlinks exist (`-links +1` → 0; max link count → 1); PBS references chunks
|
||||
by digest, never by hardlink.
|
||||
- `/usr/bin/time -v` is not installed on ep0 → exit 127, rsync never ran, and a `| grep … || true`
|
||||
wrapper swallowed it while printing a success-looking line.
|
||||
- `rsync --version | head -1` reported a working rsync 3.4.1 as "missing" — **the §12 pipe-into-head
|
||||
trap, fourth recorded instance in this project.** Both fixed by capturing the command's own `$?`.
|
||||
|
||||
## Deferred
|
||||
|
||||
1. **Old copy retained** at `/srv/pbs-felhom` (13 GB, 9,748 chunks) as the rollback. Rollback is a
|
||||
two-line `datastore.cfg` revert. Reclaim only after a new weekly offsite backup lands on the
|
||||
volume, with explicit go-ahead.
|
||||
2. **GC not run** — now unblocked by the round-trip, but left for a separate deliberate run. No GC
|
||||
schedule is configured on this PBS at all.
|
||||
3. ~~Hub PBS-DR capacity gauge not re-read.~~ **CLOSED — verified correct.** The hub operator UI
|
||||
(Offsite → PBS DR) reports `felhom-offsite (ep0)` at **97.9 GB capacity, 12.6 GB used, 13 % full**,
|
||||
agreeing with the on-box `df`. The gauge follows the datastore's configured path, so the move
|
||||
needed no hub-side change and the suspected "wrong filesystem" bug does not exist.
|
||||
4. **ep0 has no swap** (temporary 4 GB file removed; box left as found). Worth a small permanent
|
||||
swapfile — outside this runbook's scope.
|
||||
@@ -1,63 +0,0 @@
|
||||
# REPORT — F-CRIT-1 + F-A1 fixed (controller v0.179.0, 2026-07-28)
|
||||
|
||||
Docs here. Implementation, all six red-proofs and the full live replay live in
|
||||
`felhom-controller/REPORT.md`. The campaign that found both:
|
||||
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
|
||||
## What changed
|
||||
**F-CRIT-1** — an app that failed to restart after a quiesce never alarmed, for two independent
|
||||
reasons, either of which alone kept it dead: `restartAll` returned nothing (the failure was logged
|
||||
and dropped), and `classifyRunStates` whitelisted `StateStopped` on invariant I1 ("the user stopped
|
||||
it") — which the quiesce loop had made false by stopping stacks the same `docker compose down` way.
|
||||
A failed restart and a user stop are the *same* Docker state; the only difference is that the loop
|
||||
tried and could not, now surfaced by `Loop.FailedRestarts()`.
|
||||
|
||||
**F-A1** — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a
|
||||
failure. It is now contention: the tier stays DUE, is dropped before anything stops, and unending
|
||||
contention raises its own **BLOCKED** signal rather than going silent.
|
||||
|
||||
## Bounds, justified against measured reality
|
||||
- `contentionRetryAfter` **15m** — longest restore-test observed on the fleet is 12m01s; the agent's
|
||||
local restore-test wait is 10m. Caps app-stop churn at 4/hour instead of 12/hour.
|
||||
- `contentionAlarmAfter` **3h** — the agent's own PBS restore-test task is capped at 120 minutes, so
|
||||
contention outliving that is a *stuck* gate, not a busy one. 3h adds margin and is 15× the longest
|
||||
contention actually observed.
|
||||
|
||||
## Verified live, with the hub DB as arbiter — not from logs
|
||||
Same box, same day, same event type; the only difference is 409 versus a genuine error:
|
||||
|
||||
| injection | operator emails (demo-hp) |
|
||||
|---|---|
|
||||
| **409 contention** | 8 → **8** (none) |
|
||||
| **real transport failure** | 8 → **9** |
|
||||
|
||||
And for F-CRIT-1: the failed restart alarmed **9 seconds** after grace expiry with the dashboard
|
||||
banner naming the `(stopped)` state, while a **deliberate** user stop on the same box stayed silent
|
||||
through **9** dead-app scans (the positive observable that the silence is suppression, not a dead
|
||||
detector).
|
||||
|
||||
## The rule this arc earned
|
||||
Added to **both** copies of `CLAUDE.md` (live + `documentation/runbooks/workspace-CLAUDE.md`):
|
||||
**a comment asserting an invariant needs a test pinning it, or it is a wish.** Six instances in this
|
||||
project have shipped guarantees the code did not provide — `EffectiveProtected`, `newestArchiveOn`,
|
||||
the R-97a operator-only claim, `classifyRunStates`' I1, `inflight.go`'s defer claim, and
|
||||
`quiesce.go`'s spurious-failure claim. Two were found only on live hardware, and one of those had a
|
||||
green, red-proofed test suite over a production path broken two independent ways.
|
||||
|
||||
Corollary recorded with it: prefer a test that asserts the **consequence** (does the alarm fire?)
|
||||
over one that asserts the **mechanism** (does suppression expire?). R-97b's Scenario F proved the
|
||||
mechanism; the consequence was still broken.
|
||||
|
||||
## Docs touched
|
||||
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-1 and F-A1 → SHIPPED + PROVEN-LIVE.
|
||||
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — both rows struck through, closing
|
||||
section added. **All three of the campaign's alarm findings are now closed** (F-CRIT-1, F-CRIT-2,
|
||||
F-A1).
|
||||
- `documentation/runbooks/workspace-CLAUDE.md` — the invariant rule.
|
||||
|
||||
## Still open, highest first
|
||||
**Fault 4** (restic transport interruption) — four injection approaches were defeated by
|
||||
guest-bridged networking, and it is now the most valuable follow-up: F-CRIT-2 answered the phantom
|
||||
question for PBS and left the identical question open for restic. Then **R-99** (prune never removes
|
||||
phantoms) and **F-LEAK** (a failed restore-test cannot destroy its own scratch guest — observed
|
||||
again during this work).
|
||||
@@ -1,40 +0,0 @@
|
||||
# REPORT — F-CRIT-2 fixed: a failed backup no longer looks like a fresh one (2026-07-28)
|
||||
|
||||
Scope: `felhom-agent` v0.105.0 → **v0.106.0**. Docs here. Implementation detail and the full live
|
||||
re-test live in `felhom-agent/REPORT.md`; the campaign that found it is
|
||||
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
|
||||
## What changed
|
||||
`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, and NEWEST) as a
|
||||
successful backup, so the tier read fresh, went **not due**, and was never retried — seven days of
|
||||
silence on the real 168h cadence, invisible to both the R-88 breaker (defers only *due* tiers) and
|
||||
the hub deadline monitor (reads the same freshness). It now counts only *plausibly complete*
|
||||
entries via a measured 1 MiB floor; undecidable ⇒ not counted.
|
||||
|
||||
**Size is the only tier-agnostic discriminator.** `verification` and `encrypted` are absent on every
|
||||
local (dir) archive AND on a good PBS snapshot until `verify-new` catches up — gating on either
|
||||
would have rejected 100% of local backups and produced fleet-wide backup thrash. That inverse risk
|
||||
is a first-class test, red-proofed by making the filter reject everything.
|
||||
|
||||
## Verified live, not just in unit tests
|
||||
Campaign fault 2 was replayed against the fixed agent on demo-hp — phantom created, rejected and
|
||||
announced once; the tier correctly reported DUE and backed up (4,359,968,099 B landed); and the
|
||||
inverse showed **no thrash**, with 91 scheduler ticks as the positive observable that the loop was
|
||||
alive rather than dead.
|
||||
|
||||
## Settled along the way — no retention bug
|
||||
Server-side prune does **not** count phantoms toward `keep-last`: a dry-run against three real
|
||||
snapshots plus a phantom retained two real ones plus the phantom. The feared "two phantoms ⇒ zero
|
||||
real backups" does not occur. Prune never removes them either, so they accumulate one per aborted
|
||||
upload — filed as **R-99** (LOW, hygiene), not as a retention bug.
|
||||
|
||||
## Docs touched
|
||||
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-2 → SHIPPED+PROVEN-LIVE; **R-99** filed;
|
||||
**F-CRIT-1** filed as READY-HIGHEST (Campaign 8's other HIGH finding, untouched here).
|
||||
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — F-CRIT-2 row struck through and
|
||||
a closing section added.
|
||||
|
||||
## Still open, highest first
|
||||
**F-CRIT-1** — an app that fails to restart after a quiesce never alarms, for two independent
|
||||
reasons. Then fault 4 (restic transport interruption), which this fix makes more pointed: the
|
||||
phantom question is now answered for PBS and still open for restic.
|
||||
@@ -1,140 +0,0 @@
|
||||
# REPORT — F-REBOOT + F-LEAK + F-OBS, and two investigations (2026-07-28)
|
||||
|
||||
Scope in this repo: **`scripts/felhom-host-install.sh` v1.20.0 → v1.21.0** (which is where F-LEAK's
|
||||
*actual* fix lives), plus the Campaign 8 audit doc and `OPEN-ITEMS.md`. Written as
|
||||
`REPORT-freboot-fleak-fobs.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
Code companions: `felhom-agent` v0.106.0 → **v0.110.0**, `felhom-controller` v0.179.0 → **v0.180.0**.
|
||||
|
||||
**Correction to this repo's part of the story:** v1.21.0's band-scoped ACL is *not* the final F-LEAK
|
||||
fix. It works, but only **once per slot** — PVE's destroy path calls
|
||||
`AccessControl::remove_vm_access($vmid)` (`API2/LXC.pm:906`), which deletes every ACL at `/vms/<vmid>`
|
||||
(`AccessControl.pm:1898`), so **the grant is consumed by the operation it authorises**. Found by counting
|
||||
ACL rows after the first successful teardown (`/vms/990000` → 0 grants), not by reasoning about it. The
|
||||
durable fix is agent **v0.110.0**'s band-scoped fenced destroy; v1.21.0 remains valuable because it makes
|
||||
the common case need no privileged call, and it is now the *first* of two layers rather than the only one.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom.eu d0cec9d`, `felhom-agent af1c21a`, `felhom-controller fb91c8d`, all clean. Agent `0.106.0`
|
||||
and controller `0.179.0` live on both demo boxes.
|
||||
|
||||
---
|
||||
|
||||
## host-install v1.21.0 — F-LEAK, and why the fix landed *here* rather than in the agent
|
||||
|
||||
**The finding.** A restore-test whose restore **fails** leaves a scratch guest the agent cannot destroy
|
||||
(`403 missing privilege VM.Allocate`), so a half-restored guest holds its disks until a human removes
|
||||
it and the 10-slot scratch band shrinks silently.
|
||||
|
||||
**The cause is structural, not a missing privilege in the role.** `FelhomAgentGuest` is granted at
|
||||
`/pool/felhom`, and **a guest joins that pool only when its restore completes**. A failed restore
|
||||
therefore produces a guest that exists, is in no pool, and is out of the token's reach entirely.
|
||||
|
||||
**The first fix was wrong, and its own live replay is what proved it.** Agent v0.107.0 shipped a
|
||||
teardown fallback that adopted the stranded guest into the pool and retried — reasoning from
|
||||
`Pool.Allocate` on `/pool/felhom`. It fired exactly as designed and PVE refused it:
|
||||
|
||||
```
|
||||
ERROR restore-test: pool adoption failed; left for Recover vmid=990000
|
||||
err="proxmox: PUT /pools/felhom -> HTTP 500: permission denied at /vms/990000 (missing privilege ...)"
|
||||
```
|
||||
|
||||
`PUT /pools/{pool}` **also** requires `VM.Allocate` on the VM being added. **Pool membership cannot
|
||||
bootstrap its own authority.** Removed in agent v0.108.0 rather than left in place — a path that
|
||||
provably cannot work is worse than none, because it reads as a fix.
|
||||
|
||||
**What shipped instead.** `apply_scoped_acl` now grants `FelhomAgentGuest` at each `/vms/<id>` in
|
||||
`PVE_SCRATCH_VMID_MIN..PVE_SCRATCH_VMID_MAX` (990000–990009 — the band the restore-test already picks
|
||||
from), to **both** the user and the token, because the privsep-intersection rule applies here as
|
||||
everywhere.
|
||||
|
||||
Two supporting changes, both load-bearing rather than tidy-up:
|
||||
- **`remove_scoped_acl` deletes the band grants before the role delete.** PVE refuses to delete a role
|
||||
still referenced by any ACL, so omitting this would have broken the uninstall — a failure that would
|
||||
only surface on a decommission.
|
||||
- **`step_verify` asserts the band grants.** A missing one is otherwise invisible until a restore-test
|
||||
*fails*, which is precisely the case that leaked a guest in the first place.
|
||||
|
||||
### Why the grant is still not a widening — proven live, at the seam the defect lives in
|
||||
|
||||
A real PBS restore to `990000` **without `--pool`** reproduced the exact stranded state
|
||||
(`990000 stopped`; `felhom pool members: [9201]`; `990000 in pool: False`). Then, with the agent's own
|
||||
token, same guest, minutes apart:
|
||||
|
||||
| | `DELETE /nodes/<node>/lxc/990000` |
|
||||
|---|---|
|
||||
| **grant removed** (the original defect) | `403 Permission check failed (/vms/990000, VM.Allocate)` — guest still present |
|
||||
| **grant restored** (the fix) | `200 UPID:...:vzdestroy:990000:felhom-agent@pve!agent` — guest gone |
|
||||
|
||||
And it still cannot reach anything else:
|
||||
|
||||
| target | result |
|
||||
|---|---|
|
||||
| `/vms/990010` (one past the band) | **403** `Permission check failed (/vms/990010, VM.Allocate)` |
|
||||
| `/vms/100` | **403** same |
|
||||
|
||||
`990010` does not exist and PVE **still** answered 403 rather than "does not exist" — so PVE evaluates
|
||||
**permission before existence**, which makes these genuine authorization refusals rather than artifacts
|
||||
of a missing guest. Granting at `/vms` was considered and rejected: it would authorise destroying every
|
||||
guest on the box, including a co-tenant's.
|
||||
|
||||
Applied on **both** demo boxes (demo-hp and demo-felhom) so the live fleet matches the installer.
|
||||
|
||||
**A careless step of mine, recorded rather than buried.** The probe loop also issued a live `DELETE`
|
||||
against running guest **9201**. It was refused with `500 container is running` — but the *permission
|
||||
check passed* (9201 is a pool member by design), so had the guest been stopped I would have destroyed
|
||||
the live demo guest. The scratch-band probes were the safe ones; 9201 had no business in that list.
|
||||
|
||||
---
|
||||
|
||||
## Documentation changes
|
||||
|
||||
- **`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`** — F-REBOOT, F-LEAK and F-OBS
|
||||
written up as FIXED with their live evidence, including F-LEAK's refuted first attempt (recorded
|
||||
precisely *because* it looked right), plus a new **§6b** for the follow-up investigation.
|
||||
- **`documentation/backlog/OPEN-ITEMS.md`** — three findings closed, **R-100** filed.
|
||||
|
||||
---
|
||||
|
||||
## R-100 — the investigation's finding, deliberately NOT fixed
|
||||
|
||||
**A restic offsite tier that fails every night never goes stale on the hub.** This is **F-CRIT-2's
|
||||
defect class one layer up and on the other tier** — a *failed* run resetting the freshness clock — and
|
||||
it was found by asking whether F-CRIT-2's shape existed anywhere else.
|
||||
|
||||
Both halves verified in the source, not inferred:
|
||||
- **Controller:** `o.LastRun = time.Now()` is set **unconditionally** at
|
||||
`controller/internal/backup/offbox.go:716`, *outside* the `runErr` branch. The failure is recorded
|
||||
faithfully, but into a different field — `o.LastStatus = "error"` at `:725`.
|
||||
- **Hub:** `isStale()` reads **only** `off.LastRun` (`hub/internal/monitor/offsite.go:120`, `:127`,
|
||||
`:131`) and never consults `LastStatus`.
|
||||
|
||||
So a nightly restic run that fails every night keeps `LastRun` fresh, `isStale` is permanently false,
|
||||
and the staleness alarm never fires — with no successful offsite backup having occurred at all.
|
||||
|
||||
**Scope of the silence, stated precisely rather than dramatically.** `LastStatus` *does* reach the hub —
|
||||
it is parsed into the report struct and **only logged** (`offsite.go:270`); it drives no checker and no
|
||||
notification. The controller's own guest UI surfaces `LastStatus="error"`, so the failure is visible to
|
||||
someone who looks. What is missing is the **push**: the operator's fleet-wide alarm plane is silent,
|
||||
which is the plane that matters for an unattended appliance.
|
||||
|
||||
Not fixed, per this task's investigation-only scope. Fix direction: gate staleness on the last
|
||||
*successful* run rather than the last attempt — exactly what F-CRIT-2's `NewestArchiveTime` fix did for
|
||||
the PBS tier.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
`bash -n scripts/felhom-host-install.sh` clean. The ACL behaviour was verified **live on real hardware**
|
||||
rather than by dry-run, since the whole finding is about what PVE's authorizer actually does — and the
|
||||
live run is what refuted my first design.
|
||||
|
||||
## Fleet state
|
||||
Agent **0.110.0** (with the updated sudoers) and controller **0.180.0** on both demo boxes, all healthy.
|
||||
Scratch-band ACLs at 20 rows on both — re-applied after the attempt-2 destroy consumed one. No leftover scratch guests. demo-hp's `restore_test_cadence_seconds` reverted **600 → 302400**
|
||||
(a bounded change made for the replay).
|
||||
|
||||
`felhom.eu`: this repo has a **foreign uncommitted WIP file** (`documentation/PROMPT-TEMPLATE.md`) from
|
||||
another session in the shared worktree. Left untouched; my commits staged explicit paths only, per the
|
||||
never-`git add -A` rule.
|
||||
@@ -1,119 +0,0 @@
|
||||
# REPORT — ISO boot branding + single-entry GRUB menu (R-38) · website grid restored (2026-07-19)
|
||||
|
||||
> `REPORT-<topic>.md` per this repo's parallel-session rule: another session was writing in this
|
||||
> clone tonight (CAMPAIGN 7 / `DIAG-immich-restore-2026-07-19.md`), so the shared `REPORT.md` is left
|
||||
> untouched.
|
||||
|
||||
Parts 1 and 2 of the polish train. Parts 3 and 4 landed in `felhom-agent` and `felhom-controller`;
|
||||
see their own `REPORT.md`.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — scripts v1.22.0: GRUB branding + single-entry safety (R-38)
|
||||
|
||||
### What shipped
|
||||
|
||||
Every ISO is now **repacked** after `prepare-iso`. `mkimage-surgery.sh` → **`iso-repack.sh`**:
|
||||
branding and the slice-B loader swap need the same extract → modify → re-master cycle, so they share
|
||||
one pass instead of re-mastering twice. **The mkimage recipe is untouched.**
|
||||
|
||||
**The safety half — the one that matters.** The stock PVE menu offers *Graphical*, *Terminal UI*, a
|
||||
serial variant, and an **Advanced Options** submenu holding two `nomodeset` entries, three debug
|
||||
entries, *Rescue Boot*, memtest and *UEFI Firmware Settings*. Every one of those reaches the
|
||||
**manual** installer, whose first question is which disk to wipe. They are **not emitted** — not
|
||||
hidden, not password-gated. What ships is one entry, „Felhom telepítés", default, 5 s.
|
||||
|
||||
**Boot behavior is unchanged.** The `linux`/`initrd` lines are lifted **verbatim at repack time**
|
||||
from the ISO's own *Install Proxmox VE (Automated)* entry rather than frozen into a copy in this
|
||||
repo, so a PVE bump that moves the kernel path or edits the append line tracks automatically. The
|
||||
build **fails** if they cannot be found, if the append line has lost `proxmox-start-auto-installer`,
|
||||
or if `auto-installer-mode.toml` is absent — that last one because without it the single
|
||||
Felhom-labelled entry would boot a *manual* installer, i.e. exactly what this change prevents.
|
||||
|
||||
**Gates, then a re-check against the shipped artifact.** The rendered menu is asserted to have
|
||||
exactly 1 `menuentry`, 0 `submenu`s and no *live* reference to
|
||||
`proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` (comments are stripped first —
|
||||
the template's header names the dropped entries deliberately). Then the menu and theme background
|
||||
are read back **out of `final.iso`**, not out of the extract tree.
|
||||
|
||||
**The boot card.** `grub/generate-grub-background.sh` letterboxes `website/assets/og-image_2.png`
|
||||
onto a 1024×768 gfxterm canvas at repack time (ImageMagick added to the assistant image), so the boot
|
||||
screen has **one source** and not a second pre-rendered PNG to drift. The card's own subtle grid
|
||||
(measured: 4px lines of `#0D131A` on `#0D1117`, pitch 131px) is continued across the letterbox fill
|
||||
**phase-locked** to where the card's grid lands, so the fill is seamless instead of a 500px square of
|
||||
grid floating in flat navy. The generator refuses a source whose geometry no longer matches the
|
||||
measured constants — a swapped asset would misplace every line, and that only shows up on a boot
|
||||
screen nobody re-checks. Menu positioning needs a gfxmenu theme (plain `background_image` cannot move
|
||||
the menu off the wordmark), so `grub/felhom-theme.txt` puts it in the lower third the layout leaves
|
||||
empty, optically centered (measured off a canary screenshot; the comment records the measurement).
|
||||
|
||||
### Live validation — nested canary, UEFI/OVMF, PVE 9.2-1
|
||||
|
||||
Booted the built canary ISO under QEMU with OVMF and captured the framebuffer.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| GRUB renders the Felhom card | **PASS** — background + grid visible at 1024×768 |
|
||||
| Exactly one entry, selected | **PASS** — „Felhom telepítés" only |
|
||||
| Hungarian accents under gfxterm | **PASS** — „telepítés", „Indítás … másodperc múlva" render correctly |
|
||||
| Countdown visible and counting | **PASS** — 5 → 0 |
|
||||
| Auto-fires at 0 | **PASS** — serial shows ``Booting `Felhom telepítés'`` |
|
||||
| Unattended install proceeds | **PASS** — „Fetching answers for automatic installation" → auto installer |
|
||||
| Same abort as v1.21.0 | **PASS** — `ERROR: Installation failed: filter did not match any device` → `Installation aborted`; no disk touched |
|
||||
|
||||
A first build **correctly failed closed**: the banned-entry gate matched the template's own
|
||||
explanatory header. Fixed to strip comments before matching (a comment naming a removed entry is the
|
||||
point; a directive using one is the bug), which is a gate behaving as designed.
|
||||
|
||||
### Artifacts (rebuilt on 180, `/mnt/5_hdd/felhom.eu/felhom-iso/out/`)
|
||||
|
||||
| ISO | sha256 | bytes |
|
||||
|---|---|---|
|
||||
| `felhom-pve-9.2-1-v1.22.0-n100-generic-mkimage.iso` (safety) | `ff6f06ba1dbfe10f27d703afc29516001000349147426b43c9a424a0ea28bdbf` | 1 704 482 816 |
|
||||
| `felhom-pve-9.2-1-v1.22.0-n100-demo-generic-mkimage.iso` (real) | `494db0ddf859b6b152cad4d0e0d9e9cefd27255cde07e2b41aba3ac12a217888` | 1 704 482 816 |
|
||||
| `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso` (validation) | `83c61c0413c84e27b26a37bb5dfaed2fcd44fd25e3e571c7310142bd305f2f9d` | 1 705 338 880 |
|
||||
|
||||
Both shipping ISOs: `embedding 60 modules`, `El Torito boot images=2`, fs-uuid preserved, and the
|
||||
post-re-master verification confirming 1 entry + theme background inside the finished image.
|
||||
|
||||
**Deliberately not done** (per the task): no squashfs/initrd rebranding — post-GRUB screens are still
|
||||
Proxmox-branded; no disk-setup or answer-generation change; the pairing banner is untouched.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — website: the index grid background, restored
|
||||
|
||||
**Archaeology.** Not a deliberate removal. The grid lived as a fixed `body::before` in
|
||||
`index.html`'s inline `<style>` block and was dropped in **`bed8675`** ("D3 Part 2: index + kapcsolat
|
||||
on design system v2"), the commit that migrated the page onto the shared `assets/site.css`.
|
||||
`dd54e4c`, which *created* `site.css`, has no `body::before` at all — it was a porting omission and
|
||||
nothing took its place. `ccbb13a` (the other five pages) never had it. No asset was lost: the
|
||||
mechanism was pure CSS (two stacked `linear-gradient`s), which is why nothing looked missing in the
|
||||
worktree.
|
||||
|
||||
**Restoration, not redesign.** Same 50px cells, same 1px lines, same 3% opacity, same
|
||||
`position:fixed` / `z-index:-1`. One deliberate difference: the accent is the v2 `--blue` `#0083D8`
|
||||
instead of the retired legacy `#0088cc`, which `site_gates.py` bans. Scoped to `body.page-index`,
|
||||
because index is the only page that ever had it. `site.css` cache-bust bumped `?v=1` → `?v=2` across
|
||||
all seven pages (nginx caches 7 days); BOM preserved on every file.
|
||||
|
||||
**Live verification** (felhom.eu, after git-sync deploy):
|
||||
|
||||
- Desktop: grid renders behind the hero, at its original subtlety.
|
||||
- **376px viewport** (via a same-origin iframe — the browser window would not resize in this
|
||||
environment, so the narrow case was exercised for real rather than asserted): grid renders, mobile
|
||||
layout unchanged, `scrollWidth === clientWidth` so **no horizontal overflow**.
|
||||
- Computed style confirmed live: `linear-gradient(rgba(0,131,216,0.03) 1px, …)`, `50px 50px`,
|
||||
`position: fixed`, `z-index: -1`, `pointer-events: none`.
|
||||
- `python scripts/site_gates.py` — **OK** (BOM, no legacy tokens, no `<style>` blocks, cache-busted).
|
||||
|
||||
---
|
||||
|
||||
## Docs
|
||||
|
||||
- `scripts/CHANGELOG.md` — v1.22.0 entry.
|
||||
- `website/CHANGELOG.md` — grid restoration entry.
|
||||
- `documentation/backlog/ROADMAP.md` — **R-38 flipped to SHIPPED**; **R-45** (unified async-job
|
||||
feedback) and **R-46** (verification-copy browse + expiry) added; pre-invite checklist gained the
|
||||
"golden ≥ 0.147.x carries all four infra images" line.
|
||||
- Capability map: **untouched** — no capability moved. These are UX and packaging.
|
||||
@@ -1,307 +0,0 @@
|
||||
# REPORT — the universal ISO: **PUBLISHED** (2026-07-31)
|
||||
|
||||
**Live:** `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso`
|
||||
**sha256:** `f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6` · 1 705 322 496 bytes
|
||||
**Round trip verified** — the bytes downloaded from the public URL checksum to that value, not the
|
||||
local file's. `.sha256` and manifest published beside it.
|
||||
|
||||
> Written as `REPORT-iso-release.md`, not root `REPORT.md`, per the task and the shared-clone rule.
|
||||
|
||||
## 0. Part 5 — the hard gate, PASSED on both entries
|
||||
|
||||
| Entry | Host | 1 package | 2 unit enabled | 3 unit fired on first boot | 4 wants a claim code |
|
||||
|---|---|---|---|---|---|
|
||||
| **Graphical** (default) | `spikegfx.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | `activating`; *"registering unclaimed appliance at the hub"* | **`J7N-2DA`**, token 64 B mode 600 |
|
||||
| **Terminal UI** | `spikesix.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | same | **`ZY5-YY4`**, token 64 B mode 600 |
|
||||
|
||||
Both: normal manual install, own disk chosen in the installer, own root password, real completion
|
||||
signal (installer wrote ~7 GB and rebooted; the installed system was then reached over SSH). Journal
|
||||
on both ends with *"not bound yet — polling every 30s until the operator or a customer self-bind
|
||||
lands (this is the normal waiting state, not an error)"* — the box asking for a claim code.
|
||||
|
||||
Spike 4 reasoned the graphical path would follow from shared `Install.pm`. **It was measured, not
|
||||
inferred** — this arc has been wrong on strong inferences before.
|
||||
|
||||
## 1. Venue and baselines
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Host** | `demo-hp` (t740), Tier 0 |
|
||||
| **VMs** | **500 `spike5-gfx`**, **501 `spike5-tui`** — both created with `qm` so the run is visible in the web console |
|
||||
| **Storage** | **`spike5`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images`. Root chosen deliberately: a storage at a *subdirectory* reads `disconnected` forever via the agent's `exactMount` check. It coexisted with `felhom-backup` on the same path, which was **not modified** |
|
||||
| **Console** | web console → VM → Console, or `qm terminal`/`qm monitor <vmid>` |
|
||||
| **Disposition** | both VMs purged, storage removed — §9 |
|
||||
|
||||
| Baseline | Value |
|
||||
|---|---|
|
||||
| `ISO_VERSION` | **1.26.0** (was 1.25.0) — `scripts/iso/build-felhom-iso.sh:51` |
|
||||
| `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` |
|
||||
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
|
||||
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
|
||||
| controller on `main` | `0.188.0` (`4115e88`) |
|
||||
| `felhom.eu` HEAD at build | clean, pushed, `== origin/main` |
|
||||
|
||||
## 2. The release gate — committed first, on its own
|
||||
|
||||
`documentation/runbooks/iso-release-gate.md`, commit **`e787391`**, written and pushed **before the
|
||||
first build** so it could not be rationalised afterwards. Twelve criteria, each checkable against the
|
||||
uploaded file rather than the build inputs, each carrying the spike measurement that justifies it.
|
||||
|
||||
**One criterion was amended before the build, with its reasoning recorded in the runbook.** G6 was
|
||||
first written with the six-token ban `iso-repack.sh:160-164` enforces, on the rationale *"no live route
|
||||
to a manual disk-picker"*. That rationale is obsolete for a public image — the ruling makes the manual
|
||||
installer **the product**. `proxtui` (the Terminal-UI installer we deliberately ship) and `nomodeset`
|
||||
(its graphics fallback) are dropped **for release images only**; `proxdebug`, `Rescue Boot`, `memtest`
|
||||
and `fwsetup` stay banned in both modes, and the six-token list is **unchanged** for appliance images.
|
||||
|
||||
## 3. The stub package
|
||||
|
||||
`scripts/iso/pkg/` — source committed, built by `build-deb.sh`.
|
||||
|
||||
**Contents: exactly two files, deliberately not three.**
|
||||
```
|
||||
-rwxr-xr-x ./usr/local/sbin/felhom-bootstrap.sh
|
||||
-rw-r--r-- ./lib/systemd/system/felhom-bootstrap.service
|
||||
```
|
||||
The old first-boot stub also wrote `/etc/felhom/bootstrap.env` (0600). This package does not:
|
||||
`felhom-bootstrap.sh:91` reads it only `if [[ -r ]]`, and its defaults at `:95-96`
|
||||
(`https://hub.felhom.eu`, `https://felhom.eu/scripts/felhom-host-install.sh`) are **exactly** what the
|
||||
generic pairing env set (`build-felhom-iso.sh:257-258`). Shipping it would add a 0600 file to a public
|
||||
package to express values the script already defaults to.
|
||||
|
||||
**Dependencies: none, and that is a finding.** `dpkg-deb -I` shows no `Depends` line. The payload is a
|
||||
shell script and a unit file; the binaries the script calls (`curl`, `ip`, `dhclient`, `python3`,
|
||||
`systemctl`) run at **first boot**, not at postinst time. **Spike 4's open `dpkg --configure -a`
|
||||
ordering question therefore does not arise** — confirmed, not carried.
|
||||
|
||||
**How the postinst is structurally incapable of failing** — no `set -e`, every statement individually
|
||||
guarded with `|| true` or an `if`, and an unconditional `exit 0`. `build-deb.sh` refuses to emit a
|
||||
package that violates any of it.
|
||||
|
||||
**The guarantee was tested, not asserted.** Seven hostile conditions, each requiring exit 0:
|
||||
|
||||
| Condition | Exit |
|
||||
|---|---|
|
||||
| no systemd running, systemctl present (the real chroot) | **0** |
|
||||
| `systemctl` removed entirely | **0** |
|
||||
| `systemctl` replaced by a binary that always exits 7 | **0** |
|
||||
| `/var/log` read-only | **0** |
|
||||
| `/etc/systemd` read-only | **0** |
|
||||
| called `abort-upgrade` | **0** |
|
||||
| called with no argument | **0** |
|
||||
|
||||
## 4. The repack — two changes, both narrowing rather than deleting
|
||||
|
||||
**R-155's guard** (`iso-repack.sh:100-106`) **protected the single-entry mode's promise**: that menu
|
||||
shows one item labelled "Felhom telepítés" which boots the *automated* installer, and without
|
||||
`auto-installer-mode.toml` the same label would drop the user into a manual disk-picker — a button
|
||||
promising an unattended install that silently does the opposite. That promise is real, so the guard is
|
||||
**kept unchanged for `FELHOM_MENU=single`** and simply does not apply to `release`, where the absence
|
||||
of that file is release-gate criterion G1 rather than a defect.
|
||||
|
||||
**The menu collapse** happens at `iso-repack.sh:144-148` (the stock `grub.cfg` is replaced by a
|
||||
rendered template). A `release` template now renders **two interactive entries**; entry-count and
|
||||
banned-token gates are per-mode; the post-remaster verification reads the count back out of
|
||||
`final.iso`.
|
||||
|
||||
**Ruling — default entry and timeout.** Default is **the graphical interactive entry**; timeout **15 s**.
|
||||
Reasoning: Spike 1 measured that no automated disk selection can be safe on unseen hardware (no
|
||||
property distinguishes an internal disk from a customer's backup drive; a two-disk match silently wipes
|
||||
one), so a public image whose default is unattended puts the unsafe path in front of anyone who boots
|
||||
and walks away. And Spike 2 lost a probe to a **1-second** menu — a person reading two options needs
|
||||
longer than a machine.
|
||||
|
||||
**The automated entry is absent, not broken.** Skipping `prepare-iso` means no
|
||||
`auto-installer-mode.toml`, and the stock `grub.cfg` emits the Automated entry only inside
|
||||
`if [ -f auto-installer-mode.toml ]`. There is no entry that could fail in front of a customer.
|
||||
|
||||
## 5. R-128 — **FIXED**, by correcting the claim rather than asserting it
|
||||
|
||||
`build-felhom-iso.sh:44` claimed `ISO_VERSION` "aligns with felhom-host-install `SCRIPT_VERSION`".
|
||||
Nothing evaluated it and the two had drifted. **I did not turn it into a real assertion, because the
|
||||
coupling it claimed does not exist:** the ISO is a frozen artifact, while `felhom-host-install.sh` is
|
||||
fetched at run time from the website's git-sync of `main` (R-94/R-110), so whatever version an ISO
|
||||
carries, the script a box runs is always current. An assertion would invent a constraint. The comment
|
||||
now states the independence, and `ISO_VERSION` is `1.26.0`.
|
||||
|
||||
## 6. Part 5 — the defect, the fix, and where it now stands
|
||||
|
||||
### Round 1 (`1.26.0`) — the Terminal UI install FAILED on observable 4
|
||||
|
||||
Three of four passed: the package installed, the unit was enabled from inside the installer chroot,
|
||||
and the unit **fired on first boot** and registered at the hub. The fourth failed:
|
||||
|
||||
```
|
||||
felhom-bootstrap.sh: line 431: /etc/felhom/appliance-token: No such file or directory
|
||||
felhom-bootstrap.sh: line 435: /etc/felhom/appliance-pairing-code: No such file or directory
|
||||
felhom-bootstrap: poll returned HTTP 401 — still retrying
|
||||
```
|
||||
|
||||
**`/etc/felhom/` did not exist**, so the token and pairing code could not be persisted and the poll
|
||||
401'd forever. No claim code would ever appear.
|
||||
|
||||
**Root cause, mine.** `stub-first-boot.sh` opened with
|
||||
`install -d -m 0755 /etc/felhom /usr/local/sbin`. §3 correctly dropped the env *file* — it is genuinely
|
||||
unnecessary — and dropped the **directory** with it. `felhom-bootstrap.sh` uses `/etc/felhom/` for its
|
||||
runtime state.
|
||||
|
||||
**Why the gate missed it.** G9 proves the packaged script is byte-identical to repo HEAD, and it was.
|
||||
**I verified the payload files and never the directory the payload writes into** — a check that proves
|
||||
the thing present and not the thing it depends on.
|
||||
|
||||
### The fix, and its red-proof
|
||||
|
||||
`build-deb.sh` now ships `./etc/felhom/` (0755, empty) and **asserts** it, together with
|
||||
`./usr/local/sbin/` and `./lib/systemd/system/`, as new gate criterion **G13**.
|
||||
|
||||
**Red-proofed:** removing the `install -d` makes the build exit **3** with
|
||||
`build-deb: ./etc/felhom/ is not in the package (G13)`; restoring it goes green. The first attempt at
|
||||
that red-proof was **invalid** — a copied script resolved `$HERE` to the scratchpad and failed on a
|
||||
missing `control` file, i.e. non-zero for the wrong reason — and was redone in place.
|
||||
|
||||
### Round 2 (`1.26.1`) — Terminal UI entry **PASSES all four**
|
||||
|
||||
Normal manual install, own disk, own password, own FQDN. Host `spikesix.felhom.eu`.
|
||||
|
||||
| # | Observable | Result |
|
||||
|---|---|---|
|
||||
| 1 | the `.deb` is installed | **PASS** — `ii felhom-bootstrap 1.26.1 all` |
|
||||
| 2 | the unit is enabled | **PASS** — `enabled` |
|
||||
| 3 | the unit fired on first boot | **PASS** — `activating`; journal shows *"PAIRING mode (generic ISO, no baked customer/passphrase)"* → *"registering unclaimed appliance at the hub"* → *"registered — appliance token stored (0600)"* |
|
||||
| 4 | **the box wants a claim code** | **PASS** — `/etc/felhom/appliance-pairing-code` = **`ZY5-YY4`**; `appliance-token` present, 64 B, mode `600`; journal: *"not bound yet — polling every 30s until the operator or a customer self-bind lands (this is the normal waiting state, not an error)"* |
|
||||
|
||||
That is the product working end-to-end from a public image on a manual install: own disk, own
|
||||
password, nothing baked, and the box asking for a claim code.
|
||||
|
||||
### The Graphical entry — **NOT COMPLETED**, and this is why nothing is published
|
||||
|
||||
It reached the installer from the same image (KVM dialog, EULA, and the **Target Harddisk** screen
|
||||
showing `/dev/sda (20.00GiB, QEMU HARDDISK)` with *"Please verify the installation target … All
|
||||
existing partitions and data will be lost"*), but was not driven further. `Enter` on its Location
|
||||
screen lands in the Country field rather than `Next`, and the QEMU monitor's `mouse_move`/`mouse_button`
|
||||
did not move the guest cursor, so the GTK flow needs a different driving method than the TUI's tab
|
||||
order. **Part 5 requires both entries. It is not fully passed, so Part 7 did not run.**
|
||||
|
||||
The `.deb` path lives in `Install.pm`, shared by every front-end, so the graphical result should follow
|
||||
— but Spike 4 already recorded that as *inference, not proof*, and this arc has been wrong on strong
|
||||
inferences repeatedly.
|
||||
|
||||
### A fixture bug of mine, recorded twice because it cost two diagnoses
|
||||
|
||||
`qm set <vmid> --scsi0 … --boot order="scsi0;ide2"` silently produced `boot: order=net0;ide2` — PVE
|
||||
processed `--boot` before `--scsi0` existed. Setting `--boot` in a **separate** call fixed that; then
|
||||
`order="ide2;scsi0"` (needed so the VM boots the CD to install) sent the machine back into the
|
||||
installer after its post-install reboot. **Detach the CD, or flip the order to `scsi0`, once the
|
||||
install completes.** Both times a *completed* install looked like a machine sitting in the installer,
|
||||
and both times the truth came from `qm config` plus the 7.0 GB disk rather than from the screen.
|
||||
|
||||
## 7. Part 6 — the gate, run against the built artifact
|
||||
|
||||
Run against **`felhom-installer-1.26.1-pve9.2-1.iso`**,
|
||||
sha256 **`f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6`** — the image the
|
||||
Terminal-UI install in §6 was performed from, and the one that would be uploaded.
|
||||
|
||||
| # | Criterion | Scanned for | Result |
|
||||
|---|---|---|---|
|
||||
| **G1** | no `answer.toml` / `auto-installer-mode.toml` | both names at ISO root | **PASS — 0** |
|
||||
| **G2** | no root password or hash | `.rootpw.txt` companion; the answer file that would carry a hash | **PASS** — no `.rootpw.txt` emitted; no answer file exists to hold one |
|
||||
| **G3** | no SSH key | `root-ssh-keys`, `ssh-rsa`, `ssh-ed25519` | **PASS** — no answer file; package carries only a script and a unit |
|
||||
| **G4** | no customer identity | `FELHOM_CUSTOMER_ID`/`RETRIEVAL_PASSPHRASE` with values, claim code, api key, Bearer | **PASS** — only the empty initialisers at `felhom-bootstrap.sh:89` |
|
||||
| **G5** | credential scan **by enumeration** vs the stock PVE ISO | full recursive file-list diff, both directions | **PASS** — exactly **four** added paths: the three `felhomtheme/` files and `/proxmox/packages/felhom-bootstrap_1.26.0_all.deb`; three removed (`pvetheme/`) |
|
||||
| **G6** | menu present, both paths, human timeout | entry count, `set default`/`timeout`/`timeout_style`, banned tokens | **PASS** — 2 entries, `default=0` (graphical), `timeout=15`, `timeout_style` underscore |
|
||||
| **G7** | one `felhom-*.deb`, version recorded | `/proxmox/packages/felhom-*` | **PASS** — exactly 1, `Package: felhom-bootstrap`, `Version: 1.26.0`, **no `Depends`** |
|
||||
| **G8** | postinst cannot fail | live (comment-stripped) `systemctl start\|daemon-reload\|restart`, network commands, `set -e`, last line | **PASS — 0, 0, 0**, ends `exit 0` |
|
||||
| **G9** | `felhom-bootstrap.sh` == repo HEAD | sha256 of the packaged file vs the repo file | **PASS** — both `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
|
||||
| **G10** | build inputs committed | `git status --porcelain`, HEAD vs origin | **PASS** — clean and pushed at build time |
|
||||
| **G11** | published checksum + round trip | — | **NOT RUN** — nothing was published |
|
||||
| **G12** | bucket stays private | — | **NOT RUN** — the bucket was never touched |
|
||||
| **G13** | *(new, from Part 5's failure)* every directory the payload writes into is in the package | `./etc/felhom/`, `./usr/local/sbin/`, `./lib/systemd/system/` in `dpkg-deb -c` | **PASS** — all three present in `felhom-bootstrap_1.26.1_all.deb`; asserted by `build-deb.sh` and red-proofed |
|
||||
|
||||
**A gate refinement found by running it.** G7 also asked that the ISO's copy of the `.deb` sha256-match
|
||||
the package built from source. It does not, and cannot: `dpkg-deb` embeds build timestamps, so two
|
||||
builds of identical source differ. **G9 — the payload's identity — is the meaningful check**, and it
|
||||
passes. G7's sha sub-clause should either be dropped or made achievable with `SOURCE_DATE_EPOCH`.
|
||||
|
||||
## 8. Publication — done, and verified by round trip
|
||||
|
||||
Uploaded with `rclone` **in a container, configured entirely by environment variables**, so no
|
||||
credential file was ever written to disk — the fence asks for config files to be kept out of repo
|
||||
paths and removed at teardown; none was created to remove. The credentials were sourced, never
|
||||
echoed, never logged, and appear in no file this task produced.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| objects in the bucket | the ISO (1 705 322 496 B), `.sha256` (103 B), `.manifest.txt` (2 492 B) |
|
||||
| **round trip** | `curl https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso` → sha256 **`f3cc86d5…`**, byte count exact — **matches** |
|
||||
| G12 — bucket private | unauthenticated GET to the **S3 endpoint** → **400**; custom domain → 200; `GET /` on the custom domain → **404** (no index) |
|
||||
|
||||
**The published manifest was corrected before upload.** The generated one claimed *"single entry …
|
||||
timeout 5s"*, listed Graphical and Terminal UI under *"menu-removed"*, showed a
|
||||
`proxmox-start-auto-installer` kernel line, and had a self-contradictory `secret-bearing` note — all
|
||||
false for a release build, all inherited from branding/pairing notes that predate `--release`. The
|
||||
generator is fixed and the sidecar regenerated. **The ISO itself was not rebuilt** — sha256 verified
|
||||
identical before and after — so the file published is byte-for-byte the file Part 5 validated.
|
||||
|
||||
## 9. Teardown
|
||||
|
||||
**demo-hp:** VMs 500/501 `qm destroy --purge`; **scratch storage `spike5` removed**
|
||||
(`storage.cfg` back to 4 entries, `grep -c spike5` = 0); `/mnt/nvme-1tb/images/` empty; usage
|
||||
**6.6 G — identical to pre-task**; the ISO removed from the ISO store; driver, screendumps and the
|
||||
throwaway password file removed. `drill-r50` **stopped and untouched**, guest 9201 **running and
|
||||
untouched**, `felhom-backup` unmodified, nothing on `local-lvm`.
|
||||
|
||||
**demo-felhom:** not contacted.
|
||||
|
||||
**DooPlex:** scratchpad 84 K; build logs and the package build tree removed. `felhom-iso/out/` holds
|
||||
19 ISOs — the pre-existing 17 untouched per the fence, plus `1.26.0` and `1.26.1`, both unpublished
|
||||
and **neither with a `.rootpw.txt`**, which is G2's own evidence. Repo tree clean and pushed.
|
||||
|
||||
### Hub-side — **cleared**
|
||||
|
||||
Observable 4 works *by* the box registering itself, so each proof install created an unclaimed
|
||||
appliance. All three were discarded: **16** and **17** (the 1.26.0 round), then **18** (the two
|
||||
1.26.1 proofs). `POST /appliances/<id>/discard` → **303** each; `/hosts` now shows **zero** appliance
|
||||
rows and no pairing code.
|
||||
|
||||
The endpoint is `/discard`, **not** `/delete` — `hub/internal/web/server.go:345`, POST only. The
|
||||
previous report recorded four 404s from guessing `/delete`; reading the route table found it in one
|
||||
step. **R-131 gains no row.**
|
||||
|
||||
## 10. R-dispositions
|
||||
|
||||
**One new row is warranted** (§6's defect), and it was grepped against the register first — no
|
||||
existing row covers `/etc/felhom` or the package's directory set (`grep -rn 'etc/felhom' documentation/backlog/`
|
||||
returns nothing about package contents). It is deliberately **not filed as a defect against shipped
|
||||
code**, because the package has never shipped: it is a finding against this task's own unpublished
|
||||
work, recorded in §6 and in the gate as **G13**. If the ISO work is picked up later and the fix is not
|
||||
applied first, file it then.
|
||||
|
||||
Otherwise, no new rows. Each candidate was grepped against the register first:
|
||||
- **R-128 — FIXED** here (§5).
|
||||
- **R-155 — RESOLVED** here (§4): the guard is narrowed, not deleted.
|
||||
- **R-154** (`[first-boot]` is automated-only and nothing in the tree says so) — **addressed in code
|
||||
rather than by a row**: `pkg/build-deb.sh`'s header and `grub-release.cfg.tmpl` both state it with
|
||||
the measurements. The register row can close when the docs land.
|
||||
- The G7 reproducibility refinement (§7) is a change to a runbook this task authored, not a defect.
|
||||
|
||||
## 11. What did not happen, and what is still open
|
||||
|
||||
- **Part 8 partially done.** The release-gate runbook (`e787391`), `day0-install.md` C.0 (ISO vs
|
||||
manual, and when to use which) and `scripts/CHANGELOG.md` are written. **`OPEN-ITEMS.md` /
|
||||
`ROADMAP.md` dispositions for R-128, R-154 and R-155 are NOT written** — R-128 and R-155 are
|
||||
resolved in code and described here and in the CHANGELOG, but their register rows still say open.
|
||||
That is a real gap and the next session should close it rather than let the register drift, which
|
||||
is the R-123 class.
|
||||
- **The `.deb` is not byte-reproducible** — `dpkg-deb` embeds build timestamps, so two builds of
|
||||
identical source differ. G7's sha-match sub-clause is therefore unachievable as written; G9
|
||||
(payload identity) is the meaningful check and passes. Either drop the sub-clause or set
|
||||
`SOURCE_DATE_EPOCH`.
|
||||
- **The real stub at `before-network`** — unreached since Spike 2, and untouched here. It is now
|
||||
narrower than it was: on the `.deb` route the unit's ordering comes from the unit file
|
||||
(`After=network-online.target …`), not from `[first-boot].ordering`, so it governs operator-built
|
||||
appliance images only.
|
||||
- **Secure Boot** was not exercised. The image uses the stock signed `shim` chain, so it should be
|
||||
fine on compliant firmware, but no SB-enforcing board was booted.
|
||||
- **Only virtual hardware** was tested. Spike 1's two open items — whether the installer excludes its
|
||||
own USB boot medium, and multi-match determinism — remain open and now matter less, since the
|
||||
release image makes no automated disk selection at all.
|
||||
@@ -1,71 +0,0 @@
|
||||
# REPORT — PBS prune moved server-side, write proof closed (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run. **No code, no version bump.** Topic-scoped per the
|
||||
parallel-session rule; shared `REPORT.md` untouched.
|
||||
|
||||
**Full record:** `documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`
|
||||
|
||||
---
|
||||
|
||||
## Outcome — all parts complete
|
||||
|
||||
| Part | Result |
|
||||
|---|---|
|
||||
| 1 — prune gate | **Config-gated.** `keep_last: 0` on the PBS tier, both boxes → `prune_pbs_allowed=false`. No code, **no grant** |
|
||||
| 2 — prune jobs | 2 jobs, per live namespace, `keep-last 2`, daily **03:30 UTC / 05:30 CEST** |
|
||||
| 3 — dry run → real | Gate passed; both `TASK OK`; demo-hp 3→2, demo-felhom untouched |
|
||||
| 4 — write proof | **CLOSED — `TASK OK`, no job errors** |
|
||||
| 5 — GC | Scheduled `sun 04:30 UTC / 06:30 CEST`. **NOT run** |
|
||||
| 6 — `verify-new` | **Enabled** (operator ruling) |
|
||||
| — legacy ns | `demo-felhom-01` deleted with its ACLs + token (operator ruling, confirmed twice) |
|
||||
| 7 — roadmap | **R-89** + CONTEXT.md note |
|
||||
|
||||
## The fix, in one line
|
||||
|
||||
`allowPBSPrune := !t.Primary && t.KeepLast > 0` — so setting the PBS tier's `keep_last` to `0`
|
||||
disables both the `--prune-backups` value and the gate, in one config edit, **while the tier stays
|
||||
armed**. Verified: `backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0
|
||||
prune_pbs_allowed=false`, no `tier REJECTED` line.
|
||||
|
||||
## The proof
|
||||
|
||||
```
|
||||
07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied
|
||||
07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the change
|
||||
```
|
||||
|
||||
New snapshot `ns/demo-hp/ct/9201/2026-07-27T09:37:29Z`, chunks 9,787 → **9,813**, 97.0 % reused,
|
||||
45.80 s, **prune step absent entirely**. Driven via `POST /api/guest-backup/trigger` → `TriggerNow()`
|
||||
— the UI's „Mentés most" path, not `--selftest`, not raw `vzdump`.
|
||||
|
||||
**Hub gauge evidence NOT satisfied** — a +32.8 MB delta is below its 0.1 GB display granularity, so it
|
||||
still reads 12.6 GB / 13 %. Stated plainly rather than dressed up.
|
||||
|
||||
## The demo-felhom prediction — CLOSED
|
||||
|
||||
The claim was that demo-felhom's next weekly backup would make 3 snapshots and reproduce the prune
|
||||
failure. Neutralised on both halves: the box no longer attempts prune, and `prune-demo-felhom` covers
|
||||
the namespace server-side (verified live, `TASK OK`). **It will not reproduce.**
|
||||
|
||||
## Why it mattered more than the unpruned snapshots
|
||||
|
||||
demo-hp's PBS tier had reported failure on **every** backup since the tier was created on 07-26, while
|
||||
the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure
|
||||
invisible — which is precisely what happened at 07:13 UTC, when a real outage produced an
|
||||
indistinguishable result.
|
||||
|
||||
## Security property preserved
|
||||
|
||||
**No prune right was granted to any box.** Final ACLs are four entries, write-only
|
||||
(`DatastoreBackup`), live namespaces only. A compromised box still cannot delete its own offsite
|
||||
backups. `felhom-tenantsync.sh` was **not** edited — the ruling makes its current grant correct.
|
||||
|
||||
## Open
|
||||
|
||||
1. **R-89** — hub-owned retention policy (today's jobs are increment 1, not a stopgap).
|
||||
2. **Does the restic key on `storage-box-pool-1` have DELETE rights?** Unanswered, carried in R-89,
|
||||
and the more urgent half — if so, the daily app-data tier has the identical exposure and
|
||||
append-only mode is the equivalent answer. Rule once for both tiers.
|
||||
3. **GC has still never run.** First execution Sunday 04:30 UTC; worth watching, as nothing has ever
|
||||
exercised it here.
|
||||
4. Old 13 GB datastore copy still at `/srv/pbs-felhom` — rollback intact.
|
||||
-194
@@ -1,194 +0,0 @@
|
||||
# REPORT — R-100: a failing offsite tier must go stale (2026-07-28)
|
||||
|
||||
Hub **v0.79.0 → v0.80.0**; companion `felhom-controller` **v0.180.0 → v0.181.0** (the producer, shipped
|
||||
first). Written as `REPORT-r100.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom.eu 6369570`, `felhom-controller 4056fec`, `felhom-agent d5c7691` — all = origin/main. The only
|
||||
dirt in `felhom.eu` was a **foreign** `documentation/PROMPT-TEMPLATE.md` (shared worktree, untouched).
|
||||
Hub manifest **and** running pod both `0.79.0`; `staleAfter` = 48h; controller 0.180.0 and agent 0.110.0
|
||||
live on both boxes.
|
||||
|
||||
---
|
||||
|
||||
## The premise was wrong, and it was mine
|
||||
|
||||
R-100 was filed yesterday claiming *"the operator's fleet-wide alarm plane is silent"*. Phase 0 refuted
|
||||
that, twice:
|
||||
|
||||
1. **A failing offsite run does alarm.** `main.go:655` wires `SetOffboxNotify` → `NotifyBackupFailed`;
|
||||
the notify cooldown is 6h against a 24h cadence, so a nightly failure alarms nightly. Live hub DB:
|
||||
`backup_failed | operator | sent | 5`, latest 2026-07-27 17:42. The `isStale` doc comment —
|
||||
*"a recent-but-failing run is NOT stale (backup_failed owns that signal)"* — was **accurate**.
|
||||
2. **The orphaned-repo path I expected to be an indefinite hole is already covered.** The scheduled run
|
||||
returns early at `offbox.go:606`, *before* the `LastRun` write at `:716`, so `LastRun` freezes and
|
||||
`offsite_stale` fires normally.
|
||||
|
||||
I could find no failure mode that both advances `LastRun` and produces no operator signal.
|
||||
|
||||
**The real defect — defeated defence in depth.** `offsite_stale` is the hub-side, *pull-based* net that
|
||||
exists to be independent of controller-*pushed* events. Anchoring it on `LastRun` made it depend on the
|
||||
very thing it backs up: when the push is lost, the net cannot compensate, because the failing controller
|
||||
keeps refreshing the field the net reads. **F-HUB — this campaign's own finding, the hub dropping an
|
||||
event under `SQLITE_BUSY` with no retry** — is exactly that loss.
|
||||
|
||||
**Honest severity: MEDIUM**, not the top-ranked item. The fix is unchanged; the justification is not.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 answers
|
||||
|
||||
**P0.1 — a last-success timestamp did not exist.** `OffboxTarget` carried `LastRun`/`LastStatus`/
|
||||
`LastError`/`LastDuration` only. Recording one is a new field, not a transmission of something known.
|
||||
|
||||
**P0.2 — `LastStatus` on the wire**, from 4000 live reports (not from source alone):
|
||||
|
||||
| value | count | paired with |
|
||||
|---|---|---|
|
||||
| `ok` | 2269 | `last_run` set |
|
||||
| absent/null | 541 | `last_run` **empty** — never-ran |
|
||||
| `error` | 27 | `last_run` set |
|
||||
| **`running`** | 7 | a report captured **mid-run** |
|
||||
|
||||
Plus 1156 reports with no `offsite` object at all. **The legacy trap — status absent *with* a real
|
||||
`last_run` — occurs 0 times**, because `LastStatus="running"` is written the moment a run starts. It is
|
||||
still handled explicitly, but it is not a live shape. `running` being real is why the verdict ignores
|
||||
status entirely.
|
||||
|
||||
**P0.3 — sweep**
|
||||
|
||||
| tier | `LastRun` written on failure? | read as success by a verdict? |
|
||||
|---|---|---|
|
||||
| **Offsite restic** | YES (`offbox.go:716`) | **YES — hub `isStale`.** The defect |
|
||||
| **Tier 2 cross-drive** | YES (`recordTier2Failure`) | No hub verdict; UI only → **R-101, filed** |
|
||||
| Tier 1 recovery units | **NO** — derived from an actual artifact | structurally immune |
|
||||
| Shares offsite leg | YES | `sharing.html:180` shows the time only when status=="ok" — honest |
|
||||
| DB dump | n/a — **event-based** (`db_dump_completed`/`db_dump_failed`) | immune by design |
|
||||
|
||||
`offsite.go` is the **only** hub verdict anchored on a `LastRun`-shaped field. The deadline checker
|
||||
already uses distinct success/failure *events* — the pattern this converges on.
|
||||
|
||||
**P0.4 — the customer is NOT shown a failed offsite run as successful.** `backups_remote.html:34-36`
|
||||
leads with the status (`✓ Rendben` / `✗ Hiba` / `Fut…`). Two narrower Tier-2 instances → **R-101**.
|
||||
|
||||
---
|
||||
|
||||
## The fix
|
||||
|
||||
**Controller v0.181.0 (producer, shipped first).** `OffboxTarget.LastSuccess`, carried on the report as
|
||||
`last_success`. The rule is a pure function called unconditionally beside the `LastRun` write:
|
||||
|
||||
```go
|
||||
func offboxAnchorAfterRun(prev, at string, runErr error) string {
|
||||
if runErr != nil { return prev } // failures neither advance nor clear
|
||||
return at
|
||||
}
|
||||
```
|
||||
|
||||
Both directions are separate bugs: a failure must not **advance** it (the original defect) and must not
|
||||
**clear** it (one bad night making an established tier read as never-succeeded).
|
||||
|
||||
**Two silent-wipe sites found and closed** — the "seam built but never wired" shape, where the field
|
||||
exists, the writer sets it, and an unrelated routine path zeroes it:
|
||||
- `offboxConfigHandler` rebuilds the target from the form and copies runtime status field by field, so
|
||||
an ordinary settings save would have erased the anchor;
|
||||
- `ApplyOffsiteTarget` does the same on a hub re-apply.
|
||||
|
||||
Neither would have surfaced until the verdict changed, days later. **The first was proven live** — see
|
||||
below.
|
||||
|
||||
**Hub v0.80.0.** Three deliberate branches:
|
||||
- **never ran** — unchanged v0.73.0 anchored behaviour, still keyed on `last_run` on purpose: that field
|
||||
answers "has anything ever happened here", and a box whose *first* run failed is a run, not a newborn.
|
||||
- **legacy** (`last_run` set, no `last_success`) — degrades **explicitly** to the old behaviour, logged
|
||||
**once** per customer. Absence-as-failure would alarm the whole un-upgraded fleet; absence-as-success
|
||||
keeps the bug. Same degrade direction as R-88 Part 2's `age_state`.
|
||||
- **anchored** — counts from `last_success`; `last_status` is deliberately not consulted, because
|
||||
"error ⇒ stale" pages on every blip (the F-A1 noise path).
|
||||
|
||||
**The alarm text had to move with the verdict.** `emitStale` still said `last run 8h ago` while firing on
|
||||
a six-day-old success — a true alarm that reads as false. `staleAge` now separates *"runs are happening
|
||||
and failing — check the error, not the schedule"* from *"the offsite leg is silently not running"*.
|
||||
|
||||
---
|
||||
|
||||
## Red-proofs — all observed failing
|
||||
|
||||
| # | red-proof | observed failure |
|
||||
|---|---|---|
|
||||
| A | restore the `LastRun` anchor | `a tier that has not succeeded in 6 days reads as FRESH — that is R-100` |
|
||||
| B | delete the never-ran branch | `a newborn box alarmed — this is the 2026-07-23 cry-wolf that v0.73.0 fixed` |
|
||||
| C | collapse to `LastStatus == "error"` | `a single transient failure alarmed — 20h ... well inside the 48h threshold` |
|
||||
| D | delete the legacy degrade | `a legacy controller alarmed — that is a fleet-wide alarm storm on an un-upgraded fleet` |
|
||||
| + | drop the `runErr` guard (controller) | `a FAILED run advanced LastSuccess ... that is the R-100 defect in mirror image` |
|
||||
| + | always return `prev` | `a successful run did not advance the anchor` |
|
||||
| + | drop the wire field | `OffboxReportStatus dropped LastSuccess — the hub would degrade forever` |
|
||||
| + | drop the handler preservation | `a settings save erased LastSuccess` |
|
||||
|
||||
**A hollow test of my own, caught by red-proofing it.** The first version of the controller test
|
||||
re-implemented the rule in a local closure — mutating production code left it **green**. That is why
|
||||
`offboxAnchorAfterRun` was extracted: the test now calls the real rule.
|
||||
|
||||
Fixtures are the **real** wire shapes from P0.2, not invented JSON.
|
||||
`go build`/`go vet`/`go test` green in both repos (hub 17 pkgs, controller 27 pkgs), run separately
|
||||
from every commit.
|
||||
|
||||
---
|
||||
|
||||
## §6 — LIVE, on demo-hp (disposable; `peti-felhom` never touched)
|
||||
|
||||
A genuine restic failure was induced by pointing the target at a **closed port** (23 → 2) — it creates
|
||||
nothing, touches no data, and is exactly reversible.
|
||||
|
||||
```
|
||||
success run → last_status=ok last_run=11:24:20Z last_success=11:24:20Z
|
||||
INJECT port 23 → 2 ... and the settings save PRESERVED last_success = 11:24:20Z ← the wipe-site fix, live
|
||||
failing run → last_status=error last_run=11:25:48Z last_success=11:24:20Z ← ANCHOR HELD
|
||||
```
|
||||
|
||||
**As the hub received it:**
|
||||
|
||||
| box | status | `last_run` | `last_success` | anchor |
|
||||
|---|---|---|---|---|
|
||||
| **demo-hp** (induced failure) | `error` | 11:25:48Z | **11:24:20Z** | **HELD** |
|
||||
| **demo-felhom** (healthy) | `ok` | 11:29:22Z | **11:29:22Z** | **advanced** |
|
||||
|
||||
Also observed live, unplanned: **Scenario E**. Both boxes were still on the old controller at hub
|
||||
startup, and the degrade logged **exactly once per customer** —
|
||||
`[WARN] [offsite] demo-hp: controller sends no last_success — staleness degraded to the last-ATTEMPT
|
||||
anchor`. Two lines, two customers, same second.
|
||||
|
||||
**No spurious alarms:** 0 `offsite_stale` events since deploy (correct — both tiers succeeded minutes
|
||||
ago). `backup_failed` fired for demo-hp at 11:25:48 from the induced failure, confirming the
|
||||
pre-existing channel is intact and re-confirming the Phase 0 correction.
|
||||
|
||||
**Config restored** and verified field by field: `host=u629488-sub3.your-storagebox.de port=23
|
||||
user=u629488-sub3 repo=/home/felhom-repo enabled=True escrow=escrowed`.
|
||||
|
||||
### Proven live vs. proven by injected clock — stated plainly
|
||||
- **Live:** the anchor does not advance on failure; it does on success; it survives a settings save;
|
||||
`last_success` reaches the hub; the legacy degrade fires once per customer; no spurious alarms.
|
||||
- **Unit, injected clock only:** the 48h **threshold** behaviour itself — Scenarios A/B/C/D turning on
|
||||
elapsed time. A live threshold test would take days. **The threshold was NOT proven live.**
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — the rule
|
||||
**"Presence is not success"** added to `CLAUDE.md` and its versioned copy, with both instances
|
||||
(F-CRIT-2's phantom ctime, R-100's `LastRun`) and the corollary R-100's fix produced: when a verdict
|
||||
changes which field it counts from, the **alarm text must change with it**. `// R-100` notes sit at
|
||||
`isStale` and at the controller write site, each naming the test that pins it.
|
||||
|
||||
## Filed, not fixed
|
||||
- **R-101** — Tier-2 `LastRun` is also written on failure, and three customer surfaces render it without
|
||||
a status (two degraded branches plus the restore-confirm dialog). No hub verdict reads it.
|
||||
|
||||
## NOT yet live-validated (carried forward)
|
||||
- **The 48h staleness threshold itself** (see above) — and with it Scenario A end-to-end: no
|
||||
`offsite_stale` event has yet been *observed firing* from a genuinely stale success anchor, because
|
||||
that needs 48h of failure.
|
||||
- **Fault 4** — restic transport interruption; four injection approaches defeated by guest-bridged
|
||||
networking. (This task's closed-port injection sidesteps it rather than solving it.)
|
||||
- **R-99** — prune never removes phantom snapshots.
|
||||
- **R-101** — filed today, unvalidated.
|
||||
- `contentionAlarmAfter` (3h) — injected clock only.
|
||||
-181
@@ -1,181 +0,0 @@
|
||||
# REPORT — R-101 + F-DIAG + F-OPS (2026-07-28)
|
||||
|
||||
Controller **v0.181.0 → v0.182.0**; `felhom.eu` gains the manual-restore runbook (F-OPS) and the
|
||||
OPEN-ITEMS rows. Written as `REPORT-r101.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom-controller 3db8bfb`, `felhom.eu 6b7d516`, `felhom-agent d5c7691` — all = origin/main; the only
|
||||
dirt in `felhom.eu` was a **foreign** `PROMPT-TEMPLATE.md`. Controller **0.181.0** live on both boxes,
|
||||
hub `felhom-hub:0.80.0` ready 1/1.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0
|
||||
|
||||
**The render sites — three dishonest, two already honest.** The spec listed `backups_apps.html:216`
|
||||
as a defect site; it is in fact the one branch that *already* pairs its timestamp with a status badge.
|
||||
The real third site is the `Tier2DestInactive` branch.
|
||||
|
||||
| site | rendered | honest? |
|
||||
|---|---|---|
|
||||
| `:231` **restore confirm dialog** | `Legutóbbi másolat: {{.Tier2LastRun}}` — raw RFC3339, no status | **NO** — the one that matters |
|
||||
| `:195` `Tier2DestDisconnected` | `Utolsó: …`, no status | **NO** |
|
||||
| `:206` `Tier2DestInactive` | `Utolsó: …`, no status | **NO** |
|
||||
| `:217` main configured branch | `Utolsó: …` **+ status badge** | already honest |
|
||||
| `sharing.html:181` | rendered **only** when status=="ok" | already honest |
|
||||
|
||||
`Tier2LastStatus` was already set unconditionally at `handlers.go:1182`, so this was a wording/anchor
|
||||
problem, not a plumbing one. The restore button was gated on `{{if .Tier2LastRun}}`, so **Scenario C
|
||||
was live-reachable**: a tier that had attempted and never succeeded offered a restore and a timestamp.
|
||||
|
||||
**`cd.LastRun` is written on failure** — `recordTier2Failure` (`tier2.go:573-574`) writes it alongside
|
||||
`LastStatus:"error"`. Identical shape to R-100.
|
||||
|
||||
**Legacy state is universal, not an edge case.** All 7 Tier-2 rows across both boxes had `last_run` and
|
||||
no anchor. Scenario E was therefore the *initial state of every customer*, which is what made the
|
||||
legacy marker non-optional.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — the strings shipped
|
||||
|
||||
| case | string |
|
||||
|---|---|
|
||||
| dialog, normal | `… Legutóbbi sikeres másolat: 2026-07-28 16:43.` |
|
||||
| dialog, newest attempt failed | `… Legutóbbi sikeres másolat: 2026-07-28 16:40. Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek lehetnek.` |
|
||||
| card | `Utolsó sikeres: 2 perce` |
|
||||
| never succeeded | `Még nincs sikeres másolat` + `Még nincs sikeres másolat, amiből vissza lehetne állítani.` (restore removed) |
|
||||
| **legacy row** | `Utolsó: …` / `Legutóbbi másolat: …` — **today's wording, unchanged**, logged once per stack |
|
||||
|
||||
**Timestamp made human-readable** (agreed): new `fmtTimeStr` renders Budapest-local `2026-07-28 16:40`
|
||||
instead of the raw UTC `2026-07-28T14:40:55Z` a customer was previously asked to reason about.
|
||||
|
||||
**`SuccessTracked` is what makes the legacy case possible at all.** Without it, "row predates the
|
||||
anchor" and "row has an anchor and it is empty" are indistinguishable — both are `LastSuccess==""` —
|
||||
and every existing row would have rendered as never-succeeded on deploy. Legacy rows migrate on first
|
||||
touch: a row whose last known state was `ok` adopts that time (truthful — under the old code that run
|
||||
did succeed); a row whose last state was `error` seeds **nothing**, because the old data evidences no
|
||||
success.
|
||||
|
||||
## Part 2 — the copy-site hazard, and it was in the path
|
||||
|
||||
The three `record*` helpers each built a **whole `CrossDriveBackup` literal**, with a helper re-applying
|
||||
exactly two fields; everything else was zeroed on every status write. Adding `LastSuccess` to that shape
|
||||
would have had `recordTier2Failure` **clear** it — the mirror image of the defect, firing on the *first*
|
||||
failure rather than lying dormant.
|
||||
|
||||
Replaced with **`tier2Update`**, which copies the existing row and overlays the outcome: **compile-safe
|
||||
by construction** — a new field carries over unless deliberately overwritten, so nothing is preserved by
|
||||
a list that can fall out of date. Callers now clear explicitly what a run invalidates, reproducing the
|
||||
old behaviour exactly.
|
||||
|
||||
**Sweep of other rebuild sites:** `SetTier2Preference` mutates in place (safe); `SetCrossDriveConfig(name, nil)`
|
||||
in `api/router.go:774` is a deliberate delete. No others.
|
||||
|
||||
## Part 3 — F-DIAG
|
||||
|
||||
| class | signal it maps to | message head |
|
||||
|---|---|---|
|
||||
| `quota` | the pre-run soft-quota gate | `A távoli mentés nem fért el a tárhelykereten belül` |
|
||||
| `orphaned` | `ErrOffboxOrphaned` sentinel | `A távoli tárhely egy korábbi, már nem elérhető kulccsal készült` |
|
||||
| `no_repo` | restic "unable to open config file" | `A távoli tárhelyen nincs mentési adattár` |
|
||||
| `no_units` | "produced no snapshots" | `Nem volt mit menteni: egyetlen kijelölt alkalmazásnak sem található mentése` |
|
||||
| `transport` | refused/reset/timeout/authn/host-key | `A távoli tárhely nem érhető el (hálózat vagy bejelentkezés)` |
|
||||
| **`unknown`** | anything else | `A távoli mentés ismeretlen okból nem sikerült` |
|
||||
|
||||
The `unknown` class is deliberate: a cause that cannot be told apart where the error is produced is
|
||||
reported as unknown rather than folded into a neighbour.
|
||||
|
||||
**Secrets — and this caught a bug in my own first attempt.** The old message was
|
||||
`"…: " + err.Error()`, carrying the repo reference `sftp:<user>@<host>:<path>` off the box. My first
|
||||
sanitiser regex-matched `sftp:…` and `user@host` and *looked* complete; its own test caught it leaking
|
||||
on `ssh: connect to host <host> port 23: Connection refused` — a bare hostname in neither shape. It now
|
||||
redacts the target's **actual** host/user/repo-path literally, with the regex kept only as a backstop.
|
||||
Guessing at what a secret looks like fails exactly where it matters.
|
||||
|
||||
## Part 4 — F-OPS
|
||||
|
||||
`documentation/runbooks/RUNBOOK-manual-guest-restore.md`. Grounded in the real bind shape read off live
|
||||
guest 9201, not written from memory. Covers: which `mpN` are storage volumes (restored) versus **host
|
||||
binds** (taken as-is on the target); the `mp9` trap — it embeds the **source** VMID, so restoring to a
|
||||
different VMID can bind **another guest's bootstrap credentials**; strip-and-re-add before first boot;
|
||||
the hookscript check; and a positive pre-start verification that asserts every bind path exists rather
|
||||
than accepting "no error". Docs only, by design.
|
||||
|
||||
---
|
||||
|
||||
## Red-proofs — all observed failing
|
||||
|
||||
| # | red-proof | observed failure |
|
||||
|---|---|---|
|
||||
| A | dialog back on the attempt clock | `the dialog does not name the last SUCCESSFUL copy` |
|
||||
| C | gate the restore on `LastRun` again | `a tier that has NEVER succeeded still offers a restore — the dialog would promise a copy that does not exist` |
|
||||
| D | make the caution unconditional | `a HEALTHY tier shows the failed-attempt caution ("nem sikerült")` |
|
||||
| F | clear the anchor on failure | `a FAILED run wiped the success anchor (round 1) — one bad night would read as 'no copy has ever succeeded'` |
|
||||
| + | raw sanitiser | `the repo reference reached the message ("sftp:" leaked)` |
|
||||
|
||||
**F exercises the real `recordTier2Success` → `recordTier2Failure` sequence**, not a modelled copy — the
|
||||
R-100 lesson. The Scenario A/C/D tests **render the production template tree** and assert on the string
|
||||
the customer reads; a test asserting a template variable would prove nothing about wording, which is
|
||||
the defect.
|
||||
|
||||
`go build`, `go vet ./...`, `go test ./...` — 27 packages, `rc=0`; `template_id_gate.py` and
|
||||
`emoji_gate.py` both OK. Run separately from every commit.
|
||||
|
||||
---
|
||||
|
||||
## LIVE on demo-hp — the rendered dialog, which is the deliverable
|
||||
|
||||
**Legacy state** (before any run under v0.182.0) — today's wording, no fright:
|
||||
```
|
||||
Legutóbbi másolat: 2026-07-28 03:30
|
||||
```
|
||||
|
||||
Failure induced genuinely: the Tier-2 destination directory was **moved aside** and replaced by a file,
|
||||
so `mkdir …/recovery-unit` fails. (`chmod` does not work — the controller runs as root, which bypasses
|
||||
permission bits; `chattr +i` is refused, the unprivileged container lacks `CAP_LINUX_IMMUTABLE`. Both
|
||||
were tried and reported rather than glossed.) The real data was only ever moved, never deleted.
|
||||
|
||||
```
|
||||
status = error
|
||||
last_run = 2026-07-28T14:42:18Z ← ADVANCED
|
||||
last_success = 2026-07-28T14:40:55Z ← HELD
|
||||
last_error = mkdir …/paperless-ngx/recovery-unit: …
|
||||
```
|
||||
|
||||
**The rendered dialog, failed state:**
|
||||
```
|
||||
Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
|
||||
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:40.
|
||||
Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek
|
||||
lehetnek.
|
||||
```
|
||||
|
||||
**The rendered dialog, healthy state** (after restoring the destination and a successful run) — no
|
||||
caution, no tonal change:
|
||||
```
|
||||
Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
|
||||
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:43.
|
||||
```
|
||||
|
||||
Card lines: `Utolsó sikeres: 2 perce` → `Utolsó sikeres: most`.
|
||||
|
||||
**Everything restored:** destination is a directory again, 86 MB intact, mode 755, `.r101-aside` gone,
|
||||
`status=ok`, `last_success=2026-07-28T14:43:23Z`.
|
||||
|
||||
**demo-felhom is the untouched control:** all 5 rows still `tracked=None` after the deploy, rendering
|
||||
today's way, 15/15 containers up. Scenario E holding across a whole box nobody ran.
|
||||
|
||||
---
|
||||
|
||||
## NOT yet live-validated (carried forward)
|
||||
- **F-DIAG's classes** — unit-proven only. No live offsite failure of each class was induced; the
|
||||
`transport` class is the only one this arc exercised indirectly.
|
||||
- **Scenario C live** — the never-succeeded rendering is unit-proven; no fleet row is in that state
|
||||
(every row either migrated or has a real success), and manufacturing one would mean breaking a
|
||||
customer app's only Tier-2 history.
|
||||
- **The Tier-2 restore itself** was not executed — this arc changed what the dialog *says*, not what the
|
||||
restore does.
|
||||
- **R-100's 48h staleness threshold** — injected clock only.
|
||||
- **Fault 4** (restic transport interruption), **R-99**, **F-HUB**, fault 12, the three-way concurrency
|
||||
overlap — next campaign's material, untouched here.
|
||||
@@ -1,136 +0,0 @@
|
||||
# REPORT — R-106 + R-109 (+ R-122): closing the recipe-completeness set (2026-07-30)
|
||||
|
||||
Non-overwritten sibling per `CLAUDE.md:82-87` — the shared `REPORT.md` holds R-117 and is not touched.
|
||||
|
||||
Shipped: **agent v0.118.0 → v0.118.1** (`felhom-agent` `1c8a67e`, `6b5dade`) + **hub v0.83.0**
|
||||
(`felhom.eu` `acfc2b7`). Neither half is useful alone.
|
||||
|
||||
**Read §3 first if you read nothing else:** v0.118.0's R-106 half shipped INERT and live validation is
|
||||
what caught it — the recipe still said `"root"`, now with `namespace_state: resolved` beside it. Full
|
||||
account in the audit §6, filed as **R-125**.
|
||||
|
||||
## Part 0 — the answers, before the fix
|
||||
|
||||
### 0.1 Which items are actually open, and R-105/R-106's registration
|
||||
|
||||
`OPEN-ITEMS.md` calls itself "the single source of truth for open work" (`:1`), with `ROADMAP.md` keeping
|
||||
"the full history and reasoning" (`:3-4`).
|
||||
|
||||
| item | `ROADMAP.md` | `OPEN-ITEMS.md` | verdict |
|
||||
|---|---|---|---|
|
||||
| R-105 | row, `READY — 2026-07-28` (`:108`) | **absent** | **open but UNREGISTERED** |
|
||||
| R-106 | row, `READY — 2026-07-28` (`:109`) | **absent** | **open but UNREGISTERED** |
|
||||
| R-108 | row (`:111`) | row (`:50`) | registered |
|
||||
| R-109 | row (`:112`) | row (`:61`) | registered |
|
||||
|
||||
So R-109's own cell — "third recipe-completeness defect beside R-105/R-106" — was the **only** place in the
|
||||
register naming two open items. That is exactly the thread-loss the register exists to prevent, and it is
|
||||
itself a finding (filed **R-123**). Both now have rows.
|
||||
|
||||
**The set this task closes is R-106 + R-109**, matching the arc's stated definition of done (`OPEN-ITEMS.md:14`).
|
||||
**R-105 is NOT in it** and was not worked: it is M-sized and is about three *hub-held DR records* being `{}`
|
||||
(`hosts.dr_record_json`, `host_escrow.directive_json`, and the `drives` third — already traced and populated
|
||||
by the 2026-07-28 target move). Different fields, different owner, different size.
|
||||
|
||||
### 0.2 Where the recipe is generated — three producers, not two
|
||||
|
||||
| half | repo | function |
|
||||
|---|---|---|
|
||||
| host (guests/pbs/drives/pve_storage) | `felhom-agent` | `BuildDRRecipeHostHalf`, `internal/hub/dr_recipe.go:86` |
|
||||
| app (customer/apps/offsite_restic) | `felhom-controller` | `controller/internal/report/dr_recipe.go` |
|
||||
| **assembly + delivery** | `felhom.eu/hub` | `AssembleDRRecipe`, `internal/store/dr_recipe.go:104`; served by `handleDRRecipeDownload`, `internal/web/dr_recipe.go:14`, route `internal/web/server.go:439` |
|
||||
|
||||
R-109's "host-half" is therefore the **agent**, and the field must also pass the **hub's** allow-list — see §2.
|
||||
|
||||
### 0.3 What the namespace field actually contained — verified, and the brief was RIGHT
|
||||
|
||||
The eleven-session-old brief held up. Live, pre-fix, from the hub for **both** boxes:
|
||||
|
||||
```json
|
||||
"pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }
|
||||
```
|
||||
|
||||
against `/etc/pve/storage.cfg` on the same boxes:
|
||||
|
||||
```
|
||||
pbs: felhom-pbs
|
||||
datastore felhom-offsite
|
||||
namespace demo-felhom # demo-hp reads: namespace demo-hp
|
||||
```
|
||||
|
||||
Traced to source: `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo
|
||||
per item once the list is namespace-scoped via `?ns=` (`:118-120`) → always empty → `ToHub` normalises empty
|
||||
to `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it in.
|
||||
|
||||
**The authority taken, and why:** storage.cfg's `namespace` on the pbs storage. It is the same field
|
||||
`vzdump --storage <pbs>` makes PVE read, and the agent's own verify client is built from it
|
||||
(`cmd/felhom-agent/main.go:1164`). Deriving the recipe from anything else is how it drifts again.
|
||||
|
||||
## 1. R-109's ambiguity is real, in the boxes' own pre-fix recipe
|
||||
|
||||
```json
|
||||
"pve_storage": [
|
||||
{ "name": "local-lvm", "type": "lvmthin", "content": "images,rootdir" },
|
||||
{ "name": "felhom-backup", "type": "local-dir", "content": "backup" },
|
||||
{ "name": "felhom-pbs", "type": "pbs", "content": "backup" },
|
||||
{ "name": "local", "type": "local", "content": "backup,import,vztmpl,iso" }
|
||||
]
|
||||
```
|
||||
|
||||
No `backup_target` key anywhere. `felhom-backup` (live, `/mnt/hdd_1`) and `local` (`/var/lib/vz`, archives
|
||||
frozen 2026-07-28) are both `content=backup` dir storages; `local` is also the *historically* correct answer,
|
||||
which is what makes guessing it so easy.
|
||||
|
||||
## 2. R-122 — a fourth defect, found here, and it had already shipped
|
||||
|
||||
`AssembleDRRecipe`'s `hostHalfShape`/`appHalfShape` are **allow-lists** dressed as forward-compat. The
|
||||
controller has emitted `offsite_restic` since fork-4 (`controller/internal/report/dr_recipe.go:39-41`, "so DR
|
||||
knows WHERE to recover from"); `appHalfShape` never listed the key. Verified both ways:
|
||||
|
||||
- **stored**: `dr_recipe.app_half_json` carries it for all three real customers —
|
||||
`peti-felhom`, `demo-felhom` (`u629488-sub1.your-storagebox.de:23/home/felhom-repo`), `demo-hp`.
|
||||
- **delivered**: the downloaded recipe's top-level keys were
|
||||
`recipe_version, customer, guests, pbs, drives, pve_storage, apps` — **no `offsite_restic`**.
|
||||
|
||||
So a restorer reading the recipe had **no offsite location at all**, for the whole life of the feature, with
|
||||
a green suite throughout — because the test fixture `drAppHalf` is hand-written and omits the field.
|
||||
|
||||
**Deviation from the task's §7.10 ("Findings — filed as R-n, none fixed"), stated rather than absorbed:**
|
||||
I fixed it. Reasons — (a) Part 0 authorises working the real set if it differs; (b) it is the same
|
||||
symptom the task is named for (the recipe is incomplete), and the worst instance, a whole section missing;
|
||||
(c) it is in the *same two structs* R-109 forced me to edit, and leaving one of three known keys off a
|
||||
drop-list I was already correcting would be indefensible. It is filed as R-122 with a SHIPPED disposition.
|
||||
|
||||
## 3. The before/after recipe — both boxes, quoted
|
||||
|
||||
```
|
||||
demo-felhom BEFORE "namespace":"root" backup_target absent offsite_restic absent
|
||||
AFTER "namespace":"demo-felhom" backup_target {resolved, felhom-backup, /mnt/hdd_1}
|
||||
offsite_restic {u629488-sub1…}
|
||||
demo-hp BEFORE "namespace":"root" backup_target absent offsite_restic absent
|
||||
AFTER "namespace":"demo-hp" backup_target {resolved, felhom-backup, /mnt/nvme-1tb}
|
||||
offsite_restic {u629488-sub3…}
|
||||
```
|
||||
|
||||
The two boxes DISAGREEING is the point — nothing is hardcoded. And the ambiguity was not theoretical:
|
||||
on both boxes `felhom-backup` holds an archive from **07-30 04:36** while `local` stops at
|
||||
**07-28 17:5x**, frozen at the target-move date. The recipe now names the live one.
|
||||
|
||||
Full evidence, all seven red-proofs and the publish observables:
|
||||
`documentation/audits/R106-R109-recipe-completeness-2026-07-30.md`.
|
||||
|
||||
## 4. Findings filed (none of them fixed except R-122, see §2)
|
||||
|
||||
| id | finding |
|
||||
|---|---|
|
||||
| **R-122** | `AssembleDRRecipe` allow-list dropped `offsite_restic` for the feature's whole life — **FIXED here**, hub v0.83.0 |
|
||||
| **R-123** | R-105 and R-106 were `READY` in `ROADMAP.md` with no `OPEN-ITEMS.md` row — referenced only inside R-109's prose. Registered here |
|
||||
| **R-125** | v0.118.0 shipped an inert R-106 because the "production path" test injected `fakeObserver` one layer below the break — **FIXED** in v0.118.1; filed for the doctrine point (name the seam you inject at) |
|
||||
| **R-124** | The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""` and there is no namespace literally named `root` — a restorer pasting it into `pct restore --ns root` would fail. Pre-existing wire convention, deliberately unchanged; documented at `PBSRootNamespace` |
|
||||
|
||||
## 5. Not done, and why
|
||||
|
||||
- **R-105, R-108, D5** — out of scope by the task's §6. R-108 blocks D5; starting either would leave both half-done.
|
||||
- **The backup machinery** — untouched. This corrects the record, not the doing.
|
||||
- **`sess-f` (0.116.0) and `drill-r50` (0.113.0) were not upgraded** — neither was named as a venue, and `drill-r50` is fenced by the task's §6.
|
||||
- **R-124 not fixed** — changing the wire's spelling of the root namespace mid-R-106 would shift the field's meaning during the fix meant to make it trustworthy.
|
||||
@@ -1,117 +0,0 @@
|
||||
# REPORT-r116-diag — the `/disks` payload captured, R-116's mechanism isolated (2026-07-30)
|
||||
|
||||
Read-only diagnosis run by CC on DooPlex. **No code written, nothing built, nothing published.**
|
||||
Full evidence: `documentation/audits/DIAG-r116-disks-payload-2026-07-30.md`.
|
||||
|
||||
A `REPORT-*.md` sibling, not the shared `REPORT.md` (`CLAUDE.md` parallel-session rule).
|
||||
|
||||
## Outcome
|
||||
|
||||
**Both goals met.** The `/disks` read path is solved and written down verbatim, proven by a
|
||||
present-drive control run *first*; and the absent-state payload was captured, which isolates the
|
||||
mechanism.
|
||||
|
||||
**R-116 is theory #1 — "the registry-union row writes `false`" — the theory that was raised, declared
|
||||
wrong, and retracted. The retraction was the error.**
|
||||
|
||||
In the absent state `/disks` returns **4 rows, not 3**. The drive appears twice and the two facts the
|
||||
controller needs sit on different rows:
|
||||
|
||||
| row | source | `mount_path` | `guest_path` | `backup_target` |
|
||||
|---|---|---|---|---|
|
||||
| `felhom-backup` | Observe (`disks.go:196-284`) | `""` | `""` | **`true`** |
|
||||
| `694034cc-…` (the UUID) | registry union (`disks.go:297-339`) | `/mnt/cel` | `/mnt/felhom-drives/cel` | **field ABSENT ⇒ `false`** |
|
||||
|
||||
So the row holding the flag contributes **no key** to `driveTargetByPath`, and the row that owns the key
|
||||
says `false` → `isTarget[a.Path]` is `false` → generic `storage_disconnected`. On return the rows
|
||||
re-merge into one carrying both facts → specific `backup_target_restored`. Applying
|
||||
`intermediary.go:602-618` to the captured payloads gives PRESENT `True` / ABSENT `False` /
|
||||
RETURNED `True` — **the live asymmetry reproduced from payload alone.**
|
||||
|
||||
The union row's `MountPath` survives the device because the union source is the systemd **`.mount` unit
|
||||
file** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never reads the mount table. The
|
||||
dedup at `:298` therefore does not fire, because `seen` is keyed on the one field the absent state
|
||||
empties (`:290-295`).
|
||||
|
||||
**Theory #2 (the basis of the shipped v0.115.0) is false on both halves**; **#3 is false too**
|
||||
(`isTarget["/mnt/cel"]` is `false` as well). **v0.115.0 is provably inert** — its fallback calls
|
||||
`StablePathForRaw("")`, which returns `""` (`intermediary.go:69-75`), so it assigns nothing.
|
||||
|
||||
## The read path (this cost two prior sessions — it should never cost again)
|
||||
|
||||
The token plaintext exists in exactly one place: `bootstrap.json` **on the Proxmox host**. The agent's
|
||||
own store keeps SHA-256 hashes only (`tokenstore.go:26-32`), which is what defeated the earlier attempts.
|
||||
|
||||
```bash
|
||||
ssh felhom-pve
|
||||
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
|
||||
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
|
||||
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])")
|
||||
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
|
||||
```
|
||||
|
||||
Control run, live felhom-pve, drive present: **HTTP 200, 2483 bytes, 4 plausible rows** — so Part 5's
|
||||
0-rows-on-a-present-drive failure mode is excluded.
|
||||
|
||||
## Where the absent state was staged
|
||||
|
||||
No new box. The existing DooPlex **nested-PVE drill fixture** (`drill/drill.qcow2`, snapshot `virgin`) —
|
||||
my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with
|
||||
the **byte-identical live agent binary** (`sha256 f48544ad…`, `--version` 0.115.0) and every
|
||||
state-producing step through the real endpoints (`format` → `assign` → `guest-attach` →
|
||||
`backup/target`). Its present-state row matched felhom-pve's control run field-for-field before it was
|
||||
trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token
|
||||
record, no controller) are enumerated in the audit §4.
|
||||
|
||||
## Two new findings, filed not chased
|
||||
|
||||
- **R-117 (READY M) — outranks R-116.** After a detach/reattach the guest's bind is a **dead mount**:
|
||||
host is healthy on the new device node, guest still names the old one, and `ls`/write through it
|
||||
return **`EIO`** — while `/disks` reports `attached` + `bound_under_parent:true` + `backup_target:true`.
|
||||
`planDriveGates` therefore takes the `Return` branch and **restarts the customer's apps onto a dead
|
||||
namespace, reporting healthy, with no alarm on any channel.** R-113's conjunction cannot catch it:
|
||||
one half is satisfied by the stale entry, the other by the new device, and neither compares them.
|
||||
This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs.
|
||||
- **R-118 (READY XS).** An absent drive's union row reports the **root filesystem's** capacity as its own
|
||||
(46 GiB / 9.2 % for a 4 GB drive) — `statfsCapacity` at `disks.go:335-338` statfs's a bare directory on
|
||||
root. `observe.go:176-183` guards the Observe path against exactly this; the union path does not.
|
||||
`durable_id` is still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id.
|
||||
|
||||
## Register
|
||||
|
||||
`documentation/backlog/OPEN-ITEMS.md` — R-116 updated with the mechanism and the fix constraints;
|
||||
R-117 and R-118 added. The single register edit this session makes.
|
||||
|
||||
## Record correction
|
||||
|
||||
The brief's baseline `controller 0.185.1` is the version the **golden bakes**. **0.186.0** (R-114 +
|
||||
R-112, 2026-07-29) is what **demo-felhom** runs — **demo-hp is still on 0.185.1**, so the fleet is
|
||||
split, and R-114's `TargetAbsent` branch exists only on demo-felhom. Confirmed: agent **0.115.0**
|
||||
(felhom-pve) / **0.113.0** (demo-hp), hub **0.81.0** (manifest pin and live pod image agree),
|
||||
host-install **1.22.0**, `felhom.eu` HEAD `c3ce4c7`.
|
||||
|
||||
> **Correction, 2026-07-30.** As first written this section said 0.186.0 was what *both* demo boxes run.
|
||||
> That was wrong — only felhom-pve's guest was sampled and the result generalised to the fleet. demo-hp
|
||||
> re-checked directly → `0.185.1`. Fixed here and in the audit's baseline table.
|
||||
|
||||
## Teardown and fences
|
||||
|
||||
Drill guest destroyed, scratch storage removed, mount unit deleted, secrets `shred -u`'d, VM powered
|
||||
off, **`drill.qcow2` restored to `virgin`** (the golden-bake fixture is exactly as found), scratch qcow2
|
||||
and console dumps deleted. DooPlex `/mnt/5_hdd` at 24 %, unchanged.
|
||||
|
||||
Both demo boxes **read-only throughout** and re-verified after teardown: demo-hp `local-lvm` **38.83 %**
|
||||
(identical before/after and to Part 5), `drill-r50` still stopped, felhom-pve `felhom-backup` still
|
||||
active on `/dev/sdb`, both guests running, **v0.115.0 untouched**.
|
||||
|
||||
`sess-d-0452c4` now reads **STALE**, and the delete gate refuses only on ONLINE
|
||||
(`hub/internal/web/customer_delete.go:220-228`) — so it **is** now deletable; the command is recorded in
|
||||
the audit rather than executed (customer delete runs external teardown plus a DB purge). **`sess-c` is
|
||||
also still present and was not recorded by the Session-C audit** — same terms.
|
||||
|
||||
## Not done, deliberately
|
||||
|
||||
No fresh controller gate-log / hub-event correlation: that observable was already captured live and
|
||||
identically twice, and only the payload was missing. Staging a controller meant a hub customer, a
|
||||
pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8
|
||||
correlates the payload to those existing measurements and labels that step as inference.
|
||||
@@ -1,96 +0,0 @@
|
||||
# REPORT — SPIKE R-117: a dead bind that reports healthy (2026-07-30)
|
||||
|
||||
Written as `REPORT-<topic>.md`, not `REPORT.md`, per this repo's parallel-session rule and the
|
||||
established local pattern (`REPORT-r116-diag.md`, `REPORT-session-c.md`, …). The shared `REPORT.md`
|
||||
was not touched.
|
||||
|
||||
**Class: Spike.** Deliverable is a findings document. **No production code was written; no `.go` file
|
||||
in either repo was modified; nothing was built for deployment, published, or version-bumped.**
|
||||
|
||||
## Deliverables
|
||||
|
||||
| File | Action |
|
||||
|---|---|
|
||||
| `documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md` | **new** — the findings doc (Q1–Q7, evidence, recipe, probe comparison, recommendation) |
|
||||
| `documentation/backlog/OPEN-ITEMS.md` | **R-117 row rewritten** with the mechanism, the reproduction recipe and the fix constraint — the one register edit, per the brief |
|
||||
|
||||
**No CHANGELOG entry.** This repo's changelogs are per-area (`hub/`, `scripts/`, `website/`); a
|
||||
documentation-only change belongs to none of them. Stated rather than silently skipped, per standing
|
||||
rule 4.
|
||||
|
||||
## Baselines
|
||||
|
||||
agent `main` **v0.116.0** @ `d4eb259` · controller `main` **v0.186.0** @ `b331f18` ·
|
||||
`felhom.eu` HEAD `29bcfeb` · hub **live 0.82.0** · golden **0.186.0** ·
|
||||
demo-hp PVE 9.2.2 / kernel 7.0.2-6-pve, **live agent 0.113.0** (= manifest `MinAgent`; never used as
|
||||
the source of a behavioural claim — every predicate result came from a probe built from `main`).
|
||||
|
||||
## Results
|
||||
|
||||
**All seven questions answered empirically.**
|
||||
|
||||
- **Q1 — reproduced 3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp.
|
||||
**The device-node change is a consequence of the defect, not a precondition** — control test: with
|
||||
the stale bind held the drive returns as `sdc` (8:32); released, the letter is reused (`sdb`, 8:16).
|
||||
- **Q2 — two death states**: device removed ⇒ superopts gain `shutdown`, `EIO`(5) on read and write,
|
||||
host and guest; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads served from cache.
|
||||
The raw host mount is genuinely healthy in both. **No cross-device mis-identification is possible**
|
||||
on this path — the unit is fs-UUID-keyed.
|
||||
- **Q3 — confirmed at source and live.** Both halves of the R-113 conjunction compare **field 5** of a
|
||||
mountinfo line and **never read field 3 (`major:minor`)**, so neither can see that the bind and the
|
||||
raw mount name different devices. Measured `BoundUnderParent = TRUE` over an `EIO` namespace.
|
||||
- **Q4 — a pure-`/proc` check costs 0.16–0.45 ms**, cannot hang, spins up no disk, writes nothing.
|
||||
**`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow. The
|
||||
hang case is below; it is the sharpest result in the run.
|
||||
- **Q5 — the agent**, and not on balance: the controller runs inside the guest and cannot see the host
|
||||
mount tables the check needs.
|
||||
- **Q6 — recovery works in place, guest never restarted** (init PID identical). **The repair code
|
||||
already exists and three call sites already invoke it**, including the controller's `Return` branch
|
||||
*before* it restarts apps — all defeated by one early return.
|
||||
- **Q7 — a bind can die in steady state, no cycle at all.** The gate produces no action and **nothing
|
||||
is emitted on any channel.** A `Return`-branch fix cannot reach this half.
|
||||
|
||||
## Q4's hang case — measured, and it is the sharpest result
|
||||
|
||||
Against a `dmsetup suspend`ed device (I/O queues instead of returning `EIO`):
|
||||
|
||||
- **P1 and P2 completed in 364 µs / 206 µs.** They read `/proc`, so no block device is involved.
|
||||
- **`statfs` and `getdents` completed and reported HEALTHY** — on a wedged device they do not even hang.
|
||||
- **Every probe that touches the device blocked, including a buffered write with no `fsync`** — the
|
||||
`O_CREAT` metadata path needs journal access (`wchan=do_get_write_access`). There is no cheap-and-safe
|
||||
write probe.
|
||||
- **The blocked process survived `SIGTERM` and `SIGKILL`** (`stat=D`, still alive 3m50s after `kill -9`)
|
||||
and died only when the device was resumed. So **`systemctl restart felhom-agent` would hang**, leaving
|
||||
the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal
|
||||
the leak (5→5, 5→6).
|
||||
|
||||
**A timeout protects the caller's control flow and nothing else.** This turns "prefer a cheap probe" into
|
||||
a fence: **the fix must issue no block I/O.**
|
||||
|
||||
## Teardown — done, all three layers
|
||||
|
||||
Guest 9301 destroyed; `r117scratch` removed; both dm devices and both loop devices gone; `scsi_debug`
|
||||
unloaded (`/dev/sd*` back to `sda1..3`); no `r117` mounts, `/mnt` and `/root` exactly as found; `local`
|
||||
**37.02 %** against a session-start **37.00 %**. Fences re-verified *after* teardown: 9201 `running`,
|
||||
`drill-r50` `stopped`, `local-lvm` **38.84 % byte-identical**, `felhom-backup` `content backup`
|
||||
unchanged, live `/mnt/felhom-drives` intact with both submounts, agent service `active`. **Layer 3 is
|
||||
genuinely empty** — 9301 had no network interface and ran no controller, so no hub-side record was ever
|
||||
created.
|
||||
|
||||
**Ordering trap worth keeping:** a suspended dm device must be `dmsetup resume`d *before* any `umount`,
|
||||
or the teardown itself blocks on the same uninterruptible sleep.
|
||||
|
||||
## Not measured
|
||||
|
||||
No load or duration testing of the recommended check — P1/P2 were single calls, not a sustained
|
||||
reconcile loop on a many-drive box. Nothing suggests a problem (they are two `/proc` reads the code
|
||||
already performs), but it was not measured.
|
||||
|
||||
## Findings filed, none fixed
|
||||
|
||||
R-117 (mechanism + recipe), **R-117a** steady-state death with no event (HIGH, larger than R-117 as
|
||||
filed), **R-117b** `statfs`/`getdents` are hollow liveness probes, **R-117c** three untested comments
|
||||
promising "live + usable in the guest", **R-117d** the self-heal that already exists is short-circuited
|
||||
(HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing internet,
|
||||
including the WireGuard OOB path, **R-117f** an I/O liveness probe turns a wedged drive into an
|
||||
unkillable agent (HIGH — disqualifies a whole probe class).
|
||||
@@ -1,203 +0,0 @@
|
||||
# REPORT — installer-channel record correction + R-29 filing (2026-07-29)
|
||||
|
||||
Two commits, documentation only. No code, no version bump, no CHANGELOG entry, no build, no deploy,
|
||||
no box touched. Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so root `REPORT.md` (the E-2
|
||||
increment-1 report) is preserved.
|
||||
|
||||
| # | Commit | Baseline | Scope |
|
||||
|---|--------|----------|-------|
|
||||
| 1 | `d4c07873ca0c3d3e547373a9fafc0e472a6535e8` | `36d635a4cdc1`, unmoved | Retract a false R-94/E-2d finding; open R-110 |
|
||||
| 2 | (this commit) | `d4c07873`, unmoved | File R-29 to the register; three record defects; this report |
|
||||
|
||||
---
|
||||
|
||||
## Commit 1 — `d4c07873`
|
||||
|
||||
### What was false
|
||||
|
||||
`36d635a4` recorded that `felhom-bootstrap.sh` fetches the installer **from the hub**, that the hub
|
||||
therefore serves 1.19.0, and that a fresh ISO install runs the pre-E-2 installer. All three wrong.
|
||||
The claim had propagated into two `OPEN-ITEMS.md` rows, the ranking rationale, and `ROADMAP.md:149`.
|
||||
|
||||
### Confirmation table — all PASS
|
||||
|
||||
| # | Claim | Read at | Result |
|
||||
|---|-------|---------|--------|
|
||||
| F1 | bootstrap fetches from the **website** | `scripts/iso/felhom-bootstrap.sh:96` | PASS — `INSTALL_URL="${FELHOM_INSTALL_URL:-https://felhom.eu/scripts/felhom-host-install.sh}"` |
|
||||
| F2 | hub-rendered command points at the same URL | `customer_unified.html:563`, `:1262` | PASS — and **three** emission sites, not two: `:563` static, `:1262` JS error branch, **`:1267` JS success branch** |
|
||||
| F3 | website serves `/scripts/` from a git-sync tree tracking `main` | `manifests/webpage.yaml` — nginx `:74-77`, sparse-checkout CM `:211-218`, git-sync `:272-281`, init `:299-307` | PASS — `--branch=main --period=30s --link=current`; sparse-checkout `/website/` + `/scripts/`; `location /scripts/ { root …/current; }`. No image build, no ArgoCD step |
|
||||
| F4 | `hostInstallVersion` selects nothing | `configs.go:28`, `:487`; `render_test.go:219`; `customer_unified.html:494` | PASS — repo-wide grep returns exactly those 4 code sites; all other hits prose. Rendered as a text label |
|
||||
| F5 | every generated flag is parsed by 1.22.0 | generator `customer_unified.html:1206-1239` vs parser `felhom-host-install.sh:1175-1212` | PASS — `--mode --cores --memory --vmid --node --acl-storages --operator-pubkey-file --preserve-state-from --skip-provision --dry-run --preflight-only --allow-new-leaf` (+ `--customer-id`); every one a parser case. **No functional gap** |
|
||||
| F6 | installer is 1.22.0 | `scripts/felhom-host-install.sh:187` | PASS |
|
||||
| F7 | the drift test is hollow | `render_test.go:219-221` | PASS — `strings.Contains(html, hostInstallVersion)` compares the constant to itself; passes at any value |
|
||||
|
||||
### Live command 1 — what the URL actually serves
|
||||
|
||||
```
|
||||
$ curl -fsS https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 '^SCRIPT_VERSION='
|
||||
SCRIPT_VERSION="1.22.0" # the SINGLE version source (F-1): -h, the run banners, and the hub
|
||||
```
|
||||
|
||||
### Live command 2 — the drift gate's real state
|
||||
|
||||
```
|
||||
$ python3 scripts/hostinstall_gates.py; echo "exit=$?"
|
||||
ok: SCRIPT_VERSION=1.22.0
|
||||
ok: header has no version literal
|
||||
FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0 (F-1: bump both together)
|
||||
ok: age is in the installed package set
|
||||
… (six further ok lines) …
|
||||
hostinstall gates: 1 FAILURE(S)
|
||||
exit=1
|
||||
```
|
||||
|
||||
### Phase 0 source read — PAIRING reaches the same installer invocation
|
||||
|
||||
Mode selection `felhom-bootstrap.sh:537-541`: a fresh VM with no baked customer-id calls
|
||||
`run_pairing`. On HTTP 200 from `/api/v1/appliance/poll` the loop writes the hub-delivered
|
||||
`FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it, and calls
|
||||
`run_direct` **in the same invocation** (`:495-499`). `run_direct` is the single site that fetches
|
||||
`$INSTALL_URL` (`:322-330`), builds the args (`:334`) and invokes `bash "$SCRIPT_TMP" "${args[@]}"`
|
||||
(`:343`). The customer it yields is the one the operator bound — claimable. **So the ISO leg is the
|
||||
spine for E-2d**, not an obstacle to it.
|
||||
|
||||
### Rows changed
|
||||
|
||||
| ID | Before | After |
|
||||
|---|---|---|
|
||||
| R-94 (line 14) | `READY — deferred until E-2d`, blocked on E-2d, false ISO/hub framing | `READY (XS)`, blocked on nothing, retracted + re-scoped to three legs |
|
||||
| R-94 (line 16) | duplicate row, `READY #2`, 1.19.0 vs 1.20.0 | **deleted** — merged |
|
||||
| R-110 | did not exist | opened, `WAITING-ON-OPERATOR (S)` |
|
||||
| E-2d | ISO implied as obstacle | Next-action appended; ISO is the spine |
|
||||
| ranked list | 1 R-95 · 2 R-94 (high-consequence) · 3 R-86 · 4 R-87 | 1 R-95 · 2 R-94 **de-ranked** · 3 R-86 · 4 R-87 · 5 R-110 |
|
||||
|
||||
Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
|
||||
`documentation/runbooks/day0-install.md`.
|
||||
|
||||
---
|
||||
|
||||
## Commit 2 — R-29 filing + record hygiene
|
||||
|
||||
### The §1.1 ruling — is R-29 the right home for a non-design-v2 gate? **Yes. Proceeded.**
|
||||
|
||||
R-29's title says *"the design-v2 green gates"*, and `scripts/hostinstall_gates.py` is not one — it
|
||||
comes from drill F-1 (2026-07-12) and postdates the item. Four things decide it anyway:
|
||||
|
||||
1. **R-29's own audit list already spans well beyond design-v2 subject matter.** It names
|
||||
`docker_run_volume_path_gate` (docker mount safety), `offbox_rename_gate`, `app_row_dedup_gate`
|
||||
and `manifest_bearer_gate` (secrets — `runbooks/secrets.md:76`). The title is a misnomer relative
|
||||
to the item's own body.
|
||||
2. **Part (b) — "the systemic half is the real item" in R-29's words — is stated with no
|
||||
design-v2 restriction.** It is about the *enforcement mechanism*: "the gates run only when a human
|
||||
remembers to run them… decide where they run (pre-push hook, `build.sh` step, or a CI job) and make
|
||||
a red gate block the train." That is gate-agnostic and repo-wide.
|
||||
3. **`hub_confirm_gate.py` is already on R-29's list and lives in the same `scripts/` directory** as
|
||||
`hostinstall_gates.py`. Wiring one and not the other would be arbitrary.
|
||||
4. **Identical failure shape, identical genre.** Both self-describe as mechanical grep-assertions
|
||||
(`hostinstall_gates.py:2`, `hub_confirm_gate.py:1-8`); both exist, assert something true, and are
|
||||
invoked by nothing.
|
||||
|
||||
R-29 has already absorbed one independent re-raise without minting an ID (2026-07-18 rehearsal note)
|
||||
and says so explicitly. This is the third. No new ID minted.
|
||||
|
||||
### Orphan-search evidence — `hostinstall_gates.py` and `hub_confirm_gate.py`
|
||||
|
||||
Re-established at `d4c07873`. Pattern `hostinstall_gates\|hub_confirm_gate`.
|
||||
|
||||
| # | Scope | Command | Result |
|
||||
|---|-------|---------|--------|
|
||||
| S1 | `felhom.eu`, all file types | `grep -rn "$PAT" . --exclude-dir=.git` | **19 hits, zero invocations.** All are docstrings (`scripts/hostinstall_gates.py:5`, `scripts/hub_confirm_gate.py:7`), code comments (`hub/internal/web/configs.go:27`, `scripts/felhom-host-install.sh:189`) or prose (`REUSE.md:62`, `CONTEXT.md:540,564`, `hub/CHANGELOG.md:371,1292,1351,1385`, `scripts/CHANGELOG.md:483,524`, 3 files under `documentation/audits/`, `ROADMAP.md:149,158`, `OPEN-ITEMS.md:14`) |
|
||||
| S2 | sibling repos | `grep -rln "$PAT" /mnt/5_hdd/felhom.eu/git --exclude-dir=.git` minus this repo | 3 files, all in `.claude-memory/` (`MEMORY.md`, `drtier-by-default-2026-07-12.md`, `polish-batch-2026-07-13.md`). Notes, not invokers |
|
||||
| S3 | `~/.claude` | `grep -rln "$PAT" /home/kisfenyo/.claude`, and targeted on `settings*.json`, `skills/`, `hooks/` | **Zero hits in settings, skills or hooks** — where an invoker would live. Remaining hits are `file-history/` (Claude Code's own backups of files edited in past sessions) and `paste-cache/` (pasted task specs). Neither is an invocation site |
|
||||
| S4 | git hooks | `ls -1 .git/hooks/ \| grep -v '\.sample$'` | **Empty — every hook is a `.sample` stub** |
|
||||
| S5 | build files | `find . -type f \( -iname Makefile -o -iname '*.mk' -o -iname justfile -o -iname 'Taskfile*' \)` | Only `hub/Makefile`; `grep -n gate hub/Makefile` → **zero occurrences** |
|
||||
| S6 | CI | `find . -type d \( -name .github -o -name .gitea -o -name .woodpecker* -o -name .drone* -o -name .circleci \)` | **Empty — `felhom.eu` has no CI configuration at all** |
|
||||
|
||||
Of the four gates in `scripts/`, only `site_gates.py` is mandated (`CLAUDE.md:153`);
|
||||
`manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`.
|
||||
|
||||
### Rows changed
|
||||
|
||||
| ID | Before | After |
|
||||
|---|---|---|
|
||||
| **R-29** | **absent from `OPEN-ITEMS.md`** (`grep -c` → 0) while present at `ROADMAP.md:158` since before the 2026-07-27 rebuild | **opened, `READY (S for (a) / M for (b))`**, owner CC |
|
||||
| R-94 | leg (b) stood alone | leg (b) carries `→ R-29` as its class. Row otherwise untouched |
|
||||
| E-2d | cited `:322-341` for an invocation at `:343` | cited `:322-343`, with the fetch / args / call each pinned separately |
|
||||
| R-95, R-86, R-87 | `READY #1`, `READY #3`, `READY #4` | `READY` — markers dropped (see below) |
|
||||
|
||||
Not added to the ranked list under *"Why the READY rows rank this way"*: that list is a top-N
|
||||
rationale, not a complete ordering (R-99, R-102, R-103, R-104, R-108, R-109, R-89, R-92, R-93 and
|
||||
E-2d are all READY and absent from it). Ranking R-29 was not asked for and would be a priority claim
|
||||
this task has no basis to make.
|
||||
|
||||
### §2.2 — markers dropped, not renumbered
|
||||
|
||||
The `#1/#3/#4` markers duplicated ranked-list positions 1/3/4 for exactly those three rows; two
|
||||
orderings of one set is what produced the hole when the `#2` row was merged in `d4c07873`. Removing
|
||||
the duplicate leaves the ranked list as the single maintained ordering.
|
||||
|
||||
### The other two record defects
|
||||
|
||||
- `ROADMAP.md:147` — cited a non-existent **R-164**; it means controller **v0.164.0**'s
|
||||
deliberate-stop filter. Corrected. (It had already cost one max-ID scan a false positive.)
|
||||
- `CONTEXT.md:564` — asserted in the present tense that the single VERSION source is *"gated by
|
||||
`scripts/hostinstall_gates.py`"*. The gate exists, asserts exactly that, is red, and runs nowhere.
|
||||
Corrected to say the cross-check exists but is not enforced, tracked as R-94 leg (b) / R-29.
|
||||
`hub/internal/web/configs.go:27` carries the same false claim in a code comment and was left alone
|
||||
— R-94 leg (b) territory, needs a hub build.
|
||||
|
||||
Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
|
||||
`documentation/backlog/README.md`, `CONTEXT.md`, this file.
|
||||
|
||||
---
|
||||
|
||||
## Commit 3 — `de5a3e5` — the record-hygiene rider
|
||||
|
||||
Baseline `7383400a`, unmoved. Four XS items from `7383400a` §9; the three deferred observations stay
|
||||
deferred.
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `CONTEXT.md:540` | *"`scripts/hub_confirm_gate.py` enforces"* → asserts, but is not enforced (R-29). Third instance of the class after `:564` and `configs.go:27` |
|
||||
| `REUSE.md:62` | Same claim, *"enforces zero"*. The **rule stays** — never native `confirm()`/`prompt()` is correct guidance and this is a reuse-reference row — only the enforcement claim changes |
|
||||
| `OPEN-ITEMS.md:4` | Root `REPORT.md` = overwritten per-session; `REPORT-<topic>.md` = non-clobbering sibling (`CLAUDE.md:82-87`), 14 of them. Prohibition unchanged |
|
||||
| `OPEN-ITEMS.md:55` | Heading scoped to *"the **TOP** READY rows"* + a half-sentence that it is deliberately not a full ordering. **No row added to the list** |
|
||||
|
||||
`hub/internal/web/configs.go:27` — the fourth instance — left alone (R-94 leg (b), needs a hub build).
|
||||
|
||||
### Part 3 NOT done — its stated evidence is false
|
||||
|
||||
The spec proposed recording a convention: *"`ROADMAP`'s P3 `Status` column only ever takes `idea` or a
|
||||
`SHIPPED`/`PROVEN-LIVE` variant, and never takes `READY`, `BLOCKED` or `WAITING-ON-OPERATOR`"*, and
|
||||
concluded from that vocabulary split that `ROADMAP` `Status` is **disposition** while `OPEN-ITEMS`
|
||||
`State` is **live actionability**.
|
||||
|
||||
Refuted inside P3 itself (`ROADMAP.md`, lines 113–172):
|
||||
|
||||
| Line | ID | P3 `Status` cell |
|
||||
|---|---|---|
|
||||
| 151 | R-90 | `BLOCKED on Hetzner CX33 availability (operator, 2026-07-27)` |
|
||||
| 152 | R-91 | `WATCHING — gated on demo-felhom's first post-migration PBS backup` |
|
||||
| 150 | R-110 | `idea — found 2026-07-29, **WAITING-ON-OPERATOR (a ruling, not a defect)**` |
|
||||
| 137 | R-86 | `**NEXT — operator ruling 2026-07-27**` |
|
||||
|
||||
And P2's recovery-gaps sub-table (lines 99–107) uses a bare `READY` for nine rows, including `E-2`.
|
||||
|
||||
Both halves fail: the column **does** take the `OPEN-ITEMS` vocabulary, and `BLOCKED`/`WATCHING`/
|
||||
`NEXT` are live-actionability words, not dispositions. The real shape is a free-text status field
|
||||
that has absorbed both vocabularies over time — usually disposition, sometimes not. Writing the
|
||||
proposed convention down would install a **new false invariant** in the commit chain whose whole
|
||||
purpose was removing them, so per the spec's own instruction (*"If you think this ruling is wrong, say
|
||||
so and make no edit"*) `documentation/backlog/README.md` was not touched.
|
||||
|
||||
The narrow question that prompted it still resolves benignly: R-29 and R-94 reading `idea` in
|
||||
`ROADMAP` and `READY` in `OPEN-ITEMS` is **not** a defect and needs no sync. What is unsupported is
|
||||
generalising that into a rule.
|
||||
|
||||
---
|
||||
|
||||
## Standing note
|
||||
|
||||
**No gate was wired, fixed, run as a hook, or deleted by any of the three commits.** Filing R-29 is
|
||||
not doing R-29; the enforcement decision (pre-push hook / `build.sh` step / CI) and the audit of the
|
||||
remaining eight gates are R-29 part (b), M-sized, and its own task.
|
||||
@@ -1,94 +0,0 @@
|
||||
# REPORT — Session C: R-113, R-114, R-112 proven on a real box; C5 fails on a new defect (2026-07-29)
|
||||
|
||||
`RUNBOOK-session-c-2026-07-29.md`. Full evidence: `documentation/audits/SESSION-C-2026-07-29.md`.
|
||||
Root `REPORT.md` untouched.
|
||||
|
||||
## Verdicts
|
||||
|
||||
| Claim | Fix | Verdict |
|
||||
|---|---|---|
|
||||
| **C4** — offer appears and moves the target | R-112 | ✅ **PASS** |
|
||||
| **C3** — absent target tells the truth, offers nothing | R-114 | ✅ **PASS** |
|
||||
| **C5** — `backup_target_absent` + `backup_target_restored` | R-113 | ❌ **FAIL** — generic alarm, specific recovery → **R-116** |
|
||||
|
||||
**All three shipped fixes work.** R-113's gate fires in **4 seconds** (E-2d measured zero over 4½
|
||||
minutes). R-114's message is correct. R-112's banner reaches the customer. C5 fails on a **fourth,
|
||||
separate defect that was unreachable until R-113 made the gate fire at all.**
|
||||
|
||||
## What the drill box ran
|
||||
|
||||
Agent **0.114.0 from the Day-0 manifest** — the shipped binary, so C5 tested the real artifact and
|
||||
closed R-115's observation 1 for R-113. Controller **0.186.0 hand-deployed** after install (§3.1
|
||||
ruling (a)); the vouched golden bakes 0.185.1, so **C3/C4 prove the code, not the shipped golden** —
|
||||
that lag is filed against R-115, not a new ID.
|
||||
|
||||
## The three headline observables
|
||||
|
||||
**R-113** — detach at 18:43:50, gate at **18:43:54**, on exactly the shape that defeated it before:
|
||||
```
|
||||
raw /mnt/mentes : NOT mounted
|
||||
bind /mnt/felhom-drives/mentes : /dev/sdb[/felhom-data] <- the stale bind SURVIVED
|
||||
```
|
||||
|
||||
**R-114** — with the target absent: absent copy 1, **system-disk copy 0**, **offer block 0**. Both of
|
||||
E-2d's falsehoods gone.
|
||||
|
||||
**R-112** — banner element 1 and the never-configured copy in the HTML; after the wizard, the offer
|
||||
with `data-path="/mnt/felhom-drives/mentes"`. Healthy renders nothing, **proven positively**: idle
|
||||
delta 0 `/backup/tiers` calls, page-load delta **+1**, single caller ⇒ the seam ran and chose silence.
|
||||
|
||||
**Decline path proven** (registration confers no role), `restart_required:true`, agent did **not**
|
||||
self-restart, in-flight check recorded before I restarted it, wrapper created the storage at the
|
||||
drive's own mountpoint.
|
||||
|
||||
## C5's failure
|
||||
|
||||
```
|
||||
absent : Event pushed: storage_disconnected (error) <- GENERIC
|
||||
return : Event pushed: backup_target_restored (info) <- SPECIFIC
|
||||
```
|
||||
|
||||
`backup_target_absent` count **0** across the whole run. The alarm and its recovery cannot be matched
|
||||
— precisely what `notifyDriveReturned`'s own comment forbids.
|
||||
|
||||
**Root cause (R-116):** `driveTargetByPath` builds `out[GuestPath] = d.BackupTarget`, but the drive is
|
||||
**two `/disks` rows** and the flag and the guest path sit on different ones — the `felhom-backup`
|
||||
storage row carries `BackupTarget: true` and gets a guest path only while classified user-data; the
|
||||
registry union row carries the guest path and **never assigns `BackupTarget`**. Absent ⇒ they separate
|
||||
⇒ generic. Return ⇒ they rejoin ⇒ specific. v0.184.1 fixed the *keying*, not this.
|
||||
|
||||
## Mirror + over-correction guard — PASS, with a caveat
|
||||
|
||||
Non-target drive detached ⇒ `storage_disconnected`, `backup_target_absent` count 0. **Over-correction
|
||||
guard passes**: both drives present ⇒ 0 ABSENT lines, target stayed healthy — R-113's stricter presence
|
||||
did not make a healthy drive read absent. **Caveat: the mirror passes trivially**, because the target
|
||||
also produced the generic event; it confirms no over-correction but cannot confirm discrimination.
|
||||
|
||||
## Record
|
||||
|
||||
- `OPEN-ITEMS.md` — **R-113, R-114, R-112 → SHIPPED + PROVEN-LIVE**; **R-116 opened** (READY (S), P1);
|
||||
**E-2 and E-2d CLOSED as partially proven** with R-116 as the one named open leg, per the runbook's
|
||||
§9 decided-in-advance rule.
|
||||
- `ROADMAP.md` — R-116 under P1.
|
||||
- **Capability map NOT touched** — it still has **no E-2 / backup-target rows at all**, so no row could
|
||||
be moved to PROVEN-LIVE. Creating them is a design act, not a validation act. Third session running
|
||||
that this has been noted.
|
||||
|
||||
## Teardown
|
||||
|
||||
VM destroyed, storage removed, **`pvesm status` after == before** (`local-lvm` 38.78 %), guest 9201 and
|
||||
`drill-r50` untouched. **Customer ruling: DELETE**; attempted and correctly refused (`host … is
|
||||
ONLINE`) — deletable once the destroyed host ages to DOWN (>1 h), command recorded in `OPEN-ITEMS.md`.
|
||||
|
||||
## What did not happen
|
||||
|
||||
`backup_target_absent` never fired, so its severity, Hungarian copy and hub routing remain unexercised
|
||||
— R-116 blocks them. The offer was accepted via the endpoint the button POSTs, not a browser click (no
|
||||
browser automation on DooPlex); the rendered control and its non-auto-submission were verified in HTML.
|
||||
The stale bind still naming a dead device node after return was observed, not investigated.
|
||||
|
||||
## The arc
|
||||
|
||||
E-2 ends here. Its stated definition of done is **R-106 + R-109, R-108 and D5** — none of which this
|
||||
detour touched. The detour was worth taking: it found six real defects (R-111 through R-116), four of
|
||||
them customer-affecting, none of which any unit suite had caught.
|
||||
@@ -1,98 +0,0 @@
|
||||
# REPORT — tester gate: golden re-baked to 0.188.0, fresh-install proof PASSED (2026-07-31)
|
||||
|
||||
Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so the shared `REPORT.md` (E-2 increment 1) is
|
||||
not clobbered. Full record with every observable: `documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md`.
|
||||
|
||||
## Outcome
|
||||
|
||||
**§7.2 — YES: a fresh install is safe to hand to an external tester.** ISO boot → claimable,
|
||||
app-serving box in ~10 minutes unattended, and an app's data restored **from the drive with the
|
||||
guest's `app.yaml` gone**, proven readable by the application over its own TCP path.
|
||||
|
||||
**Golden 0.186.0 → 0.188.0** baked, published, vouched. **No ISO rebuilt** — Part 0 proved none was
|
||||
needed. No existing box changed; floor still v0.156.0, MinAgent still 0.113.0.
|
||||
|
||||
## Part 0 — the ISO does not need rebuilding
|
||||
|
||||
Verified against the ISO **on disk**, not from source. It bakes exactly three Felhom payloads
|
||||
(`felhom-bootstrap.sh`, its unit, the secret-free pairing env) — full-base64 match, 1 hit each — and
|
||||
**0** hits for `SCRIPT_VERSION="1.2`, `felhom-controller`, `vzdump-lxc-9100`. The installer is fetched
|
||||
at run time (`felhom-bootstrap.sh:96`) and the live URL is byte-identical to repo HEAD
|
||||
(sha `ab8b283e…`, v1.22.0, committed six days *after* the ISO). The golden arrives via the hub-vouched
|
||||
artifact manifest (`felhom-host-install.sh:423-433`). The one genuinely baked, drift-capable thing is
|
||||
`felhom-bootstrap.sh` itself — currently at repo HEAD.
|
||||
|
||||
**Proven live**, not just argued: the fresh box ran `felhom-host-install v1.22.0` and fetched golden
|
||||
**v0.188.0**, sha-verified.
|
||||
|
||||
## Part 1 — bake / publish / vouch
|
||||
|
||||
Baked **0.188.0**, not the brief's 0.187.0: 0.187.0 lacks D5, and Part 2 step 6 *is* the D5 claim, so
|
||||
that golden could not have passed the proof this task exists for. 0.188.0 satisfies R-120 anyway.
|
||||
Stated rather than absorbed, per standing rule 4.
|
||||
|
||||
GOLDEN_VERSION=0.188.0
|
||||
GOLDEN_SHA256=7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299
|
||||
|
||||
Three observables: 404 pre-gate (with a 200 control on 0.186.0 so it is not vacuous), then an
|
||||
**anonymous** download returning `http=200 bytes=649310288` and a matching sha; the manifest read back
|
||||
showing `0.188.0` selected; and the consumer call `GET https://hub.felhom.eu/api/v1/artifacts/sess-g`
|
||||
returning the pair. Plus a fourth: a real fresh box fetched and sha-verified it.
|
||||
|
||||
**R-120's gate evaluated and allowed.** Exercised both ways rather than inferred from silence —
|
||||
vouching 0.185.1 first produced `flash=golden_behind_fleet`, the logged `artifact vouch REFUSED`, and
|
||||
**no write** (the manifest still read 0.186.0); then 0.188.0 produced
|
||||
`Artifact manifest set: agent=0.118.1 golden=0.188.0`.
|
||||
|
||||
## Part 2 — the clean-install proof, on demo-hp
|
||||
|
||||
All seven steps PASS. Venue was demo-hp (Tier 0, the designated drill host) using the scratch dir
|
||||
storage at `/mnt/nvme-1tb` that `target-selection.md:38-40` names; `local-lvm`, `drill-r50` and both
|
||||
9201s untouched.
|
||||
|
||||
Highlights: real day-0 pairing → bind → install; a **real** claim (the code is emailed-only, R-119 —
|
||||
the operator relayed it), with the gate flipping `dashboard not yet claimed` → `authentication
|
||||
required`; controller **0.188.0** confirmed *from the box*; **rallly** (postgres) + **homebox**
|
||||
deployed through the real endpoints.
|
||||
|
||||
The D5 leg: recovery unit `portable-carried=2/2, withheld=0`; the carried `DB_PASSWORD` matched the
|
||||
live one **by fingerprint** (`14c8f515…`, never printed); guest `app.yaml` moved aside; restore
|
||||
returned `secrets recovered=2/2`. Step 7 read the data from **rallly's own network namespace** over
|
||||
TCP to `rallly-postgres` — not the localhost trust socket that produced D5's false pass — and the same
|
||||
path with a wrong password returned `FATAL: password authentication failed`, proving the credential
|
||||
does real work. **The discriminator held: PRE-BACKUP row = 1, POST-BACKUP row = 0.**
|
||||
|
||||
## Part 3 — runbook integrity
|
||||
|
||||
`RUNBOOK-manual-build.md` told the reader to use a "RECORDED" qemu line that is itself labelled
|
||||
*reconstructed*, and whose source says it *"was never saved"*. The real invocation is now captured
|
||||
from this bake and recorded as canonical in **§4.0**, alongside the bake/publish/teardown steps, the
|
||||
template-rot warning and where the R-120 gate actually lives. The old runbook's deviation entry is
|
||||
marked SUPERSEDED with a forward pointer.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
1. VM 310 destroyed with `--purge --destroy-unreferenced-disks 1`; `/mnt/nvme-1tb/images/` empty.
|
||||
2. `cc-scratch` removed, `storage.cfg` back to its original four entries; `felhom-backup` available
|
||||
**926 492 284 KiB before and after** — space returned exactly.
|
||||
3. Hub: **`sess-g` and its host record DELETED, full cascade** — `customer DELETE cascade COMPLETE
|
||||
for sess-g (journal #8) — full teardown`, residue purged including `appliance_registrations=1`.
|
||||
Verified positively: `/configs` and `/hosts` both loaded (10160 / 9880 bytes) with **0** hits for
|
||||
`sess-g` and 0 for the appliance UUID. The gate refused twice first (409 host ONLINE, then 400
|
||||
missing acknowledgements) — the record cannot be deleted until the destroyed box ages out of
|
||||
ONLINE, ~30 min. **`sess-f` deliberately NOT deleted** (R-131); its command is in the audit §7.1.
|
||||
Secrets shredded in the guest and on the box.
|
||||
|
||||
## Findings — filed, none fixed
|
||||
|
||||
`R-128` ISO_VERSION/SCRIPT_VERSION comment is false · `R-129` demo-hp's "no baked SSH key" is stale
|
||||
(key auth works) · `R-130` `HARD_MIN_LVM_GIB` warns and proceeds — a hard min that is not hard ·
|
||||
`R-131` `sess-f` is a fourth orphaned scratch customer · **`R-132` — `curl -w '%{redirect_url}'`
|
||||
printed the hub operator password into a session transcript; `HUB_PW` needs rotating.**
|
||||
|
||||
## Not done, deliberately
|
||||
|
||||
No ISO built; no defect fixed; no golden deployed to an existing box; no floor or MinAgent change;
|
||||
offsite/PBS-DR legs not exercised (the task forbids pointing anything at production PBS or the real
|
||||
restic offsite, so `sess-g` ran DR-tier off); Campaign 10, the demo-hp repartition and subdomain
|
||||
onboarding untouched.
|
||||
@@ -1,97 +0,0 @@
|
||||
# REPORT — SPIKE 4: can a `.deb` in the ISO deliver the stub on an interactive install? (2026-07-31)
|
||||
|
||||
> Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub
|
||||
> v0.85.0 session and the second session in a shared clone never touches it. Supersedes this file's
|
||||
> Spike 1–3 contents.
|
||||
|
||||
**Class: Spike.** Findings only — no production file changed, no release ISO built, nothing published.
|
||||
Evidence: `documentation/audits/SPIKE-universal-iso-4-2026-07-31.md`.
|
||||
|
||||
## The answer is yes, and it was measured with the negative control in the same box
|
||||
|
||||
One ISO, 15 GRUB entries, a trivial probe `.deb` injected into `/proxmox/packages/`. Two VMs on
|
||||
demo-hp built with `qm` so the run was visible in the web console: **400 interactive**, **401
|
||||
automated control**.
|
||||
|
||||
On the **interactive** install (`spikefour.felhom.eu`):
|
||||
|
||||
- the package is installed — `ii felhom-spike4-probe 0.0.1`
|
||||
- its **postinst ran** — marker file present, content intact
|
||||
- it **enabled a systemd unit**, and **that unit fired on first boot** (uptime 7.98 s, `pid1: systemd`)
|
||||
- and on **that same machine**, `proxmox-first-boot` is not installed and `/var/lib/proxmox-first-boot`
|
||||
does not exist — Spike 3's negative reproduced, not assumed
|
||||
|
||||
So the two delivery mechanisms are independent, and the one that survives the path we are actually
|
||||
shipping is the `.deb`. **The product — insert the stick, install Proxmox normally choosing your own
|
||||
disk and password, box sets itself up and waits for a claim code — is now measured rather than hoped
|
||||
for.** With one honest caveat: what was measured is a trivial probe package, not Felhom's real stub.
|
||||
Packaging the real stub and confirming pairing end-to-end is the last step before a spec (~60 min).
|
||||
|
||||
## What a postinst may and may not do
|
||||
|
||||
Identical on both paths: `pid1 = unconfigured.sh`, **no running systemd**, `/proc` and `/sys` mounted,
|
||||
and **`systemctl enable` succeeds** (it wrote the symlink). Network and DNS *happened* to be up —
|
||||
inherited from the installer's own DHCP.
|
||||
|
||||
Four constraints for the real postinst, so they get written against rather than discovered:
|
||||
|
||||
1. Never `systemctl start` or `daemon-reload` — there is no systemd running. `enable` is the only verb.
|
||||
2. **Never require the network**, despite it being present here. A box installed with the cable out
|
||||
gives a postinst no route, and a failing postinst breaks the customer's install.
|
||||
3. Never fail — guard everything, `exit 0`.
|
||||
4. Do the real work in the unit at first boot, where systemd, network and a booted kernel exist.
|
||||
|
||||
## Two smaller results
|
||||
|
||||
**The repack preserves the `.deb`, but not naively.** `xorriso … -boot_image any replay` fails with
|
||||
*"Overlapping MBR partition entries"* — and `iso-repack.sh:270-292` already documents that exact
|
||||
failure and its fix. Mirroring it produced a working image (19 El Torito entries; the `.deb` extracted
|
||||
back out is byte-identical). So this is an insertion into an extract→modify→re-master cycle our repack
|
||||
already performs, not a new build stage.
|
||||
|
||||
**Q3:** `iso-repack.sh:100-106` refuses an ISO without `auto-installer-mode.toml`. It is a guard, not
|
||||
a structural requirement, and its reasoning is sound for the shape it was written for — already R-155,
|
||||
cited exactly here. With no mode file the stock grub.cfg does not emit the Automated entry at all; with
|
||||
a mode file but no answer, that entry aborts safely and loudly.
|
||||
|
||||
## A correction I owe you from last session
|
||||
|
||||
**R-153 is retracted.** The register grep this task mandated shows R-94 already carries it verbatim at
|
||||
`OPEN-ITEMS.md:15`, status `READY (XS)`, with leg (b) being precisely "the gate fails today and is
|
||||
invoked by nothing" — and R-29, the class, says in terms *"do not mint a new ID for a new instance."*
|
||||
Spike 3 filed a duplicate.
|
||||
|
||||
**And the substantive half of that Spike 3 claim was wrong.** I wrote that the drift left the customer
|
||||
page's install-command generator "targeting a flag surface three minor versions stale." R-94 explicitly
|
||||
retracts exactly that reading: the constant selects no script — it renders as a text label, and the
|
||||
command beneath it fetches the script the website git-syncs from `main`, so **1.22.0 is what every
|
||||
install already gets**. It is a wrong number on your screen and nothing more. I overstated it.
|
||||
|
||||
## Still unknown
|
||||
|
||||
**The real stub has not been packaged** — that is the one thing between here and a build spec.
|
||||
Also unproven: `dpkg --configure -a` ordering for a package with dependencies; an ISO that never went
|
||||
through `prepare-iso` (blocked by R-155, which this spike was fenced from changing); and the Graphical
|
||||
installer, where the result should hold *a fortiori* since the `.deb` path is in `Install.pm`, shared
|
||||
by all front-ends — but that is inference, not measurement.
|
||||
|
||||
**Spike 3's Q3** — the real stub at `before-network` — **this session did not touch it.** Note it is
|
||||
now partly superseded: on the `.deb` route the unit's ordering comes from the unit file, not from
|
||||
`[first-boot].ordering`.
|
||||
|
||||
## R-rows
|
||||
|
||||
**None opened.** Each candidate was grepped against the register first: the delivery result is a
|
||||
positive finding, the postinst constraints belong in the build spec, and the repack guard is already
|
||||
R-155. **R-153 retracted** into R-94 leg (b) / R-29.
|
||||
|
||||
## Teardown
|
||||
|
||||
All three layers plus the scratch storage, verified positively. demo-hp: VMs 400/401 purged, **storage
|
||||
`spike4` removed** (`storage.cfg` back to 4, `grep -c spike4` = 0), `/mnt/nvme-1tb/images/` empty,
|
||||
**disk usage 6.6 G — identical to pre-spike**, probe ISO and driver removed, 0 loop devices,
|
||||
`drill-r50` stopped and untouched, 9201 running, `felhom-backup` unmodified, nothing on `local-lvm`.
|
||||
DooPlex: workspace scratch **4.8 GB removed**, scratchpad **3.3 GB → 88 K**, both throwaway passwords
|
||||
destroyed, 17 ISOs in `out/` untouched, no production file modified. **Hub-side: nothing created** —
|
||||
the VMs took LAN DHCP leases but never ran `felhom-host-install.sh` or contacted the hub; verified by
|
||||
fetching and searching the customer list. Nothing published.
|
||||
@@ -1,168 +1,173 @@
|
||||
# REPORT — `STATUS.md` created, and the 2026-08-02 operator decisions recorded (2026-08-02)
|
||||
# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes
|
||||
|
||||
**Overwritten** per the standing rule. The prior contents (hub v0.85.0 Network card + v0.86.0 Copy
|
||||
without reveal, 2026-07-31) have their durable record in `hub/CHANGELOG.md` and
|
||||
`documentation/audits/host-addresses-visible-2026-07-31.md`; nothing was lost by this overwrite.
|
||||
**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built
|
||||
and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the
|
||||
box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` /
|
||||
`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at
|
||||
source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**.
|
||||
|
||||
**Class: documentation only.** No code, no template, no box, no build. Repo `felhom.eu` only —
|
||||
`app-catalog-felhom.eu` was read for context and **not** modified. **No `CHANGELOG.md` entry exists
|
||||
for this change and none is missing:** this repo has no root changelog, only per-area `hub/`,
|
||||
`scripts/`, `website/` (`CLAUDE.md`), and this session touched none of those areas.
|
||||
|
||||
Baselines: `felhom.eu` @ `260a8f6`, `app-catalog-felhom.eu` @ `fd7747d`. Part 1 was derived by reading
|
||||
the register's rows and both ranking sections, not from memory.
|
||||
**Outcome: four scheduled runs, all passed, nothing triggered by hand.**
|
||||
|
||||
---
|
||||
|
||||
## 1. `STATUS.md` — the file
|
||||
## 1. Preconditions
|
||||
|
||||
Root of `felhom.eu`, so it is the first thing visible. Sections in the specified order: what works ·
|
||||
what's broken · what we're working on · waiting on you · changed since last update.
|
||||
|
||||
**Word count: 652 total, 581 excluding the header block** (`wc -w`; the header carries the
|
||||
view-not-source, not-`CONTEXT.md` and maintenance rules, which the spec requires). **That is over the
|
||||
~500 target and it is a deliberate miss, stated rather than hidden.** Five passes took it from 819 to
|
||||
652. Getting under 500 needed either dropping a mandated item or dropping the off-site-credential line
|
||||
— the register's **top-ranked** open item and the largest customer-data exposure on it. Cutting the
|
||||
biggest data risk to save forty words is the wrong trade on a page whose job is to show the operator
|
||||
what is at stake. It is 67 lines and fits a screen. **If the operator disagrees, the line to cut is
|
||||
the R-95/R-87 one** and the page drops to ~545.
|
||||
|
||||
Content, in the operator's ranking: an app can stay off after a power cut, silently (R-157) · the
|
||||
off-site copy can be erased by the box that wrote it, and has never been restored from (R-95, R-87) ·
|
||||
three of fifty-three apps saved data where backups never looked (R-156) · 20 GB of backup space
|
||||
against 50 GB of apps (R-163) · when that trips, one page says so and nothing alerts (R-158) · the
|
||||
checker exists but a person has to remember it (R-161).
|
||||
|
||||
**Constraints honoured:** no `R-n` is the subject of any sentence — every identifier is a bracketed
|
||||
pointer at the end of a line; no file paths, function names or version numbers appear; every broken
|
||||
item is stated as what a customer or the operator would notice. Shipped, watching and
|
||||
blocked-on-a-predicate rows (R-159, R-160, R-162, R-164) are absent by design.
|
||||
|
||||
**One deviation, flagged per standing rule 4.** "Waiting on you" is specified as *decisions only*, and
|
||||
it carries one non-decision: **the hub password needs rotating** (R-132, owner Viktor). It is the only
|
||||
thing on the register waiting on the operator with a live credential consequence, and omitting it from
|
||||
the operator's own page to honour a section rule would be the letter over the point. It is labelled
|
||||
*"a job, not a decision"* so the section's shape is not quietly eroded.
|
||||
|
||||
## 2. The decisions — where each one went
|
||||
|
||||
All four are in **`CONTEXT.md` as standing ruling S-5**, labelled **D-a … D-d** as in the discussion
|
||||
and deliberately kept distinct from S-3's `D1…D6`. Open work is carried as backlog rows, per the
|
||||
existing convention — no new home was created for either.
|
||||
|
||||
| Decision | Recorded | Work |
|
||||
| # | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| **D-a** — merge the backup partition away (not resize) | `CONTEXT.md` S-5 | **R-165** (new) |
|
||||
| **D-b** — desired/observed app state, own store | `CONTEXT.md` S-5 | **R-166** (new, `BLOCKED`) |
|
||||
| **D-c** — storage monitoring + backup alerts | `CONTEXT.md` S-5 | **R-167** (new) |
|
||||
| **D-d** — only DooPlex and Peti's box are protected | `CONTEXT.md` S-5 | `runbooks/target-selection.md`, this session — no row; the decision *is* the change |
|
||||
| **Maintenance rule** (Part 3) | `STATUS.md` header **and** `CLAUDE.md` § End-of-session checklist | — |
|
||||
| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) |
|
||||
| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots |
|
||||
| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) |
|
||||
| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** |
|
||||
| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one |
|
||||
| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's |
|
||||
| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window |
|
||||
|
||||
Recorded verbatim inside D-b, because it is the decision's binding constraint: *losing the state store
|
||||
must never cause an app to be deleted, restarted wrongly, or reported healthy when it is not — the
|
||||
worst acceptable outcome is re-running a backup that already ran.* Its two "establish before speccing"
|
||||
items are carried on R-166 as the reason that row is `BLOCKED` rather than `READY`.
|
||||
## 2. The due verdicts before the run, quoted
|
||||
|
||||
**Deliverable 5 asks for "the five decisions".** Part 2 defines four (D-a … D-d); the fifth deliverable
|
||||
line is the Part-3 maintenance rule, and it is in the table above. Nothing else in the task reads as a
|
||||
fifth decision — flagged rather than invented.
|
||||
```
|
||||
demo-felhom tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst"
|
||||
reason: newest settled archive … has not been proven; nothing proven on this tier yet
|
||||
tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z"
|
||||
reason: … has not been proven (last proven archive was a different one)
|
||||
|
||||
**D-a's two conditions are recorded as conditions, not commentary:** it changes the disk layout so it
|
||||
must land **before any external install**, and it removes a wall that currently fails safely so
|
||||
**R-167 ships in the same step, never after**. R-165 restates both; R-167 names R-165 as the thing it
|
||||
gates.
|
||||
demo-hp tier=felhom-backup due=true archive="…2026_08_02-04_49_29.tar.zst"
|
||||
tier=felhom-pbs due=true archive="…2026-07-28T19:19:45Z"
|
||||
```
|
||||
|
||||
**None of D-a, D-b or D-c is implemented.** No controller, agent, installer or hub file was opened for
|
||||
editing.
|
||||
**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical.
|
||||
|
||||
## 3. R-163 re-framed, and R-156's papra referral resolved
|
||||
## 3. The runs — all four SCHEDULED, none triggered
|
||||
|
||||
**R-163 is re-framed, not closed** — as instructed. State went `WAITING-ON-OPERATOR — the ratio is a
|
||||
tier-sizing ruling` → `RE-FRAMED 2026-08-02 — open, no longer waiting on a ratio`; "Blocked on" went
|
||||
from `the operator's sizing decision` to a pointer at R-165; owner `operator` → `CC`. The cell now says
|
||||
the sizing **question is withdrawn rather than answered**, that the row survives as the record of the
|
||||
constraint until the merge lands, and that the original finding follows unchanged. The intake ranking
|
||||
(item 4) was updated with it, and records that **R-165 inherits R-163's rank and is the highest-ranked
|
||||
item that must land before any external install**.
|
||||
| box | tier | due at | archive | result |
|
||||
|---|---|---|---|---|
|
||||
| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 |
|
||||
| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 |
|
||||
| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 |
|
||||
| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 |
|
||||
|
||||
**R-156's papra referral is resolved.** The referral existed because moving a mount relocates live data
|
||||
out from under a running app; with papra deployed nowhere there is nothing to strand, so the cheaper
|
||||
leg — the template mounts `/app/app-data` — is takeable without waiting on upstream.
|
||||
**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The
|
||||
only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2,
|
||||
which start nothing.
|
||||
|
||||
**The provenance is recorded with the claim, because it decides the row.** The evidence is
|
||||
`docker ps -a` on **demo-hp's guest 9201** returning empty, **supplied with the task**; this session
|
||||
**did not re-measure** — it is documentation-only and every box was fenced. The recorded scope is
|
||||
honest about its edge: it covers the one guest papra was convicted on in Campaign 10, and **no other
|
||||
customer's guest was enumerated**, so the row instructs the task that edits the template to re-check
|
||||
first. Next action on the row is the catalog edit plus `catalog_gates.py`, explicitly not done here.
|
||||
**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of
|
||||
proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following
|
||||
evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That
|
||||
is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.
|
||||
|
||||
## 4. `target-selection.md` per D-d
|
||||
**The asymmetry worth keeping:** a host-tier restore is **83–109 s**; an offsite one **300–540 s**. The
|
||||
tier an ordinary recovery uses is also the cheapest to prove.
|
||||
|
||||
The rule at the top is now D-d: **two protected machines, everything else disposable**, with the
|
||||
correction stated as a correction — the earlier caution was costing sessions and pushing drills onto
|
||||
DooPlex. The tier table's Tier 2 row is DooPlex + Peti's cluster "and, by D-d, nothing else".
|
||||
## 4. What the runs left behind
|
||||
|
||||
**Two consequences the decision did not name, both handled visibly rather than silently:**
|
||||
**Persisted state — v3, naming the archive, the tier and what was verified:**
|
||||
|
||||
- **`ep0` + the Hetzner Storage Boxes.** D-d's protected list has two machines and ep0 is not one, so
|
||||
the page no longer calls it Tier 2. It is **not** thereby scratch: it holds the PBS-DR datastore and
|
||||
the restic copy of a real customer's data — the only off-premises copy that exists. Read the narrow
|
||||
way (not protected, but not wipeable), using the page's own *fences-name-acts* rule, and **flagged
|
||||
in the page for the operator to confirm explicitly.**
|
||||
- **The shared "do not re-point either backup target" fence** on the two demo boxes was **downgraded
|
||||
from a prohibition to a stated cost**, because D-d makes both boxes freely reinstallable, which
|
||||
spends that reference configuration just as thoroughly — keeping the fence would have left the page
|
||||
self-contradicting. The reason survives: know you are spending the regression reference, and put the
|
||||
box back.
|
||||
```json
|
||||
demo-felhom felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
|
||||
proven_at 2026-08-03T22:56:45Z}
|
||||
felhom-pbs → {archive …2026-07-28T04:49:43Z, tier pbs, proven_at 2026-08-04T05:04:21Z}
|
||||
demo-hp felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
|
||||
felhom-pbs → {archive …2026-07-28T19:19:45Z, tier pbs, proven_at 2026-08-04T06:10:39Z}
|
||||
```
|
||||
|
||||
Also corrected while in the file: the *fences-name-acts* example cited the fence this edit removed, and
|
||||
demo-hp's access line asserted "no baked SSH key" — which **R-129** records as measured false on
|
||||
2026-07-31. It now points at R-129 instead of sending the next session to the hub vault for a
|
||||
credential it may not need.
|
||||
**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's
|
||||
latest report holds **two** entries, one per tier; the `local` one can only have come from the
|
||||
persisted state, because the in-memory store held only that morning's offsite run:
|
||||
|
||||
## 5. The maintenance rule
|
||||
```
|
||||
demo-felhom tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
|
||||
tier=pbs pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z tested_at=2026-08-04T05:04:21Z
|
||||
demo-hp tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z
|
||||
```
|
||||
|
||||
In two places, as specified: the `STATUS.md` header block, and a new **`## End-of-session checklist`**
|
||||
in `CLAUDE.md` — which also gathers the couplings that were previously scattered (CHANGELOG + REPORT,
|
||||
REUSE, the capability map's own end-of-session line, S-1's architecture coupling) and closes with *a
|
||||
finding goes in `OPEN-ITEMS.md` first, never only in a report, an audit or `STATUS.md`*.
|
||||
**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now:
|
||||
|
||||
`CONTEXT.md` gained a header block stating why it and `STATUS.md` are separate — same subjects,
|
||||
different readers, and `STATUS.md` holds nothing of its own. `STATUS.md` says the same from its side.
|
||||
```
|
||||
tier=felhom-pbs due=false proven="…2026-07-28T04:49:43Z"
|
||||
reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
|
||||
```
|
||||
|
||||
## 6. What could not be translated into plain language
|
||||
…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h.
|
||||
That is not a defect: it is "proved daily, on its own archive", visible one day later.
|
||||
|
||||
Asked for explicitly, because an untranslatable row usually means the row itself is unclear.
|
||||
## 5. Teardown — three layers, per box
|
||||
|
||||
- **R-29** — *"gates are enforced nowhere"*. The class is stateable ("we have checks nobody runs"), but
|
||||
its instances are four differently-broken scripts across two repos with no shared consequence, so
|
||||
every plain sentence either says nothing or misstates one instance. **R-161 is its readable
|
||||
fragment**, which is why R-161 is on the page and R-29 is not.
|
||||
- **R-123 / R-125** — process findings about how the register and how tests are written. Real, and they
|
||||
belong on the register; there is no customer-visible symptom to lead with, so they have no honest
|
||||
first sentence for this page. They are not "broken" in the operator's sense.
|
||||
- **R-133 (the plaintext break-glass credential)** — translatable, and left off only for space. It is
|
||||
the strongest candidate for the next update if something else closes.
|
||||
- **R-115 vs R-110** — separate rows, one plain-language paragraph. Merged into a single "Waiting on
|
||||
you" bullet carrying both pointers, because two adjacent bullets about publishing read as one item
|
||||
the operator has already half-decided.
|
||||
| layer | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| the machine | `pct list` → **0** entries for 990000 | **0** |
|
||||
| the host | `lvs` → **0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after |
|
||||
| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same |
|
||||
|
||||
**A register defect found while reading, filed here because the fix is not mine to guess: `R-133` is
|
||||
used TWICE** — `OPEN-ITEMS.md:80` (duplicate `domain` values accepted by the hub) and `:86` (the
|
||||
plaintext break-glass credential). Two different findings, one ID, both `READY`. One needs renumbering,
|
||||
and which one is the operator's call since both are cited from elsewhere (`CONTEXT.md` S-4 cites the
|
||||
credential one).
|
||||
Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.
|
||||
|
||||
## 7. Files changed
|
||||
## 6. What the run surfaced — three findings, two of them corrections to my own record
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `STATUS.md` | **new** — the operator page |
|
||||
| `CONTEXT.md` | S-5 (D-a … D-d); header note on the `STATUS.md` separation |
|
||||
| `documentation/backlog/OPEN-ITEMS.md` | R-165/166/167 filed; R-163 re-framed (state, blocked-on, owner, ranking); R-156's referral resolved with its provenance |
|
||||
| `documentation/runbooks/target-selection.md` | D-d rule; tier table; ep0; the demo-box fence; two stale lines |
|
||||
| `CLAUDE.md` | new `## End-of-session checklist`, carrying the `STATUS.md` maintenance rule |
|
||||
### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning
|
||||
|
||||
Nothing was built, deployed, published or touched on any host. Every claim about the register above is
|
||||
a claim about pushed source in this repo at the commit below.
|
||||
A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on
|
||||
2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by
|
||||
~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall
|
||||
(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry.
|
||||
Correlated but not established: guest 9201 was reprovisioned at 09:15–09:19, nine minutes before.
|
||||
**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**,
|
||||
and v0.123.0's probe detects the *state* but says nothing about the *transition*.
|
||||
|
||||
### R-191 (new) — every weekly offsite backup reports FAILED although it worked
|
||||
|
||||
demo-felhom, 06:49–06:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused
|
||||
incrementally) and the job then failed on the prune —
|
||||
`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job
|
||||
errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed**
|
||||
(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm
|
||||
the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that
|
||||
must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary
|
||||
— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here**
|
||||
(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two
|
||||
namespaces.
|
||||
|
||||
### Two corrections to yesterday's record
|
||||
|
||||
1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd
|
||||
**six times** (09:24 → 17:34) on that storage and privilege, and the hub raised
|
||||
`whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation
|
||||
said backups kept landing because writes go through a root path — wrong, and now corrected in the
|
||||
runbook and on the row.
|
||||
2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route;
|
||||
I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none
|
||||
since the R-86 work — and one **more** since, below.
|
||||
|
||||
**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at
|
||||
2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the
|
||||
host tier had never been proven, and the hub said so on its own, without being asked. It has not
|
||||
re-fired; the signal is edge-triggered, so the return to healthy is silent by design.
|
||||
|
||||
## 7. The capability map
|
||||
|
||||
The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo
|
||||
boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the
|
||||
teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box
|
||||
is untested and untouched, and one box proving something does not make it a fleet property.
|
||||
|
||||
## 8. Registers
|
||||
|
||||
- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the
|
||||
correction in §6.
|
||||
- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was
|
||||
the highest in use).
|
||||
- `ROADMAP.md` holds none of these rows, so nothing to collapse.
|
||||
- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under
|
||||
"What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.
|
||||
|
||||
## 9. Observations — noticed, NOT acted on
|
||||
|
||||
- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %,
|
||||
returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes
|
||||
warn about, and the offsite archive is the larger of the two. A materially bigger guest would want
|
||||
the restore pointed at `/mnt/nvme-1tb` instead.
|
||||
- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has
|
||||
none". Nothing depends on that assumption now, but the operations notes still carry it.
|
||||
- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest
|
||||
role — a 2.7× difference worth understanding before either is used to size anything.
|
||||
- **Both waiters this session produced no output** despite the runs completing; the evidence was
|
||||
gathered by direct query afterwards. A watcher that silently produces nothing is exactly the
|
||||
instrument class this project distrusts — the conclusions here rest on the boxes' own journals and
|
||||
the hub's database, not on the waiters.
|
||||
|
||||
@@ -104,7 +104,7 @@
|
||||
| `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
|
||||
| `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). |
|
||||
| `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
|
||||
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. |
|
||||
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). |
|
||||
| `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
|
||||
| `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
|
||||
| `cfg.DRTier` + offsite coupling | hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field `dr_tier` (formBool helper). |
|
||||
@@ -189,7 +189,12 @@
|
||||
|
||||
## 5. Extension points (where new features plug in)
|
||||
|
||||
- **New event type**: add to `allowedEventTypes` (hub/internal/api/handler.go ~L1063) **and** `customerMessages` (hub/internal/notify/templates.go) **and** the customer-prefs default list if customer-notifiable. Missing the first = controller POST 400s (the known gotcha).
|
||||
- **New event type — THREE registers, and which ones depend on the AUDIENCE.** Always: `allowedEventTypes` (hub/internal/api/handler.go) — missing it means the controller's POST 400s and the event vanishes (the known gotcha). Then decide the audience and stop guessing from the other registers:
|
||||
- **Operator-only** → add to `notify.operatorOnlyEvents` (hub/internal/notify/dispatcher.go) and give it **no** `customerMessages` entry. **Allowlisting alone does NOT make a type operator-only** — `FormatCustomerEmail` treats a missing `customerMessages` entry as a *fallback to the raw message*, not a block, and the only customer gate is configuration. v0.78.0 asserted the opposite in a comment and shipped the defect (R-97c). Examples: `whole_guest_backup_failed`, `recovery_unit_capture_failed`.
|
||||
- **Customer-facing with a STATIC message** → add a `customerMessages` entry (hub/internal/notify/templates.go) and the controller's `settings.DefaultEnabledEvents` if it should be on by default.
|
||||
- **Customer-facing with a DYNAMIC message** (the producer builds Hungarian text carrying names/numbers) → deliberately **no** `customerMessages` entry: `FormatCustomerEmail` PREFERS the entry over the message, so adding one silently discards the specifics. Examples: `offbox_enlarge_blocked`, `disk_health_degraded`, and since v0.89.0 `disk_warning`/`disk_critical`.
|
||||
- Pin BOTH registers in ONE test (hub/internal/api/recovery_unit_event_test.go is the model) — fixing one and not the other is the realistic mistake, and `notify.IsOperatorOnly` exists so the api package can assert it.
|
||||
- **A type in these registers with no PRODUCER is inert.** `disk_warning`/`disk_critical` were allowlisted, copy'd, default-enabled and checkbox'd from early on, and nothing in any repo emitted them until controller v0.191.0 — grep for an emitter before assuming a type works.
|
||||
- **New monitor checker**: copy hub/internal/monitor/staleness.go (§2 pattern); wire in hub/cmd/hub/main.go with an `EventNotifyFunc`; severity must be warning/error/critical to notify.
|
||||
- **New API route**: switch in `api.ServeHTTP` (handler.go ~L139); auth helper first line.
|
||||
- **New web page/action**: switch in `web.ServeHTTP` (server.go ~L182) — non-GET gets CSRF automatically; template into hub/internal/web/templates/ (embedded FS, parsed in `web.New`); new helpers into the funcMap (server.go ~L67).
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# STATUS — what works, what's broken, what's next
|
||||
|
||||
**Updated 2026-08-02.**
|
||||
**Updated 2026-08-03.**
|
||||
|
||||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
||||
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
||||
@@ -14,53 +14,76 @@ A blank machine boots the Felhom disc, installs itself unattended, and is claime
|
||||
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
||||
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
||||
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
||||
restore files and app data from the drive alone. Proven end to end on real hardware.
|
||||
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
|
||||
box six times, everything returned every time, and an app switched off deliberately stayed off.
|
||||
Proven end to end on real hardware.
|
||||
|
||||
## What's broken
|
||||
|
||||
**After a power cut, an app can stay switched off — and nothing says so.** The machine looks for apps
|
||||
that didn't come back, but looks too early and never again. In one case it decides the customer
|
||||
switched it off deliberately, so it isn't even counted as down. *(R-157)*
|
||||
- **Rebuilding a machine silently takes away its off-site app-data backup.** `demo-hp` was rebuilt on
|
||||
3 August and came up without one, and stayed that way for a day. **Fixed on 4 August** — re-issued,
|
||||
the machine picked the new password up in 15 seconds and reattached to the same repository, and you
|
||||
escrowed the key. **The underlying fault is not fixed:** the off-site password is delivered exactly
|
||||
once and a rebuilt machine cannot ask for another, so this will happen again on the next rebuild.
|
||||
The other machine survived the same rebuild only because an unused password happened to be waiting
|
||||
for it. *(R-193)*
|
||||
- **The daily email about it tells you the wrong story**, and the automatic repair that exists for
|
||||
this declines without saying why. The message says the password was never applied; it was, on
|
||||
23 July, and worked for eleven days. *(R-192)*
|
||||
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
|
||||
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
|
||||
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
|
||||
setting per machine fixes it. *(R-191)*
|
||||
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap.
|
||||
*(R-95, R-87)*
|
||||
|
||||
**The off-site copy can be erased by the machine that made it** — the credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
||||
*(R-95, R-87)*
|
||||
## What shipped recently
|
||||
|
||||
**Three apps out of fifty-three kept their data where backups never looked.** They reported healthy;
|
||||
the data would vanish on the next update. Two are fixed, the third is now clear to fix because it is
|
||||
installed nowhere. *(R-156)*
|
||||
- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
|
||||
demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
|
||||
checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
|
||||
had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
|
||||
never been tested on either machine. Both also proved their off-site copy on the same night, one
|
||||
after the other rather than at once, which is the machine deciding for itself what to do first.
|
||||
*(closes the last open half of R-86/R-185)*
|
||||
- **A backup copy the machine was never allowed to read — and could not tell you about**, on both
|
||||
demo machines. The permission was one command; the silence was the real fault, and the machine now
|
||||
checks whether it may read each copy it depends on and says so when it may not. *(R-185)*
|
||||
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
||||
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
||||
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
||||
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
|
||||
written to disk with the result and survives. This was caught happening, not predicted: a real
|
||||
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
|
||||
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
|
||||
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
|
||||
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
|
||||
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
|
||||
|
||||
**Local backups get 20 GB while apps get 50 GB.** An app that outgrows the smaller space stops being
|
||||
backed up locally — and the off-site copy is made from the local one, so that stops too. Nothing is
|
||||
lost: the last good copy is kept intact. *(R-163)*
|
||||
|
||||
**When that happens, only one page says so** — no email, no alert. The page that answers "is this app
|
||||
backed up?" is the one that stays silent. *(R-158)*
|
||||
|
||||
**The check that catches this needs a person to remember it.** One command, run by hand; nothing
|
||||
refuses a change that skipped it. *(R-161)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** the last app whose data was never saved; today's decisions written down.
|
||||
- **Next:** merging the small backup partition into the large one, with the drive-filling warning and
|
||||
the backup-failure alert in the same step.
|
||||
- **After:** rebuilding how the machine records whether an app is meant to be running.
|
||||
- **Now:** nothing outstanding.
|
||||
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
||||
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
||||
*(R-87)*
|
||||
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **How a new version reaches a machine.** Pushing the installer publishes it — half a minute later
|
||||
every new machine downloads it, with no staging and no way back but another push. And publishing is
|
||||
a step we remember rather than one the release performs, forgotten twice: a fix can be live here
|
||||
and still not reach a new machine. Nothing is installing today, so this is the cheapest moment to
|
||||
settle both. *(R-110, R-115)*
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
||||
machines to install, only which ones exist. Closing that needs either a password given to the build
|
||||
server or a check inside the hub itself. *(R-184)*
|
||||
- **Nothing else.**
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-02** — Decided: the 20 GB backup partition goes away and shares space with app data. That
|
||||
changes the disk layout, so it happens before any machine is installed outside the house.
|
||||
- **2026-08-02** — Decided: only this machine and the tester's box are protected; every other box,
|
||||
both demo boxes included, may be broken or reinstalled freely.
|
||||
- **2026-08-02** — Two of the three apps that never saved their data are fixed; this page created.
|
||||
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
||||
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
||||
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
||||
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
|
||||
that could vanish on a restart (seen happening), a release that emailed a failure for a release
|
||||
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -53,6 +53,114 @@ Risk tags: **clean** · **needs-rework** · **hazard** (entangles a delete-targe
|
||||
|
||||
---
|
||||
|
||||
## 0a. App state: desired / in-flight / observed — S-1 CONTRACT (2026-08-02, decision D-b, R-166)
|
||||
|
||||
> **This section is a live contract, not migration history** — the rest of this document is the
|
||||
> v0.33 keep/port/delete inventory. Read this before touching `stacks/`, `backup/` or `bootrecon/`.
|
||||
> Shipped in controller **v0.189.0**.
|
||||
|
||||
An app's state is **three different kinds of information**, and conflating them is what produced
|
||||
R-157 mechanism B and F-CRIT-1. They are stored differently on purpose.
|
||||
|
||||
| Kind | Question it answers | Where it lives | Persisted? |
|
||||
|---|---|---|---|
|
||||
| **Desired** | *What did the customer ask for?* | `app.yaml` → `desired_state` | Yes, beside the app's other settings |
|
||||
| **In-flight** | *Is an operation part-way through, and did it finish?* | its **own** marker file under `<data_dir>` | Yes, written before the operation and cleared after |
|
||||
| **Observed** | *Is it running, unhealthy, restarting, is its drive gone?* | nowhere | **No — rebuilt by looking** |
|
||||
|
||||
**The rule that ties them together: never derive one from another.** The defect this replaced did
|
||||
exactly that — it derived *desired* from *observed* (zero containers ⇒ "the customer stopped it"),
|
||||
and zero containers is equally what a power cut mid-compose, an interrupted deploy and an interrupted
|
||||
backup leave behind. Two real faults were therefore read as deliberate stops and stranded silently.
|
||||
|
||||
### Desired — `app.yaml`, `desired_state`
|
||||
|
||||
Tri-state: `""` (unknown) · `"running"` · `"stopped"`.
|
||||
|
||||
- **ONE OWNER: the customer's own action.** Writers are the `/api/stacks/{name}/{action}` switch,
|
||||
`DeployStack`, `UpdateOptionalConfig`'s redeploy branch, and the `.fab` import. **`StartStack` and
|
||||
`StopStack` are NOT writers** — a census found 14 callers of which only 2 are the customer; the
|
||||
rest are quiesce, the backup volume dump, offbox reconstitution, app export/restore, the storage
|
||||
gate, migration and the boot reconciler. Intent recorded in the primitive would make a nightly
|
||||
backup indistinguishable from the customer pressing Stop.
|
||||
- **Written BEFORE the act; a failed write REFUSES the act.**
|
||||
- **Absent means UNKNOWN — never "running".** Every `app.yaml` predating v0.189.0 lacks the field,
|
||||
so consumers must fall back to the pre-v0.189.0 behaviour rather than assume. A running-only
|
||||
backfill converges the unambiguous cases; **`stopped` is never inferred, from any signal.**
|
||||
|
||||
### In-flight — a marker file, one per owner
|
||||
|
||||
Two exist and they are deliberately **separate files**: `quiesce-state.json` (the whole-guest backup
|
||||
window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app —
|
||||
`backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One
|
||||
file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's
|
||||
note is a stranded app by a different route.
|
||||
|
||||
- Written **before** the stop; cleared **only** after a restart that succeeded; a **failed** restart
|
||||
keeps the marker so the next startup retries.
|
||||
- **A `defer` is not the mechanism.** A SIGKILL runs no deferred function — established on live
|
||||
hardware by Campaign 8 fault 10, where what brought the stacks back was the marker read at startup.
|
||||
- Recovery runs at startup and **completes before** the boot reconciler is launched, so an app the
|
||||
marker explains is not also reported as an unexplained boot orphan.
|
||||
|
||||
### Observed — not persisted, by design
|
||||
|
||||
`aggregateState` walks **every container** of a stack and any unhealthy or mixed result wins, so a
|
||||
partly-dead app cannot read as healthy (F-CRIT-1's shape). This requirement is met here and must not
|
||||
be re-implemented downstream. Nothing about observed state is written to disk: a controller restart
|
||||
re-observes it within one refresh, whereas persisting it risks carrying a stale verdict across the
|
||||
very restart that fixed it (the same argument as `RestartingSince`).
|
||||
|
||||
### The binding safety rule (verbatim, from decision D-b)
|
||||
|
||||
> *Losing the state store must never cause an app to be deleted, restarted wrongly, or reported
|
||||
> healthy when it is not — the worst acceptable outcome is re-running a backup that already ran.*
|
||||
|
||||
Applied: a lost or corrupt marker means the app is not auto-restarted **by that mechanism**, which is
|
||||
the pre-v0.189.0 position, not a new hazard. A lost `app.yaml` already means the app is not deployed.
|
||||
**Nothing here may make an absent file more dangerous than a present one.**
|
||||
|
||||
### Boot recovery reads desired, and asks before it acts (v0.190.0)
|
||||
|
||||
**Both boot gates now read desired state.** `bootrecon.isBootOrphan` (the R-52 sweep) and
|
||||
`shouldRecreateOnBoot` (the drive-backed recreate gate) answer the same question — *did the customer
|
||||
want this running?* — with the same three-way table, absent falling back to the pre-v0.190.0
|
||||
container count in both. Their agreement is pinned from both sides against one fixture table, because
|
||||
an import cycle prevents testing them together. R-170 closed the last gate that still guessed.
|
||||
|
||||
**The sweep observes a SETTLED fleet, not a single early sample.** It samples (name, state, container
|
||||
count) every 5 s, calls the fleet settled after 3 identical samples, and sweeps **once**, at the end.
|
||||
The window ends on settled or a 50 s budget, and the log says which. Two constraints bound it:
|
||||
|
||||
- `bootReconcileSettle + budget + one DefaultRetryDelay` must stay inside `deadAppBootGrace`, or a
|
||||
successful recovery stops being silent. This is arithmetic, pinned by a test.
|
||||
- **each sample must REFRESH first.** `GetStacks()` is the Manager's in-memory map, refreshed by the
|
||||
scheduler on its own 10 s cadence; sampling it faster without refreshing lets "settled" mean "the
|
||||
cache did not update". Found by live validation, not review.
|
||||
|
||||
A recovery completing after the grace emits a `LATE RECOVERY` warning naming the apps. The grace is
|
||||
**never** widened to make a late recovery look silent.
|
||||
|
||||
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
|
||||
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
|
||||
answer it, and the last two only became reachable once the window widened past T+5 s:
|
||||
|
||||
| Holder | Why starting would be wrong |
|
||||
|---|---|
|
||||
| the drive-absent gate | compose creates bind sources wherever the mountpoint points — the guest rootfs |
|
||||
| a quiesce | a running app inside a snapshot meant to be clean-shutdown-consistent |
|
||||
| an in-flight app-data operation | restarting an app under its own tar |
|
||||
|
||||
The rule is **not new** — the API's `startGatedByMissingDrive` already refused a customer's start on
|
||||
an absent drive. The sweep bypassed it by calling `Manager.StartStack` directly, which is what R-171
|
||||
closed. Held apps are reported separately from "still down": they are not a fault the sweep failed to
|
||||
fix, and reporting them as one is a false alarm.
|
||||
|
||||
**Read this before adding a fourth caller of `Manager.StartStack`.** That method has no gate of its
|
||||
own; every caller that is not the customer must decide for itself whether the app may run.
|
||||
|
||||
---
|
||||
|
||||
## 1. v0.33 module inventory (package → purpose, key deps)
|
||||
|
||||
| Package | Purpose | Key internal deps |
|
||||
|
||||
@@ -132,6 +132,41 @@ executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by const
|
||||
(`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self**
|
||||
(INV Part D1).
|
||||
|
||||
### Lane 2's restore-test is scheduled PER ARCHIVE GENERATION (R-86, 2026-08-03)
|
||||
|
||||
**[CONTRACT, changed 2026-08-03 — agent v0.121.0 + hub v0.91.0.]** The scheduled restore-test used to
|
||||
fire on an interval started at daemon start. It no longer does. The rule is:
|
||||
|
||||
> Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h).
|
||||
> The tier is **DUE** when **A** exists and **A has not already been proven**.
|
||||
|
||||
So a tier is proved **once per archive**, on its own archive, and the proof follows the backup rather
|
||||
than the process's uptime:
|
||||
|
||||
| tier rhythm | what is proved, and when |
|
||||
|---|---|
|
||||
| daily (host tier) | yesterday's archive, once a day |
|
||||
| weekly (offsite tier) | last week's archive, once a week |
|
||||
| newborn (no archive yet) | nothing — **UNKNOWN, never a fault** |
|
||||
|
||||
**The trap in the obvious formulation, recorded so it is not reintroduced:** *"due when the newest
|
||||
archive is ≥ 24 h old"* is never true on a **daily** tier — a new archive resets the newest-archive
|
||||
age to zero long before it reaches the lag — so the literal reading silently switches restore-testing
|
||||
off for the tier that matters most.
|
||||
|
||||
What survives unchanged: the restore-test itself (restore → boot → verify → destroy the scratch), its
|
||||
journal and crash recovery, the scratch VMID band, the one-heavy-operation gate, proof credit only on
|
||||
success, and oldest-proven ordering, which is now the tie-break **between due tiers**. A ticker
|
||||
remains, but only as the **evaluation interval** (6 h by default, chosen from a measured cost: one
|
||||
due-check is 18 ms on a local dir storage and 392 ms on the PBS tier over the WAN).
|
||||
|
||||
**The hub's half is not optional.** `restoreProvenStaleAfter` was a flat 7 days derived from the very
|
||||
cadence this replaced, and a weekly tier proved weekly reaches a proof age of **exactly** one interval
|
||||
just before its next proof — 168 h against a 168 h window. It sat ON the line, so any ordinary delay
|
||||
tipped a healthy tier into a nightly alarm. The window is now per tier, from that tier's observed
|
||||
archive interval, floored at the old 7 days, capped at 12 days (strictly inside the two-week offsite
|
||||
retention), and falling back to the tier's declared rhythm when history is too short to observe one.
|
||||
|
||||
### Why the split is right, stated once
|
||||
|
||||
**[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about
|
||||
@@ -529,6 +564,102 @@ portable secrets in the unit. **That independence is bounded by app size**, and
|
||||
> Past that the unit cannot be captured, and the app falls back to Lane 2's operator-driven
|
||||
> whole-guest route.
|
||||
|
||||
**AS OF 2026-08-02 SOMETHING NOW WARNS, AND THE ALERTING IS PART OF THIS CONTRACT (R-167 / R-158,
|
||||
decision D-c; controller v0.191.x + hub v0.89.0).** The last sentence of this section used to end
|
||||
"nothing warns when an app crosses the line". Two signals now exist and both are PROVEN-LIVE:
|
||||
|
||||
- **To the CUSTOMER, before anything fails** — `internal/fillwatch` warns per FILESYSTEM (never per
|
||||
app: one full disk holding ten apps would fire ten times) on **whichever trips first, used ≥ 85% or
|
||||
free < 5 GiB**, critical at 95% / 2 GiB, clearing at 75% / 7 GiB. **Two terms, because a percentage
|
||||
alone lies at both ends of the range this section itself documents:** 85% of a 20 G `mp1` leaves
|
||||
3 G — less than one DB-backed app's unit — while 85% of a 4 TB drive leaves 600 G. It watches the
|
||||
app-data volume, the system-data volume **and** every registered drive, which the previous
|
||||
`health_degraded` signal did not. Edge-triggered against persisted state; the hub owns cooldown.
|
||||
- **To the OPERATOR, when a capture actually fails** — `recovery_unit_capture_failed`, per app, with
|
||||
the target filesystem's used/free bytes at the moment of failure, so the *why* needs no login. It is
|
||||
**operator-tier** (`notify.operatorOnlyEvents`) and deliberately not `backup_failed`: a customer can
|
||||
take no action on a capture failure.
|
||||
|
||||
### 7.5.1 — THE CEILING THIS SECTION DESCRIBES HAS BEEN REMOVED (2026-08-03, R-165 / decision D-a)
|
||||
|
||||
**Everything above describes the SPLIT layout, which is now the legacy shape.** A golden built by
|
||||
`build-golden.sh` **v3.0.0** ships **one** data volume; `mp1` does not exist. Both consumer paths are
|
||||
binds of subdirectories of it (variant **V-c**):
|
||||
|
||||
```
|
||||
mp0 -> /var/lib/felhom ├─ docker/ --bind--> /var/lib/docker
|
||||
└─ sys_drive/ --bind--> /mnt/sys_drive
|
||||
```
|
||||
|
||||
**So the size bound below no longer applies to a box built from that golden.** A driveless app's
|
||||
recovery unit is limited by the box's actual free space, not by a partition set at build time. The
|
||||
mismatch table above (`mp0` 50 G vs `mp1` 20 G) describes what a merged box no longer has.
|
||||
|
||||
**R-175, fixed here rather than left standing.** The bound below was stated as the fleet's and was
|
||||
**one box's**: it is derived from `mp1 = 20 G`, which is demo-hp exactly and never was demo-felhom
|
||||
(`mp0 200G / mp1 50G`, where the same arithmetic gives ≈ 49 GB / ≈ 24 GB), nor the golden (`16 G / 8 G`
|
||||
before provision grew them). **Read it as a function of `mp1`, and only for a box still on the split
|
||||
layout.** Measured: `audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1.
|
||||
|
||||
**What replaced the partition's second job — the reserve.** `mp1` was also a BULKHEAD: an overflow was
|
||||
refused per app with the last good unit byte-identical, and it **could not reach `/var/lib/docker`**,
|
||||
because that was a different filesystem. On a merged box it can. Decision **B2**, shipped in controller
|
||||
**v0.192.0**, is that bulkhead made deliberate — a two-term reserve (97% used or 1 GiB free) in
|
||||
`internal/fillwatch`'s shape, sitting beyond its critical band so the customer is always warned first.
|
||||
It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
|
||||
only destroy a different app's only local copy.
|
||||
|
||||
**WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).**
|
||||
The two are deliberately different mechanisms, because conflating them is how seven failures went
|
||||
missing on 2026-08-03 without leaving a trace.
|
||||
|
||||
| | Record | Notification |
|
||||
|---|---|---|
|
||||
| what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN |
|
||||
| when | every time, unconditionally | at the end of a run, **only if something failed** |
|
||||
| gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown |
|
||||
| where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox |
|
||||
|
||||
- **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that
|
||||
is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's
|
||||
REPORT freshness, independent of any mail the box sends. That check is load-bearing for this
|
||||
design; weakening it re-opens a silent-failure path.
|
||||
- **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it.
|
||||
Deciding not to tell someone is itself an event worth recording.
|
||||
- **Deliberate skips are not failures** and never appear in the digest — a disconnected or
|
||||
decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the
|
||||
operator stops reading.
|
||||
- **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within
|
||||
the hour, because someone pressing the button is actively trying to get a backup. The periodic
|
||||
capture sweep is capped by the ordinary hourly cooldown.
|
||||
|
||||
**THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a
|
||||
**per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately
|
||||
before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the
|
||||
volume dump and the recovery-unit capture**. Those three write under one per-app root
|
||||
(`backups/primary/<app>`), which is what makes one verdict able to cover them honestly.
|
||||
|
||||
- **What it guarantees.** A refused app has **nothing written for it in that run**, its previous unit
|
||||
is **byte-identical**, it is **not stopped**, nothing anywhere is deleted, and the operator gets
|
||||
**exactly one** alert naming the app, the term that bound and the disk figures.
|
||||
- **Two terms, two questions.** *Headroom*: is the filesystem already below the reserve? *Size*: would
|
||||
THIS app's write take it below? The size estimate is the app's previous `.sql` + `.tar` on disk;
|
||||
with no history the decision degrades to headroom alone, deliberately — otherwise the first backup
|
||||
is the one that can never happen.
|
||||
- **Why it is decided lazily and not once per run.** Space changes during a run: app A's dump can put
|
||||
app B under the reserve, so a verdict taken at run start reads a disk that no longer exists.
|
||||
- **Why it is never re-decided between an app's own legs.** That is precisely the shape v0.192.0 had —
|
||||
the two dump legs unguarded and only the capture refused — under which the reserve was consumed by
|
||||
the very write it exists to bound, and the refusal's *"the previous unit is untouched"* was measured
|
||||
false. Proven live on demo-hp 2026-08-03 (R-181), fixed the same day, and re-proven by filling the
|
||||
box for each of the two terms.
|
||||
- **It sits ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal
|
||||
decided inside it would already have bounced the app it is refusing to back up.
|
||||
|
||||
**Status caveat, deliberately explicit:** every box in the field that has not been reinstalled is still
|
||||
on the split layout and everything above still describes them exactly. This subsection describes what a
|
||||
box built from golden ≥ 0.192.0 gets. Both demo boxes were reinstalled from it on 2026-08-03 (R-178).
|
||||
|
||||
Two things are deliberately **not** recorded here. **The sizing ratio is the operator's ruling**
|
||||
(**R-163**) — this section states the constraint, not a number. And **the same-device placement is
|
||||
intended, not a defect**: a driveless app's unit sits on the same SSD as its volumes, but Tier-2's
|
||||
@@ -724,7 +855,7 @@ does **not** hold as written. → **R-108**
|
||||
| ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
|
||||
| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
|
||||
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
|
||||
| R-86 (open) | Restore-tests are interval-scheduled, not backup-aligned | a tier's proof cadence is unrelated to when its archives are written |
|
||||
| ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) |
|
||||
| R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof |
|
||||
|
||||
### 10.3 Divergences that are documented elsewhere and are not re-opened here
|
||||
@@ -790,7 +921,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
|
||||
| whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C |
|
||||
| corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 |
|
||||
| the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 |
|
||||
| unattended restore-test across tiers | **IMPLEMENTED** (rotation not observed across consecutive cadences) | `00-capability-map.md:41`; LIVE per-tier timestamps this session |
|
||||
| unattended restore-test across tiers | **IMPLEMENTED**; **per-archive due-ness PROVEN-LIVE 2026-08-03** (agent v0.121.0) | `00-capability-map.md:41`; the due verdict + a real offsite run on demo-felhom (§3, Lane 2's per-archive rule) |
|
||||
| guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s |
|
||||
| quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL |
|
||||
| break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` |
|
||||
|
||||
@@ -45,7 +45,10 @@ hub, guests and PBS are UTC — every timestamp below carries its zone.
|
||||
the hub SQLite DB was taken with `kubectl exec … cat /data/hub.db > hub.db.copy` at **17:40:00 UTC**
|
||||
(113,033,216 bytes) into the session scratchpad, and every hub-DB figure below comes from that copy.
|
||||
It is a hot copy of a live database; row counts and metadata are consistent enough for an inventory
|
||||
but are a snapshot of that instant, not a transactionally consistent dump. The copy is read with
|
||||
but are a snapshot of that instant, not a transactionally consistent dump. **Do not reuse this
|
||||
command as a recipe: since hub v0.88.0 the DB is in WAL mode (R-172), so `cat /data/hub.db` alone
|
||||
yields a copy that opens cleanly and silently omits the newest writes — the `-wal` must be copied
|
||||
beside it (`documentation/operations/nodes.md`).** The copy is read with
|
||||
`mode=ro`. No hub table was written.
|
||||
|
||||
**Secret hygiene.** No secret value, key, token, password or key fingerprint is reproduced in this
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
# DIAG — does the R-166 boot sweep start an app whose data drive is absent?
|
||||
|
||||
**Date:** 2026-08-02 · **Box:** demo `felhom-pve` guest 9201 · **Controller:** v0.189.0
|
||||
**Question raised by:** reading the v0.189.0 diff, **not** by an incident. Diagnosed before any fix
|
||||
was written (task §9.2, "diagnose before theorising").
|
||||
|
||||
## Verdict
|
||||
|
||||
**CONFIRMED — the sweep starts it.** Observed directly, twice in the same run (both attempts).
|
||||
|
||||
**With a qualification that changes the severity but not the fix:** on this box the resulting
|
||||
`docker compose up -d` **failed** and **nothing was written** to the wrong disk. The mechanism that
|
||||
prevented the write is a **filesystem-permission accident that no code owns**, and the harm that DID
|
||||
occur is a different one: a false dead-app alarm for an app the drive gate is deliberately holding.
|
||||
|
||||
## The reasoning under test
|
||||
|
||||
`bootrecon` imports `stacks` alone and has no storage awareness. Since v0.189.0 `isBootOrphan`
|
||||
returns true for `desired_state: running` + zero containers. The drive-absent gate stops apps with
|
||||
`compose down` (leaving exactly zero containers) and never touches `desired_state`, because it is not
|
||||
the customer. `Manager.StartStack` has no drive gate. Therefore the sweep should start an app whose
|
||||
drive is absent.
|
||||
|
||||
## Method
|
||||
|
||||
The drive is a real USB disk (`/dev/sdb`), bound under the stable parent at
|
||||
`/mnt/felhom-drives/hdd_1`. The gate's absence signal is the agent's `BoundUnderParent`
|
||||
(`planDriveGates`, `intermediary.go:226`), so the drive was made absent **by unmounting it**, not by
|
||||
editing controller state — the state edit would have proven a different thing.
|
||||
|
||||
**Run 1 was contaminated and is reported because it produced a mechanism.** Unmounting only the
|
||||
parent bind was not enough: **the agent re-binds it within ~60 s** while `/dev/sdb` is still mounted
|
||||
at `/mnt/hdd_1`. In that run the drive gate's startup reconcile re-attached the drive and restarted
|
||||
the apps at `17:18:47`, **one second before** `bootrecon` looked at `17:18:48` — which then logged
|
||||
`no boot-orphaned apps`. That is a race that happened to go the safe way, **not** a disproof. Had it
|
||||
been reported as one, the conclusion would have been wrong.
|
||||
|
||||
Run 2 therefore unmounted **both** `/mnt/felhom-drives/hdd_1` and `/mnt/hdd_1`, and held them
|
||||
unmounted against the agent's healing for the duration.
|
||||
|
||||
## Preconditions, all verified before the observation
|
||||
|
||||
| # | Precondition | Observed |
|
||||
|---|---|---|
|
||||
| 1 | drive absent | `mount \| grep -c hdd_1` → **0** |
|
||||
| 2 | app has zero containers | `docker ps -a --filter name=calibre` → **0** |
|
||||
| 3 | intent still says running | `desired_state: running` |
|
||||
| 4 | app still deployed | `deployed: true` |
|
||||
|
||||
The gate stopped the apps itself first, on its own signal:
|
||||
|
||||
```
|
||||
17:22:07 [WARN] [gate] drive ABSENT /mnt/felhom-drives/hdd_1 — stopped+blocked 2 app(s): [calibre-web immich]
|
||||
```
|
||||
|
||||
`immich` was set to `desired_state: stopped` beforehand to scope the blast radius to one app; the
|
||||
gate is per-drive, so it stops every app on that drive regardless.
|
||||
|
||||
## The observation
|
||||
|
||||
```
|
||||
17:22:29 [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded
|
||||
17:22:34 [INFO] [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] — up to 2 attempt(s)
|
||||
17:22:34 [INFO] [stacks] Starting stack: calibre-web
|
||||
17:22:35 [WARN] [bootrecon] attempt 1/2: start "calibre-web" failed after 0.4s: exit code 1
|
||||
17:23:05 [WARN] [bootrecon] attempt 2/2: start "calibre-web" failed after 0.4s: exit code 1
|
||||
17:23:05 [WARN] [bootrecon] gave up after 2 attempt(s): recovered=[] still down=[calibre-web] (the dead-app alarm now owns these)
|
||||
```
|
||||
|
||||
**The sweep selected a drive-absent app and called `StartStack` on it.** That is the confirmation.
|
||||
|
||||
## What stopped the write, and why it must not be relied on
|
||||
|
||||
```
|
||||
Error response from daemon: error while creating mount source path
|
||||
'/mnt/felhom-drives/hdd_1/userdata/media/books':
|
||||
mkdir /mnt/felhom-drives/hdd_1/userdata: permission denied
|
||||
```
|
||||
|
||||
With the drive unbound, `/mnt/felhom-drives/hdd_1` is an empty directory on the host's `pve-root`,
|
||||
and it is **host-root-owned**:
|
||||
|
||||
```
|
||||
/mnt/felhom-drives uid=0 gid=0 mode=755
|
||||
/mnt/felhom-drives/hdd_1 uid=0 gid=0 mode=755
|
||||
```
|
||||
|
||||
Guest 9201 is an **unprivileged** LXC, so its container root is uid 100000 and cannot `mkdir` there.
|
||||
Confirmed no write occurred: `find /mnt/felhom-drives/hdd_1/` returned the directory alone, and
|
||||
`df -h /` was unchanged at `25G used / 28%` before and after.
|
||||
|
||||
**This protection is accidental.** Nothing in the controller chose it, no test pins it, and it rests
|
||||
on two conditions that are not guaranteed and are not checked anywhere:
|
||||
|
||||
1. the guest is unprivileged (a privileged guest maps root→0 and the `mkdir` succeeds);
|
||||
2. the stable-parent mountpoint directory is root-owned. When the drive **is** bound, that same path
|
||||
shows `uid=100000 gid=100000` — i.e. guest-writable. Any code path or agent version that
|
||||
pre-creates the mountpoint with guest ownership removes the protection silently.
|
||||
|
||||
It is one `chown` away from being gone, and its removal would be invisible until data landed on the
|
||||
wrong disk. It is therefore **not** a reason to leave the sweep unguarded.
|
||||
|
||||
## The harm that DID occur
|
||||
|
||||
Independent of the write question, and real on every box:
|
||||
|
||||
- the sweep burns **both** attempts and 30 s of retry delay on an app that cannot start by design;
|
||||
- it then hands the app to the **dead-app alarm** — `still down=[calibre-web] (the dead-app alarm now
|
||||
owns these)` — producing a **false alarm about an app the drive gate is deliberately holding**,
|
||||
which is exactly the noise class R-97/F-A1 exist to prevent;
|
||||
- it leaves a `Created` container behind on each attempt.
|
||||
|
||||
Before v0.189.0 none of this happened: `isBootOrphan` required `len(Containers) > 0`, and a
|
||||
gate-stopped app has zero. **This is a regression introduced by v0.189.0.**
|
||||
|
||||
## A second asymmetry found while restoring the box
|
||||
|
||||
The **API** start path already refuses this correctly. Restoring `immich` through the endpoint the UI
|
||||
calls, while the path was still flagged disconnected, returned:
|
||||
|
||||
```
|
||||
{"ok":false,"error":"A(z) /mnt/felhom-drives/hdd_1 tárhely jelenleg nem elérhető —
|
||||
az alkalmazás nem indítható, amíg a meghajtó vissza nem csatlakozik."}
|
||||
```
|
||||
|
||||
That is `startGatedByMissingDrive` (`internal/api/router.go`). So the controller already holds the
|
||||
rule "do not start an app whose drive is missing" — it is enforced on the customer's path and
|
||||
**bypassed by the sweep**, which calls `Manager.StartStack` directly. The fix is to give the sweep
|
||||
the same question to ask, not to invent a new rule.
|
||||
|
||||
## Consequence for the task
|
||||
|
||||
**Part 3 applies and is implemented first**, before the sweep's window is widened — a wider window
|
||||
makes both the false alarm and the (currently accident-blocked) write hazard wider. Fail-safe
|
||||
direction per §8.4: **cannot determine drive liveness → do not start.** Not starting is recoverable
|
||||
(the gate's `Return` branch restarts the app when the drive comes back, and the alarm reports it
|
||||
meanwhile); starting on an absent drive is not recoverable by anything automatic.
|
||||
|
||||
Register row: **R-171**, marked as a regression from v0.189.0.
|
||||
@@ -0,0 +1,220 @@
|
||||
# SPIKE — a Gitea Actions runner for the felhom gate entry points (R-168, 2026-08-02)
|
||||
|
||||
**Verdict: the mechanism works and is SHIPPED.** All six probes answered; none produced a STOP.
|
||||
The runner is unprivileged, host-mode, one registration for all four repos, and a failed run now
|
||||
sends its own alarm — measured, not assumed.
|
||||
|
||||
**Context.** Session 1 (same day) gave every felhom repo one gate entry point and a
|
||||
`.githooks/pre-push` that runs it and refuses a failing push. That hook is per-clone and
|
||||
`git push --no-verify` skips it, so nothing independent of the person pushing ever saw whether the
|
||||
gates passed. This spike built the independent half.
|
||||
|
||||
---
|
||||
|
||||
## Arrival state, re-confirmed live (all matched the anchor)
|
||||
|
||||
| Fact | Expected | Measured |
|
||||
|---|---|---|
|
||||
| Gitea version | 1.26.2 | **1.26.2** |
|
||||
| Actions enabled | all five repos | **all five** (`repo_unit` type 10 present on each) |
|
||||
| Runners registered | 0 | **0** (`action_runner` empty) |
|
||||
| Workflow runs, ever | 0 | **0** (`action_run` empty) |
|
||||
| Branch protections | 0 | **0** (`protected_branch` empty) |
|
||||
| `.gitea/` directory | none | **none in any of the five** |
|
||||
| ArgoCD `gitea` app | `path: gitea-system`, auto-sync off | **confirmed**, Synced at `420e819`, Healthy |
|
||||
|
||||
Repo visibility, which decided P3's design: the four product repos are **public**;
|
||||
`homelab-manifests` is private.
|
||||
|
||||
> **A near-miss worth recording.** My first census query reported Actions enabled on only five
|
||||
> *unrelated* repos, which looked like a baseline drift big enough to change the task. It was a
|
||||
> `| tail -5` inside my own query helper truncating the result. The measurement was never wrong;
|
||||
> the instrument was. Re-run without the pipe, all five target repos had the Actions unit. **A tool
|
||||
> that silently drops rows is indistinguishable from a finding** — the same class as the
|
||||
> `go test -run` filter that matches nothing and prints `ok`.
|
||||
|
||||
---
|
||||
|
||||
## P1 — does a registered runner pick up a job at all?
|
||||
|
||||
**Method.** Registered one runner at **owner scope** for `admin` (`repo_id=0`), label `felhom-gates`,
|
||||
via a token minted with `gitea actions generate-runner-token --scope admin` and stored out-of-band as
|
||||
`Secret/act-runner-registration`. Pushed a temporary `.gitea/workflows/probe.yml` with one
|
||||
`run: echo` step to `felhom.eu`.
|
||||
|
||||
**Measured.** Run #1 appeared against the pushed commit, was claimed by `felhom-gates-runner`, and
|
||||
finished **success**:
|
||||
|
||||
```
|
||||
felhom-gates-runner(version:v0.6.1) received task 1 of job probe, be triggered by event: push
|
||||
P1-OK runner picked up the job
|
||||
🏁 Job succeeded
|
||||
```
|
||||
|
||||
**Ruling: PASS.** No STOP.
|
||||
|
||||
---
|
||||
|
||||
## P2 — can the stock image run our checks without a container runtime?
|
||||
|
||||
**Method.** Inspected `gitea/act_runner:0.6.1` directly, then re-measured *inside a job* — because
|
||||
what matters is what the **job** sees, not what the image contains, and in host mode those are the
|
||||
same thing only if host mode is really in effect.
|
||||
|
||||
**Measured.** Stock image (Alpine Linux v3.23):
|
||||
|
||||
```
|
||||
python3: sh: python3: not found ABSENT
|
||||
git: git version 2.52.0 PRESENT
|
||||
```
|
||||
|
||||
In-job, after building the minimal image:
|
||||
|
||||
```
|
||||
P2 python3: Python 3.12.13
|
||||
P2 git: git version 2.52.0
|
||||
```
|
||||
|
||||
**Ruling: the known branch, not a failure.** Host mode works; only `python3` was missing. Built
|
||||
`gitea.dooplex.hu/admin/felhom-act-runner:0.1.0` = stock (pinned) + `python3`, nothing else, from
|
||||
`homelab-manifests/gitea-system/act-runner/Dockerfile`. Verified by deleting the local copy and
|
||||
**re-pulling from the registry** rather than trusting the push's own output.
|
||||
|
||||
**Host mode was never in doubt, so the privileged/dind pattern was never reached for** (§5/§12 of the
|
||||
task, and the reason is written into `act-runner.yaml`: DooPlex is Tier 2 and *is* the recovery
|
||||
chain).
|
||||
|
||||
---
|
||||
|
||||
## P3 — can a workflow obtain the source without JavaScript actions?
|
||||
|
||||
**Method.** No `uses:` anywhere. A plain `run:` step clones from the **in-cluster** Gitea Service
|
||||
(`http://gitea.gitea-system.svc.cluster.local:3000`) — no ingress, no TLS hop, no geo rule — and
|
||||
checks out `$GITHUB_SHA`. The four product repos are public, so no credential is needed at all.
|
||||
|
||||
**Measured.**
|
||||
|
||||
```
|
||||
P3 pushed sha = bbd62319096a1fb92f72218a28756e6fa101e87b
|
||||
P3 checked-out sha = bbd62319096a1fb92f72218a28756e6fa101e87b
|
||||
P3-OK checkout equals pushed commit
|
||||
```
|
||||
|
||||
**Ruling: PASS.** The shipped workflows use the tighter form — `git init` + `git fetch --depth 1
|
||||
origin $GITHUB_SHA` + `checkout FETCH_HEAD` — which is both shallow and pinned to the **exact pushed
|
||||
commit** rather than the branch tip, so two racing pushes cannot make a run test the wrong tree. The
|
||||
full clone in the probe took ~48 s for `felhom.eu`; the shallow fetch is materially faster.
|
||||
|
||||
---
|
||||
|
||||
## P4 — does one registration serve all four repos?
|
||||
|
||||
**Method.** The registration is owner-scoped (`owner_id=1`, `repo_id=0`). Pushed the real workflow to
|
||||
all four repos and read which runner claimed each task.
|
||||
|
||||
**Measured.**
|
||||
|
||||
```
|
||||
task repo runner_id runner
|
||||
7 felhom.eu 2 felhom-gates-runner
|
||||
8 felhom-controller 2 felhom-gates-runner
|
||||
9 felhom-agent 2 felhom-gates-runner
|
||||
10 app-catalog-felhom.eu 2 felhom-gates-runner
|
||||
```
|
||||
|
||||
**Ruling: PASS.** One registration, four repos. No per-repo registration needed.
|
||||
|
||||
---
|
||||
|
||||
## P5 — does a failed run signal anything outside the UI? *(the probe that decided Part 4)*
|
||||
|
||||
**Method.** Broke a gate deliberately and pushed it **with `--no-verify`**, which is precisely the
|
||||
bypass CI exists to catch. Run #3 concluded `failure`. Then looked for any outbound signal in the
|
||||
ten minutes around it: Gitea pod logs filtered for mail/SMTP/notification activity, and the
|
||||
`notification` table.
|
||||
|
||||
**Measured — nothing left the machine.**
|
||||
|
||||
```
|
||||
gitea pod logs, mail/smtp/notif lines since the failure : (none)
|
||||
notification rows created in the last 10 minutes : 0
|
||||
```
|
||||
|
||||
Gitea's mailer is *configured and enabled* (`[mailer] ENABLED = true`, Gmail SMTP, FROM is the
|
||||
operator's own address) and the sole user `admin` is active with
|
||||
`email_notifications_preference = enabled` — so this is not a disabled-mailer artefact at the
|
||||
config level.
|
||||
|
||||
**Honest limit on this measurement.** I did not independently prove that Gitea's SMTP path can
|
||||
*deliver*, so "no mail" cannot be split with certainty between *"1.26.2 has no action-failure
|
||||
notification"* and *"the mailer is broken"*. That distinction does not change the design: the alarm
|
||||
is built on **Resend**, a different path entirely, and Scenario C proves that path end to end. It is
|
||||
recorded here so nobody later reads this probe as a clean bill of health for Gitea's mailer.
|
||||
|
||||
**Ruling: Part 4 applies — build the alarm.** A red tick in a web UI nobody watches is exactly the
|
||||
defect R-29 filed, rebuilt one layer up.
|
||||
|
||||
---
|
||||
|
||||
## P6 — does the runner need persistent state?
|
||||
|
||||
**Method.** Two measurements, not one, because "it survived a restart" and "the PVC is load-bearing"
|
||||
are different claims.
|
||||
|
||||
- **P6a — restart with the PVC intact:** deleted the pod, let it come back.
|
||||
- **P6b — restart with state lost:** removed `/data/.runner` (exactly what ephemeral storage would
|
||||
do), deleted the pod, let it come back.
|
||||
|
||||
**Measured.**
|
||||
|
||||
| | runner list before | after | runner log |
|
||||
|---|---|---|---|
|
||||
| P6a | `1 felhom-gates-runner` | `1 felhom-gates-runner` | no re-registration; straight to `Starting runner daemon` |
|
||||
| P6b | `1 felhom-gates-runner` | `1 …` **and** `2 …` | `Registering runner…` → `Runner registered successfully` |
|
||||
|
||||
**Ruling: the PVC is load-bearing.** Without persistence, every restart mints a new registration and
|
||||
leaves the previous one behind as a permanently-offline record — the runner list would silently fill
|
||||
with corpses. Shipped with a 5 Gi Longhorn PVC (largest repo checked out is ~125 MiB).
|
||||
|
||||
**The P6b casualty was cleaned up immediately**, not left for the teardown section:
|
||||
`DELETE /api/v1/admin/actions/runners/1 → 204`, leaving exactly one runner.
|
||||
|
||||
---
|
||||
|
||||
## What the probes changed about the design
|
||||
|
||||
1. **A custom image** (P2) — stock + `python3` only, base pinned by tag.
|
||||
2. **Shallow fetch of the exact SHA** rather than a clone of the branch tip (P3).
|
||||
3. **A PVC** (P6), with the reason recorded in the manifest.
|
||||
4. **A self-sent alarm** (P5) — the task's whole second half.
|
||||
5. **A sibling clone for two of the four repos** — discovered while writing the workflows, not by a
|
||||
probe. `controller_gates.py` and `agent_gates.py` invoke the shared `reuse_refs_check.py` that
|
||||
lives in the `felhom.eu` clone next door and is deliberately never copied, and both repos'
|
||||
`REUSE.md` files cite a path that lives in the hub. Without the sibling, CI would have failed
|
||||
**closed** — correctly, but for the wrong reason. CI now reproduces the workspace's sibling
|
||||
layout, and the resulting tallies match the local hook exactly (controller 126 exact / 6 suffix /
|
||||
1 cross-repo; agent 88 / 1 / 1). **CI and the hook agree.**
|
||||
|
||||
## Two things that failed on the way, and why they are recorded
|
||||
|
||||
Both were caught because the step failed **loudly**; either would have shipped as a silent
|
||||
non-alarm if the step had swallowed its exit code.
|
||||
|
||||
- **`curl: command not found`** — the first alarm used `curl`, which the deliberately minimal image
|
||||
does not carry. Fixed by using `python3` + `urllib` rather than by growing the image: reaching for
|
||||
a bigger base to send one HTTP request is the wrong trade, and every added tool becomes something
|
||||
the next person assumes is load-bearing.
|
||||
- **Cloudflare 403, error 1010** — `api.resend.com` sits behind Cloudflare, which blocks the default
|
||||
`Python-urllib/3.x` User-Agent. **This failure looks exactly like an auth failure and is not one**,
|
||||
which is the reason it is written down: the next person to see a 403 from Resend should check the
|
||||
agent before rotating a key. Verified the fix from inside the runner image with a deliberately
|
||||
invalid payload — with a User-Agent set, Resend answers `422 missing_required_field`, i.e. the
|
||||
request reaches the API rather than the CDN.
|
||||
|
||||
## Standing limit, stated so it is not mistaken for something it is not
|
||||
|
||||
**CI here DETECTS. It does not BLOCK.** Every felhom repo pushes straight to `main` with no pull
|
||||
request, so there is no merge for a status check to gate. This is not a gap in the runner — there is
|
||||
no gate in the road. The refusing half is the local pre-push hook; this half notices when that hook
|
||||
was skipped. Making CI blocking requires branch protection and a pull-request workflow, which is a
|
||||
change to how the operator works and is **their** decision, not this task's.
|
||||
@@ -10,7 +10,7 @@
|
||||
**Class:** SPIKE (empirical validation; no product code). **Repos:** felhom.eu (this doc only);
|
||||
felhom-agent read-only for grounding (`internal/pbs/{client,pin}.go`, `configs/build-golden.sh`,
|
||||
`internal/hub/cloudflared.go`, `internal/escrow/identity.go`).
|
||||
**Probe ends:** `felhom-hetzner` = Hetzner CX23, Debian 13.4, public IPv4 `167.233.158.164`,
|
||||
**Probe ends:** `felhom-hetzner` = Hetzner CX23 (**note added 2026-08-03: rescaled to a CX33, 8 GB RAM — this spike records the machine as probed and its body is deliberately unchanged**), Debian 13.4, public IPv4 `167.233.158.164`,
|
||||
global IPv6 `2a01:4f8:...::/64` (throwaway — NOT the live jarrs.eu box) ⟷ **demo-felhom** =
|
||||
the real PVE 9.2.2 host on the operator's home line (One Hungary fixed cable, Budapest), driven
|
||||
over the existing LAN SSH path; the tunnel itself always dialed **out**.
|
||||
|
||||
@@ -0,0 +1,248 @@
|
||||
# SPIKE — measuring the `mp1` → `mp0` merge (R-165, decision D-a)
|
||||
|
||||
**Date:** 2026-08-02 · **Author:** Claude Code · **Status:** MEASUREMENT ONLY — **no layout changed**
|
||||
|
||||
> **Decision D-a is already taken** (`CONTEXT.md` S-5): the dedicated backup partition is merged away
|
||||
> rather than resized. This document measures **how**, never **whether**. It ends in a question for the
|
||||
> operator and a **STOP** — the merge itself is next session's supervised work.
|
||||
|
||||
## 0. What was and was not touched
|
||||
|
||||
Nothing was created, resized, moved or deleted. No golden was rebuilt, no guest config edited, no
|
||||
partition altered. Every figure below is a read of live state or of committed source. `ep0` and Peti's
|
||||
box were not contacted at all (`runbooks/target-selection.md`, decision D-d).
|
||||
|
||||
The monitoring D-a requires (**R-167**) shipped **before** this measurement, in controller v0.191.x and
|
||||
hub v0.89.0, and was proven on guest 9201. That ordering is the point: D-a's condition (2) says the
|
||||
monitoring lands with the merge and never after, and landing it first is strictly better.
|
||||
|
||||
---
|
||||
|
||||
## M1 — what is actually there
|
||||
|
||||
**`pct config 9201`, both demo hosts, read 2026-08-02.**
|
||||
|
||||
| | demo-felhom (N100) | demo-hp (t740) | golden default |
|
||||
|---|---|---|---|
|
||||
| `rootfs` | 32 G | 32 G | `OS_SIZE_GB=32` |
|
||||
| `mp0` `/var/lib/docker` | **200 G**, `backup=1` | **50 G**, `backup=1` | `GOLDEN_DOCKER_GB=16` |
|
||||
| `mp1` `/mnt/sys_drive` | **50 G**, `backup=1` | **20 G**, `backup=1` | `GOLDEN_SYSDATA_GB=8` |
|
||||
| `mp8` `/mnt/felhom-drives` | bind | bind | — |
|
||||
| `mp9` bootstrap | bind, `ro=1` | bind, `ro=1` | — |
|
||||
|
||||
**Occupancy (`df`, in-guest):**
|
||||
|
||||
| filesystem | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| `/` | 945 M / 32 G (4%) | 942 M / 32 G (4%) |
|
||||
| `/var/lib/docker` (`mp0`) | 13 G / 197 G (7%) | 5.4 G / 50 G (12%) |
|
||||
| `/mnt/sys_drive` (`mp1`) | **2.0 G / 50 G (5%)** | **92 M / 20 G (1%)** |
|
||||
| data drive | 21 G / 916 G (3%) — `hdd_1` | 6.7 G / 938 G (1%) — `nvme-1tb` |
|
||||
|
||||
**THE FIRST FINDING IS THAT "THE LAYOUT" IS NOT ONE THING, AND THE SPREAD IS WIDER THAN §7.5 RECORDS.**
|
||||
`architecture/07-backup-architecture.md` §7.5 states the default appliance as `mp0 50G / mp1 20G` —
|
||||
that is demo-hp exactly, and it is **not** demo-felhom, which ships **200 G / 50 G**, four and two and
|
||||
a half times larger. Both differ again from the golden's own `16 G / 8 G`, because `provision` grows
|
||||
the volumes after restore. **Any merge plan expressed as a fixed pair of numbers is already wrong for
|
||||
one of the two boxes that exist.**
|
||||
|
||||
**The corollary matters more than the numbers.** §7.5's headline bound — *"≈ 19 GB of app data for a
|
||||
file-only app, ≈ 10 GB for a DB-backed one"* — is derived from `mp1 = 20 G`. On demo-felhom, where
|
||||
`mp1` is 50 G, the real bound is ~49 GB / ~24 GB. **The architecture doc states one bound as if it
|
||||
were the fleet's, and it is one box's.** That is a documentation defect independent of the merge and
|
||||
is filed as its own row.
|
||||
|
||||
---
|
||||
|
||||
## M2 — what lives on `mp1` (it is not only backups)
|
||||
|
||||
**`du`, in-guest, read 2026-08-02.**
|
||||
|
||||
```
|
||||
demo-felhom /mnt/sys_drive/felhom-data 2.0 G
|
||||
├── backups/primary 269 M Tier-1 units of DRIVELESS apps (~30 apps)
|
||||
├── backups/secondary 1.7 G Tier-2 mirrors
|
||||
└── userdata/import 12 K the canonical drop-zone (R-75)
|
||||
|
||||
demo-hp /mnt/sys_drive/felhom-data 92 M
|
||||
├── backups/primary uptime-kuma, paperless
|
||||
├── backups/secondary paperless-ngx
|
||||
└── userdata/import paperless
|
||||
```
|
||||
|
||||
**Four distinct things would move, not one**, confirming the task's warning that a plan accounting
|
||||
only for the units is wrong:
|
||||
|
||||
1. **`backups/primary/<app>`** — the RETAINED Tier-1 recovery unit of every app with no data drive.
|
||||
Thirty apps on demo-felhom.
|
||||
2. **`backups/secondary/<app>`** — Tier-2 cross-drive mirrors, including the `_shares` pseudo-stack.
|
||||
On demo-felhom this is **1.7 G of the 2.0 G — the majority is Tier 2, not Tier 1.**
|
||||
3. **`userdata/import`** — the canonical, app-INDEPENDENT drop zone (`GetImportRoot`, R-75). It lives
|
||||
on the system drive **by contract**, not by convenience.
|
||||
4. **`userdata/`** more generally — the system-data userdata namespace for driveless apps.
|
||||
|
||||
**Observed occupancy is far below capacity on both boxes** (5% and 1%). The measured pressure today is
|
||||
zero; the constraint R-163 records is a *ceiling* problem, not a *current fill* problem. That is worth
|
||||
stating plainly because it bears on urgency, not on correctness.
|
||||
|
||||
---
|
||||
|
||||
## M3 — which merge shapes exist, and what each breaks
|
||||
|
||||
Three interacting assertions exist today, and a merge touches all three. **All three were read at
|
||||
source; none had been measured before.**
|
||||
|
||||
**(a) The golden build ASSERTS the split and ABORTS if it is absent.**
|
||||
`felhom-agent/configs/build-golden.sh:130`:
|
||||
|
||||
```
|
||||
findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . || {
|
||||
echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; }
|
||||
```
|
||||
|
||||
There is a sibling assertion for `mp0` at :126, and **two more** at :315-324 that abort if the vzdump
|
||||
log shows `excluding volume mount point mp0` or `mp1`. So the golden build fails closed on the split
|
||||
in **four** places, not one.
|
||||
|
||||
**(b) The whole-guest archive's scope is `rootfs + mp0 + mp1`,** and only because both carry
|
||||
`backup=1`. `build-golden.sh:69` records why: *"backup=1 is MANDATORY: without it vzdump EXCLUDES the
|
||||
volume (extra mountpoints default backup=0)"*.
|
||||
|
||||
**(c) `mount_parity` compares the ARCHIVE's `mpN` set against the RESTORED guest's**
|
||||
(`felhom-agent/internal/reconcile/restoretest.go:271-283`, `mountParity` at :347). Per slot it requires
|
||||
the same mount path and a restored size **not smaller** than the archive's, and it checks the reverse
|
||||
direction too. A mismatch fails the restore-test outright.
|
||||
|
||||
| shape | (a) golden assertion | (b) archive scope | (c) `mount_parity` | verdict |
|
||||
|---|---|---|---|---|
|
||||
| **S1 — one volume, two directories.** `mp1` stops existing; `/mnt/sys_drive` becomes a directory on the `mp0` filesystem | **BREAKS** — :130 aborts the build. Must be deleted, and :319's vzdump guard for `mp1` with it | **HOLDS** — scope becomes `rootfs + mp0`, still complete, because the data moved onto `mp0` | **HOLDS for new archives** (no `mp1` in archive ⇒ none required in restore). **A pre-merge archive restored into a merged guest is a different question — see below** | the shape D-a describes |
|
||||
| **S2 — two mounts, one backing pool.** `mp0` and `mp1` remain separate `mpN` slots on the same storage | **HOLDS** — both are still separate mounts | **HOLDS** unchanged | **HOLDS** unchanged | **does NOT remove the ceiling** — two filesystems still have two independent `df`s. This is thin-provisioning, not a merge, and it converts a clean per-app refusal into a shared-pool exhaustion that neither volume can see coming |
|
||||
| **S3 — grow `mp1`, keep the split** (**the shape D-a REJECTED — measured here as the baseline**) | **HOLDS** | **HOLDS** | **HOLDS** | zero structural risk, one `--sysdata-grow` value. It is *"the same wall further away"* — D-a's own words — and does not close R-163 |
|
||||
|
||||
**The one genuinely unmeasured item in M3, stated as unmeasured:** whether a **pre-merge archive**
|
||||
(carrying `mp1`) restores cleanly into the merged world. Reading `mountParity` says it should — the
|
||||
restore recreates `mp1` from the archive, so archive and restored guest agree, and parity passes. But
|
||||
**that was reasoned from source and not executed**, and this project's own record is that four
|
||||
production designs specced against unvalidated mechanisms were all wrong. **It is a one-command
|
||||
restore-test on a Tier-0 box and should be run before the merge, not after.**
|
||||
|
||||
---
|
||||
|
||||
## M4 — the bulkhead question (the important one)
|
||||
|
||||
**Today `mp1` is not only a ceiling; it is a BULKHEAD.** An app whose recovery unit outgrows the space
|
||||
is refused **per app**, its last good unit is preserved **byte-identical** (R-158's measurement), and
|
||||
— critically — **the overflow cannot reach `/var/lib/docker`**, because it is a different filesystem.
|
||||
The container runtime keeps running.
|
||||
|
||||
After S1 the same overflow lands on the filesystem Docker itself runs on. A runaway recovery-unit
|
||||
capture would fill `/var/lib/docker`, and a full Docker data-root is not a degraded state, it is a
|
||||
stopped one.
|
||||
|
||||
**This is the one place where "the merge is cheap" stops being true**, and it is why the warnings
|
||||
shipped first rather than alongside.
|
||||
|
||||
Four candidate replacements. **No choice is made here — this is the operator's ruling.**
|
||||
|
||||
| # | replacement | what it costs | what it leaves open |
|
||||
|---|---|---|---|
|
||||
| **B1 — a reserved block percentage on the merged filesystem.** `tune2fs -m` reserving N% for root; the controller runs unprivileged, so a capture cannot consume the reserve while root-owned Docker can | one `tune2fs` at build time; no code | the reserve protects *root*, not *Docker's runtime need* specifically; sizing it is a guess without a measured worst case |
|
||||
| **B2 — a refusal threshold in the capture path.** `captureAllRecoveryUnits` refuses when free space would drop below a floor, per app, and emits the alert R-158 just wired | small, local, testable; **reuses the alert that now exists** and restores the per-app-refusal semantics the bulkhead gave for free | the floor is a number needing justification; it protects the capture path only — a customer filling `mp0` through app data is untouched by it |
|
||||
| **B3 — a filesystem quota on the backup directory.** XFS project quota / ext4 project quota on `<mp0>/sys_drive/backups` | enforces at the filesystem, so **every** writer is bounded, not only the capture path | recreates a fixed ceiling — i.e. it is R-163 again inside one volume, and D-a's objection to *"a bigger number is the same wall further away"* applies to it word for word |
|
||||
| **B4 — R-167's warnings are deemed sufficient.** No hard stop; the customer is warned at 85% / 5 GiB and critically at 95% / 2 GiB | zero — it is already shipped and proven live | **a warning is not a bulkhead.** It depends on a human acting within the window, and the failure it fails to prevent is "Docker's data-root is full", which is the worst failure on the box |
|
||||
|
||||
**The measured input to that ruling:** on demo-felhom `mp1` holds 2.0 G against a 50 G ceiling, and
|
||||
`mp0` has 175 G free. The overflow scenario is not close today on either box. **B2 is the only option
|
||||
that preserves the property the bulkhead actually provided** — a per-app refusal with the last good
|
||||
unit intact — and it is the one that reuses what R-158 just built.
|
||||
|
||||
---
|
||||
|
||||
## M5 — existing boxes, and what a migration costs
|
||||
|
||||
**Read from the hub's own registers, 2026-08-02. No box was contacted.**
|
||||
|
||||
The hub's `/hosts` register holds **four** hosts, of which **two are ONLINE**:
|
||||
|
||||
| host | customer | agent | status |
|
||||
|---|---|---|---|
|
||||
| `demo-felhom-8363b5` | Demo Ügyfél | 0.119.0 | **ONLINE** |
|
||||
| `demo-hp-bb76ea` | Demo HP | 0.119.0 | **ONLINE** |
|
||||
| `drill-r50-0a4f9a` | drill-r50 | 0.113.0 | DOWN |
|
||||
| `sess-f-2670b5` | R-120 golden 0.186.0 proof | 0.116.0 | DOWN |
|
||||
|
||||
**The customer register lists five customers** — `david`, `demo-felhom`, `demo-hp`, `peti-felhom`,
|
||||
`sess-f`.
|
||||
|
||||
**THE FINDING THAT CHANGES THE COST: `peti-felhom` EXISTS AS A CUSTOMER BUT HAS NO HOST IN THE
|
||||
REGISTER.** Consistent with the long-standing *"guest not on agent node"* stop in the Peti-return
|
||||
runbook. So:
|
||||
|
||||
- **The two demo boxes are the entire measurable migrated population, and both are Tier 0 —
|
||||
disposable, per decision D-d.** For them a "migration" is not required at all: they can be
|
||||
**reinstalled** from a merged golden, which is cheaper and lower-risk than migrating in place, and
|
||||
D-d explicitly permits it.
|
||||
- **D-a's condition (1) — "it must land before any external install" — is currently SATISFIED.** No
|
||||
external box appears in the hub's host register. **This is the cheapest this decision will ever be,
|
||||
and the window is open now.**
|
||||
|
||||
**UNMEASURED, and reported as unmeasured rather than omitted:** Peti's box's actual disk layout. It
|
||||
does not report to the hub, so the hub holds no `pct config` for it, and it is protected by D-d and
|
||||
`target-selection.md` — so it was not contacted. **Whether a merged golden implies an in-place
|
||||
migration for that box, and what that costs, is not established by this spike.** Nor is whether the
|
||||
box is currently restorable at every point of such a migration. Both are inputs the operator has and
|
||||
this session does not.
|
||||
|
||||
**Also unmeasured:** the per-box in-place migration procedure itself (move `<mp1>/felhom-data` onto
|
||||
`mp0`, drop the `mp1` slot, verify) has not been executed even once on a throwaway guest. If an
|
||||
in-place migration is ever needed, **that rehearsal is the first thing to do, and its "is the box
|
||||
restorable at every point?" answer is currently unknown.**
|
||||
|
||||
---
|
||||
|
||||
## Ranked options, and a recommendation
|
||||
|
||||
**Ranked by what the measurements support, not by preference.**
|
||||
|
||||
1. **S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), shipped as a
|
||||
FRESH-INSTALL shape, with the demo boxes REINSTALLED rather than migrated.**
|
||||
It is what D-a describes; it genuinely removes the ceiling rather than moving it; the four golden
|
||||
assertions and the two vzdump guards are a bounded, greppable edit; the archive scope stays
|
||||
complete; and B2 restores the per-app refusal that is the bulkhead's real value, reusing the alert
|
||||
R-158 just wired. The migration cost for the measurable population is **zero**, because both boxes
|
||||
are Tier 0 and reinstallable.
|
||||
2. **S1 + B4 (warnings only).** Cheapest, and everything it needs is already shipped and proven live.
|
||||
Rejected as the *recommendation* only because the failure it declines to prevent — a full Docker
|
||||
data-root — is the worst one on the box, and it depends on a human acting inside the window.
|
||||
3. **S3 (grow `mp1`, keep the split).** Zero structural risk, one number. It is the measured baseline
|
||||
and D-a rejected it; recorded so the decision is compared against something.
|
||||
4. **S2 (two mounts, one pool).** Not recommended at all. It keeps both assertions satisfied while
|
||||
delivering none of the benefit, and converts a clean per-app refusal into a shared-pool exhaustion
|
||||
neither `df` can see coming — strictly worse than today.
|
||||
|
||||
**Two things to do BEFORE the merge session, both cheap, both currently unmeasured:**
|
||||
|
||||
- **Run one restore-test of a PRE-MERGE archive into a merged-layout guest on a Tier-0 box.** M3
|
||||
reasons it passes; nothing has executed it. This project's own record is that reasoning from source
|
||||
about an unvalidated mechanism has been wrong four times.
|
||||
- **Rehearse the in-place migration once on a throwaway guest**, and record whether the guest is
|
||||
restorable at every point of it. Only needed if Peti's box turns out to require migrating rather
|
||||
than reinstalling — which is the operator's information, not the hub's.
|
||||
|
||||
---
|
||||
|
||||
## THE QUESTION FOR THE OPERATOR
|
||||
|
||||
**Which shape, and what replaces the bulkhead?**
|
||||
|
||||
The recommendation is **S1 + B2**, fresh-install shape, demo boxes reinstalled. The two open inputs
|
||||
only the operator has are:
|
||||
|
||||
1. **Does Peti's box need an in-place migration, or can it be reinstalled?** The hub cannot answer
|
||||
this — the host is not in its register. It decides whether the migration rehearsal is required work
|
||||
or optional insurance.
|
||||
2. **Is B2's per-app refusal the right replacement for the bulkhead, or is B4 (warnings only)
|
||||
acceptable?** B2 costs a small amount of code and a justified floor; B4 costs nothing and is
|
||||
already live, but accepts that a full Docker data-root is reachable.
|
||||
|
||||
**STOP.** The merge is next session's work, and it is a supervised one.
|
||||
@@ -0,0 +1,183 @@
|
||||
# SPIKE R-165 Phase 0 — the two probes the merge spike left unmeasured
|
||||
|
||||
**Date:** 2026-08-03 · **Author:** Claude Code · **Status:** MEASURED — **no layout changed**
|
||||
|
||||
`SPIKE-r165-mp1-merge-2026-08-02.md` named two things as unmeasured and both are load-bearing. This
|
||||
document measures them. **It changes nothing**: no golden rebuilt, no box reinstalled, no guest config
|
||||
edited outside the throwaway probe guests, which are destroyed at the end.
|
||||
|
||||
**Host: `felhom-pve` (N100) — Tier 0.** demo-hp would have been the default per the 2026-07-25 ruling,
|
||||
but `target-selection.md` records its `local-lvm` as an **over-subscribed thin pool backing live guest
|
||||
9201** (~144 GiB allocated over ~54 GiB) where filling it corrupts every guest, and it holds **no
|
||||
container template**. felhom-pve has the template and 258 GiB free at 29% pool usage. Both are Tier 0;
|
||||
this picked the one where a probe cannot damage a live guest. **Neither DooPlex, `ep0`, nor the
|
||||
colleague's box was contacted at any point.**
|
||||
|
||||
---
|
||||
|
||||
## P1 — does a PRE-merge archive restore cleanly into the merged world?
|
||||
|
||||
**Verdict: PASS.**
|
||||
|
||||
### Method
|
||||
|
||||
The spike reasoned from `mountParity` that it should pass and said plainly that this had never been
|
||||
executed. It is now executed, on real hardware, with the real restore-test path — not a hand-assembled
|
||||
restore.
|
||||
|
||||
- **Archive:** `local:backup/vzdump-lxc-9201-2026_07_28-17_43_05.tar.zst` from **demo-hp**, 1.68 GB.
|
||||
Confirmed pre-merge by its own vzdump log rather than by assumption:
|
||||
|
||||
```
|
||||
including mount point rootfs ('/') in backup
|
||||
including mount point mp0 ('/var/lib/docker') in backup
|
||||
including mount point mp1 ('/mnt/sys_drive') in backup
|
||||
```
|
||||
|
||||
- **Command:** `felhom-agent --selftest=restore-test -archive <volid>` on demo-hp.
|
||||
|
||||
### Measured result
|
||||
|
||||
```
|
||||
"pass": true,
|
||||
"verified": "boot+running",
|
||||
"mount_parity": "ok",
|
||||
"duration_seconds": 84.2,
|
||||
"mount_inventory": [
|
||||
"mp0=/var/lib/docker (50G)",
|
||||
"mp1=/mnt/sys_drive (20G)",
|
||||
"mp8=/mnt/felhom-drives (throwaway for the archived bind)",
|
||||
"mp9=/etc/felhom-bootstrap (throwaway for the archived bind)"
|
||||
]
|
||||
=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m24s) ===
|
||||
```
|
||||
|
||||
`mountParity` was **not** relaxed, weakened or touched in any way.
|
||||
|
||||
### The limit of this result, stated rather than glossed
|
||||
|
||||
**It was run with the CURRENT agent (v0.119.0), because the merged agent does not exist yet** — Part 2
|
||||
sits after this session's STOP. What it proves is that `mountParity` compares the **archive** against
|
||||
**its own restore**, so a pre-merge archive recreates its own `mp0 + mp1` in the scratch guest and the
|
||||
two agree. That comparison never consults the *host's* golden layout, which is why the merge cannot
|
||||
invalidate it — **provided Part 2 honours its own constraint not to touch `mountParity` or the restore
|
||||
path** (§5, §12 of the task). It is an 84-second command and **should be re-run once the merged agent
|
||||
exists**, which is cheap and turns a sound inference into an observation.
|
||||
|
||||
---
|
||||
|
||||
## P2 — which S1 variant actually works on this platform?
|
||||
|
||||
**Verdict: all three probed variants are mechanically clean. They are separated by SCOPING, not by
|
||||
mechanics — and the deciding fact was not in the task's table.**
|
||||
|
||||
### Method
|
||||
|
||||
A throwaway unprivileged LXC per variant (`nesting=1,keyctl=1`, rootfs 8 G + one 10 G volume,
|
||||
`backup=1`), Docker installed from the same repo with the **same `daemon.json` the golden bakes**
|
||||
(`containerd-snapshotter: false`, overlay2, log caps). Per variant, measured at first boot and after
|
||||
**each of three reboots**:
|
||||
|
||||
1. both `/var/lib/docker` and `/mnt/sys_drive` present and **writable** (write → read back → delete,
|
||||
a positive observable rather than an `ls`);
|
||||
2. **one** filesystem — same source device **and** the same free-space figure for both paths;
|
||||
3. `dockerd` active and `docker run hello-world` succeeding;
|
||||
4. then once: the **real bootstrap propagation sequence** (`mount --rbind /mnt /mnt`,
|
||||
`mount --make-rshared /mnt`, `docker run -v /mnt:/mnt:rslave`);
|
||||
5. and: what a container mounting `/mnt:rslave` **actually sees** — the scoping check.
|
||||
|
||||
No pipe hides a non-zero; every check reports its own rc and the script aborts with `MEASURED-FAIL`.
|
||||
|
||||
### The variants
|
||||
|
||||
| | volume mounted at | then |
|
||||
|---|---|---|
|
||||
| **V-a** | `/var/lib/docker` | `/mnt/sys_drive` = bind of `/var/lib/docker/sys_drive` |
|
||||
| **V-b** | `/mnt/sys_drive` | `/var/lib/docker` = bind of `/mnt/sys_drive/docker` |
|
||||
| **V-c** | `/var/lib/felhom` *(neutral)* | **both** consumer paths are binds of subdirectories |
|
||||
|
||||
**V-c was not in the task's table.** It was probed *because* the measurements below showed V-a and V-b
|
||||
each violate a different documented invariant, and V-c is the shape that violates neither. It is
|
||||
offered as a **measured option for the operator at the STOP**, not adopted — the task is explicit that
|
||||
a variant is not chosen mid-session.
|
||||
|
||||
### Measured results
|
||||
|
||||
| check | V-a | V-b | V-c |
|
||||
|---|---|---|---|
|
||||
| both paths present + writable | **yes** | **yes** | **yes** |
|
||||
| ONE filesystem, ONE free-space figure | **yes** | **yes** | **yes** |
|
||||
| dockerd active + `docker run` — initial | **yes** | **yes** | **yes** |
|
||||
| dockerd active + `docker run` — reboots **1/2/3** | **3/3** | **3/3** | **3/3** |
|
||||
| `/mnt` propagation `shared`; container sees `/mnt` | **yes** | **yes** | **yes** |
|
||||
| both paths still real mountpoints (`findmnt` non-empty — the form the golden's assertions use) | **yes** | **yes** | **yes** |
|
||||
| container `statfs("/")` reports the merged volume | **yes** (10218772 KiB) | **yes** | **yes** |
|
||||
| **what a container mounting `/mnt:rslave` SEES** | `sys_drive` only — **8.0K** | `sys_drive` **+ `sys_drive/docker`** — **17.9M** | `sys_drive` only — **8.0K** |
|
||||
| customer data inside Docker's data-root | **YES** | no | no |
|
||||
|
||||
**The mechanical worry was misplaced.** The task flagged V-b's ordering risk — `/var/lib/docker` must
|
||||
be bound before dockerd starts. An `/etc/fstab` bind is ordered by `local-fs.target`, which precedes
|
||||
`basic.target` and therefore `docker.service`, and it held **3 reboots out of 3**. Ordering is not what
|
||||
separates these variants.
|
||||
|
||||
### What actually separates them
|
||||
|
||||
**V-b breaks a documented scoping invariant, and the measurement is the proof.** The controller
|
||||
container is started with `-v /mnt:/mnt:rslave`, and the bootstrap script's own comment states the
|
||||
scope it relies on:
|
||||
|
||||
> *"scoped to /mnt, which (Model A) holds only Felhom's felhom-data-namespace mounts, never the
|
||||
> customer's other on-drive data"*
|
||||
|
||||
Under V-b the container sees `/mnt/sys_drive/docker` — **Docker's entire data-root**, 17.9 MB on an
|
||||
empty probe box and growing with every image and every app volume. That sentence becomes false. The
|
||||
controller already holds the Docker socket, so this is **not a capability escalation** — but it puts
|
||||
Docker's internal tree inside the one path the controller's own scanners, the FileBrowser surface and
|
||||
the data-migration engine (which works *"in-process over the controller's `/mnt:/mnt:rslave` RW
|
||||
mount"*) treat as Felhom-only.
|
||||
|
||||
**V-a breaks the other one:** customer backups live inside Docker's data-root, so `du` on the data-root
|
||||
stops meaning what it says, and the ordinary operator reflex for a sick Docker — clear `/var/lib/docker`
|
||||
— destroys every local recovery unit on the box.
|
||||
|
||||
**V-c breaks neither**, at the cost of one new mount path and two fstab lines instead of one.
|
||||
|
||||
---
|
||||
|
||||
## P3 — the golden's four assertions
|
||||
|
||||
**Not yet run — it belongs to Part 1, which is after the STOP.** Recorded here so it is not lost:
|
||||
`build-golden.sh` fails closed on the split in **four** places (`:126`, `:130` separate-mount asserts;
|
||||
`:315`, `:319` vzdump-exclusion guards), and each retargeted guard must be shown to **abort** against a
|
||||
deliberately wrong shape before the golden is trusted.
|
||||
|
||||
**One measurement already de-risks it:** under all three variants both `/var/lib/docker` and
|
||||
`/mnt/sys_drive` remain **real mountpoints**, so `findmnt -no SOURCE,FSTYPE <path> | grep -q .` — the
|
||||
exact form the two existing assertions use — still returns non-empty. The assertions can be
|
||||
**retargeted with a changed message and an added guard for the single volume**, rather than rewritten
|
||||
from scratch. They must not be deleted.
|
||||
|
||||
---
|
||||
|
||||
## Teardown
|
||||
|
||||
| layer | action | evidence |
|
||||
|---|---|---|
|
||||
| **the machine** | probe guests `9401`, `9402`, `9403` destroyed; P1's scratch `990000` was torn down by the restore-test itself | `pct list` shows only `9201` |
|
||||
| **the host** | `local-lvm` **112398205 → 107204406 KiB** used (30.73% → 29.31%) — **5.19 GB actually returned**, not merely deallocated | `pvesm status` before/after |
|
||||
| **the hub** | **none created, and verified rather than assumed.** No probe claimed a box, minted a customer or registered an appliance — none ever ran a controller. The registers hold the same **5 customers** (`david`, `demo-felhom`, `demo-hp`, `peti-felhom`, `sess-f`) and same **4 hosts** (`demo-felhom-8363b5`, `demo-hp-bb76ea`, `drill-r50-0a4f9a`, `sess-f-2670b5`) as before Phase 0 | hub `/` + `/hosts` |
|
||||
|
||||
Scratch scripts and logs removed from `felhom-pve` (`/root/p2-probe.sh`, `/tmp/p2-*.log`,
|
||||
`/tmp/p1-restoretest.log`). `pct list` on both demo hosts shows only their own `9201`.
|
||||
|
||||
---
|
||||
|
||||
## What this changes about the merge
|
||||
|
||||
1. **P1 removes the restore risk from the decision.** A pre-merge archive restores clean with parity
|
||||
ok; it should be re-confirmed against the merged agent, which is one command.
|
||||
2. **The variant question is not "will it boot" — it is "which invariant do we break".** Both named
|
||||
variants work perfectly and each violates one documented guarantee. That is the operator's call and
|
||||
is the subject of this session's STOP.
|
||||
3. **V-c exists and is measured.** It costs one extra mount path and one extra fstab line, and is the
|
||||
only probed shape that keeps both guarantees.
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -116,13 +116,23 @@ root password vaulted in the hub**, `host_recovery` row `demo-hp-bb76ea` (set at
|
||||
|
||||
Retrieval (operator-side, and **shred the copy** — that DB holds every host's secret):
|
||||
|
||||
> **WAL-AWARE SINCE HUB v0.88.0 — copying `hub.db` ALONE is no longer safe.** The hub runs SQLite in
|
||||
> **WAL** mode (R-172), so a committed transaction may still live in `hub.db-wal` and not yet be in
|
||||
> the main file. A bare `cat /data/hub.db` therefore yields a copy that is **valid but stale** — it
|
||||
> opens cleanly and silently lacks the most recent writes, which is the worst failure shape for a
|
||||
> credential lookup. Copy the `-wal` beside it and let SQLite replay it on open.
|
||||
|
||||
```bash
|
||||
sudo kubectl -n felhom-system exec <hub-pod> -- cat /data/hub.db > /tmp/x.db
|
||||
sudo kubectl -n felhom-system exec <hub-pod> -- cat /data/hub.db > /tmp/x.db
|
||||
sudo kubectl -n felhom-system exec <hub-pod> -- cat /data/hub.db-wal > /tmp/x.db-wal 2>/dev/null || true
|
||||
python3 -c "import sqlite3;print(sqlite3.connect('/tmp/x.db').execute(
|
||||
\"SELECT secret FROM host_recovery WHERE host_id='demo-hp-bb76ea'\").fetchone()[0])"
|
||||
shred -u /tmp/x.db
|
||||
shred -u /tmp/x.db /tmp/x.db-wal
|
||||
```
|
||||
|
||||
The `|| true` is deliberate: an absent `-wal` is legitimate (a freshly checkpointed database), and
|
||||
must not fail the retrieval. **Shred both files** — the WAL holds the same secrets as the DB.
|
||||
|
||||
Then `sshpass -e ssh root@demo-hp` (sshpass is on DooPlex, not on the nodes).
|
||||
|
||||
**This is the lockout filed as R-61**: the ISO mints a throwaway root password per build and discards
|
||||
|
||||
@@ -3,7 +3,8 @@
|
||||
**Class:** supervised operational run. **No repo version bump** — the only commits are this record
|
||||
and the capacity note. **Nothing was deleted.**
|
||||
|
||||
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner CX23, Nuremberg.
|
||||
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner **CX33 (4 vCPU / 8 GB RAM)**, Nuremberg.
|
||||
> **Rescaled 2026-08-03** from the CX23 (2 vCPU / 3.8 GB) this runbook was written against. **The 40 GB local disk did NOT change** — this was a CPU/RAM resize — so every disk figure below still stands. The 4 GiB swapfile added on 2026-07-27 survived the resize.
|
||||
**Datastore moved:** `felhom-offsite`, `/srv/pbs-felhom` → **`/mnt/pbs-datastore`** (name unchanged).
|
||||
**Window:** 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC.
|
||||
|
||||
|
||||
@@ -229,7 +229,9 @@ as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as
|
||||
`keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
|
||||
minutes of ep0 CPU per weekly backup.
|
||||
|
||||
> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
|
||||
> Watch item (**superseded 2026-08-03**: ep0 is now a **CX33, 8 GB RAM**, and it HAS a 4 GiB swapfile
|
||||
> which survived the resize — so the pressure below is much reduced, though the shape of the concern
|
||||
> stands). As written: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
|
||||
> window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
|
||||
> `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.
|
||||
|
||||
@@ -307,7 +309,7 @@ Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13
|
||||
watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
|
||||
4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
|
||||
it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
|
||||
5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.
|
||||
5. ~~**ep0 has no swap** (3.7 GB CX23)~~ — **corrected 2026-08-03: ep0 is a CX33 with 8 GB RAM and an active 4 GiB swapfile.** See the volume runbook's OOM for the original incident.
|
||||
|
||||
## 11. Observations
|
||||
|
||||
|
||||
@@ -431,7 +431,7 @@ label. Filed under E-2.
|
||||
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
|
||||
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
|
||||
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
|
||||
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. |
|
||||
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. **CORRECTION 2026-08-03, same day, measured on the box: it DID surface as a 403, exactly as this item predicted — the earlier annotation here said otherwise and was wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST: `POST /nodes/demo-felhom/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)`. The hub raised `whole_guest_backup_failed` at the first one (*"retrying with backoff"*) and edge-triggering correctly suppressed the rest, so the operator was told once. It ALSO surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). |
|
||||
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
|
||||
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
|
||||
| 8 | The honest **single-drive label**. |
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
> firewall, and the hub-driven `felhom-peersync` reconcile surface. Re-running it on a fresh VM
|
||||
> re-creates the endpoint from nothing (that is the DR story, step 8).
|
||||
>
|
||||
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23, Debian 13,
|
||||
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23 **at the time — rescaled
|
||||
> to a CX33, 4 vCPU / 8 GB RAM, on 2026-08-03; the 40 GB local disk is unchanged**, Debian 13,
|
||||
> `167.233.158.164` / `2a01:4f8:1c16:7aa1::1`) with hub v0.32.0. The production endpoint is a
|
||||
> later re-run of this runbook on a production VM.
|
||||
>
|
||||
@@ -31,7 +32,7 @@ Parameters used throughout (adjust for a new endpoint):
|
||||
points at nothing (live-run finding). Home-resolver propagation can lag public DNS by
|
||||
minutes — a client-side `wg-quick up` that fails to resolve right after record creation
|
||||
just needs a retry.
|
||||
- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway CX23), not
|
||||
- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway box, **CX33 since 2026-08-03**), not
|
||||
any production box.
|
||||
|
||||
## 1. Base (on the box, as root)
|
||||
|
||||
@@ -29,20 +29,23 @@ prohibition as covering the act it names and nothing more.
|
||||
|---|---|---|
|
||||
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
|
||||
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
|
||||
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
|
||||
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **`ep0`** (operator ruling 2026-08-03) — and nothing else |
|
||||
|
||||
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
|
||||
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
|
||||
thing under test, the source of truth for it, and the backups, at once.
|
||||
|
||||
**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
|
||||
list names two machines and ep0 is not one of them. **That does not make them scratch, and the
|
||||
difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
|
||||
the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
|
||||
*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
|
||||
destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
|
||||
**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
|
||||
read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
|
||||
**`ep0` is Tier 2 — PROTECTED. Operator ruling, 2026-08-03.** D-d named two protected machines and did
|
||||
not name ep0 either way, so this page carried the question in writing for two days and read it the
|
||||
narrow way meanwhile (not protected, but not wipeable). The ruling settles it and **extends D-d's
|
||||
protected list to three machines**: DooPlex, Peti's cluster, ep0.
|
||||
|
||||
The reason it was never really in doubt: ep0 holds the **PBS-DR datastore and the restic copy of a
|
||||
real customer's data**, which is the only off-premises copy that exists. So *deleting datastores,
|
||||
prune jobs, tunnel config or nftables rules* was already forbidden by what it would destroy; the
|
||||
ruling makes the classification say so plainly instead of leaving each session to re-derive it.
|
||||
**Reads are fine** — including the ordinary off-site read a restore-test performs (R-86) — and it is
|
||||
never a drill target. The Hetzner Storage Boxes ride the same reasoning.
|
||||
|
||||
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
|
||||
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
|
||||
@@ -95,11 +98,15 @@ still shares a device with its guest, so a drive failure is **offsite-only recov
|
||||
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
|
||||
enrolled host. No access route from DooPlex, and nothing here needs one.
|
||||
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2, PROTECTED** (operator ruling 2026-08-03)
|
||||
|
||||
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
|
||||
path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune
|
||||
jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
|
||||
path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
|
||||
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching — the 8 GB
|
||||
is comfortable, not unbounded, and the OOM that started R-90 was a 14.46 GB restore read against
|
||||
3.8 GB. Do not delete datastores, prune jobs, tunnel config or nftables rules; never a drill target.
|
||||
**The ordinary off-site READ a restore-test performs is permitted and unchanged by the ruling**
|
||||
(R-86): the classification forbids destruction, not use. The Storage Boxes hold the restic copy —
|
||||
customer documents and photos, on a credential that can still delete (R-95).
|
||||
**Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
|
||||
produced a false "unreachable" verdict (standing rule 2).
|
||||
|
||||
@@ -178,7 +178,7 @@ turns a true alarm into one the operator dismisses.
|
||||
|
||||
### A comment asserting an invariant needs a test pinning it, or it is a wish
|
||||
|
||||
**Six instances in this project have shipped guarantees the code did not provide** — each survived
|
||||
**Seven instances in this project have shipped guarantees the code did not provide** — each survived
|
||||
review because the comment read as settled:
|
||||
|
||||
| # | Comment | What it claimed | What the code did |
|
||||
@@ -189,10 +189,14 @@ review because the comment read as settled:
|
||||
| 4 | `classifyRunStates` I1 | *"StateStopped means deliberately stopped by the user"* | quiesce stops stacks the same way — a failed restart was silent (F-CRIT-1) |
|
||||
| 5 | `inflight.go` | *"a caller that cannot acquire DEFERS"* | the backup caller recorded a failure and paged the operator (F-A1) |
|
||||
| 6 | `quiesce.go` | the agent's 409 *prevents* "a spurious failure" | on the start path it produced one (F-A1) |
|
||||
| 7 | `recovery_unit.go` B2 refusal (R-181) | *"the previous unit is untouched and NOTHING was deleted"* | *nothing deleted* held; **untouched was measured false** — the floor was checked ONLY in `captureAllRecoveryUnits`, while the two dump legs wrote the bulk into the same tree first and unguarded, so a 182,272 B tar became 2,147,666,432 B under a manifest that had not moved |
|
||||
|
||||
Two of these (4 and 5/6) were found by Campaign 8 **on live hardware**, not by review or unit tests
|
||||
— #4 had a green, red-proofed test suite over a production path that was broken two independent
|
||||
ways. So:
|
||||
Three of these (4, 5/6 and 7) were found **on live hardware**, not by review or unit tests — #4 had a
|
||||
green, red-proofed test suite over a production path that was broken two independent ways, and #7
|
||||
survived a full green suite plus three of its own red-proofs, because every one of them asserted the
|
||||
mechanism inside `captureAllRecoveryUnits` and none asserted the **consequence** across the whole
|
||||
backup run. The test that would have caught it is the one #7's fix ships: fingerprint the tree before
|
||||
and after, and compare. So:
|
||||
|
||||
- If a comment states an invariant, **name the test that pins it**, or write one.
|
||||
- If an invariant has a stated dependency (*"if either invariant changes, revisit this"*), that is
|
||||
|
||||
@@ -139,7 +139,7 @@ is not reachable, for two reasons that are each **already-recorded deliberate po
|
||||
`endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
|
||||
|
||||
So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is
|
||||
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90) and fenced by §3 — or *DR tier off* →
|
||||
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90 — **note added 2026-08-03: ep0 has since been rescaled to a CX33 with 8 GB RAM; this journal records what was true when it was written and is deliberately not revised**) and fenced by §3 — or *DR tier off* →
|
||||
no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits.
|
||||
|
||||
Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the
|
||||
|
||||
@@ -1,3 +1,264 @@
|
||||
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
|
||||
|
||||
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
|
||||
offsite tier holds two retained snapshots — `2026-07-27T19:55:41Z` and `2026-07-28T04:49:43Z` —
|
||||
**8 h 54 m apart**, because one is a healing artefact and the other a real weekly run. The mean-gap
|
||||
estimator therefore reads a **weekly** tier as nine-hourly: ×4 gives 36 h, the 7-day floor lifts it to
|
||||
168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm this whole
|
||||
task exists to prevent would have returned within a week, on the box it had just shipped to.**
|
||||
|
||||
`restoreProvenWindow` now takes `max(observed, declared)`. Observation refines a tier's rhythm
|
||||
**upward** and is ignored downward, which is right on its own terms and not merely cautious: a gap
|
||||
SHORTER than the declared rhythm is routine and means nothing — a retry, a manual run, a heal, a
|
||||
catch-up after an outage — while a gap LONGER than it is real information, saying this tier genuinely
|
||||
receives archives less often than the model assumes and its window must widen or it alarms.
|
||||
|
||||
**The cost, stated rather than hidden:** a tier that truly runs faster than its declared rhythm gets a
|
||||
wider window than it strictly needs, i.e. a slower `restore_test_stale` signal. That is the right
|
||||
direction for a signal whose message is *"unverified"*. *"Broken now"* is `restore_test_failed`, which
|
||||
is immediate and untouched.
|
||||
|
||||
Three live-derived cases added to `TestRestoreProvenWindow_Contract`, including the exact 8 h 54 m
|
||||
gap measured on the box; red-proved by restoring the tighten-too branch
|
||||
(`window(pbs, observed=8h54m) = 168h, want 288h`).
|
||||
|
||||
## v0.91.0 — a tier's staleness window learns the tier's own rhythm (2026-08-03, R-86 Part 2)
|
||||
|
||||
**This ships WITH the agent's v0.121.0, not after it.** The agent now proves a tier once per ARCHIVE
|
||||
GENERATION rather than on a 24h timer, so a tier backed up weekly is proved weekly — correctly, and
|
||||
in perfect health. `restoreProvenStaleAfter` was a flat 7 days, and its own comment derived that
|
||||
number from the cadence R-86 removes:
|
||||
|
||||
> *"the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
|
||||
> proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."*
|
||||
|
||||
A weekly tier proved weekly reaches a proof age of **exactly one interval** just before its next
|
||||
proof — 168h against a 168h window, false by a hair — so it did not merely sit near that line, it sat
|
||||
ON it, and any ordinary delay (a late backup, one deferral behind a running backup) tipped it into a
|
||||
nightly alarm about a system that is working. Shipping the agent alone would have converted the
|
||||
improvement into a false alarm.
|
||||
|
||||
**The window is now per tier**, `restoreProvenWindow(tier, observed, ok)`:
|
||||
|
||||
- **the tier's own archive interval**, observed from the host-reports the hub already holds —
|
||||
`pbs_snapshots[]` plus successful `backups[]` attributed by TARGET TYPE (the slice-A.4 rule: a
|
||||
PBS-targeted vzdump appears in both arrays, and classifying by array membership would attribute an
|
||||
offsite archive to the host tier);
|
||||
- **× 4 generations** — the settle generation plus ~3 missed opportunities, deliberately the same
|
||||
tolerance the flat constant expressed. The change is to the RHYTHM, not to the patience;
|
||||
- **floored at 7 days** — the old constant kept as a floor, so no tier is judged more tightly than
|
||||
before;
|
||||
- **capped at 12 days** — strictly inside the 2-week offsite retention with two days to spare, so a
|
||||
tier is never called stale against an archive PBS has already pruned;
|
||||
- **falling back to the DECLARED rhythm** when a box's history is too short to observe one:
|
||||
`backupStaleAfter` (26h) for the host tier and `offsiteBackupStaleAfter` (8d) for the offsite tier
|
||||
— the very thresholds the backup-freshness checker already judges those tiers against. A fresh box
|
||||
with one snapshot has no observable interval, and falling back to the FLOOR there would recreate
|
||||
the false alarm on exactly the tier this task is about.
|
||||
|
||||
**Kept, because both were earned:** absence is UNKNOWN until an anchored window has passed (R-81's
|
||||
structure, untouched), and the stale signal stays edge-triggered. `restore_test_failed` and
|
||||
`restore_test_stale` remain DISTINCT — one says your recovery is broken, the other says it is
|
||||
unverified, and the second is the one that quietly becomes the first.
|
||||
|
||||
**Every reason string now states the window it was judged against** — R-100's corollary: when a
|
||||
verdict changes what it counts from, the alarm text has to change with it, or an operator reads
|
||||
"limit 168h" under a tier actually judged at 288h and dismisses a true alarm.
|
||||
|
||||
**The window READ is unchanged in cost** (14 days), which is both enough to find proof inside the
|
||||
widest window and enough to see two generations of a weekly tier.
|
||||
|
||||
## v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (2026-08-03, R-182)
|
||||
|
||||
**Backfilled 2026-08-03 (R-86 session).** This version was built, deployed and recorded in `REPORT.md`
|
||||
and the R-182 row, but never given a CHANGELOG entry — and `REPORT.md` is overwritten every session,
|
||||
so the per-repo history under-reported what was actually running. The deployed image has been
|
||||
`felhom-hub:0.90.1` since `f21e7ca`.
|
||||
|
||||
Found by reading the first REAL digest, not by design: every app row ended with the same usage clause
|
||||
the mail already prints once on its own Filesystem line. On a two-app box that is untidy; down a list
|
||||
of a dozen it is the same forty characters twelve times, pushing the part that DIFFERS off a phone
|
||||
screen at 07:00 — the only moment that mail has to work.
|
||||
|
||||
The reserve's refusal message is authored for a single-app alert where naming the filesystem is
|
||||
right, so the message is unchanged and the DIGEST trims the duplicate when rendering.
|
||||
`trimRepeatedUsage` removes ONLY an exact "— <target path>:" suffix, so an unrelated reason is
|
||||
untouched and a reason that is nothing but the usage clause is left alone rather than emptied.
|
||||
|
||||
Also inverted the operator half of `TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer`: it
|
||||
required the operator to be e-mailed a per-app capture failure, which was correct when that event was
|
||||
the only signal and is wrong now that it is the RECORD and the digest is the notification. The
|
||||
customer-safety claim is unchanged — R-158's guarantee MOVED, it did not weaken.
|
||||
|
||||
## v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (2026-08-03, R-182)
|
||||
|
||||
**The smallest change on the board with the largest effect on trust: `processOperator`'s cooldown no
|
||||
longer returns bare.** It used to drop the event *before* `LogNotification`, so a suppressed operator
|
||||
alert and an event that never happened were indistinguishable — from the operator's side **and from
|
||||
the hub's own records**. Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two
|
||||
were mailed, and **seven left no row anywhere**. That is why the defect took a day to get the right
|
||||
way round: there was nothing to read.
|
||||
|
||||
A suppressed operator event now writes a `suppressed` row carrying the message and **the key that
|
||||
suppressed it**, so the collision is readable without reading code. This applies to **every** operator
|
||||
event, not only the one that exposed it. It deliberately does **not** change the cooldown's duration
|
||||
or semantics — it makes the drop visible, not absent.
|
||||
|
||||
**`backup_run_failures` — the per-run digest.** One operator mail at the end of a backup run listing
|
||||
every app that failed, its leg and its reason, with the counts and the target filesystem's free
|
||||
space. Added to `allowedEventTypes` **and** to `operatorOnlyEvents` — allowlisting alone does not make
|
||||
an event operator-only, and `FormatCustomerEmail` falls back to the raw English message rather than
|
||||
blocking. A test demonstrates a customer with the type in their enabled list receiving nothing.
|
||||
|
||||
**`recordOnlyEvents` — a third routing class.** Types that are STORED and RECORDED but never mailed.
|
||||
`recovery_unit_capture_failed` moves here: it is the durable per-failure record, and the digest is the
|
||||
notification. Deliberately a register rather than downgrading the severity to `info`, which would have
|
||||
had the same routing effect while relabelling a genuine failure as informational in the events table,
|
||||
the operator UI and every historical query.
|
||||
|
||||
**`cooldownRunSuffix` — the run discriminator.** A sibling of `cooldownTierSuffix` rather than a
|
||||
branch inside it, so `tier` keeps byte-identical semantics and R-97a's tests are untouched. It makes
|
||||
the cooldown effectively inert for the digest, **which is the intent**: a digest is already
|
||||
rate-limited by construction, one per run and only when something failed, so there is nothing for a
|
||||
timer to collapse — while the periodic refresh sweep sends **no** `run_id` and therefore stays under
|
||||
the ordinary hourly cooldown.
|
||||
|
||||
**The e-mail is rendered as a list, not a JSON blob** — the one operator mail with a variable-length
|
||||
payload, and a dozen apps on one line is unreadable on a phone at 07:00. An absent space reading
|
||||
renders as *unavailable*, never as zeros.
|
||||
|
||||
## v0.89.0 — the two halves of decision D-c (2026-08-02, R-167 · R-158)
|
||||
|
||||
**Decision D-c routes two new signals to two different audiences, and the hub is where that routing
|
||||
is enforced.** A customer can free space, delete files or add a drive, so a FILL WARNING is theirs. A
|
||||
customer can do nothing about a per-app backup capture failure, so it is not.
|
||||
|
||||
**New operator-only event type `recovery_unit_capture_failed`** (controller v0.191.0, R-158). Added to
|
||||
`allowedEventTypes` **and** to `notify.operatorOnlyEvents`. Deliberately NOT a reuse of
|
||||
`backup_failed`, which carries a `customerMessages` entry and sits in the controller's
|
||||
`DefaultEnabledEvents` — reusing it would email the customer, in Hungarian, about a failure they
|
||||
cannot act on. R-158's original proposal named `backup_failed`; D-c overrides it.
|
||||
|
||||
**`disk_warning` / `disk_critical` lose their generic `customerMessages` entries.** These two types
|
||||
were allowlisted here, carried Hungarian copy, sat in the controller's default enabled events and had
|
||||
a UI checkbox — and **nothing in any repo emitted them**, a complete customer pipeline with no
|
||||
producer. Controller v0.191.0 becomes that producer, and it sends a **dynamic** Hungarian message
|
||||
naming the drive and its free space. A static entry would be actively harmful: `FormatCustomerEmail`
|
||||
PREFERS the entry over the message, so keeping one would discard the label and the byte figures and
|
||||
leave the customer with *"A lemezterület 90% felett van"* — a warning with nothing to act on. Same
|
||||
reason `offbox_enlarge_blocked` and `disk_health_degraded` have no entry. The deletion itself is
|
||||
pinned by a test.
|
||||
|
||||
**New `notify.IsOperatorOnly`** — a read-only accessor so the `api` package can pin BOTH registers of
|
||||
a new event type in ONE test. Allowlisted-but-not-operator-only is invisible when the two are checked
|
||||
separately, and it is the defect v0.78.0 actually shipped. The register stays unexported so nothing
|
||||
can widen it at runtime.
|
||||
|
||||
**Tests:** 574 → **579**. The operator half is proven through the real dispatch path under the
|
||||
*breaking* configuration — the customer has the event enabled and has an email address — because that
|
||||
is the only configuration in which a missing `operatorOnlyEvents` entry is visible. Red-proof:
|
||||
removing the entry shows the customer being emailed, and the `skipped/operator_only` log row is
|
||||
asserted as a positive observable rather than inferred from an absent delivery.
|
||||
|
||||
## v0.88.0 — the WAL that never was (2026-08-02, R-172)
|
||||
|
||||
**The hub has never actually been in WAL mode.** `store.New` opened the database with
|
||||
`?_journal_mode=WAL&_busy_timeout=5000` — **mattn/go-sqlite3** syntax — while the driver is
|
||||
**modernc.org/sqlite**, whose `applyQueryParams` reads only `_pragma`, `_time_format`,
|
||||
`_time_integer_format`, `_txlock` and `_inttotime`. Everything else is **ignored without an error**.
|
||||
So the hub ran in the default rollback-journal mode with `busy_timeout=0` for its entire life, while
|
||||
its own source said otherwise — a configuration asserting an invariant the code did not provide.
|
||||
|
||||
**How it surfaced.** A false `HOST STALE` banner for `demo-felhom-8363b5` while the agent was up two
|
||||
days and reconciling normally. In rollback-journal mode a reader excludes a writer, so rendering an
|
||||
operator page can block a host report; the hub then returns **HTTP 500**, the agent logs
|
||||
`hub: report failed; keeping current interval` and **waits its full 15-minute interval**, and
|
||||
staleness fires at 30 minutes. **Two consecutive collisions = a false alarm + an operator e-mail.**
|
||||
Measured: 13 `SQLITE_BUSY` collisions in one pod lifetime, and the alarm fired twice that day
|
||||
(19:12:32 and 20:42:32 CEST) for a host that was never down.
|
||||
|
||||
**The observable that proved it:** a 128 MB `/data/hub.db` with **no `-wal`/`-shm` file beside it
|
||||
while the database was open**. In WAL mode those files must exist.
|
||||
|
||||
**The fix is one DSN, and each parameter earns its place:**
|
||||
|
||||
```
|
||||
?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate
|
||||
```
|
||||
|
||||
- `journal_mode(WAL)` — readers and one writer proceed concurrently, so a page render can no longer
|
||||
block a report. It is a property of the database FILE, so it persists once set.
|
||||
- `busy_timeout(5000)` — writers still serialise; without a timeout SQLite returns `SQLITE_BUSY`
|
||||
*immediately* rather than waiting.
|
||||
- `_txlock=immediate` — **the one that is easy to miss, and WAL + busy_timeout alone would not cover
|
||||
it.** `database/sql`'s `Begin()` is DEFERRED, so a transaction that reads then writes must upgrade
|
||||
its lock, and a failed upgrade is `SQLITE_BUSY_SNAPSHOT`, which **`busy_timeout` does not retry**.
|
||||
This store has 10+ `db.Begin()` sites and they are all write paths (customer delete/reset, wg,
|
||||
appliance, pbsdr, telemetry, log bundles). Without this the fix would leave a known un-retryable
|
||||
path open.
|
||||
|
||||
**Every test asserts what the DATABASE reports, never the DSN string** — a test on the string would
|
||||
have passed happily for the entire life of the bug. Five tests: the runtime pragma values; the
|
||||
`-wal`/`-shm` files existing beside an open DB (the production signature, pinned); a reader not
|
||||
blocking a writer (the consequence, not the mechanism); concurrent writers waiting instead of
|
||||
erroring; and racing read-then-write transactions. Plus `TestSQLiteDriverIgnoresMattnStyleParams`, a
|
||||
guard on the ROOT CAUSE: it fails if someone "tidies" the pragmas back to the familiar mattn form,
|
||||
and skips itself with instructions if a future driver starts honouring them.
|
||||
|
||||
**Red-proof:** restoring the shipped DSN reproduces the live failure exactly — `journal_mode = "delete"`,
|
||||
the `-wal` absent, and `a write FAILED while a read was open: database is locked (5) (SQLITE_BUSY)`.
|
||||
|
||||
**Operational consequence, handled rather than discovered later:** a WAL database cannot be copied by
|
||||
taking `hub.db` alone — a committed transaction may still be in `hub.db-wal`, so a bare `cat` yields
|
||||
a copy that opens cleanly and **silently omits the newest writes**. That is the worst shape for a
|
||||
credential lookup, and the break-glass retrieval in `documentation/operations/nodes.md` used exactly
|
||||
that command. Both it and the `_recovery-inventory` note are now WAL-aware (copy the `-wal`, shred
|
||||
both).
|
||||
|
||||
**Retries (options b and c in R-172) were NOT added.** With readers no longer blocking writers and
|
||||
the upgrade path covered, a `SQLITE_BUSY` reaching an HTTP handler should now be rare enough to be a
|
||||
real signal. If any appear after this, they mean something else and a retry would hide it. Revisit
|
||||
only on evidence.
|
||||
|
||||
## v0.87.0 — the Setup tab stops claiming a host-install version it cannot know (2026-08-02)
|
||||
|
||||
**R-94, all three legs, closed by deletion rather than derivation.** The customer page's Setup
|
||||
Command card read *"Day-0 host bootstrap for host-install **1.19.0**"*. The served script was
|
||||
**1.22.0**, and had been since 14 July — nineteen days of an operator-facing number that was simply
|
||||
wrong, with a version number's authority behind it.
|
||||
|
||||
**Why deriving it is not achievable honestly.** The Option-1 command downloads
|
||||
`felhom-host-install.sh` from the website **at run time**, and the website git-syncs `main` every
|
||||
thirty seconds (R-110). The hub therefore cannot know which version a given box will run — not at
|
||||
build time, not at render time. Any literal there is a guess. The comment that guarded the old const
|
||||
already half-admitted this ("Display-only… the served script is always current"). R-94(a) offered
|
||||
*derive it, or delete it*; deleting removes the drift class permanently instead of automating it.
|
||||
|
||||
**What changed.**
|
||||
- `internal/web/configs.go`: `const hostInstallVersion`, the `pageData.ScriptVersion` field and its
|
||||
assignment are **gone**. A NOTE stands in their place recording why there is deliberately no
|
||||
constant here, so the next person does not helpfully re-add one.
|
||||
- `internal/web/templates/customer_unified.html`: the sentence now says the command always fetches
|
||||
the **current** installer and renders no version at all.
|
||||
- `internal/web/render_test.go`: the assertion `strings.Contains(html, hostInstallVersion)` compared
|
||||
the constant to itself and **passed at any value** — demonstrated green with the const set to
|
||||
`9.9.9` while the served script was 1.22.0. Deleted, not replaced: there is no longer a version to
|
||||
assert. The `data-customer-id` and static-fallback assertions stay.
|
||||
- `scripts/hostinstall_gates.py` gate 1 **inverts**: it used to require the hub const to EQUAL
|
||||
`SCRIPT_VERSION`; it now asserts the hub carries **no host-install version literal at all**,
|
||||
matched in six code shapes across every `.go`/`.html` under `hub/`. Comments are deliberately not
|
||||
stripped — a `//` inside a URL string literal would truncate the scan and blind the gate — so the
|
||||
patterns match declarations, fields, assignments and the template action, never prose.
|
||||
- `scripts/felhom-host-install.sh`: **comment only**, `SCRIPT_VERSION` untouched. It claimed the gate
|
||||
keeps the hub copy equal, an invariant that no longer exists; a comment asserting an invariant the
|
||||
code does not provide is a wish.
|
||||
|
||||
**Red-proofs.** Restoring the const fails the rewritten gate 1 on three of its six shapes. The old
|
||||
`render_test.go` assertion passes with the const at `9.9.9`.
|
||||
|
||||
**Live verification:** endpoint-level (no browser on DooPlex) — the customer Setup tab is fetched and
|
||||
grepped for a version literal.
|
||||
|
||||
## v0.86.0 — Copy works without revealing, and every copy branch reports itself (2026-07-31)
|
||||
|
||||
**Found by the operator, in the way that matters: it cost a real login.** The v0.84.0 Console access
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
package api
|
||||
|
||||
import "testing"
|
||||
|
||||
// R-182 — the backup run digest needs the allowlist half, and NOT the customerMessages half.
|
||||
//
|
||||
// A new event type is a pair of register entries, and each half fails differently. For this type the
|
||||
// pair is unusual and that is the point:
|
||||
//
|
||||
// - missing from `allowedEventTypes` → POST /event returns 400 and the digest VANISHES,
|
||||
// which would rebuild the exact silence R-182 exists to end;
|
||||
// - PRESENT in `customerMessages` → the customer would be e-mailed, in Hungarian, a
|
||||
// list of which apps' backups failed and why — operator detail they can take no action on. So
|
||||
// this type must be allowlisted and must NOT have a customer message.
|
||||
//
|
||||
// The customer-facing half of decision D-c is the FILL WARNING, which fires before this and is
|
||||
// actionable (free space, delete files, add a drive). This is the operator's half.
|
||||
//
|
||||
// Operator-only routing itself is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a
|
||||
// customerMessages entry — that assumption shipped in v0.78.0 and was wrong, because
|
||||
// FormatCustomerEmail falls back to the raw message. It is pinned in
|
||||
// `internal/notify/backup_run_digest_test.go`, which demonstrates a customer with the type in their
|
||||
// enabled list receiving nothing.
|
||||
func TestBackupRunDigestIsAllowlisted(t *testing.T) {
|
||||
if !allowedEventTypes["backup_run_failures"] {
|
||||
t.Fatal("backup_run_failures must be in allowedEventTypes, or POST /event 400s and the " +
|
||||
"whole run digest is dropped at the door — the silence R-182 was filed against")
|
||||
}
|
||||
}
|
||||
|
||||
// The per-app event is the RECORD and must not be removed while the digest is the notification.
|
||||
// Deleting it would make the digest the only trace, and a digest that fails to send would then take
|
||||
// the record with it — the coupling R-182's fix exists to break.
|
||||
func TestPerAppCaptureEventStaysAllowlisted(t *testing.T) {
|
||||
if !allowedEventTypes["recovery_unit_capture_failed"] {
|
||||
t.Fatal("recovery_unit_capture_failed was removed from allowedEventTypes — it is the " +
|
||||
"durable per-failure RECORD, and the digest is only the notification; the operator " +
|
||||
"register and every historical query depend on it")
|
||||
}
|
||||
}
|
||||
@@ -1573,6 +1573,25 @@ var allowedEventTypes = map[string]bool{
|
||||
"whole_guest_backup_failed": true,
|
||||
"whole_guest_backup_recovered": true,
|
||||
|
||||
// R-158 / R-167 (controller v0.191.0, decision D-c): a per-app Tier-1 recovery-unit capture
|
||||
// failed. Until then a `[WARN]` line in the controller reached no hub channel at all — the fifth
|
||||
// instance in this project of a mechanism built and left disconnected.
|
||||
//
|
||||
// DELIBERATELY NOT `backup_failed`, for exactly the reason recorded above for the whole-guest
|
||||
// pair: that type carries a customerMessages entry AND sits in the controller's
|
||||
// DefaultEnabledEvents, so reusing it emails the CUSTOMER, in Hungarian, about a failure they
|
||||
// cannot act on. R-158's original proposal named `backup_failed`; D-c routes this to the
|
||||
// operator, and where the two disagree D-c wins.
|
||||
//
|
||||
// OPERATOR-ONLY IS ENFORCED BY `notify.operatorOnlyEvents` — see the paragraph above. This entry
|
||||
// alone does NOT make it operator-only.
|
||||
"recovery_unit_capture_failed": true,
|
||||
// R-182. The per-run backup digest: one event at the end of a run, listing every app whose
|
||||
// backup failed or was refused. Allowlisting it is NOT what keeps it away from customers —
|
||||
// `notify.operatorOnlyEvents` is (see the comment there); both entries ship together and
|
||||
// `backup_run_digest_event_test.go` pins the pair.
|
||||
"backup_run_failures": true,
|
||||
|
||||
// Controller-pushed events
|
||||
"controller_started": true,
|
||||
"claim_lockout": true, // v0.50.0 — claim/reset code brute-force lockout tripped
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
|
||||
)
|
||||
|
||||
// R-158 / R-167 (D-c) — the per-app Tier-1 recovery-unit capture failure alert.
|
||||
//
|
||||
// A new event type is a PAIR of registers, and each half fails differently:
|
||||
// - missing from allowedEventTypes → POST /event returns 400 and the event VANISHES (R-97a);
|
||||
// - missing from operatorOnlyEvents → it is delivered TO THE CUSTOMER, in raw English, about a
|
||||
// failure they can take no action on.
|
||||
//
|
||||
// The second is the quiet one — delivery "works", so nothing looks broken. It is also the defect
|
||||
// v0.78.0 actually shipped: adding a type to allowedEventTypes and ASSUMING that made it
|
||||
// operator-only. `FormatCustomerEmail` treats a missing customerMessages entry as a fallback to the
|
||||
// raw message, not a block, and the only customer gate is configuration. Both halves are pinned
|
||||
// here, in the same test, because fixing one and not the other is the realistic mistake.
|
||||
func TestRecoveryUnitCaptureFailedIsAllowlistedAndOperatorOnly(t *testing.T) {
|
||||
const et = "recovery_unit_capture_failed"
|
||||
|
||||
if !allowedEventTypes[et] {
|
||||
t.Fatalf("%s must be in allowedEventTypes, or POST /event 400s and a per-app Tier-1 backup "+
|
||||
"failure reaches no hub channel at all — which is the R-158 gap, un-fixed", et)
|
||||
}
|
||||
if !notify.IsOperatorOnly(et) {
|
||||
t.Fatalf("%s is allowlisted but NOT in notify.operatorOnlyEvents — the customer would be "+
|
||||
"emailed about a recovery-unit capture failure they can take no action on. Adding a type "+
|
||||
"to allowedEventTypes does NOT make it operator-only; that was the v0.78.0 defect "+
|
||||
"(corrected in v0.79.0/R-97c) and this is the same mistake one event later", et)
|
||||
}
|
||||
}
|
||||
|
||||
// The customer's half of D-c must NOT be operator-only — a fill warning is precisely the alert a
|
||||
// customer CAN act on (free space, delete files, add a drive). Pinned in the same file as its
|
||||
// operator sibling so the routing decision is read as one thing, which is what D-c is.
|
||||
func TestFillWarningReachesTheCustomer(t *testing.T) {
|
||||
for _, et := range []string{"disk_warning", "disk_critical"} {
|
||||
if !allowedEventTypes[et] {
|
||||
t.Fatalf("%s must stay in allowedEventTypes — it is the customer fill warning the "+
|
||||
"controller emits from v0.191.0 (R-167)", et)
|
||||
}
|
||||
if notify.IsOperatorOnly(et) {
|
||||
t.Fatalf("%s is in operatorOnlyEvents — the customer would never be warned that their "+
|
||||
"disk is filling, which is the whole customer half of decision D-c", et)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -32,15 +32,104 @@ import (
|
||||
// is a NEW monitor written straight after the third, so it copies R-81's verdict structure rather
|
||||
// than re-deriving it. A tier never proven on a newborn box is UNKNOWN, never FAILED.
|
||||
|
||||
// restoreProvenStaleAfter is how long a tier may go unproven before it is called stale.
|
||||
// ── HOW LONG MAY A TIER GO UNPROVEN? (R-86 Part 2) ───────────────────────────────────────────
|
||||
//
|
||||
// Derivation, not a guess: the restore-test cadence is 24h and rotation is oldest-first across two
|
||||
// tiers, so each tier is proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive
|
||||
// missed opportunities before alarming — loud enough to matter, quiet enough not to fire on one
|
||||
// skipped cycle (a deferral behind a long backup is normal, not a fault). It is also comfortably
|
||||
// inside the 2-week offsite retention (operator ruling 2026-07-26), so a tier is never reported
|
||||
// stale against an archive that is about to be pruned anyway.
|
||||
const restoreProvenStaleAfter = 7 * 24 * time.Hour
|
||||
// This was one flat constant, 7 days, and its comment derived that number like this:
|
||||
//
|
||||
// "the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
|
||||
// proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."
|
||||
//
|
||||
// **That premise is exactly what R-86 removed.** The agent no longer tests on an interval at all: a
|
||||
// tier is tested once per ARCHIVE GENERATION — when it holds a settled archive that has not been
|
||||
// proven. A tier backed up weekly is therefore proved weekly, by design and in perfect health, and
|
||||
// against a flat 7-day window it would sit on the line and alarm every night about a system that is
|
||||
// working. Shipping the agent's half alone would have converted the improvement into a false alarm,
|
||||
// which is why the two ship together.
|
||||
//
|
||||
// The window is now derived from **the tier's own backup rhythm**, which the hub can observe from
|
||||
// the reports it already receives, and it keeps everything the constant had earned:
|
||||
//
|
||||
// - absence is UNKNOWN until an anchored window has passed (R-81's structure, untouched);
|
||||
// - the signal stays edge-triggered;
|
||||
// - it never exceeds the offsite retention, so a tier is never called stale against an archive
|
||||
// that is about to be pruned;
|
||||
// - and it is never TIGHTER than the 7 days that were already tolerated.
|
||||
const (
|
||||
// restoreProvenGenerations is how many archive generations may pass unproven before alarming.
|
||||
// 4 = the settle lag's own generation plus ~3 missed opportunities — deliberately the same
|
||||
// tolerance the flat constant expressed, so the change is to the RHYTHM, not to the patience.
|
||||
restoreProvenGenerations = 4
|
||||
|
||||
// restoreProvenWindowFloor is the shortest window that may be applied to any tier. It is the
|
||||
// old constant, kept as a FLOOR rather than deleted: a daily tier computes 4 days from its own
|
||||
// rhythm, and tightening a live threshold is not what this task is for. A deferral behind a
|
||||
// long backup is normal, not a fault.
|
||||
restoreProvenWindowFloor = 7 * 24 * time.Hour
|
||||
|
||||
// restoreProvenWindowCap keeps the window strictly inside the 2-week offsite retention
|
||||
// (operator ruling 2026-07-26) with two days to spare. Beyond it the hub would be judging a
|
||||
// tier against an archive PBS has already pruned — an alarm nobody can act on, and the bound
|
||||
// the old constant respected in its own way.
|
||||
restoreProvenWindowCap = 12 * 24 * time.Hour
|
||||
|
||||
// restoreWindowRead is how far back the hub reads host-reports for this check: far enough to
|
||||
// find proof anywhere inside the widest window, and to see at least two archive generations of
|
||||
// a WEEKLY tier so its rhythm is observable at all.
|
||||
restoreWindowRead = 2 * restoreProvenWindowFloor
|
||||
)
|
||||
|
||||
// declaredArchiveInterval is the rhythm the hub ALREADY attributes to a tier — the same thresholds
|
||||
// the backup-freshness checker judges it against (deadline.go / deadline_tiers.go). It is the
|
||||
// fallback when a box's history is too short to observe a rhythm, and it is the right fallback
|
||||
// precisely because it is not a second opinion: if these two checkers disagreed about how often a
|
||||
// tier is expected to receive an archive, one of them would be alarming on the other's model.
|
||||
//
|
||||
// It is stated per RESTORE tier name ("local"/"pbs" — what the agent reports as source_tier), which
|
||||
// is the same split the backup tiers use under different names ("host"/"offsite").
|
||||
func declaredArchiveInterval(tier string) time.Duration {
|
||||
if tier == "pbs" {
|
||||
return offsiteBackupStaleAfter // 8 days: the weekly cadence plus a day of headroom
|
||||
}
|
||||
return backupStaleAfter // 26 hours: the daily cadence plus headroom
|
||||
}
|
||||
|
||||
// restoreProvenWindow is how long THIS tier may go unproven, given its observed archive interval.
|
||||
//
|
||||
// observedOK=false means the box's retained history did not contain two archive generations for
|
||||
// this tier, so the declared rhythm is used. That fallback matters most for exactly the tier this
|
||||
// task is about: a fresh box with a weekly offsite tier has one snapshot and no observable
|
||||
// interval, and falling back to the FLOOR there would recreate the false alarm.
|
||||
//
|
||||
// OBSERVATION MAY ONLY WIDEN, NEVER TIGHTEN — and this is not caution, it is a live measurement.
|
||||
// On demo-felhom (2026-08-03) the offsite tier's two retained snapshots are `2026-07-27T19:55:41Z`
|
||||
// and `2026-07-28T04:49:43Z`: **8 h 54 m apart**, because one is a healing artefact and the other a
|
||||
// real weekly run. A mean-gap estimate therefore reads a WEEKLY tier as nine-hourly, ×4 gives 36 h,
|
||||
// the floor lifts it to 7 days — and a weekly tier proved weekly reaches ~8.25 days of proof age, so
|
||||
// the false alarm this whole task exists to prevent would have returned within a week, on the very
|
||||
// box it shipped to.
|
||||
//
|
||||
// The asymmetry is right on its own terms too. A gap SHORTER than the declared rhythm is routine and
|
||||
// means nothing — a retry, a manual run, a heal, a catch-up after an outage. A gap LONGER than the
|
||||
// declared rhythm is real information: this tier genuinely receives archives less often than the
|
||||
// model says, and its window must widen or it alarms. So observation refines the rhythm upward and
|
||||
// is ignored downward. The cost is stated plainly: a tier that truly runs FASTER than its declared
|
||||
// rhythm gets a wider window than it strictly needs, i.e. a slower stale signal. That is the right
|
||||
// direction for a signal whose message is "unverified" — "broken NOW" is `restore_test_failed`, and
|
||||
// that one is immediate and unaffected.
|
||||
func restoreProvenWindow(tier string, observed time.Duration, observedOK bool) time.Duration {
|
||||
interval := declaredArchiveInterval(tier)
|
||||
if observedOK && observed > interval {
|
||||
interval = observed
|
||||
}
|
||||
w := time.Duration(restoreProvenGenerations) * interval
|
||||
if w < restoreProvenWindowFloor {
|
||||
w = restoreProvenWindowFloor
|
||||
}
|
||||
if w > restoreProvenWindowCap {
|
||||
w = restoreProvenWindowCap
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// Event types. Operator-tier only — see the dispatcher note in RestoreTestChecker.
|
||||
const (
|
||||
@@ -156,12 +245,13 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
return
|
||||
}
|
||||
|
||||
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-2*restoreProvenStaleAfter))
|
||||
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-restoreWindowRead))
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] restore-test check: window read failed for %s: %v", customerID, err)
|
||||
return
|
||||
}
|
||||
proven := lastProvenPerTier(rows)
|
||||
intervals := observedArchiveIntervals(rows)
|
||||
|
||||
first, ferr := c.store.GetFirstHostReportAt(customerID)
|
||||
if ferr != nil {
|
||||
@@ -170,7 +260,9 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
}
|
||||
|
||||
for _, tier := range tiers {
|
||||
v := assessRestoreProven(tier, proven[tier], first, now)
|
||||
observed, observedOK := intervals[tier]
|
||||
window := restoreProvenWindow(tier, observed, observedOK)
|
||||
v := assessRestoreProven(tier, proven[tier], first, now, window)
|
||||
key := customerID + "|" + tier
|
||||
c.mu.Lock()
|
||||
prev := c.staleStates[key]
|
||||
@@ -192,37 +284,120 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
}
|
||||
}
|
||||
|
||||
// assessRestoreProven is the per-tier verdict. PURE (now injected) so the policy is unit-tested —
|
||||
// the property that made R-81 provable, kept deliberately.
|
||||
// assessRestoreProven is the per-tier verdict. PURE (now and the window injected) so the policy is
|
||||
// unit-tested — the property that made R-81 provable, kept deliberately.
|
||||
//
|
||||
// no proof, anchor NOT elapsed → UNKNOWN (newborn box; never an alarm)
|
||||
// no proof, anchor elapsed → MISSED
|
||||
// proof older than the window → MISSED
|
||||
// otherwise → OK
|
||||
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time) backupAssessment {
|
||||
//
|
||||
// `window` is now the TIER'S OWN (R-86 Part 2) rather than one constant for every tier, and every
|
||||
// reason string states the window it was judged against. That is R-100's corollary applied here:
|
||||
// when a verdict changes what it counts from, the alarm text has to change with it, or an operator
|
||||
// reads "limit 168h" under a tier that was actually judged at 288h and dismisses a true alarm.
|
||||
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time, window time.Duration) backupAssessment {
|
||||
if provenAt.IsZero() {
|
||||
if firstReportAt.IsZero() {
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: never restore-proven, and no first-contact anchor to defer against", tier)}
|
||||
}
|
||||
watched := now.Sub(firstReportAt)
|
||||
if watched <= restoreProvenStaleAfter {
|
||||
if watched <= window {
|
||||
return backupAssessment{verdict: verdictUnknown,
|
||||
reason: fmt.Sprintf("%s tier: not restore-proven yet, but only watching for %s (grace %s since first contact %s) — newborn, not a fault",
|
||||
tier, watched.Round(time.Hour), restoreProvenStaleAfter, firstReportAt.Format(time.RFC3339))}
|
||||
tier, watched.Round(time.Hour), window, firstReportAt.Format(time.RFC3339))}
|
||||
}
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s) — the tier is unverified, not known-broken",
|
||||
tier, watched.Round(time.Hour), restoreProvenStaleAfter)}
|
||||
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
|
||||
tier, watched.Round(time.Hour), window)}
|
||||
}
|
||||
if age := now.Sub(provenAt); age > restoreProvenStaleAfter {
|
||||
if age := now.Sub(provenAt); age > window {
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s) — the tier is unverified, not known-broken",
|
||||
tier, age.Round(time.Hour), restoreProvenStaleAfter)}
|
||||
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
|
||||
tier, age.Round(time.Hour), window)}
|
||||
}
|
||||
return backupAssessment{verdict: verdictOK}
|
||||
}
|
||||
|
||||
// observedArchiveIntervals estimates how often each RESTORE tier actually receives an archive, from
|
||||
// the host-reports the hub already holds. Keyed by restore-tier name ("local"/"pbs").
|
||||
//
|
||||
// Evidence is every distinct archive timestamp in the window: `pbs_snapshots[]` for the offsite
|
||||
// tier (PBS enumerates its whole retention in each report, so one report usually settles the
|
||||
// question) and successful `backups[]` records attributed by TARGET TYPE for both tiers — the
|
||||
// slice-A.4 rule, because a PBS-targeted vzdump appears in BOTH arrays and classifying by array
|
||||
// membership would attribute an offsite archive to the host tier.
|
||||
//
|
||||
// The estimate is the MEAN gap: (newest − oldest) / (generations − 1). It needs two generations;
|
||||
// with fewer, ok=false and the caller falls back to the declared rhythm. It is deliberately crude,
|
||||
// and can afford to be: restoreProvenWindow clamps the result between a 7-day floor and a 12-day
|
||||
// cap, so the only discrimination this has to get right is "roughly daily" versus "several days or
|
||||
// slower" — which is exactly the distinction that turns a healthy weekly tier into a false alarm.
|
||||
func observedArchiveIntervals(rows []store.HostReportRow) map[string]time.Duration {
|
||||
seen := map[string]map[int64]struct{}{ // tier → set of archive unix times
|
||||
"local": {},
|
||||
"pbs": {},
|
||||
}
|
||||
add := func(tier string, t time.Time) {
|
||||
if t.IsZero() {
|
||||
return
|
||||
}
|
||||
seen[tier][t.UTC().Unix()] = struct{}{}
|
||||
}
|
||||
|
||||
for _, r := range rows {
|
||||
var hr hostReportBackups
|
||||
if json.Unmarshal([]byte(r.ReportJSON), &hr) != nil {
|
||||
continue // one malformed retained report must not blind the scan
|
||||
}
|
||||
pbs := pbsTargetSet(hr)
|
||||
for _, ps := range hr.PBSSnapshots {
|
||||
if t, ok := parseBackupTime(ps.BackupTime); ok {
|
||||
add("pbs", t)
|
||||
}
|
||||
}
|
||||
for _, b := range hr.Backups {
|
||||
if !b.Success {
|
||||
continue
|
||||
}
|
||||
t, ok := parseBackupTime(b.StartedAt)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if pbs[b.TargetID] {
|
||||
add("pbs", t)
|
||||
} else {
|
||||
add("local", t)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
out := map[string]time.Duration{}
|
||||
for tier, set := range seen {
|
||||
if len(set) < 2 {
|
||||
continue // not observable — the caller uses the declared rhythm
|
||||
}
|
||||
var oldest, newest int64
|
||||
first := true
|
||||
for ts := range set {
|
||||
if first || ts < oldest {
|
||||
oldest = ts
|
||||
}
|
||||
if first || ts > newest {
|
||||
newest = ts
|
||||
}
|
||||
first = false
|
||||
}
|
||||
span := time.Duration(newest-oldest) * time.Second
|
||||
if span <= 0 {
|
||||
continue
|
||||
}
|
||||
out[tier] = span / time.Duration(len(set)-1)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// expectedRestoreTiers names the tiers this box actually HAS, so a box without an offsite tier is
|
||||
// never reported stale for one. Same gate as Slice C's `expected`, and for the same reason: without
|
||||
// it every box lacking a tier would alarm once the anchor elapsed — absence-is-not-failure,
|
||||
|
||||
@@ -182,26 +182,33 @@ func TestRestoreTest_NewbornDoesNotAlarm(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The boundary, pinned by name so a refactor has to delete an obviously-named contract.
|
||||
// The boundary, pinned by name so a refactor has to delete an obviously-named contract. R-86 made
|
||||
// the limit per-tier, so the anchor is now measured against THE TIER'S OWN window — here the local
|
||||
// tier's, which clamps to the 7-day floor and so keeps this contract numerically identical to the
|
||||
// one the flat constant expressed.
|
||||
func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
window := restoreProvenWindow("local", 24*time.Hour, true)
|
||||
if window != restoreProvenWindowFloor {
|
||||
t.Fatalf("precondition: a daily local tier must clamp to the floor; got %s", window)
|
||||
}
|
||||
cases := []struct {
|
||||
name string
|
||||
watched time.Duration
|
||||
wantMissed bool
|
||||
}{
|
||||
{"newborn, 1h", time.Hour, false},
|
||||
{"just inside", restoreProvenStaleAfter - time.Minute, false},
|
||||
{"exactly at the limit", restoreProvenStaleAfter, false},
|
||||
{"just outside", restoreProvenStaleAfter + time.Minute, true},
|
||||
{"just inside", window - time.Minute, false},
|
||||
{"exactly at the limit", window, false},
|
||||
{"just outside", window + time.Minute, true},
|
||||
{"long past", 30 * 24 * time.Hour, true},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := assessRestoreProven("pbs", time.Time{}, now.Add(-c.watched), now)
|
||||
got := assessRestoreProven("local", time.Time{}, now.Add(-c.watched), now, window)
|
||||
if got.missed() != c.wantMissed {
|
||||
t.Fatalf("CONTRACT VIOLATED: unproven for %s (limit %s) → missed=%v, want %v (reason %q)",
|
||||
c.watched, restoreProvenStaleAfter, got.missed(), c.wantMissed, got.reason)
|
||||
c.watched, window, got.missed(), c.wantMissed, got.reason)
|
||||
}
|
||||
if !c.wantMissed && got.verdict != verdictUnknown {
|
||||
t.Fatalf("a deferred tier must be UNKNOWN (visible), not OK; got verdict=%d", got.verdict)
|
||||
@@ -215,15 +222,17 @@ func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.
|
||||
func TestRestoreTest_StaleIsSeparateFromFailure(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
stale := assessRestoreProven("pbs", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now)
|
||||
// A DAILY tier judged on its own rhythm: the window clamps to the 7-day floor.
|
||||
daily := restoreProvenWindow("local", 24*time.Hour, true)
|
||||
stale := assessRestoreProven("local", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
|
||||
if !stale.missed() {
|
||||
t.Fatalf("a tier last proven 9 days ago (limit %s) must be stale; got %q", restoreProvenStaleAfter, stale.reason)
|
||||
t.Fatalf("a daily tier last proven 9 days ago (limit %s) must be stale; got %q", daily, stale.reason)
|
||||
}
|
||||
// The wording must not read as "broken" — that is the other signal.
|
||||
if !strings.Contains(stale.reason, "unverified, not known-broken") {
|
||||
t.Fatalf("staleness must say UNVERIFIED, not broken — merging the two is the thing this avoids; got %q", stale.reason)
|
||||
}
|
||||
fresh := assessRestoreProven("pbs", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now)
|
||||
fresh := assessRestoreProven("local", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
|
||||
if fresh.verdict != verdictOK {
|
||||
t.Fatalf("a tier proven 2 days ago is fine; got verdict=%d reason=%q", fresh.verdict, fresh.reason)
|
||||
}
|
||||
@@ -283,3 +292,208 @@ func boolStr(b bool) string {
|
||||
}
|
||||
return "false"
|
||||
}
|
||||
|
||||
// ── SCENARIO G — a healthy WEEKLY tier is never reported stale (R-86 Part 2) ─────────────────
|
||||
//
|
||||
// This is the test that pins the false alarm this change would otherwise have CREATED. The agent
|
||||
// now proves a tier once per archive generation, so a weekly offsite tier is proved weekly — in
|
||||
// perfect health. Against the old flat 7-day window it would sit on the line and alarm every night.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): pin the window flat, as it was —
|
||||
//
|
||||
// - window := restoreProvenWindow(tier, observed, observedOK)
|
||||
// - window := restoreProvenWindowFloor // the pre-R-86 flat 7 days
|
||||
//
|
||||
// → --- FAIL: TestRestoreTest_HealthyWeeklyTierIsNeverStale
|
||||
//
|
||||
// week 0: a weekly tier proved on its own archive must never be stale (proof age 172h0m0s,
|
||||
// window 168h0m0s); verdict=2 reason="pbs tier: last successful restore-test was 172h0m0s ago
|
||||
// (limit 168h0m0s, this tier's own backup rhythm) — the tier is unverified, not known-broken"
|
||||
//
|
||||
// Restored. The mutation is one line because the whole of Part 2 is one decision: whose rhythm.
|
||||
//
|
||||
// NOTE, because it is the finding this test nearly hid: the FIRST version of this fixture had NO
|
||||
// jitter, and it PASSED under the mutation. A perfectly regular weekly tier's proof age reaches
|
||||
// EXACTLY 168h just before the next proof, and `age > window` is false by a hair — a hollow test
|
||||
// that would have shipped Part 1 and its false alarm together. The jitter below is what makes this
|
||||
// a test, and it is also the truth about the old constant: a healthy weekly tier did not merely sit
|
||||
// near the line, it sat ON it, so any ordinary delay tipped it over.
|
||||
func TestRestoreTest_HealthyWeeklyTierIsNeverStale(t *testing.T) {
|
||||
start := time.Date(2026, 6, 1, 3, 0, 0, 0, time.UTC)
|
||||
firstContact := start.Add(-24 * time.Hour)
|
||||
|
||||
// The observable rhythm of a weekly tier, as the hub would compute it from the reports. No
|
||||
// assertion about the window ITSELF here on purpose: that is the mechanism, and it is pinned in
|
||||
// TestRestoreProvenWindow_Contract. What this test asserts is the CONSEQUENCE — does the alarm
|
||||
// fire? — because R-97b proved a mechanism and shipped a broken consequence anyway.
|
||||
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
|
||||
|
||||
// Walk several weeks of a HEALTHY tier, with the jitter a real one has: the backup does not land
|
||||
// to the second, and a restore-test can be deferred one evaluation behind a running backup.
|
||||
//
|
||||
// The jitter is the point. A perfectly regular weekly tier's proof reaches an age of EXACTLY one
|
||||
// interval (168h) just before the next proof, and against a flat 168h window `age > window` is
|
||||
// false by a hair — so a regular fixture would pass against the very constant this task must
|
||||
// change, and prove nothing. That is the brief's "sits exactly on that line": every real-world
|
||||
// delay pushes it over, and the alarm is about a system that is working.
|
||||
settle, evalLatency := 24*time.Hour, 6*time.Hour
|
||||
archiveLate := []time.Duration{0, 4 * time.Hour, 2 * time.Hour, 6 * time.Hour, 0, 3 * time.Hour}
|
||||
deferred := []time.Duration{0, 0, 6 * time.Hour, 0, 0, 6 * time.Hour} // one evaluation behind a backup
|
||||
|
||||
archiveAt := func(week int) time.Time {
|
||||
return start.AddDate(0, 0, 7*week).Add(archiveLate[week])
|
||||
}
|
||||
provenAt := func(week int) time.Time {
|
||||
return archiveAt(week).Add(settle + evalLatency + deferred[week])
|
||||
}
|
||||
|
||||
var worst time.Duration
|
||||
for week := 0; week+1 < len(archiveLate); week++ {
|
||||
// The widest the proof's age ever gets: the instant before the NEXT week's proof lands.
|
||||
now := provenAt(week + 1).Add(-time.Second)
|
||||
age := now.Sub(provenAt(week))
|
||||
if age > worst {
|
||||
worst = age
|
||||
}
|
||||
v := assessRestoreProven("pbs", provenAt(week), firstContact, now, weekly)
|
||||
if v.verdict != verdictOK {
|
||||
t.Fatalf("week %d: a weekly tier proved on its own archive must never be stale (proof age %s, window %s); verdict=%d reason=%q",
|
||||
week, age.Round(time.Hour), weekly, v.verdict, v.reason)
|
||||
}
|
||||
}
|
||||
// The fixture must actually EXERCISE the boundary — a jitter-free walk would sit at exactly one
|
||||
// interval and pass against a flat 7-day window, which is the hollow version of this test.
|
||||
if worst <= restoreProvenWindowFloor {
|
||||
t.Fatalf("this fixture never exceeds the old flat window (worst proof age %s) — it cannot detect the defect it exists for", worst)
|
||||
}
|
||||
}
|
||||
|
||||
// ...and a weekly tier that genuinely STOPS being proved must still alarm. A window that never
|
||||
// fires is not a fix, it is a deletion.
|
||||
func TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
|
||||
|
||||
v := assessRestoreProven("pbs", now.Add(-weekly-time.Hour), now.Add(-90*24*time.Hour), now, weekly)
|
||||
if !v.missed() {
|
||||
t.Fatalf("a weekly tier unproven for longer than its own window MUST alarm; got verdict=%d reason=%q", v.verdict, v.reason)
|
||||
}
|
||||
if !strings.Contains(v.reason, weekly.String()) {
|
||||
t.Fatalf("the alarm must state the window it was judged against (R-100's corollary); got %q", v.reason)
|
||||
}
|
||||
}
|
||||
|
||||
// The window's own contract: derived from the tier's rhythm, floored, capped, and never dependent
|
||||
// on an unobservable history for the tier that would suffer most from a wrong answer.
|
||||
func TestRestoreProvenWindow_Contract(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
tier string
|
||||
observed time.Duration
|
||||
observedOK bool
|
||||
want time.Duration
|
||||
}{
|
||||
{"daily local clamps to the floor", "local", 24 * time.Hour, true, restoreProvenWindowFloor},
|
||||
{"weekly pbs widens", "pbs", 7 * 24 * time.Hour, true, restoreProvenWindowCap},
|
||||
{"3-day tier sits between", "pbs", 72 * time.Hour, true, 12 * 24 * time.Hour},
|
||||
{"unobservable local falls back to its declared rhythm", "local", 0, false, restoreProvenWindowFloor},
|
||||
{"unobservable pbs falls back WIDE, not to the floor", "pbs", 0, false, restoreProvenWindowCap},
|
||||
{"a nonsense zero interval is ignored", "pbs", 0, true, restoreProvenWindowCap},
|
||||
// MEASURED ON THE LIVE BOX, and the reason observation may only WIDEN. demo-felhom's two
|
||||
// retained PBS snapshots sit 8h54m apart (one is a healing artefact), so a mean-gap estimate
|
||||
// reads a WEEKLY tier as nine-hourly. Taking that at face value gives 4x9h = 36h → the 7-day
|
||||
// floor → and a weekly tier proved weekly (~8.25d of proof age) alarms within a week of this
|
||||
// shipping, on the box it shipped to.
|
||||
{"a short observed gap must NOT tighten a weekly tier", "pbs", 8*time.Hour + 54*time.Minute, true, restoreProvenWindowCap},
|
||||
{"a short observed gap must not tighten the host tier either", "local", 30 * time.Minute, true, restoreProvenWindowFloor},
|
||||
// ...but a tier that genuinely runs SLOWER than its declared rhythm still widens.
|
||||
{"a genuinely slower tier widens", "local", 4 * 24 * time.Hour, true, restoreProvenWindowCap},
|
||||
}
|
||||
// The relationship Part 1 depends on: a weekly tier's window must be WIDER than a daily tier's,
|
||||
// or proving weekly (which is now correct behaviour) alarms on itself.
|
||||
if restoreProvenWindow("pbs", 7*24*time.Hour, true) <= restoreProvenWindow("local", 24*time.Hour, true) {
|
||||
t.Fatal("a weekly tier must earn a wider window than a daily one — otherwise R-86's agent half alarms about itself")
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := restoreProvenWindow(c.tier, c.observed, c.observedOK)
|
||||
if got != c.want {
|
||||
t.Fatalf("window(%s, observed=%s ok=%v) = %s, want %s", c.tier, c.observed, c.observedOK, got, c.want)
|
||||
}
|
||||
if got < restoreProvenWindowFloor || got > restoreProvenWindowCap {
|
||||
t.Fatalf("every window must stay inside [%s, %s]; got %s", restoreProvenWindowFloor, restoreProvenWindowCap, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The rhythm must be OBSERVED from the reports, not assumed — including the slice-A.4 rule that a
|
||||
// PBS-targeted vzdump appears in both arrays and must be attributed by TARGET TYPE.
|
||||
func TestObservedArchiveIntervals_FromReports(t *testing.T) {
|
||||
base := time.Date(2026, 7, 1, 2, 0, 0, 0, time.UTC)
|
||||
mk := func(localAt []time.Time, pbsAt []time.Time) string {
|
||||
type stg struct{ Name, Type, Content string }
|
||||
type bk struct {
|
||||
TargetID string `json:"target_id"`
|
||||
Success bool `json:"success"`
|
||||
StartedAt string `json:"started_at"`
|
||||
}
|
||||
type snap struct {
|
||||
BackupTime string `json:"backup_time"`
|
||||
}
|
||||
payload := struct {
|
||||
StorageTargets []struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
} `json:"storage_targets"`
|
||||
Backups []bk `json:"backups"`
|
||||
PBSSnapshots []snap `json:"pbs_snapshots"`
|
||||
}{}
|
||||
payload.StorageTargets = append(payload.StorageTargets, struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
}{"felhom-backup", "dir", "backup"}, struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
}{"felhom-pbs", "pbs", "backup"})
|
||||
for _, at := range localAt {
|
||||
payload.Backups = append(payload.Backups, bk{"felhom-backup", true, at.Format(time.RFC3339)})
|
||||
}
|
||||
for _, at := range pbsAt {
|
||||
// The SAME archive appears as a vzdump record AND as a snapshot — slice A.4.
|
||||
payload.Backups = append(payload.Backups, bk{"felhom-pbs", true, at.Format(time.RFC3339)})
|
||||
payload.PBSSnapshots = append(payload.PBSSnapshots, snap{at.Format(time.RFC3339)})
|
||||
}
|
||||
b, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
rows := []store.HostReportRow{
|
||||
{ReportJSON: mk(
|
||||
[]time.Time{base, base.AddDate(0, 0, 1), base.AddDate(0, 0, 2)},
|
||||
[]time.Time{base, base.AddDate(0, 0, 7)},
|
||||
)},
|
||||
{ReportJSON: `{{{malformed`}, // must not blind the scan
|
||||
}
|
||||
|
||||
got := observedArchiveIntervals(rows)
|
||||
if d, ok := got["local"]; !ok || d != 24*time.Hour {
|
||||
t.Fatalf("a daily host tier must be observed as ~24h; got %s ok=%v", d, ok)
|
||||
}
|
||||
if d, ok := got["pbs"]; !ok || d != 7*24*time.Hour {
|
||||
t.Fatalf("a weekly offsite tier must be observed as ~7d — and its vzdump record must NOT be "+
|
||||
"counted into the host tier (slice A.4); got %s ok=%v", d, ok)
|
||||
}
|
||||
|
||||
// One generation is not a rhythm: unobservable, so the caller falls back to the declared one.
|
||||
single := []store.HostReportRow{{ReportJSON: mk(nil, []time.Time{base})}}
|
||||
if d, ok := observedArchiveIntervals(single)["pbs"]; ok {
|
||||
t.Fatalf("one archive cannot yield an interval; got %s", d)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,301 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-182 — one e-mail per backup run, and nothing dropped without a trace.
|
||||
//
|
||||
// MEASURED, NOT SUPPOSED. On 2026-08-03 nine `recovery_unit_capture_failed` events reached the hub
|
||||
// and TWO operator e-mails went out. The operator cooldown key is
|
||||
// `customerID + ":" + eventType + cooldownTierSuffix(details)`, that event carries `app` but no
|
||||
// `tier`, so the key held no app identifier: the first refused app took the hour's slot and every
|
||||
// other app's failure was discarded — **before `LogNotification`**, so it left no row on any channel
|
||||
// and could not be found afterwards.
|
||||
//
|
||||
// The operator ruled against the obvious fix (putting `app` in the key), because on a full disk that
|
||||
// is one e-mail per app. These pin the shape that replaced it.
|
||||
|
||||
// ── Scenario D — a suppressed operator event leaves a trace ───────────────────────────────────────
|
||||
|
||||
// The bare `return` at the cooldown is the whole reason this defect took a day to get the right way
|
||||
// round: there was nothing to read. A drop must be as visible in the record as a send.
|
||||
//
|
||||
// DELIBERATELY EXERCISED ON A DIFFERENT EVENT TYPE than the one that exposed the defect.
|
||||
// `recovery_unit_capture_failed` is now record-only and never reaches the cooldown at all, so using
|
||||
// it here would prove nothing. `whole_guest_backup_failed` is an ordinary operator event, and using
|
||||
// it pins §2.1's actual claim: the suppression row applies to EVERY operator event, not only the one
|
||||
// that happened to be measured.
|
||||
func TestSuppressedOperatorEvent_LeavesARow(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
// Two events of the SAME type with no discriminator — the second must be suppressed.
|
||||
d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "opengist failed", `{"app":"opengist"}`, "controller")
|
||||
d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "privatebin failed", `{"app":"privatebin"}`, "controller")
|
||||
|
||||
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
|
||||
t.Fatalf("operator mails = %d, want 1 — the premise of this test is that the second IS suppressed", got)
|
||||
}
|
||||
|
||||
rows, err := st.GetRecentNotifications("c1", 50)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var sentRows, suppressed []store2Row
|
||||
for _, r := range rows {
|
||||
if r.Channel != "operator" || r.EventType != "whole_guest_backup_failed" {
|
||||
continue
|
||||
}
|
||||
switch r.Status {
|
||||
case "sent":
|
||||
sentRows = append(sentRows, store2Row{r.Status, r.Message, r.ErrorMessage})
|
||||
case "suppressed":
|
||||
suppressed = append(suppressed, store2Row{r.Status, r.Message, r.ErrorMessage})
|
||||
}
|
||||
}
|
||||
if len(sentRows) != 1 {
|
||||
t.Fatalf("want 1 'sent' operator row, got %d", len(sentRows))
|
||||
}
|
||||
if len(suppressed) != 1 {
|
||||
t.Fatalf("want 1 'suppressed' operator row, got %d — a cooldown drop that writes NOTHING is "+
|
||||
"indistinguishable from an event that never happened, which is exactly how seven "+
|
||||
"failures went missing on 2026-08-03", len(suppressed))
|
||||
}
|
||||
// The row must name the app that was dropped, or it records that something was suppressed
|
||||
// without recording WHAT — half a fix.
|
||||
if !strings.Contains(suppressed[0].message, "privatebin") {
|
||||
t.Fatalf("the suppressed row does not name the dropped event: %q", suppressed[0].message)
|
||||
}
|
||||
// And it must carry the key, so the reason it collided is readable without reading code.
|
||||
if !strings.Contains(suppressed[0].errMsg, "key=") {
|
||||
t.Fatalf("the suppressed row does not carry the cooldown key: %q", suppressed[0].errMsg)
|
||||
}
|
||||
}
|
||||
|
||||
type store2Row struct{ status, message, errMsg string }
|
||||
|
||||
// ── Scenario E — two runs in a day each report ───────────────────────────────────────────────────
|
||||
|
||||
// The operator ruled explicitly on this: someone pressing the backup button is actively trying to
|
||||
// get a backup, and finding out tomorrow would be worse than an extra mail in a rare case.
|
||||
func TestTwoRunsInAnHour_BothReport(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
nightly := `{"run_id":"run-a","run_kind":"nightly","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
|
||||
manual := `{"run_id":"run-b","run_kind":"manual","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
|
||||
|
||||
d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", nightly, "controller")
|
||||
d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", manual, "controller")
|
||||
|
||||
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 2 {
|
||||
t.Fatalf("operator mails = %d, want 2 — the 1-hour cooldown swallowed the manual run's "+
|
||||
"digest, which is the fix reappearing one level up: press the button, the run fails, "+
|
||||
"and you are told nothing because the machine already wrote this hour", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The run discriminator must be NARROW, exactly like its `tier` sibling — empty unless the producer
|
||||
// opts in — or every existing event type's cooldown silently stops collapsing anything.
|
||||
func TestCooldownRunSuffix_EmptyForEverythingElse(t *testing.T) {
|
||||
cases := []struct{ name, details string }{
|
||||
{"no details", ""},
|
||||
{"details without run_id", `{"app":"opengist","error":"boom"}`},
|
||||
{"empty run_id value", `{"run_id":""}`},
|
||||
{"malformed json", `{{{nope`},
|
||||
{"run_id mentioned in a STRING, not as a key", `{"error":"the run_id: abc failed"}`},
|
||||
{"null details", `null`},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := cooldownRunSuffix(c.details); got != "" {
|
||||
t.Errorf("%s: suffix must be EMPTY so every other type's cooldown is unchanged, got %q", c.name, got)
|
||||
}
|
||||
}
|
||||
if got := cooldownRunSuffix(`{"run_id":"run-a"}`); got != ":run-a" {
|
||||
t.Fatalf("suffix should be the run id, got %q", got)
|
||||
}
|
||||
// The two suffixes must not interfere: a tier event still keys on its tier and nothing else.
|
||||
if got := cooldownTierSuffix(`{"tier":"felhom-pbs"}`) + cooldownRunSuffix(`{"tier":"felhom-pbs"}`); got != ":felhom-pbs" {
|
||||
t.Fatalf("a tier-only event's key changed to %q — R-97a's behaviour must be byte-identical", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario G — the customer never receives the digest ──────────────────────────────────────────
|
||||
|
||||
// v0.78.0 asserted in a COMMENT that a type with no `customerMessages` entry structurally cannot
|
||||
// reach a customer. It can: FormatCustomerEmail falls back to the raw English message and the only
|
||||
// customer gate is configuration. So this is demonstrated, not argued.
|
||||
func TestDigest_IsOperatorOnly_EvenWithAWideEnabledList(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
// A customer who has enabled EVERYTHING, including this type by name.
|
||||
if err := st.SaveNotificationPrefs("c1", "cust@example.com",
|
||||
[]string{"backup_run_failures", "node_down", "disk_warning"}, 6); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
details := `{"run_id":"run-a","run_kind":"nightly","failed":1,"attempted":3,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"}]}`
|
||||
d.ProcessEvent("c1", "backup_run_failures", "error", "1 of 3 apps failed", details, "controller")
|
||||
|
||||
if got := mailsFor(*sent, "cust@example.com"); len(got) != 0 {
|
||||
t.Fatalf("the CUSTOMER received an operator digest (%d mails) — a list of which apps' "+
|
||||
"backups failed is not something they can act on, and the raw body is English", len(got))
|
||||
}
|
||||
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
|
||||
t.Fatalf("operator mails = %d, want 1", got)
|
||||
}
|
||||
if !operatorOnlyEvents["backup_run_failures"] {
|
||||
t.Fatal("backup_run_failures is not in operatorOnlyEvents — allowlisting alone does NOT " +
|
||||
"keep it from a customer; that assumption shipped once and was wrong (v0.78.0)")
|
||||
}
|
||||
}
|
||||
|
||||
// ── Part 3 — the e-mail a person actually reads ──────────────────────────────────────────────────
|
||||
|
||||
func TestDigestEmail_ListsAppsLegsAndReasons(t *testing.T) {
|
||||
details := `{"run_id":"run-a","run_kind":"nightly","failed":3,"attempted":40,` +
|
||||
`"target_path":"/mnt/sys_drive","used_gb":64.3,"avail_gb":0.9,"total_gb":68.7,` +
|
||||
`"used_percent":94,"space_known":true,"apps":[` +
|
||||
`{"app":"opengist","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
|
||||
`{"app":"privatebin","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
|
||||
`{"app":"immich","leg":"database dump","reason":"pg_dump: connection refused"}]}`
|
||||
|
||||
subject, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error",
|
||||
"3 of 40 apps failed to back up", details)
|
||||
|
||||
// The subject must carry the counts: the operator's first decision is made from it alone.
|
||||
for _, want := range []string{"demo-hp", "3 of 40", "nightly"} {
|
||||
if !strings.Contains(subject, want) {
|
||||
t.Errorf("subject %q missing %q", subject, want)
|
||||
}
|
||||
}
|
||||
// Every app, its leg and its reason.
|
||||
for _, want := range []string{
|
||||
"opengist", "privatebin", "immich",
|
||||
"volume dump", "database dump",
|
||||
"below the reserve", "pg_dump: connection refused",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("body missing %q:\n%s", want, body)
|
||||
}
|
||||
}
|
||||
// The counts and the free space, so "one broken app" and "a full disk" read differently.
|
||||
if !strings.Contains(body, "3 of 40") {
|
||||
t.Errorf("body does not carry the failed-of-attempted count:\n%s", body)
|
||||
}
|
||||
if !strings.Contains(body, "0.9 GB free") {
|
||||
t.Errorf("body does not carry the free space:\n%s", body)
|
||||
}
|
||||
// It must NOT be a JSON blob.
|
||||
if strings.Contains(body, `"apps":[`) {
|
||||
t.Errorf("the digest rendered as raw JSON — unreadable on a phone at 07:00:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// An absent space reading must render as unavailable, never as zeros: "0 GB free" and "we could not
|
||||
// look" are opposite diagnoses, and the operator acts differently on each.
|
||||
func TestDigestEmail_UnknownSpaceIsNotZero(t *testing.T) {
|
||||
details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/x",` +
|
||||
`"space_known":false,"apps":[{"app":"a","leg":"capture","reason":"boom"}]}`
|
||||
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
|
||||
if strings.Contains(body, "0.0 GB free") {
|
||||
t.Fatalf("an unreadable filesystem rendered as zeros:\n%s", body)
|
||||
}
|
||||
if !strings.Contains(body, "unavailable") {
|
||||
t.Fatalf("an unreadable filesystem must say so:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// A payload that cannot be parsed must still produce a mail — degraded, never swallowed.
|
||||
func TestDigestEmail_UnparseableDetailsStillMails(t *testing.T) {
|
||||
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "something failed", `{{{`)
|
||||
if body == "" || !strings.Contains(body, "something failed") {
|
||||
t.Fatalf("an unparseable digest lost the mail entirely:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario C — every failure is RECORDED, e-mailed or not ──────────────────────────────────────
|
||||
|
||||
// The per-app event is the record; the digest is the notification. The record must not inherit the
|
||||
// notification's conditions — no cooldown, no preferences, no dependence on a mail going out.
|
||||
func TestPerAppFailure_IsRecordedButNotMailed(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
apps := []string{"opengist", "privatebin", "immich", "homebox", "nextcloud"}
|
||||
for _, a := range apps {
|
||||
d.ProcessEvent("c1", "recovery_unit_capture_failed", "error",
|
||||
"Recovery unit capture FAILED for \""+a+"\"", `{"app":"`+a+`"}`, "controller")
|
||||
}
|
||||
|
||||
// NOT mailed — the digest is the notification.
|
||||
if got := len(*sent); got != 0 {
|
||||
t.Fatalf("%d mail(s) sent for per-app failures — they are the RECORD; one mail per app on a "+
|
||||
"full disk is the volume problem wearing the correctness problem's clothes, which is "+
|
||||
"exactly what the operator ruled against", got)
|
||||
}
|
||||
|
||||
// But ALL FIVE recorded — this is the assertion yesterday's defect would have failed: nine
|
||||
// arrived, two were mailed, seven left no row anywhere.
|
||||
rows, err := st.GetRecentNotifications("c1", 50)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, r := range rows {
|
||||
if r.EventType == "recovery_unit_capture_failed" && r.Status == "recorded" {
|
||||
for _, a := range apps {
|
||||
if strings.Contains(r.Message, a) {
|
||||
seen[a] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(seen) != len(apps) {
|
||||
t.Fatalf("only %d of %d per-app failures were recorded (%v) — a failure that produced no row "+
|
||||
"anywhere is the measured defect of 2026-08-03", len(seen), len(apps), seen)
|
||||
}
|
||||
}
|
||||
|
||||
// The per-app reason must not repeat the filesystem figures the digest already prints once. Reviewed
|
||||
// as copy against the first real digest, not designed in the abstract.
|
||||
func TestDigestEmail_ReasonDoesNotRepeatTheUsageLine(t *testing.T) {
|
||||
reason := "refused: below the reserve (reserve: 97% used or 1.0 GiB free) — /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free"
|
||||
details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/sys_drive",` +
|
||||
`"used_gb":65,"avail_gb":0.2,"total_gb":68.7,"used_percent":95,"space_known":true,` +
|
||||
`"apps":[{"app":"opengist","leg":"whole app","reason":"` + reason + `"}]}`
|
||||
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
|
||||
|
||||
// The figures appear ONCE, on the Filesystem line — not again on every app row.
|
||||
if strings.Count(body, "65.0/68.7 GB used") != 1 {
|
||||
t.Fatalf("the usage clause appears %d times; it must appear once, on its own line — repeated "+
|
||||
"down a list of a dozen apps it pushes the part that DIFFERS off a phone screen:\n%s",
|
||||
strings.Count(body, "65.0/68.7 GB used"), body)
|
||||
}
|
||||
// But the reason itself survives — trimming must not eat the diagnosis.
|
||||
if !strings.Contains(body, "below the reserve") {
|
||||
t.Fatalf("the reason was trimmed away entirely:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// A reason naming a DIFFERENT path, or none, must be left completely alone.
|
||||
func TestTrimRepeatedUsage_LeavesUnrelatedReasonsAlone(t *testing.T) {
|
||||
for _, c := range []struct{ reason, target string }{
|
||||
{"pg_dump: connection refused", "/mnt/sys_drive"},
|
||||
{"tar failed — /mnt/other: 1/2 GB used (50%), 1 GB free", "/mnt/sys_drive"},
|
||||
{"boom", ""},
|
||||
{"", "/mnt/sys_drive"},
|
||||
} {
|
||||
if got := trimRepeatedUsage(c.reason, c.target); got != c.reason {
|
||||
t.Errorf("reason %q (target %q) was altered to %q", c.reason, c.target, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -116,6 +116,18 @@ func (d *Dispatcher) ProcessEvent(customerID, eventType, severity, message, deta
|
||||
return
|
||||
}
|
||||
|
||||
// R-182: record-only types are written down and never mailed. Placed BEFORE the severity gate
|
||||
// so the row is written whatever the severity — the record must not inherit the notification's
|
||||
// conditions, which is the coupling this whole finding is about.
|
||||
if recordOnlyEvents[eventType] {
|
||||
if err := d.store.LogNotification(customerID, eventType, severity, message, "recorded",
|
||||
"record-only: the per-run digest (backup_run_failures) carries the notification", "operator"); err != nil {
|
||||
d.logger.Printf("[WARN] Failed to record %s for %s: %v", eventType, customerID, err)
|
||||
}
|
||||
d.logger.Printf("[INFO] Recorded (not mailed) %s for %s — the run digest is the notification", eventType, customerID)
|
||||
return
|
||||
}
|
||||
|
||||
// warning / error / critical trigger notifications. "info" is an intentional non-notify (status/
|
||||
// recovery events). Anything else is UNRECOGNIZED — log it (don't silently drop), so a bad severity
|
||||
// surfaces instead of vanishing (the felhom-pve-class lesson: a critical event must never be lost).
|
||||
@@ -260,15 +272,67 @@ func cooldownTierSuffix(detailsJSON string) string {
|
||||
return ":" + d.Tier
|
||||
}
|
||||
|
||||
// cooldownRunSuffix returns ":"+run_id when the event's details carry a non-empty `run_id`, else "".
|
||||
//
|
||||
// R-182. `cooldownTierSuffix`'s sibling, and deliberately a SEPARATE function rather than an extra
|
||||
// branch inside it: `tier` keeps byte-identical semantics for every type that uses it, so R-97a's
|
||||
// behaviour and its tests are untouched by this.
|
||||
//
|
||||
// WHY A BACKUP RUN NEEDS ONE. The run digest describes ONE RUN, and a box can have two in a day —
|
||||
// the nightly one and a manual one the operator triggered *because* something looked wrong. With no
|
||||
// run-scoped discriminator the 1-hour cooldown would swallow the second, which is the failure this
|
||||
// row exists to fix, reappearing one level up: the operator presses the button, the run fails, and
|
||||
// they are told nothing because the machine already wrote that hour.
|
||||
//
|
||||
// IT MAKES THE COOLDOWN EFFECTIVELY INERT FOR THIS TYPE, AND THAT IS THE INTENT, NOT AN OVERSIGHT.
|
||||
// A digest is already rate-limited by construction — one per run, emitted only when something
|
||||
// failed — so there is nothing for a timer to collapse. The cooldown protects against a repeating
|
||||
// identical alert; a digest cannot repeat, because each run is a different run.
|
||||
//
|
||||
// NARROW, LIKE ITS SIBLING: empty unless the producer opts in by sending a `run_id`, so no existing
|
||||
// event type's cooldown behaviour changes.
|
||||
func cooldownRunSuffix(detailsJSON string) string {
|
||||
if detailsJSON == "" || !strings.Contains(detailsJSON, "\"run_id\"") {
|
||||
return ""
|
||||
}
|
||||
var d struct {
|
||||
RunID string `json:"run_id"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || d.RunID == "" {
|
||||
return ""
|
||||
}
|
||||
return ":" + d.RunID
|
||||
}
|
||||
|
||||
func (d *Dispatcher) processOperator(customerID, eventType, severity, message, detailsJSON, source string) {
|
||||
if !d.operatorOn || d.operatorEmail == "" {
|
||||
return
|
||||
}
|
||||
|
||||
cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON)
|
||||
cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON) + cooldownRunSuffix(detailsJSON)
|
||||
d.mu.Lock()
|
||||
if last, ok := d.opCooldowns[cooldownKey]; ok && time.Since(last) < 1*time.Hour {
|
||||
d.mu.Unlock()
|
||||
// R-182: RECORD THE SUPPRESSION. This used to be a bare `return` — the event was dropped
|
||||
// before any LogNotification, so a cooldown drop and an event that never happened were
|
||||
// indistinguishable from the operator's side AND from the hub's own records.
|
||||
//
|
||||
// Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two emails were
|
||||
// sent, and the other seven left NO ROW ON ANY CHANNEL. The defect that hid was serious —
|
||||
// the cooldown key carries no app identifier, so the first refused app took the slot and
|
||||
// every other app's failure that hour was discarded — but the reason it took a day to find
|
||||
// the right way round is this line: there was nothing to read.
|
||||
//
|
||||
// "We chose not to e-mail you" and "nothing happened" must never look identical. This
|
||||
// applies to EVERY operator event, not only the one that exposed it. It makes the drop
|
||||
// visible; it deliberately does NOT change the cooldown's duration or semantics.
|
||||
if err := d.store.LogNotification(customerID, eventType, severity, message,
|
||||
"suppressed", "operator cooldown 1h, key="+cooldownKey, "operator"); err != nil {
|
||||
d.logger.Printf("[WARN] Failed to record suppressed operator notification for %s/%s: %v",
|
||||
customerID, eventType, err)
|
||||
}
|
||||
d.logger.Printf("[INFO] Operator email suppressed for %s/%s — cooldown (key=%s)",
|
||||
customerID, eventType, cooldownKey)
|
||||
return
|
||||
}
|
||||
d.opCooldowns[cooldownKey] = time.Now()
|
||||
@@ -285,6 +349,33 @@ func (d *Dispatcher) processOperator(customerID, eventType, severity, message, d
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "operator")
|
||||
}
|
||||
|
||||
// recordOnlyEvents are STORED and RECORDED but never e-mailed, on either channel.
|
||||
//
|
||||
// R-182. The distinction this register exists to make is the whole of that finding: **the record and
|
||||
// the notification are different things.** A per-app backup failure must always be written down —
|
||||
// every time, unconditionally, regardless of cooldowns, preferences or whether any mail went out —
|
||||
// and it must NOT compete for an e-mail slot, because the per-run digest
|
||||
// (`backup_run_failures`) is what a person is meant to read.
|
||||
//
|
||||
// Before this, `recovery_unit_capture_failed` was both at once, and it did neither well: on
|
||||
// 2026-08-03 nine of them arrived, two were e-mailed, and the other seven were dropped by the
|
||||
// 1-hour cooldown BEFORE anything was written down. So the operator was told about one app, the
|
||||
// other apps' failures were discarded, and nothing anywhere recorded that a choice had been made.
|
||||
//
|
||||
// WHY A REGISTER AND NOT severity "info". Downgrading the severity would have the same routing
|
||||
// effect — `severityNotifies` drops info — but it would also relabel a genuine failure as
|
||||
// informational in the events table, the operator UI and every historical query, and it would
|
||||
// silently drop the X-Priority handling if the type were ever promoted back. This says what it
|
||||
// means: not silent, not urgent, RECORDED.
|
||||
//
|
||||
// IT IS NOT A WAY TO MUTE THINGS. A type belongs here only when something else carries its
|
||||
// notification. Adding one with no digest behind it rebuilds the silence R-182 was filed against.
|
||||
var recordOnlyEvents = map[string]bool{
|
||||
// The per-app Tier-1 capture/refusal failure. Its notification is the run digest, which lists
|
||||
// every failed app in one mail; this row is the durable per-failure record behind it.
|
||||
"recovery_unit_capture_failed": true,
|
||||
}
|
||||
|
||||
// operatorOnlyEvents are event types that must NEVER reach a customer, whatever their preferences say.
|
||||
//
|
||||
// R-97c. This register exists because the guarantee it provides was previously ASSERTED IN A COMMENT
|
||||
@@ -315,8 +406,29 @@ var operatorOnlyEvents = map[string]bool{
|
||||
// a consequence of another type's routing — true today, and silently untrue the moment the failed
|
||||
// event becomes customer-visible. Belt, not inference.
|
||||
"whole_guest_backup_recovered": true,
|
||||
// R-158 / R-167 (D-c). A per-app Tier-1 recovery-unit capture failure. A customer can take no
|
||||
// action on it — the causes are a full filesystem, a permission fault or a broken dump, all of
|
||||
// which the operator resolves — and the alert carries operator-grade detail (target path, byte
|
||||
// figures, the raw error). The customer's half of D-c is the FILL WARNING, which fires BEFORE
|
||||
// this and is actionable: free space, delete files, add a drive.
|
||||
"recovery_unit_capture_failed": true,
|
||||
// R-182. The per-run backup digest. It is the same class as the line above and for the same
|
||||
// reason — a customer can act on a full disk (that is the fill warning, which fires first and
|
||||
// IS customer-facing) but not on a list of which apps' backups failed and why. It also carries
|
||||
// operator-grade detail: per-app leg names, raw refusal reasons and byte figures.
|
||||
//
|
||||
// Listed here rather than relying on the absence of a `customerMessages` entry, which is NOT a
|
||||
// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
|
||||
// once (v0.78.0) and the comment above records it.
|
||||
"backup_run_failures": true,
|
||||
}
|
||||
|
||||
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
|
||||
// api package can pin BOTH registers of a new event type in one test — allowlisted-but-not-
|
||||
// operator-only is the v0.78.0 defect, and it is only visible when the two are checked together.
|
||||
// Read-only: the register itself stays unexported so nothing can widen it at runtime.
|
||||
func IsOperatorOnly(eventType string) bool { return operatorOnlyEvents[eventType] }
|
||||
|
||||
func (d *Dispatcher) processCustomer(customerID, eventType, severity, message, detailsJSON, source string) {
|
||||
// R-97c: operator-tier events stop here, BEFORE prefs are consulted — the point is that no
|
||||
// customer configuration can opt in. Logged rather than dropped, so the skip is visible in
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func discardLogger() *log.Logger { return log.New(io.Discard, "", 0) }
|
||||
|
||||
// R-158 / R-167 Scenario G — the two new signals of decision D-c sit on OPPOSITE sides of the
|
||||
// operator/customer line, and each is tested through the real dispatch path rather than by reading
|
||||
// the register.
|
||||
//
|
||||
// D-c's rule, restated: a customer can free space, delete files or add a drive, so a FILL WARNING is
|
||||
// theirs. A customer can do nothing about a recovery-unit capture failure, so it is not.
|
||||
|
||||
// The operator half. Run under the BREAKING configuration — the customer has the event explicitly
|
||||
// enabled and has an email address — because that is the only configuration in which a missing
|
||||
// operatorOnlyEvents entry is visible. This is the v0.78.0 defect, demonstrated rather than argued.
|
||||
func TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer(t *testing.T) {
|
||||
st := opOnlyStore(t)
|
||||
if err := st.SaveNotificationPrefs("c1", "customer@example.com",
|
||||
[]string{"recovery_unit_capture_failed", "backup_failed"}, 6); err != nil {
|
||||
t.Fatalf("SaveNotificationPrefs: %v", err)
|
||||
}
|
||||
|
||||
rec := &sentTo{}
|
||||
d := opOnlyDispatcher(t, st, rec)
|
||||
d.ProcessEvent("c1", "recovery_unit_capture_failed", "error",
|
||||
`Recovery unit capture FAILED for "immich" — the app has no fresh local (Tier-1) backup`,
|
||||
`{"app":"immich","used_percent":100,"space_known":true}`, "controller")
|
||||
|
||||
for _, to := range rec.to {
|
||||
if to == "customer@example.com" {
|
||||
t.Fatalf("a customer was emailed the OPERATOR-ONLY recovery_unit_capture_failed (%s) — "+
|
||||
"enabled_events must not be able to opt in to a failure they cannot act on", to)
|
||||
}
|
||||
}
|
||||
|
||||
// R-182 CHANGED WHAT THIS ASSERTS, DELIBERATELY, AND THE OLD ASSERTION IS WORTH KEEPING IN VIEW.
|
||||
//
|
||||
// Until 2026-08-03 this test required the OPERATOR to be e-mailed here, on the grounds that "the
|
||||
// alert is the whole point of R-158". That was right when this event was the only signal, and it
|
||||
// is wrong now: measured, nine of these arrived at the hub and two were mailed, because the
|
||||
// operator cooldown key carries no app identifier — so as an alarm it told the operator about one
|
||||
// app and threw the rest away.
|
||||
//
|
||||
// The type is now RECORD-ONLY: written down every time, never mailed. R-158's guarantee — the
|
||||
// operator learns WHICH app failed and WHY — is not weakened, it MOVED: the per-run digest
|
||||
// `backup_run_failures` carries every failed app in one mail, and is pinned by
|
||||
// backup_run_digest_test.go. The customer safety claim below is untouched and is the reason this
|
||||
// test still exists.
|
||||
for _, to := range rec.to {
|
||||
if to == "operator@felhom.eu" {
|
||||
t.Fatal("the operator was e-mailed a PER-APP capture failure — this type is the record " +
|
||||
"now, not the alarm. One mail per failing app on a full disk is a dozen mails, which " +
|
||||
"is the volume problem the operator ruled against; the digest is the notification")
|
||||
}
|
||||
}
|
||||
|
||||
// The RECORD must exist, always. It is what makes the digest trustworthy: if the digest is ever
|
||||
// lost, delayed or suppressed, the failures are still individually written down. An absent row is
|
||||
// equally consistent with "correctly not mailed" and "the dispatcher never ran" — the positive
|
||||
// observable is the row itself (standing rule 3).
|
||||
logs, err := st.GetRecentNotifications("c1", 20)
|
||||
if err != nil {
|
||||
t.Fatalf("GetRecentNotifications: %v", err)
|
||||
}
|
||||
found := false
|
||||
for _, l := range logs {
|
||||
if l.EventType == "recovery_unit_capture_failed" && l.Status == "recorded" &&
|
||||
strings.Contains(l.ErrorMessage, "record-only") {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("the per-app failure left no 'recorded' row — a failure that is neither mailed nor "+
|
||||
"written down is exactly the 2026-08-03 defect, rebuilt; got %d row(s)", len(logs))
|
||||
}
|
||||
}
|
||||
|
||||
// The customer half. The fill warning MUST be delivered, and it must render the controller's own
|
||||
// Hungarian text — which names the drive and the free space — rather than a generic template or the
|
||||
// raw English.
|
||||
func TestDiskFillWarning_ReachesTheCustomerInHungarianWithItsNumbers(t *testing.T) {
|
||||
st := opOnlyStore(t)
|
||||
if err := st.SaveNotificationPrefs("c1", "customer@example.com", []string{"disk_warning"}, 6); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// The exact shape the controller sends from v0.191.0.
|
||||
const hun = `A(z) „Fotók" tároló 87%-osan megtelt — 4,2 GB szabad hely maradt. ` +
|
||||
`Szabadíts fel helyet (törölj felesleges fájlokat, vagy csatlakoztass új meghajtót), ` +
|
||||
`különben a biztonsági mentések hamarosan meghiúsulnak.`
|
||||
|
||||
var bodies []string
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "operator@felhom.eu", true, discardLogger())
|
||||
d.sendEmailFn = func(to, subject, body string, headers map[string]string) error {
|
||||
if to == "customer@example.com" {
|
||||
bodies = append(bodies, subject+"\n"+body)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
d.ProcessEvent("c1", "disk_warning", "warning", hun,
|
||||
`{"label":"Fotók","used_percent":87,"avail_gb":4.2}`, "controller")
|
||||
|
||||
if len(bodies) == 0 {
|
||||
t.Fatal("the customer was NOT warned that a disk is filling — this is the customer half of " +
|
||||
"decision D-c, and nothing reached them")
|
||||
}
|
||||
got := strings.Join(bodies, "\n")
|
||||
|
||||
// The controller's dynamic text must survive. A customerMessages entry for disk_warning would
|
||||
// OVERRIDE it (templates.go prefers the entry) and discard the drive name and the free space,
|
||||
// leaving the customer with "A lemezterület 90% felett van" and nothing to act on — which is
|
||||
// exactly why v0.191.0 removes those two entries.
|
||||
if !strings.Contains(got, "Fotók") {
|
||||
t.Fatalf("the customer email does not name the drive — a generic customerMessages entry has "+
|
||||
"discarded the controller's dynamic text (templates.go priority). Got:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "4,2 GB") {
|
||||
t.Fatalf("the customer email does not carry the free-space figure. Got:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "Szabadíts fel helyet") {
|
||||
t.Fatalf("the customer email does not tell the customer what to DO. Got:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The generic entries must STAY REMOVED. A well-meaning re-add would silently re-break the test
|
||||
// above's guarantee for every future reader — this pins the deletion itself.
|
||||
func TestDiskFillTypesHaveNoGenericCustomerMessage(t *testing.T) {
|
||||
for _, et := range []string{"disk_warning", "disk_critical"} {
|
||||
if msg, ok := customerMessages[et]; ok {
|
||||
t.Fatalf("customerMessages[%q] = %q — a static entry OVERRIDES the controller's dynamic "+
|
||||
"Hungarian text (FormatCustomerEmail prefers the entry), discarding the drive name "+
|
||||
"and the free-space figure the customer needs. Same reason offbox_enlarge_blocked "+
|
||||
"and disk_health_degraded deliberately have none", et, msg)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -41,6 +42,17 @@ Severity: %s
|
||||
Time: %s
|
||||
Message: %s`, customerID, eventType, severity, now, message)
|
||||
|
||||
// R-182: the backup run digest gets a rendered list instead of a raw JSON blob. It is the one
|
||||
// operator mail that carries a VARIABLE-LENGTH payload, and a dozen apps as one line of JSON is
|
||||
// unreadable on a phone at 07:00, which is the only time it matters.
|
||||
if eventType == "backup_run_failures" {
|
||||
if rendered, sub, ok := renderBackupRunFailures(customerID, detailsJSON); ok {
|
||||
return sub, body + rendered + fmt.Sprintf("\n\nDashboard: https://hub.felhom.eu/customers/%s", customerID)
|
||||
}
|
||||
// Unparseable details fall through to the raw form below rather than losing the mail. A
|
||||
// digest that renders badly still tells the operator something; a swallowed one does not.
|
||||
}
|
||||
|
||||
if detailsJSON != "" && detailsJSON != "{}" {
|
||||
body += fmt.Sprintf("\nDetails: %s", detailsJSON)
|
||||
}
|
||||
@@ -71,9 +83,20 @@ var customerMessages = map[string]string{
|
||||
"offbox_repo_orphaned": "A távoli mentési tároló elárvult: a benne lévő mentések egy korábbi, már nem elérhető kulccsal készültek (jellemzően újratelepítés után). Új mentés a tároló visszaállításáig nem készül — nyisd meg a Távoli mentés oldalt.",
|
||||
"offbox_repo_reset": "A távoli mentési tároló visszaállítva: a régi előzmény félretéve (nem törölve), és egy üres, új tároló jött létre a mostani kulccsal.",
|
||||
|
||||
// Disk events (GUEST — the controller's own cgroup view)
|
||||
"disk_warning": "A lemezterület 90% felett van — kérjük, szabadíts fel helyet.",
|
||||
"disk_critical": "A lemezterület kritikusan magas (95%+) — azonnali beavatkozás szükséges!",
|
||||
// Disk events (GUEST — the controller's own view) — `disk_warning` / `disk_critical`.
|
||||
//
|
||||
// DELIBERATELY NO ENTRY, from hub v0.89.0 / controller v0.191.0 (R-167, decision D-c). These two
|
||||
// types were allowlisted here, carried generic Hungarian copy, sat in the controller's
|
||||
// DefaultEnabledEvents and had a UI checkbox — and NOTHING IN ANY REPO EMITTED THEM. A complete
|
||||
// customer pipeline with no producer; the sixth "built but never wired" instance in this project.
|
||||
// The controller became their producer in v0.191.0.
|
||||
//
|
||||
// The producer sends a DYNAMIC Hungarian message naming the filesystem and its free space, so a
|
||||
// static entry here would be actively harmful: FormatCustomerEmail PREFERS the entry over the
|
||||
// message, so re-adding one would discard the drive name and the byte figures and leave the
|
||||
// customer with "A lemezterület 90% felett van" — a warning with nothing to act on. Same reason
|
||||
// `offbox_enlarge_blocked` and `disk_health_degraded` have no entry. Pinned by
|
||||
// TestDiskFillTypesHaveNoGenericCustomerMessage.
|
||||
|
||||
// Host disk events (the Proxmox HOST root filesystem — distinct from the guest disk above)
|
||||
"host_disk_warning": "A házszerver alaprendszerének (Proxmox-gazda) gyökérlemeze 90% felett van — kérjük, szabadíts fel helyet (pl. régi biztonsági mentések).",
|
||||
@@ -271,3 +294,115 @@ Ha nem te kérted ezt, hagyd figyelmen kívül ezt az e-mailt.
|
||||
Felhom.eu`, link)
|
||||
return subject, body
|
||||
}
|
||||
|
||||
// ──────────────────────────────────────────────────────────────────────
|
||||
// R-182 — the backup run digest
|
||||
// ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
// backupRunFailure is one app's failed leg within a run.
|
||||
type backupRunFailure struct {
|
||||
App string `json:"app"`
|
||||
Leg string `json:"leg"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
// backupRunDetails is the digest payload the controller sends.
|
||||
type backupRunDetails struct {
|
||||
RunID string `json:"run_id"`
|
||||
RunKind string `json:"run_kind"`
|
||||
Failed int `json:"failed"`
|
||||
Attempted int `json:"attempted"`
|
||||
TargetPath string `json:"target_path"`
|
||||
UsedGB float64 `json:"used_gb"`
|
||||
AvailGB float64 `json:"avail_gb"`
|
||||
TotalGB float64 `json:"total_gb"`
|
||||
UsedPercent float64 `json:"used_percent"`
|
||||
SpaceKnown bool `json:"space_known"`
|
||||
Apps []backupRunFailure `json:"apps"`
|
||||
}
|
||||
|
||||
// renderBackupRunFailures turns the digest details into an operator-readable block and a subject
|
||||
// that says the count without being opened. Returns ok=false when the payload cannot be parsed or
|
||||
// names no apps, so the caller can fall back to the raw rendering rather than mail an empty list.
|
||||
//
|
||||
// THE SUCCESS COUNT IS NOT DECORATION. "3 of 4 apps failed" is a catastrophe and "3 of 40" is a bad
|
||||
// night; the list alone cannot tell them apart, and the operator's first decision — get up now, or
|
||||
// look after coffee — depends entirely on which it is.
|
||||
func renderBackupRunFailures(customerID, detailsJSON string) (string, string, bool) {
|
||||
if detailsJSON == "" {
|
||||
return "", "", false
|
||||
}
|
||||
var d backupRunDetails
|
||||
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || len(d.Apps) == 0 {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
kind := d.RunKind
|
||||
if kind == "" {
|
||||
kind = "backup"
|
||||
}
|
||||
subject := fmt.Sprintf("[Felhom] 🔴 %s: %d of %d apps failed to back up (%s run)",
|
||||
customerID, d.Failed, d.Attempted, kind)
|
||||
|
||||
// Column-align the app names so the leg and reason line up and the block scans vertically.
|
||||
width := 0
|
||||
for _, a := range d.Apps {
|
||||
if len(a.App) > width {
|
||||
width = len(a.App)
|
||||
}
|
||||
}
|
||||
legWidth := 0
|
||||
for _, a := range d.Apps {
|
||||
if len(a.Leg) > legWidth {
|
||||
legWidth = len(a.Leg)
|
||||
}
|
||||
}
|
||||
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "\n\nFAILED: %d of %d apps attempted in this %s run.\n\n", d.Failed, d.Attempted, kind)
|
||||
for _, a := range d.Apps {
|
||||
reason := trimRepeatedUsage(a.Reason, d.TargetPath)
|
||||
if reason == "" {
|
||||
reason = "(no reason recorded)"
|
||||
}
|
||||
fmt.Fprintf(&b, " %-*s %-*s %s\n", width, a.App, legWidth, a.Leg, reason)
|
||||
}
|
||||
|
||||
// The space figures answer "is this one broken app or a full disk" before the reasons are read.
|
||||
// An absent reading renders as unavailable, never as zeros — "0 GB free" and "we could not look"
|
||||
// are opposite diagnoses (the UnitSpace rule, same reasoning, other side of the wire).
|
||||
if d.SpaceKnown {
|
||||
fmt.Fprintf(&b, "\nFilesystem: %s — %.1f/%.1f GB used (%.0f%%), %.1f GB free\n",
|
||||
d.TargetPath, d.UsedGB, d.TotalGB, d.UsedPercent, d.AvailGB)
|
||||
} else {
|
||||
fmt.Fprintf(&b, "\nFilesystem: %s — usage unavailable (the filesystem could not be read)\n", d.TargetPath)
|
||||
}
|
||||
|
||||
b.WriteString("\nEvery failure above is also recorded individually in the notification log,\n")
|
||||
b.WriteString("whether or not this mail was sent.")
|
||||
return b.String(), subject, true
|
||||
}
|
||||
|
||||
// trimRepeatedUsage strips the trailing "— /path: X/Y GB used (Z%), W GB free" clause from a per-app
|
||||
// reason, because the digest prints those figures ONCE for the whole run on its own line.
|
||||
//
|
||||
// This is a copy fix, and it was made after reading the first real digest rather than from the
|
||||
// design. The reserve's refusal message is authored for a single-app alert, where naming the
|
||||
// filesystem is exactly right; repeated down a list of a dozen apps it is the same forty characters
|
||||
// twelve times, and it pushes the part that differs off the right-hand edge of a phone screen at
|
||||
// 07:00 — which is the only moment this mail has to work.
|
||||
//
|
||||
// It trims ONLY an exact "— <target path>:" suffix, so a reason that mentions a different path, or
|
||||
// none, is left completely alone. A reason that is nothing but the usage clause is left alone too:
|
||||
// removing everything would turn a bad line into an empty one.
|
||||
func trimRepeatedUsage(reason, targetPath string) string {
|
||||
if reason == "" || targetPath == "" {
|
||||
return reason
|
||||
}
|
||||
marker := " — " + targetPath + ":"
|
||||
i := strings.LastIndex(reason, marker)
|
||||
if i <= 0 {
|
||||
return reason
|
||||
}
|
||||
return strings.TrimSpace(reason[:i])
|
||||
}
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-172 — the connection pragmas must actually be APPLIED, not merely requested.
|
||||
//
|
||||
// THE DEFECT THESE PIN. The DSN read `?_journal_mode=WAL&_busy_timeout=5000` — mattn/go-sqlite3
|
||||
// syntax — while the driver is modernc.org/sqlite, which ignores unknown parameters WITHOUT AN
|
||||
// ERROR. The hub therefore ran in rollback-journal mode with busy_timeout=0 for its entire life,
|
||||
// and nothing said so. Any test that asserted the DSN STRING would have passed throughout.
|
||||
//
|
||||
// So every assertion below reads the value back from the DATABASE.
|
||||
|
||||
// newPragmaStore is a sibling of host_test.go's newTestStore that also returns the DB PATH, because
|
||||
// two of the checks below are about the files SQLite creates beside it.
|
||||
func newPragmaStore(t *testing.T) (*Store, string) {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "hub.db")
|
||||
s, err := New(path, log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { s.Close() })
|
||||
return s, path
|
||||
}
|
||||
|
||||
func TestStorePragmasAreActuallyApplied(t *testing.T) {
|
||||
// RED-PROOF: restore the old DSN (`?_journal_mode=WAL&_busy_timeout=5000`) and this fails with
|
||||
// journal_mode=delete, busy_timeout=0 — i.e. it reproduces the shipped bug exactly.
|
||||
// Demonstrated in hub REPORT.md.
|
||||
s, _ := newPragmaStore(t)
|
||||
|
||||
var journal string
|
||||
if err := s.db.QueryRow("PRAGMA journal_mode").Scan(&journal); err != nil {
|
||||
t.Fatalf("read journal_mode: %v", err)
|
||||
}
|
||||
if journal != "wal" {
|
||||
t.Fatalf("journal_mode = %q, want \"wal\" — in rollback-journal mode a reader blocks a writer, "+
|
||||
"so rendering an operator page can 500 a host report (R-172)", journal)
|
||||
}
|
||||
|
||||
var busy int
|
||||
if err := s.db.QueryRow("PRAGMA busy_timeout").Scan(&busy); err != nil {
|
||||
t.Fatalf("read busy_timeout: %v", err)
|
||||
}
|
||||
if busy < 5000 {
|
||||
t.Fatalf("busy_timeout = %d, want >= 5000 — without it SQLite returns SQLITE_BUSY immediately "+
|
||||
"instead of waiting, and the hub turns that into an HTTP 500", busy)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStoreWALFilesExistWhileOpen(t *testing.T) {
|
||||
// The on-disk observable that EXPOSED the bug in production, pinned as a test: in WAL mode the
|
||||
// `-wal` and `-shm` files must exist beside an open database. Their absence on the live hub is
|
||||
// what proved the pragma was never applied, so it is the check to keep.
|
||||
s, path := newPragmaStore(t)
|
||||
|
||||
// A write guarantees the WAL is materialised rather than merely configured.
|
||||
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS r172_probe (k TEXT)`); err != nil {
|
||||
t.Fatalf("probe write: %v", err)
|
||||
}
|
||||
if _, err := s.db.Exec(`INSERT INTO r172_probe (k) VALUES ('x')`); err != nil {
|
||||
t.Fatalf("probe insert: %v", err)
|
||||
}
|
||||
|
||||
for _, suffix := range []string{"-wal", "-shm"} {
|
||||
if _, err := os.Stat(path + suffix); err != nil {
|
||||
t.Fatalf("%s is missing beside an OPEN database (%v) — this is exactly the signature that "+
|
||||
"proved the live hub was NOT in WAL mode", filepath.Base(path+suffix), err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStoreReaderDoesNotBlockWriter(t *testing.T) {
|
||||
// THE CONSEQUENCE, not the mechanism. A held READ transaction — what rendering an operator page
|
||||
// does — must not make a concurrent write fail. In rollback-journal mode it does, and that is the
|
||||
// whole of R-172: `Failed to save host-report …: database is locked (5) (SQLITE_BUSY)`.
|
||||
s, _ := newPragmaStore(t)
|
||||
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS r172_probe (k TEXT)`); err != nil {
|
||||
t.Fatalf("setup: %v", err)
|
||||
}
|
||||
if _, err := s.db.Exec(`INSERT INTO r172_probe (k) VALUES ('seed')`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
// Hold a read open for the whole write.
|
||||
rows, err := s.db.Query(`SELECT k FROM r172_probe`)
|
||||
if err != nil {
|
||||
t.Fatalf("open read: %v", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
if !rows.Next() {
|
||||
t.Fatal("expected a seeded row")
|
||||
}
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := s.db.Exec(`INSERT INTO r172_probe (k) VALUES ('concurrent')`)
|
||||
done <- err
|
||||
}()
|
||||
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
t.Fatalf("a write FAILED while a read was open: %v — this is the live 500 (R-172)", err)
|
||||
}
|
||||
case <-time.After(15 * time.Second):
|
||||
t.Fatal("a write BLOCKED indefinitely while a read was open")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStoreConcurrentWritersDoNotReturnBusy(t *testing.T) {
|
||||
// Writers still serialise under WAL; busy_timeout is what turns that into a WAIT rather than an
|
||||
// error. Several concurrent writers must all succeed — the host report, the event save and a UI
|
||||
// action genuinely do overlap on this hub.
|
||||
s, _ := newPragmaStore(t)
|
||||
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS r172_probe (k TEXT)`); err != nil {
|
||||
t.Fatalf("setup: %v", err)
|
||||
}
|
||||
|
||||
const writers = 8
|
||||
var wg sync.WaitGroup
|
||||
errs := make(chan error, writers)
|
||||
for i := 0; i < writers; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if _, err := s.db.Exec(`INSERT INTO r172_probe (k) VALUES ('w')`); err != nil {
|
||||
errs <- err
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
close(errs)
|
||||
for err := range errs {
|
||||
t.Fatalf("a concurrent writer returned an error instead of waiting: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStoreTransactionUpgradeDoesNotReturnBusySnapshot(t *testing.T) {
|
||||
// `_txlock=immediate` is the parameter that is easy to leave out, and WAL + busy_timeout alone
|
||||
// would not cover this. With a DEFERRED transaction (database/sql's default), a tx that reads and
|
||||
// then writes must upgrade its lock, and a failed upgrade is SQLITE_BUSY_SNAPSHOT — which
|
||||
// busy_timeout does NOT retry. This store has 10+ db.Begin() sites and they are all write paths.
|
||||
//
|
||||
// RED-PROOF: drop `&_txlock=immediate` from sqliteDSNParams and this test becomes able to fail
|
||||
// (it is inherently racy without it, which is precisely the point — an un-retryable error that
|
||||
// appears only under contention is the worst kind to ship). Demonstrated in hub REPORT.md.
|
||||
s, _ := newPragmaStore(t)
|
||||
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS r172_probe (k TEXT)`); err != nil {
|
||||
t.Fatalf("setup: %v", err)
|
||||
}
|
||||
if _, err := s.db.Exec(`INSERT INTO r172_probe (k) VALUES ('seed')`); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
|
||||
// Two read-then-write transactions racing is the upgrade shape.
|
||||
const txs = 6
|
||||
var wg sync.WaitGroup
|
||||
errs := make(chan error, txs)
|
||||
for i := 0; i < txs; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
tx, err := s.db.Begin()
|
||||
if err != nil {
|
||||
errs <- err
|
||||
return
|
||||
}
|
||||
var n int
|
||||
if err := tx.QueryRow(`SELECT COUNT(*) FROM r172_probe`).Scan(&n); err != nil {
|
||||
tx.Rollback()
|
||||
errs <- err
|
||||
return
|
||||
}
|
||||
if _, err := tx.Exec(`INSERT INTO r172_probe (k) VALUES ('tx')`); err != nil {
|
||||
tx.Rollback()
|
||||
errs <- err
|
||||
return
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
errs <- err
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
close(errs)
|
||||
for err := range errs {
|
||||
t.Fatalf("a read-then-write transaction failed under contention: %v — this is the "+
|
||||
"SQLITE_BUSY_SNAPSHOT that busy_timeout cannot retry, and _txlock=immediate prevents", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSQLiteDriverIgnoresMattnStyleParams is the regression guard for the ROOT CAUSE, not the symptom.
|
||||
//
|
||||
// It documents, executably, that the old DSN syntax is silently ignored by this driver — so anyone
|
||||
// who "tidies" the pragmas back to the more familiar mattn form gets a failing test instead of a
|
||||
// hub that quietly reverts to rollback-journal mode for another few months.
|
||||
func TestSQLiteDriverIgnoresMattnStyleParams(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "mattnstyle.db")
|
||||
|
||||
db, err := sql.Open("sqlite", path+"?_journal_mode=WAL&_busy_timeout=5000")
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
var journal string
|
||||
if err := db.QueryRow("PRAGMA journal_mode").Scan(&journal); err != nil {
|
||||
t.Fatalf("read journal_mode: %v", err)
|
||||
}
|
||||
if journal == "wal" {
|
||||
t.Skip("this driver now honours mattn-style parameters — the R-172 trap is gone; simplify " +
|
||||
"sqliteDSNParams and delete this test")
|
||||
}
|
||||
if journal != "delete" {
|
||||
t.Fatalf("journal_mode = %q, expected the driver to IGNORE the mattn-style parameter and leave "+
|
||||
"the default; if this changed, re-read sqliteDSNParams", journal)
|
||||
}
|
||||
}
|
||||
@@ -51,9 +51,45 @@ type CustomerSummary struct {
|
||||
DiskSummary string
|
||||
}
|
||||
|
||||
// sqliteDSNParams are the connection pragmas, and getting the SYNTAX right is the whole point.
|
||||
//
|
||||
// ── R-172: this DSN was WRONG for the hub's entire life, and it failed SILENTLY ──────────────────
|
||||
//
|
||||
// It used to read `?_journal_mode=WAL&_busy_timeout=5000`. That is **mattn/go-sqlite3** syntax. This
|
||||
// hub uses **modernc.org/sqlite**, whose `applyQueryParams` reads only `_pragma`, `_time_format`,
|
||||
// `_time_integer_format`, `_txlock` and `_inttotime` — anything else is **ignored without an error**.
|
||||
// So the hub ran in the default rollback-journal mode with busy_timeout=0 while its own source said
|
||||
// otherwise: a configuration asserting an invariant the code did not provide, the same class as the
|
||||
// comments in `CLAUDE.md`'s false-invariant table.
|
||||
//
|
||||
// The observable that proved it: a 128 MB `/data/hub.db` with **no `-wal`/`-shm` file beside it while
|
||||
// the database was open**. In WAL mode those files must exist. Consequence, measured on 2026-08-02:
|
||||
// 13 `SQLITE_BUSY` collisions in one pod lifetime, each returning HTTP 500 to a host report, and two
|
||||
// consecutive misses crossing the 30-minute staleness threshold — a false `host_stale` alarm plus an
|
||||
// operator e-mail for a host that was up and healthy throughout.
|
||||
//
|
||||
// Each parameter, and why it is not optional:
|
||||
//
|
||||
// - journal_mode(WAL) — in rollback-journal mode a writer excludes readers and vice versa, so
|
||||
// rendering an operator page could block a host report. WAL lets readers and one writer proceed
|
||||
// concurrently. It is a property of the DATABASE FILE, so it persists once set.
|
||||
// - busy_timeout(5000) — writers still serialise against each other. Without a timeout SQLite
|
||||
// returns SQLITE_BUSY *immediately* rather than waiting; 5 s is far longer than any write here.
|
||||
// - txlock=immediate — THE ONE THAT IS EASY TO MISS. `database/sql`'s Begin() is DEFERRED by
|
||||
// default, so a transaction that reads and then writes must upgrade its lock, and a failed
|
||||
// upgrade returns SQLITE_BUSY_SNAPSHOT, which **busy_timeout does not retry**. This store has
|
||||
// 10+ `db.Begin()` sites and they are all write paths (customer delete/reset, wg, appliance,
|
||||
// pbsdr, telemetry, log bundles). Taking the write lock up front converts that un-retryable
|
||||
// failure into an ordinary wait covered by busy_timeout above. WAL + busy_timeout WITHOUT this
|
||||
// would leave a known un-retryable path open and ship half a fix.
|
||||
//
|
||||
// TestStorePragmasAreActuallyApplied asserts what the DATABASE reports, never what string was passed
|
||||
// — asserting the DSN would have passed happily for the entire life of the bug.
|
||||
const sqliteDSNParams = "?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate"
|
||||
|
||||
// New creates a new store and initializes the schema.
|
||||
func New(dbPath string, logger *log.Logger) (*Store, error) {
|
||||
db, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL&_busy_timeout=5000")
|
||||
db, err := sql.Open("sqlite", dbPath+sqliteDSNParams)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("opening database: %w", err)
|
||||
}
|
||||
@@ -779,7 +815,7 @@ type NotificationLogEntry struct {
|
||||
EventType string
|
||||
Severity string
|
||||
Message string
|
||||
Status string // "sent", "skipped", "failed"
|
||||
Status string // "sent", "skipped", "failed", "suppressed" (R-182: a cooldown drop, recorded rather than silent)
|
||||
ErrorMessage string
|
||||
Channel string // "operator" or "customer"
|
||||
CreatedAt time.Time
|
||||
|
||||
@@ -20,12 +20,14 @@ import (
|
||||
|
||||
var validCustomerID = regexp.MustCompile(`^[a-zA-Z0-9.\-]+$`)
|
||||
|
||||
// hostInstallVersion is the felhom-host-install.sh version the customer page's install-command
|
||||
// generator targets. Kept in sync with scripts/felhom-host-install.sh SCRIPT_VERSION — the generator
|
||||
// only ever emits flags this version parses. Display-only (the Option-1 command downloads the served
|
||||
// script, which is always current); bump when the generator's flag surface follows a new script.
|
||||
// Drift is now gated: scripts/hostinstall_gates.py asserts this const == SCRIPT_VERSION (drill F-1).
|
||||
const hostInstallVersion = "1.19.0"
|
||||
// NOTE (R-94, 2026-08-02): there is deliberately NO host-install version constant here, and the
|
||||
// Setup tab renders no version number. The hub cannot know which version a box will run: the
|
||||
// Option-1 command downloads felhom-host-install.sh from the website at run time, and the website
|
||||
// git-syncs `main` every 30s (R-110). Any build-time literal here is a guess wearing a version
|
||||
// number's authority — the previous const said 1.19.0 while the served script was 1.22.0, and had
|
||||
// been wrong since 2026-07-14. The single version source is scripts/felhom-host-install.sh's
|
||||
// SCRIPT_VERSION; scripts/hostinstall_gates.py gate 1 now asserts this file's ABSENCE of any
|
||||
// host-install version literal.
|
||||
|
||||
// validSemver matches a bare X.Y.Z controller version (the floor format). Empty is also accepted by
|
||||
// the floor handlers (clears the override).
|
||||
@@ -331,9 +333,6 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
|
||||
CSRFField template.HTML
|
||||
CSRFToken string
|
||||
|
||||
// ScriptVersion drives the install-command generator's header (GL-7). Display-only.
|
||||
ScriptVersion string
|
||||
|
||||
// Hosts (v0.47.0): the customer's enrolled hosts for the Host tab — a LIST by design
|
||||
// (1 today, N for a later HA cluster). Each entry is the hostDetailData view-model map
|
||||
// the shared host_detail_body sub-template renders.
|
||||
@@ -484,8 +483,6 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
|
||||
CSRFField: s.csrfField(r),
|
||||
CSRFToken: s.csrfToken(r),
|
||||
|
||||
ScriptVersion: hostInstallVersion,
|
||||
|
||||
Hosts: hostViews,
|
||||
}
|
||||
|
||||
|
||||
@@ -215,10 +215,11 @@ func TestTemplates_InstallGenerator(t *testing.T) {
|
||||
t.Errorf("generator control missing: %s", id)
|
||||
}
|
||||
}
|
||||
// targets the right script version + carries the client-side customer id
|
||||
if !strings.Contains(html, hostInstallVersion) {
|
||||
t.Errorf("ScriptVersion %s not rendered", hostInstallVersion)
|
||||
}
|
||||
// carries the client-side customer id. There is deliberately NO version assertion here: R-94
|
||||
// deleted the rendered host-install version, because the hub cannot know which version a box
|
||||
// will run (the script is fetched at run time). The assertion that used to sit here compared
|
||||
// hostInstallVersion to itself and passed at any value — it was demonstrated green with the
|
||||
// const set to "9.9.9" while the served script was 1.22.0.
|
||||
if !strings.Contains(html, `data-customer-id="peti-felhom"`) {
|
||||
t.Errorf("generator missing data-customer-id")
|
||||
}
|
||||
|
||||
@@ -491,9 +491,10 @@
|
||||
<section class="card">
|
||||
<h2>Setup Command</h2>
|
||||
<p class="text-muted" style="margin-bottom: 1rem; font-size: 0.85rem;">
|
||||
Day-0 host bootstrap for host-install <strong>{{.ScriptVersion}}</strong>. Run on a
|
||||
freshly-PVE-installed Proxmox <strong>host</strong> as root (create the customer in the
|
||||
hub first). It enrolls the host, installs + verifies the agent, and provisions the guest;
|
||||
Day-0 host bootstrap. The command always fetches the <strong>current</strong>
|
||||
felhom-host-install.sh from felhom.eu — there is no version to pick here. Run it
|
||||
on a freshly-PVE-installed Proxmox <strong>host</strong> as root (create the customer in
|
||||
the hub first). It enrolls the host, installs + verifies the agent, and provisions the guest;
|
||||
the in-guest controller then pulls its own <code>controller.yaml</code>. The retrieval
|
||||
passphrase is entered at the no-echo prompt — never on the command line.
|
||||
</p>
|
||||
|
||||
+1
-1
@@ -125,7 +125,7 @@ spec:
|
||||
spec:
|
||||
containers:
|
||||
- name: hub
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.86.0
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.91.1
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
|
||||
+83
-3
@@ -71,8 +71,12 @@ data:
|
||||
# Host-install script. It lives at the repo's /scripts (outside the website doc-root),
|
||||
# synced into .../current/scripts by git-sync (see the sparse-checkout ConfigMap). Served
|
||||
# as text/plain so operators can inspect it in a browser before download-then-run.
|
||||
# R-110: served from the INSTALLER TAG's tree, not the website's. The URL is unchanged
|
||||
# (https://felhom.eu/scripts/felhom-host-install.sh) — it never carried a ref, so every
|
||||
# producer of it (the bootstrap script, the hub's day-0 command) follows the tag with no
|
||||
# edit. What changed is which tree this root points at.
|
||||
location /scripts/ {
|
||||
root /usr/share/nginx/html/current;
|
||||
root /usr/share/nginx/scripts/current;
|
||||
default_type text/plain;
|
||||
}
|
||||
|
||||
@@ -213,8 +217,12 @@ metadata:
|
||||
name: git-sync-sparse-checkout
|
||||
namespace: felhom-system
|
||||
data:
|
||||
# R-110: TWO sparse-checkouts, because there are now two syncs with two different refs.
|
||||
# The website tracks `main` (a copy edit must never need a release); /scripts/ tracks the
|
||||
# installer TAG (pushing the installer must never publish it).
|
||||
sparse-checkout: |
|
||||
/website/
|
||||
sparse-checkout-scripts: |
|
||||
/scripts/
|
||||
---
|
||||
# ===================
|
||||
@@ -246,6 +254,9 @@ spec:
|
||||
- name: git-data
|
||||
mountPath: /usr/share/nginx/html
|
||||
readOnly: true
|
||||
- name: git-data-scripts
|
||||
mountPath: /usr/share/nginx/scripts
|
||||
readOnly: true
|
||||
- name: nginx-config
|
||||
mountPath: /etc/nginx/conf.d/default.conf
|
||||
subPath: default.conf
|
||||
@@ -269,11 +280,14 @@ spec:
|
||||
initialDelaySeconds: 3
|
||||
periodSeconds: 10
|
||||
|
||||
# ── The WEBSITE sync — tracks `main`, unchanged cadence ──────────────────────────────
|
||||
# Deliberately still a branch: the site is content, and a typo fix must reach felhom.eu in
|
||||
# thirty seconds without cutting a release. Only /scripts/ moved to a tag (R-110).
|
||||
- name: git-sync
|
||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||
args:
|
||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||
- --branch=main
|
||||
- --ref=main
|
||||
- --root=/git
|
||||
- --link=current
|
||||
- --period=30s
|
||||
@@ -294,13 +308,52 @@ spec:
|
||||
securityContext:
|
||||
runAsUser: 65534 # nobody
|
||||
|
||||
# ── The INSTALLER sync — tracks a TAG (R-110, operator ruling 2026-08-03) ─────────────
|
||||
# felhom-host-install.sh runs as root on a virgin machine. Before this it was served
|
||||
# straight from `main`, so pushing it WAS publishing it: within thirty seconds it was what
|
||||
# every new machine downloaded and ran, with no staging and no rollback but another push.
|
||||
#
|
||||
# Publishing is now moving this tag; rolling back is moving it back. PROVEN, not assumed:
|
||||
# git-sync v4.4.0 follows a tag AND notices a moved one — measured 2026-08-03 on a
|
||||
# throwaway sync against this very repo (`update required … local:<old> remote:<new>` →
|
||||
# `updated successfully`, one period, ~20 s).
|
||||
#
|
||||
# Bump this ref when the installer's published version changes. `hostinstall_gates.py`
|
||||
# gate 6 fails if this sync stops naming an `installer-v…` tag.
|
||||
- name: git-sync-scripts
|
||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||
args:
|
||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||
- --ref=installer-v1.24.0
|
||||
- --root=/git-scripts
|
||||
- --link=current
|
||||
- --period=30s
|
||||
- --sparse-checkout-file=/etc/git-sync-scripts/sparse-checkout
|
||||
volumeMounts:
|
||||
- name: git-data-scripts
|
||||
mountPath: /git-scripts
|
||||
- name: sparse-checkout-scripts
|
||||
mountPath: /etc/git-sync-scripts
|
||||
resources:
|
||||
requests:
|
||||
memory: "32Mi"
|
||||
cpu: "10m"
|
||||
limits:
|
||||
memory: "128Mi"
|
||||
cpu: "100m"
|
||||
securityContext:
|
||||
runAsUser: 65534 # nobody
|
||||
|
||||
# Init container: wait for first sync before nginx starts
|
||||
initContainers:
|
||||
# BOTH trees are seeded before nginx accepts traffic. The second one is why /scripts/ has
|
||||
# no 404 window across this change: a fresh pod does not become ready until the installer
|
||||
# tag has been checked out, exactly as the website already worked.
|
||||
- name: git-sync-init
|
||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||
args:
|
||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||
- --branch=main
|
||||
- --ref=main
|
||||
- --root=/git
|
||||
- --link=current
|
||||
- --one-time
|
||||
@@ -312,16 +365,43 @@ spec:
|
||||
mountPath: /etc/git-sync
|
||||
securityContext:
|
||||
runAsUser: 65534
|
||||
- name: git-sync-scripts-init
|
||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||
args:
|
||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||
- --ref=installer-v1.24.0
|
||||
- --root=/git-scripts
|
||||
- --link=current
|
||||
- --one-time
|
||||
- --sparse-checkout-file=/etc/git-sync-scripts/sparse-checkout
|
||||
volumeMounts:
|
||||
- name: git-data-scripts
|
||||
mountPath: /git-scripts
|
||||
- name: sparse-checkout-scripts
|
||||
mountPath: /etc/git-sync-scripts
|
||||
securityContext:
|
||||
runAsUser: 65534
|
||||
|
||||
volumes:
|
||||
- name: git-data
|
||||
emptyDir: {}
|
||||
- name: git-data-scripts
|
||||
emptyDir: {}
|
||||
- name: nginx-config
|
||||
configMap:
|
||||
name: nginx-config
|
||||
- name: sparse-checkout
|
||||
configMap:
|
||||
name: git-sync-sparse-checkout
|
||||
items:
|
||||
- key: sparse-checkout
|
||||
path: sparse-checkout
|
||||
- name: sparse-checkout-scripts
|
||||
configMap:
|
||||
name: git-sync-sparse-checkout
|
||||
items:
|
||||
- key: sparse-checkout-scripts
|
||||
path: sparse-checkout
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
||||
@@ -1,3 +1,243 @@
|
||||
## 1.25.0 — the off-site tier stops asking to prune (2026-08-04, R-191)
|
||||
|
||||
**A backup that worked must not report failure.** The off-site tier was written with `keep_last: 2`,
|
||||
so every weekly run uploaded its snapshot successfully and then failed the whole job on a prune the
|
||||
box's token is deliberately refused: `prune 'ct/9201': permission check failed - missing
|
||||
Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the
|
||||
operator's inbox. Every week, on both boxes, about a backup that had already succeeded.
|
||||
|
||||
**R-89 moved off-site pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay
|
||||
write-only, so a box can never delete its own off-site history. The 2026-07-26 "two weeks" ruling was
|
||||
not reversed; where it is ENFORCED moved, and this value did not follow. The tier now writes
|
||||
`keep_last: 0`, which the agent's existing guard (`allowPBSPrune = !primary && keep_last > 0`) already
|
||||
reads as "never prune from the box" — no agent change needed.
|
||||
|
||||
**VERIFIED BEFORE CHANGING IT** (read-only on ep0, 2026-08-04): prune jobs `prune-demo-felhom` and
|
||||
`prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
|
||||
and have run **every day since 2026-07-27 — 18 tasks, all `status=OK`**, the newest showing
|
||||
`retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Disabling
|
||||
the client-side prune without that check would have traded a weekly false alarm for unbounded growth.
|
||||
|
||||
A gate now asserts the off-site tier carries no client-side prune, so the value cannot drift back
|
||||
quietly. The local tier's retention is untouched.
|
||||
|
||||
## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
|
||||
|
||||
**`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
|
||||
storage and calls `felhom-backup-target-apply grant` in the same breath — a box that builds its own
|
||||
target has always been fine. The **Scenario-F arm** — *"the target already exists, leave it exactly as
|
||||
it is"* — returned without ever granting.
|
||||
|
||||
So a box whose `felhom-backup` pre-dated the install (created by the vzdump-target-move runbook, or
|
||||
surviving a reinstall) ended up with `local_backup_target: felhom-backup` while its token held
|
||||
`FelhomAgentStore` on only `local`, `local-lvm` and `felhom-pbs`. Measured on **both** demo boxes
|
||||
2026-08-03: the content API answers `{"data":[]}` through the agent's token while root lists three
|
||||
archives. That tier was invisible to the agent and never restore-tested — and nothing said so,
|
||||
because an empty listing is also what a brand-new tier returns.
|
||||
|
||||
The reuse arm now ensures the ACL through the same guarded wrapper, so both arms leave the box in the
|
||||
same state. **Scenario F is unviolated:** the storage DEFINITION is still untouched — granting the
|
||||
role the agent is supposed to have on the target this script is about to write into `agent.json` is
|
||||
finishing the job, not retargeting the box. `pveum acl modify` is idempotent, so a box that already
|
||||
has the grant is unchanged and a box whose token was rotated gets it back.
|
||||
|
||||
`$BACKUP_TARGET_ID` is deliberately **not** added to `PVE_STORAGES`, and the comment now says why: that
|
||||
list is granted in step 4/5, before the target has been resolved in step 6, and `--acl-storages`
|
||||
entries are preflight-checked for existence. The grant belongs with the resolution, which is where it
|
||||
already was for a newly created target.
|
||||
|
||||
**A gate now asserts it** (`hostinstall_gates.py`): every arm of `configure_backup_target` that
|
||||
resolves the target must also grant on it. Red-proved by reverting the reuse arm — `resolves the
|
||||
backup target in 2 place(s) but grants in only 1`.
|
||||
|
||||
## v1.23.0 — the installer is published, not pushed (2026-08-03, R-110 + R-183)
|
||||
|
||||
**Two channels moved off `main` in the same change, because either one left behind makes the other
|
||||
cosmetic.**
|
||||
|
||||
**Channel 1 — the served script.** `manifests/webpage.yaml` git-synced `/scripts/` from
|
||||
`--branch=main` on a 30 s period and nginx served that working tree, so **pushing this file WAS
|
||||
publishing it**: within half a minute it was what every new machine downloaded and ran as root, with
|
||||
no staging and no rollback but another push. The sync is now **split in two**: the website keeps
|
||||
tracking `main` at the same cadence (a copy edit must never need a release), and `/scripts/` tracks
|
||||
the tag **`installer-v<SCRIPT_VERSION>`**. Publishing is moving that tag; rolling back is moving it
|
||||
back.
|
||||
|
||||
**PROVEN, not assumed:** git-sync v4.4.0 follows a tag *and* notices a **moved** one — measured on a
|
||||
throwaway sync against this repo, `update required … local:<old> remote:<new>` → `updated
|
||||
successfully`, within one period (~20 s). The moved-tag half is what the whole publish model rests
|
||||
on, so it was measured before the manifest was touched.
|
||||
|
||||
**Channel 2 — the sixteen files the installer fetches while it runs.** `fetch_raw` pulled from
|
||||
`$AGENT_REPO/raw/branch/main`. It now pulls from **`raw/tag/v$ART_AGENT_VER`** — the agent version the
|
||||
hub has vouched and whose binary sha this script already verifies.
|
||||
|
||||
**That is a correctness fix, not only a publish-channel one (→ R-183).** These are the AGENT's
|
||||
configs — its systemd unit, its sudoers, its guarded wrappers — and a fresh install was fetching the
|
||||
**vouched binary** while taking its configs from **whatever `main` held**. Two refs, one install, and
|
||||
nothing compared them. The right ref for them was never this script's `SCRIPT_VERSION`: they do not
|
||||
live in this repo and have no relationship to its version line.
|
||||
|
||||
**No fallback to a branch.** A vouched version whose tag is missing fails loudly rather than quietly
|
||||
serving `main` — a silent fallback is the appearance of control with none of it. `felhom-agent`
|
||||
carries `v<version>` tags from now on, `release-agent.sh` creates them, and `agent_gates.py` fails if
|
||||
the vouched version is not downloadable.
|
||||
|
||||
**Channel 3 — the URL — needed no change, and that is worth recording rather than leaving as a
|
||||
silence.** `https://felhom.eu/scripts/felhom-host-install.sh` never carried a ref: the ref lives in
|
||||
the manifest. So both producers of that URL (`scripts/iso/felhom-bootstrap.sh`, the hub's day-0
|
||||
command) follow the tag with no edit — **and no hub change, so no hub version bump.**
|
||||
|
||||
**Gate 6 in `hostinstall_gates.py`** pins all three structurally, with no network so it stays in
|
||||
`--fast` and runs in CI on every push: no `raw/branch/` ref anywhere in the installer; `fetch_raw`
|
||||
still pins to `$ART_AGENT_VER`; the manifest still syncs `/scripts/` from an `installer-v…` tag and
|
||||
the website still from `main`.
|
||||
|
||||
**It deliberately does NOT assert "a tag exists for the current SCRIPT_VERSION".** That gate would go
|
||||
red on the very push that bumps the version, before publishing — and publishing being a separate
|
||||
deliberate act is the entire ruling. A gate that fails on the normal path is one people learn to
|
||||
ignore.
|
||||
|
||||
## docs — v1.22.0 exercised end to end on two real reinstalls (2026-08-03, R-178) — **no script change**
|
||||
|
||||
**Nothing shipped.** `felhom-host-install.sh` stayed at **v1.22.0**; the published copy at
|
||||
`https://felhom.eu/scripts/felhom-host-install.sh` was confirmed byte-identical to the repo copy
|
||||
(`sha256 ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4`) before use. Both demo
|
||||
boxes were uninstalled and reinstalled with it, by **two deliberately different supply paths**:
|
||||
demo-hp with `--golden <local volid>` (the `:2584` alternative), demo-felhom with
|
||||
`--force-gitea-golden` (the canonical C.3 customer command). The merge-aware `step_grows` produced
|
||||
`data +46G (->70G, ONE volume)` and `+226G (->250G)` respectively, and `fetch_verify` was observed
|
||||
succeeding against the vouched manifest for **both** artifacts on demo-felhom
|
||||
(`verified sha256 a7763d31b55b5ce7…` agent, `verified sha256 54e2a4c431daf580…` golden).
|
||||
|
||||
**Two script-side findings, filed not fixed** (the session was a runbook; §7 forbade code):
|
||||
|
||||
- **R-180** — `--archive-storage` is validated for existence (`:1583`) and for golden resolution
|
||||
(`:1661`), but never against the ACL storage set it is about to grant (the fixed default
|
||||
`local local-lvm felhom-pbs`). Staging the golden on `felhom-backup` therefore passed every
|
||||
pre-flight gate and died at **step 8/8**: `HTTP 403: permission denied at /storage/felhom-backup
|
||||
(missing privilege Datastore.AllocateSpace)` — *after* step 2 minted the token, step 4b **rotated
|
||||
and vaulted root@pam**, and step 5 installed the agent. `ARCHIVE_STORAGE ∈ PVE_STORAGES` is a
|
||||
one-line assertion over two variables both known at `:1583`.
|
||||
- **R-179** — `--uninstall` leaves the NAS network-storage systemd units behind
|
||||
(`mnt-felhom\x2ddrives-<share>.{mount,automount}`; automount left `failed`, parent bind left
|
||||
mounted). The Part E residue-diff provenance is from **v1.9.1**, which predates the feature — and
|
||||
demo-felhom, which never had a share configured, left nothing, which is exactly why a diff on such
|
||||
a box reported clean.
|
||||
|
||||
Full evidence: root `REPORT.md`.
|
||||
|
||||
## host-install: one data volume, derived from the disk (2026-08-03, R-165)
|
||||
|
||||
**Forced by a census, not planned.** `felhom-agent` v0.120.0 merges the appliance's two data volumes
|
||||
into one (decision D-a). `step_grows` computed **two** numbers and the install call passed both, so
|
||||
this script had to change with the agent or every install would have provisioned a half-sized box.
|
||||
|
||||
- **`step_grows` computes ONE total.** The old 80/20 docker-vs-sysdata split is summed: `226` where it
|
||||
was `184 + 42`, `106` where it was `84 + 22`, `46` where it was `34 + 12`. **A standard appliance
|
||||
keeps exactly the capacity it had — 250 G — it is simply no longer split by a wall.**
|
||||
- **The size still comes from the physical disk.** `step_grows` already read the thin pool's real free
|
||||
space (`lvs /dev/pve/data`); the merge only collapsed its two outputs into one. This is what makes
|
||||
the merge safe to ship: an unflagged install does **not** get the golden's 24 G base.
|
||||
- **`--sysdata-grow` is DEPRECATED but still honoured.** It is no longer auto-computed (set to 0), and
|
||||
a hand-passed value still counts because the agent **folds** it into the single volume's grow rather
|
||||
than dropping it — so an operator reproducing an old command line gets the same total.
|
||||
|
||||
## CI — a Gitea Actions runner, and a red run that reaches a person (2026-08-02, R-168)
|
||||
|
||||
**No version bump anywhere: nothing in the product repos is compiled, built or deployed by this.**
|
||||
Recorded explicitly so the omission reads as a decision rather than a miss.
|
||||
|
||||
**What this closes.** Session 1 (same day) gave every repo one gate entry point and a
|
||||
`.githooks/pre-push` that refuses a failing push. That hook is per-clone and `--no-verify` skips it,
|
||||
so nothing independent of the person pushing ever saw whether the gates passed. This is the
|
||||
independent half, and with it **R-29 CLOSES** — on the demonstrated alarm, not on a green run.
|
||||
|
||||
**`.gitea/workflows/gates.yml` (new)** — triggers on `push`, `runs-on: felhom-gates`, obtains the
|
||||
source with a shallow `git fetch` of the **exact pushed SHA** from the in-cluster Gitea Service, and
|
||||
runs `scripts/repo_gates.py --fast` and nothing else. **No `uses:` step anywhere** — JavaScript
|
||||
actions need a node runtime the host-mode runner does not have, and probe P3 measured that a plain
|
||||
`git fetch` is sufficient and lands on the pushed commit. No `|| true`; the entry point's exit code
|
||||
IS the job's result.
|
||||
|
||||
**The alarm, which is the half that matters.** Probe P5 measured that a failed run produces **no
|
||||
mail, no notification row and no log line** from Gitea. A red tick in a web UI nobody watches is
|
||||
exactly the defect R-29 filed, rebuilt one layer up — so the workflow sends its own email on failure
|
||||
via Resend (the hub's existing transactional path) and **prints the provider's accepted id**, making
|
||||
"a message left the machine" an observable. **Demonstrated, not asserted:** a deliberately broken
|
||||
commit pushed with `--no-verify` produced run #6 `failure` and
|
||||
`RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`.
|
||||
|
||||
Two traps found while building it, both worth keeping because each looks like something else:
|
||||
the runner image has **no `curl`** on purpose (python3 and git only — so the step uses `urllib`
|
||||
rather than growing the image), and `api.resend.com` sits behind **Cloudflare, which 403s the default
|
||||
`Python-urllib` User-Agent with error 1010** — a failure that reads exactly like an auth failure and
|
||||
is not one.
|
||||
|
||||
**The standing limit, written into the workflow itself: it REPORTS, it cannot REFUSE.** Every repo
|
||||
pushes straight to `main` with no pull request, so there is no merge for a status check to stand at.
|
||||
That is not a gap in the runner; there is no gate in the road. Making it blocking needs branch
|
||||
protection plus a PR workflow, which changes how the operator works → **R-169**, waiting on them.
|
||||
|
||||
**`documentation/audits/SPIKE-ci-runner-2026-08-02.md` (new)** — all six probes, method, measurement
|
||||
and ruling; none produced a STOP. Also records a near-miss worth more than the probes: a `| tail -5`
|
||||
inside my own census query silently dropped rows and looked exactly like a baseline drift big enough
|
||||
to change the task. **An instrument that can drop results silently is not a measurement.**
|
||||
|
||||
**`CLAUDE.md`** gains the matching rule from session 1's red-proofing: a `go test -run` pattern that
|
||||
matches no test prints `ok` and exits 0, so a red-proof using `-run` must first prove the filter
|
||||
matched something.
|
||||
|
||||
**`CONTEXT.md`** gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector
|
||||
that tells no one is not finished), S-10 (the runner is unprivileged because DooPlex is Tier 2), and
|
||||
S-11 (CI reproduces the workspace's sibling layout, because two entry points depend on it).
|
||||
|
||||
## Gate enforcement — one entry point per repo, and a pre-push hook (2026-08-02)
|
||||
|
||||
**No version bump: `scripts/` carries no version, and this is tooling.** Recorded explicitly so the
|
||||
omission reads as a decision rather than a miss.
|
||||
|
||||
**The census that started it.** Thirteen gate scripts exist across the four felhom repos. A full run
|
||||
on 2026-08-02 found one clean correlation: **every check a `CLAUDE.md` tells a person to run was
|
||||
passing, and two of the four nobody is told to run were failing** — `hostinstall_gates.py` since
|
||||
14 July, and `reuse_refs_check.py` on all four repos. Both failures were harmless in effect, which
|
||||
was checked line by line; nothing would have said so if they had not been.
|
||||
|
||||
**`scripts/repo_gates.py` (new)** — THE entry point for this repo. Runs `site_gates`,
|
||||
`hostinstall_gates`, `hub_confirm_gate`, `manifest_bearer_gate` and `reuse_refs_check` on this root,
|
||||
streams each gate's own output, exits worst-wins non-zero, and reports exit 2 distinctly as
|
||||
INCONCLUSIVE. **A missing gate script is a FAILURE and prints the path tried** — fail-closed, because
|
||||
a runner that quietly skips a gate is the inert-seam failure this project has shipped four times. It
|
||||
copies `app-catalog-felhom.eu/scripts/catalog_gates.py` (R-161), **not** `site_gates.py`, which is a
|
||||
gate and not a runner — copying that would have produced a ninth monolith.
|
||||
|
||||
**`scripts/reuse_refs_check.py` — resolution taught, not loosened.** RED on all four repos with 13
|
||||
findings, of which a hand audit found **zero** genuine drift: twelve were package shorthand whose
|
||||
file sits a couple of directories deeper, and `wgsync/reconciler.go`, cited by the controller, lives
|
||||
in the hub. `REUSE.md` cites by package shorthand and across repos on purpose; the tool was wrong.
|
||||
New order, first hit wins: exact → suffix → ambiguous (real citation, imprecise shorthand — not a
|
||||
failure) → sibling repo (as-is, or with the sibling's own name stripped off the front) → FAIL.
|
||||
**Every non-exact hit is printed** and every root prints a per-rule tally, because "0 failures" alone
|
||||
cannot tell a working checker from a blind one. A failure lists every resolution attempted. Evidence
|
||||
trees (`audits/`, `documentation/tests/`) are excluded from the suffix index — a copy of a file is
|
||||
not the file. An absent sibling is never a failure; an unreadable parent says so and continues.
|
||||
Result: 13/13 resolve, all four roots exit 0.
|
||||
|
||||
**`scripts/test_reuse_refs_check.py` (new, 13 tests)** — one per resolution row plus the kill
|
||||
condition. Red-proof: making `resolve()` return `exact` for an unresolvable token turns four of them
|
||||
red. **`scripts/test_repo_gates.py` (new, 3 tests)** — a SEAM test asserting each member gate's own
|
||||
distinctive stdout, never the runner's summary line; red-proofed with an inert `run_gate` that still
|
||||
prints "all felhom.eu gates OK" and exits 0.
|
||||
|
||||
**`.githooks/pre-push` (new)** — runs `repo_gates.py --fast` and refuses the push. Its honest limits
|
||||
are written into the hook itself: it is **per-clone** (`core.hooksPath` is local config; arm with
|
||||
`git config core.hooksPath .githooks`, and any manual entry-point run WARNS when a clone is unarmed)
|
||||
and **`git push --no-verify` bypasses it on purpose** — an escape hatch that cannot be reached is one
|
||||
that gets removed the first time it is inconvenient; using it must be stated in the session report.
|
||||
Measured on git 2.47.3: a relative `core.hooksPath` resolves correctly and the hook's cwd is the repo
|
||||
root whether `git push` is issued from the root or any subdirectory. The half that is neither
|
||||
per-clone nor skippable is CI — now tracked as R-168.
|
||||
|
||||
## ISO v1.26.1 — the PUBLIC installer ISO, PUBLISHED (2026-07-31)
|
||||
|
||||
**Live at `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso`**
|
||||
|
||||
+103
-18
@@ -115,8 +115,8 @@
|
||||
# (default: appliance → island 169.254.253.1:8443; byo → vmbr0 IP:8443)
|
||||
# --no-island appliance only: keep the historical LAN bind instead of the R-50 island
|
||||
# --rootfs-grow N grow OS rootfs by N GiB (default: auto-compute)
|
||||
# --datavol-grow N grow Docker-data vol by N GiB (default: auto-compute)
|
||||
# --sysdata-grow N grow user-data vol by N GiB (default: auto-compute)
|
||||
# --datavol-grow N grow the single data volume by N GiB (default: auto-compute from the pool)
|
||||
# --sysdata-grow N DEPRECATED (R-165): added to --datavol-grow; there is one volume now
|
||||
#
|
||||
# Guest cap (appliance: optional — protect a SHARED host's other guests; byo: BOTH REQUIRED —
|
||||
# the only noisy-neighbor protection on a host you do not own; needs agent >= v0.52.0):
|
||||
@@ -184,9 +184,11 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_VERSION="1.22.0" # the SINGLE version source (F-1): -h, the run banners, and the hub
|
||||
# Setup-tab copy (hub internal/web/configs.go hostInstallVersion —
|
||||
# scripts/hostinstall_gates.py asserts the two stay equal) all follow it.
|
||||
SCRIPT_VERSION="1.25.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
||||
# The hub used to carry a copy for its Setup tab; R-94 DELETED it
|
||||
# (2026-08-02) because the hub cannot know which version a box runs —
|
||||
# the Setup command fetches this script at run time. scripts/
|
||||
# hostinstall_gates.py now asserts the hub carries NO version literal.
|
||||
# 1.16.0: the FELHOM_ESCROW sudoers alias (controller-driven escrow
|
||||
# ceremony, agent v0.88.0) ships on every install/update — it rides the
|
||||
# CANONICAL sudoers fetch below (configs/felhom-agent.sudoers from the
|
||||
@@ -305,6 +307,14 @@ PVE_PRIVS_STORE="Datastore.Allocate Datastore.AllocateSpace"
|
||||
PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit"
|
||||
# Storages the agent reads/writes (archive+dump=local, restore=local-lvm, offsite DR=felhom-pbs). The
|
||||
# offsite felhom-pbs MUST be included or the agent's DR backup 403s (SPIKE residual #1). --acl-storages overrides.
|
||||
#
|
||||
# `$BACKUP_TARGET_ID` is deliberately NOT in this list, and the reason matters because the obvious
|
||||
# tidy-up is to add it (R-185, 2026-08-03). This list is granted in STEP 4/5, before
|
||||
# configure_backup_target has run in STEP 6 — so at this point the target may not exist yet, and
|
||||
# --acl-storages entries are preflight-checked for existence. The target's grant therefore belongs
|
||||
# with the target's RESOLUTION, where it already is for a freshly created one, and now also for a
|
||||
# pre-existing one. Adding it here would grant on a storage that may not exist and would still leave
|
||||
# the resolution path as the single owner of that decision, split across two places.
|
||||
PVE_STORAGES=(local local-lvm felhom-pbs)
|
||||
# E-2: the whole-guest backup target storage id, and what configure_backup_target resolved to.
|
||||
# BACKUP_TARGET_RESOLVED feeds agent.json's local_backup_target; "local" means DEGRADED (Case B).
|
||||
@@ -490,12 +500,31 @@ fetch_verify() {
|
||||
# exists, else anonymous). These are non-executable text (not the integrity-checked binary); the
|
||||
# sudoers is `visudo -cf`-validated before install, which catches corruption/tampering that would
|
||||
# matter. $1=repo-path $2=dest
|
||||
#
|
||||
# R-110 / R-183: PINNED TO THE AGENT VERSION BEING INSTALLED, never to a branch.
|
||||
#
|
||||
# These sixteen files are the AGENT's configs — its systemd unit, its sudoers, its guarded wrappers —
|
||||
# so the ref that is correct for them is the agent version this run is installing, which the hub has
|
||||
# vouched and whose binary sha this script verifies. It is NOT the installer's own SCRIPT_VERSION:
|
||||
# these files do not live in the installer's repo and have no relationship to its version line.
|
||||
#
|
||||
# Before this they came from `raw/branch/main`, which is a REAL SKEW and not only a publish-channel
|
||||
# defect (R-183): a fresh install fetched the vouched agent BINARY while taking its unit file and
|
||||
# sudoers from whatever `main` happened to hold — two refs, one install, and nothing compared them.
|
||||
#
|
||||
# NO FALLBACK TO A BRANCH. A vouched version whose tag is missing must fail loudly here rather than
|
||||
# quietly serving `main`, because a silent fallback is exactly the "appearance of control with none of
|
||||
# it" this change exists to remove. `agent_gates.py`'s published-version gate keeps the tag and the
|
||||
# vouched version in step, so this die is a backstop and not the primary control.
|
||||
fetch_raw() {
|
||||
local path="$1" dest="$2"
|
||||
# Late steps (mgmt-watchdog, OOB) can run without step 5 having resolved the manifest.
|
||||
[[ -n "$ART_AGENT_VER" ]] || resolve_artifacts
|
||||
[[ -n "$ART_AGENT_VER" ]] || die "cannot pin $path: no agent version resolved from the hub manifest"
|
||||
local -a _auth; _git_auth_args _auth
|
||||
curl -fsS "${_auth[@]}" -o "$dest" \
|
||||
"$GITEA_BASE/$GITEA_OWNER/$AGENT_REPO/raw/branch/main/$path" \
|
||||
|| die "raw fetch failed: $path"
|
||||
"$GITEA_BASE/$GITEA_OWNER/$AGENT_REPO/raw/tag/v$ART_AGENT_VER/$path" \
|
||||
|| die "raw fetch failed: $path (agent tag v$ART_AGENT_VER — is that version tagged in $AGENT_REPO?)"
|
||||
[[ -s "$dest" ]] || die "raw fetch empty: $path"
|
||||
}
|
||||
|
||||
@@ -631,8 +660,31 @@ configure_backup_target() {
|
||||
# existing id is an error, and repointing a live target is exactly the silent retarget this
|
||||
# whole arc closes.
|
||||
if pvesm status --storage "$BACKUP_TARGET_ID" >/dev/null 2>&1; then
|
||||
log_skip " backup target '$BACKUP_TARGET_ID' already exists — leaving it exactly as it is (Scenario F)"
|
||||
log_skip " backup target '$BACKUP_TARGET_ID' already exists — leaving its DEFINITION exactly as it is (Scenario F)"
|
||||
BACKUP_TARGET_RESOLVED="$BACKUP_TARGET_ID"
|
||||
# R-185: …but STILL ensure the ACL. "The storage already exists" says nothing about whether
|
||||
# the agent may READ it, and this early return is where the two came apart.
|
||||
#
|
||||
# THE DEFECT THIS CLOSES, measured on both demo boxes 2026-08-03. The CASE A path below
|
||||
# creates the storage and grants in the same breath, so a box that built its own target is
|
||||
# fine. A box whose target ALREADY existed — created by the vzdump-target-move runbook, or
|
||||
# surviving a reinstall — returned here and never granted. The result: `local_backup_target`
|
||||
# pointed at `felhom-backup` while the token held FelhomAgentStore only on local, local-lvm
|
||||
# and felhom-pbs, so the API answered `{"data":[]}` for that storage while root saw three
|
||||
# archives. The tier was invisible to the agent and never restore-tested, and nothing said so
|
||||
# — because an empty listing is also what a brand-new tier returns.
|
||||
#
|
||||
# Scenario F is UNVIOLATED: the storage definition is still untouched. Granting a role the
|
||||
# agent is supposed to have on the target this same script is about to write into
|
||||
# agent.json is not "touching the box's target", it is finishing the job. `pveum acl modify`
|
||||
# is idempotent, so a box that already has the grant is unchanged, and a box that had its
|
||||
# token rotated gets it back.
|
||||
if $DRY_RUN; then
|
||||
log_dry "felhom-backup-target-apply grant $BACKUP_TARGET_ID # R-185: ACL on a pre-existing target"
|
||||
else
|
||||
/usr/local/sbin/felhom-backup-target-apply grant "$BACKUP_TARGET_ID" \
|
||||
|| die "backup target grant failed on the pre-existing target — the agent could not read its own backup tier (R-185)"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
local mp
|
||||
@@ -1770,23 +1822,34 @@ step_token() {
|
||||
#-------------------------------------------------------------------------------
|
||||
step_grows() {
|
||||
log_step "3/8 compute volume grows"
|
||||
# Golden base: rootfs 32G + Docker-data 16G + user-data 8G (build-golden.sh).
|
||||
# Golden base since build-golden.sh v3.0.0 (R-165): rootfs 32G + ONE data volume 24G. The separate
|
||||
# 8G user-data volume was MERGED AWAY — one volume, one free-space figure, no ceiling — so there is
|
||||
# one number to compute here instead of two.
|
||||
#
|
||||
# THE SIZE IS DERIVED FROM THE PHYSICAL DISK, which is what makes the merge safe to ship: an
|
||||
# unflagged install does NOT get the golden's 24G, it gets a share of the thin pool's real free
|
||||
# space. (Before R-165 this same block already did the deriving; the merge only collapsed its
|
||||
# 80/20 docker-vs-sysdata split into a single total.)
|
||||
if [[ -z "$ROOTFS_GROW$DATAVOL_GROW$SYSDATA_GROW" ]]; then
|
||||
local free_gib
|
||||
free_gib=$(lvs --noheadings --units g -o lv_size,data_percent /dev/pve/data 2>/dev/null | awk '{gsub(/[^0-9.]/,"",$1); used=$2; print int($1*(100-used)/100)}' 2>/dev/null || echo 0)
|
||||
# Reserve headroom; split the rest ~ docker 80% / sysdata 20%; rootfs stays golden.
|
||||
# Reserve headroom; the totals below are the pre-merge pair SUMMED, so an appliance gets the
|
||||
# same capacity it did before — it is simply no longer split by a wall.
|
||||
ROOTFS_GROW=0
|
||||
if [[ "${free_gib:-0}" -ge 300 ]]; then
|
||||
DATAVOL_GROW=184; SYSDATA_GROW=42 # reproduces the standard 200G/50G appliance
|
||||
DATAVOL_GROW=226 # 184+42 -> the standard 250G appliance (was 200G+50G)
|
||||
elif [[ "${free_gib:-0}" -ge 150 ]]; then
|
||||
DATAVOL_GROW=84; SYSDATA_GROW=22
|
||||
DATAVOL_GROW=106 # 84+22
|
||||
else
|
||||
DATAVOL_GROW=34; SYSDATA_GROW=12 # minimal floors
|
||||
DATAVOL_GROW=46 # 34+12 — minimal floor
|
||||
fi
|
||||
log_info " auto-computed from ~${free_gib} GiB free"
|
||||
SYSDATA_GROW=0
|
||||
log_info " auto-computed from ~${free_gib} GiB free (ONE volume since R-165)"
|
||||
fi
|
||||
ROOTFS_GROW="${ROOTFS_GROW:-0}"; DATAVOL_GROW="${DATAVOL_GROW:-0}"; SYSDATA_GROW="${SYSDATA_GROW:-0}"
|
||||
log_info " grows: rootfs +${ROOTFS_GROW}G (->$((32+ROOTFS_GROW))G), docker +${DATAVOL_GROW}G (->$((16+DATAVOL_GROW))G), sys_drive +${SYSDATA_GROW}G (->$((8+SYSDATA_GROW))G)"
|
||||
# A hand-passed --sysdata-grow is still ACCEPTED and still counts: the agent folds it into the one
|
||||
# volume (bringup.go 4b), so an operator reproducing an old command line gets the same total.
|
||||
log_info " grows: rootfs +${ROOTFS_GROW}G (->$((32+ROOTFS_GROW))G), data +$((DATAVOL_GROW+SYSDATA_GROW))G (->$((24+DATAVOL_GROW+SYSDATA_GROW))G, ONE volume)"
|
||||
_state_mark grows
|
||||
}
|
||||
|
||||
@@ -2416,8 +2479,30 @@ for _k,_v in {"unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agen
|
||||
base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20})
|
||||
# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions —
|
||||
# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob
|
||||
# would guarantee that one of them is wrong. keep_last=2 on the offsite tier = two weeks (operator
|
||||
# ruling 2026-07-26).
|
||||
# would guarantee that one of them is wrong.
|
||||
#
|
||||
# THE OFFSITE TIER CARRIES NO CLIENT-SIDE RETENTION, AND THAT IS THE POINT (R-191, 2026-08-04).
|
||||
# It used to be written `keep_last: 2` ("two weeks", operator ruling 2026-07-26). **R-89 then moved
|
||||
# offsite pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay write-only,
|
||||
# deliberately, so that a box can never delete its own offsite history. The 2026-07-26 ruling was not
|
||||
# reversed; where it is ENFORCED moved, and this value did not follow.
|
||||
#
|
||||
# The consequence was weekly and wrong in the worst direction: vzdump UPLOADED the snapshot fine and
|
||||
# then failed the whole job on the prune the token is refused —
|
||||
# `prune 'ct/9201': permission check failed - missing Datastore.Modify|Datastore.Prune`
|
||||
# → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the operator's inbox. Every week, on
|
||||
# both boxes, about a backup that had already succeeded. A tier that cries wolf weekly is a tier
|
||||
# whose real failure nobody will see.
|
||||
#
|
||||
# VERIFIED BEFORE CHANGING IT (2026-08-04, read-only on ep0): prune jobs `prune-demo-felhom` and
|
||||
# `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
|
||||
# and have run EVERY DAY since 2026-07-27 — 18 tasks, all `status=OK`, the newest showing
|
||||
# `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Retention
|
||||
# happens; it happens THERE. **If that ever stops being true, this zero is unbounded growth** — check
|
||||
# ep0's prune jobs before assuming the offsite tier is retained.
|
||||
#
|
||||
# `keep_last: 0` means "never prune from the box" and is the value the agent's own guard reads
|
||||
# (allowPBSPrune = !primary && keep_last > 0), so no agent change is needed to honour it.
|
||||
#
|
||||
# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only
|
||||
# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent
|
||||
@@ -2428,7 +2513,7 @@ base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_sec
|
||||
# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here —
|
||||
# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start
|
||||
# writing to an offsite datastore.
|
||||
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]})
|
||||
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":0}]})
|
||||
base.setdefault('local_api', {})
|
||||
base['local_api'].setdefault('enable', True)
|
||||
base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']
|
||||
|
||||
+166
-17
@@ -5,9 +5,17 @@ drill-swept findings (DRILL-day0-vm-2026-07-12 F-1/F-7/F-9/F-10 + the ACL-narrow
|
||||
Run from the repo root: python scripts/hostinstall_gates.py
|
||||
|
||||
Gates (all must pass; non-zero exit on any failure):
|
||||
1. version — exactly ONE version source: SCRIPT_VERSION exists, the header line carries
|
||||
no version literal, and the hub Setup-tab const (hub internal/web/configs.go
|
||||
hostInstallVersion) equals SCRIPT_VERSION (F-1 structural fix)
|
||||
1. version — exactly ONE version source: SCRIPT_VERSION exists, the header line carries no
|
||||
version literal, and **the hub carries no host-install version literal at all**.
|
||||
The third assertion inverted on 2026-08-02 (R-94): it used to require the hub's
|
||||
`hostInstallVersion` const to EQUAL SCRIPT_VERSION, which is unachievable
|
||||
honestly — the Option-1 install command downloads felhom-host-install.sh from
|
||||
the website at RUN TIME and the website git-syncs `main` every 30 seconds
|
||||
(R-110), so the hub cannot know which version a given box will run. A
|
||||
build-time literal there is a guess with a version number's authority, and the
|
||||
real one drifted to 1.19.0-vs-1.22.0 and stayed wrong for 19 days. The label was
|
||||
deleted rather than derived; this gate now pins its absence (F-1 structural fix,
|
||||
second form).
|
||||
2. age — the `age` package is installed by the agent-install step (F-10)
|
||||
3. pbs-apply — configs/felhom-pbs-apply is fetched + installed to
|
||||
/usr/local/sbin/felhom-pbs-apply (F-7), and the uninstall removes it
|
||||
@@ -54,21 +62,49 @@ if re.search(r'felhom-host-install\.sh\s+v\d+\.\d+\.\d+', header):
|
||||
else:
|
||||
ok("header has no version literal")
|
||||
|
||||
# hub Setup-tab const must equal SCRIPT_VERSION (the copy the drill found at 1.12.0).
|
||||
if os.path.exists(HUB_CONFIGS) and script_ver:
|
||||
with io.open(HUB_CONFIGS, "r", encoding="utf-8") as f:
|
||||
hub_src = f.read()
|
||||
hm = re.search(r'hostInstallVersion\s*=\s*"(\d+\.\d+\.\d+)"', hub_src)
|
||||
if not hm:
|
||||
fail("hub hostInstallVersion const not found in internal/web/configs.go")
|
||||
elif hm.group(1) != script_ver:
|
||||
fail("hub Setup-tab hostInstallVersion=%s != SCRIPT_VERSION=%s (F-1: bump both together)"
|
||||
% (hm.group(1), script_ver))
|
||||
else:
|
||||
ok("hub Setup-tab hostInstallVersion matches (%s)" % hm.group(1))
|
||||
# The hub must carry NO host-install version literal at all (R-94, 2026-08-02). It cannot know
|
||||
# which version a box will run — the Option-1 command fetches the script from the website at run
|
||||
# time and the website git-syncs `main` every 30s. The const this replaced said 1.19.0 while the
|
||||
# served script was 1.22.0, and had been wrong since 2026-07-14.
|
||||
#
|
||||
# Matched in CODE SHAPES, never as bare prose: the deleted declarations, the struct field, the
|
||||
# assignment and the template action, plus a rename-proof generic form of each. Comments are
|
||||
# deliberately NOT stripped (a `//` inside a URL string literal would truncate the scan and turn
|
||||
# this gate blind); a comment that merely NAMES the identifier is allowed, and configs.go carries
|
||||
# exactly such a note explaining the absence.
|
||||
BANNED = [
|
||||
(r'\bconst\s+hostInstallVersion\b', "const hostInstallVersion"),
|
||||
(r'\bhostInstallVersion\s*=', "hostInstallVersion assignment"),
|
||||
(r'(?i)\bconst\s+\w*hostinstall\w*version\b', "a renamed host-install version const"),
|
||||
(r'\bScriptVersion\s+string\b', "ScriptVersion struct field"),
|
||||
(r'\bScriptVersion\s*:', "ScriptVersion struct assignment"),
|
||||
(r'\{\{\s*\.ScriptVersion\s*\}\}', "{{.ScriptVersion}} template action"),
|
||||
]
|
||||
HUB_DIR = os.path.join(ROOT, "hub")
|
||||
if not os.path.isdir(HUB_DIR):
|
||||
fail("hub/ not found at %s — cannot assert the absence of a host-install version literal" % HUB_DIR)
|
||||
else:
|
||||
if not os.path.exists(HUB_CONFIGS):
|
||||
fail("hub/internal/web/configs.go not found — cannot cross-check the Setup-tab version")
|
||||
scanned, hits = 0, 0
|
||||
for dirpath, dirs, files in os.walk(HUB_DIR):
|
||||
dirs[:] = [d for d in dirs if d not in (".git", "vendor", "node_modules")]
|
||||
for fn in files:
|
||||
if not (fn.endswith(".go") or fn.endswith(".html")):
|
||||
continue
|
||||
fp = os.path.join(dirpath, fn)
|
||||
scanned += 1
|
||||
with io.open(fp, "r", encoding="utf-8") as f:
|
||||
for lineno, line in enumerate(f, 1):
|
||||
for pat, what in BANNED:
|
||||
if re.search(pat, line):
|
||||
hits += 1
|
||||
fail("%s:%d carries %s — the hub must render NO host-install version "
|
||||
"(R-94: the served script is fetched at run time, so no build-time "
|
||||
"value can be true). Single source: scripts/felhom-host-install.sh "
|
||||
"SCRIPT_VERSION. Line: %s"
|
||||
% (os.path.relpath(fp, ROOT), lineno, what, line.strip()[:120]))
|
||||
if not hits:
|
||||
ok("hub carries no host-install version literal (%d .go/.html files scanned, %d shapes checked)"
|
||||
% (scanned, len(BANNED)))
|
||||
|
||||
# ── 2. age package (F-10) ───────────────────────────────────────────────────────
|
||||
# must match the REAL install invocation, not the log_dry echo (red-proof-hardened twice:
|
||||
@@ -110,6 +146,119 @@ if re.search(r'^PVE_STORAGES=\([^)]*felhom-pbs[^)]*\)', src, re.M):
|
||||
else:
|
||||
fail("felhom-pbs missing from the default PVE_STORAGES — narrowing it 403s the PBS-DR apply-bridge")
|
||||
|
||||
# ── 6. the publish channel is pinned, not floating (R-110 / R-183) ──────────────
|
||||
#
|
||||
# WHAT THIS ASSERTS, AND WHAT IT DELIBERATELY DOES NOT.
|
||||
#
|
||||
# It does NOT assert "a tag exists for the current SCRIPT_VERSION". That gate would fail the very
|
||||
# push that bumps SCRIPT_VERSION, before publishing has happened — and publishing being a SEPARATE
|
||||
# deliberate act is the whole point of R-110's ruling. A gate that goes red on the normal path is a
|
||||
# gate people learn to ignore, which is the reasoning the task's own §8.4 applies to the agent-side
|
||||
# gate; it applies here identically. "Is the vouched version actually downloadable" is a real
|
||||
# invariant and it lives where a missing artifact genuinely breaks day-0 — `felhom-agent`'s
|
||||
# `agent_gates.py`, which has the network access to answer it.
|
||||
#
|
||||
# What it asserts instead are the two STRUCTURAL regressions that would silently return the
|
||||
# installer to a floating channel, both answerable by reading files (no network, so this stays in
|
||||
# `--fast` and therefore runs in CI on every push):
|
||||
#
|
||||
# 6a. no `raw/branch/` ref anywhere in the installer — one of the sixteen agent-config fetches
|
||||
# slipping back to `main` is exactly how a channel stays floating unnoticed, and it is
|
||||
# invisible in a diff that touches one line.
|
||||
# 6b. `fetch_raw` still pins to the resolved agent version — the positive form, so the mechanism
|
||||
# cannot be quietly deleted rather than regressed.
|
||||
# 6c. the website manifest still syncs `/scripts/` from a TAG ref and the website from `main` —
|
||||
# the split is the deploy-side half of the same channel, and reverting it is one word.
|
||||
branch_refs = [l for l in lines if "raw/branch/" in l and not l.lstrip().startswith("#")]
|
||||
if branch_refs:
|
||||
fail("installer still fetches from a BRANCH ref — the run-time channel is floating again "
|
||||
"(R-110/R-183). Offending line(s): %s" % "; ".join(l.strip()[:90] for l in branch_refs))
|
||||
else:
|
||||
ok("no raw/branch/ ref in the installer — every run-time fetch is pinned")
|
||||
|
||||
if re.search(r'raw/tag/v\$ART_AGENT_VER/', src):
|
||||
ok("fetch_raw pins the agent configs to the vouched agent version")
|
||||
else:
|
||||
fail("fetch_raw no longer pins to $ART_AGENT_VER — the agent's configs and its binary can "
|
||||
"again come from different refs in one install (R-183)")
|
||||
|
||||
WEBPAGE = os.path.join(ROOT, "manifests", "webpage.yaml")
|
||||
try:
|
||||
with io.open(WEBPAGE, "r", encoding="utf-8") as f:
|
||||
wp = f.read()
|
||||
except IOError as e:
|
||||
fail("cannot read manifests/webpage.yaml to check the publish channel: %s" % e)
|
||||
wp = None
|
||||
if wp is not None:
|
||||
# The scripts sync must name a tag ref; the website sync must still track main.
|
||||
if re.search(r'--ref=installer-v', wp):
|
||||
ok("manifest: /scripts/ syncs from an installer tag")
|
||||
else:
|
||||
fail("manifests/webpage.yaml has no `--ref=installer-v…` sync — /scripts/ is not served "
|
||||
"from a tag, so pushing the installer publishes it again (R-110)")
|
||||
if re.search(r'--(branch|ref)=main', wp):
|
||||
ok("manifest: the website still tracks main (a copy edit must not need a release)")
|
||||
else:
|
||||
fail("manifests/webpage.yaml no longer tracks main for the website — pinning the SITE to "
|
||||
"the installer tag turns every copy edit into a release")
|
||||
|
||||
# ── R-185: every path that RESOLVES the backup target must also grant on it ──────────────────
|
||||
#
|
||||
# THE DEFECT THIS WOULD HAVE CAUGHT, measured on both demo boxes 2026-08-03. `configure_backup_target`
|
||||
# has two arms. The CASE A arm creates the storage and grants in the same breath. The Scenario-F arm —
|
||||
# "the target already exists, leave it alone" — returned WITHOUT granting, so a box whose target
|
||||
# pre-dated the install pointed `local_backup_target` at a storage its own token could not read. The
|
||||
# API answered `{"data":[]}` while root saw three archives, and nothing said so, because an empty
|
||||
# listing is also what a brand-new tier returns.
|
||||
#
|
||||
# The assertion is deliberately about the FUNCTION, not about PVE_STORAGES: the target's grant belongs
|
||||
# with the target's resolution (PVE_STORAGES is granted a step earlier, before the target exists), so
|
||||
# what must hold is that no arm of that function can resolve a target and skip the grant.
|
||||
fn = re.search(r'^configure_backup_target\(\)\s*\{(.*?)^\}', src, re.S | re.M)
|
||||
if not fn:
|
||||
fail("cannot find configure_backup_target() — the backup-target ACL assertion cannot run, and a "
|
||||
"check that cannot run must never report OK (R-185)")
|
||||
else:
|
||||
body = fn.group(1)
|
||||
resolutions = len(re.findall(r'BACKUP_TARGET_RESOLVED="\$BACKUP_TARGET_ID"', body))
|
||||
grants = len(re.findall(r'felhom-backup-target-apply grant', body))
|
||||
if resolutions == 0:
|
||||
fail("configure_backup_target no longer resolves BACKUP_TARGET_ID anywhere — re-read it")
|
||||
elif grants >= resolutions:
|
||||
ok("every arm that resolves the backup target also grants on it (%d resolution(s), %d grant(s))"
|
||||
% (resolutions, grants))
|
||||
else:
|
||||
fail("configure_backup_target resolves the backup target in %d place(s) but grants in only %d "
|
||||
"— an arm resolves a target the agent may not READ. That is R-185: the tier's archives are "
|
||||
"invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
|
||||
"like a brand-new tier." % (resolutions, grants))
|
||||
|
||||
# ── R-191: the OFFSITE tier must not arm a client-side prune ─────────────────────────────────
|
||||
#
|
||||
# R-89 moved offsite pruning SERVER-SIDE — ep0 runs a per-namespace prune job and box tokens stay
|
||||
# write-only, so the box is REFUSED if it asks. When this default was `keep_last: 2` the effect was a
|
||||
# weekly lie: vzdump uploaded the snapshot, then failed the whole job on the prune, and the operator
|
||||
# was told the offsite backup had failed when it had succeeded.
|
||||
#
|
||||
# The assertion is on the OFFSITE entry only. The local tier's `local_backup_retention` is untouched
|
||||
# and must stay untouched — it prunes correctly and is allowed to.
|
||||
m = re.search(r'"backup_targets":\s*\[(.*?)\]', src, re.S)
|
||||
if not m:
|
||||
fail("cannot find backup_targets in the rendered agent.json defaults — the offsite-retention "
|
||||
"assertion cannot run, and a check that cannot run must never report OK (R-191)")
|
||||
else:
|
||||
targets = m.group(1)
|
||||
kl = re.search(r'"keep_last"\s*:\s*(\d+)', targets)
|
||||
if not kl:
|
||||
fail("the offsite backup_target carries no keep_last at all — expected an explicit 0 "
|
||||
"(R-191: 0 means 'never prune from the box'; absent is not the same statement)")
|
||||
elif kl.group(1) != "0":
|
||||
fail("the offsite backup_target arms a CLIENT-SIDE prune (keep_last=%s). R-89 moved offsite "
|
||||
"pruning server-side to ep0 and box tokens are write-only, so every weekly run will "
|
||||
"upload successfully and then FAIL the job on a refused prune (R-191)." % kl.group(1))
|
||||
else:
|
||||
ok("the offsite tier arms no client-side prune (keep_last=0; retention is ep0's prune jobs)")
|
||||
|
||||
print()
|
||||
if fails:
|
||||
print("hostinstall gates: %d FAILURE(S)" % len(fails))
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""repo_gates.py — THE entry point for this repo's gates. Run from the repo root:
|
||||
|
||||
python3 scripts/repo_gates.py # every gate
|
||||
python3 scripts/repo_gates.py --fast # only gates that touch no network and no container
|
||||
# runtime (what .githooks/pre-push runs)
|
||||
|
||||
Gates, in order (all must pass; **non-zero exit on any failure**):
|
||||
|
||||
1. site website HTML: BOM, emoji, nav/footer, analytics, CDN, tokens, cache-busting
|
||||
2. hostinstall felhom-host-install.sh's five drill-swept invariants (+ R-94's absent-version)
|
||||
3. hub-confirm no native confirm()/prompt() in hub templates
|
||||
4. manifest-bearer no bearer-shaped literal anywhere in manifests/
|
||||
5. reuse-refs every path cited by this repo's REUSE.md still resolves
|
||||
|
||||
WHY THIS FILE EXISTS (2026-08-02, closing R-29 leg (a) and half of leg (b)).
|
||||
|
||||
A census of all thirteen gate scripts across the four felhom repos found one clean correlation:
|
||||
**every check a CLAUDE.md tells a person to run was passing, and two of the four nobody is told
|
||||
to run were failing** — one since 14 July. Neither failure was harmful in effect, which was
|
||||
checked line by line; nothing would have said so if they had been. The fix is not more gates, it
|
||||
is one place to run them from. `app-catalog-felhom.eu/scripts/catalog_gates.py` is the canonical
|
||||
shape (R-161) and this copies it deliberately rather than inventing a second one.
|
||||
|
||||
`site_gates.py` is a GATE — eight assertions in one file — and is NOT the model for this file. A
|
||||
runner that invokes separate gates is the shape that survives; copying site_gates would just add
|
||||
a ninth monolith.
|
||||
|
||||
FAIL-CLOSED. A gate script that is missing is a FAILURE, never a skip, and the exact path tried
|
||||
is printed. A runner that quietly drops a gate is the inert-seam failure this project has shipped
|
||||
four times.
|
||||
|
||||
EXIT CODES. Each gate returns 0 clean / 1 convicted / 2 inconclusive. This runner exits non-zero
|
||||
if any gate is non-zero, and reports 2 distinctly as INCONCLUSIVE — an undetermined result is
|
||||
never a pass, but it is not a conviction either, and the operator needs to know which they have.
|
||||
"""
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
SCRIPTS = os.path.join(ROOT, "scripts")
|
||||
|
||||
# (label, absolute script path, args, fast)
|
||||
GATES = [
|
||||
("site", os.path.join(SCRIPTS, "site_gates.py"), [], True),
|
||||
("hostinstall", os.path.join(SCRIPTS, "hostinstall_gates.py"), [], True),
|
||||
("hub-confirm", os.path.join(SCRIPTS, "hub_confirm_gate.py"), [], True),
|
||||
("manifest-bearer", os.path.join(SCRIPTS, "manifest_bearer_gate.py"), [], True),
|
||||
("reuse-refs", os.path.join(SCRIPTS, "reuse_refs_check.py"), [ROOT], True),
|
||||
]
|
||||
|
||||
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
|
||||
|
||||
|
||||
def hooks_armed_note(root):
|
||||
"""Print a WARNING (never a failure) when this clone's pre-push hook is not switched on.
|
||||
|
||||
core.hooksPath is local config and a clone does not carry it, so an unarmed clone is silent
|
||||
by construction — this is the only place it becomes visible.
|
||||
"""
|
||||
try:
|
||||
val = subprocess.check_output(["git", "config", "--get", "core.hooksPath"],
|
||||
cwd=root, stderr=subprocess.DEVNULL).decode().strip()
|
||||
except Exception:
|
||||
val = ""
|
||||
norm = val.replace("\\", "/").rstrip("/")
|
||||
if norm == ".githooks" or norm.endswith("/.githooks"):
|
||||
return
|
||||
print("WARNING: this clone is UNARMED — core.hooksPath is %s, so the pre-push hook will not\n"
|
||||
" run here. Switch it on once with: git config core.hooksPath .githooks"
|
||||
% (("'" + val + "'") if val else "unset"))
|
||||
|
||||
|
||||
def run_gate(label, path, args):
|
||||
if not os.path.exists(path):
|
||||
print("\nFAIL: gate '%s' is MISSING — tried %s" % (label, path))
|
||||
print(" A missing gate is a failure, never a skip (fail-closed).")
|
||||
return 1
|
||||
print("\n" + "=" * 78)
|
||||
print("== gate: %s (%s%s)" % (label, os.path.basename(path),
|
||||
(" " + " ".join(args)) if args else ""))
|
||||
print("=" * 78, flush=True)
|
||||
# stream the gate's own output rather than capturing it — its diagnostics are the point,
|
||||
# and a runner that swallows them makes a conviction unreadable.
|
||||
return subprocess.call([sys.executable, path] + args, cwd=ROOT)
|
||||
|
||||
|
||||
def main(argv):
|
||||
fast = "--fast" in argv
|
||||
unknown = [a for a in argv if a != "--fast"]
|
||||
if unknown:
|
||||
print("unknown argument(s): %s" % " ".join(unknown))
|
||||
print("usage: python3 scripts/repo_gates.py [--fast]")
|
||||
return 2
|
||||
|
||||
selected = [g for g in GATES if g[3] or not fast]
|
||||
skipped = [g[0] for g in GATES if not (g[3] or not fast)]
|
||||
print("repo_gates (felhom.eu) — %d gate(s)%s" % (len(selected), " [--fast]" if fast else ""))
|
||||
if skipped:
|
||||
print(" --fast SKIPPED (deliberate periodic runs, never in a hook): %s" % ", ".join(skipped))
|
||||
hooks_armed_note(ROOT)
|
||||
|
||||
results = [(label, run_gate(label, path, args)) for label, path, args, _f in selected]
|
||||
|
||||
print("\n" + "=" * 78)
|
||||
print("== summary")
|
||||
print("=" * 78)
|
||||
worst = 0
|
||||
for label, rc in results:
|
||||
print(" %-18s %-13s (exit %d)" % (label, VERDICT.get(rc, "ERROR"), rc))
|
||||
if rc != 0:
|
||||
worst = 1 if rc == 1 or worst == 1 else 2
|
||||
if worst == 0:
|
||||
print("\nall felhom.eu gates OK")
|
||||
return 0
|
||||
convicted = [l for l, rc in results if rc == 1]
|
||||
undecided = [l for l, rc in results if rc not in (0, 1)]
|
||||
if convicted:
|
||||
print("\nCONVICTED: %s" % ", ".join(convicted))
|
||||
if undecided:
|
||||
print("UNDETERMINED (never a pass): %s" % ", ".join(undecided))
|
||||
return worst
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
+166
-20
@@ -1,21 +1,142 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""REUSE.md refs check — staleness defense for the per-repo reuse maps.
|
||||
|
||||
Usage: python scripts/reuse_refs_check.py <repo-root> [<repo-root> ...]
|
||||
Usage: python3 scripts/reuse_refs_check.py <repo-root> [<repo-root> ...]
|
||||
|
||||
For each repo root given, parses <root>/REUSE.md, extracts every cited file path
|
||||
(*.go, *.py, *.html, *.css, *.yml, *.yaml, *.sh) and verifies the file exists in
|
||||
the tree. Only slash-containing (repo-relative) tokens are checked — bare filenames
|
||||
are conventions, not citations. Prints offending lines; exits non-zero if any cited
|
||||
path is missing. Symbols are NOT checked here — those are spot-verified by the
|
||||
reviewer at file:line.
|
||||
(*.go, *.py, *.html, *.css, *.yml, *.yaml, *.sh) and verifies the file EXISTS somewhere it
|
||||
could honestly be. Only slash-containing tokens are checked — bare filenames are conventions,
|
||||
not citations. Symbols are NOT checked here; those are spot-verified by the reviewer at
|
||||
file:line. Exits non-zero if any cited path resolves nowhere.
|
||||
|
||||
RESOLUTION ORDER (2026-08-02 — operator ruling; first hit wins, and every non-exact hit is
|
||||
PRINTED so a weakening of the check is visible rather than silent):
|
||||
|
||||
1. exact <root>/<token> exists — no note
|
||||
2. suffix exactly one indexed file under <root> ends with /<token>
|
||||
3. ambiguous more than one does — still OK: the citation is real, the shorthand is
|
||||
imprecise. All matches are printed and marked AMBIGUOUS.
|
||||
4. cross-repo resolved in an immediate SIBLING repo (a sibling dir containing .git), either
|
||||
as-is or with the sibling's own name stripped from the front of the token
|
||||
5. FAIL nowhere — prints the file, line, token, and EVERY resolution attempted
|
||||
|
||||
WHY THIS SHAPE. Before this, the checker demanded repo-relative paths and was RED on all four
|
||||
repos: 13 findings, and a hand audit of all 13 on 2026-08-02 found **zero** genuine drift.
|
||||
Twelve were package shorthand whose file sits one or two directories deeper
|
||||
(`appbackup/userdata.go` → `controller/internal/appbackup/userdata.go`); one,
|
||||
`wgsync/reconciler.go`, is cited by the controller's REUSE.md and lives in the HUB. REUSE.md
|
||||
cites by package shorthand and across repos on purpose — that convention is the useful one, and
|
||||
the tool was what was wrong. Rejected alternatives, recorded so they are not revisited:
|
||||
rewriting all four REUSE.md files to full paths (makes the docs worse to serve the tool), and
|
||||
deleting the checker (REUSE drift across four repos is a live risk).
|
||||
|
||||
THE POSITIVE OBSERVABLE. Every root prints a per-rule tally. "0 failures" alone cannot tell a
|
||||
working checker from a blind one — a run that suddenly resolves everything by SUFFIX is telling
|
||||
you something, and the counts are where you see it. The kill condition is pinned by
|
||||
scripts/test_reuse_refs_check.py: a citation that exists nowhere still FAILS.
|
||||
"""
|
||||
import io, os, re, sys
|
||||
|
||||
# path-looking tokens ending in a checked extension; globs (*) are conventions, not refs
|
||||
PATH_RE = re.compile(r'[A-Za-z0-9_][A-Za-z0-9_./\-]*/[A-Za-z0-9_./\-]*\.(?:go|py|html|css|yml|yaml|sh)\b')
|
||||
|
||||
# An EVIDENCE COPY of a file is not the file — never let an audit or a test-findings tree satisfy
|
||||
# a citation. `.git`/`vendor`/`node_modules` are excluded as noise.
|
||||
EXCLUDE_NAMES = {".git", "node_modules", "vendor", "audits"}
|
||||
EXCLUDE_RELPATHS = {"documentation/tests"}
|
||||
|
||||
fails = 0
|
||||
_index_cache = {}
|
||||
|
||||
|
||||
class RepoIndex(object):
|
||||
"""One walk per repo root, reused across every token and every sibling lookup."""
|
||||
|
||||
def __init__(self, root):
|
||||
self.root = root
|
||||
self.name = os.path.basename(root)
|
||||
self.files = set() # posix-style relpaths
|
||||
self.by_base = {} # basename -> [relpath, ...]
|
||||
for dirpath, dirs, filenames in os.walk(root):
|
||||
rel = os.path.relpath(dirpath, root).replace(os.sep, "/")
|
||||
if rel == ".":
|
||||
rel = ""
|
||||
dirs[:] = [d for d in dirs
|
||||
if d not in EXCLUDE_NAMES
|
||||
and ((rel + "/" + d).lstrip("/") not in EXCLUDE_RELPATHS)]
|
||||
for fn in filenames:
|
||||
p = (rel + "/" + fn).lstrip("/")
|
||||
self.files.add(p)
|
||||
self.by_base.setdefault(fn, []).append(p)
|
||||
|
||||
def exact(self, token):
|
||||
return token in self.files
|
||||
|
||||
def suffix_matches(self, token):
|
||||
base = token.rsplit("/", 1)[-1]
|
||||
return sorted(p for p in self.by_base.get(base, [])
|
||||
if p != token and p.endswith("/" + token))
|
||||
|
||||
|
||||
def index_for(root):
|
||||
root = os.path.abspath(root)
|
||||
if root not in _index_cache:
|
||||
_index_cache[root] = RepoIndex(root)
|
||||
return _index_cache[root]
|
||||
|
||||
|
||||
def find_siblings(root):
|
||||
"""Immediate sibling dirs of <root> that are themselves git working trees. One level only.
|
||||
|
||||
Returns (list_of_paths, error_message_or_None). A sibling repo that is simply absent is NEVER
|
||||
a failure — a clone in isolation must still be able to check itself.
|
||||
"""
|
||||
parent = os.path.dirname(os.path.abspath(root))
|
||||
try:
|
||||
entries = sorted(os.listdir(parent))
|
||||
except OSError as e:
|
||||
return [], "parent %s not readable (%s) — siblings were NOT searched" % (parent, e)
|
||||
sibs = []
|
||||
for e in entries:
|
||||
p = os.path.join(parent, e)
|
||||
if os.path.abspath(p) == os.path.abspath(root):
|
||||
continue
|
||||
if os.path.isdir(p) and os.path.exists(os.path.join(p, ".git")):
|
||||
sibs.append(p)
|
||||
return sibs, None
|
||||
|
||||
|
||||
def resolve(token, idx, siblings):
|
||||
"""(rule, note, tried) — rule is one of exact/suffix/ambiguous/cross-repo/None."""
|
||||
tried = ["repo-relative %s/%s" % (idx.name, token)]
|
||||
if idx.exact(token):
|
||||
return "exact", "", tried
|
||||
|
||||
tried.append("suffix search over %d indexed files in %s" % (len(idx.files), idx.name))
|
||||
m = idx.suffix_matches(token)
|
||||
if len(m) == 1:
|
||||
return "suffix", "resolved by suffix → %s" % m[0], tried
|
||||
if len(m) > 1:
|
||||
return "ambiguous", "AMBIGUOUS — %d matches: %s" % (len(m), ", ".join(m)), tried
|
||||
|
||||
for sib in siblings:
|
||||
sidx = index_for(sib)
|
||||
# a token may carry the sibling's own repo name on the front (app-catalog's REUSE.md
|
||||
# cites `felhom.eu/scripts/site_gates.py` that way) — try both forms.
|
||||
cands = [token]
|
||||
if token.startswith(sidx.name + "/"):
|
||||
cands.append(token[len(sidx.name) + 1:])
|
||||
for cand in cands:
|
||||
tried.append("sibling %s: %s" % (sidx.name, cand))
|
||||
if sidx.exact(cand):
|
||||
return "cross-repo", "cross-repo → %s/%s" % (sidx.name, cand), tried
|
||||
sm = sidx.suffix_matches(cand)
|
||||
if len(sm) == 1:
|
||||
return "cross-repo", "cross-repo (suffix) → %s/%s" % (sidx.name, sm[0]), tried
|
||||
if len(sm) > 1:
|
||||
return "cross-repo", "cross-repo AMBIGUOUS in %s — %d matches: %s" % (
|
||||
sidx.name, len(sm), ", ".join(sm)), tried
|
||||
return None, "", tried
|
||||
|
||||
|
||||
def check_repo(root):
|
||||
@@ -27,7 +148,14 @@ def check_repo(root):
|
||||
print("FAIL [%s]: no REUSE.md at %s" % (name, reuse))
|
||||
fails += 1
|
||||
return
|
||||
seen, missing = set(), 0
|
||||
idx = index_for(root)
|
||||
siblings, sib_err = find_siblings(root)
|
||||
if sib_err:
|
||||
# say so and continue — do NOT silently pretend siblings were searched
|
||||
print("NOTE [%s]: %s" % (name, sib_err))
|
||||
|
||||
seen = set()
|
||||
tally = {"exact": 0, "suffix": 0, "ambiguous": 0, "cross-repo": 0, "failed": 0}
|
||||
with io.open(reuse, encoding="utf-8") as f:
|
||||
for lineno, line in enumerate(f, 1):
|
||||
for m in PATH_RE.finditer(line):
|
||||
@@ -37,18 +165,36 @@ def check_repo(root):
|
||||
if p in seen:
|
||||
continue
|
||||
seen.add(p)
|
||||
if not os.path.isfile(os.path.join(root, p)):
|
||||
print("FAIL [%s] line %d: cited path missing: %s" % (name, lineno, p))
|
||||
missing += 1
|
||||
if missing:
|
||||
fails += missing
|
||||
else:
|
||||
print("OK [%s]: %d cited paths, all exist" % (name, len(seen)))
|
||||
rule, note, tried = resolve(p, idx, siblings)
|
||||
if rule is None:
|
||||
tally["failed"] += 1
|
||||
print("FAIL [%s] line %d: cited path resolves NOWHERE: %s" % (name, lineno, p))
|
||||
for t in tried:
|
||||
print(" tried: %s" % t)
|
||||
if not siblings and not sib_err:
|
||||
print(" tried: no sibling git repos found beside %s" % name)
|
||||
else:
|
||||
tally[rule] += 1
|
||||
if note:
|
||||
print("note [%s] line %d: %s (%s)" % (name, lineno, p, note))
|
||||
|
||||
print("%s [%s]: %d cited paths — exact %d, suffix %d, ambiguous %d, cross-repo %d, FAILED %d "
|
||||
"(siblings searched: %s)" % (
|
||||
"FAIL" if tally["failed"] else "OK ", name, len(seen),
|
||||
tally["exact"], tally["suffix"], tally["ambiguous"], tally["cross-repo"],
|
||||
tally["failed"],
|
||||
", ".join(os.path.basename(s) for s in siblings) or "none"))
|
||||
fails += tally["failed"]
|
||||
|
||||
|
||||
if len(sys.argv) < 2:
|
||||
print(__doc__)
|
||||
sys.exit(2)
|
||||
for r in sys.argv[1:]:
|
||||
check_repo(r)
|
||||
sys.exit(1 if fails else 0)
|
||||
def main(argv):
|
||||
if len(argv) < 1:
|
||||
print(__doc__)
|
||||
return 2
|
||||
for r in argv:
|
||||
check_repo(r)
|
||||
return 1 if fails else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Seam test for scripts/repo_gates.py.
|
||||
|
||||
Run: python3 scripts/test_repo_gates.py
|
||||
|
||||
WHY THIS EXISTS. An entry point is a seam by definition: a runner that LISTS a gate but never
|
||||
executes it is inert and fully green, and this project has shipped an inert seam four times. So
|
||||
the assertion is on each member gate's OWN distinctive stdout — never on the runner's summary
|
||||
line, which the runner can print without ever calling anything — plus the exit code, which is a
|
||||
runner's actual effect.
|
||||
|
||||
Red-proofed 2026-08-02: replacing run_gate's body with `return 0` (the inert runner) turns
|
||||
test_every_member_gate_actually_ran red while the summary still prints "all felhom.eu gates OK".
|
||||
"""
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
ENTRY = os.path.join(ROOT, "scripts", "repo_gates.py")
|
||||
|
||||
# (label, a substring only THAT gate can print)
|
||||
FINGERPRINTS = [
|
||||
("site", "site gates OK"),
|
||||
("hostinstall", "hostinstall gates: ALL PASS"),
|
||||
("hub-confirm", "hub confirm gate"),
|
||||
("manifest-bearer", "manifest bearer gate"),
|
||||
("reuse-refs", "cited paths — exact"),
|
||||
]
|
||||
|
||||
|
||||
class RepoGatesTest(unittest.TestCase):
|
||||
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
p = subprocess.run([sys.executable, ENTRY, "--fast"], cwd=ROOT,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
||||
cls.rc = p.returncode
|
||||
cls.out = p.stdout.decode("utf-8", "replace")
|
||||
|
||||
def test_exit_code_is_zero(self):
|
||||
self.assertEqual(self.rc, 0, self.out)
|
||||
|
||||
def test_every_member_gate_actually_ran(self):
|
||||
for label, fingerprint in FINGERPRINTS:
|
||||
self.assertIn(fingerprint, self.out,
|
||||
"gate %r is listed but its own output never appeared — an inert runner "
|
||||
"prints the summary without calling anything:\n%s" % (label, self.out))
|
||||
|
||||
def test_unknown_argument_is_rejected(self):
|
||||
p = subprocess.run([sys.executable, ENTRY, "--nope"], cwd=ROOT,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
||||
self.assertEqual(p.returncode, 2, p.stdout.decode("utf-8", "replace"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -0,0 +1,191 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Fixture tests for scripts/reuse_refs_check.py — one per row of its resolution table, plus the
|
||||
kill condition.
|
||||
|
||||
Run: python3 scripts/test_reuse_refs_check.py
|
||||
|
||||
THE ONE THAT MATTERS is test_absent_path_fails (Scenario E). The 2026-08-02 change taught the
|
||||
checker to resolve suffixes and sibling repos, which turned 13 findings green in one step. A
|
||||
checker made green by being made BLIND is a failure this project has shipped before, so the
|
||||
ability to still fail is pinned here, and the assertion is on the EXIT CODE — the effect — not on
|
||||
the summary text, which the checker can print without having decided anything.
|
||||
"""
|
||||
import importlib.util
|
||||
import io
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
SCRIPT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "reuse_refs_check.py")
|
||||
|
||||
|
||||
def load_checker():
|
||||
"""Fresh module per test — the checker keeps a global failure count and an index cache."""
|
||||
spec = importlib.util.spec_from_file_location("reuse_refs_check_under_test", SCRIPT)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
def write(path, text=""):
|
||||
d = os.path.dirname(path)
|
||||
if d and not os.path.isdir(d):
|
||||
os.makedirs(d)
|
||||
with io.open(path, "w", encoding="utf-8") as f:
|
||||
f.write(text)
|
||||
|
||||
|
||||
class ReuseRefsCheckTest(unittest.TestCase):
|
||||
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.mkdtemp(prefix="reuse-refs-")
|
||||
self.root = os.path.join(self.tmp, "myrepo")
|
||||
write(os.path.join(self.root, ".git"), "gitdir: elsewhere\n")
|
||||
|
||||
def tearDown(self):
|
||||
shutil.rmtree(self.tmp, ignore_errors=True)
|
||||
|
||||
# ── harness ──────────────────────────────────────────────────────────────────
|
||||
def run_check(self, *roots):
|
||||
"""Returns (exit_code, stdout). Exit code is the assertion that counts."""
|
||||
mod = load_checker()
|
||||
buf = io.StringIO()
|
||||
real = sys.stdout
|
||||
sys.stdout = buf
|
||||
try:
|
||||
rc = mod.main(list(roots) or [self.root])
|
||||
finally:
|
||||
sys.stdout = real
|
||||
return rc, buf.getvalue()
|
||||
|
||||
def reuse(self, body):
|
||||
write(os.path.join(self.root, "REUSE.md"), body)
|
||||
|
||||
def sibling(self, name):
|
||||
p = os.path.join(self.tmp, name)
|
||||
write(os.path.join(p, ".git"), "gitdir: elsewhere\n")
|
||||
return p
|
||||
|
||||
# ── row 1: exact ─────────────────────────────────────────────────────────────
|
||||
def test_exact_match_is_silent_and_passes(self):
|
||||
write(os.path.join(self.root, "a", "b.go"), "package a\n")
|
||||
self.reuse("see `a/b.go` for the thing\n")
|
||||
rc, out = self.run_check()
|
||||
self.assertEqual(rc, 0, out)
|
||||
self.assertIn("exact 1", out)
|
||||
self.assertNotIn("note [", out) # an exact hit prints no note
|
||||
|
||||
# ── row 2: suffix ────────────────────────────────────────────────────────────
|
||||
def test_package_shorthand_resolves_by_suffix(self):
|
||||
write(os.path.join(self.root, "controller", "internal", "pkg", "x.go"), "package pkg\n")
|
||||
self.reuse("see `pkg/x.go`\n")
|
||||
rc, out = self.run_check()
|
||||
self.assertEqual(rc, 0, out)
|
||||
self.assertIn("resolved by suffix", out)
|
||||
self.assertIn("controller/internal/pkg/x.go", out)
|
||||
self.assertIn("suffix 1", out)
|
||||
|
||||
# ── row 3: ambiguous — real citation, imprecise shorthand; NOT a failure ─────
|
||||
def test_two_suffix_matches_are_ambiguous_not_fatal(self):
|
||||
write(os.path.join(self.root, "one", "pkg", "x.go"), "package pkg\n")
|
||||
write(os.path.join(self.root, "two", "pkg", "x.go"), "package pkg\n")
|
||||
self.reuse("see `pkg/x.go`\n")
|
||||
rc, out = self.run_check()
|
||||
self.assertEqual(rc, 0, out)
|
||||
self.assertIn("AMBIGUOUS", out)
|
||||
self.assertIn("one/pkg/x.go", out)
|
||||
self.assertIn("two/pkg/x.go", out)
|
||||
self.assertIn("ambiguous 1", out)
|
||||
|
||||
# ── row 4: cross-repo, by suffix in a sibling ────────────────────────────────
|
||||
def test_sibling_repo_resolution(self):
|
||||
sib = self.sibling("otherrepo")
|
||||
write(os.path.join(sib, "hub", "internal", "wgsync", "reconciler.go"), "package wgsync\n")
|
||||
self.reuse("see `wgsync/reconciler.go`\n")
|
||||
rc, out = self.run_check()
|
||||
self.assertEqual(rc, 0, out)
|
||||
self.assertIn("cross-repo", out)
|
||||
self.assertIn("otherrepo", out)
|
||||
self.assertIn("cross-repo 1", out)
|
||||
|
||||
# ── row 4b: cross-repo where the token CARRIES the sibling's repo name ───────
|
||||
def test_sibling_repo_name_prefixed_token(self):
|
||||
sib = self.sibling("felhom.eu")
|
||||
write(os.path.join(sib, "scripts", "site_gates.py"), "# gate\n")
|
||||
self.reuse("run `felhom.eu/scripts/site_gates.py`\n")
|
||||
rc, out = self.run_check()
|
||||
self.assertEqual(rc, 0, out)
|
||||
self.assertIn("cross-repo", out)
|
||||
|
||||
def test_non_git_sibling_is_not_searched(self):
|
||||
plain = os.path.join(self.tmp, "notarepo") # no .git — not a repo, must not resolve
|
||||
write(os.path.join(plain, "pkg", "x.go"), "package pkg\n")
|
||||
self.reuse("see `pkg/x.go`\n")
|
||||
rc, out = self.run_check()
|
||||
self.assertNotEqual(rc, 0, out)
|
||||
|
||||
# ── row 5: THE KILL CONDITION (Scenario E) ──────────────────────────────────
|
||||
def test_absent_path_fails(self):
|
||||
write(os.path.join(self.root, "internal", "present.go"), "package internal\n")
|
||||
self.reuse("line one\nsee `internal/definitely_absent_xyz.go`\n")
|
||||
rc, out = self.run_check()
|
||||
self.assertNotEqual(rc, 0, "a citation that exists NOWHERE must fail:\n" + out)
|
||||
self.assertIn("definitely_absent_xyz.go", out)
|
||||
self.assertIn("line 2", out) # names the line
|
||||
self.assertIn("FAILED 1", out)
|
||||
|
||||
def test_failure_lists_every_resolution_attempted(self):
|
||||
"""CLAUDE.md standing rule: a 'not found' claim must name what was tried."""
|
||||
self.sibling("otherrepo")
|
||||
self.reuse("see `internal/definitely_absent_xyz.go`\n")
|
||||
rc, out = self.run_check()
|
||||
self.assertNotEqual(rc, 0, out)
|
||||
self.assertIn("tried: repo-relative myrepo/internal/definitely_absent_xyz.go", out)
|
||||
self.assertIn("tried: suffix search over", out)
|
||||
self.assertIn("tried: sibling otherrepo", out)
|
||||
|
||||
# ── evidence trees are not the file ─────────────────────────────────────────
|
||||
def test_evidence_copy_does_not_satisfy_a_citation(self):
|
||||
write(os.path.join(self.root, "documentation", "audits", "pkg", "x.go"), "package pkg\n")
|
||||
write(os.path.join(self.root, "documentation", "tests", "pkg", "y.go"), "package pkg\n")
|
||||
self.reuse("see `pkg/x.go` and `pkg/y.go`\n")
|
||||
rc, out = self.run_check()
|
||||
self.assertNotEqual(rc, 0, "an audits/ or documentation/tests/ copy must NOT resolve:\n" + out)
|
||||
self.assertIn("FAILED 2", out)
|
||||
|
||||
# ── a clone in isolation must still check itself ────────────────────────────
|
||||
def test_no_siblings_is_not_a_failure(self):
|
||||
write(os.path.join(self.root, "a", "b.go"), "package a\n")
|
||||
self.reuse("see `a/b.go`\n")
|
||||
rc, out = self.run_check()
|
||||
self.assertEqual(rc, 0, out)
|
||||
self.assertIn("siblings searched: none", out)
|
||||
|
||||
def test_missing_reuse_md_fails(self):
|
||||
rc, out = self.run_check()
|
||||
self.assertNotEqual(rc, 0, out)
|
||||
self.assertIn("no REUSE.md", out)
|
||||
|
||||
# ── globs stay conventions, not refs ────────────────────────────────────────
|
||||
def test_glob_is_not_a_citation(self):
|
||||
self.reuse("the gates are `scripts/*.py`\n")
|
||||
rc, out = self.run_check()
|
||||
self.assertEqual(rc, 0, out)
|
||||
self.assertIn("0 cited paths", out)
|
||||
|
||||
# ── no args → usage, exit 2 ─────────────────────────────────────────────────
|
||||
def test_no_args_is_usage_exit_2(self):
|
||||
mod = load_checker()
|
||||
buf, real = io.StringIO(), sys.stdout
|
||||
sys.stdout = buf
|
||||
try:
|
||||
rc = mod.main([])
|
||||
finally:
|
||||
sys.stdout = real
|
||||
self.assertEqual(rc, 2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: felhom-build-deploy
|
||||
description: Build, deploy, publish, or verify ANY Felhom artifact — felhom-controller image (guest 9201 bootstrap deploy), felhom-agent binary (felhom-pve), felhom-hub (GitOps/ArgoCD), the felhom.eu website (git-sync), or the app catalog. Use whenever the task says build, deploy, ship, release, publish, bump version, restart the controller/agent/hub, or verify what version is live. Contains the exact verified commands and the gotchas that silently break deploys.
|
||||
description: Build, deploy, publish, or verify ANY Felhom artifact — felhom-controller image (guest 9201 bootstrap deploy), felhom-agent binary (felhom-pve), felhom-hub (GitOps/ArgoCD), the felhom.eu website (git-sync), the app catalog, or the PUBLIC installer ISO (iso.felhom.eu). Use whenever the task says build, deploy, ship, release, publish, bump version, restart the controller/agent/hub, or verify what version is live. Contains the exact verified commands and the gotchas that silently break deploys.
|
||||
---
|
||||
|
||||
# Felhom build & deploy runbooks
|
||||
@@ -75,6 +75,72 @@ Publish to Gitea (so Day-0 self-install can fetch it): `scripts/publish-agent.sh
|
||||
`REGISTRY_*` creds. The hub's Day-0 artifact manifest must then vouch the new version — that UI is
|
||||
operator-password-gated (CC cannot); flag it as an operator follow-up.
|
||||
|
||||
## Installer ISO (felhom.eu/scripts/iso → iso.felhom.eu) — PUBLIC, irreversible
|
||||
|
||||
**Two modes, and picking the wrong one ships the wrong product.**
|
||||
|
||||
| Mode | What it is | Menu |
|
||||
|---|---|---|
|
||||
| `--release` | the **public** image. No `answer.toml`, no root password, no SSH key, no disk profile. Day-0 rides a `.deb`. | TWO interactive entries, graphical default, timeout 15s |
|
||||
| `--pairing` / `--bootstrap-env` | operator-built **appliance** image: baked answer file, baked root hash, a disk profile pinned to one machine | ONE automated entry |
|
||||
|
||||
```bash
|
||||
# build the public image (clean-tree gate first — an unpushed change does not exist)
|
||||
export FELHOM_ISO_OUT=$FELHOM_ROOT/felhom-iso/out
|
||||
bash scripts/iso/build-felhom-iso.sh \
|
||||
--pve-iso $FELHOM_ROOT/drill/proxmox-ve_9.2-1.iso \
|
||||
--iso-sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c --release
|
||||
# -> $FELHOM_ISO_OUT/felhom-installer-<VER>-pve<PVE>.iso + .sha256 + .manifest.txt (NO .rootpw.txt)
|
||||
```
|
||||
|
||||
**Before publishing, two hard gates — neither is optional and neither is a script yet:**
|
||||
|
||||
1. **`documentation/runbooks/iso-release-gate.md`** — 13 criteria, run against **the exact file you
|
||||
will upload**, not the build inputs. It is a manual checklist; `repo_gates.py` does **not** cover
|
||||
it, so nothing will remind you (R-29's shape — say so if you skip it).
|
||||
2. **A proof install from the built image on BOTH menu entries** (graphical *and* Terminal UI), each
|
||||
showing: package installed, unit `enabled`, unit fired on first boot, and a pairing code in
|
||||
`/etc/felhom/appliance-pairing-code`. Spike 4 *reasoned* the graphical path follows from shared
|
||||
`Install.pm`; the 1.26.0 run proved that reasoning insufficient in a different place — do both.
|
||||
|
||||
```bash
|
||||
# publish — rclone in a container, env-only config, so NO credential file is ever written
|
||||
source ~/.config/credentials # ISO_S3_CLIENT_AK / _SK / ISO_S3_URL — never echo, never log
|
||||
docker run --rm -v $FELHOM_ISO_OUT:/data:ro \
|
||||
-e RCLONE_CONFIG_R2_TYPE=s3 -e RCLONE_CONFIG_R2_PROVIDER=Cloudflare \
|
||||
-e RCLONE_CONFIG_R2_ACCESS_KEY_ID="$ISO_S3_CLIENT_AK" \
|
||||
-e RCLONE_CONFIG_R2_SECRET_ACCESS_KEY="$ISO_S3_CLIENT_SK" \
|
||||
-e RCLONE_CONFIG_R2_ENDPOINT="$ISO_S3_URL" \
|
||||
-e RCLONE_CONFIG_R2_REGION=auto -e RCLONE_CONFIG_R2_NO_CHECK_BUCKET=true \
|
||||
rclone/rclone:latest copy /data R2:felhom-iso --include "felhom-installer-<VER>*" --s3-chunk-size 64M
|
||||
# verify by ROUND TRIP — the downloaded bytes, not the local file
|
||||
curl -fsSL -o /tmp/rt.iso https://iso.felhom.eu/felhom-installer-<VER>-pve<PVE>.iso && sha256sum /tmp/rt.iso
|
||||
```
|
||||
|
||||
`ListBuckets` 403s — the token is object-scoped; list with `lsf R2:felhom-iso`, not `lsd R2:`.
|
||||
|
||||
### Proof-VM traps — every one of these cost a wrong diagnosis
|
||||
|
||||
- **Set `--boot` in a SEPARATE `qm set`, after the disk exists.** `qm set <id> --scsi0 … --boot order="scsi0;ide2"`
|
||||
in one call silently yields `boot: order=net0;ide2`; the VM netboots, fails, falls through to the CD.
|
||||
- **After the install, detach the CD** (`qm set <id> --delete ide2; qm set <id> --boot order="scsi0"`)
|
||||
or the machine re-enters the installer on reboot — **a completed install looks exactly like a stuck one.**
|
||||
Judge completion from `qm config` + disk usage, never from the screen.
|
||||
- **Verify focus by screendump before every `Enter`.** TUI: red-highlighted button, tab order. GTK:
|
||||
dashed focus ring, and `Enter` lands in text *fields*, not `Next`. Not checking once aborted an install.
|
||||
- **Proof installs register unclaimed appliances at the hub — discard them** or R-131 grows:
|
||||
`curl -u ":$HUB_PW" -X POST http://<hub-clusterIP>:8080/appliances/<id>/discard` → 303.
|
||||
The verb is **`/discard`**, POST only (`hub/internal/web/server.go:345`); `/delete` 404s.
|
||||
- Venue: `demo-hp`, scratch `dir` storage at **`/mnt/nvme-1tb` root** (a subdirectory reads
|
||||
`disconnected` forever — the agent's `exactMount` check). Never `local-lvm`. Remove the storage at teardown.
|
||||
|
||||
**Why the shape is what it is** (do not re-derive; four spikes measured it):
|
||||
`documentation/audits/SPIKE-universal-iso-{1,2,3,4}-2026-07-31.md`. In short — no udev property
|
||||
distinguishes an internal disk from a customer's backup drive and a two-disk filter match silently
|
||||
wipes one, so **there is no safe automated disk selection for unseen hardware**; and `[first-boot]`
|
||||
is never placed on the system by an interactive install, so day-0 rides a `.deb` in
|
||||
`/proxmox/packages/` instead (`Install.pm:1343-1372`).
|
||||
|
||||
## Hub (felhom.eu/hub → k3s, GitOps via ArgoCD app `felhom`)
|
||||
|
||||
**The manifest is the truth.** A code push + image build deploys NOTHING until `manifests/hub.yaml`'s
|
||||
|
||||
Reference in New Issue
Block a user