Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 admin
|
7dc1744eec | ||
|
admin
|
a5cd480280 | ||
|
admin
|
b0b269b28d |
@@ -213,6 +213,22 @@ local and skippable, and only CI is neither.
|
||||
- **Website** auto-deploys via git-sync; just push to `main` (live in 1–2 min). Website changes go
|
||||
through `repo_gates.py` above (it runs `site_gates.py`); new pages go into that gate's `PAGES`
|
||||
list. Emergency edits: https://files.felhom.eu. All `website/` HTML is **UTF-8 with BOM** — preserve it.
|
||||
- **THE INSTALLER DOES NOT (R-110, 2026-08-03).** `manifests/webpage.yaml` runs **two** git-syncs:
|
||||
the website from `main` as above, and `/scripts/` from the tag **`installer-v<SCRIPT_VERSION>`**.
|
||||
Pushing `scripts/felhom-host-install.sh` therefore changes nothing that any machine downloads —
|
||||
which it used to, within thirty seconds, for the one artifact that runs as **root on a virgin box**.
|
||||
- **To publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml`
|
||||
(both the sidecar and the init container), commit, and sync. `hostinstall_gates.py` gate 6
|
||||
fails if the manifest stops naming an `installer-v…` tag or if the website stops tracking `main`.
|
||||
- **To roll back:** move the tag back to the previous commit and wait ~30 s. **No ArgoCD sync and
|
||||
no deploy** — git-sync picks up a moved tag on its next period, measured live on 2026-08-03 in
|
||||
both directions. That is the emergency lever; fix forward with a new version afterwards.
|
||||
- **Do NOT pin the website to the tag.** The sparse-checkout used to cover `/website/` and
|
||||
`/scripts/` in one sync, and pinning that would turn every copy edit into a release.
|
||||
- The **URL never carries a ref** (`https://felhom.eu/scripts/felhom-host-install.sh`), so
|
||||
`felhom-bootstrap.sh` and the hub's day-0 command follow the tag with no edit — do not add one.
|
||||
- The installer's own sixteen run-time fetches are pinned separately, to `raw/tag/v$ART_AGENT_VER`
|
||||
in the **agent** repo (R-183) — they are the agent's configs, not this repo's.
|
||||
- **Manifests** are GitOps via the `felhom` app — commit to `main`, then deliberate sync.
|
||||
|
||||
## Key patterns
|
||||
|
||||
+26
@@ -97,6 +97,32 @@ not provide, and the fourth of those found on live hardware rather than by revie
|
||||
no run scope, so a refused app re-alerts on every status refresh (measured: a second identical alert
|
||||
pair 13 s after the run's). Pre-existing in v0.192.0; R-181 changed neither caller.
|
||||
|
||||
**S-15 — publishing is an act, not a side-effect of pushing (2026-08-03, R-110 + R-115 + R-183).**
|
||||
Two rulings, one shape: something became live because someone pushed, not because anyone decided.
|
||||
|
||||
- **The installer.** `/scripts/` now git-syncs the tag `installer-v<SCRIPT_VERSION>`; the **website
|
||||
keeps tracking `main`** in a second sync, because pinning both would make every copy edit a
|
||||
release. Publish = cut the next tag + bump the manifest `--ref` + sync. **Roll back = move the tag
|
||||
back**, which takes ~30 s and needs no ArgoCD sync at all — git-sync v4.4.0 follows a moved tag,
|
||||
and that half was measured before the manifest was touched because the whole model rests on it.
|
||||
- **The sixteen run-time fetches were NOT what the spec described** — sixteen, not nine, and from
|
||||
`felhom-agent`, not this repo — so no tag here could cover them. They are pinned to
|
||||
`raw/tag/v$ART_AGENT_VER` instead, which is strictly better: the agent's configs now come from the
|
||||
same ref as the agent binary being installed. That closed a real skew (**R-183**), not just a
|
||||
channel.
|
||||
- **The URL needed no change**, and that is worth knowing rather than re-deriving: it never carried
|
||||
a ref, so both producers follow the tag automatically — and no hub change means no hub bump.
|
||||
- **The agent.** `scripts/release-agent.sh` is THE release path: build → tag → publish → **verify by
|
||||
an independent download**. It does not vouch. `check-published-versions.py` refuses a `v<semver>`
|
||||
tag with no downloadable package, and **CI now runs the full gate set** rather than `--fast`,
|
||||
without which that gate would have been registered and never run.
|
||||
- **The gate's invariant is not the one specified, and P-C is why:** the hub manifest and Gitea's
|
||||
package listing are both **401** anonymously; the package download and the tags api are not. So CI
|
||||
can ask *is this installable* but not *what is vouched*. The residue is **R-184**.
|
||||
- **Neither gate asserts "the newest version is published."** That would go red on the very push
|
||||
that bumps a version, before publishing — and a gate that fails on the normal path is one people
|
||||
learn to ignore.
|
||||
|
||||
**S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).**
|
||||
Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the
|
||||
CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure
|
||||
|
||||
@@ -1,113 +0,0 @@
|
||||
# REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)
|
||||
|
||||
Follows `REPORT-campaign7/8/9.md`. Root `REPORT.md` is another session's (hub v0.85.0) and was not
|
||||
clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
|
||||
|
||||
**Full audit + evidence:** `documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`,
|
||||
`documentation/tests/campaign10-evidence-2026-07-31/`.
|
||||
|
||||
## The sentence that matters
|
||||
|
||||
**Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the
|
||||
"drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No
|
||||
resource leak over 13.5 h. Two findings: R-156 and R-157.**
|
||||
|
||||
## What was established
|
||||
|
||||
- **Venue** — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the
|
||||
`/mnt/nvme-1tb` **mount root** (a subdirectory would have emitted `storage_disconnected` for demo-hp
|
||||
all night — the exact signal I1/I2 discriminate).
|
||||
- **Baselines, all read fresh.** controller `main` 0.188.0, **golden 0.188.0 (not behind)**, agent
|
||||
0.119.0 published+vouched, hub 0.86.0, ISO **1.26.1** (`f3cc86d5…`, round-trip verified live).
|
||||
The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
|
||||
- **Isolation gate — both denials captured, each with a positive control.** The PBS control **failed
|
||||
first**: four clean-looking 403s were worthless because the token was denied on its own datastore
|
||||
too (PBS token privilege separation). Fixed, re-run, denials stand.
|
||||
- **A1** fresh install from the **published** ISO. 1.26.1 is a public release image — verified against
|
||||
its bytes that it has no auto-install path — so it was driven blind via screendump + `sendkey`
|
||||
through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which
|
||||
would otherwise have been mangled and locked the box out.
|
||||
- **A2** claimed for real; discriminator flipped `dashboard not yet claimed` → `authentication required`.
|
||||
- **A3** both drives enrolled through the **real** endpoint; `mentes` accepted as backup target via
|
||||
the offer flow, ending `degraded:false / target:felhom-backup` — the I5/I6 healthy baseline.
|
||||
Four apps healthy spanning both sides of D5's split (4 × `type: secret`, 1 × `type: password`).
|
||||
- **A4** discriminators seed and read back across all four apps; rallly's over the path
|
||||
`DATABASE_URL` actually names, not the trusted socket that produced D5's false pass.
|
||||
|
||||
## Phase B — the soak
|
||||
|
||||
Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped
|
||||
deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**).
|
||||
1 461 invariant checks in total.
|
||||
|
||||
- **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale,
|
||||
never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as
|
||||
skipped rather than silently green — it fired once.
|
||||
- **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place
|
||||
variants and 330 secret-class assertions covering both sides of D5's split.
|
||||
- **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced
|
||||
`backup_target_absent` and a clean recovery.
|
||||
- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via
|
||||
`bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names
|
||||
it. That is the case R-117's spike called "the worse half".
|
||||
- **RTO, both bands measured.** S: 66 MB → **42.0 s** / **41.4 s** across two passes (66 restores).
|
||||
M: 21.1 GB → **608 s** mean over 2 reps, both returning the correct discriminator. 327× the data
|
||||
cost 14.5× the time, giving **RTO ≈ 40 s + 26.9 s/GB** (backup ≈ 29 s + 17.4 s/GB). The fixed ~40 s
|
||||
dominates below ~1.5 GB — that is the S band, and why its numbers clustered so tightly.
|
||||
- **Capacity ceiling, and the more consequential result:** a DB-backed app's recovery unit is **1.90×**
|
||||
its data (volume tar + SQL dump). The default `/mnt/sys_drive` is **20 GB**, so on a default box the
|
||||
largest locally-backupable app is **≈ 10 GB** — **the M band does not fit at all** without a
|
||||
per-customer `SysDataGrowGB`. **RPO still not measured.**
|
||||
- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds
|
||||
flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots
|
||||
and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** —
|
||||
bounded by journald, but a lot of logging.
|
||||
- Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that
|
||||
was not first observed live.
|
||||
- **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system
|
||||
disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the
|
||||
soak would have produced green cycles that tested nothing cross-drive.
|
||||
- **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that
|
||||
never landed became a fake "stale restore", and a real one would have looked identical. Fixed and
|
||||
red-proofed before restarting.
|
||||
|
||||
## Findings
|
||||
|
||||
- **R-156 (new)** — **papra's data is neither persisted nor backed up, and it reports healthy.** The
|
||||
template mounts `papra_data:/app/data`; the app writes `/app/app-data/db/db.sqlite`. The volume is
|
||||
empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
|
||||
sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
|
||||
Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
|
||||
- **R-157 (new)** — **bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two
|
||||
mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while
|
||||
docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences,
|
||||
intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is
|
||||
exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp
|
||||
check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on
|
||||
every channel. A settle-condition fix closes A and leaves B open. Not fixed.
|
||||
- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
|
||||
(`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
|
||||
`hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
|
||||
therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by
|
||||
not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.
|
||||
|
||||
## What did not run
|
||||
|
||||
**12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place,
|
||||
kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive).
|
||||
Previously reported as 6 of 12 — that was run 1 only.
|
||||
|
||||
**Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not
|
||||
automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed.
|
||||
`kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom
|
||||
covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed.
|
||||
|
||||
**Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34–c39 clean on
|
||||
every invariant. Beyond 39 is untested, not proven clean.
|
||||
|
||||
## Teardown — OWED, nothing removed
|
||||
|
||||
Still intact: the rig is reusable for the atoms that did not run. VM 311, `c10-scratch`, PBS datastore
|
||||
`felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition:
|
||||
DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four
|
||||
orphaned scratch customers left by exactly this omission.
|
||||
@@ -1,43 +0,0 @@
|
||||
# REPORT — CAMPAIGN 7 (felhom.eu side: docs only)
|
||||
|
||||
> Written as `REPORT-campaign7.md`, **not** the shared `REPORT.md`, per the convention this run
|
||||
> added to `CLAUDE.md`: `REPORT.md` is overwritten, so a second concurrent session in this repo
|
||||
> would clobber it. This session's implementation work was in `app-catalog-felhom.eu`; here it only
|
||||
> touched documentation.
|
||||
|
||||
**Run:** 2026-07-18 evening → 2026-07-19 morning. **Class:** campaign (record-and-rank + a defined
|
||||
allowed-fix set). **Implementation repo:** `app-catalog-felhom.eu` (see its `REPORT.md`).
|
||||
|
||||
## What changed in this repo
|
||||
|
||||
| file | change |
|
||||
|---|---|
|
||||
| `documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` | **new** — method, uninstall-semantics map, trio detail, full 53-app matrix, ranked findings, coverage |
|
||||
| `documentation/backlog/ROADMAP.md` | **+3 items** — R-40 (multi-hop major upgrade path), R-41 (no standing catalog deployability check), R-42 (sidecar-major ruling) |
|
||||
| `CLAUDE.md` | REPORT.md parallel-session rule: the second session writes `REPORT-<topic>.md` |
|
||||
|
||||
No hub/agent/scripts/website code was touched (campaign scope: catalog + docs).
|
||||
|
||||
## Headline for this repo's readers
|
||||
|
||||
1. **Uninstall semantics map row PARTIAL → PROVEN** (campaign doc §2), with live evidence from all
|
||||
three trio apps: remove requires stop first; named docker volumes are **always destroyed**
|
||||
(including the app's database); HDD bind-mount data and `backups/primary/<app>` survive unless
|
||||
explicitly ticked; images are kept; `app.yaml` goes, the template stays; the per-app **offsite
|
||||
toggle survives** the uninstall while tier-2 config is cleared. The confirmation modal does warn
|
||||
about the volumes, so there is **no consent gap**.
|
||||
2. **A lying healthcheck takes an app OFF-LINE, it does not merely mislead.** Traefik will not route
|
||||
to an `unhealthy` container, so a probe that cannot execute → permanent unhealthy → **404 to the
|
||||
customer while the app serves 200 on its own port**. 7 of 53 apps were in that state.
|
||||
3. **The pre-flight gate's own signal is missing:** the 0.145.0 → 0.146.0 floor-lift emitted no
|
||||
`controller_updated` event, though the identical bootstrap path emitted one for 0.143.0 → 0.145.0
|
||||
two hours earlier (§0, finding F1). The box did converge — golden, floor and runtime all agreed —
|
||||
but the event trail under-reports version transitions.
|
||||
|
||||
## Open items owned outside this repo
|
||||
|
||||
- **plant-it / wanderer** — images do not resolve at all (neither the new tag nor the one the
|
||||
catalog already ships). Upstream research needed; recorded as findings, not deletions.
|
||||
- **gokapi** — pinned back to v1.9.6; v2 needs the seeded `config.json` regenerated. Security-
|
||||
relevant, should not sit on a superseded line indefinitely.
|
||||
- **glance** — never had a seeded `glance.yml`; proven pre-existing.
|
||||
@@ -1,62 +0,0 @@
|
||||
# REPORT — CAMPAIGN 8: the backup & restore subsystem (2026-07-27/28)
|
||||
|
||||
Adversarial, destructive, unattended run against `demo-felhom`, `demo-hp` and `ep0`.
|
||||
Full report: `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
Evidence: `DooPlex:~/campaign8/evidence/` (103 files, 35 MB, written continuously by 11 collectors).
|
||||
|
||||
**No production code was changed.** Findings are recorded and ranked, never fixed inline, per the
|
||||
campaign's own rules.
|
||||
|
||||
## Scope safety
|
||||
`peti-felhom`, its namespace and `u629488-sub2` were never touched. Phase 0 established with five
|
||||
documented probes that peti has **no data at all** in `felhom-offsite`, which is what made the
|
||||
operator-approved 100% datastore-fill safe. The 13 GB rollback copy `/srv/pbs-felhom` on ep0 is
|
||||
intact.
|
||||
|
||||
## Findings
|
||||
|
||||
| # | Finding | Severity | Class |
|
||||
|---|---|---|---|
|
||||
| F-CRIT-1 | An app that fails to restart after a quiesce **never alarms**, on any channel | **HIGH** | DEFECT |
|
||||
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT |
|
||||
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
|
||||
| F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT |
|
||||
| F-LEAK | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed | MEDIUM | DEFECT (root-caused by fault 18) |
|
||||
| F-REBOOT | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
|
||||
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
|
||||
| F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT |
|
||||
| F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational |
|
||||
|
||||
Both HIGH findings are in the same place: **the system's ability to tell you a backup did not
|
||||
happen.** Both cite the code and the comment that asserts the property the code does not provide.
|
||||
|
||||
## What is now proven that was not before
|
||||
R-88 breaker arming **and its full ladder** (15m/30m/1h/2h/4h/4h-cap) · per-tier isolation under a
|
||||
real one-tier-fails case · `whole_guest_backup_failed` end-to-end with correct tier attribution ·
|
||||
R-97c operator-only routing verified against the hub DB (zero customer rows with `status='sent'`) ·
|
||||
`whole_guest_backup_recovered` + the R-68 pairing gate firing live · **`age_state=absent`** ·
|
||||
R-97b's suppression half · the crash-recovery unquiesce by an actual SIGKILL (1 s) ·
|
||||
**R-87 — the first restic restore round-trip ever performed**, byte-verified (6/7 sha256 identical,
|
||||
the 7th explained) · R-82 one-quiesce-two-tiers · single-flight on two independent paths.
|
||||
|
||||
## Restore round-trips
|
||||
restic (R-87) · local vzdump → fresh CT · PBS offsite → fresh CT · corrupted snapshot → fails
|
||||
cleanly. `mount_parity` exact on both whole-guest tiers, `unprivileged: 1` preserved.
|
||||
|
||||
## Fleet state
|
||||
**Healthy. Nothing left broken.** All four compression knobs reverted and verified; every fault
|
||||
unwound; no leaked scratch guests, nft rules, ballast files or clock skew; ep0 datastore clean with
|
||||
zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
|
||||
|
||||
## Not tested (with reasons)
|
||||
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the
|
||||
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), fault 12 (host
|
||||
reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for
|
||||
documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings.
|
||||
|
||||
**Campaign-caused outage, stated plainly:** fault 11 took demo-hp guest 9201 down for ~9m47s
|
||||
(the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after.
|
||||
|
||||
## Note on repo conventions
|
||||
This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an
|
||||
entry — there is nothing shipped to log. The deliverable is the audit document plus this report.
|
||||
@@ -1,85 +0,0 @@
|
||||
# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28)
|
||||
|
||||
**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign,
|
||||
and findings are recorded, never fixed inline. Full write-up:
|
||||
`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`.
|
||||
Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously).
|
||||
|
||||
Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes.
|
||||
**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact.
|
||||
|
||||
## The headline — two never-proven restore paths are now proven
|
||||
|
||||
Driven through the **real endpoints the UI posts to**, over https through traefik with a real session
|
||||
and CSRF token, on live hardware.
|
||||
|
||||
| proof | result |
|
||||
|---|---|
|
||||
| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
|
||||
| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
|
||||
| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
|
||||
| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
|
||||
| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
|
||||
| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
|
||||
|
||||
The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
|
||||
at their last-backup state.**
|
||||
|
||||
## Findings — 3 defects, ranked (none fixed)
|
||||
|
||||
| # | finding | severity |
|
||||
|---|---|---|
|
||||
| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
|
||||
| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
|
||||
| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
|
||||
|
||||
Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
|
||||
sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
|
||||
artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
|
||||
so live.
|
||||
|
||||
## Mechanisms confirmed working, live
|
||||
|
||||
R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target
|
||||
breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app
|
||||
thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of
|
||||
the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on
|
||||
demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's
|
||||
sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the
|
||||
hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable;
|
||||
F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests).
|
||||
|
||||
## Where it stopped, and what remains
|
||||
|
||||
Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host
|
||||
reboot mid-backup) was deliberately not started — a large new fault class against boxes that are
|
||||
remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown.
|
||||
|
||||
**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's
|
||||
threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time
|
||||
`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign
|
||||
forbids. Reported rather than worked around. The no-code-change alternative (age the controller's
|
||||
reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended
|
||||
method next time.
|
||||
|
||||
**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an
|
||||
intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way
|
||||
concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`.
|
||||
|
||||
## Recovery
|
||||
|
||||
Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive
|
||||
observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active),
|
||||
windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on
|
||||
either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and
|
||||
the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots).
|
||||
|
||||
One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box.
|
||||
Its `until` is long past so it blocks nothing; it clears on the next successful backup or any
|
||||
controller restart (by design, not persisted). Clearing it would have cost another app outage for no
|
||||
benefit.
|
||||
|
||||
**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in
|
||||
`/opt/docker/stacks/<app>` starts a Felhom app **without its secrets** — they are injected by the
|
||||
controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a
|
||||
crash loop during recovery. Manual recovery must go through `POST /api/stacks/<name>/restart`.
|
||||
@@ -1,71 +0,0 @@
|
||||
# REPORT — DIAGNOSE immich offsite restore (felhom.eu side: docs only)
|
||||
|
||||
> Written as `REPORT-diag-immich-restore.md`, **not** the shared `REPORT.md`, per the CLAUDE.md
|
||||
> convention — `REPORT.md` is overwritten and currently holds the 2026-07-18 website refresh.
|
||||
> No code shipped in this run; findings only.
|
||||
|
||||
**Date:** 2026-07-19 · **Box:** demo-felhom (felhom-pve guest 9201), controller 0.146.0, immich v3.0.3
|
||||
|
||||
> **CLOSED IN CODE 2026-07-19 — controller v0.148.0.** Findings 1 and 2 shipped as R-43 (offsite
|
||||
> reconstitution: safety dump → stop → overwrite files → start → replay the snapshot's dump) and
|
||||
> R-44 (every run dumps before it captures; manifests carry `offsite_run_id` + `dumps_at`). Deployed
|
||||
> to demo 9201, healthy. **The §9 live acceptance has NOT run**, so no capability-map flip: the
|
||||
> offsite row is PARTIAL, the customer-restore row stays MISSING, R-3 stays DRAFT. Implementation
|
||||
> detail lives in `felhom-controller/REPORT.md`.
|
||||
|
||||
## What ran
|
||||
|
||||
A diagnosis of "immich offsite restore succeeds but photos do not reappear". No product code was
|
||||
changed: no restore/backup logic, no labels, no flashes, no `restic prune`/`forget`, no snapshot or
|
||||
escrow changes.
|
||||
|
||||
## Outcome
|
||||
|
||||
The restore did not fail. **It was never invoked on missing data, and could not have worked
|
||||
if it had been.**
|
||||
|
||||
1. Viktor deleted the 11 photos in the immich UI to test offsite restore. A UI delete means
|
||||
**trash**, not deletion — no file left the disk. Both „csak a hiányzó fájlok" runs merged
|
||||
**0 files**, correctly, and flashed success. The test proved nothing.
|
||||
2. A *valid* test would also have failed: **no offsite path loads a database.** All three buttons
|
||||
are file-only. Files would return; the library would stay empty.
|
||||
3. The shipped dump is additionally stale by design — from the 02:30 local run, never refreshed
|
||||
before a manual push. Probed: **`asset: 0`, `user: 0`, `album: 0`**.
|
||||
|
||||
**Photos:** left in trash at Viktor's instruction (recovery not wanted). All 11 files verified
|
||||
present on disk and all 11 rows intact, so an ordinary un-trash recovers them until immich's
|
||||
30-day auto-purge.
|
||||
|
||||
**Answer to "can a customer trust same-day offsite?"** For a DB-indexed app: **no — files come
|
||||
back, content does not.** The backup half is honest; the restore half cannot reconstitute the app.
|
||||
|
||||
## Decisive evidence
|
||||
|
||||
- `updatedAt` == `deletedAt` on all 11 asset rows ⇒ **no restore operation touched the DB.**
|
||||
- Unit dump `immich-postgres.sql`, 51 954 452 B, mtime **02:30 CEST** ⇒ `asset: 0 / user: 0 /
|
||||
album: 0`. The 52 MB is entirely immich's shipped geodata reference tables. It predates the admin
|
||||
user (07:56:25) and the photos (07:57).
|
||||
- **Latent hazard:** had a full restore loaded that dump, it would have written an empty DB over the
|
||||
live one, destroying the trashed rows that were the only surviving recovery path.
|
||||
|
||||
## Files written
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `documentation/audits/DIAG-immich-restore-2026-07-19.md` | new — timeline, evidence, source mechanism table, 7 findings |
|
||||
| `documentation/backlog/ROADMAP.md` | **R-43** (P2-HIGH, offsite restore cannot restore a DB) + **R-44** (P2-HIGH, manual push ships unrefreshed dump) |
|
||||
| `documentation/architecture/00-capability-map.md` | customer-restore row **stays MISSING**, gains the finding + a method note for R-3; offsite-restore row flagged *scope contested* |
|
||||
|
||||
## Needs Viktor
|
||||
|
||||
1. **Capability-map ruling (line 61).** The offsite-restore row claims PROVEN-LIVE citing
|
||||
"immich end-to-end from offsite alone" (`CAMPAIGN-6D`). That phrase is contested by this
|
||||
diagnosis. I flagged it rather than downgrading a proven row — did 6D's accept leg exercise the
|
||||
DB half, or only the file half?
|
||||
2. **Optional real red-proof.** Emptying the trash would create genuine data loss and convert the
|
||||
dump-probe inference into a live proof. Offered, **not run** — irreversible, and the probe
|
||||
already settles it.
|
||||
3. **Unreconciled: the 704.6 MiB figure.** Measured 126 MB total on the library storage. If
|
||||
704.6 MiB came off a controller Storage page, that gap is its own defect.
|
||||
4. **Orphaned pre-v3 tree** `dccc13fe…` (~55 MB across upload/thumbs/encoded-video) stranded by the
|
||||
immich 2→3 redeploy — worth a sweep policy for major redeploys.
|
||||
@@ -1,53 +0,0 @@
|
||||
# REPORT — Megosztás diagnosis (2026-07-20)
|
||||
|
||||
Topic-scoped report (parallel-session rule: shared `REPORT.md` untouched).
|
||||
|
||||
**Run:** RUNBOOK "Megosztás diagnosis — SMB unreachable from Mac + sharing-page reload loop".
|
||||
Read-only diagnosis. **No code changes, no version bumps, no builds, no restarts.**
|
||||
|
||||
**Deliverable:** `documentation/audits/DIAG-sharing-2026-07-20.md`.
|
||||
|
||||
## Verdicts
|
||||
|
||||
* **Reload loop — ROOT-CAUSED (HIGH).** `sharingStatusHandler` (`sharing_handlers.go:246`, added in
|
||||
`b5d78d1`, controller v0.147.0, 2026-07-19) coerces `idle` → `running` whenever the samba
|
||||
container is alive. `sharing.html` L320–326 treats `running` as a one-shot job-success edge and
|
||||
calls `location.reload()` 1.2 s later. The first `tick()` fires synchronously on every page load,
|
||||
so the page reloads forever. Unconditional for any customer with sharing enabled — the Megosztás
|
||||
page is currently unusable. Proven live: 6 consecutive `/sharing/status` polls all returned
|
||||
`{"phase":"running","running":true}`, and the controller log shows **no ensure job ran at all**,
|
||||
so the phase is manufactured by that line rather than left over from a stuck job.
|
||||
* **`smb://192.168.0.162` — ROOT-CAUSED.** `.162` is the Proxmox host and never was an SMB endpoint.
|
||||
smbd runs in guest 9201 and binds `192.168.0.104:445`. `nc` from the host: `.104:445` **open**,
|
||||
`.162:445` **refused**. Wrong target, stale Finder favourite.
|
||||
* **`smb://FELHOM` — OPEN, narrowed.** NetBIOS resolution works on the wire
|
||||
(`nmblookup -B 192.168.0.255 FELHOM` → `192.168.0.104 FELHOM<00>`), but the stack advertises **no
|
||||
mDNS/Bonjour** (nothing on udp/5353; the R-6 spike selected `smbd + nmbd + wsdd` only) — the
|
||||
mechanism macOS Finder prefers. Closing this needs one probe from the Mac (`smbutil lookup FELHOM`
|
||||
/ `dns-sd -B _smb._tcp`), listed in the audit's Mac test matrix.
|
||||
* **H2 (container down/crash-looping) — RULED OUT** at the first probe: `felhom-samba` Up 3 h, clean
|
||||
logs, smbd/nmbd/wsdd all bound as `infra/samba.go` intends, live `smb.conf` matches the renderer
|
||||
with no baked address literal, no PVE firewall in the path.
|
||||
|
||||
## Findings (full table + evidence in the audit doc)
|
||||
|
||||
S-1 HIGH reload loop (XS fix: latch a `sawInFlight` flag in the JS; red-proof required) ·
|
||||
S-2 MED the UI never shows the connect address, only `\\FELHOM` — customers guess IPs ·
|
||||
S-3 MED no mDNS advertisement (image slice, needs republish) ·
|
||||
S-4 LOW ensure-job phase never resets — fold into the async-job-feedback roadmap item ·
|
||||
S-5 INFO the guest's LAN IP is DHCP, so any displayed address must be read live, never cached.
|
||||
|
||||
Recommended packaging: S-1 as an immediate patch task (it bricks a shipped page), S-2 alongside it
|
||||
if the guest IP is reachable from the sharing handler, S-3 as its own slice.
|
||||
|
||||
## Also noticed
|
||||
|
||||
The **remote site's LAN is `192.168.0.0/24` — the same prefix as the DooPlex home LAN** that the
|
||||
Tailscale subnet router advertises (`192.168.0.180` shows `FAILED` in felhom-pve's neighbour table).
|
||||
A successful `ping 192.168.0.162` therefore does not by itself prove the Mac is on the remote
|
||||
segment; the Mac matrix starts by confirming which network it is on.
|
||||
|
||||
## Actions taken
|
||||
|
||||
None. Every command was a read, except a `POST /login` to obtain a session for the status polls.
|
||||
No secrets are recorded in either document.
|
||||
@@ -1,85 +0,0 @@
|
||||
# REPORT — R-111 fixed, then E-2 proven on a fresh box (2026-07-29)
|
||||
|
||||
Two phases in one session. Full evidence: `documentation/audits/E2D-fresh-vm-2026-07-29.md`.
|
||||
Root `REPORT.md` untouched.
|
||||
|
||||
## Phase 1 — R-111: the Day-0 channel now serves the current software
|
||||
|
||||
A Phase 0 gate earlier the same day stopped the E-2d run before any VM existed: a fresh box would
|
||||
have installed **agent 0.96.0 + controller 0.161.0**, ~17 and ~24 releases behind `main`.
|
||||
|
||||
| | Before | Now |
|
||||
|---|---|---|
|
||||
| agent (Gitea generic) | 0.96.0 | **0.113.0**, sha `5f3247f7…`, round-trip verified |
|
||||
| golden (Gitea generic) | 0.161.0 | **0.185.1**, sha `dba00f3e…`, embeds controller 0.185.1 |
|
||||
| hub `min_agent` | 0.93.0 | **0.113.0** (what controller v0.185.0 declares) |
|
||||
|
||||
Bake clean on every marker: `Result=success`, overlay2, **all three mounts in the archive**, 0
|
||||
FATAL/exclusions, HTTP 201, token-leak grep 0. GL-1 teardown: guest 9100 purged, secrets shredded,
|
||||
drill disk restored to `virgin`. Agent + golden moved in **one** manifest POST so it never vouched a
|
||||
new agent against an old golden. `min_agent` verified zero-impact first (all three enrolled hosts
|
||||
already at 0.113.0). Global floor deliberately **not** raised — the golden now bakes 0.185.1.
|
||||
|
||||
Commit `3dff357`.
|
||||
|
||||
## Phase 2 — the E-2d run, full ISO/PAIRING route
|
||||
|
||||
Nested PVE VM on demo-hp, one disk, outside the `felhom` pool. Bind → running controller in
|
||||
**3 m 35 s**. The install fetched exactly the artifacts published an hour earlier and restored
|
||||
`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked 20 minutes before. The publish train is
|
||||
proven end to end on a real install.
|
||||
|
||||
| Claim | Verdict |
|
||||
|---|---|
|
||||
| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** |
|
||||
| **C2** Case B fires naturally | ✅ **PROVEN** — both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort |
|
||||
| **C3** degraded banner renders **to a customer** | ⚠️ **PARTIAL** — API byte-exact; **no UI consumer exists** → **R-112** |
|
||||
| **C4** offer appears and moves the target | ⚠️ **PARTIAL** — decline path, `restart_required:true`, no self-restart, E-2a wrapper, healthy-renders-nothing all PROVEN at API level; offer equally invisible → **R-112** |
|
||||
| **C5** `backup_target_absent` end to end | ❌ **FAILED** — zero events on any channel → **R-113** |
|
||||
|
||||
## The three findings
|
||||
|
||||
**R-112 (P1)** — E-2's banner and offer have **no UI consumer**. The endpoint returns byte-exact copy;
|
||||
`grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**, and no page handler injects
|
||||
the state. Decisive contrast: templates fetch **18** distinct `/api/storage/*` endpoints;
|
||||
`backup-target` and `backup-target/assign` are the only two with zero references. v0.185.1 fixed the
|
||||
router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired.
|
||||
|
||||
**R-113 (P1)** — the drive-absent gate **cannot fire on device loss**. `planDriveGates` reads presence
|
||||
from `BoundUnderParent` = "is this path in the guest's mountinfo". The raw mount is a device-bound
|
||||
systemd unit and dies; **the agent's own bind is not device-bound and outlives the device**, so the
|
||||
gate sees "present" forever. Live: agent said `enrolled drive absent by UUID` every 20 s for 4½
|
||||
minutes, controller logged **0** `[gate]` lines, hub got **zero** events — neither the specific nor the
|
||||
generic one. Sixth instance of the class, one layer deeper: E-2b wired the seam to a condition that
|
||||
cannot occur.
|
||||
|
||||
**R-114** — on target-drive loss the message says the backup is *"on the same disk as the system"*
|
||||
(false) and offers **the drive that just vanished**. Invisible today only because of R-112 — so
|
||||
**R-114 must be fixed before R-112 is wired.**
|
||||
|
||||
Also filed as a **second instance under R-110** (not a new ID): host-install fetches **nine** files
|
||||
from `raw/branch/main` and the hub vouches a sha for **one**; E-2a's wrapper is installed 0755 to
|
||||
`/usr/local/sbin`, root-fenced in sudoers, validated only by `bash -n`.
|
||||
|
||||
## Record
|
||||
|
||||
- `OPEN-ITEMS.md` — **R-112/R-113/R-114 opened**; E-2d re-stated with results and left open for the
|
||||
residue; E-2's "NOT yet live-proven" list resolved into proven / known-broken; R-94 fully unblocked;
|
||||
R-110 extended. The drill-cleanup row was opened and then **closed the same session** once the
|
||||
teardown completed, so it is not carried in the register.
|
||||
- `ROADMAP.md` — R-112/R-113/R-114 under P1; R-111 marked SHIPPED.
|
||||
- **`architecture/00-capability-map.md` not touched** — for two reasons: the customer-facing legs are
|
||||
broken rather than proven, and the map has **no E-2 / backup-target rows at all** (worth noting
|
||||
against the ROADMAP's coupling rule).
|
||||
|
||||
## Teardown
|
||||
|
||||
VM destroyed, scratch storage removed, **`pvesm status` after == before** (`local-lvm` 38.77 %,
|
||||
byte-identical), guest 9201 and drill-r50 untouched. **Hub records removed — teardown complete.** The delete was correctly refused at four gates while the host still read ONLINE; once the destroyed host aged to DOWN (`delete-impact` → `deletable:true`) the documented cascade ran and completed: host deleted, PBS tenancy deprovisioned, claim reset, residue purged. Verified after: **0** `e2d` occurrences on the hosts page, fleet unchanged. The one purged `appliance_registrations=1` was this run's own appliance; the unrelated stale 2026-07-25 appliance (`206c8838…`) was not touched by the cascade — the operator removed it separately.
|
||||
|
||||
## One human step, and a premise correction
|
||||
|
||||
The runbook's §5.1a operator STOP (the bind) is **retired** — CC did it. But E-2d's premise that a
|
||||
fresh install yields a CC-drivable claimable customer is **wrong**: the claim code is bcrypt-hashed and
|
||||
email-only, and the gate covers everything except `/claim`, `/api/health`, `/static/`. One operator
|
||||
relay of the emailed code was required — which also proved the claim flow end to end.
|
||||
@@ -1,92 +0,0 @@
|
||||
# REPORT — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run (RUNBOOK execution). **No code changed. No version bump.**
|
||||
Written as `REPORT-<topic>.md` per the parallel-session rule — the shared `REPORT.md` was not touched.
|
||||
|
||||
**Full record with all evidence:** `documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`
|
||||
|
||||
---
|
||||
|
||||
## Outcome: DONE and verified
|
||||
|
||||
`felhom-offsite` now lives on a dedicated 100 GB Hetzner Cloud Volume instead of ep0's 40 GB root disk.
|
||||
|
||||
| | Before | After |
|
||||
|---|---|---|
|
||||
| Path | `/srv/pbs-felhom` (root disk) | **`/mnt/pbs-datastore`** (volume) |
|
||||
| Datastore total | 37.2 GB | **98 GB** (hub gauge: 97.9 GB) |
|
||||
| Used | 28.9 % | **13 %** (hub gauge: 12.6 GB, 13 %) |
|
||||
| Headroom to the 80 % warn | 19 GB | **≈65 GB** |
|
||||
| Additional customers before warn | ≈2 | **≈7–13** |
|
||||
|
||||
Datastore **name unchanged** — the PBS-DR descriptors, per-box storage ids, ACLs and namespace
|
||||
layout that R-39/R-82 made self-healing are untouched.
|
||||
|
||||
**Window:** 06:58 → 07:19 UTC (PBS down 07:00 → 07:17). **Nothing was deleted.**
|
||||
|
||||
### Acceptance evidence
|
||||
|
||||
| Gate | Result |
|
||||
|---|---|
|
||||
| Copy integrity | 13,242,207,822 = 13,242,207,822 B · **9,748 = 9,748 chunks** · 7 = 7 snapshots · `backup:backup` · itemised dry-run **0 lines** |
|
||||
| Snapshot counts per ns | `demo-felhom` 2=2, `demo-felhom-01` 3=3, `demo-hp` 2=2 |
|
||||
| atime semantics | `rw,relatime,discard` — **`relatime` present, `noatime` absent** (GC correctness) |
|
||||
| Verify job | `TASK OK`, 3/3 groups, forced re-verification of every snapshot, **0 errors** |
|
||||
| §6 mount guard | **refusal observed** — `Job … failed with result 'dependency'`; mountpoint stayed empty |
|
||||
| §8 restore round-trip | `source_tier: pbs`, `pass: true`, `mount_parity: ok`, clean teardown, 12m1s |
|
||||
|
||||
---
|
||||
|
||||
## Three findings the operator should act on
|
||||
|
||||
1. **`scratch` datastore is configured at a path that does not exist** (`/srv/pbs-scratch`).
|
||||
Pre-existing, not caused here, but now logs `ENOENT` on every PBS start. This is the PRIME RISK
|
||||
shape ("reports fine, is not there") already live in the config. **Decision needed:** remove the
|
||||
stanza or create the directory.
|
||||
|
||||
2. **The runbook's §6 acceptance test proves the wrong proposition.** `RequiresMountsFor` is a
|
||||
mount-first ordering guarantee, not a refusal — systemd silently *re-mounts* an unmounted volume
|
||||
and PBS then starts safely. The test only bites when the device is genuinely unavailable, which
|
||||
is how it was re-run and passed. **Amendment recommended in the runbook record.**
|
||||
|
||||
3. **§11 — storage box `u629193` is NOT simply unused.** No live backup path references it (no
|
||||
datastore, no restic repo, no fstab, no `known_hosts` pin; R-17 already deleted `u629193-sub1`),
|
||||
**but ep0 carries an enabled, currently-mounted sshfs unit** `mnt-pbs\x2dstoragebox.mount` →
|
||||
`/mnt/pbs-storagebox`, holding spike leftovers. Disable and remove that unit before deleting the
|
||||
box, or ep0 logs a failed mount every boot. **The deletion is the operator's console click.**
|
||||
|
||||
## Deviations from the runbook as written
|
||||
|
||||
- **The volume arrived already formatted and mounted** by Hetzner at `/mnt/HC_Volume_106469259`
|
||||
(§2 assumed neither). Operator ruled: reformat + repath. The 5 % reserve was reclaimed (`-m 0`).
|
||||
- **§8 ran on demo-felhom, not demo-hp** — DooPlex holds no SSH key for demo-hp (the G1 gap). Same
|
||||
tier, same relocated datastore, larger archive.
|
||||
- **The window was contended** by a stale 10-minute restore-test cadence on demo-felhom: the config
|
||||
had already been reverted to 3.5 days on disk, but the cadence is read once at daemon start and
|
||||
`NRestarts=0`. Restarting the agent applied it (`cadence=84h0m0s`). The in-flight test was allowed
|
||||
to finish rather than aborted.
|
||||
|
||||
## Process errors made during this run (recorded deliberately)
|
||||
|
||||
- `rsync -aHAX` **OOM-killed** ep0 (3.7 GB RAM, no swap). Cause: a PBS `.chunks/` tree pre-creates
|
||||
all 65536 shard dirs → 75,341 inodes, and `-H` retains the whole inode map. `-H` was dropped only
|
||||
after **proving** no hardlinks exist (`-links +1` → 0; max link count → 1); PBS references chunks
|
||||
by digest, never by hardlink.
|
||||
- `/usr/bin/time -v` is not installed on ep0 → exit 127, rsync never ran, and a `| grep … || true`
|
||||
wrapper swallowed it while printing a success-looking line.
|
||||
- `rsync --version | head -1` reported a working rsync 3.4.1 as "missing" — **the §12 pipe-into-head
|
||||
trap, fourth recorded instance in this project.** Both fixed by capturing the command's own `$?`.
|
||||
|
||||
## Deferred
|
||||
|
||||
1. **Old copy retained** at `/srv/pbs-felhom` (13 GB, 9,748 chunks) as the rollback. Rollback is a
|
||||
two-line `datastore.cfg` revert. Reclaim only after a new weekly offsite backup lands on the
|
||||
volume, with explicit go-ahead.
|
||||
2. **GC not run** — now unblocked by the round-trip, but left for a separate deliberate run. No GC
|
||||
schedule is configured on this PBS at all.
|
||||
3. ~~Hub PBS-DR capacity gauge not re-read.~~ **CLOSED — verified correct.** The hub operator UI
|
||||
(Offsite → PBS DR) reports `felhom-offsite (ep0)` at **97.9 GB capacity, 12.6 GB used, 13 % full**,
|
||||
agreeing with the on-box `df`. The gauge follows the datastore's configured path, so the move
|
||||
needed no hub-side change and the suspected "wrong filesystem" bug does not exist.
|
||||
4. **ep0 has no swap** (temporary 4 GB file removed; box left as found). Worth a small permanent
|
||||
swapfile — outside this runbook's scope.
|
||||
@@ -1,63 +0,0 @@
|
||||
# REPORT — F-CRIT-1 + F-A1 fixed (controller v0.179.0, 2026-07-28)
|
||||
|
||||
Docs here. Implementation, all six red-proofs and the full live replay live in
|
||||
`felhom-controller/REPORT.md`. The campaign that found both:
|
||||
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
|
||||
## What changed
|
||||
**F-CRIT-1** — an app that failed to restart after a quiesce never alarmed, for two independent
|
||||
reasons, either of which alone kept it dead: `restartAll` returned nothing (the failure was logged
|
||||
and dropped), and `classifyRunStates` whitelisted `StateStopped` on invariant I1 ("the user stopped
|
||||
it") — which the quiesce loop had made false by stopping stacks the same `docker compose down` way.
|
||||
A failed restart and a user stop are the *same* Docker state; the only difference is that the loop
|
||||
tried and could not, now surfaced by `Loop.FailedRestarts()`.
|
||||
|
||||
**F-A1** — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a
|
||||
failure. It is now contention: the tier stays DUE, is dropped before anything stops, and unending
|
||||
contention raises its own **BLOCKED** signal rather than going silent.
|
||||
|
||||
## Bounds, justified against measured reality
|
||||
- `contentionRetryAfter` **15m** — longest restore-test observed on the fleet is 12m01s; the agent's
|
||||
local restore-test wait is 10m. Caps app-stop churn at 4/hour instead of 12/hour.
|
||||
- `contentionAlarmAfter` **3h** — the agent's own PBS restore-test task is capped at 120 minutes, so
|
||||
contention outliving that is a *stuck* gate, not a busy one. 3h adds margin and is 15× the longest
|
||||
contention actually observed.
|
||||
|
||||
## Verified live, with the hub DB as arbiter — not from logs
|
||||
Same box, same day, same event type; the only difference is 409 versus a genuine error:
|
||||
|
||||
| injection | operator emails (demo-hp) |
|
||||
|---|---|
|
||||
| **409 contention** | 8 → **8** (none) |
|
||||
| **real transport failure** | 8 → **9** |
|
||||
|
||||
And for F-CRIT-1: the failed restart alarmed **9 seconds** after grace expiry with the dashboard
|
||||
banner naming the `(stopped)` state, while a **deliberate** user stop on the same box stayed silent
|
||||
through **9** dead-app scans (the positive observable that the silence is suppression, not a dead
|
||||
detector).
|
||||
|
||||
## The rule this arc earned
|
||||
Added to **both** copies of `CLAUDE.md` (live + `documentation/runbooks/workspace-CLAUDE.md`):
|
||||
**a comment asserting an invariant needs a test pinning it, or it is a wish.** Six instances in this
|
||||
project have shipped guarantees the code did not provide — `EffectiveProtected`, `newestArchiveOn`,
|
||||
the R-97a operator-only claim, `classifyRunStates`' I1, `inflight.go`'s defer claim, and
|
||||
`quiesce.go`'s spurious-failure claim. Two were found only on live hardware, and one of those had a
|
||||
green, red-proofed test suite over a production path broken two independent ways.
|
||||
|
||||
Corollary recorded with it: prefer a test that asserts the **consequence** (does the alarm fire?)
|
||||
over one that asserts the **mechanism** (does suppression expire?). R-97b's Scenario F proved the
|
||||
mechanism; the consequence was still broken.
|
||||
|
||||
## Docs touched
|
||||
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-1 and F-A1 → SHIPPED + PROVEN-LIVE.
|
||||
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — both rows struck through, closing
|
||||
section added. **All three of the campaign's alarm findings are now closed** (F-CRIT-1, F-CRIT-2,
|
||||
F-A1).
|
||||
- `documentation/runbooks/workspace-CLAUDE.md` — the invariant rule.
|
||||
|
||||
## Still open, highest first
|
||||
**Fault 4** (restic transport interruption) — four injection approaches were defeated by
|
||||
guest-bridged networking, and it is now the most valuable follow-up: F-CRIT-2 answered the phantom
|
||||
question for PBS and left the identical question open for restic. Then **R-99** (prune never removes
|
||||
phantoms) and **F-LEAK** (a failed restore-test cannot destroy its own scratch guest — observed
|
||||
again during this work).
|
||||
@@ -1,40 +0,0 @@
|
||||
# REPORT — F-CRIT-2 fixed: a failed backup no longer looks like a fresh one (2026-07-28)
|
||||
|
||||
Scope: `felhom-agent` v0.105.0 → **v0.106.0**. Docs here. Implementation detail and the full live
|
||||
re-test live in `felhom-agent/REPORT.md`; the campaign that found it is
|
||||
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
|
||||
## What changed
|
||||
`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, and NEWEST) as a
|
||||
successful backup, so the tier read fresh, went **not due**, and was never retried — seven days of
|
||||
silence on the real 168h cadence, invisible to both the R-88 breaker (defers only *due* tiers) and
|
||||
the hub deadline monitor (reads the same freshness). It now counts only *plausibly complete*
|
||||
entries via a measured 1 MiB floor; undecidable ⇒ not counted.
|
||||
|
||||
**Size is the only tier-agnostic discriminator.** `verification` and `encrypted` are absent on every
|
||||
local (dir) archive AND on a good PBS snapshot until `verify-new` catches up — gating on either
|
||||
would have rejected 100% of local backups and produced fleet-wide backup thrash. That inverse risk
|
||||
is a first-class test, red-proofed by making the filter reject everything.
|
||||
|
||||
## Verified live, not just in unit tests
|
||||
Campaign fault 2 was replayed against the fixed agent on demo-hp — phantom created, rejected and
|
||||
announced once; the tier correctly reported DUE and backed up (4,359,968,099 B landed); and the
|
||||
inverse showed **no thrash**, with 91 scheduler ticks as the positive observable that the loop was
|
||||
alive rather than dead.
|
||||
|
||||
## Settled along the way — no retention bug
|
||||
Server-side prune does **not** count phantoms toward `keep-last`: a dry-run against three real
|
||||
snapshots plus a phantom retained two real ones plus the phantom. The feared "two phantoms ⇒ zero
|
||||
real backups" does not occur. Prune never removes them either, so they accumulate one per aborted
|
||||
upload — filed as **R-99** (LOW, hygiene), not as a retention bug.
|
||||
|
||||
## Docs touched
|
||||
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-2 → SHIPPED+PROVEN-LIVE; **R-99** filed;
|
||||
**F-CRIT-1** filed as READY-HIGHEST (Campaign 8's other HIGH finding, untouched here).
|
||||
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — F-CRIT-2 row struck through and
|
||||
a closing section added.
|
||||
|
||||
## Still open, highest first
|
||||
**F-CRIT-1** — an app that fails to restart after a quiesce never alarms, for two independent
|
||||
reasons. Then fault 4 (restic transport interruption), which this fix makes more pointed: the
|
||||
phantom question is now answered for PBS and still open for restic.
|
||||
@@ -1,140 +0,0 @@
|
||||
# REPORT — F-REBOOT + F-LEAK + F-OBS, and two investigations (2026-07-28)
|
||||
|
||||
Scope in this repo: **`scripts/felhom-host-install.sh` v1.20.0 → v1.21.0** (which is where F-LEAK's
|
||||
*actual* fix lives), plus the Campaign 8 audit doc and `OPEN-ITEMS.md`. Written as
|
||||
`REPORT-freboot-fleak-fobs.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
Code companions: `felhom-agent` v0.106.0 → **v0.110.0**, `felhom-controller` v0.179.0 → **v0.180.0**.
|
||||
|
||||
**Correction to this repo's part of the story:** v1.21.0's band-scoped ACL is *not* the final F-LEAK
|
||||
fix. It works, but only **once per slot** — PVE's destroy path calls
|
||||
`AccessControl::remove_vm_access($vmid)` (`API2/LXC.pm:906`), which deletes every ACL at `/vms/<vmid>`
|
||||
(`AccessControl.pm:1898`), so **the grant is consumed by the operation it authorises**. Found by counting
|
||||
ACL rows after the first successful teardown (`/vms/990000` → 0 grants), not by reasoning about it. The
|
||||
durable fix is agent **v0.110.0**'s band-scoped fenced destroy; v1.21.0 remains valuable because it makes
|
||||
the common case need no privileged call, and it is now the *first* of two layers rather than the only one.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom.eu d0cec9d`, `felhom-agent af1c21a`, `felhom-controller fb91c8d`, all clean. Agent `0.106.0`
|
||||
and controller `0.179.0` live on both demo boxes.
|
||||
|
||||
---
|
||||
|
||||
## host-install v1.21.0 — F-LEAK, and why the fix landed *here* rather than in the agent
|
||||
|
||||
**The finding.** A restore-test whose restore **fails** leaves a scratch guest the agent cannot destroy
|
||||
(`403 missing privilege VM.Allocate`), so a half-restored guest holds its disks until a human removes
|
||||
it and the 10-slot scratch band shrinks silently.
|
||||
|
||||
**The cause is structural, not a missing privilege in the role.** `FelhomAgentGuest` is granted at
|
||||
`/pool/felhom`, and **a guest joins that pool only when its restore completes**. A failed restore
|
||||
therefore produces a guest that exists, is in no pool, and is out of the token's reach entirely.
|
||||
|
||||
**The first fix was wrong, and its own live replay is what proved it.** Agent v0.107.0 shipped a
|
||||
teardown fallback that adopted the stranded guest into the pool and retried — reasoning from
|
||||
`Pool.Allocate` on `/pool/felhom`. It fired exactly as designed and PVE refused it:
|
||||
|
||||
```
|
||||
ERROR restore-test: pool adoption failed; left for Recover vmid=990000
|
||||
err="proxmox: PUT /pools/felhom -> HTTP 500: permission denied at /vms/990000 (missing privilege ...)"
|
||||
```
|
||||
|
||||
`PUT /pools/{pool}` **also** requires `VM.Allocate` on the VM being added. **Pool membership cannot
|
||||
bootstrap its own authority.** Removed in agent v0.108.0 rather than left in place — a path that
|
||||
provably cannot work is worse than none, because it reads as a fix.
|
||||
|
||||
**What shipped instead.** `apply_scoped_acl` now grants `FelhomAgentGuest` at each `/vms/<id>` in
|
||||
`PVE_SCRATCH_VMID_MIN..PVE_SCRATCH_VMID_MAX` (990000–990009 — the band the restore-test already picks
|
||||
from), to **both** the user and the token, because the privsep-intersection rule applies here as
|
||||
everywhere.
|
||||
|
||||
Two supporting changes, both load-bearing rather than tidy-up:
|
||||
- **`remove_scoped_acl` deletes the band grants before the role delete.** PVE refuses to delete a role
|
||||
still referenced by any ACL, so omitting this would have broken the uninstall — a failure that would
|
||||
only surface on a decommission.
|
||||
- **`step_verify` asserts the band grants.** A missing one is otherwise invisible until a restore-test
|
||||
*fails*, which is precisely the case that leaked a guest in the first place.
|
||||
|
||||
### Why the grant is still not a widening — proven live, at the seam the defect lives in
|
||||
|
||||
A real PBS restore to `990000` **without `--pool`** reproduced the exact stranded state
|
||||
(`990000 stopped`; `felhom pool members: [9201]`; `990000 in pool: False`). Then, with the agent's own
|
||||
token, same guest, minutes apart:
|
||||
|
||||
| | `DELETE /nodes/<node>/lxc/990000` |
|
||||
|---|---|
|
||||
| **grant removed** (the original defect) | `403 Permission check failed (/vms/990000, VM.Allocate)` — guest still present |
|
||||
| **grant restored** (the fix) | `200 UPID:...:vzdestroy:990000:felhom-agent@pve!agent` — guest gone |
|
||||
|
||||
And it still cannot reach anything else:
|
||||
|
||||
| target | result |
|
||||
|---|---|
|
||||
| `/vms/990010` (one past the band) | **403** `Permission check failed (/vms/990010, VM.Allocate)` |
|
||||
| `/vms/100` | **403** same |
|
||||
|
||||
`990010` does not exist and PVE **still** answered 403 rather than "does not exist" — so PVE evaluates
|
||||
**permission before existence**, which makes these genuine authorization refusals rather than artifacts
|
||||
of a missing guest. Granting at `/vms` was considered and rejected: it would authorise destroying every
|
||||
guest on the box, including a co-tenant's.
|
||||
|
||||
Applied on **both** demo boxes (demo-hp and demo-felhom) so the live fleet matches the installer.
|
||||
|
||||
**A careless step of mine, recorded rather than buried.** The probe loop also issued a live `DELETE`
|
||||
against running guest **9201**. It was refused with `500 container is running` — but the *permission
|
||||
check passed* (9201 is a pool member by design), so had the guest been stopped I would have destroyed
|
||||
the live demo guest. The scratch-band probes were the safe ones; 9201 had no business in that list.
|
||||
|
||||
---
|
||||
|
||||
## Documentation changes
|
||||
|
||||
- **`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`** — F-REBOOT, F-LEAK and F-OBS
|
||||
written up as FIXED with their live evidence, including F-LEAK's refuted first attempt (recorded
|
||||
precisely *because* it looked right), plus a new **§6b** for the follow-up investigation.
|
||||
- **`documentation/backlog/OPEN-ITEMS.md`** — three findings closed, **R-100** filed.
|
||||
|
||||
---
|
||||
|
||||
## R-100 — the investigation's finding, deliberately NOT fixed
|
||||
|
||||
**A restic offsite tier that fails every night never goes stale on the hub.** This is **F-CRIT-2's
|
||||
defect class one layer up and on the other tier** — a *failed* run resetting the freshness clock — and
|
||||
it was found by asking whether F-CRIT-2's shape existed anywhere else.
|
||||
|
||||
Both halves verified in the source, not inferred:
|
||||
- **Controller:** `o.LastRun = time.Now()` is set **unconditionally** at
|
||||
`controller/internal/backup/offbox.go:716`, *outside* the `runErr` branch. The failure is recorded
|
||||
faithfully, but into a different field — `o.LastStatus = "error"` at `:725`.
|
||||
- **Hub:** `isStale()` reads **only** `off.LastRun` (`hub/internal/monitor/offsite.go:120`, `:127`,
|
||||
`:131`) and never consults `LastStatus`.
|
||||
|
||||
So a nightly restic run that fails every night keeps `LastRun` fresh, `isStale` is permanently false,
|
||||
and the staleness alarm never fires — with no successful offsite backup having occurred at all.
|
||||
|
||||
**Scope of the silence, stated precisely rather than dramatically.** `LastStatus` *does* reach the hub —
|
||||
it is parsed into the report struct and **only logged** (`offsite.go:270`); it drives no checker and no
|
||||
notification. The controller's own guest UI surfaces `LastStatus="error"`, so the failure is visible to
|
||||
someone who looks. What is missing is the **push**: the operator's fleet-wide alarm plane is silent,
|
||||
which is the plane that matters for an unattended appliance.
|
||||
|
||||
Not fixed, per this task's investigation-only scope. Fix direction: gate staleness on the last
|
||||
*successful* run rather than the last attempt — exactly what F-CRIT-2's `NewestArchiveTime` fix did for
|
||||
the PBS tier.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
`bash -n scripts/felhom-host-install.sh` clean. The ACL behaviour was verified **live on real hardware**
|
||||
rather than by dry-run, since the whole finding is about what PVE's authorizer actually does — and the
|
||||
live run is what refuted my first design.
|
||||
|
||||
## Fleet state
|
||||
Agent **0.110.0** (with the updated sudoers) and controller **0.180.0** on both demo boxes, all healthy.
|
||||
Scratch-band ACLs at 20 rows on both — re-applied after the attempt-2 destroy consumed one. No leftover scratch guests. demo-hp's `restore_test_cadence_seconds` reverted **600 → 302400**
|
||||
(a bounded change made for the replay).
|
||||
|
||||
`felhom.eu`: this repo has a **foreign uncommitted WIP file** (`documentation/PROMPT-TEMPLATE.md`) from
|
||||
another session in the shared worktree. Left untouched; my commits staged explicit paths only, per the
|
||||
never-`git add -A` rule.
|
||||
@@ -1,119 +0,0 @@
|
||||
# REPORT — ISO boot branding + single-entry GRUB menu (R-38) · website grid restored (2026-07-19)
|
||||
|
||||
> `REPORT-<topic>.md` per this repo's parallel-session rule: another session was writing in this
|
||||
> clone tonight (CAMPAIGN 7 / `DIAG-immich-restore-2026-07-19.md`), so the shared `REPORT.md` is left
|
||||
> untouched.
|
||||
|
||||
Parts 1 and 2 of the polish train. Parts 3 and 4 landed in `felhom-agent` and `felhom-controller`;
|
||||
see their own `REPORT.md`.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — scripts v1.22.0: GRUB branding + single-entry safety (R-38)
|
||||
|
||||
### What shipped
|
||||
|
||||
Every ISO is now **repacked** after `prepare-iso`. `mkimage-surgery.sh` → **`iso-repack.sh`**:
|
||||
branding and the slice-B loader swap need the same extract → modify → re-master cycle, so they share
|
||||
one pass instead of re-mastering twice. **The mkimage recipe is untouched.**
|
||||
|
||||
**The safety half — the one that matters.** The stock PVE menu offers *Graphical*, *Terminal UI*, a
|
||||
serial variant, and an **Advanced Options** submenu holding two `nomodeset` entries, three debug
|
||||
entries, *Rescue Boot*, memtest and *UEFI Firmware Settings*. Every one of those reaches the
|
||||
**manual** installer, whose first question is which disk to wipe. They are **not emitted** — not
|
||||
hidden, not password-gated. What ships is one entry, „Felhom telepítés", default, 5 s.
|
||||
|
||||
**Boot behavior is unchanged.** The `linux`/`initrd` lines are lifted **verbatim at repack time**
|
||||
from the ISO's own *Install Proxmox VE (Automated)* entry rather than frozen into a copy in this
|
||||
repo, so a PVE bump that moves the kernel path or edits the append line tracks automatically. The
|
||||
build **fails** if they cannot be found, if the append line has lost `proxmox-start-auto-installer`,
|
||||
or if `auto-installer-mode.toml` is absent — that last one because without it the single
|
||||
Felhom-labelled entry would boot a *manual* installer, i.e. exactly what this change prevents.
|
||||
|
||||
**Gates, then a re-check against the shipped artifact.** The rendered menu is asserted to have
|
||||
exactly 1 `menuentry`, 0 `submenu`s and no *live* reference to
|
||||
`proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` (comments are stripped first —
|
||||
the template's header names the dropped entries deliberately). Then the menu and theme background
|
||||
are read back **out of `final.iso`**, not out of the extract tree.
|
||||
|
||||
**The boot card.** `grub/generate-grub-background.sh` letterboxes `website/assets/og-image_2.png`
|
||||
onto a 1024×768 gfxterm canvas at repack time (ImageMagick added to the assistant image), so the boot
|
||||
screen has **one source** and not a second pre-rendered PNG to drift. The card's own subtle grid
|
||||
(measured: 4px lines of `#0D131A` on `#0D1117`, pitch 131px) is continued across the letterbox fill
|
||||
**phase-locked** to where the card's grid lands, so the fill is seamless instead of a 500px square of
|
||||
grid floating in flat navy. The generator refuses a source whose geometry no longer matches the
|
||||
measured constants — a swapped asset would misplace every line, and that only shows up on a boot
|
||||
screen nobody re-checks. Menu positioning needs a gfxmenu theme (plain `background_image` cannot move
|
||||
the menu off the wordmark), so `grub/felhom-theme.txt` puts it in the lower third the layout leaves
|
||||
empty, optically centered (measured off a canary screenshot; the comment records the measurement).
|
||||
|
||||
### Live validation — nested canary, UEFI/OVMF, PVE 9.2-1
|
||||
|
||||
Booted the built canary ISO under QEMU with OVMF and captured the framebuffer.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| GRUB renders the Felhom card | **PASS** — background + grid visible at 1024×768 |
|
||||
| Exactly one entry, selected | **PASS** — „Felhom telepítés" only |
|
||||
| Hungarian accents under gfxterm | **PASS** — „telepítés", „Indítás … másodperc múlva" render correctly |
|
||||
| Countdown visible and counting | **PASS** — 5 → 0 |
|
||||
| Auto-fires at 0 | **PASS** — serial shows ``Booting `Felhom telepítés'`` |
|
||||
| Unattended install proceeds | **PASS** — „Fetching answers for automatic installation" → auto installer |
|
||||
| Same abort as v1.21.0 | **PASS** — `ERROR: Installation failed: filter did not match any device` → `Installation aborted`; no disk touched |
|
||||
|
||||
A first build **correctly failed closed**: the banned-entry gate matched the template's own
|
||||
explanatory header. Fixed to strip comments before matching (a comment naming a removed entry is the
|
||||
point; a directive using one is the bug), which is a gate behaving as designed.
|
||||
|
||||
### Artifacts (rebuilt on 180, `/mnt/5_hdd/felhom.eu/felhom-iso/out/`)
|
||||
|
||||
| ISO | sha256 | bytes |
|
||||
|---|---|---|
|
||||
| `felhom-pve-9.2-1-v1.22.0-n100-generic-mkimage.iso` (safety) | `ff6f06ba1dbfe10f27d703afc29516001000349147426b43c9a424a0ea28bdbf` | 1 704 482 816 |
|
||||
| `felhom-pve-9.2-1-v1.22.0-n100-demo-generic-mkimage.iso` (real) | `494db0ddf859b6b152cad4d0e0d9e9cefd27255cde07e2b41aba3ac12a217888` | 1 704 482 816 |
|
||||
| `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso` (validation) | `83c61c0413c84e27b26a37bb5dfaed2fcd44fd25e3e571c7310142bd305f2f9d` | 1 705 338 880 |
|
||||
|
||||
Both shipping ISOs: `embedding 60 modules`, `El Torito boot images=2`, fs-uuid preserved, and the
|
||||
post-re-master verification confirming 1 entry + theme background inside the finished image.
|
||||
|
||||
**Deliberately not done** (per the task): no squashfs/initrd rebranding — post-GRUB screens are still
|
||||
Proxmox-branded; no disk-setup or answer-generation change; the pairing banner is untouched.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — website: the index grid background, restored
|
||||
|
||||
**Archaeology.** Not a deliberate removal. The grid lived as a fixed `body::before` in
|
||||
`index.html`'s inline `<style>` block and was dropped in **`bed8675`** ("D3 Part 2: index + kapcsolat
|
||||
on design system v2"), the commit that migrated the page onto the shared `assets/site.css`.
|
||||
`dd54e4c`, which *created* `site.css`, has no `body::before` at all — it was a porting omission and
|
||||
nothing took its place. `ccbb13a` (the other five pages) never had it. No asset was lost: the
|
||||
mechanism was pure CSS (two stacked `linear-gradient`s), which is why nothing looked missing in the
|
||||
worktree.
|
||||
|
||||
**Restoration, not redesign.** Same 50px cells, same 1px lines, same 3% opacity, same
|
||||
`position:fixed` / `z-index:-1`. One deliberate difference: the accent is the v2 `--blue` `#0083D8`
|
||||
instead of the retired legacy `#0088cc`, which `site_gates.py` bans. Scoped to `body.page-index`,
|
||||
because index is the only page that ever had it. `site.css` cache-bust bumped `?v=1` → `?v=2` across
|
||||
all seven pages (nginx caches 7 days); BOM preserved on every file.
|
||||
|
||||
**Live verification** (felhom.eu, after git-sync deploy):
|
||||
|
||||
- Desktop: grid renders behind the hero, at its original subtlety.
|
||||
- **376px viewport** (via a same-origin iframe — the browser window would not resize in this
|
||||
environment, so the narrow case was exercised for real rather than asserted): grid renders, mobile
|
||||
layout unchanged, `scrollWidth === clientWidth` so **no horizontal overflow**.
|
||||
- Computed style confirmed live: `linear-gradient(rgba(0,131,216,0.03) 1px, …)`, `50px 50px`,
|
||||
`position: fixed`, `z-index: -1`, `pointer-events: none`.
|
||||
- `python scripts/site_gates.py` — **OK** (BOM, no legacy tokens, no `<style>` blocks, cache-busted).
|
||||
|
||||
---
|
||||
|
||||
## Docs
|
||||
|
||||
- `scripts/CHANGELOG.md` — v1.22.0 entry.
|
||||
- `website/CHANGELOG.md` — grid restoration entry.
|
||||
- `documentation/backlog/ROADMAP.md` — **R-38 flipped to SHIPPED**; **R-45** (unified async-job
|
||||
feedback) and **R-46** (verification-copy browse + expiry) added; pre-invite checklist gained the
|
||||
"golden ≥ 0.147.x carries all four infra images" line.
|
||||
- Capability map: **untouched** — no capability moved. These are UX and packaging.
|
||||
@@ -1,307 +0,0 @@
|
||||
# REPORT — the universal ISO: **PUBLISHED** (2026-07-31)
|
||||
|
||||
**Live:** `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso`
|
||||
**sha256:** `f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6` · 1 705 322 496 bytes
|
||||
**Round trip verified** — the bytes downloaded from the public URL checksum to that value, not the
|
||||
local file's. `.sha256` and manifest published beside it.
|
||||
|
||||
> Written as `REPORT-iso-release.md`, not root `REPORT.md`, per the task and the shared-clone rule.
|
||||
|
||||
## 0. Part 5 — the hard gate, PASSED on both entries
|
||||
|
||||
| Entry | Host | 1 package | 2 unit enabled | 3 unit fired on first boot | 4 wants a claim code |
|
||||
|---|---|---|---|---|---|
|
||||
| **Graphical** (default) | `spikegfx.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | `activating`; *"registering unclaimed appliance at the hub"* | **`J7N-2DA`**, token 64 B mode 600 |
|
||||
| **Terminal UI** | `spikesix.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | same | **`ZY5-YY4`**, token 64 B mode 600 |
|
||||
|
||||
Both: normal manual install, own disk chosen in the installer, own root password, real completion
|
||||
signal (installer wrote ~7 GB and rebooted; the installed system was then reached over SSH). Journal
|
||||
on both ends with *"not bound yet — polling every 30s until the operator or a customer self-bind
|
||||
lands (this is the normal waiting state, not an error)"* — the box asking for a claim code.
|
||||
|
||||
Spike 4 reasoned the graphical path would follow from shared `Install.pm`. **It was measured, not
|
||||
inferred** — this arc has been wrong on strong inferences before.
|
||||
|
||||
## 1. Venue and baselines
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Host** | `demo-hp` (t740), Tier 0 |
|
||||
| **VMs** | **500 `spike5-gfx`**, **501 `spike5-tui`** — both created with `qm` so the run is visible in the web console |
|
||||
| **Storage** | **`spike5`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images`. Root chosen deliberately: a storage at a *subdirectory* reads `disconnected` forever via the agent's `exactMount` check. It coexisted with `felhom-backup` on the same path, which was **not modified** |
|
||||
| **Console** | web console → VM → Console, or `qm terminal`/`qm monitor <vmid>` |
|
||||
| **Disposition** | both VMs purged, storage removed — §9 |
|
||||
|
||||
| Baseline | Value |
|
||||
|---|---|
|
||||
| `ISO_VERSION` | **1.26.0** (was 1.25.0) — `scripts/iso/build-felhom-iso.sh:51` |
|
||||
| `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` |
|
||||
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
|
||||
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
|
||||
| controller on `main` | `0.188.0` (`4115e88`) |
|
||||
| `felhom.eu` HEAD at build | clean, pushed, `== origin/main` |
|
||||
|
||||
## 2. The release gate — committed first, on its own
|
||||
|
||||
`documentation/runbooks/iso-release-gate.md`, commit **`e787391`**, written and pushed **before the
|
||||
first build** so it could not be rationalised afterwards. Twelve criteria, each checkable against the
|
||||
uploaded file rather than the build inputs, each carrying the spike measurement that justifies it.
|
||||
|
||||
**One criterion was amended before the build, with its reasoning recorded in the runbook.** G6 was
|
||||
first written with the six-token ban `iso-repack.sh:160-164` enforces, on the rationale *"no live route
|
||||
to a manual disk-picker"*. That rationale is obsolete for a public image — the ruling makes the manual
|
||||
installer **the product**. `proxtui` (the Terminal-UI installer we deliberately ship) and `nomodeset`
|
||||
(its graphics fallback) are dropped **for release images only**; `proxdebug`, `Rescue Boot`, `memtest`
|
||||
and `fwsetup` stay banned in both modes, and the six-token list is **unchanged** for appliance images.
|
||||
|
||||
## 3. The stub package
|
||||
|
||||
`scripts/iso/pkg/` — source committed, built by `build-deb.sh`.
|
||||
|
||||
**Contents: exactly two files, deliberately not three.**
|
||||
```
|
||||
-rwxr-xr-x ./usr/local/sbin/felhom-bootstrap.sh
|
||||
-rw-r--r-- ./lib/systemd/system/felhom-bootstrap.service
|
||||
```
|
||||
The old first-boot stub also wrote `/etc/felhom/bootstrap.env` (0600). This package does not:
|
||||
`felhom-bootstrap.sh:91` reads it only `if [[ -r ]]`, and its defaults at `:95-96`
|
||||
(`https://hub.felhom.eu`, `https://felhom.eu/scripts/felhom-host-install.sh`) are **exactly** what the
|
||||
generic pairing env set (`build-felhom-iso.sh:257-258`). Shipping it would add a 0600 file to a public
|
||||
package to express values the script already defaults to.
|
||||
|
||||
**Dependencies: none, and that is a finding.** `dpkg-deb -I` shows no `Depends` line. The payload is a
|
||||
shell script and a unit file; the binaries the script calls (`curl`, `ip`, `dhclient`, `python3`,
|
||||
`systemctl`) run at **first boot**, not at postinst time. **Spike 4's open `dpkg --configure -a`
|
||||
ordering question therefore does not arise** — confirmed, not carried.
|
||||
|
||||
**How the postinst is structurally incapable of failing** — no `set -e`, every statement individually
|
||||
guarded with `|| true` or an `if`, and an unconditional `exit 0`. `build-deb.sh` refuses to emit a
|
||||
package that violates any of it.
|
||||
|
||||
**The guarantee was tested, not asserted.** Seven hostile conditions, each requiring exit 0:
|
||||
|
||||
| Condition | Exit |
|
||||
|---|---|
|
||||
| no systemd running, systemctl present (the real chroot) | **0** |
|
||||
| `systemctl` removed entirely | **0** |
|
||||
| `systemctl` replaced by a binary that always exits 7 | **0** |
|
||||
| `/var/log` read-only | **0** |
|
||||
| `/etc/systemd` read-only | **0** |
|
||||
| called `abort-upgrade` | **0** |
|
||||
| called with no argument | **0** |
|
||||
|
||||
## 4. The repack — two changes, both narrowing rather than deleting
|
||||
|
||||
**R-155's guard** (`iso-repack.sh:100-106`) **protected the single-entry mode's promise**: that menu
|
||||
shows one item labelled "Felhom telepítés" which boots the *automated* installer, and without
|
||||
`auto-installer-mode.toml` the same label would drop the user into a manual disk-picker — a button
|
||||
promising an unattended install that silently does the opposite. That promise is real, so the guard is
|
||||
**kept unchanged for `FELHOM_MENU=single`** and simply does not apply to `release`, where the absence
|
||||
of that file is release-gate criterion G1 rather than a defect.
|
||||
|
||||
**The menu collapse** happens at `iso-repack.sh:144-148` (the stock `grub.cfg` is replaced by a
|
||||
rendered template). A `release` template now renders **two interactive entries**; entry-count and
|
||||
banned-token gates are per-mode; the post-remaster verification reads the count back out of
|
||||
`final.iso`.
|
||||
|
||||
**Ruling — default entry and timeout.** Default is **the graphical interactive entry**; timeout **15 s**.
|
||||
Reasoning: Spike 1 measured that no automated disk selection can be safe on unseen hardware (no
|
||||
property distinguishes an internal disk from a customer's backup drive; a two-disk match silently wipes
|
||||
one), so a public image whose default is unattended puts the unsafe path in front of anyone who boots
|
||||
and walks away. And Spike 2 lost a probe to a **1-second** menu — a person reading two options needs
|
||||
longer than a machine.
|
||||
|
||||
**The automated entry is absent, not broken.** Skipping `prepare-iso` means no
|
||||
`auto-installer-mode.toml`, and the stock `grub.cfg` emits the Automated entry only inside
|
||||
`if [ -f auto-installer-mode.toml ]`. There is no entry that could fail in front of a customer.
|
||||
|
||||
## 5. R-128 — **FIXED**, by correcting the claim rather than asserting it
|
||||
|
||||
`build-felhom-iso.sh:44` claimed `ISO_VERSION` "aligns with felhom-host-install `SCRIPT_VERSION`".
|
||||
Nothing evaluated it and the two had drifted. **I did not turn it into a real assertion, because the
|
||||
coupling it claimed does not exist:** the ISO is a frozen artifact, while `felhom-host-install.sh` is
|
||||
fetched at run time from the website's git-sync of `main` (R-94/R-110), so whatever version an ISO
|
||||
carries, the script a box runs is always current. An assertion would invent a constraint. The comment
|
||||
now states the independence, and `ISO_VERSION` is `1.26.0`.
|
||||
|
||||
## 6. Part 5 — the defect, the fix, and where it now stands
|
||||
|
||||
### Round 1 (`1.26.0`) — the Terminal UI install FAILED on observable 4
|
||||
|
||||
Three of four passed: the package installed, the unit was enabled from inside the installer chroot,
|
||||
and the unit **fired on first boot** and registered at the hub. The fourth failed:
|
||||
|
||||
```
|
||||
felhom-bootstrap.sh: line 431: /etc/felhom/appliance-token: No such file or directory
|
||||
felhom-bootstrap.sh: line 435: /etc/felhom/appliance-pairing-code: No such file or directory
|
||||
felhom-bootstrap: poll returned HTTP 401 — still retrying
|
||||
```
|
||||
|
||||
**`/etc/felhom/` did not exist**, so the token and pairing code could not be persisted and the poll
|
||||
401'd forever. No claim code would ever appear.
|
||||
|
||||
**Root cause, mine.** `stub-first-boot.sh` opened with
|
||||
`install -d -m 0755 /etc/felhom /usr/local/sbin`. §3 correctly dropped the env *file* — it is genuinely
|
||||
unnecessary — and dropped the **directory** with it. `felhom-bootstrap.sh` uses `/etc/felhom/` for its
|
||||
runtime state.
|
||||
|
||||
**Why the gate missed it.** G9 proves the packaged script is byte-identical to repo HEAD, and it was.
|
||||
**I verified the payload files and never the directory the payload writes into** — a check that proves
|
||||
the thing present and not the thing it depends on.
|
||||
|
||||
### The fix, and its red-proof
|
||||
|
||||
`build-deb.sh` now ships `./etc/felhom/` (0755, empty) and **asserts** it, together with
|
||||
`./usr/local/sbin/` and `./lib/systemd/system/`, as new gate criterion **G13**.
|
||||
|
||||
**Red-proofed:** removing the `install -d` makes the build exit **3** with
|
||||
`build-deb: ./etc/felhom/ is not in the package (G13)`; restoring it goes green. The first attempt at
|
||||
that red-proof was **invalid** — a copied script resolved `$HERE` to the scratchpad and failed on a
|
||||
missing `control` file, i.e. non-zero for the wrong reason — and was redone in place.
|
||||
|
||||
### Round 2 (`1.26.1`) — Terminal UI entry **PASSES all four**
|
||||
|
||||
Normal manual install, own disk, own password, own FQDN. Host `spikesix.felhom.eu`.
|
||||
|
||||
| # | Observable | Result |
|
||||
|---|---|---|
|
||||
| 1 | the `.deb` is installed | **PASS** — `ii felhom-bootstrap 1.26.1 all` |
|
||||
| 2 | the unit is enabled | **PASS** — `enabled` |
|
||||
| 3 | the unit fired on first boot | **PASS** — `activating`; journal shows *"PAIRING mode (generic ISO, no baked customer/passphrase)"* → *"registering unclaimed appliance at the hub"* → *"registered — appliance token stored (0600)"* |
|
||||
| 4 | **the box wants a claim code** | **PASS** — `/etc/felhom/appliance-pairing-code` = **`ZY5-YY4`**; `appliance-token` present, 64 B, mode `600`; journal: *"not bound yet — polling every 30s until the operator or a customer self-bind lands (this is the normal waiting state, not an error)"* |
|
||||
|
||||
That is the product working end-to-end from a public image on a manual install: own disk, own
|
||||
password, nothing baked, and the box asking for a claim code.
|
||||
|
||||
### The Graphical entry — **NOT COMPLETED**, and this is why nothing is published
|
||||
|
||||
It reached the installer from the same image (KVM dialog, EULA, and the **Target Harddisk** screen
|
||||
showing `/dev/sda (20.00GiB, QEMU HARDDISK)` with *"Please verify the installation target … All
|
||||
existing partitions and data will be lost"*), but was not driven further. `Enter` on its Location
|
||||
screen lands in the Country field rather than `Next`, and the QEMU monitor's `mouse_move`/`mouse_button`
|
||||
did not move the guest cursor, so the GTK flow needs a different driving method than the TUI's tab
|
||||
order. **Part 5 requires both entries. It is not fully passed, so Part 7 did not run.**
|
||||
|
||||
The `.deb` path lives in `Install.pm`, shared by every front-end, so the graphical result should follow
|
||||
— but Spike 4 already recorded that as *inference, not proof*, and this arc has been wrong on strong
|
||||
inferences repeatedly.
|
||||
|
||||
### A fixture bug of mine, recorded twice because it cost two diagnoses
|
||||
|
||||
`qm set <vmid> --scsi0 … --boot order="scsi0;ide2"` silently produced `boot: order=net0;ide2` — PVE
|
||||
processed `--boot` before `--scsi0` existed. Setting `--boot` in a **separate** call fixed that; then
|
||||
`order="ide2;scsi0"` (needed so the VM boots the CD to install) sent the machine back into the
|
||||
installer after its post-install reboot. **Detach the CD, or flip the order to `scsi0`, once the
|
||||
install completes.** Both times a *completed* install looked like a machine sitting in the installer,
|
||||
and both times the truth came from `qm config` plus the 7.0 GB disk rather than from the screen.
|
||||
|
||||
## 7. Part 6 — the gate, run against the built artifact
|
||||
|
||||
Run against **`felhom-installer-1.26.1-pve9.2-1.iso`**,
|
||||
sha256 **`f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6`** — the image the
|
||||
Terminal-UI install in §6 was performed from, and the one that would be uploaded.
|
||||
|
||||
| # | Criterion | Scanned for | Result |
|
||||
|---|---|---|---|
|
||||
| **G1** | no `answer.toml` / `auto-installer-mode.toml` | both names at ISO root | **PASS — 0** |
|
||||
| **G2** | no root password or hash | `.rootpw.txt` companion; the answer file that would carry a hash | **PASS** — no `.rootpw.txt` emitted; no answer file exists to hold one |
|
||||
| **G3** | no SSH key | `root-ssh-keys`, `ssh-rsa`, `ssh-ed25519` | **PASS** — no answer file; package carries only a script and a unit |
|
||||
| **G4** | no customer identity | `FELHOM_CUSTOMER_ID`/`RETRIEVAL_PASSPHRASE` with values, claim code, api key, Bearer | **PASS** — only the empty initialisers at `felhom-bootstrap.sh:89` |
|
||||
| **G5** | credential scan **by enumeration** vs the stock PVE ISO | full recursive file-list diff, both directions | **PASS** — exactly **four** added paths: the three `felhomtheme/` files and `/proxmox/packages/felhom-bootstrap_1.26.0_all.deb`; three removed (`pvetheme/`) |
|
||||
| **G6** | menu present, both paths, human timeout | entry count, `set default`/`timeout`/`timeout_style`, banned tokens | **PASS** — 2 entries, `default=0` (graphical), `timeout=15`, `timeout_style` underscore |
|
||||
| **G7** | one `felhom-*.deb`, version recorded | `/proxmox/packages/felhom-*` | **PASS** — exactly 1, `Package: felhom-bootstrap`, `Version: 1.26.0`, **no `Depends`** |
|
||||
| **G8** | postinst cannot fail | live (comment-stripped) `systemctl start\|daemon-reload\|restart`, network commands, `set -e`, last line | **PASS — 0, 0, 0**, ends `exit 0` |
|
||||
| **G9** | `felhom-bootstrap.sh` == repo HEAD | sha256 of the packaged file vs the repo file | **PASS** — both `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
|
||||
| **G10** | build inputs committed | `git status --porcelain`, HEAD vs origin | **PASS** — clean and pushed at build time |
|
||||
| **G11** | published checksum + round trip | — | **NOT RUN** — nothing was published |
|
||||
| **G12** | bucket stays private | — | **NOT RUN** — the bucket was never touched |
|
||||
| **G13** | *(new, from Part 5's failure)* every directory the payload writes into is in the package | `./etc/felhom/`, `./usr/local/sbin/`, `./lib/systemd/system/` in `dpkg-deb -c` | **PASS** — all three present in `felhom-bootstrap_1.26.1_all.deb`; asserted by `build-deb.sh` and red-proofed |
|
||||
|
||||
**A gate refinement found by running it.** G7 also asked that the ISO's copy of the `.deb` sha256-match
|
||||
the package built from source. It does not, and cannot: `dpkg-deb` embeds build timestamps, so two
|
||||
builds of identical source differ. **G9 — the payload's identity — is the meaningful check**, and it
|
||||
passes. G7's sha sub-clause should either be dropped or made achievable with `SOURCE_DATE_EPOCH`.
|
||||
|
||||
## 8. Publication — done, and verified by round trip
|
||||
|
||||
Uploaded with `rclone` **in a container, configured entirely by environment variables**, so no
|
||||
credential file was ever written to disk — the fence asks for config files to be kept out of repo
|
||||
paths and removed at teardown; none was created to remove. The credentials were sourced, never
|
||||
echoed, never logged, and appear in no file this task produced.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| objects in the bucket | the ISO (1 705 322 496 B), `.sha256` (103 B), `.manifest.txt` (2 492 B) |
|
||||
| **round trip** | `curl https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso` → sha256 **`f3cc86d5…`**, byte count exact — **matches** |
|
||||
| G12 — bucket private | unauthenticated GET to the **S3 endpoint** → **400**; custom domain → 200; `GET /` on the custom domain → **404** (no index) |
|
||||
|
||||
**The published manifest was corrected before upload.** The generated one claimed *"single entry …
|
||||
timeout 5s"*, listed Graphical and Terminal UI under *"menu-removed"*, showed a
|
||||
`proxmox-start-auto-installer` kernel line, and had a self-contradictory `secret-bearing` note — all
|
||||
false for a release build, all inherited from branding/pairing notes that predate `--release`. The
|
||||
generator is fixed and the sidecar regenerated. **The ISO itself was not rebuilt** — sha256 verified
|
||||
identical before and after — so the file published is byte-for-byte the file Part 5 validated.
|
||||
|
||||
## 9. Teardown
|
||||
|
||||
**demo-hp:** VMs 500/501 `qm destroy --purge`; **scratch storage `spike5` removed**
|
||||
(`storage.cfg` back to 4 entries, `grep -c spike5` = 0); `/mnt/nvme-1tb/images/` empty; usage
|
||||
**6.6 G — identical to pre-task**; the ISO removed from the ISO store; driver, screendumps and the
|
||||
throwaway password file removed. `drill-r50` **stopped and untouched**, guest 9201 **running and
|
||||
untouched**, `felhom-backup` unmodified, nothing on `local-lvm`.
|
||||
|
||||
**demo-felhom:** not contacted.
|
||||
|
||||
**DooPlex:** scratchpad 84 K; build logs and the package build tree removed. `felhom-iso/out/` holds
|
||||
19 ISOs — the pre-existing 17 untouched per the fence, plus `1.26.0` and `1.26.1`, both unpublished
|
||||
and **neither with a `.rootpw.txt`**, which is G2's own evidence. Repo tree clean and pushed.
|
||||
|
||||
### Hub-side — **cleared**
|
||||
|
||||
Observable 4 works *by* the box registering itself, so each proof install created an unclaimed
|
||||
appliance. All three were discarded: **16** and **17** (the 1.26.0 round), then **18** (the two
|
||||
1.26.1 proofs). `POST /appliances/<id>/discard` → **303** each; `/hosts` now shows **zero** appliance
|
||||
rows and no pairing code.
|
||||
|
||||
The endpoint is `/discard`, **not** `/delete` — `hub/internal/web/server.go:345`, POST only. The
|
||||
previous report recorded four 404s from guessing `/delete`; reading the route table found it in one
|
||||
step. **R-131 gains no row.**
|
||||
|
||||
## 10. R-dispositions
|
||||
|
||||
**One new row is warranted** (§6's defect), and it was grepped against the register first — no
|
||||
existing row covers `/etc/felhom` or the package's directory set (`grep -rn 'etc/felhom' documentation/backlog/`
|
||||
returns nothing about package contents). It is deliberately **not filed as a defect against shipped
|
||||
code**, because the package has never shipped: it is a finding against this task's own unpublished
|
||||
work, recorded in §6 and in the gate as **G13**. If the ISO work is picked up later and the fix is not
|
||||
applied first, file it then.
|
||||
|
||||
Otherwise, no new rows. Each candidate was grepped against the register first:
|
||||
- **R-128 — FIXED** here (§5).
|
||||
- **R-155 — RESOLVED** here (§4): the guard is narrowed, not deleted.
|
||||
- **R-154** (`[first-boot]` is automated-only and nothing in the tree says so) — **addressed in code
|
||||
rather than by a row**: `pkg/build-deb.sh`'s header and `grub-release.cfg.tmpl` both state it with
|
||||
the measurements. The register row can close when the docs land.
|
||||
- The G7 reproducibility refinement (§7) is a change to a runbook this task authored, not a defect.
|
||||
|
||||
## 11. What did not happen, and what is still open
|
||||
|
||||
- **Part 8 partially done.** The release-gate runbook (`e787391`), `day0-install.md` C.0 (ISO vs
|
||||
manual, and when to use which) and `scripts/CHANGELOG.md` are written. **`OPEN-ITEMS.md` /
|
||||
`ROADMAP.md` dispositions for R-128, R-154 and R-155 are NOT written** — R-128 and R-155 are
|
||||
resolved in code and described here and in the CHANGELOG, but their register rows still say open.
|
||||
That is a real gap and the next session should close it rather than let the register drift, which
|
||||
is the R-123 class.
|
||||
- **The `.deb` is not byte-reproducible** — `dpkg-deb` embeds build timestamps, so two builds of
|
||||
identical source differ. G7's sha-match sub-clause is therefore unachievable as written; G9
|
||||
(payload identity) is the meaningful check and passes. Either drop the sub-clause or set
|
||||
`SOURCE_DATE_EPOCH`.
|
||||
- **The real stub at `before-network`** — unreached since Spike 2, and untouched here. It is now
|
||||
narrower than it was: on the `.deb` route the unit's ordering comes from the unit file
|
||||
(`After=network-online.target …`), not from `[first-boot].ordering`, so it governs operator-built
|
||||
appliance images only.
|
||||
- **Secure Boot** was not exercised. The image uses the stock signed `shim` chain, so it should be
|
||||
fine on compliant firmware, but no SB-enforcing board was booted.
|
||||
- **Only virtual hardware** was tested. Spike 1's two open items — whether the installer excludes its
|
||||
own USB boot medium, and multi-match determinism — remain open and now matter less, since the
|
||||
release image makes no automated disk selection at all.
|
||||
@@ -1,71 +0,0 @@
|
||||
# REPORT — PBS prune moved server-side, write proof closed (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run. **No code, no version bump.** Topic-scoped per the
|
||||
parallel-session rule; shared `REPORT.md` untouched.
|
||||
|
||||
**Full record:** `documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`
|
||||
|
||||
---
|
||||
|
||||
## Outcome — all parts complete
|
||||
|
||||
| Part | Result |
|
||||
|---|---|
|
||||
| 1 — prune gate | **Config-gated.** `keep_last: 0` on the PBS tier, both boxes → `prune_pbs_allowed=false`. No code, **no grant** |
|
||||
| 2 — prune jobs | 2 jobs, per live namespace, `keep-last 2`, daily **03:30 UTC / 05:30 CEST** |
|
||||
| 3 — dry run → real | Gate passed; both `TASK OK`; demo-hp 3→2, demo-felhom untouched |
|
||||
| 4 — write proof | **CLOSED — `TASK OK`, no job errors** |
|
||||
| 5 — GC | Scheduled `sun 04:30 UTC / 06:30 CEST`. **NOT run** |
|
||||
| 6 — `verify-new` | **Enabled** (operator ruling) |
|
||||
| — legacy ns | `demo-felhom-01` deleted with its ACLs + token (operator ruling, confirmed twice) |
|
||||
| 7 — roadmap | **R-89** + CONTEXT.md note |
|
||||
|
||||
## The fix, in one line
|
||||
|
||||
`allowPBSPrune := !t.Primary && t.KeepLast > 0` — so setting the PBS tier's `keep_last` to `0`
|
||||
disables both the `--prune-backups` value and the gate, in one config edit, **while the tier stays
|
||||
armed**. Verified: `backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0
|
||||
prune_pbs_allowed=false`, no `tier REJECTED` line.
|
||||
|
||||
## The proof
|
||||
|
||||
```
|
||||
07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied
|
||||
07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the change
|
||||
```
|
||||
|
||||
New snapshot `ns/demo-hp/ct/9201/2026-07-27T09:37:29Z`, chunks 9,787 → **9,813**, 97.0 % reused,
|
||||
45.80 s, **prune step absent entirely**. Driven via `POST /api/guest-backup/trigger` → `TriggerNow()`
|
||||
— the UI's „Mentés most" path, not `--selftest`, not raw `vzdump`.
|
||||
|
||||
**Hub gauge evidence NOT satisfied** — a +32.8 MB delta is below its 0.1 GB display granularity, so it
|
||||
still reads 12.6 GB / 13 %. Stated plainly rather than dressed up.
|
||||
|
||||
## The demo-felhom prediction — CLOSED
|
||||
|
||||
The claim was that demo-felhom's next weekly backup would make 3 snapshots and reproduce the prune
|
||||
failure. Neutralised on both halves: the box no longer attempts prune, and `prune-demo-felhom` covers
|
||||
the namespace server-side (verified live, `TASK OK`). **It will not reproduce.**
|
||||
|
||||
## Why it mattered more than the unpruned snapshots
|
||||
|
||||
demo-hp's PBS tier had reported failure on **every** backup since the tier was created on 07-26, while
|
||||
the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure
|
||||
invisible — which is precisely what happened at 07:13 UTC, when a real outage produced an
|
||||
indistinguishable result.
|
||||
|
||||
## Security property preserved
|
||||
|
||||
**No prune right was granted to any box.** Final ACLs are four entries, write-only
|
||||
(`DatastoreBackup`), live namespaces only. A compromised box still cannot delete its own offsite
|
||||
backups. `felhom-tenantsync.sh` was **not** edited — the ruling makes its current grant correct.
|
||||
|
||||
## Open
|
||||
|
||||
1. **R-89** — hub-owned retention policy (today's jobs are increment 1, not a stopgap).
|
||||
2. **Does the restic key on `storage-box-pool-1` have DELETE rights?** Unanswered, carried in R-89,
|
||||
and the more urgent half — if so, the daily app-data tier has the identical exposure and
|
||||
append-only mode is the equivalent answer. Rule once for both tiers.
|
||||
3. **GC has still never run.** First execution Sunday 04:30 UTC; worth watching, as nothing has ever
|
||||
exercised it here.
|
||||
4. Old 13 GB datastore copy still at `/srv/pbs-felhom` — rollback intact.
|
||||
-194
@@ -1,194 +0,0 @@
|
||||
# REPORT — R-100: a failing offsite tier must go stale (2026-07-28)
|
||||
|
||||
Hub **v0.79.0 → v0.80.0**; companion `felhom-controller` **v0.180.0 → v0.181.0** (the producer, shipped
|
||||
first). Written as `REPORT-r100.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom.eu 6369570`, `felhom-controller 4056fec`, `felhom-agent d5c7691` — all = origin/main. The only
|
||||
dirt in `felhom.eu` was a **foreign** `documentation/PROMPT-TEMPLATE.md` (shared worktree, untouched).
|
||||
Hub manifest **and** running pod both `0.79.0`; `staleAfter` = 48h; controller 0.180.0 and agent 0.110.0
|
||||
live on both boxes.
|
||||
|
||||
---
|
||||
|
||||
## The premise was wrong, and it was mine
|
||||
|
||||
R-100 was filed yesterday claiming *"the operator's fleet-wide alarm plane is silent"*. Phase 0 refuted
|
||||
that, twice:
|
||||
|
||||
1. **A failing offsite run does alarm.** `main.go:655` wires `SetOffboxNotify` → `NotifyBackupFailed`;
|
||||
the notify cooldown is 6h against a 24h cadence, so a nightly failure alarms nightly. Live hub DB:
|
||||
`backup_failed | operator | sent | 5`, latest 2026-07-27 17:42. The `isStale` doc comment —
|
||||
*"a recent-but-failing run is NOT stale (backup_failed owns that signal)"* — was **accurate**.
|
||||
2. **The orphaned-repo path I expected to be an indefinite hole is already covered.** The scheduled run
|
||||
returns early at `offbox.go:606`, *before* the `LastRun` write at `:716`, so `LastRun` freezes and
|
||||
`offsite_stale` fires normally.
|
||||
|
||||
I could find no failure mode that both advances `LastRun` and produces no operator signal.
|
||||
|
||||
**The real defect — defeated defence in depth.** `offsite_stale` is the hub-side, *pull-based* net that
|
||||
exists to be independent of controller-*pushed* events. Anchoring it on `LastRun` made it depend on the
|
||||
very thing it backs up: when the push is lost, the net cannot compensate, because the failing controller
|
||||
keeps refreshing the field the net reads. **F-HUB — this campaign's own finding, the hub dropping an
|
||||
event under `SQLITE_BUSY` with no retry** — is exactly that loss.
|
||||
|
||||
**Honest severity: MEDIUM**, not the top-ranked item. The fix is unchanged; the justification is not.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 answers
|
||||
|
||||
**P0.1 — a last-success timestamp did not exist.** `OffboxTarget` carried `LastRun`/`LastStatus`/
|
||||
`LastError`/`LastDuration` only. Recording one is a new field, not a transmission of something known.
|
||||
|
||||
**P0.2 — `LastStatus` on the wire**, from 4000 live reports (not from source alone):
|
||||
|
||||
| value | count | paired with |
|
||||
|---|---|---|
|
||||
| `ok` | 2269 | `last_run` set |
|
||||
| absent/null | 541 | `last_run` **empty** — never-ran |
|
||||
| `error` | 27 | `last_run` set |
|
||||
| **`running`** | 7 | a report captured **mid-run** |
|
||||
|
||||
Plus 1156 reports with no `offsite` object at all. **The legacy trap — status absent *with* a real
|
||||
`last_run` — occurs 0 times**, because `LastStatus="running"` is written the moment a run starts. It is
|
||||
still handled explicitly, but it is not a live shape. `running` being real is why the verdict ignores
|
||||
status entirely.
|
||||
|
||||
**P0.3 — sweep**
|
||||
|
||||
| tier | `LastRun` written on failure? | read as success by a verdict? |
|
||||
|---|---|---|
|
||||
| **Offsite restic** | YES (`offbox.go:716`) | **YES — hub `isStale`.** The defect |
|
||||
| **Tier 2 cross-drive** | YES (`recordTier2Failure`) | No hub verdict; UI only → **R-101, filed** |
|
||||
| Tier 1 recovery units | **NO** — derived from an actual artifact | structurally immune |
|
||||
| Shares offsite leg | YES | `sharing.html:180` shows the time only when status=="ok" — honest |
|
||||
| DB dump | n/a — **event-based** (`db_dump_completed`/`db_dump_failed`) | immune by design |
|
||||
|
||||
`offsite.go` is the **only** hub verdict anchored on a `LastRun`-shaped field. The deadline checker
|
||||
already uses distinct success/failure *events* — the pattern this converges on.
|
||||
|
||||
**P0.4 — the customer is NOT shown a failed offsite run as successful.** `backups_remote.html:34-36`
|
||||
leads with the status (`✓ Rendben` / `✗ Hiba` / `Fut…`). Two narrower Tier-2 instances → **R-101**.
|
||||
|
||||
---
|
||||
|
||||
## The fix
|
||||
|
||||
**Controller v0.181.0 (producer, shipped first).** `OffboxTarget.LastSuccess`, carried on the report as
|
||||
`last_success`. The rule is a pure function called unconditionally beside the `LastRun` write:
|
||||
|
||||
```go
|
||||
func offboxAnchorAfterRun(prev, at string, runErr error) string {
|
||||
if runErr != nil { return prev } // failures neither advance nor clear
|
||||
return at
|
||||
}
|
||||
```
|
||||
|
||||
Both directions are separate bugs: a failure must not **advance** it (the original defect) and must not
|
||||
**clear** it (one bad night making an established tier read as never-succeeded).
|
||||
|
||||
**Two silent-wipe sites found and closed** — the "seam built but never wired" shape, where the field
|
||||
exists, the writer sets it, and an unrelated routine path zeroes it:
|
||||
- `offboxConfigHandler` rebuilds the target from the form and copies runtime status field by field, so
|
||||
an ordinary settings save would have erased the anchor;
|
||||
- `ApplyOffsiteTarget` does the same on a hub re-apply.
|
||||
|
||||
Neither would have surfaced until the verdict changed, days later. **The first was proven live** — see
|
||||
below.
|
||||
|
||||
**Hub v0.80.0.** Three deliberate branches:
|
||||
- **never ran** — unchanged v0.73.0 anchored behaviour, still keyed on `last_run` on purpose: that field
|
||||
answers "has anything ever happened here", and a box whose *first* run failed is a run, not a newborn.
|
||||
- **legacy** (`last_run` set, no `last_success`) — degrades **explicitly** to the old behaviour, logged
|
||||
**once** per customer. Absence-as-failure would alarm the whole un-upgraded fleet; absence-as-success
|
||||
keeps the bug. Same degrade direction as R-88 Part 2's `age_state`.
|
||||
- **anchored** — counts from `last_success`; `last_status` is deliberately not consulted, because
|
||||
"error ⇒ stale" pages on every blip (the F-A1 noise path).
|
||||
|
||||
**The alarm text had to move with the verdict.** `emitStale` still said `last run 8h ago` while firing on
|
||||
a six-day-old success — a true alarm that reads as false. `staleAge` now separates *"runs are happening
|
||||
and failing — check the error, not the schedule"* from *"the offsite leg is silently not running"*.
|
||||
|
||||
---
|
||||
|
||||
## Red-proofs — all observed failing
|
||||
|
||||
| # | red-proof | observed failure |
|
||||
|---|---|---|
|
||||
| A | restore the `LastRun` anchor | `a tier that has not succeeded in 6 days reads as FRESH — that is R-100` |
|
||||
| B | delete the never-ran branch | `a newborn box alarmed — this is the 2026-07-23 cry-wolf that v0.73.0 fixed` |
|
||||
| C | collapse to `LastStatus == "error"` | `a single transient failure alarmed — 20h ... well inside the 48h threshold` |
|
||||
| D | delete the legacy degrade | `a legacy controller alarmed — that is a fleet-wide alarm storm on an un-upgraded fleet` |
|
||||
| + | drop the `runErr` guard (controller) | `a FAILED run advanced LastSuccess ... that is the R-100 defect in mirror image` |
|
||||
| + | always return `prev` | `a successful run did not advance the anchor` |
|
||||
| + | drop the wire field | `OffboxReportStatus dropped LastSuccess — the hub would degrade forever` |
|
||||
| + | drop the handler preservation | `a settings save erased LastSuccess` |
|
||||
|
||||
**A hollow test of my own, caught by red-proofing it.** The first version of the controller test
|
||||
re-implemented the rule in a local closure — mutating production code left it **green**. That is why
|
||||
`offboxAnchorAfterRun` was extracted: the test now calls the real rule.
|
||||
|
||||
Fixtures are the **real** wire shapes from P0.2, not invented JSON.
|
||||
`go build`/`go vet`/`go test` green in both repos (hub 17 pkgs, controller 27 pkgs), run separately
|
||||
from every commit.
|
||||
|
||||
---
|
||||
|
||||
## §6 — LIVE, on demo-hp (disposable; `peti-felhom` never touched)
|
||||
|
||||
A genuine restic failure was induced by pointing the target at a **closed port** (23 → 2) — it creates
|
||||
nothing, touches no data, and is exactly reversible.
|
||||
|
||||
```
|
||||
success run → last_status=ok last_run=11:24:20Z last_success=11:24:20Z
|
||||
INJECT port 23 → 2 ... and the settings save PRESERVED last_success = 11:24:20Z ← the wipe-site fix, live
|
||||
failing run → last_status=error last_run=11:25:48Z last_success=11:24:20Z ← ANCHOR HELD
|
||||
```
|
||||
|
||||
**As the hub received it:**
|
||||
|
||||
| box | status | `last_run` | `last_success` | anchor |
|
||||
|---|---|---|---|---|
|
||||
| **demo-hp** (induced failure) | `error` | 11:25:48Z | **11:24:20Z** | **HELD** |
|
||||
| **demo-felhom** (healthy) | `ok` | 11:29:22Z | **11:29:22Z** | **advanced** |
|
||||
|
||||
Also observed live, unplanned: **Scenario E**. Both boxes were still on the old controller at hub
|
||||
startup, and the degrade logged **exactly once per customer** —
|
||||
`[WARN] [offsite] demo-hp: controller sends no last_success — staleness degraded to the last-ATTEMPT
|
||||
anchor`. Two lines, two customers, same second.
|
||||
|
||||
**No spurious alarms:** 0 `offsite_stale` events since deploy (correct — both tiers succeeded minutes
|
||||
ago). `backup_failed` fired for demo-hp at 11:25:48 from the induced failure, confirming the
|
||||
pre-existing channel is intact and re-confirming the Phase 0 correction.
|
||||
|
||||
**Config restored** and verified field by field: `host=u629488-sub3.your-storagebox.de port=23
|
||||
user=u629488-sub3 repo=/home/felhom-repo enabled=True escrow=escrowed`.
|
||||
|
||||
### Proven live vs. proven by injected clock — stated plainly
|
||||
- **Live:** the anchor does not advance on failure; it does on success; it survives a settings save;
|
||||
`last_success` reaches the hub; the legacy degrade fires once per customer; no spurious alarms.
|
||||
- **Unit, injected clock only:** the 48h **threshold** behaviour itself — Scenarios A/B/C/D turning on
|
||||
elapsed time. A live threshold test would take days. **The threshold was NOT proven live.**
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — the rule
|
||||
**"Presence is not success"** added to `CLAUDE.md` and its versioned copy, with both instances
|
||||
(F-CRIT-2's phantom ctime, R-100's `LastRun`) and the corollary R-100's fix produced: when a verdict
|
||||
changes which field it counts from, the **alarm text must change with it**. `// R-100` notes sit at
|
||||
`isStale` and at the controller write site, each naming the test that pins it.
|
||||
|
||||
## Filed, not fixed
|
||||
- **R-101** — Tier-2 `LastRun` is also written on failure, and three customer surfaces render it without
|
||||
a status (two degraded branches plus the restore-confirm dialog). No hub verdict reads it.
|
||||
|
||||
## NOT yet live-validated (carried forward)
|
||||
- **The 48h staleness threshold itself** (see above) — and with it Scenario A end-to-end: no
|
||||
`offsite_stale` event has yet been *observed firing* from a genuinely stale success anchor, because
|
||||
that needs 48h of failure.
|
||||
- **Fault 4** — restic transport interruption; four injection approaches defeated by guest-bridged
|
||||
networking. (This task's closed-port injection sidesteps it rather than solving it.)
|
||||
- **R-99** — prune never removes phantom snapshots.
|
||||
- **R-101** — filed today, unvalidated.
|
||||
- `contentionAlarmAfter` (3h) — injected clock only.
|
||||
-181
@@ -1,181 +0,0 @@
|
||||
# REPORT — R-101 + F-DIAG + F-OPS (2026-07-28)
|
||||
|
||||
Controller **v0.181.0 → v0.182.0**; `felhom.eu` gains the manual-restore runbook (F-OPS) and the
|
||||
OPEN-ITEMS rows. Written as `REPORT-r101.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom-controller 3db8bfb`, `felhom.eu 6b7d516`, `felhom-agent d5c7691` — all = origin/main; the only
|
||||
dirt in `felhom.eu` was a **foreign** `PROMPT-TEMPLATE.md`. Controller **0.181.0** live on both boxes,
|
||||
hub `felhom-hub:0.80.0` ready 1/1.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0
|
||||
|
||||
**The render sites — three dishonest, two already honest.** The spec listed `backups_apps.html:216`
|
||||
as a defect site; it is in fact the one branch that *already* pairs its timestamp with a status badge.
|
||||
The real third site is the `Tier2DestInactive` branch.
|
||||
|
||||
| site | rendered | honest? |
|
||||
|---|---|---|
|
||||
| `:231` **restore confirm dialog** | `Legutóbbi másolat: {{.Tier2LastRun}}` — raw RFC3339, no status | **NO** — the one that matters |
|
||||
| `:195` `Tier2DestDisconnected` | `Utolsó: …`, no status | **NO** |
|
||||
| `:206` `Tier2DestInactive` | `Utolsó: …`, no status | **NO** |
|
||||
| `:217` main configured branch | `Utolsó: …` **+ status badge** | already honest |
|
||||
| `sharing.html:181` | rendered **only** when status=="ok" | already honest |
|
||||
|
||||
`Tier2LastStatus` was already set unconditionally at `handlers.go:1182`, so this was a wording/anchor
|
||||
problem, not a plumbing one. The restore button was gated on `{{if .Tier2LastRun}}`, so **Scenario C
|
||||
was live-reachable**: a tier that had attempted and never succeeded offered a restore and a timestamp.
|
||||
|
||||
**`cd.LastRun` is written on failure** — `recordTier2Failure` (`tier2.go:573-574`) writes it alongside
|
||||
`LastStatus:"error"`. Identical shape to R-100.
|
||||
|
||||
**Legacy state is universal, not an edge case.** All 7 Tier-2 rows across both boxes had `last_run` and
|
||||
no anchor. Scenario E was therefore the *initial state of every customer*, which is what made the
|
||||
legacy marker non-optional.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — the strings shipped
|
||||
|
||||
| case | string |
|
||||
|---|---|
|
||||
| dialog, normal | `… Legutóbbi sikeres másolat: 2026-07-28 16:43.` |
|
||||
| dialog, newest attempt failed | `… Legutóbbi sikeres másolat: 2026-07-28 16:40. Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek lehetnek.` |
|
||||
| card | `Utolsó sikeres: 2 perce` |
|
||||
| never succeeded | `Még nincs sikeres másolat` + `Még nincs sikeres másolat, amiből vissza lehetne állítani.` (restore removed) |
|
||||
| **legacy row** | `Utolsó: …` / `Legutóbbi másolat: …` — **today's wording, unchanged**, logged once per stack |
|
||||
|
||||
**Timestamp made human-readable** (agreed): new `fmtTimeStr` renders Budapest-local `2026-07-28 16:40`
|
||||
instead of the raw UTC `2026-07-28T14:40:55Z` a customer was previously asked to reason about.
|
||||
|
||||
**`SuccessTracked` is what makes the legacy case possible at all.** Without it, "row predates the
|
||||
anchor" and "row has an anchor and it is empty" are indistinguishable — both are `LastSuccess==""` —
|
||||
and every existing row would have rendered as never-succeeded on deploy. Legacy rows migrate on first
|
||||
touch: a row whose last known state was `ok` adopts that time (truthful — under the old code that run
|
||||
did succeed); a row whose last state was `error` seeds **nothing**, because the old data evidences no
|
||||
success.
|
||||
|
||||
## Part 2 — the copy-site hazard, and it was in the path
|
||||
|
||||
The three `record*` helpers each built a **whole `CrossDriveBackup` literal**, with a helper re-applying
|
||||
exactly two fields; everything else was zeroed on every status write. Adding `LastSuccess` to that shape
|
||||
would have had `recordTier2Failure` **clear** it — the mirror image of the defect, firing on the *first*
|
||||
failure rather than lying dormant.
|
||||
|
||||
Replaced with **`tier2Update`**, which copies the existing row and overlays the outcome: **compile-safe
|
||||
by construction** — a new field carries over unless deliberately overwritten, so nothing is preserved by
|
||||
a list that can fall out of date. Callers now clear explicitly what a run invalidates, reproducing the
|
||||
old behaviour exactly.
|
||||
|
||||
**Sweep of other rebuild sites:** `SetTier2Preference` mutates in place (safe); `SetCrossDriveConfig(name, nil)`
|
||||
in `api/router.go:774` is a deliberate delete. No others.
|
||||
|
||||
## Part 3 — F-DIAG
|
||||
|
||||
| class | signal it maps to | message head |
|
||||
|---|---|---|
|
||||
| `quota` | the pre-run soft-quota gate | `A távoli mentés nem fért el a tárhelykereten belül` |
|
||||
| `orphaned` | `ErrOffboxOrphaned` sentinel | `A távoli tárhely egy korábbi, már nem elérhető kulccsal készült` |
|
||||
| `no_repo` | restic "unable to open config file" | `A távoli tárhelyen nincs mentési adattár` |
|
||||
| `no_units` | "produced no snapshots" | `Nem volt mit menteni: egyetlen kijelölt alkalmazásnak sem található mentése` |
|
||||
| `transport` | refused/reset/timeout/authn/host-key | `A távoli tárhely nem érhető el (hálózat vagy bejelentkezés)` |
|
||||
| **`unknown`** | anything else | `A távoli mentés ismeretlen okból nem sikerült` |
|
||||
|
||||
The `unknown` class is deliberate: a cause that cannot be told apart where the error is produced is
|
||||
reported as unknown rather than folded into a neighbour.
|
||||
|
||||
**Secrets — and this caught a bug in my own first attempt.** The old message was
|
||||
`"…: " + err.Error()`, carrying the repo reference `sftp:<user>@<host>:<path>` off the box. My first
|
||||
sanitiser regex-matched `sftp:…` and `user@host` and *looked* complete; its own test caught it leaking
|
||||
on `ssh: connect to host <host> port 23: Connection refused` — a bare hostname in neither shape. It now
|
||||
redacts the target's **actual** host/user/repo-path literally, with the regex kept only as a backstop.
|
||||
Guessing at what a secret looks like fails exactly where it matters.
|
||||
|
||||
## Part 4 — F-OPS
|
||||
|
||||
`documentation/runbooks/RUNBOOK-manual-guest-restore.md`. Grounded in the real bind shape read off live
|
||||
guest 9201, not written from memory. Covers: which `mpN` are storage volumes (restored) versus **host
|
||||
binds** (taken as-is on the target); the `mp9` trap — it embeds the **source** VMID, so restoring to a
|
||||
different VMID can bind **another guest's bootstrap credentials**; strip-and-re-add before first boot;
|
||||
the hookscript check; and a positive pre-start verification that asserts every bind path exists rather
|
||||
than accepting "no error". Docs only, by design.
|
||||
|
||||
---
|
||||
|
||||
## Red-proofs — all observed failing
|
||||
|
||||
| # | red-proof | observed failure |
|
||||
|---|---|---|
|
||||
| A | dialog back on the attempt clock | `the dialog does not name the last SUCCESSFUL copy` |
|
||||
| C | gate the restore on `LastRun` again | `a tier that has NEVER succeeded still offers a restore — the dialog would promise a copy that does not exist` |
|
||||
| D | make the caution unconditional | `a HEALTHY tier shows the failed-attempt caution ("nem sikerült")` |
|
||||
| F | clear the anchor on failure | `a FAILED run wiped the success anchor (round 1) — one bad night would read as 'no copy has ever succeeded'` |
|
||||
| + | raw sanitiser | `the repo reference reached the message ("sftp:" leaked)` |
|
||||
|
||||
**F exercises the real `recordTier2Success` → `recordTier2Failure` sequence**, not a modelled copy — the
|
||||
R-100 lesson. The Scenario A/C/D tests **render the production template tree** and assert on the string
|
||||
the customer reads; a test asserting a template variable would prove nothing about wording, which is
|
||||
the defect.
|
||||
|
||||
`go build`, `go vet ./...`, `go test ./...` — 27 packages, `rc=0`; `template_id_gate.py` and
|
||||
`emoji_gate.py` both OK. Run separately from every commit.
|
||||
|
||||
---
|
||||
|
||||
## LIVE on demo-hp — the rendered dialog, which is the deliverable
|
||||
|
||||
**Legacy state** (before any run under v0.182.0) — today's wording, no fright:
|
||||
```
|
||||
Legutóbbi másolat: 2026-07-28 03:30
|
||||
```
|
||||
|
||||
Failure induced genuinely: the Tier-2 destination directory was **moved aside** and replaced by a file,
|
||||
so `mkdir …/recovery-unit` fails. (`chmod` does not work — the controller runs as root, which bypasses
|
||||
permission bits; `chattr +i` is refused, the unprivileged container lacks `CAP_LINUX_IMMUTABLE`. Both
|
||||
were tried and reported rather than glossed.) The real data was only ever moved, never deleted.
|
||||
|
||||
```
|
||||
status = error
|
||||
last_run = 2026-07-28T14:42:18Z ← ADVANCED
|
||||
last_success = 2026-07-28T14:40:55Z ← HELD
|
||||
last_error = mkdir …/paperless-ngx/recovery-unit: …
|
||||
```
|
||||
|
||||
**The rendered dialog, failed state:**
|
||||
```
|
||||
Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
|
||||
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:40.
|
||||
Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek
|
||||
lehetnek.
|
||||
```
|
||||
|
||||
**The rendered dialog, healthy state** (after restoring the destination and a successful run) — no
|
||||
caution, no tonal change:
|
||||
```
|
||||
Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
|
||||
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:43.
|
||||
```
|
||||
|
||||
Card lines: `Utolsó sikeres: 2 perce` → `Utolsó sikeres: most`.
|
||||
|
||||
**Everything restored:** destination is a directory again, 86 MB intact, mode 755, `.r101-aside` gone,
|
||||
`status=ok`, `last_success=2026-07-28T14:43:23Z`.
|
||||
|
||||
**demo-felhom is the untouched control:** all 5 rows still `tracked=None` after the deploy, rendering
|
||||
today's way, 15/15 containers up. Scenario E holding across a whole box nobody ran.
|
||||
|
||||
---
|
||||
|
||||
## NOT yet live-validated (carried forward)
|
||||
- **F-DIAG's classes** — unit-proven only. No live offsite failure of each class was induced; the
|
||||
`transport` class is the only one this arc exercised indirectly.
|
||||
- **Scenario C live** — the never-succeeded rendering is unit-proven; no fleet row is in that state
|
||||
(every row either migrated or has a real success), and manufacturing one would mean breaking a
|
||||
customer app's only Tier-2 history.
|
||||
- **The Tier-2 restore itself** was not executed — this arc changed what the dialog *says*, not what the
|
||||
restore does.
|
||||
- **R-100's 48h staleness threshold** — injected clock only.
|
||||
- **Fault 4** (restic transport interruption), **R-99**, **F-HUB**, fault 12, the three-way concurrency
|
||||
overlap — next campaign's material, untouched here.
|
||||
@@ -1,136 +0,0 @@
|
||||
# REPORT — R-106 + R-109 (+ R-122): closing the recipe-completeness set (2026-07-30)
|
||||
|
||||
Non-overwritten sibling per `CLAUDE.md:82-87` — the shared `REPORT.md` holds R-117 and is not touched.
|
||||
|
||||
Shipped: **agent v0.118.0 → v0.118.1** (`felhom-agent` `1c8a67e`, `6b5dade`) + **hub v0.83.0**
|
||||
(`felhom.eu` `acfc2b7`). Neither half is useful alone.
|
||||
|
||||
**Read §3 first if you read nothing else:** v0.118.0's R-106 half shipped INERT and live validation is
|
||||
what caught it — the recipe still said `"root"`, now with `namespace_state: resolved` beside it. Full
|
||||
account in the audit §6, filed as **R-125**.
|
||||
|
||||
## Part 0 — the answers, before the fix
|
||||
|
||||
### 0.1 Which items are actually open, and R-105/R-106's registration
|
||||
|
||||
`OPEN-ITEMS.md` calls itself "the single source of truth for open work" (`:1`), with `ROADMAP.md` keeping
|
||||
"the full history and reasoning" (`:3-4`).
|
||||
|
||||
| item | `ROADMAP.md` | `OPEN-ITEMS.md` | verdict |
|
||||
|---|---|---|---|
|
||||
| R-105 | row, `READY — 2026-07-28` (`:108`) | **absent** | **open but UNREGISTERED** |
|
||||
| R-106 | row, `READY — 2026-07-28` (`:109`) | **absent** | **open but UNREGISTERED** |
|
||||
| R-108 | row (`:111`) | row (`:50`) | registered |
|
||||
| R-109 | row (`:112`) | row (`:61`) | registered |
|
||||
|
||||
So R-109's own cell — "third recipe-completeness defect beside R-105/R-106" — was the **only** place in the
|
||||
register naming two open items. That is exactly the thread-loss the register exists to prevent, and it is
|
||||
itself a finding (filed **R-123**). Both now have rows.
|
||||
|
||||
**The set this task closes is R-106 + R-109**, matching the arc's stated definition of done (`OPEN-ITEMS.md:14`).
|
||||
**R-105 is NOT in it** and was not worked: it is M-sized and is about three *hub-held DR records* being `{}`
|
||||
(`hosts.dr_record_json`, `host_escrow.directive_json`, and the `drives` third — already traced and populated
|
||||
by the 2026-07-28 target move). Different fields, different owner, different size.
|
||||
|
||||
### 0.2 Where the recipe is generated — three producers, not two
|
||||
|
||||
| half | repo | function |
|
||||
|---|---|---|
|
||||
| host (guests/pbs/drives/pve_storage) | `felhom-agent` | `BuildDRRecipeHostHalf`, `internal/hub/dr_recipe.go:86` |
|
||||
| app (customer/apps/offsite_restic) | `felhom-controller` | `controller/internal/report/dr_recipe.go` |
|
||||
| **assembly + delivery** | `felhom.eu/hub` | `AssembleDRRecipe`, `internal/store/dr_recipe.go:104`; served by `handleDRRecipeDownload`, `internal/web/dr_recipe.go:14`, route `internal/web/server.go:439` |
|
||||
|
||||
R-109's "host-half" is therefore the **agent**, and the field must also pass the **hub's** allow-list — see §2.
|
||||
|
||||
### 0.3 What the namespace field actually contained — verified, and the brief was RIGHT
|
||||
|
||||
The eleven-session-old brief held up. Live, pre-fix, from the hub for **both** boxes:
|
||||
|
||||
```json
|
||||
"pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }
|
||||
```
|
||||
|
||||
against `/etc/pve/storage.cfg` on the same boxes:
|
||||
|
||||
```
|
||||
pbs: felhom-pbs
|
||||
datastore felhom-offsite
|
||||
namespace demo-felhom # demo-hp reads: namespace demo-hp
|
||||
```
|
||||
|
||||
Traced to source: `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo
|
||||
per item once the list is namespace-scoped via `?ns=` (`:118-120`) → always empty → `ToHub` normalises empty
|
||||
to `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it in.
|
||||
|
||||
**The authority taken, and why:** storage.cfg's `namespace` on the pbs storage. It is the same field
|
||||
`vzdump --storage <pbs>` makes PVE read, and the agent's own verify client is built from it
|
||||
(`cmd/felhom-agent/main.go:1164`). Deriving the recipe from anything else is how it drifts again.
|
||||
|
||||
## 1. R-109's ambiguity is real, in the boxes' own pre-fix recipe
|
||||
|
||||
```json
|
||||
"pve_storage": [
|
||||
{ "name": "local-lvm", "type": "lvmthin", "content": "images,rootdir" },
|
||||
{ "name": "felhom-backup", "type": "local-dir", "content": "backup" },
|
||||
{ "name": "felhom-pbs", "type": "pbs", "content": "backup" },
|
||||
{ "name": "local", "type": "local", "content": "backup,import,vztmpl,iso" }
|
||||
]
|
||||
```
|
||||
|
||||
No `backup_target` key anywhere. `felhom-backup` (live, `/mnt/hdd_1`) and `local` (`/var/lib/vz`, archives
|
||||
frozen 2026-07-28) are both `content=backup` dir storages; `local` is also the *historically* correct answer,
|
||||
which is what makes guessing it so easy.
|
||||
|
||||
## 2. R-122 — a fourth defect, found here, and it had already shipped
|
||||
|
||||
`AssembleDRRecipe`'s `hostHalfShape`/`appHalfShape` are **allow-lists** dressed as forward-compat. The
|
||||
controller has emitted `offsite_restic` since fork-4 (`controller/internal/report/dr_recipe.go:39-41`, "so DR
|
||||
knows WHERE to recover from"); `appHalfShape` never listed the key. Verified both ways:
|
||||
|
||||
- **stored**: `dr_recipe.app_half_json` carries it for all three real customers —
|
||||
`peti-felhom`, `demo-felhom` (`u629488-sub1.your-storagebox.de:23/home/felhom-repo`), `demo-hp`.
|
||||
- **delivered**: the downloaded recipe's top-level keys were
|
||||
`recipe_version, customer, guests, pbs, drives, pve_storage, apps` — **no `offsite_restic`**.
|
||||
|
||||
So a restorer reading the recipe had **no offsite location at all**, for the whole life of the feature, with
|
||||
a green suite throughout — because the test fixture `drAppHalf` is hand-written and omits the field.
|
||||
|
||||
**Deviation from the task's §7.10 ("Findings — filed as R-n, none fixed"), stated rather than absorbed:**
|
||||
I fixed it. Reasons — (a) Part 0 authorises working the real set if it differs; (b) it is the same
|
||||
symptom the task is named for (the recipe is incomplete), and the worst instance, a whole section missing;
|
||||
(c) it is in the *same two structs* R-109 forced me to edit, and leaving one of three known keys off a
|
||||
drop-list I was already correcting would be indefensible. It is filed as R-122 with a SHIPPED disposition.
|
||||
|
||||
## 3. The before/after recipe — both boxes, quoted
|
||||
|
||||
```
|
||||
demo-felhom BEFORE "namespace":"root" backup_target absent offsite_restic absent
|
||||
AFTER "namespace":"demo-felhom" backup_target {resolved, felhom-backup, /mnt/hdd_1}
|
||||
offsite_restic {u629488-sub1…}
|
||||
demo-hp BEFORE "namespace":"root" backup_target absent offsite_restic absent
|
||||
AFTER "namespace":"demo-hp" backup_target {resolved, felhom-backup, /mnt/nvme-1tb}
|
||||
offsite_restic {u629488-sub3…}
|
||||
```
|
||||
|
||||
The two boxes DISAGREEING is the point — nothing is hardcoded. And the ambiguity was not theoretical:
|
||||
on both boxes `felhom-backup` holds an archive from **07-30 04:36** while `local` stops at
|
||||
**07-28 17:5x**, frozen at the target-move date. The recipe now names the live one.
|
||||
|
||||
Full evidence, all seven red-proofs and the publish observables:
|
||||
`documentation/audits/R106-R109-recipe-completeness-2026-07-30.md`.
|
||||
|
||||
## 4. Findings filed (none of them fixed except R-122, see §2)
|
||||
|
||||
| id | finding |
|
||||
|---|---|
|
||||
| **R-122** | `AssembleDRRecipe` allow-list dropped `offsite_restic` for the feature's whole life — **FIXED here**, hub v0.83.0 |
|
||||
| **R-123** | R-105 and R-106 were `READY` in `ROADMAP.md` with no `OPEN-ITEMS.md` row — referenced only inside R-109's prose. Registered here |
|
||||
| **R-125** | v0.118.0 shipped an inert R-106 because the "production path" test injected `fakeObserver` one layer below the break — **FIXED** in v0.118.1; filed for the doctrine point (name the seam you inject at) |
|
||||
| **R-124** | The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""` and there is no namespace literally named `root` — a restorer pasting it into `pct restore --ns root` would fail. Pre-existing wire convention, deliberately unchanged; documented at `PBSRootNamespace` |
|
||||
|
||||
## 5. Not done, and why
|
||||
|
||||
- **R-105, R-108, D5** — out of scope by the task's §6. R-108 blocks D5; starting either would leave both half-done.
|
||||
- **The backup machinery** — untouched. This corrects the record, not the doing.
|
||||
- **`sess-f` (0.116.0) and `drill-r50` (0.113.0) were not upgraded** — neither was named as a venue, and `drill-r50` is fenced by the task's §6.
|
||||
- **R-124 not fixed** — changing the wire's spelling of the root namespace mid-R-106 would shift the field's meaning during the fix meant to make it trustworthy.
|
||||
@@ -1,117 +0,0 @@
|
||||
# REPORT-r116-diag — the `/disks` payload captured, R-116's mechanism isolated (2026-07-30)
|
||||
|
||||
Read-only diagnosis run by CC on DooPlex. **No code written, nothing built, nothing published.**
|
||||
Full evidence: `documentation/audits/DIAG-r116-disks-payload-2026-07-30.md`.
|
||||
|
||||
A `REPORT-*.md` sibling, not the shared `REPORT.md` (`CLAUDE.md` parallel-session rule).
|
||||
|
||||
## Outcome
|
||||
|
||||
**Both goals met.** The `/disks` read path is solved and written down verbatim, proven by a
|
||||
present-drive control run *first*; and the absent-state payload was captured, which isolates the
|
||||
mechanism.
|
||||
|
||||
**R-116 is theory #1 — "the registry-union row writes `false`" — the theory that was raised, declared
|
||||
wrong, and retracted. The retraction was the error.**
|
||||
|
||||
In the absent state `/disks` returns **4 rows, not 3**. The drive appears twice and the two facts the
|
||||
controller needs sit on different rows:
|
||||
|
||||
| row | source | `mount_path` | `guest_path` | `backup_target` |
|
||||
|---|---|---|---|---|
|
||||
| `felhom-backup` | Observe (`disks.go:196-284`) | `""` | `""` | **`true`** |
|
||||
| `694034cc-…` (the UUID) | registry union (`disks.go:297-339`) | `/mnt/cel` | `/mnt/felhom-drives/cel` | **field ABSENT ⇒ `false`** |
|
||||
|
||||
So the row holding the flag contributes **no key** to `driveTargetByPath`, and the row that owns the key
|
||||
says `false` → `isTarget[a.Path]` is `false` → generic `storage_disconnected`. On return the rows
|
||||
re-merge into one carrying both facts → specific `backup_target_restored`. Applying
|
||||
`intermediary.go:602-618` to the captured payloads gives PRESENT `True` / ABSENT `False` /
|
||||
RETURNED `True` — **the live asymmetry reproduced from payload alone.**
|
||||
|
||||
The union row's `MountPath` survives the device because the union source is the systemd **`.mount` unit
|
||||
file** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never reads the mount table. The
|
||||
dedup at `:298` therefore does not fire, because `seen` is keyed on the one field the absent state
|
||||
empties (`:290-295`).
|
||||
|
||||
**Theory #2 (the basis of the shipped v0.115.0) is false on both halves**; **#3 is false too**
|
||||
(`isTarget["/mnt/cel"]` is `false` as well). **v0.115.0 is provably inert** — its fallback calls
|
||||
`StablePathForRaw("")`, which returns `""` (`intermediary.go:69-75`), so it assigns nothing.
|
||||
|
||||
## The read path (this cost two prior sessions — it should never cost again)
|
||||
|
||||
The token plaintext exists in exactly one place: `bootstrap.json` **on the Proxmox host**. The agent's
|
||||
own store keeps SHA-256 hashes only (`tokenstore.go:26-32`), which is what defeated the earlier attempts.
|
||||
|
||||
```bash
|
||||
ssh felhom-pve
|
||||
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
|
||||
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
|
||||
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])")
|
||||
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
|
||||
```
|
||||
|
||||
Control run, live felhom-pve, drive present: **HTTP 200, 2483 bytes, 4 plausible rows** — so Part 5's
|
||||
0-rows-on-a-present-drive failure mode is excluded.
|
||||
|
||||
## Where the absent state was staged
|
||||
|
||||
No new box. The existing DooPlex **nested-PVE drill fixture** (`drill/drill.qcow2`, snapshot `virgin`) —
|
||||
my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with
|
||||
the **byte-identical live agent binary** (`sha256 f48544ad…`, `--version` 0.115.0) and every
|
||||
state-producing step through the real endpoints (`format` → `assign` → `guest-attach` →
|
||||
`backup/target`). Its present-state row matched felhom-pve's control run field-for-field before it was
|
||||
trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token
|
||||
record, no controller) are enumerated in the audit §4.
|
||||
|
||||
## Two new findings, filed not chased
|
||||
|
||||
- **R-117 (READY M) — outranks R-116.** After a detach/reattach the guest's bind is a **dead mount**:
|
||||
host is healthy on the new device node, guest still names the old one, and `ls`/write through it
|
||||
return **`EIO`** — while `/disks` reports `attached` + `bound_under_parent:true` + `backup_target:true`.
|
||||
`planDriveGates` therefore takes the `Return` branch and **restarts the customer's apps onto a dead
|
||||
namespace, reporting healthy, with no alarm on any channel.** R-113's conjunction cannot catch it:
|
||||
one half is satisfied by the stale entry, the other by the new device, and neither compares them.
|
||||
This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs.
|
||||
- **R-118 (READY XS).** An absent drive's union row reports the **root filesystem's** capacity as its own
|
||||
(46 GiB / 9.2 % for a 4 GB drive) — `statfsCapacity` at `disks.go:335-338` statfs's a bare directory on
|
||||
root. `observe.go:176-183` guards the Observe path against exactly this; the union path does not.
|
||||
`durable_id` is still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id.
|
||||
|
||||
## Register
|
||||
|
||||
`documentation/backlog/OPEN-ITEMS.md` — R-116 updated with the mechanism and the fix constraints;
|
||||
R-117 and R-118 added. The single register edit this session makes.
|
||||
|
||||
## Record correction
|
||||
|
||||
The brief's baseline `controller 0.185.1` is the version the **golden bakes**. **0.186.0** (R-114 +
|
||||
R-112, 2026-07-29) is what **demo-felhom** runs — **demo-hp is still on 0.185.1**, so the fleet is
|
||||
split, and R-114's `TargetAbsent` branch exists only on demo-felhom. Confirmed: agent **0.115.0**
|
||||
(felhom-pve) / **0.113.0** (demo-hp), hub **0.81.0** (manifest pin and live pod image agree),
|
||||
host-install **1.22.0**, `felhom.eu` HEAD `c3ce4c7`.
|
||||
|
||||
> **Correction, 2026-07-30.** As first written this section said 0.186.0 was what *both* demo boxes run.
|
||||
> That was wrong — only felhom-pve's guest was sampled and the result generalised to the fleet. demo-hp
|
||||
> re-checked directly → `0.185.1`. Fixed here and in the audit's baseline table.
|
||||
|
||||
## Teardown and fences
|
||||
|
||||
Drill guest destroyed, scratch storage removed, mount unit deleted, secrets `shred -u`'d, VM powered
|
||||
off, **`drill.qcow2` restored to `virgin`** (the golden-bake fixture is exactly as found), scratch qcow2
|
||||
and console dumps deleted. DooPlex `/mnt/5_hdd` at 24 %, unchanged.
|
||||
|
||||
Both demo boxes **read-only throughout** and re-verified after teardown: demo-hp `local-lvm` **38.83 %**
|
||||
(identical before/after and to Part 5), `drill-r50` still stopped, felhom-pve `felhom-backup` still
|
||||
active on `/dev/sdb`, both guests running, **v0.115.0 untouched**.
|
||||
|
||||
`sess-d-0452c4` now reads **STALE**, and the delete gate refuses only on ONLINE
|
||||
(`hub/internal/web/customer_delete.go:220-228`) — so it **is** now deletable; the command is recorded in
|
||||
the audit rather than executed (customer delete runs external teardown plus a DB purge). **`sess-c` is
|
||||
also still present and was not recorded by the Session-C audit** — same terms.
|
||||
|
||||
## Not done, deliberately
|
||||
|
||||
No fresh controller gate-log / hub-event correlation: that observable was already captured live and
|
||||
identically twice, and only the payload was missing. Staging a controller meant a hub customer, a
|
||||
pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8
|
||||
correlates the payload to those existing measurements and labels that step as inference.
|
||||
@@ -1,96 +0,0 @@
|
||||
# REPORT — SPIKE R-117: a dead bind that reports healthy (2026-07-30)
|
||||
|
||||
Written as `REPORT-<topic>.md`, not `REPORT.md`, per this repo's parallel-session rule and the
|
||||
established local pattern (`REPORT-r116-diag.md`, `REPORT-session-c.md`, …). The shared `REPORT.md`
|
||||
was not touched.
|
||||
|
||||
**Class: Spike.** Deliverable is a findings document. **No production code was written; no `.go` file
|
||||
in either repo was modified; nothing was built for deployment, published, or version-bumped.**
|
||||
|
||||
## Deliverables
|
||||
|
||||
| File | Action |
|
||||
|---|---|
|
||||
| `documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md` | **new** — the findings doc (Q1–Q7, evidence, recipe, probe comparison, recommendation) |
|
||||
| `documentation/backlog/OPEN-ITEMS.md` | **R-117 row rewritten** with the mechanism, the reproduction recipe and the fix constraint — the one register edit, per the brief |
|
||||
|
||||
**No CHANGELOG entry.** This repo's changelogs are per-area (`hub/`, `scripts/`, `website/`); a
|
||||
documentation-only change belongs to none of them. Stated rather than silently skipped, per standing
|
||||
rule 4.
|
||||
|
||||
## Baselines
|
||||
|
||||
agent `main` **v0.116.0** @ `d4eb259` · controller `main` **v0.186.0** @ `b331f18` ·
|
||||
`felhom.eu` HEAD `29bcfeb` · hub **live 0.82.0** · golden **0.186.0** ·
|
||||
demo-hp PVE 9.2.2 / kernel 7.0.2-6-pve, **live agent 0.113.0** (= manifest `MinAgent`; never used as
|
||||
the source of a behavioural claim — every predicate result came from a probe built from `main`).
|
||||
|
||||
## Results
|
||||
|
||||
**All seven questions answered empirically.**
|
||||
|
||||
- **Q1 — reproduced 3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp.
|
||||
**The device-node change is a consequence of the defect, not a precondition** — control test: with
|
||||
the stale bind held the drive returns as `sdc` (8:32); released, the letter is reused (`sdb`, 8:16).
|
||||
- **Q2 — two death states**: device removed ⇒ superopts gain `shutdown`, `EIO`(5) on read and write,
|
||||
host and guest; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads served from cache.
|
||||
The raw host mount is genuinely healthy in both. **No cross-device mis-identification is possible**
|
||||
on this path — the unit is fs-UUID-keyed.
|
||||
- **Q3 — confirmed at source and live.** Both halves of the R-113 conjunction compare **field 5** of a
|
||||
mountinfo line and **never read field 3 (`major:minor`)**, so neither can see that the bind and the
|
||||
raw mount name different devices. Measured `BoundUnderParent = TRUE` over an `EIO` namespace.
|
||||
- **Q4 — a pure-`/proc` check costs 0.16–0.45 ms**, cannot hang, spins up no disk, writes nothing.
|
||||
**`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow. The
|
||||
hang case is below; it is the sharpest result in the run.
|
||||
- **Q5 — the agent**, and not on balance: the controller runs inside the guest and cannot see the host
|
||||
mount tables the check needs.
|
||||
- **Q6 — recovery works in place, guest never restarted** (init PID identical). **The repair code
|
||||
already exists and three call sites already invoke it**, including the controller's `Return` branch
|
||||
*before* it restarts apps — all defeated by one early return.
|
||||
- **Q7 — a bind can die in steady state, no cycle at all.** The gate produces no action and **nothing
|
||||
is emitted on any channel.** A `Return`-branch fix cannot reach this half.
|
||||
|
||||
## Q4's hang case — measured, and it is the sharpest result
|
||||
|
||||
Against a `dmsetup suspend`ed device (I/O queues instead of returning `EIO`):
|
||||
|
||||
- **P1 and P2 completed in 364 µs / 206 µs.** They read `/proc`, so no block device is involved.
|
||||
- **`statfs` and `getdents` completed and reported HEALTHY** — on a wedged device they do not even hang.
|
||||
- **Every probe that touches the device blocked, including a buffered write with no `fsync`** — the
|
||||
`O_CREAT` metadata path needs journal access (`wchan=do_get_write_access`). There is no cheap-and-safe
|
||||
write probe.
|
||||
- **The blocked process survived `SIGTERM` and `SIGKILL`** (`stat=D`, still alive 3m50s after `kill -9`)
|
||||
and died only when the device was resumed. So **`systemctl restart felhom-agent` would hang**, leaving
|
||||
the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal
|
||||
the leak (5→5, 5→6).
|
||||
|
||||
**A timeout protects the caller's control flow and nothing else.** This turns "prefer a cheap probe" into
|
||||
a fence: **the fix must issue no block I/O.**
|
||||
|
||||
## Teardown — done, all three layers
|
||||
|
||||
Guest 9301 destroyed; `r117scratch` removed; both dm devices and both loop devices gone; `scsi_debug`
|
||||
unloaded (`/dev/sd*` back to `sda1..3`); no `r117` mounts, `/mnt` and `/root` exactly as found; `local`
|
||||
**37.02 %** against a session-start **37.00 %**. Fences re-verified *after* teardown: 9201 `running`,
|
||||
`drill-r50` `stopped`, `local-lvm` **38.84 % byte-identical**, `felhom-backup` `content backup`
|
||||
unchanged, live `/mnt/felhom-drives` intact with both submounts, agent service `active`. **Layer 3 is
|
||||
genuinely empty** — 9301 had no network interface and ran no controller, so no hub-side record was ever
|
||||
created.
|
||||
|
||||
**Ordering trap worth keeping:** a suspended dm device must be `dmsetup resume`d *before* any `umount`,
|
||||
or the teardown itself blocks on the same uninterruptible sleep.
|
||||
|
||||
## Not measured
|
||||
|
||||
No load or duration testing of the recommended check — P1/P2 were single calls, not a sustained
|
||||
reconcile loop on a many-drive box. Nothing suggests a problem (they are two `/proc` reads the code
|
||||
already performs), but it was not measured.
|
||||
|
||||
## Findings filed, none fixed
|
||||
|
||||
R-117 (mechanism + recipe), **R-117a** steady-state death with no event (HIGH, larger than R-117 as
|
||||
filed), **R-117b** `statfs`/`getdents` are hollow liveness probes, **R-117c** three untested comments
|
||||
promising "live + usable in the guest", **R-117d** the self-heal that already exists is short-circuited
|
||||
(HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing internet,
|
||||
including the WireGuard OOB path, **R-117f** an I/O liveness probe turns a wedged drive into an
|
||||
unkillable agent (HIGH — disqualifies a whole probe class).
|
||||
@@ -1,160 +0,0 @@
|
||||
# REPORT — hub v0.88.0: the WAL that never was (R-172), plus R-173 found
|
||||
|
||||
**Session artefact naming:** written as `REPORT-r172-hub-wal.md`, not `REPORT.md`, per this repo's
|
||||
parallel-session rule — the shared `REPORT.md` belongs to the controller boot-recovery session that
|
||||
ran immediately before this one and must not be clobbered.
|
||||
|
||||
**Repo:** `felhom.eu` (hub `v0.87.0` → **`v0.88.0`**) · **Trigger:** a `HOST STALE` banner the
|
||||
operator spotted on `hub.felhom.eu` after the previous session finished.
|
||||
|
||||
---
|
||||
|
||||
## 1. What the alarm actually was
|
||||
|
||||
**Not the agent, not the guest.** The agent was up **2 days**, never restarted, and actively
|
||||
reconciling; the controller was reporting normally (the header read "Last report just now", 0.190.0,
|
||||
10/10 containers). The failure was the hub **writing** the host report:
|
||||
|
||||
```
|
||||
20:26:34 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY)
|
||||
20:41:32 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY)
|
||||
20:42:32 [INFO] Host staleness: demo-felhom-8363b5 ok → stale (host_stale)
|
||||
20:42:33 [INFO] Operator email sent for demo-felhom/host_stale
|
||||
```
|
||||
|
||||
The chain, and the margin is exactly one retry wide: reports are every **15 min**, staleness fires at
|
||||
**30 min**, the hub returns **500** on `SQLITE_BUSY` without retrying, and the agent logs
|
||||
`keeping current interval` and waits a full interval without retrying either. **Two consecutive
|
||||
collisions = a false alarm.** It had already fired once that day (19:12:32, recovered 19:20:32).
|
||||
|
||||
**Was it caused by the preceding session?** Partly amplified, not caused. 13 collisions in one pod
|
||||
lifetime; **the first at 15:56 CEST, ~3 h before that session's first deploy**. 7 of 13 fell inside
|
||||
its window of ~13 controller restarts, which raises write concurrency — so the burst made a
|
||||
pre-existing fault more likely, and the fault was not new.
|
||||
|
||||
## 2. Root cause — the pragmas were never applied
|
||||
|
||||
The DSN was `?_journal_mode=WAL&_busy_timeout=5000`. That is **mattn/go-sqlite3** syntax. The driver
|
||||
is **modernc.org/sqlite v1.45.0**, whose `applyQueryParams` (confirmed at source in the module cache)
|
||||
reads only `_pragma`, `_time_format`, `_time_integer_format`, `_txlock` and `_inttotime` — and
|
||||
**silently ignores everything else**. No error, no warning.
|
||||
|
||||
So the hub ran in the default **rollback-journal** mode with **`busy_timeout=0`** for its entire life
|
||||
while its own source said WAL. In rollback-journal mode a reader excludes a writer, so rendering an
|
||||
operator page can block a host report — which is precisely the observed 500.
|
||||
|
||||
**The observable that proved it before any code changed:** a 128 MB `/data/hub.db` with **no
|
||||
`-wal`/`-shm` file beside it while the database was open**. In WAL mode those must exist.
|
||||
|
||||
This is the project's recurring class — a configuration asserting an invariant the code does not
|
||||
provide — and it is the second one this week.
|
||||
|
||||
## 3. The fix
|
||||
|
||||
```
|
||||
?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate
|
||||
```
|
||||
|
||||
| Parameter | Why it is not optional |
|
||||
|---|---|
|
||||
| `journal_mode(WAL)` | readers and one writer proceed concurrently, so a page render can no longer block a report; it is a property of the database FILE and persists once set |
|
||||
| `busy_timeout(5000)` | writers still serialise; without a timeout SQLite returns `SQLITE_BUSY` *immediately* rather than waiting |
|
||||
| `_txlock=immediate` | **the one that is easy to miss.** `database/sql`'s `Begin()` is DEFERRED, so a read-then-write transaction must upgrade its lock, and a failed upgrade is `SQLITE_BUSY_SNAPSHOT` — which **`busy_timeout` does not retry**. This store has **10+ `db.Begin()` sites and they are all write paths** (customer delete/reset, wg, appliance, pbsdr, telemetry, log bundles). WAL + busy_timeout alone would have shipped half a fix with a known un-retryable path left open |
|
||||
|
||||
**Retry options (b) and (c) from R-172 were deliberately NOT taken.** With readers no longer blocking
|
||||
writers and the upgrade path covered, a `SQLITE_BUSY` reaching a handler should now be rare enough to
|
||||
be a real signal; a retry would hide it. Revisit only on evidence.
|
||||
|
||||
## 4. Tests and the red-proof
|
||||
|
||||
**Every assertion reads the value back from the DATABASE, never the DSN string** — a string assertion
|
||||
would have passed happily for the entire life of the bug. Six tests in `internal/store/pragma_test.go`:
|
||||
|
||||
| Test | Asserts |
|
||||
|---|---|
|
||||
| `TestStorePragmasAreActuallyApplied` | runtime `journal_mode` = wal, `busy_timeout` ≥ 5000 |
|
||||
| `TestStoreWALFilesExistWhileOpen` | `-wal`/`-shm` exist beside an open DB — **the production signature, pinned** |
|
||||
| `TestStoreReaderDoesNotBlockWriter` | the CONSEQUENCE: a write during a held read succeeds |
|
||||
| `TestStoreConcurrentWritersDoNotReturnBusy` | 8 concurrent writers all wait rather than error |
|
||||
| `TestStoreTransactionUpgradeDoesNotReturnBusySnapshot` | 6 racing read-then-write transactions all commit |
|
||||
| `TestSQLiteDriverIgnoresMattnStyleParams` | guards the ROOT CAUSE: fails if the pragmas are "tidied" back to mattn form; skips itself with instructions if a future driver starts honouring them |
|
||||
|
||||
**Red-proof — restore the DSN that shipped.** Observed FAIL, then reverted with a passing control:
|
||||
|
||||
```
|
||||
journal_mode = "delete", want "wal"
|
||||
hub.db-wal is missing beside an OPEN database
|
||||
a write FAILED while a read was open: database is locked (5) (SQLITE_BUSY) ← the live error, exactly
|
||||
```
|
||||
|
||||
`go build ./... && go vet ./... && go test ./...` in `hub/` → **rc=0**. `scripts/repo_gates.py --fast`
|
||||
→ all 5 gates OK.
|
||||
|
||||
## 5. Operational consequence — handled, not discovered later
|
||||
|
||||
**A WAL database cannot be copied by taking `hub.db` alone.** A committed transaction may still live
|
||||
in `hub.db-wal`, so a bare `cat` yields a copy that **opens cleanly and silently omits the newest
|
||||
writes** — the worst possible shape for a credential lookup. The break-glass root-password retrieval
|
||||
in `documentation/operations/nodes.md` used exactly that command, and `_recovery-inventory-2026-07-28.md`
|
||||
records it as a past action that reads like a recipe. Both are now WAL-aware: copy the `-wal`
|
||||
alongside, `|| true` because an absent `-wal` is legitimate, and **shred both** (the WAL holds the
|
||||
same secrets).
|
||||
|
||||
Not hypothetical: the live `-wal` measured **729,272 bytes** during verification, all of which a bare
|
||||
`cat` would have dropped.
|
||||
|
||||
## 6. Live verification
|
||||
|
||||
- Image built and pushed; **`manifests/hub.yaml` 0.87.0 → 0.88.0** (the only thing ArgoCD deploys
|
||||
from), hard-refresh + deliberate sync (auto-sync is OFF). ArgoCD **Synced / Healthy**.
|
||||
- `/data/` now shows **`hub.db-wal` and `hub.db-shm`** beside the open DB — the exact observable whose
|
||||
absence proved the bug.
|
||||
- **Zero `SQLITE_BUSY` since the rollout.**
|
||||
- Host report landed at 21:11:33; staleness checker reports `2 ok, 0 stale` — `demo-felhom-8363b5`
|
||||
is back to `ok`.
|
||||
- `PRAGMA integrity_check` → **`ok`**, `journal_mode` → **`wal`**, all tables intact
|
||||
(hosts 4, customer_configs 6, host_recovery 4, host_reports 2756, events 2293). Read via the new
|
||||
WAL-aware copy recipe and shredded afterwards.
|
||||
|
||||
## 7. A mistake I made, and it caused a real outage
|
||||
|
||||
**I OOM-killed the hub pod three times** with a 60-concurrent page-render load test intended to prove
|
||||
the fix under contention. The pod's limit is **256 Mi**; 60 simultaneous renders of a heavy customer
|
||||
page exceeded it (`OOMKilled`, exit 137, readiness probe timeouts). The hub was unavailable for parts
|
||||
of ~6 minutes and recovered on its own.
|
||||
|
||||
Three things worth stating plainly:
|
||||
|
||||
1. **It was not the WAL change.** WAL's extra footprint is the 32 KB `-shm` mapping; the OOM was
|
||||
template rendering under concurrency I created.
|
||||
2. **I should not have run it.** DooPlex is **Tier 2 — precious**, and the hub is part of the
|
||||
recovery chain. A synthetic load test at that concurrency against a memory-limited pod on that box
|
||||
was the wrong call; the unit tests already proved the property, and the live proof needed was the
|
||||
`-wal` file plus a clean report — both of which I already had.
|
||||
3. **It did produce one piece of genuine evidence**, which does not excuse it: after three hard kills
|
||||
the WAL replayed cleanly and `integrity_check` returned `ok`, which is a real (if unplanned)
|
||||
demonstration of WAL crash-safety on this volume.
|
||||
|
||||
No data was lost. The pod is `Ready`, restart count 3, serving normally.
|
||||
|
||||
## 8. Backlog
|
||||
|
||||
- **R-172 → CLOSED**, with the root cause recorded as *the pragmas were never applied*, not as tuning.
|
||||
- **R-173 → NEW.** While checking who copies `hub.db` for the WAL change, I found `pvc/hub-data`
|
||||
carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly`
|
||||
are the only recurring jobs and both target `default`. **The hub database has no volume-level
|
||||
backup** — and it holds `host_recovery` (every box's break-glass root password), `host_escrow` +
|
||||
`host_escrow_superseded`, `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg peers.
|
||||
Filed rather than fixed: whether the exclusion is deliberate is a question for the operator, and the
|
||||
manual hot copy recorded in `_recovery-inventory` is not a backup. `grep` established the ID free.
|
||||
|
||||
## 9. Observations — noticed, NOT acted on
|
||||
|
||||
1. **The hub returns HTTP 500 for a transient lock**, which is what turned a retryable condition into
|
||||
an alarm. Left as-is deliberately (§3) so a surviving `SQLITE_BUSY` stays visible.
|
||||
2. **The agent does not retry a failed report**, so one collision costs a full 15-minute interval —
|
||||
half the staleness budget. Same reasoning; if collisions recur, this is the cheaper of the two
|
||||
retry fixes.
|
||||
3. **`store.New` sets no `SetMaxOpenConns`**, so `database/sql` may open unbounded connections against
|
||||
a single-writer database. Not changed here — WAL plus the immediate-lock covers the observed
|
||||
failure, and bounding the pool changes latency characteristics that nothing currently measures.
|
||||
@@ -1,203 +0,0 @@
|
||||
# REPORT — installer-channel record correction + R-29 filing (2026-07-29)
|
||||
|
||||
Two commits, documentation only. No code, no version bump, no CHANGELOG entry, no build, no deploy,
|
||||
no box touched. Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so root `REPORT.md` (the E-2
|
||||
increment-1 report) is preserved.
|
||||
|
||||
| # | Commit | Baseline | Scope |
|
||||
|---|--------|----------|-------|
|
||||
| 1 | `d4c07873ca0c3d3e547373a9fafc0e472a6535e8` | `36d635a4cdc1`, unmoved | Retract a false R-94/E-2d finding; open R-110 |
|
||||
| 2 | (this commit) | `d4c07873`, unmoved | File R-29 to the register; three record defects; this report |
|
||||
|
||||
---
|
||||
|
||||
## Commit 1 — `d4c07873`
|
||||
|
||||
### What was false
|
||||
|
||||
`36d635a4` recorded that `felhom-bootstrap.sh` fetches the installer **from the hub**, that the hub
|
||||
therefore serves 1.19.0, and that a fresh ISO install runs the pre-E-2 installer. All three wrong.
|
||||
The claim had propagated into two `OPEN-ITEMS.md` rows, the ranking rationale, and `ROADMAP.md:149`.
|
||||
|
||||
### Confirmation table — all PASS
|
||||
|
||||
| # | Claim | Read at | Result |
|
||||
|---|-------|---------|--------|
|
||||
| F1 | bootstrap fetches from the **website** | `scripts/iso/felhom-bootstrap.sh:96` | PASS — `INSTALL_URL="${FELHOM_INSTALL_URL:-https://felhom.eu/scripts/felhom-host-install.sh}"` |
|
||||
| F2 | hub-rendered command points at the same URL | `customer_unified.html:563`, `:1262` | PASS — and **three** emission sites, not two: `:563` static, `:1262` JS error branch, **`:1267` JS success branch** |
|
||||
| F3 | website serves `/scripts/` from a git-sync tree tracking `main` | `manifests/webpage.yaml` — nginx `:74-77`, sparse-checkout CM `:211-218`, git-sync `:272-281`, init `:299-307` | PASS — `--branch=main --period=30s --link=current`; sparse-checkout `/website/` + `/scripts/`; `location /scripts/ { root …/current; }`. No image build, no ArgoCD step |
|
||||
| F4 | `hostInstallVersion` selects nothing | `configs.go:28`, `:487`; `render_test.go:219`; `customer_unified.html:494` | PASS — repo-wide grep returns exactly those 4 code sites; all other hits prose. Rendered as a text label |
|
||||
| F5 | every generated flag is parsed by 1.22.0 | generator `customer_unified.html:1206-1239` vs parser `felhom-host-install.sh:1175-1212` | PASS — `--mode --cores --memory --vmid --node --acl-storages --operator-pubkey-file --preserve-state-from --skip-provision --dry-run --preflight-only --allow-new-leaf` (+ `--customer-id`); every one a parser case. **No functional gap** |
|
||||
| F6 | installer is 1.22.0 | `scripts/felhom-host-install.sh:187` | PASS |
|
||||
| F7 | the drift test is hollow | `render_test.go:219-221` | PASS — `strings.Contains(html, hostInstallVersion)` compares the constant to itself; passes at any value |
|
||||
|
||||
### Live command 1 — what the URL actually serves
|
||||
|
||||
```
|
||||
$ curl -fsS https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 '^SCRIPT_VERSION='
|
||||
SCRIPT_VERSION="1.22.0" # the SINGLE version source (F-1): -h, the run banners, and the hub
|
||||
```
|
||||
|
||||
### Live command 2 — the drift gate's real state
|
||||
|
||||
```
|
||||
$ python3 scripts/hostinstall_gates.py; echo "exit=$?"
|
||||
ok: SCRIPT_VERSION=1.22.0
|
||||
ok: header has no version literal
|
||||
FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0 (F-1: bump both together)
|
||||
ok: age is in the installed package set
|
||||
… (six further ok lines) …
|
||||
hostinstall gates: 1 FAILURE(S)
|
||||
exit=1
|
||||
```
|
||||
|
||||
### Phase 0 source read — PAIRING reaches the same installer invocation
|
||||
|
||||
Mode selection `felhom-bootstrap.sh:537-541`: a fresh VM with no baked customer-id calls
|
||||
`run_pairing`. On HTTP 200 from `/api/v1/appliance/poll` the loop writes the hub-delivered
|
||||
`FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it, and calls
|
||||
`run_direct` **in the same invocation** (`:495-499`). `run_direct` is the single site that fetches
|
||||
`$INSTALL_URL` (`:322-330`), builds the args (`:334`) and invokes `bash "$SCRIPT_TMP" "${args[@]}"`
|
||||
(`:343`). The customer it yields is the one the operator bound — claimable. **So the ISO leg is the
|
||||
spine for E-2d**, not an obstacle to it.
|
||||
|
||||
### Rows changed
|
||||
|
||||
| ID | Before | After |
|
||||
|---|---|---|
|
||||
| R-94 (line 14) | `READY — deferred until E-2d`, blocked on E-2d, false ISO/hub framing | `READY (XS)`, blocked on nothing, retracted + re-scoped to three legs |
|
||||
| R-94 (line 16) | duplicate row, `READY #2`, 1.19.0 vs 1.20.0 | **deleted** — merged |
|
||||
| R-110 | did not exist | opened, `WAITING-ON-OPERATOR (S)` |
|
||||
| E-2d | ISO implied as obstacle | Next-action appended; ISO is the spine |
|
||||
| ranked list | 1 R-95 · 2 R-94 (high-consequence) · 3 R-86 · 4 R-87 | 1 R-95 · 2 R-94 **de-ranked** · 3 R-86 · 4 R-87 · 5 R-110 |
|
||||
|
||||
Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
|
||||
`documentation/runbooks/day0-install.md`.
|
||||
|
||||
---
|
||||
|
||||
## Commit 2 — R-29 filing + record hygiene
|
||||
|
||||
### The §1.1 ruling — is R-29 the right home for a non-design-v2 gate? **Yes. Proceeded.**
|
||||
|
||||
R-29's title says *"the design-v2 green gates"*, and `scripts/hostinstall_gates.py` is not one — it
|
||||
comes from drill F-1 (2026-07-12) and postdates the item. Four things decide it anyway:
|
||||
|
||||
1. **R-29's own audit list already spans well beyond design-v2 subject matter.** It names
|
||||
`docker_run_volume_path_gate` (docker mount safety), `offbox_rename_gate`, `app_row_dedup_gate`
|
||||
and `manifest_bearer_gate` (secrets — `runbooks/secrets.md:76`). The title is a misnomer relative
|
||||
to the item's own body.
|
||||
2. **Part (b) — "the systemic half is the real item" in R-29's words — is stated with no
|
||||
design-v2 restriction.** It is about the *enforcement mechanism*: "the gates run only when a human
|
||||
remembers to run them… decide where they run (pre-push hook, `build.sh` step, or a CI job) and make
|
||||
a red gate block the train." That is gate-agnostic and repo-wide.
|
||||
3. **`hub_confirm_gate.py` is already on R-29's list and lives in the same `scripts/` directory** as
|
||||
`hostinstall_gates.py`. Wiring one and not the other would be arbitrary.
|
||||
4. **Identical failure shape, identical genre.** Both self-describe as mechanical grep-assertions
|
||||
(`hostinstall_gates.py:2`, `hub_confirm_gate.py:1-8`); both exist, assert something true, and are
|
||||
invoked by nothing.
|
||||
|
||||
R-29 has already absorbed one independent re-raise without minting an ID (2026-07-18 rehearsal note)
|
||||
and says so explicitly. This is the third. No new ID minted.
|
||||
|
||||
### Orphan-search evidence — `hostinstall_gates.py` and `hub_confirm_gate.py`
|
||||
|
||||
Re-established at `d4c07873`. Pattern `hostinstall_gates\|hub_confirm_gate`.
|
||||
|
||||
| # | Scope | Command | Result |
|
||||
|---|-------|---------|--------|
|
||||
| S1 | `felhom.eu`, all file types | `grep -rn "$PAT" . --exclude-dir=.git` | **19 hits, zero invocations.** All are docstrings (`scripts/hostinstall_gates.py:5`, `scripts/hub_confirm_gate.py:7`), code comments (`hub/internal/web/configs.go:27`, `scripts/felhom-host-install.sh:189`) or prose (`REUSE.md:62`, `CONTEXT.md:540,564`, `hub/CHANGELOG.md:371,1292,1351,1385`, `scripts/CHANGELOG.md:483,524`, 3 files under `documentation/audits/`, `ROADMAP.md:149,158`, `OPEN-ITEMS.md:14`) |
|
||||
| S2 | sibling repos | `grep -rln "$PAT" /mnt/5_hdd/felhom.eu/git --exclude-dir=.git` minus this repo | 3 files, all in `.claude-memory/` (`MEMORY.md`, `drtier-by-default-2026-07-12.md`, `polish-batch-2026-07-13.md`). Notes, not invokers |
|
||||
| S3 | `~/.claude` | `grep -rln "$PAT" /home/kisfenyo/.claude`, and targeted on `settings*.json`, `skills/`, `hooks/` | **Zero hits in settings, skills or hooks** — where an invoker would live. Remaining hits are `file-history/` (Claude Code's own backups of files edited in past sessions) and `paste-cache/` (pasted task specs). Neither is an invocation site |
|
||||
| S4 | git hooks | `ls -1 .git/hooks/ \| grep -v '\.sample$'` | **Empty — every hook is a `.sample` stub** |
|
||||
| S5 | build files | `find . -type f \( -iname Makefile -o -iname '*.mk' -o -iname justfile -o -iname 'Taskfile*' \)` | Only `hub/Makefile`; `grep -n gate hub/Makefile` → **zero occurrences** |
|
||||
| S6 | CI | `find . -type d \( -name .github -o -name .gitea -o -name .woodpecker* -o -name .drone* -o -name .circleci \)` | **Empty — `felhom.eu` has no CI configuration at all** |
|
||||
|
||||
Of the four gates in `scripts/`, only `site_gates.py` is mandated (`CLAUDE.md:153`);
|
||||
`manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`.
|
||||
|
||||
### Rows changed
|
||||
|
||||
| ID | Before | After |
|
||||
|---|---|---|
|
||||
| **R-29** | **absent from `OPEN-ITEMS.md`** (`grep -c` → 0) while present at `ROADMAP.md:158` since before the 2026-07-27 rebuild | **opened, `READY (S for (a) / M for (b))`**, owner CC |
|
||||
| R-94 | leg (b) stood alone | leg (b) carries `→ R-29` as its class. Row otherwise untouched |
|
||||
| E-2d | cited `:322-341` for an invocation at `:343` | cited `:322-343`, with the fetch / args / call each pinned separately |
|
||||
| R-95, R-86, R-87 | `READY #1`, `READY #3`, `READY #4` | `READY` — markers dropped (see below) |
|
||||
|
||||
Not added to the ranked list under *"Why the READY rows rank this way"*: that list is a top-N
|
||||
rationale, not a complete ordering (R-99, R-102, R-103, R-104, R-108, R-109, R-89, R-92, R-93 and
|
||||
E-2d are all READY and absent from it). Ranking R-29 was not asked for and would be a priority claim
|
||||
this task has no basis to make.
|
||||
|
||||
### §2.2 — markers dropped, not renumbered
|
||||
|
||||
The `#1/#3/#4` markers duplicated ranked-list positions 1/3/4 for exactly those three rows; two
|
||||
orderings of one set is what produced the hole when the `#2` row was merged in `d4c07873`. Removing
|
||||
the duplicate leaves the ranked list as the single maintained ordering.
|
||||
|
||||
### The other two record defects
|
||||
|
||||
- `ROADMAP.md:147` — cited a non-existent **R-164**; it means controller **v0.164.0**'s
|
||||
deliberate-stop filter. Corrected. (It had already cost one max-ID scan a false positive.)
|
||||
- `CONTEXT.md:564` — asserted in the present tense that the single VERSION source is *"gated by
|
||||
`scripts/hostinstall_gates.py`"*. The gate exists, asserts exactly that, is red, and runs nowhere.
|
||||
Corrected to say the cross-check exists but is not enforced, tracked as R-94 leg (b) / R-29.
|
||||
`hub/internal/web/configs.go:27` carries the same false claim in a code comment and was left alone
|
||||
— R-94 leg (b) territory, needs a hub build.
|
||||
|
||||
Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
|
||||
`documentation/backlog/README.md`, `CONTEXT.md`, this file.
|
||||
|
||||
---
|
||||
|
||||
## Commit 3 — `de5a3e5` — the record-hygiene rider
|
||||
|
||||
Baseline `7383400a`, unmoved. Four XS items from `7383400a` §9; the three deferred observations stay
|
||||
deferred.
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `CONTEXT.md:540` | *"`scripts/hub_confirm_gate.py` enforces"* → asserts, but is not enforced (R-29). Third instance of the class after `:564` and `configs.go:27` |
|
||||
| `REUSE.md:62` | Same claim, *"enforces zero"*. The **rule stays** — never native `confirm()`/`prompt()` is correct guidance and this is a reuse-reference row — only the enforcement claim changes |
|
||||
| `OPEN-ITEMS.md:4` | Root `REPORT.md` = overwritten per-session; `REPORT-<topic>.md` = non-clobbering sibling (`CLAUDE.md:82-87`), 14 of them. Prohibition unchanged |
|
||||
| `OPEN-ITEMS.md:55` | Heading scoped to *"the **TOP** READY rows"* + a half-sentence that it is deliberately not a full ordering. **No row added to the list** |
|
||||
|
||||
`hub/internal/web/configs.go:27` — the fourth instance — left alone (R-94 leg (b), needs a hub build).
|
||||
|
||||
### Part 3 NOT done — its stated evidence is false
|
||||
|
||||
The spec proposed recording a convention: *"`ROADMAP`'s P3 `Status` column only ever takes `idea` or a
|
||||
`SHIPPED`/`PROVEN-LIVE` variant, and never takes `READY`, `BLOCKED` or `WAITING-ON-OPERATOR`"*, and
|
||||
concluded from that vocabulary split that `ROADMAP` `Status` is **disposition** while `OPEN-ITEMS`
|
||||
`State` is **live actionability**.
|
||||
|
||||
Refuted inside P3 itself (`ROADMAP.md`, lines 113–172):
|
||||
|
||||
| Line | ID | P3 `Status` cell |
|
||||
|---|---|---|
|
||||
| 151 | R-90 | `BLOCKED on Hetzner CX33 availability (operator, 2026-07-27)` |
|
||||
| 152 | R-91 | `WATCHING — gated on demo-felhom's first post-migration PBS backup` |
|
||||
| 150 | R-110 | `idea — found 2026-07-29, **WAITING-ON-OPERATOR (a ruling, not a defect)**` |
|
||||
| 137 | R-86 | `**NEXT — operator ruling 2026-07-27**` |
|
||||
|
||||
And P2's recovery-gaps sub-table (lines 99–107) uses a bare `READY` for nine rows, including `E-2`.
|
||||
|
||||
Both halves fail: the column **does** take the `OPEN-ITEMS` vocabulary, and `BLOCKED`/`WATCHING`/
|
||||
`NEXT` are live-actionability words, not dispositions. The real shape is a free-text status field
|
||||
that has absorbed both vocabularies over time — usually disposition, sometimes not. Writing the
|
||||
proposed convention down would install a **new false invariant** in the commit chain whose whole
|
||||
purpose was removing them, so per the spec's own instruction (*"If you think this ruling is wrong, say
|
||||
so and make no edit"*) `documentation/backlog/README.md` was not touched.
|
||||
|
||||
The narrow question that prompted it still resolves benignly: R-29 and R-94 reading `idea` in
|
||||
`ROADMAP` and `READY` in `OPEN-ITEMS` is **not** a defect and needs no sync. What is unsupported is
|
||||
generalising that into a rule.
|
||||
|
||||
---
|
||||
|
||||
## Standing note
|
||||
|
||||
**No gate was wired, fixed, run as a hook, or deleted by any of the three commits.** Filing R-29 is
|
||||
not doing R-29; the enforcement decision (pre-push hook / `build.sh` step / CI) and the audit of the
|
||||
remaining eight gates are R-29 part (b), M-sized, and its own task.
|
||||
@@ -1,94 +0,0 @@
|
||||
# REPORT — Session C: R-113, R-114, R-112 proven on a real box; C5 fails on a new defect (2026-07-29)
|
||||
|
||||
`RUNBOOK-session-c-2026-07-29.md`. Full evidence: `documentation/audits/SESSION-C-2026-07-29.md`.
|
||||
Root `REPORT.md` untouched.
|
||||
|
||||
## Verdicts
|
||||
|
||||
| Claim | Fix | Verdict |
|
||||
|---|---|---|
|
||||
| **C4** — offer appears and moves the target | R-112 | ✅ **PASS** |
|
||||
| **C3** — absent target tells the truth, offers nothing | R-114 | ✅ **PASS** |
|
||||
| **C5** — `backup_target_absent` + `backup_target_restored` | R-113 | ❌ **FAIL** — generic alarm, specific recovery → **R-116** |
|
||||
|
||||
**All three shipped fixes work.** R-113's gate fires in **4 seconds** (E-2d measured zero over 4½
|
||||
minutes). R-114's message is correct. R-112's banner reaches the customer. C5 fails on a **fourth,
|
||||
separate defect that was unreachable until R-113 made the gate fire at all.**
|
||||
|
||||
## What the drill box ran
|
||||
|
||||
Agent **0.114.0 from the Day-0 manifest** — the shipped binary, so C5 tested the real artifact and
|
||||
closed R-115's observation 1 for R-113. Controller **0.186.0 hand-deployed** after install (§3.1
|
||||
ruling (a)); the vouched golden bakes 0.185.1, so **C3/C4 prove the code, not the shipped golden** —
|
||||
that lag is filed against R-115, not a new ID.
|
||||
|
||||
## The three headline observables
|
||||
|
||||
**R-113** — detach at 18:43:50, gate at **18:43:54**, on exactly the shape that defeated it before:
|
||||
```
|
||||
raw /mnt/mentes : NOT mounted
|
||||
bind /mnt/felhom-drives/mentes : /dev/sdb[/felhom-data] <- the stale bind SURVIVED
|
||||
```
|
||||
|
||||
**R-114** — with the target absent: absent copy 1, **system-disk copy 0**, **offer block 0**. Both of
|
||||
E-2d's falsehoods gone.
|
||||
|
||||
**R-112** — banner element 1 and the never-configured copy in the HTML; after the wizard, the offer
|
||||
with `data-path="/mnt/felhom-drives/mentes"`. Healthy renders nothing, **proven positively**: idle
|
||||
delta 0 `/backup/tiers` calls, page-load delta **+1**, single caller ⇒ the seam ran and chose silence.
|
||||
|
||||
**Decline path proven** (registration confers no role), `restart_required:true`, agent did **not**
|
||||
self-restart, in-flight check recorded before I restarted it, wrapper created the storage at the
|
||||
drive's own mountpoint.
|
||||
|
||||
## C5's failure
|
||||
|
||||
```
|
||||
absent : Event pushed: storage_disconnected (error) <- GENERIC
|
||||
return : Event pushed: backup_target_restored (info) <- SPECIFIC
|
||||
```
|
||||
|
||||
`backup_target_absent` count **0** across the whole run. The alarm and its recovery cannot be matched
|
||||
— precisely what `notifyDriveReturned`'s own comment forbids.
|
||||
|
||||
**Root cause (R-116):** `driveTargetByPath` builds `out[GuestPath] = d.BackupTarget`, but the drive is
|
||||
**two `/disks` rows** and the flag and the guest path sit on different ones — the `felhom-backup`
|
||||
storage row carries `BackupTarget: true` and gets a guest path only while classified user-data; the
|
||||
registry union row carries the guest path and **never assigns `BackupTarget`**. Absent ⇒ they separate
|
||||
⇒ generic. Return ⇒ they rejoin ⇒ specific. v0.184.1 fixed the *keying*, not this.
|
||||
|
||||
## Mirror + over-correction guard — PASS, with a caveat
|
||||
|
||||
Non-target drive detached ⇒ `storage_disconnected`, `backup_target_absent` count 0. **Over-correction
|
||||
guard passes**: both drives present ⇒ 0 ABSENT lines, target stayed healthy — R-113's stricter presence
|
||||
did not make a healthy drive read absent. **Caveat: the mirror passes trivially**, because the target
|
||||
also produced the generic event; it confirms no over-correction but cannot confirm discrimination.
|
||||
|
||||
## Record
|
||||
|
||||
- `OPEN-ITEMS.md` — **R-113, R-114, R-112 → SHIPPED + PROVEN-LIVE**; **R-116 opened** (READY (S), P1);
|
||||
**E-2 and E-2d CLOSED as partially proven** with R-116 as the one named open leg, per the runbook's
|
||||
§9 decided-in-advance rule.
|
||||
- `ROADMAP.md` — R-116 under P1.
|
||||
- **Capability map NOT touched** — it still has **no E-2 / backup-target rows at all**, so no row could
|
||||
be moved to PROVEN-LIVE. Creating them is a design act, not a validation act. Third session running
|
||||
that this has been noted.
|
||||
|
||||
## Teardown
|
||||
|
||||
VM destroyed, storage removed, **`pvesm status` after == before** (`local-lvm` 38.78 %), guest 9201 and
|
||||
`drill-r50` untouched. **Customer ruling: DELETE**; attempted and correctly refused (`host … is
|
||||
ONLINE`) — deletable once the destroyed host ages to DOWN (>1 h), command recorded in `OPEN-ITEMS.md`.
|
||||
|
||||
## What did not happen
|
||||
|
||||
`backup_target_absent` never fired, so its severity, Hungarian copy and hub routing remain unexercised
|
||||
— R-116 blocks them. The offer was accepted via the endpoint the button POSTs, not a browser click (no
|
||||
browser automation on DooPlex); the rendered control and its non-auto-submission were verified in HTML.
|
||||
The stale bind still naming a dead device node after return was observed, not investigated.
|
||||
|
||||
## The arc
|
||||
|
||||
E-2 ends here. Its stated definition of done is **R-106 + R-109, R-108 and D5** — none of which this
|
||||
detour touched. The detour was worth taking: it found six real defects (R-111 through R-116), four of
|
||||
them customer-affecting, none of which any unit suite had caught.
|
||||
@@ -1,98 +0,0 @@
|
||||
# REPORT — tester gate: golden re-baked to 0.188.0, fresh-install proof PASSED (2026-07-31)
|
||||
|
||||
Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so the shared `REPORT.md` (E-2 increment 1) is
|
||||
not clobbered. Full record with every observable: `documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md`.
|
||||
|
||||
## Outcome
|
||||
|
||||
**§7.2 — YES: a fresh install is safe to hand to an external tester.** ISO boot → claimable,
|
||||
app-serving box in ~10 minutes unattended, and an app's data restored **from the drive with the
|
||||
guest's `app.yaml` gone**, proven readable by the application over its own TCP path.
|
||||
|
||||
**Golden 0.186.0 → 0.188.0** baked, published, vouched. **No ISO rebuilt** — Part 0 proved none was
|
||||
needed. No existing box changed; floor still v0.156.0, MinAgent still 0.113.0.
|
||||
|
||||
## Part 0 — the ISO does not need rebuilding
|
||||
|
||||
Verified against the ISO **on disk**, not from source. It bakes exactly three Felhom payloads
|
||||
(`felhom-bootstrap.sh`, its unit, the secret-free pairing env) — full-base64 match, 1 hit each — and
|
||||
**0** hits for `SCRIPT_VERSION="1.2`, `felhom-controller`, `vzdump-lxc-9100`. The installer is fetched
|
||||
at run time (`felhom-bootstrap.sh:96`) and the live URL is byte-identical to repo HEAD
|
||||
(sha `ab8b283e…`, v1.22.0, committed six days *after* the ISO). The golden arrives via the hub-vouched
|
||||
artifact manifest (`felhom-host-install.sh:423-433`). The one genuinely baked, drift-capable thing is
|
||||
`felhom-bootstrap.sh` itself — currently at repo HEAD.
|
||||
|
||||
**Proven live**, not just argued: the fresh box ran `felhom-host-install v1.22.0` and fetched golden
|
||||
**v0.188.0**, sha-verified.
|
||||
|
||||
## Part 1 — bake / publish / vouch
|
||||
|
||||
Baked **0.188.0**, not the brief's 0.187.0: 0.187.0 lacks D5, and Part 2 step 6 *is* the D5 claim, so
|
||||
that golden could not have passed the proof this task exists for. 0.188.0 satisfies R-120 anyway.
|
||||
Stated rather than absorbed, per standing rule 4.
|
||||
|
||||
GOLDEN_VERSION=0.188.0
|
||||
GOLDEN_SHA256=7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299
|
||||
|
||||
Three observables: 404 pre-gate (with a 200 control on 0.186.0 so it is not vacuous), then an
|
||||
**anonymous** download returning `http=200 bytes=649310288` and a matching sha; the manifest read back
|
||||
showing `0.188.0` selected; and the consumer call `GET https://hub.felhom.eu/api/v1/artifacts/sess-g`
|
||||
returning the pair. Plus a fourth: a real fresh box fetched and sha-verified it.
|
||||
|
||||
**R-120's gate evaluated and allowed.** Exercised both ways rather than inferred from silence —
|
||||
vouching 0.185.1 first produced `flash=golden_behind_fleet`, the logged `artifact vouch REFUSED`, and
|
||||
**no write** (the manifest still read 0.186.0); then 0.188.0 produced
|
||||
`Artifact manifest set: agent=0.118.1 golden=0.188.0`.
|
||||
|
||||
## Part 2 — the clean-install proof, on demo-hp
|
||||
|
||||
All seven steps PASS. Venue was demo-hp (Tier 0, the designated drill host) using the scratch dir
|
||||
storage at `/mnt/nvme-1tb` that `target-selection.md:38-40` names; `local-lvm`, `drill-r50` and both
|
||||
9201s untouched.
|
||||
|
||||
Highlights: real day-0 pairing → bind → install; a **real** claim (the code is emailed-only, R-119 —
|
||||
the operator relayed it), with the gate flipping `dashboard not yet claimed` → `authentication
|
||||
required`; controller **0.188.0** confirmed *from the box*; **rallly** (postgres) + **homebox**
|
||||
deployed through the real endpoints.
|
||||
|
||||
The D5 leg: recovery unit `portable-carried=2/2, withheld=0`; the carried `DB_PASSWORD` matched the
|
||||
live one **by fingerprint** (`14c8f515…`, never printed); guest `app.yaml` moved aside; restore
|
||||
returned `secrets recovered=2/2`. Step 7 read the data from **rallly's own network namespace** over
|
||||
TCP to `rallly-postgres` — not the localhost trust socket that produced D5's false pass — and the same
|
||||
path with a wrong password returned `FATAL: password authentication failed`, proving the credential
|
||||
does real work. **The discriminator held: PRE-BACKUP row = 1, POST-BACKUP row = 0.**
|
||||
|
||||
## Part 3 — runbook integrity
|
||||
|
||||
`RUNBOOK-manual-build.md` told the reader to use a "RECORDED" qemu line that is itself labelled
|
||||
*reconstructed*, and whose source says it *"was never saved"*. The real invocation is now captured
|
||||
from this bake and recorded as canonical in **§4.0**, alongside the bake/publish/teardown steps, the
|
||||
template-rot warning and where the R-120 gate actually lives. The old runbook's deviation entry is
|
||||
marked SUPERSEDED with a forward pointer.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
1. VM 310 destroyed with `--purge --destroy-unreferenced-disks 1`; `/mnt/nvme-1tb/images/` empty.
|
||||
2. `cc-scratch` removed, `storage.cfg` back to its original four entries; `felhom-backup` available
|
||||
**926 492 284 KiB before and after** — space returned exactly.
|
||||
3. Hub: **`sess-g` and its host record DELETED, full cascade** — `customer DELETE cascade COMPLETE
|
||||
for sess-g (journal #8) — full teardown`, residue purged including `appliance_registrations=1`.
|
||||
Verified positively: `/configs` and `/hosts` both loaded (10160 / 9880 bytes) with **0** hits for
|
||||
`sess-g` and 0 for the appliance UUID. The gate refused twice first (409 host ONLINE, then 400
|
||||
missing acknowledgements) — the record cannot be deleted until the destroyed box ages out of
|
||||
ONLINE, ~30 min. **`sess-f` deliberately NOT deleted** (R-131); its command is in the audit §7.1.
|
||||
Secrets shredded in the guest and on the box.
|
||||
|
||||
## Findings — filed, none fixed
|
||||
|
||||
`R-128` ISO_VERSION/SCRIPT_VERSION comment is false · `R-129` demo-hp's "no baked SSH key" is stale
|
||||
(key auth works) · `R-130` `HARD_MIN_LVM_GIB` warns and proceeds — a hard min that is not hard ·
|
||||
`R-131` `sess-f` is a fourth orphaned scratch customer · **`R-132` — `curl -w '%{redirect_url}'`
|
||||
printed the hub operator password into a session transcript; `HUB_PW` needs rotating.**
|
||||
|
||||
## Not done, deliberately
|
||||
|
||||
No ISO built; no defect fixed; no golden deployed to an existing box; no floor or MinAgent change;
|
||||
offsite/PBS-DR legs not exercised (the task forbids pointing anything at production PBS or the real
|
||||
restic offsite, so `sess-g` ran DR-tier off); Campaign 10, the demo-hp repartition and subdomain
|
||||
onboarding untouched.
|
||||
@@ -1,97 +0,0 @@
|
||||
# REPORT — SPIKE 4: can a `.deb` in the ISO deliver the stub on an interactive install? (2026-07-31)
|
||||
|
||||
> Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub
|
||||
> v0.85.0 session and the second session in a shared clone never touches it. Supersedes this file's
|
||||
> Spike 1–3 contents.
|
||||
|
||||
**Class: Spike.** Findings only — no production file changed, no release ISO built, nothing published.
|
||||
Evidence: `documentation/audits/SPIKE-universal-iso-4-2026-07-31.md`.
|
||||
|
||||
## The answer is yes, and it was measured with the negative control in the same box
|
||||
|
||||
One ISO, 15 GRUB entries, a trivial probe `.deb` injected into `/proxmox/packages/`. Two VMs on
|
||||
demo-hp built with `qm` so the run was visible in the web console: **400 interactive**, **401
|
||||
automated control**.
|
||||
|
||||
On the **interactive** install (`spikefour.felhom.eu`):
|
||||
|
||||
- the package is installed — `ii felhom-spike4-probe 0.0.1`
|
||||
- its **postinst ran** — marker file present, content intact
|
||||
- it **enabled a systemd unit**, and **that unit fired on first boot** (uptime 7.98 s, `pid1: systemd`)
|
||||
- and on **that same machine**, `proxmox-first-boot` is not installed and `/var/lib/proxmox-first-boot`
|
||||
does not exist — Spike 3's negative reproduced, not assumed
|
||||
|
||||
So the two delivery mechanisms are independent, and the one that survives the path we are actually
|
||||
shipping is the `.deb`. **The product — insert the stick, install Proxmox normally choosing your own
|
||||
disk and password, box sets itself up and waits for a claim code — is now measured rather than hoped
|
||||
for.** With one honest caveat: what was measured is a trivial probe package, not Felhom's real stub.
|
||||
Packaging the real stub and confirming pairing end-to-end is the last step before a spec (~60 min).
|
||||
|
||||
## What a postinst may and may not do
|
||||
|
||||
Identical on both paths: `pid1 = unconfigured.sh`, **no running systemd**, `/proc` and `/sys` mounted,
|
||||
and **`systemctl enable` succeeds** (it wrote the symlink). Network and DNS *happened* to be up —
|
||||
inherited from the installer's own DHCP.
|
||||
|
||||
Four constraints for the real postinst, so they get written against rather than discovered:
|
||||
|
||||
1. Never `systemctl start` or `daemon-reload` — there is no systemd running. `enable` is the only verb.
|
||||
2. **Never require the network**, despite it being present here. A box installed with the cable out
|
||||
gives a postinst no route, and a failing postinst breaks the customer's install.
|
||||
3. Never fail — guard everything, `exit 0`.
|
||||
4. Do the real work in the unit at first boot, where systemd, network and a booted kernel exist.
|
||||
|
||||
## Two smaller results
|
||||
|
||||
**The repack preserves the `.deb`, but not naively.** `xorriso … -boot_image any replay` fails with
|
||||
*"Overlapping MBR partition entries"* — and `iso-repack.sh:270-292` already documents that exact
|
||||
failure and its fix. Mirroring it produced a working image (19 El Torito entries; the `.deb` extracted
|
||||
back out is byte-identical). So this is an insertion into an extract→modify→re-master cycle our repack
|
||||
already performs, not a new build stage.
|
||||
|
||||
**Q3:** `iso-repack.sh:100-106` refuses an ISO without `auto-installer-mode.toml`. It is a guard, not
|
||||
a structural requirement, and its reasoning is sound for the shape it was written for — already R-155,
|
||||
cited exactly here. With no mode file the stock grub.cfg does not emit the Automated entry at all; with
|
||||
a mode file but no answer, that entry aborts safely and loudly.
|
||||
|
||||
## A correction I owe you from last session
|
||||
|
||||
**R-153 is retracted.** The register grep this task mandated shows R-94 already carries it verbatim at
|
||||
`OPEN-ITEMS.md:15`, status `READY (XS)`, with leg (b) being precisely "the gate fails today and is
|
||||
invoked by nothing" — and R-29, the class, says in terms *"do not mint a new ID for a new instance."*
|
||||
Spike 3 filed a duplicate.
|
||||
|
||||
**And the substantive half of that Spike 3 claim was wrong.** I wrote that the drift left the customer
|
||||
page's install-command generator "targeting a flag surface three minor versions stale." R-94 explicitly
|
||||
retracts exactly that reading: the constant selects no script — it renders as a text label, and the
|
||||
command beneath it fetches the script the website git-syncs from `main`, so **1.22.0 is what every
|
||||
install already gets**. It is a wrong number on your screen and nothing more. I overstated it.
|
||||
|
||||
## Still unknown
|
||||
|
||||
**The real stub has not been packaged** — that is the one thing between here and a build spec.
|
||||
Also unproven: `dpkg --configure -a` ordering for a package with dependencies; an ISO that never went
|
||||
through `prepare-iso` (blocked by R-155, which this spike was fenced from changing); and the Graphical
|
||||
installer, where the result should hold *a fortiori* since the `.deb` path is in `Install.pm`, shared
|
||||
by all front-ends — but that is inference, not measurement.
|
||||
|
||||
**Spike 3's Q3** — the real stub at `before-network` — **this session did not touch it.** Note it is
|
||||
now partly superseded: on the `.deb` route the unit's ordering comes from the unit file, not from
|
||||
`[first-boot].ordering`.
|
||||
|
||||
## R-rows
|
||||
|
||||
**None opened.** Each candidate was grepped against the register first: the delivery result is a
|
||||
positive finding, the postinst constraints belong in the build spec, and the repack guard is already
|
||||
R-155. **R-153 retracted** into R-94 leg (b) / R-29.
|
||||
|
||||
## Teardown
|
||||
|
||||
All three layers plus the scratch storage, verified positively. demo-hp: VMs 400/401 purged, **storage
|
||||
`spike4` removed** (`storage.cfg` back to 4, `grep -c spike4` = 0), `/mnt/nvme-1tb/images/` empty,
|
||||
**disk usage 6.6 G — identical to pre-spike**, probe ISO and driver removed, 0 loop devices,
|
||||
`drill-r50` stopped and untouched, 9201 running, `felhom-backup` unmodified, nothing on `local-lvm`.
|
||||
DooPlex: workspace scratch **4.8 GB removed**, scratchpad **3.3 GB → 88 K**, both throwaway passwords
|
||||
destroyed, 17 ISOs in `out/` untouched, no production file modified. **Hub-side: nothing created** —
|
||||
the VMs took LAN DHCP leases but never ran `felhom-host-install.sh` or contacted the hub; verified by
|
||||
fetching and searching the customer list. Nothing published.
|
||||
@@ -1,216 +1,200 @@
|
||||
# REPORT — R-181 (the reserve guards the write that fills the disk) + R-156 (papra) + two operator rulings
|
||||
# REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed
|
||||
|
||||
**Date:** 2026-08-03 · **Repos:** `felhom-controller` (v0.192.0 → **v0.193.1**), `app-catalog-felhom.eu`, `felhom.eu` (docs only — **no hub change, no hub version bump**)
|
||||
**Date:** 2026-08-03 · **Repos:** `felhom.eu` (installer **v1.22.0 → v1.23.0**), `felhom-agent` (**no bump**)
|
||||
**Nothing was built** — no image, no binary, no golden. **Hub stays v0.89.0.**
|
||||
|
||||
## 1. Baselines — re-read on arrival, all matched §1
|
||||
## 1. Baselines — re-read on arrival, both matched §1
|
||||
|
||||
| Repo | `main` @ arrival | Version | Shipped |
|
||||
| Repo | @ arrival | Version | Result |
|
||||
|---|---|---|---|
|
||||
| `felhom-controller` | `4be6467b501b` | v0.192.0 | **v0.193.0 `fef07c3`** → **v0.193.1 `6c43bf6`** |
|
||||
| `app-catalog-felhom.eu` | `7cb58ecdf8e7` | n/a | `122bbee` |
|
||||
| `felhom.eu` | `6b5d64c1fa73` | hub v0.89.0 | docs only, **no bump** |
|
||||
| `felhom.eu` | `8360f940bfb2` | hub v0.89.0, `SCRIPT_VERSION="1.22.0"`, **0 tags** (confirmed) | installer **v1.23.0**, first tag `installer-v1.23.0` |
|
||||
| `felhom-agent` | `9dfd89cb947e` | v0.120.0 | **unchanged** — scripts and gates only |
|
||||
|
||||
All three clean (`git status --porcelain` empty, `HEAD == origin/main`) before every build.
|
||||
## 2. Part 0 — the R-182 measurement, and it REVERSED the row
|
||||
|
||||
## 2. The fix
|
||||
Filed yesterday as *"the reserve re-alerts on every status refresh"* — **too many** alerts, observed
|
||||
at the sending end. Measured at the **receiving end**, it is the opposite.
|
||||
|
||||
**One admission verdict per app per run** (`controller/internal/backup/admission.go`), taken before
|
||||
that app's **first** write and consulted by all three legs — DB dump, volume dump, unit capture. The
|
||||
three write under one per-app root (`appbackup.RecoveryUnitPath`), which is what makes one verdict
|
||||
able to cover them honestly.
|
||||
Method: the hub's SQLite copied **with its `-wal`** (4 MB and newer than the db — copying `hub.db`
|
||||
alone would have read stale data, the exact trap this project recorded before), freshness confirmed by
|
||||
the newest `notification_log` row post-dating the session.
|
||||
|
||||
- **Lazy, not run-wide.** App A's dump can put app B under the reserve; a run-start verdict reads a
|
||||
disk that no longer exists. **Never re-decided between an app's own legs** — that is the split being
|
||||
closed. **Reset per run.**
|
||||
- **Ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act, so a refused app is never
|
||||
bounced. **After** the volume-less check, which has no write to gate.
|
||||
- **Exactly one operator alert per refused app per run.** Leg order unchanged.
|
||||
- **Size term added:** *would this app's write cross the reserve?* — estimated from its previous
|
||||
`.sql` + `.tar`. **No history → headroom-only**, or the first backup becomes the one that can never
|
||||
happen; the alert says so when that applies.
|
||||
**9 `recovery_unit_capture_failed` events received today → 2 operator emails sent.**
|
||||
|
||||
## 3. Files
|
||||
| time | apps refused (events in) | operator emails out |
|
||||
|---|---|---|
|
||||
| 06:40:03 | privatebin, opengist | **opengist only** |
|
||||
| 08:59:46/47 | opengist, privatebin | **privatebin only** |
|
||||
| 08:59:59 | privatebin, opengist | **none** |
|
||||
| 09:03:00 | opengist | **none** |
|
||||
| 09:07:06 | privatebin, opengist | **none** |
|
||||
|
||||
| File | |
|
||||
**Cause, confirmed at source:** the operator cooldown key is
|
||||
`customerID + ":" + eventType + cooldownTierSuffix(details)` (`dispatcher.go:268`, 1 hour hardcoded).
|
||||
`RecoveryUnitFailureDetails` carries **`app`** and **no `tier`**, so the suffix is empty and the key
|
||||
holds **no app identifier**. The first refused app takes the slot; every other app's refusal for the
|
||||
next hour is dropped — and dropped **before `LogNotification`**, so it leaves **no row on any
|
||||
channel** and cannot be audited afterwards.
|
||||
|
||||
This is **R-97a's failure mode in a second event type**; that row's own comment states it
|
||||
(*"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20"*). `cooldownTierSuffix` was
|
||||
written narrow on purpose; `recovery_unit_capture_failed` simply never opted in.
|
||||
|
||||
**A correction I owe on yesterday's report.** It said *"one `recovery_unit_capture_failed` per app,
|
||||
HTTP 200"*. That was true of what the **controller pushed**, and a reader would take it as *the
|
||||
operator was told about each app* — which is false. The gap between an accepted event and a sent
|
||||
email is the whole of this row.
|
||||
|
||||
**Nothing was changed** (§8.5). R-182 is re-scoped with the evidence and the fix shape.
|
||||
|
||||
## 3. Probes
|
||||
|
||||
| | Question | Method | Verdict |
|
||||
|---|---|---|---|
|
||||
| **P-A** | does git-sync v4.4.0 follow a tag, and notice a **moved** one? | throwaway `docker run` git-sync against this repo, tag moved under it | **PASS both halves** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period (~20 s) |
|
||||
| **P-B** | does Gitea serve `raw/tag/<tag>/<path>`? | one fetch on a throwaway tag | **PASS** — HTTP 200, byte-identical to `raw/branch/main` |
|
||||
| **P-C** | can CI read the package registry? | anonymous fetches | **PARTIAL, and it changed the gate's design** — package **download** 200 (and **404** for a fake version, so it discriminates), **tags** api 200; package **listing** api **401**, hub artifact manifest **401** |
|
||||
|
||||
**Publish model P-A implies:** publishing is **moving the tag**; rollback is **moving it back**, in
|
||||
~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone
|
||||
(`git ls-remote --tags` → 0 at the time).
|
||||
|
||||
## 4. §8.2's three channels — enumerated
|
||||
|
||||
| Channel | Before | After | |
|
||||
|---|---|---|---|
|
||||
| 1. the served script | `main`, 30 s | **`installer-v1.23.0`** | **MOVED** — `webpage.yaml` split into two syncs |
|
||||
| 2. the run-time fetches | `raw/branch/main` | **`raw/tag/v$ART_AGENT_VER`** | **MOVED** — but see below |
|
||||
| 3. the URL producers | `main` | unchanged | **NO CHANGE NEEDED** — and that is a finding, not an omission |
|
||||
|
||||
**Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable.**
|
||||
There are **sixteen** fetches, not nine, and they come from **`felhom-agent`**, not `felhom.eu` — so
|
||||
no tag on this repo could ever have covered them, and §8.1's *"derive the tag from `SCRIPT_VERSION`"*
|
||||
was impossible for them. Raised before building; operator ruled to pin them to **the agent version
|
||||
being installed**, which the installer already resolves from the hub manifest and already sha-verifies.
|
||||
That is strictly better than any installer-derived tag: binary and configs now come from one ref.
|
||||
|
||||
**Channel 3 needed no change because the URL never carried a ref** —
|
||||
`https://felhom.eu/scripts/felhom-host-install.sh` is path-based; the ref lives in the manifest. So
|
||||
`felhom-bootstrap.sh` and the hub's day-0 command follow the tag automatically. **No hub template
|
||||
change ⇒ no hub bump**, so §1's rule was never in tension and the STOP it anticipated never arose.
|
||||
|
||||
## 5. The tag convention
|
||||
|
||||
- **Shape:** `installer-v<SCRIPT_VERSION>` in `felhom.eu` (prefixed so it cannot be read as a hub,
|
||||
agent, controller or golden version); `v<semver>` in `felhom-agent` (that repo versions one thing).
|
||||
**No new constant in the installer** — channel 2 derives its ref from `$ART_AGENT_VER` at run time,
|
||||
and channel 1's ref lives only in the manifest.
|
||||
- **Publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml` (sidecar *and*
|
||||
init container), commit, sync.
|
||||
- **Roll back:** move the tag back to the previous commit — takes ~30 s, **no ArgoCD sync, no deploy**.
|
||||
|
||||
## 6. Scenario A — proven by HTTP
|
||||
|
||||
A real commit was pushed to `main` (a marker comment in the installer) **without moving the tag**, and
|
||||
three sync periods were allowed to pass so "unchanged" means "had every chance to change":
|
||||
|
||||
```
|
||||
website tree (main): .worktrees/6a82719… <- ADVANCED to the new commit
|
||||
scripts tree (tag): .worktrees/bee6848… <- STAYED
|
||||
sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
|
||||
sha256 after push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
|
||||
marker present at the served URL? 0
|
||||
https://felhom.eu/ -> HTTP 200
|
||||
```
|
||||
|
||||
Both halves of the split in one observation: the site still tracks `main`, the installer does not.
|
||||
|
||||
## 7. Scenario B — publish and rollback, both directions
|
||||
|
||||
| act | result |
|
||||
|---|---|
|
||||
| `controller/internal/backup/admission.go` | **new** — the gate, the memo, the estimator |
|
||||
| `controller/internal/backup/admission_test.go` | **new** — 11 tests |
|
||||
| `controller/internal/backup/backup.go` | run scope + gates in the DB and volume legs |
|
||||
| `controller/internal/backup/recovery_unit.go` | `floorVerdict` size-aware; capture leg via `admitApp` |
|
||||
| `controller/internal/backup/capture_floor_test.go` | 3 call sites updated for the new signature |
|
||||
| `controller/README.md`, `REUSE.md`, `CHANGELOG.md` | |
|
||||
| `app-catalog-felhom.eu/templates/papra/docker-compose.yml` | mount moved to `/app/app-data` |
|
||||
| tag moved `bee6848 → 6a82719` | scripts tree moved in **~40 s**; served `sha256 ea2b4aa9…`; **marker present** |
|
||||
| tag moved back `→ bee6848` | scripts tree back in **~40 s**; served `sha256 2f859555…` — **exactly** the pre-publish sha; **marker gone** |
|
||||
|
||||
## 4. Tests — 28 packages `ok`, `rc=0` (read separately from any commit)
|
||||
`https://felhom.eu/` returned 200 throughout. The marker commit was then reverted, and the tag moved
|
||||
to `main`'s head — a **byte no-op**, verified by the served sha not changing.
|
||||
|
||||
All 11 new tests pass, plus the pre-existing floor suite. Refusal assertions are **sha256 tree
|
||||
fingerprints before and after**, never log lines — the defect being fixed *is* a log line the tree
|
||||
contradicted.
|
||||
## 8. Files, commits, tags
|
||||
|
||||
The DB leg cannot run without Docker (`DiscoverDatabases` shells out), so its gate is pinned by an
|
||||
**AST walk** of `backup.go` asserting `admitApp` precedes `DumpOne`. `strings.Contains` is
|
||||
insufficient: a commented-out call still contains the string.
|
||||
**`felhom.eu`** — `bee6848` (installer + gate + manifest), `6a82719` (Scenario A marker), `e79a20b`
|
||||
(marker removed), plus the docs commit below.
|
||||
`scripts/felhom-host-install.sh` · `scripts/hostinstall_gates.py` · `scripts/CHANGELOG.md` ·
|
||||
`manifests/webpage.yaml` · `CLAUDE.md` · `CONTEXT.md` · `STATUS.md` · `REPORT.md` ·
|
||||
`documentation/backlog/{OPEN-ITEMS,ROADMAP}.md` · `documentation/architecture/00-capability-map.md`
|
||||
|
||||
### Red-proofs — each demonstrated failing, then restored
|
||||
**`felhom-agent`** — `dd2d1fe` (release path + gate + CI), `0db7766` (REPORT).
|
||||
`scripts/release-agent.sh` **(new)** · `scripts/check-published-versions.py` **(new)** ·
|
||||
`scripts/agent_gates.py` · `.gitea/workflows/gates.yml` · `CLAUDE.md` · `CHANGELOG.md` · `REPORT.md`
|
||||
|
||||
**Tags created:** `felhom.eu/installer-v1.23.0` (the first tag this repo has ever had) and
|
||||
`felhom-agent/v0.120.0` (retroactive, at `cd6e267` — the commit the published binary was built from;
|
||||
`configs/` is byte-identical there and at `main`, so nothing depended on the choice).
|
||||
|
||||
## 9. Tests and red-proofs
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `felhom.eu` `repo_gates.py --fast` | all 5 gates OK |
|
||||
| `felhom-agent` `go build ./... && go vet ./...` | OK |
|
||||
| `felhom-agent` `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
|
||||
| `agent_gates.py --fast` | `published` correctly **SKIPPED** (hook must not fail on a network blip) |
|
||||
| `agent_gates.py` (full) | both OK |
|
||||
|
||||
**Red-proofs, each demonstrated failing then restored:**
|
||||
|
||||
| # | Mutation | Result |
|
||||
|---|---|---|
|
||||
| 1 | **Both** dump-leg `admitApp` gates removed (= exactly v0.192.0) | Scenario A **RED** — *"the VOLUME leg ran for a refused app"*; with the leg assertions temporarily made non-fatal, the **tree fingerprint changed** too. Also red: Scenario C, Scenario D, and the AST wiring test (which named the DB leg specifically) |
|
||||
| 2 | The entire size term removed from `floorVerdict` (both its thresholds) | Scenario D **RED** — 0 alerts where 1 was required |
|
||||
| 3a | The reserve removed entirely | Scenario F **PASSED — recorded honestly.** The specified mutation does not exercise the assertion: removing the reserve makes every app write, which overwrites and adds but **deletes nothing**, so a deletion-watching test correctly stays green |
|
||||
| 3b | A prune injected into the refusal path | Scenario F **RED** — this is the mutation that proves the test watches deletion |
|
||||
| 4 | Floor moved above the warning band (90% / 6 GiB) | `TestFloorSitsBelowTheCriticalWarningBand` **RED** |
|
||||
| C | one of the sixteen fetches reverted to `raw/branch/main` | **RED** — gate 6a *and* 6b both fired |
|
||||
| D | assertions 6a **and** 6b removed (every guard the test covers), same bad installer | **zero** mentions of the regression — the guards are what catch it |
|
||||
| 6c | the manifest before the split | **RED** on its own, before I fixed it — the gate was demonstrated red by the real pre-change state |
|
||||
| F | `v9.9.9` tagged and not published | **RED**, `binary NOT downloadable (HTTP 404 …)`, rc=1 |
|
||||
| F′ | the gate **deregistered** from `agent_gates.py`, same bad state | **rc=0, "all agent gates OK"** — restored → `CONVICTED: published`, rc=1 |
|
||||
|
||||
Every mutation removed **every** guard its test covers (#1 removed both dump-leg gates, not one).
|
||||
**Scenario F measured on real CI, not inferred.** Runs **69** and **70** are on the *same commit*
|
||||
`0db7766`: **success** before `v9.9.9` existed, **failure** after pushing it. One variable. This also
|
||||
retrospectively explains runs 67/68. **One deliberate CI failure email reached the operator — that was
|
||||
this proof, not an incident.** I could not read CI's own step log: the jobs endpoint needs a Gitea API
|
||||
token, and the only credential available (`~/.docker/config.json`) is a registry password that the API
|
||||
rejects — so the controlled before/after replaced the log rather than an assumption standing in for it.
|
||||
|
||||
## 5. Live validation — demo-hp guest 9201 (Tier 0), the method that found the defect
|
||||
## 10. No version bumps, nothing built
|
||||
|
||||
**Method:** endpoint-level — `POST /api/debug/backup/dbdump`, the exact endpoint the debug UI button
|
||||
calls, which runs the production `RunDBDumps`. No browser on DooPlex.
|
||||
`felhom-agent` **v0.120.0** unchanged (no Go code changed). Hub **v0.89.0** unchanged (no hub file
|
||||
touched). The installer's `SCRIPT_VERSION` **did** go 1.22.0 → 1.23.0 — the installer is not in §12's
|
||||
no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or
|
||||
golden was built.
|
||||
|
||||
**The instrument was re-proven before use.** demo-hp's thin pool is 53.93 GiB, so a real fill of a
|
||||
70 G volume would exhaust it and corrupt every guest. A 5 GiB `fallocate` step moved guest `df`
|
||||
1.2G → 6.2G while thin-pool `data_percent` held **36.83 → 36.83** — zero blocks allocated. Re-checked
|
||||
at every step of the fill.
|
||||
## 11. Register
|
||||
|
||||
### Headroom term — 08:59:46, 906 MB free / 99% used
|
||||
|
||||
| Observable | Result |
|
||||
| ID | Outcome |
|
||||
|---|---|
|
||||
| Tree fingerprint before | `TREE_SHA=111d1760c18d3440f700634ab325f8b8` (10 files; opengist's tar **182,272 B** — R-181's own "before" figure) |
|
||||
| Tree fingerprint after | **`111d1760c18d3440f700634ab325f8b8` — identical** |
|
||||
| Volume dumps written | **0** (baseline run at 08:58 wrote 2) |
|
||||
| `Stopping <app> for safe volume dump` | **absent** — and this is evidence, not an absence, because that line **is** present in the 08:58 baseline |
|
||||
| Operator alerts | one `recovery_unit_capture_failed` per app, severity `error`, HTTP 200 |
|
||||
| **R-110** | **CLOSED — SHIPPED** (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed |
|
||||
| **R-115** | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, no bump) |
|
||||
| **R-182** | **RE-SCOPED — the direction reversed** by Part 0's measurement; still open, now correctly described |
|
||||
| **R-183** | **NEW, and CLOSED the same session** — binary and configs came from two different refs |
|
||||
| **R-184** | **NEW, open** — nothing stops the hub vouching a version that was never released |
|
||||
|
||||
Free space restored → re-run at **09:01:33**: both apps captured normally.
|
||||
**IDs established free:** `^| \*\*R-183\*\*` / `^| \*\*R-184\*\*` in `OPEN-ITEMS.md` → **0 rows** each;
|
||||
all other hits are this session's own code and changelogs (forward references I wrote). `R-185` → 0
|
||||
hits anywhere and remains free.
|
||||
|
||||
### Size term — 09:03:00, proven separately
|
||||
## 12. Observations — noticed, documented, NOT acted on
|
||||
|
||||
Reproducing the original sequence: a real 2 GiB file planted in opengist's volume, backed up so its
|
||||
**previous** tar became **2,147,666,432 B** (the exact live figure), then the filesystem set to
|
||||
**91% used / 2.9 GB free — both headroom terms deliberately clear**.
|
||||
1. **The gate cannot see what is vouched** — filed as R-184 rather than papered over. Closing it needs
|
||||
either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails
|
||||
closed where the mistake is made, needs no new credential).
|
||||
2. **A suppressed operator alert leaves no row at all.** The cooldown returns before `LogNotification`,
|
||||
so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside
|
||||
R-182 because it is what made that row take a day to get the right way round.
|
||||
3. **`on: [push]` fires CI for tag pushes too.** Useful (it is how Scenario F was measured), but it
|
||||
means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate.
|
||||
4. **`felhom.eu` CI still runs `--fast`.** Correct today, since all its gates are network-free; if a
|
||||
network gate is ever added there, that workflow needs the same change the agent's just got.
|
||||
|
||||
- **opengist refused `(size)`** — *"this app's last backup was 2.0 GB and writing it again would cross the reserve"*
|
||||
- **privatebin ADMITTED and dumped normally** — the term is per-app, not a global halt
|
||||
- Tree unchanged; 1 volume dump instead of 2
|
||||
## 13. Teardown
|
||||
|
||||
### One honest correction to the "app not stopped" claim
|
||||
|
||||
`StartedAt` on both apps *did* move, 26 s **after** the refusal. It was the **quiesce loop** for the
|
||||
whole-guest PBS backup, which my fill had broken — not the app-data path. Its own backoff logic then
|
||||
behaved correctly (*"deferring its next quiesce by 15m so the apps are not stopped again for a backup
|
||||
that cannot succeed"*). The app-data claim rests on the **absence of the `Stopping … for safe volume
|
||||
dump` line**, which is the line that appears when that leg bounces an app.
|
||||
|
||||
## 6. The `du` measurement (§Part 1.3) — measured, then rejected
|
||||
|
||||
**66 timed runs** on demo-hp guest 9201, `docker run --rm -v <vol>:/v alpine du -sb /v`:
|
||||
**median ~355 ms per volume, range 341–404 ms** — on volumes holding **tens of KB**. The cost is
|
||||
container start-up, not the walk, so it does not shrink for small apps and only grows for real ones.
|
||||
|
||||
**Rejected**, on two grounds beyond the number: `docker run` needs the writable layer, so the
|
||||
measurement mechanism can fail under exactly the disk pressure the reserve exists to handle; and the
|
||||
previous-dump estimate measures the **artifact that will be written** rather than the live volume,
|
||||
which is the truer predictor. The previous-dump estimate stands.
|
||||
|
||||
## 7. The refusal message as shipped, and what it guarantees
|
||||
|
||||
```
|
||||
[WARN] [backup] App backup REFUSED for opengist (headroom) — refused: backing up this app would
|
||||
leave the filesystem below the reserve (reserve: 97% used or 1.0 GiB free; the filesystem is already
|
||||
below it, before this app's estimated 178.0 KB write) — /mnt/sys_drive: 64.3/68.7 GB used (94%),
|
||||
0.9 GB free; NO database dump, NO volume dump and NO recovery-unit capture was written for it, the
|
||||
previous unit is untouched and NOTHING was deleted
|
||||
```
|
||||
|
||||
**It guarantees, for that app in that run:** no DB dump, no volume dump and no capture were written;
|
||||
every file under `backups/primary/<app>` is byte-identical; the app was not stopped; nothing anywhere
|
||||
was deleted; exactly one operator alert was sent. All five verified by fingerprint above.
|
||||
|
||||
**The wording was not weakened to fit the behaviour** — the behaviour moved so the wording became
|
||||
true. What was *added* is the bound term (`headroom` / `size`) and the estimate.
|
||||
|
||||
**v0.193.1 — found by this very proof run.** The estimate was rendered fixed to two-decimal GiB, so
|
||||
opengist's real **178 KB** printed as `estimated 0.00 GiB write`, which reads as *no estimate was
|
||||
available* — the opposite of what happened. Shipped the same session because it is the same defect
|
||||
class the whole task is about. Re-verified live after redeploy: `estimated 178.0 KB write`.
|
||||
|
||||
## 8. papra (R-156, last leg)
|
||||
|
||||
**Precondition checked, not inherited** — both boxes were wiped and rebuilt today, so the 2 August
|
||||
evidence was re-measured: `docker ps -a` (**including stopped**) on **both** demo guests → no papra;
|
||||
hub `/hosts` → exactly two enrolled hosts (`demo-felhom-8363b5`, `demo-hp-bb76ea`), **zero** papra.
|
||||
|
||||
**Decided from the image, not the README:** `WORKDIR=/app`, `DATABASE_URL=file:./app-data/db/db.sqlite`,
|
||||
`DOCUMENT_STORAGE_FILESYSTEM_ROOT=./app-data/documents`, `PAPRA_CONFIG_DIR=./app-data` — and
|
||||
**`/app/data` does not exist in the image at all**.
|
||||
|
||||
**Departure from the task's stated preference order, stated because it was deliberate.** Option (1)
|
||||
(reconfigure the app to write to `/app/data`) *was* available — all three paths are env-settable. Not
|
||||
taken: it enumerates data paths, so a fourth added upstream would silently escape to the writable
|
||||
layer again — this defect re-armed and invisible. Mounting the app's own data **root** captures every
|
||||
current and future path by construction.
|
||||
|
||||
**Gate output — the arbiter, run in both directions:**
|
||||
|
||||
- fixed → `papra CLEAN`, with the self-test passing on that run: *"prober flags the R-156 signature and clears a correct template — trustworthy"*
|
||||
- reverted to `/app/data` (red-proof on the **real template**, not just the canary) → `BROKEN`: *"mount /app/data is NOT writable by the app's own uid=999"*, *"DATA in the writable layer at /app/app-data/db (db_signature=True, e.g. ['db.sqlite'])"*, *"declared volume /app/data is EMPTY"*
|
||||
- `catalog_gates.py papra` (full, not `--fast`) → **rc=0**, all three gates OK
|
||||
|
||||
**Two operational findings about the gate:** it needs **root** (it reads `/var/lib/docker/volumes`,
|
||||
mode `drwx--x---`; as a normal user its own canary fails UNDETERMINED and it correctly refuses a
|
||||
verdict — fail-closed working as designed), and it hardcodes scratch path `/srv/felhom-gate`, created
|
||||
on DooPlex. Unscoped it deploys all 53 templates; that run was aborted after 10 minutes and its
|
||||
`volgate-*` scratch projects were cleaned up.
|
||||
|
||||
## 9. §3's correction — confirmed in passing, not chased
|
||||
|
||||
`restore_points.go:57-59` takes the manifest's mtime and then `newestArtifact` over the `.sql` and
|
||||
`.tar` files, so **the newest of the three wins**. The restore point does **not** show a stale
|
||||
timestamp. Confirmed and dropped, as instructed.
|
||||
|
||||
## 10. Register
|
||||
|
||||
| ID | Change |
|
||||
|---|---|
|
||||
| **R-181** | **CLOSED — SHIPPED** (v0.193.0 + v0.193.1), with the live evidence above |
|
||||
| **R-156** | **CLOSED** — all three apps fixed |
|
||||
| **R-110** | WAITING-ON-OPERATOR → **READY**, ruling attached: **option (b), tag-tracked**, and it must cover **both** channels (the `/scripts/` git-sync *and* the nine files fetched from `raw/branch/main`) or it only half-works |
|
||||
| **R-115** | WAITING-ON-OPERATOR → **READY**, ruling attached: **mechanism (b)**, a build-side gate refusing to deploy or vouch an unpublished version; the third instance (agent v0.120.0) would have silently downgraded both demo boxes while reporting success |
|
||||
| **R-182** | **NEW.** ID established free: `grep -ro "R-182\b"` over `documentation/` and `*.md` → 2 hits, both prose in `REPORT.md` recording it as *"checked and left unused"*; `R-183` → 0 hits and remains free |
|
||||
|
||||
**R-165** is collapsed to CLOSED/PROVEN-LIVE in `ROADMAP.md`; the capability map's local-backup row
|
||||
moves to **PROVEN-LIVE, both halves**, because the live fill proved the fixed behaviour for **both**
|
||||
reserve terms.
|
||||
|
||||
## 11. Observations — noticed, documented, NOT acted on
|
||||
|
||||
1. **R-182 (filed).** The periodic status refresh (`GetFullStatus` → `captureAllRecoveryUnits`) runs
|
||||
with no admission scope, so a refused app re-alerts on every poll — measured live: a second
|
||||
identical alert pair 13 s after the run's. **Pre-existing in v0.192.0**; R-181 changed neither
|
||||
caller. Its mitigation is a *comment* claiming the hub owns cooldown — which is exactly the
|
||||
"invariant asserted in a comment with no test pinning it" shape, so verify at the hub before
|
||||
scoping.
|
||||
2. **A reserve refusal does not make the run fail.** The DB and volume legs record `SKIP`, not `FAIL`,
|
||||
so `lastDBDump.Success` stays true and the customer-facing status does not turn red. Deliberate and
|
||||
consistent with v0.192.0 (the capture refusal never set it either), and the operator alert is the
|
||||
signal — but it means "backup succeeded" and "every app was backed up" are not the same statement.
|
||||
3. **`UnitSpace.UsedPercent` and `df` disagree** — `df` reported 99% where the alert said 94%, because
|
||||
`df`'s figure accounts for ext4 reserved blocks and the floor's does not. Harmless here (the
|
||||
free-byte term bound), but a percent-term threshold is being compared against a number the operator
|
||||
cannot reproduce with `df`.
|
||||
4. **The whole-guest PBS backup fails when the volume is near-full**, pushing
|
||||
`whole_guest_backup_failed` (severity `error`). Expected under a deliberate fill, and its backoff
|
||||
behaved correctly; noted because it is collateral any future fill test will also produce.
|
||||
|
||||
## 12. Teardown
|
||||
|
||||
Fill file removed; the planted 2 GiB file removed; a final backup regenerated a correct 178 KB tar;
|
||||
`pct fstrim 9201` returned 67.5 GiB and the thin pool settled at **29.43%**, *below* its 36.83%
|
||||
baseline. The backups tree is byte-identical to the pre-test fingerprint. Guest helper scripts and the
|
||||
credential file `shred`-ed. `volgate-*` scratch compose projects removed; the unrelated 9-day-old
|
||||
`jarr-*` containers on DooPlex were left untouched. papra is **not** left deployed.
|
||||
|
||||
No `--no-verify` was used on any push; the `felhom-controller` pre-push hook ran and reported
|
||||
`gates OK` on both pushes.
|
||||
Probe container, probe sync tree and probe tag (`probe-r110-delete-me`) removed; the red-proof tag
|
||||
`v9.9.9` deleted (`git ls-remote --tags` → only `v0.120.0`); the Scenario A marker reverted from
|
||||
`main` and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl
|
||||
pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file.
|
||||
|
||||
@@ -27,34 +27,51 @@ time, and an app switched off deliberately stayed off every time.
|
||||
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
||||
*(R-95, R-87)*
|
||||
|
||||
**A full disk emails you repeatedly instead of once.** When the reserve refuses an app's backup you
|
||||
are told once by the backup run — correctly — but the page showing backup status re-checks on a timer
|
||||
and sends the same message again each time. Not new: as old as the reserve itself, and seen only
|
||||
because we watched the alerts closely while proving the fix below. Harmless if the hub already
|
||||
collapses repeats — which a comment claims and nobody has checked. *(R-182)*
|
||||
**A full disk tells you about ONE app and silently swallows the rest.** Yesterday this was written
|
||||
down the wrong way round — as *too many* emails. Measuring the receiving end reversed it: of nine
|
||||
refusals the machine reported today, **two emails were sent**. When two apps are refused in the same
|
||||
second you are told about one of them, and the other leaves no trace anywhere — not an email, not
|
||||
even a line in the log saying it was held back. So a second app can be going unbacked-up while you
|
||||
have already been told the problem is handled. It is the same fault we fixed once before for
|
||||
whole-machine backups, in a second place that never opted into the fix. *(R-182)*
|
||||
|
||||
## What shipped recently
|
||||
|
||||
**The backup partition is gone, and both demo machines run on the new shape.** Wiped and rebuilt on
|
||||
3 August and taken through the whole customer journey — set up, install an app, back it up, restore
|
||||
it. One storage area instead of two; the space a backup can use went from 19 GB to 65 GB on the small
|
||||
machine and 45 GB to 233 GB on the big one. Three reboots each, correct every time. The two were
|
||||
rebuilt deliberately differently — one from a local copy of the image, one by the ordinary customer
|
||||
route with the published fingerprint checked — so the disk shape and the delivery route are both
|
||||
proven, rather than one proven twice. Their previous demo apps and data are gone; that was the point
|
||||
of a wipe, and you approved it. *(R-165, R-178)*
|
||||
**Pushing the installer no longer publishes it.** The script that runs as root on a brand-new
|
||||
machine was copied from the main branch and served within thirty seconds, so pushing it *was*
|
||||
publishing it, with no staging and no way back but another push. It now comes from a **labelled**
|
||||
version: publishing is moving the label, and undoing it is moving the label back — about half a
|
||||
minute, no deploy. The website is untouched by this and still updates in thirty seconds, because a
|
||||
typo fix must never need a release. Proven by actually doing it: a real push changed nothing that
|
||||
anyone downloads, moving the label published it, moving it back restored the previous bytes exactly.
|
||||
|
||||
**What replaced the wall — and it now watches the right moment.** The wall was quietly doing a second
|
||||
job: keeping a runaway backup from eating the space the machine needs to keep running. That job is now
|
||||
explicit, and as first built it was checked too late — the big write happened first, unchecked, and
|
||||
only the small write after it was refused, while the message still promised your last good copy was
|
||||
untouched. **Fixed and proven on 3 August.** The machine now decides once, per app, **before it writes
|
||||
anything at all**, and that one answer covers all three steps: a refused app writes nothing, is not
|
||||
restarted, and the promise is now literally true — checked by fingerprinting every file before and
|
||||
after. It also stopped being blind to size, so an app is no longer waved through at 96% full and then
|
||||
allowed to write two gigabytes. Proven by deliberately filling a demo machine, once for each way it
|
||||
can refuse. Nothing is ever deleted to make room: every app has only one local copy, so "delete the
|
||||
oldest" would always mean destroying some other app's only copy. *(R-181)*
|
||||
**The catch that would have made it cosmetic was found and covered.** While it runs, the installer
|
||||
fetches sixteen more files — not nine, and from the *agent's* repository, not the website's. They now
|
||||
come from the same version of the agent the machine is installing. That closed a real fault nobody
|
||||
had noticed: a new machine was getting the agent's tested program and its untested settings files, in
|
||||
one install, from two different places. *(R-110, R-183)*
|
||||
|
||||
**Releasing the agent now publishes it, in one command.** Putting a built agent where a new machine
|
||||
can download it was a step someone had to remember, and it was forgotten three times in five days —
|
||||
the last time leaving both demo machines running a version nobody could download, so a rebuild would
|
||||
have quietly installed the *older* one and reported success. There is now one command that builds,
|
||||
labels, publishes and then **downloads it back to check** — and a check that refuses to stay quiet if
|
||||
a released version cannot actually be fetched. Proven by making CI fail on purpose and then go green
|
||||
again on the same code. *(R-115)*
|
||||
|
||||
**The backup partition is gone and both demo machines run on the new shape** — wiped, rebuilt and
|
||||
taken through the whole customer journey on 3 August, by two deliberately different routes so the disk
|
||||
shape and the delivery route are both proven. The space a backup can use went from 19 GB to 65 GB on
|
||||
the small machine and 45 GB to 233 GB on the big one. Their previous demo apps and data are gone; that
|
||||
was the point of a wipe, and you approved it. *(R-165, R-178)*
|
||||
|
||||
**What replaced the wall now watches the right moment.** The wall was quietly keeping a runaway
|
||||
backup from eating the space the machine needs to run. As first built, that replacement was checked
|
||||
too late — the big write happened first, unchecked — while still promising your last good copy was
|
||||
untouched. Fixed and proven on 3 August: the machine decides once, per app, **before it writes
|
||||
anything**, and that one answer covers all three steps, so a refused app writes nothing, is not
|
||||
restarted, and the promise is now literally true. It also stopped being blind to size. Nothing is ever
|
||||
deleted to make room. *(R-181)*
|
||||
|
||||
**The last of the three apps that never saved their data is fixed.** Installed nowhere, so nothing was
|
||||
stranded — checked on both demo machines and in the fleet list rather than assumed. Proven by the check
|
||||
@@ -75,24 +92,27 @@ step — but it notices quickly and tells you. *(R-29, R-161, R-168, R-169)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** both of today's items are done — the reserve and the last unsaved app. Your two decisions
|
||||
are written down and are ours to build.
|
||||
- **Next:** building those two — moving the installer onto a labelled version so publishing is one
|
||||
step you can undo, and a check that refuses to install a version nobody can download *(R-110, R-115)*.
|
||||
- **Now:** nothing outstanding from today — the reserve, the last unsaved app, and both of your
|
||||
decisions are all built and proven.
|
||||
- **Next:** the alert that tells you about one app and swallows the second *(R-182)*.
|
||||
- **After:** the off-site copy that the machine making it can still erase *(R-95, R-87)*.
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **Nothing else.** You settled both open questions on 3 August — the installer moves onto a labelled
|
||||
version, and a check will refuse to install a version nobody can download. Both are written down and
|
||||
are ours to build. *(R-110, R-115)*
|
||||
- **Nothing else.** Both decisions you took on 3 August are now built and proven. One small question
|
||||
will come back later: the automatic check cannot see which version you have told machines to
|
||||
install, only which ones exist — closing that either needs a password given to the build server or
|
||||
a check inside the hub itself. Filed, not urgent. *(R-184)*
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-03** — The reserve now guards the step that fills the disk, and its promise is true; the
|
||||
last app whose data was never saved is fixed. Both proven on a demo machine, not just in tests.
|
||||
- **2026-08-03** — Publishing became something you do rather than something that happens: the
|
||||
installer and the agent both moved onto labelled versions with a way back, and a check now refuses
|
||||
a release nobody can download. Earlier the same day: the reserve now guards the step that fills the
|
||||
disk and its promise is true, and the last app whose data was never saved is fixed. All proven on
|
||||
real machines, not just in tests.
|
||||
Earlier the same day: both demo machines wiped and rebuilt from the new base image and taken through
|
||||
set-up → install an app → back it up → restore it, with the backup space ceiling gone and measured.
|
||||
|
||||
@@ -102,10 +122,6 @@ step — but it notices quickly and tells you. *(R-29, R-161, R-168, R-169)*
|
||||
the hub's own database is in no automatic backup** — it holds every machine's emergency password.
|
||||
Filed, not yet fixed.
|
||||
|
||||
- **2026-08-02** — Boot recovery finished; six hard resets, everything back every time. Two instances
|
||||
of the same hole — starting an app whose external drive was missing — were found by reading the code
|
||||
and fixed the same day.
|
||||
|
||||
- **2026-08-02** — Thirteen mechanical checks had built up and nothing ran most of them; two were
|
||||
failing quietly. Fixed. Decided the same day: the 20 GB backup partition goes away; and only this
|
||||
machine and the tester's box are protected, every other box may be broken or reinstalled freely.
|
||||
|
||||
@@ -31,6 +31,7 @@
|
||||
|---|---|---|---|---|
|
||||
| Appliance day-0 install: golden image → first boot → auto-confirm (zero clicks) → claimable box | installer, agent, hub, golden | **PROVEN-LIVE** (nested VM) | `DRILL-day0-vm-2026-07-12`, `DRILL-day0-take2-2026-07-12` | First firing on real customer hardware pending → R-1 |
|
||||
| BYO install: `--mode byo`, mandatory caps, host-mutation disclosure, coexistence guards | installer v1.15+, agent | **PARTIAL** | `DRILL-GL6-2026-07-08` (demo box); GL-8 coexistence fixes | Peti clean-slate reinstall on proxmox2 is the first real BYO run of the current path → R-1 |
|
||||
| **The installer is PUBLISHED, not pushed — the artifact that runs as root on a virgin box is served from a version-controlled ref, and rolling back is one act** | scripts **v1.23.0** + `manifests/webpage.yaml` (R-110, operator ruling option (b)) | **PROVEN-LIVE (2026-08-03)** | `scripts/CHANGELOG.md` v1.23.0 + `REPORT.md`. **Proven by HTTP against the real URL, not from a pod's filesystem.** *Scenario A:* a real push to `main` without moving the tag left the served script **byte-identical** (`sha256 2f859555…`), and a marker comment planted in that very commit was **absent** from the served bytes, while the website tree advanced to the new commit in the same observation — both halves of the split in one measurement. *Scenario B:* moving the tag published in **~40 s** (`sha → ea2b4aa9…`, marker present) and moving it back restored **exactly** the pre-publish sha. `https://felhom.eu/` returned 200 throughout. *P-A, measured BEFORE the manifest was touched because the model rests on it:* git-sync v4.4.0 follows a tag **and notices a moved one** (`update required … local:<old> remote:<new>` → `updated successfully`) | **Two syncs, deliberately: the WEBSITE still tracks `main`.** Pinning both would turn every copy edit into a release, which makes the release meaningless and the site slow to fix. **Publish** = cut `installer-v<SCRIPT_VERSION>` + bump the manifest `--ref` + sync; **roll back** = move the tag back, which needs **no ArgoCD sync and no deploy**. Continuity is structural rather than lucky: both trees are seeded by init containers so a fresh pod is not Ready until the tag is checked out, and `maxUnavailable` rounds to 0 on one replica, so a failed scripts-init leaves the OLD pod serving — the failure direction is *no update*, never *no `/scripts/`*. **The URL never carried a ref**, so the bootstrap script and the hub's day-0 command follow the tag with no edit and **no hub change**. The installer's own sixteen run-time fetches are a separate channel pinned to the AGENT's version (**R-183**), because they are the agent's configs and not this repo's — leaving them on `main` would have made the whole change cosmetic |
|
||||
| Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; **universal secret-free / operator-bind** mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE on TWO different boards** (N100 2026-07-18; HP t740 2026-07-21) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — the full chain on real metal in a single pass:** the generic reusable pairing ISO (v1.20.0, `--loader mkimage`, SB off) booted the cheap AMI board that F1 had blocked, installed unattended, and the box **self-registered as an unclaimed appliance at 16:17:14 — the same second it first booted** (`appliance_registrations` id=3), then bound → credential-delivered → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware.** Prior nested legs: slice A `SPIKE-baremetal-iso-2026-07-16` (build gate, disk-filter fail-safe, stub→host-install fetch); slice B RUNBOOK-B (shim boots+installs OVMF SB-enforcing + SeaBIOS; `--loader mkimage` boots+installs SB-off; mkimage SB-enforcing **FAILS** `Access Denied`; surgery byte-identical); **slice C (2026-07-17): the GENERIC secret-free ISO** — box self-registers as an unclaimed appliance (`POST /api/v1/appliance/register`, one-shot poll delivery, 404-no-oracle — all live-verified through the public ingress), operator binds on the Hosts page, hub delivers credentials once; bootstrap harness proves direct(zero-appliance-calls)/pairing/delivery; artifact proven secret-free (baked env = hub URL only) | **F1 loader caveat:** `--loader mkimage` fixes cheap AMI firmware that can't USB-boot the stock GRUB — UNSIGNED → **Secure Boot must be OFF**; default `shim` keeps SB. **Slice C bind is operator-password-gated** (CC stages, Viktor binds) → the live boot→register→bind→day-0 composition + physical N100 boot fold into the supervised rehearsal (R-1). Customer-facing **self-bind page = R-27 slice 1 SHIPPED (hub v0.66.0, 2026-07-17)** — see the dedicated self-bind row | **Second board, 2026-07-21 (demo-hp, HP t740 / Ryzen V1756B / AMI M42):** the whole chain ran on virgin hardware in one pass — armed install → self-registration as an unclaimed appliance → operator bind → day-0 → running guest 9201 + agent 0.92.1 as `demo-hp-bb76ea`. **The shim loader booted with Secure Boot ENABLED**, which retires the assumption that Felhom installs need SB off — that was an N100-firmware workaround. The exact-serial disk filter took the system SSD and left the box's 1TB NVMe untouched/unenrolled on hardware it had never seen. Two failures filed rather than smoothed over: **R-59** (no DHCP → the installer baked a static fallback instead of aborting) and **R-61** (baked root password unknowable → no console access).
|
||||
| Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`<iso>.rootpw.txt`) | scripts v1.24.0 (`scripts/iso/felhom-bootstrap.sh` `network_gate`/`sweep_nics`, `build-felhom-iso.sh` rootpw emission) | **PROVEN-LIVE (nested drill — nested ≠ metal: metal proof rides the next real multi-NIC install)** | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` — dead-NIC install from the virgin v1.24.0 ISO baked the 192.168.100.2 fallback (WITH a dead default gateway), the R-59 screen painted on the console (screendump captured), and after the cable move the box swept to the working NIC, re-leased and **self-registered at the hub unaided in under a minute**; the drill also caught + fixed the stale-fallback-route trap (flush before the bounded dhclient) and verified the emitted rootpw against the installed box's shadow hash | R-59 ships as a first-boot gate, not an install-time abort (recorded deviation — the fallback is the auto-installer's own, initrd hook out of scope); sweep is structurally first-boot-only (`state.json` gate + unit done-flag condition); a box past install-start gets the screen but its interfaces are never touched |
|
||||
| Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. **Deliverability (R-4), gmail half DONE 2026-07-18:** the rehearsal's claim email was the first sent under the tightened DMARC `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu remains Viktor's open half.** (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) |
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -20,7 +20,7 @@
|
||||
|
||||
| ID | Item | Size | Status | Notes / map rows flipped |
|
||||
|----|------|------|--------|--------------------------|
|
||||
| R-115 | **Publishing is a remembered step — forgotten within eight hours of being documented as forgettable** | M | **READY** — operator ruling 2026-08-03: **mechanism (b), a build-side gate that REFUSES to deploy or vouch an unpublished version.** THIRD instance the same day (agent v0.120.0 deployed to both boxes while unpublished; a documented-path reinstall would have silently downgraded them and *succeeded*). CC's to build | A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published, so **"deployed" and "installable" are independent states that drift silently**. **Instance 1 — R-111** (morning): 17 agent releases v0.97.0–v0.113.0 stranded; a new customer would have installed without the whole R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT. Found only because the E-2d Phase 0 gate happened to look. **Instance 2 — agent 0.114.0** (same afternoon): the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, never published — which blocked Session C, since a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix. **The finding is the RECURRENCE, not either instance** — both are fixed. R-111 named this leg in its own text (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; it recurred the same day, which is the evidence that **a note is not a mechanism**. **Class: → R-29, one layer up** (a control that exists and is never walked) — deliberately NOT given a second ID. **Filed as its own item rather than reopening R-111** because R-111's finding (the channel *was* stale) is closed and verified end-to-end by the E-2d install, while the process defect that caused it is a distinct problem with a distinct fix and a distinct owner. **Operator's decision, mechanisms first:** (a) publish as a step in the build/release path so deployed and installable cannot diverge; (b) a gate that refuses to deploy an unpublished+unvouched version — strongest, fails closed; (c) a session-end checklist entry; (d) accept manual + a pre-Session-C verification. **(a)/(b) are mechanisms, (c)/(d) are reminders — and R-29's whole finding is that reminders do not hold.** No code written when filed, by design |
|
||||
| R-115 | ~~**Publishing is a remembered step — forgotten within eight hours of being documented as forgettable**~~ | M | **CLOSED — SHIPPED 2026-08-03** (`release-agent.sh` + `check-published-versions.py`, no version bump). Releasing now builds, tags, publishes and **verifies by an independent download** in one act; a `v<semver>` tag with no package fails the gate, and **CI runs the full gate set** so it actually runs. Red-proof measured on real CI: same commit, green before a tagged-unpublished version existed, red after. Residue → **R-184** | A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published, so **"deployed" and "installable" are independent states that drift silently**. **Instance 1 — R-111** (morning): 17 agent releases v0.97.0–v0.113.0 stranded; a new customer would have installed without the whole R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT. Found only because the E-2d Phase 0 gate happened to look. **Instance 2 — agent 0.114.0** (same afternoon): the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, never published — which blocked Session C, since a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix. **The finding is the RECURRENCE, not either instance** — both are fixed. R-111 named this leg in its own text (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; it recurred the same day, which is the evidence that **a note is not a mechanism**. **Class: → R-29, one layer up** (a control that exists and is never walked) — deliberately NOT given a second ID. **Filed as its own item rather than reopening R-111** because R-111's finding (the channel *was* stale) is closed and verified end-to-end by the E-2d install, while the process defect that caused it is a distinct problem with a distinct fix and a distinct owner. **Operator's decision, mechanisms first:** (a) publish as a step in the build/release path so deployed and installable cannot diverge; (b) a gate that refuses to deploy an unpublished+unvouched version — strongest, fails closed; (c) a session-end checklist entry; (d) accept manual + a pre-Session-C verification. **(a)/(b) are mechanisms, (c)/(d) are reminders — and R-29's whole finding is that reminders do not hold.** No code written when filed, by design |
|
||||
| R-116 | **The drive-absent alarm and its recovery are a mismatched pair — generic on the way out, specific on the way back** | S | idea — **PROVEN LIVE 2026-07-29** | Absent fires `storage_disconnected`; return fires `backup_target_restored`. `backup_target_absent` never fires at all (count 0 across a full Session-C run), so an operator gets an alarm they cannot match to its recovery — exactly what `notifyDriveReturned`'s own comment forbids. Root cause: `notifyDriveAbsent` (`intermediary.go:635-646`) branches on `isTarget[a.Path]` with `a.Path` the GUEST path, and `driveTargetByPath` (`:602-616`) builds it as `out[GuestPath] = d.BackupTarget` — but **the drive is TWO `/disks` rows and the flag and the guest path sit on different ones**: the `felhom-backup` storage row has `BackupTarget: true` (`felhom-agent/internal/localapi/disks.go:211`) and gets a guest path only while classified user-data, while the registry union row has the guest path and **never assigns `BackupTarget`** (`disks.go:265-267`). Absent ⇒ the flagged row loses its guest path ⇒ the union row writes `false` ⇒ generic. On return the rows rejoin ⇒ specific. v0.184.1 fixed the KEYING, not this. **Only reachable because R-113 made the gate fire at all.** Fix likely agent-side; decide the repo first. Blocks E-2's C5. Evidence: `audits/SESSION-C-2026-07-29.md` §5 |
|
||||
| R-113 | **The drive-absent gate cannot fire on device loss — E-2b's alarm is wired to an unreachable condition** | M | idea — **PROVEN LIVE 2026-07-29** | `planDriveGates` (`felhom-controller/internal/web/intermediary.go:216-262`) treats a path as present by OR-ing in `d.BoundUnderParent`, which the agent derives from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc/<pid>/mountinfo`"* (`internal/localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is not device-bound and its mountinfo entry outlives the device**, so the gate reads it as present and `notifyDriveAbsent` is never called. Live on a fresh box: target drive hot-detached, agent said `enrolled drive absent by UUID` every 20 s for 4½ min, controller logged **0** `[gate]` lines, hub received **zero** events — neither `backup_target_absent` nor the generic `storage_disconnected`. Not a virtualisation artefact (device-bound-mount vs manual-bind is the same on metal); caveat: SCSI hot-detach, physical unplug not staged. **Sixth instance of seam-built-but-never-wired — E-2b wired the seam to a condition that cannot occur.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 |
|
||||
| R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 |
|
||||
@@ -157,7 +157,7 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
|
||||
| R-97 | **The whole-guest backup tier has NO failure signal to the hub — `internal/quiesce` never notifies** | S | **SHIPPED (controller v0.177.0 + hub v0.78.0, 2026-07-27)** — **R-97a:** `quiesce.TierNotifier`, a seam (not an import) wired by an init-only setter, edge-triggered on the R-88 breaker ARMING so a failing tier is reported once per run rather than once per retry; recovery rides `recordSuccess`'s existing bool. **NEW operator-only event types** `whole_guest_backup_failed`/`_recovered` — deliberately NOT `backup_failed`, which carries a customer Hungarian template AND sits in demo-felhom's live `enabled_events`, so reusing it would have emailed the CUSTOMER about a backup they cannot act on while it was still retrying. The recovery joins `recoveredPairedDownTypes` because its `info` severity would otherwise be dropped by `severityNotifies` — the operator would hear it break and never hear it heal. **The hub's operator cooldown was keyed `customerID:eventType` alone**, so one tier would have masked the other for an hour; now narrowly extended with a `tier` suffix taken from the event details, leaving every other event type unchanged. **R-97b:** a suppression window keyed to the quiesce CYCLE (not a state test — v0.164.0's `!= StateStopped` filter cannot see an app caught MID-RESTART, which is exactly how BookStack alarmed), consumed at the same single derivation point `classifyRunStates`. Grace = **180 s**, derived from the deploy flow's 120 s health timeout and Mealie's 60 s `start_period`; it **expires**, so an app that genuinely fails to come back still alarms. **PROVEN LIVE end-to-end with a control:** the new type POSTs 200 from inside guest 9201 while a bogus type 400s, and `notification_log` shows **1 operator row, 0 customer rows**. The quiesce→notify link itself is unit-proven only. | On 2026-07-27 three whole-guest backups failed and three quiesce cycles stopped and restarted every customer app stack, and **not one `backup_failed` event reached the hub.** It is not the allowlist — the hub already carries `backup_failed` and `backup_completed` (they are emitted by the controller's *app-data* backup path). The cause is that **`internal/quiesce` does not import `internal/notify` at all**: the tier R-82 built has no route to the hub, so a whole-guest backup can fail indefinitely in silence. The loop's only trace was `app_start_failed` — **info** severity, **Hungarian**, on the **customer** channel — telling the customer BookStack was down (it had been caught mid-restart by the third cycle) without saying why, during an outage the system itself caused. So the one signal that did fire was both the wrong tier and the wrong story. **Shape:** emit `backup_failed`/`backup_completed` from `quiesceAndPollTiers` naming the TIER, operator-tier; and decide whether a quiesce-induced restart should suppress `app_start_failed` the way controller **v0.164.0**'s deliberate-stop filter does — an app the backup stopped on purpose is not a fault. R-88's breaker bounds the repetition but changes nothing about the silence |
|
||||
| R-95 | **The restic offsite tier's credential CAN DELETE — R-89's "parallel question", now ANSWERED** | M | idea — established read-only 2026-07-27 | **The exposure closed on the weekly PBS tier is fully open on the daily restic tier**, which holds the customer's actual documents and photos and is the only tier that survives losing the box. Established without mutating anything: **(1) Identity** — a per-customer *subaccount* on `storage-box-pool-1` (box 611714, bx11, `u629488`): `u629488-sub1` home `felhom-demo-felhom`, `sub2` peti-felhom, `sub3` demo-hp, each labelled `felhom-customer`. Auth is an **SSH key stored ON THE BOX** (`…/felhom-controller-data/_data/data/offbox/ssh_key`, 0600, beside `repo_password` + a pinned `known_hosts`) — customer-side, not hub-side, so a compromised guest holds it. **(2) Read-write: YES** — the API reports **`readonly=False` on all three subaccounts**, and it is not merely latent: the controller runs `restic forget --group-by host,tags --keep-daily 7 --keep-weekly … --prune` **from the box** (`backup/offbox.go:984`, also `:1070`). Delete rights are exercised on every run. **(3) Append-only: NO, and not expressible** — the repo is built as `sftp:` (`offbox.go:482`); restic's append-only mode requires the **REST server** backend, which plain SFTP cannot provide. **(4) A zero-code mitigation exists and is unused:** the box type carries `snapshot_limit=10` and the API reports `snapshot_plan=null` with **0 snapshots** and `size_snapshots=0`. Hetzner Storage Box snapshots are taken **server-side, outside the SFTP namespace** — an SFTP subaccount cannot delete them — so they are a genuine immutability layer at no extra cost and with no code change. **Rule once for both tiers, per R-89.** Options, cheapest first: enable a snapshot plan (operator click, immediate); split backup-write from prune so pruning runs somewhere the box cannot reach; or move the repo to restic's REST server with `--append-only`. Flips the capability-map row for offsite immutability |
|
||||
| R-94 | ~~A hand-synced version constant drifts, and the gate that would catch it is never run~~ | XS | **CLOSED — SHIPPED hub v0.87.0, 2026-08-02** | Closed by **deleting** the label rather than deriving it: the Setup command fetches the installer at run time from a 30 s-git-synced website (R-110), so no build-time value in the hub can be true. `hostinstall_gates.py` gate 1 inverted to pin the ABSENCE of a version literal; the tautological `render_test.go` assertion deleted (demonstrated passing at `9.9.9`). Detail: `OPEN-ITEMS.md` R-94 |
|
||||
| R-110 | **`main` is the installer's publish channel — there is no staging** | S | **READY** — operator ruling 2026-08-03: **option (b), the channel moves to a TAG**, so publishing is moving the tag and rollback is moving it back. **Must cover BOTH channels** — the nginx `/scripts/` git-sync AND the nine files the installer fetches from `raw/branch/main` — or it only half-works. CC's to build | `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a `--period=30s`, and nginx serves that working tree directly (`location /scripts/`, `root /usr/share/nginx/html/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`, `:1262`; `runbooks/day0-install.md` C.1) receives. **There is no tag, no pinned-version path, no staging copy and no rollback other than another push** — for the one artifact that runs as **root on a virgin box**, the most privileged thing Felhom ships. **Two consequences worth stating plainly:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29, so the proof run confirms what customers already receive rather than clearing it for release; and the precaution the old R-94 row recorded ("do not point every new box at an installer that has never run") **was never available to take**, because nothing points boxes at a version. **Open question for the operator, not a defect to fix blind:** should `/scripts/` serve a pinned release — a tag-tracked git-sync ref, or a versioned directory (`/scripts/1.22.0/…`) with the hub's generated command naming a version — or is `main`-tracking the accepted shape for a one-operator product where the alternative is a release ritual nobody performs? **Exposure today is zero** (no boxes are installing), which is exactly why it is cheap to decide now. Whichever way it goes, it decides whether R-94 leg (a) makes the label a *fact* (derived from the served script) or keeps it a *claim*. Flips no capability-map row — the map states what the platform does, and this changes nothing about that |
|
||||
| R-110 | ~~**`main` is the installer's publish channel — there is no staging**~~ | S | **CLOSED — SHIPPED 2026-08-03** (installer v1.23.0). `/scripts/` syncs `installer-v1.23.0`; the website still tracks `main`. Proven by HTTP: a push to `main` left the served bytes byte-identical, moving the tag published in ~40 s, moving it back restored the exact prior sha. The run-time fetches turned out to be **sixteen from the agent repo**, not nine from here — pinned to the agent version instead → **R-183** | `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a `--period=30s`, and nginx serves that working tree directly (`location /scripts/`, `root /usr/share/nginx/html/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`, `:1262`; `runbooks/day0-install.md` C.1) receives. **There is no tag, no pinned-version path, no staging copy and no rollback other than another push** — for the one artifact that runs as **root on a virgin box**, the most privileged thing Felhom ships. **Two consequences worth stating plainly:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29, so the proof run confirms what customers already receive rather than clearing it for release; and the precaution the old R-94 row recorded ("do not point every new box at an installer that has never run") **was never available to take**, because nothing points boxes at a version. **Open question for the operator, not a defect to fix blind:** should `/scripts/` serve a pinned release — a tag-tracked git-sync ref, or a versioned directory (`/scripts/1.22.0/…`) with the hub's generated command naming a version — or is `main`-tracking the accepted shape for a one-operator product where the alternative is a release ritual nobody performs? **Exposure today is zero** (no boxes are installing), which is exactly why it is cheap to decide now. Whichever way it goes, it decides whether R-94 leg (a) makes the label a *fact* (derived from the served script) or keeps it a *claim*. Flips no capability-map row — the map states what the platform does, and this changes nothing about that |
|
||||
| R-128 | **`ISO_VERSION` "aligns with SCRIPT_VERSION" was a comment nothing evaluated** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | Closed by **correcting the claim, not asserting it**: the ISO is frozen while `felhom-host-install.sh` is fetched at run time from `main` (R-94/R-110), so an assertion would invent a constraint. `build-felhom-iso.sh:45-52`. Full reasoning in `OPEN-ITEMS.md` |
|
||||
| R-154 | **`[first-boot]` is automated-install-only, and nothing in the tree said so** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | A PVE property, measured with a same-image control (`audits/SPIKE-universal-iso-3-2026-07-31.md` §2); recorded at `scripts/iso/pkg/build-deb.sh:6-11`. Superseded in practice by the `.deb` delivery route |
|
||||
| R-155 | **`iso-repack.sh` refused any ISO without `auto-installer-mode.toml`** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | **Narrowed, not deleted** — unchanged for `FELHOM_MENU=single` (`iso-repack.sh:121-128`), does not apply to `release` where the file's absence *is* gate G1. Do not remove it wholesale |
|
||||
|
||||
Reference in New Issue
Block a user