reports deleted
This commit is contained in:
@@ -1,113 +0,0 @@
|
||||
# REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)
|
||||
|
||||
Follows `REPORT-campaign7/8/9.md`. Root `REPORT.md` is another session's (hub v0.85.0) and was not
|
||||
clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
|
||||
|
||||
**Full audit + evidence:** `documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`,
|
||||
`documentation/tests/campaign10-evidence-2026-07-31/`.
|
||||
|
||||
## The sentence that matters
|
||||
|
||||
**Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the
|
||||
"drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No
|
||||
resource leak over 13.5 h. Two findings: R-156 and R-157.**
|
||||
|
||||
## What was established
|
||||
|
||||
- **Venue** — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the
|
||||
`/mnt/nvme-1tb` **mount root** (a subdirectory would have emitted `storage_disconnected` for demo-hp
|
||||
all night — the exact signal I1/I2 discriminate).
|
||||
- **Baselines, all read fresh.** controller `main` 0.188.0, **golden 0.188.0 (not behind)**, agent
|
||||
0.119.0 published+vouched, hub 0.86.0, ISO **1.26.1** (`f3cc86d5…`, round-trip verified live).
|
||||
The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
|
||||
- **Isolation gate — both denials captured, each with a positive control.** The PBS control **failed
|
||||
first**: four clean-looking 403s were worthless because the token was denied on its own datastore
|
||||
too (PBS token privilege separation). Fixed, re-run, denials stand.
|
||||
- **A1** fresh install from the **published** ISO. 1.26.1 is a public release image — verified against
|
||||
its bytes that it has no auto-install path — so it was driven blind via screendump + `sendkey`
|
||||
through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which
|
||||
would otherwise have been mangled and locked the box out.
|
||||
- **A2** claimed for real; discriminator flipped `dashboard not yet claimed` → `authentication required`.
|
||||
- **A3** both drives enrolled through the **real** endpoint; `mentes` accepted as backup target via
|
||||
the offer flow, ending `degraded:false / target:felhom-backup` — the I5/I6 healthy baseline.
|
||||
Four apps healthy spanning both sides of D5's split (4 × `type: secret`, 1 × `type: password`).
|
||||
- **A4** discriminators seed and read back across all four apps; rallly's over the path
|
||||
`DATABASE_URL` actually names, not the trusted socket that produced D5's false pass.
|
||||
|
||||
## Phase B — the soak
|
||||
|
||||
Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped
|
||||
deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**).
|
||||
1 461 invariant checks in total.
|
||||
|
||||
- **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale,
|
||||
never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as
|
||||
skipped rather than silently green — it fired once.
|
||||
- **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place
|
||||
variants and 330 secret-class assertions covering both sides of D5's split.
|
||||
- **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced
|
||||
`backup_target_absent` and a clean recovery.
|
||||
- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via
|
||||
`bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names
|
||||
it. That is the case R-117's spike called "the worse half".
|
||||
- **RTO, both bands measured.** S: 66 MB → **42.0 s** / **41.4 s** across two passes (66 restores).
|
||||
M: 21.1 GB → **608 s** mean over 2 reps, both returning the correct discriminator. 327× the data
|
||||
cost 14.5× the time, giving **RTO ≈ 40 s + 26.9 s/GB** (backup ≈ 29 s + 17.4 s/GB). The fixed ~40 s
|
||||
dominates below ~1.5 GB — that is the S band, and why its numbers clustered so tightly.
|
||||
- **Capacity ceiling, and the more consequential result:** a DB-backed app's recovery unit is **1.90×**
|
||||
its data (volume tar + SQL dump). The default `/mnt/sys_drive` is **20 GB**, so on a default box the
|
||||
largest locally-backupable app is **≈ 10 GB** — **the M band does not fit at all** without a
|
||||
per-customer `SysDataGrowGB`. **RPO still not measured.**
|
||||
- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds
|
||||
flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots
|
||||
and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** —
|
||||
bounded by journald, but a lot of logging.
|
||||
- Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that
|
||||
was not first observed live.
|
||||
- **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system
|
||||
disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the
|
||||
soak would have produced green cycles that tested nothing cross-drive.
|
||||
- **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that
|
||||
never landed became a fake "stale restore", and a real one would have looked identical. Fixed and
|
||||
red-proofed before restarting.
|
||||
|
||||
## Findings
|
||||
|
||||
- **R-156 (new)** — **papra's data is neither persisted nor backed up, and it reports healthy.** The
|
||||
template mounts `papra_data:/app/data`; the app writes `/app/app-data/db/db.sqlite`. The volume is
|
||||
empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
|
||||
sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
|
||||
Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
|
||||
- **R-157 (new)** — **bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two
|
||||
mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while
|
||||
docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences,
|
||||
intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is
|
||||
exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp
|
||||
check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on
|
||||
every channel. A settle-condition fix closes A and leaves B open. Not fixed.
|
||||
- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
|
||||
(`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
|
||||
`hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
|
||||
therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by
|
||||
not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.
|
||||
|
||||
## What did not run
|
||||
|
||||
**12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place,
|
||||
kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive).
|
||||
Previously reported as 6 of 12 — that was run 1 only.
|
||||
|
||||
**Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not
|
||||
automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed.
|
||||
`kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom
|
||||
covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed.
|
||||
|
||||
**Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34–c39 clean on
|
||||
every invariant. Beyond 39 is untested, not proven clean.
|
||||
|
||||
## Teardown — OWED, nothing removed
|
||||
|
||||
Still intact: the rig is reusable for the atoms that did not run. VM 311, `c10-scratch`, PBS datastore
|
||||
`felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition:
|
||||
DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four
|
||||
orphaned scratch customers left by exactly this omission.
|
||||
@@ -1,43 +0,0 @@
|
||||
# REPORT — CAMPAIGN 7 (felhom.eu side: docs only)
|
||||
|
||||
> Written as `REPORT-campaign7.md`, **not** the shared `REPORT.md`, per the convention this run
|
||||
> added to `CLAUDE.md`: `REPORT.md` is overwritten, so a second concurrent session in this repo
|
||||
> would clobber it. This session's implementation work was in `app-catalog-felhom.eu`; here it only
|
||||
> touched documentation.
|
||||
|
||||
**Run:** 2026-07-18 evening → 2026-07-19 morning. **Class:** campaign (record-and-rank + a defined
|
||||
allowed-fix set). **Implementation repo:** `app-catalog-felhom.eu` (see its `REPORT.md`).
|
||||
|
||||
## What changed in this repo
|
||||
|
||||
| file | change |
|
||||
|---|---|
|
||||
| `documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` | **new** — method, uninstall-semantics map, trio detail, full 53-app matrix, ranked findings, coverage |
|
||||
| `documentation/backlog/ROADMAP.md` | **+3 items** — R-40 (multi-hop major upgrade path), R-41 (no standing catalog deployability check), R-42 (sidecar-major ruling) |
|
||||
| `CLAUDE.md` | REPORT.md parallel-session rule: the second session writes `REPORT-<topic>.md` |
|
||||
|
||||
No hub/agent/scripts/website code was touched (campaign scope: catalog + docs).
|
||||
|
||||
## Headline for this repo's readers
|
||||
|
||||
1. **Uninstall semantics map row PARTIAL → PROVEN** (campaign doc §2), with live evidence from all
|
||||
three trio apps: remove requires stop first; named docker volumes are **always destroyed**
|
||||
(including the app's database); HDD bind-mount data and `backups/primary/<app>` survive unless
|
||||
explicitly ticked; images are kept; `app.yaml` goes, the template stays; the per-app **offsite
|
||||
toggle survives** the uninstall while tier-2 config is cleared. The confirmation modal does warn
|
||||
about the volumes, so there is **no consent gap**.
|
||||
2. **A lying healthcheck takes an app OFF-LINE, it does not merely mislead.** Traefik will not route
|
||||
to an `unhealthy` container, so a probe that cannot execute → permanent unhealthy → **404 to the
|
||||
customer while the app serves 200 on its own port**. 7 of 53 apps were in that state.
|
||||
3. **The pre-flight gate's own signal is missing:** the 0.145.0 → 0.146.0 floor-lift emitted no
|
||||
`controller_updated` event, though the identical bootstrap path emitted one for 0.143.0 → 0.145.0
|
||||
two hours earlier (§0, finding F1). The box did converge — golden, floor and runtime all agreed —
|
||||
but the event trail under-reports version transitions.
|
||||
|
||||
## Open items owned outside this repo
|
||||
|
||||
- **plant-it / wanderer** — images do not resolve at all (neither the new tag nor the one the
|
||||
catalog already ships). Upstream research needed; recorded as findings, not deletions.
|
||||
- **gokapi** — pinned back to v1.9.6; v2 needs the seeded `config.json` regenerated. Security-
|
||||
relevant, should not sit on a superseded line indefinitely.
|
||||
- **glance** — never had a seeded `glance.yml`; proven pre-existing.
|
||||
@@ -1,62 +0,0 @@
|
||||
# REPORT — CAMPAIGN 8: the backup & restore subsystem (2026-07-27/28)
|
||||
|
||||
Adversarial, destructive, unattended run against `demo-felhom`, `demo-hp` and `ep0`.
|
||||
Full report: `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
Evidence: `DooPlex:~/campaign8/evidence/` (103 files, 35 MB, written continuously by 11 collectors).
|
||||
|
||||
**No production code was changed.** Findings are recorded and ranked, never fixed inline, per the
|
||||
campaign's own rules.
|
||||
|
||||
## Scope safety
|
||||
`peti-felhom`, its namespace and `u629488-sub2` were never touched. Phase 0 established with five
|
||||
documented probes that peti has **no data at all** in `felhom-offsite`, which is what made the
|
||||
operator-approved 100% datastore-fill safe. The 13 GB rollback copy `/srv/pbs-felhom` on ep0 is
|
||||
intact.
|
||||
|
||||
## Findings
|
||||
|
||||
| # | Finding | Severity | Class |
|
||||
|---|---|---|---|
|
||||
| F-CRIT-1 | An app that fails to restart after a quiesce **never alarms**, on any channel | **HIGH** | DEFECT |
|
||||
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT |
|
||||
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
|
||||
| F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT |
|
||||
| F-LEAK | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed | MEDIUM | DEFECT (root-caused by fault 18) |
|
||||
| F-REBOOT | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
|
||||
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
|
||||
| F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT |
|
||||
| F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational |
|
||||
|
||||
Both HIGH findings are in the same place: **the system's ability to tell you a backup did not
|
||||
happen.** Both cite the code and the comment that asserts the property the code does not provide.
|
||||
|
||||
## What is now proven that was not before
|
||||
R-88 breaker arming **and its full ladder** (15m/30m/1h/2h/4h/4h-cap) · per-tier isolation under a
|
||||
real one-tier-fails case · `whole_guest_backup_failed` end-to-end with correct tier attribution ·
|
||||
R-97c operator-only routing verified against the hub DB (zero customer rows with `status='sent'`) ·
|
||||
`whole_guest_backup_recovered` + the R-68 pairing gate firing live · **`age_state=absent`** ·
|
||||
R-97b's suppression half · the crash-recovery unquiesce by an actual SIGKILL (1 s) ·
|
||||
**R-87 — the first restic restore round-trip ever performed**, byte-verified (6/7 sha256 identical,
|
||||
the 7th explained) · R-82 one-quiesce-two-tiers · single-flight on two independent paths.
|
||||
|
||||
## Restore round-trips
|
||||
restic (R-87) · local vzdump → fresh CT · PBS offsite → fresh CT · corrupted snapshot → fails
|
||||
cleanly. `mount_parity` exact on both whole-guest tiers, `unprivileged: 1` preserved.
|
||||
|
||||
## Fleet state
|
||||
**Healthy. Nothing left broken.** All four compression knobs reverted and verified; every fault
|
||||
unwound; no leaked scratch guests, nft rules, ballast files or clock skew; ep0 datastore clean with
|
||||
zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
|
||||
|
||||
## Not tested (with reasons)
|
||||
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the
|
||||
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), fault 12 (host
|
||||
reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for
|
||||
documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings.
|
||||
|
||||
**Campaign-caused outage, stated plainly:** fault 11 took demo-hp guest 9201 down for ~9m47s
|
||||
(the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after.
|
||||
|
||||
## Note on repo conventions
|
||||
This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an
|
||||
entry — there is nothing shipped to log. The deliverable is the audit document plus this report.
|
||||
@@ -1,85 +0,0 @@
|
||||
# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28)
|
||||
|
||||
**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign,
|
||||
and findings are recorded, never fixed inline. Full write-up:
|
||||
`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`.
|
||||
Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously).
|
||||
|
||||
Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes.
|
||||
**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact.
|
||||
|
||||
## The headline — two never-proven restore paths are now proven
|
||||
|
||||
Driven through the **real endpoints the UI posts to**, over https through traefik with a real session
|
||||
and CSRF token, on live hardware.
|
||||
|
||||
| proof | result |
|
||||
|---|---|
|
||||
| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
|
||||
| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
|
||||
| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
|
||||
| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
|
||||
| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
|
||||
| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
|
||||
|
||||
The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
|
||||
at their last-backup state.**
|
||||
|
||||
## Findings — 3 defects, ranked (none fixed)
|
||||
|
||||
| # | finding | severity |
|
||||
|---|---|---|
|
||||
| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
|
||||
| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
|
||||
| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
|
||||
|
||||
Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
|
||||
sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
|
||||
artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
|
||||
so live.
|
||||
|
||||
## Mechanisms confirmed working, live
|
||||
|
||||
R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target
|
||||
breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app
|
||||
thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of
|
||||
the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on
|
||||
demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's
|
||||
sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the
|
||||
hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable;
|
||||
F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests).
|
||||
|
||||
## Where it stopped, and what remains
|
||||
|
||||
Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host
|
||||
reboot mid-backup) was deliberately not started — a large new fault class against boxes that are
|
||||
remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown.
|
||||
|
||||
**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's
|
||||
threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time
|
||||
`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign
|
||||
forbids. Reported rather than worked around. The no-code-change alternative (age the controller's
|
||||
reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended
|
||||
method next time.
|
||||
|
||||
**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an
|
||||
intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way
|
||||
concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`.
|
||||
|
||||
## Recovery
|
||||
|
||||
Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive
|
||||
observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active),
|
||||
windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on
|
||||
either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and
|
||||
the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots).
|
||||
|
||||
One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box.
|
||||
Its `until` is long past so it blocks nothing; it clears on the next successful backup or any
|
||||
controller restart (by design, not persisted). Clearing it would have cost another app outage for no
|
||||
benefit.
|
||||
|
||||
**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in
|
||||
`/opt/docker/stacks/<app>` starts a Felhom app **without its secrets** — they are injected by the
|
||||
controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a
|
||||
crash loop during recovery. Manual recovery must go through `POST /api/stacks/<name>/restart`.
|
||||
@@ -1,71 +0,0 @@
|
||||
# REPORT — DIAGNOSE immich offsite restore (felhom.eu side: docs only)
|
||||
|
||||
> Written as `REPORT-diag-immich-restore.md`, **not** the shared `REPORT.md`, per the CLAUDE.md
|
||||
> convention — `REPORT.md` is overwritten and currently holds the 2026-07-18 website refresh.
|
||||
> No code shipped in this run; findings only.
|
||||
|
||||
**Date:** 2026-07-19 · **Box:** demo-felhom (felhom-pve guest 9201), controller 0.146.0, immich v3.0.3
|
||||
|
||||
> **CLOSED IN CODE 2026-07-19 — controller v0.148.0.** Findings 1 and 2 shipped as R-43 (offsite
|
||||
> reconstitution: safety dump → stop → overwrite files → start → replay the snapshot's dump) and
|
||||
> R-44 (every run dumps before it captures; manifests carry `offsite_run_id` + `dumps_at`). Deployed
|
||||
> to demo 9201, healthy. **The §9 live acceptance has NOT run**, so no capability-map flip: the
|
||||
> offsite row is PARTIAL, the customer-restore row stays MISSING, R-3 stays DRAFT. Implementation
|
||||
> detail lives in `felhom-controller/REPORT.md`.
|
||||
|
||||
## What ran
|
||||
|
||||
A diagnosis of "immich offsite restore succeeds but photos do not reappear". No product code was
|
||||
changed: no restore/backup logic, no labels, no flashes, no `restic prune`/`forget`, no snapshot or
|
||||
escrow changes.
|
||||
|
||||
## Outcome
|
||||
|
||||
The restore did not fail. **It was never invoked on missing data, and could not have worked
|
||||
if it had been.**
|
||||
|
||||
1. Viktor deleted the 11 photos in the immich UI to test offsite restore. A UI delete means
|
||||
**trash**, not deletion — no file left the disk. Both „csak a hiányzó fájlok" runs merged
|
||||
**0 files**, correctly, and flashed success. The test proved nothing.
|
||||
2. A *valid* test would also have failed: **no offsite path loads a database.** All three buttons
|
||||
are file-only. Files would return; the library would stay empty.
|
||||
3. The shipped dump is additionally stale by design — from the 02:30 local run, never refreshed
|
||||
before a manual push. Probed: **`asset: 0`, `user: 0`, `album: 0`**.
|
||||
|
||||
**Photos:** left in trash at Viktor's instruction (recovery not wanted). All 11 files verified
|
||||
present on disk and all 11 rows intact, so an ordinary un-trash recovers them until immich's
|
||||
30-day auto-purge.
|
||||
|
||||
**Answer to "can a customer trust same-day offsite?"** For a DB-indexed app: **no — files come
|
||||
back, content does not.** The backup half is honest; the restore half cannot reconstitute the app.
|
||||
|
||||
## Decisive evidence
|
||||
|
||||
- `updatedAt` == `deletedAt` on all 11 asset rows ⇒ **no restore operation touched the DB.**
|
||||
- Unit dump `immich-postgres.sql`, 51 954 452 B, mtime **02:30 CEST** ⇒ `asset: 0 / user: 0 /
|
||||
album: 0`. The 52 MB is entirely immich's shipped geodata reference tables. It predates the admin
|
||||
user (07:56:25) and the photos (07:57).
|
||||
- **Latent hazard:** had a full restore loaded that dump, it would have written an empty DB over the
|
||||
live one, destroying the trashed rows that were the only surviving recovery path.
|
||||
|
||||
## Files written
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `documentation/audits/DIAG-immich-restore-2026-07-19.md` | new — timeline, evidence, source mechanism table, 7 findings |
|
||||
| `documentation/backlog/ROADMAP.md` | **R-43** (P2-HIGH, offsite restore cannot restore a DB) + **R-44** (P2-HIGH, manual push ships unrefreshed dump) |
|
||||
| `documentation/architecture/00-capability-map.md` | customer-restore row **stays MISSING**, gains the finding + a method note for R-3; offsite-restore row flagged *scope contested* |
|
||||
|
||||
## Needs Viktor
|
||||
|
||||
1. **Capability-map ruling (line 61).** The offsite-restore row claims PROVEN-LIVE citing
|
||||
"immich end-to-end from offsite alone" (`CAMPAIGN-6D`). That phrase is contested by this
|
||||
diagnosis. I flagged it rather than downgrading a proven row — did 6D's accept leg exercise the
|
||||
DB half, or only the file half?
|
||||
2. **Optional real red-proof.** Emptying the trash would create genuine data loss and convert the
|
||||
dump-probe inference into a live proof. Offered, **not run** — irreversible, and the probe
|
||||
already settles it.
|
||||
3. **Unreconciled: the 704.6 MiB figure.** Measured 126 MB total on the library storage. If
|
||||
704.6 MiB came off a controller Storage page, that gap is its own defect.
|
||||
4. **Orphaned pre-v3 tree** `dccc13fe…` (~55 MB across upload/thumbs/encoded-video) stranded by the
|
||||
immich 2→3 redeploy — worth a sweep policy for major redeploys.
|
||||
@@ -1,53 +0,0 @@
|
||||
# REPORT — Megosztás diagnosis (2026-07-20)
|
||||
|
||||
Topic-scoped report (parallel-session rule: shared `REPORT.md` untouched).
|
||||
|
||||
**Run:** RUNBOOK "Megosztás diagnosis — SMB unreachable from Mac + sharing-page reload loop".
|
||||
Read-only diagnosis. **No code changes, no version bumps, no builds, no restarts.**
|
||||
|
||||
**Deliverable:** `documentation/audits/DIAG-sharing-2026-07-20.md`.
|
||||
|
||||
## Verdicts
|
||||
|
||||
* **Reload loop — ROOT-CAUSED (HIGH).** `sharingStatusHandler` (`sharing_handlers.go:246`, added in
|
||||
`b5d78d1`, controller v0.147.0, 2026-07-19) coerces `idle` → `running` whenever the samba
|
||||
container is alive. `sharing.html` L320–326 treats `running` as a one-shot job-success edge and
|
||||
calls `location.reload()` 1.2 s later. The first `tick()` fires synchronously on every page load,
|
||||
so the page reloads forever. Unconditional for any customer with sharing enabled — the Megosztás
|
||||
page is currently unusable. Proven live: 6 consecutive `/sharing/status` polls all returned
|
||||
`{"phase":"running","running":true}`, and the controller log shows **no ensure job ran at all**,
|
||||
so the phase is manufactured by that line rather than left over from a stuck job.
|
||||
* **`smb://192.168.0.162` — ROOT-CAUSED.** `.162` is the Proxmox host and never was an SMB endpoint.
|
||||
smbd runs in guest 9201 and binds `192.168.0.104:445`. `nc` from the host: `.104:445` **open**,
|
||||
`.162:445` **refused**. Wrong target, stale Finder favourite.
|
||||
* **`smb://FELHOM` — OPEN, narrowed.** NetBIOS resolution works on the wire
|
||||
(`nmblookup -B 192.168.0.255 FELHOM` → `192.168.0.104 FELHOM<00>`), but the stack advertises **no
|
||||
mDNS/Bonjour** (nothing on udp/5353; the R-6 spike selected `smbd + nmbd + wsdd` only) — the
|
||||
mechanism macOS Finder prefers. Closing this needs one probe from the Mac (`smbutil lookup FELHOM`
|
||||
/ `dns-sd -B _smb._tcp`), listed in the audit's Mac test matrix.
|
||||
* **H2 (container down/crash-looping) — RULED OUT** at the first probe: `felhom-samba` Up 3 h, clean
|
||||
logs, smbd/nmbd/wsdd all bound as `infra/samba.go` intends, live `smb.conf` matches the renderer
|
||||
with no baked address literal, no PVE firewall in the path.
|
||||
|
||||
## Findings (full table + evidence in the audit doc)
|
||||
|
||||
S-1 HIGH reload loop (XS fix: latch a `sawInFlight` flag in the JS; red-proof required) ·
|
||||
S-2 MED the UI never shows the connect address, only `\\FELHOM` — customers guess IPs ·
|
||||
S-3 MED no mDNS advertisement (image slice, needs republish) ·
|
||||
S-4 LOW ensure-job phase never resets — fold into the async-job-feedback roadmap item ·
|
||||
S-5 INFO the guest's LAN IP is DHCP, so any displayed address must be read live, never cached.
|
||||
|
||||
Recommended packaging: S-1 as an immediate patch task (it bricks a shipped page), S-2 alongside it
|
||||
if the guest IP is reachable from the sharing handler, S-3 as its own slice.
|
||||
|
||||
## Also noticed
|
||||
|
||||
The **remote site's LAN is `192.168.0.0/24` — the same prefix as the DooPlex home LAN** that the
|
||||
Tailscale subnet router advertises (`192.168.0.180` shows `FAILED` in felhom-pve's neighbour table).
|
||||
A successful `ping 192.168.0.162` therefore does not by itself prove the Mac is on the remote
|
||||
segment; the Mac matrix starts by confirming which network it is on.
|
||||
|
||||
## Actions taken
|
||||
|
||||
None. Every command was a read, except a `POST /login` to obtain a session for the status polls.
|
||||
No secrets are recorded in either document.
|
||||
@@ -1,85 +0,0 @@
|
||||
# REPORT — R-111 fixed, then E-2 proven on a fresh box (2026-07-29)
|
||||
|
||||
Two phases in one session. Full evidence: `documentation/audits/E2D-fresh-vm-2026-07-29.md`.
|
||||
Root `REPORT.md` untouched.
|
||||
|
||||
## Phase 1 — R-111: the Day-0 channel now serves the current software
|
||||
|
||||
A Phase 0 gate earlier the same day stopped the E-2d run before any VM existed: a fresh box would
|
||||
have installed **agent 0.96.0 + controller 0.161.0**, ~17 and ~24 releases behind `main`.
|
||||
|
||||
| | Before | Now |
|
||||
|---|---|---|
|
||||
| agent (Gitea generic) | 0.96.0 | **0.113.0**, sha `5f3247f7…`, round-trip verified |
|
||||
| golden (Gitea generic) | 0.161.0 | **0.185.1**, sha `dba00f3e…`, embeds controller 0.185.1 |
|
||||
| hub `min_agent` | 0.93.0 | **0.113.0** (what controller v0.185.0 declares) |
|
||||
|
||||
Bake clean on every marker: `Result=success`, overlay2, **all three mounts in the archive**, 0
|
||||
FATAL/exclusions, HTTP 201, token-leak grep 0. GL-1 teardown: guest 9100 purged, secrets shredded,
|
||||
drill disk restored to `virgin`. Agent + golden moved in **one** manifest POST so it never vouched a
|
||||
new agent against an old golden. `min_agent` verified zero-impact first (all three enrolled hosts
|
||||
already at 0.113.0). Global floor deliberately **not** raised — the golden now bakes 0.185.1.
|
||||
|
||||
Commit `3dff357`.
|
||||
|
||||
## Phase 2 — the E-2d run, full ISO/PAIRING route
|
||||
|
||||
Nested PVE VM on demo-hp, one disk, outside the `felhom` pool. Bind → running controller in
|
||||
**3 m 35 s**. The install fetched exactly the artifacts published an hour earlier and restored
|
||||
`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked 20 minutes before. The publish train is
|
||||
proven end to end on a real install.
|
||||
|
||||
| Claim | Verdict |
|
||||
|---|---|
|
||||
| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** |
|
||||
| **C2** Case B fires naturally | ✅ **PROVEN** — both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort |
|
||||
| **C3** degraded banner renders **to a customer** | ⚠️ **PARTIAL** — API byte-exact; **no UI consumer exists** → **R-112** |
|
||||
| **C4** offer appears and moves the target | ⚠️ **PARTIAL** — decline path, `restart_required:true`, no self-restart, E-2a wrapper, healthy-renders-nothing all PROVEN at API level; offer equally invisible → **R-112** |
|
||||
| **C5** `backup_target_absent` end to end | ❌ **FAILED** — zero events on any channel → **R-113** |
|
||||
|
||||
## The three findings
|
||||
|
||||
**R-112 (P1)** — E-2's banner and offer have **no UI consumer**. The endpoint returns byte-exact copy;
|
||||
`grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**, and no page handler injects
|
||||
the state. Decisive contrast: templates fetch **18** distinct `/api/storage/*` endpoints;
|
||||
`backup-target` and `backup-target/assign` are the only two with zero references. v0.185.1 fixed the
|
||||
router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired.
|
||||
|
||||
**R-113 (P1)** — the drive-absent gate **cannot fire on device loss**. `planDriveGates` reads presence
|
||||
from `BoundUnderParent` = "is this path in the guest's mountinfo". The raw mount is a device-bound
|
||||
systemd unit and dies; **the agent's own bind is not device-bound and outlives the device**, so the
|
||||
gate sees "present" forever. Live: agent said `enrolled drive absent by UUID` every 20 s for 4½
|
||||
minutes, controller logged **0** `[gate]` lines, hub got **zero** events — neither the specific nor the
|
||||
generic one. Sixth instance of the class, one layer deeper: E-2b wired the seam to a condition that
|
||||
cannot occur.
|
||||
|
||||
**R-114** — on target-drive loss the message says the backup is *"on the same disk as the system"*
|
||||
(false) and offers **the drive that just vanished**. Invisible today only because of R-112 — so
|
||||
**R-114 must be fixed before R-112 is wired.**
|
||||
|
||||
Also filed as a **second instance under R-110** (not a new ID): host-install fetches **nine** files
|
||||
from `raw/branch/main` and the hub vouches a sha for **one**; E-2a's wrapper is installed 0755 to
|
||||
`/usr/local/sbin`, root-fenced in sudoers, validated only by `bash -n`.
|
||||
|
||||
## Record
|
||||
|
||||
- `OPEN-ITEMS.md` — **R-112/R-113/R-114 opened**; E-2d re-stated with results and left open for the
|
||||
residue; E-2's "NOT yet live-proven" list resolved into proven / known-broken; R-94 fully unblocked;
|
||||
R-110 extended. The drill-cleanup row was opened and then **closed the same session** once the
|
||||
teardown completed, so it is not carried in the register.
|
||||
- `ROADMAP.md` — R-112/R-113/R-114 under P1; R-111 marked SHIPPED.
|
||||
- **`architecture/00-capability-map.md` not touched** — for two reasons: the customer-facing legs are
|
||||
broken rather than proven, and the map has **no E-2 / backup-target rows at all** (worth noting
|
||||
against the ROADMAP's coupling rule).
|
||||
|
||||
## Teardown
|
||||
|
||||
VM destroyed, scratch storage removed, **`pvesm status` after == before** (`local-lvm` 38.77 %,
|
||||
byte-identical), guest 9201 and drill-r50 untouched. **Hub records removed — teardown complete.** The delete was correctly refused at four gates while the host still read ONLINE; once the destroyed host aged to DOWN (`delete-impact` → `deletable:true`) the documented cascade ran and completed: host deleted, PBS tenancy deprovisioned, claim reset, residue purged. Verified after: **0** `e2d` occurrences on the hosts page, fleet unchanged. The one purged `appliance_registrations=1` was this run's own appliance; the unrelated stale 2026-07-25 appliance (`206c8838…`) was not touched by the cascade — the operator removed it separately.
|
||||
|
||||
## One human step, and a premise correction
|
||||
|
||||
The runbook's §5.1a operator STOP (the bind) is **retired** — CC did it. But E-2d's premise that a
|
||||
fresh install yields a CC-drivable claimable customer is **wrong**: the claim code is bcrypt-hashed and
|
||||
email-only, and the gate covers everything except `/claim`, `/api/health`, `/static/`. One operator
|
||||
relay of the emailed code was required — which also proved the claim flow end to end.
|
||||
@@ -1,92 +0,0 @@
|
||||
# REPORT — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run (RUNBOOK execution). **No code changed. No version bump.**
|
||||
Written as `REPORT-<topic>.md` per the parallel-session rule — the shared `REPORT.md` was not touched.
|
||||
|
||||
**Full record with all evidence:** `documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`
|
||||
|
||||
---
|
||||
|
||||
## Outcome: DONE and verified
|
||||
|
||||
`felhom-offsite` now lives on a dedicated 100 GB Hetzner Cloud Volume instead of ep0's 40 GB root disk.
|
||||
|
||||
| | Before | After |
|
||||
|---|---|---|
|
||||
| Path | `/srv/pbs-felhom` (root disk) | **`/mnt/pbs-datastore`** (volume) |
|
||||
| Datastore total | 37.2 GB | **98 GB** (hub gauge: 97.9 GB) |
|
||||
| Used | 28.9 % | **13 %** (hub gauge: 12.6 GB, 13 %) |
|
||||
| Headroom to the 80 % warn | 19 GB | **≈65 GB** |
|
||||
| Additional customers before warn | ≈2 | **≈7–13** |
|
||||
|
||||
Datastore **name unchanged** — the PBS-DR descriptors, per-box storage ids, ACLs and namespace
|
||||
layout that R-39/R-82 made self-healing are untouched.
|
||||
|
||||
**Window:** 06:58 → 07:19 UTC (PBS down 07:00 → 07:17). **Nothing was deleted.**
|
||||
|
||||
### Acceptance evidence
|
||||
|
||||
| Gate | Result |
|
||||
|---|---|
|
||||
| Copy integrity | 13,242,207,822 = 13,242,207,822 B · **9,748 = 9,748 chunks** · 7 = 7 snapshots · `backup:backup` · itemised dry-run **0 lines** |
|
||||
| Snapshot counts per ns | `demo-felhom` 2=2, `demo-felhom-01` 3=3, `demo-hp` 2=2 |
|
||||
| atime semantics | `rw,relatime,discard` — **`relatime` present, `noatime` absent** (GC correctness) |
|
||||
| Verify job | `TASK OK`, 3/3 groups, forced re-verification of every snapshot, **0 errors** |
|
||||
| §6 mount guard | **refusal observed** — `Job … failed with result 'dependency'`; mountpoint stayed empty |
|
||||
| §8 restore round-trip | `source_tier: pbs`, `pass: true`, `mount_parity: ok`, clean teardown, 12m1s |
|
||||
|
||||
---
|
||||
|
||||
## Three findings the operator should act on
|
||||
|
||||
1. **`scratch` datastore is configured at a path that does not exist** (`/srv/pbs-scratch`).
|
||||
Pre-existing, not caused here, but now logs `ENOENT` on every PBS start. This is the PRIME RISK
|
||||
shape ("reports fine, is not there") already live in the config. **Decision needed:** remove the
|
||||
stanza or create the directory.
|
||||
|
||||
2. **The runbook's §6 acceptance test proves the wrong proposition.** `RequiresMountsFor` is a
|
||||
mount-first ordering guarantee, not a refusal — systemd silently *re-mounts* an unmounted volume
|
||||
and PBS then starts safely. The test only bites when the device is genuinely unavailable, which
|
||||
is how it was re-run and passed. **Amendment recommended in the runbook record.**
|
||||
|
||||
3. **§11 — storage box `u629193` is NOT simply unused.** No live backup path references it (no
|
||||
datastore, no restic repo, no fstab, no `known_hosts` pin; R-17 already deleted `u629193-sub1`),
|
||||
**but ep0 carries an enabled, currently-mounted sshfs unit** `mnt-pbs\x2dstoragebox.mount` →
|
||||
`/mnt/pbs-storagebox`, holding spike leftovers. Disable and remove that unit before deleting the
|
||||
box, or ep0 logs a failed mount every boot. **The deletion is the operator's console click.**
|
||||
|
||||
## Deviations from the runbook as written
|
||||
|
||||
- **The volume arrived already formatted and mounted** by Hetzner at `/mnt/HC_Volume_106469259`
|
||||
(§2 assumed neither). Operator ruled: reformat + repath. The 5 % reserve was reclaimed (`-m 0`).
|
||||
- **§8 ran on demo-felhom, not demo-hp** — DooPlex holds no SSH key for demo-hp (the G1 gap). Same
|
||||
tier, same relocated datastore, larger archive.
|
||||
- **The window was contended** by a stale 10-minute restore-test cadence on demo-felhom: the config
|
||||
had already been reverted to 3.5 days on disk, but the cadence is read once at daemon start and
|
||||
`NRestarts=0`. Restarting the agent applied it (`cadence=84h0m0s`). The in-flight test was allowed
|
||||
to finish rather than aborted.
|
||||
|
||||
## Process errors made during this run (recorded deliberately)
|
||||
|
||||
- `rsync -aHAX` **OOM-killed** ep0 (3.7 GB RAM, no swap). Cause: a PBS `.chunks/` tree pre-creates
|
||||
all 65536 shard dirs → 75,341 inodes, and `-H` retains the whole inode map. `-H` was dropped only
|
||||
after **proving** no hardlinks exist (`-links +1` → 0; max link count → 1); PBS references chunks
|
||||
by digest, never by hardlink.
|
||||
- `/usr/bin/time -v` is not installed on ep0 → exit 127, rsync never ran, and a `| grep … || true`
|
||||
wrapper swallowed it while printing a success-looking line.
|
||||
- `rsync --version | head -1` reported a working rsync 3.4.1 as "missing" — **the §12 pipe-into-head
|
||||
trap, fourth recorded instance in this project.** Both fixed by capturing the command's own `$?`.
|
||||
|
||||
## Deferred
|
||||
|
||||
1. **Old copy retained** at `/srv/pbs-felhom` (13 GB, 9,748 chunks) as the rollback. Rollback is a
|
||||
two-line `datastore.cfg` revert. Reclaim only after a new weekly offsite backup lands on the
|
||||
volume, with explicit go-ahead.
|
||||
2. **GC not run** — now unblocked by the round-trip, but left for a separate deliberate run. No GC
|
||||
schedule is configured on this PBS at all.
|
||||
3. ~~Hub PBS-DR capacity gauge not re-read.~~ **CLOSED — verified correct.** The hub operator UI
|
||||
(Offsite → PBS DR) reports `felhom-offsite (ep0)` at **97.9 GB capacity, 12.6 GB used, 13 % full**,
|
||||
agreeing with the on-box `df`. The gauge follows the datastore's configured path, so the move
|
||||
needed no hub-side change and the suspected "wrong filesystem" bug does not exist.
|
||||
4. **ep0 has no swap** (temporary 4 GB file removed; box left as found). Worth a small permanent
|
||||
swapfile — outside this runbook's scope.
|
||||
@@ -1,63 +0,0 @@
|
||||
# REPORT — F-CRIT-1 + F-A1 fixed (controller v0.179.0, 2026-07-28)
|
||||
|
||||
Docs here. Implementation, all six red-proofs and the full live replay live in
|
||||
`felhom-controller/REPORT.md`. The campaign that found both:
|
||||
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
|
||||
## What changed
|
||||
**F-CRIT-1** — an app that failed to restart after a quiesce never alarmed, for two independent
|
||||
reasons, either of which alone kept it dead: `restartAll` returned nothing (the failure was logged
|
||||
and dropped), and `classifyRunStates` whitelisted `StateStopped` on invariant I1 ("the user stopped
|
||||
it") — which the quiesce loop had made false by stopping stacks the same `docker compose down` way.
|
||||
A failed restart and a user stop are the *same* Docker state; the only difference is that the loop
|
||||
tried and could not, now surfaced by `Loop.FailedRestarts()`.
|
||||
|
||||
**F-A1** — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a
|
||||
failure. It is now contention: the tier stays DUE, is dropped before anything stops, and unending
|
||||
contention raises its own **BLOCKED** signal rather than going silent.
|
||||
|
||||
## Bounds, justified against measured reality
|
||||
- `contentionRetryAfter` **15m** — longest restore-test observed on the fleet is 12m01s; the agent's
|
||||
local restore-test wait is 10m. Caps app-stop churn at 4/hour instead of 12/hour.
|
||||
- `contentionAlarmAfter` **3h** — the agent's own PBS restore-test task is capped at 120 minutes, so
|
||||
contention outliving that is a *stuck* gate, not a busy one. 3h adds margin and is 15× the longest
|
||||
contention actually observed.
|
||||
|
||||
## Verified live, with the hub DB as arbiter — not from logs
|
||||
Same box, same day, same event type; the only difference is 409 versus a genuine error:
|
||||
|
||||
| injection | operator emails (demo-hp) |
|
||||
|---|---|
|
||||
| **409 contention** | 8 → **8** (none) |
|
||||
| **real transport failure** | 8 → **9** |
|
||||
|
||||
And for F-CRIT-1: the failed restart alarmed **9 seconds** after grace expiry with the dashboard
|
||||
banner naming the `(stopped)` state, while a **deliberate** user stop on the same box stayed silent
|
||||
through **9** dead-app scans (the positive observable that the silence is suppression, not a dead
|
||||
detector).
|
||||
|
||||
## The rule this arc earned
|
||||
Added to **both** copies of `CLAUDE.md` (live + `documentation/runbooks/workspace-CLAUDE.md`):
|
||||
**a comment asserting an invariant needs a test pinning it, or it is a wish.** Six instances in this
|
||||
project have shipped guarantees the code did not provide — `EffectiveProtected`, `newestArchiveOn`,
|
||||
the R-97a operator-only claim, `classifyRunStates`' I1, `inflight.go`'s defer claim, and
|
||||
`quiesce.go`'s spurious-failure claim. Two were found only on live hardware, and one of those had a
|
||||
green, red-proofed test suite over a production path broken two independent ways.
|
||||
|
||||
Corollary recorded with it: prefer a test that asserts the **consequence** (does the alarm fire?)
|
||||
over one that asserts the **mechanism** (does suppression expire?). R-97b's Scenario F proved the
|
||||
mechanism; the consequence was still broken.
|
||||
|
||||
## Docs touched
|
||||
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-1 and F-A1 → SHIPPED + PROVEN-LIVE.
|
||||
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — both rows struck through, closing
|
||||
section added. **All three of the campaign's alarm findings are now closed** (F-CRIT-1, F-CRIT-2,
|
||||
F-A1).
|
||||
- `documentation/runbooks/workspace-CLAUDE.md` — the invariant rule.
|
||||
|
||||
## Still open, highest first
|
||||
**Fault 4** (restic transport interruption) — four injection approaches were defeated by
|
||||
guest-bridged networking, and it is now the most valuable follow-up: F-CRIT-2 answered the phantom
|
||||
question for PBS and left the identical question open for restic. Then **R-99** (prune never removes
|
||||
phantoms) and **F-LEAK** (a failed restore-test cannot destroy its own scratch guest — observed
|
||||
again during this work).
|
||||
@@ -1,40 +0,0 @@
|
||||
# REPORT — F-CRIT-2 fixed: a failed backup no longer looks like a fresh one (2026-07-28)
|
||||
|
||||
Scope: `felhom-agent` v0.105.0 → **v0.106.0**. Docs here. Implementation detail and the full live
|
||||
re-test live in `felhom-agent/REPORT.md`; the campaign that found it is
|
||||
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
|
||||
## What changed
|
||||
`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, and NEWEST) as a
|
||||
successful backup, so the tier read fresh, went **not due**, and was never retried — seven days of
|
||||
silence on the real 168h cadence, invisible to both the R-88 breaker (defers only *due* tiers) and
|
||||
the hub deadline monitor (reads the same freshness). It now counts only *plausibly complete*
|
||||
entries via a measured 1 MiB floor; undecidable ⇒ not counted.
|
||||
|
||||
**Size is the only tier-agnostic discriminator.** `verification` and `encrypted` are absent on every
|
||||
local (dir) archive AND on a good PBS snapshot until `verify-new` catches up — gating on either
|
||||
would have rejected 100% of local backups and produced fleet-wide backup thrash. That inverse risk
|
||||
is a first-class test, red-proofed by making the filter reject everything.
|
||||
|
||||
## Verified live, not just in unit tests
|
||||
Campaign fault 2 was replayed against the fixed agent on demo-hp — phantom created, rejected and
|
||||
announced once; the tier correctly reported DUE and backed up (4,359,968,099 B landed); and the
|
||||
inverse showed **no thrash**, with 91 scheduler ticks as the positive observable that the loop was
|
||||
alive rather than dead.
|
||||
|
||||
## Settled along the way — no retention bug
|
||||
Server-side prune does **not** count phantoms toward `keep-last`: a dry-run against three real
|
||||
snapshots plus a phantom retained two real ones plus the phantom. The feared "two phantoms ⇒ zero
|
||||
real backups" does not occur. Prune never removes them either, so they accumulate one per aborted
|
||||
upload — filed as **R-99** (LOW, hygiene), not as a retention bug.
|
||||
|
||||
## Docs touched
|
||||
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-2 → SHIPPED+PROVEN-LIVE; **R-99** filed;
|
||||
**F-CRIT-1** filed as READY-HIGHEST (Campaign 8's other HIGH finding, untouched here).
|
||||
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — F-CRIT-2 row struck through and
|
||||
a closing section added.
|
||||
|
||||
## Still open, highest first
|
||||
**F-CRIT-1** — an app that fails to restart after a quiesce never alarms, for two independent
|
||||
reasons. Then fault 4 (restic transport interruption), which this fix makes more pointed: the
|
||||
phantom question is now answered for PBS and still open for restic.
|
||||
@@ -1,140 +0,0 @@
|
||||
# REPORT — F-REBOOT + F-LEAK + F-OBS, and two investigations (2026-07-28)
|
||||
|
||||
Scope in this repo: **`scripts/felhom-host-install.sh` v1.20.0 → v1.21.0** (which is where F-LEAK's
|
||||
*actual* fix lives), plus the Campaign 8 audit doc and `OPEN-ITEMS.md`. Written as
|
||||
`REPORT-freboot-fleak-fobs.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
Code companions: `felhom-agent` v0.106.0 → **v0.110.0**, `felhom-controller` v0.179.0 → **v0.180.0**.
|
||||
|
||||
**Correction to this repo's part of the story:** v1.21.0's band-scoped ACL is *not* the final F-LEAK
|
||||
fix. It works, but only **once per slot** — PVE's destroy path calls
|
||||
`AccessControl::remove_vm_access($vmid)` (`API2/LXC.pm:906`), which deletes every ACL at `/vms/<vmid>`
|
||||
(`AccessControl.pm:1898`), so **the grant is consumed by the operation it authorises**. Found by counting
|
||||
ACL rows after the first successful teardown (`/vms/990000` → 0 grants), not by reasoning about it. The
|
||||
durable fix is agent **v0.110.0**'s band-scoped fenced destroy; v1.21.0 remains valuable because it makes
|
||||
the common case need no privileged call, and it is now the *first* of two layers rather than the only one.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom.eu d0cec9d`, `felhom-agent af1c21a`, `felhom-controller fb91c8d`, all clean. Agent `0.106.0`
|
||||
and controller `0.179.0` live on both demo boxes.
|
||||
|
||||
---
|
||||
|
||||
## host-install v1.21.0 — F-LEAK, and why the fix landed *here* rather than in the agent
|
||||
|
||||
**The finding.** A restore-test whose restore **fails** leaves a scratch guest the agent cannot destroy
|
||||
(`403 missing privilege VM.Allocate`), so a half-restored guest holds its disks until a human removes
|
||||
it and the 10-slot scratch band shrinks silently.
|
||||
|
||||
**The cause is structural, not a missing privilege in the role.** `FelhomAgentGuest` is granted at
|
||||
`/pool/felhom`, and **a guest joins that pool only when its restore completes**. A failed restore
|
||||
therefore produces a guest that exists, is in no pool, and is out of the token's reach entirely.
|
||||
|
||||
**The first fix was wrong, and its own live replay is what proved it.** Agent v0.107.0 shipped a
|
||||
teardown fallback that adopted the stranded guest into the pool and retried — reasoning from
|
||||
`Pool.Allocate` on `/pool/felhom`. It fired exactly as designed and PVE refused it:
|
||||
|
||||
```
|
||||
ERROR restore-test: pool adoption failed; left for Recover vmid=990000
|
||||
err="proxmox: PUT /pools/felhom -> HTTP 500: permission denied at /vms/990000 (missing privilege ...)"
|
||||
```
|
||||
|
||||
`PUT /pools/{pool}` **also** requires `VM.Allocate` on the VM being added. **Pool membership cannot
|
||||
bootstrap its own authority.** Removed in agent v0.108.0 rather than left in place — a path that
|
||||
provably cannot work is worse than none, because it reads as a fix.
|
||||
|
||||
**What shipped instead.** `apply_scoped_acl` now grants `FelhomAgentGuest` at each `/vms/<id>` in
|
||||
`PVE_SCRATCH_VMID_MIN..PVE_SCRATCH_VMID_MAX` (990000–990009 — the band the restore-test already picks
|
||||
from), to **both** the user and the token, because the privsep-intersection rule applies here as
|
||||
everywhere.
|
||||
|
||||
Two supporting changes, both load-bearing rather than tidy-up:
|
||||
- **`remove_scoped_acl` deletes the band grants before the role delete.** PVE refuses to delete a role
|
||||
still referenced by any ACL, so omitting this would have broken the uninstall — a failure that would
|
||||
only surface on a decommission.
|
||||
- **`step_verify` asserts the band grants.** A missing one is otherwise invisible until a restore-test
|
||||
*fails*, which is precisely the case that leaked a guest in the first place.
|
||||
|
||||
### Why the grant is still not a widening — proven live, at the seam the defect lives in
|
||||
|
||||
A real PBS restore to `990000` **without `--pool`** reproduced the exact stranded state
|
||||
(`990000 stopped`; `felhom pool members: [9201]`; `990000 in pool: False`). Then, with the agent's own
|
||||
token, same guest, minutes apart:
|
||||
|
||||
| | `DELETE /nodes/<node>/lxc/990000` |
|
||||
|---|---|
|
||||
| **grant removed** (the original defect) | `403 Permission check failed (/vms/990000, VM.Allocate)` — guest still present |
|
||||
| **grant restored** (the fix) | `200 UPID:...:vzdestroy:990000:felhom-agent@pve!agent` — guest gone |
|
||||
|
||||
And it still cannot reach anything else:
|
||||
|
||||
| target | result |
|
||||
|---|---|
|
||||
| `/vms/990010` (one past the band) | **403** `Permission check failed (/vms/990010, VM.Allocate)` |
|
||||
| `/vms/100` | **403** same |
|
||||
|
||||
`990010` does not exist and PVE **still** answered 403 rather than "does not exist" — so PVE evaluates
|
||||
**permission before existence**, which makes these genuine authorization refusals rather than artifacts
|
||||
of a missing guest. Granting at `/vms` was considered and rejected: it would authorise destroying every
|
||||
guest on the box, including a co-tenant's.
|
||||
|
||||
Applied on **both** demo boxes (demo-hp and demo-felhom) so the live fleet matches the installer.
|
||||
|
||||
**A careless step of mine, recorded rather than buried.** The probe loop also issued a live `DELETE`
|
||||
against running guest **9201**. It was refused with `500 container is running` — but the *permission
|
||||
check passed* (9201 is a pool member by design), so had the guest been stopped I would have destroyed
|
||||
the live demo guest. The scratch-band probes were the safe ones; 9201 had no business in that list.
|
||||
|
||||
---
|
||||
|
||||
## Documentation changes
|
||||
|
||||
- **`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`** — F-REBOOT, F-LEAK and F-OBS
|
||||
written up as FIXED with their live evidence, including F-LEAK's refuted first attempt (recorded
|
||||
precisely *because* it looked right), plus a new **§6b** for the follow-up investigation.
|
||||
- **`documentation/backlog/OPEN-ITEMS.md`** — three findings closed, **R-100** filed.
|
||||
|
||||
---
|
||||
|
||||
## R-100 — the investigation's finding, deliberately NOT fixed
|
||||
|
||||
**A restic offsite tier that fails every night never goes stale on the hub.** This is **F-CRIT-2's
|
||||
defect class one layer up and on the other tier** — a *failed* run resetting the freshness clock — and
|
||||
it was found by asking whether F-CRIT-2's shape existed anywhere else.
|
||||
|
||||
Both halves verified in the source, not inferred:
|
||||
- **Controller:** `o.LastRun = time.Now()` is set **unconditionally** at
|
||||
`controller/internal/backup/offbox.go:716`, *outside* the `runErr` branch. The failure is recorded
|
||||
faithfully, but into a different field — `o.LastStatus = "error"` at `:725`.
|
||||
- **Hub:** `isStale()` reads **only** `off.LastRun` (`hub/internal/monitor/offsite.go:120`, `:127`,
|
||||
`:131`) and never consults `LastStatus`.
|
||||
|
||||
So a nightly restic run that fails every night keeps `LastRun` fresh, `isStale` is permanently false,
|
||||
and the staleness alarm never fires — with no successful offsite backup having occurred at all.
|
||||
|
||||
**Scope of the silence, stated precisely rather than dramatically.** `LastStatus` *does* reach the hub —
|
||||
it is parsed into the report struct and **only logged** (`offsite.go:270`); it drives no checker and no
|
||||
notification. The controller's own guest UI surfaces `LastStatus="error"`, so the failure is visible to
|
||||
someone who looks. What is missing is the **push**: the operator's fleet-wide alarm plane is silent,
|
||||
which is the plane that matters for an unattended appliance.
|
||||
|
||||
Not fixed, per this task's investigation-only scope. Fix direction: gate staleness on the last
|
||||
*successful* run rather than the last attempt — exactly what F-CRIT-2's `NewestArchiveTime` fix did for
|
||||
the PBS tier.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
`bash -n scripts/felhom-host-install.sh` clean. The ACL behaviour was verified **live on real hardware**
|
||||
rather than by dry-run, since the whole finding is about what PVE's authorizer actually does — and the
|
||||
live run is what refuted my first design.
|
||||
|
||||
## Fleet state
|
||||
Agent **0.110.0** (with the updated sudoers) and controller **0.180.0** on both demo boxes, all healthy.
|
||||
Scratch-band ACLs at 20 rows on both — re-applied after the attempt-2 destroy consumed one. No leftover scratch guests. demo-hp's `restore_test_cadence_seconds` reverted **600 → 302400**
|
||||
(a bounded change made for the replay).
|
||||
|
||||
`felhom.eu`: this repo has a **foreign uncommitted WIP file** (`documentation/PROMPT-TEMPLATE.md`) from
|
||||
another session in the shared worktree. Left untouched; my commits staged explicit paths only, per the
|
||||
never-`git add -A` rule.
|
||||
@@ -1,119 +0,0 @@
|
||||
# REPORT — ISO boot branding + single-entry GRUB menu (R-38) · website grid restored (2026-07-19)
|
||||
|
||||
> `REPORT-<topic>.md` per this repo's parallel-session rule: another session was writing in this
|
||||
> clone tonight (CAMPAIGN 7 / `DIAG-immich-restore-2026-07-19.md`), so the shared `REPORT.md` is left
|
||||
> untouched.
|
||||
|
||||
Parts 1 and 2 of the polish train. Parts 3 and 4 landed in `felhom-agent` and `felhom-controller`;
|
||||
see their own `REPORT.md`.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — scripts v1.22.0: GRUB branding + single-entry safety (R-38)
|
||||
|
||||
### What shipped
|
||||
|
||||
Every ISO is now **repacked** after `prepare-iso`. `mkimage-surgery.sh` → **`iso-repack.sh`**:
|
||||
branding and the slice-B loader swap need the same extract → modify → re-master cycle, so they share
|
||||
one pass instead of re-mastering twice. **The mkimage recipe is untouched.**
|
||||
|
||||
**The safety half — the one that matters.** The stock PVE menu offers *Graphical*, *Terminal UI*, a
|
||||
serial variant, and an **Advanced Options** submenu holding two `nomodeset` entries, three debug
|
||||
entries, *Rescue Boot*, memtest and *UEFI Firmware Settings*. Every one of those reaches the
|
||||
**manual** installer, whose first question is which disk to wipe. They are **not emitted** — not
|
||||
hidden, not password-gated. What ships is one entry, „Felhom telepítés", default, 5 s.
|
||||
|
||||
**Boot behavior is unchanged.** The `linux`/`initrd` lines are lifted **verbatim at repack time**
|
||||
from the ISO's own *Install Proxmox VE (Automated)* entry rather than frozen into a copy in this
|
||||
repo, so a PVE bump that moves the kernel path or edits the append line tracks automatically. The
|
||||
build **fails** if they cannot be found, if the append line has lost `proxmox-start-auto-installer`,
|
||||
or if `auto-installer-mode.toml` is absent — that last one because without it the single
|
||||
Felhom-labelled entry would boot a *manual* installer, i.e. exactly what this change prevents.
|
||||
|
||||
**Gates, then a re-check against the shipped artifact.** The rendered menu is asserted to have
|
||||
exactly 1 `menuentry`, 0 `submenu`s and no *live* reference to
|
||||
`proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` (comments are stripped first —
|
||||
the template's header names the dropped entries deliberately). Then the menu and theme background
|
||||
are read back **out of `final.iso`**, not out of the extract tree.
|
||||
|
||||
**The boot card.** `grub/generate-grub-background.sh` letterboxes `website/assets/og-image_2.png`
|
||||
onto a 1024×768 gfxterm canvas at repack time (ImageMagick added to the assistant image), so the boot
|
||||
screen has **one source** and not a second pre-rendered PNG to drift. The card's own subtle grid
|
||||
(measured: 4px lines of `#0D131A` on `#0D1117`, pitch 131px) is continued across the letterbox fill
|
||||
**phase-locked** to where the card's grid lands, so the fill is seamless instead of a 500px square of
|
||||
grid floating in flat navy. The generator refuses a source whose geometry no longer matches the
|
||||
measured constants — a swapped asset would misplace every line, and that only shows up on a boot
|
||||
screen nobody re-checks. Menu positioning needs a gfxmenu theme (plain `background_image` cannot move
|
||||
the menu off the wordmark), so `grub/felhom-theme.txt` puts it in the lower third the layout leaves
|
||||
empty, optically centered (measured off a canary screenshot; the comment records the measurement).
|
||||
|
||||
### Live validation — nested canary, UEFI/OVMF, PVE 9.2-1
|
||||
|
||||
Booted the built canary ISO under QEMU with OVMF and captured the framebuffer.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| GRUB renders the Felhom card | **PASS** — background + grid visible at 1024×768 |
|
||||
| Exactly one entry, selected | **PASS** — „Felhom telepítés" only |
|
||||
| Hungarian accents under gfxterm | **PASS** — „telepítés", „Indítás … másodperc múlva" render correctly |
|
||||
| Countdown visible and counting | **PASS** — 5 → 0 |
|
||||
| Auto-fires at 0 | **PASS** — serial shows ``Booting `Felhom telepítés'`` |
|
||||
| Unattended install proceeds | **PASS** — „Fetching answers for automatic installation" → auto installer |
|
||||
| Same abort as v1.21.0 | **PASS** — `ERROR: Installation failed: filter did not match any device` → `Installation aborted`; no disk touched |
|
||||
|
||||
A first build **correctly failed closed**: the banned-entry gate matched the template's own
|
||||
explanatory header. Fixed to strip comments before matching (a comment naming a removed entry is the
|
||||
point; a directive using one is the bug), which is a gate behaving as designed.
|
||||
|
||||
### Artifacts (rebuilt on 180, `/mnt/5_hdd/felhom.eu/felhom-iso/out/`)
|
||||
|
||||
| ISO | sha256 | bytes |
|
||||
|---|---|---|
|
||||
| `felhom-pve-9.2-1-v1.22.0-n100-generic-mkimage.iso` (safety) | `ff6f06ba1dbfe10f27d703afc29516001000349147426b43c9a424a0ea28bdbf` | 1 704 482 816 |
|
||||
| `felhom-pve-9.2-1-v1.22.0-n100-demo-generic-mkimage.iso` (real) | `494db0ddf859b6b152cad4d0e0d9e9cefd27255cde07e2b41aba3ac12a217888` | 1 704 482 816 |
|
||||
| `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso` (validation) | `83c61c0413c84e27b26a37bb5dfaed2fcd44fd25e3e571c7310142bd305f2f9d` | 1 705 338 880 |
|
||||
|
||||
Both shipping ISOs: `embedding 60 modules`, `El Torito boot images=2`, fs-uuid preserved, and the
|
||||
post-re-master verification confirming 1 entry + theme background inside the finished image.
|
||||
|
||||
**Deliberately not done** (per the task): no squashfs/initrd rebranding — post-GRUB screens are still
|
||||
Proxmox-branded; no disk-setup or answer-generation change; the pairing banner is untouched.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — website: the index grid background, restored
|
||||
|
||||
**Archaeology.** Not a deliberate removal. The grid lived as a fixed `body::before` in
|
||||
`index.html`'s inline `<style>` block and was dropped in **`bed8675`** ("D3 Part 2: index + kapcsolat
|
||||
on design system v2"), the commit that migrated the page onto the shared `assets/site.css`.
|
||||
`dd54e4c`, which *created* `site.css`, has no `body::before` at all — it was a porting omission and
|
||||
nothing took its place. `ccbb13a` (the other five pages) never had it. No asset was lost: the
|
||||
mechanism was pure CSS (two stacked `linear-gradient`s), which is why nothing looked missing in the
|
||||
worktree.
|
||||
|
||||
**Restoration, not redesign.** Same 50px cells, same 1px lines, same 3% opacity, same
|
||||
`position:fixed` / `z-index:-1`. One deliberate difference: the accent is the v2 `--blue` `#0083D8`
|
||||
instead of the retired legacy `#0088cc`, which `site_gates.py` bans. Scoped to `body.page-index`,
|
||||
because index is the only page that ever had it. `site.css` cache-bust bumped `?v=1` → `?v=2` across
|
||||
all seven pages (nginx caches 7 days); BOM preserved on every file.
|
||||
|
||||
**Live verification** (felhom.eu, after git-sync deploy):
|
||||
|
||||
- Desktop: grid renders behind the hero, at its original subtlety.
|
||||
- **376px viewport** (via a same-origin iframe — the browser window would not resize in this
|
||||
environment, so the narrow case was exercised for real rather than asserted): grid renders, mobile
|
||||
layout unchanged, `scrollWidth === clientWidth` so **no horizontal overflow**.
|
||||
- Computed style confirmed live: `linear-gradient(rgba(0,131,216,0.03) 1px, …)`, `50px 50px`,
|
||||
`position: fixed`, `z-index: -1`, `pointer-events: none`.
|
||||
- `python scripts/site_gates.py` — **OK** (BOM, no legacy tokens, no `<style>` blocks, cache-busted).
|
||||
|
||||
---
|
||||
|
||||
## Docs
|
||||
|
||||
- `scripts/CHANGELOG.md` — v1.22.0 entry.
|
||||
- `website/CHANGELOG.md` — grid restoration entry.
|
||||
- `documentation/backlog/ROADMAP.md` — **R-38 flipped to SHIPPED**; **R-45** (unified async-job
|
||||
feedback) and **R-46** (verification-copy browse + expiry) added; pre-invite checklist gained the
|
||||
"golden ≥ 0.147.x carries all four infra images" line.
|
||||
- Capability map: **untouched** — no capability moved. These are UX and packaging.
|
||||
@@ -1,307 +0,0 @@
|
||||
# REPORT — the universal ISO: **PUBLISHED** (2026-07-31)
|
||||
|
||||
**Live:** `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso`
|
||||
**sha256:** `f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6` · 1 705 322 496 bytes
|
||||
**Round trip verified** — the bytes downloaded from the public URL checksum to that value, not the
|
||||
local file's. `.sha256` and manifest published beside it.
|
||||
|
||||
> Written as `REPORT-iso-release.md`, not root `REPORT.md`, per the task and the shared-clone rule.
|
||||
|
||||
## 0. Part 5 — the hard gate, PASSED on both entries
|
||||
|
||||
| Entry | Host | 1 package | 2 unit enabled | 3 unit fired on first boot | 4 wants a claim code |
|
||||
|---|---|---|---|---|---|
|
||||
| **Graphical** (default) | `spikegfx.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | `activating`; *"registering unclaimed appliance at the hub"* | **`J7N-2DA`**, token 64 B mode 600 |
|
||||
| **Terminal UI** | `spikesix.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | same | **`ZY5-YY4`**, token 64 B mode 600 |
|
||||
|
||||
Both: normal manual install, own disk chosen in the installer, own root password, real completion
|
||||
signal (installer wrote ~7 GB and rebooted; the installed system was then reached over SSH). Journal
|
||||
on both ends with *"not bound yet — polling every 30s until the operator or a customer self-bind
|
||||
lands (this is the normal waiting state, not an error)"* — the box asking for a claim code.
|
||||
|
||||
Spike 4 reasoned the graphical path would follow from shared `Install.pm`. **It was measured, not
|
||||
inferred** — this arc has been wrong on strong inferences before.
|
||||
|
||||
## 1. Venue and baselines
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Host** | `demo-hp` (t740), Tier 0 |
|
||||
| **VMs** | **500 `spike5-gfx`**, **501 `spike5-tui`** — both created with `qm` so the run is visible in the web console |
|
||||
| **Storage** | **`spike5`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images`. Root chosen deliberately: a storage at a *subdirectory* reads `disconnected` forever via the agent's `exactMount` check. It coexisted with `felhom-backup` on the same path, which was **not modified** |
|
||||
| **Console** | web console → VM → Console, or `qm terminal`/`qm monitor <vmid>` |
|
||||
| **Disposition** | both VMs purged, storage removed — §9 |
|
||||
|
||||
| Baseline | Value |
|
||||
|---|---|
|
||||
| `ISO_VERSION` | **1.26.0** (was 1.25.0) — `scripts/iso/build-felhom-iso.sh:51` |
|
||||
| `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` |
|
||||
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
|
||||
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
|
||||
| controller on `main` | `0.188.0` (`4115e88`) |
|
||||
| `felhom.eu` HEAD at build | clean, pushed, `== origin/main` |
|
||||
|
||||
## 2. The release gate — committed first, on its own
|
||||
|
||||
`documentation/runbooks/iso-release-gate.md`, commit **`e787391`**, written and pushed **before the
|
||||
first build** so it could not be rationalised afterwards. Twelve criteria, each checkable against the
|
||||
uploaded file rather than the build inputs, each carrying the spike measurement that justifies it.
|
||||
|
||||
**One criterion was amended before the build, with its reasoning recorded in the runbook.** G6 was
|
||||
first written with the six-token ban `iso-repack.sh:160-164` enforces, on the rationale *"no live route
|
||||
to a manual disk-picker"*. That rationale is obsolete for a public image — the ruling makes the manual
|
||||
installer **the product**. `proxtui` (the Terminal-UI installer we deliberately ship) and `nomodeset`
|
||||
(its graphics fallback) are dropped **for release images only**; `proxdebug`, `Rescue Boot`, `memtest`
|
||||
and `fwsetup` stay banned in both modes, and the six-token list is **unchanged** for appliance images.
|
||||
|
||||
## 3. The stub package
|
||||
|
||||
`scripts/iso/pkg/` — source committed, built by `build-deb.sh`.
|
||||
|
||||
**Contents: exactly two files, deliberately not three.**
|
||||
```
|
||||
-rwxr-xr-x ./usr/local/sbin/felhom-bootstrap.sh
|
||||
-rw-r--r-- ./lib/systemd/system/felhom-bootstrap.service
|
||||
```
|
||||
The old first-boot stub also wrote `/etc/felhom/bootstrap.env` (0600). This package does not:
|
||||
`felhom-bootstrap.sh:91` reads it only `if [[ -r ]]`, and its defaults at `:95-96`
|
||||
(`https://hub.felhom.eu`, `https://felhom.eu/scripts/felhom-host-install.sh`) are **exactly** what the
|
||||
generic pairing env set (`build-felhom-iso.sh:257-258`). Shipping it would add a 0600 file to a public
|
||||
package to express values the script already defaults to.
|
||||
|
||||
**Dependencies: none, and that is a finding.** `dpkg-deb -I` shows no `Depends` line. The payload is a
|
||||
shell script and a unit file; the binaries the script calls (`curl`, `ip`, `dhclient`, `python3`,
|
||||
`systemctl`) run at **first boot**, not at postinst time. **Spike 4's open `dpkg --configure -a`
|
||||
ordering question therefore does not arise** — confirmed, not carried.
|
||||
|
||||
**How the postinst is structurally incapable of failing** — no `set -e`, every statement individually
|
||||
guarded with `|| true` or an `if`, and an unconditional `exit 0`. `build-deb.sh` refuses to emit a
|
||||
package that violates any of it.
|
||||
|
||||
**The guarantee was tested, not asserted.** Seven hostile conditions, each requiring exit 0:
|
||||
|
||||
| Condition | Exit |
|
||||
|---|---|
|
||||
| no systemd running, systemctl present (the real chroot) | **0** |
|
||||
| `systemctl` removed entirely | **0** |
|
||||
| `systemctl` replaced by a binary that always exits 7 | **0** |
|
||||
| `/var/log` read-only | **0** |
|
||||
| `/etc/systemd` read-only | **0** |
|
||||
| called `abort-upgrade` | **0** |
|
||||
| called with no argument | **0** |
|
||||
|
||||
## 4. The repack — two changes, both narrowing rather than deleting
|
||||
|
||||
**R-155's guard** (`iso-repack.sh:100-106`) **protected the single-entry mode's promise**: that menu
|
||||
shows one item labelled "Felhom telepítés" which boots the *automated* installer, and without
|
||||
`auto-installer-mode.toml` the same label would drop the user into a manual disk-picker — a button
|
||||
promising an unattended install that silently does the opposite. That promise is real, so the guard is
|
||||
**kept unchanged for `FELHOM_MENU=single`** and simply does not apply to `release`, where the absence
|
||||
of that file is release-gate criterion G1 rather than a defect.
|
||||
|
||||
**The menu collapse** happens at `iso-repack.sh:144-148` (the stock `grub.cfg` is replaced by a
|
||||
rendered template). A `release` template now renders **two interactive entries**; entry-count and
|
||||
banned-token gates are per-mode; the post-remaster verification reads the count back out of
|
||||
`final.iso`.
|
||||
|
||||
**Ruling — default entry and timeout.** Default is **the graphical interactive entry**; timeout **15 s**.
|
||||
Reasoning: Spike 1 measured that no automated disk selection can be safe on unseen hardware (no
|
||||
property distinguishes an internal disk from a customer's backup drive; a two-disk match silently wipes
|
||||
one), so a public image whose default is unattended puts the unsafe path in front of anyone who boots
|
||||
and walks away. And Spike 2 lost a probe to a **1-second** menu — a person reading two options needs
|
||||
longer than a machine.
|
||||
|
||||
**The automated entry is absent, not broken.** Skipping `prepare-iso` means no
|
||||
`auto-installer-mode.toml`, and the stock `grub.cfg` emits the Automated entry only inside
|
||||
`if [ -f auto-installer-mode.toml ]`. There is no entry that could fail in front of a customer.
|
||||
|
||||
## 5. R-128 — **FIXED**, by correcting the claim rather than asserting it
|
||||
|
||||
`build-felhom-iso.sh:44` claimed `ISO_VERSION` "aligns with felhom-host-install `SCRIPT_VERSION`".
|
||||
Nothing evaluated it and the two had drifted. **I did not turn it into a real assertion, because the
|
||||
coupling it claimed does not exist:** the ISO is a frozen artifact, while `felhom-host-install.sh` is
|
||||
fetched at run time from the website's git-sync of `main` (R-94/R-110), so whatever version an ISO
|
||||
carries, the script a box runs is always current. An assertion would invent a constraint. The comment
|
||||
now states the independence, and `ISO_VERSION` is `1.26.0`.
|
||||
|
||||
## 6. Part 5 — the defect, the fix, and where it now stands
|
||||
|
||||
### Round 1 (`1.26.0`) — the Terminal UI install FAILED on observable 4
|
||||
|
||||
Three of four passed: the package installed, the unit was enabled from inside the installer chroot,
|
||||
and the unit **fired on first boot** and registered at the hub. The fourth failed:
|
||||
|
||||
```
|
||||
felhom-bootstrap.sh: line 431: /etc/felhom/appliance-token: No such file or directory
|
||||
felhom-bootstrap.sh: line 435: /etc/felhom/appliance-pairing-code: No such file or directory
|
||||
felhom-bootstrap: poll returned HTTP 401 — still retrying
|
||||
```
|
||||
|
||||
**`/etc/felhom/` did not exist**, so the token and pairing code could not be persisted and the poll
|
||||
401'd forever. No claim code would ever appear.
|
||||
|
||||
**Root cause, mine.** `stub-first-boot.sh` opened with
|
||||
`install -d -m 0755 /etc/felhom /usr/local/sbin`. §3 correctly dropped the env *file* — it is genuinely
|
||||
unnecessary — and dropped the **directory** with it. `felhom-bootstrap.sh` uses `/etc/felhom/` for its
|
||||
runtime state.
|
||||
|
||||
**Why the gate missed it.** G9 proves the packaged script is byte-identical to repo HEAD, and it was.
|
||||
**I verified the payload files and never the directory the payload writes into** — a check that proves
|
||||
the thing present and not the thing it depends on.
|
||||
|
||||
### The fix, and its red-proof
|
||||
|
||||
`build-deb.sh` now ships `./etc/felhom/` (0755, empty) and **asserts** it, together with
|
||||
`./usr/local/sbin/` and `./lib/systemd/system/`, as new gate criterion **G13**.
|
||||
|
||||
**Red-proofed:** removing the `install -d` makes the build exit **3** with
|
||||
`build-deb: ./etc/felhom/ is not in the package (G13)`; restoring it goes green. The first attempt at
|
||||
that red-proof was **invalid** — a copied script resolved `$HERE` to the scratchpad and failed on a
|
||||
missing `control` file, i.e. non-zero for the wrong reason — and was redone in place.
|
||||
|
||||
### Round 2 (`1.26.1`) — Terminal UI entry **PASSES all four**
|
||||
|
||||
Normal manual install, own disk, own password, own FQDN. Host `spikesix.felhom.eu`.
|
||||
|
||||
| # | Observable | Result |
|
||||
|---|---|---|
|
||||
| 1 | the `.deb` is installed | **PASS** — `ii felhom-bootstrap 1.26.1 all` |
|
||||
| 2 | the unit is enabled | **PASS** — `enabled` |
|
||||
| 3 | the unit fired on first boot | **PASS** — `activating`; journal shows *"PAIRING mode (generic ISO, no baked customer/passphrase)"* → *"registering unclaimed appliance at the hub"* → *"registered — appliance token stored (0600)"* |
|
||||
| 4 | **the box wants a claim code** | **PASS** — `/etc/felhom/appliance-pairing-code` = **`ZY5-YY4`**; `appliance-token` present, 64 B, mode `600`; journal: *"not bound yet — polling every 30s until the operator or a customer self-bind lands (this is the normal waiting state, not an error)"* |
|
||||
|
||||
That is the product working end-to-end from a public image on a manual install: own disk, own
|
||||
password, nothing baked, and the box asking for a claim code.
|
||||
|
||||
### The Graphical entry — **NOT COMPLETED**, and this is why nothing is published
|
||||
|
||||
It reached the installer from the same image (KVM dialog, EULA, and the **Target Harddisk** screen
|
||||
showing `/dev/sda (20.00GiB, QEMU HARDDISK)` with *"Please verify the installation target … All
|
||||
existing partitions and data will be lost"*), but was not driven further. `Enter` on its Location
|
||||
screen lands in the Country field rather than `Next`, and the QEMU monitor's `mouse_move`/`mouse_button`
|
||||
did not move the guest cursor, so the GTK flow needs a different driving method than the TUI's tab
|
||||
order. **Part 5 requires both entries. It is not fully passed, so Part 7 did not run.**
|
||||
|
||||
The `.deb` path lives in `Install.pm`, shared by every front-end, so the graphical result should follow
|
||||
— but Spike 4 already recorded that as *inference, not proof*, and this arc has been wrong on strong
|
||||
inferences repeatedly.
|
||||
|
||||
### A fixture bug of mine, recorded twice because it cost two diagnoses
|
||||
|
||||
`qm set <vmid> --scsi0 … --boot order="scsi0;ide2"` silently produced `boot: order=net0;ide2` — PVE
|
||||
processed `--boot` before `--scsi0` existed. Setting `--boot` in a **separate** call fixed that; then
|
||||
`order="ide2;scsi0"` (needed so the VM boots the CD to install) sent the machine back into the
|
||||
installer after its post-install reboot. **Detach the CD, or flip the order to `scsi0`, once the
|
||||
install completes.** Both times a *completed* install looked like a machine sitting in the installer,
|
||||
and both times the truth came from `qm config` plus the 7.0 GB disk rather than from the screen.
|
||||
|
||||
## 7. Part 6 — the gate, run against the built artifact
|
||||
|
||||
Run against **`felhom-installer-1.26.1-pve9.2-1.iso`**,
|
||||
sha256 **`f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6`** — the image the
|
||||
Terminal-UI install in §6 was performed from, and the one that would be uploaded.
|
||||
|
||||
| # | Criterion | Scanned for | Result |
|
||||
|---|---|---|---|
|
||||
| **G1** | no `answer.toml` / `auto-installer-mode.toml` | both names at ISO root | **PASS — 0** |
|
||||
| **G2** | no root password or hash | `.rootpw.txt` companion; the answer file that would carry a hash | **PASS** — no `.rootpw.txt` emitted; no answer file exists to hold one |
|
||||
| **G3** | no SSH key | `root-ssh-keys`, `ssh-rsa`, `ssh-ed25519` | **PASS** — no answer file; package carries only a script and a unit |
|
||||
| **G4** | no customer identity | `FELHOM_CUSTOMER_ID`/`RETRIEVAL_PASSPHRASE` with values, claim code, api key, Bearer | **PASS** — only the empty initialisers at `felhom-bootstrap.sh:89` |
|
||||
| **G5** | credential scan **by enumeration** vs the stock PVE ISO | full recursive file-list diff, both directions | **PASS** — exactly **four** added paths: the three `felhomtheme/` files and `/proxmox/packages/felhom-bootstrap_1.26.0_all.deb`; three removed (`pvetheme/`) |
|
||||
| **G6** | menu present, both paths, human timeout | entry count, `set default`/`timeout`/`timeout_style`, banned tokens | **PASS** — 2 entries, `default=0` (graphical), `timeout=15`, `timeout_style` underscore |
|
||||
| **G7** | one `felhom-*.deb`, version recorded | `/proxmox/packages/felhom-*` | **PASS** — exactly 1, `Package: felhom-bootstrap`, `Version: 1.26.0`, **no `Depends`** |
|
||||
| **G8** | postinst cannot fail | live (comment-stripped) `systemctl start\|daemon-reload\|restart`, network commands, `set -e`, last line | **PASS — 0, 0, 0**, ends `exit 0` |
|
||||
| **G9** | `felhom-bootstrap.sh` == repo HEAD | sha256 of the packaged file vs the repo file | **PASS** — both `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
|
||||
| **G10** | build inputs committed | `git status --porcelain`, HEAD vs origin | **PASS** — clean and pushed at build time |
|
||||
| **G11** | published checksum + round trip | — | **NOT RUN** — nothing was published |
|
||||
| **G12** | bucket stays private | — | **NOT RUN** — the bucket was never touched |
|
||||
| **G13** | *(new, from Part 5's failure)* every directory the payload writes into is in the package | `./etc/felhom/`, `./usr/local/sbin/`, `./lib/systemd/system/` in `dpkg-deb -c` | **PASS** — all three present in `felhom-bootstrap_1.26.1_all.deb`; asserted by `build-deb.sh` and red-proofed |
|
||||
|
||||
**A gate refinement found by running it.** G7 also asked that the ISO's copy of the `.deb` sha256-match
|
||||
the package built from source. It does not, and cannot: `dpkg-deb` embeds build timestamps, so two
|
||||
builds of identical source differ. **G9 — the payload's identity — is the meaningful check**, and it
|
||||
passes. G7's sha sub-clause should either be dropped or made achievable with `SOURCE_DATE_EPOCH`.
|
||||
|
||||
## 8. Publication — done, and verified by round trip
|
||||
|
||||
Uploaded with `rclone` **in a container, configured entirely by environment variables**, so no
|
||||
credential file was ever written to disk — the fence asks for config files to be kept out of repo
|
||||
paths and removed at teardown; none was created to remove. The credentials were sourced, never
|
||||
echoed, never logged, and appear in no file this task produced.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| objects in the bucket | the ISO (1 705 322 496 B), `.sha256` (103 B), `.manifest.txt` (2 492 B) |
|
||||
| **round trip** | `curl https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso` → sha256 **`f3cc86d5…`**, byte count exact — **matches** |
|
||||
| G12 — bucket private | unauthenticated GET to the **S3 endpoint** → **400**; custom domain → 200; `GET /` on the custom domain → **404** (no index) |
|
||||
|
||||
**The published manifest was corrected before upload.** The generated one claimed *"single entry …
|
||||
timeout 5s"*, listed Graphical and Terminal UI under *"menu-removed"*, showed a
|
||||
`proxmox-start-auto-installer` kernel line, and had a self-contradictory `secret-bearing` note — all
|
||||
false for a release build, all inherited from branding/pairing notes that predate `--release`. The
|
||||
generator is fixed and the sidecar regenerated. **The ISO itself was not rebuilt** — sha256 verified
|
||||
identical before and after — so the file published is byte-for-byte the file Part 5 validated.
|
||||
|
||||
## 9. Teardown
|
||||
|
||||
**demo-hp:** VMs 500/501 `qm destroy --purge`; **scratch storage `spike5` removed**
|
||||
(`storage.cfg` back to 4 entries, `grep -c spike5` = 0); `/mnt/nvme-1tb/images/` empty; usage
|
||||
**6.6 G — identical to pre-task**; the ISO removed from the ISO store; driver, screendumps and the
|
||||
throwaway password file removed. `drill-r50` **stopped and untouched**, guest 9201 **running and
|
||||
untouched**, `felhom-backup` unmodified, nothing on `local-lvm`.
|
||||
|
||||
**demo-felhom:** not contacted.
|
||||
|
||||
**DooPlex:** scratchpad 84 K; build logs and the package build tree removed. `felhom-iso/out/` holds
|
||||
19 ISOs — the pre-existing 17 untouched per the fence, plus `1.26.0` and `1.26.1`, both unpublished
|
||||
and **neither with a `.rootpw.txt`**, which is G2's own evidence. Repo tree clean and pushed.
|
||||
|
||||
### Hub-side — **cleared**
|
||||
|
||||
Observable 4 works *by* the box registering itself, so each proof install created an unclaimed
|
||||
appliance. All three were discarded: **16** and **17** (the 1.26.0 round), then **18** (the two
|
||||
1.26.1 proofs). `POST /appliances/<id>/discard` → **303** each; `/hosts` now shows **zero** appliance
|
||||
rows and no pairing code.
|
||||
|
||||
The endpoint is `/discard`, **not** `/delete` — `hub/internal/web/server.go:345`, POST only. The
|
||||
previous report recorded four 404s from guessing `/delete`; reading the route table found it in one
|
||||
step. **R-131 gains no row.**
|
||||
|
||||
## 10. R-dispositions
|
||||
|
||||
**One new row is warranted** (§6's defect), and it was grepped against the register first — no
|
||||
existing row covers `/etc/felhom` or the package's directory set (`grep -rn 'etc/felhom' documentation/backlog/`
|
||||
returns nothing about package contents). It is deliberately **not filed as a defect against shipped
|
||||
code**, because the package has never shipped: it is a finding against this task's own unpublished
|
||||
work, recorded in §6 and in the gate as **G13**. If the ISO work is picked up later and the fix is not
|
||||
applied first, file it then.
|
||||
|
||||
Otherwise, no new rows. Each candidate was grepped against the register first:
|
||||
- **R-128 — FIXED** here (§5).
|
||||
- **R-155 — RESOLVED** here (§4): the guard is narrowed, not deleted.
|
||||
- **R-154** (`[first-boot]` is automated-only and nothing in the tree says so) — **addressed in code
|
||||
rather than by a row**: `pkg/build-deb.sh`'s header and `grub-release.cfg.tmpl` both state it with
|
||||
the measurements. The register row can close when the docs land.
|
||||
- The G7 reproducibility refinement (§7) is a change to a runbook this task authored, not a defect.
|
||||
|
||||
## 11. What did not happen, and what is still open
|
||||
|
||||
- **Part 8 partially done.** The release-gate runbook (`e787391`), `day0-install.md` C.0 (ISO vs
|
||||
manual, and when to use which) and `scripts/CHANGELOG.md` are written. **`OPEN-ITEMS.md` /
|
||||
`ROADMAP.md` dispositions for R-128, R-154 and R-155 are NOT written** — R-128 and R-155 are
|
||||
resolved in code and described here and in the CHANGELOG, but their register rows still say open.
|
||||
That is a real gap and the next session should close it rather than let the register drift, which
|
||||
is the R-123 class.
|
||||
- **The `.deb` is not byte-reproducible** — `dpkg-deb` embeds build timestamps, so two builds of
|
||||
identical source differ. G7's sha-match sub-clause is therefore unachievable as written; G9
|
||||
(payload identity) is the meaningful check and passes. Either drop the sub-clause or set
|
||||
`SOURCE_DATE_EPOCH`.
|
||||
- **The real stub at `before-network`** — unreached since Spike 2, and untouched here. It is now
|
||||
narrower than it was: on the `.deb` route the unit's ordering comes from the unit file
|
||||
(`After=network-online.target …`), not from `[first-boot].ordering`, so it governs operator-built
|
||||
appliance images only.
|
||||
- **Secure Boot** was not exercised. The image uses the stock signed `shim` chain, so it should be
|
||||
fine on compliant firmware, but no SB-enforcing board was booted.
|
||||
- **Only virtual hardware** was tested. Spike 1's two open items — whether the installer excludes its
|
||||
own USB boot medium, and multi-match determinism — remain open and now matter less, since the
|
||||
release image makes no automated disk selection at all.
|
||||
@@ -1,71 +0,0 @@
|
||||
# REPORT — PBS prune moved server-side, write proof closed (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run. **No code, no version bump.** Topic-scoped per the
|
||||
parallel-session rule; shared `REPORT.md` untouched.
|
||||
|
||||
**Full record:** `documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`
|
||||
|
||||
---
|
||||
|
||||
## Outcome — all parts complete
|
||||
|
||||
| Part | Result |
|
||||
|---|---|
|
||||
| 1 — prune gate | **Config-gated.** `keep_last: 0` on the PBS tier, both boxes → `prune_pbs_allowed=false`. No code, **no grant** |
|
||||
| 2 — prune jobs | 2 jobs, per live namespace, `keep-last 2`, daily **03:30 UTC / 05:30 CEST** |
|
||||
| 3 — dry run → real | Gate passed; both `TASK OK`; demo-hp 3→2, demo-felhom untouched |
|
||||
| 4 — write proof | **CLOSED — `TASK OK`, no job errors** |
|
||||
| 5 — GC | Scheduled `sun 04:30 UTC / 06:30 CEST`. **NOT run** |
|
||||
| 6 — `verify-new` | **Enabled** (operator ruling) |
|
||||
| — legacy ns | `demo-felhom-01` deleted with its ACLs + token (operator ruling, confirmed twice) |
|
||||
| 7 — roadmap | **R-89** + CONTEXT.md note |
|
||||
|
||||
## The fix, in one line
|
||||
|
||||
`allowPBSPrune := !t.Primary && t.KeepLast > 0` — so setting the PBS tier's `keep_last` to `0`
|
||||
disables both the `--prune-backups` value and the gate, in one config edit, **while the tier stays
|
||||
armed**. Verified: `backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0
|
||||
prune_pbs_allowed=false`, no `tier REJECTED` line.
|
||||
|
||||
## The proof
|
||||
|
||||
```
|
||||
07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied
|
||||
07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the change
|
||||
```
|
||||
|
||||
New snapshot `ns/demo-hp/ct/9201/2026-07-27T09:37:29Z`, chunks 9,787 → **9,813**, 97.0 % reused,
|
||||
45.80 s, **prune step absent entirely**. Driven via `POST /api/guest-backup/trigger` → `TriggerNow()`
|
||||
— the UI's „Mentés most" path, not `--selftest`, not raw `vzdump`.
|
||||
|
||||
**Hub gauge evidence NOT satisfied** — a +32.8 MB delta is below its 0.1 GB display granularity, so it
|
||||
still reads 12.6 GB / 13 %. Stated plainly rather than dressed up.
|
||||
|
||||
## The demo-felhom prediction — CLOSED
|
||||
|
||||
The claim was that demo-felhom's next weekly backup would make 3 snapshots and reproduce the prune
|
||||
failure. Neutralised on both halves: the box no longer attempts prune, and `prune-demo-felhom` covers
|
||||
the namespace server-side (verified live, `TASK OK`). **It will not reproduce.**
|
||||
|
||||
## Why it mattered more than the unpruned snapshots
|
||||
|
||||
demo-hp's PBS tier had reported failure on **every** backup since the tier was created on 07-26, while
|
||||
the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure
|
||||
invisible — which is precisely what happened at 07:13 UTC, when a real outage produced an
|
||||
indistinguishable result.
|
||||
|
||||
## Security property preserved
|
||||
|
||||
**No prune right was granted to any box.** Final ACLs are four entries, write-only
|
||||
(`DatastoreBackup`), live namespaces only. A compromised box still cannot delete its own offsite
|
||||
backups. `felhom-tenantsync.sh` was **not** edited — the ruling makes its current grant correct.
|
||||
|
||||
## Open
|
||||
|
||||
1. **R-89** — hub-owned retention policy (today's jobs are increment 1, not a stopgap).
|
||||
2. **Does the restic key on `storage-box-pool-1` have DELETE rights?** Unanswered, carried in R-89,
|
||||
and the more urgent half — if so, the daily app-data tier has the identical exposure and
|
||||
append-only mode is the equivalent answer. Rule once for both tiers.
|
||||
3. **GC has still never run.** First execution Sunday 04:30 UTC; worth watching, as nothing has ever
|
||||
exercised it here.
|
||||
4. Old 13 GB datastore copy still at `/srv/pbs-felhom` — rollback intact.
|
||||
-194
@@ -1,194 +0,0 @@
|
||||
# REPORT — R-100: a failing offsite tier must go stale (2026-07-28)
|
||||
|
||||
Hub **v0.79.0 → v0.80.0**; companion `felhom-controller` **v0.180.0 → v0.181.0** (the producer, shipped
|
||||
first). Written as `REPORT-r100.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom.eu 6369570`, `felhom-controller 4056fec`, `felhom-agent d5c7691` — all = origin/main. The only
|
||||
dirt in `felhom.eu` was a **foreign** `documentation/PROMPT-TEMPLATE.md` (shared worktree, untouched).
|
||||
Hub manifest **and** running pod both `0.79.0`; `staleAfter` = 48h; controller 0.180.0 and agent 0.110.0
|
||||
live on both boxes.
|
||||
|
||||
---
|
||||
|
||||
## The premise was wrong, and it was mine
|
||||
|
||||
R-100 was filed yesterday claiming *"the operator's fleet-wide alarm plane is silent"*. Phase 0 refuted
|
||||
that, twice:
|
||||
|
||||
1. **A failing offsite run does alarm.** `main.go:655` wires `SetOffboxNotify` → `NotifyBackupFailed`;
|
||||
the notify cooldown is 6h against a 24h cadence, so a nightly failure alarms nightly. Live hub DB:
|
||||
`backup_failed | operator | sent | 5`, latest 2026-07-27 17:42. The `isStale` doc comment —
|
||||
*"a recent-but-failing run is NOT stale (backup_failed owns that signal)"* — was **accurate**.
|
||||
2. **The orphaned-repo path I expected to be an indefinite hole is already covered.** The scheduled run
|
||||
returns early at `offbox.go:606`, *before* the `LastRun` write at `:716`, so `LastRun` freezes and
|
||||
`offsite_stale` fires normally.
|
||||
|
||||
I could find no failure mode that both advances `LastRun` and produces no operator signal.
|
||||
|
||||
**The real defect — defeated defence in depth.** `offsite_stale` is the hub-side, *pull-based* net that
|
||||
exists to be independent of controller-*pushed* events. Anchoring it on `LastRun` made it depend on the
|
||||
very thing it backs up: when the push is lost, the net cannot compensate, because the failing controller
|
||||
keeps refreshing the field the net reads. **F-HUB — this campaign's own finding, the hub dropping an
|
||||
event under `SQLITE_BUSY` with no retry** — is exactly that loss.
|
||||
|
||||
**Honest severity: MEDIUM**, not the top-ranked item. The fix is unchanged; the justification is not.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 answers
|
||||
|
||||
**P0.1 — a last-success timestamp did not exist.** `OffboxTarget` carried `LastRun`/`LastStatus`/
|
||||
`LastError`/`LastDuration` only. Recording one is a new field, not a transmission of something known.
|
||||
|
||||
**P0.2 — `LastStatus` on the wire**, from 4000 live reports (not from source alone):
|
||||
|
||||
| value | count | paired with |
|
||||
|---|---|---|
|
||||
| `ok` | 2269 | `last_run` set |
|
||||
| absent/null | 541 | `last_run` **empty** — never-ran |
|
||||
| `error` | 27 | `last_run` set |
|
||||
| **`running`** | 7 | a report captured **mid-run** |
|
||||
|
||||
Plus 1156 reports with no `offsite` object at all. **The legacy trap — status absent *with* a real
|
||||
`last_run` — occurs 0 times**, because `LastStatus="running"` is written the moment a run starts. It is
|
||||
still handled explicitly, but it is not a live shape. `running` being real is why the verdict ignores
|
||||
status entirely.
|
||||
|
||||
**P0.3 — sweep**
|
||||
|
||||
| tier | `LastRun` written on failure? | read as success by a verdict? |
|
||||
|---|---|---|
|
||||
| **Offsite restic** | YES (`offbox.go:716`) | **YES — hub `isStale`.** The defect |
|
||||
| **Tier 2 cross-drive** | YES (`recordTier2Failure`) | No hub verdict; UI only → **R-101, filed** |
|
||||
| Tier 1 recovery units | **NO** — derived from an actual artifact | structurally immune |
|
||||
| Shares offsite leg | YES | `sharing.html:180` shows the time only when status=="ok" — honest |
|
||||
| DB dump | n/a — **event-based** (`db_dump_completed`/`db_dump_failed`) | immune by design |
|
||||
|
||||
`offsite.go` is the **only** hub verdict anchored on a `LastRun`-shaped field. The deadline checker
|
||||
already uses distinct success/failure *events* — the pattern this converges on.
|
||||
|
||||
**P0.4 — the customer is NOT shown a failed offsite run as successful.** `backups_remote.html:34-36`
|
||||
leads with the status (`✓ Rendben` / `✗ Hiba` / `Fut…`). Two narrower Tier-2 instances → **R-101**.
|
||||
|
||||
---
|
||||
|
||||
## The fix
|
||||
|
||||
**Controller v0.181.0 (producer, shipped first).** `OffboxTarget.LastSuccess`, carried on the report as
|
||||
`last_success`. The rule is a pure function called unconditionally beside the `LastRun` write:
|
||||
|
||||
```go
|
||||
func offboxAnchorAfterRun(prev, at string, runErr error) string {
|
||||
if runErr != nil { return prev } // failures neither advance nor clear
|
||||
return at
|
||||
}
|
||||
```
|
||||
|
||||
Both directions are separate bugs: a failure must not **advance** it (the original defect) and must not
|
||||
**clear** it (one bad night making an established tier read as never-succeeded).
|
||||
|
||||
**Two silent-wipe sites found and closed** — the "seam built but never wired" shape, where the field
|
||||
exists, the writer sets it, and an unrelated routine path zeroes it:
|
||||
- `offboxConfigHandler` rebuilds the target from the form and copies runtime status field by field, so
|
||||
an ordinary settings save would have erased the anchor;
|
||||
- `ApplyOffsiteTarget` does the same on a hub re-apply.
|
||||
|
||||
Neither would have surfaced until the verdict changed, days later. **The first was proven live** — see
|
||||
below.
|
||||
|
||||
**Hub v0.80.0.** Three deliberate branches:
|
||||
- **never ran** — unchanged v0.73.0 anchored behaviour, still keyed on `last_run` on purpose: that field
|
||||
answers "has anything ever happened here", and a box whose *first* run failed is a run, not a newborn.
|
||||
- **legacy** (`last_run` set, no `last_success`) — degrades **explicitly** to the old behaviour, logged
|
||||
**once** per customer. Absence-as-failure would alarm the whole un-upgraded fleet; absence-as-success
|
||||
keeps the bug. Same degrade direction as R-88 Part 2's `age_state`.
|
||||
- **anchored** — counts from `last_success`; `last_status` is deliberately not consulted, because
|
||||
"error ⇒ stale" pages on every blip (the F-A1 noise path).
|
||||
|
||||
**The alarm text had to move with the verdict.** `emitStale` still said `last run 8h ago` while firing on
|
||||
a six-day-old success — a true alarm that reads as false. `staleAge` now separates *"runs are happening
|
||||
and failing — check the error, not the schedule"* from *"the offsite leg is silently not running"*.
|
||||
|
||||
---
|
||||
|
||||
## Red-proofs — all observed failing
|
||||
|
||||
| # | red-proof | observed failure |
|
||||
|---|---|---|
|
||||
| A | restore the `LastRun` anchor | `a tier that has not succeeded in 6 days reads as FRESH — that is R-100` |
|
||||
| B | delete the never-ran branch | `a newborn box alarmed — this is the 2026-07-23 cry-wolf that v0.73.0 fixed` |
|
||||
| C | collapse to `LastStatus == "error"` | `a single transient failure alarmed — 20h ... well inside the 48h threshold` |
|
||||
| D | delete the legacy degrade | `a legacy controller alarmed — that is a fleet-wide alarm storm on an un-upgraded fleet` |
|
||||
| + | drop the `runErr` guard (controller) | `a FAILED run advanced LastSuccess ... that is the R-100 defect in mirror image` |
|
||||
| + | always return `prev` | `a successful run did not advance the anchor` |
|
||||
| + | drop the wire field | `OffboxReportStatus dropped LastSuccess — the hub would degrade forever` |
|
||||
| + | drop the handler preservation | `a settings save erased LastSuccess` |
|
||||
|
||||
**A hollow test of my own, caught by red-proofing it.** The first version of the controller test
|
||||
re-implemented the rule in a local closure — mutating production code left it **green**. That is why
|
||||
`offboxAnchorAfterRun` was extracted: the test now calls the real rule.
|
||||
|
||||
Fixtures are the **real** wire shapes from P0.2, not invented JSON.
|
||||
`go build`/`go vet`/`go test` green in both repos (hub 17 pkgs, controller 27 pkgs), run separately
|
||||
from every commit.
|
||||
|
||||
---
|
||||
|
||||
## §6 — LIVE, on demo-hp (disposable; `peti-felhom` never touched)
|
||||
|
||||
A genuine restic failure was induced by pointing the target at a **closed port** (23 → 2) — it creates
|
||||
nothing, touches no data, and is exactly reversible.
|
||||
|
||||
```
|
||||
success run → last_status=ok last_run=11:24:20Z last_success=11:24:20Z
|
||||
INJECT port 23 → 2 ... and the settings save PRESERVED last_success = 11:24:20Z ← the wipe-site fix, live
|
||||
failing run → last_status=error last_run=11:25:48Z last_success=11:24:20Z ← ANCHOR HELD
|
||||
```
|
||||
|
||||
**As the hub received it:**
|
||||
|
||||
| box | status | `last_run` | `last_success` | anchor |
|
||||
|---|---|---|---|---|
|
||||
| **demo-hp** (induced failure) | `error` | 11:25:48Z | **11:24:20Z** | **HELD** |
|
||||
| **demo-felhom** (healthy) | `ok` | 11:29:22Z | **11:29:22Z** | **advanced** |
|
||||
|
||||
Also observed live, unplanned: **Scenario E**. Both boxes were still on the old controller at hub
|
||||
startup, and the degrade logged **exactly once per customer** —
|
||||
`[WARN] [offsite] demo-hp: controller sends no last_success — staleness degraded to the last-ATTEMPT
|
||||
anchor`. Two lines, two customers, same second.
|
||||
|
||||
**No spurious alarms:** 0 `offsite_stale` events since deploy (correct — both tiers succeeded minutes
|
||||
ago). `backup_failed` fired for demo-hp at 11:25:48 from the induced failure, confirming the
|
||||
pre-existing channel is intact and re-confirming the Phase 0 correction.
|
||||
|
||||
**Config restored** and verified field by field: `host=u629488-sub3.your-storagebox.de port=23
|
||||
user=u629488-sub3 repo=/home/felhom-repo enabled=True escrow=escrowed`.
|
||||
|
||||
### Proven live vs. proven by injected clock — stated plainly
|
||||
- **Live:** the anchor does not advance on failure; it does on success; it survives a settings save;
|
||||
`last_success` reaches the hub; the legacy degrade fires once per customer; no spurious alarms.
|
||||
- **Unit, injected clock only:** the 48h **threshold** behaviour itself — Scenarios A/B/C/D turning on
|
||||
elapsed time. A live threshold test would take days. **The threshold was NOT proven live.**
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — the rule
|
||||
**"Presence is not success"** added to `CLAUDE.md` and its versioned copy, with both instances
|
||||
(F-CRIT-2's phantom ctime, R-100's `LastRun`) and the corollary R-100's fix produced: when a verdict
|
||||
changes which field it counts from, the **alarm text must change with it**. `// R-100` notes sit at
|
||||
`isStale` and at the controller write site, each naming the test that pins it.
|
||||
|
||||
## Filed, not fixed
|
||||
- **R-101** — Tier-2 `LastRun` is also written on failure, and three customer surfaces render it without
|
||||
a status (two degraded branches plus the restore-confirm dialog). No hub verdict reads it.
|
||||
|
||||
## NOT yet live-validated (carried forward)
|
||||
- **The 48h staleness threshold itself** (see above) — and with it Scenario A end-to-end: no
|
||||
`offsite_stale` event has yet been *observed firing* from a genuinely stale success anchor, because
|
||||
that needs 48h of failure.
|
||||
- **Fault 4** — restic transport interruption; four injection approaches defeated by guest-bridged
|
||||
networking. (This task's closed-port injection sidesteps it rather than solving it.)
|
||||
- **R-99** — prune never removes phantom snapshots.
|
||||
- **R-101** — filed today, unvalidated.
|
||||
- `contentionAlarmAfter` (3h) — injected clock only.
|
||||
-181
@@ -1,181 +0,0 @@
|
||||
# REPORT — R-101 + F-DIAG + F-OPS (2026-07-28)
|
||||
|
||||
Controller **v0.181.0 → v0.182.0**; `felhom.eu` gains the manual-restore runbook (F-OPS) and the
|
||||
OPEN-ITEMS rows. Written as `REPORT-r101.md` so the shared `REPORT.md` is not clobbered.
|
||||
|
||||
## Baselines (reconfirmed, not copied)
|
||||
`felhom-controller 3db8bfb`, `felhom.eu 6b7d516`, `felhom-agent d5c7691` — all = origin/main; the only
|
||||
dirt in `felhom.eu` was a **foreign** `PROMPT-TEMPLATE.md`. Controller **0.181.0** live on both boxes,
|
||||
hub `felhom-hub:0.80.0` ready 1/1.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0
|
||||
|
||||
**The render sites — three dishonest, two already honest.** The spec listed `backups_apps.html:216`
|
||||
as a defect site; it is in fact the one branch that *already* pairs its timestamp with a status badge.
|
||||
The real third site is the `Tier2DestInactive` branch.
|
||||
|
||||
| site | rendered | honest? |
|
||||
|---|---|---|
|
||||
| `:231` **restore confirm dialog** | `Legutóbbi másolat: {{.Tier2LastRun}}` — raw RFC3339, no status | **NO** — the one that matters |
|
||||
| `:195` `Tier2DestDisconnected` | `Utolsó: …`, no status | **NO** |
|
||||
| `:206` `Tier2DestInactive` | `Utolsó: …`, no status | **NO** |
|
||||
| `:217` main configured branch | `Utolsó: …` **+ status badge** | already honest |
|
||||
| `sharing.html:181` | rendered **only** when status=="ok" | already honest |
|
||||
|
||||
`Tier2LastStatus` was already set unconditionally at `handlers.go:1182`, so this was a wording/anchor
|
||||
problem, not a plumbing one. The restore button was gated on `{{if .Tier2LastRun}}`, so **Scenario C
|
||||
was live-reachable**: a tier that had attempted and never succeeded offered a restore and a timestamp.
|
||||
|
||||
**`cd.LastRun` is written on failure** — `recordTier2Failure` (`tier2.go:573-574`) writes it alongside
|
||||
`LastStatus:"error"`. Identical shape to R-100.
|
||||
|
||||
**Legacy state is universal, not an edge case.** All 7 Tier-2 rows across both boxes had `last_run` and
|
||||
no anchor. Scenario E was therefore the *initial state of every customer*, which is what made the
|
||||
legacy marker non-optional.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — the strings shipped
|
||||
|
||||
| case | string |
|
||||
|---|---|
|
||||
| dialog, normal | `… Legutóbbi sikeres másolat: 2026-07-28 16:43.` |
|
||||
| dialog, newest attempt failed | `… Legutóbbi sikeres másolat: 2026-07-28 16:40. Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek lehetnek.` |
|
||||
| card | `Utolsó sikeres: 2 perce` |
|
||||
| never succeeded | `Még nincs sikeres másolat` + `Még nincs sikeres másolat, amiből vissza lehetne állítani.` (restore removed) |
|
||||
| **legacy row** | `Utolsó: …` / `Legutóbbi másolat: …` — **today's wording, unchanged**, logged once per stack |
|
||||
|
||||
**Timestamp made human-readable** (agreed): new `fmtTimeStr` renders Budapest-local `2026-07-28 16:40`
|
||||
instead of the raw UTC `2026-07-28T14:40:55Z` a customer was previously asked to reason about.
|
||||
|
||||
**`SuccessTracked` is what makes the legacy case possible at all.** Without it, "row predates the
|
||||
anchor" and "row has an anchor and it is empty" are indistinguishable — both are `LastSuccess==""` —
|
||||
and every existing row would have rendered as never-succeeded on deploy. Legacy rows migrate on first
|
||||
touch: a row whose last known state was `ok` adopts that time (truthful — under the old code that run
|
||||
did succeed); a row whose last state was `error` seeds **nothing**, because the old data evidences no
|
||||
success.
|
||||
|
||||
## Part 2 — the copy-site hazard, and it was in the path
|
||||
|
||||
The three `record*` helpers each built a **whole `CrossDriveBackup` literal**, with a helper re-applying
|
||||
exactly two fields; everything else was zeroed on every status write. Adding `LastSuccess` to that shape
|
||||
would have had `recordTier2Failure` **clear** it — the mirror image of the defect, firing on the *first*
|
||||
failure rather than lying dormant.
|
||||
|
||||
Replaced with **`tier2Update`**, which copies the existing row and overlays the outcome: **compile-safe
|
||||
by construction** — a new field carries over unless deliberately overwritten, so nothing is preserved by
|
||||
a list that can fall out of date. Callers now clear explicitly what a run invalidates, reproducing the
|
||||
old behaviour exactly.
|
||||
|
||||
**Sweep of other rebuild sites:** `SetTier2Preference` mutates in place (safe); `SetCrossDriveConfig(name, nil)`
|
||||
in `api/router.go:774` is a deliberate delete. No others.
|
||||
|
||||
## Part 3 — F-DIAG
|
||||
|
||||
| class | signal it maps to | message head |
|
||||
|---|---|---|
|
||||
| `quota` | the pre-run soft-quota gate | `A távoli mentés nem fért el a tárhelykereten belül` |
|
||||
| `orphaned` | `ErrOffboxOrphaned` sentinel | `A távoli tárhely egy korábbi, már nem elérhető kulccsal készült` |
|
||||
| `no_repo` | restic "unable to open config file" | `A távoli tárhelyen nincs mentési adattár` |
|
||||
| `no_units` | "produced no snapshots" | `Nem volt mit menteni: egyetlen kijelölt alkalmazásnak sem található mentése` |
|
||||
| `transport` | refused/reset/timeout/authn/host-key | `A távoli tárhely nem érhető el (hálózat vagy bejelentkezés)` |
|
||||
| **`unknown`** | anything else | `A távoli mentés ismeretlen okból nem sikerült` |
|
||||
|
||||
The `unknown` class is deliberate: a cause that cannot be told apart where the error is produced is
|
||||
reported as unknown rather than folded into a neighbour.
|
||||
|
||||
**Secrets — and this caught a bug in my own first attempt.** The old message was
|
||||
`"…: " + err.Error()`, carrying the repo reference `sftp:<user>@<host>:<path>` off the box. My first
|
||||
sanitiser regex-matched `sftp:…` and `user@host` and *looked* complete; its own test caught it leaking
|
||||
on `ssh: connect to host <host> port 23: Connection refused` — a bare hostname in neither shape. It now
|
||||
redacts the target's **actual** host/user/repo-path literally, with the regex kept only as a backstop.
|
||||
Guessing at what a secret looks like fails exactly where it matters.
|
||||
|
||||
## Part 4 — F-OPS
|
||||
|
||||
`documentation/runbooks/RUNBOOK-manual-guest-restore.md`. Grounded in the real bind shape read off live
|
||||
guest 9201, not written from memory. Covers: which `mpN` are storage volumes (restored) versus **host
|
||||
binds** (taken as-is on the target); the `mp9` trap — it embeds the **source** VMID, so restoring to a
|
||||
different VMID can bind **another guest's bootstrap credentials**; strip-and-re-add before first boot;
|
||||
the hookscript check; and a positive pre-start verification that asserts every bind path exists rather
|
||||
than accepting "no error". Docs only, by design.
|
||||
|
||||
---
|
||||
|
||||
## Red-proofs — all observed failing
|
||||
|
||||
| # | red-proof | observed failure |
|
||||
|---|---|---|
|
||||
| A | dialog back on the attempt clock | `the dialog does not name the last SUCCESSFUL copy` |
|
||||
| C | gate the restore on `LastRun` again | `a tier that has NEVER succeeded still offers a restore — the dialog would promise a copy that does not exist` |
|
||||
| D | make the caution unconditional | `a HEALTHY tier shows the failed-attempt caution ("nem sikerült")` |
|
||||
| F | clear the anchor on failure | `a FAILED run wiped the success anchor (round 1) — one bad night would read as 'no copy has ever succeeded'` |
|
||||
| + | raw sanitiser | `the repo reference reached the message ("sftp:" leaked)` |
|
||||
|
||||
**F exercises the real `recordTier2Success` → `recordTier2Failure` sequence**, not a modelled copy — the
|
||||
R-100 lesson. The Scenario A/C/D tests **render the production template tree** and assert on the string
|
||||
the customer reads; a test asserting a template variable would prove nothing about wording, which is
|
||||
the defect.
|
||||
|
||||
`go build`, `go vet ./...`, `go test ./...` — 27 packages, `rc=0`; `template_id_gate.py` and
|
||||
`emoji_gate.py` both OK. Run separately from every commit.
|
||||
|
||||
---
|
||||
|
||||
## LIVE on demo-hp — the rendered dialog, which is the deliverable
|
||||
|
||||
**Legacy state** (before any run under v0.182.0) — today's wording, no fright:
|
||||
```
|
||||
Legutóbbi másolat: 2026-07-28 03:30
|
||||
```
|
||||
|
||||
Failure induced genuinely: the Tier-2 destination directory was **moved aside** and replaced by a file,
|
||||
so `mkdir …/recovery-unit` fails. (`chmod` does not work — the controller runs as root, which bypasses
|
||||
permission bits; `chattr +i` is refused, the unprivileged container lacks `CAP_LINUX_IMMUTABLE`. Both
|
||||
were tried and reported rather than glossed.) The real data was only ever moved, never deleted.
|
||||
|
||||
```
|
||||
status = error
|
||||
last_run = 2026-07-28T14:42:18Z ← ADVANCED
|
||||
last_success = 2026-07-28T14:40:55Z ← HELD
|
||||
last_error = mkdir …/paperless-ngx/recovery-unit: …
|
||||
```
|
||||
|
||||
**The rendered dialog, failed state:**
|
||||
```
|
||||
Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
|
||||
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:40.
|
||||
Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek
|
||||
lehetnek.
|
||||
```
|
||||
|
||||
**The rendered dialog, healthy state** (after restoring the destination and a successful run) — no
|
||||
caution, no tonal change:
|
||||
```
|
||||
Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
|
||||
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:43.
|
||||
```
|
||||
|
||||
Card lines: `Utolsó sikeres: 2 perce` → `Utolsó sikeres: most`.
|
||||
|
||||
**Everything restored:** destination is a directory again, 86 MB intact, mode 755, `.r101-aside` gone,
|
||||
`status=ok`, `last_success=2026-07-28T14:43:23Z`.
|
||||
|
||||
**demo-felhom is the untouched control:** all 5 rows still `tracked=None` after the deploy, rendering
|
||||
today's way, 15/15 containers up. Scenario E holding across a whole box nobody ran.
|
||||
|
||||
---
|
||||
|
||||
## NOT yet live-validated (carried forward)
|
||||
- **F-DIAG's classes** — unit-proven only. No live offsite failure of each class was induced; the
|
||||
`transport` class is the only one this arc exercised indirectly.
|
||||
- **Scenario C live** — the never-succeeded rendering is unit-proven; no fleet row is in that state
|
||||
(every row either migrated or has a real success), and manufacturing one would mean breaking a
|
||||
customer app's only Tier-2 history.
|
||||
- **The Tier-2 restore itself** was not executed — this arc changed what the dialog *says*, not what the
|
||||
restore does.
|
||||
- **R-100's 48h staleness threshold** — injected clock only.
|
||||
- **Fault 4** (restic transport interruption), **R-99**, **F-HUB**, fault 12, the three-way concurrency
|
||||
overlap — next campaign's material, untouched here.
|
||||
@@ -1,136 +0,0 @@
|
||||
# REPORT — R-106 + R-109 (+ R-122): closing the recipe-completeness set (2026-07-30)
|
||||
|
||||
Non-overwritten sibling per `CLAUDE.md:82-87` — the shared `REPORT.md` holds R-117 and is not touched.
|
||||
|
||||
Shipped: **agent v0.118.0 → v0.118.1** (`felhom-agent` `1c8a67e`, `6b5dade`) + **hub v0.83.0**
|
||||
(`felhom.eu` `acfc2b7`). Neither half is useful alone.
|
||||
|
||||
**Read §3 first if you read nothing else:** v0.118.0's R-106 half shipped INERT and live validation is
|
||||
what caught it — the recipe still said `"root"`, now with `namespace_state: resolved` beside it. Full
|
||||
account in the audit §6, filed as **R-125**.
|
||||
|
||||
## Part 0 — the answers, before the fix
|
||||
|
||||
### 0.1 Which items are actually open, and R-105/R-106's registration
|
||||
|
||||
`OPEN-ITEMS.md` calls itself "the single source of truth for open work" (`:1`), with `ROADMAP.md` keeping
|
||||
"the full history and reasoning" (`:3-4`).
|
||||
|
||||
| item | `ROADMAP.md` | `OPEN-ITEMS.md` | verdict |
|
||||
|---|---|---|---|
|
||||
| R-105 | row, `READY — 2026-07-28` (`:108`) | **absent** | **open but UNREGISTERED** |
|
||||
| R-106 | row, `READY — 2026-07-28` (`:109`) | **absent** | **open but UNREGISTERED** |
|
||||
| R-108 | row (`:111`) | row (`:50`) | registered |
|
||||
| R-109 | row (`:112`) | row (`:61`) | registered |
|
||||
|
||||
So R-109's own cell — "third recipe-completeness defect beside R-105/R-106" — was the **only** place in the
|
||||
register naming two open items. That is exactly the thread-loss the register exists to prevent, and it is
|
||||
itself a finding (filed **R-123**). Both now have rows.
|
||||
|
||||
**The set this task closes is R-106 + R-109**, matching the arc's stated definition of done (`OPEN-ITEMS.md:14`).
|
||||
**R-105 is NOT in it** and was not worked: it is M-sized and is about three *hub-held DR records* being `{}`
|
||||
(`hosts.dr_record_json`, `host_escrow.directive_json`, and the `drives` third — already traced and populated
|
||||
by the 2026-07-28 target move). Different fields, different owner, different size.
|
||||
|
||||
### 0.2 Where the recipe is generated — three producers, not two
|
||||
|
||||
| half | repo | function |
|
||||
|---|---|---|
|
||||
| host (guests/pbs/drives/pve_storage) | `felhom-agent` | `BuildDRRecipeHostHalf`, `internal/hub/dr_recipe.go:86` |
|
||||
| app (customer/apps/offsite_restic) | `felhom-controller` | `controller/internal/report/dr_recipe.go` |
|
||||
| **assembly + delivery** | `felhom.eu/hub` | `AssembleDRRecipe`, `internal/store/dr_recipe.go:104`; served by `handleDRRecipeDownload`, `internal/web/dr_recipe.go:14`, route `internal/web/server.go:439` |
|
||||
|
||||
R-109's "host-half" is therefore the **agent**, and the field must also pass the **hub's** allow-list — see §2.
|
||||
|
||||
### 0.3 What the namespace field actually contained — verified, and the brief was RIGHT
|
||||
|
||||
The eleven-session-old brief held up. Live, pre-fix, from the hub for **both** boxes:
|
||||
|
||||
```json
|
||||
"pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }
|
||||
```
|
||||
|
||||
against `/etc/pve/storage.cfg` on the same boxes:
|
||||
|
||||
```
|
||||
pbs: felhom-pbs
|
||||
datastore felhom-offsite
|
||||
namespace demo-felhom # demo-hp reads: namespace demo-hp
|
||||
```
|
||||
|
||||
Traced to source: `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo
|
||||
per item once the list is namespace-scoped via `?ns=` (`:118-120`) → always empty → `ToHub` normalises empty
|
||||
to `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it in.
|
||||
|
||||
**The authority taken, and why:** storage.cfg's `namespace` on the pbs storage. It is the same field
|
||||
`vzdump --storage <pbs>` makes PVE read, and the agent's own verify client is built from it
|
||||
(`cmd/felhom-agent/main.go:1164`). Deriving the recipe from anything else is how it drifts again.
|
||||
|
||||
## 1. R-109's ambiguity is real, in the boxes' own pre-fix recipe
|
||||
|
||||
```json
|
||||
"pve_storage": [
|
||||
{ "name": "local-lvm", "type": "lvmthin", "content": "images,rootdir" },
|
||||
{ "name": "felhom-backup", "type": "local-dir", "content": "backup" },
|
||||
{ "name": "felhom-pbs", "type": "pbs", "content": "backup" },
|
||||
{ "name": "local", "type": "local", "content": "backup,import,vztmpl,iso" }
|
||||
]
|
||||
```
|
||||
|
||||
No `backup_target` key anywhere. `felhom-backup` (live, `/mnt/hdd_1`) and `local` (`/var/lib/vz`, archives
|
||||
frozen 2026-07-28) are both `content=backup` dir storages; `local` is also the *historically* correct answer,
|
||||
which is what makes guessing it so easy.
|
||||
|
||||
## 2. R-122 — a fourth defect, found here, and it had already shipped
|
||||
|
||||
`AssembleDRRecipe`'s `hostHalfShape`/`appHalfShape` are **allow-lists** dressed as forward-compat. The
|
||||
controller has emitted `offsite_restic` since fork-4 (`controller/internal/report/dr_recipe.go:39-41`, "so DR
|
||||
knows WHERE to recover from"); `appHalfShape` never listed the key. Verified both ways:
|
||||
|
||||
- **stored**: `dr_recipe.app_half_json` carries it for all three real customers —
|
||||
`peti-felhom`, `demo-felhom` (`u629488-sub1.your-storagebox.de:23/home/felhom-repo`), `demo-hp`.
|
||||
- **delivered**: the downloaded recipe's top-level keys were
|
||||
`recipe_version, customer, guests, pbs, drives, pve_storage, apps` — **no `offsite_restic`**.
|
||||
|
||||
So a restorer reading the recipe had **no offsite location at all**, for the whole life of the feature, with
|
||||
a green suite throughout — because the test fixture `drAppHalf` is hand-written and omits the field.
|
||||
|
||||
**Deviation from the task's §7.10 ("Findings — filed as R-n, none fixed"), stated rather than absorbed:**
|
||||
I fixed it. Reasons — (a) Part 0 authorises working the real set if it differs; (b) it is the same
|
||||
symptom the task is named for (the recipe is incomplete), and the worst instance, a whole section missing;
|
||||
(c) it is in the *same two structs* R-109 forced me to edit, and leaving one of three known keys off a
|
||||
drop-list I was already correcting would be indefensible. It is filed as R-122 with a SHIPPED disposition.
|
||||
|
||||
## 3. The before/after recipe — both boxes, quoted
|
||||
|
||||
```
|
||||
demo-felhom BEFORE "namespace":"root" backup_target absent offsite_restic absent
|
||||
AFTER "namespace":"demo-felhom" backup_target {resolved, felhom-backup, /mnt/hdd_1}
|
||||
offsite_restic {u629488-sub1…}
|
||||
demo-hp BEFORE "namespace":"root" backup_target absent offsite_restic absent
|
||||
AFTER "namespace":"demo-hp" backup_target {resolved, felhom-backup, /mnt/nvme-1tb}
|
||||
offsite_restic {u629488-sub3…}
|
||||
```
|
||||
|
||||
The two boxes DISAGREEING is the point — nothing is hardcoded. And the ambiguity was not theoretical:
|
||||
on both boxes `felhom-backup` holds an archive from **07-30 04:36** while `local` stops at
|
||||
**07-28 17:5x**, frozen at the target-move date. The recipe now names the live one.
|
||||
|
||||
Full evidence, all seven red-proofs and the publish observables:
|
||||
`documentation/audits/R106-R109-recipe-completeness-2026-07-30.md`.
|
||||
|
||||
## 4. Findings filed (none of them fixed except R-122, see §2)
|
||||
|
||||
| id | finding |
|
||||
|---|---|
|
||||
| **R-122** | `AssembleDRRecipe` allow-list dropped `offsite_restic` for the feature's whole life — **FIXED here**, hub v0.83.0 |
|
||||
| **R-123** | R-105 and R-106 were `READY` in `ROADMAP.md` with no `OPEN-ITEMS.md` row — referenced only inside R-109's prose. Registered here |
|
||||
| **R-125** | v0.118.0 shipped an inert R-106 because the "production path" test injected `fakeObserver` one layer below the break — **FIXED** in v0.118.1; filed for the doctrine point (name the seam you inject at) |
|
||||
| **R-124** | The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""` and there is no namespace literally named `root` — a restorer pasting it into `pct restore --ns root` would fail. Pre-existing wire convention, deliberately unchanged; documented at `PBSRootNamespace` |
|
||||
|
||||
## 5. Not done, and why
|
||||
|
||||
- **R-105, R-108, D5** — out of scope by the task's §6. R-108 blocks D5; starting either would leave both half-done.
|
||||
- **The backup machinery** — untouched. This corrects the record, not the doing.
|
||||
- **`sess-f` (0.116.0) and `drill-r50` (0.113.0) were not upgraded** — neither was named as a venue, and `drill-r50` is fenced by the task's §6.
|
||||
- **R-124 not fixed** — changing the wire's spelling of the root namespace mid-R-106 would shift the field's meaning during the fix meant to make it trustworthy.
|
||||
@@ -1,117 +0,0 @@
|
||||
# REPORT-r116-diag — the `/disks` payload captured, R-116's mechanism isolated (2026-07-30)
|
||||
|
||||
Read-only diagnosis run by CC on DooPlex. **No code written, nothing built, nothing published.**
|
||||
Full evidence: `documentation/audits/DIAG-r116-disks-payload-2026-07-30.md`.
|
||||
|
||||
A `REPORT-*.md` sibling, not the shared `REPORT.md` (`CLAUDE.md` parallel-session rule).
|
||||
|
||||
## Outcome
|
||||
|
||||
**Both goals met.** The `/disks` read path is solved and written down verbatim, proven by a
|
||||
present-drive control run *first*; and the absent-state payload was captured, which isolates the
|
||||
mechanism.
|
||||
|
||||
**R-116 is theory #1 — "the registry-union row writes `false`" — the theory that was raised, declared
|
||||
wrong, and retracted. The retraction was the error.**
|
||||
|
||||
In the absent state `/disks` returns **4 rows, not 3**. The drive appears twice and the two facts the
|
||||
controller needs sit on different rows:
|
||||
|
||||
| row | source | `mount_path` | `guest_path` | `backup_target` |
|
||||
|---|---|---|---|---|
|
||||
| `felhom-backup` | Observe (`disks.go:196-284`) | `""` | `""` | **`true`** |
|
||||
| `694034cc-…` (the UUID) | registry union (`disks.go:297-339`) | `/mnt/cel` | `/mnt/felhom-drives/cel` | **field ABSENT ⇒ `false`** |
|
||||
|
||||
So the row holding the flag contributes **no key** to `driveTargetByPath`, and the row that owns the key
|
||||
says `false` → `isTarget[a.Path]` is `false` → generic `storage_disconnected`. On return the rows
|
||||
re-merge into one carrying both facts → specific `backup_target_restored`. Applying
|
||||
`intermediary.go:602-618` to the captured payloads gives PRESENT `True` / ABSENT `False` /
|
||||
RETURNED `True` — **the live asymmetry reproduced from payload alone.**
|
||||
|
||||
The union row's `MountPath` survives the device because the union source is the systemd **`.mount` unit
|
||||
file** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never reads the mount table. The
|
||||
dedup at `:298` therefore does not fire, because `seen` is keyed on the one field the absent state
|
||||
empties (`:290-295`).
|
||||
|
||||
**Theory #2 (the basis of the shipped v0.115.0) is false on both halves**; **#3 is false too**
|
||||
(`isTarget["/mnt/cel"]` is `false` as well). **v0.115.0 is provably inert** — its fallback calls
|
||||
`StablePathForRaw("")`, which returns `""` (`intermediary.go:69-75`), so it assigns nothing.
|
||||
|
||||
## The read path (this cost two prior sessions — it should never cost again)
|
||||
|
||||
The token plaintext exists in exactly one place: `bootstrap.json` **on the Proxmox host**. The agent's
|
||||
own store keeps SHA-256 hashes only (`tokenstore.go:26-32`), which is what defeated the earlier attempts.
|
||||
|
||||
```bash
|
||||
ssh felhom-pve
|
||||
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
|
||||
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
|
||||
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])")
|
||||
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
|
||||
```
|
||||
|
||||
Control run, live felhom-pve, drive present: **HTTP 200, 2483 bytes, 4 plausible rows** — so Part 5's
|
||||
0-rows-on-a-present-drive failure mode is excluded.
|
||||
|
||||
## Where the absent state was staged
|
||||
|
||||
No new box. The existing DooPlex **nested-PVE drill fixture** (`drill/drill.qcow2`, snapshot `virgin`) —
|
||||
my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with
|
||||
the **byte-identical live agent binary** (`sha256 f48544ad…`, `--version` 0.115.0) and every
|
||||
state-producing step through the real endpoints (`format` → `assign` → `guest-attach` →
|
||||
`backup/target`). Its present-state row matched felhom-pve's control run field-for-field before it was
|
||||
trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token
|
||||
record, no controller) are enumerated in the audit §4.
|
||||
|
||||
## Two new findings, filed not chased
|
||||
|
||||
- **R-117 (READY M) — outranks R-116.** After a detach/reattach the guest's bind is a **dead mount**:
|
||||
host is healthy on the new device node, guest still names the old one, and `ls`/write through it
|
||||
return **`EIO`** — while `/disks` reports `attached` + `bound_under_parent:true` + `backup_target:true`.
|
||||
`planDriveGates` therefore takes the `Return` branch and **restarts the customer's apps onto a dead
|
||||
namespace, reporting healthy, with no alarm on any channel.** R-113's conjunction cannot catch it:
|
||||
one half is satisfied by the stale entry, the other by the new device, and neither compares them.
|
||||
This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs.
|
||||
- **R-118 (READY XS).** An absent drive's union row reports the **root filesystem's** capacity as its own
|
||||
(46 GiB / 9.2 % for a 4 GB drive) — `statfsCapacity` at `disks.go:335-338` statfs's a bare directory on
|
||||
root. `observe.go:176-183` guards the Observe path against exactly this; the union path does not.
|
||||
`durable_id` is still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id.
|
||||
|
||||
## Register
|
||||
|
||||
`documentation/backlog/OPEN-ITEMS.md` — R-116 updated with the mechanism and the fix constraints;
|
||||
R-117 and R-118 added. The single register edit this session makes.
|
||||
|
||||
## Record correction
|
||||
|
||||
The brief's baseline `controller 0.185.1` is the version the **golden bakes**. **0.186.0** (R-114 +
|
||||
R-112, 2026-07-29) is what **demo-felhom** runs — **demo-hp is still on 0.185.1**, so the fleet is
|
||||
split, and R-114's `TargetAbsent` branch exists only on demo-felhom. Confirmed: agent **0.115.0**
|
||||
(felhom-pve) / **0.113.0** (demo-hp), hub **0.81.0** (manifest pin and live pod image agree),
|
||||
host-install **1.22.0**, `felhom.eu` HEAD `c3ce4c7`.
|
||||
|
||||
> **Correction, 2026-07-30.** As first written this section said 0.186.0 was what *both* demo boxes run.
|
||||
> That was wrong — only felhom-pve's guest was sampled and the result generalised to the fleet. demo-hp
|
||||
> re-checked directly → `0.185.1`. Fixed here and in the audit's baseline table.
|
||||
|
||||
## Teardown and fences
|
||||
|
||||
Drill guest destroyed, scratch storage removed, mount unit deleted, secrets `shred -u`'d, VM powered
|
||||
off, **`drill.qcow2` restored to `virgin`** (the golden-bake fixture is exactly as found), scratch qcow2
|
||||
and console dumps deleted. DooPlex `/mnt/5_hdd` at 24 %, unchanged.
|
||||
|
||||
Both demo boxes **read-only throughout** and re-verified after teardown: demo-hp `local-lvm` **38.83 %**
|
||||
(identical before/after and to Part 5), `drill-r50` still stopped, felhom-pve `felhom-backup` still
|
||||
active on `/dev/sdb`, both guests running, **v0.115.0 untouched**.
|
||||
|
||||
`sess-d-0452c4` now reads **STALE**, and the delete gate refuses only on ONLINE
|
||||
(`hub/internal/web/customer_delete.go:220-228`) — so it **is** now deletable; the command is recorded in
|
||||
the audit rather than executed (customer delete runs external teardown plus a DB purge). **`sess-c` is
|
||||
also still present and was not recorded by the Session-C audit** — same terms.
|
||||
|
||||
## Not done, deliberately
|
||||
|
||||
No fresh controller gate-log / hub-event correlation: that observable was already captured live and
|
||||
identically twice, and only the payload was missing. Staging a controller meant a hub customer, a
|
||||
pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8
|
||||
correlates the payload to those existing measurements and labels that step as inference.
|
||||
@@ -1,96 +0,0 @@
|
||||
# REPORT — SPIKE R-117: a dead bind that reports healthy (2026-07-30)
|
||||
|
||||
Written as `REPORT-<topic>.md`, not `REPORT.md`, per this repo's parallel-session rule and the
|
||||
established local pattern (`REPORT-r116-diag.md`, `REPORT-session-c.md`, …). The shared `REPORT.md`
|
||||
was not touched.
|
||||
|
||||
**Class: Spike.** Deliverable is a findings document. **No production code was written; no `.go` file
|
||||
in either repo was modified; nothing was built for deployment, published, or version-bumped.**
|
||||
|
||||
## Deliverables
|
||||
|
||||
| File | Action |
|
||||
|---|---|
|
||||
| `documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md` | **new** — the findings doc (Q1–Q7, evidence, recipe, probe comparison, recommendation) |
|
||||
| `documentation/backlog/OPEN-ITEMS.md` | **R-117 row rewritten** with the mechanism, the reproduction recipe and the fix constraint — the one register edit, per the brief |
|
||||
|
||||
**No CHANGELOG entry.** This repo's changelogs are per-area (`hub/`, `scripts/`, `website/`); a
|
||||
documentation-only change belongs to none of them. Stated rather than silently skipped, per standing
|
||||
rule 4.
|
||||
|
||||
## Baselines
|
||||
|
||||
agent `main` **v0.116.0** @ `d4eb259` · controller `main` **v0.186.0** @ `b331f18` ·
|
||||
`felhom.eu` HEAD `29bcfeb` · hub **live 0.82.0** · golden **0.186.0** ·
|
||||
demo-hp PVE 9.2.2 / kernel 7.0.2-6-pve, **live agent 0.113.0** (= manifest `MinAgent`; never used as
|
||||
the source of a behavioural claim — every predicate result came from a probe built from `main`).
|
||||
|
||||
## Results
|
||||
|
||||
**All seven questions answered empirically.**
|
||||
|
||||
- **Q1 — reproduced 3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp.
|
||||
**The device-node change is a consequence of the defect, not a precondition** — control test: with
|
||||
the stale bind held the drive returns as `sdc` (8:32); released, the letter is reused (`sdb`, 8:16).
|
||||
- **Q2 — two death states**: device removed ⇒ superopts gain `shutdown`, `EIO`(5) on read and write,
|
||||
host and guest; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads served from cache.
|
||||
The raw host mount is genuinely healthy in both. **No cross-device mis-identification is possible**
|
||||
on this path — the unit is fs-UUID-keyed.
|
||||
- **Q3 — confirmed at source and live.** Both halves of the R-113 conjunction compare **field 5** of a
|
||||
mountinfo line and **never read field 3 (`major:minor`)**, so neither can see that the bind and the
|
||||
raw mount name different devices. Measured `BoundUnderParent = TRUE` over an `EIO` namespace.
|
||||
- **Q4 — a pure-`/proc` check costs 0.16–0.45 ms**, cannot hang, spins up no disk, writes nothing.
|
||||
**`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow. The
|
||||
hang case is below; it is the sharpest result in the run.
|
||||
- **Q5 — the agent**, and not on balance: the controller runs inside the guest and cannot see the host
|
||||
mount tables the check needs.
|
||||
- **Q6 — recovery works in place, guest never restarted** (init PID identical). **The repair code
|
||||
already exists and three call sites already invoke it**, including the controller's `Return` branch
|
||||
*before* it restarts apps — all defeated by one early return.
|
||||
- **Q7 — a bind can die in steady state, no cycle at all.** The gate produces no action and **nothing
|
||||
is emitted on any channel.** A `Return`-branch fix cannot reach this half.
|
||||
|
||||
## Q4's hang case — measured, and it is the sharpest result
|
||||
|
||||
Against a `dmsetup suspend`ed device (I/O queues instead of returning `EIO`):
|
||||
|
||||
- **P1 and P2 completed in 364 µs / 206 µs.** They read `/proc`, so no block device is involved.
|
||||
- **`statfs` and `getdents` completed and reported HEALTHY** — on a wedged device they do not even hang.
|
||||
- **Every probe that touches the device blocked, including a buffered write with no `fsync`** — the
|
||||
`O_CREAT` metadata path needs journal access (`wchan=do_get_write_access`). There is no cheap-and-safe
|
||||
write probe.
|
||||
- **The blocked process survived `SIGTERM` and `SIGKILL`** (`stat=D`, still alive 3m50s after `kill -9`)
|
||||
and died only when the device was resumed. So **`systemctl restart felhom-agent` would hang**, leaving
|
||||
the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal
|
||||
the leak (5→5, 5→6).
|
||||
|
||||
**A timeout protects the caller's control flow and nothing else.** This turns "prefer a cheap probe" into
|
||||
a fence: **the fix must issue no block I/O.**
|
||||
|
||||
## Teardown — done, all three layers
|
||||
|
||||
Guest 9301 destroyed; `r117scratch` removed; both dm devices and both loop devices gone; `scsi_debug`
|
||||
unloaded (`/dev/sd*` back to `sda1..3`); no `r117` mounts, `/mnt` and `/root` exactly as found; `local`
|
||||
**37.02 %** against a session-start **37.00 %**. Fences re-verified *after* teardown: 9201 `running`,
|
||||
`drill-r50` `stopped`, `local-lvm` **38.84 % byte-identical**, `felhom-backup` `content backup`
|
||||
unchanged, live `/mnt/felhom-drives` intact with both submounts, agent service `active`. **Layer 3 is
|
||||
genuinely empty** — 9301 had no network interface and ran no controller, so no hub-side record was ever
|
||||
created.
|
||||
|
||||
**Ordering trap worth keeping:** a suspended dm device must be `dmsetup resume`d *before* any `umount`,
|
||||
or the teardown itself blocks on the same uninterruptible sleep.
|
||||
|
||||
## Not measured
|
||||
|
||||
No load or duration testing of the recommended check — P1/P2 were single calls, not a sustained
|
||||
reconcile loop on a many-drive box. Nothing suggests a problem (they are two `/proc` reads the code
|
||||
already performs), but it was not measured.
|
||||
|
||||
## Findings filed, none fixed
|
||||
|
||||
R-117 (mechanism + recipe), **R-117a** steady-state death with no event (HIGH, larger than R-117 as
|
||||
filed), **R-117b** `statfs`/`getdents` are hollow liveness probes, **R-117c** three untested comments
|
||||
promising "live + usable in the guest", **R-117d** the self-heal that already exists is short-circuited
|
||||
(HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing internet,
|
||||
including the WireGuard OOB path, **R-117f** an I/O liveness probe turns a wedged drive into an
|
||||
unkillable agent (HIGH — disqualifies a whole probe class).
|
||||
@@ -1,160 +0,0 @@
|
||||
# REPORT — hub v0.88.0: the WAL that never was (R-172), plus R-173 found
|
||||
|
||||
**Session artefact naming:** written as `REPORT-r172-hub-wal.md`, not `REPORT.md`, per this repo's
|
||||
parallel-session rule — the shared `REPORT.md` belongs to the controller boot-recovery session that
|
||||
ran immediately before this one and must not be clobbered.
|
||||
|
||||
**Repo:** `felhom.eu` (hub `v0.87.0` → **`v0.88.0`**) · **Trigger:** a `HOST STALE` banner the
|
||||
operator spotted on `hub.felhom.eu` after the previous session finished.
|
||||
|
||||
---
|
||||
|
||||
## 1. What the alarm actually was
|
||||
|
||||
**Not the agent, not the guest.** The agent was up **2 days**, never restarted, and actively
|
||||
reconciling; the controller was reporting normally (the header read "Last report just now", 0.190.0,
|
||||
10/10 containers). The failure was the hub **writing** the host report:
|
||||
|
||||
```
|
||||
20:26:34 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY)
|
||||
20:41:32 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY)
|
||||
20:42:32 [INFO] Host staleness: demo-felhom-8363b5 ok → stale (host_stale)
|
||||
20:42:33 [INFO] Operator email sent for demo-felhom/host_stale
|
||||
```
|
||||
|
||||
The chain, and the margin is exactly one retry wide: reports are every **15 min**, staleness fires at
|
||||
**30 min**, the hub returns **500** on `SQLITE_BUSY` without retrying, and the agent logs
|
||||
`keeping current interval` and waits a full interval without retrying either. **Two consecutive
|
||||
collisions = a false alarm.** It had already fired once that day (19:12:32, recovered 19:20:32).
|
||||
|
||||
**Was it caused by the preceding session?** Partly amplified, not caused. 13 collisions in one pod
|
||||
lifetime; **the first at 15:56 CEST, ~3 h before that session's first deploy**. 7 of 13 fell inside
|
||||
its window of ~13 controller restarts, which raises write concurrency — so the burst made a
|
||||
pre-existing fault more likely, and the fault was not new.
|
||||
|
||||
## 2. Root cause — the pragmas were never applied
|
||||
|
||||
The DSN was `?_journal_mode=WAL&_busy_timeout=5000`. That is **mattn/go-sqlite3** syntax. The driver
|
||||
is **modernc.org/sqlite v1.45.0**, whose `applyQueryParams` (confirmed at source in the module cache)
|
||||
reads only `_pragma`, `_time_format`, `_time_integer_format`, `_txlock` and `_inttotime` — and
|
||||
**silently ignores everything else**. No error, no warning.
|
||||
|
||||
So the hub ran in the default **rollback-journal** mode with **`busy_timeout=0`** for its entire life
|
||||
while its own source said WAL. In rollback-journal mode a reader excludes a writer, so rendering an
|
||||
operator page can block a host report — which is precisely the observed 500.
|
||||
|
||||
**The observable that proved it before any code changed:** a 128 MB `/data/hub.db` with **no
|
||||
`-wal`/`-shm` file beside it while the database was open**. In WAL mode those must exist.
|
||||
|
||||
This is the project's recurring class — a configuration asserting an invariant the code does not
|
||||
provide — and it is the second one this week.
|
||||
|
||||
## 3. The fix
|
||||
|
||||
```
|
||||
?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate
|
||||
```
|
||||
|
||||
| Parameter | Why it is not optional |
|
||||
|---|---|
|
||||
| `journal_mode(WAL)` | readers and one writer proceed concurrently, so a page render can no longer block a report; it is a property of the database FILE and persists once set |
|
||||
| `busy_timeout(5000)` | writers still serialise; without a timeout SQLite returns `SQLITE_BUSY` *immediately* rather than waiting |
|
||||
| `_txlock=immediate` | **the one that is easy to miss.** `database/sql`'s `Begin()` is DEFERRED, so a read-then-write transaction must upgrade its lock, and a failed upgrade is `SQLITE_BUSY_SNAPSHOT` — which **`busy_timeout` does not retry**. This store has **10+ `db.Begin()` sites and they are all write paths** (customer delete/reset, wg, appliance, pbsdr, telemetry, log bundles). WAL + busy_timeout alone would have shipped half a fix with a known un-retryable path left open |
|
||||
|
||||
**Retry options (b) and (c) from R-172 were deliberately NOT taken.** With readers no longer blocking
|
||||
writers and the upgrade path covered, a `SQLITE_BUSY` reaching a handler should now be rare enough to
|
||||
be a real signal; a retry would hide it. Revisit only on evidence.
|
||||
|
||||
## 4. Tests and the red-proof
|
||||
|
||||
**Every assertion reads the value back from the DATABASE, never the DSN string** — a string assertion
|
||||
would have passed happily for the entire life of the bug. Six tests in `internal/store/pragma_test.go`:
|
||||
|
||||
| Test | Asserts |
|
||||
|---|---|
|
||||
| `TestStorePragmasAreActuallyApplied` | runtime `journal_mode` = wal, `busy_timeout` ≥ 5000 |
|
||||
| `TestStoreWALFilesExistWhileOpen` | `-wal`/`-shm` exist beside an open DB — **the production signature, pinned** |
|
||||
| `TestStoreReaderDoesNotBlockWriter` | the CONSEQUENCE: a write during a held read succeeds |
|
||||
| `TestStoreConcurrentWritersDoNotReturnBusy` | 8 concurrent writers all wait rather than error |
|
||||
| `TestStoreTransactionUpgradeDoesNotReturnBusySnapshot` | 6 racing read-then-write transactions all commit |
|
||||
| `TestSQLiteDriverIgnoresMattnStyleParams` | guards the ROOT CAUSE: fails if the pragmas are "tidied" back to mattn form; skips itself with instructions if a future driver starts honouring them |
|
||||
|
||||
**Red-proof — restore the DSN that shipped.** Observed FAIL, then reverted with a passing control:
|
||||
|
||||
```
|
||||
journal_mode = "delete", want "wal"
|
||||
hub.db-wal is missing beside an OPEN database
|
||||
a write FAILED while a read was open: database is locked (5) (SQLITE_BUSY) ← the live error, exactly
|
||||
```
|
||||
|
||||
`go build ./... && go vet ./... && go test ./...` in `hub/` → **rc=0**. `scripts/repo_gates.py --fast`
|
||||
→ all 5 gates OK.
|
||||
|
||||
## 5. Operational consequence — handled, not discovered later
|
||||
|
||||
**A WAL database cannot be copied by taking `hub.db` alone.** A committed transaction may still live
|
||||
in `hub.db-wal`, so a bare `cat` yields a copy that **opens cleanly and silently omits the newest
|
||||
writes** — the worst possible shape for a credential lookup. The break-glass root-password retrieval
|
||||
in `documentation/operations/nodes.md` used exactly that command, and `_recovery-inventory-2026-07-28.md`
|
||||
records it as a past action that reads like a recipe. Both are now WAL-aware: copy the `-wal`
|
||||
alongside, `|| true` because an absent `-wal` is legitimate, and **shred both** (the WAL holds the
|
||||
same secrets).
|
||||
|
||||
Not hypothetical: the live `-wal` measured **729,272 bytes** during verification, all of which a bare
|
||||
`cat` would have dropped.
|
||||
|
||||
## 6. Live verification
|
||||
|
||||
- Image built and pushed; **`manifests/hub.yaml` 0.87.0 → 0.88.0** (the only thing ArgoCD deploys
|
||||
from), hard-refresh + deliberate sync (auto-sync is OFF). ArgoCD **Synced / Healthy**.
|
||||
- `/data/` now shows **`hub.db-wal` and `hub.db-shm`** beside the open DB — the exact observable whose
|
||||
absence proved the bug.
|
||||
- **Zero `SQLITE_BUSY` since the rollout.**
|
||||
- Host report landed at 21:11:33; staleness checker reports `2 ok, 0 stale` — `demo-felhom-8363b5`
|
||||
is back to `ok`.
|
||||
- `PRAGMA integrity_check` → **`ok`**, `journal_mode` → **`wal`**, all tables intact
|
||||
(hosts 4, customer_configs 6, host_recovery 4, host_reports 2756, events 2293). Read via the new
|
||||
WAL-aware copy recipe and shredded afterwards.
|
||||
|
||||
## 7. A mistake I made, and it caused a real outage
|
||||
|
||||
**I OOM-killed the hub pod three times** with a 60-concurrent page-render load test intended to prove
|
||||
the fix under contention. The pod's limit is **256 Mi**; 60 simultaneous renders of a heavy customer
|
||||
page exceeded it (`OOMKilled`, exit 137, readiness probe timeouts). The hub was unavailable for parts
|
||||
of ~6 minutes and recovered on its own.
|
||||
|
||||
Three things worth stating plainly:
|
||||
|
||||
1. **It was not the WAL change.** WAL's extra footprint is the 32 KB `-shm` mapping; the OOM was
|
||||
template rendering under concurrency I created.
|
||||
2. **I should not have run it.** DooPlex is **Tier 2 — precious**, and the hub is part of the
|
||||
recovery chain. A synthetic load test at that concurrency against a memory-limited pod on that box
|
||||
was the wrong call; the unit tests already proved the property, and the live proof needed was the
|
||||
`-wal` file plus a clean report — both of which I already had.
|
||||
3. **It did produce one piece of genuine evidence**, which does not excuse it: after three hard kills
|
||||
the WAL replayed cleanly and `integrity_check` returned `ok`, which is a real (if unplanned)
|
||||
demonstration of WAL crash-safety on this volume.
|
||||
|
||||
No data was lost. The pod is `Ready`, restart count 3, serving normally.
|
||||
|
||||
## 8. Backlog
|
||||
|
||||
- **R-172 → CLOSED**, with the root cause recorded as *the pragmas were never applied*, not as tuning.
|
||||
- **R-173 → NEW.** While checking who copies `hub.db` for the WAL change, I found `pvc/hub-data`
|
||||
carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly`
|
||||
are the only recurring jobs and both target `default`. **The hub database has no volume-level
|
||||
backup** — and it holds `host_recovery` (every box's break-glass root password), `host_escrow` +
|
||||
`host_escrow_superseded`, `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg peers.
|
||||
Filed rather than fixed: whether the exclusion is deliberate is a question for the operator, and the
|
||||
manual hot copy recorded in `_recovery-inventory` is not a backup. `grep` established the ID free.
|
||||
|
||||
## 9. Observations — noticed, NOT acted on
|
||||
|
||||
1. **The hub returns HTTP 500 for a transient lock**, which is what turned a retryable condition into
|
||||
an alarm. Left as-is deliberately (§3) so a surviving `SQLITE_BUSY` stays visible.
|
||||
2. **The agent does not retry a failed report**, so one collision costs a full 15-minute interval —
|
||||
half the staleness budget. Same reasoning; if collisions recur, this is the cheaper of the two
|
||||
retry fixes.
|
||||
3. **`store.New` sets no `SetMaxOpenConns`**, so `database/sql` may open unbounded connections against
|
||||
a single-writer database. Not changed here — WAL plus the immediate-lock covers the observed
|
||||
failure, and bounding the pool changes latency characteristics that nothing currently measures.
|
||||
@@ -1,203 +0,0 @@
|
||||
# REPORT — installer-channel record correction + R-29 filing (2026-07-29)
|
||||
|
||||
Two commits, documentation only. No code, no version bump, no CHANGELOG entry, no build, no deploy,
|
||||
no box touched. Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so root `REPORT.md` (the E-2
|
||||
increment-1 report) is preserved.
|
||||
|
||||
| # | Commit | Baseline | Scope |
|
||||
|---|--------|----------|-------|
|
||||
| 1 | `d4c07873ca0c3d3e547373a9fafc0e472a6535e8` | `36d635a4cdc1`, unmoved | Retract a false R-94/E-2d finding; open R-110 |
|
||||
| 2 | (this commit) | `d4c07873`, unmoved | File R-29 to the register; three record defects; this report |
|
||||
|
||||
---
|
||||
|
||||
## Commit 1 — `d4c07873`
|
||||
|
||||
### What was false
|
||||
|
||||
`36d635a4` recorded that `felhom-bootstrap.sh` fetches the installer **from the hub**, that the hub
|
||||
therefore serves 1.19.0, and that a fresh ISO install runs the pre-E-2 installer. All three wrong.
|
||||
The claim had propagated into two `OPEN-ITEMS.md` rows, the ranking rationale, and `ROADMAP.md:149`.
|
||||
|
||||
### Confirmation table — all PASS
|
||||
|
||||
| # | Claim | Read at | Result |
|
||||
|---|-------|---------|--------|
|
||||
| F1 | bootstrap fetches from the **website** | `scripts/iso/felhom-bootstrap.sh:96` | PASS — `INSTALL_URL="${FELHOM_INSTALL_URL:-https://felhom.eu/scripts/felhom-host-install.sh}"` |
|
||||
| F2 | hub-rendered command points at the same URL | `customer_unified.html:563`, `:1262` | PASS — and **three** emission sites, not two: `:563` static, `:1262` JS error branch, **`:1267` JS success branch** |
|
||||
| F3 | website serves `/scripts/` from a git-sync tree tracking `main` | `manifests/webpage.yaml` — nginx `:74-77`, sparse-checkout CM `:211-218`, git-sync `:272-281`, init `:299-307` | PASS — `--branch=main --period=30s --link=current`; sparse-checkout `/website/` + `/scripts/`; `location /scripts/ { root …/current; }`. No image build, no ArgoCD step |
|
||||
| F4 | `hostInstallVersion` selects nothing | `configs.go:28`, `:487`; `render_test.go:219`; `customer_unified.html:494` | PASS — repo-wide grep returns exactly those 4 code sites; all other hits prose. Rendered as a text label |
|
||||
| F5 | every generated flag is parsed by 1.22.0 | generator `customer_unified.html:1206-1239` vs parser `felhom-host-install.sh:1175-1212` | PASS — `--mode --cores --memory --vmid --node --acl-storages --operator-pubkey-file --preserve-state-from --skip-provision --dry-run --preflight-only --allow-new-leaf` (+ `--customer-id`); every one a parser case. **No functional gap** |
|
||||
| F6 | installer is 1.22.0 | `scripts/felhom-host-install.sh:187` | PASS |
|
||||
| F7 | the drift test is hollow | `render_test.go:219-221` | PASS — `strings.Contains(html, hostInstallVersion)` compares the constant to itself; passes at any value |
|
||||
|
||||
### Live command 1 — what the URL actually serves
|
||||
|
||||
```
|
||||
$ curl -fsS https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 '^SCRIPT_VERSION='
|
||||
SCRIPT_VERSION="1.22.0" # the SINGLE version source (F-1): -h, the run banners, and the hub
|
||||
```
|
||||
|
||||
### Live command 2 — the drift gate's real state
|
||||
|
||||
```
|
||||
$ python3 scripts/hostinstall_gates.py; echo "exit=$?"
|
||||
ok: SCRIPT_VERSION=1.22.0
|
||||
ok: header has no version literal
|
||||
FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0 (F-1: bump both together)
|
||||
ok: age is in the installed package set
|
||||
… (six further ok lines) …
|
||||
hostinstall gates: 1 FAILURE(S)
|
||||
exit=1
|
||||
```
|
||||
|
||||
### Phase 0 source read — PAIRING reaches the same installer invocation
|
||||
|
||||
Mode selection `felhom-bootstrap.sh:537-541`: a fresh VM with no baked customer-id calls
|
||||
`run_pairing`. On HTTP 200 from `/api/v1/appliance/poll` the loop writes the hub-delivered
|
||||
`FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it, and calls
|
||||
`run_direct` **in the same invocation** (`:495-499`). `run_direct` is the single site that fetches
|
||||
`$INSTALL_URL` (`:322-330`), builds the args (`:334`) and invokes `bash "$SCRIPT_TMP" "${args[@]}"`
|
||||
(`:343`). The customer it yields is the one the operator bound — claimable. **So the ISO leg is the
|
||||
spine for E-2d**, not an obstacle to it.
|
||||
|
||||
### Rows changed
|
||||
|
||||
| ID | Before | After |
|
||||
|---|---|---|
|
||||
| R-94 (line 14) | `READY — deferred until E-2d`, blocked on E-2d, false ISO/hub framing | `READY (XS)`, blocked on nothing, retracted + re-scoped to three legs |
|
||||
| R-94 (line 16) | duplicate row, `READY #2`, 1.19.0 vs 1.20.0 | **deleted** — merged |
|
||||
| R-110 | did not exist | opened, `WAITING-ON-OPERATOR (S)` |
|
||||
| E-2d | ISO implied as obstacle | Next-action appended; ISO is the spine |
|
||||
| ranked list | 1 R-95 · 2 R-94 (high-consequence) · 3 R-86 · 4 R-87 | 1 R-95 · 2 R-94 **de-ranked** · 3 R-86 · 4 R-87 · 5 R-110 |
|
||||
|
||||
Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
|
||||
`documentation/runbooks/day0-install.md`.
|
||||
|
||||
---
|
||||
|
||||
## Commit 2 — R-29 filing + record hygiene
|
||||
|
||||
### The §1.1 ruling — is R-29 the right home for a non-design-v2 gate? **Yes. Proceeded.**
|
||||
|
||||
R-29's title says *"the design-v2 green gates"*, and `scripts/hostinstall_gates.py` is not one — it
|
||||
comes from drill F-1 (2026-07-12) and postdates the item. Four things decide it anyway:
|
||||
|
||||
1. **R-29's own audit list already spans well beyond design-v2 subject matter.** It names
|
||||
`docker_run_volume_path_gate` (docker mount safety), `offbox_rename_gate`, `app_row_dedup_gate`
|
||||
and `manifest_bearer_gate` (secrets — `runbooks/secrets.md:76`). The title is a misnomer relative
|
||||
to the item's own body.
|
||||
2. **Part (b) — "the systemic half is the real item" in R-29's words — is stated with no
|
||||
design-v2 restriction.** It is about the *enforcement mechanism*: "the gates run only when a human
|
||||
remembers to run them… decide where they run (pre-push hook, `build.sh` step, or a CI job) and make
|
||||
a red gate block the train." That is gate-agnostic and repo-wide.
|
||||
3. **`hub_confirm_gate.py` is already on R-29's list and lives in the same `scripts/` directory** as
|
||||
`hostinstall_gates.py`. Wiring one and not the other would be arbitrary.
|
||||
4. **Identical failure shape, identical genre.** Both self-describe as mechanical grep-assertions
|
||||
(`hostinstall_gates.py:2`, `hub_confirm_gate.py:1-8`); both exist, assert something true, and are
|
||||
invoked by nothing.
|
||||
|
||||
R-29 has already absorbed one independent re-raise without minting an ID (2026-07-18 rehearsal note)
|
||||
and says so explicitly. This is the third. No new ID minted.
|
||||
|
||||
### Orphan-search evidence — `hostinstall_gates.py` and `hub_confirm_gate.py`
|
||||
|
||||
Re-established at `d4c07873`. Pattern `hostinstall_gates\|hub_confirm_gate`.
|
||||
|
||||
| # | Scope | Command | Result |
|
||||
|---|-------|---------|--------|
|
||||
| S1 | `felhom.eu`, all file types | `grep -rn "$PAT" . --exclude-dir=.git` | **19 hits, zero invocations.** All are docstrings (`scripts/hostinstall_gates.py:5`, `scripts/hub_confirm_gate.py:7`), code comments (`hub/internal/web/configs.go:27`, `scripts/felhom-host-install.sh:189`) or prose (`REUSE.md:62`, `CONTEXT.md:540,564`, `hub/CHANGELOG.md:371,1292,1351,1385`, `scripts/CHANGELOG.md:483,524`, 3 files under `documentation/audits/`, `ROADMAP.md:149,158`, `OPEN-ITEMS.md:14`) |
|
||||
| S2 | sibling repos | `grep -rln "$PAT" /mnt/5_hdd/felhom.eu/git --exclude-dir=.git` minus this repo | 3 files, all in `.claude-memory/` (`MEMORY.md`, `drtier-by-default-2026-07-12.md`, `polish-batch-2026-07-13.md`). Notes, not invokers |
|
||||
| S3 | `~/.claude` | `grep -rln "$PAT" /home/kisfenyo/.claude`, and targeted on `settings*.json`, `skills/`, `hooks/` | **Zero hits in settings, skills or hooks** — where an invoker would live. Remaining hits are `file-history/` (Claude Code's own backups of files edited in past sessions) and `paste-cache/` (pasted task specs). Neither is an invocation site |
|
||||
| S4 | git hooks | `ls -1 .git/hooks/ \| grep -v '\.sample$'` | **Empty — every hook is a `.sample` stub** |
|
||||
| S5 | build files | `find . -type f \( -iname Makefile -o -iname '*.mk' -o -iname justfile -o -iname 'Taskfile*' \)` | Only `hub/Makefile`; `grep -n gate hub/Makefile` → **zero occurrences** |
|
||||
| S6 | CI | `find . -type d \( -name .github -o -name .gitea -o -name .woodpecker* -o -name .drone* -o -name .circleci \)` | **Empty — `felhom.eu` has no CI configuration at all** |
|
||||
|
||||
Of the four gates in `scripts/`, only `site_gates.py` is mandated (`CLAUDE.md:153`);
|
||||
`manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`.
|
||||
|
||||
### Rows changed
|
||||
|
||||
| ID | Before | After |
|
||||
|---|---|---|
|
||||
| **R-29** | **absent from `OPEN-ITEMS.md`** (`grep -c` → 0) while present at `ROADMAP.md:158` since before the 2026-07-27 rebuild | **opened, `READY (S for (a) / M for (b))`**, owner CC |
|
||||
| R-94 | leg (b) stood alone | leg (b) carries `→ R-29` as its class. Row otherwise untouched |
|
||||
| E-2d | cited `:322-341` for an invocation at `:343` | cited `:322-343`, with the fetch / args / call each pinned separately |
|
||||
| R-95, R-86, R-87 | `READY #1`, `READY #3`, `READY #4` | `READY` — markers dropped (see below) |
|
||||
|
||||
Not added to the ranked list under *"Why the READY rows rank this way"*: that list is a top-N
|
||||
rationale, not a complete ordering (R-99, R-102, R-103, R-104, R-108, R-109, R-89, R-92, R-93 and
|
||||
E-2d are all READY and absent from it). Ranking R-29 was not asked for and would be a priority claim
|
||||
this task has no basis to make.
|
||||
|
||||
### §2.2 — markers dropped, not renumbered
|
||||
|
||||
The `#1/#3/#4` markers duplicated ranked-list positions 1/3/4 for exactly those three rows; two
|
||||
orderings of one set is what produced the hole when the `#2` row was merged in `d4c07873`. Removing
|
||||
the duplicate leaves the ranked list as the single maintained ordering.
|
||||
|
||||
### The other two record defects
|
||||
|
||||
- `ROADMAP.md:147` — cited a non-existent **R-164**; it means controller **v0.164.0**'s
|
||||
deliberate-stop filter. Corrected. (It had already cost one max-ID scan a false positive.)
|
||||
- `CONTEXT.md:564` — asserted in the present tense that the single VERSION source is *"gated by
|
||||
`scripts/hostinstall_gates.py`"*. The gate exists, asserts exactly that, is red, and runs nowhere.
|
||||
Corrected to say the cross-check exists but is not enforced, tracked as R-94 leg (b) / R-29.
|
||||
`hub/internal/web/configs.go:27` carries the same false claim in a code comment and was left alone
|
||||
— R-94 leg (b) territory, needs a hub build.
|
||||
|
||||
Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
|
||||
`documentation/backlog/README.md`, `CONTEXT.md`, this file.
|
||||
|
||||
---
|
||||
|
||||
## Commit 3 — `de5a3e5` — the record-hygiene rider
|
||||
|
||||
Baseline `7383400a`, unmoved. Four XS items from `7383400a` §9; the three deferred observations stay
|
||||
deferred.
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `CONTEXT.md:540` | *"`scripts/hub_confirm_gate.py` enforces"* → asserts, but is not enforced (R-29). Third instance of the class after `:564` and `configs.go:27` |
|
||||
| `REUSE.md:62` | Same claim, *"enforces zero"*. The **rule stays** — never native `confirm()`/`prompt()` is correct guidance and this is a reuse-reference row — only the enforcement claim changes |
|
||||
| `OPEN-ITEMS.md:4` | Root `REPORT.md` = overwritten per-session; `REPORT-<topic>.md` = non-clobbering sibling (`CLAUDE.md:82-87`), 14 of them. Prohibition unchanged |
|
||||
| `OPEN-ITEMS.md:55` | Heading scoped to *"the **TOP** READY rows"* + a half-sentence that it is deliberately not a full ordering. **No row added to the list** |
|
||||
|
||||
`hub/internal/web/configs.go:27` — the fourth instance — left alone (R-94 leg (b), needs a hub build).
|
||||
|
||||
### Part 3 NOT done — its stated evidence is false
|
||||
|
||||
The spec proposed recording a convention: *"`ROADMAP`'s P3 `Status` column only ever takes `idea` or a
|
||||
`SHIPPED`/`PROVEN-LIVE` variant, and never takes `READY`, `BLOCKED` or `WAITING-ON-OPERATOR`"*, and
|
||||
concluded from that vocabulary split that `ROADMAP` `Status` is **disposition** while `OPEN-ITEMS`
|
||||
`State` is **live actionability**.
|
||||
|
||||
Refuted inside P3 itself (`ROADMAP.md`, lines 113–172):
|
||||
|
||||
| Line | ID | P3 `Status` cell |
|
||||
|---|---|---|
|
||||
| 151 | R-90 | `BLOCKED on Hetzner CX33 availability (operator, 2026-07-27)` |
|
||||
| 152 | R-91 | `WATCHING — gated on demo-felhom's first post-migration PBS backup` |
|
||||
| 150 | R-110 | `idea — found 2026-07-29, **WAITING-ON-OPERATOR (a ruling, not a defect)**` |
|
||||
| 137 | R-86 | `**NEXT — operator ruling 2026-07-27**` |
|
||||
|
||||
And P2's recovery-gaps sub-table (lines 99–107) uses a bare `READY` for nine rows, including `E-2`.
|
||||
|
||||
Both halves fail: the column **does** take the `OPEN-ITEMS` vocabulary, and `BLOCKED`/`WATCHING`/
|
||||
`NEXT` are live-actionability words, not dispositions. The real shape is a free-text status field
|
||||
that has absorbed both vocabularies over time — usually disposition, sometimes not. Writing the
|
||||
proposed convention down would install a **new false invariant** in the commit chain whose whole
|
||||
purpose was removing them, so per the spec's own instruction (*"If you think this ruling is wrong, say
|
||||
so and make no edit"*) `documentation/backlog/README.md` was not touched.
|
||||
|
||||
The narrow question that prompted it still resolves benignly: R-29 and R-94 reading `idea` in
|
||||
`ROADMAP` and `READY` in `OPEN-ITEMS` is **not** a defect and needs no sync. What is unsupported is
|
||||
generalising that into a rule.
|
||||
|
||||
---
|
||||
|
||||
## Standing note
|
||||
|
||||
**No gate was wired, fixed, run as a hook, or deleted by any of the three commits.** Filing R-29 is
|
||||
not doing R-29; the enforcement decision (pre-push hook / `build.sh` step / CI) and the audit of the
|
||||
remaining eight gates are R-29 part (b), M-sized, and its own task.
|
||||
@@ -1,94 +0,0 @@
|
||||
# REPORT — Session C: R-113, R-114, R-112 proven on a real box; C5 fails on a new defect (2026-07-29)
|
||||
|
||||
`RUNBOOK-session-c-2026-07-29.md`. Full evidence: `documentation/audits/SESSION-C-2026-07-29.md`.
|
||||
Root `REPORT.md` untouched.
|
||||
|
||||
## Verdicts
|
||||
|
||||
| Claim | Fix | Verdict |
|
||||
|---|---|---|
|
||||
| **C4** — offer appears and moves the target | R-112 | ✅ **PASS** |
|
||||
| **C3** — absent target tells the truth, offers nothing | R-114 | ✅ **PASS** |
|
||||
| **C5** — `backup_target_absent` + `backup_target_restored` | R-113 | ❌ **FAIL** — generic alarm, specific recovery → **R-116** |
|
||||
|
||||
**All three shipped fixes work.** R-113's gate fires in **4 seconds** (E-2d measured zero over 4½
|
||||
minutes). R-114's message is correct. R-112's banner reaches the customer. C5 fails on a **fourth,
|
||||
separate defect that was unreachable until R-113 made the gate fire at all.**
|
||||
|
||||
## What the drill box ran
|
||||
|
||||
Agent **0.114.0 from the Day-0 manifest** — the shipped binary, so C5 tested the real artifact and
|
||||
closed R-115's observation 1 for R-113. Controller **0.186.0 hand-deployed** after install (§3.1
|
||||
ruling (a)); the vouched golden bakes 0.185.1, so **C3/C4 prove the code, not the shipped golden** —
|
||||
that lag is filed against R-115, not a new ID.
|
||||
|
||||
## The three headline observables
|
||||
|
||||
**R-113** — detach at 18:43:50, gate at **18:43:54**, on exactly the shape that defeated it before:
|
||||
```
|
||||
raw /mnt/mentes : NOT mounted
|
||||
bind /mnt/felhom-drives/mentes : /dev/sdb[/felhom-data] <- the stale bind SURVIVED
|
||||
```
|
||||
|
||||
**R-114** — with the target absent: absent copy 1, **system-disk copy 0**, **offer block 0**. Both of
|
||||
E-2d's falsehoods gone.
|
||||
|
||||
**R-112** — banner element 1 and the never-configured copy in the HTML; after the wizard, the offer
|
||||
with `data-path="/mnt/felhom-drives/mentes"`. Healthy renders nothing, **proven positively**: idle
|
||||
delta 0 `/backup/tiers` calls, page-load delta **+1**, single caller ⇒ the seam ran and chose silence.
|
||||
|
||||
**Decline path proven** (registration confers no role), `restart_required:true`, agent did **not**
|
||||
self-restart, in-flight check recorded before I restarted it, wrapper created the storage at the
|
||||
drive's own mountpoint.
|
||||
|
||||
## C5's failure
|
||||
|
||||
```
|
||||
absent : Event pushed: storage_disconnected (error) <- GENERIC
|
||||
return : Event pushed: backup_target_restored (info) <- SPECIFIC
|
||||
```
|
||||
|
||||
`backup_target_absent` count **0** across the whole run. The alarm and its recovery cannot be matched
|
||||
— precisely what `notifyDriveReturned`'s own comment forbids.
|
||||
|
||||
**Root cause (R-116):** `driveTargetByPath` builds `out[GuestPath] = d.BackupTarget`, but the drive is
|
||||
**two `/disks` rows** and the flag and the guest path sit on different ones — the `felhom-backup`
|
||||
storage row carries `BackupTarget: true` and gets a guest path only while classified user-data; the
|
||||
registry union row carries the guest path and **never assigns `BackupTarget`**. Absent ⇒ they separate
|
||||
⇒ generic. Return ⇒ they rejoin ⇒ specific. v0.184.1 fixed the *keying*, not this.
|
||||
|
||||
## Mirror + over-correction guard — PASS, with a caveat
|
||||
|
||||
Non-target drive detached ⇒ `storage_disconnected`, `backup_target_absent` count 0. **Over-correction
|
||||
guard passes**: both drives present ⇒ 0 ABSENT lines, target stayed healthy — R-113's stricter presence
|
||||
did not make a healthy drive read absent. **Caveat: the mirror passes trivially**, because the target
|
||||
also produced the generic event; it confirms no over-correction but cannot confirm discrimination.
|
||||
|
||||
## Record
|
||||
|
||||
- `OPEN-ITEMS.md` — **R-113, R-114, R-112 → SHIPPED + PROVEN-LIVE**; **R-116 opened** (READY (S), P1);
|
||||
**E-2 and E-2d CLOSED as partially proven** with R-116 as the one named open leg, per the runbook's
|
||||
§9 decided-in-advance rule.
|
||||
- `ROADMAP.md` — R-116 under P1.
|
||||
- **Capability map NOT touched** — it still has **no E-2 / backup-target rows at all**, so no row could
|
||||
be moved to PROVEN-LIVE. Creating them is a design act, not a validation act. Third session running
|
||||
that this has been noted.
|
||||
|
||||
## Teardown
|
||||
|
||||
VM destroyed, storage removed, **`pvesm status` after == before** (`local-lvm` 38.78 %), guest 9201 and
|
||||
`drill-r50` untouched. **Customer ruling: DELETE**; attempted and correctly refused (`host … is
|
||||
ONLINE`) — deletable once the destroyed host ages to DOWN (>1 h), command recorded in `OPEN-ITEMS.md`.
|
||||
|
||||
## What did not happen
|
||||
|
||||
`backup_target_absent` never fired, so its severity, Hungarian copy and hub routing remain unexercised
|
||||
— R-116 blocks them. The offer was accepted via the endpoint the button POSTs, not a browser click (no
|
||||
browser automation on DooPlex); the rendered control and its non-auto-submission were verified in HTML.
|
||||
The stale bind still naming a dead device node after return was observed, not investigated.
|
||||
|
||||
## The arc
|
||||
|
||||
E-2 ends here. Its stated definition of done is **R-106 + R-109, R-108 and D5** — none of which this
|
||||
detour touched. The detour was worth taking: it found six real defects (R-111 through R-116), four of
|
||||
them customer-affecting, none of which any unit suite had caught.
|
||||
@@ -1,98 +0,0 @@
|
||||
# REPORT — tester gate: golden re-baked to 0.188.0, fresh-install proof PASSED (2026-07-31)
|
||||
|
||||
Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so the shared `REPORT.md` (E-2 increment 1) is
|
||||
not clobbered. Full record with every observable: `documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md`.
|
||||
|
||||
## Outcome
|
||||
|
||||
**§7.2 — YES: a fresh install is safe to hand to an external tester.** ISO boot → claimable,
|
||||
app-serving box in ~10 minutes unattended, and an app's data restored **from the drive with the
|
||||
guest's `app.yaml` gone**, proven readable by the application over its own TCP path.
|
||||
|
||||
**Golden 0.186.0 → 0.188.0** baked, published, vouched. **No ISO rebuilt** — Part 0 proved none was
|
||||
needed. No existing box changed; floor still v0.156.0, MinAgent still 0.113.0.
|
||||
|
||||
## Part 0 — the ISO does not need rebuilding
|
||||
|
||||
Verified against the ISO **on disk**, not from source. It bakes exactly three Felhom payloads
|
||||
(`felhom-bootstrap.sh`, its unit, the secret-free pairing env) — full-base64 match, 1 hit each — and
|
||||
**0** hits for `SCRIPT_VERSION="1.2`, `felhom-controller`, `vzdump-lxc-9100`. The installer is fetched
|
||||
at run time (`felhom-bootstrap.sh:96`) and the live URL is byte-identical to repo HEAD
|
||||
(sha `ab8b283e…`, v1.22.0, committed six days *after* the ISO). The golden arrives via the hub-vouched
|
||||
artifact manifest (`felhom-host-install.sh:423-433`). The one genuinely baked, drift-capable thing is
|
||||
`felhom-bootstrap.sh` itself — currently at repo HEAD.
|
||||
|
||||
**Proven live**, not just argued: the fresh box ran `felhom-host-install v1.22.0` and fetched golden
|
||||
**v0.188.0**, sha-verified.
|
||||
|
||||
## Part 1 — bake / publish / vouch
|
||||
|
||||
Baked **0.188.0**, not the brief's 0.187.0: 0.187.0 lacks D5, and Part 2 step 6 *is* the D5 claim, so
|
||||
that golden could not have passed the proof this task exists for. 0.188.0 satisfies R-120 anyway.
|
||||
Stated rather than absorbed, per standing rule 4.
|
||||
|
||||
GOLDEN_VERSION=0.188.0
|
||||
GOLDEN_SHA256=7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299
|
||||
|
||||
Three observables: 404 pre-gate (with a 200 control on 0.186.0 so it is not vacuous), then an
|
||||
**anonymous** download returning `http=200 bytes=649310288` and a matching sha; the manifest read back
|
||||
showing `0.188.0` selected; and the consumer call `GET https://hub.felhom.eu/api/v1/artifacts/sess-g`
|
||||
returning the pair. Plus a fourth: a real fresh box fetched and sha-verified it.
|
||||
|
||||
**R-120's gate evaluated and allowed.** Exercised both ways rather than inferred from silence —
|
||||
vouching 0.185.1 first produced `flash=golden_behind_fleet`, the logged `artifact vouch REFUSED`, and
|
||||
**no write** (the manifest still read 0.186.0); then 0.188.0 produced
|
||||
`Artifact manifest set: agent=0.118.1 golden=0.188.0`.
|
||||
|
||||
## Part 2 — the clean-install proof, on demo-hp
|
||||
|
||||
All seven steps PASS. Venue was demo-hp (Tier 0, the designated drill host) using the scratch dir
|
||||
storage at `/mnt/nvme-1tb` that `target-selection.md:38-40` names; `local-lvm`, `drill-r50` and both
|
||||
9201s untouched.
|
||||
|
||||
Highlights: real day-0 pairing → bind → install; a **real** claim (the code is emailed-only, R-119 —
|
||||
the operator relayed it), with the gate flipping `dashboard not yet claimed` → `authentication
|
||||
required`; controller **0.188.0** confirmed *from the box*; **rallly** (postgres) + **homebox**
|
||||
deployed through the real endpoints.
|
||||
|
||||
The D5 leg: recovery unit `portable-carried=2/2, withheld=0`; the carried `DB_PASSWORD` matched the
|
||||
live one **by fingerprint** (`14c8f515…`, never printed); guest `app.yaml` moved aside; restore
|
||||
returned `secrets recovered=2/2`. Step 7 read the data from **rallly's own network namespace** over
|
||||
TCP to `rallly-postgres` — not the localhost trust socket that produced D5's false pass — and the same
|
||||
path with a wrong password returned `FATAL: password authentication failed`, proving the credential
|
||||
does real work. **The discriminator held: PRE-BACKUP row = 1, POST-BACKUP row = 0.**
|
||||
|
||||
## Part 3 — runbook integrity
|
||||
|
||||
`RUNBOOK-manual-build.md` told the reader to use a "RECORDED" qemu line that is itself labelled
|
||||
*reconstructed*, and whose source says it *"was never saved"*. The real invocation is now captured
|
||||
from this bake and recorded as canonical in **§4.0**, alongside the bake/publish/teardown steps, the
|
||||
template-rot warning and where the R-120 gate actually lives. The old runbook's deviation entry is
|
||||
marked SUPERSEDED with a forward pointer.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
1. VM 310 destroyed with `--purge --destroy-unreferenced-disks 1`; `/mnt/nvme-1tb/images/` empty.
|
||||
2. `cc-scratch` removed, `storage.cfg` back to its original four entries; `felhom-backup` available
|
||||
**926 492 284 KiB before and after** — space returned exactly.
|
||||
3. Hub: **`sess-g` and its host record DELETED, full cascade** — `customer DELETE cascade COMPLETE
|
||||
for sess-g (journal #8) — full teardown`, residue purged including `appliance_registrations=1`.
|
||||
Verified positively: `/configs` and `/hosts` both loaded (10160 / 9880 bytes) with **0** hits for
|
||||
`sess-g` and 0 for the appliance UUID. The gate refused twice first (409 host ONLINE, then 400
|
||||
missing acknowledgements) — the record cannot be deleted until the destroyed box ages out of
|
||||
ONLINE, ~30 min. **`sess-f` deliberately NOT deleted** (R-131); its command is in the audit §7.1.
|
||||
Secrets shredded in the guest and on the box.
|
||||
|
||||
## Findings — filed, none fixed
|
||||
|
||||
`R-128` ISO_VERSION/SCRIPT_VERSION comment is false · `R-129` demo-hp's "no baked SSH key" is stale
|
||||
(key auth works) · `R-130` `HARD_MIN_LVM_GIB` warns and proceeds — a hard min that is not hard ·
|
||||
`R-131` `sess-f` is a fourth orphaned scratch customer · **`R-132` — `curl -w '%{redirect_url}'`
|
||||
printed the hub operator password into a session transcript; `HUB_PW` needs rotating.**
|
||||
|
||||
## Not done, deliberately
|
||||
|
||||
No ISO built; no defect fixed; no golden deployed to an existing box; no floor or MinAgent change;
|
||||
offsite/PBS-DR legs not exercised (the task forbids pointing anything at production PBS or the real
|
||||
restic offsite, so `sess-g` ran DR-tier off); Campaign 10, the demo-hp repartition and subdomain
|
||||
onboarding untouched.
|
||||
@@ -1,97 +0,0 @@
|
||||
# REPORT — SPIKE 4: can a `.deb` in the ISO deliver the stub on an interactive install? (2026-07-31)
|
||||
|
||||
> Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub
|
||||
> v0.85.0 session and the second session in a shared clone never touches it. Supersedes this file's
|
||||
> Spike 1–3 contents.
|
||||
|
||||
**Class: Spike.** Findings only — no production file changed, no release ISO built, nothing published.
|
||||
Evidence: `documentation/audits/SPIKE-universal-iso-4-2026-07-31.md`.
|
||||
|
||||
## The answer is yes, and it was measured with the negative control in the same box
|
||||
|
||||
One ISO, 15 GRUB entries, a trivial probe `.deb` injected into `/proxmox/packages/`. Two VMs on
|
||||
demo-hp built with `qm` so the run was visible in the web console: **400 interactive**, **401
|
||||
automated control**.
|
||||
|
||||
On the **interactive** install (`spikefour.felhom.eu`):
|
||||
|
||||
- the package is installed — `ii felhom-spike4-probe 0.0.1`
|
||||
- its **postinst ran** — marker file present, content intact
|
||||
- it **enabled a systemd unit**, and **that unit fired on first boot** (uptime 7.98 s, `pid1: systemd`)
|
||||
- and on **that same machine**, `proxmox-first-boot` is not installed and `/var/lib/proxmox-first-boot`
|
||||
does not exist — Spike 3's negative reproduced, not assumed
|
||||
|
||||
So the two delivery mechanisms are independent, and the one that survives the path we are actually
|
||||
shipping is the `.deb`. **The product — insert the stick, install Proxmox normally choosing your own
|
||||
disk and password, box sets itself up and waits for a claim code — is now measured rather than hoped
|
||||
for.** With one honest caveat: what was measured is a trivial probe package, not Felhom's real stub.
|
||||
Packaging the real stub and confirming pairing end-to-end is the last step before a spec (~60 min).
|
||||
|
||||
## What a postinst may and may not do
|
||||
|
||||
Identical on both paths: `pid1 = unconfigured.sh`, **no running systemd**, `/proc` and `/sys` mounted,
|
||||
and **`systemctl enable` succeeds** (it wrote the symlink). Network and DNS *happened* to be up —
|
||||
inherited from the installer's own DHCP.
|
||||
|
||||
Four constraints for the real postinst, so they get written against rather than discovered:
|
||||
|
||||
1. Never `systemctl start` or `daemon-reload` — there is no systemd running. `enable` is the only verb.
|
||||
2. **Never require the network**, despite it being present here. A box installed with the cable out
|
||||
gives a postinst no route, and a failing postinst breaks the customer's install.
|
||||
3. Never fail — guard everything, `exit 0`.
|
||||
4. Do the real work in the unit at first boot, where systemd, network and a booted kernel exist.
|
||||
|
||||
## Two smaller results
|
||||
|
||||
**The repack preserves the `.deb`, but not naively.** `xorriso … -boot_image any replay` fails with
|
||||
*"Overlapping MBR partition entries"* — and `iso-repack.sh:270-292` already documents that exact
|
||||
failure and its fix. Mirroring it produced a working image (19 El Torito entries; the `.deb` extracted
|
||||
back out is byte-identical). So this is an insertion into an extract→modify→re-master cycle our repack
|
||||
already performs, not a new build stage.
|
||||
|
||||
**Q3:** `iso-repack.sh:100-106` refuses an ISO without `auto-installer-mode.toml`. It is a guard, not
|
||||
a structural requirement, and its reasoning is sound for the shape it was written for — already R-155,
|
||||
cited exactly here. With no mode file the stock grub.cfg does not emit the Automated entry at all; with
|
||||
a mode file but no answer, that entry aborts safely and loudly.
|
||||
|
||||
## A correction I owe you from last session
|
||||
|
||||
**R-153 is retracted.** The register grep this task mandated shows R-94 already carries it verbatim at
|
||||
`OPEN-ITEMS.md:15`, status `READY (XS)`, with leg (b) being precisely "the gate fails today and is
|
||||
invoked by nothing" — and R-29, the class, says in terms *"do not mint a new ID for a new instance."*
|
||||
Spike 3 filed a duplicate.
|
||||
|
||||
**And the substantive half of that Spike 3 claim was wrong.** I wrote that the drift left the customer
|
||||
page's install-command generator "targeting a flag surface three minor versions stale." R-94 explicitly
|
||||
retracts exactly that reading: the constant selects no script — it renders as a text label, and the
|
||||
command beneath it fetches the script the website git-syncs from `main`, so **1.22.0 is what every
|
||||
install already gets**. It is a wrong number on your screen and nothing more. I overstated it.
|
||||
|
||||
## Still unknown
|
||||
|
||||
**The real stub has not been packaged** — that is the one thing between here and a build spec.
|
||||
Also unproven: `dpkg --configure -a` ordering for a package with dependencies; an ISO that never went
|
||||
through `prepare-iso` (blocked by R-155, which this spike was fenced from changing); and the Graphical
|
||||
installer, where the result should hold *a fortiori* since the `.deb` path is in `Install.pm`, shared
|
||||
by all front-ends — but that is inference, not measurement.
|
||||
|
||||
**Spike 3's Q3** — the real stub at `before-network` — **this session did not touch it.** Note it is
|
||||
now partly superseded: on the `.deb` route the unit's ordering comes from the unit file, not from
|
||||
`[first-boot].ordering`.
|
||||
|
||||
## R-rows
|
||||
|
||||
**None opened.** Each candidate was grepped against the register first: the delivery result is a
|
||||
positive finding, the postinst constraints belong in the build spec, and the repack guard is already
|
||||
R-155. **R-153 retracted** into R-94 leg (b) / R-29.
|
||||
|
||||
## Teardown
|
||||
|
||||
All three layers plus the scratch storage, verified positively. demo-hp: VMs 400/401 purged, **storage
|
||||
`spike4` removed** (`storage.cfg` back to 4, `grep -c spike4` = 0), `/mnt/nvme-1tb/images/` empty,
|
||||
**disk usage 6.6 G — identical to pre-spike**, probe ISO and driver removed, 0 loop devices,
|
||||
`drill-r50` stopped and untouched, 9201 running, `felhom-backup` unmodified, nothing on `local-lvm`.
|
||||
DooPlex: workspace scratch **4.8 GB removed**, scratchpad **3.3 GB → 88 K**, both throwaway passwords
|
||||
destroyed, 17 ISOs in `out/` untouched, no production file modified. **Hub-side: nothing created** —
|
||||
the VMs took LAN DHCP leases but never ran `felhom-host-install.sh` or contacted the hub; verified by
|
||||
fetching and searching the customer list. Nothing published.
|
||||
Reference in New Issue
Block a user