diff --git a/REPORT-campaign10.md b/REPORT-campaign10.md deleted file mode 100644 index 0f29432..0000000 --- a/REPORT-campaign10.md +++ /dev/null @@ -1,113 +0,0 @@ -# REPORT — Campaign 10, two-storage adversarial soak (2026-07-31) - -Follows `REPORT-campaign7/8/9.md`. Root `REPORT.md` is another session's (hub v0.85.0) and was not -clobbered — same shared-clone reasoning as `REPORT-iso-release.md`. - -**Full audit + evidence:** `documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`, -`documentation/tests/campaign10-evidence-2026-07-31/`. - -## The sentence that matters - -**Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the -"drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No -resource leak over 13.5 h. Two findings: R-156 and R-157.** - -## What was established - -- **Venue** — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the - `/mnt/nvme-1tb` **mount root** (a subdirectory would have emitted `storage_disconnected` for demo-hp - all night — the exact signal I1/I2 discriminate). -- **Baselines, all read fresh.** controller `main` 0.188.0, **golden 0.188.0 (not behind)**, agent - 0.119.0 published+vouched, hub 0.86.0, ISO **1.26.1** (`f3cc86d5…`, round-trip verified live). - The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129. -- **Isolation gate — both denials captured, each with a positive control.** The PBS control **failed - first**: four clean-looking 403s were worthless because the token was denied on its own datastore - too (PBS token privilege separation). Fixed, re-run, denials stand. -- **A1** fresh install from the **published** ISO. 1.26.1 is a public release image — verified against - its bytes that it has no auto-install path — so it was driven blind via screendump + `sendkey` - through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which - would otherwise have been mangled and locked the box out. -- **A2** claimed for real; discriminator flipped `dashboard not yet claimed` → `authentication required`. -- **A3** both drives enrolled through the **real** endpoint; `mentes` accepted as backup target via - the offer flow, ending `degraded:false / target:felhom-backup` — the I5/I6 healthy baseline. - Four apps healthy spanning both sides of D5's split (4 × `type: secret`, 1 × `type: password`). -- **A4** discriminators seed and read back across all four apps; rallly's over the path - `DATABASE_URL` actually names, not the trusted socket that produced D5's false pass. - -## Phase B — the soak - -Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped -deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**). -1 461 invariant checks in total. - -- **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale, - never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as - skipped rather than silently green — it fired once. -- **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place - variants and 330 secret-class assertions covering both sides of D5's split. -- **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced - `backup_target_absent` and a clean recovery. -- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via - `bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names - it. That is the case R-117's spike called "the worse half". -- **RTO, both bands measured.** S: 66 MB → **42.0 s** / **41.4 s** across two passes (66 restores). - M: 21.1 GB → **608 s** mean over 2 reps, both returning the correct discriminator. 327× the data - cost 14.5× the time, giving **RTO ≈ 40 s + 26.9 s/GB** (backup ≈ 29 s + 17.4 s/GB). The fixed ~40 s - dominates below ~1.5 GB — that is the S band, and why its numbers clustered so tightly. -- **Capacity ceiling, and the more consequential result:** a DB-backed app's recovery unit is **1.90×** - its data (volume tar + SQL dump). The default `/mnt/sys_drive` is **20 GB**, so on a default box the - largest locally-backupable app is **≈ 10 GB** — **the M band does not fit at all** without a - per-customer `SysDataGrowGB`. **RPO still not measured.** -- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds - flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots - and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** — - bounded by journald, but a lot of logging. -- Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that - was not first observed live. -- **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system - disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the - soak would have produced green cycles that tested nothing cross-drive. -- **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that - never landed became a fake "stale restore", and a real one would have looked identical. Fixed and - red-proofed before restarting. - -## Findings - -- **R-156 (new)** — **papra's data is neither persisted nor backed up, and it reports healthy.** The - template mounts `papra_data:/app/data`; the app writes `/app/app-data/db/db.sqlite`. The volume is - empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB - sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its - Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed. -- **R-157 (new)** — **bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two - mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while - docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences, - intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is - exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp - check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on - every channel. A settle-condition fix closes A and leaves B open. Not fixed. -- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier - (`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred, - `hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign - therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by - not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows. - -## What did not run - -**12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place, -kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive). -Previously reported as 6 of 12 — that was run 1 only. - -**Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not -automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed. -`kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom -covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed. - -**Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34–c39 clean on -every invariant. Beyond 39 is untested, not proven clean. - -## Teardown — OWED, nothing removed - -Still intact: the rig is reusable for the atoms that did not run. VM 311, `c10-scratch`, PBS datastore -`felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition: -DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four -orphaned scratch customers left by exactly this omission. diff --git a/REPORT-campaign7.md b/REPORT-campaign7.md deleted file mode 100644 index 7ef6f74..0000000 --- a/REPORT-campaign7.md +++ /dev/null @@ -1,43 +0,0 @@ -# REPORT — CAMPAIGN 7 (felhom.eu side: docs only) - -> Written as `REPORT-campaign7.md`, **not** the shared `REPORT.md`, per the convention this run -> added to `CLAUDE.md`: `REPORT.md` is overwritten, so a second concurrent session in this repo -> would clobber it. This session's implementation work was in `app-catalog-felhom.eu`; here it only -> touched documentation. - -**Run:** 2026-07-18 evening → 2026-07-19 morning. **Class:** campaign (record-and-rank + a defined -allowed-fix set). **Implementation repo:** `app-catalog-felhom.eu` (see its `REPORT.md`). - -## What changed in this repo - -| file | change | -|---|---| -| `documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` | **new** — method, uninstall-semantics map, trio detail, full 53-app matrix, ranked findings, coverage | -| `documentation/backlog/ROADMAP.md` | **+3 items** — R-40 (multi-hop major upgrade path), R-41 (no standing catalog deployability check), R-42 (sidecar-major ruling) | -| `CLAUDE.md` | REPORT.md parallel-session rule: the second session writes `REPORT-.md` | - -No hub/agent/scripts/website code was touched (campaign scope: catalog + docs). - -## Headline for this repo's readers - -1. **Uninstall semantics map row PARTIAL → PROVEN** (campaign doc §2), with live evidence from all - three trio apps: remove requires stop first; named docker volumes are **always destroyed** - (including the app's database); HDD bind-mount data and `backups/primary/` survive unless - explicitly ticked; images are kept; `app.yaml` goes, the template stays; the per-app **offsite - toggle survives** the uninstall while tier-2 config is cleared. The confirmation modal does warn - about the volumes, so there is **no consent gap**. -2. **A lying healthcheck takes an app OFF-LINE, it does not merely mislead.** Traefik will not route - to an `unhealthy` container, so a probe that cannot execute → permanent unhealthy → **404 to the - customer while the app serves 200 on its own port**. 7 of 53 apps were in that state. -3. **The pre-flight gate's own signal is missing:** the 0.145.0 → 0.146.0 floor-lift emitted no - `controller_updated` event, though the identical bootstrap path emitted one for 0.143.0 → 0.145.0 - two hours earlier (§0, finding F1). The box did converge — golden, floor and runtime all agreed — - but the event trail under-reports version transitions. - -## Open items owned outside this repo - -- **plant-it / wanderer** — images do not resolve at all (neither the new tag nor the one the - catalog already ships). Upstream research needed; recorded as findings, not deletions. -- **gokapi** — pinned back to v1.9.6; v2 needs the seeded `config.json` regenerated. Security- - relevant, should not sit on a superseded line indefinitely. -- **glance** — never had a seeded `glance.yml`; proven pre-existing. diff --git a/REPORT-campaign8.md b/REPORT-campaign8.md deleted file mode 100644 index 07d1a1c..0000000 --- a/REPORT-campaign8.md +++ /dev/null @@ -1,62 +0,0 @@ -# REPORT — CAMPAIGN 8: the backup & restore subsystem (2026-07-27/28) - -Adversarial, destructive, unattended run against `demo-felhom`, `demo-hp` and `ep0`. -Full report: `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`. -Evidence: `DooPlex:~/campaign8/evidence/` (103 files, 35 MB, written continuously by 11 collectors). - -**No production code was changed.** Findings are recorded and ranked, never fixed inline, per the -campaign's own rules. - -## Scope safety -`peti-felhom`, its namespace and `u629488-sub2` were never touched. Phase 0 established with five -documented probes that peti has **no data at all** in `felhom-offsite`, which is what made the -operator-approved 100% datastore-fill safe. The 13 GB rollback copy `/srv/pbs-felhom` on ep0 is -intact. - -## Findings - -| # | Finding | Severity | Class | -|---|---|---|---| -| F-CRIT-1 | An app that fails to restart after a quiesce **never alarms**, on any channel | **HIGH** | DEFECT | -| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT | -| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) | -| F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT | -| F-LEAK | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed | MEDIUM | DEFECT (root-caused by fault 18) | -| F-REBOOT | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT | -| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT | -| F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT | -| F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational | - -Both HIGH findings are in the same place: **the system's ability to tell you a backup did not -happen.** Both cite the code and the comment that asserts the property the code does not provide. - -## What is now proven that was not before -R-88 breaker arming **and its full ladder** (15m/30m/1h/2h/4h/4h-cap) · per-tier isolation under a -real one-tier-fails case · `whole_guest_backup_failed` end-to-end with correct tier attribution · -R-97c operator-only routing verified against the hub DB (zero customer rows with `status='sent'`) · -`whole_guest_backup_recovered` + the R-68 pairing gate firing live · **`age_state=absent`** · -R-97b's suppression half · the crash-recovery unquiesce by an actual SIGKILL (1 s) · -**R-87 — the first restic restore round-trip ever performed**, byte-verified (6/7 sha256 identical, -the 7th explained) · R-82 one-quiesce-two-tiers · single-flight on two independent paths. - -## Restore round-trips -restic (R-87) · local vzdump → fresh CT · PBS offsite → fresh CT · corrupted snapshot → fails -cleanly. `mount_parity` exact on both whole-guest tiers, `unprivileged: 1` preserved. - -## Fleet state -**Healthy. Nothing left broken.** All four compression knobs reverted and verified; every fault -unwound; no leaked scratch guests, nft rules, ballast files or clock skew; ep0 datastore clean with -zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8. - -## Not tested (with reasons) -Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the -most valuable follow-up**, because F-CRIT-2 raises the same question for restic), fault 12 (host -reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for -documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings. - -**Campaign-caused outage, stated plainly:** fault 11 took demo-hp guest 9201 down for ~9m47s -(the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after. - -## Note on repo conventions -This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an -entry — there is nothing shipped to log. The deliverable is the audit document plus this report. diff --git a/REPORT-campaign9.md b/REPORT-campaign9.md deleted file mode 100644 index 7cee9fb..0000000 --- a/REPORT-campaign9.md +++ /dev/null @@ -1,85 +0,0 @@ -# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28) - -**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign, -and findings are recorded, never fixed inline. Full write-up: -`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`. -Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously). - -Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes. -**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact. - -## The headline — two never-proven restore paths are now proven - -Driven through the **real endpoints the UI posts to**, over https through traefik with a real session -and CSRF token, on live hardware. - -| proof | result | -|---|---| -| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) | -| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) | -| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy | -| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 | -| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy | -| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 | - -The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return -at their last-backup state.** - -## Findings — 3 defects, ranked (none fixed) - -| # | finding | severity | -|---|---|---| -| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** | -| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** | -| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** | - -Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog -sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an -artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed -so live. - -## Mechanisms confirmed working, live - -R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target -breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app -thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of -the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on -demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's -sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the -hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable; -F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests). - -## Where it stopped, and what remains - -Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host -reboot mid-backup) was deliberately not started — a large new fault class against boxes that are -remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown. - -**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's -threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time -`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign -forbids. Reported rather than worked around. The no-code-change alternative (age the controller's -reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended -method next time. - -**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an -intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way -concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`. - -## Recovery - -Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive -observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active), -windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on -either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and -the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots). - -One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box. -Its `until` is long past so it blocks nothing; it clears on the next successful backup or any -controller restart (by design, not persisted). Clearing it would have cost another app outage for no -benefit. - -**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in -`/opt/docker/stacks/` starts a Felhom app **without its secrets** — they are injected by the -controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a -crash loop during recovery. Manual recovery must go through `POST /api/stacks//restart`. diff --git a/REPORT-diag-immich-restore.md b/REPORT-diag-immich-restore.md deleted file mode 100644 index cd8bd62..0000000 --- a/REPORT-diag-immich-restore.md +++ /dev/null @@ -1,71 +0,0 @@ -# REPORT — DIAGNOSE immich offsite restore (felhom.eu side: docs only) - -> Written as `REPORT-diag-immich-restore.md`, **not** the shared `REPORT.md`, per the CLAUDE.md -> convention — `REPORT.md` is overwritten and currently holds the 2026-07-18 website refresh. -> No code shipped in this run; findings only. - -**Date:** 2026-07-19 · **Box:** demo-felhom (felhom-pve guest 9201), controller 0.146.0, immich v3.0.3 - -> **CLOSED IN CODE 2026-07-19 — controller v0.148.0.** Findings 1 and 2 shipped as R-43 (offsite -> reconstitution: safety dump → stop → overwrite files → start → replay the snapshot's dump) and -> R-44 (every run dumps before it captures; manifests carry `offsite_run_id` + `dumps_at`). Deployed -> to demo 9201, healthy. **The §9 live acceptance has NOT run**, so no capability-map flip: the -> offsite row is PARTIAL, the customer-restore row stays MISSING, R-3 stays DRAFT. Implementation -> detail lives in `felhom-controller/REPORT.md`. - -## What ran - -A diagnosis of "immich offsite restore succeeds but photos do not reappear". No product code was -changed: no restore/backup logic, no labels, no flashes, no `restic prune`/`forget`, no snapshot or -escrow changes. - -## Outcome - -The restore did not fail. **It was never invoked on missing data, and could not have worked -if it had been.** - -1. Viktor deleted the 11 photos in the immich UI to test offsite restore. A UI delete means - **trash**, not deletion — no file left the disk. Both „csak a hiányzó fájlok" runs merged - **0 files**, correctly, and flashed success. The test proved nothing. -2. A *valid* test would also have failed: **no offsite path loads a database.** All three buttons - are file-only. Files would return; the library would stay empty. -3. The shipped dump is additionally stale by design — from the 02:30 local run, never refreshed - before a manual push. Probed: **`asset: 0`, `user: 0`, `album: 0`**. - -**Photos:** left in trash at Viktor's instruction (recovery not wanted). All 11 files verified -present on disk and all 11 rows intact, so an ordinary un-trash recovers them until immich's -30-day auto-purge. - -**Answer to "can a customer trust same-day offsite?"** For a DB-indexed app: **no — files come -back, content does not.** The backup half is honest; the restore half cannot reconstitute the app. - -## Decisive evidence - -- `updatedAt` == `deletedAt` on all 11 asset rows ⇒ **no restore operation touched the DB.** -- Unit dump `immich-postgres.sql`, 51 954 452 B, mtime **02:30 CEST** ⇒ `asset: 0 / user: 0 / - album: 0`. The 52 MB is entirely immich's shipped geodata reference tables. It predates the admin - user (07:56:25) and the photos (07:57). -- **Latent hazard:** had a full restore loaded that dump, it would have written an empty DB over the - live one, destroying the trashed rows that were the only surviving recovery path. - -## Files written - -| File | Change | -|---|---| -| `documentation/audits/DIAG-immich-restore-2026-07-19.md` | new — timeline, evidence, source mechanism table, 7 findings | -| `documentation/backlog/ROADMAP.md` | **R-43** (P2-HIGH, offsite restore cannot restore a DB) + **R-44** (P2-HIGH, manual push ships unrefreshed dump) | -| `documentation/architecture/00-capability-map.md` | customer-restore row **stays MISSING**, gains the finding + a method note for R-3; offsite-restore row flagged *scope contested* | - -## Needs Viktor - -1. **Capability-map ruling (line 61).** The offsite-restore row claims PROVEN-LIVE citing - "immich end-to-end from offsite alone" (`CAMPAIGN-6D`). That phrase is contested by this - diagnosis. I flagged it rather than downgrading a proven row — did 6D's accept leg exercise the - DB half, or only the file half? -2. **Optional real red-proof.** Emptying the trash would create genuine data loss and convert the - dump-probe inference into a live proof. Offered, **not run** — irreversible, and the probe - already settles it. -3. **Unreconciled: the 704.6 MiB figure.** Measured 126 MB total on the library storage. If - 704.6 MiB came off a controller Storage page, that gap is its own defect. -4. **Orphaned pre-v3 tree** `dccc13fe…` (~55 MB across upload/thumbs/encoded-video) stranded by the - immich 2→3 redeploy — worth a sweep policy for major redeploys. diff --git a/REPORT-diag-sharing-2026-07-20.md b/REPORT-diag-sharing-2026-07-20.md deleted file mode 100644 index 5e575a3..0000000 --- a/REPORT-diag-sharing-2026-07-20.md +++ /dev/null @@ -1,53 +0,0 @@ -# REPORT — Megosztás diagnosis (2026-07-20) - -Topic-scoped report (parallel-session rule: shared `REPORT.md` untouched). - -**Run:** RUNBOOK "Megosztás diagnosis — SMB unreachable from Mac + sharing-page reload loop". -Read-only diagnosis. **No code changes, no version bumps, no builds, no restarts.** - -**Deliverable:** `documentation/audits/DIAG-sharing-2026-07-20.md`. - -## Verdicts - -* **Reload loop — ROOT-CAUSED (HIGH).** `sharingStatusHandler` (`sharing_handlers.go:246`, added in - `b5d78d1`, controller v0.147.0, 2026-07-19) coerces `idle` → `running` whenever the samba - container is alive. `sharing.html` L320–326 treats `running` as a one-shot job-success edge and - calls `location.reload()` 1.2 s later. The first `tick()` fires synchronously on every page load, - so the page reloads forever. Unconditional for any customer with sharing enabled — the Megosztás - page is currently unusable. Proven live: 6 consecutive `/sharing/status` polls all returned - `{"phase":"running","running":true}`, and the controller log shows **no ensure job ran at all**, - so the phase is manufactured by that line rather than left over from a stuck job. -* **`smb://192.168.0.162` — ROOT-CAUSED.** `.162` is the Proxmox host and never was an SMB endpoint. - smbd runs in guest 9201 and binds `192.168.0.104:445`. `nc` from the host: `.104:445` **open**, - `.162:445` **refused**. Wrong target, stale Finder favourite. -* **`smb://FELHOM` — OPEN, narrowed.** NetBIOS resolution works on the wire - (`nmblookup -B 192.168.0.255 FELHOM` → `192.168.0.104 FELHOM<00>`), but the stack advertises **no - mDNS/Bonjour** (nothing on udp/5353; the R-6 spike selected `smbd + nmbd + wsdd` only) — the - mechanism macOS Finder prefers. Closing this needs one probe from the Mac (`smbutil lookup FELHOM` - / `dns-sd -B _smb._tcp`), listed in the audit's Mac test matrix. -* **H2 (container down/crash-looping) — RULED OUT** at the first probe: `felhom-samba` Up 3 h, clean - logs, smbd/nmbd/wsdd all bound as `infra/samba.go` intends, live `smb.conf` matches the renderer - with no baked address literal, no PVE firewall in the path. - -## Findings (full table + evidence in the audit doc) - -S-1 HIGH reload loop (XS fix: latch a `sawInFlight` flag in the JS; red-proof required) · -S-2 MED the UI never shows the connect address, only `\\FELHOM` — customers guess IPs · -S-3 MED no mDNS advertisement (image slice, needs republish) · -S-4 LOW ensure-job phase never resets — fold into the async-job-feedback roadmap item · -S-5 INFO the guest's LAN IP is DHCP, so any displayed address must be read live, never cached. - -Recommended packaging: S-1 as an immediate patch task (it bricks a shipped page), S-2 alongside it -if the guest IP is reachable from the sharing handler, S-3 as its own slice. - -## Also noticed - -The **remote site's LAN is `192.168.0.0/24` — the same prefix as the DooPlex home LAN** that the -Tailscale subnet router advertises (`192.168.0.180` shows `FAILED` in felhom-pve's neighbour table). -A successful `ping 192.168.0.162` therefore does not by itself prove the Mac is on the remote -segment; the Mac matrix starts by confirming which network it is on. - -## Actions taken - -None. Every command was a read, except a `POST /login` to obtain a session for the status polls. -No secrets are recorded in either document. diff --git a/REPORT-e2d.md b/REPORT-e2d.md deleted file mode 100644 index d207c57..0000000 --- a/REPORT-e2d.md +++ /dev/null @@ -1,85 +0,0 @@ -# REPORT — R-111 fixed, then E-2 proven on a fresh box (2026-07-29) - -Two phases in one session. Full evidence: `documentation/audits/E2D-fresh-vm-2026-07-29.md`. -Root `REPORT.md` untouched. - -## Phase 1 — R-111: the Day-0 channel now serves the current software - -A Phase 0 gate earlier the same day stopped the E-2d run before any VM existed: a fresh box would -have installed **agent 0.96.0 + controller 0.161.0**, ~17 and ~24 releases behind `main`. - -| | Before | Now | -|---|---|---| -| agent (Gitea generic) | 0.96.0 | **0.113.0**, sha `5f3247f7…`, round-trip verified | -| golden (Gitea generic) | 0.161.0 | **0.185.1**, sha `dba00f3e…`, embeds controller 0.185.1 | -| hub `min_agent` | 0.93.0 | **0.113.0** (what controller v0.185.0 declares) | - -Bake clean on every marker: `Result=success`, overlay2, **all three mounts in the archive**, 0 -FATAL/exclusions, HTTP 201, token-leak grep 0. GL-1 teardown: guest 9100 purged, secrets shredded, -drill disk restored to `virgin`. Agent + golden moved in **one** manifest POST so it never vouched a -new agent against an old golden. `min_agent` verified zero-impact first (all three enrolled hosts -already at 0.113.0). Global floor deliberately **not** raised — the golden now bakes 0.185.1. - -Commit `3dff357`. - -## Phase 2 — the E-2d run, full ISO/PAIRING route - -Nested PVE VM on demo-hp, one disk, outside the `felhom` pool. Bind → running controller in -**3 m 35 s**. The install fetched exactly the artifacts published an hour earlier and restored -`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked 20 minutes before. The publish train is -proven end to end on a real install. - -| Claim | Verdict | -|---|---| -| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** | -| **C2** Case B fires naturally | ✅ **PROVEN** — both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort | -| **C3** degraded banner renders **to a customer** | ⚠️ **PARTIAL** — API byte-exact; **no UI consumer exists** → **R-112** | -| **C4** offer appears and moves the target | ⚠️ **PARTIAL** — decline path, `restart_required:true`, no self-restart, E-2a wrapper, healthy-renders-nothing all PROVEN at API level; offer equally invisible → **R-112** | -| **C5** `backup_target_absent` end to end | ❌ **FAILED** — zero events on any channel → **R-113** | - -## The three findings - -**R-112 (P1)** — E-2's banner and offer have **no UI consumer**. The endpoint returns byte-exact copy; -`grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**, and no page handler injects -the state. Decisive contrast: templates fetch **18** distinct `/api/storage/*` endpoints; -`backup-target` and `backup-target/assign` are the only two with zero references. v0.185.1 fixed the -router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired. - -**R-113 (P1)** — the drive-absent gate **cannot fire on device loss**. `planDriveGates` reads presence -from `BoundUnderParent` = "is this path in the guest's mountinfo". The raw mount is a device-bound -systemd unit and dies; **the agent's own bind is not device-bound and outlives the device**, so the -gate sees "present" forever. Live: agent said `enrolled drive absent by UUID` every 20 s for 4½ -minutes, controller logged **0** `[gate]` lines, hub got **zero** events — neither the specific nor the -generic one. Sixth instance of the class, one layer deeper: E-2b wired the seam to a condition that -cannot occur. - -**R-114** — on target-drive loss the message says the backup is *"on the same disk as the system"* -(false) and offers **the drive that just vanished**. Invisible today only because of R-112 — so -**R-114 must be fixed before R-112 is wired.** - -Also filed as a **second instance under R-110** (not a new ID): host-install fetches **nine** files -from `raw/branch/main` and the hub vouches a sha for **one**; E-2a's wrapper is installed 0755 to -`/usr/local/sbin`, root-fenced in sudoers, validated only by `bash -n`. - -## Record - -- `OPEN-ITEMS.md` — **R-112/R-113/R-114 opened**; E-2d re-stated with results and left open for the - residue; E-2's "NOT yet live-proven" list resolved into proven / known-broken; R-94 fully unblocked; - R-110 extended. The drill-cleanup row was opened and then **closed the same session** once the - teardown completed, so it is not carried in the register. -- `ROADMAP.md` — R-112/R-113/R-114 under P1; R-111 marked SHIPPED. -- **`architecture/00-capability-map.md` not touched** — for two reasons: the customer-facing legs are - broken rather than proven, and the map has **no E-2 / backup-target rows at all** (worth noting - against the ROADMAP's coupling rule). - -## Teardown - -VM destroyed, scratch storage removed, **`pvesm status` after == before** (`local-lvm` 38.77 %, -byte-identical), guest 9201 and drill-r50 untouched. **Hub records removed — teardown complete.** The delete was correctly refused at four gates while the host still read ONLINE; once the destroyed host aged to DOWN (`delete-impact` → `deletable:true`) the documented cascade ran and completed: host deleted, PBS tenancy deprovisioned, claim reset, residue purged. Verified after: **0** `e2d` occurrences on the hosts page, fleet unchanged. The one purged `appliance_registrations=1` was this run's own appliance; the unrelated stale 2026-07-25 appliance (`206c8838…`) was not touched by the cascade — the operator removed it separately. - -## One human step, and a premise correction - -The runbook's §5.1a operator STOP (the bind) is **retired** — CC did it. But E-2d's premise that a -fresh install yields a CC-drivable claimable customer is **wrong**: the claim code is bcrypt-hashed and -email-only, and the gate covers everything except `/claim`, `/api/health`, `/static/`. One operator -relay of the emailed code was required — which also proved the claim flow end to end. diff --git a/REPORT-ep0-datastore-volume-2026-07-27.md b/REPORT-ep0-datastore-volume-2026-07-27.md deleted file mode 100644 index 258772f..0000000 --- a/REPORT-ep0-datastore-volume-2026-07-27.md +++ /dev/null @@ -1,92 +0,0 @@ -# REPORT — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27) - -**Class:** supervised operational run (RUNBOOK execution). **No code changed. No version bump.** -Written as `REPORT-.md` per the parallel-session rule — the shared `REPORT.md` was not touched. - -**Full record with all evidence:** `documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md` - ---- - -## Outcome: DONE and verified - -`felhom-offsite` now lives on a dedicated 100 GB Hetzner Cloud Volume instead of ep0's 40 GB root disk. - -| | Before | After | -|---|---|---| -| Path | `/srv/pbs-felhom` (root disk) | **`/mnt/pbs-datastore`** (volume) | -| Datastore total | 37.2 GB | **98 GB** (hub gauge: 97.9 GB) | -| Used | 28.9 % | **13 %** (hub gauge: 12.6 GB, 13 %) | -| Headroom to the 80 % warn | 19 GB | **≈65 GB** | -| Additional customers before warn | ≈2 | **≈7–13** | - -Datastore **name unchanged** — the PBS-DR descriptors, per-box storage ids, ACLs and namespace -layout that R-39/R-82 made self-healing are untouched. - -**Window:** 06:58 → 07:19 UTC (PBS down 07:00 → 07:17). **Nothing was deleted.** - -### Acceptance evidence - -| Gate | Result | -|---|---| -| Copy integrity | 13,242,207,822 = 13,242,207,822 B · **9,748 = 9,748 chunks** · 7 = 7 snapshots · `backup:backup` · itemised dry-run **0 lines** | -| Snapshot counts per ns | `demo-felhom` 2=2, `demo-felhom-01` 3=3, `demo-hp` 2=2 | -| atime semantics | `rw,relatime,discard` — **`relatime` present, `noatime` absent** (GC correctness) | -| Verify job | `TASK OK`, 3/3 groups, forced re-verification of every snapshot, **0 errors** | -| §6 mount guard | **refusal observed** — `Job … failed with result 'dependency'`; mountpoint stayed empty | -| §8 restore round-trip | `source_tier: pbs`, `pass: true`, `mount_parity: ok`, clean teardown, 12m1s | - ---- - -## Three findings the operator should act on - -1. **`scratch` datastore is configured at a path that does not exist** (`/srv/pbs-scratch`). - Pre-existing, not caused here, but now logs `ENOENT` on every PBS start. This is the PRIME RISK - shape ("reports fine, is not there") already live in the config. **Decision needed:** remove the - stanza or create the directory. - -2. **The runbook's §6 acceptance test proves the wrong proposition.** `RequiresMountsFor` is a - mount-first ordering guarantee, not a refusal — systemd silently *re-mounts* an unmounted volume - and PBS then starts safely. The test only bites when the device is genuinely unavailable, which - is how it was re-run and passed. **Amendment recommended in the runbook record.** - -3. **§11 — storage box `u629193` is NOT simply unused.** No live backup path references it (no - datastore, no restic repo, no fstab, no `known_hosts` pin; R-17 already deleted `u629193-sub1`), - **but ep0 carries an enabled, currently-mounted sshfs unit** `mnt-pbs\x2dstoragebox.mount` → - `/mnt/pbs-storagebox`, holding spike leftovers. Disable and remove that unit before deleting the - box, or ep0 logs a failed mount every boot. **The deletion is the operator's console click.** - -## Deviations from the runbook as written - -- **The volume arrived already formatted and mounted** by Hetzner at `/mnt/HC_Volume_106469259` - (§2 assumed neither). Operator ruled: reformat + repath. The 5 % reserve was reclaimed (`-m 0`). -- **§8 ran on demo-felhom, not demo-hp** — DooPlex holds no SSH key for demo-hp (the G1 gap). Same - tier, same relocated datastore, larger archive. -- **The window was contended** by a stale 10-minute restore-test cadence on demo-felhom: the config - had already been reverted to 3.5 days on disk, but the cadence is read once at daemon start and - `NRestarts=0`. Restarting the agent applied it (`cadence=84h0m0s`). The in-flight test was allowed - to finish rather than aborted. - -## Process errors made during this run (recorded deliberately) - -- `rsync -aHAX` **OOM-killed** ep0 (3.7 GB RAM, no swap). Cause: a PBS `.chunks/` tree pre-creates - all 65536 shard dirs → 75,341 inodes, and `-H` retains the whole inode map. `-H` was dropped only - after **proving** no hardlinks exist (`-links +1` → 0; max link count → 1); PBS references chunks - by digest, never by hardlink. -- `/usr/bin/time -v` is not installed on ep0 → exit 127, rsync never ran, and a `| grep … || true` - wrapper swallowed it while printing a success-looking line. -- `rsync --version | head -1` reported a working rsync 3.4.1 as "missing" — **the §12 pipe-into-head - trap, fourth recorded instance in this project.** Both fixed by capturing the command's own `$?`. - -## Deferred - -1. **Old copy retained** at `/srv/pbs-felhom` (13 GB, 9,748 chunks) as the rollback. Rollback is a - two-line `datastore.cfg` revert. Reclaim only after a new weekly offsite backup lands on the - volume, with explicit go-ahead. -2. **GC not run** — now unblocked by the round-trip, but left for a separate deliberate run. No GC - schedule is configured on this PBS at all. -3. ~~Hub PBS-DR capacity gauge not re-read.~~ **CLOSED — verified correct.** The hub operator UI - (Offsite → PBS DR) reports `felhom-offsite (ep0)` at **97.9 GB capacity, 12.6 GB used, 13 % full**, - agreeing with the on-box `df`. The gauge follows the datastore's configured path, so the move - needed no hub-side change and the suspected "wrong filesystem" bug does not exist. -4. **ep0 has no swap** (temporary 4 GB file removed; box left as found). Worth a small permanent - swapfile — outside this runbook's scope. diff --git a/REPORT-fcrit1-fa1.md b/REPORT-fcrit1-fa1.md deleted file mode 100644 index 3e37816..0000000 --- a/REPORT-fcrit1-fa1.md +++ /dev/null @@ -1,63 +0,0 @@ -# REPORT — F-CRIT-1 + F-A1 fixed (controller v0.179.0, 2026-07-28) - -Docs here. Implementation, all six red-proofs and the full live replay live in -`felhom-controller/REPORT.md`. The campaign that found both: -`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`. - -## What changed -**F-CRIT-1** — an app that failed to restart after a quiesce never alarmed, for two independent -reasons, either of which alone kept it dead: `restartAll` returned nothing (the failure was logged -and dropped), and `classifyRunStates` whitelisted `StateStopped` on invariant I1 ("the user stopped -it") — which the quiesce loop had made false by stopping stacks the same `docker compose down` way. -A failed restart and a user stop are the *same* Docker state; the only difference is that the loop -tried and could not, now surfaced by `Loop.FailedRestarts()`. - -**F-A1** — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a -failure. It is now contention: the tier stays DUE, is dropped before anything stops, and unending -contention raises its own **BLOCKED** signal rather than going silent. - -## Bounds, justified against measured reality -- `contentionRetryAfter` **15m** — longest restore-test observed on the fleet is 12m01s; the agent's - local restore-test wait is 10m. Caps app-stop churn at 4/hour instead of 12/hour. -- `contentionAlarmAfter` **3h** — the agent's own PBS restore-test task is capped at 120 minutes, so - contention outliving that is a *stuck* gate, not a busy one. 3h adds margin and is 15× the longest - contention actually observed. - -## Verified live, with the hub DB as arbiter — not from logs -Same box, same day, same event type; the only difference is 409 versus a genuine error: - -| injection | operator emails (demo-hp) | -|---|---| -| **409 contention** | 8 → **8** (none) | -| **real transport failure** | 8 → **9** | - -And for F-CRIT-1: the failed restart alarmed **9 seconds** after grace expiry with the dashboard -banner naming the `(stopped)` state, while a **deliberate** user stop on the same box stayed silent -through **9** dead-app scans (the positive observable that the silence is suppression, not a dead -detector). - -## The rule this arc earned -Added to **both** copies of `CLAUDE.md` (live + `documentation/runbooks/workspace-CLAUDE.md`): -**a comment asserting an invariant needs a test pinning it, or it is a wish.** Six instances in this -project have shipped guarantees the code did not provide — `EffectiveProtected`, `newestArchiveOn`, -the R-97a operator-only claim, `classifyRunStates`' I1, `inflight.go`'s defer claim, and -`quiesce.go`'s spurious-failure claim. Two were found only on live hardware, and one of those had a -green, red-proofed test suite over a production path broken two independent ways. - -Corollary recorded with it: prefer a test that asserts the **consequence** (does the alarm fire?) -over one that asserts the **mechanism** (does suppression expire?). R-97b's Scenario F proved the -mechanism; the consequence was still broken. - -## Docs touched -- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-1 and F-A1 → SHIPPED + PROVEN-LIVE. -- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — both rows struck through, closing - section added. **All three of the campaign's alarm findings are now closed** (F-CRIT-1, F-CRIT-2, - F-A1). -- `documentation/runbooks/workspace-CLAUDE.md` — the invariant rule. - -## Still open, highest first -**Fault 4** (restic transport interruption) — four injection approaches were defeated by -guest-bridged networking, and it is now the most valuable follow-up: F-CRIT-2 answered the phantom -question for PBS and left the identical question open for restic. Then **R-99** (prune never removes -phantoms) and **F-LEAK** (a failed restore-test cannot destroy its own scratch guest — observed -again during this work). diff --git a/REPORT-fcrit2.md b/REPORT-fcrit2.md deleted file mode 100644 index 469a303..0000000 --- a/REPORT-fcrit2.md +++ /dev/null @@ -1,40 +0,0 @@ -# REPORT — F-CRIT-2 fixed: a failed backup no longer looks like a fresh one (2026-07-28) - -Scope: `felhom-agent` v0.105.0 → **v0.106.0**. Docs here. Implementation detail and the full live -re-test live in `felhom-agent/REPORT.md`; the campaign that found it is -`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`. - -## What changed -`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, and NEWEST) as a -successful backup, so the tier read fresh, went **not due**, and was never retried — seven days of -silence on the real 168h cadence, invisible to both the R-88 breaker (defers only *due* tiers) and -the hub deadline monitor (reads the same freshness). It now counts only *plausibly complete* -entries via a measured 1 MiB floor; undecidable ⇒ not counted. - -**Size is the only tier-agnostic discriminator.** `verification` and `encrypted` are absent on every -local (dir) archive AND on a good PBS snapshot until `verify-new` catches up — gating on either -would have rejected 100% of local backups and produced fleet-wide backup thrash. That inverse risk -is a first-class test, red-proofed by making the filter reject everything. - -## Verified live, not just in unit tests -Campaign fault 2 was replayed against the fixed agent on demo-hp — phantom created, rejected and -announced once; the tier correctly reported DUE and backed up (4,359,968,099 B landed); and the -inverse showed **no thrash**, with 91 scheduler ticks as the positive observable that the loop was -alive rather than dead. - -## Settled along the way — no retention bug -Server-side prune does **not** count phantoms toward `keep-last`: a dry-run against three real -snapshots plus a phantom retained two real ones plus the phantom. The feared "two phantoms ⇒ zero -real backups" does not occur. Prune never removes them either, so they accumulate one per aborted -upload — filed as **R-99** (LOW, hygiene), not as a retention bug. - -## Docs touched -- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-2 → SHIPPED+PROVEN-LIVE; **R-99** filed; - **F-CRIT-1** filed as READY-HIGHEST (Campaign 8's other HIGH finding, untouched here). -- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — F-CRIT-2 row struck through and - a closing section added. - -## Still open, highest first -**F-CRIT-1** — an app that fails to restart after a quiesce never alarms, for two independent -reasons. Then fault 4 (restic transport interruption), which this fix makes more pointed: the -phantom question is now answered for PBS and still open for restic. diff --git a/REPORT-freboot-fleak-fobs.md b/REPORT-freboot-fleak-fobs.md deleted file mode 100644 index 8d593b0..0000000 --- a/REPORT-freboot-fleak-fobs.md +++ /dev/null @@ -1,140 +0,0 @@ -# REPORT — F-REBOOT + F-LEAK + F-OBS, and two investigations (2026-07-28) - -Scope in this repo: **`scripts/felhom-host-install.sh` v1.20.0 → v1.21.0** (which is where F-LEAK's -*actual* fix lives), plus the Campaign 8 audit doc and `OPEN-ITEMS.md`. Written as -`REPORT-freboot-fleak-fobs.md` so the shared `REPORT.md` is not clobbered. - -Code companions: `felhom-agent` v0.106.0 → **v0.110.0**, `felhom-controller` v0.179.0 → **v0.180.0**. - -**Correction to this repo's part of the story:** v1.21.0's band-scoped ACL is *not* the final F-LEAK -fix. It works, but only **once per slot** — PVE's destroy path calls -`AccessControl::remove_vm_access($vmid)` (`API2/LXC.pm:906`), which deletes every ACL at `/vms/` -(`AccessControl.pm:1898`), so **the grant is consumed by the operation it authorises**. Found by counting -ACL rows after the first successful teardown (`/vms/990000` → 0 grants), not by reasoning about it. The -durable fix is agent **v0.110.0**'s band-scoped fenced destroy; v1.21.0 remains valuable because it makes -the common case need no privileged call, and it is now the *first* of two layers rather than the only one. - -## Baselines (reconfirmed, not copied) -`felhom.eu d0cec9d`, `felhom-agent af1c21a`, `felhom-controller fb91c8d`, all clean. Agent `0.106.0` -and controller `0.179.0` live on both demo boxes. - ---- - -## host-install v1.21.0 — F-LEAK, and why the fix landed *here* rather than in the agent - -**The finding.** A restore-test whose restore **fails** leaves a scratch guest the agent cannot destroy -(`403 missing privilege VM.Allocate`), so a half-restored guest holds its disks until a human removes -it and the 10-slot scratch band shrinks silently. - -**The cause is structural, not a missing privilege in the role.** `FelhomAgentGuest` is granted at -`/pool/felhom`, and **a guest joins that pool only when its restore completes**. A failed restore -therefore produces a guest that exists, is in no pool, and is out of the token's reach entirely. - -**The first fix was wrong, and its own live replay is what proved it.** Agent v0.107.0 shipped a -teardown fallback that adopted the stranded guest into the pool and retried — reasoning from -`Pool.Allocate` on `/pool/felhom`. It fired exactly as designed and PVE refused it: - -``` -ERROR restore-test: pool adoption failed; left for Recover vmid=990000 - err="proxmox: PUT /pools/felhom -> HTTP 500: permission denied at /vms/990000 (missing privilege ...)" -``` - -`PUT /pools/{pool}` **also** requires `VM.Allocate` on the VM being added. **Pool membership cannot -bootstrap its own authority.** Removed in agent v0.108.0 rather than left in place — a path that -provably cannot work is worse than none, because it reads as a fix. - -**What shipped instead.** `apply_scoped_acl` now grants `FelhomAgentGuest` at each `/vms/` in -`PVE_SCRATCH_VMID_MIN..PVE_SCRATCH_VMID_MAX` (990000–990009 — the band the restore-test already picks -from), to **both** the user and the token, because the privsep-intersection rule applies here as -everywhere. - -Two supporting changes, both load-bearing rather than tidy-up: -- **`remove_scoped_acl` deletes the band grants before the role delete.** PVE refuses to delete a role - still referenced by any ACL, so omitting this would have broken the uninstall — a failure that would - only surface on a decommission. -- **`step_verify` asserts the band grants.** A missing one is otherwise invisible until a restore-test - *fails*, which is precisely the case that leaked a guest in the first place. - -### Why the grant is still not a widening — proven live, at the seam the defect lives in - -A real PBS restore to `990000` **without `--pool`** reproduced the exact stranded state -(`990000 stopped`; `felhom pool members: [9201]`; `990000 in pool: False`). Then, with the agent's own -token, same guest, minutes apart: - -| | `DELETE /nodes//lxc/990000` | -|---|---| -| **grant removed** (the original defect) | `403 Permission check failed (/vms/990000, VM.Allocate)` — guest still present | -| **grant restored** (the fix) | `200 UPID:...:vzdestroy:990000:felhom-agent@pve!agent` — guest gone | - -And it still cannot reach anything else: - -| target | result | -|---|---| -| `/vms/990010` (one past the band) | **403** `Permission check failed (/vms/990010, VM.Allocate)` | -| `/vms/100` | **403** same | - -`990010` does not exist and PVE **still** answered 403 rather than "does not exist" — so PVE evaluates -**permission before existence**, which makes these genuine authorization refusals rather than artifacts -of a missing guest. Granting at `/vms` was considered and rejected: it would authorise destroying every -guest on the box, including a co-tenant's. - -Applied on **both** demo boxes (demo-hp and demo-felhom) so the live fleet matches the installer. - -**A careless step of mine, recorded rather than buried.** The probe loop also issued a live `DELETE` -against running guest **9201**. It was refused with `500 container is running` — but the *permission -check passed* (9201 is a pool member by design), so had the guest been stopped I would have destroyed -the live demo guest. The scratch-band probes were the safe ones; 9201 had no business in that list. - ---- - -## Documentation changes - -- **`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`** — F-REBOOT, F-LEAK and F-OBS - written up as FIXED with their live evidence, including F-LEAK's refuted first attempt (recorded - precisely *because* it looked right), plus a new **§6b** for the follow-up investigation. -- **`documentation/backlog/OPEN-ITEMS.md`** — three findings closed, **R-100** filed. - ---- - -## R-100 — the investigation's finding, deliberately NOT fixed - -**A restic offsite tier that fails every night never goes stale on the hub.** This is **F-CRIT-2's -defect class one layer up and on the other tier** — a *failed* run resetting the freshness clock — and -it was found by asking whether F-CRIT-2's shape existed anywhere else. - -Both halves verified in the source, not inferred: -- **Controller:** `o.LastRun = time.Now()` is set **unconditionally** at - `controller/internal/backup/offbox.go:716`, *outside* the `runErr` branch. The failure is recorded - faithfully, but into a different field — `o.LastStatus = "error"` at `:725`. -- **Hub:** `isStale()` reads **only** `off.LastRun` (`hub/internal/monitor/offsite.go:120`, `:127`, - `:131`) and never consults `LastStatus`. - -So a nightly restic run that fails every night keeps `LastRun` fresh, `isStale` is permanently false, -and the staleness alarm never fires — with no successful offsite backup having occurred at all. - -**Scope of the silence, stated precisely rather than dramatically.** `LastStatus` *does* reach the hub — -it is parsed into the report struct and **only logged** (`offsite.go:270`); it drives no checker and no -notification. The controller's own guest UI surfaces `LastStatus="error"`, so the failure is visible to -someone who looks. What is missing is the **push**: the operator's fleet-wide alarm plane is silent, -which is the plane that matters for an unattended appliance. - -Not fixed, per this task's investigation-only scope. Fix direction: gate staleness on the last -*successful* run rather than the last attempt — exactly what F-CRIT-2's `NewestArchiveTime` fix did for -the PBS tier. - ---- - -## Verification - -`bash -n scripts/felhom-host-install.sh` clean. The ACL behaviour was verified **live on real hardware** -rather than by dry-run, since the whole finding is about what PVE's authorizer actually does — and the -live run is what refuted my first design. - -## Fleet state -Agent **0.110.0** (with the updated sudoers) and controller **0.180.0** on both demo boxes, all healthy. -Scratch-band ACLs at 20 rows on both — re-applied after the attempt-2 destroy consumed one. No leftover scratch guests. demo-hp's `restore_test_cadence_seconds` reverted **600 → 302400** -(a bounded change made for the replay). - -`felhom.eu`: this repo has a **foreign uncommitted WIP file** (`documentation/PROMPT-TEMPLATE.md`) from -another session in the shared worktree. Left untouched; my commits staged explicit paths only, per the -never-`git add -A` rule. diff --git a/REPORT-iso-branding-website-grid-2026-07-19.md b/REPORT-iso-branding-website-grid-2026-07-19.md deleted file mode 100644 index 10a2852..0000000 --- a/REPORT-iso-branding-website-grid-2026-07-19.md +++ /dev/null @@ -1,119 +0,0 @@ -# REPORT — ISO boot branding + single-entry GRUB menu (R-38) · website grid restored (2026-07-19) - -> `REPORT-.md` per this repo's parallel-session rule: another session was writing in this -> clone tonight (CAMPAIGN 7 / `DIAG-immich-restore-2026-07-19.md`), so the shared `REPORT.md` is left -> untouched. - -Parts 1 and 2 of the polish train. Parts 3 and 4 landed in `felhom-agent` and `felhom-controller`; -see their own `REPORT.md`. - ---- - -## Part 1 — scripts v1.22.0: GRUB branding + single-entry safety (R-38) - -### What shipped - -Every ISO is now **repacked** after `prepare-iso`. `mkimage-surgery.sh` → **`iso-repack.sh`**: -branding and the slice-B loader swap need the same extract → modify → re-master cycle, so they share -one pass instead of re-mastering twice. **The mkimage recipe is untouched.** - -**The safety half — the one that matters.** The stock PVE menu offers *Graphical*, *Terminal UI*, a -serial variant, and an **Advanced Options** submenu holding two `nomodeset` entries, three debug -entries, *Rescue Boot*, memtest and *UEFI Firmware Settings*. Every one of those reaches the -**manual** installer, whose first question is which disk to wipe. They are **not emitted** — not -hidden, not password-gated. What ships is one entry, „Felhom telepítés", default, 5 s. - -**Boot behavior is unchanged.** The `linux`/`initrd` lines are lifted **verbatim at repack time** -from the ISO's own *Install Proxmox VE (Automated)* entry rather than frozen into a copy in this -repo, so a PVE bump that moves the kernel path or edits the append line tracks automatically. The -build **fails** if they cannot be found, if the append line has lost `proxmox-start-auto-installer`, -or if `auto-installer-mode.toml` is absent — that last one because without it the single -Felhom-labelled entry would boot a *manual* installer, i.e. exactly what this change prevents. - -**Gates, then a re-check against the shipped artifact.** The rendered menu is asserted to have -exactly 1 `menuentry`, 0 `submenu`s and no *live* reference to -`proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` (comments are stripped first — -the template's header names the dropped entries deliberately). Then the menu and theme background -are read back **out of `final.iso`**, not out of the extract tree. - -**The boot card.** `grub/generate-grub-background.sh` letterboxes `website/assets/og-image_2.png` -onto a 1024×768 gfxterm canvas at repack time (ImageMagick added to the assistant image), so the boot -screen has **one source** and not a second pre-rendered PNG to drift. The card's own subtle grid -(measured: 4px lines of `#0D131A` on `#0D1117`, pitch 131px) is continued across the letterbox fill -**phase-locked** to where the card's grid lands, so the fill is seamless instead of a 500px square of -grid floating in flat navy. The generator refuses a source whose geometry no longer matches the -measured constants — a swapped asset would misplace every line, and that only shows up on a boot -screen nobody re-checks. Menu positioning needs a gfxmenu theme (plain `background_image` cannot move -the menu off the wordmark), so `grub/felhom-theme.txt` puts it in the lower third the layout leaves -empty, optically centered (measured off a canary screenshot; the comment records the measurement). - -### Live validation — nested canary, UEFI/OVMF, PVE 9.2-1 - -Booted the built canary ISO under QEMU with OVMF and captured the framebuffer. - -| Check | Result | -|---|---| -| GRUB renders the Felhom card | **PASS** — background + grid visible at 1024×768 | -| Exactly one entry, selected | **PASS** — „Felhom telepítés" only | -| Hungarian accents under gfxterm | **PASS** — „telepítés", „Indítás … másodperc múlva" render correctly | -| Countdown visible and counting | **PASS** — 5 → 0 | -| Auto-fires at 0 | **PASS** — serial shows ``Booting `Felhom telepítés'`` | -| Unattended install proceeds | **PASS** — „Fetching answers for automatic installation" → auto installer | -| Same abort as v1.21.0 | **PASS** — `ERROR: Installation failed: filter did not match any device` → `Installation aborted`; no disk touched | - -A first build **correctly failed closed**: the banned-entry gate matched the template's own -explanatory header. Fixed to strip comments before matching (a comment naming a removed entry is the -point; a directive using one is the bug), which is a gate behaving as designed. - -### Artifacts (rebuilt on 180, `/mnt/5_hdd/felhom.eu/felhom-iso/out/`) - -| ISO | sha256 | bytes | -|---|---|---| -| `felhom-pve-9.2-1-v1.22.0-n100-generic-mkimage.iso` (safety) | `ff6f06ba1dbfe10f27d703afc29516001000349147426b43c9a424a0ea28bdbf` | 1 704 482 816 | -| `felhom-pve-9.2-1-v1.22.0-n100-demo-generic-mkimage.iso` (real) | `494db0ddf859b6b152cad4d0e0d9e9cefd27255cde07e2b41aba3ac12a217888` | 1 704 482 816 | -| `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso` (validation) | `83c61c0413c84e27b26a37bb5dfaed2fcd44fd25e3e571c7310142bd305f2f9d` | 1 705 338 880 | - -Both shipping ISOs: `embedding 60 modules`, `El Torito boot images=2`, fs-uuid preserved, and the -post-re-master verification confirming 1 entry + theme background inside the finished image. - -**Deliberately not done** (per the task): no squashfs/initrd rebranding — post-GRUB screens are still -Proxmox-branded; no disk-setup or answer-generation change; the pairing banner is untouched. - ---- - -## Part 2 — website: the index grid background, restored - -**Archaeology.** Not a deliberate removal. The grid lived as a fixed `body::before` in -`index.html`'s inline `