From c16697c21afe514ccaeaff7ad349ff26c6aed0c0 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 9 Nov 2026 10:19:36 +0100 Subject: [PATCH] demo-hp local tier refused again: space freed; decision 285 + 56 amended (old infra/rc images); R-1366, R-1367 filed; R-548 note Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- REPORT.md | 62 +++++++++++++------ STATUS.md | 2 +- .../architecture/09-update-architecture.md | 15 +++++ documentation/backlog/OPEN-ITEMS.md | 8 ++- documentation/backlog/STATUS-HISTORY.md | 4 ++ 5 files changed, 68 insertions(+), 23 deletions(-) diff --git a/REPORT.md b/REPORT.md index 2c35e791..3bd29a86 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,24 +1,48 @@ -# REPORT — morning check, and hub 0.179.1 (2026-11-08, attended) +# REPORT — demo-hp's full local tier, and what the whole-box copy holds (2026-11-09, attended) -**Plain words first.** Last night's fixes worked. The N100 installed its Proxmox updates (no refusal) and a new kernel, -and restarted itself. demo-hp made its first local whole-box copy in 10 days. The hub still said the N100 „needs a -restart" after it had restarted; hub 0.179.1 fixes that. demo-hp does need a restart; the operator chose to wait for its -kernel step. +**Plain words first.** demo-hp did not restart on 11-08/09. Its local whole-box copy was refused for space again +(12.5 GiB free, 13.3 GiB needed), so the night's OS run and its kernel step waited. With the operator's yes, CC freed +space and filed the causes. The controller now deletes two kinds of old images by itself (v0.351.0, in code, not +released). -**Box facts (SSH).** demo-felhom: pve step `DONE upgraded=33` (05:32), kernel 7.0.14-28 one-shot boot 05:35, `KERNEL GOOD` -(default now 7.0.14-28), 0 host processes on replaced libraries, guest 9201 running. demo-hp: local backup `completed` -02:43 (`local:backup/vzdump-lxc-9201-2026_11_08-02_30_38`), pve step `upgraded=35`, no kernel step yet (7.0.14-28 pending), -systemd/sshd/kvm/zed/rrdcached on replaced libraries. +## What the whole-box copy holds (demo-hp guest 9201, `docker system df -v`, measured) -**Hub 0.179.1** (`osupdates.layerLine`): the host's last boot (crash guard `last_boot_at`) cuts the host layer's -reboot-needed run, less one minute of clock skew; a claim after the boot starts a new run. Feeds the System page and -the 14-day alarm. Test `TestRebootNeeded_ClearedByAHostBootAfterTheClaim`, red on two mutants; `go test ./...` green. -Deployed: image built from `ddbcabeb`, manifest `6b91a215`, ArgoCD sync of `Deployment/hub` only, rollout ok, pod -logs `felhom-hub 0.179.1 starting`. **Live (endpoint method — `GET /system` with CC's token, the page the operator -opens):** only demo-hp is listed. Evidence: `documentation/audits/hub-reboot-flag-2026-11-08/`. +| Part | Size | +|---|---| +| Images used by containers | 13.2 GB | +| The apps' undo images (decision 53, on purpose) | 9.8 GB | +| Old controller images (25 of them test builds) | 2.2 GB | +| Old infrastructure and helper images | 1.35 GB | +| Docker volumes (app data) | ~1.5 GB | +| `sys_drive` | 1.3 GB | -**Remote restart buttons (the operator's question).** The host page has „Restart controller" and „Reboot the guest" -per guest (agent actions, D65). There is no button that reboots the Proxmox host; the host reboots only by the kernel -step. Not a defect — the agent's action list is closed by design. +Packed: 9.8 GB, on a 39 GB system disk. The N100 writes its copy to its 1 TB backup drive (`felhom-backup`), 1.5 GB. +Tester 1 and demo-hp write to the system disk (`local`). The same old-image pattern was on all three boxes (43–48% of +image space unused; most of it is undo images). -**Registers.** No row opened or closed (the hub fault fixed in the session, size rule). System poster: no fact changed. +## What was done on demo-hp (operator's yes) + +- Inside guest 9201: 36 unused images deleted by name (25 old controller test builds; traefik v3.6.7/v3.7.13; + cloudflared 2026.6.0/2026.9.3; Samba 1.1.0; FileBrowser 1.3.3; alpine:3.20, curl, hello-world). The undo images, + the previous controller (0.349.0) and the previous pins were kept. Images 27.56 → 24.15 GB. +- On the host: installer ISO 1.30.0 removed from `local` (`pvesm free`); its sha256 `590d7b9b…` matched the copy on + DooPlex and the file on iso.felhom.eu (HTTP 200). The old golden archive 0.236.0 stays (no other copy found). +- `local`: 12.5 → 14.0 GiB free, above the 13.3 GiB the space check asks. +- Evidence off-git: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/` (before, cleanup, journal noise). + +**Not yet observed:** tonight's local copy, OS run and kernel step on demo-hp. Positive observable: a new +`vzdump-lxc-9201-2026_11_10-*.tar.zst` in `/var/lib/vz/dump/`, then the host's boot time after the kernel step. + +## Product changes + +- Controller v0.351.0 (felhom-controller CHANGELOG): decision 56 amended (old `-rc` controller images go) and decision + 285 (pinned infrastructure images keep the pin and the previous pin). Six mutants red-proved. +- `09-update-architecture.md` §3: decision 285 added, decision 56 amended. + +## Register + +Rows before 223, after 225; opened 2 (R-1366 the whole-box copy carries every image — operator decision; R-1367 the +host journal's sudo session noise), closed 0. R-548 extended with the 11-09 refusal and the clean-up. + +Architecture read: `07-backup-architecture.md` (tiers), `09-update-architecture.md` §3 (decisions 53, 56). +System poster: no fact changed (no backup tier, time or retention moved). diff --git a/STATUS.md b/STATUS.md index a30c0765..6d8506d6 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-11-08.** The night went right: the N100 installed its Proxmox updates and a new kernel and restarted itself; demo-hp made its first local whole-box copy in 10 days. Hub 0.179.1 no longer says „needs a restart" for a box that has restarted; demo-hp still needs one (its kernel step, likely tonight). +**Updated 2026-11-09.** demo-hp did not restart, because its local whole-box copy was refused for space again and the night update waited; CC freed space (old images, an old installer file), so tonight's copy fits. Controller 0.351.0 (in code, not released) deletes old test builds and old system images by itself. Two new rows: the whole-box copy is mostly app programs (your decision, R-1366), and the host log noise (R-1367). ## What works - **The Plex attack did not reach Felhom's secrets** (R-1241). diff --git a/documentation/architecture/09-update-architecture.md b/documentation/architecture/09-update-architecture.md index a4102460..baa518b8 100644 --- a/documentation/architecture/09-update-architecture.md +++ b/documentation/architecture/09-update-architecture.md @@ -716,6 +716,11 @@ R-636's louder repeated alarm. image back (`:289`). After a good swap that image is "the one before" the running one — so „the one before it" here and R-745's „rolls back to the RUNNING image" name the same image, seen before and after the swap. The controller never hands the agent an older roll-back target. + *Amended 2026-11-09 (operator, „go with A" — decision 285's session):* a test build (`X.Y.Z-rc.N`) is a version below + its release, so an old one is deleted like an old release; one above the running version or between the previous + and the running one stays; with no swap record the previous is the highest RELEASE below the running one (a test + build only when no release is there). Measured cause: 25 old test builds (2.2 GB) on demo-hp's guest, all left + alone because every `-rc` tag was. Controller v0.351.0, `TestControllerRetention_OldTestBuildsGo`. ### 2026-10-01 — decided by CC unattended, operator may reverse @@ -1225,6 +1230,16 @@ Wanderer's Meilisearch went 256 → 384 MB and was measured again (it sizes itse „a person — on the box, or the operator on the hub". Built as the controller's `clear_hold` operator action (`04` §6.1), cleared by the running controller (R-1343). *Operator, 2026-10-29 12:10.* +### 2026-11-09 (attended) — old infrastructure images are deleted like old controller images + +285. **A box keeps, per pinned infrastructure image (traefik, cloudflared, FileBrowser, Samba), the pin, every newer + version, the previous pin and every image a container uses; older and untagged images of that repository are + deleted** by exact name/ID, never forced or pruned, not while an app pulls (R-863) or the controller swaps itself — + decision 56's shape. *Operator, 2026-11-09 („go with A": the product fixes for demo-hp's full local backup tier, + R-548).* **Why:** nothing deleted them; each pin is 65–195 MB and moves every few weeks on every customer box, and every + image rides the whole-guest copy (mp0 holds Docker's data). **Cost:** a hand roll-back two pins back pulls the image + again. Controller v0.351.0, `internal/stacks/infra_image_retention.go`, `TestInfraRetention_*`. + ### 2026-10-28 (day, attended) — RomM kept; the runtime cache fills itself in CI 277. **D61 = keep RomM offered** (R-1330) while no RomM release carries Node ≥ 24.22.1; the first one that does is taken diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 0ed6bea2..07686e2c 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -168,7 +168,7 @@ stopping line that lies. | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC | | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:** **OPEN — rank P3-LOW; owner: CC (after R-784)** **2026-11-02:** v1.10.0 and v1.11.0 read clean of HTTP2Shell (Node 24.22.1); v1.9.0 is still bad. **The big step v0.17.3 → v1.11.0 is proven on the drill:** bench, 9202 (peak 29.4 %), the undo (`audits/security-2026-11-02/bench/sparkyfitness/`, `walks/sparkyfitness-drill/`). Not written: R-784 (the licence) stands before it. | — | — | CC | -## Backup & restore — 46 rows (P2 4, P3 24, P4 18) +## Backup & restore — 47 rows (P2 4, P3 25, P4 18) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -181,10 +181,11 @@ stopping line that lies. | **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC | | **R-433** | Backup & restore | P3 | **A sub-account cannot reach ANY Storage Box snapshot, by any name — so clause (b) of the 2026-09-01 R-95 re-scope ("the rest is recoverable file by file") is NOT SUPPORTED.** MEASURED 2026-09-01 on `demo-hp` over the credential the box already holds, read-only, no delete verb issued. **The sweep:** a batched `stat -c %n` over Hetzner's port-23 restricted shell (500–600 paths per round trip, stdout carrying only paths that exist) tried **777,600** names of the vendor form `YYYY-MM-DDTHH-MM-SS` across nine full days at second granularity, and 126 alternative shapes — **zero resolved.** **The control is what makes the zero mean anything:** the identical 600-name batch with one real path appended returned it in 6 of 6 batches. **The structural cause:** `/home` (the customer data, `u629488-sub3`) is **st_dev 0,82**; `/.zfs/snapshot` is **st_dev 0,276**; `/home/.zfs` does not exist. A snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding `felhom-repo`. **What still stands:** clause (a) — the box can delete its live repository but cannot WRITE into `/.zfs/snapshot` — is unchanged and re-confirmed. **What is now open again:** the only routes to the older copy are a panel rollback of the WHOLE Storage Box (deletes newer snapshots, hits every customer on it) and the provider API (fenced by §11-D, and `hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method at all**, so it needs new code regardless). **NOT ESTABLISHED, and it is the question that decides whether per-file recovery exists for anyone:** whether the MAIN account can see the snapshots. No main-account credential exists in this project. **The ranking is Viktor's; I am not re-ranking R-95 — but the argument that moved it down is the argument this drill removed.** `audits/evidence-drill-r95-recovery-2026-09-01/` **BLOCKED-ON-PROVIDER 2026-09-01.** The one question that can move this is drafted and ready to send: **Question 1** of `documentation/runbooks/provider-questions-2026-09-01.md` — *can the MAIN account retrieve individual files from a snapshot, without a whole-box restore?* **Neither answer leaves this row where it is:** "yes" makes per-file recovery real but permanently operator-only, and closes this; "no, full restore only" means the snapshots do not bound a single customer's exposure at all, because using them costs every other customer on the box their newer snapshots — **and R-95 becomes urgent.** Nothing here can progress without it, and it is not CC's to send (§11-D). Tracked by a dated check in the DUE-CHECKS block. **⚠ A TRAP ON THE WAY TO THIS ANSWER, found by the operator 2026-09-01 and armed against in the questions file: HETZNER HAS TWO SIMILARLY-NAMED PRODUCTS WHOSE DOCS SAY OPPOSITE THINGS.** **Storage SHARE** (a managed Nextcloud, NOT us) documents *"Currently, we only support restores for the full backup ZFS snapshot to a specific point in time"* (`docs.hetzner.com/storage/storage-share/faq/backup-snapshot/`). **Storage BOX** (ours) documents the opposite — *"You can download individual files or entire directories as usual"* (`docs.hetzner.com/storage/storage-box/snapshots/`). **A web search for the obvious phrasing surfaces the SHARE page, and it reads like a definitive NO.** Tell them apart by the giveaways: the Share page talks about *Nextcloud's data cache*, a *database dump* and the *konsoleH* interface, and never mentions Storage Box. **Why this is filed and not left as trivia: a support agent could answer Question 1 from the wrong page, and that answer would push this row to the top of the register for no reason.** Question 1 now names the product, quotes the Storage Box line, and says explicitly that we know what the Share FAQ says. **If an answer cites full-snapshot-only, check WHICH PRODUCT it is about before acting on it.** **Nothing in this repository ever leaned on the Share claim** — verified by grep at the time; the only vendor line we cite is the Storage Box one. **RE-DATED 2026-09-15 → 2026-09-22 (due-checks gate):** the operator mailbox read through the Gmail connector (`(from:hetzner OR subject:hetzner OR "Storage Box") after:2026/09/01`) holds no Hetzner reply — one match, our own `offsite_snapshots_dropped` alarm. Whether the two tickets were ever sent is not visible to CC; if they were not, sending them is the operator's act. **-- ANSWERED. Hetzner replied on ticket #2026090103040671; the operator supplied the thread on 2026-09-22 after this session searched for it and WRONGLY reported no answer (see the instrument note below and R-628).** **Q1 - can the MAIN account read individual files and directories out of a specific snapshot, without restoring the whole box?** Hetzner: *"With the main account it should be possible to download files and directories from the snapshot as it is stated in the documentation"*, citing `docs.hetzner.com/storage/storage-box/snapshots#access-to-snapshots`; and separately *"A restore of a snapshot will revert the Storage Box completely to the state it was in when the snapshot was taken."* **So the Storage BOX documentation governs, not the Storage Share FAQ** - which is exactly the trap `provider-questions-2026-09-01.md` warned the reader about, and the answer came back on the right side of it. **File-level snapshot access is a MAIN-ACCOUNT capability; the sub-account cannot do it**, which matches R-433's own measured finding (777,600 names swept from a sub-account, zero hits). **HEDGE, kept because it is in the reply: "should be possible" is not "is", and nobody has yet read a file out of a snapshot from the main account on `u629488`. That is now the measurement this row needs, and it is cheap.** **Q2 - is `--append-only` enforced by Hetzner, or taken from what the client sends?** Hetzner: *"You can use --append-only and it would look like this: `command="rclone serve restic --stdio --append-only path/to/repo" `"*, with `fluix.one/blog/hetzner-restic-append-only/`. **So it is enforced by US, by pinning a FORCED COMMAND to the SSH key in the Storage Box's `authorized_keys`** - not by Hetzner globally, and not by anything the client asks for. A key without that prefix still gets a deleting server. **This is the answer R-95 has been blocked on** and it says append-only IS expressible on a Storage Box, which R-95 records as impossible over SFTP - see R-95 and R-436. **NOT YET MEASURED, and that distinction is the whole of what this row is worth: this is a support answer, not a proof.** What is owed is a real forced-command key on `u629488`, a restic `forget --prune` through it that is REFUSED, and a backup through it that still succeeds. | **BLOCKED** — **BLOCKED-ON-PROVIDER — Question 1 of `runbooks/provider-questions-2026-09-01.md`** | — | — | CC + operator | | **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC | -| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. **2026-11-03: demo-hp reached it too.** Its local tier has refused every try since 2026-10-30 („local has 9.4 GiB free; the last archive of guest 9201 was 9.1 GiB"; the disk is 39 GB, the newest archive 10-29), every 15 minutes: the mails `whole_guest_backup_failed` (08:11 CET, 2026-11-03) and `expected_backup_missed` (05:00 CET, „newest backup is 120h old"). PBS and off-site are not this tier. Not caused by the release. **2026-11-07 — the box half again (operator's yes):** still refused every ~4 h („local has 9.4 GiB free … needs about 12.4 GiB"), `expected_backup_missed` daily to 216 h, and the OS updates held behind it (`os_update_stale`). The journal had grown to 3.8 GB (journald's default cap on a 39 GB root); shrunk to 500 MB (`journalctl --vacuum-size=500M`, the whole journal saved first, off-git on DooPlex) and the August test archive of guest 9100 removed (`pvesm free`): `/` 9.5 GB → 14 GB free. The journal will grow back, so this buys weeks; the product half above is still the fix. `audits/demo-check-2026-11-07/`. | — | — | CC | +| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. **2026-11-03: demo-hp reached it too.** Its local tier has refused every try since 2026-10-30 („local has 9.4 GiB free; the last archive of guest 9201 was 9.1 GiB"; the disk is 39 GB, the newest archive 10-29), every 15 minutes: the mails `whole_guest_backup_failed` (08:11 CET, 2026-11-03) and `expected_backup_missed` (05:00 CET, „newest backup is 120h old"). PBS and off-site are not this tier. Not caused by the release. **2026-11-07 — the box half again (operator's yes):** still refused every ~4 h („local has 9.4 GiB free … needs about 12.4 GiB"), `expected_backup_missed` daily to 216 h, and the OS updates held behind it (`os_update_stale`). The journal had grown to 3.8 GB (journald's default cap on a 39 GB root); shrunk to 500 MB (`journalctl --vacuum-size=500M`, the whole journal saved first, off-git on DooPlex) and the August test archive of guest 9100 removed (`pvesm free`): `/` 9.5 GB → 14 GB free. The journal will grow back, so this buys weeks; the product half above is still the fix. `audits/demo-check-2026-11-07/`. **2026-11-09 — refused again** („local has 12.5 GiB free; the last archive of guest 9201 was 9.8 GiB, so a new one needs about 13.3 GiB"): the 11-08 copy had grown 9.1 → 9.8 GiB, so the night's OS run and its kernel step waited, and the host was not restarted. With the operator's yes: 36 unused old controller, infrastructure and helper images deleted inside the guest (3.4 GB of the source) and installer ISO 1.30.0 removed from `local` (its sha256 matched the copy on DooPlex and iso.felhom.eu): 14.0 GiB free. The causes are filed: R-1366 (the copy carries every image), R-1367 (the journal); controller v0.351.0 deletes the old images by itself. Evidence off-git: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/`. | — | — | CC | | **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) | | **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC | | **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC | +| **R-1366** | Backup & restore | P3 | **The whole-guest copy carries every Docker image, so it is mostly programs we could download again.** MEASURED 2026-11-09 on demo-hp guest 9201 (`docker system df -v`): the 32 GB source holds 27.5 GB of images — 13.2 GB used by containers, 9.8 GB the apps' undo images (decision 53, on purpose), 3.5 GB old controller and infrastructure images (deleted by hand the same day; controller v0.351.0 deletes them by itself) — and about 1.5 GB of Docker volumes plus 1.3 GB `sys_drive`. Packed: 9.8 GB, on a 39 GB system disk (R-548). The N100's copy is 1.5 GB (few apps). A copy without the images would be about a tenth of the size, but a whole-guest restore is the disaster path and would then need every image pulled again — an upstream tag can be gone (R-933 is the mirror). This is a design choice about the disaster path, not a defect. | OPEN — filed 2026-11-09 | operator decision | Options to put to the operator: (a) keep the images in every tier and size the local target (a backup drive, or the local tier skipped when it cannot fit — R-548); (b) the LOCAL tier only leaves the images out (`vzdump --exclude-path` on Docker's `image/` and `overlay2/`) and its restore re-pulls, PBS and off-site keep them; (c) no tier keeps them, after R-933. Measure first on scratch 9202: a whole-guest restore of a copy without images, every app back and the time it took. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/` (off-git, DooPlex) | CC | | **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC | | **R-164** | Backup & restore | P4 | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC | | **R-213** | Backup & restore | P4 | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **DEFERRED** — operator ruling 2026-10-12 (S18, `09` §3 decision 213): designed after the first customers. Stays P4; re-dated: look again when the first paying household has run 30 days. **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | After the first customers | Operator + CC | @@ -269,12 +270,13 @@ stopping line that lies. | **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC | | **R-1312** | Backup & restore | P3 | **The hub customer page's off-site switch did not reach the box: on Tester 1 (2026-10-27, operator-attended, ~14:00–14:12 UTC) the box's saved off-site target stayed `enabled` and its `controller.yaml` `offsite.enabled: true` (unchanged since 2026-10-04) while the hub showed off-site OFF.** So what the switch stops on the box side is not known: the controller reads `cfg.Offsite.Enabled` only at start (`cmd/controller/main.go:890`), and nothing rewrote the file. Either the switch is hub-side only by design (then the page must say so), or a box keeps pushing off-site while the operator believes it is off. Found by the R-1218 live attempt (R-1218 closed 2026-10-27 with its live proof waived by the operator). | OPEN | — | Read the hub handler behind the switch and what it changes (config, tenancy, storage access); then one attended run: switch off, watch the box's next off-site push (does it run, does it fail at the storage) | CC | -## Box system & updates — 16 rows (P3 10, P4 6) +## Box system & updates — 17 rows (P3 11, P4 6) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC | | **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | +| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC | | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC | | **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC | | **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC | diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index cebe8eb2..4d402112 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.** +## Moved from STATUS on 2026-11-09 + +**Updated 2026-11-08.** The night went right: the N100 installed its Proxmox updates and a new kernel and restarted itself; demo-hp made its first local whole-box copy in 10 days. Hub 0.179.1 no longer says „needs a restart" for a box that has restarted; demo-hp still needs one (its kernel step, likely tonight). + ## Moved from STATUS on 2026-11-08 **Updated 2026-11-07 (late).** The three test boxes run agent 0.172.1 (you vouched and signed it): the N100's nightly Proxmox updates are no longer refused, and demo-hp has space for its local whole-box copy again. DooPlex's old copy of the website is retired; the installer's version pin now lives in `INSTALLER-TAG`, and two browser headers are back on the public site.