Test-box check 2026-11-07: demo-hp space freed (R-548 note), N100 pve-step R6 diary entry (agent 0.172.1), STATUS do-list, R-1228 due date moved (asked)
gates / gates (push) Successful in 1m58s
gates / gates (push) Successful in 1m58s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
adminandClaude Opus 5.5 committed
1 parent
6d52329e3f
commit
7c6d3236df
8 files changed
+81
-29
No files matched your search
@@ -1,30 +1,31 @@
|
||||
# REPORT — read-back of the first real kernel night (2026-10-07 → 08)
|
||||
# REPORT — test-box check after a week away (2026-11-07, attended)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| demo-felhom — step ran | **PASS.** Whole-guest backup 04:35; guest/host/Proxmox steps 04:37–04:38; kernel **staged 04:39:04 as exactly the told 7.0.14-20** (the sources offered 7.0.14-22 — R-898's fix held); restart 04:39:08; boot on 7.0.14-20; judged healthy **04:40:38** (38 s after the agent started); **7.0.14-20 is the default**, flag empty, step `good`. |
|
||||
| demo-felhom — apps | Away about **1.5 min** (restart 04:39:08 → every container healthy at the 04:40:38 verdict). |
|
||||
| demo-felhom — alarms | None: no `host_stale`, no backup failure mail (demo-felhom has no drive apps, so R-897's fix was not exercised there). Crash guard armed, 0 unclean boots. Hub logged staged → judging → applied. |
|
||||
| demo-hp — step ran | **NO.** No whole-guest backup that night: the last one was yesterday's morning press at 08:49:53; with the 24 h cadence it came due at 08:49 today, after the window closed (08:30). No backup → no night leg → no kernel step. Nothing changed: running and default 7.0.14-20. Filed **R-899**. |
|
||||
| Reply-To | **Proven by you:** your reply to the test mail (2026-10-07 20:29) went to admin@felhom.eu. |
|
||||
| Your inbox overnight | Only Tester 2's expected missed-backup mails (the laptop is off). |
|
||||
**Plain words first.** All three test boxes (N100, HP, Tester 1) run, report to the hub, and all apps are healthy.
|
||||
Two faults were found. demo-hp made no local whole-box copy for 9 days because its system disk was too full; CC freed
|
||||
space with the operator's yes. The N100 refused its nightly Proxmox updates three nights in a row; agent 0.172.1 fixes
|
||||
it and waits for the operator's vouch and signed jobs (commands in `felhom-agent/REPORT.md`).
|
||||
|
||||
**Rows: 126 before → 127 after. Opened 1 (R-899). Closed 0.**
|
||||
**Method.** Two channels: the hub's database copied from the running pod WITH its WAL (events since 2026-10-31,
|
||||
`hosts`, `reports`, OS releases), and SSH to each box (agent journal, `pct list`, `docker ps` in guest 9201, `df`,
|
||||
`apt-get -s`). The two agree. The DB copy was deleted from the scratchpad after reading.
|
||||
|
||||
**What happens next, by itself:**
|
||||
- demo-hp is still due 7.0.14-22. A new household mail is allowed after 14:38 today (20 h after the last); its
|
||||
whole-guest backup is due inside tonight's window → the step should run on the night 8→9.
|
||||
- demo-felhom now sees 7.0.14-22 pending → due; its household gets a mail after 09:00; it should step to 22 tonight too.
|
||||
- After that both run 7.0.14-22, and "Approve kernel set" can appear. Read back the morning of 2026-10-09.
|
||||
**What the week shows.**
|
||||
- Every night: DB dumps and drive copies done on all three. Both hosts rebooted 2026-10-29 ~05:00 for the kernel step.
|
||||
- 2026-11-01 14:19: all three hosts stale together for ~1 h, then recovered — same minute on all three, so a line or
|
||||
hub-side gap, not a box fault.
|
||||
- Tester4 (2026-11-03): CC's proof VM, its host already deleted from the hub — the STATUS do-item is removed.
|
||||
- Tester 1 (2026-11-02): radicale stopped as unhealthy twice, and a disk warning at 89 %; it is at 15 % now.
|
||||
- All three: „Root files behind" (bundle 0.170.0). The 0.172.1 bundle job clears it.
|
||||
|
||||
**Seen, not fixed:** the after-boot "judging" report carries ring 1 (the agent reads its ring from the hub's block, which
|
||||
it has not fetched yet in the first second after a boot). Cosmetic: the hub's approval reads its own ring list, not
|
||||
this field. Not fixed today because it needs an agent release and delivery for a label.
|
||||
**Done.**
|
||||
- demo-hp: whole journal saved off-git (`/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/`), then
|
||||
`journalctl --vacuum-size=500M` (freed 3.3 GB) and `pvesm free` of the 2026-08-21 archive of guest 9100: `/` 9.5 → 14 GB
|
||||
free; the preflight asks 12.4 GiB. Recorded under R-548. The journal will grow back (journald default cap).
|
||||
- felhom-agent v0.172.1 released (not vouched) — see that repo's REPORT.
|
||||
- N100 diary entry; STATUS updated (new do-list, the Tester4 item removed).
|
||||
|
||||
Evidence: `documentation/audits/kernel-night-2026-10-07/readback/`.
|
||||
**Not yet observed.** demo-hp's next local-tier try (every ~4 h) has not run since the cleanup. Positive observable:
|
||||
`backup: completed … target=local` in its agent journal.
|
||||
|
||||
## Decisions for you
|
||||
|
||||
1. **R-899 — a daytime "back up now" press skips the next night's backup (and its updates).** My pick: count nights,
|
||||
not 24 hours (the backup is due when the last one is older than ~20 h at the window's start). If you do nothing: each
|
||||
daytime press costs one night, and the kernel step waits one more day.
|
||||
**Registers.** No row opened or closed (the GRUB fault is fixed in the session, size rule). Evidence:
|
||||
`documentation/audits/demo-check-2026-11-07/`. System poster: no fact changed.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-11-03.** Every box runs controller 0.350.0 with Traefik v3.7.17; the hole is closed, proven on the scratch box. Golden 0.350.0 and agent 0.172.0 are vouched (10:18 CET), hub 0.179.0 is live, and Paperless 3, n8n 3, Mealie and Termix are offered.
|
||||
**Updated 2026-11-07.** CC checked the three test boxes: all run and all apps are healthy. demo-hp had no local whole-box copy for 9 days (its disk was full); CC freed space. The N100 refused its nightly Proxmox updates (a GRUB package in the wrong group); agent 0.172.1 fixes it and waits for your vouch and signed jobs.
|
||||
|
||||
## What works
|
||||
- **The Plex attack did not reach Felhom's secrets** (R-1241).
|
||||
@@ -49,7 +49,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). (since 10-14)
|
||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. (since 10-14)
|
||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). (since 10-12)
|
||||
**2026-11-03.** 1. From about 11:00 CET: delete host `Tester4-e286d3` on the Hosts page (CC's proof VM, gone). If not: stale mails. 2. To the homelab session: R-1355.
|
||||
**2026-11-07.** 1. Vouch agent 0.172.1 (sha `505a0594…`, bundle `3e7acaee…`; `felhom-agent/CHANGELOG.md`). 2. Send each of the three test boxes two signed jobs: `agent_update` 0.172.1, then `agent_config_update` 0.172.1 (commands in `felhom-agent/REPORT.md`). If not: the N100 gets no Proxmox updates and mails `os_update_failed` every morning; all three keep the „Root files behind" warning. **2026-11-03.** To the homelab session: R-1355.
|
||||
**2026-10-30.** On the hub's Hosts page: Discard the appliance of 10:44:59 (uuid d3529d2d…) and appliance 49 (CC's proof VMs). If not: dead entries alarm. **2026-10-29 (release run).** 1. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes.
|
||||
**2026-10-23 do list.**
|
||||
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
Full demo-hp journal (2026-10-04..2026-11-07, 49 MB gz) kept off-git on DooPlex: /mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/
|
||||
@@ -0,0 +1,21 @@
|
||||
== before
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/mapper/pve-root 39G 28G 9.5G 75% /
|
||||
Archived and active journals take up 3.8G in the file system.
|
||||
local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst tar.zst backup 656970239 9100
|
||||
local:backup/vzdump-lxc-9201-2026_10_29-04_52_27.tar.zst tar.zst backup 9765465774 9201
|
||||
local:backup/felhom-golden-0.236.0.tar.zst tar.zst backup 654115664
|
||||
VMID Status Lock Name
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
9401 stopped upgrade-harness
|
||||
== act
|
||||
Vacuuming done, freed 3.3G of archived journals from /var/log/journal/9756a908fef046fda6a56428d3fefe3c.
|
||||
Vacuuming done, freed 0B of archived journals from /run/log/journal.
|
||||
Removed volume 'local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst'
|
||||
== after
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/mapper/pve-root 39G 24G 14G 64% /
|
||||
Archived and active journals take up 468M in the file system.
|
||||
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
|
||||
local dir active 40453376 24241552 14124708 59.92%
|
||||
@@ -0,0 +1,24 @@
|
||||
## os-apply refusals (journal)
|
||||
Nov 04 05:13:34 demo-felhom felhom-os-apply[1475023]: os-apply: START release=ring0-20261104T041155Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0
|
||||
Nov 05 05:16:58 demo-felhom felhom-os-apply[3114895]: os-apply: START release=ring0-20261105T041548Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0
|
||||
Nov 05 05:17:05 demo-felhom felhom-os-apply[3115256]: os-apply: REFUSED: R6 the plan would touch grub-efi-amd64, which is not in the plan
|
||||
Nov 06 05:21:34 demo-felhom felhom-os-apply[561830]: os-apply: START release=ring0-20261106T042042Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0
|
||||
Nov 06 05:21:42 demo-felhom felhom-os-apply[562230]: os-apply: REFUSED: R6 the plan would touch grub-efi-amd64, which is not in the plan
|
||||
Nov 07 05:26:34 demo-felhom felhom-os-apply[2202999]: os-apply: START release=ring0-20261107T042542Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0
|
||||
Nov 07 05:26:42 demo-felhom felhom-os-apply[2203373]: os-apply: REFUSED: R6 the plan would touch grub-efi-amd64, which is not in the plan
|
||||
|
||||
## read-only simulation
|
||||
Inst proxmox-grub [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64]) []
|
||||
Inst grub-efi-amd64 [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64]) []
|
||||
Inst grub2-common [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64]) []
|
||||
Inst grub-efi-amd64-unsigned [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64]) []
|
||||
Inst grub-efi-amd64-bin [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64]) []
|
||||
Inst grub-pc-bin [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64]) []
|
||||
Inst grub-common [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64]) [grub-efi-amd64-signed:amd64 ]
|
||||
Inst grub-efi-amd64-signed [1+2.12+9+pmx2] (1+2.12+9+pmx4 Proxmox Debian Repository:stable [amd64])
|
||||
|
||||
proxmox-grub
|
||||
|Depends: grub-efi-amd64
|
||||
|Depends: grub-efi-ia32
|
||||
Depends: <grub-efi-arm64>
|
||||
Depends: grub-common
|
||||
@@ -181,7 +181,7 @@ stopping line that lies.
|
||||
| **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC |
|
||||
| **R-433** | Backup & restore | P3 | **A sub-account cannot reach ANY Storage Box snapshot, by any name — so clause (b) of the 2026-09-01 R-95 re-scope ("the rest is recoverable file by file") is NOT SUPPORTED.** MEASURED 2026-09-01 on `demo-hp` over the credential the box already holds, read-only, no delete verb issued. **The sweep:** a batched `stat -c %n` over Hetzner's port-23 restricted shell (500–600 paths per round trip, stdout carrying only paths that exist) tried **777,600** names of the vendor form `YYYY-MM-DDTHH-MM-SS` across nine full days at second granularity, and 126 alternative shapes — **zero resolved.** **The control is what makes the zero mean anything:** the identical 600-name batch with one real path appended returned it in 6 of 6 batches. **The structural cause:** `/home` (the customer data, `u629488-sub3`) is **st_dev 0,82**; `/.zfs/snapshot` is **st_dev 0,276**; `/home/.zfs` does not exist. A snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding `felhom-repo`. **What still stands:** clause (a) — the box can delete its live repository but cannot WRITE into `/.zfs/snapshot` — is unchanged and re-confirmed. **What is now open again:** the only routes to the older copy are a panel rollback of the WHOLE Storage Box (deletes newer snapshots, hits every customer on it) and the provider API (fenced by §11-D, and `hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method at all**, so it needs new code regardless). **NOT ESTABLISHED, and it is the question that decides whether per-file recovery exists for anyone:** whether the MAIN account can see the snapshots. No main-account credential exists in this project. **The ranking is Viktor's; I am not re-ranking R-95 — but the argument that moved it down is the argument this drill removed.** `audits/evidence-drill-r95-recovery-2026-09-01/` **BLOCKED-ON-PROVIDER 2026-09-01.** The one question that can move this is drafted and ready to send: **Question 1** of `documentation/runbooks/provider-questions-2026-09-01.md` — *can the MAIN account retrieve individual files from a snapshot, without a whole-box restore?* **Neither answer leaves this row where it is:** "yes" makes per-file recovery real but permanently operator-only, and closes this; "no, full restore only" means the snapshots do not bound a single customer's exposure at all, because using them costs every other customer on the box their newer snapshots — **and R-95 becomes urgent.** Nothing here can progress without it, and it is not CC's to send (§11-D). Tracked by a dated check in the DUE-CHECKS block. **⚠ A TRAP ON THE WAY TO THIS ANSWER, found by the operator 2026-09-01 and armed against in the questions file: HETZNER HAS TWO SIMILARLY-NAMED PRODUCTS WHOSE DOCS SAY OPPOSITE THINGS.** **Storage SHARE** (a managed Nextcloud, NOT us) documents *"Currently, we only support restores for the full backup ZFS snapshot to a specific point in time"* (`docs.hetzner.com/storage/storage-share/faq/backup-snapshot/`). **Storage BOX** (ours) documents the opposite — *"You can download individual files or entire directories as usual"* (`docs.hetzner.com/storage/storage-box/snapshots/`). **A web search for the obvious phrasing surfaces the SHARE page, and it reads like a definitive NO.** Tell them apart by the giveaways: the Share page talks about *Nextcloud's data cache*, a *database dump* and the *konsoleH* interface, and never mentions Storage Box. **Why this is filed and not left as trivia: a support agent could answer Question 1 from the wrong page, and that answer would push this row to the top of the register for no reason.** Question 1 now names the product, quotes the Storage Box line, and says explicitly that we know what the Share FAQ says. **If an answer cites full-snapshot-only, check WHICH PRODUCT it is about before acting on it.** **Nothing in this repository ever leaned on the Share claim** — verified by grep at the time; the only vendor line we cite is the Storage Box one. **RE-DATED 2026-09-15 → 2026-09-22 (due-checks gate):** the operator mailbox read through the Gmail connector (`(from:hetzner OR subject:hetzner OR "Storage Box") after:2026/09/01`) holds no Hetzner reply — one match, our own `offsite_snapshots_dropped` alarm. Whether the two tickets were ever sent is not visible to CC; if they were not, sending them is the operator's act. **-- ANSWERED. Hetzner replied on ticket #2026090103040671; the operator supplied the thread on 2026-09-22 after this session searched for it and WRONGLY reported no answer (see the instrument note below and R-628).** **Q1 - can the MAIN account read individual files and directories out of a specific snapshot, without restoring the whole box?** Hetzner: *"With the main account it should be possible to download files and directories from the snapshot as it is stated in the documentation"*, citing `docs.hetzner.com/storage/storage-box/snapshots#access-to-snapshots`; and separately *"A restLine truncated
|
||||
| **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC |
|
||||
| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. **2026-11-03: demo-hp reached it too.** Its local tier has refused every try since 2026-10-30 („local has 9.4 GiB free; the last archive of guest 9201 was 9.1 GiB"; the disk is 39 GB, the newest archive 10-29), every 15 minutes: the mails `whole_guest_backup_failed` (08:11 CET, 2026-11-03) and `expected_backup_missed` (05:00 CET, „newest backup is 120h old"). PBS and off-site are not this tier. Not caused by the release. | — | — | CC |
|
||||
| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. **2026-11-03: demo-hp reached it too.** Its local tier has refused every try since 2026-10-30 („local has 9.4 GiB free; the last archive of guest 9201 was 9.1 GiB"; the disk is 39 GB, the newest archive 10-29), every 15 minutes: the mails `whole_guest_backup_failed` (08:11 CET, 2026-11-03) and `expected_backup_missed` (05:00 CET, „newest backup is 120h old"). PBS and off-site are not this tier. Not caused by the release. **2026-11-07 — the box half again (operator's yes):** still refused every ~4 h („local has 9.4 GiB free … needs about 12.4 GiB"), `expected_backup_missed` daily to 216 h, and the OS updates held behind it (`os_update_stale`). The journal had grown to 3.8 GB (journald's default cap on a 39 GB root); shrunk to 500 MB (`journalctl --vacuum-size=500M`, tLine truncated
|
||||
| **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) |
|
||||
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
|
||||
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
|
||||
@@ -210,7 +210,7 @@ stopping line that lies.
|
||||
| **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. **2026-10-25:** the kept-data choice no longer takes such a future-dated copy as the newest (R-1271, controller `bd9f4f7`); retention is unchanged — this row. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC |
|
||||
| **R-1212** | Backup & restore | P4 | **Two household texts on the backup path do not follow the box's language:** the share-restore summary (`SharesRestoreResult.FlashMessage`, `controller/internal/backup/shares_restore.go`) is built from Hungarian Go literals, and the off-site `LastError` for an interrupted run is the Hungarian literal „megszakadt futás (a vezérlő újraindult futás közben)" (`controller/internal/backup/backup.go`). An English household reads Hungarian there. Found by the night helper 2026-10-21. | OPEN | — | Bundle keys for both (rendered in the box language when written, R-557), with parity fixtures | CC |
|
||||
| **R-1221** | Backup & restore | P4 | **The nightly whole-box backup starts ~1.5–5 min later each night, until one night is skipped (after ~70–80 nights with no press between).** REAL, low harm (night 2026-10-22; split from R-1189 (b)): agent `localapi/server.go:1184` is `age >= Cadence` with the age counted from the archive START; the controller polls every 5 min (`quiesce.go:150`) and the archive starts after the apps stop. Measured on demo-hp: 04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40. At W+6h one night is skipped (44 h is below the 48 h valve, `quiesce.go:807-818`); the hub's 26 h line (`deadline.go:29`) mails the operator. Any manual or off-cycle run resets the drift. Evidence `audits/night-2026-10-22/P3-candidates/demohp-whole-box-completions.txt`. | OPEN | — | Fix: inside the night window a tier is due at `age >= cadence - 4h`; the agent answers due-ness, so it is agent code (a unit test with an injected clock) and rides an agent release | CC |
|
||||
| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). **2026-10-30: SWITCHED** (operator yes, 10:02:35 UTC): `felhom.eu` and `www` are Cloudflare Pages (`felhom-site`, uploaded by `scripts/publish-site-cloudflare.sh`); 30 of 30 pages and the installer byte-identical to `installer-v1.36.1` from outside; a fresh install from the public ISO 1.30.0 (throwaway VM 3461 on demo-hp) fetched the installer from the test address and from `felhom.eu` and passed the preflight both times; the contact form posts to `api.felhom.eu` (DooPlex mailer, a test mail arrived). `audits/release-2026-10-30/p3-cloudflare/`. **What is left:** a new install still needs DooPlex for the hub (claim, config, first report) and Gitea (the agent package, the golden) — by design (D35); DooPlex's webpage copy runs until 2026-11-06 as the way back (DUE-CHECKS), then the operator decides to retire it. NARROWED | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) |
|
||||
| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). **2026-10-30: SWITCHED** (operator yes, 10:02:35 UTC): `felhom.eu` and `www` are Cloudflare Pages (`felhom-site`, uploaded by `scripts/publish-site-cloudflare.sh`); 30 of 30 pages and the installer byte-identical to `installer-v1.36.1` from outside; a fresh install from the public ISO 1.30.0 (throwaway VM 3461 on demo-hp) fetched the installer from the test address and from `felhom.eu` and passed the preflight both times; the contact form posts to `api.felhom.eu` (DooPlex mailer, a test mail arrived). `audits/release-2026-10-30/p3-cloudflare/`. **What is left:** a new install still needs DooPlex for the hub (claim, config, first report) and Gitea (the agent package, the golden) — by design (D35); DooPlex's webpage copy runs until 2026-11-06 as the way back (DUE-CHECKS), then the operator decides to retire it. NARROWED **2026-11-07: asked** (demo-check session, in chat); the due date moved to 2026-11-14 to wait for the answer. | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) |
|
||||
| **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator |
|
||||
| **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC |
|
||||
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
|
||||
@@ -406,6 +406,6 @@ stopping line that lies.
|
||||
| R-1035 | 2026-11-11 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row |
|
||||
| R-528 | 2026-11-12 | no failed oom_check / unexplained app_oom since 2026-10-12; detail in the R-528 row |
|
||||
| R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row |
|
||||
| R-1228 | 2026-11-06 | D35 way back ends: ask the operator to retire DooPlex's felhom.eu/www webpage copy (ingress + git-sync), or keep it longer; detail in the R-1228 row |
|
||||
| R-1228 | 2026-11-14 | D35 way back ends: ask the operator to retire DooPlex's felhom.eu/www webpage copy (ingress + git-sync), or keep it longer; detail in the R-1228 row |
|
||||
<!-- DUE-CHECKS-END -->
|
||||
| **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC |
|
||||
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-11-07
|
||||
|
||||
**Updated 2026-11-03.** Every box runs controller 0.350.0 with Traefik v3.7.17; the hole is closed, proven on the scratch box. Golden 0.350.0 and agent 0.172.0 are vouched (10:18 CET), hub 0.179.0 is live, and Paperless 3, n8n 3, Mealie and Termix are offered.
|
||||
|
||||
## Moved from STATUS on 2026-11-03
|
||||
|
||||
**Updated 2026-11-03 (morning).** On your word, golden 0.349.0 is baked; the code-push gate is green again. It needs your vouch (first line of the release list). It carries Traefik v3.7.16 until controller 0.350.0 (R-1354).
|
||||
|
||||
@@ -52,3 +52,4 @@ what happened, who noticed, and the register row it became. Prose here is not a
|
||||
the gap the Proxmox host's clock read 14 s behind DooPlex and demo-hp: `chrony` has failed at every boot since at least
|
||||
2026-10-16 („Could not send notification to $NOTIFY_SOCKET"); demo-hp, same package and options, runs it. Register:
|
||||
R-1300 (the release), R-1301 (the clock). Evidence: `audits/cloudflared-2026.10.3-2026-10-26/` (`obs-n100-clock.txt`). (CC)
|
||||
- **2026-11-05..07 — the nightly Proxmox update step refused three nights in a row.** `proxmox-grub` 2.12-9+pmx4 needs `grub-efi-amd64` of the same version; the update wrapper put `proxmox-grub` in the Proxmox group (its boot-package rule matched only names that START with `grub`), so the whole step was refused (R6) and 34 Proxmox updates waited; an `os_update_failed` mail each morning. Apps not touched. Agent 0.172.1 puts `proxmox-grub` with the boot packages; it reaches the box by the operator's vouch and two signed jobs. Register: fixed in the session (size rule), recorded in the agent's CHANGELOG. Evidence: `audits/demo-check-2026-11-07/n100-pve-step-R6.txt`. (CC)
|
||||
Reference in new issue
Block a user