lists: the Clock list (70 jobs) and the End-dates list (39 things), 9 register rows, 4 decisions
gates / gates (push) Successful in 2m2s

Read-only review of every job on a timer and of everything that ends on a date,
with measurements from the three test boxes. Nothing ends in the next 14 days.
Register 236 -> 245 (R-1379..R-1387), pointer lines on twelve rows, D76-D79 in
STATUS.md. Part C: felhom-agent's clone address holds no token now; the other
clones wait for the operator's yes. R-1035's 4-week check moved to 2026-11-18
(operator read needed).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-11 11:54:42 +01:00
1 parent 43e244ad8f
commit 978666dde3
81 files changed
+26738 -20

No files matched your search

+28 -19
View File
@@ -168,7 +168,7 @@ stopping line that lies.
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:** **OPEN — rank P3-LOW; owner: CC (after R-784)** **2026-11-02:** v1.10.0 and v1.11.0 read clean of HTTP2Shell (Node 24.22.1); v1.9.0 is still bad. **The big step v0.17.3 → v1.11.0 is proven on the drill:** bench, 9202 (peak 29.4 %), the undo (`audits/security-2026-11-02/bench/sparkyfitness/`, `walks/sparkyfitness-drill/`). Not written: R-784 (the licence) stands before it. | — | — | CC |
## Backup & restore — 48 rows (P2 4, P3 26, P4 18)
## Backup & restore — 49 rows (P2 4, P3 27, P4 18)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -177,7 +177,7 @@ stopping line that lies.
| **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | NARROWED — **2026-10-23 (D32):** the sheet is current — names and places only, one A4, HU + EN (`runbooks/break-glass-sheet.md`); every place checked to exist. LEFT: the operator prints it, fills it by hand, keeps it outside the home. **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G1/G3/G10):** unchanged; the sheet is still the one act that breaks the circle, and S5's offline form is unknown to CC — on the sheet too (decide list in STATUS). | operator: print the sheet | Operator: print + store the sheet; then close | operator |
| **R-1254** | Backup & restore | P2 | **The hub saves a household's new escrow package in TWO writes; a failure between them leaves the new repository hash beside the OLD identity package, and the box then marks itself escrowed — on a total loss the written code opens nothing.** Found 2026-10-24 (HUNT-3 H1), read at `file:line`: `hub/internal/api/handler.go:1415` `SaveHostEscrow` commits blob + hash, `:1445` `SaveHostDRBundle` sets `identity_blob` in a second statement (`internal/store/store.go:3788-3796`; the upsert `:3490-3511` never touches it); the controller's confirm reads the hash (`felhom-controller/controller/internal/report/escrow_confirm.go:195-217`); the agent sends once. Same family (H2): a lost or late answer (30 s) tells the household „the previous code still works” when the hub may already hold the new package (`felhom-agent/internal/localapi/escrow_ceremony.go:79,532-534,639-641`). Static. **The hub half is fixed on main 2026-10-24 night** (felhom.eu `dcf4f84c`; Part 3): key, hash and identity blob in ONE transaction (`SaveHostEscrowWithIdentity`), a bad identity refused before any write; hub image `0.170.0-rc.r1254` built locally (`67db584a45d7`), NOT pushed, NOT deployed. The agent ceremony half (H2: „the previous code still works” after a lost answer) is still open. | **NARROWED** — **released 2026-10-29:** agent 0.168.0 (vouched) + controller 0.347.0 (the floor); a ceremony ran on both (the happy path); the lost-answer branch has tests only (a live proof needs a hub answer lost on purpose). **the hub half CLOSED 2026-10-25:** hub **0.170.0** deployed on the operator's yes (manifest bump, ArgoCD sync of the hub Deployment only, image digest `6478498d…`; 30 minutes and more with no alarm mail, the mailbox as the control); Tester 1 ran the escrow ceremony end to end through the page's endpoints and the hub stored the new package in one transaction („the new package is on the hub", `audits/release-2026-10-26/escrow-ceremony-tester1.txt`). The ceremony half stays open. **2026-10-28 night: the ceremony half FIXED ON MAIN** (agent `5ddd124` CI 2473, controller `e33cffa` CI 2474; tests red first; `03` §8 [DESIGN] 2026-10-28): after a failed upload the agent asks the hub which package it holds (the identity blob compared) — the new one → success, another → 502 and „the previous code still works", no answer → 504 that promises nothing; the controller words 504 and its own failed call as „Nem tudjuk biztosan… Őrizd meg mindkét kódot…" and waits 60 s for the claim/confirm (was 15 s, shorter than the agent's 30 s upload). | the hub release (attended); the agent half | deploy hub v0.170.0 (Monday list); then the ceremony asks the hub what it holds before it says „the previous code still works” | CC |
| **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC |
| **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC |
| **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC08-3). | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC |
| **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC |
| **R-433** | Backup & restore | P3 | **A sub-account cannot reach ANY Storage Box snapshot, by any name — so clause (b) of the 2026-09-01 R-95 re-scope ("the rest is recoverable file by file") is NOT SUPPORTED.** MEASURED 2026-09-01 on `demo-hp` over the credential the box already holds, read-only, no delete verb issued. **The sweep:** a batched `stat -c %n` over Hetzner's port-23 restricted shell (500–600 paths per round trip, stdout carrying only paths that exist) tried **777,600** names of the vendor form `YYYY-MM-DDTHH-MM-SS` across nine full days at second granularity, and 126 alternative shapes — **zero resolved.** **The control is what makes the zero mean anything:** the identical 600-name batch with one real path appended returned it in 6 of 6 batches. **The structural cause:** `/home` (the customer data, `u629488-sub3`) is **st_dev 0,82**; `/.zfs/snapshot` is **st_dev 0,276**; `/home/.zfs` does not exist. A snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding `felhom-repo`. **What still stands:** clause (a) — the box can delete its live repository but cannot WRITE into `/.zfs/snapshot` — is unchanged and re-confirmed. **What is now open again:** the only routes to the older copy are a panel rollback of the WHOLE Storage Box (deletes newer snapshots, hits every customer on it) and the provider API (fenced by §11-D, and `hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method at all**, so it needs new code regardless). **NOT ESTABLISHED, and it is the question that decides whether per-file recovery exists for anyone:** whether the MAIN account can see the snapshots. No main-account credential exists in this project. **The ranking is Viktor's; I am not re-ranking R-95 — but the argument that moved it down is the argument this drill removed.** `audits/evidence-drill-r95-recovery-2026-09-01/` **BLOCKED-ON-PROVIDER 2026-09-01.** The one question that can move this is drafted and ready to send: **Question 1** of `documentation/runbooks/provider-questions-2026-09-01.md` — *can the MAIN account retrieve individual files from a snapshot, without a whole-box restore?* **Neither answer leaves this row where it is:** "yes" makes per-file recovery real but permanently operator-only, and closes this; "no, full restore only" means the snapshots do not bound a single customer's exposure at all, because using them costs every other customer on the box their newer snapshots — **and R-95 becomes urgent.** Nothing here can progress without it, and it is not CC's to send (§11-D). Tracked by a dated check in the DUE-CHECKS block. **⚠ A TRAP ON THE WAY TO THIS ANSWER, found by the operator 2026-09-01 and armed against in the questions file: HETZNER HAS TWO SIMILARLY-NAMED PRODUCTS WHOSE DOCS SAY OPPOSITE THINGS.** **Storage SHARE** (a managed Nextcloud, NOT us) documents *"Currently, we only support restores for the full backup ZFS snapshot to a specific point in time"* (`docs.hetzner.com/storage/storage-share/faq/backup-snapshot/`). **Storage BOX** (ours) documents the opposite — *"You can download individual files or entire directories as usual"* (`docs.hetzner.com/storage/storage-box/snapshots/`). **A web search for the obvious phrasing surfaces the SHARE page, and it reads like a definitive NO.** Tell them apart by the giveaways: the Share page talks about *Nextcloud's data cache*, a *database dump* and the *konsoleH* interface, and never mentions Storage Box. **Why this is filed and not left as trivia: a support agent could answer Question 1 from the wrong page, and that answer would push this row to the top of the register for no reason.** Question 1 now names the product, quotes the Storage Box line, and says explicitly that we know what the Share FAQ says. **If an answer cites full-snapshot-only, check WHICH PRODUCT it is about before acting on it.** **Nothing in this repository ever leaned on the Share claim** — verified by grep at the time; the only vendor line we cite is the Storage Box one. **RE-DATED 2026-09-15 → 2026-09-22 (due-checks gate):** the operator mailbox read through the Gmail connector (`(from:hetzner OR subject:hetzner OR "Storage Box") after:2026/09/01`) holds no Hetzner reply — one match, our own `offsite_snapshots_dropped` alarm. Whether the two tickets were ever sent is not visible to CC; if they were not, sending them is the operator's act. **-- ANSWERED. Hetzner replied on ticket #2026090103040671; the operator supplied the thread on 2026-09-22 after this session searched for it and WRONGLY reported no answer (see the instrument note below and R-628).** **Q1 - can the MAIN account read individual files and directories out of a specific snapshot, without restoring the whole box?** Hetzner: *"With the main account it should be possible to download files and directories from the snapshot as it is stated in the documentation"*, citing `docs.hetzner.com/storage/storage-box/snapshots#access-to-snapshots`; and separately *"A restLine truncated
| **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place O1, listed from documents only (the off-site box was not read in this run — second pass). | — | — | CC |
@@ -204,22 +204,23 @@ stopping line that lies.
| **R-1173** | Backup & restore | P3 | **R-921's consequence (no app stop for an off-site tier the agent will refuse) has not been seen live.** Agent 0.161.0 serves `host_busy` (proven live on demo-hp 2026-10-20 during a press: busy `backup:local`), and controller ≥ 0.335.0 reads it — but the case only arises on the night schedule (a manual press backs up the local tier only), so the no-stop branch is proven by its red-proved test alone. | OPEN — filed 2026-10-20 **2026-10-23 dated check — DONE (the other outcome):** demo-hp's off-site (PBS) tier ran 2026-10-22 20:42:53–20:49:48 UTC with nothing in flight (quiesce → started → snapshotted → early resume → done); no `host is busy` case arose, so the R-921 no-stop branch is still proven by its test alone. Tonight only the local tier was due (refused by space, R-548). Kept open; no new date — it reads itself the night a busy host meets a due off-site tier. | `audits/night2-2026-10-20/P1/1b-host-busy-poll.txt` | Read demo-hp's controller log after the night its off-site (pbs) tier is due (~2026-10-22): a `host is busy … no app is stopped (R-921)` line, or the off-site tier starting with nothing in flight | CC |
| **R-1184** | Backup & restore | P3 | **A cut-off shared-folder restore can be „placed" and reported restored while files are missing; the download is then deleted.** Readiness is only „the scratch folder is not empty" (`internal/backup/shares_restore.go:131-140`) — the R-358 class fixed for apps, not for shares; 30-min download window (`web/offbox_handlers.go:716`); fixed 2 GiB free-space gate that fails open (`:107`). CONFIRMED in code. Data-loss hunt 2026-10-20 (C1 reviewer 1, R3). | OPEN | — | Fix: a completion marker after restic succeeds, as for apps | CC |
| **R-1185** | Backup & restore | P3 | **A database dump that fails its own content check has already replaced last night's good dump, and the second drive copies it an hour later.** `appbackup/dbdump.go:388-422`: size check, rename over the good file (`:399`), then `ValidateDump` (`:419`). Volume tars are promoted with no content check either (`backup.go:902`). Tier 1 is one point per app by design (07 §6.1); the ORDER is the defect. PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/`, a docker stub prints a header-only dump): `ValidateDump` says invalid („dump file too small"), `DumpOneTo` returns no error, and last night's good dump (with its `CREATE TABLE`) is gone from disk. Data-loss hunt 2026-10-20 (D5). | OPEN | — | Fix: validate the temp file before the rename; keep the last good dump on a failed check | CC |
| **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC |
| **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC03-5, CC09). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC |
| **R-1188** | Backup & restore | P4 | **Restore-path candidates from the data-loss hunt — what is left is unlikely or a sentence:** (b) a failed `volume rm` is ignored before the untar, which then lands in the old volume (`backup/restore.go:218`; `StopStack` is `compose down`, so the rm rarely fails); (d) the full off-site download stops at 30 min per press and the household is not told to press again (`web/offbox_handlers.go:455`; the scratch is kept, so a second press probably continues — not measured); (e) place and full restore re-read the LATEST snapshot, not the downloaded one (`offbox_restore.go:675-679`, `_ = id`). Closed parts: (a) with R-1195 (controller 0.338.0), (c) in controller 0.337.0 (`TestR1188c_ALeftoverTempDoesNotBlockTheRetry`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — **2026-10-22 night: re-ranked P3→P4**: (a) and (c) are shipped; the rest is low-chance or a sentence. | — | (d): measure a second press on a large download on 9202, then write the sentence; (b), (e): unit tests when the file is next touched | CC |
| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt — what is left:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it: the agent turns a future archive time into a negative age (`localapi/server.go:1179-1192`); half-live 2026-10-21 (B6 #8, `s8-wrong-clock/README.md`: a local archive dated 2028-10-21); agent code, so it rides an agent release and the operator's vouch; (c) the whole-guest quiesce waits for the catch-up and the update leg, not for a restore (`quiesce.go:213-275, 565`); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — a design question; (g) spring forward 2027-03-28 moves 02:30 onto the 03:30 second copy; (h) self-update busy check before the pull only (`updater.go:532-545, 764`; not re-read); (i) a double-clicked restore shows failed while it runs (check and begin not atomic, `web/handlers.go:1860, 1881`). **Moved out 2026-10-22 night:** (b) the skipped nights after an off-cycle run were FIXED by R-899 (controller 0.304.0; demo-hp's journal shows a night run after each of three off-cycle runs since), the creep → R-1221; (d) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** the catch-up now runs the off-site proof and the integrity check once after its legs, the false comment corrected; red-proved (`nightchain/r1189d_test.go`, `cmd/controller/r1189d_wiring_test.go`); (e) → R-1222 (hub). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — **2026-10-23: (d) released in controller 0.340.0;** (a), (c), (f), (g) stay as written. OPEN — 2026-10-22 night: (b) half closed, (d) fixed in code, (e) split out. | — | (a) with the next agent release (a unit test with a future archive); (c) a 9202 recipe; (g) a unit test with the 2027 clock; (f) to the operator at the next design review | CC |
| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt — what is left:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it: the agent turns a future archive time into a negative age (`localapi/server.go:1179-1192`); half-live 2026-10-21 (B6 #8, `s8-wrong-clock/README.md`: a local archive dated 2028-10-21); agent code, so it rides an agent release and the operator's vouch; (c) the whole-guest quiesce waits for the catch-up and the update leg, not for a restore (`quiesce.go:213-275, 565`); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — a design question; (g) spring forward 2027-03-28 moves 02:30 onto the 03:30 second copy; (h) self-update busy check before the pull only (`updater.go:532-545, 764`; not re-read); (i) a double-clicked restore shows failed while it runs (check and begin not atomic, `web/handlers.go:1860, 1881`). **Moved out 2026-10-22 night:** (b) the skipped nights after an off-cycle run were FIXED by R-899 (controller 0.304.0; demo-hp's journal shows a night run after each of three off-cycle runs since), the creep → R-1221; (d) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** the catch-up now runs the off-site proof and the integrity check once after its legs, the false comment corrected; red-proved (`nightchain/r1189d_test.go`, `cmd/controller/r1189d_wiring_test.go`); (e) → R-1222 (hub). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CA01-6, CA01-8). | OPEN — **2026-10-23: (d) released in controller 0.340.0;** (a), (c), (f), (g) stay as written. OPEN — 2026-10-22 night: (b) half closed, (d) fixed in code, (e) split out. | — | (a) with the next agent release (a unit test with a future archive); (c) a 9202 recipe; (g) a unit test with the 2027 clock; (f) to the operator at the next design review | CC |
| **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt — what is left:** (c) PLAUSIBLE: after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists — a design question against `07` §6.1's Tier-3 gate (the operator's); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`) — (e), (f), (h) not re-read. **Moved out 2026-10-22 night:** (a) → R-1218, (b) → R-1219, (g) → R-1220. Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN | — | (c) a read-only fingerprint compare on a reinstalled test box; (e), (f), (h) re-read, then unit tests; (d) to the operator at the next design review | CC |
| **R-1198** | Backup & restore | P3 | **A backup disk the family adds later is never offered as the place for the whole-box backup; the page keeps offering the disk that holds the apps' own data.** Rehearsal 2026-10-20 B6 #13 (`audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/`): a 50 GB disk plugged in and initialised as „Mentolemez" → the backups page still offers only „Családi adatlemez — Kijelölöm"; `internal/web/backup_target_offer.go:101` `firstOfferableDrive` offers the FIRST eligible drive, never a choice, and the drive's card has no „use for backups". | OPEN | — | Offer every eligible drive (largest empty first) and let the household pick | CC |
| **R-1200** | Backup & restore | P3 | **A box installed in the afternoon gets no whole-box backup on its first night: the install's own backup starts the 24 h clock, so the family's whole first day reaches a whole-box copy only on the SECOND night (~38 h after install).** Rehearsal 2026-10-20 B5 (`audits/rehearsal-2026-10-20/B5-night/README.md`): Tester4's install backup at 12:09Z; at the 02:30–06:30Z window the agent answered „within cadence window" (`felhom-agent internal/localapi/server.go:1187`, `age >= tier.Cadence`, default 24 h). R-899 (closed) fixed the same shape for a daytime PRESS only. The other tiers did run that night (database copy, second drive, off-site — all PASS), so app data was covered; the whole-box (DR) tier was not. | **OPEN** — a design question: should the first scheduled night always run (like a press), or is one tier enough on night one? | — | decide; if „always run": treat the install backup like a press in the due rule, with a test on the first night | CC |
| **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. **2026-10-25:** the kept-data choice no longer takes such a future-dated copy as the newest (R-1271, controller `bd9f4f7`); retention is unchanged — this row. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC |
| **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CH02; class R-1384). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. **2026-10-25:** the kept-data choice no longer takes such a future-dated copy as the newest (R-1271, controller `bd9f4f7`); retention is unchanged — this row. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC |
| **R-1212** | Backup & restore | P4 | **Two household texts on the backup path do not follow the box's language:** the share-restore summary (`SharesRestoreResult.FlashMessage`, `controller/internal/backup/shares_restore.go`) is built from Hungarian Go literals, and the off-site `LastError` for an interrupted run is the Hungarian literal „megszakadt futás (a vezérlő újraindult futás közben)" (`controller/internal/backup/backup.go`). An English household reads Hungarian there. Found by the night helper 2026-10-21. | OPEN | — | Bundle keys for both (rendered in the box language when written, R-557), with parity fixtures | CC |
| **R-1221** | Backup & restore | P4 | **The nightly whole-box backup starts ~1.5–5 min later each night, until one night is skipped (after ~70–80 nights with no press between).** REAL, low harm (night 2026-10-22; split from R-1189 (b)): agent `localapi/server.go:1184` is `age >= Cadence` with the age counted from the archive START; the controller polls every 5 min (`quiesce.go:150`) and the archive starts after the apps stop. Measured on demo-hp: 04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40. At W+6h one night is skipped (44 h is below the 48 h valve, `quiesce.go:807-818`); the hub's 26 h line (`deadline.go:29`) mails the operator. Any manual or off-cycle run resets the drift. Evidence `audits/night-2026-10-22/P3-candidates/demohp-whole-box-completions.txt`. | OPEN | — | Fix: inside the night window a tier is due at `age >= cadence - 4h`; the agent answers due-ness, so it is agent code (a unit test with an injected clock) and rides an agent release | CC |
| **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator |
| **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC |
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place O1, listed from documents only (the off-site box was not read in this run — second pass). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EC23-7). | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB05). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place O1, listed from documents only (the off-site box was not read in this run — second pass). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EC23-7). | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
| **R-1362** | Backup & restore | P3 | **With the internet down, the Backups page takes over 2 minutes (135 s and 145 s measured; 5 s with only the hub down).** Walked 2026-11-03 on Tester 1: every other page answers at once on the home network. The page waits on something off-site inside its handler; a household that lost its line and opens exactly this page sees a hang. `audits/security-2026-11-02/part5/outage-wan.log` | OPEN — filed 2026-11-03 | — | name the wait with a timing, then give it a short deadline and show „the off-site state could not be read" | CC |
| **R-1374** | Backup & restore | P3 | **Four kinds of copy are kept with no end named, and nobody is reminded of them.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC02, EC23, EC18, EC27; read in source; 9202 has no off-site store, so not measured). (1) **Off-site snapshots of a removed app**: the removal keeps them by ruling (closed R-474), and the weekly clean-up keeps the last snapshots of each app for ever (`internal/backup/offbox_window.go:80-82`), also for an app that is gone. No rule says when they end. (2) **A whole restore from the second drive** moves the old files beside the restored ones (`internal/backup/tier2_whole.go:52`, `:132`, `:345`); nothing removes them later and no page lists them. (3) **Exported app files** stay in the export folder of the drive; the pages can start, list and read an export, none deletes one, and nothing ages it (`internal/web/handler_export.go:30-87`). **Members (answers marked N):** EC02-4, EC23-4, EC18-4, EC27-4. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D72 = yes, by the household's own choice; `09` §3 decision 292).** Not built. | — | Controller (D72): when the household pressed „delete backup data” for an app, its off-site copies end 30 days later, at the next clean-up after that day (the number of a restore copy, D62); when the household kept the backups, the off-site copies stay. The page shows the end date before anything is deleted — write that into `07-backup-architecture.md` before the build. Parts (2) and (3): a delete on the export page; a line on the kept-data page for the files a whole restore set aside. | CC |
| **R-1385** | Backup & restore | P3 | **Two timed jobs can meet with no lock and no order (12 answers).** Found 2026-11-11 (the Clock list, class W3, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED on the night of 2026-11-11: on Tester 1 and the N100 the whole-guest backup started at about 05:30 local, the minute of the off-site proof restore (CC09-5); on demo-hp one whole-guest backup started at 02:30:38, with the dump leg; on the N100 the whole-guest backup starts about 5 minutes later each day. From source: the controller's auto-update default is the minute the whole-guest gate opens (CC11-5); the second-drive copy and the abandon sweep wait only for the clock (CC02-5, CC19-5, CC21-5); the PBS verify can run beside a backup (CA13-5, CA04-5); the timers on DooPlex and ep0 are not ordered against the hub's copies (CH05-8, CH24-5, CH25-3, CH26-5, CH26-6; from documents only, not measured). No harm was seen; the durations are from small test stores. | OPEN — filed 2026-11-11 (clock list) | — | Measure one night with a large store on Tier 0 (as a task), then decide per pair: a shared flag, a fixed offset, or „harmless, written down”. | CC |
## Storage & devices — 15 rows (P3 6, P4 9)
@@ -255,13 +256,13 @@ stopping line that lies.
| **R-1258** | Security & access | P3 | **ep0's iperf3 line-test server takes any tunnel peer's test: one box (root on its host) can keep it busy for the fleet or ask a long reverse test that fills ep0's uplink.** Found 2026-10-24 (SEC-2 X2): `felhom.eu/scripts/ep0-iperf3/felhom-iperf3.service:17` — no authorized users, no server-side duration or rate cap; one client at a time (`felhom-agent/internal/linetest/iperf.go:263-264` → `busy`; 6 tries a day). Static (no ep0 change in this run). | OPEN | — | per-box iperf3 users or server-side caps, tried on a scratch VM first | CC |
| **R-1307** | Security & access | P3 | **Packages inside our own images are not watched** — the Debian packages of the controller image, the Alpine packages of the hub and Samba images (Samba's version is whatever Alpine 3.24 ships). The security watch (R-1303) reads only the base image's point release (BEHIND); Debian's DSAs and Alpine's secdb are not read. `audits/security-watch-2026-10-26/SPIKE.md` „cannot see" | OPEN — filed 2026-10-26 | — | add Alpine secdb (machine-readable JSON per release) and Debian's tracker for the packages the images install, or a scanner run on the built images in CI | CC |
| **R-955** | Security & access | P4 | **The cross-box check of the hub API is done in source and tests but not fully proven: if a later check masks a missing id check, a box's key could read or change another box's hub record without any test noticing.** Read 2026-10-12: every box-facing `/api/v1` route takes the box from its own key or refuses a different id; 14 routes already had a refusal test, 5 got one (`crossbox_refusal_test.go`). Left: red-prove the 4 new tests that were not red-proved (as the restore-directive one was), and the live read-only check with one demo box's key against the other demo box's id. | OPEN — filed 2026-10-12 by the security review | — | Red-prove the 4 tests; run the live GET check on the two demo boxes | CC |
| **R-924** | Security & access | P2 | **The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment.** MEASURED 2026-10-09: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` (0600 `kisfenyo`); `ssh-keygen -y -P ''` opens all three (no passphrase); `/mnt/5_hdd/felhom.eu` is in no backup set, the off-site job does not carry them. `04` §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; `04` §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). `runbooks/total-loss-of-dooplex.md` S6. | NARROWED — **2026-10-23 (D32):** S6a/S6b are on the current sheet. LEFT: the operator prints it. **NARROWED 2026-10-09 — operator ruling: option A** (print both AND copy them). The three key files and their public halves ride the nightly encrypted DooPlex→ep0 copy (`scripts/dooplex-offsite/`, a missing key refuses the push); the Sunday restore test proves each private key still derives its own public half (never printed). **LEFT:** the operator prints S6a/S6b (`runbooks/break-glass-sheet.md`); the keys still have no passphrase (custody ruling 1 of 2026-09-16 stands until the first sale). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G2):** the three key files are on `sda1`, the same disk as DooPlex's local backup tree; their only off-box copy is the S1-encrypted nightly one. Passphrase: unchanged by the custody ruling. | operator: print | Operator: print the two keys; then close | operator |
| **R-924** | Security & access | P2 | **The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment.** MEASURED 2026-10-09: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` (0600 `kisfenyo`); `ssh-keygen -y -P ''` opens all three (no passphrase); `/mnt/5_hdd/felhom.eu` is in no backup set, the off-site job does not carry them. `04` §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; `04` §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). `runbooks/total-loss-of-dooplex.md` S6. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB12). | NARROWED — **2026-10-23 (D32):** S6a/S6b are on the current sheet. LEFT: the operator prints it. **NARROWED 2026-10-09 — operator ruling: option A** (print both AND copy them). The three key files and their public halves ride the nightly encrypted DooPlex→ep0 copy (`scripts/dooplex-offsite/`, a missing key refuses the push); the Sunday restore test proves each private key still derives its own public half (never printed). **LEFT:** the operator prints S6a/S6b (`runbooks/break-glass-sheet.md`); the keys still have no passphrase (custody ruling 1 of 2026-09-16 stands until the first sale). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G2):** the three key files are on `sda1`, the same disk as DooPlex's local backup tree; their only off-box copy is the S1-encrypted nightly one. Passphrase: unchanged by the custody ruling. | operator: print | Operator: print the two keys; then close | operator |
| **R-132** | Security & access | P3 | **`curl -w '%{redirect_url}'` reconstructs the request URL WITH its basic-auth credential** — so a `-u ":$HUB_PW"` call that never put the password in a URL still printed it **Merged 2026-10-05 from R-350 (duplicate):** (1) 2026-08-20 occurrence: POST /configuration/artifacts answers 303; leak lives only in the CC transcript under ~/.claude/projects/ on DooPlex, not in git/evidence (checked then). (2) `-v` and `--libcurl` also re-render the credential, not only %{redirect_url}; confirm redirects with %{http_code} + follow-up GET. (3) Rotation path: hub /configuration form (current_password/new_password/confirm_password); DB override wins over ConfigMap (break-glass); CC can rotate file-to-file without printing (operator-present-one-time-secrets) if asked. | **NARROWED 2026-10-20 — the rotation is DONE:** the operator changed the hub password on 2026-10-15 and the `HUB_PW` line is gone (`audits/hub-tokens-2026-10-15/C153-live.txt:18`; CC uses its own token since hub 0.152.0). **Left, the operator's:** should the break-glass `password_hash` in `manifests/hub.yaml` (a public repository) be emptied or replaced? The DB override wins at login, so the old hash no longer opens the hub. **WAITING-ON-OPERATOR** — **ACTION: rotate `HUB_PW`** **Folded R-580 2026-10-03** (the same `curl -w %{redirect_url}` credential echo, seen again 2026-09-18). **Note 2026-10-12 (security review):** `manifests/hub.yaml` also carries the hub login's bcrypt hash in this public repository (the break-glass field); when `HUB_PW` is rotated, decide whether that field should stay empty. | the operator's word on the `manifests/hub.yaml` hash field | Happened on 2026-07-31 while red-proofing the R-120 gate: the hub operator password was written to the session transcript by the write-out format, not by the request. `-u` is safe; the *reporting* was not. Rule: read the redirect from `-D -` and grep `^Location:`, never `%{redirect_url}`, on any authenticated call. Rotate the hub password (`/configuration` → Login password; ConfigMap `auth.password_hash` is the reset path) and update `~/.config/credentials` | Viktor |
| **R-137** | Security & access | P3 | **Cloudflare geo-WAF rules are zone-scoped and non-namespaced — four cross-tenant faults.** `globalRuleDesc = "[felhom-geo] Global"` (`waf.go:18`) is one literal description per ZONE; `appRuleDescPrefix` keys by app name with no customer (`waf.go:21`); `BuildGlobalExpression` has no positive hostname scoping (`waf.go:241`); `applyDiff` deletes every `[felhom-geo]` rule not in THIS box's desired set (`geosync.go:320`) | READY (M) — **blocks shared-zone onboarding** | — | With two customers in one zone: they overwrite each other's Global rule forever; one customer's country policy applies zone-wide; per-app rules collide by name; and disabling the feature for one (or the hub's `RemoveGeoRules`) wipes them all. Interim mitigation, no code: keep geo-restriction OFF for every shared-zone customer. Fix = namespace descriptions by `customer_id` + add `http.host ends_with "<domain>"` to both expressions — a TWO-REPO change (controller + hub `RemoveGeoRules`). Same audit §5.1 | CC |
| **R-775** | Security & access | P3 | **[P2-MEDIUM] Grimmory: a stranger's 5 wrong sign-ins lock EVERY visitor out of the web login for 15 minutes — so Grimmory was not published.** MEASURED 2026-10-01 on 9202 (drill catalog, v3.4.1, through traefik): after 5 wrong tries for `admin` every further sign-in answered 429 — the household's right password AND a different name — and stayed 429 for 10+ minutes of retries. Read in the jar: `AuthRateLimitService` — Caffeine `expireAfterWrite(ofMinutes(15))`, `MAX_ATTEMPTS 5`, keys `login:ip:` and `login:user:`; Spring `forward-headers-strategy: native` takes the address from X-Forwarded-For, and behind the tunnel every visitor is the tunnel container's address (R-753) — the wger shape (R-752), with no setting to change it. Everything else in the checklist passed (bench + box step v3.4.1 → v3.5.0, gate by its own probe, OPDS through traefik); two smaller findings for the publishing session: on a reinstall over the first install's kept books, a new upload was saved to the drive but not added to the library (`box/grimmory/reinstall-c1.txt`, not investigated); and the remove + restore round trip (2.5) cannot be shown on 9202 for a drive app — its backup lives on the scratch drive, which is not a registered drive (R-756). The template waits in `audits/new-apps-2026-10-01/wip/grimmory/`. **Needs (operator):** (A) publish with a sentence on the page that wrong guesses by others can lock the login for 15 minutes (MEASURED: during the lock an e-reader's OPDS feed still answered 200 with its own login, wrong 401 — `box/grimmory/opds-under-lock.txt`), or (B) wait until the box passes each visitor's real address (R-753). `audits/new-apps-2026-10-01/box/grimmory/throttle.txt` **-- 2026-10-01 (evening):** option B's precondition SHIPPED (controller v0.286.1, R-753): Grimmory's Tomcat RemoteIpValve walks from the right and counts `172.16.0.0/12` as a proxy (READ in source, Spring Boot 4.1.1 — not yet measured with Grimmory's own lock), so `login:ip:` becomes per visitor; `login:user:` still lets a stranger lock the public name `admin` 15 min. A third route was spiked and passed: Grimmory behind the permanent family gate with its e-reader paths excepted (R-780). Recommendation: publish behind the family gate if R-780 is built; otherwise B with a measured 3.6. **UPDATE 2026-10-02 — NARROWED, Grimmory PUBLISHED behind the family gate:** a stranger cannot reach Grimmory's web sign-in at all (6 tries through the simulated tunnel: the gate's 401, then the household signs in 200 — `audits/family-gate-2026-10-02/A/items.txt`), and the e-reader exceptions keep Grimmory's own login. The 2.5 round trip now WORKS on 9202 (`audits/family-gate-2026-10-02/B/box/life.txt`). **What is left:** a family member past the gate can still lock a NAME (the admin's) for 15 minutes with 5 wrong tries — hard-coded in Grimmory; and the reinstall-over-kept-books finding (`new-apps-2026-10-01/box/grimmory/reinstall-c1.txt`) is still not investigated. | **WATCHING — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P2→P3: Grimmory is now behind the family gate; only a family member can still lock a name.** | — | — | CC |
| **R-908** | Security & access | P3 | **An old Resend API key is still in the history of the `homelab-manifests` repository; it was replaced on 2026-06-29, but whether it is also revoked at Resend is unknown.** FOUND 2026-10-08 by the contact-mailer source search (`audits/mailer-source-2026-10-08/SEARCH.md`, „A side finding"): commit c9648cd (2026-02-05, „added mailer pod") put a key literal in the old `felhom-system/contact-mailer.yaml` comment; the file was deleted in ee93b50 but the history keeps it. Compared by hash only, never printed: it is NOT today's key (`Secret/resend-api`, rotated 2026-06-29, felhom.eu feea0606). If the old key still works at Resend, anyone with read access to that repository can send mail as felhom.eu. | **OPEN — owner: operator** **Note 2026-10-12 (security review):** gitleaks 8.24.3 with its default rules did NOT flag this key in the full history scan, so a scanner gate alone would not have caught it (R-986). | — | In the Resend console, check that the old key is revoked (revoke it if not); rewriting the repository's history is NOT needed once it is revoked | operator |
| **R-525** | Security & access | P4 | **[P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured.** Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. **What it needs:** a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | **READY — rank P3-LOW; owner: CC (spike)** **Re-ranked 2026-10-03: P3->P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed.** | — | — | CC |
| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `<!-- source -->` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restarLine truncated
| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `<!-- source -->` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restarLine truncated
| **R-1243** | Security & access | P2 | **The hub's Longhorn backups sit unencrypted in the NFS folder any DooPlex pod can read (R-1241) — the two there now (2026-10-18, 2026-10-23) hold every household's Cloudflare and tunnel tokens in plaintext, because they predate hub 0.168.0.** Measured 2026-10-23 night (`audits/hub-token-seal-2026-10-23/` §3): the hub volume is `encrypted=false` and in Longhorn's `default` backup group (daily 04:00 + weekly Sunday 05:00, retain 1 each); the hub's own copy to ep0 is encrypted at rest (16/16 snapshots `crypt-mode: encrypt`, key on DooPlex's root disk and on paper) and in transit (SSH tunnel + pinned PBS TLS), every 6 h, restore-tested weekly. DECIDE: (a) an encrypted Longhorn volume for the hub (~5–10 min downtime, a homelab StorageClass + key in the cluster) or (b) take the hub volume out of the Longhorn backup — ep0's copy is the backup; the access log then has no backup until it rides the ep0 push. **CC's pick: (b).** And: delete the two plain copies (the operator's call; with (b) they never rotate out). | **VERIFY** — **2026-10-24: D40 put into effect** (operator ruling 07:22): the hub PVC `hub-data` carries `recurring-job-group.longhorn.io/default: disabled` (felhom.eu `3363ffa8`, ArgoCD sync of the PVC only) and the Longhorn volume `pvc-486c9809…` reads `disabled` too (labelled live to match, unchanged after 2 min); the NFS folder holds no copy of the volume (no `pvc-486c9809*` directory, no backup config naming it; Longhorn lists no backup volume for it; 53 other volumes backed up this morning). Closes after the dated check: tonight's 04:00 job makes no new hub backup while the others run. WAITING-ON-OPERATOR **2026-10-25 dated check DONE (read-only, the restore-truth session):** `kubectl -n longhorn-system get backups.longhorn.io` lists 106 backups made today (the newest 04:08:28Z — the job ran) and none of `pvc-486c9809` — the hub volume is out of the Longhorn backup as D40 meant. | the operator's pick; a homelab change for either option | Operator: pick (a) or (b), and say whether to delete the two plain copies after hub 0.168.0 is live | CC (the dated check) |
| **R-1355** | Security & access | P2 | **DooPlex's own Gitea 1.26.2 is inside 19 new notices (2026-11-02, fixed 28.3.0: `GHSA-8p3x` OAuth2 account takeover, `GHSA-37xv` and `GHSA-8g99` public-only tokens reach private data, `GHSA-32h8` Actions secrets) — on the host that holds our code and registry.** Also new on DooPlex: Vaultwarden 1.37.4 (fix 1.37.5), Uptime Kuma 2.5.6 (fix 2.5.9), Paperless 3.3.4 (fix 3.4.0). Felhom changes nothing on DooPlex. Read in `documentation/security-watch/state.json`, group DooPlex. | OPEN — filed 2026-11-03 | the homelab session | add them to `audits/http2shell-2026-10-27/HOMELAB-LIST.md`'s work; which notices reach a self-hosted Gitea with our settings is the homelab session's first read | operator (homelab session) |
| **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator |
@@ -274,12 +275,12 @@ stopping line that lies.
| **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator |
| **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC |
## Box system & updates — 18 rows (P3 13, P4 5)
## Box system & updates — 21 rows (P3 16, P4 5)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` **2026-11-11:** the Clock list measured it again (32.6 s behind, time service not running) and leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CA03-6; class R-1384). | OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC02, CC06-6, CC07-6, CC22-6). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
@@ -290,15 +291,18 @@ stopping line that lies.
| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **2026-10-24 02:4xZ dated check read (weekend-night session, read only):** Tester 1 (8 GB host, guest 4 GB, controller 0.341.0) ran its 03:30 copy at 01:30:00–01:30:13Z (3 apps, 157 MB + 86 MB + 16 KB; no „capped copy container did not start" WARN, no leftover `felhom-tier2-copy-*`); the guest cgroup's memory „full" stall counter on the host (`/sys/fs/cgroup/lxc/9201/memory.pressure`) read **total 444 µs** over the whole 11.1 h since the host started (15:32Z), so „full" stayed far under 20 % (20 % of 10 s = 2 s); 0 OOM kills. **Not shown:** that the copy ran in the capped container (the INFO log does not name it; Docker keeps only ~200 events, from 02:36Z). **Weak load:** 0.24 GB, not the soak's large copy — so the row stays WATCHING; close after a night whose copy moves ≥ 1 GB, read from the same counter. WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC |
| **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
| **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC |
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC |
| **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC |
| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. Decision D76 in `STATUS.md`. | OPEN — filed 2026-11-11 (end-dates list) | — | After D76: the agent reads the certificate file on its own host and takes the new fingerprint (design in `03-host-agent.md` first); prove on a scratch host with a forced renewal. | CC |
| **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC |
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. Decision D79 in `STATUS.md`. | OPEN — filed 2026-11-11 (clock list) | — | After D79: the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
## Monitoring & notifications — 26 rows (P2 3, P3 15, P4 8)
## Monitoring & notifications — 28 rows (P2 3, P3 17, P4 8)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -323,13 +327,15 @@ stopping line that lies.
| **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC |
| **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC |
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC |
| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` **2026-11-11:** the Clock list's class „nobody is told” (R-1383) is beside this row; it does not repeat it. | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC |
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. | OPEN — filed 2026-11-11 (end-dates list) | — | After D77: the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. Decision D78 in `STATUS.md`. | OPEN — filed 2026-11-11 (clock list) | — | After D78: one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
## Hub & operator — 16 rows (P2 1, P3 4, P4 11)
## Hub & operator — 17 rows (P2 1, P3 5, P4 11)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -344,11 +350,12 @@ stopping line that lies.
| **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC |
| **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator |
| **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |
| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CH23-6). | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
| **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC |
| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). **2026-11-10 read (the hub's start after a deploy, `audits/release-space-fixes-2026-11-10/hub/10-watch.txt`):** 11 slow lines in the first 2 minutes of the new pod, the longest 19.1 s (`GetHostReportsSince`), then the OS-report reads (3–4 s) and the two prunes (2.7 s, 4.2 s); none in the 28 minutes after. The 13:02 UTC snapshot's `VACUUM INTO` took 127.8 s. The same statements as the 11-02 read; no new one. | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC |
| **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC |
| **R-1376** | Hub & operator | P3 | **Records of a thing that is gone stay in six places.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH00, EH02, EH21, EA00, EA11, EA27; read in source, `documentation/audits/endings-list-2026-11-10/evidence/hub-delete-statements.txt`; no hub database was read). (1) **Hub:** thirteen tables have no delete statement anywhere. Deleting a box leaves its update settings, its update reports and its kernel notices; deleting a household leaves its off-site key rows, off-site window rows and abandon requests. Most are small; the update reports of each box grow with time. (2) **Host uninstall** leaves the mount units and host mounts of household drives (said when R-179 closed; no open row carried it). (3) **A drive wiped by a signed job** keeps a record and a mount unit that name the old filesystem (agent `internal/signedjobs/wipe.go:61-110`). (4) **The token of a destroyed guest** stays valid until a new guest takes its number: the token store has no remove (agent `internal/localapi/tokenstore.go:127-210`). **Members (answers marked N):** EH00-4, EH02-4, EH21-4, EA00-4, EA11-4, EA27-4. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D73 = yes, with one exception; `09` §3 decision 293).** Not built. | — | Hub (D73): add the tables with no delete to the two purges and to the daily clean-up; the delete journal (`host_deletions`, `customer_resets`) stays on purpose, and the privacy text says so. Agent: remove a guest's token when the guest is destroyed. Parts (2) and (3) have no ruling of their own. | CC |
| **R-1386** | Hub & operator | P3 | **Timed jobs with no time limit and no measured duration (25 answers).** Found 2026-11-11 (the Clock list, class W4, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source; the hub was not measured, by ruling). The hub's 60-second sweep runs its checks one after the other with no limit for each: one slow check delays all the others, the liveness check too (CH19-3, CH19-8, CH01-3, CH08-3, CH08-8, CH10-3, CH13-8, CH18-3, CH06-8, CH02-3); the 07:10 audit has 10 minutes, measured with three boxes only (CH11-3, CH11-8). On the box: the interval jobs have no limit of their own (CC11-3, CC12-3, CC13-3, CC14-3, CC15-3, CC16-3, CC20-3, CA06-3, CA10-3, CA10-8, CA13-8, CA15-3, CA19-3). Every measured duration is from a small test store. | OPEN — filed 2026-11-11 (clock list) | — | Hub: log each sweep step's duration and alarm when one sweep takes longer than its interval; then read a week of numbers and set limits. Box: the same for the controller's interval jobs. | CC |
## Business & legal — 14 rows (P2 4, P3 2, P4 8)
@@ -367,9 +374,9 @@ stopping line that lies.
| **R-930** | Business & legal | P3 | **The site's primary call-to-action fails WCAG AA: white on `--blue` #0083D8 measures 4.01:1 at 16px/600, against a 4.5:1 floor.** MEASURED 2026-10-10 in the browser, on the LIVE home page and on the redesign preview: identical, so this is **pre-existing and site-wide**, not a regression of the redesign. It is the „Jelentkezem tesztelőnek" button and every `.cta-button` on every page. `.cta-button-secondary` is fine (7.17:1). Three fixes, all needing the operator because each touches brand: (a) darken `--blue` until the ratio clears 4.5 (changes a token used across hub, controller and website); (b) make the label 700 weight at 18.66px+, which moves it into WCAG's „large text" band where 3:1 applies and 4.01 passes, at the cost of a bigger button everywhere; (c) accept it and record the decision. **Not changed unilaterally**: `felhom-web-design` §6 puts a new accent colour behind the operator's word. | **WAITING-ON-OPERATOR** — filed 2026-10-10 by the home-page redesign session; evidence `audits/redesign-2026-10-10/AUDIT.md` | operator's choice between (a), (b), (c) | pick one of the three | operator |
| **R-989** | Business & legal | P4 | **Operator: read `documentation/audits/selfhst-survey-2026-10-12/FELHOM.md` §6 and rule on each recommendation.** READ-ONLY research 2026-10-12: the selfh.st self-hosting survey 2023–2025 (1,869 / 3,655 / 4,081 respondents, hobby users) against our catalog, features, pitch and hardware. Ten ranked recommendations, none acted on; the top three: a supported-hardware page (none exists — finding, §4), off-site backup as a *why* reason in the pitch (only 25.2 % of the crowd backs up off-site), Syncthing as the largest catalog gap (25.9 %). Tables: `TABLE.md`; method and limits: `METHOD.md`. | **WAITING-ON-OPERATOR** — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages (the 2026 survey replaces it) | operator |
| **R-990** | Business & legal | P4 | **Operator: read `documentation/audits/forum-research-2026-10-12/FELHOM.md` §5 and rule on each recommendation.** READ-ONLY research 2026-10-12: three years (#7943–#11092, 2,908 posts, 195 people, 82 beginner threads) of the public PROHARDVER! topic „Home server / házi szerver építése”, tagged and counted by person (`FINDINGS.md`, method and spot-check in `METHOD.md`; no username or post text in git). Ten ranked recommendations, none acted on; the top three: a „which machine works” page (8 of 33 beginners who name their hardware own only a Synology, a router disk or a Raspberry Pi — the installer cannot run there; the same gap is R-989's first recommendation from a different source, and the shipped-page half is R-991), „you do not choose an operating system — we do” (the biggest beginner question, 20 of 82 threads), and „bring your old office PC” as the cheap entry. Catalog gap (≥ 3 people): a DLNA server (7), an ad/DNS blocker (5), a camera recorder (5). | **WAITING-ON-OPERATOR** — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages | operator |
| **R-1035** | Business & legal | P4 | **Did the „felhő tárhely” page earn anything? Two dated checks.** **Baseline (reported by the operator from Search Console, 2026-10-14, last 3 months):** 1 click, 48 impressions, CTR 2.1 %, average position **73.6**; „felhő tárhely” 19 impressions, „felhőtárhely” 5, „felhő alapú tárhely” 3. CC measured separately that the phrase „felhő tárhely” appeared **0 times** on the home, FAQ and backup pages — Google was showing the home page for those queries by accident. `/sajat-felho-tarhely` + `/en/own-cloud-storage` published **2026-10-14**; full baseline, price sources and the power provenance in `audits/seo-2026-10-14/BASELINE.md`. **What to measure, both times:** impressions and average position for „felhő tárhely” and „saját felhő tárhely”, plus the new page's own impressions, clicks and position. **A null result at +4 weeks means nothing on its own** — a new page on a site at position 73.6 takes weeks to months to move, and „felhő tárhely” alone is held by Microsoft, Google and the big list sites; the page aims at the two longer phrases. That is why there are two checks and not one, and why neither should be read as a verdict on the copy. **There is no Search Console connector in this project, so CC cannot read these numbers** — the operator reads them, CC records and compares. **Blocked until the operator asks for indexing** (Search Console → URL-ellenőrzés → the URL → „Indexelés kérése”): until that happens, or Google finds the page itself, both checks read zero for a reason that is not the page. **This is a slow channel and does not replace talking to people.** | **WATCHING** | operator: request indexing, then supply the Search Console read | 2026-11-11: first read (+4 weeks); 2026-12-09: second (+8 weeks), compared with the first | operator + CC |
| **R-1035** | Business & legal | P4 | **Did the „felhő tárhely” page earn anything? Two dated checks.** **Baseline (reported by the operator from Search Console, 2026-10-14, last 3 months):** 1 click, 48 impressions, CTR 2.1 %, average position **73.6**; „felhő tárhely” 19 impressions, „felhőtárhely” 5, „felhő alapú tárhely” 3. CC measured separately that the phrase „felhő tárhely” appeared **0 times** on the home, FAQ and backup pages — Google was showing the home page for those queries by accident. `/sajat-felho-tarhely` + `/en/own-cloud-storage` published **2026-10-14**; full baseline, price sources and the power provenance in `audits/seo-2026-10-14/BASELINE.md`. **What to measure, both times:** impressions and average position for „felhő tárhely” and „saját felhő tárhely”, plus the new page's own impressions, clicks and position. **A null result at +4 weeks means nothing on its own** — a new page on a site at position 73.6 takes weeks to months to move, and „felhő tárhely” alone is held by Microsoft, Google and the big list sites; the page aims at the two longer phrases. That is why there are two checks and not one, and why neither should be read as a verdict on the copy. **There is no Search Console connector in this project, so CC cannot read these numbers** — the operator reads them, CC records and compares. **Blocked until the operator asks for indexing** (Search Console → URL-ellenőrzés → the URL → „Indexelés kérése”): until that happens, or Google finds the page itself, both checks read zero for a reason that is not the page. **This is a slow channel and does not replace talking to people.** **2026-11-11:** the 4-week check came due today in a run with no operator present; only the operator can read Search Console, so the date moved to 2026-11-18 (the gate's rule allows a move with a reason). The 8-week check keeps its date. | **WATCHING** | operator: request indexing, then supply the Search Console read | 2026-11-11: first read (+4 weeks); 2026-12-09: second (+8 weeks), compared with the first | operator + CC |
## Process & tooling — 32 rows (P3 8, P4 24)
## Process & tooling — 34 rows (P3 8, P4 26)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -415,8 +422,10 @@ stopping line that lies.
|---|---|---|
| R-1293 | 2027-03-29 | the spring clock change: the hub's 02:00 snapshot and the 02:30 push on 2027-03-28 (02:00–03:00 does not exist) — which file each push took; detail in the R-1293 row |
| R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row |
| R-1035 | 2026-11-11 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row |
| R-1035 | 2026-11-18 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row |
| R-528 | 2026-11-12 | no failed oom_check / unexplained app_oom since 2026-10-12; detail in the R-528 row |
| R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row |
<!-- DUE-CHECKS-END -->
| **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC |
| **R-1381** | Process & tooling | P4 | **No document says when eleven things end, or where to read the date (11 answers).** Found 2026-11-11 (the End-dates list, class T3, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; search S18 found no document that lists end dates). OS support for Debian 13 and Proxmox 9 (DB17-2, DB17-3), the tunnel client's oldest allowed version (DB19-2), the installer ISO (DH19-3), and the outside accounts and tokens (DH23-2, DH24-2, DH27-2, DH28-2, DH29-2, DH31-2, DH32-2). This run could not read these dates (no account call, by its fences). The operator reads them once: the table „Only you can read these dates” in `documentation/audits/end-dates-list-2026-11-11/README.md`. | OPEN — filed 2026-11-11 (end-dates list) | — | When the operator has read the dates: write them in one place (a short table in `documentation/operations/`), and give the due-checks block the ones inside a year. | CC |
| **R-1387** | Process & tooling | P4 | **A document and the source disagree on a time, in four places (4 answers).** Found 2026-11-11 (the Clock list, class W5, `documentation/audits/clock-list-2026-11-11/TABLE.md`). (1) `documentation/architecture/03-host-agent.md` names `Restart=always` and a watchdog for the agent's unit; the unit has `Restart=on-failure` and no watchdog (CA18-6, CA18-8). Which one is the design is not stated, so this session did not edit either. (2) The hub-database copy alarm: 13 hours in one place, 26 in another (CH23-7; a DooPlex procedure, listed for the operator, not edited). (3) Old „24 h / 02:30” text for the hub's daily job (CH17-6). **Seen beside the lists, same kind, not in the count:** (4) the comment at `felhom-controller/controller/cmd/controller/main.go:1412` says 14 days where the code keeps restore copies 30 days (D62); (5) `documentation/architecture/07-backup-architecture.md:455-457` cites `main.go` lines that have moved; (6) `documentation/operations/tailscale.md` says „both nodes” where three test hosts exist, and Tailscale was not running on demo-hp and Tester 1 on 2026-11-11 (not known if that is meant); (7) the threat model and the recovery inventory do not say the same about how often the hub bearer is rotated. | OPEN — filed 2026-11-11 (clock list) | — | Operator or a task: say which side is right for (1) and (2); then one docs commit. (3) is a text fix in the same commit. | CC |
+4
View File
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-11-11 (midday)
**Updated 2026-11-10 (evening).** Three releases are live: the agent that cleans its own update folder (you vouched it), the controller that sends one warning for one full disk (every box takes it by itself), and the hub that sends the host-disk alarm to you only. You ruled D71–D75 at 15:19 and none is built; the smallest system disk is 200 GB, and what the installer does below 200 GB (refuse, or warn) is open (R-1370). The report of the run is `REPORT-release-space-fixes-2026-11-10.md`.
## Moved from STATUS on 2026-11-10 (evening)
**Updated 2026-11-10 (late afternoon).** You ruled D71–D75 at 15:19 (`documentation/architecture/09-update-architecture.md` §3, decisions 291–296); none is built. The smallest system disk is 200 GB, and what the installer does below 200 GB (refuse, or warn) is open (R-1370). The report of this run is `REPORT-release-space-fixes-2026-11-10.md`.