diff --git a/CONTEXT.md b/CONTEXT.md index 1f44b9eb..45104a32 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,11 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-23 morning — the operator's „go" for the day brief: the releases, D31 as built, D32–D36 (decide rows in `STATUS-HISTORY.md`).** +> **Releases:** controller 0.340.0 (global floor 06:41 UTC, MinAgent 0.158.1); agent 0.163.0 (binary sha256 561b5316…, bundle 3afb88d9…, vouched by the operator 07:58 UTC; delivered to the three test boxes by signed jobs through a step bundle `0.163.0-step1` — the D4 drop-in is a new path, R-880); installer 1.36.0 (tag `installer-v1.36.0` → 83388c08, both webpage pins); hub 0.167.0 (sha256:177860a9…, 08:27 UTC); the website `0bb57a4f`. +> **D31** = as built (the re-issue reads the newest deletion of a KEY-HOLDING host; an install that got the password but no recovery package is rotated without a press). **D32** = yes: the sheet is names and places only, one A4, HU + EN; the operator prints and fills it. **D33** = yes: `scripts/release-copy/` nightly 01:40, newest 3 image tags (controller, hub, samba) + 5 agent versions, skopeo `dir:` (digest-true — why not `docker save`: a re-push re-compresses), ep0 `/srv/felhom-release-copy` under the one limited user `felhom-relcopy` (rrsync), cap 3 GB enforced by the sender, a read-only registry token of its own. **D34** = yes: hub snapshots at 02/08/14/20 (catch-up > 6 h), DooPlex push :30 after, limit 7 h + a wait for a snapshot being written (R-1234: 2 h was wrong), alarm 13 h; ep0 retention unchanged (keep-daily 14, keep-weekly 8). **D35** = yes, after the release week (R-1228, dated 2026-10-30). **D36** = yes, both sentences, published. +> **Added on the operator's word the same day (not soaked):** D34's hub half went into 0.167.0 (asked before the build). Record: `audits/day-2026-10-23/`, `REPORT-day-2026-10-23.md`. + > **2026-10-22 evening — operator rulings D29, D30, D31 (CC's picks accepted; the decide rows now in `STATUS-HISTORY.md`).** > **D29** = no: the controller repository stays public for now (R-925). **D30** = the agent makes a new WireGuard key and > registers it on the existing peer path (R-1204). **D31** = the hub skips a deleted install that never made a key when it diff --git a/REPORT-day-2026-10-23.md b/REPORT-day-2026-10-23.md new file mode 100644 index 00000000..e8d5fdee --- /dev/null +++ b/REPORT-day-2026-10-23.md @@ -0,0 +1,103 @@ +# REPORT — day 2026-10-23: the releases and the DooPlex hour (the operator's „go") + +Design read: `07-backup-architecture.md` §6.3/§8, `09-update-architecture.md` §3 decision 250, `03-host-agent.md`, +`05-hub-architecture.md` §16.3, `runbooks/total-loss-of-dooplex.md`, `audits/dooplex-survival-2026-10-22/PLAN.md`. +Evidence: `documentation/audits/day-2026-10-23/01`–`14`. + +## Five lines + +1. Released: controller 0.340.0 (global floor), agent 0.163.0 (vouched by the operator), installer 1.36.0, hub 0.167.0, + the website with D36's two sentences. +2. Root SSH from the LAN is key-only on all three test boxes; new installs get it from installer 1.36.0. +3. The hub database now leaves every 6 hours (restored back byte-identical); the released images and agent packages + leave nightly to ep0 (pulled back, every sha256 OK, one image loaded on a scratch box). +4. One false failure mail at 10:28 (my install of the 6-hourly copy) — filed R-1234, fixed and red-proved the same hour. +5. Register 191 → 184; the operator: print the sheet, Hosts page clean-up, D37 + D38. + +## Per box — versions before → after, and the SSH checks + +| Box | controller | agent | root-file bundle | `sshd -T` root | sshd reloaded | key login | felhom-sshd | swap = RAM (9201) | +|---|---|---|---|---|---|---|---|---| +| Tester 1 | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 (via 0.163.0-step1) | without-password | yes, 09:27:38 | ok | active, `sshd -t` ok | 4096/4096 | +| demo-hp | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:19:44 | ok | active, `sshd -t` ok | 25898/25898 | +| demo-felhom | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:24:42 | ok | active, `sshd -t` ok | 12288/12288 | +| hub | 0.165.0 → 0.167.0 | | | | | | | | + +Not proven (said plainly): the real root password over SSH (CC does not hold it — a dummy one is refused, but OpenSSH logs +only „Failed password"); a felhom-sshd connection (it admits only the operator's tunnel peer); the physical console (the +drop-in does not touch it; `passwd -S root` = P). + +## Digests, sha256, CI by commit + +| What | Value | +|---|---| +| controller 0.340.0 | `sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621` — `caaa86e`, CI 2245 success | +| agent 0.163.0 binary / bundle | `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c` / `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8` — `1109092` CI 2246, `b4721ee` CI 2249 success | +| step bundle 0.163.0-step1 | `6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf` | +| installer 1.36.0 (served = tag) | `de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec` — `83388c08` CI 2247 success; pins `2aebf1bd` | +| hub 0.167.0 | `sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0` — built at `d306b8c2` (hub/ = `be7e97dc`, CI 2250 success; `d306b8c2` CI 2251 success); manifest `34979d5d` | +| website | `0bb57a4f` | +| D34 fix (R-1234) | `52af38df` | +| prometheus rules | homelab-manifests `1bbef892` | + +CI of the last commits (`2aebf1bd`, `0bb57a4f`, `34979d5d`, `52af38df` and this close-out): see the final message. + +## Parts + +- **1 — controller 0.340.0:** 9202 smoke, per-customer floors, three boxes green, box walk + https-only rc 0, global floor + 06:41 UTC (MinAgent 0.158.1), read back with the WAL (`01`–`06`). +- **2 — agent + installer:** release, a step bundle because the D4 drop-in is a new path (R-880), three signed jobs per + box (`07`); the operator's vouch read back from the DB and the live page (`10`); installer tag, both pins, a + webpage-only sync, the served script = the tag (`11`). +- **3 — hub 0.167.0:** built from a clean clone; on the operator's word it also carries D34's 6-hourly snapshot (asked + before the build; not soaked); hub-only sync; 30-minute watch: 0 hub events, 0 errors, all boxes reporting, no hub mail + (`14`). +- **4 — website:** D3/D5 patch + D36 HU + EN; site gates; published; read back live: new text present, old gone (control: + the previous commit), screenshots at 390 and 1366 px, no sideways scroll (`12-website-readback/`). +- **5 — D34:** retention unchanged, server-side on ep0 (`prune-operator-hubdb` 03:45, keep-daily 14, keep-weekly 8 — the + current day's up to 4 copies until the next prune). Hand run 501.6 MiB / 24.5 MiB uploaded in 4 s; the newest copy + restored from ep0 with the read-only token = the pod's snapshot by sha256 `a9ff4c8a…`, integrity ok, 4 hosts, holds + today's vouch. Alarm 13 h. Runbook updated (`13`). +- **6 — D33:** ep0 before: `/` 14 GB free (the data disk's root IS the PBS datastore — not used). One user + `felhom-relcopy` (locked, rrsync-confined: shell and `..` refused), one directory `/srv/felhom-release-copy`. Cap 3 GB, + enforced by the sender (ep0 has no quota). **skopeo, not `docker save`:** `dir:` keeps the registry's bytes, so a push + back keeps the digest; no Docker daemon. A read-only registry token of its own, minted in the Gitea pod straight into a + 0600 file. First run 676 MB, 91 s; pulled back: 141 files OK, 9 images re-verified (digests match independent records), + controller 0.340.0 loaded on 9202 (ID = config digest). Alarm `FelhomReleaseCopyStale`. Restore steps in + `total-loss-of-dooplex.md` (`08`). +- **7 — D32:** the sheet is names and places only, HU + EN, one A4 (rendered: 1 page; a copy in the operator's home + folder); every place checked to exist. demo-hp: three superseded ISOs removed by name, `/` 8.5 → 14 GB free (`09`). + R-548: the box half done, the product fix proposed (R-921's pattern). +- **8 — close-out:** CONTEXT rulings, STATUS (decide rows D32–D36 → history, D37/D38 new, the do list, D35 dated + 2026-10-30), REPORTs and CHANGELOGs; the system poster's facts file updated, its hub-DB label edited (text), and a new + arrow asked for in STATUS („System poster needs a refresh"). + +## Register — 191 → 184; opened 4, closed 11 + +Opened: R-1234 (closed the same hour), R-1235, R-1236, R-1237. Closed: R-49, R-982, R-985, R-991, R-1204, R-1205, +R-1225, R-1226, R-1227, R-1230, R-1234. Updated: R-1214 (released; dated check 2026-10-24), R-1187, R-1189, R-232, +R-548, R-923, R-924, R-1228 (D35, dated 2026-10-30), R-1183's check text (four snapshot slots). + +## Observations + +- The D34 first push refused the hub's own catch-up and mailed admin@ — FILED: R-1234 (closed, fixed `52af38df`). +- The release copy is proven to load, not to restore a registry; it follows the newest versions, not the floor/vouch; no + golden — FILED: R-1235. +- Gitea OOM-killed at its 1 GiB limit, the last at 06:41:10 UTC during the release — FILED: R-1236 (STATUS D37). +- The night sweep's mail: one-time test-box codes in one session record — FILED: R-1237. +- The hub DB snapshot is 501 MiB, not 24 MB — NOT-A-FINDING: 24 MB is the compressed upload, which is what ep0 stores. +- felhom-sshd does not answer from the LAN — NOT-A-FINDING: its firewall admits only the operator peer (H1, by design). +- A dummy root password logs „Failed password", not a policy refusal — NOT-A-FINDING: OpenSSH's behaviour under + `prohibit-password`; the policy is proven by `sshd -T` and the reload. +- The vouch left `artifact_wrapper_sha256` empty — NOT-A-FINDING: an optional field; the wrapper rides the vouched bundle. +- Guests 9202, 9401 (demo-hp) and a transient 990000 (Tester 1) have swap ≠ RAM — NOT-A-FINDING: outside the felhom pool. +- `known_hosts` holds an older key for 192.168.0.155 (demo-hp's guest address) — NOT-A-FINDING: guests are reached by + `pct exec`; nothing was changed. +- The D4 drop-in needed a step bundle — NOT-A-FINDING: the documented R-880 path for a new bundle path. + +## Teardown — three layers + +Machine: 9202 — the test image untagged, the tar removed. Host: demo-hp `/tmp` tar removed; three ISOs removed on purpose +(Part 7); DooPlex: the read-back copies and the hub-DB scratch restore shredded/removed. Hub: nothing provisioned; the +per-customer floors cleared after the global floor. Left on purpose: the ep0 user + directory, the DooPlex release-copy +job, the 6-hourly hub-DB timer, both alarms. diff --git a/STATUS.md b/STATUS.md index e7df5bc1..1a0f3298 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night PASSED (no failed check at night, every night job passed, nothing lost). Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35). +**Updated 2026-10-23 (day).** Released on your go: controller 0.340.0 (the floor), agent 0.163.0 (vouched; root SSH by key only on the three test boxes), installer 1.36.0, hub 0.167.0, and the website with your two sentences. The hub database now leaves every 6 hours and the released images and agent packages every night, both read back from ep0; the break-glass sheet is current — please print it. One false failure mail reached you at 10:28 (my install of the 6-hour copy, fixed in the same hour). ## What works - A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4). @@ -21,7 +21,7 @@ - The scratch box cannot use agent features (R-1083). ## What we work on -- Dated checks, run by CC. 2026-10-23: no app stop when the host is busy (R-1173). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts). +- Dated checks, run by CC. 2026-10-24: the 8 GB box's first night copy on 0.340.0 (R-1214). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes). ## Waiting on you — decide @@ -29,11 +29,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| -| D36 | R-1225, R-1226 | Two website sentences before the publish: „8 GB runs the basics; the photo app needs 16 GB” and „what you open from outside the home passes Cloudflare's network, readable there; the backups leave only encrypted”? | **Yes, both** (the patch's own aim is the whole truth) | — | the patch publishes with both promises a little too strong | — | -| D35 | R-1228 | After the release week, serve the website and the installer from Cloudflare, like the ISO? | **Yes, later** (no new installs stop when DooPlex is down) | keep them home: after a DooPlex loss, no install until it is back | the same | — | -| D34 | R-232 | Copy the hub database to ep0 every 6 hours instead of every night? | **Yes** (0 €, about 30 minutes, one DooPlex change) | keep nightly: up to a day of hub changes lost, e.g. a new box's recovery package | the same | — | -| D33 | R-1227 | Copy the released images and agent packages to ep0 every night? | **Yes** (0 €, about 2 hours of CC, a few GB on ep0) | a timed rebuild drill instead: about 3 hours once, and rebuilt images are not the same | after a DooPlex loss no box updates, reinstalls or installs until everything is rebuilt | — | -| D32 | R-923, R-924 | Print the break-glass sheet now (eight keys and logins + ep0's fingerprint) and keep it outside the home? | **Yes** (30 minutes; nothing else opens the off-site copies after a loss) | an encrypted USB stick outside the home: ~10 €, 1 hour, one more passphrase | after a DooPlex loss no off-site copy opens | — | +| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | +| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — | | D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — | | D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — | @@ -48,17 +45,14 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14. 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12. -**Morning, 2026-10-23 — in this order.** -1. Hold or go: CC says go — the test night passed on a fresh 8 GB box (no failed check at night, every night job passed, nothing lost; one gap: no whole-box copy to ep0, R-1232). One question from the review: D31 as built also covers an install that got the whole-box password but no recovery package — its password is rotated without your press (R-1205). CC: go. If not: revert one hub commit before item 4. -2. Release controller 0.340.0 (main after tonight; tested as rc.8 on the scratch box, rc.6 on the test box): the safe night copy, the catch-up fix, R-49, R-1187, R-1189, R-1214. If not: these wait. -3. Release the agent (the tunnel repair, its tunnel field, the root-SSH bundle) and the installer tag with the root-SSH removal (R-1204, R-982); then vouch. If not: a clock-ahead box stays cut off; root keeps its password on the LAN. -4. Build and deploy hub 0.167.0 (D31, the tunnel alarm and its clearing). If not: a dead tunnel stays silent. -5. Publish the website patch (D3, D5), with D36's two sentences if yes. If not: the old wording stays. -6. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef (tonight's test box). If not: two dead entries alarm. -7. The DooPlex hour: D32–D35 in the table. If not: a DooPlex loss stays as the survey describes. +**Day, 2026-10-23 — your do list.** +1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens. +2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef. If not: two dead entries alarm. +3. D37 and D38 in the table (two small yes/no). If not: Gitea may restart in release hours; no golden in the release copy. **The rest** +- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy. - Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12. - Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09. - Two leftovers on DooPlex: the Facebook `first-post/` folder in `leftovers/2026-10-14/` (keep, commit or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). If not: they stay. Since 2026-10-14. -- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784); print the break-glass sheet and the signing keys (R-923, R-924). +- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784);. - Later (yours, not urgent): R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902. diff --git a/documentation/architecture/felhom-system-poster.facts.md b/documentation/architecture/felhom-system-poster.facts.md index 4867f46a..452cb1b0 100644 --- a/documentation/architecture/felhom-system-poster.facts.md +++ b/documentation/architecture/felhom-system-poster.facts.md @@ -101,6 +101,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1– - one namespace **per household** (whole-guest copies, each household's own key), with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.* - the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys). + - outside PBS, a plain folder: the **release copy** (images + agent packages, D33). Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly. Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data." - **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys. @@ -121,8 +122,9 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1– Arrows from DooPlex (dashed, orange → ep0): - **00:20 "DooPlex copy"**: the code (614 MB), Gitea's database and settings, the **password manager**, the nightly secrets export, the **signing keys**. Encrypted on DooPlex with the **DooPlex off-site key**. The key that writes it **cannot delete** old copies. Tested back every Sunday. -- **02:30 "Hub database"**: encrypted with the **hub-DB key**. Tested back every Sunday at 04:30. -- **Left out on purpose:** the container registry (27.7 GB) — "rebuilt from the code". +- **02:30, 08:30, 14:30, 20:30 "Hub database"** (every 6 hours since 2026-10-23, D34): encrypted with the **hub-DB key**. Tested back every Sunday at 04:30. +- **01:40 "Release copy"** (since 2026-10-23, D33): the newest 3 released images of the controller, hub and samba, and the newest 5 agent packages — not encrypted (nothing secret), into a plain folder on ep0 (`/srv/felhom-release-copy`, its own limited user, outside PBS), about 0.7 GB, capped at 3 GB. +- **Left out on purpose:** the rest of the container registry (older tags, test builds, non-Felhom images) — "rebuilt from the code". Arrow ep0 → DooPlex (dashed, 05:00): "**ep0-copy** pull: a second copy of ep0 at home. Keeps 8 weekly copies. A daily job removes a deleted customer's copy within 30 days." **The test machines** (small, plain tiles near DooPlex, labelled "disposable"): diff --git a/documentation/architecture/felhom-system-poster.html b/documentation/architecture/felhom-system-poster.html index 048a22be..3ab0e626 100644 --- a/documentation/architecture/felhom-system-poster.html +++ b/documentation/architecture/felhom-system-poster.html @@ -133,7 +133,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
The hub never connects into a box.
00:20 DooPlex copy
DooPlex off-site key
-
02:30 hub database
+
every 6 h: hub database
hub-DB key
05:00 ep0-copy pull
ciphertext only
diff --git a/documentation/audits/day-2026-10-23/01-9202-smoke-0340.txt b/documentation/audits/day-2026-10-23/01-9202-smoke-0340.txt new file mode 100644 index 00000000..fea6dc34 --- /dev/null +++ b/documentation/audits/day-2026-10-23/01-9202-smoke-0340.txt @@ -0,0 +1,6 @@ +== 9202 smoke 06:37:25 +switched to gitea.dooplex.hu/admin/felhom-controller:0.340.0 at 06:37:28 +healthy after ~10s: Up 7 seconds (healthy) +2026/10/23 06:37:28 [INFO] [tier2] start-up: no leftover second-drive copy container +errors, unhealthy+restarting: 0 0 +login page: 200 diff --git a/documentation/audits/day-2026-10-23/02-per-customer-floors.txt b/documentation/audits/day-2026-10-23/02-per-customer-floors.txt new file mode 100644 index 00000000..1b1e3611 --- /dev/null +++ b/documentation/audits/day-2026-10-23/02-per-customer-floors.txt @@ -0,0 +1,3 @@ +tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set +demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set +demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set diff --git a/documentation/audits/day-2026-10-23/03-release-0340-boxes.txt b/documentation/audits/day-2026-10-23/03-release-0340-boxes.txt new file mode 100644 index 00000000..303d46c5 --- /dev/null +++ b/documentation/audits/day-2026-10-23/03-release-0340-boxes.txt @@ -0,0 +1,15 @@ +### root@192.168.0.154 +containers=10 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 53 seconds (healthy) +errors=0 +copy-containers=0 +2026/10/23 06:38:21 [INFO] [tier2] start-up: no leftover second-drive copy container +### hp +containers=24 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 54 seconds (healthy) +errors=0 +copy-containers=0 +2026/10/23 06:38:22 [INFO] [tier2] start-up: no leftover second-drive copy container +### felhom-pve +containers=9 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 58 seconds (healthy) +errors=0 +copy-containers=0 +2026/10/23 06:38:05 [INFO] [tier2] start-up: no leftover second-drive copy container diff --git a/documentation/audits/day-2026-10-23/04-https-only.txt b/documentation/audits/day-2026-10-23/04-https-only.txt new file mode 100644 index 00000000..8682d689 --- /dev/null +++ b/documentation/audits/day-2026-10-23/04-https-only.txt @@ -0,0 +1,33 @@ +control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -) +== tester-1 (enkicsifelhom.hu): 6 public name(s) + OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1 + OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 + OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1 + OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1 + OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1 +== demo-hp (enkisfelhom.hu): 14 public name(s) + OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1 + OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1 + OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1 + OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1 + OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1 + OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1 + OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 + OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1 + OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1 + OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1 + OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1 + OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1 + OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1 +== demo-felhom (demo-felhom.eu): 8 public name(s) + OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1 + OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1 + OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1 + OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 + OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1 + OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1 + OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1 + OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1 +== every public name redirects http -> https diff --git a/documentation/audits/day-2026-10-23/05-files-link-and-copy.txt b/documentation/audits/day-2026-10-23/05-files-link-and-copy.txt new file mode 100644 index 00000000..b0f26a77 --- /dev/null +++ b/documentation/audits/day-2026-10-23/05-files-link-and-copy.txt @@ -0,0 +1,13 @@ +== tester-1 (dashboard login with the household password, the press a browser makes) +python3: can't open file '/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/38fde0c6-e964-4ec1-b0c7-56e62d675225/scratchpad/n23/fileswalk.py': [Errno 2] No such file or directory +== demo-hp (no session): GET /__files/open -> HTTP/2 302 location: /login +== demo-felhom (no session): GET /__files/open -> HTTP/2 302 location: /login +== tester-1 06:40:35 +Fájlok press: GET /__files/open -> ['HTTP/2 302'] https://files.enkicsifelhom.hu/_felhom/sso +manual second-drive copy: 200 2. mentés elindítva +2026/10/23 06:40:43 [INFO] [backup] Tier 2 copied bookstack → /mnt/felhom-drives/adatlemez/backups/secondary/bookstack (157.0 MB, 0 leg(s), 1s) +2026/10/23 06:40:46 [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx (85.9 MB, 1 leg(s), 2s) [SSD: state-only] +2026/10/23 06:40:48 [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/adatlemez/backups/secondary/privatebin (16.4 KB, 0 leg(s), 0s) +2026/10/23 06:40:48 [INFO] [backup] Tier 2 run complete: 3 app(s) processed (incl. volume-only — F6) +copy-containers-left=0 +[exited with code 0] diff --git a/documentation/audits/day-2026-10-23/06-global-floor.txt b/documentation/audits/day-2026-10-23/06-global-floor.txt new file mode 100644 index 00000000..042cc4c3 --- /dev/null +++ b/documentation/audits/day-2026-10-23/06-global-floor.txt @@ -0,0 +1,5 @@ +== global floor 06:41:04: HTTP/1.1 303 See Other Location: /configuration?flash=floor_set +clear tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set +clear demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set +clear demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set +read back (with WAL): [('min_controller_version', '0.340.0', '2026-10-23 06:41:04'), ('min_controller_version_declared_min_agent', '0.340.0=0.158.1', '2026-10-23 06:41:04')] diff --git a/documentation/audits/day-2026-10-23/07-agent-0163-boxes.txt b/documentation/audits/day-2026-10-23/07-agent-0163-boxes.txt new file mode 100644 index 00000000..bba54d37 --- /dev/null +++ b/documentation/audits/day-2026-10-23/07-agent-0163-boxes.txt @@ -0,0 +1,78 @@ +# Part 2 — agent 0.163.0 (operator go 2026-10-23), read 2026-10-23T07:28:53Z + +## release (scripts/release-agent.sh at felhom-agent 1109092, CI 2246 green; CHANGELOG b4721ee CI 2249 green) +[release-agent] built ok: sha256 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c +[release-agent] config bundle built: sha256 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 +[publish-agent] upload OK (HTTP 201) +[publish-agent] round-trip GET verified (sha256 matches) +[release-agent] config bundle published (HTTP 201) + sha256 : 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c + bundle : 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 (felhom-config-bundle.json — vouch it with the agent) + * [new tag] v0.163.0 -> v0.163.0 + +## step bundle (R-880: the D4 drop-in is a NEW path; the boxes ran bundle 0.160.0 = 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f) +0.163.0-step1 = base 0.160.0 with only /usr/local/sbin/felhom-os-apply replaced; sha256 6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf; PUT 201; read back by download: same sha + +## delivery: three signed jobs per box (felhom-opsign, key felhom-op-1): agent_update 06:48Z, agent_config_update 0.163.0-step1 06:59Z, agent_config_update 0.163.0 07:19Z + +## hp (192.168.0.104) 2026-10-23T07:26:29Z +agent: felhom-agent 0.163.0 +bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:19:44Z + 1 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok + 2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok +guest 9201: memory: 25898 swap: 25898 +guest 9202: memory: 25898 swap: 512 +guest 9401: memory: 10240 swap: 0 +drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:19 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf +drop-in content: PermitRootLogin prohibit-password +sshd -T: permitrootlogin without-password +root password set (passwd -S): P +felhom-sshd: active; its root policy: permitrootlogin without-password +felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 } +main sshd reloaded after the drop-in: Oct 23 09:19:44 demo-hp sshd[1010]: Received SIGHUP; restarting. Oct 23 09:19:44 demo-hp systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server. +pveproxy: active +main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4 +key login: ok +root password attempt rc=255 (non-zero = refused) +Oct 23 09:26:37 demo-hp sshd-session[3026730]: Failed password for root from 192.168.0.180 port 34150 ssh2 +## root@192.168.0.154 (192.168.0.154) 2026-10-23T07:28:00Z +agent: felhom-agent 0.163.0 +bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:27:38Z + 2 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok + 2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok +guest 9201: memory: 4096 swap: 4096 +guest 990000: memory: 4096 swap: 512 +drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:27 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf +drop-in content: PermitRootLogin prohibit-password +sshd -T: permitrootlogin without-password +root password set (passwd -S): P +felhom-sshd: active; its root policy: permitrootlogin without-password +felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 } +main sshd reloaded after the drop-in: Oct 23 09:27:38 felhom sshd[890]: Received SIGHUP; restarting. Oct 23 09:27:38 felhom systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server. +pveproxy: active +main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4 +key login: ok +root password attempt rc=255 (non-zero = refused) +Oct 23 09:28:08 felhom sshd-session[4184378]: Failed password for root from 192.168.0.180 port 33442 ssh2 +## felhom-pve (192.168.0.162) 2026-10-23T07:28:11Z +agent: felhom-agent 0.163.0 +bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:24:43Z + 2 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok + 2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok +guest 9201: memory: 12288 swap: 12288 +drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:24 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf +drop-in content: PermitRootLogin prohibit-password +sshd -T: permitrootlogin without-password +root password set (passwd -S): P +felhom-sshd: active; its root policy: permitrootlogin without-password +felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 } +main sshd reloaded after the drop-in: Oct 23 09:24:42 demo-felhom sshd[1011]: Received SIGHUP; restarting. Oct 23 09:24:42 demo-felhom systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server +pveproxy: active +main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4 +key login: ok +root password attempt rc=255 (non-zero = refused) + +## notes +- felhom-sshd :8822 admits only the operator peer 10.77.0.250 over wg-felhom (H1); a connection test from DooPlex times out BY DESIGN — proven: active, listening, sshd -t ok, firewall set. +- a dummy root password is logged as 'Failed password' (OpenSSH refuses a root password silently under prohibit-password); the policy proof is sshd -T + the SIGHUP reload after the drop-in. The real root password was not tried (CC does not hold it). +- guests 9202, 9401 (demo-hp) and the transient 990000 (Tester 1, gone a minute later) are outside the felhom pool; swap = RAM applies to pool guests (9201 on all three: swap = memory). diff --git a/documentation/audits/day-2026-10-23/08-release-copy-D33.txt b/documentation/audits/day-2026-10-23/08-release-copy-D33.txt new file mode 100644 index 00000000..0a9d038f --- /dev/null +++ b/documentation/audits/day-2026-10-23/08-release-copy-D33.txt @@ -0,0 +1,53 @@ +# Part 6 — D33: the released images and agent packages to ep0 nightly (operator yes 2026-10-23), 2026-10-23T07:51:56Z + +## ep0 — free space first, then ONE user and ONE directory (nothing else on ep0 changed) +before: /dev/sda1 38G 23G used 14G free 63% (/mnt/pbs-datastore is the PBS datastore root — not used) +created: user felhom-relcopy (system, uid 999, password locked 'L', shell /bin/sh for the forced command), /srv/felhom-release-copy (0750) with data/ and .ssh/authorized_keys: + command="/usr/bin/rrsync /srv/felhom-release-copy/data",restrict ssh-ed25519 +fence test: rsync --list-only OK; a shell command → 'rrsync error: SSH_ORIGINAL_COMMAND does not run rsync'; '..' → 'do not use .. in arg' +after the first copy: /dev/sda1 38G 24G used 13G free 65%; /srv/felhom-release-copy/data 677M +cap: CAP_MB=3072, enforced by the sender (ep0's root fs has no quota; a quota would change more than one directory) + +## DooPlex — the job (felhom.eu d306b8c2): /usr/local/sbin/felhom-release-copy + .service + .timer (01:40), /etc/felhom-release-copy/{env,id_ed25519,known_hosts,registry-token} +registry token: a NEW Gitea access token for user admin, name felhom-release-copy-d33, scope read:package ONLY — minted inside the gitea pod straight into /etc/felhom-release-copy/registry-token (0600 root); never printed +known_hosts: ep0's ed25519 host key, SHA256:NGtcS5v3K+9UjMTM6X5Ib0CoChdhqj9bezLz1hOUqgY — equal to the key already pinned for root@ep0 +skopeo, not docker save: skopeo dir: keeps the registry's manifest bytes and compressed blobs, so a push back gives the SAME digest (docker save re-compresses → a new digest); no Docker daemon needed + +## the first run (systemctl start, under the unit's sandbox) +2026-10-23T09:48:33+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-controller: keeping 0.338.0, 0.339.0, 0.340.0 +2026-10-23T09:48:37+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-hub: keeping 0.164.0, 0.165.0, 0.166.0 +2026-10-23T09:48:43+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-samba: keeping 1.0.0, 1.1.0, 1.2.0 +2026-10-23T09:48:47+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-agent: keeping 0.159.2, 0.160.0, 0.161.0, 0.162.0, 0.163.0 +2026-10-23T09:48:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: staged and verified: 676 MB, new this run: felhom-controller:0.338.0, felhom-controller:0.339.0, felhom-controller:0.340.0, felhom-hub:0.164.0, felhom-hub:0.165.0, felhom-hub:0.166.0, felhom-samba:1.0.0, felhom-samba:1.1.0, felhom-samba:1.2.0 +2026-10-23T09:49:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: pushed to ep0 and compared by checksum: no difference (91 s) +2026-10-23T09:49:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: success signal written +2026-10-23T09:49:51+02:00 dooplex systemd[1]: felhom-release-copy.service: Consumed 30.416s CPU time, 743.1M memory peak. + +## read back from ep0 (rsync pull with the job key → /var/lib/felhom-release-copy/readback) +sha256sum -c SHA256SUMS: every file OK (141 files), 677 MB +every image re-verified (top manifest = its digest, every blob = its name): + felhom-controller 0.338.0 sha256:e8a529f395bfe8c0e7b87d5d98929a6f8e9834aa13c0ab77bd214ee9e7e53433 + felhom-controller 0.339.0 sha256:fee71aafceb4a7d5f24ae4a6a273a8d51eb26e0550fe9d12d4d86955f6a343a8 + felhom-controller 0.340.0 sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621 + felhom-hub 0.164.0 sha256:931594e333a4e8a9784eff2fb144d022ab46f89dd6a865a93e6100dc28dbd9dc + felhom-hub 0.165.0 sha256:c6180fb79a2ff7d5546a704c68b634b9207afb0a0ecd524c74b0310434532537 + felhom-hub 0.166.0 sha256:829667bb289c790bc99719e88dad8f7d08faaa9f662df1e9a06b1774915a1af6 + felhom-samba 1.0.0 sha256:0fa9fae3eaa9d48836c855a4884f0751a90ff050dc9e7d17e41d93bb434499c0 + felhom-samba 1.1.0 sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10 + felhom-samba 1.2.0 sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9 + agent 0.159.2 binary 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9 bundle 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9 + agent 0.160.0 binary 7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1 bundle 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f + agent 0.161.0 binary ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5 bundle 485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47 + agent 0.162.0 binary 5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928 bundle ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5 + agent 0.163.0 binary 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c bundle 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 +independent controls (other channels): controller 0.340.0 = the release digest from build.sh (03-release-0340-boxes.txt); hub 0.165.0 = the running hub pod's imageID; hub 0.164.0 = hub/CHANGELOG; agent 0.163.0 = release-agent.sh today; agent 0.161.0 = night2-2026-10-20/P1/1b; bundle 0.160.0 = the three boxes' installed bundle_sha256 + +## load on a scratch machine (9202 on demo-hp, Tier 0) +the ep0 copy of felhom-controller 0.340.0 → skopeo docker-archive (amd64) → pct push 9202 → docker load: + Loaded image: felhom-restorecopy-test:0.340.0; ID=sha256:3f34d74519463c338422594b70c31d9c5beabed48847134c6d26b5a95ebbae2b = the copy's amd64 config digest; a container ran (ls /) + teardown: image untagged (0 left), the tar removed on 9202, on demo-hp /tmp and on DooPlex + +timer: Sat 2026-10-24 01:42:16 CEST 15h - - felhom-release-copy.timer felhom-release-copy.service + +## alarm (homelab-manifests 1bbef892): FelhomReleaseCopyStale (> 26 h or absent, warning) — promtool check: 7 rules OK; promtool test: quiet at 25 h, fires at 27 h +synced ConfigMap mon-system/prometheus-rules only; POST /-/reload 200; /api/v1/rules: FelhomReleaseCopyStale inactive ok; metric age at read ~3235 s diff --git a/documentation/audits/day-2026-10-23/09-demohp-space.txt b/documentation/audits/day-2026-10-23/09-demohp-space.txt new file mode 100644 index 00000000..e264ac0b --- /dev/null +++ b/documentation/audits/day-2026-10-23/09-demohp-space.txt @@ -0,0 +1,10 @@ +no guest references the three +BEFORE: /dev/mapper/pve-root 39G 29G 8.5G 77% / +Removed volume 'local:iso/felhom-installer-1.27.1-pve9.2-1.iso' +removed local:iso/felhom-installer-1.27.1-pve9.2-1.iso +Removed volume 'local:iso/felhom-installer-1.28.0-pve9.2-1.iso' +removed local:iso/felhom-installer-1.28.0-pve9.2-1.iso +Removed volume 'local:iso/felhom-installer-1.29.0-pve9.2-1.iso' +removed local:iso/felhom-installer-1.29.0-pve9.2-1.iso +AFTER: /dev/mapper/pve-root 39G 24G 14G 64% / +local:iso/felhom-installer-1.30.0-pve9.2-1.iso iso iso 1705324544 diff --git a/documentation/audits/day-2026-10-23/10-vouch-readback.txt b/documentation/audits/day-2026-10-23/10-vouch-readback.txt new file mode 100644 index 00000000..7b735c27 --- /dev/null +++ b/documentation/audits/day-2026-10-23/10-vouch-readback.txt @@ -0,0 +1,16 @@ +# The vouch read-back — agent 0.163.0 (the operator vouched on the hub, answered 'Vouched now'), read 2026-10-23T08:00:42Z + +## channel 1: hub DB (hub.db + -wal + -shm in ONE tar from the pod), hub_settings +artifact_agent_sha256|561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c|2026-10-23 07:58:46 +artifact_agent_version|0.163.0|2026-10-23 07:58:46 +artifact_bundle_sha256|3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8|2026-10-23 07:58:46 +artifact_golden_sha256|3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f|2026-10-23 07:58:46 +artifact_golden_version|0.331.0|2026-10-23 07:58:46 +artifact_min_agent|0.158.1|2026-10-23 07:58:46 +artifact_wrapper_sha256||2026-10-23 07:58:46 + +## channel 2: the running hub's operator page GET /configuration (CC's token), the selected options +value="0.163.0" data-sha="561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c" selected>0.163.0 +value="0.331.0" data-sha="3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f" selected>0.331.0 + +agent 0.163.0 sha256 = the release (561b5316…); bundle sha256 = the release (3afb88d9…). Golden 0.331.0 unchanged. The floor's MinAgent stays 0.158.1 (artifact_min_agent 0.158.1; controller 0.340.0 declares 0.158.1). diff --git a/documentation/audits/day-2026-10-23/11-installer-1360.txt b/documentation/audits/day-2026-10-23/11-installer-1360.txt new file mode 100644 index 00000000..afea1c90 --- /dev/null +++ b/documentation/audits/day-2026-10-23/11-installer-1360.txt @@ -0,0 +1,13 @@ +# Installer 1.36.0 published (operator yes 2026-10-23), read 2026-10-23T08:16:46Z + +1. bump: felhom.eu 83388c08 (SCRIPT_VERSION 1.36.0; scripts/test_hostinstall.py 40 passed; CI 2247 success) +2. tag: installer-v1.36.0 (annotated d00f868c) -> 83388c08, pushed +3. pins: manifests/webpage.yaml both --ref=installer-v1.36.0 (sidecar git-sync-scripts + init git-sync-scripts-init), commit 2aebf1bd; + ArgoCD: only Deployment felhom-system/felhom-webpage was OutOfSync; synced that ONE resource at revision 2aebf1bd; rollout ok; + live args: git-sync-scripts-init ['--ref=installer-v1.36.0'], git-sync-scripts ['--ref=installer-v1.36.0'], website git-sync ['--ref=main'] (unchanged) +4. read back: curl https://felhom.eu/scripts/felhom-host-install.sh + SCRIPT_VERSION="1.36.0" # the SINGLE version source (F-1): -h and the run banners follow it. + sha256 served = de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec + sha256 tag file = de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec + the drop-in name 10-felhom-root-key-only appears 2 times (the uninstall's removal) +Rollback: move the tag back to installer-v1.35.0's commit and wait ~30 s (no sync). diff --git a/documentation/audits/day-2026-10-23/12-website-readback/README.txt b/documentation/audits/day-2026-10-23/12-website-readback/README.txt new file mode 100644 index 00000000..e1799100 --- /dev/null +++ b/documentation/audits/day-2026-10-23/12-website-readback/README.txt @@ -0,0 +1,4 @@ +# Website publish read-back (felhom.eu 0bb57a4f, operator yes 2026-10-23) +Method: headless Chromium (playwright 1.61.1 image) on the LIVE site; the FAQ answer opened by a click; 390x844 (phone) and 1366x900 (computer). +No page scrolls sideways at either size (scrollWidth == viewport on all 10 shots). +Text: see text-check.txt (new sentences present; the old 'only encrypted (the remote backup)' gone; control: HEAD~1 had it twice). diff --git a/documentation/audits/day-2026-10-23/12-website-readback/computer-biztonsagimentes.png b/documentation/audits/day-2026-10-23/12-website-readback/computer-biztonsagimentes.png new file mode 100644 index 00000000..c1a55aaa Binary files /dev/null and b/documentation/audits/day-2026-10-23/12-website-readback/computer-biztonsagimentes.png differ diff --git a/documentation/audits/day-2026-10-23/12-website-readback/computer-en-download.png b/documentation/audits/day-2026-10-23/12-website-readback/computer-en-download.png new file mode 100644 index 00000000..6bf3dca8 Binary files /dev/null and b/documentation/audits/day-2026-10-23/12-website-readback/computer-en-download.png differ diff --git a/documentation/audits/day-2026-10-23/12-website-readback/computer-en-faq.png b/documentation/audits/day-2026-10-23/12-website-readback/computer-en-faq.png new file mode 100644 index 00000000..c84b3dae Binary files /dev/null and b/documentation/audits/day-2026-10-23/12-website-readback/computer-en-faq.png differ diff --git a/documentation/audits/day-2026-10-23/12-website-readback/computer-gyik.png b/documentation/audits/day-2026-10-23/12-website-readback/computer-gyik.png new file mode 100644 index 00000000..8eb662dc Binary files /dev/null and b/documentation/audits/day-2026-10-23/12-website-readback/computer-gyik.png differ diff --git a/documentation/audits/day-2026-10-23/12-website-readback/computer-letoltes.png b/documentation/audits/day-2026-10-23/12-website-readback/computer-letoltes.png new file mode 100644 index 00000000..cc44cdf7 Binary files /dev/null and b/documentation/audits/day-2026-10-23/12-website-readback/computer-letoltes.png differ diff --git a/documentation/audits/day-2026-10-23/12-website-readback/phone-biztonsagimentes.png b/documentation/audits/day-2026-10-23/12-website-readback/phone-biztonsagimentes.png new file mode 100644 index 00000000..020ee3d4 Binary files /dev/null and b/documentation/audits/day-2026-10-23/12-website-readback/phone-biztonsagimentes.png differ diff --git a/documentation/audits/day-2026-10-23/12-website-readback/phone-en-download.png b/documentation/audits/day-2026-10-23/12-website-readback/phone-en-download.png new file mode 100644 index 00000000..1e4c0b23 Binary files /dev/null and b/documentation/audits/day-2026-10-23/12-website-readback/phone-en-download.png differ diff --git a/documentation/audits/day-2026-10-23/12-website-readback/phone-en-faq.png b/documentation/audits/day-2026-10-23/12-website-readback/phone-en-faq.png new file mode 100644 index 00000000..579644bb Binary files /dev/null and b/documentation/audits/day-2026-10-23/12-website-readback/phone-en-faq.png differ diff --git a/documentation/audits/day-2026-10-23/12-website-readback/phone-gyik.png b/documentation/audits/day-2026-10-23/12-website-readback/phone-gyik.png new file mode 100644 index 00000000..5cb4fd0a Binary files /dev/null and b/documentation/audits/day-2026-10-23/12-website-readback/phone-gyik.png differ diff --git a/documentation/audits/day-2026-10-23/12-website-readback/phone-letoltes.png b/documentation/audits/day-2026-10-23/12-website-readback/phone-letoltes.png new file mode 100644 index 00000000..63491a4a Binary files /dev/null and b/documentation/audits/day-2026-10-23/12-website-readback/phone-letoltes.png differ diff --git a/documentation/audits/day-2026-10-23/12-website-readback/text-check.txt b/documentation/audits/day-2026-10-23/12-website-readback/text-check.txt new file mode 100644 index 00000000..4f04c23a --- /dev/null +++ b/documentation/audits/day-2026-10-23/12-website-readback/text-check.txt @@ -0,0 +1,7 @@ +waited ~0 s +NEW letoltes 'az alapokat futtatja' count 1 (want 2 on the FAQ: visible + JSON-LD; 1 on download) +NEW en/download 'runs the basics; the photo app needs 16 GB' count 1 (want 2 on the FAQ: visible + JSON-LD; 1 on download) +NEW gyik 'a Cloudflare h' count 4 (want 2 on the FAQ: visible + JSON-LD; 1 on download) +NEW en/faq "passes through Cloudflare's network and is readable there" count 2 (want 2 on the FAQ: visible + JSON-LD; 1 on download) +OLD gyik 'otthonodat (a t' live 0 (want 0); control HEAD~1 2 (want 2) +OLD en/faq 'leaves your home only encrypted (the remote backup)' live 0 (want 0); control HEAD~1 2 (want 2) diff --git a/documentation/audits/day-2026-10-23/13-hubdb-6h-D34.txt b/documentation/audits/day-2026-10-23/13-hubdb-6h-D34.txt new file mode 100644 index 00000000..90cecf34 --- /dev/null +++ b/documentation/audits/day-2026-10-23/13-hubdb-6h-D34.txt @@ -0,0 +1,43 @@ +# Part 5 — D34: the hub DB off-site every 6 hours (operator yes 2026-10-23), 2026-10-23T08:41:15Z + +## what changed +- hub 0.167.0 (deployed 08:27Z, image sha256:177860a9…): snapshots at 02:00 08:00 14:00 20:00 Budapest; start-up catch-up when the newest > 6 h. Hub log at start: + 2026/10/23 10:28:12 [INFO] db-snapshot 08:00: next run at 2026-10-24 08:00 CEST (in 21h31m48s) + 2026/10/23 10:28:12 [INFO] db-snapshot 14:00: next run at 2026-10-23 14:00 CEST (in 3h31m48s) + 2026/10/23 10:28:12 [INFO] db-snapshot 20:00: next run at 2026-10-23 20:00 CEST (in 9h31m48s) + 2026/10/23 10:28:12 [INFO] db-snapshot 02:00: next run at 2026-10-24 02:00 CEST (in 15h31m48s) + 2026/10/23 10:30:06 [INFO] db snapshot written: hub-20261023T082812Z.db (525996032 bytes, 1m54.537s); pruned 1, keeping 2 +- DooPlex felhom-hub-db-backup (felhom.eu 52af38df): timer 02:30 08:30 14:30 20:30; waits ≤ 20 min for a snapshot being written; refuses a newest snapshot older than 7 h +- R-1234 (closed): the first version (2 h, no wait, be7e97dc) refused the timer's own catch-up at 10:28 CEST and mailed admin@ once — fixed in 52af38df + +## retention (ep0, server-side, unchanged): prune-job prune-operator-hubdb, ns operator, 03:45, keep-daily 14, keep-weekly 8 +=> the current day's copies (up to 4) stay until the next 03:45 prune, then the newest of each day for 14 days and one per week for 8 weeks. Size per copy: 501.6 MiB raw, 24.5 MiB uploaded (compressed, dedup'd chunks). + +## run once by hand (systemctl start felhom-hub-db-backup.service, 10:39 CEST) +2026-10-23T10:39:33+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: snapshot hub-20261023T082812Z.db, 11 min old +2026-10-23T10:39:45+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: checked: 525996032 bytes, integrity ok, 4 host(s) +2026-10-23T10:39:48+02:00 dooplex felhom-hub-db-backup[2377041]: hubdb.pxar: had to backup 501.629 MiB of 501.629 MiB (compressed 24.471 MiB) in 2.47 s (average 203.437 MiB/s) +2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2377041]: Duration: 3.21s +2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: pushed hub-20261023T082812Z.db to ep0 (ns operator) in 4 s +2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: success signal written + +## read back: restore the NEWEST copy from ep0 with the READ-ONLY token into a scratch dir (shredded after) +15 copies on ep0, oldest 2026-10-10T00:30:56Z +newest copy on ep0: host/dooplex-hub/2026-10-23T08:39:45Z +Using encryption key from '/etc/felhom-hub-backup/enc.key'.. +Fingerprint: b2:19:bf:36:3b:97:3d:6c +restore complete (501.629 MiB processed in 5.4s, average 92.519 MiB/s) +restored size: 525996032 bytes +restored sha256: a9ff4c8a0d0fe34da72411d9c165e573069f5601492d1e625578464f56adbde7 +integrity_check: ok +SELECT COUNT(*) FROM hosts: 4 +SELECT value FROM hub_settings WHERE key='artifact_agent_version': 0.163.0 +pod snapshot hub-20261023T082812Z.db: 525996032 bytes, sha256 a9ff4c8a0d0fe34da72411d9c165e573069f5601492d1e625578464f56adbde7 (kubectl exec sha256sum — a different channel from the restore) +=> byte-identical; it holds today's vouch (artifact_agent_version 0.163.0, set 07:58Z) — a nightly copy could not + +## timers +Fri 2026-10-23 14:30:08 CEST 3h 48min Fri 2026-10-23 10:28:30 CEST 12min ago felhom-hub-db-backup.timer felhom-hub-db-backup.service + +## alarm (homelab-manifests 1bbef892): HubDBBackupStale 26 h -> 13 h (two missed 6-hourly pushes) +promtool test: quiet at 12 h, fires at 14 h; RED-PROOF: the rule at 26 h fails the 14 h case +in force: /api/v1/rules shows the 13 * 3600 expression, inactive ok; metric age at read ~237 s diff --git a/documentation/audits/day-2026-10-23/14-hub-0167-watch.txt b/documentation/audits/day-2026-10-23/14-hub-0167-watch.txt new file mode 100644 index 00000000..4aa41d8f --- /dev/null +++ b/documentation/audits/day-2026-10-23/14-hub-0167-watch.txt @@ -0,0 +1,21 @@ +# Part 3 — hub 0.167.0 deploy and the 30-minute watch (operator yes 2026-10-23), read 2026-10-23T08:56:31Z + +build: clean clone at d306b8c2 (hub/ unchanged since be7e97dc, CI 2250 success; d306b8c2 CI 2251 success before the deploy); hub tests rc 0 (28 packages) +image: gitea.dooplex.hu/admin/felhom-hub:0.167.0 sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0 +manifest 34979d5d; ArgoCD: only Deployment felhom-system/hub OutOfSync; synced that one resource; rollout ok 08:27:34Z; pod imageID = the digest above; Synced/Healthy + +## watch 08:27:34Z → 08:56:31Z — channel 1: the hub DB (with WAL) + log +events since the deploy: 0 (newest event 07:07:10Z Tester4 host_down — the expected dead test box; 161 events since 2026-10-22: the table is live) +host report|demo-felhom-8363b5|2026-10-23 08:54:54|2 +host report|demo-hp-bb76ea|2026-10-23 08:49:41|2 +host report|tester-1-d70be4|2026-10-23 08:42:35|1 +controller report|demo-felhom|2026-10-23 08:53:20|3 +controller report|demo-hp|2026-10-23 08:53:24|3 +controller report|tester-1|2026-10-23 08:53:23|3 +hub log since deploy: 0 [ERROR]; 'osupdates: alarm check done … 0 alarm(s) sent'; 'Host staleness checker initialized: 3 ok, 1 already alarmed (seeded silently, R-1001)' — Tester4 not re-mailed + +## channel 2: the operator mailbox (Gmail connector, to:admin@felhom.eu newer_than:1d) +after 08:27Z: ONE mail, 08:28:32Z '[DooPlex backup] FAILED: felhom-hub-db-backup.service' — NOT from the hub: CC's install of the 6-hourly timer (R-1234, closed, fixed 52af38df). No hub mail after the deploy. +before the deploy (control that the channel delivers): 07:07:11Z Tester4 host_down, 07:04:11Z node_down, 06:22Z/06:19Z stale — the hub's mails do arrive + +verdict: no new false alarm from the hub in the 30 minutes after the deploy; one CC-caused DooPlex mail, filed and fixed diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 7337ec9e..c0afa505 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -30,6 +30,16 @@ | Row | What | Closed | Evidence | |---|---|---|---| +| **R-49** | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** (P3) | CLOSED 2026-10-23 — released in controller **0.340.0** (D21 = B: a template may exclude a named volume; Immich's model cache is out of every backup); the global floor 0.340.0 (MinAgent 0.158.1) since 06:41 UTC, all three test boxes on it | `audits/day-2026-10-23/` 03, 06 | +| **R-1204** | **A box that runs once with its clock ahead loses its WireGuard tunnel to ep0 — the PBS whole-box copies and the operator's door — until the real date passes the wrong one, and nothing says why.** (P2) | CLOSED 2026-10-23 — the sign reported and the heal (D30, the agent re-keys on the existing peer path) in agent **0.163.0** (vouched by the operator 07:58 UTC; on Tester 1, demo-hp, demo-felhom by signed jobs), the alarm and its clearing in hub **0.167.0** (deployed 08:27 UTC). The live heal was proven on the test night (`audits/night-2026-10-23/P4-testnight/`) | `audits/day-2026-10-23/` 07, and the hub watch in 14 | +| **R-1205** | **After a box death, a SECOND install that is deleted before it makes a key leaves the whole-box (PBS) tier waiting for the operator's re-issue.** (P2) | CLOSED 2026-10-23 — D31 in hub **0.167.0** (deployed 08:27 UTC, image sha256:177860a9…): the re-issue reads the newest deletion of a host that held a key | `audits/day-2026-10-23/` 14 | +| **R-1230** | **A reinstalled box that waits for the household's recovery code was reported „stuck” after an hour: a false operator mail, a needless re-issue of the off-site login, and a controller restart on the box.** (P3) | CLOSED 2026-10-23 — hub **0.167.0** (deployed 08:27 UTC): the delivery checker stands down for `awaiting_recovery_key` | `audits/day-2026-10-23/` 14 | +| **R-982** | **The Proxmox host accepts root password login over SSH from the LAN (measured on demo-hp), so anyone on the household network can try the root password.** (P3) | CLOSED 2026-10-23 — D4 delivered: the bundle drop-in in agent **0.163.0** (config bundle sha256 3afb88d9…, via a step bundle 0.163.0-step1 because it adds a path, R-880) on all three test boxes — `sshd -T` `permitrootlogin without-password`, sshd reloaded after the drop-in, a key login works, felhom-sshd active; new installs: installer **1.36.0** published (tag `installer-v1.36.0`, served sha256 de8bf5f3… = the tag). The real root password was not tried over SSH (CC does not hold it); the console is untouched by design | `audits/day-2026-10-23/` 07, 11 | +| **R-985** | **Our zero-knowledge wording is stronger than the product: on a live box the operator can reach the off-site backup key (through the vaulted root credential and the operator tunnel) without the customer's recovery code.** (P3) | CLOSED 2026-10-23 — the D3 wording published on the website (felhom.eu `0bb57a4f`), read back live at phone and computer size | `audits/day-2026-10-23/` 12-website-readback | +| **R-991** | **Nothing a buyer can read says which machines Felhom installs on, and a quarter of the forum beginners own only machines it cannot install on.** (P3) | CLOSED 2026-10-23 — the D5 minimum machine published (felhom.eu `0bb57a4f`), read back live | `audits/day-2026-10-23/` 12-website-readback | +| **R-1225** | **The website's minimum machine (8 GB) cannot run the photo app, and the download page does not say so.** (P3) | CLOSED 2026-10-23 — D36: „8 GB runs the basics; the photo app needs 16 GB” on both download pages, published `0bb57a4f`, read back live | `audits/day-2026-10-23/` 12-website-readback | +| **R-1226** | **Two FAQ answers say the household's data leaves the home only encrypted (the remote backup) — not true for data reached from outside.** (P2) | CLOSED 2026-10-23 — D36: both FAQ answers (visible + JSON-LD, HU + EN) now say what is opened from outside passes Cloudflare's network and is readable there; the backups leave only encrypted. Published `0bb57a4f`; live: the new sentence 2× per FAQ page, the old one 0× (control: the previous commit had it 2×) | `audits/day-2026-10-23/` 12-website-readback | +| **R-1227** | **The container registry and the agent packages exist only on DooPlex — after its loss no box updates, reinstalls or installs.** (P2) | CLOSED 2026-10-23 — D33 built (felhom.eu `d306b8c2`): nightly 01:40 the newest 3 release tags of controller, hub, samba (skopeo `dir:`, digest-true) and the newest 5 agent versions to ep0 `/srv/felhom-release-copy` (one new user `felhom-relcopy`, rrsync-confined; cap 3 GB by the sender); first run 676 MB in 91 s; pulled back, every sha256 OK, every image re-verified; controller 0.340.0 from the copy loaded on 9202 (ID = config digest); alarm `FelhomReleaseCopyStale`; restore steps in `total-loss-of-dooplex.md`. Follow-ups: R-1235 | `audits/day-2026-10-23/` 08-release-copy-D33.txt | | **R-1234** | **D34's first push refused the hub's own catch-up and mailed a false „hub-db-backup failed" to admin@.** Installing the 6-hourly timer at 10:28 made systemd catch up the missed 08:30 slot at once (`Persistent=true`), while the freshly deployed hub 0.167.0 was still writing its start-up snapshot (`hub-*.db.tmp`); the only finished one was 02:00 (8 h), above the new 2-hour limit → „the hub stopped snapshotting", `OnFailure` mail id `01a167e0…`. The 2 h limit was also stricter than the hub's own 6-hour catch-up, so any restart could repeat it. (P3) | CLOSED 2026-10-23 day, the same session — the push waits (≤ 20 min) for a snapshot being written, and the limit is one slot gap + 1 h (7 h); tests `test_d34_waits_for_a_snapshot_being_written` (reproduces the 10:28 run when the wait is removed), `…_a_restart_within_one_slot_is_not_an_alarm`, `…_two_missed_slots_are_refused`, each red-proved | `audits/day-2026-10-23/13-hubdb-6h-D34.txt` | ## 2026-10-22 night — the seven decisions put into effect (full text of the rows: `git show 1100a344:documentation/backlog/OPEN-ITEMS.md`) diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 4d4f1d46..fff4272c 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -119,7 +119,6 @@ stopping line that lies. | **R-980** | Install & onboarding | P3 | **A person cannot check that the installer is ours: the ISO's first boot runs the install script from a movable tag with no hash or signature, and the ISO checksum sits on the same website as the download.** Fix: a signed manifest (or signature) checked by the ISO before it runs the script, and the public key printed in the guide. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC | | **R-250** | Install & onboarding | P3 | **A customer create can fail fail-closed because the host-key scan ladder is shorter than the DNS/AAAA settle time.** Found 2026-08-07 creating the fifth walk's venue. `POST /configs/new` with off-site enabled provisions the Storage Box sub-account and then scans its SSH host key to pin it — **fail-closed by design** (`offsite.go:111-121`, *"don't serve a descriptor the controller can't verify"*), with `defaultScanBackoff` = 2+4+8+16+30 ≈ **60 s**, sized by its own comment to *"the observed DNS propagation lag"*. **Measured, both halves:** the first create exhausted the ladder — five `no such host`, then `dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachable` (the name had just begun resolving, **AAAA-first, into a pod with no IPv6 route**) — and the hub logged `[ERROR] offsite provision for walk5`. An **identical second POST succeeded** ~70 s later on its own final rung (`shared already provisioned … (subaccount 285351)`). **Total settle ≈ 100 s against a 60 s budget.** The endpoint was never the problem: verified afterwards from both the node and the hub pod, by name, `SSH-2.0-OpenSSH_9.6p1`. **Two distinct things are wrong and should not be merged:** (1) the budget is sized against DNS *existence*, but what actually bit is the **AAAA-before-A** window, a different and longer phenomenon — lengthen the ladder and/or prefer the A record for the scan dial; (2) **the operator is told nothing actionable** — the create fails with a generic error, the remedy is "press it again", and nothing says so. The retry is genuinely safe (idempotent on the label; confirmed afterwards — `one_time_secrets` 1, one sub-account, no double-provision) **but that safety is invisible to the person deciding whether pressing again will double-charge them.** **Severity LOW-MEDIUM:** self-clearing, no data at risk — but it is the first thing a new customer's provisioning does and it fails looking like an outage. | **VERIFY** — **2026-10-12: hub 0.147.0 live** (S29, decision 224): the host-key scan dials IPv4 first, waits ~2 min (was ~1), and when it still runs out says the store was created and it is safe to press Save again (`TestScannerDialsIPv4First`, `TestScanOutOfRetriesSaysSafeToPressAgain`, red-proved). Known limit: the hub Ingress reads for 60 s, so a very slow first try can show a gateway timeout while provisioning finishes in the background (pressing again is safe). **READY** — operator ruling 2026-10-12 (S29, `09` §3 decision 224): yes — the scan prefers IPv4, waits longer, and the error says „safe to press again". **READY** — owner Viktor | — | The next real off-site setup (a new customer): read the hub log for the scan; success or the new message → close | CC | | **R-516** | Install & onboarding | P3 | **[P3-LOW] English a customer meets on a fresh box and its apps' first screens — enumerated by the big night.** MEASURED 2026-09-14 (BIGNIGHT, VM 333, ISO 1.27.1, controller 0.242.0). Felhom-owned: (1) the dashboard menu item **„Debug"**; (2) the dashboard CPU tile **„Load: 0.29 / 0.39 / 0.37"**; (3) the launcher tile **„Filebrowser"** opens a login in English with no Felhom text (R-513); (4) the storage page mixes formal „Adjon hozzá / Csatlakoztasson" with the product's „te". App first screens a household meets before any Felhom text helps: (5) **Uptime Kuma 2.4 opens on „Which database would you like to use?"** (SQLite / Embedded MariaDB, „Next") — the app card's „Első lépések" does not mention it; (6) PrivateBin, Gokapi, AdventureLog and FileBrowser UIs are English (the apps' own). Already rows: the Proxmox installer screens (R-495, answered by the guide), `wiki.DOMAIN` (R-498). **Fix shape:** rename „Debug"/„Load" (controller); add the Uptime Kuma database step to its card, or pre-seed `db-config.json` for SQLite in the template (catalog). **Added by F4 (20:04:54Z):** (7) the storage page prints the disconnect time as a raw ISO UTC string „Leválasztva: 2026-09-14T19:58:02Z"; (8) the „Meghajtó leválasztva" banner appears twice on every page; (9) „4 telepített alkalmazás nem fut — nézze meg a rendszermonitort" uses the formal form. **Added by F7 (20:50–21:00Z, system disk at 95 %):** (10) a banner on every page in English, „**SSD disk usage high: 90%**"; (11) the dashboard tile reads „Rendszer (/) 61.8 GB / 68.7 GB (**90%**)" while `df` reports **95 %** (reserved blocks ignored), and „(/)" labels the data volume `/mnt/sys_drive`; the deploy page says nothing about free disk. **Added by the i18n spike (2026-09-17, controller v0.247.0):** (12) **six formal („ön") forms in the converted dashboard copy** — „Olvassa be telefonnal" (launcher QR hint), „Biztosan kikapcsolja a megosztást?" (launcher), „Ha újratelepíti … importálnia kell" (the layout's remove-app modal), „Kérjük, vegye fel a kapcsolatot" (backups empty state). They were NOT fixed — a localisation release may not change Hungarian bytes — and are now COUNTED by `controller/scripts/i18n_missing_gate.py` (`HU_FORMAL_CEILING` = 6, a ratchet: a new one convicts, fixing one lowers it). The inventory (`audits/I18N-INVENTORY-2026-09-17.md`) is the list this row closes against in localisation slice 6 (R-561). **Extended 2026-09-17 by slice 1 (controller v0.248.0–v0.250.0):** the converted copy now counts **16** formal forms (`HU_FORMAL_CEILING` 6 → 10 → 12 → 16, each raise stated, Hungarian unchanged by rule) — and the count is an UNDER-count: the gate's stem list sees 4 on the release C pages (storage, drive wizards, debug), while a wider list („írja be”, „adja meg”, „adjon hozzá”, „válassza ki”, „biztosan eltávolítja”, „engedélyezze”, „hozzon létre”, „kattintson”) finds at least **22** keys there alone (login „Adja meg a jelszavát”, the NAS guide, the storage confirms). Widen the stems when this row is worked; the ceiling rises with them. **NARROWED 2026-09-20 by localisation slice 6's walk, item by item** (`audits/i18n-slice6-2026-09-20/R-516-item-by-item.md`): item 2 FIXED (slice 1), items 3 and 6 APP-OWNED (and for an English household item 6 is an advantage), item 11 is a wrong NUMBER rather than a language and needs its own row, item 1 is correct English for an English reader and still open for a Hungarian one. **Items 4, 7, 8, 9 and 10 are NOT DECIDABLE FROM AN ENGLISH WALK** - they are about Hungarian copy quality, a disconnected drive and a full disk, none of which this walk had. **What this row is now waiting for is a HUNGARIAN walk on a box with a second drive**, not another English one; saying it closed would be the kind of closure that makes a register stop meaning anything. **-- 2026-10-08:** the recovery screen's own answers are Go literals in Hungarian only — 14 `renderRecovery` messages in `controller/internal/web/recovery_handlers.go` (seen: the safe default „A művelet nem fejeződött be…" rendered on an English page in a red-proof run). Today's new R-304 sentence is a bundle key in both languages; the 14 old ones are not. | **NARROWED 2026-09-20 - rank P3-LOW; owner: CC; needs a Hungarian walk** **2026-10-05 (burn-down night): items 1 and 4 FIXED on controller `main`** (`d7efa7b`): the menu says „Hibakeresés” (English „Debug”); the storage pages use the te-form (formal-form ceiling 17 → 14; 116 parity fixtures changed by exactly those bytes). Item 2 was already fixed. **Left:** items 5–6 (apps' own screens) and the rest. **2026-10-06: items 1 and 4 DELIVERED** (controller v0.298.0). **2026-10-06 (burn-down night, later): items 7–10 and 12 FIXED on controller `main`** (`e4774e6`, `2e5ef7e`): the disconnect time in local time; one banner per unplugged drive; „nézd meg”; the disk/memory/CPU/temperature banners in the household's language (`health.*` keys; the wire text to the hub unchanged and pinned); the last eleven counted formal forms are te-form (ceiling 14 → 0; 119 parity fixtures changed by exactly those bytes). **Left:** items 5–6 (apps' own screens); about 20 formal forms the gate does not count yet („írja be”, „adja meg”, „hozzon létre”, …) on the debug, storage, network-storage and security pages. Item 11 moved to its own row **R-889**. **2026-10-06 02:40: the bundle has NO formal form left** (controller `4c3c203`, unreleased): 49 sentences in the te-form; the gate's stem list widened (it counted 61 on the previous bundle, 0 now) with decoys and one listed third-person exception. **Left (narrowed):** Felhom-owned formal forms in Go and template literals the gate cannot read yet — the network-storage attach errors, escrow and share handlers, the single-copy notice, a settings refusal, the SMART mail, a fill-watch „Kérjük”, and the setup wizard pages; each moves into the bundle in the te-form. Items 5–6 are the apps' own screens (not ours). **2026-10-06: items 7–10, 12 and the bundle's 49 formal forms DELIVERED** in controller v0.299.0. **2026-10-06 night: the Go-literal leftover moved** (controller main `3d6ba28`, unreleased): 22 Go-literal messages — escrow handler (6), share password page (2), export upload (1), network-storage uid/remove/attach failures (13) — are bundle keys (`api.escrow.*`, `api.share.*`, `api.export.*`, `api.netstorage.*`) in the te-form with English (14 were formal); a detached attach failure renders in the reader's language; the NAS refusal says „Válassz". `TestR516_GoLiteralFormalFormsAreGone` (source scan + positive control), `TestR516_NetAddFailureFollowsTheReader`; red-proofs `audits/night-burndown-2026-10-06/ctrl/R-516-red*.txt`. **Left:** the setup wizard and recovery-info.txt (R-554); the SMART and fill-watch texts (the hub event's message — wire text, needs the household-copy producer shape); items 5–6 (apps' own screens). | — | — | CC | -| **R-991** | Install & onboarding | P3 | **Nothing a buyer can read says which machines Felhom installs on, and a quarter of the forum beginners own only machines it cannot install on.** FOUND 2026-10-12 (forum research, `audits/forum-research-2026-10-12/FELHOM.md` §2). Searched `documentation/architecture/`, `website/` (`gyik`, `technologiak`, `letoltes`) and `scripts/felhom-host-install.sh`: no CPU-architecture, RAM, disk or network floor is written anywhere. The download page says only that the installer is built on Proxmox VE 9.2 — x86-64 only (inferred from Proxmox, not stated on the page). Of the 33 forum beginners who name what they own, 8 own only a Synology/QNAP/other NAS box, a router with a USB disk or a Raspberry Pi. The selfh.st survey reading reached the same gap independently (R-989, its recommendation 1). **Needs (operator):** the floor itself (RAM and disk minimum; whether thin clients and old laptops with 4–8 GB are accepted) — then one short section on the download page and in the FAQ. Not changed here: the research task forbade website edits. | READY — the website patch waits for the operator's publish. (was WAITING-ON-OPERATOR) — **2026-10-20: R-993 merged here** (one decision, D5): the proposal to approve is `audits/sizing-2026-10-13/SIZING.md` §5 — minimum x86-64, 4 cores (N100 class), 8 GB, 256 GB SSD; recommended 16 GB, 512 GB SSD; out: ARM / Raspberry Pi, Synology-QNAP, routers, cloud VPS. R-993 also measured that the installer has no floor on the HOST (`autosize_guest_caps` limits only the guest), so whatever minimum is approved, nothing enforces it at install today (full text: `git show 5c487313:documentation/backlog/OPEN-ITEMS.md`). filed 2026-10-12 **2026-10-22 night — D5 ruled (operator 15:47: approve the minimum machine).** The hardware section for `letoltes.html` and `gyik.html`, both languages, is in the same ready website patch as D3 (`audits/night-2026-10-22/P2-D3-D5/website-D3-D5.patch`). Not filed: an install-time host floor — R-993 measured that nothing enforces one; whether to refuse a too-small host is a new question for after the publish. | — | Operator: say „publish”; CC pushes the patch; then close. Ask then whether the installer should refuse a host below the floor | CC | | **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC | | **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator | @@ -157,16 +156,15 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest **2026-10-23 night — survey again, read only (`audits/dooplex-survival-2026-10-22/PLAN.md`):** both nightly off-site jobs and both weekly restore tests green (copies 2026-10-22, tests 2026-10-18); the local tree is still the same box (2026-10-22 03:16 ok). New in the survey: the hub DB copy is up to 24 h old (G5: every 6 h — the operator's yes; a DooPlex change). | — | — | operator | +| **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | NARROWED — **2026-10-23 (operator yes D33, D34):** the hub DB now leaves every 6 hours (hub 0.167.0 + `scripts/hub-db-backup/`, alarm at 13 h; restored back byte-identical) and the released images and agent packages leave nightly (`scripts/release-copy/`, R-1227 closed). (c)–(g) unchanged. **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest **2026-10-23 night — survey again, read only (`audits/dooplex-survival-2026-10-22/PLAN.md`):** both nightly off-site jobs and both weekly restore tests green (copies 2026-10-22, tests 2026-10-18); the local tree is still the same box (2026-10-22 03:16 ok). New in the survey: the hub DB copy is up to 24 h old (G5: every 6 h — the operator's yes; a DooPlex change). | — | — | operator | | **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):** `_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). **FIRST VERDICT READ 2026-10-21 (rehearsal session, read-only):** the 07:10 CEST run judged three tiers against yesterday — demo-felhom 5, demo-hp 14, tester-1 4 new snapshot files, „each inside a reported box run"; nothing vanished, nothing unexplained; Tester4 stored its first baseline (44 files); Tester-2's own listing failed (no `snapshots/` dir — the box has never pushed). No `offsite_snapshots_vanished` / `_unexplained` mail in the operator mailbox (second channel). Evidence `audits/due-checks-2026-10-21/R-895-hub-0710.txt`. The positive alarm path is still proven only by the unit red-proofs. | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC | -| **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G1/G3/G10):** unchanged; the sheet is still the one act that breaks the circle, and S5's offline form is unknown to CC — on the sheet too (decide list in STATUS). | operator: print the sheet | Operator: print + store the sheet; then close | operator | -| **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | READY — waits for the controller 0.340.0 release. **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** **READY 2026-10-20 — measured and planned, nothing changed; the decision is the operator's (STATUS D21).** Re-measured on 9202 (Immich v3.2.4): off-site ~1.33 GB = model cache 824 MB (62 %), raw database folder 283 MB (21 %), Immich's own backups 115 MB (9 %), logical dump 54 MB, originals 49 MB, thumbnails 2 MB — 102 MB must survive. A: leave out all three re-fetchable parts (−92 %; needs a never-run restore from the dump alone on the vectorchord image). B: only the model cache (−62 %; needs one restore with an empty cache). Pick: B now, A's other parts after their proof (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`). **2026-10-22 night — D21 = B built and proven, waits for the controller release.** A template may now exclude a named volume from every backup (`.felhom.yml` `backup.volumes`, class `excluded`; controller main `6a17c95`, unreleased, rides 0.340.0; guarded: only a declared name, never all of an app's volumes; the old tar is removed by the stale clean-up). The catalog's Immich template carries it (`ce33e3b`, live, inert before 0.340.0). On 9202: Immich's backup unit 1 160 553 832 → 336 788 031 B; with the cache volume removed, Immich started and its ML answered (HTTP 200, 512-dim) after 9.2 s, the cache refilled to 611 MB (`audits/night-2026-10-22/P2-D21/`). | — | Release controller 0.340.0 (STATUS morning list); close then. A's other parts (the raw DB folder, Immich's own backups) after their restore proof — a new row when started | CC | +| **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | NARROWED — **2026-10-23 (D32):** the sheet is current — names and places only, one A4, HU + EN (`runbooks/break-glass-sheet.md`); every place checked to exist. LEFT: the operator prints it, fills it by hand, keeps it outside the home. **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G1/G3/G10):** unchanged; the sheet is still the one act that breaks the circle, and S5's offline form is unknown to CC — on the sheet too (decide list in STATUS). | operator: print the sheet | Operator: print + store the sheet; then close | operator | | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC | | **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC | | **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC | | **R-433** | Backup & restore | P3 | **A sub-account cannot reach ANY Storage Box snapshot, by any name — so clause (b) of the 2026-09-01 R-95 re-scope ("the rest is recoverable file by file") is NOT SUPPORTED.** MEASURED 2026-09-01 on `demo-hp` over the credential the box already holds, read-only, no delete verb issued. **The sweep:** a batched `stat -c %n` over Hetzner's port-23 restricted shell (500–600 paths per round trip, stdout carrying only paths that exist) tried **777,600** names of the vendor form `YYYY-MM-DDTHH-MM-SS` across nine full days at second granularity, and 126 alternative shapes — **zero resolved.** **The control is what makes the zero mean anything:** the identical 600-name batch with one real path appended returned it in 6 of 6 batches. **The structural cause:** `/home` (the customer data, `u629488-sub3`) is **st_dev 0,82**; `/.zfs/snapshot` is **st_dev 0,276**; `/home/.zfs` does not exist. A snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding `felhom-repo`. **What still stands:** clause (a) — the box can delete its live repository but cannot WRITE into `/.zfs/snapshot` — is unchanged and re-confirmed. **What is now open again:** the only routes to the older copy are a panel rollback of the WHOLE Storage Box (deletes newer snapshots, hits every customer on it) and the provider API (fenced by §11-D, and `hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method at all**, so it needs new code regardless). **NOT ESTABLISHED, and it is the question that decides whether per-file recovery exists for anyone:** whether the MAIN account can see the snapshots. No main-account credential exists in this project. **The ranking is Viktor's; I am not re-ranking R-95 — but the argument that moved it down is the argument this drill removed.** `audits/evidence-drill-r95-recovery-2026-09-01/` **BLOCKED-ON-PROVIDER 2026-09-01.** The one question that can move this is drafted and ready to send: **Question 1** of `documentation/runbooks/provider-questions-2026-09-01.md` — *can the MAIN account retrieve individual files from a snapshot, without a whole-box restore?* **Neither answer leaves this row where it is:** "yes" makes per-file recovery real but permanently operator-only, and closes this; "no, full restore only" means the snapshots do not bound a single customer's exposure at all, because using them costs every other customer on the box their newer snapshots — **and R-95 becomes urgent.** Nothing here can progress without it, and it is not CC's to send (§11-D). Tracked by a dated check in the DUE-CHECKS block. **⚠ A TRAP ON THE WAY TO THIS ANSWER, found by the operator 2026-09-01 and armed against in the questions file: HETZNER HAS TWO SIMILARLY-NAMED PRODUCTS WHOSE DOCS SAY OPPOSITE THINGS.** **Storage SHARE** (a managed Nextcloud, NOT us) documents *"Currently, we only support restores for the full backup ZFS snapshot to a specific point in time"* (`docs.hetzner.com/storage/storage-share/faq/backup-snapshot/`). **Storage BOX** (ours) documents the opposite — *"You can download individual files or entire directories as usual"* (`docs.hetzner.com/storage/storage-box/snapshots/`). **A web search for the obvious phrasing surfaces the SHARE page, and it reads like a definitive NO.** Tell them apart by the giveaways: the Share page talks about *Nextcloud's data cache*, a *database dump* and the *konsoleH* interface, and never mentions Storage Box. **Why this is filed and not left as trivia: a support agent could answer Question 1 from the wrong page, and that answer would push this row to the top of the register for no reason.** Question 1 now names the product, quotes the Storage Box line, and says explicitly that we know what the Share FAQ says. **If an answer cites full-snapshot-only, check WHICH PRODUCT it is about before acting on it.** **Nothing in this repository ever leaned on the Share claim** — verified by grep at the time; the only vendor line we cite is the Storage Box one. **RE-DATED 2026-09-15 → 2026-09-22 (due-checks gate):** the operator mailbox read through the Gmail connector (`(from:hetzner OR subject:hetzner OR "Storage Box") after:2026/09/01`) holds no Hetzner reply — one match, our own `offsite_snapshots_dropped` alarm. Whether the two tickets were ever sent is not visible to CC; if they were not, sending them is the operator's act. **-- ANSWERED. Hetzner replied on ticket #2026090103040671; the operator supplied the thread on 2026-09-22 after this session searched for it and WRONGLY reported no answer (see the instrument note below and R-628).** **Q1 - can the MAIN account read individual files and directories out of a specific snapshot, without restoring the whole box?** Hetzner: *"With the main account it should be possible to download files and directories from the snapshot as it is stated in the documentation"*, citing `docs.hetzner.com/storage/storage-box/snapshots#access-to-snapshots`; and separately *"A restore of a snapshot will revert the Storage Box completely to the state it was in when the snapshot was taken."* **So the Storage BOX documentation governs, not the Storage Share FAQ** - which is exactly the trap `provider-questions-2026-09-01.md` warned the reader about, and the answer came back on the right side of it. **File-level snapshot access is a MAIN-ACCOUNT capability; the sub-account cannot do it**, which matches R-433's own measured finding (777,600 names swept from a sub-account, zero hits). **HEDGE, kept because it is in the reply: "should be possible" is not "is", and nobody has yet read a file out of a snapshot from the main account on `u629488`. That is now the measurement this row needs, and it is cheap.** **Q2 - is `--append-only` enforced by Hetzner, or taken from what the client sends?** Hetzner: *"You can use --append-only and it would look like this: `command="rclone serve restic --stdio --append-only path/to/repo" `"*, with `fluix.one/blog/hetzner-restic-append-only/`. **So it is enforced by US, by pinning a FORCED COMMAND to the SSH key in the Storage Box's `authorized_keys`** - not by Hetzner globally, and not by anything the client asks for. A key without that prefix still gets a deleting server. **This is the answer R-95 has been blocked on** and it says append-only IS expressible on a Storage Box, which R-95 records as impossible over SFTP - see R-95 and R-436. **NOT YET MEASURED, and that distinction is the whole of what this row is worth: this is a support answer, not a proof.** What is owed is a real forced-command key on `u629488`, a restic `forget --prune` through it that is REFUSED, and a backup through it that still succeeds. | **BLOCKED** — **BLOCKED-ON-PROVIDER — Question 1 of `runbooks/provider-questions-2026-09-01.md`** | — | — | CC + operator | | **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC | -| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). | — | — | CC | +| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. | — | — | CC | | **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) | | **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC | | **R-164** | Backup & restore | P4 | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC | @@ -185,25 +183,23 @@ stopping line that lies. | **R-1184** | Backup & restore | P3 | **A cut-off shared-folder restore can be „placed" and reported restored while files are missing; the download is then deleted.** Readiness is only „the scratch folder is not empty" (`internal/backup/shares_restore.go:131-140`) — the R-358 class fixed for apps, not for shares; 30-min download window (`web/offbox_handlers.go:716`); fixed 2 GiB free-space gate that fails open (`:107`). CONFIRMED in code. Data-loss hunt 2026-10-20 (C1 reviewer 1, R3). | OPEN | — | Fix: a completion marker after restic succeeds, as for apps | CC | | **R-1185** | Backup & restore | P3 | **A database dump that fails its own content check has already replaced last night's good dump, and the second drive copies it an hour later.** `appbackup/dbdump.go:388-422`: size check, rename over the good file (`:399`), then `ValidateDump` (`:419`). Volume tars are promoted with no content check either (`backup.go:902`). Tier 1 is one point per app by design (07 §6.1); the ORDER is the defect. PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/`, a docker stub prints a header-only dump): `ValidateDump` says invalid („dump file too small"), `DumpOneTo` returns no error, and last night's good dump (with its `CREATE TABLE`) is gone from disk. Data-loss hunt 2026-10-20 (D5). | OPEN | — | Fix: validate the temp file before the rename; keep the last good dump on a failed check | CC | | **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC | -| **R-1187** | Backup & restore | P2 | **A `.fab` export or import can lose or mix household data — code-confirmed paths from the data-loss hunt:** (a) an export silently drops a whole drive folder that holds a symlink and still ends „done” (`appexport/fabplan.go:292-310`, `export.go:712`; a FIFO also hangs the export on `os.Open`); (b) the import handles only folders and regular files: no owners set, symlinks dropped (`restore.go:1138-1183`); (c) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** an import over an installed app that is not running merged the old data with the bundle's (`docker cp` never deletes) — reproduced on 9202 before the fix (stopped: a marker written after the export SURVIVED the import; running control: replaced), now refused with a Hungarian sentence and the app untouched; red-proved (`r1187c_test.go`), proven on 9202 rc.2 (`audits/night-2026-10-22/P3-candidates/r1187c-9202.txt`); (d) PLAUSIBLE: no restore-busy check in `StartImport`, no undo, no record (`restore.go:181-186`); (f) the bundle's HDD_PATH is trusted, an empty one gives a relative path (`restore.go:589-631`); (g) the DB import misses postgis images (adventurelog, dawarich: `classifyDBImage`) and runs psql without ON_ERROR_STOP (`restore.go:1047-1056, 933-943`; the dump is `--clean --if-exists`, so ON_ERROR_STOP is safe). (e) was R-1191. Research `audits/night-2026-10-22/P3-candidates/analysis.md`; first list `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | OPEN — **2026-10-22 night: re-ranked P3→P2** ((a) and (b) lose household files on a move to a new box with no error). (c) fixed in code; (a), (b), (f), (g) read in code, not live-proven. **2026-10-23 night — fresh review (`REPORT-night-2026-10-23.md`, C2, C3):** two more import paths still merge — a running app whose `RemoveStackVolumes` fails falls back to `StopStack` and goes on (`appexport/restore.go:375-380`), and `restoreHDDData` extracts drive-folder tars over the existing folder without clearing it (`restore.go:602-655`) for running and stopped apps alike. Both exist in 0.339.0 — not a hold for 0.340.0. The (c) refusal also catches an unhealthy or restarting app (`IsStackRunning`: running/degraded only); its text now says so and names removal (controller `943c517`). Recovery code: fix only with a red-proof + a 9202 proof. | — | Fix (a)+(b) with a red-proof and a 9202 proof (capture code — the night rule); (g) small fix with a 9202 postgis app; (d) after (a) | CC | +| **R-1187** | Backup & restore | P2 | **A `.fab` export or import can lose or mix household data — code-confirmed paths from the data-loss hunt:** (a) an export silently drops a whole drive folder that holds a symlink and still ends „done” (`appexport/fabplan.go:292-310`, `export.go:712`; a FIFO also hangs the export on `os.Open`); (b) the import handles only folders and regular files: no owners set, symlinks dropped (`restore.go:1138-1183`); (c) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** an import over an installed app that is not running merged the old data with the bundle's (`docker cp` never deletes) — reproduced on 9202 before the fix (stopped: a marker written after the export SURVIVED the import; running control: replaced), now refused with a Hungarian sentence and the app untouched; red-proved (`r1187c_test.go`), proven on 9202 rc.2 (`audits/night-2026-10-22/P3-candidates/r1187c-9202.txt`); (d) PLAUSIBLE: no restore-busy check in `StartImport`, no undo, no record (`restore.go:181-186`); (f) the bundle's HDD_PATH is trusted, an empty one gives a relative path (`restore.go:589-631`); (g) the DB import misses postgis images (adventurelog, dawarich: `classifyDBImage`) and runs psql without ON_ERROR_STOP (`restore.go:1047-1056, 933-943`; the dump is `--clean --if-exists`, so ON_ERROR_STOP is safe). (e) was R-1191. Research `audits/night-2026-10-22/P3-candidates/analysis.md`; first list `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | OPEN — **2026-10-23: (c) and the C3/N9 refusal texts released in controller 0.340.0;** (a), (b), (f), (g) and the two import paths of C2 stay open. OPEN — **2026-10-22 night: re-ranked P3→P2** ((a) and (b) lose household files on a move to a new box with no error). (c) fixed in code; (a), (b), (f), (g) read in code, not live-proven. **2026-10-23 night — fresh review (`REPORT-night-2026-10-23.md`, C2, C3):** two more import paths still merge — a running app whose `RemoveStackVolumes` fails falls back to `StopStack` and goes on (`appexport/restore.go:375-380`), and `restoreHDDData` extracts drive-folder tars over the existing folder without clearing it (`restore.go:602-655`) for running and stopped apps alike. Both exist in 0.339.0 — not a hold for 0.340.0. The (c) refusal also catches an unhealthy or restarting app (`IsStackRunning`: running/degraded only); its text now says so and names removal (controller `943c517`). Recovery code: fix only with a red-proof + a 9202 proof. | — | Fix (a)+(b) with a red-proof and a 9202 proof (capture code — the night rule); (g) small fix with a 9202 postgis app; (d) after (a) | CC | | **R-1188** | Backup & restore | P4 | **Restore-path candidates from the data-loss hunt — what is left is unlikely or a sentence:** (b) a failed `volume rm` is ignored before the untar, which then lands in the old volume (`backup/restore.go:218`; `StopStack` is `compose down`, so the rm rarely fails); (d) the full off-site download stops at 30 min per press and the household is not told to press again (`web/offbox_handlers.go:455`; the scratch is kept, so a second press probably continues — not measured); (e) place and full restore re-read the LATEST snapshot, not the downloaded one (`offbox_restore.go:675-679`, `_ = id`). Closed parts: (a) with R-1195 (controller 0.338.0), (c) in controller 0.337.0 (`TestR1188c_ALeftoverTempDoesNotBlockTheRetry`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — **2026-10-22 night: re-ranked P3→P4**: (a) and (c) are shipped; the rest is low-chance or a sentence. | — | (d): measure a second press on a large download on 9202, then write the sentence; (b), (e): unit tests when the file is next touched | CC | -| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt — what is left:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it: the agent turns a future archive time into a negative age (`localapi/server.go:1179-1192`); half-live 2026-10-21 (B6 #8, `s8-wrong-clock/README.md`: a local archive dated 2028-10-21); agent code, so it rides an agent release and the operator's vouch; (c) the whole-guest quiesce waits for the catch-up and the update leg, not for a restore (`quiesce.go:213-275, 565`); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — a design question; (g) spring forward 2027-03-28 moves 02:30 onto the 03:30 second copy; (h) self-update busy check before the pull only (`updater.go:532-545, 764`; not re-read); (i) a double-clicked restore shows failed while it runs (check and begin not atomic, `web/handlers.go:1860, 1881`). **Moved out 2026-10-22 night:** (b) the skipped nights after an off-cycle run were FIXED by R-899 (controller 0.304.0; demo-hp's journal shows a night run after each of three off-cycle runs since), the creep → R-1221; (d) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** the catch-up now runs the off-site proof and the integrity check once after its legs, the false comment corrected; red-proved (`nightchain/r1189d_test.go`, `cmd/controller/r1189d_wiring_test.go`); (e) → R-1222 (hub). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — 2026-10-22 night: (b) half closed, (d) fixed in code, (e) split out. | — | (a) with the next agent release (a unit test with a future archive); (c) a 9202 recipe; (g) a unit test with the 2027 clock; (f) to the operator at the next design review | CC | +| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt — what is left:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it: the agent turns a future archive time into a negative age (`localapi/server.go:1179-1192`); half-live 2026-10-21 (B6 #8, `s8-wrong-clock/README.md`: a local archive dated 2028-10-21); agent code, so it rides an agent release and the operator's vouch; (c) the whole-guest quiesce waits for the catch-up and the update leg, not for a restore (`quiesce.go:213-275, 565`); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — a design question; (g) spring forward 2027-03-28 moves 02:30 onto the 03:30 second copy; (h) self-update busy check before the pull only (`updater.go:532-545, 764`; not re-read); (i) a double-clicked restore shows failed while it runs (check and begin not atomic, `web/handlers.go:1860, 1881`). **Moved out 2026-10-22 night:** (b) the skipped nights after an off-cycle run were FIXED by R-899 (controller 0.304.0; demo-hp's journal shows a night run after each of three off-cycle runs since), the creep → R-1221; (d) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** the catch-up now runs the off-site proof and the integrity check once after its legs, the false comment corrected; red-proved (`nightchain/r1189d_test.go`, `cmd/controller/r1189d_wiring_test.go`); (e) → R-1222 (hub). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — **2026-10-23: (d) released in controller 0.340.0;** (a), (c), (f), (g) stay as written. OPEN — 2026-10-22 night: (b) half closed, (d) fixed in code, (e) split out. | — | (a) with the next agent release (a unit test with a future archive); (c) a 9202 recipe; (g) a unit test with the 2027 clock; (f) to the operator at the next design review | CC | | **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt — what is left:** (c) PLAUSIBLE: after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists — a design question against `07` §6.1's Tier-3 gate (the operator's); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`) — (e), (f), (h) not re-read. **Moved out 2026-10-22 night:** (a) → R-1218, (b) → R-1219, (g) → R-1220. Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN | — | (c) a read-only fingerprint compare on a reinstalled test box; (e), (f), (h) re-read, then unit tests; (d) to the operator at the next design review | CC | | **R-1198** | Backup & restore | P3 | **A backup disk the family adds later is never offered as the place for the whole-box backup; the page keeps offering the disk that holds the apps' own data.** Rehearsal 2026-10-20 B6 #13 (`audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/`): a 50 GB disk plugged in and initialised as „Mentolemez" → the backups page still offers only „Családi adatlemez — Kijelölöm"; `internal/web/backup_target_offer.go:101` `firstOfferableDrive` offers the FIRST eligible drive, never a choice, and the drive's card has no „use for backups". | OPEN | — | Offer every eligible drive (largest empty first) and let the household pick | CC | | **R-1200** | Backup & restore | P3 | **A box installed in the afternoon gets no whole-box backup on its first night: the install's own backup starts the 24 h clock, so the family's whole first day reaches a whole-box copy only on the SECOND night (~38 h after install).** Rehearsal 2026-10-20 B5 (`audits/rehearsal-2026-10-20/B5-night/README.md`): Tester4's install backup at 12:09Z; at the 02:30–06:30Z window the agent answered „within cadence window" (`felhom-agent internal/localapi/server.go:1187`, `age >= tier.Cadence`, default 24 h). R-899 (closed) fixed the same shape for a daytime PRESS only. The other tiers did run that night (database copy, second drive, off-site — all PASS), so app data was covered; the whole-box (DR) tier was not. | **OPEN** — a design question: should the first scheduled night always run (like a press), or is one tier enough on night one? | — | decide; if „always run": treat the install backup like a press in the due rule, with a test on the first night | CC | -| **R-1204** | Backup & restore | P2 | **A box that runs once with its clock ahead loses its WireGuard tunnel to ep0 — the PBS whole-box copies and the operator's door — until the real date passes the wrong one, and nothing says why.** Rehearsal 2026-10-20 B6 #8 (`audits/rehearsal-2026-10-20/B6-failures/s8-wrong-clock/`): Tester4 booted with its clock at 2028-10-21 for 13 min, then the real time. ep0's `wg show` (read-only): Tester4's last handshake 07:40:47Z (clock at 2028), none in the 14 min after the return, the other peers 21–96 s. WireGuard refuses a handshake initiation whose TAI64N timestamp is not newer than the last one accepted from that peer, so the peer stays dead until 2028 or until ep0 resets it. Effects seen: every agent PBS call times out (15 s), the controller's due-check errors, the backups page says „A host-ügynök jelenleg nem elérhető". Likely triggers in the field: a dead CMOS battery, a hand-set BIOS clock, a VM restored with a wrong RTC. | READY — agent release + hub 0.167.0 deploy. **OPEN** — reproduced once live; not reset on ep0 (read-only in this run; the peer still shows the stale handshake). The box is destroyed in B7, which ends it for Tester4. **2026-10-22 night — detection built, not live; the heal is decision D30.** Agent main `a0c88e5` (unreleased): the heartbeat reports `handshake_future_s` when the last handshake is more than 60 s in the future (`TestR1204_AFutureHandshakeIsReported`). Hub 0.166.0 (`e267c5e2`, image built, NOT deployed): `wg_tunnel_stale` / `wg_tunnel_ok` to the operator only, edge-triggered, when a box reports a future handshake or one older than 6 h (healthy boxes: at most 113–124 s over 24 h). The heal options: A) the agent makes a new WireGuard key and registers it through the existing peer path (`hub/internal/api/wg.go` `RegisterWGPeerForHost` + the wgsync push to ep0) — a new key has no stored timestamp; B) ep0 removes and re-adds the same peer on a hub-signed request — a new write path on ep0. **2026-10-23 night — D30 built (operator 2026-10-22: the agent makes a new key on the existing peer path).** Agent main `ae69ad4` (unreleased): a handshake stamped in the future for 15 min → a new key pair (the old one kept as `private.key.prev`), registered through `POST /hosts//wg`; the hub re-keys in place (same /32) and its full-list push swaps the key on ep0 — nothing new on ep0. Hub main `84751cb0` (0.167.0 source): the new key's handshake clears `wg_tunnel_stale` (pinned). **Live on the test box (Tester4-16fdef, nested VM 9344, `audits/night-2026-10-23/P2-D30/`):** run 1 (clock +2 h, back) — the tunnel healed by itself in ~3 min, because ep0 started a handshake of its own (it had traffic for the box) and the box accepts ep0's timestamps; the agent saw the future stamp and correctly did NOT re-key. Run 2 (the rehearsal's condition: ep0's handshake starts dropped on the box, a test-box firewall rule) — the tunnel stayed dead as in the rehearsal; 15 min later the agent re-keyed and the hub took the new key (`existed=false generation=3 sync=ok`, same 10.77.0.5). **What R-1204's failure needs, measured:** a box that runs ahead AND an ep0 with nothing to send it — so it is rarer than the row said; D30 covers it. | — | Release the agent (with D4) and deploy hub 0.167.0 (STATUS morning list); close then. The reboot gap is R-1223 | CC | -| **R-1205** | Backup & restore | P2 | **After a box death, a SECOND install that is deleted before it makes a key leaves the whole-box (PBS) tier waiting for the operator's re-issue.** Seen live 2026-10-21 (fix-1 proof, `audits/fixes-2026-10-21/P1-live/`): the rebuilt Tester4 box enrolled and the hub logged `pbsdr auto-provision for Tester4 (WG-registration hook): the endpoint already holds a PBS token for Tester4 but the hub has no descriptor — use the explicit "Re-issue PBS credentials" action`. The F-14 auto-re-issue reads the customer's LATEST host deletion, which was the short-lived rehearsal box (B7: deleted un-acked — it never held an escrow row); the R-1176 carry reads the newest ACKED deletion and worked. A household whose first reinstall attempt is abandoned gets no whole-box copy until the operator presses. | READY — hub 0.167.0 deploy. WAITING-ON-OPERATOR — **2026-10-22 night (read, `hub/internal/web/pbsdr.go:236-248`, `store.go:3293-3297`):** the F-14 gate is deliberate (operator ruling 2026-07-13), so this is a design question, not a defect — decision D31 in STATUS. **2026-10-23 night — D31 built (operator 2026-10-22: skip a deleted install that never made a key).** Hub main `84751cb0` (0.167.0 source, not built): the F-14 gate reads the newest deletion of a host that HELD A KEY (`LatestKeyHoldingHostDeletion`, `escrow_acked = 1`); tests `TestLatestKeyHoldingHostDeletion_SkipsAKeylessInstall` + `TestPBSDR_D31_AKeylessInstallBetweenDoesNotBlockTheReissue`, red-proved. **Review note (`REPORT-night-2026-10-23.md`, N1) — the operator's hold-or-go:** „never made a key” is measured as „held no escrow row”; an install that got a PBS secret (through an earlier automatic re-issue) but was deleted before its escrow ceremony is skipped too, so the next install's re-issue rotates THAT deleted install's secret without an acknowledgement. CC: go — a rotation revokes the deleted box's secret (the safer outcome) and the household's copies stay; the stricter gate (skip only deletions with no PBS descriptor) would leave R-1205's own live case manual. | the hub deploy (the operator's yes) | Deploy hub 0.167.0; close then | CC | | **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC | | **R-1212** | Backup & restore | P4 | **Two household texts on the backup path do not follow the box's language:** the share-restore summary (`SharesRestoreResult.FlashMessage`, `controller/internal/backup/shares_restore.go`) is built from Hungarian Go literals, and the off-site `LastError` for an interrupted run is the Hungarian literal „megszakadt futás (a vezérlő újraindult futás közben)" (`controller/internal/backup/backup.go`). An English household reads Hungarian there. Found by the night helper 2026-10-21. | OPEN | — | Bundle keys for both (rendered in the box language when written, R-557), with parity fixtures | CC | | **R-1218** | Backup & restore | P2 | **A recovery ceremony done while off-site is switched off (or cleared) seals no off-site repository password, and the hub then demotes the GOOD older package to operator-only custody: the household's own recovery code no longer opens its off-site copy.** REAL in code (night 2026-10-22, read; split from R-1190 (a)): `web/escrow_handlers.go:217-224` stages the password only if `OffboxConfigured()` (`backup/offbox.go:695-710`), false when the target is cleared or disabled — although `ClearOffboxTarget` keeps the password file on purpose (`offbox_clear.go` header); the agent then seals no password (`cmd/felhom-agent/main.go:2981-3003`) and uploads sha ""; hub `SaveHostEscrow` sees `curSHA != ""` and moves the good package to retained custody; `OffsiteRecoveryOffer` reads "" as „covers no key” (`offbox.go:1855-1873`) and says nothing. Evidence `audits/night-2026-10-22/P3-candidates/analysis.md` §R-1190(a). | OPEN — proposed fix, not built (key-handling code: a fix needs a red-proof AND a live proof, and 9202 has no hub to prove it with) | — | Fix (~6 lines): stage when the password FILE exists (`PushOffboxPasswordForEscrow`, `offbox.go:800`, reads only the file). Red-proof: a web test that the stage runs with off-site cleared; a hub store test pinning „abc”→"" ⇒ superseded (and refuse that demotion). Live proof on a test box with a hub (demo-hp 9201) | CC | | **R-1219** | Backup & restore | P2 | **During a recovery offer the backups page hides the „create your recovery code” ceremony (it would replace the key that opens the old off-site copy), but the global top bar still links to it and the start handler does not refuse.** REAL in code (night 2026-10-22, read; split from R-1190 (b)): `web/escrow_banner.go:23-35` ignores `recoveryOffer()`; `templates/backups_remote.html:144-160` hides it and its own comment names the harm (R-199); `escrow_handlers.go:180-245` starts anyway; reachable in offer shape (c), `offbox.go:1866-1870`. Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — proposed fix, not built (key-handling code) | — | Fix: the bar hides its link during an offer, and the start handler answers 409 with a Hungarian sentence; red-proof with a handler test; live proof with an offer state on a test box with a hub | CC | | **R-1220** | Backup & restore | P2 | **„Megőrzött adatok” (kept data) can list an INSTALLED app's live folder as kept — and offer to delete it — when that app's app.yaml cannot be read.** REAL in code (night 2026-10-22, read; split from R-1190 (g)): a nil AppConfig drops the app from `liveDriveBinds` (`stacks/kept.go:220-235, 336-360, 372-389`); a parse error gives nil (`deploy.go:1113-1125`). Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — proposed fix, not built (delete-path code) | — | Fix: an app whose app.yaml is unreadable counts as live (fail closed); its folders are never listed as kept. Red-proof: a unit test with a broken app.yaml; 9202 proof with a throwaway app | CC | | **R-1221** | Backup & restore | P4 | **The nightly whole-box backup starts ~1.5–5 min later each night, until one night is skipped (after ~70–80 nights with no press between).** REAL, low harm (night 2026-10-22; split from R-1189 (b)): agent `localapi/server.go:1184` is `age >= Cadence` with the age counted from the archive START; the controller polls every 5 min (`quiesce.go:150`) and the archive starts after the apps stop. Measured on demo-hp: 04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40. At W+6h one night is skipped (44 h is below the 48 h valve, `quiesce.go:807-818`); the hub's 26 h line (`deadline.go:29`) mails the operator. Any manual or off-cycle run resets the drift. Evidence `audits/night-2026-10-22/P3-candidates/demohp-whole-box-completions.txt`. | OPEN | — | Fix: inside the night window a tier is due at `age >= cadence - 4h`; the agent answers due-ness, so it is agent code (a unit test with an injected clock) and rides an agent release | CC | -| **R-1227** | Backup & restore | P2 | **The container registry and the agent packages exist only on DooPlex — after its loss no box updates, reinstalls or installs.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G7): `/data/gitea/packages` is 38.6 GB, Longhorn on DooPlex only; the nightly off-site copy leaves it out on purpose (2026-10-09: „rebuilt from the code”), and a rebuild was never timed; a rebuilt image is not the same digest. | OPEN | the operator's yes (a DooPlex change) | Operator decides (STATUS): A) copy what a box can be offered (released controller/hub tags, agent packages, infra images) nightly to ep0 `operator` — CC's pick; B) a timed rebuild drill and accept the outage | operator | -| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | operator | +| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). OPEN | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) | | **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator | | **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC | | **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go` `newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC | +| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC | ## Storage & devices — 11 rows (P3 4, P4 7) @@ -226,12 +222,11 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-1139** | Security & access | P2 | **Tester 2's break-glass root password was not rotated with the others (R-954, operator decision D1 = A, 2026-10-17): the box has been offline since 2026-10-04.** demo-hp, Tester 1 and demo-felhom were rotated and proven on 2026-10-17 (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | OPEN — filed 2026-10-17 **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a fresh install sets and vaults a new console password).** | Tester 2 reporting again | When it reports: the installer's step 4b alone (`P3/rotate-step4b.sh` shape), then the vaulted value logs in (PVE API 200) and the old one is refused | CC | -| **R-985** | Security & access | P3 | **Our zero-knowledge wording is stronger than the product: on a live box the operator can reach the off-site backup key (through the vaulted root credential and the operator tunnel) without the customer's recovery code.** Design-shaped; decision for the operator in `STATUS.md`. Fix options: change the wording, or remove operator root reach to the key. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | READY — the website patch waits for the operator's publish. (was WAITING-ON-OPERATOR — design-proposal) **2026-10-22 night — D3 ruled (operator 15:47: change the wording now).** The volunteer guides carry the new wording (felhom.eu `0639cf07`); the website change is a ready patch, not pushed, because a website push publishes (`audits/night-2026-10-22/P2-D3-D5/website-D3-D5.patch`, applies cleanly). | — | Operator: say „publish”; CC applies the patch, runs `site_gates.py`, pushes; then close (look again after the first sale, per D3) | CC | | **R-1042** | Security & access | P3 | **The boxes' Cloudflare keys cannot change zone settings, so Cloudflare's own „Always Use HTTPS” cannot be switched on.** Measured 2026-10-14 on all three zones (`audits/https-only-2026-10-14/``1A-1-zones-and-keys.txt`): each `cf_api_token` is valid and sees only its own zone, but GET/PATCH `settings/always_use_https`, `ssl`, `security_header` answer 403; the redirect and transform rule phases too. The https-only fix therefore lives on the box (decision 246). A second layer at Cloudflare needs „Zone Settings: Edit” (and, for headers, „Transform Rules”) added to each customer's key — the hub's zone-reach check stays. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | a key permission only the operator can grant | Optional: widen each customer key, then switch „Always Use HTTPS” on per zone | operator | | **R-1104** | Security & access | P3 | **Check https-only on Tester 2 when it reports again.** R-1033 (plain http on every public name) is closed on the three boxes that report (35 names → 301, 2026-10-16). Tester 2 (`sajatfelhom.hu`) has been off since 2026-10-04 on controller 0.292.0; its addresses answer Cloudflare 530 (tunnel down), so nothing is served in plain http while it is off. When it comes back it takes the floor (≥ 0.312.0 carries the redirect). | OPEN — filed 2026-10-16 | Tester 2 reporting again | Run `scripts/https_only_check.py` with Tester 2 in it; close | CC | | **R-1141** | Security & access | P3 | **`felhom-host-install --rotate-recovery` exists only as part of a full installer re-run, which also reinstalls the vouched agent and the root files.** On a box running a newer agent than the vouched one (demo-hp on the candidate, 2026-10-17) it would move the agent back; on a household box it is a much larger act than a password change. The 2026-10-17 rotation therefore ran the installer's own `step_break_glass` ALONE (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | **NARROWED 2026-10-18 — built on `main`, not published.** `felhom-host-install.sh --rotate-recovery-only` runs step 4b alone (felhom.eu `7d64038f`): reads the hub, host id and host key from the agent config, refuses byo and every other mode flag, asks the hub's `/healthz` before the password changes, never touches the agent or root files. The default install path is unchanged in behaviour (harness verdicts and 9 transcripts identical old vs new; demo-hp `--preflight-only` identical; `--rotate-recovery-only --dry-run` on demo-hp changed nothing). `step_break_glass` moved, byte-identical, ahead of the argument parse. `break-glass.md` §5 points at it. Publishes only with the next `installer-v…` tag. | the next installer tag | Cut the next `installer-v…` tag (with the other untagged installer changes); close | CC | | **R-955** | Security & access | P4 | **The cross-box check of the hub API is done in source and tests but not fully proven: if a later check masks a missing id check, a box's key could read or change another box's hub record without any test noticing.** Read 2026-10-12: every box-facing `/api/v1` route takes the box from its own key or refuses a different id; 14 routes already had a refusal test, 5 got one (`crossbox_refusal_test.go`). Left: red-prove the 4 new tests that were not red-proved (as the restore-directive one was), and the live read-only check with one demo box's key against the other demo box's id. | OPEN — filed 2026-10-12 by the security review | — | Red-prove the 4 tests; run the live GET check on the two demo boxes | CC | -| **R-924** | Security & access | P2 | **The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment.** MEASURED 2026-10-09: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` (0600 `kisfenyo`); `ssh-keygen -y -P ''` opens all three (no passphrase); `/mnt/5_hdd/felhom.eu` is in no backup set, the off-site job does not carry them. `04` §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; `04` §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). `runbooks/total-loss-of-dooplex.md` S6. | **NARROWED 2026-10-09 — operator ruling: option A** (print both AND copy them). The three key files and their public halves ride the nightly encrypted DooPlex→ep0 copy (`scripts/dooplex-offsite/`, a missing key refuses the push); the Sunday restore test proves each private key still derives its own public half (never printed). **LEFT:** the operator prints S6a/S6b (`runbooks/break-glass-sheet.md`); the keys still have no passphrase (custody ruling 1 of 2026-09-16 stands until the first sale). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G2):** the three key files are on `sda1`, the same disk as DooPlex's local backup tree; their only off-box copy is the S1-encrypted nightly one. Passphrase: unchanged by the custody ruling. | operator: print | Operator: print the two keys; then close | operator | +| **R-924** | Security & access | P2 | **The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment.** MEASURED 2026-10-09: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` (0600 `kisfenyo`); `ssh-keygen -y -P ''` opens all three (no passphrase); `/mnt/5_hdd/felhom.eu` is in no backup set, the off-site job does not carry them. `04` §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; `04` §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). `runbooks/total-loss-of-dooplex.md` S6. | NARROWED — **2026-10-23 (D32):** S6a/S6b are on the current sheet. LEFT: the operator prints it. **NARROWED 2026-10-09 — operator ruling: option A** (print both AND copy them). The three key files and their public halves ride the nightly encrypted DooPlex→ep0 copy (`scripts/dooplex-offsite/`, a missing key refuses the push); the Sunday restore test proves each private key still derives its own public half (never printed). **LEFT:** the operator prints S6a/S6b (`runbooks/break-glass-sheet.md`); the keys still have no passphrase (custody ruling 1 of 2026-09-16 stands until the first sale). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G2):** the three key files are on `sda1`, the same disk as DooPlex's local backup tree; their only off-box copy is the S1-encrypted nightly one. Passphrase: unchanged by the custody ruling. | operator: print | Operator: print the two keys; then close | operator | | **R-132** | Security & access | P3 | **`curl -w '%{redirect_url}'` reconstructs the request URL WITH its basic-auth credential** — so a `-u ":$HUB_PW"` call that never put the password in a URL still printed it **Merged 2026-10-05 from R-350 (duplicate):** (1) 2026-08-20 occurrence: POST /configuration/artifacts answers 303; leak lives only in the CC transcript under ~/.claude/projects/ on DooPlex, not in git/evidence (checked then). (2) `-v` and `--libcurl` also re-render the credential, not only %{redirect_url}; confirm redirects with %{http_code} + follow-up GET. (3) Rotation path: hub /configuration form (current_password/new_password/confirm_password); DB override wins over ConfigMap (break-glass); CC can rotate file-to-file without printing (operator-present-one-time-secrets) if asked. | **NARROWED 2026-10-20 — the rotation is DONE:** the operator changed the hub password on 2026-10-15 and the `HUB_PW` line is gone (`audits/hub-tokens-2026-10-15/C153-live.txt:18`; CC uses its own token since hub 0.152.0). **Left, the operator's:** should the break-glass `password_hash` in `manifests/hub.yaml` (a public repository) be emptied or replaced? The DB override wins at login, so the old hash no longer opens the hub. **WAITING-ON-OPERATOR** — **ACTION: rotate `HUB_PW`** **Folded R-580 2026-10-03** (the same `curl -w %{redirect_url}` credential echo, seen again 2026-09-18). **Note 2026-10-12 (security review):** `manifests/hub.yaml` also carries the hub login's bcrypt hash in this public repository (the break-glass field); when `HUB_PW` is rotated, decide whether that field should stay empty. | the operator's word on the `manifests/hub.yaml` hash field | Happened on 2026-07-31 while red-proofing the R-120 gate: the hub operator password was written to the session transcript by the write-out format, not by the request. `-u` is safe; the *reporting* was not. Rule: read the redirect from `-D -` and grep `^Location:`, never `%{redirect_url}`, on any authenticated call. Rotate the hub password (`/configuration` → Login password; ConfigMap `auth.password_hash` is the reset path) and update `~/.config/credentials` | Viktor | | **R-137** | Security & access | P3 | **Cloudflare geo-WAF rules are zone-scoped and non-namespaced — four cross-tenant faults.** `globalRuleDesc = "[felhom-geo] Global"` (`waf.go:18`) is one literal description per ZONE; `appRuleDescPrefix` keys by app name with no customer (`waf.go:21`); `BuildGlobalExpression` has no positive hostname scoping (`waf.go:241`); `applyDiff` deletes every `[felhom-geo]` rule not in THIS box's desired set (`geosync.go:320`) | READY (M) — **blocks shared-zone onboarding** | — | With two customers in one zone: they overwrite each other's Global rule forever; one customer's country policy applies zone-wide; per-app rules collide by name; and disabling the feature for one (or the hub's `RemoveGeoRules`) wipes them all. Interim mitigation, no code: keep geo-restriction OFF for every shared-zone customer. Fix = namespace descriptions by `customer_id` + add `http.host ends_with ""` to both expressions — a TWO-REPO change (controller + hub `RemoveGeoRules`). Same audit §5.1 | CC | | **R-775** | Security & access | P3 | **[P2-MEDIUM] Grimmory: a stranger's 5 wrong sign-ins lock EVERY visitor out of the web login for 15 minutes — so Grimmory was not published.** MEASURED 2026-10-01 on 9202 (drill catalog, v3.4.1, through traefik): after 5 wrong tries for `admin` every further sign-in answered 429 — the household's right password AND a different name — and stayed 429 for 10+ minutes of retries. Read in the jar: `AuthRateLimitService` — Caffeine `expireAfterWrite(ofMinutes(15))`, `MAX_ATTEMPTS 5`, keys `login:ip:` and `login:user:`; Spring `forward-headers-strategy: native` takes the address from X-Forwarded-For, and behind the tunnel every visitor is the tunnel container's address (R-753) — the wger shape (R-752), with no setting to change it. Everything else in the checklist passed (bench + box step v3.4.1 → v3.5.0, gate by its own probe, OPDS through traefik); two smaller findings for the publishing session: on a reinstall over the first install's kept books, a new upload was saved to the drive but not added to the library (`box/grimmory/reinstall-c1.txt`, not investigated); and the remove + restore round trip (2.5) cannot be shown on 9202 for a drive app — its backup lives on the scratch drive, which is not a registered drive (R-756). The template waits in `audits/new-apps-2026-10-01/wip/grimmory/`. **Needs (operator):** (A) publish with a sentence on the page that wrong guesses by others can lock the login for 15 minutes (MEASURED: during the lock an e-reader's OPDS feed still answered 200 with its own login, wrong 401 — `box/grimmory/opds-under-lock.txt`), or (B) wait until the box passes each visitor's real address (R-753). `audits/new-apps-2026-10-01/box/grimmory/throttle.txt` **-- 2026-10-01 (evening):** option B's precondition SHIPPED (controller v0.286.1, R-753): Grimmory's Tomcat RemoteIpValve walks from the right and counts `172.16.0.0/12` as a proxy (READ in source, Spring Boot 4.1.1 — not yet measured with Grimmory's own lock), so `login:ip:` becomes per visitor; `login:user:` still lets a stranger lock the public name `admin` 15 min. A third route was spiked and passed: Grimmory behind the permanent family gate with its e-reader paths excepted (R-780). Recommendation: publish behind the family gate if R-780 is built; otherwise B with a measured 3.6. **UPDATE 2026-10-02 — NARROWED, Grimmory PUBLISHED behind the family gate:** a stranger cannot reach Grimmory's web sign-in at all (6 tries through the simulated tunnel: the gate's 401, then the household signs in 200 — `audits/family-gate-2026-10-02/A/items.txt`), and the e-reader exceptions keep Grimmory's own login. The 2.5 round trip now WORKS on 9202 (`audits/family-gate-2026-10-02/B/box/life.txt`). **What is left:** a family member past the gate can still lock a NAME (the admin's) for 15 minutes with 5 wrong tries — hard-coded in Grimmory; and the reinstall-over-kept-books finding (`new-apps-2026-10-01/box/grimmory/reinstall-c1.txt`) is still not investigated. | **WATCHING — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P2→P3: Grimmory is now behind the family gate; only a family member can still lock a name.** | — | — | CC | @@ -244,17 +239,17 @@ stopping line that lies. | **R-1050** | Security & access | P4 | **Raise HSTS to one year on or after 2026-11-15, if no https trouble was reported.** `09` §3 decision 249 (D2 = A, operator 2026-10-15): HSTS stays `max-age=86400` now; after a month with no https trouble it becomes one year (`31536000`); never preload, never `includeSubDomains`. The value is `HSTSMaxAgeSeconds` in `felhom-controller/controller/internal/infra/infra.go`, pinned by `TestRenderForwardedHeaders_HTTPSOnly`. „Trouble” = any report of a name that will not open, or a box answering http after the floor. | **WATCHING** — filed 2026-10-15 | the date | On or after 2026-11-15: read the register and the operator's mail for https trouble since 2026-10-15; none → a controller release with one year (test updated), bake → vouch → floor | CC | | **R-1054** | Security & access | P3 | **The hub's operator/global API key is still one bearer that can do everything on `/api/v1`, including reading every box's break-glass root password (`GET /admin/hosts/{id}/recovery-credential`).** Hub v0.152.0 gave the web surface scoped tokens (`05` §17) but tokens are refused on `/api/v1` by design. Who uses the global key today is not listed in one place. Fix: list its consumers; move each operator use to a scoped token on the web side (or a token class on the API), and keep the global key only where a box path needs it — or retire it. | OPEN — filed 2026-10-15 by the token build | — | List the global key's consumers (pod env, scripts, runbooks) | CC | | **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f ` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC | +| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC | ## Box system & updates — 12 rows (P2 1, P3 7, P4 4) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC | -| **R-982** | Box system & updates | P3 | **The Proxmox host accepts root password login over SSH from the LAN (measured on demo-hp), so anyone on the household network can try the root password.** Design-shaped: the break-glass password exists for the console. Fix: key-only SSH for root; the password stays for the console. Evidence: `audits/security-review-2026-10-12/evidence/demo-hp-host-reads.txt`. | BLOCKED — on the agent release (the operator's vouch) and an installer tag. OPEN — design-proposal **2026-10-22 night — D4 ruled (operator 15:47: root SSH from the LAN by key only; the console password stays) and built as two ready patches, not pushed:** the agent's config bundle gains a sshd drop-in (`/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`, `PermitRootLogin prohibit-password`; Debian reads the drop-ins first and the first value wins; felhom-sshd on 8822 has its own config) and the installer's uninstall removes it. Proven on a fresh install from ISO 1.30.0 (nested VM 9343, destroyed): before, password SSH worked; after, password SSH refused, key works, PVE web password login 200, console login works (`audits/night-2026-10-22/P2-D4/`). Not pushed: the bundle half reaches a box only with an agent release, and the installer half only with a new `installer-v…` tag (R-110; a push to `main` alone publishes nothing) — both forbidden tonight. **2026-10-23 night — the D4 patches v2 after a fresh review (`REPORT-night-2026-10-23.md`, A1–A7):** the bundle uses `systemctl try-reload-or-restart ssh` (a stopped or socket-activated sshd no longer refuses the bundle and freezes the route), reads `sshd -T` back and says loudly when the drop-in is not in force (no Include, an earlier file, a Match block — `rep.bundle.sshd_dropin_not_in_force`, not a refusal), and an undone bundle reloads sshd again; the drop-in comment cites `runbooks/break-glass.md`; the installer's uninstall warns when the reload fails. Tests RootKeyOnly 6, the two new ones red-proved; installer tests 40/40 with the patch. Files: `audits/night-2026-10-23/P3-review/felhom-agent-bundle-D4-v2.patch`, `felhom.eu-installer-D4-v2.patch` (both apply to tonight's main). A box whose own sshd config is already broken still refuses the bundle (`sshd -t`) — kept: a reload of a broken config is worse. Live on the test box: password SSH refused, key works, web password 200 (by hand, the bundle's steps). | the next agent release + installer tag | Apply the v2 patches with the next agent release and installer tag (both halves together); check on demo-hp after delivery | CC | | **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC | | **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC | | **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC | -| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC | +| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC | | **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator | | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC | | **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement | — | — | CC | @@ -285,7 +280,6 @@ stopping line that lies. | **R-1222** | Monitoring & notifications | P3 | **A household that sets an EVENING backup time gets a false „database dump missed” error every morning.** REAL in code (night 2026-10-22; split from R-1189 (e)): the hub's 05:00 check wants a dump since Budapest midnight (`hub/internal/monitor/deadline.go:416-419, 555-565`), so an evening dump (the banner suggests evening times) is always „yesterday”; on 2026-10-25 the check also runs at 04:00 (R-1183). Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN | — | Fix: the check wants a dump within the last 26 h (or since the box's own scheduled time); a hub test with an evening schedule; rides a hub deploy | CC | | **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC | | **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC | -| **R-1230** | Monitoring & notifications | P3 | **A reinstalled box that waits for the household's recovery code was reported „stuck” after an hour: a false operator mail, a needless re-issue of the off-site login, and a controller restart on the box.** Measured on the test night 2026-10-23 (Tester4-16fdef, `audits/night-2026-10-23/P4-testnight/`): the box declared `offsite.state=awaiting_recovery_key`; the hub's delivery check knew only `needs_credential`, read „consumed, never applied” and at 20:16 UTC mailed `offsite_delivery_stuck` and re-staged the password (the config bump restarted the controller — one 502 on the outside probe). The controller's own comment said this branch is skipped — false. | READY — hub 0.167.0 source `14388e21`: the checker stands down for that declared state; `TestR1230_AHeldBoxIsNotStuck`, red-proved. | the hub deploy (the operator's yes) | Deploy hub 0.167.0; close then | CC | | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC | @@ -359,8 +353,7 @@ stopping line that lies. | **R-1152** | Process & tooling | P4 | **felhom CI's 5-minute job timeout is inert** (act_runner 0.6.1 does not enforce a job-level `timeout-minutes`, R-1136), so a hung step runs up to the runner's 3 h. Move it to a step-level `timeout-minutes` on each workflow's gate step (e.g. 20 min for felhom.eu, which runs ~7 min), measured first: a step timeout fails the job AND the `if: failure()` alarm still runs. Needs a scratch repo (the admin token cannot create one — the operator, or a token with `write:user`). Found 2026-10-18. | OPEN — filed 2026-10-18 | — | Measure on a scratch repo, then the four workflows | CC | | **R-1168** | Process & tooling | P3 | **The security scanner only warns: a newly committed secret or a called vulnerability is listed and the push goes on.** R-986 shipped the gate warn-first on purpose (2026-10-19, `scripts/security_scan_gate.py`): every hit is printed, none refuses. Turning it into a refusal needs a quiet period first (no unexplained WARN for some weeks), and a decision on what refuses (a new gitleaks hit; a called vulnerability with a fix available). Also: CI has no Go, so govulncheck runs only in the pre-push hook. | OPEN — filed 2026-10-19 | `audits/night-2026-10-19/P7/` | After ~2 quiet weeks: a `FELHOM_SCAN_STRICT` mode that exits 1 on a new hit, and Go (or a binary govulncheck mode) in CI | CC | | **R-1174** | Process & tooling | P4 | **The status gate becomes a refusal on 2026-10-27.** `scripts/status_view_gate.py` (2026-10-20) checks root `STATUS.md` on every push: at most 50 lines and 6,000 characters outside the decide table, one „Updated" paragraph of at most 3 sentences with no „Earlier" chain, English outside quotes, every cited R-n an open row, no decided, closed or done row in the decide table, the P1 row under „What is broken" or „decide", and every open row whose Owner names the operator under „decide" or „do". Until `SWITCH_DATE` (2026-10-27, one constant in the gate) a conviction prints a WARNING and exits 0; from that date it exits 1. INCONCLUSIVE (no sections, no register) exits 2 at once. Decoys: `scripts/test_status_view_gate.py` (39 cases; each check red-proved by removing it, the date red-proved on both sides), run by `test_gate_decoys.py` and `script_tests_gate.py`. | WATCHING — filed 2026-10-20 | the date | On or after 2026-10-27: see one real refusal (exit 1 on a page that breaks a check — a hand run on a copy is enough), write it here, clear the due check, close | CC | -| **R-1225** | Process & tooling | P3 | **The website's minimum machine (8 GB) cannot run the photo app, and the download page does not say so.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, W2) of the D5 patch: the estimator (felhom.eu `37cb83c1`) says Immich alone „does not fit” on 8 GB; the patch says only „check memory on the estimator”. | OPEN | the operator's wording (a product promise) | Operator: add one clause to the D5 hardware section („8 GB runs the basics; the photo app needs 16 GB”) before the publish, or keep it as is | operator | -| **R-1226** | Process & tooling | P2 | **Two FAQ answers say the household's data leaves the home only encrypted (the remote backup) — not true for data reached from outside.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, W3; pre-existing, not from the D3/D5 patch): `13` §6 accepts that Cloudflare can read tunnel traffic (R-904), and the box reports to the hub. The same answers were just rewritten for „the whole truth” (D3). | OPEN | the operator's wording (a product promise) | Operator: one sentence („data you open from outside the home passes Cloudflare's network, readable there; the backups leave only encrypted”) — CC adds it to the website patch on the word | operator | +| **R-1236** | Process & tooling | P3 | **Gitea (the code and the registry) is OOM-killed at its 1 GiB limit — 4 restarts in 17 days, the last at 06:41:10 UTC on 2026-10-23, six seconds after the global floor moved three boxes to a new controller image.** Read 2026-10-23 (`kubectl get pod`: `lastState.terminated.reason OOMKilled`, exit 137; kernel: „git invoked oom-killer”; limit `memory: 1Gi`, request 256Mi; 620 Mi in use at rest). During a restart a box update, an agent package fetch, a CI clone or a push fails. Not changed: DooPlex's configuration is the operator's. | OPEN | the operator's yes (a DooPlex change) | Operator: raise the limit (e.g. 2 GiB; DooPlex has the RAM) — CC's pick; or keep it and accept a restart in release hours | operator |