diff --git a/CONTEXT.md b/CONTEXT.md index 1aef5d52..309c02cd 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -63,6 +63,13 @@ > STATUS:** the default limit's size (CC: 50 % of the measured upload) and who is told (CC: the family only about the > qBittorrent limit they can change; the unexplained alarms to the operator only — the R-388 rule). +> **2026-10-24 (evening, unattended) — review round 2** (`audits/review-round-2-2026-10-24/README.md`). The 17 architecture +> documents checked against `main`: facts corrected at `file:line` (EDITS.md), rulings and promises left for the operator +> (R-1266, D45), undocumented decisions written as notes (R-1265, D44). New: **R-1251 (P1)** a cut restore of a drive-backed +> app is put back over newer data (shown with the real `Recover`); **R-1252 (P2)** a rejected backup block makes the +> second-drive reconcile delete the household's copies (reproduced on 9202). Controller `b01bb4b` (unreleased, next +> v0.343.0): a media restart skips household-stopped apps (R-721); the FileBrowser SSO header file appears whole. + > **2026-10-23 ~12:00 — operator rulings by starting the D39 brief (`09` §3 decision 255; R-1239 closed).** **D39 = B:** a > `security_fix` ladder step — box walk on 9202 required, bench optional, taken by the night leg. **No operator word per > incident:** a public advisory id (CVE or GHSA) + its https:// notice is the authority; the session that writes the step diff --git a/REPORT-review-round-2-2026-10-24.md b/REPORT-review-round-2-2026-10-24.md new file mode 100644 index 00000000..033a4a79 --- /dev/null +++ b/REPORT-review-round-2-2026-10-24.md @@ -0,0 +1,33 @@ +# REPORT — review round 2 (2026-10-24, unattended): the documents against the code, data loss and security + +**Plain words first.** +- The 17 design documents were checked against the code: 493 claims, 121 out of date. About 130 places are corrected or + marked with a dated note; no rule, fence or promise was changed. +- About 30 things the code does that no document named are now written down for you to confirm (D44), and seven places + where a document's ruling or promise says something the code does not do are waiting for you (D45). +- The worst data-loss finding: a power cut during a restore of an app on a drive can later pour old data over the + household's new data (R-1251, P1). Proven with the real recovery code; it needs a fix in an attended run. +- Also proven, on the scratch box: one wrong line in an app's backup settings deletes that app's second-drive copy + (R-1252, P2). +- The worst security finding: the „root logs in by key only" SSH rule also reaches a host the household owns (BYO), + where we promised to change nothing (R-1257, D43). Nothing serious was found in the hub's or the controller's new routes. + +**Evidence and the full tables:** `documentation/audits/review-round-2-2026-10-24/README.md`; every document edit with +its `file:line` in `EDITS.md`; the reviewers' candidates in `reviewers/`; the two proofs in `proofs/`. + +**Commits (all CI green by hash):** controller `b01bb4b` (two small fixes, red-proved; unreleased) — CI 2294 / #480; +controller `2449223` (REUSE.md); felhom-agent `f5c3d77` and catalog `227a544` (the rule-file alignment); felhom.eu +`77015997` (documents batch 1) and this commit (batch 2, the register, STATUS, CONTEXT). The CI numbers of the last four +are in the final report. + +**Register:** before 190, after 207; opened 17 (R-1251–R-1267), closed 0. + +**Instruction files edited (facts only, named here):** `.claude/rules/unprompted-work.md` in felhom-agent, the catalog +and the workspace root — the missing „A version number is pushed once" paragraph (before 95 lines, md5 3b071ad2; after +byte-identical to the other two, b1891bd4); `felhom.eu/.claude/rules/live-probes.md:41` — `/opt/felhom/stacks/` → +`/opt/docker/stacks//app.yaml` (the real path, and the folder stays after a clean remove). + +**System poster:** the facts file and the html changed together (text only, the hub-DB push times); no new drawing. + +**Machines:** 9202 used for the R-1252 proof and put back (one difference: Immich's new empty volume, R-1267). DooPlex, +ep0, household boxes and the hub: not changed. No release, deploy, vouch or publish. diff --git a/STATUS.md b/STATUS.md index d7d0e956..155f2ed1 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-24 (afternoon, attended).** The two DooPlex backup alarms had one cause: yesterday's reboot changed which password the database takes, so the database copy job could not log in, and the Gitea off-site copy then refused its old copy. Fixed and cleared at 12:51 local; critical alarms now repeat every 4 hours (all backup alarms in one mail), warnings every 12. A box whose torrent app seeds all day can now get its weekly line test (proven on demo-hp, not yet released). +**Updated 2026-10-24 (evening, unattended review).** The 17 design documents were checked against the code; about 130 old facts are corrected, no rule changed. Two new data-loss holes are proven; both need a fix in an attended run. Two small controller faults are fixed, not released. ## What works - **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241). @@ -18,6 +18,7 @@ - The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15). ## What is broken +- **P1 (new):** a power cut while an app on a drive is restored can later pour old data over new data (R-1251); one wrong line in an app's backup settings deletes its second-drive copy (R-1252, P2). - **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours). - Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). The scratch box cannot use agent features (R-1083). - DooPlex: something sets the shared checkout to „bare" (R-935). @@ -31,6 +32,9 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| +| D45 | R-1266 | Six places where a design document says one thing and the code does another (two update rules, the golden check, „the hub never connects into a box", „the operator cannot open the off-site copy", the tunnel plan). Should the text follow the code? | **Yes — the text follows the code** (each is how the system works today; you read every changed sentence) | the code follows the text: six code changes, one of them to the box's door for you | the documents keep saying what is not true | — | +| D44 | R-1265 | Confirm about 25 behaviours the code has but no document named (now written as notes), and that MeTube's downloads are not backed up? | **Yes, all** (each was read in the code; none adds risk) | name the ones to change; each becomes a row | the notes stay marked „undocumented" | — | +| D43 | R-1257 | The SSH rule „root logs in by key only" also reaches a box the household owns (a BYO host), where we promised to change nothing on the host. Keep it there too? | **No — skip it on BYO hosts** (it is their machine; we said we would not touch it) | keep it: a BYO owner who uses a root password over SSH loses it at the next update, and the installer screen must say so | it reaches every BYO host at its next update | — | | D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | | D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — | | D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — | diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 109e0636..a838f3b0 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -246,7 +246,7 @@ | [G12](capability-map-detail/G.md#g12) | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | — | the 2026-07-13 hub closing bundle (no evidence document) | | [G13](capability-map-detail/G.md#g13) | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | — | hub v0.54 (2026-07-13; no evidence document) | | [G14](capability-map-detail/G.md#g14) | The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail | hub v0.135.0 (R-604, R-530) | **IMPLEMENTED** | the page live; the alarm and the mail unit-proven | `audits/hub-safety-2026-10-05/partD/` (2026-10-05) | -| [G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) | +| [G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a consistent copy every 6 hours since D34 (nightly before 2026-10-23; corrected 2026-10-24), encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) | | [G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) | | [G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) | | [G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) | diff --git a/documentation/architecture/02-controller-module-map.md b/documentation/architecture/02-controller-module-map.md index 8e578740..573d11f8 100644 --- a/documentation/architecture/02-controller-module-map.md +++ b/documentation/architecture/02-controller-module-map.md @@ -105,7 +105,9 @@ Tri-state: `""` (unknown) · `"running"` · `"stopped"`. ### In-flight — a marker file, one per owner -Two exist and they are deliberately **separate files**: `quiesce-state.json` (the whole-guest backup +Two exist and they are deliberately **separate files** *(four in-flight records by 2026-10-24: also `update-journal.json` +— the guarded update's crash journal read by `RecoverUpdates`, `internal/stacks/update.go:49,1306` — and +`.felhom-install-pending` — an interrupted install, R-681, `install_interrupted.go:28`; noted 2026-10-24)*: `quiesce-state.json` (the whole-guest backup window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app — `backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's @@ -169,7 +171,8 @@ A recovery completing after the grace emits a `LATE RECOVERY` warning naming the **Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders -answer it, and the last two only became reachable once the window widened past T+5 s: +answer it *(four since slice 4: check „1b", a guarded update moving the app — `IsUpdating`, `cmd/controller/main.go:2459-2477`; +noted 2026-10-24)*, and the last two only became reachable once the window widened past T+5 s: | Holder | Why starting would be wrong | |---|---| @@ -218,6 +221,13 @@ own; every caller that is not the customer must decide for itself whether the ap --- +> *Dated note 2026-10-24 (review round 2): this inventory is v0.33's. On `main` (`321df1f`) 22 packages under +> `controller/internal/` have no row here: `aftercrash`, `agentapi`, `appbackup`, `backupwindow`, `bootrecon`, +> `bootstrap`, `channelhealth`, `crashboot`, `dockerexec`, `family`, `fillwatch`, `i18n`, `infra`, `logx`, `mailrelay`, +> `mediawire`, `nightchain`, `offsiteapply`, `privpolicy`, `quiesce`, `sockheal`, `upwatch` (an `ls` of the tree). Each +> package's own doc comment is its description; `sockheal` (re-heal after the guest's Docker socket is recreated, +> R-860) and `channelhealth` (a periodic proof of the controller→agent channel) are in no architecture document.* + ## 2. Classification table (per package/file) ### `cmd/` @@ -266,6 +276,11 @@ own; every caller that is not the customer must decide for itself whether the ap |---|---|---|---| | `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework | +*As built (noted 2026-10-24, review round 2): the `cloudflare/` package is still wired — with `cf_api_token` set the +controller builds a `GeoSyncManager` and calls the Cloudflare API (`cmd/controller/main.go:1992-2000`; the field is still in +`config.go:160`). The DELETE classification above is the plan, not the state; whether any box sets the token is not +readable from source.* + ### `config/`, `crypto/`, `util/` | File | Class | Reason | Risk | |---|---|---|---| @@ -325,6 +340,7 @@ household's file names. The household's lost list is shown on the app page (`los | File | Class | Reason | Risk | |---|---|---|---| | `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean | +| *(as built 2026-10-24: both files below are LIVE — the updater decides the version and the floor and hands the container swap to the agent, `AgentSwapper`, `internal/selfupdate/updater.go:38-47`; `state.go` keeps `update-state.json`)* | | | | | `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean | | `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean | @@ -387,7 +403,8 @@ dashboard alert is an `Alert` with two placement fields: `PageOnly` (the pages i page) and `Inline` (rendered by the page template in place, not by the layout's banner). `GetAlerts` returns the list (endpoint-drift, agent-channel and dead-app alerts first, then the rest sorted error > warning > info, capped at five plus an overflow line), and `layout.html` paints only those that are not `Inline` and match the -page; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones. +page *(as built since R-906 — noted 2026-10-24: pages use `GetBannerAlerts(page, lang)`, which filters per page BEFORE the +cap, `internal/web/alerts.go:353`, `handlers.go:144`)*; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones. **Exactly one warning is inline today:** the „storage is not on a separate drive" health warning. It is `PageOnly: dashboard, monitoring` and `Inline: true`, so it sits quietly under the storage bars; **every other @@ -821,7 +838,7 @@ R-438. The three facts below are the ones a reader needs before touching any of `Syncer.copyTemplates` (`sync/sync.go:319`) walks every directory in the catalog cache and copies `docker-compose.yml` and `.felhom.yml` into the matching stack folder. **There is no test of whether the app is deployed.** The only guard is a sha256 content compare (`copyIfChanged`) and the only -exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351`), plus +exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351` then, `:399` now; `copyTemplates` is now `sync.go:436`), plus one immediate sync at controller start (`sync.go:98`). **SINCE v0.235.0** this walk still happens and `.felhom.yml` is still copied unconditionally, but the @@ -835,6 +852,7 @@ and they stay that way until something else acts. ### 2. Every lifecycle path resolves that disagreement, silently, by upgrading +*(This section is the 2026-09 measurement; `UpdateStack` was deleted in v0.237.0 — the button calls `StartGuardedUpdate`, `09` §6.1 — noted 2026-10-24.)* `StartStack`, `RestartStack` and `UpdateStack` (`stacks/manager.go:1029 / 1133 / 1170`) all end in `docker compose up -d`. `up -d` makes the container match the file, **and pulls the image itself if it is absent** — measured at 18.3 s with a pull versus 0.5 s without, against a negative control @@ -887,7 +905,9 @@ measured; that the restore writes to that path is read, not measured. beállítások csak olvashatók.", `deploy.html`). This is a design, not an oversight: a changed value would need a guarded re-deploy that nothing performs. **Finding (2026-09-15):** the catalog field flag `locked_after_deploy` (`stacks/metadata.go`) is parsed and read by **no** controller code — every field -is read-only after install whatever the catalog says. So a page must never tell a household to change +is read-only after install whatever the catalog says. *(Dated note 2026-10-24: it is now read into the app's +`LockedFields` (`internal/stacks/deploy.go:106,534`) and enforced only by `UpdateStackConfig`, which nothing calls — so +„read-only whatever the catalog says" still holds in practice.)* So a page must never tell a household to change a value after install. **Vaultwarden (R-512)** follows the design instead of breaking it: registration is closed by default and the household is invited from the admin panel, measured to work without mail (stranger 400, invite 200, invited 200). diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index 98e0c79a..b8e65467 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -308,6 +308,18 @@ A controller can only `POST /rollback` (or snapshot/backup) **its own** guest token → guest and authorizes per guest, so a compromised controller's blast radius is **self-scoped and bounded** to its own guest. +**[DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through R-1265]:** the surface listed above is +the slice-8 design; on `main` the local API registers **41** routes (`felhom-agent/internal/localapi/server.go`; 24 of +them are not listed above — the backup tiers and target, disk candidates, decommission, format status, guest attach, +the network storage routes, the controller swap, the escrow stage/preflight/ceremony routes, the debug logs, guest +memory). Among them the controller can start a HOST action: `POST /host/line-test/run` (about 55 s; bounded by the due +date, the window, 6 tries a day and the heavy-operation gate — R-1250, unreleased). The local API also listens on the +R-50 island bridge (host `169.254.253.1`, guest `net1` `169.254.253.2/30`, `internal/config/config.go:260-269`). The +weekly line test keeps its state in the OOB state dir (`line-test.json`) and has an off switch, `line_test.disable`. +The served WireGuard block carries `oob_peer_ip` and `oob_operator_ssh_key` (`internal/hub/report.go`). The D30 re-key +runs only inside the tunnel watchdog, and its 15-minute count lives in memory — an agent restart starts it again (the +once-an-hour bound survives in `wg/last-rekey`). + ### 6a. Implementation (slice 8A — implemented) **Status: implemented** (agent v0.10.0 `internal/localapi`; controller v0.35.0 `internal/bootstrap` diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 5dbef462..830be65e 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -253,7 +253,7 @@ It carries guest sizing, `pve_storage[]`, and an app half with per-app `storage_ the identity bundle's shape is `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}` (`felhom-agent/internal/escrow/identity.go:26-39`). -**[DESIGN] Off-site deletion custody (decisions 68–69, 2026-10-03) — BUILT hub v0.127.0 / controller v0.289.1, live on both demo boxes.** The restic repository password stays on the box only. The box's off-site key is append-only (pinned in `authorized_keys`, written by the hub — the key registrar); the box never receives the sub-account password, which the hub stores encrypted at rest. Old snapshots are pruned by the box itself, only in a weekly window the hub opens, behind a fake-snapshot guard (R-822); the window has run live on both demo boxes since 2026-10-05 (R-95, closed). The guard's residual — past-dated fakes steering the keeps, and a window check that trusts the box's own counts — is R-822 and R-895. **Hub v0.158.0 (built 2026-10-18, deploy waits on the operator) adds the hub's own check:** the 07:10 key audit compares the repository's `snapshots/` listing, over the hub's own login, with yesterday's — files gone that no clean-up window explains raise `offsite_snapshots_vanished` (error), files no reported box run explains raise `offsite_snapshots_unexplained` (warning); both operator-only (R-895 slice 2). A Felhom-side pruner holding repository passwords is rejected. +**[DESIGN] Off-site deletion custody (decisions 68–69, 2026-10-03) — BUILT hub v0.127.0 / controller v0.289.1, live on both demo boxes.** The restic repository password stays on the box only. The box's off-site key is append-only (pinned in `authorized_keys`, written by the hub — the key registrar); the box never receives the sub-account password, which the hub stores encrypted at rest. Old snapshots are pruned by the box itself, only in a weekly window the hub opens, behind a fake-snapshot guard (R-822); the window has run live on both demo boxes since 2026-10-05 (R-95, closed). The guard's residual — past-dated fakes steering the keeps, and a window check that trusts the box's own counts — is R-822 and R-895. **Hub v0.158.0 (built 2026-10-18; deployed the same day on the operator's yes — corrected 2026-10-24) adds the hub's own check:** the 07:10 key audit compares the repository's `snapshots/` listing, over the hub's own login, with yesterday's — files gone that no clean-up window explains raise `offsite_snapshots_vanished` (error), files no reported box run explains raise `offsite_snapshots_unexplained` (warning); both operator-only (R-895 slice 2). A Felhom-side pruner holding repository passwords is rejected. **[DESIGN, built hub 2026-10-07 — R-32 option A, `09` §3 decision 167] What RESET does to the household's off-site copy.** On the shared pool box a sub-account is a LOGIN, not the data: deleting it leaves its folder (measured @@ -380,7 +380,7 @@ never moved by an update or an undo, and the unit restore accepts it. | Tier | Location | Captures | Cadence (LIVE) | Retention (LIVE) | Encrypted | |---|---|---|---|---|---| -| **Tier-1** recovery unit | `/backups/primary//` on the app's own drive | compose + `.felhom.yml` + **secret-stripped** `app.yaml`, `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no | +| **Tier-1** recovery unit | `/backups/primary//` on the app's own drive | compose + `.felhom.yml` + `app.yaml` (0600 — since D5 / schema 2 it carries the PORTABLE secrets, `recovery_unit.go:25`; „secret-stripped" corrected 2026-10-24), `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no | | **Tier-2** cross-drive | `/backups/secondary//` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/` + `userdata/` | nightly at W+60m | mirror (rsync) | no | | **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) | | **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/felhom` (its `docker/` and `sys_drive/` folders, bound in the guest to `/var/lib/docker` and `/mnt/sys_drive` — §7.5.1; `mp1` no longer exists, corrected 2026-10-15) | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` → `--prune-backups keep-daily=3`: one copy per day, 3 days (keep-last until agent 0.160.0, R-1149) | **no** | @@ -437,7 +437,7 @@ container must keep its id). Ring 1 never takes it in the night leg. Measured: t **[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered -(`cmd/controller/main.go:604-607`). Both boxes run W = `02:30`. +(`backupwindow.LegTimes`, called at `cmd/controller/main.go:1258-1264`; line corrected 2026-10-24). Both boxes run W = `02:30`. **[FACT, controller v0.271.0, 2026-09-25] A fourth leg: automatic app updates, CHAINED after the off-site leg** (`09` §3 decisions 11 and 20, §6.4 part 7). The `offbox-backup` job runs the update leg when its off-site @@ -448,8 +448,10 @@ night the whole-guest backup starts later inside its own window; it is never ski numbers: `audits/DRILL-night-2026-09-25.md` Part D. **[FACT, 2026-09-25] demo-hp's local whole-guest target is its host ROOT disk; retention there is now -keep-last=1** (operator ruling 2026-09-24, option A; R-684). PVE prunes AFTER a successful backup, so the target -must hold keep-last + 1 archives during the run — measured: a 9201 archive is 8.18 GB, the root disk 39 GB. The +keep-last=1** (operator ruling 2026-09-24, option A; R-684) — *since R-1149 (operator D17, 2026-10-19) the local tier's +spec is `keep-daily=N` (`felhom-agent/internal/config/config.go:553-559`), so demo-hp is keep-daily=1; corrected +2026-10-24*. PVE prunes AFTER a successful backup, so the target +must hold N + 1 archives during the run — measured: a 9201 archive is 8.18 GB, the root disk 39 GB. The warning that should precede such a failure is R-685. **[FACT, 2026-10-14] A power cut during the whole-guest backup leaves a half-written archive, and from agent v0.156.1 the @@ -731,7 +733,8 @@ definition is written or a named volume replaced — a failed placement, volume the live definition back and HOLDS the app stopped (`restore_mixed`, operator-cleared); the household reads that it needs our help; the operator gets a `backup_run_failures` line. The hold is persisted before the stop. Placed files ALONE (no version change, no volume replaced) still restart after a successful rollback (`TestR379_ScenarioA`): that -mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893. **`[FACT]` +mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893 *(shipped since: +D20 in controller v0.336.0 and D22 in v0.338.0, below — noted 2026-10-24)*. **`[FACT]` 2026-10-19 (measured on 9202, controller 0.335.0, `audits/night-2026-10-19/P3/`):** a forced replay failure after the volume was replaced left the NEWEST rows (the undo dump) on top of the backup's volume — a table only the backup had stayed (37 tables, the live app had 36); the definition did not move (same version); the app was held and a start @@ -753,7 +756,7 @@ the app running, no hold, the copy (69 MB, 0.9 s) gone; a normal restore of the known limit above is closed for named volumes; placed FILES are still not taken back (the plain-files mix, as ruled). **`[DESIGN]` 2026-10-21 — the put-back covers EVERY restore, and a power cut (operator ruling D22 = yes, 2026-10-21 -20:05; controller `main`, unreleased).** (1) **The local unit restore** — the app's own unit, the second drive's unit, +20:05; released in controller v0.338.0 — corrected 2026-10-24).** (1) **The local unit restore** — the app's own unit, the second drive's unit, the second drive's „Teljes visszaállítás" and a kept load all run `RestoreFromRecoveryUnitAtWith` — now takes the same measure → copy → put-back (`offbox_restore_undo.go`, one implementation, no second copy of the logic): the copy must fit beside the unit before anything stops; ANY failure after the stop (volume replay, definition write, DB-only start, DB @@ -973,6 +976,7 @@ both. Files written under different pins (a failed leg) make the unit `mixed`. |---|---|---| | own unit (Tier 1), second-drive mirror (Tier 2), kept-data Load | present, `compose/` names the data's pins | the data starts under that definition; if it differs from what ran, the restore pages' first sentence names the backup and the version (the kept-data Load has its own sentence and does not add this one) | | same | present, `compose/` names OTHER pins, or `mixed` | **refused before anything is touched** (`ErrUnitVersionMismatch`): „…ezért a visszaállítás nem indult el. Az alkalmazás érintetlen" | +| off-site (Tier 3) | present, `mixed`, or a definition that does not belong to its data | **refused before anything is touched** like the rows above (`unitVersionCheck`, `offbox_reconstitute.go:826-833`; row added 2026-10-24) | | off-site (Tier 3) | present, the snapshot's version differs from what runs | the snapshot unit's definition is written into the stack dir (and pinned) right after the stop, before any file, volume or database is touched; the database service is resolved from it | | any | absent — a unit or snapshot written before v0.275.0, or with an unstamped file | restores **as before** (its captured definition; the off-site path the live one), with a WARN naming it | @@ -1341,12 +1345,13 @@ Added 2026-08-02 from `audits/SPIKE-recovery-unit-space-2026-08-02.md` and which is true and is the whole story for a drive-resident app. **It is not the whole story for an app with no data drive, and that case was undocumented until now.** -**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go:245-255`) +**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go`, now near line 441) returns the app's `HDD_PATH` if it has one and **`systemDataPath` otherwise** — what `internal/appbackup/paths.go:26-27` calls *"the SSD-only system-data fallback"*. Nothing deletes a unit -after it is copied onward: the only prune is F5 (`backup.go:1053-1112`), which removes residue on an +after it is copied onward: the only prune is F5 (`pruneStalePrimaryDirs`, `backup.go`, now near line 1514), which removes residue on an **old** drive when an app **moves**. So for every app without a data drive, `mp1` holds the kept copy -indefinitely. +indefinitely. *(Dated note 2026-10-24: `mp1` no longer exists — §6.1 / line 386; the system data now lives under the +one `mp0` volume's `sys_drive/` folder, `felhom-agent/configs/build-golden.sh`.)* **What a unit contains, which bounds the problem** (`internal/backup/recovery_unit.go:20-25`): `compose/` + `db-dumps/` + `volume-dumps/` (named-volume tars) + `manifest.json`. **It never contains @@ -1495,7 +1500,9 @@ crosses the line — **R-158**. > Eleven rows carry a blank RTO — 4, 5, 8, 9, 10, 11, 11b, 12, 13, 14, 15 — and only six of those are > deferred work. **Row 5** has no recovery to time (a derived copy), **row 13** has no route by > design, **row 14** is proven and merely never stopwatched, **row 11b** is a note not a row, and -> **row 15 is an open DEFECT (R-104) that this line does NOT cover.** +> **row 15 was an open DEFECT (R-104) that this line did NOT cover** — *R-104 was CLOSED 2026-10-05 (controller +v0.297.0: a lock that survives the self-heal gets the class `locked`, its own note and a retry, `offbox.go:156-162`; +CLOSED-ITEMS.md; corrected 2026-10-24).* > > **NO STATUS MOVED ON THE DAY THIS WAS WRITTEN, because nothing was proven that day.** A stopping > line that promotes a row is a stopping line that lies. Full text, and the conditions that reopen @@ -1528,13 +1535,13 @@ crosses the line — **R-158**. | 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human | | 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** **`[BETA-DEFERRED]`** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) | | 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** **`[BETA-DEFERRED]`** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) **`[FACT]` 2026-10-03 (decision 70): ep0's datastore now has a nightly copy on DooPlex** (PBS pull-sync `ep0-felhom-offsite`, `remove-vanished false`, weekly verify, failures mailed to the operator); first pull 201 s, 12 GB, 4 of 4 snapshots, matching ep0 namespace for namespace. Restore route: `runbooks/ep0-datastore-copy.md` — never walked. `audits/offsite-lock-build-2026-10-03/partF/`. **`[FACT]` 2026-10-04: the restore route from the DooPlex copy was WALKED** on demo-hp (scratch VMID): listed in 2 s, restored in 186 s (15 GB logical), data read by `pct mount`; the restored config carries `onboot: 1` and the host's real drive binds — strip them on a restore beside the original (R-834). The copy now keeps 8 weekly copies (decision 71). `runbooks/ep0-datastore-copy.md` route 1. | -| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recovered, so nothing was timed.** `audits/evidence-drill-r95-recovery-2026-09-01/`. **`[FACT]` 2026-10-03 (decisions 68–69, hub v0.127.0, controller v0.289.1): THE BOX CAN NO LONGER DELETE ITS RESTIC HISTORY.** Both demo boxes run on an append-only key the hub pinned; a delete from the box is refused (`403 Forbidden`, measured on each, counts 13→13 and 100→100); the box never receives the sub-account password (the old endpoint answers 410, measured with demo-hp's own key); the hub reads every key file daily. Retention runs only in a hub-opened window behind a fake-snapshot guard — mechanics proven live (window 1 on demo-hp: opened, guard refused, closed in 3 s, operator mailed); **a real prune inside a window is NOT yet proven** (R-824). Residual: an add-only attacker can still fill the quota and plant past-dated snapshots (R-822). `audits/offsite-lock-build-2026-10-03/`. **`[FACT]` 2026-10-04 (controller v0.290.0, hub v0.128.0):** the guard now skips same-day-superseded young snapshots instead of refusing; window 2 on demo-hp ran without refusal and removed nothing (127→127 — every candidate was a young same-day copy); weekly windows are ON fleet-wide. **A window that actually removes snapshots has still not been observed** (first candidates age past 8 days around 2026-10-11). The household's set-aside deletion is the hub's after 7 days (decision 74), proven live on a planted scratch dir. **`[FACT]` 2026-10-05 (controller v0.294.0, R-867 CLOSED): A WINDOW THAT REMOVES SNAPSHOTS IS OBSERVED, on both demo boxes.** The guard's line is now built from the policy's own constants (keep-daily 7 CALENDAR days, `09` decision 104) — the 8-day age line sat inside the keep window and refused every honest window (demo-felhom 2026-10-05 02:15 UTC, an error mail). One window each, opened by the operator's one-shot grant and run by the household's "run now": demo-felhom 16 → 14, demo-hp 145 → 127 — the removed snapshots are exactly those the policy predicted; the hub's rows read `pruned` with the same counts, no drop event, no mail, the key files clean. The UNATTENDED weekly window has not yet been observed removing (next due ~2026-10-11/12). Residual unchanged: R-822 (past-dated gap-fills steer older keeps). `audits/night-fixes-2026-10-05/partA/`. **`[DESIGN]` 2026-10-08 (R-435, `09` §3 decision 191; hub main, unreleased): on a pinned tier even ONE snapshot that disappears outside a hub-opened window raises an error mail** — each window explains at most its hub-set cap; NAS tiers keep the half-rule; the count is the box's own and net (`08` §6.5). **2026-10-08 (R-893, decision 192; controller main, unreleased):** a failed off-site replay of one app after a moved definition or a replaced volume now HOLDS the app stopped for support instead of starting it on mixed data (§6.3); „put back exactly as it was" is the next slice. **`[DESIGN]` 2026-10-18 (R-973; agent `main`, unreleased):** re-running the whole-guest backup to push out older restore points is capped per guest and tier (§6.1, the R-973 note); the local tier stays floodable within a day (R-1149). | +| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recovered, so nothing was timed.** `audits/evidence-drill-r95-recovery-2026-09-01/`. **`[FACT]` 2026-10-03 (decisions 68–69, hub v0.127.0, controller v0.289.1): THE BOX CAN NO LONGER DELETE ITS RESTIC HISTORY.** Both demo boxes run on an append-only key the hub pinned; a delete from the box is refused (`403 Forbidden`, measured on each, counts 13→13 and 100→100); the box never receives the sub-account password (the old endpoint answers 410, measured with demo-hp's own key); the hub reads every key file daily. Retention runs only in a hub-opened window behind a fake-snapshot guard — mechanics proven live (window 1 on demo-hp: opened, guard refused, closed in 3 s, operator mailed); **a real prune inside a window is NOT yet proven** (R-824). Residual: an add-only attacker can still fill the quota and plant past-dated snapshots (R-822). `audits/offsite-lock-build-2026-10-03/`. **`[FACT]` 2026-10-04 (controller v0.290.0, hub v0.128.0):** the guard now skips same-day-superseded young snapshots instead of refusing; window 2 on demo-hp ran without refusal and removed nothing (127→127 — every candidate was a young same-day copy); weekly windows are ON fleet-wide. **A window that actually removes snapshots has still not been observed** (first candidates age past 8 days around 2026-10-11). The household's set-aside deletion is the hub's after 7 days (decision 74), proven live on a planted scratch dir. **`[FACT]` 2026-10-05 (controller v0.294.0, R-867 CLOSED): A WINDOW THAT REMOVES SNAPSHOTS IS OBSERVED, on both demo boxes.** The guard's line is now built from the policy's own constants (keep-daily 7 CALENDAR days, `09` decision 104) — the 8-day age line sat inside the keep window and refused every honest window (demo-felhom 2026-10-05 02:15 UTC, an error mail). One window each, opened by the operator's one-shot grant and run by the household's "run now": demo-felhom 16 → 14, demo-hp 145 → 127 — the removed snapshots are exactly those the policy predicted; the hub's rows read `pruned` with the same counts, no drop event, no mail, the key files clean. The UNATTENDED weekly window has not yet been observed removing (next due ~2026-10-11/12). Residual unchanged: R-822 (past-dated gap-fills steer older keeps). `audits/night-fixes-2026-10-05/partA/`. **`[DESIGN]` 2026-10-08 (R-435, `09` §3 decision 191; hub main, unreleased): on a pinned tier even ONE snapshot that disappears outside a hub-opened window raises an error mail** — each window explains at most its hub-set cap; NAS tiers keep the half-rule; the count is the box's own and net (`08` §6.5). **2026-10-08 (R-893, decision 192; released — corrected 2026-10-24):** a failed off-site replay of one app after a moved definition or a replaced volume now HOLDS the app stopped for support instead of starting it on mixed data (§6.3); „put back exactly as it was" is the next slice. **`[DESIGN]` 2026-10-18 (R-973; released in agent v0.160.0 — corrected 2026-10-24):** re-running the whole-guest backup to push out older restore points is capped per guest and tier (§6.1, the R-973 note); the local tier stays floodable within a day (R-1149). | | 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its encrypted database copy on ep0 (6-hourly, `scripts/hub-db-backup`; **since 2026-10-24 the hub volume is no longer in DooPlex's Longhorn backup**, D40 / R-1243) | operator (`kubectl`) | | 6 h (ep0, 14 daily + 8 weekly) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) | | 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** **`[BETA-DEFERRED]`** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) | | 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** **`[BETA-DEFERRED]`** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D **`[FACT]` 2026-10-03:** the PBS (whole-guest) off-site copy now also exists OFF Hetzner — DooPlex pulls ep0's datastore nightly (decision 70). The restic tier is still Hetzner-only. | | 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) | | 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix | -| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** | +| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **FIXED 2026-10-05 (R-104, v0.297.0; was DEFECT — corrected 2026-10-24)** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** | ### 8.2 The ONE exception to row 5's derived-copy rule (R-403, controller v0.230.0) @@ -1599,12 +1606,33 @@ refuses a missing one — then the in-process copy runs). **The container must n fix, 2026-10-23 night): a fixed name and the label `felhom.tier2-copy=1`; on any error, the timeout included, it is removed by name; at controller start, before the scheduler and the web server, every labelled container is removed; and a restore is refused while any second-drive copy runs or a leftover may still run (the other direction of R-1180 — -a `--delete` beside a restore removes what the restore brings back). **`[FACT]` 9202, rc.5 +a `--delete` beside a restore removes what the restore brings back). *(As built — noted 2026-10-24: the refusal is in the +web layer's `restoreOpBlocked` (`internal/web/restore_wizard.go:226`); the install-time „use my kept data" load checks +only `RestoreStatus`/`IsRunning` (`internal/api/kept_install.go:89-92`) — a register row of review round 2.)* **`[FACT]` 9202, rc.5 (`audits/night-2026-10-23/P1-straycopy/`):** a controller restart in the middle of a 16 GB copy → the new controller removed the old container at start; a restore probe during the copy was refused; the redone copy equals the source; a 15-second test limit → the container removed, the partial copy stopped growing. Controller: `internal/backup/tier2_copy_lifecycle.go`. +### 8.2c Decisions in code that no document recorded [DESIGN — undocumented, found 2026-10-24, review round 2 — the operator confirms them through one register row] + +Read at `file:line` on `main` (controller `321df1f`, hub `44d19042`) by the main session: + +- **A second-drive copy is limited to 60 minutes**; a stray copy container that cannot be removed blocks every + restore and is retried every minute; copies into one destination run one at a time and are never killed by name + while running (`felhom-controller/controller/internal/backup/tier2_copy_lifecycle.go:40,113-121,150-172`). +- **The household's „Eltávolítás" of a kept copy (D25) is refused while any backup, restore or second-drive copy runs** + (`internal/web/kept_copies.go:54-58`). +- **A start that fails AFTER a successful put-back** leaves the app stopped and NOT held, keeps the copy, and tells the + household `restore_failed_put_back_not_started` (`internal/backup/offbox_restore_undo.go:245-247`). +- **The hub's off-site delivery checker** (`hub/internal/monitor/offsite_delivery.go`): a box that took no off-site + credential is reported `offsite_delivery_stuck` (operator) and healed by `maybeHeal` (a burned vs a stuck shape); + a box that declares it waits for the household's recovery code (`awaiting_recovery_key`) is skipped — no event, no + heal (R-1230, `:103-110`). +- **The release copy (D33)** — nightly 01:40, the newest 3 image tags and 5 agent packages to ep0 under its own + limited user, capped by the sender (`felhom.eu/scripts/release-copy/`) — has no row in this matrix; its homes are + the poster facts and `runbooks/total-loss-of-dooplex.md`. + ### 8.1 The blank cells, listed explicitly Per the rule that a blank is a finding, here they are: diff --git a/documentation/architecture/08-alarm-ladder.md b/documentation/architecture/08-alarm-ladder.md index a5d9a61c..61f7aa4c 100644 --- a/documentation/architecture/08-alarm-ladder.md +++ b/documentation/architecture/08-alarm-ladder.md @@ -593,7 +593,7 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me 94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`). While the box owns it, qBittorrent's page says: „A qBittorrent feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod." -6. **Limits, stated:** an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from +6. **Limits, stated:** *(added 2026-10-24, review round 2: ANY viewer explains any amount of upload, so a hacked app can send freely while one real playback runs — `upwatch/watch.go:275,299`; R-1262)*; an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container (Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line. diff --git a/documentation/architecture/09-update-architecture.md b/documentation/architecture/09-update-architecture.md index 5ef51c2e..07d6645f 100644 --- a/documentation/architecture/09-update-architecture.md +++ b/documentation/architecture/09-update-architecture.md @@ -65,6 +65,13 @@ build is added to both in the same commit. ### 1.1 The button +> *Dated note 2026-10-24 (review round 2): §1 is the 2026-09 MEASUREMENT before the arc. `UpdateStack` no longer exists +> (deleted in v0.237.0, controller `REUSE.md`); the button calls `StartGuardedUpdate` — backup precondition, safety +> dump, pin journal, undo copy, health wait, undo (`internal/api/router.go:794`; `internal/stacks/update.go`). The line +> numbers below are the measurement's; e.g. `copyTemplates` is now `sync.go:436`, `copyIfChanged` `:700`, the `15m` +> default `config.go:399`, `RestartStack` `manager.go:1454` (it now runs `up -d --force-recreate` and errors when no +> container restarted, R-1026).* + `Manager.UpdateStack` (`felhom-controller/controller/internal/stacks/manager.go:1199`) is two compose commands and nothing else: @@ -267,7 +274,8 @@ builds them (`audits/update-rulings-2026-09-23/`). next night. **Why:** the update then leans on the freshest copy the box ever has, and there is no second window for anyone to set or misread. **Replaces:** §3b Q1's recommendation of a fixed 02:30–05:00. *For the builder:* `stacks.update_window` (`config.go` L195, default `03:00-05:00`) - exists and **no Go code reads it** — it appears only in `config.go` (field + default), + exists and **no Go code reads it** *(dated note 2026-10-24: the field is now removed — no controller source names + it; `controller/README.md` „`stacks.update_window` is removed")* — it appears only in `config.go` (field + default), `setup/handlers.go` L482 (written into a new box's `controller.yaml`) and `configs/controller.yaml.example`. Remove it or fold it into the chain; do not build a second window. @@ -424,7 +432,8 @@ R-636's louder repeated alarm. (a) broke the second. Shipped as controller **v0.269.1** — a second release in the session, instead of shipping (a) to the fleet with the floor. *Note 2026-09-30 (R-740) — the cost line above is not what the code does.* „(or the automatic leg)": the - automatic leg never presses a digest-only change (`unattended.go:433–435`, skip `no_test_record`). And the + automatic leg never presses a digest-only change (`unattended.go:433–435` then, `:449-451` now — line corrected + 2026-10-24 — skip `no_test_record`). And the catalog never writes a re-test of a floating tag at a new digest (the writer refuses a step that moves no image), so a same-name upstream fix reaches no box by either route today. Measured, with the options, in R-740; the decision is the operator's (STATUS 2026-09-30). This decision's ruling — the sync never moves @@ -1981,7 +1990,7 @@ moves a stack dir. | writer | pin source | |---|---| | the deploy path (`runComposeDeploy`) | the template just deployed from | -| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull | +| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull — *as built: `UpdateStack` was deleted in v0.237.0; the guarded update writes the pin (`advancePinTo`, the `pinning` phase before `pulling`), and two more paths move a pin BACK — `pinBack` and the undo's `restoreDefinition` (`internal/stacks/update.go`, `undo.go`; noted 2026-10-24)* | | the restore (`stackAdapter.RecreateStackDefinitionFromUnit`) | the recovery unit's captured compose — **this closes R-441** | | `Manager.AdoptPins` | the observation, once, and only when complete AND matching | @@ -2047,7 +2056,7 @@ earlier feature is the failure mode to look for whenever a file changes meaning. | **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 | | **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | **SHIPPED + PROVEN LIVE, controller v0.237.0 (job) + v0.238.0 (page) + v0.238.1 (2026-09-13); any backup tier since v0.239.0 (§3 decision 8)** — §6.1 | | **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 | -| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | OPEN — R-450; rulings 2026-09-23; build order §6.4 | +| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | ~~OPEN — R-450~~ **BUILT** — the leg in `internal/stacks/unattended.go`, wired into the off-site job; R-450 CLOSED 2026-10-05 (capability map B11) *(noted 2026-10-24, review round 2)* | | **7** | **A fleet sweep pipeline** — the operator can see, and move, how far behind every box is. | OPEN — R-451; ruled 2026-09-23 (decision 18), built later | ### 6.1 Slice 4 as SHIPPED (controller v0.237.0 + v0.238.0, 2026-09-13) @@ -2062,12 +2071,13 @@ closed by construction: nothing reports an update complete on the compose exit c | # | phase | what happens | on failure | |---|---|---|---| | 0 | refusals (409, before the intent is recorded) | held (R-439), busy (backup/restore/app-data op/quiesce), migration, already updating, deploying, memory (the deploy's `memoryVerdict`, releasing the app's own request), disk (**fixed 2 GB floor** — image size unknown without a registry), **no copy on ANY tier and no backup can be taken now** (since v0.239.0; before it, no restorable Tier-2 copy) | nothing moves, nothing is recorded | -| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) | nothing moves | +| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) — *an app whose data lives in bind-mounted files walks second drive → OFF-SITE → own unit (decision 9, R-479, v0.241.0; `internal/backup/update_guard.go` `updateTierOrderBindData`; noted 2026-10-24)* | nothing moves | | 2 | `backing-up` — only when no tier holds a fresh copy | `RunAppBackupNow`: this app's DB dump → volume dump → unit capture (marked proven current) → Tier-2 copy, whose failure is a WARN since v0.239.0 | refused with the backup's own error; nothing moves | | 3 | `safety-dump` | `WriteUpdateSafetyDump` (R-361's undo copy) — **before the pin moves** | refused; nothing moves | | 4 | `pinning` | the previous definition is copied aside and journaled, then the pin advances | pin put back | | 5 | `pulling` | `compose pull` | **pin and definition PUT BACK** — nothing ran (Scenario E) | | 5a | `copying` **(v0.263.0)** | `compose stop`, then each NAMED volume `cp -a` into `.pre-update-` by a helper that writes a finished-marker last (bind folders never) | copies removed, **pin put back, the old version started again** — nothing new ran | +| 5b | `converting` **(v0.273.0, row added 2026-10-24)** | only on a step whose ladder entry carries `engine_conversion`: the box converts the PostgreSQL major (dump from the old engine, load into the new, check) — `internal/stacks/pgconvert.go` | the undo puts the old engine's data back | | 6 | `starting` | `compose up -d --remove-orphans` | **UNDO** (below), HOLD only if the undo fails | | 7 | `verifying` | the `.felhom.yml` health check through the existing probe **when it resolves to a container**, else 60 s of every container running and none restarting; bounded by `update.health_timeout` | **UNDO** (below), HOLD only if the undo fails. Before v0.263.0: stop + HOLD, the pin stays (Scenario F) | | 7a | `undoing` **(v0.263.0)** | every copy validated (marker) BEFORE anything is poured back; volumes emptied and refilled; definition, pin and the pinned version's `.felhom.yml` record put back from the job's own copies; `up`; health with the OLD version's probe | HOLD, the sentence prefixed *„A frissítés nem sikerült, és az automatikus visszaállítás sem."* + the data state (`untouched` / `half` / `not_started`); copies kept | @@ -2116,7 +2126,7 @@ the harness has proven it, is slice 6's. > copy taken before the pin moved**, so the old version meets pre-migration data. Spiked by hand on > 9202 the same day — §6.1a. -**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6. +**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6. *(Noted 2026-10-24: stepping is built — the ladder, one tested step per press, decision 14.)* ### 6.1a The undo (decision 15) — SPIKED BY HAND, then BUILT: controller v0.263.2 (2026-09-23) @@ -2130,7 +2140,8 @@ only the live box could show, fixed the same day:** the undo's probe was never a current probe held the app `unhealthy` (v0.263.1), and the "old" `.felhom.yml` taken at update time was already the new one, because `.felhom.yml` flows in on every catalog sync (v0.263.2: the pinned version's file is now recorded in `applied-meta/` whenever a version is pinned). **Residual (R-646):** -an app pinned before v0.263.2 has no such record until its next pin. **Two more, found by the chaos hour of +an app pinned before v0.263.2 has no such record until its next pin. *(Closed since: `BackfillAppliedMeta` records it at +start for every app CURRENT with the catalog, and names the rest — `internal/stacks/undo.go:646-652`; noted 2026-10-24.)* **Two more, found by the chaos hour of 2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** the undo finds an app's volumes by their compose label, and a restore recreates them WITHOUT it — so after any restore the undo copies nothing (R-658, P1); and a held file-leg app is pointed at a restore that refuses a database-only copy, with no route left on a box without @@ -2270,7 +2281,7 @@ breaks. --- -### 6.2 Slice 6, as it will be built (OPEN — R-450; ruled 2026-09-23, decisions 11–17) +### 6.2 Slice 6, as it will be built (~~OPEN — R-450~~ BUILT, R-450 closed 2026-10-05 — noted 2026-10-24; ruled 2026-09-23, decisions 11–17) **The shape the rulings fix. The build order and costs are §6.4.** Rewritten 2026-09-23; the earlier "as it would be built" draft keyed on the window of Q1's recommendation and on @@ -2341,7 +2352,7 @@ retried. A working caller in this exact shape exists as evidence, not product: **The per-box switch key is `app_update.unattended`**, default **true** (decision 12). ⚠ **NOT `auto_update` — that name is TAKEN, and by the very thing this must not collide with.** -`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281, default +`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281 then, L278-279 now; default **04:30** at L422) are the CONTROLLER's own update. Two settings with that name, one meaning the controller and one meaning apps, is the kind of collision that is only discovered by an operator who turned off the wrong one. The app-scoped key is `app_update.*`. **And `stacks.update_window` is @@ -2371,12 +2382,12 @@ what the part can do to a household's data if it is wrong, not how likely that i | # | part | rulings / rows | cost | depends on | risk to customer data | |---|---|---|---|---|---| -| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold`: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. | +| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold` *(as SHIPPED — noted 2026-10-24: the copy is the FOLDER copy decision 19 chose: every named volume `cp -a` into a sibling volume with a finished-marker, put back on failure; no database load — `internal/stacks/undo.go:26-36`; the text that follows is the design before the bake-off)*: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. | | **2** | **SHIPPED — controller v0.264.0 + hub v0.120.0, proven live on 9202 and 9201 2026-09-23** (`audits/undo-fleet-2026-09-23/`). **The update sentences in the household's language** (R-606) and a mail when an automatic update is undone or held: events `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, on by default, mailed in the household's language with the app named in the subject; per-app cooldown on both legs. Leftovers: R-647. | R-606, 15 | **1** | — | none | | **3** | **SHIPPED — controller v0.264.0, proven live on 9202 2026-09-23.** **A disabled notifier says so** (R-620), so the mail of part 2 can be measured on a scratch box at all. | R-620 | **0.5** | — | none | | **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only | -| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates//steps/.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape | -| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. **The box half (compare, render `name:tag@sha256`) is not built.** **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low | +| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates//steps/.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). *(Closed since: a step's own `steps/.felhom.yml` is read when present — `ladder.go:127-132`, v0.269.0; noted 2026-10-24.)* **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape | +| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. ~~**The box half (compare, render `name:tag@sha256`) is not built.**~~ *Built — the box renders the tested digest, carries it on sync, and the badge reads the tested digest (`internal/stacks/digest.go:15-24`, `updateorder.go`; noted 2026-10-24).* **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low | | **7** | **SHIPPED — controller v0.271.0 (`9cf13a3`), proven on 9202 over four simulated nights 2026-09-24/25 and watched through the demo boxes' first real automatic night (`audits/DRILL-night-2026-09-25.md`).** Decisions 31–33 taken unattended. Gaps a scratch box cannot close: R-687; no resume after a restart: R-686. *(Before:)* **SPIKED 2026-09-24 (night, Part C) — the measured build brief is §6.4.2 below.** **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has ENDED (on every exit path, skips included), one app at a time, one step per press, `app_update.unattended` default ON (decision 12), `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it, and the full-system backup's gate waits for it until W+5h (decision 20). | 11, 12, 14, 15, 20 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe | | **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none | | **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none | @@ -2459,7 +2470,7 @@ During the db-dump leg every press was refused `busy` (transient, retried next p update's own finish should refresh them. - **An Update pressed on an app already at the head runs the whole guarded update** — backup, pull, restart, `done` — with no refusal (R-679: navidrome restarted four times for nothing). The leg never - presses an app whose pin equals the head; the preflight should refuse with reason `current`. + presses an app whose pin equals the head; the preflight should refuse with reason `current` (as built the reason key is `already_current`, `update.go:432` — noted 2026-10-24). - **The product does not remember a failed step.** After vikunja's undo the badge still offers the same step; only the caller's memory kept it from a re-press. Decision 15 needs it on the box: the failed `to` is recorded per app and the leg skips it until the catalog's ladder changes. @@ -2541,6 +2552,24 @@ and E are the ones that unblock a decision; leg A is the one that takes the time harness on DooPlex for the image-side ones. +### 6.4.x Decisions in code no section records [DESIGN — undocumented, found 2026-10-24, review round 2 — the operator confirms them through one register row] + +Read at `file:line` on controller `main` (`321df1f`) by the main session: + +- **`build.sh --push` also refuses a DIRTY tree, a HEAD that is not `origin/main`, and a directory that is not a git + checkout** — `controller/build.sh:144-166`. §0 names only the tag-exists and unreachable-registry refusals. +- **The leg retries a passing refusal ONCE in the same night** (`busy`, `updating`, `deploying`, `migrating`, + `self_updating`), after `RetryWait` — not „the next pass/night" (`internal/stacks/unattended.go:65-66`, the pass loop). +- **The leg skips an app the household stopped** (`stopped_by_household`, R-721): an update would start it + (`unattended.go:416-418`). +- **A manual run of the chain** (R-705, the debug action) gives the leg a deadline of its start + the leg's normal + length (`backupwindow.UpdateLegLengthMin`), not W+5h (`unattended.go:232-238`). +- **Two preflight refusals are plain skips to the leg:** `already_current` and `engine_no_test` + (`internal/stacks/update.go:432,454`). +- **The render table (§5.4) has one more row:** when the catalog template's images cannot be read, the syncer leaves + the app's compose file alone for that cycle (`internal/sync/sync.go:622-627`); and the box's R-971 allow-list runs + before ANY compose write (`cmd/controller/main.go:421`, `SetAdmitFn`). + ## 6.5 The drill catalog and the image store — the standing method for update drills **Why this section exists.** On 2026-09-21 an afternoon session put a deliberately broken image into @@ -2629,7 +2658,9 @@ customer box has credentials for it. The store listens on the guest's loopback o ### 7.1 The record (slice 1) `Manager.recordInstalledImages` (`felhom-controller/controller/internal/stacks/installed.go`) runs -after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`, +after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy` +*(as built — noted 2026-10-24: `UpdateStack` is gone; the callers are `StartStack`, `RestartStack`, the deploy path, the +end of the guarded update and the end of the undo — `manager.go`, `deploy.go`, `update.go`, `undo.go`)*, and writes `app.yaml`: ```yaml @@ -2705,7 +2736,7 @@ Version strings stay in the logs, the API and the hub. The 8th, immich's own ghcr build, is UNMEASURED — ghcr exposes no anonymous last-modified. So on demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved. Digest-level comparison needs the catalog to record the digest at push time — **R-446**, put to the - operator as §3b **Q6**, recommended YES. + operator as §3b **Q6**, recommended YES. *(Built since for every ref with a ladder entry — `updateorder.go`, `digest.go`; the limitation remains only for a ref no entry records; noted 2026-10-24.)* 2. **~~Nothing enforces `catalog_since`.~~ Enforced by the pre-push hook since 2026-09-13 (R-452, `app-catalog-felhom.eu/scripts/check-catalog-since.py`); CI's shallow clone still skips it out loud.** A commit that moves an `image:` line and forgets the date under-reports how far behind a box is. The gates runner fetches at `--depth 1` and has no parent commit to diff against, so the gate needs a deeper fetch — **R-452**. @@ -2730,13 +2761,18 @@ Version strings stay in the logs, the API and the hub. refuses without a restorable, proven Tier-2 copy, backs up first when that copy is stale, takes a safety dump, and holds an app that does not come up. **What stays true:** it still has no automatic rollback (deliberately, §6.1) and can still attempt a multi-major jump the app will refuse (R-40) — - that now ends HELD rather than crash-looping behind a green button. + that now ends HELD rather than crash-looping behind a green button. *(Dated note 2026-10-24: a copy on ANY tier now + satisfies the precondition (v0.239.0), the box undoes a failed update itself (decision 15, v0.263.2), and the + ladder climbs one tested step per press (decision 14) — `internal/backup/update_guard.go`, `internal/stacks/undo.go`, + `ladder.go`.)* 7. ~~**An engine major can be applied without its datadir upgrade, and nothing notices.**~~ **CLOSED 2026-09-13 for MariaDB (R-459):** every `mariadb:` sidecar carries `MARIADB_AUTO_UPGRADE=1`, and the harness shows the conversion running on the E3/E3b edges (§3 decision 5). **What stays true:** the PostgreSQL half (R-463) has no equivalent — the image performs no `pg_upgrade` — and the engine-major rule (§3 precaution 3, R-469) is what keeps both engines inside their major until - Slice 4 gives the Update button a backup. + Slice 4 gives the Update button a backup. *(Dated note 2026-10-24: MariaDB is lifted — its major is its own edge; a + PostgreSQL major may pass with a two-venue proof and an `engine_conversion` mark, and the box converts it + (v0.273.0) — `app-catalog-felhom.eu/scripts/check-engine-major.py`.)* 8. ~~**Only three of 53 apps have ever had an upgrade measured.**~~ **WIDENED 2026-09-21 to 21 EDGES ACROSS 19 APPS** (`audits/DRILL-update-night-2026-09-21.md`), on scratch guest 9202 through the product's own guarded Update, against a PRIVATE DRILL CATALOG so the live catalog diff --git a/documentation/architecture/10-localisation.md b/documentation/architecture/10-localisation.md index e50a489b..23c5a323 100644 --- a/documentation/architecture/10-localisation.md +++ b/documentation/architecture/10-localisation.md @@ -128,7 +128,9 @@ the hub's e-mails follow.** `expired` pinned to Hungarian so the language cannot become the oracle the text refuses to be. Full design: `05-hub-architecture.md` §15. **R-555 closed** — the `language` allowlist entry is out of `wire_contract_gate.py` and the gate now checks the field for real. -- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece:** the hub stores a per-customer language, renders it into +- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece** *(Part B is built + too — the controller reads `customer.language` as its start language since v0.256.0, `internal/config/config.go:149-154`; + corrected 2026-10-24)*: the hub stores a per-customer language, renders it into `controller.yaml` next to `customer.id/name/domain/email` (`hub/internal/configgen/configgen.go`), and the box uses it **only while the household has never chosen** (`settings.json` empty). The household's own choice always wins; the hub's e-mails follow the language the box **reports**, which @@ -184,7 +186,8 @@ bundles. They now judge the page **as rendered in each language** (`controller/s - **hu** — the product speaks in „te". The converted copy carries formal („ön") forms (R-516). They are not fixed by a localisation release (§1), so the gate counts them against `HU_FORMAL_CEILING` as a ratchet: a new one convicts, fixing one lowers the ceiling. The ceiling followed the conversion: - 6 (spike) → 10 (release A) → 12 (release B) → **16** (release C). **[FACT] Measured limit:** the stem + 6 (spike) → 10 (release A) → 12 (release B) → **16** (release C) → … → **0** (R-516 item 12, the last eleven in the + te-form — `felhom-controller/controller/scripts/i18n_missing_gate.py:41`; corrected 2026-10-24). **[FACT] Measured limit:** the stem list is short — on the release C pages it counts 4 formal forms, while a wider list („írja be", „adja meg", „válassza", „biztosan eltávolítja", „engedélyezze", …) finds at least 22 keys (R-516). - **en** — second person, plain: no „please", no „kindly". @@ -201,7 +204,8 @@ vissza") — R-564. The hub copy gate and the catalog have no language concept - **[DESIGN] The hub's operator pages** — the operator reads them; they stay English (decision 1). - **[DESIGN] The first-boot wizard** (`controller/internal/setup/`, 8 pages, 95 strings) — out of scope and obsolete (decision 4, 2026-09-17; `02-controller-module-map.md` L56). A household still reaches - it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**. + it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**. *(Deleted since: + `internal/setup/` is gone (v0.308.0); `bootstrap.WaitForConfig` shows a waiting page instead — corrected 2026-10-24.)* - **[FACT]** App containers' own UIs (Uptime Kuma, PrivateBin…) — not Felhom's copy (R-516 items 5-6). --- @@ -226,7 +230,7 @@ i18n: ``` Why: one app's copy stays in one file a reviewer sees whole; the controller's `yaml.Unmarshal` ignores -unknown keys (`internal/stacks/metadata.go` L316 — **verified: this repo constructs no `yaml.Decoder` +unknown keys (`internal/stacks/metadata.go` L316 then, L475 now — **verified: this repo constructs no `yaml.Decoder` at all, so there is no `KnownFields` anywhere to turn strict**), so older controllers are unaffected; a missing English field falls back field by field. @@ -234,7 +238,7 @@ a missing English field falls back field by field. reached only through `LocalizeStacks` / `LocalizeStackPtr` / `Stack.MetaFor`. - **`For("hu")` is the parsed struct with `I18n` cleared and nothing else**, deep-compared against all - 53 real catalog files (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`). + 53 real catalog files (the identity fixtures are still those 53; the catalog has 65 apps — noted 2026-10-24) (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`). §1 therefore holds by construction for catalog copy, and `LocalizeStacks` returns the INPUT SLICE for Hungarian rather than a copy, so nothing on the Hungarian path is even touched. - **Fallback is field by field, and blank counts as absent** (ruling 3). A half-translated app is a @@ -307,10 +311,10 @@ for the English page test (R-565), arriving again in a different repo. So `app-catalog-felhom.eu/scripts/check-copy-i18n.py` folds and matches word-bounded stems, with a positive and two negative controls printed on every run. -**[FACT] The gate**, fifth row of `catalog_gates.py`, static, in the pre-push hook: +**[FACT] The gate**, fifth row of `catalog_gates.py` (the sixth since — corrected 2026-10-24), static, in the pre-push hook: 1. **FREEZE** — every Hungarian copy string equals `scripts/copy_freeze/hu.json`, captured in its own - commit before any translation. Runs on all 53 apps whatever scope is named. A new app must be + commit before any translation. Runs on every app in the freeze file (65 on 2026-10-24) whatever scope is named. A new app must be admitted with `--add-app NAME --reason "…"`. 2. **STRUCTURE** — the `en` block may carry copy fields and nothing else; every key-matched entry must have a Hungarian twin, or it is INERT on the box and the translator never learns. diff --git a/documentation/architecture/11-os-updates.md b/documentation/architecture/11-os-updates.md index 6ab2d929..54985599 100644 --- a/documentation/architecture/11-os-updates.md +++ b/documentation/architecture/11-os-updates.md @@ -11,7 +11,7 @@ > | | | > |---|---| -> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1).** Ratification is Viktor's review, not an editor's. | +> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1); steps 2–6 are BUILT by §8's own list (corrected 2026-10-24 — the ratification is still the operator's).** Ratification is Viktor's review, not an editor's. | > | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. | > | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 | > | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. | @@ -79,7 +79,7 @@ boxes installed. Nobody keeps a hand-written list. - **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5). - **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since - 2026-06-11) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release + 2026-06-11 — *`2026.9.3` today, corrected 2026-10-24*) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release — the app-image question (`09`), not an OS package. The gap is R-838. --- @@ -101,9 +101,10 @@ gets the Docker engine that was current when its golden was baked, and keeps it. ~~**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist (`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.~~ -**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:58` -(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:107`) and `:194` -(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:628`). Nothing else. +**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:68` +(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:133`) and `:228` +(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:659`). Nothing else. *(Line numbers +corrected 2026-10-24.)* `apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4 proposes how to avoid that. @@ -250,7 +251,7 @@ approval of that day stays in force (decided by CC unattended — operator may r versions. A broken-into hub can choose an older version or no version. It cannot make a box install a package that the publisher did not sign. -### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]` +### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]` — *BUILT and installed since (§8.1); the agent's sudoers holds only the `--plan` line, no `--repair-only` (`felhom-agent/configs/felhom-agent.sudoers:337`); noted 2026-10-24* `felhom-os-apply` — root-owned (`0755 root:root`), installed by the installer beside `felhom-selfupdate-guarded`; the agent's sudoers gets exactly `/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json` and @@ -315,7 +316,7 @@ PID 1 or `lxc-start`, or a kernel was installed. Decision 96. Before it, the signed `agent_update` replaced only the binary; wrappers, units and sudoers lines reached an installed box by reinstall or by hand. -- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22: sudoers ×2, the +- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22 then, **32** counted 2026-10-24 — plus the vzdump-partial-clean, thin-check, internal-place and priv-apply wrappers, the guest hook, the shared-parent unit, the GRUB one-shot pair and the sshd root-key-only drop-in; sudoers ×2, the five wrappers, the crash guard and its units + config, the agent and rollback units, the start-limit drop-in, the mgmt watchdog and its tmpfiles/units, the OOB belt's four files). `scripts/build-config-bundle.py` builds the bundle from it reproducibly; `release-agent.sh` publishes it beside the binary; the hub vouches its sha with the agent (exact-name @@ -393,6 +394,9 @@ must never overlap a backup, a restore-test or a self-update.~~ - **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release, how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest approved release raises an alarm (`08`). +- **[FACT, added 2026-10-24 — hub v0.169.0] The System page's „Upload line" cell** shows the best of the last SUCCESSFUL + weekly line test (R-1247), amber when it is older than 15 days or was never measured after an attempt — judged on + `last_ok_at`, never on the attempt time (`hub/internal/web/system.go` `lineStaleAfter`, `lineTestCell`). - **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held / reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed, @@ -480,7 +484,9 @@ Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-do **What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve, libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on 2026-10-07. **What it never touches:** a kernel, boot, firmware or microcode name (`HOST_SLOW_RE`: `proxmox-kernel*`, -`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`) +`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`, +and — read 2026-10-24 in `felhom-os-apply` — `linux-(image|headers|kbuild|modules|base)*`, `firmware-*`, +`proxmox-secure-boot-support`) — that is R-836's lane (decision 164). No reboot. - **Wrapper** (`felhom-agent/configs/felhom-os-apply`): layer `pve`, lane `slow` only. Origin `Proxmox Debian diff --git a/documentation/architecture/13-threat-model.md b/documentation/architecture/13-threat-model.md index ece43ae3..f042b334 100644 --- a/documentation/architecture/13-threat-model.md +++ b/documentation/architecture/13-threat-model.md @@ -78,6 +78,13 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980). | **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971). **Re-check (2026-10-24, review round 2):** the box checks every rendered compose against its allow-list before any write since controller v0.328.0 (`internal/privpolicy/privpolicy.go`) | | **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (deployed 2026-10-24 — corrected 2026-10-24; `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, released 2026-10-24 in controller v0.342.0 + hub v0.169.0 — corrected 2026-10-24) detects | +**[DESIGN — undocumented in `01` §5, found 2026-10-24, review round 2 — confirmed through R-1265]:** the file manager's one +sign-in (R-1100) — FileBrowser trusts its proxy header blindly, and every app container can reach `filebrowser:80` +directly; the only guard is that the header NAME is a per-box secret, which traefik's forwardAuth strips from every +client request and sets only for a browser holding the dashboard's files pass +(`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`, `internal/web/files_sso.go`). The name file's +creation race was fixed by review round 2 (controller `b01bb4b`). + ## 6. What we accept (do not re-report) - **Cloudflare can read tunnel traffic** — R-904, `09` §3 decision 184. diff --git a/documentation/architecture/14-host-power-off.md b/documentation/architecture/14-host-power-off.md index 263e519a..bd93d0b6 100644 --- a/documentation/architecture/14-host-power-off.md +++ b/documentation/architecture/14-host-power-off.md @@ -22,7 +22,7 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42 | 2 | controller `POST /api/server/poweroff` (`internal/web/poweroff_handlers.go`) | dashboard login (`RequireAuth` + CSRF), POST only, the word again, the controller's own backup/restore/app-update check, the `host_poweroff` feature gate (MinAgent 0.155.0) | | 3 | agent `POST /host/poweroff` (`internal/localapi/hostpoweroff.go`) | `withGuest` token scope; the guards (§3); the record (`internal/poweroff`) | | 4 | agent → hub | a host report carrying `deliberate_off.requested_at`, waited for until the hub ACCEPTED it (≤ 60 s, `Loop.SetReportedHook`) | -| 5 | agent | `pct shutdown --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors, `FELHOM_POWEROFF`) | +| 5 | agent | `pct shutdown --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors: `pct ^shutdown [0-9]+ --timeout [0-9]+$` in `FELHOM_GUESTHOOK`, and `systemctl poweroff` in `FELHOM_POWEROFF` — `felhom-agent/configs/felhom-agent.sudoers:81-90`; corrected 2026-10-24) | | 6 | hub (`store/deliberateoff.go`) | `hosts.deliberate_off_at`; silence alarms held 14 days (§4) | | 7 | the household | presses the machine's power button; the first report of the new boot has no `deliberate_off` → the hub clears it | @@ -33,11 +33,11 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42 | first 10 min after boot | `too_soon_after_boot` | `TooSoonAfterBoot` | | one accepted request per boot (file keyed by `/proc/stat` btime) | `already_requested_this_boot` | `SecondRequestSameBootRefused`, `poweroff/state_test.go` | | any other guest or VM running (`/proc`: `lxc-start -n N`, `kvm -id N`) | `other_guests_running` | `RefusedWhileOtherGuestsRun`, `RunningGuestsFromProc` | -| heavy-operation gate held (backup, restore test, OS step, sweep, trim) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` | +| heavy-operation gate held (backup, restore test, OS step, sweep, trim, and since agent v0.164.0 the weekly upload line test — corrected 2026-10-24) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` | | backup job, controller swap, drive format, NAS verify, escrow ceremony, Proxmox vzdump | `backup_running` / `controller_update_running` / `drive_operation_running` / `escrow_ceremony_running` | `RefusedDuringVzdump`, `FailsClosedWithoutTaskReader` | +| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` | **„A Proxmox vzdump runs” (agent v0.156.1, R-1008):** a vzdump task for the guest in Proxmox's active list whose worker process still lives — Proxmox's own rule (the UPID's pid + pstart against `/proc//stat`). After a power cut the dead task stays listed for ~30–60 s; it no longer counts. Doubt (an unparsable task, an unreadable `/proc`) still refuses. The same answer serves the start-up stale-lock pass, the guest-power watchdog and the controller supervisor (`stalelock_inflight_test.go`). -| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` | **[DESIGN]** A cancelled power-off leaves the box on (gate released) — better a box that did not switch off than a backup, update or format cut in the middle. **[DESIGN]** A failed guest shutdown does not stop the power-off: Proxmox's @@ -86,5 +86,15 @@ alternative (an out-of-band confirmation) needs a second channel the box does no ## 6. Not done here -- The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000). +- ~~The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).~~ Published — `website/gyik.html:128,603` (corrected 2026-10-24). - The BIOS „after power loss" value of the N100 was not read (no monitor); the household text promises only the button. + +## Decisions in code no section records [DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through one register row] + +- **A re-enroll while the household power-off hold is active counts as „gone"** (a reinstall over the old host), not + „live" (R-1154; `hub/internal/api/reenroll.go:67-68`). +- **Setting and clearing the deliberate-off time each record an info event** — `host_deliberate_off`, + `host_deliberate_off_ended` — on the customer's event list; neither is mailed (`hub/internal/api/deliberate_off.go:12-13`). +- **The confirmation word is accepted in three forms:** „KIKAPCSOLÁS", the unaccented „KIKAPCSOLAS" (a foreign keyboard) + and „TURN OFF" (`felhom-controller/controller/internal/web/poweroff_handlers.go:28-29`). + diff --git a/documentation/architecture/15-media-library.md b/documentation/architecture/15-media-library.md index fd894a55..4954a418 100644 --- a/documentation/architecture/15-media-library.md +++ b/documentation/architecture/15-media-library.md @@ -165,7 +165,8 @@ directory) with that tag. Facts the link code needs: **[FACT]** An unlisted writable `${…_PATH}` bind is MANDATORY (every tier, off-site included); a plain host path is in no capture set (`stacks/classify_binds.go` `classifyRoot`). Measured over the live templates: Transmission's unlisted `watch` bind goes off-site; the download folders are `optional`, so a second drive gets a nightly copy of every -download (R-1019). **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's +download (R-1019). *(Fixed since: catalog `c50adfc` marks the download and watch folders `excluded` — +`app-catalog-felhom.eu/templates/transmission/.felhom.yml:46-51`; noted 2026-10-24.)* **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's media is no single app's data; whether the library gets its own household-level backup is a separate decision. ## 8. Removal and export assume one drive per app @@ -192,6 +193,13 @@ household's existing files under `media/torrents` stay where they are. Evidence: `audits/media-drive-events-2026-10-14/` (K1–K7). „Idempotent" = a second run changes nothing (K2: run 2 left every entry, folder and library as it was). „Household" = a step only the household can take. +> *Dated note 2026-10-24 (review round 2): this table is the 2026-10-14 MEASUREMENT („today" = then). As built (§11): +> on a drive event the controller only REWRITES the binds; the media apps restart when the household presses +> „Újraindítás most" or „Később" (03:30) — the file browser alone recreates itself (`internal/stacks/media_library.go`, +> `internal/web/media_events.go`); a safe removal keeps the drive REGISTERED and its return re-binds with no household +> step (`SetEjected`, `media_events.go`); and since controller `b01bb4b` (unreleased) neither restart recreates an app +> the household stopped (R-721).* + | event → app | action | measured call | idempotent | household | evidence | |---|---|---|---|---|---| | **added** — every media app | the compose gains the drive's bind → `up -d` recreates it (the compose CHANGED, so today's restart works) | controller compose `up -d` | yes | no | K5-1: 1–10 s per app | @@ -258,3 +266,9 @@ save path when its client is saved before the wiring; **a Plex status probe (`/l unclaimed Plex because Plex answers 401 for a moment while it starts — never gate on a status a starting app also returns** (G7f). + + +**[DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through one register row]:** the „Később" +03:30 media restart is an IN-MEMORY timer — a controller restart forgets it and only the notice stays +(`internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`); the Sonarr/Radarr API key is READ from `/config/config.xml`, the „set at +install" option §6 calls cleaner was not built (`internal/mediawire/mediawire.go:336-345`). diff --git a/documentation/architecture/16-storage-physical-identity.md b/documentation/architecture/16-storage-physical-identity.md index cc141cb5..48d725b3 100644 --- a/documentation/architecture/16-storage-physical-identity.md +++ b/documentation/architecture/16-storage-physical-identity.md @@ -15,8 +15,8 @@ | Place | Found by | Identified by | Mounted | What lives there | The backup page says | |---|---|---|---|---|---| | Host root / PVE `local` | PVE storage `local` [FACT] A/internal/storage/observe.go:424; never a candidate [FACT] A/internal/storage/role.go:53-79 | `path:` / `store:local` [FACT] A/internal/storage/durableid.go:54-67; role system | host `/`, `/var/lib/vz` | the **local whole-guest copies** (vzdump, retention 3) [FACT] E/scripts/felhom-host-install.sh:730-734; `07` §6.1 | `backup.target.degraded` „…ugyanazon a lemezen van, mint a rendszer…" [FACT] C/internal/web/backup_target_offer.go:78-100 | -| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — a Go constant, not i18n [FACT] backup.go:359-361, :1387-1410 | -| Enrolled drive (`/mnt/` on the host → `/mnt/felhom-drives/`) | `GET /disks/candidates`: **whole disks only** [FACT] A/internal/storage/candidates.go:33-53 | mount: `uuid:`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 | +| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — *corrected 2026-10-24:* the i18n keys `note.backup.single_copy` / `note.backup.single_copy_unproven`, picked by the agent's verdict [FACT] C/internal/backup/backup.go:1485-1494 | +| Enrolled drive (`/mnt/` on the host → `/mnt/felhom-drives/`) | `GET /disks/candidates`: whole disks, each carrying its nested partition candidates (empty / filesystem / in use; corrected 2026-10-24) [FACT] A/internal/storage/candidates.go:84-90 | mount: `uuid:`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 | | Network share | customer input [FACT] A/internal/localapi/netstorage.go:36-40 | none („no durable-id, no SMART") [FACT] netstorage.go:15-19 | `.automount` under `/mnt/felhom-drives/` [FACT] A/internal/storage/netmount.go:18-31 | browse only; never media, never Tier-2 [FACT] media_library.go:83; tier2.go:141-143 | `err.backup.halozati_tarolo_nem_lehet_a_2` [FACT] tier2.go:37-40 | | USB disk | same as an enrolled drive; transport never decides [FACT] C/internal/settings/settings.go:609-614 | same; the bridge may pass the real identity (N100: yes) [FACT] spike case 4a | same | same | same; SMART may be missing → „Nincs adat" | @@ -126,7 +126,7 @@ binds both. **Creating or deleting partitions: never from the dashboard in this ## 7. The installer **[FACT]** Today `step_grows` gives the guest's one data volume (`mp0`) 70 / 130 / 250 GiB by the thin pool's free space -(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh:2261-2275). Measured pools: 128 GB +(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh `step_grows`, now at :2475 — line corrected 2026-10-24). Measured pools: 128 GB disk → 54 GiB, 200 GB → 113 GiB, 512 GB → 349 GiB (PVE default layout; root capped at 96 GiB). **[DESIGN] The rule:** `mp0` = 70 GiB (pool < 300), 130 (300–800), 250 (≥ 800). Reserve = **max(16 GiB, 10 % of the @@ -231,10 +231,25 @@ sizes ruled above in §7): the public installer 1.33.0, agent 0.158.1 and golden 0.318.0 vouched) → data volume 100 GiB, internal place 213 GiB, registered as the default drive with no press; Jellyfin installed with no extra step onto it; a whole-box copy holds nothing of it; partitions formatted and used through the wizard; the honest backup offer on a VM. **Not as written:** an empty part -asks for the typed confirmation (the probe reads the parent's partition table — R-1093); a 200 GB disk gets no internal +asks for the typed confirmation (the probe reads the parent's partition table — R-1093) *(fixed since in agent v0.160.0: +the parent's PTTYPE is set aside when `blkid -p` proves the node is a partition of that scheme — `git tag --contains +f7b7308`; noted 2026-10-24)*; a 200 GB disk gets no internal place under the ruled sizes (15 GiB < 20). Also in build B: R-1084 (the whole-box backup offer asks the same-disk question) and R-1082 (`/mnt/sys_drive` is never offered to attach, by device). A bundle that adds a path needs a step bundle on boxes with an older wrapper (R16, R-880): 0.158.1 was delivered through `0.158.1-step1`. +## Decisions in code no section records [DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through one register row] + +Read at `file:line` on `main` by the main session: + +- **`systemd-detect-virt --vm` is read once per agent process**; empty output becomes `unknown`, which is not `none`, + so every disk then reads unknown/vm — the fail-closed direction (`felhom-agent/internal/storage/identity.go:195-205`). +- **A path the controller cannot read counts as the SAME disk** (R-668, v0.269.0): no second copy is ever claimed on + it (`felhom-controller/controller/internal/backup/backup.go:1436-1450`). +- **The internal place („Belső tárhely") refuses eject, decommission and wipe** — the controller (`refuseInternalLifecycle`) + and the agent both — and allows moving its data to another drive (`internal/web/internal_place.go:20-26`). +- **The internal place's size subtracts EVERY thin volume already promised in the pool**, not only the 32 GiB rootfs and + the data volume (`felhom-agent/configs/felhom-internal-place:11-14`). + diff --git a/documentation/architecture/_recovery-inventory-2026-07-28.md b/documentation/architecture/_recovery-inventory-2026-07-28.md index 6dad4a35..3f9f9461 100644 --- a/documentation/architecture/_recovery-inventory-2026-07-28.md +++ b/documentation/architecture/_recovery-inventory-2026-07-28.md @@ -5,6 +5,21 @@ > No matrix is designed here, no target architecture is proposed, and no gap is ranked. > > **This is the input to a redesign, not the redesign.** +> +> **WHAT HAS MOVED SINCE (banner added 2026-10-24, review round 2 — this inventory records 2026-07-28 and is kept as +> it was; each line below was read at `file:line` on `main` that day):** +> - The guest has ONE data volume, `mp0` at `/var/lib/felhom`, whose `docker/` and `sys_drive/` folders are bound in the +> guest; there is no `mp1` (`felhom-agent/configs/build-golden.sh:31-38,99`) — the „rootfs + mp0 + mp1" lines are history. +> - A schema-2 recovery unit (D5) carries the PORTABLE secrets; a restore takes the database password from the unit +> (R-1106) — `felhom-controller/controller/internal/backup/recovery_unit.go:24-27`, `restore_dbcreds.go`. +> - The second-drive copy's `recovery-unit/` IS read: `RestoreTier2Unit` restores the full unit from the mirror (R-102) +> — `internal/backup/tier2_restore.go:197`. +> - The off-site reconstitution replays the snapshot's named-volume tars, volumes first (R-354) — +> `internal/backup/offbox_reconstitute.go`. +> - ep0's per-household prune jobs are keep-weekly 2 at 03:30 (R-1162/R-1165, `scripts/felhom-tenantsync.sh:187-197`); +> the local whole-guest tier is keep-daily=3 (R-1149, `felhom-agent/internal/config/config.go:553-559`). +> - restic is the pinned official release, checked by version and checksum (R-1105, `felhom-controller/controller/Dockerfile:43-46`), +> no longer Debian's 0.14.0. ## 0. Method, baselines, and what "verified" means here diff --git a/documentation/audits/review-round-2-2026-10-24/9202/at-end-2026-10-24.txt b/documentation/audits/review-round-2-2026-10-24/9202/at-end-2026-10-24.txt new file mode 100644 index 00000000..a57692d3 --- /dev/null +++ b/documentation/audits/review-round-2-2026-10-24/9202/at-end-2026-10-24.txt @@ -0,0 +1,17 @@ +Sat Oct 24 14:05:27 UTC 2026 +gitea.dooplex.hu/admin/felhom-controller:0.342.0 + repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 6 hours (healthy) +filebrowser gtstef/filebrowser:1.5.6-stable Up 23 hours (healthy) +immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up About an hour (healthy) +immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up About an hour (healthy) +immich-redis redis:7-alpine Up About an hour (healthy) +immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up About an hour (healthy) +paperless-postgres postgres:18-alpine Up About an hour (healthy) +paperless-redis redis:7-alpine Up About an hour (healthy) +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up About an hour (healthy) +traefik traefik:v3.7.13 Up 23 hours +324af98a77bf1b5a6e0427187c6e1991 - +3 +/dev/loop0 32G 2.7G 27G 10% / +/dev/nvme0n1 938G 239G 652G 27% /mnt/felhom-drives/scratch_hdd diff --git a/documentation/audits/review-round-2-2026-10-24/EDITS.md b/documentation/audits/review-round-2-2026-10-24/EDITS.md index 2b659092..abf0f117 100644 --- a/documentation/audits/review-round-2-2026-10-24/EDITS.md +++ b/documentation/audits/review-round-2-2026-10-24/EDITS.md @@ -94,3 +94,102 @@ Rules, fences, „never" lines and promises were NOT edited — those are rows f - [S18] R-970 „does not hold" → the routes are removed, pinned by `r970_no_hub_key_door_test.go`. - [S19] R-971 „design gap" → `privpolicy.Admit` before every write (controller v0.328.0). - [S20] „hub 0.168.0 deploy pending", „R-1244 not released" → deployed / released 2026-10-24. + +## 07-backup-architecture.md (DOC-D1) +- [S2/C3] Tier-1 „secret-stripped app.yaml" → carries the portable secrets (D5) — `felhom-controller/controller/internal/backup/recovery_unit.go:24-27`. +- [S7] `main.go:604-607` → `backupwindow.LegTimes` at `cmd/controller/main.go:1258-1264`. +- [S9] demo-hp „keep-last=1" → a dated note: keep-daily=N since R-1149 — `felhom-agent/internal/config/config.go:553-559`. +- [S1] D22 „controller main, unreleased" → released v0.338.0. +- [S11] §6.6 table → a row: the off-site restore also refuses a mismatched/mixed snapshot first — `internal/backup/offbox_reconstitute.go:826-833`. +- [S8] line citations moved (`GetAppDrivePath`, F5); `mp1` → a dated note (gone). +- [S3/C1] row 15 R-104 „open DEFECT" ×2 → FIXED 2026-10-05, v0.297.0 — `internal/backup/offbox.go:156-162`; CLOSED-ITEMS.md:895. +- [S6] row 10 R-893 / R-973 „unreleased" → released; „the next slice on R-893" → note: D20 (v0.336.0), D22 (v0.338.0). +- [S5] „Hub v0.158.0 … deploy waits" → deployed 2026-10-18 12:16 (hub CHANGELOG:288). +- [S10] „a restore is refused while any second-drive copy runs" → a dated note: the kept-data load checks only RestoreStatus/IsRunning (`internal/api/kept_install.go:89-92`) — a row. +- [U1/U2/U6 + hub U2/U3 + U7 → §8.2c, DESIGN — undocumented notes, one row for the operator]: the 60-min copy limit and stray handling; Eltávolítás refused during backups; start-after-put-back failure; the hub's off-site delivery checker (`offsite_delivery_stuck`, `maybeHeal`, the R-1230 skip — `hub/internal/monitor/offsite_delivery.go:103-110`); the release copy (D33) not in the matrix. +- NOT edited here: C4 (a stale code comment in `felhom-agent/internal/localapi/backup_cap.go:15`) — a row; C2 (G15 „nightly") — corrected in the capability map below. + +## 09-update-architecture.md (DOC-C2) — dated notes; no §3 ruling text edited +- [S1/S2/S3] §1 (the 2026-09 measurement) → one dated note: `UpdateStack` gone, `StartGuardedUpdate`; moved line numbers. +- [S16] `stacks.update_window` „no Go code reads it" (inside ruling 11) → a dated note: removed. +- [S17] `unattended.go:433–435` → `:449-451`. +- [S4] §5.3 writers → note: `advancePinTo`; `pinBack`, the undo's `restoreDefinition`. +- [S7/C1] slice 6 „OPEN — R-450" ×2 → BUILT, R-450 closed 2026-10-05. +- [S12] R-646 residual → closed by `BackfillAppliedMeta` — `internal/stacks/undo.go:646-652`. +- [S18] `config.go L280-281` → L278-279. +- [S9] part 1 „DB up alone … empty-then-load" → note: the shipped folder copy — `undo.go:26-36`. +- [S10] „a step has no .felhom.yml of its own" → closed — `ladder.go:127-132`. +- [S11] „the box half … is not built" → built — `digest.go:15-24`. +- [S19] reason `current` → `already_current` — `update.go:432`. +- [S14] digest comparison „R-446 §3b Q6" → built for every ref with a ladder entry. +- [S13] §8.6 / „stepping is slice 6" → notes: any tier, the undo, the ladder. +- [S15] engine-major „until Slice 4" → note: MariaDB lifted, PostgreSQL by two-venue proof. +- [S5] recordInstalledImages callers → note: the real callers. +- [S8] §6.1 phase table → the bind-data tier order; a `converting` row (v0.273.0, `pgconvert.go`). +- [U1–U6, S6 → §6.4.x, DESIGN — undocumented notes]: build.sh's dirty/HEAD/non-git refusals (`controller/build.sh:144-166`); the once-retry; stopped_by_household; the manual leg's deadline; already_current/engine_no_test; the unreadable-template render row and R-971 admission. +- NOT edited (RULINGS — rows for the operator): C4 decision 13's „CompareImageRefs moves to the catalog gate" (not built); C5 decision 14's „never jumps" vs a person's press on an app older than the ladder; C7 decision 250 item 5's „by its AGE" vs the gate's proxy. + +## 00-capability-map.md +- 20 PROVEN-LIVE rows marked „re-check (2026-10-24)" in their Scope cell, each with the commit that changed its mechanism after its evidence (status words unchanged): a7, a8, a10, a11, a17, b1, b2, b7, b11, c3, c4, c5, c6, d6, d10, d11, e17, g7, g15, g18. Each commit's files read with `git show --stat`. +- [DOC-D1 C2] G15 „a nightly consistent copy" → every 6 hours since D34. + +## 16-storage-physical-identity.md (DOC-D2) +- [S2] the single-copy notice „a Go constant, not i18n" → the i18n keys — `felhom-controller/controller/internal/backup/backup.go:1485-1494`. +- [S3] `/disks/candidates` „whole disks only" → with nested partition candidates — `felhom-agent/internal/storage/candidates.go:84-90`. +- [S4] `felhom-host-install.sh:2261-2275` → `step_grows` at `:2475`. +- [S1] the R-1093 „not as written" → a note: fixed in agent v0.160.0 (`git tag --contains f7b7308`). +- [U4–U7 → DESIGN — undocumented notes]: detect-virt once; unreadable = same disk (R-668); the internal place's lifecycle refusals; its size rule. + +## 15-media-library.md (DOC-D2) +- [S5/S6/C2] the §10 measurement table → one dated note before it: binds rewritten, restart on the household's press, the drive kept registered, and `b01bb4b`'s stopped-app rule. +- [S7] Transmission's watch/download folders → note: `excluded` since catalog `c50adfc` (`templates/transmission/.felhom.yml:46-51`). +- [U1/U3 → DESIGN — undocumented note]: the „Később" timer is in memory (`media_events.go` `handleMediaRestart`); the Sonarr/Radarr key read from `config.xml` (`mediawire.go:336-345`). +- [U2 — FIXED in code, not a note]: a media restart started a household-stopped app → controller `b01bb4b`. + +## _recovery-inventory-2026-07-28.md (DOC-D2 S8–S13) — a WHAT HAS MOVED SINCE banner; the inventory body kept as its date's record +- one `mp0`, no `mp1` (`felhom-agent/configs/build-golden.sh:31-38,99`); portable secrets in the unit; `RestoreTier2Unit` (`tier2_restore.go:197`); volume tars replayed (R-354); ep0 keep-weekly 2 at 03:30 (`scripts/felhom-tenantsync.sh:187-197`); local keep-daily=3; restic pinned 0.19.1 (`Dockerfile:43-52`). +- runbooks/RUNBOOK-manual-guest-restore.md [DOC-D2 C3]: a dated note after the example config — one `mp0` at /var/lib/felhom, no `mp1` (golden v2.1.0+); the procedure unchanged. + +## 14-host-power-off.md (DOC-E) +- [S2] „two exact sudoers vectors, FELHOM_POWEROFF" → `pct shutdown` in FELHOM_GUESTHOOK + `systemctl poweroff` in FELHOM_POWEROFF — `felhom-agent/configs/felhom-agent.sudoers:81-90`. +- [S1] heavy-gate holders → plus the line test (agent v0.164.0). +- [S3] the website FAQ line „a later task" → published — `website/gyik.html:128,603`. +- [S4] a paragraph that split the §3 table → moved below the table (rendering only). +- [U1–U3 → DESIGN — undocumented notes]: a re-enroll during the hold is „gone" (`hub/internal/api/reenroll.go:67-68`); the two info events; the three confirmation words. + +## 11-os-updates.md (DOC-E) +- [S8] status „§8 step 2 BUILT" → steps 2–6 built by §8's own list (the ratification left to the operator). +- [S5] cloudflared `2026.6.0` → `2026.9.3` today — `felhom-controller/controller/internal/infra/infra.go:26`. +- [S6] apt sudoers/caller line numbers → `sudoers:68,228`, `lanresolver.go:133`, `wgtunnel/manager.go:659`. +- [S7] §5.4.1 „DRAFT … nothing installed" → note: built; sudoers holds only `--plan` (`sudoers:337`). +- [S9] BUNDLE_FILES 22 → 32 (counted in `felhom-agent/configs/felhom-os-apply`). +- [S10] HOST_SLOW_RE list → plus `linux-*`, `firmware-*`, `proxmox-secure-boot-support` (read from `felhom-os-apply`). +- [DOC-E U5 / DOC-C1 U5] §5.7 → the System page's „Upload line" cell (`hub/internal/web/system.go` `lineStaleAfter`). + +## 10-localisation.md (DOC-E) +- [S12] „Part B is the remaining piece" → built (v0.256.0) — `felhom-controller/controller/internal/config/config.go:149-154`. +- [S13] formal-form ceiling 16 → 0 — `felhom-controller/controller/scripts/i18n_missing_gate.py:41`. +- [S11/C1] the setup wizard „still reachable … R-554" → deleted (`internal/setup` absent; `bootstrap.WaitForConfig`). +- [S15] `metadata.go L316` → L475. +- [S14] „53 real catalog files" (fixtures, still 53) / „fifth row" → sixth / „all 53 apps" → the freeze file's 65. + +## 02-controller-module-map.md (DOC-E) — dated notes; the v0.33 plan tables kept +- [S16/C3] two in-flight markers → four (`update-journal.json`, `.felhom-install-pending`). +- [S17] three StartGate holders → four (1b `IsUpdating`, `cmd/controller/main.go:2459-2477`). +- [S18] `GetAlerts` → `GetBannerAlerts(page, lang)` (R-906) — `internal/web/alerts.go:353`. +- [S19] „no longer calls the Cloudflare API" → note: `cloudflare/` still wired with `cf_api_token` (`main.go:1992-2000`). +- [S20] `state.go`/`updater.go` DELETE → note: both live, `AgentSwapper` (`internal/selfupdate/updater.go:38-47`). +- [S21] `sync.go:319`, `config.go:351` → `:436`, `:399`. +- [S22] `locked_after_deploy` „read by no code" → note: read into `LockedFields`, enforced only by an uncalled function. +- [C2] §2 `UpdateStack` → note: deleted in v0.237.0. +- [U4] §1 → a note naming the 22 packages with no row (an `ls` of `controller/internal/`). + +## felhom-controller/REUSE.md (DOC-D1 C5/U8, DOC-C2 C3) +- `CrashPutBack … (D22 / R-1195, unreleased)` → `v0.338.0`. +- `recordInstalledImages` callers `StartStack/RestartStack/UpdateStack/runComposeDeploy` → the real callers; `UpdateStack` deleted in v0.237.0. +- A new row: `Tier2CopyBlocksRestore` / `ClearLeftoverTier2Copies` / `MarkTier2CopyForTest` + the `tier2Docker` seam (`internal/backup/tier2_copy_lifecycle.go:85,255,319,341`). + +## Late additions (the disposition check before the README) +- 03 §6 [DOC-B U1–U5, U8; DOC-A U4 → DESIGN — undocumented note]: 41 routes (24 unlisted), the controller-started line test, the island bridge, the line-test state and switch, the WG block's OOB fields, the D30 count in memory. +- 13 §5 [DOC-A U1 → DESIGN — undocumented note]: the file manager's one sign-in trust (`filebrowser_sso.go:15-22`). +- 08 §9 item 6 [SEC-1 X1]: the limits list names „any viewer explains any upload" (R-1262). diff --git a/documentation/audits/review-round-2-2026-10-24/README.md b/documentation/audits/review-round-2-2026-10-24/README.md new file mode 100644 index 00000000..4c3d6c37 --- /dev/null +++ b/documentation/audits/review-round-2-2026-10-24/README.md @@ -0,0 +1,90 @@ +# Review round 2 — the architecture documents against the code, and a fresh data-loss and security look (2026-10-24, unattended) + +**Baselines (read at start, 14:42 CEST):** felhom.eu `44d19042`, felhom-controller `321df1f` (v0.342.0 released; +„next v0.343.0" unreleased on top), felhom-agent `5e05627` (v0.164.0; „next v0.165.0"), app-catalog `5e196e1`, hub +deployed 0.169.0. Register: 190 rows. + +**Method.** Thirteen read-only reviewers (each prompt carried the brief's fences word for word — +`reviewers/FENCES-given-to-every-reviewer.md`): seven on the documents (the brief's four groups, two of them split, plus a seventh for +`02`/`10`/`11`/`14`, which the groups did not name), four on data loss (restore and put-back; delete and retention; +keys; time and concurrency), two on security (hub + controller; agent + ep0 + installer + the DooPlex→ep0 jobs). Their +answers are CANDIDATES (`reviewers/`). The main session read every item it acted on at `file:line`; the edits and +their evidence are in `EDITS.md`. + +## Part 1 — the documents (claims checked by the reviewers: 493; stale reported: 121) + +| Document | Verdict | +|---|---| +| `01` topology & trust | 7 stale facts corrected; 2 product statements contradict the code → R-1266 (D45) | +| `03` host agent | 9 corrected (shipped releases, key files, listen address, swap, cloudflared's 4th state); a note: 41 local-API routes, 24 unlisted; break-glass text → R-1266 | +| `04` control-plane authorization | 3 corrected; a note: a signed op's window has no length cap → R-1265 | +| `05` hub | 9 corrected (the 6-hourly DB copy, version literals, the fifth bind-link path, 44 event types, the retired DR record) | +| `06` off-site connectivity | 7 dated notes (the design-of-record kept); „no inbound listener" → R-1266 | +| `07` backup | 12 corrected (portable secrets, R-104 fixed, keep-daily, D22/D20 released, a refusal row); §8.2c notes → R-1265; the kept-data load beside a copy → R-1256 | +| `08` alarm ladder | 10 corrected (per-type cooldowns, R-723, per-app grain, R-998 skip, §8.1 built, §9 released, the operator-only box alarms); one ruling line kept with a dated note (operator Q1 made `backup_integrity_failed` a default) | +| `09` updates | 19 dated notes and corrections (the §1 measurement, slice 6 built, the undo as shipped, digests, phase table); §6.4.x notes → R-1265; three rulings the code does not do as written → R-1266 | +| `10` localisation | 7 corrected (Part B built, ceiling 0, the wizard deleted, counts) | +| `11` OS updates | 7 corrected (cloudflared 2026.9.3, sudoers lines, the wrapper built, 32 bundle files, HOST_SLOW_RE); the System page's „Upload line" cell added | +| `12` agent tooling | 3 corrected; the five rule copies aligned (the instruction-file edits below) | +| `13` threat model | 11 facts corrected, every verdict cell left as it was and marked „re-check"; a note on the file manager's sign-in trust | +| `14` power-off | 4 corrected (sudoers vectors, line-test holder, FAQ published, the table split); notes → R-1265 | +| `15` media library | a dated note on the §10 measurement table; R-1019's fix; notes → R-1265 | +| `16` storage identity | 4 corrected; notes → R-1265 | +| `02` controller module map | 8 dated notes (markers, holders, banner alerts, cloudflare still wired, updater live, LockedFields, 22 packages with no row) | +| `_hub-review.md` / `_recovery-inventory-2026-07-28.md` | a HISTORICAL banner / a „what has moved since" banner — both dated records kept as they were | +| `00` capability map | 20 PROVEN-LIVE rows marked „re-check" with the commit that changed their mechanism (no status lowered); G15's „nightly" corrected | +| `felhom-system-poster` facts + html | the timeline's hub-DB push (text only; no drawing needed) | + +**Rules, fences and promises were not edited.** Seven items that need the operator are in R-1266 (D45); about 30 +behaviours with no document are written as `[DESIGN — undocumented, found 2026-10-24]` notes for the operator to confirm +(R-1265, D44). + +**Instruction files edited (unprompted-work §5 — facts only):** the workspace root, felhom-agent and app-catalog copies +of `.claude/rules/unprompted-work.md` got the missing „A version number is pushed once" paragraph (now byte-identical to +the other two); `felhom.eu/.claude/rules/live-probes.md:41` — the stack path `/opt/felhom/stacks/` → `/opt/docker/ +stacks//app.yaml`. Details: `EDITS.md`. + +## Parts 2 and 3 — findings, ranked + +| # | Sev | Finding | What was proven | Row | +|---|---|---|---|---| +| 1 | **P1** | A power cut during a restore of a drive-backed app: the app runs on the cut restore's data, and a later controller restart pours the pre-restore copy back over newer writes | the real `Recover` code in a lean clone: boot 1 „copy gone → refused → marker kept", boot 2 put-back after the app ran (`proofs/H1-drive-cut-restore/`); the boot recreate's missing check read at `intermediary.go:470-476` | R-1251 | +| 2 | P2 | One rejected backup block makes the night's second-drive copy delete the app's household-file copies | **reproduced on 9202**: control kept the copy twice; one bogus path → „removed stale dest dir userdata/media" (`proofs/H2-bad-backup-block/`) | R-1252 | +| 3 | P2 | An export with „stop first" overwrites another app's restore marker | static, read at `file:line` | R-1253 | +| 4 | P2 | The hub saves the escrow package in two writes; a lost answer tells the household „the previous code still works" | static | R-1254 | +| 5 | P2 | D4's sshd drop-in reaches BYO hosts (against `11` §1); the in-force check and the all-or-nothing `sshd -t` | static | R-1257 (D43) | +| 6 | P3 | Escrow carry / set-aside name edges | static | R-1255 | +| 7 | P3 | Four restore edges (definition write, merge, a mirror race, the unused copy) | static | R-1256 | +| 8 | P3 | ep0's iperf3 takes any tunnel peer's test | static (no ep0 change) | R-1258 | +| 9 | P3 | The DooPlex→ep0 copies: the vouched agent can drop out; deletions mid-transfer; no future bound; **2026-10-25: the 02:30 push re-sends the 20:00 snapshot** | DST timing verified on DooPlex (`systemd-analyze`, Go) | R-1259 | +| 10 | P3 | Spring 2027: the leg's deadline vs the gate's clock; two legs at one instant | Go's mapping verified | R-1260 | +| 11 | P3 | The unreleased R-1250 line hold: two seeders, a removed app, alt-speed | static | R-1261 | +| 12 | P3 | Any viewer explains any upload (the upload watch) | static; `08`'s limits list now names it | R-1262 | +| 13 | P4 | Small items (a mislabelled operator mail, undeclared volumes on Remove, mount views, stale comments, a DNS-name password, rrsync) | static | R-1263 | +| 14 | P4 | Skills tooling (no size cap on the grandfathered skill; a new skill not installed) | read | R-1264 | +| 15 | P4 | Immich leaves an empty anonymous volume per recreate (296 on 9202); photos unaffected | read in the running image | R-1267 | + +**Fixed in this session (small, not recovery code, red-proved, commit only — controller `b01bb4b`, CI 2294 / #480 success):** +a media restart („Újraindítás most", „Később", the safe removal) no longer starts an app the household stopped (R-721; +two tests, each seen failing with the rule off); the file manager's header name file appears whole (a race the +existing test caught 44 times in 300 runs on clean `main`; 0 in 300 after, 0 in 50 under `-race`). + +**NOT-A-FINDING:** SEC-1 X2 (a passphrase enroll after the operator's host delete gets ciphertext — the window D19 +accepts); SEC-1 X3 (the refused-passphrase mail is operator-only with a one-hour cooldown); DOC-D1 U4 (the fit check +that passes on a statfs error — the copy then fails and the app restarts untouched, `restore_unit.go:502-512`); +HUNT-3 D31's comment wording (no data effect); Immich's `/data` volume for data placement (R-1267 holds the leak); +the stack folder left after a clean removal (it is every catalog app's template folder). + +## Machines + +- **9202** (Tier 0): as found at 12:46 UTC (`9202/as-found-2026-10-24.txt`); at the end (`9202/at-end-…`) the same + containers and images, healthy (restarted by the test night chains), the catalog on the live repo, the controller + 0.342.0. The only difference: Immich's running container uses a new empty anonymous volume (R-1267); the two others + my test runs made were removed by name. audiobookshelf installed and removed through the product; my marker file + removed; its second-drive copy removed by the product's remove. +- **DooPlex:** reads, lean clones in the scratchpad, the gates; no service changed. **ep0, household boxes, the hub:** + not touched (hub reads only, none needed in the end). + +## Teardown (three layers) + +Machine: 9202 as above. Host (demo-hp): nothing provisioned. Hub: nothing changed. diff --git a/documentation/audits/review-round-2-2026-10-24/reviewers/FENCES-given-to-every-reviewer.md b/documentation/audits/review-round-2-2026-10-24/reviewers/FENCES-given-to-every-reviewer.md new file mode 100644 index 00000000..ae1788eb --- /dev/null +++ b/documentation/audits/review-round-2-2026-10-24/reviewers/FENCES-given-to-every-reviewer.md @@ -0,0 +1,37 @@ +## FENCES — the brief's, word for word (operator ruling 2026-10-07: every helper prompt carries them in full) + +From the brief's §0 Scope: +- **Class:** review + verification, unattended. **Code is not changed** except a fix that is small, not in recovery code, + and has a red-proof (commit only). Documents may be corrected for facts proven at `file:line` (PROMPT-TEMPLATE §9 + rule 9: facts yes, rules and fences never without the operator). +- **Machines:** DooPlex (lean clones, builds, read-only reads), 9202 for proofs (put back as found). No household box, no + ep0 change, no release. +- **Coordination:** the catalog sweep runs in parallel on demo-hp's nested VM and writes to `felhom.eu`. `git pull + --rebase` before each push; never remove its lines. **Do not use 9202 between 18:00 and 20:00 if the sweep asks for it** + (it should not; it has its own VM). +- **Reviewers:** read-only subagents; only the main session touches machines, git and the register. + +From the brief's Rules: +1. Verify every claim at `file:line` before acting; a reviewer's word is a candidate, not a finding. 2. Documents: facts +yes; rules, fences, „never" lines and protected-machine text never without the operator. 3. Code: only small, non-recovery +fixes with a red-proof, commit only. 4. No release, deploy, vouch, publish, ep0 or DooPlex change. 5. Secrets: never +printed or committed. 6. 9202 put back as found. 7. Every finding a row, a fix, or `NOT-A-FINDING: `. 8. Stop at a +clean point. 9. No branches; no „Co-Authored-By". + +From the workspace CLAUDE.md (this host is production infrastructure): +- NEVER run `docker system prune`, `docker image prune -a`, or any global Docker cleanup here. +- NEVER touch k3s data dirs, Longhorn mounts, PBS datastores, or Gitea storage. +- Destructive disk/guest operations belong to felhom-pve via the agent — never on this host. +- Secrets: never `source` or `cat` a credentials file; never print a token, password, key or `PlexOnlineToken`. + +## WHAT THIS MEANS FOR YOU, THE HELPER (binding) + +You are a **read-only reviewer**. You may ONLY read files in `/mnt/5_hdd/felhom.eu/git/` (Read, Grep, Glob, and +read-only shell such as `grep`, `rg`, `sed -n`, `git log`, `git show`, `git blame`). You may NOT: edit or write any file; +run `git` commands that change anything (no commit, checkout, reset, stash, pull, push, worktree); run `go test`, `go +build`, any gate script, `docker`, `kubectl`, `ssh`, `curl` to any host, `sec`; touch 9202, demo-hp, the N100, Tester 1, +Tester 2, ep0, the hub or DooPlex's services. Only the main session touches machines, git and the register. If you +think something needs a live test, write the test plan in your answer — do not run it. + +Repos (read their `main` working trees): `felhom.eu` (hub in `hub/`, docs in `documentation/`, scripts in `scripts/`), +`felhom-controller` (Go in `controller/`), `felhom-agent`, `app-catalog-felhom.eu`. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 4d3ca7d6..d8b00692 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -92,12 +92,13 @@ deferred work, and collapsing them into one number is how a blank stops meaning * **row 13** — `NONE for host-loss` **by design**; R exists in zero system copies, so there is no route. * **row 14** — `PROVEN`; the break-glass route works and has simply never been stopwatched. * **row 15** — an **open DEFECT** (R-104, the stale-lock path), not a deferred recovery. **It is NOT - inside this stopping line** and must not be read as parked by it. + inside this stopping line** and must not be read as parked by it. *(R-104 was CLOSED 2026-10-05, controller v0.297.0 — + CLOSED-ITEMS.md; noted 2026-10-24.)* * **row 11b** — a consequences note attached to row 11, not a recovery row of its own. **What is NOT deferred and stays open:** **R-95** and **R-433**, both `BLOCKED-ON-PROVIDER` behind `documentation/runbooks/provider-questions-2026-09-01.md`; **R-435**, documentation only; and -**R-104**, row 15's defect. The alarm text (**R-434**) was fixed the same day and is closed. +**R-104**, row 15's defect *(closed 2026-10-05, v0.297.0 — noted 2026-10-24)*. The alarm text (**R-434**) was fixed the same day and is closed. **THIS REOPENS IF:** a customer-facing recovery path is found broken; **or** Hetzner's answers change what the snapshots are worth (either answer in Part 3's file can do it — "full restore only" makes @@ -122,7 +123,7 @@ stopping line that lies. | **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC | | **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator | -## Apps & catalog — 15 rows (P3 6, P4 9) +## Apps & catalog — 16 rows (P3 6, P4 10) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -141,6 +142,7 @@ stopping line that lies. | **R-1043** | Apps & catalog | P4 | **Sonarr, Radarr and Prowlarr build their login redirect as an absolute `http://` URL.** They trust forwarded headers only from their „Trusted networks” (the household set `192.168.0.0/24`; traefik is `172.18.x`), so they ignore `X-Forwarded-Proto: https` and answer `Location: http:///login` to an https visitor. Since controller v0.312.0 that hop is sent back to https by traefik (a GET, no password), so it is cosmetic; a template could seed traefik's network as trusted. (`audits/https-only-2026-10-14/``1A-3-4.txt`) | **OPEN** — filed 2026-10-14 | — | Seed the docker network as a trusted proxy in the *arr templates (with the R-1030 seed), measure the redirect | CC | | **R-1099** | Apps & catalog | P4 | **The N100 may hold RomM games in the old upload place, now shown under a fake „roms” platform.** Until 2026-10-16 RomM's own upload wrote to the drive's `roms/roms//`; after the mapping fix (catalog `fb18471`, R-1089) those files are at RomM's `library/roms/roms/…`. Measured on 9201: such a game showed as missing in its platform and again under a platform „roms” (`audits/romm-ghost-2026-10-16/B/b-after.txt`). Files put in with the file manager are not affected — they are now found. The N100 runs RomM (hub `/apps/romm`); CC does not log in to the lived-in box. | OPEN — filed 2026-10-16 | Operator: look at `roms/roms/` on the N100's drive in the file manager | If it holds platform folders: move them up one level into `roms/`, then run „Scan” in RomM; if it is empty, close | Operator | | **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | WATCHING — **2026-10-22 night:** not reproducible tonight; the Immich server log of 19:38 is gone (Immich restarted at 02:16 for the off-site dump). What the household sees: the upload fails after 10 minutes and the Immich phone app sends it again. | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC | +| **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. | OPEN | — | map `/data` explicitly (or set `IMMICH_MEDIA_LOCATION=/usr/src/app/upload`) through a tested catalog step; clean the old volumes through the product on each box | CC | ## App updates — 5 rows (P2 1, P3 4) @@ -153,13 +155,17 @@ stopping line that lies. | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC | | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC | -## Backup & restore — 47 rows (P2 11, P3 19, P4 17) +## Backup & restore — 52 rows (P1 1, P2 11, P3 22, P4 18) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| +| **R-1251** | Backup & restore | P1 | **A power cut during a restore of a DRIVE-backed app: the app then runs on the cut restore's data, and a later controller restart pours the pre-restore copy back over everything written since — the D22 promise is false for every drive-backed app.** Found 2026-10-24 (review round 2, HUNT-1 H1). Read at `file:line` (controller `321df1f`): the undo copy lives on the app's drive (`internal/backup/offbox_restore_undo.go:47-48`); `AppStopGuard.Recover` runs at start BEFORE the drive is bound, finds the copy „gone” and restarts „as before” (`appstop_marker.go:362-367`); the gated start refuses (drive absent) → marker and Undo KEPT (`:383-387,408-411`); the drive binds and the boot recreate starts the app checking only „held” (`internal/web/intermediary.go:470-476`); the household writes; the next controller start (e.g. the nightly self-update) finds the copy present and pours it back. **Proven with the real `Recover` code** in a lean clone (`audits/review-round-2-2026-10-24/``proofs/H1-drive-cut-restore/`); NOT reproduced on 9202 (its drive is mounted at boot and its agent API is off, so the recreate never runs there). Same family: H2 — a FAILED crash put-back leaves the app stopped and unheld, so Start or the recreate runs it and a later restart pours an old copy back (`appstop_marker.go:368-371`; `api/router.go:690-709`); H7 — Stop pressed during a restore skips the put-back after a cut (`appstop_marker.go:356-361`). | **OPEN** | recovery code — not fixed in an unattended review | an attended fix: an active marker holding an Undo for an app blocks EVERY start path (Recover's, the boot recreate, Start, the update) until the put-back ran; a copy on an UNBOUND drive waits, it is not „gone”; the drill on Tester 1 (a real bind at boot) | CC | | **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | NARROWED — **2026-10-23 (operator yes D33, D34):** the hub DB now leaves every 6 hours (hub 0.167.0 + `scripts/hub-db-backup/`, alarm at 13 h; restored back byte-identical) and the released images and agent packages leave nightly (`scripts/release-copy/`, R-1227 closed). (c)–(g) unchanged. **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest **2026-10-24 (attended): `DooplexGiteaOffsiteStale` fired from 00:57Z — not this path's fault.** The 00:20 push refused, as designed, a 12 h old Gitea dump: the PostgreSQL dump CronJob had failed every run since 2026-10-23 16:00Z (`28P01`) because the window reboot restarted the database and CNPG put the bootstrap secret's password in force (homelab `531023d`, operator's pick: the CronJob only). Fixed; dump, push and restore test run by hand (26 456 files, 12 repos `git fsck`, `gitea.dump` 1114 entries); cleared 10:51Z (`audits/dooplex-alarms-2026-10-24/`). Backup alarms now arrive as ONE mail every 4 h (homelab `22d7697`). **2026-10-23 night — survey again, read only (`audits/dooplex-survival-2026-10-22/PLAN.md`):** both nightly off-site jobs and both weekly restore tests green (copies 2026-10-22, tests 2026-10-18); the local tree is still the same box (2026-10-22 03:16 ok). New in the survey: the hub DB copy is up to 24 h old (G5: every 6 h — the operator's yes; a DooPlex change). | — | — | operator | | **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):** `_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). **FIRST VERDICT READ 2026-10-21 (rehearsal session, read-only):** the 07:10 CEST run judged three tiers against yesterday — demo-felhom 5, demo-hp 14, tester-1 4 new snapshot files, „each inside a reported box run"; nothing vanished, nothing unexplained; Tester4 stored its first baseline (44 files); Tester-2's own listing failed (no `snapshots/` dir — the box has never pushed). No `offsite_snapshots_vanished` / `_unexplained` mail in the operator mailbox (second channel). Evidence `audits/due-checks-2026-10-21/R-895-hub-0710.txt`. The positive alarm path is still proven only by the unit red-proofs. | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC | | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | NARROWED — **2026-10-23 (D32):** the sheet is current — names and places only, one A4, HU + EN (`runbooks/break-glass-sheet.md`); every place checked to exist. LEFT: the operator prints it, fills it by hand, keeps it outside the home. **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G1/G3/G10):** unchanged; the sheet is still the one act that breaks the circle, and S5's offline form is unknown to CC — on the sheet too (decide list in STATUS). | operator: print the sheet | Operator: print + store the sheet; then close | operator | +| **R-1252** | Backup & restore | P2 | **One rejected backup block in an app's `.felhom.yml` makes that night's second-drive copy DELETE the app's copies of the household's files.** REPRODUCED on 9202 2026-10-24 (`audits/review-round-2-2026-10-24/``proofs/H2-bad-backup-block/`): audiobookshelf with a marker file in `media/audiobooks`; two unchanged runs kept the copy (the positive control); one bogus path in the block → „backup block rejected … matches no compose bind” → the legacy legs (appdata only) → „Tier 2 reconcile: removed stale dest dir userdata/media”; the original on the drive untouched. Mechanism: `internal/stacks/metadata.go:550-557` (any defect rejects the WHOLE block) → `appbackup/classify.go:217-221` → `backup/tier2_capture.go:84-100` → `tier2.go:492` `tier2Reconcile` (`:325-329`), against D25's own header (`tier2_kept.go:17-18`). Same family (static): a D25 kept copy is reconciled away when its leg leaves the declared set and its record stays (HUNT-2 H2, `tier2.go:485-492`); Remove with `remove_backups` deletes kept copies and leaves their records (HUNT-2 H9, `api/router.go:1086-1087`). | OPEN | — | an attended fix in the backup code: a rejected or absent classification never reconciles (keep every dest dir that night, alarm instead); a kept copy stays kept; Remove drops the kept records it deletes; the red-proof is this run's 9202 steps | CC | +| **R-1253** | Backup & restore | P2 | **An export with „stop the app first” during another app's restore overwrites the ONE app-stop marker file — the restore's crash record and its Undo are lost; a power cut then leaves the restored app on mixed data with no put-back.** Found 2026-10-24 (HUNT-1 H3): `AppStopGuard.Begin` writes unconditionally (`internal/backup/appstop_marker.go:279-292`) and `End()` removes the file (`:306-320`); the exporter takes the guard (`internal/appexport/export.go:139-143,262-266`) and neither `StartExport` (`:173-178`) nor the web handlers (`internal/web/handler_export*.go`, grep: no restore check) look for a running restore. Static. | OPEN | — | refuse an export while a restore or an app-stop marker is active (or one marker per operation); a test with two Begins | CC | +| **R-1254** | Backup & restore | P2 | **The hub saves a household's new escrow package in TWO writes; a failure between them leaves the new repository hash beside the OLD identity package, and the box then marks itself escrowed — on a total loss the written code opens nothing.** Found 2026-10-24 (HUNT-3 H1), read at `file:line`: `hub/internal/api/handler.go:1415` `SaveHostEscrow` commits blob + hash, `:1445` `SaveHostDRBundle` sets `identity_blob` in a second statement (`internal/store/store.go:3788-3796`; the upsert `:3490-3511` never touches it); the controller's confirm reads the hash (`felhom-controller/controller/internal/report/escrow_confirm.go:195-217`); the agent sends once. Same family (H2): a lost or late answer (30 s) tells the household „the previous code still works” when the hub may already hold the new package (`felhom-agent/internal/localapi/escrow_ceremony.go:79,532-534,639-641`). Static. | OPEN | — | one transaction for both writes (a store test: the second write fails → nothing stored); the ceremony asks the hub what it holds before it says „the previous code still works\” | CC | | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC | | **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC | | **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC | @@ -167,6 +173,9 @@ stopping line that lies. | **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC | | **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. | — | — | CC | | **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) | +| **R-1255** | Backup & restore | P3 | **Escrow edge cases (review round 2, HUNT-3 H3/H4):** the R-1176 carry moves the newest kept package even when it holds no repository password (`hub/internal/store/escrow_carry.go:100-110` checks `identity_blob` only), while the right one stays under the deleted host (`store.go:3533-3534`); two set-asides on the same UTC day reuse a name and the purge can delete a still-stored copy's package (`hub/internal/offsitekeys/offsitekeys.go:378-385`; `store.go:4295-4300`). Static. | OPEN | — | carry the newest package that has a hash; a unique set-aside name per run; store tests | CC | +| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. | OPEN | — | one attended pass over the restore paths with these four as tests | CC | +| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC | | **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC | | **R-164** | Backup & restore | P4 | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC | | **R-213** | Backup & restore | P4 | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **DEFERRED** — operator ruling 2026-10-12 (S18, `09` §3 decision 213): designed after the first customers. Stays P4; re-dated: look again when the first paying household has run 30 days. **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | After the first customers | Operator + CC | @@ -201,6 +210,7 @@ stopping line that lies. | **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC | | **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go` `newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC | | **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC | +| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC | ## Storage & devices — 11 rows (P3 4, P4 7) @@ -218,16 +228,18 @@ stopping line that lies. | **R-1094** | Storage & devices | P4 | **The whole-disk format refusal tells the household to format the free part — which is refused while the other part is in use.** SEEN 2026-10-16 (controller 0.318.0): 409 „Ezen a lemezen egy használatban lévő meghajtó van, ezért a teljes lemez nem formázható. Csak a szabad részét formázhatod." — since agent 0.158.1 a part next to a mounted part can only be USED (the root wrapper trusts no „Felhom's own" claim). Evidence: `audits/storage-identity-buildB-2026-10-15/live/14-samedisk-refusal-offer.txt`. | READY | — | Reword `err.web.meghajto_hasznalatban_formazas` (hu + en): „… nem formázható. A szabad részét használatba veheted." Next controller release; the copy gates run. | CC | | **R-1213** | Storage & devices | P4 | **Operator detail still on two household pages:** the registered drive card's `ext4 · /dev/sdb1 · Model` line (`templates/storage.html`; W14 hid only the agent drive cards), and the heading „Távoli mentés (3. mentés) — titkosított, offsite" with the English „offsite". Found by the night helper 2026-10-21 (W11/W14 follow-up). | OPEN | — | Move the line behind „Részletek"; „…titkosított, házon kívüli" | CC | -## Security & access — 23 rows (P1 2, P2 3, P3 12, P4 6) +## Security & access — 25 rows (P1 2, P2 4, P3 13, P4 6) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-1241** | Security & access | P1 | **2026-10-24: the Felhom half is DONE** — hub 0.168.0 deployed (the tokens sealed; every copy leaving the volume is clean, the live file's old bytes accepted by the operator, R-975 closed), the hub volume out of the Longhorn backup (D40, R-1243 VERIFY), the two old plain copies deleted by the operator. **What stays here is the homelab half:** the open NFS export and Longhorn's login (the homelab session). **A hacked app pod on DooPlex can read the hub's nightly database copies — with every customer's unsealed Cloudflare, tunnel and git token (R-975) — through the Longhorn NFS export, and by a published file-handle escape the operator signing keys on the same disk: whoever breaks one homelab app can take over the boxes' tunnels and sign agent updates.** MEASURED 2026-10-23 (Plex2Shell impact audit, `audits/plex2shell-felhom-impact-2026-10-23/`): export `/mnt/5_hdd/backup/longhorn-pvc` rw, `no_root_squash`, `no_subtree_check`, a sub-directory of sda1, which also holds `/mnt/5_hdd/felhom.eu/felhom-op-*` (no passphrase, R-924); `secure` is void in pods (containerd sets `ip_unprivileged_port_start=0`; a uid-1000 probe with no capabilities bound port 700); the four `felhom-system` volumes have `encrypted=false`; Longhorn's API on :9500 answers any pod with no login (backups, targets, volumes). In the Plex2Shell window neither door was used — verdict NOT REACHED (nfsd accepted 0 new connections 10-10 → 10-23, positive control; no off-schedule Longhorn backup or volume). 2026-10-23 ~15:18Z, operator yes: the export was narrowed from `*` to `10.42.0.0/16` — the home network and the tailnet are closed, pods are not (Longhorn connects from its pod address). Not exercised: the escape itself (Tier-2 machine). | OPEN | the homelab session (DooPlex host + cluster); the operator (R-924) | Homelab: a NetworkPolicy so only `longhorn-system` reaches :2049/:111 and :9500, or the backup store on its own filesystem. Felhom: R-975 sealed in hub 0.168.0 (deploy pending); the hub's plain Longhorn copies → R-1243; a passphrase on the signing keys (R-924) | operator | | **R-1139** | Security & access | P2 | **Tester 2's break-glass root password was not rotated with the others (R-954, operator decision D1 = A, 2026-10-17): the box has been offline since 2026-10-04.** demo-hp, Tester 1 and demo-felhom were rotated and proven on 2026-10-17 (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | OPEN — filed 2026-10-17 **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a fresh install sets and vaults a new console password).** | Tester 2 reporting again | When it reports: the installer's step 4b alone (`P3/rotate-step4b.sh` shape), then the vaulted value logs in (PVE API 200) and the old one is refused | CC | +| **R-1257** | Security & access | P2 | **D4's root-key-only sshd drop-in reaches BYO hosts too, while `11` §1 says a BYO host gets guest and Docker updates only — never a host change — and the installer's BYO screen lists what it leaves alone.** Found 2026-10-24 (SEC-2 X1), read at `file:line`: `felhom-agent/configs/felhom-os-apply:230` writes the drop-in in every mode (`plan_writes` holds back only policy `oob`, `:1985-2000`; `check_appliance` guards only the OS layer, `:590,1070`); `felhom.eu/scripts/felhom-host-install.sh` (the BYO screen). A BYO owner who logs in to root by password over SSH loses that at the next bundle. Same area: the „in force” check probes only `addr=192.0.2.1`, so a `Match Address ` block reads as in force (X3, `:2101-2110`); a failing `sshd -t` refuses and undoes the WHOLE bundle (X4, `:2102,2112-2115`). | **WAITING-ON-OPERATOR** | the operator's choice (D43) | rule D43; then CC changes the bundle policy and the check | operator | | **R-1042** | Security & access | P3 | **The boxes' Cloudflare keys cannot change zone settings, so Cloudflare's own „Always Use HTTPS” cannot be switched on.** Measured 2026-10-14 on all three zones (`audits/https-only-2026-10-14/``1A-1-zones-and-keys.txt`): each `cf_api_token` is valid and sees only its own zone, but GET/PATCH `settings/always_use_https`, `ssl`, `security_header` answer 403; the redirect and transform rule phases too. The https-only fix therefore lives on the box (decision 246). A second layer at Cloudflare needs „Zone Settings: Edit” (and, for headers, „Transform Rules”) added to each customer's key — the hub's zone-reach check stays. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | a key permission only the operator can grant | Optional: widen each customer key, then switch „Always Use HTTPS” on per zone | operator | | **R-1242** | Security & access | P3 | **The demo NAS export `/mnt/5_hdd/felhom-data` → 192.168.0.162 (demo-felhom) is a sub-directory of DooPlex's sda1 with `no_subtree_check`, so a compromised demo-felhom host could walk the whole disk as uid 101000 and read every world-readable file there (the GPG secrets exports are 0644, every repo, the public keys).** Measured 2026-10-23 (`audits/plex2shell-felhom-impact-2026-10-23/` Part 1; `exportfs -v`: `all_squash,anonuid=101000`). 0600 files (the signing keys) stay out of reach. Not exercised. | OPEN | a DooPlex host change (fence) | Give the demo share its own filesystem, or `subtree_check`; or remove other-read from `/mnt/5_hdd/backup/secrets/exports` | operator | | **R-1104** | Security & access | P3 | **Check https-only on Tester 2 when it reports again.** R-1033 (plain http on every public name) is closed on the three boxes that report (35 names → 301, 2026-10-16). Tester 2 (`sajatfelhom.hu`) has been off since 2026-10-04 on controller 0.292.0; its addresses answer Cloudflare 530 (tunnel down), so nothing is served in plain http while it is off. When it comes back it takes the floor (≥ 0.312.0 carries the redirect). | OPEN — filed 2026-10-16 | Tester 2 reporting again | Run `scripts/https_only_check.py` with Tester 2 in it; close | CC | | **R-1141** | Security & access | P3 | **`felhom-host-install --rotate-recovery` exists only as part of a full installer re-run, which also reinstalls the vouched agent and the root files.** On a box running a newer agent than the vouched one (demo-hp on the candidate, 2026-10-17) it would move the agent back; on a household box it is a much larger act than a password change. The 2026-10-17 rotation therefore ran the installer's own `step_break_glass` ALONE (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | **NARROWED 2026-10-18 — built on `main`, not published.** `felhom-host-install.sh --rotate-recovery-only` runs step 4b alone (felhom.eu `7d64038f`): reads the hub, host id and host key from the agent config, refuses byo and every other mode flag, asks the hub's `/healthz` before the password changes, never touches the agent or root files. The default install path is unchanged in behaviour (harness verdicts and 9 transcripts identical old vs new; demo-hp `--preflight-only` identical; `--rotate-recovery-only --dry-run` on demo-hp changed nothing). `step_break_glass` moved, byte-identical, ahead of the argument parse. `break-glass.md` §5 points at it. Publishes only with the next `installer-v…` tag. | the next installer tag | Cut the next `installer-v…` tag (with the other untagged installer changes); close | CC | +| **R-1258** | Security & access | P3 | **ep0's iperf3 line-test server takes any tunnel peer's test: one box (root on its host) can keep it busy for the fleet or ask a long reverse test that fills ep0's uplink.** Found 2026-10-24 (SEC-2 X2): `felhom.eu/scripts/ep0-iperf3/felhom-iperf3.service:17` — no authorized users, no server-side duration or rate cap; one client at a time (`felhom-agent/internal/linetest/iperf.go:263-264` → `busy`; 6 tries a day). Static (no ep0 change in this run). | OPEN | — | per-box iperf3 users or server-side caps, tried on a scratch VM first | CC | | **R-955** | Security & access | P4 | **The cross-box check of the hub API is done in source and tests but not fully proven: if a later check masks a missing id check, a box's key could read or change another box's hub record without any test noticing.** Read 2026-10-12: every box-facing `/api/v1` route takes the box from its own key or refuses a different id; 14 routes already had a refusal test, 5 got one (`crossbox_refusal_test.go`). Left: red-prove the 4 new tests that were not red-proved (as the restore-directive one was), and the live read-only check with one demo box's key against the other demo box's id. | OPEN — filed 2026-10-12 by the security review | — | Red-prove the 4 tests; run the live GET check on the two demo boxes | CC | | **R-924** | Security & access | P2 | **The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment.** MEASURED 2026-10-09: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` (0600 `kisfenyo`); `ssh-keygen -y -P ''` opens all three (no passphrase); `/mnt/5_hdd/felhom.eu` is in no backup set, the off-site job does not carry them. `04` §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; `04` §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). `runbooks/total-loss-of-dooplex.md` S6. | NARROWED — **2026-10-23 (D32):** S6a/S6b are on the current sheet. LEFT: the operator prints it. **NARROWED 2026-10-09 — operator ruling: option A** (print both AND copy them). The three key files and their public halves ride the nightly encrypted DooPlex→ep0 copy (`scripts/dooplex-offsite/`, a missing key refuses the push); the Sunday restore test proves each private key still derives its own public half (never printed). **LEFT:** the operator prints S6a/S6b (`runbooks/break-glass-sheet.md`); the keys still have no passphrase (custody ruling 1 of 2026-09-16 stands until the first sale). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G2):** the three key files are on `sda1`, the same disk as DooPlex's local backup tree; their only off-box copy is the S1-encrypted nightly one. Passphrase: unchanged by the custody ruling. | operator: print | Operator: print the two keys; then close | operator | | **R-132** | Security & access | P3 | **`curl -w '%{redirect_url}'` reconstructs the request URL WITH its basic-auth credential** — so a `-u ":$HUB_PW"` call that never put the password in a URL still printed it **Merged 2026-10-05 from R-350 (duplicate):** (1) 2026-08-20 occurrence: POST /configuration/artifacts answers 303; leak lives only in the CC transcript under ~/.claude/projects/ on DooPlex, not in git/evidence (checked then). (2) `-v` and `--libcurl` also re-render the credential, not only %{redirect_url}; confirm redirects with %{http_code} + follow-up GET. (3) Rotation path: hub /configuration form (current_password/new_password/confirm_password); DB override wins over ConfigMap (break-glass); CC can rotate file-to-file without printing (operator-present-one-time-secrets) if asked. | **NARROWED 2026-10-20 — the rotation is DONE:** the operator changed the hub password on 2026-10-15 and the `HUB_PW` line is gone (`audits/hub-tokens-2026-10-15/C153-live.txt:18`; CC uses its own token since hub 0.152.0). **Left, the operator's:** should the break-glass `password_hash` in `manifests/hub.yaml` (a public repository) be emptied or replaced? The DB override wins at login, so the old hash no longer opens the hub. **WAITING-ON-OPERATOR** — **ACTION: rotate `HUB_PW`** **Folded R-580 2026-10-03** (the same `curl -w %{redirect_url}` credential echo, seen again 2026-09-18). **Note 2026-10-12 (security review):** `manifests/hub.yaml` also carries the hub login's bcrypt hash in this public repository (the break-glass field); when `HUB_PW` is rotated, decide whether that field should stay empty. | the operator's word on the `manifests/hub.yaml` hash field | Happened on 2026-07-31 while red-proofing the R-120 gate: the hub operator password was written to the session transcript by the write-out format, not by the request. `-u` is safe; the *reporting* was not. Rule: read the redirect from `-D -` and grep `^Location:`, never `%{redirect_url}`, on any authenticated call. Rotate the hub password (`/configuration` → Login password; ConfigMap `auth.password_hash` is the reset path) and update `~/.config/credentials` | Viktor | @@ -250,6 +262,7 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| +| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC | | **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC | | **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC | @@ -263,10 +276,12 @@ stopping line that lies. | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC | -## Monitoring & notifications — 23 rows (P2 2, P3 14, P4 7) +## Monitoring & notifications — 22 rows (P2 1, P3 15, P4 6) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| +| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | OPEN | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC | +| **R-1262** | Monitoring & notifications | P3 | **The upload watch treats ANY viewer as an explanation for any amount of upload, so a hacked Plex can send freely while one family member watches.** Found 2026-10-24 (SEC-1 X1): `felhom-controller/controller/internal/upwatch/watch.go:275,299`; `08` §9 states the rule, its limits list did not name this case. | OPEN | — | propose a cap (viewers × a bitrate ceiling) for the operator's R-1244 design; `08`'s limits list names the case meanwhile | CC | | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC | | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | | **R-1250** | Monitoring & notifications | P3 | **A qBittorrent that seeds all day is never line-tested, so it never gets the limit that exists for it.** The weekly test (R-1247) starts only after 10 minutes in which the host's uplink sent < 2 Mbit/s each minute; a seed at the line's speed keeps it busy through every window, so no test, no tested line, and no qBittorrent limit (D41 uses the tested line only). Found 2026-10-24 by reading the rule against the case it serves; not yet seen on a box (no test box seeds). | **NARROWED 2026-10-24 — FIXED on main, not released** (brief 2026-10-24 Part 4, commit only): agent `29bcf44`+`1e087bb` (`waiting_for_quiet`, `POST /host/line-test/run`), controller `046b808` (`internal/upwatch/linehold.go`: record first, hold at 1 KiB/s, test, the exact old limit back; the start-up check after a kill; the 50 % rule waits). Red-proofs in both CHANGELOGs. **Live on demo-hp with a real seed** (10 Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; held: uplink 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, back to 1 249 280 B/s exactly; a kill -9 mid-test left the record and the start-up check put the limit back (`audits/dooplex-alarms-2026-10-24/part4-r1250-live-proof.txt`). The live run used probes over plain HTTP (the pinned agent client is unit-tested, not live). | the next release train (agent ≥ 0.165.0 needs the operator's vouch) | Release agent + controller in the next normal train; on a seeding box read one Tuesday's `line test on request` line; then close | CC | @@ -326,11 +341,13 @@ stopping line that lies. | **R-990** | Business & legal | P4 | **Operator: read `documentation/audits/forum-research-2026-10-12/FELHOM.md` §5 and rule on each recommendation.** READ-ONLY research 2026-10-12: three years (#7943–#11092, 2,908 posts, 195 people, 82 beginner threads) of the public PROHARDVER! topic „Home server / házi szerver építése”, tagged and counted by person (`FINDINGS.md`, method and spot-check in `METHOD.md`; no username or post text in git). Ten ranked recommendations, none acted on; the top three: a „which machine works” page (8 of 33 beginners who name their hardware own only a Synology, a router disk or a Raspberry Pi — the installer cannot run there; the same gap is R-989's first recommendation from a different source, and the shipped-page half is R-991), „you do not choose an operating system — we do” (the biggest beginner question, 20 of 82 threads), and „bring your old office PC” as the cheap entry. Catalog gap (≥ 3 people): a DLNA server (7), an ad/DNS blocker (5), a camera recorder (5). | **WAITING-ON-OPERATOR** — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages | operator | | **R-1035** | Business & legal | P4 | **Did the „felhő tárhely” page earn anything? Two dated checks.** **Baseline (reported by the operator from Search Console, 2026-10-14, last 3 months):** 1 click, 48 impressions, CTR 2.1 %, average position **73.6**; „felhő tárhely” 19 impressions, „felhőtárhely” 5, „felhő alapú tárhely” 3. CC measured separately that the phrase „felhő tárhely” appeared **0 times** on the home, FAQ and backup pages — Google was showing the home page for those queries by accident. `/sajat-felho-tarhely` + `/en/own-cloud-storage` published **2026-10-14**; full baseline, price sources and the power provenance in `audits/seo-2026-10-14/BASELINE.md`. **What to measure, both times:** impressions and average position for „felhő tárhely” and „saját felhő tárhely”, plus the new page's own impressions, clicks and position. **A null result at +4 weeks means nothing on its own** — a new page on a site at position 73.6 takes weeks to months to move, and „felhő tárhely” alone is held by Microsoft, Google and the big list sites; the page aims at the two longer phrases. That is why there are two checks and not one, and why neither should be read as a verdict on the copy. **There is no Search Console connector in this project, so CC cannot read these numbers** — the operator reads them, CC records and compares. **Blocked until the operator asks for indexing** (Search Console → URL-ellenőrzés → the URL → „Indexelés kérése”): until that happens, or Google finds the page itself, both checks read zero for a reason that is not the page. **This is a slow channel and does not replace talking to people.** | **WATCHING** | operator: request indexing, then supply the Search Console read | 2026-11-11: first read (+4 weeks); 2026-12-09: second (+8 weeks), compared with the first | operator + CC | -## Process & tooling — 32 rows (P2 1, P3 8, P4 23) +## Process & tooling — 33 rows (P3 9, P4 24) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-1083** | Process & tooling | P3 | **The scratch box 9202 cannot use any feature that needs the agent: its `bootstrap.json` has no `local_api` block, so its controller never gets an agent client.** MEASURED 2026-10-15 (storage-identity build A): 9202 put on controller 0.317.0 by hand to show that a NAS share no longer hides the one-copy notice (R-1075); its `bootstrap.json` (read on the box) holds no `local_api`, so the controller has no agent to ask and keeps the old `st_dev` rule — the check could not tell old from new — 9202 was put back on 0.314.0. Every agent-coupled feature (drives, same-disk, format, network storage) is untestable on the scratch box today, and the NAS leg of build A is proven by a red-proofed unit test only. Evidence: `audits/storage-identity-buildA-2026-10-15/9202-upgrade.txt`. | **OPEN — the exact gap, read 2026-10-19 (no agent change tonight):** the agent mints a per-guest local-API token in ONE place, `--selftest=provision`'s back half (`cmd/felhom-agent/main.go` `bh.Provision` after `engine.RunBringUp`), which only runs for a guest it brings up from a golden archive in the same call. There is no product way to mint for an EXISTING guest (9202) — no subcommand, no local-API route; the token store `/var/lib/felhom-agent/local-tokens.log` is written only by that path. Options: an agent subcommand `--mint-guest-token ` (owner-only, writes bootstrap's `local_api`), or re-provision 9202 through `selftest=provision` (destroys and rebuilds the scratch guest — not done; rule 6). Until then 9202 is controller-only. OPEN **2026-10-20 — left out of agent 0.161.0 on purpose:** the code would be small (a root-only subcommand that mints the token and adds `local_api` to an existing guest's `bootstrap.json`), but an agent key makes 9202's controller book the agent's whole-guest backups, one tier of which writes to ep0 (the demo-hp namespace) — and 9202 is outside the `felhom` pool. Needs, before it ships: a way to give a scratch guest a key with NO whole-guest tiers (or the operator's word that 9202 may write to ep0), and 9202 in the pool. | — | Give 9202 a local-API token the normal way (the agent mints it for its guest), or record in `runbooks/target-selection.md` that 9202 is controller-only; then show the NAS notice live. | CC | +| **R-1265** | Process & tooling | P3 | **Decisions found in code with no document — written as `[DESIGN — undocumented, found 2026-10-24]` notes; the operator confirms them (review round 2).** The notes: `04` §2 (a signed op's window has no maximum length; opsign's default 30 min); `07` §8.2c (the 60-min copy limit and stray handling; Eltávolítás refused during a backup; a start that fails after a put-back is not held; the hub's off-site delivery checker and its R-1230 skip; the release copy has no matrix row); `09` §6.4.x (build.sh's three refusals; the once-retry; stopped_by_household; the manual leg's deadline; two preflight refusals; the unreadable-template row and the R-971 admission); `14` (a re-enroll during the hold is „gone”; two info events; three confirmation words); `15` (the „Később” timer is in memory; the Sonarr/Radarr key read from config.xml); `16` (detect-virt read once; an unreadable path = the same disk; the internal place's lifecycle refusals and its size rule); and R-1019's extension to MeTube, NZBGet and SABnzbd (catalog `c50adfc`: the brief named only qBittorrent and Transmission; MeTube's downloads may be gone from their source site — HUNT-2 H3). | **WAITING-ON-OPERATOR** | the operator's word (D44) | confirm all, or name the ones to change; a confirmed note loses its „undocumented\” tag | operator | +| **R-1266** | Process & tooling | P3 | **Rulings and promises in the documents that the code does not do as written — the operator rules whether the text or the code moves (review round 2).** (1) `09` decision 13: „the tag rule (`CompareImageRefs`) moves to the catalog gate” — it did not (`felhom-controller/controller/internal/stacks/updateorder.go:134`; `09` §6.4 part 4 says so); (2) decision 14: a box „never jumps” — a person's press on an app older than the ladder takes the catalog's current definition (`ladder.go:34-36,175-176`; the night leg does not jump); (3) decision 250 item 5: the golden gate judges „by its AGE … while the floor is above it” — the gate cannot read the floor and uses „release ahead” (`scripts/golden_currency_gate.py:8-12`); (4) `01` §1/§4 „the hub never connects into a box”, „no inbound control endpoint”, „no standing operator backdoor” — the hub pokes a box (UDP to its WG /32:51822 via ep0, `hub/internal/poke/client.go`; agent `internal/poke/poke.go:35`) and the operator's WG peer reaches felhom-sshd (`06` §4.5); `03` §5's break-glass „via the Cloudflare Tunnel behind Access” and `06` §6's „no inbound listener” are the same old text; (5) `01` §8: the operator „cannot open / read” the off-site backup — R-985 already softened the website; (6) `06` §8's plan that the agent manages cloudflared — `03` §2 says it will not. | **WAITING-ON-OPERATOR** | the operator's choice (D45) | „the text follows the code\” → CC rewrites each and the operator reads the diff; „the code follows the text\” → a row per item | operator | | **R-1240** | Process & tooling | P4 | **Scratch guest 9202 holds 34 saved copies of its `controller.yaml`, each carrying the catalog's git token** (`controller.yaml.pre-`, 2026-09-21 → 2026-10-17, root 0600, inside the guest's controller volume). Every drill session that repointed 9202 saved one and none removed it after its byte-identical restore. Found 2026-10-23 (the D39 session removed its own after `cmp` proved it identical). Not read: whether the token in them is still live (R-925 rotated tokens). | **OPEN** | — | read which token the copies carry (by hash, never printed); if the live one, delete the copies; add „remove the save after an identical restore" to `repoint.py restore` | CC | | **R-1081** | Process & tooling | P4 | **Two published controller tags hold content that is not what their number says: `0.315.0` and `0.316.3`.** Both were caused by CC on 2026-10-15, and both by the same shape — a guard that was not on the path the work takes. **`0.316.3`:** the build directory's `build.sh` is a symlink into the SHARED clone's working copy, which predated the `FELHOM_REPO_DIR` override, so the override was silently ignored and the build copied that tree — which held 31 uncommitted files of another session's physical-disk storage work. The banner printed `Commit: 0225255` with no `Repo:` line. **Nobody deployed it; that was luck, not a control.** Fixed at the root in controller v0.316.4: `build.sh` now refuses `--push` from a tree that is dirty, not at `origin/main`, or not a git checkout, with four test cases red-proved. **Neither tag is re-pushed or deleted** — a version number is pushed once, and that applies to numbers CC spoiled. **No box runs either:** demo-hp 9201 is on 0.317.0, the floor is 0.314.0, and the button offers the highest tag. **`0.315.0`:** Caused by CC on 2026-10-15 while proving R-1080's own fix: the guard was added to `controller/build.sh` in the repo, but the documented release command runs `$FELHOM_ROOT/build/felhom-controller/build.sh`, which was a **separate copy taken 2026-07-18** and had no guard. Running the refusal against a real existing tag therefore rebuilt and re-pushed 0.315.0 — a fourth time — with v0.316.0's code under it (digest `8887343011d6` → `b809201841f7`). **Fixed at the root in the same session:** the build directory's script is now a symlink to the repo's, the old copy kept beside it as `build.sh.stale-copy-2026-07-18`, and the refusal was then proven to fire from the documented path. **No box is affected:** nothing runs 0.315.0 (demo-hp 9201 is on 0.316.2), the floor is 0.314.0, and the household button offers the highest tag. What remains is two tags that must never be deployed and are no longer reproducible from the registry. | **OPEN** — filed 2026-10-15; nothing to do unless someone needs one of those two images, and then the answer is the current release | — | Leave them. If an audit ever needs the real 0.315.0 or 0.316.3, build it from the matching commit under a NEW number and say so | CC | | **R-1034** | Process & tooling | P3 | **Gate 17 checks the FAQ JSON-LD on two pages only, by a hard-coded list — a FAQPage anywhere else is unchecked.** `scripts/site_gates.py` gate 17 loops `for p in ("gyik.html", os.path.join("en", "faq.html"))`, so its „JSON-LD equals the visible questions and answers” guarantee stops at those two files. **Since 2026-10-14 two more pages carry a FAQPage block the gate never reads:** `/sajat-felho-tarhely` and `/en/own-cloud-storage`. Their JSON-LD was *derived* from the visible headings and paragraphs by a script and then verified verbatim against the rendered text, so it is correct today — but nothing stops the next copy edit moving the visible text and leaving the structured data behind, which is exactly the failure gate 17 exists to catch, and the one Google penalises. **Why it is not a one-line fix:** gate 17's extractor is written for the FAQ pages' accordion markup (`