Review round 2 (batch 2 + close-out): 02/07/09/10/11/14/15/16/inventory facts corrected at file:line (EDITS.md), undocumented decisions as notes (R-1265), rulings left for the operator (R-1266); register 190 -> 207 (R-1251..R-1267 opened: R-1251 P1 cut drive-app restore put back over newer data, R-1252 P2 rejected backup block deletes second-drive copies - reproduced on 9202); STATUS D43-D45; CONTEXT; audit README
gates / gates (push) Successful in 13m10s

This commit is contained in:
2026-10-24 16:12:30 +02:00
parent 77015997f8
commit 876d92367e
23 changed files with 551 additions and 70 deletions
+7
View File
@@ -63,6 +63,13 @@
> STATUS:** the default limit's size (CC: 50 % of the measured upload) and who is told (CC: the family only about the
> qBittorrent limit they can change; the unexplained alarms to the operator only — the R-388 rule).
> **2026-10-24 (evening, unattended) — review round 2** (`audits/review-round-2-2026-10-24/README.md`). The 17 architecture
> documents checked against `main`: facts corrected at `file:line` (EDITS.md), rulings and promises left for the operator
> (R-1266, D45), undocumented decisions written as notes (R-1265, D44). New: **R-1251 (P1)** a cut restore of a drive-backed
> app is put back over newer data (shown with the real `Recover`); **R-1252 (P2)** a rejected backup block makes the
> second-drive reconcile delete the household's copies (reproduced on 9202). Controller `b01bb4b` (unreleased, next
> v0.343.0): a media restart skips household-stopped apps (R-721); the FileBrowser SSO header file appears whole.
> **2026-10-23 ~12:00 — operator rulings by starting the D39 brief (`09` §3 decision 255; R-1239 closed).** **D39 = B:** a
> `security_fix` ladder step — box walk on 9202 required, bench optional, taken by the night leg. **No operator word per
> incident:** a public advisory id (CVE or GHSA) + its https:// notice is the authority; the session that writes the step
+33
View File
@@ -0,0 +1,33 @@
# REPORT — review round 2 (2026-10-24, unattended): the documents against the code, data loss and security
**Plain words first.**
- The 17 design documents were checked against the code: 493 claims, 121 out of date. About 130 places are corrected or
marked with a dated note; no rule, fence or promise was changed.
- About 30 things the code does that no document named are now written down for you to confirm (D44), and seven places
where a document's ruling or promise says something the code does not do are waiting for you (D45).
- The worst data-loss finding: a power cut during a restore of an app on a drive can later pour old data over the
household's new data (R-1251, P1). Proven with the real recovery code; it needs a fix in an attended run.
- Also proven, on the scratch box: one wrong line in an app's backup settings deletes that app's second-drive copy
(R-1252, P2).
- The worst security finding: the „root logs in by key only" SSH rule also reaches a host the household owns (BYO),
where we promised to change nothing (R-1257, D43). Nothing serious was found in the hub's or the controller's new routes.
**Evidence and the full tables:** `documentation/audits/review-round-2-2026-10-24/README.md`; every document edit with
its `file:line` in `EDITS.md`; the reviewers' candidates in `reviewers/`; the two proofs in `proofs/`.
**Commits (all CI green by hash):** controller `b01bb4b` (two small fixes, red-proved; unreleased) — CI 2294 / #480;
controller `2449223` (REUSE.md); felhom-agent `f5c3d77` and catalog `227a544` (the rule-file alignment); felhom.eu
`77015997` (documents batch 1) and this commit (batch 2, the register, STATUS, CONTEXT). The CI numbers of the last four
are in the final report.
**Register:** before 190, after 207; opened 17 (R-1251–R-1267), closed 0.
**Instruction files edited (facts only, named here):** `.claude/rules/unprompted-work.md` in felhom-agent, the catalog
and the workspace root — the missing „A version number is pushed once" paragraph (before 95 lines, md5 3b071ad2; after
byte-identical to the other two, b1891bd4); `felhom.eu/.claude/rules/live-probes.md:41` — `/opt/felhom/stacks/<name>` →
`/opt/docker/stacks/<name>/app.yaml` (the real path, and the folder stays after a clean remove).
**System poster:** the facts file and the html changed together (text only, the hub-DB push times); no new drawing.
**Machines:** 9202 used for the R-1252 proof and put back (one difference: Immich's new empty volume, R-1267). DooPlex,
ep0, household boxes and the hub: not changed. No release, deploy, vouch or publish.
+5 -1
View File
@@ -2,7 +2,7 @@
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-10-24 (afternoon, attended).** The two DooPlex backup alarms had one cause: yesterday's reboot changed which password the database takes, so the database copy job could not log in, and the Gitea off-site copy then refused its old copy. Fixed and cleared at 12:51 local; critical alarms now repeat every 4 hours (all backup alarms in one mail), warnings every 12. A box whose torrent app seeds all day can now get its weekly line test (proven on demo-hp, not yet released).
**Updated 2026-10-24 (evening, unattended review).** The 17 design documents were checked against the code; about 130 old facts are corrected, no rule changed. Two new data-loss holes are proven; both need a fix in an attended run. Two small controller faults are fixed, not released.
## What works
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
@@ -18,6 +18,7 @@
- The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15).
## What is broken
- **P1 (new):** a power cut while an app on a drive is restored can later pour old data over new data (R-1251); one wrong line in an app's backup settings deletes its second-drive copy (R-1252, P2).
- **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). The scratch box cannot use agent features (R-1083).
- DooPlex: something sets the shared checkout to „bare" (R-935).
@@ -31,6 +32,9 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|---|---|---|---|---|---|---|
| D45 | R-1266 | Six places where a design document says one thing and the code does another (two update rules, the golden check, „the hub never connects into a box", „the operator cannot open the off-site copy", the tunnel plan). Should the text follow the code? | **Yes — the text follows the code** (each is how the system works today; you read every changed sentence) | the code follows the text: six code changes, one of them to the box's door for you | the documents keep saying what is not true | — |
| D44 | R-1265 | Confirm about 25 behaviours the code has but no document named (now written as notes), and that MeTube's downloads are not backed up? | **Yes, all** (each was read in the code; none adds risk) | name the ones to change; each becomes a row | the notes stay marked „undocumented" | — |
| D43 | R-1257 | The SSH rule „root logs in by key only" also reaches a box the household owns (a BYO host), where we promised to change nothing on the host. Keep it there too? | **No — skip it on BYO hosts** (it is their machine; we said we would not touch it) | keep it: a BYO owner who uses a root password over SSH loses it at the next update, and the installer screen must say so | it reaches every BYO host at its next update | — |
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
@@ -246,7 +246,7 @@
| <a id="g12"></a>[G12](capability-map-detail/G.md#g12) | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | — | the 2026-07-13 hub closing bundle (no evidence document) |
| <a id="g13"></a>[G13](capability-map-detail/G.md#g13) | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | — | hub v0.54 (2026-07-13; no evidence document) |
| <a id="g14"></a>[G14](capability-map-detail/G.md#g14) | The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail | hub v0.135.0 (R-604, R-530) | **IMPLEMENTED** | the page live; the alarm and the mail unit-proven | `audits/hub-safety-2026-10-05/partD/` (2026-10-05) |
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a consistent copy every 6 hours since D34 (nightly before 2026-10-23; corrected 2026-10-24), encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
@@ -105,7 +105,9 @@ Tri-state: `""` (unknown) · `"running"` · `"stopped"`.
### In-flight — a marker file, one per owner
Two exist and they are deliberately **separate files**: `quiesce-state.json` (the whole-guest backup
Two exist and they are deliberately **separate files** *(four in-flight records by 2026-10-24: also `update-journal.json`
— the guarded update's crash journal read by `RecoverUpdates`, `internal/stacks/update.go:49,1306` — and
`.felhom-install-pending` — an interrupted install, R-681, `install_interrupted.go:28`; noted 2026-10-24)*: `quiesce-state.json` (the whole-guest backup
window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app —
`backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One
file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's
@@ -169,7 +171,8 @@ A recovery completing after the grace emits a `LATE RECOVERY` warning naming the
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
answer it, and the last two only became reachable once the window widened past T+5 s:
answer it *(four since slice 4: check „1b", a guarded update moving the app — `IsUpdating`, `cmd/controller/main.go:2459-2477`;
noted 2026-10-24)*, and the last two only became reachable once the window widened past T+5 s:
| Holder | Why starting would be wrong |
|---|---|
@@ -218,6 +221,13 @@ own; every caller that is not the customer must decide for itself whether the ap
---
> *Dated note 2026-10-24 (review round 2): this inventory is v0.33's. On `main` (`321df1f`) 22 packages under
> `controller/internal/` have no row here: `aftercrash`, `agentapi`, `appbackup`, `backupwindow`, `bootrecon`,
> `bootstrap`, `channelhealth`, `crashboot`, `dockerexec`, `family`, `fillwatch`, `i18n`, `infra`, `logx`, `mailrelay`,
> `mediawire`, `nightchain`, `offsiteapply`, `privpolicy`, `quiesce`, `sockheal`, `upwatch` (an `ls` of the tree). Each
> package's own doc comment is its description; `sockheal` (re-heal after the guest's Docker socket is recreated,
> R-860) and `channelhealth` (a periodic proof of the controller→agent channel) are in no architecture document.*
## 2. Classification table (per package/file)
### `cmd/`
@@ -266,6 +276,11 @@ own; every caller that is not the customer must decide for itself whether the ap
|---|---|---|---|
| `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework |
*As built (noted 2026-10-24, review round 2): the `cloudflare/` package is still wired — with `cf_api_token` set the
controller builds a `GeoSyncManager` and calls the Cloudflare API (`cmd/controller/main.go:1992-2000`; the field is still in
`config.go:160`). The DELETE classification above is the plan, not the state; whether any box sets the token is not
readable from source.*
### `config/`, `crypto/`, `util/`
| File | Class | Reason | Risk |
|---|---|---|---|
@@ -325,6 +340,7 @@ household's file names. The household's lost list is shown on the app page (`los
| File | Class | Reason | Risk |
|---|---|---|---|
| `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean |
| *(as built 2026-10-24: both files below are LIVE — the updater decides the version and the floor and hands the container swap to the agent, `AgentSwapper`, `internal/selfupdate/updater.go:38-47`; `state.go` keeps `update-state.json`)* | | | |
| `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean |
| `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean |
@@ -387,7 +403,8 @@ dashboard alert is an `Alert` with two placement fields: `PageOnly` (the pages i
page) and `Inline` (rendered by the page template in place, not by the layout's banner). `GetAlerts` returns
the list (endpoint-drift, agent-channel and dead-app alerts first, then the rest sorted error > warning > info,
capped at five plus an overflow line), and `layout.html` paints only those that are not `Inline` and match the
page; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
page *(as built since R-906 — noted 2026-10-24: pages use `GetBannerAlerts(page, lang)`, which filters per page BEFORE the
cap, `internal/web/alerts.go:353`, `handlers.go:144`)*; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
**Exactly one warning is inline today:** the „storage is not on a separate drive" health warning. It is
`PageOnly: dashboard, monitoring` and `Inline: true`, so it sits quietly under the storage bars; **every other
@@ -821,7 +838,7 @@ R-438. The three facts below are the ones a reader needs before touching any of
`Syncer.copyTemplates` (`sync/sync.go:319`) walks every directory in the catalog cache and copies
`docker-compose.yml` and `.felhom.yml` into the matching stack folder. **There is no test of whether
the app is deployed.** The only guard is a sha256 content compare (`copyIfChanged`) and the only
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351`), plus
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351` then, `:399` now; `copyTemplates` is now `sync.go:436`), plus
one immediate sync at controller start (`sync.go:98`).
**SINCE v0.235.0** this walk still happens and `.felhom.yml` is still copied unconditionally, but the
@@ -835,6 +852,7 @@ and they stay that way until something else acts.
### 2. Every lifecycle path resolves that disagreement, silently, by upgrading
*(This section is the 2026-09 measurement; `UpdateStack` was deleted in v0.237.0 — the button calls `StartGuardedUpdate`, `09` §6.1 — noted 2026-10-24.)*
`StartStack`, `RestartStack` and `UpdateStack` (`stacks/manager.go:1029 / 1133 / 1170`) all end in
`docker compose up -d`. `up -d` makes the container match the file, **and pulls the image itself if it
is absent** — measured at 18.3 s with a pull versus 0.5 s without, against a negative control
@@ -887,7 +905,9 @@ measured; that the restore writes to that path is read, not measured.
beállítások csak olvashatók.", `deploy.html`). This is a design, not an oversight: a changed value would
need a guarded re-deploy that nothing performs. **Finding (2026-09-15):** the catalog field flag
`locked_after_deploy` (`stacks/metadata.go`) is parsed and read by **no** controller code — every field
is read-only after install whatever the catalog says. So a page must never tell a household to change
is read-only after install whatever the catalog says. *(Dated note 2026-10-24: it is now read into the app's
`LockedFields` (`internal/stacks/deploy.go:106,534`) and enforced only by `UpdateStackConfig`, which nothing calls — so
„read-only whatever the catalog says" still holds in practice.)* So a page must never tell a household to change
a value after install. **Vaultwarden (R-512)** follows the design instead of breaking it: registration
is closed by default and the household is invited from the admin panel, measured to work without mail
(stranger 400, invite 200, invited 200).
@@ -308,6 +308,18 @@ A controller can only `POST /rollback` (or snapshot/backup) **its own** guest
token → guest and authorizes per guest, so a compromised controller's blast radius is
**self-scoped and bounded** to its own guest.
**[DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through R-1265]:** the surface listed above is
the slice-8 design; on `main` the local API registers **41** routes (`felhom-agent/internal/localapi/server.go`; 24 of
them are not listed above — the backup tiers and target, disk candidates, decommission, format status, guest attach,
the network storage routes, the controller swap, the escrow stage/preflight/ceremony routes, the debug logs, guest
memory). Among them the controller can start a HOST action: `POST /host/line-test/run` (about 55 s; bounded by the due
date, the window, 6 tries a day and the heavy-operation gate — R-1250, unreleased). The local API also listens on the
R-50 island bridge (host `169.254.253.1`, guest `net1` `169.254.253.2/30`, `internal/config/config.go:260-269`). The
weekly line test keeps its state in the OOB state dir (`line-test.json`) and has an off switch, `line_test.disable`.
The served WireGuard block carries `oob_peer_ip` and `oob_operator_ssh_key` (`internal/hub/report.go`). The D30 re-key
runs only inside the tunnel watchdog, and its 15-minute count lives in memory — an agent restart starts it again (the
once-an-hour bound survives in `wg/last-rekey`).
### 6a. Implementation (slice 8A — implemented)
**Status: implemented** (agent v0.10.0 `internal/localapi`; controller v0.35.0 `internal/bootstrap`
File diff suppressed because one or more lines are too long
@@ -593,7 +593,7 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me
94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`).
While the box owns it, qBittorrent's page says: „A qBittorrent
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
6. **Limits, stated:** an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
6. **Limits, stated:** *(added 2026-10-24, review round 2: ANY viewer explains any amount of upload, so a hacked app can send freely while one real playback runs — `upwatch/watch.go:275,299`; R-1262)*; an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood
whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container
(Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line.
@@ -65,6 +65,13 @@ build is added to both in the same commit.
### 1.1 The button
> *Dated note 2026-10-24 (review round 2): §1 is the 2026-09 MEASUREMENT before the arc. `UpdateStack` no longer exists
> (deleted in v0.237.0, controller `REUSE.md`); the button calls `StartGuardedUpdate` — backup precondition, safety
> dump, pin journal, undo copy, health wait, undo (`internal/api/router.go:794`; `internal/stacks/update.go`). The line
> numbers below are the measurement's; e.g. `copyTemplates` is now `sync.go:436`, `copyIfChanged` `:700`, the `15m`
> default `config.go:399`, `RestartStack` `manager.go:1454` (it now runs `up -d --force-recreate` and errors when no
> container restarted, R-1026).*
`Manager.UpdateStack` (`felhom-controller/controller/internal/stacks/manager.go:1199`) is two compose
commands and nothing else:
@@ -267,7 +274,8 @@ builds them (`audits/update-rulings-2026-09-23/`).
next night. **Why:** the update then leans on the freshest copy the box ever has, and there is no
second window for anyone to set or misread. **Replaces:** §3b Q1's recommendation of a fixed
02:30–05:00. *For the builder:* `stacks.update_window` (`config.go` L195, default `03:00-05:00`)
exists and **no Go code reads it** — it appears only in `config.go` (field + default),
exists and **no Go code reads it** *(dated note 2026-10-24: the field is now removed — no controller source names
it; `controller/README.md` „`stacks.update_window` is removed")* — it appears only in `config.go` (field + default),
`setup/handlers.go` L482 (written into a new box's `controller.yaml`) and
`configs/controller.yaml.example`. Remove it or fold it into the chain; do not build a second
window.
@@ -424,7 +432,8 @@ R-636's louder repeated alarm.
(a) broke the second. Shipped as controller **v0.269.1** — a second release in the session, instead of
shipping (a) to the fleet with the floor.
*Note 2026-09-30 (R-740) — the cost line above is not what the code does.* „(or the automatic leg)": the
automatic leg never presses a digest-only change (`unattended.go:433–435`, skip `no_test_record`). And the
automatic leg never presses a digest-only change (`unattended.go:433–435` then, `:449-451` now — line corrected
2026-10-24 — skip `no_test_record`). And the
catalog never writes a re-test of a floating tag at a new digest (the writer refuses a step that moves no
image), so a same-name upstream fix reaches no box by either route today. Measured, with the options, in
R-740; the decision is the operator's (STATUS 2026-09-30). This decision's ruling — the sync never moves
@@ -1981,7 +1990,7 @@ moves a stack dir.
| writer | pin source |
|---|---|
| the deploy path (`runComposeDeploy`) | the template just deployed from |
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull |
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull — *as built: `UpdateStack` was deleted in v0.237.0; the guarded update writes the pin (`advancePinTo`, the `pinning` phase before `pulling`), and two more paths move a pin BACK — `pinBack` and the undo's `restoreDefinition` (`internal/stacks/update.go`, `undo.go`; noted 2026-10-24)* |
| the restore (`stackAdapter.RecreateStackDefinitionFromUnit`) | the recovery unit's captured compose — **this closes R-441** |
| `Manager.AdoptPins` | the observation, once, and only when complete AND matching |
@@ -2047,7 +2056,7 @@ earlier feature is the failure mode to look for whenever a file changes meaning.
| **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 |
| **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | **SHIPPED + PROVEN LIVE, controller v0.237.0 (job) + v0.238.0 (page) + v0.238.1 (2026-09-13); any backup tier since v0.239.0 (§3 decision 8)** — §6.1 |
| **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 |
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | OPEN — R-450; rulings 2026-09-23; build order §6.4 |
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | ~~OPEN — R-450~~ **BUILT** — the leg in `internal/stacks/unattended.go`, wired into the off-site job; R-450 CLOSED 2026-10-05 (capability map B11) *(noted 2026-10-24, review round 2)* |
| **7** | **A fleet sweep pipeline** — the operator can see, and move, how far behind every box is. | OPEN — R-451; ruled 2026-09-23 (decision 18), built later |
### 6.1 Slice 4 as SHIPPED (controller v0.237.0 + v0.238.0, 2026-09-13)
@@ -2062,12 +2071,13 @@ closed by construction: nothing reports an update complete on the compose exit c
| # | phase | what happens | on failure |
|---|---|---|---|
| 0 | refusals (409, before the intent is recorded) | held (R-439), busy (backup/restore/app-data op/quiesce), migration, already updating, deploying, memory (the deploy's `memoryVerdict`, releasing the app's own request), disk (**fixed 2 GB floor** — image size unknown without a registry), **no copy on ANY tier and no backup can be taken now** (since v0.239.0; before it, no restorable Tier-2 copy) | nothing moves, nothing is recorded |
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) | nothing moves |
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) — *an app whose data lives in bind-mounted files walks second drive → OFF-SITE → own unit (decision 9, R-479, v0.241.0; `internal/backup/update_guard.go` `updateTierOrderBindData`; noted 2026-10-24)* | nothing moves |
| 2 | `backing-up` — only when no tier holds a fresh copy | `RunAppBackupNow`: this app's DB dump → volume dump → unit capture (marked proven current) → Tier-2 copy, whose failure is a WARN since v0.239.0 | refused with the backup's own error; nothing moves |
| 3 | `safety-dump` | `WriteUpdateSafetyDump` (R-361's undo copy) — **before the pin moves** | refused; nothing moves |
| 4 | `pinning` | the previous definition is copied aside and journaled, then the pin advances | pin put back |
| 5 | `pulling` | `compose pull` | **pin and definition PUT BACK** — nothing ran (Scenario E) |
| 5a | `copying` **(v0.263.0)** | `compose stop`, then each NAMED volume `cp -a` into `<volume>.pre-update-<stamp>` by a helper that writes a finished-marker last (bind folders never) | copies removed, **pin put back, the old version started again** — nothing new ran |
| 5b | `converting` **(v0.273.0, row added 2026-10-24)** | only on a step whose ladder entry carries `engine_conversion`: the box converts the PostgreSQL major (dump from the old engine, load into the new, check) — `internal/stacks/pgconvert.go` | the undo puts the old engine's data back |
| 6 | `starting` | `compose up -d --remove-orphans` | **UNDO** (below), HOLD only if the undo fails |
| 7 | `verifying` | the `.felhom.yml` health check through the existing probe **when it resolves to a container**, else 60 s of every container running and none restarting; bounded by `update.health_timeout` | **UNDO** (below), HOLD only if the undo fails. Before v0.263.0: stop + HOLD, the pin stays (Scenario F) |
| 7a | `undoing` **(v0.263.0)** | every copy validated (marker) BEFORE anything is poured back; volumes emptied and refilled; definition, pin and the pinned version's `.felhom.yml` record put back from the job's own copies; `up`; health with the OLD version's probe | HOLD, the sentence prefixed *„A frissítés nem sikerült, és az automatikus visszaállítás sem."* + the data state (`untouched` / `half` / `not_started`); copies kept |
@@ -2116,7 +2126,7 @@ the harness has proven it, is slice 6's.
> copy taken before the pin moved**, so the old version meets pre-migration data. Spiked by hand on
> 9202 the same day — §6.1a.
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6.
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6. *(Noted 2026-10-24: stepping is built — the ladder, one tested step per press, decision 14.)*
### 6.1a The undo (decision 15) — SPIKED BY HAND, then BUILT: controller v0.263.2 (2026-09-23)
@@ -2130,7 +2140,8 @@ only the live box could show, fixed the same day:** the undo's probe was never a
current probe held the app `unhealthy` (v0.263.1), and the "old" `.felhom.yml` taken at update time
was already the new one, because `.felhom.yml` flows in on every catalog sync (v0.263.2: the pinned
version's file is now recorded in `applied-meta/` whenever a version is pinned). **Residual (R-646):**
an app pinned before v0.263.2 has no such record until its next pin. **Two more, found by the chaos hour of
an app pinned before v0.263.2 has no such record until its next pin. *(Closed since: `BackfillAppliedMeta` records it at
start for every app CURRENT with the catalog, and names the rest — `internal/stacks/undo.go:646-652`; noted 2026-10-24.)* **Two more, found by the chaos hour of
2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** the undo finds an app's volumes by their compose
label, and a restore recreates them WITHOUT it — so after any restore the undo copies nothing (R-658, P1); and a
held file-leg app is pointed at a restore that refuses a database-only copy, with no route left on a box without
@@ -2270,7 +2281,7 @@ breaks.
---
### 6.2 Slice 6, as it will be built (OPEN — R-450; ruled 2026-09-23, decisions 11–17)
### 6.2 Slice 6, as it will be built (~~OPEN — R-450~~ BUILT, R-450 closed 2026-10-05 — noted 2026-10-24; ruled 2026-09-23, decisions 11–17)
**The shape the rulings fix. The build order and costs are §6.4.** Rewritten 2026-09-23; the
earlier "as it would be built" draft keyed on the window of Q1's recommendation and on
@@ -2341,7 +2352,7 @@ retried. A working caller in this exact shape exists as evidence, not product:
**The per-box switch key is `app_update.unattended`**, default **true** (decision 12).
⚠ **NOT `auto_update` — that name is TAKEN, and by the very thing this must not collide with.**
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281, default
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281 then, L278-279 now; default
**04:30** at L422) are the CONTROLLER's own update. Two settings with that name, one meaning the
controller and one meaning apps, is the kind of collision that is only discovered by an operator who
turned off the wrong one. The app-scoped key is `app_update.*`. **And `stacks.update_window` is
@@ -2371,12 +2382,12 @@ what the part can do to a household's data if it is wrong, not how likely that i
| # | part | rulings / rows | cost | depends on | risk to customer data |
|---|---|---|---|---|---|
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold`: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold` *(as SHIPPED — noted 2026-10-24: the copy is the FOLDER copy decision 19 chose: every named volume `cp -a` into a sibling volume with a finished-marker, put back on failure; no database load — `internal/stacks/undo.go:26-36`; the text that follows is the design before the bake-off)*: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
| **2** | **SHIPPED — controller v0.264.0 + hub v0.120.0, proven live on 9202 and 9201 2026-09-23** (`audits/undo-fleet-2026-09-23/`). **The update sentences in the household's language** (R-606) and a mail when an automatic update is undone or held: events `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, on by default, mailed in the household's language with the app named in the subject; per-app cooldown on both legs. Leftovers: R-647. | R-606, 15 | **1** | — | none |
| **3** | **SHIPPED — controller v0.264.0, proven live on 9202 2026-09-23.** **A disabled notifier says so** (R-620), so the mail of part 2 can be measured on a scratch box at all. | R-620 | **0.5** | — | none |
| **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/<to>.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only |
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. **The box half (compare, render `name:tag@sha256`) is not built.** **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). *(Closed since: a step's own `steps/<key>.felhom.yml` is read when present — `ladder.go:127-132`, v0.269.0; noted 2026-10-24.)* **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. ~~**The box half (compare, render `name:tag@sha256`) is not built.**~~ *Built — the box renders the tested digest, carries it on sync, and the badge reads the tested digest (`internal/stacks/digest.go:15-24`, `updateorder.go`; noted 2026-10-24).* **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
| **7** | **SHIPPED — controller v0.271.0 (`9cf13a3`), proven on 9202 over four simulated nights 2026-09-24/25 and watched through the demo boxes' first real automatic night (`audits/DRILL-night-2026-09-25.md`).** Decisions 31–33 taken unattended. Gaps a scratch box cannot close: R-687; no resume after a restart: R-686. *(Before:)* **SPIKED 2026-09-24 (night, Part C) — the measured build brief is §6.4.2 below.** **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has ENDED (on every exit path, skips included), one app at a time, one step per press, `app_update.unattended` default ON (decision 12), `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it, and the full-system backup's gate waits for it until W+5h (decision 20). | 11, 12, 14, 15, 20 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe |
| **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none |
| **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none |
@@ -2459,7 +2470,7 @@ During the db-dump leg every press was refused `busy` (transient, retried next p
update's own finish should refresh them.
- **An Update pressed on an app already at the head runs the whole guarded update** — backup, pull,
restart, `done` — with no refusal (R-679: navidrome restarted four times for nothing). The leg never
presses an app whose pin equals the head; the preflight should refuse with reason `current`.
presses an app whose pin equals the head; the preflight should refuse with reason `current` (as built the reason key is `already_current`, `update.go:432` — noted 2026-10-24).
- **The product does not remember a failed step.** After vikunja's undo the badge still offers the
same step; only the caller's memory kept it from a re-press. Decision 15 needs it on the box: the
failed `to` is recorded per app and the leg skips it until the catalog's ladder changes.
@@ -2541,6 +2552,24 @@ and E are the ones that unblock a decision; leg A is the one that takes the time
harness on DooPlex for the image-side ones.
### 6.4.x Decisions in code no section records [DESIGN — undocumented, found 2026-10-24, review round 2 — the operator confirms them through one register row]
Read at `file:line` on controller `main` (`321df1f`) by the main session:
- **`build.sh --push` also refuses a DIRTY tree, a HEAD that is not `origin/main`, and a directory that is not a git
checkout** — `controller/build.sh:144-166`. §0 names only the tag-exists and unreachable-registry refusals.
- **The leg retries a passing refusal ONCE in the same night** (`busy`, `updating`, `deploying`, `migrating`,
`self_updating`), after `RetryWait` — not „the next pass/night" (`internal/stacks/unattended.go:65-66`, the pass loop).
- **The leg skips an app the household stopped** (`stopped_by_household`, R-721): an update would start it
(`unattended.go:416-418`).
- **A manual run of the chain** (R-705, the debug action) gives the leg a deadline of its start + the leg's normal
length (`backupwindow.UpdateLegLengthMin`), not W+5h (`unattended.go:232-238`).
- **Two preflight refusals are plain skips to the leg:** `already_current` and `engine_no_test`
(`internal/stacks/update.go:432,454`).
- **The render table (§5.4) has one more row:** when the catalog template's images cannot be read, the syncer leaves
the app's compose file alone for that cycle (`internal/sync/sync.go:622-627`); and the box's R-971 allow-list runs
before ANY compose write (`cmd/controller/main.go:421`, `SetAdmitFn`).
## 6.5 The drill catalog and the image store — the standing method for update drills
**Why this section exists.** On 2026-09-21 an afternoon session put a deliberately broken image into
@@ -2629,7 +2658,9 @@ customer box has credentials for it. The store listens on the guest's loopback o
### 7.1 The record (slice 1)
`Manager.recordInstalledImages` (`felhom-controller/controller/internal/stacks/installed.go`) runs
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`,
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`
*(as built — noted 2026-10-24: `UpdateStack` is gone; the callers are `StartStack`, `RestartStack`, the deploy path, the
end of the guarded update and the end of the undo — `manager.go`, `deploy.go`, `update.go`, `undo.go`)*,
and writes `app.yaml`:
```yaml
@@ -2705,7 +2736,7 @@ Version strings stay in the logs, the API and the hub.
The 8th, immich's own ghcr build, is UNMEASURED — ghcr exposes no anonymous last-modified. So on
demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.
Digest-level comparison needs the catalog to record the digest at push time — **R-446**, put to the
operator as §3b **Q6**, recommended YES.
operator as §3b **Q6**, recommended YES. *(Built since for every ref with a ladder entry — `updateorder.go`, `digest.go`; the limitation remains only for a ref no entry records; noted 2026-10-24.)*
2. **~~Nothing enforces `catalog_since`.~~ Enforced by the pre-push hook since 2026-09-13 (R-452, `app-catalog-felhom.eu/scripts/check-catalog-since.py`); CI's shallow clone still skips it out loud.** A commit that moves an `image:` line and forgets the date
under-reports how far behind a box is. The gates runner fetches at `--depth 1` and has no parent
commit to diff against, so the gate needs a deeper fetch — **R-452**.
@@ -2730,13 +2761,18 @@ Version strings stay in the logs, the API and the hub.
refuses without a restorable, proven Tier-2 copy, backs up first when that copy is stale, takes a
safety dump, and holds an app that does not come up. **What stays true:** it still has no automatic
rollback (deliberately, §6.1) and can still attempt a multi-major jump the app will refuse (R-40) —
that now ends HELD rather than crash-looping behind a green button.
that now ends HELD rather than crash-looping behind a green button. *(Dated note 2026-10-24: a copy on ANY tier now
satisfies the precondition (v0.239.0), the box undoes a failed update itself (decision 15, v0.263.2), and the
ladder climbs one tested step per press (decision 14) — `internal/backup/update_guard.go`, `internal/stacks/undo.go`,
`ladder.go`.)*
7. ~~**An engine major can be applied without its datadir upgrade, and nothing notices.**~~ **CLOSED
2026-09-13 for MariaDB (R-459):** every `mariadb:` sidecar carries `MARIADB_AUTO_UPGRADE=1`, and the
harness shows the conversion running on the E3/E3b edges (§3 decision 5). **What stays true:** the
PostgreSQL half (R-463) has no equivalent — the image performs no `pg_upgrade` — and the
engine-major rule (§3 precaution 3, R-469) is what keeps both engines inside their major until
Slice 4 gives the Update button a backup.
Slice 4 gives the Update button a backup. *(Dated note 2026-10-24: MariaDB is lifted — its major is its own edge; a
PostgreSQL major may pass with a two-venue proof and an `engine_conversion` mark, and the box converts it
(v0.273.0) — `app-catalog-felhom.eu/scripts/check-engine-major.py`.)*
8. ~~**Only three of 53 apps have ever had an upgrade measured.**~~ **WIDENED 2026-09-21 to 21
EDGES ACROSS 19 APPS** (`audits/DRILL-update-night-2026-09-21.md`), on scratch guest 9202
through the product's own guarded Update, against a PRIVATE DRILL CATALOG so the live catalog
+11 -7
View File
@@ -128,7 +128,9 @@ the hub's e-mails follow.**
`expired` pinned to Hungarian so the language cannot become the oracle the text refuses to be.
Full design: `05-hub-architecture.md` §15. **R-555 closed** — the `language` allowlist entry is out
of `wire_contract_gate.py` and the gate now checks the field for real.
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece:** the hub stores a per-customer language, renders it into
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece** *(Part B is built
too — the controller reads `customer.language` as its start language since v0.256.0, `internal/config/config.go:149-154`;
corrected 2026-10-24)*: the hub stores a per-customer language, renders it into
`controller.yaml` next to `customer.id/name/domain/email` (`hub/internal/configgen/configgen.go`),
and the box uses it **only while the household has never chosen** (`settings.json` empty). The
household's own choice always wins; the hub's e-mails follow the language the box **reports**, which
@@ -184,7 +186,8 @@ bundles. They now judge the page **as rendered in each language** (`controller/s
- **hu** — the product speaks in „te". The converted copy carries formal („ön") forms (R-516). They are
not fixed by a localisation release (§1), so the gate counts them against `HU_FORMAL_CEILING` as a
ratchet: a new one convicts, fixing one lowers the ceiling. The ceiling followed the conversion:
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C). **[FACT] Measured limit:** the stem
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C) → … → **0** (R-516 item 12, the last eleven in the
te-form — `felhom-controller/controller/scripts/i18n_missing_gate.py:41`; corrected 2026-10-24). **[FACT] Measured limit:** the stem
list is short — on the release C pages it counts 4 formal forms, while a wider list („írja be",
„adja meg", „válassza", „biztosan eltávolítja", „engedélyezze", …) finds at least 22 keys (R-516).
- **en** — second person, plain: no „please", no „kindly".
@@ -201,7 +204,8 @@ vissza") — R-564. The hub copy gate and the catalog have no language concept
- **[DESIGN] The hub's operator pages** — the operator reads them; they stay English (decision 1).
- **[DESIGN] The first-boot wizard** (`controller/internal/setup/`, 8 pages, 95 strings) — out of scope
and obsolete (decision 4, 2026-09-17; `02-controller-module-map.md` L56). A household still reaches
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**.
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**. *(Deleted since:
`internal/setup/` is gone (v0.308.0); `bootstrap.WaitForConfig` shows a waiting page instead — corrected 2026-10-24.)*
- **[FACT]** App containers' own UIs (Uptime Kuma, PrivateBin…) — not Felhom's copy (R-516 items 5-6).
---
@@ -226,7 +230,7 @@ i18n:
```
Why: one app's copy stays in one file a reviewer sees whole; the controller's `yaml.Unmarshal` ignores
unknown keys (`internal/stacks/metadata.go` L316 — **verified: this repo constructs no `yaml.Decoder`
unknown keys (`internal/stacks/metadata.go` L316 then, L475 now — **verified: this repo constructs no `yaml.Decoder`
at all, so there is no `KnownFields` anywhere to turn strict**), so older controllers are unaffected;
a missing English field falls back field by field.
@@ -234,7 +238,7 @@ a missing English field falls back field by field.
reached only through `LocalizeStacks` / `LocalizeStackPtr` / `Stack.MetaFor`.
- **`For("hu")` is the parsed struct with `I18n` cleared and nothing else**, deep-compared against all
53 real catalog files (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
53 real catalog files (the identity fixtures are still those 53; the catalog has 65 apps — noted 2026-10-24) (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
§1 therefore holds by construction for catalog copy, and `LocalizeStacks` returns the INPUT SLICE
for Hungarian rather than a copy, so nothing on the Hungarian path is even touched.
- **Fallback is field by field, and blank counts as absent** (ruling 3). A half-translated app is a
@@ -307,10 +311,10 @@ for the English page test (R-565), arriving again in a different repo. So
`app-catalog-felhom.eu/scripts/check-copy-i18n.py` folds and matches word-bounded stems, with a
positive and two negative controls printed on every run.
**[FACT] The gate**, fifth row of `catalog_gates.py`, static, in the pre-push hook:
**[FACT] The gate**, fifth row of `catalog_gates.py` (the sixth since — corrected 2026-10-24), static, in the pre-push hook:
1. **FREEZE** — every Hungarian copy string equals `scripts/copy_freeze/hu.json`, captured in its own
commit before any translation. Runs on all 53 apps whatever scope is named. A new app must be
commit before any translation. Runs on every app in the freeze file (65 on 2026-10-24) whatever scope is named. A new app must be
admitted with `--add-app NAME --reason "…"`.
2. **STRUCTURE** — the `en` block may carry copy fields and nothing else; every key-matched entry must
have a Hungarian twin, or it is INERT on the box and the translator never learns.
+14 -8
View File
@@ -11,7 +11,7 @@
> | | |
> |---|---|
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1).** Ratification is Viktor's review, not an editor's. |
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1); steps 2–6 are BUILT by §8's own list (corrected 2026-10-24 — the ratification is still the operator's).** Ratification is Viktor's review, not an editor's. |
> | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. |
> | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 |
> | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. |
@@ -79,7 +79,7 @@ boxes installed. Nobody keeps a hand-written list.
- **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).
- **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in
the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since
2026-06-11) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
2026-06-11 — *`2026.9.3` today, corrected 2026-10-24*) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
— the app-image question (`09`), not an OS package. The gap is R-838.
---
@@ -101,9 +101,10 @@ gets the Docker engine that was current when its golden was baked, and keeps it.
~~**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist
(`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.~~
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:58`
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:107`) and `:194`
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:628`). Nothing else.
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:68`
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:133`) and `:228`
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:659`). Nothing else. *(Line numbers
corrected 2026-10-24.)*
`apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4
proposes how to avoid that.
@@ -250,7 +251,7 @@ approval of that day stays in force (decided by CC unattended — operator may r
versions. A broken-into hub can choose an older version or no version. It cannot make a box install a
package that the publisher did not sign.
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]`
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]` — *BUILT and installed since (§8.1); the agent's sudoers holds only the `--plan` line, no `--repair-only` (`felhom-agent/configs/felhom-agent.sudoers:337`); noted 2026-10-24*
`felhom-os-apply` — root-owned (`0755 root:root`), installed by the installer beside `felhom-selfupdate-guarded`;
the agent's sudoers gets exactly `/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json` and
@@ -315,7 +316,7 @@ PID 1 or `lxc-start`, or a kernel was installed.
Decision 96. Before it, the signed `agent_update` replaced only the binary; wrappers, units and sudoers lines reached
an installed box by reinstall or by hand.
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22: sudoers ×2, the
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22 then, **32** counted 2026-10-24 — plus the vzdump-partial-clean, thin-check, internal-place and priv-apply wrappers, the guest hook, the shared-parent unit, the GRUB one-shot pair and the sshd root-key-only drop-in; sudoers ×2, the
five wrappers, the crash guard and its units + config, the agent and rollback units, the start-limit drop-in, the mgmt
watchdog and its tmpfiles/units, the OOB belt's four files). `scripts/build-config-bundle.py` builds the bundle from it
reproducibly; `release-agent.sh` publishes it beside the binary; the hub vouches its sha with the agent (exact-name
@@ -393,6 +394,9 @@ must never overlap a backup, a restore-test or a self-update.~~
- **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release,
how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest
approved release raises an alarm (`08`).
- **[FACT, added 2026-10-24 — hub v0.169.0] The System page's „Upload line" cell** shows the best of the last SUCCESSFUL
weekly line test (R-1247), amber when it is older than 15 days or was never measured after an attempt — judged on
`last_ok_at`, never on the attempt time (`hub/internal/web/system.go` `lineStaleAfter`, `lineTestCell`).
- **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with
buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held /
reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed,
@@ -480,7 +484,9 @@ Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-do
**What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve,
libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on
2026-10-07. **What it never touches:** a kernel, boot, firmware or microcode name (`HOST_SLOW_RE`: `proxmox-kernel*`,
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`)
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`,
and — read 2026-10-24 in `felhom-os-apply` — `linux-(image|headers|kbuild|modules|base)*`, `firmware-*`,
`proxmox-secure-boot-support`)
— that is R-836's lane (decision 164). No reboot.
- **Wrapper** (`felhom-agent/configs/felhom-os-apply`): layer `pve`, lane `slow` only. Origin `Proxmox Debian
@@ -78,6 +78,13 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971). **Re-check (2026-10-24, review round 2):** the box checks every rendered compose against its allow-list before any write since controller v0.328.0 (`internal/privpolicy/privpolicy.go`) |
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (deployed 2026-10-24 — corrected 2026-10-24; `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, released 2026-10-24 in controller v0.342.0 + hub v0.169.0 — corrected 2026-10-24) detects |
**[DESIGN — undocumented in `01` §5, found 2026-10-24, review round 2 — confirmed through R-1265]:** the file manager's one
sign-in (R-1100) — FileBrowser trusts its proxy header blindly, and every app container can reach `filebrowser:80`
directly; the only guard is that the header NAME is a per-box secret, which traefik's forwardAuth strips from every
client request and sets only for a browser holding the dashboard's files pass
(`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`, `internal/web/files_sso.go`). The name file's
creation race was fixed by review round 2 (controller `b01bb4b`).
## 6. What we accept (do not re-report)
- **Cloudflare can read tunnel traffic** — R-904, `09` §3 decision 184.
@@ -22,7 +22,7 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
| 2 | controller `POST /api/server/poweroff` (`internal/web/poweroff_handlers.go`) | dashboard login (`RequireAuth` + CSRF), POST only, the word again, the controller's own backup/restore/app-update check, the `host_poweroff` feature gate (MinAgent 0.155.0) |
| 3 | agent `POST /host/poweroff` (`internal/localapi/hostpoweroff.go`) | `withGuest` token scope; the guards (§3); the record (`internal/poweroff`) |
| 4 | agent → hub | a host report carrying `deliberate_off.requested_at`, waited for until the hub ACCEPTED it (≤ 60 s, `Loop.SetReportedHook`) |
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors, `FELHOM_POWEROFF`) |
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors: `pct ^shutdown [0-9]+ --timeout [0-9]+$` in `FELHOM_GUESTHOOK`, and `systemctl poweroff` in `FELHOM_POWEROFF` — `felhom-agent/configs/felhom-agent.sudoers:81-90`; corrected 2026-10-24) |
| 6 | hub (`store/deliberateoff.go`) | `hosts.deliberate_off_at`; silence alarms held 14 days (§4) |
| 7 | the household | presses the machine's power button; the first report of the new boot has no `deliberate_off` → the hub clears it |
@@ -33,11 +33,11 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
| first 10 min after boot | `too_soon_after_boot` | `TooSoonAfterBoot` |
| one accepted request per boot (file keyed by `/proc/stat` btime) | `already_requested_this_boot` | `SecondRequestSameBootRefused`, `poweroff/state_test.go` |
| any other guest or VM running (`/proc`: `lxc-start -n N`, `kvm -id N`) | `other_guests_running` | `RefusedWhileOtherGuestsRun`, `RunningGuestsFromProc` |
| heavy-operation gate held (backup, restore test, OS step, sweep, trim) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
| heavy-operation gate held (backup, restore test, OS step, sweep, trim, and since agent v0.164.0 the weekly upload line test — corrected 2026-10-24) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
| backup job, controller swap, drive format, NAS verify, escrow ceremony, Proxmox vzdump | `backup_running` / `controller_update_running` / `drive_operation_running` / `escrow_ceremony_running` | `RefusedDuringVzdump`, `FailsClosedWithoutTaskReader` |
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
**„A Proxmox vzdump runs” (agent v0.156.1, R-1008):** a vzdump task for the guest in Proxmox's active list whose worker process still lives — Proxmox's own rule (the UPID's pid + pstart against `/proc/<pid>/stat`). After a power cut the dead task stays listed for ~30–60 s; it no longer counts. Doubt (an unparsable task, an unreadable `/proc`) still refuses. The same answer serves the start-up stale-lock pass, the guest-power watchdog and the controller supervisor (`stalelock_inflight_test.go`).
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
**[DESIGN]** A cancelled power-off leaves the box on (gate released) — better a box that did not switch off than a
backup, update or format cut in the middle. **[DESIGN]** A failed guest shutdown does not stop the power-off: Proxmox's
@@ -86,5 +86,15 @@ alternative (an out-of-band confirmation) needs a second channel the box does no
## 6. Not done here
- The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).
- ~~The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).~~ Published — `website/gyik.html:128,603` (corrected 2026-10-24).
- The BIOS „after power loss" value of the N100 was not read (no monitor); the household text promises only the button.
## Decisions in code no section records [DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through one register row]
- **A re-enroll while the household power-off hold is active counts as „gone"** (a reinstall over the old host), not
„live" (R-1154; `hub/internal/api/reenroll.go:67-68`).
- **Setting and clearing the deliberate-off time each record an info event** — `host_deliberate_off`,
`host_deliberate_off_ended` — on the customer's event list; neither is mailed (`hub/internal/api/deliberate_off.go:12-13`).
- **The confirmation word is accepted in three forms:** „KIKAPCSOLÁS", the unaccented „KIKAPCSOLAS" (a foreign keyboard)
and „TURN OFF" (`felhom-controller/controller/internal/web/poweroff_handlers.go:28-29`).
+15 -1
View File
@@ -165,7 +165,8 @@ directory) with that tag. Facts the link code needs:
**[FACT]** An unlisted writable `${…_PATH}` bind is MANDATORY (every tier, off-site included); a plain host path is in
no capture set (`stacks/classify_binds.go` `classifyRoot`). Measured over the live templates: Transmission's unlisted
`watch` bind goes off-site; the download folders are `optional`, so a second drive gets a nightly copy of every
download (R-1019). **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
download (R-1019). *(Fixed since: catalog `c50adfc` marks the download and watch folders `excluded` —
`app-catalog-felhom.eu/templates/transmission/.felhom.yml:46-51`; noted 2026-10-24.)* **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
media is no single app's data; whether the library gets its own household-level backup is a separate decision.
## 8. Removal and export assume one drive per app
@@ -192,6 +193,13 @@ household's existing files under `media/torrents` stay where they are.
Evidence: `audits/media-drive-events-2026-10-14/` (K1–K7). „Idempotent" = a second run changes nothing (K2: run 2 left every entry, folder and library as it
was). „Household" = a step only the household can take.
> *Dated note 2026-10-24 (review round 2): this table is the 2026-10-14 MEASUREMENT („today" = then). As built (§11):
> on a drive event the controller only REWRITES the binds; the media apps restart when the household presses
> „Újraindítás most" or „Később" (03:30) — the file browser alone recreates itself (`internal/stacks/media_library.go`,
> `internal/web/media_events.go`); a safe removal keeps the drive REGISTERED and its return re-binds with no household
> step (`SetEjected`, `media_events.go`); and since controller `b01bb4b` (unreleased) neither restart recreates an app
> the household stopped (R-721).*
| event → app | action | measured call | idempotent | household | evidence |
|---|---|---|---|---|---|
| **added** — every media app | the compose gains the drive's bind → `up -d` recreates it (the compose CHANGED, so today's restart works) | controller compose `up -d` | yes | no | K5-1: 1–10 s per app |
@@ -258,3 +266,9 @@ save path when its client is saved before the wiring; **a Plex status probe (`/l
unclaimed Plex because Plex answers 401 for a moment while it starts — never gate on a status a starting app also
returns** (G7f).
**[DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through one register row]:** the „Később"
03:30 media restart is an IN-MEMORY timer — a controller restart forgets it and only the notice stays
(`internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`); the Sonarr/Radarr API key is READ from `/config/config.xml`, the „set at
install" option §6 calls cleaner was not built (`internal/mediawire/mediawire.go:336-345`).
@@ -15,8 +15,8 @@
| Place | Found by | Identified by | Mounted | What lives there | The backup page says |
|---|---|---|---|---|---|
| Host root / PVE `local` | PVE storage `local` [FACT] A/internal/storage/observe.go:424; never a candidate [FACT] A/internal/storage/role.go:53-79 | `path:` / `store:local` [FACT] A/internal/storage/durableid.go:54-67; role system | host `/`, `/var/lib/vz` | the **local whole-guest copies** (vzdump, retention 3) [FACT] E/scripts/felhom-host-install.sh:730-734; `07` §6.1 | `backup.target.degraded` „…ugyanazon a lemezen van, mint a rendszer…" [FACT] C/internal/web/backup_target_offer.go:78-100 |
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — a Go constant, not i18n [FACT] backup.go:359-361, :1387-1410 |
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: **whole disks only** [FACT] A/internal/storage/candidates.go:33-53 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — *corrected 2026-10-24:* the i18n keys `note.backup.single_copy` / `note.backup.single_copy_unproven`, picked by the agent's verdict [FACT] C/internal/backup/backup.go:1485-1494 |
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: whole disks, each carrying its nested partition candidates (empty / filesystem / in use; corrected 2026-10-24) [FACT] A/internal/storage/candidates.go:84-90 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
| Network share | customer input [FACT] A/internal/localapi/netstorage.go:36-40 | none („no durable-id, no SMART") [FACT] netstorage.go:15-19 | `.automount` under `/mnt/felhom-drives/<name>` [FACT] A/internal/storage/netmount.go:18-31 | browse only; never media, never Tier-2 [FACT] media_library.go:83; tier2.go:141-143 | `err.backup.halozati_tarolo_nem_lehet_a_2` [FACT] tier2.go:37-40 |
| USB disk | same as an enrolled drive; transport never decides [FACT] C/internal/settings/settings.go:609-614 | same; the bridge may pass the real identity (N100: yes) [FACT] spike case 4a | same | same | same; SMART may be missing → „Nincs adat" |
@@ -126,7 +126,7 @@ binds both. **Creating or deleting partitions: never from the dashboard in this
## 7. The installer
**[FACT]** Today `step_grows` gives the guest's one data volume (`mp0`) 70 / 130 / 250 GiB by the thin pool's free space
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh:2261-2275). Measured pools: 128 GB
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh `step_grows`, now at :2475 — line corrected 2026-10-24). Measured pools: 128 GB
disk → 54 GiB, 200 GB → 113 GiB, 512 GB → 349 GiB (PVE default layout; root capped at 96 GiB).
**[DESIGN] The rule:** `mp0` = 70 GiB (pool < 300), 130 (300–800), 250 (≥ 800). Reserve = **max(16 GiB, 10 % of the
@@ -231,10 +231,25 @@ sizes ruled above in §7):
the public installer 1.33.0, agent 0.158.1 and golden 0.318.0 vouched) → data volume 100 GiB, internal place 213 GiB,
registered as the default drive with no press; Jellyfin installed with no extra step onto it; a whole-box copy holds nothing
of it; partitions formatted and used through the wizard; the honest backup offer on a VM. **Not as written:** an empty part
asks for the typed confirmation (the probe reads the parent's partition table — R-1093); a 200 GB disk gets no internal
asks for the typed confirmation (the probe reads the parent's partition table — R-1093) *(fixed since in agent v0.160.0:
the parent's PTTYPE is set aside when `blkid -p` proves the node is a partition of that scheme — `git tag --contains
f7b7308`; noted 2026-10-24)*; a 200 GB disk gets no internal
place under the ruled sizes (15 GiB < 20).
Also in build B: R-1084 (the whole-box backup offer asks the same-disk question) and R-1082 (`/mnt/sys_drive` is never
offered to attach, by device). A bundle that adds a path needs a step bundle on boxes with an older wrapper (R16, R-880):
0.158.1 was delivered through `0.158.1-step1`.
## Decisions in code no section records [DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through one register row]
Read at `file:line` on `main` by the main session:
- **`systemd-detect-virt --vm` is read once per agent process**; empty output becomes `unknown`, which is not `none`,
so every disk then reads unknown/vm — the fail-closed direction (`felhom-agent/internal/storage/identity.go:195-205`).
- **A path the controller cannot read counts as the SAME disk** (R-668, v0.269.0): no second copy is ever claimed on
it (`felhom-controller/controller/internal/backup/backup.go:1436-1450`).
- **The internal place („Belső tárhely") refuses eject, decommission and wipe** — the controller (`refuseInternalLifecycle`)
and the agent both — and allows moving its data to another drive (`internal/web/internal_place.go:20-26`).
- **The internal place's size subtracts EVERY thin volume already promised in the pool**, not only the 32 GiB rootfs and
the data volume (`felhom-agent/configs/felhom-internal-place:11-14`).
@@ -5,6 +5,21 @@
> No matrix is designed here, no target architecture is proposed, and no gap is ranked.
>
> **This is the input to a redesign, not the redesign.**
>
> **WHAT HAS MOVED SINCE (banner added 2026-10-24, review round 2 — this inventory records 2026-07-28 and is kept as
> it was; each line below was read at `file:line` on `main` that day):**
> - The guest has ONE data volume, `mp0` at `/var/lib/felhom`, whose `docker/` and `sys_drive/` folders are bound in the
> guest; there is no `mp1` (`felhom-agent/configs/build-golden.sh:31-38,99`) — the „rootfs + mp0 + mp1" lines are history.
> - A schema-2 recovery unit (D5) carries the PORTABLE secrets; a restore takes the database password from the unit
> (R-1106) — `felhom-controller/controller/internal/backup/recovery_unit.go:24-27`, `restore_dbcreds.go`.
> - The second-drive copy's `recovery-unit/` IS read: `RestoreTier2Unit` restores the full unit from the mirror (R-102)
> — `internal/backup/tier2_restore.go:197`.
> - The off-site reconstitution replays the snapshot's named-volume tars, volumes first (R-354) —
> `internal/backup/offbox_reconstitute.go`.
> - ep0's per-household prune jobs are keep-weekly 2 at 03:30 (R-1162/R-1165, `scripts/felhom-tenantsync.sh:187-197`);
> the local whole-guest tier is keep-daily=3 (R-1149, `felhom-agent/internal/config/config.go:553-559`).
> - restic is the pinned official release, checked by version and checksum (R-1105, `felhom-controller/controller/Dockerfile:43-46`),
> no longer Debian's 0.14.0.
## 0. Method, baselines, and what "verified" means here
@@ -0,0 +1,17 @@
Sat Oct 24 14:05:27 UTC 2026
gitea.dooplex.hu/admin/felhom-controller:0.342.0
repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 6 hours (healthy)
filebrowser gtstef/filebrowser:1.5.6-stable Up 23 hours (healthy)
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up About an hour (healthy)
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up About an hour (healthy)
immich-redis redis:7-alpine Up About an hour (healthy)
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up About an hour (healthy)
paperless-postgres postgres:18-alpine Up About an hour (healthy)
paperless-redis redis:7-alpine Up About an hour (healthy)
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up About an hour (healthy)
traefik traefik:v3.7.13 Up 23 hours
324af98a77bf1b5a6e0427187c6e1991 -
3
/dev/loop0 32G 2.7G 27G 10% /
/dev/nvme0n1 938G 239G 652G 27% /mnt/felhom-drives/scratch_hdd
@@ -94,3 +94,102 @@ Rules, fences, „never" lines and promises were NOT edited — those are rows f
- [S18] R-970 „does not hold" → the routes are removed, pinned by `r970_no_hub_key_door_test.go`.
- [S19] R-971 „design gap" → `privpolicy.Admit` before every write (controller v0.328.0).
- [S20] „hub 0.168.0 deploy pending", „R-1244 not released" → deployed / released 2026-10-24.
## 07-backup-architecture.md (DOC-D1)
- [S2/C3] Tier-1 „secret-stripped app.yaml" → carries the portable secrets (D5) — `felhom-controller/controller/internal/backup/recovery_unit.go:24-27`.
- [S7] `main.go:604-607` → `backupwindow.LegTimes` at `cmd/controller/main.go:1258-1264`.
- [S9] demo-hp „keep-last=1" → a dated note: keep-daily=N since R-1149 — `felhom-agent/internal/config/config.go:553-559`.
- [S1] D22 „controller main, unreleased" → released v0.338.0.
- [S11] §6.6 table → a row: the off-site restore also refuses a mismatched/mixed snapshot first — `internal/backup/offbox_reconstitute.go:826-833`.
- [S8] line citations moved (`GetAppDrivePath`, F5); `mp1` → a dated note (gone).
- [S3/C1] row 15 R-104 „open DEFECT" ×2 → FIXED 2026-10-05, v0.297.0 — `internal/backup/offbox.go:156-162`; CLOSED-ITEMS.md:895.
- [S6] row 10 R-893 / R-973 „unreleased" → released; „the next slice on R-893" → note: D20 (v0.336.0), D22 (v0.338.0).
- [S5] „Hub v0.158.0 … deploy waits" → deployed 2026-10-18 12:16 (hub CHANGELOG:288).
- [S10] „a restore is refused while any second-drive copy runs" → a dated note: the kept-data load checks only RestoreStatus/IsRunning (`internal/api/kept_install.go:89-92`) — a row.
- [U1/U2/U6 + hub U2/U3 + U7 → §8.2c, DESIGN — undocumented notes, one row for the operator]: the 60-min copy limit and stray handling; Eltávolítás refused during backups; start-after-put-back failure; the hub's off-site delivery checker (`offsite_delivery_stuck`, `maybeHeal`, the R-1230 skip — `hub/internal/monitor/offsite_delivery.go:103-110`); the release copy (D33) not in the matrix.
- NOT edited here: C4 (a stale code comment in `felhom-agent/internal/localapi/backup_cap.go:15`) — a row; C2 (G15 „nightly") — corrected in the capability map below.
## 09-update-architecture.md (DOC-C2) — dated notes; no §3 ruling text edited
- [S1/S2/S3] §1 (the 2026-09 measurement) → one dated note: `UpdateStack` gone, `StartGuardedUpdate`; moved line numbers.
- [S16] `stacks.update_window` „no Go code reads it" (inside ruling 11) → a dated note: removed.
- [S17] `unattended.go:433–435` → `:449-451`.
- [S4] §5.3 writers → note: `advancePinTo`; `pinBack`, the undo's `restoreDefinition`.
- [S7/C1] slice 6 „OPEN — R-450" ×2 → BUILT, R-450 closed 2026-10-05.
- [S12] R-646 residual → closed by `BackfillAppliedMeta` — `internal/stacks/undo.go:646-652`.
- [S18] `config.go L280-281` → L278-279.
- [S9] part 1 „DB up alone … empty-then-load" → note: the shipped folder copy — `undo.go:26-36`.
- [S10] „a step has no .felhom.yml of its own" → closed — `ladder.go:127-132`.
- [S11] „the box half … is not built" → built — `digest.go:15-24`.
- [S19] reason `current` → `already_current` — `update.go:432`.
- [S14] digest comparison „R-446 §3b Q6" → built for every ref with a ladder entry.
- [S13] §8.6 / „stepping is slice 6" → notes: any tier, the undo, the ladder.
- [S15] engine-major „until Slice 4" → note: MariaDB lifted, PostgreSQL by two-venue proof.
- [S5] recordInstalledImages callers → note: the real callers.
- [S8] §6.1 phase table → the bind-data tier order; a `converting` row (v0.273.0, `pgconvert.go`).
- [U1–U6, S6 → §6.4.x, DESIGN — undocumented notes]: build.sh's dirty/HEAD/non-git refusals (`controller/build.sh:144-166`); the once-retry; stopped_by_household; the manual leg's deadline; already_current/engine_no_test; the unreadable-template render row and R-971 admission.
- NOT edited (RULINGS — rows for the operator): C4 decision 13's „CompareImageRefs moves to the catalog gate" (not built); C5 decision 14's „never jumps" vs a person's press on an app older than the ladder; C7 decision 250 item 5's „by its AGE" vs the gate's proxy.
## 00-capability-map.md
- 20 PROVEN-LIVE rows marked „re-check (2026-10-24)" in their Scope cell, each with the commit that changed its mechanism after its evidence (status words unchanged): a7, a8, a10, a11, a17, b1, b2, b7, b11, c3, c4, c5, c6, d6, d10, d11, e17, g7, g15, g18. Each commit's files read with `git show --stat`.
- [DOC-D1 C2] G15 „a nightly consistent copy" → every 6 hours since D34.
## 16-storage-physical-identity.md (DOC-D2)
- [S2] the single-copy notice „a Go constant, not i18n" → the i18n keys — `felhom-controller/controller/internal/backup/backup.go:1485-1494`.
- [S3] `/disks/candidates` „whole disks only" → with nested partition candidates — `felhom-agent/internal/storage/candidates.go:84-90`.
- [S4] `felhom-host-install.sh:2261-2275` → `step_grows` at `:2475`.
- [S1] the R-1093 „not as written" → a note: fixed in agent v0.160.0 (`git tag --contains f7b7308`).
- [U4–U7 → DESIGN — undocumented notes]: detect-virt once; unreadable = same disk (R-668); the internal place's lifecycle refusals; its size rule.
## 15-media-library.md (DOC-D2)
- [S5/S6/C2] the §10 measurement table → one dated note before it: binds rewritten, restart on the household's press, the drive kept registered, and `b01bb4b`'s stopped-app rule.
- [S7] Transmission's watch/download folders → note: `excluded` since catalog `c50adfc` (`templates/transmission/.felhom.yml:46-51`).
- [U1/U3 → DESIGN — undocumented note]: the „Később" timer is in memory (`media_events.go` `handleMediaRestart`); the Sonarr/Radarr key read from `config.xml` (`mediawire.go:336-345`).
- [U2 — FIXED in code, not a note]: a media restart started a household-stopped app → controller `b01bb4b`.
## _recovery-inventory-2026-07-28.md (DOC-D2 S8–S13) — a WHAT HAS MOVED SINCE banner; the inventory body kept as its date's record
- one `mp0`, no `mp1` (`felhom-agent/configs/build-golden.sh:31-38,99`); portable secrets in the unit; `RestoreTier2Unit` (`tier2_restore.go:197`); volume tars replayed (R-354); ep0 keep-weekly 2 at 03:30 (`scripts/felhom-tenantsync.sh:187-197`); local keep-daily=3; restic pinned 0.19.1 (`Dockerfile:43-52`).
- runbooks/RUNBOOK-manual-guest-restore.md [DOC-D2 C3]: a dated note after the example config — one `mp0` at /var/lib/felhom, no `mp1` (golden v2.1.0+); the procedure unchanged.
## 14-host-power-off.md (DOC-E)
- [S2] „two exact sudoers vectors, FELHOM_POWEROFF" → `pct shutdown` in FELHOM_GUESTHOOK + `systemctl poweroff` in FELHOM_POWEROFF — `felhom-agent/configs/felhom-agent.sudoers:81-90`.
- [S1] heavy-gate holders → plus the line test (agent v0.164.0).
- [S3] the website FAQ line „a later task" → published — `website/gyik.html:128,603`.
- [S4] a paragraph that split the §3 table → moved below the table (rendering only).
- [U1–U3 → DESIGN — undocumented notes]: a re-enroll during the hold is „gone" (`hub/internal/api/reenroll.go:67-68`); the two info events; the three confirmation words.
## 11-os-updates.md (DOC-E)
- [S8] status „§8 step 2 BUILT" → steps 2–6 built by §8's own list (the ratification left to the operator).
- [S5] cloudflared `2026.6.0` → `2026.9.3` today — `felhom-controller/controller/internal/infra/infra.go:26`.
- [S6] apt sudoers/caller line numbers → `sudoers:68,228`, `lanresolver.go:133`, `wgtunnel/manager.go:659`.
- [S7] §5.4.1 „DRAFT … nothing installed" → note: built; sudoers holds only `--plan` (`sudoers:337`).
- [S9] BUNDLE_FILES 22 → 32 (counted in `felhom-agent/configs/felhom-os-apply`).
- [S10] HOST_SLOW_RE list → plus `linux-*`, `firmware-*`, `proxmox-secure-boot-support` (read from `felhom-os-apply`).
- [DOC-E U5 / DOC-C1 U5] §5.7 → the System page's „Upload line" cell (`hub/internal/web/system.go` `lineStaleAfter`).
## 10-localisation.md (DOC-E)
- [S12] „Part B is the remaining piece" → built (v0.256.0) — `felhom-controller/controller/internal/config/config.go:149-154`.
- [S13] formal-form ceiling 16 → 0 — `felhom-controller/controller/scripts/i18n_missing_gate.py:41`.
- [S11/C1] the setup wizard „still reachable … R-554" → deleted (`internal/setup` absent; `bootstrap.WaitForConfig`).
- [S15] `metadata.go L316` → L475.
- [S14] „53 real catalog files" (fixtures, still 53) / „fifth row" → sixth / „all 53 apps" → the freeze file's 65.
## 02-controller-module-map.md (DOC-E) — dated notes; the v0.33 plan tables kept
- [S16/C3] two in-flight markers → four (`update-journal.json`, `.felhom-install-pending`).
- [S17] three StartGate holders → four (1b `IsUpdating`, `cmd/controller/main.go:2459-2477`).
- [S18] `GetAlerts` → `GetBannerAlerts(page, lang)` (R-906) — `internal/web/alerts.go:353`.
- [S19] „no longer calls the Cloudflare API" → note: `cloudflare/` still wired with `cf_api_token` (`main.go:1992-2000`).
- [S20] `state.go`/`updater.go` DELETE → note: both live, `AgentSwapper` (`internal/selfupdate/updater.go:38-47`).
- [S21] `sync.go:319`, `config.go:351` → `:436`, `:399`.
- [S22] `locked_after_deploy` „read by no code" → note: read into `LockedFields`, enforced only by an uncalled function.
- [C2] §2 `UpdateStack` → note: deleted in v0.237.0.
- [U4] §1 → a note naming the 22 packages with no row (an `ls` of `controller/internal/`).
## felhom-controller/REUSE.md (DOC-D1 C5/U8, DOC-C2 C3)
- `CrashPutBack … (D22 / R-1195, unreleased)` → `v0.338.0`.
- `recordInstalledImages` callers `StartStack/RestartStack/UpdateStack/runComposeDeploy` → the real callers; `UpdateStack` deleted in v0.237.0.
- A new row: `Tier2CopyBlocksRestore` / `ClearLeftoverTier2Copies` / `MarkTier2CopyForTest` + the `tier2Docker` seam (`internal/backup/tier2_copy_lifecycle.go:85,255,319,341`).
## Late additions (the disposition check before the README)
- 03 §6 [DOC-B U1–U5, U8; DOC-A U4 → DESIGN — undocumented note]: 41 routes (24 unlisted), the controller-started line test, the island bridge, the line-test state and switch, the WG block's OOB fields, the D30 count in memory.
- 13 §5 [DOC-A U1 → DESIGN — undocumented note]: the file manager's one sign-in trust (`filebrowser_sso.go:15-22`).
- 08 §9 item 6 [SEC-1 X1]: the limits list names „any viewer explains any upload" (R-1262).
@@ -0,0 +1,90 @@
# Review round 2 — the architecture documents against the code, and a fresh data-loss and security look (2026-10-24, unattended)
**Baselines (read at start, 14:42 CEST):** felhom.eu `44d19042`, felhom-controller `321df1f` (v0.342.0 released;
„next v0.343.0" unreleased on top), felhom-agent `5e05627` (v0.164.0; „next v0.165.0"), app-catalog `5e196e1`, hub
deployed 0.169.0. Register: 190 rows.
**Method.** Thirteen read-only reviewers (each prompt carried the brief's fences word for word —
`reviewers/FENCES-given-to-every-reviewer.md`): seven on the documents (the brief's four groups, two of them split, plus a seventh for
`02`/`10`/`11`/`14`, which the groups did not name), four on data loss (restore and put-back; delete and retention;
keys; time and concurrency), two on security (hub + controller; agent + ep0 + installer + the DooPlex→ep0 jobs). Their
answers are CANDIDATES (`reviewers/`). The main session read every item it acted on at `file:line`; the edits and
their evidence are in `EDITS.md`.
## Part 1 — the documents (claims checked by the reviewers: 493; stale reported: 121)
| Document | Verdict |
|---|---|
| `01` topology & trust | 7 stale facts corrected; 2 product statements contradict the code → R-1266 (D45) |
| `03` host agent | 9 corrected (shipped releases, key files, listen address, swap, cloudflared's 4th state); a note: 41 local-API routes, 24 unlisted; break-glass text → R-1266 |
| `04` control-plane authorization | 3 corrected; a note: a signed op's window has no length cap → R-1265 |
| `05` hub | 9 corrected (the 6-hourly DB copy, version literals, the fifth bind-link path, 44 event types, the retired DR record) |
| `06` off-site connectivity | 7 dated notes (the design-of-record kept); „no inbound listener" → R-1266 |
| `07` backup | 12 corrected (portable secrets, R-104 fixed, keep-daily, D22/D20 released, a refusal row); §8.2c notes → R-1265; the kept-data load beside a copy → R-1256 |
| `08` alarm ladder | 10 corrected (per-type cooldowns, R-723, per-app grain, R-998 skip, §8.1 built, §9 released, the operator-only box alarms); one ruling line kept with a dated note (operator Q1 made `backup_integrity_failed` a default) |
| `09` updates | 19 dated notes and corrections (the §1 measurement, slice 6 built, the undo as shipped, digests, phase table); §6.4.x notes → R-1265; three rulings the code does not do as written → R-1266 |
| `10` localisation | 7 corrected (Part B built, ceiling 0, the wizard deleted, counts) |
| `11` OS updates | 7 corrected (cloudflared 2026.9.3, sudoers lines, the wrapper built, 32 bundle files, HOST_SLOW_RE); the System page's „Upload line" cell added |
| `12` agent tooling | 3 corrected; the five rule copies aligned (the instruction-file edits below) |
| `13` threat model | 11 facts corrected, every verdict cell left as it was and marked „re-check"; a note on the file manager's sign-in trust |
| `14` power-off | 4 corrected (sudoers vectors, line-test holder, FAQ published, the table split); notes → R-1265 |
| `15` media library | a dated note on the §10 measurement table; R-1019's fix; notes → R-1265 |
| `16` storage identity | 4 corrected; notes → R-1265 |
| `02` controller module map | 8 dated notes (markers, holders, banner alerts, cloudflare still wired, updater live, LockedFields, 22 packages with no row) |
| `_hub-review.md` / `_recovery-inventory-2026-07-28.md` | a HISTORICAL banner / a „what has moved since" banner — both dated records kept as they were |
| `00` capability map | 20 PROVEN-LIVE rows marked „re-check" with the commit that changed their mechanism (no status lowered); G15's „nightly" corrected |
| `felhom-system-poster` facts + html | the timeline's hub-DB push (text only; no drawing needed) |
**Rules, fences and promises were not edited.** Seven items that need the operator are in R-1266 (D45); about 30
behaviours with no document are written as `[DESIGN — undocumented, found 2026-10-24]` notes for the operator to confirm
(R-1265, D44).
**Instruction files edited (unprompted-work §5 — facts only):** the workspace root, felhom-agent and app-catalog copies
of `.claude/rules/unprompted-work.md` got the missing „A version number is pushed once" paragraph (now byte-identical to
the other two); `felhom.eu/.claude/rules/live-probes.md:41` — the stack path `/opt/felhom/stacks/<name>` → `/opt/docker/
stacks/<name>/app.yaml`. Details: `EDITS.md`.
## Parts 2 and 3 — findings, ranked
| # | Sev | Finding | What was proven | Row |
|---|---|---|---|---|
| 1 | **P1** | A power cut during a restore of a drive-backed app: the app runs on the cut restore's data, and a later controller restart pours the pre-restore copy back over newer writes | the real `Recover` code in a lean clone: boot 1 „copy gone → refused → marker kept", boot 2 put-back after the app ran (`proofs/H1-drive-cut-restore/`); the boot recreate's missing check read at `intermediary.go:470-476` | R-1251 |
| 2 | P2 | One rejected backup block makes the night's second-drive copy delete the app's household-file copies | **reproduced on 9202**: control kept the copy twice; one bogus path → „removed stale dest dir userdata/media" (`proofs/H2-bad-backup-block/`) | R-1252 |
| 3 | P2 | An export with „stop first" overwrites another app's restore marker | static, read at `file:line` | R-1253 |
| 4 | P2 | The hub saves the escrow package in two writes; a lost answer tells the household „the previous code still works" | static | R-1254 |
| 5 | P2 | D4's sshd drop-in reaches BYO hosts (against `11` §1); the in-force check and the all-or-nothing `sshd -t` | static | R-1257 (D43) |
| 6 | P3 | Escrow carry / set-aside name edges | static | R-1255 |
| 7 | P3 | Four restore edges (definition write, merge, a mirror race, the unused copy) | static | R-1256 |
| 8 | P3 | ep0's iperf3 takes any tunnel peer's test | static (no ep0 change) | R-1258 |
| 9 | P3 | The DooPlex→ep0 copies: the vouched agent can drop out; deletions mid-transfer; no future bound; **2026-10-25: the 02:30 push re-sends the 20:00 snapshot** | DST timing verified on DooPlex (`systemd-analyze`, Go) | R-1259 |
| 10 | P3 | Spring 2027: the leg's deadline vs the gate's clock; two legs at one instant | Go's mapping verified | R-1260 |
| 11 | P3 | The unreleased R-1250 line hold: two seeders, a removed app, alt-speed | static | R-1261 |
| 12 | P3 | Any viewer explains any upload (the upload watch) | static; `08`'s limits list now names it | R-1262 |
| 13 | P4 | Small items (a mislabelled operator mail, undeclared volumes on Remove, mount views, stale comments, a DNS-name password, rrsync) | static | R-1263 |
| 14 | P4 | Skills tooling (no size cap on the grandfathered skill; a new skill not installed) | read | R-1264 |
| 15 | P4 | Immich leaves an empty anonymous volume per recreate (296 on 9202); photos unaffected | read in the running image | R-1267 |
**Fixed in this session (small, not recovery code, red-proved, commit only — controller `b01bb4b`, CI 2294 / #480 success):**
a media restart („Újraindítás most", „Később", the safe removal) no longer starts an app the household stopped (R-721;
two tests, each seen failing with the rule off); the file manager's header name file appears whole (a race the
existing test caught 44 times in 300 runs on clean `main`; 0 in 300 after, 0 in 50 under `-race`).
**NOT-A-FINDING:** SEC-1 X2 (a passphrase enroll after the operator's host delete gets ciphertext — the window D19
accepts); SEC-1 X3 (the refused-passphrase mail is operator-only with a one-hour cooldown); DOC-D1 U4 (the fit check
that passes on a statfs error — the copy then fails and the app restarts untouched, `restore_unit.go:502-512`);
HUNT-3 D31's comment wording (no data effect); Immich's `/data` volume for data placement (R-1267 holds the leak);
the stack folder left after a clean removal (it is every catalog app's template folder).
## Machines
- **9202** (Tier 0): as found at 12:46 UTC (`9202/as-found-2026-10-24.txt`); at the end (`9202/at-end-…`) the same
containers and images, healthy (restarted by the test night chains), the catalog on the live repo, the controller
0.342.0. The only difference: Immich's running container uses a new empty anonymous volume (R-1267); the two others
my test runs made were removed by name. audiobookshelf installed and removed through the product; my marker file
removed; its second-drive copy removed by the product's remove.
- **DooPlex:** reads, lean clones in the scratchpad, the gates; no service changed. **ep0, household boxes, the hub:**
not touched (hub reads only, none needed in the end).
## Teardown (three layers)
Machine: 9202 as above. Host (demo-hp): nothing provisioned. Hub: nothing changed.
@@ -0,0 +1,37 @@
## FENCES — the brief's, word for word (operator ruling 2026-10-07: every helper prompt carries them in full)
From the brief's §0 Scope:
- **Class:** review + verification, unattended. **Code is not changed** except a fix that is small, not in recovery code,
and has a red-proof (commit only). Documents may be corrected for facts proven at `file:line` (PROMPT-TEMPLATE §9
rule 9: facts yes, rules and fences never without the operator).
- **Machines:** DooPlex (lean clones, builds, read-only reads), 9202 for proofs (put back as found). No household box, no
ep0 change, no release.
- **Coordination:** the catalog sweep runs in parallel on demo-hp's nested VM and writes to `felhom.eu`. `git pull
--rebase` before each push; never remove its lines. **Do not use 9202 between 18:00 and 20:00 if the sweep asks for it**
(it should not; it has its own VM).
- **Reviewers:** read-only subagents; only the main session touches machines, git and the register.
From the brief's Rules:
1. Verify every claim at `file:line` before acting; a reviewer's word is a candidate, not a finding. 2. Documents: facts
yes; rules, fences, „never" lines and protected-machine text never without the operator. 3. Code: only small, non-recovery
fixes with a red-proof, commit only. 4. No release, deploy, vouch, publish, ep0 or DooPlex change. 5. Secrets: never
printed or committed. 6. 9202 put back as found. 7. Every finding a row, a fix, or `NOT-A-FINDING: <reason>`. 8. Stop at a
clean point. 9. No branches; no „Co-Authored-By".
From the workspace CLAUDE.md (this host is production infrastructure):
- NEVER run `docker system prune`, `docker image prune -a`, or any global Docker cleanup here.
- NEVER touch k3s data dirs, Longhorn mounts, PBS datastores, or Gitea storage.
- Destructive disk/guest operations belong to felhom-pve via the agent — never on this host.
- Secrets: never `source` or `cat` a credentials file; never print a token, password, key or `PlexOnlineToken`.
## WHAT THIS MEANS FOR YOU, THE HELPER (binding)
You are a **read-only reviewer**. You may ONLY read files in `/mnt/5_hdd/felhom.eu/git/` (Read, Grep, Glob, and
read-only shell such as `grep`, `rg`, `sed -n`, `git log`, `git show`, `git blame`). You may NOT: edit or write any file;
run `git` commands that change anything (no commit, checkout, reset, stash, pull, push, worktree); run `go test`, `go
build`, any gate script, `docker`, `kubectl`, `ssh`, `curl` to any host, `sec`; touch 9202, demo-hp, the N100, Tester 1,
Tester 2, ep0, the hub or DooPlex's services. Only the main session touches machines, git and the register. If you
think something needs a live test, write the test plan in your answer — do not run it.
Repos (read their `main` working trees): `felhom.eu` (hub in `hub/`, docs in `documentation/`, scripts in `scripts/`),
`felhom-controller` (Go in `controller/`), `felhom-agent`, `app-catalog-felhom.eu`.
File diff suppressed because one or more lines are too long
+4
View File
@@ -33,6 +33,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-23 (afternoon).** Your D39 = B is built and live: an app with a public security hole can now get its fix at night on every box, after one test on the test box — controller 0.341.0 is the floor since 13:24 (your yes). The test: Plex moved by itself at night on the test box, and an older controller safely waited for a hand press. Nothing needs you.
## Moved from STATUS on 2026-10-24 (evening)
**Updated 2026-10-24 (afternoon, attended).** The two DooPlex backup alarms had one cause: yesterday's reboot changed which password the database takes, so the database copy job could not log in, and the Gitea off-site copy then refused its old copy. Fixed and cleared at 12:51 local; critical alarms now repeat every 4 hours (all backup alarms in one mail), warnings every 12. A box whose torrent app seeds all day can now get its weekly line test (proven on demo-hp, not yet released).
## Moved from STATUS on 2026-10-23 (afternoon)
**Updated 2026-10-23 (day).** Released on your go: controller 0.340.0 (the floor), agent 0.163.0 (vouched; root SSH by key only on the three test boxes), installer 1.36.0, hub 0.167.0, and the website with your two sentences. The hub database now leaves every 6 hours and the released images and agent packages every night, both read back from ep0; the break-glass sheet is current — please print it. One false failure mail reached you at 10:28 (my install of the 6-hour copy, fixed in the same hour).
@@ -31,6 +31,11 @@ mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=
hookscript: local:snippets/felhom-guest-hook.sh
```
*Dated note 2026-10-24 (review round 2): a guest built from golden `build-golden.sh` v2.1.0 or later has ONE data volume,
`mp0` at `/var/lib/felhom` (its `docker/` and `sys_drive/` folders are bound inside the guest), and **no `mp1`**
(`felhom-agent/configs/build-golden.sh:31-38,99`). The procedure below is unchanged: volume mounts (whatever their
number) come back from the archive; only the host BINDS are stripped.*
**Two kinds of `mpN`, and only one is dangerous:**
| kind | looks like | on restore |