Review round 2 (batch 2 + close-out): 02/07/09/10/11/14/15/16/inventory facts corrected at file:line (EDITS.md), undocumented decisions as notes (R-1265), rulings left for the operator (R-1266); register 190 -> 207 (R-1251..R-1267 opened: R-1251 P1 cut drive-app restore put back over newer data, R-1252 P2 rejected backup block deletes second-drive copies - reproduced on 9202); STATUS D43-D45; CONTEXT; audit README
gates / gates (push) Successful in 13m10s
gates / gates (push) Successful in 13m10s
This commit is contained in:
@@ -63,6 +63,13 @@
|
||||
> STATUS:** the default limit's size (CC: 50 % of the measured upload) and who is told (CC: the family only about the
|
||||
> qBittorrent limit they can change; the unexplained alarms to the operator only — the R-388 rule).
|
||||
|
||||
> **2026-10-24 (evening, unattended) — review round 2** (`audits/review-round-2-2026-10-24/README.md`). The 17 architecture
|
||||
> documents checked against `main`: facts corrected at `file:line` (EDITS.md), rulings and promises left for the operator
|
||||
> (R-1266, D45), undocumented decisions written as notes (R-1265, D44). New: **R-1251 (P1)** a cut restore of a drive-backed
|
||||
> app is put back over newer data (shown with the real `Recover`); **R-1252 (P2)** a rejected backup block makes the
|
||||
> second-drive reconcile delete the household's copies (reproduced on 9202). Controller `b01bb4b` (unreleased, next
|
||||
> v0.343.0): a media restart skips household-stopped apps (R-721); the FileBrowser SSO header file appears whole.
|
||||
|
||||
> **2026-10-23 ~12:00 — operator rulings by starting the D39 brief (`09` §3 decision 255; R-1239 closed).** **D39 = B:** a
|
||||
> `security_fix` ladder step — box walk on 9202 required, bench optional, taken by the night leg. **No operator word per
|
||||
> incident:** a public advisory id (CVE or GHSA) + its https:// notice is the authority; the session that writes the step
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# REPORT — review round 2 (2026-10-24, unattended): the documents against the code, data loss and security
|
||||
|
||||
**Plain words first.**
|
||||
- The 17 design documents were checked against the code: 493 claims, 121 out of date. About 130 places are corrected or
|
||||
marked with a dated note; no rule, fence or promise was changed.
|
||||
- About 30 things the code does that no document named are now written down for you to confirm (D44), and seven places
|
||||
where a document's ruling or promise says something the code does not do are waiting for you (D45).
|
||||
- The worst data-loss finding: a power cut during a restore of an app on a drive can later pour old data over the
|
||||
household's new data (R-1251, P1). Proven with the real recovery code; it needs a fix in an attended run.
|
||||
- Also proven, on the scratch box: one wrong line in an app's backup settings deletes that app's second-drive copy
|
||||
(R-1252, P2).
|
||||
- The worst security finding: the „root logs in by key only" SSH rule also reaches a host the household owns (BYO),
|
||||
where we promised to change nothing (R-1257, D43). Nothing serious was found in the hub's or the controller's new routes.
|
||||
|
||||
**Evidence and the full tables:** `documentation/audits/review-round-2-2026-10-24/README.md`; every document edit with
|
||||
its `file:line` in `EDITS.md`; the reviewers' candidates in `reviewers/`; the two proofs in `proofs/`.
|
||||
|
||||
**Commits (all CI green by hash):** controller `b01bb4b` (two small fixes, red-proved; unreleased) — CI 2294 / #480;
|
||||
controller `2449223` (REUSE.md); felhom-agent `f5c3d77` and catalog `227a544` (the rule-file alignment); felhom.eu
|
||||
`77015997` (documents batch 1) and this commit (batch 2, the register, STATUS, CONTEXT). The CI numbers of the last four
|
||||
are in the final report.
|
||||
|
||||
**Register:** before 190, after 207; opened 17 (R-1251–R-1267), closed 0.
|
||||
|
||||
**Instruction files edited (facts only, named here):** `.claude/rules/unprompted-work.md` in felhom-agent, the catalog
|
||||
and the workspace root — the missing „A version number is pushed once" paragraph (before 95 lines, md5 3b071ad2; after
|
||||
byte-identical to the other two, b1891bd4); `felhom.eu/.claude/rules/live-probes.md:41` — `/opt/felhom/stacks/<name>` →
|
||||
`/opt/docker/stacks/<name>/app.yaml` (the real path, and the folder stays after a clean remove).
|
||||
|
||||
**System poster:** the facts file and the html changed together (text only, the hub-DB push times); no new drawing.
|
||||
|
||||
**Machines:** 9202 used for the R-1252 proof and put back (one difference: Immich's new empty volume, R-1267). DooPlex,
|
||||
ep0, household boxes and the hub: not changed. No release, deploy, vouch or publish.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-10-24 (afternoon, attended).** The two DooPlex backup alarms had one cause: yesterday's reboot changed which password the database takes, so the database copy job could not log in, and the Gitea off-site copy then refused its old copy. Fixed and cleared at 12:51 local; critical alarms now repeat every 4 hours (all backup alarms in one mail), warnings every 12. A box whose torrent app seeds all day can now get its weekly line test (proven on demo-hp, not yet released).
|
||||
**Updated 2026-10-24 (evening, unattended review).** The 17 design documents were checked against the code; about 130 old facts are corrected, no rule changed. Two new data-loss holes are proven; both need a fix in an attended run. Two small controller faults are fixed, not released.
|
||||
|
||||
## What works
|
||||
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
|
||||
@@ -18,6 +18,7 @@
|
||||
- The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15).
|
||||
|
||||
## What is broken
|
||||
- **P1 (new):** a power cut while an app on a drive is restored can later pour old data over new data (R-1251); one wrong line in an app's backup settings deletes its second-drive copy (R-1252, P2).
|
||||
- **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
|
||||
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). The scratch box cannot use agent features (R-1083).
|
||||
- DooPlex: something sets the shared checkout to „bare" (R-935).
|
||||
@@ -31,6 +32,9 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
|
||||
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||
|---|---|---|---|---|---|---|
|
||||
| D45 | R-1266 | Six places where a design document says one thing and the code does another (two update rules, the golden check, „the hub never connects into a box", „the operator cannot open the off-site copy", the tunnel plan). Should the text follow the code? | **Yes — the text follows the code** (each is how the system works today; you read every changed sentence) | the code follows the text: six code changes, one of them to the box's door for you | the documents keep saying what is not true | — |
|
||||
| D44 | R-1265 | Confirm about 25 behaviours the code has but no document named (now written as notes), and that MeTube's downloads are not backed up? | **Yes, all** (each was read in the code; none adds risk) | name the ones to change; each becomes a row | the notes stay marked „undocumented" | — |
|
||||
| D43 | R-1257 | The SSH rule „root logs in by key only" also reaches a box the household owns (a BYO host), where we promised to change nothing on the host. Keep it there too? | **No — skip it on BYO hosts** (it is their machine; we said we would not touch it) | keep it: a BYO owner who uses a root password over SSH loses it at the next update, and the installer screen must say so | it reaches every BYO host at its next update | — |
|
||||
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
|
||||
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
|
||||
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
|
||||
|
||||
@@ -246,7 +246,7 @@
|
||||
| <a id="g12"></a>[G12](capability-map-detail/G.md#g12) | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | — | the 2026-07-13 hub closing bundle (no evidence document) |
|
||||
| <a id="g13"></a>[G13](capability-map-detail/G.md#g13) | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | — | hub v0.54 (2026-07-13; no evidence document) |
|
||||
| <a id="g14"></a>[G14](capability-map-detail/G.md#g14) | The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail | hub v0.135.0 (R-604, R-530) | **IMPLEMENTED** | the page live; the alarm and the mail unit-proven | `audits/hub-safety-2026-10-05/partD/` (2026-10-05) |
|
||||
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a consistent copy every 6 hours since D34 (nightly before 2026-10-23; corrected 2026-10-24), encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
|
||||
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
|
||||
|
||||
@@ -105,7 +105,9 @@ Tri-state: `""` (unknown) · `"running"` · `"stopped"`.
|
||||
|
||||
### In-flight — a marker file, one per owner
|
||||
|
||||
Two exist and they are deliberately **separate files**: `quiesce-state.json` (the whole-guest backup
|
||||
Two exist and they are deliberately **separate files** *(four in-flight records by 2026-10-24: also `update-journal.json`
|
||||
— the guarded update's crash journal read by `RecoverUpdates`, `internal/stacks/update.go:49,1306` — and
|
||||
`.felhom-install-pending` — an interrupted install, R-681, `install_interrupted.go:28`; noted 2026-10-24)*: `quiesce-state.json` (the whole-guest backup
|
||||
window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app —
|
||||
`backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One
|
||||
file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's
|
||||
@@ -169,7 +171,8 @@ A recovery completing after the grace emits a `LATE RECOVERY` warning naming the
|
||||
|
||||
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
|
||||
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
|
||||
answer it, and the last two only became reachable once the window widened past T+5 s:
|
||||
answer it *(four since slice 4: check „1b", a guarded update moving the app — `IsUpdating`, `cmd/controller/main.go:2459-2477`;
|
||||
noted 2026-10-24)*, and the last two only became reachable once the window widened past T+5 s:
|
||||
|
||||
| Holder | Why starting would be wrong |
|
||||
|---|---|
|
||||
@@ -218,6 +221,13 @@ own; every caller that is not the customer must decide for itself whether the ap
|
||||
|
||||
---
|
||||
|
||||
> *Dated note 2026-10-24 (review round 2): this inventory is v0.33's. On `main` (`321df1f`) 22 packages under
|
||||
> `controller/internal/` have no row here: `aftercrash`, `agentapi`, `appbackup`, `backupwindow`, `bootrecon`,
|
||||
> `bootstrap`, `channelhealth`, `crashboot`, `dockerexec`, `family`, `fillwatch`, `i18n`, `infra`, `logx`, `mailrelay`,
|
||||
> `mediawire`, `nightchain`, `offsiteapply`, `privpolicy`, `quiesce`, `sockheal`, `upwatch` (an `ls` of the tree). Each
|
||||
> package's own doc comment is its description; `sockheal` (re-heal after the guest's Docker socket is recreated,
|
||||
> R-860) and `channelhealth` (a periodic proof of the controller→agent channel) are in no architecture document.*
|
||||
|
||||
## 2. Classification table (per package/file)
|
||||
|
||||
### `cmd/`
|
||||
@@ -266,6 +276,11 @@ own; every caller that is not the customer must decide for itself whether the ap
|
||||
|---|---|---|---|
|
||||
| `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework |
|
||||
|
||||
*As built (noted 2026-10-24, review round 2): the `cloudflare/` package is still wired — with `cf_api_token` set the
|
||||
controller builds a `GeoSyncManager` and calls the Cloudflare API (`cmd/controller/main.go:1992-2000`; the field is still in
|
||||
`config.go:160`). The DELETE classification above is the plan, not the state; whether any box sets the token is not
|
||||
readable from source.*
|
||||
|
||||
### `config/`, `crypto/`, `util/`
|
||||
| File | Class | Reason | Risk |
|
||||
|---|---|---|---|
|
||||
@@ -325,6 +340,7 @@ household's file names. The household's lost list is shown on the app page (`los
|
||||
| File | Class | Reason | Risk |
|
||||
|---|---|---|---|
|
||||
| `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean |
|
||||
| *(as built 2026-10-24: both files below are LIVE — the updater decides the version and the floor and hands the container swap to the agent, `AgentSwapper`, `internal/selfupdate/updater.go:38-47`; `state.go` keeps `update-state.json`)* | | | |
|
||||
| `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean |
|
||||
| `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean |
|
||||
|
||||
@@ -387,7 +403,8 @@ dashboard alert is an `Alert` with two placement fields: `PageOnly` (the pages i
|
||||
page) and `Inline` (rendered by the page template in place, not by the layout's banner). `GetAlerts` returns
|
||||
the list (endpoint-drift, agent-channel and dead-app alerts first, then the rest sorted error > warning > info,
|
||||
capped at five plus an overflow line), and `layout.html` paints only those that are not `Inline` and match the
|
||||
page; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
|
||||
page *(as built since R-906 — noted 2026-10-24: pages use `GetBannerAlerts(page, lang)`, which filters per page BEFORE the
|
||||
cap, `internal/web/alerts.go:353`, `handlers.go:144`)*; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
|
||||
|
||||
**Exactly one warning is inline today:** the „storage is not on a separate drive" health warning. It is
|
||||
`PageOnly: dashboard, monitoring` and `Inline: true`, so it sits quietly under the storage bars; **every other
|
||||
@@ -821,7 +838,7 @@ R-438. The three facts below are the ones a reader needs before touching any of
|
||||
`Syncer.copyTemplates` (`sync/sync.go:319`) walks every directory in the catalog cache and copies
|
||||
`docker-compose.yml` and `.felhom.yml` into the matching stack folder. **There is no test of whether
|
||||
the app is deployed.** The only guard is a sha256 content compare (`copyIfChanged`) and the only
|
||||
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351`), plus
|
||||
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351` then, `:399` now; `copyTemplates` is now `sync.go:436`), plus
|
||||
one immediate sync at controller start (`sync.go:98`).
|
||||
|
||||
**SINCE v0.235.0** this walk still happens and `.felhom.yml` is still copied unconditionally, but the
|
||||
@@ -835,6 +852,7 @@ and they stay that way until something else acts.
|
||||
|
||||
### 2. Every lifecycle path resolves that disagreement, silently, by upgrading
|
||||
|
||||
*(This section is the 2026-09 measurement; `UpdateStack` was deleted in v0.237.0 — the button calls `StartGuardedUpdate`, `09` §6.1 — noted 2026-10-24.)*
|
||||
`StartStack`, `RestartStack` and `UpdateStack` (`stacks/manager.go:1029 / 1133 / 1170`) all end in
|
||||
`docker compose up -d`. `up -d` makes the container match the file, **and pulls the image itself if it
|
||||
is absent** — measured at 18.3 s with a pull versus 0.5 s without, against a negative control
|
||||
@@ -887,7 +905,9 @@ measured; that the restore writes to that path is read, not measured.
|
||||
beállítások csak olvashatók.", `deploy.html`). This is a design, not an oversight: a changed value would
|
||||
need a guarded re-deploy that nothing performs. **Finding (2026-09-15):** the catalog field flag
|
||||
`locked_after_deploy` (`stacks/metadata.go`) is parsed and read by **no** controller code — every field
|
||||
is read-only after install whatever the catalog says. So a page must never tell a household to change
|
||||
is read-only after install whatever the catalog says. *(Dated note 2026-10-24: it is now read into the app's
|
||||
`LockedFields` (`internal/stacks/deploy.go:106,534`) and enforced only by `UpdateStackConfig`, which nothing calls — so
|
||||
„read-only whatever the catalog says" still holds in practice.)* So a page must never tell a household to change
|
||||
a value after install. **Vaultwarden (R-512)** follows the design instead of breaking it: registration
|
||||
is closed by default and the household is invited from the admin panel, measured to work without mail
|
||||
(stranger 400, invite 200, invited 200).
|
||||
|
||||
@@ -308,6 +308,18 @@ A controller can only `POST /rollback` (or snapshot/backup) **its own** guest
|
||||
token → guest and authorizes per guest, so a compromised controller's blast radius is
|
||||
**self-scoped and bounded** to its own guest.
|
||||
|
||||
**[DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through R-1265]:** the surface listed above is
|
||||
the slice-8 design; on `main` the local API registers **41** routes (`felhom-agent/internal/localapi/server.go`; 24 of
|
||||
them are not listed above — the backup tiers and target, disk candidates, decommission, format status, guest attach,
|
||||
the network storage routes, the controller swap, the escrow stage/preflight/ceremony routes, the debug logs, guest
|
||||
memory). Among them the controller can start a HOST action: `POST /host/line-test/run` (about 55 s; bounded by the due
|
||||
date, the window, 6 tries a day and the heavy-operation gate — R-1250, unreleased). The local API also listens on the
|
||||
R-50 island bridge (host `169.254.253.1`, guest `net1` `169.254.253.2/30`, `internal/config/config.go:260-269`). The
|
||||
weekly line test keeps its state in the OOB state dir (`line-test.json`) and has an off switch, `line_test.disable`.
|
||||
The served WireGuard block carries `oob_peer_ip` and `oob_operator_ssh_key` (`internal/hub/report.go`). The D30 re-key
|
||||
runs only inside the tunnel watchdog, and its 15-minute count lives in memory — an agent restart starts it again (the
|
||||
once-an-hour bound survives in `wg/last-rekey`).
|
||||
|
||||
### 6a. Implementation (slice 8A — implemented)
|
||||
|
||||
**Status: implemented** (agent v0.10.0 `internal/localapi`; controller v0.35.0 `internal/bootstrap`
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -593,7 +593,7 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me
|
||||
94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`).
|
||||
While the box owns it, qBittorrent's page says: „A qBittorrent
|
||||
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
|
||||
6. **Limits, stated:** an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
|
||||
6. **Limits, stated:** *(added 2026-10-24, review round 2: ANY viewer explains any amount of upload, so a hacked app can send freely while one real playback runs — `upwatch/watch.go:275,299`; R-1262)*; an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
|
||||
a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood
|
||||
whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container
|
||||
(Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line.
|
||||
|
||||
@@ -65,6 +65,13 @@ build is added to both in the same commit.
|
||||
|
||||
### 1.1 The button
|
||||
|
||||
> *Dated note 2026-10-24 (review round 2): §1 is the 2026-09 MEASUREMENT before the arc. `UpdateStack` no longer exists
|
||||
> (deleted in v0.237.0, controller `REUSE.md`); the button calls `StartGuardedUpdate` — backup precondition, safety
|
||||
> dump, pin journal, undo copy, health wait, undo (`internal/api/router.go:794`; `internal/stacks/update.go`). The line
|
||||
> numbers below are the measurement's; e.g. `copyTemplates` is now `sync.go:436`, `copyIfChanged` `:700`, the `15m`
|
||||
> default `config.go:399`, `RestartStack` `manager.go:1454` (it now runs `up -d --force-recreate` and errors when no
|
||||
> container restarted, R-1026).*
|
||||
|
||||
`Manager.UpdateStack` (`felhom-controller/controller/internal/stacks/manager.go:1199`) is two compose
|
||||
commands and nothing else:
|
||||
|
||||
@@ -267,7 +274,8 @@ builds them (`audits/update-rulings-2026-09-23/`).
|
||||
next night. **Why:** the update then leans on the freshest copy the box ever has, and there is no
|
||||
second window for anyone to set or misread. **Replaces:** §3b Q1's recommendation of a fixed
|
||||
02:30–05:00. *For the builder:* `stacks.update_window` (`config.go` L195, default `03:00-05:00`)
|
||||
exists and **no Go code reads it** — it appears only in `config.go` (field + default),
|
||||
exists and **no Go code reads it** *(dated note 2026-10-24: the field is now removed — no controller source names
|
||||
it; `controller/README.md` „`stacks.update_window` is removed")* — it appears only in `config.go` (field + default),
|
||||
`setup/handlers.go` L482 (written into a new box's `controller.yaml`) and
|
||||
`configs/controller.yaml.example`. Remove it or fold it into the chain; do not build a second
|
||||
window.
|
||||
@@ -424,7 +432,8 @@ R-636's louder repeated alarm.
|
||||
(a) broke the second. Shipped as controller **v0.269.1** — a second release in the session, instead of
|
||||
shipping (a) to the fleet with the floor.
|
||||
*Note 2026-09-30 (R-740) — the cost line above is not what the code does.* „(or the automatic leg)": the
|
||||
automatic leg never presses a digest-only change (`unattended.go:433–435`, skip `no_test_record`). And the
|
||||
automatic leg never presses a digest-only change (`unattended.go:433–435` then, `:449-451` now — line corrected
|
||||
2026-10-24 — skip `no_test_record`). And the
|
||||
catalog never writes a re-test of a floating tag at a new digest (the writer refuses a step that moves no
|
||||
image), so a same-name upstream fix reaches no box by either route today. Measured, with the options, in
|
||||
R-740; the decision is the operator's (STATUS 2026-09-30). This decision's ruling — the sync never moves
|
||||
@@ -1981,7 +1990,7 @@ moves a stack dir.
|
||||
| writer | pin source |
|
||||
|---|---|
|
||||
| the deploy path (`runComposeDeploy`) | the template just deployed from |
|
||||
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull |
|
||||
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull — *as built: `UpdateStack` was deleted in v0.237.0; the guarded update writes the pin (`advancePinTo`, the `pinning` phase before `pulling`), and two more paths move a pin BACK — `pinBack` and the undo's `restoreDefinition` (`internal/stacks/update.go`, `undo.go`; noted 2026-10-24)* |
|
||||
| the restore (`stackAdapter.RecreateStackDefinitionFromUnit`) | the recovery unit's captured compose — **this closes R-441** |
|
||||
| `Manager.AdoptPins` | the observation, once, and only when complete AND matching |
|
||||
|
||||
@@ -2047,7 +2056,7 @@ earlier feature is the failure mode to look for whenever a file changes meaning.
|
||||
| **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 |
|
||||
| **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | **SHIPPED + PROVEN LIVE, controller v0.237.0 (job) + v0.238.0 (page) + v0.238.1 (2026-09-13); any backup tier since v0.239.0 (§3 decision 8)** — §6.1 |
|
||||
| **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 |
|
||||
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | OPEN — R-450; rulings 2026-09-23; build order §6.4 |
|
||||
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | ~~OPEN — R-450~~ **BUILT** — the leg in `internal/stacks/unattended.go`, wired into the off-site job; R-450 CLOSED 2026-10-05 (capability map B11) *(noted 2026-10-24, review round 2)* |
|
||||
| **7** | **A fleet sweep pipeline** — the operator can see, and move, how far behind every box is. | OPEN — R-451; ruled 2026-09-23 (decision 18), built later |
|
||||
|
||||
### 6.1 Slice 4 as SHIPPED (controller v0.237.0 + v0.238.0, 2026-09-13)
|
||||
@@ -2062,12 +2071,13 @@ closed by construction: nothing reports an update complete on the compose exit c
|
||||
| # | phase | what happens | on failure |
|
||||
|---|---|---|---|
|
||||
| 0 | refusals (409, before the intent is recorded) | held (R-439), busy (backup/restore/app-data op/quiesce), migration, already updating, deploying, memory (the deploy's `memoryVerdict`, releasing the app's own request), disk (**fixed 2 GB floor** — image size unknown without a registry), **no copy on ANY tier and no backup can be taken now** (since v0.239.0; before it, no restorable Tier-2 copy) | nothing moves, nothing is recorded |
|
||||
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) | nothing moves |
|
||||
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) — *an app whose data lives in bind-mounted files walks second drive → OFF-SITE → own unit (decision 9, R-479, v0.241.0; `internal/backup/update_guard.go` `updateTierOrderBindData`; noted 2026-10-24)* | nothing moves |
|
||||
| 2 | `backing-up` — only when no tier holds a fresh copy | `RunAppBackupNow`: this app's DB dump → volume dump → unit capture (marked proven current) → Tier-2 copy, whose failure is a WARN since v0.239.0 | refused with the backup's own error; nothing moves |
|
||||
| 3 | `safety-dump` | `WriteUpdateSafetyDump` (R-361's undo copy) — **before the pin moves** | refused; nothing moves |
|
||||
| 4 | `pinning` | the previous definition is copied aside and journaled, then the pin advances | pin put back |
|
||||
| 5 | `pulling` | `compose pull` | **pin and definition PUT BACK** — nothing ran (Scenario E) |
|
||||
| 5a | `copying` **(v0.263.0)** | `compose stop`, then each NAMED volume `cp -a` into `<volume>.pre-update-<stamp>` by a helper that writes a finished-marker last (bind folders never) | copies removed, **pin put back, the old version started again** — nothing new ran |
|
||||
| 5b | `converting` **(v0.273.0, row added 2026-10-24)** | only on a step whose ladder entry carries `engine_conversion`: the box converts the PostgreSQL major (dump from the old engine, load into the new, check) — `internal/stacks/pgconvert.go` | the undo puts the old engine's data back |
|
||||
| 6 | `starting` | `compose up -d --remove-orphans` | **UNDO** (below), HOLD only if the undo fails |
|
||||
| 7 | `verifying` | the `.felhom.yml` health check through the existing probe **when it resolves to a container**, else 60 s of every container running and none restarting; bounded by `update.health_timeout` | **UNDO** (below), HOLD only if the undo fails. Before v0.263.0: stop + HOLD, the pin stays (Scenario F) |
|
||||
| 7a | `undoing` **(v0.263.0)** | every copy validated (marker) BEFORE anything is poured back; volumes emptied and refilled; definition, pin and the pinned version's `.felhom.yml` record put back from the job's own copies; `up`; health with the OLD version's probe | HOLD, the sentence prefixed *„A frissítés nem sikerült, és az automatikus visszaállítás sem."* + the data state (`untouched` / `half` / `not_started`); copies kept |
|
||||
@@ -2116,7 +2126,7 @@ the harness has proven it, is slice 6's.
|
||||
> copy taken before the pin moved**, so the old version meets pre-migration data. Spiked by hand on
|
||||
> 9202 the same day — §6.1a.
|
||||
|
||||
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6.
|
||||
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6. *(Noted 2026-10-24: stepping is built — the ladder, one tested step per press, decision 14.)*
|
||||
|
||||
### 6.1a The undo (decision 15) — SPIKED BY HAND, then BUILT: controller v0.263.2 (2026-09-23)
|
||||
|
||||
@@ -2130,7 +2140,8 @@ only the live box could show, fixed the same day:** the undo's probe was never a
|
||||
current probe held the app `unhealthy` (v0.263.1), and the "old" `.felhom.yml` taken at update time
|
||||
was already the new one, because `.felhom.yml` flows in on every catalog sync (v0.263.2: the pinned
|
||||
version's file is now recorded in `applied-meta/` whenever a version is pinned). **Residual (R-646):**
|
||||
an app pinned before v0.263.2 has no such record until its next pin. **Two more, found by the chaos hour of
|
||||
an app pinned before v0.263.2 has no such record until its next pin. *(Closed since: `BackfillAppliedMeta` records it at
|
||||
start for every app CURRENT with the catalog, and names the rest — `internal/stacks/undo.go:646-652`; noted 2026-10-24.)* **Two more, found by the chaos hour of
|
||||
2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** the undo finds an app's volumes by their compose
|
||||
label, and a restore recreates them WITHOUT it — so after any restore the undo copies nothing (R-658, P1); and a
|
||||
held file-leg app is pointed at a restore that refuses a database-only copy, with no route left on a box without
|
||||
@@ -2270,7 +2281,7 @@ breaks.
|
||||
|
||||
---
|
||||
|
||||
### 6.2 Slice 6, as it will be built (OPEN — R-450; ruled 2026-09-23, decisions 11–17)
|
||||
### 6.2 Slice 6, as it will be built (~~OPEN — R-450~~ BUILT, R-450 closed 2026-10-05 — noted 2026-10-24; ruled 2026-09-23, decisions 11–17)
|
||||
|
||||
**The shape the rulings fix. The build order and costs are §6.4.** Rewritten 2026-09-23; the
|
||||
earlier "as it would be built" draft keyed on the window of Q1's recommendation and on
|
||||
@@ -2341,7 +2352,7 @@ retried. A working caller in this exact shape exists as evidence, not product:
|
||||
**The per-box switch key is `app_update.unattended`**, default **true** (decision 12).
|
||||
|
||||
⚠ **NOT `auto_update` — that name is TAKEN, and by the very thing this must not collide with.**
|
||||
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281, default
|
||||
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281 then, L278-279 now; default
|
||||
**04:30** at L422) are the CONTROLLER's own update. Two settings with that name, one meaning the
|
||||
controller and one meaning apps, is the kind of collision that is only discovered by an operator who
|
||||
turned off the wrong one. The app-scoped key is `app_update.*`. **And `stacks.update_window` is
|
||||
@@ -2371,12 +2382,12 @@ what the part can do to a household's data if it is wrong, not how likely that i
|
||||
|
||||
| # | part | rulings / rows | cost | depends on | risk to customer data |
|
||||
|---|---|---|---|---|---|
|
||||
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold`: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
|
||||
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold` *(as SHIPPED — noted 2026-10-24: the copy is the FOLDER copy decision 19 chose: every named volume `cp -a` into a sibling volume with a finished-marker, put back on failure; no database load — `internal/stacks/undo.go:26-36`; the text that follows is the design before the bake-off)*: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
|
||||
| **2** | **SHIPPED — controller v0.264.0 + hub v0.120.0, proven live on 9202 and 9201 2026-09-23** (`audits/undo-fleet-2026-09-23/`). **The update sentences in the household's language** (R-606) and a mail when an automatic update is undone or held: events `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, on by default, mailed in the household's language with the app named in the subject; per-app cooldown on both legs. Leftovers: R-647. | R-606, 15 | **1** | — | none |
|
||||
| **3** | **SHIPPED — controller v0.264.0, proven live on 9202 2026-09-23.** **A disabled notifier says so** (R-620), so the mail of part 2 can be measured on a scratch box at all. | R-620 | **0.5** | — | none |
|
||||
| **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/<to>.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only |
|
||||
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
|
||||
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. **The box half (compare, render `name:tag@sha256`) is not built.** **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
|
||||
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). *(Closed since: a step's own `steps/<key>.felhom.yml` is read when present — `ladder.go:127-132`, v0.269.0; noted 2026-10-24.)* **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
|
||||
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. ~~**The box half (compare, render `name:tag@sha256`) is not built.**~~ *Built — the box renders the tested digest, carries it on sync, and the badge reads the tested digest (`internal/stacks/digest.go:15-24`, `updateorder.go`; noted 2026-10-24).* **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
|
||||
| **7** | **SHIPPED — controller v0.271.0 (`9cf13a3`), proven on 9202 over four simulated nights 2026-09-24/25 and watched through the demo boxes' first real automatic night (`audits/DRILL-night-2026-09-25.md`).** Decisions 31–33 taken unattended. Gaps a scratch box cannot close: R-687; no resume after a restart: R-686. *(Before:)* **SPIKED 2026-09-24 (night, Part C) — the measured build brief is §6.4.2 below.** **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has ENDED (on every exit path, skips included), one app at a time, one step per press, `app_update.unattended` default ON (decision 12), `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it, and the full-system backup's gate waits for it until W+5h (decision 20). | 11, 12, 14, 15, 20 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe |
|
||||
| **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none |
|
||||
| **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none |
|
||||
@@ -2459,7 +2470,7 @@ During the db-dump leg every press was refused `busy` (transient, retried next p
|
||||
update's own finish should refresh them.
|
||||
- **An Update pressed on an app already at the head runs the whole guarded update** — backup, pull,
|
||||
restart, `done` — with no refusal (R-679: navidrome restarted four times for nothing). The leg never
|
||||
presses an app whose pin equals the head; the preflight should refuse with reason `current`.
|
||||
presses an app whose pin equals the head; the preflight should refuse with reason `current` (as built the reason key is `already_current`, `update.go:432` — noted 2026-10-24).
|
||||
- **The product does not remember a failed step.** After vikunja's undo the badge still offers the
|
||||
same step; only the caller's memory kept it from a re-press. Decision 15 needs it on the box: the
|
||||
failed `to` is recorded per app and the leg skips it until the catalog's ladder changes.
|
||||
@@ -2541,6 +2552,24 @@ and E are the ones that unblock a decision; leg A is the one that takes the time
|
||||
harness on DooPlex for the image-side ones.
|
||||
|
||||
|
||||
### 6.4.x Decisions in code no section records [DESIGN — undocumented, found 2026-10-24, review round 2 — the operator confirms them through one register row]
|
||||
|
||||
Read at `file:line` on controller `main` (`321df1f`) by the main session:
|
||||
|
||||
- **`build.sh --push` also refuses a DIRTY tree, a HEAD that is not `origin/main`, and a directory that is not a git
|
||||
checkout** — `controller/build.sh:144-166`. §0 names only the tag-exists and unreachable-registry refusals.
|
||||
- **The leg retries a passing refusal ONCE in the same night** (`busy`, `updating`, `deploying`, `migrating`,
|
||||
`self_updating`), after `RetryWait` — not „the next pass/night" (`internal/stacks/unattended.go:65-66`, the pass loop).
|
||||
- **The leg skips an app the household stopped** (`stopped_by_household`, R-721): an update would start it
|
||||
(`unattended.go:416-418`).
|
||||
- **A manual run of the chain** (R-705, the debug action) gives the leg a deadline of its start + the leg's normal
|
||||
length (`backupwindow.UpdateLegLengthMin`), not W+5h (`unattended.go:232-238`).
|
||||
- **Two preflight refusals are plain skips to the leg:** `already_current` and `engine_no_test`
|
||||
(`internal/stacks/update.go:432,454`).
|
||||
- **The render table (§5.4) has one more row:** when the catalog template's images cannot be read, the syncer leaves
|
||||
the app's compose file alone for that cycle (`internal/sync/sync.go:622-627`); and the box's R-971 allow-list runs
|
||||
before ANY compose write (`cmd/controller/main.go:421`, `SetAdmitFn`).
|
||||
|
||||
## 6.5 The drill catalog and the image store — the standing method for update drills
|
||||
|
||||
**Why this section exists.** On 2026-09-21 an afternoon session put a deliberately broken image into
|
||||
@@ -2629,7 +2658,9 @@ customer box has credentials for it. The store listens on the guest's loopback o
|
||||
### 7.1 The record (slice 1)
|
||||
|
||||
`Manager.recordInstalledImages` (`felhom-controller/controller/internal/stacks/installed.go`) runs
|
||||
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`,
|
||||
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`
|
||||
*(as built — noted 2026-10-24: `UpdateStack` is gone; the callers are `StartStack`, `RestartStack`, the deploy path, the
|
||||
end of the guarded update and the end of the undo — `manager.go`, `deploy.go`, `update.go`, `undo.go`)*,
|
||||
and writes `app.yaml`:
|
||||
|
||||
```yaml
|
||||
@@ -2705,7 +2736,7 @@ Version strings stay in the logs, the API and the hub.
|
||||
The 8th, immich's own ghcr build, is UNMEASURED — ghcr exposes no anonymous last-modified. So on
|
||||
demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.
|
||||
Digest-level comparison needs the catalog to record the digest at push time — **R-446**, put to the
|
||||
operator as §3b **Q6**, recommended YES.
|
||||
operator as §3b **Q6**, recommended YES. *(Built since for every ref with a ladder entry — `updateorder.go`, `digest.go`; the limitation remains only for a ref no entry records; noted 2026-10-24.)*
|
||||
2. **~~Nothing enforces `catalog_since`.~~ Enforced by the pre-push hook since 2026-09-13 (R-452, `app-catalog-felhom.eu/scripts/check-catalog-since.py`); CI's shallow clone still skips it out loud.** A commit that moves an `image:` line and forgets the date
|
||||
under-reports how far behind a box is. The gates runner fetches at `--depth 1` and has no parent
|
||||
commit to diff against, so the gate needs a deeper fetch — **R-452**.
|
||||
@@ -2730,13 +2761,18 @@ Version strings stay in the logs, the API and the hub.
|
||||
refuses without a restorable, proven Tier-2 copy, backs up first when that copy is stale, takes a
|
||||
safety dump, and holds an app that does not come up. **What stays true:** it still has no automatic
|
||||
rollback (deliberately, §6.1) and can still attempt a multi-major jump the app will refuse (R-40) —
|
||||
that now ends HELD rather than crash-looping behind a green button.
|
||||
that now ends HELD rather than crash-looping behind a green button. *(Dated note 2026-10-24: a copy on ANY tier now
|
||||
satisfies the precondition (v0.239.0), the box undoes a failed update itself (decision 15, v0.263.2), and the
|
||||
ladder climbs one tested step per press (decision 14) — `internal/backup/update_guard.go`, `internal/stacks/undo.go`,
|
||||
`ladder.go`.)*
|
||||
7. ~~**An engine major can be applied without its datadir upgrade, and nothing notices.**~~ **CLOSED
|
||||
2026-09-13 for MariaDB (R-459):** every `mariadb:` sidecar carries `MARIADB_AUTO_UPGRADE=1`, and the
|
||||
harness shows the conversion running on the E3/E3b edges (§3 decision 5). **What stays true:** the
|
||||
PostgreSQL half (R-463) has no equivalent — the image performs no `pg_upgrade` — and the
|
||||
engine-major rule (§3 precaution 3, R-469) is what keeps both engines inside their major until
|
||||
Slice 4 gives the Update button a backup.
|
||||
Slice 4 gives the Update button a backup. *(Dated note 2026-10-24: MariaDB is lifted — its major is its own edge; a
|
||||
PostgreSQL major may pass with a two-venue proof and an `engine_conversion` mark, and the box converts it
|
||||
(v0.273.0) — `app-catalog-felhom.eu/scripts/check-engine-major.py`.)*
|
||||
8. ~~**Only three of 53 apps have ever had an upgrade measured.**~~ **WIDENED 2026-09-21 to 21
|
||||
EDGES ACROSS 19 APPS** (`audits/DRILL-update-night-2026-09-21.md`), on scratch guest 9202
|
||||
through the product's own guarded Update, against a PRIVATE DRILL CATALOG so the live catalog
|
||||
|
||||
@@ -128,7 +128,9 @@ the hub's e-mails follow.**
|
||||
`expired` pinned to Hungarian so the language cannot become the oracle the text refuses to be.
|
||||
Full design: `05-hub-architecture.md` §15. **R-555 closed** — the `language` allowlist entry is out
|
||||
of `wire_contract_gate.py` and the gate now checks the field for real.
|
||||
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece:** the hub stores a per-customer language, renders it into
|
||||
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece** *(Part B is built
|
||||
too — the controller reads `customer.language` as its start language since v0.256.0, `internal/config/config.go:149-154`;
|
||||
corrected 2026-10-24)*: the hub stores a per-customer language, renders it into
|
||||
`controller.yaml` next to `customer.id/name/domain/email` (`hub/internal/configgen/configgen.go`),
|
||||
and the box uses it **only while the household has never chosen** (`settings.json` empty). The
|
||||
household's own choice always wins; the hub's e-mails follow the language the box **reports**, which
|
||||
@@ -184,7 +186,8 @@ bundles. They now judge the page **as rendered in each language** (`controller/s
|
||||
- **hu** — the product speaks in „te". The converted copy carries formal („ön") forms (R-516). They are
|
||||
not fixed by a localisation release (§1), so the gate counts them against `HU_FORMAL_CEILING` as a
|
||||
ratchet: a new one convicts, fixing one lowers the ceiling. The ceiling followed the conversion:
|
||||
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C). **[FACT] Measured limit:** the stem
|
||||
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C) → … → **0** (R-516 item 12, the last eleven in the
|
||||
te-form — `felhom-controller/controller/scripts/i18n_missing_gate.py:41`; corrected 2026-10-24). **[FACT] Measured limit:** the stem
|
||||
list is short — on the release C pages it counts 4 formal forms, while a wider list („írja be",
|
||||
„adja meg", „válassza", „biztosan eltávolítja", „engedélyezze", …) finds at least 22 keys (R-516).
|
||||
- **en** — second person, plain: no „please", no „kindly".
|
||||
@@ -201,7 +204,8 @@ vissza") — R-564. The hub copy gate and the catalog have no language concept
|
||||
- **[DESIGN] The hub's operator pages** — the operator reads them; they stay English (decision 1).
|
||||
- **[DESIGN] The first-boot wizard** (`controller/internal/setup/`, 8 pages, 95 strings) — out of scope
|
||||
and obsolete (decision 4, 2026-09-17; `02-controller-module-map.md` L56). A household still reaches
|
||||
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**.
|
||||
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**. *(Deleted since:
|
||||
`internal/setup/` is gone (v0.308.0); `bootstrap.WaitForConfig` shows a waiting page instead — corrected 2026-10-24.)*
|
||||
- **[FACT]** App containers' own UIs (Uptime Kuma, PrivateBin…) — not Felhom's copy (R-516 items 5-6).
|
||||
|
||||
---
|
||||
@@ -226,7 +230,7 @@ i18n:
|
||||
```
|
||||
|
||||
Why: one app's copy stays in one file a reviewer sees whole; the controller's `yaml.Unmarshal` ignores
|
||||
unknown keys (`internal/stacks/metadata.go` L316 — **verified: this repo constructs no `yaml.Decoder`
|
||||
unknown keys (`internal/stacks/metadata.go` L316 then, L475 now — **verified: this repo constructs no `yaml.Decoder`
|
||||
at all, so there is no `KnownFields` anywhere to turn strict**), so older controllers are unaffected;
|
||||
a missing English field falls back field by field.
|
||||
|
||||
@@ -234,7 +238,7 @@ a missing English field falls back field by field.
|
||||
reached only through `LocalizeStacks` / `LocalizeStackPtr` / `Stack.MetaFor`.
|
||||
|
||||
- **`For("hu")` is the parsed struct with `I18n` cleared and nothing else**, deep-compared against all
|
||||
53 real catalog files (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
|
||||
53 real catalog files (the identity fixtures are still those 53; the catalog has 65 apps — noted 2026-10-24) (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
|
||||
§1 therefore holds by construction for catalog copy, and `LocalizeStacks` returns the INPUT SLICE
|
||||
for Hungarian rather than a copy, so nothing on the Hungarian path is even touched.
|
||||
- **Fallback is field by field, and blank counts as absent** (ruling 3). A half-translated app is a
|
||||
@@ -307,10 +311,10 @@ for the English page test (R-565), arriving again in a different repo. So
|
||||
`app-catalog-felhom.eu/scripts/check-copy-i18n.py` folds and matches word-bounded stems, with a
|
||||
positive and two negative controls printed on every run.
|
||||
|
||||
**[FACT] The gate**, fifth row of `catalog_gates.py`, static, in the pre-push hook:
|
||||
**[FACT] The gate**, fifth row of `catalog_gates.py` (the sixth since — corrected 2026-10-24), static, in the pre-push hook:
|
||||
|
||||
1. **FREEZE** — every Hungarian copy string equals `scripts/copy_freeze/hu.json`, captured in its own
|
||||
commit before any translation. Runs on all 53 apps whatever scope is named. A new app must be
|
||||
commit before any translation. Runs on every app in the freeze file (65 on 2026-10-24) whatever scope is named. A new app must be
|
||||
admitted with `--add-app NAME --reason "…"`.
|
||||
2. **STRUCTURE** — the `en` block may carry copy fields and nothing else; every key-matched entry must
|
||||
have a Hungarian twin, or it is INERT on the box and the translator never learns.
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
|
||||
> | | |
|
||||
> |---|---|
|
||||
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1).** Ratification is Viktor's review, not an editor's. |
|
||||
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1); steps 2–6 are BUILT by §8's own list (corrected 2026-10-24 — the ratification is still the operator's).** Ratification is Viktor's review, not an editor's. |
|
||||
> | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. |
|
||||
> | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 |
|
||||
> | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. |
|
||||
@@ -79,7 +79,7 @@ boxes installed. Nobody keeps a hand-written list.
|
||||
- **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).
|
||||
- **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in
|
||||
the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since
|
||||
2026-06-11) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
|
||||
2026-06-11 — *`2026.9.3` today, corrected 2026-10-24*) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
|
||||
— the app-image question (`09`), not an OS package. The gap is R-838.
|
||||
|
||||
---
|
||||
@@ -101,9 +101,10 @@ gets the Docker engine that was current when its golden was baked, and keeps it.
|
||||
|
||||
~~**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist
|
||||
(`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.~~
|
||||
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:58`
|
||||
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:107`) and `:194`
|
||||
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:628`). Nothing else.
|
||||
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:68`
|
||||
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:133`) and `:228`
|
||||
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:659`). Nothing else. *(Line numbers
|
||||
corrected 2026-10-24.)*
|
||||
`apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4
|
||||
proposes how to avoid that.
|
||||
|
||||
@@ -250,7 +251,7 @@ approval of that day stays in force (decided by CC unattended — operator may r
|
||||
versions. A broken-into hub can choose an older version or no version. It cannot make a box install a
|
||||
package that the publisher did not sign.
|
||||
|
||||
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]`
|
||||
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]` — *BUILT and installed since (§8.1); the agent's sudoers holds only the `--plan` line, no `--repair-only` (`felhom-agent/configs/felhom-agent.sudoers:337`); noted 2026-10-24*
|
||||
|
||||
`felhom-os-apply` — root-owned (`0755 root:root`), installed by the installer beside `felhom-selfupdate-guarded`;
|
||||
the agent's sudoers gets exactly `/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json` and
|
||||
@@ -315,7 +316,7 @@ PID 1 or `lxc-start`, or a kernel was installed.
|
||||
Decision 96. Before it, the signed `agent_update` replaced only the binary; wrappers, units and sudoers lines reached
|
||||
an installed box by reinstall or by hand.
|
||||
|
||||
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22: sudoers ×2, the
|
||||
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22 then, **32** counted 2026-10-24 — plus the vzdump-partial-clean, thin-check, internal-place and priv-apply wrappers, the guest hook, the shared-parent unit, the GRUB one-shot pair and the sshd root-key-only drop-in; sudoers ×2, the
|
||||
five wrappers, the crash guard and its units + config, the agent and rollback units, the start-limit drop-in, the mgmt
|
||||
watchdog and its tmpfiles/units, the OOB belt's four files). `scripts/build-config-bundle.py` builds the bundle from it
|
||||
reproducibly; `release-agent.sh` publishes it beside the binary; the hub vouches its sha with the agent (exact-name
|
||||
@@ -393,6 +394,9 @@ must never overlap a backup, a restore-test or a self-update.~~
|
||||
- **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release,
|
||||
how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest
|
||||
approved release raises an alarm (`08`).
|
||||
- **[FACT, added 2026-10-24 — hub v0.169.0] The System page's „Upload line" cell** shows the best of the last SUCCESSFUL
|
||||
weekly line test (R-1247), amber when it is older than 15 days or was never measured after an attempt — judged on
|
||||
`last_ok_at`, never on the attempt time (`hub/internal/web/system.go` `lineStaleAfter`, `lineTestCell`).
|
||||
- **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with
|
||||
buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held /
|
||||
reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed,
|
||||
@@ -480,7 +484,9 @@ Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-do
|
||||
**What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve,
|
||||
libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on
|
||||
2026-10-07. **What it never touches:** a kernel, boot, firmware or microcode name (`HOST_SLOW_RE`: `proxmox-kernel*`,
|
||||
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`)
|
||||
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`,
|
||||
and — read 2026-10-24 in `felhom-os-apply` — `linux-(image|headers|kbuild|modules|base)*`, `firmware-*`,
|
||||
`proxmox-secure-boot-support`)
|
||||
— that is R-836's lane (decision 164). No reboot.
|
||||
|
||||
- **Wrapper** (`felhom-agent/configs/felhom-os-apply`): layer `pve`, lane `slow` only. Origin `Proxmox Debian
|
||||
|
||||
@@ -78,6 +78,13 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
|
||||
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971). **Re-check (2026-10-24, review round 2):** the box checks every rendered compose against its allow-list before any write since controller v0.328.0 (`internal/privpolicy/privpolicy.go`) |
|
||||
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (deployed 2026-10-24 — corrected 2026-10-24; `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, released 2026-10-24 in controller v0.342.0 + hub v0.169.0 — corrected 2026-10-24) detects |
|
||||
|
||||
**[DESIGN — undocumented in `01` §5, found 2026-10-24, review round 2 — confirmed through R-1265]:** the file manager's one
|
||||
sign-in (R-1100) — FileBrowser trusts its proxy header blindly, and every app container can reach `filebrowser:80`
|
||||
directly; the only guard is that the header NAME is a per-box secret, which traefik's forwardAuth strips from every
|
||||
client request and sets only for a browser holding the dashboard's files pass
|
||||
(`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`, `internal/web/files_sso.go`). The name file's
|
||||
creation race was fixed by review round 2 (controller `b01bb4b`).
|
||||
|
||||
## 6. What we accept (do not re-report)
|
||||
|
||||
- **Cloudflare can read tunnel traffic** — R-904, `09` §3 decision 184.
|
||||
|
||||
@@ -22,7 +22,7 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
|
||||
| 2 | controller `POST /api/server/poweroff` (`internal/web/poweroff_handlers.go`) | dashboard login (`RequireAuth` + CSRF), POST only, the word again, the controller's own backup/restore/app-update check, the `host_poweroff` feature gate (MinAgent 0.155.0) |
|
||||
| 3 | agent `POST /host/poweroff` (`internal/localapi/hostpoweroff.go`) | `withGuest` token scope; the guards (§3); the record (`internal/poweroff`) |
|
||||
| 4 | agent → hub | a host report carrying `deliberate_off.requested_at`, waited for until the hub ACCEPTED it (≤ 60 s, `Loop.SetReportedHook`) |
|
||||
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors, `FELHOM_POWEROFF`) |
|
||||
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors: `pct ^shutdown [0-9]+ --timeout [0-9]+$` in `FELHOM_GUESTHOOK`, and `systemctl poweroff` in `FELHOM_POWEROFF` — `felhom-agent/configs/felhom-agent.sudoers:81-90`; corrected 2026-10-24) |
|
||||
| 6 | hub (`store/deliberateoff.go`) | `hosts.deliberate_off_at`; silence alarms held 14 days (§4) |
|
||||
| 7 | the household | presses the machine's power button; the first report of the new boot has no `deliberate_off` → the hub clears it |
|
||||
|
||||
@@ -33,11 +33,11 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
|
||||
| first 10 min after boot | `too_soon_after_boot` | `TooSoonAfterBoot` |
|
||||
| one accepted request per boot (file keyed by `/proc/stat` btime) | `already_requested_this_boot` | `SecondRequestSameBootRefused`, `poweroff/state_test.go` |
|
||||
| any other guest or VM running (`/proc`: `lxc-start -n N`, `kvm -id N`) | `other_guests_running` | `RefusedWhileOtherGuestsRun`, `RunningGuestsFromProc` |
|
||||
| heavy-operation gate held (backup, restore test, OS step, sweep, trim) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
|
||||
| heavy-operation gate held (backup, restore test, OS step, sweep, trim, and since agent v0.164.0 the weekly upload line test — corrected 2026-10-24) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
|
||||
| backup job, controller swap, drive format, NAS verify, escrow ceremony, Proxmox vzdump | `backup_running` / `controller_update_running` / `drive_operation_running` / `escrow_ceremony_running` | `RefusedDuringVzdump`, `FailsClosedWithoutTaskReader` |
|
||||
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
|
||||
|
||||
**„A Proxmox vzdump runs” (agent v0.156.1, R-1008):** a vzdump task for the guest in Proxmox's active list whose worker process still lives — Proxmox's own rule (the UPID's pid + pstart against `/proc/<pid>/stat`). After a power cut the dead task stays listed for ~30–60 s; it no longer counts. Doubt (an unparsable task, an unreadable `/proc`) still refuses. The same answer serves the start-up stale-lock pass, the guest-power watchdog and the controller supervisor (`stalelock_inflight_test.go`).
|
||||
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
|
||||
|
||||
**[DESIGN]** A cancelled power-off leaves the box on (gate released) — better a box that did not switch off than a
|
||||
backup, update or format cut in the middle. **[DESIGN]** A failed guest shutdown does not stop the power-off: Proxmox's
|
||||
@@ -86,5 +86,15 @@ alternative (an out-of-band confirmation) needs a second channel the box does no
|
||||
|
||||
## 6. Not done here
|
||||
|
||||
- The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).
|
||||
- ~~The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).~~ Published — `website/gyik.html:128,603` (corrected 2026-10-24).
|
||||
- The BIOS „after power loss" value of the N100 was not read (no monitor); the household text promises only the button.
|
||||
|
||||
## Decisions in code no section records [DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through one register row]
|
||||
|
||||
- **A re-enroll while the household power-off hold is active counts as „gone"** (a reinstall over the old host), not
|
||||
„live" (R-1154; `hub/internal/api/reenroll.go:67-68`).
|
||||
- **Setting and clearing the deliberate-off time each record an info event** — `host_deliberate_off`,
|
||||
`host_deliberate_off_ended` — on the customer's event list; neither is mailed (`hub/internal/api/deliberate_off.go:12-13`).
|
||||
- **The confirmation word is accepted in three forms:** „KIKAPCSOLÁS", the unaccented „KIKAPCSOLAS" (a foreign keyboard)
|
||||
and „TURN OFF" (`felhom-controller/controller/internal/web/poweroff_handlers.go:28-29`).
|
||||
|
||||
|
||||
@@ -165,7 +165,8 @@ directory) with that tag. Facts the link code needs:
|
||||
**[FACT]** An unlisted writable `${…_PATH}` bind is MANDATORY (every tier, off-site included); a plain host path is in
|
||||
no capture set (`stacks/classify_binds.go` `classifyRoot`). Measured over the live templates: Transmission's unlisted
|
||||
`watch` bind goes off-site; the download folders are `optional`, so a second drive gets a nightly copy of every
|
||||
download (R-1019). **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
|
||||
download (R-1019). *(Fixed since: catalog `c50adfc` marks the download and watch folders `excluded` —
|
||||
`app-catalog-felhom.eu/templates/transmission/.felhom.yml:46-51`; noted 2026-10-24.)* **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
|
||||
media is no single app's data; whether the library gets its own household-level backup is a separate decision.
|
||||
|
||||
## 8. Removal and export assume one drive per app
|
||||
@@ -192,6 +193,13 @@ household's existing files under `media/torrents` stay where they are.
|
||||
Evidence: `audits/media-drive-events-2026-10-14/` (K1–K7). „Idempotent" = a second run changes nothing (K2: run 2 left every entry, folder and library as it
|
||||
was). „Household" = a step only the household can take.
|
||||
|
||||
> *Dated note 2026-10-24 (review round 2): this table is the 2026-10-14 MEASUREMENT („today" = then). As built (§11):
|
||||
> on a drive event the controller only REWRITES the binds; the media apps restart when the household presses
|
||||
> „Újraindítás most" or „Később" (03:30) — the file browser alone recreates itself (`internal/stacks/media_library.go`,
|
||||
> `internal/web/media_events.go`); a safe removal keeps the drive REGISTERED and its return re-binds with no household
|
||||
> step (`SetEjected`, `media_events.go`); and since controller `b01bb4b` (unreleased) neither restart recreates an app
|
||||
> the household stopped (R-721).*
|
||||
|
||||
| event → app | action | measured call | idempotent | household | evidence |
|
||||
|---|---|---|---|---|---|
|
||||
| **added** — every media app | the compose gains the drive's bind → `up -d` recreates it (the compose CHANGED, so today's restart works) | controller compose `up -d` | yes | no | K5-1: 1–10 s per app |
|
||||
@@ -258,3 +266,9 @@ save path when its client is saved before the wiring; **a Plex status probe (`/l
|
||||
unclaimed Plex because Plex answers 401 for a moment while it starts — never gate on a status a starting app also
|
||||
returns** (G7f).
|
||||
|
||||
|
||||
|
||||
**[DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through one register row]:** the „Később"
|
||||
03:30 media restart is an IN-MEMORY timer — a controller restart forgets it and only the notice stays
|
||||
(`internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`); the Sonarr/Radarr API key is READ from `/config/config.xml`, the „set at
|
||||
install" option §6 calls cleaner was not built (`internal/mediawire/mediawire.go:336-345`).
|
||||
|
||||
@@ -15,8 +15,8 @@
|
||||
| Place | Found by | Identified by | Mounted | What lives there | The backup page says |
|
||||
|---|---|---|---|---|---|
|
||||
| Host root / PVE `local` | PVE storage `local` [FACT] A/internal/storage/observe.go:424; never a candidate [FACT] A/internal/storage/role.go:53-79 | `path:` / `store:local` [FACT] A/internal/storage/durableid.go:54-67; role system | host `/`, `/var/lib/vz` | the **local whole-guest copies** (vzdump, retention 3) [FACT] E/scripts/felhom-host-install.sh:730-734; `07` §6.1 | `backup.target.degraded` „…ugyanazon a lemezen van, mint a rendszer…" [FACT] C/internal/web/backup_target_offer.go:78-100 |
|
||||
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — a Go constant, not i18n [FACT] backup.go:359-361, :1387-1410 |
|
||||
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: **whole disks only** [FACT] A/internal/storage/candidates.go:33-53 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
|
||||
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — *corrected 2026-10-24:* the i18n keys `note.backup.single_copy` / `note.backup.single_copy_unproven`, picked by the agent's verdict [FACT] C/internal/backup/backup.go:1485-1494 |
|
||||
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: whole disks, each carrying its nested partition candidates (empty / filesystem / in use; corrected 2026-10-24) [FACT] A/internal/storage/candidates.go:84-90 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
|
||||
| Network share | customer input [FACT] A/internal/localapi/netstorage.go:36-40 | none („no durable-id, no SMART") [FACT] netstorage.go:15-19 | `.automount` under `/mnt/felhom-drives/<name>` [FACT] A/internal/storage/netmount.go:18-31 | browse only; never media, never Tier-2 [FACT] media_library.go:83; tier2.go:141-143 | `err.backup.halozati_tarolo_nem_lehet_a_2` [FACT] tier2.go:37-40 |
|
||||
| USB disk | same as an enrolled drive; transport never decides [FACT] C/internal/settings/settings.go:609-614 | same; the bridge may pass the real identity (N100: yes) [FACT] spike case 4a | same | same | same; SMART may be missing → „Nincs adat" |
|
||||
|
||||
@@ -126,7 +126,7 @@ binds both. **Creating or deleting partitions: never from the dashboard in this
|
||||
## 7. The installer
|
||||
|
||||
**[FACT]** Today `step_grows` gives the guest's one data volume (`mp0`) 70 / 130 / 250 GiB by the thin pool's free space
|
||||
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh:2261-2275). Measured pools: 128 GB
|
||||
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh `step_grows`, now at :2475 — line corrected 2026-10-24). Measured pools: 128 GB
|
||||
disk → 54 GiB, 200 GB → 113 GiB, 512 GB → 349 GiB (PVE default layout; root capped at 96 GiB).
|
||||
|
||||
**[DESIGN] The rule:** `mp0` = 70 GiB (pool < 300), 130 (300–800), 250 (≥ 800). Reserve = **max(16 GiB, 10 % of the
|
||||
@@ -231,10 +231,25 @@ sizes ruled above in §7):
|
||||
the public installer 1.33.0, agent 0.158.1 and golden 0.318.0 vouched) → data volume 100 GiB, internal place 213 GiB,
|
||||
registered as the default drive with no press; Jellyfin installed with no extra step onto it; a whole-box copy holds nothing
|
||||
of it; partitions formatted and used through the wizard; the honest backup offer on a VM. **Not as written:** an empty part
|
||||
asks for the typed confirmation (the probe reads the parent's partition table — R-1093); a 200 GB disk gets no internal
|
||||
asks for the typed confirmation (the probe reads the parent's partition table — R-1093) *(fixed since in agent v0.160.0:
|
||||
the parent's PTTYPE is set aside when `blkid -p` proves the node is a partition of that scheme — `git tag --contains
|
||||
f7b7308`; noted 2026-10-24)*; a 200 GB disk gets no internal
|
||||
place under the ruled sizes (15 GiB < 20).
|
||||
|
||||
Also in build B: R-1084 (the whole-box backup offer asks the same-disk question) and R-1082 (`/mnt/sys_drive` is never
|
||||
offered to attach, by device). A bundle that adds a path needs a step bundle on boxes with an older wrapper (R16, R-880):
|
||||
0.158.1 was delivered through `0.158.1-step1`.
|
||||
|
||||
## Decisions in code no section records [DESIGN — undocumented, found 2026-10-24, review round 2 — confirmed through one register row]
|
||||
|
||||
Read at `file:line` on `main` by the main session:
|
||||
|
||||
- **`systemd-detect-virt --vm` is read once per agent process**; empty output becomes `unknown`, which is not `none`,
|
||||
so every disk then reads unknown/vm — the fail-closed direction (`felhom-agent/internal/storage/identity.go:195-205`).
|
||||
- **A path the controller cannot read counts as the SAME disk** (R-668, v0.269.0): no second copy is ever claimed on
|
||||
it (`felhom-controller/controller/internal/backup/backup.go:1436-1450`).
|
||||
- **The internal place („Belső tárhely") refuses eject, decommission and wipe** — the controller (`refuseInternalLifecycle`)
|
||||
and the agent both — and allows moving its data to another drive (`internal/web/internal_place.go:20-26`).
|
||||
- **The internal place's size subtracts EVERY thin volume already promised in the pool**, not only the 32 GiB rootfs and
|
||||
the data volume (`felhom-agent/configs/felhom-internal-place:11-14`).
|
||||
|
||||
|
||||
@@ -5,6 +5,21 @@
|
||||
> No matrix is designed here, no target architecture is proposed, and no gap is ranked.
|
||||
>
|
||||
> **This is the input to a redesign, not the redesign.**
|
||||
>
|
||||
> **WHAT HAS MOVED SINCE (banner added 2026-10-24, review round 2 — this inventory records 2026-07-28 and is kept as
|
||||
> it was; each line below was read at `file:line` on `main` that day):**
|
||||
> - The guest has ONE data volume, `mp0` at `/var/lib/felhom`, whose `docker/` and `sys_drive/` folders are bound in the
|
||||
> guest; there is no `mp1` (`felhom-agent/configs/build-golden.sh:31-38,99`) — the „rootfs + mp0 + mp1" lines are history.
|
||||
> - A schema-2 recovery unit (D5) carries the PORTABLE secrets; a restore takes the database password from the unit
|
||||
> (R-1106) — `felhom-controller/controller/internal/backup/recovery_unit.go:24-27`, `restore_dbcreds.go`.
|
||||
> - The second-drive copy's `recovery-unit/` IS read: `RestoreTier2Unit` restores the full unit from the mirror (R-102)
|
||||
> — `internal/backup/tier2_restore.go:197`.
|
||||
> - The off-site reconstitution replays the snapshot's named-volume tars, volumes first (R-354) —
|
||||
> `internal/backup/offbox_reconstitute.go`.
|
||||
> - ep0's per-household prune jobs are keep-weekly 2 at 03:30 (R-1162/R-1165, `scripts/felhom-tenantsync.sh:187-197`);
|
||||
> the local whole-guest tier is keep-daily=3 (R-1149, `felhom-agent/internal/config/config.go:553-559`).
|
||||
> - restic is the pinned official release, checked by version and checksum (R-1105, `felhom-controller/controller/Dockerfile:43-46`),
|
||||
> no longer Debian's 0.14.0.
|
||||
|
||||
## 0. Method, baselines, and what "verified" means here
|
||||
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
Sat Oct 24 14:05:27 UTC 2026
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.342.0
|
||||
repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 6 hours (healthy)
|
||||
filebrowser gtstef/filebrowser:1.5.6-stable Up 23 hours (healthy)
|
||||
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up About an hour (healthy)
|
||||
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up About an hour (healthy)
|
||||
immich-redis redis:7-alpine Up About an hour (healthy)
|
||||
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up About an hour (healthy)
|
||||
paperless-postgres postgres:18-alpine Up About an hour (healthy)
|
||||
paperless-redis redis:7-alpine Up About an hour (healthy)
|
||||
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up About an hour (healthy)
|
||||
traefik traefik:v3.7.13 Up 23 hours
|
||||
324af98a77bf1b5a6e0427187c6e1991 -
|
||||
3
|
||||
/dev/loop0 32G 2.7G 27G 10% /
|
||||
/dev/nvme0n1 938G 239G 652G 27% /mnt/felhom-drives/scratch_hdd
|
||||
@@ -94,3 +94,102 @@ Rules, fences, „never" lines and promises were NOT edited — those are rows f
|
||||
- [S18] R-970 „does not hold" → the routes are removed, pinned by `r970_no_hub_key_door_test.go`.
|
||||
- [S19] R-971 „design gap" → `privpolicy.Admit` before every write (controller v0.328.0).
|
||||
- [S20] „hub 0.168.0 deploy pending", „R-1244 not released" → deployed / released 2026-10-24.
|
||||
|
||||
## 07-backup-architecture.md (DOC-D1)
|
||||
- [S2/C3] Tier-1 „secret-stripped app.yaml" → carries the portable secrets (D5) — `felhom-controller/controller/internal/backup/recovery_unit.go:24-27`.
|
||||
- [S7] `main.go:604-607` → `backupwindow.LegTimes` at `cmd/controller/main.go:1258-1264`.
|
||||
- [S9] demo-hp „keep-last=1" → a dated note: keep-daily=N since R-1149 — `felhom-agent/internal/config/config.go:553-559`.
|
||||
- [S1] D22 „controller main, unreleased" → released v0.338.0.
|
||||
- [S11] §6.6 table → a row: the off-site restore also refuses a mismatched/mixed snapshot first — `internal/backup/offbox_reconstitute.go:826-833`.
|
||||
- [S8] line citations moved (`GetAppDrivePath`, F5); `mp1` → a dated note (gone).
|
||||
- [S3/C1] row 15 R-104 „open DEFECT" ×2 → FIXED 2026-10-05, v0.297.0 — `internal/backup/offbox.go:156-162`; CLOSED-ITEMS.md:895.
|
||||
- [S6] row 10 R-893 / R-973 „unreleased" → released; „the next slice on R-893" → note: D20 (v0.336.0), D22 (v0.338.0).
|
||||
- [S5] „Hub v0.158.0 … deploy waits" → deployed 2026-10-18 12:16 (hub CHANGELOG:288).
|
||||
- [S10] „a restore is refused while any second-drive copy runs" → a dated note: the kept-data load checks only RestoreStatus/IsRunning (`internal/api/kept_install.go:89-92`) — a row.
|
||||
- [U1/U2/U6 + hub U2/U3 + U7 → §8.2c, DESIGN — undocumented notes, one row for the operator]: the 60-min copy limit and stray handling; Eltávolítás refused during backups; start-after-put-back failure; the hub's off-site delivery checker (`offsite_delivery_stuck`, `maybeHeal`, the R-1230 skip — `hub/internal/monitor/offsite_delivery.go:103-110`); the release copy (D33) not in the matrix.
|
||||
- NOT edited here: C4 (a stale code comment in `felhom-agent/internal/localapi/backup_cap.go:15`) — a row; C2 (G15 „nightly") — corrected in the capability map below.
|
||||
|
||||
## 09-update-architecture.md (DOC-C2) — dated notes; no §3 ruling text edited
|
||||
- [S1/S2/S3] §1 (the 2026-09 measurement) → one dated note: `UpdateStack` gone, `StartGuardedUpdate`; moved line numbers.
|
||||
- [S16] `stacks.update_window` „no Go code reads it" (inside ruling 11) → a dated note: removed.
|
||||
- [S17] `unattended.go:433–435` → `:449-451`.
|
||||
- [S4] §5.3 writers → note: `advancePinTo`; `pinBack`, the undo's `restoreDefinition`.
|
||||
- [S7/C1] slice 6 „OPEN — R-450" ×2 → BUILT, R-450 closed 2026-10-05.
|
||||
- [S12] R-646 residual → closed by `BackfillAppliedMeta` — `internal/stacks/undo.go:646-652`.
|
||||
- [S18] `config.go L280-281` → L278-279.
|
||||
- [S9] part 1 „DB up alone … empty-then-load" → note: the shipped folder copy — `undo.go:26-36`.
|
||||
- [S10] „a step has no .felhom.yml of its own" → closed — `ladder.go:127-132`.
|
||||
- [S11] „the box half … is not built" → built — `digest.go:15-24`.
|
||||
- [S19] reason `current` → `already_current` — `update.go:432`.
|
||||
- [S14] digest comparison „R-446 §3b Q6" → built for every ref with a ladder entry.
|
||||
- [S13] §8.6 / „stepping is slice 6" → notes: any tier, the undo, the ladder.
|
||||
- [S15] engine-major „until Slice 4" → note: MariaDB lifted, PostgreSQL by two-venue proof.
|
||||
- [S5] recordInstalledImages callers → note: the real callers.
|
||||
- [S8] §6.1 phase table → the bind-data tier order; a `converting` row (v0.273.0, `pgconvert.go`).
|
||||
- [U1–U6, S6 → §6.4.x, DESIGN — undocumented notes]: build.sh's dirty/HEAD/non-git refusals (`controller/build.sh:144-166`); the once-retry; stopped_by_household; the manual leg's deadline; already_current/engine_no_test; the unreadable-template render row and R-971 admission.
|
||||
- NOT edited (RULINGS — rows for the operator): C4 decision 13's „CompareImageRefs moves to the catalog gate" (not built); C5 decision 14's „never jumps" vs a person's press on an app older than the ladder; C7 decision 250 item 5's „by its AGE" vs the gate's proxy.
|
||||
|
||||
## 00-capability-map.md
|
||||
- 20 PROVEN-LIVE rows marked „re-check (2026-10-24)" in their Scope cell, each with the commit that changed its mechanism after its evidence (status words unchanged): a7, a8, a10, a11, a17, b1, b2, b7, b11, c3, c4, c5, c6, d6, d10, d11, e17, g7, g15, g18. Each commit's files read with `git show --stat`.
|
||||
- [DOC-D1 C2] G15 „a nightly consistent copy" → every 6 hours since D34.
|
||||
|
||||
## 16-storage-physical-identity.md (DOC-D2)
|
||||
- [S2] the single-copy notice „a Go constant, not i18n" → the i18n keys — `felhom-controller/controller/internal/backup/backup.go:1485-1494`.
|
||||
- [S3] `/disks/candidates` „whole disks only" → with nested partition candidates — `felhom-agent/internal/storage/candidates.go:84-90`.
|
||||
- [S4] `felhom-host-install.sh:2261-2275` → `step_grows` at `:2475`.
|
||||
- [S1] the R-1093 „not as written" → a note: fixed in agent v0.160.0 (`git tag --contains f7b7308`).
|
||||
- [U4–U7 → DESIGN — undocumented notes]: detect-virt once; unreadable = same disk (R-668); the internal place's lifecycle refusals; its size rule.
|
||||
|
||||
## 15-media-library.md (DOC-D2)
|
||||
- [S5/S6/C2] the §10 measurement table → one dated note before it: binds rewritten, restart on the household's press, the drive kept registered, and `b01bb4b`'s stopped-app rule.
|
||||
- [S7] Transmission's watch/download folders → note: `excluded` since catalog `c50adfc` (`templates/transmission/.felhom.yml:46-51`).
|
||||
- [U1/U3 → DESIGN — undocumented note]: the „Később" timer is in memory (`media_events.go` `handleMediaRestart`); the Sonarr/Radarr key read from `config.xml` (`mediawire.go:336-345`).
|
||||
- [U2 — FIXED in code, not a note]: a media restart started a household-stopped app → controller `b01bb4b`.
|
||||
|
||||
## _recovery-inventory-2026-07-28.md (DOC-D2 S8–S13) — a WHAT HAS MOVED SINCE banner; the inventory body kept as its date's record
|
||||
- one `mp0`, no `mp1` (`felhom-agent/configs/build-golden.sh:31-38,99`); portable secrets in the unit; `RestoreTier2Unit` (`tier2_restore.go:197`); volume tars replayed (R-354); ep0 keep-weekly 2 at 03:30 (`scripts/felhom-tenantsync.sh:187-197`); local keep-daily=3; restic pinned 0.19.1 (`Dockerfile:43-52`).
|
||||
- runbooks/RUNBOOK-manual-guest-restore.md [DOC-D2 C3]: a dated note after the example config — one `mp0` at /var/lib/felhom, no `mp1` (golden v2.1.0+); the procedure unchanged.
|
||||
|
||||
## 14-host-power-off.md (DOC-E)
|
||||
- [S2] „two exact sudoers vectors, FELHOM_POWEROFF" → `pct shutdown` in FELHOM_GUESTHOOK + `systemctl poweroff` in FELHOM_POWEROFF — `felhom-agent/configs/felhom-agent.sudoers:81-90`.
|
||||
- [S1] heavy-gate holders → plus the line test (agent v0.164.0).
|
||||
- [S3] the website FAQ line „a later task" → published — `website/gyik.html:128,603`.
|
||||
- [S4] a paragraph that split the §3 table → moved below the table (rendering only).
|
||||
- [U1–U3 → DESIGN — undocumented notes]: a re-enroll during the hold is „gone" (`hub/internal/api/reenroll.go:67-68`); the two info events; the three confirmation words.
|
||||
|
||||
## 11-os-updates.md (DOC-E)
|
||||
- [S8] status „§8 step 2 BUILT" → steps 2–6 built by §8's own list (the ratification left to the operator).
|
||||
- [S5] cloudflared `2026.6.0` → `2026.9.3` today — `felhom-controller/controller/internal/infra/infra.go:26`.
|
||||
- [S6] apt sudoers/caller line numbers → `sudoers:68,228`, `lanresolver.go:133`, `wgtunnel/manager.go:659`.
|
||||
- [S7] §5.4.1 „DRAFT … nothing installed" → note: built; sudoers holds only `--plan` (`sudoers:337`).
|
||||
- [S9] BUNDLE_FILES 22 → 32 (counted in `felhom-agent/configs/felhom-os-apply`).
|
||||
- [S10] HOST_SLOW_RE list → plus `linux-*`, `firmware-*`, `proxmox-secure-boot-support` (read from `felhom-os-apply`).
|
||||
- [DOC-E U5 / DOC-C1 U5] §5.7 → the System page's „Upload line" cell (`hub/internal/web/system.go` `lineStaleAfter`).
|
||||
|
||||
## 10-localisation.md (DOC-E)
|
||||
- [S12] „Part B is the remaining piece" → built (v0.256.0) — `felhom-controller/controller/internal/config/config.go:149-154`.
|
||||
- [S13] formal-form ceiling 16 → 0 — `felhom-controller/controller/scripts/i18n_missing_gate.py:41`.
|
||||
- [S11/C1] the setup wizard „still reachable … R-554" → deleted (`internal/setup` absent; `bootstrap.WaitForConfig`).
|
||||
- [S15] `metadata.go L316` → L475.
|
||||
- [S14] „53 real catalog files" (fixtures, still 53) / „fifth row" → sixth / „all 53 apps" → the freeze file's 65.
|
||||
|
||||
## 02-controller-module-map.md (DOC-E) — dated notes; the v0.33 plan tables kept
|
||||
- [S16/C3] two in-flight markers → four (`update-journal.json`, `.felhom-install-pending`).
|
||||
- [S17] three StartGate holders → four (1b `IsUpdating`, `cmd/controller/main.go:2459-2477`).
|
||||
- [S18] `GetAlerts` → `GetBannerAlerts(page, lang)` (R-906) — `internal/web/alerts.go:353`.
|
||||
- [S19] „no longer calls the Cloudflare API" → note: `cloudflare/` still wired with `cf_api_token` (`main.go:1992-2000`).
|
||||
- [S20] `state.go`/`updater.go` DELETE → note: both live, `AgentSwapper` (`internal/selfupdate/updater.go:38-47`).
|
||||
- [S21] `sync.go:319`, `config.go:351` → `:436`, `:399`.
|
||||
- [S22] `locked_after_deploy` „read by no code" → note: read into `LockedFields`, enforced only by an uncalled function.
|
||||
- [C2] §2 `UpdateStack` → note: deleted in v0.237.0.
|
||||
- [U4] §1 → a note naming the 22 packages with no row (an `ls` of `controller/internal/`).
|
||||
|
||||
## felhom-controller/REUSE.md (DOC-D1 C5/U8, DOC-C2 C3)
|
||||
- `CrashPutBack … (D22 / R-1195, unreleased)` → `v0.338.0`.
|
||||
- `recordInstalledImages` callers `StartStack/RestartStack/UpdateStack/runComposeDeploy` → the real callers; `UpdateStack` deleted in v0.237.0.
|
||||
- A new row: `Tier2CopyBlocksRestore` / `ClearLeftoverTier2Copies` / `MarkTier2CopyForTest` + the `tier2Docker` seam (`internal/backup/tier2_copy_lifecycle.go:85,255,319,341`).
|
||||
|
||||
## Late additions (the disposition check before the README)
|
||||
- 03 §6 [DOC-B U1–U5, U8; DOC-A U4 → DESIGN — undocumented note]: 41 routes (24 unlisted), the controller-started line test, the island bridge, the line-test state and switch, the WG block's OOB fields, the D30 count in memory.
|
||||
- 13 §5 [DOC-A U1 → DESIGN — undocumented note]: the file manager's one sign-in trust (`filebrowser_sso.go:15-22`).
|
||||
- 08 §9 item 6 [SEC-1 X1]: the limits list names „any viewer explains any upload" (R-1262).
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# Review round 2 — the architecture documents against the code, and a fresh data-loss and security look (2026-10-24, unattended)
|
||||
|
||||
**Baselines (read at start, 14:42 CEST):** felhom.eu `44d19042`, felhom-controller `321df1f` (v0.342.0 released;
|
||||
„next v0.343.0" unreleased on top), felhom-agent `5e05627` (v0.164.0; „next v0.165.0"), app-catalog `5e196e1`, hub
|
||||
deployed 0.169.0. Register: 190 rows.
|
||||
|
||||
**Method.** Thirteen read-only reviewers (each prompt carried the brief's fences word for word —
|
||||
`reviewers/FENCES-given-to-every-reviewer.md`): seven on the documents (the brief's four groups, two of them split, plus a seventh for
|
||||
`02`/`10`/`11`/`14`, which the groups did not name), four on data loss (restore and put-back; delete and retention;
|
||||
keys; time and concurrency), two on security (hub + controller; agent + ep0 + installer + the DooPlex→ep0 jobs). Their
|
||||
answers are CANDIDATES (`reviewers/`). The main session read every item it acted on at `file:line`; the edits and
|
||||
their evidence are in `EDITS.md`.
|
||||
|
||||
## Part 1 — the documents (claims checked by the reviewers: 493; stale reported: 121)
|
||||
|
||||
| Document | Verdict |
|
||||
|---|---|
|
||||
| `01` topology & trust | 7 stale facts corrected; 2 product statements contradict the code → R-1266 (D45) |
|
||||
| `03` host agent | 9 corrected (shipped releases, key files, listen address, swap, cloudflared's 4th state); a note: 41 local-API routes, 24 unlisted; break-glass text → R-1266 |
|
||||
| `04` control-plane authorization | 3 corrected; a note: a signed op's window has no length cap → R-1265 |
|
||||
| `05` hub | 9 corrected (the 6-hourly DB copy, version literals, the fifth bind-link path, 44 event types, the retired DR record) |
|
||||
| `06` off-site connectivity | 7 dated notes (the design-of-record kept); „no inbound listener" → R-1266 |
|
||||
| `07` backup | 12 corrected (portable secrets, R-104 fixed, keep-daily, D22/D20 released, a refusal row); §8.2c notes → R-1265; the kept-data load beside a copy → R-1256 |
|
||||
| `08` alarm ladder | 10 corrected (per-type cooldowns, R-723, per-app grain, R-998 skip, §8.1 built, §9 released, the operator-only box alarms); one ruling line kept with a dated note (operator Q1 made `backup_integrity_failed` a default) |
|
||||
| `09` updates | 19 dated notes and corrections (the §1 measurement, slice 6 built, the undo as shipped, digests, phase table); §6.4.x notes → R-1265; three rulings the code does not do as written → R-1266 |
|
||||
| `10` localisation | 7 corrected (Part B built, ceiling 0, the wizard deleted, counts) |
|
||||
| `11` OS updates | 7 corrected (cloudflared 2026.9.3, sudoers lines, the wrapper built, 32 bundle files, HOST_SLOW_RE); the System page's „Upload line" cell added |
|
||||
| `12` agent tooling | 3 corrected; the five rule copies aligned (the instruction-file edits below) |
|
||||
| `13` threat model | 11 facts corrected, every verdict cell left as it was and marked „re-check"; a note on the file manager's sign-in trust |
|
||||
| `14` power-off | 4 corrected (sudoers vectors, line-test holder, FAQ published, the table split); notes → R-1265 |
|
||||
| `15` media library | a dated note on the §10 measurement table; R-1019's fix; notes → R-1265 |
|
||||
| `16` storage identity | 4 corrected; notes → R-1265 |
|
||||
| `02` controller module map | 8 dated notes (markers, holders, banner alerts, cloudflare still wired, updater live, LockedFields, 22 packages with no row) |
|
||||
| `_hub-review.md` / `_recovery-inventory-2026-07-28.md` | a HISTORICAL banner / a „what has moved since" banner — both dated records kept as they were |
|
||||
| `00` capability map | 20 PROVEN-LIVE rows marked „re-check" with the commit that changed their mechanism (no status lowered); G15's „nightly" corrected |
|
||||
| `felhom-system-poster` facts + html | the timeline's hub-DB push (text only; no drawing needed) |
|
||||
|
||||
**Rules, fences and promises were not edited.** Seven items that need the operator are in R-1266 (D45); about 30
|
||||
behaviours with no document are written as `[DESIGN — undocumented, found 2026-10-24]` notes for the operator to confirm
|
||||
(R-1265, D44).
|
||||
|
||||
**Instruction files edited (unprompted-work §5 — facts only):** the workspace root, felhom-agent and app-catalog copies
|
||||
of `.claude/rules/unprompted-work.md` got the missing „A version number is pushed once" paragraph (now byte-identical to
|
||||
the other two); `felhom.eu/.claude/rules/live-probes.md:41` — the stack path `/opt/felhom/stacks/<name>` → `/opt/docker/
|
||||
stacks/<name>/app.yaml`. Details: `EDITS.md`.
|
||||
|
||||
## Parts 2 and 3 — findings, ranked
|
||||
|
||||
| # | Sev | Finding | What was proven | Row |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **P1** | A power cut during a restore of a drive-backed app: the app runs on the cut restore's data, and a later controller restart pours the pre-restore copy back over newer writes | the real `Recover` code in a lean clone: boot 1 „copy gone → refused → marker kept", boot 2 put-back after the app ran (`proofs/H1-drive-cut-restore/`); the boot recreate's missing check read at `intermediary.go:470-476` | R-1251 |
|
||||
| 2 | P2 | One rejected backup block makes the night's second-drive copy delete the app's household-file copies | **reproduced on 9202**: control kept the copy twice; one bogus path → „removed stale dest dir userdata/media" (`proofs/H2-bad-backup-block/`) | R-1252 |
|
||||
| 3 | P2 | An export with „stop first" overwrites another app's restore marker | static, read at `file:line` | R-1253 |
|
||||
| 4 | P2 | The hub saves the escrow package in two writes; a lost answer tells the household „the previous code still works" | static | R-1254 |
|
||||
| 5 | P2 | D4's sshd drop-in reaches BYO hosts (against `11` §1); the in-force check and the all-or-nothing `sshd -t` | static | R-1257 (D43) |
|
||||
| 6 | P3 | Escrow carry / set-aside name edges | static | R-1255 |
|
||||
| 7 | P3 | Four restore edges (definition write, merge, a mirror race, the unused copy) | static | R-1256 |
|
||||
| 8 | P3 | ep0's iperf3 takes any tunnel peer's test | static (no ep0 change) | R-1258 |
|
||||
| 9 | P3 | The DooPlex→ep0 copies: the vouched agent can drop out; deletions mid-transfer; no future bound; **2026-10-25: the 02:30 push re-sends the 20:00 snapshot** | DST timing verified on DooPlex (`systemd-analyze`, Go) | R-1259 |
|
||||
| 10 | P3 | Spring 2027: the leg's deadline vs the gate's clock; two legs at one instant | Go's mapping verified | R-1260 |
|
||||
| 11 | P3 | The unreleased R-1250 line hold: two seeders, a removed app, alt-speed | static | R-1261 |
|
||||
| 12 | P3 | Any viewer explains any upload (the upload watch) | static; `08`'s limits list now names it | R-1262 |
|
||||
| 13 | P4 | Small items (a mislabelled operator mail, undeclared volumes on Remove, mount views, stale comments, a DNS-name password, rrsync) | static | R-1263 |
|
||||
| 14 | P4 | Skills tooling (no size cap on the grandfathered skill; a new skill not installed) | read | R-1264 |
|
||||
| 15 | P4 | Immich leaves an empty anonymous volume per recreate (296 on 9202); photos unaffected | read in the running image | R-1267 |
|
||||
|
||||
**Fixed in this session (small, not recovery code, red-proved, commit only — controller `b01bb4b`, CI 2294 / #480 success):**
|
||||
a media restart („Újraindítás most", „Később", the safe removal) no longer starts an app the household stopped (R-721;
|
||||
two tests, each seen failing with the rule off); the file manager's header name file appears whole (a race the
|
||||
existing test caught 44 times in 300 runs on clean `main`; 0 in 300 after, 0 in 50 under `-race`).
|
||||
|
||||
**NOT-A-FINDING:** SEC-1 X2 (a passphrase enroll after the operator's host delete gets ciphertext — the window D19
|
||||
accepts); SEC-1 X3 (the refused-passphrase mail is operator-only with a one-hour cooldown); DOC-D1 U4 (the fit check
|
||||
that passes on a statfs error — the copy then fails and the app restarts untouched, `restore_unit.go:502-512`);
|
||||
HUNT-3 D31's comment wording (no data effect); Immich's `/data` volume for data placement (R-1267 holds the leak);
|
||||
the stack folder left after a clean removal (it is every catalog app's template folder).
|
||||
|
||||
## Machines
|
||||
|
||||
- **9202** (Tier 0): as found at 12:46 UTC (`9202/as-found-2026-10-24.txt`); at the end (`9202/at-end-…`) the same
|
||||
containers and images, healthy (restarted by the test night chains), the catalog on the live repo, the controller
|
||||
0.342.0. The only difference: Immich's running container uses a new empty anonymous volume (R-1267); the two others
|
||||
my test runs made were removed by name. audiobookshelf installed and removed through the product; my marker file
|
||||
removed; its second-drive copy removed by the product's remove.
|
||||
- **DooPlex:** reads, lean clones in the scratchpad, the gates; no service changed. **ep0, household boxes, the hub:**
|
||||
not touched (hub reads only, none needed in the end).
|
||||
|
||||
## Teardown (three layers)
|
||||
|
||||
Machine: 9202 as above. Host (demo-hp): nothing provisioned. Hub: nothing changed.
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
## FENCES — the brief's, word for word (operator ruling 2026-10-07: every helper prompt carries them in full)
|
||||
|
||||
From the brief's §0 Scope:
|
||||
- **Class:** review + verification, unattended. **Code is not changed** except a fix that is small, not in recovery code,
|
||||
and has a red-proof (commit only). Documents may be corrected for facts proven at `file:line` (PROMPT-TEMPLATE §9
|
||||
rule 9: facts yes, rules and fences never without the operator).
|
||||
- **Machines:** DooPlex (lean clones, builds, read-only reads), 9202 for proofs (put back as found). No household box, no
|
||||
ep0 change, no release.
|
||||
- **Coordination:** the catalog sweep runs in parallel on demo-hp's nested VM and writes to `felhom.eu`. `git pull
|
||||
--rebase` before each push; never remove its lines. **Do not use 9202 between 18:00 and 20:00 if the sweep asks for it**
|
||||
(it should not; it has its own VM).
|
||||
- **Reviewers:** read-only subagents; only the main session touches machines, git and the register.
|
||||
|
||||
From the brief's Rules:
|
||||
1. Verify every claim at `file:line` before acting; a reviewer's word is a candidate, not a finding. 2. Documents: facts
|
||||
yes; rules, fences, „never" lines and protected-machine text never without the operator. 3. Code: only small, non-recovery
|
||||
fixes with a red-proof, commit only. 4. No release, deploy, vouch, publish, ep0 or DooPlex change. 5. Secrets: never
|
||||
printed or committed. 6. 9202 put back as found. 7. Every finding a row, a fix, or `NOT-A-FINDING: <reason>`. 8. Stop at a
|
||||
clean point. 9. No branches; no „Co-Authored-By".
|
||||
|
||||
From the workspace CLAUDE.md (this host is production infrastructure):
|
||||
- NEVER run `docker system prune`, `docker image prune -a`, or any global Docker cleanup here.
|
||||
- NEVER touch k3s data dirs, Longhorn mounts, PBS datastores, or Gitea storage.
|
||||
- Destructive disk/guest operations belong to felhom-pve via the agent — never on this host.
|
||||
- Secrets: never `source` or `cat` a credentials file; never print a token, password, key or `PlexOnlineToken`.
|
||||
|
||||
## WHAT THIS MEANS FOR YOU, THE HELPER (binding)
|
||||
|
||||
You are a **read-only reviewer**. You may ONLY read files in `/mnt/5_hdd/felhom.eu/git/` (Read, Grep, Glob, and
|
||||
read-only shell such as `grep`, `rg`, `sed -n`, `git log`, `git show`, `git blame`). You may NOT: edit or write any file;
|
||||
run `git` commands that change anything (no commit, checkout, reset, stash, pull, push, worktree); run `go test`, `go
|
||||
build`, any gate script, `docker`, `kubectl`, `ssh`, `curl` to any host, `sec`; touch 9202, demo-hp, the N100, Tester 1,
|
||||
Tester 2, ep0, the hub or DooPlex's services. Only the main session touches machines, git and the register. If you
|
||||
think something needs a live test, write the test plan in your answer — do not run it.
|
||||
|
||||
Repos (read their `main` working trees): `felhom.eu` (hub in `hub/`, docs in `documentation/`, scripts in `scripts/`),
|
||||
`felhom-controller` (Go in `controller/`), `felhom-agent`, `app-catalog-felhom.eu`.
|
||||
File diff suppressed because one or more lines are too long
@@ -33,6 +33,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-23 (afternoon).** Your D39 = B is built and live: an app with a public security hole can now get its fix at night on every box, after one test on the test box — controller 0.341.0 is the floor since 13:24 (your yes). The test: Plex moved by itself at night on the test box, and an older controller safely waited for a hand press. Nothing needs you.
|
||||
|
||||
## Moved from STATUS on 2026-10-24 (evening)
|
||||
|
||||
**Updated 2026-10-24 (afternoon, attended).** The two DooPlex backup alarms had one cause: yesterday's reboot changed which password the database takes, so the database copy job could not log in, and the Gitea off-site copy then refused its old copy. Fixed and cleared at 12:51 local; critical alarms now repeat every 4 hours (all backup alarms in one mail), warnings every 12. A box whose torrent app seeds all day can now get its weekly line test (proven on demo-hp, not yet released).
|
||||
|
||||
## Moved from STATUS on 2026-10-23 (afternoon)
|
||||
|
||||
**Updated 2026-10-23 (day).** Released on your go: controller 0.340.0 (the floor), agent 0.163.0 (vouched; root SSH by key only on the three test boxes), installer 1.36.0, hub 0.167.0, and the website with your two sentences. The hub database now leaves every 6 hours and the released images and agent packages every night, both read back from ep0; the break-glass sheet is current — please print it. One false failure mail reached you at 10:28 (my install of the 6-hour copy, fixed in the same hour).
|
||||
|
||||
@@ -31,6 +31,11 @@ mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=
|
||||
hookscript: local:snippets/felhom-guest-hook.sh
|
||||
```
|
||||
|
||||
*Dated note 2026-10-24 (review round 2): a guest built from golden `build-golden.sh` v2.1.0 or later has ONE data volume,
|
||||
`mp0` at `/var/lib/felhom` (its `docker/` and `sys_drive/` folders are bound inside the guest), and **no `mp1`**
|
||||
(`felhom-agent/configs/build-golden.sh:31-38,99`). The procedure below is unchanged: volume mounts (whatever their
|
||||
number) come back from the archive; only the host BINDS are stripped.*
|
||||
|
||||
**Two kinds of `mpN`, and only one is dangerous:**
|
||||
|
||||
| kind | looks like | on restore |
|
||||
|
||||
Reference in New Issue
Block a user