diff --git a/REPORT-campaign12-class-sweep.md b/REPORT-campaign12-class-sweep.md new file mode 100644 index 0000000..4bc3fe8 --- /dev/null +++ b/REPORT-campaign12-class-sweep.md @@ -0,0 +1,66 @@ +# REPORT — Campaign 12, the class sweep (2026-08-08, unattended) + +*A non-overwritten `REPORT-.md` sibling, per `CLAUDE.md:82-87` — a parallel session shares this +clone and the shared `REPORT.md` was not touched.* + +## What ran + +**Part 1 — the bake.** Golden **0.208.0** baked on the drill VM, published, and round-trip verified: +656 150 362 B, sha256 `ba668f59…5ffb82`, and `./etc/felhom-controller-image` read **out of the +downloaded archive** says `felhom-controller:0.208.0`. All acceptance markers green, `Result=success`, +bake VM destroyed and the drill disk restored to `virgin`. Token never on a command line (`grep -c` = +0 on the committed log, **with a control proving the grep works**). **NOT VOUCHED — that is the one +thing awaiting the operator.** Evidence: `documentation/tests/golden-0.208.0-2026-08-08/`. + +**Parts 2–4 — the sweep.** Seven defect classes swept for siblings, analysis only. Report: +`documentation/audits/CAMPAIGN-12-class-sweep-2026-08-08.md`. + +## Result + +**Eight new register rows, R-256 … R-263** (highest ID moved from R-255), grouped by class in +`backlog/OPEN-ITEMS.md`. **C1 produced no new instance** and has no row. + +The sharpest finding is **R-260**: the agent reports `operator_key_configured` on every heartbeat and +the hub has no field for it, so the check that answers *"can the operator get into this box"* returns +`ok` for a box with no operator key installed. Seven more dropped fields are censused with it. + +**Part 4's ranking is the campaign's most valuable output** and lives in `backlog/ROADMAP.md` as +**G-1 … G-8**. The recommendation: **gate C5** (a cross-repo tag-reachability check — cheap, +`--fast`-eligible, and it would have caught every instance in R-260's census on the commit that +introduced them), and **do not gate C6**, because the standard tool was measured against a planted +probe and found blind to the exact shape both known instances have. + +## Controls — every class states whether its method re-found the known instances + +| class | control | +|---|---| +| C1 | shipped gate **2 of 3**, verified by replaying the pre-fix templates from `8dbbc98^` | +| C2 | **2 of 2**, re-found as fixed | +| C3 | **2 of 3 re-found, 1 as fixed** — and swept COMPLETELY (all 9 sites) | +| C4 | fix pattern (R-225 `StatsKnown`) re-found intact | +| C5 | **re-found** — and a correction owed: the task called `escrow_stale` closed; it is R-247, `READY` | +| C6 | **`deadcode` 0 of 2** (blind spot measured with a planted probe); bespoke method **1 of 2** | +| C7 | **weakest** — the nine known are closed, so the sample re-finds the pattern, not the instances | + +## Honest limits + +- **C7 sampled 60 of 2652 production invariant comments** and **none of the 1440 test comments**. The + task asked explicitly for the tests' own claims; that half is **owed, not done**. +- **C2 sampled 19 of 221 refusal strings.** +- **C3 and C4 covered the controller only** — not the hub, not the agent. +- **No finding was reproduced on a live box.** Source reading only, as §6 rule 2 requires. R-258 and + R-259 are the two most worth confirming live before they are fixed. +- **Seven suspicions were investigated and DISPROVED**, including two of my own methods; §6 of the + report names each. +- **`STATUS.md` is 100 lines, 7 over its 93-line "one screen".** The overflow is the campaign's own + entry plus the operator approval; the eight findings themselves are in the register, as required. +- **This session did not run inside `tmux`**, contrary to the workspace `CLAUDE.md`. + +## Gates and hygiene + +`python3 scripts/repo_gates.py --fast` → **all 7 gates OK**, including `golden-currency`, which was +correctly **RED on arrival** and is green after the bake. **No `--no-verify` was used anywhere** — the +bypass the task authorised was not needed. + +No product code changed. Nothing was deployed beyond the golden. No machine was touched beyond the +bake VM, which was torn down. diff --git a/STATUS.md b/STATUS.md index 80c7294..d10db24 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,6 +1,6 @@ # STATUS — what works, what's broken, what's next -**Updated 2026-08-08.** +**Updated 2026-08-08 (overnight — Campaign 12).** > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates > part of it in plain words, and **nothing may exist only here**. Not `CONTEXT.md`, which is technical @@ -24,59 +24,65 @@ time**: someone with a browser and their recovery code got everything back with inside the machine at any point**. From logging in to seeing what is in the store took **72 seconds**. *(R-201 — closed.)* -**And the two rough edges the walk found are gone.** After a rebuild the restore used to stop dead -twice — once because the machine no longer recognised its own drives, once because it refused to -restore an app that was not installed while promising, three lines above, that it would reinstall it. -Both now say what has happened, say that nothing is lost, and link to the one screen that fixes it. -*(R-252, R-253 — both closed 2026-08-08.)* +**And the two rough edges that walk found are gone** — after a rebuild the restore used to stop dead +twice; both refusals now say what happened, that nothing is lost, and link to the screen that fixes +it. *(R-252, R-253 — closed 2026-08-08.)* ## What's broken -- **Nothing new is broken.** The three known secret-in-page faults are all fixed; what remains is that - the *check* against a fourth covers 4 pages out of 27, and the cheap check that covers all of them is - blind to the exact shape that shipped. Filed rather than papered over. *(R-255)* +- **Nothing new is broken.** All three secret-in-page faults are fixed; what remains is that the + *check* against a fourth covers 4 pages of 27, and the cheap one covering all of them is blind to + the shape that actually shipped. *(R-255)* - **The machine's own screen keeps telling an already-paired box to pair itself** — 25 minutes after it was paired, on a screen that promises it refreshes itself. *(R-214, R-235)* - **A rebuilt machine cannot create a new recovery code at all.** *(R-221)* - **A backup that covered nothing still calls itself „Sikeres".** The state is honest; the word is not. *(R-240)* -- **A machine waiting for its recovery code can stop making off-site backups without alarming us.** - Measured on 7 August: after a *rebuild* we ARE told, promptly and correctly. The gap is narrower than - it read — it is a machine that reaches the state without a working tier behind it. *(R-243)* +- **A machine waiting for its recovery code can stop backing up off-site without alarming us.** After + a *rebuild* we ARE told; the gap is a box reaching that state with no working tier behind it. *(R-243)* - **The card offering to reopen set-aside backups promises more than we can deliver** — we keep the old sealed package, but nothing can open it. *(R-202)* - **Deleting a customer leaves rows behind** on every test machine ever torn down, while reporting a clean teardown. No secrets involved, but it accumulates with each walk. *(R-244)* - **Putting restored files back where they belong is still a manual step.** *(R-213)* -## Found today +## Found overnight — we went looking for siblings, on purpose -- **The last two passwords are out of the page source**, the same way as yesterday's: the page carries - only a yes/no, and the value is fetched when you ask for it — and the asking is now recorded, which - it never was. One of them was an app's own first-login password, read live out of the running app. - *(R-254, both sites fixed.)* -- **How much was actually exposed: nothing we can find.** The only app that generates a first-login - password isn't installed anywhere, and of the three apps actually running on the demo machine, none - uses a generated secret. **But nothing recorded reads** — that was part of the fault — so this is - "no evidence of exposure", not "proof there was none". No passwords need changing on that basis; - the call is yours. -- **One of the two turned out not to be a fault.** The deploy form's hidden password field is - deliberate: a form must submit what it saves, so the value you wrote down is the one stored. +Every fault this month came in one of seven shapes; overnight we looked for the others **by shape** — +reading and counting only, nothing changed or deployed, no machine touched. Eight new items, none +urgent. Full working: `audits/CAMPAIGN-12-class-sweep-2026-08-08.md`. + +- **The one that matters.** The machine tells us whether your engineer's emergency key is really + installed — and **we throw that sentence away before reading it**, so a box can read as "we can get + in" when nobody can. Same shape in seven other places. *(R-260)* +- **A green tick meaning only "a backup exists"** — any age counts, and it reddens on a problem + measured across all apps rather than that one. *(R-258)* +- **A disk we failed to measure is drawn as an empty, healthy one** — "0.0 GB / 0.0 GB (0%)" in the + normal colour, while the right pattern sits two files away. *(R-259)* +- **Two comments promise more than the code does** (one guarded by a test that cannot fail on the + thing it names), and **two refusals still stop at "no"**, one in half-English. + *(R-262, R-263, R-256, R-257)* +- **Which shapes can a machine catch from now on?** That mattered more than the list: **one is cheap + and worth doing** (the thrown-away-sentence one, which would have caught every instance on the day + it was written), two need a small decision first, and **two are honestly not automatable** — + including the one that looked most automatable, where the standard tool was tested and found blind. + Recorded, not built. *(ROADMAP G-1…G-8)* ## What we're working on -- **Widening the check** so a fourth secret-in-a-page is caught by a machine rather than by - someone looking. *(R-255)* -- **Proving the hub really keeps the old sealed key** when a machine re-seals. Never run outside a - test; needs a second deliberate wipe and its own session. *(R-198)* +- **Widening the check** so a fourth secret-in-a-page is caught by a machine, not by someone. *(R-255)* +- **Proving the hub really keeps the old sealed key** when a machine re-seals — needs its own session + and a second deliberate wipe. *(R-198)* ## Waiting on you -- **Nothing blocking.** Today's release needs a new base image before installations receive it — same - as yesterday, same answer: it is ours to do, not yours. *(R-242)* - -*R-245 — whether an undecided household is auto-abandoned after 30 days — was settled on 7 August: we -do not build it, and the reasoning is recorded.* +- **One approval: the new base image.** Tonight's release is baked, published and byte-checked, and + **installations still receive yesterday's version until you press Save.** Hub → Configuration → + Day-0 artifacts → Golden **0.208.0** → Save. One field moves; the other two are already right and + were checked. Reversible — re-select 0.207.0 and Save. *(R-242)* +- **Third time in three days, so worth a minute.** A check now catches the *baking* being forgotten; + **nothing catches the approval being forgotten** — tonight's bake proved it, going green before the + approval existed. Two ways to close it are written up, neither built. *(ROADMAP G-8)* ## DooPlex infrastructure — separate from the product @@ -90,5 +96,5 @@ being readable.* anyone else saw it. *(R-132)* - **One thing to read after DooPlex next restarts** — the second-SSD move has never survived a reboot; it writes PASS/FAIL to `/var/log/felhom-store-postboot-check.log`. On PASS, 34 GB comes back. *(R-209a)* -- **Backup scripts on DooPlex are unversioned host state.** *(R-231)* -- **Instruction-file follow-ups**, each needing a decision rather than an edit. *(R-229, R-230)* +- **Backup scripts on DooPlex are unversioned host state** *(R-231)*, and the instruction-file + follow-ups each need a decision rather than an edit *(R-229, R-230)*. diff --git a/documentation/audits/CAMPAIGN-12-class-sweep-2026-08-08.md b/documentation/audits/CAMPAIGN-12-class-sweep-2026-08-08.md new file mode 100644 index 0000000..99aafcd --- /dev/null +++ b/documentation/audits/CAMPAIGN-12-class-sweep-2026-08-08.md @@ -0,0 +1,429 @@ +# CAMPAIGN 12 — the class sweep (2026-08-08, unattended) + +**Look for the siblings of everything this arc found, by class rather than by feature.** Nothing was +fixed, nothing was deployed beyond Part 1's golden, and no machine was touched beyond the bake VM, +which was torn down. + +Baselines re-read on arrival, all three clean and equal to `origin/main`: + +| Component | Version | commit | +|---|---|---| +| `felhom-controller` | released **v0.208.0** | `37b5ba08a704` | +| `felhom-agent` | v0.127.0 | `703db166e762` | +| hub | v0.98.0 deployed | `59527d00f9c4` | +| `app-catalog-felhom.eu` | — | `459766c` | + +Highest register ID on arrival **R-255**; re-checked before minting, unmoved. This session opens +**R-256 … R-263**. + +--- + +## 1. The bake — and the one approval awaiting the operator + +**Golden 0.208.0 is baked, published and round-trip verified. It is NOT vouched.** Full evidence: +`documentation/tests/golden-0.208.0-2026-08-08/` (README + the 324-line `bake.log`). + +| | | +|---|---| +| version | **0.208.0** (derived from the controller image, which is a required argument) | +| size | **656 150 362 B** | +| sha256 | **`ba668f59f063511dda122a5b05900f1350573cdc9619980872f87a58755ffb82`** | +| round trip | downloaded back from Gitea, hashed independently on DooPlex — **size and sha identical** | +| **the delivered artifact's own claim** | `./etc/felhom-controller-image` read **out of the downloaded archive** → `gitea.dooplex.hu/admin/felhom-controller:0.208.0` | +| fourth confirmation | the sha the hub's `/configuration` dropdown offers for 0.208.0 matches the bake exactly | + +**Acceptance markers, each grepped verbatim against the real log:** `docker OK (overlay2; data-root +/var/lib/docker)` ×1 · `including mount point` ×2 (rootfs **and mp0**, no mp1) · `upload OK (HTTP 201)` +×1 · `excluding` ×0 · `FATAL` ×0 · `mp1` ×0. `Result=success`, `ExecMainStatus=0`. + +**Hygiene.** The token was copied **file → file** and read from a `0600` file **inside** the VM by a +runner script; `systemctl show golden-bake -p Environment -p ExecStart | grep -c -F ` → **0**. +The committed log greps **0** for the literal token, **and the grep was shown to work first** — the +same grep on a copy with the token appended returned **1**, then the copy was shredded. + +**404 pre-gate with a control:** `…/0.207.0/golden.tar.zst` → **200** (the URL shape is right), +`…/0.208.0/golden.tar.zst` → **404** (nothing to overwrite). + +**Teardown:** `pct destroy 9100 --purge`, `pct list` empty; token/runner/script/log `shred -u`'d and +`/root` residue clean; `poweroff`; qemu confirmed gone via `ps -eo comm | grep -c qemu-system-x86` → 0 +(never `pgrep -f`, which self-matches); **`qemu-img snapshot -a virgin` restored** and the snapshot +list re-read. + +### ⚠ THE ONE THING ASKED OF THE OPERATOR + +**Vouch golden 0.208.0** — hub → Configuration → Day-0 artifacts → Golden `0.208.0` → Save. + +| field | now | after | why | +|---|---|---|---| +| `golden_version` | 0.207.0 | **0.208.0** | **the only field that moves** | +| `agent_version` | 0.127.0 | 0.127.0 | already ≥ `MinAgent 0.127.0` — read from the controller CHANGELOG header, checked not skipped | +| `min_agent` | 0.127.0 | 0.127.0 | equals the declared `MinAgent`; **not above `agent_version`** (the R-216 shape hub v0.97.0 holds) | + +**Vouching is reversible** — re-select 0.207.0 and Save. The 0.207.0 package was not deleted (the +publish step's pre-delete targets only its own version), so a rollback is a form submission. + +**Until that Save, a machine installed tonight still receives controller 0.207.0.** + +--- + +## 2. What golden is vouched, and the currency gate's state + +- **Vouched at the time of writing: golden 0.207.0**, agent 0.127.0, `min_agent` 0.127.0, + `min_controller_version` 0.156.0 — read live from the hub, not from a document. +- `golden_currency_gate.py` was **RED on arrival** (`controller v0.208.0 is released and NO golden + carries it`) exactly as §1 of the task predicted, and is **GREEN after the bake**. +- **No `--no-verify` was needed anywhere in this session.** + +--- + +## 3. Per class — new instances, the control, the method, and its blind spots + +### C1 — a secret in a response body + +**New instances: 0** on the surface the class names (a response body a customer can fetch). + +**Control — the shipped gate re-finds 2 of the 3 known instances, measured not assumed.** The three +pre-fix templates were restored from history (`8dbbc98^`) into a scratch copy of the controller and +`scripts/secret_in_markup_gate.py` was run against them. It convicted +`settings_security.html:408`, `:430` and `app_info.html:185`, and **did not convict** +`deploy.html:482` (`value="{{$val}}"`) — R-255's stated blind spot, now verified by replay rather +than by reading its docstring. The gate passes clean over today's 36 templates. + +**Method (the new half).** The shipped gate is name-based on the TEMPLATE side. This sweep is its +mirror on the GO side: every assignment into template data, a JSON response map or a render struct +whose **right-hand side** names a secret, regardless of what the **key** is called — the shape the +gate cannot see. 27 sites matched; 5 under a key that does not itself announce a secret. + +All 5 resolve to non-findings, and the reasoning is recorded in §6. + +**Blind spots.** One hop only — a secret laundered through a neutrally-named local two or more hops +from its source is invisible. Struct-literal page data is covered only in the field-assignment form. +A secret fetched inside a helper the handler calls is invisible. And the whole sweep reasons about +**source**, never about rendered bytes — the runtime body assertion remains the only check that +catches all three known shapes, and it still covers 4 of 27 pages. + +### C2 — a refusal that names no route + +**New instances: 2 (R-256, R-257).** + +**Control.** The known instances (R-252, R-253) were re-read in their pre-fix form at `8dbbc98^` and +in today's tree. The method distinguishes them correctly: the pre-fix strings state a reason and stop +(*"nincs mentés felhasználói adattal"*), and today's `backups_restore.html:79` names the route — +*„Meghajtók", „Meglévő meghajtó csatolása". Utána gyere vissza ide.* **2 of 2 re-found, as fixed.** + +**Method + what was SAMPLED.** 221 distinct customer-visible refusal strings exist across the +controller's production code. **This was NOT swept exhaustively.** The sample is the three files on +the surface where both known instances lived — `recovery_handlers.go`, `offbox_handlers.go`, +`restore_wizard.go` — yielding 19 distinct refusals, all read by hand. **The other ~202 strings were +not examined.** + +**Blind spots.** Refusals composed at render time from template fragments are invisible to a +string-literal grep; so are refusals whose text lives in the HTML rather than in Go. "Names a route" +is a judgement, not a predicate — two readers could disagree on the borderline cases, and three of +the 19 were called borderline and left unfiled. + +### C3 — a success verdict over an incomplete state + +**New instances: 1 (R-258). This class was swept COMPLETELY, not sampled.** + +**Method.** Every assignment of a success-valued status in the controller's production code: +**9 sites**, all read. That is the whole population, which is why this is the one class where +"no others exist" is a claim the method can support. + +**Control — 2 of 3 known instances re-found, and the third re-found as fixed.** `offbox.go:1040` +still carries R-240 verbatim: a configured box with nothing selected reports `LastStatus = "ok"` with +the warning *„Sikeres — nincs mentésre jelölt alkalmazás"*. R-234's fix is visible in the same block +(`missingUnprotected` → `incomplete`). R-203's mandatory-gap fix likewise. + +**Blind spots.** It sees only *literal* success values assigned to a name matching +`Status|State|Verdict|Result`. A verdict expressed as a boolean, an enum constant, or the absence of +an error is invisible — and the controller does use booleans for verdicts elsewhere. It covers the +controller only; the hub and agent were not swept for this class. + +### C4 — an absence reported as a fact + +**New instances: 1 (R-259).** + +**Control — the fix pattern re-found.** R-225's `StatsKnown` is exemplary and the sweep re-finds it +intact: on `backups_remote.html` every derived figure — size, snapshot count, quota text and the +**bar itself** — is behind `{{if .Offbox.StatsKnown}}`, with the honest alternative text +*„a pillanatképek száma még ismeretlen"*. Its own comment states the principle this class is about: +*"a 0%-wide bar over an unread store is a picture of emptiness, and a picture is a claim."* + +**Method.** Enumerate the rendered counts/sizes/percentages in the 36 templates, then ask of each +whether a `*Known`-style companion exists and whether the producing Go call can fail into a zero. + +**Blind spots.** Name-based on the companion flag — a three-state encoding done some other way (a +pointer, a sentinel, a separate error field) reads as "unguarded" and needs a human look; two +candidates were dismissed that way. Only the controller's templates were covered; the hub's operator +UI was not. + +### C5 — a field one side sends and the other drops + +**New instances: 1 register row (R-260) carrying a measured census of 8+ dropped fields.** +**This class was the highest-yield, as predicted.** + +**Control — re-found.** The known instance is live: the hub's `store.EscrowStatus` carries +`Stale bool json:"escrow_stale,omitempty"` and the controller's `report.EscrowStatus` has no matching +field. **Correction to the task's framing, recorded rather than dropped: this is R-247 and it is +`READY`, not closed.** The live repo is the source (rule 7). + +**Method — and the first one was wrong.** Attempt 1 paired structs across repos by json-tag-set +overlap. It false-positived badly, because the hub decodes the same controller report through +**several ad-hoc anonymous structs** — `offsite.state` and `offsite.abandon_purge_requested` looked +dropped and are in fact consumed at `api/handler.go:530` and `store/store.go:1490`. Attempt 2 is the +**tag-reachability test**: for every json tag emitted on a real wire, does that literal string occur +**anywhere** in the receiving repo? A tag that occurs nowhere cannot be decoded by any struct, named +or anonymous. That is definitive, and it is the version that generalises into a gate. + +**Blind spots, several of them earned during the sweep.** (a) The emit sets initially included +*decode* structs — `hostReportPayload`, `applianceRegisterReq`, `WireDesiredState` — producing +nonsense in the reverse direction; those rows were struck. (b) The struct extractor **flattens +anonymous nested structs**, which produced a phantom asymmetry on `hostStorageTarget` (§6). (c) 29 +generic tag names (`name`, `type`, `state`, `status`, …) were excluded by name because a repo-wide +string test is meaningless for them — **a genuine drop of a generically-named field would be missed.** +(d) It cannot tell a deliberate ignore from an accidental one; the hub's `hostReportPayload` documents +that it ignores several collections on purpose, and those were not counted. + +### C6 — built and never wired + +**New instances: 1 (R-261). The more valuable output is the method result, not the list.** + +**Control — and the standard tool FAILS it.** `golang.org/x/tools/cmd/deadcode` was run against all +three binaries and **re-found neither** known instance. A **planted probe** measured why: an +unreachable exported *function* added to the hub's store package is reported; an unreachable exported +*method* on `*Store`, in the same file, is **not**. RTA treats the method set of a dynamically-used +type as potentially reachable. **Both known instances are methods** — `(*Store).MarkEscrowStale` and +`(*RestoreTestState).Snapshot`, both confirmed still callerless today — which is exactly the shape it +cannot see. + +A method-aware textual sweep was then written. It re-finds **1 of 2** (`MarkEscrowStale`); it misses +`RestoreTestState.Snapshot` because a `.Snapshot()` call on a *different* type in +`guestnet/report.go` satisfies its name-based test. **So: 1 of 2, and stated as 1 of 2.** + +**Findings after triage: small, and mostly inert — which is itself the result.** 137 raw candidates → +31 whose doc comment claims a consumer → after removing method-values-as-callbacks +(`PBSDRAutoProvision`), cross-repo name reuse (`DebugLogs`, `AppStopGuard`) and template dispatch, +five genuinely have only their own declaration. Four of those five are **inert**: the data reaches its +stated consumer by another route. `OffboxOrphanedRenamedTo` says *"for the card copy"* and has no +caller — **but R-228 is genuinely closed**, because the card reads the settings field +`.Offbox.OrphanedRenamedTo` directly in `backups_remote.html:80`. Same for `OffboxEscrowState`. +**An unwired accessor whose data reaches its consumer another way is dead code, not a defect**, and +ranking it as one would inflate this class. + +The one that is not inert is R-261. + +**Blind spots.** Go **templates** call methods with no `.Name(` syntax; method **values** passed as +callbacks likewise; reflection and struct-literal function fields likewise. Same-name methods on +different types are conflated, which makes the sweep conservative — it under-reports and never +over-reports. A method used only inside its own file counts as used, so whole-subsystem orphans are +invisible. + +### C7 — a comment asserting an invariant the code does not hold + +**New instances: 2 (R-262, R-263), both found while sweeping other classes.** + +**Method + what was SAMPLED — this is the class where the honesty matters most.** There are **4092** +comments across the three repos matching the invariant vocabulary (`never` / `always` / `must not` / +`cannot` / `is the only` / `guarantees` / `impossible`) — 2652 in production code and 1440 in tests. +**Exhaustive is not possible in one session and was not attempted.** The sample is the **60 production +comments containing "is the only"**, chosen because a uniqueness claim is the one form that is +*mechanically falsifiable* by a grep. Of those 60, the safety-bearing ones were tested by hand. +**The other ~2592 production comments and all 1440 test comments were not examined** — and the task's +own instruction to check the tests' claims is therefore **only partly discharged**: the two instances +below are production comments. + +**Control.** The nine known instances are the repo's own table (in the `felhom-testing` skill) and are +closed; the sample re-finds the *pattern* rather than the instances, so **C7 has the weakest control +of the seven and should be read that way.** + +Both new instances are stated in the register. The sharper one, R-262, is the campaign in miniature: +a comment claiming a contract is mirrored *field-for-field* and *guarded by a key-set test*, where the +contract is two fields short **and the fixture the test reads omits the same two fields** — the +instrument is blind in exactly the place the defect is. + +--- + +## 4. Which classes can be gated — the ranking + +Ranked by what a gate would be worth, using this campaign's own instance counts as the evidence. + +| rank | class | gateable? | cost | what it would MISS | +|---|---|---|---|---| +| **1** | **C5** | **Yes — cheaply and definitively** | ~150 lines of Python, no network, `--fast`-eligible | generically-named tags; a deliberate ignore looks like an accident, so it needs an allowlist-with-reasons like the C1 gate has | +| **2** | **C3** | **Yes** | small — the whole population is 9 sites | verdicts expressed as booleans or as the absence of an error | +| **3** | **C4** | **Partly** | medium | the companion-flag convention must be adopted deliberately; a gate can enforce "every rendered figure has a `*Known`" only once that is the house style | +| **4** | **C1** | **Already gated, partly** | completing it is the expensive one | see below | +| **5** | **C7** | **Narrowly** | small for the narrow form | only uniqueness claims are mechanical; the other ~97% of the vocabulary is not | +| **6** | **C2** | **No, not mechanically** | — | "names a route" is a judgement; a gate could only enforce a *convention* | +| **7** | **C6** | **No — and this is the finding** | — | the off-the-shelf tool is blind to the project's own shape, measured | + +**C5 is the clear first.** The tag-reachability test is a string comparison over two repos' source: it +convicts definitively (a tag occurring nowhere in the receiver cannot be decoded), needs no network, +and would have caught the escrow_stale instance, the `operator_key_configured` instance and every +other row in R-260's census on the commit that introduced them. **It is the cheapest high-value gate +available to this project.** Its one design requirement is an allowlist with stated reasons, because +the hub deliberately ignores some collections. + +**C6 does not hold, and saying so is the point.** The class *looks* the most mechanical of the seven +and it is the one where the standard tool measurably fails. Any gate here would have to be written +from scratch, would be conservative by construction, and — on this campaign's evidence — would spend +most of its output on inert dead accessors. **Recommendation: do not gate C6. Gate C5 instead, which +catches a strict subset of the same "the answer was available and discarded" family with none of the +ambiguity.** + +**C1's existing gate, stated honestly.** `secret_in_markup_gate.py` covers all 36 templates on the +name-based check and is **blind to the neutral-page-data-key shape** — measured this session by replay, +not merely quoted: it convicts 2 of the 3 known instances and not the third. The runtime body +assertion, which catches all three, covers **4 of 27 pages**. Completing it means constructing each +remaining page's data in a test — the cost is per-page and real, and it is the honest reason it has +not been done. The Go-side mirror written for this campaign is a cheap partial third net and is **not +recommended as a gate on its own**: it produced 27 candidates and 0 findings, which is a bad +signal-to-noise ratio to put in front of every push. + +**Nothing was built.** These are candidates recorded in `backlog/ROADMAP.md`. + +--- + +## 5. Harness faults, separated from product faults + +**Harness, not product:** + +1. **My first C5 method was wrong** and would have produced ~10 false findings had it been believed — + struct-pair diffing cannot see a receiver that decodes through ad-hoc anonymous structs. Replaced, + not patched over. +2. **My struct extractor flattens anonymous nested structs**, which manufactured a phantom + `hostStorageTarget` asymmetry (§6). +3. **A `grep -v host_oob.go` excluded `monitor/host_oob.go` too**, and I briefly believed + `GetHostOOBStates` had no caller. Caught by checking before writing it down. Excluding by basename + is a trap worth naming. +4. **`deadcode` failing its control** is a harness fault in the sense that matters here — but it is + reported as a *result* in §3/C6, because "the standard tool cannot see this project's shape" is + exactly what Part 4 was asked to establish. + +**Session hygiene, recorded:** this session did **not** run inside `tmux`, contrary to the workspace +`CLAUDE.md`. Nothing was lost, but the whole unattended run was one dropped connection from dying. + +--- + +## 6. Suspicions investigated and DISPROVED + +A campaign reporting only what it found is half a campaign. + +1. **`offsite.state` / `offsite.abandon_purge_requested` are dropped by the hub.** **DISPROVED.** + The hub's `monitor.offsiteReport` has neither, but both are consumed through separate ad-hoc + structs — `api/handler.go:530` and `store/store.go:1490`. This is what killed method 1. +2. **A restore-test mount-parity mismatch is invisible to the hub.** **DISPROVED, and it mattered.** + `mount_parity` and `mount_inventory` genuinely are dropped, but a mismatch sets `res.Err` and + returns **before** `res.Pass = true` (`reconcile/restoretest.go:277-283`), so the *verdict* travels + as `pass:false` with the delta in the error string. What is lost is the *depth* of a pass, not the + failure. R-260 states it that way rather than the way it first looked. +3. **`hostStorageTarget` does not mirror the agent field-for-field.** **DISPROVED.** The 11 "missing" + SMART fields are nested under `smart` on both sides; my extractor flattened the hub's anonymous + inner struct. That half of the comment is TRUE — which is what makes the `hostRestoreTest` half + (R-262) a real finding rather than a sweeping accusation. +4. **`GetHostOOBStates` has no caller (a C6 instance).** **DISPROVED** — `monitor/host_oob.go:40,77`. + My own grep excluded it. +5. **`Validation.TableCount` reports 0 tables over a validation that could not run (a C4 instance).** + **DISPROVED.** `backups_apps.html:74-82` is correctly three-stated: `Valid` → the count, + `Error` → *„Hiba"*, neither → *„Az érvényesítés nem futott le"*. A good example of the class handled + well. +6. **The controller leaks a NAS password into a response body (a C1 instance).** **DISPROVED.** + `netstorage_handlers.go:163` puts it in a **request to the agent**, documented in place as never + persisted and never in the job status. +7. **`OffboxOrphanedRenamedTo` being callerless means R-228 regressed.** **DISPROVED** — the card reads + the settings field directly (`backups_remote.html:80`). R-228 is genuinely closed. + +--- + +## 7. What was SAMPLED rather than swept, exactly + +| class | swept completely | sampled | NOT examined | +|---|---|---|---| +| C1 | all 36 templates (shipped gate); 27 Go-side candidate sites | — | the 23 pages with no runtime body assertion | +| C2 | — | 3 files / 19 refusals on the recovery-restore-offbox surface | **~202 of 221 refusal strings**; all refusals composed in HTML | +| C3 | **yes — all 9 success-status sites in the controller** | — | the hub and the agent entirely; boolean/enum verdicts | +| C4 | the 36 controller templates' rendered figures | — | the hub operator UI | +| C5 | **yes — all 4 wire directions, 465 emitted tags** | — | 29 generically-named tags, excluded by name | +| C6 | all 3 binaries via `deadcode`; 1323 exported methods textually | 31 of 137 candidates triaged by hand | the 106 candidates whose doc comment claims no consumer | +| C7 | — | **60 of 2652** production invariant comments ("is the only" only) | **~2592 production + all 1440 test comments** | + +--- + +## 8. R-242's untouched half — what would catch a skipped vouch earliest + +**Recorded, not built.** This bake demonstrated the gap rather than arguing it: +`golden_currency_gate.py` **flipped from red to green the moment the evidence DIRECTORY existed** — +before the round-trip download had finished, and with no vouch anywhere near it. Its positive signal +is "a directory named after the version is present in the repo". + +**Earliest catch, in order of how early it fires:** + +1. **Earliest and cheapest — make the vouch the thing that writes the evidence.** The gate cannot read + the hub (its docstring records why: a network-reading gate is not `--fast`, so it would run in + neither the pre-push hook nor CI). But the *bake session* can: after the operator saves, re-read + `/configuration` and write the observed `golden_version` into the evidence README as a machine- + readable line. The gate then requires that line, not merely the directory. It still cannot detect a + lie, but it detects the **forgetting**, which is the actual failure mode the gate was built for. +2. **Loudest — a hub-side check, not a repo-side one.** The hub already knows both numbers: the + vouched `golden_version` and the newest controller the fleet reports. A daily monitor raising + `golden_behind_release` would catch a skipped vouch **and** a vouch that silently rolled back, which + nothing in git can ever see. It fires within a day rather than at the next push. +3. **Weakest — a checklist item.** R-242 already was a rule without a mechanism, and it recurred the + next day. Recording it again is not a fix. + +**Recommendation: (1) now, (2) when the hub is next touched.** (1) closes the recurrence that has cost +two releases in two days; (2) closes the class. + +--- + +## 9. Register rows opened, grouped by class + +Highest ID **moved: R-255 → R-263.** + +| class | rows | +|---|---| +| C1 | *none — no new instance* | +| C2 | **R-256**, **R-257** | +| C3 | **R-258** | +| C4 | **R-259** | +| C5 | **R-260** | +| C6 | **R-261** | +| C7 | **R-262**, **R-263** | + +Gating candidates from Part 4 are items in `backlog/ROADMAP.md`, not register rows. + +--- + +## 10. What did not run, and why + +- **The vouch.** Deliberate — §3 of the task halts for it, and the campaign does not depend on it. +- **Any fix.** Deliberate — §0 forbids it; a campaign that fixes as it goes measures a moving target. +- **C7 exhaustively.** Not possible in one session: 4092 comments. Sampled and declared. +- **C2 exhaustively.** 221 refusals; 19 read. Sampled and declared. +- **C3/C4 on the hub and agent.** Only the controller was swept. Not attempted, not a claim. +- **The tests' invariant claims (C7's second half).** 1440 test comments; none examined. The task asked + for this explicitly and it is **owed**, not done. +- **Live validation of any finding.** Every finding in this report is from source reading. **None was + reproduced on a live box** — deliberately, since §6 rule 2 permits no machine beyond the bake VM. + R-258 and R-259 are the two most worth confirming live before they are fixed. + +--- + +## 11. Observations — noticed and NOT acted on + +1. **The `felhom-golden` package list holds 14 goldens** back to 0.136.0, none ever pruned. Not a + problem today (a rollback needs the old package to exist), but nothing decides when one may go. +2. **The hub's `host-report.golden.json` fixture contains `cpu_temp_c` and `loadavg`**, which no hub + struct decodes. A fixture carrying keys the receiver cannot read is a quiet way for a "contract + test" to look broader than it is — the same shape as R-262, one level down. +3. **`readDiskUsage` (R-259's site) is one of several collectors that return silently on error**, + leaving the caller's zero value. The class may be wider than the one rendered figure this campaign + measured; only the dashboard's system-disk meter was traced to a customer-visible surface. +4. **The re-enroll response echoes `new_api_key` back to the caller that sent it** + (`api/dr.go:148`). Not a leak — the box already holds it — but it puts a live credential in any + transcript of that response, and the endpoint is one an operator drives by hand. +5. **The hub operator UI renders `RetrievalPassword` into page data** (`web/configs.go:672`, `:1262`). + Outside C1 as defined (operator-only, not a customer-fetchable page), so not filed — but it is the + R-249 shape on the other surface, and the operator UI has no equivalent of the controller's gate. +6. **This session did not run inside `tmux`**, contrary to the workspace `CLAUDE.md`. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 24cf014..4383943 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -369,6 +369,26 @@ unconditional promise, which CAMPAIGN-11 §7 step 7 measured the customer-facing | **R-210** | **Which of the 345 local images may be deleted — 131 controller tags and 62 hub tags exist ONLY on this box and are not recoverable by `docker pull`** | **WAITING-ON-OPERATOR** — NEW 2026-08-05 | an operator ruling | **Nothing was deleted; this is a list, not an action.** The registry was queried directly: `felhom-controller` has **76** tags in Gitea vs **207** locally, `felhom-hub` **45** vs **107**. The **131 + 62 local-only tags are all OLD** — controller `0.39.0`–`0.135.0` plus `v0.35.0`–`v0.39.0`, hub `0.9.0`–`0.57.0` plus `v0.7.2`–`v0.13.0` — while everything from controller `0.136.0` and hub `0.58.0` upward IS in the registry and therefore re-pullable. **Size the prize honestly before spending a decision on it:** per-tag sizes sum to 139.29 GB, but that double-counts shared layers — `docker system df` puts the **real** dedup'd image footprint at **31.02 GB with 27.02 GB reclaimable**, i.e. an order of magnitude less than the build cache P3 already returned. `docker image prune -a` would remove 343 of 345 (only `redis:7-alpine` and `postgres:16-alpine` are held by running containers). **CC's view: not worth doing for the space** — it buys ~27 GB against 199 GB now free, and its only real benefit is dropping unrecoverable clutter | operator | | **R-211** | **Prometheus has no config-reloader — a rules change reaches the pod and is never read** | **READY (S) — NEW 2026-08-05** | — | Found while verifying R-205 rather than by looking for it. The `mon-system/prometheus` Deployment runs **one** container (`prom/prometheus:v3.12.0`) with **no `configmap-reload`/`prometheus-config-reloader` sidecar**. After the ArgoCD sync the updated `node-housekeeping-alerts.yml` was present **inside the pod** (`grep -c "and on(instance)"` → 3 on the mounted symlink) while the Prometheus **rules API still served the old expression** — for **4+ minutes**, with no error anywhere. It only took effect after an explicit `POST /-/reload`. **The consequence is general, not specific to R-205: every rule edit in this repo since the stack was built has silently not applied until something happened to restart the pod** — so "committed and synced" has never meant "in force", and ArgoCD reporting `Synced/Healthy` is true and beside the point. `--web.enable-lifecycle` IS already set, so the fix is small: add a reloader sidecar watching the ConfigMap, or a `checksum/config` pod annotation so a rules change rolls the pod. **Same class as the four *built-but-never-wired* seams** — the control exists, nothing walks it | CC | +## CAMPAIGN 12 — the class sweep, 2026-08-08 (unattended) + +Eight rows, **grouped by class so the classes are visible as classes**. Full method, controls, blind +spots and the Part-4 gating ranking: `audits/CAMPAIGN-12-class-sweep-2026-08-08.md`. Gating candidates +are in `ROADMAP.md`, not here. **C1 produced no new instance** and has no row, deliberately. + +**⚠ A correction the campaign owed to its own brief:** the task described C5's `escrow_stale` instance +as "closed individually". **It is not — it is R-247, `READY`.** The live repo is the source. + +| ID | What | State | +|---|---|---| +| **R-256** | **C2 — „A mentéskezelő nem elérhető." names no route at all.** `web/offbox_handlers.go:47` and `:181` flash this to the customer on the off-site backup surface. It states an internal component's unavailability in the operator's vocabulary („mentéskezelő" = the backup Manager object), gives no reason the customer can act on, and names no next step — not "try again in a few minutes", not "contact support", not a page to go to. **Contrast, in the same subsystem and shipped the same week:** R-252's fix reads *„Meghajtók", „Meglévő meghajtó csatolása". Utána gyere vissza ide.* — a route. **Severity is low and stated so it is not over-ranked:** the condition is a nil backup manager, which on a healthy box does not occur; this is about the copy, not a broken path. Found by the C2 sample (19 refusals on the recovery/restore/offbox surface; **~202 of the repo's 221 refusal strings were NOT examined**) | **READY** — owner Viktor | +| **R-257** | **C2 — „Az offsite tároló nincs elárvult állapotban." puts an English loanword and an internal state name in front of a Hungarian household customer, and names no route.** `web/offbox_handlers.go:270` (the Go error it mirrors is `backup/offbox.go:343`). „Offsite" is untranslated; „elárvult állapot" is the codebase's own `OffboxOrphaned()` predicate surfacing verbatim. A customer who pressed a button and got this cannot tell whether something failed, whether they did something wrong, or what to do instead. **This is a refusal that is CORRECT and fail-closed and still a dead end** — the same shape R-241 recorded for `--recover-offsite-install`. **Fix shape, not a decision:** say what the customer tried to do, why it does not apply right now, and where to look — or, since this is a state they cannot reach deliberately, do not offer the action at all | **READY** — owner Viktor | +| **R-258** | **C3 — the customer's per-app backup tick is green on the PRESENCE of a restore point, and its only red condition is a GLOBAL one.** `web/handlers.go:1240-1248`: `row.Tier1LastStatus = "ok"` whenever `ListRestorePoints(app)` returns ≥1 point, and `"error"` only when `status.LastDBDump != nil && !status.LastDBDump.Success` — where `LastDBDump` is the box's **single most-recent DB dump** (`backup/backup.go:1064`, `m.lastDBDump`), **not this app's**. Three consequences, in increasing order of how wrong they look to a customer: (a) an app whose own last backup failed shows a **green tick** provided any older restore point exists and some *other* app dumped successfully afterwards; (b) an app with **no database at all** takes the `nil` branch and is green on presence alone; (c) the tick asserts nothing about **recency** — a restore point from three weeks ago is as green as one from last night, while `row.Tier1LastRun` beside it carries the real (old) time. **It is customer-visible:** `templates/backups_apps.html:173-174` renders it as a check or a cross. **This is the project's own "presence is not success" rule as a UI badge** — the artifact's existence is being read as a successful result, which is what the workspace `CLAUDE.md` section of that name forbids. **Found by a COMPLETE sweep, not a sample:** all 9 success-status assignment sites in the controller were read; this is the only new one. **NOT reproduced live** — source reading only | **READY** — owner Viktor | +| **R-259** | **C4 — a disk read that FAILS renders as „0.0 GB / 0.0 GB (0%)" in the nominal colour, on the dashboard's most-looked-at meter.** `system/info_linux.go:259-264`: `readDiskUsage` logs at DEBUG and **returns**, leaving the caller's `TotalGB`/`UsedGB`/`AvailGB`/`Percent` at their zero values. `dashboard.html:57-63` then renders `{{fmtGB .SystemInfo.DiskUsedGB}} / {{fmtGB .SystemInfo.DiskTotalGB}} ({{printf "%.0f" .SystemInfo.DiskPercent}}%)` plus a meter whose fill is `width:0%`, and `usageColor(0)` returns **`"nominal"`** (`web/funcmap.go:189-196`) — so **a failed measurement is drawn as a healthy, empty disk.** There is **no `*Known` companion for the system disk** (grep: 0). **The fix pattern already exists in this codebase, two files away, with the reasoning written out:** R-225's `StatsKnown` guards every derived off-site figure and its own comment says *"a 0%-wide bar over an unread store is a picture of emptiness, and a picture is a claim"* (`backups_remote.html:60-62`). The dashboard does not use it. **It also travels:** `report/builder.go:94` puts the same zeroed `TotalGB`/`UsedGB` into the host report, so a failed statfs reaches the hub as a 0-of-0 disk. **Observation attached, not filed separately:** `readDiskUsage` is one of several collectors in that file that return silently on error; only this one was traced to a customer-visible surface. **NOT reproduced live** | **READY** — owner Viktor | +| **R-260** | **C5 — the agent reports at least eight decision-bearing facts the hub models NOWHERE, and the sharpest one blinds the check that answers „can the operator get into this box".** Measured 2026-08-08 by a tag-reachability test over all four wire directions (465 emitted json tags; a tag whose literal string occurs nowhere in the receiving repo cannot be decoded by any struct, named or anonymous). **THE ONE THAT MATTERS: `operator_key_configured`.** The agent emits it every heartbeat (`hub/report.go:178` — *"operator authorized_key installed"*); the hub's OOB decoder (`store/host_oob.go:38-45`) mirrors **5 of the agent's 8** OOB fields and has no field for it, nor for `wg_handshake_age_s` or `healed_at`. So `oobDegraded` (`monitor/host_oob.go:60-70`) tests config-invalid and (peer-configured AND not-active-or-not-reachable) — and **a box with felhom-sshd active, reachable, valid config, peer configured and NO OPERATOR KEY INSTALLED is `ok`.** The agent knows and says so; the hub throws it away. The checker's own doc comment claims it *"answers 'can the operator get into this box right now, and if not, why' proactively"* — see R-262's class. **Break-glass is the recovery chain (G1), which is what makes this the top row of the census.** **The rest of the census, hub-side absent (0 occurrences anywhere in `felhom.eu/hub/`):** `guest_net` — **the entire per-guest network-health object**, populated every cycle at `hub/collect.go:290`, carrying `dhclient_alive`, `has_route`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`; `selfupdate_pending` + `selfupdate_pending_version` — a staged-but-unapplied agent update is invisible to the fleet view; `healed_recently` (mgmt-plane); `operator_key_configured`, `wg_handshake_age_s`, `healed_at` (OOB); `mount_parity` + `mount_inventory` (restore-test, see R-262); `cpu_temp_c`, `loadavg`, `memory_total_bytes`, `memory_used_bytes`, `uptime_seconds` (host metrics); `model_name` (SMART); `applied_at` (PBS-DR). **Controller→hub, hub-side absent:** `reporting_disabled`, `config_hash`, `last_db_dump`, `last_integrity_check`, `migrated_to`, `catalog_ref`, `storage_bindings`, `container_path`, `subpath`, `temperature_celsius`, `load_avg_{1,5,15}`, `memory_{total,used}_mb`, `uptime_seconds`. **⚠ STATED PRECISELY, because it first looked worse than it is:** the hub stores the whole report as `report_json`, so nothing is lost from the DATABASE — every hub consumer, without exception, re-unmarshals it into a typed struct, so nothing reaches a check, an alarm, a notification or a screen. And a **restore-test mount-parity mismatch is NOT hidden**: it sets `res.Err` and returns before `res.Pass = true`, so `pass:false` travels; what is lost is the *depth* of a pass, not the failure. **The class is what this row is about, not any single field** — this is `escrow_stale` (R-247) for the fourth, fifth and sixth time, and §4 of the campaign report argues it is the cheapest high-value gate available to this project. **Blind spot of the method, recorded:** 29 generically-named tags (`name`, `state`, `status`, …) were excluded by name, so a drop of a generically-named field would be missed | **READY** — owner Viktor | +| **R-261** | **C6 — `CountSelfBindTokens` exists so that callers can assert an invariant, and no production caller asserts it.** `hub/internal/store/selfbind.go:106-111`. Its doc comment: *"it exists so callers can assert the 'after this runs, the only live link is one we just issued — or none' invariant that the auto-mint at customer-create / RESET-completion depends on."* **Census: only its own declaration in production; the two callers are `selfbind_automint_test.go:29` and `customer_delete_test.go:510`.** Tests are not callers (the campaign's rule), so the invariant the auto-mint *depends on* is checked in the test suite and never at the moment it matters. **This is the smallest of the eight rows and is filed at its true size, because the rest of the C6 sweep found INERT dead accessors rather than defects:** `OffboxOrphanedRenamedTo` and `OffboxEscrowState` have no caller but their data reaches the card another way (the template reads the settings field directly, `backups_remote.html:80`) — **R-228 is genuinely closed, and the sweep's first reading that it had regressed was wrong.** **The more consequential C6 result is a method result and is in the report, not here:** `golang.org/x/tools/cmd/deadcode` re-finds **neither** known instance, and a planted probe measured why — it reports an unreachable exported FUNCTION and not an unreachable exported METHOD on a widely-used type, and both known instances are methods | **READY** — owner Viktor | +| **R-262** | **C7 — a comment claims a cross-repo contract is mirrored „field-for-field" and „the key-set tests guard drift"; it is two fields short, AND THE FIXTURE THE TEST READS OMITS THE SAME TWO FIELDS.** `hub/internal/api/handler.go:682-687` covers `hostBackup` **and** `hostRestoreTest`. **It is TRUE of `hostBackup`** (verified field-for-field against `agent/internal/hub/Backup`). **It is FALSE of `hostRestoreTest`:** the agent emits `mount_parity` and `mount_inventory` (`hub/report.go:432-433`, populated in production from `reconcile/restoretest.go:277-283` via `backup/runner.go:517`), and the hub has no field for either — **0 occurrences in the entire hub repo** outside the CHANGELOG. **The guard is blind in exactly the place the drift is:** `TestHostReport_GoldenContract` reads `testdata/host-report.golden.json`, the two copies of which are byte-identical as required — and **neither contains `mount_parity` or `mount_inventory` at all**, so the key sets agree on a shape that is not the shape the agent sends. A test that cannot fail on the drift it names is the R-97b lesson (*prove the consequence, not the mechanism*) landing on a contract test. **Consequence, stated precisely:** the verdict is not lost (a parity mismatch fails the test before `Pass` is set), but the hub cannot distinguish a full-fidelity restore-test pass from a boot-only one, for any agent, ever. **Fix shape, not a decision:** add the two fields and put them in the fixture — or narrow the comment to name `hostBackup` only and say plainly that `hostRestoreTest` is a subset. **Attached observation:** the same fixture carries `cpu_temp_c` and `loadavg`, which no hub struct decodes — a fixture carrying keys the receiver cannot read is the same shape one level down | **READY** — owner Viktor | +| **R-263** | **C7 — „This is the ONLY writer of `StoragePath.BackupTarget`" is false, and nothing pins it.** `settings/settings.go:1317-1319`, on `SetBackupTarget`. **`ClearBackupTarget` (`:1358-1363`) also writes the field**, 17 lines below, in the same file. **The GUARANTEE the comment protects is intact and that is why this is filed small:** the sentence continues *"registration must never set it (E-2 §3: a drive never acquires a role by appearing)"*, and `ClearBackupTarget` only ever writes `false`, so no path other than `SetBackupTarget` **grants** the role. **What is wrong is the claim as written, and the absence of anything holding it:** `backup_target_role_test.go` exercises the behaviour and asserts nothing about writer uniqueness, so if a third writer appeared tomorrow — one that granted — the comment would still read as settled and the suite would still be green. **This is the class's own definition:** an invariant asserted in prose with no test pinning it. **Fix shape:** one word (*"the only writer that GRANTS the role"*) plus a test that fails when a second granting writer appears. **Method honesty:** found in a sample of **60 of 2652** production invariant comments — the "is the only" form only, chosen because a uniqueness claim is the one form a grep can falsify. **~2592 production and all 1440 test invariant comments were NOT examined**, so C7 has the weakest coverage of the seven classes and the task's instruction to check the tests' own claims is **owed, not discharged** | **READY** — owner Viktor | + ## Why the TOP READY rows rank this way diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 7db6960..e7ddca4 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -128,6 +128,24 @@ | R-109 | ~~The DR recipe records no backup target~~ | XS | **SHIPPED** — agent v0.118.1 + hub v0.83.0, 2026-07-30 | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` (the scheduler's own function) + the mountpoint that separates `/mnt/hdd_1` from `/var/lib/vz`; unresolvable is recorded as unresolvable, never defaulted. Required an unscoped HUB half: `AssembleDRRecipe` allow-lists top-level keys → **R-122**, a fourth recipe-completeness defect that had already shipped (`offsite_restic` dropped for the feature's whole life). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 | | E-2 | **Drive-role machinery around the moved vzdump target.** The 2026-07-28 runbook proved the architecture change by hand on both demo boxes; this is the machinery: a **backup-target role** on `StoragePath` beside `Schedulable`/`IsDefault`/`Kind`; **assignment in the storage wizard** (suggest by attribute, refuse the absurd, never decide by transport or `removable` — on the reference hardware demo-felhom's target IS a USB HDD and BOTH drives report `removable=0`); **unassigned drives do nothing automatically**; **stickiness** (never silently retarget); `felhom-host-install.sh` creating the target with `--is_mountpoint 1` **and** issuing the `FelhomAgentStore` ACL; **absent-target policy**; **retention/space accounting** on a drive the customer shares; the honest **single-drive label**; remaining fleet migration | M | READY — 2026-07-28 | Full scope + rationale in `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §7. Two traps already paid for live: the storage `path` must BE the mountpoint or the agent reports the target `disconnected` forever (`internal/storage/observe.go:321`), and the per-storage `FelhomAgentStore` grant is mandatory or every backup 403s. Absent-drive behaviour today is **fail-loudly, no silent retarget** (`is_mountpoint 1` proven live) — which is NOT the intended fall-back-and-alarm design. Flips: matrix row 4 | +## Gating candidates — Campaign 12, Part 4 (2026-08-08) + +**Ranked by what a gate would be worth, using Campaign 12's own instance counts as the evidence.** +Nothing here was built; §4 of `audits/CAMPAIGN-12-class-sweep-2026-08-08.md` carries the reasoning and +the measurements. The recurring lesson these rank against: **a pattern found three times is not closed +by looking a fourth time.** + +| Rank | Item | Size | Status | Notes | +|----|------|------|--------|-------| +| **G-1** | **Gate C5 — the cross-repo tag-reachability check.** For every json tag emitted on a real wire (agent→hub, controller→hub, agent→controller, hub→both), assert that the literal tag string occurs somewhere in the receiving repo. A tag occurring nowhere cannot be decoded by any struct, named **or anonymous** — which is what makes this definitive where struct-pair diffing is not. | S (~150 lines Python) | **candidate — recommended first** | **The cheapest high-value gate available to this project.** No network, no container runtime → `--fast`-eligible, so it runs in BOTH the pre-push hook and CI (the constraint that forced `golden_currency_gate.py` to check the bake rather than the vouch). It would have caught `escrow_stale` (R-247), `operator_key_configured` and every other row of R-260's census **on the commit that introduced them**. **Design requirement, not optional:** an allowlist with stated reasons, in the shape `secret_in_markup_gate.py` already uses — the hub deliberately ignores some collections and says so. **Known miss:** generically-named tags (`name`, `state`, `status`, …), where a repo-wide string test is meaningless; 29 were excluded by name in the sweep. → R-260 | +| **G-2** | **Gate C3 — a success verdict may not be set where an incompleteness signal is in scope.** Assert that every literal success-status assignment either has no gap/skip/missing signal available at that point, or consults it. | S | candidate | The whole population in the controller is **9 sites** — Campaign 12 read all of them, which is why this class is the one where "no others exist" is supportable. Small enough to gate by enumeration rather than by inference. **Known miss:** verdicts expressed as booleans, enum constants, or the absence of an error — and the controller does use those elsewhere. Instances: R-240 (open), R-258 (new). | +| **G-3** | **Gate C4 — every rendered count/size/percentage needs a `*Known` companion.** | M | candidate — **needs a convention decision first** | R-225's `StatsKnown` is the house pattern and it is exemplary; a gate can only enforce it once it is *the* house style. Today three-state is also encoded with pointers and with separate error fields, both legitimate, and a name-based gate reads those as unguarded. **The decision owed is "how does this codebase say 'we could not look'", not "should we gate it".** Instance: R-259. | +| **G-4** | **Complete C1's runtime body assertion — 4 of 27 pages today.** | L | candidate — the expensive one, and honestly so | `secret_in_markup_gate.py` covers all 36 templates on the NAME-based check and is **blind to a secret under a neutral page-data key** — verified 2026-08-08 by replaying the three pre-fix templates through it: it convicts 2 of 3 and not the third. The runtime assertion catches all three; extending it means constructing each remaining page's data in a test, which is a per-page cost and is the real reason it has not been done. **Do NOT adopt the Go-side mirror Campaign 12 wrote as a gate on its own** — 27 candidates, 0 findings is bad signal-to-noise in front of every push. → R-255 | +| **G-5** | **Gate C7, narrowly — uniqueness claims only.** A comment saying "X is the ONLY writer/place/caller of Y" is mechanically falsifiable; assert it. | S | candidate — narrow by construction | Covers ~60 of the 2652 production invariant comments. The other ~97% of the vocabulary (`never`, `always`, `must not`, `guarantees`) is not mechanical and a gate must not pretend otherwise. Instance: R-263. | +| **G-6** | **C2 — NOT mechanically gateable.** | — | **recorded as a no** | "Names a route" is a judgement, not a predicate. The most a check could do is enforce a *convention* (e.g. every customer-visible refusal string ends in an imperative clause), which would be gamed rather than followed. Better served by the UI-copy review the `felhom-ui-design` skill already governs. Instances: R-256, R-257. | +| **G-7** | **C6 — NOT gateable, and the measurement is the finding.** | — | **recorded as a no, with evidence** | The class *looks* the most mechanical of the seven and is the one where the off-the-shelf tool measurably fails: `golang.org/x/tools/cmd/deadcode` re-found **neither** known instance, and a **planted probe** showed why — it reports an unreachable exported FUNCTION but not an unreachable exported METHOD on a widely-used type, and both known instances are methods. A bespoke gate would be conservative by construction and would spend most of its output on **inert dead accessors** (4 of the 5 residual candidates were inert). **Recommendation: gate C5 instead** — it catches a strict subset of the same "the answer was available and discarded" family with none of the ambiguity. → R-261 | +| **G-8** | **R-242's untouched half — catch a SKIPPED VOUCH.** | S then M | candidate — **the one with a live recurrence** | Measured during Campaign 12's own bake: `golden_currency_gate.py` **flipped red→green the moment the evidence DIRECTORY existed**, before the round-trip download finished and with no vouch near it. **(a) Cheapest, now:** have the bake session re-read `/configuration` after the operator's Save and write the observed `golden_version` into the evidence README as a machine-readable line; the gate then requires that line rather than the directory. Detects the FORGETTING, which is the actual failure mode. **(b) Loudest, when the hub is next touched:** a hub-side daily check comparing the vouched golden against the newest controller the fleet reports — it fires within a day and catches a silent ROLLBACK too, which nothing in git can ever see. **(c) A checklist item is not a fix** — R-242 already was a rule without a mechanism and it recurred the next day. → R-242 | + ## P3 — post-alpha | ID | Item | Size | Status | Notes | diff --git a/documentation/tests/golden-0.208.0-2026-08-08/README.md b/documentation/tests/golden-0.208.0-2026-08-08/README.md new file mode 100644 index 0000000..58c3a89 --- /dev/null +++ b/documentation/tests/golden-0.208.0-2026-08-08/README.md @@ -0,0 +1,140 @@ +# Golden 0.208.0 — baked, published, round-trip verified, **NOT VOUCHED** (2026-08-08) + +**Closes the delivery gap opened by controller v0.208.0** (R-254 — the last two secrets leaving the +page source, plus the gate against a fourth). Before this bake `golden_currency_gate.py` was correctly +red and **a machine installed tonight would have received 0.207.0**. Evidence: `bake.log` (324 lines, +beside this file). + +**⚠ THE VOUCH IS DELIBERATELY NOT DONE.** Campaign 12 §3 halts here and asks the operator for it. The +hub still serves **0.207.0** to fresh installs until someone presses Save. See "The vouch, awaiting +the operator" below. + +## Where it ran + +The **drill VM on DooPlex** (`/mnt/5_hdd/felhom.eu/drill/drill.qcow2`, internal snapshot `virgin`) — +the accepted Tier-2 exception **for bakes**, per `runbooks/target-selection.md`. Launched with the +canonical §4.0 line (`if=virtio` is load-bearing; the guest expects `/dev/vda`), cold-booted because +the snapshot is disk-only (0 B `VM_SIZE`). **Restored to `virgin` afterwards**, snapshot list re-read. + +Liveness was checked with `ps -eo comm | grep -c qemu-system-x86`, never `pgrep -f`, which +self-matches its own command line and reports a false "still running". + +## The inputs, listed rather than assumed + +| | | +|---|---| +| host | `drill-pve`, `pve-manager/9.2.2` (kernel 7.0.2-6-pve), 40 G free on `/` | +| template | **`debian-13-standard_13.6-1_amd64.tar.zst`** — listed with `pveam available` on the day, because *the exact point release rots*; checksum verified on download | +| build script | `felhom-agent/configs/build-golden.sh` **v3.0.0**, from a clean tree at `703db166e762` (`git status --porcelain` empty, `HEAD == origin/main`, both checked before the copy) | +| controller baked | `gitea.dooplex.hu/admin/felhom-controller:0.208.0` — a **required** argument; `GOLDEN_VERSION` is derived from it. Registry manifest confirmed HTTP 200 before the bake started | + +## The token never crossed a shell + +Copied **file → file** (`scp ~/.gitea-token → /root/.gitea-token`, `0600`), and the bake launched +through a runner script **inside** the VM that reads the file itself. The older `--setenv=` form put +the value where `systemctl show` prints it. + +``` +systemctl show golden-bake -p Environment -p ExecStart | grep -c -F "$(cat /root/.gitea-token)" + → 0 +``` + +## The 404 pre-gate, with a control so a 404 cannot mean "wrong URL" + +``` +felhom-golden/0.207.0/golden.tar.zst → 200 ← the control: the URL SHAPE is right +felhom-golden/0.208.0/golden.tar.zst → 404 ← the pre-gate: nothing to overwrite +``` + +The filename is `golden.tar.zst`, **not** `felhom-golden-.tar.zst` — a wrong URL 404s for the +wrong reason and passes even when the version already exists (R-233). + +## Acceptance markers — each grepped verbatim against this run's real log + +| marker | count | verbatim | +|---|---|---| +| storage driver | **1** | ` docker OK (overlay2; data-root /var/lib/docker)` | +| mount points — rootfs **and mp0**, and **there is no mp1** | **2** | `INFO: including mount point rootfs ('/') in backup` · `INFO: including mount point mp0 ('/var/lib/felhom') in backup` | +| upload | **1** | `[golden] upload OK (HTTP 201)` | +| `excluding` | **0** | — | +| `FATAL` | **0** | — | +| `mp1` | **0** | — (it stopped existing in `build-golden.sh` v3.0.0, R-165) | + +`Result=success`, `ExecMainStatus=0`. Archive 625 MB; volid +`local:backup/vzdump-lxc-9100-2026_08_07-22_38_43.tar.zst` (rootfs 32 G + ONE data volume 24 G at +`/var/lib/felhom`, all in the archive). + +## The publish, and the round trip — the published BYTES, not the local file + +``` +GOLDEN_VERSION=0.208.0 +GOLDEN_SHA256=ba668f59f063511dda122a5b05900f1350573cdc9619980872f87a58755ffb82 +``` + +Downloaded back from `https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.208.0/golden.tar.zst`: + +| | | +|---|---| +| size | **656 150 362 B** — identical to what the bake reported | +| sha256 | **`ba668f59…5ffb82`** — hashed independently on DooPlex; identical | +| **`./etc/felhom-controller-image` read OUT of the downloaded archive** | **`gitea.dooplex.hu/admin/felhom-controller:0.208.0`** | + +**That last line is the one that matters:** it is not a claim about what was built, it is the delivered +artifact saying which controller it will start. + +**A fourth, independent confirmation:** the sha the hub's own dropdown offers for 0.208.0 +(`data-sha="ba668f59…"`, read from `/configuration`) matches the bake and the round trip. The package +the hub would serve is the one baked here. + +## The vouch, awaiting the operator — a THREE-field change, all three checked + +`MinAgent` was **read from the golden's controller `CHANGELOG.md` header**, not assumed: +`## v0.208.0 — … — MinAgent 0.127.0`. + +| field | now (live `hub_settings`) | after the vouch | why | +|---|---|---|---| +| `golden_version` | 0.207.0 | **0.208.0** | the new bake — **this is the only field that moves** | +| `agent_version` | 0.127.0 | **0.127.0** (unchanged) | already **≥ MinAgent 0.127.0** — checked, not skipped | +| `min_agent` | 0.127.0 | **0.127.0** (unchanged) | equals the controller's declared `MinAgent` | + +**`min_agent` is not above `agent_version`** — that is the R-216 shape, which hub v0.97.0 HOLDS rather +than serving past. **In the event this is a one-field move, and it is recorded as three because +checking all three is the point:** `golden_version` alone ships a controller onto an older agent than +it declares it needs. + +**How:** hub → Configuration → Day-0 artifacts → Golden `0.208.0` → Save. The R-120 gate sits on that +save and REFUSES a golden older than the newest controller the fleet reports; a refusal redirects to +`?flash=golden_behind_fleet` and writes nothing. + +**Vouching is reversible** — re-select `0.207.0` and Save. The 0.207.0 package is **not** deleted by +this bake (the publish step's pre-delete targets only its own version), so rolling back is a form +submission, not a rebuild. + +## Teardown + +`pct destroy 9100 --purge` (`pct list` empty after) · token, runner script, build script and in-VM log +`shred -u`'d, `/root` residue listing clean (`.bashrc .forward .profile .ssh` only) · `poweroff` · +qemu confirmed exited via `ps -eo comm | grep -c qemu-system-x86` → **0** · +**`qemu-img snapshot -a virgin` restored**, snapshot list re-read. + +**Token-leak grep on the log that is COMMITTED here — and a `0` is only evidence once the grep is +shown to work:** + +``` +grep -c -F "" bake.log → 0 +grep -c -F "" → 1 ← the control; copy then shredded +``` + +## What this does and does not change + +**Does:** the golden exists, is published, and is byte-verified. `golden_currency_gate.py` is **green** +again. + +**Does NOT:** **a fresh Day-0 install still lands on controller 0.207.0.** The gate checks the BAKE, +not the vouch — its own docstring says so, and this bake demonstrated the gap concretely: + +> **The gate flipped from red to green the moment this evidence DIRECTORY existed** — before the +> round-trip download had finished, and with no vouch anywhere near it. Its positive signal is "a +> directory named after the version is present in the repo". That is R-242's remaining half, measured +> rather than argued: **nothing anywhere catches a skipped vouch.** Recorded, not built — see the +> Campaign 12 report §8. diff --git a/documentation/tests/golden-0.208.0-2026-08-08/bake.log b/documentation/tests/golden-0.208.0-2026-08-08/bake.log new file mode 100644 index 0000000..6885c99 --- /dev/null +++ b/documentation/tests/golden-0.208.0-2026-08-08/bake.log @@ -0,0 +1,324 @@ +[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.208.0 +[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) … + Logical volume "vm-9100-disk-0" created. + Logical volume pve/vm-9100-disk-0 changed. +Creating filesystem with 8388608 4k blocks and 2097152 inodes +Filesystem UUID: 264a5264-ce5b-4cbd-9df0-57815cd6afb7 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, + 4096000, 7962624 + Logical volume "vm-9100-disk-1" created. + Logical volume pve/vm-9100-disk-1 changed. +Creating filesystem with 6291456 4k blocks and 1572864 inodes +Filesystem UUID: 39af4c84-4e25-465c-b45f-1f893f74b7d0 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, +extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst' +Total bytes read: 553512960 (528MiB, 168MiB/s) +Detected container architecture: amd64 +Creating SSH host key 'ssh_host_rsa_key' - this may take some time ... +done: SHA256:T1eshshAyQmblC2kzN0Orl8OUpUUcz436Okl9Ormfbk root@felhom-golden +Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ... +done: SHA256:N66mrs1433iZzugSQMgS5M/V14khU/dyjK8qCg+MB08 root@felhom-golden +Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ... +done: SHA256:tfiZgN0Pz+e/ZbICqw/WGJ3yGHVUf7F6DAx6Mbtku7c root@felhom-golden +[golden] starting + installing Docker (official repo, trixie channel) … +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation … +[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds … +[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) … +Unable to find image 'hello-world:latest' locally +latest: Pulling from library/hello-world +4f55086f7dd0: Pulling fs layer +4f55086f7dd0: Download complete +4f55086f7dd0: Pull complete +Digest: sha256:7f4da0fc94bcece205a8c0b6f4d11c8196924654ffe5c4d1aa439b7f632048b2 +Status: Downloaded newer image for hello-world:latest + docker OK (overlay2; data-root /var/lib/docker) + /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4 + /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4 + both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576 +[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.208.0 (no registry cred at deploy) … + +WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'. +Configure a credential helper to remove this warning. See +https://docs.docker.com/go/credential-store/ + +0.208.0: Pulling from admin/felhom-controller +039e6f9f9752: Pulling fs layer +80fbb63a1167: Pulling fs layer +9d6c7a1331cb: Pulling fs layer +3f0ff4614786: Pulling fs layer +97ccbd1fb75e: Pulling fs layer +ff4c40ddcdf8: Pulling fs layer +3f0ff4614786: Waiting +97ccbd1fb75e: Waiting +ff4c40ddcdf8: Waiting +039e6f9f9752: Verifying Checksum +039e6f9f9752: Download complete +9d6c7a1331cb: Verifying Checksum +9d6c7a1331cb: Download complete +3f0ff4614786: Verifying Checksum +3f0ff4614786: Download complete +97ccbd1fb75e: Verifying Checksum +97ccbd1fb75e: Download complete +ff4c40ddcdf8: Verifying Checksum +ff4c40ddcdf8: Download complete +80fbb63a1167: Verifying Checksum +80fbb63a1167: Download complete +039e6f9f9752: Pull complete +80fbb63a1167: Pull complete +9d6c7a1331cb: Pull complete +3f0ff4614786: Pull complete +97ccbd1fb75e: Pull complete +ff4c40ddcdf8: Pull complete +Digest: sha256:398b8fdfed6f36cc3b5e0ef960b608c604d2be4378e60829cb1236419f9de9a7 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.208.0 +gitea.dooplex.hu/admin/felhom-controller:0.208.0 +[golden] asking the controller which infra images it manages … +[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 … +v3.6.7: Pulling from library/traefik +589002ba0eae: Pulling fs layer +ef63511ea6cc: Pulling fs layer +0738e5cb835e: Pulling fs layer +3e6813f70c64: Pulling fs layer +3e6813f70c64: Waiting +ef63511ea6cc: Download complete +589002ba0eae: Verifying Checksum +589002ba0eae: Download complete +3e6813f70c64: Verifying Checksum +3e6813f70c64: Download complete +0738e5cb835e: Verifying Checksum +0738e5cb835e: Download complete +589002ba0eae: Pull complete +ef63511ea6cc: Pull complete +0738e5cb835e: Pull complete +3e6813f70c64: Pull complete +Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a +Status: Downloaded newer image for traefik:v3.6.7 +docker.io/library/traefik:v3.6.7 +2026.6.0: Pulling from cloudflare/cloudflared +47de5dd0b812: Pulling fs layer +c172f21841df: Pulling fs layer +99515e7b4d35: Pulling fs layer +99ba982a9142: Pulling fs layer +d6b1b89eccac: Pulling fs layer +2780920e5dbf: Pulling fs layer +7c12895b777b: Pulling fs layer +3214acf345c0: Pulling fs layer +52630fc75a18: Pulling fs layer +dd64bf2dd177: Pulling fs layer +b839dfae01f6: Pulling fs layer +ebddc55facdc: Pulling fs layer +bdfd7f7e5bf6: Pulling fs layer +2d4d7adf6272: Pulling fs layer +40008157d8d2: Pulling fs layer +bd8962e29291: Pulling fs layer +cac2ae0193cb: Pulling fs layer +74d1dac84ecc: Pulling fs layer +99ba982a9142: Waiting +dd64bf2dd177: Waiting +b839dfae01f6: Waiting +ebddc55facdc: Waiting +bdfd7f7e5bf6: Waiting +2d4d7adf6272: Waiting +40008157d8d2: Waiting +bd8962e29291: Waiting +cac2ae0193cb: Waiting +74d1dac84ecc: Waiting +d6b1b89eccac: Waiting +2780920e5dbf: Waiting +7c12895b777b: Waiting +3214acf345c0: Waiting +52630fc75a18: Waiting +47de5dd0b812: Download complete +c172f21841df: Verifying Checksum +c172f21841df: Download complete +99515e7b4d35: Verifying Checksum +99515e7b4d35: Download complete +47de5dd0b812: Pull complete +99ba982a9142: Verifying Checksum +99ba982a9142: Download complete +d6b1b89eccac: Verifying Checksum +d6b1b89eccac: Download complete +2780920e5dbf: Verifying Checksum +2780920e5dbf: Download complete +7c12895b777b: Download complete +3214acf345c0: Verifying Checksum +3214acf345c0: Download complete +52630fc75a18: Verifying Checksum +52630fc75a18: Download complete +dd64bf2dd177: Verifying Checksum +dd64bf2dd177: Download complete +b839dfae01f6: Verifying Checksum +b839dfae01f6: Download complete +ebddc55facdc: Verifying Checksum +ebddc55facdc: Download complete +bdfd7f7e5bf6: Verifying Checksum +bdfd7f7e5bf6: Download complete +2d4d7adf6272: Verifying Checksum +2d4d7adf6272: Download complete +c172f21841df: Pull complete +bd8962e29291: Verifying Checksum +bd8962e29291: Download complete +40008157d8d2: Verifying Checksum +40008157d8d2: Download complete +cac2ae0193cb: Verifying Checksum +74d1dac84ecc: Verifying Checksum +74d1dac84ecc: Download complete +99515e7b4d35: Pull complete +99ba982a9142: Pull complete +d6b1b89eccac: Pull complete +2780920e5dbf: Pull complete +7c12895b777b: Pull complete +3214acf345c0: Pull complete +52630fc75a18: Pull complete +dd64bf2dd177: Pull complete +b839dfae01f6: Pull complete +ebddc55facdc: Pull complete +bdfd7f7e5bf6: Pull complete +2d4d7adf6272: Pull complete +40008157d8d2: Pull complete +bd8962e29291: Pull complete +cac2ae0193cb: Pull complete +74d1dac84ecc: Pull complete +Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f +Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0 +docker.io/cloudflare/cloudflared:2026.6.0 +1.3.3-stable: Pulling from gtstef/filebrowser +6a0ac1617861: Pulling fs layer +ef8806083e82: Pulling fs layer +b74107c861c7: Pulling fs layer +adc935def003: Pulling fs layer +4f4fb700ef54: Pulling fs layer +18695ccc900a: Pulling fs layer +45d119d5c397: Pulling fs layer +dac52db4fc51: Pulling fs layer +6d598f86b2f2: Pulling fs layer +8aa349c8396c: Pulling fs layer +adc935def003: Waiting +8aa349c8396c: Waiting +4f4fb700ef54: Waiting +18695ccc900a: Waiting +45d119d5c397: Waiting +dac52db4fc51: Waiting +6d598f86b2f2: Waiting +6a0ac1617861: Verifying Checksum +6a0ac1617861: Download complete +adc935def003: Verifying Checksum +adc935def003: Download complete +4f4fb700ef54: Verifying Checksum +4f4fb700ef54: Download complete +b74107c861c7: Verifying Checksum +b74107c861c7: Download complete +ef8806083e82: Verifying Checksum +ef8806083e82: Download complete +45d119d5c397: Verifying Checksum +45d119d5c397: Download complete +6a0ac1617861: Pull complete +dac52db4fc51: Verifying Checksum +dac52db4fc51: Download complete +6d598f86b2f2: Verifying Checksum +6d598f86b2f2: Download complete +18695ccc900a: Verifying Checksum +18695ccc900a: Download complete +8aa349c8396c: Verifying Checksum +8aa349c8396c: Download complete +ef8806083e82: Pull complete +b74107c861c7: Pull complete +adc935def003: Pull complete +4f4fb700ef54: Pull complete +18695ccc900a: Pull complete +45d119d5c397: Pull complete +dac52db4fc51: Pull complete +6d598f86b2f2: Pull complete +8aa349c8396c: Pull complete +Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c +Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable +docker.io/gtstef/filebrowser:1.3.3-stable +1.1.0: Pulling from admin/felhom-samba +897d797d2723: Pulling fs layer +3051591aa250: Pulling fs layer +ce57a3f93416: Pulling fs layer +fb94eeec2fe1: Pulling fs layer +fb94eeec2fe1: Waiting +ce57a3f93416: Verifying Checksum +ce57a3f93416: Download complete +fb94eeec2fe1: Verifying Checksum +fb94eeec2fe1: Download complete +897d797d2723: Verifying Checksum +897d797d2723: Download complete +897d797d2723: Pull complete +3051591aa250: Verifying Checksum +3051591aa250: Download complete +3051591aa250: Pull complete +ce57a3f93416: Pull complete +fb94eeec2fe1: Pull complete +Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0 +gitea.dooplex.hu/admin/felhom-samba:1.1.0 +[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'. +[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'. +[golden] baking the first-boot SSH host-key regeneration unit (F3) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'. +[golden] identity-clean + minimize … +[golden] stop + archive … +INFO: including mount point rootfs ('/') in backup +INFO: including mount point mp0 ('/var/lib/felhom') in backup +INFO: archive file size: 625MB +INFO: Finished Backup of VM 9100 (00:00:35) +[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_08_07-22_38_43.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive) +[golden] publishing golden (656150362 bytes, sha256 ba668f59f063511d…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.208.0/golden.tar.zst +[golden] pre-delete existing: HTTP 404 (404/204 expected) +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.208.0 +GOLDEN_SHA256=ba668f59f063511dda122a5b05900f1350573cdc9619980872f87a58755ffb82 +[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.208.0 / ba668f59f063511dda122a5b05900f1350573cdc9619980872f87a58755ffb82 +[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)