felhom-controller/REPORT.md

REPORT — v0.234.0, update arc slices 1, 1b & 2 (2026-09-03)

Overwritten each run. This records the most recent implementation only.

v0.234.0 (2026-09-03) — read this first. v0.233.0 shipped with "the record only appears after the next lifecycle action" written down as an ACCEPTED LIMITATION. The operator found it the next morning and it was a defect: OpenGist on demo-felhom, up 15 hours, running exactly the catalog pin, showing no badge at all. On a quiet box "fills in gradually" means "never", and a feature that fills itself in on an event nobody triggers is, on the quiet installations, not shipped. Manager.BackfillInstalledImages now seeds the absences at startup by READING containers — it starts nothing, restarts nothing and writes no compose file. Proven live on both demo boxes (§6b). A test of mine also went red overnight and that is §6c.

Read this second: one claim in the task turned out to be wrong, and it is a path, not a fact. The task cites source as internal/stacks/deploy.go, internal/web/funcmap.go and so on. The Go module lives under controller/ — every one of those is controller/internal/.... Every line number in the task was EXACT against the baseline (AppConfig 99, runComposeDeploy 395, LoadAppConfig 792, SaveAppConfig 806, SensitiveEnvVars 889, stackEnv 1233, composeExecCustomEnv 1268, execCommand 1344, logPostStartStatus 1382, checkLocalImages 1410, Metadata 14, isOperationalState 90) — checked, not assumed. The rest of §10 is at the end.

---

1. Confirmed baselines — none had moved

repo	task's baseline	found	note
felhom-controller	960d29b0612c	960d29b0612c	matched
felhom.eu	56c7e373a3e2	56c7e373a3e2	matched
app-catalog-felhom.eu	5d8f25f61189	5d8f25f61189	matched
felhom-agent	4586f0f7f6d1	4586f0f7f6d1	not touched

Highest R- id: 445, confirmed. Minted R-446..R-453 (eight, one more than the task anticipated — R-453 is the credentials-file finding in §7 — a mistake of mine, not the box's).

2. Files created and modified

Created: controller/internal/stacks/installed.go, controller/internal/stacks/installed_test.go, controller/internal/web/updatebadge.go, controller/internal/web/updatebadge_test.go.

Modified: controller/internal/stacks/deploy.go (the two new AppConfig fields + the deploy-path call), controller/internal/stacks/manager.go (the seam field, Stack.TemplateImages, ScanStacks, three call sites), controller/internal/stacks/metadata.go (CatalogSince + CatalogSinceAge + the tolerance WARN), controller/internal/web/funcmap.go, controller/internal/web/templates/app_info.html, controller/internal/web/templates/stacks.html, plus CHANGELOG.md, CONTEXT.md, REUSE.md, controller/README.md.

3. Commits pushed to main

repo	commit	what
felhom-controller	8025304acc0a6737	v0.233.0 — the record and the badge
felhom-controller	38d28b5b624c	v0.234.0 — the startup backfill + the calendar-bomb fix
app-catalog-felhom.eu	69761cf91bfc	catalog_since on all 53 apps + the CLAUDE.md rule
app-catalog-felhom.eu	8220f8dc…	the catalog's own REPORT
felhom.eu	6035dfcc3ae1	09-update-architecture.md, the register, roadmap, capability map, STATUS, live evidence
felhom.eu	e86cf42e0ba5	R-454..R-456, filed because the observations gate refused a report that filed none
felhom.eu	bc47dd4ef997	v0.234.0 docs: the living architecture doc's limitation 3 struck, R-457, capability map, STATUS

No branches. All three gate runs passed on push (controller: 13 gates OK + 1 advisory, see §9).

4. Tests, and both companion red-proofs

1707 → 1729 test functions (+22; +17 in v0.233.0, +5 in v0.234.0). 28 packages, 0 FAIL, go build ./... && go vet ./... && go test ./....

group	what it pins
A	a MULTI-service fixture records one entry PER COMPOSE SERVICE with digests and a parseable RFC3339 at; an image with no RepoDigests is recorded with an EMPTY digest, never skipped; a partial read is recorded AND logged with the count and the missing service names
B	the record follows the CONTAINER, not the file — the fixture's compose says v2.8.5 while the container runs v2.8.6 and the record carries v2.8.6; an unchanged observation does NOT rewrite app.yaml and at is carried forward
C	an unwritable app.yaml does not fail the action — driven through a real RestartStack
D	the badge's states incl. no-record-renders-nothing, no version string ever, and nothing badged that cannot be judged (undeployed / protected / orphaned); plus a RENDER of both production templates
E	the wiring — RestartStack reached end to end, plus an AST walk of the four call sites
F	catalog_since tolerance: absent, blank, tegnap, 18/07/2026, 2026-13-45, and a FUTURE date

Three companion red-proofs, each run, observed failing, and reverted (2026-09-02):

#	mutation	observed failure
1	give recordInstalledImages an error return and make RestartStack return it	TestGroupC fails: "restart must SUCCEED even when the record cannot be written: … permission denied" — the customer's app refuses to start because a note could not be written
2	the no-record guard returns updateCurrent instead of updateUnknown	three sub-tests fail, incl. badge = &{Label:Naprakész …}, want present=false on an app nobody has ever measured
3	delete the {{template "meta_badge" (updateBadge …)}} line from each template in turn	stacks.html → "the behind badge is missing from stacks"; app_info.html → the same on app_info

All three reverted; the full suite re-run green afterwards.

Seam discipline. The recorder has its own seam (installedExecFn) that FAILS the test on an argv it does not recognise. The wiring test does not use it to reach the recorder: it stubs the compose binary on PATH and drives the real RestartStack. The AST walk exists because a strings.Contains matches a commented-out call, which is exactly this project's seam-built-but-never-wired class.

5. Deployed version

$ ssh hp "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"
gitea.dooplex.hu/admin/felhom-controller:0.233.0 Up 19 seconds (healthy)

Image digest sha256:df5940ccf5a548ceee9065a2cc55a467f8941c636138441e0d77e320dc2023ab. Previous: 0.232.0.

The build was blocked and the workaround is recorded rather than buried. Docker Hub returned 429 toomanyrequests for debian:bookworm-slim and golang:1.24-bookworm, so build.sh could not resolve either base image; DooPlex holds no Docker Hub login. Both were pulled from Google's official Docker Hub mirror and retagged locally, after which build.sh ran unmodified:

debian:bookworm-slim      image id  sha256:88200866dfff7ea7f5cbcb6ec7c8a701889efe6fe859fe64d6990e4b07ea4171
golang:1.24-bookworm      image id  sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac

MEASURED — the identity claim is no longer taken on trust. The throttle cleared 40 minutes later and the check was run rather than left for someone else:

• docker pull docker.io/library/debian:bookworm-slim and …/golang:1.24-bookworm both answered Status: Image is up to date, i.e. Docker Hub's own manifest resolved to the images already local — the ones the mirror supplied and the ones 0.233.0 was built from. The image ids above are unchanged after the Hub pull. That is the decisive test: same bytes, by Docker's own resolution.
• Independently, the two manifest indexes are identical between the registries: sha256 of the docker manifest inspect body is 959bc47a76ff713a… (debian) and de3a17b36657e232… (golang) on both docker.io/library/… and mirror.gcr.io/library/….

So the shipped 0.233.0 image is byte-for-byte what a Docker-Hub build would have produced. No host configuration was changed and no daemon was restarted. (Note for the next reader: the two sha256 values above are local IMAGE IDs, not manifest-list digests — the two are easy to confuse and were confused once during this check.)

6. Live validation — what was proven, and by which method

Full evidence: felhom.eu/documentation/tests/VALIDATION-update-slice12-2026-09-02.md.

Method, stated: the BOOT RECONCILER — bootrecon.Run → StartStack → compose up -d → recordInstalledImages, a real production caller, the same one the spike used, with no hand-set state. Not the customer's Restart button, and §7 is why.

Ground truth was read from the containers before anything was touched, independently:

bentopdf      ghcr.io/alam00000/bentopdf:v2.8.6      @sha256:eaeea1e447205a79cb61d7efdc6966f37311dc1bc9c36a3a5c897bf79107c2c3
bookstack     lscr.io/linuxserver/bookstack:26.05.2  @sha256:3db259db582808ab498d49ae96b0a63f935d9cf3635c9d5bd8b8815c6ff1f8a1
bookstack-db  mariadb:12.3                           @sha256:a02fe89cb597d4375812b2eac90cf9d0775d4686daa7f7cc750ebbcad7525bbc

Before: nine deployed apps, ZERO with a record. After, the multi-service case — the one that matters, because one entry for the whole stack is the wrong outcome:

18:28:48 bootrecon.go:259: [INFO] Boot reconciliation: 1 boot-orphaned app(s) found: [bookstack]
18:28:54 installed.go:408: [INFO] [stacks] installed-images bookstack: recorded 2 service(s)
                                  (bookstack=lscr.io/linuxserver/bookstack:26.05.2 (sha256:3db259db5828…),
                                   bookstack-db=mariadb:12.3 (sha256:a02fe89cb597…))

installed_images:
    bookstack:
        ref: lscr.io/linuxserver/bookstack:26.05.2
        digest: sha256:3db259db582808ab498d49ae96b0a63f935d9cf3635c9d5bd8b8815c6ff1f8a1
        at: "2026-09-02T18:28:54Z"
    bookstack-db:
        ref: mariadb:12.3
        digest: sha256:a02fe89cb597d4375812b2eac90cf9d0775d4686daa7f7cc750ebbcad7525bbc
        at: "2026-09-02T18:28:54Z"

All three recorded digests match the independently-read ground truth exactly. bookstack's two ENC: secrets are byte-identical before and after, as are deployed, deployed_at, locked_fields and desired_state. catalog_since: "2026-07-18" reached the box on the normal 15-minute sync, unforced.

The badge, quoted from the LIVE pages

<span class="tag tag-ok" title="Ez az alkalmazás a legfrissebb elérhető változatot futtatja.">Naprakész</span>
<span class="tag tag-warn" title="Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.">Frissítés elérhető — 52 napja</span>

Byte-identical to the task's string table. Three of the four states are live, on BOTH surfaces:

state	where	observed
current	/stacks ×2, /apps/bookstack ×1	„Naprakész", tag-ok
behind, age known	/stacks ×1, /apps/bentopdf ×1	„Frissítés elérhető — 52 napja", tag-warn
no record	/apps/docmost — a DEPLOYED app that has not restarted since the upgrade	nothing rendered. The load-bearing case: absent is UNKNOWN, not current
behind, age unknown	—	not reachable live: all 53 catalog apps now carry a valid catalog_since. TestGroupF only.

52 napja is arithmetic on a real value, not a placeholder: bentopdf's catalog_since is 2026-07-12, the run is 2026-09-02.

How the behind state was staged, stated because it matters: bentopdf's docker-compose.yml tag was edited v2.8.6 → v2.8.5 and nothing else — no restart, no up -d, the container never touched — then reverted, sha256 equal both sides (39679e28cdd6…), diff empty, and the badge returned to „Naprakész". So the RENDER is measured; the syncer's own half stays separately measured in the spike. bentopdf was chosen because it has no database, no volume and no data of any kind.

Nothing about updating changed, checked on the live page in the behind state: update ×1, restart ×1, stop ×1 for bentopdf. The badge is wired to nothing.

ASCII-fragment counts, grep -oF, with positive AND negative controls

fragment	/stacks	/apps/bookstack	/apps/docmost	/stacks (behind)	/apps/bentopdf (behind)
Naprak	2	1	0	1	0
napja	0	0	0	1	1
52 napja	0	0	0	1	1
BookStack / Docmost (positive)	1 / 1	4 / 0	0 / 5	—	—
zzz-never-present (negative)	0	0	0	0	0
Nem-karbantartott-XYZ (negative)	0	0	0	—	—

The same fragments in the DEPLOYED BINARY, as a second observable on the shipped artifact:

Hungarian strings, in the DEPLOYED binary, ASCII fragments with both controls — a positive observable on the shipped artifact, and not a rendered page:

fragment	count
Naprak	3	positive
Friss	25	positive
napja	2	positive
legfrissebb	1	positive (the hover text)
zzz-never-present-control	0	negative control
Nem-karbantartott-XYZ	0	negative control

End state: all three containers back on the same images, both named volumes untouched, both apps healthy. This run provisioned nothing — no guest, no storage, no hub record — so there is no teardown to report on any of the three layers.

6b. v0.234.0 — the startup backfill, proven live on both boxes

The honest complication first: by the time 0.234.0 was ready, all nine deployed apps on demo-hp and the one on demo-felhom already carried records — the overnight backup cycle had restarted them and v0.233.0's recorder fired on every one (opengist is stamped 2026-09-03T00:31:11Z). The natural fleet state could no longer exercise the new code.

The pre-0.233.0 shape was therefore recreated on demo-hp (Tier 0): the installed_images: block was deleted from privatebin (1 service) and romm (3 services) — a record, never data — and the controller restarted. Ground truth was read from the containers first.

09:59:45 installed-images backfill: privatebin recorded 1 service(s) (privatebin/pdo:2.0.5 (sha256:8a2cac16eff6…))
09:59:45 installed-images backfill: romm recorded 3 service(s) (rommapp/romm:5.0.0 (sha256:91f6611eca5a…),
                                    mariadb:11.4 (sha256:4f1d8d202fcf…), redis:7-alpine (sha256:ff02b58f971e…))
09:59:45 installed-images backfill: 2 app(s) recorded, 7 already had a record, 0 left unrecorded

• Exactly the two stripped apps were seeded; the other seven were untouched — the never-overwrite rule observed, not asserted.
• Every digest matches the independently-read ground truth.
• /stacks then carried „Naprakész" ×9, one per deployed app, negative control zzz-never-present at 0.
• On demo-felhom the operator's own case renders the badge on /stacks?filter=running and /apps/opengist.
• Backups removed from the box; nothing provisioned, no app started, stopped or upgraded.

NOT staged live, and the choice is the point: the refusal half — that a partial observation is not seeded — needs a degraded app, and manufacturing one is a dead-app alarm candidate. This project has already paid for 61 false customer e-mails from that class (R-330), so it is covered by TestGroupG_BackfillRefusesAPartialObservation with a companion red-proof (remove the guard → "backfilled 1, want 0") and recorded here as unproven-live.

6c. A test of mine was a calendar bomb, and the suite caught it

TestGroupD_BadgeRendersOnBothSurfaces hardcoded a fixture catalog_since: "2026-07-18" and the expected string "Frissítés elérhető — 46 napja". The pure badge tests inject a clock; the render test cannot — it goes through the production templates, which call the funcmap entry, which reads time.Now(). Green on 2026-09-02, red on 2026-09-03 with "the behind badge is missing" on both surfaces, because the true answer had become 47.

Fixed by deriving the fixture: catalog_since is computed as today minus 46 days, so it asserts the real number through the real clock and cannot rot. FILED: R-457, which also names six other test files carrying both a date literal and time.Now() — as unchecked candidates, not accusations, because mixing the two is only a defect where the literal feeds a clock-evaluated assertion.

7. NOT yet live-validated — an explicit list

Everything the task asked to be validated live, was. What remains:

0. The backfill's REFUSAL of a partial observation — §6b explains why it was not staged live.

1. The fourth badge state, „Frissítés elérhető" WITHOUT an age. It needs an app whose catalog_since is absent, malformed or future-dated, and all 53 catalog apps now carry a valid one. Covered by TestGroupF (absent, blank, tegnap, 18/07/2026, 2026-13-45, future-dated).

2. POST /api/stacks/{name}/restart / /deploy as the trigger. The recorder was reached live through StartStack (the boot reconciler — a real production caller), and in a unit test through a real RestartStack. UpdateStack and the deploy path are covered by the AST walk. The restart ENDPOINT itself was not fired live; the code path it reaches was.

3. The abandoned + update double-badge, unit-tested only — no live app is abandoned on either box.

4. Any behaviour on a box other than demo-hp. demo-felhom is still on 0.232.0, deliberately: it was not in scope and its one deployed app adds nothing the multi-service case did not prove.

A correction of my own, stated before anything else in this section

I reported the vaulted dashboard password as stale on BOTH demo boxes, and it was not. ~/.config/credentials quotes its values with single quotes; my extraction stripped only double quotes, so the quote characters were sent as part of the password. The operator corrected it in one line and the retry returned 302 + felhom_session.

The instrumentation lesson is the part worth keeping, and R-453 now carries it: I quoted the controller's auth.go:176 [WARN] Failed login as the discriminator. It is a true one and it separates wrong password from wrong Host header — and that is ALL it separates. It cannot tell a wrong password from wrong password HANDLING, and I read it as though it could. A discriminator that rules out one alternative does not rule in the remaining one. This is the second time this file's quoting has produced a confident wrong verdict, so the fix filed is one shared extraction helper rather than a resolution to be careful.

8. Register — 194 rows before, 205 after. Nothing closed.

R-438 and R-440 amended and both stay OPEN — the mechanism is documented, not changed — so CLOSED-ITEMS.md is untouched, and this run's compression sweep is a no-op that says so.

New: R-446 floating-tag honesty (P2, CC) · R-447 slice 3, BLOCKED on an operator ruling (P1) · R-448 slice 4 (P2) · R-449 slice 5 (P2) · R-450 slice 6 (P2) · R-451 slice 7 (P3) · R-452 the catalog_since gate, deferred because the runner fetches at --depth 1 (P3) · R-453 (rewritten) the credentials file's SINGLE quotes, and a discriminator read past what it discriminates (P3) · R-454 five gofmt-unclean test files with no gate (P3) · R-455 DooPlex has no Docker Hub login and the ceiling now blocks builds (P2, WAITING-ON-OPERATOR) · R-456 a partly-dead stack is not a boot orphan and that is written down nowhere (P3).

9. Observations — noticed, documented, NOT acted on

1. The golden is one release behind. The push gate advises: newest released controller 0.233.0, newest golden baked 0.232.0. Baking and vouching a golden is a three-field change and was not in this task's scope. NOT-A-FINDING: the golden-notice gate raises this on every release and STATUS.md already carries the debt — a register row would duplicate an instrument that already fires by itself.
2. Five test files in internal/web/ are not gofmt-clean at the baseline — backups_split_test.go, claim_code_naming_test.go, disk_health_test.go, r400_debug_routes_test.go, recovery_test.go. Pre-existing; both files I added are clean. Not reformatted, under the minimal-changes rule. FILED: R-454.
3. The catalog's non-static gates could not be run to a verdict on this host. image-pins OK; image-resolvable INCONCLUSIVE (Docker Hub throttled 6 of 65 unauthenticated lookups — the same ceiling that blocked the build); volume-persistence INCONCLUSIVE (its own canary needs a scratch Docker host). Neither is caused by the change, and the runner exited 0. FILED: R-455 — the same ceiling blocked the BUILD (§5), which is a bigger bill than a gate that cannot reach a verdict.
4. bentopdf's catalog_since needed a decision, not just a script. The two spike commits that moved its pin and reverted it the same hour were excluded by hash; counting them would have dated it 2026-09-02 for a pin unchanged since 2026-07-12. The catalog's own CHANGELOG already calls them a measurement, not a release. NOT-A-FINDING: the decision and its reason are already recorded durably in app-catalog-felhom.eu's CHANGELOG.md and REPORT.md; it is a settled call, not an open question.
5. The boot reconciler does not treat a partly-dead stack as a boot orphan. Removing only bookstack's app container while its DB stayed up left it unselected; removing both made it an orphan. Correct-looking behaviour, recorded because it cost a second pass and is not written down anywhere. FILED: R-456.

10. Every claim in the task that turned out to be wrong, named

1. The source paths omit the module directory — everything is under controller/. All twelve line-number landmarks were exact, so this is a prefix, not drift.
2. app_info.html ~L48 and stacks.html ~L89 point at neighbouring places, not at the badge. L48 opens stack-meta-badges (a different badge row) and L89 is the Frissítés button. The meta_badge calls the new badge had to join are at L13 and L42. Both were found by reading, as instructed.
3. "No STOP in this task ... nothing is waiting on the operator." True, and it held — one operator turn was spent, and it was spent correcting a mistake of mine (§7), not on a decision the task owed them. STATUS.md item 9 records the result and asks for nothing.
4. Scenario A's stated route, POST /api/stacks/{name}/deploy, was not fired. Deploying an app just to prove the recorder would have left a throwaway on a live box; the recorder was reached live through StartStack instead (the boot reconciler), and the deploy call site is covered by a multi-service unit fixture plus the AST walk.
5. "446 looks next" — correct, and ELEVEN were needed rather than seven (R-446..R-456), because the observations gate refuses a report whose observations name no row, which is the right behaviour and is why four extra rows exist instead of four paragraphs that die with this file.
6. Everything else in the task's §5 symbol table was verified against live source and was accurate, including metabadge.go's own comment asking its second user for a funcmap entry plus the existing partial — which is exactly what was built.