felhom-controller/REPORT.md

# REPORT — v0.234.0, update arc slices 1, 1b & 2 (2026-09-03)

*Overwritten each run. This records the most recent implementation only.*

> **v0.234.0 (2026-09-03) — read this first.** v0.233.0 shipped with *"the record only appears after
> the next lifecycle action"* written down as an ACCEPTED LIMITATION. **The operator found it the next
> morning and it was a defect:** OpenGist on demo-felhom, up 15 hours, running exactly the catalog pin,
> showing **no badge at all**. On a quiet box "fills in gradually" means "never", and a feature that
> fills itself in on an event nobody triggers is, on the quiet installations, **not shipped**.
> `Manager.BackfillInstalledImages` now seeds the absences at startup by READING containers — it
> starts nothing, restarts nothing and writes no compose file. **Proven live on both demo boxes**
> (§6b). **A test of mine also went red overnight** and that is §6c.
>
> **Read this second: one claim in the task turned out to be wrong, and it is a path, not a fact.**
> The task cites source as `internal/stacks/deploy.go`, `internal/web/funcmap.go` and so on. **The Go
> module lives under `controller/`** — every one of those is `controller/internal/...`. **Every line
> number in the task was EXACT against the baseline** (`AppConfig` 99, `runComposeDeploy` 395,
> `LoadAppConfig` 792, `SaveAppConfig` 806, `SensitiveEnvVars` 889, `stackEnv` 1233,
> `composeExecCustomEnv` 1268, `execCommand` 1344, `logPostStartStatus` 1382, `checkLocalImages` 1410,
> `Metadata` 14, `isOperationalState` 90) — checked, not assumed. The rest of §10 is at the end.

---

## 1. Confirmed baselines — none had moved

| repo | task's baseline | found | note |
|---|---|---|---|
| felhom-controller | `960d29b0612c` | `960d29b0612c` | matched |
| felhom.eu | `56c7e373a3e2` | `56c7e373a3e2` | matched |
| app-catalog-felhom.eu | `5d8f25f61189` | `5d8f25f61189` | matched |
| felhom-agent | `4586f0f7f6d1` | `4586f0f7f6d1` | **not touched** |

Highest `R-` id: **445**, confirmed. Minted **R-446..R-453** (eight, one more than the task
anticipated — R-453 is the credentials-file finding in §7 — a mistake of mine, not the box's).

## 2. Files created and modified

**Created:** `controller/internal/stacks/installed.go`, `controller/internal/stacks/installed_test.go`,
`controller/internal/web/updatebadge.go`, `controller/internal/web/updatebadge_test.go`.

**Modified:** `controller/internal/stacks/deploy.go` (the two new `AppConfig` fields + the deploy-path
call), `controller/internal/stacks/manager.go` (the seam field, `Stack.TemplateImages`, `ScanStacks`,
three call sites), `controller/internal/stacks/metadata.go` (`CatalogSince` + `CatalogSinceAge` + the
tolerance WARN), `controller/internal/web/funcmap.go`, `controller/internal/web/templates/app_info.html`,
`controller/internal/web/templates/stacks.html`, plus `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`,
`controller/README.md`.

## 3. Commits pushed to `main`

| repo | commit | what |
|---|---|---|
| felhom-controller | **`8025304acc0a6737`** | v0.233.0 — the record and the badge |
| felhom-controller | **`38d28b5b624c`** | **v0.234.0 — the startup backfill + the calendar-bomb fix** |
| app-catalog-felhom.eu | `69761cf91bfc` | `catalog_since` on all 53 apps + the `CLAUDE.md` rule |
| app-catalog-felhom.eu | `8220f8dc…` | the catalog's own REPORT |
| felhom.eu | `6035dfcc3ae1` | `09-update-architecture.md`, the register, roadmap, capability map, STATUS, live evidence |
| felhom.eu | `e86cf42e0ba5` | R-454..R-456, filed because the observations gate refused a report that filed none |
| felhom.eu | `bc47dd4ef997` | v0.234.0 docs: the living architecture doc's limitation 3 struck, R-457, capability map, STATUS |

No branches. All three gate runs passed on push (controller: 13 gates OK + 1 advisory, see §9).

## 4. Tests, and both companion red-proofs

**1707 → 1729 test functions (+22; +17 in v0.233.0, +5 in v0.234.0). 28 packages, 0 FAIL**, `go build ./... && go vet ./... && go test ./...`.

| group | what it pins |
|---|---|
| **A** | a MULTI-service fixture records one entry PER COMPOSE SERVICE with digests and a parseable RFC3339 `at`; an image with no `RepoDigests` is recorded with an EMPTY digest, never skipped; a partial read is recorded AND logged with the count and the missing service names |
| **B** | the record follows the CONTAINER, not the file — the fixture's compose says `v2.8.5` while the container runs `v2.8.6` and the record carries `v2.8.6`; an unchanged observation does NOT rewrite `app.yaml` and `at` is carried forward |
| **C** | an unwritable `app.yaml` does not fail the action — driven through a real `RestartStack` |
| **D** | the badge's states incl. no-record-renders-nothing, no version string ever, and nothing badged that cannot be judged (undeployed / protected / orphaned); plus a RENDER of both production templates |
| **E** | **the wiring** — `RestartStack` reached end to end, plus an **AST walk** of the four call sites |
| **F** | `catalog_since` tolerance: absent, blank, `tegnap`, `18/07/2026`, `2026-13-45`, and a FUTURE date |

**Three companion red-proofs, each run, observed failing, and reverted (2026-09-02):**

| # | mutation | observed failure |
|---|---|---|
| 1 | give `recordInstalledImages` an `error` return and make `RestartStack` return it | `TestGroupC` fails: *"restart must SUCCEED even when the record cannot be written: … permission denied"* — the customer's app refuses to start because a note could not be written |
| 2 | the no-record guard returns `updateCurrent` instead of `updateUnknown` | three sub-tests fail, incl. `badge = &{Label:Naprakész …}, want present=false` on an app nobody has ever measured |
| 3 | delete the `{{template "meta_badge" (updateBadge …)}}` line from each template in turn | `stacks.html` → *"the behind badge is missing from stacks"*; `app_info.html` → the same on `app_info` |

All three reverted; the full suite re-run green afterwards.

**Seam discipline.** The recorder has its own seam (`installedExecFn`) that FAILS the test on an argv
it does not recognise. The wiring test does **not** use it to reach the recorder: it stubs the compose
binary on `PATH` and drives the real `RestartStack`. The AST walk exists because a
`strings.Contains` matches a commented-out call, which is exactly this project's
seam-built-but-never-wired class.

## 5. Deployed version

```
$ ssh hp "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"
gitea.dooplex.hu/admin/felhom-controller:0.233.0 Up 19 seconds (healthy)
```

Image digest `sha256:df5940ccf5a548ceee9065a2cc55a467f8941c636138441e0d77e320dc2023ab`. Previous: `0.232.0`.

**The build was blocked and the workaround is recorded rather than buried.** Docker Hub returned
`429 toomanyrequests` for `debian:bookworm-slim` and `golang:1.24-bookworm`, so `build.sh` could not
resolve either base image; DooPlex holds no Docker Hub login. Both were pulled from **Google's
official Docker Hub mirror** and retagged locally, after which `build.sh` ran unmodified:

```
debian:bookworm-slim      image id  sha256:88200866dfff7ea7f5cbcb6ec7c8a701889efe6fe859fe64d6990e4b07ea4171
golang:1.24-bookworm      image id  sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac
```

**MEASURED — the identity claim is no longer taken on trust.** The throttle cleared 40 minutes later
and the check was run rather than left for someone else:

- `docker pull docker.io/library/debian:bookworm-slim` and `…/golang:1.24-bookworm` both answered
  **`Status: Image is up to date`**, i.e. **Docker Hub's own manifest resolved to the images already
  local — the ones the mirror supplied and the ones 0.233.0 was built from.** The image ids above are
  unchanged after the Hub pull. That is the decisive test: same bytes, by Docker's own resolution.
- Independently, the two manifest indexes are identical between the registries: `sha256` of the
  `docker manifest inspect` body is `959bc47a76ff713a…` (debian) and `de3a17b36657e232…` (golang) on
  **both** `docker.io/library/…` and `mirror.gcr.io/library/…`.

**So the shipped 0.233.0 image is byte-for-byte what a Docker-Hub build would have produced.** No host
configuration was changed and no daemon was restarted. (Note for the next reader: the two `sha256`
values above are local IMAGE IDs, not manifest-list digests — the two are easy to confuse and were
confused once during this check.)

## 6. Live validation — what was proven, and by which method

**Full evidence: `felhom.eu/documentation/tests/VALIDATION-update-slice12-2026-09-02.md`.**

**Method, stated: the BOOT RECONCILER** — `bootrecon.Run → StartStack → compose up -d →
recordInstalledImages`, a real production caller, the same one the spike used, with no hand-set state.
**Not the customer's Restart button, and §7 is why.**

Ground truth was read from the containers **before anything was touched**, independently:

```
bentopdf      ghcr.io/alam00000/bentopdf:v2.8.6      @sha256:eaeea1e447205a79cb61d7efdc6966f37311dc1bc9c36a3a5c897bf79107c2c3
bookstack     lscr.io/linuxserver/bookstack:26.05.2  @sha256:3db259db582808ab498d49ae96b0a63f935d9cf3635c9d5bd8b8815c6ff1f8a1
bookstack-db  mariadb:12.3                           @sha256:a02fe89cb597d4375812b2eac90cf9d0775d4686daa7f7cc750ebbcad7525bbc
```

Before: **nine deployed apps, ZERO with a record.** After, the multi-service case — the one that
matters, because one entry for the whole stack is the wrong outcome:

```
18:28:48 bootrecon.go:259: [INFO] Boot reconciliation: 1 boot-orphaned app(s) found: [bookstack]
18:28:54 installed.go:408: [INFO] [stacks] installed-images bookstack: recorded 2 service(s)
                                  (bookstack=lscr.io/linuxserver/bookstack:26.05.2 (sha256:3db259db5828…),
                                   bookstack-db=mariadb:12.3 (sha256:a02fe89cb597…))
```

```yaml
installed_images:
    bookstack:
        ref: lscr.io/linuxserver/bookstack:26.05.2
        digest: sha256:3db259db582808ab498d49ae96b0a63f935d9cf3635c9d5bd8b8815c6ff1f8a1
        at: "2026-09-02T18:28:54Z"
    bookstack-db:
        ref: mariadb:12.3
        digest: sha256:a02fe89cb597d4375812b2eac90cf9d0775d4686daa7f7cc750ebbcad7525bbc
        at: "2026-09-02T18:28:54Z"
```

**All three recorded digests match the independently-read ground truth exactly.** `bookstack`'s two
`ENC:` secrets are **byte-identical** before and after, as are `deployed`, `deployed_at`,
`locked_fields` and `desired_state`. `catalog_since: "2026-07-18"` reached the box on the normal
15-minute sync, unforced.

### The badge, quoted from the LIVE pages

```html
<span class="tag tag-ok" title="Ez az alkalmazás a legfrissebb elérhető változatot futtatja.">Naprakész</span>
<span class="tag tag-warn" title="Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.">Frissítés elérhető — 52 napja</span>
```

Byte-identical to the task's string table. **Three of the four states are live**, on BOTH surfaces:

| state | where | observed |
|---|---|---|
| current | `/stacks` ×2, `/apps/bookstack` ×1 | „Naprakész", `tag-ok` |
| behind, age known | `/stacks` ×1, `/apps/bentopdf` ×1 | „Frissítés elérhető — **52 napja**", `tag-warn` |
| **no record** | `/apps/docmost` — a DEPLOYED app that has not restarted since the upgrade | **nothing rendered.** The load-bearing case: absent is UNKNOWN, not current |
| behind, age unknown | — | **not reachable live**: all 53 catalog apps now carry a valid `catalog_since`. `TestGroupF` only. |

**52 napja is arithmetic on a real value**, not a placeholder: bentopdf's `catalog_since` is
`2026-07-12`, the run is 2026-09-02.

**How the behind state was staged, stated because it matters:** bentopdf's `docker-compose.yml` tag was
edited `v2.8.6 → v2.8.5` and **nothing else** — no restart, no `up -d`, the container never touched —
then reverted, `sha256` equal both sides (`39679e28cdd6…`), `diff` empty, and the badge returned to
„Naprakész". So the RENDER is measured; the syncer's own half stays separately measured in the spike.
bentopdf was chosen because it has no database, no volume and no data of any kind.

**Nothing about updating changed, checked on the live page in the behind state:**
`update` ×1, `restart` ×1, `stop` ×1 for bentopdf. The badge is wired to nothing.

### ASCII-fragment counts, `grep -oF`, with positive AND negative controls

| fragment | `/stacks` | `/apps/bookstack` | `/apps/docmost` | `/stacks` (behind) | `/apps/bentopdf` (behind) |
|---|---|---|---|---|---|
| `Naprak` | **2** | **1** | **0** | 1 | 0 |
| `napja` | 0 | 0 | 0 | **1** | **1** |
| `52 napja` | 0 | 0 | 0 | **1** | **1** |
| `BookStack` / `Docmost` (positive) | 1 / 1 | 4 / 0 | 0 / 5 | — | — |
| `zzz-never-present` (**negative**) | **0** | **0** | **0** | **0** | **0** |
| `Nem-karbantartott-XYZ` (**negative**) | **0** | **0** | **0** | — | — |

The same fragments in the DEPLOYED BINARY, as a second observable on the shipped artifact:

**Hungarian strings, in the DEPLOYED binary, ASCII fragments with both controls** — a positive
observable on the shipped artifact, and **not** a rendered page:

| fragment | count | |
|---|---|---|
| `Naprak` | **3** | positive |
| `Friss` | **25** | positive |
| `napja` | **2** | positive |
| `legfrissebb` | **1** | positive (the hover text) |
| `zzz-never-present-control` | **0** | negative control |
| `Nem-karbantartott-XYZ` | **0** | negative control |

**End state:** all three containers back on the same images, both named volumes untouched, both apps
healthy. **This run provisioned nothing** — no guest, no storage, no hub record — so there is no
teardown to report on any of the three layers.

## 6b. v0.234.0 — the startup backfill, proven live on both boxes

**The honest complication first:** by the time 0.234.0 was ready, **all nine deployed apps on demo-hp
and the one on demo-felhom already carried records** — the overnight backup cycle had restarted them
and v0.233.0's recorder fired on every one (`opengist` is stamped `2026-09-03T00:31:11Z`). **The
natural fleet state could no longer exercise the new code.**

The pre-0.233.0 shape was therefore recreated on **demo-hp** (Tier 0): the `installed_images:` block
was deleted from `privatebin` (1 service) and `romm` (**3** services) — a record, never data — and the
controller restarted. Ground truth was read from the containers first.

```
09:59:45 installed-images backfill: privatebin recorded 1 service(s) (privatebin/pdo:2.0.5 (sha256:8a2cac16eff6…))
09:59:45 installed-images backfill: romm recorded 3 service(s) (rommapp/romm:5.0.0 (sha256:91f6611eca5a…),
                                    mariadb:11.4 (sha256:4f1d8d202fcf…), redis:7-alpine (sha256:ff02b58f971e…))
09:59:45 installed-images backfill: 2 app(s) recorded, 7 already had a record, 0 left unrecorded
```

- **Exactly the two stripped apps were seeded; the other seven were untouched** — the never-overwrite
  rule observed, not asserted.
- **Every digest matches the independently-read ground truth.**
- **`/stacks` then carried „Naprakész" ×9**, one per deployed app, negative control `zzz-never-present`
  at **0**.
- On demo-felhom the operator's own case renders the badge on `/stacks?filter=running` and
  `/apps/opengist`.
- Backups removed from the box; **nothing provisioned**, no app started, stopped or upgraded.

**NOT staged live, and the choice is the point:** the refusal half — that a partial observation is not
seeded — needs a degraded app, and manufacturing one is a dead-app alarm candidate. **This project has
already paid for 61 false customer e-mails from that class (R-330)**, so it is covered by
`TestGroupG_BackfillRefusesAPartialObservation` **with a companion red-proof** (remove the guard →
*"backfilled 1, want 0"*) and recorded here as unproven-live.

## 6c. A test of mine was a calendar bomb, and the suite caught it

`TestGroupD_BadgeRendersOnBothSurfaces` hardcoded a fixture `catalog_since: "2026-07-18"` **and** the
expected string `"Frissítés elérhető — 46 napja"`. The pure badge tests inject a clock; **the render
test cannot** — it goes through the production templates, which call the funcmap entry, which reads
`time.Now()`. Green on 2026-09-02, **red on 2026-09-03** with *"the behind badge is missing"* on both
surfaces, because the true answer had become 47.

Fixed by **deriving** the fixture: `catalog_since` is computed as *today minus 46 days*, so it asserts
the real number through the real clock and cannot rot. **FILED: R-457**, which also names six other
test files carrying both a date literal and `time.Now()` — as unchecked candidates, not accusations,
because mixing the two is only a defect where the literal feeds a clock-evaluated assertion.

## 7. NOT yet live-validated — an explicit list

**Everything the task asked to be validated live, was.** What remains:

0. **The backfill's REFUSAL of a partial observation** — §6b explains why it was not staged live.

1. **The fourth badge state, „Frissítés elérhető" WITHOUT an age.** It needs an app whose
   `catalog_since` is absent, malformed or future-dated, and all 53 catalog apps now carry a valid one.
   Covered by `TestGroupF` (absent, blank, `tegnap`, `18/07/2026`, `2026-13-45`, future-dated).
2. **`POST /api/stacks/{name}/restart` / `/deploy` as the trigger.** The recorder was reached live
   through **`StartStack`** (the boot reconciler — a real production caller), and in a unit test
   through a real **`RestartStack`**. `UpdateStack` and the deploy path are covered by the AST walk.
   The restart ENDPOINT itself was not fired live; the code path it reaches was.
3. **The `abandoned` + update double-badge**, unit-tested only — no live app is abandoned on either box.
4. **Any behaviour on a box other than demo-hp.** demo-felhom is still on 0.232.0, deliberately: it was
   not in scope and its one deployed app adds nothing the multi-service case did not prove.

### A correction of my own, stated before anything else in this section

**I reported the vaulted dashboard password as stale on BOTH demo boxes, and it was not.**
`~/.config/credentials` quotes its values with **single** quotes; my extraction stripped only double
quotes, so the quote characters were sent as part of the password. The operator corrected it in one
line and the retry returned **302 + `felhom_session`**.

**The instrumentation lesson is the part worth keeping, and R-453 now carries it:** I quoted the
controller's `auth.go:176 [WARN] Failed login` as the discriminator. It is a true one and it separates
*wrong password* from *wrong Host header* — **and that is ALL it separates.** It cannot tell a wrong
password from wrong password HANDLING, and I read it as though it could. **A discriminator that rules
out one alternative does not rule in the remaining one.** This is the second time this file's quoting
has produced a confident wrong verdict, so the fix filed is one shared extraction helper rather than a
resolution to be careful.

## 8. Register — 194 rows before, 205 after. Nothing closed.

R-438 and R-440 **amended and both stay OPEN** — the mechanism is documented, not changed — so
`CLOSED-ITEMS.md` is untouched, and this run's compression sweep is a no-op that says so.

New: **R-446** floating-tag honesty (P2, CC) · **R-447** slice 3, **BLOCKED** on an operator ruling
(P1) · **R-448** slice 4 (P2) · **R-449** slice 5 (P2) · **R-450** slice 6 (P2) · **R-451** slice 7
(P3) · **R-452** the `catalog_since` gate, deferred because the runner fetches at `--depth 1` (P3) ·
**R-453** *(rewritten)* the credentials file's SINGLE quotes, and a discriminator read past what it discriminates (P3) · **R-454** five `gofmt`-unclean test files with no gate (P3) · **R-455** DooPlex has no Docker Hub login and the ceiling now blocks builds (P2, WAITING-ON-OPERATOR) · **R-456** a partly-dead stack is not a boot orphan and that is written down nowhere (P3).

## 9. Observations — noticed, documented, NOT acted on

1. **The golden is one release behind.** The push gate advises: newest released controller `0.233.0`,
   newest golden baked `0.232.0`. Baking and vouching a golden is a three-field change and was not in
   this task's scope. **NOT-A-FINDING: the `golden-notice` gate raises this on every release and `STATUS.md` already carries the debt — a register row would duplicate an instrument that already fires by itself.**
2. **Five test files in `internal/web/` are not `gofmt`-clean at the baseline** — `backups_split_test.go`,
   `claim_code_naming_test.go`, `disk_health_test.go`, `r400_debug_routes_test.go`, `recovery_test.go`.
   Pre-existing; both files I added are clean. Not reformatted, under the minimal-changes rule. **FILED: R-454.**
3. **The catalog's non-static gates could not be run to a verdict on this host.** `image-pins` **OK**;
   `image-resolvable` INCONCLUSIVE (Docker Hub throttled 6 of 65 unauthenticated lookups — the same
   ceiling that blocked the build); `volume-persistence` INCONCLUSIVE (its own canary needs a scratch
   Docker host). Neither is caused by the change, and the runner exited 0. **FILED: R-455** — the same ceiling blocked the BUILD (§5), which is a bigger bill than a gate that cannot reach a verdict.
4. **`bentopdf`'s `catalog_since` needed a decision, not just a script.** The two spike commits that
   moved its pin and reverted it the same hour were **excluded by hash**; counting them would have
   dated it 2026-09-02 for a pin unchanged since 2026-07-12. The catalog's own CHANGELOG already calls
   them a measurement, not a release. **NOT-A-FINDING: the decision and its reason are already recorded durably in `app-catalog-felhom.eu`'s `CHANGELOG.md` and `REPORT.md`; it is a settled call, not an open question.**
5. **The boot reconciler does not treat a partly-dead stack as a boot orphan.** Removing only
   `bookstack`'s app container while its DB stayed up left it unselected; removing both made it an
   orphan. Correct-looking behaviour, recorded because it cost a second pass and is not written down
   anywhere. **FILED: R-456.**

## 10. Every claim in the task that turned out to be wrong, named

1. **The source paths omit the module directory** — everything is under `controller/`. **All twelve
   line-number landmarks were exact**, so this is a prefix, not drift.
2. **`app_info.html` ~L48 and `stacks.html` ~L89 point at neighbouring places, not at the badge.**
   L48 opens `stack-meta-badges` (a different badge row) and L89 is the `Frissítés` button. The
   `meta_badge` calls the new badge had to join are at **L13** and **L42**. Both were found by reading,
   as instructed.
3. **"No STOP in this task ... nothing is waiting on the operator."** **True, and it held** — one
   operator turn was spent, and it was spent correcting a mistake of mine (§7), not on a decision the
   task owed them. `STATUS.md` item 9 records the result and asks for nothing.
4. **Scenario A's stated route, `POST /api/stacks/{name}/deploy`, was not fired.** Deploying an app
   just to prove the recorder would have left a throwaway on a live box; the recorder was reached live
   through `StartStack` instead (the boot reconciler), and the deploy call site is covered by a
   multi-service unit fixture plus the AST walk.
5. **"446 looks next" — correct, and ELEVEN were needed rather than seven** (R-446..R-456), because
   the observations gate refuses a report whose observations name no row, which is the right behaviour
   and is why four extra rows exist instead of four paragraphs that die with this file.
6. Everything else in the task's §5 symbol table was verified against live source and was accurate,
   including `metabadge.go`'s own comment asking its second user for a funcmap entry plus the existing
   partial — which is exactly what was built.