Files
felhom.eu/documentation/tests/VALIDATION-update-slice12-2026-09-02.md
T
admin bc47dd4ef9
gates / gates (push) Successful in 18s
v0.234.0: a known limitation written on 2026-09-02 was a defect by the next morning
The operator looked at demo-felhom and found OpenGist - up 15 hours, running
exactly the catalog pin, showing no badge at all. 09-update-architecture.md had
recorded that as an accepted limitation the day before: 'the fleet view fills in
gradually'. On a quiet box gradually means never, and a feature that fills itself
in on an event nobody triggers is, on the quiet installations, not shipped. That
limitation row is now struck with the reason kept.

The living document gains slice 1b, the two admission rules of the backfill (it
never overwrites, and it refuses to seed a partial observation because the badge
reads a service-count mismatch as BEHIND), and the note that the same field having
two writers with two different admission rules is deliberate.

Live evidence added: all nine apps already had records by the time 0.234.0 was
ready, so the natural fleet state could no longer exercise the new code - said
plainly rather than papered over. The pre-0.233.0 shape was recreated on demo-hp
by stripping two records; the backfill re-seeded exactly those two with digests
matching independently-read ground truth and left the other seven alone.

The refusal half was deliberately NOT staged live: it needs a degraded app, and
manufacturing one risks the false-customer-email class that already cost 61 mails
(R-330). Unit-tested with a red-proof, and recorded as unproven-live.

R-457: a test that hardcodes a date and asserts an age derived from it is green
only on the day it is written. Mine was, and it went red overnight. Six other
files carry both a date literal and time.Now() - named as candidates, not accused.
2026-09-03 12:01:44 +02:00

15 KiB
Raw Blame History

VALIDATION — update arc slices 1 & 2, live on demo-hp (2026-09-02)

Controller 0.233.0, guest 9201 on demo-hp (Tier 0 — disposable). Evidence copied off the box at the end of the phase that produced it, not at the end of the session.

Method: endpoint/production-path level. claude-in-chrome is not available on DooPlex. Which path was used, stated exactly: the BOOT RECONCILER, bootrecon.Run → StackProvider.StartStack → compose up -d → recordInstalledImages — a REAL production caller, the same one SPIKE-app-update-2026-09-01 §2 variant 1c-ii used, and no hand-set state anywhere. Why not the customer's Restart button: §4.0 — the dashboard login was lost to a mistake of mine for part of the run, and the record half was measured before it was recovered.


1. Deployed version

$ pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'
gitea.dooplex.hu/admin/felhom-controller:0.233.0 Up 19 seconds (healthy)

Previous: 0.232.0. Image digest sha256:df5940ccf5a548ceee9065a2cc55a467f8941c636138441e0d77e320dc2023ab.

2. The record — PROVEN LIVE, on a single-service AND a multi-service app

2.0 Before — nine deployed apps, ZERO with a record

bentopdf deployed=1 installed_images=0        paperless-ngx deployed=1 installed_images=0
bookstack deployed=1 installed_images=0       privatebin    deployed=1 installed_images=0
calibre-web deployed=1 installed_images=0     romm          deployed=1 installed_images=0
docmost deployed=1 installed_images=0
kimai deployed=1 installed_images=0
opengist deployed=1 installed_images=0

That is the legacy case, and it is the state the badge must render NOTHING for.

2.0b Ground truth, read from the containers BEFORE anything was touched

bentopdf      ref=ghcr.io/alam00000/bentopdf:v2.8.6      rd=…@sha256:eaeea1e447205a79cb61d7efdc6966f37311dc1bc9c36a3a5c897bf79107c2c3
bookstack     ref=lscr.io/linuxserver/bookstack:26.05.2  rd=…@sha256:3db259db582808ab498d49ae96b0a63f935d9cf3635c9d5bd8b8815c6ff1f8a1
bookstack-db  ref=mariadb:12.3                           rd=…@sha256:a02fe89cb597d4375812b2eac90cf9d0775d4686daa7f7cc750ebbcad7525bbc

This is the discriminator. Every digest below is compared against these, taken independently.

2.1 Single service — bentopdf

Staged exactly as the spike stages a boot orphan: docker rm -f bentopdf (chosen because it has no database, no volume and no data of any kind), then the controller restarted so the reconciler runs. desired_state: running was left untouched. Nothing else was staged; the reconciler selected the app on its own.

18:27:32 main.go:2162:    [INFO] [bootrecon] boot window: fleet settled after 30s — sweeping
18:27:32 bootrecon.go:259: [INFO] [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [bentopdf]
18:27:32 manager.go:1066:  [INFO] [stacks] Starting stack: bentopdf
18:27:32 manager.go:1081:  [INFO] [stacks] Stack bentopdf started successfully (took 0.3s)
18:27:32 installed.go:408: [INFO] [stacks] installed-images bentopdf: recorded 1 service(s) (bentopdf=ghcr.io/alam00000/bentopdf:v2.8.6 (sha256:eaeea1e44720…))

/opt/docker/stacks/bentopdf/app.yaml after:

installed_images:
    bentopdf:
        ref: ghcr.io/alam00000/bentopdf:v2.8.6
        digest: sha256:eaeea1e447205a79cb61d7efdc6966f37311dc1bc9c36a3a5c897bf79107c2c3
        at: "2026-09-02T18:27:32Z"

Digest MATCHES §2.0b exactly.

2.2 Multi-service — bookstack. The case that matters.

One entry for the whole stack is the wrong outcome, and only a multi-container app can tell.

Same staging (docker rm -f bookstack bookstack-db; both volumes are NAMED — bookstack_bookstack_config, bookstack_bookstack_db_data — and were verified present, so no data was at risk), then the controller restarted.

18:28:48 bootrecon.go:259: [INFO] [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [bookstack]
18:28:48 manager.go:1066:  [INFO] [stacks] Starting stack: bookstack
18:28:54 installed.go:408: [INFO] [stacks] installed-images bookstack: recorded 2 service(s) (bookstack=lscr.io/linuxserver/bookstack:26.05.2 (sha256:3db259db5828…), bookstack-db=mariadb:12.3 (sha256:a02fe89cb597…))
18:28:54 bootrecon.go:303: [INFO] [bootrecon] Boot reconciliation complete: 1 app(s) recovered in 1 attempt(s)
installed_images:
    bookstack:
        ref: lscr.io/linuxserver/bookstack:26.05.2
        digest: sha256:3db259db582808ab498d49ae96b0a63f935d9cf3635c9d5bd8b8815c6ff1f8a1
        at: "2026-09-02T18:28:54Z"
    bookstack-db:
        ref: mariadb:12.3
        digest: sha256:a02fe89cb597d4375812b2eac90cf9d0775d4686daa7f7cc750ebbcad7525bbc
        at: "2026-09-02T18:28:54Z"

TWO entries, keyed by COMPOSE SERVICE NAME, both digests matching §2.0b exactly.

A NOTE ON WHY IT TOOK TWO PASSES, because it is a real fact about the reconciler and not a mishap. The first pass removed only the bookstack app container and left bookstack-db running; the reconciler did not select bookstack — it found [bentopdf] alone. A stack with one live member is not a boot orphan to it. Removing the DB container as well made the whole stack orphaned and it was selected on the next pass. Both apps are running and healthy at the end (§5).

2.3 The encrypted secrets survived the write — checked, not assumed

bookstack's app.yaml holds two encrypted values. Before and after the record was written, both ENC: strings are byte-identical, and deployed, deployed_at, locked_fields and desired_state are unchanged. This is the copy-and-overlay property (R-100) holding under a new field.

3. catalog_since reached the box by the NORMAL 15-minute sync

No force, no hand-edit:

$ grep -n catalog_since /opt/docker/stacks/bookstack/.felhom.yml
13:catalog_since: "2026-07-18"

4. The badge render — PROVEN LIVE on both surfaces, three of the four states

4.0 A false diagnosis of my own, corrected here rather than buried

The first pass reported "the vaulted dashboard password is stale on both demo boxes", with the controller's own [WARN] Failed login quoted as the discriminator. The password was fine. The extraction was wrong. ~/.config/credentials quotes its values with single quotes and the sed used stripped only double quotes, so the literal ' characters were sent as part of the password. The operator said so, one retry with sed "s/^['\"]//;s/['\"]$//" returned HTTP 302 + felhom_session, and everything below followed.

This is exactly the trap the credentials-file-values-are-quoted memory records — it was applied half-way. The controller's log was a true observation and a misleading one: Failed login proves the BYTES did not match the hash; it says nothing about whose fault that is. A discriminator that separates "wrong password" from "wrong host header" does not separate "wrong password" from "wrong password handling", and I read it as if it did. Register row R-453 was opened on the wrong premise and is corrected there.

4.1 „Naprakész" — quoted from the live pages

GET /apps/bookstack and GET /stacks, session-authenticated, Host: felhom.enkisfelhom.hu:

<span class="tag tag-ok" title="Ez az alkalmazás a legfrissebb elérhető változatot futtatja.">Naprakész</span>

Byte-identical to the string table in the task. On /stacks it appears exactly TWICE — the two apps that have a record — while the seven other deployed apps render NOTHING. That is the no-record case, observed live and not inferred.

4.2 „Frissítés elérhető — N napja" — the age comes from catalog_since, live

To reach the behind state the badge needs the template to pin something the container is not running. Staged on bentopdf — the app with no database, no volume and no data of any kind — by editing only its docker-compose.yml tag v2.8.6 → v2.8.5 and touching nothing else. The container was never restarted and no up -d ran; the file is the badge's input, and this is the same shape the syncer produces on its own. ScanStacks (2-minute cadence) picked it up:

<span class="tag tag-warn" title="Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.">Frissítés elérhető — 52 napja</span>

52 napja is arithmetic on a real catalog value, not a placeholder: bentopdf's catalog_since is 2026-07-12 and the run is 2026-09-02 — 52 days. It rendered on the app page and the app list.

REVERTED IMMEDIATELY, and verified byte-identical: sha256 before and after both 39679e28cdd6ee8f46e359290b4d631cf234a1cfdae374cd84c8fe7588870ed1, diff empty, container still ghcr.io/alam00000/bentopdf:v2.8.6 Up 17 minutes (healthy) throughout. The badge then returned to „Naprakész" with napja count 0.

GRADED HONESTLY: the compose file was placed by hand, not by the syncer. What is proven is the render — compose file → ScanStacks → TemplateImages → updateBadge → page — with the file carrying exactly what a sync would have written. The syncer's own half is separately measured in SPIKE-app-update-2026-09-01 §3.

4.3 ASCII-fragment counts, with a positive and a negative control

Accented text is never grepped directly here. grep -oF, so no . is a wildcard — the correction the spike had to make on itself.

fragment /stacks (current) /apps/bookstack /apps/docmost (legacy) /stacks (behind) /apps/bentopdf (behind)
Naprak 2 1 0 1 0
napja 0 0 0 1 1
52 napja 0 0 0 1 1
legfrissebb (hover) 2 1 0 — —
BookStack (positive control) 1 4 0 — —
Docmost (positive control) 1 0 5 — —
zzz-never-present (negative control) 0 0 0 0 0
Nem-karbantartott-XYZ (negative control) 0 0 0 — —

/apps/docmost is the load-bearing column: a deployed app with no record renders no badge at all, live. Absent is UNKNOWN, and it is not rendered as current.

4.4 Nothing about updating changed — checked on the live page

In the behind state, /stacks still carries exactly one of each for bentopdf:

stackAction(event, 'bentopdf', 'update')    ×1
stackAction(event, 'bentopdf', 'restart')   ×1
stackAction(event, 'bentopdf', 'stop')      ×1

The badge is wired to nothing.

4.5 The one state NOT reachable live

„Frissítés elérhető" with no age — it needs an app whose catalog_since is absent, malformed or future-dated, and all 53 catalog apps now carry a valid one. Covered by TestGroupF, which walks absent, blank, tegnap, 18/07/2026, 2026-13-45 and a future date.

4b. v0.234.0 — the startup backfill, PROVEN LIVE 2026-09-03

Why it exists. §4 of this file, written 2026-09-02, recorded "the record only appears after the next lifecycle action" as an accepted limitation. The operator looked at demo-felhom the next morning and found OpenGist — up 15 hours, running exactly the catalog pin, showing no badge at all. On a quiet box "fills in gradually" means "never".

The staging, and why it was needed

By the time v0.234.0 was ready, all nine deployed apps on demo-hp and the one on demo-felhom already had records — the overnight backup cycle had restarted them and v0.233.0's recorder had fired on every one (opengist's record is stamped 2026-09-03T00:31:11Z). So the natural fleet state could no longer exercise the new code, and saying that plainly matters more than a green log line.

The pre-0.233.0 shape was therefore recreated on demo-hp (Tier 0): the installed_images: block was deleted from privatebin (1 service) and romm (3 services) — a record, never data — and the controller restarted. Ground truth was read from the containers first, independently.

The result

09:59:45 installed.go:519: [INFO] installed-images backfill: privatebin recorded 1 service(s) (privatebin=privatebin/pdo:2.0.5 (sha256:8a2cac16eff6…))
09:59:45 installed.go:519: [INFO] installed-images backfill: romm recorded 3 service(s) (romm=rommapp/romm:5.0.0 (sha256:91f6611eca5a…), romm-db=mariadb:11.4 (sha256:4f1d8d202fcf…), romm-redis=redis:7-alpine (sha256:ff02b58f971e…))
09:59:45 installed.go:525: [INFO] installed-images backfill: 2 app(s) recorded, 7 already had a record, 0 left unrecorded
  • Exactly the two stripped apps were seeded, and the other seven were not touched — the never-overwrite rule, observed rather than asserted.
  • Every digest matches the ground truth read from the containers beforehand: privatebin/pdo@sha256:8a2cac16eff6caed4dc622e7b3ebd0c0ffcb28dfc0eba0b9c335636552eadac1, rommapp/romm@sha256:91f6611eca5a4dafc4f4a1d72a1ed7dd66a11375d939f28410dc1d1de0b80b1b, mariadb@sha256:4f1d8d202fcf7bcb3902f63af09f9c1a050c2922a89652f22abaec0d4f015e83.
  • The re-seeded records are identical to the ones stripped, apart from at — which is correct: it records when THIS observation was first made.
  • /stacks on demo-hp then carried „Naprakész" ×9 — one for every deployed app — with the negative control zzz-never-present at 0 and napja at 0 (nothing is behind on that box).
  • On demo-felhom the operator's own case renders: <span class="tag tag-ok" …>Naprakész</span> on /stacks?filter=running and on /apps/opengist.

What was NOT proven live, and why the choice was made

The refusal half — that a partial observation is not seeded — was not staged on a live box. Doing so means stopping one container of a multi-service app to make it degraded, and a degraded app is a dead-app alarm candidate; this project has already paid for 61 false customer e-mails from exactly that class (R-330), and manufacturing one to demonstrate a guard is a bad trade. It is covered by TestGroupG_BackfillRefusesAPartialObservation with a companion red-proof — removing the guard makes it fail with "backfilled 1, want 0". Stated as unproven-live rather than left unmentioned.

Teardown

The two app.yaml backups taken on the box were removed. Nothing was provisioned and no app was started, stopped or upgraded at any point.

5. End state — nothing left broken, nothing provisioned

$ pct exec 9201 -- docker ps --format '{{.Names}}|{{.Image}}|{{.Status}}' | grep -E 'bookstack|bentopdf'
bookstack|lscr.io/linuxserver/bookstack:26.05.2|Up 10 seconds (health: starting)
bookstack-db|mariadb:12.3|Up 16 seconds (healthy)
bentopdf|ghcr.io/alam00000/bentopdf:v2.8.6|Up About a minute (healthy)

All three containers are back on the SAME images they ran before, both named volumes untouched, and both apps carry a complete record. This run provisioned nothing — no guest, no storage, no hub record — so there is no teardown to report on any of the three layers.