The operator looked at demo-felhom and found OpenGist - up 15 hours, running exactly the catalog pin, showing no badge at all. 09-update-architecture.md had recorded that as an accepted limitation the day before: 'the fleet view fills in gradually'. On a quiet box gradually means never, and a feature that fills itself in on an event nobody triggers is, on the quiet installations, not shipped. That limitation row is now struck with the reason kept. The living document gains slice 1b, the two admission rules of the backfill (it never overwrites, and it refuses to seed a partial observation because the badge reads a service-count mismatch as BEHIND), and the note that the same field having two writers with two different admission rules is deliberate. Live evidence added: all nine apps already had records by the time 0.234.0 was ready, so the natural fleet state could no longer exercise the new code - said plainly rather than papered over. The pre-0.233.0 shape was recreated on demo-hp by stripping two records; the backfill re-seeded exactly those two with digests matching independently-read ground truth and left the other seven alone. The refusal half was deliberately NOT staged live: it needs a degraded app, and manufacturing one risks the false-customer-email class that already cost 61 mails (R-330). Unit-tested with a red-proof, and recorded as unproven-live. R-457: a test that hardcodes a date and asserts an age derived from it is green only on the day it is written. Mine was, and it went red overnight. Six other files carry both a date literal and time.Now() - named as candidates, not accused.
15 KiB
VALIDATION — update arc slices 1 & 2, live on demo-hp (2026-09-02)
Controller 0.233.0, guest 9201 on demo-hp (Tier 0 — disposable). Evidence copied off the box
at the end of the phase that produced it, not at the end of the session.
Method: endpoint/production-path level. claude-in-chrome is not available on DooPlex.
Which path was used, stated exactly: the BOOT RECONCILER, bootrecon.Run → StackProvider.StartStack → compose up -d → recordInstalledImages — a REAL production caller, the same one
SPIKE-app-update-2026-09-01 §2 variant 1c-ii used, and no hand-set state anywhere.
Why not the customer's Restart button: §4.0 — the dashboard login was lost to a mistake of mine
for part of the run, and the record half was measured before it was recovered.
1. Deployed version
$ pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'
gitea.dooplex.hu/admin/felhom-controller:0.233.0 Up 19 seconds (healthy)
Previous: 0.232.0. Image digest sha256:df5940ccf5a548ceee9065a2cc55a467f8941c636138441e0d77e320dc2023ab.
2. The record — PROVEN LIVE, on a single-service AND a multi-service app
2.0 Before — nine deployed apps, ZERO with a record
bentopdf deployed=1 installed_images=0 paperless-ngx deployed=1 installed_images=0
bookstack deployed=1 installed_images=0 privatebin deployed=1 installed_images=0
calibre-web deployed=1 installed_images=0 romm deployed=1 installed_images=0
docmost deployed=1 installed_images=0
kimai deployed=1 installed_images=0
opengist deployed=1 installed_images=0
That is the legacy case, and it is the state the badge must render NOTHING for.
2.0b Ground truth, read from the containers BEFORE anything was touched
bentopdf ref=ghcr.io/alam00000/bentopdf:v2.8.6 rd=…@sha256:eaeea1e447205a79cb61d7efdc6966f37311dc1bc9c36a3a5c897bf79107c2c3
bookstack ref=lscr.io/linuxserver/bookstack:26.05.2 rd=…@sha256:3db259db582808ab498d49ae96b0a63f935d9cf3635c9d5bd8b8815c6ff1f8a1
bookstack-db ref=mariadb:12.3 rd=…@sha256:a02fe89cb597d4375812b2eac90cf9d0775d4686daa7f7cc750ebbcad7525bbc
This is the discriminator. Every digest below is compared against these, taken independently.
2.1 Single service — bentopdf
Staged exactly as the spike stages a boot orphan: docker rm -f bentopdf (chosen because it has no
database, no volume and no data of any kind), then the controller restarted so the reconciler runs.
desired_state: running was left untouched. Nothing else was staged; the reconciler selected the app
on its own.
18:27:32 main.go:2162: [INFO] [bootrecon] boot window: fleet settled after 30s — sweeping
18:27:32 bootrecon.go:259: [INFO] [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [bentopdf]
18:27:32 manager.go:1066: [INFO] [stacks] Starting stack: bentopdf
18:27:32 manager.go:1081: [INFO] [stacks] Stack bentopdf started successfully (took 0.3s)
18:27:32 installed.go:408: [INFO] [stacks] installed-images bentopdf: recorded 1 service(s) (bentopdf=ghcr.io/alam00000/bentopdf:v2.8.6 (sha256:eaeea1e44720…))
/opt/docker/stacks/bentopdf/app.yaml after:
installed_images:
bentopdf:
ref: ghcr.io/alam00000/bentopdf:v2.8.6
digest: sha256:eaeea1e447205a79cb61d7efdc6966f37311dc1bc9c36a3a5c897bf79107c2c3
at: "2026-09-02T18:27:32Z"
Digest MATCHES §2.0b exactly.
2.2 Multi-service — bookstack. The case that matters.
One entry for the whole stack is the wrong outcome, and only a multi-container app can tell.
Same staging (docker rm -f bookstack bookstack-db; both volumes are NAMED —
bookstack_bookstack_config, bookstack_bookstack_db_data — and were verified present, so no data
was at risk), then the controller restarted.
18:28:48 bootrecon.go:259: [INFO] [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [bookstack]
18:28:48 manager.go:1066: [INFO] [stacks] Starting stack: bookstack
18:28:54 installed.go:408: [INFO] [stacks] installed-images bookstack: recorded 2 service(s) (bookstack=lscr.io/linuxserver/bookstack:26.05.2 (sha256:3db259db5828…), bookstack-db=mariadb:12.3 (sha256:a02fe89cb597…))
18:28:54 bootrecon.go:303: [INFO] [bootrecon] Boot reconciliation complete: 1 app(s) recovered in 1 attempt(s)
installed_images:
bookstack:
ref: lscr.io/linuxserver/bookstack:26.05.2
digest: sha256:3db259db582808ab498d49ae96b0a63f935d9cf3635c9d5bd8b8815c6ff1f8a1
at: "2026-09-02T18:28:54Z"
bookstack-db:
ref: mariadb:12.3
digest: sha256:a02fe89cb597d4375812b2eac90cf9d0775d4686daa7f7cc750ebbcad7525bbc
at: "2026-09-02T18:28:54Z"
TWO entries, keyed by COMPOSE SERVICE NAME, both digests matching §2.0b exactly.
A NOTE ON WHY IT TOOK TWO PASSES, because it is a real fact about the reconciler and not a mishap.
The first pass removed only the bookstack app container and left bookstack-db running; the
reconciler did not select bookstack — it found [bentopdf] alone. A stack with one live member is
not a boot orphan to it. Removing the DB container as well made the whole stack orphaned and it was
selected on the next pass. Both apps are running and healthy at the end (§5).
2.3 The encrypted secrets survived the write — checked, not assumed
bookstack's app.yaml holds two encrypted values. Before and after the record was written, both
ENC: strings are byte-identical, and deployed, deployed_at, locked_fields and
desired_state are unchanged. This is the copy-and-overlay property (R-100) holding under a new field.
3. catalog_since reached the box by the NORMAL 15-minute sync
No force, no hand-edit:
$ grep -n catalog_since /opt/docker/stacks/bookstack/.felhom.yml
13:catalog_since: "2026-07-18"
4. The badge render — PROVEN LIVE on both surfaces, three of the four states
4.0 A false diagnosis of my own, corrected here rather than buried
The first pass reported "the vaulted dashboard password is stale on both demo boxes", with the
controller's own [WARN] Failed login quoted as the discriminator. The password was fine. The
extraction was wrong. ~/.config/credentials quotes its values with single quotes and the sed
used stripped only double quotes, so the literal ' characters were sent as part of the password.
The operator said so, one retry with sed "s/^['\"]//;s/['\"]$//" returned HTTP 302 +
felhom_session, and everything below followed.
This is exactly the trap the credentials-file-values-are-quoted memory records — it was applied
half-way. The controller's log was a true observation and a misleading one: Failed login proves the
BYTES did not match the hash; it says nothing about whose fault that is. A discriminator that
separates "wrong password" from "wrong host header" does not separate "wrong password" from "wrong
password handling", and I read it as if it did. Register row R-453 was opened on the wrong premise
and is corrected there.
4.1 „Naprakész" — quoted from the live pages
GET /apps/bookstack and GET /stacks, session-authenticated, Host: felhom.enkisfelhom.hu:
<span class="tag tag-ok" title="Ez az alkalmazás a legfrissebb elérhető változatot futtatja.">Naprakész</span>
Byte-identical to the string table in the task. On /stacks it appears exactly TWICE — the two
apps that have a record — while the seven other deployed apps render NOTHING. That is the
no-record case, observed live and not inferred.
4.2 „Frissítés elérhető — N napja" — the age comes from catalog_since, live
To reach the behind state the badge needs the template to pin something the container is not running.
Staged on bentopdf — the app with no database, no volume and no data of any kind — by editing
only its docker-compose.yml tag v2.8.6 → v2.8.5 and touching nothing else. The container was
never restarted and no up -d ran; the file is the badge's input, and this is the same shape the
syncer produces on its own. ScanStacks (2-minute cadence) picked it up:
<span class="tag tag-warn" title="Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.">Frissítés elérhető — 52 napja</span>
52 napja is arithmetic on a real catalog value, not a placeholder: bentopdf's catalog_since is
2026-07-12 and the run is 2026-09-02 — 52 days. It rendered on the app page and the app list.
REVERTED IMMEDIATELY, and verified byte-identical: sha256 before and after both
39679e28cdd6ee8f46e359290b4d631cf234a1cfdae374cd84c8fe7588870ed1, diff empty, container still
ghcr.io/alam00000/bentopdf:v2.8.6 Up 17 minutes (healthy) throughout. The badge then returned to
„Naprakész" with napja count 0.
GRADED HONESTLY: the compose file was placed by hand, not by the syncer. What is proven is the
render — compose file → ScanStacks → TemplateImages → updateBadge → page — with the file
carrying exactly what a sync would have written. The syncer's own half is separately measured in
SPIKE-app-update-2026-09-01 §3.
4.3 ASCII-fragment counts, with a positive and a negative control
Accented text is never grepped directly here. grep -oF, so no . is a wildcard — the correction the
spike had to make on itself.
| fragment | /stacks (current) |
/apps/bookstack |
/apps/docmost (legacy) |
/stacks (behind) |
/apps/bentopdf (behind) |
|---|---|---|---|---|---|
Naprak |
2 | 1 | 0 | 1 | 0 |
napja |
0 | 0 | 0 | 1 | 1 |
52 napja |
0 | 0 | 0 | 1 | 1 |
legfrissebb (hover) |
2 | 1 | 0 | — | — |
BookStack (positive control) |
1 | 4 | 0 | — | — |
Docmost (positive control) |
1 | 0 | 5 | — | — |
zzz-never-present (negative control) |
0 | 0 | 0 | 0 | 0 |
Nem-karbantartott-XYZ (negative control) |
0 | 0 | 0 | — | — |
/apps/docmost is the load-bearing column: a deployed app with no record renders no badge at all,
live. Absent is UNKNOWN, and it is not rendered as current.
4.4 Nothing about updating changed — checked on the live page
In the behind state, /stacks still carries exactly one of each for bentopdf:
stackAction(event, 'bentopdf', 'update') ×1
stackAction(event, 'bentopdf', 'restart') ×1
stackAction(event, 'bentopdf', 'stop') ×1
The badge is wired to nothing.
4.5 The one state NOT reachable live
„Frissítés elérhető" with no age — it needs an app whose catalog_since is absent, malformed or
future-dated, and all 53 catalog apps now carry a valid one. Covered by TestGroupF, which walks
absent, blank, tegnap, 18/07/2026, 2026-13-45 and a future date.
4b. v0.234.0 — the startup backfill, PROVEN LIVE 2026-09-03
Why it exists. §4 of this file, written 2026-09-02, recorded "the record only appears after the next lifecycle action" as an accepted limitation. The operator looked at demo-felhom the next morning and found OpenGist — up 15 hours, running exactly the catalog pin, showing no badge at all. On a quiet box "fills in gradually" means "never".
The staging, and why it was needed
By the time v0.234.0 was ready, all nine deployed apps on demo-hp and the one on demo-felhom already
had records — the overnight backup cycle had restarted them and v0.233.0's recorder had fired on
every one (opengist's record is stamped 2026-09-03T00:31:11Z). So the natural fleet state could no
longer exercise the new code, and saying that plainly matters more than a green log line.
The pre-0.233.0 shape was therefore recreated on demo-hp (Tier 0): the installed_images: block
was deleted from privatebin (1 service) and romm (3 services) — a record, never data — and the
controller restarted. Ground truth was read from the containers first, independently.
The result
09:59:45 installed.go:519: [INFO] installed-images backfill: privatebin recorded 1 service(s) (privatebin=privatebin/pdo:2.0.5 (sha256:8a2cac16eff6…))
09:59:45 installed.go:519: [INFO] installed-images backfill: romm recorded 3 service(s) (romm=rommapp/romm:5.0.0 (sha256:91f6611eca5a…), romm-db=mariadb:11.4 (sha256:4f1d8d202fcf…), romm-redis=redis:7-alpine (sha256:ff02b58f971e…))
09:59:45 installed.go:525: [INFO] installed-images backfill: 2 app(s) recorded, 7 already had a record, 0 left unrecorded
- Exactly the two stripped apps were seeded, and the other seven were not touched — the never-overwrite rule, observed rather than asserted.
- Every digest matches the ground truth read from the containers beforehand:
privatebin/pdo@sha256:8a2cac16eff6caed4dc622e7b3ebd0c0ffcb28dfc0eba0b9c335636552eadac1,rommapp/romm@sha256:91f6611eca5a4dafc4f4a1d72a1ed7dd66a11375d939f28410dc1d1de0b80b1b,mariadb@sha256:4f1d8d202fcf7bcb3902f63af09f9c1a050c2922a89652f22abaec0d4f015e83. - The re-seeded records are identical to the ones stripped, apart from
at— which is correct: it records when THIS observation was first made. /stackson demo-hp then carried „Naprakész" ×9 — one for every deployed app — with the negative controlzzz-never-presentat 0 andnapjaat 0 (nothing is behind on that box).- On demo-felhom the operator's own case renders:
<span class="tag tag-ok" …>Naprakész</span>on/stacks?filter=runningand on/apps/opengist.
What was NOT proven live, and why the choice was made
The refusal half — that a partial observation is not seeded — was not staged on a live box. Doing
so means stopping one container of a multi-service app to make it degraded, and a degraded app is a
dead-app alarm candidate; this project has already paid for 61 false customer e-mails from exactly
that class (R-330), and manufacturing one to demonstrate a guard is a bad trade. It is covered by
TestGroupG_BackfillRefusesAPartialObservation with a companion red-proof — removing the guard
makes it fail with "backfilled 1, want 0". Stated as unproven-live rather than left unmentioned.
Teardown
The two app.yaml backups taken on the box were removed. Nothing was provisioned and no app was
started, stopped or upgraded at any point.
5. End state — nothing left broken, nothing provisioned
$ pct exec 9201 -- docker ps --format '{{.Names}}|{{.Image}}|{{.Status}}' | grep -E 'bookstack|bentopdf'
bookstack|lscr.io/linuxserver/bookstack:26.05.2|Up 10 seconds (health: starting)
bookstack-db|mariadb:12.3|Up 16 seconds (healthy)
bentopdf|ghcr.io/alam00000/bentopdf:v2.8.6|Up About a minute (healthy)
All three containers are back on the SAME images they ran before, both named volumes untouched, and both apps carry a complete record. This run provisioned nothing — no guest, no storage, no hub record — so there is no teardown to report on any of the three layers.