felhom.eu/REPORT.md

REPORT — SPIKE: what an app update actually does, and which other paths do it too (2026-09-01)

Task class: Spike. No production code was written in any repo. Output is a findings doc, register rows, two architecture updates and one operator decision.

Findings doc: documentation/audits/SPIKE-app-update-2026-09-01.md Evidence: documentation/audits/evidence-spike-app-update-2026-09-01/ (19 files)

---

1. The Phase 1 answer, stated unambiguously

YES — docker compose up -d upgrades an app whose compose file has already moved, and the Restart button does it.

Produced by variants 1a and 1b, and independently by 1c-ii:

• 1a (target image ABSENT locally): the restart took 18.3 s, ended with the container on the new tag, and left the new image in the local store where it had not been — so up -d performed a network pull, in a code path that contains no pull step.
• 1b (target image PRESENT): 0.5 s, container recreated onto the new tag, image id unchanged.
• 1d, the negative control (file NOT edited): container id, image id, digest and StartedAt all identical — up -d did not even recreate. The method can show "no change" when nothing changed.
• 1c-ii (unattended): the boot reconciler brought a failed app back on the new version with nobody pressing anything.

And the answer to 1c is NO, which narrows the exposure: a hard guest reset upgraded nothing. Docker's restart: unless-stopped restored the existing containers on the old image and the reconciler logged its own verdict — Boot reconciliation: no boot-orphaned apps (nothing to start).

Per the task's own rule, no fix is proposed. The behaviour may have been chosen — RestartStack says so in a comment — so it goes to the operator as a decision.

---

2. Confirmed baselines actually used

Repo	at §1 of the task	measured at start	end	moved?
felhom-controller	960d29b0612c	960d29b0612c	960d29b0612c	no — read only
felhom.eu	1d59353df437	1d59353df437	this session's documents	as planned
app-catalog-felhom.eu	29edad9c5bf4	29edad9c5bf4	5d8f25f, tree identical to 29edad9c5bf4	reverted
felhom-agent	4586f0f7f6d1	4586f0f7f6d1	4586f0f7f6d1	not touched

None had moved since the task was written. Live controller on demo-hp: 0.232.0.

---

3. Per-phase results, each with its control

Phase	Result	Control
0	R-438, R-439, R-440 filed, each with rank and owner	ids confirmed free by grepping all three register files (R-437 was highest)
1	THE GATE: YES. 1a/1b upgrade; 1c does not; 1c-ii upgrades unattended	1d — unchanged file, no change at all
2	Sync rewrote a DEPLOYED app's file at 17:45:17Z; container unchanged; nothing told the customer	positive (BentoPDF, 4 hits) + negative (zzz-never-present, 0) on both customer pages
3a	Pull failure: HTTP 500, app untouched and still running	re-run as a Restart: also 500, app still ran
3b	HTTP 200 "update completed" over a crash-looping app; alarm fired 5m16s later	the alarm is a positive observable (Event pushed: app_start_failed), plus the detector heartbeat 1 currently down
4	demo-hp and demo-felhom: zero drift by tag. But 2 of 6 floating pins have already MOVED upstream	both fully-pinned images (romm:5.0.0, pdo:2.0.5) were SAME
5	Existing safety dump is database-only; DB dumps are 48 KB–395 KB; the file half is the cost, and it does not fit for a large app	demo-hp is too young to price it — stated, and Campaign 10's measured figures cited instead
6	App data CANNOT be rolled back. Old image refuses to start on migrated data	returning to 32.0.9 restored both seeded markers byte-identical — the data is not destroyed, only the downgrade is blocked

---

4. The exact symbols that bring an app back — found by reading

path	file:symbol
boot reconciler	felhom-controller/controller/internal/bootrecon/bootrecon.go:269 — Reconciler.Run → StartStack
its scheduler	controller/cmd/controller/main.go:2127 — runBootReconcile, called at :450
app-stop guard	controller/internal/backup/appstop_marker.go:283 — AppStopGuard.Recover → StartStack
its hold-aware wrapper	controller/cmd/controller/main.go:2008 — gatedAppStopStarter.StartStack
drive-return gate	controller/internal/web/intermediary.go:222 — Server.restartStacks → StartStack
guest-boot change	controller/internal/web/intermediary.go:458 — Server.processGuestBootChange
quiesce restart	controller/internal/quiesce/quiesce.go:733 — Loop.restartAll
off-site reconstitution	controller/internal/backup/offbox_reconstitute.go:692 — Manager.ReconstituteFromOffsite

Full table of all 13 non-API call sites: findings doc §8.

---

5. Register rows opened, updated or re-ranked

row	what	owner
R-438	opened P1-HIGH, then updated with the live evidence (sync overwrite measured; consequence measured; the in-source design intent found and recorded, which narrows it)	VIKTOR rules, CC measures
R-439	opened P3-LOW, then updated — the severity survives but its stated reason was imprecise (isOperationalState counts restarting/degraded as operational)	CC
R-440	opened P2-MEDIUM, then updated: two floating pins have ALREADY moved, with a passing control	CC
R-441	NEW — the restore path and the sync disagree about the image, and the sync wins within 15 minutes	CC measures, VIKTOR rules
R-442	NEW — P1-HIGH: remove_hdd_data:true is inert with no paths.hdd_path; 128 MB left, API said neither removed nor preserved	CC
R-443	NEW — the Update button reports success over an app it has broken	CC proposes, VIKTOR rules
R-444	NEW — nothing runs pct fstrim; demo-hp's thin pool held ~23.8 GB of freed blocks	CC
R-445	NEW — hub app telemetry outlives the app and sets a fleet-wide recommendation	VIKTOR rules, CC implements

Nothing was closed and nothing was re-ranked. The ranking of R-438 relative to existing rows is Viktor's, and I have not moved anything.

---

6. Claims in the task that turned out to be wrong, named

1. "a power cut … is an unattended three-major-version upgrade" — not as stated. A plain power cut upgraded nothing (measured). The unattended upgrade needs "and the app did not come back". The exposure is smaller than the operator page claims.
2. "Five other code paths end in compose up -d" — thirteen non-API call sites, nine files.
3. "Phase 4 — demo-hp, demo-felhom and Peti's box" — Peti's box is DOWN, not enrolled, and runbooks/target-selection.md:161 says "No access route from DooPlex". Two boxes measured live; Peti's row is UNKNOWN, with what is knowable taken read-only from the hub and the catalog history.
4. R-439's severity reason — right conclusion, imprecise reason (see §5).
5. R-440's "23 pins" — exactly right (79 image lines / 53 apps / 66 distinct; 23 with no patch component). One arguable 24th named rather than rounded away.
6. The catalog history figures — spot-checked and all correct: 153 commits, 53 apps, 0 files with upgrade metadata, and all four multi-major bumps confirmed by commit hash.
7. My own method, corrected in-flight: the first customer-page search used grep -o "2.8.6", whose unescaped . produced two false hits; grep -F gives zero. The controls caught it.

---

7. Evidence handling

Evidence was written directly into documentation/audits/evidence-spike-app-update-2026-09-01/ on DooPlex as each phase produced it — before every revert, including the intermediate ones. The catalog revert (18:10:29Z), the compose reverts, the guest reset and the Nextcloud teardown all happened after their evidence was already off the machine. Nothing was lost and nothing had to be reproduced.

---

8. Observations

1. The restore path writes the recovery unit's OLD image pin into the live stack dir, and the catalog syncer overwrites it again within 15 minutes. The overwrite half is measured on demo-hp; that the restore writes to that same path is read at cmd/controller/main.go:2570, not measured — both halves are graded as such in the row. FILED: R-441

2. remove_hdd_data: true removed nothing. 128 MB of app data stayed on the drive while the API returned 200 with hdd_paths_removed:null, hdd_paths_preserved:null. Root cause established with controls: Paths.HDDPath has no default and demo-hp's controller.yaml does not set it, so ParseComposeHDDMounts returns nil on its first line. FILED: R-442

3. An update can report success over an app it has just broken. HTTP 200 and "updated successfully" while the container was already crash-looping; the truth reached the customer 5m16s later through the dead-app alarm rather than through the update itself. FILED: R-443

4. The PVE thin pool was holding ~23.8 GB of blocks the guest had already freed, and fstrim from inside the unprivileged container is refused; pct fstrim from the host reclaimed it. Nothing runs it on the fleet. FILED: R-444

5. The hub kept app telemetry for an app that no longer exists anywhere, and it now sets a fleet-wide suggested memory limit for Nextcloud derived from a 15-minute crash-looping throwaway. Retained rather than cleared, because the reset is irreversible and on the operator's own surface. FILED: R-445

6. The syncer's debug hash line cannot show what it claims to show. logFileHashes (internal/sync/sync.go:386) reads the destination after the write, so it printed src=2ebbbda3765b2b21, dst=2ebbbda3765b2b21 (changed) — the same hash twice, with the word "changed". NOT-A-FINDING: it is DEBUG-only and the Updated <app>/<file> INFO line immediately above it carries the fact correctly, so nothing is lost and no behaviour is wrong. Recorded so the next person reading a sync log does not try to learn from those two hashes, which cannot differ.

---

9. Teardown — all three layers

Layer 1 — the machine. bentopdf restored to its catalog tag v2.8.6, digest sha256:eaeea1e447205a79…, byte-identical to the run's baseline. The throwaway nextcloud stack removed via the product's own endpoint; containers, all three volumes and app.yaml gone. The 128 MB the product failed to remove (observation 2) deleted by hand along with its backup dirs; find /mnt -iname "*nextcloud*" returns nothing. Five images this run pulled removed by targeted docker rmi — no prune of any kind was run anywhere. No guest was created; guest 9201 was hard-reset once by design and returned with all nine apps.

Layer 2 — the host. local-lvm 68.97% → 70.91% during the run → 26.78% after pct fstrim 9201. The run's ~1.05 GiB was returned and 23.8 GB more that predated it. Guest filesystems back to pre-run values (/ 957 M, /mnt/sys_drive 12 G, hdd_1 5.5 G).

Layer 3 — the hub. This run provisioned NOTHING. No customer record and no appliance record was created; the customers list is unchanged at five rows, identical to the list read at the start. The existing demo-hp customer was used. What the run did create is hub events (app_start_failed, plus deploy/remove for the throwaway app) — retained deliberately, because the event log is an append-only record and deleting from it to tidy a test damages the surface this project relies on for history. One residue is retained rather than cleared with the reason and the exact one-line command recorded in the findings doc §13 (observation 5).

Nothing on demo-felhom, ep0, DooPlex or Peti's box was modified. Peti's box was never contacted.

---

10. Final verification

felhom-controller: git status --porcelain → empty
                   HEAD = origin/main = 960d29b0612c
                   go build ./...  → OK
                   go vet ./...    → OK
                   go test ./...   → rc=0, 28 packages, 0 FAIL

The controller tree was left untouched, and that is proven rather than asserted.

---

11. My own mistakes

1. My first customer-page search used a regex where I needed a literal. grep -o "2.8.6" treats . as a wildcard and reported two hits on a page that contains none. I caught it only because the task requires a control on every such search, and re-ran with grep -F. The rule earned its place in the same session it was applied.
2. My first poll for "nextcloud is healthy" matched the wrong container. The break condition matched healthy anywhere in the line and fired on nextcloud-redis. Corrected to an exact-name filter. No result depended on it.
3. I renumbered STATUS.md badly on the first attempt, leaving the list running 1–6, 8, 9 with no item 7, and left the section's own lead line saying one thing was waiting when there were two. Both fixed. This is the ranking-paragraph-goes-stale trap (R-405) in miniature, and it appeared within minutes of my writing about it.