4 Commits

Author	SHA1	Message	Date
admin	177c75781e	REPORT: CI run 285 is GREEN on the golden-bake commit - the golden debt closed the gate	2026-08-31 16:26:20 +02:00
admin	2263245cf2	golden 0.230.0 baked, vouched, floor raised - demo-felhom moved itself off the R-403 build (R-410 filed) ... GOLDEN_SHA256 9287f7cef5f13166276e8406005e3f28004004510c5184f1c1c7377f7aafad2e, 657 873 700 B. Evidence documentation/tests/golden-0.230.0-2026-08-31/. WHY IT WAS OWED: the newest golden was 0.229.0, which IS the build R-403 says deletes a good copy. Every fresh install and the whole fleet floor still carried it. golden_currency_gate.py had been red across dddcc80, 6e550ae, 130f7a6 and 32a4c35. THREE INDEPENDENT READERS agreed before anything was vouched: the bake's own print, the round trip of the PUBLISHED bytes (HTTP 200, 657873700 B, same sha), and the hub's Day-0 dropdown reading Gitea on a different code path. And the delivered artifact names the controller it will start - ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.230.0, with 19382 entries under var/lib/felhom/docker/. BOTH PRE-GATES were shown able to see something before their zeroes were believed: the 404 pre-gate, and the token-leak grep which returns 0 on the committed log and 1 on a seeded throwaway copy. The transient unit's own properties were grepped for the token too - 0, with the same seeded positive control returning 1. Acceptance markers counted on the COMMITTED log: 1/1/1/1 present, 0/0 absent, and the zeroes are believable because the same including-mount-point pattern returns two real lines on that file. THE VOUCH IS A THREE-FIELD CHANGE and only one field moved, which is stated rather than left to look careless: golden_version 0.229.0 -> 0.230.0; agent_version 0.130.0 and min_agent 0.129.0 UNCHANGED because v0.230.0's CHANGELOG header says MinAgent 0.129.0 and 0.129.0 <= 0.130.0, so this is not the R-216 shape. The 303 flash was not treated as proof - the page was re-read and golden_behind_fleet confirmed absent. THE FLOOR is a separate setting and was raised on the operator's explicit answer: min_controller_version 0.229.0 -> 0.230.0. THE POSITIVE OBSERVABLE, from the agent's own journal on demo-felhom, which was still running the defective 0.229.0: 16:21:30 controller-swap: image file written, restarting bootstrap target=...0.230.0 16:21:40 controller-swap: new controller healthy target=...0.230.0 Both boxes now 0.230.0 healthy. Honest note: the polling loop's first read already said 0.230.0, so the transition was not seen by the loop - the journal is the evidence. R-410 FILED, found while the gate went green: golden_currency_gate.py is satisfied by a DIRECTORY NAME (EVIDENCE_RE against os.listdir, :89,:123). I created the evidence directory before the bake finished and the gate would have passed at that moment. It already declares that it does not check the vouch; it does not declare that the bake check is a filename check. Fix: read the GOLDEN_SHA256= line out of the directory's bake.log, with a red-proof on an empty directory. R-242 updated - seventh debt, paid the same day, twice in one day. Teardown: pct destroy 9100 --purge, shred -u AFTER the log was copied out, poweroff, qemu confirmed exited with ps -eo comm (not pgrep -f, which self-matches), disk reverted to virgin. All 13 gates green - the first push this session that needed no --no-verify. Ceiling R-409 -> R-410.	2026-08-31 16:25:35 +02:00
admin	32a4c35c9c	REPORT: record the CI verdict by run id - 283 red on the pre-existing golden debt, and 281 proves it is pre-existing	2026-08-31 16:00:03 +02:00
admin	130f7a6eba	R-87 SPIKE: measured, do not build it as written (R-407..R-409 filed) ... Spike. NO production code. No version bump, no build, no deploy, no golden. felhom-controller and felhom-agent were READ ONLY. The fleet stays on v0.230.0. Q1 restic is 0.14.0 (go1.19.8, bookworm 12.15) - the four source comments asserting it are CONFIRMED, not corrected. Q2 --verify DOES exist and is NOT a content check. Red-proof: one byte changed in a restored 160 MB tar with size and mtime preserved passed clean, rc=0. Verify took 131 ms on a 213 MB / 7-file tree, which cannot be hashing. A size or mtime mismatch causes a silent re-download, not a failure. Controls: --target 1 hit, four post-0.14 flags and a nonsense string 0 hits each. Neither --verify nor --no-lock appears anywhere in the controller source. Q3 no reference for "correct" exists. restic ls --json carries no content hash in 0.14.0, and the unit manifest hashes 4918 B of a 213231242 B unit - 0.0023 percent, the config files and not the dumps or the tars. R-409. Q4 it is CHEAP. All 8 apps / 774378123 B logical restored back to back in 25 s, against 40257 ms for the weekly 100 percent check beside it. Individual restores 2253-3978 ms regardless of size: cost is per-snapshot round-trip plus ~1 s per 200 MB. Peak scratch is the app's full logical size. The 1.1 MB restic cache is index only and hides nothing (--no-cache 5423 ms vs cached 3198 ms, trees byte-identical). Q5 skip-if-busy stays right at 25 s against a 2m52s nightly backup. But RestoreOffboxScratch takes NO acquireRunning, while offbox_integrity.go:28 asserts every off-site operation does. R-408. Q6 observed with a positively-controlled lock sampler: restic restore takes NO lock; restic check DOES (locks 0 -> 1 for nine samples -> 0 across the check, zero across two restores). The product writes anyway - unlockStale runs `restic unlock`, a delete verb, before every restore (offbox_restore.go:289). The task's lead was right in direction and wrong in mechanism. R-95's constraint IS satisfiable: --no-lock plus skipping unlockStale writes nothing, and both mechanisms exist unused. Neither was fixed - the task forbids it. offbox_integrity.go:255's "It NEVER writes to the repository" is R-407. Q7 THE DECIDING ONE: of R-353/354/356/358/403 an unattended scratch-restore would have caught ONE (R-356). The value is elsewhere, and the weekly check structurally cannot reach it: `check` proves the stored bytes are the stored bytes, never that we stored the RIGHT thing. A hollow unit backs up, checks at 100 percent and restores cleanly and recovers nothing - R-403, measured in bytes on 31 August. RECOMMENDATION: option C, the NARROW test - one app a night, restored to scratch, checked against its own manifest.json through the existing unitCarriesData, scratch deleted, the SNAPSHOT recorded as the proof. Options A (do not build) and B (scheduled attended drill) considered explicitly; B is weakest because it is what already happens. R-87 should be RE-SCOPED, not built as written, and that is Viktor's call - the row stays open carrying the verdict and STATUS.md item 4 asks it in plain words. Also corrected in 07-backup-architecture.md: matrix rows 4 and 10 both said "the depth that ships ON does not re-read pack contents (R-399)". R-399 CLOSED in v0.228.0 and the depth is 100 percent. Two stale cells, fixed, and the spike verdict added beside them. Row 4's verdict is UNCHANGED by the spike and now says so. Teardown: all three layers, none of them "nothing was created" - 6 files on the PVE host, 9 in the guest, 5 plus 2 run-flags in the container, all removed and verified empty. The four scratch directories this session's restores created were removed; three that pre-date the session were left alone. Two state changes recorded rather than hidden: the control integrity run recorded its verdict (depth structure -> 100%, due-ness +7 days), and four restores appear in the controller log. Nothing was written to the off-site repository by hand. Evidence: documentation/audits/evidence-spike-restic-restore-2026-08-31/ - 31 files, every one pulled off the box BEFORE teardown (R-320). golden-currency is RED at this commit and was already red at dddcc80. Pre-existing, not this session's debt. Second --no-verify push of the day for that reason; R-404's count goes six -> seven and its row says so. Ceiling R-406 -> R-409.	2026-08-31 15:59:09 +02:00