245 Commits

Author	SHA1	Message	Date
admin	fc737b0fc0	installer v1.28.0: the removal genuinely reverses the installation (R-316) ... v1.27.0's fix worked exactly once per machine. Measured on drill-r50 from virgin, on the PUBLISHED v1.27.0, before anything was changed: cycle 1 recorded 'no' and freed :53; cycle 2 recorded 'yes' and left dnsmasq running on 0.0.0.0:53; cycle 3 refused, exit 1. Every box already in the field is at cycle 2, and a reinstall onto a machine that has had Felhom is cycle 2 by definition. Why cycle 2 says yes: the preflight's ownership question is dpkg-query package presence and nothing else - not the absence of a record. Stopping the unit and leaving the package made our own package read as the household's one cycle later. Now the uninstall removes the package when the record says we installed it. Order unchanged and load-bearing: read the record, act, then delete the state file that holds it. TWO packages are recorded, because dnsmasq ships the unit and dnsmasq-base ships /usr/sbin/dnsmasq, and each is taken back only if we added it. The dependency check is a SIMULATION, not a guess: apt-get -s purge is asked what it would remove and the purge proceeds only if that set is a subset of ours; otherwise stop+disable, naming the package that blocked it. Never interactive, never fatal, and the success is re-queried rather than read off an exit code. Watched: three fixed cycles -> install 3 PASSES; a household resolver untouched; a dependent package not purged and named; no record -> untouched with the command named. Red-proofs with the mutation asserted applied: remove the purge -> cycle 3 refuses in those exact words; remove the ownership check -> a household resolver is purged; infer ownership -> the guess is taken. Also: R-317 (the agent stats a path dnsmasq-base owns to decide whether to install dnsmasq - pre-existing, now reachable), R-318 (no honest ownership marker exists for existing boxes; the preflight message is the mechanism), and the status page's decisions section rewritten to say what each decision costs and what doing nothing selects.	2026-08-13 08:16:04 +02:00
admin	d102ca5767	Vouch delivered fleet-wide; the MinAgent hold observed firing and releasing for the first time ... Operator vouched golden 0.214.0 / agent 0.129.0 / min agent 0.129.0 and raised the floor to 0.214.0. Both artifact shas in hub_settings match the bake and the release byte-for-byte. With the floor at 0.214.0 and demo-hp still on agent 0.128.0, the hub HELD the controller floor - 'agent 0.128.0 < MinAgent 0.129.0 (controller floor withheld)' - and released it 6 seconds after the agent was brought up. That is the R-216 guard doing exactly what it exists for, seen firing for the first time, and it is the argument for declaring MinAgent in the CHANGELOG header. Both demo boxes verified on the boxes: controller 0.214.0, agent 0.129.0, healthy.	2026-08-12 21:14:38 +02:00
admin	684cd2eb11	R-265 third sighting: the jobs API names the failing step when the log 404s, and a re-run disambiguates	2026-08-12 19:18:44 +02:00
admin	4906aeb3f9	R-311 proven live (HTTP 422 on hardware); R-308 WITHDRAWN — my quoting bug, not a stale credential ... The live test read as a FAILURE for twenty minutes because I stripped only double quotes from a credentials value wrapped in SINGLE ones, sending a literal ' as part of the recovery code. Correctly unquoted: the old code returns 422 with opens_retained=true and the supersession date; a wrong code still returns 400. The same bug produced the R-308 finding in the previous report. The dashboard password is fine - HTTP 302 with a session cookie on the first try. Third time this project has produced a wrong 'the credential is stale' verdict from that one trap.	2026-08-12 19:05:58 +02:00
admin	6362bb6cb6	hub v0.103.0 — a host can read the packages we kept for it (R-311) ... ListSupersededEscrow had zero production callers for nineteen days. It is the only reader of a retained identity_blob, so the retention shipped in v0.93.0 was material the product could not reach - proven on the fixture 2026-08-12, where a code that opens a retained package was answered as a code that opened nothing. New GET /api/v1/hosts/<id>/escrow/retained: self-scoped exactly as the current-row GET, same recovery-mode gate, same audit event written BEFORE the bytes leave, capped at 16. Rows with a NULL identity_blob are WITHHELD and returned as unopenable_count. They retain the PBS key, not the repository password, so they can never open what the caller is asking about; serving them would have the agent try packages that cannot succeed and would let the screen claim an earlier package is openable on exactly the boxes the original defect hurt. The count is returned because their existence is load-bearing and underivable. The trade, stated rather than waved through: the hub still cannot read any of it - sealed bytes in, sealed bytes out, no decrypt path, no recovery code ever held. What widens is volume, bounded by self-scope, the recovery-mode gate and the cap. The response is a NAMED TYPE, not a map, so the wire-contract gate can resolve it; the wire is declared as a fourth ROOT and the gate now checks 182 tags rather than 174. A positive control shows that check is name-presence, not decodability - filed as R-315 rather than reported as coverage. Six tests through the real endpoint; four red-proofs asserted applied.	2026-08-12 18:43:07 +02:00
admin	1d5f2b8bb6	DRILL: the retained key works, and the customer cannot reach it ... Three verdicts, kept separate because collapsing them is how this assumption survived a week. (a) The material IS retained. host_escrow_superseded id 11 is the first retained row in fleet history to carry identity_blob (572 B), byte-identical to the pre-supersession row (sha256 a10032341c8584ed...). (b) The retained material DOES open the old store. Unsealed with the old recovery code it yielded a password byte-identical to the pre-change one, and restored three planted files byte-identical from a store the box itself could no longer open - including a Hungarian accented filename verified as raw bytes. Negative control ran first and failed closed. (c) The customer has NO route, and is misinformed. ListSupersededEscrow has zero production callers; the recovery path selects FROM host_escrow. Asked with the code that had just worked by hand, the product answered "the recovery code did not open the sealed bundle". A valid code for retained history is reported as a bad code - the R-224 class again. R-304, rank 1. Both installer faults were watched happening first, so installer-v1.27.0 is now published (tag + both webpage.yaml refs). Pre-fix: the box came up on controller 0.98.3 against a vouched 0.213.0, below the floor and below the version carrying the recovery screen; and our own uninstall left dnsmasq on 0.0.0.0:53 so our own next install refused. R-297 and R-300 CLOSED. Also filed R-305 (the dnsmasq fix fires once per machine - the leftover returns on the second reinstall, proven), R-306 (--preflight-only writes state it says it does not), R-307 (a live abandon countdown on demo-felhom, firing 2026-08-24 - operator decision), R-308 (stored controller password stale), R-309 (the day-0 runbook's publication claim has been false since R-110), R-310 (two edges). Ceiling R-303 -> R-310. Capability map moved: the retention claim is now marked operator-only. Phase A logs did not survive the intermediate revert; recorded.	2026-08-12 17:41:56 +02:00
admin	fbe1155fbb	R-302 docs: register rows, the two rules earned twice, STATUS ... Closes R-296 (verified: shipped in v0.212.0) and R-301 (premise confirmed, fixed in v0.213.0). Files R-302 with WHY the obvious condition was rejected, and R-303 for the missing markOrphaned guard - the co-render is now harmless, not impossible. Bake evidence for golden 0.213.0.	2026-08-12 15:38:11 +02:00
admin	890a474ff2	STATUS back to one screen; close R-280 and R-294 ... 211 lines -> one screen. Moves closed items out, corrects the tester paragraph, states the floor situation as the operator's one-field call, and stops asking him to decide something that shipped.	2026-08-12 14:04:35 +02:00
admin	125aec1be2	R-300: uninstall no longer leaves dnsmasq blocking the next install ... Removing the snippet and restarting left dnsmasq enabled and unconstrained on 0.0.0.0:53, so the next byo install's preflight refused and the customer went debugging a home network that was never at fault. Ownership is recorded at preflight (the only moment it is a fact - the package is installed by the agent, not this script) and honoured at removal. Boxes already in the field carry no record and fail safe to restart-only, with the reason and the command logged; the preflight message covers them instead. Not observed live - no installer-v1.27.0 tag is cut. Files R-299..R-301.	2026-08-12 14:03:40 +02:00
admin	f76cbf0ec7	Part 0: correct the PETI record - the mitigation it named does not exist ... The row said a drive failure there means offsite-only recovery. Re-read from the hub's own store: no host row (deleted 2026-07-15 08:56:22, escrow_acked=0), no escrow of any kind, and offsite backup never ran once (escrow_state pending, snapshot_count 0 - the fork-4 guard working, not a fault). The local app-data repo was empty too and the whole-guest vzdump shares the failing device. If that drive fails today, everything on it is lost. Records the fact and leaves the parked/not-parked ruling open - that is the operator's call and does not need restating to be true. STATUS.md no longer reads the absence of a hub record as reassuring.	2026-08-12 13:43:07 +02:00
admin	eb600872f2	R-297: installer compares a local golden against the manifest before using it ... Step 7 short-circuited on any local golden archive with no version compare, no digest and no warning, so the manifest sha256 was consulted only on the fetch path. Local discovery is newest-by-filename: correct by recency, never by verification. A box could reinstall from a stale archive and come back below the version where the offsite recovery screen exists. Digest first, then the baked controller tag. An auto-discovered mismatch re-fetches the vouched golden; an operator-named mismatch refuses. An unreadable manifest refuses rather than passing. Not published: installer-v1.26.0 is deliberately not cut until a fresh install has been observed taking a stale local golden on drill-r50. Also files R-295..R-298.	2026-08-10 13:57:50 +02:00
admin	11a5c3bd92	R-265 second sighting: a CI run failed and its log cannot be retrieved ... felhom.eu run 293 (67eced8, the demo-felhom recovery commit) FAILED, and /actions/jobs/293/logs returns HTTP 500 while runs 292 and 294 both return 200 with 12290 bytes. The failure is real and its reason is not establishable -- exactly what this row was filed for, now seen a second time. What can be said: the same tree passes the FULL gate set locally, all eight OK, and run 294 on the next commit passed in CI, so nothing persisted. What cannot: why 293 failed. The process failure is mine. The checklist says confirm your own push's CI run by ID; I pushed 67eced8 without doing so, and it surfaced only because the next session happened to list recent runs. A pull check that is skipped is not a check.	2026-08-10 11:17:37 +02:00
admin	c04f933d0b	The census answers no, three receipts found, and the prune was on file all along ... CENSUS (read-only, hub store, tester's machine not contacted): no machine that is not ours can be in the state that cost demo-felhom its history. The hub holds escrow for three hosts; both demo boxes lost their pre-fix key in the same four hours on 2026-08-04; peti-felhom and david have no host row and no escrow at all. A control ran FIRST and had to pass -- the query returned "present (572 bytes)" for a host known to have material and "absent (NULL)" for one known not to. Corrected my own instrument on the way: a date-only comparison mislabelled both losses as after the fix, so the in-force moment is now pinned from the hub's first post-fix escrow row (11:11:37Z), which independently agrees with the register. PART 1 ESTABLISHED. The prune is recorded inside R-267 -- the row about the Configuration page being slow -- because pruning artifacts is what made that page fast. Arithmetic checks (23+7=30, plus three versions that only surfaced after the first thirty moved them onto page one = 33) and the PAGINATED listing shows both generics at exactly ten. R-291's blocking condition is released: the operator was being asked to establish something already written down. And my counter-argument yesterday was wrong in exactly the way R-267 warns about -- "containers hold 19" came from an unpaginated query; paginated they hold 270 and 169. RECEIPTS: three restored (drives.enrol, backup.tier1, fail.lost-recovery-code), each citing the document that walked it; the map already read PROVEN-LIVE for all three, so this follows the map rather than raising a status in the view. NINE HONEST GREYS. fault.selfheal's best hit argues against it -- an incident recording self-heal's absence through a 1h15m outage. THE DECAY RULE FIRED FOR THE FIRST TIME. backup.restore-proof has a receipt from 28 July and is superseded anyway: demo-hp's restore-test failed 5 August and the box has since been rebuilt. A claim about a continuing behaviour cannot rest on an old observation. The capability map still reads PROVEN-LIVE and is now the thing out of step -- recorded, not silently rewritten. PART 4 specified, not implemented. The orphan card promises restorability the box rendering it cannot evaluate: the discriminator is on the hub and no wire field carries it. A conditional promise the system cannot evaluate is the same defect as an unconditional false one, so the copy stops promising, says what happens, and names a route. Ships with the next controller change so one bake covers both.	2026-08-10 11:15:37 +02:00
admin	67eced8fbf	demo-felhom is protected again, and the authorised recovery could never have worked ... Checked before acting, and the check is the finding. The box's local key and the hub's sealed escrow key hash to the SAME value (c60c8bc737a6b7c6...), and that key answers "wrong password or no key found" against its own repository. Running the recovery would have returned a key the box already held and which was already proven not to open the store. The store was written under 48741892f0ef4d59... -- host_escrow_superseded id=4, superseded 2026-08-04 07:20:08, identity_blob NULL. The restic password lives only in the identity bundle (escrow/identity.go:39, read by recover.go:91), so it is unrecoverable by construction; the surviving K-escrow payload is 64 bytes, a wrapped key, far too small to carry it. Same shape the register already records for demo-hp, four hours the wrong side of the retention fix. Took the operator's stated fallback instead: the orphan reset through the customer's own card. Old store moved aside, never deleted, to /home/felhom-repo.orphaned-20260810 (1.2 GB); fresh repository under the current key; offbox_repo_reset audited hub-side. Then PROVEN rather than assumed -- last_status ok, 10s, and the snapshot's CONTENTS listed: opengist compose files, manifest.json and volume-dumps/opengist_opengist_data.tar. Not an empty backup calling itself successful. R-202 gains hard evidence: the orphan card promises those set-aside backups may be restorable later with their recovery code. For these 1.2 GB that is false and unfixable, and it is said to the customers most likely to read it.	2026-08-10 10:10:43 +02:00
admin	985f0ba63c	Record the guards, the narrowing, and the two things I could not do ... R-273's owed guards are both built and closed. R-291 records what CI stopped covering and why, so it can be widened deliberately rather than discovered. R-292 is new and was found by a test failing for the wrong reason: artifact_sha_invalid conflates "version missing", "registry unreachable" and "bad sha" into one message. v0.102.0 works around it by ORDERING -- the probes run first, so an unreachable registry is reported as unreachable -- but the message itself is untouched. CONTEXT gains the rule this session is about: a check and the policy it enforces must read the same number from the same place, or they drift and the drift looks like a defect in something else. Two corollaries, both of which cost something: a bounded check must print what it stopped covering on every run, and an unreadable policy is INCONCLUSIVE rather than unbounded. Stated in the report rather than glossed: Part 4 (finding receipts for the twelve downgraded claims) was NOT done and is a shortfall, not a decision -- splitting it would have produced exactly the half-checked green the exercise exists to prevent. Part 5 was droppable and dropped. The tag-push green is not re-proved tonight and is not claimed; the evidence offered is runs 190 and 216.	2026-08-09 19:28:45 +02:00
admin	36bcd12543	Deploy hub v0.102.0, and record that the package deleter is still not established ... manifests/hub.yaml 0.101.0 -> 0.102.0. Image built and pushed, and verified served by the registry before the bump rather than after. R-287 corrected on two counts. My own sentence "no DELETE on the packages API appears in 48h of Gitea router logs" is WITHDRAWN: kubectl logs on the Gitea pod now returns nothing older than 2026-08-09 16:35 and contains zero api/packages lines even for requests I made myself, so the log never covered the window and its silence was never evidence. A second attempt to attribute the deletion also failed and the deleter remains NOT ESTABLISHED. Sources exhausted: no register row records a package prune (R-210 is WAITING-ON-OPERATOR, says "Nothing was deleted; this is a list, not an action", and concerns local Docker images); package_version has no soft-delete column so a deletion leaves no row; Gitea's action feed carries no package operation at all in the window; and a uniform newest-ten cap is not visible -- felhom-agent generic holds 10 but the container packages hold 19 each. It may be unestablishable from this side: Gitea keeps no package-deletion trail.	2026-08-09 19:15:17 +02:00
admin	6088afcbed	Verify the standing picture against source: 12 downgrades, and the decay ran both ways ... 55 claims verified. Twelve moved, all downwards: walked 32 -> 20, built 5 -> 17. Register ceiling R-284 -> R-290. THE RULE DID NOT FIRE THE WAY IT WAS EXPECTED TO. Not one downgrade came from code moving under an old proof. All twelve came from step 1 of the same rule -- the cited evidence does not exist. Measured: of the 28 capability-map rows behind the page's claims, 8 carry a tests/ or audits/ path and 20 carry prose only. The green dots were drawn from rows that cite an argument, not a walk (R-290). The map, not the dataset, is what needs fixing -- it still says PROVEN-LIVE for all twelve. And once it ran backwards: fault.operator-email looked contradicted by R-182, but live source shows the backup_run_failures digest allowlisted, operator-only and templated, with recovery_unit_capture_failed now record-only. The claim is right and the REGISTER ROW is stale (R-289). The session went looking for stale proofs and found a stale defect. R-281 WITHDRAWN -- wrong in both directions, settled by the operator's mailbox. The tripwire DID fire (escrow_blob_served 10:19:41Z = 12:19 CEST) and false error-severity alarms fired too, for deliberate attended work (R-285). The measurement's cause is ESTABLISHED: the P7 query copied hub.db without hub.db-wal, and the signature is exact -- it reported "2 events all day, newest 00:30:07", and the rows at or before 00:30:07 number exactly 2. Timezone and wrong-key were tested and refuted. The control had been drawn from the same stale snapshot as the measurement, which is why it agreed (R-286). Part 4: NO WORKFLOW CHANGED, deliberately. The gate is not ref-sensitive -- it enumerates from the Gitea tags API, and both previous tag pushes passed. The red is TRUE: run 267 saw v0.120.0 downloadable, run 284 on the same commit saw 404. Who deleted the package is NOT established and is not guessed (R-287). The page is now generated from where-felhom-stands.yaml by scripts/render_stands.py: static, zero script tags, every moved status carrying a visible "changed, was X" chip. The React bundle -- whose content was gzip+base64 inside a JS module map -- is kept as a dated snapshot. scripts/check_stands.py gates the data and convicted 51 problems in my own first draft before the staged positive control ever ran.	2026-08-09 18:40:49 +02:00
admin	1d6f1c522d	Rehearsal 2026-08-09 COMPLETE: data BYTE-IDENTICAL, journey needs a shell twice ... The walk finished. All four planted files came back byte-identical out of snapshot 41c830db, including two Hungarian accented filenames verified as RAW NAME BYTES (NFC preserved) — the discriminator the Gate 0 positive control was built for, having been watched failing on an NFC->NFD rename that renders the same. Unlock 21s, restore 13.2s. It finished only because a terminal was available twice: - R-273 CLOSED. v0.128.0 was published as a package and never git-tagged, so every install died at 5/8. Tag pushed on operator instruction after an INDEPENDENT download proved the package sha equalled the vouched value; --resume then reached Day-0 SUCCESS in 3m49s on controller 0.210.0. The two guards that would stop the class recurring are still owed. - R-280 NEW, rank 1. A reinstalled box cannot re-attach its own data drive by any dashboard route: /api/disks/candidates returns empty because both lists are built from the UNCLAIMED-disk scan, and the drive is claimed precisely because it is also the backup target. Correct for "initialise", over-broad for "attach", which is non-destructive by definition. The restore page meanwhile says "Ez ket kattintas" and points at that empty list. Cleared by POSTing /mnt/sys_drive — an internal path no household could produce. Also new: R-281 the hub said NOTHING through the entire reinstall and the sealed-backup tripwire did not fire on a real unseal (positive control: 2 events all day fleet-wide); R-282 one code with three names and a mail pointing at a page the box does not show; R-283 hub reads "Claimed 18d ago" while the box serves its setup page; R-284 "almost full" over a 93%-free store. R-274 NARROWED by measurement rather than left as written: the resume path fetched the vouched golden correctly, because --resume skips the preflight that does local discovery. What survives is real — discovery is sort|tail -1 with no manifest comparison — but a FRESH install taking a stale golden is still not observed, and the row says so. Two of my own claims were refuted by test and are recorded as refuted, not quietly dropped: the leftover sudoers file is inert (sudo skips dotted names), and demo-hp's off-site tier was healthy all along.	2026-08-09 12:35:26 +02:00
admin	b1afbb8a4d	Rehearsal 2026-08-09: the walk stops at P3 — R-273 blocks every install fleet-wide ... P1 uninstall, P2 preflight, P3 install. The install FAILED at step 5/8 in 44s, and the two rank-1 findings are both on the setting-up path a tester's visit is made of. Eleven register rows minted (R-269..R-279); ceiling moves 268 -> 279. R-273 (RANK 1) — the hub vouches agent 0.128.0; that version was published as a Gitea PACKAGE but never git-tagged. Since R-183 the installer correctly pins its config fetches to raw/tag/v<vouched>, so every fresh install and every reinstall now 404s as root, mid-install. Measured: main 200, v0.127.0 200, v0.128.0 404. This is R-184 arriving; release-agent.sh:23 already documents the exact hazard. Existing boxes are fine (self-update takes the binary from the registry). NOT fixed here — publishing a release tag is outward-facing and the runbook says stop and report. One command unblocks it; it is in STATUS.md. R-272 (RANK 1) — Felhom's own uninstall leaves the condition that makes Felhom's own reinstall refuse. It installs dnsmasq at day-0, then on teardown removes the snippet and RESTARTS the daemon unconstrained (process start time lands inside the uninstall window), which grabs 0.0.0.0:53; the next preflight then refuses, and the message reads as though the owner's LAN DNS is at fault. R-274 — a local golden is adopted with no version and no sha check; the manifest vouch is consulted only on the fetch path. demo-hp's local copy is controller 0.192.0 against a vouched 0.210.0, and below the 0.200.0 where the recovery screen shipped. Not yet observed end-to-end (R-273 killed step 5 first). Also: R-275 orphaned credential backups + uid reuse, R-276 the wg tunnel outlives the uninstall, R-269/270/271 from the token rotation, R-277 three hub surfaces misreport a healthy off-site tier, R-278 demo-felhom six days unprotected, R-279 no operator-triggerable off-site run. Two hypotheses of mine were tested and REFUTED rather than shipped as findings: the leftover sudoers file is inert (sudo skips dotted filenames), and demo-hp's off-site tier was healthy all along - I had misread the hub and said so. STATUS.md records the three rulings §8.3 asked for, with the floor CORRECTED to its live value 0.200.0 and the count corrected to twenty.	2026-08-09 10:59:44 +02:00
admin	56f8aa611c	R-267 closed: 26.2s -> 5.4s cold / 0.14s warm, and two corrections to my own measurements ... Registry pruned to the newest 10 per package on the operator's confirmed rule. 33 deletions, all HTTP 204; the live-vouched golden 0.210.0, agent 0.128.0 and floor 0.127.0 were asserted into the KEEP set BEFORE any DELETE was issued and verified still fetchable after. TWO CORRECTIONS TO WHAT I REPORTED EARLIER, both recorded rather than quietly dropped: 1. 'Only 50 generic versions exist' was NOT a count, it was a PAGE LIMIT. ?limit=1000 returns at most 50, and the 50 I measured was exactly the cap. Three older agent versions (0.81.0/0.80.0/0.79.0) only became visible after the first 30 deletions moved them onto page one. An unpaginated listing is not evidence of a total — this repo's own 'an empty listing is not evidence of emptiness' rule, walked into while measuring it. 2. The operator's 'reduce the number of artifacts' was the better call and my measurement said otherwise. I reported it helps sub-linearly and is not the lever. Measured after: trimming to 10+10 took the COLD load from 13.4s to 5.4s, a 2.5x improvement on exactly the path the memo cannot help, because the fan-out is per-version. drill-r50 runs agent 0.113.0, now deleted; flagged before deleting, disposable nested drill VM, only its re-download path is gone.	2026-08-08 20:44:58 +02:00
admin	efe9dfd15d	R-267 CLOSED (26.2s -> 0.24s warm); R-268 filed: I printed a live token into a transcript ... R-267 closed by hub v0.101.0. Measured after the 60s memo: cold 13.4s, warm 0.24-0.33s. The operator sees a quarter-second except at most once a minute. R-268 filed against myself. Setting up R-221's live drill, a one-liner meant to list bootstrap.json's KEYS printed the local_api object whole, including its token, for guest 9201. Reported rather than quietly rotated, because a secret reaching a transcript is a finding whatever its blast radius. Exposure assessed rather than assumed, and it is small: the token opens only the agent's per-guest local API on the island bridge between that host and that one guest, self-scoped to guest 9201, not routable from the LAN or internet, on a Tier-0 disposable box with no customer data. Using it already requires code execution there, at which point an attacker has more than the token. Rotation exists (TokenStore.Mint, last-write-wins per VMID) but must also rewrite the guest's bootstrap.json or the controller loses agent access — an operator-timed act, not a background one. The general fix is upstream: reading secret-bearing JSON should go through a helper that prints keys and never values, the discipline the golden bake already uses for the Gitea token. R-221 also recorded as PROVEN ON HARDWARE in STATUS.	2026-08-08 20:13:22 +02:00
admin	1c14b91d6f	R-267: the Configuration page, measured — 26.2s to ~10s, and what is left ... Reported as 'almost minutes'. The guess that it hashes artifacts on page load DOES NOT HOLD and the code already said so: Gitea stores the sha and the hub reads it as metadata. The cost was latency x count, fixed in three legs (hub v0.100.0-0.100.2), each found by refusing to accept a number that did not match the arithmetic. Measured 26.2s -> mean 9.85s over 8 samples (min 5.13, max 18.13). The remaining dominant cost is the package SEARCH, 0.20-3.8s per dropdown depending on load, which concurrency does not help; 16 concurrent file-metadata calls take 0.58s by comparison. EVERY NUMBER IS CONTAMINATED and the row says so: taken on DooPlex at load average 7-11 while this same session was building images, running two Go suites and baking a golden. The same search measured 3.8s in-cluster and 0.44s from the host ninety seconds later. Re-measure on an idle box. Both operator proposals answered on the measurement rather than deferred to: pruning artifacts helps sub-linearly (only 50 versions exist) and is worth doing for its own sake; storing the hash in the hub DB is NOT recommended, because Gitea is already the store and a copy would be a second source of truth the operator reads to confirm a vouch. The lever that would work — an in-memory cache with a TTL — is left OPEN because it trades dropdown freshness for speed, which is an operator decision.	2026-08-08 18:03:00 +02:00
admin	4a4a1e245a	R-265 CI timeout + golden 0.210.0 baked; R-221/R-259/R-258 closed, R-266 minted, G-3 unblocked ... Four defects of one family, all shipped today: something the box already knows, thrown away or drawn as its opposite. Agent v0.128.0, controller v0.210.0. NO HUB CODE, no hub bump, no ArgoCD sync. R-265 (this repo). timeout-minutes: 5 on the gates job — every honest run in the observed session finished in 18-34s, so this is ~9x the slowest and far under whatever reaped run 264 at 834s with no log. The alarm mail now carries Elapsed (start stamp via $GITHUB_ENV; an absent stamp prints "unknown (no start stamp)", never a bogus 1.7-billion-second figure) and its "names itself in the run log" sentence is qualified so it cannot mislead when there is no log. ⚠ THE UNKNOWN IS NOT CLOSED. Whether the if: failure() alarm fires for a REAPED job is still unverified. The timeout makes the reap unreachable in practice; it does not answer what happens in one. Demonstrating it means deliberately hanging a run on main, which would leave the branch red for a parallel session. Said in the workflow comment, the changelog, R-265 and the report — none of them claiming it is answered. GOLDEN 0.210.0 baked, published, round-trip verified, NOT VOUCHED. The currency gate went red the moment the controller was bumped — correct — and is closed by the bake, never --no-verify. No --no-verify anywhere this session. ⚠ THE AGENT WAS NOT PUBLISHED UNTIL THIS SESSION CHECKED, AND IT MATTERED. R-221's fix is in the AGENT, and a fresh install takes its agent from the Day-0 manifest. The binary had been hand-deployed to felhom-pve and never published, so agent_version 0.128.0 was not selectable and a fresh install would have received 0.127.0 — the golden would have carried the controller fixes and NOT the one the headline defect needed. Caught by checking each Day-0 value was FETCHABLE rather than assuming. Published from the live-deployed bytes, sha-verified across the hop first. Registers. R-221, R-259, R-258, R-265 CLOSED. R-266 MINTED (READY): the failed root statfs still travels to the hub as a 0-of-0 disk; ranked LOW because it is the quiet direction — it can only miss a true alarm, never raise a false one — and it is now a two-repo wire change governed by G-1's gate. Highest ID moved R-265 -> R-266. CONTEXT S-39 rules the convention this project was missing: "we do not know" is never drawn as "fine", and the codebase has ONE way of saying it — an explicit ...Known bool companion checked in the template. ROADMAP G-3 was explicitly blocked on that decision and is unblocked; what remains there is a survey-and-convert of existing sites, not the gate. Capability map row 93 CHECKED and it was NOT claiming something untrue — it is about the operator notification path. But its narrative ("the page you open to ask whether ONE app is backed up") invites the wrong reading, and the adjacent thing WAS false until v0.210.0, so the row now records that the two halves disagreed and only the operator half was true. Six red-proofs across the two code repos, each with the mutation asserted applied. The one that matters: Part 1 Scenario A FAILED against today's tree, with the intended message. Part 1's operator-present live validation is OWED and is the session's STOP. repo_gates --fast: all 8 OK.	2026-08-08 16:52:39 +02:00
admin	4f5784894e	R-265: a CI run failed with NO LOG, and the alarm points at a log that does not exist ... Run 264 (650cc8a, a DOCUMENTATION-ONLY commit) failed between two greens of byte-identical gate code. Not waved away as a flake, because this project's own record is that a "known flake" can be a true positive. MEASURED. Every other run this session: 18-34s, log present (HTTP 200). Run 264: 834s (07:12:40 -> 07:26:34 UTC) and GET /actions/jobs/264/logs returns HTTP 500 - "264.log.zst: file does not exist". The act-runner pod never restarted (0 restarts, 5d17h), so the job hung and was reaped; the runner did not die. NOT A GATE FINDING, on four independent facts: the diff from the green before it is Markdown only; the same content is green two commits later (265, 33s); the gate code is identical across 263/264/265; and 260-262, which WERE real gate failures, each failed in under 35s WITH a log. THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately recorded as such. DooPlex was doing heavy work in that window (139 MB kubectl cp, a go run compiling the whole hub module), which is a plausible contention story - but 40 cores at load ~5 does not establish it, so it is filed as a hypothesis rather than asserted as a cause. THE FINDING THAT MATTERS IS SECOND-ORDER, and it is gates.yml's own purpose turned against it. The workflow exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and its alarm mail says "The failing gate names itself in the run log." There is no run log. An operator following that sentence finds nothing and cannot tell a reap from a conviction. Unverified and worse: the alarm step is `if: failure()` and whether it ran at all for a reaped job is unknown - if it did not, this was a red CI that alarmed nobody. Fix shapes recorded, none built: surface duration + log-presence in the alarm; an explicit timeout-minutes under the reap so it fails fast and loudly WITH a log; and one deliberate test of whether the alarm fires on a reaped job, because until that runs, "CI alarms on failure" is an assumption.	2026-08-08 09:29:51 +02:00
admin	b080ecf411	hub v0.99.0 — the hub can see whether the operator can get in (R-260); G-1 gate closes, R-247 closes ... oobDegraded tested five things and the sixth never arrived. The agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the SAME version that introduced the `oob` stanza carrying it — and store.HostOOBRow mirrored five of the agent's eight OOB fields. With no field for it, encoding/json discarded the fact on arrival, so a box with felhom-sshd active, reachable, a valid config and a configured peer reported `ok` with NO OPERATOR KEY INSTALLED AT ALL. Not a wrong answer: an answer to a question nobody was asking. `operator_peer_configured`, which the hub did read, only says the peer IP is in desired-state — that OOB is MEANT to work, not that entry is possible. Now decoded: operator_key_configured, plus wg_handshake_age_s and healed_at. The last two ride the ALERT TEXT and are deliberately NOT in the predicate — widening a check beyond the fact that is now arriving is how a check stops being read. SCENARIO F, decided on a measurement rather than a preference. operator_key_configured decodes as a POINTER: nil = the agent never said, reported distinctly and never as ok. The version gate was rejected because the field and its stanza shipped in the SAME agent version (v0.72.0), so a stanza without the field cannot come from any released agent; the fleet is 0.113.0/0.127.0 and the vouched floor is 0.127.0. Handled explicitly anyway and pinned, because "cannot happen" is a claim this project has been burned by. THE MESSAGE NAMES THE FAULT. oobDegradedReason is the single source for both predicate and text, so the alert can never name a different fault from the one that fired. The old form derived it separately and had a vocabulary of two — unreachable, or config invalid — with no way to say the key is missing. The operator reads this at 07:00. TESTS DRIVE THE DECODE BOUNDARY. Every hub OOB test before this built a HostOOBRow by hand, and a test written that way CANNOT SEE A FIELD THAT NEVER DECODES — which is how this held a green suite for five weeks. The pre-existing fixture oobReport() also omitted the field, so those scenarios ran against a report shape no released agent produces (same family as R-262). Both fixed. Red-proofs, 8 expected outcomes and 0 wrong, each with the mutation asserted applied: dropping the field returns the false ok; an unconditional check alerts a healthy box; unknown-as-ok restores the silent pass. G-1 CLOSED — scripts/wire_contract_gate.py shipped as ranked, built BEFORE the fixes and seen failing on 40 fields (documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md). Two instrument defects the control caught first: a substring false negative (grep -F healed_at matched privsep_healed_at) and treating dr_recipe as wholly opaque when its top-level sections ARE decoded through an allow-list that already cost offsite_restic (R-122). The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo: R-260 said "at least eight DECISION-BEARING facts", never eight tags. The real count is 40. R-260 CLOSED (class gated, sharpest instance fixed). R-247 CLOSED (controller v0.209.0). R-264 MINTED and OPEN — the 21 facts with no consumer, allowlisted with reasons so that gating the class could not be mistaken for deciding them. Still open and named: R-246, R-255..R-259, R-261..R-263, and C7's test-comment half. Capability map checked: it claims OOB access is implemented, never monitored, so no row was untrue; what was untrue sat one layer down and the row now records it. repo_gates --fast: all 8 OK. go build/vet/test green in hub, run separately from this commit.	2026-08-08 08:47:02 +02:00
admin	b7fb2117ae	CAMPAIGN 12 — the class sweep: golden 0.208.0 baked (awaiting vouch), R-256..R-263 filed, gating ranked ... Part 1. Golden 0.208.0 baked on the drill VM, published and ROUND-TRIP VERIFIED — 656 150 362 B, sha256 ba668f59..5ffb82, and ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.208.0. Acceptance markers all green (overlay2 x1, mount points x2 rootfs+mp0, upload HTTP 201 x1, excluding/FATAL/mp1 x0), Result=success. Token file->file, read inside the VM; systemctl show grep = 0; committed-log grep = 0 WITH a control proving the grep works. Bake VM destroyed, drill disk restored to virgin. NOT VOUCHED — the campaign halts there deliberately. golden_currency_gate.py was correctly RED on arrival and is green after the bake. No --no-verify was needed anywhere in this session. Parts 2-4. Seven defect classes swept for siblings by class rather than by feature. Analysis only: no product code, nothing deployed, no machine touched beyond the bake VM. Eight new rows R-256..R-263 (ceiling moved from R-255), grouped by class in OPEN-ITEMS.md. C1 produced no new instance and has no row. The sharpest is R-260: the agent reports operator_key_configured every heartbeat, the hub has no field for it, so the check that answers "can the operator get into this box" returns ok for a box with no operator key installed. Every class states whether its method re-found the known instances, because a method that cannot re-find them has not been shown to work: C1 2/3 (verified by replaying the pre-fix templates), C2 2/2, C3 2/3 + 1 as fixed, C4 fix-pattern re-found, C5 re-found, C6 deadcode 0/2 and bespoke 1/2, C7 weakest and said so. Blind spots stated per class; seven suspicions investigated and DISPROVED, including two of my own methods. Part 4's ranking is in ROADMAP.md as G-1..G-8. Gate C5 (cross-repo tag reachability — cheap, --fast-eligible, would have caught every R-260 instance on the introducing commit). Do NOT gate C6: golang.org/x/tools/cmd/deadcode was measured against a PLANTED probe and is blind to unreachable METHODS on widely-used types, which is exactly the shape both known instances have. R-242's untouched half is recorded, not built: this bake demonstrated it, the currency gate flipping green the moment the evidence DIRECTORY existed, before the round trip finished and with no vouch near it. Correction the campaign owed its own brief: escrow_stale was described as closed; it is R-247 and READY. The live repo is the source. Sampled rather than swept, exactly: C7 60 of 2652 production invariant comments and NONE of the 1440 test comments (that half is owed); C2 19 of 221 refusals; C3/C4 controller only. No finding was reproduced live. STATUS.md is 100 lines against its 93-line one screen.	2026-08-07 23:07:42 +02:00
admin	59527d00f9	R-254 CLOSED both sites (controller v0.208.0); R-255 filed; R-242 red again ... R-254 site one was the same defect and is fixed the same way. Site two was NOT the defect the row described: the pre-deploy hidden input is deliberate (a form must carry what it submits, README §318) and was left alone; the indefensible one was the readonly display input on an ALREADY-DEPLOYED app, where nothing is submitted. The premise that this broke a repo rule does not hold and is recorded rather than dropped: no line in the repo says 'no silent auto-fill'. What exists is CONTEXT.md:2070, about accidental EMPTY-password deployments. §7.3 measured on the fleet: site one's code path has never run (crafty-controller is the only app declaring initial_credentials and is deployed nowhere); site two's exposure is also empty (demo-hp runs three apps, none with a generated secret field). HONEST LIMIT: that is a current-state measurement, and nothing recorded reads — which was part of the fault. No evidence of exposure, and no mechanism that could have produced evidence either way. Rotation not indicated by anything measured. R-255 NEW: the guard covers 4 of 27 pages at runtime, and the cheap all-templates gate is blind to the shape that actually shipped (a secret under a neutral page-data key) — both verified, both stated in the gate's own docstring. Filed rather than declaring a partial guard complete. R-242 red a second time in 24h; --no-verify declared. The cadence is the argument for its other half: nothing gates the vouch.	2026-08-07 21:27:20 +02:00
admin	f651b31a7a	golden 0.207.0 BAKED, PUBLISHED, ROUND-TRIP VERIFIED and VOUCHED — the currency gate goes green ... Closes the delivery gap v0.207.0 opened this session. Until now the gate was correctly red and a machine installed today would have received 0.206.0 — the release written, tested and pushed, and not delivered. Round trip is the evidence, not the build log: the published bytes were downloaded back (656 879 192 B, sha256 20ec9602…22995, both identical to what the bake reported) and ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.207.0 — the delivered artifact naming the controller it will start. Acceptance markers were the ones R-233 re-captured from a real log: docker OK (overlay2…) x1, including mount point rootfs AND mp0 x2 (there is no mp1 since build-golden.sh v3.0.0), upload OK (HTTP 201) x1, excluding 0, FATAL 0. The 404 pre-gate ran WITH a control so a 404 could not mean 'wrong URL': 0.206.0 -> 200, 0.207.0 -> 404. The token never crossed a shell — copied file->file, read by a runner script inside the VM; systemctl show grep for the value returned 0. The token-leak grep on the COMMITTED log returned 0, and that 0 is evidence because a planted copy returned 1 before being shredded. Vouch was a three-field change with all three checked deliberately: MinAgent 0.127.0 read from the golden's controller CHANGELOG header, agent_version already >= it, min_agent not above agent_version (not the R-216 shape). Verified by re-reading the manifest rather than trusting the flash. The R-120 gate did not refuse. Drill VM restored to virgin; qemu confirmed exited with ps -eo comm, not a self-matching pgrep -f. R-242: the bake half is done and the --no-verify bypass declared earlier today is now historical. Its remaining half is UNCHANGED — nothing gates the VOUCH itself, so a baked-but-unvouched golden still passes the currency gate silently.	2026-08-07 20:59:37 +02:00
admin	c1dec41328	walk5 venue TORN DOWN — census 168 rows -> 67, and R-244 grew by 30 as predicted ... Operator-confirmed. Stopped under a name guard (demo-hp carries its own 9201), aged past the hub's stale_threshold read from the DEPLOYED ConfigMap (30m), and polled delete-impact until deletable:true — treating an empty response as retry, never as success. Cascade + qm destroy --purge, guarded a second time. Every layer verified absent against a positive control that must survive and does: VM 300 drill-r50 and demo-hp's own guest 9201 still there; ep0 namespaces demo-felhom + demo-hp still there; wg peers .2 .3 .4 .250 still on the live wg0; hub rows for demo-felhom, demo-hp, peti-felhom untouched. 16.64 GiB returned against 17 G measured. RECORDED FOR THE NEXT TEARDOWN: the WG peer is removed on a ~5-minute SCHEDULE, not by the cascade. Immediately after the delete the hub row was gone while 10.77.0.5 was still on ep0's live wg0; wgsync had last run 37 seconds before the cascade, and the next push (4 peers) removed it, verified on the live interface at 16:57:07Z. The previous ledger checked this after it had already converged, so it read as instantaneous — a teardown that checks too soon would file a false finding. R-244 grew by 30 rows (app_log_issues), PREDICTED in the pre-run enumeration rather than discovered afterwards. Running total across torn-down venues ~101. Nothing here claims a clean teardown. Storage Box layer evidenced from the hub's own deprovision log: the HETZNER_API token in ~/.config/credentials cannot see box 611421 (subaccounts -> 404, storage_boxes -> 200 with 0 entries) — it is scoped to another project.	2026-08-07 18:58:55 +02:00
admin	1fc38761a8	R-249/R-252/R-253 CLOSED (controller v0.207.0); R-254 filed; R-201's state field corrected ... R-201: the STATE FIELD recorded only PASSED + PROVEN-LIVE 2026-08-04 — the data half from the drill — while the body carried yesterday's journey pass. Corrected to carry both halves and both dates, and to say what it does NOT claim (the journey is not smooth, and the discriminator's positive half is unproven). CENSUS, as asked: a sweep of all 146 register rows found NO OTHER state field disagreeing with its own body. Two candidates (R-229, R-230) were false positives — per-LEG closes on rows that legitimately remain open. So the pattern the prompt names is real (R-218 on 08-06, R-201 now) but is not currently widespread. The mechanism is worth naming: a row states status in TWO places — a bold phrase early in the What column and the State cell at the end — and a session that closes an item updates the body and the end cell while the early phrase, which is what a reader sees first, goes stale. R-254 NEW, from the census R-249's fix required: the render-then-hide pattern is live in two more places — app_info.html puts a REAL per-install app password in a hidden span, and deploy.html renders a generated secret into a value= attribute. Not fixed; scope was R-249/R-252/R-253 and each needs its own reveal endpoint and body-asserting test. R-242: the golden-currency gate FAILED as designed — v0.207.0 is released and no golden carries it. This push used --no-verify, declared here and in the report. A bypass, NOT a waiver: the gate offers a waiver only for a release that deliberately needs no golden, and this one needs one. A bake + vouch is owed. STATUS.md: 93 lines.	2026-08-07 18:17:59 +02:00
admin	3f4fb3825f	R-201 CLOSED — the unaided recovery journey passes, both halves, on the fifth walk ... Capability map: the unaided-recovery row turns FAILED -> PROVEN-LIVE, scoped, with what it still does not claim stated in the row itself: shape (c) did not fire positively (with the mint guard holding there is no local key, so the offer comes from shape (a)); and 'unaided' here means possible-without-a-shell, not obvious, because two obstacles are unsignposted. OPEN-ITEMS: R-201 closed with its evidence. Five new rows R-249..R-253 (the retrieval passphrase in page HTML; the host-key scan ladder vs AAAA settle; the listing's per-tag rows; the two unsignposted restore steps). R-243 annotated rather than re-filed: on a REBUILD offsite_delivery_stuck does not skip, so the row's gap is narrower than it reads. STATUS.md: headline changed, and trimmed 97 -> 92 lines rather than extended, per its own header. Teardown recorded as OWED with its before-measurements, the stop-and-age gate, and the positive controls that must survive.	2026-08-07 17:24:27 +02:00
admin	7850469d5b	golden 0.206.0 VOUCHED; demo-hp's stale flag cleared; session report ... VOUCHED with the operator's approval, verified from the stored hub_settings rather than the flash: golden_version 0.205.0 -> 0.206.0, sha c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e. agent_version and min_agent both stayed 0.127.0. wrapper_sha256 was carried through explicitly, because the handler CLEARS it when omitted. THE GATE WAS CONVICTED BEFORE THE BAKE AND IS OK AFTER IT - red to green on the same command, which is its proof that it measures something real. It went green on the BAKE, not the vouch; that limitation is stated in its docstring and stays open on R-242. THE STALE FLAG WAS WRONG AND IS CLEARED, with the operator's approval. One row, identity-matched on host_id and guarded on stale_at IS NOT NULL; changes() returned 1. Verified end to end, not just in the database: the hub serves the hash again, the box recorded it at 11:10:19Z, and it is byte-identical to the key that box is using - so shape (c) compares, matches and correctly stays silent. The false warning is gone, PROVEN WITH A POSITIVE CONTROL rather than an absent line: 0 escrow-confirm lines since the restart while 5 scheduler lines in the same window prove the box was logging. R-246 records the clearance and keeps the column ruling open: stale_at has NO production writer, changes what a customer is told, and can be seen by nobody who would look for it. Either give it an evidential setter or retire it. STATUS.md finished at 87 lines (from 258). Waiting-on-you is now genuinely empty: the base image is approved and live, and R-245 was re-filed as a decision taken with quota as its reopening condition. Session report: REPORT-clear-the-ground-2026-08-08.md - the six spike questions each answered with method and measurement, Q4 said plainly (only a database read), Q6 said loudly (a fresh box CANNOT reach this state, so the next walk cannot meet it), and three observations noticed but not acted on.	2026-08-07 13:13:16 +02:00
admin	f39b51782a	R-244: measure the teardown residue rather than estimate it — still owed ... Read-only census, no truncation. app_log_issues holds 1309 rows; 71 reference a torn-down venue (finalwalk, c11, rewalk, part4). Of those 44 are ORPHANS, naming only torn-down customers and safely deletable, and 27 are SHARED with a live customer and must be de-referenced, never deleted. 1238 untouched. The 27 are exactly why the leg was never written, and why a DELETE ... WHERE customer LIKE would destroy a live customer's issue history. NOT fixed here, and the reason is stated on the row rather than left implied: the fix is hub code, this session's scope forbade a hub version bump, and a hand-run SQL mutation over 71 rows - 27 needing surgical de-referencing - with no tested code path and no red-proof is the shape that goes wrong on a live database. The next session now starts from data instead of a guess.	2026-08-07 13:01:09 +02:00
admin	ceac5e0deb	STATUS.md rebuilt (258 -> 85); R-245 re-filed as decided; R-246/247/248 filed ... STATUS.md REBUILT FROM THE REGISTER, not trimmed. Its own header says one screen; it had reached 258 lines, having been 83 four days ago. The three named defects, all fixed: 1. the "waiting on you" list asked the operator to decide the RECOVERY SCREEN, built and shipped 2026-08-05, and to approve an orphaned-backup deletion the register records as DONE the same day; 2. a stray line reading only "- **Nothing.**" sat mid-list; 3. the DooPlex infrastructure work was mixed in with the product's. Infrastructure is now under ITS OWN HEADING rather than dropped, and the reason is stated on the page: these are real asks that need the operator, but they concern the machine this is built on, not what a customer receives. Dropping them would lose real work; mixing them is why the page stopped being readable. The 100-line "what shipped recently" log is gone. That is what the per-repo CHANGELOGs and the register are for, and restating it here is what made the page grow back. R-245 RE-FILED as a decision taken, not a question pending. It sat as WAITING-ON-OPERATOR for a day with nothing actually pending - it was settled on 2026-08-07. It keeps the whole reasoning and now carries the condition that would REOPEN it, which the reasoning already named: QUOTA, old set-aside history blocking new backups. A condition, not a calendar. AUDIT OF EVERY WAITING-ON-OPERATOR ROW, parsing the state column exactly rather than grepping for the phrase (which over-matches rows that merely mention it): exactly ONE row carried it - R-245 - and it was a settled decision. So zero rows were genuinely waiting, and the drift was caught while it was still a single row. R-246/R-247/R-248 file the read-only stale-blob spike's findings. R-242 updated: it recurred within a day, and shape (b) is now built - with the vouch half explicitly still open on that row rather than being papered over.	2026-08-07 12:59:52 +02:00
admin	9657334fb7	R-241 FIXED: registers, capability map, STATUS, hub CHANGELOG v0.98.0 ... R-241 closed against controller v0.206.0 + hub v0.98.0, following the spike's ruling rather than the obvious reading. The row records what the fix does AND the two real bugs the tests caught rather than review - a missing t.Enabled (caught by an EXISTING test) and a missing falling-edge sync that reintroduced the very defect the epoch exists to fix. R-243 UPDATED, not closed: the STATE it describes can no longer be entered (the mint guard), and what replaces it is VISIBLE rather than silent - the box declares awaiting_recovery_key and the customer is offered the screen. But the ALARM GAP is untouched, for the same three reasons, so a box whose customer never acts still stops backing up with no operator signal. The remaining work is an operator-side signal for a box held past some age, deliberately not bundled into R-241's fix. R-245 NEW - WAITING-ON-OPERATOR, recorded and NOT built: should an undecided customer be auto-abandoned after 30 days? The operator's proposal is recorded WITH the reasoning against it, so the decision can be revisited properly: a reinstall implies a person, so nobody is absent; a customer who cannot find their code gets in touch, which is why the operator LEVERS were the thing worth building; the cost is the customer's own storage allowance; and the real harm is QUOTA, which is a condition, not a calendar. If it is ever built, build it to trigger on the harm with a dated warning, never on a date alone. The capability map's recovery-journey row STAYS FAIL. These are fixes, not a walk - nothing here walked a customer end to end, and the row goes green only when one completes with no operator intervention AND a byte-identical sentinel. R-214, R-202 and R-240 are still open. STATUS compressed rather than extended, per its own one-screen rule, and the "rebuilding throws away the off-site history" line corrected: the cause is fixed, so leaving it as a live defect would be false. hub CHANGELOG v0.98.0 for the superseded-package purge. Highest register ID moves R-244 -> R-245.	2026-08-07 12:13:09 +02:00
admin	094e93e828	finalwalk teardown complete; R-244 filed; session report ... All five layers gone, each verified with a positive control that must survive and does: VM 324 + 4 disks -> absent (VM 300 drill-r50 remains) hub: 13 tables at 0, incl. BOTH escrow tables (demo-felhom/demo-hp/peti remain) Storage Box u629488-sub4 -> gone (sub1/2/3 remain) ep0 PBS ns finalwalk -> gone (demo-felhom, demo-hp remain) WireGuard 10.77.0.5 -> gone from the LIVE wg show on ep0, not just the hub DB (.2/.3/.4/.250 remain) 14.06 GiB reclaimed against 15 G measured before deletion. R shredded with a planted-copy control: plant -> search finds both -> shred -> the same search finds 0. The zero was not believed until the instrument was proven. R-244 (NEW): a FULL census after the cascade logged COMPLETE full teardown found 61 rows still matching finalwalk. Four sources are deliberate provenance; the fifth, app_log_issues (29 rows), is NOT covered by the residue purge - and it is systematic: c11 40, rewalk 20, part4 24 still present from the 2026-08-06 teardown, whose ledger recorded zero occurrences. That claim used a narrower query than a census and does not hold; the correction is recorded in both the prior ledger and the register rather than the measurement quietly redone. No secret material is involved. The table is a fleet-wide aggregate: 12 of the 29 rows are finalwalk-only orphans, 17 are shared with LIVE customers and must be de-referenced, not deleted - very likely why the leg was never written. Not fixed; a cascade change needs its own red-proof. Lesson, and it is the reusable part: a per-table absence query is not a census.	2026-08-07 09:59:16 +02:00
admin	db578cd44d	R-239 CLOSED (golden 0.205.0 vouched); R-241 ruling into the map and STATUS ... R-239: the operator approved the vouch this session. golden_version 0.203.0 -> 0.205.0 (+ derived sha); agent_version and min_agent both stayed 0.127.0, because the new golden's MinAgent is also 0.127.0 - so in the event it was a ONE-field change, not three. wrapper_sha256 was carried through explicitly: the handler reads it from the form and CLEARS it when omitted. Verified from the stored hub_settings (WAL-aware copy), not from the flash. The R-120 gate passed exactly - the newest controller the fleet reports is 0.205.0, so a 0.204.0 golden would have been refused. R-241: the capability map's recovery-journey row and STATUS carry the spike's ruling - a MINTING defect, not a screen-predicate defect. The row stays FAIL: delivery is not a journey, and R-241 is diagnosed, not fixed. R-242 and R-243 surfaced in STATUS in plain language.	2026-08-07 09:27:22 +02:00
admin	71c43f87c2	SPIKE R-241 — ruled: a MINTING defect, not a screen-predicate defect ... Read-only spike on the standing venue (demo-hp VM 324 finalwalk-appliance), read before anything else touched the fleet. No product code written. THE RULING. The recovery screen was telling the truth. There genuinely was nothing recoverable under the key the box held, because the box minted that key itself over the top of a sealed package it already knew the hub held. Fixing the predicate would paper over a box quietly making its own history unopenable. Three measurements carry it: 1. WriteOffboxSecrets (offbox.go:411) mints on ONE input - does the file exist. Its two neighbours in the same file both consult GetHubEscrowIdentityPresent(). Same fact on three paths, used on two. 2. The flag was the PRECONDITION of the chain that reached the minting: the retry job only logs when the declaration is live, and the venue logged it at 02:48:03Z - 30 minutes and six ticks before the mint. 3. The box KNEW and discarded it: at 03:28:03Z, 35 minutes before the customer looked, escrow_confirm.go:154 computed the exact discriminator and logged the mismatch. Never persisted, never surfaced. And the hub explicitly disclaims doing this - offsiteheal's package doc: 'credential automatic, key customer-present ... must not quietly widen'. Also answered: Q2 (shape (b) is structurally unreachable - the escrow gate sits upstream of the only orphan producer, with a positive control that the scheduler was alive); Q7 (the 'create a new code' button does NOT destroy the data - R-198's retention holds - but it converts a self-service recovery into one needing an unbuilt read path, and it re-enables the screen while invalidating the code that screen accepts). New rows: R-242 - a release that changes customer-visible behaviour is not delivered until a golden carries it, and nothing enforces that. Recorded with three proposed shapes; deliberately NOT built. R-243 - a box in this state silently stops backing up and NO alarm fires: three individually-correct exclusions leave one state unobserved. Highest register ID moves R-241 -> R-243.	2026-08-07 08:53:16 +02:00
admin	2228c0bff6	final walk COMPLETE — data PASS, journey FAIL; R-241 filed ... THE DATA: PASS. All three sentinels byte-identical out of snapshot f5c53b03, including the 12 MB binary and the accented Hungarian filename whose NAME BYTES are identical too. Disk -> restic -> SFTP -> Storage Box -> rebuilt machine -> disk, intact. THE JOURNEY: FAIL, and further from the line than the previous walk. The claim worked first try (302 in 0.164s). Then: / lands on the launcher with no recovery pointer, /recovery 302s away, and the remote page offers to CREATE a new recovery code — which would orphan the history the customer's code protects. There is no field anywhere to enter the code they hold. The operator's documented remedy also refuses, correctly and fail-closed. Recovery needed three guest command lines. R-241 — and the cause is a success this same walk proved six hours earlier. OffsiteRecoveryOffer() shows the screen only when (a) there is NO repository password (pristine rebuild) or (b) one exists but the history will not open under it. Overnight the credential self-heal collected the staged credential and applied the tier, writing a FRESH key at 03:18Z — so (a) is false; and (b) is unreachable because orphan detection needs a run, and runs are blocked by escrow_state=pending. The gap is self-locking. Measured keys: on-disk 9b4a9a9d... vs recovered-from-R 30ef574f... This is R-218's shape one level up: succeeding at the self-heal stopped the box OFFERING the recovery it still needed. Registers: R-201 moved to its outcome; R-241 filed; capability map's recovery row stays FAIL with both halves and the cause named; STATUS rewritten for the operator. Highest ID R-238 -> R-241. The venue is left with the recovered key in place and the self-heal key moved aside, never deleted. Teardown still owed.	2026-08-07 06:45:36 +02:00
admin	2d2d8d3ddc	final walk Phase A + the five checks; R-239 (delivery gap) and R-240 filed ... Phase A complete on a NEW venue (demo-hp VM 324, customer finalwalk). Installed from the published ISO — verified byte-identical to iso.felhom.eu — claimed, app deployed, three sentinels placed, escrow ceremony 6/6 with restic_pw_sealed=true, R held in one 0600 file on DooPlex, off-site backup taken and the three sentinels PROVEN IN THE SNAPSHOT BY NAME (snapshot 1da4f80d, 12.0 MB). THE HEADLINE: a fresh install lands on agent 0.127.0 (vouched, no hand upgrade — that half works) and controller 0.203.0, while the newest is 0.205.0. The vouched golden still bakes 0.203.0, so tonight's box has neither R-237 nor R-234. Filed as R-239: a delivery gap, not a regression. Every closure claimed for those two is true of the code and false of the fleet until a golden is baked and vouched. The five checks, on that box: T1 ok + the old bare-count warning (no next step) T2 FAILS — both runs say "elindult", the second as a success flash T3 FAILS — toggle off hides an existing snapshot; wizard 302s away T4 ok, and the wording is still "Sikeres" beside "nothing is selected" -> R-240 T5 PASSES — driven as a browser does, prepare -> full_prep -> confirm -> commit completed; R-238 confirmed a harness artifact, not a product defect HARNESS FAULT, caught by the §4 gate: the first off-site run reported ok with a 26.6 KB repo — impossible for a 12 MB sentinel. I had placed the files under .../adatok/felhom-data/userdata/... while this box's namespace root is /mnt/felhom-drives/adatok directly. The product was correct throughout; proving by listing rather than by a green status is what caught it before any destruction. Nothing destroyed. The venue stands.	2026-08-07 00:42:59 +02:00
admin	feed748325	R-234 root-caused and CLOSED; R-218's state field corrected ... R-234 was filed as "toggling an app on leaves it without a bundle, so the first run skips it". Measured on demo-hp: that state does not survive a run — the off-site run's own pre-dump phase calls captureAllRecoveryUnits for every DEPLOYED stack, through admitApp, before the push, and a unit moved aside was RECREATED. The actual cause was the single-flight: the manual run was dropped because an earlier one was still going, runOffboxBackup returned nil, the handler had already answered "A tavoli mentes elindult", and the card then showed the PREVIOUS run's green verdict. Fixed in controller v0.205.0 and proven live on demo-hp: a second request while one is in flight now says "Mar fut egy tavoli mentes — ez a keres nem inditott ujat. A most lathato eredmeny meg a korabbi futase", as a flash_error. Independently, and a real gap on its own: a run that skipped an app the customer selected is now `incomplete`, not `ok`. Selected+deployed with no unit counts; selected-but-undeployed is named with what to do but does NOT count, because a box left amber by an app somebody removed is a status nobody reads. R-218's state field read REOPENED while the same row's body already recorded the fix shipped in v0.203.0 and proven live. Corrected to CLOSED, keeping the over-claim history — it is why the row is worded as it is. Capability map: the off-site capture row's `incomplete` sentence widened to cover a whole-app skip, and it still does not claim a newly-selected app is protected by the next run — for a deployed app it is, for an undeployed one the card says so. Still open, deliberately: R-213, R-202, R-214, R-235.	2026-08-06 22:03:58 +02:00
admin	70b62608b0	teardown of two venues; R-218/R-220 CLOSED, R-236 WITHDRAWN, R-238 reclassified ... Part 0 — c11 and rewalk destroyed, three layers each plus the off-site side and the WireGuard peer, via the hub's own cascade (external teardown FIRST, DB purge LAST). 37.3 GB reclaimed on c11-scratch, matching the 20G+17G measured. Positive control after each: part4 must still be found, and was. ep0 namespaces now exactly demo-felhom, demo-hp, part4. The cascade refuses to delete a live host and there is no decommission endpoint, so both boxes were stopped and aged past the hub's 30m stale_threshold first. Register corrections — the durable record was wrong about two shipped fixes: R-218 REOPENED -> CLOSED. Shipped controller v0.203.0, proven live on a genuinely rebuilt box: the hub re-staged at 13:24:57Z and the box collected it on a tick, no guest command line, no operator action. R-220 "OPEN — NOT FIXED" -> CLOSED. Shipped agent v0.127.0, proven live after a real guest purge with both raw mounts still on the surviving host: /disks/candidates returned both drives (before: two empty lists) and both re-attached through the customer endpoint. R-236 WITHDRAWN — I FILED THIS WRONGLY. The hub log shows offsiteheal re-staged the stored secret at 13:24:57Z after its documented two-report debounce, with no provider credential minted. My Re-issue at 13:26:39Z came 102s LATER, was redundant, and minted an unnecessary provider credential (subaccount 284735) — the very double-issue the offsite-delivery guard warns about once a minute in the log. "Nincs teendod" is true; I did not wait ~16 minutes. Operational lesson, not a product defect. R-238 reclassified as a harness artifact (mode=full without confirm=1 is step 1 of a deliberate two-step and starts no job by design); its real residue — the total silence of that step — is fixed in controller v0.204.0. R-237 CLOSED by controller v0.204.0.	2026-08-06 16:53:38 +02:00
admin	5810e1f552	Part 4 half 2: upgrade path PASSES, recovery journey dead-ends (R-236/237/238) ... Destroyed guest 9201 (guarded on hostname — demo-hp also has a 9201), wiped both drives leaving the raw /mnt/<name> mounts in place, reinstalled from the published installer v1.25.0. PASS — the upgrade path: agent 0.127.0 -> 0.127.0 across the rebuild. The previous re-walk DOWNGRADED 0.126.0 -> 0.125.0 (R-216 part 4). Controller 0.203.0 from the baked golden, Day-0 provision SUCCESS in ~2m50s, nothing applied by hand. R-220 FIXED, proven on a genuinely rebuilt box: with both devices mounted raw AND managed, /disks/candidates offered both drives (before the fix: two empty lists), and both re-attached through the customer endpoint. R-218's consume half proven live end to end: credential re-staged at 13:26:32Z, collected on a tick, tier configured — no guest command line. Previously this exact situation was dead end 1 and needed a guest shell. FAIL — the data half, NOT reached. Three new findings, in the order they bite: R-236 nothing re-stages the off-site one-time password after a rebuild, so the "Nincs teendod" promise stalls until an operator re-issues. The retry job is NOT at fault — it retries correctly and logs why. R-237 the restore surface is keyed on installed + remote-marked apps, so after a rebuild the customer is shown no backups at all — a circular dead end. R-238 "Teljes visszaallitas elokeszitese" returns 302 and does NOTHING: no job, no state change, no error, no line in the controller's own debug ring. mode=unit on the same form works. This is why the sentinel byte-identity criterion was not met — nothing came back, rather than coming back wrong. No guest command line was used at any point in either half — the property the previous walk could not claim.	2026-08-06 16:03:04 +02:00
admin	ac451f9667	Part 4 half 1: fresh install lands on vouched 0.127.0/0.203.0 with no hand upgrade ... Venue: demo-hp VM 323 part4-appliance / guest 9201 / hub customer part4. Bind 13:53:40 -> agent 0.127.0 ONLINE 13:55:34, no "floor held", controller 0.203.0 from the baked golden. The previous re-walk's reinstall DOWNGRADED the agent (R-216 part 4); this one did not, because the vouch moved all three fields (golden_version + agent_version + min_agent) together. R-218 on a real installation: offsite-credential-retry registered (every 5m0s) and ticking, 0 "credential retry" lines, and the tier configured unaided 14s after the settle gate — the previous walk's dead end 1 is gone. Stated precisely in the journal: this venue proves the SILENT half only; the collecting half was proven separately with a positive control. R-225 visible in both states on a live box: "még ismeretlen" before any snapshot, "12.0 MB / 50 GB" after. Unknown is not zero. Fixture complete with NO guest command line — the claim used the emailed code through the real /claim form (the hub keeps only a hash), and the sentinels were written over SMB with the share mounted so the accented filename never crossed a shell (verified as hex on both sides, no U+FFFD). Two new findings, both "presence is not success": R-234 an offsite run reports success while omitting an app just toggled on — found ONLY because the pre-destruction verification restore was run instead of trusting the green tick. R-235 the console still shows the pairing banner 25 min after the bind, while promising it refreshes itself. Half 2 (destroy -> reinstall -> recover -> compare) is owed and NOT started; the venue is standing and its teardown is owed.	2026-08-06 14:56:14 +02:00
admin	1c47e3b6fd	golden 0.203.0 baked + published; runbook acceptance markers fixed (R-233) ... Bake evidence: documentation/tests/golden-0.203.0-2026-08-06/ (bake.log + README). sha256 3039c6ffa7a5a8b2d959daddb2895c58b44de70f8d4f4a7e12ad4b1c0d61dc88, verified by an independent round-trip download and by reading /etc/felhom-controller-image out of the published archive itself. NOT vouched — the hub still serves 0.201.0. R-233: RUNBOOK-manual-build.md §4.1 named three pass markers, two of which the script cannot print (`overlay2 OK` does not exist; `mp1` stopped existing in build-golden.sh v3.0.0 under R-165), and a 404 pre-gate URL with the wrong filename, which would 404 for the wrong reason and pass even when the version already existed. A grep for an impossible string reads 0 forever and 0 is indistinguishable from failure. Markers re-captured from the real log; token handling moved off the command line into an in-VM runner script; a positive control is now required on the token-leak grep; the vouch step rewritten as the three-field change it is (golden_version + agent_version + min_agent).	2026-08-06 13:23:22 +02:00
admin	a1a542b9a7	RECON: what DooPlex's own backup actually protects (R-232) — read-only, nothing changed ... Five sets, 14/14 successful runs in 14 days, and a file restored byte-identical from the data repo -- the first time a restore has ever been demonstrated here. What the failure matrix exposes: notify_failure is a no-op (NOTIFY_ON_FAILURE=true, webhook commented out) so a failed backup tells nobody; NOTHING leaves the box (Longhorn's target is DooPlex itself over NFS, the only outbound-looking cron pulls inbound for another project); the backup tree is a single writable path with no append-only; two sets are same-disk (.claude-memory and the postgres dumps, whose source sits inside the backup tree); Longhorn retains 1 generation; secrets/restic-repo has never held a snapshot because backup-secrets.sh never calls restic; and /opt/backup/docs/BACKUP-RESTORE.md does not exist though the unit advertises it. NOT a finding: the restic passphrase. On-box copy is on a different disk from the backups and the operator holds an offline copy out of band, so disk loss is recoverable. The residual is only that it is operator-held rather than system-held escrow. A stale lock in the data repo is MINE, from the previous task's verification. Fenced from clearing it; recorded with its impact and the one command that fixes it.	2026-08-06 12:16:38 +02:00
admin	5ca5082e7c	docs: close out the instruction arc — t740 corrected on evidence, registers, ledger, S-37 ... target-selection.md said demo-hp has no off-site tier. Measured first: pvesm list felhom-pbs on the box returns two snapshots in demo-hp's OWN namespace (2026-07-28, 2026-08-04) against ep0's felhom-offsite. The claim was TRUE WHEN WRITTEN and went stale when F10 resolved 2026-07-23. The measurement is kept in an HTML comment beside the corrected sentence. This file decides which machine may be destroyed, so the sentence was load-bearing, not cosmetic. R-229(b) CLOSED (agent 175 -> 99 eff). R-230(b) CLOSED (symlink, proven from fresh sessions). R-230(a) part-actioned -- three false statements fixed, WARN loop added, bulk ruling still owed. S-37: a claim in an instruction file is checked, not trusted.	2026-08-06 11:49:22 +02:00
admin	9c10c5d600	docs: ledger, registers and standing ruling S-36 for the part-2 instruction/memory work (R-229) ... R-229 legs (a) and (c) CLOSED. R-230 opened for what this deliberately left -- the ruling owed on auto-written staleness, the symlink decision, the spec-as-failing-test pilot. R-231 opened for /opt/backup/scripts/ being unversioned host state, found while adding the memory store to the backup set: the same class this task was closing, inside the fix for it. S-36: the auto-memory store is backed up, never committed; no memory file is ever deleted; a rule file created mid-session is NOT live in that session. Second session in this repo, so REPORT-<topic>.md per CLAUDE.md; shared REPORT.md untouched.	2026-08-06 11:01:13 +02:00
admin	c21bcf84f7	docs+gate: instruction files cannot silently regrow (R-229) ... New shared scripts/instructions_gate.py, registered in controller_gates.py and agent_gates.py, never copied into a sibling repo (the reuse_refs_check.py precedent). 20 fixture tests, all asserting the effect: exit code AND that the message names the file and the reason. It is a consistency gate, not a budget gate, and the failure message says so. A /context reading measured the instruction files at 15k tokens against 869k free in a 1M window -- space is not the constraint, and a future reader must not re-derive the wrong reason. The 200-line ceiling is adherence guidance; a file nobody can hold in their head is where contradictions hide, and five were found here. Checks run against effective text (HTML comments stripped, because they are stripped before injection): the line ceiling; every .claude/rules/*.md declares paths: or an explicit unconditional: true; no component version literal; no TEMPORARY block carrying a past date; and the workspace-root CLAUDE.md is byte-identical to its versioned copy -- the live file sits outside any git repo, so that copy is its only version-controlled record. Two traps recorded so they are not reintroduced: a bare \d+\.\d+\.\d+ matches the first three octets of every IPv4 (the gate excludes dotted quads, or it fails on 192.168.0.180 in the agent's own file); and unconditional: true is NOT a Claude Code feature but this project's own marker. Workspace-root CLAUDE.md 208 -> 182 lines (142 effective), copy kept identical. The nine-instance invariant table moved into the felhom-testing skill, which triggers when writing or reviewing a test; all three directive bullets stayed in the core. felhom.eu/CLAUDE.md got surgical corrections only and is knowingly still over the ceiling at 227 effective lines -- closing it needs the restructure R-229 defers, said plainly rather than quietly absorbed. CONTEXT.md gains standing ruling S-35. OPEN-ITEMS.md gains R-229. Docs only -- no Go, no version bump, nothing built or deployed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JJc8sAGRWmavP3rMtdpkr2	2026-08-06 09:38:52 +02:00
admin	d30c2a51ed	R-224..R-228 CLOSED: registers, capability map, campaign annotation, STATUS ... Five closed in controller v0.202.0 + agent v0.126.0, each with its live or red-proof evidence in the row. Five explicitly still open and named as such rather than left to inference: R-214, R-220, R-221, R-213, R-202 — and R-220 is flagged as currently worked around BY HAND on the campaign venue, which is the only reason an app could be deployed there. The capability map's recovery row STAYS FAIL and says why: fixes are not a re-walk, nothing walked a customer end to end, and the customer-facing messages were NOT re-driven live because /recovery correctly retires itself once the old data is set aside — restoring that state is the reconfiguration the task forbade. The campaign document is ANNOTATED, not rewritten: it records what was true when it ran, and that is its value. workspace-CLAUDE.md gains comment-vs-code entry 9 — the escrow header said the errors were 'DISTINCT on purpose' and named THREE situations while a fourth was folded into one of them, and a green test named the defect and did not prevent it because it asserted a STRING one layer below the merge. ROADMAP needed no collapse — it carries no rows for these IDs.	2026-08-06 08:34:04 +02:00