1059 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
admin	3bf62b95bb	fix(gate): the wire-contract search shelled out to grep and read its failure as a finding ... CI convicted ALL 174 checked tags while the pre-push hook was green. Cause, read from the run log rather than guessed at the second attempt: the search used `grep -rnE --include=…`, and the CI runner's image carries python3 and git and deliberately little else — its grep does not support `--include`, so stdout was empty and the gate read empty as "the tag is absent". That is a gate silently treating a tool failure as a finding, which is worse than no gate, and it is exactly the error-swallowing this repo forbids. A green from it would have been just as untrustworthy as the red. Fixed by removing the dependency, not by working around it: the search is now pure Python — one token index per receiving repo, built in a single pass, no subprocess. Faster too (one walk instead of ~350 greps), and unreadable-file / empty-repo cases now exit 2 INCONCLUSIVE rather than reporting absence. THE BEFORE CAPTURE WAS RE-VERIFIED, NOT RE-GENERATED — the stronger claim. All 40 fields recorded in BEFORE.md were re-tested against the new implementation: agree=40, disagree=0, i.e. exactly the four this session fixed are now present and the other 36 still absent. The number 40 stands under both implementations; only the mechanism changed. The whole-token property survives by construction — a token index treats `healed_at` and `privsep_healed_at` as distinct tokens. This is the THIRD instrument defect this gate's own controls caught before it was trusted, after the substring false negative and the dr_recipe over-opacity. The first two were caught by re-finding the known instances; this one by the CI-versus-hook disagreement the workflow's alarm mail explicitly says outranks whatever the push was for.	2026-08-08 09:10:59 +02:00
admin	436abf39d5	ci: give the wire-contract gate the sibling clone it needs (the hook and CI disagreed) ... CI went red on the two G-1 commits while the local pre-push hook was GREEN — which this workflow's own alarm mail says outranks whatever the push was for, because it is a finding about the gates themselves. It was. CAUSE, reproduced rather than guessed: scripts/wire_contract_gate.py compares what one component EMITS against what the other can RECEIVE, so it needs the SOURCE of the controller and the agent. The workflow already fetched the controller (for golden-currency) but not the agent, so the gate exited 2 INCONCLUSIVE. Reproduced locally by pointing the gate at a nonexistent agent path: same exit 2, same message. THE LESSON, and it is the reusable half: the pre-push hook runs on a workstation where every sibling is a real clone, so a gate that needs a sibling passes there and is inconclusive here. **The two automated homes are not interchangeable, and a NEW GATE MUST BE CHECKED IN BOTH.** Recorded in the step's own comment beside the fetch. Fixed by giving the gate what it needs. NOT by letting it skip when a sibling is absent — that is the fail-open shape, and it would leave the gate running in NEITHER home, which is the R-29 census failure this runner was built to end and which golden_currency_gate.py's docstring already warns about at length.	2026-08-08 09:07:40 +02:00
admin	9771fd9c27	deploy: hub 0.99.0 — the manifest is the truth (R-260) ... A built image deploys nothing until this tag moves in git and the app is synced. felhom-hub:0.99.0 confirmed present in the registry (manifest HTTP 200) before the bump.	2026-08-08 09:03:23 +02:00
admin	2ce3c2a0f2	golden 0.209.0 BAKED, PUBLISHED, ROUND-TRIP VERIFIED — the currency gate goes green (R-242) ... The G-1 session released controller v0.209.0 (R-247), which made golden_currency_gate.py correctly red and REFUSED THE PUSH: no golden carried the newest release. The honest answer to that is the bake it asks for, not --no-verify. The gate's own docstring says the cost of a trip is one bake, which is the operation this project wants to be routine. 656 697 956 B, sha256 c9c4bcd6..e818ff. Round-tripped: the published bytes downloaded back, hashed independently, size and sha identical, and ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.209.0 — the delivered artifact naming the controller it will start. Acceptance markers all green (overlay2 x1, mount points x2 rootfs+mp0, upload HTTP 201 x1, excluding/FATAL/mp1 x0), Result=success, ExecMainStatus=0. 404 pre-gate with a 200 control on 0.208.0 so a 404 could not mean "wrong URL". Token file->file into a 0600 file read inside the VM; systemctl show grep = 0; committed-log grep = 0 WITH a control returning 1 to prove the grep works. Bake VM destroyed, /root residue clean, qemu confirmed gone, drill disk restored to virgin. IT ALSO CONSOLIDATES THE OPERATOR'S APPROVAL. Golden 0.208.0 was baked last night and never vouched; 0.209.0 contains everything it did plus R-247, so it supersedes rather than wastes it. One Save, not two — STATUS.md updated accordingly and back to its 93-line screen. NOT VOUCHED. Fresh installs still land on 0.207.0 until the operator saves. And R-242's untouched half showed itself again: this gate flipped green on the presence of the evidence DIRECTORY, with no vouch anywhere near it. Recorded, not built — ROADMAP G-8. repo_gates --fast: all 8 OK, including wire-contract and golden-currency.	2026-08-08 09:01:26 +02:00
admin	b080ecf411	hub v0.99.0 — the hub can see whether the operator can get in (R-260); G-1 gate closes, R-247 closes ... oobDegraded tested five things and the sixth never arrived. The agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the SAME version that introduced the `oob` stanza carrying it — and store.HostOOBRow mirrored five of the agent's eight OOB fields. With no field for it, encoding/json discarded the fact on arrival, so a box with felhom-sshd active, reachable, a valid config and a configured peer reported `ok` with NO OPERATOR KEY INSTALLED AT ALL. Not a wrong answer: an answer to a question nobody was asking. `operator_peer_configured`, which the hub did read, only says the peer IP is in desired-state — that OOB is MEANT to work, not that entry is possible. Now decoded: operator_key_configured, plus wg_handshake_age_s and healed_at. The last two ride the ALERT TEXT and are deliberately NOT in the predicate — widening a check beyond the fact that is now arriving is how a check stops being read. SCENARIO F, decided on a measurement rather than a preference. operator_key_configured decodes as a POINTER: nil = the agent never said, reported distinctly and never as ok. The version gate was rejected because the field and its stanza shipped in the SAME agent version (v0.72.0), so a stanza without the field cannot come from any released agent; the fleet is 0.113.0/0.127.0 and the vouched floor is 0.127.0. Handled explicitly anyway and pinned, because "cannot happen" is a claim this project has been burned by. THE MESSAGE NAMES THE FAULT. oobDegradedReason is the single source for both predicate and text, so the alert can never name a different fault from the one that fired. The old form derived it separately and had a vocabulary of two — unreachable, or config invalid — with no way to say the key is missing. The operator reads this at 07:00. TESTS DRIVE THE DECODE BOUNDARY. Every hub OOB test before this built a HostOOBRow by hand, and a test written that way CANNOT SEE A FIELD THAT NEVER DECODES — which is how this held a green suite for five weeks. The pre-existing fixture oobReport() also omitted the field, so those scenarios ran against a report shape no released agent produces (same family as R-262). Both fixed. Red-proofs, 8 expected outcomes and 0 wrong, each with the mutation asserted applied: dropping the field returns the false ok; an unconditional check alerts a healthy box; unknown-as-ok restores the silent pass. G-1 CLOSED — scripts/wire_contract_gate.py shipped as ranked, built BEFORE the fixes and seen failing on 40 fields (documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md). Two instrument defects the control caught first: a substring false negative (grep -F healed_at matched privsep_healed_at) and treating dr_recipe as wholly opaque when its top-level sections ARE decoded through an allow-list that already cost offsite_restic (R-122). The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo: R-260 said "at least eight DECISION-BEARING facts", never eight tags. The real count is 40. R-260 CLOSED (class gated, sharpest instance fixed). R-247 CLOSED (controller v0.209.0). R-264 MINTED and OPEN — the 21 facts with no consumer, allowlisted with reasons so that gating the class could not be mistaken for deciding them. Still open and named: R-246, R-255..R-259, R-261..R-263, and C7's test-comment half. Capability map checked: it claims OOB access is implemented, never monitored, so no row was untrue; what was untrue sat one layer down and the row now records it. repo_gates --fast: all 8 OK. go build/vet/test green in hub, run separately from this commit.	2026-08-08 08:47:02 +02:00
admin	560f0d4451	G-1: a gate for the dropped field — built first, and seen failing on 40 ... Campaign 12 ranked this first of eight gating candidates. It is built BEFORE the fixes it finds, because last night an off-the-shelf tool for a neighbouring class (deadcode, for C6) was made to prove itself first and found NEITHER of the two defects it was meant for. A gate nobody has watched fail has not been shown to work. scripts/wire_contract_gate.py, registered in repo_gates.py as --fast (no network, no container, so it runs in BOTH the pre-push hook and CI — the R-29 constraint). THE TEST. For every json tag reachable from a declared wire ROOT, does that literal tag occur anywhere in the receiving repo's production Go or templates? A tag occurring nowhere cannot be decoded by any struct, named OR anonymous. That last clause is why a string test is used instead of comparing struct to struct: Campaign 12's first attempt paired types by shape and false-positived badly, because the hub decodes one report through several ad-hoc anonymous structs. RESULT ON TODAY'S TREE: 210 tags checked across 3 declared wires, 51 skipped (generic / opaque / allowlisted), 40 CONVICTED. Captured verbatim in documentation/tests/wire-contract-gate-2026-08-08/ BEFORE.md, which is deliverable 1 of this session. The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo rather than quoted: R-260's wording was "at least eight DECISION-BEARING facts", not eight tags in total. The real count on the three declared wires is 40, and R-260's own census already listed more than eight. Recorded because this prompt's own rule 6 says not to quote a document as source. TWO THINGS THE CONTROL CAUGHT, both before the gate was trusted: 1. A SUBSTRING FALSE NEGATIVE. `grep -F healed_at` also matches `privsep_healed_at`, so a genuinely dropped field read as received — and R-260 named healed_at, so its absence from the output was the tell. Now a whole-token regex; healed_at is convicted. 2. dr_recipe IS NOT WHOLLY OPAQUE. The hub stores each half as json.RawMessage and re-emits nested shapes verbatim, so the LEAVES are genuinely not on this wire. But the TOP-LEVEL SECTION KEYS are decoded by hostHalfShape/appHalfShape, and those are ALLOW-LISTS: a section an emitter adds is silently dropped until named in both. That already cost `offsite_restic` (R-122). So the gate is opaque BELOW depth 1, not opaque — the sections are checked and pass. Self-test: `--selftest` plants an unreachable tag on a real root in a throwaway copy and asserts conviction. Verified: exit 1, planted tag named. Blind spots are in the module docstring AND in the gate's own output, because Campaign 12's C1 guard turned out blind to one of the three shapes it was written for: generic tag names are not checked; reachability of a NAME is not use of a VALUE; only declared ROOTS are covered, and the hub's desired-state (served as raw stored JSON, no typed emitter) and the agent local API are NOT. Allowlist entries carry a stated reason. A quiet exclusion is a dropped field with paperwork. Not pushed alone: the fixes follow in the next commit so main is never red on this check.	2026-08-08 08:28:53 +02:00
admin	b7fb2117ae	CAMPAIGN 12 — the class sweep: golden 0.208.0 baked (awaiting vouch), R-256..R-263 filed, gating ranked ... Part 1. Golden 0.208.0 baked on the drill VM, published and ROUND-TRIP VERIFIED — 656 150 362 B, sha256 ba668f59..5ffb82, and ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.208.0. Acceptance markers all green (overlay2 x1, mount points x2 rootfs+mp0, upload HTTP 201 x1, excluding/FATAL/mp1 x0), Result=success. Token file->file, read inside the VM; systemctl show grep = 0; committed-log grep = 0 WITH a control proving the grep works. Bake VM destroyed, drill disk restored to virgin. NOT VOUCHED — the campaign halts there deliberately. golden_currency_gate.py was correctly RED on arrival and is green after the bake. No --no-verify was needed anywhere in this session. Parts 2-4. Seven defect classes swept for siblings by class rather than by feature. Analysis only: no product code, nothing deployed, no machine touched beyond the bake VM. Eight new rows R-256..R-263 (ceiling moved from R-255), grouped by class in OPEN-ITEMS.md. C1 produced no new instance and has no row. The sharpest is R-260: the agent reports operator_key_configured every heartbeat, the hub has no field for it, so the check that answers "can the operator get into this box" returns ok for a box with no operator key installed. Every class states whether its method re-found the known instances, because a method that cannot re-find them has not been shown to work: C1 2/3 (verified by replaying the pre-fix templates), C2 2/2, C3 2/3 + 1 as fixed, C4 fix-pattern re-found, C5 re-found, C6 deadcode 0/2 and bespoke 1/2, C7 weakest and said so. Blind spots stated per class; seven suspicions investigated and DISPROVED, including two of my own methods. Part 4's ranking is in ROADMAP.md as G-1..G-8. Gate C5 (cross-repo tag reachability — cheap, --fast-eligible, would have caught every R-260 instance on the introducing commit). Do NOT gate C6: golang.org/x/tools/cmd/deadcode was measured against a PLANTED probe and is blind to unreachable METHODS on widely-used types, which is exactly the shape both known instances have. R-242's untouched half is recorded, not built: this bake demonstrated it, the currency gate flipping green the moment the evidence DIRECTORY existed, before the round trip finished and with no vouch near it. Correction the campaign owed its own brief: escrow_stale was described as closed; it is R-247 and READY. The live repo is the source. Sampled rather than swept, exactly: C7 60 of 2652 production invariant comments and NONE of the 1440 test comments (that half is owed); C2 19 of 221 refusals; C3/C4 controller only. No finding was reproduced live. STATUS.md is 100 lines against its 93-line one screen.	2026-08-07 23:07:42 +02:00
admin	59527d00f9	R-254 CLOSED both sites (controller v0.208.0); R-255 filed; R-242 red again ... R-254 site one was the same defect and is fixed the same way. Site two was NOT the defect the row described: the pre-deploy hidden input is deliberate (a form must carry what it submits, README §318) and was left alone; the indefensible one was the readonly display input on an ALREADY-DEPLOYED app, where nothing is submitted. The premise that this broke a repo rule does not hold and is recorded rather than dropped: no line in the repo says 'no silent auto-fill'. What exists is CONTEXT.md:2070, about accidental EMPTY-password deployments. §7.3 measured on the fleet: site one's code path has never run (crafty-controller is the only app declaring initial_credentials and is deployed nowhere); site two's exposure is also empty (demo-hp runs three apps, none with a generated secret field). HONEST LIMIT: that is a current-state measurement, and nothing recorded reads — which was part of the fault. No evidence of exposure, and no mechanism that could have produced evidence either way. Rotation not indicated by anything measured. R-255 NEW: the guard covers 4 of 27 pages at runtime, and the cheap all-templates gate is blind to the shape that actually shipped (a secret under a neutral page-data key) — both verified, both stated in the gate's own docstring. Filed rather than declaring a partial guard complete. R-242 red a second time in 24h; --no-verify declared. The cadence is the argument for its other half: nothing gates the vouch.	2026-08-07 21:27:20 +02:00
admin	f651b31a7a	golden 0.207.0 BAKED, PUBLISHED, ROUND-TRIP VERIFIED and VOUCHED — the currency gate goes green ... Closes the delivery gap v0.207.0 opened this session. Until now the gate was correctly red and a machine installed today would have received 0.206.0 — the release written, tested and pushed, and not delivered. Round trip is the evidence, not the build log: the published bytes were downloaded back (656 879 192 B, sha256 20ec9602…22995, both identical to what the bake reported) and ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.207.0 — the delivered artifact naming the controller it will start. Acceptance markers were the ones R-233 re-captured from a real log: docker OK (overlay2…) x1, including mount point rootfs AND mp0 x2 (there is no mp1 since build-golden.sh v3.0.0), upload OK (HTTP 201) x1, excluding 0, FATAL 0. The 404 pre-gate ran WITH a control so a 404 could not mean 'wrong URL': 0.206.0 -> 200, 0.207.0 -> 404. The token never crossed a shell — copied file->file, read by a runner script inside the VM; systemctl show grep for the value returned 0. The token-leak grep on the COMMITTED log returned 0, and that 0 is evidence because a planted copy returned 1 before being shredded. Vouch was a three-field change with all three checked deliberately: MinAgent 0.127.0 read from the golden's controller CHANGELOG header, agent_version already >= it, min_agent not above agent_version (not the R-216 shape). Verified by re-reading the manifest rather than trusting the flash. The R-120 gate did not refuse. Drill VM restored to virgin; qemu confirmed exited with ps -eo comm, not a self-matching pgrep -f. R-242: the bake half is done and the --no-verify bypass declared earlier today is now historical. Its remaining half is UNCHANGED — nothing gates the VOUCH itself, so a baked-but-unvouched golden still passes the currency gate silently.	2026-08-07 20:59:37 +02:00
admin	c1dec41328	walk5 venue TORN DOWN — census 168 rows -> 67, and R-244 grew by 30 as predicted ... Operator-confirmed. Stopped under a name guard (demo-hp carries its own 9201), aged past the hub's stale_threshold read from the DEPLOYED ConfigMap (30m), and polled delete-impact until deletable:true — treating an empty response as retry, never as success. Cascade + qm destroy --purge, guarded a second time. Every layer verified absent against a positive control that must survive and does: VM 300 drill-r50 and demo-hp's own guest 9201 still there; ep0 namespaces demo-felhom + demo-hp still there; wg peers .2 .3 .4 .250 still on the live wg0; hub rows for demo-felhom, demo-hp, peti-felhom untouched. 16.64 GiB returned against 17 G measured. RECORDED FOR THE NEXT TEARDOWN: the WG peer is removed on a ~5-minute SCHEDULE, not by the cascade. Immediately after the delete the hub row was gone while 10.77.0.5 was still on ep0's live wg0; wgsync had last run 37 seconds before the cascade, and the next push (4 peers) removed it, verified on the live interface at 16:57:07Z. The previous ledger checked this after it had already converged, so it read as instantaneous — a teardown that checks too soon would file a false finding. R-244 grew by 30 rows (app_log_issues), PREDICTED in the pre-run enumeration rather than discovered afterwards. Running total across torn-down venues ~101. Nothing here claims a clean teardown. Storage Box layer evidenced from the hub's own deprovision log: the HETZNER_API token in ~/.config/credentials cannot see box 611421 (subaccounts -> 404, storage_boxes -> 200 with 0 entries) — it is scoped to another project.	2026-08-07 18:58:55 +02:00
admin	1fc38761a8	R-249/R-252/R-253 CLOSED (controller v0.207.0); R-254 filed; R-201's state field corrected ... R-201: the STATE FIELD recorded only PASSED + PROVEN-LIVE 2026-08-04 — the data half from the drill — while the body carried yesterday's journey pass. Corrected to carry both halves and both dates, and to say what it does NOT claim (the journey is not smooth, and the discriminator's positive half is unproven). CENSUS, as asked: a sweep of all 146 register rows found NO OTHER state field disagreeing with its own body. Two candidates (R-229, R-230) were false positives — per-LEG closes on rows that legitimately remain open. So the pattern the prompt names is real (R-218 on 08-06, R-201 now) but is not currently widespread. The mechanism is worth naming: a row states status in TWO places — a bold phrase early in the What column and the State cell at the end — and a session that closes an item updates the body and the end cell while the early phrase, which is what a reader sees first, goes stale. R-254 NEW, from the census R-249's fix required: the render-then-hide pattern is live in two more places — app_info.html puts a REAL per-install app password in a hidden span, and deploy.html renders a generated secret into a value= attribute. Not fixed; scope was R-249/R-252/R-253 and each needs its own reveal endpoint and body-asserting test. R-242: the golden-currency gate FAILED as designed — v0.207.0 is released and no golden carries it. This push used --no-verify, declared here and in the report. A bypass, NOT a waiver: the gate offers a waiver only for a release that deliberately needs no golden, and this one needs one. A bake + vouch is owed. STATUS.md: 93 lines.	2026-08-07 18:17:59 +02:00
admin	3f4fb3825f	R-201 CLOSED — the unaided recovery journey passes, both halves, on the fifth walk ... Capability map: the unaided-recovery row turns FAILED -> PROVEN-LIVE, scoped, with what it still does not claim stated in the row itself: shape (c) did not fire positively (with the mint guard holding there is no local key, so the offer comes from shape (a)); and 'unaided' here means possible-without-a-shell, not obvious, because two obstacles are unsignposted. OPEN-ITEMS: R-201 closed with its evidence. Five new rows R-249..R-253 (the retrieval passphrase in page HTML; the host-key scan ladder vs AAAA settle; the listing's per-tag rows; the two unsignposted restore steps). R-243 annotated rather than re-filed: on a REBUILD offsite_delivery_stuck does not skip, so the row's gap is narrower than it reads. STATUS.md: headline changed, and trimmed 97 -> 92 lines rather than extended, per its own header. Teardown recorded as OWED with its before-measurements, the stop-and-age gate, and the positive controls that must survive.	2026-08-07 17:24:27 +02:00
admin	0691bc59a5	walk5 (R-201): Phase B + the verdict — BOTH HALVES PASS ... THE DATA: PASS. All three sentinels byte-identical out of snapshot 5b0f20f7, including the accented filename's bytes, read back as bytes from the live path. THE JOURNEY: PASS — the first time in five walks. Zero guest command lines were needed to progress; the previous walk needed three. The reset-code hatch was used once, in Phase A only. §5's observation, which stands on its own whatever the verdict: at 14:58:52Z the rebuilt box collected its re-staged credential, configured the transport, and REFUSED TO MINT a repository password over the sealed package the hub holds. At the equivalent moment the previous walk minted a fresh key and lost the journey silently at 03:18. Sampled every 20s from T0: no key at any moment. Honest about which shape fired: with the mint guard holding there is no local key, so the offer comes from shape (a), not shape (c). Shape (c) was measured in Phase A in its NEGATIVE half (equal hashes, correctly silent). This proves the mint guard positively and the discriminator negatively. RTO 71.7s login to open store, of which 12.44s was the unseal and ~22s my own CSRF harness retry. Two new customer-facing obstacles, neither needing a shell but neither signposted: the restore refuses on unattached drives, and refuses because the app is not installed on a page that says the restore reinstalls it.	2026-08-07 17:18:37 +02:00
admin	879007aaec	walk5 (R-201, fifth walk): Phase A journal — fixture built, sentinels proved by name ... Written before the destruction, per §9.9. A fresh install landed on the VOUCHED set with no hand upgrade (controller 0.206.0, agent 0.127.0) — R-239's delivery gap is closed for this run, which is the first of the five walks where the box under test is the box a customer receives. Both §4.6 pre-destruction checks pass, neither previously exercised on a clean box: the recovery offer is correctly SILENT (shape (c) compares equal — the two key hashes are byte-identical on box and hub), and the restore page lists the app with the future-backup toggle OFF (R-237's fix, which the last walk measured failing). Also recorded: the §4.5 gate caught a harness fault (a toggle sent as enabled=1 rather than enabled=on) that had produced a green 'ok' over a zero-snapshot repository — the exact shape the gate exists for.	2026-08-07 15:05:35 +02:00
admin	10af63d0b0	report: record the CI breakage the new gate caused, and its fix ... CI checks out one repo shallow, so the golden-currency gate found no sibling controller clone, exited 2 (INCONCLUSIVE) and turned CI red on every push. Caught by pulling the run status rather than assuming it. Fixed by fetching the controller repo in CI - not by letting the gate skip, which would have been fail-open and would have left it running in neither of its automated homes. Runs 244 and 245 confirm green. Also records the reusable lesson: the pre-push hook and CI do not see the same filesystem, so a gate that reads a sibling repo can pass locally and break CI.	2026-08-07 13:16:32 +02:00
admin	6f25e02828	CI: give the golden-currency gate the sibling clone it needs, instead of letting it skip ... CI checks out ONE repo, shallow. The R-242 gate reads the newest RELEASED controller from the sibling clone's CHANGELOG.md - the same sibling assumption reuse_refs_check.py and instructions_gate.py already make - so in CI it was exiting 2 (INCONCLUSIVE) and CI went red on every push. Caught by checking the CI result rather than assuming it: runs 241, 242 and 243 all failed while the gates were green locally. A PERMANENTLY-RED CI IS THE DETECTOR-NOBODY-HEARS FAILURE THIS WORKFLOW EXISTS TO PREVENT - people stop reading it, and then it catches nothing. So the fix is to give the gate what it needs, not to let it skip when the sibling is absent: a skip would be the fail-open shape this project keeps removing, and the gate would then run in NEITHER of its two automated homes (the pre-push hook and CI). Depth 1, pinned to main, plain git - no JavaScript-action step, per the workflow's own rule about the runner having python3 and git and nothing else. If the fetch fails the gate still reports INCONCLUSIVE rather than passing.	2026-08-07 13:15:06 +02:00
admin	7850469d5b	golden 0.206.0 VOUCHED; demo-hp's stale flag cleared; session report ... VOUCHED with the operator's approval, verified from the stored hub_settings rather than the flash: golden_version 0.205.0 -> 0.206.0, sha c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e. agent_version and min_agent both stayed 0.127.0. wrapper_sha256 was carried through explicitly, because the handler CLEARS it when omitted. THE GATE WAS CONVICTED BEFORE THE BAKE AND IS OK AFTER IT - red to green on the same command, which is its proof that it measures something real. It went green on the BAKE, not the vouch; that limitation is stated in its docstring and stays open on R-242. THE STALE FLAG WAS WRONG AND IS CLEARED, with the operator's approval. One row, identity-matched on host_id and guarded on stale_at IS NOT NULL; changes() returned 1. Verified end to end, not just in the database: the hub serves the hash again, the box recorded it at 11:10:19Z, and it is byte-identical to the key that box is using - so shape (c) compares, matches and correctly stays silent. The false warning is gone, PROVEN WITH A POSITIVE CONTROL rather than an absent line: 0 escrow-confirm lines since the restart while 5 scheduler lines in the same window prove the box was logging. R-246 records the clearance and keeps the column ruling open: stale_at has NO production writer, changes what a customer is told, and can be seen by nobody who would look for it. Either give it an evidential setter or retire it. STATUS.md finished at 87 lines (from 258). Waiting-on-you is now genuinely empty: the base image is approved and live, and R-245 was re-filed as a decision taken with quota as its reopening condition. Session report: REPORT-clear-the-ground-2026-08-08.md - the six spike questions each answered with method and measurement, Q4 said plainly (only a database read), Q6 said loudly (a fresh box CANNOT reach this state, so the next walk cannot meet it), and three observations noticed but not acted on.	2026-08-07 13:13:16 +02:00
admin	721297ed5e	golden 0.206.0 baked, published, round-trip verified — NOT vouched; the gate goes green ... Controller v0.206.0 shipped the R-241 fixes on 2026-08-07 while the vouched golden still carried 0.205.0, so a machine installed this morning would have received neither - and the next recovery walk would have measured the old behaviour and failed for a reason nothing to do with the walk. Same gap as R-239, one day after R-239 was closed. version 0.206.0 sha256 c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e size 656,750,694 bytes (uncompressed 2,003,138,560) MinAgent 0.127.0 Round-trip verified rather than trusted: the published bytes were fetched back, re-hashed independently (match), zstd-tested, and ./etc/felhom-controller-image was read OUT of the download -> felhom-controller:0.206.0. That last step is the one that matters, because GOLDEN_VERSION is derived from the tag argument and could be right over stale content. All acceptance markers pass; unit Result=success ExecMainStatus=0. Secret hygiene: token file->file, in-VM runner so it never reached a command line (unit-property grep 0), literal-value leak grep on the COMMITTED log 0 - with a positive control proving the grep works before the 0 was believed. Bake VM torn down: CT 9100 purged, secrets shredded, qemu observed gone via ps -eo comm, drill.qcow2 reverted to virgin. THE GATE BUILT EARLIER THIS SESSION NOW PASSES. It was shown CONVICTED against the pre-bake state and is OK now - red to green on the same check, the same command, which is its proof that it measures something real. Note it went green on the BAKE, not the vouch: that is its stated limitation, and the vouch is still pending the operator. NOT VOUCHED - the operator's act. Only one field moves: golden_version 0.205.0 -> 0.206.0 (+ its derived sha). agent_version and min_agent both stay 0.127.0. wrapper_sha256 is unchanged but is CLEARED if omitted from the POST.	2026-08-07 13:05:40 +02:00
admin	f39b51782a	R-244: measure the teardown residue rather than estimate it — still owed ... Read-only census, no truncation. app_log_issues holds 1309 rows; 71 reference a torn-down venue (finalwalk, c11, rewalk, part4). Of those 44 are ORPHANS, naming only torn-down customers and safely deletable, and 27 are SHARED with a live customer and must be de-referenced, never deleted. 1238 untouched. The 27 are exactly why the leg was never written, and why a DELETE ... WHERE customer LIKE would destroy a live customer's issue history. NOT fixed here, and the reason is stated on the row rather than left implied: the fix is hub code, this session's scope forbade a hub version bump, and a hand-run SQL mutation over 71 rows - 27 needing surgical de-referencing - with no tested code path and no red-proof is the shape that goes wrong on a live database. The next session now starts from data instead of a guess.	2026-08-07 13:01:09 +02:00
admin	ceac5e0deb	STATUS.md rebuilt (258 -> 85); R-245 re-filed as decided; R-246/247/248 filed ... STATUS.md REBUILT FROM THE REGISTER, not trimmed. Its own header says one screen; it had reached 258 lines, having been 83 four days ago. The three named defects, all fixed: 1. the "waiting on you" list asked the operator to decide the RECOVERY SCREEN, built and shipped 2026-08-05, and to approve an orphaned-backup deletion the register records as DONE the same day; 2. a stray line reading only "- **Nothing.**" sat mid-list; 3. the DooPlex infrastructure work was mixed in with the product's. Infrastructure is now under ITS OWN HEADING rather than dropped, and the reason is stated on the page: these are real asks that need the operator, but they concern the machine this is built on, not what a customer receives. Dropping them would lose real work; mixing them is why the page stopped being readable. The 100-line "what shipped recently" log is gone. That is what the per-repo CHANGELOGs and the register are for, and restating it here is what made the page grow back. R-245 RE-FILED as a decision taken, not a question pending. It sat as WAITING-ON-OPERATOR for a day with nothing actually pending - it was settled on 2026-08-07. It keeps the whole reasoning and now carries the condition that would REOPEN it, which the reasoning already named: QUOTA, old set-aside history blocking new backups. A condition, not a calendar. AUDIT OF EVERY WAITING-ON-OPERATOR ROW, parsing the state column exactly rather than grepping for the phrase (which over-matches rows that merely mention it): exactly ONE row carried it - R-245 - and it was a settled decision. So zero rows were genuinely waiting, and the drift was caught while it was still a single row. R-246/R-247/R-248 file the read-only stale-blob spike's findings. R-242 updated: it recurred within a day, and shape (b) is now built - with the vouch half explicitly still open on that row rather than being papered over.	2026-08-07 12:59:52 +02:00
admin	3ca9a7bbe6	R-242: build the gate the rule described — a release without a golden now fails the push ... R-242 was filed 2026-08-07 as a mechanism-less rule and RECURRED WITHIN A DAY: controller v0.206.0 shipped the R-241 fixes while the vouched golden still carried 0.205.0, so a machine installed this morning would have received neither. Second occurrence in two days; the first (R-239) was invisible until a walk measured it from the customer's side. SHOWN FAILING FIRST, against today's state, before anything was baked - that is the gate's red-proof and the whole point of building it before the bake: newest released controller : 0.206.0 newest golden baked : 0.205.0 GOLDEN CURRENCY GATE FAILED ... A machine installed right now would receive v0.205.0 - the release is written, tested and pushed, and NOT delivered. Entry point exits 1; summary reports CONVICTED: golden-currency. *** THIS PUSH USED --no-verify, to push past the gate's OWN conviction. *** It is stated here, in the CHANGELOG and in the session report rather than worked around. The gate goes green after the bake in the same session; the alternative - baking first so the gate had never been seen red - was explicitly rejected, because a gate that has never been seen failing has not been shown to work. IT IS --fast, AND THAT FORCED THE DESIGN. Both the pre-push hook and CI run repo_gates.py --fast, which by contract selects only gates touching no network. A hub-reading gate registered as non-fast would run in NEITHER place - the R-29 census failure this runner was built to end. SO IT CHECKS THE BAKE, NOT THE VOUCH. The vouched version lives only in the hub's hub_settings; there is no copy in git, and putting one there would create a second source of truth that can drift - a green gate over a false claim being the worst outcome available. A bake without a vouch still passes. That gap is real, is stated in the docstring, and stays on R-242 rather than being hidden. The recurrence this gate exists for was a missing BAKE. IT COMPARES VERSIONS, NOT BEHAVIOUR, so a release that changed nothing customer-visible also trips it. Accepted deliberately: judging "customer-visible" by hand is what failed twice, and the cost of a false trip is one bake. A waiver belongs in the register, never in a habit of bypassing. Inconclusive (exit 2) on an absent controller clone or an unparseable header: not knowing is never a pass.	2026-08-07 12:53:43 +02:00
admin	b12f8ec2f3	deploy hub v0.98.0 — the R-241 superseded-package purge ... The manifest is the truth: the image was built and pushed, but nothing deploys until this tag moves and the app is synced.	2026-08-07 12:20:59 +02:00
admin	9657334fb7	R-241 FIXED: registers, capability map, STATUS, hub CHANGELOG v0.98.0 ... R-241 closed against controller v0.206.0 + hub v0.98.0, following the spike's ruling rather than the obvious reading. The row records what the fix does AND the two real bugs the tests caught rather than review - a missing t.Enabled (caught by an EXISTING test) and a missing falling-edge sync that reintroduced the very defect the epoch exists to fix. R-243 UPDATED, not closed: the STATE it describes can no longer be entered (the mint guard), and what replaces it is VISIBLE rather than silent - the box declares awaiting_recovery_key and the customer is offered the screen. But the ALARM GAP is untouched, for the same three reasons, so a box whose customer never acts still stops backing up with no operator signal. The remaining work is an operator-side signal for a box held past some age, deliberately not bundled into R-241's fix. R-245 NEW - WAITING-ON-OPERATOR, recorded and NOT built: should an undecided customer be auto-abandoned after 30 days? The operator's proposal is recorded WITH the reasoning against it, so the decision can be revisited properly: a reinstall implies a person, so nobody is absent; a customer who cannot find their code gets in touch, which is why the operator LEVERS were the thing worth building; the cost is the customer's own storage allowance; and the real harm is QUOTA, which is a condition, not a calendar. If it is ever built, build it to trigger on the harm with a dated warning, never on a date alone. The capability map's recovery-journey row STAYS FAIL. These are fixes, not a walk - nothing here walked a customer end to end, and the row goes green only when one completes with no operator intervention AND a byte-identical sentinel. R-214, R-202 and R-240 are still open. STATUS compressed rather than extended, per its own one-screen rule, and the "rebuilding throws away the off-site history" line corrected: the cause is fixed, so leaving it as a live defect would be false. hub CHANGELOG v0.98.0 for the superseded-package purge. Highest register ID moves R-244 -> R-245.	2026-08-07 12:13:09 +02:00
admin	ac4b2a4ba9	hub: drop the retained recovery package when the box declares its set-aside history deleted (R-241) ... The hub half of the controller's abandonment countdown, and the ONLY reason felhom.eu was touched for R-241 at all. A customer who abandons their old off-site history gets a 14-day countdown. At the end of it the controller deletes the set-aside restic store and then DECLARES offsite.abandon_purge_requested in its report until the retained sealed package that protected that store is gone too. Removing only one half leaves a state that asks a question nobody can answer: a package that opens nothing, or ciphertext nobody can ever decrypt. store.PurgeSupersededEscrowForCustomer is the one place R-198's retention is ever undone, and its doc comment says why that is legitimate here. It NEVER touches host_escrow - the current package covers the key the box is using now and is what makes its live backups recoverable. Only host_escrow_superseded rows go. The handler acts on the box's DECLARATION, never an inference, on the same principle as offsite.state: the hub cannot see that a remote store was deleted and the box can. It is placed immediately BEFORE the ACK is built, deliberately. GetEscrowStatusForCustomer is read after it runs, so the SAME response that carries the request's effect is what closes the box's two-phase commit - no second round-trip, and no window in which the box believes it is still owed. The declaration repeats on every report until that ACK stops reporting a superseded package, so a lost request retries by itself. A purge failure is logged at ERROR and never swallowed: the box keeps declaring, so it retries, but an operator must be able to see that the two halves are apart right now. An idempotent re-declaration (already purged, the box has not yet seen the confirming ACK) logs at DEBUG and is not an error. Audit event offsite_abandon_purged is hub-internal, like the pbsdr_* and offsite_selfheal_* events - allowedEventTypes governs the box-pushed POST /event surface, not this. No agent change. No deletion has been performed against any real store. Green: go build, go vet, go test ./... all clean in hub/; repo gates OK.	2026-08-07 11:48:10 +02:00
admin	094e93e828	finalwalk teardown complete; R-244 filed; session report ... All five layers gone, each verified with a positive control that must survive and does: VM 324 + 4 disks -> absent (VM 300 drill-r50 remains) hub: 13 tables at 0, incl. BOTH escrow tables (demo-felhom/demo-hp/peti remain) Storage Box u629488-sub4 -> gone (sub1/2/3 remain) ep0 PBS ns finalwalk -> gone (demo-felhom, demo-hp remain) WireGuard 10.77.0.5 -> gone from the LIVE wg show on ep0, not just the hub DB (.2/.3/.4/.250 remain) 14.06 GiB reclaimed against 15 G measured before deletion. R shredded with a planted-copy control: plant -> search finds both -> shred -> the same search finds 0. The zero was not believed until the instrument was proven. R-244 (NEW): a FULL census after the cascade logged COMPLETE full teardown found 61 rows still matching finalwalk. Four sources are deliberate provenance; the fifth, app_log_issues (29 rows), is NOT covered by the residue purge - and it is systematic: c11 40, rewalk 20, part4 24 still present from the 2026-08-06 teardown, whose ledger recorded zero occurrences. That claim used a narrower query than a census and does not hold; the correction is recorded in both the prior ledger and the register rather than the measurement quietly redone. No secret material is involved. The table is a fleet-wide aggregate: 12 of the 29 rows are finalwalk-only orphans, 17 are shared with LIVE customers and must be de-referenced, not deleted - very likely why the leg was never written. Not fixed; a cascade change needs its own red-proof. Lesson, and it is the reusable part: a per-table absence query is not a census.	2026-08-07 09:59:16 +02:00
admin	db578cd44d	R-239 CLOSED (golden 0.205.0 vouched); R-241 ruling into the map and STATUS ... R-239: the operator approved the vouch this session. golden_version 0.203.0 -> 0.205.0 (+ derived sha); agent_version and min_agent both stayed 0.127.0, because the new golden's MinAgent is also 0.127.0 - so in the event it was a ONE-field change, not three. wrapper_sha256 was carried through explicitly: the handler reads it from the form and CLEARS it when omitted. Verified from the stored hub_settings (WAL-aware copy), not from the flash. The R-120 gate passed exactly - the newest controller the fleet reports is 0.205.0, so a 0.204.0 golden would have been refused. R-241: the capability map's recovery-journey row and STATUS carry the spike's ruling - a MINTING defect, not a screen-predicate defect. The row stays FAIL: delivery is not a journey, and R-241 is diagnosed, not fixed. R-242 and R-243 surfaced in STATUS in plain language.	2026-08-07 09:27:22 +02:00
admin	08b75e602e	golden 0.205.0 baked, published and round-trip verified — NOT vouched (R-239) ... Closes the delivery gap's build half. The vouched golden carried controller 0.203.0 while 0.205.0 was released, so a machine installed last night got neither R-237 (restore list keyed on the store) nor R-234 (skipped-app verdict). Both were measured from the customer's side on that box. version 0.205.0 sha256 8f49b2e8ccbc86a49df821fee9fb00c07293758811d3d0f0512dd0cf5fd54ee8 size 656,937,561 bytes (uncompressed 2,003,343,360) MinAgent 0.127.0 Round-trip verified rather than trusted: the published bytes were fetched back, re-hashed (match), zstd-tested, and ./etc/felhom-controller-image was read OUT of the downloaded archive -> felhom-controller:0.205.0. That last step is the one that matters, because GOLDEN_VERSION is derived from the tag argument and could have been right over stale content. Acceptance markers all pass; unit Result=success ExecMainStatus=0. Secret hygiene: token file->file, in-VM runner so it never reached a command line (unit-property grep 0), literal-value leak grep on the COMMITTED log 0 - with a positive control proving the grep works before the 0 was believed. Bake VM torn down: CT 9100 purged, secrets shredded, qemu observed gone via ps -eo comm, drill.qcow2 reverted to virgin. NOT VOUCHED - that is the operator's act. Only ONE field actually moves: golden_version 0.203.0 -> 0.205.0 (+ its derived sha). agent_version and min_agent both stay 0.127.0, because the new golden's MinAgent is also 0.127.0. The R-120 gate passes exactly: the newest controller the fleet reports is 0.205.0, so a 0.204.0 golden would have been refused.	2026-08-07 09:05:59 +02:00
admin	71c43f87c2	SPIKE R-241 — ruled: a MINTING defect, not a screen-predicate defect ... Read-only spike on the standing venue (demo-hp VM 324 finalwalk-appliance), read before anything else touched the fleet. No product code written. THE RULING. The recovery screen was telling the truth. There genuinely was nothing recoverable under the key the box held, because the box minted that key itself over the top of a sealed package it already knew the hub held. Fixing the predicate would paper over a box quietly making its own history unopenable. Three measurements carry it: 1. WriteOffboxSecrets (offbox.go:411) mints on ONE input - does the file exist. Its two neighbours in the same file both consult GetHubEscrowIdentityPresent(). Same fact on three paths, used on two. 2. The flag was the PRECONDITION of the chain that reached the minting: the retry job only logs when the declaration is live, and the venue logged it at 02:48:03Z - 30 minutes and six ticks before the mint. 3. The box KNEW and discarded it: at 03:28:03Z, 35 minutes before the customer looked, escrow_confirm.go:154 computed the exact discriminator and logged the mismatch. Never persisted, never surfaced. And the hub explicitly disclaims doing this - offsiteheal's package doc: 'credential automatic, key customer-present ... must not quietly widen'. Also answered: Q2 (shape (b) is structurally unreachable - the escrow gate sits upstream of the only orphan producer, with a positive control that the scheduler was alive); Q7 (the 'create a new code' button does NOT destroy the data - R-198's retention holds - but it converts a self-service recovery into one needing an unbuilt read path, and it re-enables the screen while invalidating the code that screen accepts). New rows: R-242 - a release that changes customer-visible behaviour is not delivered until a golden carries it, and nothing enforces that. Recorded with three proposed shapes; deliberately NOT built. R-243 - a box in this state silently stops backing up and NO alarm fires: three individually-correct exclusions leave one state unobserved. Highest register ID moves R-241 -> R-243.	2026-08-07 08:53:16 +02:00
admin	2228c0bff6	final walk COMPLETE — data PASS, journey FAIL; R-241 filed ... THE DATA: PASS. All three sentinels byte-identical out of snapshot f5c53b03, including the 12 MB binary and the accented Hungarian filename whose NAME BYTES are identical too. Disk -> restic -> SFTP -> Storage Box -> rebuilt machine -> disk, intact. THE JOURNEY: FAIL, and further from the line than the previous walk. The claim worked first try (302 in 0.164s). Then: / lands on the launcher with no recovery pointer, /recovery 302s away, and the remote page offers to CREATE a new recovery code — which would orphan the history the customer's code protects. There is no field anywhere to enter the code they hold. The operator's documented remedy also refuses, correctly and fail-closed. Recovery needed three guest command lines. R-241 — and the cause is a success this same walk proved six hours earlier. OffsiteRecoveryOffer() shows the screen only when (a) there is NO repository password (pristine rebuild) or (b) one exists but the history will not open under it. Overnight the credential self-heal collected the staged credential and applied the tier, writing a FRESH key at 03:18Z — so (a) is false; and (b) is unreachable because orphan detection needs a run, and runs are blocked by escrow_state=pending. The gap is self-locking. Measured keys: on-disk 9b4a9a9d... vs recovered-from-R 30ef574f... This is R-218's shape one level up: succeeding at the self-heal stopped the box OFFERING the recovery it still needed. Registers: R-201 moved to its outcome; R-241 filed; capability map's recovery row stays FAIL with both halves and the cause named; STATUS rewritten for the operator. Highest ID R-238 -> R-241. The venue is left with the recovered key in place and the self-heal key moved aside, never deleted. Teardown still owed.	2026-08-07 06:45:36 +02:00
admin	1ff6f8f8e0	final walk §7: the credential chain runs end to end, unaided, on an UNCLAIMED box ... Three questions answered from the hub's own log, not inferred: 1. the rebuilt, still-unclaimed box DOES report (host-report + Received report) 2. it DOES declare offsite.state=needs_credential, and offsite-delivery correctly declines once a minute, naming internal/offsiteheal as the owner 3. offsiteheal re-staged UNAIDED at 03:15Z, after two reports carried the declaration, with no provider credential minted — about 32 minutes after the rebuild, matching the documented 2x15-minute debounce And then the box COLLECTED it on its own 5-minute tick: [offsite-apply] credential retry: the staged credential was collected and the tier applied That success line shipped in v0.203.0 and this is the FIRST time it has been seen live: yesterday's walk only produced its sibling before I intervened at 102s and mistook my own button press for the cause — the error that produced R-236 and forced its withdrawal. Here nobody touched anything and the box was not even claimed. R-218's consume half, R-236's withdrawal and the previous walk's dead end 1 are all settled by one unattended observation.	2026-08-07 05:23:03 +02:00
admin	1b490c8cbf	final walk: destroyed, rebuilt, and HALTED at the claim screen ... Destroyed 02:40:31Z (guarded on hostname — demo-hp also has a guest 9201), drives wiped to 20K with the mounts deliberately left in place because the surviving raw mount IS the R-220 condition. Reinstalled through the published day-0 path, installer v1.25.0 fetched live; Day-0 provision SUCCESS in 2m32s. R-239 measured a second time, from the other side: the rebuild landed on agent 0.127.0 (no downgrade, no hand upgrade — that half is right) and controller 0.203.0. The box a customer would recover on tonight also lacks R-234 and R-237. The machine is AT THE CLAIM SCREEN awaiting the operator. A claim code has already been requested through the customer-facing path and emailed, so the morning is paste-a-code rather than request-then-paste. The reset-code hatch was NOT used and will not be: it is a guest command line and would fail the rule the walk measures. Stated plainly in the journal: journey steps from the destruction onward are driven over HTTP from the appliance to the guest's island address, as a browser would; some instrumentation reads are guest command lines and are counted as such, but none changed state or was needed to progress the journey.	2026-08-07 04:46:11 +02:00
admin	502078bebf	final walk: the soak DID catch a scheduled cycle; both directions recorded ... Window 2 (01:56-02:36Z) caught the off-site nightly at 02:15:03Z — snaps 1->2, last_run advanced with no prompting. Window 1 had already caught the local legs (db-dump 00:30:25Z, tier + fill-watch 01:30:19Z). My 04:15 prediction was wrong in the other direction; recorded rather than quietly fixed. What should have fired and did not: NOTHING. Six registered jobs were never seen in the log and none is a finding — four are quiet by construction (scheduler.go:267, quiet := job.Interval <= 30s) and two run every 6h, outside a 4h window. Checked the source rather than filing four phantom defects. What fired and should not have: nothing. No alert, notification, email or digest. The only WARN lines in five hours were three of mine. Observation kept: offbox-backup ticked twice 62s apart and produced ONE new snapshot — the second silently dropped by the single-flight, which is correct for the nightly path and is the same mechanism that produced R-234 on the manual path.	2026-08-07 04:39:42 +02:00
admin	f873c557ca	final walk: the soak, and the window correction stated as mine ... Window 1 (22:44-01:54Z, untouched): every periodic job fired at its declared cadence — agent-channel-health 185x/1m, stack-scan 108x/2m, the 5-minute trio 44x each, hub-report 14x/15m. offsite-credential-retry ran 44 times and did no work and said nothing, which is R-218's asserted healthy-box behaviour on a box that never saw the defect. No alert, notification or digest fired; nothing on the must-not list fired. The off-site state never moved during that window — and that is NOT a finding. The daily jobs run on the controller's clock and the guest is UTC while the appliance is CEST, so the nightly local (~02:30) and off-site (~04:15) fall at 02:30Z and 04:15Z. I sized the window against CEST and it closed at 01:54Z, before either. Reporting 'the nightly did not fire' would have been a false finding produced by a badly-chosen window. Window 2 (corrective, 01:56-02:35Z) covers the 02:30Z local backup. The 04:15Z off-site nightly is deliberately NOT covered and is recorded as not run with the reason: leaving the machine at the claim screen before 07:00 is the primary deliverable, and waiting would have put the destruction at ~06:35 CEST with no margin. The off-site tier was exercised four times manually instead, including a full listing by name.	2026-08-07 03:57:18 +02:00
admin	2d2d8d3ddc	final walk Phase A + the five checks; R-239 (delivery gap) and R-240 filed ... Phase A complete on a NEW venue (demo-hp VM 324, customer finalwalk). Installed from the published ISO — verified byte-identical to iso.felhom.eu — claimed, app deployed, three sentinels placed, escrow ceremony 6/6 with restic_pw_sealed=true, R held in one 0600 file on DooPlex, off-site backup taken and the three sentinels PROVEN IN THE SNAPSHOT BY NAME (snapshot 1da4f80d, 12.0 MB). THE HEADLINE: a fresh install lands on agent 0.127.0 (vouched, no hand upgrade — that half works) and controller 0.203.0, while the newest is 0.205.0. The vouched golden still bakes 0.203.0, so tonight's box has neither R-237 nor R-234. Filed as R-239: a delivery gap, not a regression. Every closure claimed for those two is true of the code and false of the fleet until a golden is baked and vouched. The five checks, on that box: T1 ok + the old bare-count warning (no next step) T2 FAILS — both runs say "elindult", the second as a success flash T3 FAILS — toggle off hides an existing snapshot; wizard 302s away T4 ok, and the wording is still "Sikeres" beside "nothing is selected" -> R-240 T5 PASSES — driven as a browser does, prepare -> full_prep -> confirm -> commit completed; R-238 confirmed a harness artifact, not a product defect HARNESS FAULT, caught by the §4 gate: the first off-site run reported ok with a 26.6 KB repo — impossible for a 12 MB sentinel. I had placed the files under .../adatok/felhom-data/userdata/... while this box's namespace root is /mnt/felhom-drives/adatok directly. The product was correct throughout; proving by listing rather than by a green status is what caught it before any destruction. Nothing destroyed. The venue stands.	2026-08-07 00:42:59 +02:00
admin	feed748325	R-234 root-caused and CLOSED; R-218's state field corrected ... R-234 was filed as "toggling an app on leaves it without a bundle, so the first run skips it". Measured on demo-hp: that state does not survive a run — the off-site run's own pre-dump phase calls captureAllRecoveryUnits for every DEPLOYED stack, through admitApp, before the push, and a unit moved aside was RECREATED. The actual cause was the single-flight: the manual run was dropped because an earlier one was still going, runOffboxBackup returned nil, the handler had already answered "A tavoli mentes elindult", and the card then showed the PREVIOUS run's green verdict. Fixed in controller v0.205.0 and proven live on demo-hp: a second request while one is in flight now says "Mar fut egy tavoli mentes — ez a keres nem inditott ujat. A most lathato eredmeny meg a korabbi futase", as a flash_error. Independently, and a real gap on its own: a run that skipped an app the customer selected is now `incomplete`, not `ok`. Selected+deployed with no unit counts; selected-but-undeployed is named with what to do but does NOT count, because a box left amber by an app somebody removed is a status nobody reads. R-218's state field read REOPENED while the same row's body already recorded the fix shipped in v0.203.0 and proven live. Corrected to CLOSED, keeping the over-claim history — it is why the row is worded as it is. Capability map: the off-site capture row's `incomplete` sentence widened to cover a whole-app skip, and it still does not claim a newly-selected app is protected by the next run — for a deployed app it is, for an undeployed one the card says so. Still open, deliberately: R-213, R-202, R-214, R-235.	2026-08-06 22:03:58 +02:00
admin	fca85498e1	Part 5: the last venue torn down after its findings were written ... part4 (VM 323) was kept while Parts 1 and 3 read it — it was standing in the exact failed half-B state and tearing it down first would have destroyed the evidence. Destroyed after the diagnoses were committed (70b6260): VM, hub host, hub customer, u629488-sub6, ep0 namespace, wg peer 10.77.0.7, all verified absent with demo-hp as the positive control. Final: demo-hp holds only VM 300 (drill-r50) and its own CT 9201; ep0 holds only demo-felhom and demo-hp. 53.4 GB reclaimed across the three venues. The cascade purges all escrow ciphertext, so R_C11_REWALK and R_PART4 in the operator's credentials file are now inert.	2026-08-06 17:25:20 +02:00
admin	190c432f3a	capability map: the recovery journey stays FAIL, with the two dead ends closed and vouched ... R-218 and R-220 shipped (controller v0.203.0 / agent v0.127.0) and are proven live on a genuinely rebuilt box; golden 0.203.0 + agent 0.127.0 + min_agent 0.127.0 are vouched, so the delivery gap the re-walk recorded is gone. The row stays FAIL because the walk did not finish: it stopped at R-237 (the restore list was keyed on installed-and-toggled apps), now fixed in v0.204.0 and proven live — but no sentinel was restored, so the data half is unproven in either direction for that venue. R-238 reclassified (harness artifact, real residue fixed); R-236 withdrawn.	2026-08-06 17:02:55 +02:00
admin	70b62608b0	teardown of two venues; R-218/R-220 CLOSED, R-236 WITHDRAWN, R-238 reclassified ... Part 0 — c11 and rewalk destroyed, three layers each plus the off-site side and the WireGuard peer, via the hub's own cascade (external teardown FIRST, DB purge LAST). 37.3 GB reclaimed on c11-scratch, matching the 20G+17G measured. Positive control after each: part4 must still be found, and was. ep0 namespaces now exactly demo-felhom, demo-hp, part4. The cascade refuses to delete a live host and there is no decommission endpoint, so both boxes were stopped and aged past the hub's 30m stale_threshold first. Register corrections — the durable record was wrong about two shipped fixes: R-218 REOPENED -> CLOSED. Shipped controller v0.203.0, proven live on a genuinely rebuilt box: the hub re-staged at 13:24:57Z and the box collected it on a tick, no guest command line, no operator action. R-220 "OPEN — NOT FIXED" -> CLOSED. Shipped agent v0.127.0, proven live after a real guest purge with both raw mounts still on the surviving host: /disks/candidates returned both drives (before: two empty lists) and both re-attached through the customer endpoint. R-236 WITHDRAWN — I FILED THIS WRONGLY. The hub log shows offsiteheal re-staged the stored secret at 13:24:57Z after its documented two-report debounce, with no provider credential minted. My Re-issue at 13:26:39Z came 102s LATER, was redundant, and minted an unnecessary provider credential (subaccount 284735) — the very double-issue the offsite-delivery guard warns about once a minute in the log. "Nincs teendod" is true; I did not wait ~16 minutes. Operational lesson, not a product defect. R-238 reclassified as a harness artifact (mode=full without confirm=1 is step 1 of a deliberate two-step and starts no job by design); its real residue — the total silence of that step — is fixed in controller v0.204.0. R-237 CLOSED by controller v0.204.0.	2026-08-06 16:53:38 +02:00
admin	5810e1f552	Part 4 half 2: upgrade path PASSES, recovery journey dead-ends (R-236/237/238) ... Destroyed guest 9201 (guarded on hostname — demo-hp also has a 9201), wiped both drives leaving the raw /mnt/<name> mounts in place, reinstalled from the published installer v1.25.0. PASS — the upgrade path: agent 0.127.0 -> 0.127.0 across the rebuild. The previous re-walk DOWNGRADED 0.126.0 -> 0.125.0 (R-216 part 4). Controller 0.203.0 from the baked golden, Day-0 provision SUCCESS in ~2m50s, nothing applied by hand. R-220 FIXED, proven on a genuinely rebuilt box: with both devices mounted raw AND managed, /disks/candidates offered both drives (before the fix: two empty lists), and both re-attached through the customer endpoint. R-218's consume half proven live end to end: credential re-staged at 13:26:32Z, collected on a tick, tier configured — no guest command line. Previously this exact situation was dead end 1 and needed a guest shell. FAIL — the data half, NOT reached. Three new findings, in the order they bite: R-236 nothing re-stages the off-site one-time password after a rebuild, so the "Nincs teendod" promise stalls until an operator re-issues. The retry job is NOT at fault — it retries correctly and logs why. R-237 the restore surface is keyed on installed + remote-marked apps, so after a rebuild the customer is shown no backups at all — a circular dead end. R-238 "Teljes visszaallitas elokeszitese" returns 302 and does NOTHING: no job, no state change, no error, no line in the controller's own debug ring. mode=unit on the same form works. This is why the sentinel byte-identity criterion was not met — nothing came back, rather than coming back wrong. No guest command line was used at any point in either half — the property the previous walk could not claim.	2026-08-06 16:03:04 +02:00
admin	ac451f9667	Part 4 half 1: fresh install lands on vouched 0.127.0/0.203.0 with no hand upgrade ... Venue: demo-hp VM 323 part4-appliance / guest 9201 / hub customer part4. Bind 13:53:40 -> agent 0.127.0 ONLINE 13:55:34, no "floor held", controller 0.203.0 from the baked golden. The previous re-walk's reinstall DOWNGRADED the agent (R-216 part 4); this one did not, because the vouch moved all three fields (golden_version + agent_version + min_agent) together. R-218 on a real installation: offsite-credential-retry registered (every 5m0s) and ticking, 0 "credential retry" lines, and the tier configured unaided 14s after the settle gate — the previous walk's dead end 1 is gone. Stated precisely in the journal: this venue proves the SILENT half only; the collecting half was proven separately with a positive control. R-225 visible in both states on a live box: "még ismeretlen" before any snapshot, "12.0 MB / 50 GB" after. Unknown is not zero. Fixture complete with NO guest command line — the claim used the emailed code through the real /claim form (the hub keeps only a hash), and the sentinels were written over SMB with the share mounted so the accented filename never crossed a shell (verified as hex on both sides, no U+FFFD). Two new findings, both "presence is not success": R-234 an offsite run reports success while omitting an app just toggled on — found ONLY because the pre-destruction verification restore was run instead of trusting the green tick. R-235 the console still shows the pairing banner 25 min after the bind, while promising it refreshes itself. Half 2 (destroy -> reinstall -> recover -> compare) is owed and NOT started; the venue is standing and its teardown is owed.	2026-08-06 14:56:14 +02:00
admin	1c47e3b6fd	golden 0.203.0 baked + published; runbook acceptance markers fixed (R-233) ... Bake evidence: documentation/tests/golden-0.203.0-2026-08-06/ (bake.log + README). sha256 3039c6ffa7a5a8b2d959daddb2895c58b44de70f8d4f4a7e12ad4b1c0d61dc88, verified by an independent round-trip download and by reading /etc/felhom-controller-image out of the published archive itself. NOT vouched — the hub still serves 0.201.0. R-233: RUNBOOK-manual-build.md §4.1 named three pass markers, two of which the script cannot print (`overlay2 OK` does not exist; `mp1` stopped existing in build-golden.sh v3.0.0 under R-165), and a 404 pre-gate URL with the wrong filename, which would 404 for the wrong reason and pass even when the version already existed. A grep for an impossible string reads 0 forever and 0 is indistinguishable from failure. Markers re-captured from the real log; token handling moved off the command line into an in-VM runner script; a positive control is now required on the token-leak grep; the vouch step rewritten as the three-field change it is (golden_version + agent_version + min_agent).	2026-08-06 13:23:22 +02:00
admin	0c4411e54b	R-201 re-walk: the data PASSES again, the journey still FAILS — two dead ends, down from four ... Asked Campaign 11 Phase 1's question a second time, on the fixed build, on a NEW appliance (VM 322, customer rewalk). The Campaign 11 venue was untouched. THE DATA: PASS. All three sentinels byte-identical out of the pre-destruction snapshot a7bc23bd in 23s through the customer's own restore flow — including a 12 MB binary and an accented Hungarian filename whose NAME BYTES are identical too (verified as hex, not as rendered text). THE JOURNEY: FAIL, two dead ends against Phase 1's four. 1. R-218's CONSUME half. The hub re-staged the credential at 11:44:57 saying 'the box re-consumes on its next cycle'; a full cycle ran at 11:55:46/54 (with a positive control that it ran) and it did not. A census of the customer-reachable actions found none that fetches it. Only a command line INSIDE THE GUEST moved it — 18s, confirming nothing was wrong with the credential, target or key: only the trigger. R-218's row said SHIPPED and over-claimed; it is corrected to REOPENED for the consume half. 2. R-220. Drives still unenrollable after a rebuild, needing a Proxmox-host unmount; without it no app redeploys and the restore page stays empty. Unaided RTO STILL UNDEFINED. Attended: +45s key placed, +24m12s tier up, +30m13s data verified. The 30m must not be quoted as the customer number. What passed and is new: the recovery screen appeared WITHOUT being sought, answered all three questions with a seal date matching the hub exactly, the emailed reset code worked first try, the unlock was a real 1.528s unseal, and R-225's fix was seen working in the wild (unknown, not a false zero). R-216 part 4 reproduced live: the reinstall downgraded the hand-installed agent 0.126.0 -> 0.125.0. DELIVERY GAP recorded as owed and NOT conflated with the journey: a fresh install landed on controller 0.201.0 / agent 0.125.0 — the vouched versions, neither carrying the fixes — installed by hand. Nothing was vouched. Capability map row STAYS FAIL. Campaign 11 doc gets a dated ADDENDUM, not a rewrite.	2026-08-06 12:18:29 +02:00
admin	a1a542b9a7	RECON: what DooPlex's own backup actually protects (R-232) — read-only, nothing changed ... Five sets, 14/14 successful runs in 14 days, and a file restored byte-identical from the data repo -- the first time a restore has ever been demonstrated here. What the failure matrix exposes: notify_failure is a no-op (NOTIFY_ON_FAILURE=true, webhook commented out) so a failed backup tells nobody; NOTHING leaves the box (Longhorn's target is DooPlex itself over NFS, the only outbound-looking cron pulls inbound for another project); the backup tree is a single writable path with no append-only; two sets are same-disk (.claude-memory and the postgres dumps, whose source sits inside the backup tree); Longhorn retains 1 generation; secrets/restic-repo has never held a snapshot because backup-secrets.sh never calls restic; and /opt/backup/docs/BACKUP-RESTORE.md does not exist though the unit advertises it. NOT a finding: the restic passphrase. On-box copy is on a different disk from the backups and the operator holds an offline copy out of band, so disk loss is recoverable. The residual is only that it is operator-held rather than system-held escrow. A stale lock in the data repo is MINE, from the previous task's verification. Fenced from clearing it; recorded with its impact and the one command that fixes it.	2026-08-06 12:16:38 +02:00
admin	b1bf1f271c	REPORT + STATUS: instruction arc close-out ... Operator line for the one change with real-world consequence: the runbook that says which machine may be wrecked claimed the HP box keeps no off-site copy, and it keeps two. Checked against the machine, not corrected from memory.	2026-08-06 11:50:28 +02:00
admin	5ca5082e7c	docs: close out the instruction arc — t740 corrected on evidence, registers, ledger, S-37 ... target-selection.md said demo-hp has no off-site tier. Measured first: pvesm list felhom-pbs on the box returns two snapshots in demo-hp's OWN namespace (2026-07-28, 2026-08-04) against ep0's felhom-offsite. The claim was TRUE WHEN WRITTEN and went stale when F10 resolved 2026-07-23. The measurement is kept in an HTML comment beside the corrected sentence. This file decides which machine may be destroyed, so the sentence was load-bearing, not cosmetic. R-229(b) CLOSED (agent 175 -> 99 eff). R-230(b) CLOSED (symlink, proven from fresh sessions). R-230(a) part-actioned -- three false statements fixed, WARN loop added, bulk ruling still owed. S-37: a claim in an instruction file is checked, not trusted.	2026-08-06 11:49:22 +02:00
admin	f49b1f390b	workspace: the root CLAUDE.md becomes a symlink; check 5 learns two shapes (R-230(b)) ... Two files kept identical by hand and one check is a divergence class policed forever; one file reachable by two paths cannot diverge at all. install_workspace.py now links by default, MIGRATES an existing regular file (backing it up first and SAYING SO if it differed -- that difference is the last chance to notice an unsynced edit), and keeps --copy for a clone that wants the old shape. Check 5 asserts a different thing per shape: for a link, that it points at the versioned copy and resolves to a real file; for two files, byte-identity as before. A dangling link is worse than a diverged copy -- the instructions load NOTHING and there is no content left to notice is wrong -- so that case is red-proofed. NOT yet proven to LOAD: that needs a fresh session and a hook line, which is Phase 7. If it does not load, this reverts to the copy.	2026-08-06 11:41:36 +02:00
admin	15fa5273ba	gate: check 7 (register citations) + content WARNings on the memory index ... Check 7 catches "cites a register item and calls it open when it is not" -- the R-168 class, four files, one self-contradicting. Trigger is an openness CLAIM, not any citation: policing every mention would fire on ~30 legitimate provenance citations and the gate would be switched off. Deliberate deviation from the task's literal wording, to keep it alive. Two bugs found by the check's own red-proofs, both of which would have shipped: - the state marker is not self-closing (**SHIPPED - text**), so the first parser read R-168 itself as OPEN -- a gate that cannot convict its founding case is decoration; - the CLOSED exemption was line-wide, so "shipped" in a title pardoned "OPEN R-25b". Check 6 gains WARN-only content classes on MEMORY.md. Link targets are stripped first: the earlier scan reported three expired statements, all three false (dates in filenames), while missing the one real expired claim, whose deadline was written ~08-02 with no ISO date. 39 -> 60 assertions. All four runners green.	2026-08-06 11:39:34 +02:00
admin	92a076c239	docs: backup verified IN the repository — snapshot b587f775, 118/118 + 40/40 (R-229) ... 58,158 files / 405.865 GiB in 27:56; 994 MiB added, 77 MiB stored -- the one-time full re-read cost I/O, not storage, as predicted. Also records a false alarm I raised against my own instrument: `restic ls <snapshot> <path>` reported 0 of 40 archived memories because restic 0.18.0's path filter does not recurse into subdirectories. The unfiltered listing shows all 40. An instrument that can drop results silently is not a measurement -- and this one was measuring a safety property.	2026-08-06 11:22:34 +02:00
admin	6a835ad9ec	Re-walk R-201: Phase A record, written BEFORE anything is destroyed ... The fixture is built and the gate passed: all three sentinels are in snapshot a7bc23bd BY NAME and at the right sizes, and the accented filename survived into restic intact. Recorded because they are results, not incidentals: - what a FRESH install actually landed on — controller 0.201.0, agent 0.125.0, golden 0.201.0. Neither carries the fixes. Brought to 0.202.0/0.126.0 BY HAND, and the delivery gap is recorded as owed: this proves the journey, NOT that a customer would receive that build. - baseline drift caught on arrival: one documentation-only R-229 commit per repo, and the highest register ID is R-229, not the runbook's R-228. - the runbook's installer baseline (1.25.0) is stale; the published artifact is 1.26.1, verified byte-identical to demo-hp's local copy. - credentials DISCOVERED not assumed: HUB_PW mapped and verified live; nothing else was needed from the file. A harness fault worth keeping: the accented sentinel's filename was destroyed at creation by my own base64->bash->pct exec chain, producing U+FFFD replacement characters. A Python decode('utf-8') check called it VALID UTF-8, because U+FFFD is valid UTF-8 — only a hex dump of the filename bytes exposed it. Rewritten from explicit bytes inside Python on the guest. R claimed one-shot, streamed file->file to a 0600 store on DooPlex, never rendered, shape verified without printing (10 words, 75 chars). The runbook's §5.5-vs-§9.4 tension about where R may live is flagged in the journal rather than resolved silently.	2026-08-06 11:19:19 +02:00
admin	415371862d	REUSE + STATUS: map the new scripts, correct a second stale "not wired" claim (R-229) ... REUSE.md line 62 claimed hub_confirm_gate.py is "not wired to run (R-29)" -- it is gate 3 of repo_gates.py, which both the pre-push hook and CI run. Second instance of the same stale-claim class found today, in a third file. Added rows for repo_gates.py (the runner), instructions_gate.py, install_workspace.py and rules_report.py, plus the extension point: a new .claude/rules/*.md is NOT live in the session that created it -- verify from a fresh session, never from the frontmatter. STATUS.md: one operator-facing entry -- the assistant's notes had no backup; they do now, with the same-disk and no-off-site caveats stated rather than implied.	2026-08-06 11:05:33 +02:00