58 Commits

Author	SHA1	Message	Date
admin	4a4a1e245a	R-265 CI timeout + golden 0.210.0 baked; R-221/R-259/R-258 closed, R-266 minted, G-3 unblocked ... Four defects of one family, all shipped today: something the box already knows, thrown away or drawn as its opposite. Agent v0.128.0, controller v0.210.0. NO HUB CODE, no hub bump, no ArgoCD sync. R-265 (this repo). timeout-minutes: 5 on the gates job — every honest run in the observed session finished in 18-34s, so this is ~9x the slowest and far under whatever reaped run 264 at 834s with no log. The alarm mail now carries Elapsed (start stamp via $GITHUB_ENV; an absent stamp prints "unknown (no start stamp)", never a bogus 1.7-billion-second figure) and its "names itself in the run log" sentence is qualified so it cannot mislead when there is no log. ⚠ THE UNKNOWN IS NOT CLOSED. Whether the if: failure() alarm fires for a REAPED job is still unverified. The timeout makes the reap unreachable in practice; it does not answer what happens in one. Demonstrating it means deliberately hanging a run on main, which would leave the branch red for a parallel session. Said in the workflow comment, the changelog, R-265 and the report — none of them claiming it is answered. GOLDEN 0.210.0 baked, published, round-trip verified, NOT VOUCHED. The currency gate went red the moment the controller was bumped — correct — and is closed by the bake, never --no-verify. No --no-verify anywhere this session. ⚠ THE AGENT WAS NOT PUBLISHED UNTIL THIS SESSION CHECKED, AND IT MATTERED. R-221's fix is in the AGENT, and a fresh install takes its agent from the Day-0 manifest. The binary had been hand-deployed to felhom-pve and never published, so agent_version 0.128.0 was not selectable and a fresh install would have received 0.127.0 — the golden would have carried the controller fixes and NOT the one the headline defect needed. Caught by checking each Day-0 value was FETCHABLE rather than assuming. Published from the live-deployed bytes, sha-verified across the hop first. Registers. R-221, R-259, R-258, R-265 CLOSED. R-266 MINTED (READY): the failed root statfs still travels to the hub as a 0-of-0 disk; ranked LOW because it is the quiet direction — it can only miss a true alarm, never raise a false one — and it is now a two-repo wire change governed by G-1's gate. Highest ID moved R-265 -> R-266. CONTEXT S-39 rules the convention this project was missing: "we do not know" is never drawn as "fine", and the codebase has ONE way of saying it — an explicit ...Known bool companion checked in the template. ROADMAP G-3 was explicitly blocked on that decision and is unblocked; what remains there is a survey-and-convert of existing sites, not the gate. Capability map row 93 CHECKED and it was NOT claiming something untrue — it is about the operator notification path. But its narrative ("the page you open to ask whether ONE app is backed up") invites the wrong reading, and the adjacent thing WAS false until v0.210.0, so the row now records that the two halves disagreed and only the operator half was true. Six red-proofs across the two code repos, each with the mutation asserted applied. The one that matters: Part 1 Scenario A FAILED against today's tree, with the intended message. Part 1's operator-present live validation is OWED and is the session's STOP. repo_gates --fast: all 8 OK.	2026-08-08 16:52:39 +02:00
admin	dd55a3f98c	correct the vouch state: the operator vouched 0.208.0 DURING this session ... Recorded on arrival as 0.207.0, re-read from live hub_settings at the end and it is 0.208.0 — the operator acted while the session ran. The ask is therefore 0.208.0 -> 0.209.0, not 0.207.0 -> 0.209.0, and STATUS.md plus the golden evidence now say so. Caught only because the state was re-read rather than carried forward from the arrival note. A fact recorded at the start of a long session is a fact about the start of the session.	2026-08-08 09:25:25 +02:00
admin	2ce3c2a0f2	golden 0.209.0 BAKED, PUBLISHED, ROUND-TRIP VERIFIED — the currency gate goes green (R-242) ... The G-1 session released controller v0.209.0 (R-247), which made golden_currency_gate.py correctly red and REFUSED THE PUSH: no golden carried the newest release. The honest answer to that is the bake it asks for, not --no-verify. The gate's own docstring says the cost of a trip is one bake, which is the operation this project wants to be routine. 656 697 956 B, sha256 c9c4bcd6..e818ff. Round-tripped: the published bytes downloaded back, hashed independently, size and sha identical, and ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.209.0 — the delivered artifact naming the controller it will start. Acceptance markers all green (overlay2 x1, mount points x2 rootfs+mp0, upload HTTP 201 x1, excluding/FATAL/mp1 x0), Result=success, ExecMainStatus=0. 404 pre-gate with a 200 control on 0.208.0 so a 404 could not mean "wrong URL". Token file->file into a 0600 file read inside the VM; systemctl show grep = 0; committed-log grep = 0 WITH a control returning 1 to prove the grep works. Bake VM destroyed, /root residue clean, qemu confirmed gone, drill disk restored to virgin. IT ALSO CONSOLIDATES THE OPERATOR'S APPROVAL. Golden 0.208.0 was baked last night and never vouched; 0.209.0 contains everything it did plus R-247, so it supersedes rather than wastes it. One Save, not two — STATUS.md updated accordingly and back to its 93-line screen. NOT VOUCHED. Fresh installs still land on 0.207.0 until the operator saves. And R-242's untouched half showed itself again: this gate flipped green on the presence of the evidence DIRECTORY, with no vouch anywhere near it. Recorded, not built — ROADMAP G-8. repo_gates --fast: all 8 OK, including wire-contract and golden-currency.	2026-08-08 09:01:26 +02:00
admin	b080ecf411	hub v0.99.0 — the hub can see whether the operator can get in (R-260); G-1 gate closes, R-247 closes ... oobDegraded tested five things and the sixth never arrived. The agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the SAME version that introduced the `oob` stanza carrying it — and store.HostOOBRow mirrored five of the agent's eight OOB fields. With no field for it, encoding/json discarded the fact on arrival, so a box with felhom-sshd active, reachable, a valid config and a configured peer reported `ok` with NO OPERATOR KEY INSTALLED AT ALL. Not a wrong answer: an answer to a question nobody was asking. `operator_peer_configured`, which the hub did read, only says the peer IP is in desired-state — that OOB is MEANT to work, not that entry is possible. Now decoded: operator_key_configured, plus wg_handshake_age_s and healed_at. The last two ride the ALERT TEXT and are deliberately NOT in the predicate — widening a check beyond the fact that is now arriving is how a check stops being read. SCENARIO F, decided on a measurement rather than a preference. operator_key_configured decodes as a POINTER: nil = the agent never said, reported distinctly and never as ok. The version gate was rejected because the field and its stanza shipped in the SAME agent version (v0.72.0), so a stanza without the field cannot come from any released agent; the fleet is 0.113.0/0.127.0 and the vouched floor is 0.127.0. Handled explicitly anyway and pinned, because "cannot happen" is a claim this project has been burned by. THE MESSAGE NAMES THE FAULT. oobDegradedReason is the single source for both predicate and text, so the alert can never name a different fault from the one that fired. The old form derived it separately and had a vocabulary of two — unreachable, or config invalid — with no way to say the key is missing. The operator reads this at 07:00. TESTS DRIVE THE DECODE BOUNDARY. Every hub OOB test before this built a HostOOBRow by hand, and a test written that way CANNOT SEE A FIELD THAT NEVER DECODES — which is how this held a green suite for five weeks. The pre-existing fixture oobReport() also omitted the field, so those scenarios ran against a report shape no released agent produces (same family as R-262). Both fixed. Red-proofs, 8 expected outcomes and 0 wrong, each with the mutation asserted applied: dropping the field returns the false ok; an unconditional check alerts a healthy box; unknown-as-ok restores the silent pass. G-1 CLOSED — scripts/wire_contract_gate.py shipped as ranked, built BEFORE the fixes and seen failing on 40 fields (documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md). Two instrument defects the control caught first: a substring false negative (grep -F healed_at matched privsep_healed_at) and treating dr_recipe as wholly opaque when its top-level sections ARE decoded through an allow-list that already cost offsite_restic (R-122). The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo: R-260 said "at least eight DECISION-BEARING facts", never eight tags. The real count is 40. R-260 CLOSED (class gated, sharpest instance fixed). R-247 CLOSED (controller v0.209.0). R-264 MINTED and OPEN — the 21 facts with no consumer, allowlisted with reasons so that gating the class could not be mistaken for deciding them. Still open and named: R-246, R-255..R-259, R-261..R-263, and C7's test-comment half. Capability map checked: it claims OOB access is implemented, never monitored, so no row was untrue; what was untrue sat one layer down and the row now records it. repo_gates --fast: all 8 OK. go build/vet/test green in hub, run separately from this commit.	2026-08-08 08:47:02 +02:00
admin	b7fb2117ae	CAMPAIGN 12 — the class sweep: golden 0.208.0 baked (awaiting vouch), R-256..R-263 filed, gating ranked ... Part 1. Golden 0.208.0 baked on the drill VM, published and ROUND-TRIP VERIFIED — 656 150 362 B, sha256 ba668f59..5ffb82, and ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.208.0. Acceptance markers all green (overlay2 x1, mount points x2 rootfs+mp0, upload HTTP 201 x1, excluding/FATAL/mp1 x0), Result=success. Token file->file, read inside the VM; systemctl show grep = 0; committed-log grep = 0 WITH a control proving the grep works. Bake VM destroyed, drill disk restored to virgin. NOT VOUCHED — the campaign halts there deliberately. golden_currency_gate.py was correctly RED on arrival and is green after the bake. No --no-verify was needed anywhere in this session. Parts 2-4. Seven defect classes swept for siblings by class rather than by feature. Analysis only: no product code, nothing deployed, no machine touched beyond the bake VM. Eight new rows R-256..R-263 (ceiling moved from R-255), grouped by class in OPEN-ITEMS.md. C1 produced no new instance and has no row. The sharpest is R-260: the agent reports operator_key_configured every heartbeat, the hub has no field for it, so the check that answers "can the operator get into this box" returns ok for a box with no operator key installed. Every class states whether its method re-found the known instances, because a method that cannot re-find them has not been shown to work: C1 2/3 (verified by replaying the pre-fix templates), C2 2/2, C3 2/3 + 1 as fixed, C4 fix-pattern re-found, C5 re-found, C6 deadcode 0/2 and bespoke 1/2, C7 weakest and said so. Blind spots stated per class; seven suspicions investigated and DISPROVED, including two of my own methods. Part 4's ranking is in ROADMAP.md as G-1..G-8. Gate C5 (cross-repo tag reachability — cheap, --fast-eligible, would have caught every R-260 instance on the introducing commit). Do NOT gate C6: golang.org/x/tools/cmd/deadcode was measured against a PLANTED probe and is blind to unreachable METHODS on widely-used types, which is exactly the shape both known instances have. R-242's untouched half is recorded, not built: this bake demonstrated it, the currency gate flipping green the moment the evidence DIRECTORY existed, before the round trip finished and with no vouch near it. Correction the campaign owed its own brief: escrow_stale was described as closed; it is R-247 and READY. The live repo is the source. Sampled rather than swept, exactly: C7 60 of 2652 production invariant comments and NONE of the 1440 test comments (that half is owed); C2 19 of 221 refusals; C3/C4 controller only. No finding was reproduced live. STATUS.md is 100 lines against its 93-line one screen.	2026-08-07 23:07:42 +02:00
admin	59527d00f9	R-254 CLOSED both sites (controller v0.208.0); R-255 filed; R-242 red again ... R-254 site one was the same defect and is fixed the same way. Site two was NOT the defect the row described: the pre-deploy hidden input is deliberate (a form must carry what it submits, README §318) and was left alone; the indefensible one was the readonly display input on an ALREADY-DEPLOYED app, where nothing is submitted. The premise that this broke a repo rule does not hold and is recorded rather than dropped: no line in the repo says 'no silent auto-fill'. What exists is CONTEXT.md:2070, about accidental EMPTY-password deployments. §7.3 measured on the fleet: site one's code path has never run (crafty-controller is the only app declaring initial_credentials and is deployed nowhere); site two's exposure is also empty (demo-hp runs three apps, none with a generated secret field). HONEST LIMIT: that is a current-state measurement, and nothing recorded reads — which was part of the fault. No evidence of exposure, and no mechanism that could have produced evidence either way. Rotation not indicated by anything measured. R-255 NEW: the guard covers 4 of 27 pages at runtime, and the cheap all-templates gate is blind to the shape that actually shipped (a secret under a neutral page-data key) — both verified, both stated in the gate's own docstring. Filed rather than declaring a partial guard complete. R-242 red a second time in 24h; --no-verify declared. The cadence is the argument for its other half: nothing gates the vouch.	2026-08-07 21:27:20 +02:00
admin	f651b31a7a	golden 0.207.0 BAKED, PUBLISHED, ROUND-TRIP VERIFIED and VOUCHED — the currency gate goes green ... Closes the delivery gap v0.207.0 opened this session. Until now the gate was correctly red and a machine installed today would have received 0.206.0 — the release written, tested and pushed, and not delivered. Round trip is the evidence, not the build log: the published bytes were downloaded back (656 879 192 B, sha256 20ec9602…22995, both identical to what the bake reported) and ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.207.0 — the delivered artifact naming the controller it will start. Acceptance markers were the ones R-233 re-captured from a real log: docker OK (overlay2…) x1, including mount point rootfs AND mp0 x2 (there is no mp1 since build-golden.sh v3.0.0), upload OK (HTTP 201) x1, excluding 0, FATAL 0. The 404 pre-gate ran WITH a control so a 404 could not mean 'wrong URL': 0.206.0 -> 200, 0.207.0 -> 404. The token never crossed a shell — copied file->file, read by a runner script inside the VM; systemctl show grep for the value returned 0. The token-leak grep on the COMMITTED log returned 0, and that 0 is evidence because a planted copy returned 1 before being shredded. Vouch was a three-field change with all three checked deliberately: MinAgent 0.127.0 read from the golden's controller CHANGELOG header, agent_version already >= it, min_agent not above agent_version (not the R-216 shape). Verified by re-reading the manifest rather than trusting the flash. The R-120 gate did not refuse. Drill VM restored to virgin; qemu confirmed exited with ps -eo comm, not a self-matching pgrep -f. R-242: the bake half is done and the --no-verify bypass declared earlier today is now historical. Its remaining half is UNCHANGED — nothing gates the VOUCH itself, so a baked-but-unvouched golden still passes the currency gate silently.	2026-08-07 20:59:37 +02:00
admin	1fc38761a8	R-249/R-252/R-253 CLOSED (controller v0.207.0); R-254 filed; R-201's state field corrected ... R-201: the STATE FIELD recorded only PASSED + PROVEN-LIVE 2026-08-04 — the data half from the drill — while the body carried yesterday's journey pass. Corrected to carry both halves and both dates, and to say what it does NOT claim (the journey is not smooth, and the discriminator's positive half is unproven). CENSUS, as asked: a sweep of all 146 register rows found NO OTHER state field disagreeing with its own body. Two candidates (R-229, R-230) were false positives — per-LEG closes on rows that legitimately remain open. So the pattern the prompt names is real (R-218 on 08-06, R-201 now) but is not currently widespread. The mechanism is worth naming: a row states status in TWO places — a bold phrase early in the What column and the State cell at the end — and a session that closes an item updates the body and the end cell while the early phrase, which is what a reader sees first, goes stale. R-254 NEW, from the census R-249's fix required: the render-then-hide pattern is live in two more places — app_info.html puts a REAL per-install app password in a hidden span, and deploy.html renders a generated secret into a value= attribute. Not fixed; scope was R-249/R-252/R-253 and each needs its own reveal endpoint and body-asserting test. R-242: the golden-currency gate FAILED as designed — v0.207.0 is released and no golden carries it. This push used --no-verify, declared here and in the report. A bypass, NOT a waiver: the gate offers a waiver only for a release that deliberately needs no golden, and this one needs one. A bake + vouch is owed. STATUS.md: 93 lines.	2026-08-07 18:17:59 +02:00
admin	3f4fb3825f	R-201 CLOSED — the unaided recovery journey passes, both halves, on the fifth walk ... Capability map: the unaided-recovery row turns FAILED -> PROVEN-LIVE, scoped, with what it still does not claim stated in the row itself: shape (c) did not fire positively (with the mint guard holding there is no local key, so the offer comes from shape (a)); and 'unaided' here means possible-without-a-shell, not obvious, because two obstacles are unsignposted. OPEN-ITEMS: R-201 closed with its evidence. Five new rows R-249..R-253 (the retrieval passphrase in page HTML; the host-key scan ladder vs AAAA settle; the listing's per-tag rows; the two unsignposted restore steps). R-243 annotated rather than re-filed: on a REBUILD offsite_delivery_stuck does not skip, so the row's gap is narrower than it reads. STATUS.md: headline changed, and trimmed 97 -> 92 lines rather than extended, per its own header. Teardown recorded as OWED with its before-measurements, the stop-and-age gate, and the positive controls that must survive.	2026-08-07 17:24:27 +02:00
admin	7850469d5b	golden 0.206.0 VOUCHED; demo-hp's stale flag cleared; session report ... VOUCHED with the operator's approval, verified from the stored hub_settings rather than the flash: golden_version 0.205.0 -> 0.206.0, sha c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e. agent_version and min_agent both stayed 0.127.0. wrapper_sha256 was carried through explicitly, because the handler CLEARS it when omitted. THE GATE WAS CONVICTED BEFORE THE BAKE AND IS OK AFTER IT - red to green on the same command, which is its proof that it measures something real. It went green on the BAKE, not the vouch; that limitation is stated in its docstring and stays open on R-242. THE STALE FLAG WAS WRONG AND IS CLEARED, with the operator's approval. One row, identity-matched on host_id and guarded on stale_at IS NOT NULL; changes() returned 1. Verified end to end, not just in the database: the hub serves the hash again, the box recorded it at 11:10:19Z, and it is byte-identical to the key that box is using - so shape (c) compares, matches and correctly stays silent. The false warning is gone, PROVEN WITH A POSITIVE CONTROL rather than an absent line: 0 escrow-confirm lines since the restart while 5 scheduler lines in the same window prove the box was logging. R-246 records the clearance and keeps the column ruling open: stale_at has NO production writer, changes what a customer is told, and can be seen by nobody who would look for it. Either give it an evidential setter or retire it. STATUS.md finished at 87 lines (from 258). Waiting-on-you is now genuinely empty: the base image is approved and live, and R-245 was re-filed as a decision taken with quota as its reopening condition. Session report: REPORT-clear-the-ground-2026-08-08.md - the six spike questions each answered with method and measurement, Q4 said plainly (only a database read), Q6 said loudly (a fresh box CANNOT reach this state, so the next walk cannot meet it), and three observations noticed but not acted on.	2026-08-07 13:13:16 +02:00
admin	ceac5e0deb	STATUS.md rebuilt (258 -> 85); R-245 re-filed as decided; R-246/247/248 filed ... STATUS.md REBUILT FROM THE REGISTER, not trimmed. Its own header says one screen; it had reached 258 lines, having been 83 four days ago. The three named defects, all fixed: 1. the "waiting on you" list asked the operator to decide the RECOVERY SCREEN, built and shipped 2026-08-05, and to approve an orphaned-backup deletion the register records as DONE the same day; 2. a stray line reading only "- **Nothing.**" sat mid-list; 3. the DooPlex infrastructure work was mixed in with the product's. Infrastructure is now under ITS OWN HEADING rather than dropped, and the reason is stated on the page: these are real asks that need the operator, but they concern the machine this is built on, not what a customer receives. Dropping them would lose real work; mixing them is why the page stopped being readable. The 100-line "what shipped recently" log is gone. That is what the per-repo CHANGELOGs and the register are for, and restating it here is what made the page grow back. R-245 RE-FILED as a decision taken, not a question pending. It sat as WAITING-ON-OPERATOR for a day with nothing actually pending - it was settled on 2026-08-07. It keeps the whole reasoning and now carries the condition that would REOPEN it, which the reasoning already named: QUOTA, old set-aside history blocking new backups. A condition, not a calendar. AUDIT OF EVERY WAITING-ON-OPERATOR ROW, parsing the state column exactly rather than grepping for the phrase (which over-matches rows that merely mention it): exactly ONE row carried it - R-245 - and it was a settled decision. So zero rows were genuinely waiting, and the drift was caught while it was still a single row. R-246/R-247/R-248 file the read-only stale-blob spike's findings. R-242 updated: it recurred within a day, and shape (b) is now built - with the vouch half explicitly still open on that row rather than being papered over.	2026-08-07 12:59:52 +02:00
admin	9657334fb7	R-241 FIXED: registers, capability map, STATUS, hub CHANGELOG v0.98.0 ... R-241 closed against controller v0.206.0 + hub v0.98.0, following the spike's ruling rather than the obvious reading. The row records what the fix does AND the two real bugs the tests caught rather than review - a missing t.Enabled (caught by an EXISTING test) and a missing falling-edge sync that reintroduced the very defect the epoch exists to fix. R-243 UPDATED, not closed: the STATE it describes can no longer be entered (the mint guard), and what replaces it is VISIBLE rather than silent - the box declares awaiting_recovery_key and the customer is offered the screen. But the ALARM GAP is untouched, for the same three reasons, so a box whose customer never acts still stops backing up with no operator signal. The remaining work is an operator-side signal for a box held past some age, deliberately not bundled into R-241's fix. R-245 NEW - WAITING-ON-OPERATOR, recorded and NOT built: should an undecided customer be auto-abandoned after 30 days? The operator's proposal is recorded WITH the reasoning against it, so the decision can be revisited properly: a reinstall implies a person, so nobody is absent; a customer who cannot find their code gets in touch, which is why the operator LEVERS were the thing worth building; the cost is the customer's own storage allowance; and the real harm is QUOTA, which is a condition, not a calendar. If it is ever built, build it to trigger on the harm with a dated warning, never on a date alone. The capability map's recovery-journey row STAYS FAIL. These are fixes, not a walk - nothing here walked a customer end to end, and the row goes green only when one completes with no operator intervention AND a byte-identical sentinel. R-214, R-202 and R-240 are still open. STATUS compressed rather than extended, per its own one-screen rule, and the "rebuilding throws away the off-site history" line corrected: the cause is fixed, so leaving it as a live defect would be false. hub CHANGELOG v0.98.0 for the superseded-package purge. Highest register ID moves R-244 -> R-245.	2026-08-07 12:13:09 +02:00
admin	db578cd44d	R-239 CLOSED (golden 0.205.0 vouched); R-241 ruling into the map and STATUS ... R-239: the operator approved the vouch this session. golden_version 0.203.0 -> 0.205.0 (+ derived sha); agent_version and min_agent both stayed 0.127.0, because the new golden's MinAgent is also 0.127.0 - so in the event it was a ONE-field change, not three. wrapper_sha256 was carried through explicitly: the handler reads it from the form and CLEARS it when omitted. Verified from the stored hub_settings (WAL-aware copy), not from the flash. The R-120 gate passed exactly - the newest controller the fleet reports is 0.205.0, so a 0.204.0 golden would have been refused. R-241: the capability map's recovery-journey row and STATUS carry the spike's ruling - a MINTING defect, not a screen-predicate defect. The row stays FAIL: delivery is not a journey, and R-241 is diagnosed, not fixed. R-242 and R-243 surfaced in STATUS in plain language.	2026-08-07 09:27:22 +02:00
admin	2228c0bff6	final walk COMPLETE — data PASS, journey FAIL; R-241 filed ... THE DATA: PASS. All three sentinels byte-identical out of snapshot f5c53b03, including the 12 MB binary and the accented Hungarian filename whose NAME BYTES are identical too. Disk -> restic -> SFTP -> Storage Box -> rebuilt machine -> disk, intact. THE JOURNEY: FAIL, and further from the line than the previous walk. The claim worked first try (302 in 0.164s). Then: / lands on the launcher with no recovery pointer, /recovery 302s away, and the remote page offers to CREATE a new recovery code — which would orphan the history the customer's code protects. There is no field anywhere to enter the code they hold. The operator's documented remedy also refuses, correctly and fail-closed. Recovery needed three guest command lines. R-241 — and the cause is a success this same walk proved six hours earlier. OffsiteRecoveryOffer() shows the screen only when (a) there is NO repository password (pristine rebuild) or (b) one exists but the history will not open under it. Overnight the credential self-heal collected the staged credential and applied the tier, writing a FRESH key at 03:18Z — so (a) is false; and (b) is unreachable because orphan detection needs a run, and runs are blocked by escrow_state=pending. The gap is self-locking. Measured keys: on-disk 9b4a9a9d... vs recovered-from-R 30ef574f... This is R-218's shape one level up: succeeding at the self-heal stopped the box OFFERING the recovery it still needed. Registers: R-201 moved to its outcome; R-241 filed; capability map's recovery row stays FAIL with both halves and the cause named; STATUS rewritten for the operator. Highest ID R-238 -> R-241. The venue is left with the recovered key in place and the self-heal key moved aside, never deleted. Teardown still owed.	2026-08-07 06:45:36 +02:00
admin	feed748325	R-234 root-caused and CLOSED; R-218's state field corrected ... R-234 was filed as "toggling an app on leaves it without a bundle, so the first run skips it". Measured on demo-hp: that state does not survive a run — the off-site run's own pre-dump phase calls captureAllRecoveryUnits for every DEPLOYED stack, through admitApp, before the push, and a unit moved aside was RECREATED. The actual cause was the single-flight: the manual run was dropped because an earlier one was still going, runOffboxBackup returned nil, the handler had already answered "A tavoli mentes elindult", and the card then showed the PREVIOUS run's green verdict. Fixed in controller v0.205.0 and proven live on demo-hp: a second request while one is in flight now says "Mar fut egy tavoli mentes — ez a keres nem inditott ujat. A most lathato eredmeny meg a korabbi futase", as a flash_error. Independently, and a real gap on its own: a run that skipped an app the customer selected is now `incomplete`, not `ok`. Selected+deployed with no unit counts; selected-but-undeployed is named with what to do but does NOT count, because a box left amber by an app somebody removed is a status nobody reads. R-218's state field read REOPENED while the same row's body already recorded the fix shipped in v0.203.0 and proven live. Corrected to CLOSED, keeping the over-claim history — it is why the row is worded as it is. Capability map: the off-site capture row's `incomplete` sentence widened to cover a whole-app skip, and it still does not claim a newly-selected app is protected by the next run — for a deployed app it is, for an undeployed one the card says so. Still open, deliberately: R-213, R-202, R-214, R-235.	2026-08-06 22:03:58 +02:00
admin	0c4411e54b	R-201 re-walk: the data PASSES again, the journey still FAILS — two dead ends, down from four ... Asked Campaign 11 Phase 1's question a second time, on the fixed build, on a NEW appliance (VM 322, customer rewalk). The Campaign 11 venue was untouched. THE DATA: PASS. All three sentinels byte-identical out of the pre-destruction snapshot a7bc23bd in 23s through the customer's own restore flow — including a 12 MB binary and an accented Hungarian filename whose NAME BYTES are identical too (verified as hex, not as rendered text). THE JOURNEY: FAIL, two dead ends against Phase 1's four. 1. R-218's CONSUME half. The hub re-staged the credential at 11:44:57 saying 'the box re-consumes on its next cycle'; a full cycle ran at 11:55:46/54 (with a positive control that it ran) and it did not. A census of the customer-reachable actions found none that fetches it. Only a command line INSIDE THE GUEST moved it — 18s, confirming nothing was wrong with the credential, target or key: only the trigger. R-218's row said SHIPPED and over-claimed; it is corrected to REOPENED for the consume half. 2. R-220. Drives still unenrollable after a rebuild, needing a Proxmox-host unmount; without it no app redeploys and the restore page stays empty. Unaided RTO STILL UNDEFINED. Attended: +45s key placed, +24m12s tier up, +30m13s data verified. The 30m must not be quoted as the customer number. What passed and is new: the recovery screen appeared WITHOUT being sought, answered all three questions with a seal date matching the hub exactly, the emailed reset code worked first try, the unlock was a real 1.528s unseal, and R-225's fix was seen working in the wild (unknown, not a false zero). R-216 part 4 reproduced live: the reinstall downgraded the hand-installed agent 0.126.0 -> 0.125.0. DELIVERY GAP recorded as owed and NOT conflated with the journey: a fresh install landed on controller 0.201.0 / agent 0.125.0 — the vouched versions, neither carrying the fixes — installed by hand. Nothing was vouched. Capability map row STAYS FAIL. Campaign 11 doc gets a dated ADDENDUM, not a rewrite.	2026-08-06 12:18:29 +02:00
admin	a1a542b9a7	RECON: what DooPlex's own backup actually protects (R-232) — read-only, nothing changed ... Five sets, 14/14 successful runs in 14 days, and a file restored byte-identical from the data repo -- the first time a restore has ever been demonstrated here. What the failure matrix exposes: notify_failure is a no-op (NOTIFY_ON_FAILURE=true, webhook commented out) so a failed backup tells nobody; NOTHING leaves the box (Longhorn's target is DooPlex itself over NFS, the only outbound-looking cron pulls inbound for another project); the backup tree is a single writable path with no append-only; two sets are same-disk (.claude-memory and the postgres dumps, whose source sits inside the backup tree); Longhorn retains 1 generation; secrets/restic-repo has never held a snapshot because backup-secrets.sh never calls restic; and /opt/backup/docs/BACKUP-RESTORE.md does not exist though the unit advertises it. NOT a finding: the restic passphrase. On-box copy is on a different disk from the backups and the operator holds an offline copy out of band, so disk loss is recoverable. The residual is only that it is operator-held rather than system-held escrow. A stale lock in the data repo is MINE, from the previous task's verification. Fenced from clearing it; recorded with its impact and the one command that fixes it.	2026-08-06 12:16:38 +02:00
admin	b1bf1f271c	REPORT + STATUS: instruction arc close-out ... Operator line for the one change with real-world consequence: the runbook that says which machine may be wrecked claimed the HP box keeps no off-site copy, and it keeps two. Checked against the machine, not corrected from memory.	2026-08-06 11:50:28 +02:00
admin	415371862d	REUSE + STATUS: map the new scripts, correct a second stale "not wired" claim (R-229) ... REUSE.md line 62 claimed hub_confirm_gate.py is "not wired to run (R-29)" -- it is gate 3 of repo_gates.py, which both the pre-push hook and CI run. Second instance of the same stale-claim class found today, in a third file. Added rows for repo_gates.py (the runner), instructions_gate.py, install_workspace.py and rules_report.py, plus the extension point: a new .claude/rules/*.md is NOT live in the session that created it -- verify from a fresh session, never from the frontmatter. STATUS.md: one operator-facing entry -- the assistant's notes had no backup; they do now, with the same-disk and no-off-site caveats stated rather than implied.	2026-08-06 11:05:33 +02:00
admin	d30c2a51ed	R-224..R-228 CLOSED: registers, capability map, campaign annotation, STATUS ... Five closed in controller v0.202.0 + agent v0.126.0, each with its live or red-proof evidence in the row. Five explicitly still open and named as such rather than left to inference: R-214, R-220, R-221, R-213, R-202 — and R-220 is flagged as currently worked around BY HAND on the campaign venue, which is the only reason an app could be deployed there. The capability map's recovery row STAYS FAIL and says why: fixes are not a re-walk, nothing walked a customer end to end, and the customer-facing messages were NOT re-driven live because /recovery correctly retires itself once the old data is set aside — restoring that state is the reconfiguration the task forbade. The campaign document is ANNOTATED, not rewritten: it records what was true when it ran, and that is its value. workspace-CLAUDE.md gains comment-vs-code entry 9 — the escrow header said the errors were 'DISTINCT on purpose' and named THREE situations while a fourth was folded into one of them, and a green test named the defect and did not prevent it because it asserted a STRING one layer below the merge. ROADMAP needed no collapse — it carries no rows for these IDs.	2026-08-06 08:34:04 +02:00
admin	0f8c3cc848	STATUS: what the overnight stress test found, in plain words ... Leads with what held — nothing lost a byte, the set-aside really is set aside (verified 12.5 MB untouched at the far end), a wrong code refused three times with nothing written and no lockout, the alarm fired and cleared itself. Then the four new findings in operator language: R-224 (our own systems being down is reported to the customer as a bad recovery code — the same lie as yesterday through a different door, and the machine had not even tried the code: three hundredths of a second against a real attempt's one second), R-226 (a mistyping customer can no longer be told to check their typing), R-225 (0 snapshots / 0 GB shown above a paragraph saying the store holds backups), R-228 (the set-aside backups become invisible). States explicitly that nothing was fixed last night, on purpose.	2026-08-06 00:01:51 +02:00
admin	3a539ea530	docs: R-223 CLOSED by a golden rebake — and it was never a dropdown ... The first vouch of agent 0.125.0 was REFUSED by R-120's gate: the artifacts form saves as a unit, and golden 0.192.0 was older than the newest controller the fleet reports. The golden had been stale since before controller 0.193.0, so the Day-0 manifest had been effectively unvouchable for days and nobody had cause to notice. Correcting STATUS.md and OPEN-ITEMS, which said "one dropdown". Golden 0.201.0 baked in the drill VM, round-trip verified from Gitea (e730d7cab343eb35...f007654, 658165766 B), teardown clean, 0 token hits in the saved log. Manifest now agent=0.125.0 golden=0.201.0 min_agent=0.125.0, read back after the save. The held floor: the HOLD is established positively (hub log + the box's own "floor still unknown"). Its RELEASE is not — no HELD line and no dashboard reason are both absences, the served path logs nothing by design, and the box-side positive needs a restart the venue cannot take before Phase 2. Recorded as deduction plus a pinned test, not a live positive.	2026-08-05 22:29:30 +02:00
admin	1a0f7db92f	docs: CAMPAIGN-11 — the journey FAILED, R-198's retention PROVEN, six findings fixed ... Registers and evidence for the campaign and its fix pass. OPEN-ITEMS: R-214..R-223. Six SHIPPED (R-215/216/217/218/219/222); three deliberately still open and each blocks a real flow (R-214 console banner, R-220 drives unenrollable after a rebuild, R-221 a rebuilt box cannot run the escrow ceremony); R-223 minted and WAITING-ON-OPERATOR (vouch agent 0.125.0). R-213 and R-202 untouched. Capability map: a new row for the customer's UNAIDED journey, recorded FAILED and staying failed until a re-walk passes — fixes are not a journey. The existing rebuild row is corrected where it said R-198's retention was unit-proven only: it was proven in production on the first supersession since the fix, identity_blob retained at 572 B byte-length exact. CLAUDE.md comment-vs-code table: eighth entry — ResolveManagedFloor, the first where the false invariant was a GUARD rather than a comment alone. STATUS: the headline is now "the backup promise is proved, the recovery journey is not", and the one thing waiting on the operator.	2026-08-05 18:04:13 +02:00
admin	f45b1f6761	docs: R-193 CLOSED (the recovery screen); R-213 minted for the put-back ... - OPEN-ITEMS: R-193 CLOSED with both 2026-08-05 rulings (unlocking and restoring are separate; 'I do not want the old data' moves the store aside after a double confirmation), and the shape-(b) reasoning — WriteOffboxSecrets auto-generates a repository password on re-apply, so the literal 'fresh data area' trigger would have opened a window that closes by itself. - R-213 MINTED (R-212 was and still is the highest, re-checked for the second writer): putting files back in place, with the live-versus-backup comparison named as its requirement. Not started, deliberately. - capability map: the 'needs someone who knows to look' qualifier is GONE; what remains is stated narrowly — no correct-code run through the page, the put-back is out of scope, and the journey has not been re-walked end to end. - 07-backup-architecture 7.0: a fifth row, and where the screen deliberately stops. - CONTEXT: standing ruling S-34. - STATUS: the headline change and the two things still owed as proof. No hub change and no hub bump.	2026-08-05 12:56:55 +02:00
admin	63e0ac01f2	R-212 CLOSED: the three orphaned stores deleted after a corrected list (~1.45 GB) ... The register said 'two set-aside stores, ~1.2 GB'. Measured before touching anything: THREE set-aside stores totalling ~1.45 GB, and the thing that was exactly 1.2 GB was demo-felhom's LIVE felhom-repo. Matching on the size would have deleted a working repository. The operator was shown the corrected list and confirmed 'delete all three'. Deleted: demo-felhom orphaned-20260717 (1.4 G) + orphaned-20260718 (3.0 M); demo-hp orphaned-20260804 (43 M). Both LIVE repos untouched, confirmed by full listings before and after on each account. Proof nothing live was caught: a real off-site run on demo-hp immediately afterwards returned status ok, orphaned false, no error, 6 snapshots. Method note recorded for the next session: the storage box has a RESTRICTED shell. 'test -d X && rm -rf -- X' returns 'Command not found' and does nothing (it failed CLOSED, verified by an unchanged listing); 'rm -r <path>' as one simple command is the working form.	2026-08-05 11:15:24 +02:00
admin	4faebe2926	docs: R-204 ALL FOUR items closed; R-193 credential half; R-192 by replacement; R-212 filed ... - OPEN-ITEMS: R-204 all four CLOSED with both 2026-08-05 rulings recorded (the declared-state trigger and its four-meanings-of-absence reasoning; the recovery preview's dashboard-password exposure accepted as metadata, not content). R-193's credential half CLOSED, screen + deletion still open. R-192 CLOSED by REPLACEMENT. R-202 untouched. - R-212 MINTED (R-211 was the highest, grepped): the orphaned-ciphertext deletion HALTED at its STOP because the measured paths do not match the register — three set-aside stores totalling ~1.45 GB, and the thing that is exactly 1.2 GB is demo-felhom's LIVE repo. Nothing was deleted. - capability map: all four interventions closed; the row KEEPS a qualifier for a new reason — no step needs an operator, but there is no customer-facing recovery screen, and the journey has not been re-walked end to end. - 07-backup-architecture 7.0: the four-step table updated; the declaration-vs- inference reasoning and the credential-automatic/key-customer-present split. - CONTEXT: standing ruling S-33. - STATUS: the headline change and the deletion STOP. - REPORT-r204-item4.md rather than REPORT.md: a parallel session is active in this shared clone.	2026-08-05 11:06:41 +02:00
admin	c917251eeb	docs: P6 EXECUTED — containerd store moved to SSD2; reboot deferred (R-209, R-209a) ... Operator ruled "proceed" after reading the pre-analysis; the storageReserved condition CC made the move conditional on was applied with it. The pre-analysis is retained verbatim as §11-bis so the ruling can be audited against what it was given. MOVED WITH ZERO LOSS, verified BEFORE the original was touched on four independent observables: 550,891 = 550,891 filesystem entries; 448 = 448 trusted.overlay xattrs (load-bearing — overlayfs stacking rides them, which is why rsync -X is not optional); 37,243 = 37,243 hardlinks; byte-identical meta.db sha256. Verified again after: identical image/tag/volume ID SETS, cache 2.782 GB / 38 records, ceiling still 30 GiB, pg 4 DBs / 31 tables / 175,135,767 B, redis DBSIZE 2437. End-to-end proof was a real build on the relocated store, rc=0 — not an inventory count. k3s was never at risk, and that was established BEFORE stopping anything: it runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and ~160 pods stayed up. Only the two jarr-* dev containers were affected. TRAP FOUND WHILE PROVING THE GUARD, and it is the reusable part: RequiresMountsFor on a path with NO mount unit is a SILENT NO-OP — containerd started normally against an absent-but-unmounted path. A typo'd guard buys nothing and says nothing: the built-but-never-wired shape again. The guard was therefore verified positively at the unit level (Requires= AND After=mnt-ssd_2.mount on both units), and refusal proven with a genuinely absent DEVICE via a temporary synthetic .mount unit — /mnt/ssd_2 hosts 12 live Longhorn replicas and must never be unmounted, and editing fstab on a production host risks emergency mode at boot. Result: "Job containerd.service/start failed with result 'dependency'", is-active inactive. It refused rather than creating an empty store on the root disk. storageReserved on SSD2 0 -> 80 GB; still Schedulable=True at 76.34%. NOT PROVEN, AND NOT GLOSSED (R-209a): the operator ruled DO NOT REBOOT, so the move has not survived a reboot — which is this task's own standard for validated. The MECHANISM is proven; the CONSEQUENCE (that a real boot mounts /mnt/ssd_2 before containerd starts, in this host's actual ordering) is not, and mount-ordering reasoning is exactly the class this project has been burned by. So: the pre-move tree stays MOVED ASIDE, not deleted (34.3 GB, which is why / sits at 54% and not lower), and felhom-store-postboot-check.service — oneshot, enabled, dry-run PASS at install — validates it automatically at the next boot, asserting positively that /var/lib/containerd does NOT exist. Read /var/log/felhom-store-postboot-check.log after the next boot; on PASS the rollback copy can go and ~34 GB returns. Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).	2026-08-05 10:27:40 +02:00
admin	ee9d9bf203	docs: SPIKE — DooPlex build-cache containment; R-205..R-211 (2026-08-05) ... Spike output only; no production Go code. The one shipped change rides in homelab-manifests 6808a4b (R-205, the monitoring rule). VERDICT: mechanism confirmed, with one correction and one refutation. - CONFIRMED: builder.gc IS honoured under the containerd worker and DOES evict. Proven by naming a 440 MB `go mod download` record present at build N and absent by N+2 — not by absence of an error. - CORRECTED: honoured ONLY in the `policy` array form. The flat form is SILENTLY ignored — daemon starts, logs nothing, keeps its defaults. `dockerd --validate` returned "configuration OK" for a bogus key AND for a config that then crashed the daemon. The oracle is `docker buildx inspect`. - REFUTED: Docker's `data-root` would NOT move the cache — it moves 0.62 GB. The 181.4 GB belongs to the system containerd (`root` in /etc/containerd/config.toml). P3 (operator-approved) executed: prune claimed 156.9 GB, the filesystem returned 150.35 GB (the 6.5 GB gap is layers shared with images), SYNCHRONOUSLY — / went 86% -> 53% used, and Longhorn's default disk went Schedulable=False (DiskPressure) -> Schedulable=True (18.85% -> 50.32%). P7 root-caused the largest item and it is NOT the cap: all 208 `go mod download` records had Usage count 1. Isolated by controlled builds — same VERSION build-arg -> CACHED, new VERSION -> executed, byte-identical tree. `ARG VERSION`/`ARG GIT_COMMIT` sit ABOVE the module-download step, and a RUN's cache key includes the stage environment. Both Dockerfiles have it. One line each to fix -> R-208. P6 NOT EXECUTED — stops at the operator, as specified. Pre-analysis: the move is safe as measured (+38.8 pp above the 25% floor) but SSD2 is the only Longhorn disk with storageReserved=0 and is overcommitted 6.9x; at full inflation the move lands 12 pp BELOW the floor. The prune removed the move's urgency, so CC recommends against it unless ~80 GB is reserved on SSD2 -> R-209. Register: R-205 (CLOSED, shipped), R-206 (Ansible: cap + prune + narrowed Docker ban), R-207 (DRY_RUN guard), R-208 (ARG ordering), R-209/R-210 (operator), R-211 (Prometheus has no config-reloader — rules changes have never applied until something restarted the pod; found while verifying R-205). Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).	2026-08-05 10:02:46 +02:00
admin	0dbd954fec	docs: R-196 closed, R-204 items 1-3 closed, item 4 open (R-193) ... - OPEN-ITEMS: R-196 CLOSED; R-204 items 1-3 CLOSED with item 4 named and its dependency stated. Header restates that R-202, the 1.2 GB ciphertext deletion and R-198's still-unit-proven retention all REMAIN OPEN. - capability map: the recovery row keeps its 'with a person present' qualifier, names which crutch remains, and cites the three now gone. - 07-backup-architecture: new 7.0 - what a customer can and cannot do ALONE, the four steps in a table with status. This is the section a future reader will use to answer that question. - CONTEXT: standing ruling S-32, superseding S-31 steps 2-5. - STATUS: rewritten to one screen per its own header; removes a corrupted half-overwritten section left from the drill session. - ROADMAP: R-196 and R-204 collapsed.	2026-08-05 07:37:35 +02:00
admin	2a7ac03c47	R-201 PASSED: a customer's file survived a machine rebuild and came back byte-identical	2026-08-04 23:18:47 +02:00
admin	b228fd102d	R-201 night run: the off-site key IS recoverable after a real rebuild (proven); the verdict is blocked by R-204	2026-08-04 22:33:01 +02:00
admin	73fb595e38	R-203 shipped: the app and its backup agree, and 'ok' means it — R-201 unblocked	2026-08-04 18:52:14 +02:00
admin	a0c4b607a6	drill R-201: prepared and HALTED before the wipe — a mandatory data directory was missing from the off-site snapshot while the run said ok (R-203)	2026-08-04 15:00:06 +02:00
admin	846253baa8	docs: the offsite key is proven recoverable (R-199 closed, R-200 half, R-201 scoped)	2026-08-04 13:55:25 +02:00
admin	91cabdde1b	hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196) ... R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site data key, which is the secret the retention was built to preserve. And because the copy happens as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups stay recoverable. Both demo boxes crossed that line on 2026-08-04. - identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion. - ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob. - CountCurrentEscrowWithIdentity is the census of who the fix protects. - Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone. - Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why the pre-existing retention test stayed green for two months asserting the mechanism. R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity blob retained, a changed key is "this history now depends on an older recovery code", not a loss. R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and regressed, each stating its own measurement; the regressed text withdraws the Re-issue recommendation. Every self-heal refusal leaves a notification_log row with its reason. The guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in the alert text so the limitation travels with the number. offsite_delivery_stuck and offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither has a customerMessages entry, which is not a block. R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the restic repo password. It resets the PROVIDER password and cannot touch the repo password, which is generated on the box. All five corrected; the staleness mark documented as precautionary. The BEHAVIOUR stays open. Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).	2026-08-04 12:56:58 +02:00
admin	3f2b7bc023	RECON: trace the offsite DR chain link by link — it does not join up (R-198..R-201) ... Read-only recon of the escrow -> recovery chain, from a dead node to an open repository. No production code, no build, no version bump. Headline: the hub's superseded-escrow retention does NOT retain the offsite repository password. host_escrow_superseded has no identity_blob column and demoteCurrentEscrowTx copies only the K-escrow blob, so what survives a supersession is the PBS datastore key, not the restic repo password. The next escrow ceremony -- which the system tells a rebuilt box's customer to run -- destroys the last copy. Both demo boxes crossed that line on 2026-08-04. Also established: - the hub's blob-serving endpoints (re-enroll / restore-directive) have zero callers anywhere: agent, hub UI, scripts, runbooks (R-199) - POST /backup/offbox/inject-password is routed and handled but no template contains the form (R-200) - nothing in the recovery path has ever been exercised; the one live round-trip proof (2026-06-10) predates the ResticRepoPassword field (R-201) - a fail-closed mint refusal IS implementable: the report ACK already carries escrow{identity_blob_present, restic_pw_sha256} and the controller discards it whenever no offbox target exists Corrections: yesterday's spike annotated (candidate (b) overturned in part -- unattended recovery is impossible, customer-present is not); capability-map retention claim struck through and replaced with what the code does. Deliverable: documentation/audits/RECON-offsite-dr-chain-2026-08-04.md Register: new R-198..R-201; R-193 and R-192 updated; STATUS.md refreshed.	2026-08-04 12:16:04 +02:00
admin	b7c5cfc061	docs: R-193 spike report, operator STATUS, and the ROADMAP R-1 scope correction (F3 covers HOST re-enroll, not a guest rebuild)	2026-08-04 11:12:08 +02:00
admin	a684a81796	docs: R-191 closed, R-190 mitigation shipped (mechanism open), R-194 filed ... R-191: the gate was run first and it mattered — ep0's prune jobs have run daily since 2026-07-27 (18 tasks, all OK). THREE of my own queries said the opposite and all three were broken instruments (worker-type vs worker_type, prune vs prunejob, proxmox-backup vs proxmox-backup-proxy); acting on any would have disabled the only pruning attempt while reporting that nothing prunes. R-190: the row now reflects the MITIGATION, not the cause, and says so. Includes that v0.124.0's transition record failed in production while every unit test passed, and Part 3's single pass — the token-rotation class is real and documented in our own installer but does not fit this box (same token still authenticates, the other three grants survived, no installer run evidenced); ruled out on this box, not fleet-wide. R-194: PVE caches permissions (~40s and ~16min measured), so a single permission read is a lagging indicator and may explain when R-190's refusal surfaced. 07-backup-architecture: the offsite retention row was RIGHT and the configuration disagreed with it — noted, since a doc that states a contract does not enforce it.	2026-08-04 10:04:25 +02:00
admin	33f0ab2f33	R-193: demo-hp offsite restored on the box; escrow correction recorded ... Re-issue through the designed endpoint: fresh password staged 07:11:51, settle-gate GO, consumed 15s later at 07:12:06, offsite configured at 07:12:09 against the SAME sub-account (275124) and repo path. Both boxes now escrowed. CORRECTION to this session's own reading: the escrow did NOT recover by itself. escrow_state went pending -> escrowed 15s after the apply and I inferred automation; the operator had run the ceremony. It needed a human on both boxes — demo-hp at 07:16:02, demo-felhom at 07:20:28 (its escrow had been pending since its own rebuild the day before). A 15-second state change is not evidence of automation, which is the same class as reading an absent log line as success. Snapshot continuity is NOT established: both boxes report snapshot_count 0, but the run-history keys are ABSENT rather than zeroed — a controller that has never run an offbox backup, not an empty repo. The 04:15 run decides it.	2026-08-04 09:23:08 +02:00
admin	2584dfb938	R-193: a guest rebuild silently drops the offsite tier; R-192 cause established ... Operator confirms no hub-side offsite config change, so the regression was not an action. Evidence: demo-hp's controller went 0.187.0 -> 0.192.0 at 06:12:18 with a new config hash and the agent re-keyed its leaf three minutes earlier — a guest rebuild. The last pre-rebuild report shows the tier fully healthy: escrowed, last success 02:16:39Z, 15 snapshots, 40.9 MB. No offsite object in the 108 reports since. Mechanism: the restic credential is delivered once. demo-hp consumed its secret on 2026-07-23; the rebuilt controller has no copy and no way to request another. demo-felhom survived the SAME rebuild only because its secret was still unconsumed — it consumed it four seconds after its config hash changed and was reporting offsite again 76 seconds later. That difference was luck, not design. Also sharpens R-192: the self-heal's guard refuses when any report since the consume carried an offbox target, but that query reads the OLDEST 500 reports — all of which predate the rebuild. Healthy history before a rebuild is not evidence the credential still works, which is why the automation that exists for this case declined to act.	2026-08-04 09:05:53 +02:00
admin	185661062a	STATUS: back to one screen	2026-08-04 08:58:04 +02:00
admin	967ddf2f44	R-192: offsite_delivery_stuck states the opposite of what it measured ... demo-hp has been emailing daily since 2026-08-03 06:12 UTC that '500 reports since carry no offbox target — the credential is likely burned'. Measured against the hub's own data: all 500 of those reports DO carry one. The credential was consumed 2026-07-23 and applied successfully; the controller reported an offsite object continuously until 2026-08-03 05:59:21, then it vanished at 06:12:19 and has been absent for 108 reports. A regressed apply, not a burn. Two defects. maybeEmitStuck builds its message from ReportsSinceConsume while hardcoding 'carry no offbox target' and never consults OffsiteReportsSinceConsume, so the recommended action targets a failure that did not occur. And maybeHeal refuses SILENTLY on exactly that field ('regressed-apply shape -> operator's call', a bare return), so there is a daily email with the wrong story, no heal, and nothing saying why the heal declined. The underlying condition is real: that box currently has no off-site copy of its customer app data, since 08:12 CEST on 2026-08-03. What removed it is not established.	2026-08-04 08:57:53 +02:00
admin	d692c46db0	RUNBOOK: the host tier is restore-PROVEN unattended on both demo boxes ... Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite 540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest, booted, verified and destroyed itself; zero 990000 guests or volumes afterwards and both local-lvm figures returned to their pre-run values. Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for the first time: never-proven sorted first, each box took its HOST tier, deferred the offsite one, and picked it up on the next evaluation six hours later. The host-tier proofs reached the hub through R-189's merge — demo-felhom's report carries two tiers, and the local one can only have come from disk. The capability map's optimistic half is cashed, with its scope stated: these two boxes, not the fleet. Surfaced and filed rather than fixed: - R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with a reinstall, any logged pveum activity and any cluster-log entry ruled out. - R-191: every weekly offsite backup uploads successfully and then fails the job on a prune the box is deliberately not allowed to do (R-89 moved it server-side; both boxes still arm keep_last=2). Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the write path (six, with the hub raising whole_guest_backup_failed at the first), and my earlier "no restore_test_* events" was produced by grepping a 404 page.	2026-08-04 08:13:43 +02:00
admin	e3187c86d5	docs: R-185 closed — the silence as well as the grant ... - OPEN-ITEMS: R-185 closed with the measurement, the corrected root cause (the installer's Scenario-F reuse arm, not PVE_STORAGES), and the live sequence. Records that demo-hp carried the same drift and was fixed too. - capability map: the whole-guest row's HOST-tier half was OPTIMISTIC and now says so — that tier was not merely unproven, it was unprovable on both demo boxes, and every live proof cited was on the offsite tier. - vzdump-target-move runbook: its item 5 predicted this; annotated (not rewritten) with what actually happened — the create arm did grant, the reuse arm did not, and it surfaced as a silent unreadable tier rather than the 403 the item expected, because vzdump writes through a root path. - CONTEXT: S-21 (an empty listing cannot distinguish forbidden from newborn; the measured trap that an ungranted path answers with INHERITED privileges) and S-22 (the Scenario-F arm must finish the job). - STATUS: rewritten for the operator, back to one screen.	2026-08-03 19:02:25 +02:00
admin	81d4438630	docs: R-189, R-188 and R-186 closed (agent v0.122.0); R-185 stays open ... - OPEN-ITEMS: all three rows closed with their measurements. R-189 records the migration consequence seen live (a pre-R-189 record has no tier and is not reportable, so upgrading does not retroactively make an old proof visible). R-188 records both directions demonstrated and the fixture cleaned up. R-186 carries the before/after sha values including the CGO_ENABLED discrepancy. The priority list states explicitly that R-185 is the one left open from that group and that this session did not touch it. - capability map: the restore-proof row now says the evidence path itself had a gap — the proof reached the hub only because no restart intervened — and what closed it. - CONTEXT: S-19 (a proof is durable and reportable, a failure is neither, and why) and S-20 (the release ordering, what each step protects, and how a binary is verified). - STATUS: rewritten for the operator, back to one screen.	2026-08-03 16:46:49 +02:00
admin	c102832892	docs: R-86 closed and proven live; ep0 recorded as protected; R-185/186/187 filed ... - OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal reading is never true on a daily tier); R-87 re-ranked UP because R-86 built most of what it waited for; R-185 (the agent cannot list demo-felhom's host backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's sha is not reproducible from its tag), R-187 (R-115's publish leg had never actually run) filed. R-184 was the highest ID in use. - ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row itself proposed — which would have been the never-fires version. - 07-backup-architecture: new contract section — restore-testing is per ARCHIVE GENERATION, with the trap and what did not change (S-1). - 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on the 635 s due-triggered offsite run, with the restart and teardown evidence. - CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines). Numbered 17/18 because S-14 and S-15 were already duplicated in the file. - STATUS: rewritten for the operator, trimmed back to one screen.	2026-08-03 15:28:32 +02:00
admin	e34b614e5b	docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected ... R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The hub's notification_log for the run reads: two per-app failures RECORDED, one digest SENT naming both, and the customer channel SKIPPED with operator_only. Against the measured previous behaviour — two failures, one email naming one app, one leaving no trace anywhere. Scenario D proved itself on an event I had not planned: disk_critical alarmed on two filesystems, the second was collapsed by the cooldown, and that collapse is now visible WITH ITS KEY. Yesterday it would have left nothing at all. A gap the spec did not anticipate is recorded with its fix: the per-app event also fires from the periodic sweep, outside any run, so making it record-only would have created a NEW silence. The sweep emits a digest too, with no run_id, so it stays under the ordinary hourly cooldown. ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile SURVIVED the resize and is active (checked, because a resize is a stop/start). The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere. Five documents corrected — three of which the task's list did not name, found by searching. Two audit/evidence documents ANNOTATED, body untouched: they record what was true when written and that is their value. R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable, not unbounded — the original OOM was a 14.46 GB restore — so the restore-test cadence should still be paced, just not by fear of the endpoint. target-selection.md's "D-d did not name ep0 either way" is deliberately left standing. It is the operator's question, not CC's. STATUS.md 127 -> 83 lines, items rather than sentences.	2026-08-03 14:00:54 +02:00
admin	b0b269b28d	docs: R-110 + R-115 closed, R-182 re-scoped by measurement, R-183/R-184 filed ... R-182's direction REVERSED by Part 0's measurement. Filed yesterday as "the reserve re-alerts on every status refresh" — too many alerts, seen at the sending end. Measured at the receiving end: 9 events received today, 2 operator emails sent. When two apps are refused in the same second the operator is told about ONE; the other is dropped before LogNotification, so it leaves no row on any channel and cannot be audited. The operator cooldown key is customerID:eventType(+tier) and the capture-failed event carries `app` but no `tier`, so the key has no app identifier. Same failure mode as R-97a, in a second event type that never opted into the narrow fix. Nothing changed — Part 0 was investigation only. Correction owed: yesterday's report said "one recovery_unit_capture_failed per app, HTTP 200". True of what the CONTROLLER pushed; a reader would take it as "the operator was told about each app", which is false. R-110 CLOSED (installer v1.23.0). Both channels moved. The spec's mechanism for channel 2 rested on a factual error — the run-time fetches are sixteen, not nine, and come from felhom-agent, not this repo — so no tag here could cover them; pinned to the agent version being installed instead, on the operator's ruling. Channel 3 needed no change: the URL never carried a ref, so no hub change and no hub bump. R-115 CLOSED. release-agent.sh builds, tags, publishes and verifies by an independent download; check-published-versions.py refuses a tag with no package; CI now runs the full gate set so it actually runs. R-183 NEW+CLOSED: a fresh install fetched the vouched agent binary and its sixteen config files from two different refs, and nothing compared them. R-184 NEW: nothing stops the hub vouching a version that was never released. The R-115 gate cannot see it — measured, the hub manifest and Gitea's package listing are both 401 anonymously. capability map: new PROVEN-LIVE row for the published installer channel. STATUS.md 138 -> 127 lines.	2026-08-03 12:44:08 +02:00
admin	fb652024ea	docs: R-181 closed, R-156 closed, R-110 + R-115 rulings recorded, R-182 filed ... R-181 CLOSED (controller v0.193.0 + v0.193.1) and proven live on demo-hp for BOTH reserve terms. The reserve is now a per-app, per-run ADMISSION decision taken before the app's first write and covering all three write legs, and it gained a size term. The refusal's wording was not weakened; the behaviour moved so it became true, verified by sha256 tree fingerprint. R-156 CLOSED — papra's template mounts the app's own data root. Precondition re-measured rather than inherited (both boxes were wiped today). Part 4, documentation only, nothing built: - R-110 WAITING-ON-OPERATOR -> READY. Ruling: option (b), the installer's publish channel moves to a TAG. Recorded with the condition that decides whether it works at all — it must cover BOTH the /scripts/ git-sync and the nine files the installer fetches from raw/branch/main. - R-115 WAITING-ON-OPERATOR -> READY. Ruling: mechanism (b), a build-side gate refusing to deploy or vouch an unpublished version. The third instance (agent v0.120.0) would have silently downgraded both demo boxes while succeeding. R-182 NEW: the periodic status refresh has no admission scope, so a refused app re-alerts on every poll (measured: a second alert pair 13s after the run's). Pre-existing in v0.192.0; deliberately not fixed in the R-181 task. capability map: the local-backup row moves to PROVEN-LIVE in BOTH halves. ROADMAP: R-165 collapses to CLOSED; R-181 collapsed into it. 07-backup-architecture.md: the reserve's contract stated as what the code provides (S-1 — an architectural contract changed in the same session). STATUS.md trimmed 150 -> 111 lines, "What's broken" no longer holds shipped work, and the stale "After:" line (pointing at work that shipped on 2 August) is fixed.	2026-08-03 11:36:16 +02:00
admin	aa62449694	R-178 CLOSED: both demo boxes reinstalled from the merged golden and proven ... Two boxes, two DIFFERENT supply paths, so the session proved the disk shape and the delivery route rather than one of them twice. demo-hp (layout proof, --golden <local volid>): mp0 at /var/lib/felhom, backup=1, 70G, no mp1; /var/lib/docker and /mnt/sys_drive both real mounts of its subdirectories via fstab; one df figure and one device id (64519) on all three paths; reboots 3/3 with the binds surviving each. demo-felhom (pipeline proof, --force-gitea-golden): fetch_verify succeeding against the vouched manifest for BOTH artifacts -- 'verified sha256 54e2a4c431daf580... matches the hub manifest' for the golden, a7763d31... for the agent. 250G single volume, grep -c '^mp1:' = 0, reboots 3/3. Journey proven on both, endpoint-level: claim -> deploy -> back up -> restore, with a planted marker returning byte-identical on each box. Ceiling measured gone: 65 GiB and 233 GiB available to a recovery unit, against 19 and 45. R-165 -> IMPLEMENTED, not PROVEN-LIVE, on the operator's ruling. B2, which that row records as the bulkhead's replacement, fired live for the first time and does refuse per app, delete nothing and alert -- but it is checked only in captureAllRecoveryUnits while runVolumeDumps writes the bulk unguarded, and its 'the previous unit is untouched' claim was measured false (182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). -> R-181. New: R-179 (uninstall leaves NAS network-storage units), R-180 (--archive-storage not cross-checked against the ACL grant; 403 at step 8/8 after root@pam is rotated), R-181. Third instance of R-115 recorded (agent 0.120.0 unpublished). No code written, no version bumps -- this was a runbook.	2026-08-03 09:34:15 +02:00