b080ecf411b236417678825a3493d11266523074
73 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 admin
|
560f0d4451 |
G-1: a gate for the dropped field — built first, and seen failing on 40
Campaign 12 ranked this first of eight gating candidates. It is built BEFORE the fixes it finds, because last night an off-the-shelf tool for a neighbouring class (deadcode, for C6) was made to prove itself first and found NEITHER of the two defects it was meant for. A gate nobody has watched fail has not been shown to work. scripts/wire_contract_gate.py, registered in repo_gates.py as --fast (no network, no container, so it runs in BOTH the pre-push hook and CI — the R-29 constraint). THE TEST. For every json tag reachable from a declared wire ROOT, does that literal tag occur anywhere in the receiving repo's production Go or templates? A tag occurring nowhere cannot be decoded by any struct, named OR anonymous. That last clause is why a string test is used instead of comparing struct to struct: Campaign 12's first attempt paired types by shape and false-positived badly, because the hub decodes one report through several ad-hoc anonymous structs. RESULT ON TODAY'S TREE: 210 tags checked across 3 declared wires, 51 skipped (generic / opaque / allowlisted), 40 CONVICTED. Captured verbatim in documentation/tests/wire-contract-gate-2026-08-08/ BEFORE.md, which is deliverable 1 of this session. The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo rather than quoted: R-260's wording was "at least eight DECISION-BEARING facts", not eight tags in total. The real count on the three declared wires is 40, and R-260's own census already listed more than eight. Recorded because this prompt's own rule 6 says not to quote a document as source. TWO THINGS THE CONTROL CAUGHT, both before the gate was trusted: 1. A SUBSTRING FALSE NEGATIVE. `grep -F healed_at` also matches `privsep_healed_at`, so a genuinely dropped field read as received — and R-260 named healed_at, so its absence from the output was the tell. Now a whole-token regex; healed_at is convicted. 2. dr_recipe IS NOT WHOLLY OPAQUE. The hub stores each half as json.RawMessage and re-emits nested shapes verbatim, so the LEAVES are genuinely not on this wire. But the TOP-LEVEL SECTION KEYS are decoded by hostHalfShape/appHalfShape, and those are ALLOW-LISTS: a section an emitter adds is silently dropped until named in both. That already cost `offsite_restic` (R-122). So the gate is opaque BELOW depth 1, not opaque — the sections are checked and pass. Self-test: `--selftest` plants an unreachable tag on a real root in a throwaway copy and asserts conviction. Verified: exit 1, planted tag named. Blind spots are in the module docstring AND in the gate's own output, because Campaign 12's C1 guard turned out blind to one of the three shapes it was written for: generic tag names are not checked; reachability of a NAME is not use of a VALUE; only declared ROOTS are covered, and the hub's desired-state (served as raw stored JSON, no typed emitter) and the agent local API are NOT. Allowlist entries carry a stated reason. A quiet exclusion is a dropped field with paperwork. Not pushed alone: the fixes follow in the next commit so main is never red on this check. |
||
|
admin
|
b7fb2117ae |
CAMPAIGN 12 — the class sweep: golden 0.208.0 baked (awaiting vouch), R-256..R-263 filed, gating ranked
gates / gates (push) Successful in 20s
Part 1. Golden 0.208.0 baked on the drill VM, published and ROUND-TRIP VERIFIED — 656 150 362 B, sha256 ba668f59..5ffb82, and ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.208.0. Acceptance markers all green (overlay2 x1, mount points x2 rootfs+mp0, upload HTTP 201 x1, excluding/FATAL/mp1 x0), Result=success. Token file->file, read inside the VM; systemctl show grep = 0; committed-log grep = 0 WITH a control proving the grep works. Bake VM destroyed, drill disk restored to virgin. NOT VOUCHED — the campaign halts there deliberately. golden_currency_gate.py was correctly RED on arrival and is green after the bake. No --no-verify was needed anywhere in this session. Parts 2-4. Seven defect classes swept for siblings by class rather than by feature. Analysis only: no product code, nothing deployed, no machine touched beyond the bake VM. Eight new rows R-256..R-263 (ceiling moved from R-255), grouped by class in OPEN-ITEMS.md. C1 produced no new instance and has no row. The sharpest is R-260: the agent reports operator_key_configured every heartbeat, the hub has no field for it, so the check that answers "can the operator get into this box" returns ok for a box with no operator key installed. Every class states whether its method re-found the known instances, because a method that cannot re-find them has not been shown to work: C1 2/3 (verified by replaying the pre-fix templates), C2 2/2, C3 2/3 + 1 as fixed, C4 fix-pattern re-found, C5 re-found, C6 deadcode 0/2 and bespoke 1/2, C7 weakest and said so. Blind spots stated per class; seven suspicions investigated and DISPROVED, including two of my own methods. Part 4's ranking is in ROADMAP.md as G-1..G-8. Gate C5 (cross-repo tag reachability — cheap, --fast-eligible, would have caught every R-260 instance on the introducing commit). Do NOT gate C6: golang.org/x/tools/cmd/deadcode was measured against a PLANTED probe and is blind to unreachable METHODS on widely-used types, which is exactly the shape both known instances have. R-242's untouched half is recorded, not built: this bake demonstrated it, the currency gate flipping green the moment the evidence DIRECTORY existed, before the round trip finished and with no vouch near it. Correction the campaign owed its own brief: escrow_stale was described as closed; it is R-247 and READY. The live repo is the source. Sampled rather than swept, exactly: C7 60 of 2652 production invariant comments and NONE of the 1440 test comments (that half is owed); C2 19 of 221 refusals; C3/C4 controller only. No finding was reproduced live. STATUS.md is 100 lines against its 93-line one screen. |
||
|
admin
|
f651b31a7a |
golden 0.207.0 BAKED, PUBLISHED, ROUND-TRIP VERIFIED and VOUCHED — the currency gate goes green
gates / gates (push) Successful in 22s
Closes the delivery gap v0.207.0 opened this session. Until now the gate was correctly red and a machine installed today would have received 0.206.0 — the release written, tested and pushed, and not delivered. Round trip is the evidence, not the build log: the published bytes were downloaded back (656 879 192 B, sha256 20ec9602…22995, both identical to what the bake reported) and ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.207.0 — the delivered artifact naming the controller it will start. Acceptance markers were the ones R-233 re-captured from a real log: docker OK (overlay2…) x1, including mount point rootfs AND mp0 x2 (there is no mp1 since build-golden.sh v3.0.0), upload OK (HTTP 201) x1, excluding 0, FATAL 0. The 404 pre-gate ran WITH a control so a 404 could not mean 'wrong URL': 0.206.0 -> 200, 0.207.0 -> 404. The token never crossed a shell — copied file->file, read by a runner script inside the VM; systemctl show grep for the value returned 0. The token-leak grep on the COMMITTED log returned 0, and that 0 is evidence because a planted copy returned 1 before being shredded. Vouch was a three-field change with all three checked deliberately: MinAgent 0.127.0 read from the golden's controller CHANGELOG header, agent_version already >= it, min_agent not above agent_version (not the R-216 shape). Verified by re-reading the manifest rather than trusting the flash. The R-120 gate did not refuse. Drill VM restored to virgin; qemu confirmed exited with ps -eo comm, not a self-matching pgrep -f. R-242: the bake half is done and the --no-verify bypass declared earlier today is now historical. Its remaining half is UNCHANGED — nothing gates the VOUCH itself, so a baked-but-unvouched golden still passes the currency gate silently. |
||
|
admin
|
c1dec41328 |
walk5 venue TORN DOWN — census 168 rows -> 67, and R-244 grew by 30 as predicted
gates / gates (push) Failing after 13s
Operator-confirmed. Stopped under a name guard (demo-hp carries its own 9201), aged past the hub's stale_threshold read from the DEPLOYED ConfigMap (30m), and polled delete-impact until deletable:true — treating an empty response as retry, never as success. Cascade + qm destroy --purge, guarded a second time. Every layer verified absent against a positive control that must survive and does: VM 300 drill-r50 and demo-hp's own guest 9201 still there; ep0 namespaces demo-felhom + demo-hp still there; wg peers .2 .3 .4 .250 still on the live wg0; hub rows for demo-felhom, demo-hp, peti-felhom untouched. 16.64 GiB returned against 17 G measured. RECORDED FOR THE NEXT TEARDOWN: the WG peer is removed on a ~5-minute SCHEDULE, not by the cascade. Immediately after the delete the hub row was gone while 10.77.0.5 was still on ep0's live wg0; wgsync had last run 37 seconds before the cascade, and the next push (4 peers) removed it, verified on the live interface at 16:57:07Z. The previous ledger checked this after it had already converged, so it read as instantaneous — a teardown that checks too soon would file a false finding. R-244 grew by 30 rows (app_log_issues), PREDICTED in the pre-run enumeration rather than discovered afterwards. Running total across torn-down venues ~101. Nothing here claims a clean teardown. Storage Box layer evidenced from the hub's own deprovision log: the HETZNER_API token in ~/.config/credentials cannot see box 611421 (subaccounts -> 404, storage_boxes -> 200 with 0 entries) — it is scoped to another project. |
||
|
admin
|
3f4fb3825f |
R-201 CLOSED — the unaided recovery journey passes, both halves, on the fifth walk
gates / gates (push) Successful in 17s
Capability map: the unaided-recovery row turns FAILED -> PROVEN-LIVE, scoped, with what it still does not claim stated in the row itself: shape (c) did not fire positively (with the mint guard holding there is no local key, so the offer comes from shape (a)); and 'unaided' here means possible-without-a-shell, not obvious, because two obstacles are unsignposted. OPEN-ITEMS: R-201 closed with its evidence. Five new rows R-249..R-253 (the retrieval passphrase in page HTML; the host-key scan ladder vs AAAA settle; the listing's per-tag rows; the two unsignposted restore steps). R-243 annotated rather than re-filed: on a REBUILD offsite_delivery_stuck does not skip, so the row's gap is narrower than it reads. STATUS.md: headline changed, and trimmed 97 -> 92 lines rather than extended, per its own header. Teardown recorded as OWED with its before-measurements, the stop-and-age gate, and the positive controls that must survive. |
||
|
admin
|
0691bc59a5 |
walk5 (R-201): Phase B + the verdict — BOTH HALVES PASS
THE DATA: PASS. All three sentinels byte-identical out of snapshot 5b0f20f7, including the accented filename's bytes, read back as bytes from the live path. THE JOURNEY: PASS — the first time in five walks. Zero guest command lines were needed to progress; the previous walk needed three. The reset-code hatch was used once, in Phase A only. §5's observation, which stands on its own whatever the verdict: at 14:58:52Z the rebuilt box collected its re-staged credential, configured the transport, and REFUSED TO MINT a repository password over the sealed package the hub holds. At the equivalent moment the previous walk minted a fresh key and lost the journey silently at 03:18. Sampled every 20s from T0: no key at any moment. Honest about which shape fired: with the mint guard holding there is no local key, so the offer comes from shape (a), not shape (c). Shape (c) was measured in Phase A in its NEGATIVE half (equal hashes, correctly silent). This proves the mint guard positively and the discriminator negatively. RTO 71.7s login to open store, of which 12.44s was the unseal and ~22s my own CSRF harness retry. Two new customer-facing obstacles, neither needing a shell but neither signposted: the restore refuses on unattached drives, and refuses because the app is not installed on a page that says the restore reinstalls it. |
||
|
admin
|
879007aaec |
walk5 (R-201, fifth walk): Phase A journal — fixture built, sentinels proved by name
gates / gates (push) Successful in 21s
Written before the destruction, per §9.9. A fresh install landed on the VOUCHED set with no hand upgrade (controller 0.206.0, agent 0.127.0) — R-239's delivery gap is closed for this run, which is the first of the five walks where the box under test is the box a customer receives. Both §4.6 pre-destruction checks pass, neither previously exercised on a clean box: the recovery offer is correctly SILENT (shape (c) compares equal — the two key hashes are byte-identical on box and hub), and the restore page lists the app with the future-backup toggle OFF (R-237's fix, which the last walk measured failing). Also recorded: the §4.5 gate caught a harness fault (a toggle sent as enabled=1 rather than enabled=on) that had produced a green 'ok' over a zero-snapshot repository — the exact shape the gate exists for. |
||
|
admin
|
721297ed5e |
golden 0.206.0 baked, published, round-trip verified — NOT vouched; the gate goes green
gates / gates (push) Failing after 10m43s
Controller v0.206.0 shipped the R-241 fixes on 2026-08-07 while the vouched golden still carried 0.205.0, so a machine installed this morning would have received neither - and the next recovery walk would have measured the old behaviour and failed for a reason nothing to do with the walk. Same gap as R-239, one day after R-239 was closed. version 0.206.0 sha256 c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e size 656,750,694 bytes (uncompressed 2,003,138,560) MinAgent 0.127.0 Round-trip verified rather than trusted: the published bytes were fetched back, re-hashed independently (match), zstd-tested, and ./etc/felhom-controller-image was read OUT of the download -> felhom-controller:0.206.0. That last step is the one that matters, because GOLDEN_VERSION is derived from the tag argument and could be right over stale content. All acceptance markers pass; unit Result=success ExecMainStatus=0. Secret hygiene: token file->file, in-VM runner so it never reached a command line (unit-property grep 0), literal-value leak grep on the COMMITTED log 0 - with a positive control proving the grep works before the 0 was believed. Bake VM torn down: CT 9100 purged, secrets shredded, qemu observed gone via ps -eo comm, drill.qcow2 reverted to virgin. THE GATE BUILT EARLIER THIS SESSION NOW PASSES. It was shown CONVICTED against the pre-bake state and is OK now - red to green on the same check, the same command, which is its proof that it measures something real. Note it went green on the BAKE, not the vouch: that is its stated limitation, and the vouch is still pending the operator. NOT VOUCHED - the operator's act. Only one field moves: golden_version 0.205.0 -> 0.206.0 (+ its derived sha). agent_version and min_agent both stay 0.127.0. wrapper_sha256 is unchanged but is CLEARED if omitted from the POST. |
||
|
admin
|
094e93e828 |
finalwalk teardown complete; R-244 filed; session report
gates / gates (push) Successful in 13s
All five layers gone, each verified with a positive control that must
survive and does:
VM 324 + 4 disks -> absent (VM 300 drill-r50 remains)
hub: 13 tables at 0, incl. BOTH escrow tables (demo-felhom/demo-hp/peti remain)
Storage Box u629488-sub4 -> gone (sub1/2/3 remain)
ep0 PBS ns finalwalk -> gone (demo-felhom, demo-hp remain)
WireGuard 10.77.0.5 -> gone from the LIVE wg show on ep0, not just
the hub DB (.2/.3/.4/.250 remain)
14.06 GiB reclaimed against 15 G measured before deletion.
R shredded with a planted-copy control: plant -> search finds both ->
shred -> the same search finds 0. The zero was not believed until the
instrument was proven.
R-244 (NEW): a FULL census after the cascade logged COMPLETE full teardown
found 61 rows still matching finalwalk. Four sources are deliberate
provenance; the fifth, app_log_issues (29 rows), is NOT covered by the
residue purge - and it is systematic: c11 40, rewalk 20, part4 24 still
present from the 2026-08-06 teardown, whose ledger recorded zero
occurrences. That claim used a narrower query than a census and does not
hold; the correction is recorded in both the prior ledger and the register
rather than the measurement quietly redone.
No secret material is involved. The table is a fleet-wide aggregate: 12 of
the 29 rows are finalwalk-only orphans, 17 are shared with LIVE customers
and must be de-referenced, not deleted - very likely why the leg was never
written. Not fixed; a cascade change needs its own red-proof.
Lesson, and it is the reusable part: a per-table absence query is not a
census.
|
||
|
admin
|
08b75e602e |
golden 0.205.0 baked, published and round-trip verified — NOT vouched (R-239)
gates / gates (push) Successful in 13s
Closes the delivery gap's build half. The vouched golden carried controller 0.203.0 while 0.205.0 was released, so a machine installed last night got neither R-237 (restore list keyed on the store) nor R-234 (skipped-app verdict). Both were measured from the customer's side on that box. version 0.205.0 sha256 8f49b2e8ccbc86a49df821fee9fb00c07293758811d3d0f0512dd0cf5fd54ee8 size 656,937,561 bytes (uncompressed 2,003,343,360) MinAgent 0.127.0 Round-trip verified rather than trusted: the published bytes were fetched back, re-hashed (match), zstd-tested, and ./etc/felhom-controller-image was read OUT of the downloaded archive -> felhom-controller:0.205.0. That last step is the one that matters, because GOLDEN_VERSION is derived from the tag argument and could have been right over stale content. Acceptance markers all pass; unit Result=success ExecMainStatus=0. Secret hygiene: token file->file, in-VM runner so it never reached a command line (unit-property grep 0), literal-value leak grep on the COMMITTED log 0 - with a positive control proving the grep works before the 0 was believed. Bake VM torn down: CT 9100 purged, secrets shredded, qemu observed gone via ps -eo comm, drill.qcow2 reverted to virgin. NOT VOUCHED - that is the operator's act. Only ONE field actually moves: golden_version 0.203.0 -> 0.205.0 (+ its derived sha). agent_version and min_agent both stay 0.127.0, because the new golden's MinAgent is also 0.127.0. The R-120 gate passes exactly: the newest controller the fleet reports is 0.205.0, so a 0.204.0 golden would have been refused. |
||
|
admin
|
2228c0bff6 |
final walk COMPLETE — data PASS, journey FAIL; R-241 filed
gates / gates (push) Successful in 15s
THE DATA: PASS. All three sentinels byte-identical out of snapshot f5c53b03, including the 12 MB binary and the accented Hungarian filename whose NAME BYTES are identical too. Disk -> restic -> SFTP -> Storage Box -> rebuilt machine -> disk, intact. THE JOURNEY: FAIL, and further from the line than the previous walk. The claim worked first try (302 in 0.164s). Then: / lands on the launcher with no recovery pointer, /recovery 302s away, and the remote page offers to CREATE a new recovery code — which would orphan the history the customer's code protects. There is no field anywhere to enter the code they hold. The operator's documented remedy also refuses, correctly and fail-closed. Recovery needed three guest command lines. R-241 — and the cause is a success this same walk proved six hours earlier. OffsiteRecoveryOffer() shows the screen only when (a) there is NO repository password (pristine rebuild) or (b) one exists but the history will not open under it. Overnight the credential self-heal collected the staged credential and applied the tier, writing a FRESH key at 03:18Z — so (a) is false; and (b) is unreachable because orphan detection needs a run, and runs are blocked by escrow_state=pending. The gap is self-locking. Measured keys: on-disk 9b4a9a9d... vs recovered-from-R 30ef574f... This is R-218's shape one level up: succeeding at the self-heal stopped the box OFFERING the recovery it still needed. Registers: R-201 moved to its outcome; R-241 filed; capability map's recovery row stays FAIL with both halves and the cause named; STATUS rewritten for the operator. Highest ID R-238 -> R-241. The venue is left with the recovered key in place and the self-heal key moved aside, never deleted. Teardown still owed. |
||
|
admin
|
1ff6f8f8e0 |
final walk §7: the credential chain runs end to end, unaided, on an UNCLAIMED box
gates / gates (push) Successful in 12s
Three questions answered from the hub's own log, not inferred:
1. the rebuilt, still-unclaimed box DOES report (host-report + Received report)
2. it DOES declare offsite.state=needs_credential, and offsite-delivery correctly
declines once a minute, naming internal/offsiteheal as the owner
3. offsiteheal re-staged UNAIDED at 03:15Z, after two reports carried the
declaration, with no provider credential minted — about 32 minutes after the
rebuild, matching the documented 2x15-minute debounce
And then the box COLLECTED it on its own 5-minute tick:
[offsite-apply] credential retry: the staged credential was collected and the
tier applied
That success line shipped in v0.203.0 and this is the FIRST time it has been seen
live: yesterday's walk only produced its sibling before I intervened at 102s and
mistook my own button press for the cause — the error that produced R-236 and
forced its withdrawal. Here nobody touched anything and the box was not even
claimed. R-218's consume half, R-236's withdrawal and the previous walk's dead
end 1 are all settled by one unattended observation.
|
||
|
admin
|
1b490c8cbf |
final walk: destroyed, rebuilt, and HALTED at the claim screen
gates / gates (push) Successful in 27s
Destroyed 02:40:31Z (guarded on hostname — demo-hp also has a guest 9201), drives wiped to 20K with the mounts deliberately left in place because the surviving raw mount IS the R-220 condition. Reinstalled through the published day-0 path, installer v1.25.0 fetched live; Day-0 provision SUCCESS in 2m32s. R-239 measured a second time, from the other side: the rebuild landed on agent 0.127.0 (no downgrade, no hand upgrade — that half is right) and controller 0.203.0. The box a customer would recover on tonight also lacks R-234 and R-237. The machine is AT THE CLAIM SCREEN awaiting the operator. A claim code has already been requested through the customer-facing path and emailed, so the morning is paste-a-code rather than request-then-paste. The reset-code hatch was NOT used and will not be: it is a guest command line and would fail the rule the walk measures. Stated plainly in the journal: journey steps from the destruction onward are driven over HTTP from the appliance to the guest's island address, as a browser would; some instrumentation reads are guest command lines and are counted as such, but none changed state or was needed to progress the journey. |
||
|
admin
|
502078bebf |
final walk: the soak DID catch a scheduled cycle; both directions recorded
gates / gates (push) Successful in 20s
Window 2 (01:56-02:36Z) caught the off-site nightly at 02:15:03Z — snaps 1->2, last_run advanced with no prompting. Window 1 had already caught the local legs (db-dump 00:30:25Z, tier + fill-watch 01:30:19Z). My 04:15 prediction was wrong in the other direction; recorded rather than quietly fixed. What should have fired and did not: NOTHING. Six registered jobs were never seen in the log and none is a finding — four are quiet by construction (scheduler.go:267, quiet := job.Interval <= 30s) and two run every 6h, outside a 4h window. Checked the source rather than filing four phantom defects. What fired and should not have: nothing. No alert, notification, email or digest. The only WARN lines in five hours were three of mine. Observation kept: offbox-backup ticked twice 62s apart and produced ONE new snapshot — the second silently dropped by the single-flight, which is correct for the nightly path and is the same mechanism that produced R-234 on the manual path. |
||
|
admin
|
f873c557ca |
final walk: the soak, and the window correction stated as mine
gates / gates (push) Successful in 20s
Window 1 (22:44-01:54Z, untouched): every periodic job fired at its declared cadence — agent-channel-health 185x/1m, stack-scan 108x/2m, the 5-minute trio 44x each, hub-report 14x/15m. offsite-credential-retry ran 44 times and did no work and said nothing, which is R-218's asserted healthy-box behaviour on a box that never saw the defect. No alert, notification or digest fired; nothing on the must-not list fired. The off-site state never moved during that window — and that is NOT a finding. The daily jobs run on the controller's clock and the guest is UTC while the appliance is CEST, so the nightly local (~02:30) and off-site (~04:15) fall at 02:30Z and 04:15Z. I sized the window against CEST and it closed at 01:54Z, before either. Reporting 'the nightly did not fire' would have been a false finding produced by a badly-chosen window. Window 2 (corrective, 01:56-02:35Z) covers the 02:30Z local backup. The 04:15Z off-site nightly is deliberately NOT covered and is recorded as not run with the reason: leaving the machine at the claim screen before 07:00 is the primary deliverable, and waiting would have put the destruction at ~06:35 CEST with no margin. The off-site tier was exercised four times manually instead, including a full listing by name. |
||
|
admin
|
2d2d8d3ddc |
final walk Phase A + the five checks; R-239 (delivery gap) and R-240 filed
gates / gates (push) Failing after 13m22s
Phase A complete on a NEW venue (demo-hp VM 324, customer finalwalk). Installed
from the published ISO — verified byte-identical to iso.felhom.eu — claimed, app
deployed, three sentinels placed, escrow ceremony 6/6 with restic_pw_sealed=true,
R held in one 0600 file on DooPlex, off-site backup taken and the three sentinels
PROVEN IN THE SNAPSHOT BY NAME (snapshot 1da4f80d, 12.0 MB).
THE HEADLINE: a fresh install lands on agent 0.127.0 (vouched, no hand upgrade —
that half works) and controller 0.203.0, while the newest is 0.205.0. The vouched
golden still bakes 0.203.0, so tonight's box has neither R-237 nor R-234. Filed as
R-239: a delivery gap, not a regression. Every closure claimed for those two is
true of the code and false of the fleet until a golden is baked and vouched.
The five checks, on that box:
T1 ok + the old bare-count warning (no next step)
T2 FAILS — both runs say "elindult", the second as a success flash
T3 FAILS — toggle off hides an existing snapshot; wizard 302s away
T4 ok, and the wording is still "Sikeres" beside "nothing is selected" -> R-240
T5 PASSES — driven as a browser does, prepare -> full_prep -> confirm -> commit
completed; R-238 confirmed a harness artifact, not a product defect
HARNESS FAULT, caught by the §4 gate: the first off-site run reported ok with a
26.6 KB repo — impossible for a 12 MB sentinel. I had placed the files under
.../adatok/felhom-data/userdata/... while this box's namespace root is
/mnt/felhom-drives/adatok directly. The product was correct throughout; proving by
listing rather than by a green status is what caught it before any destruction.
Nothing destroyed. The venue stands.
|
||
|
admin
|
fca85498e1 |
Part 5: the last venue torn down after its findings were written
gates / gates (push) Successful in 8s
part4 (VM 323) was kept while Parts 1 and 3 read it — it was standing in the exact
failed half-B state and tearing it down first would have destroyed the evidence.
Destroyed after the diagnoses were committed (
|
||
|
admin
|
70b62608b0 |
teardown of two venues; R-218/R-220 CLOSED, R-236 WITHDRAWN, R-238 reclassified
gates / gates (push) Successful in 18s
Part 0 — c11 and rewalk destroyed, three layers each plus the off-site side and
the WireGuard peer, via the hub's own cascade (external teardown FIRST, DB purge
LAST). 37.3 GB reclaimed on c11-scratch, matching the 20G+17G measured. Positive
control after each: part4 must still be found, and was. ep0 namespaces now exactly
demo-felhom, demo-hp, part4.
The cascade refuses to delete a live host and there is no decommission endpoint,
so both boxes were stopped and aged past the hub's 30m stale_threshold first.
Register corrections — the durable record was wrong about two shipped fixes:
R-218 REOPENED -> CLOSED. Shipped controller v0.203.0, proven live on a
genuinely rebuilt box: the hub re-staged at 13:24:57Z and the box
collected it on a tick, no guest command line, no operator action.
R-220 "OPEN — NOT FIXED" -> CLOSED. Shipped agent v0.127.0, proven live after a
real guest purge with both raw mounts still on the surviving host:
/disks/candidates returned both drives (before: two empty lists) and both
re-attached through the customer endpoint.
R-236 WITHDRAWN — I FILED THIS WRONGLY. The hub log shows offsiteheal re-staged
the stored secret at 13:24:57Z after its documented two-report debounce, with no
provider credential minted. My Re-issue at 13:26:39Z came 102s LATER, was
redundant, and minted an unnecessary provider credential (subaccount 284735) —
the very double-issue the offsite-delivery guard warns about once a minute in the
log. "Nincs teendod" is true; I did not wait ~16 minutes. Operational lesson, not
a product defect.
R-238 reclassified as a harness artifact (mode=full without confirm=1 is step 1 of
a deliberate two-step and starts no job by design); its real residue — the total
silence of that step — is fixed in controller v0.204.0.
R-237 CLOSED by controller v0.204.0.
|
||
|
admin
|
5810e1f552 |
Part 4 half 2: upgrade path PASSES, recovery journey dead-ends (R-236/237/238)
gates / gates (push) Successful in 8s
Destroyed guest 9201 (guarded on hostname — demo-hp also has a 9201), wiped both
drives leaving the raw /mnt/<name> mounts in place, reinstalled from the
published installer v1.25.0.
PASS — the upgrade path:
agent 0.127.0 -> 0.127.0 across the rebuild. The previous re-walk DOWNGRADED
0.126.0 -> 0.125.0 (R-216 part 4). Controller 0.203.0 from the baked golden,
Day-0 provision SUCCESS in ~2m50s, nothing applied by hand.
R-220 FIXED, proven on a genuinely rebuilt box: with both devices mounted raw
AND managed, /disks/candidates offered both drives (before the fix: two empty
lists), and both re-attached through the customer endpoint.
R-218's consume half proven live end to end: credential re-staged at 13:26:32Z,
collected on a tick, tier configured — no guest command line. Previously this
exact situation was dead end 1 and needed a guest shell.
FAIL — the data half, NOT reached. Three new findings, in the order they bite:
R-236 nothing re-stages the off-site one-time password after a rebuild, so the
"Nincs teendod" promise stalls until an operator re-issues. The retry job
is NOT at fault — it retries correctly and logs why.
R-237 the restore surface is keyed on installed + remote-marked apps, so after
a rebuild the customer is shown no backups at all — a circular dead end.
R-238 "Teljes visszaallitas elokeszitese" returns 302 and does NOTHING: no job,
no state change, no error, no line in the controller's own debug ring.
mode=unit on the same form works. This is why the sentinel byte-identity
criterion was not met — nothing came back, rather than coming back wrong.
No guest command line was used at any point in either half — the property the
previous walk could not claim.
|
||
|
admin
|
ac451f9667 |
Part 4 half 1: fresh install lands on vouched 0.127.0/0.203.0 with no hand upgrade
gates / gates (push) Successful in 9s
Venue: demo-hp VM 323 part4-appliance / guest 9201 / hub customer part4.
Bind 13:53:40 -> agent 0.127.0 ONLINE 13:55:34, no "floor held", controller
0.203.0 from the baked golden. The previous re-walk's reinstall DOWNGRADED the
agent (R-216 part 4); this one did not, because the vouch moved all three fields
(golden_version + agent_version + min_agent) together.
R-218 on a real installation: offsite-credential-retry registered (every 5m0s)
and ticking, 0 "credential retry" lines, and the tier configured unaided 14s
after the settle gate — the previous walk's dead end 1 is gone. Stated precisely
in the journal: this venue proves the SILENT half only; the collecting half was
proven separately with a positive control.
R-225 visible in both states on a live box: "még ismeretlen" before any
snapshot, "12.0 MB / 50 GB" after. Unknown is not zero.
Fixture complete with NO guest command line — the claim used the emailed code
through the real /claim form (the hub keeps only a hash), and the sentinels were
written over SMB with the share mounted so the accented filename never crossed a
shell (verified as hex on both sides, no U+FFFD).
Two new findings, both "presence is not success":
R-234 an offsite run reports success while omitting an app just toggled on —
found ONLY because the pre-destruction verification restore was run
instead of trusting the green tick.
R-235 the console still shows the pairing banner 25 min after the bind, while
promising it refreshes itself.
Half 2 (destroy -> reinstall -> recover -> compare) is owed and NOT started; the
venue is standing and its teardown is owed.
|
||
|
admin
|
1c47e3b6fd |
golden 0.203.0 baked + published; runbook acceptance markers fixed (R-233)
gates / gates (push) Successful in 13s
Bake evidence: documentation/tests/golden-0.203.0-2026-08-06/ (bake.log + README). sha256 3039c6ffa7a5a8b2d959daddb2895c58b44de70f8d4f4a7e12ad4b1c0d61dc88, verified by an independent round-trip download and by reading /etc/felhom-controller-image out of the published archive itself. NOT vouched — the hub still serves 0.201.0. R-233: RUNBOOK-manual-build.md §4.1 named three pass markers, two of which the script cannot print (`overlay2 OK` does not exist; `mp1` stopped existing in build-golden.sh v3.0.0 under R-165), and a 404 pre-gate URL with the wrong filename, which would 404 for the wrong reason and pass even when the version already existed. A grep for an impossible string reads 0 forever and 0 is indistinguishable from failure. Markers re-captured from the real log; token handling moved off the command line into an in-VM runner script; a positive control is now required on the token-leak grep; the vouch step rewritten as the three-field change it is (golden_version + agent_version + min_agent). |
||
|
admin
|
0c4411e54b |
R-201 re-walk: the data PASSES again, the journey still FAILS — two dead ends, down from four
gates / gates (push) Successful in 9s
Asked Campaign 11 Phase 1's question a second time, on the fixed build, on a
NEW appliance (VM 322, customer rewalk). The Campaign 11 venue was untouched.
THE DATA: PASS. All three sentinels byte-identical out of the pre-destruction
snapshot a7bc23bd in 23s through the customer's own restore flow — including a
12 MB binary and an accented Hungarian filename whose NAME BYTES are identical
too (verified as hex, not as rendered text).
THE JOURNEY: FAIL, two dead ends against Phase 1's four.
1. R-218's CONSUME half. The hub re-staged the credential at 11:44:57 saying
'the box re-consumes on its next cycle'; a full cycle ran at 11:55:46/54
(with a positive control that it ran) and it did not. A census of the
customer-reachable actions found none that fetches it. Only a command line
INSIDE THE GUEST moved it — 18s, confirming nothing was wrong with the
credential, target or key: only the trigger. R-218's row said SHIPPED and
over-claimed; it is corrected to REOPENED for the consume half.
2. R-220. Drives still unenrollable after a rebuild, needing a Proxmox-host
unmount; without it no app redeploys and the restore page stays empty.
Unaided RTO STILL UNDEFINED. Attended: +45s key placed, +24m12s tier up,
+30m13s data verified. The 30m must not be quoted as the customer number.
What passed and is new: the recovery screen appeared WITHOUT being sought,
answered all three questions with a seal date matching the hub exactly, the
emailed reset code worked first try, the unlock was a real 1.528s unseal, and
R-225's fix was seen working in the wild (unknown, not a false zero).
R-216 part 4 reproduced live: the reinstall downgraded the hand-installed agent
0.126.0 -> 0.125.0.
DELIVERY GAP recorded as owed and NOT conflated with the journey: a fresh
install landed on controller 0.201.0 / agent 0.125.0 — the vouched versions,
neither carrying the fixes — installed by hand. Nothing was vouched.
Capability map row STAYS FAIL. Campaign 11 doc gets a dated ADDENDUM, not a
rewrite.
|
||
|
admin
|
6a835ad9ec |
Re-walk R-201: Phase A record, written BEFORE anything is destroyed
gates / gates (push) Successful in 8s
The fixture is built and the gate passed: all three sentinels are in snapshot
a7bc23bd BY NAME and at the right sizes, and the accented filename survived
into restic intact.
Recorded because they are results, not incidentals:
- what a FRESH install actually landed on — controller 0.201.0, agent 0.125.0,
golden 0.201.0. Neither carries the fixes. Brought to 0.202.0/0.126.0 BY HAND,
and the delivery gap is recorded as owed: this proves the journey, NOT that a
customer would receive that build.
- baseline drift caught on arrival: one documentation-only R-229 commit per repo,
and the highest register ID is R-229, not the runbook's R-228.
- the runbook's installer baseline (1.25.0) is stale; the published artifact is
1.26.1, verified byte-identical to demo-hp's local copy.
- credentials DISCOVERED not assumed: HUB_PW mapped and verified live; nothing
else was needed from the file.
A harness fault worth keeping: the accented sentinel's filename was destroyed at
creation by my own base64->bash->pct exec chain, producing U+FFFD replacement
characters. A Python decode('utf-8') check called it VALID UTF-8, because U+FFFD
is valid UTF-8 — only a hex dump of the filename bytes exposed it. Rewritten
from explicit bytes inside Python on the guest.
R claimed one-shot, streamed file->file to a 0600 store on DooPlex, never
rendered, shape verified without printing (10 words, 75 chars). The runbook's
§5.5-vs-§9.4 tension about where R may live is flagged in the journal rather
than resolved silently.
|
||
|
admin
|
9c1d05d360 |
CAMPAIGN-11: hygiene, what-did-not-run, venue end state, and the session report
gates / gates (push) Successful in 8s
The recovery codes are shredded with the plant->find->shred->fail-to-find
control the brief asks for, and THE CONTROL PAID FOR ITSELF ON ITS FIRST RUN:
it found the Phase 0 code in ~/.config/credentials as R_CAMPAIGN_11 — a copy
this session did not create and would never have looked for. Without it, a
'codes shredded' claim would have been false. That key was removed from the
shared file with a verified diff (every other line identical, nine keys intact)
and HUB_PW re-tested at hub:200.
Consequence stated plainly rather than left to be discovered:
/home/felhom-repo.orphaned-20260805 (12 535 KB, the three Phase 0 sentinels) is
now permanently unopenable — which is what the set-aside screen promises, and
teardown removes it anyway.
Venue left WORKING and said so: ONLINE, 4 containers healthy, backup target not
degraded, off-site on 2 snapshots. Two things a future session needs: the raw
/mnt/{adatok,mentes} mounts are deliberately left unmounted (R-220's
workaround), and the appliance root credential was shredded — re-fetch it from
the hub.
REPORT-campaign11-phase24.md rather than REPORT.md, per the repo's
parallel-session rule.
No product code changed. No version bumped.
|
||
|
admin
|
df6081e60b |
CAMPAIGN-11 Phase 4: the soak passed, with two absences answered honestly
gates / gates (push) Successful in 8s
All five daily jobs fired exactly once and completed on time. The 04:15 offsite run produced snapshot_count 1 -> 2, last_status ok, 24.9s — unprompted, on a box rebuilt twice and set aside four hours earlier. Nothing on the must-not list fired, including offsiteheal: that is R-218's NEGATIVE control passing (a box with a target must not declare), which does not substitute for its positive half, still owed. Investigated and DISPROVED: tier2-backup completing in 118ms looked like a silent no-op; the copy is real and on the backup drive (818.5 KB verified). Correction to my own pre-registration: backup_run_digest is a TEST FILENAME, not an event type. The real one is backup_run_failures, a failures digest whose silence on a clean night is correct — reporting it as a miss would have been a finding I invented. What survives: the offsite run emits no hub event while both lesser tiers announce success; failures and staleness are covered elsewhere, so it is a consistency wrinkle, recorded not filed. Two absences answered rather than assumed: the restore-test's silence was PRE-REGISTERED as correct (6h eval, 24h settle); the agent's whole-guest tier is left explicitly UNRESOLVED because routine local-api calls are not logged at INFO — a 5h search returns 0 on a box that demonstrably served such calls, so the instrument cannot answer. No product code changed. |
||
|
admin
|
07998dcff2 |
CAMPAIGN-11 Phase 4: pre-registered expectations, both directions, before the window
gates / gates (push) Successful in 9s
Written BEFORE the soak so the comparison cannot be fitted afterwards. Sources are the box's own [scheduler] Registered lines and the agent's backup tier armed lines, read at 23:56. MUST happen: db-dump 02:30, tier2-backup + fill-watch 03:30, metrics-prune 04:00, offbox-backup 04:15 (snapshot_count 1 -> 2), a backup_run_digest event, reports every ~15m. MUST NOT happen: host_stale/host_recovered, offsite_selfheal_restaged (this is also R-218's negative control — a box WITH a target must not declare), offbox_repo_orphaned, offsite_repo_key_changed, escrow blob SERVED, any controller self-update. Genuinely uncertain and recorded as such: the restore-test has a 24h settle and every archive here is hours old, so NOT running is correct, not a gap. |
||
|
admin
|
f88faad50b |
CAMPAIGN-11 Phase 2 complete: F7/F9/F10/F11, and R-228 (the set-aside history is invisible)
gates / gates (push) Successful in 8s
F11 PASS ok->stale->ok, an operator mail each way, customer mail correctly
pairing-gated. DOWN (>1h) not reached — stated, not implied.
F7 the set-aside PASSES decisively: move-aside to
/home/felhom-repo.orphaned-20260805, mtime and du -s (12535 KB) and
snapshot f3d9cd67 all unchanged; a fresh empty repo beside it.
NOTHING DELETED.
the afterwards FAILS -> R-228: settings.json records
"orphaned_renamed_to" and OrphanedRenamedTo has ZERO references in
any template or handler. 12.5 MB of retained customer data at a path
the box knows and never shows. Seam built, never wired.
F9 R-215's fix PROVEN LIVE on a narrower shape: with recoveryOffer() false,
GET /recovery now 302s instead of rendering its story. The literal
never-had-offsite precondition was not staged (it needs a rebuild,
which the brief forbids before Phase 4) — recorded as partial.
F10 NOT INJECTED, harness not product. Three attempts, each with a control:
the app, then the controller's monitor, then the run itself recreate
the mandatory directory within ~1 s. Kept as an observation: at
capture the directory held only a recreated metadata.db and NOT the
customer's file, and the run still said ok.
Also reproduced live for a third time: R-220 (candidates empty; unmounting only
the raw mounts flips both drives back) — still OPEN.
Venue left WORKING: 4 containers healthy, backup target not degraded, off-site
fresh repo last_status ok, escrow escrowed.
No product code changed.
|
||
|
admin
|
04ac465da6 |
CAMPAIGN-11 Phases 2+4: the fault journal, the campaign document, and hub v0.97.1's missing heading
gates / gates (push) Successful in 8s
Phase 2 (eleven injected faults) and the §4 positives that were owed. - §4.1 MEASURED, twice: the box's rendered GetFloor() is 0.200.0, and a cold-started controller logs "settle-gate: GO — at/above floor 0.200.0" against the same line reading "floor still unknown" while the hold was in force. Also corrects the brief's plan: SetFloor's line is u.dbg(), gated on logging.level=debug and written to the logger, so it can NEVER reach the debug ring — a restart alone would not have produced it. - §4.2 still NOT measured, deliberately: the venue has an off-site target, so needsOffsiteCredential correctly returns false. Recorded, not inferred from the unit test. - F1 PARTIAL, F2 PASS, F3 FAIL, F4 FAIL, F5 PASS, F6 PASS, F8 PARTIAL. F3+F4: a hub outage and a stopped agent are both rendered as "this code does not open your package", in 0.056 s and 0.030 s — no unseal attempted. The agent's own err field distinguishes them exactly and it is discarded at the HTTP boundary; the R-216 capability gate answers source=version and cannot see reachability. - R-217's fix HOLDS under exactly its fault (F5), verified with the false-claim strings absent and accented positive controls present. hub/CHANGELOG.md: v0.97.1 had no heading of its own — the change was written into the v0.97.0 entry while the deployed tag is 0.97.1. Given its own entry, marked as added retroactively. Second occurrence of the class (agent 0.90.1). Evidence: documentation/tests/campaign11-evidence-2026-08-05/journal-phase24.md No product code changed. |
||
|
admin
|
3a539ea530 |
docs: R-223 CLOSED by a golden rebake — and it was never a dropdown
gates / gates (push) Successful in 11s
The first vouch of agent 0.125.0 was REFUSED by R-120's gate: the artifacts form saves as a unit, and golden 0.192.0 was older than the newest controller the fleet reports. The golden had been stale since before controller 0.193.0, so the Day-0 manifest had been effectively unvouchable for days and nobody had cause to notice. Correcting STATUS.md and OPEN-ITEMS, which said "one dropdown". Golden 0.201.0 baked in the drill VM, round-trip verified from Gitea (e730d7cab343eb35...f007654, 658165766 B), teardown clean, 0 token hits in the saved log. Manifest now agent=0.125.0 golden=0.201.0 min_agent=0.125.0, read back after the save. The held floor: the HOLD is established positively (hub log + the box's own "floor still unknown"). Its RELEASE is not — no HELD line and no dashboard reason are both absences, the served path logs nothing by design, and the box-side positive needs a restart the venue cannot take before Phase 2. Recorded as deduction plus a pinned test, not a live positive. |
||
|
admin
|
1a0f7db92f |
docs: CAMPAIGN-11 — the journey FAILED, R-198's retention PROVEN, six findings fixed
gates / gates (push) Successful in 8s
Registers and evidence for the campaign and its fix pass. OPEN-ITEMS: R-214..R-223. Six SHIPPED (R-215/216/217/218/219/222); three deliberately still open and each blocks a real flow (R-214 console banner, R-220 drives unenrollable after a rebuild, R-221 a rebuilt box cannot run the escrow ceremony); R-223 minted and WAITING-ON-OPERATOR (vouch agent 0.125.0). R-213 and R-202 untouched. Capability map: a new row for the customer's UNAIDED journey, recorded FAILED and staying failed until a re-walk passes — fixes are not a journey. The existing rebuild row is corrected where it said R-198's retention was unit-proven only: it was proven in production on the first supersession since the fix, identity_blob retained at 572 B byte-length exact. CLAUDE.md comment-vs-code table: eighth entry — ResolveManagedFloor, the first where the false invariant was a GUARD rather than a comment alone. STATUS: the headline is now "the backup promise is proved, the recovery journey is not", and the one thing waiting on the operator. |
||
|
admin
|
e34b614e5b |
docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected
gates / gates (push) Successful in 7s
R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The hub's notification_log for the run reads: two per-app failures RECORDED, one digest SENT naming both, and the customer channel SKIPPED with operator_only. Against the measured previous behaviour — two failures, one email naming one app, one leaving no trace anywhere. Scenario D proved itself on an event I had not planned: disk_critical alarmed on two filesystems, the second was collapsed by the cooldown, and that collapse is now visible WITH ITS KEY. Yesterday it would have left nothing at all. A gap the spec did not anticipate is recorded with its fix: the per-app event also fires from the periodic sweep, outside any run, so making it record-only would have created a NEW silence. The sweep emits a digest too, with no run_id, so it stays under the ordinary hourly cooldown. ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile SURVIVED the resize and is active (checked, because a resize is a stop/start). The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere. Five documents corrected — three of which the task's list did not name, found by searching. Two audit/evidence documents ANNOTATED, body untouched: they record what was true when written and that is their value. R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable, not unbounded — the original OOM was a 14.46 GB restore — so the restore-test cadence should still be paced, just not by fear of the endpoint. target-selection.md's "D-d did not name ep0 either way" is deliberately left standing. It is the operator's question, not CC's. STATUS.md 127 -> 83 lines, items rather than sentences. |
||
|
admin
|
5f35aa0346 |
Campaign 10: M-band RTO measured — RTO ~= 40s + 26.9s/GB, and a capacity ceiling that matters more
The S figures (66 MB -> 42.0s, two passes agreeing to 0.6s) had a spread tight enough to prove fixed work dominates, which is exactly why they said nothing about M. Second point taken 327x larger, same app, same method: clock from restore request to the app serving the correct discriminator. rallly's postgres volume grown 66 MB -> 21.1 GB (200k rows, STORAGE EXTERNAL so TOAST cannot compress it into a fake number). Two reps: rep 1 backup 406.4s unit 41149 MB RTO 624.5s discriminator correct rep 2 backup 387.2s unit 41133 MB RTO 591.8s discriminator correct 327x the data cost 14.5x the time - strongly sub-linear: RTO ~= 40s + 26.9 s/GB backup ~= 29s + 17.4 s/GB 10 GB -> 5.2 min 20 GB -> 9.6 min (measured 10.1) 100 GB -> 46 min The fixed ~40s dominates below ~1.5 GB, which IS the S band and explains its tight clustering. The more consequential result is capacity. A DB-backed app's recovery unit is 1.90x its data (volume tar PLUS SQL dump): 21.1 GB produced a 40.2 GB unit. The default appliance ships /mnt/sys_drive at 20 GB, so the largest app that can hold a local Tier-1/2 recovery unit on a default box is about 10 GB - and that fills the volume. The M band does not fit on a default box at all; this test only reached 21 GB because sys_drive was first grown 20G -> 70G with the same operation the product performs via SysDataGrowGB. A tier-sizing decision, not a defect, but it is invisible until an app crosses it. Caveats stated in the doc: two points define a line but do not test linearity; the 1.90x is DB-app-specific and a file-only app should be nearer 1.0x (inferred, not measured); synthetic incompressible data; one app, one box. |
||
|
admin
|
405a795e32 |
Campaign 10: RESOLVED — the backup_target_* silence was transient and self-recovered; SQLITE_BUSY drops are absorbed by retry
Both halves of the disposition were run and neither survived as a finding. The I1/I1-pair violations cluster at cycles 31-33 and nowhere else across 39 cycles; c34-c39 are clean, so it recovered with no intervention. Final tally I1 37 PASS / 2 VIOLATION, I1-pair 36 PASS / 3 VIOLATION. On the quiesced box one slow detach with 4 minutes either side produced a perfect pair. And the alarming false-healthy (mentes bound=False while degraded=false) does not survive quiescence - I had been reading the two halves at different instants of a detach. No R-n. Separately cleared: the hub's SQLITE_BUSY event drops. 7 in 24h including one for the real customer demo-felhom, and the hub does return 500 with notification dispatch only after a successful save - so a lost event would be a lost alarm. But the controller retries 3 times and ZERO events exhausted their attempts; the 07:04:39 drop landed at 07:04:42. Nothing lost. Only cosmetic note: the ERROR line reads like data loss and is not. |
||
|
admin
|
1931dfcb0c |
Campaign 10: R-157 second mechanism — the zero-container case, which is SILENT
The 4th hard-reset failure had a different signature, verified not assumed: all of rallly healthy, papra missing entirely with state=stopped deployed=True containers=0. Zero containers is exactly what bootrecon deliberately never touches, because the UI's Stop is compose down which removes containers - but a hard reset landing during a compose operation produces the identical state. The signature the safety rule depends on cannot distinguish the two. Worse: in that state the deadapp check reported 0 currently down while a deployed app was not running. No app_start_failed, no banner. That is the workspace's own false-invariant #4 (F-CRIT-1, StateStopped assumed deliberate) recurring through a hard reset rather than quiesce. NOT filed as new - CLAUDE.md already records it - but confirmed live on 0.188.0 via a new path. papra returned after ~15 min, later than the harness's 10-min window, so this instance was slow rather than permanent and the doc says so. What restarted it is not established. Mechanism A (Exited, missed by the unsettled snapshot) alarms; mechanism B (zero containers) is invisible on every channel. A settle-condition fix closes A only. |
||
|
admin
|
2d64ee7241 |
Campaign 10: OPEN observation — backup_target_* pair went silent under rapid cycling
Three I1/I1-pair violations in ~5 minutes, all "expected event absent". Recorded as an OPEN observation, NOT a finding: the system was mid-abuse when it was seen, and a verdict taken on a system being hammered is worth little. Established: it is not hub-side suppression and not a truncated log. The hub pod has 0 restarts over 43h and the controller's own log matches it line for line, so the events were never emitted. It is specific to the backup_target_* pair - the generic storage_disconnected/reconnected pair for the other drive kept firing normally throughout the same window. Also sampled, and the more serious half if it survives quiescence: mentes reads bound_under_parent=False while the backup-target state simultaneously reports degraded=false. Those cannot both be right - a false healthy on the backup target is I5/I6's failure mode. NOT established: whether the pair recovers once cycling stops (the harness detaches every ~2 min; a customer does not), whether the 02:25:37 controller restart is implicated, and whether the degraded=false sample was transient. Disposition written into the doc: after the run ends, quiesce with both drives attached, then do ONE slow detach/reattach and see whether the pair fires. That distinguishes "does not survive rapid cycling" from "the target alarm has silently stopped working", which would be severe. |
||
|
admin
|
3d4c5365c1 |
Campaign 10: correct R-157 — the failure is INTERMITTENT (3 of 6), not deterministic
The first write-up said R-157 reproduced "at the same cycle in both runs - deterministic, not a coincidence". Wrong. The cycle numbers matched only because the runner's RNG is seeded so both runs drew the same permutation. The failure itself is a coin flip: run 2b's four hard resets went PASS(c2), FAIL(c10), PASS(c18), FAIL(c26); run 2a went PASS(c2), FAIL(c10). Three failures in six. The correction matters because it changes what kind of bug this is, and it strengthens rather than weakens the root cause: intermittency is exactly what a race against container-state settling predicts, whereas a wrong predicate would fail every time. Signature is identical on all three occurrences: rallly Exited 255 with rallly-postgres healthy, bootrecon reporting "no boot-orphaned apps" about 5s after controller start, and the container count still churning after the sweep (third occurrence 01:05: refresh 8, bootrecon 01:05:13, then 8 -> 7 -> 8). |
||
|
admin
|
7f6b00375b |
Campaign 10: R-157 — bootrecon's start-once sweep misses the boot orphan it exists to recover
Reproduced twice, two independent runs, same cycle (the runner's RNG is seeded so both drew the same permutation - deterministic, not coincidence). A hard reset mid-backup brought everything back except the app half of the DB-backed stack: rallly left Exited 255, oom=false, restarts=0, its own log ending "Ready" - it died healthy - while rallly-postgres returned healthy. 20:28:13 Status refresh: 8 containers across 55 stacks <-- docker ps -a shows NINE 20:28:18 [bootrecon] Boot reconciliation: no boot-orphaned apps 20:28:25 Status refresh: 7 ... 8 containers <-- still churning AFTER the sweep 20:39:14 [deadapp] 20 scans, 5 deployed evaluated, 1 currently down The predicate is sound: once settled the controller reports rallly state=degraded containers=2, and IsDownState includes StateDegraded, so len>0 && IsDownState holds. The SNAPSHOT was wrong. bootrecon fires as a goroutine ~5s after start while docker is still restoring containers, and is start-once by design, so it never re-checks. Consequence: the app stays down indefinitely. Detection is perfect and recovery never happens - R-52's original shape, an alarm with no recovery. Not fixed. Distinguished from this campaign's two earlier HARNESS defects: both drives bound, every other app returned incl. the drive-backed one, only the app half of a two-container stack missing while its DB is healthy, and it surfaced through the fixed check written for exactly this. |
||
|
admin
|
80db2c103a |
Campaign 10: full write-up of the run-2a harness defects
The previous commit message was truncated by an unescaped paren in the shell, so the fix detail and the product observations were lost from the record. This adds them as evidence, where they belong. Covers: the cc_proof table showing no C010-A row at all (the seed never landed); both harness defects; why an ambiguous I7 justified stopping a 10-cycle run; the red-proofed controls; and two transient product observations recorded but NOT filed as findings - the health probe naming the DB container on the app's port for about 70s during recovery, and a ValidateDump WARN on a dump taken while the app was down. |
||
|
admin
|
9ca57e591b |
Campaign 10: two run-2a violations were HARNESS defects, not product defects — fixed
Run 2a hit its first two violations at cycle 10 and BOTH trace to my harness, not the product. Recorded in full because a check that fails for the wrong reason is as corrosive as one that passes for the wrong reason. HARD-RESET VM returned=True canaries_intact=False I7 want=C10-C010-A-194530 got=C10-C009-A-192929 restore_ok=True Root cause, evidenced: the cc_proof table's highest row is C10-C009-A — there is NO C010-A row at all, so the seed never landed. The hard-reset atom ran earlier in the same cycle and left rallly Exited(255); atom_restore_verify called seed() and never checked its return value, so an unwritten generation became a fake stale |
||
|
admin
|
ac6c05bd7b |
Campaign 10: add monotonic-growth sampling — the half the invariants cannot see
I1-I11 are CORRECTNESS invariants: they answer 'is the system telling the truth this cycle'. All 586 of them passed in run 1 while nothing at all watched whether disk usage, snapshot count, log volume, fd count or RSS climbs. Accumulation is exactly what depth was for, and it was missing from the invariant list. Adds c10growth.py (Campaign 2's controller_rss.tsv precedent, widened to 19 metrics) sampling every 90s as a SEPARATE process, so the in-flight run 2 did not have to be restarted. Attributes every sample to a cycle by reading the runner's status.txt, and records NA rather than dying when the box is down during a hard-reset or reboot atom. c10growth_report.py turns it into Campaign 2's table shape (start/end/min/max/ slope-per-cycle) and splits verdicts by class: growth in RSS/fd/volumes/images/ restarts is a LEAK; growth in backup storage or the qcow2 is expected accumulation, reported with a projection to cycle 45. Caught a bug in the sampler itself on the first analysis: MENTES_USED_MB appeared to jump 623 -> 5667 MB, which is exactly ROOT_USED_MB — when a drive is detached, /mnt/<name> reverts to a plain directory on root and df silently reports the ROOT filesystem. The same class of error as the agent's exactMount check, in the measurement code. Gated on mountpoint and red-proofed both ways: a real mount returns a number, a non-mount returns NA. |
||
|
admin
|
816c59c43a |
Campaign 10: R-117 Q7 (fs aborted in place, device present) proven PASS; extended atom set
The case R-117's spike called the worse half — a drive dying with no detach/return cycle, which before agent v0.117.0 emitted nothing on any channel indefinitely. Box runs 0.119.0. Aborted ext4 in place (abort,emergency_ro; device still present): bound_under_parent went false, storage_disconnected fired, the storage page named the stopped app, and calibre-web (whose library binds that drive) was STOPPED rather than restarted onto the dead namespace. Recovery needed a full device close, not a remount — exactly as the fix intends (BindAborted => quiet no-op). Runner extended with the 7 atom families run 1 skipped: abort-fs-in-place, kill-agent-mid-backup, hard-reset-VM-mid-write, reboot-VM, concurrent backup+restore, concurrent backup+detach, fill-drive-near-full. Also fixes a run-1 flaw recorded in the audit: reboot was appended AFTER the shuffle so it never interleaved with a detach; heavy atoms are now permuted in with the rest. Run-1 evidence preserved as *-run1.* (cycle numbering restarts per run). |
||
|
admin
|
69f896d3cd |
Campaign 10 Phase B: 27 cycles, 586 invariant checks, 0 violations
Ran the soak on the Phase A rig. Ended on its own deadline — no watchdog halt, no atom exception, no I11 breach. I1 28+28 pairs, I2 28+28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135, I11 28. Zero violations. The row counts are themselves the no-silent-skip check: I3/I4 twice per cycle (both drives), I10 = 5 secret-class fields x 27, REBOOT on cycles 7/14/21 only. I7 is the headline: 28 restores, 28 correct discriminators — never stale, never empty. RTO (Tier 1, rallly, 66 MB): min 38.8s, median 42.0s, p90 42.5s, max 44.3s. That is the S band's lower end ONLY; the 5.5s spread over 28 runs says fixed work dominates, so nothing extrapolates to M or L. RPO not measured. Every atom and invariant was proven BY HAND before automation — the runner asserts nothing that was not first observed live. Caught a Phase A gap before starting: no app had HDD_PATH, so all data sat on the system disk and I3 could never have fired. Deployed calibre-web onto adatok first; otherwise the run would have produced 27 green cycles that tested nothing cross-drive. Investigated and DISPROVED a suspected defect (audit 5.2): /api/disks reports state=attached for a physically absent drive, and intermediary.go:230 really does compute presence from State=="attached". It is inert — planDriveGates only gates paths under /mnt/felhom-drives/ and uses BoundUnderParent there, which was correctly false. The gate fired; the storage page showed "Meghajtó leválasztva". No R-n minted. Honest gaps: 6 of ~12 atom families ran. Not run — Tier 3 (structurally un-isolatable), abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms, fill-drive-near-full. I8 not checked, I9 not automated (cited from the tester-gate run, not re-claimed). kill_controller is NOT mid-backup and reboot_guest never interleaved with a detach. 27 cycles does not answer the brief's question about drift at the thirty-eighth. Teardown still OWED, including hub customer c10-soak (disposition: DELETE). |
||
|
admin
|
4691aa1a35 |
Campaign 10: Phase A complete + gated; Phase B not run; R-156 filed
Phase A passed every gate on a fresh box built from the PUBLISHED ISO 1.26.1: install, claim, two drives enrolled through the real endpoints with the backup target healthy, four apps spanning both sides of D5's secret split, and a working discriminator across all four. Isolation gate: both denials captured, each with a positive control. The PBS control FAILED first — four clean-looking 403s were worthless because the token was denied on its own datastore too (PBS token privilege separation). Fixed and re-run; the denials stand. R-156 (new, register grepped): papra's data is neither persisted nor backed up, and it reports healthy. The template mounts papra_data:/app/data; the app writes /app/app-data/db/db.sqlite. Volume empty and root-owned against a -rootless image, real DB in the container writable layer, healthcheck only probes the HTTP port. Its Tier-1/2 backup is real, verifiable and contains nothing. Not fixed. Tier 3 could not be isolated so it was not run: offsite hard-requires the DR tier (configs.go:1300) and the DR tier only provisions on ep0 (per-endpoint allocation deferred, hub/README.md:260). Both are recorded deliberate positions, so no R-n minted. The campaign touched neither ep0 nor the Storage Box. Phase B did not start. Phase A was budgeted at ~1h and took ~5.5h (1.26.1 is a public release image with no auto-install path, so the install was a blind screendump+sendkey walk). That left the runner — which judges eleven invariants and fires destructive atoms unattended — to be written at 04:00 with ~3h of night left. Stopped on the brief's own fence: a rig producing false negatives is worse than no rig. The rig is built and idle; teardown is OWED and itemised, including hub customer c10-soak (disposition: DELETE). |
||
|
admin
|
aa9c08f0dd |
docs(rehearsal): N100 S8 wrap — VALIDATION, map flips, ROADMAP R-30..R-39, R-3 draft
The 2026-07-18 N100 rehearsal ran the complete final-product flow on real metal
in one pass: RESET -> generic pairing ISO v1.20.0 -> customer self-bind -> day-0
-> managed-floor lift -> escrow ceremony -> offsite snapshots. No code changed;
every finding is recorded and ranked, none fixed.
VALIDATION-n100-rehearsal-2026-07-18.md — run context, a UTC-normalised timeline
built from the hub events stream / hub DB / controller log / bootstrap + agent
journals, per-ledger verdicts for S1-S8 + ledgers 8 and 9, 12 findings, the
not-exercised list, and 7 discrepancies against the brief.
Headline wall-clocks: bind -> credential 26 s; bind -> controller running the
current version 2 min 44 s; managed floor 0.143.0 -> 0.145.0 in 5 s unattended
(initiated_by: auto-floor); escrow ceremony -> offsite enabled 12 s; drive enrol
30.3 s. No post-bind leg stalled, which is the immediacy row's real-onboarding
proof.
Capability map (10 citations added):
- Bare-metal Felhom ISO PARTIAL -> PROVEN-LIVE (F1 closed on metal)
- Customer self-bind (slice 1) IMPLEMENTED -> PROVEN-LIVE (customer_selfbind)
- Guest RAM resize (R-24) IMPLEMENTED -> PROVEN-LIVE (shrink AND grow)
- Customer RESET two real firings + verified external teardown
- Escrow ceremony first live wizard firing
- Immediacy row "real-onboarding proof pending" cleared
- Publish train box-side floor lift proven on a fresh install
- Customer claim R-4 gmail half (Inbox under p=quarantine)
- Offsite orphan guard staged live leg fired on its own
- DR tier by default candidate PROVEN-LIVE upgrade WITHDRAWN (R-39)
Not flipped, as instructed: customer-performs-restore, BYO, DLNA, multi-user.
ROADMAP — collapsed R-1 (appliance half done, Peti half survives), R-21
(physically closed), R-24, R-27 slice 1, R-4. New ranked items:
P2-HIGH R-39 PBS DR applied-but-dead R-30 liveness from the wait channel
R-31 async offsite + status R-32 RESET base-dir purge
R-33 bootstrap quiet-poll
P2 R-34 backup lifecycle R-35 config-apply session survival
R-36 post-RESET offsite prompt R-27c console-passphrase bind
P3 R-37 post-RESET health card R-38 installer GRUB slice
Plus a pre-invite checklist (golden 0.145.x rebuild, freemail.hu, C6, R-11).
R-39 is NEW and was not on the brief: the PBS DR descriptor auto-provisions and
the agent converges state=applied, but pvesm reports 401 Unauthorized/inactive
and a direct probe 401s on every endpoint including /version while WG is healthy.
The hub minted a second token secret two minutes after the agent applied the
first and consumed_at is still NULL; the converged state machine will not
re-apply, and the agent's verify loop cannot read the credential to notice it
(non-root read of a file it writes through a root wrapper). Rank is provisional
pending Viktor.
R-3 draft: all four [REFINE] slots filled, self-bind made the default path with
"send the link BEFORE the customer sees the console", the measured wall-clock
table added, and interim operator workarounds for R-31/R-36/R-39. C6 (renumbered
C7) is marked as the single unexecuted step and keeps the doc a DRAFT.
Evidence bundle: 180:~/n100-rehearsal/ (10 files + MANIFEST.md), collected before
the box was unplugged for travel. Secrets read only to run probes; recorded as
lengths and metadata, never values.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
|
||
|
admin
|
3172df1927 |
scripts v1.18.0 — R-21 slice B: firmware loader option --loader shim|mkimage (F1)
Closes N100 F1 (HIGH): cheap AMI (AN3PLUS 0.01-class) UEFI firmware can't relocate the ISO's stock signed GRUB from USB (relocation 0x0). The run's live grub-mkimage workaround is now a first-class pipeline mode. - build-felhom-iso.sh: --loader shim|mkimage (default shim, byte-for-byte unchanged; profile-settable FELHOM_LOADER; --loader wins). Loud banner + manifest loader:/grub-mkimage: fields + -mkimage filename suffix. - mkimage-surgery.sh (new): post-prepare-iso, in the assistant container. Builds a monolithic grub-mkimage loader from the ISO's own GRUB (module set from its grub.cfg; embedded search --fs-uuid -> configfile the real menu). Swaps it into the ISO9660 tree (real lowercase path) + the efi.img ESP; xorriso re-master preserves BIOS-hybrid + UEFI + GPT-ESP, drops only Apple HFS+/APM. Recipe from the N100 run evidence, not re-derived. - Dockerfile.assistant: grub-common + grub-efi-amd64-bin + mtools + dosfstools. profiles/n100.profile (new, mkimage + SB-off note). - Validated on nested VM 311 (RUNBOOK-B legs): leg1 shim boots+installs under OVMF SB-enforcing + SeaBIOS; leg2 mkimage boots+installs under SB-off; leg3 (red-proof) mkimage under SB-enforcing FAILS Access Denied (unsigned -> SB must be OFF); leg4 surgery byte-identical payload. bash -n + shellcheck clean. Physical N100 closure folds into the rehearsal (n100-safety match-nothing ISO built + sha-recorded, unbooted). PXE stays a deferred R-21 note. |
||
|
admin
|
1c737db4f4 | docs: N100 F5/F6/F7 dispositions (fixed), capability map (fresh-USB wizard PROVEN-LIVE), ROADMAP (R-16 doc-drift closed, R-24 guest-resources-as-desired-state), iso README BIOS G3 note, REPORT/CONTEXT | ||
|
admin
|
7747a16ff1 |
feat(hub): v0.57.0 reinstall-of-existing-customer arc — claim/offsite/escrow continuity
F2 claim re-issue on clean-slate re-enroll (ReissueForReenroll, host-enroll mint path, single-bump, reset code; hub never stores the password so fork B). F3 offsite re-issue on re-enroll (ReissueOffsiteForCustomer, same machinery as the manual button). 2.3 escrow honesty (red-proofed): re-issuing offsite marks the escrow stale (MarkEscrowStale), withholds the mismatched restic hash from auto-confirm, DR checklist shows stale not done. Events: claim_reissued_reenroll / offsite_reissued / escrow_stale. Controller + scripts unchanged (source contradicted both premises): the controller reads escrow prereqs live from the agent; the installer can't know the descriptor-provisioned storage id. F4 root fix is agent-side -> ROADMAP R-22; demo unblocked live (Part 0 ACL grant). VALIDATION doc F2 erratum + F3/F4 dispositions. Green gate + Scenario-C red-proof pass. |
||
|
admin
|
dd961a66bb |
docs(tests): N100 bare-metal physical validation run (R-21) — findings + current state
Supervised clean-slate reinstall of the demo N100 from a pipeline-built Felhom ISO + onboarding rehearsal. Core objectives GREEN: first-boot chain reached rc-0 on the first attempt on real hardware (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict -> key on MAC+UUID. Hard obstacle was firmware not the pipeline: this cheap AMI AN3PLUS 0.01 board won't UEFI-boot the ISO's GRUB from USB (relocation 0x0; SB-off + shim-bypass don't help), worked around live with a grub-mkimage loader from the box's own GRUB (slice-B input). 7 findings logged (F1 firmware loader HIGH; F2/F3/F4 reused-customer edges -> R-1/Peti; F5 guest-RAM, F6 drive-init mount, F7 back-route). Full doc in documentation/tests/; ROADMAP R-21 + REPORT + CONTEXT updated. No production code changed. |
||
|
admin
|
1a1e42addb |
docs: campaign-2 R1/R2 root-cause correction + DR bind-mount source spike
Correction: campaign-2's "R1" (bind-mount restore needs root) was a SYMPTOM of an empty felhom pool -> agent lacked VM.Audit on 9201 -> restore-test's existing, correct bindMountOverrides never ran. Live-proven: pool healed (Part A) -> restore-test PASSES (neutralizes 2 binds, boot+running, clean teardown, 4m35s). Report rec #1 (make pct restore work on bind mounts) superseded. Fixes shipped: agent v0.74.0 (pool re-assertion), controller v0.103.0 (F-C2-1). Spike (SPIKE-dr-bindmount-source-2026-07-07): DR bring-up source options — PBS pct.conf.blob blocked by encryption under the non-root token; DR recipe lacks mpN; mp8/mp9 are STRUCTURAL CONSTANTS -> recommend a small known-constant override reusing bindMountOverrides. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 |
||
|
admin
|
7714f1f6e4 |
docs(tests): Campaign 2 report + evidence — DR primitives, self-update, storage, faults, endurance
Full-system validation on demo 9201 (controller v0.102.0 / agent v0.73.0). Headline: ONE genuine pilot-critical finding — the non-root agent cannot pct-restore the standard Felhom guest because it has bind-mount mountpoints (mp8 /mnt/felhom-drives, mp9 bootstrap) and pct restore refuses bind mounts unless root. This breaks BOTH --selftest=restore-test AND the offsite DR bring-up (R1), linked to a missing VM.Audit/VM.Allocate on the agent token (R2). The large raw FAIL count is dominated by one harness bug (auth-session expiry mis-scoring P3-P9) plus D1-injection/soak-probe/ dryrun-journal/escrow-blob calibration issues; the product where actually exercised was healthy with ZERO resource leaks over ~7h. Demo node healed to pre-campaign state (auth off, notifications restored, no scratch leaks). No production code changed. Evidence snapshot + README under campaign2-evidence-2026-07-07/ (cookie jars scrubbed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 |