Commit Graph

1038 Commits

Author SHA1 Message Date
admin b12f8ec2f3 deploy hub v0.98.0 — the R-241 superseded-package purge
gates / gates (push) Successful in 11s
The manifest is the truth: the image was built and pushed, but nothing
deploys until this tag moves and the app is synced.
2026-08-07 12:20:59 +02:00
admin 9657334fb7 R-241 FIXED: registers, capability map, STATUS, hub CHANGELOG v0.98.0
gates / gates (push) Successful in 14s
R-241 closed against controller v0.206.0 + hub v0.98.0, following the spike's
ruling rather than the obvious reading. The row records what the fix does AND
the two real bugs the tests caught rather than review - a missing t.Enabled
(caught by an EXISTING test) and a missing falling-edge sync that
reintroduced the very defect the epoch exists to fix.

R-243 UPDATED, not closed: the STATE it describes can no longer be entered
(the mint guard), and what replaces it is VISIBLE rather than silent - the box
declares awaiting_recovery_key and the customer is offered the screen. But the
ALARM GAP is untouched, for the same three reasons, so a box whose customer
never acts still stops backing up with no operator signal. The remaining work
is an operator-side signal for a box held past some age, deliberately not
bundled into R-241's fix.

R-245 NEW - WAITING-ON-OPERATOR, recorded and NOT built: should an undecided
customer be auto-abandoned after 30 days? The operator's proposal is recorded
WITH the reasoning against it, so the decision can be revisited properly: a
reinstall implies a person, so nobody is absent; a customer who cannot find
their code gets in touch, which is why the operator LEVERS were the thing
worth building; the cost is the customer's own storage allowance; and the real
harm is QUOTA, which is a condition, not a calendar. If it is ever built,
build it to trigger on the harm with a dated warning, never on a date alone.

The capability map's recovery-journey row STAYS FAIL. These are fixes, not a
walk - nothing here walked a customer end to end, and the row goes green only
when one completes with no operator intervention AND a byte-identical
sentinel. R-214, R-202 and R-240 are still open.

STATUS compressed rather than extended, per its own one-screen rule, and the
"rebuilding throws away the off-site history" line corrected: the cause is
fixed, so leaving it as a live defect would be false.

hub CHANGELOG v0.98.0 for the superseded-package purge.

Highest register ID moves R-244 -> R-245.
2026-08-07 12:13:09 +02:00
admin ac4b2a4ba9 hub: drop the retained recovery package when the box declares its set-aside history deleted (R-241)
The hub half of the controller's abandonment countdown, and the ONLY reason
felhom.eu was touched for R-241 at all.

A customer who abandons their old off-site history gets a 14-day countdown. At
the end of it the controller deletes the set-aside restic store and then
DECLARES offsite.abandon_purge_requested in its report until the retained
sealed package that protected that store is gone too. Removing only one half
leaves a state that asks a question nobody can answer: a package that opens
nothing, or ciphertext nobody can ever decrypt.

store.PurgeSupersededEscrowForCustomer is the one place R-198's retention is
ever undone, and its doc comment says why that is legitimate here. It NEVER
touches host_escrow - the current package covers the key the box is using now
and is what makes its live backups recoverable. Only host_escrow_superseded
rows go.

The handler acts on the box's DECLARATION, never an inference, on the same
principle as offsite.state: the hub cannot see that a remote store was deleted
and the box can.

It is placed immediately BEFORE the ACK is built, deliberately.
GetEscrowStatusForCustomer is read after it runs, so the SAME response that
carries the request's effect is what closes the box's two-phase commit - no
second round-trip, and no window in which the box believes it is still owed.
The declaration repeats on every report until that ACK stops reporting a
superseded package, so a lost request retries by itself.

A purge failure is logged at ERROR and never swallowed: the box keeps
declaring, so it retries, but an operator must be able to see that the two
halves are apart right now. An idempotent re-declaration (already purged, the
box has not yet seen the confirming ACK) logs at DEBUG and is not an error.

Audit event offsite_abandon_purged is hub-internal, like the pbsdr_* and
offsite_selfheal_* events - allowedEventTypes governs the box-pushed
POST /event surface, not this.

No agent change. No deletion has been performed against any real store.

Green: go build, go vet, go test ./... all clean in hub/; repo gates OK.
2026-08-07 11:48:10 +02:00
admin 094e93e828 finalwalk teardown complete; R-244 filed; session report
gates / gates (push) Successful in 13s
All five layers gone, each verified with a positive control that must
survive and does:
  VM 324 + 4 disks   -> absent (VM 300 drill-r50 remains)
  hub: 13 tables at 0, incl. BOTH escrow tables (demo-felhom/demo-hp/peti remain)
  Storage Box u629488-sub4 -> gone (sub1/2/3 remain)
  ep0 PBS ns finalwalk     -> gone (demo-felhom, demo-hp remain)
  WireGuard 10.77.0.5      -> gone from the LIVE wg show on ep0, not just
                              the hub DB (.2/.3/.4/.250 remain)
14.06 GiB reclaimed against 15 G measured before deletion.

R shredded with a planted-copy control: plant -> search finds both ->
shred -> the same search finds 0. The zero was not believed until the
instrument was proven.

R-244 (NEW): a FULL census after the cascade logged COMPLETE full teardown
found 61 rows still matching finalwalk. Four sources are deliberate
provenance; the fifth, app_log_issues (29 rows), is NOT covered by the
residue purge - and it is systematic: c11 40, rewalk 20, part4 24 still
present from the 2026-08-06 teardown, whose ledger recorded zero
occurrences. That claim used a narrower query than a census and does not
hold; the correction is recorded in both the prior ledger and the register
rather than the measurement quietly redone.

No secret material is involved. The table is a fleet-wide aggregate: 12 of
the 29 rows are finalwalk-only orphans, 17 are shared with LIVE customers
and must be de-referenced, not deleted - very likely why the leg was never
written. Not fixed; a cascade change needs its own red-proof.

Lesson, and it is the reusable part: a per-table absence query is not a
census.
2026-08-07 09:59:16 +02:00
admin db578cd44d R-239 CLOSED (golden 0.205.0 vouched); R-241 ruling into the map and STATUS
gates / gates (push) Successful in 14s
R-239: the operator approved the vouch this session. golden_version
0.203.0 -> 0.205.0 (+ derived sha); agent_version and min_agent both stayed
0.127.0, because the new golden's MinAgent is also 0.127.0 - so in the event
it was a ONE-field change, not three. wrapper_sha256 was carried through
explicitly: the handler reads it from the form and CLEARS it when omitted.
Verified from the stored hub_settings (WAL-aware copy), not from the flash.
The R-120 gate passed exactly - the newest controller the fleet reports is
0.205.0, so a 0.204.0 golden would have been refused.

R-241: the capability map's recovery-journey row and STATUS carry the spike's
ruling - a MINTING defect, not a screen-predicate defect. The row stays FAIL:
delivery is not a journey, and R-241 is diagnosed, not fixed.

R-242 and R-243 surfaced in STATUS in plain language.
2026-08-07 09:27:22 +02:00
admin 08b75e602e golden 0.205.0 baked, published and round-trip verified — NOT vouched (R-239)
gates / gates (push) Successful in 13s
Closes the delivery gap's build half. The vouched golden carried controller
0.203.0 while 0.205.0 was released, so a machine installed last night got
neither R-237 (restore list keyed on the store) nor R-234 (skipped-app
verdict). Both were measured from the customer's side on that box.

  version  0.205.0
  sha256   8f49b2e8ccbc86a49df821fee9fb00c07293758811d3d0f0512dd0cf5fd54ee8
  size     656,937,561 bytes (uncompressed 2,003,343,360)
  MinAgent 0.127.0

Round-trip verified rather than trusted: the published bytes were fetched
back, re-hashed (match), zstd-tested, and ./etc/felhom-controller-image was
read OUT of the downloaded archive -> felhom-controller:0.205.0. That last
step is the one that matters, because GOLDEN_VERSION is derived from the tag
argument and could have been right over stale content.

Acceptance markers all pass; unit Result=success ExecMainStatus=0.
Secret hygiene: token file->file, in-VM runner so it never reached a command
line (unit-property grep 0), literal-value leak grep on the COMMITTED log 0 -
with a positive control proving the grep works before the 0 was believed.

Bake VM torn down: CT 9100 purged, secrets shredded, qemu observed gone via
ps -eo comm, drill.qcow2 reverted to virgin.

NOT VOUCHED - that is the operator's act. Only ONE field actually moves:
golden_version 0.203.0 -> 0.205.0 (+ its derived sha). agent_version and
min_agent both stay 0.127.0, because the new golden's MinAgent is also
0.127.0. The R-120 gate passes exactly: the newest controller the fleet
reports is 0.205.0, so a 0.204.0 golden would have been refused.
2026-08-07 09:05:59 +02:00
admin 71c43f87c2 SPIKE R-241 — ruled: a MINTING defect, not a screen-predicate defect
gates / gates (push) Successful in 6s
Read-only spike on the standing venue (demo-hp VM 324 finalwalk-appliance),
read before anything else touched the fleet. No product code written.

THE RULING. The recovery screen was telling the truth. There genuinely was
nothing recoverable under the key the box held, because the box minted that
key itself over the top of a sealed package it already knew the hub held.
Fixing the predicate would paper over a box quietly making its own history
unopenable.

Three measurements carry it:
  1. WriteOffboxSecrets (offbox.go:411) mints on ONE input - does the file
     exist. Its two neighbours in the same file both consult
     GetHubEscrowIdentityPresent(). Same fact on three paths, used on two.
  2. The flag was the PRECONDITION of the chain that reached the minting:
     the retry job only logs when the declaration is live, and the venue
     logged it at 02:48:03Z - 30 minutes and six ticks before the mint.
  3. The box KNEW and discarded it: at 03:28:03Z, 35 minutes before the
     customer looked, escrow_confirm.go:154 computed the exact
     discriminator and logged the mismatch. Never persisted, never surfaced.

And the hub explicitly disclaims doing this - offsiteheal's package doc:
'credential automatic, key customer-present ... must not quietly widen'.

Also answered: Q2 (shape (b) is structurally unreachable - the escrow gate
sits upstream of the only orphan producer, with a positive control that the
scheduler was alive); Q7 (the 'create a new code' button does NOT destroy
the data - R-198's retention holds - but it converts a self-service recovery
into one needing an unbuilt read path, and it re-enables the screen while
invalidating the code that screen accepts).

New rows:
  R-242 - a release that changes customer-visible behaviour is not delivered
          until a golden carries it, and nothing enforces that. Recorded
          with three proposed shapes; deliberately NOT built.
  R-243 - a box in this state silently stops backing up and NO alarm fires:
          three individually-correct exclusions leave one state unobserved.

Highest register ID moves R-241 -> R-243.
2026-08-07 08:53:16 +02:00
admin 2228c0bff6 final walk COMPLETE — data PASS, journey FAIL; R-241 filed
gates / gates (push) Successful in 15s
THE DATA: PASS. All three sentinels byte-identical out of snapshot f5c53b03,
including the 12 MB binary and the accented Hungarian filename whose NAME BYTES
are identical too. Disk -> restic -> SFTP -> Storage Box -> rebuilt machine ->
disk, intact.

THE JOURNEY: FAIL, and further from the line than the previous walk. The claim
worked first try (302 in 0.164s). Then: / lands on the launcher with no recovery
pointer, /recovery 302s away, and the remote page offers to CREATE a new recovery
code — which would orphan the history the customer's code protects. There is no
field anywhere to enter the code they hold. The operator's documented remedy also
refuses, correctly and fail-closed. Recovery needed three guest command lines.

R-241 — and the cause is a success this same walk proved six hours earlier.
OffsiteRecoveryOffer() shows the screen only when (a) there is NO repository
password (pristine rebuild) or (b) one exists but the history will not open under
it. Overnight the credential self-heal collected the staged credential and applied
the tier, writing a FRESH key at 03:18Z — so (a) is false; and (b) is unreachable
because orphan detection needs a run, and runs are blocked by escrow_state=pending.
The gap is self-locking. Measured keys: on-disk 9b4a9a9d... vs recovered-from-R
30ef574f... This is R-218's shape one level up: succeeding at the self-heal stopped
the box OFFERING the recovery it still needed.

Registers: R-201 moved to its outcome; R-241 filed; capability map's recovery row
stays FAIL with both halves and the cause named; STATUS rewritten for the operator.
Highest ID R-238 -> R-241.

The venue is left with the recovered key in place and the self-heal key moved
aside, never deleted. Teardown still owed.
2026-08-07 06:45:36 +02:00
admin 1ff6f8f8e0 final walk §7: the credential chain runs end to end, unaided, on an UNCLAIMED box
gates / gates (push) Successful in 12s
Three questions answered from the hub's own log, not inferred:
  1. the rebuilt, still-unclaimed box DOES report (host-report + Received report)
  2. it DOES declare offsite.state=needs_credential, and offsite-delivery correctly
     declines once a minute, naming internal/offsiteheal as the owner
  3. offsiteheal re-staged UNAIDED at 03:15Z, after two reports carried the
     declaration, with no provider credential minted — about 32 minutes after the
     rebuild, matching the documented 2x15-minute debounce

And then the box COLLECTED it on its own 5-minute tick:
  [offsite-apply] credential retry: the staged credential was collected and the
  tier applied

That success line shipped in v0.203.0 and this is the FIRST time it has been seen
live: yesterday's walk only produced its sibling before I intervened at 102s and
mistook my own button press for the cause — the error that produced R-236 and
forced its withdrawal. Here nobody touched anything and the box was not even
claimed. R-218's consume half, R-236's withdrawal and the previous walk's dead
end 1 are all settled by one unattended observation.
2026-08-07 05:23:03 +02:00
admin 1b490c8cbf final walk: destroyed, rebuilt, and HALTED at the claim screen
gates / gates (push) Successful in 27s
Destroyed 02:40:31Z (guarded on hostname — demo-hp also has a guest 9201), drives
wiped to 20K with the mounts deliberately left in place because the surviving raw
mount IS the R-220 condition. Reinstalled through the published day-0 path,
installer v1.25.0 fetched live; Day-0 provision SUCCESS in 2m32s.

R-239 measured a second time, from the other side: the rebuild landed on agent
0.127.0 (no downgrade, no hand upgrade — that half is right) and controller
0.203.0. The box a customer would recover on tonight also lacks R-234 and R-237.

The machine is AT THE CLAIM SCREEN awaiting the operator. A claim code has already
been requested through the customer-facing path and emailed, so the morning is
paste-a-code rather than request-then-paste. The reset-code hatch was NOT used and
will not be: it is a guest command line and would fail the rule the walk measures.

Stated plainly in the journal: journey steps from the destruction onward are driven
over HTTP from the appliance to the guest's island address, as a browser would;
some instrumentation reads are guest command lines and are counted as such, but
none changed state or was needed to progress the journey.
2026-08-07 04:46:11 +02:00
admin 502078bebf final walk: the soak DID catch a scheduled cycle; both directions recorded
gates / gates (push) Successful in 20s
Window 2 (01:56-02:36Z) caught the off-site nightly at 02:15:03Z — snaps 1->2,
last_run advanced with no prompting. Window 1 had already caught the local legs
(db-dump 00:30:25Z, tier + fill-watch 01:30:19Z). My 04:15 prediction was wrong in
the other direction; recorded rather than quietly fixed.

What should have fired and did not: NOTHING. Six registered jobs were never seen
in the log and none is a finding — four are quiet by construction
(scheduler.go:267, quiet := job.Interval <= 30s) and two run every 6h, outside a
4h window. Checked the source rather than filing four phantom defects.

What fired and should not have: nothing. No alert, notification, email or digest.
The only WARN lines in five hours were three of mine.

Observation kept: offbox-backup ticked twice 62s apart and produced ONE new
snapshot — the second silently dropped by the single-flight, which is correct for
the nightly path and is the same mechanism that produced R-234 on the manual path.
2026-08-07 04:39:42 +02:00
admin f873c557ca final walk: the soak, and the window correction stated as mine
gates / gates (push) Successful in 20s
Window 1 (22:44-01:54Z, untouched): every periodic job fired at its declared
cadence — agent-channel-health 185x/1m, stack-scan 108x/2m, the 5-minute trio 44x
each, hub-report 14x/15m. offsite-credential-retry ran 44 times and did no work
and said nothing, which is R-218's asserted healthy-box behaviour on a box that
never saw the defect. No alert, notification or digest fired; nothing on the
must-not list fired.

The off-site state never moved during that window — and that is NOT a finding.
The daily jobs run on the controller's clock and the guest is UTC while the
appliance is CEST, so the nightly local (~02:30) and off-site (~04:15) fall at
02:30Z and 04:15Z. I sized the window against CEST and it closed at 01:54Z, before
either. Reporting 'the nightly did not fire' would have been a false finding
produced by a badly-chosen window.

Window 2 (corrective, 01:56-02:35Z) covers the 02:30Z local backup. The 04:15Z
off-site nightly is deliberately NOT covered and is recorded as not run with the
reason: leaving the machine at the claim screen before 07:00 is the primary
deliverable, and waiting would have put the destruction at ~06:35 CEST with no
margin. The off-site tier was exercised four times manually instead, including a
full listing by name.
2026-08-07 03:57:18 +02:00
admin 2d2d8d3ddc final walk Phase A + the five checks; R-239 (delivery gap) and R-240 filed
gates / gates (push) Failing after 13m22s
Phase A complete on a NEW venue (demo-hp VM 324, customer finalwalk). Installed
from the published ISO — verified byte-identical to iso.felhom.eu — claimed, app
deployed, three sentinels placed, escrow ceremony 6/6 with restic_pw_sealed=true,
R held in one 0600 file on DooPlex, off-site backup taken and the three sentinels
PROVEN IN THE SNAPSHOT BY NAME (snapshot 1da4f80d, 12.0 MB).

THE HEADLINE: a fresh install lands on agent 0.127.0 (vouched, no hand upgrade —
that half works) and controller 0.203.0, while the newest is 0.205.0. The vouched
golden still bakes 0.203.0, so tonight's box has neither R-237 nor R-234. Filed as
R-239: a delivery gap, not a regression. Every closure claimed for those two is
true of the code and false of the fleet until a golden is baked and vouched.

The five checks, on that box:
  T1 ok + the old bare-count warning (no next step)
  T2 FAILS — both runs say "elindult", the second as a success flash
  T3 FAILS — toggle off hides an existing snapshot; wizard 302s away
  T4 ok, and the wording is still "Sikeres" beside "nothing is selected" -> R-240
  T5 PASSES — driven as a browser does, prepare -> full_prep -> confirm -> commit
     completed; R-238 confirmed a harness artifact, not a product defect

HARNESS FAULT, caught by the §4 gate: the first off-site run reported ok with a
26.6 KB repo — impossible for a 12 MB sentinel. I had placed the files under
.../adatok/felhom-data/userdata/... while this box's namespace root is
/mnt/felhom-drives/adatok directly. The product was correct throughout; proving by
listing rather than by a green status is what caught it before any destruction.

Nothing destroyed. The venue stands.
2026-08-07 00:42:59 +02:00
admin feed748325 R-234 root-caused and CLOSED; R-218's state field corrected
gates / gates (push) Successful in 18s
R-234 was filed as "toggling an app on leaves it without a bundle, so the first run
skips it". Measured on demo-hp: that state does not survive a run — the off-site
run's own pre-dump phase calls captureAllRecoveryUnits for every DEPLOYED stack,
through admitApp, before the push, and a unit moved aside was RECREATED.

The actual cause was the single-flight: the manual run was dropped because an
earlier one was still going, runOffboxBackup returned nil, the handler had already
answered "A tavoli mentes elindult", and the card then showed the PREVIOUS run's
green verdict. Fixed in controller v0.205.0 and proven live on demo-hp: a second
request while one is in flight now says "Mar fut egy tavoli mentes — ez a keres nem
inditott ujat. A most lathato eredmeny meg a korabbi futase", as a flash_error.

Independently, and a real gap on its own: a run that skipped an app the customer
selected is now `incomplete`, not `ok`. Selected+deployed with no unit counts;
selected-but-undeployed is named with what to do but does NOT count, because a box
left amber by an app somebody removed is a status nobody reads.

R-218's state field read REOPENED while the same row's body already recorded the
fix shipped in v0.203.0 and proven live. Corrected to CLOSED, keeping the
over-claim history — it is why the row is worded as it is.

Capability map: the off-site capture row's `incomplete` sentence widened to cover a
whole-app skip, and it still does not claim a newly-selected app is protected by
the next run — for a deployed app it is, for an undeployed one the card says so.

Still open, deliberately: R-213, R-202, R-214, R-235.
2026-08-06 22:03:58 +02:00
admin fca85498e1 Part 5: the last venue torn down after its findings were written
gates / gates (push) Successful in 8s
part4 (VM 323) was kept while Parts 1 and 3 read it — it was standing in the exact
failed half-B state and tearing it down first would have destroyed the evidence.
Destroyed after the diagnoses were committed (70b6260): VM, hub host, hub customer,
u629488-sub6, ep0 namespace, wg peer 10.77.0.7, all verified absent with demo-hp as
the positive control.

Final: demo-hp holds only VM 300 (drill-r50) and its own CT 9201; ep0 holds only
demo-felhom and demo-hp. 53.4 GB reclaimed across the three venues.

The cascade purges all escrow ciphertext, so R_C11_REWALK and R_PART4 in the
operator's credentials file are now inert.
2026-08-06 17:25:20 +02:00
admin 190c432f3a capability map: the recovery journey stays FAIL, with the two dead ends closed and vouched
gates / gates (push) Successful in 8s
R-218 and R-220 shipped (controller v0.203.0 / agent v0.127.0) and are proven live
on a genuinely rebuilt box; golden 0.203.0 + agent 0.127.0 + min_agent 0.127.0 are
vouched, so the delivery gap the re-walk recorded is gone. The row stays FAIL
because the walk did not finish: it stopped at R-237 (the restore list was keyed
on installed-and-toggled apps), now fixed in v0.204.0 and proven live — but no
sentinel was restored, so the data half is unproven in either direction for that
venue. R-238 reclassified (harness artifact, real residue fixed); R-236 withdrawn.
2026-08-06 17:02:55 +02:00
admin 70b62608b0 teardown of two venues; R-218/R-220 CLOSED, R-236 WITHDRAWN, R-238 reclassified
gates / gates (push) Successful in 18s
Part 0 — c11 and rewalk destroyed, three layers each plus the off-site side and
the WireGuard peer, via the hub's own cascade (external teardown FIRST, DB purge
LAST). 37.3 GB reclaimed on c11-scratch, matching the 20G+17G measured. Positive
control after each: part4 must still be found, and was. ep0 namespaces now exactly
demo-felhom, demo-hp, part4.

The cascade refuses to delete a live host and there is no decommission endpoint,
so both boxes were stopped and aged past the hub's 30m stale_threshold first.

Register corrections — the durable record was wrong about two shipped fixes:
  R-218 REOPENED -> CLOSED. Shipped controller v0.203.0, proven live on a
        genuinely rebuilt box: the hub re-staged at 13:24:57Z and the box
        collected it on a tick, no guest command line, no operator action.
  R-220 "OPEN — NOT FIXED" -> CLOSED. Shipped agent v0.127.0, proven live after a
        real guest purge with both raw mounts still on the surviving host:
        /disks/candidates returned both drives (before: two empty lists) and both
        re-attached through the customer endpoint.

R-236 WITHDRAWN — I FILED THIS WRONGLY. The hub log shows offsiteheal re-staged
the stored secret at 13:24:57Z after its documented two-report debounce, with no
provider credential minted. My Re-issue at 13:26:39Z came 102s LATER, was
redundant, and minted an unnecessary provider credential (subaccount 284735) —
the very double-issue the offsite-delivery guard warns about once a minute in the
log. "Nincs teendod" is true; I did not wait ~16 minutes. Operational lesson, not
a product defect.

R-238 reclassified as a harness artifact (mode=full without confirm=1 is step 1 of
a deliberate two-step and starts no job by design); its real residue — the total
silence of that step — is fixed in controller v0.204.0.

R-237 CLOSED by controller v0.204.0.
2026-08-06 16:53:38 +02:00
admin 5810e1f552 Part 4 half 2: upgrade path PASSES, recovery journey dead-ends (R-236/237/238)
gates / gates (push) Successful in 8s
Destroyed guest 9201 (guarded on hostname — demo-hp also has a 9201), wiped both
drives leaving the raw /mnt/<name> mounts in place, reinstalled from the
published installer v1.25.0.

PASS — the upgrade path:
  agent 0.127.0 -> 0.127.0 across the rebuild. The previous re-walk DOWNGRADED
  0.126.0 -> 0.125.0 (R-216 part 4). Controller 0.203.0 from the baked golden,
  Day-0 provision SUCCESS in ~2m50s, nothing applied by hand.

R-220 FIXED, proven on a genuinely rebuilt box: with both devices mounted raw
AND managed, /disks/candidates offered both drives (before the fix: two empty
lists), and both re-attached through the customer endpoint.

R-218's consume half proven live end to end: credential re-staged at 13:26:32Z,
collected on a tick, tier configured — no guest command line. Previously this
exact situation was dead end 1 and needed a guest shell.

FAIL — the data half, NOT reached. Three new findings, in the order they bite:
  R-236 nothing re-stages the off-site one-time password after a rebuild, so the
        "Nincs teendod" promise stalls until an operator re-issues. The retry job
        is NOT at fault — it retries correctly and logs why.
  R-237 the restore surface is keyed on installed + remote-marked apps, so after
        a rebuild the customer is shown no backups at all — a circular dead end.
  R-238 "Teljes visszaallitas elokeszitese" returns 302 and does NOTHING: no job,
        no state change, no error, no line in the controller's own debug ring.
        mode=unit on the same form works. This is why the sentinel byte-identity
        criterion was not met — nothing came back, rather than coming back wrong.

No guest command line was used at any point in either half — the property the
previous walk could not claim.
2026-08-06 16:03:04 +02:00
admin ac451f9667 Part 4 half 1: fresh install lands on vouched 0.127.0/0.203.0 with no hand upgrade
gates / gates (push) Successful in 9s
Venue: demo-hp VM 323 part4-appliance / guest 9201 / hub customer part4.
Bind 13:53:40 -> agent 0.127.0 ONLINE 13:55:34, no "floor held", controller
0.203.0 from the baked golden. The previous re-walk's reinstall DOWNGRADED the
agent (R-216 part 4); this one did not, because the vouch moved all three fields
(golden_version + agent_version + min_agent) together.

R-218 on a real installation: offsite-credential-retry registered (every 5m0s)
and ticking, 0 "credential retry" lines, and the tier configured unaided 14s
after the settle gate — the previous walk's dead end 1 is gone. Stated precisely
in the journal: this venue proves the SILENT half only; the collecting half was
proven separately with a positive control.

R-225 visible in both states on a live box: "még ismeretlen" before any
snapshot, "12.0 MB / 50 GB" after. Unknown is not zero.

Fixture complete with NO guest command line — the claim used the emailed code
through the real /claim form (the hub keeps only a hash), and the sentinels were
written over SMB with the share mounted so the accented filename never crossed a
shell (verified as hex on both sides, no U+FFFD).

Two new findings, both "presence is not success":
  R-234 an offsite run reports success while omitting an app just toggled on —
        found ONLY because the pre-destruction verification restore was run
        instead of trusting the green tick.
  R-235 the console still shows the pairing banner 25 min after the bind, while
        promising it refreshes itself.

Half 2 (destroy -> reinstall -> recover -> compare) is owed and NOT started; the
venue is standing and its teardown is owed.
2026-08-06 14:56:14 +02:00
admin 1c47e3b6fd golden 0.203.0 baked + published; runbook acceptance markers fixed (R-233)
gates / gates (push) Successful in 13s
Bake evidence: documentation/tests/golden-0.203.0-2026-08-06/ (bake.log + README).
sha256 3039c6ffa7a5a8b2d959daddb2895c58b44de70f8d4f4a7e12ad4b1c0d61dc88, verified
by an independent round-trip download and by reading /etc/felhom-controller-image
out of the published archive itself. NOT vouched — the hub still serves 0.201.0.

R-233: RUNBOOK-manual-build.md §4.1 named three pass markers, two of which the
script cannot print (`overlay2 OK` does not exist; `mp1` stopped existing in
build-golden.sh v3.0.0 under R-165), and a 404 pre-gate URL with the wrong
filename, which would 404 for the wrong reason and pass even when the version
already existed. A grep for an impossible string reads 0 forever and 0 is
indistinguishable from failure. Markers re-captured from the real log; token
handling moved off the command line into an in-VM runner script; a positive
control is now required on the token-leak grep; the vouch step rewritten as the
three-field change it is (golden_version + agent_version + min_agent).
2026-08-06 13:23:22 +02:00
admin 0c4411e54b R-201 re-walk: the data PASSES again, the journey still FAILS — two dead ends, down from four
gates / gates (push) Successful in 9s
Asked Campaign 11 Phase 1's question a second time, on the fixed build, on a
NEW appliance (VM 322, customer rewalk). The Campaign 11 venue was untouched.

THE DATA: PASS. All three sentinels byte-identical out of the pre-destruction
snapshot a7bc23bd in 23s through the customer's own restore flow — including a
12 MB binary and an accented Hungarian filename whose NAME BYTES are identical
too (verified as hex, not as rendered text).

THE JOURNEY: FAIL, two dead ends against Phase 1's four.
 1. R-218's CONSUME half. The hub re-staged the credential at 11:44:57 saying
    'the box re-consumes on its next cycle'; a full cycle ran at 11:55:46/54
    (with a positive control that it ran) and it did not. A census of the
    customer-reachable actions found none that fetches it. Only a command line
    INSIDE THE GUEST moved it — 18s, confirming nothing was wrong with the
    credential, target or key: only the trigger. R-218's row said SHIPPED and
    over-claimed; it is corrected to REOPENED for the consume half.
 2. R-220. Drives still unenrollable after a rebuild, needing a Proxmox-host
    unmount; without it no app redeploys and the restore page stays empty.

Unaided RTO STILL UNDEFINED. Attended: +45s key placed, +24m12s tier up,
+30m13s data verified. The 30m must not be quoted as the customer number.

What passed and is new: the recovery screen appeared WITHOUT being sought,
answered all three questions with a seal date matching the hub exactly, the
emailed reset code worked first try, the unlock was a real 1.528s unseal, and
R-225's fix was seen working in the wild (unknown, not a false zero).

R-216 part 4 reproduced live: the reinstall downgraded the hand-installed agent
0.126.0 -> 0.125.0.

DELIVERY GAP recorded as owed and NOT conflated with the journey: a fresh
install landed on controller 0.201.0 / agent 0.125.0 — the vouched versions,
neither carrying the fixes — installed by hand. Nothing was vouched.

Capability map row STAYS FAIL. Campaign 11 doc gets a dated ADDENDUM, not a
rewrite.
2026-08-06 12:18:29 +02:00
admin a1a542b9a7 RECON: what DooPlex's own backup actually protects (R-232) — read-only, nothing changed
gates / gates (push) Successful in 9s
Five sets, 14/14 successful runs in 14 days, and a file restored byte-identical from the data repo
-- the first time a restore has ever been demonstrated here.

What the failure matrix exposes: notify_failure is a no-op (NOTIFY_ON_FAILURE=true, webhook
commented out) so a failed backup tells nobody; NOTHING leaves the box (Longhorn's target is
DooPlex itself over NFS, the only outbound-looking cron pulls inbound for another project); the
backup tree is a single writable path with no append-only; two sets are same-disk (.claude-memory
and the postgres dumps, whose source sits inside the backup tree); Longhorn retains 1 generation;
secrets/restic-repo has never held a snapshot because backup-secrets.sh never calls restic; and
/opt/backup/docs/BACKUP-RESTORE.md does not exist though the unit advertises it.

NOT a finding: the restic passphrase. On-box copy is on a different disk from the backups and the
operator holds an offline copy out of band, so disk loss is recoverable. The residual is only that
it is operator-held rather than system-held escrow.

A stale lock in the data repo is MINE, from the previous task's verification. Fenced from clearing
it; recorded with its impact and the one command that fixes it.
2026-08-06 12:16:38 +02:00
admin b1bf1f271c REPORT + STATUS: instruction arc close-out
gates / gates (push) Successful in 8s
Operator line for the one change with real-world consequence: the runbook that says which machine
may be wrecked claimed the HP box keeps no off-site copy, and it keeps two. Checked against the
machine, not corrected from memory.
2026-08-06 11:50:28 +02:00
admin 5ca5082e7c docs: close out the instruction arc — t740 corrected on evidence, registers, ledger, S-37
gates / gates (push) Successful in 8s
target-selection.md said demo-hp has no off-site tier. Measured first: pvesm list felhom-pbs on the
box returns two snapshots in demo-hp's OWN namespace (2026-07-28, 2026-08-04) against ep0's
felhom-offsite. The claim was TRUE WHEN WRITTEN and went stale when F10 resolved 2026-07-23. The
measurement is kept in an HTML comment beside the corrected sentence. This file decides which
machine may be destroyed, so the sentence was load-bearing, not cosmetic.

R-229(b) CLOSED (agent 175 -> 99 eff). R-230(b) CLOSED (symlink, proven from fresh sessions).
R-230(a) part-actioned -- three false statements fixed, WARN loop added, bulk ruling still owed.

S-37: a claim in an instruction file is checked, not trusted.
2026-08-06 11:49:22 +02:00
admin f49b1f390b workspace: the root CLAUDE.md becomes a symlink; check 5 learns two shapes (R-230(b))
gates / gates (push) Successful in 7s
Two files kept identical by hand and one check is a divergence class policed forever; one file
reachable by two paths cannot diverge at all. install_workspace.py now links by default, MIGRATES
an existing regular file (backing it up first and SAYING SO if it differed -- that difference is
the last chance to notice an unsynced edit), and keeps --copy for a clone that wants the old shape.

Check 5 asserts a different thing per shape: for a link, that it points at the versioned copy and
resolves to a real file; for two files, byte-identity as before. A dangling link is worse than a
diverged copy -- the instructions load NOTHING and there is no content left to notice is wrong --
so that case is red-proofed.

NOT yet proven to LOAD: that needs a fresh session and a hook line, which is Phase 7. If it does
not load, this reverts to the copy.
2026-08-06 11:41:36 +02:00
admin 15fa5273ba gate: check 7 (register citations) + content WARNings on the memory index
gates / gates (push) Successful in 8s
Check 7 catches "cites a register item and calls it open when it is not" -- the R-168 class, four
files, one self-contradicting. Trigger is an openness CLAIM, not any citation: policing every
mention would fire on ~30 legitimate provenance citations and the gate would be switched off.
Deliberate deviation from the task's literal wording, to keep it alive.

Two bugs found by the check's own red-proofs, both of which would have shipped:
- the state marker is not self-closing (**SHIPPED - text**), so the first parser read R-168 itself
  as OPEN -- a gate that cannot convict its founding case is decoration;
- the CLOSED exemption was line-wide, so "shipped" in a title pardoned "OPEN R-25b".

Check 6 gains WARN-only content classes on MEMORY.md. Link targets are stripped first: the earlier
scan reported three expired statements, all three false (dates in filenames), while missing the one
real expired claim, whose deadline was written ~08-02 with no ISO date.

39 -> 60 assertions. All four runners green.
2026-08-06 11:39:34 +02:00
admin 92a076c239 docs: backup verified IN the repository — snapshot b587f775, 118/118 + 40/40 (R-229)
gates / gates (push) Successful in 8s
58,158 files / 405.865 GiB in 27:56; 994 MiB added, 77 MiB stored -- the one-time full re-read
cost I/O, not storage, as predicted.

Also records a false alarm I raised against my own instrument: `restic ls <snapshot> <path>`
reported 0 of 40 archived memories because restic 0.18.0's path filter does not recurse into
subdirectories. The unfiltered listing shows all 40. An instrument that can drop results silently
is not a measurement -- and this one was measuring a safety property.
2026-08-06 11:22:34 +02:00
admin 6a835ad9ec Re-walk R-201: Phase A record, written BEFORE anything is destroyed
gates / gates (push) Successful in 8s
The fixture is built and the gate passed: all three sentinels are in snapshot
a7bc23bd BY NAME and at the right sizes, and the accented filename survived
into restic intact.

Recorded because they are results, not incidentals:
- what a FRESH install actually landed on — controller 0.201.0, agent 0.125.0,
  golden 0.201.0. Neither carries the fixes. Brought to 0.202.0/0.126.0 BY HAND,
  and the delivery gap is recorded as owed: this proves the journey, NOT that a
  customer would receive that build.
- baseline drift caught on arrival: one documentation-only R-229 commit per repo,
  and the highest register ID is R-229, not the runbook's R-228.
- the runbook's installer baseline (1.25.0) is stale; the published artifact is
  1.26.1, verified byte-identical to demo-hp's local copy.
- credentials DISCOVERED not assumed: HUB_PW mapped and verified live; nothing
  else was needed from the file.

A harness fault worth keeping: the accented sentinel's filename was destroyed at
creation by my own base64->bash->pct exec chain, producing U+FFFD replacement
characters. A Python decode('utf-8') check called it VALID UTF-8, because U+FFFD
is valid UTF-8 — only a hex dump of the filename bytes exposed it. Rewritten
from explicit bytes inside Python on the guest.

R claimed one-shot, streamed file->file to a 0600 store on DooPlex, never
rendered, shape verified without printing (10 words, 75 chars). The runbook's
§5.5-vs-§9.4 tension about where R may live is flagged in the journal rather
than resolved silently.
2026-08-06 11:19:19 +02:00
admin 415371862d REUSE + STATUS: map the new scripts, correct a second stale "not wired" claim (R-229)
gates / gates (push) Successful in 8s
REUSE.md line 62 claimed hub_confirm_gate.py is "not wired to run (R-29)" -- it is gate 3 of
repo_gates.py, which both the pre-push hook and CI run. Second instance of the same stale-claim
class found today, in a third file.

Added rows for repo_gates.py (the runner), instructions_gate.py, install_workspace.py and
rules_report.py, plus the extension point: a new .claude/rules/*.md is NOT live in the session
that created it -- verify from a fresh session, never from the frontmatter.

STATUS.md: one operator-facing entry -- the assistant's notes had no backup; they do now, with
the same-disk and no-off-site caveats stated rather than implied.
2026-08-06 11:05:33 +02:00
admin a0b50eb48f docs: R-168 is CLOSED — correct the stale "CI is still owed" sentence carried into the trim (R-229)
gates / gates (push) Successful in 8s
Found by the checklist's own last item (confirm your push's CI run by run ID). Corrected here and
in the three sibling repos. Ledger J2 records the two lessons: a trim is a VOLUME operation and
carries stale claims forward unless each is re-checked, and the gate cannot catch this class --
"this register item is closed" is not mechanically checkable from the instruction file.
2026-08-06 11:03:40 +02:00
admin 9c10c5d600 docs: ledger, registers and standing ruling S-36 for the part-2 instruction/memory work (R-229)
gates / gates (push) Successful in 8s
R-229 legs (a) and (c) CLOSED. R-230 opened for what this deliberately left -- the ruling owed on
auto-written staleness, the symlink decision, the spec-as-failing-test pilot. R-231 opened for
/opt/backup/scripts/ being unversioned host state, found while adding the memory store to the
backup set: the same class this task was closing, inside the fix for it.

S-36: the auto-memory store is backed up, never committed; no memory file is ever deleted;
a rule file created mid-session is NOT live in that session.

Second session in this repo, so REPORT-<topic>.md per CLAUDE.md; shared REPORT.md untouched.
2026-08-06 11:01:13 +02:00
admin f65ea89a24 workspace: version the root CLAUDE.md + InstructionsLoaded hook, and report which rules fire (R-229)
gates / gates (push) Successful in 8s
install_workspace.py lays down the two things that shaped every session while existing on one
host only. Unlike install_skills.py the targets are LIVE CONFIG, so: timestamped backup before
every write, settings.json MERGED (this script owns exactly one key), a diverged CLAUDE.md
reported rather than silently resolved, and an unparseable settings.json refused outright.

Proven: all 7 top-level settings keys survived byte-identically, and run 2 wrote nothing.

rules_report.py surfaces the column that matters -- rules that have NEVER fired, which are
mis-globbed or dead. 6 of 9 on first run. The hook now self-rotates at 5 MB.

The memory store is BACKED UP, NOT COMMITTED (auto-written, may name hosts/paths): added to
dooplex-backup.service's User Data component. /opt/backup/scripts/ is itself unversioned host
state -- filed, not fixed here.
2026-08-06 10:57:01 +02:00
admin f27aed87cd gate: instructions_gate check 6 — the auto-memory index (R-229)
gates / gates (push) Successful in 15s
MEMORY.md is the larger half of what loads before a word is typed (8.4k tokens vs the root
CLAUDE.md's 6.6k) and is the one instruction file nobody hand-edits, so nothing was watching it.

Three deliberately different outcomes, each pinned by a test: over-ceiling FAILS (auto-memory
drops content past the limit with no error), an orphan WARNS (the store is outside git), and an
absent store PASSES while PRINTING its reason -- asserted on the reason text, because a pass with
no reason is indistinguishable from a gate that stopped running.

39 assertions (was 20). Red-proof run against the real store, not a fixture.
2026-08-06 10:48:59 +02:00
admin 3a9dd81e18 docs+gate: felhom.eu/CLAUDE.md becomes core + path-scoped rules; instructions gate registered (R-229)
gates / gates (push) Successful in 8s
227 -> 115 effective lines, split into .claude/rules/{hub,website,manifests,docs}.md, and
repo_gates.py gains gate 6. Trim first, register second: a registered-but-failing gate refuses
every push through the pre-push hook, which is why this repo -- the one that OWNS the gate --
was the only one not running it.

Register discipline and the R-110 installer fence deliberately stayed in the core; both have
triggers no fixed glob covers, and scoping them would have rebuilt the failure class they exist
to prevent.

Scoping proven from the InstructionsLoaded hook log in two fresh sessions, not from frontmatter.
2026-08-06 10:41:42 +02:00
admin c21bcf84f7 docs+gate: instruction files cannot silently regrow (R-229)
gates / gates (push) Successful in 7s
New shared scripts/instructions_gate.py, registered in controller_gates.py and
agent_gates.py, never copied into a sibling repo (the reuse_refs_check.py
precedent). 20 fixture tests, all asserting the effect: exit code AND that the
message names the file and the reason.

It is a consistency gate, not a budget gate, and the failure message says so. A
/context reading measured the instruction files at 15k tokens against 869k free in
a 1M window -- space is not the constraint, and a future reader must not re-derive
the wrong reason. The 200-line ceiling is adherence guidance; a file nobody can
hold in their head is where contradictions hide, and five were found here.

Checks run against effective text (HTML comments stripped, because they are
stripped before injection): the line ceiling; every .claude/rules/*.md declares
paths: or an explicit unconditional: true; no component version literal; no
TEMPORARY block carrying a past date; and the workspace-root CLAUDE.md is
byte-identical to its versioned copy -- the live file sits outside any git repo,
so that copy is its only version-controlled record.

Two traps recorded so they are not reintroduced: a bare \d+\.\d+\.\d+ matches the
first three octets of every IPv4 (the gate excludes dotted quads, or it fails on
192.168.0.180 in the agent's own file); and unconditional: true is NOT a Claude
Code feature but this project's own marker.

Workspace-root CLAUDE.md 208 -> 182 lines (142 effective), copy kept identical.
The nine-instance invariant table moved into the felhom-testing skill, which
triggers when writing or reviewing a test; all three directive bullets stayed in
the core. felhom.eu/CLAUDE.md got surgical corrections only and is knowingly still
over the ceiling at 227 effective lines -- closing it needs the restructure R-229
defers, said plainly rather than quietly absorbed.

CONTEXT.md gains standing ruling S-35. OPEN-ITEMS.md gains R-229.

Docs only -- no Go, no version bump, nothing built or deployed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JJc8sAGRWmavP3rMtdpkr2
2026-08-06 09:38:52 +02:00
admin d30c2a51ed R-224..R-228 CLOSED: registers, capability map, campaign annotation, STATUS
gates / gates (push) Successful in 7s
Five closed in controller v0.202.0 + agent v0.126.0, each with its live or
red-proof evidence in the row. Five explicitly still open and named as such
rather than left to inference: R-214, R-220, R-221, R-213, R-202 — and R-220 is
flagged as currently worked around BY HAND on the campaign venue, which is the
only reason an app could be deployed there.

The capability map's recovery row STAYS FAIL and says why: fixes are not a
re-walk, nothing walked a customer end to end, and the customer-facing messages
were NOT re-driven live because /recovery correctly retires itself once the old
data is set aside — restoring that state is the reconfiguration the task forbade.

The campaign document is ANNOTATED, not rewritten: it records what was true when
it ran, and that is its value.

workspace-CLAUDE.md gains comment-vs-code entry 9 — the escrow header said the
errors were 'DISTINCT on purpose' and named THREE situations while a fourth was
folded into one of them, and a green test named the defect and did not prevent
it because it asserted a STRING one layer below the merge.

ROADMAP needed no collapse — it carries no rows for these IDs.
2026-08-06 08:34:04 +02:00
admin 453e4503a9 capability map: the recovery row stays FAIL — what Phases 2+4 add, and what they do not
gates / gates (push) Successful in 9s
Says explicitly that these faults are NOT a re-walk, so the row cannot go green
on them. What they add: the BACKUP promise strengthened (the offsite tier ran
itself at 04:15 on a twice-rebuilt box, snapshot_count 1->2; all five daily jobs
fired once; nothing on the must-not list fired), R-217 and R-215 proven live
under exactly their faults, and the set-aside proved not to delete (12 535 KB
byte-exact at the far end).

What they do NOT add: any progress on the JOURNEY. R-224 is Phase 1's headline
defect relocated from the version channel to the transport — a hub outage and a
stopped agent are both reported as a bad recovery code, in 0.056s and 0.030s,
with no unseal attempted. Plus R-225, R-226, R-228.

Also records that §4.1 is now MEASURED rather than deduced, and §4.2's positive
half still is not.
2026-08-06 04:33:27 +02:00
admin 9c1d05d360 CAMPAIGN-11: hygiene, what-did-not-run, venue end state, and the session report
gates / gates (push) Successful in 8s
The recovery codes are shredded with the plant->find->shred->fail-to-find
control the brief asks for, and THE CONTROL PAID FOR ITSELF ON ITS FIRST RUN:
it found the Phase 0 code in ~/.config/credentials as R_CAMPAIGN_11 — a copy
this session did not create and would never have looked for. Without it, a
'codes shredded' claim would have been false. That key was removed from the
shared file with a verified diff (every other line identical, nine keys intact)
and HUB_PW re-tested at hub:200.

Consequence stated plainly rather than left to be discovered:
/home/felhom-repo.orphaned-20260805 (12 535 KB, the three Phase 0 sentinels) is
now permanently unopenable — which is what the set-aside screen promises, and
teardown removes it anyway.

Venue left WORKING and said so: ONLINE, 4 containers healthy, backup target not
degraded, off-site on 2 snapshots. Two things a future session needs: the raw
/mnt/{adatok,mentes} mounts are deliberately left unmounted (R-220's
workaround), and the appliance root credential was shredded — re-fetch it from
the hub.

REPORT-campaign11-phase24.md rather than REPORT.md, per the repo's
parallel-session rule.

No product code changed. No version bumped.
2026-08-06 04:32:56 +02:00
admin df6081e60b CAMPAIGN-11 Phase 4: the soak passed, with two absences answered honestly
gates / gates (push) Successful in 8s
All five daily jobs fired exactly once and completed on time. The 04:15 offsite
run produced snapshot_count 1 -> 2, last_status ok, 24.9s — unprompted, on a box
rebuilt twice and set aside four hours earlier. Nothing on the must-not list
fired, including offsiteheal: that is R-218's NEGATIVE control passing (a box
with a target must not declare), which does not substitute for its positive
half, still owed.

Investigated and DISPROVED: tier2-backup completing in 118ms looked like a
silent no-op; the copy is real and on the backup drive (818.5 KB verified).

Correction to my own pre-registration: backup_run_digest is a TEST FILENAME, not
an event type. The real one is backup_run_failures, a failures digest whose
silence on a clean night is correct — reporting it as a miss would have been a
finding I invented. What survives: the offsite run emits no hub event while both
lesser tiers announce success; failures and staleness are covered elsewhere, so
it is a consistency wrinkle, recorded not filed.

Two absences answered rather than assumed: the restore-test's silence was
PRE-REGISTERED as correct (6h eval, 24h settle); the agent's whole-guest tier is
left explicitly UNRESOLVED because routine local-api calls are not logged at INFO
— a 5h search returns 0 on a box that demonstrably served such calls, so the
instrument cannot answer.

No product code changed.
2026-08-06 04:21:13 +02:00
admin 0f8c3cc848 STATUS: what the overnight stress test found, in plain words
gates / gates (push) Successful in 7s
Leads with what held — nothing lost a byte, the set-aside really is set aside
(verified 12.5 MB untouched at the far end), a wrong code refused three times
with nothing written and no lockout, the alarm fired and cleared itself.

Then the four new findings in operator language: R-224 (our own systems being
down is reported to the customer as a bad recovery code — the same lie as
yesterday through a different door, and the machine had not even tried the
code: three hundredths of a second against a real attempt's one second),
R-226 (a mistyping customer can no longer be told to check their typing),
R-225 (0 snapshots / 0 GB shown above a paragraph saying the store holds
backups), R-228 (the set-aside backups become invisible).

States explicitly that nothing was fixed last night, on purpose.
2026-08-06 00:01:51 +02:00
admin 95e2fba8ca CAMPAIGN-11 doc: invariants, RTO, teardown-owed, hygiene
gates / gates (push) Successful in 7s
Invariants across the phase boundaries: I1/I4(product)/I5/I7 held; I3 breached
twice (R-227's raw Bad Gateway, and R-220's refusal naming an impossible
action, reproduced live a third time); I6 breached twice (R-224, R-225). I2 is
recorded as untested rather than passed, because F10 could not be injected.
I4 was breached by the HARNESS, not the product — sourcing ~/.config/credentials
echoed two demo-box recovery codes into the transcript.

RTO unchanged and explicitly not re-measured: Phase 2's faults are not a
re-walk. Phase 1's 61 attended minutes stand; the unaided number stays
undefined.

Teardown recorded as OWED with all three layers named plus the off-site side —
including that the campaign's sub-account now holds TWO repos, the fresh one
and the 12 535 KB set-aside history, and that demo-felhom/demo-hp namespaces on
ep0 must not be touched.
2026-08-06 00:00:53 +02:00
admin 07998dcff2 CAMPAIGN-11 Phase 4: pre-registered expectations, both directions, before the window
gates / gates (push) Successful in 9s
Written BEFORE the soak so the comparison cannot be fitted afterwards. Sources
are the box's own [scheduler] Registered lines and the agent's backup tier armed
lines, read at 23:56.

MUST happen: db-dump 02:30, tier2-backup + fill-watch 03:30, metrics-prune
04:00, offbox-backup 04:15 (snapshot_count 1 -> 2), a backup_run_digest event,
reports every ~15m.
MUST NOT happen: host_stale/host_recovered, offsite_selfheal_restaged (this is
also R-218's negative control — a box WITH a target must not declare),
offbox_repo_orphaned, offsite_repo_key_changed, escrow blob SERVED, any
controller self-update.
Genuinely uncertain and recorded as such: the restore-test has a 24h settle and
every archive here is hours old, so NOT running is correct, not a gap.
2026-08-05 23:59:19 +02:00
admin ce6fafa14f CAMPAIGN-11: R-228 (set-aside history invisible) + campaign doc F7/F9/F10/F11 sections
The move-aside is correct and verified byte-for-byte; what follows it is not.
settings.json records orphaned_renamed_to and OrphanedRenamedTo has zero
references in any template or handler — 12.5 MB of deliberately retained
customer data at a path the box knows and never shows. Fifth instance of the
project's seam-built-but-never-wired pattern.

Campaign document gains the F7/F9/F10/F11 write-ups: F7 split (set-aside PASS,
afterwards FAIL), F9 partial (R-215's gate proven live on a narrower shape),
F10 NOT INJECTED (harness — three self-healing attempts), F11 PASS.

No product code changed.
2026-08-05 23:59:09 +02:00
admin f88faad50b CAMPAIGN-11 Phase 2 complete: F7/F9/F10/F11, and R-228 (the set-aside history is invisible)
gates / gates (push) Successful in 8s
F11 PASS  ok->stale->ok, an operator mail each way, customer mail correctly
          pairing-gated. DOWN (>1h) not reached — stated, not implied.
F7  the set-aside PASSES decisively: move-aside to
          /home/felhom-repo.orphaned-20260805, mtime and du -s (12535 KB) and
          snapshot f3d9cd67 all unchanged; a fresh empty repo beside it.
          NOTHING DELETED.
    the afterwards FAILS -> R-228: settings.json records
          "orphaned_renamed_to" and OrphanedRenamedTo has ZERO references in
          any template or handler. 12.5 MB of retained customer data at a path
          the box knows and never shows. Seam built, never wired.
F9  R-215's fix PROVEN LIVE on a narrower shape: with recoveryOffer() false,
          GET /recovery now 302s instead of rendering its story. The literal
          never-had-offsite precondition was not staged (it needs a rebuild,
          which the brief forbids before Phase 4) — recorded as partial.
F10 NOT INJECTED, harness not product. Three attempts, each with a control:
          the app, then the controller's monitor, then the run itself recreate
          the mandatory directory within ~1 s. Kept as an observation: at
          capture the directory held only a recreated metadata.db and NOT the
          customer's file, and the run still said ok.

Also reproduced live for a third time: R-220 (candidates empty; unmounting only
the raw mounts flips both drives back) — still OPEN.

Venue left WORKING: 4 containers healthy, backup target not degraded, off-site
fresh repo last_status ok, escrow escrowed.
No product code changed.
2026-08-05 23:55:52 +02:00
admin 259f48aedc OPEN-ITEMS: CAMPAIGN-11 Phase 2 findings R-224..R-227, and what Phase 2 recorded against existing rows
gates / gates (push) Successful in 7s
R-224  every non-code failure on the unlock path is reported as a statement
       about the customer's code. F3 (hub REJECTed) and F4 (agent stopped) both
       returned M4 for a CORRECT current code, in 0.056 s and 0.030 s — neither
       compatible with a real unseal (~1.0 s of scrypt). The agent's err field
       distinguishes fetch-failure from unseal-failure and both become one HTTP
       400; the controller has no reach-failure branch. The vocabulary exists
       and is wired to agentClient() construction, not to the RPC. R-216's gate
       answers source=version and cannot see reachability.
R-225  the store reports 0 snapshots / 0 GB when it cannot read it, above a
       card saying it holds backups. Measured over SFTP: snapshot f3d9cd67 and
       12 535 KB are really there. snapshot_count/repo_size_bytes are ABSENT
       from settings.json after a rebuild — unknown renders as zero.
R-226  M1, the only message that tells a customer to check their typing, is
       unreachable on any box that has re-escrowed (M4's branch precedes it).
R-227  a restart mid-unlock returns a raw English "Bad Gateway" (I3).

Recorded against existing rows: R-216 §4.1 now MEASURED twice (and a correction
— SetFloor's DEBUG line can never reach the debug ring, so a restart alone
would not have produced it); R-218's live half still NOT measured, deliberately,
because the venue has a target; R-217 and R-215's fixes HELD under their faults.
R-199's back-pointer was already present in the capability map — the brief's
assumption that it was owed is stale.

No product code changed.
2026-08-05 23:28:54 +02:00
admin 04ac465da6 CAMPAIGN-11 Phases 2+4: the fault journal, the campaign document, and hub v0.97.1's missing heading
gates / gates (push) Successful in 8s
Phase 2 (eleven injected faults) and the §4 positives that were owed.

- §4.1 MEASURED, twice: the box's rendered GetFloor() is 0.200.0, and a
  cold-started controller logs "settle-gate: GO — at/above floor 0.200.0"
  against the same line reading "floor still unknown" while the hold was in
  force. Also corrects the brief's plan: SetFloor's line is u.dbg(), gated on
  logging.level=debug and written to the logger, so it can NEVER reach the
  debug ring — a restart alone would not have produced it.
- §4.2 still NOT measured, deliberately: the venue has an off-site target, so
  needsOffsiteCredential correctly returns false. Recorded, not inferred from
  the unit test.
- F1 PARTIAL, F2 PASS, F3 FAIL, F4 FAIL, F5 PASS, F6 PASS, F8 PARTIAL.
  F3+F4: a hub outage and a stopped agent are both rendered as "this code does
  not open your package", in 0.056 s and 0.030 s — no unseal attempted. The
  agent's own err field distinguishes them exactly and it is discarded at the
  HTTP boundary; the R-216 capability gate answers source=version and cannot
  see reachability.
- R-217's fix HOLDS under exactly its fault (F5), verified with the false-claim
  strings absent and accented positive controls present.

hub/CHANGELOG.md: v0.97.1 had no heading of its own — the change was written
into the v0.97.0 entry while the deployed tag is 0.97.1. Given its own entry,
marked as added retroactively. Second occurrence of the class (agent 0.90.1).

Evidence: documentation/tests/campaign11-evidence-2026-08-05/journal-phase24.md
No product code changed.
2026-08-05 23:13:47 +02:00
admin 3a539ea530 docs: R-223 CLOSED by a golden rebake — and it was never a dropdown
gates / gates (push) Successful in 11s
The first vouch of agent 0.125.0 was REFUSED by R-120's gate: the artifacts form saves as
a unit, and golden 0.192.0 was older than the newest controller the fleet reports. The
golden had been stale since before controller 0.193.0, so the Day-0 manifest had been
effectively unvouchable for days and nobody had cause to notice. Correcting STATUS.md and
OPEN-ITEMS, which said "one dropdown".

Golden 0.201.0 baked in the drill VM, round-trip verified from Gitea
(e730d7cab343eb35...f007654, 658165766 B), teardown clean, 0 token hits in the saved log.
Manifest now agent=0.125.0 golden=0.201.0 min_agent=0.125.0, read back after the save.

The held floor: the HOLD is established positively (hub log + the box's own "floor still
unknown"). Its RELEASE is not — no HELD line and no dashboard reason are both absences,
the served path logs nothing by design, and the box-side positive needs a restart the
venue cannot take before Phase 2. Recorded as deduction plus a pinned test, not a live
positive.
2026-08-05 22:29:30 +02:00
admin 1a0f7db92f docs: CAMPAIGN-11 — the journey FAILED, R-198's retention PROVEN, six findings fixed
gates / gates (push) Successful in 8s
Registers and evidence for the campaign and its fix pass.

OPEN-ITEMS: R-214..R-223. Six SHIPPED (R-215/216/217/218/219/222); three deliberately
still open and each blocks a real flow (R-214 console banner, R-220 drives unenrollable
after a rebuild, R-221 a rebuilt box cannot run the escrow ceremony); R-223 minted and
WAITING-ON-OPERATOR (vouch agent 0.125.0). R-213 and R-202 untouched.

Capability map: a new row for the customer's UNAIDED journey, recorded FAILED and staying
failed until a re-walk passes — fixes are not a journey. The existing rebuild row is
corrected where it said R-198's retention was unit-proven only: it was proven in production
on the first supersession since the fix, identity_blob retained at 572 B byte-length exact.

CLAUDE.md comment-vs-code table: eighth entry — ResolveManagedFloor, the first where the
false invariant was a GUARD rather than a comment alone.

STATUS: the headline is now "the backup promise is proved, the recovery journey is not",
and the one thing waiting on the operator.
2026-08-05 18:04:13 +02:00
admin 79e31ac24b manifests: hub 0.97.0 -> 0.97.1 (held-floor reason matches its cause)
gates / gates (push) Successful in 9s
2026-08-05 17:54:33 +02:00
admin a7f1d277b1 hub: the held-floor REASON must match the CAUSE (CAMPAIGN-11 follow-on)
gates / gates (push) Successful in 7s
v0.97.0 introduced a second hold reason and left both surfaces printing the first. The
freshly deployed hub logged, for the campaign box:

  managed floor HELD for c11: agent "0.125.0" < MinAgent 0.113.0

which is FALSE — 0.125.0 is ABOVE 0.113.0. That box is held because its floor sits above
the vouched golden, not because of its agent. CLAUDE.md's corollary exactly: when a verdict
changes which field it counts from, the alarm text has to change with it, or a true alarm
reads as one to dismiss.

Both the ACK log line and the Hosts-dashboard HeldReason now come from one
ManagedFloorDecision.HoldReason(), and TestResolveManagedFloor_HoldReasonMatchesTheCause
pins each reason to its cause.
2026-08-05 17:53:24 +02:00