The live test read as a FAILURE for twenty minutes because I stripped only double
quotes from a credentials value wrapped in SINGLE ones, sending a literal ' as part
of the recovery code. Correctly unquoted: the old code returns 422 with
opens_retained=true and the supersession date; a wrong code still returns 400.
The same bug produced the R-308 finding in the previous report. The dashboard
password is fine - HTTP 302 with a session cookie on the first try. Third time this
project has produced a wrong 'the credential is stale' verdict from that one trap.
ListSupersededEscrow had zero production callers for nineteen days. It is the only
reader of a retained identity_blob, so the retention shipped in v0.93.0 was
material the product could not reach - proven on the fixture 2026-08-12, where a
code that opens a retained package was answered as a code that opened nothing.
New GET /api/v1/hosts/<id>/escrow/retained: self-scoped exactly as the current-row
GET, same recovery-mode gate, same audit event written BEFORE the bytes leave,
capped at 16. Rows with a NULL identity_blob are WITHHELD and returned as
unopenable_count - they retain the PBS key, not the repository password, so they
can never open what the caller is asking about, and serving them would let the
screen claim an earlier package is openable on exactly the boxes the original
defect hurt. The count is returned because their existence is load-bearing and
underivable by the caller.
The trade, stated rather than waved through: the hub still cannot read any of it -
sealed bytes in, sealed bytes out, no decrypt path, no recovery code ever held.
What widens is volume, bounded by self-scope, the recovery-mode gate and the cap.
The response is a NAMED TYPE, not a map, so the wire-contract gate can resolve it;
the wire is declared as a fourth ROOT and the gate now checks 182 tags rather than
174. A positive control shows that check is name-presence, not decodability -
filed as R-315 rather than reported as coverage.
Also: golden 0.214.0 baked, published and round-trip verified; the countdown on
demo-felhom cancelled on the operator's ruling (R-307); the spike that halted
Part 3 recorded as R-312; the set-aside store found unrecoverable as R-313.
Six hub tests through the real endpoint; four red-proofs asserted applied.
ListSupersededEscrow had zero production callers for nineteen days. It is the only
reader of a retained identity_blob, so the retention shipped in v0.93.0 was
material the product could not reach - proven on the fixture 2026-08-12, where a
code that opens a retained package was answered as a code that opened nothing.
New GET /api/v1/hosts/<id>/escrow/retained: self-scoped exactly as the current-row
GET, same recovery-mode gate, same audit event written BEFORE the bytes leave,
capped at 16.
Rows with a NULL identity_blob are WITHHELD and returned as unopenable_count.
They retain the PBS key, not the repository password, so they can never open what
the caller is asking about; serving them would have the agent try packages that
cannot succeed and would let the screen claim an earlier package is openable on
exactly the boxes the original defect hurt. The count is returned because their
existence is load-bearing and underivable.
The trade, stated rather than waved through: the hub still cannot read any of it -
sealed bytes in, sealed bytes out, no decrypt path, no recovery code ever held.
What widens is volume, bounded by self-scope, the recovery-mode gate and the cap.
The response is a NAMED TYPE, not a map, so the wire-contract gate can resolve it;
the wire is declared as a fourth ROOT and the gate now checks 182 tags rather than
174. A positive control shows that check is name-presence, not decodability -
filed as R-315 rather than reported as coverage.
Six tests through the real endpoint; four red-proofs asserted applied.
Three verdicts, kept separate because collapsing them is how this assumption
survived a week.
(a) The material IS retained. host_escrow_superseded id 11 is the first retained
row in fleet history to carry identity_blob (572 B), byte-identical to the
pre-supersession row (sha256 a10032341c8584ed...).
(b) The retained material DOES open the old store. Unsealed with the old recovery
code it yielded a password byte-identical to the pre-change one, and restored
three planted files byte-identical from a store the box itself could no longer
open - including a Hungarian accented filename verified as raw bytes. Negative
control ran first and failed closed.
(c) The customer has NO route, and is misinformed. ListSupersededEscrow has zero
production callers; the recovery path selects FROM host_escrow. Asked with the
code that had just worked by hand, the product answered "the recovery code did
not open the sealed bundle". A valid code for retained history is reported as a
bad code - the R-224 class again. R-304, rank 1.
Both installer faults were watched happening first, so installer-v1.27.0 is now
published (tag + both webpage.yaml refs). Pre-fix: the box came up on controller
0.98.3 against a vouched 0.213.0, below the floor and below the version carrying
the recovery screen; and our own uninstall left dnsmasq on 0.0.0.0:53 so our own
next install refused. R-297 and R-300 CLOSED.
Also filed R-305 (the dnsmasq fix fires once per machine - the leftover returns
on the second reinstall, proven), R-306 (--preflight-only writes state it says it
does not), R-307 (a live abandon countdown on demo-felhom, firing 2026-08-24 -
operator decision), R-308 (stored controller password stale), R-309 (the day-0
runbook's publication claim has been false since R-110), R-310 (two edges).
Ceiling R-303 -> R-310. Capability map moved: the retention claim is now marked
operator-only. Phase A logs did not survive the intermediate revert; recorded.
Closes R-296 (verified: shipped in v0.212.0) and R-301 (premise confirmed, fixed
in v0.213.0). Files R-302 with WHY the obvious condition was rejected, and R-303
for the missing markOrphaned guard - the co-render is now harmless, not
impossible. Bake evidence for golden 0.213.0.
211 lines -> one screen. Moves closed items out, corrects the tester paragraph,
states the floor situation as the operator's one-field call, and stops asking
him to decide something that shipped.
Removing the snippet and restarting left dnsmasq enabled and unconstrained on
0.0.0.0:53, so the next byo install's preflight refused and the customer went
debugging a home network that was never at fault.
Ownership is recorded at preflight (the only moment it is a fact - the package
is installed by the agent, not this script) and honoured at removal. Boxes
already in the field carry no record and fail safe to restart-only, with the
reason and the command logged; the preflight message covers them instead.
Not observed live - no installer-v1.27.0 tag is cut. Files R-299..R-301.
The spec listed backups_remote.html:98 as 'Accurate; keep'. It ended with the
same unevaluable promise as surface 2, in the plural - and because the spec's
own guard was written against the singular form, it could not catch it either.
Both corrected; implemented in controller v0.212.0 (R-299).
The row said a drive failure there means offsite-only recovery. Re-read from the
hub's own store: no host row (deleted 2026-07-15 08:56:22, escrow_acked=0), no
escrow of any kind, and offsite backup never ran once (escrow_state pending,
snapshot_count 0 - the fork-4 guard working, not a fault). The local app-data
repo was empty too and the whole-guest vzdump shares the failing device. If that
drive fails today, everything on it is lost.
Records the fact and leaves the parked/not-parked ruling open - that is the
operator's call and does not need restating to be true.
STATUS.md no longer reads the absence of a hub record as reassuring.
GOLDEN_SHA256=8593516889eb93fe1691410d7306be8cb87ee835b8d2378740eb34022272f849
Round-trip verified on the served bytes. Not vouched - that is the operator's.
Step 7 short-circuited on any local golden archive with no version compare, no
digest and no warning, so the manifest sha256 was consulted only on the fetch
path. Local discovery is newest-by-filename: correct by recency, never by
verification. A box could reinstall from a stale archive and come back below the
version where the offsite recovery screen exists.
Digest first, then the baked controller tag. An auto-discovered mismatch
re-fetches the vouched golden; an operator-named mismatch refuses. An unreadable
manifest refuses rather than passing.
Not published: installer-v1.26.0 is deliberately not cut until a fresh install
has been observed taking a stale local golden on drill-r50.
Also files R-295..R-298.
felhom.eu run 293 (67eced8, the demo-felhom recovery commit) FAILED, and
/actions/jobs/293/logs returns HTTP 500 while runs 292 and 294 both return 200
with 12290 bytes. The failure is real and its reason is not establishable --
exactly what this row was filed for, now seen a second time.
What can be said: the same tree passes the FULL gate set locally, all eight OK,
and run 294 on the next commit passed in CI, so nothing persisted. What cannot:
why 293 failed.
The process failure is mine. The checklist says confirm your own push's CI run by
ID; I pushed 67eced8 without doing so, and it surfaced only because the next
session happened to list recent runs. A pull check that is skipped is not a check.
CENSUS (read-only, hub store, tester's machine not contacted): no machine that is
not ours can be in the state that cost demo-felhom its history. The hub holds
escrow for three hosts; both demo boxes lost their pre-fix key in the same four
hours on 2026-08-04; peti-felhom and david have no host row and no escrow at all.
A control ran FIRST and had to pass -- the query returned "present (572 bytes)"
for a host known to have material and "absent (NULL)" for one known not to.
Corrected my own instrument on the way: a date-only comparison mislabelled both
losses as after the fix, so the in-force moment is now pinned from the hub's first
post-fix escrow row (11:11:37Z), which independently agrees with the register.
PART 1 ESTABLISHED. The prune is recorded inside R-267 -- the row about the
Configuration page being slow -- because pruning artifacts is what made that page
fast. Arithmetic checks (23+7=30, plus three versions that only surfaced after the
first thirty moved them onto page one = 33) and the PAGINATED listing shows both
generics at exactly ten. R-291's blocking condition is released: the operator was
being asked to establish something already written down. And my counter-argument
yesterday was wrong in exactly the way R-267 warns about -- "containers hold 19"
came from an unpaginated query; paginated they hold 270 and 169.
RECEIPTS: three restored (drives.enrol, backup.tier1, fail.lost-recovery-code),
each citing the document that walked it; the map already read PROVEN-LIVE for all
three, so this follows the map rather than raising a status in the view. NINE
HONEST GREYS. fault.selfheal's best hit argues against it -- an incident recording
self-heal's absence through a 1h15m outage.
THE DECAY RULE FIRED FOR THE FIRST TIME. backup.restore-proof has a receipt from
28 July and is superseded anyway: demo-hp's restore-test failed 5 August and the
box has since been rebuilt. A claim about a continuing behaviour cannot rest on an
old observation. The capability map still reads PROVEN-LIVE and is now the thing
out of step -- recorded, not silently rewritten.
PART 4 specified, not implemented. The orphan card promises restorability the box
rendering it cannot evaluate: the discriminator is on the hub and no wire field
carries it. A conditional promise the system cannot evaluate is the same defect as
an unconditional false one, so the copy stops promising, says what happens, and
names a route. Ships with the next controller change so one bake covers both.
Checked before acting, and the check is the finding. The box's local key and the
hub's sealed escrow key hash to the SAME value (c60c8bc737a6b7c6...), and that
key answers "wrong password or no key found" against its own repository. Running
the recovery would have returned a key the box already held and which was already
proven not to open the store.
The store was written under 48741892f0ef4d59... -- host_escrow_superseded id=4,
superseded 2026-08-04 07:20:08, identity_blob NULL. The restic password lives
only in the identity bundle (escrow/identity.go:39, read by recover.go:91), so it
is unrecoverable by construction; the surviving K-escrow payload is 64 bytes, a
wrapped key, far too small to carry it. Same shape the register already records
for demo-hp, four hours the wrong side of the retention fix.
Took the operator's stated fallback instead: the orphan reset through the
customer's own card. Old store moved aside, never deleted, to
/home/felhom-repo.orphaned-20260810 (1.2 GB); fresh repository under the current
key; offbox_repo_reset audited hub-side. Then PROVEN rather than assumed --
last_status ok, 10s, and the snapshot's CONTENTS listed: opengist compose files,
manifest.json and volume-dumps/opengist_opengist_data.tar. Not an empty backup
calling itself successful.
R-202 gains hard evidence: the orphan card promises those set-aside backups may
be restorable later with their recovery code. For these 1.2 GB that is false and
unfixable, and it is said to the customers most likely to read it.
Powered up 2026-08-10 ~09:26 CEST. Both unblocked on the hub, both OK on the
approved pair (agent 0.128.0, controller 0.210.0). No false alarm on power-up --
the mute did its job and was removed as the banner said it must be.
The hub briefly read "Guests 0/2": the agent's first post-boot report genuinely
said stopped, because it caught the guests mid-start. Both corrected to running
on the next cycle (07:40:52Z, 07:43:39Z). Transient, not a defect -- confirmed by
waiting for the observable rather than assuming it.
demo-hp's off-site repository still opens with the box's own credential, 18
snapshots intact including yesterday's rehearsal files, so tonight's 04:15 run
has what it needs. demo-felhom's is ORPHANED with no successful run ever; the
offsite_stale mail it sent this morning is a TRUE alarm and the remedy is the
customer-present recovery ceremony, which needs the operator.
R-273's owed guards are both built and closed. R-291 records what CI stopped
covering and why, so it can be widened deliberately rather than discovered.
R-292 is new and was found by a test failing for the wrong reason:
artifact_sha_invalid conflates "version missing", "registry unreachable" and
"bad sha" into one message. v0.102.0 works around it by ORDERING -- the probes
run first, so an unreachable registry is reported as unreachable -- but the
message itself is untouched.
CONTEXT gains the rule this session is about: a check and the policy it enforces
must read the same number from the same place, or they drift and the drift looks
like a defect in something else. Two corollaries, both of which cost something:
a bounded check must print what it stopped covering on every run, and an
unreadable policy is INCONCLUSIVE rather than unbounded.
Stated in the report rather than glossed: Part 4 (finding receipts for the twelve
downgraded claims) was NOT done and is a shortfall, not a decision -- splitting
it would have produced exactly the half-checked green the exercise exists to
prevent. Part 5 was droppable and dropped. The tag-push green is not re-proved
tonight and is not claimed; the evidence offered is runs 190 and 216.
manifests/hub.yaml 0.101.0 -> 0.102.0. Image built and pushed, and verified
served by the registry before the bump rather than after.
R-287 corrected on two counts. My own sentence "no DELETE on the packages API
appears in 48h of Gitea router logs" is WITHDRAWN: kubectl logs on the Gitea pod
now returns nothing older than 2026-08-09 16:35 and contains zero api/packages
lines even for requests I made myself, so the log never covered the window and
its silence was never evidence.
A second attempt to attribute the deletion also failed and the deleter remains
NOT ESTABLISHED. Sources exhausted: no register row records a package prune
(R-210 is WAITING-ON-OPERATOR, says "Nothing was deleted; this is a list, not an
action", and concerns local Docker images); package_version has no soft-delete
column so a deletion leaves no row; Gitea's action feed carries no package
operation at all in the window; and a uniform newest-ten cap is not visible --
felhom-agent generic holds 10 but the container packages hold 19 each. It may be
unestablishable from this side: Gitea keeps no package-deletion trail.
The guard owed since Friday morning. Agent v0.128.0 was published as a package
and never git-tagged; it was vouched here; and because felhom-host-install.sh
fetches an agent's configs from raw/tag/v<version>/configs/, every fresh install
and reinstall died at step 5 of 8, as root, on a virgin machine, for most of a
day. handleSetArtifacts is the sole UI path to SetArtifactManifest, so the check
belongs here and nowhere else.
TWO LEGS, because both failed inside two days: the TAG (missing, R-273) and the
PACKAGE (pruned from under a still-tagged version, R-287). Either alone catches
one of them.
It asserts configs/felhom-mkfs-guarded.sh -- the FIRST of the installer's sixteen
fetch_raw calls and literally the file whose 404 broke Friday. A test pins the
constant, because probing a path that merely exists is how it stayed invisible.
The golden gets the package leg only: it has no config tree, so a tag probe would
assert something the installer never does.
"Could not verify" refuses too, with its own message. No override -- the registry
is the operator's own server, so if it is unreachable the vouch can wait.
ORDERING IS LOAD-BEARING AND A FAILING TEST FOUND IT. The probes run before
resolveArtifactSHA, whose flash conflates "missing", "unreachable" and "bad sha".
Probing first means an unreachable registry is reported as unreachable.
Five scenarios each naming the wrong outcome; three red-proofs, mutations asserted
applied and reverted. With the tag check removed, scenario A reports artifacts_set
-- Friday's exact defect returns.
55 claims verified. Twelve moved, all downwards: walked 32 -> 20, built 5 -> 17.
Register ceiling R-284 -> R-290.
THE RULE DID NOT FIRE THE WAY IT WAS EXPECTED TO. Not one downgrade came from
code moving under an old proof. All twelve came from step 1 of the same rule --
the cited evidence does not exist. Measured: of the 28 capability-map rows
behind the page's claims, 8 carry a tests/ or audits/ path and 20 carry prose
only. The green dots were drawn from rows that cite an argument, not a walk
(R-290). The map, not the dataset, is what needs fixing -- it still says
PROVEN-LIVE for all twelve.
And once it ran backwards: fault.operator-email looked contradicted by R-182,
but live source shows the backup_run_failures digest allowlisted, operator-only
and templated, with recovery_unit_capture_failed now record-only. The claim is
right and the REGISTER ROW is stale (R-289). The session went looking for stale
proofs and found a stale defect.
R-281 WITHDRAWN -- wrong in both directions, settled by the operator's mailbox.
The tripwire DID fire (escrow_blob_served 10:19:41Z = 12:19 CEST) and false
error-severity alarms fired too, for deliberate attended work (R-285). The
measurement's cause is ESTABLISHED: the P7 query copied hub.db without hub.db-wal,
and the signature is exact -- it reported "2 events all day, newest 00:30:07",
and the rows at or before 00:30:07 number exactly 2. Timezone and wrong-key were
tested and refuted. The control had been drawn from the same stale snapshot as
the measurement, which is why it agreed (R-286).
Part 4: NO WORKFLOW CHANGED, deliberately. The gate is not ref-sensitive -- it
enumerates from the Gitea tags API, and both previous tag pushes passed. The red
is TRUE: run 267 saw v0.120.0 downloadable, run 284 on the same commit saw 404.
Who deleted the package is NOT established and is not guessed (R-287).
The page is now generated from where-felhom-stands.yaml by scripts/render_stands.py:
static, zero script tags, every moved status carrying a visible "changed, was X"
chip. The React bundle -- whose content was gzip+base64 inside a JS module map --
is kept as a dated snapshot. scripts/check_stands.py gates the data and convicted
51 problems in my own first draft before the staged positive control ever ran.
Operator is moving back from the vacation home. Guests shut down cleanly first
(checked: no running PVE task, no guest lock, no vzdump/restic in flight — the
"reboot mid-vzdump leaves a stale lock and the guest won't autostart" trap),
then the hosts. Confirmed off at the FABRIC rather than inferred from silence:
the tailnet is healthy, dooplex is up, and both peers report "offline, last
seen 1m ago". An absent ping is not evidence; an offline peer on a working
tailnet is.
Both customers blocked on the hub so the 30-minute host_stale and 60-minute
host_down transitions do not fire four false alarms during the drive.
IsCustomerBlocked gates every monitor plus the notification intake; it does NOT
gate config pull or report intake, so the boxes return normally on power-up.
STATUS.md carries a banner at the very top because the mute has a tail: while
blocked, a box that FAILS to come back is equally silent. Unblocking on arrival
is now the first thing anyone reads.
The walk finished. All four planted files came back byte-identical out of
snapshot 41c830db, including two Hungarian accented filenames verified as RAW
NAME BYTES (NFC preserved) — the discriminator the Gate 0 positive control was
built for, having been watched failing on an NFC->NFD rename that renders the
same. Unlock 21s, restore 13.2s.
It finished only because a terminal was available twice:
- R-273 CLOSED. v0.128.0 was published as a package and never git-tagged, so
every install died at 5/8. Tag pushed on operator instruction after an
INDEPENDENT download proved the package sha equalled the vouched value;
--resume then reached Day-0 SUCCESS in 3m49s on controller 0.210.0. The two
guards that would stop the class recurring are still owed.
- R-280 NEW, rank 1. A reinstalled box cannot re-attach its own data drive by
any dashboard route: /api/disks/candidates returns empty because both lists
are built from the UNCLAIMED-disk scan, and the drive is claimed precisely
because it is also the backup target. Correct for "initialise", over-broad for
"attach", which is non-destructive by definition. The restore page meanwhile
says "Ez ket kattintas" and points at that empty list. Cleared by POSTing
/mnt/sys_drive — an internal path no household could produce.
Also new: R-281 the hub said NOTHING through the entire reinstall and the
sealed-backup tripwire did not fire on a real unseal (positive control: 2 events
all day fleet-wide); R-282 one code with three names and a mail pointing at a
page the box does not show; R-283 hub reads "Claimed 18d ago" while the box
serves its setup page; R-284 "almost full" over a 93%-free store.
R-274 NARROWED by measurement rather than left as written: the resume path
fetched the vouched golden correctly, because --resume skips the preflight that
does local discovery. What survives is real — discovery is sort|tail -1 with no
manifest comparison — but a FRESH install taking a stale golden is still not
observed, and the row says so.
Two of my own claims were refuted by test and are recorded as refuted, not
quietly dropped: the leftover sudoers file is inert (sudo skips dotted names),
and demo-hp's off-site tier was healthy all along.
P1 uninstall, P2 preflight, P3 install. The install FAILED at step 5/8 in 44s,
and the two rank-1 findings are both on the setting-up path a tester's visit is
made of. Eleven register rows minted (R-269..R-279); ceiling moves 268 -> 279.
R-273 (RANK 1) — the hub vouches agent 0.128.0; that version was published as a
Gitea PACKAGE but never git-tagged. Since R-183 the installer correctly pins its
config fetches to raw/tag/v<vouched>, so every fresh install and every reinstall
now 404s as root, mid-install. Measured: main 200, v0.127.0 200, v0.128.0 404.
This is R-184 arriving; release-agent.sh:23 already documents the exact hazard.
Existing boxes are fine (self-update takes the binary from the registry).
NOT fixed here — publishing a release tag is outward-facing and the runbook says
stop and report. One command unblocks it; it is in STATUS.md.
R-272 (RANK 1) — Felhom's own uninstall leaves the condition that makes Felhom's
own reinstall refuse. It installs dnsmasq at day-0, then on teardown removes the
snippet and RESTARTS the daemon unconstrained (process start time lands inside
the uninstall window), which grabs 0.0.0.0:53; the next preflight then refuses,
and the message reads as though the owner's LAN DNS is at fault.
R-274 — a local golden is adopted with no version and no sha check; the manifest
vouch is consulted only on the fetch path. demo-hp's local copy is controller
0.192.0 against a vouched 0.210.0, and below the 0.200.0 where the recovery
screen shipped. Not yet observed end-to-end (R-273 killed step 5 first).
Also: R-275 orphaned credential backups + uid reuse, R-276 the wg tunnel outlives
the uninstall, R-269/270/271 from the token rotation, R-277 three hub surfaces
misreport a healthy off-site tier, R-278 demo-felhom six days unprotected,
R-279 no operator-triggerable off-site run.
Two hypotheses of mine were tested and REFUTED rather than shipped as findings:
the leftover sudoers file is inert (sudo skips dotted filenames), and demo-hp's
off-site tier was healthy all along - I had misread the hub and said so.
STATUS.md records the three rulings §8.3 asked for, with the floor CORRECTED to
its live value 0.200.0 and the count corrected to twenty.
Venue demo-hp, operator-approved at STOP 1. Records the state that P1 destroys,
plus seven pre-walk findings, while they can still be checked against a live box.
R-268 CLOSED — the leaked per-guest local-API token is rotated and the rotation
is PROVEN in both directions (old refused, new accepted, channel up with a
positive observable). Rotating it surfaced three defects:
- an out-of-process rotation does NOT revoke the old token. The daemon serves
Lookup from a stale index and re-reads only on a MISS, so a superseded token
is a direct hit. Red-proved in a unit probe AND live on hardware; the shipped
RemintCoherence test passes only because it looks up the NEW token first.
- R-268's own recipe is incomplete: ensureLocalAPI returns early on a present
local_api block, so writing bootstrap.json is not enough — the controller
serves the old token from controller.yaml across restarts.
- the agent-channel alarm never closes: the UP branch does not notify from an
unseeded state, and the alarm's own remedy ("re-bootstrap") resets it.
Gate 0 complete: dataset planted in the Calibre library (coverage verified, not
assumed) with two Hungarian accented filenames; the comparator watched FAILING
three ways including an NFC->NFD rename that renders identically; off-site run
driven through the product's own button; restore point recorded by identity as
snapshot 41c830db, confirmed to carry all four files.
Also corrects the record: demo-hp's off-site tier is HEALTHY. Three hub surfaces
agreed it was absent and all three mislead — the panel showing 0 snapshots renders
the LOCAL tier, 162 KB rounds to 0.0 GB, and a two-day-old stuck event reads as
current. And the managed-update floor is live at 0.200.0, not 0.156.0.
Registry pruned to the newest 10 per package on the operator's confirmed rule. 33 deletions, all
HTTP 204; the live-vouched golden 0.210.0, agent 0.128.0 and floor 0.127.0 were asserted into the
KEEP set BEFORE any DELETE was issued and verified still fetchable after.
TWO CORRECTIONS TO WHAT I REPORTED EARLIER, both recorded rather than quietly dropped:
1. 'Only 50 generic versions exist' was NOT a count, it was a PAGE LIMIT. ?limit=1000 returns at most
50, and the 50 I measured was exactly the cap. Three older agent versions (0.81.0/0.80.0/0.79.0)
only became visible after the first 30 deletions moved them onto page one. An unpaginated listing
is not evidence of a total — this repo's own 'an empty listing is not evidence of emptiness' rule,
walked into while measuring it.
2. The operator's 'reduce the number of artifacts' was the better call and my measurement said
otherwise. I reported it helps sub-linearly and is not the lever. Measured after: trimming to
10+10 took the COLD load from 13.4s to 5.4s, a 2.5x improvement on exactly the path the memo
cannot help, because the fan-out is per-version.
drill-r50 runs agent 0.113.0, now deleted; flagged before deleting, disposable nested drill VM, only
its re-download path is gone.
Operator-present drill on demo-felhom guest 9201. Preflight refused with the key removed, went green
after one 60s tick with the SAME MainPID (1993397 both sides, so no restart), and all 45 config keys
came back identical. Positive control run before the change so the green afterwards is a measurement,
not an artefact of the probe. Marker sha unchanged throughout.
R-267 closed by hub v0.101.0. Measured after the 60s memo: cold 13.4s, warm 0.24-0.33s. The operator
sees a quarter-second except at most once a minute.
R-268 filed against myself. Setting up R-221's live drill, a one-liner meant to list bootstrap.json's
KEYS printed the local_api object whole, including its token, for guest 9201. Reported rather than
quietly rotated, because a secret reaching a transcript is a finding whatever its blast radius.
Exposure assessed rather than assumed, and it is small: the token opens only the agent's per-guest
local API on the island bridge between that host and that one guest, self-scoped to guest 9201, not
routable from the LAN or internet, on a Tier-0 disposable box with no customer data. Using it already
requires code execution there, at which point an attacker has more than the token.
Rotation exists (TokenStore.Mint, last-write-wins per VMID) but must also rewrite the guest's
bootstrap.json or the controller loses agent access — an operator-timed act, not a background one.
The general fix is upstream: reading secret-bearing JSON should go through a helper that prints keys
and never values, the discipline the golden bake already uses for the Gitea token.
R-221 also recorded as PROVEN ON HARDWARE in STATUS.
v0.100.x removed the serialisation: 26.2s -> ~9.85s mean. What remained was one Gitea package SEARCH
per dropdown at 0.20-3.8s depending on load, which concurrency cannot help.
Memoised for 60s IN MEMORY. The TTL was ruled by the operator against the workflow that cares: a
bake-and-vouch session publishes an artifact and comes straight here to select it, so a minute is
short enough not to be noticed and long enough that every reload in that session is instant.
NOT persisted. Gitea IS the store for both the version list and the sha; a copy in hub_settings would
be a second source of truth that can drift from the registry it describes, and the operator reads the
sha here to confirm what they are about to vouch. An in-memory cache dies with the process and can
never be mistaken for a record.
A failed resolve is NOT cached — a blip must not pin an empty dropdown for a minute. But an empty
list from a package that genuinely has no versions IS cached, because 'we found nothing' and 'we
could not look' are different answers (CONTEXT S-39, applied to a list instead of a figure).
THE FIRST VERSION OF THIS GOT THAT WRONG: the comment said only successful resolves were cached and
the code cached the empty list anyway. TestArtifactChoices_FailureIsNotCached caught it before it
shipped — which is the argument for writing the test that asserts the comment, and the same class
this session spent the day closing.
go build/vet/test green, go test -race clean, run separately from this commit.
Reported as 'almost minutes'. The guess that it hashes artifacts on page load DOES NOT HOLD and the
code already said so: Gitea stores the sha and the hub reads it as metadata. The cost was latency x
count, fixed in three legs (hub v0.100.0-0.100.2), each found by refusing to accept a number that
did not match the arithmetic.
Measured 26.2s -> mean 9.85s over 8 samples (min 5.13, max 18.13). The remaining dominant cost is the
package SEARCH, 0.20-3.8s per dropdown depending on load, which concurrency does not help; 16
concurrent file-metadata calls take 0.58s by comparison.
EVERY NUMBER IS CONTAMINATED and the row says so: taken on DooPlex at load average 7-11 while this
same session was building images, running two Go suites and baking a golden. The same search measured
3.8s in-cluster and 0.44s from the host ninety seconds later. Re-measure on an idle box.
Both operator proposals answered on the measurement rather than deferred to: pruning artifacts helps
sub-linearly (only 50 versions exist) and is worth doing for its own sake; storing the hash in the
hub DB is NOT recommended, because Gitea is already the store and a copy would be a second source of
truth the operator reads to confirm a vouch. The lever that would work — an in-memory cache with a
TTL — is left OPEN because it trades dropdown freshness for speed, which is an operator decision.
Third and last leg, found the same way as the second — by not accepting that the numbers matched the
arithmetic when they did not. After the fan-out and the side-by-side resolve the page was ~11.9s mean
where ~3s was predicted.
Cause: the client used http.DefaultTransport, whose MaxIdleConnsPerHost is 2. Above that Go opens a
connection per request and discards it after, so under a 16-way fan-out almost every call paid a
fresh TCP setup AND a fresh authentication. Authentication is the expensive half: unauthenticated
/api/v1/version answers in ~0.03s while an authenticated package call takes ~0.24s against the same
Gitea instance.
Transport sized to the fan-out: MaxIdleConnsPerHost 16, MaxConnsPerHost 16 as a ceiling so a large
package list can never stampede Gitea harder than the fan-out needs, IdleConnTimeout 90s.
Follow-up to the fan-out, and the reason for it is worth recording: THE FIRST FIX DID LESS THAN THE
ARITHMETIC PREDICTED. Concurrency took the page from 26.2s to ~11-18s, not the ~2s expected, so the
gap was chased instead of declared closed.
What it found: the slowest single call on the page is not a per-version sha lookup at all, it is the
PACKAGE SEARCH (/api/v1/packages/admin?type=generic&q=...), measured in-cluster at 1.1-2.2s each
against ~0.24s for a file's metadata. Per-version fan-out cannot touch it — there is one search per
package and they ran in series.
The two dropdowns are independent, so they now resolve side by side, overlapping both searches and
both fan-outs. go test -race clean on the new concurrent paths.
Gitea's latency on this box is load-dependent and varies 2-4x between samples, so the CHANGELOG
quotes a range rather than a single pair of numbers.
felhom-hub:0.100.0 confirmed present in the registry (manifest HTTP 200) before this bump. A built
image deploys nothing until this tag moves in git and the app is synced.
MEASURED, NOT GUESSED: GET /configuration -> HTTP 200 in 26.2s.
The reasonable guess was that it hashes the artifacts on page load. It does not, and the code already
said so: Gitea stores each package file's sha256 and gitea.FileSHA256 reads it as metadata — "a cheap
metadata call, the artifact bytes are never downloaded". The cost was never CPU.
IT WAS LATENCY x COUNT. artifactChoices made ONE SERIAL round-trip per version, for two packages,
capped at 20 each: 2 x (1 version list + 20 sha lookups) = 42 sequential requests at ~0.6s each out
through the public ingress. 42 x 0.6 = 26s, which is what the clock said.
1. The sha lookups now run CONCURRENTLY, bounded at 8 in flight. Order preserved by writing into a
slot rather than appending — the dropdown is newest-first, and a scrambled sha would show the
operator a hash belonging to a DIFFERENT artifact. A failed lookup still drops that version only.
2. The client talks to Gitea IN-CLUSTER (http://gitea.gitea-system.svc.cluster.local:3000,
overridable via GITEA_API_URL). Measured from the hub pod: 0.11s against 0.26-1.16s, because the
public path adds DNS, the ingress hop and a TLS handshake to each of the 42. Plain HTTP is safe
ONLY because it never leaves the cluster network — the registry token rides the Authorization
header, so this must not point at a public host without TLS. Unreachable -> the existing graceful
degradation to manual text entry, unchanged.
DELIBERATELY NOT DONE: caching the sha in the hub's own database. That was the other half of the
proposal and it is the wrong shape. Gitea already IS the store; a copy in hub_settings would be a
second source of truth that can drift from the registry it describes — and the operator reads exactly
this value to confirm what they are about to vouch, so a stale one would be a confident wrong answer.
The same reasoning golden_currency_gate.py already records for the vouched version. With the fan-out,
a cold load needs no cache to be fast.
The cap stays at 20 and now bounds the FAN-OUT too, not just the rendered list.
Tests pin order (and that each sha belongs to its own version), per-version failure isolation, and
THE CONCURRENCY ITSELF — a wall-clock assertion plus an in-flight counter, so a fast run cannot be
luck, and an upper bound so a large package list cannot stampede Gitea. Red-proof: reverting to the
serial loop takes 861ms where the concurrent one takes 150ms, and the test fails naming the
26-second page.
go build / go vet / go test ./... green (18 packages), run separately from this commit.
Four defects of one family, all shipped today: something the box already knows, thrown away or drawn
as its opposite. Agent v0.128.0, controller v0.210.0. NO HUB CODE, no hub bump, no ArgoCD sync.
R-265 (this repo). timeout-minutes: 5 on the gates job — every honest run in the observed session
finished in 18-34s, so this is ~9x the slowest and far under whatever reaped run 264 at 834s with no
log. The alarm mail now carries Elapsed (start stamp via $GITHUB_ENV; an absent stamp prints
"unknown (no start stamp)", never a bogus 1.7-billion-second figure) and its "names itself in the run
log" sentence is qualified so it cannot mislead when there is no log.
⚠ THE UNKNOWN IS NOT CLOSED. Whether the if: failure() alarm fires for a REAPED job is still
unverified. The timeout makes the reap unreachable in practice; it does not answer what happens in
one. Demonstrating it means deliberately hanging a run on main, which would leave the branch red for
a parallel session. Said in the workflow comment, the changelog, R-265 and the report — none of them
claiming it is answered.
GOLDEN 0.210.0 baked, published, round-trip verified, NOT VOUCHED. The currency gate went red the
moment the controller was bumped — correct — and is closed by the bake, never --no-verify. No
--no-verify anywhere this session.
⚠ THE AGENT WAS NOT PUBLISHED UNTIL THIS SESSION CHECKED, AND IT MATTERED. R-221's fix is in the
AGENT, and a fresh install takes its agent from the Day-0 manifest. The binary had been hand-deployed
to felhom-pve and never published, so agent_version 0.128.0 was not selectable and a fresh install
would have received 0.127.0 — the golden would have carried the controller fixes and NOT the one the
headline defect needed. Caught by checking each Day-0 value was FETCHABLE rather than assuming.
Published from the live-deployed bytes, sha-verified across the hop first.
Registers. R-221, R-259, R-258, R-265 CLOSED. R-266 MINTED (READY): the failed root statfs still
travels to the hub as a 0-of-0 disk; ranked LOW because it is the quiet direction — it can only miss
a true alarm, never raise a false one — and it is now a two-repo wire change governed by G-1's gate.
Highest ID moved R-265 -> R-266.
CONTEXT S-39 rules the convention this project was missing: "we do not know" is never drawn as
"fine", and the codebase has ONE way of saying it — an explicit ...Known bool companion checked in
the template. ROADMAP G-3 was explicitly blocked on that decision and is unblocked; what remains
there is a survey-and-convert of existing sites, not the gate.
Capability map row 93 CHECKED and it was NOT claiming something untrue — it is about the operator
notification path. But its narrative ("the page you open to ask whether ONE app is backed up")
invites the wrong reading, and the adjacent thing WAS false until v0.210.0, so the row now records
that the two halves disagreed and only the operator half was true.
Six red-proofs across the two code repos, each with the mutation asserted applied. The one that
matters: Part 1 Scenario A FAILED against today's tree, with the intended message.
Part 1's operator-present live validation is OWED and is the session's STOP.
repo_gates --fast: all 8 OK.
Run 264 (650cc8a, a DOCUMENTATION-ONLY commit) failed between two greens of byte-identical gate code.
Not waved away as a flake, because this project's own record is that a "known flake" can be a true
positive.
MEASURED. Every other run this session: 18-34s, log present (HTTP 200). Run 264: 834s (07:12:40 ->
07:26:34 UTC) and GET /actions/jobs/264/logs returns HTTP 500 - "264.log.zst: file does not exist".
The act-runner pod never restarted (0 restarts, 5d17h), so the job hung and was reaped; the runner
did not die.
NOT A GATE FINDING, on four independent facts: the diff from the green before it is Markdown only;
the same content is green two commits later (265, 33s); the gate code is identical across 263/264/265;
and 260-262, which WERE real gate failures, each failed in under 35s WITH a log.
THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately recorded as such. DooPlex was doing heavy
work in that window (139 MB kubectl cp, a go run compiling the whole hub module), which is a
plausible contention story - but 40 cores at load ~5 does not establish it, so it is filed as a
hypothesis rather than asserted as a cause.
THE FINDING THAT MATTERS IS SECOND-ORDER, and it is gates.yml's own purpose turned against it. The
workflow exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and
its alarm mail says "The failing gate names itself in the run log." There is no run log. An operator
following that sentence finds nothing and cannot tell a reap from a conviction. Unverified and worse:
the alarm step is `if: failure()` and whether it ran at all for a reaped job is unknown - if it did
not, this was a red CI that alarmed nobody.
Fix shapes recorded, none built: surface duration + log-presence in the alarm; an explicit
timeout-minutes under the reap so it fails fast and loudly WITH a log; and one deliberate test of
whether the alarm fires on a reaped job, because until that runs, "CI alarms on failure" is an
assumption.
Recorded on arrival as 0.207.0, re-read from live hub_settings at the end and it is 0.208.0 — the
operator acted while the session ran. The ask is therefore 0.208.0 -> 0.209.0, not 0.207.0 ->
0.209.0, and STATUS.md plus the golden evidence now say so.
Caught only because the state was re-read rather than carried forward from the arrival note. A fact
recorded at the start of a long session is a fact about the start of the session.
The gate's third instrument defect and the reusable lesson underneath it: a gate needing a sibling
clone passes on a workstation and is INCONCLUSIVE in CI, and a gate must not shell out to a tool the
CI image may not have. Three red runs (260-262), each of which mailed the operator, before 263 went
green. CONTEXT S-38 gains the two-homes rule.
CI convicted ALL 174 checked tags while the pre-push hook was green. Cause, read from the run log
rather than guessed at the second attempt: the search used `grep -rnE --include=…`, and the CI
runner's image carries python3 and git and deliberately little else — its grep does not support
`--include`, so stdout was empty and the gate read empty as "the tag is absent".
That is a gate silently treating a tool failure as a finding, which is worse than no gate, and it is
exactly the error-swallowing this repo forbids. A green from it would have been just as untrustworthy
as the red.
Fixed by removing the dependency, not by working around it: the search is now pure Python — one
token index per receiving repo, built in a single pass, no subprocess. Faster too (one walk instead
of ~350 greps), and unreadable-file / empty-repo cases now exit 2 INCONCLUSIVE rather than reporting
absence.
THE BEFORE CAPTURE WAS RE-VERIFIED, NOT RE-GENERATED — the stronger claim. All 40 fields recorded in
BEFORE.md were re-tested against the new implementation: agree=40, disagree=0, i.e. exactly the four
this session fixed are now present and the other 36 still absent. The number 40 stands under both
implementations; only the mechanism changed. The whole-token property survives by construction — a
token index treats `healed_at` and `privsep_healed_at` as distinct tokens.
This is the THIRD instrument defect this gate's own controls caught before it was trusted, after the
substring false negative and the dr_recipe over-opacity. The first two were caught by re-finding the
known instances; this one by the CI-versus-hook disagreement the workflow's alarm mail explicitly
says outranks whatever the push was for.
CI went red on the two G-1 commits while the local pre-push hook was GREEN — which this workflow's
own alarm mail says outranks whatever the push was for, because it is a finding about the gates
themselves. It was.
CAUSE, reproduced rather than guessed: scripts/wire_contract_gate.py compares what one component
EMITS against what the other can RECEIVE, so it needs the SOURCE of the controller and the agent.
The workflow already fetched the controller (for golden-currency) but not the agent, so the gate
exited 2 INCONCLUSIVE. Reproduced locally by pointing the gate at a nonexistent agent path: same
exit 2, same message.
THE LESSON, and it is the reusable half: the pre-push hook runs on a workstation where every sibling
is a real clone, so a gate that needs a sibling passes there and is inconclusive here. **The two
automated homes are not interchangeable, and a NEW GATE MUST BE CHECKED IN BOTH.** Recorded in the
step's own comment beside the fetch.
Fixed by giving the gate what it needs. NOT by letting it skip when a sibling is absent — that is the
fail-open shape, and it would leave the gate running in NEITHER home, which is the R-29 census
failure this runner was built to end and which golden_currency_gate.py's docstring already warns
about at length.
A built image deploys nothing until this tag moves in git and the app is synced.
felhom-hub:0.99.0 confirmed present in the registry (manifest HTTP 200) before the bump.
admin