Commit Graph

408 Commits

Author SHA1 Message Date
admin e1b8269be0 restore test takes only this box's archives (R-727): an archive encrypted with another key is another box's
gates / gates (push) Successful in 16s
Red-proof RP39. Released as v0.138.0 by release-agent.sh.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
v0.138.0
2026-09-30 10:33:52 +02:00
admin 5c68c869b6 docs: v0.137.0 vouched with golden 0.276.0 (2026-09-28)
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-28 09:45:08 +02:00
admin 728d12b1a0 CHANGELOG: v0.137.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 14:01:26 +02:00
admin dd81866b16 REPORT: v0.136.0 and v0.137.0
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
v0.137.0
2026-09-27 14:01:02 +02:00
admin 3ef095fb71 agent: a guest outside the agent's ACL is not a known guest (R-689, v0.136.0 regression)
gates / gates (push) Successful in 14s
PVE answers 403 permission denied, not "does not exist", for a vmid outside the felhom pool;
v0.136.0 turned that into a lookup failure and the local tier read UNKNOWN every evaluation
(measured on demo-hp). Such an archive is skipped. Red-proofed; verified read-only on demo-hp
with the pre-release binary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 14:00:42 +02:00
admin 7c986915ca CHANGELOG: v0.136.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 13:46:22 +02:00
admin 16dbc83221 agent: the restore test takes only archives of a guest that still exists (R-689, second half)
gates / gates (push) Successful in 14s
Measured on demo-hp right after v0.135.0: with the golden skipped the pick fell to a leftover
archive of guest 9100, deleted in August. "does not exist" skips it; any other lookup error
makes the tier unknown. Red-proofed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
v0.136.0
2026-09-27 13:45:56 +02:00
admin 9555a7f93b REPORT: v0.135.0 released and signed-delivered to both demo hosts
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 13:02:10 +02:00
admin 9ff937d8fb CHANGELOG: v0.135.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 12:16:17 +02:00
admin d4be12ca95 agent: the restore test proves only backups of a guest (R-689)
gates / gates (push) Successful in 13s
The golden template in local:backup/ was picked as the newest settled archive on demo-hp and
failed every 6 h. Candidates are now vzdump-<type>-<vmid> files or PBS ct|vm/<vmid> snapshots
with a reported vmid. Red-proofed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
v0.135.0
2026-09-27 12:15:42 +02:00
admin 7403c2a838 REPORT + CONTEXT: v0.133.0 and v0.134.0 delivered to both demo boxes; restore test back on
gates / gates (push) Successful in 13s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-25 04:55:26 +02:00
admin 309e368731 CHANGELOG: v0.134.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-24 22:49:19 +02:00
admin 0722b2cdb0 agent: a whole-box backup that cannot fit its local target is skipped with a reason before anything starts (R-685)
gates / gates (push) Successful in 13s
Free space is read from GET /nodes/<node>/storage — GET /storage carries no usage (found live, before release).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
v0.134.0
2026-09-24 22:48:51 +02:00
admin 4fe2f81a32 CHANGELOG + REPORT + CONTEXT: v0.133.0 released (tag + package verified by download), not delivered
gates / gates (push) Successful in 13s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-24 16:25:47 +02:00
admin 9bdb4dae8f v0.133.0: a restore-test can never fill a box's disk; leftovers retried on a timer (R-672, R-673)
gates / gates (push) Successful in 13s
Space preflight before anything is created (uncompressed size from the vzdump log / PBS
snapshot, x1.2 + 5 GiB, thin metadata, off the tested guest's pool when another storage
is eligible, unknown refuses, reported as a non-pass result). Failed scratch teardown and
the stale-lock sweep retried every 10 min (the sweep under the heavy-op gate). A thin
pool crossing 90% requests an immediate host report. Six red-proofs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
v0.133.0
2026-09-24 16:23:15 +02:00
admin d9864a94bf CHANGELOG: v0.132.0 vouched for Day-0 installs with golden 0.246.0 (operator decision)
gates / gates (push) Successful in 16s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-17 12:25:49 +02:00
admin 77cd70f7c0 REPORT: agent v0.132.0 - slow crash loop, signed delivery, live proof with the production window
gates / gates (push) Successful in 13s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-17 11:30:45 +02:00
admin 1030abd7d6 CHANGELOG: v0.132.0 released (tag + package verified by download)
gates / gates (push) Successful in 11s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-17 10:27:16 +02:00
admin 18d03bd437 v0.132.0: the slow crash-loop counter (R-539, operator ruling 3 of 2026-09-16)
gates / gates (push) Successful in 12s
Beside the unchanged 3-in-15-minutes brake, a second counter: restarts the
supervisor performed in the last 24 hours. At the fifth the heartbeat stanza
sets slow_crashloop_since (moving at most once per 24 h), slow_crashloop and
restarts_24h; hub v0.117.0 mints controller_slow_crashloop (warning,
operator-only) when the timestamp moves. It never stops restarting.

Persisted per guest (tmp+rename, 0600) so an agent restart or reboot does not
reset it - unlike the fast record, whose reason for staying in memory (a
persisted give-up outliving the fix) does not apply to a counter that only
warns. Deliberate kills count. The startup line prints the new limits.

Red-proofs seen failing: no counter; the once-per-24h guard removed ('the
operator would be mailed per restart'); the save removed ('Restarts24h:1'
after an agent restart). Negative control: restarts 7 h apart never raise it.
go build/vet/test ./... green, 30 packages.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
v0.132.0
2026-09-17 10:26:42 +02:00
admin e98b857684 REPORT: v0.131.0 supervisor + per-tier status, delivery and live validation
gates / gates (push) Successful in 13s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-15 11:37:07 +02:00
admin dcdeb3d16d CHANGELOG: v0.131.0 released (tag + package verified by download)
gates / gates (push) Successful in 12s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-15 09:50:47 +02:00
admin 610804b98d v0.131.0: controller supervisor (R-523); per-tier backup status + tier storage presence (R-517/R-518)
gates / gates (push) Successful in 11s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
v0.131.0
2026-09-15 09:49:44 +02:00
admin 4586f0f7f6 re-run CI against a register that now carries R-421
gates / gates (push) Successful in 9s
The earlier run convicted correctly: instructions_gate found this repo citing R-421 while
felhom.eu's OPEN-ITEMS.md did not yet have the row. My ordering, not the gate's fault - the register
lives in felhom.eu, so a repo citing a new row must be pushed after it.
2026-09-01 12:45:52 +02:00
admin 205e22babe decoy sweep: no gate changed here, and that is the result (R-421)
gates / gates (push) Failing after 12s
All 29 gate scripts across the four repos were read and DECOYED - the label constructed without the
fact, the gate run, the verdict recorded. 16 were fooled. None of them were in this repo.

A decoy that nobody would write proves nothing, so the attempts that turned out illegitimate were
WITHDRAWN rather than counted. Both of this repo were withdrawn, and both are named in the audit.

The gates here that could not be given a plausible decoy are listed BY NAME in
felhom.eu/scripts/decoy_coverage_gate.py EXEMPT (R-426) as UNTESTED - not as sound. A gate nobody
tried to fool is UNKNOWN, and calling it sound would be the same confident guess this sweep exists
to find.

Survey table: felhom.eu/documentation/audits/AUDIT-gate-decoys-2026-09-01.md
2026-09-01 12:38:58 +02:00
admin 058b945064 gate 11: register the shared observations gate
gates / gates (push) Successful in 10s
felhom.eu/scripts/observations_gate.py, invoked across the workspace like
reuse_refs_check.py and instructions_gate.py. This repo's REPORT.md has no
observations section today, so the gate passes quietly - it is registered for
the session that writes one.
2026-08-23 13:53:20 +02:00
admin 40d857b527 CHANGELOG: the fleet runs the published bytes, not the proof build (R-349)
gates / gates (push) Successful in 7s
Both boxes were first given a hand build: same source, same version
string, different bytes (256e0829 vs the published a56a92a7), because
release-agent.sh builds with -trimpath -buildvcs=false and a hand build
does not.

Nothing would have corrected it. The boxes already reported 0.130.0, so
self-update saw the vouched version as installed and would have done
nothing, forever. Every version check in the system compares the STRING.

Both reinstalled from the downloaded package; both now report a56a92a7.
2026-08-20 12:53:37 +02:00
admin 7ae6990bac v0.130.0 released: tag + package published, heading now claims it
gates / gates (push) Successful in 8s
sha256 a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
tag v0.130.0 at 7569f34, 14,141,158 bytes.

Reproducible: rebuilding with -trimpath -buildvcs=false matches the
published artifact byte for byte (R-186's property, checked not assumed).

The heading said UNRELEASED while the fix was hand-installed on demo-hp
only -- publishing then would have pushed it onto the control box through
self-update. release-complete convicted on the release heading and was
right to; the answer was to stop claiming a release, not to bypass it.

NOT VOUCHED by this commit. Vouching is the separate operator act.
2026-08-20 12:47:25 +02:00
admin 7569f34aeb CHANGELOG: correct a claim this session made and then disproved
gates / gates (push) Successful in 8s
The v0.130.0 draft said the 388 descriptors already stuck on ep0 would
persist until the PBS proxy restarted. Measured within the hour: they
clear when the AGENT restarts. demo-hp released its 199 in one second
(415 -> 216 fd); demo-felhom released the remaining 203 (220 -> 17 fd in
under two seconds). 17 is precisely ep0's t0 baseline of 2026-08-18.

They were held on both sides. Closing either side ends them. ep0 was
read-only throughout and its proxy PID never changed.
v0.130.0
2026-08-20 12:39:12 +02:00
admin ede49b610d R-344: restore the idle-connection timeout our hand-rolled transports lost
gates / gates (push) Successful in 7s
Every client here pins TLS, so none can use http.DefaultTransport and each
hand-rolls its own. A composite literal takes IdleConnTimeout ZERO, which
means retain idle connections forever -- not "use a sane default".
pbsTargetsFromPVE builds a fresh pbs.Client every cycle and drops the
previous one, and an abandoned http.Transport does not close its
connections. One stranded socket per cycle, on both sides, forever.

Measured: 388 established connections on ep0 over 46 h, 194 per box, zero
closed in a 31-minute window. pvestatd and proxmox-backup-client made
162,404 requests in the same window and leaked none.

New leaf package internal/httpx owns the default (90s, http.DefaultTransport's
own value) and NewTransport, which returns a FRESH transport per call and
treats <=0 as "use the default", never "no timeout". pbs.Config gains
IdleConnTimeout for tests only.

hub and proxmox carried the same missing default and are corrected here for
consistency. Neither contributed to the ep0 leak -- both are built once per
process and neither talks to ep0:8007.

Tests count connections SERVER-side and model the abandonment, so they pin
the consequence rather than the field. Two red-proofs, both seen failing:
removing the timeout -> "still holds 5 open connection(s), want 0";
DisableKeepAlives -> "3 sequential requests over 3 connection(s), want 1"
(the leak test PASSES under that one -- it is the worse-fix guard that
catches it).

Not released: hand-installed on demo-hp only so demo-felhom stays the
control. CHANGELOG heading stays UNRELEASED until the publish is authorised.
2026-08-20 11:09:39 +02:00
admin f17ed11599 REPORT: agent v0.129.0 — the retained-package recovery class, released and deployed
gates / gates (push) Successful in 21s
2026-08-12 18:49:51 +02:00
admin 1db56bf837 v0.129.0 — a correct code for an earlier package stops being called wrong (R-311)
gates / gates (push) Successful in 14s
Yesterday's drill proved a retained escrow package opens a set-aside store and
restores planted files byte-identical, while this agent answered the customer's
correct code with "the recovery code did not open the sealed bundle". Nothing had
ever tried the retained packages, so a correct-but-earlier code and a mistype were
genuinely indistinguishable.

OffsiteKeyRecoverer gains an optional FetchRetained, consulted ONLY after the
current package refuses, so the ordinary recovery pays nothing for it and cannot
fail because of it. A match returns ErrCodeOpensRetained wrapped in a
RetainedOpenedError carrying the supersession date - no material, no code, no
password. The local API answers 422: a FIFTH status added to the R-224 switch,
never a restructuring of it.

Fail-safe in every direction. Nil fetcher, a hub too old for the route (404 is a
clean "none"), a transport failure, a malformed package: each leaves the original
refusal standing. Attempts bounded at 6 because each unwrap is ~1s of scrypt.

Seven tests with REAL age crypto - the two situations are indistinguishable AT
THE UNWRAP, so a faked unwrap would prove nothing. Red-proof asserted applied:
remove the retained lookup and the fail-closed wrong-code error returns, which is
the lie in those exact words.
v0.129.0
2026-08-12 18:40:00 +02:00
admin 53d047a6c1 Two guards, one number: bound the published check to the retention it must live with
gates / gates (push) Successful in 17s
Gates only. No release, no version bump, no binary published; the agent stays
v0.128.0 at 28ba8593b8 and nothing on a customer's machine changes.

THE COUPLING DEFECT. The registry stopped serving 0.120.0 and older while
check-published-versions.py demanded every tag still be downloadable. Both rules
are sensible and together they are impossible, so CI went red at a commit whose
own run had been GREEN the day before -- and would have gone red again at the
next publish when 0.121.0 was evicted. scripts/retention-policy.json is now THE
number and both readers take it from there.

WHAT CI NO LONGER COVERS, and it prints this on every run rather than leaving it
to be discovered: a released version older than the retention window is no longer
asserted downloadable. Its git tag and its config tree ARE still asserted -- only
the binary's presence is dropped. A missing policy file is INCONCLUSIVE (exit 2),
never silently unbounded.

THE NUMBER IS NOT A LOCATED RULING and the file says so in its own header. Ten is
what the registry demonstrably holds; no register row records a prune, R-210 says
"Nothing was deleted; this is a list, not an action" and concerns local Docker
images, and container packages hold 19 each. The principled bound is the hub's
vouched min_agent floor -- nothing can install below it -- and that is the
recorded follow-up.

check-release-complete.py is the tag half as a machine. release-agent.sh already
warned that "a released version without a git tag 404s a box mid-install, as
root" and the step was still missed, so this is a gate and not a reminder. Legs
1-2 need no network and run in --fast, so the pre-push hook is the earliest
catch. Red-proved by repointing the CHANGELOG head at an unreleased v0.129.0:
both legs convicted and each named its fix command.

Three controls run: green at 10 naming what it dropped; widened to 11 the evicted
version re-enters and convicts; policy removed gives INCONCLUSIVE naming the path.
2026-08-09 19:05:05 +02:00
admin 28ba8593b8 v0.128.0 — R-221: the escrow seed is asserted every tick, not remembered once
gates / gates (push) Failing after 28s
A rebuilt box could not run the escrow ceremony AT ALL, with no way forward from inside the product.
This was the only open item blocking a customer from something we promise them.

MECHANISM, established at file:line rather than assumed. The preflight refuses on
escrow.pbs_storage_id; the pbsdr bridge writes that key; and it wrote it from exactly one place —
finishConverged, reached only on the paths that actually converge.

The marker and the key live in different places and die at different times. The marker is host-side
(<agent-state>/pbsdr/marker.json). The key is in agent.json, which step_agent_config renders from
`base = {}` unless an explicit --preserve-from is given (felhom.eu/scripts/felhom-host-install.sh:
2396 the step, :2449 the render, :2579 the O_TRUNC write; the flag :1246, defaulting empty at :256)
— AND THE RENDER NEVER WRITES AN escrow SECTION AT ALL (grep over the whole heredoc: zero hits). So
a rebuild keeps the marker and takes the key: same descriptor, same hash, early return, and the seed
never runs again into a config that no longer has it.

A rebuild is only the case that was measured. The same hole opens for a hand-edited or restored
config, which is the honest reason this is a seam fix rather than an installer fix: the seed must be
a thing the loop ASSERTS, not a thing it did once.

Apply now re-asserts the seed BEFORE the idempotent early return. seedEscrowStorageID is unchanged
and still never clobbers a different existing value — an operator's own choice outranks the
descriptor's, with a warning naming both.

THE EARLY RETURN IS KEPT. It stops a converged box re-running Proxmox operations every 60s, and
TestSeedReasserted_OnConvergedTick_WithZeroProxmoxCalls asserts ZERO recorded runner calls on that
tick, so a "fix" that simply deleted the return would fail. Cost: one small file read plus a JSON
parse per tick, no exec, no network, early-returning once the value matches.

A seed failure can never un-converge the box: Warn plus a message on the published status, exactly
as finishConverged does it — no marker write, no state change.

Tests drive the REAL Apply with a real temp-dir agent.json and a call-recording runner; calling
seedEscrowStorageID directly cannot see the early return, which IS the defect. Production wiring
(pbsdr.NewManager(..., cfg.SourcePath, ...)) is asserted by walking main.go's AST, not by
strings.Contains, which a commented-out call also satisfies.

Red-proofs, each with the mutation asserted applied: removing the new call makes Scenario A fail on
today's tree (it did, with the intended message); removing the early return makes the
zero-Proxmox-calls assertion fail (it did).

go build / go vet / go test ./... green (29 packages), run separately from this commit.
v0.128.0
2026-08-08 16:29:13 +02:00
admin 6981450110 docs: a comment claimed the hub reads a field it has no field for (R-260)
gates / gates (push) Successful in 26s
Comment-only; no behaviour, no wire change, no version bump, nothing to rebuild.

HostReport.SelfUpdatePending / SelfUpdatePendingVersion carried "The hub reads an absent field as
pending=false, the correct default." The hub has NO FIELD for either, so it reads nothing — present
or absent — and encoding/json discards them on arrival. The sentence described an intent rather than
the code and read as settled long enough that a class sweep had to find it.

The emission is correct and stays: the agent reports the truth and the fault is entirely in the
receiving. The missing consumer is R-264 (OPEN). felhom.eu/scripts/wire_contract_gate.py now refuses
any NEW field of this shape and records the existing ones as reasoned allowlist entries.
2026-08-08 08:46:17 +02:00
admin 703db166e7 v0.127.0: a mount Felhom itself made is not 'something else' (R-220)
gates / gates (push) Successful in 8s
After a rebuild the customer's own drives could not be re-attached: candidates
returned initialize:[] attach:[] while both drives sat there, and the deploy
refused with 'choose an attached drive from the list' — a list that was empty.
Measured live three times.

Mechanism: enrolment mounts a drive TWICE, at /mnt/felhom-drives/<name> and at
the raw /mnt/<name> it creates on the host. The host survives a guest rebuild;
the controller's registry does not. So classifyClaim saw a mount outside the
managed prefix and concluded 'claimed by something else' — about our own mount.

The fix is CORROBORATED, not a widened prefix: a non-managed mountpoint is
forgiven only when the SAME device is also mounted under the managed path, a
pairing only our enrolment produces. A disk another system uses — /srv/data,
/media/x, even /mnt/someone-elses-disk — has no counterpart and is STILL
refused, with its own test and a red-proof showing an over-wide fix offering it
for formatting.

Read from /proc/mounts deliberately: the lsblk invocation is pinned verbatim in
configs/felhom-agent.sudoers, so using the plural MOUNTPOINTS would have coupled
this to a sudoers rollout. /proc/mounts is world-readable — no sudo, no new
allowlisted command, no config change.

Fail-safe: an unreadable mount table corroborates NOTHING, so the device
classifies exactly as before. 'Could not corroborate' must never read as 'ours'.

29 packages ok, vet clean, agent gates OK.
v0.127.0
2026-08-06 12:55:28 +02:00
admin aa74294a7d docs: felhom-agent CLAUDE.md becomes a core plus path-scoped rules (R-229 leg b)
gates / gates (push) Successful in 8s
175 -> 99 effective lines. New .claude/rules/{proxmox,localapi,backup,storage}.md alongside the
existing health-checks.md. The release section points at the felhom-build-deploy skill rather than
restating a table that drifts from the script; the layout section's per-package annotations moved
into the rule file for their area instead of being deleted.

Kept in the core because it is the only part re-injected after /compact: the root-CLI fence and its
three exceptions, the destructive-op gate, prove-ownership (audit A1), the gate entry point, the F9
live-validation fence, and the checklist.

health-checks.md overlaps localapi.md and storage.md on three globs -- deliberate, both load,
stated in each file. Go build/vet/test green and unchanged.
2026-08-06 11:28:27 +02:00
admin 5b2666e3a2 docs: R-168 is CLOSED — the "CI is still owed" sentence was stale (R-229 part 2)
gates / gates (push) Successful in 11s
Corrected in all four instruction files across all four repos. Found while confirming this
session own push by run ID, which is precisely the check that catches it.

In felhom-agent/CLAUDE.md the sentence contradicted the same file release section, which
already said R-168 mails the failure -- a contradiction inside one instruction file, the exact
class the R-229 work exists to find.

REPORT.md deliberately NOT overwritten in the sibling repos: a one-line docs correction must not
destroy the record of their last real implementation.
2026-08-06 11:02:59 +02:00
admin 062a7027ab docs: remove expired and contradictory blocks from CLAUDE.md (R-229)
gates / gates (push) Successful in 8s
Surgical corrections only; the file is deliberately NOT restructured (deferred).

Deleted the expired TEMPORARY block. It read "felhom-pve is at a remote site
(until ~2026-08-02) ... Delete this block on return" and was still being read as
current fact on 2026-08-06, four days past its own deadline, while
felhom-controller/CLAUDE.md asserted the opposite. The location-independence fact
worth keeping (localapi binds 169.254.253.1:8443 on vmbr9 since the R-50 island
migration) moved to an HTML comment.

Every component version literal is gone from effective text, including the
--version reading and the go.mod Go directive. Versions change several times a
day; ask the hub's /hosts + /configs or the box.

The drill-VM claim and the host addresses now point at
documentation/operations/nodes.md, which already stated both correctly. This
file's drill-VM claim was the correct one -- confirmed by qm list on demo-hp.

The R-115/R-188/R-186 release narratives moved to an HTML comment and to the
felhom-build-deploy skill; the directives stayed (never hand-roll the build; the
build -> tag -> publish -> push order; reproducible -trimpath -buildvcs=false).

The health-check block-I/O rule became .claude/rules/health-checks.md, scoped to
the five packages where health checks are written. It had been duplicated from
felhom.eu/CLAUDE.md with a note explaining that that file does not load in an
agent-only session -- correct reasoning, made obsolete by path-scoped rules.

agent_gates.py registers the shared instructions gate.

Docs only -- no Go, no version bump, nothing built or deployed.
Ledger: felhom.eu/documentation/audits/LEDGER-instruction-trim-2026-08-06.md

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JJc8sAGRWmavP3rMtdpkr2
2026-08-06 09:38:38 +02:00
admin a2e914f683 v0.126.0: a fetch failure is not a wrong recovery code (R-224)
gates / gates (push) Successful in 7s
A hub the agent could not reach was reported to the customer as a bad recovery
code. Measured live 2026-08-05 (CAMPAIGN-11 F3): hub firewalled off, a CORRECT
current code, and the customer told it did not open their package — in 0.0556s
against ~1.0s for a real unseal. No unseal was attempted.

The discriminator existed here and this boundary threw it away: recover.go
fails at four distinguishable points and the local-api handler had cases for
two, with a default answering 'the recovery code did not open the sealed
bundle, OR the bundle could not be fetched'.

escrow.ErrBundleFetch now joins the fetch leg and the handler routes it to 502
with its own words — the code was NOT used. 502 not 4xx: the request was not
bad, an upstream dependency failed. Four situations, four statuses: 502 fetch /
400 fetched-and-refused / 404 no bundle / 409 predates the field. The
controller classifies on the STATUS and never parses the sentence.

A GREEN TEST NAMED THIS DEFECT AND DID NOT PREVENT IT.
TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct has said since v0.125.0
that the operator must not be sent to re-read their code because the hub was
unreachable — and passed throughout, because it asserted this package's error
STRING one layer below the merge, and a string is not something a caller can
branch on. Re-pointed at the sentinel, with a consequence-level twin asserting
the status.

Red-proofs: removing the %w join fails the sentinel test; deleting the handler
case makes fetch and wrong-code both answer 400 with the wrong-code sentence.

29 packages ok, vet clean, agent gates OK.
v0.126.0
2026-08-06 07:55:15 +02:00
admin 0404f60e6a pre-push: refuse a push from a clone outside the felhom workspace (R-204 rider)
gates / gates (push) Successful in 10s
The workspace root is already documented (workspace-CLAUDE.md, the workspace-root
CLAUDE.md 'stay inside it') and work drifted into a home directory anyway. A rule
that has failed once as a reminder is not fixed by writing it down again, so it is
now asserted where it can bite.

A push is the right trigger: throwaway clones under /tmp for probes and red-proofs
never push, so nothing legitimate breaks. Symlinks are resolved on both sides; an
absent workspace root SKIPS the check rather than failing it, so this cannot brick
a legitimate clone on another machine. The only bypass is the documented
--no-verify, whose use is already reportable.

Identical in all four repos.
2026-08-05 10:46:39 +02:00
admin 3f5f61b716 docs: R-199 links 6-8 — CONTEXT + REPORT (proven live on demo-felhom)
gates / gates (push) Successful in 7s
2026-08-04 13:56:38 +02:00
admin 6d7904786c agent v0.125.0: open the sealed bundle, return one field (R-199 links 7-8)
gates / gates (push) Successful in 7s
Link 7's only production caller was a --selftest reading R from an env var. Link 8 did not
exist: that selftest writes the whole bundle JSON and its success message named
"tunnel_token + pbs_token" -- accurate when written, a misstatement since v0.77.0 sealed the
offsite repository password into the same bundle. It now names what THIS bundle carried and
what it did not.

POST /escrow/recover-offsite-password: the controller supplies R, the agent fetches this
host's own blob from the hub (self-scoped by the per-host key), unseals it, and returns ONLY
the offsite restic repository password plus its sha256. Not the tunnel token, not the PBS
token, not the WG key -- the controller is a trust tier down and needs none of them.

R: in memory for one call, cleared on every path, never on disk, never in argv, never logged,
never echoed. A test redirects TMPDIR and asserts the tree is EMPTY afterwards -- emptiness
rather than a content scan, because a content scan is defeated by a later call overwriting the
leaked file, which is how the first version of that test passed its own red-proof while R sat
on disk.

Three distinct outcomes: no blob (404), a bundle that opens but predates the field (409), a
code that does not open it (400, fail-closed at the KDF, nothing written).

The wiring is asserted by an AST walk from func main() to the Options field, not by grep.
v0.125.0
2026-08-04 13:41:12 +02:00
admin 856a127cd6 v0.124.1: the repair record must survive the probe that did NOT feed the hub (R-190)
gates / gates (push) Successful in 6s
v0.124.0's transition record never reached the hub, and only the live run showed
it. The capability reported degraded for "one cycle" — the probe call that did the
repair. But probeAll is invoked independently by the self-check log and by the
collector building a host report. On the demo box the repairing call was the log's
(09:39:34, journal shows the self-repair and degraded=1) and the report three
seconds later found the grant present and sent ok. The agent's journal had the
record; the hub had nothing. That is the silence R-190 is about, re-created inside
its own mitigation, with every unit test green.

Fixed with a latch on TIME, not call count: a confirmed repair reports for 20
minutes, which exceeds the 900s report interval, so at least one report must carry
it. It clears on its own and is per tier.

Two hollow tests caught and fixed on the way — one asserting a value it built
itself, one asserting the latch helper rather than the path consuming it (its
red-proof duly passed). The decisions now live in storeGrantHealthyVerdict and
storeGrantRepairedVerdict and the tests call those.
v0.124.1
2026-08-04 09:44:56 +02:00
admin 257c4d85c0 v0.124.0: a lost storage grant repairs itself, and says that it was lost (R-190)
gates / gates (push) Successful in 7s
R-190 is a grant that worked at 04:44 on 2026-08-03 and was gone by 09:24, with a
reinstall, logged pveum activity and cluster-log entries all ruled out. The cause
is open; the resilience need not wait for it.

Everything needed already existed and had only ever been called once: the root
wrapper's `grant` verb, its sudoers vector (`grant *`, any storage id — confirmed,
not assumed), and the exact command. The verb had only ever run at storage
creation — the "built but never wired" shape in a verb rather than a seam.

The probe now runs that wrapper on a missing grant and re-reads ONCE to confirm,
the pbsdr R-22 shape including its restraint.

The record is the half that matters. A repair leaving only "ok" behind destroys
the only evidence a permission vanished, so a recurring loss becomes undetectable
— worse than the fault. A confirmed repair therefore reports DEGRADED for exactly
one cycle with the explanation in Feature, because that is the field the hub puts
in the operator's email (Reason does not travel). Nothing new was built: the hub's
existing ok->degraded->ok edge is the channel, so one loss produces one alert pair.
No wire change, no hub change, no new event type.

Bounded at one attempt per tier per hour: a storage can be unreadable for reasons
an ACL cannot fix, and re-granting every cycle is a repair loop wearing a fix's
clothes. A failed repair never masks the fault.
v0.124.0
2026-08-04 09:38:27 +02:00
admin 72161f6cf0 REPORT: correct the manifest commit hash (311dc06)
gates / gates (push) Successful in 7s
2026-08-03 19:04:44 +02:00
admin 03b58cec0a REPORT + CONTEXT + REUSE: R-185 closed, with the corrected root cause
gates / gates (push) Successful in 7s
The installer defect was NOT PVE_STORAGES as the row and the task assumed: the
create arm of configure_backup_target grants, the Scenario-F reuse arm did not.
Also records the measured trap (an ungranted path answers with INHERITED
privileges, not empty and not 403), the deviation from the spec's suggested
Prober generalisation in favour of the existing poolReadStatus precedent, the
hollow test caught before it shipped, and that demo-hp carried the same drift and
was fixed.
2026-08-03 19:04:15 +02:00
admin fe14bc62c0 v0.123.0: a tier the box cannot READ now says so (R-185)
gates / gates (push) Successful in 7s
The missing grant is one command; the silence was the defect. On demo-felhom the
agent's token has FelhomAgentStore on local, local-lvm and felhom-pbs — and not
on felhom-backup, the storage the same installer configured as
local_backup_target. That storage answers {"data":[]} through the token while
root sees three archives.

An empty listing is what a FORBIDDEN tier and a NEWBORN tier both return, so
pickForThisRun skipped it as "no settled archive yet" and the tier was never
restore-testable on that box. The permission question, unlike the listing, has a
definite answer, so it is asked directly: Client.Permissions reads
/access/permissions as the agent's OWN token, and one capability.Status per
configured tier reports it — composed around the sudo prober, the way the
pool-read check already is.

Measured first, because the obvious reading is wrong: an ungranted path answers
neither empty nor 403, but with the privileges inherited from the box-wide grant
(Sys.Audit, SDN.Use, Datastore.Audit). Checking for Datastore.Audit would report
a blinded storage healthy — red-proved. The probe tests for
Datastore.AllocateSpace.

The probed set comes from the box's own config, never a fixed list: a hardcoded
probe list is the defect reproduced inside the fix. Critical, because the hub
alerts only on critical — except the "local" fallback target, which is reported
but does not page. It never looks at content, so it cannot alarm on a newborn
tier; it never reports ok when it could not ask. Status wire shape unchanged, so
no hub change.
v0.123.0
2026-08-03 18:53:47 +02:00
admin 0b28eae7bb REPORT: R-189/R-188/R-186 — live evidence, the three sha values, and the observations
gates / gates (push) Successful in 6s
Scenario A proven on demo-felhom against the exact observation that filed R-189:
a 675 s offsite restore-test passed, the agent was restarted 11 seconds later
(inside the reporting window), and the hub's very next report carried
'1 restore-tests' where the same sequence produced 0 this morning. The hub's own
database holds the archive, the tier, the pass and the ORIGINAL test time, with
the run mechanics deliberately zero.

Also records the property the validation surfaced: the state holds one proof per
tier, so proving an older archive re-arms a newer one — confirmed live after the
defaults were restored.
2026-08-03 16:58:35 +02:00
admin 7581f8140a v0.122.0: three ways the signals lied about themselves (R-189, R-188, R-186)
gates / gates (push) Successful in 7s
All three are the reporting and release path misreporting its own work. No
customer machine, no backup, no restore, no data. The restore-test itself and
when it runs are unchanged.

R-189 — a passing restore-test no longer vanishes on a restart. restore_tests[]
came only from the in-memory store, whose comment ("lost on restart; the cadence
re-populates") was true under a timer and stopped being true when R-86 made the
agent refuse to re-test a proven archive: the proof is then not repeated for a
whole archive generation. Observed live — a 14.5 GB offsite PASS reached no
host-report because the agent was restarted 2m43s later. RestoreTestState now
carries tier + verified beside the archive and renders reportable entries; the
collector merges them, one per tier, newest by TestedAt. It refuses to lie: a
record missing archive-or-tier produces no entry, and run mechanics are not
re-invented. Only successes are persisted, and the asymmetry is now written where
it will be read.

R-188 — a correct release stops emailing a failure. Only the tag PUSH moved
(build -> tag locally -> publish -> push tag): the push wakes CI, and a tag
visible before its package made the gate correctly fail a correct release about
half the time. The old order's invariant is asserted directly instead — the gate
now refuses a published version with no tag, as a bounded probe that prints its
own coverage, because the package listing api is still 401 without a token.

R-186 — a released binary can be verified by rebuilding it. -trimpath
-buildvcs=false: same source, same bytes, tag or no tag. Measured. publish-agent's
fallback also forced CGO_ENABLED=0 and produced a 74 KB different binary for the
same version; both paths now build identically. CLAUDE.md records the command.
v0.122.0
2026-08-03 16:40:18 +02:00
admin 3d0a1d615d REPORT: restart proof, R-188/R-189, and the restored defaults
gates / gates (push) Successful in 7s
2026-08-03 15:36:46 +02:00