felhom.eu/REPORT.md

felhom.eu — task reports

Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.

TASK-B — R-39 fleet fix + R-50b(a) · hub v0.67.0 → v0.68.0

Date: 2026-07-21 · Baseline: c35da9d (clean, == origin/main) → 54a4644. Companion: agent v0.91.2 (felhom-agent, see that repo's REPORT.md for the agent half, STOP-1 evidence and the four red-proofs).

Status

Leg	Status
Hub v0.68.0	SHIPPED + DEPLOYED — GitOps manifest bump 0.67.0→0.68.0, ArgoCD Synced/Healthy, rollout complete, pod 1/1
Agent v0.91.2	shipped + published + deployed to felhom-pve
Hub v0.68.1	SHIPPED + DEPLOYED — fixes the Configuration layout the v0.68.0 field broke
STOP-1	done + verified
STOP-3 — manifest save	DONE by the operator 2026-07-21 08:31:12Z. All six fields persisted, including artifact_wrapper_sha256 = 104db0a4… (matches the agent's reported hash → drift gauge reads ok) and artifact_min_agent = 0.91.2.
STOP-2 — live re-issue	DONE + PROVEN 2026-07-21. Chain closed in 13 seconds; see below.

STOP-2 — the proof (the whole task's reason to exist)

The operator pressed Re-issue PBS credentials. The identical click on 2026-07-18 did nothing.

hub   08:39:31Z  fresh mint, generation 0 -> 1
                 descriptor gains "secret_generation": 1
                 (token_id + fingerprint BYTE-IDENTICAL — the re-key shape that was invisible)
agent 10:39:34   felhom-pbs-apply read felhom-pbs      <- leg (b): the read that was impossible
agent 10:39:38   ERROR "REJECTED ... applied and DEAD"  previous_state=applied
                                                        <- leg (c): the R-39 state, loud at last
hub   08:39:45Z  consumed_at stamped
agent 10:39:45   "one-time token secret consumed" secret_len=36
                                                        <- leg (a): NO short-circuit
agent 10:39:45   felhom-pbs-apply reconcile (set-only, no --server)
agent 10:39:47   "pbsdr: converged" state=applied

(host CEST = UTC+2; hub timestamps UTC.)

Corroboration: the agent marker hash moved to afbb3b41… — in the failure it was byte-identical to the pre-reissue marker, which was the single-line proof of the defect. consumed_at stamped. The on-disk secret's mtime moved 2026-07-18 20:28:52 → 2026-07-21 10:39:45. A live probe with the NEW credential returns 200. Three consecutive hub reports trace the entire state machine applied → auth_failed → applied. Zero pbsdr_selfheal escalations fired, with exactly ONE mint, ONE consume and no consumed-failed.json — the box healed through the descriptor path before the damper was ever needed.

First attempt, worth recording: the operator initially pressed the offsite re-issue — there are two distinct Re-issue actions and my instruction said only "press Re-issue". Harmless to PBS-DR, but it rotated the restic password and correctly marked the escrow stale, so the recovery-code ceremony had to be re-run (done). Name the surface explicitly in future runbook steps.

What shipped hub-side

• host_pbs_secrets.generation — a monotonic per-host counter advanced by every fresh MINT and by nothing else, stamped into the descriptor as secret_generation. Since the agent re-applies on the descriptor's CONTENT HASH and a re-key returns byte-identical token_id / fingerprint / datastore / namespace, this is the only field that moves — and therefore the thing that re-arms a converged agent.
  • A re-stage deliberately does not advance it (same secret, unchanged descriptor content).
  • omitempty is load-bearing: emitting a zero would shift every pre-existing descriptor's hash at once — a fleet-wide spurious re-apply.
  • Deviation from spec, deliberate: the brief said to reuse "the new row's id … no schema change". There is no row id — the table is host_id PRIMARY KEY, UPSERTed last-write-wins — and created_at collides for two mints in one second. An additive counter column (existing idempotent ALTER TABLE idiom) is the only monotonic source. Verified applied on the live DB after deploy.
• pbsdrheal gains an auth_failed trigger — a new trigger in the existing machine, escalating to a fresh mint (never a re-stage, which would re-feed the secret PBS just rejected) through the existing damper, so a 401 flap cannot become a secret-minting chain.
• consumed_at honesty gauge — an unconsumed secret past a 15-minute grace under a box reporting applied is the exact 2026-07-18 fingerprint and a disagreement no single tier can detect alone. Surfaced with its own event, deliberately as a SURFACE not a heal: auto-re-issuing would mint a second secret on top of an unconsumed one, which is the mint/consume race R-39(a) recorded.
• Corrected a comment that stated a falsehood — ReissuePBSDR claimed it refreshed the descriptor "with the NEW token_id/fingerprint". False for a re-key, and believing it is why nobody expected the descriptor to come back identical.
• R-50b(a) — ArtifactManifest.WrapperSHA256 + operator field + host-page drift surface. An unknown on either side reads as quiet, never as drift.

Method notes worth keeping

• P2 confirmed GitOps-only, and the trap is real: build.sh itself prints kubectl set image … as its deploy hint, contradicting CLAUDE.md. Not used. Worth fixing in the script — it will mislead exactly the session that trusts tool output over the runbook.
• P4 read the fleet from a temporary copy of the hub DB, which carries live credentials (api_key, host_pbs_secrets.value). Copy shredded immediately after each read. Result: one enrolled host, so the MinAgent raise strands nobody.
• Tests include a flow-level ReissuePBSDR test against a fake that models a real re-key (identical token/fingerprint, rotated secret only). Its red-proof fails on the assertion with both byte-identical blocks printed — the July-18 defect reproduced in a unit test.

For the operator

STOP-2 (one click): press Re-issue PBS credentials for the demo customer. Expected: fresh secret row → secret_generation 0 → 1 (the live descriptor has no such key today) → poke → agent re-applies with no short-circuit → fresh secret consumed → reconcile rc-0 → probe 200 → tier active. The July-18 negative — the same click doing nothing — is the historical red-proof.

STOP-3 (manifest save): Agent 0.91.2 / sha256 34d309be429473f3f0ab34e3185e17b22463a341b30bf46e162306bff4aec22a / PBS wrapper sha256 104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3 / MinAgent 0.91.2. Superseded, do not vouch: 0.91.0 (inert probe leg), 0.91.1. 0.90.1 correctly stays 404.

Residual

R-50b (b)/(c) remain open — the wrapper is still fetched unversioned from raw/branch/main; this release makes drift visible, it does not fix the channel. The 0440 sudoers file is not agent-readable, so its drift stays invisible. The DR-tier capability-map row is deliberately not upgraded to PROVEN-LIVE until STOP-2 supplies the evidence.