felhom.eu/REPORT.md

felhom.eu — task reports

Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.

TASK-B — R-39 fleet fix + R-50b(a) · hub v0.67.0 → v0.68.0

Date: 2026-07-21 · Baseline: c35da9d (clean, == origin/main) → 54a4644. Companion: agent v0.91.2 (felhom-agent, see that repo's REPORT.md for the agent half, STOP-1 evidence and the four red-proofs).

Status

Leg	Status
Hub v0.68.0	SHIPPED + DEPLOYED — GitOps manifest bump 0.67.0→0.68.0, ArgoCD Synced/Healthy, rollout complete, pod 1/1
Agent v0.91.2	shipped + published + deployed to felhom-pve
Hub v0.68.1	SHIPPED + DEPLOYED — fixes the Configuration layout the v0.68.0 field broke
STOP-1	done + verified
STOP-3 — manifest save	DONE by the operator 2026-07-21 08:31:12Z. All six fields persisted, including artifact_wrapper_sha256 = 104db0a4… (matches the agent's reported hash → drift gauge reads ok) and artifact_min_agent = 0.91.2.
STOP-2 — live re-issue	NOT DONE — the wrong Re-issue was pressed. The hub recorded offsite_reissued (the restic/Hetzner offsite credential), not the PBS-DR one. Evidence: host_pbs_secrets is untouched (created + consumed 2026-07-18, generation = 0) and the pbs_dr descriptor still carries no secret_generation key. See §STOP-2 below.

STOP-2 — what actually happened, and the side effect

The button pressed was the offsite re-issue, not PBS DR. Hub events at 08:29:29Z: offsite_reissued followed immediately by escrow_stale ("A helyreállítási kulcs-letét elavult az offsite jelszó cseréje miatt").

Consequence that needs an action: rotating the offsite restic password makes the escrow blob seal a password that no longer opens the repo, so the hub correctly stops advertising "ceremony done" (the designed v0.57.0 stale_at behaviour — not a bug, but a real follow-up). The recovery-code ceremony has to be re-run before the offsite leg is trustworthy again.

PBS-DR is untouched and healthy — a live probe with the stored credential still returns 200 and the secret file's mtime is unchanged (2026-07-18 20:28:52), which is exactly consistent with "no PBS re-key happened".

The R-39 proof therefore still needs the PBS DR re-issue specifically.

What shipped hub-side

• host_pbs_secrets.generation — a monotonic per-host counter advanced by every fresh MINT and by nothing else, stamped into the descriptor as secret_generation. Since the agent re-applies on the descriptor's CONTENT HASH and a re-key returns byte-identical token_id / fingerprint / datastore / namespace, this is the only field that moves — and therefore the thing that re-arms a converged agent.
  • A re-stage deliberately does not advance it (same secret, unchanged descriptor content).
  • omitempty is load-bearing: emitting a zero would shift every pre-existing descriptor's hash at once — a fleet-wide spurious re-apply.
  • Deviation from spec, deliberate: the brief said to reuse "the new row's id … no schema change". There is no row id — the table is host_id PRIMARY KEY, UPSERTed last-write-wins — and created_at collides for two mints in one second. An additive counter column (existing idempotent ALTER TABLE idiom) is the only monotonic source. Verified applied on the live DB after deploy.
• pbsdrheal gains an auth_failed trigger — a new trigger in the existing machine, escalating to a fresh mint (never a re-stage, which would re-feed the secret PBS just rejected) through the existing damper, so a 401 flap cannot become a secret-minting chain.
• consumed_at honesty gauge — an unconsumed secret past a 15-minute grace under a box reporting applied is the exact 2026-07-18 fingerprint and a disagreement no single tier can detect alone. Surfaced with its own event, deliberately as a SURFACE not a heal: auto-re-issuing would mint a second secret on top of an unconsumed one, which is the mint/consume race R-39(a) recorded.
• Corrected a comment that stated a falsehood — ReissuePBSDR claimed it refreshed the descriptor "with the NEW token_id/fingerprint". False for a re-key, and believing it is why nobody expected the descriptor to come back identical.
• R-50b(a) — ArtifactManifest.WrapperSHA256 + operator field + host-page drift surface. An unknown on either side reads as quiet, never as drift.

Method notes worth keeping

• P2 confirmed GitOps-only, and the trap is real: build.sh itself prints kubectl set image … as its deploy hint, contradicting CLAUDE.md. Not used. Worth fixing in the script — it will mislead exactly the session that trusts tool output over the runbook.
• P4 read the fleet from a temporary copy of the hub DB, which carries live credentials (api_key, host_pbs_secrets.value). Copy shredded immediately after each read. Result: one enrolled host, so the MinAgent raise strands nobody.
• Tests include a flow-level ReissuePBSDR test against a fake that models a real re-key (identical token/fingerprint, rotated secret only). Its red-proof fails on the assertion with both byte-identical blocks printed — the July-18 defect reproduced in a unit test.

For the operator

STOP-2 (one click): press Re-issue PBS credentials for the demo customer. Expected: fresh secret row → secret_generation 0 → 1 (the live descriptor has no such key today) → poke → agent re-applies with no short-circuit → fresh secret consumed → reconcile rc-0 → probe 200 → tier active. The July-18 negative — the same click doing nothing — is the historical red-proof.

STOP-3 (manifest save): Agent 0.91.2 / sha256 34d309be429473f3f0ab34e3185e17b22463a341b30bf46e162306bff4aec22a / PBS wrapper sha256 104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3 / MinAgent 0.91.2. Superseded, do not vouch: 0.91.0 (inert probe leg), 0.91.1. 0.90.1 correctly stays 404.

Residual

R-50b (b)/(c) remain open — the wrapper is still fetched unversioned from raw/branch/main; this release makes drift visible, it does not fix the channel. The 0440 sudoers file is not agent-readable, so its drift stays invisible. The DR-tier capability-map row is deliberately not upgraded to PROVEN-LIVE until STOP-2 supplies the evidence.