113 lines
7.3 KiB
Markdown
113 lines
7.3 KiB
Markdown
# felhom.eu — task reports
|
|
|
|
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
|
|
|
# TASK-B — R-39 fleet fix + R-50b(a) · hub **v0.67.0 → v0.68.0**
|
|
|
|
**Date:** 2026-07-21 · **Baseline:** `c35da9d` (clean, == `origin/main`) → `54a4644`.
|
|
Companion: agent **v0.91.2** (`felhom-agent`, see that repo's `REPORT.md` for the agent half,
|
|
STOP-1 evidence and the four red-proofs).
|
|
|
|
## Status
|
|
|
|
| Leg | Status |
|
|
|---|---|
|
|
| Hub v0.68.0 | **SHIPPED + DEPLOYED** — GitOps manifest bump `0.67.0→0.68.0`, ArgoCD `Synced/Healthy`, rollout complete, pod 1/1 |
|
|
| Agent v0.91.2 | shipped + published + deployed to felhom-pve |
|
|
| Hub v0.68.1 | **SHIPPED + DEPLOYED** — fixes the Configuration layout the v0.68.0 field broke |
|
|
| STOP-1 | done + verified |
|
|
| **STOP-3 — manifest save** | **DONE by the operator 2026-07-21 08:31:12Z.** All six fields persisted, including `artifact_wrapper_sha256 = 104db0a4…` (matches the agent's reported hash → drift gauge reads **ok**) and `artifact_min_agent = 0.91.2`. |
|
|
| **STOP-2 — live re-issue** | **DONE + PROVEN 2026-07-21.** Chain closed in **13 seconds**; see below. |
|
|
|
|
## STOP-2 — the proof (the whole task's reason to exist)
|
|
|
|
The operator pressed **Re-issue PBS credentials**. The identical click on 2026-07-18 did nothing.
|
|
|
|
```
|
|
hub 08:39:31Z fresh mint, generation 0 -> 1
|
|
descriptor gains "secret_generation": 1
|
|
(token_id + fingerprint BYTE-IDENTICAL — the re-key shape that was invisible)
|
|
agent 10:39:34 felhom-pbs-apply read felhom-pbs <- leg (b): the read that was impossible
|
|
agent 10:39:38 ERROR "REJECTED ... applied and DEAD" previous_state=applied
|
|
<- leg (c): the R-39 state, loud at last
|
|
hub 08:39:45Z consumed_at stamped
|
|
agent 10:39:45 "one-time token secret consumed" secret_len=36
|
|
<- leg (a): NO short-circuit
|
|
agent 10:39:45 felhom-pbs-apply reconcile (set-only, no --server)
|
|
agent 10:39:47 "pbsdr: converged" state=applied
|
|
```
|
|
|
|
(host CEST = UTC+2; hub timestamps UTC.)
|
|
|
|
**Corroboration:** the agent marker hash moved to `afbb3b41…` — in the failure it was *byte-identical*
|
|
to the pre-reissue marker, which was the single-line proof of the defect. `consumed_at` stamped. The
|
|
on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`. A live probe with the NEW
|
|
credential returns **200**. Three consecutive hub reports trace the entire state machine
|
|
`applied → auth_failed → applied`. **Zero** `pbsdr_selfheal` escalations fired, with exactly ONE mint,
|
|
ONE consume and no `consumed-failed.json` — the box healed through the descriptor path before the
|
|
damper was ever needed.
|
|
|
|
**First attempt, worth recording:** the operator initially pressed the **offsite** re-issue — there are
|
|
two distinct Re-issue actions and my instruction said only "press Re-issue". Harmless to PBS-DR, but it
|
|
rotated the restic password and correctly marked the escrow **stale**, so the recovery-code ceremony
|
|
had to be re-run (done). Name the surface explicitly in future runbook steps.
|
|
|
|
## What shipped hub-side
|
|
|
|
- **`host_pbs_secrets.generation`** — a monotonic per-host counter advanced by every fresh MINT and
|
|
by nothing else, stamped into the descriptor as `secret_generation`. Since the agent re-applies on
|
|
the descriptor's CONTENT HASH and a re-key returns byte-identical `token_id` / `fingerprint` /
|
|
`datastore` / `namespace`, this is the only field that moves — and therefore the thing that
|
|
re-arms a converged agent.
|
|
- A **re-stage** deliberately does not advance it (same secret, unchanged descriptor content).
|
|
- `omitempty` is load-bearing: emitting a zero would shift every pre-existing descriptor's hash at
|
|
once — a fleet-wide spurious re-apply.
|
|
- **Deviation from spec, deliberate:** the brief said to reuse "the new row's id … no schema
|
|
change". There is no row id — the table is `host_id PRIMARY KEY`, UPSERTed last-write-wins — and
|
|
`created_at` collides for two mints in one second. An additive counter column (existing
|
|
idempotent `ALTER TABLE` idiom) is the only monotonic source. **Verified applied on the live DB
|
|
after deploy.**
|
|
- **`pbsdrheal` gains an `auth_failed` trigger** — a new trigger in the existing machine, escalating
|
|
to a fresh mint (never a re-stage, which would re-feed the secret PBS just rejected) through the
|
|
**existing damper**, so a 401 flap cannot become a secret-minting chain.
|
|
- **`consumed_at` honesty gauge** — an unconsumed secret past a 15-minute grace under a box reporting
|
|
`applied` is the exact 2026-07-18 fingerprint and a disagreement **no single tier can detect
|
|
alone**. Surfaced with its own event, deliberately as a SURFACE not a heal: auto-re-issuing would
|
|
mint a second secret on top of an unconsumed one, which is the mint/consume race R-39(a) recorded.
|
|
- **Corrected a comment that stated a falsehood** — `ReissuePBSDR` claimed it refreshed the
|
|
descriptor "with the NEW token_id/fingerprint". False for a re-key, and believing it is why nobody
|
|
expected the descriptor to come back identical.
|
|
- **R-50b(a)** — `ArtifactManifest.WrapperSHA256` + operator field + host-page drift surface. **An
|
|
unknown on either side reads as quiet, never as drift.**
|
|
|
|
## Method notes worth keeping
|
|
|
|
- **P2 confirmed GitOps-only, and the trap is real:** `build.sh` itself prints
|
|
`kubectl set image …` as its deploy hint, contradicting `CLAUDE.md`. Not used. Worth fixing in the
|
|
script — it will mislead exactly the session that trusts tool output over the runbook.
|
|
- **P4 read the fleet from a temporary copy of the hub DB**, which carries live credentials
|
|
(`api_key`, `host_pbs_secrets.value`). Copy shredded immediately after each read. Result: **one
|
|
enrolled host**, so the MinAgent raise strands nobody.
|
|
- Tests include a **flow-level** `ReissuePBSDR` test against a fake that models a real re-key
|
|
(identical token/fingerprint, rotated secret only). Its red-proof fails on the assertion with both
|
|
byte-identical blocks printed — the July-18 defect reproduced in a unit test.
|
|
|
|
## For the operator
|
|
|
|
**STOP-2 (one click):** press **Re-issue PBS credentials** for the demo customer. Expected:
|
|
fresh secret row → `secret_generation` **0 → 1** (the live descriptor has no such key today) → poke →
|
|
agent re-applies with **no short-circuit** → fresh secret consumed → reconcile rc-0 → probe 200 → tier
|
|
`active`. The July-18 negative — the same click doing nothing — is the historical red-proof.
|
|
|
|
**STOP-3 (manifest save):** Agent `0.91.2` / sha256 `34d309be429473f3f0ab34e3185e17b22463a341b30bf46e162306bff4aec22a` /
|
|
**PBS wrapper sha256** `104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3` /
|
|
MinAgent `0.91.2`. Superseded, do not vouch: 0.91.0 (inert probe leg), 0.91.1. `0.90.1` correctly
|
|
stays 404.
|
|
|
|
## Residual
|
|
|
|
R-50b **(b)/(c) remain open** — the wrapper is still fetched unversioned from `raw/branch/main`; this
|
|
release makes drift visible, it does not fix the channel. The 0440 sudoers file is not agent-readable,
|
|
so its drift stays invisible. The DR-tier capability-map row is deliberately **not** upgraded to
|
|
PROVEN-LIVE until STOP-2 supplies the evidence.
|