Reconciler observed waiting_secret across two reports (16:52+17:07 UTC), re-staged
the stored secret at 17:10:00 (no ep0 token, no gen bump); agent re-consumed
17:10:26 and converged (applied) 17:10:28 — no operator click. Demo host untouched
(scoped). Fleet widening remains a deliberate follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Deploy the PBS-DR self-heal reconciler scoped to the drill guest only for the
first supervised run; widen to the fleet (remove the env) in a follow-up sync
once the converged-host no-op is confirmed live.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Implements SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44). A box re-installed/rolled
back onto its stable host_id loses its agent-side converged marker; the hub
keeps the enabled descriptor + a CONSUMED one-time secret, the WG peer persists
(changed==false, cascade can't re-fire), so the agent sits in waiting_secret
forever. The missing piece is a consumable secret, not the descriptor.
New internal/pbsdrheal reconciler (5m, wgsync shape): for enabled+provisioned
hosts whose latest report pbs_dr.state is a stuck state past a >=2-distinct-report
debounce, re-stage the stored secret (store.RestageHostPBSSecret: clear
consumed_at, no ep0 call, NO generation bump); escalate to Re-issue (web
ReissuePBSDR) only when no secret is stored or the agent reports consumed_failed.
Converged/disabled/verify_failed/DR-OFF = no-op. PBSDRHEAL_ONLY_HOST scopes a
supervised rollout. Scenarios A-F + all six red-proofs verified. No agent change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Refutes the surviving theory: on the drill host the pbs_dr descriptor is present
and enabled, the WG peer exists, verify passes — the block is a one-time secret
consumed 07-12 that no path re-mints after the agent loses its converged marker
(re-install/rollback; same WG pubkey -> changed==false -> cascade can't re-fire).
Live-proven: staging any consumable secret converges in one tick (existing token,
zero ep0 churn); re-asserting a converged descriptor is a clean idempotent no-op.
Safe re-trigger = re-serve a secret gated on agent waiting_secret/consumed_failed,
never blind-timer Reissue. Design-inputs table handed to the self-heal TASK spec.
Drill left CONVERGED (P-DAY0-DEEP PBS-DR leg now GREEN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Fresh-appliance PBS-DR consume races the WG handshake at install (fingerprint probe timeout),
consumes nothing, and does not re-request the descriptor once the tunnel is up; reused-peer
re-provision never re-issues it. No operational re-trigger. Escrow ceremony + offsite round-trip
blocked on this fresh guest → friend-alpha alpha #1 escrow must be supervised live (customer-holds-R).
Drill deviation (CC-held R) never exercised. Recorded, not fixed inline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
immich enlargement blocked at real 50 GiB quota via sparse mandatory fixture; unit-only
continues; EnlargedBlocked edge-fires once; both operator + customer emails delivered;
fully reverted. G4 notification email restored via the real form (wipe-bug regression).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Core-promise legs proven on real data: .fab >=1GiB full circle (byte-identical mandatory,
zero sibling contamination, DB round-trip) + SQ3 immich offsite restore-to-live (functional
from offsite alone). Deferred: P3-BROWSER/P3-DELIVERY/P-TIER2-deep/P4-timing/P-DAY0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Unattended close-out of the C6 remainder. Flagship finding C6B-F1 (CRITICAL):
.fab export silently produces hollow, data-free bundles for the standard
${USERDATA_PATH} needs_hdd convention (12/13 media apps) — sonarr 4.17 GB -> a
2308-byte config-only bundle, past the v0.125.0 anti-hollow guard. Transport
(upload/download, real CF edge, 413, byte-integrity) all PASS. C6B-F2 (MED,
confounded): net-storage removal leaves an orphaned autofs mount. Browser-
interactive + deep backup-tier items split to CAMPAIGN-6C. No code changed; no
secret/R/blob committed; campaign credential left for Viktor to rotate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
CAMPAIGN-5 findings doc (2026-07-14): NAS re-arm ring core + v0.129.0 fix live-proof
Findings-only. v0.129.0 F-A/F-B/F-C all CONFIRMED FIXED live on the fleet. NAS
re-arm ring core PASS on a fresh campaign NFS share: F10 start-limit clear +
reset-failed+rearmed, F9 per-share verdict (no empty sweep), F1/F2 clean removal,
F12 unit cycle-free. F8 improved (stub/mounted:false during outage) but reachable:true
still server-level. Reboot half of the matrix + upload/browser planes deferred with
procedures. No credential/R/blob committed.
Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd
@
hub v0.54.0: change operator login password from the Configuration UI
Adds a "Login password" card on /configuration. The password was previously
settable only via the hub-config ConfigMap (auth.password_hash) + redeploy.
- store: hub_settings key operator_password_hash + Get/SetOperatorPasswordHash
- server: passwordHash field -> configPasswordHash (seed); new
effectivePasswordHash() (DB override wins, else seed) is now the single
source for the CSRF gate, RequireAuth, and handleLogin
- POST /configuration/password (handleChangePassword): requires current
password, 8-72 byte new + confirm, bcrypt cost 10, persists DB override;
existing sessions kept valid; ConfigMap stays the break-glass reset path
- UI: current/new/confirm form + inline mismatch pre-check + 6 flashes
- tests + red-proofs: override precedence, happy-path via handleLogin,
wrong-current rejection, mismatch/too-short/no-op, template render
- docs: CHANGELOG, README (auth+config), REUSE, REPORT
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd
@
The global bearer (api.report_api_key) leaves manifests/hub.yaml: the
ConfigMap field is an empty placeholder; the Deployment injects
REPORT_API_KEY from out-of-band Secret/report-api (deliberately NOT
optional - a missing Secret fails Ready rather than booting a hub with an
empty bearer). main.go gains the REPORT_API_KEY env override (the
RESEND_API_KEY twin). scripts/manifest_secret_gate.py blocks bearer-shaped
(64-hex) literals across manifests/ (felhom.secret.yaml's pre-existing
umami APP_SECRET is a visible non-fatal KNOWN-BACKLOG line - secrets.md
backlog, out of scope here).
Ordering (load-bearing): create Secret/report-api with the CURRENT value
BEFORE syncing; this manifest change rides the same sync as the 0.53.0
image (older images ignore the env override). The exposed git-history copy
dies only with the supervised ROTATION - procedure + consumer list in
documentation/runbooks/secrets.md ("Operator/global bearer key"); both
publish runbooks' ROTATION notes now point at that disposition. The
controller repo's example-config copy of the literal is scrubbed
(felhom-controller 51c871a).
Red-proof E: reintroducing a 64-hex literal into manifests/hub.yaml ->
gate exit 1 (recorded); restored clean.
Customer status (dashboard row, /configs list, detail header + strip) is
now worst(controllerDerived, hostStatusOf(each expected host)) via the ONE
staleness definition (Server.hostStatus, hosts.go - shared with the
HostStalenessChecker; no second threshold). Any host down/stale caps the
customer at WARN with a cause chip naming the host ("host down: <id>");
pending (never-reported) hosts worsen only once the customer has reported
(onboarding exclusion). The three previously-inlined controller-status
chains collapse into controllerStatus() (rollup.go). Display + derivation
only - checker alerting untouched.
Live shape pinned (drill-1 / Peti cluster): host down 23h + controller
report minutes old rendered a GREEN row - TestRollup_DeadHostMasking now
fails that exact outcome. Red-proof: short-circuiting foldHostStatus to
controller-only flips C + two D subtests red ("dashboard row is GREEN
over a 23h-dead host").
pbsdrProvisionAtom, on tenantsync token_exists: consult the customer's
MOST RECENT host-deletion record. escrow_acked=true -> invoke the
EXISTING tenantsync Reissue op, store the audit event (pbsdr_auto_reissue,
hub-source, "Previous key destroyed (acknowledged deletion) - credentials
re-issued automatically."), proceed to secret+descriptor as a normal
provision. No record / un-acked -> the pre-existing refusal, byte-unchanged
(never-silently-re-key law; operator ruling 2026-07-13).
Scenario A test drives the REAL handler path over a REAL escrow-ack delete;
scenario B pins the exact non-effect (zero Reissue calls, no state).
Red-proof: bypassing the gate flips both B subtests to 303 (silent re-key).
Every DeleteHost writes a provenance row INSIDE the cascade tx: host_id,
customer_id, deleted_at, escrow_acked. escrow_acked = ack given over a
PRESENT escrow row (acknowledged destruction, not a vacuous checkbox).
LatestHostDeletion(customer) serves the F-14 gate — newest record only,
so an old acked deletion never whitelists a newer un-acked one. No
backfill by design: pre-record deletions stay on the manual path.
Red-proof: dropping the in-tx INSERT fails TestDeleteHost_ProvenanceRecord
("no deletion record written") + the part-2 scenario-A test.
admin