F2 claim re-issue on clean-slate re-enroll (ReissueForReenroll, host-enroll mint path,
single-bump, reset code; hub never stores the password so fork B). F3 offsite re-issue on
re-enroll (ReissueOffsiteForCustomer, same machinery as the manual button). 2.3 escrow honesty
(red-proofed): re-issuing offsite marks the escrow stale (MarkEscrowStale), withholds the
mismatched restic hash from auto-confirm, DR checklist shows stale not done. Events:
claim_reissued_reenroll / offsite_reissued / escrow_stale.
Controller + scripts unchanged (source contradicted both premises): the controller reads escrow
prereqs live from the agent; the installer can't know the descriptor-provisioned storage id. F4
root fix is agent-side -> ROADMAP R-22; demo unblocked live (Part 0 ACL grant). VALIDATION doc
F2 erratum + F3/F4 dispositions. Green gate + Scenario-C red-proof pass.
Supervised clean-slate reinstall of the demo N100 from a pipeline-built Felhom ISO +
onboarding rehearsal. Core objectives GREEN: first-boot chain reached rc-0 on the first
attempt on real hardware (closes slice A's operator-gated boundary), serial-filter safety
proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict -> key on
MAC+UUID. Hard obstacle was firmware not the pipeline: this cheap AMI AN3PLUS 0.01 board
won't UEFI-boot the ISO's GRUB from USB (relocation 0x0; SB-off + shim-bypass don't help),
worked around live with a grub-mkimage loader from the box's own GRUB (slice-B input).
7 findings logged (F1 firmware loader HIGH; F2/F3/F4 reused-customer edges -> R-1/Peti;
F5 guest-RAM, F6 drive-init mount, F7 back-route). Full doc in documentation/tests/;
ROADMAP R-21 + REPORT + CONTEXT updated. No production code changed.
scripts/iso/: a DooPlex pipeline (build-felhom-iso.sh + Dockerfile.assistant) that
turns the official PVE ISO into a Felhom auto-install ISO whose first-boot stub
installs a retry-forever felhom-bootstrap unit which unattended-fetches
felhom-host-install.sh from the public felhom.eu/scripts channel and runs it until the
host is enrolled + a guest provisioned. host-install is UNMODIFIED (invoked only).
- build gates the answer on validate-answer OUTPUT text, never $? (spike S1 exit-0 trap)
- stub is from-iso, fully-up, exactly-once; retry unit owns all network work (S8a)
- retry-vs-resume encoded once: plain first, --resume when install state exists (v1.11.3)
- secret-bearing (embeds the retrieval passphrase): supervised/single-use; env shredded on success
Validated on VM 310: build gate + red-proof, disk-filter fail-safe, chain + retry,
resume-decision, exactly-once, no-net retry+recovery. Terminal host-install rc-0 success
operator-gated (drill customer needs the password-gated create-UI). scripts v1.16.0;
ROADMAP R-21 -> in-progress. Detail in REPORT.md.
Opened every citation in 00-capability-map.md and judged each of the 55 rows
against the specific scenario (not topic adjacency). 23 confirmed (incl. 5
MISSING proven-absent by code sweep), 26 citation-corrected, 6 demoted; no
upgrades. Status: 32 PROVEN-LIVE / 15 IMPLEMENTED / 3 PARTIAL / 5 MISSING
(was 38 PROVEN-LIVE).
Demotions PROVEN-LIVE->IMPLEMENTED: B6 (post-deploy config — T-PAGE-ALL is
render-only), E7 (FileBrowser access — no browse/download proof), F1 (health
email — cited P3-DELIVERY is the enlarge-block trigger), F4 (metrics — cited
legs are H1/H2 harness artifacts), F6 (operator alerting — no corpus doc),
G6 (offsite fill/staleness/freeze — no live-fired leg, freeze inconclusive).
Key corrections: D6 cited CAMPAIGN-3 F10/F11/F12 which are the CRITICAL
failures — fixes shipped agent v0.85 and were re-validated live in CAMPAIGN-4
/6A; C6 cited offbox-restic doc for a PBS-restore claim -> CAMPAIGN-2 T-P9;
C1/C5/B5 cited defect-discovery docs instead of the fix-validation legs.
Closed ROADMAP R-20: operator-key pinning is fully automatic in day-0
(felhom-host-install.sh step_agent_config, L2044/2146-2156/2332-2337).
Header draft caveat swapped for the verified stamp. Full per-row verdict
table in REPORT.md.
The ~215 lines flagged as "unknown uncommitted code" (hub/internal/notify/,
claim/engine.go, store.go) were in-flight files from a parallel CC session on
the customer-claim arc, snapshotted before it committed. All landed cleanly
(6b40eb8 v0.50.0, a1d0450 v0.54.0, e205a2d); v0.55.0 shipped. Tree is clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
Closes the two deferred deep campaigns from 6D on live infra: tier-2 v2 layout
engine (A1-A4) and F7 atomic-write + restic C2 self-heal (B1-B4). No production
code shipped. Two LOW findings (F-6E-1 demo drive drift; T-6E-1 DB-dump dir-fsync
parity). Drill repo torn down; fixtures reverted; box healthy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Reconciler observed waiting_secret across two reports (16:52+17:07 UTC), re-staged
the stored secret at 17:10:00 (no ep0 token, no gen bump); agent re-consumed
17:10:26 and converged (applied) 17:10:28 — no operator click. Demo host untouched
(scoped). Fleet widening remains a deliberate follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Deploy the PBS-DR self-heal reconciler scoped to the drill guest only for the
first supervised run; widen to the fleet (remove the env) in a follow-up sync
once the converged-host no-op is confirmed live.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Implements SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44). A box re-installed/rolled
back onto its stable host_id loses its agent-side converged marker; the hub
keeps the enabled descriptor + a CONSUMED one-time secret, the WG peer persists
(changed==false, cascade can't re-fire), so the agent sits in waiting_secret
forever. The missing piece is a consumable secret, not the descriptor.
New internal/pbsdrheal reconciler (5m, wgsync shape): for enabled+provisioned
hosts whose latest report pbs_dr.state is a stuck state past a >=2-distinct-report
debounce, re-stage the stored secret (store.RestageHostPBSSecret: clear
consumed_at, no ep0 call, NO generation bump); escalate to Re-issue (web
ReissuePBSDR) only when no secret is stored or the agent reports consumed_failed.
Converged/disabled/verify_failed/DR-OFF = no-op. PBSDRHEAL_ONLY_HOST scopes a
supervised rollout. Scenarios A-F + all six red-proofs verified. No agent change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Refutes the surviving theory: on the drill host the pbs_dr descriptor is present
and enabled, the WG peer exists, verify passes — the block is a one-time secret
consumed 07-12 that no path re-mints after the agent loses its converged marker
(re-install/rollback; same WG pubkey -> changed==false -> cascade can't re-fire).
Live-proven: staging any consumable secret converges in one tick (existing token,
zero ep0 churn); re-asserting a converged descriptor is a clean idempotent no-op.
Safe re-trigger = re-serve a secret gated on agent waiting_secret/consumed_failed,
never blind-timer Reissue. Design-inputs table handed to the self-heal TASK spec.
Drill left CONVERGED (P-DAY0-DEEP PBS-DR leg now GREEN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Fresh-appliance PBS-DR consume races the WG handshake at install (fingerprint probe timeout),
consumes nothing, and does not re-request the descriptor once the tunnel is up; reused-peer
re-provision never re-issues it. No operational re-trigger. Escrow ceremony + offsite round-trip
blocked on this fresh guest → friend-alpha alpha #1 escrow must be supervised live (customer-holds-R).
Drill deviation (CC-held R) never exercised. Recorded, not fixed inline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
immich enlargement blocked at real 50 GiB quota via sparse mandatory fixture; unit-only
continues; EnlargedBlocked edge-fires once; both operator + customer emails delivered;
fully reverted. G4 notification email restored via the real form (wipe-bug regression).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Core-promise legs proven on real data: .fab >=1GiB full circle (byte-identical mandatory,
zero sibling contamination, DB round-trip) + SQ3 immich offsite restore-to-live (functional
from offsite alone). Deferred: P3-BROWSER/P3-DELIVERY/P-TIER2-deep/P4-timing/P-DAY0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Unattended close-out of the C6 remainder. Flagship finding C6B-F1 (CRITICAL):
.fab export silently produces hollow, data-free bundles for the standard
${USERDATA_PATH} needs_hdd convention (12/13 media apps) — sonarr 4.17 GB -> a
2308-byte config-only bundle, past the v0.125.0 anti-hollow guard. Transport
(upload/download, real CF edge, 413, byte-integrity) all PASS. C6B-F2 (MED,
confounded): net-storage removal leaves an orphaned autofs mount. Browser-
interactive + deep backup-tier items split to CAMPAIGN-6C. No code changed; no
secret/R/blob committed; campaign credential left for Viktor to rotate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
CAMPAIGN-5 findings doc (2026-07-14): NAS re-arm ring core + v0.129.0 fix live-proof
Findings-only. v0.129.0 F-A/F-B/F-C all CONFIRMED FIXED live on the fleet. NAS
re-arm ring core PASS on a fresh campaign NFS share: F10 start-limit clear +
reset-failed+rearmed, F9 per-share verdict (no empty sweep), F1/F2 clean removal,
F12 unit cycle-free. F8 improved (stub/mounted:false during outage) but reachable:true
still server-level. Reboot half of the matrix + upload/browser planes deferred with
procedures. No credential/R/blob committed.
Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd
@
hub v0.54.0: change operator login password from the Configuration UI
Adds a "Login password" card on /configuration. The password was previously
settable only via the hub-config ConfigMap (auth.password_hash) + redeploy.
- store: hub_settings key operator_password_hash + Get/SetOperatorPasswordHash
- server: passwordHash field -> configPasswordHash (seed); new
effectivePasswordHash() (DB override wins, else seed) is now the single
source for the CSRF gate, RequireAuth, and handleLogin
- POST /configuration/password (handleChangePassword): requires current
password, 8-72 byte new + confirm, bcrypt cost 10, persists DB override;
existing sessions kept valid; ConfigMap stays the break-glass reset path
- UI: current/new/confirm form + inline mismatch pre-check + 6 flashes
- tests + red-proofs: override precedence, happy-path via handleLogin,
wrong-current rejection, mismatch/too-short/no-op, template render
- docs: CHANGELOG, README (auth+config), REUSE, REPORT
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd
@
The global bearer (api.report_api_key) leaves manifests/hub.yaml: the
ConfigMap field is an empty placeholder; the Deployment injects
REPORT_API_KEY from out-of-band Secret/report-api (deliberately NOT
optional - a missing Secret fails Ready rather than booting a hub with an
empty bearer). main.go gains the REPORT_API_KEY env override (the
RESEND_API_KEY twin). scripts/manifest_secret_gate.py blocks bearer-shaped
(64-hex) literals across manifests/ (felhom.secret.yaml's pre-existing
umami APP_SECRET is a visible non-fatal KNOWN-BACKLOG line - secrets.md
backlog, out of scope here).
Ordering (load-bearing): create Secret/report-api with the CURRENT value
BEFORE syncing; this manifest change rides the same sync as the 0.53.0
image (older images ignore the env override). The exposed git-history copy
dies only with the supervised ROTATION - procedure + consumer list in
documentation/runbooks/secrets.md ("Operator/global bearer key"); both
publish runbooks' ROTATION notes now point at that disposition. The
controller repo's example-config copy of the literal is scrubbed
(felhom-controller 51c871a).
Red-proof E: reintroducing a 64-hex literal into manifests/hub.yaml ->
gate exit 1 (recorded); restored clean.
Customer status (dashboard row, /configs list, detail header + strip) is
now worst(controllerDerived, hostStatusOf(each expected host)) via the ONE
staleness definition (Server.hostStatus, hosts.go - shared with the
HostStalenessChecker; no second threshold). Any host down/stale caps the
customer at WARN with a cause chip naming the host ("host down: <id>");
pending (never-reported) hosts worsen only once the customer has reported
(onboarding exclusion). The three previously-inlined controller-status
chains collapse into controllerStatus() (rollup.go). Display + derivation
only - checker alerting untouched.
Live shape pinned (drill-1 / Peti cluster): host down 23h + controller
report minutes old rendered a GREEN row - TestRollup_DeadHostMasking now
fails that exact outcome. Red-proof: short-circuiting foldHostStatus to
controller-only flips C + two D subtests red ("dashboard row is GREEN
over a 23h-dead host").
pbsdrProvisionAtom, on tenantsync token_exists: consult the customer's
MOST RECENT host-deletion record. escrow_acked=true -> invoke the
EXISTING tenantsync Reissue op, store the audit event (pbsdr_auto_reissue,
hub-source, "Previous key destroyed (acknowledged deletion) - credentials
re-issued automatically."), proceed to secret+descriptor as a normal
provision. No record / un-acked -> the pre-existing refusal, byte-unchanged
(never-silently-re-key law; operator ruling 2026-07-13).
Scenario A test drives the REAL handler path over a REAL escrow-ack delete;
scenario B pins the exact non-effect (zero Reissue calls, no state).
Red-proof: bypassing the gate flips both B subtests to 303 (silent re-key).
Every DeleteHost writes a provenance row INSIDE the cascade tx: host_id,
customer_id, deleted_at, escrow_acked. escrow_acked = ack given over a
PRESENT escrow row (acknowledged destruction, not a vacuous checkbox).
LatestHostDeletion(customer) serves the F-14 gate — newest record only,
so an old acked deletion never whitelists a newer un-acked one. No
backfill by design: pre-record deletions stay on the manual path.
Red-proof: dropping the in-tx INSERT fails TestDeleteHost_ProvenanceRecord
("no deletion record written") + the part-2 scenario-A test.
admin