F2 claim re-issue on clean-slate re-enroll (ReissueForReenroll, host-enroll mint path,
single-bump, reset code; hub never stores the password so fork B). F3 offsite re-issue on
re-enroll (ReissueOffsiteForCustomer, same machinery as the manual button). 2.3 escrow honesty
(red-proofed): re-issuing offsite marks the escrow stale (MarkEscrowStale), withholds the
mismatched restic hash from auto-confirm, DR checklist shows stale not done. Events:
claim_reissued_reenroll / offsite_reissued / escrow_stale.
Controller + scripts unchanged (source contradicted both premises): the controller reads escrow
prereqs live from the agent; the installer can't know the descriptor-provisioned storage id. F4
root fix is agent-side -> ROADMAP R-22; demo unblocked live (Part 0 ACL grant). VALIDATION doc
F2 erratum + F3/F4 dispositions. Green gate + Scenario-C red-proof pass.
Supervised clean-slate reinstall of the demo N100 from a pipeline-built Felhom ISO +
onboarding rehearsal. Core objectives GREEN: first-boot chain reached rc-0 on the first
attempt on real hardware (closes slice A's operator-gated boundary), serial-filter safety
proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict -> key on
MAC+UUID. Hard obstacle was firmware not the pipeline: this cheap AMI AN3PLUS 0.01 board
won't UEFI-boot the ISO's GRUB from USB (relocation 0x0; SB-off + shim-bypass don't help),
worked around live with a grub-mkimage loader from the box's own GRUB (slice-B input).
7 findings logged (F1 firmware loader HIGH; F2/F3/F4 reused-customer edges -> R-1/Peti;
F5 guest-RAM, F6 drive-init mount, F7 back-route). Full doc in documentation/tests/;
ROADMAP R-21 + REPORT + CONTEXT updated. No production code changed.
scripts/iso/: a DooPlex pipeline (build-felhom-iso.sh + Dockerfile.assistant) that
turns the official PVE ISO into a Felhom auto-install ISO whose first-boot stub
installs a retry-forever felhom-bootstrap unit which unattended-fetches
felhom-host-install.sh from the public felhom.eu/scripts channel and runs it until the
host is enrolled + a guest provisioned. host-install is UNMODIFIED (invoked only).
- build gates the answer on validate-answer OUTPUT text, never $? (spike S1 exit-0 trap)
- stub is from-iso, fully-up, exactly-once; retry unit owns all network work (S8a)
- retry-vs-resume encoded once: plain first, --resume when install state exists (v1.11.3)
- secret-bearing (embeds the retrieval passphrase): supervised/single-use; env shredded on success
Validated on VM 310: build gate + red-proof, disk-filter fail-safe, chain + retry,
resume-decision, exactly-once, no-net retry+recovery. Terminal host-install rc-0 success
operator-gated (drill customer needs the password-gated create-UI). scripts v1.16.0;
ROADMAP R-21 -> in-progress. Detail in REPORT.md.
Audit doc for the Day-0 VM drill: appliance install, floor-at-first-report,
escrow ceremony + auto-confirm FIRST LIVE FIRING (~7.5 min, zero clicks),
offsite backup + restore round-trip, PBS-DR/WG fork (Peti-sequence rehearsal).
Corrects day0-install.md A.2 (git creds optional since v1.11.2, anonymous
fetch is the normal shape) and RUNBOOK-escrow-ceremony.md (identity-only mode
does NOT exist — F-6; age prereq — F-10). REPORT.md overwritten; CONTEXT.md
one-liner added.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
journalctl proved boot -1 came up at 23:31 WITHOUT networking: the agent automount
template's After/Wants=network-online.target cycles with local-fs.target and systemd
deleted networking.service to break it; the 06:45 power-cycle boot hit the same cycle
and deleted the automount instead. Every boot with an enrolled share is a coin flip
until the template drops that ordering. Recovery executed: automount started, apps
healed, privatebin removed via real flow, verification backup green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
Full hub->Hetzner->controller loop validated on demo-felhom against the new
dedicated-project pool box; 2 snapshots on the new box, key-only. F1/F3 fixed
mid-run (hub v0.38.1, controller v0.106.1); F2/F4/F5/F6 recorded open.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Descriptor.HostFingerprint (SHA256, non-secret), captured at provision via an
x/crypto/ssh keyscan (SSHHostKeyScanner — dials :23, grabs the host key from the
handshake, no ssh binary). Fail-closed: nil scanner or scan failure → error (don't
serve a descriptor the controller can't verify). Pairs with controller v0.106.0
which re-scans + refuses on mismatch (no blind TOFU).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Full cycle on the demo: enable→stage→real escrow-create (K+restic-pw under a
fresh R, uploaded to hub, staged wiped, K byte-unchanged)→atomicity proof (run
skipped before confirm, proceeded after — gate opens exactly on confirm). DR
money shot: restic password recovered from the PRODUCTION hub blob (via
identity-consume with R) is byte-identical to the live pw (sha 24c44f49) and
decrypted the real box repo; inject→restore endpoints reproduced a 45MB db-dump
byte-perfect. Demo left escrowed (un-paused); R held out-of-band; no secret in
logs; repo_password unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Real endpoints, fresh box sub-account. Discovery proof: immich (undeployed,
USB-only unit — the v0.103.0 mis-resolution case) → ok/1 snapshot on the USB
path; 45MB db-dump restore sha256-matched. No-silent-success on the live box:
0/N → error + backup_failed alert; partial → ok + LastWarning. Persists across
restart. Forks 1+2 confirmed (direct; sub-account /home/<name>, ssh-copy-id -s
-f, console key field still broken F3); fork-4 custody surface recorded, escrow
design OPEN (repo pw + key not independently recoverable). Kept as standing
demo-offsite.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Uninstall: NEW 4b4 removes the self-update artifacts (guarded wrapper,
.prev/.new.* A/B slots, rollback unit, limits drop-in — derived from
configs/felhom-selfupdate-guarded); enrolled drives under /mnt/felhom-drives
unmounted plain-umount-only (busy = warn+guidance, never -l/-f; root bind
guarded); both modes end with a KEPT-vs-WIPED statement (drives/PBS/hub
record/escrow/vaulted recovery credential live on); guest-only mode prints
the vmid's bind-store drives.
Key-pin: OPERATOR_KEY_* constants (empty until the operator ceremony) +
--operator-pubkey-file (argv-validated; comment=key_id required; file
overrides constants) -> authz.signers per the agent SignerKey schema;
no-keys-resolved KEEPS preserved signers; verify reports armed/dormant WARN.
Harness: +13 static cases + GL4-D parity + GL4-INV + PVE-tier GL4 H-U
(full-uninstall dry transcript vs live 9201). 28/28 on felhom-pve;
red-proofs RP-1..3 run->fail->revert. GO-LIVE-PACKAGE.md ABSENT AGAIN ->
status in CONTEXT.md. Live teardown/armed-pin = GL-6.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Agent = the live felhom-pve bytes (sha 1ec3f588..76af05, round-trip verified);
golden baked clean-room in the drill VM (build-golden v2.0.0, controller
0.103.0, all integrity gates green, sha 8481e8a1..6026e) and published 201.
Both anon-fetch-verified (Gate 3c: packages world-readable -> G3 read-only
token rotation cannot break fetches; installer-side empty-token die is the
real precondition). Drill VM restored to virgin. OPERATOR NEXT: Day-0
manifest bump to the four recorded values.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
admin