Virgin-ISO nested drill closed the train: dead-NIC install baked the
fallback (incl. the dead default gateway), the R-59 screen painted
(capture committed beside the spike doc), the cable move healed +
registered at the hub in 23s unaided, and the build's rootpw file
matched the installed box's shadow hash. R-59 SHIPPED with the recorded
deviation (first-boot gate; installer-initrd abort out of scope by
operator ack). R-60 SHIPPED (spike + drill cited; F-P9 route-flush fix
included). R-61 slice 1 SHIPPED. New R-62 row (hub delete-dialog
cosmetics, XS). Capability map: new PROVEN-LIVE row (nested != metal,
said so). Cleanup verified: felhom-pve interfaces byte-identical,
bridge/VMs/ISO removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Live drill finding (nested leg, 2026-07-22): the installer's no-DHCP
fallback bakes not just the 192.168.100.2 static address but a DEFAULT
ROUTE via 192.168.100.1, and dhclient-script never replaces an existing
default route — so the sweep's hub probe rode the dead gateway and
failed even though the candidate's lease had landed, and the gate
oscillated (lease -> probe fail -> byte-identical restore -> screen).
The candidate probe now flushes vmbr0's addresses+routes after the
re-point, letting the lease install its own; the restore path and each
next candidate re-apply configured state via ifreload. Validated live
by hot-patching the drill box: heal in 33s from restart — self-heal
log, inet-dhcp persist, .felhom-bak, hub registration + pairing banner.
Full virgin-ISO drill follows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
network_gate() runs before mode dispatch. Hub reachable -> return with
ZERO new behavior (asserted). Unreachable -> diagnose (physical-NIC
table with raise-first carrier read; installer 192.168.100.2-fallback
signature named when present), then — ONLY while no install state file
exists, doubly confined by the unit's ConditionPathExists=!done-flag —
sweep: per candidate re-point vmbr0's bridge-ports (atomic tmp+mv,
always derived from the pristine copy) -> ifreload -> bounded dhclient
(20s) -> hub probe (any HTTP status = reachable; the hub answers 302).
First success persists winner + DHCP addressing (original kept as
interfaces.felhom-bak) and proceeds; total failure restores the config
BYTE-IDENTICALLY, paints a calm Hungarian console screen (spec copy,
print_pairing_banner pattern) and retries every 60s with the unit held
in activating — waiting is not failing (v1.21.0). Trigger is always
hub-unreachable, never the fallback signature (spike F-P6: the
truth-changed-after-install variant has no signature). Design inputs:
SPIKE-firstboot-nic-sweep-2026-07-22 F-P1..F-P8.
Harness: +4 scenarios (G1 zero-behavior in D incl. zero-sleeps +
fixture-untouched, G2 sweep-success/persist/proceed, G3 sweep-fail
byte-identical + screen content, G4 state-file-present = never sweep),
gate seams (FELHOM_NET_SYS/INTERFACES_FILE/CONSOLE_DEV), ip/ifreload/
dhclient fakes, hub-probe oracle incl. follow:<nic> mode. Red-proofs
run and restored: (A) restore-on-failure dropped -> G3 byte-identical
FAILs; (B) state gate removed -> G4 zero-call FAILs; (C) early return
removed -> all five G1 invariants FAIL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Nested VM 9310 on felhom-pve (throwaway vmbr9 rig): re-point
bridge-ports -> ifreload (0.36s, raises the port itself) -> dhclient
lease 3.3s -> hub HTTPS 302 in 160ms; a dead candidate costs exactly
its timeout bound (rc=124 at 20s). Bonus finding: the auto-installer
picks the NIC that leases and bakes the lease as STATIC — so wrong-NIC
boxes come in two variants (192.168.100.2 fallback bake vs
truth-changed-after-install), and the gate must trigger on
hub-unreachable, never on the fallback signature. Eight design inputs
(F-P1..F-P8) recorded for the Part-3 gate. STOP-0 discharged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
The old pairing/delivery scenarios expected a non-zero exit on an
unbound 204 poll — the one-poll-per-invocation design v1.21.0 (R-33)
deliberately removed; against the current script they would hang on a
real sleep. Now: a PATH-faked sleep counts the waits and flips the poll
to 200 after 3 cycles, so one scenario proves the whole v1.21.0 shape
in a single invocation (register -> in-script 204 waits -> delivery ->
host-install -> done-flag, exit 0), plus a 410 crash-window scenario
(still exits non-zero on purpose). Runaway guard: fake sleep kills the
loop after 25 calls. Assistant image gains python3 (the bootstrap's
JSON parsing needs it; PVE ships it on the real box) — the harness runs
in that image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
The ISO build now writes the minted throwaway root plaintext to a 0600
sibling file (<iso>.rootpw.txt: password + ISO name + build date) — the
single record of truth. Never stdout/logs/manifest; the manifest carries
only a pointer line. OUT_ISO naming hoisted above the mint (the sibling
is named after the ISO); the mint itself and ROOT_PLAIN's lifetime are
unchanged beyond the one file write. FELHOM_ISO_KEEP_WORK=1 debug escape
added for the harness. New test/rootpw-emission.sh: dry-run emission,
0600, plaintext<->answer-hash cross-check (openssl -6 -salt), no
plaintext on stdout, manifest-heredoc guard. Red-proof run: plaintext
injected into the manifest heredoc -> harness FAILs -> restored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Per the 2026-07-21 refresh brief: R-39 interim blocks (B4/E1) and the R-36
manual-Save block (C4) deleted — both shipped and proven live; freemail.hu
gate proven (R-4 COMPLETE); golden/floor-lift note now cites two shapes
(rehearsal + virgin HP t740 day-0 lift 0.153.0->0.156.0); A3 loader table
per operations/nodes.md (N100=mkimage/SB-off per record, HP t740=shim/SB
ENABLED); B2 multi-NIC cabled-port gotcha (R-59/R-60 pending); new A5 gate
(agent >=0.93.0 deployed box-side before the first escrow ceremony); D
offboarding pointer to §G (R-25b). DRAFT status and the C7 graduation gate
unchanged. ROADMAP R-25b pointer follows the rename.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Found validating v0.69.0 against the live hub. demo-vm-felhom was deleted
on 07-18 and was still on the Customers list AND still raising offsite_stale
(10 events, latest 07-21 17:34, operator email at 19:34) — because
GetCustomers() is report-derived and no lifecycle tier ever deleted a report.
New leg 3 (residue), before the record purge: reports, app_telemetry,
app_log_tails, log_tail_requests, customer_notifications, plus the
credential-bearing appliance_registrations and selfbind_tokens. Audit
(events, notification_log) and F-14 provenance still survive.
Ghost customers are now deletable: 404 means "nothing here", not "no config
row". With no config row the offsite descriptor is unknowable, so the Hetzner
and descriptor legs record skipped_no_config rather than a bare "skipped".
Two more red-proofs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
POST /configs/{id}/delete now runs hosts -> RESET -> purge behind three
acknowledgements, a typed customer-id, a stale-preview check and the
ONLINE-host refusal (every gate before any write, so a refusal has zero
side effects). The shallow handleConfigDelete is gone.
Two invariants are asserted, not just commented: ruling 3 is preserved by
construction (leg 2 never sees a host row) and retained escrow custody is
purged exactly once, in leg 3 (leg 2 runs with purgeEscrow=false).
handleCustomerReset's committed half was extracted as commitCustomerReset;
the standalone RESET path is byte-identical to v0.68.1 and its suite is
untouched. Five red-proofs run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
New documentation/operations/nodes.md: HP t740 hardware, disks (SanDisk system
SSD 182195804614; Toshiba 1TB NVMe 58BS11AFT8MQ PRESENT AND UNENROLLED, still
NTFS, do not touch), the five-NIC map and the trap that cost the first install,
and the access path - no SSH key is baked, auth is the hub-vaulted G1
break-glass password.
tailscale.md gains demo-hp, the operator-lab-exception warning so a future
product-shape audit does not conclude the product ships tailscale, and the
--accept-dns evidence: the join omitted the flag, MagicDNS rewrote
/etc/resolv.conf, and it was reverted. Harmless at the vacation site, would
have bitten silently at home where split-horizon matters.
OPEN: key expiry still enabled on demo-hp (2027-01-17) - needs an admin-console
toggle or an API token; a pre-auth key cannot do it.
R-59 no-DHCP install must hard-abort (it baked 192.168.100.2 static and
completed - a box that can never call home). R-60 first-boot NIC sweep
self-heal. R-61 the baked root password must be knowable; a fixed well-known
password is explicitly rejected.
Positive evidence same-session: R-21 slice C PROVEN on a SECOND, virgin board
(HP t740) - and the shim loader booted with Secure Boot ENABLED, retiring the
assumption that Felhom installs need SB off. Fresh-box floor lift
0.153.0 -> 0.156.0 during day-0 cited on the publish-train row.
ISO README gains the t740 five-NIC trap: the 4-port igb card gets no lease,
the onboard r8169 port does.
R-58 records the operator ruling (2026-07-21) with the argument verbatim: the
installer should list available storage devices, excluding the install media,
and let one be selected. Third ISO mode alongside unattended-serial and
match-nothing-safety; unattended stays the appliance/factory mode. Slice 1 is
the abort-screen candidate table, same enumeration code, and it collapses the
two-boot dance on its own. Matters most for BYO/reinstall, where the serial is
unknown and a wrong guess is destructive.
ISO README gains the HP section: shim proven on this board by the safety boot,
the uncommitted-armed-profile pattern, verify-from-inside-the-ISO, and a
pointer to the prior-LVM abort that is the one likely failure on a
second-hand disk.
R-55's reboot leg ran operator-present on 9201: immich UI-stopped -> stayed
stopped across pct reboot, calibre-web recreated, zero alerts, ~15s.
R-57 records the lifecycle mechanism with the operator's abandoned-app
requirements verbatim and plant-it as the motivating case, including why the
retired/ directory move was wrong and the v0.158.1 pointer-receiver defect.
PROMPT-TEMPLATE: standard 'For the operator' plain-language section, mandatory
for M+ tasks and anything with a STOP.
ROADMAP rulings (operator, 2026-07-21): R-25b full-teardown cascade with three
acks + typed name (M-sized, spec to follow, no longer blocks R-3); R-11 channel
= direct Messenger, doc is the architect's; R-42 option (a), sidecars follow the
app; R-17 delete the archive - spike-lite found NO tooling verb targets it, so
it is an operator console action; R-4 complete (freemail.hu verified).
R-55 + R-41 slice 1 marked shipped; new R-56 (app difficulty classification -
the constructive half of the glance ruling).
scripts/build-hub.sh v1.23.0: the hub build script was outside any repo. Adopted
verbatim + versioned; the build-dir path is now a symlink to it.
felhom-testing skill: the ~1/5 recovery-code 'known flake' is retired - it was a
real defect the test was correctly detecting.
Mirrors profiles/n100.profile with one deliberate difference: the loader
default stays at the stock signed shim chain. mkimage exists to work around
the N100's AMI firmware GRUB relocation defect, not as a Felhom requirement,
and it costs Secure Boot. HP business firmware should take shim - so try it
first and carry mkimage as the fallback.
Ships the match-nothing disk filter, so built as-is it IS the hp-safety
pre-flight ISO. Arming instructions point at an uncommitted working profile,
per the n100-demo precedent.
Dead primary: degraded in 13 s, exactly one app_start_failed, banner rendered and
self-cleared. Boot orphan: recovered in one attempt with zero alerts. Dead dhclient:
detected in 57 s on process liveness while the lease was still live, healed 120 s after the
kill — the tunnel never dropped, so the outage was prevented rather than observed.
P1 answered as a by-product: bookstack StartedAt == the moment bootrecon StartStack
returned, so unless-stopped did NOT resurrect it. F5 hypothesis confirmed.
New R-55, surfaced by the leg designed to prove the opposite: the boot bind gate recreates
and STARTS every deployed drive-backed app unconditionally, so a customer Stop does not
survive a reboot for those apps. Predates R-52 and does not implicate it, but it narrows
R-52's practical scope and needs a ruling.
R-51's roadmap diagnosis is corrected at the source: aggregation returned StateRunning
("partial") for a running/stopped mix, so the stack read RUNNING and IsDownState was never
consulted about at all — the constraint that row protects was never in tension
with the fix.
New R-54 row closes the INCIDENT-guest-dhclient-killed-2026-07-20 §5 OPEN RISK, and records
the design fact that makes it work: liveness of the DHCP client is itself a probe, because
the damage is timed and the address outlives its cause by 1-2 hours. The static-guest leg is
deliberately deferred to R-50.
New capability-map row is IMPLEMENTED, not PROVEN-LIVE: one leg is live (the watchdog's
healthy cycle on felhom-pve), the three that matter are destructive and operator-present and
have not run.
PROMPT-TEMPLATE §10 gains the seam-discipline row, including that a strings.Contains source
assertion is NOT sufficient — a commented-out call still contains the string.
The operator pressed Re-issue PBS credentials and the chain closed in 13 seconds. The
identical click on 2026-07-18 did nothing at all.
hub 08:39:31Z fresh mint, generation 0 -> 1; descriptor gains secret_generation: 1
(token_id + fingerprint BYTE-IDENTICAL — the invisible re-key shape)
agent 10:39:34 felhom-pbs-apply read felhom-pbs (leg b: the impossible read)
agent 10:39:38 ERROR REJECTED ... applied and DEAD, previous_state=applied
(leg c: the R-39 state, loud)
hub 08:39:45Z consumed_at stamped
agent 10:39:45 one-time token secret consumed (leg a: NO short-circuit)
agent 10:39:45 reconcile (set-only, no --server)
agent 10:39:47 pbsdr: converged state=applied
Corroboration: marker hash moved to afbb3b41… (it was byte-identical to the pre-reissue
marker in the failure); secret mtime 2026-07-18 -> 2026-07-21 10:39:45; new credential
probes 200; three consecutive reports trace applied -> auth_failed -> applied; ZERO
self-heal escalations, one mint, one consume, no consumed-failed.json — the box healed
through the descriptor path before the damper was ever needed.
Recorded for future runbooks: the operator first pressed the OFFSITE re-issue (two
distinct Re-issue actions exist). Harmless to PBS-DR, but it rotated the restic password
and correctly marked the escrow STALE, so the ceremony had to be re-run. Name the surface
explicitly next time.
The v0.68.0 row wrapped itself in a <div>, but the artifacts <form> IS the CSS grid
(display:grid, no inner container). The stray </div> closed the surrounding card from
inside the form and the new <div> was never closed — it swallowed the submit button and
ran to </form>, so the row rendered outside the card and Save landed inline. Reported by
the operator on first use.
The field still submitted (it stayed inside the form), so this was layout damage rather
than data loss, but the unbalanced markup put every section below it in the wrong
container.
Fixed as plain grid cells (grid-column: 2/4), no nested elements.
There was no render assertion on this form at all, which is why a hand-edit broke it
silently. The new test asserts the field is inside the form, the button has not escaped,
the form contains ZERO divs, whole-page div balance holds, and the sections after it
survive. Red-proofed against the broken shape.
R-39's three legs are closed and deployed: the hub stamps a monotonic secret_generation
so a re-key finally moves the descriptor hash; the wrapper gains a narrow read verb so
the non-root agent can read the credential it writes; and ProbeAuth turns a 401 into a
loud auth_failed the existing damper escalates to a fresh mint. Plus a consumed_at
honesty gauge for the applied-but-never-consumed disagreement.
Recorded in the R-39 row, because both are the kind of thing a future reader needs:
- A load-bearing fact the spec did not flag, checked rather than trusted: Apply bails out
if the storage status probe ERRORS and adopt converges without consuming when the
storage reads active, so the fix depended on PVE's 401 behaviour. PVE's storage_info
wraps activation in eval{} and leaves active=0, so a 401 returns HTTP 200 with
active:0 — never an API error. The chain is sound by proof, not inference.
- A defect I shipped and caught: v0.91.0 built the probe seam and main.go never wired it,
so the leg was inert while every test passed. Same class as controller v0.154.0 the day
before. Fixed in v0.91.1 (artifact superseded, not overwritten); v0.91.2 made a healthy
probe observable so "no auth_failed" can never again be confused with "never probed".
The DR-tier capability row is deliberately NOT upgraded to PROVEN-LIVE: the decisive
evidence is STOP-2, the operator pressing Re-issue and the box converging where the
identical click did nothing on 2026-07-18.
R-50b(a) shipped — wrapper sha256 in the manifest + agent reporting + host drift surface,
with unknown-on-either-side reading as quiet rather than drift. (b)/(c) remain open: the
wrapper is still fetched unversioned from raw/branch/main.
Completes the hub half of R-39's fleet fix on top of the generation core (c484aa2).
pbsdrheal gains an auth_failed TRIGGER — a new trigger in the existing machine, not a
new machine. A box whose credential PBS rejects escalates to a fresh mint, never a
re-stage (which would re-feed the secret PBS just rejected), through the EXISTING damper:
a 401 flap must not become a secret-minting chain. With the generation stamp this closes
the loop end to end — agent proves the 401, hub re-keys, generation advances, descriptor
hash moves, agent re-consumes.
consumed_at honesty gauge: a staged secret still unconsumed past a 15-minute grace while
the box reports `applied` is surfaced with its own event. That is the exact 2026-07-18
fingerprint and a disagreement no single tier can see alone. Deliberately a SURFACE, not
a heal — auto-re-issuing on it would mint a second secret on top of an unconsumed one,
which is the mint/consume race R-39(a) already recorded. One event per distinct report,
and an honestly-stuck box does not double-report (its unconsumed secret is the symptom
being healed, not a contradiction).
R-50b(a): ArtifactManifest.WrapperSHA256 + operator field + host-page drift surface. The
PBS wrapper is root-owned 0755 and the pinned sudoers vector, yet installed unversioned
from raw/branch/main and absent from every manifest. Agents >=0.91.0 report the installed
hash; a mismatch is surfaced. An unknown on EITHER side reads as quiet, never as drift —
lighting every host amber on rollout day is how a warning becomes background noise. The
delivery channel itself stays R-50b(b)/(c).
Compatibility unchanged: safe for 0.90.0 agents (unknown JSON key dropped); the re-arm
and auth-honesty guarantees need agent >=0.91.0, so MinAgent moves only after the fleet
has self-updated.
Tests: auth_failed escalate/debounce/recovery-forgets-streak; honesty gauge incl. grace
window, the restage edge (consumed_at deliberately NULLed), consumed-never-alarms, and
honest-stuck-no-double-report; wrapper drift incl. both unknown directions. Red-proof run
at the assertion level: removing the auth_failed arm fails the escalation tests with
reissues=0.
admin