The unattended offsite restore-test on demo-felhom passed: 14.46 GB archive,
duration_s=635.07 (10m35s), then it rotated to the local tier. Persisted state
confirms the credit: {"felhom-pbs": "2026-07-27T06:14:42Z"}.
CORRECTION: I estimated ~2 hours for this restore. It took 10m35s. I derived
the estimate from a download rate measured during the FAILED attempt, which was
running under contention; the real link does ~1.4 GB/min. I then used that wrong
figure to raise a design concern — that the heavy-op gate would block backups
for hours on this box — which at 10 minutes largely evaporates. An estimate
extrapolated from a degraded measurement is not a measurement.
The SPEC's closing risk note is corrected in place, with the original left
visible for the lesson.
R-86 (NEXT, operator ruling 2026-07-27): backup-ALIGNED restore-test scheduling
— test a tier ~1 day after ITS OWN backup. R-85 schedules on a free-running
interval, which cannot express 'the day after the PBS backup': any fixed offset
drifts, so alignment would be luck. Shape: trigger from the tier's own last
successful backup rather than a clock. Interim in force: 302400s (3.5d), which
lands each tier ~weekly — the cadence half of the ruling, not the alignment half.
R-87: the restic app-data offsite tier is NEVER restore-tested. R-85 covers
whole-guest vzdump tiers only; the agent has no restic surface. That is arguably
the tier that matters most — the only one that survives losing the box AND
carries the customer's app data, since the whole-guest snapshot excludes the
bind-mounted drives. Exactly the state PBS was in before R-85.
REPORT.md: the full R-80 -> R-85 arc, including a section on the seven mistakes
I made and the two recurring shapes behind them (inferring behaviour from an
artifact instead of the code that consumes it; reading a result without its exit
code). Records demo-felhom's restore-test as IN FLIGHT at close, with the manual
recovery step if the deferred restart watcher does not complete.
Hub gate green (17 packages, rc=0).
- ROADMAP: R-85 row. Code SHIPPED; rotation NOT YET OBSERVED LIVE, stated as
such rather than written as done.
- Capability map: a new row for UNATTENDED restore-proof, IMPLEMENTED not
PROVEN-LIVE, kept distinct from the R-82 row that a MANUAL selftest earned.
That distinction is the same one the activation-vs-arrival split made.
- 03-host-agent §8: the scheduler covers every tier, oldest-proven first; the
spec is per-run; a restore-test joins the one-heavy-op gate. The safety
properties that must not be re-derived are listed.
- 07: restore-proof recorded as a per-tier property. Doc still NOT ratified.
- 06: corrects S4.1's 'the offsite restore-test now runs unattended' — it
silently stopped being true when local_backup_target was retargeted to 'local',
the SECOND time in that doc that a correct mechanism was broken by its input
changing underneath it.
- CONTEXT + REUSE.
Hub gate green (17 packages, rc=0).
Written against verified state, not assumption. Records three gaps the original
Phase 5 ordering does not cover:
1. hub v0.77.0 is COMMITTED BUT NOT DEPLOYED — manifest pins 0.76.0 and the pod
runs 0.76.0, so the R-85 signal exists only in git. The original Phase 5 only
mentions the agent.
2. R-85 has no ROADMAP row.
3. The agent CHANGELOG says v0.104.0-dev; an ldflags version disagreeing with
the CHANGELOG is the reconciliation problem hub 0.73.2 already caused.
One ordering correction: THE HUB GOES FIRST. Agent v0.104.0 makes the offsite
tier testable; hub v0.77.0 makes a failure audible. Agent-first means rotation
begins with nothing listening — two tiers able to fail silently instead of one,
which is the fault R-85 exists to end. Also drops the retired drill box, so the
rollout is demo-hp -> demo-felhom.
Names the phase's most likely SILENT failure: the new rotation state lands at
/var/lib/felhom-agent/restore-test-state.json and the agent is non-root. If that
is not writable, RecordSuccess warns and continues — a quiet return to one tier
being starved, not a crash.
Flags for operator judgement: demo-felhom's 14.46 GB offsite archive makes its
unattended restore-test a ~2h operation every other day, holding the heavy-op
gate throughout. Ruled when the only measured restore was demo-hp's 4 minutes.
Restore round-trip on demo-hp: pass=true, verified=boot+running,
mount_parity=ok, source_tier=pbs (the v0.100.0 fix — the earlier attempt said
'local' and died at 600s), 4m5s restore+boot+verify+teardown, clean teardown
with no 403 and no leak. That last point confirms 06's reading that the
teardown 403 was a phantom, and corrects my earlier framing of it as a standing
privilege gap.
Multi-tier quiesce driven through the REAL UI endpoint (authed+CSRF):
exactly ONE stop/start pair with BOTH backups inside it, local-first/PBS-last,
app quiesced through the non-last tier, early resume on the last tier's
snapshot. Total downtime 1m27s for both tiers; app healthy after.
Capability map row upgraded IMPLEMENTED -> PROVEN-LIVE, kept distinct from the
DR-tier row above which proves ACTIVATION not ARRIVAL. Remaining gaps recorded:
the SCHEDULED restore-test still only selects the primary tier (manual path
proven, unattended not), and the hub infers cadence from storage type.
R-84 resolved by asking the STORAGE rather than persisting the store: ground
truth, so a pruned archive correctly stops counting where a persisted record
would keep claiming a backup that no longer exists. Proven live on both boxes
with the in-memory store cold.
demo-hp's FIRST EVER offsite backup landed (4.25 GB) — the R-82 finding closed
on the box where it was worst. Controller v0.175.0 deployed to both boxes.
Slice D.1 — host-install 1.20.0: a FRESH box defaults to local-daily +
offsite-weekly (felhom-pbs, 604800s, keep_last=2). setdefault semantics proven
both ways: fresh gets the tier, an UPGRADE preserves the existing backup block
verbatim — so an in-place upgrade can never silently start writing to an
offsite datastore. Existing boxes are migrated explicitly.
Slice E:
- 07-backup-architecture.md: honest status header per CONTEXT ruling S-2, with
an explicit STALE-outside-the-PBS-tier verdict (the controller tiers were last
verified 41 controller versions ago). The PBS row claimed 'PBS on DooPlex'
(the retired spike store) with no cadence; it now names felhom-pbs ->
felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2. NOT marked
ratified — that is Viktor's review of the section 10 list. Discharges R-83.
- 06-offsite-connectivity.md: the target-split remaining-work note collapsed
(shipped), and records HOW S4.1's tier-aware timeout silently regressed — the
mechanism was never removed, its INPUT changed when local_backup_target was
retargeted to 'local'. Also notes S4.1 already diagnosed the teardown 403 as a
phantom (a timeout consequence, not an ACL gap).
- capability map: new row for recurring offsite backups actually LANDING, as
distinct from the existing row proving ACTIVATION. IMPLEMENTED, not
PROVEN-LIVE — the restore round-trip has not completed under the fixed code.
- ROADMAP: R-82 SHIPPED with its remaining gate named, R-83 DISCHARGED, R-84
left open.
- CONTEXT + REPORT: the arc, including the mid-arc correction I had to make.
Per the operator's 2026-07-26 ruling: datastore growth deferred, R-82 proceeds.
Records the measurements so the constraint is written down rather than carried
in a session: 37.2 GB total / 10.8 GB used, no cross-customer dedup (per-tenant
encryption keys), 80% warn reached at roughly the second additional customer,
and the pvesm 0/0/0 reporting artifact that means operators must read fill from
the hub gauge. Also records what the tier does and does not carry, and the
conditional on the P0.1 weekly verdict (Tier-3 offsite must be healthy).
Doc NOT marked ratified — that stays Viktor's review (R-83).
Read-only gate. No code, no backup triggered, no config changed.
P0.1 weekly CONFIRMED: the only 7-day-exposed state is the non-SMB half of
settings.json (storage_paths, app_backup toggles, notification prefs,
password_hash, launcher token) — none catastrophic, all recoverable. The two
items that would have overturned it, encryption.key and the offbox
credentials, are STABLE files unchanged since first boot, so a 7-day-old copy
is byte-identical. Everything referentially coupled to app state is carried
daily by Tier-1/2/3, and Tier-3 offsite was verified running and ok on both
production boxes. CONDITIONAL: drill-r50 has offsite:null, so the verdict does
not hold there.
P0.2 RESOLVED, benign: PBS returns HTTP 200 with zeroed usage to the
namespace-scoped token (DatastoreBackup, not Datastore.Audit), so PVE prints
0/0/0. Ground truth via the hub ep0 df op: 28.9% of 37.2 GB. Writes work —
the 07-18 snapshot is owned by this token, 9.74 GB, verify ok.
P0.3 STOP: 37.2 GB datastore, 10.8 GB used, 80% alert at 29.8 GB. Encrypted
per-tenant backups do not dedup across customers. Current 3 boxes project to
~15-21 GB weekly keep-3; each additional customer costs ~5-10 GB, so the 80%
alert fires at roughly the SECOND additional customer — inside the alpha
horizon. Needs a ruling on datastore size / retention before recurring writes.
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class.
On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp
and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel
claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump
archives were on disk. Cause: the agent backup store is in-memory, so the
R-50 fleet restart emptied `backups` until the next run, and the hub read
empty as "no backup exists".
- assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`;
absence is UNKNOWN until it outlives an anchored window. Still pure.
- store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own
retained history (bounded 7-day lookback, early-exit on fresh evidence) —
"when did I last SEE evidence of a backup?" The anchor was free: the hub
already retains 90 days. No agent change, no new persisted state.
- store.GetFirstHostReportAt anchors absence at first contact, reusing the
existing 26h threshold as the grace (no new knob, the v0.73.0 shape).
- Deferrals logged + counted; reason strings kept distinct.
- backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would
alarm six days in seven) and owned by R-82.
Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live
message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all
three now silent.
Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
Read-only diagnostic. No fix applied.
Verdict: FALSE ALARM for demo-felhom + demo-hp, true-but-premature for
drill-r50. Fired ONCE (2026-07-26 03:00 UTC), not nightly. No real
external customer notified.
Root cause: felhom-agent/internal/backup/store.go is in-memory by design;
an agent restart empties the host-report 'backups' array until the next
backup runs. The R-50 island migration restarted the fleet on 2026-07-25
12:44 UTC, opening an ~18h blind window that the 03:00 UTC deadline check
fell into. Hub's assessBackupFreshness reads the empty array as 'no
backup exists'.
Separate real finding: the PBS/offsite-DR tier has no schedule at all
(local_backup_target=local, no jobs.cfg) — demo-felhom has one snapshot
from 07-18, demo-hp has zero since pbs_dr was applied 07-21.
The allowlist entry is REQUIRED, not cosmetic: handleEvent 400s an unknown
event_type, so controller v0.173.0's new drift alert would be silently inert
without it. Shipped with the controller that emits it.
Docs:
- RUNBOOK-local-api-endpoint-drift.md — how to repair a drift, including the
step everyone will want to skip (establish which value is CORRECT from what
the agent is actually bound to, rather than assuming bootstrap.json wins) and
what success looks like (SILENCE, not a "recovered" line, because a fresh
controller's healthy first observation is not logged). Records both
2026-07-26 repairs.
- ROADMAP: R-77 shipped; R-78 the local_api authority ruling, with the
clobber-a-working-channel risk spelled out in BOTH directions so it is not
resolved opportunistically; R-79 the whole-surface English-strings sweep;
R-80 expected_backup_missed, flagged as likely outranking R-77 because 7.3
days of stale backup materially exceeds the ~1.5-day channel outage, so the
causal link the DIAG hedged on cannot be the whole story.
- Capability map: note against the drive-wizard row (every agent-backed
capability rides this channel) that a silent drift class is now detected.
NO row status flips — detection is not prevention.
Read-only diagnostic; nothing restarted, reconfigured or deployed.
CAUSE (H2, supported): the island migration rewrote bootstrap.json
(169.254.253.1:8443) but the controller reads controller.yaml, which still
holds the pre-island LAN address on both boxes -- mtimes 2026-07-21 and
2026-07-23, i.e. days BEFORE the migration. ensureLocalAPI
(bootstrap.go:254) returns early when local_api.endpoint is non-empty, so it
fills a MISSING endpoint and never reconciles a STALE one. The agent no longer
binds the LAN address, hence "connection refused".
H1 refuted: every island leg is healthy on both hosts -- agent active and bound
169.254.253.1:8443, vmbr9 up, guest eth1 up, TCP from the guest to the island
is OPEN. The island works; nothing is using it.
H3 refuted: hub-persisted events put the first alarm at 2026-07-25 12:44/12:46,
~17.5h before the v0.172.0 deploy. Today's re-fires are restart artifacts --
channel state is in-memory by design.
H4 refuted: felhom-agent is active on both hosts.
Also: the logged "up->down" is an ARTIFACT -- the debounce branch seeds an
unseeded state to "up", so the channel was born-down and has never been up.
Symptom B confirmed: EffectiveProtected gates cloudflared on a token predicate
but samba on smb.Enabled alone, so sharing-without-password reports FAIL for a
deliberately-undeployed stack. The function's own doc comment asserts the parity
that is broken. Caused by leg 4 of the R-75 task. Blast radius: zero emails so
far, but demo-hp flipped ok->fail and the health_critical path has reached the
CUSTOMER channel historically. The English issue strings are whole-surface, not
a one-off.
Fixes described, none applied.
New documentation/controller/import-and-data-paths.md: the canonical import root
(and why it is NOT a registered StoragePath), the three data_paths roles, the
Fork-3 validation asymmetry, the class-driven copy rule, and the seven
invariants a future change must not break.
Capability map "File access via browser" — status DELIBERATELY UNCHANGED. The
drop-zone now has its own FileBrowser source and the app page carries a deep
link, both verified live, but nothing drove the FileBrowser HTTP UI (no browser
on DooPlex), so the row's standing "browse is exercised in no doc" caveat still
holds and PROVEN-LIVE remains unearned.
R-75 collapsed to its shipped one-liner. R-76 left open — this task does not fix
it, and nothing built here assumes an import/* directory stays 2775.
R-75 (spiked, GO) names the capability-map row it would flip: "File access via
browser" (00-capability-map.md line 96), currently IMPLEMENTED with the caveat
that browse/download through FileBrowser is exercised in no doc. Carries the
mandatory determinism constraint from P6 (sort + red-proof, or FileBrowser
force-recreates on every sync pass), the zero-removals invariant for
`documents`, the url.PathEscape-not-QueryEscape trap, and the four design forks
with evidence + recommendation, all awaiting operator ruling.
R-76 is minted for the two PRE-EXISTING defects the spike surfaced and
deliberately did not fix: FileBrowser Quantum creating 0644/0755 without
propagating setgid (breaking the shared-group chain one level below any
customer-created folder -- latent only because every userdata-touching app runs
uid 1000), and import/calibre living at 755 on demo-felhom where its same-app
sibling media/books is 2775.
Source: audits/SPIKE-catalog-data-paths-2026-07-26.md
Verdict GO, with one mandatory constraint.
Phase 0 HOLDS: the customer-facing path set is fully derivable from data the
controller already parses (ParseComposeClassifiableBinds), and
ValidateBackupSpec's refusal set already covers the path-safety class
data_paths: needs -- so the annotation-only design introduces no new
filesystem-write primitive. No catalog folder is reachable only via env
indirection; zero templates use long `type: bind` syntax.
Found off-brief: the derivation is ALREADY LIVE at deploy time
(stacks/manager.go:183 ensureUserdataMounts, "the deploy belt"), proven by a
clean two-box control -- media/podcasts exists on demo-felhom where
audiobookshelf is deployed, not on demo-hp, and is in no skeleton.
P0 GO: derived set 14 dirs vs hardcoded 14 (+media/podcasts, -documents); all
three hand-verified anchors match. P1 GO: 4 sources incl. nested + accented
names all index ready. P2 GO: deep-link template constructible in Go from
(sourceName, relPath) alone; login redirect preserves the target. P3 GO for the
feature, with a pre-existing setgid-chain break recorded. P4 GO: userdata/import
is shareable, guard unchanged, no live share created. P5: before-state inventory
captured read-only on both boxes. P6 GO-conditional: the naive derivation
produced 20 distinct outputs from 20 identical runs -- a guaranteed FileBrowser
force-recreate loop -- fixed by one sort. P7: 12 sources safe server-side.
No production code, no version bump, no live mutation.
- B2 demo-hp + B3 demo-felhom migrated to the island (agent 0.96.0), apps
served throughout (0 container restarts), island /storage 200, LAN DNS pinned
to the LAN IP, hub reports 0.96.0. No rollback.
- capability-map 'site/network change' row PARTIAL -> PROVEN-LIVE
- ROADMAP R-50 -> SHIPPED (fleet-migrated); add R-74 (island on Peti's cluster)
- nodes.md: both boxes island-bound, agent 0.96.0
- A4: scratch guest provisioned from golden on the island-configured drill came
up with net1 automatically (zero manual edits) — the v0.96.0 provisioning path
proven live. Spike method caveat CLOSED.
- REPORT: Phase A done + vouched 0.96.0; B0/B1/A4 done; B2/B3 await operator go.
Idempotent LAN->island migration procedure with rollback table + abort criteria
(firewall LAST). Validated verbatim on drill VM 300: rolled to r50pre, migrated,
island /storage 200, LAN DNS held on the LAN IP (Finding-1 pin), apps healthy,
hub reports 0.96.0. No rollback fired.
Provisioned nested-PVE drill 'drill-r50' (qm300 on demo-hp) via the v1.25.0
nested-vm ISO through the real day-0, then ran the R-50 empirical spike:
- vmbr9 portless island bridge + guest island NIC hot-add (LAN undisturbed)
- F1 replay money shot: LAN move survives on the island; LAN-literal bind
reproduces the 2026-07-20 daemon-exit bug verbatim
- dnsmasq trap confirmed live + lan_resolver.host_ip fix proven
- pin address-independent (leaf SHA-256 unchanged, HTTP 200 over island)
- survival matrix: agent/guest/host-cold-reboot all return on the island
Docs: SPIKE verdict BLOCKED->GO, ROADMAP R-50 SPIKED->GO, nodes.md drill VM,
REPORT overwrite.
Read-only probes on demo-felhom: both disks (system SSD /dev/sda, USB /dev/sdb) report
PASSED via the exact allowlisted 'smartctl -a -j <dev>'. Card shows Nincs adat because the
agent never reads: 'local' is a dir on LVM pve-root (backing='' + smartDeviceFor has no
dm/LVM branch), and the USB is surfaced via the non-enriched driveTargets.Known union path.
-d sat NOT needed (bridge passes SMART through; sudoers grants only -a -j). Graded fix
directions B(low-risk, USB) > A(system SSD, medium) > C(reject). STOP — no fix implemented.
New capability-map row (IMPLEMENTED; §13 endpoint-level live on 9201). Records the
ruling: member accounts are superseded by the capability-URL guest share for
launcher sharing; per-member tile visibility parked under the SSO/members arc (R-15).
Updated the launcher row's member-coupling note and R-15 accordingly.
Virgin-ISO nested drill closed the train: dead-NIC install baked the
fallback (incl. the dead default gateway), the R-59 screen painted
(capture committed beside the spike doc), the cable move healed +
registered at the hub in 23s unaided, and the build's rootpw file
matched the installed box's shadow hash. R-59 SHIPPED with the recorded
deviation (first-boot gate; installer-initrd abort out of scope by
operator ack). R-60 SHIPPED (spike + drill cited; F-P9 route-flush fix
included). R-61 slice 1 SHIPPED. New R-62 row (hub delete-dialog
cosmetics, XS). Capability map: new PROVEN-LIVE row (nested != metal,
said so). Cleanup verified: felhom-pve interfaces byte-identical,
bridge/VMs/ISO removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Nested VM 9310 on felhom-pve (throwaway vmbr9 rig): re-point
bridge-ports -> ifreload (0.36s, raises the port itself) -> dhclient
lease 3.3s -> hub HTTPS 302 in 160ms; a dead candidate costs exactly
its timeout bound (rc=124 at 20s). Bonus finding: the auto-installer
picks the NIC that leases and bakes the lease as STATIC — so wrong-NIC
boxes come in two variants (192.168.100.2 fallback bake vs
truth-changed-after-install), and the gate must trigger on
hub-unreachable, never on the fallback signature. Eight design inputs
(F-P1..F-P8) recorded for the Part-3 gate. STOP-0 discharged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Per the 2026-07-21 refresh brief: R-39 interim blocks (B4/E1) and the R-36
manual-Save block (C4) deleted — both shipped and proven live; freemail.hu
gate proven (R-4 COMPLETE); golden/floor-lift note now cites two shapes
(rehearsal + virgin HP t740 day-0 lift 0.153.0->0.156.0); A3 loader table
per operations/nodes.md (N100=mkimage/SB-off per record, HP t740=shim/SB
ENABLED); B2 multi-NIC cabled-port gotcha (R-59/R-60 pending); new A5 gate
(agent >=0.93.0 deployed box-side before the first escrow ceremony); D
offboarding pointer to §G (R-25b). DRAFT status and the C7 graduation gate
unchanged. ROADMAP R-25b pointer follows the rename.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Found validating v0.69.0 against the live hub. demo-vm-felhom was deleted
on 07-18 and was still on the Customers list AND still raising offsite_stale
(10 events, latest 07-21 17:34, operator email at 19:34) — because
GetCustomers() is report-derived and no lifecycle tier ever deleted a report.
New leg 3 (residue), before the record purge: reports, app_telemetry,
app_log_tails, log_tail_requests, customer_notifications, plus the
credential-bearing appliance_registrations and selfbind_tokens. Audit
(events, notification_log) and F-14 provenance still survive.
Ghost customers are now deletable: 404 means "nothing here", not "no config
row". With no config row the offsite descriptor is unknowable, so the Hetzner
and descriptor legs record skipped_no_config rather than a bare "skipped".
Two more red-proofs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
POST /configs/{id}/delete now runs hosts -> RESET -> purge behind three
acknowledgements, a typed customer-id, a stale-preview check and the
ONLINE-host refusal (every gate before any write, so a refusal has zero
side effects). The shallow handleConfigDelete is gone.
Two invariants are asserted, not just commented: ruling 3 is preserved by
construction (leg 2 never sees a host row) and retained escrow custody is
purged exactly once, in leg 3 (leg 2 runs with purgeEscrow=false).
handleCustomerReset's committed half was extracted as commitCustomerReset;
the standalone RESET path is byte-identical to v0.68.1 and its suite is
untouched. Five red-proofs run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
New documentation/operations/nodes.md: HP t740 hardware, disks (SanDisk system
SSD 182195804614; Toshiba 1TB NVMe 58BS11AFT8MQ PRESENT AND UNENROLLED, still
NTFS, do not touch), the five-NIC map and the trap that cost the first install,
and the access path - no SSH key is baked, auth is the hub-vaulted G1
break-glass password.
tailscale.md gains demo-hp, the operator-lab-exception warning so a future
product-shape audit does not conclude the product ships tailscale, and the
--accept-dns evidence: the join omitted the flag, MagicDNS rewrote
/etc/resolv.conf, and it was reverted. Harmless at the vacation site, would
have bitten silently at home where split-horizon matters.
OPEN: key expiry still enabled on demo-hp (2027-01-17) - needs an admin-console
toggle or an API token; a pre-auth key cannot do it.
R-59 no-DHCP install must hard-abort (it baked 192.168.100.2 static and
completed - a box that can never call home). R-60 first-boot NIC sweep
self-heal. R-61 the baked root password must be knowable; a fixed well-known
password is explicitly rejected.
Positive evidence same-session: R-21 slice C PROVEN on a SECOND, virgin board
(HP t740) - and the shim loader booted with Secure Boot ENABLED, retiring the
assumption that Felhom installs need SB off. Fresh-box floor lift
0.153.0 -> 0.156.0 during day-0 cited on the publish-train row.
ISO README gains the t740 five-NIC trap: the 4-port igb card gets no lease,
the onboard r8169 port does.
R-58 records the operator ruling (2026-07-21) with the argument verbatim: the
installer should list available storage devices, excluding the install media,
and let one be selected. Third ISO mode alongside unattended-serial and
match-nothing-safety; unattended stays the appliance/factory mode. Slice 1 is
the abort-screen candidate table, same enumeration code, and it collapses the
two-boot dance on its own. Matters most for BYO/reinstall, where the serial is
unknown and a wrong guess is destructive.
ISO README gains the HP section: shim proven on this board by the safety boot,
the uncommitted-armed-profile pattern, verify-from-inside-the-ISO, and a
pointer to the prior-LVM abort that is the one likely failure on a
second-hand disk.
R-55's reboot leg ran operator-present on 9201: immich UI-stopped -> stayed
stopped across pct reboot, calibre-web recreated, zero alerts, ~15s.
R-57 records the lifecycle mechanism with the operator's abandoned-app
requirements verbatim and plant-it as the motivating case, including why the
retired/ directory move was wrong and the v0.158.1 pointer-receiver defect.
Claude Code
admin