The v0.150.0 'orphaned dhclient' cleanup killed guest 9201's DHCP client, not a
host one: LXC guests share the host PID namespace, so the missing eth0, pidfile
and lease file were all true ON THE HOST and all irrelevant. Lease expired ~80
min later, guest lost its address, tunnel/hub/catalog/agent all went dark.
Restored; records the one-command ownership check (/proc/<pid>/cgroup) and
leaves the unsupervised-dhclient risk open for a ruling with R-50.
Also records against R-45 the lesson the loop taught: a phase a client answers
with a one-shot action must be an EDGE the registry serves once, never a level
synthesised into the phase channel.
felhom-pve joins the tailnet as a host package (100.70.170.35, expiry
disabled); DooPlex already advertised 192.168.0.0/24 via its GitOps k3s
tailscale pod (100.107.87.53). ssh felhom-pve now targets the tailnet IP
(direct-over-LAN at home, tunnel when away); felhom-pve-lan = LAN fallback.
--accept-dns=false on the host. Host converted static->DHCP (reservation
keeps .162 at home). Measured: --accept-routes hairpins the local subnet
at home -> travel-only opt-in. PBS is offsite (own WireGuard tunnel), needs
none of this. Recorded not fixed: controller<->agent plane hard-pinned to
192.168.0.162 (agent listen_addr + guest bootstrap.json) -> not yet portable
off-LAN. No code changed. Full doc: documentation/operations/tailscale.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMForrE4c1wZxd9LukxYVt
VERDICT. H1 confirmed: the reconstitution never ran. /backup/offbox/reconstitute
was never hit (reconstituted/safety-dump/replay line counts all 0); the operator
clicked the old missing-only button, which merged 34 files and left the DB
untouched. H2 rejected (the full path never executed; staging used the correct
snapshot 49e7cb46). H3 rejected (no replay happened; DB was genuinely asset:0).
R-44 is exonerated and working: 49e7cb46 carries offsite_run_id 20260719T102851Z
and its dump probes to asset:11 / user:1 — a properly coherent pair, against
round 1's asset:0 / user:0.
H4 (NEW, the more important result). Phase-3 recovery ran the real sequence and
the v0.148.0 path executed correctly — safety dump, stop, start, replay — then
the replay ABORTED: relation "clip_index" already exists. Proven to the second:
10:58:25 controller begins replay
10:58:33 immich-server "Reindexing clip_index" -> "Reindexed clip_index"
10:58:35 dump's CREATE INDEX clip_index fails, exit 3
The replay races the app's own schema repair, because reconstitution starts the
stack before replaying (ImportDump needs a live container). The photos survived
only because pg_dump emits COPY data before CREATE INDEX — luck, not design. The
same start-then-replay shape is on the LOCAL path, so it is a class defect.
Recovery: PARTIAL and STOPPED as instructed. All 11 assets are back and visible
by every DB criterion (status=active, deletedAt null, all files resolve), but the
operation reported failure and immich reports schema drift. Not clean acceptance
evidence; no second attempt made; safety dump read only.
Findings -> R-47 (the replay race), R-48 (restore controls separable only by
layout — the proven cause of this incident), R-49 (1.1 GB of a 1.2 GB immich
backup is ML model cache + a postgres data-dir tar duplicating the .sql dump +
immich's own nightly dump; recorded, not changed).
RULINGS LANDED.
- 00-capability-map.md:61 -> PARTIAL, scope-corrected: 6D's destruction hit the
FILE TREE only; the DB survived in its named volume, so "end-to-end from
offsite alone" overclaimed scope.
- The 704.6 MiB "discrepancy" -> CLOSED, not a defect: it was immich's own
Tarhely widget, never a controller page. Same for round 2's 650MiB->1.4GiB.
- customer-restore row: partial evidence recorded, NOT flipped.
Diagnosis only — no code, label or layout changes; no forget/prune.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
ROADMAP R-43/R-44 → SHIPPED with what actually landed:
- R-43: ReconstituteFromOffsite — safety dump → stop → files overwritten to
the snapshot version → start → the snapshot's own dump replayed from the
SCRATCH unit (the live unit is still never overwritten). Invariants:
nothing is ever deleted, and the undo is verified on disk before the act.
- R-44: every run, manual AND nightly, refreshes dumps + units BEFORE the
capture, so each snapshot is a coherent {DB@T, files@T} pair stamped with
offsite_run_id + dumps_at. Nightly ordering is now structural rather than a
coincidence of two scheduler entries at 02:30 and 04:15.
DIAG doc gains a "fixed in v0.148.0" header pointing at both.
Capability map deliberately NOT flipped: the DB half now exists but is PARTIAL,
not PROVEN-LIVE — shipped and deployed with no live acceptance behind it. The
customer-restore row stays MISSING, now blocked only on the evidence run rather
than on missing capability, and R-3 stays DRAFT.
Still open: the 00-capability-map.md:61 ruling — did CAMPAIGN-6D's "immich
end-to-end from offsite alone" exercise the DB half, or only the file half?
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
R-38 flips to SHIPPED with the live-canary evidence. Two new items registered
from tonight's work: R-45 (unified async-job feedback — five one-off progress
surfaces now exist and agree on nothing; the two lessons any framework must
encode are "probe the terminal state, never infer it" and "a source reporting
nothing is normal, not broken") and R-46 (verification copies are now visible
but still cannot be browsed and never expire — the invisible-result gap and the
disk-lifecycle gap, designed together because they are the same object).
Pre-invite checklist gains the golden note: build-golden v2.1.0 makes goldens
>= 0.147.x carry all four infra images; no rebuild for this alone.
REPORT-<topic>.md per the parallel-session rule — another session is writing in
this clone tonight, so the shared REPORT.md is left alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Viktor deleted 11 immich photos to test offsite restore; both place runs
merged 0 files and flashed success. Two-layered root cause:
- A UI delete means TRASH, not deletion — no file left the disk, so there
was genuinely nothing missing to restore. The test proved nothing.
- A valid test would also have failed: no offsite path loads a database.
All three buttons are file-only (offbox_restore.go:171-232, :346-421).
Files return; the library stays empty. The only DB replay is
RestoreFromRecoveryUnit, reachable solely from the LOCAL restore panel.
Compounding: the manual push never refreshes dumps (offbox_handlers.go:
203-227) — today's unit dump is from 02:30 and probes to asset:0 / user:0 /
album:0, predating both the admin user and the photos. Had a full restore
loaded it, it would have written an empty DB over the live one, destroying
the trashed rows that were the only surviving recovery path.
Answer to "can a customer trust same-day offsite?" for a DB-indexed app:
no — files come back, content does not.
No product code changed. Photos left in trash (recovery not wanted).
Capability map: customer-restore row stays MISSING by finding, not by
absence of attempt; offsite-restore row flagged scope-contested pending
a ruling on whether CAMPAIGN-6D exercised the DB half.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
Fills the gap flagged in §8.1. Upstream notes for bookstack, immich, nextcloud,
n8n, grafana, vikunja, privatebin, radarr, romm; plus the ones this sweep found
EMPIRICALLY on a real deploy (outline PGSSLMODE, tandoor port+ALLOWED_HOSTS,
zipline DATABASE_URL/healthcheck rename, homebox tag prefix + pepper, wger
DJANGO_DB_* + :8000) -- which are the more useful half, being what actually
happens rather than what a changelog remembered to mention.
Four majors (calcom, claper, gramps-web, onlyoffice) deployed clean but have no
retrieved upstream one-liner; recorded as not-retrieved, not fabricated.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
An offsite run after the trio's fresh redeploy shows the snapshot TAGS are stable
across uninstall -> volume-destroy -> reinstall: calibre-web 243ad08c -> 9698ba74
and immich 6df12205 -> ea511336 both keep their exact tag sets, so a
'restic --tag <app>' query still spans the reinstall boundary and pre-uninstall
restore points stay addressable.
Side effect: the standing 'kimaradtak: bookstack' offsite warning is now CLEARED.
bookstack had no offsite snapshot because it had no tier-1 backup to push; the
fresh redeploy produced one, so its first-ever offsite snapshot landed (run ok,
1m13s, 6 snapshots, 211.6 MB, last_warning None).
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Adds the CAMPAIGN 7 audit doc (53/53 apps deployed and validated on the demo box
through the real controller endpoints; 45 pass, 13 template fixes, ranked
findings) and the uninstall-semantics map row it proves out: remove requires stop
first, named docker volumes are ALWAYS destroyed including the DB, HDD data and
backups survive unless explicitly ticked, and the per-app offsite toggle outlives
the app while tier-2 config is cleared.
ROADMAP +3:
R-40 the update path cannot express a multi-hop major upgrade (nextcloud
forbids skipping majors; the template moved 31 -> 34)
R-41 no standing 'does every template still deploy?' check -- the sweep was
the first thing to try, and found 5 apps that had NEVER been deployable
R-42 ruling needed on DB/cache sidecar majors (deliberately not bumped)
CLAUDE.md: REPORT.md is overwritten, so a second concurrent session in this repo
writes REPORT-<topic>.md instead -- applied to this run's own report.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
R-36: both halves delivered — the enabled-but-unprovisioned warning on the
customer page (reusing the same predicate the offsite re-issue handler refuses
on), and the related sub-item, auto-minting the self-bind link at customer
creation AND RESET completion so the console banner's promised email is already
true. Records the gap found while wiring it: PurgeCustomerResetDBState does not
clear selfbind_tokens, so a pre-RESET link would have survived the reset; the
skip paths now clear stale tokens.
R-37: the post-RESET staleness banner, narrow by design — an in-flight reset
does not trigger it, it clears itself on the first post-RESET report, and ties
resolve to STALE because SQLite timestamps are second-resolution and a
same-second report almost certainly predates the reset.
Both red-proofed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Golden 0.146.0 baked on the drill VM and published to gitea:
felhom-golden/0.146.0/golden.tar.zst
sha256 4834c703162c5437467a329144b1a523019bf5693ab9d439558be7323587e955
612696588 B (584 MB archive), controller 0.146.0 confirmed baked in
All pass markers green: Result=success/ExecMainStatus=0, 0 FATAL/exclusions,
docker OK (overlay2), ALL THREE mounts included (rootfs + mp0 /var/lib/docker +
mp1 /mnt/sys_drive), pre-delete HTTP 404 (the pre-gate — version did not exist),
upload HTTP 201.
Integrity verified independently of the build host: anonymous GET | sha256sum
matches byte-for-byte, ranged GET 206, content-length matches the bake's bytes.
The version now appears in the hub dropdown (0.136.0, 0.143.0, 0.146.0).
Teardown per GL-1: log copied out as evidence first
(180:/mnt/5_hdd/felhom.eu/drill/bake-0.146.0.log), guest 9100 purged, token +
script + log shredded in-VM, VM off, qemu confirmed gone via `ps -eo comm` (not
the self-matching pgrep -f), drill disk reverted to the virgin snapshot exactly
as found. Token-leak grep = 0 against the LITERAL token value, on the bake log
and both ISO build logs from this session.
REMAINING is operator-only and password-gated: Day-0 manifest Golden -> 0.146.0
(Agent stays 0.90.0, MinAgent stays 0.90.0 — v0.146.0 declares no new agent
coupling), then the floor -> v0.146.0 saved LAST.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
ROADMAP:
- R-39 gains the full live diagnosis and REFUTES the brief's hypothesis. The
generation IS bumped (SetHostDesired bumps unconditionally, 2->3) and
applyPBSDR is exonerated, so no hub fix was shipped. The real mechanism is a
signal mismatch: the hub's re-consume signal is a generation bump + poke,
while the agent re-applies on a change of the DESCRIPTOR CONTENT HASH
(manager.go ~L235). An ep0 re-issue re-keys the secret of an EXISTING token,
so token_id/fingerprint are unchanged, the descriptor is byte-identical, the
hash never moves, and the fresh secret is never consumed -> 401 forever.
Proof: consumed-failed.json carries the same hash a4e5424... as the marker
written two minutes before the re-issue.
Records the second defect found while healing (wrapper reconcile passing
--server, fixed in agent v0.90.1), marks the box HEALED with evidence
(pvesm active, token 200, a real 9.7 GB encrypted backup listed PBS-side),
and leaves the fleet fix explicitly pending its own spec.
- R-33 collapses to SHIPPED (scripts v1.21.0), incl. why
TimeoutStartSec=infinity is the load-bearing half.
- Pre-invite checklist: golden target moves 0.145.x -> 0.146.0 and notes it is
now MORE stale, since v0.146.0 is live on the demo box while the golden still
bakes 0.143.0.
REPORT overwritten with the train: R-39 diagnosis verbatim + heal evidence, the
two ISO shas with the byte-identical-payload verification, the nav polish and
why the screenshot leg could not be done (the demo controller password is
customer-owned since the claim flow, so the build-server credentials are stale),
Phase 4 skipped cleanly, and Phase 5 deferred rather than half-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The 2026-07-18 N100 rehearsal ran the complete final-product flow on real metal
in one pass: RESET -> generic pairing ISO v1.20.0 -> customer self-bind -> day-0
-> managed-floor lift -> escrow ceremony -> offsite snapshots. No code changed;
every finding is recorded and ranked, none fixed.
VALIDATION-n100-rehearsal-2026-07-18.md — run context, a UTC-normalised timeline
built from the hub events stream / hub DB / controller log / bootstrap + agent
journals, per-ledger verdicts for S1-S8 + ledgers 8 and 9, 12 findings, the
not-exercised list, and 7 discrepancies against the brief.
Headline wall-clocks: bind -> credential 26 s; bind -> controller running the
current version 2 min 44 s; managed floor 0.143.0 -> 0.145.0 in 5 s unattended
(initiated_by: auto-floor); escrow ceremony -> offsite enabled 12 s; drive enrol
30.3 s. No post-bind leg stalled, which is the immediacy row's real-onboarding
proof.
Capability map (10 citations added):
- Bare-metal Felhom ISO PARTIAL -> PROVEN-LIVE (F1 closed on metal)
- Customer self-bind (slice 1) IMPLEMENTED -> PROVEN-LIVE (customer_selfbind)
- Guest RAM resize (R-24) IMPLEMENTED -> PROVEN-LIVE (shrink AND grow)
- Customer RESET two real firings + verified external teardown
- Escrow ceremony first live wizard firing
- Immediacy row "real-onboarding proof pending" cleared
- Publish train box-side floor lift proven on a fresh install
- Customer claim R-4 gmail half (Inbox under p=quarantine)
- Offsite orphan guard staged live leg fired on its own
- DR tier by default candidate PROVEN-LIVE upgrade WITHDRAWN (R-39)
Not flipped, as instructed: customer-performs-restore, BYO, DLNA, multi-user.
ROADMAP — collapsed R-1 (appliance half done, Peti half survives), R-21
(physically closed), R-24, R-27 slice 1, R-4. New ranked items:
P2-HIGH R-39 PBS DR applied-but-dead R-30 liveness from the wait channel
R-31 async offsite + status R-32 RESET base-dir purge
R-33 bootstrap quiet-poll
P2 R-34 backup lifecycle R-35 config-apply session survival
R-36 post-RESET offsite prompt R-27c console-passphrase bind
P3 R-37 post-RESET health card R-38 installer GRUB slice
Plus a pre-invite checklist (golden 0.145.x rebuild, freemail.hu, C6, R-11).
R-39 is NEW and was not on the brief: the PBS DR descriptor auto-provisions and
the agent converges state=applied, but pvesm reports 401 Unauthorized/inactive
and a direct probe 401s on every endpoint including /version while WG is healthy.
The hub minted a second token secret two minutes after the agent applied the
first and consumed_at is still NULL; the converged state machine will not
re-apply, and the agent's verify loop cannot read the credential to notice it
(non-root read of a file it writes through a root wrapper). Rank is provisional
pending Viktor.
R-3 draft: all four [REFINE] slots filled, self-bind made the default path with
"send the link BEFORE the customer sees the console", the measured wall-clock
table added, and interim operator workarounds for R-31/R-36/R-39. C6 (renumbered
C7) is marked as the single unexecuted step and keeps the doc a DRAFT.
Evidence bundle: 180:~/n100-rehearsal/ (10 files + MANIFEST.md), collected before
the box was unplugged for travel. Secrets read only to run probes; recorded as
lengths and metadata, never values.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Origin: R-7b close-out (felhom-controller REPORT section 4f). Two parts:
(a) the finding is benign — estimate.go:179 mounts a NAMED VOLUME (daemon-side, no
host path), the same shape as three already-allowlisted entries, so the fix is a
3-line ALLOWLIST addition with its WHY, NOT a docker-cp rewrite;
(b) the systemic half: the gates run only when a human remembers, so this one sat
red from v0.129.0 (2026-07-14) to v0.145.0 while REPORTs said green. Second
instance of the class after the v0.123.0 'Windows green gate silently red' note.
Lists the full gate inventory to audit for the same rot.
Drops the false 'no offsite target on the demo box' clause from the ROADMAP row,
the capability-map SMB row and sharing.md. Cites offsite snapshots e0b9d723 /
4e2b15ec and the restore round-trip results. Root cause (guessed settings key) is
recorded in felhom-controller REPORT section 7b.
- capability map: SMB row KNOWN GAP cleared -> share data rides both tiers; the
offsite leg + restore round-trip flagged as not-yet-live-exercised
- ROADMAP R-7b: idea -> SHIPPED, with the Model B' rationale and the live evidence
- controller/sharing.md: the KNOWN GAP block replaced by the execution contract;
operator note corrected — samba IS liveness-monitored since v0.145.0
Viktor's human leg closed the last gate: both shares open from the Windows
Network view, an interactive Explorer save landed as uid 1000, and a write into
the read-only share was refused with the folder untouched. Capability map row
flipped to PROVEN-LIVE with that evidence; ROADMAP R-7 + sharing.md updated.
R-7b (shares classified but not in any live backup run) remains open.
controller/sharing.md (code-verified vs controller v0.144.0 + felhom-samba
1.0.0); capability map 'Files from Windows Explorer / Mac Finder (SMB server)'
MISSING -> IMPLEMENTED (PROVEN-LIVE pending Viktor's Explorer leg); ROADMAP R-7
-> shipped-slice-1 with the slice-2 remainder, and the backup design fork split
out as R-7b (shares are classified but not in any live backup run yet).
Human Explorer leg exposed the split: FELHOM-SPIKE renders (WSD PASS) but the
double-click fails 0x80070035 — flat name resolves by no path (DNS/LLMNR/NetBIOS
all silent; disable netbios=yes). By-IP mount works => SMB is healthy, the gap is
name resolution. Fix verified live: adding nmbd (NetBIOS) => nbtstat lists
FELHOM-SPIKE, ping resolves, \FELHOM-SPIKE\spike-share mounts by name. R-7 must
ship smbd+wsdd+nmbd (+avahi/.local), not wsdd alone.
Verdict: appliance guest is LAN-bridged; multicast discovery works only in the
guest netns (guest-direct or docker --network host) — default bridge is deaf to
LAN multicast. Real samba+wsdd on host-net: Windows 11 ProbeMatch + 445 + SMB
round-trip PASS; SSDP MediaServer:1 reaches LAN clients. R-7 => host-network
LAN-bound; R-8 Jellyfin-DLNA plausible if host-network. ROADMAP R-6 -> spiked,
R-7/R-8 unblocked. S4.4 Explorer render pending human.
Let a customer bind their own freshly-installed appliance without the
operator: operator "Send self-bind link" mints a 7-day tokenized
capability link, emailed (Hungarian, sibling sender) to the customer, who
opens a public /bind/<token> page and proves two factors — the console
pairing code shown on the box screen + their retrieval passphrase — and
the hub stages the bind via the same BindAppliance (provenance
customer_selfbind). The box's ~30s appliance poll delivers.
Viktor's three rulings verbatim: console pairing code (no appliance list
ever rendered), operator-sent tokenized link, 5-attempt lockout ->
"call support". Wrong code == wrong passphrase (one generic failure, no
oracle, both factors compared unconditionally); expiry falls back to
operator-bind unchanged.
THE TRAP: one public prefix /bind/, exempt from auth+CSRF at both /login
gate sites via a single isPublicBindPath predicate (tight trailing-slash
match; ServeMux ..-cleans; handler rejects '/' in token). 9 tests
(Scenarios A-F + F1/F2); 4 red-proofs verified red-then-green (lockout,
oracle, widened-prefix, single-active). GC verdict: no appliance GC ->
the 7-day TTL stands alone. Controller/agent untouched; R-27b deferred.
Green: full hub build/vet/test (17 ok) + bash -n + hub confirm gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.
Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.
- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).
Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
Closes N100 F1 (HIGH): cheap AMI (AN3PLUS 0.01-class) UEFI firmware can't
relocate the ISO's stock signed GRUB from USB (relocation 0x0). The run's live
grub-mkimage workaround is now a first-class pipeline mode.
- build-felhom-iso.sh: --loader shim|mkimage (default shim, byte-for-byte
unchanged; profile-settable FELHOM_LOADER; --loader wins). Loud banner +
manifest loader:/grub-mkimage: fields + -mkimage filename suffix.
- mkimage-surgery.sh (new): post-prepare-iso, in the assistant container. Builds
a monolithic grub-mkimage loader from the ISO's own GRUB (module set from its
grub.cfg; embedded search --fs-uuid -> configfile the real menu). Swaps it into
the ISO9660 tree (real lowercase path) + the efi.img ESP; xorriso re-master
preserves BIOS-hybrid + UEFI + GPT-ESP, drops only Apple HFS+/APM. Recipe from
the N100 run evidence, not re-derived.
- Dockerfile.assistant: grub-common + grub-efi-amd64-bin + mtools + dosfstools.
profiles/n100.profile (new, mkimage + SB-off note).
- Validated on nested VM 311 (RUNBOOK-B legs): leg1 shim boots+installs under
OVMF SB-enforcing + SeaBIOS; leg2 mkimage boots+installs under SB-off; leg3
(red-proof) mkimage under SB-enforcing FAILS Access Denied (unsigned -> SB must
be OFF); leg4 surgery byte-identical payload. bash -n + shellcheck clean.
Physical N100 closure folds into the rehearsal (n100-safety match-nothing ISO
built + sha-recorded, unbooted). PXE stays a deferred R-21 note.
- DeleteHost(deleteEscrow) demotes current host_escrow into host_escrow_superseded (copy-before-delete, same tx), spares existing; one shared demoteCurrentEscrowTx (reused by SaveHostEscrow). F-14 provenance/gate unchanged.
- DeleteCustomerConfig now purges both escrow tables for all the customer's hosts incl. already-deleted (F-14 provenance UNION) — the one true purge point.
- Wording: checkbox/refusal/Danger-zone → demotion. S6b OBSOLETE. Red-proofs TestDeleteHost_Demotes + TestDeleteCustomer_Purges + wording guard.
admin