F11 PASS ok->stale->ok, an operator mail each way, customer mail correctly
pairing-gated. DOWN (>1h) not reached — stated, not implied.
F7 the set-aside PASSES decisively: move-aside to
/home/felhom-repo.orphaned-20260805, mtime and du -s (12535 KB) and
snapshot f3d9cd67 all unchanged; a fresh empty repo beside it.
NOTHING DELETED.
the afterwards FAILS -> R-228: settings.json records
"orphaned_renamed_to" and OrphanedRenamedTo has ZERO references in
any template or handler. 12.5 MB of retained customer data at a path
the box knows and never shows. Seam built, never wired.
F9 R-215's fix PROVEN LIVE on a narrower shape: with recoveryOffer() false,
GET /recovery now 302s instead of rendering its story. The literal
never-had-offsite precondition was not staged (it needs a rebuild,
which the brief forbids before Phase 4) — recorded as partial.
F10 NOT INJECTED, harness not product. Three attempts, each with a control:
the app, then the controller's monitor, then the run itself recreate
the mandatory directory within ~1 s. Kept as an observation: at
capture the directory held only a recreated metadata.db and NOT the
customer's file, and the run still said ok.
Also reproduced live for a third time: R-220 (candidates empty; unmounting only
the raw mounts flips both drives back) — still OPEN.
Venue left WORKING: 4 containers healthy, backup target not degraded, off-site
fresh repo last_status ok, escrow escrowed.
No product code changed.
Phase 2 (eleven injected faults) and the §4 positives that were owed.
- §4.1 MEASURED, twice: the box's rendered GetFloor() is 0.200.0, and a
cold-started controller logs "settle-gate: GO — at/above floor 0.200.0"
against the same line reading "floor still unknown" while the hold was in
force. Also corrects the brief's plan: SetFloor's line is u.dbg(), gated on
logging.level=debug and written to the logger, so it can NEVER reach the
debug ring — a restart alone would not have produced it.
- §4.2 still NOT measured, deliberately: the venue has an off-site target, so
needsOffsiteCredential correctly returns false. Recorded, not inferred from
the unit test.
- F1 PARTIAL, F2 PASS, F3 FAIL, F4 FAIL, F5 PASS, F6 PASS, F8 PARTIAL.
F3+F4: a hub outage and a stopped agent are both rendered as "this code does
not open your package", in 0.056 s and 0.030 s — no unseal attempted. The
agent's own err field distinguishes them exactly and it is discarded at the
HTTP boundary; the R-216 capability gate answers source=version and cannot
see reachability.
- R-217's fix HOLDS under exactly its fault (F5), verified with the false-claim
strings absent and accented positive controls present.
hub/CHANGELOG.md: v0.97.1 had no heading of its own — the change was written
into the v0.97.0 entry while the deployed tag is 0.97.1. Given its own entry,
marked as added retroactively. Second occurrence of the class (agent 0.90.1).
Evidence: documentation/tests/campaign11-evidence-2026-08-05/journal-phase24.md
No product code changed.
The first vouch of agent 0.125.0 was REFUSED by R-120's gate: the artifacts form saves as
a unit, and golden 0.192.0 was older than the newest controller the fleet reports. The
golden had been stale since before controller 0.193.0, so the Day-0 manifest had been
effectively unvouchable for days and nobody had cause to notice. Correcting STATUS.md and
OPEN-ITEMS, which said "one dropdown".
Golden 0.201.0 baked in the drill VM, round-trip verified from Gitea
(e730d7cab343eb35...f007654, 658165766 B), teardown clean, 0 token hits in the saved log.
Manifest now agent=0.125.0 golden=0.201.0 min_agent=0.125.0, read back after the save.
The held floor: the HOLD is established positively (hub log + the box's own "floor still
unknown"). Its RELEASE is not — no HELD line and no dashboard reason are both absences,
the served path logs nothing by design, and the box-side positive needs a restart the
venue cannot take before Phase 2. Recorded as deduction plus a pinned test, not a live
positive.
Registers and evidence for the campaign and its fix pass.
OPEN-ITEMS: R-214..R-223. Six SHIPPED (R-215/216/217/218/219/222); three deliberately
still open and each blocks a real flow (R-214 console banner, R-220 drives unenrollable
after a rebuild, R-221 a rebuilt box cannot run the escrow ceremony); R-223 minted and
WAITING-ON-OPERATOR (vouch agent 0.125.0). R-213 and R-202 untouched.
Capability map: a new row for the customer's UNAIDED journey, recorded FAILED and staying
failed until a re-walk passes — fixes are not a journey. The existing rebuild row is
corrected where it said R-198's retention was unit-proven only: it was proven in production
on the first supersession since the fix, identity_blob retained at 572 B byte-length exact.
CLAUDE.md comment-vs-code table: eighth entry — ResolveManagedFloor, the first where the
false invariant was a GUARD rather than a comment alone.
STATUS: the headline is now "the backup promise is proved, the recovery journey is not",
and the one thing waiting on the operator.
R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The
hub's notification_log for the run reads: two per-app failures RECORDED, one
digest SENT naming both, and the customer channel SKIPPED with operator_only.
Against the measured previous behaviour — two failures, one email naming one
app, one leaving no trace anywhere.
Scenario D proved itself on an event I had not planned: disk_critical alarmed on
two filesystems, the second was collapsed by the cooldown, and that collapse is
now visible WITH ITS KEY. Yesterday it would have left nothing at all.
A gap the spec did not anticipate is recorded with its fix: the per-app event
also fires from the periodic sweep, outside any run, so making it record-only
would have created a NEW silence. The sweep emits a digest too, with no run_id,
so it stays under the ordinary hourly cooldown.
ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile
SURVIVED the resize and is active (checked, because a resize is a stop/start).
The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere.
Five documents corrected — three of which the task's list did not name, found by
searching. Two audit/evidence documents ANNOTATED, body untouched: they record
what was true when written and that is their value.
R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable,
not unbounded — the original OOM was a 14.46 GB restore — so the restore-test
cadence should still be paced, just not by fear of the endpoint.
target-selection.md's "D-d did not name ep0 either way" is deliberately left
standing. It is the operator's question, not CC's.
STATUS.md 127 -> 83 lines, items rather than sentences.
The S figures (66 MB -> 42.0s, two passes agreeing to 0.6s) had a spread tight
enough to prove fixed work dominates, which is exactly why they said nothing about
M. Second point taken 327x larger, same app, same method: clock from restore
request to the app serving the correct discriminator.
rallly's postgres volume grown 66 MB -> 21.1 GB (200k rows, STORAGE EXTERNAL so
TOAST cannot compress it into a fake number). Two reps:
rep 1 backup 406.4s unit 41149 MB RTO 624.5s discriminator correct
rep 2 backup 387.2s unit 41133 MB RTO 591.8s discriminator correct
327x the data cost 14.5x the time - strongly sub-linear:
RTO ~= 40s + 26.9 s/GB backup ~= 29s + 17.4 s/GB
10 GB -> 5.2 min 20 GB -> 9.6 min (measured 10.1) 100 GB -> 46 min
The fixed ~40s dominates below ~1.5 GB, which IS the S band and explains its tight
clustering.
The more consequential result is capacity. A DB-backed app's recovery unit is
1.90x its data (volume tar PLUS SQL dump): 21.1 GB produced a 40.2 GB unit. The
default appliance ships /mnt/sys_drive at 20 GB, so the largest app that can hold
a local Tier-1/2 recovery unit on a default box is about 10 GB - and that fills the
volume. The M band does not fit on a default box at all; this test only reached
21 GB because sys_drive was first grown 20G -> 70G with the same operation the
product performs via SysDataGrowGB. A tier-sizing decision, not a defect, but it
is invisible until an app crosses it.
Caveats stated in the doc: two points define a line but do not test linearity; the
1.90x is DB-app-specific and a file-only app should be nearer 1.0x (inferred, not
measured); synthetic incompressible data; one app, one box.
Both halves of the disposition were run and neither survived as a finding.
The I1/I1-pair violations cluster at cycles 31-33 and nowhere else across 39
cycles; c34-c39 are clean, so it recovered with no intervention. Final tally I1
37 PASS / 2 VIOLATION, I1-pair 36 PASS / 3 VIOLATION. On the quiesced box one slow
detach with 4 minutes either side produced a perfect pair. And the alarming
false-healthy (mentes bound=False while degraded=false) does not survive
quiescence - I had been reading the two halves at different instants of a detach.
No R-n.
Separately cleared: the hub's SQLITE_BUSY event drops. 7 in 24h including one for
the real customer demo-felhom, and the hub does return 500 with notification
dispatch only after a successful save - so a lost event would be a lost alarm. But
the controller retries 3 times and ZERO events exhausted their attempts; the
07:04:39 drop landed at 07:04:42. Nothing lost. Only cosmetic note: the ERROR line
reads like data loss and is not.
The 4th hard-reset failure had a different signature, verified not assumed: all of
rallly healthy, papra missing entirely with state=stopped deployed=True
containers=0. Zero containers is exactly what bootrecon deliberately never touches,
because the UI's Stop is compose down which removes containers - but a hard reset
landing during a compose operation produces the identical state. The signature the
safety rule depends on cannot distinguish the two.
Worse: in that state the deadapp check reported 0 currently down while a deployed
app was not running. No app_start_failed, no banner. That is the workspace's own
false-invariant #4 (F-CRIT-1, StateStopped assumed deliberate) recurring through a
hard reset rather than quiesce. NOT filed as new - CLAUDE.md already records it -
but confirmed live on 0.188.0 via a new path.
papra returned after ~15 min, later than the harness's 10-min window, so this
instance was slow rather than permanent and the doc says so. What restarted it is
not established.
Mechanism A (Exited, missed by the unsettled snapshot) alarms; mechanism B (zero
containers) is invisible on every channel. A settle-condition fix closes A only.
Three I1/I1-pair violations in ~5 minutes, all "expected event absent". Recorded as
an OPEN observation, NOT a finding: the system was mid-abuse when it was seen, and
a verdict taken on a system being hammered is worth little.
Established: it is not hub-side suppression and not a truncated log. The hub pod
has 0 restarts over 43h and the controller's own log matches it line for line, so
the events were never emitted. It is specific to the backup_target_* pair - the
generic storage_disconnected/reconnected pair for the other drive kept firing
normally throughout the same window.
Also sampled, and the more serious half if it survives quiescence: mentes reads
bound_under_parent=False while the backup-target state simultaneously reports
degraded=false. Those cannot both be right - a false healthy on the backup target
is I5/I6's failure mode.
NOT established: whether the pair recovers once cycling stops (the harness detaches
every ~2 min; a customer does not), whether the 02:25:37 controller restart is
implicated, and whether the degraded=false sample was transient.
Disposition written into the doc: after the run ends, quiesce with both drives
attached, then do ONE slow detach/reattach and see whether the pair fires. That
distinguishes "does not survive rapid cycling" from "the target alarm has silently
stopped working", which would be severe.
The first write-up said R-157 reproduced "at the same cycle in both runs -
deterministic, not a coincidence". Wrong. The cycle numbers matched only because
the runner's RNG is seeded so both runs drew the same permutation. The failure
itself is a coin flip: run 2b's four hard resets went PASS(c2), FAIL(c10),
PASS(c18), FAIL(c26); run 2a went PASS(c2), FAIL(c10). Three failures in six.
The correction matters because it changes what kind of bug this is, and it
strengthens rather than weakens the root cause: intermittency is exactly what a
race against container-state settling predicts, whereas a wrong predicate would
fail every time.
Signature is identical on all three occurrences: rallly Exited 255 with
rallly-postgres healthy, bootrecon reporting "no boot-orphaned apps" about 5s
after controller start, and the container count still churning after the sweep
(third occurrence 01:05: refresh 8, bootrecon 01:05:13, then 8 -> 7 -> 8).
Reproduced twice, two independent runs, same cycle (the runner's RNG is seeded so
both drew the same permutation - deterministic, not coincidence).
A hard reset mid-backup brought everything back except the app half of the
DB-backed stack: rallly left Exited 255, oom=false, restarts=0, its own log ending
"Ready" - it died healthy - while rallly-postgres returned healthy.
20:28:13 Status refresh: 8 containers across 55 stacks <-- docker ps -a shows NINE
20:28:18 [bootrecon] Boot reconciliation: no boot-orphaned apps
20:28:25 Status refresh: 7 ... 8 containers <-- still churning AFTER the sweep
20:39:14 [deadapp] 20 scans, 5 deployed evaluated, 1 currently down
The predicate is sound: once settled the controller reports rallly state=degraded
containers=2, and IsDownState includes StateDegraded, so len>0 && IsDownState
holds. The SNAPSHOT was wrong. bootrecon fires as a goroutine ~5s after start
while docker is still restoring containers, and is start-once by design, so it
never re-checks.
Consequence: the app stays down indefinitely. Detection is perfect and recovery
never happens - R-52's original shape, an alarm with no recovery. Not fixed.
Distinguished from this campaign's two earlier HARNESS defects: both drives bound,
every other app returned incl. the drive-backed one, only the app half of a
two-container stack missing while its DB is healthy, and it surfaced through the
fixed check written for exactly this.
The previous commit message was truncated by an unescaped paren in the shell, so
the fix detail and the product observations were lost from the record. This adds
them as evidence, where they belong.
Covers: the cc_proof table showing no C010-A row at all (the seed never landed);
both harness defects; why an ambiguous I7 justified stopping a 10-cycle run; the
red-proofed controls; and two transient product observations recorded but NOT
filed as findings - the health probe naming the DB container on the app's port for
about 70s during recovery, and a ValidateDump WARN on a dump taken while the app
was down.
Run 2a hit its first two violations at cycle 10 and BOTH trace to my harness, not
the product. Recorded in full because a check that fails for the wrong reason is
as corrosive as one that passes for the wrong reason.
HARD-RESET VM returned=True canaries_intact=False
I7 want=C10-C010-A-194530 got=C10-C009-A-192929 restore_ok=True
Root cause, evidenced: the cc_proof table's highest row is C10-C009-A — there is
NO C010-A row at all, so the seed never landed. The hard-reset atom ran earlier in
the same cycle and left rallly Exited(255); atom_restore_verify called seed() and
never checked its return value, so an unwritten generation became a fake stale
I1-I11 are CORRECTNESS invariants: they answer 'is the system telling the truth
this cycle'. All 586 of them passed in run 1 while nothing at all watched whether
disk usage, snapshot count, log volume, fd count or RSS climbs. Accumulation is
exactly what depth was for, and it was missing from the invariant list.
Adds c10growth.py (Campaign 2's controller_rss.tsv precedent, widened to 19
metrics) sampling every 90s as a SEPARATE process, so the in-flight run 2 did not
have to be restarted. Attributes every sample to a cycle by reading the runner's
status.txt, and records NA rather than dying when the box is down during a
hard-reset or reboot atom.
c10growth_report.py turns it into Campaign 2's table shape (start/end/min/max/
slope-per-cycle) and splits verdicts by class: growth in RSS/fd/volumes/images/
restarts is a LEAK; growth in backup storage or the qcow2 is expected
accumulation, reported with a projection to cycle 45.
Caught a bug in the sampler itself on the first analysis: MENTES_USED_MB appeared
to jump 623 -> 5667 MB, which is exactly ROOT_USED_MB — when a drive is detached,
/mnt/<name> reverts to a plain directory on root and df silently reports the ROOT
filesystem. The same class of error as the agent's exactMount check, in the
measurement code. Gated on mountpoint and red-proofed both ways: a real mount
returns a number, a non-mount returns NA.
The case R-117's spike called the worse half — a drive dying with no detach/return
cycle, which before agent v0.117.0 emitted nothing on any channel indefinitely.
Box runs 0.119.0. Aborted ext4 in place (abort,emergency_ro; device still present):
bound_under_parent went false, storage_disconnected fired, the storage page named
the stopped app, and calibre-web (whose library binds that drive) was STOPPED
rather than restarted onto the dead namespace. Recovery needed a full device close,
not a remount — exactly as the fix intends (BindAborted => quiet no-op).
Runner extended with the 7 atom families run 1 skipped: abort-fs-in-place,
kill-agent-mid-backup, hard-reset-VM-mid-write, reboot-VM, concurrent
backup+restore, concurrent backup+detach, fill-drive-near-full. Also fixes a run-1
flaw recorded in the audit: reboot was appended AFTER the shuffle so it never
interleaved with a detach; heavy atoms are now permuted in with the rest.
Run-1 evidence preserved as *-run1.* (cycle numbering restarts per run).
Ran the soak on the Phase A rig. Ended on its own deadline — no watchdog halt,
no atom exception, no I11 breach.
I1 28+28 pairs, I2 28+28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135,
I11 28. Zero violations. The row counts are themselves the no-silent-skip check:
I3/I4 twice per cycle (both drives), I10 = 5 secret-class fields x 27, REBOOT on
cycles 7/14/21 only.
I7 is the headline: 28 restores, 28 correct discriminators — never stale, never
empty. RTO (Tier 1, rallly, 66 MB): min 38.8s, median 42.0s, p90 42.5s, max
44.3s. That is the S band's lower end ONLY; the 5.5s spread over 28 runs says
fixed work dominates, so nothing extrapolates to M or L. RPO not measured.
Every atom and invariant was proven BY HAND before automation — the runner
asserts nothing that was not first observed live.
Caught a Phase A gap before starting: no app had HDD_PATH, so all data sat on the
system disk and I3 could never have fired. Deployed calibre-web onto adatok
first; otherwise the run would have produced 27 green cycles that tested nothing
cross-drive.
Investigated and DISPROVED a suspected defect (audit 5.2): /api/disks reports
state=attached for a physically absent drive, and intermediary.go:230 really does
compute presence from State=="attached". It is inert — planDriveGates only gates
paths under /mnt/felhom-drives/ and uses BoundUnderParent there, which was
correctly false. The gate fired; the storage page showed "Meghajtó leválasztva".
No R-n minted.
Honest gaps: 6 of ~12 atom families ran. Not run — Tier 3 (structurally
un-isolatable), abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write,
reboot-VM, both concurrency atoms, fill-drive-near-full. I8 not checked, I9 not
automated (cited from the tester-gate run, not re-claimed). kill_controller is
NOT mid-backup and reboot_guest never interleaved with a detach. 27 cycles does
not answer the brief's question about drift at the thirty-eighth.
Teardown still OWED, including hub customer c10-soak (disposition: DELETE).
Phase A passed every gate on a fresh box built from the PUBLISHED ISO 1.26.1:
install, claim, two drives enrolled through the real endpoints with the backup
target healthy, four apps spanning both sides of D5's secret split, and a
working discriminator across all four.
Isolation gate: both denials captured, each with a positive control. The PBS
control FAILED first — four clean-looking 403s were worthless because the token
was denied on its own datastore too (PBS token privilege separation). Fixed and
re-run; the denials stand.
R-156 (new, register grepped): papra's data is neither persisted nor backed up,
and it reports healthy. The template mounts papra_data:/app/data; the app writes
/app/app-data/db/db.sqlite. Volume empty and root-owned against a -rootless
image, real DB in the container writable layer, healthcheck only probes the HTTP
port. Its Tier-1/2 backup is real, verifiable and contains nothing. Not fixed.
Tier 3 could not be isolated so it was not run: offsite hard-requires the DR
tier (configs.go:1300) and the DR tier only provisions on ep0 (per-endpoint
allocation deferred, hub/README.md:260). Both are recorded deliberate positions,
so no R-n minted. The campaign touched neither ep0 nor the Storage Box.
Phase B did not start. Phase A was budgeted at ~1h and took ~5.5h (1.26.1 is a
public release image with no auto-install path, so the install was a blind
screendump+sendkey walk). That left the runner — which judges eleven invariants
and fires destructive atoms unattended — to be written at 04:00 with ~3h of
night left. Stopped on the brief's own fence: a rig producing false negatives is
worse than no rig. The rig is built and idle; teardown is OWED and itemised,
including hub customer c10-soak (disposition: DELETE).
The 2026-07-18 N100 rehearsal ran the complete final-product flow on real metal
in one pass: RESET -> generic pairing ISO v1.20.0 -> customer self-bind -> day-0
-> managed-floor lift -> escrow ceremony -> offsite snapshots. No code changed;
every finding is recorded and ranked, none fixed.
VALIDATION-n100-rehearsal-2026-07-18.md — run context, a UTC-normalised timeline
built from the hub events stream / hub DB / controller log / bootstrap + agent
journals, per-ledger verdicts for S1-S8 + ledgers 8 and 9, 12 findings, the
not-exercised list, and 7 discrepancies against the brief.
Headline wall-clocks: bind -> credential 26 s; bind -> controller running the
current version 2 min 44 s; managed floor 0.143.0 -> 0.145.0 in 5 s unattended
(initiated_by: auto-floor); escrow ceremony -> offsite enabled 12 s; drive enrol
30.3 s. No post-bind leg stalled, which is the immediacy row's real-onboarding
proof.
Capability map (10 citations added):
- Bare-metal Felhom ISO PARTIAL -> PROVEN-LIVE (F1 closed on metal)
- Customer self-bind (slice 1) IMPLEMENTED -> PROVEN-LIVE (customer_selfbind)
- Guest RAM resize (R-24) IMPLEMENTED -> PROVEN-LIVE (shrink AND grow)
- Customer RESET two real firings + verified external teardown
- Escrow ceremony first live wizard firing
- Immediacy row "real-onboarding proof pending" cleared
- Publish train box-side floor lift proven on a fresh install
- Customer claim R-4 gmail half (Inbox under p=quarantine)
- Offsite orphan guard staged live leg fired on its own
- DR tier by default candidate PROVEN-LIVE upgrade WITHDRAWN (R-39)
Not flipped, as instructed: customer-performs-restore, BYO, DLNA, multi-user.
ROADMAP — collapsed R-1 (appliance half done, Peti half survives), R-21
(physically closed), R-24, R-27 slice 1, R-4. New ranked items:
P2-HIGH R-39 PBS DR applied-but-dead R-30 liveness from the wait channel
R-31 async offsite + status R-32 RESET base-dir purge
R-33 bootstrap quiet-poll
P2 R-34 backup lifecycle R-35 config-apply session survival
R-36 post-RESET offsite prompt R-27c console-passphrase bind
P3 R-37 post-RESET health card R-38 installer GRUB slice
Plus a pre-invite checklist (golden 0.145.x rebuild, freemail.hu, C6, R-11).
R-39 is NEW and was not on the brief: the PBS DR descriptor auto-provisions and
the agent converges state=applied, but pvesm reports 401 Unauthorized/inactive
and a direct probe 401s on every endpoint including /version while WG is healthy.
The hub minted a second token secret two minutes after the agent applied the
first and consumed_at is still NULL; the converged state machine will not
re-apply, and the agent's verify loop cannot read the credential to notice it
(non-root read of a file it writes through a root wrapper). Rank is provisional
pending Viktor.
R-3 draft: all four [REFINE] slots filled, self-bind made the default path with
"send the link BEFORE the customer sees the console", the measured wall-clock
table added, and interim operator workarounds for R-31/R-36/R-39. C6 (renumbered
C7) is marked as the single unexecuted step and keeps the doc a DRAFT.
Evidence bundle: 180:~/n100-rehearsal/ (10 files + MANIFEST.md), collected before
the box was unplugged for travel. Secrets read only to run probes; recorded as
lengths and metadata, never values.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Closes N100 F1 (HIGH): cheap AMI (AN3PLUS 0.01-class) UEFI firmware can't
relocate the ISO's stock signed GRUB from USB (relocation 0x0). The run's live
grub-mkimage workaround is now a first-class pipeline mode.
- build-felhom-iso.sh: --loader shim|mkimage (default shim, byte-for-byte
unchanged; profile-settable FELHOM_LOADER; --loader wins). Loud banner +
manifest loader:/grub-mkimage: fields + -mkimage filename suffix.
- mkimage-surgery.sh (new): post-prepare-iso, in the assistant container. Builds
a monolithic grub-mkimage loader from the ISO's own GRUB (module set from its
grub.cfg; embedded search --fs-uuid -> configfile the real menu). Swaps it into
the ISO9660 tree (real lowercase path) + the efi.img ESP; xorriso re-master
preserves BIOS-hybrid + UEFI + GPT-ESP, drops only Apple HFS+/APM. Recipe from
the N100 run evidence, not re-derived.
- Dockerfile.assistant: grub-common + grub-efi-amd64-bin + mtools + dosfstools.
profiles/n100.profile (new, mkimage + SB-off note).
- Validated on nested VM 311 (RUNBOOK-B legs): leg1 shim boots+installs under
OVMF SB-enforcing + SeaBIOS; leg2 mkimage boots+installs under SB-off; leg3
(red-proof) mkimage under SB-enforcing FAILS Access Denied (unsigned -> SB must
be OFF); leg4 surgery byte-identical payload. bash -n + shellcheck clean.
Physical N100 closure folds into the rehearsal (n100-safety match-nothing ISO
built + sha-recorded, unbooted). PXE stays a deferred R-21 note.
F2 claim re-issue on clean-slate re-enroll (ReissueForReenroll, host-enroll mint path,
single-bump, reset code; hub never stores the password so fork B). F3 offsite re-issue on
re-enroll (ReissueOffsiteForCustomer, same machinery as the manual button). 2.3 escrow honesty
(red-proofed): re-issuing offsite marks the escrow stale (MarkEscrowStale), withholds the
mismatched restic hash from auto-confirm, DR checklist shows stale not done. Events:
claim_reissued_reenroll / offsite_reissued / escrow_stale.
Controller + scripts unchanged (source contradicted both premises): the controller reads escrow
prereqs live from the agent; the installer can't know the descriptor-provisioned storage id. F4
root fix is agent-side -> ROADMAP R-22; demo unblocked live (Part 0 ACL grant). VALIDATION doc
F2 erratum + F3/F4 dispositions. Green gate + Scenario-C red-proof pass.
Supervised clean-slate reinstall of the demo N100 from a pipeline-built Felhom ISO +
onboarding rehearsal. Core objectives GREEN: first-boot chain reached rc-0 on the first
attempt on real hardware (closes slice A's operator-gated boundary), serial-filter safety
proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict -> key on
MAC+UUID. Hard obstacle was firmware not the pipeline: this cheap AMI AN3PLUS 0.01 board
won't UEFI-boot the ISO's GRUB from USB (relocation 0x0; SB-off + shim-bypass don't help),
worked around live with a grub-mkimage loader from the box's own GRUB (slice-B input).
7 findings logged (F1 firmware loader HIGH; F2/F3/F4 reused-customer edges -> R-1/Peti;
F5 guest-RAM, F6 drive-init mount, F7 back-route). Full doc in documentation/tests/;
ROADMAP R-21 + REPORT + CONTEXT updated. No production code changed.
Correction: campaign-2's "R1" (bind-mount restore needs root) was a SYMPTOM of an
empty felhom pool -> agent lacked VM.Audit on 9201 -> restore-test's existing,
correct bindMountOverrides never ran. Live-proven: pool healed (Part A) ->
restore-test PASSES (neutralizes 2 binds, boot+running, clean teardown, 4m35s).
Report rec #1 (make pct restore work on bind mounts) superseded. Fixes shipped:
agent v0.74.0 (pool re-assertion), controller v0.103.0 (F-C2-1).
Spike (SPIKE-dr-bindmount-source-2026-07-07): DR bring-up source options — PBS
pct.conf.blob blocked by encryption under the non-root token; DR recipe lacks mpN;
mp8/mp9 are STRUCTURAL CONSTANTS -> recommend a small known-constant override
reusing bindMountOverrides.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Full-system validation on demo 9201 (controller v0.102.0 / agent v0.73.0). Headline:
ONE genuine pilot-critical finding — the non-root agent cannot pct-restore the
standard Felhom guest because it has bind-mount mountpoints (mp8 /mnt/felhom-drives,
mp9 bootstrap) and pct restore refuses bind mounts unless root. This breaks BOTH
--selftest=restore-test AND the offsite DR bring-up (R1), linked to a missing
VM.Audit/VM.Allocate on the agent token (R2). The large raw FAIL count is dominated
by one harness bug (auth-session expiry mis-scoring P3-P9) plus D1-injection/soak-probe/
dryrun-journal/escrow-blob calibration issues; the product where actually exercised was
healthy with ZERO resource leaks over ~7h. Demo node healed to pre-campaign state
(auth off, notifications restored, no scratch leaks). No production code changed.
Evidence snapshot + README under campaign2-evidence-2026-07-07/ (cookie jars scrubbed).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
sys_drive=disk-2 distinct, selftest [PASS] System data path; disk-full ENOSPC
confined to disk-2 (/ + docker-data unaffected, apps healthy); mp1 in vzdump;
SSD-fallback app (radarr) data lands on disk-2. 2.5 size=50G; reboot-survival
in 3.3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
1.6 deploy/HDD_PATH stable, 1.7 eject clean stable removal (v0.80 ruled out) +
impact names apps, 1.9 migrate flips stable->stable w/ verify-before-cleanup,
1.10 decommission via raw agentWhere + reconnect restore, 1.11 backups under
stable namespace. FINDING F1: restic binary still apt-installed (Dockerfile:58)
though all restic code moved to agent — dead weight. navidrome now on flash.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
Read-only diagnosis: restart-manager works (natural-exit probe RestartCount climbs;
manualRestart=false); docker kill is a manual stop so restart is correctly suppressed.
Definitive: kill -9 the controller PID (real crash) -> unless-stopped auto-restarts it,
RestartCount=1 healthy in 5s. Daemon stable (NRestarts=0, no OOM). Finding #1 downgraded
in the campaign-2 doc. Separate minor obs: boot-time drive-ordering race. No fix applied.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Storage lifecycle (unblocked by v0.74.0), break-and-recover C1-C8, backup edge cases.
Headline finding: docker restart-policy not honored -> a crashed felhom-controller has
no automatic recovery (oneshot bootstrap + dead restart policy). Plus disconnect-recreate
race, agent loopback whitelist, carried-forward #1 gaps. Demo restored to baseline.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confirmed H5 (ephemeral-port exhaustion): agentClient() builds a new http.Transport
per call (IdleConnTimeout:0, never CloseIdleConnections) -> ~5.8k leaked idle ESTABLISHED
sockets/day to 162:8443, exhausts the 28k ephemeral range in ~5 days of controller uptime
-> EADDRNOTAVAIL. :8006 immune (controller never dials pveproxy). Cleared by restart.
H1/H2/H3/H4/H6 ruled out with positive evidence. Fix is controller-side (reuse one
Client), NOT an agent rebind. No changes applied.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validated both unvalidated 10D mechanisms: (1) identity-bundle escrow round-trip
via age scrypt+AEAD (recover on a secret-less box, wrong-R fails closed), (2)
Cloudflare tunnel re-establishment — running the recovered token on a new box
routes the hostname there immediately (no DNS change); the old connector is a
hot standby, superseded in routing but not auto-retired -> 10D must rotate the
tunnel/PBS token + retire the stale connector for host-loss security. Redacted;
secrets shredded; live demo untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validated escrow consumption end-to-end on a genuinely key-less box against
the real felhom-spike datastore: recover K from (blob,R) via the real
escrow.Unwrap, restore REAL data (spike-lxc rootfs, 2.5G) with the recovered
key only, wrong-R fails closed (no plausible-but-wrong key), live K
byte-unchanged. Redacted (no R/K/secret). GO to spec 10C + build 10D.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Doc-only spike (no hub code change). Validated on demo-felhom (guest 8200,
torn down): (1) guest->host HTTPS over vmbr0 with fingerprint-pin + bearer +
self-scoping (200/401/403, wrong-pin TLS fail, no firewall rule needed);
(2) config-mount + golden-baked bootstrap unit deploys+runs the controller
(docker login/pull/run v0.34.0) with no pct exec. Verdict: GO to 8A spec.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Validated wrap->lose->unwrap->restore on a fenced throwaway: the R-recovered key
decrypts a real encrypted snapshot. Pins the PBS-native command sequence (key
change-passphrase --kdf scrypt/none), the pty requirement (F-A1: TTY-only, env var
ignored) + the echo caveat (F-A2: discard pty output so R can't leak), the blob
format/size, and the R format (EFF wordlist, >=128-bit). No K/R/token value recorded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SPIKE-RUNBOOK Slice 7 Phase 0, executed live on demo-felhom. Golden base
(Debian 13 + Docker, nesting=1,keyctl=1, identity-cleaned) built as root@pam,
archived, then token-restored to a throwaway guest and brought up LINK-UP with
the FelhomAgent token (restore/config/resize/start all token-covered).
Key findings:
- MAC reset is UNCONDITIONAL — vzrestore preserves the archived MAC (F1).
- hostname reset is host-side token config (F2).
- machine-id auto-regenerates on first boot (free); SSH host keys do NOT —
ssh.service fails, agent must run ssh-keygen -A guest-side OR bake a first-boot
unit (F3, the one surface-widening design consequence).
- keyctl-through-restore is functional (Docker hello-world in the restored guest);
storage driver overlayfs (F5/F6).
- Settles the §9 / doc-13 identity-reset field list for the provision path.
Verdict: READY to spec the unified bring-up reconcile job (Phase 7.1).
Golden archive kept; both spike guests torn down.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Empirical PBS validation before the slice-6 Phase B spec. Records: PBS install on
Debian-13 DooPlex (trixie key ships in proxmox-archive-keyring, no standalone .gpg),
datastore + cert fingerprint, the PBS privsep gotcha (grant role on user AND token),
the encrypted pbs storage + key location (/etc/pve/priv/storage/<id>.enc), the snapshot
volid format + native fields (→ PBSSnapshot shape), restore-from-PBS works unchanged,
the verify mechanism (server-side; agent drives it remotely via the PBS API, result read
from snapshot verification.state), no operator-token privilege gap, and zero-knowledge
confirmed (server can't decrypt without the client key). PBS+datastore+storage left up
for Phase B; no secrets committed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin