VERDICT. H1 confirmed: the reconstitution never ran. /backup/offbox/reconstitute
was never hit (reconstituted/safety-dump/replay line counts all 0); the operator
clicked the old missing-only button, which merged 34 files and left the DB
untouched. H2 rejected (the full path never executed; staging used the correct
snapshot 49e7cb46). H3 rejected (no replay happened; DB was genuinely asset:0).
R-44 is exonerated and working: 49e7cb46 carries offsite_run_id 20260719T102851Z
and its dump probes to asset:11 / user:1 — a properly coherent pair, against
round 1's asset:0 / user:0.
H4 (NEW, the more important result). Phase-3 recovery ran the real sequence and
the v0.148.0 path executed correctly — safety dump, stop, start, replay — then
the replay ABORTED: relation "clip_index" already exists. Proven to the second:
10:58:25 controller begins replay
10:58:33 immich-server "Reindexing clip_index" -> "Reindexed clip_index"
10:58:35 dump's CREATE INDEX clip_index fails, exit 3
The replay races the app's own schema repair, because reconstitution starts the
stack before replaying (ImportDump needs a live container). The photos survived
only because pg_dump emits COPY data before CREATE INDEX — luck, not design. The
same start-then-replay shape is on the LOCAL path, so it is a class defect.
Recovery: PARTIAL and STOPPED as instructed. All 11 assets are back and visible
by every DB criterion (status=active, deletedAt null, all files resolve), but the
operation reported failure and immich reports schema drift. Not clean acceptance
evidence; no second attempt made; safety dump read only.
Findings -> R-47 (the replay race), R-48 (restore controls separable only by
layout — the proven cause of this incident), R-49 (1.1 GB of a 1.2 GB immich
backup is ML model cache + a postgres data-dir tar duplicating the .sql dump +
immich's own nightly dump; recorded, not changed).
RULINGS LANDED.
- 00-capability-map.md:61 -> PARTIAL, scope-corrected: 6D's destruction hit the
FILE TREE only; the DB survived in its named volume, so "end-to-end from
offsite alone" overclaimed scope.
- The 704.6 MiB "discrepancy" -> CLOSED, not a defect: it was immich's own
Tarhely widget, never a controller page. Same for round 2's 650MiB->1.4GiB.
- customer-restore row: partial evidence recorded, NOT flipped.
Diagnosis only — no code, label or layout changes; no forget/prune.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
ROADMAP R-43/R-44 → SHIPPED with what actually landed:
- R-43: ReconstituteFromOffsite — safety dump → stop → files overwritten to
the snapshot version → start → the snapshot's own dump replayed from the
SCRATCH unit (the live unit is still never overwritten). Invariants:
nothing is ever deleted, and the undo is verified on disk before the act.
- R-44: every run, manual AND nightly, refreshes dumps + units BEFORE the
capture, so each snapshot is a coherent {DB@T, files@T} pair stamped with
offsite_run_id + dumps_at. Nightly ordering is now structural rather than a
coincidence of two scheduler entries at 02:30 and 04:15.
DIAG doc gains a "fixed in v0.148.0" header pointing at both.
Capability map deliberately NOT flipped: the DB half now exists but is PARTIAL,
not PROVEN-LIVE — shipped and deployed with no live acceptance behind it. The
customer-restore row stays MISSING, now blocked only on the evidence run rather
than on missing capability, and R-3 stays DRAFT.
Still open: the 00-capability-map.md:61 ruling — did CAMPAIGN-6D's "immich
end-to-end from offsite alone" exercise the DB half, or only the file half?
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
Viktor deleted 11 immich photos to test offsite restore; both place runs
merged 0 files and flashed success. Two-layered root cause:
- A UI delete means TRASH, not deletion — no file left the disk, so there
was genuinely nothing missing to restore. The test proved nothing.
- A valid test would also have failed: no offsite path loads a database.
All three buttons are file-only (offbox_restore.go:171-232, :346-421).
Files return; the library stays empty. The only DB replay is
RestoreFromRecoveryUnit, reachable solely from the LOCAL restore panel.
Compounding: the manual push never refreshes dumps (offbox_handlers.go:
203-227) — today's unit dump is from 02:30 and probes to asset:0 / user:0 /
album:0, predating both the admin user and the photos. Had a full restore
loaded it, it would have written an empty DB over the live one, destroying
the trashed rows that were the only surviving recovery path.
Answer to "can a customer trust same-day offsite?" for a DB-indexed app:
no — files come back, content does not.
No product code changed. Photos left in trash (recovery not wanted).
Capability map: customer-restore row stays MISSING by finding, not by
absence of attempt; offsite-restore row flagged scope-contested pending
a ruling on whether CAMPAIGN-6D exercised the DB half.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
The 2026-07-18 N100 rehearsal ran the complete final-product flow on real metal
in one pass: RESET -> generic pairing ISO v1.20.0 -> customer self-bind -> day-0
-> managed-floor lift -> escrow ceremony -> offsite snapshots. No code changed;
every finding is recorded and ranked, none fixed.
VALIDATION-n100-rehearsal-2026-07-18.md — run context, a UTC-normalised timeline
built from the hub events stream / hub DB / controller log / bootstrap + agent
journals, per-ledger verdicts for S1-S8 + ledgers 8 and 9, 12 findings, the
not-exercised list, and 7 discrepancies against the brief.
Headline wall-clocks: bind -> credential 26 s; bind -> controller running the
current version 2 min 44 s; managed floor 0.143.0 -> 0.145.0 in 5 s unattended
(initiated_by: auto-floor); escrow ceremony -> offsite enabled 12 s; drive enrol
30.3 s. No post-bind leg stalled, which is the immediacy row's real-onboarding
proof.
Capability map (10 citations added):
- Bare-metal Felhom ISO PARTIAL -> PROVEN-LIVE (F1 closed on metal)
- Customer self-bind (slice 1) IMPLEMENTED -> PROVEN-LIVE (customer_selfbind)
- Guest RAM resize (R-24) IMPLEMENTED -> PROVEN-LIVE (shrink AND grow)
- Customer RESET two real firings + verified external teardown
- Escrow ceremony first live wizard firing
- Immediacy row "real-onboarding proof pending" cleared
- Publish train box-side floor lift proven on a fresh install
- Customer claim R-4 gmail half (Inbox under p=quarantine)
- Offsite orphan guard staged live leg fired on its own
- DR tier by default candidate PROVEN-LIVE upgrade WITHDRAWN (R-39)
Not flipped, as instructed: customer-performs-restore, BYO, DLNA, multi-user.
ROADMAP — collapsed R-1 (appliance half done, Peti half survives), R-21
(physically closed), R-24, R-27 slice 1, R-4. New ranked items:
P2-HIGH R-39 PBS DR applied-but-dead R-30 liveness from the wait channel
R-31 async offsite + status R-32 RESET base-dir purge
R-33 bootstrap quiet-poll
P2 R-34 backup lifecycle R-35 config-apply session survival
R-36 post-RESET offsite prompt R-27c console-passphrase bind
P3 R-37 post-RESET health card R-38 installer GRUB slice
Plus a pre-invite checklist (golden 0.145.x rebuild, freemail.hu, C6, R-11).
R-39 is NEW and was not on the brief: the PBS DR descriptor auto-provisions and
the agent converges state=applied, but pvesm reports 401 Unauthorized/inactive
and a direct probe 401s on every endpoint including /version while WG is healthy.
The hub minted a second token secret two minutes after the agent applied the
first and consumed_at is still NULL; the converged state machine will not
re-apply, and the agent's verify loop cannot read the credential to notice it
(non-root read of a file it writes through a root wrapper). Rank is provisional
pending Viktor.
R-3 draft: all four [REFINE] slots filled, self-bind made the default path with
"send the link BEFORE the customer sees the console", the measured wall-clock
table added, and interim operator workarounds for R-31/R-36/R-39. C6 (renumbered
C7) is marked as the single unexecuted step and keeps the doc a DRAFT.
Evidence bundle: 180:~/n100-rehearsal/ (10 files + MANIFEST.md), collected before
the box was unplugged for travel. Secrets read only to run probes; recorded as
lengths and metadata, never values.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Drops the false 'no offsite target on the demo box' clause from the ROADMAP row,
the capability-map SMB row and sharing.md. Cites offsite snapshots e0b9d723 /
4e2b15ec and the restore round-trip results. Root cause (guessed settings key) is
recorded in felhom-controller REPORT section 7b.
- capability map: SMB row KNOWN GAP cleared -> share data rides both tiers; the
offsite leg + restore round-trip flagged as not-yet-live-exercised
- ROADMAP R-7b: idea -> SHIPPED, with the Model B' rationale and the live evidence
- controller/sharing.md: the KNOWN GAP block replaced by the execution contract;
operator note corrected — samba IS liveness-monitored since v0.145.0
Viktor's human leg closed the last gate: both shares open from the Windows
Network view, an interactive Explorer save landed as uid 1000, and a write into
the read-only share was refused with the folder untouched. Capability map row
flipped to PROVEN-LIVE with that evidence; ROADMAP R-7 + sharing.md updated.
R-7b (shares classified but not in any live backup run) remains open.
controller/sharing.md (code-verified vs controller v0.144.0 + felhom-samba
1.0.0); capability map 'Files from Windows Explorer / Mac Finder (SMB server)'
MISSING -> IMPLEMENTED (PROVEN-LIVE pending Viktor's Explorer leg); ROADMAP R-7
-> shipped-slice-1 with the slice-2 remainder, and the backup design fork split
out as R-7b (shares are classified but not in any live backup run yet).
Let a customer bind their own freshly-installed appliance without the
operator: operator "Send self-bind link" mints a 7-day tokenized
capability link, emailed (Hungarian, sibling sender) to the customer, who
opens a public /bind/<token> page and proves two factors — the console
pairing code shown on the box screen + their retrieval passphrase — and
the hub stages the bind via the same BindAppliance (provenance
customer_selfbind). The box's ~30s appliance poll delivers.
Viktor's three rulings verbatim: console pairing code (no appliance list
ever rendered), operator-sent tokenized link, 5-attempt lockout ->
"call support". Wrong code == wrong passphrase (one generic failure, no
oracle, both factors compared unconditionally); expiry falls back to
operator-bind unchanged.
THE TRAP: one public prefix /bind/, exempt from auth+CSRF at both /login
gate sites via a single isPublicBindPath predicate (tight trailing-slash
match; ServeMux ..-cleans; handler rejects '/' in token). 9 tests
(Scenarios A-F + F1/F2); 4 red-proofs verified red-then-green (lockout,
oracle, widened-prefix, single-active). GC verdict: no appliance GC ->
the 7-day TTL stands alone. Controller/agent untouched; R-27b deferred.
Green: full hub build/vet/test (17 ok) + bash -n + hub confirm gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
Closes N100 F1 (HIGH): cheap AMI (AN3PLUS 0.01-class) UEFI firmware can't
relocate the ISO's stock signed GRUB from USB (relocation 0x0). The run's live
grub-mkimage workaround is now a first-class pipeline mode.
- build-felhom-iso.sh: --loader shim|mkimage (default shim, byte-for-byte
unchanged; profile-settable FELHOM_LOADER; --loader wins). Loud banner +
manifest loader:/grub-mkimage: fields + -mkimage filename suffix.
- mkimage-surgery.sh (new): post-prepare-iso, in the assistant container. Builds
a monolithic grub-mkimage loader from the ISO's own GRUB (module set from its
grub.cfg; embedded search --fs-uuid -> configfile the real menu). Swaps it into
the ISO9660 tree (real lowercase path) + the efi.img ESP; xorriso re-master
preserves BIOS-hybrid + UEFI + GPT-ESP, drops only Apple HFS+/APM. Recipe from
the N100 run evidence, not re-derived.
- Dockerfile.assistant: grub-common + grub-efi-amd64-bin + mtools + dosfstools.
profiles/n100.profile (new, mkimage + SB-off note).
- Validated on nested VM 311 (RUNBOOK-B legs): leg1 shim boots+installs under
OVMF SB-enforcing + SeaBIOS; leg2 mkimage boots+installs under SB-off; leg3
(red-proof) mkimage under SB-enforcing FAILS Access Denied (unsigned -> SB must
be OFF); leg4 surgery byte-identical payload. bash -n + shellcheck clean.
Physical N100 closure folds into the rehearsal (n100-safety match-nothing ISO
built + sha-recorded, unbooted). PXE stays a deferred R-21 note.
- DeleteHost(deleteEscrow) demotes current host_escrow into host_escrow_superseded (copy-before-delete, same tx), spares existing; one shared demoteCurrentEscrowTx (reused by SaveHostEscrow). F-14 provenance/gate unchanged.
- DeleteCustomerConfig now purges both escrow tables for all the customer's hosts incl. already-deleted (F-14 provenance UNION) — the one true purge point.
- Wording: checkbox/refusal/Danger-zone → demotion. S6b OBSOLETE. Red-proofs TestDeleteHost_Demotes + TestDeleteCustomer_Purges + wording guard.
Opened every citation in 00-capability-map.md and judged each of the 55 rows
against the specific scenario (not topic adjacency). 23 confirmed (incl. 5
MISSING proven-absent by code sweep), 26 citation-corrected, 6 demoted; no
upgrades. Status: 32 PROVEN-LIVE / 15 IMPLEMENTED / 3 PARTIAL / 5 MISSING
(was 38 PROVEN-LIVE).
Demotions PROVEN-LIVE->IMPLEMENTED: B6 (post-deploy config — T-PAGE-ALL is
render-only), E7 (FileBrowser access — no browse/download proof), F1 (health
email — cited P3-DELIVERY is the enlarge-block trigger), F4 (metrics — cited
legs are H1/H2 harness artifacts), F6 (operator alerting — no corpus doc),
G6 (offsite fill/staleness/freeze — no live-fired leg, freeze inconclusive).
Key corrections: D6 cited CAMPAIGN-3 F10/F11/F12 which are the CRITICAL
failures — fixes shipped agent v0.85 and were re-validated live in CAMPAIGN-4
/6A; C6 cited offbox-restic doc for a PBS-restore claim -> CAMPAIGN-2 T-P9;
C1/C5/B5 cited defect-discovery docs instead of the fix-validation legs.
Closed ROADMAP R-20: operator-key pinning is fully automatic in day-0
(felhom-host-install.sh step_agent_config, L2044/2146-2156/2332-2337).
Header draft caveat swapped for the verified stamp. Full per-row verdict
table in REPORT.md.
felhom-host-install.sh installs felhom-selfupdate-guarded (sh -n), the rollback
unit + the [Unit] start-limit drop-in (daemon-reload) so day-0 boxes get
operator-signed self-update from birth; non-fatal on pre-D1 agent repos.
03-host-agent.md §11 updated to the shipped mechanism (signed op, A/B wrapper,
OnFailure rollback, tuned start-limit backstop).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
escrow-create -storage felhom-offsite -identity-bundle -directive -upload on
felhom-pve: fresh R (operator-recorded off-transcript) wraps K + WG key + offsite
PBS token; uploaded to hub. Independently verified in host_escrow (demo-felhom-01:
key_fingerprint=offsite b0:fe:2a…, identity blob 499B, directive present,
created_at=now) with zero knowledge of R. S5 (DR consume) now UNBLOCKED.
This re-run also remediated an earlier escrow whose R had leaked into an agent
transcript (last-write-wins overwrote it with the clean-R blob).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Offsite restore-test now runs unattended on the agent-token path (agent v0.68.0):
tier-aware restore-task timeout (pbs→120m, local→10m). The scratch-band
VM.Allocate follow-up was a phantom — the scratch restores INTO /pool/felhom
(ACL already grants it); the earlier 403 was the 10m-timeout consequence. No
ACL/host-install change. doc-06 S4 row + runbook §4b updated.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The CGNAT smoke test's OPEN DECISION is made: fleet-wide, permanent,
family-agnostic client MTU 1280 (RFC 8200 IPv6-minimum floor), implemented in
felhom-agent v0.65.0.
- §4.3: OPEN DECISION → made decision + rationale + rejected alternatives
(auto-probe/per-type = fragility for a non-metric; MSS-clamp = unneeded).
"ships MTU 1420" → 1280; corrected "fixed-line unaffected" (DS-Lite ~1452,
PPPoE would also black-hole at 1420 — one floor covers all).
- §4.2: 1280 is family-agnostic so MTU correctness is decoupled from v4/v6
resolution; v4-pin noted as a separate optional determinism item.
- §3.2 wire-field note + §8 S6 row: MTU 1420 → 1280; S6 residual now only the
true-CGNAT-SIM retest.
- REPORT.md overwritten.
Docs-only; the constant change is in felhom-agent v0.65.0.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
S3 appendix CGNAT smoke test run live on felhom-pve (agent v0.64.0) with a
USB-tethered phone. Tunnel established + held its NAT mapping across a 32-min
idle soak on PersistentKeepalive=25 alone (0 stalls, final ping no-bounce).
- §7: open-validation CLOSED; two honest caveats kept (SIM had a public mobile
IPv4 so true CGNAT 100.64/10 not reproduced — retest-when-able; + MTU).
- §4.3: mobile outer path MTU ~1400 black-holed bulk data at the shipped 1420;
MTU 1340 restored the PBS page. OPEN DECISION: permanent vs per-connection.
- §4.4: keepalive 32-min mobile-NAT hold datapoint.
- §4.2: dual-stack v6 trap (wg-quick prefers endpoint AAAA, rides un-NATed v6).
- §8: S6 row CGNAT closure marked done.
- REPORT.md: overwritten with this operational-validation summary.
Runbook: RUNBOOK-s3-cgnat-smoke. Docs-only; no agent/hub/controller code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Records the settled transport decisions (plain WG, host-side agent-managed pilot,
one hub-driven endpoint VM running WG+PBS, hub source-of-truth over WireDesiredState,
one datastore + per-customer namespaces, relay-through-DooPlex rejected). Grounded
at file:line vs felhom.eu@bf099f6 + felhom-agent@4ba1b14. Backlog line resolved to
cite spike + design doc. Docs-only; slices are separate future tasks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
- 05-hub-architecture.md: stale 'felhom-hub v0.6.3' -> v0.11.0 (design-draft note).
- auth-hub-sync-integrations.md: full onlyoffice:nextcloud occ command sequence.
- storage-monitoring-metrics.md: metricsDBPath verified to coincide with the volume-backed
data_dir on the bootstrap guest (persists; hardcoding is latent fragility only).
- REORG-NOTES: gaps 2/3 CLOSED, gap 5 partially closed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agent-only implementation (felhom-agent v0.17.0 escrow.Consume); no hub code
change. 10C done; 10D is the last piece of slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add DELETE /hosts/{id}/jobs/{job_id} (per-host self-scoped, idempotent) so the
agent clears a job after executing or terminally rejecting it. The hub stores
the operator-signed blobs opaquely (no signing key — cannot forge or open);
the agent verifies + executes. Doc 03 §4/§6/§9 updated (operator-signed path
live; 8C wipe completes; 10B done).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Serve operator intent to authenticated hosts: PUT /admin/hosts/{id}/desired-state
(global key) bumps desired_generation; GET /hosts/{id}/desired-state + /jobs are
per-host self-scoped; the host-report envelope now carries the real generation +
has_signed_ops. New signed_jobs table + store methods. Desired-state stored/served
opaquely (agent owns the schema). Cross-repo golden (envelope + desired-state)
byte-identical with felhom-agent; doc 03 §4/§9 updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the cross-repo host-report golden byte-identical with felhom-agent
(host.cpu_temp_c). Document GET /host/metrics in doc 03 section 6 and define
slice 9 in the section 9 roadmap. No hub code change / no version bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin