Closes the two deferred deep campaigns from 6D on live infra: tier-2 v2 layout
engine (A1-A4) and F7 atomic-write + restic C2 self-heal (B1-B4). No production
code shipped. Two LOW findings (F-6E-1 demo drive drift; T-6E-1 DB-dump dir-fsync
parity). Drill repo torn down; fixtures reverted; box healthy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Refutes the surviving theory: on the drill host the pbs_dr descriptor is present
and enabled, the WG peer exists, verify passes — the block is a one-time secret
consumed 07-12 that no path re-mints after the agent loses its converged marker
(re-install/rollback; same WG pubkey -> changed==false -> cascade can't re-fire).
Live-proven: staging any consumable secret converges in one tick (existing token,
zero ep0 churn); re-asserting a converged descriptor is a clean idempotent no-op.
Safe re-trigger = re-serve a secret gated on agent waiting_secret/consumed_failed,
never blind-timer Reissue. Design-inputs table handed to the self-heal TASK spec.
Drill left CONVERGED (P-DAY0-DEEP PBS-DR leg now GREEN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Fresh-appliance PBS-DR consume races the WG handshake at install (fingerprint probe timeout),
consumes nothing, and does not re-request the descriptor once the tunnel is up; reused-peer
re-provision never re-issues it. No operational re-trigger. Escrow ceremony + offsite round-trip
blocked on this fresh guest → friend-alpha alpha #1 escrow must be supervised live (customer-holds-R).
Drill deviation (CC-held R) never exercised. Recorded, not fixed inline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
immich enlargement blocked at real 50 GiB quota via sparse mandatory fixture; unit-only
continues; EnlargedBlocked edge-fires once; both operator + customer emails delivered;
fully reverted. G4 notification email restored via the real form (wipe-bug regression).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Core-promise legs proven on real data: .fab >=1GiB full circle (byte-identical mandatory,
zero sibling contamination, DB round-trip) + SQ3 immich offsite restore-to-live (functional
from offsite alone). Deferred: P3-BROWSER/P3-DELIVERY/P-TIER2-deep/P4-timing/P-DAY0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Unattended close-out of the C6 remainder. Flagship finding C6B-F1 (CRITICAL):
.fab export silently produces hollow, data-free bundles for the standard
${USERDATA_PATH} needs_hdd convention (12/13 media apps) — sonarr 4.17 GB -> a
2308-byte config-only bundle, past the v0.125.0 anti-hollow guard. Transport
(upload/download, real CF edge, 413, byte-integrity) all PASS. C6B-F2 (MED,
confounded): net-storage removal leaves an orphaned autofs mount. Browser-
interactive + deep backup-tier items split to CAMPAIGN-6C. No code changed; no
secret/R/blob committed; campaign credential left for Viktor to rotate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
CAMPAIGN-5 findings doc (2026-07-14): NAS re-arm ring core + v0.129.0 fix live-proof
Findings-only. v0.129.0 F-A/F-B/F-C all CONFIRMED FIXED live on the fleet. NAS
re-arm ring core PASS on a fresh campaign NFS share: F10 start-limit clear +
reset-failed+rearmed, F9 per-share verdict (no empty sweep), F1/F2 clean removal,
F12 unit cycle-free. F8 improved (stub/mounted:false during outage) but reachable:true
still server-level. Reboot half of the matrix + upload/browser planes deferred with
procedures. No credential/R/blob committed.
Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd
@
Audit doc for the Day-0 VM drill: appliance install, floor-at-first-report,
escrow ceremony + auto-confirm FIRST LIVE FIRING (~7.5 min, zero clicks),
offsite backup + restore round-trip, PBS-DR/WG fork (Peti-sequence rehearsal).
Corrects day0-install.md A.2 (git creds optional since v1.11.2, anonymous
fetch is the normal shape) and RUNBOOK-escrow-ceremony.md (identity-only mode
does NOT exist — F-6; age prereq — F-10). REPORT.md overwritten; CONTEXT.md
one-liner added.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
journalctl proved boot -1 came up at 23:31 WITHOUT networking: the agent automount
template's After/Wants=network-online.target cycles with local-fs.target and systemd
deleted networking.service to break it; the 06:45 power-cycle boot hit the same cycle
and deleted the automount instead. Every boot with an enrolled share is a coin flip
until the template drops that ordering. Recovery executed: automount started, apps
healed, privatebin removed via real flow, verification backup green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
Supervised live proof on the demo (agent 0.82.0 + controller 0.115.0): after pct reboot 9201 the
autofs trigger does not re-propagate into the guest; an in-guest access sees an empty dir and does
not trigger the host mount. Root cause: shared->slave propagation + no agent network-mount reassert
on guest reboot. FAIL protocol honored (nothing repaired live); fix spec'd in backlog.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Unattended live soak of the build-complete offsite arc. F4 re-issue + restore
round-trip proved live (9/9 byte-identical, immich db-dump exact). Crash
recovery surfaced a HIGH resilience gap: crash mid-prune leaves a stale restic
exclusive lock the controller can't self-heal → subsequent runs fail until
manual unlock. Demo restored to start-state. Track B (quota) + hub-orchestrated
A blocked by session-gating → Peti runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Full hub->Hetzner->controller loop validated on demo-felhom against the new
dedicated-project pool box; 2 snapshots on the new box, key-only. F1/F3 fixed
mid-run (hub v0.38.1, controller v0.106.1); F2/F4/F5/F6 recorded open.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Full cycle on the demo: enable→stage→real escrow-create (K+restic-pw under a
fresh R, uploaded to hub, staged wiped, K byte-unchanged)→atomicity proof (run
skipped before confirm, proceeded after — gate opens exactly on confirm). DR
money shot: restic password recovered from the PRODUCTION hub blob (via
identity-consume with R) is byte-identical to the live pw (sha 24c44f49) and
decrypted the real box repo; inject→restore endpoints reproduced a 45MB db-dump
byte-perfect. Demo left escrowed (un-paused); R held out-of-band; no secret in
logs; repo_password unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Money shot (Probe 3): a password recovered from an age-under-R escrow of the
real repo password DECRYPTED the real Storage-Box repo (listed immich +
audiobookshelf snapshots) — working key, not just byte-match. Probe 1:
round-trip byte-perfect + wrong-R fails closed (0 plaintext). Probe 2:
controller→agent POST-body transport already proven (withGuest) — stage
endpoint is additive. Probe 4: WriteOffboxSecrets honors a pre-placed password
(DR re-injection), but config endpoint carries no password → explicit inject
step needed. All probes throwaway/discarded; live password + repo untouched.
Impl blueprint for the fork-4 spec inside.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Real endpoints, fresh box sub-account. Discovery proof: immich (undeployed,
USB-only unit — the v0.103.0 mis-resolution case) → ok/1 snapshot on the USB
path; 45MB db-dump restore sha256-matched. No-silent-success on the live box:
0/N → error + backup_failed alert; partial → ok + LastWarning. Persists across
restart. Forks 1+2 confirmed (direct; sub-account /home/<name>, ssh-copy-id -s
-f, console key field still broken F3); fork-4 custody surface recorded, escrow
design OPEN (repo pw + key not independently recoverable). Kept as standing
demo-offsite.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Validated the controller offbox restic-SFTP tier direct from the residential
demo box (IPv4-only egress) end-to-end through the real endpoints: reach on
:23 (PASS), chown-immune restic init (works where PBS-direct failed),
config→run→restore green, sub-account isolation confirmed. Residential upload
~4.3 MB/s. Concentrator NOT needed. KEY BLOCKER F1: offbox reads recovery unit
from the app data-drive namespace root, not the actual backup-drive location →
silently backs up 0 apps (ok status) on multi-drive setups. Spike only,
synthetic data, torn down.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Wired the 1TB BX11 Storage Box to ep0 as a durable key-auth SSHFS mount
(/mnt/pbs-storagebox, reboot-proven). Findings: direct PBS datastore UNFIT
(box rejects chown -> ChunkStore::create fails); loopback-ext4-on-SSHFS FIT
with caveats (GC 7s/verify 12s/restore 11s all green, no GC hang). Raw mount
~327MB/s write, ~69MB/s read. Spike only, synthetic data, torn down.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Correction: campaign-2's "R1" (bind-mount restore needs root) was a SYMPTOM of an
empty felhom pool -> agent lacked VM.Audit on 9201 -> restore-test's existing,
correct bindMountOverrides never ran. Live-proven: pool healed (Part A) ->
restore-test PASSES (neutralizes 2 binds, boot+running, clean teardown, 4m35s).
Report rec #1 (make pct restore work on bind mounts) superseded. Fixes shipped:
agent v0.74.0 (pool re-assertion), controller v0.103.0 (F-C2-1).
Spike (SPIKE-dr-bindmount-source-2026-07-07): DR bring-up source options — PBS
pct.conf.blob blocked by encryption under the non-root token; DR recipe lacks mpN;
mp8/mp9 are STRUCTURAL CONSTANTS -> recommend a small known-constant override
reusing bindMountOverrides.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Pilot-blockers bundle shipped + live-validated: agent v0.73.0 (F2 mount-role
fallback), controller v0.102.0 (async restore family, F4 re-adjudicated as a UX
bug not a restore bug), catalog healthcheck sweep. All three accepted live on the
demo node.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Correct the end-state: uptime-kuma redeployed healthy, notifications restored;
vaultwarden serves (127.0.0.1 /alive 200) but reads docker-unhealthy due to a
localhost->IPv6 healthcheck probe quirk (reproduced on fresh redeploy);
bookstack unhealthy at app level (HTTP 500, pre-existing). Recommend catalog
healthchecks use 127.0.0.1 not localhost. Neither is a P1-P3 defect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
admin