Unattended live soak of the build-complete offsite arc. F4 re-issue + restore
round-trip proved live (9/9 byte-identical, immich db-dump exact). Crash
recovery surfaced a HIGH resilience gap: crash mid-prune leaves a stale restic
exclusive lock the controller can't self-heal → subsequent runs fail until
manual unlock. Demo restored to start-state. Track B (quota) + hub-orchestrated
A blocked by session-gating → Peti runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Full hub->Hetzner->controller loop validated on demo-felhom against the new
dedicated-project pool box; 2 snapshots on the new box, key-only. F1/F3 fixed
mid-run (hub v0.38.1, controller v0.106.1); F2/F4/F5/F6 recorded open.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Full cycle on the demo: enable→stage→real escrow-create (K+restic-pw under a
fresh R, uploaded to hub, staged wiped, K byte-unchanged)→atomicity proof (run
skipped before confirm, proceeded after — gate opens exactly on confirm). DR
money shot: restic password recovered from the PRODUCTION hub blob (via
identity-consume with R) is byte-identical to the live pw (sha 24c44f49) and
decrypted the real box repo; inject→restore endpoints reproduced a 45MB db-dump
byte-perfect. Demo left escrowed (un-paused); R held out-of-band; no secret in
logs; repo_password unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Money shot (Probe 3): a password recovered from an age-under-R escrow of the
real repo password DECRYPTED the real Storage-Box repo (listed immich +
audiobookshelf snapshots) — working key, not just byte-match. Probe 1:
round-trip byte-perfect + wrong-R fails closed (0 plaintext). Probe 2:
controller→agent POST-body transport already proven (withGuest) — stage
endpoint is additive. Probe 4: WriteOffboxSecrets honors a pre-placed password
(DR re-injection), but config endpoint carries no password → explicit inject
step needed. All probes throwaway/discarded; live password + repo untouched.
Impl blueprint for the fork-4 spec inside.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Real endpoints, fresh box sub-account. Discovery proof: immich (undeployed,
USB-only unit — the v0.103.0 mis-resolution case) → ok/1 snapshot on the USB
path; 45MB db-dump restore sha256-matched. No-silent-success on the live box:
0/N → error + backup_failed alert; partial → ok + LastWarning. Persists across
restart. Forks 1+2 confirmed (direct; sub-account /home/<name>, ssh-copy-id -s
-f, console key field still broken F3); fork-4 custody surface recorded, escrow
design OPEN (repo pw + key not independently recoverable). Kept as standing
demo-offsite.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Validated the controller offbox restic-SFTP tier direct from the residential
demo box (IPv4-only egress) end-to-end through the real endpoints: reach on
:23 (PASS), chown-immune restic init (works where PBS-direct failed),
config→run→restore green, sub-account isolation confirmed. Residential upload
~4.3 MB/s. Concentrator NOT needed. KEY BLOCKER F1: offbox reads recovery unit
from the app data-drive namespace root, not the actual backup-drive location →
silently backs up 0 apps (ok status) on multi-drive setups. Spike only,
synthetic data, torn down.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Wired the 1TB BX11 Storage Box to ep0 as a durable key-auth SSHFS mount
(/mnt/pbs-storagebox, reboot-proven). Findings: direct PBS datastore UNFIT
(box rejects chown -> ChunkStore::create fails); loopback-ext4-on-SSHFS FIT
with caveats (GC 7s/verify 12s/restore 11s all green, no GC hang). Raw mount
~327MB/s write, ~69MB/s read. Spike only, synthetic data, torn down.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Correction: campaign-2's "R1" (bind-mount restore needs root) was a SYMPTOM of an
empty felhom pool -> agent lacked VM.Audit on 9201 -> restore-test's existing,
correct bindMountOverrides never ran. Live-proven: pool healed (Part A) ->
restore-test PASSES (neutralizes 2 binds, boot+running, clean teardown, 4m35s).
Report rec #1 (make pct restore work on bind mounts) superseded. Fixes shipped:
agent v0.74.0 (pool re-assertion), controller v0.103.0 (F-C2-1).
Spike (SPIKE-dr-bindmount-source-2026-07-07): DR bring-up source options — PBS
pct.conf.blob blocked by encryption under the non-root token; DR recipe lacks mpN;
mp8/mp9 are STRUCTURAL CONSTANTS -> recommend a small known-constant override
reusing bindMountOverrides.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Pilot-blockers bundle shipped + live-validated: agent v0.73.0 (F2 mount-role
fallback), controller v0.102.0 (async restore family, F4 re-adjudicated as a UX
bug not a restore bug), catalog healthcheck sweep. All three accepted live on the
demo node.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Correct the end-state: uptime-kuma redeployed healthy, notifications restored;
vaultwarden serves (127.0.0.1 /alive 200) but reads docker-unhealthy due to a
localhost->IPv6 healthcheck probe quirk (reproduced on fresh redeploy);
bookstack unhealthy at app level (HTTP 500, pre-existing). Recommend catalog
healthchecks use 127.0.0.1 not localhost. Neither is a P1-P3 defect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Pilot-gap CLOSED: F3 volume-tar backup (incl. volume-only apps, bit-identical),
vaultwarden restore drill (bit-identical), Tier-2 additive-only restore, off-box
SFTP round-trip — all proven on v0.101.0 + F1-fixed template with real data on a
healthy app set. Full F1-fix acceptance. F4 resolved as NON-bug (restore is a
long synchronous op; traefik proxy read-timeout returns 000/502 while the
controller completes it). ZERO product bugs in P1-P3; ~18 throwaway-harness bugs
fixed (path/sha/DB-client/proxy-timeout/test-ordering). No production code changed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
F1: original "shim sets SMTP_FROM w/o SMTP_HOST" hypothesis was WRONG (shim
injects nothing when off) — real cause is defined-empty SMTP vars + upstream
_enable_smtp default true → crash; fixed via _ENABLE_SMTP gate (catalog),
live-validated. F2 verdict: REAL finding — roleForMountPath resolves an enrolled
user-data drive that isn't a PVE storage to system (fail-safe over-refusal)
while /disks calls it user-data; agent-side fix deferred. F3: fixed in
controller v0.101.0 (git subprocess deadline).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Validates a second sshd (own port/config/host-key/AuthorizedKeysFile/unit) as
Felhom's OOB entry point, coexisting with the customer's stock sshd on :22:
robust port probe-and-claim (skip/idempotent/loud-exhaustion), four-axis
coexistence independence, tunnel-scoped nft, reload-gate robustness, operator
identity isolated via AuthorizedKeysFile location, clean uninstall.
#1 TASK constraint (caught live): a second sshd unit must NEVER declare
RuntimeDirectory=sshd — it removes the SHARED /run/sshd privsep dir on stop and
takes the stock sshd down (LAN SSH lockout; recovered via PVE console + mkdir
/run/sshd). Use a tmpfiles.d entry instead. All spike artifacts removed; baseline
re-verified.
Docs-only; no code/hub/agent/manifest change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Validates operator-inbound access over the existing offsite WG arc (doc 06):
operator peer forwarded operator->box only, box sshd gated to the operator /32,
§4.5 box<->box isolation intact (both negatives counter-proven), mutual repair
real (agent self-healed a stopped tunnel in ~15s unaided). One TASK-shaping gap:
the operator /32 must be a RENDERED conf field — a runtime `wg set` is wiped by
the agent's own self-heal. All live-arc changes reverted + baseline re-verified.
Docs-only; no code/hub/agent/manifest change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
GO with two design-reshaping surprises: OnFailure= fires on EVERY crash on
systemd 257 (rollback triggers at first post-apply crash; marker-guard +
idempotency mandatory), and deliberate restarts consume start-limit budget.
S1 trap confirmed (default shape never reaches failed — 35 starts/180s);
tuned [Unit] StartLimitIntervalSec=120 + Burst=4 → failed in ~20s;
[Service] placement is an asymmetric half-applied trap; reset-failed required.
S2: all restart variants reliable, detached systemd-run wins. S3: A/B flip
atomic under kill -9, all verbs + guards pass. S4: StateDir + sudoers shapes OK.
Dummy artifacts only; real agent uptime unchanged (verified).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Keep-side restore validated live: class-A DB replay, fail-closed data-key
gate + proven non-destruction, resettable-secret discrimination, Tier-2
class-C copy, and guardrails (traversal/missing/single-flight/restart).
Findings: F1 (HIGH) UI restore dead — /api/backup/snapshots has no handler;
F2 no one-click in-place class-C restore; F3 named-volume data never backed
up (DumpAppVolumes* has no caller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
P0-P7 probe ladder on real ends (demo-felhom PVE host <-> throwaway Hetzner).
Verdict: GO at H1 — plain host-side WG, box-dials-out to a small public endpoint;
Headscale is a separable fleet layer, not a traversal necessity. Line is plain-NAT
(not CGNAT) with zero IPv6 — recorded honestly; CGNAT vantage deferred to Peti VM 110.
Real 2 GiB worst-case PBS backup over the tunnel at the full home uplink, pin intact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Live-probed on felhom-pve under the PRODUCTION scoped token vs root:
enumeration IS pool-filtered (T1: token sees [9201] of 4 guests); pool read
403s naming Pool.Audit (T2); Pool.Audit@/pool/felhom alone suffices (T3,
throwaway identity, torn down); /cluster/resources withholds the pool field
without Pool.Audit (T7); local ownership records all partial (T5).
Recommendation for the A1 impl spec (now unblocked): Pool.Audit added to
FelhomAgentGuest + GET /pools/felhom cross-check in staleLockController,
fail-safe skip on read failure. Appendix: committed-secrets rotation
micro-runbook (operator follow-up).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Companion to CAMPAIGN-pool-effects: red-teams the four axes the pool campaign
did not cover (provenance-gating of destructive ops, sudoers↔fine-gate coverage,
local-API within-scope blast radius, mkfs/claim classifier, signed-vs-self-serve
boundary). Verifies the v0.60.0 F1 fix is complete; finds no new HIGH/MED. One
F1-sibling (stale-lock pool-blind reaper, LOW) + LOW/INFO defense-in-depth
residuals. Read-only static trace against felhom-agent @cc93dae/v0.60.0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Companion to felhom-agent v0.60.0 (proof-of-launch destroy gating + restore-
test band-advance). New documentation/runbooks/provisioning.md; campaign doc
updated with the resolution status and the operator manifest-bump follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full validation of the felhom pool + scoped-token migration on felhom-pve:
- Track A: complete agent op surface under the scoped token — 16/16 pass
- Track B: vmid origin traced (externally supplied); adversarial collision
probes prove the compensating rollback/teardown attempts to destroy a
guest the job never created (contained only by the pool ACL)
- Track C: containment + 3b-fix intact; Track D: no pool leaks, adopt/
uninstall pool handling correct
- Findings doc: documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md
No code change, no version bump. Demo left as found.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Read-only spike on felhom-pve. Verdicts: (SQ1) customer drives need NOT be PVE
storages — track in the intent registry (durable-id + host mount + bind), the
dir-storage is legacy. (SQ2) an unclaimed-disk filter (OS/LVM-PV/ZFS/PVE-storage/
VM-CT/mounted/RO/registry signals, fail-safe exclude-on-unknown) returns EXACTLY
{/dev/sdd} on the demo, never sda/sdb/sdc. (SQ3) mkfs sudoers is /dev/* — the
agent code is the only guard; DataBearing (wipe-confirm) is insufficient (OS disk
is data-bearing), so a mandatory agent unclaimed-gate + guarded-mkfs-wrapper
sudoers + a shared-box operator gate for self-serve format. (SQ4) new
/disks/candidates raw-scan endpoint feeding both wizards, registry-based
enrollment. (SQ5) migrate existing drives to the registry. Implementation plan +
open risks included. No product code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live supervised validation on felhom-pve: NO 403 on drive management under the
scoped token (agent has no storage-registration/pvesm path — all host-ops +
Datastore.Audit which is now box-wide). 3b-fix confirmed in the real UI (drives
Regisztrálva, no detach alert). A full physical wizard enroll couldn't complete
because a raw new device (SD-card reader, no PVE-storage entry) doesn't surface
in the init wizard (Observe() lists PVE storages only) — ACL-independent; flagged
as a separate new-drive-registration follow-up. No code change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Throwaway-token spike on felhom-pve: the agent's 16 privs scope from / to
/pool/felhom + /storage/<targets> (+Pool.Allocate) without breaking the flow.
CRUX proven: restore a fresh vmid INTO the pool works at /pool/felhom (T1 200).
T1-T7 all pass; blast radius contained (out-of-pool guests 403). Findings:
3-role layout (per-path subsets need separate roles); Datastore.Audit filters
so needn't be box-wide; Store scope MUST include felhom-pbs (offsite default);
restore-test scratch band (9990xx) not in pool → 3b must handle. All spike
objects cleaned up; live FelhomAgent/9201 untouched. No product code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Categorized STALE/DEAD/MISSING/POLISH/CORRECTNESS findings for the hub web
GUI. No GUI code changed, no version bump — findings doc only, for operator
prioritization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
Part A: hub-config operator_email/operator_enabled (committed earlier);
operator + customer email paths proven end-to-end via real Resend sends.
Append the "Findings closed" section to the TESTRUN audit (Part A proof,
F2-a root cause + deploy fix, F2-b fix + live validation, pending §D
supervised reboot) and remove the stale duplicate "Final state" block.
REPORT overwritten. hub/CHANGELOG entry. No hub image change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
Phase 1: capability->hub operator alert fired live (event captured, English, names cap+feature);
email is operator-config-gated in demo (not a defect). Phase 2: drive re-propagation + zero false
pages PASS, but TWO findings — F2-a the §4 doubling did NOT self-correct (boot script runs as root,
never needed the grant → hypothesis falsified), F2-b a vzdump backup in flight at reboot left a stale
snapshot-delete lock blocking guest autostart. Phase 3 referenced (host_leaf_changed proven on v0.22.1).
Final parity known-good.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
Q1: 6 modes classified — pin-mismatch/401/refused/timeout/no-such-host are RUNTIME (per-call),
malformed-fp is a CONSTRUCTION error (latches via sync.Once). Q2: across a clean agent restart the
memoized /api/disks path self-heals in ~1s with NO controller restart (Up 7h); a fresh-client probe
agrees at every phase. Recommend: probe GET /storage via the memoized client, classify, N>=2 debounce
for transient reasons, hub relay. Throwaway probe removed, no config changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
admin