The KEXINIT-reset diagnostic, the three-layer model, PVE-web-console recovery with
the vaulted root@pam credential, the /run/sshd one-line fix, finding a recurring
clobber, and the keep-the-180-token-fresh standing note.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Validates a second sshd (own port/config/host-key/AuthorizedKeysFile/unit) as
Felhom's OOB entry point, coexisting with the customer's stock sshd on :22:
robust port probe-and-claim (skip/idempotent/loud-exhaustion), four-axis
coexistence independence, tunnel-scoped nft, reload-gate robustness, operator
identity isolated via AuthorizedKeysFile location, clean uninstall.
#1 TASK constraint (caught live): a second sshd unit must NEVER declare
RuntimeDirectory=sshd — it removes the SHARED /run/sshd privsep dir on stop and
takes the stock sshd down (LAN SSH lockout; recovered via PVE console + mkdir
/run/sshd). Use a tmpfiles.d entry instead. All spike artifacts removed; baseline
re-verified.
Docs-only; no code/hub/agent/manifest change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Validates operator-inbound access over the existing offsite WG arc (doc 06):
operator peer forwarded operator->box only, box sshd gated to the operator /32,
§4.5 box<->box isolation intact (both negatives counter-proven), mutual repair
real (agent self-healed a stopped tunnel in ~15s unaided). One TASK-shaping gap:
the operator /32 must be a RENDERED conf field — a runtime `wg set` is wiped by
the agent's own self-heal. All live-arc changes reverted + baseline re-verified.
Docs-only; no code/hub/agent/manifest change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
felhom-host-install.sh installs felhom-selfupdate-guarded (sh -n), the rollback
unit + the [Unit] start-limit drop-in (daemon-reload) so day-0 boxes get
operator-signed self-update from birth; non-fatal on pre-D1 agent repos.
03-host-agent.md §11 updated to the shipped mechanism (signed op, A/B wrapper,
OnFailure rollback, tuned start-limit backstop).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
GO with two design-reshaping surprises: OnFailure= fires on EVERY crash on
systemd 257 (rollback triggers at first post-apply crash; marker-guard +
idempotency mandatory), and deliberate restarts consume start-limit budget.
S1 trap confirmed (default shape never reaches failed — 35 starts/180s);
tuned [Unit] StartLimitIntervalSec=120 + Burst=4 → failed in ~20s;
[Service] placement is an asymmetric half-applied trap; reset-failed required.
S2: all restart variants reliable, detached systemd-run wins. S3: A/B flip
atomic under kill -9, all verbs + guards pass. S4: StateDir + sudoers shapes OK.
Dummy artifacts only; real agent uptime unchanged (verified).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Keep-side restore validated live: class-A DB replay, fail-closed data-key
gate + proven non-destruction, resettable-secret discrimination, Tier-2
class-C copy, and guardrails (traversal/missing/single-flight/restart).
Findings: F1 (HIGH) UI restore dead — /api/backup/snapshots has no handler;
F2 no one-click in-place class-C restore; F3 named-volume data never backed
up (DumpAppVolumes* has no caller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
escrow-create -storage felhom-offsite -identity-bundle -directive -upload on
felhom-pve: fresh R (operator-recorded off-transcript) wraps K + WG key + offsite
PBS token; uploaded to hub. Independently verified in host_escrow (demo-felhom-01:
key_fingerprint=offsite b0:fe:2a…, identity blob 499B, directive present,
created_at=now) with zero knowledge of R. S5 (DR consume) now UNBLOCKED.
This re-run also remediated an earlier escrow whose R had leaked into an agent
transcript (last-write-wins overwrote it with the clean-R blob).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Offsite restore-test now runs unattended on the agent-token path (agent v0.68.0):
tier-aware restore-task timeout (pbs→120m, local→10m). The scratch-band
VM.Allocate follow-up was a phantom — the scratch restores INTO /pool/felhom
(ACL already grants it); the earlier 403 was the 10m-timeout consequence. No
ACL/host-install change. doc-06 S4 row + runbook §4b updated.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The CGNAT smoke test's OPEN DECISION is made: fleet-wide, permanent,
family-agnostic client MTU 1280 (RFC 8200 IPv6-minimum floor), implemented in
felhom-agent v0.65.0.
- §4.3: OPEN DECISION → made decision + rationale + rejected alternatives
(auto-probe/per-type = fragility for a non-metric; MSS-clamp = unneeded).
"ships MTU 1420" → 1280; corrected "fixed-line unaffected" (DS-Lite ~1452,
PPPoE would also black-hole at 1420 — one floor covers all).
- §4.2: 1280 is family-agnostic so MTU correctness is decoupled from v4/v6
resolution; v4-pin noted as a separate optional determinism item.
- §3.2 wire-field note + §8 S6 row: MTU 1420 → 1280; S6 residual now only the
true-CGNAT-SIM retest.
- REPORT.md overwritten.
Docs-only; the constant change is in felhom-agent v0.65.0.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
S3 appendix CGNAT smoke test run live on felhom-pve (agent v0.64.0) with a
USB-tethered phone. Tunnel established + held its NAT mapping across a 32-min
idle soak on PersistentKeepalive=25 alone (0 stalls, final ping no-bounce).
- §7: open-validation CLOSED; two honest caveats kept (SIM had a public mobile
IPv4 so true CGNAT 100.64/10 not reproduced — retest-when-able; + MTU).
- §4.3: mobile outer path MTU ~1400 black-holed bulk data at the shipped 1420;
MTU 1340 restored the PBS page. OPEN DECISION: permanent vs per-connection.
- §4.4: keepalive 32-min mobile-NAT hold datapoint.
- §4.2: dual-stack v6 trap (wg-quick prefers endpoint AAAA, rides un-NATed v6).
- §8: S6 row CGNAT closure marked done.
- REPORT.md: overwritten with this operational-validation summary.
Runbook: RUNBOOK-s3-cgnat-smoke. Docs-only; no agent/hub/controller code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
All 8 steps run on the dev endpoint; live-run corrections folded in: enterprise-
repo removal after PBS install, proxmox-backup-client is a separate package,
throwaway-token namespace proof, wg-show-dump-leaks-the-private-key warning
(incident: first server key leaked to session log -> rotated on the spot),
AAAA-must-be-::1 + DNS-propagation-lag notes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Records the settled transport decisions (plain WG, host-side agent-managed pilot,
one hub-driven endpoint VM running WG+PBS, hub source-of-truth over WireDesiredState,
one datastore + per-customer namespaces, relay-through-DooPlex rejected). Grounded
at file:line vs felhom.eu@bf099f6 + felhom-agent@4ba1b14. Backlog line resolved to
cite spike + design doc. Docs-only; slices are separate future tasks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
P0-P7 probe ladder on real ends (demo-felhom PVE host <-> throwaway Hetzner).
Verdict: GO at H1 — plain host-side WG, box-dials-out to a small public endpoint;
Headscale is a separable fleet layer, not a traversal necessity. Line is plain-NAT
(not CGNAT) with zero IPv6 — recorded honestly; CGNAT vantage deferred to Peti VM 110.
Real 2 GiB worst-case PBS backup over the tunnel at the full home uplink, pin intact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Live-probed on felhom-pve under the PRODUCTION scoped token vs root:
enumeration IS pool-filtered (T1: token sees [9201] of 4 guests); pool read
403s naming Pool.Audit (T2); Pool.Audit@/pool/felhom alone suffices (T3,
throwaway identity, torn down); /cluster/resources withholds the pool field
without Pool.Audit (T7); local ownership records all partial (T5).
Recommendation for the A1 impl spec (now unblocked): Pool.Audit added to
FelhomAgentGuest + GET /pools/felhom cross-check in staleLockController,
fail-safe skip on read failure. Appendix: committed-secrets rotation
micro-runbook (operator follow-up).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Tokens, component semantics (meter/tag/metarow/panel/list/section-h/
stats/buttons), the state truth tables incl. the stopped->neutral
exception-color change, vendored typography (latin-ext) + Lucide
iconography, chart palette, and the grep-gate enforcement list.
Shipped in controller v0.96.0; D1-D4 build from this document.
Companion to CAMPAIGN-pool-effects: red-teams the four axes the pool campaign
did not cover (provenance-gating of destructive ops, sudoers↔fine-gate coverage,
local-API within-scope blast radius, mkfs/claim classifier, signed-vs-self-serve
boundary). Verifies the v0.60.0 F1 fix is complete; finds no new HIGH/MED. One
F1-sibling (stale-lock pool-blind reaper, LOW) + LOW/INFO defense-in-depth
residuals. Read-only static trace against felhom-agent @cc93dae/v0.60.0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Companion to felhom-agent v0.60.0 (proof-of-launch destroy gating + restore-
test band-advance). New documentation/runbooks/provisioning.md; campaign doc
updated with the resolution status and the operator manifest-bump follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full validation of the felhom pool + scoped-token migration on felhom-pve:
- Track A: complete agent op surface under the scoped token — 16/16 pass
- Track B: vmid origin traced (externally supplied); adversarial collision
probes prove the compensating rollback/teardown attempts to destroy a
guest the job never created (contained only by the pool ACL)
- Track C: containment + 3b-fix intact; Track D: no pool leaks, adopt/
uninstall pool handling correct
- Findings doc: documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md
No code change, no version bump. Demo left as found.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Read-only spike on felhom-pve. Verdicts: (SQ1) customer drives need NOT be PVE
storages — track in the intent registry (durable-id + host mount + bind), the
dir-storage is legacy. (SQ2) an unclaimed-disk filter (OS/LVM-PV/ZFS/PVE-storage/
VM-CT/mounted/RO/registry signals, fail-safe exclude-on-unknown) returns EXACTLY
{/dev/sdd} on the demo, never sda/sdb/sdc. (SQ3) mkfs sudoers is /dev/* — the
agent code is the only guard; DataBearing (wipe-confirm) is insufficient (OS disk
is data-bearing), so a mandatory agent unclaimed-gate + guarded-mkfs-wrapper
sudoers + a shared-box operator gate for self-serve format. (SQ4) new
/disks/candidates raw-scan endpoint feeding both wizards, registry-based
enrollment. (SQ5) migrate existing drives to the registry. Implementation plan +
open risks included. No product code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live supervised validation on felhom-pve: NO 403 on drive management under the
scoped token (agent has no storage-registration/pvesm path — all host-ops +
Datastore.Audit which is now box-wide). 3b-fix confirmed in the real UI (drives
Regisztrálva, no detach alert). A full physical wizard enroll couldn't complete
because a raw new device (SD-card reader, no PVE-storage entry) doesn't surface
in the init wizard (Observe() lists PVE storages only) — ACL-independent; flagged
as a separate new-drive-registration follow-up. No code change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Throwaway-token spike on felhom-pve: the agent's 16 privs scope from / to
/pool/felhom + /storage/<targets> (+Pool.Allocate) without breaking the flow.
CRUX proven: restore a fresh vmid INTO the pool works at /pool/felhom (T1 200).
T1-T7 all pass; blast radius contained (out-of-pool guests 403). Findings:
3-role layout (per-path subsets need separate roles); Datastore.Audit filters
so needn't be box-wide; Store scope MUST include felhom-pbs (offsite default);
restore-test scratch band (9990xx) not in pool → 3b must handle. All spike
objects cleaned up; live FelhomAgent/9201 untouched. No product code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Categorized STALE/DEAD/MISSING/POLISH/CORRECTNESS findings for the hub web
GUI. No GUI code changed, no version bump — findings doc only, for operator
prioritization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
Part A: hub-config operator_email/operator_enabled (committed earlier);
operator + customer email paths proven end-to-end via real Resend sends.
Append the "Findings closed" section to the TESTRUN audit (Part A proof,
F2-a root cause + deploy fix, F2-b fix + live validation, pending §D
supervised reboot) and remove the stale duplicate "Final state" block.
REPORT overwritten. hub/CHANGELOG entry. No hub image change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
Phase 1: capability->hub operator alert fired live (event captured, English, names cap+feature);
email is operator-config-gated in demo (not a defect). Phase 2: drive re-propagation + zero false
pages PASS, but TWO findings — F2-a the §4 doubling did NOT self-correct (boot script runs as root,
never needed the grant → hypothesis falsified), F2-b a vzdump backup in flight at reboot left a stale
snapshot-delete lock blocking guest autostart. Phase 3 referenced (host_leaf_changed proven on v0.22.1).
Final parity known-good.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
Q1: 6 modes classified — pin-mismatch/401/refused/timeout/no-such-host are RUNTIME (per-call),
malformed-fp is a CONSTRUCTION error (latches via sync.Once). Q2: across a clean agent restart the
memoized /api/disks path self-heals in ~1s with NO controller restart (Up 7h); a fresh-client probe
agrees at every phase. Recommend: probe GET /storage via the memoized client, classify, N>=2 debounce
for transient reasons, hub relay. Throwaway probe removed, no config changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
admin