Operator ruling at the GL-6 drill's Gate 0: the Felhom artifacts are
world-readable by design; an EMPTY git.username/git.token in the customer
config now WARNS and fetches anonymously instead of dying at step 5/8
(the hub-vouched sha256 stays the integrity root; a configured credential
is still used when present; curl auth args are conditional because -u with
an empty token 401s even on public content). All 12 installer fetch
targets validated 200 anonymously before shipping. Harness: +GL6-ANON
shape case; GL4-C2 assertion updated for the v1.11.1 pinned constants
(the benign overrides-notice is not a die). Drill record carries the
deviation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Agent v0.76.0 live + published (9828c5f7..f50b, the exact felhom-pve bytes,
anon-fetch re-verified). Manifest bump goes straight to 0.76.0; 0.75.0
superseded unpublished; 0.74.0 must not be vouched (broken guest-loss DR).
Measured full-fidelity runtime: 3m4s local tier.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Uninstall: NEW 4b4 removes the self-update artifacts (guarded wrapper,
.prev/.new.* A/B slots, rollback unit, limits drop-in — derived from
configs/felhom-selfupdate-guarded); enrolled drives under /mnt/felhom-drives
unmounted plain-umount-only (busy = warn+guidance, never -l/-f; root bind
guarded); both modes end with a KEPT-vs-WIPED statement (drives/PBS/hub
record/escrow/vaulted recovery credential live on); guest-only mode prints
the vmid's bind-store drives.
Key-pin: OPERATOR_KEY_* constants (empty until the operator ceremony) +
--operator-pubkey-file (argv-validated; comment=key_id required; file
overrides constants) -> authz.signers per the agent SignerKey schema;
no-keys-resolved KEEPS preserved signers; verify reports armed/dormant WARN.
Harness: +13 static cases + GL4-D parity + GL4-INV + PVE-tier GL4 H-U
(full-uninstall dry transcript vs live 9201). 28/28 on felhom-pve;
red-proofs RP-1..3 run->fail->revert. GO-LIVE-PACKAGE.md ABSENT AGAIN ->
status in CONTEXT.md. Live teardown/armed-pin = GL-6.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Agent = the live felhom-pve bytes (sha 1ec3f588..76af05, round-trip verified);
golden baked clean-room in the drill VM (build-golden v2.0.0, controller
0.103.0, all integrity gates green, sha 8481e8a1..6026e) and published 201.
Both anon-fetch-verified (Gate 3c: packages world-readable -> G3 read-only
token rotation cannot break fetches; installer-side empty-token die is the
real precondition). Drill VM restored to virgin. OPERATOR NEXT: Day-0
manifest bump to the four recorded values.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Correction: campaign-2's "R1" (bind-mount restore needs root) was a SYMPTOM of an
empty felhom pool -> agent lacked VM.Audit on 9201 -> restore-test's existing,
correct bindMountOverrides never ran. Live-proven: pool healed (Part A) ->
restore-test PASSES (neutralizes 2 binds, boot+running, clean teardown, 4m35s).
Report rec #1 (make pct restore work on bind mounts) superseded. Fixes shipped:
agent v0.74.0 (pool re-assertion), controller v0.103.0 (F-C2-1).
Spike (SPIKE-dr-bindmount-source-2026-07-07): DR bring-up source options — PBS
pct.conf.blob blocked by encryption under the non-root token; DR recipe lacks mpN;
mp8/mp9 are STRUCTURAL CONSTANTS -> recommend a small known-constant override
reusing bindMountOverrides.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Full-system validation on demo 9201 (controller v0.102.0 / agent v0.73.0). Headline:
ONE genuine pilot-critical finding — the non-root agent cannot pct-restore the
standard Felhom guest because it has bind-mount mountpoints (mp8 /mnt/felhom-drives,
mp9 bootstrap) and pct restore refuses bind mounts unless root. This breaks BOTH
--selftest=restore-test AND the offsite DR bring-up (R1), linked to a missing
VM.Audit/VM.Allocate on the agent token (R2). The large raw FAIL count is dominated
by one harness bug (auth-session expiry mis-scoring P3-P9) plus D1-injection/soak-probe/
dryrun-journal/escrow-blob calibration issues; the product where actually exercised was
healthy with ZERO resource leaks over ~7h. Demo node healed to pre-campaign state
(auth off, notifications restored, no scratch leaks). No production code changed.
Evidence snapshot + README under campaign2-evidence-2026-07-07/ (cookie jars scrubbed).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Pilot-blockers bundle shipped + live-validated: agent v0.73.0 (F2 mount-role
fallback), controller v0.102.0 (async restore family, F4 re-adjudicated as a UX
bug not a restore bug), catalog healthcheck sweep. All three accepted live on the
demo node.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Correct the end-state: uptime-kuma redeployed healthy, notifications restored;
vaultwarden serves (127.0.0.1 /alive 200) but reads docker-unhealthy due to a
localhost->IPv6 healthcheck probe quirk (reproduced on fresh redeploy);
bookstack unhealthy at app level (HTTP 500, pre-existing). Recommend catalog
healthchecks use 127.0.0.1 not localhost. Neither is a P1-P3 defect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Pilot-gap CLOSED: F3 volume-tar backup (incl. volume-only apps, bit-identical),
vaultwarden restore drill (bit-identical), Tier-2 additive-only restore, off-box
SFTP round-trip — all proven on v0.101.0 + F1-fixed template with real data on a
healthy app set. Full F1-fix acceptance. F4 resolved as NON-bug (restore is a
long synchronous op; traefik proxy read-timeout returns 000/502 while the
controller completes it). ZERO product bugs in P1-P3; ~18 throwaway-harness bugs
fixed (path/sha/DB-client/proxy-timeout/test-ordering). No production code changed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
F1: original "shim sets SMTP_FROM w/o SMTP_HOST" hypothesis was WRONG (shim
injects nothing when off) — real cause is defined-empty SMTP vars + upstream
_enable_smtp default true → crash; fixed via _ENABLE_SMTP gate (catalog),
live-validated. F2 verdict: REAL finding — roleForMountPath resolves an enrolled
user-data drive that isn't a PVE storage to system (fail-safe over-refusal)
while /disks calls it user-data; agent-side fix deferred. F3: fixed in
controller v0.101.0 (git subprocess deadline).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The KEXINIT-reset diagnostic, the three-layer model, PVE-web-console recovery with
the vaulted root@pam credential, the /run/sshd one-line fix, finding a recurring
clobber, and the keep-the-180-token-fresh standing note.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Validates a second sshd (own port/config/host-key/AuthorizedKeysFile/unit) as
Felhom's OOB entry point, coexisting with the customer's stock sshd on :22:
robust port probe-and-claim (skip/idempotent/loud-exhaustion), four-axis
coexistence independence, tunnel-scoped nft, reload-gate robustness, operator
identity isolated via AuthorizedKeysFile location, clean uninstall.
#1 TASK constraint (caught live): a second sshd unit must NEVER declare
RuntimeDirectory=sshd — it removes the SHARED /run/sshd privsep dir on stop and
takes the stock sshd down (LAN SSH lockout; recovered via PVE console + mkdir
/run/sshd). Use a tmpfiles.d entry instead. All spike artifacts removed; baseline
re-verified.
Docs-only; no code/hub/agent/manifest change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Validates operator-inbound access over the existing offsite WG arc (doc 06):
operator peer forwarded operator->box only, box sshd gated to the operator /32,
§4.5 box<->box isolation intact (both negatives counter-proven), mutual repair
real (agent self-healed a stopped tunnel in ~15s unaided). One TASK-shaping gap:
the operator /32 must be a RENDERED conf field — a runtime `wg set` is wiped by
the agent's own self-heal. All live-arc changes reverted + baseline re-verified.
Docs-only; no code/hub/agent/manifest change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
felhom-host-install.sh installs felhom-selfupdate-guarded (sh -n), the rollback
unit + the [Unit] start-limit drop-in (daemon-reload) so day-0 boxes get
operator-signed self-update from birth; non-fatal on pre-D1 agent repos.
03-host-agent.md §11 updated to the shipped mechanism (signed op, A/B wrapper,
OnFailure rollback, tuned start-limit backstop).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
GO with two design-reshaping surprises: OnFailure= fires on EVERY crash on
systemd 257 (rollback triggers at first post-apply crash; marker-guard +
idempotency mandatory), and deliberate restarts consume start-limit budget.
S1 trap confirmed (default shape never reaches failed — 35 starts/180s);
tuned [Unit] StartLimitIntervalSec=120 + Burst=4 → failed in ~20s;
[Service] placement is an asymmetric half-applied trap; reset-failed required.
S2: all restart variants reliable, detached systemd-run wins. S3: A/B flip
atomic under kill -9, all verbs + guards pass. S4: StateDir + sudoers shapes OK.
Dummy artifacts only; real agent uptime unchanged (verified).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Keep-side restore validated live: class-A DB replay, fail-closed data-key
gate + proven non-destruction, resettable-secret discrimination, Tier-2
class-C copy, and guardrails (traversal/missing/single-flight/restart).
Findings: F1 (HIGH) UI restore dead — /api/backup/snapshots has no handler;
F2 no one-click in-place class-C restore; F3 named-volume data never backed
up (DumpAppVolumes* has no caller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
escrow-create -storage felhom-offsite -identity-bundle -directive -upload on
felhom-pve: fresh R (operator-recorded off-transcript) wraps K + WG key + offsite
PBS token; uploaded to hub. Independently verified in host_escrow (demo-felhom-01:
key_fingerprint=offsite b0:fe:2a…, identity blob 499B, directive present,
created_at=now) with zero knowledge of R. S5 (DR consume) now UNBLOCKED.
This re-run also remediated an earlier escrow whose R had leaked into an agent
transcript (last-write-wins overwrote it with the clean-R blob).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Offsite restore-test now runs unattended on the agent-token path (agent v0.68.0):
tier-aware restore-task timeout (pbs→120m, local→10m). The scratch-band
VM.Allocate follow-up was a phantom — the scratch restores INTO /pool/felhom
(ACL already grants it); the earlier 403 was the 10m-timeout consequence. No
ACL/host-install change. doc-06 S4 row + runbook §4b updated.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The CGNAT smoke test's OPEN DECISION is made: fleet-wide, permanent,
family-agnostic client MTU 1280 (RFC 8200 IPv6-minimum floor), implemented in
felhom-agent v0.65.0.
- §4.3: OPEN DECISION → made decision + rationale + rejected alternatives
(auto-probe/per-type = fragility for a non-metric; MSS-clamp = unneeded).
"ships MTU 1420" → 1280; corrected "fixed-line unaffected" (DS-Lite ~1452,
PPPoE would also black-hole at 1420 — one floor covers all).
- §4.2: 1280 is family-agnostic so MTU correctness is decoupled from v4/v6
resolution; v4-pin noted as a separate optional determinism item.
- §3.2 wire-field note + §8 S6 row: MTU 1420 → 1280; S6 residual now only the
true-CGNAT-SIM retest.
- REPORT.md overwritten.
Docs-only; the constant change is in felhom-agent v0.65.0.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
S3 appendix CGNAT smoke test run live on felhom-pve (agent v0.64.0) with a
USB-tethered phone. Tunnel established + held its NAT mapping across a 32-min
idle soak on PersistentKeepalive=25 alone (0 stalls, final ping no-bounce).
- §7: open-validation CLOSED; two honest caveats kept (SIM had a public mobile
IPv4 so true CGNAT 100.64/10 not reproduced — retest-when-able; + MTU).
- §4.3: mobile outer path MTU ~1400 black-holed bulk data at the shipped 1420;
MTU 1340 restored the PBS page. OPEN DECISION: permanent vs per-connection.
- §4.4: keepalive 32-min mobile-NAT hold datapoint.
- §4.2: dual-stack v6 trap (wg-quick prefers endpoint AAAA, rides un-NATed v6).
- §8: S6 row CGNAT closure marked done.
- REPORT.md: overwritten with this operational-validation summary.
Runbook: RUNBOOK-s3-cgnat-smoke. Docs-only; no agent/hub/controller code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
All 8 steps run on the dev endpoint; live-run corrections folded in: enterprise-
repo removal after PBS install, proxmox-backup-client is a separate package,
throwaway-token namespace proof, wg-show-dump-leaks-the-private-key warning
(incident: first server key leaked to session log -> rotated on the spot),
AAAA-must-be-::1 + DNS-propagation-lag notes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Records the settled transport decisions (plain WG, host-side agent-managed pilot,
one hub-driven endpoint VM running WG+PBS, hub source-of-truth over WireDesiredState,
one datastore + per-customer namespaces, relay-through-DooPlex rejected). Grounded
at file:line vs felhom.eu@bf099f6 + felhom-agent@4ba1b14. Backlog line resolved to
cite spike + design doc. Docs-only; slices are separate future tasks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
P0-P7 probe ladder on real ends (demo-felhom PVE host <-> throwaway Hetzner).
Verdict: GO at H1 — plain host-side WG, box-dials-out to a small public endpoint;
Headscale is a separable fleet layer, not a traversal necessity. Line is plain-NAT
(not CGNAT) with zero IPv6 — recorded honestly; CGNAT vantage deferred to Peti VM 110.
Real 2 GiB worst-case PBS backup over the tunnel at the full home uplink, pin intact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Live-probed on felhom-pve under the PRODUCTION scoped token vs root:
enumeration IS pool-filtered (T1: token sees [9201] of 4 guests); pool read
403s naming Pool.Audit (T2); Pool.Audit@/pool/felhom alone suffices (T3,
throwaway identity, torn down); /cluster/resources withholds the pool field
without Pool.Audit (T7); local ownership records all partial (T5).
Recommendation for the A1 impl spec (now unblocked): Pool.Audit added to
FelhomAgentGuest + GET /pools/felhom cross-check in staleLockController,
fail-safe skip on read failure. Appendix: committed-secrets rotation
micro-runbook (operator follow-up).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Tokens, component semantics (meter/tag/metarow/panel/list/section-h/
stats/buttons), the state truth tables incl. the stopped->neutral
exception-color change, vendored typography (latin-ext) + Lucide
iconography, chart palette, and the grep-gate enforcement list.
Shipped in controller v0.96.0; D1-D4 build from this document.
Companion to CAMPAIGN-pool-effects: red-teams the four axes the pool campaign
did not cover (provenance-gating of destructive ops, sudoers↔fine-gate coverage,
local-API within-scope blast radius, mkfs/claim classifier, signed-vs-self-serve
boundary). Verifies the v0.60.0 F1 fix is complete; finds no new HIGH/MED. One
F1-sibling (stale-lock pool-blind reaper, LOW) + LOW/INFO defense-in-depth
residuals. Read-only static trace against felhom-agent @cc93dae/v0.60.0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Companion to felhom-agent v0.60.0 (proof-of-launch destroy gating + restore-
test band-advance). New documentation/runbooks/provisioning.md; campaign doc
updated with the resolution status and the operator manifest-bump follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full validation of the felhom pool + scoped-token migration on felhom-pve:
- Track A: complete agent op surface under the scoped token — 16/16 pass
- Track B: vmid origin traced (externally supplied); adversarial collision
probes prove the compensating rollback/teardown attempts to destroy a
guest the job never created (contained only by the pool ACL)
- Track C: containment + 3b-fix intact; Track D: no pool leaks, adopt/
uninstall pool handling correct
- Findings doc: documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md
No code change, no version bump. Demo left as found.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Read-only spike on felhom-pve. Verdicts: (SQ1) customer drives need NOT be PVE
storages — track in the intent registry (durable-id + host mount + bind), the
dir-storage is legacy. (SQ2) an unclaimed-disk filter (OS/LVM-PV/ZFS/PVE-storage/
VM-CT/mounted/RO/registry signals, fail-safe exclude-on-unknown) returns EXACTLY
{/dev/sdd} on the demo, never sda/sdb/sdc. (SQ3) mkfs sudoers is /dev/* — the
agent code is the only guard; DataBearing (wipe-confirm) is insufficient (OS disk
is data-bearing), so a mandatory agent unclaimed-gate + guarded-mkfs-wrapper
sudoers + a shared-box operator gate for self-serve format. (SQ4) new
/disks/candidates raw-scan endpoint feeding both wizards, registry-based
enrollment. (SQ5) migrate existing drives to the registry. Implementation plan +
open risks included. No product code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live supervised validation on felhom-pve: NO 403 on drive management under the
scoped token (agent has no storage-registration/pvesm path — all host-ops +
Datastore.Audit which is now box-wide). 3b-fix confirmed in the real UI (drives
Regisztrálva, no detach alert). A full physical wizard enroll couldn't complete
because a raw new device (SD-card reader, no PVE-storage entry) doesn't surface
in the init wizard (Observe() lists PVE storages only) — ACL-independent; flagged
as a separate new-drive-registration follow-up. No code change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin