sys_drive=disk-2 distinct, selftest [PASS] System data path; disk-full ENOSPC
confined to disk-2 (/ + docker-data unaffected, apps healthy); mp1 in vzdump;
SSD-fallback app (radarr) data lands on disk-2. 2.5 size=50G; reboot-survival
in 3.3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
1.6 deploy/HDD_PATH stable, 1.7 eject clean stable removal (v0.80 ruled out) +
impact names apps, 1.9 migrate flips stable->stable w/ verify-before-cleanup,
1.10 decommission via raw agentWhere + reconnect restore, 1.11 backups under
stable namespace. FINDING F1: restic binary still apt-installed (Dockerfile:58)
though all restic code moved to agent — dead weight. navidrome now on flash.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
Root cause: controller handleStorageRegister registers the raw /mnt/<name> path
instead of the stable /mnt/felhom-drives/<name> path the agent binds the drive at.
Drives ARE live in guest+container at the stable path; controller watches an empty
rootfs placeholder. Not v0.40.0 mp1; not stale agent state. Read-only diagnosis,
no fix applied. Recommended fix direction documented.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
Read-only diagnosis: restart-manager works (natural-exit probe RestartCount climbs;
manualRestart=false); docker kill is a manual stop so restart is correctly suppressed.
Definitive: kill -9 the controller PID (real crash) -> unless-stopped auto-restarts it,
RestartCount=1 healthy in 5s. Daemon stable (NRestarts=0, no OOM). Finding #1 downgraded
in the campaign-2 doc. Separate minor obs: boot-time drive-ordering race. No fix applied.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Storage lifecycle (unblocked by v0.74.0), break-and-recover C1-C8, backup edge cases.
Headline finding: docker restart-policy not honored -> a crashed felhom-controller has
no automatic recovery (oneshot bootstrap + dead restart policy). Plus disconnect-recreate
race, agent loopback whitelist, carried-forward #1 gaps. Demo restored to baseline.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confirmed H5 (ephemeral-port exhaustion): agentClient() builds a new http.Transport
per call (IdleConnTimeout:0, never CloseIdleConnections) -> ~5.8k leaked idle ESTABLISHED
sockets/day to 162:8443, exhausts the 28k ephemeral range in ~5 days of controller uptime
-> EADDRNOTAVAIL. :8006 immune (controller never dials pveproxy). Cleared by restart.
H1/H2/H3/H4/H6 ruled out with positive evidence. Fix is controller-side (reuse one
Client), NOT an agent rebind. No changes applied.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tracks felhom-agent v0.39.0, which dropped role + restic_repo_coord from the host-half
drives[]. Hub reads drives as json.RawMessage (verbatim passthrough) → no store/handler
change needed. Dropped role from the store golden + drHostHalf fixture; re-synced the api
host-report golden byte-identical with the agent copy (it previously lacked dr_recipe
entirely). Added TestAssembleDRRecipe_V1DriveShape regression guard (demonstrated to fail
when the fixture re-adds role, then reverted). No behavior change; redeploy optional.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DR recipe slice (hub half), grounded in SPIKE-dr-recipe-2026-06-16. The hub
receives two additive dr_recipe halves on the existing report paths (agent
storage/guest/PBS on host-report; controller customer/apps on the controller
report), stores them PLAINTEXT in a DEDICATED dr_recipe table keyed by customer
(each half preserves the other), and AssembleDRRecipe stitches them into one
operator-readable recipe (ignore-unknown + version-skew tolerant).
View: a DR-recipe panel on the customer page + GET /customers/{id}/dr-recipe.json
download (operator-auth, no secrets to redact). Plaintext-at-rest is correct —
the recipe is the clean inverse of the retired infra-backup.
Tests: store round-trip (each half preserves the other), assemble-matches-golden,
ignore-unknown + version skew, partial halves, no-secrets sweep. Manifest tag
bumped to v0.13.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Part 0 (live): flash apps on 9201 were down due to an operator pct reboot at
10:26 UTC + a boot-ordering race — dockerd auto-starts unless-stopped flash apps
~18s before the agent re-binds felhom-flash, so the create-time bind mkdir fails
(permission denied) and RestartCount=0 never retries. Drive healthy, data intact,
no USB drop, durable-id fine, drive-gate uninvolved. v0.70.0 self-restart RULED
OUT (container restart, not a guest reboot; +38min after exits). Fix: restarted
the 7 apps via the controller (drive present) — all Up. Flagged the intermediary
mount app-start race as an architectural gap.
Parts 1-3 (cited): characterized escrow (K + identity under recovery code R,
fingerprint-gated, hub zero-knowledge) + PBS whole-CT contents (rootfs/secrets in,
external drives out) + capstone DONE vs PENDING (agent-side recovery orchestration
not wired, syncer.go:92). Defined the secret-free DR recipe (guest sizing + drive
durable-id/role inventory + PVE storage + app bindings + PBS coords), sourced from
facts the agent/controller already hold, landing in the reserved
WireDesiredState.storage_manifest placeholder. Field-by-field boundary proof + a
no-secrets test spec. Fork list + recommendation: spec/emit/store the recipe now,
defer re-enrollment auth to slice 10D, never touch the escrow/PBS secret path.
No code changes, no version bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase-1 of SPIKE-infra-backup-2026-06-15. The infra-backup mechanism was dead
since slice 8C yet stored plaintext customer secrets at rest (app-secret key,
restic password, Cloudflare tokens) — a zero-knowledge violation — and its
absence made the daily expected_backup_missed email fire for healthy customers.
- Repoint monitor.CheckBackupDeadlines backup half to the agent host-report's
PBS snapshots (+vzdump): alarm only on no-backup / >26h stale / verify failed.
Keep the db_dump half. No host-report → no backup alarm (liveness owns that).
New store.GetLatestHostReportJSON. Tests incl. a companion that fails pre-fix.
- Remove the infra-backup endpoints, store methods/types, and operator panel;
/recovery now returns config_yaml only.
- migrate(): DROP infra_backup_versions/infra_backups + VACUUM (+wal_checkpoint)
to physically reclaim the plaintext pages, gated on table existence.
Flagged out-of-scope: exposed creds need operator rotation; legacy reports table
holds historical plaintext restic_password rows (separate leak, not purged here).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Report-only audit. Confirms the infra-backup push died at slice 8C
(controller v0.37.0) — builder deleted, PushInfraBackup +
NotifyBackupCompleted left as caller-less stubs; agent never
re-implemented it. Same root cause as the daily deadline-check email
(no backup_completed emitter). Live hub records inspected read-only:
plaintext secrets at rest (AES master key + restic pw + Cloudflare
tokens) = zero-knowledge violation; disk_layout is bare-metal-era.
Recommends phased retire-now / revive-concept-later (secret-free DR
recipe) + fork list. No code changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live-validated on demo guest 9201 (felhom-flash throwaway): shared/slave mount
propagation delivers host->guest drive swaps live with no restart and no pct
change; bind source = permanent stable dir => inherently C1-immune (clean boot
with drive absent). nsenter fallback non-viable on unprivileged LXC; ro-tmpfs
placeholder is the CAP_DAC_OVERRIDE-proof fail-close. Report-only, no code change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live unattended UI exercise of migration/decommission/re-enroll/eject +
feature sweep on shipped controller v0.66.2 / agent v0.32.0 (guest 9201).
No production code changed.
Migration engine solid (all moves checksum-verified identical, sources
cleaned, HDD_PATH flipped, apps healthy). Lifecycle gaps found:
- C1 (critical): decommission/eject leave a stale LXC mp (no pct delete)
-> guest fails pre-start hook on next reboot, ALL apps down. Recovered
live via pct set --delete mp1.
- H1: /api/storage/{disconnect,reconnect,restart-apps} are 404 (unrouted)
though settings.js calls them.
- H2: no UI button for /api/storage/decommission.
- H3: no one-click re-enroll for a decommissioned drive.
- M1: decommissioning the default drive doesn't reassign the default.
- M2: eject does NOT auto-remount (intent-gated like decommission).
- M3: userdata/import/calibre created 755 at enroll; migrate-all doesn't
re-assert setgid on pre-existing dirs (16/17 preserved 2775).
- M4: felhom-usb mount unit had stale UUID; M6: test-email fails (no SMTP).
End-state sane: apps healthy on felhom-flash (now default), both data
drives Aktiv, reboot-safe, sda/PBS/controller untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Report-only spike + one live throwaway FORMAT experiment on /dev/sdc.
No production code, no version bump.
- Q1: raw-disk discovery gap confirmed — Observe AND localapi GET /disks
are both built only from PVE storages (ListStorage/NodeStorage); a
not-yet-storage disk is invisible. proxmox client has no /disks/* method.
- Q3: safety classifier (RoleForRawDevice/SystemDisks/RoleForStorage)
already protects sda/felhom-pbs live, but needs two hardening checks:
whole-disk-granularity enrolled-subtraction + LVM-PV membership.
- Q6: FORMAT path proven end-to-end live — pvesh disks/directory create
-> agent Observe -> controller "Regisztralas" enroll. sdc left as the
clean enrolled felhom-flash B3 migration target. sda/sdb untouched.
- Premise correction: felhom-pbs is remote (no local PBS disk).
- Fork list for the build spec in the audit doc + REPORT.md.
Maps current namespace dir creation/ownership (live on 9201), the 9-app/13-mount
media blast radius, run-identity table (FileBrowser=uid1000, komga/audiobookshelf=root),
LXC idmap, and proposes the setgid GID-1000 convention. Flags: MigrateAll's merge walk
drops setgid+owner on userdata. Ends with a 7-item fork list. No code/deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contradicts the new->enrolled manual-enrollment model; manual enroll is
by design. The additive-discovery sub-point was adopted (controller
v0.64.0 A1) but only for paths deployed apps reference, not auto-register
of agent-attached drives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HDD attached at the agent layer (F9 bind) but never entered the controller storage
registry: enroll/register flows (runStorageInit/runStorageAttach/handleStorageRegister)
are the only paths that call registerStoragePath, and F9 attached via the agent directly;
AutoDiscoverStoragePaths is a one-time seed scanning only deployed-app HDD_PATHs. So the
HDD is 'Nem regisztrált' + absent from the deploy dropdown (GetSchedulableStoragePaths).
Verdict: F9 NOT closed. Fix direction: auto-register-on-attach + clearer manual register.
Relocates the 2026-06-14 live-drive findings + fixspec from the felhom-controller
repo root into documentation/audits/ (alongside the bughunt-reconcile/deep-sweep
records), and preserves the M18/M19 implementable fix-plans (from the deleted
controller fix/m18 + fix/m19 branches) into a new documentation/backlog/. Part of
the trunk-based no-branches reconciliation.
- 05-hub-architecture.md: stale 'felhom-hub v0.6.3' -> v0.11.0 (design-draft note).
- auth-hub-sync-integrations.md: full onlyoffice:nextcloud occ command sequence.
- storage-monitoring-metrics.md: metricsDBPath verified to coincide with the volume-backed
data_dir on the bootstrap guest (persists; hardcoding is latent fragility only).
- REORG-NOTES: gaps 2/3 CLOSED, gap 5 partially closed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New documentation/controller/ subtree (module map + deploy/stack-lifecycle, backup,
storage/monitoring/metrics, auth/hub/sync/integrations) grounded in current source;
top-level documentation/README.md index across controller/agent/platform/hub/audits;
REORG-NOTES with the verification ledger + flagged doc-gaps. Supersedes (keeps) the
v0.33 controller planning map. Additive only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings the 2026-06-13 audit + reconciliation under documentation/audits/ with
per-finding remediation status (controller v0.59.0 fixes deployed to demo;
AGENT-001 fix prepared/pending review).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validated both unvalidated 10D mechanisms: (1) identity-bundle escrow round-trip
via age scrypt+AEAD (recover on a secret-less box, wrong-R fails closed), (2)
Cloudflare tunnel re-establishment — running the recovered token on a new box
routes the hostname there immediately (no DNS change); the old connector is a
hot standby, superseded in routing but not auto-retired -> 10D must rotate the
tunnel/PBS token + retire the stale connector for host-loss security. Redacted;
secrets shredded; live demo untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agent-only implementation (felhom-agent v0.17.0 escrow.Consume); no hub code
change. 10C done; 10D is the last piece of slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add DELETE /hosts/{id}/jobs/{job_id} (per-host self-scoped, idempotent) so the
agent clears a job after executing or terminally rejecting it. The hub stores
the operator-signed blobs opaquely (no signing key — cannot forge or open);
the agent verifies + executes. Doc 03 §4/§6/§9 updated (operator-signed path
live; 8C wipe completes; 10B done).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin