Operator sets a minimum controller version (FLOOR), per-customer defaulting to a
global floor; the report ACK returns the effective floor + latest_version so the
controller auto-updates to the floor when below it (latest stays the opt-in button).
- store: min_controller_version column + hub_settings global floor + Effective/
Get/SetGlobal/SetMin resolution + config/env DEFAULT_MIN_CONTROLLER_VERSION
- handler: report ACK {min_controller_version, latest_version}; LatestVersionProvider
- web: global floor editor + per-customer override form + Floor column (English)
- tests: floor resolution + ACK + render; override-precedence red-proof verified
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
Operator-run Day-0 automation for a fresh PVE host: pveum token -> hub
POST /host-enroll (single secret, option C) -> agent config -> felhom-agent
--selftest=provision -> verify. Idempotent/resumable (state.json), --dry-run,
--resume, --force, --preserve-from; --mode dr is a documented 10D stub.
Live-validated on felhom-pve: authorized wipe of demo guest 9201 + full
re-provision from the golden -> controller config-pull + public tunnel HTTP 200
+ host-report of 9201; idempotent --resume no-op. Found+fixed one ordering bug
during the run (token ACL must be applied AFTER token rotation — token-remove
purges the ACL).
Adds scripts/README.md (operator runbook) + scripts/CHANGELOG.md; REPORT
overwritten; spike doc pointer added. No service code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
New POST /api/v1/host-enroll (handleHostEnroll): X-Retrieval-Password authed,
body {customer_id} -> {host_id, api_key}. Mint-once-reuse (201 first, 200
reuse) so re-running the host-bootstrap never orphans a running agent's key;
auth checked before any mint. Backed by new Store.GetHostByCustomer
(ORDER BY updated_at DESC LIMIT 1, idx_hosts_customer).
GET /config/{id} and global-key POST /admin/hosts left untouched. Exact-match
route (path == "/host-enroll") to avoid the /hosts/ prefix collision.
Tests: host_enroll_test.go (mint/reuse/401-no-mint/404/400) + GetHostByCustomer
store test; companion red-proof verified always-mint fails the reuse assertion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
Empirically validates the composed Day-0 chain against the LIVE hub: host
mint (POST /admin/hosts), agent host-auth via the real internal/hub client
(host-report + FetchDesiredState), and controller config-pull
(GET /config/{id} + X-Retrieval-Password). All links + wrong-cases pass.
Option A works end-to-end with zero new code; option C (single-secret enroll)
feasible + recommended for cutover. Throwaway customer/host cleaned up.
No production code, no version bump (spike).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
sys_drive=disk-2 distinct, selftest [PASS] System data path; disk-full ENOSPC
confined to disk-2 (/ + docker-data unaffected, apps healthy); mp1 in vzdump;
SSD-fallback app (radarr) data lands on disk-2. 2.5 size=50G; reboot-survival
in 3.3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
1.6 deploy/HDD_PATH stable, 1.7 eject clean stable removal (v0.80 ruled out) +
impact names apps, 1.9 migrate flips stable->stable w/ verify-before-cleanup,
1.10 decommission via raw agentWhere + reconnect restore, 1.11 backups under
stable namespace. FINDING F1: restic binary still apt-installed (Dockerfile:58)
though all restic code moved to agent — dead weight. navidrome now on flash.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
Root cause: controller handleStorageRegister registers the raw /mnt/<name> path
instead of the stable /mnt/felhom-drives/<name> path the agent binds the drive at.
Drives ARE live in guest+container at the stable path; controller watches an empty
rootfs placeholder. Not v0.40.0 mp1; not stale agent state. Read-only diagnosis,
no fix applied. Recommended fix direction documented.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
Read-only diagnosis: restart-manager works (natural-exit probe RestartCount climbs;
manualRestart=false); docker kill is a manual stop so restart is correctly suppressed.
Definitive: kill -9 the controller PID (real crash) -> unless-stopped auto-restarts it,
RestartCount=1 healthy in 5s. Daemon stable (NRestarts=0, no OOM). Finding #1 downgraded
in the campaign-2 doc. Separate minor obs: boot-time drive-ordering race. No fix applied.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Storage lifecycle (unblocked by v0.74.0), break-and-recover C1-C8, backup edge cases.
Headline finding: docker restart-policy not honored -> a crashed felhom-controller has
no automatic recovery (oneshot bootstrap + dead restart policy). Plus disconnect-recreate
race, agent loopback whitelist, carried-forward #1 gaps. Demo restored to baseline.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confirmed H5 (ephemeral-port exhaustion): agentClient() builds a new http.Transport
per call (IdleConnTimeout:0, never CloseIdleConnections) -> ~5.8k leaked idle ESTABLISHED
sockets/day to 162:8443, exhausts the 28k ephemeral range in ~5 days of controller uptime
-> EADDRNOTAVAIL. :8006 immune (controller never dials pveproxy). Cleared by restart.
H1/H2/H3/H4/H6 ruled out with positive evidence. Fix is controller-side (reuse one
Client), NOT an agent rebind. No changes applied.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tracks felhom-agent v0.39.0, which dropped role + restic_repo_coord from the host-half
drives[]. Hub reads drives as json.RawMessage (verbatim passthrough) → no store/handler
change needed. Dropped role from the store golden + drHostHalf fixture; re-synced the api
host-report golden byte-identical with the agent copy (it previously lacked dr_recipe
entirely). Added TestAssembleDRRecipe_V1DriveShape regression guard (demonstrated to fail
when the fixture re-adds role, then reverted). No behavior change; redeploy optional.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DR recipe slice (hub half), grounded in SPIKE-dr-recipe-2026-06-16. The hub
receives two additive dr_recipe halves on the existing report paths (agent
storage/guest/PBS on host-report; controller customer/apps on the controller
report), stores them PLAINTEXT in a DEDICATED dr_recipe table keyed by customer
(each half preserves the other), and AssembleDRRecipe stitches them into one
operator-readable recipe (ignore-unknown + version-skew tolerant).
View: a DR-recipe panel on the customer page + GET /customers/{id}/dr-recipe.json
download (operator-auth, no secrets to redact). Plaintext-at-rest is correct —
the recipe is the clean inverse of the retired infra-backup.
Tests: store round-trip (each half preserves the other), assemble-matches-golden,
ignore-unknown + version skew, partial halves, no-secrets sweep. Manifest tag
bumped to v0.13.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Part 0 (live): flash apps on 9201 were down due to an operator pct reboot at
10:26 UTC + a boot-ordering race — dockerd auto-starts unless-stopped flash apps
~18s before the agent re-binds felhom-flash, so the create-time bind mkdir fails
(permission denied) and RestartCount=0 never retries. Drive healthy, data intact,
no USB drop, durable-id fine, drive-gate uninvolved. v0.70.0 self-restart RULED
OUT (container restart, not a guest reboot; +38min after exits). Fix: restarted
the 7 apps via the controller (drive present) — all Up. Flagged the intermediary
mount app-start race as an architectural gap.
Parts 1-3 (cited): characterized escrow (K + identity under recovery code R,
fingerprint-gated, hub zero-knowledge) + PBS whole-CT contents (rootfs/secrets in,
external drives out) + capstone DONE vs PENDING (agent-side recovery orchestration
not wired, syncer.go:92). Defined the secret-free DR recipe (guest sizing + drive
durable-id/role inventory + PVE storage + app bindings + PBS coords), sourced from
facts the agent/controller already hold, landing in the reserved
WireDesiredState.storage_manifest placeholder. Field-by-field boundary proof + a
no-secrets test spec. Fork list + recommendation: spec/emit/store the recipe now,
defer re-enrollment auth to slice 10D, never touch the escrow/PBS secret path.
No code changes, no version bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase-1 of SPIKE-infra-backup-2026-06-15. The infra-backup mechanism was dead
since slice 8C yet stored plaintext customer secrets at rest (app-secret key,
restic password, Cloudflare tokens) — a zero-knowledge violation — and its
absence made the daily expected_backup_missed email fire for healthy customers.
- Repoint monitor.CheckBackupDeadlines backup half to the agent host-report's
PBS snapshots (+vzdump): alarm only on no-backup / >26h stale / verify failed.
Keep the db_dump half. No host-report → no backup alarm (liveness owns that).
New store.GetLatestHostReportJSON. Tests incl. a companion that fails pre-fix.
- Remove the infra-backup endpoints, store methods/types, and operator panel;
/recovery now returns config_yaml only.
- migrate(): DROP infra_backup_versions/infra_backups + VACUUM (+wal_checkpoint)
to physically reclaim the plaintext pages, gated on table existence.
Flagged out-of-scope: exposed creds need operator rotation; legacy reports table
holds historical plaintext restic_password rows (separate leak, not purged here).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Report-only audit. Confirms the infra-backup push died at slice 8C
(controller v0.37.0) — builder deleted, PushInfraBackup +
NotifyBackupCompleted left as caller-less stubs; agent never
re-implemented it. Same root cause as the daily deadline-check email
(no backup_completed emitter). Live hub records inspected read-only:
plaintext secrets at rest (AES master key + restic pw + Cloudflare
tokens) = zero-knowledge violation; disk_layout is bare-metal-era.
Recommends phased retire-now / revive-concept-later (secret-free DR
recipe) + fork list. No code changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live-validated on demo guest 9201 (felhom-flash throwaway): shared/slave mount
propagation delivers host->guest drive swaps live with no restart and no pct
change; bind source = permanent stable dir => inherently C1-immune (clean boot
with drive absent). nsenter fallback non-viable on unprivileged LXC; ro-tmpfs
placeholder is the CAP_DAC_OVERRIDE-proof fail-close. Report-only, no code change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live unattended UI exercise of migration/decommission/re-enroll/eject +
feature sweep on shipped controller v0.66.2 / agent v0.32.0 (guest 9201).
No production code changed.
Migration engine solid (all moves checksum-verified identical, sources
cleaned, HDD_PATH flipped, apps healthy). Lifecycle gaps found:
- C1 (critical): decommission/eject leave a stale LXC mp (no pct delete)
-> guest fails pre-start hook on next reboot, ALL apps down. Recovered
live via pct set --delete mp1.
- H1: /api/storage/{disconnect,reconnect,restart-apps} are 404 (unrouted)
though settings.js calls them.
- H2: no UI button for /api/storage/decommission.
- H3: no one-click re-enroll for a decommissioned drive.
- M1: decommissioning the default drive doesn't reassign the default.
- M2: eject does NOT auto-remount (intent-gated like decommission).
- M3: userdata/import/calibre created 755 at enroll; migrate-all doesn't
re-assert setgid on pre-existing dirs (16/17 preserved 2775).
- M4: felhom-usb mount unit had stale UUID; M6: test-email fails (no SMTP).
End-state sane: apps healthy on felhom-flash (now default), both data
drives Aktiv, reboot-safe, sda/PBS/controller untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Report-only spike + one live throwaway FORMAT experiment on /dev/sdc.
No production code, no version bump.
- Q1: raw-disk discovery gap confirmed — Observe AND localapi GET /disks
are both built only from PVE storages (ListStorage/NodeStorage); a
not-yet-storage disk is invisible. proxmox client has no /disks/* method.
- Q3: safety classifier (RoleForRawDevice/SystemDisks/RoleForStorage)
already protects sda/felhom-pbs live, but needs two hardening checks:
whole-disk-granularity enrolled-subtraction + LVM-PV membership.
- Q6: FORMAT path proven end-to-end live — pvesh disks/directory create
-> agent Observe -> controller "Regisztralas" enroll. sdc left as the
clean enrolled felhom-flash B3 migration target. sda/sdb untouched.
- Premise correction: felhom-pbs is remote (no local PBS disk).
- Fork list for the build spec in the audit doc + REPORT.md.
Maps current namespace dir creation/ownership (live on 9201), the 9-app/13-mount
media blast radius, run-identity table (FileBrowser=uid1000, komga/audiobookshelf=root),
LXC idmap, and proposes the setgid GID-1000 convention. Flags: MigrateAll's merge walk
drops setgid+owner on userdata. Ends with a 7-item fork list. No code/deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contradicts the new->enrolled manual-enrollment model; manual enroll is
by design. The additive-discovery sub-point was adopted (controller
v0.64.0 A1) but only for paths deployed apps reference, not auto-register
of agent-attached drives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HDD attached at the agent layer (F9 bind) but never entered the controller storage
registry: enroll/register flows (runStorageInit/runStorageAttach/handleStorageRegister)
are the only paths that call registerStoragePath, and F9 attached via the agent directly;
AutoDiscoverStoragePaths is a one-time seed scanning only deployed-app HDD_PATHs. So the
HDD is 'Nem regisztrált' + absent from the deploy dropdown (GetSchedulableStoragePaths).
Verdict: F9 NOT closed. Fix direction: auto-register-on-attach + clearer manual register.
Relocates the 2026-06-14 live-drive findings + fixspec from the felhom-controller
repo root into documentation/audits/ (alongside the bughunt-reconcile/deep-sweep
records), and preserves the M18/M19 implementable fix-plans (from the deleted
controller fix/m18 + fix/m19 branches) into a new documentation/backlog/. Part of
the trunk-based no-branches reconciliation.
- 05-hub-architecture.md: stale 'felhom-hub v0.6.3' -> v0.11.0 (design-draft note).
- auth-hub-sync-integrations.md: full onlyoffice:nextcloud occ command sequence.
- storage-monitoring-metrics.md: metricsDBPath verified to coincide with the volume-backed
data_dir on the bootstrap guest (persists; hardcoding is latent fragility only).
- REORG-NOTES: gaps 2/3 CLOSED, gap 5 partially closed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin