docs: REPORT.md for hub v0.12.0 infra-backup retirement
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -4,127 +4,80 @@
|
||||
|
||||
---
|
||||
|
||||
# REPORT — SPIKE: stable intermediary mount (swap the backing drive underneath) (2026-06-15)
|
||||
# Hub v0.12.0 — retire Infra Backup, purge plaintext secrets, fix backup-deadline email
|
||||
|
||||
## Type
|
||||
Architecture spike — report-only + live throwaway experiments on `felhom-flash` (`/dev/sdc1`) on demo guest 9201 (`felhom-pve`, PVE 9.2.2, unprivileged LXC). No production code, no version bump. Full doc: [documentation/audits/SPIKE-intermediary-mount-2026-06-15.md](documentation/audits/SPIKE-intermediary-mount-2026-06-15.md). Decides the *bigger architecture* behind the **C1 critical bug** surfaced in the B3 run below; the self-heal hook ships separately as the immediate C1 net.
|
||||
**Date:** 2026-06-16
|
||||
**Scope:** Phase-1 of `documentation/audits/SPIKE-infra-backup-2026-06-15.md` (hub side).
|
||||
**Deployed:** `felhom-hub:v0.12.0` live on k3s (ArgoCD `felhom` app Synced); commit `0635640`.
|
||||
**Companion repo:** `felhom-controller` v0.69.0 (own REPORT there).
|
||||
|
||||
## Result
|
||||
**Viable and strictly better than today's per-drive `pct set -mpN` model.** Bind a *permanent* stable host dir into the guest once (one provision-time reboot), then mount/unmount the real drive **underneath it host-side** — changes reach the running guest **live, no restart, no `pct` change** — via `shared`(host)/`slave`(guest) mount propagation that PVE/LXC delivers automatically. Proven both directions (mount appears / unmount disappears), both bind forms.
|
||||
## What shipped
|
||||
|
||||
**Inherently C1-immune (headline):** rebooted guest 9201 with the drive **absent** → clean boot in 12 s, because the `mp` bind source is the always-present stable dir, not `<drive>/felhom-data` (which vanishes today → guest brick). End-state: 9201 restored to original config (mp0/1/2/9), 25 containers healthy; all experiment mounts/dirs/artifacts removed; `felhom-usb`/`sda`/PBS/`local(-lvm)` never touched.
|
||||
1. **Backup-deadline check repointed to PBS freshness** (`internal/monitor/deadline.go`).
|
||||
The backup half no longer queries for a `backup_completed` event (nothing emits it post-slice-8C,
|
||||
so it fired daily for every healthy customer). It now reads the customer's **latest agent
|
||||
host-report** (`store.GetLatestHostReportJSON`) and raises `expected_backup_missed` only on
|
||||
positive evidence: no PBS snapshot / successful vzdump at all, newest backup older than **26h**, or
|
||||
the newest PBS snapshot's `verify_state == "failed"`. A fresh-but-unverified snapshot is **not** a
|
||||
failure (PBS verifies on its own cadence — alarming on it would recreate the false alarm). The
|
||||
**db-dump half is unchanged**. A customer with **no host-report** gets no backup alarm here
|
||||
(liveness is the host-staleness checker's job).
|
||||
|
||||
## Findings / decisions
|
||||
- **Q1 propagation:** ✅ live no-restart, both forms; guest receives binds as `shared,slave` (no custom `lxc.mount.entry` flags needed).
|
||||
- **Q2 nsenter fallback:** ❌ not viable on the unprivileged LXC (`mount: must be superuser`; block device absent from guest `/dev`). **Use propagation.**
|
||||
- **Q3 idmap:** ✅ guest uid/gid 1000 → host 101000 read/write unchanged; no regression.
|
||||
- **Q4 fail-closed:** ✅ naturally fail-closed (bare stable dir host-root-owned/unmapped → guest can't write). **`chmod 0000` is reliable only on a host-root-owned dir** — container-root's `CAP_DAC_OVERRIDE` bypasses 0000 on a *mapped*-owned dir (proven). **ro tmpfs placeholder = the capability-proof EROFS fail-close**, auto-shadowed by the drive on reconnect, auto-re-exposed on eject. App-stop stays primary.
|
||||
- **Q5 C1 immunity:** ✅ reboot-with-drive-absent boots clean.
|
||||
- **Q6 round-trip:** ✅ eject/reconnect = pure host `mount`/`umount`, no restart; reboot mid-eject = no brick.
|
||||
- **Q7/Q8 fit:** parent-bind makes enroll/plug/eject/reconnect **all restart-free**; only whole-guest decommission touches the bind. Migration/FileBrowser cleaner. **Confinement correction:** bind only `<raw>/felhom-data` onto the stable path (proven — customer's `dump/images/private/lost+found` stay out of the guest), not the raw whole-drive mount.
|
||||
2. **Infra Backup feature removed** (`api/handler.go`, `store/store.go`, `web/configs.go`,
|
||||
`templates/customer_unified.html`, `templates/customer.html`): the `POST/GET /api/v1/infra-backup[…]`
|
||||
endpoints + handlers, the store methods/types (`SaveInfraBackup`/`GetInfraBackup`/`GetInfraBackupByID`/
|
||||
`GetInfraBackupMeta`/`ListInfraBackupVersions`/`pruneInfraBackups`, `InfraBackupMeta`/`InfraBackupVersion`),
|
||||
and the operator "Infra Backup" panel. `GET /api/v1/recovery/{id}` now returns **config_yaml only**.
|
||||
The customer-page config-drift badge (diffed against the stored controller.yaml) is hidden; the live
|
||||
"Show Diff" path is unaffected.
|
||||
|
||||
## Fork picks (for the build spec)
|
||||
parent-bind form · propagation (not nsenter) · ro-placeholder fail-close (+ host-root-owned dir as 2nd layer) · `HDD_PATH` kept identical via in-guest bind/symlink *or* repoint (decide) · agent mounts `<raw>/felhom-data` onto the stable path. See the doc's FORK LIST + current→intermediary migration sketch.
|
||||
3. **Plaintext secret purge** (`store/store.go migrate()`): `DROP infra_backup_versions; DROP
|
||||
infra_backups; VACUUM; wal_checkpoint(TRUNCATE)` — gated on table existence so normal restarts skip
|
||||
it. VACUUM physically reclaims the freed pages so the plaintext keys/tokens are not merely delinked.
|
||||
|
||||
---
|
||||
## Gate (STEP 1a) — PASSED before any change
|
||||
|
||||
# REPORT — B3: live unattended storage-lifecycle exercise + feature sweep (2026-06-15)
|
||||
The demo's latest host-report (id 699, agent v0.36.7) carried **5 PBS snapshots, all `verify=ok`,
|
||||
newest `2026-06-15T18:41:29Z` (13.9 h old, < 26 h)** plus a matching successful vzdump. The repoint's
|
||||
data source is present and fresh, so the repoint alone clears the email.
|
||||
|
||||
## Type
|
||||
Live, unattended, full-UI exercise of migration / decommission / re-enroll / eject + a broad feature sweep on shipped **controller v0.66.2 / agent v0.32.0** (demo guest 9201). No production code changed. Full doc: [documentation/audits/B3-storage-lifecycle-2026-06-15.md](documentation/audits/B3-storage-lifecycle-2026-06-15.md).
|
||||
## Verification (live, non-hollow)
|
||||
|
||||
## Result
|
||||
Migration engine is **solid** (every move checksum-verified identical, sources cleaned, HDD_PATH flipped, apps healthy). The decommission/eject/re-enroll **lifecycle has serious gaps**, incl. one critical guest-bricking bug and three unrouted endpoints the UI calls. End-state is sane: all apps healthy on **felhom-flash (now default)**, both data drives Aktív, sda/PBS/controller untouched.
|
||||
- **Build + tests:** `go build ./... && go test ./...` green. `deadline_test.go` covers fresh+verified→
|
||||
quiet (the **companion**), stale→alarm, failed-verify→alarm, no-host-report→quiet, db-dump half
|
||||
preserved, and a pure `assessBackupFreshness` table. Red-proof: with the old event-based half
|
||||
temporarily restored, the companion + no-report + db-dump-preserved tests **fail**, while the
|
||||
pure-helper test still passes — the behavioral tests are sensitive to the logic.
|
||||
- **Deploy:** hub v0.12.0 rolled out; startup log: `Retired infra-backup: dropped 2 table(s) and
|
||||
VACUUMed to reclaim plaintext pages`.
|
||||
- **Tables gone:** `.tables` on the live `/data/hub.db` shows no `infra_backup*`. DB shrank
|
||||
**65.8 MB → 52.2 MB**; no `-wal` sidecar lingers.
|
||||
- **Secret purge proof (grep on the live post-VACUUM DB file):** the infra-backup-exclusive markers
|
||||
are gone — `encryption_key_b64` 17→**0**, `controller_config_b64` 11→**0**, and the actual
|
||||
AES-key value from spike record 109 present→**0** (physically reclaimed, not just delinked). The
|
||||
residual `cf_api_token`/`cf_tunnel_token` hits are in `customer_configs.config_json` (the hub's
|
||||
legitimate config store), not the infra-backup blobs (which held CF tokens base64-encoded inside the
|
||||
now-zero `controller_config_b64`).
|
||||
- **Endpoints retired:** `POST /api/v1/infra-backup` and `GET …/versions` return **404**; control
|
||||
`POST /api/v1/report` still returns **401** (routed, unauth) — confirming the 404s are route removal,
|
||||
not a blanket failure.
|
||||
- **Demo email fix:** the next 03:00 deadline run cannot be observed within this session, but the live
|
||||
demo host-report (fresh verified PBS, 13.9 h) exercises the not-missed path, and demo emits
|
||||
`db_dump_completed` daily (never missed) — so `expected_backup_missed` will no longer fire.
|
||||
|
||||
UI-driven per A4 where the dashboard exposes the action (per-app migrate + migrate-all driven via the real buttons, native `confirm()` auto-accepted); operations the UI lacks a button for / calls an unrouted endpoint for were driven via the controller's own JSON API in-guest (correct `Host` header, bypassing the body-mangling proxy) — each gap documented as a finding.
|
||||
## Flagged for the operator (out of scope here)
|
||||
|
||||
## Findings (→ next-slice fixes)
|
||||
- 🔴 **C1 (critical):** decommission/eject never `pct set --delete mpN`, so the stale LXC bind **bricks the guest on next reboot** (pre-start hook exit 255, ALL apps down). Recovered live via manual `pct set --delete mp1`.
|
||||
- 🟠 **H1:** `/api/storage/disconnect`, `/api/storage/reconnect`, `/api/storage/restart-apps` are **404 (unrouted)** though settings.js calls them.
|
||||
- 🟠 **H2:** no UI button for `/api/storage/decommission` (B2b flow unreachable from the dashboard).
|
||||
- 🟠 **H3:** no one-click re-enroll for a decommissioned drive (only "Eltávolítás").
|
||||
- 🟡 **M1:** decommissioning the default drive **doesn't reassign the default** (orphaned, no failover).
|
||||
- 🟡 **M2:** eject does **NOT auto-remount** on reconnect — intent-gated-skipped like decommission (premise contradicted); difference is bind-record retention + soft-marker.
|
||||
- 🟡 **M3:** `userdata/import/calibre` created **755** at enroll (skeleton gap) and migrate-all doesn't re-assert setgid on pre-existing dirs (16/17 dirs preserved 2775).
|
||||
- 🟡 **M4:** felhom-usb's systemd mount unit had a **stale UUID** (couldn't auto-mount); fixed in restore. **M6:** test-email fails (no SMTP).
|
||||
- **Rotate the exposed credentials** that were in the dropped blobs (Cloudflare API + tunnel tokens
|
||||
for `demo-felhom.eu` first; hub/session secrets if shared). They remain valid until rotated; the
|
||||
purge removes the at-rest copy but not their validity.
|
||||
- **Separate historical leak:** the legacy `reports` table holds thousands of rows with a plaintext
|
||||
`restic_password` value from old controller versions. The **live** controller no longer sends it
|
||||
(removed in controller v0.69.0), but the historical rows persist — a distinct purge/rotation
|
||||
decision, deliberately not done here.
|
||||
- **`peti-felhom` is a defunct customer marked `active`** (no host-report or controller report since
|
||||
Feb 2026). Its daily `expected_backup_missed` stops with this change (no host-report → no backup
|
||||
alarm), but it should be marked inactive to silence all residual noise.
|
||||
|
||||
## Passes
|
||||
Per-app migration, migrate-all (7 apps, checksums identical, setgid 16/17), decommission state (intent/unmount/prune/Kiváltva), no-auto-rebind invariant (intent-gated), missing-storage badges (8×), re-enroll marker-clear, encrypted backup→PBS (snapshot landed), FileBrowser umask (PID1 0002 → 2775 group-writable + collab write), fresh-app deploy+lifecycle+logs, monitoring page.
|
||||
## Out of scope (untouched)
|
||||
|
||||
---
|
||||
|
||||
# REPORT — SPIKE: self-serve raw-drive plug-and-enroll (GATE) (2026-06-15)
|
||||
|
||||
## Type
|
||||
GATE spike — report-only + one live throwaway experiment on `/dev/sdc`. **No production code, no version bump, no service restart.** Build spec is gated on the fork picks below.
|
||||
Full doc: [documentation/audits/SPIKE-raw-drive-enroll-2026-06-15.md](documentation/audits/SPIKE-raw-drive-enroll-2026-06-15.md).
|
||||
|
||||
## One-screen summary
|
||||
The plug-and-enroll flow is mostly assemblable from primitives that already exist. The genuine **net-new** is (a) a **raw-disk discovery source** — the agent's `Observe` and localapi `GET /disks` are *both* built only from PVE storages (`ListStorage`/`NodeStorage`), so a disk that isn't already a PVE storage is invisible — and (b) a **create-storage call** (the agent creates no PVE storages today). The safety classifier **already exists** (`RoleForRawDevice`, `SystemDisks`, `RoleForStorage` in `internal/storage/role.go`) and correctly protects the live system disk, but needs **two hardening additions** before it can drive a *discovery* list.
|
||||
|
||||
Validated live on `felhom-pve`:
|
||||
- **sda** (ESP + sole LVM PV `pve` VG holding root, local-lvm, **guest 9201's disks**) → PROTECTED (system); pinned via `/boot/efi`=`sda2`.
|
||||
- **sdb** (`felhom-usb`, ext4 `/mnt/felhom-usb`) → PROTECTED (enrolled data).
|
||||
- **sdc** (removable junk iso9660+EFI+hfsplus) → OFFERABLE.
|
||||
- **Premise correction:** `felhom-pbs` is a **remote** `pbs:` storage (server 192.168.0.180) — **no local PBS disk** exists; it's protected at the storage tier, not a raw-disk tier.
|
||||
|
||||
**Live FORMAT experiment (serial-fenced to sdc):** `pvesh create …/disks/directory --name felhom-flash --device /dev/sdc --filesystem ext4 --add_storage 1` did partition+mkfs+mount+register in one call (fs-UUID mount unit, `is_mountpoint 1`, `/mnt/pve/felhom-flash`). It **immediately appeared in the controller's agent-drive view as user-data / data-bearing / "Nem regisztrált" with a one-click `Regisztrálás`** — no agent restart. sdc left as the clean enrolled `felhom-flash` B3 migration target.
|
||||
|
||||
## The two safety-gate hardening gaps (must close)
|
||||
1. **Whole-disk-granularity enrolled-subtraction** — `deviceRole` matches a registered storage by exact `BackingDevice==device`; a discovery flow asks about whole disk `/dev/sdb` while `felhom-usb` backs `/dev/sdb1`, so the match fails and sdb would be offered for format. Compare at whole-disk granularity.
|
||||
2. **LVM-PV / guest-image membership** — `SystemDisks` resolves the system disk from the mount table only; a non-boot disk holding `local-lvm` PVs / guest images would be missed. Add a `disks/lvm` PV-membership check.
|
||||
|
||||
## Fork list (operator decides)
|
||||
1. Discovery source: `disks/list`+`disks/lvm`+mount-table+Observe (recommended) ± `lsblk` for display.
|
||||
2. Safety signals: mount-table-only (ships, but can offer a non-boot LVM disk) vs **hardened** (the 2 checks above — recommended).
|
||||
3. Mount path: PVE default `/mnt/pve/<name>` vs **`/mnt/<name>`** via `EnsureMount`+`pvesm add dir` (recommended, matches felhom-usb).
|
||||
4. Format-vs-enroll-as-is: single supported-fs partition → as-is; else format; format on ambiguity.
|
||||
5. Endpoints: `GET /disks/raw` + `POST /disks/init` + `POST /disks/attach-existing`, `withGuest`-scoped, agent re-derives the gate.
|
||||
6. v1 scope: **format-only first** (proven live) + enroll-as-is fast-follow (recommended) vs both in v1.
|
||||
7. Normalize `content` (vs PVE all-types default); typed proxmox-client create method (vs host-side `pvesh`).
|
||||
|
||||
---
|
||||
|
||||
# REPORT — Slice 10D (hub half): DR capstone — recovery mode + re-enroll + directive serving (hub v0.11.0) (2026-06-10)
|
||||
|
||||
## Type
|
||||
|
||||
TASK (CC-implemented). The hub half of the slice-10 DR capstone (closes slice 10). Pairs with
|
||||
`felhom-agent` v0.18.0 (identity escrow + restore-mode consumption).
|
||||
|
||||
## What changed (hub)
|
||||
|
||||
The hub ORCHESTRATES recovery but holds **no usable secret and no Cloudflare write-power** — a
|
||||
compromised hub can at most hand out **opaque** blobs (they need `R`, which the hub never has) + rotate
|
||||
its own per-host credential. It cannot hijack a customer's tunnel (the destructive rotation is the
|
||||
operator's job).
|
||||
|
||||
### API
|
||||
- **`PUT/DELETE /admin/hosts/{id}/recovery-mode`** (global key) — arm/disable recovery mode with a
|
||||
bounded TTL (clamped [60s, 4h], default 30m → **auto-expires**). Directive + re-enroll are served
|
||||
ONLY while active.
|
||||
- **`POST /hosts/{id}/re-enroll`** — gated ONLY on recovery mode (the lost box has no old key). Rotates
|
||||
the host's API key to the new box's key (**old box revoked**) + returns the directive + opaque blobs.
|
||||
- **`GET /hosts/{id}/restore-directive`** (re-enrolled key, recovery-gated) — re-fetch.
|
||||
- The slice-7 escrow upload now also accepts the **identity blob** + **non-secret directive** (additive).
|
||||
|
||||
### Store
|
||||
- `hosts.recovery_mode_until`; `host_escrow.identity_blob` + `directive_json`. Methods:
|
||||
`SetRecoveryMode`/`ClearRecoveryMode`, `RotateHostAPIKey`, `SaveHostDRBundle`/`GetHostDRBundle`.
|
||||
|
||||
## Tests (green)
|
||||
- re-enroll refused without recovery mode (403); recovery-arm is global-key-only; re-enroll **rotates +
|
||||
revokes** (old key→401, new key→200); directive served only in recovery mode + **expires**; clear
|
||||
disables re-enroll.
|
||||
|
||||
## Docs
|
||||
- Doc 03 §9 (10D done → **SLICE 10 CLOSED**) + the host-loss DR flow with the **operator-side rotation**
|
||||
model (hub orchestrates + read-only verifies; the operator deletes the stale connector + rotates the
|
||||
tunnel/PBS token from a trusted environment).
|
||||
|
||||
## Deferred (non-blocking, per the locked model)
|
||||
- The Config DR/Recovery **web UI** (functional today via the recovery-mode admin API) + a small
|
||||
operator rotation CLI. **No Cloudflare write-credential is in the hub by design.**
|
||||
|
||||
## Pending
|
||||
- Build + deploy hub v0.11.0 + agent v0.18.0; run the operator-in-the-loop DR drill (throwaway identity).
|
||||
Credential rotation; the Komga healthcheck; the secret-free DR "recipe" (the later DR slice).
|
||||
|
||||
Reference in New Issue
Block a user