From 4e3f6c48d4d7114031944d56b6cb3b3713f569f3 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 18 Jul 2026 13:35:00 +0200 Subject: [PATCH] docs(R-7b): share backup EXECUTION shipped (controller v0.145.0, Model B') MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - capability map: SMB row KNOWN GAP cleared -> share data rides both tiers; the offsite leg + restore round-trip flagged as not-yet-live-exercised - ROADMAP R-7b: idea -> SHIPPED, with the Model B' rationale and the live evidence - controller/sharing.md: the KNOWN GAP block replaced by the execution contract; operator note corrected — samba IS liveness-monitored since v0.145.0 --- .../architecture/00-capability-map.md | 2 +- documentation/backlog/ROADMAP.md | 2 +- documentation/controller/sharing.md | 57 +++++++++++++++---- 3 files changed, 48 insertions(+), 13 deletions(-) diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 9c70e42..0de5698 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -85,7 +85,7 @@ | LAN access when internet is down (lan_resolver) | agent | **IMPLEMENTED** | — | Never drilled as a customer experience ("net down — can I reach my photos?") → R-19 | | Phone photo backup | immich (classified) | **PROVEN-LIVE** | 6D end-to-end restore proof | | | Documents/OCR | paperless-ngx (classified) | **PROVEN-LIVE** | `CAMPAIGN-6C` 4P-1 (deploy paperless-ngx, ingest 3 docs via consume flow, OCR + PDF/A ~90s) + 4P-2/3/5 | Consume-folder ingestion awkward without SMB → R-7 | -| Files from Windows Explorer / Mac Finder (SMB **server**) | controller v0.144.0 + `felhom-samba:1.0.0` | **PROVEN-LIVE** | `felhom-controller` REPORT.md (v0.144.0) + `controller/sharing.md`; transport verdict `audits/SPIKE-lan-discovery-2026-07-18.md` | „Megosztás" page: enable + one household password + shares (new folder or picked existing, per-share read-only). Fourth protected infra stack (host-net, smbd+**nmbd**+wsdd). Live on demo: 445 reachable, NetBIOS `FELHOM` resolves, write/read byte-compare PASS, write to a read-only share REFUSED, SMB writes land as uid 1000. **Explorer leg PASSED 2026-07-18 (Viktor):** Network → FELHOM → both shares open; a real Explorer *save* into `dokumentumok` landed owned uid 1000, and a write into the read-only `filmek` was refused by Windows with the folder left untouched. KNOWN GAP: share data is backup-**classified** (mandatory/optional per share) but not yet in a live tier-2/offsite RUN — reported design fork, see REPORT §4 | +| Files from Windows Explorer / Mac Finder (SMB **server**) | controller v0.145.0 + `felhom-samba:1.0.0` | **PROVEN-LIVE** | `felhom-controller` REPORT.md (v0.144.0) + `controller/sharing.md`; transport verdict `audits/SPIKE-lan-discovery-2026-07-18.md` | „Megosztás" page: enable + one household password + shares (new folder or picked existing, per-share read-only). Fourth protected infra stack (host-net, smbd+**nmbd**+wsdd). Live on demo: 445 reachable, NetBIOS `FELHOM` resolves, write/read byte-compare PASS, write to a read-only share REFUSED, SMB writes land as uid 1000. **Explorer leg PASSED 2026-07-18 (Viktor):** Network → FELHOM → both shares open; a real Explorer *save* into `dokumentumok` landed owned uid 1000, and a write into the read-only `filmek` was refused by Windows with the folder left untouched. Share data now RIDES BOTH BACKUP TIERS (R-7b, controller v0.145.0, Model B′ sibling shares source): tier-2 cross-drive legs + an offsite `_shares` restic snapshot carrying the share definitions and the credential copy, with a „Megosztások" restore. Tier-2 leg + samba liveness alert PROVEN-LIVE on demo; the offsite leg and the restore round-trip are unit-covered but NOT yet live-exercised (the demo box has no offsite target) | | Media to TV via DLNA | — | **MISSING** | — | Jellyfin app exists; DLNA/SSDP unvalidated → R-6, R-8 | | File access via browser | FileBrowser (infra app, auto-mount sync) | **IMPLEMENTED** | FileBrowser runs healthy + userdata-bound (`storage-lifecycle-acceptance-2026-06-15`, `CAMPAIGN-3`) | Actual browse/download **through FileBrowser** is exercised in no doc. (Cited `CAMPAIGN-2` T-PAGE-ALL renders only the controller dashboard pages, not FileBrowser.) Demoted | | Forgot dashboard password → instant reset code | controller v0.123, hub | **PROVEN-LIVE** | `DRILL-day0-take2-2026-07-12` F-15 (live re-run of the exact failure path: hash applied 1s after request, code accepted first try) | | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index d86c002..808d4e8 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -32,7 +32,7 @@ | R-5 | Hub: offsite storage visibility — RESTIC box aggregate (v0.64.0) + PBS DR datastore (v0.65.0), each with fill + operator alert; Offsite tabs + dual dashboard gauges | M | **SHIPPED (hub v0.64.0 + v0.65.0 + tenantsync v1.2.0, 2026-07-17)** | Restic: `monitor.OffsiteBoxChecker` (throttled Hetzner GET; fill 80/90% + oversub 2.0×; scope `"pool-box"`). PBS DR (v0.65.0, Option A): read-only `usage` op on the ep0 `felhom-tenantsync` forced command (`df`, no mutation) → `monitor.PBSDRBoxChecker` (fill 80/90%, states ok/unavailable/degraded, scope `"pbsdr-box"`); `/offsite` Restic/PBS-DR tabs + two gauges; hub deploy ⟂ ep0 update (graceful "n/a"). **Both data pipelines PROVEN-LIVE** (restic `0.2% full`; PBS `19.1% full, 7.1 GB of 37.2 GB` after ep0 → v1.2.0). Thresholds pending Viktor (named keys; separate restic/PBS pairs). OPEN: UI render unit-only (password-gated), alert emails unit + red-proof only (both nominal). See hub v0.64.0 + v0.65.0 REPORTs | | R-6 | **Spike: LAN service discovery from the guest** — SSDP multicast (UDP 1900, DLNA), WSD (Windows discovery), mDNS; host-network vs macvlan; is the customer LXC LAN-bridged in appliance deployments? | M | **spiked (2026-07-18)** | **VERDICT: appliance guest IS LAN-bridged (own DHCP lease on the household /24); multicast discovery works ONLY in the guest netns — guest-direct or Docker `--network host` (SSDP/mDNS/WSD all PASS both ways); the default docker bridge is categorically DEAF to LAN multicast (WSD/mDNS RX FAIL, unicast-publish PASS). Real samba+wsdd on host-net → Windows 11 ProbeMatch + FELHOM-SPIKE renders in Explorer + 445 + authenticated SMB round-trip all PASS; real SSDP `MediaServer:1` advert reaches both LAN clients. → R-7 SMB stack MUST be host-network LAN-bound; R-8 Jellyfin-DLNA plausible if host-network. Caveat: `vmbr0 multicast_snooping=1` worked only because the household router is a live querier — customer LANs w/ snooping+no-querier, and Peti's BYO bridge, are UNTESTED gaps.** **S4b (human leg, the sharpest finding): wsdd makes the box VISIBLE but the Explorer double-click FAILS `0x80070035` — WSD gives no name resolution; the flat `\\FELHOM-SPIKE` resolved by no path. Adding `nmbd` (NetBIOS) fixed it live (flat name resolves + mounts). → R-7 needs smbd+wsdd+nmbd (+avahi/.local for modern clients), not wsdd alone.** Doc: `audits/SPIKE-lan-discovery-2026-07-18.md`. | | R-7 | SMB **server** share — LAN file sharing as an embedded controller feature („Megosztás") | L | **SHIPPED slice 1 (controller v0.144.0 + `felhom-samba:1.0.0`, 2026-07-18)** | Flips map row *Files from Windows Explorer / Mac Finder (SMB server)* MISSING → **IMPLEMENTED**. Delivered as the FOURTH protected infra stack (host-net, smbd+**nmbd**+wsdd, `bind interfaces only = lo eth0`), NOT a catalog app. „Megosztás" → „Hálózati megosztás": enable + ONE household password (STDIN→smbpasswd, never persisted — only `user_set`), shares table, create flow (new folder under `/shares/` or an existing folder via a guarded picker). Picker guard = absolute → EvalSymlinks → containment in a registered LIVE root → deny-listed system subtree (derived from `ProtectedHDDPaths`, provably a subset) → is-a-dir, with UNIFORM refusals (no filesystem oracle). Per-share read-only enforced in smb.conf AND as a `:ro` bind. `force user = felhom` → SMB writes land as uid 1000. Live-validated on demo through the real endpoints + a Win11 workstation (445 True, NetBIOS `FELHOM` resolves, write/read byte-compare PASS, **write to a read-only share REFUSED**, WSD ProbeMatch). **Explorer leg PASSED 2026-07-18 (Viktor):** both shares open from the Network view; an Explorer save into the writable share landed as uid 1000 and a write into the read-only share was refused, folder untouched. Slice 1 is fully PROVEN-LIVE. Docs: `controller/sharing.md`; ship report `felhom-controller/REPORT.md`. **Slice 2 remainder:** avahi/`.local` for modern+Apple clients, curated app-folder presets (one-click paperless `consume/`), per-share users + guest access, recycle bin. **Backup gap split out → R-7b.** | -| R-7b | **Share data is backup-CLASSIFIED but not backed up** (design fork reported by R-7 slice 1, not improvised). `ClassifiedBinds("samba")` correctly returns mandatory/optional per share, but no live run consumes it: `backup.RunTier2` short-circuits on `os.Stat(unitDir)` BEFORE calling `GetStackClassifiedBinds`, and the offsite runner enumerates `settings.GetOffboxApps()`. Both engines are **recovery-unit shaped**, which a share-only infra stack has not. Decide the model: give samba a synthetic recovery unit, teach the tier engines a unit-less class-driven source, or keep shares tier-2-only. | M | **idea (needs Viktor ruling)** | Origin: R-7 slice 1 (`felhom-controller` REPORT §4). **Customer-facing consequence: until this lands, files dropped on a share are NOT in any backup — do not promise otherwise.** [R4] intended new shares to default to the full product backup promise (mandatory = offsite + tier-2); the classification honours that, the execution does not yet. Also blocks a clean "every SMB-writable path has a backup class" claim in the R-7 docs | +| R-7b | **Share backup EXECUTION** — put share data into the live tier-2 + offsite runs (the design fork reported by R-7 slice 1) | M | **SHIPPED (controller v0.145.0, 2026-07-18)** | **Viktor's ruling: Model B′ — a SIBLING shares source.** New, additive job/leg code reusing the proven primitives (tier-2 mirror seam, restic wrappers, soft-quota/enlargement gate, status recorders) while leaving **every per-app engine path byte-identical** — NOT a synthetic recovery unit (breaks on multi-drive shares, wraps 1 KB of JSON in dump machinery) and NOT engine-loop surgery. The B′ invariant is enforced by test in both tiers, red-proofed. Tier 2 → `RunSharesTier2` (legs grouped by SOURCE drive → `backups/secondary/_shares//`, payload at `_payload/`, layout marker LAST). Tier 3 → `runOffboxSharesLeg`: ONE extra `restic backup --tag felhom-offbox --tag _shares` placed after the app loop and BEFORE retention, so `forget --group-by host,tags` covers the new group with no flag change; a quota-blocked push degrades to the **manifest only, never to nothing**. Restore → „Megosztások" on `/backups/restore`: scratch, then a missing-only merge whose every destination is PREFIX-ASSERTED against live storage roots, definitions merged existing-wins, then `ReconcileSamba`, then the credential. The **payload** (`_shares-manifest.json` + a best-effort secret-bearing `passdb.tar`) is what makes DR return files + configuration + password rather than loose bytes. **Fold-in: samba joins the liveness set** — `EffectiveProtected` adds the CONTAINER `felhom-samba` exactly while sharing is on. **Live-validated on demo:** real `/api/backup/tier2` trigger → `_shares` tree + marker + payload on the cross-drive target, mirrored file md5-identical, payload 0600 preserved; samba stopped → `health_critical` pushed and hub-accepted (200) → self-healed. **NOT live-exercised (no offsite target on the demo box): the restic `_shares` leg and the restore round-trip** — unit-covered + red-proofed; needs a box with an escrowed offsite target (Viktor). Findings: the reserved-name assumption was FALSE (`nbNameRe` accepted „_shares" as a share name — now refused); the alert/e-mail pipeline needed NO change and adds no new event type. Docs: `controller/sharing.md`; ship report `felhom-controller/REPORT.md`. | | R-8 | DLNA (**gate input now exists — R-6 spiked 2026-07-18: SSDP reaches LAN clients from host-net**): validate Jellyfin's built-in DLNA server first; only add minidlna to the catalog if Jellyfin-DLNA fails | S | idea (unblocked) | Don't add catalog weight before proving the cheap path. **R-6 confirmed the cheap path is physically viable — Jellyfin DLNA must run host-network (same multicast constraint as R-7)** | | R-9 | Uninstaller trio (from 07-15 Peti session): cluster-aware `felhom_guests` guard (node-local `pct list` deletes cluster-wide pveum objects); saferemove detection + time estimate + opt-in `--quick-remove` (never mutate `storage.cfg`); smarter `restore_storage` default for BYO clusters (shared storage, not local-lvm) | M | idea | Second item's rejected alternative (temp-disable-and-restore) stays rejected — crash window silently downgrades cluster wipe policy | | R-10 | T-6E-1: DB-dump dir-fsync asymmetry (LOW, confirmed in 6E) | XS | idea | One-line hardening; batch with the next controller task | diff --git a/documentation/controller/sharing.md b/documentation/controller/sharing.md index ca579c0..37b228f 100644 --- a/documentation/controller/sharing.md +++ b/documentation/controller/sharing.md @@ -142,21 +142,56 @@ The new-folder flow uses a separate, strictly tighter check (`sharingResolveStor `smb.conf` and the passdb are config, not customer data, and are never classified. -> **KNOWN GAP — share data is classified but not yet backed up.** Making this seam correct does not by -> itself put share data into a live run: `backup.RunTier2` short-circuits on a missing *recovery unit* -> before it ever calls `GetStackClassifiedBinds`, and the offsite runner enumerates -> `settings.GetOffboxApps()`. Both engines are recovery-unit shaped, which a share-only infra stack -> has not. Teaching them about one is a structural change, so it was reported as a design fork rather -> than improvised inside the engines. **Until it is resolved, do not tell a customer that files -> dropped on a share are backed up.** See `felhom-controller/REPORT.md` §4. +## Backup EXECUTION — the sibling shares source (R-7b, controller v0.145.0) + +The classification above is now *executed*. It did not get there through `ClassifiedBinds` — the +engines stayed recovery-unit shaped and Model B′ deliberately left every per-app path byte-identical. +Instead `internal/backup` runs a **sibling shares source** off the same registry: + +| Tier | Entry point | What it writes | +|---|---|---| +| 2 (cross-drive) | `RunSharesTier2`, after the per-stack loop in the same run | `backups/secondary/_shares//` per source drive, `_payload/`, `.felhom-tier2-layout` written **LAST** | +| 3 (offsite) | `runOffboxSharesLeg`, after the per-app loop and **before** retention | ONE `restic backup --tag felhom-offbox --tag _shares` = payload dir + every MANDATORY share | +| restore | „Megosztások" on `/backups/restore` | scratch → missing-only merge, each destination prefix-asserted against LIVE storage roots | + +**The payload is the point.** `_shares-manifest.json` carries the share *definitions* (sorted → +byte-deterministic, so an unchanged registry gives the mirror nothing to rewrite) and `passdb.tar` +carries the household credential, best-effort. Without it a restore hands the customer their files +back and an empty „Megosztás" page. The credential copy is **secret-bearing**: 0600, encrypted inside +restic, never logged at INFO and never in a report or a committed file. + +**Degradation contract.** A quota-blocked offsite push falls back to the **manifest only, never to +nothing** — definitions protection must not regress because the files stopped fitting. + +**Restore semantics.** Files merge missing-only (a live file is never overwritten). Definitions merge +**existing-wins**: a restore must never silently flip a live share's read-only or „Felhőmentés" +setting, and anything skipped is named in the flash. Then `ReconcileSamba` re-renders `smb.conf`, then +the credential goes back into the named volume. + +**`_shares` is a reserved key** (restic tag, dest root, status record). `ValidateSMBShareName` refuses +a leading underscore — note this was a REAL gap, not a formality: `nbNameRe` begins with +`[A-Za-z0-9_]`, so „_shares" was previously an accepted share name. The key never reaches a customer +surface; `backup.DisplayStackName` maps it to „Megosztások" at the notification and prose boundaries, +while the persisted set, the tag and the paths keep the raw key. + +> **Live-validation status (2026-07-18).** The tier-2 leg is PROVEN-LIVE on the demo box through the +> real `/api/backup/tier2` endpoint (tree + marker + payload on the cross-drive target, mirrored file +> md5-identical, payload 0600 preserved). **The offsite `_shares` leg and the restore round-trip are +> unit-covered and red-proofed but NOT yet live-exercised** — the demo box has no offsite target +> configured. They need a box with an escrowed offsite repo. ## Operator notes - samba is protected in **code** (`config.alwaysProtectedStacks`), because `cfg.Stacks.Protected` comes - from the golden-generated `controller.yaml` and predates the feature. Consequence: - `monitor.EffectiveProtected` does **not** monitor samba liveness — a dead samba container raises no - "protected container missing" issue. Deliberate for slice 1 (the safe direction: no false alarms - while the feature is off). + from the golden-generated `controller.yaml` and predates the feature. **Since v0.145.0 (R-7b) it is + also monitored for liveness:** `monitor.EffectiveProtected` takes the SMB settings and adds the + container `infra.SambaContainerName` exactly while sharing is ON, so a dead sharing service raises + the standard protected-container issue → `health_critical` → alert → Hungarian degradation e-mail, + while a box that never enabled sharing stays quiet (both directions unit-tested and red-proofed). + Note the CONTAINER name (`felhom-samba`) is deliberately not the stack name (`samba`) — the health + check docker-inspects container names. PROVEN-LIVE 2026-07-18: stopping the container produced + `Protected container not running: felhom-samba`, status `fail`, and a hub-accepted `health_critical` + event; the next tick's `EnsureBaseStack` self-healed it. - Discovery depends on the household LAN. The R-6 spike's §7d caveat applies: a customer network with IGMP snooping enabled and **no querier** could prune the discovery multicast, and a BYO topology may put the guest on a bridge with no L2 path to the household LAN. The page carries an honest hint that