diff --git a/documentation/audits/SPIKE-restic-snapshot-shape-2026-07-14.md b/documentation/audits/SPIKE-restic-snapshot-shape-2026-07-14.md new file mode 100644 index 0000000..f634942 --- /dev/null +++ b/documentation/audits/SPIKE-restic-snapshot-shape-2026-07-14.md @@ -0,0 +1,299 @@ +# SPIKE — restic snapshot shape, retention grouping, stats semantics (SP-1/SP-2/SP-3) — 2026-07-14 + +**Class:** spike (findings only — no production code, no config, no schema edits; the only write +outside scratch is this doc). Gates the **Task 3a** (offsite tier policy engine) spec, which is +BLOCKED until these three verdicts land (spike-first gate). + +## Verdict up front + +1. **SP-1 (§9 quota accounting):** `restic stats --json` (no mode = restore-size over ALL + snapshots) **multiplies** — it reports Σ over every retained snapshot of that snapshot's logical + size (N=5 → exactly 5× one snapshot). `stats --mode raw-data --json` reports the **actual + deduplicated+compressed repo bytes** (one physical copy; grows only by the inter-snapshot delta). + Quota MUST switch to `--mode raw-data`. Architecture-doc assumption **holds**. +2. **SP-2 (§6 forget flag):** the production `forget` call with **default grouping (host,paths) + STRANDS** old-shape snapshots — the unit-only group keeps its own full 7-daily/4-weekly/6-monthly + floor **forever** because no new snapshot ever enters it. `--group-by host,tags` collapses old + + new shape into one ``-keyed group so the old shape **ages out naturally**, and keeps + different apps in **separate groups**. Add `--group-by host,tags`. Architecture-doc assumption + **holds**. +3. **SP-3 (§7 restore):** multi-path restore reconstructs **absolute source paths under `--target`**; + `--include ` gives a clean **unit-only** restore; **uid/gid/mode incl. the + setgid bit are preserved** on a root restore; `snapshots --json` `paths` lists all captured paths + (no manifest change needed). **One contradiction:** ⚠️ **ARCHITECTURE IMPACT** — restic 0.14.0 + does **NOT** error on a nonexistent source path (§2 decision 5 / §2.5 premise); it skips with a + warning, **exits 0**, and silently creates a partial snapshot. The stat-filter is still required — + but for the opposite reason the doc states. + +--- + +## 0. Baselines (live-verified at session start) + +| Item | Value | How verified | +|---|---|---| +| felhom.eu `main` head (incl. architecture-doc + §10 confirmations) | **`eff107d3`** (`eff107d` "confirmations"; doc added in `82280b1`) | `git ls-remote origin` == local HEAD | +| felhom-controller live image on guest 9201 | **`gitea.dooplex.hu/admin/felhom-controller:0.132.0`** (Up, healthy) | `docker ps` on 9201 | +| restic version (inside the felhom-controller container — the production binary) | **`restic 0.14.0 compiled with go1.19.8 on linux/amd64`** | `docker exec felhom-controller restic version` | +| Scratch mount | `/mnt/sys_drive` — dedicated device (`pve-vm-9201-disk-2`, 50 G, 47 G free, own fs, NOT guest rootfs) | `df -h` inside container; container sees `/mnt` via `rslave` bind | + +**Production invocation shapes mirrored (read-only from `internal/backup/offbox.go`, v0.132.0):** +backup `backup --tag felhom-offbox --tag ` (`:575`) · forget +`forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune` (`:595`) · stats `stats --json` +(no mode) (`:700`) · restore `restore latest --tag --target ` (`:732`). + +**Semantics assumption (stated explicitly):** restic's stats / forget / restore behaviors are +**repo-format-level, not backend-level**. A local directory repo therefore reproduces the exact +Storage-Box-over-SFTP semantics under test; SFTP would add latency and failure modes but change none +of the accounting/grouping/restore-shape answers. All timings below are LOCAL-repo and explicitly +**not** SFTP-representative. The scratch repo used restic repo-format v2 (0.14 default; compression +`auto`) — test data is `/dev/urandom` (incompressible) so raw-data bytes reflect real storage cost. + +**Scratch discipline:** everything under `/mnt/sys_drive/felhom-spike-scratch/` (repo, `repo2`, +synthetic urandom sources, restore targets, throwaway 64-hex password in `pw`). No production path +touched: the offbox key/repo_password/known_hosts live under `DataDir/offbox` (a docker volume), not +under `/mnt`; the configured Hetzner target was never contacted. Cleanup proof in §Cleanup. + +--- + +## Part 1 — SP-1: stats semantics (quota accounting, §9) + +**Question:** in a repo where a large path is captured in N snapshots, what does (a) `stats --json` +(production call, restore-size, all snapshots) report vs (b) `stats --mode raw-data --json` vs +(c) `stats latest --json`? + +**Protocol:** fresh repo; `unit/` = 5 files × 1 MiB (5,246,976 B) + `bigdata/` = 4 files × 64 MiB +(268,439,552 B, urandom); **5 snapshots of the same unchanged paths** +`backup /unit /bigdata --tag felhom-offbox --tag spiketest`; record the three stats; then +append 20 MiB urandom to `bigdata/`, take a 6th snapshot, re-record. + +**Unchanged-run behavior (itself a finding):** restic 0.14.0 creates a **new snapshot every run** +even when nothing changed — 5 distinct IDs (`9f66cd47, 3df2edcd, 1195267a, 1324877b, 528a324b`), +each `Added to the repository: 0 B` after the first. It does NOT skip unchanged runs. + +**Transcript (verbatim JSON):** + +``` +# N=5 +$ restic stats --json +{"total_size":1368391680,"total_file_count":70,"snapshots_count":5} +$ restic stats --mode raw-data --json +{"total_size":273700446,"total_file_count":0,"total_blob_count":185,"snapshots_count":5} +$ restic stats latest --json +{"total_size":273678336,"total_file_count":14,"snapshots_count":1} + +# after +20 MiB delta → N=6 (backup #6: "Added to the repository: 20.015 MiB") +$ restic stats --json +{"total_size":1663041536,"total_file_count":85,"snapshots_count":6} +$ restic stats --mode raw-data --json +{"total_size":294681014,"total_file_count":0,"total_blob_count":203,"snapshots_count":6} +$ restic stats latest --json +{"total_size":294649856,"total_file_count":15,"snapshots_count":1} +``` + +**Arithmetic:** + +| | `stats --json` (default) | `stats --mode raw-data` | `stats latest` | +|---|---|---|---| +| N=5 | 1,368,391,680 = **5 × 273,678,336** | 273,700,446 (≈261 MiB, one physical copy) | 273,678,336 (one snapshot) | +| N=6 | 1,663,041,536 = 5×273,678,336 + 294,649,856 | 294,681,014 (**+20,980,568 B ≈ 20 MiB** vs N=5) | 294,649,856 | + +- Default mode `total_size` = **Σ over all snapshots of each snapshot's restore size** + (`total_file_count` 70 = 5×14). **Per-snapshot multiplication CONFIRMED.** +- raw-data `total_size` at N=5 ≈ the single physical copy (261 MiB); at N=6 it grew by **exactly the + 20 MiB delta**, not by another full copy. **Cross-snapshot dedup, no multiplication CONFIRMED.** + +**Verdict:** +- `stats --json` (default/restore-size) counts **the logical restore size summed across every + retained snapshot** — for ~17 retained snapshots of a mandatory library it reports ~17× the disk. +- `stats --mode raw-data` counts **the actual deduplicated+compressed bytes the repo occupies** — + what the customer's Storage Box really fills. +- **Quota accounting for the new shape should read `--mode raw-data` because it is the only mode + whose value tracks real Storage Box fill; the default mode multiplies by the retained-snapshot + count and would trip the ≥100% gate at ~17× reality.** (Confirms §9 direction / decision #6.) + +--- + +## Part 2 — SP-2: retention grouping across the shape change (§6) + +**Question:** when an app's path set changes (unit-only → unit+userdata), does the production +`forget` strand old-shape snapshots in a permanently-retained group? Does `--group-by host,tags` +fix it without merging different apps? + +**Protocol (fresh `repo2`; `--time` backdating):** appA old shape = 12 daily `backup ` +snapshots tagged `felhom-offbox,appA`, 2026-06-19…06-30; appA new shape = 12 daily +`backup ` same tags, 2026-07-03…07-14; appB = 3 `backup ` tagged +`felhom-offbox,appB`, 2026-07-12…14 (separation control). Then the production policy +`forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --dry-run` with default grouping, and again +with `--group-by host,tags`. **Dry-run only — no snapshot destroyed.** + +**Transcript (A) — DEFAULT grouping (host,paths), three groups:** + +``` +Group [paths=p2unit] (OLD shape appA) → keep 8, remove 4 + keep: 06-21(weekly) 06-24 06-25 06-26 06-27 06-28(d+w) 06-29 06-30(d+w+monthly) + remove: 06-19 06-20 06-22 06-23 +Group [paths=p2big,p2unit] (NEW shape appA) → keep 8, remove 4 + keep: 07-05(weekly) 07-08 07-09 07-10 07-11 07-12(d+w) 07-13 07-14(d+w+monthly) + remove: 07-03 07-04 07-06 07-07 +Group [paths=p2unitB] (appB) → keep 3, remove 0 +``` + +**Transcript (B) — `--group-by host,tags`, two groups:** + +``` +Group [tags=appA,felhom-offbox] (OLD+NEW appA in ONE group) → keep 10, remove 14 + keep: 06-28(weekly) 06-30(monthly) ← only 2 old-shape survive, as legit historical reps + 07-05(weekly) 07-08 07-09 07-10 07-11 07-12(d+w) 07-13 07-14(d+w+monthly) + remove: 06-19 06-20 06-21 06-22 06-23 06-24 06-25 06-26 06-27 06-29 ← 10 of 12 old-shape gone + 07-03 07-04 07-06 07-07 +Group [tags=appB,felhom-offbox] (appB) → keep 3, remove 0 ← separate group +``` + +**Interpretation (per the guardrail):** +- **Default grouping strands: YES.** The old-shape `[p2unit]` group retains **8** June snapshots + under its own independent 7d/4w/6m floor. Because every subsequent appA push carries the new + `[p2unit,p2big]` path set, **no new snapshot ever enters the `[p2unit]` group** — nothing pushes + those 8 out of the keep-daily-7 / keep-weekly-4 / keep-monthly-6 windows. They are retained + **forever** → permanent zombie quota cost. This is the stranding signature: *a separate old-shape + group keeping its own full floor with no inflow to age it*. +- **`--group-by host,tags` ages old shape correctly: YES.** The `` tag set is identical + across old and new shape, so all 24 appA snapshots share **one** group. Only **06-28 (weekly)** + and **06-30 (monthly)** survive — as genuine historical weekly/monthly representatives (correct + AGING, not stranding); the other 10 old-shape snapshots are removed immediately, and as new daily + pushes accumulate 06-28/06-30 fall out of the weekly-4/monthly-6 windows and age out too. +- **Per-app group separation preserved under tags grouping: YES.** appB + (`tags=appB,felhom-offbox`) forms its **own** group — the tag LIST is the group key (restic sorts + it; input `felhom-offbox,appA` displays as `appA,felhom-offbox`). appA and appB never merge. + +**Verdict:** default grouping strands (evidence: the 8-snapshot orphan `[p2unit]` group with no +inflow); `--group-by host,tags` ages the old shape correctly (10 of 12 old-shape removed at once, +the 2 survivors are legit reps); per-app separation preserved (appB stays its own group). +**The 3a forget invocation should be +`forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --group-by host,tags --prune`.** + +--- + +## Part 3 — SP-3: multi-path restore shape, selectivity, fidelity, missing-source (§7) + +Reused the Part-1 repo; added a fidelity fixture inside `bigdata/` and took a 7th snapshot +(`8ce4ec93`, tag `spiketest`), which became `latest`. + +**3.1 Full-restore tree shape.** `restore latest --tag spiketest --target ` → + +``` +/mnt/sys_drive/felhom-spike-scratch/unit/u1..u5.bin +/mnt/sys_drive/felhom-spike-scratch/bigdata/{b1..b4,b5_delta,owned_1000}.bin, setgid_dir/ +``` + +**Verdict (§7.1):** absolute source paths are reconstructed verbatim under `--target` +(`/`) — the scratch-relocation design must account for the full absolute +prefix under the chosen scratch dir. + +**3.2 Unit-only selective restore.** Four `--include` forms, each into its own target: + +| `--include` form | unit files | bigdata files | +|---|---|---| +| `/mnt/sys_drive/felhom-spike-scratch/unit` (absolute source path) | 5 | 0 | +| `/mnt/sys_drive/felhom-spike-scratch/unit/*` | 5 | 0 | +| `*/unit/*` | 5 | 0 | +| `unit` (bare component) | 5 | 0 | + +All four isolate the unit tree. The bare `unit` form works only because restic matches a path +**component** anywhere and nothing else is named `unit`; it would also match a stray `unit` file +under userdata. **Verdict (§7.2): use `--include `** (the exact `src` restic was +handed at backup, i.e. `backups/primary/`) — unambiguous, exact-subtree, no false matches. + +**3.3 Ownership/mode fidelity (root restore).** Fixture before backup vs after restore: + +``` +source : owned_1000.bin uid=1000 gid=1000 mode=644 | setgid_dir uid=0 gid=1000 mode=2775 +restored: owned_1000.bin uid=1000 gid=1000 mode=644 | setgid_dir uid=0 gid=1000 mode=2775 +``` + +**Verdict (§5 fidelity row):** restic root-restore preserves uid/gid, mode, **and the setgid bit +(2775)** exactly — correct for the `SharedContentGID=1000` / mode-2775 userdata convention. + +**3.4 Nonexistent source path.** ⚠️ **ARCHITECTURE IMPACT.** +`backup /unit /does-not-exist --tag felhom-offbox --tag spiketest2`: + +``` +/mnt/sys_drive/felhom-spike-scratch/does-not-exist does not exist, skipping +using parent snapshot 8ce4ec93 +processed 5 files, 5.000 MiB in 0:00 +snapshot 98bc38ef saved +[backup exit=0] +$ restic snapshots --tag spiketest2 --json +[{... "paths":["/mnt/sys_drive/felhom-spike-scratch/unit"], "tags":["felhom-offbox","spiketest2"], + "short_id":"98bc38ef"}] +``` + +restic 0.14.0 **does NOT error** on a nonexistent source path. It prints "does not exist, skipping" +to stderr, **exits 0**, and **silently creates a snapshot containing only the paths that existed**. +This **contradicts** §2 decision 5 / §2.5, which state "restic errors on a nonexistent source path, +so the capture set is stat-filtered before invocation." The stat-filter decision itself is still +correct — but its stated justification is empirically false, and the real justification is +**stronger/opposite**: without the pre-invocation stat-filter, a missing mandatory path produces a +**partial snapshot with a success exit code and no loud signal**, so the §2.5 "loud WARN" would +never fire off a restic error — the controller must stat-filter to *detect* the absence itself. +**Flagged; not redesigned here (rule 6).** 3a must not rely on a nonzero restic exit to catch a +missing mandatory path. + +**3.5 Shape introspection.** `snapshots latest --tag spiketest --json` → +`"paths":["/mnt/sys_drive/felhom-spike-scratch/bigdata","/mnt/sys_drive/felhom-spike-scratch/unit"]` +— the `paths` field lists **all** captured paths. **Verdict (§6): the snapshot's own `paths` +metadata is an accurate shape record; the "no recovery-unit manifest change / no SchemaVersion bump" +claim holds.** + +**3.6 Timing.** Full restore of the ~276 MB set (294,744,064 B) = **1.25 s** wall-clock — LOCAL repo, +**NOT** SFTP-representative; only a sanity check that the shape is nowhere near the 30-min handler +ceiling (§12). SFTP over residential upstream will dominate; 3a should size the ceiling from a real +SFTP restore, not this number. + +--- + +## Decision inputs for Task 3a + +| SP | Verdict | Resolves | Architecture-doc status | +|---|---|---|---| +| **SP-1** | Quota must read `restic stats --mode raw-data --json` (dedup+compressed real bytes). Default `stats --json` = Σ per-snapshot restore size = N× multiplication (N=5 → exactly 5×). | **§9** accounting switch / decision #6 | Assumption **HOLDS** — direction confirmed | +| **SP-2** | 3a forget = `forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --group-by host,tags --prune`. Default (host,paths) strands 8 old-shape snapshots in an orphan group forever; tags-grouping ages them out and keeps apps separate. | **§6** retention flag | Assumption **HOLDS** — stranding proven, fix proven | +| **SP-3.1/3.2** | Restore reconstructs `/`; unit-only = `restore … --include `. | **§7.1 / §7.2** restore shape + unit-only default | Assumption **HOLDS** | +| **SP-3.3** | uid/gid/mode + setgid bit preserved on root restore. | **§5** restic fidelity row | Assumption **HOLDS** | +| **SP-3.5** | `snapshots --json` `paths` lists all captured paths — accurate shape record. | **§6** "no manifest change" claim | Assumption **HOLDS** | +| **SP-3.4** | ⚠️ **ARCHITECTURE IMPACT** — restic 0.14.0 does NOT error on a missing source path: skips-with-warning, **exit 0**, silent partial snapshot. §2 decision 5 / §2.5 premise ("restic errors") is FALSE. Stat-filter still required, but because restic gives **no** error signal (not because it errors). | **§2 decision 5 / §2.5** stat-filter justification | **CONTRADICTED** (decision survives; rationale must be rewritten; 3a must not depend on a nonzero exit) | + +--- + +## Cleanup (part of the deliverable) + +`rm -rf /mnt/sys_drive/felhom-spike-scratch` executed inside the container. Proof — `/mnt/sys_drive` +after cleanup (scratch gone; only the pre-existing namespace + `lost+found` remain), and free space +returned to the pre-spike baseline: + +``` +$ ls -la /mnt/sys_drive +drwxr-xr-x felhom-data +drwx------ lost+found +$ df -h /mnt/sys_drive # 2.8M used / 47G free — identical to session start +/dev/mapper/pve-vm--9201--disk--2 50G 2.8M 47G 1% /mnt/sys_drive +``` + +No secrets written to this doc or the commit (the throwaway repo password lived only in scratch and +was removed with it). No production path modified. + +## Observations (documented, not acted on) + +- restic 0.14.0's "new snapshot every run even when unchanged" means an offbox schedule that runs + and finds no changes still adds a snapshot to the retention window — expected, and exactly why the + keep-daily/weekly/monthly policy exists; noted for the 3a quota-growth mental model. +- `stats --mode raw-data` returns `total_file_count:0` (it counts blobs, `total_blob_count`, not + files) — a 3a consumer must read `total_size`, not `total_file_count`, from the raw-data JSON. +- The pinned restic is **0.14.0** (2022-era). The exit-code-on-missing-path behavior (SP-3.4) + changed in later restic releases (newer versions exit 3 on skipped paths); if the container's + restic is ever bumped, the SP-3.4 assumption must be re-checked — but the stat-filter design is + correct under both behaviors. + +--- + +*Baselines re-verified live at session start. Findings-doc commit hash on felhom.eu `main`: recorded +at commit time (see the commit that adds this file).*