felhom.eu/documentation/audits/SPIKE-restic-snapshot-shape-2026-07-14.md

SPIKE — restic snapshot shape, retention grouping, stats semantics (SP-1/SP-2/SP-3) — 2026-07-14

Class: spike (findings only — no production code, no config, no schema edits; the only write outside scratch is this doc). Gates the Task 3a (offsite tier policy engine) spec, which is BLOCKED until these three verdicts land (spike-first gate).

Verdict up front

1. SP-1 (§9 quota accounting): restic stats --json (no mode = restore-size over ALL snapshots) multiplies — it reports Σ over every retained snapshot of that snapshot's logical size (N=5 → exactly 5× one snapshot). stats --mode raw-data --json reports the actual deduplicated+compressed repo bytes (one physical copy; grows only by the inter-snapshot delta). Quota MUST switch to --mode raw-data. Architecture-doc assumption holds.
2. SP-2 (§6 forget flag): the production forget call with default grouping (host,paths) STRANDS old-shape snapshots — the unit-only group keeps its own full 7-daily/4-weekly/6-monthly floor forever because no new snapshot ever enters it. --group-by host,tags collapses old + new shape into one <stack>-keyed group so the old shape ages out naturally, and keeps different apps in separate groups. Add --group-by host,tags. Architecture-doc assumption holds.
3. SP-3 (§7 restore): multi-path restore reconstructs absolute source paths under --target; --include <absolute-unit-path> gives a clean unit-only restore; uid/gid/mode incl. the setgid bit are preserved on a root restore; snapshots --json paths lists all captured paths (no manifest change needed). One contradiction: ⚠️ ARCHITECTURE IMPACT — restic 0.14.0 does NOT error on a nonexistent source path (§2 decision 5 / §2.5 premise); it skips with a warning, exits 0, and silently creates a partial snapshot. The stat-filter is still required — but for the opposite reason the doc states.

---

0. Baselines (live-verified at session start)

Item	Value	How verified
felhom.eu main head (incl. architecture-doc + §10 confirmations)	eff107d3 (eff107d "confirmations"; doc added in 82280b1)	git ls-remote origin == local HEAD
felhom-controller live image on guest 9201	gitea.dooplex.hu/admin/felhom-controller:0.132.0 (Up, healthy)	docker ps on 9201
restic version (inside the felhom-controller container — the production binary)	restic 0.14.0 compiled with go1.19.8 on linux/amd64	docker exec felhom-controller restic version
Scratch mount	/mnt/sys_drive — dedicated device (pve-vm-9201-disk-2, 50 G, 47 G free, own fs, NOT guest rootfs)	df -h inside container; container sees /mnt via rslave bind

Production invocation shapes mirrored (read-only from internal/backup/offbox.go, v0.132.0): backup backup --tag felhom-offbox --tag <stack> <src> (:575) · forget forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune (:595) · stats stats --json (no mode) (:700) · restore restore latest --tag <stack> --target <dir> (:732).

Semantics assumption (stated explicitly): restic's stats / forget / restore behaviors are repo-format-level, not backend-level. A local directory repo therefore reproduces the exact Storage-Box-over-SFTP semantics under test; SFTP would add latency and failure modes but change none of the accounting/grouping/restore-shape answers. All timings below are LOCAL-repo and explicitly not SFTP-representative. The scratch repo used restic repo-format v2 (0.14 default; compression auto) — test data is /dev/urandom (incompressible) so raw-data bytes reflect real storage cost.

Scratch discipline: everything under /mnt/sys_drive/felhom-spike-scratch/ (repo, repo2, synthetic urandom sources, restore targets, throwaway 64-hex password in pw). No production path touched: the offbox key/repo_password/known_hosts live under DataDir/offbox (a docker volume), not under /mnt; the configured Hetzner target was never contacted. Cleanup proof in §Cleanup.

---

Part 1 — SP-1: stats semantics (quota accounting, §9)

Question: in a repo where a large path is captured in N snapshots, what does (a) stats --json (production call, restore-size, all snapshots) report vs (b) stats --mode raw-data --json vs (c) stats latest --json?

Protocol: fresh repo; unit/ = 5 files × 1 MiB (5,246,976 B) + bigdata/ = 4 files × 64 MiB (268,439,552 B, urandom); 5 snapshots of the same unchanged paths backup <abs>/unit <abs>/bigdata --tag felhom-offbox --tag spiketest; record the three stats; then append 20 MiB urandom to bigdata/, take a 6th snapshot, re-record.

Unchanged-run behavior (itself a finding): restic 0.14.0 creates a new snapshot every run even when nothing changed — 5 distinct IDs (9f66cd47, 3df2edcd, 1195267a, 1324877b, 528a324b), each Added to the repository: 0 B after the first. It does NOT skip unchanged runs.

Transcript (verbatim JSON):

# N=5
$ restic stats --json
{"total_size":1368391680,"total_file_count":70,"snapshots_count":5}
$ restic stats --mode raw-data --json
{"total_size":273700446,"total_file_count":0,"total_blob_count":185,"snapshots_count":5}
$ restic stats latest --json
{"total_size":273678336,"total_file_count":14,"snapshots_count":1}

# after +20 MiB delta → N=6 (backup #6: "Added to the repository: 20.015 MiB")
$ restic stats --json
{"total_size":1663041536,"total_file_count":85,"snapshots_count":6}
$ restic stats --mode raw-data --json
{"total_size":294681014,"total_file_count":0,"total_blob_count":203,"snapshots_count":6}
$ restic stats latest --json
{"total_size":294649856,"total_file_count":15,"snapshots_count":1}

Arithmetic:

	stats --json (default)	stats --mode raw-data	stats latest
N=5	1,368,391,680 = 5 × 273,678,336	273,700,446 (≈261 MiB, one physical copy)	273,678,336 (one snapshot)
N=6	1,663,041,536 = 5×273,678,336 + 294,649,856	294,681,014 (+20,980,568 B ≈ 20 MiB vs N=5)	294,649,856

• Default mode total_size = Σ over all snapshots of each snapshot's restore size (total_file_count 70 = 5×14). Per-snapshot multiplication CONFIRMED.
• raw-data total_size at N=5 ≈ the single physical copy (261 MiB); at N=6 it grew by exactly the 20 MiB delta, not by another full copy. Cross-snapshot dedup, no multiplication CONFIRMED.

Verdict:

• stats --json (default/restore-size) counts the logical restore size summed across every retained snapshot — for ~17 retained snapshots of a mandatory library it reports ~17× the disk.
• stats --mode raw-data counts the actual deduplicated+compressed bytes the repo occupies — what the customer's Storage Box really fills.
• Quota accounting for the new shape should read --mode raw-data because it is the only mode whose value tracks real Storage Box fill; the default mode multiplies by the retained-snapshot count and would trip the ≥100% gate at ~17× reality. (Confirms §9 direction / decision #6.)

---

Part 2 — SP-2: retention grouping across the shape change (§6)

Question: when an app's path set changes (unit-only → unit+userdata), does the production forget strand old-shape snapshots in a permanently-retained group? Does --group-by host,tags fix it without merging different apps?

Protocol (fresh repo2; --time backdating): appA old shape = 12 daily backup <unit> snapshots tagged felhom-offbox,appA, 2026-06-19…06-30; appA new shape = 12 daily backup <unit> <big> same tags, 2026-07-03…07-14; appB = 3 backup <unitB> tagged felhom-offbox,appB, 2026-07-12…14 (separation control). Then the production policy forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --dry-run with default grouping, and again with --group-by host,tags. Dry-run only — no snapshot destroyed.

Transcript (A) — DEFAULT grouping (host,paths), three groups:

Group [paths=p2unit]            (OLD shape appA) → keep 8, remove 4
  keep:   06-21(weekly) 06-24 06-25 06-26 06-27 06-28(d+w) 06-29 06-30(d+w+monthly)
  remove: 06-19 06-20 06-22 06-23
Group [paths=p2big,p2unit]      (NEW shape appA) → keep 8, remove 4
  keep:   07-05(weekly) 07-08 07-09 07-10 07-11 07-12(d+w) 07-13 07-14(d+w+monthly)
  remove: 07-03 07-04 07-06 07-07
Group [paths=p2unitB]           (appB)           → keep 3, remove 0

Transcript (B) — --group-by host,tags, two groups:

Group [tags=appA,felhom-offbox]  (OLD+NEW appA in ONE group) → keep 10, remove 14
  keep:   06-28(weekly) 06-30(monthly)   ← only 2 old-shape survive, as legit historical reps
          07-05(weekly) 07-08 07-09 07-10 07-11 07-12(d+w) 07-13 07-14(d+w+monthly)
  remove: 06-19 06-20 06-21 06-22 06-23 06-24 06-25 06-26 06-27 06-29   ← 10 of 12 old-shape gone
          07-03 07-04 07-06 07-07
Group [tags=appB,felhom-offbox]  (appB) → keep 3, remove 0   ← separate group

Interpretation (per the guardrail):

• Default grouping strands: YES. The old-shape [p2unit] group retains 8 June snapshots under its own independent 7d/4w/6m floor. Because every subsequent appA push carries the new [p2unit,p2big] path set, no new snapshot ever enters the [p2unit] group — nothing pushes those 8 out of the keep-daily-7 / keep-weekly-4 / keep-monthly-6 windows. They are retained forever → permanent zombie quota cost. This is the stranding signature: a separate old-shape group keeping its own full floor with no inflow to age it.
• --group-by host,tags ages old shape correctly: YES. The <stack> tag set is identical across old and new shape, so all 24 appA snapshots share one group. Only 06-28 (weekly) and 06-30 (monthly) survive — as genuine historical weekly/monthly representatives (correct AGING, not stranding); the other 10 old-shape snapshots are removed immediately, and as new daily pushes accumulate 06-28/06-30 fall out of the weekly-4/monthly-6 windows and age out too.
• Per-app group separation preserved under tags grouping: YES. appB (tags=appB,felhom-offbox) forms its own group — the tag LIST is the group key (restic sorts it; input felhom-offbox,appA displays as appA,felhom-offbox). appA and appB never merge.

Verdict: default grouping strands (evidence: the 8-snapshot orphan [p2unit] group with no inflow); --group-by host,tags ages the old shape correctly (10 of 12 old-shape removed at once, the 2 survivors are legit reps); per-app separation preserved (appB stays its own group). The 3a forget invocation should be forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --group-by host,tags --prune.

---

Part 3 — SP-3: multi-path restore shape, selectivity, fidelity, missing-source (§7)

Reused the Part-1 repo; added a fidelity fixture inside bigdata/ and took a 7th snapshot (8ce4ec93, tag spiketest), which became latest.

3.1 Full-restore tree shape. restore latest --tag spiketest --target <t> →

<t>/mnt/sys_drive/felhom-spike-scratch/unit/u1..u5.bin
<t>/mnt/sys_drive/felhom-spike-scratch/bigdata/{b1..b4,b5_delta,owned_1000}.bin, setgid_dir/

Verdict (§7.1): absolute source paths are reconstructed verbatim under --target (<target>/<abs-source-path>) — the scratch-relocation design must account for the full absolute prefix under the chosen scratch dir.

3.2 Unit-only selective restore. Four --include forms, each into its own target:

--include form	unit files	bigdata files
/mnt/sys_drive/felhom-spike-scratch/unit (absolute source path)	5	0
/mnt/sys_drive/felhom-spike-scratch/unit/*	5	0
*/unit/*	5	0
unit (bare component)	5	0

All four isolate the unit tree. The bare unit form works only because restic matches a path component anywhere and nothing else is named unit; it would also match a stray unit file under userdata. Verdict (§7.2): use --include <absolute-unit-path> (the exact src restic was handed at backup, i.e. backups/primary/<app>) — unambiguous, exact-subtree, no false matches.

3.3 Ownership/mode fidelity (root restore). Fixture before backup vs after restore:

source :  owned_1000.bin uid=1000 gid=1000 mode=644 | setgid_dir uid=0 gid=1000 mode=2775
restored: owned_1000.bin uid=1000 gid=1000 mode=644 | setgid_dir uid=0 gid=1000 mode=2775

Verdict (§5 fidelity row): restic root-restore preserves uid/gid, mode, and the setgid bit (2775) exactly — correct for the SharedContentGID=1000 / mode-2775 userdata convention.

3.4 Nonexistent source path. ⚠️ ARCHITECTURE IMPACT. backup <abs>/unit <abs>/does-not-exist --tag felhom-offbox --tag spiketest2:

/mnt/sys_drive/felhom-spike-scratch/does-not-exist does not exist, skipping
using parent snapshot 8ce4ec93
processed 5 files, 5.000 MiB in 0:00
snapshot 98bc38ef saved
[backup exit=0]
$ restic snapshots --tag spiketest2 --json
[{... "paths":["/mnt/sys_drive/felhom-spike-scratch/unit"], "tags":["felhom-offbox","spiketest2"],
  "short_id":"98bc38ef"}]

restic 0.14.0 does NOT error on a nonexistent source path. It prints "does not exist, skipping" to stderr, exits 0, and silently creates a snapshot containing only the paths that existed. This contradicts §2 decision 5 / §2.5, which state "restic errors on a nonexistent source path, so the capture set is stat-filtered before invocation." The stat-filter decision itself is still correct — but its stated justification is empirically false, and the real justification is stronger/opposite: without the pre-invocation stat-filter, a missing mandatory path produces a partial snapshot with a success exit code and no loud signal, so the §2.5 "loud WARN" would never fire off a restic error — the controller must stat-filter to detect the absence itself. Flagged; not redesigned here (rule 6). 3a must not rely on a nonzero restic exit to catch a missing mandatory path.

3.5 Shape introspection. snapshots latest --tag spiketest --json → "paths":["/mnt/sys_drive/felhom-spike-scratch/bigdata","/mnt/sys_drive/felhom-spike-scratch/unit"] — the paths field lists all captured paths. Verdict (§6): the snapshot's own paths metadata is an accurate shape record; the "no recovery-unit manifest change / no SchemaVersion bump" claim holds.

3.6 Timing. Full restore of the ~276 MB set (294,744,064 B) = 1.25 s wall-clock — LOCAL repo, NOT SFTP-representative; only a sanity check that the shape is nowhere near the 30-min handler ceiling (§12). SFTP over residential upstream will dominate; 3a should size the ceiling from a real SFTP restore, not this number.

---

Decision inputs for Task 3a

SP	Verdict	Resolves	Architecture-doc status
SP-1	Quota must read restic stats --mode raw-data --json (dedup+compressed real bytes). Default stats --json = Σ per-snapshot restore size = N× multiplication (N=5 → exactly 5×).	§9 accounting switch / decision #6	Assumption HOLDS — direction confirmed
SP-2	3a forget = forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --group-by host,tags --prune. Default (host,paths) strands 8 old-shape snapshots in an orphan group forever; tags-grouping ages them out and keeps apps separate.	§6 retention flag	Assumption HOLDS — stranding proven, fix proven
SP-3.1/3.2	Restore reconstructs <target>/<abs-source-path>; unit-only = restore … --include <absolute-unit-path>.	§7.1 / §7.2 restore shape + unit-only default	Assumption HOLDS
SP-3.3	uid/gid/mode + setgid bit preserved on root restore.	§5 restic fidelity row	Assumption HOLDS
SP-3.5	snapshots --json paths lists all captured paths — accurate shape record.	§6 "no manifest change" claim	Assumption HOLDS
SP-3.4	⚠️ ARCHITECTURE IMPACT — restic 0.14.0 does NOT error on a missing source path: skips-with-warning, exit 0, silent partial snapshot. §2 decision 5 / §2.5 premise ("restic errors") is FALSE. Stat-filter still required, but because restic gives no error signal (not because it errors).	§2 decision 5 / §2.5 stat-filter justification	CONTRADICTED (decision survives; rationale must be rewritten; 3a must not depend on a nonzero exit)

---

Cleanup (part of the deliverable)

rm -rf /mnt/sys_drive/felhom-spike-scratch executed inside the container. Proof — /mnt/sys_drive after cleanup (scratch gone; only the pre-existing namespace + lost+found remain), and free space returned to the pre-spike baseline:

$ ls -la /mnt/sys_drive
drwxr-xr-x felhom-data
drwx------ lost+found
$ df -h /mnt/sys_drive        # 2.8M used / 47G free — identical to session start
/dev/mapper/pve-vm--9201--disk--2  50G  2.8M  47G  1% /mnt/sys_drive

No secrets written to this doc or the commit (the throwaway repo password lived only in scratch and was removed with it). No production path modified.

Observations (documented, not acted on)

• restic 0.14.0's "new snapshot every run even when unchanged" means an offbox schedule that runs and finds no changes still adds a snapshot to the retention window — expected, and exactly why the keep-daily/weekly/monthly policy exists; noted for the 3a quota-growth mental model.
• stats --mode raw-data returns total_file_count:0 (it counts blobs, total_blob_count, not files) — a 3a consumer must read total_size, not total_file_count, from the raw-data JSON.
• The pinned restic is 0.14.0 (2022-era). The exit-code-on-missing-path behavior (SP-3.4) changed in later restic releases (newer versions exit 3 on skipped paths); if the container's restic is ever bumped, the SP-3.4 assumption must be re-checked — but the stat-filter design is correct under both behaviors.

---

Baselines re-verified live at session start. Findings-doc commit hash on felhom.eu main: recorded at commit time (see the commit that adds this file).