admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SPIKE restic snapshot shape (SP-1/SP-2/SP-3): stats mode, forget grouping, restore shape

Browse Source 
SP-1: quota must use stats --mode raw-data (default restore-size multiplies N x).
SP-2: default forget grouping (host,paths) strands old-shape snapshots forever;
      --group-by host,tags ages them out, keeps apps separate.
SP-3: multi-path restore reconstructs abs paths under --target; --include <abs-unit>
      = unit-only; uid/gid/setgid preserved; snapshots paths field is the shape record.
ARCHITECTURE IMPACT: restic 0.14.0 does NOT error on a missing source path (exit 0,
      silent partial snapshot) -- contradicts §2.5 premise; stat-filter still required.
restic 0.14.0 in felhom-controller:0.132.0 container; scratch on demo 9201, cleaned up.
This commit is contained in:
authentik Default Admin
2026-07-14 21:18:11 +02:00
parent eff107d324
commit 0771d58b58
1 changed files with 299 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/SPIKE-restic-snapshot-shape-2026-07-14.md
+299
View File
@@ -0,0 +1,299 @@
# SPIKE — restic snapshot shape, retention grouping, stats semantics (SP-1/SP-2/SP-3) — 2026-07-14
**Class:** spike (findings only — no production code, no config, no schema edits; the only write
outside scratch is this doc). Gates the **Task 3a** (offsite tier policy engine) spec, which is
BLOCKED until these three verdicts land (spike-first gate).
## Verdict up front
1. **SP-1 (§9 quota accounting):** `restic stats --json` (no mode = restore-size over ALL
snapshots) **multiplies** — it reports Σ over every retained snapshot of that snapshot's logical
size (N=5 → exactly 5× one snapshot). `stats --mode raw-data --json` reports the **actual
deduplicated+compressed repo bytes** (one physical copy; grows only by the inter-snapshot delta).
Quota MUST switch to `--mode raw-data`. Architecture-doc assumption **holds**.
2. **SP-2 (§6 forget flag):** the production `forget` call with **default grouping (host,paths)
STRANDS** old-shape snapshots — the unit-only group keeps its own full 7-daily/4-weekly/6-monthly
floor **forever** because no new snapshot ever enters it. `--group-by host,tags` collapses old +
new shape into one `<stack>`-keyed group so the old shape **ages out naturally**, and keeps
different apps in **separate groups**. Add `--group-by host,tags`. Architecture-doc assumption
**holds**.
3. **SP-3 (§7 restore):** multi-path restore reconstructs **absolute source paths under `--target`**;
`--include <absolute-unit-path>` gives a clean **unit-only** restore; **uid/gid/mode incl. the
setgid bit are preserved** on a root restore; `snapshots --json` `paths` lists all captured paths
(no manifest change needed). **One contradiction:** ⚠️ **ARCHITECTURE IMPACT** — restic 0.14.0
does **NOT** error on a nonexistent source path (§2 decision 5 / §2.5 premise); it skips with a
warning, **exits 0**, and silently creates a partial snapshot. The stat-filter is still required —
but for the opposite reason the doc states.
---
## 0. Baselines (live-verified at session start)
| Item | Value | How verified |
|---|---|---|
| felhom.eu `main` head (incl. architecture-doc + §10 confirmations) | **`eff107d3`** (`eff107d` "confirmations"; doc added in `82280b1`) | `git ls-remote origin` == local HEAD |
| felhom-controller live image on guest 9201 | **`gitea.dooplex.hu/admin/felhom-controller:0.132.0`** (Up, healthy) | `docker ps` on 9201 |
| restic version (inside the felhom-controller container — the production binary) | **`restic 0.14.0 compiled with go1.19.8 on linux/amd64`** | `docker exec felhom-controller restic version` |
| Scratch mount | `/mnt/sys_drive` — dedicated device (`pve-vm-9201-disk-2`, 50 G, 47 G free, own fs, NOT guest rootfs) | `df -h` inside container; container sees `/mnt` via `rslave` bind |
**Production invocation shapes mirrored (read-only from `internal/backup/offbox.go`, v0.132.0):**
backup `backup --tag felhom-offbox --tag <stack> <src>` (`:575`) · forget
`forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune` (`:595`) · stats `stats --json`
(no mode) (`:700`) · restore `restore latest --tag <stack> --target <dir>` (`:732`).
**Semantics assumption (stated explicitly):** restic's stats / forget / restore behaviors are
**repo-format-level, not backend-level**. A local directory repo therefore reproduces the exact
Storage-Box-over-SFTP semantics under test; SFTP would add latency and failure modes but change none
of the accounting/grouping/restore-shape answers. All timings below are LOCAL-repo and explicitly
**not** SFTP-representative. The scratch repo used restic repo-format v2 (0.14 default; compression
`auto`) — test data is `/dev/urandom` (incompressible) so raw-data bytes reflect real storage cost.
**Scratch discipline:** everything under `/mnt/sys_drive/felhom-spike-scratch/` (repo, `repo2`,
synthetic urandom sources, restore targets, throwaway 64-hex password in `pw`). No production path
touched: the offbox key/repo_password/known_hosts live under `DataDir/offbox` (a docker volume), not
under `/mnt`; the configured Hetzner target was never contacted. Cleanup proof in §Cleanup.
---
## Part 1 — SP-1: stats semantics (quota accounting, §9)
**Question:** in a repo where a large path is captured in N snapshots, what does (a) `stats --json`
(production call, restore-size, all snapshots) report vs (b) `stats --mode raw-data --json` vs
(c) `stats latest --json`?
**Protocol:** fresh repo; `unit/` = 5 files × 1 MiB (5,246,976 B) + `bigdata/` = 4 files × 64 MiB
(268,439,552 B, urandom); **5 snapshots of the same unchanged paths**
`backup <abs>/unit <abs>/bigdata --tag felhom-offbox --tag spiketest`; record the three stats; then
append 20 MiB urandom to `bigdata/`, take a 6th snapshot, re-record.
**Unchanged-run behavior (itself a finding):** restic 0.14.0 creates a **new snapshot every run**
even when nothing changed — 5 distinct IDs (`9f66cd47, 3df2edcd, 1195267a, 1324877b, 528a324b`),
each `Added to the repository: 0 B` after the first. It does NOT skip unchanged runs.
**Transcript (verbatim JSON):**
```
# N=5
$ restic stats --json
{"total_size":1368391680,"total_file_count":70,"snapshots_count":5}
$ restic stats --mode raw-data --json
{"total_size":273700446,"total_file_count":0,"total_blob_count":185,"snapshots_count":5}
$ restic stats latest --json
{"total_size":273678336,"total_file_count":14,"snapshots_count":1}
# after +20 MiB delta → N=6 (backup #6: "Added to the repository: 20.015 MiB")
$ restic stats --json
{"total_size":1663041536,"total_file_count":85,"snapshots_count":6}
$ restic stats --mode raw-data --json
{"total_size":294681014,"total_file_count":0,"total_blob_count":203,"snapshots_count":6}
$ restic stats latest --json
{"total_size":294649856,"total_file_count":15,"snapshots_count":1}
```
**Arithmetic:**
| | `stats --json` (default) | `stats --mode raw-data` | `stats latest` |
|---|---|---|---|
| N=5 | 1,368,391,680 = **5 × 273,678,336** | 273,700,446 (≈261 MiB, one physical copy) | 273,678,336 (one snapshot) |
| N=6 | 1,663,041,536 = 5×273,678,336 + 294,649,856 | 294,681,014 (**+20,980,568 B ≈ 20 MiB** vs N=5) | 294,649,856 |
- Default mode `total_size` = **Σ over all snapshots of each snapshot's restore size**
(`total_file_count` 70 = 5×14). **Per-snapshot multiplication CONFIRMED.**
- raw-data `total_size` at N=5 ≈ the single physical copy (261 MiB); at N=6 it grew by **exactly the
20 MiB delta**, not by another full copy. **Cross-snapshot dedup, no multiplication CONFIRMED.**
**Verdict:**
- `stats --json` (default/restore-size) counts **the logical restore size summed across every
retained snapshot** — for ~17 retained snapshots of a mandatory library it reports ~17× the disk.
- `stats --mode raw-data` counts **the actual deduplicated+compressed bytes the repo occupies** —
what the customer's Storage Box really fills.
- **Quota accounting for the new shape should read `--mode raw-data` because it is the only mode
whose value tracks real Storage Box fill; the default mode multiplies by the retained-snapshot
count and would trip the ≥100% gate at ~17× reality.** (Confirms §9 direction / decision #6.)
---
## Part 2 — SP-2: retention grouping across the shape change (§6)
**Question:** when an app's path set changes (unit-only → unit+userdata), does the production
`forget` strand old-shape snapshots in a permanently-retained group? Does `--group-by host,tags`
fix it without merging different apps?
**Protocol (fresh `repo2`; `--time` backdating):** appA old shape = 12 daily `backup <unit>`
snapshots tagged `felhom-offbox,appA`, 2026-06-19…06-30; appA new shape = 12 daily
`backup <unit> <big>` same tags, 2026-07-03…07-14; appB = 3 `backup <unitB>` tagged
`felhom-offbox,appB`, 2026-07-12…14 (separation control). Then the production policy
`forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --dry-run` with default grouping, and again
with `--group-by host,tags`. **Dry-run only — no snapshot destroyed.**
**Transcript (A) — DEFAULT grouping (host,paths), three groups:**
```
Group [paths=p2unit] (OLD shape appA) → keep 8, remove 4
keep: 06-21(weekly) 06-24 06-25 06-26 06-27 06-28(d+w) 06-29 06-30(d+w+monthly)
remove: 06-19 06-20 06-22 06-23
Group [paths=p2big,p2unit] (NEW shape appA) → keep 8, remove 4
keep: 07-05(weekly) 07-08 07-09 07-10 07-11 07-12(d+w) 07-13 07-14(d+w+monthly)
remove: 07-03 07-04 07-06 07-07
Group [paths=p2unitB] (appB) → keep 3, remove 0
```
**Transcript (B) — `--group-by host,tags`, two groups:**
```
Group [tags=appA,felhom-offbox] (OLD+NEW appA in ONE group) → keep 10, remove 14
keep: 06-28(weekly) 06-30(monthly) ← only 2 old-shape survive, as legit historical reps
07-05(weekly) 07-08 07-09 07-10 07-11 07-12(d+w) 07-13 07-14(d+w+monthly)
remove: 06-19 06-20 06-21 06-22 06-23 06-24 06-25 06-26 06-27 06-29 ← 10 of 12 old-shape gone
07-03 07-04 07-06 07-07
Group [tags=appB,felhom-offbox] (appB) → keep 3, remove 0 ← separate group
```
**Interpretation (per the guardrail):**
- **Default grouping strands: YES.** The old-shape `[p2unit]` group retains **8** June snapshots
under its own independent 7d/4w/6m floor. Because every subsequent appA push carries the new
`[p2unit,p2big]` path set, **no new snapshot ever enters the `[p2unit]` group** — nothing pushes
those 8 out of the keep-daily-7 / keep-weekly-4 / keep-monthly-6 windows. They are retained
**forever** → permanent zombie quota cost. This is the stranding signature: *a separate old-shape
group keeping its own full floor with no inflow to age it*.
- **`--group-by host,tags` ages old shape correctly: YES.** The `<stack>` tag set is identical
across old and new shape, so all 24 appA snapshots share **one** group. Only **06-28 (weekly)**
and **06-30 (monthly)** survive — as genuine historical weekly/monthly representatives (correct
AGING, not stranding); the other 10 old-shape snapshots are removed immediately, and as new daily
pushes accumulate 06-28/06-30 fall out of the weekly-4/monthly-6 windows and age out too.
- **Per-app group separation preserved under tags grouping: YES.** appB
(`tags=appB,felhom-offbox`) forms its **own** group — the tag LIST is the group key (restic sorts
it; input `felhom-offbox,appA` displays as `appA,felhom-offbox`). appA and appB never merge.
**Verdict:** default grouping strands (evidence: the 8-snapshot orphan `[p2unit]` group with no
inflow); `--group-by host,tags` ages the old shape correctly (10 of 12 old-shape removed at once,
the 2 survivors are legit reps); per-app separation preserved (appB stays its own group).
**The 3a forget invocation should be
`forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --group-by host,tags --prune`.**
---
## Part 3 — SP-3: multi-path restore shape, selectivity, fidelity, missing-source (§7)
Reused the Part-1 repo; added a fidelity fixture inside `bigdata/` and took a 7th snapshot
(`8ce4ec93`, tag `spiketest`), which became `latest`.
**3.1 Full-restore tree shape.** `restore latest --tag spiketest --target <t>` →
```
<t>/mnt/sys_drive/felhom-spike-scratch/unit/u1..u5.bin
<t>/mnt/sys_drive/felhom-spike-scratch/bigdata/{b1..b4,b5_delta,owned_1000}.bin, setgid_dir/
```
**Verdict (§7.1):** absolute source paths are reconstructed verbatim under `--target`
(`<target>/<abs-source-path>`) — the scratch-relocation design must account for the full absolute
prefix under the chosen scratch dir.
**3.2 Unit-only selective restore.** Four `--include` forms, each into its own target:
| `--include` form | unit files | bigdata files |
|---|---|---|
| `/mnt/sys_drive/felhom-spike-scratch/unit` (absolute source path) | 5 | 0 |
| `/mnt/sys_drive/felhom-spike-scratch/unit/*` | 5 | 0 |
| `*/unit/*` | 5 | 0 |
| `unit` (bare component) | 5 | 0 |
All four isolate the unit tree. The bare `unit` form works only because restic matches a path
**component** anywhere and nothing else is named `unit`; it would also match a stray `unit` file
under userdata. **Verdict (§7.2): use `--include <absolute-unit-path>`** (the exact `src` restic was
handed at backup, i.e. `backups/primary/<app>`) — unambiguous, exact-subtree, no false matches.
**3.3 Ownership/mode fidelity (root restore).** Fixture before backup vs after restore:
```
source : owned_1000.bin uid=1000 gid=1000 mode=644 | setgid_dir uid=0 gid=1000 mode=2775
restored: owned_1000.bin uid=1000 gid=1000 mode=644 | setgid_dir uid=0 gid=1000 mode=2775
```
**Verdict (§5 fidelity row):** restic root-restore preserves uid/gid, mode, **and the setgid bit
(2775)** exactly — correct for the `SharedContentGID=1000` / mode-2775 userdata convention.
**3.4 Nonexistent source path.** ⚠️ **ARCHITECTURE IMPACT.**
`backup <abs>/unit <abs>/does-not-exist --tag felhom-offbox --tag spiketest2`:
```
/mnt/sys_drive/felhom-spike-scratch/does-not-exist does not exist, skipping
using parent snapshot 8ce4ec93
processed 5 files, 5.000 MiB in 0:00
snapshot 98bc38ef saved
[backup exit=0]
$ restic snapshots --tag spiketest2 --json
[{... "paths":["/mnt/sys_drive/felhom-spike-scratch/unit"], "tags":["felhom-offbox","spiketest2"],
"short_id":"98bc38ef"}]
```
restic 0.14.0 **does NOT error** on a nonexistent source path. It prints "does not exist, skipping"
to stderr, **exits 0**, and **silently creates a snapshot containing only the paths that existed**.
This **contradicts** §2 decision 5 / §2.5, which state "restic errors on a nonexistent source path,
so the capture set is stat-filtered before invocation." The stat-filter decision itself is still
correct — but its stated justification is empirically false, and the real justification is
**stronger/opposite**: without the pre-invocation stat-filter, a missing mandatory path produces a
**partial snapshot with a success exit code and no loud signal**, so the §2.5 "loud WARN" would
never fire off a restic error — the controller must stat-filter to *detect* the absence itself.
**Flagged; not redesigned here (rule 6).** 3a must not rely on a nonzero restic exit to catch a
missing mandatory path.
**3.5 Shape introspection.** `snapshots latest --tag spiketest --json` →
`"paths":["/mnt/sys_drive/felhom-spike-scratch/bigdata","/mnt/sys_drive/felhom-spike-scratch/unit"]`
— the `paths` field lists **all** captured paths. **Verdict (§6): the snapshot's own `paths`
metadata is an accurate shape record; the "no recovery-unit manifest change / no SchemaVersion bump"
claim holds.**
**3.6 Timing.** Full restore of the ~276 MB set (294,744,064 B) = **1.25 s** wall-clock — LOCAL repo,
**NOT** SFTP-representative; only a sanity check that the shape is nowhere near the 30-min handler
ceiling (§12). SFTP over residential upstream will dominate; 3a should size the ceiling from a real
SFTP restore, not this number.
---
## Decision inputs for Task 3a
| SP | Verdict | Resolves | Architecture-doc status |
|---|---|---|---|
| **SP-1** | Quota must read `restic stats --mode raw-data --json` (dedup+compressed real bytes). Default `stats --json` = Σ per-snapshot restore size = N× multiplication (N=5 → exactly 5×). | **§9** accounting switch / decision #6 | Assumption **HOLDS** — direction confirmed |
| **SP-2** | 3a forget = `forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --group-by host,tags --prune`. Default (host,paths) strands 8 old-shape snapshots in an orphan group forever; tags-grouping ages them out and keeps apps separate. | **§6** retention flag | Assumption **HOLDS** — stranding proven, fix proven |
| **SP-3.1/3.2** | Restore reconstructs `<target>/<abs-source-path>`; unit-only = `restore … --include <absolute-unit-path>`. | **§7.1 / §7.2** restore shape + unit-only default | Assumption **HOLDS** |
| **SP-3.3** | uid/gid/mode + setgid bit preserved on root restore. | **§5** restic fidelity row | Assumption **HOLDS** |
| **SP-3.5** | `snapshots --json` `paths` lists all captured paths — accurate shape record. | **§6** "no manifest change" claim | Assumption **HOLDS** |
| **SP-3.4** | ⚠️ **ARCHITECTURE IMPACT** — restic 0.14.0 does NOT error on a missing source path: skips-with-warning, **exit 0**, silent partial snapshot. §2 decision 5 / §2.5 premise ("restic errors") is FALSE. Stat-filter still required, but because restic gives **no** error signal (not because it errors). | **§2 decision 5 / §2.5** stat-filter justification | **CONTRADICTED** (decision survives; rationale must be rewritten; 3a must not depend on a nonzero exit) |
---
## Cleanup (part of the deliverable)
`rm -rf /mnt/sys_drive/felhom-spike-scratch` executed inside the container. Proof — `/mnt/sys_drive`
after cleanup (scratch gone; only the pre-existing namespace + `lost+found` remain), and free space
returned to the pre-spike baseline:
```
$ ls -la /mnt/sys_drive
drwxr-xr-x felhom-data
drwx------ lost+found
$ df -h /mnt/sys_drive # 2.8M used / 47G free — identical to session start
/dev/mapper/pve-vm--9201--disk--2 50G 2.8M 47G 1% /mnt/sys_drive
```
No secrets written to this doc or the commit (the throwaway repo password lived only in scratch and
was removed with it). No production path modified.
## Observations (documented, not acted on)
- restic 0.14.0's "new snapshot every run even when unchanged" means an offbox schedule that runs
and finds no changes still adds a snapshot to the retention window — expected, and exactly why the
keep-daily/weekly/monthly policy exists; noted for the 3a quota-growth mental model.
- `stats --mode raw-data` returns `total_file_count:0` (it counts blobs, `total_blob_count`, not
files) — a 3a consumer must read `total_size`, not `total_file_count`, from the raw-data JSON.
- The pinned restic is **0.14.0** (2022-era). The exit-code-on-missing-path behavior (SP-3.4)
changed in later restic releases (newer versions exit 3 on skipped paths); if the container's
restic is ever bumped, the SP-3.4 assumption must be re-checked — but the stat-filter design is
correct under both behaviors.
---
*Baselines re-verified live at session start. Findings-doc commit hash on felhom.eu `main`: recorded
at commit time (see the commit that adds this file).*