300 lines
18 KiB
Markdown
300 lines
18 KiB
Markdown
# SPIKE — restic snapshot shape, retention grouping, stats semantics (SP-1/SP-2/SP-3) — 2026-07-14
|
||
|
||
**Class:** spike (findings only — no production code, no config, no schema edits; the only write
|
||
outside scratch is this doc). Gates the **Task 3a** (offsite tier policy engine) spec, which is
|
||
BLOCKED until these three verdicts land (spike-first gate).
|
||
|
||
## Verdict up front
|
||
|
||
1. **SP-1 (§9 quota accounting):** `restic stats --json` (no mode = restore-size over ALL
|
||
snapshots) **multiplies** — it reports Σ over every retained snapshot of that snapshot's logical
|
||
size (N=5 → exactly 5× one snapshot). `stats --mode raw-data --json` reports the **actual
|
||
deduplicated+compressed repo bytes** (one physical copy; grows only by the inter-snapshot delta).
|
||
Quota MUST switch to `--mode raw-data`. Architecture-doc assumption **holds**.
|
||
2. **SP-2 (§6 forget flag):** the production `forget` call with **default grouping (host,paths)
|
||
STRANDS** old-shape snapshots — the unit-only group keeps its own full 7-daily/4-weekly/6-monthly
|
||
floor **forever** because no new snapshot ever enters it. `--group-by host,tags` collapses old +
|
||
new shape into one `<stack>`-keyed group so the old shape **ages out naturally**, and keeps
|
||
different apps in **separate groups**. Add `--group-by host,tags`. Architecture-doc assumption
|
||
**holds**.
|
||
3. **SP-3 (§7 restore):** multi-path restore reconstructs **absolute source paths under `--target`**;
|
||
`--include <absolute-unit-path>` gives a clean **unit-only** restore; **uid/gid/mode incl. the
|
||
setgid bit are preserved** on a root restore; `snapshots --json` `paths` lists all captured paths
|
||
(no manifest change needed). **One contradiction:** ⚠️ **ARCHITECTURE IMPACT** — restic 0.14.0
|
||
does **NOT** error on a nonexistent source path (§2 decision 5 / §2.5 premise); it skips with a
|
||
warning, **exits 0**, and silently creates a partial snapshot. The stat-filter is still required —
|
||
but for the opposite reason the doc states.
|
||
|
||
---
|
||
|
||
## 0. Baselines (live-verified at session start)
|
||
|
||
| Item | Value | How verified |
|
||
|---|---|---|
|
||
| felhom.eu `main` head (incl. architecture-doc + §10 confirmations) | **`eff107d3`** (`eff107d` "confirmations"; doc added in `82280b1`) | `git ls-remote origin` == local HEAD |
|
||
| felhom-controller live image on guest 9201 | **`gitea.dooplex.hu/admin/felhom-controller:0.132.0`** (Up, healthy) | `docker ps` on 9201 |
|
||
| restic version (inside the felhom-controller container — the production binary) | **`restic 0.14.0 compiled with go1.19.8 on linux/amd64`** | `docker exec felhom-controller restic version` |
|
||
| Scratch mount | `/mnt/sys_drive` — dedicated device (`pve-vm-9201-disk-2`, 50 G, 47 G free, own fs, NOT guest rootfs) | `df -h` inside container; container sees `/mnt` via `rslave` bind |
|
||
|
||
**Production invocation shapes mirrored (read-only from `internal/backup/offbox.go`, v0.132.0):**
|
||
backup `backup --tag felhom-offbox --tag <stack> <src>` (`:575`) · forget
|
||
`forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune` (`:595`) · stats `stats --json`
|
||
(no mode) (`:700`) · restore `restore latest --tag <stack> --target <dir>` (`:732`).
|
||
|
||
**Semantics assumption (stated explicitly):** restic's stats / forget / restore behaviors are
|
||
**repo-format-level, not backend-level**. A local directory repo therefore reproduces the exact
|
||
Storage-Box-over-SFTP semantics under test; SFTP would add latency and failure modes but change none
|
||
of the accounting/grouping/restore-shape answers. All timings below are LOCAL-repo and explicitly
|
||
**not** SFTP-representative. The scratch repo used restic repo-format v2 (0.14 default; compression
|
||
`auto`) — test data is `/dev/urandom` (incompressible) so raw-data bytes reflect real storage cost.
|
||
|
||
**Scratch discipline:** everything under `/mnt/sys_drive/felhom-spike-scratch/` (repo, `repo2`,
|
||
synthetic urandom sources, restore targets, throwaway 64-hex password in `pw`). No production path
|
||
touched: the offbox key/repo_password/known_hosts live under `DataDir/offbox` (a docker volume), not
|
||
under `/mnt`; the configured Hetzner target was never contacted. Cleanup proof in §Cleanup.
|
||
|
||
---
|
||
|
||
## Part 1 — SP-1: stats semantics (quota accounting, §9)
|
||
|
||
**Question:** in a repo where a large path is captured in N snapshots, what does (a) `stats --json`
|
||
(production call, restore-size, all snapshots) report vs (b) `stats --mode raw-data --json` vs
|
||
(c) `stats latest --json`?
|
||
|
||
**Protocol:** fresh repo; `unit/` = 5 files × 1 MiB (5,246,976 B) + `bigdata/` = 4 files × 64 MiB
|
||
(268,439,552 B, urandom); **5 snapshots of the same unchanged paths**
|
||
`backup <abs>/unit <abs>/bigdata --tag felhom-offbox --tag spiketest`; record the three stats; then
|
||
append 20 MiB urandom to `bigdata/`, take a 6th snapshot, re-record.
|
||
|
||
**Unchanged-run behavior (itself a finding):** restic 0.14.0 creates a **new snapshot every run**
|
||
even when nothing changed — 5 distinct IDs (`9f66cd47, 3df2edcd, 1195267a, 1324877b, 528a324b`),
|
||
each `Added to the repository: 0 B` after the first. It does NOT skip unchanged runs.
|
||
|
||
**Transcript (verbatim JSON):**
|
||
|
||
```
|
||
# N=5
|
||
$ restic stats --json
|
||
{"total_size":1368391680,"total_file_count":70,"snapshots_count":5}
|
||
$ restic stats --mode raw-data --json
|
||
{"total_size":273700446,"total_file_count":0,"total_blob_count":185,"snapshots_count":5}
|
||
$ restic stats latest --json
|
||
{"total_size":273678336,"total_file_count":14,"snapshots_count":1}
|
||
|
||
# after +20 MiB delta → N=6 (backup #6: "Added to the repository: 20.015 MiB")
|
||
$ restic stats --json
|
||
{"total_size":1663041536,"total_file_count":85,"snapshots_count":6}
|
||
$ restic stats --mode raw-data --json
|
||
{"total_size":294681014,"total_file_count":0,"total_blob_count":203,"snapshots_count":6}
|
||
$ restic stats latest --json
|
||
{"total_size":294649856,"total_file_count":15,"snapshots_count":1}
|
||
```
|
||
|
||
**Arithmetic:**
|
||
|
||
| | `stats --json` (default) | `stats --mode raw-data` | `stats latest` |
|
||
|---|---|---|---|
|
||
| N=5 | 1,368,391,680 = **5 × 273,678,336** | 273,700,446 (≈261 MiB, one physical copy) | 273,678,336 (one snapshot) |
|
||
| N=6 | 1,663,041,536 = 5×273,678,336 + 294,649,856 | 294,681,014 (**+20,980,568 B ≈ 20 MiB** vs N=5) | 294,649,856 |
|
||
|
||
- Default mode `total_size` = **Σ over all snapshots of each snapshot's restore size**
|
||
(`total_file_count` 70 = 5×14). **Per-snapshot multiplication CONFIRMED.**
|
||
- raw-data `total_size` at N=5 ≈ the single physical copy (261 MiB); at N=6 it grew by **exactly the
|
||
20 MiB delta**, not by another full copy. **Cross-snapshot dedup, no multiplication CONFIRMED.**
|
||
|
||
**Verdict:**
|
||
- `stats --json` (default/restore-size) counts **the logical restore size summed across every
|
||
retained snapshot** — for ~17 retained snapshots of a mandatory library it reports ~17× the disk.
|
||
- `stats --mode raw-data` counts **the actual deduplicated+compressed bytes the repo occupies** —
|
||
what the customer's Storage Box really fills.
|
||
- **Quota accounting for the new shape should read `--mode raw-data` because it is the only mode
|
||
whose value tracks real Storage Box fill; the default mode multiplies by the retained-snapshot
|
||
count and would trip the ≥100% gate at ~17× reality.** (Confirms §9 direction / decision #6.)
|
||
|
||
---
|
||
|
||
## Part 2 — SP-2: retention grouping across the shape change (§6)
|
||
|
||
**Question:** when an app's path set changes (unit-only → unit+userdata), does the production
|
||
`forget` strand old-shape snapshots in a permanently-retained group? Does `--group-by host,tags`
|
||
fix it without merging different apps?
|
||
|
||
**Protocol (fresh `repo2`; `--time` backdating):** appA old shape = 12 daily `backup <unit>`
|
||
snapshots tagged `felhom-offbox,appA`, 2026-06-19…06-30; appA new shape = 12 daily
|
||
`backup <unit> <big>` same tags, 2026-07-03…07-14; appB = 3 `backup <unitB>` tagged
|
||
`felhom-offbox,appB`, 2026-07-12…14 (separation control). Then the production policy
|
||
`forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --dry-run` with default grouping, and again
|
||
with `--group-by host,tags`. **Dry-run only — no snapshot destroyed.**
|
||
|
||
**Transcript (A) — DEFAULT grouping (host,paths), three groups:**
|
||
|
||
```
|
||
Group [paths=p2unit] (OLD shape appA) → keep 8, remove 4
|
||
keep: 06-21(weekly) 06-24 06-25 06-26 06-27 06-28(d+w) 06-29 06-30(d+w+monthly)
|
||
remove: 06-19 06-20 06-22 06-23
|
||
Group [paths=p2big,p2unit] (NEW shape appA) → keep 8, remove 4
|
||
keep: 07-05(weekly) 07-08 07-09 07-10 07-11 07-12(d+w) 07-13 07-14(d+w+monthly)
|
||
remove: 07-03 07-04 07-06 07-07
|
||
Group [paths=p2unitB] (appB) → keep 3, remove 0
|
||
```
|
||
|
||
**Transcript (B) — `--group-by host,tags`, two groups:**
|
||
|
||
```
|
||
Group [tags=appA,felhom-offbox] (OLD+NEW appA in ONE group) → keep 10, remove 14
|
||
keep: 06-28(weekly) 06-30(monthly) ← only 2 old-shape survive, as legit historical reps
|
||
07-05(weekly) 07-08 07-09 07-10 07-11 07-12(d+w) 07-13 07-14(d+w+monthly)
|
||
remove: 06-19 06-20 06-21 06-22 06-23 06-24 06-25 06-26 06-27 06-29 ← 10 of 12 old-shape gone
|
||
07-03 07-04 07-06 07-07
|
||
Group [tags=appB,felhom-offbox] (appB) → keep 3, remove 0 ← separate group
|
||
```
|
||
|
||
**Interpretation (per the guardrail):**
|
||
- **Default grouping strands: YES.** The old-shape `[p2unit]` group retains **8** June snapshots
|
||
under its own independent 7d/4w/6m floor. Because every subsequent appA push carries the new
|
||
`[p2unit,p2big]` path set, **no new snapshot ever enters the `[p2unit]` group** — nothing pushes
|
||
those 8 out of the keep-daily-7 / keep-weekly-4 / keep-monthly-6 windows. They are retained
|
||
**forever** → permanent zombie quota cost. This is the stranding signature: *a separate old-shape
|
||
group keeping its own full floor with no inflow to age it*.
|
||
- **`--group-by host,tags` ages old shape correctly: YES.** The `<stack>` tag set is identical
|
||
across old and new shape, so all 24 appA snapshots share **one** group. Only **06-28 (weekly)**
|
||
and **06-30 (monthly)** survive — as genuine historical weekly/monthly representatives (correct
|
||
AGING, not stranding); the other 10 old-shape snapshots are removed immediately, and as new daily
|
||
pushes accumulate 06-28/06-30 fall out of the weekly-4/monthly-6 windows and age out too.
|
||
- **Per-app group separation preserved under tags grouping: YES.** appB
|
||
(`tags=appB,felhom-offbox`) forms its **own** group — the tag LIST is the group key (restic sorts
|
||
it; input `felhom-offbox,appA` displays as `appA,felhom-offbox`). appA and appB never merge.
|
||
|
||
**Verdict:** default grouping strands (evidence: the 8-snapshot orphan `[p2unit]` group with no
|
||
inflow); `--group-by host,tags` ages the old shape correctly (10 of 12 old-shape removed at once,
|
||
the 2 survivors are legit reps); per-app separation preserved (appB stays its own group).
|
||
**The 3a forget invocation should be
|
||
`forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --group-by host,tags --prune`.**
|
||
|
||
---
|
||
|
||
## Part 3 — SP-3: multi-path restore shape, selectivity, fidelity, missing-source (§7)
|
||
|
||
Reused the Part-1 repo; added a fidelity fixture inside `bigdata/` and took a 7th snapshot
|
||
(`8ce4ec93`, tag `spiketest`), which became `latest`.
|
||
|
||
**3.1 Full-restore tree shape.** `restore latest --tag spiketest --target <t>` →
|
||
|
||
```
|
||
<t>/mnt/sys_drive/felhom-spike-scratch/unit/u1..u5.bin
|
||
<t>/mnt/sys_drive/felhom-spike-scratch/bigdata/{b1..b4,b5_delta,owned_1000}.bin, setgid_dir/
|
||
```
|
||
|
||
**Verdict (§7.1):** absolute source paths are reconstructed verbatim under `--target`
|
||
(`<target>/<abs-source-path>`) — the scratch-relocation design must account for the full absolute
|
||
prefix under the chosen scratch dir.
|
||
|
||
**3.2 Unit-only selective restore.** Four `--include` forms, each into its own target:
|
||
|
||
| `--include` form | unit files | bigdata files |
|
||
|---|---|---|
|
||
| `/mnt/sys_drive/felhom-spike-scratch/unit` (absolute source path) | 5 | 0 |
|
||
| `/mnt/sys_drive/felhom-spike-scratch/unit/*` | 5 | 0 |
|
||
| `*/unit/*` | 5 | 0 |
|
||
| `unit` (bare component) | 5 | 0 |
|
||
|
||
All four isolate the unit tree. The bare `unit` form works only because restic matches a path
|
||
**component** anywhere and nothing else is named `unit`; it would also match a stray `unit` file
|
||
under userdata. **Verdict (§7.2): use `--include <absolute-unit-path>`** (the exact `src` restic was
|
||
handed at backup, i.e. `backups/primary/<app>`) — unambiguous, exact-subtree, no false matches.
|
||
|
||
**3.3 Ownership/mode fidelity (root restore).** Fixture before backup vs after restore:
|
||
|
||
```
|
||
source : owned_1000.bin uid=1000 gid=1000 mode=644 | setgid_dir uid=0 gid=1000 mode=2775
|
||
restored: owned_1000.bin uid=1000 gid=1000 mode=644 | setgid_dir uid=0 gid=1000 mode=2775
|
||
```
|
||
|
||
**Verdict (§5 fidelity row):** restic root-restore preserves uid/gid, mode, **and the setgid bit
|
||
(2775)** exactly — correct for the `SharedContentGID=1000` / mode-2775 userdata convention.
|
||
|
||
**3.4 Nonexistent source path.** ⚠️ **ARCHITECTURE IMPACT.**
|
||
`backup <abs>/unit <abs>/does-not-exist --tag felhom-offbox --tag spiketest2`:
|
||
|
||
```
|
||
/mnt/sys_drive/felhom-spike-scratch/does-not-exist does not exist, skipping
|
||
using parent snapshot 8ce4ec93
|
||
processed 5 files, 5.000 MiB in 0:00
|
||
snapshot 98bc38ef saved
|
||
[backup exit=0]
|
||
$ restic snapshots --tag spiketest2 --json
|
||
[{... "paths":["/mnt/sys_drive/felhom-spike-scratch/unit"], "tags":["felhom-offbox","spiketest2"],
|
||
"short_id":"98bc38ef"}]
|
||
```
|
||
|
||
restic 0.14.0 **does NOT error** on a nonexistent source path. It prints "does not exist, skipping"
|
||
to stderr, **exits 0**, and **silently creates a snapshot containing only the paths that existed**.
|
||
This **contradicts** §2 decision 5 / §2.5, which state "restic errors on a nonexistent source path,
|
||
so the capture set is stat-filtered before invocation." The stat-filter decision itself is still
|
||
correct — but its stated justification is empirically false, and the real justification is
|
||
**stronger/opposite**: without the pre-invocation stat-filter, a missing mandatory path produces a
|
||
**partial snapshot with a success exit code and no loud signal**, so the §2.5 "loud WARN" would
|
||
never fire off a restic error — the controller must stat-filter to *detect* the absence itself.
|
||
**Flagged; not redesigned here (rule 6).** 3a must not rely on a nonzero restic exit to catch a
|
||
missing mandatory path.
|
||
|
||
**3.5 Shape introspection.** `snapshots latest --tag spiketest --json` →
|
||
`"paths":["/mnt/sys_drive/felhom-spike-scratch/bigdata","/mnt/sys_drive/felhom-spike-scratch/unit"]`
|
||
— the `paths` field lists **all** captured paths. **Verdict (§6): the snapshot's own `paths`
|
||
metadata is an accurate shape record; the "no recovery-unit manifest change / no SchemaVersion bump"
|
||
claim holds.**
|
||
|
||
**3.6 Timing.** Full restore of the ~276 MB set (294,744,064 B) = **1.25 s** wall-clock — LOCAL repo,
|
||
**NOT** SFTP-representative; only a sanity check that the shape is nowhere near the 30-min handler
|
||
ceiling (§12). SFTP over residential upstream will dominate; 3a should size the ceiling from a real
|
||
SFTP restore, not this number.
|
||
|
||
---
|
||
|
||
## Decision inputs for Task 3a
|
||
|
||
| SP | Verdict | Resolves | Architecture-doc status |
|
||
|---|---|---|---|
|
||
| **SP-1** | Quota must read `restic stats --mode raw-data --json` (dedup+compressed real bytes). Default `stats --json` = Σ per-snapshot restore size = N× multiplication (N=5 → exactly 5×). | **§9** accounting switch / decision #6 | Assumption **HOLDS** — direction confirmed |
|
||
| **SP-2** | 3a forget = `forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --group-by host,tags --prune`. Default (host,paths) strands 8 old-shape snapshots in an orphan group forever; tags-grouping ages them out and keeps apps separate. | **§6** retention flag | Assumption **HOLDS** — stranding proven, fix proven |
|
||
| **SP-3.1/3.2** | Restore reconstructs `<target>/<abs-source-path>`; unit-only = `restore … --include <absolute-unit-path>`. | **§7.1 / §7.2** restore shape + unit-only default | Assumption **HOLDS** |
|
||
| **SP-3.3** | uid/gid/mode + setgid bit preserved on root restore. | **§5** restic fidelity row | Assumption **HOLDS** |
|
||
| **SP-3.5** | `snapshots --json` `paths` lists all captured paths — accurate shape record. | **§6** "no manifest change" claim | Assumption **HOLDS** |
|
||
| **SP-3.4** | ⚠️ **ARCHITECTURE IMPACT** — restic 0.14.0 does NOT error on a missing source path: skips-with-warning, **exit 0**, silent partial snapshot. §2 decision 5 / §2.5 premise ("restic errors") is FALSE. Stat-filter still required, but because restic gives **no** error signal (not because it errors). | **§2 decision 5 / §2.5** stat-filter justification | **CONTRADICTED** (decision survives; rationale must be rewritten; 3a must not depend on a nonzero exit) |
|
||
|
||
---
|
||
|
||
## Cleanup (part of the deliverable)
|
||
|
||
`rm -rf /mnt/sys_drive/felhom-spike-scratch` executed inside the container. Proof — `/mnt/sys_drive`
|
||
after cleanup (scratch gone; only the pre-existing namespace + `lost+found` remain), and free space
|
||
returned to the pre-spike baseline:
|
||
|
||
```
|
||
$ ls -la /mnt/sys_drive
|
||
drwxr-xr-x felhom-data
|
||
drwx------ lost+found
|
||
$ df -h /mnt/sys_drive # 2.8M used / 47G free — identical to session start
|
||
/dev/mapper/pve-vm--9201--disk--2 50G 2.8M 47G 1% /mnt/sys_drive
|
||
```
|
||
|
||
No secrets written to this doc or the commit (the throwaway repo password lived only in scratch and
|
||
was removed with it). No production path modified.
|
||
|
||
## Observations (documented, not acted on)
|
||
|
||
- restic 0.14.0's "new snapshot every run even when unchanged" means an offbox schedule that runs
|
||
and finds no changes still adds a snapshot to the retention window — expected, and exactly why the
|
||
keep-daily/weekly/monthly policy exists; noted for the 3a quota-growth mental model.
|
||
- `stats --mode raw-data` returns `total_file_count:0` (it counts blobs, `total_blob_count`, not
|
||
files) — a 3a consumer must read `total_size`, not `total_file_count`, from the raw-data JSON.
|
||
- The pinned restic is **0.14.0** (2022-era). The exit-code-on-missing-path behavior (SP-3.4)
|
||
changed in later restic releases (newer versions exit 3 on skipped paths); if the container's
|
||
restic is ever bumped, the SP-3.4 assumption must be re-checked — but the stat-filter design is
|
||
correct under both behaviors.
|
||
|
||
---
|
||
|
||
*Baselines re-verified live at session start. Findings-doc added to felhom.eu `main` at commit
|
||
**`0771d58`** (this recorded-hash line committed as the immediate follow-up).*
|