diff --git a/REPORT-r87-spike.md b/REPORT-r87-spike.md new file mode 100644 index 00000000..29558b70 --- /dev/null +++ b/REPORT-r87-spike.md @@ -0,0 +1,176 @@ +# REPORT — R-87 spike: can the off-site copy be restore-tested without a person? (2026-08-31) + +Written as `REPORT-r87-spike.md`, not `REPORT.md`: this repo's `CLAUDE.md` says the shared report is +overwritten and two sessions clobber each other. + +**Findings document (the deliverable):** `documentation/audits/SPIKE-restic-restore-test-2026-08-31.md` +**Evidence:** `documentation/audits/evidence-spike-restic-restore-2026-08-31/` — 31 files. + +--- + +## 1. Baselines, re-checked at the start + +| repo | `main` @ | version | matched the task's stated baseline? | +|---|---|---|---| +| `felhom-controller` | `2d802d75e88616d86cbade8a0e16965c2b85771c` | v0.230.0 | yes | +| `felhom.eu` | `dddcc808be95d1c89b276b4d791491bad3c96bba` | — | yes; clean tree, `HEAD == origin/main` | +| `felhom-agent` | `058b945` | v0.130.0 | yes | + +## 2. Part 1 — the register correction + +- **The mis-filing commit:** `ef6ac6f`, 2026-08-22, *"One register, enforced by a gate; closed work + compressed into siblings (R-376..R-378)"*. Established by `git log -S '**R-87**'` on both register + files — it is the only commit that added the row to `CLOSED-ITEMS.md` and the only one that removed + it from `OPEN-ITEMS.md`. **The history settled it; no guess was needed.** +- **It is a survivor of R-378, not a separate incident.** R-378 records six rows moved wrongly by that + same sweep and restored in the same session. R-87 is a **seventh it missed**, and it escaped because + its state cell led with `READY` and carried the word `closed` later, describing a different row. +- **My own count, reproduced:** **1** mis-filed row under the leading-verdict predicate. The same scan + convicts **3** if it reads the whole state cell (R-224 and R-260 are false positives — long prose + verdicts containing "open"/"OPEN") and **144** if it reads the whole row. The task author's count of + one is confirmed, and only under the predicate R-378 argues for. +- **The gate:** `scripts/closed_register_gate.py`. Two rules. **Red-proof rule 1:** a planted `READY` + row convicts by name, rc=1; removing it leaves the file byte-identical. **Red-proof rule 2:** a + planted duplicate id convicts, rc=1. **Negative control:** run against the files *as pushed* + (`HEAD:`), it convicts R-87 at L72 and R-398 at L139, rc=1. **Registered LAST**, as the 12th gate in + `repo_gates.py`, after it was green. +- **Also corrected:** `R-398` had a row in both registers (a deliberate cross-reference stub). Now + prose beneath the table. + +## 3. Q1–Q7, one paragraph each + +**Q1 — restic 0.14.0**, `go1.19.8`, Debian bookworm 12.15, from the running container. The four source +comments asserting 0.14.0 are **confirmed**. Method: `docker exec felhom-controller restic version`. + +**Q2 — `--verify` exists and is NOT a content check.** `restic restore --help` lists +`--verify verify restored files content`; positive control `--target` = 1 hit, negative controls +`--delete/--dry-run/--overwrite/--sparse` and a nonsense string = 0 hits each. Neither `--verify` nor +`--no-lock` appears anywhere in the controller source (`grep -rn` rc=1, with `--json`/`--target` as +the positive control). **Red-proof:** one byte changed in a restored 160 MB tar with size and mtime +preserved — `restore --verify` **passed clean, rc=0**. Verify took **131 ms** on a 213 MB / 7-file +tree, which cannot be hashing. A size or mtime mismatch causes a silent **re-download**, not a +failure. **So restic cannot tell us a restore produced correct files.** + +**Q3 — no reference for "correct" exists today.** `restic ls --json` file nodes in 0.14.0 carry no +content hash. The recovery unit's `manifest.json` hashes three config files — **4 918 B of a +213 231 242 B unit, 0.0023 %** — and not the DB dump or the volume tars. A planted sentinel is a drill +technique and does not transfer; the live data drifts. **What the manifest CAN answer is +completeness**, through the existing `unitCarriesData` (`r403_hollow.go:40`), with no new metadata. +Filed as R-409. + +**Q4 — ~4 s per app, 25 s for all eight, cheaper than the weekly check.** Through the product's own +path: docmost unit 9 s, kimai full 11 s. Raw restic, all 8 snapshots / **774 378 123 B logical** back +to back: **25 s**, individual times 2 253–3 978 ms *regardless of size* (185 KB → 2.25 s, 213 MB → +3.20 s). The cost is per-snapshot round-trip plus ≈ 1 s per 200 MB. Peak scratch = the app's full +logical size, 213 272 202 B for the largest. The restic cache is **1.1 MB** (index only) and does not +hide the cost: `--no-cache` 5 423 ms vs cached 3 198 ms, trees byte-identical. **Against R-359's +35.0 s / 39.2 s: the same 100 % check re-measured today is 40 257 ms — so restore-testing the whole +box costs LESS than one weekly check.** Extrapolation to 10×/100× is in the findings doc, **labelled +as extrapolation**, with scratch space named as the constraint that binds before time does; the +single-store hole is R-401's. + +**Q5 — skip-if-busy stays right, and a bigger thing is wrong.** Scheduler registrations read off the +box (CEST): db-dump 02:30, tier2 03:30, **offbox-backup 04:15 (2m52s measured)**, abandon-sweep 05:10, +**offsite-integrity 06:00 (40.3 s)**. A 25 s hold is seconds, not minutes, and there is an empty gap +04:18–06:00. **But `RestoreOffboxScratch` takes no `acquireRunning` at all** — nine non-test callers, +it is not one — while `offbox_integrity.go:28` asserts *"Every off-site operation takes +`acquireRunning`"*. `restore_wizard.go:174` records the same fact independently. Filed as **R-408**. + +**Q6 — the restore itself writes nothing; the product writes anyway; and `check` writes a lock.** +Observed with a lock sampler and an argv sampler, both inside the container, the repo URL redacted at +source. **Positive control:** across the product's integrity run the repo went `locks=0` → +`locks=1 id=81fd4d42…` for nine consecutive samples → `locks=0`. **The same instrument saw zero locks +across two restores**, so `restic restore` in 0.14.0 does not lock. Observed argv for one restore: +`snapshots latest --tag --json`, then **`unlock`**, then `restore --target …`. The middle +one is `unlockStale` (`offbox_restore.go:289`), unconditional, a **delete verb**. **§5's lead was +right in direction and wrong in mechanism** — the write is `unlockStale`, not `resticStep`'s +escalation. **The constraint IS satisfiable:** `--no-lock` + skipping `unlockStale` writes nothing, +and both mechanisms exist in 0.14.0 unused. `offbox_integrity.go:255`'s *"It NEVER writes to the +repository"* is filed as **R-407**. **Neither was fixed** — §7 forbids it. + +**Q7 — one of five.** R-353 (a *local* restore path) **no**; R-354 (no volume-replay leg, after the +scratch) **no**; **R-356 (refused every driveless app) YES — five of eight apps on this box would +have fired it on the first night**; R-358 (needs a part-way failure) **no**; R-403 (destroyed a +*local* copy) **no as filed, yes for the shape**. **The honest verdict is "few", and it points +elsewhere:** `check` proves the stored bytes are the stored bytes, never that we stored the *right* +thing. A hollow unit backs up, checks at 100 % and restores cleanly, and recovers nothing — R-403, +measured in bytes nine days ago. Nothing asks that question on any tier. + +## 4. Recommendation + +Three options with costs and a do-nothing outcome are in the findings document. **I would pick option +C — the narrow test:** one app a night, restored to scratch, checked against its own `manifest.json`, +scratch deleted, the snapshot recorded as the proof. ~4 s and ≤ 213 MB per night; catches R-356 and +the R-403 class; needs no new metadata. **It must use `--no-lock`, skip `unlockStale`, and take +`acquireRunning`** — all three established by this spike. **Options A (do not build) and B (scheduled +attended drill) were considered explicitly and are argued in the document; B is the weakest, because +it is what already happens.** **R-87 should be RE-SCOPED, not built as written — and that is Viktor's +call**, so the row stays open carrying the verdict, and `STATUS.md` item 4 asks it in plain words. + +## 5. Evidence + +`documentation/audits/evidence-spike-restic-restore-2026-08-31/`, 31 files, numbered by question. +**Every file was pulled off the box before any teardown** (R-320) — including the two in-container +sampler logs, which were `cat`ed to DooPlex before the container `/tmp` was cleared. + +## 6. Probes removed — all three layers, and none of them is "nothing was created" + +| layer | created | after teardown | +|---|---|---| +| PVE host `/root` | 6 scripts + one 0600 password file | `ls \| grep` → nothing | +| guest 9201 `/root`, `/tmp` | 9 files | grep → nothing | +| container `/tmp` | 5 files + 2 run-flags | `/tmp` lists empty; no restic process left | + +**Scratch directories:** the four created by this session's restores (`docmost`, `kimai`, +`privatebin`, `opengist`) were removed. Three (`bookstack`, `calibre-web`, `paperless-ngx`) pre-date +this session and were **left alone**. The local password copy was `shred -u`'d. + +**Two state changes recorded rather than hidden:** the integrity check run as the lock positive +control **recorded its verdict** (`last_integrity_check` → `2026-08-31T13:41:28Z`, depth `structure` → +**`100%`**, due-ness advanced 7 days), and four restores plus two logins appear in the controller log. +**Nothing was written to the off-site repository by hand.** + +## 7. Register + +| id | action | +|---|---| +| **R-87** | **moved back to `OPEN-ITEMS.md`** (verbatim from `ef6ac6f^`, beside R-95), then updated with the spike verdict and a re-scope proposal | +| **R-398** | de-tabled in `CLOSED-ITEMS.md`; the open row is the record | +| **R-404** | bypass count corrected six → **seven** (this session's Part 1 push) | +| **R-405** | filed + **CLOSED** — the mis-file, the reproduced count, the gate | +| **R-406** | filed — two findings share the id R-133 | +| **R-407** | filed — `restic check` takes a lock; the comment says it never writes | +| **R-408** | filed — `RestoreOffboxScratch` takes no `acquireRunning` | +| **R-409** | filed — the unit manifest hashes 0.002 % of the unit | + +**Register size:** `OPEN-ITEMS.md` **165 → 171** rows (+R-87 restored, +R-405..R-409); `CLOSED-ITEMS.md` **153 → 151** (−R-87, −R-398). +Ceiling **R-404 → R-409**. + +`python3 scripts/unproven.py --summary`: 55 claims, walked 20 / partial 17 / built 14 / missing 4, +**NOT WALKED 35 of 55 — unchanged by this session**, which shipped no product claim. + +## 8. No controller code changed and no golden is owed + +`felhom-controller` and `felhom-agent` were **read only**. No version bump, no build, no deploy, no +golden. The fleet stays on v0.230.0. **The one golden debt that exists — v0.230.0 released with the +newest bake at 0.229.0 — was already red at `dddcc80` before this session started** and belongs to +that release, not to this task; `golden_currency_gate.py` was the only failing gate at every point in +this session, before and after. + +**`git push --no-verify` was used, twice, for exactly that reason** — records-only pushes meeting the +golden gate. That is R-404's subject and the count is updated in its row. + +## 9. Observations — noticed, not acted on + +- `paperless-ngx` has an off-site snapshot under that tag and none under `paperless`; `filebrowser` + has none at all (it is infrastructure, so that may be correct). Not chased. +- `CLOSED-ITEMS.md` rows **R-399** and **R-400** supply two columns where the table declares four — + they render with no `Shipped` and no `Evidence`. The new gate warns rather than convicts, because an + empty state cell is not an open state word. +- Two rows (**R-309**, **R-351**) carry a `|` inside their body, shifting their own cells. Named as + the gate's first residual hole. +- The controller image has **no `ps` and no `python3`**. `/proc/*/cmdline` is the substitute that + works, and it is worth knowing before writing any probe that runs in there. +- The guest scheduler logs in **CEST**, not UTC — `offbox-backup scheduled for 2026-09-01 04:15 CEST` + against a `last_run` of `02:17:57Z`. Consistent, and the opposite of what the project memory says + about guest time. diff --git a/STATUS.md b/STATUS.md index a8d814ea..9d9069bb 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,6 +1,10 @@ # STATUS — what works, what's broken, what's next -**Updated 2026-08-31 — the weekly off-site check now re-reads your actual data, not just the list of +**Updated 2026-08-31 (second pass) — I measured whether the box could test its own off-site +restore without you. It can, and it is cheap — but not in the shape we had written down, so +there is a decision for you in item 4. No code changed today.** + +**Earlier 2026-08-31 — the weekly off-site check now re-reads your actual data, not just the list of it. A third of the debug page did nothing and no longer exists. 0.228.0 is baked, vouched and delivered; both demo machines are on it and nothing is waiting on you about this release.** @@ -24,9 +28,9 @@ nothing.* controller image; no customer action, no data migration, no credential change. 3. **Whether a documents-only push should still be checked for a missing golden** (R-404). We have now - skipped that check **six times**, each time for a written reason: it runs on every push to the + skipped that check **seven times**, each time for a written reason: it runs on every push to the website/documentation repository, including pushes that change nothing a machine installs. - **A guard we correctly skip six times is teaching us to skip it.** + **A guard we correctly skip seven times is teaching us to skip it.** **The case for narrowing it:** a documents-only push cannot be the one that finishes a release, so only checking pushes that touch real code would fire on exactly the risky ones and end the habit. **The case against:** the check was earned — a release went out while machines were still being @@ -35,11 +39,25 @@ nothing.* **If you do nothing:** nothing breaks, the skipping stays routine, and the count keeps rising. I have NOT changed it; this is yours to decide and mine to build. -4. **Whether to change the hub password** (R-350). I printed it into my own session log on 20 August. +4. **Whether to have the box check its own off-site RESTORE every night** (R-87). I measured it + today instead of guessing. It is cheap: restoring **every** app on `demo-hp` — 8 backups, 774 MB — + took **25 seconds**, less than the 40 seconds the weekly check beside it already takes. But it + would catch **one** of the five restore faults we found by hand in the last six days, so the + version the old note asked for is not worth building. + **The version that IS worth building is a different question:** the weekly check proves the stored + bytes are the stored bytes. It cannot tell us we stored the **wrong thing** — an empty recovery + package backs up, checks and restores perfectly and gives the customer nothing back. That is not a + theory; it happened on 31 August (R-403). A nightly check of one app against its own packing list + would catch it and needs nothing new built underneath. + **If you do nothing:** the weekly check keeps being right about the bytes, and the first empty + package will be found by a customer trying to restore. + **My pick:** build the narrow version. **Yours to decide**, and I changed no code today. + +5. **Whether to change the hub password** (R-350). I printed it into my own session log on 20 August. Not in git, not in any saved file — in the log on this machine. **If you do nothing:** it stays as it is, at the risk you accept by leaving it. I can change it without ever showing you the new one. -5. **`demo-hp`'s network setup does not match our own notes** (R-338) — the machine works, the page is +6. **`demo-hp`'s network setup does not match our own notes** (R-338) — the machine works, the page is wrong, or the other way round. **If you do nothing:** the page keeps misleading the next session, as it misled one by an hour. diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 29c91008..ad7980f0 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -98,7 +98,7 @@ with nothing but their dashboard password. No operator, no ticket, no scheduling | 1 | „Visszaállítás indítása" | `POST /backup/restore` | destructive — rebuilds the app from its Tier-1 unit | | 2 | „Fájlok visszaállítása" | `POST /backup/tier2/restore` | additive, missing-only | | 3 | „Visszaállítás a távoli tárolóból" | `POST /backup/offbox/restore` | non-destructive, to a verification copy | -| 4 | „Helyreállítás az élő adatok közé" | `POST /backup/offbox/place` | additive, missing-only **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open), and the depth that ships ON does not re-read pack contents (R-399).** | +| 4 | „Helyreállítás az élő adatok közé" | `POST /backup/offbox/place` | additive, missing-only **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. | | 5 | „Teljes visszaállítás (fájlok + adatbázis)" | `POST /backup/offbox/reconstitute` | destructive to files + DB, never deleting | | 6 | „Megosztások visszaállítása" + place | `POST /backup/shares/{restore,place}` | additive | | 7 | `.fab` import | `POST` → `apiImportStart` | destructive re-import of one app | @@ -869,7 +869,7 @@ crosses the line — **R-158**. | 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human | | 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) | | 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) | -| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open), and the depth that ships ON does not re-read pack contents (R-399).** | +| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. | | 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) | | 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) | | 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D | @@ -1079,7 +1079,7 @@ does **not** hold as written. → **R-108** | ~~R-359~~ | ~~The off-site restic store is never verified by anything, ever~~ | **CLOSED 2026-08-30, controller v0.227.0/v0.227.1.** A daily `offsite-integrity` job on **due-ness, not a weekday**; it takes the single-writer flag and SKIPS rather than waits (`resticStep` escalates to `unlock --remove-all` and is only safe while that flag is held). Three outcomes — skipped / unreachable / failed — because 'I could not look' is not 'I looked and it is broken'. **⚠ The depth that ships ON does NOT catch silent corruption:** measured, a pack corrupted without a size change returned `no errors were found`, exit 0; only `--read-data*` caught it. Choosing the depth is **R-399** | | ~~R-397~~ | ~~`NotifyIntegrityOK`/`NotifyIntegrityFailed` had no caller and the product advertised a weekly check that did not exist~~ | **CLOSED 2026-08-30, controller v0.227.0.** Sixth built-but-never-wired instance: hub allowlist, Hungarian text, settings checkbox and debug button all existed; only the caller did not. `ok` is severity `info` and mails nobody by design | | ~~R-399~~ | ~~The check reads the catalogue and never the data~~ | **CLOSED 2026-08-31, controller v0.228.0.** `monitoring.integrity.read_data_subset` now defaults to **`100%`**, so the weekly check downloads and re-hashes every stored byte. **The fact that made it necessary, and the sentence that should stop anyone turning it back down to save four seconds: the structure check PASSED a size-preserving pack corruption.** Measured on `demo-hp` 2026-08-30 — plain `restic check` reported `no errors were found` and exited 0 over a pack damaged without a size change; every read-data form caught it. Cost on that 134 MB store: 35.0 s structure vs 39.2 s at 100%. `off` (any case) returns a box to structure depth; an empty value means *not configured*, therefore the default; a malformed value WARNs and falls back to the DEFAULT, never to structure. A completed check over 5 minutes logs an operator WARN naming R-401 — **one data point, on one 134 MB store, so no rotation schedule, size threshold or bandwidth budget was invented from it.** Proven live at both depths 2026-08-31 with the restic argv observed from the guest | -| **R-87 (open) — AND IT IS NOT R-359** | The restic tier is never restore-TESTED | matrix row 4's route has no unattended proof. **Stated explicitly because the two rows sit next to each other and are easy to conflate: a `check` proves the STORE IS READABLE; a restore-test proves DATA COMES BACK OUT.** v0.227.0 did the first and nothing else. This row is untouched by it | +| **R-87 (open — SPIKED 2026-08-31, RE-SCOPE PROPOSED) — AND IT IS NOT R-359** | The restic tier is never restore-TESTED | **SPIKE VERDICT, `audits/SPIKE-restic-restore-test-2026-08-31.md`:** build the NARROW version, not the row as written. **Measured:** a scratch restore of all 8 apps costs **25 s / ≤213 MB scratch**, LESS than the 40.3 s weekly check beside it; but restic 0.14.0's `--verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving corruption passed clean, red-proofed), `restic ls --json` carries no content hash, and the unit manifest hashes **4 918 B of a 213 231 242 B unit** — so **no reference for "correct" exists** (R-409). **Of the five drill-found restore defects R-353/354/356/358/403, an unattended scratch-restore would have caught ONE (R-356).** The value is elsewhere and the weekly check structurally cannot reach it: `check` proves the stored bytes are the stored bytes, never that we stored the RIGHT thing — a hollow unit backs up, checks and restores cleanly and recovers nothing (R-403, measured 2026-08-31). **Proposed re-scope, Viktor's call:** *prove the off-site snapshot still CONTAINS a recoverable unit* — one app a night, restored to scratch, checked against its own `manifest.json` via the existing `unitCarriesData`. Must use `--no-lock` and skip `unlockStale` (R-95's constraint is otherwise violated — R-407/R-408 record what the path writes today) and must take `acquireRunning`, which `RestoreOffboxScratch` does not. | matrix row 4's route has no unattended proof. **Stated explicitly because the two rows sit next to each other and are easy to conflate: a `check` proves the STORE IS READABLE; a restore-test proves DATA COMES BACK OUT.** v0.227.0 did the first and nothing else. This row is untouched by it | ### 10.3 Divergences that are documented elsewhere and are not re-opened here diff --git a/documentation/audits/SPIKE-restic-restore-test-2026-08-31.md b/documentation/audits/SPIKE-restic-restore-test-2026-08-31.md new file mode 100644 index 00000000..9e383297 --- /dev/null +++ b/documentation/audits/SPIKE-restic-restore-test-2026-08-31.md @@ -0,0 +1,465 @@ +# SPIKE — can the off-site (restic) copy be restore-tested without a person? (R-87) + +**Date:** 2026-08-31 · **Venue:** `demo-hp` (Tier 0, disposable), guest 9201, controller **v0.230.0** +**Class:** spike. **No production code was written.** No version bump, no build, no deploy, no golden. +**Evidence:** `documentation/audits/evidence-spike-restic-restore-2026-08-31/` — 31 files, all pulled +off the box before teardown (R-320). + +**Baselines re-confirmed at the start of the session:** `felhom-controller` `main` @ +`2d802d75e88616d86cbade8a0e16965c2b85771c`, `v0.230.0`; `felhom.eu` @ +`dddcc808be95d1c89b276b4d791491bad3c96bba` (clean tree, `HEAD == origin/main`); `felhom-agent` +`058b945`, `v0.130.0`. + +--- + +## The one-paragraph answer + +**restic 0.14.0 cannot tell us a restore produced correct files, and neither can anything else the +box holds today** — `--verify` exists but checks size and modification time, not content, and the +recovery unit's own manifest records a hash for 4 918 bytes of a 213 231 242-byte unit. **But an +unattended restore-test is far cheaper than expected:** restoring *every* app on the box — 8 +snapshots, 774 MB logical — took **25 seconds**, against **40.3 seconds** for the weekly integrity +check sitting beside it. And the question worth asking is not the one R-87 was filed for. Of the five +restore-path defects human drills found between 2026-08-26 and 2026-08-31, an unattended +scratch-restore would have caught **one**. What it *would* catch, and what the weekly check +structurally cannot, is a snapshot that is perfectly intact and contains **nothing recoverable** — +the R-403 shape, measured live nine days ago. **Recommendation: build the narrow version (option C +below), and do not build the thing R-87 asks for.** + +--- + +## Part 1 — the register correction (done, pushed as `6e550ae`) + +**When and where it went wrong.** R-87 was moved into `CLOSED-ITEMS.md` by commit **`ef6ac6f`**, +2026-08-22, *"One register, enforced by a gate; closed work compressed into siblings +(R-376..R-378)"*. Established from `git log -S`, not inferred: that commit is the only one that ever +added an R-87 row to `CLOSED-ITEMS.md` and the only one that removed it from `OPEN-ITEMS.md`. + +**It is a survivor of R-378, not a separate incident.** R-378 records that same sweep moving six +still-open rows — R-123, R-190, R-214, R-264, R-295, R-352 — and restoring them verbatim in the same +session. **R-87 is a seventh it missed.** Its state cell read +`READY — RE-RANKED UP 2026-08-03 (R-86 closed)`: the leading verdict is `READY`, and the word +`closed` later in the same cell describes a *different* row. Nine days in the wrong file, while +`OPEN-ITEMS.md`'s ranking paragraph ranked it fourth and pointed at nothing. + +**The count, reproduced independently — and the predicate decides the answer.** + +| predicate | rows convicted in `CLOSED-ITEMS.md` (of 151) | which | +|---|---|---| +| open word anywhere in the **row** | 144 | meaningless | +| open word anywhere in the **state cell** | 3 | R-87, **R-224**, **R-260** | +| open word in the **leading verdict** | **1** | R-87 | + +R-224 and R-260 are genuinely closed; their long prose verdicts merely contain the words "open" and +"OPEN". **The task author's count of one is right, and it is right only under the leading-verdict +predicate** — which is R-378's own lesson, restated by measurement. + +**The gate:** `scripts/closed_register_gate.py`, two rules — no open state word leading a +`CLOSED-ITEMS.md` row's verdict, and no `R-` id with a row in both registers. Red-proofed on both +rules; negative-controlled against the files **as they were pushed**, where it convicts R-87 by name +and exits 1. Registered as the 12th gate in `repo_gates.py` **after** it was green. Four residual +holes are named in its docstring. + +**One thing the second rule turned up:** `R-398` also had a row in both registers — a deliberate +cross-reference stub. It is now prose beneath the table, not a table row. + +**A duplicate this session did NOT fix:** `OPEN-ITEMS.md` carries two unrelated findings both +numbered **R-133** (`:267` hub `customer_id` uniqueness; `:273` plaintext break-glass credential). +Filed as **R-406**; deliberately not gated, because a within-register duplicate rule would fail on a +pre-existing row and a registered-but-failing gate refuses every push. + +--- + +## Q1 — what restic is actually running? + +**Answer: restic 0.14.0, and every source comment asserting that is correct.** + +From the running container on `demo-hp`, not from the Dockerfile: + +``` +restic 0.14.0 compiled with go1.19.8 on linux/amd64 +/usr/bin/restic +/etc/debian_version → 12.15 (Debian bookworm, as the Dockerfile says) +``` + +Method: `pct exec 9201 -- docker exec felhom-controller restic version`, rc=0. Evidence +`02-q1-restic-version.txt`. The comments at `offbox_capture.go:15`, `offbox_restore.go:21`, +`offbox.go:726`, `offbox_progress.go:48,66` are **confirmed, not corrected**. + +--- + +## Q2 — what can that version verify about a RESTORE? + +**Answer: `--verify` exists, and it does NOT verify content. It cannot tell us a restore produced +correct files.** + +**`--verify` is real.** `restic restore --help` on the running container lists +`--verify verify restored files content`. Controls, because a `grep -c 0` must be earned: + +| control | result | +|---|---| +| positive — `--target` present | 1 hit | +| `--verify` present | 1 hit, quoted verbatim above | +| negative — `--delete`, `--dry-run`, `--overwrite`, `--sparse` (all post-0.14) | 0 hits each | +| negative — `ZZZ-NOT-A-FLAG` | 0 hits | + +Evidence `03-q2-restore-help.txt`, `04-q2-controls.txt`. **`--verify` and `--no-lock` both exist in +0.14.0 and neither appears anywhere in the controller source** — `grep -rn` over +`felhom-controller/controller/` returns rc=1 for both, with `--json`/`--target` as the positive +control (`05-q2-codebase-verify-grep.txt`). + +**What `--verify` actually checks — measured, with a red-proof and a negative control.** + +| test | what was done | result | +|---|---|---| +| cost | virgin restore of kimai (213 231 242 B, 7 files) with `--verify` | verify itself **131 ms**; restore+verify 3 438 ms | +| **red-proof** | corrupt one byte in a restored 160 MB tar, **size and mtime preserved**, re-run `restore --verify` | **PASSED clean, rc=0** — the corruption was not detected | +| control (size) | truncate the same file by 1 byte, re-run | restic silently **re-downloaded** it; verify reported "7 files, 133 ms", rc=0 | +| control (mtime) | corrupt content and bump mtime, re-run | restic silently **re-downloaded** it; rc=0 | +| negative control | same command against the untouched copy | identical output — so "passed" carries no information | + +**131 ms cannot hash 213 MB.** Combined with the red-proof, `--verify` in 0.14.0 is a size-and-mtime +reconciliation that re-fetches anything that disagrees. It is useful — it makes a restore +self-repairing — and it is **not** a content check. Evidence `20-`, `21-`, `22-`. + +**This is the point at which the spike's shape changed**, per §9's instruction to stop and +reconsider after Q1/Q2: the tool cannot supply the reference, so Q3 became the hard question. + +--- + +## Q3 — if not restic, then what is the reference for "correct"? + +**Answer: there is none available to an unattended test today. The one hash record that exists covers +0.002 % of a unit's bytes.** + +| candidate | verdict | why | +|---|---|---| +| restic `--verify` | **rejected** | size + mtime only — Q2's red-proof | +| the snapshot's own metadata | **rejected** | `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps — **no content hash** (`24-q3-hash-coverage.txt`) | +| a hash the controller already records | **rejected as a content reference, kept as a completeness one** | see below | +| `restic check --read-data` | **already shipped, different question** | it proves the STORE's packs, never the restored files | +| a planted sentinel | **rejected** | a drill technique. An unattended test may not write data into a customer's app to have something to look for | +| the live data | **rejected** | it drifts by design; the snapshot is 12 h old by the time a check runs | +| restore twice and compare | **rejected** | proves determinism, not correctness | + +**The hash record that exists, and its exact coverage.** The recovery unit's `manifest.json` carries +a `checksums` object. For kimai, measured on the restored unit: + +``` +checksums: .felhom.yml (2 235 B), app.yaml (488 B), docker-compose.yml (2 195 B) = 4 918 B +files in the unit: + kimai-mariadb.sql 48 217 B + + kimai_kimai_db_data.tar 160 331 776 B + + kimai_kimai_var.tar 52 845 056 B + + manifest.json 1 275 B total 213 231 242 B +``` + +**4 918 of 213 231 242 bytes — 0.0023 %.** The three config files are hashed; the database dump and +the two volume tars, which are the recoverable data, are not. Filed as **R-409**. + +**What the manifest CAN answer is a different and better question.** It declares `db_dumps` and +`volume_dumps` by name, and `unitCarriesData` (`r403_hollow.go:40`) already asks it. A restored unit +can therefore be checked for **completeness** — does every file the manifest declares exist — with +no new metadata, no new reference, and no content hash. That is the whole of the recommendation in +§Recommendation. + +--- + +## Q4 — what does one restore-test cost? + +**Answer: about 4 seconds per app and 25 seconds for the whole box — cheaper than the weekly +integrity check it would sit beside. The cost is dominated by per-snapshot round-trip, not by data +volume.** + +**Through the product's own path** (`POST /backup/offbox/restore` → `RestoreOffboxScratch`), timed +from the POST to the completion log line: + +| app | mode | logical size | wall clock | +|---|---|---|---| +| docmost | unit | 118 207 270 B | **9 s** (13:40:10 → 13:40:19) | +| kimai | full | 213 231 242 B | **11 s** (13:42:54 → 13:43:05) | +| opengist | unit | 185 664 B | ~8 s | + +**Raw restic, all eight snapshots back to back** (`26-q4-all-apps-cost.txt`): + +| app | logical | restored to disk | ms | +|---|---|---|---| +| privatebin | 2 123 006 | 2 159 870 | 2 628 | +| opengist | 185 664 | 222 528 | 2 253 | +| calibre-web | 5 808 703 | 5 878 335 | 3 642 | +| paperless-ngx | 83 473 654 | 83 547 382 | 3 062 | +| bookstack | 166 641 768 | 166 682 728 | 2 745 | +| docmost | 118 207 270 | 118 248 230 | 3 676 | +| romm | 184 706 816 | 184 747 776 | 3 978 | +| kimai | 213 231 242 | 213 272 202 | 3 202 | +| **all eight** | **774 378 123** | **774 759 051** | **25 s** | + +**185 KB takes 2.25 s and 213 MB takes 3.20 s.** Nearly all of it is fixed per-snapshot cost — +opening the repo, loading the index, one SFTP session. Data adds roughly **1 s per 200 MB** +(≈ 67 MB/s on this link). + +**The cache is not hiding the cost.** `/root/.cache/restic` is **1.1 MB** — index and metadata only. +Cached 3 198 ms vs `--no-cache` 5 423 ms for kimai; the trees are byte-identical (`diff -r` → YES). +Pack data always crosses the wire. Evidence `25-q4-cache-effect.txt`. + +**Peak scratch:** the restore writes the **full logical size** — 213 272 202 B for the largest app. +Sequential-with-cleanup needs only the largest app; all-at-once needs 774 MB. + +**Against R-359's numbers.** R-359 measured 35.0 s at structure depth and 39.2 s at 100 % read-data +on a 140 829 678 B store. Re-measured today through the product's own debug button on a +141 959 062 B store: **40 257 ms at 100 %** (`15-q5-integrity-run.txt`). So: + +> **restore-testing every app on the box (25 s) costs LESS than one weekly integrity check (40 s).** +> Same order of magnitude, and on the cheaper side of it. + +**Extrapolation — labelled as extrapolation.** R-401 records that the check's cost tracks the index +and a read-data run's tracks the data. **A restore tracks the data too, plus a fixed per-snapshot +cost.** With 8 apps and the measured ≈ 67 MB/s: + +| store | fixed (8 × ~2 s) | data | total | +|---|---|---|---| +| today, 774 MB logical | 16 s | ~9 s | **25 s (measured)** | +| 10× — 7.7 GB | 16 s | ~115 s | ~2.2 min | +| 100× — 77 GB | 16 s | ~19 min | ~19 min | + +**Why it may not hold.** One store, one link, one afternoon. This is DooPlex→Hetzner; a customer's +domestic line is the real variable, and restore is the *download* direction, usually the faster one +on such a line. Scratch space becomes the binding constraint long before time does: at 100× the +largest app would need ~21 GB of free scratch, and `RestoreOffboxScratch`'s headroom gate would start +refusing. **Unknown, and what would settle it:** a measurement on a store above 10 GB. None exists. +This is the same single-data-point hole R-401 already owns. + +--- + +## Q5 — contention + +**Answer: skip-if-busy stays right, because the operation is seconds and not minutes — but the +restore path takes NO single-writer flag at all today, which is a bigger problem than contention.** + +**The configured window on this box**, read from the scheduler's own registrations rather than +assumed (`28-q5-schedule.txt`, times are **CEST** — the guest scheduler runs in local time, not UTC): + +| job | time | measured duration | +|---|---|---| +| db-dump | 02:30 | — | +| tier2-backup | 03:30 | — | +| **offbox-backup** | **04:15** | **2m52s** (last run, `last_duration`) | +| offsite-abandon-sweep | 05:10 | — | +| **offsite-integrity** | **06:00** | **40.3 s** at 100 % depth | + +A restore-test of all eight apps holds anything it holds for **25 s** — one seventh of the nightly +off-site backup, and shorter than the check beside it. There is an empty gap from ~04:18 to 06:00. +**Skip-if-busy remains the right policy**, and it is not the "minutes rather than seconds" case the +task worried about. The `integrityCheckTimeout` reasoning transfers unchanged. + +**The real finding here.** `offbox_integrity.go:28` states the invariant: + +> *"Every off-site operation takes `acquireRunning` for exactly that reason."* + +**It does not.** `grep -rn 'acquireRunning()'` finds nine non-test callers; `RestoreOffboxScratch` +(`offbox_restore.go:234`) is **not** among them. `restore_wizard.go:174` says the same thing +independently — *"`RestoreOffboxScratch` never acquires it at all"* — and the UI works around it with +a separate display flag. So a scheduled, unattended caller built on `RestoreOffboxScratch` today +would run with **no single-writer flag**, which is precisely the hazard the integrity file's header +is shaped around. **A comment asserting an invariant with no test pinning it.** Filed as **R-408**. + +--- + +## Q6 — what does a restore-test WRITE to the repository? + +**Answer: `restic restore` writes nothing — it does not even take a lock. The product's restore path +writes anyway, because it runs `restic unlock` before every restore. And `restic check` — the thing +whose comment says it never writes — DOES take a lock.** + +**Method.** Two observers inside the container, both proven before being believed: + +1. a lock sampler running `restic list locks --no-lock` every ~4 s (the observer itself never writes); +2. an argv sampler reading `/proc/*/cmdline` every 0.2 s, with the repo URL and `sftp.command` + redacted at the source. + +**Positive control for the lock sampler — it works.** The product's own integrity check ran +13:41:28 → 13:42:11: + +``` +13:41:30 locks=0 +13:41:34 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf + ... nine consecutive samples, one lock ... +13:42:10 locks=1 ids=81fd… +13:42:14 locks=0 +``` + +**So `restic check` writes a lock file to the repository.** `offbox_integrity.go:255` says +*"It NEVER writes to the repository: `check` is a read verb, and nothing here prunes, forgets, +unlocks or backs up."* The three named verbs are correct; the sentence's headline is not. Filed as +**R-407**. + +**The restore, with the same proven instrument:** locks=0 across every sample inside both restore +windows (13:40:11/:15/:19 for docmost, 13:42:57/:43:01/:05 for kimai). **restic 0.14.0's `restore` +does not lock the repository.** + +**What the product runs, observed argv (redacted), one restore of `opengist`:** + +``` +13:44:41 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:44 restic -r sftp: -o sftp.command= unlock +13:44:46 restic -r sftp: -o sftp.command= restore b5aa8f9b --target … --include … +``` + +Three invocations. The middle one is `unlockStale` (`offbox_restore.go:289` → `offbox.go:743`), which +runs **unconditionally before every restore** and is a **delete verb against `locks/`**. With no +stale lock present it deletes nothing — but it is a write-capable command on the exact path R-87 +wants to run unattended. `resticStep`'s escalation to `unlock --remove-all` (`offbox.go:760-775`) +fires only on `repository is already locked`, which a restore cannot provoke by itself now that we +know restore takes no lock. + +**Against §5's constraint** — *"R-95 still applies: that credential can delete, so a restic +restore-test must never be able to write to the repo"*: + +- **The lead in the task was right in direction and wrong in mechanism.** The write is not + `resticStep`'s escalation; it is `unlockStale`, one line earlier and unconditional. +- **The constraint IS satisfiable, cheaply, and both mechanisms already exist in 0.14.0 and are + unused:** a restore-test that passes `--no-lock` and skips `unlockStale` writes nothing to the + repository at all. That is a design note for whoever builds it — **this spike did not fix it**, per + §7. + +--- + +## Q7 — what would an unattended restore-test catch that the weekly 100 % check does not? + +**Answer: of the five defects human drills found in the last six days, one. But that is the wrong +scoreboard, and the right one has a much better answer.** + +| defect | would an unattended scratch-restore have caught it? | how / why not | +|---|---|---| +| **R-353** — a **local** unit restore reported a bare completion whether it returned a dataset or nothing | **NO** | different code path entirely (`restore_unit.go`). An off-site restore-test never enters it | +| **R-354** — the off-site full restore has no named-volume replay leg: the tar reaches the scratch and is never replayed | **NO** | the defect is *after* the scratch. A test that stops at the scratch sees a correct scratch. Going further means replaying into a live app, which an unattended test must not do | +| **R-356** — the off-site restore refused every app with no data drive (40 of 53 catalogue templates) | **YES** | the test calls `RestoreOffboxScratch`, gets a refusal, and `rerr != nil`. On this box five of eight apps live on `/mnt/sys_drive` with no drive — it would have fired on the first night | +| **R-358** — `OffboxFullScratchReady` asked "non-empty directory", which is what a *failed* restic run leaves | **NO** | needs a restore that fails part-way. On a healthy run the broken and the fixed predicate agree | +| **R-403** — a hollow unit mirrored over a complete one with `--delete`; 120 082 104 B → 7 036 B, recorded as success | **NO as filed** — but **YES for the shape** | R-403 destroyed a *local* second-drive copy. What a restore-test sees is the consequence: once a hollow unit is captured off-site, the snapshot is perfectly intact and contains nothing recoverable | + +**One of five. If the question is "does our restore code work", the honest answer is: the drills +already answer it, they answer it better, and automating a worse version of it is not worth an +evening.** + +**The scoreboard that matters is different, and the weekly check structurally cannot play on it.** + +> `restic check --read-data-subset=100%` proves that **the bytes we stored are the bytes we stored.** +> It cannot tell us **we stored the wrong thing.** + +A hollow recovery unit — no database dump, no volume tar — backs up cleanly, checks cleanly at 100 % +depth, restores cleanly, and recovers nothing. **R-403 proved that shape is real, on this fleet, nine +days ago, measured in bytes.** Nothing in the product asks the question today, on any tier, at any +cadence. A restore-test is simply the cheapest place to ask it, because the manifest that answers it +travels inside the snapshot. + +--- + +## Recommendation to Viktor — three options, with costs + +**Option A — do not build it.** +*Cost:* nothing. *What you get:* the weekly 100 % check keeps proving the stored bytes; the restore +code keeps being proven by your drills. +*What you lose:* nothing that has bitten yet — and the R-403 shape stays invisible until a customer +needs the data. **This is a defensible answer** and it is the one R-87's original framing deserves. + +**Option B — a scheduled attended drill instead.** +*Cost:* one of your evenings, monthly. *What you get:* everything a person can see, including the +R-354 class that stops at the scratch. +*Honest objection:* this is what already happens, and it is what found all five defects. Scheduling +it changes nothing except that it now has a date. **Low value for the price.** + +**Option C — build the NARROW unattended test: one app per night, rotating, restored to scratch, and +checked against its own manifest. RECOMMENDED.** + +*What it does, in one sentence:* restore the newest off-site snapshot of one app into the throwaway +scratch, assert that every file the unit's `manifest.json` declares is present, record which snapshot +was proved, delete the scratch. + +*Measured cost, not estimated:* **~4 s and ≤ 213 MB of scratch per night** (one app), or 25 s for all +eight. Off-site traffic: one restore's worth, ≈ the app's size. **Less than the weekly integrity +check already running beside it.** No new metadata, no new reference, no content hashes — it reuses +`unitCarriesData`'s existing manifest read. + +*What it catches:* R-356 outright, and the R-403 class — a snapshot that is intact and empty — which +nothing else in the product asks about. +*What it does not catch, stated so nobody expects it to:* R-353, R-354, R-358. Those stay drill work. + +*Three things it must be built with, all established by this spike:* +1. **`--no-lock`, and skip `unlockStale`** — then it writes nothing to the repository and R-95's + constraint is honoured for real (Q6). +2. **Take `acquireRunning`** — `RestoreOffboxScratch` does not, and the whole off-site + single-writer story assumes every operation does (Q5, R-408). +3. **Record the SNAPSHOT it proved, not a timestamp** — R-87's own row already says this, and R-86 + built the per-archive due-ness model to copy. + +**If you do nothing:** the weekly check keeps running and keeps being right about the bytes. The +first time a hollow unit reaches the off-site store, nothing will notice, and the discovery will be a +customer's restore. That is not a hypothetical shape — it is R-403, measured on 2026-08-31. + +**I would pick C**, scoped exactly as above. It is cheaper than the check beside it, it asks a +question nothing else asks, and it needs no invention. + +**R-87 itself should be RE-SCOPED, not built as written** — from *"restore-test the restic tier"* to +*"prove the off-site snapshot still contains a recoverable unit"*. That is your call, so R-87 stays +open carrying this verdict. + +--- + +## Register rows opened by this spike + +| id | what | +|---|---| +| **R-405** | R-87 was mis-filed by `ef6ac6f`; corrected + gated (CLOSED same session) | +| **R-406** | two unrelated findings share the id R-133 in `OPEN-ITEMS.md` | +| **R-407** | `restic check` DOES take a repository lock; `offbox_integrity.go:255` says it never writes | +| **R-408** | `RestoreOffboxScratch` takes no `acquireRunning`; `offbox_integrity.go:28` asserts every off-site operation does | +| **R-409** | the recovery-unit manifest hashes 4 918 B of a 213 231 242 B unit — the data files have no recorded hash | + +--- + +## Probes, teardown and side-effects + +**All three layers, and two of them really are "nothing was left".** + +| layer | created | removed | +|---|---|---| +| PVE host `demo-hp` `/root` | 6 helper scripts + one 0600 password file | all removed; `ls \| grep` returns nothing | +| guest 9201 `/root`, `/tmp` | 9 helper/session files + `spike-env.sh` | all removed; grep returns nothing | +| container `/tmp` | `spike-env.sh`, two samplers, two logs, two run-flags | all removed; `/tmp` lists empty | + +**Scratch directories:** four (`docmost`, `kimai`, `privatebin`, `opengist`) were created by the +restores this spike drove and were **removed**. Three (`bookstack`, `calibre-web`, `paperless-ngx`) +pre-date this session and were **left alone** — this spike never restored them. + +**Two deliberate state changes on the box, recorded rather than hidden:** + +1. The integrity check I ran to control the lock observer **recorded its verdict** — the box's + `last_integrity_check` moved to `2026-08-31T13:41:28Z`, `ok=true`, and + `last_integrity_depth` moved from `structure` to **`100%`**. Due-ness advanced by seven days. That + is the product behaving correctly; it is not a repair and it is not damage. +2. Two logins and four restores appear in the controller log and in the customer-visible restore-op + history. + +**Nothing was written to the off-site repository by hand.** No prune, no forget, no `unlock` issued +by me. Every write observed in Q6 was the product's own. + +**No controller code changed. No golden is owed.** The fleet stays on v0.230.0. The one delivery debt +that exists — `golden_currency_gate.py` red because v0.230.0 has no golden — was **already red at +`dddcc80`** before this session began, and belongs to the v0.230.0 release, not to this task. + +--- + +## Observations noticed and not acted on + +- **`paperless-ngx` and `filebrowser` have no off-site snapshot** under the tags the inventory uses — + `paperless` returns nothing and `paperless-ngx` returns one; `filebrowser` returns none at all. + `filebrowser` is infrastructure, so that may be correct. Not chased. +- **`CLOSED-ITEMS.md` has two malformed rows** — R-399 and R-400 supply two columns where the table + declares four, so they render with no `Shipped` and no `Evidence`. The new gate prints them as a + warning rather than convicting, because an empty state cell is not an open state word. +- **Two rows carry a `|` inside their body** (R-309, R-351), which shifts their own cells. Named as + the new gate's first residual hole. +- **The controller image has no `ps` and no `python3`** — worth knowing before writing any probe that + runs inside it. `/proc/*/cmdline` is the substitute that works. +- **`git push --no-verify` was used for this session's Part 1 commit** — bypass **#7**, for the reason + R-404 exists: a documentation-only push met `golden_currency_gate.py`. Recorded here because R-404 + counts them. diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/00-venue.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/00-venue.txt new file mode 100644 index 00000000..82b786f3 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/00-venue.txt @@ -0,0 +1,4 @@ +demo-hp + 15:32:56 up 9 days, 21:48, 1 user, load average: 0.70, 0.63, 0.57 +VMID Status Lock Name +9201 running demo-hp diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/01-guest-docker-ps.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/01-guest-docker-ps.txt new file mode 100644 index 00000000..b6918853 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/01-guest-docker-ps.txt @@ -0,0 +1,20 @@ +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.230.0 Up 59 minutes (healthy) diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/02-q1-restic-version.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/02-q1-restic-version.txt new file mode 100644 index 00000000..df4bba49 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/02-q1-restic-version.txt @@ -0,0 +1,4 @@ +restic 0.14.0 compiled with go1.19.8 on linux/amd64 +restic-rc=0 +/usr/bin/restic +12.15 diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/03-q2-restore-help.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/03-q2-restore-help.txt new file mode 100644 index 00000000..d0d36f76 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/03-q2-restore-help.txt @@ -0,0 +1,64 @@ + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + +The "restore" command extracts the data from a snapshot from the repository to +a directory. + +The special snapshot "latest" can be used to restore the latest snapshot in the +repository. + +EXIT STATUS +=========== + +Exit status is 0 if the command was successful, and non-zero if there was any error. + +Usage: + restic restore [flags] snapshotID + +Flags: + -e, --exclude pattern exclude a pattern (can be specified multiple times) + -h, --help help for restore + -H, --host stringArray only consider snapshots for this host when the snapshot ID is "latest" (can be specified multiple times) + --iexclude --exclude same as --exclude but ignores the casing of filenames + --iinclude --include same as --include but ignores the casing of filenames + -i, --include pattern include a pattern, exclude everything else (can be specified multiple times) + --path path only consider snapshots which include this (absolute) path for snapshot ID "latest" + --tag taglist only consider snapshots which include this taglist for snapshot ID "latest" (default []) + -t, --target string directory to extract data to + --verify verify restored files content + +Global Flags: + --cacert file file to load root certificates from (default: use system certificates) + --cache-dir directory set the cache directory. (default: use system default cache directory) + --cleanup-cache auto remove old cache directories + --compression mode compression mode (only available for repository format version 2), one of (auto|off|max) (default auto) + --insecure-tls skip TLS certificate verification when connecting to the repository (insecure) + --json set output mode to JSON for commands that support it + --key-hint key key ID of key to try decrypting first (default: $RESTIC_KEY_HINT) + --limit-download int limits downloads to a maximum rate in KiB/s. (default: unlimited) + --limit-upload int limits uploads to a maximum rate in KiB/s. (default: unlimited) + --no-cache do not use a local cache + --no-lock do not lock the repository, this allows some operations on read-only repositories + -o, --option key=value set extended option (key=value, can be specified multiple times) + --pack-size uint set target pack size in MiB, created pack files may be larger (default: $RESTIC_PACK_SIZE) + --password-command command shell command to obtain the repository password from (default: $RESTIC_PASSWORD_COMMAND) + -p, --password-file file file to read the repository password from (default: $RESTIC_PASSWORD_FILE) + -q, --quiet do not output comprehensive progress report + -r, --repo repository repository to backup to or restore from (default: $RESTIC_REPOSITORY) + --repository-file file file to read the repository location from (default: $RESTIC_REPOSITORY_FILE) + --tls-client-cert file path to a file containing PEM encoded TLS client certificate and private key + -v, --verbose n be verbose (specify multiple times or a level using --verbose=n, max level/times is 3) +=== rc=0 === diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/04-q2-controls.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/04-q2-controls.txt new file mode 100644 index 00000000..41fd7997 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/04-q2-controls.txt @@ -0,0 +1,11 @@ +--- POSITIVE CONTROL: a flag that MUST be there --- +1 +--- the verify flag, verbatim --- + --verify verify restored files content +--- NEGATIVE CONTROL: flags added AFTER 0.14, must be absent --- +--delete hits=0 +--dry-run hits=0 +--overwrite hits=0 +--sparse hits=0 +--- NEGATIVE CONTROL 2: a string that cannot be there --- +0 diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/05-q2-codebase-verify-grep.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/05-q2-codebase-verify-grep.txt new file mode 100644 index 00000000..4389c2de --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/05-q2-codebase-verify-grep.txt @@ -0,0 +1,8 @@ +=== '--verify' anywhere in controller/ (all files, incl gitignored cmd/) === +grep-rc=1 + +=== POSITIVE CONTROL: '--no-lock' === +grep-rc=1 + +=== every restic flag token used in internal/backup/ (non-test) === +"--delete" "--group-by" "--ignore-existing" "--include" "--itemize-changes" "--json" "--keep-daily" "--keep-monthly" "--keep-weekly" "--mode" "--prune" "--read-data" "--remove-all" "--rm" "--tag" "--target" diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/06-guest-state.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/06-guest-state.txt new file mode 100644 index 00000000..9e313ff9 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/06-guest-state.txt @@ -0,0 +1,48 @@ +felhom-controller Up About an hour (healthy) +docmost Up About an hour (healthy) +docmost-redis Up About an hour (healthy) +docmost-postgres Up About an hour (healthy) +romm Up 4 hours (healthy) +romm-db Up 4 hours (healthy) +romm-redis Up 4 hours (healthy) +privatebin Up 4 hours (healthy) +paperless-webserver Up 4 hours (healthy) +paperless-postgres Up 4 hours (healthy) +paperless-redis Up 4 hours (healthy) +opengist Up 4 hours (healthy) +kimai Up 4 hours (healthy) +kimai-db Up 4 hours (healthy) +calibre-web Up 4 hours (healthy) +bookstack Up 4 hours (healthy) +bookstack-db Up 4 hours (healthy) +filebrowser Up 9 days (healthy) +cloudflared Up 9 days +traefik Up 9 days +=== guest disk === +Filesystem Size Used Avail Use% Mounted on +/dev/mapper/pve-vm--9201--disk--0 32G 957M 29G 4% / +/dev/mapper/pve-vm--9201--disk--1 69G 12G 55G 18% /var/lib/felhom +/dev/mapper/pve-root 39G 17G 21G 46% /mnt/felhom-drives +none 492K 4.0K 488K 1% /dev +udev 14G 0 14G 0% /dev/tty +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/0e82b836f4879d75df7380c71546ade764fc20c0f1c7e1d7905ab790ddbc7a4e/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/1924a292e532a1948a03c280bce4b21f48f109765d427ca110044f5e1469fcfe/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/ba706ad1711d7d7d8b31250f124caac3e889a66be994e9bba959d7e9542ee76a/merged +/dev/nvme0n1 938G 6.0G 885G 1% /mnt/felhom-drives/hdd_1 +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/c52ca60013e4af6d6d93a4ca471d9c1dffc7f99a9923869e4d72f47e17dfa44f/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/e79e25da71576531d24348a67d88e2c7d1c9e4105622ae95cd7a40bd6c52cb57/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/8131f6f54062f0094af0e03a1ad26a7c77ed6991e27a10211403cfbb7cb479a1/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/932ec3a6439333b5ecf8b39cb68795e15bdd1dc4a7fb831cae46bd9625ebaeb2/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/a2d41f6c1de3963c93bce576a41d0922695ec4d1460b4ebdd9a398245028182e/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/c0efcc2fbbb80d78ee5d764a3f9f6ab9e622d1df2faac473068aa0486f2c3d66/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/b976192342358497830be3cd329d63ca9546550da2f849b2baa44c041ded1c17/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/a3d1768da870b1d86015d6850275ebba59cb905e4415f9cd399c1bf15b6d08e1/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/a0f134a5b31237fbc56ea1fb38113d3a0352f9d4a3d36e5e40b3cff874b42366/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/1400ead8a09768fd52c279ea50cffbd5b87d061ef83fb93a0fa0685d7e0b027b/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/3723f8b898e197ad12b8fcaef66c08569cc9040d1efd1cba7d8797b5a7f0b14f/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/25ee94f6cfa646a342dd751ca14f58e56f3227aaaf9160d26409c4cc23994453/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/9e92315ab21675a0f271efb4c8eb0ed792a4de69dd58ec5e51300dfa3852c4b9/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/bb1ad0c258566bc3d4013171500fa6a7ed30a80ee620bc93ef8adb939eb0c166/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/2461db88cd4e3b967ab86cdc6a41de3f62b433ccbc1ca0424876a79ac0ba8d10/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/ba1fcae5ed394c9dbdee666392cc61400ef700a69f30ad98fe2de38ef43d7a24/merged +overlay 69G 12G 55G 18% /var/lib/docker/overlay2/688b974f0cd67fd37641d3ce2adb9c696ab09074a966dbbbd0654fd47fc5b57e/merged diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/07-offbox-state.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/07-offbox-state.txt new file mode 100644 index 00000000..668ecadd --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/07-offbox-state.txt @@ -0,0 +1,13 @@ +last_run '2026-08-31T02:17:57Z' +last_success '2026-08-31T02:17:57Z' +last_status 'ok' +last_duration '2m52s' +schedule 'daily' +repo_size_bytes 141959062 +repo_size_human '135.4 MB' +snapshot_count 67 +stats_known True +quota_gb 50 +last_integrity_check '2026-08-31T08:31:18Z' +last_integrity_ok True +last_integrity_depth 'structure' diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/08-q6-locks-before.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/08-q6-locks-before.txt new file mode 100644 index 00000000..8bccc136 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/08-q6-locks-before.txt @@ -0,0 +1,8 @@ +--- locks BEFORE (observer uses --no-lock) --- +lock-count=0 +--- snapshots for docmost --- +ID Time Host Tags Paths +-------------------------------------------------------------------------------------------------------------------- +a07c36a1 2026-08-31 02:17:11 demo-hp felhom-offbox,docmost /mnt/sys_drive/felhom-data/backups/primary/docmost +-------------------------------------------------------------------------------------------------------------------- +1 snapshots diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/09-q4-snapshot-sizes.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/09-q4-snapshot-sizes.txt new file mode 100644 index 00000000..d30f9442 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/09-q4-snapshot-sizes.txt @@ -0,0 +1,13 @@ +=== repo totals === +{"total_size":5958072637,"total_file_count":1380,"snapshots_count":67} + +=== per-app latest snapshot: id, size === +docmost a07c36a1 {"total_size":118207270,"total_file_count":20,"snapshots_count":1} +romm df7ad253 {"total_size":184706816,"total_file_count":18,"snapshots_count":1} +privatebin bb81b9da {"total_size":2123006,"total_file_count":13,"snapshots_count":1} +paperless NO-SNAPSHOT +opengist b5aa8f9b {"total_size":185664,"total_file_count":13,"snapshots_count":1} +kimai e7dcf8fd {"total_size":213231242,"total_file_count":16,"snapshots_count":1} +calibre-web ad2ae49c {"total_size":5808703,"total_file_count":38,"snapshots_count":1} +bookstack 91154be7 {"total_size":166641768,"total_file_count":19,"snapshots_count":1} +filebrowser NO-SNAPSHOT diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/10-q4-pre-restore.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/10-q4-pre-restore.txt new file mode 100644 index 00000000..912e34f5 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/10-q4-pre-restore.txt @@ -0,0 +1,6 @@ +observer running; first samples: +13:39:17 locks=0 +ERR +=== scratch BEFORE === +ls: cannot access '/mnt/felhom-drives/*/felhom-data/backups/offsite-restore/': No such file or directory +949346897920 diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/11-q4-restore-post.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/11-q4-restore-post.txt new file mode 100644 index 00000000..2fd5ef41 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/11-q4-restore-post.txt @@ -0,0 +1,2 @@ +t0=2026-08-31T13:40:10.095555740Z +post_http=302 post_time=0.010129 diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/12-q4-restore-log.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/12-q4-restore-log.txt new file mode 100644 index 00000000..b9d3d727 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/12-q4-restore-log.txt @@ -0,0 +1,6 @@ +DONE after 2 polls +=== controller log, restore window === +2026/08/31 13:40:10 auth.go:134: [DEBUG] [web] auth: valid session for POST /backup/offbox/restore +2026/08/31 13:40:10 server.go:393: [DEBUG] [web] ServeHTTP: POST /backup/offbox/restore from 172.18.0.4:34642 +2026/08/31 13:40:19 offbox_restore.go:298: [INFO] [offbox] restored docmost (a07c36a1, full=false) → /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost +2026/08/31 13:40:19 offbox_handlers.go:372: [INFO] [web] off-box restore docmost completed (full=false, async) diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/13-q4-q6-after-unit.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/13-q4-q6-after-unit.txt new file mode 100644 index 00000000..e1818695 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/13-q4-q6-after-unit.txt @@ -0,0 +1,33 @@ +=== lock observer samples === +13:39:55 rc=0 locks=0 ids= +13:39:59 rc=0 locks=0 ids= +13:40:03 rc=0 locks=0 ids= +13:40:07 rc=0 locks=0 ids= +13:40:11 rc=0 locks=0 ids= +13:40:15 rc=0 locks=0 ids= +13:40:19 rc=0 locks=0 ids= +13:40:23 rc=0 locks=0 ids= +13:40:27 rc=0 locks=0 ids= + +=== scratch AFTER (unit restore) === +118761552 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T140924Z-docmost-postgres.sql +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T140713Z-docmost-postgres.sql +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T140501Z-docmost-postgres.sql +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T135827Z-docmost-postgres.sql +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/docmost-postgres.sql +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/compose/.felhom.yml +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/compose/docker-compose.yml +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/compose/app.yaml +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/volume-dumps/docmost_docmost_postgres_data.tar +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/volume-dumps/docmost_docmost_redis_data.tar +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/volume-dumps/docmost_docmost_storage.tar +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/manifest.json +/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/.felhom-restore-complete.json + +file count: 16 +=== marker === +{"schema":1,"snapshot_id":"a07c36a1","full":false,"finished_at":"2026-08-31T13:40:19Z"} diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/14-q4-all-snapshots.json b/documentation/audits/evidence-spike-restic-restore-2026-08-31/14-q4-all-snapshots.json new file mode 100644 index 00000000..548f0eae --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/14-q4-all-snapshots.json @@ -0,0 +1 @@ +[{"time":"2026-08-09T08:30:38.439582803Z","parent":"48bdfe343537e7abaf96400b525a92ec02086af3db6b7c3ab5dcb17ff68f4e6e","tree":"c37deae059897218d2a3dda39fe32c88a96d4c9c2b648065918ef3289080f662","paths":["/mnt/sys_drive/felhom-data/backups/primary/calibre-web","/mnt/sys_drive/felhom-data/userdata/media/books"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","calibre-web"],"id":"41c830dbe11ecee825f77d222f0cd97534049645784383e444fc019bd86a016f","short_id":"41c830db"},{"time":"2026-08-09T08:30:49.47172713Z","parent":"f2120d883c5d5c0081e698e620b6a35e52781317bebd1a6281735a4b7cf8c9ec","tree":"0ed5695c68d3f8d1eb50e58e675d498df4aa0804e2c2cb38b131fb7c9e384194","paths":["/mnt/sys_drive/felhom-data/backups/primary/opengist"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","opengist"],"id":"9e38b84c2ed88523a66ebad1a0a4cd996d7fba25da14b1ce3616bc3a3dd31751","short_id":"9e38b84c"},{"time":"2026-08-09T08:30:53.465490024Z","parent":"c365f01a8460a0211b33224682d7b2c24bcf481cfec40d1bad0307610438b3e6","tree":"8bb670471f388d5ad6464c2de2d1986d5c92d4bb77f8b822ac8042805d4f7a0e","paths":["/mnt/sys_drive/felhom-data/backups/primary/privatebin"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","privatebin"],"id":"78b93f0486555ea7d6e7526293501889cbabadee68ad946d6b2209c9e5da0f36","short_id":"78b93f04"},{"time":"2026-08-23T02:16:25.235054714Z","parent":"245932e40463e5a1d42b193759d6cf20abe00a7739efa8d687c83ae60f8f0dae","tree":"8ca10d1f7843222004e8c3a13b0f2fd19e5a26e46b2e0231703cb09b5bc59fe6","paths":["/mnt/sys_drive/felhom-data/backups/primary/bookstack"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","bookstack"],"id":"f06ad130a272a6963ea2ce22c76333c896a65460ec098ea83096b20aa89d7599","short_id":"f06ad130"},{"time":"2026-08-23T02:16:32.300678847Z","parent":"8d85d3dd62565e1f9c1b241d72f79fc0a4a3733284a955de08197ac2a64f7a4a","tree":"e2c1ba9ddad6c3a7df23764f76ec6c85dc5b8a26db74921f6d72ab0a3db2c950","paths":["/mnt/felhom-drives/hdd_1/backups/primary/calibre-web","/mnt/felhom-drives/hdd_1/userdata/media/books"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","calibre-web"],"id":"e68f6444d6241e496aef0b606fc736178133f90d198973471dc01c1acdc9b282","short_id":"e68f6444"},{"time":"2026-08-23T02:16:36.102571228Z","parent":"2630a25eb56271d5c32c12781e84ee2f3ad30a73d81a83ee5e1e49986c8894ac","tree":"0985393ce93e24fe42bcf883a1dabbd562e71129d9c25036cfc8e4da6f75ad88","paths":["/mnt/sys_drive/felhom-data/backups/primary/privatebin"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","privatebin"],"id":"d1a614b532b7eb7bfdccc749a33261a257ed30c11a936c7fb537a1cc8dfb4b94","short_id":"d1a614b5"},{"time":"2026-08-23T02:16:39.222268239Z","parent":"750b7b4d7a8d7923e7e9515a5c8f98570e7499500b4a2f5259fc8c05e7459da2","tree":"456cf7a9dd60961293b3cfd0df814c54c846a66124f737373f860a2c0964c056","paths":["/mnt/sys_drive/felhom-data/backups/primary/docmost"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","docmost"],"id":"e23e34a5871b95a2590a8bcb6d9037affbe494b72477d4424d822a0d9c26d2a1","short_id":"e23e34a5"},{"time":"2026-08-23T02:16:45.641580054Z","parent":"12b443ddd8ee5327f947f0baca3ff82f4746cce43deef4d641a84d80ca48ede0","tree":"a8beab17d7a19f5f5be306f573d355f0e70a89a9fad6977d20618f6bcd7284ae","paths":["/mnt/sys_drive/felhom-data/backups/primary/kimai"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","kimai"],"id":"5c3492ad763fb53510a8587945d2532063bef0fb2aa2e521293434469bed996d","short_id":"5c3492ad"},{"time":"2026-08-23T02:16:55.433584543Z","parent":"1be5b6402bbb16a769850276132c6dc88d2221e4c0f990b0c961f18b9060ba25","tree":"a975dce83f23053c189fde8c80241fd0ab67589dae19044186731260368e2c1b","paths":["/mnt/sys_drive/felhom-data/backups/primary/opengist"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","opengist"],"id":"82ef86926a61e8c86789073d839f7d08adc9b35753adf074cc535a63c3863ee2","short_id":"82ef8692"},{"time":"2026-08-23T02:16:58.57383511Z","parent":"1a230d746bc1349c917178e55ba12c531c23bc25921c0cc59705b3e56546641c","tree":"c073d2cd0692f3234612ff04a18835dbf95fff5b7360d9a824494b8ffeb1f021","paths":["/mnt/felhom-drives/hdd_1/appdata/paperless/media","/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","paperless-ngx"],"id":"778e8feb5fb6927ac7c10ba77fb0d357e339c47a96279b81fbecb831519d86ec","short_id":"778e8feb"},{"time":"2026-08-23T02:17:04.82144334Z","parent":"efcbc48301a727beeea495c42406f5a1ad59e97ba3f8f355e736aa48d7e68ebe","tree":"cecced990ed1a38569ca7c37c90fd16790e5c7fbf994f8f518f0f2a9d635fa8b","paths":["/mnt/felhom-drives/hdd_1/backups/primary/romm"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","romm"],"id":"8cc0e4b958b322bd5e6f17ad6a00cf4f070bafec96ad67a76adbfb85a47d6313","short_id":"8cc0e4b9"},{"time":"2026-08-25T02:16:29.549013249Z","parent":"75301c50a599eca4193ce09f366f85b3b6254e081a9b44e683cd297187888238","tree":"23c71aa9f827c56c6606334fa25bdb0f02186ab4641f7455247ee6a9dbea33b7","paths":["/mnt/sys_drive/felhom-data/backups/primary/privatebin"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","privatebin"],"id":"324e2dc838da5bdd3d96e78c3ba2a847d395fa68752ad2413c60f9bb6f32644d","short_id":"324e2dc8"},{"time":"2026-08-25T02:16:32.843227614Z","parent":"0f6889acbdb2131b618f2a7a24d8727adba46e49e1abe72db16a60fd7f8617c0","tree":"ae68cbcbb10f9d85ca4ab708f3b00c61e4dc6ba5075fd6d1e3019977187a3441","paths":["/mnt/sys_drive/felhom-data/backups/primary/bookstack"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","bookstack"],"id":"50b2b6b79dabb87717a96a0d367418145d6ab09f635349fd1315acbcc209c4f9","short_id":"50b2b6b7"},{"time":"2026-08-25T02:16:36.976442915Z","parent":"d7d808bedcdf667b9245fd63e629258caaae4b6ddb6c57d27410a20f39cad64c","tree":"f48628e8120c727c617b2a0fba9229067ff79dea4baceac4c2c8e0423451f90f","paths":["/mnt/felhom-drives/hdd_1/backups/primary/calibre-web","/mnt/felhom-drives/hdd_1/userdata/media/books"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","calibre-web"],"id":"f48b55010b037d3d22f975258d74440917ab4e7827c7b7a33bf7c4406f0e5003","short_id":"f48b5501"},{"time":"2026-08-25T02:16:40.772849688Z","parent":"eadbdbc9e1afb2bcb2ff089f8e121eea1f41a050b43ec77e3b911d58d9bb13e2","tree":"ef082fedfa2dd0c5aaf060e1884bfa7790bc7daba78669f7284fe2eb6b257705","paths":["/mnt/sys_drive/felhom-data/backups/primary/docmost"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","docmost"],"id":"d457b2c852cbac13417006df2825d8b2c27fc137f51df8684428374ff97124ac","short_id":"d457b2c8"},{"time":"2026-08-25T02:16:45.519179095Z","parent":"473f3912230b627725cd6651d728122b9ebaa199f83b9c2672f249c181da97bc","tree":"fcbad3fdab7dd5e357bab6fef3b02245ae95872444eafef55734343002de0885","paths":["/mnt/sys_drive/felhom-data/backups/primary/kimai"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","kimai"],"id":"2ebc28a45af12517f6cfcdf820d126a34ae0fbcaff024ca834f228fe056e08d1","short_id":"2ebc28a4"},{"time":"2026-08-25T02:16:55.42365237Z","parent":"6002f97943d0557c3883eb1ca0b3196b642c3b34f3a0f646e864ea418c9d49ca","tree":"a4e62a6df5ca8acf580258ca5ab7bb0b6fe064752c611ef0444dc0227bddac76","paths":["/mnt/sys_drive/felhom-data/backups/primary/opengist"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","opengist"],"id":"24cc2845d81a0954ba099a8d320d7c544fa5f82f795a63396998d059b2713b62","short_id":"24cc2845"},{"time":"2026-08-25T02:16:58.68540818Z","parent":"25fd9adacfeca821b3ec09b583f4f9fe39724fb7e1f1f8aba11c14ebca6a8f6f","tree":"cda8733befa1ba107c4d5d514e04726d7d095661bc899d8e642db6f713efdb93","paths":["/mnt/felhom-drives/hdd_1/backups/primary/romm"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","romm"],"id":"001a85575e0a7e3cb886d352062e77559dcc691a8ad1530450c28a7b0519234d","short_id":"001a8557"},{"time":"2026-08-25T02:17:04.930967505Z","parent":"b18a2476d2e8f538a5ef08a6879ee03ab23270bb8747931c579ae6905f8e5dfe","tree":"a0a3f9a3734665fbff13fe454e9953416c5d98f82958acf709e7305cdaca5f2c","paths":["/mnt/felhom-drives/hdd_1/appdata/paperless/media","/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","paperless-ngx"],"id":"ab0bb7f6c833564d6a0b86a88bfa8591950b3d3dca7a0e13047a723af210e46a","short_id":"ab0bb7f6"},{"time":"2026-08-26T02:16:29.71933113Z","parent":"2ebc28a45af12517f6cfcdf820d126a34ae0fbcaff024ca834f228fe056e08d1","tree":"aaf0584a302de2cb2a9d000e8d21b53d57d162ba9e556bbc3b9377a37470774a","paths":["/mnt/sys_drive/felhom-data/backups/primary/kimai"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","kimai"],"id":"936a30c3e89999c8842e0239d6b851d018c0e49dfeeaed46cf5cb75e32a2fe5e","short_id":"936a30c3"},{"time":"2026-08-26T02:16:39.913504559Z","parent":"24cc2845d81a0954ba099a8d320d7c544fa5f82f795a63396998d059b2713b62","tree":"c9dbed4e140d6cfde6fa060642a03cb85a90ad72761afb7440e2bec0d5da1398","paths":["/mnt/sys_drive/felhom-data/backups/primary/opengist"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","opengist"],"id":"13c3cf9da0f10efb838e7891bc2dd05cf0a46de8bdb23d490c01fede3bb27fc2","short_id":"13c3cf9d"},{"time":"2026-08-26T02:16:43.227326488Z","parent":"001a85575e0a7e3cb886d352062e77559dcc691a8ad1530450c28a7b0519234d","tree":"e95420b3adb3ee90bfb13a0054689d9689d7f0c2a2d565d97d437f2c18546e86","paths":["/mnt/felhom-drives/hdd_1/backups/primary/romm"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","romm"],"id":"9e3474ec426cffd42495edf41c27187ee50443c1d004791f9f5becd20637b9d4","short_id":"9e3474ec"},{"time":"2026-08-26T02:16:49.375441041Z","parent":"ab0bb7f6c833564d6a0b86a88bfa8591950b3d3dca7a0e13047a723af210e46a","tree":"6394fcdbbade7ad8fee605b03b8a743912c3efbf002c25c0ea965f8a533188dc","paths":["/mnt/felhom-drives/hdd_1/appdata/paperless/media","/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","paperless-ngx"],"id":"a2aec191d34a56d053112ebcf044a825d3d533ac5435c8225d20bb05ccfed91d","short_id":"a2aec191"},{"time":"2026-08-26T02:16:55.945101164Z","parent":"324e2dc838da5bdd3d96e78c3ba2a847d395fa68752ad2413c60f9bb6f32644d","tree":"40b42006ba9322cc6f0619e5b35c95d17a30f9cec8f03015d8c33061090025fd","paths":["/mnt/sys_drive/felhom-data/backups/primary/privatebin"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","privatebin"],"id":"389a3a6dba9f0eebb79cba847a42fc7a6daf6a0a22f27f23803acc8cfa3c6d1f","short_id":"389a3a6d"},{"time":"2026-08-26T02:16:59.254988612Z","parent":"50b2b6b79dabb87717a96a0d367418145d6ab09f635349fd1315acbcc209c4f9","tree":"9e3809fabdf6fc748ac4bcc5fa10f5cac2efa8da44b5660fd3811f71b1e44b57","paths":["/mnt/sys_drive/felhom-data/backups/primary/bookstack"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","bookstack"],"id":"c7408e7db0fcecddc45d778e530be6847cca5c9f76ca7b904f388142d88933e1","short_id":"c7408e7d"},{"time":"2026-08-26T02:17:03.463474567Z","parent":"f48b55010b037d3d22f975258d74440917ab4e7827c7b7a33bf7c4406f0e5003","tree":"1372ca8f79e13117f3ba0ae9bad484966e004853663196564fe6989f66b760d7","paths":["/mnt/felhom-drives/hdd_1/backups/primary/calibre-web","/mnt/felhom-drives/hdd_1/userdata/media/books"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","calibre-web"],"id":"f40c8fd9b3dfc486b5aa6adbe9b88efa3f847cefac1133d08e1e028f30b54941","short_id":"f40c8fd9"},{"time":"2026-08-26T02:17:07.343795007Z","parent":"d457b2c852cbac13417006df2825d8b2c27fc137f51df8684428374ff97124ac","tree":"b3e1cab1f40bcc7f642735acea8561b9aae6ad7fafd14861ed77215e177e3c8d","paths":["/mnt/sys_drive/felhom-data/backups/primary/docmost"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","docmost"],"id":"ea5e9a258b036a1759297c77f3bc0d1eaef8d880054348428331109e977d6d1b","short_id":"ea5e9a25"},{"time":"2026-08-27T02:16:26.968788853Z","parent":"a2aec191d34a56d053112ebcf044a825d3d533ac5435c8225d20bb05ccfed91d","tree":"07d0c4ac846f243f7ffb2d28864684222f54511333d546f50ec3032d3c14da8f","paths":["/mnt/felhom-drives/hdd_1/appdata/paperless/media","/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","paperless-ngx"],"id":"e6ee92aceaa6beb849a8e970d55a208b6158684dd888960e4f5e534c38993fa5","short_id":"e6ee92ac"},{"time":"2026-08-27T02:16:34.027545958Z","parent":"389a3a6dba9f0eebb79cba847a42fc7a6daf6a0a22f27f23803acc8cfa3c6d1f","tree":"a0ab150c58b68efe86dc8a20da433ec335481e1264f0f422675d6db89a15ddac","paths":["/mnt/sys_drive/felhom-data/backups/primary/privatebin"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","privatebin"],"id":"7326694728a760867d367165c20ec313b0ebf3dc39d08c1144b3bb968fe07e1e","short_id":"73266947"},{"time":"2026-08-27T02:16:37.200277709Z","parent":"c7408e7db0fcecddc45d778e530be6847cca5c9f76ca7b904f388142d88933e1","tree":"df9df92fe3a313422c607d69d5f728f9824f9756069b97984bf6ca4c4ad50393","paths":["/mnt/sys_drive/felhom-data/backups/primary/bookstack"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","bookstack"],"id":"794d8219271346ad5d6daa920d940de99c1a7f6c3d7920a21c318d157d6f771f","short_id":"794d8219"},{"time":"2026-08-27T02:16:41.364487503Z","parent":"f40c8fd9b3dfc486b5aa6adbe9b88efa3f847cefac1133d08e1e028f30b54941","tree":"02dd1fe5072a7a7ef003bfe50da52f4903523225f7bd3ce72e0f3a58d7d11952","paths":["/mnt/felhom-drives/hdd_1/backups/primary/calibre-web","/mnt/felhom-drives/hdd_1/userdata/media/books"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","calibre-web"],"id":"b1e4d6a5b2ca6ab59326fd9d7dc404f77a83c359e7825809244f1ed2bbaf85ce","short_id":"b1e4d6a5"},{"time":"2026-08-27T02:16:45.272914231Z","parent":"ea5e9a258b036a1759297c77f3bc0d1eaef8d880054348428331109e977d6d1b","tree":"9e7a1b85527458e64c220f707244c78f131f7967856067844cbfdd4b06e01bfb","paths":["/mnt/sys_drive/felhom-data/backups/primary/docmost"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","docmost"],"id":"7eea66d2076544c32dfd38cace4900b780115f4e83797b432d2a3adb2e669ea0","short_id":"7eea66d2"},{"time":"2026-08-27T02:16:50.321522487Z","parent":"936a30c3e89999c8842e0239d6b851d018c0e49dfeeaed46cf5cb75e32a2fe5e","tree":"221091a30e9da02c76d4c793e3acdd817d9b054ce998c5c5c61789bda925781f","paths":["/mnt/sys_drive/felhom-data/backups/primary/kimai"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","kimai"],"id":"b9db5519167c8b31300a7e96a29eb59d583b31b5e7312ba5e8367a20d44a8ecd","short_id":"b9db5519"},{"time":"2026-08-27T02:17:00.293326136Z","parent":"13c3cf9da0f10efb838e7891bc2dd05cf0a46de8bdb23d490c01fede3bb27fc2","tree":"cd0715aac889147b6efb71fa047567051526ff163d0a67e46a13821c17cbf363","paths":["/mnt/sys_drive/felhom-data/backups/primary/opengist"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","opengist"],"id":"0751dad2f53fad7ee63caae93b047f3f126712e373890911b8ae45dae219a143","short_id":"0751dad2"},{"time":"2026-08-27T02:17:03.542311119Z","parent":"9e3474ec426cffd42495edf41c27187ee50443c1d004791f9f5becd20637b9d4","tree":"51c764d2a042a7137b2ad4f558af29e402b0722543f611fdc1df4812226d2069","paths":["/mnt/felhom-drives/hdd_1/backups/primary/romm"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","romm"],"id":"3714731cc908c7257202c27e34788718c377b7b6bd045417a39cc88dbedc8767","short_id":"3714731c"},{"time":"2026-08-28T02:16:30.038082307Z","parent":"794d8219271346ad5d6daa920d940de99c1a7f6c3d7920a21c318d157d6f771f","tree":"42b50ec36f91c7a99092df10b9f84753eb03cc72692aa20d7725e230aacabcab","paths":["/mnt/sys_drive/felhom-data/backups/primary/bookstack"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","bookstack"],"id":"8dcbd53ab23524b30b20f8ff2e6ebb5cd79d0985d918ab2c795d70cc51f269f0","short_id":"8dcbd53a"},{"time":"2026-08-28T02:16:34.255924303Z","parent":"b1e4d6a5b2ca6ab59326fd9d7dc404f77a83c359e7825809244f1ed2bbaf85ce","tree":"d9ea6bae930fc09a57fa13aa4c2355463b67ddafbc2a4007538112be21cce504","paths":["/mnt/felhom-drives/hdd_1/backups/primary/calibre-web","/mnt/felhom-drives/hdd_1/userdata/media/books"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","calibre-web"],"id":"5a635785b0e5fb49ce895f22709022da3984db6bc20c2b579e74caf4fee08484","short_id":"5a635785"},{"time":"2026-08-28T02:16:38.042186988Z","parent":"7eea66d2076544c32dfd38cace4900b780115f4e83797b432d2a3adb2e669ea0","tree":"0bab87395807761d8a03e2549c44d1a41c975e6cf908b0b3d6d582227a912552","paths":["/mnt/sys_drive/felhom-data/backups/primary/docmost"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","docmost"],"id":"035e923ff2c37e666ee10cdb5740df1991c69e123dd0d796dfce77ed08e3875a","short_id":"035e923f"},{"time":"2026-08-28T02:16:43.110076287Z","parent":"b9db5519167c8b31300a7e96a29eb59d583b31b5e7312ba5e8367a20d44a8ecd","tree":"dadf187a28c163a72d0588a73331407dc435e1381d9e095ecdc532430b004d72","paths":["/mnt/sys_drive/felhom-data/backups/primary/kimai"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","kimai"],"id":"6f613bbd0cde4abb726180946b352bb3a1dfb5e688eafc01bb6add0162e4c728","short_id":"6f613bbd"},{"time":"2026-08-28T02:16:53.04098882Z","parent":"0751dad2f53fad7ee63caae93b047f3f126712e373890911b8ae45dae219a143","tree":"fb24d05f3379682743fa8723b202f898cd7bb4422267466d8fa7e93afdd3c444","paths":["/mnt/sys_drive/felhom-data/backups/primary/opengist"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","opengist"],"id":"f0c6ce706347b8fc942d3f34fff27af54fff990e1b6376e247669e72f4845f38","short_id":"f0c6ce70"},{"time":"2026-08-28T02:16:56.203850626Z","parent":"3714731cc908c7257202c27e34788718c377b7b6bd045417a39cc88dbedc8767","tree":"d0a4c16bb331108973f6b709ef260cbc8cae2b1ebdc99baa81ba1eb4c0fbeab7","paths":["/mnt/felhom-drives/hdd_1/backups/primary/romm"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","romm"],"id":"02f1de9d0fbe3b0d02aad96104297bc0676b7def88caae80af971dbe3e5af713","short_id":"02f1de9d"},{"time":"2026-08-28T02:17:02.21772672Z","parent":"e6ee92aceaa6beb849a8e970d55a208b6158684dd888960e4f5e534c38993fa5","tree":"9e0c707cf905ee799c79f3eeb1b96a11d85ff52a620954cfa63f728ad6da3a94","paths":["/mnt/felhom-drives/hdd_1/appdata/paperless/media","/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","paperless-ngx"],"id":"726c1ce833c6eaa88df88e37c3ca771d9053b151f28738220a1eca762af020ff","short_id":"726c1ce8"},{"time":"2026-08-28T02:17:08.921774992Z","parent":"7326694728a760867d367165c20ec313b0ebf3dc39d08c1144b3bb968fe07e1e","tree":"0bea5be1b9c110d017ac2167f7a9a3a31c53c25094408a4bef3f0a96b31b3239","paths":["/mnt/sys_drive/felhom-data/backups/primary/privatebin"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","privatebin"],"id":"c0e9dc25548efffe6c0f562d80b81e43711b48b3458c34ff30fc47217e902e2d","short_id":"c0e9dc25"},{"time":"2026-08-29T02:16:26.341833273Z","parent":"8dcbd53ab23524b30b20f8ff2e6ebb5cd79d0985d918ab2c795d70cc51f269f0","tree":"cf47d94cb016509e99da1c243365e30ec0b18a3b2d4f22a74fbd9fc9708931f1","paths":["/mnt/sys_drive/felhom-data/backups/primary/bookstack"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","bookstack"],"id":"0b90da52e042de34dbb8bc95937a78941b3951da8d109822b51ab3dbdf6268c0","short_id":"0b90da52"},{"time":"2026-08-29T02:16:30.960058165Z","parent":"5a635785b0e5fb49ce895f22709022da3984db6bc20c2b579e74caf4fee08484","tree":"2dd71e771f77c02b1687ea4d40767040bff967fe1a4b0d04d727579a81de79ba","paths":["/mnt/felhom-drives/hdd_1/backups/primary/calibre-web","/mnt/felhom-drives/hdd_1/userdata/media/books"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","calibre-web"],"id":"6aee181813a75591808b0939e9f071ca1815d9a813d23781190164f6dbf4d131","short_id":"6aee1818"},{"time":"2026-08-29T02:16:34.712960315Z","parent":"035e923ff2c37e666ee10cdb5740df1991c69e123dd0d796dfce77ed08e3875a","tree":"d6e90514d2b00e9840ba8782c11863a96027a053a6db955e7396272d39d7a694","paths":["/mnt/sys_drive/felhom-data/backups/primary/docmost"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","docmost"],"id":"7aea3e6dc6be80da4e54aee2bfda77c39f4a9254309ccb5e93b3c4427e64c44b","short_id":"7aea3e6d"},{"time":"2026-08-29T02:16:39.666366906Z","parent":"6f613bbd0cde4abb726180946b352bb3a1dfb5e688eafc01bb6add0162e4c728","tree":"cf33a871bda612a13f3a82c096c938349fa9f3509022ceac0de544d2b72f128f","paths":["/mnt/sys_drive/felhom-data/backups/primary/kimai"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","kimai"],"id":"2f043b35bb1232c4c012c5fb6470d6dd3c78f86dbb9c2e2a476cfd336eb9097f","short_id":"2f043b35"},{"time":"2026-08-29T02:16:49.647953065Z","parent":"f0c6ce706347b8fc942d3f34fff27af54fff990e1b6376e247669e72f4845f38","tree":"a28b37f7101c77008cc24b1af7a8f1cf72f5bbdd601f123c74303bc4ce4b7b6f","paths":["/mnt/sys_drive/felhom-data/backups/primary/opengist"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","opengist"],"id":"fa862b02f344aedad9d8493601e2f5bb32789156ca22439725c7eff5643e9ac7","short_id":"fa862b02"},{"time":"2026-08-29T02:16:52.865504533Z","parent":"02f1de9d0fbe3b0d02aad96104297bc0676b7def88caae80af971dbe3e5af713","tree":"2846855549f0994ab8f8339dfac64280ab251db47175369baab94bf008d4767b","paths":["/mnt/felhom-drives/hdd_1/backups/primary/romm"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","romm"],"id":"3f93ec623efbcfe28720779f27d746e496d82e955941f6ac299f08e9247cd7a0","short_id":"3f93ec62"},{"time":"2026-08-29T02:16:58.930628566Z","parent":"726c1ce833c6eaa88df88e37c3ca771d9053b151f28738220a1eca762af020ff","tree":"bf012cf5b32ae105af10a40b0e73b01d837255786628cdc3cad630db937e0825","paths":["/mnt/felhom-drives/hdd_1/appdata/paperless/media","/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","paperless-ngx"],"id":"ebade956b3a9100616f7e9eaff8da4f6b2b85350452db7763a6f2ad03487d4dc","short_id":"ebade956"},{"time":"2026-08-29T02:17:06.268014413Z","parent":"c0e9dc25548efffe6c0f562d80b81e43711b48b3458c34ff30fc47217e902e2d","tree":"c58f71ef0e45c1b71e0240134abc63966d38d53a39ef30cfca8038d0cf7925fa","paths":["/mnt/sys_drive/felhom-data/backups/primary/privatebin"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","privatebin"],"id":"983a03657196b4a7fb02b40d0a79ae78dee8322b7424554a97d7f1e3723a2da2","short_id":"983a0365"},{"time":"2026-08-30T02:16:29.510729824Z","parent":"3f93ec623efbcfe28720779f27d746e496d82e955941f6ac299f08e9247cd7a0","tree":"de00a19fed246208443885abaa150c31572f57ab3279f49cfa43238a1494d821","paths":["/mnt/felhom-drives/hdd_1/backups/primary/romm"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","romm"],"id":"9ae0aa88de83bc49362837c1af11b1261372fb2f8c3ac30484090f2d88cc1246","short_id":"9ae0aa88"},{"time":"2026-08-30T02:16:35.481119219Z","parent":"ebade956b3a9100616f7e9eaff8da4f6b2b85350452db7763a6f2ad03487d4dc","tree":"601ccf42806ceb45e0319f860ef99b18699e986409229c34e02bd2008240dd90","paths":["/mnt/felhom-drives/hdd_1/appdata/paperless/media","/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","paperless-ngx"],"id":"2404511dafdc7a65036b0f2e631ae1d791f6fb9d85f061f2f59bc8d1d99ca870","short_id":"2404511d"},{"time":"2026-08-30T02:16:42.235422711Z","parent":"983a03657196b4a7fb02b40d0a79ae78dee8322b7424554a97d7f1e3723a2da2","tree":"ee23f497f6f1d38a997e61c324e85378dfaf2758e318d690a348b25b8b0089f0","paths":["/mnt/sys_drive/felhom-data/backups/primary/privatebin"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","privatebin"],"id":"4a5db84d0efa2d5508c5d4ebeaa57fce218d1819f44803d2e7f4bd1211d893a6","short_id":"4a5db84d"},{"time":"2026-08-30T02:16:45.483014529Z","parent":"0b90da52e042de34dbb8bc95937a78941b3951da8d109822b51ab3dbdf6268c0","tree":"792e35bc9481a7731c95e91471810b3da4001a4417ad0bd80f3640d14473516d","paths":["/mnt/sys_drive/felhom-data/backups/primary/bookstack"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","bookstack"],"id":"ea40c93af59d7aa23a0534c26ef60931b7f73079068aab35496c38f7cbb98877","short_id":"ea40c93a"},{"time":"2026-08-30T02:16:49.631431901Z","parent":"6aee181813a75591808b0939e9f071ca1815d9a813d23781190164f6dbf4d131","tree":"10dd4e46b43e75cb46f64bdfd83615c095e78cce0dcc87148fab59a8eebf3a81","paths":["/mnt/felhom-drives/hdd_1/backups/primary/calibre-web","/mnt/felhom-drives/hdd_1/userdata/media/books"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","calibre-web"],"id":"37b7b51a43e6e755ca56bc37a969e53e59ca84287b9a2dae26302a513249b2f4","short_id":"37b7b51a"},{"time":"2026-08-30T02:16:53.456588837Z","parent":"7aea3e6dc6be80da4e54aee2bfda77c39f4a9254309ccb5e93b3c4427e64c44b","tree":"b533b29b187c58eee1cbb2eac5d9ee9f8845804c5a2e27c4a8e525aec921fbca","paths":["/mnt/sys_drive/felhom-data/backups/primary/docmost"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","docmost"],"id":"ee67d85d949ae401447ed6bd12df4f79f363b4a616d21f60ea0e6ebbdbc89d19","short_id":"ee67d85d"},{"time":"2026-08-30T02:16:58.549995966Z","parent":"2f043b35bb1232c4c012c5fb6470d6dd3c78f86dbb9c2e2a476cfd336eb9097f","tree":"f6c25da1b9c9df0f0c53af35358ca18f8e682bf303b4dc3871ea136bf43650d0","paths":["/mnt/sys_drive/felhom-data/backups/primary/kimai"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","kimai"],"id":"84542ec89b96fd05d504ca33e5f391a959c2e42e94353ca53e2e46929791d576","short_id":"84542ec8"},{"time":"2026-08-30T02:17:08.365249494Z","parent":"fa862b02f344aedad9d8493601e2f5bb32789156ca22439725c7eff5643e9ac7","tree":"129240b222e96d013d533d4bb021e1771ef165d1f3d7f8c44432cd4958ff67fe","paths":["/mnt/sys_drive/felhom-data/backups/primary/opengist"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","opengist"],"id":"a5c5c973f57df3b2a6dfee798f09a5b8a37f70415af0a871620def4589eb2d40","short_id":"a5c5c973"},{"time":"2026-08-31T02:16:30.041113131Z","parent":"84542ec89b96fd05d504ca33e5f391a959c2e42e94353ca53e2e46929791d576","tree":"00bbec71aced8301be6100c4ac5de6142defbbf0d24b44898a5dfff059e9969f","paths":["/mnt/sys_drive/felhom-data/backups/primary/kimai"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","kimai"],"id":"e7dcf8fde91a54f655175b60da2e45cfa7b907be78bc9ac5ead5601f3eaf077d","short_id":"e7dcf8fd"},{"time":"2026-08-31T02:16:42.562119597Z","parent":"4a5db84d0efa2d5508c5d4ebeaa57fce218d1819f44803d2e7f4bd1211d893a6","tree":"0e599f5c06ab7ba1e93b82ddc48463219fdd68d5a27a75d19e63c919e2fda875","paths":["/mnt/sys_drive/felhom-data/backups/primary/privatebin"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","privatebin"],"id":"bb81b9da0e87f5699d6e2caad4b100a29109cf55e456487a845a263fac4c734f","short_id":"bb81b9da"},{"time":"2026-08-31T02:16:45.969574107Z","parent":"a5c5c973f57df3b2a6dfee798f09a5b8a37f70415af0a871620def4589eb2d40","tree":"ff6d457dafc14e4a197fb7edfb7abb58b3d0c308dd9f4d66ccd956b12ade2099","paths":["/mnt/sys_drive/felhom-data/backups/primary/opengist"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","opengist"],"id":"b5aa8f9b77ead28047b4404684ef75abcd9b1b75db6f791ff5f0af9411543b65","short_id":"b5aa8f9b"},{"time":"2026-08-31T02:16:49.343798426Z","parent":"2404511dafdc7a65036b0f2e631ae1d791f6fb9d85f061f2f59bc8d1d99ca870","tree":"1c250ab771e426599b55fa327022160c5c22ee5e727f61de1f26ede5d9e46847","paths":["/mnt/felhom-drives/hdd_1/appdata/paperless/media","/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","paperless-ngx"],"id":"c3f1b55cb22c57ba63dd4b5ff8d1b166396875349b18ec93fb516707698c11e9","short_id":"c3f1b55c"},{"time":"2026-08-31T02:16:56.748327217Z","parent":"9ae0aa88de83bc49362837c1af11b1261372fb2f8c3ac30484090f2d88cc1246","tree":"3ff96abdf54d93034bf36396849f58d3abab5d75d1e776c93ce34e031d830a97","paths":["/mnt/felhom-drives/hdd_1/backups/primary/romm"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","romm"],"id":"df7ad253fb2f80cb9f5adc0f1237ad2b575fb41f673fcc706b2bfcbe1b6e9235","short_id":"df7ad253"},{"time":"2026-08-31T02:17:02.838774129Z","parent":"ea40c93af59d7aa23a0534c26ef60931b7f73079068aab35496c38f7cbb98877","tree":"e3c295207c002192a62f906aba2d211eafc4bdcd85e6984bf68633ff294d3c2a","paths":["/mnt/sys_drive/felhom-data/backups/primary/bookstack"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","bookstack"],"id":"91154be7d390b8dc842363409ad11ef1d781322e5c2d14a3666d6a6abecd48dc","short_id":"91154be7"},{"time":"2026-08-31T02:17:07.036789777Z","parent":"37b7b51a43e6e755ca56bc37a969e53e59ca84287b9a2dae26302a513249b2f4","tree":"3ed260200bcc23f89f6317d832223743b7e5252843a5dabd3ee80e93b8224468","paths":["/mnt/felhom-drives/hdd_1/backups/primary/calibre-web","/mnt/felhom-drives/hdd_1/userdata/media/books"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","calibre-web"],"id":"ad2ae49c90a5a1a12986403be6d7941c36735484b1540117e5bc421333ebc80f","short_id":"ad2ae49c"},{"time":"2026-08-31T02:17:11.003679016Z","parent":"ee67d85d949ae401447ed6bd12df4f79f363b4a616d21f60ea0e6ebbdbc89d19","tree":"c4cb6a57b8d4772ce5d3cb2b564a339d29f571224bd14199d0925348c260326b","paths":["/mnt/sys_drive/felhom-data/backups/primary/docmost"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","docmost"],"id":"a07c36a1217ef769a66946ce85cbaa1786b8aea569d8605c4cf0551e51f4b3d8","short_id":"a07c36a1"}] diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/15-q5-integrity-run.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/15-q5-integrity-run.txt new file mode 100644 index 00000000..57da66f4 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/15-q5-integrity-run.txt @@ -0,0 +1,5 @@ +t0=13:41:28 +{"data":{"depth":"100%","duration_ms":40257,"ok":true,"read_data_subset":"100%","skip_reason":"","skipped":false,"unreachable":false},"message":"Az ellenőrzés rendben lezajlott","ok":true} + +http=200 time_total=42.701280 +t1=13:42:11 diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/16-q6-locks-during-check.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/16-q6-locks-during-check.txt new file mode 100644 index 00000000..242b63f4 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/16-q6-locks-during-check.txt @@ -0,0 +1 @@ +sh: 1: Syntax error: Unterminated quoted string diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/16-q6-locks-full.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/16-q6-locks-full.txt new file mode 100644 index 00000000..55c2d418 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/16-q6-locks-full.txt @@ -0,0 +1,170 @@ +13:39:55 rc=0 locks=0 ids= +13:39:59 rc=0 locks=0 ids= +13:40:03 rc=0 locks=0 ids= +13:40:07 rc=0 locks=0 ids= +13:40:11 rc=0 locks=0 ids= +13:40:15 rc=0 locks=0 ids= +13:40:19 rc=0 locks=0 ids= +13:40:23 rc=0 locks=0 ids= +13:40:27 rc=0 locks=0 ids= +13:40:31 rc=0 locks=0 ids= +13:40:35 rc=0 locks=0 ids= +13:40:39 rc=0 locks=0 ids= +13:40:43 rc=0 locks=0 ids= +13:40:47 rc=0 locks=0 ids= +13:40:50 rc=0 locks=0 ids= +13:40:54 rc=0 locks=0 ids= +13:40:58 rc=0 locks=0 ids= +13:41:02 rc=0 locks=0 ids= +13:41:06 rc=0 locks=0 ids= +13:41:10 rc=0 locks=0 ids= +13:41:14 rc=0 locks=0 ids= +13:41:18 rc=0 locks=0 ids= +13:41:22 rc=0 locks=0 ids= +13:41:26 rc=0 locks=0 ids= +13:41:30 rc=0 locks=0 ids= +13:41:34 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf +13:41:38 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf +13:41:42 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf +13:41:46 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf +13:41:50 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf +13:41:54 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf +13:41:58 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf +13:42:02 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf +13:42:06 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf +13:42:10 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf +13:42:14 rc=0 locks=0 ids= +13:42:18 rc=0 locks=0 ids= +13:42:22 rc=0 locks=0 ids= +13:42:26 rc=0 locks=0 ids= +13:42:30 rc=0 locks=0 ids= +13:42:34 rc=0 locks=0 ids= +13:42:38 rc=0 locks=0 ids= +13:42:42 rc=0 locks=0 ids= +13:42:46 rc=0 locks=0 ids= +13:42:49 rc=0 locks=0 ids= +13:42:53 rc=0 locks=0 ids= +13:42:57 rc=0 locks=0 ids= +13:43:01 rc=0 locks=0 ids= +13:43:05 rc=0 locks=0 ids= +13:43:09 rc=0 locks=0 ids= +13:43:13 rc=0 locks=0 ids= +13:43:17 rc=0 locks=0 ids= +13:43:21 rc=0 locks=0 ids= +13:43:25 rc=0 locks=0 ids= +13:43:29 rc=0 locks=0 ids= +13:43:33 rc=0 locks=0 ids= +13:43:37 rc=0 locks=0 ids= +13:43:41 rc=0 locks=0 ids= +13:43:45 rc=0 locks=0 ids= +13:43:49 rc=0 locks=0 ids= +13:43:53 rc=0 locks=0 ids= +13:43:57 rc=0 locks=0 ids= +13:44:01 rc=0 locks=0 ids= +13:44:05 rc=0 locks=1 ids=9222fe46cd531ebe2c729af8ef3912bd77bea8cbe72b0527ce7702d5db678c4f +13:44:08 rc=0 locks=0 ids= +13:44:12 rc=0 locks=0 ids= +13:44:16 rc=0 locks=0 ids= +13:44:20 rc=0 locks=0 ids= +13:44:24 rc=0 locks=0 ids= +13:44:28 rc=0 locks=0 ids= +13:44:32 rc=0 locks=0 ids= +13:44:36 rc=0 locks=0 ids= +13:44:40 rc=0 locks=0 ids= +13:44:44 rc=0 locks=0 ids= +13:44:48 rc=0 locks=1 ids=7a045a2731f7cdbb2aebf000018f040ed4f81d3ede59ed55921975cfbd8d4a61 +13:44:52 rc=0 locks=0 ids= +13:44:56 rc=0 locks=0 ids= +13:45:00 rc=0 locks=0 ids= +13:45:04 rc=0 locks=0 ids= +13:45:08 rc=0 locks=0 ids= +13:45:12 rc=0 locks=0 ids= +13:45:16 rc=0 locks=0 ids= +13:45:20 rc=0 locks=0 ids= +13:45:24 rc=0 locks=0 ids= +13:45:28 rc=0 locks=0 ids= +13:45:31 rc=0 locks=0 ids= +13:45:35 rc=0 locks=0 ids= +13:45:39 rc=0 locks=0 ids= +13:45:43 rc=0 locks=0 ids= +13:45:47 rc=0 locks=0 ids= +13:45:51 rc=0 locks=0 ids= +13:45:55 rc=0 locks=0 ids= +13:45:59 rc=0 locks=0 ids= +13:46:03 rc=0 locks=0 ids= +13:46:07 rc=0 locks=0 ids= +13:46:11 rc=0 locks=0 ids= +13:46:15 rc=0 locks=0 ids= +13:46:19 rc=0 locks=0 ids= +13:46:23 rc=0 locks=0 ids= +13:46:27 rc=0 locks=0 ids= +13:46:31 rc=0 locks=0 ids= +13:46:35 rc=0 locks=0 ids= +13:46:39 rc=0 locks=0 ids= +13:46:43 rc=0 locks=0 ids= +13:46:47 rc=0 locks=0 ids= +13:46:51 rc=0 locks=0 ids= +13:46:54 rc=0 locks=0 ids= +13:46:58 rc=0 locks=0 ids= +13:47:02 rc=0 locks=0 ids= +13:47:06 rc=0 locks=0 ids= +13:47:10 rc=0 locks=0 ids= +13:47:14 rc=0 locks=0 ids= +13:47:18 rc=0 locks=0 ids= +13:47:22 rc=0 locks=0 ids= +13:47:26 rc=0 locks=0 ids= +13:47:30 rc=0 locks=0 ids= +13:47:34 rc=0 locks=0 ids= +13:47:38 rc=0 locks=0 ids= +13:47:42 rc=0 locks=0 ids= +13:47:46 rc=0 locks=0 ids= +13:47:50 rc=0 locks=0 ids= +13:47:54 rc=0 locks=0 ids= +13:47:58 rc=0 locks=0 ids= +13:48:01 rc=0 locks=0 ids= +13:48:05 rc=0 locks=0 ids= +13:48:09 rc=0 locks=0 ids= +13:48:13 rc=0 locks=0 ids= +13:48:17 rc=0 locks=0 ids= +13:48:21 rc=0 locks=0 ids= +13:48:25 rc=0 locks=0 ids= +13:48:29 rc=0 locks=0 ids= +13:48:33 rc=0 locks=0 ids= +13:48:37 rc=0 locks=0 ids= +13:48:41 rc=0 locks=0 ids= +13:48:45 rc=0 locks=0 ids= +13:48:49 rc=0 locks=0 ids= +13:48:53 rc=0 locks=0 ids= +13:48:57 rc=0 locks=0 ids= +13:49:01 rc=0 locks=0 ids= +13:49:05 rc=0 locks=0 ids= +13:49:09 rc=0 locks=0 ids= +13:49:13 rc=0 locks=0 ids= +13:49:17 rc=0 locks=0 ids= +13:49:21 rc=0 locks=0 ids= +13:49:25 rc=0 locks=0 ids= +13:49:29 rc=0 locks=0 ids= +13:49:33 rc=0 locks=0 ids= +13:49:36 rc=0 locks=0 ids= +13:49:40 rc=0 locks=0 ids= +13:49:44 rc=0 locks=0 ids= +13:49:48 rc=0 locks=0 ids= +13:49:52 rc=0 locks=0 ids= +13:49:56 rc=0 locks=0 ids= +13:50:00 rc=0 locks=0 ids= +13:50:04 rc=0 locks=0 ids= +13:50:08 rc=0 locks=0 ids= +13:50:12 rc=0 locks=0 ids= +13:50:16 rc=0 locks=0 ids= +13:50:20 rc=0 locks=0 ids= +13:50:24 rc=0 locks=0 ids= +13:50:28 rc=0 locks=0 ids= +13:50:32 rc=0 locks=0 ids= +13:50:36 rc=0 locks=0 ids= +13:50:40 rc=0 locks=0 ids= +13:50:43 rc=0 locks=0 ids= +13:50:47 rc=0 locks=0 ids= +13:50:51 rc=0 locks=0 ids= +13:50:55 rc=0 locks=0 ids= +13:50:59 rc=0 locks=0 ids= +13:51:03 rc=0 locks=0 ids= diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/17-q4-candidate-sizes.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/17-q4-candidate-sizes.txt new file mode 100644 index 00000000..f1ef8491 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/17-q4-candidate-sizes.txt @@ -0,0 +1,6 @@ +c3f1b55c {"total_size":83473654,"total_file_count":29,"snapshots_count":1} + +ad2ae49c {"total_size":5808703,"total_file_count":38,"snapshots_count":1} + +e7dcf8fd {"total_size":213231242,"total_file_count":16,"snapshots_count":1} + diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/18-q4-kimai-full-post.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/18-q4-kimai-full-post.txt new file mode 100644 index 00000000..658132e5 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/18-q4-kimai-full-post.txt @@ -0,0 +1,10 @@ +13:42:54 +t0=2026-08-31T13:42:54.542010807Z +post_http=302 post_time=0.009733 +2026/08/31 13:42:11 offbox_integrity.go:306: [INFO] [offbox] integrity: check PASSED in 40s (structure, index, and 100% of the pack data re-read) +2026/08/31 13:42:54 auth.go:134: [DEBUG] [web] auth: valid session for POST /backup/offbox/restore +2026/08/31 13:42:54 server.go:393: [DEBUG] [web] ServeHTTP: POST /backup/offbox/restore from 172.18.0.4:34642 +2026/08/31 13:43:05 offbox_restore.go:298: [INFO] [offbox] restored kimai (e7dcf8fd, full=true) → /mnt/felhom-drives/hdd_1/backups/offsite-restore/kimai +2026/08/31 13:43:05 offbox_handlers.go:372: [INFO] [web] off-box restore kimai completed (full=true, async) +--- +213231328 /mnt/felhom-drives/hdd_1/backups/offsite-restore/kimai diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/19-q6-argv-full.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/19-q6-argv-full.txt new file mode 100644 index 00000000..85dc262c --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/19-q6-argv-full.txt @@ -0,0 +1,58 @@ +13:44:39 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:40 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:40 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:40 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:41 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:42 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:42 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:42 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:42 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:43 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:43 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:43 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:43 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:43 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:43 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:44 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:44 restic -r sftp: -o sftp.command= unlock +13:44:44 restic -r sftp: -o sftp.command= unlock +13:44:45 restic -r sftp: -o sftp.command= unlock +13:44:45 restic -r sftp: -o sftp.command= unlock +13:44:46 restic -r sftp: -o sftp.command= unlock +13:44:46 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:46 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:46 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:47 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:47 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:47 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:47 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:47 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:48 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:48 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:48 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:48 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:48 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:49 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:51 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:51 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:51 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:52 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:52 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:55 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:55 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:55 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:56 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:56 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:58 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:58 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:59 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:59 restic -r sftp: -o sftp.command= list locks --no-lock +13:45:00 restic -r sftp: -o sftp.command= list locks --no-lock +13:45:00 restic -r sftp: -o sftp.command= list locks --no-lock +13:45:02 restic -r sftp: -o sftp.command= list locks --no-lock +13:45:02 restic -r sftp: -o sftp.command= list locks --no-lock +13:45:03 restic -r sftp: -o sftp.command= list locks --no-lock +13:45:03 restic -r sftp: -o sftp.command= list locks --no-lock +13:45:04 restic -r sftp: -o sftp.command= list locks --no-lock +13:45:04 restic -r sftp: -o sftp.command= list locks --no-lock +13:45:06 restic -r sftp: -o sftp.command= list locks --no-lock diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/19-q6-argv.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/19-q6-argv.txt new file mode 100644 index 00000000..20061b1a --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/19-q6-argv.txt @@ -0,0 +1,48 @@ +t0=2026-08-31T13:44:41.569463100Z +post_http=302 post_time=0.010061 +=== distinct restic argv observed (timestamps stripped) === +restic -r sftp: -o sftp.command= list locks --no-lock +restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +restic -r sftp: -o sftp.command= unlock +=== first+last timestamp per verb === +13:44:39 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:40 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:40 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:40 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:41 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:42 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:42 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:42 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:42 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:43 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:43 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:43 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:43 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:43 restic -r sftp: -o sftp.command= snapshots latest --tag opengist --json +13:44:43 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:44 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:44 restic -r sftp: -o sftp.command= unlock +13:44:44 restic -r sftp: -o sftp.command= unlock +13:44:45 restic -r sftp: -o sftp.command= unlock +13:44:45 restic -r sftp: -o sftp.command= unlock +13:44:46 restic -r sftp: -o sftp.command= unlock +13:44:46 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:46 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:46 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:47 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:47 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:47 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:47 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:47 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:48 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:48 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:48 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:48 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:48 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:49 restic -r sftp: -o sftp.command= restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist +13:44:51 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:51 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:51 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:52 restic -r sftp: -o sftp.command= list locks --no-lock +13:44:52 restic -r sftp: -o sftp.command= list locks --no-lock diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/20-q2-verify-cost.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/20-q2-verify-cost.txt new file mode 100644 index 00000000..ebecc070 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/20-q2-verify-cost.txt @@ -0,0 +1,11 @@ +=== A: restore WITHOUT --verify (kimai e7dcf8fd, 213 MB) === +sh: 7: bc: not found +rc=0 wall=s bytes=213272202 +restoring to /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/a + +=== B: restore WITH --verify, same snapshot === +sh: 12: bc: not found +rc=0 wall=s bytes=213272202 +restoring to /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b +verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b +finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b (took 132ms) diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/21-q2-verify-redproof.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/21-q2-verify-redproof.txt new file mode 100644 index 00000000..7ba7e38d --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/21-q2-verify-redproof.txt @@ -0,0 +1,18 @@ +A no-verify rc=0 wall_ms=2990 bytes=213272202 +B with-verify rc=0 wall_ms=4756 bytes=213272202 +verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b +finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b (took 126ms) + +=== RED-PROOF: corrupt ONE restored byte, keep size AND mtime, re-run restore --verify === +victim=kimai_kimai_db_data.tar size=160331776 +after edit: size=160331776 (must be unchanged) sha=dd962a1097077e39 orig-sha=81ce628f45418d93 +re-run rc=0 +restoring to /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b +verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b +finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b (took 125ms) + +=== NEGATIVE CONTROL: same command on the UNTOUCHED copy /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/a === +rc=0 +restoring to /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/a +verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/a +finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/a (took 132ms) diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/22-q2-verify-semantics.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/22-q2-verify-semantics.txt new file mode 100644 index 00000000..90b3c940 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/22-q2-verify-semantics.txt @@ -0,0 +1,19 @@ +=== CASE 1: SIZE changed (truncate 1 byte) === +size=160331775 +rc=0 +verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b +finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b (took 133ms) +size after=160331776 sha=81ce628f45418d93 + +=== CASE 2: content corrupt + mtime BUMPED === +sha before rerun=51ae3c7762768fe4 +rc=0 +verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b +finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b (took 128ms) +sha after rerun =81ce628f45418d93 orig=81ce628f45418d93 + +=== CASE 3: content corrupt, size+mtime PRESERVED, restore into a VIRGIN target === +(this is the shape an unattended test would meet: a fresh restore, then a check) +virgin restore+verify rc=0 wall_ms=3438 +verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/g +finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/g (took 131ms) diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/23-q3-reference-candidates.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/23-q3-reference-candidates.txt new file mode 100644 index 00000000..5008088d --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/23-q3-reference-candidates.txt @@ -0,0 +1,46 @@ +=== does the recovery unit manifest carry hashes? === +{ + "schema_version": 2, + "app_name": "kimai", + "display_name": "Kimai", + "controller_version": "0.227.1", + "created_at": "2026-08-31T02:16:24Z", + "drive": "/mnt/sys_drive", + "namespace_root": "/mnt/sys_drive/felhom-data", + "image_pins": [ + "kimai/kimai2:apache-2.57.0", + "mariadb:11.6" + ], + "secret_env_vars": [ + "DB_PASSWORD", + "ADMIN_PASSWORD" + ], + "data_key_env_vars": null, + "secret_source": "portable secrets (data keys, DB passwords, internal signing secrets) are IN this unit's compose/app.yaml (0600); internet-reachable admin logins are NOT, and come from the guest's app.yaml or are regenerated on restore", + "config_files": [ + "docker-compose.yml", + ".felhom.yml", + "app.yaml" + ], + "db_dumps": [ + "kimai-mariadb.sql" + ], + "volume_dumps": [ + "kimai_kimai_db_data.tar", + "kimai_kimai_var.tar" + ], + "checksums": { + ".felhom.yml": "95beb458a42391af4bf78429decfbc60537d8d53d61bea68c38687bcc7fa0d23", + "app.yaml": "9192dec10e5aae76c2787d0e82ad39115d43f7011519f854bc06534f930c3e9c", + "docker-compose.yml": "c25029f8bd16e6252dceda0b4e87917a8b9ce0b9ad6bdbfc82fa8d4f630e6310" + }, + "portable_secret_env_vars": [ + "DB_PASSWORD" + ], + "o + +=== does restic ls --json expose per-file content hashes? === +{"time":"2026-08-31T02:16:30.041113131Z","parent":"84542ec89b96fd05d504ca33e5f391a959c2e42e94353ca53e2e46929791d576","tree":"00bbec71aced8301be6100c4ac5de6142defbbf0d24b44898a5dfff059e9969f","paths":["/mnt/sys_drive/felhom-data/backups/primary/kimai"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","kimai"],"id":"e7dcf8fde91a54f655175b60da2e45cfa7b907be78bc9ac5ead5601f3eaf077d","sho +{"name":"mnt","type":"dir","path":"/mnt","uid":0,"gid":0,"mode":2147484141,"permissions":"drwxr-xr-x","mtime":"2026-08-21T16:00:39.422914481Z","atime":"2026-08-21T16:00:39.422914481Z","ctime":"2026-08-21T16:00:39.422914481Z","struct_type":"node"} +{"name":"sys_drive","type":"dir","path":"/mnt/sys_drive","uid":0,"gid":0,"mode":2147484141,"permissions":"drwxr-xr-x","mtime":"2026-08-21T16:01:03.794262171Z","atime":"2026-08-21T16:01:03.794262171Z","ctime":"2026-08-21T16:01:03.794262171Z","struct_type":"node"} +{"name":"felhom-data","type":"dir","path":"/mnt/sys_drive/felhom-data","uid":0,"gid":0,"mode":2147484141,"permissions":"drwxr-xr-x","mtime":"2026-08-21T16:31:03.891800035Z","atime":"2026-08-21T16:31:03.891800035Z","ctime":"2026-08-21T16:31:03.891800035Z","struct_type":"node"} diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/24-q3-hash-coverage.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/24-q3-hash-coverage.txt new file mode 100644 index 00000000..3e343105 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/24-q3-hash-coverage.txt @@ -0,0 +1,18 @@ +=== a FILE node from restic ls --json === +{"name":".felhom.yml","type":"file","path":"/mnt/sys_drive/felhom-data/backups/primary/kimai/compose/.felhom.yml","uid":0,"gid":0,"size":2235,"mode":420,"permissions":"-rw-r--r--","mtime":"2026-08-31T02:16:24.917098021Z","atime":"2026-08-31T02:16:24.917098021Z","ctime":"2026-08-31T02:16:24.918098034Z","struct_type":"node"} +{"name":"app.yaml","type":"file","path":"/mnt/sys_drive/felhom-data/backups/primary/kimai/compose/app.yaml","uid":0,"gid":0,"size":488,"mode":384,"permissions":"-rw-------","mtime":"2026-08-31T02:16:24.918098034Z","atime":"2026-08-31T02:16:24.918098034Z","ctime":"2026-08-31T02:16:24.918098034Z","struct_type":"node"} + +=== the full checksums block of the kimai manifest === + "checksums": { + ".felhom.yml": "95beb458a42391af4bf78429decfbc60537d8d53d61bea68c38687bcc7fa0d23", + "app.yaml": "9192dec10e5aae76c2787d0e82ad39115d43f7011519f854bc06534f930c3e9c", + "docker-compose.yml": "c25029f8bd16e6252dceda0b4e87917a8b9ce0b9ad6bdbfc82fa8d4f630e6310" + }, +=== files actually in the unit === + 48217 /mnt/sys_drive/felhom-data/backups/primary/kimai/db-dumps/kimai-mariadb.sql + 2235 /mnt/sys_drive/felhom-data/backups/primary/kimai/compose/.felhom.yml + 2195 /mnt/sys_drive/felhom-data/backups/primary/kimai/compose/docker-compose.yml + 488 /mnt/sys_drive/felhom-data/backups/primary/kimai/compose/app.yaml + 160331776 /mnt/sys_drive/felhom-data/backups/primary/kimai/volume-dumps/kimai_kimai_db_data.tar + 52845056 /mnt/sys_drive/felhom-data/backups/primary/kimai/volume-dumps/kimai_kimai_var.tar + 1275 /mnt/sys_drive/felhom-data/backups/primary/kimai/manifest.json diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/25-q4-cache-effect.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/25-q4-cache-effect.txt new file mode 100644 index 00000000..fa541cb7 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/25-q4-cache-effect.txt @@ -0,0 +1,10 @@ +=== restic local cache present? === +1.1M /root/.cache/restic + +=== kimai 213 MB: cached vs --no-cache (the WAN-truth number) === +cached rc=0 ms=3198 +--no-cache rc=0 ms=5423 +identical trees: YES + +=== peak scratch bytes for the biggest app === +213272202 /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/i diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/26-q4-all-apps-cost.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/26-q4-all-apps-cost.txt new file mode 100644 index 00000000..b1f505a5 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/26-q4-all-apps-cost.txt @@ -0,0 +1,9 @@ +privatebin snap=bb81b9da logical= 2123006 restored= 2159870 ms= 2628 rc=0 +opengist snap=b5aa8f9b logical= 185664 restored= 222528 ms= 2253 rc=0 +calibre-web snap=ad2ae49c logical= 5808703 restored= 5878335 ms= 3642 rc=0 +paperless-ngx snap=c3f1b55c logical= 83473654 restored= 83547382 ms= 3062 rc=0 +bookstack snap=91154be7 logical= 166641768 restored= 166682728 ms= 2745 rc=0 +docmost snap=a07c36a1 logical= 118207270 restored= 118248230 ms= 3676 rc=0 +romm snap=df7ad253 logical= 184706816 restored= 184747776 ms= 3978 rc=0 +kimai snap=e7dcf8fd logical= 213231242 restored= 213272202 ms= 3202 rc=0 +ALL EIGHT, back to back: 25s diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/27-q5-single-writer-flag.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/27-q5-single-writer-flag.txt new file mode 100644 index 00000000..a1da6d89 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/27-q5-single-writer-flag.txt @@ -0,0 +1,41 @@ +=== every acquireRunning caller (non-test) === +internal/backup/offbox_integrity.go:272: if err := m.acquireRunning(); err != nil { +internal/backup/offbox_restore.go:549: if err := m.acquireRunning(); err != nil { +internal/backup/offbox_reconstitute.go:491: if err := m.acquireRunning(); err != nil { +internal/backup/backup.go:473: if err := m.acquireRunning(); err != nil { +internal/backup/backup.go:935:func (m *Manager) AcquireRunningForTest() error { return m.acquireRunning() } +internal/backup/backup.go:939:func (m *Manager) acquireRunning() error { +internal/backup/shares_restore.go:203: if err := m.acquireRunning(); err != nil { +internal/backup/offbox.go:864: if err := m.acquireRunning(); err != nil { +internal/backup/tier2_restore.go:183:// do NOT add an acquireRunning() here. A second acquire would refuse the restore it is guarding. +internal/backup/tier2_restore.go:347: if err := m.acquireRunning(); err != nil { + +=== does RestoreOffboxScratch acquire it? === +NO acquireRunning in RestoreOffboxScratch + +=== restore_wizard.go:170-190 === +// restoreOpInFlight reports whether a restore op is in flight, FOR DISPLAY. +// +// **Use this, not `Manager.IsRunning()`.** The Manager carries two different booleans and they are +// not interchangeable: +// +// - `m.running` (read by `IsRunning`) is the CONCURRENCY single-flight. It is acquired *inside* +// the restore function, on the background goroutine — and `RestoreOffboxScratch` never acquires +// it at all. So for the verification restore and the full-restore preparation — the wizard's two +// most-used actions, and the long ones, since they stream from restic — `IsRunning()` is false +// for the entire operation. +// - `m.opRunning` (read by `RestoreStatus`) is the DISPLAY flag, set synchronously by +// `BeginRestoreOp` in the handler *before* the goroutine launches and cleared by `EndRestoreOp`. +// It covers all four offsite actions with no start-up window. +// +// v0.154.0 shipped with `IsRunning()` here, which made the execution step unreachable for +// `RestoreOffboxScratch`: the page offered all three intents, with live buttons, while a restore was +// downloading — and the progress banner (which polls the op status) contradicted it on the same +// screen. Caught by the operator on the first live click-through. +func restoreOpInFlight(st backup.RestoreOpStatus) bool { + return st.Running +} + +// restoreOpBlocked reports whether a NEW restore must be refused right now, and returns the +// Hungarian refusal to show. It reads BOTH flags, deliberately: +// diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/28-q5-schedule.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/28-q5-schedule.txt new file mode 100644 index 00000000..a1458df0 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/28-q5-schedule.txt @@ -0,0 +1,111 @@ +=== scheduled off-site jobs (main.go) === +663: sched.Every("status-refresh", 10*time.Second, func(ctx context.Context) error { +666: sched.Every("stack-scan", 2*time.Minute, func(ctx context.Context) error { +669: sched.Every("health-probes", 10*time.Second, func(ctx context.Context) error { +679: sched.Every("system-health", healthInterval, func(ctx context.Context) error { +726: sched.Every("deadapp-check", 30*time.Second, func(ctx context.Context) error { +742: sched.Every("ring-spill", 30*time.Second, func(ctx context.Context) error { +876: sched.Daily("db-dump", dbLeg, func(ctx context.Context) error { +914: sched.Every("offsite-credential-retry", 5*time.Minute, func(ctx context.Context) error { +937: sched.Every("backup-cache", 5*time.Minute, func(ctx context.Context) error { +962: sched.Daily("tier2-backup", tier2Leg, func(ctx context.Context) error { +1095: sched.Daily("offbox-backup", offboxLeg, func(ctx context.Context) error { +1109: sched.Daily("offsite-abandon-sweep", "05:10", func(ctx context.Context) error { +1139: sched.Daily("offsite-integrity", "06:00", func(ctx context.Context) error { +1147: sched.Daily("metrics-prune", "04:00", func(ctx context.Context) error { +1198: sched.Daily("fill-watch", "03:30", func(ctx context.Context) error { return fillWatcher.Check() }) +1228: sched.Every("hub-report", pushInterval, func(ctx context.Context) error { +1256: sched.Every("selfupdate-check", checkInterval, func(ctx context.Context) error { +1266: sched.Daily("selfupdate-auto", cfg.SelfUpdate.AutoUpdateTime, func(ctx context.Context) error { +1293: sched.Daily("asset-sync", cfg.Assets.SyncSchedule, func(ctx context.Context) error { +1464: sched.Every("geo-verify", 6*time.Hour, func(ctx context.Context) error { + sync_schedule: "05:00" +backup: + db_dump_schedule: "02:30" + enabled: true + prune_schedule: weekly + restic_password_file: /opt/docker/felhom-controller/data/restic-password + restic_schedule: "03:00" + retention: + keep_daily: 7 + keep_monthly: 6 + keep_weekly: 4 +customer: + domain: enkisfelhom.hu + email: doodoo21@freemail.hu + id: demo-hp + name: Demo HP + telegram_chat_id: "" +git: + branch: main +-- + health_check_schedule: "06:00" + healthchecks_base: https://status.felhom.eu + system_health_interval: 5m + thresholds: + backup_max_age_hours: 36 + cpu_warn_percent: 90 + disk_crit_percent: 90 + disk_warn_percent: 80 + memory_warn_percent: 85 + temperature_warn_celsius: 75 +internal/backupwindow/backupwindow.go:54:func LegTimes(start string) (db, tier2, offbox string) { +internal/backupwindow/backupwindow.go-55- m, err := ParseHHMM(start) +internal/backupwindow/backupwindow.go-56- if err != nil { +internal/backupwindow/backupwindow.go-57- return "", "", "" +internal/backupwindow/backupwindow.go-58- } +internal/backupwindow/backupwindow.go-59- return FmtHHMM(m), FmtHHMM(m + tier2OffsetMin), FmtHHMM(m + offboxOffsetMin) +internal/backupwindow/backupwindow.go-60-} +internal/backupwindow/backupwindow.go-61- +internal/backupwindow/backupwindow.go-62-// GateWindow returns the whole-guest backup gate bounds [W+2h, W+6h) as HH:MM strings (for the UI +internal/backupwindow/backupwindow.go-63-// "kb. – között" line and the gate-denial log). Empty strings on an invalid start. +internal/backupwindow/backupwindow.go-64-func GateWindow(start string) (from, to string) { +internal/backupwindow/backupwindow.go-65- m, err := ParseHHMM(start) +internal/backupwindow/backupwindow.go-66- if err != nil { +internal/backupwindow/backupwindow.go-67- return "", "" +internal/backupwindow/backupwindow.go-68- } +internal/backupwindow/backupwindow.go-69- return FmtHHMM(m + gateStartMin), FmtHHMM(m + gateEndMin) +internal/backupwindow/backupwindow.go-70-} +internal/backupwindow/backupwindow.go-71- +internal/backupwindow/backupwindow.go-72-// EffectiveWindow resolves the active window by precedence: a valid settings value wins over a valid +internal/backupwindow/backupwindow.go-73-// controller.yaml value, which wins over DefaultWindow. An empty or corrupted value simply falls +internal/backupwindow/backupwindow.go-74-// through — so a bad settings string degrades to the yaml default rather than breaking scheduling. +internal/backupwindow/backupwindow.go-75-func EffectiveWindow(settingsVal, yamlVal string) string { +internal/backupwindow/backupwindow.go-76- if Valid(settingsVal) == nil { +internal/backupwindow/backupwindow.go-77- return settingsVal +internal/backupwindow/backupwindow.go-78- } +internal/backupwindow/backupwindow.go-79- if Valid(yamlVal) == nil { +internal/backupwindow/backupwindow.go-80- return yamlVal +internal/backupwindow/backupwindow.go-81- } +internal/backupwindow/backupwindow.go-82- return DefaultWindow +internal/backupwindow/backupwindow.go-83-} +10:// DefaultWindow is the last-resort window when neither settings nor controller.yaml supplies one. +12:const DefaultWindow = "02:30" +17: tier2OffsetMin = 60 // tier-2 mirror at W+60m +18: offboxOffsetMin = 105 // off-box copy at W+105m +2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="status-refresh" interval=10s totalJobs=1 +2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="stack-scan" interval=2m0s totalJobs=2 +2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="health-probes" interval=10s totalJobs=3 +2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="system-health" interval=5m0s totalJobs=4 +2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="deadapp-check" interval=30s totalJobs=5 +2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="ring-spill" interval=30s totalJobs=6 +2026/08/31 12:33:29 scheduler.go:132: [INFO] [scheduler] Daily job db-dump scheduled for 2026-09-01 02:30 CEST +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="db-dump" schedule="02:30" nextRun=2026-09-01T02:30:00+02:00 totalJobs=7 +2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="offsite-credential-retry" interval=5m0s totalJobs=8 +2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="backup-cache" interval=5m0s totalJobs=9 +2026/08/31 12:33:29 scheduler.go:132: [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-09-01 03:30 CEST +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="tier2-backup" schedule="03:30" nextRun=2026-09-01T03:30:00+02:00 totalJobs=10 +2026/08/31 12:33:29 scheduler.go:132: [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-09-01 04:15 CEST +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offbox-backup" schedule="04:15" nextRun=2026-09-01T04:15:00+02:00 totalJobs=11 +2026/08/31 12:33:29 scheduler.go:132: [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-09-01 05:10 CEST +2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offsite-abandon-sweep" schedule="05:10" nextRun=2026-09-01T05:10:00+02:00 totalJobs=12 +2026/08/31 12:33:29 scheduler.go:132: [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-09-01 06:00 CEST +grep: write error: Broken pipe diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/29-teardown.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/29-teardown.txt new file mode 100644 index 00000000..e24fb843 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/29-teardown.txt @@ -0,0 +1,39 @@ +=== LAYER 3: container /tmp === +/tmp: 1: Syntax error: Unterminated quoted string +remaining spike files in container: [] +wc: invalid option -- '"' +Try 'wc --help' for more information. +restic processes still running: + +=== LAYER 3b: the scratch dirs my probes created === +161146787 /mnt/felhom-drives/hdd_1/backups/offsite-restore/bookstack +5808705 /mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web +118761552 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost +213231328 /mnt/felhom-drives/hdd_1/backups/offsite-restore/kimai +185751 /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist +72345368 /mnt/felhom-drives/hdd_1/backups/offsite-restore/paperless-ngx +2123093 /mnt/felhom-drives/hdd_1/backups/offsite-restore/privatebin +after: [bookstack +calibre-web +paperless-ngx] + +=== LAYER 2: guest /root === +remaining in guest: [] +=== LAYER 1: PVE host /root === +ssh: Could not resolve hostname hp: Name or service not known +=== LAYER 1: PVE host demo-hp /root === +host-leftovers-rc=1 (1 = none found = clean) + +=== LAYER 3 re-verified: container === +container /tmp listed above + +=== LAYER 2 re-verified: guest === +guest-leftovers-rc=1 + +=== the app still healthy after all this === +felhom-controller Up About an hour (healthy) +docmost Up About an hour (healthy) +docmost-redis Up About an hour (healthy) +docmost-postgres Up About an hour (healthy) +romm Up 4 hours (healthy) +romm-db Up 4 hours (healthy) diff --git a/documentation/audits/evidence-spike-restic-restore-2026-08-31/30-side-effect-integrity-verdict.txt b/documentation/audits/evidence-spike-restic-restore-2026-08-31/30-side-effect-integrity-verdict.txt new file mode 100644 index 00000000..d03ae0d2 --- /dev/null +++ b/documentation/audits/evidence-spike-restic-restore-2026-08-31/30-side-effect-integrity-verdict.txt @@ -0,0 +1,3 @@ +last_integrity_check": "2026-08-31T13:41:28Z" +last_integrity_ok": true +last_integrity_depth": "100%" diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 0dc8f651..da411a82 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -225,7 +225,7 @@ unconditional promise, which CAMPAIGN-11 §7 step 7 measured the customer-facing | **R-121** | **A BOX's installed agent can sit releases behind the vouched one and nothing notices — the R-120 gate does not cover it.** demo-hp ran agent **0.113.0** while the hub vouched **0.116.0**, through the whole R-116/R-117 arc, and no signal existed on any channel | **READY (S) — NEW 2026-07-30** | — | **Fourth instance of the drift family** (R-111 golden's agent 17 releases behind, R-115 built+deployed but never published, R-120 golden a controller behind — and now installed-vs-vouched on a live box). **Confirmed at source that R-120's gate cannot catch it:** `hub/internal/web/configs.go:1165-1169` compares `goldenVer` against `store.NewestReportedControllerVersion()` — it is a **golden-artifact vs fleet-CONTROLLER** check and says nothing about the agent installed on a box. **`MinAgent` does not cover it either:** it is used to HOLD the controller floor for a box whose agent is too old (`hub/internal/api/handler.go:530-538`, `store.go:1857`) — protective, not an alarm — and demo-hp's 0.113.0 **equalled** `min_agent` 0.113.0, so even a floor comparison was satisfied. **The cost, measured:** R-117's whole subject is the R-113 conjunction, which landed in **0.114.0** — so the designated drill host could not exercise the code under investigation at all, and the R-117 spike had to route every predicate result through an out-of-repo probe built from `main` instead of the installed agent (`audits/SPIKE-r117-bind-liveness-2026-07-30.md` §1, §2.3). Discovered because the R-117 task made bringing the box current an explicit prerequisite. **Fix shape (not implemented):** the hub already receives `AgentVersion` on every host report, and already has semver comparison in Go — the missing piece is a checker comparing reported agent vs the **vouched** agent and surfacing it, operator-tier. Note the honest tension: a box legitimately lags between publish and deploy, so this wants a staleness window rather than an instant alarm | CC | | **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC | | **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC | -| **R-87** | The restic tier is never restore-tested | **READY — RE-RANKED UP 2026-08-03 (R-86 closed)** | — | Design a controller-side test (no scratch-guest analogue transfers). **Most of what this row needed now exists.** R-86 built the piece that was missing: a tier is proved **per archive generation**, on its own rhythm, with the proof recorded as *which archive* — which is exactly the shape a weekly-ish restic tier needs, and the reason this row could not simply reuse the whole-guest scheduler before. What remains is genuinely restic-specific and is NOT a scheduling problem: there is no scratch-guest analogue, so the test has to be a controller-side restore of a bounded sample into a throwaway path, with its own definition of "proved". **Two things to carry over rather than re-derive:** the proof must record the SNAPSHOT it proved (not a timestamp), and the hub's staleness window must learn this tier's rhythm the way `restoreProvenWindow` now does — a restic tier on a weekly cadence lands on the same false-alarm line the flat 7 days did. **And R-95 still applies:** that credential can delete, so a restic restore-test must never be able to write to the repo | CC | +| **R-87** | The restic tier is never restore-tested | **READY — SPIKED 2026-08-31, RE-SCOPE PROPOSED (a DECISION for Viktor). Was: READY — RE-RANKED UP 2026-08-03 (R-86 closed)** | — | **SPIKE VERDICT — `audits/SPIKE-restic-restore-test-2026-08-31.md`. Do NOT build this row as written.** **Q1/Q2 measured:** restic is **0.14.0** (the four source comments asserting it are correct); `--verify` DOES exist and is **not a content check** — a one-byte corruption of a restored 160 MB tar with size and mtime preserved **passed clean**, and verify took 131 ms on a 213 MB tree, which cannot be hashing. **Q3:** no reference for "correct" exists — `restic ls --json` carries no content hash in 0.14.0, and the unit manifest hashes **4 918 B of 213 231 242 B** (R-409). **Q4 measured on demo-hp:** one app ≈ 2.3–4.0 s; **all 8 apps / 774 MB logical = 25 s**, against **40.3 s** for the weekly 100% check beside it — a restore-test is CHEAPER than the check. Peak scratch = the app's full logical size (213 MB largest). Cost is dominated by per-snapshot round-trip, not data: 185 KB takes 2.25 s and 213 MB takes 3.20 s. **Q5:** 25 s against a 2m52s nightly backup — skip-if-busy stays right; **but `RestoreOffboxScratch` takes NO `acquireRunning` (R-408)**. **Q6 observed with a positively-controlled lock sampler:** `restic restore` takes **no lock at all**; the product writes anyway because `unlockStale` runs `restic unlock` — a DELETE verb — before every restore (`offbox_restore.go:289`); and `restic check` DOES take a lock (R-407). **R-95's constraint IS satisfiable:** `--no-lock` + skipping `unlockStale` writes nothing, and both mechanisms exist in 0.14.0 and are unused. **Q7, the deciding one:** of R-353/R-354/R-356/R-358/R-403, an unattended scratch-restore would have caught **ONE (R-356)**. **PROPOSED RE-SCOPE:** from *restore-test the tier* to *prove the off-site snapshot still CONTAINS a recoverable unit* — one app a night, restored to scratch, checked against its own `manifest.json` through the existing `unitCarriesData` (`r403_hollow.go:40`), scratch deleted, the SNAPSHOT recorded as the proof. That catches the one thing the weekly check structurally cannot: **`check` proves the stored bytes are the stored bytes, never that we stored the RIGHT thing** — a hollow unit backs up, checks and restores cleanly and recovers nothing (R-403, measured in bytes 2026-08-31). **Viktor decides: build the narrow version, or close this row as answered by R-359.** | CC | | **R-191** | **Every weekly offsite backup UPLOADS successfully and then FAILS the job on a prune the box is deliberately not allowed to do — on both demo boxes.** Measured on demo-felhom 2026-08-04 06:49–06:53: the upload completed (223 s, 629 MiB of 1.874 GiB, **67.2 % reused incrementally**), then `ERROR: prune 'ct/9201': proxmox-backup-client failed: Error: permission check failed - missing Datastore.Modify\|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `ERROR: Backup of VM 9201 failed - error pruning backups` → `TASK ERROR: job errors`. The hub raised `whole_guest_backup_failed` | **CLOSED — SHIPPED 2026-08-04** (installer **1.25.0**; both live boxes corrected) | — | **This is R-89's rule not reaching the config.** R-89 moved PBS pruning SERVER-SIDE — *"boxes set `keep_last: 0`, ep0 runs prune jobs; box tokens stay write-only, never widen the grant"*. The token behaves exactly as designed: it refuses. But **both** demo boxes still arm the offsite tier with `keep_last=2 prune_pbs_allowed=true` (`backup_targets: [{target_id: felhom-pbs, cadence_seconds: 604800, keep_last: 2}]`), so every run asks for a prune that must fail. **The data is SAFE and that is why this is not a P1:** the snapshot lands before the prune is attempted; what is wrong is the job's VERDICT and the weekly operator e-mail it produces. **But it is corrosive in the specific way this project keeps finding:** a backup that reports FAILED while succeeding trains the operator to discount `whole_guest_backup_failed`, which is the same alert that would carry a real one — and it is exactly the failure the R-100 corollary warns about, an alarm whose text is true and whose trigger is not the thing you would act on. **Fix is one config line per box** (`keep_last: 0` on the PBS tier) plus whatever writes it on a fresh install; **deliberately NOT applied in this session** — the session was a runbook with an explicit "change nothing, and if a change appears necessary, stop and report" rule, and a retention field on a live backup tier is not a change to slip into an observation run. **Check before fixing:** whether ep0's prune jobs actually cover these two namespaces, or the snapshots simply accumulate once the box stops asking **THE GATE WAS RUN FIRST, AND IT MATTERED.** Before disabling anything, ep0 was read (read-only, Tier 2): prune jobs `prune-demo-felhom` and `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, `schedule 03:30`, `keep-last 2`, comment *"R-82 retention keep-last=2, server-side (box tokens are write-only)"* — and they have run **every day since 2026-07-27: 18 tasks, all `status=OK`**. The newest task log reads `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` / `keep ct/9201/2026-07-27…` / `keep ct/9201/2026-07-28…` / `TASK OK`. Retention happens, and it happens there. **A METHODOLOGICAL WARNING WORTH MORE THAN THE FIX.** Three separate queries said the OPPOSITE — *no prune jobs have ever run* — and **all three were broken instruments**: `worker-type` where the field is `worker_type`; the value `prune` where the worker type is `prunejob`; and `journalctl -u proxmox-backup` where the unit is `proxmox-backup-proxy`. A fourth reading (3 snapshots under keep-last 2) was mis-framed by CC and self-corrected — the third snapshot had landed AFTER that day's 03:30 window. Acting on any of them would have disabled the only pruning ATTEMPT while reporting that nothing prunes: a weekly false alarm traded for unbounded growth on the protected endpoint, invisible for months. **The gate is what caught it, and only because it demanded evidence rather than a verdict.** **Shipped:** installer **1.25.0** writes `keep_last: 0` on the offsite tier (the agent's existing guard `allowPBSPrune = !primary && keep_last > 0` already reads that as *never prune from the box* — no agent change), the justifying paragraph is rewritten to say where retention lives and cite R-89, and `hostinstall_gates.py` asserts it (red-proved: pinning `keep_last: 2` back fails the gate). **Both live boxes corrected in their own config** — `backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false` on demo-felhom and demo-hp, with the local tier untouched at `keep_last=3`. Served over HTTPS at `1.25.0` with `"keep_last":0` in the served bytes. **STILL TO OBSERVE:** the next weekly offsite run completing OK end-to-end. The change removes the failing step; the *schedule* proving it is next week's event, and this row should carry that line when it happens. | CC | | **R-194** | **PVE's permission cache delays every grant-state verdict by an unknown amount, so "the agent can read it" and "the ACL exists" are not the same measurement.** Observed twice while validating R-190's self-repair on demo-felhom 2026-08-04: both ACL rows for `/storage/felhom-backup` were deleted, and `GET /access/permissions` continued to report `Datastore.AllocateSpace` present — for **~40 s** in one run and **~16 minutes** in another. During that window the capability probe reads healthy and the self-repair does not fire | **OPEN** | — | **Why it matters beyond the delay:** it puts a floor under how fast a lost grant can be noticed, it makes any single permission read a lagging indicator, and — the interesting part — **it is a candidate contributor to R-190's own timeline**: a grant removed at an unknown moment could keep working until a cache expiry, which is exactly the shape of *worked at 04:44, refused at 09:24*. That does not explain what removed it, but it may explain when the refusal SURFACED, and the two have been treated as the same instant. **Not a defect in our code** — it is PVE behaviour, and the mitigation already tolerates it (the repair fires on the next probe after the cache clears). **What is worth deciding:** whether the store-grant probe should ALSO consult the storage content listing as a second signal, since that appeared to reflect the loss immediately (`{"data":[]}` while the permission read still said present) — two signals disagreeing is itself information, and today only one of them is read | CC | | **R-200** | **The DR password-injection seam has a handler, a route and tests — and no form.** `POST /backup/offbox/inject-password` is routed (`controller/internal/web/server.go:510`) to `offboxInjectPasswordHandler` (`offbox_handlers.go:174-196`) → `InjectOffboxPassword` (`backup/offbox.go:541`). **No template in the repository contains that path or any form posting to it** (grep over `internal/web/templates/`: one unrelated hit, an XSS comment) | **PLUMBING COMPLETE** (controller **v0.196.0**); **the FORM is not built — still open** | — | **The tenth instance of this project's built-but-never-wired class, and the exact shape `CLAUDE.md` and `felhom.eu/CLAUDE.md`'s seam-wiring rule were written for:** handler tests that POST directly (`offbox_escrow_test.go:167,180`) prove nothing about reachability. To use the only implemented recovery seam today, a person must hand-craft an authenticated POST with a session cookie and CSRF token. **Note the layering while fixing it:** this form takes a **64-hex repo password**, not a recovery code (`offboxRepoPwPattern`, `offbox.go:543`) — they are different secrets at different layers, and the operator's 2026-08-04 ruling asks for a form that takes **R**. Build the R form and treat this one as the operator/DR fallback it was written as, but **ship it with a render test per branch of whatever gate it sits behind**. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 9 **THE DIAGNOSTIC HALF IS DONE AND IT ANSWERED THE QUESTION.** `--recover-offsite-check` is a `docker exec` escape hatch in the shape of `--print-reset-code`: R on **STDIN** (never argv, never `ps`, never shell history, never a transcript), fetch+unseal via the agent, and a verdict of **two sha256 hashes**. **It compares and never installs** — the recovered password is not written to `offbox/repo_password`; a test asserts the data dir is byte-unchanged and its red-proof (adding the install call) fails it. Confirmed live: `repo_password` mtime still `2026-08-03 07:18:02` after the successful check at `2026-08-04 11:49`. **Exit codes are load-bearing** — `0` match, `2` a clean MISMATCH, `1` a step failed; "it failed" and "it worked and disagreed" must never share a status because only one is a finding about the system. A box with no local password reports distinctly (the rebuilt-box shape, where the next step is to INSTALL rather than compare). **WHAT IS NOT BUILT, deliberately:** no card, no form, no preview, no customer-facing text — building an interface on top of a chain nobody had walked is how the preceding three weeks went wrong. **What remains for this row:** link 9 (the recovered password placed so `WriteOffboxSecrets` keeps it) and the customer-facing shape the operator ruled on 2026-08-04 (yell → R form → preview → proceed), which is now priced against a chain that exists rather than one that is assumed **PART 0 SHIPPED 2026-08-04 (v0.196.0):** `--recover-offsite-install` is the sibling of the check — same fetch/unseal path, same STDIN discipline for R — and it **places** the recovered password via `InjectOffboxPassword`. The confirmation is a SECOND invocation (`--confirm-install`): without it, both hashes print and nothing is written, so the operator sees the comparison before a write is possible. Three outcomes named distinctly: **installed** (no local password — the rebuilt-box shape), **unchanged** (identical key, nothing written), **refused** (a DIFFERENT key present — installing would clobber the key the current repository is encrypted under; exit 2, no force offered). It re-reads the file after writing rather than trusting the call. **Red-proof observed:** removing the confirmation gate makes the dry run write the password. The R-persistence test carries a **positive control** (a planted copy found, then removed and not found). **NOT YET EXERCISED AGAINST A LIVE RECOVERY** — the R-201 drill halted before step 9, so this is unit-proven only. **What remains for this row:** the customer-facing shape the operator ruled on 2026-08-04 (yell → recovery-code form → preview → proceed) | CC | @@ -455,7 +455,14 @@ there is one ranking to maintain rather than two. a drift gate nobody runs and a test that compares a constant to itself. Cheap and worth doing; not high-consequence, and it blocks nothing. 3. ~~**R-86**~~ — **CLOSED 2026-08-03**, agent v0.121.0 + hub v0.91.0, proven live on demo-felhom. -4. **R-87** — **re-ranked UP**: R-86 built most of what it was waiting for (per-archive due-ness, a +4. **R-87** — **SPIKED 2026-08-31 and now a DECISION, not work.** It also spent 2026-08-22..31 in + `CLOSED-ITEMS.md` by mistake while this paragraph ranked it fourth and pointed at nothing + (R-405). The spike measured it rather than designing it: a scratch restore of all 8 apps costs + 25 s against the 40.3 s weekly check, but it would have caught ONE of the five drill-found + restore defects. **Recommendation: build the NARROW version — prove the snapshot still + CONTAINS a recoverable unit — or close the row.** Viktor's call; see + `audits/SPIKE-restic-restore-test-2026-08-31.md`. *The 2026-08-03 rationale, kept:* + R-86 built most of what it was waiting for (per-archive due-ness, a proof that names its archive, and a staleness window that learns a tier's rhythm). What is left is restic-specific — there is no scratch-guest analogue — so it still needs its own design, but it is no longer waiting on a scheduling model that did not exist. @@ -581,9 +588,12 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-104** | **An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach.** `resticStep` has `unlock --remove-all` (`internal/backup/offbox.go:634-648`) but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`:77-93`) has no lock case → `"other"` → fail-fast; `ClassifyOffsiteFailure` likewise, so the operator is told *„A távoli mentés ismeretlen okból nem sikerült"* for a precisely-known, self-healable condition **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-28, size S, roadmap state `READY — 2026-07-28`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. ****PARTLY STALE, checked against live source 2026-08-22 — migrated as written per the task rule, with the staleness named rather than edited away.** The self-heal this row calls unreachable was built: `resticStep` escalates to `unlock --remove-all` and retries once (`internal/backup/offbox.go:763-768`), and `unlockStale` runs before every off-site run and restore (`:1274`, `offbox_restore.go:261`). Its premise that the probe fails first is also doubtful: the probe is `restic cat config`, a read that takes no lock. **What REMAINS true:** `ClassifyOffsiteFailure` (`offbox.go:179-193`) still has no lock case, so if a lock ever did survive both layers the customer would still be told an unknown reason. Re-rank on that basis, not on the original text.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | **Was C9-F3.** Reachable by any interruption — container restart, OOM, network drop, host reboot mid-backup. The tier stays dead until a human runs `restic unlock --remove-all`. Flips: the offsite row in map §C; `07` §8 row 15 | CC | | **R-105** | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-28, size M, roadmap state `READY — 2026-07-28`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. ****PARTLY FIXED BY ITS OWN UPDATE.** The `drives` third was traced and populated on both demo boxes on 2026-07-28 (the enrolled drives were never PVE storages, so `isUserDataDrive` never saw them). The other two thirds — `hosts.dr_record_json` and `host_escrow.directive_json` — were NOT re-verified this session and are carried as written.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. **UPDATE 2026-07-28 (vzdump-target move): the `drives` third is TRACED and now POPULATED on both demo boxes.** Cause: the enrolled data drives were never PVE storages at all — only agent-generated systemd mounts — so they never entered `report.StorageTargets` and `isUserDataDrive` never saw them. Giving each drive a `dir` storage at its own mountpoint supplied all three required fields at once (type `local-dir`, fs-UUID durable id, mount path), and the recipe now emits `uuid:91d2dc2d-…`/`/mnt/nvme-1tb` on demo-hp and `uuid:47a3361a-…`/`/mnt/hdd_1` o | CC | | **R-340** | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** **REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc//fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC | -| **R-404** | **DECISION FOR VIKTOR — should a documentation-only push be subject to the golden-currency gate?** `git push --no-verify` has now been used **six times**, each with a recorded reason, because `repo_gates.py --fast` runs `golden_currency_gate.py` on every push to `felhom.eu` including pushes that touch only `documentation/`. **A guard that is correctly bypassed six times is training everyone to bypass it**, and the seventh bypass will be faster to reach for than the sixth | **OPEN — a DECISION, not work. Filed 2026-08-31, deliberately NOT acted on** | — | **FOR narrowing it:** a docs-only push cannot be the push that finishes a release, so scoping the gate to pushes that touch `controller/`, `agent/` or `hub/` code is arguably not a weakening at all — it would fire on exactly the pushes that can create the gap and on no others. It would also end the habit, which is the real cost being paid now. **AGAINST narrowing it:** the gate was earned by a real recurrence — v0.206.0 shipped while the vouched golden carried 0.205.0, and v0.204.0/v0.205.0 before it — and every narrowing of a guard risks the thing it was built for coming back. The gate is also deliberately `--fast` so that BOTH the pre-push hook and CI run it (R-29's census failure); a scoped version must stay in both or it runs in neither. **IF VIKTOR DOES NOTHING:** the bypass stays routine and the count keeps rising; nothing breaks, and the guard quietly stops being one. **Owner: Viktor decides, CC builds. This task did NOT change the gate.** | Viktor | +| **R-404** | **DECISION FOR VIKTOR — should a documentation-only push be subject to the golden-currency gate?** `git push --no-verify` has now been used **seven times** (the seventh 2026-08-31, the R-87 spike's records-only Part 1 commit `6e550ae`), each with a recorded reason, because `repo_gates.py --fast` runs `golden_currency_gate.py` on every push to `felhom.eu` including pushes that touch only `documentation/`. **A guard that is correctly bypassed six times is training everyone to bypass it**, and the seventh bypass will be faster to reach for than the sixth | **OPEN — a DECISION, not work. Filed 2026-08-31, deliberately NOT acted on** | — | **FOR narrowing it:** a docs-only push cannot be the push that finishes a release, so scoping the gate to pushes that touch `controller/`, `agent/` or `hub/` code is arguably not a weakening at all — it would fire on exactly the pushes that can create the gap and on no others. It would also end the habit, which is the real cost being paid now. **AGAINST narrowing it:** the gate was earned by a real recurrence — v0.206.0 shipped while the vouched golden carried 0.205.0, and v0.204.0/v0.205.0 before it — and every narrowing of a guard risks the thing it was built for coming back. The gate is also deliberately `--fast` so that BOTH the pre-push hook and CI run it (R-29's census failure); a scoped version must stay in both or it runs in neither. **IF VIKTOR DOES NOTHING:** the bypass stays routine and the count keeps rising; nothing breaks, and the guard quietly stops being one. **Owner: Viktor decides, CC builds. This task did NOT change the gate.** | Viktor | | **R-405** | **R-87 sat in `CLOSED-ITEMS.md` for nine days while it was still open, and the register's own ranking paragraph ranked it fourth pointing at nothing.** Established from history, not inferred: it was moved by the 2026-08-22 compression sweep, commit `ef6ac6f` (*One register, enforced by a gate; closed work compressed into siblings, R-376..R-378*) — the same commit and the same defect class R-378 records. **R-378 caught six — R-123, R-190, R-214, R-264, R-295, R-352 — and missed a seventh.** R-87 escaped because its state cell read `READY — RE-RANKED UP 2026-08-03 (R-86 closed)`: the leading verdict is `READY` and the word `closed` later in the same cell describes a **different** row. **Count reproduced independently 2026-08-31, and the predicate decides the answer:** matching an open word anywhere in the state cell convicts **three** of 151 rows (R-87, plus R-224 and R-260, both genuinely closed with the words "open"/"OPEN" inside long prose verdicts); matching the **leading verdict** convicts exactly **one**, R-87; matching the whole row convicts **144**. **Fixed this session:** the row is back in `OPEN-ITEMS.md` verbatim from `ef6ac6f^`, next to R-95 where it sat before, and `scripts/closed_register_gate.py` is the 12th gate. Red-proofed both rules and negative-controlled against the pushed pre-fix files, where it convicts R-87 by name. **`R-398` was ALSO in both registers** — a deliberate cross-reference stub — and is now prose beneath the table rather than a row, because a row in both files is what rule 2 convicts on. | **CLOSED 2026-08-31 — corrected + gated in the same session** | R-378 | Nothing further. The gate's four residual holes are named in its docstring; hole 4 is R-406. | CC | | **R-406** | **Two unrelated findings in `OPEN-ITEMS.md` share the identifier R-133.** `OPEN-ITEMS.md:267` is *the hub enforces uniqueness on `customer_id` only* (`domain` is `TEXT NOT NULL DEFAULT ''` with no UNIQUE/CHECK); `OPEN-ITEMS.md:273` is *the vaulted break-glass console credential is PLAINTEXT AT REST*. Different subjects, different owners, one number. Found 2026-08-31 while measuring duplicate ids for R-405's gate — **this is the only such collision in either register** (measured: no id appears twice in `CLOSED-ITEMS.md`, and R-88a/R-88b and R-209/R-209a are distinct suffixed ids, not duplicates). **Why the gate does NOT check for it:** a within-register duplicate rule would fail on this pre-existing row, and a registered-but-failing gate refuses every push. **Renumbering is not obviously safe** — `R-133` is cited elsewhere and a blind renumber breaks whichever citation meant the other one. | **OPEN — LOW** | R-405 | Establish which of the two `R-133` citations exist outside the register, then renumber the one with fewer (or none) and add the within-register duplicate rule to `closed_register_gate.py`. Do NOT renumber before grepping the citations. | CC | +| **R-407** | **`restic check` DOES write a lock file to the repository, and the comment above it says it never writes.** `offbox_integrity.go:255` reads *"CheckOffboxIntegrity runs one off-site integrity check. It NEVER writes to the repository: `check` is a read verb, and nothing here prunes, forgets, unlocks or backs up."* The three named verbs are correct; the headline is not. **OBSERVED 2026-08-31 on demo-hp**, with a lock sampler that was positively controlled before it was believed: across the product's own integrity run (13:41:28→13:42:11) the repository went `locks=0` → `locks=1 id=81fd4d4200d848466e18cb7a9d8e0d43…` for nine consecutive samples → `locks=0`. The same instrument saw **zero** locks across two restores, so it is not reporting a constant. **Why it matters and why it is LOW rather than ignorable:** R-95's whole constraint is phrased as "must never be able to write to the repo", and a comment stating a guarantee the code does not provide is this project's most-repeated failure — nine instances. The lock itself is correct behaviour and there is no defect in the check; **the defect is the sentence.** | **OPEN — LOW (a comment, not behaviour)** | R-87, R-359 | Correct the sentence in place — say the check takes a repository LOCK and writes nothing else — and pin it with a test, or pass `--no-lock` and make the sentence true. Do NOT delete the sentence: R-360's rule is that a doc comment claiming a guard is why nobody looks for the missing guard. Evidence: `audits/evidence-spike-restic-restore-2026-08-31/16-q6-locks-full.txt`. | CC | +| **R-408** | **`RestoreOffboxScratch` takes NO single-writer flag, and the file that depends on that invariant states it as universal.** `offbox_integrity.go:28` reads *"Every off-site operation takes `acquireRunning` for exactly that reason"* — the reason being that `resticStep` escalates to `unlock --remove-all` and is safe only because the in-process mutex proves no sibling operation is live. **Measured 2026-08-31:** `grep -rn 'acquireRunning()'` finds nine non-test callers and `RestoreOffboxScratch` (`offbox_restore.go:234`) is not among them. `restore_wizard.go:174` records the same fact independently — *"`RestoreOffboxScratch` never acquires it at all"* — and the UI works around it with a separate display flag (`opRunning`), so the gap is known at the web layer and unknown at the one that reasons about repository safety. **The exposure today is small and the exposure tomorrow is not:** the web handler's `restoreOpBlocked()` fences the only caller that exists, and restic's restore takes no lock (R-407's sampler), so nothing currently collides. **An unattended restore-test — R-87 — would be the first caller with no web handler in front of it.** | **OPEN — MEDIUM** | R-87, R-359 | Decide ONE way: either `RestoreOffboxScratch` takes `acquireRunning` (and every existing caller is re-checked for the double-acquire refusal `tier2_restore.go:183` warns about), or `offbox_integrity.go:28`'s sentence is corrected to name the exception. Whichever is chosen, **pin it with a test** — this is a comment asserting an invariant with nothing holding it. Do this BEFORE R-87 ships anything. | CC | +| **R-409** | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC |