R-87 SPIKE: measured, do not build it as written (R-407..R-409 filed)
gates / gates (push) Failing after 17s
gates / gates (push) Failing after 17s
Spike. NO production code. No version bump, no build, no deploy, no golden.
felhom-controller and felhom-agent were READ ONLY. The fleet stays on v0.230.0.
Q1 restic is 0.14.0 (go1.19.8, bookworm 12.15) - the four source comments asserting
it are CONFIRMED, not corrected.
Q2 --verify DOES exist and is NOT a content check. Red-proof: one byte changed in a
restored 160 MB tar with size and mtime preserved passed clean, rc=0. Verify took
131 ms on a 213 MB / 7-file tree, which cannot be hashing. A size or mtime mismatch
causes a silent re-download, not a failure. Controls: --target 1 hit, four post-0.14
flags and a nonsense string 0 hits each. Neither --verify nor --no-lock appears
anywhere in the controller source.
Q3 no reference for "correct" exists. restic ls --json carries no content hash in
0.14.0, and the unit manifest hashes 4918 B of a 213231242 B unit - 0.0023 percent,
the config files and not the dumps or the tars. R-409.
Q4 it is CHEAP. All 8 apps / 774378123 B logical restored back to back in 25 s, against
40257 ms for the weekly 100 percent check beside it. Individual restores 2253-3978 ms
regardless of size: cost is per-snapshot round-trip plus ~1 s per 200 MB. Peak scratch
is the app's full logical size. The 1.1 MB restic cache is index only and hides nothing
(--no-cache 5423 ms vs cached 3198 ms, trees byte-identical).
Q5 skip-if-busy stays right at 25 s against a 2m52s nightly backup. But
RestoreOffboxScratch takes NO acquireRunning, while offbox_integrity.go:28 asserts
every off-site operation does. R-408.
Q6 observed with a positively-controlled lock sampler: restic restore takes NO lock;
restic check DOES (locks 0 -> 1 for nine samples -> 0 across the check, zero across two
restores). The product writes anyway - unlockStale runs `restic unlock`, a delete verb,
before every restore (offbox_restore.go:289). The task's lead was right in direction and
wrong in mechanism. R-95's constraint IS satisfiable: --no-lock plus skipping unlockStale
writes nothing, and both mechanisms exist unused. Neither was fixed - the task forbids it.
offbox_integrity.go:255's "It NEVER writes to the repository" is R-407.
Q7 THE DECIDING ONE: of R-353/354/356/358/403 an unattended scratch-restore would have
caught ONE (R-356). The value is elsewhere, and the weekly check structurally cannot
reach it: `check` proves the stored bytes are the stored bytes, never that we stored the
RIGHT thing. A hollow unit backs up, checks at 100 percent and restores cleanly and
recovers nothing - R-403, measured in bytes on 31 August.
RECOMMENDATION: option C, the NARROW test - one app a night, restored to scratch, checked
against its own manifest.json through the existing unitCarriesData, scratch deleted, the
SNAPSHOT recorded as the proof. Options A (do not build) and B (scheduled attended drill)
considered explicitly; B is weakest because it is what already happens. R-87 should be
RE-SCOPED, not built as written, and that is Viktor's call - the row stays open carrying
the verdict and STATUS.md item 4 asks it in plain words.
Also corrected in 07-backup-architecture.md: matrix rows 4 and 10 both said "the depth
that ships ON does not re-read pack contents (R-399)". R-399 CLOSED in v0.228.0 and the
depth is 100 percent. Two stale cells, fixed, and the spike verdict added beside them.
Row 4's verdict is UNCHANGED by the spike and now says so.
Teardown: all three layers, none of them "nothing was created" - 6 files on the PVE host,
9 in the guest, 5 plus 2 run-flags in the container, all removed and verified empty. The
four scratch directories this session's restores created were removed; three that
pre-date the session were left alone. Two state changes recorded rather than hidden: the
control integrity run recorded its verdict (depth structure -> 100%, due-ness +7 days),
and four restores appear in the controller log. Nothing was written to the off-site
repository by hand.
Evidence: documentation/audits/evidence-spike-restic-restore-2026-08-31/ - 31 files,
every one pulled off the box BEFORE teardown (R-320).
golden-currency is RED at this commit and was already red at dddcc80. Pre-existing, not
this session's debt. Second --no-verify push of the day for that reason; R-404's count
goes six -> seven and its row says so.
Ceiling R-406 -> R-409.
This commit is contained in:
@@ -0,0 +1,465 @@
|
||||
# SPIKE — can the off-site (restic) copy be restore-tested without a person? (R-87)
|
||||
|
||||
**Date:** 2026-08-31 · **Venue:** `demo-hp` (Tier 0, disposable), guest 9201, controller **v0.230.0**
|
||||
**Class:** spike. **No production code was written.** No version bump, no build, no deploy, no golden.
|
||||
**Evidence:** `documentation/audits/evidence-spike-restic-restore-2026-08-31/` — 31 files, all pulled
|
||||
off the box before teardown (R-320).
|
||||
|
||||
**Baselines re-confirmed at the start of the session:** `felhom-controller` `main` @
|
||||
`2d802d75e88616d86cbade8a0e16965c2b85771c`, `v0.230.0`; `felhom.eu` @
|
||||
`dddcc808be95d1c89b276b4d791491bad3c96bba` (clean tree, `HEAD == origin/main`); `felhom-agent`
|
||||
`058b945`, `v0.130.0`.
|
||||
|
||||
---
|
||||
|
||||
## The one-paragraph answer
|
||||
|
||||
**restic 0.14.0 cannot tell us a restore produced correct files, and neither can anything else the
|
||||
box holds today** — `--verify` exists but checks size and modification time, not content, and the
|
||||
recovery unit's own manifest records a hash for 4 918 bytes of a 213 231 242-byte unit. **But an
|
||||
unattended restore-test is far cheaper than expected:** restoring *every* app on the box — 8
|
||||
snapshots, 774 MB logical — took **25 seconds**, against **40.3 seconds** for the weekly integrity
|
||||
check sitting beside it. And the question worth asking is not the one R-87 was filed for. Of the five
|
||||
restore-path defects human drills found between 2026-08-26 and 2026-08-31, an unattended
|
||||
scratch-restore would have caught **one**. What it *would* catch, and what the weekly check
|
||||
structurally cannot, is a snapshot that is perfectly intact and contains **nothing recoverable** —
|
||||
the R-403 shape, measured live nine days ago. **Recommendation: build the narrow version (option C
|
||||
below), and do not build the thing R-87 asks for.**
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — the register correction (done, pushed as `6e550ae`)
|
||||
|
||||
**When and where it went wrong.** R-87 was moved into `CLOSED-ITEMS.md` by commit **`ef6ac6f`**,
|
||||
2026-08-22, *"One register, enforced by a gate; closed work compressed into siblings
|
||||
(R-376..R-378)"*. Established from `git log -S`, not inferred: that commit is the only one that ever
|
||||
added an R-87 row to `CLOSED-ITEMS.md` and the only one that removed it from `OPEN-ITEMS.md`.
|
||||
|
||||
**It is a survivor of R-378, not a separate incident.** R-378 records that same sweep moving six
|
||||
still-open rows — R-123, R-190, R-214, R-264, R-295, R-352 — and restoring them verbatim in the same
|
||||
session. **R-87 is a seventh it missed.** Its state cell read
|
||||
`READY — RE-RANKED UP 2026-08-03 (R-86 closed)`: the leading verdict is `READY`, and the word
|
||||
`closed` later in the same cell describes a *different* row. Nine days in the wrong file, while
|
||||
`OPEN-ITEMS.md`'s ranking paragraph ranked it fourth and pointed at nothing.
|
||||
|
||||
**The count, reproduced independently — and the predicate decides the answer.**
|
||||
|
||||
| predicate | rows convicted in `CLOSED-ITEMS.md` (of 151) | which |
|
||||
|---|---|---|
|
||||
| open word anywhere in the **row** | 144 | meaningless |
|
||||
| open word anywhere in the **state cell** | 3 | R-87, **R-224**, **R-260** |
|
||||
| open word in the **leading verdict** | **1** | R-87 |
|
||||
|
||||
R-224 and R-260 are genuinely closed; their long prose verdicts merely contain the words "open" and
|
||||
"OPEN". **The task author's count of one is right, and it is right only under the leading-verdict
|
||||
predicate** — which is R-378's own lesson, restated by measurement.
|
||||
|
||||
**The gate:** `scripts/closed_register_gate.py`, two rules — no open state word leading a
|
||||
`CLOSED-ITEMS.md` row's verdict, and no `R-` id with a row in both registers. Red-proofed on both
|
||||
rules; negative-controlled against the files **as they were pushed**, where it convicts R-87 by name
|
||||
and exits 1. Registered as the 12th gate in `repo_gates.py` **after** it was green. Four residual
|
||||
holes are named in its docstring.
|
||||
|
||||
**One thing the second rule turned up:** `R-398` also had a row in both registers — a deliberate
|
||||
cross-reference stub. It is now prose beneath the table, not a table row.
|
||||
|
||||
**A duplicate this session did NOT fix:** `OPEN-ITEMS.md` carries two unrelated findings both
|
||||
numbered **R-133** (`:267` hub `customer_id` uniqueness; `:273` plaintext break-glass credential).
|
||||
Filed as **R-406**; deliberately not gated, because a within-register duplicate rule would fail on a
|
||||
pre-existing row and a registered-but-failing gate refuses every push.
|
||||
|
||||
---
|
||||
|
||||
## Q1 — what restic is actually running?
|
||||
|
||||
**Answer: restic 0.14.0, and every source comment asserting that is correct.**
|
||||
|
||||
From the running container on `demo-hp`, not from the Dockerfile:
|
||||
|
||||
```
|
||||
restic 0.14.0 compiled with go1.19.8 on linux/amd64
|
||||
/usr/bin/restic
|
||||
/etc/debian_version → 12.15 (Debian bookworm, as the Dockerfile says)
|
||||
```
|
||||
|
||||
Method: `pct exec 9201 -- docker exec felhom-controller restic version`, rc=0. Evidence
|
||||
`02-q1-restic-version.txt`. The comments at `offbox_capture.go:15`, `offbox_restore.go:21`,
|
||||
`offbox.go:726`, `offbox_progress.go:48,66` are **confirmed, not corrected**.
|
||||
|
||||
---
|
||||
|
||||
## Q2 — what can that version verify about a RESTORE?
|
||||
|
||||
**Answer: `--verify` exists, and it does NOT verify content. It cannot tell us a restore produced
|
||||
correct files.**
|
||||
|
||||
**`--verify` is real.** `restic restore --help` on the running container lists
|
||||
`--verify verify restored files content`. Controls, because a `grep -c 0` must be earned:
|
||||
|
||||
| control | result |
|
||||
|---|---|
|
||||
| positive — `--target` present | 1 hit |
|
||||
| `--verify` present | 1 hit, quoted verbatim above |
|
||||
| negative — `--delete`, `--dry-run`, `--overwrite`, `--sparse` (all post-0.14) | 0 hits each |
|
||||
| negative — `ZZZ-NOT-A-FLAG` | 0 hits |
|
||||
|
||||
Evidence `03-q2-restore-help.txt`, `04-q2-controls.txt`. **`--verify` and `--no-lock` both exist in
|
||||
0.14.0 and neither appears anywhere in the controller source** — `grep -rn` over
|
||||
`felhom-controller/controller/` returns rc=1 for both, with `--json`/`--target` as the positive
|
||||
control (`05-q2-codebase-verify-grep.txt`).
|
||||
|
||||
**What `--verify` actually checks — measured, with a red-proof and a negative control.**
|
||||
|
||||
| test | what was done | result |
|
||||
|---|---|---|
|
||||
| cost | virgin restore of kimai (213 231 242 B, 7 files) with `--verify` | verify itself **131 ms**; restore+verify 3 438 ms |
|
||||
| **red-proof** | corrupt one byte in a restored 160 MB tar, **size and mtime preserved**, re-run `restore --verify` | **PASSED clean, rc=0** — the corruption was not detected |
|
||||
| control (size) | truncate the same file by 1 byte, re-run | restic silently **re-downloaded** it; verify reported "7 files, 133 ms", rc=0 |
|
||||
| control (mtime) | corrupt content and bump mtime, re-run | restic silently **re-downloaded** it; rc=0 |
|
||||
| negative control | same command against the untouched copy | identical output — so "passed" carries no information |
|
||||
|
||||
**131 ms cannot hash 213 MB.** Combined with the red-proof, `--verify` in 0.14.0 is a size-and-mtime
|
||||
reconciliation that re-fetches anything that disagrees. It is useful — it makes a restore
|
||||
self-repairing — and it is **not** a content check. Evidence `20-`, `21-`, `22-`.
|
||||
|
||||
**This is the point at which the spike's shape changed**, per §9's instruction to stop and
|
||||
reconsider after Q1/Q2: the tool cannot supply the reference, so Q3 became the hard question.
|
||||
|
||||
---
|
||||
|
||||
## Q3 — if not restic, then what is the reference for "correct"?
|
||||
|
||||
**Answer: there is none available to an unattended test today. The one hash record that exists covers
|
||||
0.002 % of a unit's bytes.**
|
||||
|
||||
| candidate | verdict | why |
|
||||
|---|---|---|
|
||||
| restic `--verify` | **rejected** | size + mtime only — Q2's red-proof |
|
||||
| the snapshot's own metadata | **rejected** | `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps — **no content hash** (`24-q3-hash-coverage.txt`) |
|
||||
| a hash the controller already records | **rejected as a content reference, kept as a completeness one** | see below |
|
||||
| `restic check --read-data` | **already shipped, different question** | it proves the STORE's packs, never the restored files |
|
||||
| a planted sentinel | **rejected** | a drill technique. An unattended test may not write data into a customer's app to have something to look for |
|
||||
| the live data | **rejected** | it drifts by design; the snapshot is 12 h old by the time a check runs |
|
||||
| restore twice and compare | **rejected** | proves determinism, not correctness |
|
||||
|
||||
**The hash record that exists, and its exact coverage.** The recovery unit's `manifest.json` carries
|
||||
a `checksums` object. For kimai, measured on the restored unit:
|
||||
|
||||
```
|
||||
checksums: .felhom.yml (2 235 B), app.yaml (488 B), docker-compose.yml (2 195 B) = 4 918 B
|
||||
files in the unit: + kimai-mariadb.sql 48 217 B
|
||||
+ kimai_kimai_db_data.tar 160 331 776 B
|
||||
+ kimai_kimai_var.tar 52 845 056 B
|
||||
+ manifest.json 1 275 B total 213 231 242 B
|
||||
```
|
||||
|
||||
**4 918 of 213 231 242 bytes — 0.0023 %.** The three config files are hashed; the database dump and
|
||||
the two volume tars, which are the recoverable data, are not. Filed as **R-409**.
|
||||
|
||||
**What the manifest CAN answer is a different and better question.** It declares `db_dumps` and
|
||||
`volume_dumps` by name, and `unitCarriesData` (`r403_hollow.go:40`) already asks it. A restored unit
|
||||
can therefore be checked for **completeness** — does every file the manifest declares exist — with
|
||||
no new metadata, no new reference, and no content hash. That is the whole of the recommendation in
|
||||
§Recommendation.
|
||||
|
||||
---
|
||||
|
||||
## Q4 — what does one restore-test cost?
|
||||
|
||||
**Answer: about 4 seconds per app and 25 seconds for the whole box — cheaper than the weekly
|
||||
integrity check it would sit beside. The cost is dominated by per-snapshot round-trip, not by data
|
||||
volume.**
|
||||
|
||||
**Through the product's own path** (`POST /backup/offbox/restore` → `RestoreOffboxScratch`), timed
|
||||
from the POST to the completion log line:
|
||||
|
||||
| app | mode | logical size | wall clock |
|
||||
|---|---|---|---|
|
||||
| docmost | unit | 118 207 270 B | **9 s** (13:40:10 → 13:40:19) |
|
||||
| kimai | full | 213 231 242 B | **11 s** (13:42:54 → 13:43:05) |
|
||||
| opengist | unit | 185 664 B | ~8 s |
|
||||
|
||||
**Raw restic, all eight snapshots back to back** (`26-q4-all-apps-cost.txt`):
|
||||
|
||||
| app | logical | restored to disk | ms |
|
||||
|---|---|---|---|
|
||||
| privatebin | 2 123 006 | 2 159 870 | 2 628 |
|
||||
| opengist | 185 664 | 222 528 | 2 253 |
|
||||
| calibre-web | 5 808 703 | 5 878 335 | 3 642 |
|
||||
| paperless-ngx | 83 473 654 | 83 547 382 | 3 062 |
|
||||
| bookstack | 166 641 768 | 166 682 728 | 2 745 |
|
||||
| docmost | 118 207 270 | 118 248 230 | 3 676 |
|
||||
| romm | 184 706 816 | 184 747 776 | 3 978 |
|
||||
| kimai | 213 231 242 | 213 272 202 | 3 202 |
|
||||
| **all eight** | **774 378 123** | **774 759 051** | **25 s** |
|
||||
|
||||
**185 KB takes 2.25 s and 213 MB takes 3.20 s.** Nearly all of it is fixed per-snapshot cost —
|
||||
opening the repo, loading the index, one SFTP session. Data adds roughly **1 s per 200 MB**
|
||||
(≈ 67 MB/s on this link).
|
||||
|
||||
**The cache is not hiding the cost.** `/root/.cache/restic` is **1.1 MB** — index and metadata only.
|
||||
Cached 3 198 ms vs `--no-cache` 5 423 ms for kimai; the trees are byte-identical (`diff -r` → YES).
|
||||
Pack data always crosses the wire. Evidence `25-q4-cache-effect.txt`.
|
||||
|
||||
**Peak scratch:** the restore writes the **full logical size** — 213 272 202 B for the largest app.
|
||||
Sequential-with-cleanup needs only the largest app; all-at-once needs 774 MB.
|
||||
|
||||
**Against R-359's numbers.** R-359 measured 35.0 s at structure depth and 39.2 s at 100 % read-data
|
||||
on a 140 829 678 B store. Re-measured today through the product's own debug button on a
|
||||
141 959 062 B store: **40 257 ms at 100 %** (`15-q5-integrity-run.txt`). So:
|
||||
|
||||
> **restore-testing every app on the box (25 s) costs LESS than one weekly integrity check (40 s).**
|
||||
> Same order of magnitude, and on the cheaper side of it.
|
||||
|
||||
**Extrapolation — labelled as extrapolation.** R-401 records that the check's cost tracks the index
|
||||
and a read-data run's tracks the data. **A restore tracks the data too, plus a fixed per-snapshot
|
||||
cost.** With 8 apps and the measured ≈ 67 MB/s:
|
||||
|
||||
| store | fixed (8 × ~2 s) | data | total |
|
||||
|---|---|---|---|
|
||||
| today, 774 MB logical | 16 s | ~9 s | **25 s (measured)** |
|
||||
| 10× — 7.7 GB | 16 s | ~115 s | ~2.2 min |
|
||||
| 100× — 77 GB | 16 s | ~19 min | ~19 min |
|
||||
|
||||
**Why it may not hold.** One store, one link, one afternoon. This is DooPlex→Hetzner; a customer's
|
||||
domestic line is the real variable, and restore is the *download* direction, usually the faster one
|
||||
on such a line. Scratch space becomes the binding constraint long before time does: at 100× the
|
||||
largest app would need ~21 GB of free scratch, and `RestoreOffboxScratch`'s headroom gate would start
|
||||
refusing. **Unknown, and what would settle it:** a measurement on a store above 10 GB. None exists.
|
||||
This is the same single-data-point hole R-401 already owns.
|
||||
|
||||
---
|
||||
|
||||
## Q5 — contention
|
||||
|
||||
**Answer: skip-if-busy stays right, because the operation is seconds and not minutes — but the
|
||||
restore path takes NO single-writer flag at all today, which is a bigger problem than contention.**
|
||||
|
||||
**The configured window on this box**, read from the scheduler's own registrations rather than
|
||||
assumed (`28-q5-schedule.txt`, times are **CEST** — the guest scheduler runs in local time, not UTC):
|
||||
|
||||
| job | time | measured duration |
|
||||
|---|---|---|
|
||||
| db-dump | 02:30 | — |
|
||||
| tier2-backup | 03:30 | — |
|
||||
| **offbox-backup** | **04:15** | **2m52s** (last run, `last_duration`) |
|
||||
| offsite-abandon-sweep | 05:10 | — |
|
||||
| **offsite-integrity** | **06:00** | **40.3 s** at 100 % depth |
|
||||
|
||||
A restore-test of all eight apps holds anything it holds for **25 s** — one seventh of the nightly
|
||||
off-site backup, and shorter than the check beside it. There is an empty gap from ~04:18 to 06:00.
|
||||
**Skip-if-busy remains the right policy**, and it is not the "minutes rather than seconds" case the
|
||||
task worried about. The `integrityCheckTimeout` reasoning transfers unchanged.
|
||||
|
||||
**The real finding here.** `offbox_integrity.go:28` states the invariant:
|
||||
|
||||
> *"Every off-site operation takes `acquireRunning` for exactly that reason."*
|
||||
|
||||
**It does not.** `grep -rn 'acquireRunning()'` finds nine non-test callers; `RestoreOffboxScratch`
|
||||
(`offbox_restore.go:234`) is **not** among them. `restore_wizard.go:174` says the same thing
|
||||
independently — *"`RestoreOffboxScratch` never acquires it at all"* — and the UI works around it with
|
||||
a separate display flag. So a scheduled, unattended caller built on `RestoreOffboxScratch` today
|
||||
would run with **no single-writer flag**, which is precisely the hazard the integrity file's header
|
||||
is shaped around. **A comment asserting an invariant with no test pinning it.** Filed as **R-408**.
|
||||
|
||||
---
|
||||
|
||||
## Q6 — what does a restore-test WRITE to the repository?
|
||||
|
||||
**Answer: `restic restore` writes nothing — it does not even take a lock. The product's restore path
|
||||
writes anyway, because it runs `restic unlock` before every restore. And `restic check` — the thing
|
||||
whose comment says it never writes — DOES take a lock.**
|
||||
|
||||
**Method.** Two observers inside the container, both proven before being believed:
|
||||
|
||||
1. a lock sampler running `restic list locks --no-lock` every ~4 s (the observer itself never writes);
|
||||
2. an argv sampler reading `/proc/*/cmdline` every 0.2 s, with the repo URL and `sftp.command`
|
||||
redacted at the source.
|
||||
|
||||
**Positive control for the lock sampler — it works.** The product's own integrity check ran
|
||||
13:41:28 → 13:42:11:
|
||||
|
||||
```
|
||||
13:41:30 locks=0
|
||||
13:41:34 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf
|
||||
... nine consecutive samples, one lock ...
|
||||
13:42:10 locks=1 ids=81fd…
|
||||
13:42:14 locks=0
|
||||
```
|
||||
|
||||
**So `restic check` writes a lock file to the repository.** `offbox_integrity.go:255` says
|
||||
*"It NEVER writes to the repository: `check` is a read verb, and nothing here prunes, forgets,
|
||||
unlocks or backs up."* The three named verbs are correct; the sentence's headline is not. Filed as
|
||||
**R-407**.
|
||||
|
||||
**The restore, with the same proven instrument:** locks=0 across every sample inside both restore
|
||||
windows (13:40:11/:15/:19 for docmost, 13:42:57/:43:01/:05 for kimai). **restic 0.14.0's `restore`
|
||||
does not lock the repository.**
|
||||
|
||||
**What the product runs, observed argv (redacted), one restore of `opengist`:**
|
||||
|
||||
```
|
||||
13:44:41 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:44 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
13:44:46 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target … --include …
|
||||
```
|
||||
|
||||
Three invocations. The middle one is `unlockStale` (`offbox_restore.go:289` → `offbox.go:743`), which
|
||||
runs **unconditionally before every restore** and is a **delete verb against `locks/`**. With no
|
||||
stale lock present it deletes nothing — but it is a write-capable command on the exact path R-87
|
||||
wants to run unattended. `resticStep`'s escalation to `unlock --remove-all` (`offbox.go:760-775`)
|
||||
fires only on `repository is already locked`, which a restore cannot provoke by itself now that we
|
||||
know restore takes no lock.
|
||||
|
||||
**Against §5's constraint** — *"R-95 still applies: that credential can delete, so a restic
|
||||
restore-test must never be able to write to the repo"*:
|
||||
|
||||
- **The lead in the task was right in direction and wrong in mechanism.** The write is not
|
||||
`resticStep`'s escalation; it is `unlockStale`, one line earlier and unconditional.
|
||||
- **The constraint IS satisfiable, cheaply, and both mechanisms already exist in 0.14.0 and are
|
||||
unused:** a restore-test that passes `--no-lock` and skips `unlockStale` writes nothing to the
|
||||
repository at all. That is a design note for whoever builds it — **this spike did not fix it**, per
|
||||
§7.
|
||||
|
||||
---
|
||||
|
||||
## Q7 — what would an unattended restore-test catch that the weekly 100 % check does not?
|
||||
|
||||
**Answer: of the five defects human drills found in the last six days, one. But that is the wrong
|
||||
scoreboard, and the right one has a much better answer.**
|
||||
|
||||
| defect | would an unattended scratch-restore have caught it? | how / why not |
|
||||
|---|---|---|
|
||||
| **R-353** — a **local** unit restore reported a bare completion whether it returned a dataset or nothing | **NO** | different code path entirely (`restore_unit.go`). An off-site restore-test never enters it |
|
||||
| **R-354** — the off-site full restore has no named-volume replay leg: the tar reaches the scratch and is never replayed | **NO** | the defect is *after* the scratch. A test that stops at the scratch sees a correct scratch. Going further means replaying into a live app, which an unattended test must not do |
|
||||
| **R-356** — the off-site restore refused every app with no data drive (40 of 53 catalogue templates) | **YES** | the test calls `RestoreOffboxScratch`, gets a refusal, and `rerr != nil`. On this box five of eight apps live on `/mnt/sys_drive` with no drive — it would have fired on the first night |
|
||||
| **R-358** — `OffboxFullScratchReady` asked "non-empty directory", which is what a *failed* restic run leaves | **NO** | needs a restore that fails part-way. On a healthy run the broken and the fixed predicate agree |
|
||||
| **R-403** — a hollow unit mirrored over a complete one with `--delete`; 120 082 104 B → 7 036 B, recorded as success | **NO as filed** — but **YES for the shape** | R-403 destroyed a *local* second-drive copy. What a restore-test sees is the consequence: once a hollow unit is captured off-site, the snapshot is perfectly intact and contains nothing recoverable |
|
||||
|
||||
**One of five. If the question is "does our restore code work", the honest answer is: the drills
|
||||
already answer it, they answer it better, and automating a worse version of it is not worth an
|
||||
evening.**
|
||||
|
||||
**The scoreboard that matters is different, and the weekly check structurally cannot play on it.**
|
||||
|
||||
> `restic check --read-data-subset=100%` proves that **the bytes we stored are the bytes we stored.**
|
||||
> It cannot tell us **we stored the wrong thing.**
|
||||
|
||||
A hollow recovery unit — no database dump, no volume tar — backs up cleanly, checks cleanly at 100 %
|
||||
depth, restores cleanly, and recovers nothing. **R-403 proved that shape is real, on this fleet, nine
|
||||
days ago, measured in bytes.** Nothing in the product asks the question today, on any tier, at any
|
||||
cadence. A restore-test is simply the cheapest place to ask it, because the manifest that answers it
|
||||
travels inside the snapshot.
|
||||
|
||||
---
|
||||
|
||||
## Recommendation to Viktor — three options, with costs
|
||||
|
||||
**Option A — do not build it.**
|
||||
*Cost:* nothing. *What you get:* the weekly 100 % check keeps proving the stored bytes; the restore
|
||||
code keeps being proven by your drills.
|
||||
*What you lose:* nothing that has bitten yet — and the R-403 shape stays invisible until a customer
|
||||
needs the data. **This is a defensible answer** and it is the one R-87's original framing deserves.
|
||||
|
||||
**Option B — a scheduled attended drill instead.**
|
||||
*Cost:* one of your evenings, monthly. *What you get:* everything a person can see, including the
|
||||
R-354 class that stops at the scratch.
|
||||
*Honest objection:* this is what already happens, and it is what found all five defects. Scheduling
|
||||
it changes nothing except that it now has a date. **Low value for the price.**
|
||||
|
||||
**Option C — build the NARROW unattended test: one app per night, rotating, restored to scratch, and
|
||||
checked against its own manifest. RECOMMENDED.**
|
||||
|
||||
*What it does, in one sentence:* restore the newest off-site snapshot of one app into the throwaway
|
||||
scratch, assert that every file the unit's `manifest.json` declares is present, record which snapshot
|
||||
was proved, delete the scratch.
|
||||
|
||||
*Measured cost, not estimated:* **~4 s and ≤ 213 MB of scratch per night** (one app), or 25 s for all
|
||||
eight. Off-site traffic: one restore's worth, ≈ the app's size. **Less than the weekly integrity
|
||||
check already running beside it.** No new metadata, no new reference, no content hashes — it reuses
|
||||
`unitCarriesData`'s existing manifest read.
|
||||
|
||||
*What it catches:* R-356 outright, and the R-403 class — a snapshot that is intact and empty — which
|
||||
nothing else in the product asks about.
|
||||
*What it does not catch, stated so nobody expects it to:* R-353, R-354, R-358. Those stay drill work.
|
||||
|
||||
*Three things it must be built with, all established by this spike:*
|
||||
1. **`--no-lock`, and skip `unlockStale`** — then it writes nothing to the repository and R-95's
|
||||
constraint is honoured for real (Q6).
|
||||
2. **Take `acquireRunning`** — `RestoreOffboxScratch` does not, and the whole off-site
|
||||
single-writer story assumes every operation does (Q5, R-408).
|
||||
3. **Record the SNAPSHOT it proved, not a timestamp** — R-87's own row already says this, and R-86
|
||||
built the per-archive due-ness model to copy.
|
||||
|
||||
**If you do nothing:** the weekly check keeps running and keeps being right about the bytes. The
|
||||
first time a hollow unit reaches the off-site store, nothing will notice, and the discovery will be a
|
||||
customer's restore. That is not a hypothetical shape — it is R-403, measured on 2026-08-31.
|
||||
|
||||
**I would pick C**, scoped exactly as above. It is cheaper than the check beside it, it asks a
|
||||
question nothing else asks, and it needs no invention.
|
||||
|
||||
**R-87 itself should be RE-SCOPED, not built as written** — from *"restore-test the restic tier"* to
|
||||
*"prove the off-site snapshot still contains a recoverable unit"*. That is your call, so R-87 stays
|
||||
open carrying this verdict.
|
||||
|
||||
---
|
||||
|
||||
## Register rows opened by this spike
|
||||
|
||||
| id | what |
|
||||
|---|---|
|
||||
| **R-405** | R-87 was mis-filed by `ef6ac6f`; corrected + gated (CLOSED same session) |
|
||||
| **R-406** | two unrelated findings share the id R-133 in `OPEN-ITEMS.md` |
|
||||
| **R-407** | `restic check` DOES take a repository lock; `offbox_integrity.go:255` says it never writes |
|
||||
| **R-408** | `RestoreOffboxScratch` takes no `acquireRunning`; `offbox_integrity.go:28` asserts every off-site operation does |
|
||||
| **R-409** | the recovery-unit manifest hashes 4 918 B of a 213 231 242 B unit — the data files have no recorded hash |
|
||||
|
||||
---
|
||||
|
||||
## Probes, teardown and side-effects
|
||||
|
||||
**All three layers, and two of them really are "nothing was left".**
|
||||
|
||||
| layer | created | removed |
|
||||
|---|---|---|
|
||||
| PVE host `demo-hp` `/root` | 6 helper scripts + one 0600 password file | all removed; `ls \| grep` returns nothing |
|
||||
| guest 9201 `/root`, `/tmp` | 9 helper/session files + `spike-env.sh` | all removed; grep returns nothing |
|
||||
| container `/tmp` | `spike-env.sh`, two samplers, two logs, two run-flags | all removed; `/tmp` lists empty |
|
||||
|
||||
**Scratch directories:** four (`docmost`, `kimai`, `privatebin`, `opengist`) were created by the
|
||||
restores this spike drove and were **removed**. Three (`bookstack`, `calibre-web`, `paperless-ngx`)
|
||||
pre-date this session and were **left alone** — this spike never restored them.
|
||||
|
||||
**Two deliberate state changes on the box, recorded rather than hidden:**
|
||||
|
||||
1. The integrity check I ran to control the lock observer **recorded its verdict** — the box's
|
||||
`last_integrity_check` moved to `2026-08-31T13:41:28Z`, `ok=true`, and
|
||||
`last_integrity_depth` moved from `structure` to **`100%`**. Due-ness advanced by seven days. That
|
||||
is the product behaving correctly; it is not a repair and it is not damage.
|
||||
2. Two logins and four restores appear in the controller log and in the customer-visible restore-op
|
||||
history.
|
||||
|
||||
**Nothing was written to the off-site repository by hand.** No prune, no forget, no `unlock` issued
|
||||
by me. Every write observed in Q6 was the product's own.
|
||||
|
||||
**No controller code changed. No golden is owed.** The fleet stays on v0.230.0. The one delivery debt
|
||||
that exists — `golden_currency_gate.py` red because v0.230.0 has no golden — was **already red at
|
||||
`dddcc80`** before this session began, and belongs to the v0.230.0 release, not to this task.
|
||||
|
||||
---
|
||||
|
||||
## Observations noticed and not acted on
|
||||
|
||||
- **`paperless-ngx` and `filebrowser` have no off-site snapshot** under the tags the inventory uses —
|
||||
`paperless` returns nothing and `paperless-ngx` returns one; `filebrowser` returns none at all.
|
||||
`filebrowser` is infrastructure, so that may be correct. Not chased.
|
||||
- **`CLOSED-ITEMS.md` has two malformed rows** — R-399 and R-400 supply two columns where the table
|
||||
declares four, so they render with no `Shipped` and no `Evidence`. The new gate prints them as a
|
||||
warning rather than convicting, because an empty state cell is not an open state word.
|
||||
- **Two rows carry a `|` inside their body** (R-309, R-351), which shifts their own cells. Named as
|
||||
the new gate's first residual hole.
|
||||
- **The controller image has no `ps` and no `python3`** — worth knowing before writing any probe that
|
||||
runs inside it. `/proc/*/cmdline` is the substitute that works.
|
||||
- **`git push --no-verify` was used for this session's Part 1 commit** — bypass **#7**, for the reason
|
||||
R-404 exists: a documentation-only push met `golden_currency_gate.py`. Recorded here because R-404
|
||||
counts them.
|
||||
@@ -0,0 +1,4 @@
|
||||
demo-hp
|
||||
15:32:56 up 9 days, 21:48, 1 user, load average: 0.70, 0.63, 0.57
|
||||
VMID Status Lock Name
|
||||
9201 running demo-hp
|
||||
@@ -0,0 +1,20 @@
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = "UTF-8",
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.230.0 Up 59 minutes (healthy)
|
||||
@@ -0,0 +1,4 @@
|
||||
restic 0.14.0 compiled with go1.19.8 on linux/amd64
|
||||
restic-rc=0
|
||||
/usr/bin/restic
|
||||
12.15
|
||||
@@ -0,0 +1,64 @@
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = "UTF-8",
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
|
||||
The "restore" command extracts the data from a snapshot from the repository to
|
||||
a directory.
|
||||
|
||||
The special snapshot "latest" can be used to restore the latest snapshot in the
|
||||
repository.
|
||||
|
||||
EXIT STATUS
|
||||
===========
|
||||
|
||||
Exit status is 0 if the command was successful, and non-zero if there was any error.
|
||||
|
||||
Usage:
|
||||
restic restore [flags] snapshotID
|
||||
|
||||
Flags:
|
||||
-e, --exclude pattern exclude a pattern (can be specified multiple times)
|
||||
-h, --help help for restore
|
||||
-H, --host stringArray only consider snapshots for this host when the snapshot ID is "latest" (can be specified multiple times)
|
||||
--iexclude --exclude same as --exclude but ignores the casing of filenames
|
||||
--iinclude --include same as --include but ignores the casing of filenames
|
||||
-i, --include pattern include a pattern, exclude everything else (can be specified multiple times)
|
||||
--path path only consider snapshots which include this (absolute) path for snapshot ID "latest"
|
||||
--tag taglist only consider snapshots which include this taglist for snapshot ID "latest" (default [])
|
||||
-t, --target string directory to extract data to
|
||||
--verify verify restored files content
|
||||
|
||||
Global Flags:
|
||||
--cacert file file to load root certificates from (default: use system certificates)
|
||||
--cache-dir directory set the cache directory. (default: use system default cache directory)
|
||||
--cleanup-cache auto remove old cache directories
|
||||
--compression mode compression mode (only available for repository format version 2), one of (auto|off|max) (default auto)
|
||||
--insecure-tls skip TLS certificate verification when connecting to the repository (insecure)
|
||||
--json set output mode to JSON for commands that support it
|
||||
--key-hint key key ID of key to try decrypting first (default: $RESTIC_KEY_HINT)
|
||||
--limit-download int limits downloads to a maximum rate in KiB/s. (default: unlimited)
|
||||
--limit-upload int limits uploads to a maximum rate in KiB/s. (default: unlimited)
|
||||
--no-cache do not use a local cache
|
||||
--no-lock do not lock the repository, this allows some operations on read-only repositories
|
||||
-o, --option key=value set extended option (key=value, can be specified multiple times)
|
||||
--pack-size uint set target pack size in MiB, created pack files may be larger (default: $RESTIC_PACK_SIZE)
|
||||
--password-command command shell command to obtain the repository password from (default: $RESTIC_PASSWORD_COMMAND)
|
||||
-p, --password-file file file to read the repository password from (default: $RESTIC_PASSWORD_FILE)
|
||||
-q, --quiet do not output comprehensive progress report
|
||||
-r, --repo repository repository to backup to or restore from (default: $RESTIC_REPOSITORY)
|
||||
--repository-file file file to read the repository location from (default: $RESTIC_REPOSITORY_FILE)
|
||||
--tls-client-cert file path to a file containing PEM encoded TLS client certificate and private key
|
||||
-v, --verbose n be verbose (specify multiple times or a level using --verbose=n, max level/times is 3)
|
||||
=== rc=0 ===
|
||||
@@ -0,0 +1,11 @@
|
||||
--- POSITIVE CONTROL: a flag that MUST be there ---
|
||||
1
|
||||
--- the verify flag, verbatim ---
|
||||
--verify verify restored files content
|
||||
--- NEGATIVE CONTROL: flags added AFTER 0.14, must be absent ---
|
||||
--delete hits=0
|
||||
--dry-run hits=0
|
||||
--overwrite hits=0
|
||||
--sparse hits=0
|
||||
--- NEGATIVE CONTROL 2: a string that cannot be there ---
|
||||
0
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
=== '--verify' anywhere in controller/ (all files, incl gitignored cmd/) ===
|
||||
grep-rc=1
|
||||
|
||||
=== POSITIVE CONTROL: '--no-lock' ===
|
||||
grep-rc=1
|
||||
|
||||
=== every restic flag token used in internal/backup/ (non-test) ===
|
||||
"--delete" "--group-by" "--ignore-existing" "--include" "--itemize-changes" "--json" "--keep-daily" "--keep-monthly" "--keep-weekly" "--mode" "--prune" "--read-data" "--remove-all" "--rm" "--tag" "--target"
|
||||
@@ -0,0 +1,48 @@
|
||||
felhom-controller Up About an hour (healthy)
|
||||
docmost Up About an hour (healthy)
|
||||
docmost-redis Up About an hour (healthy)
|
||||
docmost-postgres Up About an hour (healthy)
|
||||
romm Up 4 hours (healthy)
|
||||
romm-db Up 4 hours (healthy)
|
||||
romm-redis Up 4 hours (healthy)
|
||||
privatebin Up 4 hours (healthy)
|
||||
paperless-webserver Up 4 hours (healthy)
|
||||
paperless-postgres Up 4 hours (healthy)
|
||||
paperless-redis Up 4 hours (healthy)
|
||||
opengist Up 4 hours (healthy)
|
||||
kimai Up 4 hours (healthy)
|
||||
kimai-db Up 4 hours (healthy)
|
||||
calibre-web Up 4 hours (healthy)
|
||||
bookstack Up 4 hours (healthy)
|
||||
bookstack-db Up 4 hours (healthy)
|
||||
filebrowser Up 9 days (healthy)
|
||||
cloudflared Up 9 days
|
||||
traefik Up 9 days
|
||||
=== guest disk ===
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/mapper/pve-vm--9201--disk--0 32G 957M 29G 4% /
|
||||
/dev/mapper/pve-vm--9201--disk--1 69G 12G 55G 18% /var/lib/felhom
|
||||
/dev/mapper/pve-root 39G 17G 21G 46% /mnt/felhom-drives
|
||||
none 492K 4.0K 488K 1% /dev
|
||||
udev 14G 0 14G 0% /dev/tty
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/0e82b836f4879d75df7380c71546ade764fc20c0f1c7e1d7905ab790ddbc7a4e/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/1924a292e532a1948a03c280bce4b21f48f109765d427ca110044f5e1469fcfe/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/ba706ad1711d7d7d8b31250f124caac3e889a66be994e9bba959d7e9542ee76a/merged
|
||||
/dev/nvme0n1 938G 6.0G 885G 1% /mnt/felhom-drives/hdd_1
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/c52ca60013e4af6d6d93a4ca471d9c1dffc7f99a9923869e4d72f47e17dfa44f/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/e79e25da71576531d24348a67d88e2c7d1c9e4105622ae95cd7a40bd6c52cb57/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/8131f6f54062f0094af0e03a1ad26a7c77ed6991e27a10211403cfbb7cb479a1/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/932ec3a6439333b5ecf8b39cb68795e15bdd1dc4a7fb831cae46bd9625ebaeb2/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/a2d41f6c1de3963c93bce576a41d0922695ec4d1460b4ebdd9a398245028182e/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/c0efcc2fbbb80d78ee5d764a3f9f6ab9e622d1df2faac473068aa0486f2c3d66/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/b976192342358497830be3cd329d63ca9546550da2f849b2baa44c041ded1c17/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/a3d1768da870b1d86015d6850275ebba59cb905e4415f9cd399c1bf15b6d08e1/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/a0f134a5b31237fbc56ea1fb38113d3a0352f9d4a3d36e5e40b3cff874b42366/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/1400ead8a09768fd52c279ea50cffbd5b87d061ef83fb93a0fa0685d7e0b027b/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/3723f8b898e197ad12b8fcaef66c08569cc9040d1efd1cba7d8797b5a7f0b14f/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/25ee94f6cfa646a342dd751ca14f58e56f3227aaaf9160d26409c4cc23994453/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/9e92315ab21675a0f271efb4c8eb0ed792a4de69dd58ec5e51300dfa3852c4b9/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/bb1ad0c258566bc3d4013171500fa6a7ed30a80ee620bc93ef8adb939eb0c166/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/2461db88cd4e3b967ab86cdc6a41de3f62b433ccbc1ca0424876a79ac0ba8d10/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/ba1fcae5ed394c9dbdee666392cc61400ef700a69f30ad98fe2de38ef43d7a24/merged
|
||||
overlay 69G 12G 55G 18% /var/lib/docker/overlay2/688b974f0cd67fd37641d3ce2adb9c696ab09074a966dbbbd0654fd47fc5b57e/merged
|
||||
@@ -0,0 +1,13 @@
|
||||
last_run '2026-08-31T02:17:57Z'
|
||||
last_success '2026-08-31T02:17:57Z'
|
||||
last_status 'ok'
|
||||
last_duration '2m52s'
|
||||
schedule 'daily'
|
||||
repo_size_bytes 141959062
|
||||
repo_size_human '135.4 MB'
|
||||
snapshot_count 67
|
||||
stats_known True
|
||||
quota_gb 50
|
||||
last_integrity_check '2026-08-31T08:31:18Z'
|
||||
last_integrity_ok True
|
||||
last_integrity_depth 'structure'
|
||||
@@ -0,0 +1,8 @@
|
||||
--- locks BEFORE (observer uses --no-lock) ---
|
||||
lock-count=0
|
||||
--- snapshots for docmost ---
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------------------------------------------------------------
|
||||
a07c36a1 2026-08-31 02:17:11 demo-hp felhom-offbox,docmost /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
--------------------------------------------------------------------------------------------------------------------
|
||||
1 snapshots
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
=== repo totals ===
|
||||
{"total_size":5958072637,"total_file_count":1380,"snapshots_count":67}
|
||||
|
||||
=== per-app latest snapshot: id, size ===
|
||||
docmost a07c36a1 {"total_size":118207270,"total_file_count":20,"snapshots_count":1}
|
||||
romm df7ad253 {"total_size":184706816,"total_file_count":18,"snapshots_count":1}
|
||||
privatebin bb81b9da {"total_size":2123006,"total_file_count":13,"snapshots_count":1}
|
||||
paperless NO-SNAPSHOT
|
||||
opengist b5aa8f9b {"total_size":185664,"total_file_count":13,"snapshots_count":1}
|
||||
kimai e7dcf8fd {"total_size":213231242,"total_file_count":16,"snapshots_count":1}
|
||||
calibre-web ad2ae49c {"total_size":5808703,"total_file_count":38,"snapshots_count":1}
|
||||
bookstack 91154be7 {"total_size":166641768,"total_file_count":19,"snapshots_count":1}
|
||||
filebrowser NO-SNAPSHOT
|
||||
@@ -0,0 +1,6 @@
|
||||
observer running; first samples:
|
||||
13:39:17 locks=0
|
||||
ERR
|
||||
=== scratch BEFORE ===
|
||||
ls: cannot access '/mnt/felhom-drives/*/felhom-data/backups/offsite-restore/': No such file or directory
|
||||
949346897920
|
||||
@@ -0,0 +1,2 @@
|
||||
t0=2026-08-31T13:40:10.095555740Z
|
||||
post_http=302 post_time=0.010129
|
||||
@@ -0,0 +1,6 @@
|
||||
DONE after 2 polls
|
||||
=== controller log, restore window ===
|
||||
2026/08/31 13:40:10 auth.go:134: [DEBUG] [web] auth: valid session for POST /backup/offbox/restore
|
||||
2026/08/31 13:40:10 server.go:393: [DEBUG] [web] ServeHTTP: POST /backup/offbox/restore from 172.18.0.4:34642
|
||||
2026/08/31 13:40:19 offbox_restore.go:298: [INFO] [offbox] restored docmost (a07c36a1, full=false) → /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost
|
||||
2026/08/31 13:40:19 offbox_handlers.go:372: [INFO] [web] off-box restore docmost completed (full=false, async)
|
||||
@@ -0,0 +1,33 @@
|
||||
=== lock observer samples ===
|
||||
13:39:55 rc=0 locks=0 ids=
|
||||
13:39:59 rc=0 locks=0 ids=
|
||||
13:40:03 rc=0 locks=0 ids=
|
||||
13:40:07 rc=0 locks=0 ids=
|
||||
13:40:11 rc=0 locks=0 ids=
|
||||
13:40:15 rc=0 locks=0 ids=
|
||||
13:40:19 rc=0 locks=0 ids=
|
||||
13:40:23 rc=0 locks=0 ids=
|
||||
13:40:27 rc=0 locks=0 ids=
|
||||
|
||||
=== scratch AFTER (unit restore) ===
|
||||
118761552 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T140924Z-docmost-postgres.sql
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T140713Z-docmost-postgres.sql
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T140501Z-docmost-postgres.sql
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T135827Z-docmost-postgres.sql
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/docmost-postgres.sql
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/compose/.felhom.yml
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/compose/docker-compose.yml
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/compose/app.yaml
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/volume-dumps/docmost_docmost_postgres_data.tar
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/volume-dumps/docmost_docmost_redis_data.tar
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/volume-dumps/docmost_docmost_storage.tar
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/manifest.json
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/.felhom-restore-complete.json
|
||||
|
||||
file count: 16
|
||||
=== marker ===
|
||||
{"schema":1,"snapshot_id":"a07c36a1","full":false,"finished_at":"2026-08-31T13:40:19Z"}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,5 @@
|
||||
t0=13:41:28
|
||||
{"data":{"depth":"100%","duration_ms":40257,"ok":true,"read_data_subset":"100%","skip_reason":"","skipped":false,"unreachable":false},"message":"Az ellenőrzés rendben lezajlott","ok":true}
|
||||
|
||||
http=200 time_total=42.701280
|
||||
t1=13:42:11
|
||||
+1
@@ -0,0 +1 @@
|
||||
sh: 1: Syntax error: Unterminated quoted string
|
||||
@@ -0,0 +1,170 @@
|
||||
13:39:55 rc=0 locks=0 ids=
|
||||
13:39:59 rc=0 locks=0 ids=
|
||||
13:40:03 rc=0 locks=0 ids=
|
||||
13:40:07 rc=0 locks=0 ids=
|
||||
13:40:11 rc=0 locks=0 ids=
|
||||
13:40:15 rc=0 locks=0 ids=
|
||||
13:40:19 rc=0 locks=0 ids=
|
||||
13:40:23 rc=0 locks=0 ids=
|
||||
13:40:27 rc=0 locks=0 ids=
|
||||
13:40:31 rc=0 locks=0 ids=
|
||||
13:40:35 rc=0 locks=0 ids=
|
||||
13:40:39 rc=0 locks=0 ids=
|
||||
13:40:43 rc=0 locks=0 ids=
|
||||
13:40:47 rc=0 locks=0 ids=
|
||||
13:40:50 rc=0 locks=0 ids=
|
||||
13:40:54 rc=0 locks=0 ids=
|
||||
13:40:58 rc=0 locks=0 ids=
|
||||
13:41:02 rc=0 locks=0 ids=
|
||||
13:41:06 rc=0 locks=0 ids=
|
||||
13:41:10 rc=0 locks=0 ids=
|
||||
13:41:14 rc=0 locks=0 ids=
|
||||
13:41:18 rc=0 locks=0 ids=
|
||||
13:41:22 rc=0 locks=0 ids=
|
||||
13:41:26 rc=0 locks=0 ids=
|
||||
13:41:30 rc=0 locks=0 ids=
|
||||
13:41:34 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf
|
||||
13:41:38 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf
|
||||
13:41:42 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf
|
||||
13:41:46 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf
|
||||
13:41:50 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf
|
||||
13:41:54 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf
|
||||
13:41:58 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf
|
||||
13:42:02 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf
|
||||
13:42:06 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf
|
||||
13:42:10 rc=0 locks=1 ids=81fd4d4200d848466e18cb7a9d8e0d4300d95707c68417449178c3023531eecf
|
||||
13:42:14 rc=0 locks=0 ids=
|
||||
13:42:18 rc=0 locks=0 ids=
|
||||
13:42:22 rc=0 locks=0 ids=
|
||||
13:42:26 rc=0 locks=0 ids=
|
||||
13:42:30 rc=0 locks=0 ids=
|
||||
13:42:34 rc=0 locks=0 ids=
|
||||
13:42:38 rc=0 locks=0 ids=
|
||||
13:42:42 rc=0 locks=0 ids=
|
||||
13:42:46 rc=0 locks=0 ids=
|
||||
13:42:49 rc=0 locks=0 ids=
|
||||
13:42:53 rc=0 locks=0 ids=
|
||||
13:42:57 rc=0 locks=0 ids=
|
||||
13:43:01 rc=0 locks=0 ids=
|
||||
13:43:05 rc=0 locks=0 ids=
|
||||
13:43:09 rc=0 locks=0 ids=
|
||||
13:43:13 rc=0 locks=0 ids=
|
||||
13:43:17 rc=0 locks=0 ids=
|
||||
13:43:21 rc=0 locks=0 ids=
|
||||
13:43:25 rc=0 locks=0 ids=
|
||||
13:43:29 rc=0 locks=0 ids=
|
||||
13:43:33 rc=0 locks=0 ids=
|
||||
13:43:37 rc=0 locks=0 ids=
|
||||
13:43:41 rc=0 locks=0 ids=
|
||||
13:43:45 rc=0 locks=0 ids=
|
||||
13:43:49 rc=0 locks=0 ids=
|
||||
13:43:53 rc=0 locks=0 ids=
|
||||
13:43:57 rc=0 locks=0 ids=
|
||||
13:44:01 rc=0 locks=0 ids=
|
||||
13:44:05 rc=0 locks=1 ids=9222fe46cd531ebe2c729af8ef3912bd77bea8cbe72b0527ce7702d5db678c4f
|
||||
13:44:08 rc=0 locks=0 ids=
|
||||
13:44:12 rc=0 locks=0 ids=
|
||||
13:44:16 rc=0 locks=0 ids=
|
||||
13:44:20 rc=0 locks=0 ids=
|
||||
13:44:24 rc=0 locks=0 ids=
|
||||
13:44:28 rc=0 locks=0 ids=
|
||||
13:44:32 rc=0 locks=0 ids=
|
||||
13:44:36 rc=0 locks=0 ids=
|
||||
13:44:40 rc=0 locks=0 ids=
|
||||
13:44:44 rc=0 locks=0 ids=
|
||||
13:44:48 rc=0 locks=1 ids=7a045a2731f7cdbb2aebf000018f040ed4f81d3ede59ed55921975cfbd8d4a61
|
||||
13:44:52 rc=0 locks=0 ids=
|
||||
13:44:56 rc=0 locks=0 ids=
|
||||
13:45:00 rc=0 locks=0 ids=
|
||||
13:45:04 rc=0 locks=0 ids=
|
||||
13:45:08 rc=0 locks=0 ids=
|
||||
13:45:12 rc=0 locks=0 ids=
|
||||
13:45:16 rc=0 locks=0 ids=
|
||||
13:45:20 rc=0 locks=0 ids=
|
||||
13:45:24 rc=0 locks=0 ids=
|
||||
13:45:28 rc=0 locks=0 ids=
|
||||
13:45:31 rc=0 locks=0 ids=
|
||||
13:45:35 rc=0 locks=0 ids=
|
||||
13:45:39 rc=0 locks=0 ids=
|
||||
13:45:43 rc=0 locks=0 ids=
|
||||
13:45:47 rc=0 locks=0 ids=
|
||||
13:45:51 rc=0 locks=0 ids=
|
||||
13:45:55 rc=0 locks=0 ids=
|
||||
13:45:59 rc=0 locks=0 ids=
|
||||
13:46:03 rc=0 locks=0 ids=
|
||||
13:46:07 rc=0 locks=0 ids=
|
||||
13:46:11 rc=0 locks=0 ids=
|
||||
13:46:15 rc=0 locks=0 ids=
|
||||
13:46:19 rc=0 locks=0 ids=
|
||||
13:46:23 rc=0 locks=0 ids=
|
||||
13:46:27 rc=0 locks=0 ids=
|
||||
13:46:31 rc=0 locks=0 ids=
|
||||
13:46:35 rc=0 locks=0 ids=
|
||||
13:46:39 rc=0 locks=0 ids=
|
||||
13:46:43 rc=0 locks=0 ids=
|
||||
13:46:47 rc=0 locks=0 ids=
|
||||
13:46:51 rc=0 locks=0 ids=
|
||||
13:46:54 rc=0 locks=0 ids=
|
||||
13:46:58 rc=0 locks=0 ids=
|
||||
13:47:02 rc=0 locks=0 ids=
|
||||
13:47:06 rc=0 locks=0 ids=
|
||||
13:47:10 rc=0 locks=0 ids=
|
||||
13:47:14 rc=0 locks=0 ids=
|
||||
13:47:18 rc=0 locks=0 ids=
|
||||
13:47:22 rc=0 locks=0 ids=
|
||||
13:47:26 rc=0 locks=0 ids=
|
||||
13:47:30 rc=0 locks=0 ids=
|
||||
13:47:34 rc=0 locks=0 ids=
|
||||
13:47:38 rc=0 locks=0 ids=
|
||||
13:47:42 rc=0 locks=0 ids=
|
||||
13:47:46 rc=0 locks=0 ids=
|
||||
13:47:50 rc=0 locks=0 ids=
|
||||
13:47:54 rc=0 locks=0 ids=
|
||||
13:47:58 rc=0 locks=0 ids=
|
||||
13:48:01 rc=0 locks=0 ids=
|
||||
13:48:05 rc=0 locks=0 ids=
|
||||
13:48:09 rc=0 locks=0 ids=
|
||||
13:48:13 rc=0 locks=0 ids=
|
||||
13:48:17 rc=0 locks=0 ids=
|
||||
13:48:21 rc=0 locks=0 ids=
|
||||
13:48:25 rc=0 locks=0 ids=
|
||||
13:48:29 rc=0 locks=0 ids=
|
||||
13:48:33 rc=0 locks=0 ids=
|
||||
13:48:37 rc=0 locks=0 ids=
|
||||
13:48:41 rc=0 locks=0 ids=
|
||||
13:48:45 rc=0 locks=0 ids=
|
||||
13:48:49 rc=0 locks=0 ids=
|
||||
13:48:53 rc=0 locks=0 ids=
|
||||
13:48:57 rc=0 locks=0 ids=
|
||||
13:49:01 rc=0 locks=0 ids=
|
||||
13:49:05 rc=0 locks=0 ids=
|
||||
13:49:09 rc=0 locks=0 ids=
|
||||
13:49:13 rc=0 locks=0 ids=
|
||||
13:49:17 rc=0 locks=0 ids=
|
||||
13:49:21 rc=0 locks=0 ids=
|
||||
13:49:25 rc=0 locks=0 ids=
|
||||
13:49:29 rc=0 locks=0 ids=
|
||||
13:49:33 rc=0 locks=0 ids=
|
||||
13:49:36 rc=0 locks=0 ids=
|
||||
13:49:40 rc=0 locks=0 ids=
|
||||
13:49:44 rc=0 locks=0 ids=
|
||||
13:49:48 rc=0 locks=0 ids=
|
||||
13:49:52 rc=0 locks=0 ids=
|
||||
13:49:56 rc=0 locks=0 ids=
|
||||
13:50:00 rc=0 locks=0 ids=
|
||||
13:50:04 rc=0 locks=0 ids=
|
||||
13:50:08 rc=0 locks=0 ids=
|
||||
13:50:12 rc=0 locks=0 ids=
|
||||
13:50:16 rc=0 locks=0 ids=
|
||||
13:50:20 rc=0 locks=0 ids=
|
||||
13:50:24 rc=0 locks=0 ids=
|
||||
13:50:28 rc=0 locks=0 ids=
|
||||
13:50:32 rc=0 locks=0 ids=
|
||||
13:50:36 rc=0 locks=0 ids=
|
||||
13:50:40 rc=0 locks=0 ids=
|
||||
13:50:43 rc=0 locks=0 ids=
|
||||
13:50:47 rc=0 locks=0 ids=
|
||||
13:50:51 rc=0 locks=0 ids=
|
||||
13:50:55 rc=0 locks=0 ids=
|
||||
13:50:59 rc=0 locks=0 ids=
|
||||
13:51:03 rc=0 locks=0 ids=
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
c3f1b55c {"total_size":83473654,"total_file_count":29,"snapshots_count":1}
|
||||
|
||||
ad2ae49c {"total_size":5808703,"total_file_count":38,"snapshots_count":1}
|
||||
|
||||
e7dcf8fd {"total_size":213231242,"total_file_count":16,"snapshots_count":1}
|
||||
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
13:42:54
|
||||
t0=2026-08-31T13:42:54.542010807Z
|
||||
post_http=302 post_time=0.009733
|
||||
2026/08/31 13:42:11 offbox_integrity.go:306: [INFO] [offbox] integrity: check PASSED in 40s (structure, index, and 100% of the pack data re-read)
|
||||
2026/08/31 13:42:54 auth.go:134: [DEBUG] [web] auth: valid session for POST /backup/offbox/restore
|
||||
2026/08/31 13:42:54 server.go:393: [DEBUG] [web] ServeHTTP: POST /backup/offbox/restore from 172.18.0.4:34642
|
||||
2026/08/31 13:43:05 offbox_restore.go:298: [INFO] [offbox] restored kimai (e7dcf8fd, full=true) → /mnt/felhom-drives/hdd_1/backups/offsite-restore/kimai
|
||||
2026/08/31 13:43:05 offbox_handlers.go:372: [INFO] [web] off-box restore kimai completed (full=true, async)
|
||||
---
|
||||
213231328 /mnt/felhom-drives/hdd_1/backups/offsite-restore/kimai
|
||||
@@ -0,0 +1,58 @@
|
||||
13:44:39 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:40 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:40 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:40 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:41 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:42 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:42 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:42 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:42 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:44 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:44 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
13:44:44 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
13:44:45 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
13:44:45 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
13:44:46 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
13:44:46 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:46 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:46 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:47 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:47 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:47 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:47 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:47 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:48 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:48 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:48 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:48 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:48 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:49 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:51 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:51 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:51 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:52 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:52 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:55 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:55 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:55 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:56 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:56 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:58 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:58 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:59 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:59 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:45:00 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:45:00 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:45:02 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:45:02 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:45:03 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:45:03 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:45:04 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:45:04 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:45:06 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
@@ -0,0 +1,48 @@
|
||||
t0=2026-08-31T13:44:41.569463100Z
|
||||
post_http=302 post_time=0.010061
|
||||
=== distinct restic argv observed (timestamps stripped) ===
|
||||
restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
=== first+last timestamp per verb ===
|
||||
13:44:39 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:40 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:40 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:40 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:41 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:42 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:42 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:42 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:42 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> snapshots latest --tag opengist --json
|
||||
13:44:43 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:44 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:44 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
13:44:44 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
13:44:45 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
13:44:45 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
13:44:46 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> unlock
|
||||
13:44:46 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:46 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:46 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:47 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:47 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:47 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:47 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:47 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:48 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:48 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:48 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:48 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:48 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:49 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> restore b5aa8f9b --target /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist --include /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
13:44:51 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:51 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:51 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:52 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
13:44:52 restic -r sftp:<REPO-REDACTED> -o sftp.command=<REDACTED> list locks --no-lock
|
||||
@@ -0,0 +1,11 @@
|
||||
=== A: restore WITHOUT --verify (kimai e7dcf8fd, 213 MB) ===
|
||||
sh: 7: bc: not found
|
||||
rc=0 wall=s bytes=213272202
|
||||
restoring <Snapshot e7dcf8fd of [/mnt/sys_drive/felhom-data/backups/primary/kimai] at 2026-08-31 02:16:30.041113131 +0000 UTC by root@demo-hp> to /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/a
|
||||
|
||||
=== B: restore WITH --verify, same snapshot ===
|
||||
sh: 12: bc: not found
|
||||
rc=0 wall=s bytes=213272202
|
||||
restoring <Snapshot e7dcf8fd of [/mnt/sys_drive/felhom-data/backups/primary/kimai] at 2026-08-31 02:16:30.041113131 +0000 UTC by root@demo-hp> to /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b
|
||||
verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b
|
||||
finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b (took 132ms)
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
A no-verify rc=0 wall_ms=2990 bytes=213272202
|
||||
B with-verify rc=0 wall_ms=4756 bytes=213272202
|
||||
verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b
|
||||
finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b (took 126ms)
|
||||
|
||||
=== RED-PROOF: corrupt ONE restored byte, keep size AND mtime, re-run restore --verify ===
|
||||
victim=kimai_kimai_db_data.tar size=160331776
|
||||
after edit: size=160331776 (must be unchanged) sha=dd962a1097077e39 orig-sha=81ce628f45418d93
|
||||
re-run rc=0
|
||||
restoring <Snapshot e7dcf8fd of [/mnt/sys_drive/felhom-data/backups/primary/kimai] at 2026-08-31 02:16:30.041113131 +0000 UTC by root@demo-hp> to /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b
|
||||
verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b
|
||||
finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b (took 125ms)
|
||||
|
||||
=== NEGATIVE CONTROL: same command on the UNTOUCHED copy /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/a ===
|
||||
rc=0
|
||||
restoring <Snapshot e7dcf8fd of [/mnt/sys_drive/felhom-data/backups/primary/kimai] at 2026-08-31 02:16:30.041113131 +0000 UTC by root@demo-hp> to /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/a
|
||||
verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/a
|
||||
finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/a (took 132ms)
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
=== CASE 1: SIZE changed (truncate 1 byte) ===
|
||||
size=160331775
|
||||
rc=0
|
||||
verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b
|
||||
finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b (took 133ms)
|
||||
size after=160331776 sha=81ce628f45418d93
|
||||
|
||||
=== CASE 2: content corrupt + mtime BUMPED ===
|
||||
sha before rerun=51ae3c7762768fe4
|
||||
rc=0
|
||||
verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b
|
||||
finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/b (took 128ms)
|
||||
sha after rerun =81ce628f45418d93 orig=81ce628f45418d93
|
||||
|
||||
=== CASE 3: content corrupt, size+mtime PRESERVED, restore into a VIRGIN target ===
|
||||
(this is the shape an unattended test would meet: a fresh restore, then a check)
|
||||
virgin restore+verify rc=0 wall_ms=3438
|
||||
verifying files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/g
|
||||
finished verifying 7 files in /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/g (took 131ms)
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
=== does the recovery unit manifest carry hashes? ===
|
||||
{
|
||||
"schema_version": 2,
|
||||
"app_name": "kimai",
|
||||
"display_name": "Kimai",
|
||||
"controller_version": "0.227.1",
|
||||
"created_at": "2026-08-31T02:16:24Z",
|
||||
"drive": "/mnt/sys_drive",
|
||||
"namespace_root": "/mnt/sys_drive/felhom-data",
|
||||
"image_pins": [
|
||||
"kimai/kimai2:apache-2.57.0",
|
||||
"mariadb:11.6"
|
||||
],
|
||||
"secret_env_vars": [
|
||||
"DB_PASSWORD",
|
||||
"ADMIN_PASSWORD"
|
||||
],
|
||||
"data_key_env_vars": null,
|
||||
"secret_source": "portable secrets (data keys, DB passwords, internal signing secrets) are IN this unit's compose/app.yaml (0600); internet-reachable admin logins are NOT, and come from the guest's app.yaml or are regenerated on restore",
|
||||
"config_files": [
|
||||
"docker-compose.yml",
|
||||
".felhom.yml",
|
||||
"app.yaml"
|
||||
],
|
||||
"db_dumps": [
|
||||
"kimai-mariadb.sql"
|
||||
],
|
||||
"volume_dumps": [
|
||||
"kimai_kimai_db_data.tar",
|
||||
"kimai_kimai_var.tar"
|
||||
],
|
||||
"checksums": {
|
||||
".felhom.yml": "95beb458a42391af4bf78429decfbc60537d8d53d61bea68c38687bcc7fa0d23",
|
||||
"app.yaml": "9192dec10e5aae76c2787d0e82ad39115d43f7011519f854bc06534f930c3e9c",
|
||||
"docker-compose.yml": "c25029f8bd16e6252dceda0b4e87917a8b9ce0b9ad6bdbfc82fa8d4f630e6310"
|
||||
},
|
||||
"portable_secret_env_vars": [
|
||||
"DB_PASSWORD"
|
||||
],
|
||||
"o
|
||||
|
||||
=== does restic ls --json expose per-file content hashes? ===
|
||||
{"time":"2026-08-31T02:16:30.041113131Z","parent":"84542ec89b96fd05d504ca33e5f391a959c2e42e94353ca53e2e46929791d576","tree":"00bbec71aced8301be6100c4ac5de6142defbbf0d24b44898a5dfff059e9969f","paths":["/mnt/sys_drive/felhom-data/backups/primary/kimai"],"hostname":"demo-hp","username":"root","tags":["felhom-offbox","kimai"],"id":"e7dcf8fde91a54f655175b60da2e45cfa7b907be78bc9ac5ead5601f3eaf077d","sho
|
||||
{"name":"mnt","type":"dir","path":"/mnt","uid":0,"gid":0,"mode":2147484141,"permissions":"drwxr-xr-x","mtime":"2026-08-21T16:00:39.422914481Z","atime":"2026-08-21T16:00:39.422914481Z","ctime":"2026-08-21T16:00:39.422914481Z","struct_type":"node"}
|
||||
{"name":"sys_drive","type":"dir","path":"/mnt/sys_drive","uid":0,"gid":0,"mode":2147484141,"permissions":"drwxr-xr-x","mtime":"2026-08-21T16:01:03.794262171Z","atime":"2026-08-21T16:01:03.794262171Z","ctime":"2026-08-21T16:01:03.794262171Z","struct_type":"node"}
|
||||
{"name":"felhom-data","type":"dir","path":"/mnt/sys_drive/felhom-data","uid":0,"gid":0,"mode":2147484141,"permissions":"drwxr-xr-x","mtime":"2026-08-21T16:31:03.891800035Z","atime":"2026-08-21T16:31:03.891800035Z","ctime":"2026-08-21T16:31:03.891800035Z","struct_type":"node"}
|
||||
@@ -0,0 +1,18 @@
|
||||
=== a FILE node from restic ls --json ===
|
||||
{"name":".felhom.yml","type":"file","path":"/mnt/sys_drive/felhom-data/backups/primary/kimai/compose/.felhom.yml","uid":0,"gid":0,"size":2235,"mode":420,"permissions":"-rw-r--r--","mtime":"2026-08-31T02:16:24.917098021Z","atime":"2026-08-31T02:16:24.917098021Z","ctime":"2026-08-31T02:16:24.918098034Z","struct_type":"node"}
|
||||
{"name":"app.yaml","type":"file","path":"/mnt/sys_drive/felhom-data/backups/primary/kimai/compose/app.yaml","uid":0,"gid":0,"size":488,"mode":384,"permissions":"-rw-------","mtime":"2026-08-31T02:16:24.918098034Z","atime":"2026-08-31T02:16:24.918098034Z","ctime":"2026-08-31T02:16:24.918098034Z","struct_type":"node"}
|
||||
|
||||
=== the full checksums block of the kimai manifest ===
|
||||
"checksums": {
|
||||
".felhom.yml": "95beb458a42391af4bf78429decfbc60537d8d53d61bea68c38687bcc7fa0d23",
|
||||
"app.yaml": "9192dec10e5aae76c2787d0e82ad39115d43f7011519f854bc06534f930c3e9c",
|
||||
"docker-compose.yml": "c25029f8bd16e6252dceda0b4e87917a8b9ce0b9ad6bdbfc82fa8d4f630e6310"
|
||||
},
|
||||
=== files actually in the unit ===
|
||||
48217 /mnt/sys_drive/felhom-data/backups/primary/kimai/db-dumps/kimai-mariadb.sql
|
||||
2235 /mnt/sys_drive/felhom-data/backups/primary/kimai/compose/.felhom.yml
|
||||
2195 /mnt/sys_drive/felhom-data/backups/primary/kimai/compose/docker-compose.yml
|
||||
488 /mnt/sys_drive/felhom-data/backups/primary/kimai/compose/app.yaml
|
||||
160331776 /mnt/sys_drive/felhom-data/backups/primary/kimai/volume-dumps/kimai_kimai_db_data.tar
|
||||
52845056 /mnt/sys_drive/felhom-data/backups/primary/kimai/volume-dumps/kimai_kimai_var.tar
|
||||
1275 /mnt/sys_drive/felhom-data/backups/primary/kimai/manifest.json
|
||||
@@ -0,0 +1,10 @@
|
||||
=== restic local cache present? ===
|
||||
1.1M /root/.cache/restic
|
||||
|
||||
=== kimai 213 MB: cached vs --no-cache (the WAN-truth number) ===
|
||||
cached rc=0 ms=3198
|
||||
--no-cache rc=0 ms=5423
|
||||
identical trees: YES
|
||||
|
||||
=== peak scratch bytes for the biggest app ===
|
||||
213272202 /mnt/felhom-drives/hdd_1/backups/offsite-restore/.spike-probe/i
|
||||
@@ -0,0 +1,9 @@
|
||||
privatebin snap=bb81b9da logical= 2123006 restored= 2159870 ms= 2628 rc=0
|
||||
opengist snap=b5aa8f9b logical= 185664 restored= 222528 ms= 2253 rc=0
|
||||
calibre-web snap=ad2ae49c logical= 5808703 restored= 5878335 ms= 3642 rc=0
|
||||
paperless-ngx snap=c3f1b55c logical= 83473654 restored= 83547382 ms= 3062 rc=0
|
||||
bookstack snap=91154be7 logical= 166641768 restored= 166682728 ms= 2745 rc=0
|
||||
docmost snap=a07c36a1 logical= 118207270 restored= 118248230 ms= 3676 rc=0
|
||||
romm snap=df7ad253 logical= 184706816 restored= 184747776 ms= 3978 rc=0
|
||||
kimai snap=e7dcf8fd logical= 213231242 restored= 213272202 ms= 3202 rc=0
|
||||
ALL EIGHT, back to back: 25s
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
=== every acquireRunning caller (non-test) ===
|
||||
internal/backup/offbox_integrity.go:272: if err := m.acquireRunning(); err != nil {
|
||||
internal/backup/offbox_restore.go:549: if err := m.acquireRunning(); err != nil {
|
||||
internal/backup/offbox_reconstitute.go:491: if err := m.acquireRunning(); err != nil {
|
||||
internal/backup/backup.go:473: if err := m.acquireRunning(); err != nil {
|
||||
internal/backup/backup.go:935:func (m *Manager) AcquireRunningForTest() error { return m.acquireRunning() }
|
||||
internal/backup/backup.go:939:func (m *Manager) acquireRunning() error {
|
||||
internal/backup/shares_restore.go:203: if err := m.acquireRunning(); err != nil {
|
||||
internal/backup/offbox.go:864: if err := m.acquireRunning(); err != nil {
|
||||
internal/backup/tier2_restore.go:183:// do NOT add an acquireRunning() here. A second acquire would refuse the restore it is guarding.
|
||||
internal/backup/tier2_restore.go:347: if err := m.acquireRunning(); err != nil {
|
||||
|
||||
=== does RestoreOffboxScratch acquire it? ===
|
||||
NO acquireRunning in RestoreOffboxScratch
|
||||
|
||||
=== restore_wizard.go:170-190 ===
|
||||
// restoreOpInFlight reports whether a restore op is in flight, FOR DISPLAY.
|
||||
//
|
||||
// **Use this, not `Manager.IsRunning()`.** The Manager carries two different booleans and they are
|
||||
// not interchangeable:
|
||||
//
|
||||
// - `m.running` (read by `IsRunning`) is the CONCURRENCY single-flight. It is acquired *inside*
|
||||
// the restore function, on the background goroutine — and `RestoreOffboxScratch` never acquires
|
||||
// it at all. So for the verification restore and the full-restore preparation — the wizard's two
|
||||
// most-used actions, and the long ones, since they stream from restic — `IsRunning()` is false
|
||||
// for the entire operation.
|
||||
// - `m.opRunning` (read by `RestoreStatus`) is the DISPLAY flag, set synchronously by
|
||||
// `BeginRestoreOp` in the handler *before* the goroutine launches and cleared by `EndRestoreOp`.
|
||||
// It covers all four offsite actions with no start-up window.
|
||||
//
|
||||
// v0.154.0 shipped with `IsRunning()` here, which made the execution step unreachable for
|
||||
// `RestoreOffboxScratch`: the page offered all three intents, with live buttons, while a restore was
|
||||
// downloading — and the progress banner (which polls the op status) contradicted it on the same
|
||||
// screen. Caught by the operator on the first live click-through.
|
||||
func restoreOpInFlight(st backup.RestoreOpStatus) bool {
|
||||
return st.Running
|
||||
}
|
||||
|
||||
// restoreOpBlocked reports whether a NEW restore must be refused right now, and returns the
|
||||
// Hungarian refusal to show. It reads BOTH flags, deliberately:
|
||||
//
|
||||
@@ -0,0 +1,111 @@
|
||||
=== scheduled off-site jobs (main.go) ===
|
||||
663: sched.Every("status-refresh", 10*time.Second, func(ctx context.Context) error {
|
||||
666: sched.Every("stack-scan", 2*time.Minute, func(ctx context.Context) error {
|
||||
669: sched.Every("health-probes", 10*time.Second, func(ctx context.Context) error {
|
||||
679: sched.Every("system-health", healthInterval, func(ctx context.Context) error {
|
||||
726: sched.Every("deadapp-check", 30*time.Second, func(ctx context.Context) error {
|
||||
742: sched.Every("ring-spill", 30*time.Second, func(ctx context.Context) error {
|
||||
876: sched.Daily("db-dump", dbLeg, func(ctx context.Context) error {
|
||||
914: sched.Every("offsite-credential-retry", 5*time.Minute, func(ctx context.Context) error {
|
||||
937: sched.Every("backup-cache", 5*time.Minute, func(ctx context.Context) error {
|
||||
962: sched.Daily("tier2-backup", tier2Leg, func(ctx context.Context) error {
|
||||
1095: sched.Daily("offbox-backup", offboxLeg, func(ctx context.Context) error {
|
||||
1109: sched.Daily("offsite-abandon-sweep", "05:10", func(ctx context.Context) error {
|
||||
1139: sched.Daily("offsite-integrity", "06:00", func(ctx context.Context) error {
|
||||
1147: sched.Daily("metrics-prune", "04:00", func(ctx context.Context) error {
|
||||
1198: sched.Daily("fill-watch", "03:30", func(ctx context.Context) error { return fillWatcher.Check() })
|
||||
1228: sched.Every("hub-report", pushInterval, func(ctx context.Context) error {
|
||||
1256: sched.Every("selfupdate-check", checkInterval, func(ctx context.Context) error {
|
||||
1266: sched.Daily("selfupdate-auto", cfg.SelfUpdate.AutoUpdateTime, func(ctx context.Context) error {
|
||||
1293: sched.Daily("asset-sync", cfg.Assets.SyncSchedule, func(ctx context.Context) error {
|
||||
1464: sched.Every("geo-verify", 6*time.Hour, func(ctx context.Context) error {
|
||||
sync_schedule: "05:00"
|
||||
backup:
|
||||
db_dump_schedule: "02:30"
|
||||
enabled: true
|
||||
prune_schedule: weekly
|
||||
restic_password_file: /opt/docker/felhom-controller/data/restic-password
|
||||
restic_schedule: "03:00"
|
||||
retention:
|
||||
keep_daily: 7
|
||||
keep_monthly: 6
|
||||
keep_weekly: 4
|
||||
customer:
|
||||
domain: enkisfelhom.hu
|
||||
email: doodoo21@freemail.hu
|
||||
id: demo-hp
|
||||
name: Demo HP
|
||||
telegram_chat_id: ""
|
||||
git:
|
||||
branch: main
|
||||
--
|
||||
health_check_schedule: "06:00"
|
||||
healthchecks_base: https://status.felhom.eu
|
||||
system_health_interval: 5m
|
||||
thresholds:
|
||||
backup_max_age_hours: 36
|
||||
cpu_warn_percent: 90
|
||||
disk_crit_percent: 90
|
||||
disk_warn_percent: 80
|
||||
memory_warn_percent: 85
|
||||
temperature_warn_celsius: 75
|
||||
internal/backupwindow/backupwindow.go:54:func LegTimes(start string) (db, tier2, offbox string) {
|
||||
internal/backupwindow/backupwindow.go-55- m, err := ParseHHMM(start)
|
||||
internal/backupwindow/backupwindow.go-56- if err != nil {
|
||||
internal/backupwindow/backupwindow.go-57- return "", "", ""
|
||||
internal/backupwindow/backupwindow.go-58- }
|
||||
internal/backupwindow/backupwindow.go-59- return FmtHHMM(m), FmtHHMM(m + tier2OffsetMin), FmtHHMM(m + offboxOffsetMin)
|
||||
internal/backupwindow/backupwindow.go-60-}
|
||||
internal/backupwindow/backupwindow.go-61-
|
||||
internal/backupwindow/backupwindow.go-62-// GateWindow returns the whole-guest backup gate bounds [W+2h, W+6h) as HH:MM strings (for the UI
|
||||
internal/backupwindow/backupwindow.go-63-// "kb. <from>–<to> között" line and the gate-denial log). Empty strings on an invalid start.
|
||||
internal/backupwindow/backupwindow.go-64-func GateWindow(start string) (from, to string) {
|
||||
internal/backupwindow/backupwindow.go-65- m, err := ParseHHMM(start)
|
||||
internal/backupwindow/backupwindow.go-66- if err != nil {
|
||||
internal/backupwindow/backupwindow.go-67- return "", ""
|
||||
internal/backupwindow/backupwindow.go-68- }
|
||||
internal/backupwindow/backupwindow.go-69- return FmtHHMM(m + gateStartMin), FmtHHMM(m + gateEndMin)
|
||||
internal/backupwindow/backupwindow.go-70-}
|
||||
internal/backupwindow/backupwindow.go-71-
|
||||
internal/backupwindow/backupwindow.go-72-// EffectiveWindow resolves the active window by precedence: a valid settings value wins over a valid
|
||||
internal/backupwindow/backupwindow.go-73-// controller.yaml value, which wins over DefaultWindow. An empty or corrupted value simply falls
|
||||
internal/backupwindow/backupwindow.go-74-// through — so a bad settings string degrades to the yaml default rather than breaking scheduling.
|
||||
internal/backupwindow/backupwindow.go-75-func EffectiveWindow(settingsVal, yamlVal string) string {
|
||||
internal/backupwindow/backupwindow.go-76- if Valid(settingsVal) == nil {
|
||||
internal/backupwindow/backupwindow.go-77- return settingsVal
|
||||
internal/backupwindow/backupwindow.go-78- }
|
||||
internal/backupwindow/backupwindow.go-79- if Valid(yamlVal) == nil {
|
||||
internal/backupwindow/backupwindow.go-80- return yamlVal
|
||||
internal/backupwindow/backupwindow.go-81- }
|
||||
internal/backupwindow/backupwindow.go-82- return DefaultWindow
|
||||
internal/backupwindow/backupwindow.go-83-}
|
||||
10:// DefaultWindow is the last-resort window when neither settings nor controller.yaml supplies one.
|
||||
12:const DefaultWindow = "02:30"
|
||||
17: tier2OffsetMin = 60 // tier-2 mirror at W+60m
|
||||
18: offboxOffsetMin = 105 // off-box copy at W+105m
|
||||
2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: status-refresh (every 10s)
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="status-refresh" interval=10s totalJobs=1
|
||||
2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s)
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="stack-scan" interval=2m0s totalJobs=2
|
||||
2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: health-probes (every 10s)
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="health-probes" interval=10s totalJobs=3
|
||||
2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: system-health (every 5m0s)
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="system-health" interval=5m0s totalJobs=4
|
||||
2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s)
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="deadapp-check" interval=30s totalJobs=5
|
||||
2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: ring-spill (every 30s)
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="ring-spill" interval=30s totalJobs=6
|
||||
2026/08/31 12:33:29 scheduler.go:132: [INFO] [scheduler] Daily job db-dump scheduled for 2026-09-01 02:30 CEST
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="db-dump" schedule="02:30" nextRun=2026-09-01T02:30:00+02:00 totalJobs=7
|
||||
2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s)
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="offsite-credential-retry" interval=5m0s totalJobs=8
|
||||
2026/08/31 12:33:29 scheduler.go:102: [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s)
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="backup-cache" interval=5m0s totalJobs=9
|
||||
2026/08/31 12:33:29 scheduler.go:132: [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-09-01 03:30 CEST
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="tier2-backup" schedule="03:30" nextRun=2026-09-01T03:30:00+02:00 totalJobs=10
|
||||
2026/08/31 12:33:29 scheduler.go:132: [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-09-01 04:15 CEST
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offbox-backup" schedule="04:15" nextRun=2026-09-01T04:15:00+02:00 totalJobs=11
|
||||
2026/08/31 12:33:29 scheduler.go:132: [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-09-01 05:10 CEST
|
||||
2026/08/31 12:33:29 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offsite-abandon-sweep" schedule="05:10" nextRun=2026-09-01T05:10:00+02:00 totalJobs=12
|
||||
2026/08/31 12:33:29 scheduler.go:132: [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-09-01 06:00 CEST
|
||||
grep: write error: Broken pipe
|
||||
@@ -0,0 +1,39 @@
|
||||
=== LAYER 3: container /tmp ===
|
||||
/tmp: 1: Syntax error: Unterminated quoted string
|
||||
remaining spike files in container: []
|
||||
wc: invalid option -- '"'
|
||||
Try 'wc --help' for more information.
|
||||
restic processes still running:
|
||||
|
||||
=== LAYER 3b: the scratch dirs my probes created ===
|
||||
161146787 /mnt/felhom-drives/hdd_1/backups/offsite-restore/bookstack
|
||||
5808705 /mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web
|
||||
118761552 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost
|
||||
213231328 /mnt/felhom-drives/hdd_1/backups/offsite-restore/kimai
|
||||
185751 /mnt/felhom-drives/hdd_1/backups/offsite-restore/opengist
|
||||
72345368 /mnt/felhom-drives/hdd_1/backups/offsite-restore/paperless-ngx
|
||||
2123093 /mnt/felhom-drives/hdd_1/backups/offsite-restore/privatebin
|
||||
after: [bookstack
|
||||
calibre-web
|
||||
paperless-ngx]
|
||||
|
||||
=== LAYER 2: guest /root ===
|
||||
remaining in guest: []
|
||||
=== LAYER 1: PVE host /root ===
|
||||
ssh: Could not resolve hostname hp: Name or service not known
|
||||
=== LAYER 1: PVE host demo-hp /root ===
|
||||
host-leftovers-rc=1 (1 = none found = clean)
|
||||
|
||||
=== LAYER 3 re-verified: container ===
|
||||
container /tmp listed above
|
||||
|
||||
=== LAYER 2 re-verified: guest ===
|
||||
guest-leftovers-rc=1
|
||||
|
||||
=== the app still healthy after all this ===
|
||||
felhom-controller Up About an hour (healthy)
|
||||
docmost Up About an hour (healthy)
|
||||
docmost-redis Up About an hour (healthy)
|
||||
docmost-postgres Up About an hour (healthy)
|
||||
romm Up 4 hours (healthy)
|
||||
romm-db Up 4 hours (healthy)
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
last_integrity_check": "2026-08-31T13:41:28Z"
|
||||
last_integrity_ok": true
|
||||
last_integrity_depth": "100%"
|
||||
Reference in New Issue
Block a user