R-436 measured on the provider: append-only forced key HOLDS (403 on every delete), but the sub-account password defeats it (R-820); design proposal + ep0 options
gates / gates (push) Successful in 29s
gates / gates (push) Successful in 29s
Spike, no product change. Venue u629488-sub4 (tester-1, operator ruling); scratch repo removed, authorized_keys restored byte-identical. Closed R-436 (due-check cleared), R-430. Opened R-820, R-821, R-822. R-95 and R-342 updated. 07 §D [FACT] block. STATUS: two operator decisions. Register 326 -> 327. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -1367,6 +1367,19 @@ answered by calling the provider API with the production token. Accept explicitl
|
||||
> spike stopped here rather than answering it by acting**, and left it as R-429 — ten minutes in
|
||||
> the Storage Box panel, and it re-ranks R-95. `audits/SPIKE-r95-offsite-delete-2026-09-01.md` §Q1.
|
||||
|
||||
> **`[FACT]` 2026-10-03 — append-only on the Storage Box, MEASURED (R-436).** On the provider
|
||||
> (`u629488-sub4`, scratch repo, since removed), a key pinned in `authorized_keys` to
|
||||
> `command="rclone serve restic --stdio --append-only <dir>",restrict` backs up, lists, restores and
|
||||
> checks, and every delete is refused: `blob not removed, server response: 403 Forbidden (403)`; the
|
||||
> same command through an unpinned key deletes. The pinned key gets no shell, sftp, scp, rsync or port
|
||||
> forward. **But the PASSWORD logs in on ports 22 and 23 and can rewrite `authorized_keys`, and a box can
|
||||
> obtain that password from the hub's off-site self-heal at will** — so the pin protects nothing until
|
||||
> the box stops receiving the password (R-820). Port 22 accepts no OpenSSH-format key; 23 does.
|
||||
> Locks: a crash lock blocks `check`, not `backup`; `unlock --remove-all` clears it through the pinned
|
||||
> key. An add-only key can still plant future-dated snapshots that make the box's retention policy
|
||||
> select every real snapshot (R-822). Who prunes is a PROPOSAL awaiting the operator, not design:
|
||||
> `audits/offsite-append-only-2026-10-03/DESIGN.md`; evidence `…/live/`, `…/lab/`.
|
||||
|
||||
**E. `local` vzdump shares a physical device with the guest it backs up.** LIVE on both hosts:
|
||||
`/var/lib/vz` (the archive target) and `local-lvm` (the guest's rootfs and both `backup=1`
|
||||
mountpoints) are both on `/dev/sda3` → VG `pve`. The tier therefore protects against **corruption
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
# Off-site append-only — who deletes old backups when the box cannot? (PROPOSAL, 2026-10-03)
|
||||
|
||||
**Status: a PROPOSAL for the operator to rule on. Nothing here is built. Nothing here is `[DESIGN]`.**
|
||||
Evidence: `live/` (the provider, `u629488-sub4`) and `lab/` (OpenSSH + rclone v1.75.1 on DooPlex).
|
||||
Baselines: felhom.eu `f4c5466`, controller `0945332` (v0.288.0, restic **0.14.0** Debian `0.14.0-1+b5`).
|
||||
|
||||
## 1. What was measured (the facts this design stands on)
|
||||
|
||||
| # | Fact | Where |
|
||||
|---|---|---|
|
||||
| F1 | A key whose `authorized_keys` line is `command="rclone serve restic --stdio --append-only <dir>",restrict` can `init`, `backup`, `snapshots`, `restore` (bytes identical) and `check`. | `live/E1-E3`, `live/E4-E6`, `live/C2-A5` |
|
||||
| F2 | Through that key `forget <id> --prune`, `forget --keep-last 1` and a real `prune` are REFUSED: `blob not removed, server response: 403 Forbidden (403)`, rc=1, snapshot count unchanged. Each refusal costs **~45–48 s** of restic retries. A refused `prune` has already **written a new index** first (`check`: duplicate index, non-critical). | `live/E4-E6` |
|
||||
| F3 | Control: the same `forget` through a key WITHOUT the forced command deletes (1/1 files deleted). | `live/E4-E6` (C1) |
|
||||
| F4 | The client's requested path and flags are ignored: the pinned directory is served even when the client names another path, and the client never asked for `--append-only` (restic sends `serve restic --stdio --b2-hard-delete`). | `live/C2-A5`, `live/B2` |
|
||||
| F5 | The forced key cannot get a shell, `sftp`, `scp`, `rsync` or a port forward (`administratively prohibited`); a command such as `rm -rf <dir>` just starts the forced rclone. | `live/B2`, `live/B1-B2` |
|
||||
| F6 | Port **22 accepts no OpenSSH-format key at all** (both test keys refused there); keys work on port **23** only — the port the product uses (`hub/internal/offsite/offsite.go` `sftpPort = 23`). **The PASSWORD logs in on BOTH ports**, and through it `.ssh/authorized_keys` was read and REWRITTEN (that is how the test keys were installed). | `live/B1-B2`, this session |
|
||||
| F7 | Crash locks: a killed backup leaves a lock. The next **backup is not blocked**; an **exclusive** operation (`check` — the weekly integrity job) **is**. Plain `unlock` prints `successfully removed locks` and removes nothing (the lock is not yet stale — by design, R-430 reproduced); `unlock --remove-all` **does remove it** — the append-only server allows lock deletion. | `live/C2-A5`, `lab/A5` |
|
||||
| F8 | A crashed upload leaves unreferenced packs (lab: 17) that only a deleting key can clean. | `lab/A5` |
|
||||
| F9 | The append-only server refuses delete/overwrite of existing files (403) and path escapes (`../`, `%2e%2e`, `..%2f` → 400), but ACCEPTS new files of any name inside the repo, including a new `keys/` entry (quota can be filled). | `lab/B-rclone-server-surface` (lab rclone; the provider's rclone version cannot be read — `rclone version` is not a shell command there) |
|
||||
| F10 | **Retention poisoning.** Through the add-only key, 13 future-dated empty snapshots with the same host+tag make the box's exact policy (`--group-by host,tags --keep-daily 7 --keep-weekly 4 --keep-monthly 6`) select **all 3 real snapshots for removal**. | `lab/C3-retention-poisoning` |
|
||||
| F11 | The provider's rclone creates `~/.config/rclone/` (empty) in the sub-account home on first use. | `live/TEARDOWN` |
|
||||
|
||||
From the code (read, not run):
|
||||
|
||||
| # | Fact | Where |
|
||||
|---|---|---|
|
||||
| K1 | The box can obtain its sub-account password **at will**: it declares `needs_credential` in two reports, the hub's off-site self-heal re-arms the **stored** value (`RestageOneTimeSecret`; the value is never cleared after a consume) or escalates to a provider re-issue, and the box consumes it with its own API key. | `hub/internal/offsiteheal/reconciler.go`, `hub/internal/store/store.go` `ConsumeOneTimeSecret`/`RestageOneTimeSecret`, `controller/internal/offsiteapply/seams.go` |
|
||||
| K2 | The hub DB holds every sub-account password in the clear, indefinitely (`one_time_secrets.value`). | `hub/internal/store/store.go` |
|
||||
| K3 | Four box features need a normal (deleting) login, not just the two `forget` sites: `forget --prune` after a run (`offbox.go` ~1422), over quota (`offbox.go` ~1793), the orphan **move-aside** (`mv`, `offbox.go` `resetOrphanedRepo`), and the customer-chosen **abandonment** (`rm -rf`, `offbox_abandon.go` `AbandonSweep`). | controller `0945332` |
|
||||
| K4 | restic has no prune-only credential: every key unlocks the same master key, so whoever prunes can read every backup in that repo. | restic design (documented, not measured) |
|
||||
|
||||
## 2. The prerequisite every option shares — the lock is worthless until this is closed
|
||||
|
||||
**F6 + K1: a box that is broken into can get the password, log in on port 23 with it, and rewrite
|
||||
`authorized_keys` — removing the forced command from its own key.** So before any lock ships:
|
||||
|
||||
1. **The box never receives the sub-account password.** The hub becomes the key registrar: the box
|
||||
generates its key pair and sends the PUBLIC key to the hub; the hub (which already holds the
|
||||
password, K2) writes the forced line into `authorized_keys` over SFTP port 23. The off-site
|
||||
self-heal re-arms the HUB's install job, not a password for the box.
|
||||
2. **The hub checks the file.** Each day the hub reads `authorized_keys` and alarms if any line lacks
|
||||
the forced prefix (outside an open clean-up window). This is also the answer to "an operator adds a
|
||||
key later": a key without the prefix re-opens deletion, so a check — not a rule — catches it.
|
||||
3. Optional hardening, operator's call: rotate the sub-account password after each hub use and keep it
|
||||
only in the hub (K2 stays true either way: **the hub can already delete every household's off-site
|
||||
history today** — new row).
|
||||
|
||||
## 3. The options
|
||||
|
||||
### Option 1 — a clean-up window opened by the hub (the box keeps pruning its own repo)
|
||||
|
||||
The box normally holds only its add-only key. Once a week the hub adds a second, deleting key line for
|
||||
the same box key pair (or a second key the box holds) for ~15 minutes; the box runs its retention; the
|
||||
hub removes the line and checks the file.
|
||||
|
||||
- **Custody: unchanged.** The repository password never leaves the box (07 §8a stays true).
|
||||
- **What a compromised box can do:** delete during a window. Detection backstop: R-431 (a fall of more
|
||||
than half), plus a new per-window check — the hub compares the snapshot count before and after and
|
||||
alarms on a fall larger than the retention could cause.
|
||||
- **Poisoning (F10) must be guarded even here**, because an attacker who was on the box and left can
|
||||
plant future snapshots that the next honest window then obeys: before `forget`, refuse when any
|
||||
snapshot is dated in the future or newer than the newest the hub has seen reported; run `--dry-run`
|
||||
first and abort if it would remove more snapshots than the policy can remove in a week.
|
||||
- **Build cost:** hub: key registrar + window open/close + `authorized_keys` check (SFTP with the
|
||||
password it holds — no provider API, no main account). Controller: switch the transport from `sftp:`
|
||||
to `rclone:` with `-o rclone.program="ssh -p 23 … -i <key> … rclone"` (restic 0.14.0 is enough, F1;
|
||||
**rclone is NOT needed in the image**); move `forget --prune` (both sites, together — R-191) into
|
||||
the window; move the move-aside and the abandonment to the hub (K3).
|
||||
- **Money:** none.
|
||||
|
||||
### Option 2 — clean-up runs off the box
|
||||
|
||||
A Felhom-side worker prunes each repo with a deleting key.
|
||||
|
||||
- **Custody: CHANGES.** restic has no prune-only key (K4): the worker must hold every repository
|
||||
password in a form it can use unattended, and could read every household's backups. That changes a
|
||||
promise to the customer (07 §8a) — not CC's to change.
|
||||
- **Poisoning (F10):** same guard needed.
|
||||
- **Build cost:** a new always-on worker in the recovery path, its own credentials store, its own
|
||||
failure alarms — plus everything in §2.
|
||||
|
||||
### Option 3 — never delete; let the quota grow
|
||||
|
||||
- **Money:** small. The pool box (BX11, 1 TB) was €4.06/month when recorded (`SPIKE-ep0-storagebox-
|
||||
2026-07-09`; Hetzner raised prices in April 2026 — re-check) ≈ **€0.004 per GB per month**. A
|
||||
household adding 10 GB a year unpruned costs ≈ **€0.50 a year**. The demo boxes cannot calibrate
|
||||
this (0.5 GB and 0.0 GB used today), so 10 GB/year is an assumption, stated as one.
|
||||
- **The real cost is the quota (50–100 GB):** at the quota `offbox_fit.go` refuses new pushes and the
|
||||
over-quota `forget` (K3) would itself be refused, so the household's off-site copy STOPS. Crash
|
||||
leftovers (F8) and refused-prune index files (F2) also accumulate.
|
||||
- Good as an **interim**, not an end state.
|
||||
|
||||
## 4. Recommendation
|
||||
|
||||
**Option 1, with Option 3 as the interim while it is built.** Order:
|
||||
|
||||
1. §2 first (key registrar + the `authorized_keys` check) — without it nothing below protects anything.
|
||||
2. Switch every box to the forced key with **no** box-side retention (Option 3 interim). Quotas today
|
||||
are 50–100 GB against ≤0.5 GB used, so this costs nothing for months.
|
||||
3. Then the weekly window (Option 1) with the poisoning guard.
|
||||
|
||||
Why not Option 2: it trades a box-level risk for a custody change that lets one Felhom machine read
|
||||
every household's backups, and it adds an always-on service to the recovery path.
|
||||
|
||||
## 5. Migration, rotation, restore
|
||||
|
||||
- **Existing repos (both demo boxes, any household):** only the `authorized_keys` line and the
|
||||
box's transport change; the repository bytes are not touched. **Not measured:** reading a repo
|
||||
written over `sftp:` through `rclone:` — restic's on-disk layout is backend-independent, so it is
|
||||
expected to work; measure on the scratch account before any household. History is kept.
|
||||
- **Key rotation:** the hub writes the new forced line, the box switches, the hub removes the old line —
|
||||
the same registrar path.
|
||||
- **Restore** works through the add-only key (F1) — no second key is needed for the household.
|
||||
- **Locks:** the self-heal (`resticStep` → `unlock --remove-all`) keeps working under this transport
|
||||
(F7). R-430's fear does not apply to it.
|
||||
- **Failure speed:** with today's code a forced key makes each night's `forget --prune` fail after
|
||||
~45 s per refused file and grow the index (F2) — the two `forget` sites must leave the box in the
|
||||
same change that switches the key.
|
||||
@@ -0,0 +1,33 @@
|
||||
# R-436 exit test — written BEFORE any command ran (2026-10-03 ~11:10 CEST)
|
||||
|
||||
The lock HOLDS only if ALL of these are true on the LIVE Storage Box (`u629488`, scratch venue
|
||||
`u629488-sub4`, a dedicated repo path that is NOT `felhom-repo`):
|
||||
|
||||
| # | Through the FORCED key (`command="rclone serve restic --stdio --append-only <path>"`) | Must be |
|
||||
|---|---|---|
|
||||
| E1 | `restic init` (scratch repo) | succeeds |
|
||||
| E2 | `restic backup` of a small tree | succeeds, snapshot count +1 |
|
||||
| E3 | `restic snapshots`, `restic restore` of one file (bytes compared) | succeeds |
|
||||
| E4 | `restic forget <id> --prune` | REFUSED — verbatim error quoted; count unchanged |
|
||||
| E5 | `restic prune` | REFUSED — verbatim; `check` clean |
|
||||
| E6 | `restic forget --keep-last 1` | REFUSED — verbatim; count unchanged |
|
||||
| E7 | the same key asking for a shell / `sftp` / `scp` / `rsync` / a port forward | REFUSED |
|
||||
| E8 | the same key, client asks for a plain `rclone serve restic --stdio` (no flag) | still append-only |
|
||||
|
||||
Controls (without them a refusal means nothing):
|
||||
|
||||
| # | Control | Must be |
|
||||
|---|---|---|
|
||||
| C1 | the same `forget --prune` through a key WITHOUT the forced command | SUCCEEDS (the test can see a delete) |
|
||||
| C2 | the forced key, client names a DIFFERENT repo path | recorded as observed (expected: the pinned path is served regardless) |
|
||||
|
||||
Locks (R-430): kill a backup mid-run through the forced key; record whether the next backup
|
||||
wedges, and what `unlock` / `unlock --remove-all` report AND what remains on disk.
|
||||
|
||||
**Verdict rule:** E1–E8 as stated and C1 succeeding ⇒ the lock holds *for that key*. Part B then asks
|
||||
whether any OTHER credential reachable from a box defeats it; if one does, the lock alone is not
|
||||
protection, and that is reported first.
|
||||
|
||||
A LOCAL LAB (`lab/`) runs the same matrix against OpenSSH + rclone on DooPlex. It measures rclone's
|
||||
and restic's behaviour; it CANNOT stand in for the provider's sshd honouring `command=`. Only `live/`
|
||||
can close R-436.
|
||||
@@ -0,0 +1,29 @@
|
||||
# R-342 — the ep0 datastore safeguard: options (2026-10-03, read only, nothing changed on ep0)
|
||||
|
||||
**The gap:** ep0's server snapshot covers the 38 GB system disk, not `/mnt/pbs-datastore` (a separate
|
||||
100 GB Hetzner Cloud Volume). The hub's Offsite page read today: datastore `felhom-offsite`,
|
||||
**19.5 GB used of 97.9 GB (20 %)**. ep0 holds the only off-premises copy of the households'
|
||||
whole-box backups. Those backups are **client-side encrypted per customer** (`encryption-key` in each
|
||||
box's `storage.cfg`, 07 §8a) — a copy elsewhere holds ciphertext only.
|
||||
|
||||
## A correction first
|
||||
|
||||
**R-342's first candidate does not exist.** Hetzner Cloud has **no Volume snapshots**: server
|
||||
snapshots and backups exclude attached volumes, and volume snapshots have been an open feature request
|
||||
since 2019 (hetznercloud/csi-driver issue #79; simplebackups.com Hetzner note). Not measured on the
|
||||
panel; documented by third parties and consistent with R-342's own evidence file.
|
||||
|
||||
## The options
|
||||
|
||||
| | What | Money / month | Setup | What it protects |
|
||||
|---|---|---|---|---|
|
||||
| **A. PBS pull-sync to DooPlex's PBS** | A sync job on DooPlex's existing PBS pulls the `felhom-offsite` datastore from ep0 (read-only token on ep0, `DatastoreReader`), e.g. nightly | **€0** (DooPlex `/mnt/5_hdd` has 5.5 TB free; ~20 GB now) | ~1–2 h: a PBS remote + sync job + a read-only token; one route from DooPlex to ep0 :8007 must be chosen (the tunnel or the public address) | A lost/corrupted ep0 datastore, a bad ep0 procedure, and losing Hetzner entirely (a different provider and site). Ciphertext only — no custody change. **Cost:** a new job on DooPlex (Tier 2 — the operator's call, not CC's) |
|
||||
| **B. Written acceptance + a copy before each risky procedure** | Accept that the datastore is unprotected in normal running; any runbook step that can touch `/mnt/pbs-datastore` first copies it off (e.g. to DooPlex, `rsync -a` **without** `-H` — `-H` was measured running out of memory on a PBS chunk store here, and PBS uses no hard links) | €0 | ~10 min to write the rule; ~20–40 min per procedure | Only the procedure itself. Not disk loss, not the provider, not a mistake between procedures |
|
||||
| ~~Hetzner Volume snapshot~~ | does not exist | — | — | — |
|
||||
| A second Hetzner volume / server | another volume ≈ €0.057/GB/month (April 2026 price, third-party source) → 100 GB ≈ **€5.70** + a server to attach it to | €5–10 | hours | Disk loss only; same provider, same failure domain (07 §D) |
|
||||
|
||||
## Recommendation
|
||||
|
||||
**A.** It costs no money, protects against everything B protects against and more, and keeps the
|
||||
custody model because the data is already encrypted per customer. Its one real cost is a new job on
|
||||
DooPlex, which is why it is the operator's decision.
|
||||
@@ -0,0 +1,43 @@
|
||||
$ c.sh forced spike-repo init
|
||||
rclone: 2026/10/03 09:07:47 NOTICE: Config file "/home/sub/.config/rclone/rclone.conf" not found - using defaults
|
||||
created restic repository 54423dc079 at rclone:lab:spike-repo
|
||||
|
||||
Please note that knowledge of your password is required to access
|
||||
the repository. Losing your password means that your data is
|
||||
irrecoverably lost.
|
||||
[rc=0]
|
||||
|
||||
$ c.sh forced spike-repo backup /d/tree --host labbox --tag app1
|
||||
rclone: 2026/10/03 09:07:51 NOTICE: Config file "/home/sub/.config/rclone/rclone.conf" not found - using defaults
|
||||
no parent snapshot found, will read all files
|
||||
|
||||
Files: 4 new, 0 changed, 0 unmodified
|
||||
Dirs: 3 new, 0 changed, 0 unmodified
|
||||
Added to the repository: 588.177 KiB (587.567 KiB stored)
|
||||
|
||||
processed 4 files, 585.948 KiB in 0:00
|
||||
snapshot 63a075dc saved
|
||||
[rc=0]
|
||||
|
||||
$ c.sh forced spike-repo backup /d/tree --host labbox --tag app1
|
||||
rclone: 2026/10/03 09:07:53 NOTICE: Config file "/home/sub/.config/rclone/rclone.conf" not found - using defaults
|
||||
using parent snapshot 63a075dc
|
||||
|
||||
Files: 0 new, 0 changed, 4 unmodified
|
||||
Dirs: 0 new, 0 changed, 3 unmodified
|
||||
Added to the repository: 0 B (0 B stored)
|
||||
|
||||
processed 4 files, 585.948 KiB in 0:00
|
||||
snapshot 066a039d saved
|
||||
[rc=0]
|
||||
|
||||
$ c.sh forced spike-repo snapshots
|
||||
rclone: 2026/10/03 09:07:55 NOTICE: Config file "/home/sub/.config/rclone/rclone.conf" not found - using defaults
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
63a075dc 2026-10-03 09:07:50 labbox app1 /d/tree
|
||||
066a039d 2026-10-03 09:07:52 labbox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
2 snapshots
|
||||
[rc=0]
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
server files before: snapshots=2 keys=1 index=1 data=2 locks=0
|
||||
$ c.sh forced spike-repo restore 63a075dc --target /d/restored --include /d/tree/sub/note.txt
|
||||
restoring <Snapshot 63a075dc of [/d/tree] at 2026-10-03 09:07:50.656818298 +0000 UTC by root@labbox> to /d/restored
|
||||
[rc=0]
|
||||
|
||||
restored bytes:
|
||||
$ c.sh forced spike-repo forget 63a075dc --prune
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 720.254544ms: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 873.42004ms: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 1.054928461s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 1.560325776s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 3.004145903s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 2.147653057s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 3.739082318s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 5.099891944s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 10.263247495s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 19.514091959s: blob not removed, server response: 403 Forbidden (403)
|
||||
unable to remove <snapshot/63a075dc45> from the repository
|
||||
[0:48] 0.00% 0 / 1 files deleted
|
||||
|
||||
blob not removed, server response: 403 Forbidden (403)
|
||||
github.com/restic/restic/internal/backend/rest.(*Backend).Remove
|
||||
github.com/restic/restic/internal/backend/rest/rest.go:399
|
||||
github.com/restic/restic/internal/backend.(*RetryBackend).Remove.func1
|
||||
github.com/restic/restic/internal/backend/backend_retry.go:108
|
||||
github.com/cenkalti/backoff.RetryNotifyWithTimer
|
||||
github.com/cenkalti/backoff/retry.go:55
|
||||
github.com/cenkalti/backoff.RetryNotify
|
||||
github.com/cenkalti/backoff/retry.go:34
|
||||
github.com/restic/restic/internal/backend.(*RetryBackend).retry
|
||||
github.com/restic/restic/internal/backend/backend_retry.go:46
|
||||
github.com/restic/restic/internal/backend.(*RetryBackend).Remove
|
||||
github.com/restic/restic/internal/backend/backend_retry.go:107
|
||||
github.com/restic/restic/internal/cache.(*Backend).Remove
|
||||
github.com/restic/restic/internal/cache/backend.go:38
|
||||
main.deleteFiles.func2
|
||||
github.com/restic/restic/cmd/restic/delete.go:47
|
||||
golang.org/x/sync/errgroup.(*Group).Go.func1
|
||||
golang.org/x/sync/errgroup/errgroup.go:75
|
||||
runtime.goexit
|
||||
runtime/asm_amd64.s:1594
|
||||
[rc=1]
|
||||
|
||||
$ c.sh forced spike-repo prune
|
||||
loading indexes...
|
||||
loading all snapshots...
|
||||
finding data that is still in use for 2 snapshots
|
||||
[0:00] 100.00% 2 / 2 snapshots
|
||||
|
||||
searching used packs...
|
||||
collecting packs for deletion and repacking
|
||||
[0:00] 100.00% 2 / 2 packs processed
|
||||
|
||||
|
||||
to repack: 0 blobs / 0 B
|
||||
this removes: 0 blobs / 0 B
|
||||
to delete: 0 blobs / 0 B
|
||||
total prune: 0 blobs / 0 B
|
||||
remaining: 8 blobs / 587.247 KiB
|
||||
unused size after prune: 0 B (0.00% of remaining size)
|
||||
|
||||
done
|
||||
[rc=0]
|
||||
|
||||
$ c.sh forced spike-repo forget --keep-last 1
|
||||
Applying Policy: keep 1 latest snapshots
|
||||
keep 1 snapshots:
|
||||
ID Time Host Tags Reasons Paths
|
||||
-----------------------------------------------------------------------------
|
||||
066a039d 2026-10-03 09:07:52 labbox app1 last snapshot /d/tree
|
||||
-----------------------------------------------------------------------------
|
||||
1 snapshots
|
||||
|
||||
remove 1 snapshots:
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
63a075dc 2026-10-03 09:07:50 labbox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
1 snapshots
|
||||
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 720.254544ms: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 873.42004ms: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 1.054928461s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 1.560325776s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 3.004145903s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 2.147653057s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 3.739082318s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 5.099891944s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 10.263247495s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<snapshot/63a075dc45>) returned error, retrying after 19.514091959s: blob not removed, server response: 403 Forbidden (403)
|
||||
unable to remove <snapshot/63a075dc45> from the repository
|
||||
[0:48] 0.00% 0 / 1 files deleted
|
||||
|
||||
blob not removed, server response: 403 Forbidden (403)
|
||||
github.com/restic/restic/internal/backend/rest.(*Backend).Remove
|
||||
github.com/restic/restic/internal/backend/rest/rest.go:399
|
||||
github.com/restic/restic/internal/backend.(*RetryBackend).Remove.func1
|
||||
github.com/restic/restic/internal/backend/backend_retry.go:108
|
||||
github.com/cenkalti/backoff.RetryNotifyWithTimer
|
||||
github.com/cenkalti/backoff/retry.go:55
|
||||
github.com/cenkalti/backoff.RetryNotify
|
||||
github.com/cenkalti/backoff/retry.go:34
|
||||
github.com/restic/restic/internal/backend.(*RetryBackend).retry
|
||||
github.com/restic/restic/internal/backend/backend_retry.go:46
|
||||
github.com/restic/restic/internal/backend.(*RetryBackend).Remove
|
||||
github.com/restic/restic/internal/backend/backend_retry.go:107
|
||||
github.com/restic/restic/internal/cache.(*Backend).Remove
|
||||
github.com/restic/restic/internal/cache/backend.go:38
|
||||
main.deleteFiles.func2
|
||||
github.com/restic/restic/cmd/restic/delete.go:47
|
||||
golang.org/x/sync/errgroup.(*Group).Go.func1
|
||||
golang.org/x/sync/errgroup/errgroup.go:75
|
||||
runtime.goexit
|
||||
runtime/asm_amd64.s:1594
|
||||
[rc=1]
|
||||
|
||||
$ c.sh forced spike-repo snapshots
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
63a075dc 2026-10-03 09:07:50 labbox app1 /d/tree
|
||||
066a039d 2026-10-03 09:07:52 labbox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
2 snapshots
|
||||
[rc=0]
|
||||
|
||||
$ c.sh forced spike-repo check
|
||||
using temporary cache in /tmp/restic-check-cache-3313406781
|
||||
create exclusive lock for repository
|
||||
load indexes
|
||||
check all packs
|
||||
check snapshots, trees and blobs
|
||||
[0:00] 100.00% 2 / 2 snapshots
|
||||
|
||||
no errors were found
|
||||
[rc=0]
|
||||
|
||||
server files after: snapshots=2 keys=1 index=1 data=2 locks=0
|
||||
cmp restored vs source: IDENTICAL (hello-r436)
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
## E5 made real: unreferenced data exists, then prune through the FORCED key
|
||||
$ c.sh forced spike-repo backup /d/tree2 --host labbox --tag app2
|
||||
no parent snapshot found, will read all files
|
||||
|
||||
Files: 1 new, 0 changed, 0 unmodified
|
||||
Dirs: 2 new, 0 changed, 0 unmodified
|
||||
Added to the repository: 293.934 KiB (293.869 KiB stored)
|
||||
|
||||
processed 1 files, 292.969 KiB in 0:00
|
||||
snapshot 4137acee saved
|
||||
[rc=0]
|
||||
|
||||
app2 snapshot = 4137acee
|
||||
## C1 control: the PLAIN key (no forced command) removes a snapshot
|
||||
$ c.sh plain spike-repo forget 4137acee
|
||||
rclone: 2026/10/03 09:10:36 CRITICAL: Failed to create file system for "lab:spike-repo": didn't find section in config file ("lab")
|
||||
Fatal: unable to open repository at rclone:lab:spike-repo: error talking HTTP to rclone: Get "http://localhost/file-5577006791947779410": unexpected EOF
|
||||
[rc=1]
|
||||
|
||||
server files now: snapshots=3 keys=1 index=2 data=4 locks=0
|
||||
## E5: prune through the FORCED key must now try to delete a pack
|
||||
$ c.sh forced spike-repo prune
|
||||
loading indexes...
|
||||
loading all snapshots...
|
||||
finding data that is still in use for 3 snapshots
|
||||
[0:00] 100.00% 3 / 3 snapshots
|
||||
|
||||
searching used packs...
|
||||
collecting packs for deletion and repacking
|
||||
[0:00] 100.00% 4 / 4 packs processed
|
||||
|
||||
|
||||
to repack: 0 blobs / 0 B
|
||||
this removes: 0 blobs / 0 B
|
||||
to delete: 0 blobs / 0 B
|
||||
total prune: 0 blobs / 0 B
|
||||
remaining: 12 blobs / 880.956 KiB
|
||||
unused size after prune: 0 B (0.00% of remaining size)
|
||||
|
||||
done
|
||||
[rc=0]
|
||||
|
||||
server files after forced prune: snapshots=3 keys=1 index=2 data=4 locks=0
|
||||
## C1 control: prune through the PLAIN key
|
||||
$ c.sh plain spike-repo prune
|
||||
rclone: 2026/10/03 09:10:39 CRITICAL: Failed to create file system for "lab:spike-repo": didn't find section in config file ("lab")
|
||||
Fatal: unable to open repository at rclone:lab:spike-repo: error talking HTTP to rclone: Get "http://localhost/file-5577006791947779410": unexpected EOF
|
||||
[rc=1]
|
||||
|
||||
server files after plain prune: snapshots=3 keys=1 index=2 data=4 locks=0
|
||||
## C2: forced key, client names a DIFFERENT repo path (other-repo)
|
||||
$ c.sh forced other-repo snapshots
|
||||
ID Time Host Tags Paths
|
||||
---------------------------------------------------------------
|
||||
63a075dc 2026-10-03 09:07:50 labbox app1 /d/tree
|
||||
066a039d 2026-10-03 09:07:52 labbox app1 /d/tree
|
||||
4137acee 2026-10-03 09:10:32 labbox app2 /d/tree2
|
||||
---------------------------------------------------------------
|
||||
3 snapshots
|
||||
[rc=0]
|
||||
|
||||
$ c.sh forced other-repo init
|
||||
Fatal: create repository at rclone:lab:other-repo failed: Fatal: config file already exists
|
||||
|
||||
[rc=1]
|
||||
|
||||
spike-repo
|
||||
+105
@@ -0,0 +1,105 @@
|
||||
## (re-run; the first attempt named an rclone remote 'lab:' the PLAIN server has no config for — the plain key runs rclone with the CLIENT's path, which is the point)
|
||||
## C1 control: the PLAIN key (no forced command) removes snapshot 4137acee
|
||||
$ c.sh plain spike-repo forget 4137acee
|
||||
[0:00] 100.00% 1 / 1 files deleted
|
||||
|
||||
[rc=0]
|
||||
|
||||
server files now: snapshots=2 keys=1 index=2 data=4 locks=0
|
||||
## E5: prune through the FORCED key — unreferenced pack now exists
|
||||
$ c.sh forced spike-repo prune
|
||||
loading indexes...
|
||||
loading all snapshots...
|
||||
finding data that is still in use for 2 snapshots
|
||||
[0:00] 100.00% 2 / 2 snapshots
|
||||
|
||||
searching used packs...
|
||||
collecting packs for deletion and repacking
|
||||
[0:00] 100.00% 4 / 4 packs processed
|
||||
|
||||
|
||||
to repack: 0 blobs / 0 B
|
||||
this removes: 0 blobs / 0 B
|
||||
to delete: 4 blobs / 293.709 KiB
|
||||
total prune: 4 blobs / 293.709 KiB
|
||||
remaining: 8 blobs / 587.247 KiB
|
||||
unused size after prune: 0 B (0.00% of remaining size)
|
||||
|
||||
rebuilding index
|
||||
[0:00] 100.00% 2 / 2 packs processed
|
||||
|
||||
deleting obsolete index files
|
||||
Remove(<index/c4e8a9de5e>) returned error, retrying after 720.254544ms: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/f5b53fa5e4>) returned error, retrying after 582.280027ms: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/f5b53fa5e4>) returned error, retrying after 703.28564ms: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/c4e8a9de5e>) returned error, retrying after 693.478123ms: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/f5b53fa5e4>) returned error, retrying after 1.335175957s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/c4e8a9de5e>) returned error, retrying after 636.341646ms: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/c4e8a9de5e>) returned error, retrying after 1.107876242s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/f5b53fa5e4>) returned error, retrying after 1.007386063s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/c4e8a9de5e>) returned error, retrying after 2.027308147s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/f5b53fa5e4>) returned error, retrying after 2.569756966s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/c4e8a9de5e>) returned error, retrying after 4.987726727s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/f5b53fa5e4>) returned error, retrying after 2.711970641s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/f5b53fa5e4>) returned error, retrying after 5.015617898s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/c4e8a9de5e>) returned error, retrying after 4.659096946s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/f5b53fa5e4>) returned error, retrying after 8.277195667s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/c4e8a9de5e>) returned error, retrying after 6.689436284s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/c4e8a9de5e>) returned error, retrying after 10.163166574s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/f5b53fa5e4>) returned error, retrying after 15.10932531s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/c4e8a9de5e>) returned error, retrying after 13.811796615s: blob not removed, server response: 403 Forbidden (403)
|
||||
Remove(<index/f5b53fa5e4>) returned error, retrying after 13.516432903s: blob not removed, server response: 403 Forbidden (403)
|
||||
unable to remove <index/c4e8a9de5e> from the repository
|
||||
unable to remove <index/f5b53fa5e4> from the repository
|
||||
[0:45] 0.00% 0 / 2 files deleted
|
||||
|
||||
Fatal: blob not removed, server response: 403 Forbidden (403)
|
||||
[rc=1]
|
||||
|
||||
server files after forced prune: snapshots=2 keys=1 index=3 data=4 locks=0
|
||||
$ c.sh forced spike-repo check
|
||||
using temporary cache in /tmp/restic-check-cache-3704335306
|
||||
create exclusive lock for repository
|
||||
load indexes
|
||||
pack 1d5237362eb08ed6ae49c8d57dc28123e665d6efadb34bb1771764f874ae2ba9 contained in several indexes: {2ac5484c c4e8a9de}
|
||||
pack f3a63740213e99cde370d4f8f7f63d808c0e921237ed522678c730e3e81391e5 contained in several indexes: {2ac5484c c4e8a9de}
|
||||
This is non-critical, you can run `restic rebuild-index' to correct this
|
||||
check all packs
|
||||
check snapshots, trees and blobs
|
||||
[0:00] 100.00% 2 / 2 snapshots
|
||||
|
||||
no errors were found
|
||||
[rc=0]
|
||||
|
||||
## C1 control: prune through the PLAIN key
|
||||
$ c.sh plain spike-repo prune
|
||||
loading indexes...
|
||||
loading all snapshots...
|
||||
finding data that is still in use for 2 snapshots
|
||||
[0:00] 100.00% 2 / 2 snapshots
|
||||
|
||||
searching used packs...
|
||||
collecting packs for deletion and repacking
|
||||
[0:00] 100.00% 4 / 4 packs processed
|
||||
|
||||
|
||||
to repack: 0 blobs / 0 B
|
||||
this removes: 0 blobs / 0 B
|
||||
to delete: 4 blobs / 293.709 KiB
|
||||
total prune: 4 blobs / 293.709 KiB
|
||||
remaining: 8 blobs / 587.247 KiB
|
||||
unused size after prune: 0 B (0.00% of remaining size)
|
||||
|
||||
rebuilding index
|
||||
[0:00] 100.00% 2 / 2 packs processed
|
||||
|
||||
deleting obsolete index files
|
||||
[0:00] 100.00% 3 / 3 files deleted
|
||||
|
||||
removing 2 old packs
|
||||
[0:00] 100.00% 2 / 2 files deleted
|
||||
|
||||
done
|
||||
[rc=0]
|
||||
|
||||
server files after plain prune: snapshots=2 keys=1 index=1 data=2 locks=0
|
||||
@@ -0,0 +1,70 @@
|
||||
## A5: kill a backup mid-run through the FORCED key (docker kill -s KILL on the client = a crash)
|
||||
killing r436-cli-2801178 at 09:12:15
|
||||
locks on server after the crash:
|
||||
-rw-r--r-- 1 sub sub 141 09:12:12 b70bc18cbb4e907bc81308a463dd0a86839ef7200eb74c8ba8baf20d51f74fca
|
||||
## next BACKUP (shared lock) from a NEW container (new hostname — as after a controller recreate)
|
||||
$ c.sh forced spike-repo backup /d/tree --host labbox --tag app1
|
||||
using parent snapshot 066a039d
|
||||
|
||||
Files: 0 new, 0 changed, 4 unmodified
|
||||
Dirs: 0 new, 1 changed, 2 unmodified
|
||||
Added to the repository: 325 B (274 B stored)
|
||||
|
||||
processed 4 files, 585.948 KiB in 0:00
|
||||
snapshot 16f27a03 saved
|
||||
[rc=0]
|
||||
|
||||
## next EXCLUSIVE op (check) — does the stale lock wedge it?
|
||||
$ c.sh forced spike-repo check
|
||||
using temporary cache in /tmp/restic-check-cache-108793622
|
||||
create exclusive lock for repository
|
||||
unable to create lock in backend: repository is already locked by PID 1 on 6999b25ca937 by root (UID 0, GID 0)
|
||||
lock was created at 2026-10-03 09:12:12 (9.203992203s ago)
|
||||
storage ID b70bc18c
|
||||
the `unlock` command can be used to remove stale locks
|
||||
[rc=1]
|
||||
|
||||
## plain unlock (stale-only)
|
||||
$ c.sh forced spike-repo unlock
|
||||
successfully removed locks
|
||||
[rc=0]
|
||||
|
||||
locks:
|
||||
-rw-r--r-- 1 sub sub 141 09:12:12 b70bc18cbb4e907bc81308a463dd0a86839ef7200eb74c8ba8baf20d51f74fca
|
||||
## unlock --remove-all
|
||||
$ c.sh forced spike-repo unlock --remove-all
|
||||
successfully removed locks
|
||||
[rc=0]
|
||||
|
||||
locks:
|
||||
## check again
|
||||
$ c.sh forced spike-repo check
|
||||
using temporary cache in /tmp/restic-check-cache-3364593304
|
||||
create exclusive lock for repository
|
||||
load indexes
|
||||
check all packs
|
||||
pack 1a9dd015f3b76fd43ed4f4019bb15d024a36763d2e52f1a9216284a53f9be971: not referenced in any index
|
||||
pack 8184618eaa974e3094796144f9b466468113f9e2bcd82194f9ca58f959d0f822: not referenced in any index
|
||||
pack 63e40e54bedfc638a55e9fe5225231edebc84bb7263b4e4ca25b5933188ebb1c: not referenced in any index
|
||||
pack 97b9da6b48256ec083071902fea61ef56ea5a18f3af869f7ec6eb2ead7772674: not referenced in any index
|
||||
pack b213484fe31fd640b6b519a5ddbe486bfeed32688fea8a0185ff98dcf7165eb3: not referenced in any index
|
||||
pack 5ccfd475d12254508662e5940c923ae132dc0a63b48ba65ff8e3970925f264fb: not referenced in any index
|
||||
pack 0b27c44a9453ac56fb8c171447927d6a45b30ad9233aaa71e4dd9db9a3102b43: not referenced in any index
|
||||
pack af691cecf756d65918ffd1430d8bebdfb720b657ba1ab24c37fd42e1506185ca: not referenced in any index
|
||||
pack 14883d87a878ea91e66a471e892eb1a2f17cc916dcf3be3f6f32f9eb353b755c: not referenced in any index
|
||||
pack 70c49c3190eba6522d6b493b2c35009bd8a562edd5fb706a0ef29a865c56d52a: not referenced in any index
|
||||
pack fa2445b724ba511629fb1dadc75a86f6ae413a61147f7152e50d58effba4ea88: not referenced in any index
|
||||
pack 3902d8453121f05f590775ff617782e0da7027553edcc81305a5629abf6bcde9: not referenced in any index
|
||||
pack f86e1c71608c4ddcde8ad66ada62c42a0459b798f4ce4bc1f8a76a4728a85b1d: not referenced in any index
|
||||
pack 4a080216c5d17e9f2b5eb91cc797901d7f858b86f97cbedb20192e6a90160599: not referenced in any index
|
||||
pack 10a1dfa78d932c22e68f4bf59fde5e5fdb997d79711d67b8029285ab4c9ae934: not referenced in any index
|
||||
pack 70fb6c036df351cc2f9f46ad69ecc8d234f635b4deee44fbc711e7f81174ea62: not referenced in any index
|
||||
pack c7d8a4a140ac753cb9e9232a014e521b5d419c5c29cb29f4713e7b8b71cd0e47: not referenced in any index
|
||||
17 additional files were found in the repo, which likely contain duplicate data.
|
||||
This is non-critical, you can run `restic prune` to correct this.
|
||||
check snapshots, trees and blobs
|
||||
[0:00] 100.00% 3 / 3 snapshots
|
||||
|
||||
no errors were found
|
||||
[rc=0]
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
rclone: rclone v1.75.1 (lab; the provider runs its own version)
|
||||
T1 DELETE snapshots/<existing> -> 403 Forbidden
|
||||
T2 POST over existing snapshots/<id> -> 403 Forbidden
|
||||
T3 POST over existing config -> 403 Forbidden
|
||||
T4 DELETE data/<existing pack> -> 403 Forbidden
|
||||
T5 POST ../.ssh/authorized_keys -> 400 Bad Request
|
||||
T6 POST %2e%2e/.ssh/authorized_keys -> 400 Bad Request
|
||||
T7 POST data/..%2f..%2f.ssh/x -> 400 Bad Request
|
||||
T8 POST a NEW keys/aaaa -> 200
|
||||
T9 POST arbitrary top-level foo -> 200
|
||||
T10 DELETE the new keys/aaaa -> 403 Forbidden
|
||||
T11 POST a NEW locks/bbbb then DELETE -> 200 / 200
|
||||
after: snapshot present: yes; config unchanged: yes; authorized_keys unchanged: yes; stray files:
|
||||
/home/sub:
|
||||
spike-repo
|
||||
|
||||
/home/sub/.ssh:
|
||||
authorized_keys
|
||||
/home/sub/spike-repo:
|
||||
config
|
||||
data
|
||||
foo
|
||||
index
|
||||
keys
|
||||
locks
|
||||
snapshots
|
||||
|
||||
/home/sub/spike-repo/keys:
|
||||
aaaa
|
||||
f651e7eec6b06d2594a748cb05cfaca39f7488092af5c4da5c332a7c3a7a8e5e
|
||||
|
||||
/home/sub/spike-repo/locks:
|
||||
@@ -0,0 +1,73 @@
|
||||
## C3 — retention poisoning. An attacker holding ONLY the forced key + the repo password ADDS snapshots (allowed) dated in the future, same host+tag.
|
||||
real snapshots before:
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
63a075dc 2026-10-03 09:07:50 labbox app1 /d/tree
|
||||
066a039d 2026-10-03 09:07:52 labbox app1 /d/tree
|
||||
16f27a03 2026-10-03 09:12:18 labbox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
3 snapshots
|
||||
after the attacker's adds (forced key, all succeeded):
|
||||
ID Time Host Tags
|
||||
---------------------------------------------
|
||||
63a075dc 2026-10-03 09:07:50 labbox app1
|
||||
066a039d 2026-10-03 09:07:52 labbox app1
|
||||
16f27a03 2026-10-03 09:12:18 labbox app1
|
||||
cf04458a 2027-01-01 03:00:00 labbox app1
|
||||
19394898 2027-01-02 03:00:00 labbox app1
|
||||
bfaf4619 2027-01-03 03:00:00 labbox app1
|
||||
3ba72ec1 2027-01-04 03:00:00 labbox app1
|
||||
8b3b6385 2027-01-05 03:00:00 labbox app1
|
||||
7034c9da 2027-01-06 03:00:00 labbox app1
|
||||
2c18c893 2027-01-07 03:00:00 labbox app1
|
||||
f0caba56 2027-02-15 03:00:00 labbox app1
|
||||
cb7c4ec4 2027-03-15 03:00:00 labbox app1
|
||||
064066cf 2027-04-15 03:00:00 labbox app1
|
||||
bc83265e 2027-05-15 03:00:00 labbox app1
|
||||
af39799e 2027-06-15 03:00:00 labbox app1
|
||||
e887befe 2027-07-15 03:00:00 labbox app1
|
||||
---------------------------------------------
|
||||
16 snapshots
|
||||
## the HONEST retention (the box's exact policy) run later by whoever holds a deleting key — DRY RUN:
|
||||
Applying Policy: keep 7 daily, 4 weekly, 6 monthly snapshots
|
||||
keep 7 snapshots:
|
||||
ID Time Host Tags Reasons Paths
|
||||
---------------------------------------------------------------------------------
|
||||
2c18c893 2027-01-07 03:00:00 labbox app1 daily snapshot /d/empty
|
||||
f0caba56 2027-02-15 03:00:00 labbox app1 daily snapshot /d/empty
|
||||
monthly snapshot
|
||||
cb7c4ec4 2027-03-15 03:00:00 labbox app1 daily snapshot /d/empty
|
||||
monthly snapshot
|
||||
064066cf 2027-04-15 03:00:00 labbox app1 daily snapshot /d/empty
|
||||
weekly snapshot
|
||||
monthly snapshot
|
||||
bc83265e 2027-05-15 03:00:00 labbox app1 daily snapshot /d/empty
|
||||
weekly snapshot
|
||||
monthly snapshot
|
||||
af39799e 2027-06-15 03:00:00 labbox app1 daily snapshot /d/empty
|
||||
weekly snapshot
|
||||
monthly snapshot
|
||||
e887befe 2027-07-15 03:00:00 labbox app1 daily snapshot /d/empty
|
||||
weekly snapshot
|
||||
monthly snapshot
|
||||
---------------------------------------------------------------------------------
|
||||
7 snapshots
|
||||
|
||||
remove 9 snapshots:
|
||||
ID Time Host Tags Paths
|
||||
---------------------------------------------------------------
|
||||
63a075dc 2026-10-03 09:07:50 labbox app1 /d/tree
|
||||
066a039d 2026-10-03 09:07:52 labbox app1 /d/tree
|
||||
16f27a03 2026-10-03 09:12:18 labbox app1 /d/tree
|
||||
cf04458a 2027-01-01 03:00:00 labbox app1 /d/empty
|
||||
19394898 2027-01-02 03:00:00 labbox app1 /d/empty
|
||||
bfaf4619 2027-01-03 03:00:00 labbox app1 /d/empty
|
||||
3ba72ec1 2027-01-04 03:00:00 labbox app1 /d/empty
|
||||
8b3b6385 2027-01-05 03:00:00 labbox app1 /d/empty
|
||||
7034c9da 2027-01-06 03:00:00 labbox app1 /d/empty
|
||||
---------------------------------------------------------------
|
||||
9 snapshots
|
||||
|
||||
Would have removed the following snapshots:
|
||||
{066a039d 16f27a03 19394898 3ba72ec1 63a075dc 7034c9da 8b3b6385 bfaf4619 cf04458a}
|
||||
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
## after B2: spike-r436 still there (the 'rm -rf' through the forced key ran rclone, not rm)
|
||||
$ password p23: ls
|
||||
felhom-repo
|
||||
spike-r436
|
||||
[rc=0]
|
||||
|
||||
## port 22 controls
|
||||
$ plain p22: sftp ls .ssh
|
||||
u629488-sub4@u629488-sub4.your-storagebox.de: Permission denied (publickey,password).
|
||||
Connection closed
|
||||
[rc=255]
|
||||
|
||||
$ password p22: sftp ls .ssh (stdin batch)
|
||||
Connected to u629488-sub4.your-storagebox.de.
|
||||
sftp> ls -la .ssh
|
||||
drwx------ 2 u629488-sub4 u629488 3 Sep 16 16:02 .
|
||||
drwxr-xr-x 6 u629488-sub4 u629488 6 Oct 3 11:20 ..
|
||||
-rw------- 1 u629488-sub4 u629488 510 Oct 3 11:19 authorized_keys
|
||||
sftp> bye
|
||||
[rc=0]
|
||||
|
||||
## B2 port forward, positive test: open -L, then try to use it
|
||||
$ connect through the forward
|
||||
[rc=0]
|
||||
|
||||
ssh -L log:
|
||||
channel 1: open failed: administratively prohibited: open failed
|
||||
@@ -0,0 +1,78 @@
|
||||
## B2 — the FORCED key asked for anything else (port 23 and 22)
|
||||
$ forced p23: shell command 'ls -la .ssh'
|
||||
2026/10/03 11:24:27 NOTICE: Config file "/home/.config/rclone/rclone.conf" not found - using defaults
|
||||
[rc=0]
|
||||
|
||||
$ forced p23: 'rm -rf spike-r436'
|
||||
2026/10/03 11:24:28 NOTICE: Config file "/home/.config/rclone/rclone.conf" not found - using defaults
|
||||
[rc=0]
|
||||
|
||||
$ forced p23: sftp subsystem
|
||||
Connection closed
|
||||
[rc=255]
|
||||
|
||||
$ forced p23: scp download
|
||||
scp: Connection closed
|
||||
[rc=255]
|
||||
|
||||
$ forced p23: rsync --server
|
||||
2026/10/03 11:24:50 NOTICE: Config file "/home/.config/rclone/rclone.conf" not found - using defaults
|
||||
protocol version mismatch -- is your shell clean?
|
||||
(see the rsync manpage for an explanation)
|
||||
rsync error: protocol incompatibility (code 2) at compat.c(622) [Receiver=3.4.1]
|
||||
[rc=2]
|
||||
|
||||
$ forced p23: local port forward
|
||||
[rc=124]
|
||||
|
||||
$ forced p23: rclone serve restic WITHOUT the flag, other path
|
||||
2026/10/03 11:25:16 NOTICE: Config file "/home/.config/rclone/rclone.conf" not found - using defaults
|
||||
[rc=0]
|
||||
|
||||
$ forced p22: shell command 'ls -la .ssh'
|
||||
u629488-sub4@u629488-sub4.your-storagebox.de: Permission denied (publickey,password).
|
||||
[rc=255]
|
||||
|
||||
$ forced p22: 'rm -rf spike-r436'
|
||||
u629488-sub4@u629488-sub4.your-storagebox.de: Permission denied (publickey,password).
|
||||
[rc=255]
|
||||
|
||||
$ forced p22: sftp subsystem
|
||||
u629488-sub4@u629488-sub4.your-storagebox.de: Permission denied (publickey,password).
|
||||
Connection closed
|
||||
[rc=255]
|
||||
|
||||
$ forced p22: scp download
|
||||
u629488-sub4@u629488-sub4.your-storagebox.de: Permission denied (publickey,password).
|
||||
scp: Connection closed
|
||||
[rc=255]
|
||||
|
||||
$ forced p22: rsync --server
|
||||
u629488-sub4@u629488-sub4.your-storagebox.de: Permission denied (publickey,password).
|
||||
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
|
||||
rsync error: unexplained error (code 255) at io.c(232) [Receiver=3.4.1]
|
||||
[rc=255]
|
||||
|
||||
$ forced p22: local port forward
|
||||
u629488-sub4@u629488-sub4.your-storagebox.de: Permission denied (publickey,password).
|
||||
[rc=255]
|
||||
|
||||
$ forced p22: rclone serve restic WITHOUT the flag, other path
|
||||
u629488-sub4@u629488-sub4.your-storagebox.de: Permission denied (publickey,password).
|
||||
[rc=255]
|
||||
|
||||
## B-control — the PLAIN key (what every box holds today) on port 23
|
||||
$ plain p23: ls -la .ssh
|
||||
total 3
|
||||
drwx------ 2 u629488-sub4 1061 3 Sep 16 16:02 .
|
||||
drwxr-xr-x 6 u629488-sub4 1061 6 Oct 3 11:20 ..
|
||||
-rw------- 1 u629488-sub4 1061 510 Oct 3 11:19 authorized_keys
|
||||
[rc=0]
|
||||
|
||||
$ plain p23: sftp ls .ssh
|
||||
sftp> ls -la .ssh
|
||||
drwx------ ? u629488-sub4 1061 3 Sep 16 18:02 .ssh/.
|
||||
drwxr-xr-x ? u629488-sub4 1061 6 Oct 3 13:20 .ssh/..
|
||||
-rw------- ? u629488-sub4 1061 510 Oct 3 13:19 .ssh/authorized_keys
|
||||
[rc=0]
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
## C2: forced key, client names another path
|
||||
$ lc.sh forced other-path snapshots
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
d807418c 2026-10-03 11:20:19 livebox app1 /d/tree
|
||||
68885c41 2026-10-03 11:20:22 livebox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
2 snapshots
|
||||
[rc=0]
|
||||
|
||||
## A5: crash a backup mid-run (docker kill -s KILL)
|
||||
killing r436-live-3883038 at 11:31:14
|
||||
locks after crash:
|
||||
-rw-rw-r-- 1 u629488-sub4 1061 141 Oct 3 11:31 96a4f54c6477793c31050da87ad5e835bbbed2ce76549634a1ccf6cf5a065524
|
||||
$ lc.sh forced spike-r436 backup /d/tree --host livebox --tag app1
|
||||
using parent snapshot 68885c41
|
||||
|
||||
Files: 0 new, 0 changed, 4 unmodified
|
||||
Dirs: 0 new, 1 changed, 2 unmodified
|
||||
Added to the repository: 325 B (274 B stored)
|
||||
|
||||
processed 4 files, 585.948 KiB in 0:00
|
||||
snapshot 8f720f1d saved
|
||||
[rc=0]
|
||||
|
||||
$ lc.sh forced spike-r436 check
|
||||
using temporary cache in /tmp/restic-check-cache-2564422886
|
||||
create exclusive lock for repository
|
||||
unable to create lock in backend: repository is already locked by PID 1 on 91671d11c1a8 by root (UID 0, GID 0)
|
||||
lock was created at 2026-10-03 11:31:08 (14.102493775s ago)
|
||||
storage ID 96a4f54c
|
||||
the `unlock` command can be used to remove stale locks
|
||||
[rc=1]
|
||||
|
||||
## plain unlock
|
||||
$ lc.sh forced spike-r436 unlock
|
||||
successfully removed locks
|
||||
[rc=0]
|
||||
|
||||
locks:
|
||||
-rw-rw-r-- 1 u629488-sub4 1061 141 Oct 3 11:31 96a4f54c6477793c31050da87ad5e835bbbed2ce76549634a1ccf6cf5a065524
|
||||
## unlock --remove-all
|
||||
$ lc.sh forced spike-r436 unlock --remove-all
|
||||
successfully removed locks
|
||||
[rc=0]
|
||||
|
||||
locks:
|
||||
$ lc.sh forced spike-r436 check
|
||||
using temporary cache in /tmp/restic-check-cache-3405178813
|
||||
create exclusive lock for repository
|
||||
load indexes
|
||||
pack c67a24b49d4e41a91d6de9448cb7c72623b6eb12ee8a50b7f4db2e2dfb465480 contained in several indexes: {85027deb a5b0181c}
|
||||
pack 92475c24d7b0495584f168c7a73e6f0e35c9c27881c2ecbb0e18695b889cfb7f contained in several indexes: {85027deb a5b0181c}
|
||||
This is non-critical, you can run `restic rebuild-index' to correct this
|
||||
check all packs
|
||||
pack 9783961d590d0675569230cf7662548d03914af0eebe1d4b1eb6b5573dafb9a8: not referenced in any index
|
||||
pack 5aa289972c6799f5a18c991f3f2c36a2ac8fcdeb41ed5b2708807a774bc730ae: not referenced in any index
|
||||
pack f16746415ed296cc92e78df06859b3fd2c0ba3deb73a31c97e8ed301ee7c214a: not referenced in any index
|
||||
pack 51690a831d4a96fbeed192e0ff0ce879d309994ccb044ead6a1f2096baeea610: not referenced in any index
|
||||
pack 38026bf50bc4028ef383ea19fbdeff362a925d6ecd4be66a3c635b8796fe623e: not referenced in any index
|
||||
5 additional files were found in the repo, which likely contain duplicate data.
|
||||
This is non-critical, you can run `restic prune` to correct this.
|
||||
check snapshots, trees and blobs
|
||||
[0:00] 100.00% 3 / 3 snapshots
|
||||
|
||||
no errors were found
|
||||
[rc=0]
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
Sat Oct 3 11:20:12 UTC 2026
|
||||
$ lc.sh forced spike-r436 init
|
||||
rclone: 2026/10/03 11:20:16 NOTICE: Config file "/home/.config/rclone/rclone.conf" not found - using defaults
|
||||
created restic repository 3b129d9736 at rclone:spike-r436
|
||||
|
||||
Please note that knowledge of your password is required to access
|
||||
the repository. Losing your password means that your data is
|
||||
irrecoverably lost.
|
||||
[rc=0]
|
||||
|
||||
$ lc.sh forced spike-r436 backup /d/tree --host livebox --tag app1
|
||||
rclone: 2026/10/03 11:20:20 NOTICE: Config file "/home/.config/rclone/rclone.conf" not found - using defaults
|
||||
no parent snapshot found, will read all files
|
||||
|
||||
Files: 4 new, 0 changed, 0 unmodified
|
||||
Dirs: 3 new, 0 changed, 0 unmodified
|
||||
Added to the repository: 588.177 KiB (587.535 KiB stored)
|
||||
|
||||
processed 4 files, 585.948 KiB in 0:00
|
||||
snapshot d807418c saved
|
||||
[rc=0]
|
||||
|
||||
$ lc.sh forced spike-r436 backup /d/tree --host livebox --tag app1
|
||||
rclone: 2026/10/03 11:20:23 NOTICE: Config file "/home/.config/rclone/rclone.conf" not found - using defaults
|
||||
using parent snapshot d807418c
|
||||
|
||||
Files: 0 new, 0 changed, 4 unmodified
|
||||
Dirs: 0 new, 0 changed, 3 unmodified
|
||||
Added to the repository: 0 B (0 B stored)
|
||||
|
||||
processed 4 files, 585.948 KiB in 0:00
|
||||
snapshot 68885c41 saved
|
||||
[rc=0]
|
||||
|
||||
$ lc.sh forced spike-r436 snapshots
|
||||
rclone: 2026/10/03 11:20:26 NOTICE: Config file "/home/.config/rclone/rclone.conf" not found - using defaults
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
d807418c 2026-10-03 11:20:19 livebox app1 /d/tree
|
||||
68885c41 2026-10-03 11:20:22 livebox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
2 snapshots
|
||||
[rc=0]
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
server files (via password shell, find): snapshots=0 index=0 data=0 keys=0 locks=0
|
||||
$ lc.sh forced spike-r436 restore d807418c --target /d/restored --include /d/tree/sub/note.txt
|
||||
restoring <Snapshot d807418c of [/d/tree] at 2026-10-03 11:20:19.591226798 +0000 UTC by root@livebox> to /d/restored
|
||||
[rc=0]
|
||||
|
||||
restored vs source: IDENTICAL
|
||||
## E4
|
||||
$ lc.sh forced spike-r436 forget d807418c --prune
|
||||
[0:48] 0.00% 0 / 1 files deleted
|
||||
|
||||
unable to remove <snapshot/d807418cbc> from the repository
|
||||
blob not removed, server response: 403 Forbidden (403)
|
||||
[rc=1]
|
||||
|
||||
## E6
|
||||
$ lc.sh forced spike-r436 forget --keep-last 1
|
||||
Applying Policy: keep 1 latest snapshots
|
||||
keep 1 snapshots:
|
||||
ID Time Host Tags Reasons Paths
|
||||
-----------------------------------------------------------------------------
|
||||
68885c41 2026-10-03 11:20:22 livebox app1 last snapshot /d/tree
|
||||
-----------------------------------------------------------------------------
|
||||
1 snapshots
|
||||
|
||||
remove 1 snapshots:
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
d807418c 2026-10-03 11:20:19 livebox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
1 snapshots
|
||||
|
||||
unable to remove <snapshot/d807418cbc> from the repository
|
||||
[0:48] 0.00% 0 / 1 files deleted
|
||||
|
||||
blob not removed, server response: 403 Forbidden (403)
|
||||
[rc=1]
|
||||
|
||||
server files: snapshots=0 index=0 data=0 keys=0 locks=0
|
||||
## E5 made real: app2 snapshot, removed by the PLAIN key (C1), then prune through the FORCED key
|
||||
$ lc.sh forced spike-r436 backup /d/tree2 --host livebox --tag app2
|
||||
no parent snapshot found, will read all files
|
||||
|
||||
Files: 1 new, 0 changed, 0 unmodified
|
||||
Dirs: 2 new, 0 changed, 0 unmodified
|
||||
Added to the repository: 293.934 KiB (293.869 KiB stored)
|
||||
|
||||
processed 1 files, 292.969 KiB in 0:00
|
||||
snapshot 973e0dac saved
|
||||
[rc=0]
|
||||
|
||||
app2 = 973e0dac
|
||||
## C1 control: PLAIN key forget
|
||||
$ lc.sh plain spike-r436 forget 973e0dac
|
||||
[0:00] 100.00% 1 / 1 files deleted
|
||||
|
||||
[rc=0]
|
||||
|
||||
server files: snapshots=0 index=0 data=0 keys=0 locks=0
|
||||
## E5
|
||||
$ lc.sh forced spike-r436 prune
|
||||
loading indexes...
|
||||
loading all snapshots...
|
||||
finding data that is still in use for 2 snapshots
|
||||
[0:00] 100.00% 2 / 2 snapshots
|
||||
|
||||
searching used packs...
|
||||
collecting packs for deletion and repacking
|
||||
[0:00] 100.00% 4 / 4 packs processed
|
||||
|
||||
|
||||
to repack: 0 blobs / 0 B
|
||||
this removes: 0 blobs / 0 B
|
||||
to delete: 4 blobs / 293.709 KiB
|
||||
total prune: 4 blobs / 293.709 KiB
|
||||
remaining: 8 blobs / 587.215 KiB
|
||||
unused size after prune: 0 B (0.00% of remaining size)
|
||||
|
||||
rebuilding index
|
||||
[0:00] 100.00% 2 / 2 packs processed
|
||||
|
||||
deleting obsolete index files
|
||||
unable to remove <index/a5b0181c03> from the repository
|
||||
unable to remove <index/3d1291b4e3> from the repository
|
||||
[0:45] 0.00% 0 / 2 files deleted
|
||||
|
||||
Fatal: blob not removed, server response: 403 Forbidden (403)
|
||||
[rc=1]
|
||||
|
||||
server files: snapshots=0 index=0 data=0 keys=0 locks=0
|
||||
$ lc.sh forced spike-r436 snapshots
|
||||
ID Time Host Tags Paths
|
||||
--------------------------------------------------------------
|
||||
d807418c 2026-10-03 11:20:19 livebox app1 /d/tree
|
||||
68885c41 2026-10-03 11:20:22 livebox app1 /d/tree
|
||||
--------------------------------------------------------------
|
||||
2 snapshots
|
||||
[rc=0]
|
||||
|
||||
$ lc.sh forced spike-r436 check
|
||||
using temporary cache in /tmp/restic-check-cache-3559416072
|
||||
create exclusive lock for repository
|
||||
load indexes
|
||||
pack c67a24b49d4e41a91d6de9448cb7c72623b6eb12ee8a50b7f4db2e2dfb465480 contained in several indexes: {85027deb a5b0181c}
|
||||
pack 92475c24d7b0495584f168c7a73e6f0e35c9c27881c2ecbb0e18695b889cfb7f contained in several indexes: {85027deb a5b0181c}
|
||||
This is non-critical, you can run `restic rebuild-index' to correct this
|
||||
check all packs
|
||||
check snapshots, trees and blobs
|
||||
[0:00] 100.00% 2 / 2 snapshots
|
||||
|
||||
no errors were found
|
||||
[rc=0]
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
## teardown 2026-10-03T11:31:45Z
|
||||
Connected to u629488-sub4.your-storagebox.de.
|
||||
sftp> put ak.orig .ssh/authorized_keys
|
||||
Uploading ak.orig to /home/.ssh/authorized_keys
|
||||
authorized_keys sha256 now : 795e715315973740bed25d99df39aebb159b79867423f72be9c92f7ed1072c94
|
||||
authorized_keys sha256 orig: 795e715315973740bed25d99df39aebb159b79867423f72be9c92f7ed1072c94
|
||||
rm spike-r436: rc=0
|
||||
home now: . .. .config .ssh felhom-repo
|
||||
forced key after teardown: u629488-sub4@u629488-sub4.your-storagebox.de: Permission denied (publickey,password).
|
||||
plain key after teardown: u629488-sub4@u629488-sub4.your-storagebox.de: Permission denied (publickey,password).
|
||||
## .config was NOT present before the test (initial listing: .ssh, felhom-repo) — created by the provider's rclone:
|
||||
.config
|
||||
└── rclone
|
||||
|
||||
2 directories, 0 files
|
||||
home after: . .. .ssh felhom-repo
|
||||
@@ -26,6 +26,17 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-03 — off-site append-only, measured on the provider (R-436, R-430)
|
||||
|
||||
> Spike, no product change. Evidence and design: `audits/offsite-append-only-2026-10-03/`.
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-436** | **Hetzner's `--append-only` forced command holds for the key it is pinned to.** On `u629488-sub4` (tester-1's, operator-ruled venue; scratch repo `spike-r436`, removed): pinned key `command="rclone serve restic --stdio --append-only spike-r436",restrict` — `init`, two `backup`s, `snapshots`, `restore` (bytes identical), `check` OK; `forget d807418c --prune`, `forget --keep-last 1`, `prune` → `blob not removed, server response: 403 Forbidden (403)`, rc=1, count unchanged; control with an unpinned key: `1 / 1 files deleted`. The client's path and flags are ignored; no shell, sftp, scp, rsync or port forward (`administratively prohibited`). **Reasoning kept: the pin protects a repository only if no other route can rewrite `authorized_keys` — and the password can (R-820).** | CLOSED 2026-10-03 — MEASURED; the due-check (2026-10-06) is cleared by this measurement | `live/E1-E3-init-backup.txt`, `live/E4-E6-deletes-and-C1.txt`, `live/B2-forced-key-misuse.txt`, `live/TEARDOWN.txt` (authorized_keys restored, sha256 identical) |
|
||||
| **R-430** | **`restic unlock` prints `successfully removed locks` after removing nothing — by design; and `unlock --remove-all` DOES work through the append-only key.** Measured live and in the lab: a crash lock (not yet stale: under 30 min, new hostname) survives plain `unlock`, which still prints success; `--remove-all` removes it because the rclone append-only server allows lock deletion. A crash lock blocks `check`, not `backup`. So `resticStep`'s self-heal stays valid under R-436's transport; the earlier sticky-directory model does not describe it. | CLOSED 2026-10-03 — ANSWERED; not a precondition for the rclone transport | `live/C2-A5-locks.txt`, `lab/A5-locks.txt` |
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-03 — the triage: finished rows moved out of the open register
|
||||
|
||||
> Every row below sat in `OPEN-ITEMS.md` with a finished LEADING verdict (or was verified finished against
|
||||
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user