R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The
hub's notification_log for the run reads: two per-app failures RECORDED, one
digest SENT naming both, and the customer channel SKIPPED with operator_only.
Against the measured previous behaviour — two failures, one email naming one
app, one leaving no trace anywhere.
Scenario D proved itself on an event I had not planned: disk_critical alarmed on
two filesystems, the second was collapsed by the cooldown, and that collapse is
now visible WITH ITS KEY. Yesterday it would have left nothing at all.
A gap the spec did not anticipate is recorded with its fix: the per-app event
also fires from the periodic sweep, outside any run, so making it record-only
would have created a NEW silence. The sweep emits a digest too, with no run_id,
so it stays under the ordinary hourly cooldown.
ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile
SURVIVED the resize and is active (checked, because a resize is a stop/start).
The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere.
Five documents corrected — three of which the task's list did not name, found by
searching. Two audit/evidence documents ANNOTATED, body untouched: they record
what was true when written and that is their value.
R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable,
not unbounded — the original OOM was a 14.46 GB restore — so the restore-test
cadence should still be paced, just not by fear of the endpoint.
target-selection.md's "D-d did not name ep0 either way" is deliberately left
standing. It is the operator's question, not CC's.
STATUS.md 127 -> 83 lines, items rather than sentences.
R-181 CLOSED (controller v0.193.0 + v0.193.1) and proven live on demo-hp for
BOTH reserve terms. The reserve is now a per-app, per-run ADMISSION decision
taken before the app's first write and covering all three write legs, and it
gained a size term. The refusal's wording was not weakened; the behaviour moved
so it became true, verified by sha256 tree fingerprint.
R-156 CLOSED — papra's template mounts the app's own data root. Precondition
re-measured rather than inherited (both boxes were wiped today).
Part 4, documentation only, nothing built:
- R-110 WAITING-ON-OPERATOR -> READY. Ruling: option (b), the installer's
publish channel moves to a TAG. Recorded with the condition that decides
whether it works at all — it must cover BOTH the /scripts/ git-sync and the
nine files the installer fetches from raw/branch/main.
- R-115 WAITING-ON-OPERATOR -> READY. Ruling: mechanism (b), a build-side gate
refusing to deploy or vouch an unpublished version. The third instance (agent
v0.120.0) would have silently downgraded both demo boxes while succeeding.
R-182 NEW: the periodic status refresh has no admission scope, so a refused app
re-alerts on every poll (measured: a second alert pair 13s after the run's).
Pre-existing in v0.192.0; deliberately not fixed in the R-181 task.
capability map: the local-backup row moves to PROVEN-LIVE in BOTH halves.
ROADMAP: R-165 collapses to CLOSED; R-181 collapsed into it.
07-backup-architecture.md: the reserve's contract stated as what the code
provides (S-1 — an architectural contract changed in the same session).
STATUS.md trimmed 150 -> 111 lines, "What's broken" no longer holds shipped
work, and the stale "After:" line (pointing at work that shipped on 2 August)
is fixed.
07-backup-architecture.md gains §7.5.1 (S-1: the contract changed in the same
session): the ceiling §7.5 describes no longer exists for a box built from
golden >= 0.192.0, the bulkhead's replacement is recorded, and R-175 is FIXED
here rather than left standing — the bound is restated as a function of mp1
and scoped to split-layout boxes, naming all three real shapes.
Capability map: new row as IMPLEMENTED, deliberately NOT proven-live, with
the missing leg named — no box has been reinstalled from the golden, and
"the golden baked" is not "a box built from it works".
R-163 CLOSED: the ceiling it recorded stops existing. R-176(a) answered by
P1; (b) WITHDRAWN, since every node is reinstalled rather than migrated.
R-178 filed for the reinstalls, which were not done this session.
CONTEXT S-13 (the variant chosen on measurement; pruning rejected with its
reason) and S-14 (prove first, then vouch — the golden is published but
deliberately unvouched, because vouching is what makes a fresh install pick
up a layout no box has been proven from).
STATUS: plain-language section; both operator questions now answered, so the
waiting-on-you item is cleared. Two older entries trimmed so the page did
not grow.
R-167/R-158 shipped and proven live (controller v0.191.x, hub v0.89.0):
two new capability-map rows PROVEN-LIVE with live citations, and
07-backup-architecture.md §7.5's closing claim "nothing warns when an app
crosses the line" is now false and rewritten (S-1: an architectural
contract changed in the same session). §7.5 also gains the caveat that its
size bound is ONE BOX'S, not the fleet's.
Part 3 SPIKE (audits/SPIKE-r165-mp1-merge-2026-08-02.md): M1-M5 measured,
NO layout touched. Three findings the merge session must not re-derive:
"the layout" is not one thing (200G/50G vs 50G/20G vs 16G/8G); mp1 is a
BULKHEAD and not only a ceiling, so after the merge an overflow reaches
/var/lib/docker; the golden fails closed on the split in four places.
D-a's condition (1) is currently SATISFIED — no external box is in the
hub's register, and both demo boxes are Tier 0 and reinstallable.
Recommendation given, choice NOT made — it ends at the operator's ruling.
CONTEXT.md S-11 (D-c's routing, and why R-158's own backup_failed proposal
was overruled) and S-12 (the monitoring landed BEFORE the merge).
STATUS.md gains the plain-language section and the merge decision, with two
older entries trimmed so the page did not grow.
New rows R-174 (closed same session), R-175, R-176, R-177; each ID grepped
free before minting.
Nine rows into OPEN-ITEMS.md and ROADMAP.md, matching each file's column shape.
R-156 and R-157 had lived only in audit documents - the identical "minted in a
spike doc and never carried across" failure the register already records for
R-153/R-154/R-155, caught by the catalog sweep's own section 8.0 while it was
happening. R-158 was minted by a second session the same day for an unrelated
finding, which is why the sweep's proposals were renumbered R-159..R-162 at filing
time. All nine IDs verified free in BOTH backlog files before use.
Part 0 settled the question the sizing item depended on, by reading:
mp1 is RETENTION, not staging, and neither of the two framings was right. A unit
is the KEPT copy on the app's OWN drive (backup.go:245-255); for an app with no
HDD_PATH the namespace falls back to the system SSD - "the SSD-only system-data
fallback" (appbackup/paths.go:26-27). There is no post-copy deletion: the only
prune is F5 residue-on-old-drives when an app MOVES (backup.go:1053-1112). So mp1
retains the units of driveless apps only - not every app, but not transient
either. Confirmed against the spike: sys_drive held exactly the four driveless
apps and not calibre-web, which had a drive and was still backed up.
A unit is volume tars + DB dumps only, never mp8 userdata
(recovery_unit.go:20-25), so a 1 TB photo library can never overflow one. And mp1
gates the WHOLE chain, not just Tier 1: Tier-2 mirrors the unit "(always)" from
RecoveryUnitPath (tier2.go:302,368) and Tier-3 carries it, so a unit that cannot
be written leaves both with nothing to copy.
Part 2 fired on both triggers - retention, and the fallback undocumented - so
07-backup-architecture.md gains section 7.5. Section 6.1 said a unit lives "on the
app's own drive", which is true and was the whole story only for drive-resident
apps; the no-drive case was undocumented, as was the sizing constraint. 7.5
records the mp0-50G-vs-mp1-20G mismatch, the measured ratios (DB app up to ~2x,
21.1GB -> 40.2GB; file-only 1.00x), and the bound this puts on D5's Lane-1
independence: restorable from the drive alone only while the unit still fits -
about 19 GB file-only, about 10 GB DB-backed. No number proposed; the ratio is the
operator's ruling (R-163).
R-159/R-160 marked SHIPPED only after verifying the template changes are in
app-catalog origin/main, and R-156's gate likewise (check-volume-persistence.py
present). papra is NOT fixed - referred - so R-156 stays open on that one app.
Ranked, with one line of reasoning each: R-157 first (an app can stay down
indefinitely with mechanism B silent on every channel), then R-161 (the gate
exists and nothing runs it, which is why R-156's class recurs - R-29's record is
three orphaned gates and one enforced), R-156, R-163, R-158, R-164, R-162.
Records controller v0.188.0 across the four coupled artifacts.
07-backup-architecture.md is the owning doc:
- new 7.4 = the recovery chain AFTER D5 (7.1 leg 1 superseded; leg 2,
the living-app dependency, explicitly unchanged so this is not read
as more than it is)
- 7.3 collapsed to history, with the correction that the target as
written (data_key-only) was tested in Part 0 and rejected
- 3 records that the two-lane split is now real, not just intended
- matrix rows 3 / 3c (new) / 13; 10.1 D5 itself shipped
Also: new capability-map row, D5 collapsed in ROADMAP + OPEN-ITEMS,
and R-127 filed in both (data_key flag unreliable; O4 can regenerate a
DB password that no longer matches the restored data directory).
The audit is named D5-drive-alone-restore rather than "...secrets..."
because .gitignore blocks *secret* -- a guard worth respecting, not
forcing past.
Four-artifact update per the coupling rule, plus the audit.
07-backup-architecture.md: §10.1 retitled CLOSED with the ruling and the D5
sentence; the FileBrowser network-share row flipped YES->NO, closed at the
PLACEMENT rather than at the bind; the exposure chain annotated with the fifth
surface (decommission-with-migrate guarded only its source) and the correction
that the boundary is the deploy POST, not the dropdown; §7.3 retitled UNBLOCKED;
register row collapsed; open question F answered.
00-capability-map.md: new §D row PROVEN-LIVE, with the un-exercised legs named —
the deploy-POST and decommission refusals are unit-tested, not live-fired.
OPEN-ITEMS.md: R-108 dispositioned; D5 given its OWN row as READY/UNBLOCKED (it
had existed only inside other rows' prose — the R-123 thread-loss pattern);
R-126 registered.
ROADMAP.md: R-108 collapsed to a shipped one-liner; R-126 added.
R-126 filed not fixed: a .fab bundle (plaintext secrets, optional password) can
be exported ONTO a NAS. Split out of R-108 rather than folded in — it is an
explicit customer-chosen export destination, not a browsing surface reaching a
backup tree, so it was never part of D5's precondition.
Live evidence: same-box before/after on demo-felhom through the real authenticated
endpoint, the network-specific refusal on demo-hp, non-effect verified in the
registry, and R-67's share-root bind diffed byte-identical across the deploy.
Replaces the 2026-07-14 DRAFT entirely - it was verified against controller v0.132.0,
51 versions stale, while being cited as authoritative. Now verified against controller
0.183.0 / agent 0.110.0 / hub 0.80.0 / catalog 4252121. NOT ratified.
Every statement is labelled [DESIGN] (a decision from the architecture discussion) or
[FACT] (file:line, live output, or a citation to _recovery-inventory-2026-07-28.md).
Phase 0 verified D5's premise and it FAILED. Ten read surfaces were checked; nine are
clean (SMB under both namespace shapes, FileBrowser for local drives, .fab import and
download, /api/debug/*, all three ServeFile sites, registering the backup dir as a
drive). One is not: FileBrowser binds a NETWORK share at its ROOT while local drives
are userdata-scoped, and no IsNetwork() filter guards the deploy dropdown, the per-app
migrate targets, or handleStorageMigrateApp. So an app namespace - and under D5 its
plaintext secrets - can land somewhere a browsing surface serves with download:true.
Not a leak today (the unit's app.yaml is secret-stripped); it becomes one under D5.
D5 is therefore written as a BLOCKED target, not as settled. -> R-108
Core artifact is the failure -> recovery matrix (15 rows). RTO cells carry only
measured durations; 12 are blank and each blank is listed explicitly as a finding.
RPO cells carry the configured cadence that bounds RPO, labelled as such - no RPO has
ever been measured from an incident and none is invented.
The 7/53 vs 9/43/1 Tier-2 coverage counts are both recorded, unresolved; neither adopted.
Coupling (S-1): capability map section C now cites the matrix instead of restating
routes; ROADMAP gains R-102..R-108; OPEN-ITEMS gains only the in-flight R-108 plus
C9-F1b/F3/F4 -> R-n cross-references; CONTEXT records D1-D6 as standing ruling S-3.
No code, no version bump, no deploy, no box touched.
The last open item from the datastore relocation. Hub operator UI
(Offsite -> PBS DR) reports felhom-offsite (ep0) at 97.9 GB capacity,
12.6 GB used, 13% full - agreeing with the on-box df (98 G / 13 G / 13%).
The gauge follows the datastore's CONFIGURED PATH, so the relocation
required no hub-side change. RUNBOOK section 10.3 warned that a stale
37.2 GB reading would mean the gauge reads the wrong filesystem and
would be a real bug worth a roadmap item - it does not, and there is
no bug.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ARoadHBf8rHoscfiqeVZn
Supervised runbook execution. No code change, no version bump.
felhom-offsite moved from ep0's 40 GB root disk (/srv/pbs-felhom) to a
dedicated 100 GB Hetzner Cloud Volume (/mnt/pbs-datastore, ext4 -m 0,
by-id fstab, relatime). Datastore NAME unchanged, so the PBS-DR
descriptors, per-box storage ids, ACLs and namespaces are untouched.
Capacity: 37.2 GB -> 98 GB total, 28.9% -> 13% used, headroom to the 80%
warn 19 GB -> ~65 GB. This CLEARS the R-82 Phase 0 P0.3 STOP. Per-tenant
encryption still precludes cross-customer dedup, so the slope is
unchanged - the volume buys runway, not a better cost model.
Verified: byte totals and chunk counts identical (9748), 7/7 snapshots
across all three namespaces, backup:backup ownership, clean itemised
dry-run, full verify job TASK OK with 0 errors, and a restore round-trip
(source_tier pbs, pass true, mount_parity ok, clean teardown).
Nothing deleted - the original 13 GB stays at /srv/pbs-felhom as the
rollback until a new weekly backup lands. GC deliberately not run.
Three findings recorded:
- the `scratch` datastore points at a non-existent path (pre-existing;
now logs ENOENT every start) - operator decision
- the runbook's S6 guard test proves the wrong proposition:
RequiresMountsFor re-mounts rather than refusing, so the test only
bites when the device is genuinely unavailable (re-run that way, and
the refusal was observed) - amendment recommended
- S11: storage box u629193 has no live backup path, BUT ep0 carries an
enabled sshfs mount unit against it that must be removed before the
box is deleted
Deviations: the volume arrived pre-formatted and mounted; S8 ran on
demo-felhom rather than demo-hp (no SSH key for demo-hp); the window was
contended by a stale in-memory 10-minute restore-test cadence whose
config had already been reverted on disk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ARoadHBf8rHoscfiqeVZn
- ROADMAP: R-85 row. Code SHIPPED; rotation NOT YET OBSERVED LIVE, stated as
such rather than written as done.
- Capability map: a new row for UNATTENDED restore-proof, IMPLEMENTED not
PROVEN-LIVE, kept distinct from the R-82 row that a MANUAL selftest earned.
That distinction is the same one the activation-vs-arrival split made.
- 03-host-agent §8: the scheduler covers every tier, oldest-proven first; the
spec is per-run; a restore-test joins the one-heavy-op gate. The safety
properties that must not be re-derived are listed.
- 07: restore-proof recorded as a per-tier property. Doc still NOT ratified.
- 06: corrects S4.1's 'the offsite restore-test now runs unattended' — it
silently stopped being true when local_backup_target was retargeted to 'local',
the SECOND time in that doc that a correct mechanism was broken by its input
changing underneath it.
- CONTEXT + REUSE.
Hub gate green (17 packages, rc=0).
Slice D.1 — host-install 1.20.0: a FRESH box defaults to local-daily +
offsite-weekly (felhom-pbs, 604800s, keep_last=2). setdefault semantics proven
both ways: fresh gets the tier, an UPGRADE preserves the existing backup block
verbatim — so an in-place upgrade can never silently start writing to an
offsite datastore. Existing boxes are migrated explicitly.
Slice E:
- 07-backup-architecture.md: honest status header per CONTEXT ruling S-2, with
an explicit STALE-outside-the-PBS-tier verdict (the controller tiers were last
verified 41 controller versions ago). The PBS row claimed 'PBS on DooPlex'
(the retired spike store) with no cadence; it now names felhom-pbs ->
felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2. NOT marked
ratified — that is Viktor's review of the section 10 list. Discharges R-83.
- 06-offsite-connectivity.md: the target-split remaining-work note collapsed
(shipped), and records HOW S4.1's tier-aware timeout silently regressed — the
mechanism was never removed, its INPUT changed when local_backup_target was
retargeted to 'local'. Also notes S4.1 already diagnosed the teardown 403 as a
phantom (a timeout consequence, not an ACL gap).
- capability map: new row for recurring offsite backups actually LANDING, as
distinct from the existing row proving ACTIVATION. IMPLEMENTED, not
PROVEN-LIVE — the restore round-trip has not completed under the fixed code.
- ROADMAP: R-82 SHIPPED with its remaining gate named, R-83 DISCHARGED, R-84
left open.
- CONTEXT + REPORT: the arc, including the mid-arc correction I had to make.
Per the operator's 2026-07-26 ruling: datastore growth deferred, R-82 proceeds.
Records the measurements so the constraint is written down rather than carried
in a session: 37.2 GB total / 10.8 GB used, no cross-customer dedup (per-tenant
encryption keys), 80% warn reached at roughly the second additional customer,
and the pvesm 0/0/0 reporting artifact that means operators must read fill from
the hub gauge. Also records what the tier does and does not carry, and the
conditional on the P0.1 weekly verdict (Tier-3 offsite must be healthy).
Doc NOT marked ratified — that stays Viktor's review (R-83).
admin
Claude Code