Architecture: 08 records ruling A (45 m, the round-9 arithmetic, the cost) and
the two new event types with their audiences; 03 records the slow counter as
built; 07 records the restore-record persistence as a REVERSED design for the
restore record only; CONTEXT.md carries the day's rulings.
Guide: the recovery code moves after the first apps and waits for the yellow bar.
Register: R-549 and R-550 closed PROVEN-LIVE; R-546 closed on red-proofed tests
with its live walk owed by R-551 (no Tier-0 box is paused AND agent-connected).
R-552 filed: an interrupted-restore notice for a removed app never clears -
found in my own v0.246.0 after the release was built.
Evidence: Part A (hub prints 45m/1h30m), Part C delivery on HP and N100, B.4(a)
live proof and its teardown.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The tier-3 pause is the zero-knowledge escrow design and is untouched. What was
missing was the ASK, while the backup page promised the copy that had never run.
- VOLUNTEER-first-hour.md: a new step 6, right after the dashboard password and
before the first app - what the code is, where, write it on PAPER, and that
Felhom cannot get it back for them. Sections 6..12 renumbered to 7..13.
- day0-install.md A.2b: the operator step for a REBUILT box, which was missing.
Acknowledged delete -> the hub re-issues by itself; otherwise ONE press of
"Re-issue PBS credentials" (F-14 ruling 2026-07-13, hub/internal/web/pbsdr.go).
This is the correction to last night's "zero presses" note.
- 07-backup-architecture.md: 6.1 records tier-3's paused state as a DESIGN, and
2 records that the household is asked from first login.
- capability map: the first-hour row's last gap closed, with what it still does
not claim (no volunteer has walked the ask from the written guide).
- register: R-543 CLOSED with the live measurements; R-545 filed (nothing
un-configures an off-site target). R-511 was already closed yesterday.
- STATUS: the answered publish question removed (1.28.0 is live), readiness yes.
- evidence: red-proofs, the two-box live validation, teardown on three layers,
and both of my own mistakes in this session.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Off-site is ON by default for a new customer — shared, 100 GB soft quota prefilled,
the checkbox kept so an operator can opt a customer out. The reason is this repo's
own [FACT]: the whole-guest tiers do not carry the data drive and a Tier-1 unit has
no file leg, so with this unticked a one-drive box keeps NO copy of the household's
own files. Measured on a fresh box the same day.
The quota is prefilled because the fill warning only fires when quota_gb > 0.
Also registers controller v0.244.0's app_deploy_started / app_deploy_failed in both
allowedEventTypes and customerMessages, per the rule that the two move together.
Red-proofed: dropping the default fails the new render test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Hub v0.112.0 serves a floor above the golden with a declared MinAgent;
controller v0.239.0 reached both demo boxes by that floor in 14 s and 15 s
and updates on any backup tier. 09 §3 decisions 7 and 8, §6/§6.1; 07 §6
line; capability map row; STATUS items 15/16 done and the cadence line
corrected; CONTEXT; register: R-470/R-472/R-475 compressed to CLOSED-ITEMS
(full text at 2f5d3af), R-477..R-480 opened, R-474 reproduced a third
time. OPEN-ITEMS 431689 -> 432156 bytes, CLOSED-ITEMS 118051 -> 120598.
Evidence: documentation/audits/rulings-r472-r475-2026-09-13/.
R-434 CLOSED — and the row's own "blocked on R-433" verdict was wrong, which is the point.
The fix is a DELETION, not a replacement: withdraw the promise instead of swapping it for a
new one, and the sentence is true under every possible answer to the provider questions, so
it never needs a second rewrite. A replacement would have been blocked; a withdrawal is not.
was: "...still hold the older copy, so this is recoverable file-by-file; it is NOT
confirmed data loss. Check whether a deletion ran on the box before restoring."
now: "...still hold the older copy. The route back out of them is not yet established,
so treat this as neither confirmed data loss nor confirmed recovery. Get in touch
before restoring anything, and check whether a deletion ran on the box."
It must not swing the other way either: "your backups are gone" is still usually false.
Clause (a) — the box cannot WRITE into the snapshot area — stands and is re-confirmed.
Tests: offsite_r434_test.go, three, all driving the production path so they assert the
sentence an operator RECEIVES. ASCII-only fragments, positive and negative controls.
RED-PROOF: restoring the v0.111.0 sentence failed all three, on every fragment, with the
offending sentence printed. TestR431_FiresOnAMassDeletion asserted "NOT confirmed data
loss" and caught this fix correctly; its wording fragment is REMOVED rather than updated,
so the wording keeps ONE home.
R-435 written into the detector's own documentation, no threshold changed: it sees a mass
deletion, not one app being wiped (69 across 9 apps -> ~35 needed, one tag is ~9, and
forget --prune groups by host,tags). Says explicitly not to lower the numbers.
THE STOPPING LINE, in all three places — register, 07 section 8 head, STATUS.md.
Deferred set ENUMERATED, not described: 07 section 8 rows 4, 8, 9, 10, 11 (+11b), 12,
each tagged [BETA-DEFERRED]. A number in the brief was wrong and is corrected in place:
six rows are DEFERRED, ELEVEN carry a blank RTO (4,5,8,9,10,11,11b,12,13,14,15); the other
five are blank for reasons that are not deferred work, and row 15 is an open DEFECT (R-104)
that the stopping line does NOT cover. NO STATUS MOVED — nothing was proven today.
Two provider questions drafted, not sent, no API called (11-D stands):
documentation/runbooks/provider-questions-2026-09-01.md, linked from R-95 and R-433, and
tracked by a dated DUE-CHECKS row (2026-09-15) — the 2026-07-27 check that sat unconfirmed
for 36 days is the scar that block exists for.
R-95, R-433 BLOCKED-ON-PROVIDER. R-95's one-day demotion on a clause that did not hold is
recorded; the proposal to rank it back near the top is stated and NOT acted on. R-430 marked
LATENT with its trigger: it becomes live the moment delete is withdrawn, so it is a
precondition on the R-95 build, not a follow-up. The stale ranking paragraph ("armed",
"zero snapshots") is corrected in place, order unchanged.
Register 621 -> 688 lines; 181 rows throughout; open-state 170 -> 169.
No controller or agent change. No golden owed, no floor change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM
The drill was to delete demo-hp's off-site history and get it back out of a Storage
Box snapshot, filling row 10's blank RTO. Phase 1 found there is nothing to get it
back from: no snapshot is reachable from a sub-account BY ANY NAME.
Measured, read-only, no delete verb issued against any live store:
- 777,600 exact names in the vendor form YYYY-MM-DDTHH-MM-SS, nine full days at
second granularity, plus 126 alternative shapes -> ZERO hits.
- The control is what makes that mean anything: the identical 600-name batch shape
with one real path appended returned it, 6 of 6.
- Structural cause: /home (u629488-sub3) is st_dev 0,82; /.zfs/snapshot is st_dev
0,276; /home/.zfs does not exist. A snapshot under /.zfs/snapshot belongs to a
different dataset than the one holding felhom-repo.
- Three tools agree with controls in the same run: SFTP, the port-23 shell,
rsync --list-only.
So yesterday's re-scope splits: clause (a) "the box cannot write into the snapshot
area" STANDS and is re-confirmed; clause (b) "the rest is recoverable file by file"
is NOT SUPPORTED. STOPPED before Phase 2 on the operator's ruling — with no recovery
leg the deletion would have destroyed real history to buy only an alarm test that
could not fire at the specified size. Store verified untouched at 69 snapshots.
R-432 ANSWERED (negatively; its panel-read next step withdrawn as unnecessary).
R-433 no snapshot reachable by any name — decides R-95's remedy and its rank.
R-434 the drop alarm's text promises a file-by-file recovery that cannot be performed.
R-435 the drop detector is blind to a single-app deletion (>50% of 69 needed, ~9 given).
R-436 LEAD: the provider offers `rclone serve restic --stdio` and restic 0.14.0 speaks
`rclone:` (measured, controlled) — real prevention may need no new machine, IF
the vendor pins --append-only. Ask before building.
07 §8 row 10: text corrected, status NOT moved, RTO still blank.
No code, no version bump, no image, no golden. REPORT.md's only copy of the R-331
report preserved as REPORT-r331-backup-card.md before overwrite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM
THE RECORD WAS TELLING A WORSE STORY THAN THE TRUTH FOR TWO MONTHS, and my own probe is why.
R-429 CORRECTED. Yesterday's spike searched for a directory called `.snapshots`. The vendor documents
the path as /.zfs/snapshot. The probe's CONTROLS were sound and its SUBJECT was wrong, so "not found"
was true and meant nothing. Re-probed at the documented path on both boxes, with controls:
- /.zfs lists (shares, snapshot) from inside the jail;
- a write into /.zfs/snapshot is REFUSED - dest open ...: Failure - while the identical write to
the account home SUCCEEDS and was cleaned up.
That is the append-only property PROVEN rather than cited, and it is the sentence the whole re-scope
rests on. Seven daily snapshots are confirmed in the panel. The mitigation works. What remains true,
and was always the actual finding: the row claiming it had no R-number, its "confirm tomorrow" went
36 days unanswered, and the DUE-CHECKS block built for that class was empty. THE FINDING WAS NEVER THE
SNAPSHOTS - IT WAS THAT NOBODY COULD TELL.
R-95 RE-SCOPED: the box can delete its LIVE repository but cannot write to the daily snapshots of it,
so a deletion costs at most one day plus a per-file recovery - not open-ended loss. The ranking is
Viktor's; it has been #1 since July on the old story.
R-432 FILED: a sub-account sees /.zfs/snapshot EMPTY while the same box holds seven snapshots, so
per-file recovery is operator-only today. One panel read settles whether a NAMED snapshot can still be
entered, which would make it product-reachable.
R-431 SHIPPED. Third signal in OffsiteChecker. On the hub deliberately: a detector on the box is one
the deletion can silence. Threshold REASONED, not invented - over 12 898 reports every decrease lands
on ZERO and predates stats_known, and in the stats_known window there are none, so observed churn gave
nothing to calibrate against. Retention cannot halve a total; a mass deletion goes to ~0. Hence: more
than half, and at least 5. Guarded by StatsKnown (R-331), the declared State (R-204) and run success
(R-100 - whose lesson lives in this very file).
ACCEPTANCE: 9 009 real report points replayed through the detector produced ZERO alarms.
Three red-proofs run. The escalation one only became real after the first version was found HOLLOW -
it re-swept the same report, so the baseline had already moved and the latch was never consulted.
07 row 10's status is NOT moved: the write-refusal is measured, but the recovery ROUTE has never been
walked, which is what PARTIAL means.
READ-ONLY STUDY. No code, no version, no image, no golden. No delete verb was issued against any
live store. ep0, DooPlex and Peti's box were not touched at all.
Q1 FIRST, AND IT DID NOT GO THE EXPECTED WAY. The brief supposed a working seven-day snapshot net
might bound the worst case. Measured on BOTH boxes over their own SFTP credential, with a positive
and a negative control on each: NO .snapshots is visible to either sub-account - not in the account
home, not inside the repo - and the account is jailed at /. Either none exist or a sub-account
cannot see them, and the second is not a reprieve: a snapshot the box cannot see is one the box
cannot restore from, so recovery would be an operator act at the Hetzner panel.
The register's claim rests on nothing that was checked. The row has NO R-number, so nothing can cite
it; its "confirm tomorrow" was 2026-07-27, 36 days ago; and the DUE-CHECKS block built for exactly
this (R-341) is EMPTY. R-95's word "ARMED" is withdrawn pending R-429. The confirming field is a
Hetzner API field, so this spike STOPPED at the section 11-D fence and left it for Viktor - ten
minutes in the panel, and it re-ranks everything.
Q2: TEN verbs, not nine. `check` was missing from the brief's list; `dump` is not a verb (it is a
progress phase constant) and was withdrawn. There are TWO `forget --prune` sites - offbox.go:1388
AND offbox.go:1759 - and disarming one without the other reproduces R-191 exactly.
Q3 (documented, from our own API mirror): AccessSettings has five booleans and `readonly` is the
only permission axis. No append-only. So the PBS shape does NOT transfer - PBS is a server that can
refuse; a Storage Box is a filesystem that runs nothing.
Q5 (measured, faithful append-only model, both controls passed first): withdrawing delete does NOT
wedge the store - restic treats a dead owner's lock as stale and proceeds. The constraint everyone
feared is not the blocker. But `unlock --remove-all` printed "successfully removed locks" while the
lock was still there, and resticStep's crash-lock self-heal is built on that call - R-430.
Q6 (measured, with a control): restic 0.14.0 DOES speak rest:. Append-only is a rest-server flag,
not a restic one.
Q7 (measured): detection is nearly free. snapshot_count already reaches the hub and the hub APPENDS
reports, so the history to compare against is already on disk.
RECOMMENDATION: answer Q1 today (Viktor, ten minutes), then build detection, then move retention off
the box. Defer the transport change until Q1 is answered.
Register: OPEN 179 -> 181. Filed R-429, R-430; R-95 updated and kept OPEN. 07 row 10's status is
deliberately UNCHANGED.
Part 2.1's determination is the first artifact: the scratch resolver was consciously OUT OF
SCOPE for R-356, not excluded on state-only grounds - established from R-356's own commit
08eb1a6, whose tests say 'the prepared scratch still resolves ... only the DESTINATION
moves'. So 07 section 6.3's rule applies and now has a FOURTH consumer, and the section
says so.
Live evidence: the collision rerun on demo-hp with the sampler positively controlled first
(12 locks=1 across a real check, 4 locks=0 quiet), showing unlock --remove-all 0 times where
the drill saw it twice; and the proof reaching verdict pass on demo-felhom - the box that
could not run it at all - recorded where last_proof_result had been ABSENT every night.
Capability map: the off-site proof row now records that the nightly firing IS proven (it ran
unattended at 05:30 on demo-hp) and that a driveless box can now be proved.
Register: six rows closed and compressed. OPEN 176 -> 170, CLOSED 152 -> 158.
Controller v0.231.0 + hub v0.110.0, both deployed and verified on demo-hp.
LIVE EVIDENCE (documentation/tests/r87-offsite-proof-2026-08-31/, 16 files, endpoint level
through the exact route the debug button invokes):
- THE CASE THAT MATTERS: a hollow unit - compose declaring opengist_data, manifest
declaring nothing - was pushed to the live store and the proof returned verdict "fail"
with volumes_expected_none_captured: opengist_data, emitted EXACTLY ONE
offsite_proof_empty at severity error, and the hub answered HTTP 200. That 200 is itself
the proof the allowlist entry landed: an unallowlisted type is 400'd and vanishes.
- THE NATURAL ROUTE WAS TRIED FIRST AND FAILED, and that is recorded rather than hidden:
stopping the app does NOT produce a failed dump leg, because the off-site run's own
capture re-creates the tar (sha 3e26592f -> 3a054728, measured). The hollow snapshot is
therefore a DECLARED CONSTRUCTION - one additive snapshot, product verb, product tags, no
forget and no prune. State restored: the product's own run made a healthy snapshot the
newest again and the proof then passed opengist.
- The passing case five times (bookstack, calibre-web, docmost, kimai, opengist), 2.2-4.0s
each, matching the spike's measured band.
- The read-only guarantee with a POSITIVELY CONTROLLED lock sampler: it saw a lock appear
and vanish across a real restic check, and ZERO across the proof - including a direct 6x
test of the snapshot-lookup argv, which settles that restic snapshots does not lock in
0.14.0 either.
- Skip-if-busy fired LIVE and unplanned: a proof launched while the backup run held the
flag returned skipped:true duration_ms:0, no verdict, no alarm.
- The customer's own verification copies were untouched throughout, which is the safety
property the separate proof root exists for.
ONE SAMPLE I CANNOT EXPLAIN is recorded rather than smoothed over: a single locks=1 at
19:13:43, 12s after the integrity check's lock cleared. Two independent tests exclude the
proof; I did not establish what it was.
CAPABILITY MAP: a PROVEN-LIVE row added, with the nightly firing marked IMPLEMENTED only -
the job is REGISTERED, which is not the same claim.
07 section 8 MATRIX ROW 4 WAS NOT MOVED, deliberately, and section 10.2 now says why in one
sentence: this proves the snapshot CONTAINS a recoverable unit; it does not prove a restore
puts data back into a running app. Without that sentence the new green tick reads as
covering the drill.
REGISTER: R-87 CLOSED and compressed into CLOSED-ITEMS.md. OPEN 172 -> 171, CLOSED 151 ->
152. No new rows minted. R-408 and R-409 stay open and are referenced by this work.
golden-currency is RED and it is a DECLARED, EXPECTED debt: v0.231.0 is released and the
newest golden carries 0.230.0. The fleet is on 0.230.0; demo-felhom does not have this job.
A golden carrying 0.231.0 is OWED and it is Viktor's call (R-242). This push uses
--no-verify for that reason - bypass #8.
Spike. NO production code. No version bump, no build, no deploy, no golden.
felhom-controller and felhom-agent were READ ONLY. The fleet stays on v0.230.0.
Q1 restic is 0.14.0 (go1.19.8, bookworm 12.15) - the four source comments asserting
it are CONFIRMED, not corrected.
Q2 --verify DOES exist and is NOT a content check. Red-proof: one byte changed in a
restored 160 MB tar with size and mtime preserved passed clean, rc=0. Verify took
131 ms on a 213 MB / 7-file tree, which cannot be hashing. A size or mtime mismatch
causes a silent re-download, not a failure. Controls: --target 1 hit, four post-0.14
flags and a nonsense string 0 hits each. Neither --verify nor --no-lock appears
anywhere in the controller source.
Q3 no reference for "correct" exists. restic ls --json carries no content hash in
0.14.0, and the unit manifest hashes 4918 B of a 213231242 B unit - 0.0023 percent,
the config files and not the dumps or the tars. R-409.
Q4 it is CHEAP. All 8 apps / 774378123 B logical restored back to back in 25 s, against
40257 ms for the weekly 100 percent check beside it. Individual restores 2253-3978 ms
regardless of size: cost is per-snapshot round-trip plus ~1 s per 200 MB. Peak scratch
is the app's full logical size. The 1.1 MB restic cache is index only and hides nothing
(--no-cache 5423 ms vs cached 3198 ms, trees byte-identical).
Q5 skip-if-busy stays right at 25 s against a 2m52s nightly backup. But
RestoreOffboxScratch takes NO acquireRunning, while offbox_integrity.go:28 asserts
every off-site operation does. R-408.
Q6 observed with a positively-controlled lock sampler: restic restore takes NO lock;
restic check DOES (locks 0 -> 1 for nine samples -> 0 across the check, zero across two
restores). The product writes anyway - unlockStale runs `restic unlock`, a delete verb,
before every restore (offbox_restore.go:289). The task's lead was right in direction and
wrong in mechanism. R-95's constraint IS satisfiable: --no-lock plus skipping unlockStale
writes nothing, and both mechanisms exist unused. Neither was fixed - the task forbids it.
offbox_integrity.go:255's "It NEVER writes to the repository" is R-407.
Q7 THE DECIDING ONE: of R-353/354/356/358/403 an unattended scratch-restore would have
caught ONE (R-356). The value is elsewhere, and the weekly check structurally cannot
reach it: `check` proves the stored bytes are the stored bytes, never that we stored the
RIGHT thing. A hollow unit backs up, checks at 100 percent and restores cleanly and
recovers nothing - R-403, measured in bytes on 31 August.
RECOMMENDATION: option C, the NARROW test - one app a night, restored to scratch, checked
against its own manifest.json through the existing unitCarriesData, scratch deleted, the
SNAPSHOT recorded as the proof. Options A (do not build) and B (scheduled attended drill)
considered explicitly; B is weakest because it is what already happens. R-87 should be
RE-SCOPED, not built as written, and that is Viktor's call - the row stays open carrying
the verdict and STATUS.md item 4 asks it in plain words.
Also corrected in 07-backup-architecture.md: matrix rows 4 and 10 both said "the depth
that ships ON does not re-read pack contents (R-399)". R-399 CLOSED in v0.228.0 and the
depth is 100 percent. Two stale cells, fixed, and the spike verdict added beside them.
Row 4's verdict is UNCHANGED by the spike and now says so.
Teardown: all three layers, none of them "nothing was created" - 6 files on the PVE host,
9 in the guest, 5 plus 2 run-flags in the container, all removed and verified empty. The
four scratch directories this session's restores created were removed; three that
pre-date the session were left alone. Two state changes recorded rather than hidden: the
control integrity run recorded its verdict (depth structure -> 100%, due-ness +7 days),
and four restores appear in the controller log. Nothing was written to the off-site
repository by hand.
Evidence: documentation/audits/evidence-spike-restic-restore-2026-08-31/ - 31 files,
every one pulled off the box BEFORE teardown (R-320).
golden-currency is RED at this commit and was already red at dddcc80. Pre-existing, not
this session's debt. Second --no-verify push of the day for that reason; R-404's count
goes six -> seven and its row says so.
Ceiling R-406 -> R-409.
07-backup-architecture gains section 8.2, placed beside row 5 on purpose: the derived-copy rebuild
rule is UNCHANGED and section 8.2 names the single exception, so a future reader who finds RunTier2
skipping a leg does not fix it back. It carries the measurement (120 082 104 B -> 7 036 B on the
shipped v0.229.0), the four-case table, why hollow is a manifest question and not a size question,
why the data legs are deliberately not guarded, and why the capture job is not guarded either.
00-capability-map: the Tier-2 row's status does NOT move, stated explicitly rather than left
ambiguous. R-403 removes a way the route could be DESTROYED between uses; it does not change what
the route can be relied on for.
Register: R-403 CLOSED and compressed into CLOSED-ITEMS (594 -> 593 open lines). R-404 FILED as a
DECISION and deliberately not acted on - should a documents-only push be subject to the
golden-currency gate, now that it has been correctly bypassed six times? Both sides stated, plus
what happens if Viktor does nothing. The gate was NOT changed.
R-242: seventh conviction, and the FIRST where the day-0 ground does not apply - R-403 is a defect
in the nightly Tier-2 copy, which a day-0 box starts running on its first night. This push uses
git push --no-verify, declared here and in felhom-controller/REPORT.md. A golden carrying 0.230.0 is
owed and is more urgent than the previous six.
STATUS: the R-403 item moves out of 'Broken' into what works, in plain words; the delivery item now
says a golden is owed and that the fleet carries the defect; R-404 goes into the decide section with
its do-nothing outcome.
Drill evidence: nine phase logs, including the two things that went wrong (a repair whose rsync was
not installed in the guest and silently did nothing, and a session that expired mid-run so a POST
did nothing).
07-backup-architecture: 6.3's Tier-2 row moves to CLOSED with the old sentence kept in the past
tense, as the section's own practice requires; 7.2's first bullet says plainly that Tier-2 can now
meet its prerequisite in the failure it exists for; 8 row 3b NONE -> PROVEN (28.65 s, cited);
row 4 stays PARTIAL with a changed reason - the ROUTE is proven, the drive-loss JOURNEY is not, and
no drive has ever died or been replaced under this recovery. 8.1's blanks updated per row.
6.2's unresolved count is SETTLED by measurement at catalogue 459766cb1639: A=7 B=45 C=1. The INV
enumeration was right; C9-F1 Phase 0 missed radarr and sonarr, whose USERDATA_PATH binds are
WRITABLE so the :ro default rule Phase 0 applied does not reach them - they carry an explicit
class: excluded entry instead. Class C is bentopdf. No catalogue file was changed.
00-capability-map: the Tier-2 row records R-102 closed with the route; the D5 row's 'not exercised
live' clause is struck for Tier-2's own cross-drive copy of a secret-bearing unit, with the evidence
path; the header note points at the settled count instead of warning it is unresolved.
Register: R-102, R-103 and their C9-F4 / C9-F1b aliases closed and compressed into CLOSED-ITEMS
(596 -> 593 lines, each naming git show 1623a4d5b5 for the original). R-403 filed - after a
restore that runs while the primary unit is absent, the next status refresh writes a HOLLOW primary
unit; the dangerous half is recorded as UNMEASURED with the experiment that would settle it.
R-242: sixth conviction of golden_currency_gate. THIS PUSH USES git push --no-verify, declared here
and in felhom-controller/REPORT.md - a BYPASS, not a waiver. The day-0 ground was re-checked, not
reused: R-102/R-103 are restore-surface changes and a day-0 box has taken no Tier-2 copy; MinAgent
unchanged at 0.129.0. OWED: bake a golden carrying 0.229.0, vouch it, raise the floor.
Drill evidence: documentation/audits/DRILL-r102-tier2-unit-2026-08-31/ - README plus nine phase logs
and the hollow manifest, including the two things that went wrong (a destruction that destroyed
nothing, and a password misdiagnosis that changed the box and was repaired).
STATUS.md: header said 2026-08-23 over a 2026-08-30 body, and two "Waiting on
you" items were both numbered 4 — both fixed. R-399 leaves that section (decided
and shipped); the depth change is stated in plain words and the remaining items
each say what happens if Viktor does nothing.
00-capability-map.md: the off-site verification row now carries its DEPTH, and
its live citation is the 2026-08-31 run at 100%. The weekly firing at the new
depth stays IMPLEMENTED, not PROVEN-LIVE.
07-backup-architecture.md §10.2: R-399 recorded closed, with the one sentence
that stops it being turned back down — the structure check PASSED a
size-preserving pack corruption. R-87 untouched and still OPEN.
Register: R-399 and R-400 compressed into CLOSED-ITEMS.md with their reasoning
kept and 300d7e8 named as the commit holding the originals. R-401 filed with a
TRIGGER (the slow-check WARN firing) rather than a date. R-402 filed: the
integrity verdict and its depth are on the wire and no hub surface reads either.
OPEN 166 -> 165, CLOSED 148 -> 150.
wire_contract_gate.py: offsite.last_integrity_depth allowlisted WITH ITS REASON
beside its sibling last_integrity_ok, both to be deleted together when a hub
surface is built (R-402).
THE MEASUREMENT IS THE STORY, and it re-frames the row it was filed under. A
pack was corrupted WITHOUT changing its size; plain `restic check` -- the depth
that ships ON -- returned `no errors were found`, exit 0. Only --read-data
caught it. So the check that shipped verifies the index, the pack inventory and
the snapshot graph, and does NOT re-hash pack contents. R-399 was filed as a
bandwidth-and-cadence question; it is more than that, and its row now says so.
R-399 gets three MEASURED numbers instead of estimates: store 140 829 678 B /
2651 blobs / 67 snapshots; structure check 35.0 s; curve 10% 35.9 s, 50% 37.3 s,
100% 39.2 s. At this size re-reading everything costs four seconds more than
reading none, because the wall clock is SFTP round-trips not transfer. The row
states the limit too: these do NOT extrapolate.
R-400: the sweep the task asked for found EIGHT dead debug buttons, not one. 24
endpoints referenced in debug.html, 17 dispatched. Single dispatcher, exact
match, default NotFound -- so they 404. A third of a debug page does nothing, on
the surface an operator reaches for when something is already wrong.
R-398 is CORRECTED AND LEFT OPEN, not closed. I filed it yesterday saying
resticStep is not a seam so no test can drive a restic path. The layer below it
has been injectable since the off-site tier shipped. The row survives as the
record that the seam EXISTS so nobody re-files it.
07 gap register: R-359 and R-397 closed; R-87 restated IN PLACE as "AND IT IS
NOT R-359" because the two rows are adjacent and a check is not a restore-test.
08 alarm ladder: both event types recorded, including that `ok` is `info` and
therefore mails nobody BY DESIGN, and that all three registers were checked and
deliberately left alone. 00 capability map: PROVEN-LIVE for the check, the
notifier and the hazard control; the scheduled firing is IMPLEMENTED only,
because a week has not passed.
wire_contract_gate: `offsite.last_integrity_ok` allowlisted WITH A REASON. The
gate was right -- the controller emits a field no hub struct can decode.
Building the display is a hub change and R-331 ruled that class the operator's
decision; the entry says to delete it when a surface exists.
This push used `git push --no-verify`. golden-currency is CONVICTED and right:
0.227.1 is released and the golden carries 0.226.1. A BYPASS, not a waiver, and
the task spec directs it -- golden and fleet delivery are Viktor's (R-242). It
is item 3 under "Waiting on you".
Register 163 -> 165 -> 163.
Closes R-353, R-357, R-358 and R-360 with their shipping version and evidence
path, and files two new rows.
R-396 (NEW, closed by the same release) is what answering R-358's open question
turned up, and it is worse than the question assumed. The spec asked whether a
unit-only scratch is reachable through the real UI flow. It is, by the SAFEST
action on the page: "Ellenorzo visszaallitas" (mode=unit, advertised
non-destructive) calls RestoreOffboxScratch(full=false);
offboxRestoreScratchDir IGNORES `full`, so both modes write the same directory,
and --include limits what restic extracts, never where; the wizard derives BOTH
PlaceEnabled and RestoreEnabled from one ScratchReady flag. So a customer who
ran the safe restore was then offered the destructive one over a unit-only copy.
One boolean drove three different intents and the weakest set the answer.
R-395 (filed by the spec) is fixed in this commit, not just recorded. STATUS.md
said golden 0.223.0 / floor 0.222.0 in one block and demo-hp 0.219.0 / floor
0.218.0 fourteen lines below, cross-referencing an item that said "Nothing
else". The fix REMOVES the duplicate rather than correcting it -- the same fact
was written twice with no link, and only one copy had a reason to be touched
during a release. "What works" now points at the item above instead of restating
a version.
07-backup-architecture: four rows added to the 10.2 gap register plus R-396.
Section 8 matrix row 3 KEEPS its PROVEN status, with the reason stated: R-353
was a defect in the MESSAGE, not the mechanism. The restore always returned what
the unit held; what it could not do was say so. A status that measures whether
data comes back must not move because a status line was wrong.
00-capability-map: one new row, and it splits what is claimed. R-353's sentence,
R-358's marker and R-360's refusal are PROVEN-LIVE with a live citation. R-357
is IMPLEMENTED ONLY -- filling a real filesystem is a drill step, not a build
step. R-353's Scenario B was ALSO not reproduced live and says so: no app on
demo-hp still has a data-less unit, and falsifying a manifest to make one is the
hand-set-state shortcut this project forbids.
This push used `git push --no-verify`. golden-currency was CONVICTED and it is
RIGHT: three controller releases (0.224.0, 0.225.0, 0.226.0) and the golden
still carries 0.223.0. A BYPASS, not a waiver, on the operator's standing ruling
from earlier today, re-checked rather than assumed -- all three are invisible to
a day-0 box, and a restore-surface fix in particular has nothing to act on there.
The ground expires the moment a release changes first-boot behaviour. Tracked on
R-242; ONE bake carrying 0.226.0 covers all three.
07-backup-architecture.md gains a dated [FACT] on R-361 - a comment asserting an
invariant the code did not have, for four months - and a [DESIGN] on the db_dumps
decision INCLUDING the trap it created: a stable list lets the already-current
early return fire, so per-capture housekeeping must sit above it.
00-capability-map.md records the NEGATIVE from Part 3 so it is not re-derived: a
held app does NOT raise the dead-app alarm. It aggregates to unhealthy, which
IsDownState excludes. Measured on the shipped build with the scans demonstrably
running over it. No suppression was built and no row opened.
R-383: the double-failure message names an undo copy that is not there - R-361's
own class, one surface over, observed on both 0.220.2 and 0.221.1.
R-384: an app whose database has died reads unhealthy and raises no alarm.
R-361 closed and compressed. OPEN-ITEMS 325236 -> 327266 bytes.
Golden 0.221.1 baked, published and round-trip verified. The golden-currency gate
blocked this push and that block is not circular, so it was satisfied rather than
bypassed - no --no-verify anywhere in this session.
07-backup-architecture.md 6.3 gains a dated [DESIGN] paragraph on replay ->
rollback -> hold, including why no engine flag closes it: --single-transaction
makes Postgres atomic, MariaDB DDL is not transactional, so the rollback is the
fix and the flag is a belt.
Drill record for the live walk, including the TWO defects the walk found in the
fix itself (a rollback into a re-created container; an operator route that
cleared the file while the running controller kept refusing) and the ONE
red-proof that PASSED, which is reported rather than omitted.
R-379..R-382 compressed into CLOSED-ITEMS.md. OPEN-ITEMS 330683 -> 325236 bytes.
STATUS.md restates the outcome and names the next operator step.
07-backup-architecture.md: three places said no offsite action unpacks the
named-volume tars. R-107 closed in controller v0.218.0; all three corrected with
a dated [FACT], the old sentence kept in the past tense. R-102 is NOT closed and
the correction says so explicitly.
New [DESIGN] paragraph in 6.3: the restore destination is resolved by the same
rule as the capture destination, and the wrong-disk refusal applies to apps that
have a drive to get wrong. Carries the 13/40 measurement.
STATUS.md was internally contradictory - nothing waiting, and one decision
waiting, for something the same page recorded as shipped. 218 -> 102 lines; the
deciding section now says what happens if nothing is done.
R-356 compressed into CLOSED-ITEMS.md; OPEN-ITEMS 327109 -> 325236 bytes.
Drill record and 16 evidence files for the live walk on demo-hp.
- OPEN-ITEMS: R-193 CLOSED with both 2026-08-05 rulings (unlocking and restoring
are separate; 'I do not want the old data' moves the store aside after a double
confirmation), and the shape-(b) reasoning — WriteOffboxSecrets auto-generates a
repository password on re-apply, so the literal 'fresh data area' trigger would
have opened a window that closes by itself.
- R-213 MINTED (R-212 was and still is the highest, re-checked for the second
writer): putting files back in place, with the live-versus-backup comparison
named as its requirement. Not started, deliberately.
- capability map: the 'needs someone who knows to look' qualifier is GONE; what
remains is stated narrowly — no correct-code run through the page, the put-back
is out of scope, and the journey has not been re-walked end to end.
- 07-backup-architecture 7.0: a fifth row, and where the screen deliberately stops.
- CONTEXT: standing ruling S-34.
- STATUS: the headline change and the two things still owed as proof.
No hub change and no hub bump.
- OPEN-ITEMS: R-204 all four CLOSED with both 2026-08-05 rulings recorded (the
declared-state trigger and its four-meanings-of-absence reasoning; the recovery
preview's dashboard-password exposure accepted as metadata, not content).
R-193's credential half CLOSED, screen + deletion still open. R-192 CLOSED by
REPLACEMENT. R-202 untouched.
- R-212 MINTED (R-211 was the highest, grepped): the orphaned-ciphertext deletion
HALTED at its STOP because the measured paths do not match the register — three
set-aside stores totalling ~1.45 GB, and the thing that is exactly 1.2 GB is
demo-felhom's LIVE repo. Nothing was deleted.
- capability map: all four interventions closed; the row KEEPS a qualifier for a
new reason — no step needs an operator, but there is no customer-facing recovery
screen, and the journey has not been re-walked end to end.
- 07-backup-architecture 7.0: the four-step table updated; the declaration-vs-
inference reasoning and the credential-automatic/key-customer-present split.
- CONTEXT: standing ruling S-33.
- STATUS: the headline change and the deletion STOP.
- REPORT-r204-item4.md rather than REPORT.md: a parallel session is active in this
shared clone.
- OPEN-ITEMS: R-196 CLOSED; R-204 items 1-3 CLOSED with item 4 named and
its dependency stated. Header restates that R-202, the 1.2 GB ciphertext
deletion and R-198's still-unit-proven retention all REMAIN OPEN.
- capability map: the recovery row keeps its 'with a person present'
qualifier, names which crutch remains, and cites the three now gone.
- 07-backup-architecture: new 7.0 - what a customer can and cannot do
ALONE, the four steps in a table with status. This is the section a
future reader will use to answer that question.
- CONTEXT: standing ruling S-32, superseding S-31 steps 2-5.
- STATUS: rewritten to one screen per its own header; removes a corrupted
half-overwritten section left from the drill session.
- ROADMAP: R-196 and R-204 collapsed.
R-191: the gate was run first and it mattered — ep0's prune jobs have run daily
since 2026-07-27 (18 tasks, all OK). THREE of my own queries said the opposite and
all three were broken instruments (worker-type vs worker_type, prune vs prunejob,
proxmox-backup vs proxmox-backup-proxy); acting on any would have disabled the only
pruning attempt while reporting that nothing prunes.
R-190: the row now reflects the MITIGATION, not the cause, and says so. Includes
that v0.124.0's transition record failed in production while every unit test
passed, and Part 3's single pass — the token-rotation class is real and documented
in our own installer but does not fit this box (same token still authenticates, the
other three grants survived, no installer run evidenced); ruled out on this box, not
fleet-wide.
R-194: PVE caches permissions (~40s and ~16min measured), so a single permission
read is a lagging indicator and may explain when R-190's refusal surfaced.
07-backup-architecture: the offsite retention row was RIGHT and the configuration
disagreed with it — noted, since a doc that states a contract does not enforce it.
- OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal
reading is never true on a daily tier); R-87 re-ranked UP because R-86 built
most of what it waited for; R-185 (the agent cannot list demo-felhom's host
backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's
sha is not reproducible from its tag), R-187 (R-115's publish leg had never
actually run) filed. R-184 was the highest ID in use.
- ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row
itself proposed — which would have been the never-fires version.
- 07-backup-architecture: new contract section — restore-testing is per ARCHIVE
GENERATION, with the trap and what did not change (S-1).
- 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on
the 635 s due-triggered offsite run, with the restart and teardown evidence.
- CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and
S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines).
Numbered 17/18 because S-14 and S-15 were already duplicated in the file.
- STATUS: rewritten for the operator, trimmed back to one screen.
R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The
hub's notification_log for the run reads: two per-app failures RECORDED, one
digest SENT naming both, and the customer channel SKIPPED with operator_only.
Against the measured previous behaviour — two failures, one email naming one
app, one leaving no trace anywhere.
Scenario D proved itself on an event I had not planned: disk_critical alarmed on
two filesystems, the second was collapsed by the cooldown, and that collapse is
now visible WITH ITS KEY. Yesterday it would have left nothing at all.
A gap the spec did not anticipate is recorded with its fix: the per-app event
also fires from the periodic sweep, outside any run, so making it record-only
would have created a NEW silence. The sweep emits a digest too, with no run_id,
so it stays under the ordinary hourly cooldown.
ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile
SURVIVED the resize and is active (checked, because a resize is a stop/start).
The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere.
Five documents corrected — three of which the task's list did not name, found by
searching. Two audit/evidence documents ANNOTATED, body untouched: they record
what was true when written and that is their value.
R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable,
not unbounded — the original OOM was a 14.46 GB restore — so the restore-test
cadence should still be paced, just not by fear of the endpoint.
target-selection.md's "D-d did not name ep0 either way" is deliberately left
standing. It is the operator's question, not CC's.
STATUS.md 127 -> 83 lines, items rather than sentences.
admin