36 Commits

Author SHA1 Message Date
admin c917251eeb docs: P6 EXECUTED — containerd store moved to SSD2; reboot deferred (R-209, R-209a)
gates / gates (push) Successful in 7s
Operator ruled "proceed" after reading the pre-analysis; the storageReserved
condition CC made the move conditional on was applied with it. The pre-analysis
is retained verbatim as §11-bis so the ruling can be audited against what it
was given.

MOVED WITH ZERO LOSS, verified BEFORE the original was touched on four
independent observables: 550,891 = 550,891 filesystem entries; 448 = 448
trusted.overlay xattrs (load-bearing — overlayfs stacking rides them, which is
why rsync -X is not optional); 37,243 = 37,243 hardlinks; byte-identical
meta.db sha256. Verified again after: identical image/tag/volume ID SETS,
cache 2.782 GB / 38 records, ceiling still 30 GiB, pg 4 DBs / 31 tables /
175,135,767 B, redis DBSIZE 2437. End-to-end proof was a real build on the
relocated store, rc=0 — not an inventory count.

k3s was never at risk, and that was established BEFORE stopping anything: it
runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and
~160 pods stayed up. Only the two jarr-* dev containers were affected.

TRAP FOUND WHILE PROVING THE GUARD, and it is the reusable part:
RequiresMountsFor on a path with NO mount unit is a SILENT NO-OP — containerd
started normally against an absent-but-unmounted path. A typo'd guard buys
nothing and says nothing: the built-but-never-wired shape again. The guard was
therefore verified positively at the unit level (Requires= AND
After=mnt-ssd_2.mount on both units), and refusal proven with a genuinely
absent DEVICE via a temporary synthetic .mount unit — /mnt/ssd_2 hosts 12 live
Longhorn replicas and must never be unmounted, and editing fstab on a
production host risks emergency mode at boot. Result:
"Job containerd.service/start failed with result 'dependency'", is-active
inactive. It refused rather than creating an empty store on the root disk.

storageReserved on SSD2 0 -> 80 GB; still Schedulable=True at 76.34%.

NOT PROVEN, AND NOT GLOSSED (R-209a): the operator ruled DO NOT REBOOT, so the
move has not survived a reboot — which is this task's own standard for
validated. The MECHANISM is proven; the CONSEQUENCE (that a real boot mounts
/mnt/ssd_2 before containerd starts, in this host's actual ordering) is not,
and mount-ordering reasoning is exactly the class this project has been burned
by. So: the pre-move tree stays MOVED ASIDE, not deleted (34.3 GB, which is why
/ sits at 54% and not lower), and felhom-store-postboot-check.service — oneshot,
enabled, dry-run PASS at install — validates it automatically at the next boot,
asserting positively that /var/lib/containerd does NOT exist. Read
/var/log/felhom-store-postboot-check.log after the next boot; on PASS the
rollback copy can go and ~34 GB returns.

Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).
2026-08-05 10:27:40 +02:00
admin ee9d9bf203 docs: SPIKE — DooPlex build-cache containment; R-205..R-211 (2026-08-05)
gates / gates (push) Successful in 7s
Spike output only; no production Go code. The one shipped change rides in
homelab-manifests 6808a4b (R-205, the monitoring rule).

VERDICT: mechanism confirmed, with one correction and one refutation.

- CONFIRMED: builder.gc IS honoured under the containerd worker and DOES evict.
  Proven by naming a 440 MB `go mod download` record present at build N and
  absent by N+2 — not by absence of an error.
- CORRECTED: honoured ONLY in the `policy` array form. The flat form is
  SILENTLY ignored — daemon starts, logs nothing, keeps its defaults.
  `dockerd --validate` returned "configuration OK" for a bogus key AND for a
  config that then crashed the daemon. The oracle is `docker buildx inspect`.
- REFUTED: Docker's `data-root` would NOT move the cache — it moves 0.62 GB.
  The 181.4 GB belongs to the system containerd (`root` in
  /etc/containerd/config.toml).

P3 (operator-approved) executed: prune claimed 156.9 GB, the filesystem
returned 150.35 GB (the 6.5 GB gap is layers shared with images), SYNCHRONOUSLY
— / went 86% -> 53% used, and Longhorn's default disk went
Schedulable=False (DiskPressure) -> Schedulable=True (18.85% -> 50.32%).

P7 root-caused the largest item and it is NOT the cap: all 208 `go mod download`
records had Usage count 1. Isolated by controlled builds — same VERSION build-arg
-> CACHED, new VERSION -> executed, byte-identical tree. `ARG VERSION`/`ARG
GIT_COMMIT` sit ABOVE the module-download step, and a RUN's cache key includes
the stage environment. Both Dockerfiles have it. One line each to fix -> R-208.

P6 NOT EXECUTED — stops at the operator, as specified. Pre-analysis: the move is
safe as measured (+38.8 pp above the 25% floor) but SSD2 is the only Longhorn
disk with storageReserved=0 and is overcommitted 6.9x; at full inflation the move
lands 12 pp BELOW the floor. The prune removed the move's urgency, so CC
recommends against it unless ~80 GB is reserved on SSD2 -> R-209.

Register: R-205 (CLOSED, shipped), R-206 (Ansible: cap + prune + narrowed Docker
ban), R-207 (DRY_RUN guard), R-208 (ARG ordering), R-209/R-210 (operator),
R-211 (Prometheus has no config-reloader — rules changes have never applied
until something restarted the pod; found while verifying R-205).

Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).
2026-08-05 10:02:46 +02:00
admin 0dbd954fec docs: R-196 closed, R-204 items 1-3 closed, item 4 open (R-193)
gates / gates (push) Successful in 7s
- OPEN-ITEMS: R-196 CLOSED; R-204 items 1-3 CLOSED with item 4 named and
  its dependency stated. Header restates that R-202, the 1.2 GB ciphertext
  deletion and R-198's still-unit-proven retention all REMAIN OPEN.
- capability map: the recovery row keeps its 'with a person present'
  qualifier, names which crutch remains, and cites the three now gone.
- 07-backup-architecture: new 7.0 - what a customer can and cannot do
  ALONE, the four steps in a table with status. This is the section a
  future reader will use to answer that question.
- CONTEXT: standing ruling S-32, superseding S-31 steps 2-5.
- STATUS: rewritten to one screen per its own header; removes a corrupted
  half-overwritten section left from the drill session.
- ROADMAP: R-196 and R-204 collapsed.
2026-08-05 07:37:35 +02:00
admin 975a690fbe manifests: hub 0.95.0 (R-196 / R-204 item 2)
gates / gates (push) Successful in 7s
2026-08-05 07:19:57 +02:00
admin 5c7d67102d CHANGELOG: hub v0.95.0 (R-196 / R-204 item 2)
gates / gates (push) Successful in 7s
2026-08-05 07:17:37 +02:00
admin d1a8edb332 R-196 / R-204 item 2: a re-issue no longer marks a healthy escrow stale (hub v0.95.0)
ReissueCredentials marked the escrow stale on every re-issue, on precautionary
grounds — the box's re-apply MIGHT mint a fresh repository password. It usually
does not. A stale flag withholds restic_pw_sha256 from the ACK, which stops the
controller's auto-confirm, which leaves EscrowState pending, which makes
OffboxRunnable false: every off-site backup refused on a box whose key was never
in doubt — and the customer told to re-run the one ceremony that would have
superseded the key just recovered.

The case it guessed at is measured elsewhere: the controller's Scenario-F
re-check compares the sealed hash against the live repo password on every ACK
(and the mark was BLINDING it by emptying that hash), and R-197's
offsite_repo_key_changed fires on a proven difference across a supersession.

offsite_reissued is unchanged. MarkEscrowStale is kept without a caller so a
future EVIDENTIAL writer has the mechanism, with a test pinning it live.
TestReissue_InvalidatesEscrow is replaced by its exact inverse.
2026-08-05 07:17:29 +02:00
admin 2a7ac03c47 R-201 PASSED: a customer's file survived a machine rebuild and came back byte-identical
gates / gates (push) Successful in 7s
2026-08-04 23:18:47 +02:00
admin b228fd102d R-201 night run: the off-site key IS recoverable after a real rebuild (proven); the verdict is blocked by R-204
gates / gates (push) Successful in 6s
2026-08-04 22:33:01 +02:00
admin 24acc8d816 REPORT: R-203 shipped and proven live; R-201 unblocked
gates / gates (push) Successful in 8s
2026-08-04 18:53:13 +02:00
admin 73fb595e38 R-203 shipped: the app and its backup agree, and 'ok' means it — R-201 unblocked
gates / gates (push) Successful in 7s
2026-08-04 18:52:14 +02:00
admin a0c4b607a6 drill R-201: prepared and HALTED before the wipe — a mandatory data directory was missing from the off-site snapshot while the run said ok (R-203)
gates / gates (push) Successful in 8s
2026-08-04 15:00:06 +02:00
admin 846253baa8 docs: the offsite key is proven recoverable (R-199 closed, R-200 half, R-201 scoped)
gates / gates (push) Successful in 8s
2026-08-04 13:55:25 +02:00
admin dd089265e8 manifests: hub 0.93.0 -> 0.94.0 (R-199 box-authenticated escrow retrieval)
gates / gates (push) Successful in 8s
2026-08-04 13:40:26 +02:00
admin 435f4a5229 hub v0.94.0: a box can fetch its own sealed recovery package (R-199 link 6)
gates / gates (push) Successful in 7s
Link 6 of the recovery chain had no client. The hub has served the identity blob since
slice 10D from handleReEnroll / handleGetRestoreDirective, gated on operator-armed recovery
mode and the global key -- and nothing in the agent, the hub UI, any script or any runbook
ever called either. The only documented retrieval was sqlite3 writefile() by hand on a
kubectl cp-ed database.

GET /api/v1/hosts/{host_id}/escrow is the box-authenticated mirror of the PUT that put the
blob there. Self-scoped (a per-host key reads only its own; global may read any). A host with
no bundle gets 200 {present:false} -- a 404 is indistinguishable from an unknown host and a
bare empty 200 from a zero-length blob.

THE TRADE IS RECORDED IN THE HANDLER, not inferred: obtaining the blob used to require the
operator to arm recovery mode; now whoever controls a rebuilt box can obtain it with that
box's own credential. They still cannot open it -- the hub has never held R and a wrong code
fails closed at age's scrypt KDF. The mitigation is that every retrieval raises
escrow_blob_served (warning, operator-only), recorded before the bytes leave.

escrowSelfServiceRetrieval is the single decision point: flip it to false and the endpoint
additionally requires recovery mode, changing nothing else.

The operator-driven DR path is untouched, pinned by a test. Red-proofs observed: removing the
ownership check serves host B's blob to host A; removing the record makes it silent.
2026-08-04 13:39:27 +02:00
admin 9f31956201 REPORT: hub v0.93.0 (R-198 retention fix, R-197 changed-key signal, R-192 honesty half, R-196 five comments, R-202 filed)
gates / gates (push) Successful in 7s
2026-08-04 13:00:38 +02:00
admin 40687b0921 manifests: hub 0.92.0 -> 0.93.0 (R-198 escrow retention + R-197/R-192 honesty pass)
gates / gates (push) Successful in 7s
2026-08-04 12:58:04 +02:00
admin 91cabdde1b hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196)
gates / gates (push) Successful in 7s
R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.

  - identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
    routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
  - ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
  - CountCurrentEscrowWithIdentity is the census of who the fix protects.
  - Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
  - Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
    the pre-existing retention test stayed green for two months asserting the mechanism.

R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.

R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.

R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.

Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
2026-08-04 12:56:58 +02:00
admin 3f2b7bc023 RECON: trace the offsite DR chain link by link — it does not join up (R-198..R-201)
gates / gates (push) Successful in 7s
Read-only recon of the escrow -> recovery chain, from a dead node to an open
repository. No production code, no build, no version bump.

Headline: the hub's superseded-escrow retention does NOT retain the offsite
repository password. host_escrow_superseded has no identity_blob column and
demoteCurrentEscrowTx copies only the K-escrow blob, so what survives a
supersession is the PBS datastore key, not the restic repo password. The next
escrow ceremony -- which the system tells a rebuilt box's customer to run --
destroys the last copy. Both demo boxes crossed that line on 2026-08-04.

Also established:
- the hub's blob-serving endpoints (re-enroll / restore-directive) have zero
  callers anywhere: agent, hub UI, scripts, runbooks (R-199)
- POST /backup/offbox/inject-password is routed and handled but no template
  contains the form (R-200)
- nothing in the recovery path has ever been exercised; the one live
  round-trip proof (2026-06-10) predates the ResticRepoPassword field (R-201)
- a fail-closed mint refusal IS implementable: the report ACK already carries
  escrow{identity_blob_present, restic_pw_sha256} and the controller discards
  it whenever no offbox target exists

Corrections: yesterday's spike annotated (candidate (b) overturned in part --
unattended recovery is impossible, customer-present is not); capability-map
retention claim struck through and replaced with what the code does.

Deliverable: documentation/audits/RECON-offsite-dr-chain-2026-08-04.md
Register: new R-198..R-201; R-193 and R-192 updated; STATUS.md refreshed.
2026-08-04 12:16:04 +02:00
admin d26f49ad68 REPORT: record CI run 74 (docs commit) green
gates / gates (push) Successful in 7s
2026-08-04 11:12:54 +02:00
admin b7c5cfc061 docs: R-193 spike report, operator STATUS, and the ROADMAP R-1 scope correction (F3 covers HOST re-enroll, not a guest rebuild)
gates / gates (push) Successful in 8s
2026-08-04 11:12:08 +02:00
admin f581ac1349 manifests: hub 0.91.1 -> 0.92.0 (R-195 phantom-customer alarm)
gates / gates (push) Successful in 6s
2026-08-04 11:05:41 +02:00
admin 7fff45d688 R-195: a customer with no machine ever bound does not alarm (hub v0.92.0) + R-193/R-192 spike
gates / gates (push) Successful in 7s
Part 4 (ships): `david` — a prospective customer with hosts=0, host_deletions=0,
reports=0 — e-mailed an expected_dbdump_missed ERROR at 03:00 UTC three mornings
running. The existing down-skip could never cover it: it reads the staleness
checker's state, which is seeded from a query over the `reports` table, so a
customer that never reported has no state at all and GetState() returns "" rather
than "down". store.HasEverBoundHost (hosts row OR host_deletions tombstone) is
consulted once per customer at the top of the deadline loop. The discriminator is
"was a host EVER bound", never "has a report arrived" — a box installed and never
heard from is a real fault and keeps alarming. Fail-OPEN on a read error. Red-proof
observed: removing the guard fails with `got [expected_dbdump_missed]`, verbatim the
event david sent.

Parts 0-3 (spike, NO production code for R-193/R-192):
audits/SPIKE-offsite-credential-recovery-2026-08-04.md establishes that the one-shot
provider password is the RECOVERABLE secret and the restic repository password is the
irreplaceable one — and that a guest rebuild mints a fresh one, orphaning the previous
off-site history. Measured without touching a box, by comparing
host_escrow.restic_pw_sha256 against host_escrow_superseded: BOTH demo boxes changed
(demo-hp 15 snapshots / 40.9 MB, demo-felhom 36 snapshots / 1.14 GB). demo-felhom's
"lucky" 76-second recovery restored delivery and not the repository, silently, for 13h.
ReissueCredentials does NOT rotate the restic password (R-39's record and two hub
comments are wrong -> R-196); candidate (b) is not implementable against a
zero-knowledge escrow; candidate (a) already exists as F3 and is wired to the wrong
event. Ends in ranked options and an unanswered question for the operator.

R-195 SHIPPED; R-196 + R-197 filed; R-192 + R-193 updated, neither closed.
2026-08-04 11:04:39 +02:00
admin f456835bbc REPORT/R-190: the alert pair closed — one loss, one email, one recovery
gates / gates (push) Successful in 8s
The 20-minute latch expired at 10:20:29 and the hub logged degraded -> ok
(agent_capability_recovered) at 10:30:40. Final state on both boxes: agent
0.124.1, two ACL rows on /storage/felhom-backup.
2026-08-04 10:31:46 +02:00
admin e06920d930 R-188: the reorder moved the race rather than removing it (observed on the v0.124.0 release)
gates / gates (push) Successful in 7s
CI run 28 failed and run 29 passed on the same sha. The tag was created locally at
07:38:36Z; run 28 started 07:38:31Z (commit push) and run 29 at 07:38:41Z (tag
push). The old race was 'tag before package'; the new one is 'package before its
tag is pushed' — the window the converse probe exists to detect, and it did.

Not confirmed from the run log (the endpoint 404s), so recorded as the
timing-consistent explanation rather than a proven one. The gate passes locally
with all 7 versions installable and none untagged. Both obvious closures are the
'make the gate blind for a window' shape this row refused, so the options are to
accept a seconds-wide window or to make releases quiet to CI another way.
2026-08-04 10:08:10 +02:00
admin 813239d8d3 REPORT: R-191 closed, R-190 mitigation shipped, and the three broken instruments
gates / gates (push) Successful in 8s
Part 0's gate PASSED — ep0 prunes both namespaces daily since 2026-07-27 (18
tasks, all OK) — but three of my own queries said the opposite and all three were
broken instruments. Acting on them would have disabled the only pruning attempt
while filing a finding that nothing prunes.

Also records that v0.124.0's transition record failed in production with a green
test suite, that two red-proofs did not fail on the first attempt (one could not
compile, one asserted a helper rather than the path), and that two hollow tests
were caught in one file.
2026-08-04 10:06:12 +02:00
admin a684a81796 docs: R-191 closed, R-190 mitigation shipped (mechanism open), R-194 filed
gates / gates (push) Successful in 7s
R-191: the gate was run first and it mattered — ep0's prune jobs have run daily
since 2026-07-27 (18 tasks, all OK). THREE of my own queries said the opposite and
all three were broken instruments (worker-type vs worker_type, prune vs prunejob,
proxmox-backup vs proxmox-backup-proxy); acting on any would have disabled the only
pruning attempt while reporting that nothing prunes.

R-190: the row now reflects the MITIGATION, not the cause, and says so. Includes
that v0.124.0's transition record failed in production while every unit test
passed, and Part 3's single pass — the token-rotation class is real and documented
in our own installer but does not fit this box (same token still authenticates, the
other three grants survived, no installer run evidenced); ruled out on this box, not
fleet-wide.

R-194: PVE caches permissions (~40s and ~16min measured), so a single permission
read is a lagging indicator and may explain when R-190's refusal surfaced.

07-backup-architecture: the offsite retention row was RIGHT and the configuration
disagreed with it — noted, since a doc that states a contract does not enforce it.
2026-08-04 10:04:25 +02:00
admin 71662336aa manifests: /scripts/ syncs installer-v1.24.0 -> installer-v1.25.0 (R-191)
gates / gates (push) Successful in 7s
2026-08-04 09:46:21 +02:00
admin 51871a7ea6 installer 1.25.0: the off-site tier stops asking to prune (R-191)
gates / gates (push) Successful in 8s
Every weekly off-site run uploaded successfully and then failed the job on a prune
the box's token is deliberately refused — R-89 moved off-site pruning server-side
to ep0 and box tokens stay write-only. The 2026-07-26 'two weeks' ruling was not
reversed; where it is enforced moved, and keep_last: 2 did not follow. Now 0,
which the agent's existing guard already reads as 'never prune from the box'.

Verified read-only on ep0 before changing it: both namespaces have a prune job at
03:30 keep-last 2 that has run every day since 2026-07-27 — 18 tasks, all OK, the
newest keeping exactly two. Without that check this would have traded a weekly
false alarm for unbounded growth.

A gate asserts the offsite tier carries no client-side prune. The local tier is
untouched.
2026-08-04 09:46:09 +02:00
admin 33f0ab2f33 R-193: demo-hp offsite restored on the box; escrow correction recorded
gates / gates (push) Successful in 7s
Re-issue through the designed endpoint: fresh password staged 07:11:51, settle-gate
GO, consumed 15s later at 07:12:06, offsite configured at 07:12:09 against the SAME
sub-account (275124) and repo path. Both boxes now escrowed.

CORRECTION to this session's own reading: the escrow did NOT recover by itself.
escrow_state went pending -> escrowed 15s after the apply and I inferred automation;
the operator had run the ceremony. It needed a human on both boxes — demo-hp at
07:16:02, demo-felhom at 07:20:28 (its escrow had been pending since its own rebuild
the day before). A 15-second state change is not evidence of automation, which is the
same class as reading an absent log line as success.

Snapshot continuity is NOT established: both boxes report snapshot_count 0, but the
run-history keys are ABSENT rather than zeroed — a controller that has never run an
offbox backup, not an empty repo. The 04:15 run decides it.
2026-08-04 09:23:08 +02:00
admin 2584dfb938 R-193: a guest rebuild silently drops the offsite tier; R-192 cause established
gates / gates (push) Successful in 8s
Operator confirms no hub-side offsite config change, so the regression was not an
action. Evidence: demo-hp's controller went 0.187.0 -> 0.192.0 at 06:12:18 with a
new config hash and the agent re-keyed its leaf three minutes earlier — a guest
rebuild. The last pre-rebuild report shows the tier fully healthy: escrowed, last
success 02:16:39Z, 15 snapshots, 40.9 MB. No offsite object in the 108 reports
since.

Mechanism: the restic credential is delivered once. demo-hp consumed its secret on
2026-07-23; the rebuilt controller has no copy and no way to request another.
demo-felhom survived the SAME rebuild only because its secret was still unconsumed
— it consumed it four seconds after its config hash changed and was reporting
offsite again 76 seconds later. That difference was luck, not design.

Also sharpens R-192: the self-heal's guard refuses when any report since the
consume carried an offbox target, but that query reads the OLDEST 500 reports —
all of which predate the rebuild. Healthy history before a rebuild is not evidence
the credential still works, which is why the automation that exists for this case
declined to act.
2026-08-04 09:05:53 +02:00
admin 185661062a STATUS: back to one screen
gates / gates (push) Successful in 7s
2026-08-04 08:58:04 +02:00
admin 967ddf2f44 R-192: offsite_delivery_stuck states the opposite of what it measured
gates / gates (push) Successful in 7s
demo-hp has been emailing daily since 2026-08-03 06:12 UTC that '500 reports
since carry no offbox target — the credential is likely burned'. Measured against
the hub's own data: all 500 of those reports DO carry one. The credential was
consumed 2026-07-23 and applied successfully; the controller reported an offsite
object continuously until 2026-08-03 05:59:21, then it vanished at 06:12:19 and
has been absent for 108 reports. A regressed apply, not a burn.

Two defects. maybeEmitStuck builds its message from ReportsSinceConsume while
hardcoding 'carry no offbox target' and never consults OffsiteReportsSinceConsume,
so the recommended action targets a failure that did not occur. And maybeHeal
refuses SILENTLY on exactly that field ('regressed-apply shape -> operator's
call', a bare return), so there is a daily email with the wrong story, no heal,
and nothing saying why the heal declined.

The underlying condition is real: that box currently has no off-site copy of its
customer app data, since 08:12 CEST on 2026-08-03. What removed it is not
established.
2026-08-04 08:57:53 +02:00
admin d692c46db0 RUNBOOK: the host tier is restore-PROVEN unattended on both demo boxes
gates / gates (push) Successful in 7s
Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite
540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest,
booted, verified and destroyed itself; zero 990000 guests or volumes afterwards
and both local-lvm figures returned to their pre-run values.

Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for
the first time: never-proven sorted first, each box took its HOST tier, deferred
the offsite one, and picked it up on the next evaluation six hours later. The
host-tier proofs reached the hub through R-189's merge — demo-felhom's report
carries two tiers, and the local one can only have come from disk.

The capability map's optimistic half is cashed, with its scope stated: these two
boxes, not the fleet.

Surfaced and filed rather than fixed:
- R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with
  a reinstall, any logged pveum activity and any cluster-log entry ruled out.
- R-191: every weekly offsite backup uploads successfully and then fails the job
  on a prune the box is deliberately not allowed to do (R-89 moved it
  server-side; both boxes still arm keep_last=2).

Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the
write path (six, with the hub raising whole_guest_backup_failed at the first), and
my earlier "no restore_test_* events" was produced by grepping a 404 page.
2026-08-04 08:13:43 +02:00
admin b93ee06abc R-190 filed; two corrections to yesterday's R-185 record
gates / gates (push) Successful in 8s
CORRECTION 1 — the runbook annotation and the R-185 row both said the drift did
not surface as a 403 because writes go through a root path. That is WRONG.
demo-felhom's local-api backup jobs 403'd six times between 09:24 and 17:34 CEST
on exactly that storage and privilege, and the hub raised whole_guest_backup_failed
at the first with edge-triggering suppressing the rest. The impact was not only an
unreadable tier: the agent's own whole-guest backups to it were failing.

CORRECTION 2 — on this box the grant was LOST, not never issued. A vzdump by the
agent's token to that storage completed OK at 04:44:50 the same morning; the first
403 is 09:24:56. Ruled out by measurement: a host reinstall (uptime 12 days), any
pveum/ACL/user.cfg activity in syslog 04:00-10:00, any ACL entry in the cluster
log. Correlated but not established: guest 9201 was reprovisioned nine minutes
before the first failure.

R-190 files the unexplained disappearance, and notes that the new store-grant
probe detects the STATE but says nothing about the TRANSITION.
2026-08-03 20:11:09 +02:00
admin e3187c86d5 docs: R-185 closed — the silence as well as the grant
gates / gates (push) Successful in 8s
- OPEN-ITEMS: R-185 closed with the measurement, the corrected root cause (the
  installer's Scenario-F reuse arm, not PVE_STORAGES), and the live sequence.
  Records that demo-hp carried the same drift and was fixed too.
- capability map: the whole-guest row's HOST-tier half was OPTIMISTIC and now
  says so — that tier was not merely unproven, it was unprovable on both demo
  boxes, and every live proof cited was on the offsite tier.
- vzdump-target-move runbook: its item 5 predicted this; annotated (not
  rewritten) with what actually happened — the create arm did grant, the reuse
  arm did not, and it surfaced as a silent unreadable tier rather than the 403
  the item expected, because vzdump writes through a root path.
- CONTEXT: S-21 (an empty listing cannot distinguish forbidden from newborn; the
  measured trap that an ungranted path answers with INHERITED privileges) and
  S-22 (the Scenario-F arm must finish the job).
- STATUS: rewritten for the operator, back to one screen.
2026-08-03 19:02:25 +02:00
admin 311dc06c13 manifests: /scripts/ syncs installer-v1.23.0 -> installer-v1.24.0 (R-185)
gates / gates (push) Successful in 8s
Both refs — the git-sync sidecar and its init container. A fresh pod must not
serve a different installer from a running one.
2026-08-03 18:59:05 +02:00
44 changed files with 4997 additions and 284 deletions
+309
View File
@@ -17,6 +17,315 @@
## Standing rulings ## Standing rulings
**S-32 — THREE OF S-31's FOUR MANUAL STEPS ARE CLOSED (2026-08-05, R-204 items 13 / R-196).
controller v0.198.0 + hub v0.95.0. Read this BEFORE S-31 — it supersedes S-31's steps 25.**
- **Step 1 (the reset code) — CLOSED.** `web.effectiveClaimCode` now calls `settings.ReloadClaimCode()`
before applying the settings-vs-config precedence, so a code minted by the SEPARATE
`--print-reset-code` process is visible without a restart. **The precedence rule is unchanged**
the defect was the freshness of the settings value, not which source wins. It is a READ-THROUGH on
the claim path only, deliberately **not** a watcher, a signal handler or a TTL: a TTL would leave a
window in which a superseded code still works, which is worse than the bug. `effectiveClaimCode`
now returns an error and **every caller fails closed** (`claimGateActive` keeps the gate UP; an
absent settings file is NOT an error — a pre-first-save box legitimately falls back to the
controller.yaml bake). **S-31's "restart the controller between minting and claiming" is obsolete.**
- **Steps 24 (the false staleness) — CLOSED.** `offsite.ReissueCredentials` no longer calls
`MarkEscrowStale` and no longer emits `escrow_stale`; `offsite_reissued` is untouched. **The
mechanism that made this expensive, recorded because it is not obvious:** `stale_at` makes
`GetEscrowStatusForCustomer` WITHHOLD `restic_pw_sha256` — which is the exact value the controller's
Scenario-F re-check (`report.EscrowAutoConfirmer.reconcileEscrowed`) compares against its live repo
password on every ACK. **The precautionary mark was blinding the measurement that would have told
the truth**, leaving the box able to report only the hash-LESS reason, which is false. Coverage
after removal: that per-ACK comparison (continuous, box-side) plus R-197's `offsite_repo_key_changed`
(edge-triggered, hub-side, at a supersession). **`MarkEscrowStale` is KEPT with no caller** — the
flag is still read by the ACK, the operator card and the PBS-DR view, and the right way to set it is
a future caller that has MEASURED a change; `TestEscrowStaleMechanism_StillWithholdsAndClears` stops
it decaying meanwhile. **The manual `/backup/offbox/confirm-escrow` workaround is no longer needed
after a Re-issue.**
- **Step 5 (the restore default) — CLOSED.** `restoreScratchOutcomeMsg` states, for `mode=unit`, what
came back, that the customer's own files did NOT, and the step that gets them; the wizard's intent
card 1 states its scope before the choice. **The `mode=full` size gate is untouched** and pinned as
unchanged. **The default stays `unit`:** all three wizard forms set `mode` explicitly, so changing
it would alter nothing the customer sees while silently changing a hand-crafted mode-less POST.
- **WHAT REMAINS is S-31's step 1-before-the-list: a rebuilt box still cannot obtain an off-site
credential unaided** (the one-time password was consumed by its predecessor), so an operator
Re-issue is still required. That is **R-193**, it needs an operator ruling, and it is why the
capability-map recovery row keeps its *"with a person present"* qualifier.
- **Live-validated on demo boxes, endpoint-level** (no browser on DooPlex): reset codes on
demo-felhom 9201 with `restarts=0` across the mints; the unit-restore outcome and the full-restore
size-gate reveal on demo-hp 9201, using `privatebin` so the drill's `calibre-web` scratch was not
touched. **The Part 2 change was NOT fired live on demo-hp** — a Re-issue there was out of scope.
**S-31 — THE DRILL PASSED: a customer's file survives a machine rebuild and comes back byte-identical.
The capability is proven; the customer JOURNEY is four undocumented manual steps (2026-08-04, R-201/R-204).**
**Proven on hardware, end to end:** demo-hp's controller data volume was destroyed and the sentinel
deleted from disk. The recovery code produced `8a9e33aa4da6…` (byte-identical to the pre-wipe on-disk
key AND the hub's independent record); it installed on the bare box; the **existing repository opened**
(`repo_state: null`, 3 snapshots, 42 026 B = the pre-wipe size — not a fresh history); and the customer
restore flow returned the sentinel at `643166269103a25c…`, **byte-identical**. `identity_blob` was
unchanged throughout and **no ceremony was run at any point** (superseded rows still 2).
**But it took four manual steps that are in no design document (R-204), all measured:**
1. a rebuilt controller cannot configure its off-site tier — the one-time password was consumed by its
predecessor (`no unconsumed offsite password`, R-193);
2. the Re-issue that fixes that sets `stale_at` **while `restic_pw_sha256` is unchanged** (R-196);
3. a stale escrow makes the hub withhold the hash from the ACK → `EscrowAutoConfirmer` can never flip
`pending → escrowed``OffboxRunnable` refuses every run;
4. it was cleared with the **manual** confirm (`/backup/offbox/confirm-escrow`), NOT a ceremony — a
ceremony would have superseded the identity blob and destroyed the recovered key;
5. and the restore's **default `mode=unit` returns the recovery unit, not the customer's files**
`mode=full` is what returns documents. A customer following the obvious path gets the wrong thing
and is told nothing.
**Before any of it, a rebuilt box is unclaimed** and the claim gate intercepts every controller
endpoint. **The local escape hatch does not work unaided:** `--print-reset-code` writes the new hash to
`settings.json` while the running controller keeps its old copy in memory, so the code never matches —
**restart the controller between minting and claiming.**
*Facts a future session needs:*
- **A guest rebuild in this fleet is a controller-DATA-VOLUME loss, not a guest reprovision.** The
2026-08-03 incident R-193 is filed against ran with guest 9201 up throughout — no `pct destroy`, no
`pct restore`, no `--selftest=provision`. Reproduce it that way.
- **A good snapshot is not durable against a later bad run on the same day.** `forget --keep-daily 7
--group-by host,tags` keeps one per tag per day; a later, worse snapshot evicts a good one.
- **Never run a ceremony while a recovery is in flight** — it supersedes the identity blob. Under
v0.93.0 the old blob is retained, but nothing serves a superseded blob back (R-199).
**S-30 — the R-201 drill was PREPARED and HALTED BEFORE THE WIPE (2026-08-04). Nothing was wiped.**
It stopped at step 4 because the sentinel file was **not in the off-site snapshot** while the run
reported `ok` (**R-203**): the deploy-time `${USERDATA_PATH}` root and the backup-time `NamespaceRoot`
disagree for an app whose `HDD_PATH` equals `system_data_path` — and the same compose used both roots.
Wiping would have destroyed the only copy of the sentinel and proven nothing.
*Established live on the way, all firsts, and all worth carrying:*
- **A rebuilt box's off-site run REFUSES** — `repo_state: orphaned`, the orphan card, and
`offbox_repo_orphaned` at the hub. It does **not** silently start a fresh history. Closes R-193's Q3.
- **The orphan reset works**: move-aside to `/home/felhom-repo.orphaned-20260804`, never delete.
- **demo-hp's pre-rebuild off-site history is permanently unrecoverable** — its key is in superseded
row id 3 with `identity_blob` NULL, superseded **four hours before** v0.93.0 fixed the retention.
- **Neither pre-existing off-site-toggled app has a restorable file leg** — both are named-volume-only,
which the tier tars but the customer restore never unpacks. Any drill needing a recoverable file
needs an app with a **mandatory userdata path**; `calibre-web` is now deployed on demo-hp as that
fixture, with the sentinel in place.
*Still not established, unchanged:* **no file has ever been restored from an off-site backup after a
wipe**, and Part 0's install path (controller v0.196.0) has never run against a live recovery.
**S-29 — a box may fetch its OWN sealed recovery blob with its OWN credential; the operator-driven DR
path is a separate thing and stays gated (2026-08-04, R-199; hub v0.94.0 + agent v0.125.0 + controller
v0.195.0).**
**The decision, so it is never re-litigated from scratch.** Self-service recovery needs the box to
obtain its own sealed package. The only retrieval that existed was operator-gated behind recovery mode
and the global key (`dr.go`), which cannot serve a customer-present flow. `GET /hosts/{id}/escrow` now
serves that ONE object to its authenticated owner.
- **Why it is safe:** the blob is opaque; the hub has never held R and has no decrypt path; a wrong
code fails closed at age's scrypt KDF (measured live — `exit status 1`, no plaintext, nothing
written); the caller already authenticates as this host for its report, desired state, WG
registration and PBS token, so this adds no new identity, only a new object.
- **What it changes, which is the whole of the trade:** obtaining the blob used to require the
OPERATOR to arm recovery mode. It now needs only that box's credential. **Fewer parties are
required.** The mitigation is that the capability is AUDITED, not silent: every retrieval raises
`escrow_blob_served` (warning, operator-only), written before the bytes leave.
- **The override is one constant.** `api.escrowSelfServiceRetrieval` — false re-imposes the
recovery-mode requirement on this endpoint and changes nothing else. Do not scatter the condition.
- **`handleReEnroll` / `handleGetRestoreDirective` are UNTOUCHED** and must stay so: they rotate the
host API key and serve the K-escrow and the directive as well. Different object, different gate.
Pinned by `TestEscrowGet_OperatorDRPathUnchanged`.
**Facts a future session needs before touching this area:**
- **The unseal lives in the AGENT and must stay there.** `age` is an agent runtime dependency
(`/usr/bin/age`, present on both demo hosts) and is deliberately absent from the controller image;
the blob is host-scoped and its only writer is the agent under the per-host key.
- **The agent returns ONE field.** Not the tunnel token, not the PBS token, not the WG key — the
controller is a trust tier down. Narrowing cost nothing and is not recoverable later.
- **R's rules:** in memory for one call, cleared on every path, never on disk, never in argv, never
logged, never echoed. The controller CLI reads it from STDIN for exactly that reason. Verified live:
0 log lines, 0 files, 0 leftover staging dirs, with a positive control proving the search worked.
- **The chain today: links 18 walked, 911 not.** The KEY comes back. Nothing installs it, reopens a
repository with it, or restores a file — R-200's remaining half and R-201.
**S-28 — the escrow retention now covers the OFF-SITE data key, and customer-present recovery is the
accepted design, which makes that retention load-bearing (2026-08-04, R-198/R-197; hub v0.93.0).**
**The contract changed, and this is the sentence to carry:** when an escrow blob supersedes another,
`demoteCurrentEscrowTx` retains **both** sealed keys — the K-escrow `blob` (PBS datastore key) **and**
`identity_blob` (the age-wrapped bundle carrying the restic **repository** password). Until v0.93.0 it
retained only the first, so the retention preserved the whole-guest key and destroyed the off-site data
key, and **the destroying act was the escrow ceremony a rebuilt box asks its customer to run.**
*Why this is load-bearing rather than tidy:* the operator ruled on 2026-08-04 that **candidate (c) is
refused — no repository password is retained on the Proxmox host.** With no host-retained copy, the
**customer-present recovery path is the only way back from a rebuild**, and it runs entirely through
the retained identity blob. A session that touches escrow custody is touching the single copy.
*Facts a future session needs before touching this area:*
- **The ordering is load-bearing and invisible from the routine.** `SaveHostDRBundle` writes
`identity_blob` **after** `SaveHostEscrow` returns (the escrow PUT, `api/handler.go`), so at demote
time `host_escrow` still holds the OLD identity blob. Invert that and the retained bytes become the
NEW blob filed under the OLD hash — recoverable-looking and wrong. Pinned by
`TestSaveHostEscrow_RetainsIdentityBlob`.
- **`demoteCurrentEscrowTx` is shared by re-escrow and host-delete.** Both have a test; fixing one
caller instead of the routine is a demonstrated red-proof, not a hypothetical.
- **Nothing was backfilled and nothing could be** — pre-v0.93.0 retained rows have no identity blob and
their source rows are already overwritten. Both demo boxes' pre-2026-08-04 repository passwords are
gone permanently, recovery code or not.
- **A retained key is not a recovery.** The hub's blob-serving endpoints have no client (R-199), the
injection seam has no form (R-200), and the path has never been exercised (R-201). v0.93.0 makes the
key survive; it does not hand it back.
- **`offsite_repo_key_changed`** (R-197, warning, operator-only, edge-triggered on a supersession where
both hashes are known and differ) is the evidential signal that a box's off-site data key moved. It
carries **no hash value**. `MarkEscrowStale` is **precautionary**, not evidential — see S-26(a).
**S-27 — a customer with NO machine ever bound is UNKNOWN, silently; one that was bound and went quiet
still alarms (2026-08-04, R-195; hub v0.92.0).** Operator ruling, implemented as
`store.HasEverBoundHost` (live `hosts` row OR `host_deletions` tombstone) consulted once at the top of
`CheckBackupDeadlines`' loop. **The discriminator is "was a host EVER bound", never "has a report
arrived"** — a box that was installed and never phoned home is a real fault. Fail-**open** on a read
error; the deferral is logged with its own counter; the R-81 anchored-verdict structure is untouched.
*The mechanism is worth carrying, because the guard that should have covered this was keyed off the
wrong thing:* the existing down-skip reads `StalenessChecker.GetState()`, whose map is seeded from
`store.GetCustomers()` — **a query over the `reports` table**. A customer with zero reports is in no
row, gets no state, and `GetState()` returns `""` not `"down"`, so **the skip misses exactly the
customer it would most obviously cover.** `david` (created 2026-08-01, no machine) e-mailed an
`expected_dbdump_missed` ERROR three mornings running; `peti-felhom` — active, host deleted
2026-07-15 — does not, because its 482 old reports make it `down`. Generalise it: **a "skip the dead"
guard built on evidence of life cannot see something that was never alive.**
**S-26 — the one-shot secret is the recoverable one; the irreplaceable one is minted fresh on every
guest rebuild (2026-08-04, R-193 spike — `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`).
No code shipped for it; the decision is the operator's.**
Two sentences, because they are what every future session needs before touching this area:
- **Q1 — what is one-shot.** Of the three secrets in the offsite path, the hub's **one-time
provider password** is one-shot but freely re-issuable, the box's **SFTP key** is regenerated on
every apply by design, and the **restic repository password** — the DATA key, which the agent's own
source calls *"irreplaceable"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the only one
nothing can restage.
- **Q2 — what a rebuild costs.** A rebuilt controller **mints a brand-new repository password**
(`WriteOffboxSecrets`, `offbox.go:392`, generates whenever `<DataDir>/offbox/repo_password` is
absent) and **no automatic path ever consults the escrowed one** — `InjectOffboxPassword` has exactly
one caller in the whole repo, a web form a human pastes into — so **every guest rebuild orphans the
previous off-site history.**
*Measured without touching a box, and the method is reusable:* `host_escrow.restic_pw_sha256` vs
`host_escrow_superseded.restic_pw_sha256`. demo-hp `8e03eddf…`→`8a9e33aa…` (15 snapshots / 40.9 MB
orphaned); demo-felhom `48741892…`→`c60c8bc7…` (**36 snapshots / 1.14 GB**). **demo-felhom is the half
that matters:** R-193 recorded it as having survived the rebuild by luck, and it did — its *delivery*
recovered in 76 s off a stale staged secret. **Its repository did not**, and nothing said so for 13 h.
*Luck restored the plumbing, not the data.*
*Three corrections that must not be re-inherited.* **(a)** `ReissueCredentials` does **not** rotate the
restic password — R-39's record, `offsite.go:198-201` and `api/handler.go:1067-1069` all say it does,
and all three are wrong (→ **R-196**, the eighth entry in `CLAUDE.md`'s table). **(b)** Candidate (b),
*"recoverable from escrow at re-bootstrap"*, is **not implementable** — the escrow is R-wrapped and
zero-knowledge (D6), so only a customer-present ceremony can open it, which is the manual form that
already exists. **(c)** Candidate (a), *"the hub restages automatically"*, **already exists** as
`reissueOnReenroll`'s F3 leg and is wired to the wrong event — it sits behind `handleHostEnroll`'s
mint-once-reuse short-circuit, and a **guest** rebuild leaves the `hosts` row intact. **Shipping it
first would have made both boxes look healthy on 2026-08-04 while their snapshots were orphaned** —
strictly worse than the current loud failure.
*The candidate nobody had named,* and the only one aimed at the actual harm: **the agent survives a
guest rebuild**, already receives the repo password over the pinned local API
(`POST /escrow/stage-secret`) and already writes it to a fixed 0600 path — it merely **wipes** it after
the ceremony. Retaining and serving it back needs no new seam. Its price is one real trade: a copy of
the data key at rest on the Proxmox host. **That trade is the operator's to make and the spike does not
make it.**
**S-24 — offsite retention is ep0's, and the box asks for none (2026-08-04, R-191; installer 1.25.0).**
R-89 moved offsite pruning server-side and box tokens stay write-only. The 2026-07-26 "two weeks"
ruling was not reversed — **where it is ENFORCED moved, and the installer's `keep_last: 2` did not
follow**, so every weekly run uploaded fine and then failed the job on a refused prune. The offsite
tier now writes `keep_last: 0`, which the agent's existing guard already reads as *never prune from
the box*; the local tier is untouched. A gate asserts it. **Before switching it off, ep0 was read:**
both namespaces have a prune job at 03:30 keep-last 2 that has run daily since 2026-07-27 (18 tasks,
all OK). **If that ever stops, `keep_last: 0` is unbounded growth** — check ep0's prune jobs before
assuming the offsite tier is retained.
**S-25 — a lost storage grant repairs itself, and the repair is REPORTED (2026-08-04, R-190; agent
v0.124.1).** On a missing grant the agent runs the existing root wrapper (`felhom-backup-target-apply
grant <id>`, already sudoers-permitted for any id) and re-reads once — the pbsdr R-22 shape. Bounded
at one attempt per tier per hour.
*The record is the point, and getting it wrong is easy.* A repair that leaves only `ok` behind
destroys the only evidence a permission vanished. v0.124.0 reported degraded for "one cycle" and
**that failed in production while every unit test passed**: `probeAll` is called independently by the
self-check log and by the collector, the repairing call was the log's, and the host-report three
seconds later sent `ok`. The fix is a latch on **time** (20 min > the 900 s report interval), so at
least one report must carry it. The explanation rides `Feature`, because that is the field the hub
interpolates into the operator's e-mail — `Reason` does not travel. **Nothing new was built:** the
hub's existing ok→degraded→ok edge is the channel.
*Caveat measured live:* **PVE caches permissions** (~40 s and ~16 min observed), so detection lags the
loss and a single permission read is a lagging indicator → R-194.
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
(2026-08-04). Scope: those two boxes, not the fleet.**
Four SCHEDULED runs overnight, none triggered by hand: demo-felhom host **83.8 s** / offsite
**540.4 s**; demo-hp host **109.3 s** / offsite **300.1 s**. Every one restored into a scratch guest,
booted, verified and destroyed itself.
*What this closes.* Until yesterday every live restore-proof this project held was on the OFFSITE
tier. The on-box tier — the one an ordinary recovery uses — had never been proven on either box, and
not because it failed: the agent could not read the storage it lives on (R-185), so it never saw an
archive there to test.
*What was observed for the first time.* Both boxes had BOTH tiers due simultaneously. Never-proven
sorts first, so each took its host tier, deferred the offsite one, and picked that up on the next
evaluation six hours later — R-86's ordering and the one-heavy-operation gate, working together,
unsupervised. The host-tier proof then reached the hub through R-189's merge, which is that path
carrying a host-tier entry for the first time.
*The asymmetry worth remembering:* a host-tier restore is **83109 s**; an offsite one is
**300540 s**. The tier that matters for an ordinary recovery is also the cheapest to prove.
**S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission
question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).**
*The defect.* On both demo boxes the agent's token had `FelhomAgentStore` on `local`, `local-lvm` and
`felhom-pbs` and **not** on `felhom-backup` — the storage the same installer configured as
`local_backup_target`. The content API answered `{"data":[]}` through the token while root listed
three archives. `pickForThisRun` skipped the tier as *"no settled archive yet"*, which is exactly
what a brand-new tier reports, so the host tier was never restore-testable and nothing said so.
*The rule.* The permission question has a definite answer where the listing does not. `Permissions`
reads `/access/permissions?path=/storage/<target>` **as the agent's own token** — asking as root
answers a different question and always says yes — and one `capability.Status` per configured tier
reports it. The probed set comes from `BackupTiers()`, never a fixed list: a hardcoded probe list is
the defect reproduced inside the fix.
*The measured trap, because the obvious reading is wrong.* An ungranted path answers **neither empty
nor 403**: it carries the privileges inherited from the box-wide `/` grant
(`Sys.Audit, SDN.Use, Datastore.Audit`). Testing for path-presence, or for `Datastore.Audit`, reports
a blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**, and re-measuring is required
before that constant is ever changed.
*Criticality, weighed once.* Critical, because the hub alerts only on critical and a non-critical
entry would ride the report and alert nobody. **Except** the `local` fallback target, which
host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it
does not page — turning an ordinary documented setup into an alert is how a signal becomes something
an operator archives unread. It never consults content, so it cannot alarm on a newborn tier by
construction, and it never reports ok when it could not ask.
**S-22 — the installer's Scenario-F arm must finish the job, not just leave the definition alone
(2026-08-03, R-185).** `configure_backup_target` has two arms. Case A creates the storage and grants
in the same breath. The reuse arm — *"the target already exists"* — returned **without granting**, and
that, not `PVE_STORAGES`, is where the drift came from: a box whose target pre-dated the install
(the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token
could not read. The reuse arm now ensures the ACL through the same guarded wrapper. **Scenario F is
unviolated** — the storage DEFINITION is untouched, and granting the role the agent is supposed to
have on the target this script is about to write into `agent.json` is finishing the job, not
retargeting the box. `$BACKUP_TARGET_ID` stays OUT of `PVE_STORAGES`: that list is granted a step
before the target is resolved, and `--acl-storages` entries are preflight-checked for existence.
A gate asserts every arm that resolves the target also grants on it.
**S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is **S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is
the design (2026-08-03, R-189; agent v0.122.0).** the design (2026-08-03, R-189; agent v0.122.0).**
+132 -109
View File
@@ -1,137 +1,160 @@
# REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03) # REPORT — R-196 / R-204 item 2 (hub v0.95.0), 2026-08-05
> **A LATER SESSION THE SAME DAY (R-189 · R-188 · R-186) touched this repo for REGISTERS AND DOCS **A re-issue no longer marks a healthy escrow stale.** One behaviour change, one register closed, and
> ONLY** — `OPEN-ITEMS.md`, `STATUS.md`, `CONTEXT.md` (S-19, S-20) and the capability map. **No hub the coverage claim proved rather than assumed. The controller's half of R-204 (items 1 and 3) is
> code changed and the hub was not bumped**: the defect was the agent no longer sending `felhom-controller` v0.198.0.
> `restore_tests[]`, not the hub failing to read them. That session's report is
> `felhom-agent/REPORT.md`. What follows is the last hub IMPLEMENTATION, v0.91.1, which is what this
> file is for.
**Date:** 2026-08-03 · **Repo:** `felhom.eu` hub **v0.90.1 → v0.91.0 → v0.91.1**
(`323f45a`, `046df30`), manifests `687fedd`, `ff2655c`. Ships **with** `felhom-agent` v0.121.0, not
after it. Ruling recorded: **ep0 is Tier 2, protected**.
## 1. Baselines, re-read on arrival ## 1. Baselines, re-read on arrival
| Repo | `main` @ commit | Version | Matched §1? | | Repo | Expected | Found |
|---|---|---|---| |---|---|---|
| `felhom.eu` | `e34b614e5b65` | hub CHANGELOG top entry `v0.90.0`; `manifests/hub.yaml` ran **`0.90.1`** | **Yes — and the discrepancy was real** | | `felhom.eu` | `2a7ac03c4726` / hub v0.94.0 (deployed `felhom-hub:0.94.0`) | **exact match**, tree clean, `HEAD == origin/main` |
The missing `v0.90.1` CHANGELOG entry is backfilled. It was built, deployed and recorded in **§3.2's landmark had DRIFTED, and the drift changed the work.** The task described
`REPORT.md` + the R-182 row on 2026-08-03 (`f21e7ca`) but never given a CHANGELOG entry — and `offsite.go:222-231` under a known-consequence comment saying the mark was made on a false premise.
`REPORT.md` is overwritten every session, so the per-repo history under-reported what was running. That comment had already been rewritten by the R-196 comment-correction commit, and the version on
`main` gave a **non-false** ground for the mark: *"the box's re-apply may mint a fresh repository
password (it does exactly that whenever `<DataDir>/offbox/repo_password` is absent — the
guest-rebuild shape)"*. So the question was no longer "delete a comment's lie" but "is the shape it
guards actually covered elsewhere?" — which is Scenario D, and §8.2 says to stop and report if it is
not. It is; §3 below is the evidence.
## 2. Why Part 2 could not ship later ## 2. What changed
The agent now proves a tier **once per archive generation**, so a tier backed up weekly is proved `offsite.ReissueCredentials` no longer calls `MarkEscrowStale` and no longer emits the `escrow_stale`
weekly — correctly, in perfect health. `restoreProvenStaleAfter` was a flat 7 days and its own comment event. **`offsite_reissued` is untouched** and still fires on every re-issue. The known-consequence
derived that number *from the cadence R-86 removes*. comment is rewritten to record what was done, when, and why — with the disagreement below stated in
it rather than absorbed.
The brief said a weekly tier would "sit exactly on that line". That is literally true, and the **What the mark actually cost, established mechanically rather than asserted** (this is why it was a
arithmetic is worth stating: proofs land at `archive + settle`, so just before the next proof the blocker and not a nit):
current one is **exactly one interval old** — 168 h against a 168 h window, and `age > window` is false
by a hair. It did not sit near the line, it sat **on** it. Any ordinary delay — a backup landing late,
one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm.
## 3. The window as derived 1. `stale_at` set → `GetEscrowStatusForCustomer` **withholds** `restic_pw_sha256` from the report ACK.
2. With no hash, the controller's SLICE-3 auto-confirm returns early and cannot flip
`pending → escrowed`.
3. `OffboxRunnable() = OffboxConfigured() && EscrowState == "escrowed"` → **every off-site backup
refused**, indefinitely, on a box whose repository key was never in doubt.
4. The customer is told to re-run the recovery ceremony — which mints a new recovery code and
supersedes the sealed blob. **During a recovery that is the one act that would have destroyed the
key just recovered.**
``` ## 3. Scenario D — the evidence that the removed marking is covered
interval = declaredArchiveInterval(tier) // host 26h, offsite 8d
if observedOK && observed > interval { interval = observed } // observation may only WIDEN
window = clamp(4 * interval, floor 7d, cap 12d)
```
- **4 generations** — the settle generation plus ~3 missed opportunities: deliberately the same The mark was precautionary and aimed at ONE shape: a re-issue followed by a box that mints a fresh
tolerance the flat constant expressed. The change is to the *rhythm*, not to the patience. repository password (the guest-rebuild shape, where `offbox/repo_password` is absent). That shape is
- **The declared rhythm is `backupStaleAfter` (26 h) / `offsiteBackupStaleAfter` (8 d)** — the very measured in two independent places, and **the mark was blinding one of them**:
thresholds the backup-freshness checker already judges those tiers against. Not a second opinion:
if the two checkers disagreed about how often a tier receives an archive, one would be alarming on
the other's model.
- **The observation** comes from reports the hub already holds: `pbs_snapshots[]` plus successful
`backups[]` attributed by **target type** (slice-A.4 — a PBS-targeted vzdump appears in *both*
arrays, and classifying by array membership would attribute an offsite archive to the host tier).
Mean gap over distinct archive times; fewer than two generations ⇒ unobservable.
- **Floor 7 d** — the old constant kept as a floor, so no tier is judged more tightly than before.
- **Cap 12 d** — strictly inside the 2-week offsite retention with two days to spare, so a tier is
never called stale against an archive PBS has already pruned.
| tier | interval used | window | proof age in health | verdict | - **Continuous, box-side — the real coverage.** `report.EscrowAutoConfirmer.reconcileEscrowed`
|---|---|---|---|---| (controller) compares the ACK's sealed `restic_pw_sha256` against the box's CURRENT local repo
| daily host | 26 h declared | 4×26 h → **floor 168 h** | ≤ ~54 h | OK — numerically unchanged from before | password on **every report ACK**, raising the stale flag, the customer card and the
| weekly offsite | 8 d declared (7 d observed loses to it) | 4×8 d → **cap 288 h** | ≤ ~198 h | OK, 3.75 d of margin | „create a new recovery code" CTA on a mismatch. That is a **measurement**, not a guess, and it is
| newborn offsite | 8 d declared | **288 h** | — | UNKNOWN until the anchor passes | continuous rather than edge-triggered. Pinned by the controller's
`TestEscrowStale_MismatchWarnsOnceAndFlags` — re-run green this session.
**And step 1 above was blinding it:** a stale flag empties the very hash that comparison needs, so
the box could only reach the hash-LESS branch and report *"the hub's current blob carries NO
password hash"* — which is false. Removing the mark restores the true signal.
- **Edge-triggered, hub-side.** R-197's `offsite_repo_key_changed` fires from
`api.maybeEmitRepoKeyChanged` on a proven hash difference across a supersession and pages the
operator. **Red-proved:** removing the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut`
made `TestEscrowPut_ChangedRepoKey_RaisesSignal` fail with *"the repository key demonstrably changed
and NO signal was raised"*, while the two silence tests stayed green.
## 4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box **Disagreement recorded, per the R-96 standing rule.** Scenario D as written asks that a real key
change also *"mark the escrow stale"*. **It must not, and nothing was changed to make it.** The hub
learns of a real change at the instant a supersession **seals the new password** — i.e. when the
escrow is at its freshest. Marking it stale there would ask the customer for a ceremony to fix the
ceremony that just ran. The correct consequence at that instant is the operator alarm, which is
exactly what R-197 already does. This is recorded in the code comment, the CHANGELOG and OPEN-ITEMS,
not only here.
demo-felhom's offsite tier holds two retained snapshots, `2026-07-27T19:55:41Z` and ## 4. `MarkEscrowStale` is kept with no caller — deliberately
`2026-07-28T04:49:43Z`**8 h 54 m apart**, because one is a healing artefact and the other a real
weekly run. A mean-gap estimate therefore reads a **weekly** tier as nine-hourly: ×4 = 36 h, the floor
lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm
this whole task exists to prevent would have returned within a week, on the box it had just shipped
to.**
Fixed by `max(observed, declared)`. A gap *shorter* than the declared rhythm is routine and means Per task §5 it was not to be modified, and it is not deleted either. The `stale_at` flag remains live
nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap *longer* than it is real and correct — read by the ACK, the operator config card and the PBS-DR view — and the right way to
information. **Cost, stated rather than hidden:** a tier that truly runs faster than its declared set it is a **future EVIDENTIAL caller** that has measured a key change rather than guessed at one.
rhythm gets a wider window than it needs, i.e. a slower `restore_test_stale`. Right direction for a Its doc comment now says so plainly instead of naming a caller that no longer exists, and
signal meaning *unverified**broken now* is `restore_test_failed`, immediate and untouched. `TestEscrowStaleMechanism_StillWithholdsAndClears` keeps the mechanism from decaying to inert while
nothing writes it (the seam-built-but-never-wired shape, in reverse).
## 5. Kept, because it was earned The schema comment and `EscrowStatus.Stale`'s comment were corrected the same way — each of the three
previously asserted a writer that is now gone.
- **Absence is UNKNOWN** until an anchored window has passed (R-81's structure, unchanged). ## 5. Files modified
- **The stale signal stays edge-triggered** (`staleStates` untouched).
- **`restore_test_failed` and `restore_test_stale` stay DISTINCT** — one says your recovery is broken, | File | Change |
the other that it is unverified, and the second is the one that quietly becomes the first. |---|---|
- **Every reason string now states the window it was judged against** — R-100's corollary: when a | `hub/internal/offsite/offsite.go` | the pessimistic `MarkEscrowStale` + `escrow_stale` event removed; comment rewritten to record the change, the coverage and the disagreement |
verdict changes what it counts from, the alarm text must change with it, or an operator reads | `hub/internal/offsite/offsite_test.go` | `TestReissue_InvalidatesEscrow` **replaced in place by its exact inverse** `TestReissue_DoesNotMarkAHealthyEscrowStale`; new `TestEscrowStaleMechanism_StillWithholdsAndClears` |
"limit 168h" under a tier judged at 288 h and dismisses a true alarm. | `hub/internal/store/store.go` | three comments corrected (`MarkEscrowStale`, the `stale_at` schema note, `EscrowStatus.Stale`) — each had named a writer that no longer exists |
- **The window READ is unchanged in cost** (14 days) — enough to find proof inside the widest window | `manifests/hub.yaml` | image tag `0.94.0``0.95.0` |
and to see two generations of a weekly tier. | `hub/CHANGELOG.md`, `CONTEXT.md`, `STATUS.md`, `documentation/…` | v0.95.0 entry; ruling **S-32**; the register and architecture updates below |
**Commits on `main`:** `d1a8edb` (behaviour + tests + comments) · `5c7d671` (CHANGELOG) ·
`975a690` (manifest bump).
**Deploy:** built + pushed `felhom-hub:0.95.0`, bumped `manifests/hub.yaml`, pushed, then a
**deliberate ArgoCD hard-refresh + sync** (auto-sync stays off; no `kubectl set image` anywhere).
Result: app `felhom` **Synced / Healthy**, `deploy/hub` rolled out, running image
`gitea.dooplex.hu/admin/felhom-hub:0.95.0`, startup log clean (offsite provisioning, pool-box checker,
PBS-DR reconciler and all six host checkers initialised; `Listening on :8080`).
## 6. Tests and red-proofs ## 6. Tests and red-proofs
| Test | Asserts | Mutation | Observed | Green gate: `cd hub && go build ./... && go vet ./... && go test ./...`**rc=0**.
`python3 scripts/repo_gates.py --fast`**all five gates OK**.
| Test | Result | Red-proof — what was mutated | Outcome |
|---|---|---|---| |---|---|---|---|
| `TestRestoreTest_HealthyWeeklyTierIsNeverStale` | a healthy weekly tier **with jitter** is never stale across 6 weeks | window pinned flat at `restoreProvenWindowFloor` | **FAIL**`week 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s"` | | `TestReissue_DoesNotMarkAHealthyEscrowStale` (C) | PASS | restored the pessimistic `MarkEscrowStale` block in `ReissueCredentials`, exactly as it was | **FAILED***"a re-issue marked a HEALTHY escrow stale…"* |
| `TestRestoreProvenWindow_Contract` | floor, cap, declared fallback, **observation may only widen** | `observed > interval``observed > 0` | **FAIL**`window(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s` | | `TestEscrowStaleMechanism_StillWithholdsAndClears` | PASS | same mutation | **stayed GREEN** — correctly: the mutation restores a *caller*, not a break in the mechanism. That split is the evidence Scenario C's assertion is about the caller and not the flag. |
| `TestObservedArchiveIntervals_FromReports` | rhythm observed per tier; a PBS vzdump is **not** counted into the host tier | — | pass | | `TestEscrowPut_ChangedRepoKey_RaisesSignal` (D) | PASS | removed the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut` | **FAILED***"the repository key demonstrably changed and NO signal was raised"* |
| `TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms` | a window that never fires is a deletion, not a fix | — | pass | | `TestEscrowPut_UnchangedRepoKey_Silent`, `TestEscrowPut_HashlessSupersession_NoSignal` | PASS | same | stayed green — the detector's silence branches are independent |
| existing R-85/R-81 suite | anchored UNKNOWN, edge-trigger, distinct events, newborn silence | — | pass, unchanged | | controller `TestEscrowStale_MismatchWarnsOnceAndFlags` | PASS | — (cited as the continuous-coverage pin) | — |
**A hollow test caught by its own red-proof.** The first Scenario-G fixture used a perfectly regular Scenario C asserts the **consequence** (the ACK still carries the hash, so auto-confirm can proceed)
weekly tier and **PASSED under the flat-window mutation**, because a regular tier's proof age lands on rather than the mechanism (that a function was not called), because the hash is what the drill's
exactly 168 h and `>` lets it through. It would have shipped Part 1 and its false alarm together while blockage actually turned on. It also asserts that `offsite_reissued` still fires — removing a false
looking like proof. The jitter is what makes it a test — and it is also the truth about the old alarm must not remove the true notice.
constant.
## 7. Deployment ## 7. Live validation
| Step | Evidence | **Per task §12 point 5, a live re-issue was NOT run, and must not have been on demo-hp** — it would
|---|---| have been a credential rotation on the box holding the drill's evidence. Part 2 is proved by test and
| Images built + pushed | `felhom-hub:0.91.0`, then `0.91.1` | by the deployment being live and healthy. The controller-side halves of R-204 were validated live and
| Manifest bumped in git | `687fedd`, `ff2655c` — never `kubectl set image` | are reported in `felhom-controller/REPORT.md`.
| ArgoCD deliberate sync | `sync=Synced health=Healthy` |
| Running image | `gitea.dooplex.hu/admin/felhom-hub:0.91.1` |
| Startup log | checkers initialized; `Listening on :8080` |
## 8. Part 3 — the ep0 ruling, recorded ## 8. Register and documentation
`runbooks/target-selection.md` had carried *"D-d did not name ep0 either way. Confirm it explicitly"* - **`OPEN-ITEMS.md`** — **R-196 → CLOSED (hub v0.95.0)**; **R-204 → items 13 CLOSED, item 4 OPEN
for two days. **Operator ruling, 2026-08-03: ep0 is protected.** Recorded three ways — the tier table (→ R-193)** with its dependency named. The header block is updated and states explicitly that
names it, the standing question is gone, and its per-machine section is retitled **Tier 2, PROTECTED**. **R-202**, **the ~1.2 GB orphaned-ciphertext deletion** and **R-198's retention (still UNIT-PROVEN
`CONTEXT.md` S-18 records that this **extends D-d's protected list to three machines** (DooPlex, ONLY — the second deliberate wipe is the next item)** all **remain open**, so nothing is presumed
Peti's cluster, ep0) and that it is a *classification*, not new prohibitions: destroying datastores, closed by association. R-201 is recorded as PASSED. **R-204 is still the highest ID; nothing new
prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and **the was minted.**
ordinary off-site read a restore-test performs remains permitted** — which this session then exercised. - **`architecture/00-capability-map.md`** — the recovery row now says three of the four crutches are
gone, names the fixes and their evidence, and states that **item 4 (R-193) is the one that remains**
and is why the row **keeps its "with a person present" qualifier**. The **R-199 back-pointer was
already present** on the adjacent key-recovery row (added when that row was last corrected), so it
needed no further action — verified, not assumed.
- **`architecture/07-backup-architecture.md`** — **new §7.0, "What a customer can and cannot do
ALONE"**: the four steps in a table with what each cost and its status, plus the honest current
answer. This is the section a future reader will use to answer the question.
- **`documentation/backlog/ROADMAP.md`** — R-196 and R-204 collapsed per the coupling rule.
- **`CONTEXT.md`** — new standing ruling **S-32**, which supersedes S-31's steps 25 and carries the
blinding mechanism, the fail-closed rule and the "no TTL" reasoning forward.
- **`STATUS.md`** — rewritten to **one screen** (191 → ~90 lines) per its own header. It also had a
corrupted, half-overwritten "What we're working on" section left from the drill session, which is
now gone. Next item stated as the retention drill.
## 9. Registers and docs **CI:** felhom.eu runs **154** (`5c7d671`, code) and **155** (`975a690`, manifest) — both success.
`--no-verify` was **not** used; the pre-push gate ran and passed on every push.
- **R-86 → CLOSED**, shipped + proven live, with the trap in its own wording recorded on the row. ## 9. Observations — noticed, NOT acted on
- **R-87 → re-ranked UP**: R-86 built most of what it waited for. What remains is restic-specific.
- **R-185 / R-186 / R-187 → filed** (grep established 185187 free; R-184 was the highest in use). - **`allowedEventTypes` still lists `escrow_stale`**, which after this change has **no producer** in
- `ROADMAP.md` R-86 collapsed, keeping the reasoning and **correcting the shape the row itself either repo. It is inert rather than harmful; removing an allowlist entry is a behaviour change and
proposed** — which was the never-fires version. is out of this session's scope.
- `STATUS.md` rewritten for the operator, trimmed back to one screen. - `MarkEscrowStale` is now dead code by call-graph. Kept on purpose (§4 above) — but if a future
- `07-backup-architecture.md` §3 carries the new contract (S-1); `00-capability-map.md`'s restore-test session's linter or cleanup pass proposes deleting it, the reason it exists is in its doc comment
row upgraded with its live citation; `CONTEXT.md` gains S-17 (the rule + the trap + the config key) and in the test that exercises it.
and S-18 (ep0). - `/` on DooPlex is at **86%** used — under the 90% abort line, but worth watching before large builds.
+3
View File
@@ -75,8 +75,11 @@
| `(*Store).GetHostRecoveryMeta` + `(*Server).handleHostRevealRecoveryCredential` | hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | `(hostID) (*HostRecoveryMeta, error)` · `POST /hosts/{id}/reveal-recovery-credential` | The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | **Use `GetHostRecoveryMeta` on any page-render path** — its struct and its `SELECT` both omit the `secret` column, so it cannot leak one; `GetHostRecoveryCredential` (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE `recovery_credential_revealed` event via `SaveEvent` and calls NO dispatcher (the `handleRequestLogTail` shape). `api/handler.go handleAdminGetRecoveryCredential` (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. | | `(*Store).GetHostRecoveryMeta` + `(*Server).handleHostRevealRecoveryCredential` | hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | `(hostID) (*HostRecoveryMeta, error)` · `POST /hosts/{id}/reveal-recovery-credential` | The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | **Use `GetHostRecoveryMeta` on any page-render path** — its struct and its `SELECT` both omit the `secret` column, so it cannot leak one; `GetHostRecoveryCredential` (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE `recovery_credential_revealed` event via `SaveEvent` and calls NO dispatcher (the `handleRequestLogTail` shape). `api/handler.go handleAdminGetRecoveryCredential` (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. |
| `host_detail_body` sub-template | hub/internal/web/templates/host_detail_body.html | `{{template "host_detail_body" .}}` | Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with `{{.HostID}}` (the customer page renders N instances). | | `host_detail_body` sub-template | hub/internal/web/templates/host_detail_body.html | `{{template "host_detail_body" .}}` | Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with `{{.HostID}}` (the customer page renders N instances). |
| `(*Store).ListHostsByCustomer` | hub/internal/store/store.go (~L1620) | `(customerID) ([]Host, error)` | A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. | | `(*Store).ListHostsByCustomer` | hub/internal/store/store.go (~L1620) | `(customerID) ([]Host, error)` | A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. |
| `(*Store).HasEverBoundHost` (v0.92.0, R-195) | hub/internal/store/store.go | `(customerID) (bool, error)` | Any verdict that must not fire for a customer with **no machine ever bound** — "was anything ever expected of this customer" | `hosts` row **OR** `host_deletions` tombstone. **NOT a liveness check and never a substitute for one:** a box that was bound and went silent returns `true` and must keep alarming — that is the case any change here breaks first (pinned by `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`). Callers **fail OPEN** on its error: an unreadable binding must never SUPPRESS an alarm. Do **not** re-derive this from report presence — `store.GetCustomers()` (and therefore the staleness checker's `down` state) is a query over `reports`, so a never-reported customer has no state at all, which is exactly how the daily false alarm reached `david`. |
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). | | `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. | | `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
| `handleHostEscrowGet` + `escrowSelfServiceRetrieval` | hub/internal/api/handler.go | `GET /hosts/{id}/escrow` | The box-authenticated MIRROR of the escrow PUT — self-scoped by the per-host key. **`escrowSelfServiceRetrieval` is THE single decision point** for whether a box may read its own blob without operator-armed recovery mode (§8.2 vs §8.3, R-199): flip it, do not scatter the condition. **Never merge this with `dr.go`'s `handleReEnroll`/`handleGetRestoreDirective`** — those rotate the host API key and serve the K-escrow and directive too, and keep their recovery-mode gate (pinned by `TestEscrowGet_OperatorDRPathUnchanged`). Every successful retrieval MUST raise `escrow_blob_served` before the bytes leave; that audit row is the mitigation the trade rests on. |
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts**`blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** `SaveHostDRBundle` writes `identity_blob` AFTER `SaveHostEscrow` returns, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. | | `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. | | `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. | | `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
+90 -54
View File
@@ -1,6 +1,6 @@
# STATUS — what works, what's broken, what's next # STATUS — what works, what's broken, what's next
**Updated 2026-08-03.** **Updated 2026-08-05.**
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**, > page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
@@ -13,71 +13,107 @@
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer, A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
who sets their own password. They install apps from a catalogue of fifty-three, share files over the who sets their own password. They install apps from a catalogue of fifty-three, share files over the
home network, and open apps from a launcher or a shared link. Backups run on their own to three home network, and open apps from a launcher or a shared link. Backups run on their own to three
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can places — the machine's drive, a second drive, and an encrypted off-site copy.
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
box six times, everything returned every time, and an app switched off deliberately stayed off. **And the whole backup promise is now proved.** On 4 August we destroyed a machine on purpose and
Proven end to end on real hardware. deleted a marked file from its disk. Using the recovery code you saved: the backup key came back
**identical, character for character**; the existing off-site store **opened** rather than starting
over; and the file was restored **byte for byte identical**. *(R-201)*
## What's broken ## What's broken
- **A customer still cannot do that recovery alone — but only one step is left.** Getting from "the
key is recoverable" to "the file is back" took four steps that appeared in no instructions. **Three
are fixed today** *(R-204)*: the local reset-code tool works on the first try instead of needing the
controller restarted; re-issuing the storage credential no longer falsely marks the recovery key
"stale" (which used to stop every off-site backup and invite the one act that would have destroyed
the recovered key); and the everyday restore now **says in plain Hungarian that it returned the
app's settings and database and not your documents**, and names the button that does. **The step
that remains is the first one:** a rebuilt machine cannot get a storage credential by itself,
because the one-time password was used up by its predecessor — so you still have to press
Re-issue. That is a design decision waiting on you, below. *(R-193)*
- **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key
that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one. Both demo
machines did this on 34 August — 51 backups (~1.2 GB) between them. The old key now survives the
recovery ceremony, and a changed key now raises an alarm the same day, but **the rebuild itself
still starts a fresh history**. *(R-193)*
- **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
says the old backups may later be restorable with the matching recovery code. That is true for
machines that re-seal from now on and **false for anything already orphaned** — and the machine
cannot tell which case it is in. We deliberately did **not** patch the sentence: a conditional
promise that can still be wrong is worse there than a vague one. *(R-202)*
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can - **The off-site copy can be erased by the machine that made it.** The credential that writes it can
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy. also delete it. A daily snapshot is armed as a stopgap. *(R-95, R-87)*
*(R-95, R-87)*
- **Nothing else open that affects a customer.** The three faults that were on this list yesterday —
the reserve watching the wrong step, the last app whose data was never saved, and the alert that
told you about one app and swallowed the rest — are all fixed and proven on a real machine.
## What shipped recently ## What shipped recently
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked - **2026-08-05** — **The DooPlex server's disk is out of danger: 86% full → 54%, and the storage layer
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was is unstuck.** The cause was leftover working data from building our own software — 157 GB of it,
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a growing about 5 GB a day, which nothing was allowed to delete. **148 GB came back in 86 seconds.**
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now The storage layer had already stopped accepting new copies of any volume onto that disk; that is
written to disk with the result and survives. This was caught happening, not predicted: a real fixed the same day. A **30 GB ceiling** is now in place and was **proved to work by deliberately
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a overfilling it and watching it evict** — not by assuming the setting took. Two things that failed
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now quietly around it: the warning meant to catch exactly this **could never fire** (fixed and proved),
published after the binary, and a new check catches the opposite mistake so nothing is traded away. and the weekly cleanup is still forbidden from touching the thing that grows (next session).
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you *(R-205 … R-211)*
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)* - **2026-08-05****The build data now lives on the second SSD, moved with nothing lost.** All 345
- **Each backup is now proved, instead of the clock being obeyed.** The machine used to re-test a images, every saved volume and both development databases came through identical — checked before
backup every twenty-four hours on a timer that restarted whenever the software was updated, so the the original was touched and again afterwards, and confirmed by running a real build on the moved
test happened at an arbitrary time and a fresh backup could sit unproven while an old one was copy. **The server's own services never went down:** Gitea, the registry, the hub and the backup
re-checked. Now a copy is tested once, about a day after it is made, and not again until there is a system run on a separate system and stayed up throughout. The second SSD now also **reserves 80 GB**
newer one — so the daily copy is proved daily and the weekly off-site copy weekly, each on its own. for this, so the storage layer can no longer quietly claim the space and repeat what happened to the
The alert that says "this copy has not been proved lately" learned each copy's own rhythm in the first disk. The old copy is kept as the way back until the machine next restarts. *(R-209)*
same change; without that it would have started complaining every night about a system that is - **2026-08-05****Three of the four recovery crutches removed.** The reset code works first time;
working. *(R-86)* a credential re-issue no longer blocks off-site backups on a healthy machine; and the default
- **A failed backup now tells you about every app, once**, with every failure written down whether or restore no longer quietly returns the wrong thing. *(R-204 items 13, R-196)*
not it is emailed, and a message the machine decides not to send now leaves a record saying so. - **2026-08-04 (night)****The drill PASSED**, end to end, on real hardware. *(R-201)*
*(R-182)* - **2026-08-04** — The folder-left-out-of-the-backup problem fixed both halves: the app and its backup
look in the same directory, and a backup that misses a folder marked essential reports *incomplete*
instead of success. *(R-203)*
- **2026-08-04** — The hub now keeps the off-site backup key when a machine re-seals, instead of only
the whole-machine one, and a machine can fetch its own sealed package back. *(R-198, R-199)*
- **2026-08-04** — The daily false alarm about David is gone; a vanished permission now repairs itself
**and says it had to**; the weekly off-site backup stopped reporting failure after a successful
upload. *(R-195, R-190, R-191)*
## What we're working on ## What we're working on
- **Now:** nothing outstanding. - **Next: the retention proof.** The hub keeping the old sealed key when a machine re-seals is the one
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing remaining link that has **never run outside a test**. Proving it needs a second deliberate wipe on
tests unattended. Most of the machinery it needed arrived with the restore-test change below. the spare demo machine, and it is its own procedure. *(R-198)*
*(R-87)* - **Then:** the orphaned-backup deletion you asked for (below), and the off-site copy the machine can
- **After:** the off-site copy that the machine making it can still erase. *(R-95)* still erase. *(R-193, R-95)*
## Waiting on you ## Waiting on you
- **One thing to read after the machine next restarts — and nothing to do until then.** You told me
not to restart DooPlex, so I did not, and the move to the second SSD has therefore never been
through a restart. It works right now and nothing was lost, but a restart is the one test that
matters for this kind of change, and it has not happened. **I made it check itself:** whenever the
machine next starts, for any reason, it writes a plain PASS or FAIL line to
`/var/log/felhom-store-postboot-check.log`. **If it says PASS, the old copy can be deleted and 34 GB
comes back.** Until then I have deliberately kept that old copy, which is the only quick way back —
it is why the disk sits at 54% rather than lower. *(R-209a)*
- **One list to rule on: 193 old images that exist only on this machine.** 131 controller versions and
62 hub versions are not in the registry, so they cannot be re-downloaded — all of them old
(controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing
was deleted.** Worth knowing before you spend time on it: they only account for about 27 GB against
199 GB now free, so this is about clutter, not space. *(R-210)*
- **The one-shot credential decision — this is now the last thing between a customer and an unaided
recovery.** A rebuilt machine has no storage credential of its own, so an operator must press
Re-issue. Everything after that point is now self-service. Deciding how a rebuilt machine should get
a credential is the remaining design question. *(R-193, R-204 item 4)*
- **The recovery screen you described has been priced, and it can be built.** A freshly installed
machine that finds a sealed package waiting should say so, offer a box for the recovery code, and
show what would come back before doing anything. One thing to weigh, deliberately not decided: that
screen is reachable by anyone with the household's dashboard password, and the preview reveals
backup dates and app names. *(R-193)*
- **The orphaned backups on the storage box — you said delete, and it is still owed.** About 1.2 GB
across the two demo machines, in set-aside stores nobody can open and nothing prunes. It wants its
own session rather than riding along with other work. *(R-193)*
- *(decided 4 Aug)* You chose **not** to keep a copy of the backup key on the Proxmox host, which
makes the customer's own recovery code the only route back from a rebuild. *(R-193)*
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
session log; nothing suggests anyone else saw it. *(R-132)* session log; nothing suggests anyone else saw it. *(R-132)*
- **One small question, not urgent.** The automatic check cannot see which version you have told - **One small question, not urgent.** The automatic version check cannot see which version you have
machines to install, only which ones exist. Closing that needs either a password given to the build told machines to install, only which ones exist. *(R-184)*
server or a check inside the hub itself. *(R-184)*
- **Nothing else.** The question about whether the off-site endpoint counts as protected is
**settled — it does**, and the machine list now says so instead of asking.
## Changed since last update
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
that could vanish on a restart (seen happening), a release that emailed a failure for a release
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
- **2026-08-03** — Backups are now proved one at a time, each about a day after it is made, instead of
on a timer; the "not proved lately" alert learned each copy's own rhythm so it does not complain
about a healthy weekly copy. You settled that the off-site endpoint is protected, and the machine
list records it. One thing found while testing: on the small demo machine the agent cannot see its
own local backups at all — a permission that was never granted — so that copy has never been
test-restored there. Written down, not yet fixed. *(R-86, R-185)*
File diff suppressed because one or more lines are too long
@@ -378,6 +378,60 @@ nicety.
**loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An **loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An
honest signup-time tradeoff, not a hidden one. honest signup-time tradeoff, not a hidden one.
#### Custody across generations — what the hub retains when a blob is superseded (hub v0.60.0; **contract corrected hub v0.93.0, R-198**)
A ceremony that seals a *different* passphrase does not destroy the blob it replaces: the hub copies
the current `host_escrow` row into `host_escrow_superseded` **before** overwriting it, so a previous
generation stays openable with the recovery code that sealed it. The same routine
(`demoteCurrentEscrowTx`) is used when a host is deleted, which **demotes** custody rather than
destroying it. The hub never decrypts either table.
**The contract as originally shipped was incomplete, and the gap is worth recording because it was
invisible for two months.** An escrow generation is **two** sealed artifacts — the K-escrow `blob`
(the PBS datastore key `K`) and `identity_blob` (the age-wrapped `IdentityBundle`, which carries the
offsite **restic repository password** as well as the tunnel/PBS tokens and the WG key). Retention
covered only the first. So a supersession preserved the whole-guest key and destroyed the off-site
data key — and because the copy happens as the new blob lands, **the destroying act was the ceremony
itself**, i.e. the step a rebuilt box asks its customer to perform. Corrected in hub v0.93.0: the
retained row carries both.
**Two consequences that belong in this doc rather than a changelog:**
- **Generational custody is now the recovery precondition for the offsite tier.** The operator ruled
on 2026-08-04 that no repository password is retained on the Proxmox host (R-193 candidate (c),
refused). With no host-side copy, a rebuilt box's only route back to its previous off-site history
is *the customer's older recovery code opening the retained identity blob* — which exists only if
the supersession that replaced it happened on hub v0.93.0 or later.
- **Retention is custody, not recovery, and this doc must not be read as claiming otherwise.** The
hub's restore-mode serving of these blobs (§ slice 10D) has **no client** in the agent, the
controller's recovered-password injection seam has **no user interface**, and no part of the chain
has ever been exercised outside a unit test — the one live identity round-trip (2026-06-10)
predates the `ResticRepoPassword` field. Tracked as R-199/R-200/R-201; evidence
`documentation/audits/RECON-offsite-dr-chain-2026-08-04.md`.
#### Serving a blob back to its own box (hub v0.94.0 + agent v0.125.0, R-199)
Retention without retrieval is custody, not recovery. Since 2026-08-04 a host reads its own sealed
identity blob from `GET /api/v1/hosts/{host_id}/escrow` — self-scoped by the per-host key, the mirror
of the PUT that stored it — and opens it locally with the customer's recovery code via the agent's
`POST /escrow/recover-offsite-password`, which returns **only** the offsite repository password.
**This is a deliberate widening of who can obtain the ciphertext**, recorded in `CONTEXT.md` S-29 and
in the handler itself: it previously took an operator arming recovery mode, and now takes only that
box's own credential. The blob remains unopenable without R, and every retrieval raises an
operator-visible `escrow_blob_served` event. The operator-driven re-enroll/restore-directive path is
unchanged and keeps its recovery-mode gate.
**Proven end to end on demo-felhom, 2026-08-04:** the recovered repository password's sha256 was
byte-identical to the one on the box, and to the hash the hub had independently recorded. **The chain
stops there** — nothing installs a recovered password, reopens a repository with it, or restores a
file.
**The honesty property above is unchanged and now has a second edge:** losing `R` and the box makes
the offsite backups unrecoverable by anyone — *and* an older generation's data is recoverable only
with **that generation's** recovery code, which the customer must still hold. A current recovery code
does not open a previous generation's blob.
## 9. Provisioning & DR flows ## 9. Provisioning & DR flows
**Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs **Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs
@@ -278,7 +278,10 @@ The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no | | **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) | | **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
| **Plane-2** whole-guest, local | `local:``/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** | | **Plane-2** whole-guest, local | `local:``/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` | **yes** (per-customer key) | | **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30`**and the box asks for none** (R-191) | **yes** (per-customer key) |
> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed **[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
@@ -348,6 +351,26 @@ about whether the bytes can be turned back into a working system, and a system c
completely while having no executable recovery route for a given failure. That is not a hypothetical completely while having no executable recovery route for a given failure. That is not a hypothetical
here — §8 has rows where it is the actual state. here — §8 has rows where it is the actual state.
### 7.0 What a customer can and cannot do ALONE — the four steps the drill found
> **[FACT] Added 2026-08-05.** The 2026-08-04 R-201 drill (`audits/DRILL-r201-night-run-2026-08-04.md`)
> is the first end-to-end proof that a customer's file survives a machine rebuild and comes back
> byte-identical. **It passed with an operator present, and four manual interventions stood between
> "the key is recoverable" and "the file is back" — none of which was in any design document.** They
> are recorded here because this is the section a future reader will use to answer *"can the customer
> do this alone?"*, and until 2026-08-05 the answer was no for reasons nothing wrote down.
| # | The step | Why it stopped a customer | Status |
|---|---|---|---|
| 1 | **The reset code was refused on the first try** | `--print-reset-code` runs as a SEPARATE process (`docker exec`); it persists a new code while the running controller keeps the old one in its cache, so the code the customer is told to type never matches. **The controller had to be restarted in between**, which nothing said. Two attempts failed during the drill before that was worked out. | **CLOSED — controller v0.198.0** (R-204 item 1). `effectiveClaimCode` reads through to the persisted state. Read-through, not a TTL: a TTL would leave a window in which a superseded code still works. Fails closed on an unreadable state. Proven live on demo-felhom 9201 with nothing restarted. |
| 2 | **Re-issuing the off-site credential marked the recovery escrow "stale"** | A stale escrow withholds `restic_pw_sha256` from the report ACK → the controller's auto-confirm cannot flip `pending→escrowed``OffboxRunnable()` is false → **every off-site backup refused**. The customer is then told to re-run the recovery ceremony, **which is the one act that would have destroyed the key just recovered.** The repository password had not changed at all. | **CLOSED — hub v0.95.0** (R-204 item 2 / R-196). The precautionary mark is gone. The real case is measured by the controller's Scenario-F hash re-check each ACK — which the mark was BLINDING by emptying that very hash — and by R-197's `offsite_repo_key_changed` at a supersession. |
| 3 | **The restore's default returned the wrong thing, silently** | `mode=unit` restores the recovery unit — the app's definition, configuration and database dumps — and **not the customer's files**; the userdata in the same snapshot is excluded by `--include`. The outcome message was one sentence for both modes and named neither scope. On the last step of a disaster recovery, the default quietly did not do what the person asked. | **CLOSED — controller v0.198.0** (R-204 item 3). The unit outcome names what came back, what did not, and the step that gets it; the wizard's intent card states its scope before the choice. **The size gate on `mode=full` is untouched**, and the default stays `unit`. |
| 4 | **A rebuilt box cannot obtain an off-site credential unaided** | The one-time provider password was spent by its predecessor, so the rebuilt guest has nothing to authenticate with and an **operator Re-issue** is required. | **OPEN → R-193.** This is a design decision (the one-shot credential model), not a defect, and it needs an operator ruling. **It is the reason the capability-map row keeps its "with a person present" qualifier.** |
> **[FACT] The honest current answer:** three of the four are gone. **A customer alone still cannot
> complete a rebuild recovery**, because step 4 requires the operator. Everything after step 4 —
> claim, restore, verify — is now self-service.
### 7.1 The dependency graph ### 7.1 The dependency graph
> **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below > **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below
@@ -0,0 +1,265 @@
# DRILL — R-201, the night run: **PASS. The customer's file came back, byte-identical.**
**Date:** 2026-08-04, 21:3023:15 · **Box:** `demo-hp` (Tier 0) · unattended, then **finished with the
operator present**. The box is up, healthy, and fully re-armed.
> **THE VERDICT: PASS.**
>
> ```
> pre-wipe sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
> restored sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
> ```
>
> A machine was destroyed on purpose — controller data volume gone, the customer's file deleted from
> disk — and **the file came back byte-identical**, through the customer's own recovery code and the
> customer's own restore flow. **This is the first time the Felhom backup story has been proved end to
> end.**
>
> The key recovered byte-identical (`8a9e33aa4da6…`), the existing repository **opened rather than
> being re-initialised** (`repo_state: null`, 3 snapshots, 42 026 B — the pre-wipe size exactly), and
> the pre-wipe snapshot `e6132ae5` was still there with the file inside it.
>
> **It took four undocumented steps to get there**, each measured, and they are the other half of the
> result — see §3. A customer hitting this alone today would not get their data back.
---
## 1. The verdict — **PASS**
| | |
|---|---|
| sentinel sha256, pre-wipe (step 3) | `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` |
| sentinel sha256, restored (step 10) | **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** |
| **match** | **YES — byte-identical** |
| snapshot count at step 9 | **3** — the pre-wipe count. **NOT 1.** No fresh history was started |
| repo size at step 9 | **42 026 B** — the pre-wipe size exactly |
| `repo_state` | **`null`** — the repository OPENED. Not orphaned |
The restore ran through the customer flow (`POST /backup/offbox/restore`, `mode=full`, two-step
confirm), out of snapshot `e6132ae5` — the one taken **before** the wipe.
**A detail worth keeping:** the first restore attempt used the default `mode=unit`, which recovers the
recovery unit only and **not** the userdata leg. `mode=full` is what returns the customer's files. A
customer told to "restore from off-site" would, on the default, get their app definition back and not
their documents — and nothing in that outcome says so.
## 2. What was proven, in order, on hardware
### Step 6 — the wipe
The controller's data volume was destroyed and the bootstrap re-ran. Fresh data directory, everything
stamped `2026-08-04 20:00:2x`:
```
encryption.key 32 B 2026-08-04 20:00:21 ← brand new: every pre-wipe app secret is undecryptable
claimed = None ← the fresh-install signal
offbox = null ← no off-site target
```
**Faithful to the incident, and deliberately so.** The 2026-08-03 rebuild that R-193 is filed against
was **not** a guest reprovision — the journal shows guest 9201 running continuously through that
window, with no `pct destroy`, no `pct restore` and no `--selftest=provision`. What changed was the
controller and its data volume. Reproducing *that* is what the drill needs; a guest destroy plus an
unrehearsed provisioning chain, improvised unattended, is what §8.10 exists to prevent.
### Step 7 — the assertion that keeps recovery possible: **PASSED**
```
host_escrow (demo-hp-bb76ea), AFTER the wipe:
identity_blob = 572 bytes ← unchanged
restic_pw_sha256 = 8a9e33aa4da6… ← unchanged
updated_at = 2026-08-04 11:11:37 ← unchanged; nothing re-escrowed
stale_at = NULL
```
**No ceremony was run and nothing re-escrowed itself.** The sealed key survived the rebuild untouched.
### Step 8 — **THE KEY CAME BACK**
```
=== offsite key recovery check (R-200) — compares, never installs ===
recovered sha256: 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a
[FAIL] there is no repository password on this box to compare against
(the recovery itself SUCCEEDED — this box simply has no local key. That is the
rebuilt-box shape, where the next step is to INSTALL rather than compare.)
```
Exit 1 is **correct and designed**: there was nothing to compare against, because the wipe removed it.
The recovered hash matches the pre-wipe on-disk key and the hub's own record — three independent
sources agreeing, one of them recovered through the full chain (hub → agent → unseal → extract) on a
box that had just lost everything it knew.
### Step 9a — installed cleanly
```
=== offsite key recovery INSTALL (R-200) ===
on-disk sha256: (none — this box has no repository password)
recovered sha256: 8a9e33aa4da6…
[INSTALLED] the recovered repository password is in place and reads back identical.
```
The "installed" branch — the rebuilt-box shape v0.196.0 was written for — took its first real run.
### Step 9b — the apply kept it
After the target was reconfigured, the on-disk key was still `8a9e33aa4da6…`. `WriteOffboxSecrets`
found the file present and kept it, exactly as documented.
---
## 3. The wall — three blockers, each measured
### (a) R-193, reconfirmed live: a rebuilt controller cannot configure its off-site tier
```
[INFO] [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.197.0)
[WARN] [offsite-apply] reconcile: offsite-apply: consume one-time password:
no unconsumed offsite password (already consumed or none provisioned)
(retries on next config refresh/restart)
```
The ledger, measured: `demo-hp` one-time secret created `07:11:51`, consumed `07:12:06` — **by the
previous controller**. The rebuilt one has nothing to consume and retries forever. Exactly the state
demo-hp sat in for 25 hours on 2026-08-03.
**Remedy:** an operator Re-issue. Performed here through the designed endpoint
(`POST /configs/demo-hp/offsite-reissue`, HTTP 303), after which the target configured normally.
### (b) R-196, measured live — and it lands squarely on the recovery path
```
host_escrow (demo-hp-bb76ea), after the Re-issue:
stale_at = 2026-08-04 20:15:49 ← set
restic_pw_sha256 = 8a9e33aa4da6… ← UNCHANGED. The key did not move.
```
**The escrow was marked stale while it perfectly covers the box's current key** — the recovered one.
That is R-196's false staleness, and here it is not a cosmetic lie: a stale escrow makes the hub
**withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending →
escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses to run any off-site backup**.
> **The remedy for (a) disables the recovery it was needed for.** The only documented way to clear a
> stale escrow is a fresh ceremony — which supersedes the identity blob, i.e. **destroys the very key
> being recovered**. Under hub v0.93.0 the old blob is now *retained*, but nothing can serve a
> superseded blob back (R-199's inventory: the endpoint serves the CURRENT row only).
**Superseded rows still number 2** — no ceremony was run tonight. The key is intact.
### (c) The claim gate — undocumented as a recovery step
```
[DEBUG] [web] claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)
```
A rebuilt box is **unclaimed** (`claimed = None`, fresh `settings.json`), and the claim gate correctly
intercepts every non-claim route. **So no controller endpoint can be driven at all** — not the manual
escrow confirm, not the backup trigger — until the customer re-claims the box. Both POSTs returned 302
to the claim page and neither reached its handler; `last_run` stayed null, which is why the earlier
reading of "the run was refused" needed this second look to be accurate.
This is correct behaviour and it is **not wrong** — but it is a step in the customer's recovery journey
that appears in no design document, and it comes *before* anything else can happen.
---
## 4. How it was finished, and the fourth undocumented step
The unattended run stopped at the wall in §3 rather than improvising further at 22:35 (§8.10). With the
operator present the remaining path was walked, and it took **four steps a customer would have to
discover**:
1. **Re-issue** the offsite credential (hub, operator-only) — otherwise the box can never configure its
tier. §3(a).
2. **Re-claim the box.** The local escape hatch (`--print-reset-code`) mints a code — **and does not
work on its own**: it writes the new hash to `settings.json` while the RUNNING controller keeps its
old copy in memory, so `effectiveClaimCode()` never sees it and every attempt returns
*"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming.** Two
claim attempts failed on this before it was diagnosed; the generation counter reached 6.
3. **Confirm the escrow** manually (`/backup/offbox/confirm-escrow`) → `escrowed`, so `OffboxRunnable`
permits a run. **Not a ceremony** — the identity blob was never superseded (still 2 retained rows).
4. **Restore with `mode=full`.** The default `mode=unit` returns the recovery unit and **not** the
customer's files.
**None of these four is in any design document, and the drill needed all four.**
## 5. §5's five conditions, as recorded before the wipe
| # | Condition | Evidence |
|---|---|---|
| 1 | §4 confirmed, sentinel **listed by name** | snapshot `e6132ae5` (19:36:26) — `-rw-r--r-- 1000 1000 181 … /DRILL-SENTINEL.txt` |
| 2 | deliberate rollback archive, **verified** | `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst`, 1 606 765 083 B; **full zstd stream read OK** (4 867 573 760 B uncompressed); the sentinel confirmed **inside** it |
| 3 | §3's option | see §6 |
| 4 | demo-felhom untouched | `health=ok`, `escrow_state=escrowed`, not touched at any point |
| 5 | free space | nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 % |
**A precondition had drifted and was repaired before the wipe, not worked around.** The staged
snapshot no longer contained the sentinel: the afternoon's `[WARN] mandatory data path missing`
experiment produced a *later* same-day calibre-web snapshot, and restic's `forget --keep-daily 7
--group-by host,tags` had pruned the good one in its favour. The fixture on disk was correct, so one
backup re-established it and it was re-verified by listing. **Lesson worth carrying: a good snapshot is
not durable against a later bad run on the same day.**
## 6. §3 — the recovery code
**Option B as already in place, with a strict improvement: no new copy was created, so nothing needed
shredding.**
The operator placed `R_DEMO-HP` in their own `~/.config/credentials` on DooPlex (mode `0600`) two
sessions ago, deliberately, for this purpose. This session read it from there and **piped it to stdin**
for each of the two invocations that needed it. It was never an argument, never exported, never written
to a second file, and never logged.
**Nothing was shredded, and that is the point:** the operator's own permanent store is theirs, not a
drill artefact, and destroying it would have destroyed their record. Because no additional copy was
made, there is nothing left behind to prove gone — a stronger position than option B's
create-then-shred.
**Verified afterwards** with the planted-copy positive control, exactly as on 2026-08-04 afternoon —
see §8.
---
## 7. The state the box is left in — healthy and re-armed
```
controller felhom-controller:0.197.0, healthy, CLAIMED
apps privatebin opengist calibre-web filebrowser cloudflared traefik — all serving
escrow_state escrowed (no ceremony was run; identity_blob untouched)
repo key 8a9e33aa4da6… THE RECOVERED KEY — and the repository opens with it
off-site 3 snapshots, 42 026 B, repo_state null; all three apps re-toggled for off-site
sentinel restored to its live location, sha256 643166269103a25c… — verified
```
The verification copy under `backups/offsite-restore/calibre-web/` is **left in place as evidence**.
## 8. R persisted nowhere — searched, with a positive control
Swept the agent journal, the controller container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`,
`/root` on demo-hp for the recovery code: **0 hits**, and **0** leftover `felhom-idesc-*` unseal
staging directories. A planted copy was found by the same sweep (**1**) and not found after shredding
(**0**), so the instrument is shown sensitive rather than assumed to be.
---
## 9. Teardown — three layers
| layer | state |
|---|---|
| the guest | **Nothing torn down.** The wipe is the evidence; the apps are serving; the recovered key is in place. The scratch band is empty — no restore-test guest was created tonight. |
| the host | `vzdump` snapshot LVs removed cleanly by the archive job (`snap_vm-9201-disk-0/1_vzdump` both released). One new archive, 1.6 GB, on `nvme-1tb` (883 GB free). Nothing deleted. |
| the hub | **No new customer records.** `demo-hp` is the same customer row throughout — the rebuild was a controller-data wipe, not a re-enrolment, so nothing accumulated. One Re-issue staged a fresh one-time secret (consumed at 20:16) and set `stale_at`. The two superseded escrow rows are **unchanged** — no ceremony ran. |
**Nothing was deleted on the storage endpoint**, including the ~1.2 GB of previously-orphaned
ciphertext. Ruled, still owed, and deliberately not ridden along with a drill.
## 10. Part 2 — not run
Its gate is *"the drill PASSED; its evidence is written down; and there is time."* The first two are
now met. **The third was not** — it was 23:15 after a full destructive cycle, and Part 2 is a second
wipe. It is deliberately left for a session that starts fresh, with drill 1's result already recorded
so a second wipe cannot overwrite it.
**R-198's retention therefore remains unit-proven only.** Nothing has yet superseded a key in
production, so the retention's live behaviour is still unobserved — and it is now the last unproven
link in the chain this drill exercised. The check is cheap: wipe again, do NOT recover, let a ceremony
seal a different password, then read the superseded row's `identity_blob`.
@@ -0,0 +1,311 @@
# DRILL RECORD — R-201, the wipe-and-recover proof: **PREPARED, HALTED BEFORE THE WIPE**
**Date:** 2026-08-04 · **Box:** `demo-hp` (Tier 0, the designated drill host) · **Nothing was wiped.**
**Outcome:** the drill did not reach its verdict. It was halted at step 4 by a defect that makes the
verdict unobtainable — and that defect is worth more than the drill.
> **THE HEADLINE.** A customer-declared **mandatory** data directory was **silently absent from the
> off-site snapshot**, while the backup reported `ok` with three snapshots. The only trace is one
> `[WARN]` line inside the controller container. The hub, the card and the counters all say the backup
> succeeded. → **R-203**
>
> **Nothing irreversible happened.** The wipe never ran. `demo-hp` is left in a BETTER state than it
> started: it now has a working off-site repository (it had an orphaned one), three snapshots, and a
> sentinel file on disk.
---
## 1. The verdict — not reached, and why that is the correct outcome
The drill's pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established that
the sentinel is not in the off-site snapshot at all.** Wiping the box would therefore have:
- destroyed the sentinel, which exists only on that box;
- proven nothing about recovery, because there would be nothing to recover;
- and done so *after* the point of no return.
The runbook's own rule applies: *"If this session finds itself writing code beyond Part 0, stop. That
means a precondition was wrong, and the finding outranks the drill."* A precondition was wrong. It was
one the runbook's P1P6 table did not contain, because nobody knew to look for it.
**Sentinel sha256 (step 3), recorded and still on the box:**
`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`
at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt` (181 B).
---
## 2. R-203 — the defect that halted the drill
**Measured, twice, on the live box.**
| what | path | exists? |
|---|---|---|
| the app's live bind, where the customer's files actually land | `/mnt/sys_drive/userdata/media/books` | **YES** (the sentinel is here) |
| the path the off-site capture set treats as the mandatory directory | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** |
The controller's own log, verbatim:
```
[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite:
/mnt/sys_drive/felhom-data/userdata/media/books
[INFO] [offbox] backed up calibre-web (…/backups/primary/calibre-web, 0 mandatory path(s))
[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s
```
**The run reported `ok`.** `last_status: ok`, `last_success` stamped, `snapshot_count: 3`. Nothing
customer-visible, nothing hub-visible and nothing in any counter says the mandatory directory was
dropped. This is `CLAUDE.md`'s recurring shape — *a path the customer thinks is protected is not in the
snapshot* — and the code even has the right words for it in a WARN nobody reads.
**The mechanism, from source.** `NamespaceRoot(drivePath, inGuestDrive)`
(`appbackup/paths.go:28-33`) appends `felhom-data` **when the drive IS the system data path**
(`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`,
`backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is documented and computed
as **`<HDD_PATH>/userdata`** (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive`
and an app deployed at `HDD_PATH=/mnt/sys_drive`, those two produce different directories.
**And the same compose file used BOTH roots.** From `docker inspect calibre-web`:
```
bind /mnt/sys_drive/felhom-data/userdata/import/calibre -> /cwa-book-ingest ← felhom-data root
bind /mnt/sys_drive/userdata/media/books -> /calibre-library ← NO felhom-data root
```
`${IMPORT_PATH}` resolved *with* the segment; `${USERDATA_PATH}` resolved *without* it. One deploy, one
template, two roots.
### What is measured and what is not — stated because the scope changes the fix
- **MEASURED:** the two paths disagree; the mandatory directory is absent from the snapshot; the run
reports `ok`; the only signal is a container-log WARN.
- **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a *supported* choice. It was chosen because
demo-hp's only registered drive, `Felhom-Share`, is a NAS and was **correctly refused** as an app
namespace (R-108's `RefuseAsAppNamespace`, working as designed). `/mnt/sys_drive` was **accepted**
(HTTP 202) rather than refused.
**Either branch is a defect, which is why this is filed regardless:**
- if the system drive **is** a supported app namespace → the userdata path resolution is wrong for
every app deployed on it, and their mandatory directories are silently unprotected;
- if it is **not** supported → the deploy accepted a namespace it should have refused, exactly as it
refused the NAS one call earlier, and the refusal that exists is not reaching this case.
**What must NOT be concluded from this drill:** that off-site backups are broken generally. The two
pre-existing apps (`opengist`, `privatebin`) declare **no** mandatory userdata paths — everything they
own is in named volumes — so they are unaffected, and their snapshots are real.
---
## 3. Preconditions, each measured
| # | Precondition | Result |
|---|---|---|
| **P1** | operator holds the recovery code | **PASS**`R_DEMO-HP`, recorded by the operator in the DooPlex credentials file (`~/.config/credentials`). *The code itself appears nowhere in this record.* |
| **P2** | demo-hp on agent v0.125.0 + controller v0.196.0 | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed the published v0.125.0 (sha `f7d8339b53d9…`, verified against the release) and controller v0.196.0. `age` present at `/usr/bin/age`. |
| **P3** | the current `identity_blob` seals the repository under test | **PASS with a caveat that reshaped the drill**`identity_blob` = 572 B, `restic_pw_sha256` = `8a9e33aa4da6…`. But **no repository existed under that key** (see §4). |
| **P4** | a deliberate, verified whole-guest archive as rollback | **NOT TAKEN** — deliberately. It is only needed for the wipe, and the wipe did not happen. |
| **P5** | demo-felhom untouched and healthy | **PASS** — not touched at any point. |
| **P6** | space | **PASS**`felhom-backup` 927 GB free, `local-lvm` 30.9 % used, guest 64 GB free. |
---
## 4. Step-by-step, with every observable
### Step 1 — starting state (hub, read-only)
`demo-hp-bb76ea`: `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`, created `2026-08-04T11:11:37Z`, `escrow_state: escrowed`.
Report: `snapshot_count: 0`, `repo_size_bytes: 0`, **no `last_run`/`last_status` at all** — the shape of
a controller that has never run an off-site backup in this lifetime.
### Step 1b — the repository was ORPHANED, and this is the first live proof of a prediction
The 2026-08-04 spike predicted from source — and could not measure — that a rebuilt box's next
off-site run would hit a **third** outcome: neither reattaching the old snapshots nor silently starting
a fresh history, but **refusing**. Measured here, twice over.
**Read-only probe first** (`restic cat config` with the current key, writes nothing):
```
Fatal: wrong password or no key found
```
— the exact string `classifyResticProbe` maps to `"orphaned"`.
**Then the real run, through the customer UI endpoint** (`POST /backup/offbox/run`):
```
[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous,
no-longer-available key; runs will skip until reset
[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset)
[INFO] Event pushed: offbox_repo_orphaned (warning) — A távoli mentési tároló elárvult …
```
`repo_state: "orphaned"`, `orphaned_at: 2026-08-04T12:37:30Z`, `last_status: "error"`, the orphan card
rendered on `/backups/remote`, and the event reached the hub (HTTP 200).
> **The system stopped and said so. It did not quietly start a new history over the old one.**
> That closes R-193's open Q3 — and it is the good half of this month's story.
**Why the repository was orphaned:** the 15 snapshots / 40.9 MB at `/home/felhom-repo` were written
under key `8e03eddf9ff7…` before the 2026-08-03 rebuild. That key survives only in superseded escrow
row **id 3**, which carries **`identity_blob` = NULL** — it was superseded at `2026-08-04 07:15:36`,
**four hours before** hub v0.93.0 fixed the retention. Unrecoverable, permanently, with or without a
recovery code.
### Step 1c — the reset (operator-authorised)
The orphan card's own reset, confirmed by the operator during the session:
```
[WARN] [offbox] resetting orphaned repo (operator-confirmed (claimed)):
move-aside /home/felhom-repo -> /home/felhom-repo.orphaned-20260804, then re-init
[INFO] [offbox] orphaned repo reset complete — old history set aside at
/home/felhom-repo.orphaned-20260804 (move-aside, not deleted); fresh repo initialized
[INFO] Event pushed: offbox_repo_reset (info)
```
**Nothing was deleted.** The reset path had never run in anger before; it works.
### Steps 23 — the recovery code and the sentinel
The operator ran the ceremonies for **both** boxes earlier the same day and saved the codes. Measured
hub-side: the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z`
(demo-felhom), and `restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated
them as same-password re-ceremonies: **no superseded row, no `offsite_repo_key_changed`**. That is
v0.93.0's Scenario E, live.
**A file-leg app had to be deployed**, and this is a precondition the runbook did not anticipate:
neither off-site-toggled app on the box (`opengist`, `privatebin`) has a restorable file leg — both
keep everything in **named volumes**, which the off-site tier backs up as tars but the customer restore
flow **never unpacks** (`offbox_reconstitute.go`). A sentinel in either would have been unrecoverable
by design.
`calibre-web` was chosen: it declares `userdata: media/books class: mandatory`, and it is a
single-container app. Deployed through the real API (`POST /api/stacks/calibre-web/deploy`, HTTP 202),
toggled for off-site, and a Tier-1 recovery unit captured
(`Recovery unit captured for calibre-web → …/backups/primary/calibre-web`).
Sentinel written, `sync`ed, hashed: **`643166269103a25c…`**, 181 B.
### Step 4 — the pre-wipe off-site backup: **`ok`, and wrong**
```
last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped
snapshot_count 3 · repo_size_bytes 30 636 · repo_state null
```
Three snapshots, three apps, 37 s — and **the sentinel is in none of them**, per §2.
### Steps 511 — NOT RUN
Step 5 (verify the pre-wipe snapshot contents) would have confirmed the absence a second way; it is
moot given §2. Steps 611 (archive, wipe, reinstall, recover, install, restore, compare) were **not
started**. The §7 STOP was never reached, because the drill failed its own precondition first.
---
## 5. Part 0 — shipped, tested, deployed (R-200's plumbing half)
`--recover-offsite-install` (controller **v0.196.0**, commit `1b1366b`): same fetch → unseal → extract
path as `--recover-offsite-check`, same STDIN discipline for R, but it **places** the recovered
password via `InjectOffboxPassword`.
- **Confirmation is a second invocation.** Without `--confirm-install` it prints both hashes and writes
nothing. A single interactive prompt would have had to share stdin with R.
- **Three outcomes, named distinctly:** *installed* (no local password — the rebuilt-box shape),
*unchanged* (identical key already present, nothing written), *refused* (a DIFFERENT key present;
installing would clobber the key the current repository is encrypted under — exit 2, no force offered).
- It re-reads the file after writing rather than trusting the call's return.
**Tests + red-proof.** `go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK.
Removing the confirmation gate makes the dry run write the password and fails
`TestRecoverAndInstall_InstallsOnABareBox` with *"the DRY RUN wrote the password"*. The R-persistence
test carries a **positive control** — a planted copy of the code is found by the sweep, then removed
and not found — because an absence check is worth only what its sensitivity is.
**It was deployed to demo-hp and never exercised against a live recovery**, because the drill halted
before step 9. Its unit proof stands; its live proof does not exist.
---
## 6. What this drill did and did not establish
**Established, live, for the first time:**
1. A rebuilt box's off-site repository is **orphaned and the run refuses** — the spike's predicted
third outcome, measured. It does not silently start a fresh history.
2. The **orphan reset works**: move-aside, never delete, fresh repo initialised, event pushed.
3. **A customer-declared mandatory data directory can be silently absent from the off-site snapshot
while the run reports `ok`** (R-203).
4. demo-hp's pre-rebuild off-site history is **permanently unrecoverable** — its key was destroyed four
hours before the fix that would have kept it.
**NOT established — and unchanged from before this session:**
- **No file has ever been restored from an off-site backup after a wipe.** R-201's question is still
open, and its pass condition is unchanged.
- Part 0's install path has never run against a live recovery.
- The v0.93.0 `identity_blob` retention is **still unit-proven only** — nothing in this session
superseded a key, so nothing exercised it.
---
## 7. State left behind, and teardown
**Deliberately not torn down** — this is evidence, and the box is better off than it was:
| layer | state |
|---|---|
| the guest | `calibre-web` deployed and running, off-site-toggled, sentinel file in place. **Kept** as the drill fixture for the resumed run — it is the only app on either demo box with a restorable file leg. |
| the off-site repository | fresh, working, 3 snapshots, 30 636 B. The old 15-snapshot history is **set aside**, not deleted, at `/home/felhom-repo.orphaned-20260804`. |
| the host | agent v0.125.0, controller v0.196.0, `pvesm` unchanged apart from normal usage. |
| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` events recorded for `demo-hp`. No scratch customer records were created — **nothing was reinstalled**. |
**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — that act is ruled and still owed,
and §8.3 of the runbook forbids riding it along with a drill. The reset added
`/home/felhom-repo.orphaned-20260804` (≈41 MB) to what is set aside.
**No `R` persisted anywhere** — the recovery code was never used in this session. It was read only to
confirm the key exists in the credentials file; no unseal was performed on demo-hp.
---
## 7b. UPDATE, same day — the blocker is FIXED and the drill is ready to resume
Controller **v0.197.0** shipped both halves of R-203:
- **the paths agree.** On demo-hp the live bind moved from `/mnt/sys_drive/userdata/media/books` to
`/mnt/sys_drive/felhom-data/userdata/media/books` — the directory the capture set looks in. The
capture log went from `0 mandatory path(s)` to **`1 mandatory path(s)`**.
- **`ok` means it.** A run that cannot capture a MANDATORY directory now reports **`incomplete`**,
names the app and the folders, and raises the operator digest — instead of `ok` with a warning
beside it.
**And the sentinel is in the snapshot, listed by name:**
```
$ restic ls -l latest --tag calibre-web
-rw-r--r-- 1000 1000 181 2026-08-04 12:53:06 /mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
```
sha256 **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** — byte-identical to §1,
verified after the fix's migration moved the file to the corrected directory.
**One live check could NOT be reproduced, and is recorded rather than claimed.** Hiding the books
directory to watch the `incomplete` verdict fire on hardware did not work: the running container's
bind mount **recreated** the directory, so `os.Stat` succeeded and there was no gap. That is itself
worth knowing — a bind-mounted directory cannot easily be "missing" while its app runs, so the
mandatory-gap condition arises in practice when the path resolves somewhere the app never binds (the
R-203 case), not when a live app's own directory vanishes. The verdict is proven by the run-level
test, which drives the real `RunOffboxBackup` and asserts both `incomplete` and the operator signal.
The fixture was restored and the sentinel re-verified at the same hash.
## 8. To resume the drill
1. **Fix or scope R-203.** Until the mandatory userdata path lands in the snapshot, no sentinel can
survive a wipe and the drill cannot reach its verdict.
2. Re-run steps 45 and confirm the sentinel IS in the snapshot — by listing it, not by a green status.
3. Then P4 (the deliberate archive), the §7 STOP, and steps 611 as written.
Everything else is already in place: the code, the versions, the recovery code, the working repository,
the file-leg app and the sentinel.
@@ -0,0 +1,531 @@
# RECON — can a customer actually get their backups back?
**Date:** 2026-08-04 · **Items:** R-193, R-192 (+ new: R-198…R-201) · **Class:** recon — **no production code**
**Session scope:** read-only everywhere. Nothing built, deployed, wiped, rotated, re-issued or deleted.
> **Headline, before the detail.** The chain is **not** assembled, and the reason is worse than "two
> links are manual". **The hub's superseded-escrow retention — the feature built for exactly this
> incident — does not retain the offsite repository password.** `host_escrow_superseded` has no
> `identity_blob` column, and `demoteCurrentEscrowTx` copies only the K-escrow blob. So the last copy
> of a repo password is destroyed by the **next escrow ceremony** — the very act the system tells the
> customer to perform when their escrow goes `pending` after a rebuild. The recovery window closes
> because the remedy closes it.
---
## 1. Confirmed baselines (re-read on arrival)
| Repo | `main` @ commit | tree | Version |
|------|-----------------|------|---------|
| `felhom-agent` | `856a127cd60b0be797867c162f0ae49c7b64993e` | clean | v0.124.1 |
| `felhom-controller` | `0887fd676d608e7c103fa13b3e87536ad2481811` | clean | v0.194.0 |
| `felhom.eu` | `d26f49ad680e2be2ce2888fd58dff7dee886e4ad` | clean | hub v0.92.0 |
All three match the prompt's cited commits exactly. `git status --porcelain` empty in each.
**Highest register ID: R-197** (`OPEN-ITEMS.md` R-196/R-197 were minted by yesterday's spike;
R-194/R-195 by the R-190 arc). Grepped across `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`,
`CONTEXT.md`. New items in this session therefore start at **R-198**.
---
## 2. Instruments — what each can and cannot answer
| # | Instrument | Exact read | Known limit |
|---|---|---|---|
| I1 | source | the three repos at the commits above | the authority; every claim below cites a file:line |
| I2 | hub SQLite snapshot | `kubectl -n felhom-system exec pod/hub-6775889d95-gc8sk -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | the `-wal` is load-bearing (copied; `PRAGMA integrity_check``ok`) |
| I2-fresh | freshness proof | `MAX(host_reports.received_at)` = `2026-08-04 10:00:40`, `datetime('now')` = `2026-08-04 10:03:48` | **positive observable**: newest row 3 m 08 s old, not "the query did not error" |
| I3 | live filesystem, demo-felhom guest 9201 | `ssh felhom-pve "pct exec 9201 -- ls -la --time-style=full-iso …"` | read-only `ls`; mtimes only, no file contents read |
| I4 | caller census | `grep -rn <symbol> --include=*.go` across all four repos | proves absence of a caller in Go; templates checked separately |
**Not used, deliberately:** no ceremony, no re-issue, no rotation, no `selftest=identity-consume`, no
write of any kind, nothing touched on the storage endpoint.
---
## 3. Q1 — the chain, link by link
From a dead node to an open repository. **Status vocabulary as specified.**
| # | Link | Component / function | Who invokes it | Status |
|---|---|---|---|---|
| 1 | repo password is minted | controller `Manager.WriteOffboxSecrets` → mint at `internal/backup/offbox.go:392-401`; generator `:405` | `ApplyOffsiteTarget` (`offbox.go:474`) ← `ConfigureOffbox` ← the apply-bridge (`offsiteapply.go:221,256`) | **PROVEN-LIVE** — fires on every fresh data dir; measured twice this week (I2 hash change, I3 mtime) |
| 2 | password staged to the agent | controller `PushOffboxPasswordForEscrow` (`offbox.go:530`) → agent `POST /escrow/stage-secret` (`localapi/server.go:518`) → `escrow.StagedResticPasswordPath()` = `/var/lib/felhom-agent/escrow-stage/restic_repo_password`, 0600 (`identity.go:44-47`) | the controller enable/ceremony path | **PROVEN-LIVE** |
| 3 | sealed under R at the ceremony | `AttachResticPassword` (`identity.go:73`) → `WrapIdentityBundle``WrapIdentity` (`identity.go:114`, `age -p -a`, scrypt + ChaCha20-Poly1305) | agent `--selftest=escrow-create`, driven by the controller wizard (`/escrow/ceremony`, `localapi/server.go:525-527`) | **PROVEN-LIVE** — customer-facing wizard first live firing 2026-07-18 (`00-capability-map.md`, escrow row) |
| 4 | stored by the hub | `PUT /api/v1/hosts/<id>/escrow` (`api/handler.go:239`, `handleHostEscrowPut :1104`) → `host_escrow.identity_blob` | agent `main.go:2937` | **PROVEN-LIVE** — measured: `identity_blob` present, 572 B, for **both** hosts (I2) |
| 5 | **old blob retained on supersession** | `demoteCurrentEscrowTx` (`store/store.go:2547-2556`) → `host_escrow_superseded` | `SaveHostEscrow`, `DeleteHost` | **BROKEN FOR THIS PURPOSE.** The INSERT copies `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256`**not `identity_blob`**, and the table has **no such column** (`PRAGMA table_info`, I2). The K-escrow survives; **the restic repo password does not** |
| 6 | served back to a re-enrolling box | hub `handleReEnroll` (`api/dr.go:101`) and `handleGetRestoreDirective` (`:155`), returning `identity_escrow_b64`; recovery-mode gated, armed with the **global operator key** (`:34`) | **NOBODY.** I4: zero callers in `felhom-agent` (no `ReEnroll` symbol, no `/re-enroll` or `/restore-directive` string anywhere in the agent's hub client, `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`); zero in the hub UI (no template mentions them); zero in `scripts/` or any runbook | **implemented-never-exercised, and CLIENTLESS.** The only documented retrieval is by hand: `sqlite3 "SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …"` on a `kubectl cp`-ed `hub.db` (project memory, 2026-07-04 S5 prep) |
| 7 | unsealed with R | `UnwrapIdentityBundle` (`identity.go:172`) → `UnwrapIdentity` (`:139`) | **`runSelftestIdentityConsume` only** (`cmd/felhom-agent/main.go:2845-2900`), reading R from `FELHOM_RECOVERY_CODE`. I4: no other production caller | **manual-only — the only caller is a `--selftest` mode** |
| 8 | the repo password extracted | none. The selftest writes the **whole bundle JSON** to `-keydest` 0600 (`main.go:2872`) and prints *"identity recovered (tunnel_token + pbs_token)"* — it does not mention, extract or route `restic_repo_password` at all | a human reads the JSON | **missing** |
| 9 | placed on the rebuilt controller | `InjectOffboxPassword` (`offbox.go:541`) | one caller: `offboxInjectPasswordHandler` (`web/offbox_handlers.go:189`) at `POST /backup/offbox/inject-password` (`web/server.go:510`). **I4: no template in the repo contains that path or any form posting to it** | **manual-only, and UI-unreachable** — the handler exists, the form does not. A person must POST it by hand with a valid session cookie + CSRF token |
| 10 | the existing repo opens | `ensureOffboxRepo` (`offbox.go:653`): `restic cat config` → success clears `RepoState` | the scheduled/manual offbox run | **implemented-never-exercised** on a *recovered* password (exercised constantly on a live one) |
| 11 | a backup or restore proves it | — | — | **never** |
### The chain in one sentence
Links 14 are real and proven. **Link 5 silently drops the payload.** Links 69 are each individually
manual, and link 6 has no client at all. Link 11 has never happened.
---
## 4. Q2 — where the chain stops today
**The first link that is not automatic is #5, and it is not merely manual — it is lossy.**
Two distinct stopping points, and the order matters:
**(a) If the box has NOT re-escrowed since the rebuild** — the current `host_escrow.identity_blob`
still seals the *old* password. Recovery is possible in principle, and a person must:
1. Arm recovery mode on the hub (`PUT /api/v1/admin/hosts/<id>/recovery-mode`, global operator key) —
or skip it entirely, because nothing consumes the endpoint it gates;
2. `kubectl cp` / `exec cat` the hub DB and `sqlite3 … writefile('/root/idblob', identity_blob)`;
3. copy the blob to a machine with the agent binary and `age`;
4. `FELHOM_RECOVERY_CODE=… felhom-agent --selftest=identity-consume -blob /root/idblob -keydest /root/bundle.json`;
5. open `bundle.json` by hand and read `restic_repo_password` out of it;
6. log in to the customer's dashboard and `POST /backup/offbox/inject-password` **by hand** (curl with
session + CSRF) — there is no form;
7. re-run the apply so `WriteOffboxSecrets` finds the file present and keeps it;
8. trigger a run and hope `restic cat config` succeeds.
Eight manual steps, five of them requiring operator-tier access to the hub, one requiring the customer
to be present with R, and one (step 6) with no user interface at all.
**(b) If the box HAS re-escrowed** — which is what the system pushes the customer to do, because a
rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`,
`offbox.go:570`), and the card says *„Helyreállítási kód szükséges"* — then the old `identity_blob`
has been **overwritten** by `SaveHostEscrow` and **not carried** into the superseded row. At that point
**no procedure exists**, with or without R, at any level of effort. The password is gone.
Both demo boxes are in state (b) as of 2026-08-04 07:15:36 (demo-hp) and 07:20:08 (demo-felhom) — I2,
`host_escrow.updated_at`. This is an **independent, and much stronger, reason** the 51 orphaned
snapshots are unrecoverable than "nobody kept the recovery codes". Keeping R would not have helped.
---
## 5. Q3 — can a rebuilt controller know it should recover?
### RULING: **YES — the hub already tells it, on every single report, and the controller throws the answer away.**
**What the box knows at the mint decision.** `WriteOffboxSecrets` is called from
`ApplyOffsiteTarget`, driven by the apply-bridge with the hub's offsite **descriptor**, whose fields
are `host, user, port, repo_path, host_fingerprint, quota_gb` (`offsiteapply.go:198-204`). **Nothing
about escrow is in scope at that point.** Locally, the only signal is the one the mint branch already
reads: `os.Stat(m.offboxPwPath())` returning `IsNotExist` — which *is* the fresh-data-dir signal, at
exactly the right line, but cannot distinguish "never configured" from "rebuilt".
**What the box is nonetheless already told.** Every report ACK carries an `escrow` object
(`api/handler.go:504-510`), consumed as `report.EscrowStatus`
(`internal/report/escrow_confirm.go:19-23`):
```go
type EscrowStatus struct {
IdentityBlobPresent bool `json:"identity_blob_present"`
ResticPwSHA256 string `json:"restic_pw_sha256"`
CreatedAt string `json:"created_at"`
}
```
That is precisely *"the hub has a blob, and here is the hash of the password it covers"* — arriving
every report cycle, on a channel that already exists, with no new API and no new secret.
**And it is discarded.** `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:75-84`) returns
immediately unless `Pending()` or `Escrowed()` — both of which require `OffboxConfigured()` **and** a
non-nil target (`cmd/controller/main.go:580-590`). On a freshly rebuilt box, before the offsite apply,
there is no target, so the status is dropped on the floor. It is never persisted anywhere: I4 finds
exactly **one** reference to the ACK field in the whole controller
(`cmd/controller/main.go:641: escrowConfirmer.Reconcile(resp.Escrow)`).
**Therefore a fail-closed refusal IS implementable**, with no new hub endpoint:
- persist the last-seen ACK escrow status alongside settings (the precedent exists — `ClaimSync`
caches the hub's claim-code state into `settings.json` idempotently by generation,
`internal/report/claim_sync.go:39-53`, and it does so *set-only, never cleared on hub silence*,
which is the correct shape here too);
- at the mint branch, refuse when the password file is absent **and** the cached status says
`identity_blob_present && restic_pw_sha256 != ""`.
**One ordering caveat, stated rather than assumed.** On a truly fresh box the apply can race the first
ACK. Measured on demo-hp: first post-rebuild report `2026-08-03 06:12:19`, offsite apply
`2026-08-04 07:12` — the ACK preceded the apply by 25 hours, but that gap was an incident, not a
design. A fail-closed rule handles the race correctly by construction: refuse to mint until the escrow
status is known, and retry next cycle (the apply-bridge is already built to retry and to persist
nothing on a partial failure, `offsiteapply.go:1-6`).
**Corroborating measurement that the data dir really is fresh on a rebuild** (I3, demo-felhom guest
9201, `/var/lib/docker/volumes/felhom-controller-data/_data/data/`):
| file | mtime |
|---|---|
| the data directory's parent | `2026-08-03 07:17:25` |
| `encryption.key` (32 B) | `2026-08-03 07:17:26` |
| `offbox/` | `2026-08-03 07:18:02` |
| `offbox/repo_password` (64 B) | `2026-08-03 07:18:02` |
The volume, the app-secret encryption key and the repo password are all newborn, 36 seconds apart.
Corroborated hub-side (I2): `json_extract(report_json,'$.claimed')` reads **1 → 0 → 1** across the
rebuild on both boxes — demo-hp `0` in 5 reports `2026-08-03 06:12:19``06:21:03`; demo-felhom `0` in 4
reports `07:17:54``07:19:46`. `Claimed` is set only by `SetClaimed()`, whose sole caller is the claim
submit handler (`web/claim.go:349`), so the return to `1` was a **human re-claim**, not automation.
> **Note, not acted on:** a fresh `encryption.key` means every pre-rebuild `app.yaml` `ENC:` secret is
> undecryptable — the C-1 circular dependency in `_recovery-inventory-2026-07-28.md` fired on both
> demo boxes this week. That is outside R-193's scope and is recorded as an observation (§13).
---
## 6. Q4 — what the stale-escrow detector actually covers
### RULING: **the prompt's premise conflates two different mechanisms. `escrow_confirm.go` DOES fire on a mint. But it fires into a log file, and the remedy it prescribes destroys the last copy of the old key.**
**The conflation, stated because it matters.** Yesterday's spike's sentence *"wired to the one path
that doesn't change the key and absent from the one that does"* is about the **hub's `escrow_stale`
event**, emitted by `ReissueCredentials` (`hub/internal/offsite/offsite.go:198-201`) — that is R-196,
and it stands. It is **not** about `controller/internal/report/escrow_confirm.go`. Two different
detectors; only one of them was ever claimed to be misplaced.
**What `escrow_confirm.go` does on a mint, from source.** After a rebuild the apply creates a target
with `EscrowState: "pending"` (the `cur == nil` fallthrough, `offbox.go:482-494`). So `Pending()` is
true, the pending branch runs (`escrow_confirm.go:86-107`), the hub's hash (sealing the OLD password)
is compared against `LocalHash()` (the NEW minted one), they differ, and it logs:
```
[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password
(hub hash %.12s… != local %.12s…) — run the escrow ceremony … staying pending
```
**So: confirmed, it fires.** The detector is not absent from the mint path.
**Three limits, each of which matters more than the fact that it fires.**
1. **It is a controller-local log line.** No event, no e-mail, no hub-side signal. `StaleBlob()` — the
flag the web card renders — is set **only** by the escrowed branch (`reconcileEscrowed`,
`:130-165`); the pending branch sets no display flag. The operator learns nothing.
2. **Its prescription is the destructive act.** *"run the escrow ceremony"* → a new ceremony →
`SaveHostEscrow` overwrites `host_escrow.identity_blob``demoteCurrentEscrowTx` does not carry it
**the old repo password ceases to exist**. The detector correctly identifies the situation and
then tells the customer to close the door.
3. **It cannot distinguish the two situations that need different answers** — "you re-keyed
deliberately" and "your machine was rebuilt and your history is about to be orphaned" produce the
identical warning.
So: **less needs building than the prompt feared for the *detection*, and more needs building for the
*consequence*.** The comparison exists; what does not exist is a path from that comparison to either a
signal or a recovery.
---
## 7. Q5 — is the superseded-escrow retention real, and what does it retain?
### RULING: **the retention is real, it retains the wrong blob, and nothing can read it back.**
**What it retains — source and live, agreeing.** `demoteCurrentEscrowTx` (`store/store.go:2547-2556`):
```sql
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
FROM host_escrow WHERE host_id = ?
```
`identity_blob` is not in the column list. Live schema (I2, `PRAGMA table_info`):
| table | columns |
|---|---|
| `host_escrow` | `host_id, blob, key_fingerprint, posture, created_at, updated_at, **identity_blob**, directive_json, restic_pw_sha256, stale_at` |
| `host_escrow_superseded` | `id, host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at` |
**`blob` is the K-escrow — the PBS datastore encryption key**, wrapped by the PBS-native scrypt path.
**`identity_blob` is the age-wrapped `IdentityBundle`** — the one carrying `ResticRepoPassword`
(`felhom-agent/internal/escrow/identity.go:34-39`), plus `TunnelToken`, `PBSToken` and
`WGPrivateKey`. Live sizes confirm they are different objects: both hosts' current rows hold
`blob` = 383 B **and** `identity_blob` = 572 B; both superseded rows hold `blob` = 383 B and nothing
else.
**So, exactly what a customer with the matching OLD recovery code could recover from a retained blob:**
- ✅ the PBS datastore encryption key **K** — i.e. the Plane-2 whole-guest offsite ciphertext;
- ❌ **not** the restic repository password — the Tier-3 offsite app-data history;
- ❌ not the tunnel token, not the PBS access token, not the WireGuard private key.
**And there is no read path.** I4: `ListSupersededEscrow` (`store/store.go:2617`) has **zero
production callers** — only two tests. The only production surface is `CountSupersededEscrow`, used
for a number on the host page (`web/hosts.go:573`) and a log line (`api/handler.go:1151`). The schema
comment says so itself: *"The ACK and restore-serving read `host_escrow` (the CURRENT row) — never
this table."* Retrieval is `sqlite3` on the hub pod, by hand, by the operator.
**The claim this contradicts.** `00-capability-map.md` (escrow-ceremony row) records hub v0.60.0 as:
> *"a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS
> it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code
> (turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable')."*
The same claim appears in the schema comment (`store.go:370-375`) and, in Hungarian, on the customer's
own orphan card (`controller/internal/web/templates/backups_remote.html:66,69`):
> *„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók
> lehetnek."* · *„A régi előzmény félretéve marad (nem törlődik), és a hozzá tartozó helyreállítási
> kóddal később visszaállítható lehet."*
**For the offsite restic repository — the incident it names — that is false.** The retained blob does
not contain the key that opens it. This is the **ninth** entry in `CLAUDE.md`'s table of comments
asserting an invariant the code does not provide, and the first that is also **customer-facing copy**.
**R-198**, and the capability-map correction in §9.
---
## 8. Q6 — has any part of this ever been exercised?
**Method:** read every audit, validation doc, findings file and REPORT that mentions escrow consume,
identity-consume, re-enroll or inject-password; then check what the cited evidence actually covers.
### Proven
| what | evidence | scope of the proof |
|---|---|---|
| ceremony (create + seal + upload) | `00-capability-map.md` escrow row; `tests/VALIDATION-n100-rehearsal-2026-07-18.md` S6 — customer drove the wizard, R claimed one-shot 16:56:39, auto-confirmed 16:56:41 | the **create** side only |
| identity-bundle round-trip with a real R on a secret-less box | `tests/slice10d-identity-restore-spike-findings.md` §1, 2026-06-10 — wrap→unwrap byte-identical (`sha256` match); wrong-R **fails closed**, no plaintext emitted | **PRE-FORK-4.** The bundle then was `{tunnel_token, pbs_token}`. `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** (`felhom-agent/CHANGELOG.md:2344`) — a month later. **The proof does not cover the field this whole question is about** |
| `ResticRepoPassword` survives the wrap/unwrap | agent v0.77.0 CHANGELOG: *"`IdentityBundle` round-trip carries `ResticRepoPassword` byte-exact + not-in-blob + wrong-R fails closed"* | **unit test only** |
| `InjectOffboxPassword` places and refuses correctly | `controller/internal/backup/offbox_test.go:952-990`, incl. the companion showing a fresh mint without inject | **unit test only** |
### Never exercised — named in those words
- **A blob has never been served to a box.** Link 6 has no client (§3).
- **A fork-4 bundle has never been unsealed with a real R outside a unit test.** The only production
caller is a `--selftest` mode; the one live consume ever prepared (S5 Part 4-B, 2026-07-04) was
explicitly **deferred and never run** — its own resume note still reads *"DEFERRED 2026-07-04, not
run"* (project memory; `06-offsite-connectivity.md:327` marks the destructive drill operator-gated).
- **A recovered repo password has never been injected into a controller.**
- **An existing offsite repository has never been reopened with a recovered password.**
- **No restore of any kind has ever been performed from a recovered secret.**
`_recovery-inventory-2026-07-28.md` already said most of this and said it accurately — A.2.7 records
*"the **consume** side is proven at spike level … the destructive in-place drill is explicitly
operator-gated and unrun"*, and C.1 row 1 says *"escrow **creation** proven live; **consume never
exercised destructively**"*. **That document is right and this session confirms it.** What it did not
catch — because it was written from `host_escrow`, not from the supersession path — is Q5.
**Capability-map correction owed:** not for a "recovery proven" claim (no row makes one), but for the
**retention** claim quoted in §7. See §9.
---
## 9. Q7 — the operator's ruled shape, assessed row by row
> **Operator ruling, 2026-08-04, verbatim:** *If a node is a fresh install AND the hub has a recovery
> blob, then the controller should yell that recovery is available, and provide a form for the customer
> to enter the recovery key. After unlocking the blob, the controller should show what will be
> recovered before proceeding.*
| The ruling requires | Exists? | What it needs, and what it costs |
|---|---|---|
| the controller knows it is a **fresh install** | **YES, at the right line** | `os.Stat(m.offboxPwPath())``IsNotExist` is already the mint branch (`offbox.go:392`). It is trustworthy as *"this controller has no repo password"*, which is the only fact the decision needs; it does **not** by itself distinguish first-ever install from rebuild — that is what the next row supplies. Corroborators exist but are weaker: `Claimed` is set-only and was measured re-set by a human within minutes (§5); a fresh `encryption.key` is a stronger signal but is not currently read by anything. **Cost: zero — the branch already exists.** |
| the controller knows **the hub has a blob** | **YES on the wire, NO in the code** | `EscrowStatus{IdentityBlobPresent, ResticPwSHA256, CreatedAt}` arrives on every report ACK and is discarded when no offbox target exists (§5). **Needs:** persist it (the `ClaimSync` set-only-by-generation pattern is the precedent, `claim_sync.go:39-53`). **Cost: small — controller only, one settings field, one reconciler, no hub change.** |
| it can **yell** rather than silently mint | **NO** | The refusal belongs **inside `WriteOffboxSecrets`, at the mint branch** — the one place that decides. Returning a new sentinel (`ErrOffboxRecoveryAvailable`, sibling of `ErrOffboxOrphaned`) propagates through `ApplyOffsiteTarget``ConfigureOffbox` → the apply-bridge, which already treats a failed configure as "persist nothing, retry next cycle" (`offsiteapply.go:5-6, 256`) — so a refusal is **fail-safe by construction**, not a new failure mode. Meanwhile the customer sees a card on `/backups/remote` (the orphan card at `backups_remote.html:60-70` is the exact precedent) and offsite runs stay blocked, which they already are (`OffboxRunnable`). **Cost: small-medium — controller only.** **Design hazard to name:** a first-ever install of a *re-used customer record* would also trip this; the refusal must be dismissible ("start fresh anyway"), and dismissal must be the act that mints. |
| a **form takes R** | **NO — and R is not a password** | The controller UI **emits** R (`backups_escrow.html:235-236`, `/backup/escrow/…/claim` returns it once) and has **never accepted** one. The only thing resembling an entry point is `POST /backup/offbox/inject-password`, which takes a **64-hex repo password** (`offboxRepoPwPattern`, `offbox.go:543`) — a different secret at a different layer — and **has no form in any template** (§3 link 9). **Cost: small for the form; the hard part is the next row.** |
| the blob is **unsealed** with R | **NO — and there is a real boundary** | Unsealing is `age -d`. `ageBinary = "/usr/bin/age"` (`identity.go:24`) and the code live in the **agent**. The **controller image does not ship `age`** (`controller/Dockerfile:56-70` — restic, openssh-client, sshpass, sqlite3, docker-cli; no age) and must not: the agent is the privileged tier by design. **The crossing already has a transport and a precedent:** the pinned local API carries `POST /escrow/ceremony` + `GET /escrow/ceremony/status` + `POST /escrow/ceremony/claim` (`localapi/server.go:524-527`) — i.e. the agent **already** handles R server-side for the create direction and hands the result back exactly once. **Needs: one new agent local-API endpoint** (`POST /escrow/unseal`: blob + R in, the single `restic_repo_password` field out, R never logged, R never persisted) — the exact mirror of the claim endpoint. **Plus** a way for the blob to reach the agent: the agent's hub client has no escrow **GET** (only the `PUT` at `main.go:2937`), and `dr.go`'s re-enroll is the wrong tool (it rotates the host API key and needs operator-armed recovery mode). **Needs: a second small hub endpoint**`GET /hosts/<id>/escrow`, per-host-key authed, serving opaque bytes to the box that owns them. Zero-knowledge is untouched: the hub still never decrypts, and the bytes are useless without R. **Cost: medium — three repos, two new endpoints, both narrow.** |
| **"show what will be recovered" before proceeding** | **the primitive EXISTS; the flow does not** | `restic snapshots --json` + `restic stats --mode raw-data --json` are already how the box counts snapshots and sizes the repo (`offboxRecordStats`, `offbox.go:1234-1265`). Both are **read-only against the repository** — nothing is written, no lock is taken, no `init`. So a preview can show: **snapshot count, per-snapshot time, host, tags (= app names), paths, and the deduplicated repo size** — all before committing. **Committing requires only that the recovered password be written to `offboxPwPath()`**, and even that is reversible in principle. **The clean shape:** unseal → hold the recovered password **in memory** → run the two read-only probes against the existing repo → render → and only on the customer's confirmation call `InjectOffboxPassword`. **Cost: small — one handler, one template, no new restic capability.** **Caveat, measured not assumed:** `offboxRecordStats` today also persists `RepoSizeBytes` into settings as a side effect; a preview path must not reuse it verbatim. |
| **proceeding** actually recovers | **NO end-to-end** | `InjectOffboxPassword` → next `WriteOffboxSecrets` keeps the file → `ensureOffboxRepo` probes and clears `RepoState`. Every piece is implemented; **none has ever run against a recovered password** (§8). **Cost: zero new code, but it is precisely what the drill in §10 has to prove.** |
### Things the ruling implies that do not exist, priced
| missing piece | cost | note |
|---|---|---|
| persisted ACK escrow status + fail-closed mint refusal | **S** | controller only; the `ClaimSync` pattern applies directly |
| the "recovery is available" card + dismissal | **S** | one template, one flag; the orphan card is the template precedent |
| a customer-facing **recovery-code** form (distinct from the password form) | **S** | the password form's handler exists but has no UI; both need building |
| agent local-API `POST /escrow/unseal` | **M** | new privileged surface; mirror of `/escrow/ceremony/claim` |
| hub `GET /hosts/<id>/escrow` (opaque, per-host key) | **S** | new read surface on the hub; zero-knowledge preserved |
| preview handler + template driven by read-only restic probes | **S** | primitive already exists |
| **carrying `identity_blob` into `host_escrow_superseded`** | **S** | one column, one `SELECT` list — **and without it the whole feature protects only boxes that have not yet re-escrowed** (→ R-198). This is a prerequisite, not a nice-to-have |
### The security question the ruling raises — stated, not answered
**What gates the surface today.** The controller dashboard is behind `RequireAuth`
(`internal/web/auth.go:52`): a bcrypt password (`effectivePasswordHash()` — customer-set in
`settings.json`, else operator-provisioned in `controller.yaml`), a 7-day session cookie, 5 failed
logins per minute then lockout, and `CsrfProtect` on every mutating route. The claim gate serves only
the claim page on an unclaimed box. So a recovery form would sit **behind the customer's dashboard
password**, on a controller reachable at whatever address that box's dashboard is reachable at.
**What a preview would expose to someone who guesses or steals R.** Not file contents — but the
snapshot list is not nothing: **backup dates and cadence, the app names (restic tags), the paths, and
the repository size.** That is a map of the household's digital life, and it is precisely the metadata
the zero-knowledge posture otherwise keeps out of everyone's hands including the operator's.
**Three specifics the operator should weigh, without this session choosing:**
1. **Guessing R is not the threat; holding the dashboard password is.** R is ten EFF words
(~129 bits, `00-capability-map.md` agent v0.93.0 note) and `age`'s scrypt is deliberately slow —
online guessing is not a route. But **anyone with the dashboard password can already restore live
offsite data** (`_recovery-inventory-2026-07-28.md` A.1.3A.1.6). The marginal new exposure is
confined to the *orphaned/previous* repository, which today nobody can read at all.
2. **The form is an oracle.** A correct/incorrect R answer distinguishable in the UI turns the
dashboard into a verification service for a stolen recovery code. `UnwrapIdentity` already
fails closed with a single generic error (`identity.go:157`) — the UI must not do better than that,
and should rate-limit like the login path does.
3. **R will pass through the agent.** Whatever the form does, R crosses the local API to a
root-adjacent daemon. The ceremony already does this in the other direction, so the precedent is
set — but it means R is momentarily present in agent process memory on the Proxmox host, which is
the same host D6 says the operator has root on. **This is the same trade as spike option (c), in a
smaller and time-bounded form**, and it should be decided alongside it rather than separately.
---
## 10. Q8 — the drill design
**Why a deliberate cycle.** No recovery code exists for either demo box, and — per §7 — even one would
no longer open the old repositories. The proof therefore requires creating R, keeping it, and walking
the full circle. **This section is the design; the drill is not run in this session.**
### Target and preconditions
| | |
|---|---|
| **Box** | **`demo-hp`** (HP t740, node `felhom-host`, host `demo-hp-bb76ea`). Tier 0 — disposable, and the designated drill host by the 2026-07-25 operator ruling. **Not demo-felhom** — it is the N100 that also carries the ISO/rehearsal lineage; leave one demo box in a known-good state throughout |
| **Guest** | the customer guest on that host (rebuild target) |
| **Data** | one deliberately identifiable app + a sentinel file whose content is a known string and whose sha256 is recorded before the wipe |
| **Prereq** | the box is healthy, offsite target configured and `escrow_state: escrowed`, and its **current** `host_escrow.identity_blob` seals the password of the repo that will be tested |
| **Duration** | ~34 h wall clock, of which ~45 min is operator-attended (steps 2, 5, 8, 9) |
| **Blocking dependency** | **steps 89 cannot be executed as designed today** — links 6, 8 and 9 of the chain do not exist as automation. The drill as written uses the manual substitutes (§4a) and **that is the point**: it measures the *existing* chain honestly before anything is built |
### The cycle
| # | Step | Who | Observable that must be recorded |
|---|---|---|---|
| 1 | Record the starting state: `host_escrow.restic_pw_sha256`, `identity_blob` length, `$.offsite.snapshot_count`, `repo_size_bytes` | CC | a hash and two counters, from the hub DB |
| 2 | **Run the escrow ceremony through the customer wizard and KEEP R** | **HUMAN** | R written to the operator's password manager under a named entry; **its storage location recorded in the drill report — the code itself never** |
| 3 | Deploy an app, write the sentinel file, record its sha256 | CC | sha256 + path |
| 4 | Run an off-site backup through the real UI endpoint; wait for completion | CC | `$.offsite.last_status = ok`, `snapshot_count` increased by ≥1, `last_success` stamped |
| 5 | **Verify the snapshot from outside**: `restic snapshots --json` against the repo | CC | snapshot id + tag + timestamp — the pre-wipe truth to compare against |
| 6 | **Wipe**: destroy the guest and reinstall it (the same procedure that produced the 2026-08-03 rebuild) | **HUMAN** decision, CC executes | new `controller_started`; `claimed` drops to 0; a new `encryption.key` mtime |
| 7 | Let the box come back and reach the mint. **Do NOT run a new ceremony** | CC | the controller log carries the `escrow-confirm` mismatch WARN (§6); `escrow_state: pending`; `host_escrow.identity_blob` **unchanged** (this is the assertion that keeps recovery possible) |
| 8 | **Recover**: export `identity_blob` from the hub, unseal with the kept R, extract `restic_repo_password` | **HUMAN** (R is theirs) | `identity-consume` exits 0; the bundle JSON contains a non-empty `restic_repo_password`; **the value is never printed or logged** |
| 9 | **Inject** the recovered password (`POST /backup/offbox/inject-password`, by hand — there is no form), re-apply, run an off-site backup | **HUMAN**-assisted | `restic cat config` succeeds; **`snapshot_count` ≥ the step-5 count + 1**, not 1 |
| 10 | **Restore the sentinel** through the customer restore flow and compare sha256 | CC | **byte-identical sha256 to step 3** |
| 11 | Record everything, tear down nothing that would hide the evidence | CC | drill report |
### The single observation that makes this a FAILURE rather than a pass
> **Step 10's sha256 does not match step 3's — or step 9 reports a snapshot count of 1.**
A snapshot count of 1 means a **new, empty history** was started over the old one: the repository did
not reopen, and every earlier snapshot is orphaned. **Explicitly: "the repository opened" is not the
pass condition.** `restic cat config` succeeding proves the password is right; it does not prove the
customer's data came back. Only the sha256 comparison does.
### Secondary failures worth recording separately (each is a finding, not a pass/fail)
- Step 7 finds `identity_blob` **changed** — something re-escrowed automatically → the recovery window
closes without a human act (would be new and serious).
- Step 8's bundle has an **empty** `restic_repo_password` — a pre-fork-4 or hash-less blob shape.
- Step 9's `ensureOffboxRepo` classifies `orphaned` despite the correct password → the classifier or
the transport, not the key.
### What could make it fail for uninteresting reasons
The `age` binary absent on the machine running step 8; the hub DB export producing a truncated blob
(use `writefile`, verify the byte length against `length(identity_blob)` = 572); the CSRF token in
step 9 (`felhom_session` is dropped by curl's cookie jar — project memory
`controller-api-headless-access`); and the R-71a settle gate deferring the apply in step 9 (wait for
its GO line rather than retrying).
### Rollback, and whether the loss is acceptable
**If the recovery does not work, what has been lost is:** demo-hp's guest, its app data since the last
whole-guest archive, and its off-site history. **On that box that is acceptable** — it is Tier 0,
explicitly disposable, and it is the box the target-selection runbook names for drills. **The
rollback** is a whole-guest restore from the local/PBS archive taken immediately before step 6, which
is a proven path on this fleet (restore-test passed on demo-hp's host tier in 109.3 s on 2026-08-04).
**Take that archive deliberately as part of step 6 and verify it exists before destroying anything** —
do not rely on the nightly.
**Not acceptable, and therefore not proposed:** running this on demo-felhom (leaves no healthy demo
box), on DooPlex (Tier 2, it *is* the recovery chain), or on any real customer's box.
---
## 11. Every query used
| # | Source | Query / read | Value read |
|---|---|---|---|
| 1 | git | `git -C <repo> rev-parse HEAD`; `git status --porcelain` | the three commits in §1; all empty |
| 2 | hub DB | `PRAGMA integrity_check` | `ok` |
| 3 | hub DB | `SELECT MAX(received_at) FROM host_reports; SELECT datetime('now')` | `2026-08-04 10:00:40` / `2026-08-04 10:03:48` |
| 4 | hub DB | `SELECT host_id, length(blob), length(identity_blob), substr(restic_pw_sha256,1,12), created_at, updated_at FROM host_escrow` | `demo-felhom-8363b5, 383, 572, c60c8bc737a6, 2026-08-04T07:20:07Z, 2026-08-04 07:20:08` · `demo-hp-bb76ea, 383, 572, 8a9e33aa4da6, 2026-08-04T07:15:36Z, 2026-08-04 07:15:36` |
| 5 | hub DB | `SELECT id, host_id, length(blob), substr(restic_pw_sha256,1,12), created_at, superseded_at FROM host_escrow_superseded` | `3, demo-hp-bb76ea, 383, 8e03eddf9ff7, 2026-07-23T10:01:17Z, 2026-08-04 07:15:36` · `4, demo-felhom-8363b5, 383, 48741892f0ef, 2026-07-21T08:38:33Z, 2026-08-04 07:20:08` |
| 6 | hub DB | `PRAGMA table_info(host_escrow)` / `(host_escrow_superseded)` | the two column lists in §7 — `identity_blob` present in the first, **absent from the second** |
| 7 | hub DB | latest report per customer: `json_extract(report_json,'$.claimed' / '$.offsite.escrow_state' / '.snapshot_count' / '.repo_state' / '.last_status' / '.last_run')` | demo-felhom `2026-08-04 09:49:42`: `1, escrowed, 0, NULL, NULL, NULL` · demo-hp `2026-08-04 09:56:57`: `1, escrowed, 0, NULL, NULL, NULL` |
| 8 | hub DB | `claimed` grouped over reports since `2026-08-03 05:00` | demo-hp: `1` ×124, **`0` ×5 (06:12:19→06:21:03)** · demo-felhom: `1` ×120, **`0` ×4 (07:17:54→07:19:46)** |
| 9 | hub DB | `SELECT customer_id, event_type, severity, created_at FROM events WHERE created_at > '2026-08-03 00:00'` | no `escrow_stale`, no `offsite_reissued`, no `offbox_repo_orphaned`, no claim event in the window; `controller_started` ×4 demo-hp / ×4 demo-felhom around the rebuilds |
| 10 | guest 9201 (I3) | `ls -la --time-style=full-iso …/felhom-controller-data/_data/data{,/offbox}` | the mtime table in §5 |
| 11 | source (I4) | `grep -rn "InjectOffboxPassword" --include=*.go .` | 1 definition, 1 handler caller, 5 test uses, 1 comment — **no production caller besides the handler** |
| 12 | source (I4) | `grep -rn "inject" internal/web/templates/` | one unrelated hit (an XSS comment) — **no form** |
| 13 | source (I4) | `grep -rni "reenroll\|re_enroll" --include=*.go .` in `felhom-agent` | **zero hits** |
| 14 | source (I4) | `grep -rn "re-enroll\|recovery-mode\|restore-directive" --include=*.sh --include=*.md --include=*.py` workspace-wide | only design docs + project memory; no script, no runbook step |
| 15 | source (I4) | `grep -rn "ListSupersededEscrow"` in `felhom.eu` | 2 test callers, 0 production |
| 16 | source (I4) | `grep -rn "\.Escrow\b" --include=*.go .` in `felhom-controller` | one hit: `cmd/controller/main.go:641` |
| 17 | source | `felhom-agent/CHANGELOG.md` | `v0.77.0 — fork-4: escrow the offsite restic repo password under R (2026-07-09)` |
| 18 | source | `controller/Dockerfile:36-70` | runtime package list — **no `age`** |
---
## 12. What could not be established
Named, not guessed.
1. **The 2026-08-05 ~02:15 UTC scheduled run's outcome.** Still unmeasured — both boxes became
runnable only on 2026-08-04 morning and neither has run since (query 7: `last_run` absent on both).
Yesterday's spike predicts `ErrOffboxOrphaned`; that prediction is **unchanged and still unproven**.
2. **Whether demo-hp's data dir was fresh.** Established for demo-felhom by direct mtime measurement
(I3). demo-hp was not measured — it has no baked SSH key and reaching it needs the break-glass
credential, which this read-only session did not use. The hub-side `claimed 1→0→1` signature is
identical on both boxes, which is suggestive, not decisive.
3. **Whether the orphaned ciphertext still exists at the provider.** Unchanged from yesterday: reading
the endpoint needs the SFTP credential, which was not touched.
4. **Whether any pre-2026-07-21 escrow generation ever existed for demo-felhom.** The superseded table
holds two rows in the entire database; earlier generations are not retained (and, per §7, would not
have carried the identity blob anyway).
5. **Whether `age` is present on any machine that would run step 8 of the drill.** Not checked — the
agent declares `/usr/bin/age` as a runtime dependency but this session did not verify it on either
demo host.
---
## 13. Observations — noticed, not acted on
1. **A fresh `encryption.key` on every rebuild** (I3: 32 B, mtime `2026-08-03 07:17:26`) means every
pre-rebuild `app.yaml` `ENC:` secret on that box is undecryptable. This is exactly circular
dependency **C-1** in `_recovery-inventory-2026-07-28.md`, and it fired live on both demo boxes this
week without anyone naming it. It is a strictly larger blast radius than the offsite repo and it is
**not** covered by R-193. Worth its own item if the operator agrees it is not already implied by C-1.
2. **`Claimed` returning to `1` within minutes** of a rebuild, with no event of any kind recording it,
means "the customer claimed this box" has no audit trail. There is no `customer_claimed` event type.
3. **`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting** — harmless today,
a trap for anyone reusing it as a read-only preview primitive (§9).
4. **`handleReEnroll` returns the new API key in the response body** (`dr.go:148`) while also logging
the rotation. Not a defect (the box needs it), but it is a secret in an HTTP response on an endpoint
nothing currently calls — worth a second look before anything starts calling it.
---
## 14. The operator's decisions — stated, and NOT answered here
1. **Run the drill (§10) or not?** It is the only thing that converts *"the pieces look right"* into
*"a customer got their data back"*. It costs demo-hp for an afternoon and, if it fails, that box's
history. **Recommendation: yes — but fix `identity_blob` retention (R-198) first**, because a drill
that walks the current chain will pass or fail on a chain that is missing a link everyone believed
was there.
2. **The orphaned snapshots at the endpoint** (40.9 MB + 1.14 GB, presumed present, §12.3). They are
**unrecoverable by two independent mechanisms** now: no recovery code was kept, *and* the sealed
passwords were destroyed by the 2026-08-04 re-ceremonies. They consume the 50 GB soft quota
indefinitely; nothing prunes a moved-aside restic repo. **Delete, or leave?** This session did not
act on it and states it as the operator's.
3. **The unchanged R-193 question** — agent-retained repo password (spike option c) versus accepting
that every rebuild starts a new repository. **This session adds one input to that decision:** the
"accept it, provided the system says so loudly" branch currently rests on a retention feature that
does not retain the relevant secret. Under option (d)-alone as it stands today, *"the old history
stays recoverable with your recovery code"* is not true. Either R-198 is fixed, or that branch has
to be re-described honestly to the customer.
</content>
</invoke>
@@ -0,0 +1,826 @@
# SPIKE — DooPlex build-cache containment (cap at 30 GB, relocate to SSD2)
**Date:** 2026-08-05 · **Host:** DooPlex (192.168.0.180) — **Tier 2, protected**, not a drill target
**Class:** Spike. §1, §2, §3, §4, §13-verify, §15 apply; §5–§12 skipped (no production Go code written).
**Anchor:** `dooplex_report.md` (2026-08-05 read-only disk audit). Every number in it was treated as a
claim to re-measure, not a fact to carry forward.
> Unit convention: **GB = decimal (10⁹)**, because `df -B1`, `du -B1` and Prometheus emit raw bytes.
> `docker buildx inspect` renders **GiB** (2³⁰) — where that matters it is labelled.
---
## Verdict
**The mechanism is confirmed, with one correction and one refutation, and the follow-up is safe to spec.**
- **Confirmed:** a `builder.gc` policy in `/etc/docker/daemon.json` **is** honoured under the containerd
worker, and it **does** evict. Proven by naming a 440 MB cache record present at one build and absent
two builds later — not by the absence of an error.
- **Corrected:** it is honoured **only in the `policy` array form**. The flat form
(`{"gc": {"reservedSpace": …}}`) is **silently ignored** — the daemon starts, logs nothing, and keeps
its built-in defaults. `dockerd --validate` is worthless here: it returned `configuration OK` for a
bogus key **and** for a config that then crashed the daemon.
- **Refuted:** the audit's proposal to change Docker's `data-root` would **not** move the build cache. It
moves 0.62 GB. The 181.4 GB belongs to the **system containerd**, relocated via `root` in
`/etc/containerd/config.toml`.
- **The prune is done and it was the whole rescue.** `/` went **86% → 53% used**, and Longhorn's default
disk went **`Schedulable=False (DiskPressure)``Schedulable=True`**. The live consequence is cleared.
- **The largest single finding is not the cap.** 90.5 GB of the 157 GB was one layer re-downloading on
every build, and the root cause is a **one-line Dockerfile ordering defect**`ARG VERSION` declared
above the module-download step. Fixing it removes the source; the cap only bounds the symptom.
- **P6 (the SSD2 move) was NOT executed** — it stops at the operator, as specified. The pre-analysis
says the move is safe as measured but carries a real Longhorn risk, and the prune removed its urgency.
---
## 1. Confirmed baselines
| Repo | `main` @ commit at spike start | Drift vs the task's stated baseline |
|---|---|---|
| `homelab-manifests` | `5188058d0e43` | none — matched |
| `felhom.eu` | `0dbd954fec90` | none — matched |
Both trees clean (`git status --porcelain` empty) and `HEAD == origin/main` before any commit.
**Register:** highest allocated at baseline was **R-204**, re-confirmed in `OPEN-ITEMS.md`. **R-205**
onward were free and are allocated here (R-205 … R-211).
**One path correction:** `homelab-manifests` is **not** in `/mnt/5_hdd/felhom.eu/git/` with the other
repos — it lives at **`/home/kisfenyo/git/homelab-manifests`**.
---
## 2. P0 — arrival state, expected vs measured
Measured live before anything was changed. **Values that matched are recorded, not omitted.**
| Fact | Expected (audit) | **Measured** | |
|---|---|---|---|
| Root fs size | 477,393,182,720 B | **477,393,182,720 B** | match |
| Root fs used | 387.3 GB | **387,332,755,456 B** | match (+10.8 MB in 2 h) |
| Root fs avail | 65.7 GB | **65,734,934,528 B** | match |
| Root fs percent | 86% | **86%** | match |
| Docker server version | 29.6.2 | **29.6.2** (build `3d80467`) | match |
| Storage driver | overlayfs, containerd snapshotter | **`overlayfs`, `driver-type: io.containerd.snapshotter.v1`** | match |
| Docker Root Dir | `/var/lib/docker`, ~0.6 GB | **`/var/lib/docker`, 618,541,056 B** | match |
| containerd root | `/var/lib/containerd`, ~181.4 GB | **181,394,223,104 B** | match |
| Build cache total / reclaimable | 157 GB / 147.8 GB | **157 GB / 147.8 GB**, 1446 records | match |
| Images / containers | 345 images, 2 running | **345 unique** (350 tag lines), **2 running** | match |
| `/etc/docker/daemon.json` | does not exist | **does not exist** (`/etc/docker/` exists, empty) | match |
| `/etc/buildkit/` | does not exist | **does not exist** | match |
| SSD2 device / mount / fs / free | ~240 GB, ~203 GB free | **`/dev/sde2`, `/mnt/ssd_2`, ext4, 235,148,750,848 B fs, 203,034,353,664 B free** | match |
| SSD1 Longhorn | `Schedulable=False (DiskPressure)` | **`Schedulable=False`, `DiskPressure`, 18.85% avail** | match |
| HDD2 Longhorn | also False | **`Schedulable=False`, `DiskPressure`, 24.91% avail** | match |
| Housekeeping timer | active, last Sun 2026-08-02 | **active**, last **2026-08-02 05:35:10 CEST**, next 08-09, `ExecMainStatus=0` | match |
**Prohibition honoured:** `node-housekeeping.sh` was **not run**, in any mode. Its state was read from
systemd, from the textfile on disk, and from Prometheus. The textfile's mtime is still
`2026-08-02 05:35:11` — the 8-week reclaim history is intact.
**What the audit did not measure — the SSD2 Longhorn record** (`Crucial-SSD-240G`,
`/mnt/ssd_2/data/longhorn`):
| field | value |
|---|---:|
| `storageMaximum` | 235,148,750,848 |
| `storageAvailable` | 214,958,080,000 (**91.41%**) |
| `storageScheduled` | 139,586,437,120 |
| `storageReserved` | **0** |
| condition | **`Schedulable=True`** |
| replicas | **12** |
**Two things stand out and both matter for P6.** SSD2 is the **only** Longhorn disk reserving **zero**
(HDD2 and HDD4 each reserve 500 GB). And `storageScheduled` (139.6 GB) exceeds actual usage
(`df`: 20,094,939,136 B) by **6.9×** — Longhorn is heavily overcommitted on this disk.
---
## 3. P1 — which daemon owns the bytes
> **The single configuration change that relocates the build cache is `root = "<path>"` in
> `/etc/containerd/config.toml`, owned by `containerd.service` — the SYSTEM containerd
> (PID 2698677, socket `/run/containerd/containerd.sock`).**
**The audit's `data-root` proposal is refuted, and the split is measured:**
| tree | size | moved by `data-root`? |
|---|---:|---|
| `/var/lib/docker` | **618,541,056 B** (volumes 321 MB, **buildkit metadata 282 MB**, containers 14 MB) | yes |
| `/var/lib/containerd` | **181,394,223,104 B** (overlayfs snapshots **170,357,870,592 B**, content 10.98 GB) | **no** |
`dockerd` runs as `/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock` with **no
systemd drop-ins**. BuildKit keeps its *record database* under Docker Root Dir (0.28 GB) but its *layer
bytes* in containerd's snapshotter — which is why `data-root` looks plausible and is wrong.
In `/etc/containerd/config.toml` the `root` key is **present but commented out**, so it defaults to
`/var/lib/containerd`. Uncommenting it is the whole change.
**k3s is unaffected — confirmed, not assumed.** k3s runs a **separate** containerd instance with its own
socket (`/run/k3s/containerd/containerd.sock`) and its own root
(`/var/lib/rancher/k3s/agent/containerd`, **79,669,473,280 B**). Nothing in this spike touches it.
---
## 4. P2 — which GC key schema this release accepts
**`dockerd --validate` exists on 29.6.2 and is not an oracle for this subtree.** Measured:
| candidate | `--validate` | reality |
|---|---|---|
| `defaultKeepStorage` (old flat form) | `configuration OK` | — |
| `policy` array | `configuration OK` | **honoured** |
| `reservedSpace` / `maxUsedSpace` / `minFreeSpace` flat | `configuration OK` | **silently ignored** |
| **`totallyBogusKey`** | **`configuration OK`** | proves the validator does no field checking |
| `policy` with `"filter": [three values]` | **`configuration OK`** | **crashed the daemon on restart** |
> **The real oracle is `docker buildx inspect`, which renders the *effective* policy.** That is an
> assertion of effect; `--validate` and "no error in the log" are both assertions of absence, and both
> were measured to lie here.
**The flat form is silently ignored.** With `{"gc":{"enabled":true,"reservedSpace":"1GB","maxUsedSpace":"2GB","minFreeSpace":"10GB"}}`
applied and the daemon restarted, `buildx inspect` still reported the built-in defaults, and
`journalctl -u docker.service` carried **no warning, no deprecation notice and no error**.
**The `filter` key takes ONE value per policy entry, not an array.** An array is fatal at startup:
```
error initializing buildkit: error creating buildkit instance: filters expect only one value
```
The daemon then entered a systemd restart loop. **This is also how the two `unless-stopped` dev
containers ended up stopped** — and they did **not** come back when the daemon later started cleanly;
they needed a manual `docker start`. A follow-up that templates this file must **restart and verify**,
never validate-and-assume.
**Documented statement on containerd-worker applicability:** none found in `dockerd --help` or the
installed docs. Recorded as **unknown from documentation**, and settled empirically by P5.
**The default policy is not "no policy".** Before any change, `buildx inspect` reported built-in rules
with **Reserved 41.91 GiB / Max Used 331.6 GiB / Min Free 82.89 GiB**. The audit's "there is no BuildKit
GC policy on this host" is *almost* right — there was one, sized so large it could never bind: the cache
at 157 GB was less than half the 331.6 GiB ceiling.
### The validated `daemon.json`, verbatim (final state, left in place deliberately)
```json
{
"builder": {
"gc": {
"enabled": true,
"policy": [
{ "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" },
{ "all": true, "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" }
]
}
}
}
```
Effective policy confirmed live (note the **GiB** rendering — `30GB` in JSON becomes a 30 GiB rule):
```
GC Policy rule#0: All: false Reserved Space: 10GiB Max Used Space: 30GiB Min Free Space: 20GiB
GC Policy rule#1: All: true Reserved Space: 10GiB Max Used Space: 30GiB Min Free Space: 20GiB
```
**No deprecation warning was emitted** for any accepted form.
---
## 5. P3 — the prune: claimed, actual, settle
Pre-checked: no build in progress, P0 and P7 captured first.
| | value |
|---|---:|
| **Claimed by `docker builder prune -af`** | **156.9 GB** (1444 records deleted, 86 s wall-clock) |
| **Actual filesystem reclaim** | **150,353,829,888 B (150.35 GB)** |
| **Time to settle** | **synchronous — full reclaim visible at t+0** |
Settle poll (t+0 already showed +150,354,477,056 B; two consecutive readings <50 MB apart at t+60s;
the residual 0.4 MB and 0.3 MB steps are ordinary background writes, not lagging reclaim):
| t | avail | step delta |
|---|---:|---:|
| 0 s | 215,947,522,048 | — |
| 30 s | 215,947,141,120 | 380,928 |
| 60 s | 215,946,874,880 | 266,240 |
**The 6.5 GB gap between claimed and actual is not an error** — it is the layers still shared with
retained images (`Shared: 8.6 GB` before the prune). **A weekly job must measure the filesystem, not
trust the command's `Total:`.**
**Unlike containerd's image GC, this returns bytes synchronously**, so the follow-up needs no
`settle_imagefs` equivalent — but the settle poll is what *established* that, and it cost 60 seconds.
| | before | after |
|---|---:|---:|
| `/` used | 387,474,644,992 B (**86%**) | 240,301,133,824 B (**53%**) |
| `/` avail | 65,593,044,992 B | 215,946,874,880 B |
| `/var/lib/containerd` | 181,537,714,176 B | 31,160,053,760 B |
| build cache | 156.9 GB / 1443 records | **0 B / 0 records** |
| images | 345 | **345** (untouched) |
| dev containers | 2 healthy | **2 healthy** |
### The live consequence, cleared
Longhorn re-evaluated within 60 s:
| disk | before | after |
|---|---|---|
| `default-disk-…` (SSD1, `/var/lib/longhorn/`) | `Schedulable=False (DiskPressure)`, 89,967,820,800 B, **18.85%** | **`Schedulable=True`**, 240,228,761,600 B, **50.32%** |
| `HDD2` | `Schedulable=False (DiskPressure)`, 24.91% | **unchanged — still False, 24.91%** |
**HDD2 is untouched by this work and remains unschedulable.** It is a separate item, stated here so it
is not read as fixed by association.
---
## 6. P5 — proving the ceiling is enforced
**Rollback stated before the first mutation** and never needed beyond the recovery below:
`sudo rm /etc/docker/daemon.json && sudo systemctl restart docker` — the file did not exist, so removal
is a complete revert.
Test ceiling **2 GB** (below one build's working set, deliberately, so eviction is reachable).
### Build-by-build totals
| step | build cache total | reclaimable | records |
|---|---:|---:|---:|
| baseline (post-prune) | 0 B | 0 B | 0 |
| controller 1 | 2.583 GB | 2.583 GB | 22 |
| hub 1 | **3.703 GB** ← peak | 3.703 GB | 37 |
| controller 2 | 3.525 GB | 3.525 GB | 37 |
| hub 2 | 3.051 GB | 3.051 GB | 38 |
| controller 3 | 2.817 GB | 2.817 GB | 36 |
| hub 3 | **2.782 GB** | 2.782 GB | 38 |
**The total stops rising and trends down** — the cap binds. Records plateau at 3638.
### The named eviction — the assertion that matters
Set-differencing the record IDs, **4 records present at controller-build-1 were absent by
controller-build-3**, including the exact class that accumulated 208 copies:
| ID | size | created | description |
|---|---:|---|---|
| **`9ak8egnxrg6rkibic2zo44u1`** | **440 MB** | 07:28:34 | **`mount / from exec /bin/sh -c go mod download \|\| true`** |
| `kkic530irq4em04kywzwrsjp` | 214.6 MB | 07:28:37 | `… go build -ldflags …` |
| `yvityhta4wppwa8cxs525c3g` | 58.83 MB | 07:28:36 | `… go mod tidy` |
| `mvttjfrimnt1ttt0ogry75ue` | 49.8 MB | 07:28:35 | `[builder 5/7] COPY . .` |
Across the whole sequence: `h1→h3` evicted **17** records, `c2→c3` evicted **10**.
### Tolerance, stated honestly
Steady state settled at **2.782.82 GB against a 2 GiB (2.147 GB) rule — about 30% over**, with a peak
of **3.70 GB (~72% over)** mid-sequence.
**This overshoot is an artefact of the test, not a property of the cap, and should not be extrapolated to
30 GB.** A single controller build produces ~2.6 GB of cache in one shot, so a 2 GB ceiling was set
*below one build's working set* — GC cannot get under it without deleting the cache it just created. At
30 GB the same working set is ~9% of the ceiling and GC has ample room.
### GC is build-triggered, not timer-driven
Six minutes idle, polled each minute: **total unchanged at 2.782 GB, records unchanged at 38.** The
cache does not shrink on its own. A burst of builds can overshoot and *stay* overshot until the next
build — which is the argument for keeping a weekly `docker builder prune` as a backstop (R-206), rather
than relying on the cap alone.
### Then the ruled 30 GB
Applied, restarted, and confirmed by the effective policy (§4). Both dev containers came back healthy
and **their data is intact — asserted positively, not by absence of complaint**:
| observable | before | after |
|---|---:|---:|
| postgres `count(*) from pg_database` | 4 | **4** |
| postgres `pg_database_size('jarr_dev')` | 175,135,767 | **175,135,767** |
| postgres public tables | 31 | **31** |
| redis `DBSIZE` | 2437 | **2437** |
| docker volumes | 8 | **8, identical names** |
---
## 7. P7 — why the module-download layer re-runs
**Root cause found and proven: `ARG VERSION` / `ARG GIT_COMMIT` are declared in the builder stage
*above* the module-download step, so a new version value invalidates every `RUN` beneath them.**
### The evidence chain
**1. Every record was used exactly once.** All **208** `go mod download` records (206 at audit time, +2
from this spike's builds) carried **`Usage count: 1`**, spanning 2026-07-06 → 2026-08-05, totalling
91.36 GB at 438440 MB each. Not one was ever reused.
**2. Two builds through `build.sh`, byte-identical tree, both missed.**
| step | build 1 | build 2 |
|---|---|---|
| `#8 WORKDIR /build` | CACHED | CACHED |
| `#9 COPY go.mod ./` | **CACHED** | **CACHED** |
| **`#10 RUN go mod download \|\| true`** | **executed** | **executed** |
| `#11 COPY . .` | executed | executed |
| `#12 RUN go mod tidy` | executed | executed |
| `#13 RUN … go build …` | executed | executed |
| `#14/#15/#16` (stage-1 `RUN`s) | CACHED | CACHED |
`go.mod` was **byte-identical** (`d8ded3f0798c…`) in the repo and in the workspace, mtime unchanged
(`2026-07-24 11:54:08`).
**3. `build.sh`'s churn was ruled out by fingerprinting.** Content-hashing all 737 files across
`rm -rf` + `cp -a` + host-side `go mod tidy`: **`tidy` changes nothing**, and the reproduced tree is
**content-identical** to the live workspace. `go.mod`'s inode number was even identical across the churn.
**4. A direct `docker build` on the untouched workspace cached *everything*** — including
`RUN go mod download` — proving BuildKit's cache is healthy and the Dockerfile is cacheable.
**5. The decisive isolation.** Same workspace, same command, only the build-arg value varied:
| run | `--build-arg VERSION` | `COPY go.mod ./` | **`RUN go mod download`** |
|---|---|---|---|
| E1 | **new** (`spike-p7e`) | CACHED | **executed** |
| E2 | **same** (`spike-p7e`) | CACHED | **CACHED** |
**The mechanism, and it explains the `#9 CACHED / #10 executed` signature exactly:** a `COPY`'s cache key
is content-based and carries no environment, so it hits; a `RUN`'s key includes the stage **environment**,
which contains every `ARG` declared above it. Since every real build passes a fresh `VERSION` (and often
a fresh `GIT_COMMIT`), the module-download layer is invalidated **on every build, without exception**
206 builds, 206 retained 440 MB snapshots, 90.5 GB.
**Both Dockerfiles have it.** `felhom.eu/hub/Dockerfile` declares `ARG VERSION` / `ARG BUILD_TIME` above
`COPY go.mod go.sum*``RUN go mod download`. **And because both produce byte-identical `buildx du`
description strings, the 208 records are a COMBINED count** and must not be attributed to one project.
**Fix shape — not applied here, filed as R-208:** move the `ARG` declarations down to just above the
final `go build`. One line per Dockerfile. **This is worth more than the cap and the move combined:** the
cap bounds the symptom at 30 GB; this removes the source.
---
## 8. P4 — the image store, enumerated (nothing deleted)
345 unique images across 350 tag lines (5 double-tagged), **0 dangling**. Only two are held by running
containers: `redis:7-alpine` and `postgres:16-alpine`. `docker image prune -a` would remove **343**.
**Size the prize before spending a decision on it.** Per-tag sizes sum to 139.29 GB, but that
double-counts shared layers. `docker system df` puts the **real dedup'd footprint at 31.02 GB, of which
27.02 GB is reclaimable** — roughly a fifth of what the build-cache prune already returned.
| GB (per-tag sum) | tags | repository |
|---:|---:|---|
| 118.44 | **207** | `gitea.dooplex.hu/admin/felhom-controller` |
| 7.25 | **107** | `gitea.dooplex.hu/admin/felhom-hub` |
| 4.34 | 5 | `gitea.dooplex.hu/admin/jarr` |
| 1.69 | 1 | `ghcr.io/seanmorley15/adventurelog-backend` |
| 0.94 | 1 | `ghcr.io/papra-hq/papra` |
| 0.72 | 1 | `codewithcj/sparkyfitness_server` |
| 0.72 | 1 | `postgis/postgis` |
| 0.59 | 3 | `gitea.dooplex.hu/admin/sparkyfitness-export` |
| (23 more, each ≤0.53 GB) | | |
### Registry presence — which are recoverable
Queried against the live Gitea registry v2 API:
| repo | tags in registry | tags local | **local-only (NOT re-pullable)** |
|---|---:|---:|---:|
| `felhom-controller` | 76 | 207 | **131** |
| `felhom-hub` | 45 | 107 | **62** |
**The local-only tags are all old.** Controller `0.39.0``0.135.0` plus `v0.35.0``v0.39.0`; hub
`0.9.0``0.57.0` plus `v0.7.2``v0.13.0`. **Everything from controller `0.136.0` and hub `0.58.0`
upward is in the registry** and therefore recoverable by `docker pull`.
**This is the list the operator still owes a ruling on (R-210).** CC's view: not worth doing for the
space — ~27 GB against 199 GB now free. The only real benefit is dropping unrecoverable clutter.
---
## 9. P8 — what else writes to Docker
**The bounded list of build triggers on the host Docker daemon is: interactive/manual runs only.**
- **18 systemd timers** enumerated — none invokes docker. (`node-housekeeping`, `dooplex-backup`,
`apt-daily*`, `sysstat*`, `logrotate`, `fstrim`, `e2scrub_all`, `smart-fancontrol`, …)
- **Root crontab: empty** (comments only). **User crontab: one entry**, `sync-hetzner-backups.sh`,
writes under `/mnt/5_hdd`.
- **`/etc/cron.d`: 4 real files** — anacron, e2scrub_all, sysstat, zfsutils. None touches docker.
**The hole the audit left open is now closed with positive evidence.** The audit concluded "no CI runner
exists on this host" from systemd/process/crontab alone — but a Gitea Actions runner **does** exist as a
**cluster pod** (R-168, 2026-08-02). It cannot reach the host daemon:
- **162 pods swept across all namespaces: `0` mount any `docker`/`containerd` hostPath.**
- `act-runner` (`gitea-system`, image `felhom-act-runner:0.1.0`) has one volume — a PVC at `/data`.
`privileged: false`, `allowPrivilegeEscalation: false`, **all capabilities dropped**, no docker socket.
- Its Dockerfile states the absence is deliberate: *"DELIBERATELY NOT INSTALLED: a Go toolchain, Node, or
Docker … There is no container runtime in here on purpose."*
- `gates.yml` runs exactly one command — `python3 scripts/<repo>_gates.py --fast` — plus a git fetch and
a failure-only Resend alarm. **No `docker build` anywhere.**
**So the audit's ~7 builds/day attribution to interactive use is confirmed**, and now on a complete
enumeration rather than a partial one.
---
## 10. P9 — the monitoring rule fix (shipped)
**File:** `homelab-manifests/mon-system/prometheus-rules.yaml`, alert
`RootFsPressureDespiteHousekeeping`. **Commit `6808a4b`.**
**Change:** `and`**`and on(instance)`**. A bare `and` requires identical label sets; the left vector
carries `{device,fstype,instance,job,mountpoint}` and the right — a node_exporter *textfile* metric —
carries only `{instance,job}`. Nothing ever matched.
### Red-proof (required, and both numbers recorded)
**Threshold substituted `0.25``0.5`, and the reason is stated rather than hidden:** P3's prune had
already taken `/` to 44.6% available, so the shipped threshold's left side is now legitimately false. The
defect is in **label matching**, not in the threshold, so the proof is run at a threshold where both
sides are individually true — which is exactly the condition the original rule was meant to catch.
| expression | `result_count` |
|---|---:|
| left side alone (`avail/size < 0.5`) | **1** |
| right side alone (`time() last_success < 1209600`) | **1** |
| **as shipped — bare `and`** | **0** |
| **fixed — `and on(instance)`** | **1** |
For completeness, at the shipped 0.25 threshold **both** forms return 0 today — because the left side is
false now, not because the fix failed.
**Annotation rewritten.** It pointed at Longhorn replicas/snapshots (~10% of the growth). It now names
`docker buildx du` first, records the 30 GB cap in `/etc/docker/daemon.json`, and notes that a reverted
or unapplied cap looks exactly like this alert firing — so the alert text carries its own diagnosis.
### Deployment — GitOps, no `kubectl set image`, no live ConfigMap edit
ArgoCD app `monitoring` (auto-sync **off**): hard-refresh → `OutOfSync` at `6808a4be54a7` → deliberate
sync → **`Synced` / `Healthy`**, operation phase **`Succeeded`**.
**Verified in Prometheus, not assumed from ArgoCD:**
```
query : … < 0.25 and on (instance) time() - node_housekeeping_last_success_timestamp_seconds < 1209600
state : inactive health : ok lastError : (none)
```
`inactive` is now the **correct** answer — `/` really is fine at 44.6% available.
> **A near-miss worth recording: I invented a full commit SHA from ArgoCD's truncated display and the
> sync failed with `upload-pack: not our ref`.** It failed loudly and cost one retry. It would not have
> failed loudly had the fabricated SHA happened to exist.
### The finding P9 turned up on the way — R-211
**There is no config-reloader.** The `mon-system/prometheus` Deployment runs **one** container
(`prom/prometheus:v3.12.0`) with no reloader sidecar. After the sync the updated rules file was present
**inside the pod** (`grep -c "and on(instance)"` → 3 on the mounted symlink) while the Prometheus rules
API **still served the old expression for 4+ minutes**, with no error anywhere. It took effect only after
an explicit `POST /-/reload` (HTTP 200; `--web.enable-lifecycle` was already set).
**The consequence is general:** every rule edit in this repo has silently not applied until something
happened to restart the pod. `Synced/Healthy` was true and beside the point. Filed as **R-211**.
---
## 11. P6 — the SSD2 move: **EXECUTED 2026-08-05 on operator authorisation**
> **Executed after the operator ruled "proceed", having read the pre-analysis below. The
> `storageReserved` condition CC named was applied with it. The operator subsequently ruled
> explicitly: DO NOT REBOOT DooPlex — so the reboot validation is DEFERRED, and what that leaves
> unproven is stated in §11.9 rather than glossed.**
### 11.0 Execution record
**Blast radius, established before anything was stopped:** k3s runs a **separate** containerd
(`/run/k3s/containerd/containerd.sock`), so **Gitea, the registry, the hub, PBS, Longhorn and all
~160 pods stayed up throughout**. The only workloads affected were the two `jarr-*` dev containers.
`ctr -n moby containers list` confirmed exactly those two on the system containerd.
| # | step | result |
|---|---|---|
| 1 | stop `docker.service`, `docker.socket`, `containerd.service` | all `inactive` |
| 2 | `mkdir /mnt/ssd_2/containerd` (mode 711, root:root) | ok |
| 3 | `rsync -aHAX --numeric-ids` | **550,891 files, 33,276,370,041 B, 4 m 58 s** |
| 4 | `mv /var/lib/containerd → /var/lib/containerd.pre-move-2026-08-05` | **moved aside, never deleted** |
| 5 | `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` (backup kept) | one-line diff |
| 6 | `RequiresMountsFor=/mnt/ssd_2` drop-ins on **both** `containerd.service` and `docker.service` | resolved into both units |
| 7 | start + verify | see below |
| 8 | `storageReserved = 80 GB` on SSD2 | applied |
**`-X` is load-bearing** — overlayfs snapshots carry `trusted.overlay.*` xattrs, and losing them
would silently corrupt layer stacking.
### 11.1 Copy verified BEFORE the original was touched
Four independent observables, all matching:
| check | source | destination |
|---|---:|---:|
| filesystem entries | 550,891 | **550,891** |
| `trusted.overlay` xattrs | 448 | **448** |
| files with link count >1 (hardlinks) | 37,243 | **37,243** |
| `io.containerd.metadata.v1.bolt/meta.db` sha256 | `c60ecfb4ee134255…` | **`c60ecfb4ee134255…`** |
(Byte totals differ by +14 MB — ext4 block allocation, not content.)
### 11.2 Nothing was lost
| | pre-move | post-move |
|---|---:|---:|
| unique images | 345 | **345 — identical ID set** |
| tag lines | 350 | **350 — identical set** |
| docker volumes | 8 | **8 — identical set** |
| build cache | 2.782 GB / 38 records | **2.782 GB / 38 records** |
| GC ceiling | 30 GiB | **30 GiB, still in force** |
| postgres databases | 4 | **4** |
| postgres `jarr_dev` bytes | 175,135,767 | **175,135,767** |
| postgres public tables | 31 | **31** |
| redis `DBSIZE` | 2437 | **2437** |
**End-to-end proof, not just inventory:** a real `./build.sh` run against the relocated store
completed `rc=0`, produced an image, and grew the cache 2.782 → 4.012 GB. The scratch tag was removed
and `:latest` restored to the anchor digest afterwards.
### 11.3 The store is demonstrably on SSD2
```
containerd config dump → root = '/mnt/ssd_2/containerd'
du /mnt/ssd_2/containerd → 35,500,183,552 B
/var/lib/containerd → does not exist
```
### 11.4 The guard — and a trap found while proving it
**First attempt did NOT refuse, and that is a finding worth more than the test.** With
`RequiresMountsFor=/mnt/absent-guard-test-2026-08-05` — a path with **no mount unit at all**
containerd **started normally**. `RequiresMountsFor` on a path systemd has no mount unit for is a
**silent no-op**. So a typo in the guard path buys nothing and says nothing: the same
*built-but-never-wired* shape this project has shipped repeatedly.
**So the guard was verified positively, at the unit level:**
```
containerd.service Requires=mnt-ssd_2.mount After=mnt-ssd_2.mount
docker.service Requires=mnt-ssd_2.mount
```
**Then the refusal itself was proven with a genuinely absent DEVICE.** `/mnt/ssd_2` hosts **12 live
Longhorn replicas** and must never be unmounted, and editing `fstab` on a production host risks
emergency mode at boot — so a **temporary synthetic `.mount` unit** pointing at a nonexistent UUID
was used instead, with a temporary `RequiresMountsFor` on the real `containerd.service`:
```
A dependency job for containerd.service failed.
systemd: Dependency failed for containerd.service - containerd container runtime.
systemd: containerd.service: Job containerd.service/start failed with result 'dependency'.
containerd is-active: inactive
```
**containerd REFUSED to start rather than creating an empty store on the root disk** — exactly the
required behaviour, and exactly the signature the ep0 runbook describes. All test artifacts were
removed; only `10-ssd2-mount.conf` remains on each unit and the guard resolves to `/mnt/ssd_2` alone.
### 11.5 Longhorn — the condition CC made the move conditional on
`storageReserved` on SSD2 raised **0 → 80,000,000,000 B**, covering the containerd tree so Longhorn's
scheduler can no longer oversubscribe into the space the store needs.
| disk | available | reserved | `Schedulable` |
|---|---:|---:|---|
| **Crucial-SSD-240G (SSD2)** | 179,516,211,200 (**76.34%**) | **80,000,000,000** | **True** |
| default-disk (SSD1) | 236,978,176,000 (49.64%) | 143,217,954,816 | **True** |
| HDD4 | 1,793,798,963,200 (60.82%) | 536,870,912,000 | True |
| HDD2 | 2,470,864,486,400 (24.91%) | 536,870,912,000 | **False** — pre-existing, untouched |
SSD2 remains schedulable with the reserve applied. **Rollback: patch `storageReserved` back to 0.**
### 11.6 A standing post-boot assertion was installed
`felhom-store-postboot-check.service` (oneshot, `After=docker.service`, enabled) runs
`/usr/local/sbin/felhom-store-postboot-check.sh` at **every** boot and logs to
`/var/log/felhom-store-postboot-check.log`. It asserts, with **positive** observables:
- `/mnt/ssd_2` is mounted;
- containerd's effective `root` is `/mnt/ssd_2/containerd`;
- **`/var/lib/containerd` does NOT exist** — i.e. no empty store was recreated on the root disk, which
is the precise trap the guard exists to prevent;
- docker sees ≥100 images;
- both dev containers are running.
Dry-run at install time: **RESULT: PASS** on all six lines. It was proven working *before* being
relied upon.
### 11.7 Disk position after the move
| | value |
|---|---:|
| `/` | 224 GB used, **199 GB free, 54%** |
| `/mnt/ssd_2` | 52 GB used, **157 GB free, 25%** |
| store on SSD2 | 35,500,183,552 B |
| **rollback copy retained on `/`** | **34,299,498,496 B** |
**`/` did not drop below 54% because the rollback copy is deliberately still there.** Reclaiming it
returns ~34 GB — see §11.9.
### 11.8 Rollback, complete and one sequence
1. `sudo systemctl stop docker docker.socket containerd`
2. `sudo mv /var/lib/containerd.pre-move-2026-08-05 /var/lib/containerd`
3. `sudo cp /etc/containerd/config.toml.pre-move-2026-08-05 /etc/containerd/config.toml`
4. `sudo rm -rf /etc/systemd/system/{containerd,docker}.service.d/10-ssd2-mount.conf`
5. `sudo systemctl disable --now felhom-store-postboot-check.service && sudo rm /etc/systemd/system/felhom-store-postboot-check.service /usr/local/sbin/felhom-store-postboot-check.sh`
6. `sudo kubectl patch nodes.longhorn.io -n longhorn-system dooplex --type merge -p '{"spec":{"disks":{"Crucial-SSD-240G":{"storageReserved":0}}}}'`
7. `sudo systemctl daemon-reload && sudo systemctl start containerd docker`
8. `sudo rm -rf /mnt/ssd_2/containerd`
### 11.9 **NOT PROVEN: the reboot. Stated, not glossed.**
**The operator ruled explicitly that DooPlex must not be rebooted, so the reboot validation did not
happen.** Uptime is unbroken (7 weeks 6 days, since 2026-06-10) — verified, not assumed.
**By this task's own standard — *"a move that has not survived a reboot has not been validated"* — the
move is NOT fully validated,** and it would be dishonest to close it as if it were. Precisely:
- **PROVEN (mechanism):** the guard is wired into both units (`Requires=` + `After=mnt-ssd_2.mount`),
and containerd **refuses to start** when a required mount's device is genuinely absent.
- **NOT PROVEN (consequence):** that a real boot sequence mounts `/mnt/ssd_2` before containerd starts
*on this host, in the real ordering*. Mount-ordering reasoning is exactly the class this project has
been burned by, and the mechanism proof does not substitute for it. `CLAUDE.md` prefers the
consequence over the mechanism; here only the mechanism is available.
**Two consequences follow, and both are deliberate:**
1. **The rollback copy `/var/lib/containerd.pre-move-2026-08-05` (34.3 GB) STAYS** until a reboot has
validated the move. Deleting it now would trade a cheap 34 GB for the only cheap way back.
2. **The next reboot — whenever it happens, planned or not — validates this automatically.** The
post-boot service writes `RESULT: PASS` or `RESULT: FAIL` to
`/var/log/felhom-store-postboot-check.log`. **Read that file after the next boot**; on `PASS`, the
rollback copy can be removed and ~34 GB returned to `/`.
Tracked as **R-209a**.
---
## 11-bis. P6 — the pre-analysis the ruling was made on
*(Retained as written, before execution, so the ruling can be audited against what it was given.)*
**First, the framing changed underneath this probe.** P3's prune took `/` to 53% used and un-stuck
SSD1's Longhorn disk. **The move is no longer a rescue; it is a deliberate choice.** Per standing rule 4,
CC's disagreement is stated rather than absorbed: **the cap alone resolved the problem the move was meant
to solve, and the move carries a Longhorn risk the cap does not.**
### Mechanism — three candidates
| | mechanism | survives package upgrade | expressible in Ansible | visible reading the host cold |
|---|---|---|---|---|
| **(a)** | **`root = "/mnt/ssd_2/…"` in `/etc/containerd/config.toml`** | **yes** — dpkg conffile, local edits preserved | **yes** — a template, the file is already managed-shaped | **yes** — one line in the daemon's own config |
| (b) | systemd mount unit / fstab bind onto `/var/lib/containerd` | yes | yes | **weak** — invisible unless you check `findmnt` |
| (c) | symlink | fragile | yes | **weak**, and overlayfs mount semantics through a symlink are unproven here |
**Recommendation: (a).** It is the only one that a person reading the host cold can *see*, and the key
already exists in the file (commented out).
### The boot-order trap, and its guard
If containerd or Docker starts before `/mnt/ssd_2` is mounted, the daemon **creates an empty store on the
root disk** and 345 images appear to have vanished.
**Guard:** `RequiresMountsFor=/mnt/ssd_2` on **`containerd.service`** and **`docker.service`**. Neither
has it today — `containerd.service` carries only `After=network.target dbus.service`.
> **`RequiresMountsFor` RE-MOUNTS; it does not refuse.** systemd pulls the mount unit in and *starts* it,
> so an unmounted-but-available device is silently re-mounted and everything looks fine. It refuses only
> when the device is genuinely **unavailable**. **Testing it by unmounting proves nothing** — it must be
> tested by pointing at an absent device. This is the ep0 datastore lesson
> (`RUNBOOK-ep0-datastore-volume-2026-07-27.md`), earned on this project.
**And a move that has not survived a reboot has not been validated.** Mount-ordering reasoning is exactly
the class this project has been burned by.
### The Longhorn consequence — the part that decides it
`Crucial-SSD-240G`, `storageMaximum` 235,148,750,848, **25% floor = 58,787,187,712**:
| scenario | resulting `storageAvailable` | as % | vs the 25% floor |
|---|---:|---:|---:|
| today, untouched | 214,958,080,000 | 91.41% | +66.4 pp |
| move cache only (30 GB) | 184,958,080,000 | 78.66% | **+53.7 pp** |
| move whole containerd tree today (34.3 GB) | 180,658,569,216 | 76.83% | +51.8 pp |
| **move at steady state (~31.5 GB images + 30 GB cache ≈ 65 GB)** | 149,958,080,000 | **63.77%** | **+38.8 pp** |
**As measured, the move is safe by a wide margin — nowhere near the 10-pp caution band.**
**But the worst case is not comfortable, and it is not hypothetical enough to omit.** `storageScheduled`
on SSD2 is **139,586,437,120** while actual usage is **20,094,939,136** — a 6.9× overcommit. If those 12
volumes ever inflate to their scheduled size:
| scenario | available | as % | vs floor |
|---|---:|---:|---|
| + cache only (30 GB) | 65,562,313,728 | 27.88% | +2.9 pp — inside the caution band |
| **+ steady state (65 GB)** | 30,562,313,728 | **13.00%** | **12.0 pp → `Schedulable=False`** |
That is precisely the failure that just took SSD1 out of service.
> **Recommendation: proceed only together with setting Longhorn's `storageReserved` on SSD2 to cover the
> containerd tree (~80 GB).** SSD2 reserving **zero** while HDD2 and HDD4 each reserve 500 GB is an
> anomaly in its own right, and it is what lets Longhorn oversubscribe into the space the cache needs.
> Without that, CC recommends **not moving** — the cap has already done the job.
### The sequence, prepared and not executed
Each step with its rollback. **Do not run without the operator.**
| # | step | rollback |
|---|---|---|
| 1 | `systemctl stop docker docker.socket containerd` | `systemctl start containerd docker` |
| 2 | `mkdir -p /mnt/ssd_2/containerd` | `rmdir` |
| 3 | `rsync -aHAX --numeric-ids /var/lib/containerd/ /mnt/ssd_2/containerd/` (small now — ~34 GB, and ~2.8 GB of it is cache) | delete the copy; the original is untouched |
| 4 | `mv /var/lib/containerd /var/lib/containerd.pre-move` (**move aside, never delete**) | `mv` back |
| 5 | set `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` | restore the commented line |
| 6 | drop-in `RequiresMountsFor=/mnt/ssd_2` on `containerd.service` **and** `docker.service`; `systemctl daemon-reload` | delete the drop-ins |
| 7 | `systemctl start containerd docker`; **verify `docker images` = 345, both dev containers up, `du` confirms the store's real path** | steps 5→4→1 in reverse |
| 8 | set SSD2 `storageReserved` ≈ 80 GB in the Longhorn node CR | set back to 0 |
| 9 | **`reboot`, then re-verify everything in step 7** | as above |
| 10 | test the guard by pointing at an **absent** device — an unmount alone proves nothing | restore fstab |
| 11 | only after a clean reboot + a full build: `rm -rf /var/lib/containerd.pre-move` | — none; do this last |
---
## 12. Teardown
**Nothing was provisioned.** No machine, no VM, no LXC guest, no hub record, no customer, no Longhorn
volume, no PVC. There is nothing to decommission.
**Cleaned up:**
- The six candidate `daemon.json` files written for P2 (in the session scratch dir, never in
`/etc/docker/`) — **removed**.
- The scratch workspace fingerprint tree (`ws_sim`) — **removed**.
- **Nine scratch image tags** created by the probe builds — `felhom-spike:{p7c,e1,e2}`,
`felhom-controller:{spike-p7a,spike-p7b,spike-p7d,evict-c1,evict-c2,evict-c3}`,
`felhom-hub:{evict-h1,evict-h2,evict-h3}`**all removed**.
- `felhom-controller:latest` **restored** to `sha256:bdcd4a4ae8e006f1…` (0.198.0), byte-matching the
anchor captured before the first build. Image count back to **345**, the P0 baseline.
**Intentional state left in place, recorded here so it does not read as a leftover:**
**`/etc/docker/daemon.json` exists**, with the content in §4. It did not exist before this session.
Removing it plus a daemon restart is a complete revert.
**Not cleaned up, deliberately:** the ~2.78 GB of build cache from the eviction sequence. It is the
working cache under a live 30 GB policy, not scratch.
---
## 13. Final state
| | at arrival | **now** |
|---|---:|---:|
| `/` used | 387,332,755,456 B (**86%**) | **240,301,133,824 B (54%)** |
| `/` available | 65,734,934,528 B | **212,766,556,160 B** |
| `/var/lib/containerd` | 181,394,223,104 B | **34,299,510,784 B** |
| build cache | 157 GB / 1446 records | **2.782 GB / 38 records, capped at 30 GiB** |
| images | 345 | **345** |
| dev containers | 2 healthy | **2 healthy, data verified intact** |
| SSD1 Longhorn | **`Schedulable=False (DiskPressure)`** | **`Schedulable=True`, 50.32%** |
| `RootFsPressureDespiteHousekeeping` | **dead — could never fire** | **live, `health=ok`, red-proven** |
| growth driver | unbounded, ~5.25 GB/day | **capped; source diagnosed (R-208)** |
---
## 14. Observations — noticed, deliberately not acted on
1. **`build.sh` exists in two copies** — `felhom-controller/controller/build.sh` (in git) and
`/mnt/5_hdd/felhom.eu/build/felhom-controller/build.sh` (the operative one). They differ by **one
comment line**. The hub solved this with a symlink into the repo; the controller has not. Not fixed —
out of scope, but it is a drift channel.
2. **`build.sh` tags `:latest` on every build**, including throwaway version tags. A spike build silently
moved `latest` off 0.198.0; it was restored from an anchor captured beforehand. A `--no-latest` flag,
or tagging `latest` only on `--push`, would remove the trap.
3. **The default BuildKit GC policy is derived from disk size**, so a bigger root disk would have raised
the 331.6 GiB ceiling too. The cap needs to be explicit no matter how much disk is added.
4. **`node-housekeeping.sh`'s own `avail_bytes()` uses `df … | tail -1`** — the truncating-instrument
shape this project has been bitten by twice. It is *correct* here (`df` on one path emits exactly one
data row), so it was left alone, but it is worth knowing it is there.
5. **HDD2 remains `Schedulable=False (DiskPressure)`** at 24.91%, a hair under the 25% floor with 2.47 TB
free and its 500 GB reserve satisfied. Untouched by this work, and not fixed by it.
6. **The `mon-system` Prometheus has `--web.enable-admin-api`** as well as `--web.enable-lifecycle`. The
admin API permits series deletion. Noted, not acted on.
7. **Longhorn on SSD2 is overcommitted 6.9×** (139.6 GB scheduled vs 20.1 GB used). That is normal for
thin provisioning, but combined with `storageReserved = 0` it is what makes the P6 worst case reach
the floor.
---
## 15. Register
Allocated this session: **R-205** (CLOSED — the monitoring rule, shipped), **R-206** (Ansible: cap +
prune + narrowed Docker ban), **R-207** (the `DRY_RUN` guard), **R-208** (the `ARG`-ordering cache miss —
the largest remaining win), **R-209** (WAITING-ON-OPERATOR — whether to move to SSD2 at all),
**R-210** (WAITING-ON-OPERATOR — the image list), **R-211** (no Prometheus config-reloader).
@@ -0,0 +1,540 @@
# SPIKE — what is actually one-shot, and what a guest rebuild really costs
**Date:** 2026-08-04 · **Items:** R-193 (rebuild drops offsite), R-192 (the alert says the opposite of
what it measured) · **Class:** spike — **no production code shipped for R-193 or R-192**
**Baselines read:** `felhom.eu` @ `f456835bbcc4` (hub v0.91.1) · `felhom-controller` @ `0887fd67` ·
`felhom-agent` @ `856a127`
> **This document ends in ranked options and a STOP.** It establishes facts; it does not choose.
> The operator's question is stated, unanswered, at the end.
---
> ### ANNOTATION 2026-08-04 (later the same day) — read this before acting on §Q8
>
> A follow-up recon traced the escrow→recovery chain link by link
> (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **Two corrections to this document, and they pull
> in opposite directions.**
>
> **(1) Q8 candidate (b) — "not implementable as specified" — is TOO STRONG and is overturned in
> part.** Zero-knowledge means the *hub* cannot open the blob; the *customer* can, with R, which is
> what the ceremony exists for. The mechanism to hand a rebuilt box its old repo password back —
> customer-present, not unattended — is a real design, and the operator has since ruled on its shape.
> What (b) got right is narrower than what it said: **an UNATTENDED rebuild cannot recover from a
> zero-knowledge escrow.** A customer-present one can. The recon prices that shape row by row.
>
> **(2) A LOAD-BEARING FACT THIS SPIKE DID NOT CHECK, and it makes (d)-alone weaker than ranked
> here.** `host_escrow_superseded` **has no `identity_blob` column**, and `demoteCurrentEscrowTx`
> (`hub/internal/store/store.go:2547-2556`) copies only the K-escrow `blob`. So the retained
> "superseded" blob holds the **PBS datastore key**, not the **restic repository password**. The
> retention feature does not retain the secret it was built to retain, and the **next escrow ceremony
> — which the system tells the customer to run whenever a rebuild leaves the escrow `pending` — is
> what destroys the last copy.** Both demo boxes crossed that line on 2026-08-04 at 07:15:36 and
> 07:20:08. → **R-198**.
>
> **Consequence for this document's recommendation.** "Ship (d), then decide (c)" still holds, but
> (d)'s implicit promise — *the old history stays recoverable with the recovery code* — is **false as
> the code stands**, and is repeated verbatim to the customer in Hungarian on the orphan card. (d) is
> not complete without R-198.
>
> **Unchanged and still correct:** Q1, Q2, Q4 (R-196), Q5, Q6, Q7 (R-192) and the Q8 (a)/(c)/(d)
> analyses. **Q3 remains UNMEASURED** — neither box has run since; the 2026-08-05 ~02:15 UTC run is
> still the decisive one.
---
## 0. Why this exists
Two candidate fixes were named when R-193 was filed, and neither could be weighed, because two prior
session reports described the same mechanism in contradictory terms:
- R-39's record: Re-issue *"rotates the restic password and makes the escrow stale"*.
- R-193's own resolution note: the one-time password is *"only the transport credential used once to
install the box's own SSH key"*.
Those two worlds imply very different fates for the fifteen existing snapshots. This spike settles it
**from source first, live state second**, so a measurement can contradict the code rather than be
interpreted by it.
**The headline, before the detail:** *both prior claims are wrong in the way that matters.* Re-issue
does **not** touch the restic repository password — but a **guest rebuild does**, silently, on every
box, with nothing marking the escrow stale and nothing telling anyone. Both demo boxes lost repository
continuity on 2026-08-03/04. The one that "recovered by luck" lost it too.
---
## 1. The instruments, and what each one can and cannot answer
Stated up front so a wrong instrument is visible here rather than inside a conclusion (§8 rule 2).
| # | Instrument | Exact query / read | Known limit |
|---|---|---|---|
| I1 | hub SQLite snapshot | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale at copy time (mtime 08:34 vs wal 10:45 CEST). Copying only the main file would have produced confident answers from a stale file — the exact R-3 shape. |
| I1-check | freshness proof | `PRAGMA integrity_check``ok`; `MAX(host_reports.received_at)` = `2026-08-04 08:45:38`, `datetime('now')` = `08:49:46` | **Positive observable**: newest row 4 min old. Not "the query returned no error". |
| I2 | `host_escrow.restic_pw_sha256` + `host_escrow_superseded.restic_pw_sha256` | see §2 | sha256 of a 256-bit random secret — non-reversible, safe to compare. Answers *"did the repo password change"* **without touching a box or a credential**. |
| I3 | `reports.report_json → $.offsite` | grouped distinct-object query with first/last seen | The report echoes the box's own target state; absence of the key = no target configured. |
| I4 | `one_time_secrets` | `SELECT customer_id, created_at, consumed_at` | **`customer_id` is the PRIMARY KEY** — one row per customer, last-write-wins. It **cannot** answer "how many credentials were delivered"; a Re-issue destroys the prior row. demo-hp's 2026-07-23 consume is no longer in this table. |
| I5 | source | `hub/internal/offsite/*`, `hub/internal/api/*`, `controller/internal/{backup,offsiteapply}`, `agent/internal/escrow` | — |
**Not used, deliberately:** no `ReissueCredentials`, no ceremony, no config change, no deletion, no
write to any box. Nothing on either box or on the storage endpoint was altered by this session.
---
## Q2 — Does a rebuilt controller reuse the escrowed repository password, or mint a new one?
### **RULING: it MINTS A NEW ONE. The escrowed password is never consulted on any automatic path.**
**Method — source.** The apply-bridge's terminal step is
`Enabler.ConfigureOffbox(...)` (`controller/internal/offsiteapply/offsiteapply.go:221,256`), which
reaches `Manager.ApplyOffsiteTarget``WriteOffboxSecrets`
(`controller/internal/backup/offbox.go:370`). Quoted verbatim, `offbox.go:392-401`:
```go
// Auto-generate the repo password once (0600), never log it.
if _, err := os.Stat(m.offboxPwPath()); os.IsNotExist(err) {
pw, gerr := generateOffboxPassword()
...
if werr := os.WriteFile(m.offboxPwPath(), []byte(pw), 0o600); werr != nil {
```
`offboxPwPath()` is `<DataDir>/offbox/repo_password` (`offbox.go:363`). A rebuilt guest has a fresh
data dir, so the file is absent, so **a fresh 256-bit password is minted**.
**The only path that recovers the old one is `InjectOffboxPassword`** (`offbox.go:541`), whose sole
caller in the entire repo is `offboxInjectPasswordHandler`
(`controller/internal/web/offbox_handlers.go:189`) — a **web form where a human pastes the recovered
password**. Verified by census: `grep -rn "InjectOffboxPassword" --include=*.go .` → three hits, one
definition, one caller, one comment. **The apply-bridge never calls it.**
**Method — live, and it is decisive without touching a box (I2).** The hub already stores the sha256
of the escrowed repo password, so the question is answerable as a hash comparison:
| host | superseded (pre-rebuild) | current (post-rebuild) | verdict |
|---|---|---|---|
| `demo-hp-bb76ea` | `8e03eddf9ff7…` created `2026-07-23T10:01:17Z` | `8a9e33aa4da6…` created `2026-08-04T07:15:36Z` | **CHANGED** |
| `demo-felhom-8363b5` | `48741892f0ef…` created `2026-07-21T08:38:33Z` | `c60c8bc737a6…` created `2026-08-04T07:20:07Z` | **CHANGED** |
**Both boxes minted a new repository password.** demo-hp's pre-rebuild escrow (`8e03…`, sealed eight
minutes after its 2026-07-23 09:53:41 apply) is the key to its 15 snapshots. That key now exists
**only inside a superseded, R-wrapped escrow blob** that the hub cannot open.
### **This is bigger than R-193 as filed, and it hits the box R-193 called lucky.**
R-193 records demo-felhom as having *"survived the SAME rebuild by luck"* — 76 seconds of downtime.
Measured (I3), demo-felhom's pre-rebuild offsite object was:
```
last_run 2026-08-03T02:17:44Z · last_status ok · snapshot_count 36 · repo_size_bytes 1 136 685 919
```
**36 snapshots, 1.14 GB.** Since 2026-08-03 07:19:10 UTC it has reported `snapshot_count: 0,
repo_size_bytes: 0` in every one of 109 reports, and its repo password hash changed.
> **The luck recovered DELIVERY. It did not recover the REPOSITORY.** The staged secret restored the
> transport in 76 seconds and the box then minted a brand-new repo password anyway. The contrast
> R-193 draws between the two boxes is real for the *credential*, and **false for the data**: both
> boxes lost repository continuity, one loudly and one silently. The silent one is worse.
---
## Q3 — What did the 04:15 run report?
### **RULING: UNMEASURED — the decisive run has not happened yet. And the binary the question offers is the wrong one; source says the answer will be NEITHER 15 nor 1.**
**Why it has not happened.** The scheduled off-box run fires at ~02:15 UTC (04:15 CEST). Measured
from I3 — `last_run` values `2026-08-02T02:15:52Z`, `2026-08-03T02:16:39Z` on demo-hp;
`2026-08-01T02:17:54Z`, `2026-08-02T02:17:01Z`, `2026-08-03T02:17:44Z` on demo-felhom. **Daily, both
boxes.** Then:
| box | why 2026-08-04 02:15 UTC produced nothing |
|---|---|
| demo-hp | no offsite target existed at all — the `offsite` key was **ABSENT** from every report between `2026-08-03 06:12:19` and `2026-08-04 07:12:01` UTC (111 reports). The re-issue landed at 07:15:47, five hours after the window. |
| demo-felhom | the target existed but `escrow_state: pending` from `2026-08-03 07:19:10` to `2026-08-04 07:20:13`. `OffboxRunnable` requires configured **AND** escrowed (`offbox.go:564-569`), so the fork-4 gate blocked the run. The ceremony landed at 07:20:28. |
Both boxes became runnable only this morning. **The decisive run is 2026-08-05 ~02:15 UTC on both.**
Reported as unmeasured rather than inferred.
### What source says will happen — and it is a third outcome
Same sub-account, same repo path, new password. Measured coordinates (I3, `$.dr_recipe`):
`u629488-sub3.your-storagebox.de:/home/felhom-repo`**unchanged** across the whole incident, and
`repoPath` is a compile-time constant (`hub/internal/offsite/offsite.go:85`). So the next run opens an
**existing repo with the wrong key**, which the codebase already has a name for
(`controller/internal/backup/offbox.go:67-93`):
```go
// ErrOffboxOrphaned is the sentinel returned when the offsite repo exists but is keyed under a
// passphrase this controller no longer has (the reinstall shape) ...
case strings.Contains(s, "wrong password or no key found"):
return "orphaned"
```
and `ensureOffboxRepo` (`offbox.go:663-679`) branches on claim state:
```go
if !m.settings.GetClaimed() { // UNCLAIMED → auto move-aside + re-init
...
m.markOrphaned()
return ErrOffboxOrphaned // CLAIMED → skip the run, show the orphan card
```
Both boxes report `claimed: 1` (I3, `$.claimed`, latest report each). **So the predicted outcome is:
the run REFUSES with `ErrOffboxOrphaned`, produces no snapshot at all, and waits for the customer to
confirm a reset.** Not 15 (reattached) and not 1 (silently fresh) — a **third** outcome the question's
framing did not contain. That is the good news half: the system is not going to quietly start a new
history over the old one. It will stop and say so.
**Consequence if the customer confirms the reset:** `resetOrphanedRepo` (`offbox.go:270-314`) moves the
old repo aside and re-inits — *"move-aside, not deleted"*. The 40.9 MB + 1.14 GB of old ciphertext
survives, unreadable without the superseded escrow, **and keeps consuming the 50 GB soft quota
indefinitely.** Nothing prunes a moved-aside restic repo (S-24 covers ep0's PBS namespaces, a
different tier).
**To measure it tomorrow:** re-read `$.offsite.last_status` / `last_error` on both boxes after
02:20 UTC, and `$.offsite.repo_state` for the orphan flag. **Predicted `ErrOffboxOrphaned`, not a
snapshot count.** Whoever picks this up should record which of the three actually occurred — a
prediction from source is not a measurement.
---
## Q1 — What is one-shot, and what is not?
Three secrets, and they have almost nothing in common.
| | **① storage-provider password** | **② the box's SFTP key** | **③ restic repository password** |
|---|---|---|---|
| **What it is** | the Hetzner sub-account / box password | ed25519 keypair for `sftp` transport | the **data** key — restic's repo encryption |
| **Generated by** | **hub**`genPassword()` (`offsite.go:481`) | **controller**`KeyGen.Generate()` (`offsiteapply.go:234`) | **controller**`generateOffboxPassword()`, 32 random bytes hex (`offbox.go:406`) |
| **Stored where** | `one_time_secrets` (hub, plaintext, **one row per customer**) | `<DataDir>/offbox/ssh_key` 0600 (box only) | `<DataDir>/offbox/repo_password` 0600 (box) + inside the R-wrapped escrow blob (hub, opaque) |
| **Delivered how** | `GET …/offsite/consume-password`, **served exactly once** then marked consumed (`api/offsite.go:9-31`) | never delivered — installed onto the provider by ssh-copy-id using ① | never delivered anywhere; it never leaves the box except into the escrow ceremony |
| **Re-issuable?** | **YES**`ReissueCredentials` resets it at the provider, any time, operator-initiated | **YES** — regenerated on every full apply; `dr_recipe.go:45` states it plainly: *"the SFTP access key is regenerated at DR"* | **NO automatic path.** Recoverable only by unsealing the escrow with the customer's recovery code and pasting it into `offboxInjectPasswordHandler` |
| **What a guest rebuild does to it** | nothing (it lives on the hub / at the provider) | destroys it; a fresh one is minted and installed — **harmless** | **destroys it; a fresh one is minted — CATASTROPHIC for the existing repo** |
| **Escrowed?** | no | no | **yes**`IdentityBundle.ResticRepoPassword` |
The agent's own source names the asymmetry, and it is the sentence the whole spike turns on
(`felhom-agent/internal/escrow/identity.go:35-39`):
> `ResticRepoPassword` … **It is the DATA key for the offsite tier — irreplaceable (unlike the SFTP
> access key, which is regenerable at DR).**
**The one-shot thing (①) is the recoverable one. The irreplaceable thing (③) is the one nothing
re-stages.** R-193 was filed against ①. The damage is in ③.
**Instrument caveat, worth carrying (I4):** `one_time_secrets` has `customer_id` as PRIMARY KEY, and
`SaveOneTimeSecret` is last-write-wins **by design** (the R-39(a) guard comment in
`monitor/offsite_delivery.go:26-30` depends on it). So the delivery ledger holds only the newest
credential per customer: demo-hp's 2026-07-23 consume, on which R-192's whole diagnosis rests, was
**overwritten** by the 2026-08-04 Re-issue and is no longer readable from that table.
---
## Q4 — Which claim about Re-issue is true?
### **RULING: `ReissueCredentials` touches ONLY the provider credential (①). It does NOT touch the restic repository password. R-39's record is wrong — and the hub's own comments repeat the same wrong claim in three places.**
**Method — source, quoted rather than summarised.** `ReissueCredentials`
(`hub/internal/offsite/offsite.go:150-228`) does exactly four things:
1. `genPassword()`;
2. `ResetSubaccountPassword` / `ResetBoxPassword` + `WaitAction`**the provider account password**;
3. `Store.SaveOneTimeSecret(customerID, pw)` — stage ① for delivery;
4. bookkeeping: `MarkEscrowStale` + `offsite_reissued` + `escrow_stale` events.
There is **no reference to a restic password anywhere in the function**, and none is possible — the
repo password is generated on the box and never leaves it except into the escrow blob. The hub does
not hold it in any openable form (`api/handler.go`: *"The hub stores the bytes and NEVER decrypts them
(it has no recovery code)"*).
**But step 4 is justified by a claim that is false**, `offsite.go:198-201`:
> `// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing`
> `// key-escrow blob — which sealed the OLD password — is now STALE.`
The escrow *is* marked stale; the stated reason for marking it is not something this function does.
The same false premise appears twice more: `api/handler.go:1067-1069` (*"The re-issuer resets the
restic repo password, which makes the OLD escrow blob stale"*) and `OPEN-ITEMS.md`'s R-193 row, which
inherited it from R-39.
> **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does
> not provide** — and the first where the comment is not merely unenforced but factually describes a
> different function. It survived because it reads as settled and because its *effect* (a stale escrow)
> is real, so nobody checked the *cause*.
**Consequence, and it is a live defect, not a documentation nit.** On the ordinary Re-issue shape —
a consumed-but-failed install on a box that still has its `repo_password` file — the box re-applies,
`WriteOffboxSecrets` finds the file present and **keeps it**, and the repo password is unchanged. The
hub has nonetheless told the customer, in Hungarian, that their recovery escrow is stale and asked
them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.****R-196**.
**Corroboration against what actually happened on demo-hp yesterday.** The escrow went `pending` after
the Re-issue and the task asks: if the repo password was untouched, what explains that? **It is fully
explained, and not by the Re-issue.** Two independent mechanisms, both hub-side-invisible:
1. `ApplyOffsiteTarget` (`offbox.go:482-494`) carries `EscrowState` over from the *existing* target —
but the rebuilt guest had **no existing target**, so `cur` was nil, so the fallthrough
`if tgt.EscrowState != "escrowed" { tgt.EscrowState = "pending" }` fired.
2. The box had minted a **new** repo password moments earlier (Q2), so `pending` was *correct* — the
new key genuinely was not escrowed.
**The Re-issue's `escrow_stale` event fired at 07:11:51 for the wrong reason and happened to be true
for a different one.** Coincidence, not correctness. The proof that these are independent is
**demo-felhom**: it received **no Re-issue at all** (measured — its only `escrow_stale` /
`offsite_reissued` events in the entire history are dated `2026-07-21 08:29:29`), its repo password
changed anyway, and **nothing marked its escrow stale for thirteen hours.**
> **The mechanism is exactly inverted.** `escrow_stale` is wired to the one path that does **not**
> change the repo password, and absent from the path that **does**.
---
## Q5 — Why did one box recover itself and the other not?
### **RULING: confirmed from the ledger — an unconsumed staged secret, thirteen days old. And the recovery was partial in a way the narrative missed.**
**Method — I4, the `one_time_secrets` ledger, no values read:**
| customer | `created_at` | `consumed_at` |
|---|---|---|
| `demo-felhom` | `2026-07-21 08:29:29` | `2026-08-03 07:17:58` |
| `demo-hp` | `2026-08-04 07:11:51` | `2026-08-04 07:12:06` |
demo-felhom's row was **created 2026-07-21 and sat UNCONSUMED for 13 days** — traceable to a Re-issue
at that exact timestamp (its `offsite_reissued` + `escrow_stale` event pair, `2026-07-21 08:29:29`).
When the rebuild dropped its target on 2026-08-03, the fresh controller consumed the spare at 07:17:58
and was reporting `offsite` again by 07:19:10. **76 seconds, on a credential nobody had planned to
leave there.** Confirmed from the ledger, not the narrative.
**What would have happened without one:** exactly demo-hp — `consume`**404** (`api/offsite.go:20-22`,
`no unconsumed offsite password`), the apply-bridge returns
`offsite-apply: consume one-time password: …` and retries forever, and the tier stays absent until an
operator acts. demo-hp sat there **25 hours** (`2026-08-03 06:12:19``2026-08-04 07:12:01`).
**The correction to the contrast (see Q2):** demo-felhom's spare restored *delivery* only. Its 36
snapshots / 1.14 GB were orphaned by the same rebuild. **Neither box's data survived; one box's
plumbing did.**
*Not established:* demo-felhom's escrow generation before 2026-07-21. `host_escrow_superseded` holds
only two rows in the whole database (ids 3 and 4, one per box, both superseded 2026-08-04), so whether
the 2026-07-21 Re-issue's ceremony re-sealed an *unchanged* password — the predicted false-staleness
shape — **cannot be shown from this data**. Stated as unknown rather than asserted.
---
## Q6 — Why is delivery one-shot at all?
### **RULING: the design's reason is that ① is a LIVE PROVIDER PASSWORD, not a bootstrap token; and the honest answer to the security question is that an automatic restage weakens EXPOSURE WINDOW, not authentication.**
**The design's own reasons, from source:**
- `api/offsite.go:9-12`*"serves the one-time transient offsite password to the controller **EXACTLY
ONCE** … The value is returned once then marked consumed — a second call 404s. NEVER logged."*
- `offsite.go:145-149`*"the **EXPLICIT** operator recovery for a consumed-password dead-end … It is
**NOT** implicit rotation: ProvisionOffsite never calls this."*
- `offsiteapply.go:1-6` — the apply-bridge is *"idempotent (a descriptor hash marker prevents
**re-consuming a spent password**)"*, i.e. the design treats a wasted consume as a real hazard, which
R-71a's whole settle-gate exists to prevent.
**The security half, stated rather than skipped.** The question is: *could a machine that is not the
real one obtain a credential via an automatic restage?*
**What stands there today is authentication, and a restage does not remove it.** The consume endpoint
is gated by `checkAuthCustomer` (`api/offsite.go:14-18`) — **the customer's API key, the same
credential as the config pull**. Anything that could consume a restaged password can already pull the
customer's whole config. So an automatic restage **grants no new access to an unauthenticated party**.
**What it genuinely weakens is the exposure window.** Today a live Hetzner sub-account password is
fetchable only in the seconds after an operator deliberately staged one. Under an automatic restage it
becomes fetchable **whenever the box reports no offsite target** — a condition an attacker holding the
API key can arguably induce. The credential's blast radius at the provider is the customer's own
sub-account (label-scoped; `ReissueCredentials` refuses unless the label lookup finds **exactly one**,
`offsite.go:164,180`), so this is a widening of *when*, not of *what*.
**And one further guard must survive any change** — R-39(a), `monitor/offsite_delivery.go:26-30`:
`SaveOneTimeSecret` is last-write-wins, so restaging on top of an **unconsumed** secret clobbers a
password a box may be about to consume. Any automatic restage must re-read the row immediately before
acting and refuse unless it is still a **consumed** row. That guard is correct and is not what R-192 is
about.
---
## Q7 — R-192's guard: exactly what it reads, and why it read wrong
### **RULING: both halves confirmed, with numbers. The guard reads the OLDEST 500 reports since the consume, and for demo-hp all 500 predated the rebuild by six days.**
**The query, quoted** (`hub/internal/store/store.go:987`):
```sql
SELECT report_json FROM reports WHERE customer_id = ? AND received_at > ? ORDER BY id LIMIT 500
```
`ORDER BY id` ascending with `LIMIT 500` = **the oldest 500**, not the newest.
**Reproduced against the live data (I1)**, with demo-hp's real consume anchor `2026-07-23 09:53:41`
(recovered from R-192's record — it is no longer in `one_time_secrets`, see I4):
| measurement | value |
|---|---|
| `total` (what the guard sees) | **500** |
| `withOffsite` (what the guard sees) | **500** |
| oldest report in that window | `2026-07-23 09:53:47` |
| newest report in that window | **`2026-07-28 11:17:40`** |
| true `total` over the same window, no LIMIT | 1174 |
| true `withOffsite` over the same window, no LIMIT | 1063 (⇒ 111 without — matching the 111 ABSENT reports in I3) |
**The whole 500-report evidence set ends 2026-07-28 — six days before the 2026-08-03 rebuild.**
**Defect (a) — the message is exactly inverted.** `maybeEmitStuck`
(`monitor/offsite_delivery.go:~110`) interpolates `status.ReportsSinceConsume` into a string that
hardcodes the phrase *"carry no offbox target"*, and never reads `OffsiteReportsSinceConsume`. The
live e-mail, quoted in full from the events table:
> *"Offsite delivery stuck: one-time password consumed 284h19m0s ago and **500 report(s) since carry
> no offbox target** — the credential is likely burned (apply died between consume and persist).
> Re-issue delivers a fresh one."*
`OffsiteReportsSinceConsume` for that same call was **500**. Every one of them carried a target. The
message states the precise negation of its own measurement, and prescribes a remedy for a failure mode
that did not occur.
**Defect (b) — the heal refuses silently, on stale evidence.** `maybeHeal` returns bare on
`status.OffsiteReportsSinceConsume != 0` (it was 500), with **no log line**. `offsite_credential_restaged`
has never fired for any customer — confirmed: zero rows of that event type in the database.
**What the guard SHOULD discriminate on — the shape, not the code.** The guard's intent is right:
*"there is offbox evidence, so this is a regressed apply, not a burned credential — the operator's
call."* Its error is that **it asks the question of the wrong time period**. Offbox evidence from
before a rebuild is not evidence that the credential still works. The discriminator must be
**recency-bounded and rebuild-aware**: judge on evidence *after the newest `controller_started` /
config-hash change*, or on the latest N reports — never on "everything since the consume". And a
refusal must **name its reason in the log**, because "we chose not to heal" and "the heal never ran"
must not look identical (the S-16 rule, one layer down).
**Do not fix it in isolation.** Under Q2's finding, a successful auto-restage on demo-hp would have
restored the transport and the box would *still* have minted a new repo password. **The heal cannot
protect the data; it can only protect the plumbing.** Whatever shape it takes must say so.
---
## Q8 — What each candidate would cost
**A fourth option exists and is the important one**, because (a) and (b) both address ① and Q2 shows
the damage is in ③.
### (a) The hub restages automatically when a re-enrolled box reports no offsite
**A version of this ALREADY EXISTS and is wired to the wrong event.** `reissueOnReenroll`
(`hub/internal/api/handler.go:1051-1084`), leg **F3**:
> *"F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
> one-time password only ever reached the OLD controller)."*
It is called from `handleHostEnroll` — but **after** a mint-once-reuse short-circuit
(`handler.go:1004-1016`): `if existing != nil { …return… }`. **A guest rebuild leaves the `hosts` row
intact** (measured: `demo-hp-bb76ea` is the same host_id throughout the incident; only the agent leaf
re-keyed, `host_leaf_changed` 2026-08-03 06:09:40), so re-enroll returns the existing credential and
**F3 is never reached.**
- **Changes:** hub only. Either lower the trigger from host-enrollment to a controller-level rebuild
signal (the `config_hash` change + `controller_started` pair the hub already receives and already
logs), or fix R-192's guard so the existing R-71c self-heal can act.
- **Weakens:** the exposure window in Q6. Needs the R-39(a) unconsumed-secret guard kept intact.
- **Does NOT solve:** ③. The box still mints a new repo password and still orphans the repo. **This
option restores the plumbing to a box whose data key is already gone.**
- **Cost:** small. **Value against the actual harm: near zero.**
### (b) The credential becomes recoverable from escrow at re-bootstrap
> **ANNOTATED 2026-08-04 (recon) — this ruling is overturned in part. See the annotation at the top.**
> The constraint is real but narrower than written: an **unattended** rebuild cannot recover from a
> zero-knowledge escrow; a **customer-present** one can, and the operator has ruled on that shape.
> The recon also found that the chain this option would use is clientless at three links and lossy at
> a fourth (`RECON-offsite-dr-chain-2026-08-04.md` §3–§4).
- **BLOCKED AS STATED, and this is a hard constraint, not an estimate.** The escrow blob is
**R-wrapped and zero-knowledge** — the hub holds opaque bytes and has no recovery code
(`api/handler.go`, and D6 in `CONTEXT.md` S-3). **A rebuilt box cannot unseal it without the
customer's recovery code.** Any design that says "recover it at re-bootstrap" is describing a
customer-present ceremony, i.e. what the existing manual `offboxInjectPasswordHandler` already is.
- **Does NOT solve:** an unattended rebuild, which is the whole scenario.
- **Cost: not implementable as specified.**
### (c) — NOT PREVIOUSLY NAMED — the agent retains the repo password across a guest rebuild
The agent lives **on the Proxmox host** and survives a guest rebuild. It already receives the repo
password today: the controller pushes it over the pinned local API
(`POST /escrow/stage-secret``agent/internal/localapi/escrow_stage.go`), and the agent writes it to
the fixed 0600 path `/var/lib/felhom-agent/escrow-stage/restic_repo_password`
(`agent/internal/escrow/identity.go:44-46`) **transiently — wiped by the ceremony**
(`WipeStagedResticPassword`). The seam, the transport, the pinning and the file are all already built.
- **Changes:** agent (retain rather than wipe; serve back over the same pinned local API) + controller
(on a fresh data dir, ask the agent before minting — the `InjectOffboxPassword` seam already exists
and is exactly the right shape).
- **Weakens:** the escrow's *zero-knowledge* posture in one specific way — the irreplaceable data key
would now sit at rest on the Proxmox host, which the operator has root on. **Measured against D6,
that is not a new exposure in principle** (D6 already states plainly that *"the operator cannot read
customer data"* was never the security property, and the operator holds root on every box) — but it
IS a new copy, and copies are the thing R-133 is already open about for the hub DB. **This is a real
trade and it is the operator's to make.**
- **Solves:** exactly the failure that occurred, unattended, on every rebuild, without any hub
involvement and without any credential leaving the premises.
- **Cost:** medium. Two repos, one existing seam each.
### (d) Do neither — keep the manual Re-issue, and fix what LIES
- **Changes:** R-192's message + refusal log; correct the three false comments about what Re-issue
rotates; and — the part that actually matters — **make a repo-password change VISIBLE**. The hub
already receives `restic_pw_sha256` on every escrow upload and already stores it. Comparing the new
hash against the superseded one is a two-line verdict, and it is the signal that was missing on
demo-felhom for thirteen hours → **R-197**. (Note the shape: **this spike answered its own hardest
question with a comparison the hub could be making automatically and is not.** Both values were
already in the database; nothing read them.)
- **Weakens:** nothing.
- **Does NOT solve:** the data loss. Every rebuild still orphans the repo; the operator simply finds
out the same day instead of never.
- **Cost:** small.
### Ranked, with a recommendation
| rank | option | why |
|---|---|---|
| **1** | **(d) now — the honesty pass** | It is cheap, it weakens nothing, and **it is the only option that helps the incident already in flight.** The alarm that lies is worse than no alarm; the repo-password-changed detector is the one signal whose absence let demo-felhom lose 1.14 GB of history silently. Ship this regardless of what is decided about the rest. |
| **2** | **(c) — agent-retained repo password** | The only candidate that addresses ③, which is where the harm is. Every seam it needs exists. It costs one deliberate trade-off the operator must actually make. |
| **3** | **(a) — auto-restage** | Worth doing **after** (c), never instead of it. On its own it restores the transport to a box whose data key is already gone, and would have made both boxes look healthy on 2026-08-04 while the snapshots were orphaned — a *strictly worse* outcome than the current loud failure. |
| **4** | **(b) — recover from escrow at re-bootstrap** | Not implementable as stated; the escrow is zero-knowledge by design. |
**Recommendation: ship (d), then decide (c).** And do not ship (a) first — it would have hidden this.
---
## What could not be established
Named rather than guessed (§8 rule 4).
1. **The 2026-08-05 02:15 UTC run's actual outcome** (Q3). Predicted `ErrOffboxOrphaned` from source +
claim state; **not measured**.
2. **Whether the orphaned ciphertext still exists at the provider.** 40.9 MB (demo-hp) + 1.14 GB
(demo-felhom) are *presumed* present at `/home/felhom-repo` — reading the endpoint needs the SFTP
credential, which this session did not touch. The Hetzner API has no directory-listing surface.
3. **Whether demo-felhom's 2026-07-21 Re-issue re-sealed an unchanged password** (the predicted
false-staleness shape, Q4/Q5). Only two `host_escrow_superseded` rows exist in the whole database;
the generation before 2026-07-21 is not retained.
4. **What removed demo-hp's offsite target at the rebuild** is fully explained (fresh data dir), but
**why the guest was rebuilt on both boxes on consecutive mornings** is outside this spike.
---
## The operator's question — stated, and NOT answered here
> **The irreplaceable secret in the offsite tier is the restic repository password. It is generated on
> the box, it is destroyed by a guest rebuild, and the only copy that survives is sealed under the
> customer's recovery code — which nothing but a human can open. Every machine is going to be
> reinstalled.**
>
> **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host, where
> you already have root — or do you want it to stay sealed under the customer's recovery code and
> accept that every rebuild starts a new repository and orphans the old one, provided the system says
> so loudly on the day it happens?**
Option (c) is the first. Option (d) alone is the second. **This spike does not choose.**
File diff suppressed because one or more lines are too long
+12 -1
View File
@@ -26,7 +26,7 @@
| R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css`**0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints**`backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | | R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css`**0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints**`backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 |
| R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | | R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 |
| R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/<customer>`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 | | R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/<customer>`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 |
| R-1 | **Peti convergence***the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path | | R-1 | **Peti convergence***the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path. **SCOPE-CORRECTED 2026-08-04 (R-193 spike, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`): "auto-fire on re-enrollment" means HOST re-enrollment, and only when the host record was DELETED first.** `reissueOnReenroll` sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`, `api/handler.go:1004-1016`), so a **guest** rebuild — which leaves the `hosts` row intact — never reaches F2 or F3. Measured live: demo-hp's guest was rebuilt 2026-08-03, `demo-hp-bb76ea` was unchanged throughout, F3 did not fire, and the box had **no off-site copy for 25 hours** until an operator ran Re-issue by hand. Peti's clean-slate (a real host delete + re-enroll) still walks the supported path; **the arc does not cover a guest rebuild, and that is the gap R-193 owns.** Note also that F3 alone would NOT have saved the data — the rebuilt controller mints a fresh restic repo password regardless (spike Q2) |
| R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) | | R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
| R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b | | R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none``p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** | | R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none``p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** |
@@ -57,6 +57,17 @@
| R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. **Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17):** the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above | | R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. **Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17):** the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above |
| R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/<id>` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. | | R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/<id>` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. |
| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** | | R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
| R-198 | **The superseded-escrow retention kept the K-escrow and dropped the identity blob** — so it preserved the PBS key and destroyed the offsite restic repository password, with the escrow CEREMONY as the destroying act | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | Corrects the escrow-ceremony capability-map row's v0.60.0 retention claim (was struck through 2026-08-04, now restated as what ships). One column + one SELECT list, in the shared `demoteCurrentEscrowTx`, so both callers are fixed at once. Nothing backfillable. **Load-bearing, not tidy:** with R-193(c) refused, customer-present recovery is the only route back from a rebuild and it runs entirely through the retained identity blob |
| R-197 | **The hub stored both halves of "did this box's offsite data key change" and compared them nowhere** | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | New `offsite_repo_key_changed` (warning, operator-only, edge-triggered, no hash values). Flips nothing in the capability map — it adds a signal, not a capability |
| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **CLOSED — behaviour shipped hub v0.95.0 (2026-08-05)** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping **Collapsed 2026-08-05.** The comment correction landed in v0.93.0; the BEHAVIOUR landed in v0.95.0 with R-204 item 2 — the precautionary `MarkEscrowStale` and its `escrow_stale` event are gone from `ReissueCredentials`. Removing a signal was safe because the real case is measured twice over, and the mark was **blinding** one of them: a stale flag empties `restic_pw_sha256` from the ACK, which is what the controller's per-ACK re-check compares against. |
| R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one |
| R-199 | **The hub served recovery blobs on endpoints with no client anywhere** — the DR capstone's retrieval had never been called by the agent, the hub UI, a script or a runbook | M | **SHIPPED + PROVEN-LIVE (hub v0.94.0 + agent v0.125.0, 2026-08-04)** | Adds the capability-map row *"the offsite repository password can be RECOVERED from the sealed escrow"* as PROVEN-LIVE — for the KEY only, explicitly not for a restore. Chain links 68 walked on demo-felhom: recovered sha256 == on-disk sha256 == the hub's stored hash. Carries a deliberate security trade (a box's own credential now suffices where an operator-armed recovery mode was required), recorded in the handler and overridable by one constant |
| R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed |
| R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
| R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
| R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 |
| R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **ITEMS 13 SHIPPED (controller v0.198.0 + hub v0.95.0, 2026-08-05); ITEM 4 OPEN → R-193** | Collapsed per the coupling rule. The fix taken was R-196's — remove the guess, keep the two MEASURED signals (the controller's per-ACK hash re-check, which the stale flag was blinding, and R-197's supersession detector). Plus the claim-code read-through and the restore-scope wording. **Flips the capability-map recovery row from four crutches to one**, and that one is R-193's operator Re-issue, which keeps the row's *with a person present* qualifier. Does NOT touch R-198's retention, which is still unit-proven and needs the second drill |
| R-201 | **The wipe-and-recover drill** | L | **PASSED + PROVEN-LIVE (2026-08-04 night)** | Flips the off-site app-data row to *"a customer's file survives a machine rebuild and comes back"* — PROVEN-LIVE, scoped to demo-hp and to a controller-data rebuild. The journey took four undocumented manual steps (R-204); the capability is real, the customer experience is not yet |
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE | | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L11811219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L21462156, reinstall preserves existing), verified at L23322337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) | | R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L11811219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L21462156, reinstall preserves existing), verified at L23322337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor``06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0``06:57:19` `image file written, restarting bootstrap``06:57:21` container StartedAt + UpdateState `completed_at``06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f | | R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor``06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0``06:57:19` `image file written, restarting bootstrap``06:57:21` container StartedAt + UpdateState `completed_at``06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |
@@ -431,7 +431,7 @@ label. Filed under E-2.
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). | | 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. | | 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. | | 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. | | 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. **CORRECTION 2026-08-03, same day, measured on the box: it DID surface as a 403, exactly as this item predicted — the earlier annotation here said otherwise and was wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST: `POST /nodes/demo-felhom/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)`. The hub raised `whole_guest_backup_failed` at the first one (*"retrying with backoff"*) and edge-triggering correctly suppressed the rest, so the operator was told once. It ALSO surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). |
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. | | 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. | | 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
| 8 | The honest **single-drive label**. | | 8 | The honest **single-drive label**. |
+234
View File
@@ -1,3 +1,237 @@
## v0.95.0 — a re-issue no longer marks a healthy escrow stale (2026-08-05, R-196 / R-204 item 2)
`ReissueCredentials` called `MarkEscrowStale` on every re-issue that found an escrow row. The mark was
**precautionary, not evidential**: it guessed that the box's re-apply MIGHT mint a fresh repository
password. On the ordinary re-issue shape — a box that still holds its `<DataDir>/offbox/repo_password`
— the password does not change, so it marked a HEALTHY escrow stale. The 2026-08-04 recovery drill
(R-201) is what promoted this from a nit to a blocker.
**What the mark actually did**, because "asks for an unnecessary ceremony" understates it:
1. `stale_at` set → `GetEscrowStatusForCustomer` WITHHOLDS `restic_pw_sha256` from the report ACK.
2. With no hash, the controller's SLICE-3 auto-confirm cannot flip pending→escrowed.
3. `OffboxRunnable() = OffboxConfigured() && EscrowState=="escrowed"` → **every off-site backup
refused, indefinitely**, on a box whose key was never in doubt.
4. The customer is told to re-run the recovery ceremony — which mints a new recovery code and
supersedes the sealed blob. During a recovery that is the one act that would have destroyed the key
just recovered.
A precautionary flag that stops the data protection it is guarding is not conservative.
**Removing it leaves no gap — the case it guessed at is measured elsewhere, and better.**
- **Continuous, box-side:** the controller compares the ACK's sealed hash against its CURRENT local repo
password on every report ACK (`report.EscrowAutoConfirmer.reconcileEscrowed`, the Scenario-F
re-check). In the guest-rebuild shape — the only shape where a re-issue is followed by a fresh
repository password — that mismatches within one report cycle and raises the stale card plus the
„create a new recovery code" CTA. Pinned by the controller's `TestEscrowStale_MismatchWarnsOnceAndFlags`.
**And the mark was BLINDING it:** by emptying the hash (step 1) it removed the very value that
comparison needs, leaving the box able to report only the hash-LESS reason, which is false.
- **Edge-triggered, hub-side:** R-197's `offsite_repo_key_changed` fires on a proven hash difference
across a supersession and pages the operator. Pinned by `TestEscrowPut_ChangedRepoKey_RaisesSignal`;
red-proved by removing the `maybeEmitRepoKeyChanged` call.
`offsite_reissued` is unchanged and still always fires — the customer must still learn that the
credential moved. Only the stale marking and its `escrow_stale` event are gone.
**`MarkEscrowStale` is kept, not deleted, and now has no caller** — deliberately. The `stale_at` flag
is still live and correct (read by the ACK, the operator card and the PBS-DR view); the right way to
set it is a future caller that has MEASURED a key change rather than guessed at one. Its doc now says
so, and `TestEscrowStaleMechanism_StillWithholdsAndClears` keeps the mechanism from decaying while
nothing writes it.
**Disagreement recorded** (the R-96 standing rule): the task's Scenario D asked that a real key change
also "mark the escrow stale". It must not, and nothing was changed to make it — the hub learns of a
real change at the moment a supersession SEALS THE NEW PASSWORD, i.e. when the escrow is freshest.
Marking it stale there would ask for a ceremony to fix the ceremony that just ran. The correct
consequence at that instant is the operator alarm, which is what R-197 already does.
`TestReissue_InvalidatesEscrow` is replaced in place by `TestReissue_DoesNotMarkAHealthyEscrowStale`,
its exact inverse, so the reversal is visible in `git log -L`. The old test was not wrong about the
effect it measured; it was wrong about the cause, and it pinned the defect.
## v0.94.0 — a box can fetch its own sealed recovery package (2026-08-04, R-199 link 6)
**Chain link 6 had no client.** The hub has served the identity blob since slice 10D, from
`handleReEnroll` / `handleGetRestoreDirective` — gated on operator-armed recovery mode and the global
key. Nothing in the agent, the hub UI, any script or any runbook ever called either. The only
documented retrieval was `sqlite3 … writefile(…)` by hand on a `kubectl cp`-ed database.
**`GET /api/v1/hosts/{host_id}/escrow`** — the box-authenticated MIRROR of the PUT that put the blob
there. Self-scoped: a per-host key reads only its own; the global key may read any, the same asymmetry
the PUT has. A host with no sealed bundle gets `200 {present:false}` — a clean answer, because a 404
is indistinguishable from an unknown host and a bare empty 200 from a zero-length blob, and neither
of those is what "no ceremony has run yet" means.
**THE TRADE, RECORDED IN THE HANDLER RATHER THAN INFERRED.** Before this, obtaining the blob required
the OPERATOR to arm recovery mode. Now whoever controls a rebuilt box can obtain it with that box's
own credential. That is a real reduction in the number of parties required, and it is the whole of the
trade — they still cannot open it, because the hub has never held R and a wrong code fails closed at
age's scrypt KDF. **The mitigation is that the capability is audited rather than silent:** every
successful retrieval raises `escrow_blob_served` (warning, operator-only), recorded before the bytes
leave. A silent capability on this object is the shape the last two weeks were spent removing.
`escrowSelfServiceRetrieval` is a single named constant — the §8.2/§8.3 decision point. Flipping it to
false re-imposes the recovery-mode requirement on this endpoint and changes nothing else, so the
operator can overrule the trade at the cost of a boolean rather than a redesign.
**The operator-driven DR path is untouched** — same gate, same behaviour, pinned by a test that
exercises re-enroll and restore-directive with recovery mode off and on. Red-proofs observed: removing
the ownership check makes a cross-host read succeed (host A served host B's blob); removing the audit
record makes the retrieval silent.
**Not in this release:** the customer-facing flow. No card, no form, no preview — those are designed on
ground that has been walked, and R-200/R-201 are that walk.
## v0.93.0 — the retention keeps the key it was built to keep, and three things stop lying (2026-08-04, R-198/R-197/R-196/R-192)
### R-198 — the superseded-escrow retention was preserving the wrong key, and the ceremony was destroying the right one
`host_escrow_superseded` shipped in v0.60.0 with a `blob` column — the **K-escrow**, i.e. the PBS
datastore encryption key. `identity_blob` was added to `host_escrow` **later**, by the slice-10D
`ALTER TABLE`, and **never to the retained table**. The offsite **restic repository password** — the
data key for every off-site app-data snapshot — lives inside `identity_blob`. So
`demoteCurrentEscrowTx`, whose own comment calls it *"THE ONE escrow row-copy routine"*, retained the
whole-guest key and silently dropped the off-site data key: **precisely the secret the reinstall-orphan
incident the feature was built for was about.**
**What makes it the most serious defect this project has recorded is not the missing column — it is
which act destroys the key.** The copy happens as the new blob overwrites the old, so the destroying
act is the **escrow ceremony**: the exact thing a rebuilt box tells its customer to run, on a card
that promises in Hungarian that the old backups stay recoverable with their recovery code. Both demo
boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08), and their previous repository passwords
are now unrecoverable whether or not anyone kept the recovery codes.
- `host_escrow_superseded` gains `identity_blob BLOB` (in the CREATE for fresh databases **and** as an
additive `ALTER TABLE`, following the existing idiom; tolerated on re-run, changes no existing row).
- `demoteCurrentEscrowTx` carries it — so **both** callers are fixed by one change: the re-escrow path
(`SaveHostEscrow`) and the host-delete demotion (`DeleteHost`).
- `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob
is reachable from Go at all. `CountCurrentEscrowWithIdentity` is the census of who the fix protects.
- The comment above the table now records that the ruling stated there was not met, and what it cost.
- **Nothing is backfillable and the code says so:** rows superseded before this were written without
the identity blob and their source rows are already overwritten.
**Tests assert the CONSEQUENCE, not the mechanism** — the pre-existing
`TestSaveHostEscrow_RetainsSuperseded` was green throughout, because it asserted that a retained row
exists with the old K-blob. `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still
yield a repository password, and pins the load-bearing ordering (the identity blob is written **after**
`SaveHostEscrow`, so the demote sees the previous generation). `TestDeleteHost_DemotesIdentityBlob`
proves the shared routine through its **other** caller — a shared routine tested through one caller is
how a fix gets believed on a path nobody exercised. Red-proofs: dropping the column from the copy fails
both; fixing only the re-escrow caller fails the delete one while the re-escrow one passes.
### R-197 — the hub held both halves of the evidence and compared them nowhere
`restic_pw_sha256` is stored on the current escrow and carried to the retained row on every
supersession. Comparing the two is what answered the 2026-08-04 recon's hardest question in one query —
and nothing in the hub did it. demo-felhom's repository password changed on 2026-08-03, orphaning 36
snapshots / 1.14 GB, and **no event, e-mail, card or log line said so for thirteen hours.**
`SaveHostEscrow` now returns the hash it replaced, and `handleHostEscrowPut` raises
**`offsite_repo_key_changed`** (warning, operator-only, edge-triggered once per supersession) when both
hashes are known and differ. The in-between shapes are logged rather than dropped, so *"we chose not to
alarm"* and *"the check did not run"* never look identical. **No hash value travels** — not in the
message, not in the details; the event names the host, the customer and the retained count.
**Severity chosen for the world this release creates, not the one it inherits:** before R-198 a changed
key meant the previous history was unopenable by anyone, ever, which would have argued for `error`.
From v0.93.0 the superseding ceremony retains the old identity blob, so the situation is *"this
customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss.
`warning` also routes: the dispatcher treats `info` as an intentional non-notify.
### R-192 (partially) — the alert describes what was measured, and a refusal leaves a trace
demo-hp e-mailed this daily from 2026-08-03: *"one-time password consumed 284h19m0s ago and 500
report(s) since carry no offbox target — the credential is likely burned … Re-issue delivers a fresh
one."* `OffsiteReportsSinceConsume` for that same call was **500**: every one of those reports carried a
target. The message stated the precise negation of its own measurement and prescribed a remedy for a
failure mode that had not occurred — while the self-heal correctly declined, **in complete silence**.
- The one stuck state is now reported as the two situations it actually covers, **burned** and
**regressed**, each stating its own measurement and carrying its own recommendation (the regressed
text explicitly withdraws Re-issue and points at what removes an offbox target — a guest rebuild,
R-193). `offsite_reports_since_consume` is in the details for the first time.
- **The counting window is named in the text.** `CountReportsOffsiteSince` reads `ORDER BY id LIMIT
500` — the **oldest** 500 reports after the consume — so on a long-lived box the counts describe the
start of the window, not the present. **That guard defect stays OPEN (R-192)** because its correct
shape depends on the recovery chain that is not yet assembled (R-199/R-200/R-201); naming the window
is how it stays visible instead of being laundered into a confident sentence.
- **Every self-heal refusal now leaves a record** — a `notification_log` row on the operator channel
with status `refused` and its reason, the R-182 suppressed-e-mail precedent. It rides the stuck
event's 24 h cadence, so it sits beside the e-mail it explains rather than accumulating per tick.
**The guard's logic is unchanged**: the set of situations in which the heal fires is what it was.
- `offsite_delivery_stuck` and `offsite_credential_restaged` are added to `operatorOnlyEvents`. Neither
was ever registered, and neither has a `customerMessages` entry — which is **not** a block, so a
customer with a configured recipient was in line for an English e-mail about one-time passwords being
*"likely burned"*. Narrowing only; the operator channel is untouched.
### R-196 — five comments corrected (the spec expected three)
`ReissueCredentials` resets the **provider account** password. It does not and cannot touch the restic
repository password, which is generated on the box and never leaves it except sealed under R. Five
places in the hub said otherwise: `offsite/offsite.go` (the `MarkEscrowStale` justification),
`api/handler.go` (the F3 re-enroll comment), and three in `store/store.go` (the `stale_at` ALTER, the
`MarkEscrowStale` doc comment, and `EscrowStatus.Stale`). **That false belief sent two separate
investigations the wrong way in one day.** All five now say what the code does, name the correction and
cite the recon; the staleness mark is documented as **precautionary**, not evidential. **The behaviour
is unchanged and R-196 stays OPEN** — the false-staleness alarm it causes is a behaviour change and
must not ride a comment-correction release.
### Not in this release
**R-199, R-200 and R-201 remain open** — the hub's blob-serving endpoints still have no client, the
password-injection seam still has no form, and nothing in the recovery path has ever been exercised.
**Part 5 hit its gate and the orphan card is untouched** — see R-202: a single ACK boolean can say
whether a retained recoverable blob exists, but not whether one covers *the key this orphaned
repository was written under*, and a conditional promise that is still capable of being false is worse
on that surface than a hedged sentence.
## v0.92.0 — a customer with no machine ever bound does not alarm (2026-08-04, R-195)
`david` is a real prospective customer whose record was created 2026-08-01 with **no host ever bound**
`hosts=0`, `host_deletions=0`, `host_reports=0`, `reports=0` — and it e-mailed an
`expected_dbdump_missed` **error** at 03:00 UTC on 2026-08-02, 08-03 and 08-04. Nothing has ever been
expected of that customer, so the honest verdict is UNKNOWN, not missed.
**The mechanism is the interesting half, because the guard that should have covered this is keyed off
the wrong thing.** `CheckBackupDeadlines` already skips nodes that are `down`, and that skip reads
`StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — **a query over the
`reports` table**. A customer with zero reports is in no row, gets no state at all, and `GetState()`
returns `""` rather than `"down"`. **The skip misses exactly the customer it would most obviously
cover.** Corroborated on live data: `peti-felhom` is active with a host deleted 2026-07-15 and does
*not* alarm, because it has 482 old reports and is therefore `down`. The backup half was already safe
(`reportJSON == ""` → skip); the DB-dump half had no guard at all.
- **`store.HasEverBoundHost(customerID)`** — a live `hosts` row **OR** a `host_deletions` tombstone.
Consulted once per customer at the top of the deadline loop; a false verdict skips the customer with
an INFO line and its own counter in the summary.
- **The discriminator is "was a host EVER bound", NOT "has a report arrived", and that is the whole
design.** A box that was installed, bound and then went silent is a real fault and must keep
alarming — it has a `hosts` row, so it is judged. Only a customer that never had a machine is
UNKNOWN. The tombstone is included for the same reason: a customer whose host was deleted *had* one,
and that judgement belongs to the staleness checker, not to this predicate.
- **Fail-OPEN on a read error.** An unreadable binding must never SUPPRESS a real alarm; it logs and
judges anyway.
- **The anchored-verdict structure is untouched** (R-81/v0.73.0). This is the same invariant — absence
is UNKNOWN until an anchor elapses — applied one level up, at the question of whether there is a
subject at all.
**Tests, including the one that matters.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`
is a real shape — a machine installed and never phoned home — and it is indistinguishable from `david`
on every signal except the one the guard reads. If the guard is ever "simplified" to key off report
presence, customer age or a name pattern, it goes red. Plus a bound-then-went-quiet case, a
deleted-host case, and the predicate's own three inputs. **Companion red-proof observed:** removing the
guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]`
verbatim the event `david` sent three mornings running. Restored after.
**`david`'s record was not modified.** The record is correct; the alarm was what was wrong.
**Shipped alongside a spike that deliberately shipped NO code** —
`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md` (R-193/R-192): what is one-shot
in the offsite path, and what a guest rebuild really costs. It ends in ranked options and a question
for the operator; R-196 and R-197 were minted from it and are not implemented here.
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2) ## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's **Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
+203
View File
@@ -0,0 +1,203 @@
package api
import (
"encoding/base64"
"encoding/json"
"net/http"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// R-199 (hub v0.94.0) — the box-authenticated retrieval of a host's OWN sealed identity blob.
// This is a new surface on the most sensitive object in the system; these tests exist to pin the
// three properties that make it defensible: it is self-scoped, it is honest when there is nothing to
// serve, and it is never silent.
func seedEscrowedHost(t *testing.T, st *store.Store, hostID, customerID, apiKey string, identity []byte) {
t.Helper()
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: apiKey}); err != nil {
t.Fatal(err)
}
if _, _, err := st.SaveHostEscrow(hostID, []byte("k-escrow"), "fp", "zero_knowledge", "2026-08-04T11:00:00Z", "SHA"); err != nil {
t.Fatal(err)
}
if len(identity) > 0 {
if err := st.SaveHostDRBundle(hostID, identity, `{}`); err != nil {
t.Fatal(err)
}
}
}
// The happy path: a host reads back exactly the bytes it uploaded, verbatim.
func TestEscrowGet_ServesOwnBlobVerbatim(t *testing.T) {
h, st, _ := newTestHandler(t)
identity := []byte("\x00\x01age-wrapped-identity-bundle\xff")
seedEscrowedHost(t, st, "h1", "c1", "HKEY", identity)
rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", "")
if rr.Code != http.StatusOK {
t.Fatalf("GET escrow = %d, want 200 (%s)", rr.Code, rr.Body.String())
}
var out struct {
HostID string `json:"host_id"`
Present bool `json:"present"`
B64 string `json:"identity_escrow_b64"`
}
if err := json.Unmarshal(rr.Body.Bytes(), &out); err != nil {
t.Fatal(err)
}
if !out.Present || out.HostID != "h1" {
t.Fatalf("unexpected envelope: %+v", out)
}
got, err := base64.StdEncoding.DecodeString(out.B64)
if err != nil {
t.Fatal(err)
}
if string(got) != string(identity) {
t.Fatal("the served blob is not the stored blob — the hub must return ciphertext verbatim")
}
}
// Scenario C — a box can fetch ONLY its own blob.
// RED-PROOF: delete the `!isGlobal && authHostID != pathHostID` check in handleHostEscrowGet →
// cross-host retrieval succeeds → this FAILS. Without that line, any host key is a fleet-wide reader
// of every customer's sealed bundle.
func TestEscrowGet_CrossHostRefused(t *testing.T) {
h, st, _ := newTestHandler(t)
seedEscrowedHost(t, st, "hostA", "custA", "KEY-A", []byte("A-identity"))
seedEscrowedHost(t, st, "hostB", "custB", "KEY-B", []byte("B-identity-SECRET"))
rr := do(h, http.MethodGet, "/hosts/hostB/escrow", "KEY-A", "")
if rr.Code != http.StatusForbidden {
t.Fatalf("host A reading host B's blob = %d, want 403 — a cross-host read of a sealed bundle "+
"must be impossible on every code path (body: %s)", rr.Code, rr.Body.String())
}
if b := rr.Body.String(); len(b) > 0 && (contains(b, "B-identity") || contains(b, base64.StdEncoding.EncodeToString([]byte("B-identity-SECRET")))) {
t.Fatal("the refusal body leaked the other host's blob")
}
// Unauthenticated is refused too, and does not leak which hosts exist.
if rr := do(h, http.MethodGet, "/hosts/hostB/escrow", "", ""); rr.Code != http.StatusUnauthorized {
t.Fatalf("unauthenticated GET = %d, want 401", rr.Code)
}
// The global operator key MAY read any — the same asymmetry the PUT has.
if rr := do(h, http.MethodGet, "/hosts/hostB/escrow", globalKey, ""); rr.Code != http.StatusOK {
t.Fatalf("global key GET = %d, want 200", rr.Code)
}
}
func contains(hay, needle string) bool {
return len(needle) > 0 && len(hay) >= len(needle) && (func() bool {
for i := 0; i+len(needle) <= len(hay); i++ {
if hay[i:i+len(needle)] == needle {
return true
}
}
return false
})()
}
// Scenario D — a host with no sealed bundle gets a CLEAN answer: 200 with present=false. Not a 404
// (indistinguishable from an unknown host), not an empty 200 without the flag (indistinguishable
// from a zero-length blob). Both of those read as a fault to a caller whose situation is simply
// "no ceremony has run yet".
func TestEscrowGet_NoBlobIsCleanNone(t *testing.T) {
h, st, _ := newTestHandler(t)
seedEscrowedHost(t, st, "h1", "c1", "HKEY", nil) // K-escrow only, no identity blob
rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", "")
if rr.Code != http.StatusOK {
t.Fatalf("no-blob GET = %d, want 200 with present=false", rr.Code)
}
var out struct {
Present bool `json:"present"`
B64 string `json:"identity_escrow_b64"`
}
json.Unmarshal(rr.Body.Bytes(), &out)
if out.Present || out.B64 != "" {
t.Fatalf("a host with no bundle must report present=false and no bytes, got %+v", out)
}
// An unknown host is a DIFFERENT answer — the two must not collapse into one.
if rr := do(h, http.MethodGet, "/hosts/nope/escrow", globalKey, ""); rr.Code != http.StatusNotFound {
t.Fatalf("unknown host = %d, want 404 (distinct from a known host with no bundle)", rr.Code)
}
}
// Scenario E — every successful retrieval leaves a durable record naming the host.
// RED-PROOF: remove the SaveEvent call in handleHostEscrowGet → no event → this FAILS. A silent
// capability on this object is the whole reason §8.2's trade is acceptable at all.
func TestEscrowGet_LeavesARecord(t *testing.T) {
h, st, _ := newTestHandler(t)
seedEscrowedHost(t, st, "h1", "c1", "HKEY", []byte("age-wrapped"))
if rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", ""); rr.Code != http.StatusOK {
t.Fatalf("GET = %d", rr.Code)
}
ev, err := st.GetLatestEventByType("c1", eventEscrowBlobServed)
if err != nil {
t.Fatal(err)
}
if ev == nil {
t.Fatal("R-199: a sealed recovery blob was served and NOTHING recorded it — the audit row is " +
"the mitigation that makes this endpoint's trade acceptable")
}
if ev.Severity != "warning" {
t.Errorf("severity = %q, want warning: info is an intentional non-notify, so the operator would "+
"never hear that the recovery path was used", ev.Severity)
}
if !contains(ev.Message, "h1") {
t.Errorf("the record must name the host, got %q", ev.Message)
}
// The record must not carry the blob itself — it is an audit row, not a second copy.
if contains(ev.Message, "age-wrapped") || contains(ev.DetailsJSON, "age-wrapped") ||
contains(ev.DetailsJSON, base64.StdEncoding.EncodeToString([]byte("age-wrapped"))) {
t.Errorf("the audit record embedded the blob: msg=%q details=%q", ev.Message, ev.DetailsJSON)
}
// A "no blob" answer is NOT a retrieval and must not raise one.
seedEscrowedHost(t, st, "h2", "c2", "KEY2", nil)
do(h, http.MethodGet, "/hosts/h2/escrow", "KEY2", "")
if ev, _ := st.GetLatestEventByType("c2", eventEscrowBlobServed); ev != nil {
t.Fatal("a present=false answer served no blob and must raise no retrieval record")
}
}
// Operator-tier by construction, registered in the same commit that mints the type.
func TestEscrowBlobServed_IsOperatorOnly(t *testing.T) {
if !notify.IsOperatorOnly(eventEscrowBlobServed) {
t.Fatalf("%s is not registered operator-only — a customer would receive operator-grade English "+
"about their sealed recovery bundle being handed out", eventEscrowBlobServed)
}
}
// Scenario G — the operator-driven DR path is UNTOUCHED: same gate, same behaviour. The new sibling
// must not loosen it, and must not share it.
func TestEscrowGet_OperatorDRPathUnchanged(t *testing.T) {
h, st, _ := newTestHandler(t)
seedEscrowedHost(t, st, "h1", "c1", "HKEY", []byte("age-wrapped"))
// re-enroll without recovery mode: still refused.
if rr := do(h, http.MethodPost, "/hosts/h1/re-enroll", "HKEY", `{"new_api_key":"x"}`); rr.Code != http.StatusForbidden {
t.Fatalf("re-enroll without recovery mode = %d, want 403 — the new endpoint must not loosen it", rr.Code)
}
// restore-directive without recovery mode: still refused.
if rr := do(h, http.MethodGet, "/hosts/h1/restore-directive", "HKEY", ""); rr.Code != http.StatusForbidden {
t.Fatalf("restore-directive without recovery mode = %d, want 403", rr.Code)
}
// …and arming it still works, i.e. the old path is functional, not merely closed.
if err := st.SetRecoveryMode("h1", time.Now().UTC().Add(10*time.Minute)); err != nil {
t.Fatal(err)
}
if rr := do(h, http.MethodGet, "/hosts/h1/restore-directive", "HKEY", ""); rr.Code != http.StatusOK {
t.Fatalf("restore-directive WITH recovery mode = %d, want 200", rr.Code)
}
// The new endpoint does NOT depend on recovery mode (that is the §8.2 trade, made explicit here so
// a future change to escrowSelfServiceRetrieval is visible as a test change).
if err := st.ClearRecoveryMode("h1"); err != nil {
t.Fatal(err)
}
if rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", ""); rr.Code != http.StatusOK {
t.Fatalf("self-service escrow GET with recovery mode OFF = %d, want 200 (escrowSelfServiceRetrieval=true)", rr.Code)
}
}
@@ -0,0 +1,146 @@
package api
import (
"encoding/base64"
"encoding/json"
"net/http"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// R-197 (v0.93.0) — a changed offsite repository key must be REPORTED. Both halves of the comparison
// (host_escrow.restic_pw_sha256 and the row it replaces) have been in the database since SLICE 3 and
// nothing read them: demo-felhom's repository password changed on 2026-08-03, orphaning 36 snapshots /
// 1.14 GB, and no event, e-mail, card or log line said so for thirteen hours.
//
// Driven through the real endpoint (PUT /hosts/<id>/escrow), not by calling the emitter, so the wiring
// is part of what is proven.
// escrowBodyWithHash is escrowBody plus the sealed-password hash and an identity blob — the fork-4
// upload shape. Hash VALUES here are test fixtures, never real secrets.
func escrowBodyWithHash(blob []byte, resticPwSHA, identity string) string {
m := map[string]string{
"blob_b64": base64.StdEncoding.EncodeToString(blob),
"key_fingerprint": "ab:cd:ef",
"posture": "zero_knowledge",
"created_at": "2026-08-04T05:00:00Z",
"restic_pw_sha256": resticPwSHA,
}
if identity != "" {
m["identity_blob_b64"] = base64.StdEncoding.EncodeToString([]byte(identity))
}
b, _ := json.Marshal(m)
return string(b)
}
// Scenario D — a new escrow sealing a DIFFERENT repo password raises exactly one operator signal,
// naming the host, carrying NO hash value.
// RED-PROOF: remove the maybeEmitRepoKeyChanged call from handleHostEscrowPut (or make its predicate
// always fall through) → no event → this FAILS.
func TestEscrowPut_ChangedRepoKey_RaisesSignal(t *testing.T) {
h, st, _ := newTestHandler(t)
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-gen1")); rr.Code != http.StatusOK {
t.Fatalf("first PUT = %d (%s)", rr.Code, rr.Body.String())
}
// Generation 1 is onboarding, not a change — nothing may fire yet.
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
t.Fatalf("a FIRST escrow must not raise %s: %+v", eventRepoKeyChanged, ev)
}
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_GEN2", "age-gen2")); rr.Code != http.StatusOK {
t.Fatalf("second PUT = %d (%s)", rr.Code, rr.Body.String())
}
ev, err := st.GetLatestEventByType("c1", eventRepoKeyChanged)
if err != nil {
t.Fatal(err)
}
if ev == nil {
t.Fatal("R-197: the repository key demonstrably changed and NO signal was raised — this is the " +
"thirteen-hour silence that let demo-felhom lose 1.14 GB unremarked")
}
if ev.Severity != "warning" {
t.Errorf("severity = %q, want warning (info is an intentional non-notify — the operator would not be told)", ev.Severity)
}
if !strings.Contains(ev.Message, "h1") {
t.Errorf("message must name the host, got %q", ev.Message)
}
// NO HASH VALUE may travel — not in the message, not in the details.
for _, forbidden := range []string{"SHA_GEN1", "SHA_GEN2"} {
if strings.Contains(ev.Message, forbidden) || strings.Contains(ev.DetailsJSON, forbidden) {
t.Errorf("a repo-password hash value leaked into the event (%s): msg=%q details=%q", forbidden, ev.Message, ev.DetailsJSON)
}
}
// Exactly one, not one per anything.
evs, _ := st.GetRecentEvents("c1", 50)
n := 0
for _, e := range evs {
if e.EventType == eventRepoKeyChanged {
n++
}
}
if n != 1 {
t.Fatalf("%s fired %d times for one supersession, want 1 (edge-triggered)", eventRepoKeyChanged, n)
}
}
// Scenario E — an UNCHANGED key says nothing. A customer re-running a ceremony for good reasons must
// not be punished for it with an alarm.
func TestEscrowPut_UnchangedRepoKey_Silent(t *testing.T) {
h, st, _ := newTestHandler(t)
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_SAME", "age-1"))
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_SAME", "age-2"))
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
t.Fatalf("a same-hash re-ceremony must be SILENT, got: %+v", ev)
}
// And it must not have created a superseded row either (pre-existing contract, re-asserted here
// because Scenario E's silence would also be produced by the supersede branch never running).
if n, _ := st.CountSupersededEscrow("h1"); n != 0 {
t.Fatalf("same-hash re-upload created %d superseded row(s), want 0", n)
}
}
// The in-between shapes must not be silent-and-indistinguishable either: a hash-less NEW blob cannot
// show whether the key changed, and saying nothing at all would look identical to "unchanged".
// Asserted at the level the code offers — no event (correct: nothing was measured) and no crash.
func TestEscrowPut_HashlessSupersession_NoSignal(t *testing.T) {
h, st, _ := newTestHandler(t)
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-1"))
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "", "age-2")); rr.Code != http.StatusOK {
t.Fatalf("hash-less supersession PUT = %d", rr.Code)
}
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
t.Fatalf("a hash-less new blob cannot EVIDENCE a change; it must not claim one: %+v", ev)
}
// The retention still ran (the hashes differ, so this IS a supersession) — the key is kept even
// though the change cannot be evidenced.
if n, _ := st.CountSupersededEscrow("h1"); n != 1 {
t.Fatalf("superseded rows = %d, want 1", n)
}
}
// R-97a's lesson, pinned in the SAME commit that mints the type: an operator-tier event that is not in
// operatorOnlyEvents reaches customers as raw English, because a missing customerMessages entry is NOT
// a block. Checked here rather than in notify so the type's two registers are asserted together.
func TestRepoKeyChanged_IsOperatorOnly(t *testing.T) {
if !notify.IsOperatorOnly(eventRepoKeyChanged) {
t.Fatalf("%s is not registered operator-only — a customer would receive operator-grade English "+
"about escrow custody", eventRepoKeyChanged)
}
// The two R-192 types have the same property and the same reason (see the register's comment).
for _, et := range []string{"offsite_delivery_stuck", "offsite_credential_restaged"} {
if !notify.IsOperatorOnly(et) {
t.Errorf("%s is not registered operator-only", et)
}
}
}
+1 -1
View File
@@ -155,7 +155,7 @@ func TestReportACK_EscrowStatus(t *testing.T) {
} }
// escrow row with identity blob + hash → the ACK carries all three fields // escrow row with identity blob + hash → the ACK carries all three fields
if _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil { if _, _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := st.SaveHostDRBundle("hv1", []byte("identity-blob"), "{}"); err != nil { if err := st.SaveHostDRBundle("hv1", []byte("identity-blob"), "{}"); err != nil {
+200 -4
View File
@@ -239,6 +239,12 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"): case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow") hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
h.handleHostEscrowPut(w, r, hostID) h.handleHostEscrowPut(w, r, hostID)
// R-199 (v0.94.0): the box-authenticated MIRROR of the PUT above — a host reads back its own
// opaque identity blob so it can be unsealed with the customer's recovery code. Distinct from the
// operator-driven DR path in dr.go, which stays exactly as it is (see handleHostEscrowGet).
case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
h.handleHostEscrowGet(w, r, hostID)
// G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the // G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the
// operator retrieves it via the /admin/ path (global key only). // operator retrieves it via the /admin/ path (global key only).
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"): case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"):
@@ -1065,9 +1071,13 @@ func (h *Handler) reissueOnReenroll(cc *store.CustomerConfig) {
} }
} }
// F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the // F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
// one-time password only ever reached the OLD controller). The re-issuer resets the restic repo // one-time password only ever reached the OLD controller). ⚠ CORRECTED 2026-08-04 (R-196): this
// password, which makes the OLD escrow blob stale — the offsite provisioner invalidates the // used to claim "the re-issuer resets the restic repo password, which makes the OLD escrow blob
// escrow (2.3) and emits both events. Skips silently when offsite isn't provisioned/enabled. // stale". It does not and cannot — the re-issuer resets the PROVIDER account password; the
// repository password is generated on the box and never leaves it except sealed under R. The
// provisioner does mark the escrow stale, but PRECAUTIONARILY (see the reasoning at
// offsite.ReissueCredentials), not because this call rotated anything.
// Skips silently when offsite isn't provisioned/enabled.
if h.offsiteReissuer != nil { if h.offsiteReissuer != nil {
if err := h.offsiteReissuer(context.Background(), cc.CustomerID); err != nil { if err := h.offsiteReissuer(context.Background(), cc.CustomerID); err != nil {
h.logger.Printf("[WARN] offsite re-issue on re-enroll for %s failed: %v", cc.CustomerID, err) h.logger.Printf("[WARN] offsite re-issue on re-enroll for %s failed: %v", cc.CustomerID, err)
@@ -1141,7 +1151,7 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
// Store the OPAQUE bytes. No decrypt path exists — the hub cannot open this. Part B (v0.60.0): // Store the OPAQUE bytes. No decrypt path exists — the hub cannot open this. Part B (v0.60.0):
// when this upload supersedes a DIFFERENT-passphrase old blob, the old one is RETAINED (not // when this upload supersedes a DIFFERENT-passphrase old blob, the old one is RETAINED (not
// overwritten) so its recovery-code-recoverable history survives (Viktor's data-first ruling). // overwritten) so its recovery-code-recoverable history survives (Viktor's data-first ruling).
superseded, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256) superseded, prevPwSHA, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256)
if serr != nil { if serr != nil {
h.logger.Printf("[ERROR] Failed to store escrow for host %s: %v", pathHostID, serr) h.logger.Printf("[ERROR] Failed to store escrow for host %s: %v", pathHostID, serr)
http.Error(w, "Internal error", http.StatusInternalServerError) http.Error(w, "Internal error", http.StatusInternalServerError)
@@ -1157,6 +1167,10 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
"A korábbi helyreállítási csomag megőrizve (új kulcs érkezett).", string(details), "hub"); eerr != nil { "A korábbi helyreállítási csomag megőrizve (új kulcs érkezett).", string(details), "hub"); eerr != nil {
h.logger.Printf("[WARN] escrow_superseded event save failed for %s: %v", pathHostID, eerr) h.logger.Printf("[WARN] escrow_superseded event save failed for %s: %v", pathHostID, eerr)
} }
// R-197: the box's offsite DATA key demonstrably changed. Both halves of that comparison
// have been stored since SLICE 3 and nothing read them — demo-felhom's key changed on
// 2026-08-03 and nothing said so for thirteen hours.
h.maybeEmitRepoKeyChanged(host.CustomerID, pathHostID, prevPwSHA, req.ResticPwSHA256, n)
} }
} }
// Slice 10D.1: optionally store the IDENTITY escrow blob + the non-secret DR directive alongside // Slice 10D.1: optionally store the IDENTITY escrow blob + the non-secret DR directive alongside
@@ -1185,6 +1199,188 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
w.Write([]byte(`{"status":"ok"}`)) w.Write([]byte(`{"status":"ok"}`))
} }
// eventRepoKeyChanged (R-197) — the box's offsite restic REPOSITORY password changed, proven by the
// hub's own stored hashes. Hub-internal (not in allowedEventTypes, like escrow_superseded) and
// registered operator-only in notify.operatorOnlyEvents.
const eventRepoKeyChanged = "offsite_repo_key_changed"
// eventEscrowBlobServed (R-199) — a host retrieved its own sealed identity blob. Hub-internal,
// operator-only. See handleHostEscrowGet for why every retrieval is loud.
const eventEscrowBlobServed = "escrow_blob_served"
// escrowSelfServiceRetrieval is THE SINGLE DECISION POINT for the §8.2/§8.3 trade (R-199).
//
// true (§8.2, shipped v0.94.0) — a host may read its own blob whenever it authenticates as itself.
// false (§8.3, the fallback) — the same read additionally requires operator-armed recovery mode.
//
// It is one condition on purpose: the operator may overrule the trade below, and switching must cost a
// boolean rather than a redesign. Everything else in the recovery chain is identical either way.
const escrowSelfServiceRetrieval = true
// handleHostEscrowGet serves a host its OWN opaque identity-escrow blob (R-199, v0.94.0).
//
// WHAT THIS GIVES OUT, WHY IT IS SAFE, AND WHAT IT CHANGES ABOUT WHO IS REQUIRED — recorded here so the
// next reader finds the trade rather than inferring it (the dr.go header convention).
//
// WHAT: the age-wrapped `IdentityBundle` — opaque ciphertext. It carries the offsite restic repository
// password, the tunnel token, the PBS token and the WG key. The hub stores these bytes and has no
// decrypt path; the recovery code R that opens them exists only in the customer's hands.
//
// WHY IT IS SAFE TO GIVE OUT: the blob is useless without R (age scrypt + ChaCha20-Poly1305; a wrong R
// fails closed at the KDF, never to a plausible-but-wrong bundle), and a 10-word EFF code carries ~129
// bits. The caller already authenticates as this host for its report, its desired state, its WG
// registration and its PBS token — this adds no new identity, only a new object, and it is the exact
// MIRROR of the PUT above, which is how the blob got here in the first place.
//
// WHAT IT CHANGES, STATED PLAINLY BECAUSE IT IS THE WHOLE OF THE TRADE: before this, obtaining the blob
// required the OPERATOR to arm recovery mode with the global key (dr.go). Now whoever controls a
// rebuilt box can obtain it with that box's own credential. That is a real reduction in the number of
// parties required. They still cannot open it. The mitigation is that the capability is AUDITED rather
// than silent: every successful retrieval raises an operator event (below), because a silent capability
// is the shape this project has spent two weeks removing.
//
// THE OPERATOR-DRIVEN DR PATH IS UNTOUCHED. `handleReEnroll` / `handleGetRestoreDirective` keep their
// recovery-mode gate and their global-key arming, and they serve the K-escrow and the directive as
// well. This endpoint serves ONE object to ONE authenticated owner. Do not merge them.
func (h *Handler) handleHostEscrowGet(w http.ResponseWriter, r *http.Request, pathHostID string) {
authHostID, _, isGlobal, ok := h.checkAuthHost(r)
if !ok {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
if pathHostID == "" {
http.Error(w, "Missing host_id", http.StatusBadRequest)
return
}
// SELF-SCOPED: a per-host key reads only its OWN escrow. The global operator key may read any —
// the same asymmetry the PUT has. Without this line any host key is a fleet-wide blob reader.
if !isGlobal && authHostID != pathHostID {
h.logger.Printf("[WARN] escrow GET REFUSED: host %s asked for %s's blob (self-scope)", authHostID, pathHostID)
http.Error(w, "Forbidden: host_id mismatch", http.StatusForbidden)
return
}
host, err := h.store.GetHost(pathHostID)
if err != nil {
http.Error(w, "Internal error", http.StatusInternalServerError)
return
}
if host == nil {
http.Error(w, "Unknown host_id", http.StatusNotFound)
return
}
// §8.3 fallback lives here and nowhere else.
if !escrowSelfServiceRetrieval && !host.InRecoveryMode(time.Now().UTC()) {
h.logger.Printf("[WARN] escrow GET REFUSED for %s — self-service retrieval is disabled and recovery mode is not armed", pathHostID)
http.Error(w, "Forbidden: host not in recovery mode (operator must arm it)", http.StatusForbidden)
return
}
bundle, berr := h.store.GetHostDRBundle(pathHostID)
if berr != nil {
h.logger.Printf("[ERROR] escrow GET for %s: %v", pathHostID, berr)
http.Error(w, "Internal error", http.StatusInternalServerError)
return
}
// A host with no sealed bundle gets a CLEAN ANSWER, not a fault: 200 with present=false. A 404
// here would be indistinguishable from an unknown host, and an empty 200 without the flag would be
// indistinguishable from a zero-length blob — both read as "something is broken" to a caller whose
// situation is simply "no ceremony has run yet".
if bundle == nil || len(bundle.IdentityBlob) == 0 {
h.logger.Printf("[INFO] escrow GET for %s: no identity blob stored (no ceremony has run)", pathHostID)
writeJSON(w, http.StatusOK, map[string]any{"host_id": pathHostID, "present": false, "identity_escrow_b64": ""})
return
}
// THE MITIGATION (§8.2). Recorded BEFORE the bytes leave, so a retrieval cannot be served without
// its audit row; a save failure is logged and does NOT block the response (the blob is opaque and
// refusing it would break a recovery over an audit hiccup — but the log line always exists).
//
// SEVERITY = warning, i.e. it reaches the operator by e-mail. Retrieval is not routine today: it
// happens during a recovery and nowhere else. IF a customer-facing self-service flow ever makes it
// routine, revisit this — but revisit it deliberately, do not let it decay to info because the
// mail became annoying.
if host.CustomerID != "" {
msg := fmt.Sprintf("Recovery blob served: host %s retrieved its own sealed identity escrow (%d opaque bytes). "+
"This is the recovery path in use — the blob cannot be opened without the customer's recovery code, which the hub never holds. "+
"If no recovery is in progress on that box, investigate.", pathHostID, len(bundle.IdentityBlob))
details, _ := json.Marshal(map[string]any{
"host_id": pathHostID,
"blob_bytes": len(bundle.IdentityBlob),
"self_scope": !isGlobal,
})
if _, eerr := h.store.SaveEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub"); eerr != nil {
h.logger.Printf("[WARN] %s event save FAILED for %s (serving anyway): %v", eventEscrowBlobServed, pathHostID, eerr)
} else if h.dispatcher != nil {
go h.dispatcher.ProcessEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub")
}
}
h.logger.Printf("[WARN] escrow blob SERVED to host %s (%d opaque bytes, self_scope=%v) — recovery path in use",
pathHostID, len(bundle.IdentityBlob), !isGlobal)
writeJSON(w, http.StatusOK, map[string]any{
"host_id": pathHostID,
"present": true,
"identity_escrow_b64": base64.StdEncoding.EncodeToString(bundle.IdentityBlob),
})
}
// maybeEmitRepoKeyChanged raises ONE operator signal per supersession when the sealed offsite repo
// password demonstrably changed. Both hashes have been stored since SLICE 3 (host_escrow and, since
// v0.60.0, host_escrow_superseded) and NOTHING compared them: demo-felhom's repository password
// changed on 2026-08-03, orphaning 36 snapshots / 1.14 GB, and no event, e-mail, card or log line
// said so for thirteen hours — the comparison that eventually found it is this one
// (audits/RECON-offsite-dr-chain-2026-08-04.md, R-197).
//
// THE PREDICATE IS DELIBERATELY NARROW: both hashes known AND different. A first-ever hash (prev "")
// is onboarding, not a change; a hash-less supersession (now "") cannot show a change happened; an
// identical hash is a re-ceremony of the SAME password, which is a normal healthy act and must stay
// silent or a customer is punished for re-running a ceremony. The in-between shapes are LOGGED rather
// than dropped, so "we chose not to alarm" and "the check did not run" never look identical.
//
// SEVERITY = warning, chosen for the world v0.93.0 creates rather than the one it inherits. Before
// R-198 a changed key meant the previous history was unopenable by anyone, ever — that would have
// argued for error. From v0.93.0 the superseding ceremony RETAINS the old identity blob, so the
// previous history stays recoverable with the recovery code that sealed it: the situation is "this
// customer's off-site history now depends on an older recovery code", which is operator-actionable
// (check the orphan card, expect a fresh repository) and is not a loss. warning also routes: the
// dispatcher notifies on warning/error/critical and treats info as an intentional non-notify, and
// the whole point of this row is that the operator learns on the day.
//
// EDGE-TRIGGERED: called only from the superseded branch of the escrow PUT, i.e. once per
// supersession, never per report. No timer lives here — the dispatcher owns cooldown.
//
// NO HASH VALUE TRAVELS. The message and the details name the host, the customer and the retained
// count only. The hashes are non-reversible, but a hash is still a fingerprint of a live secret and
// this project's rule is that values do not leave the store.
func (h *Handler) maybeEmitRepoKeyChanged(customerID, hostID, prevSHA, newSHA string, retained int) {
switch {
case prevSHA == "":
h.logger.Printf("[INFO] escrow for host %s: no previous repo-password hash recorded (first hash, or a legacy hash-less blob) — repo-key-change check not applicable", hostID)
return
case newSHA == "":
h.logger.Printf("[WARN] escrow for host %s: the NEW blob carries no repo-password hash (hash-less supersession) — whether the repository key changed CANNOT be determined from the hub's data", hostID)
return
case prevSHA == newSHA:
return // same password re-sealed: a healthy re-ceremony (Scenario E — silence is correct)
}
msg := fmt.Sprintf("Offsite repository key CHANGED for host %s: the new escrow seals a different repository password than the one it replaced. "+
"The previous off-site history is no longer opened by this box's current key. The superseding blob was retained (%d held), so that history stays recoverable with the recovery code that sealed it — verify the box's off-site tier reports a repository rather than an orphan card, and expect the next backup to start a fresh history.",
hostID, retained)
details, _ := json.Marshal(map[string]any{
"host_id": hostID,
"retained_count": retained,
"repo_key": "changed", // never the hash values
})
if _, err := h.store.SaveEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub"); err != nil {
h.logger.Printf("[WARN] %s event save failed for %s: %v", eventRepoKeyChanged, hostID, err)
return // audit row first: an e-mail without its event row lies (the OffsiteChecker convention)
}
h.logger.Printf("[WARN] offsite repository key CHANGED for host %s (customer %s) — previous history now depends on the superseded recovery code; %d retained blob(s)",
hostID, customerID, retained)
if h.dispatcher != nil {
go h.dispatcher.ProcessEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub")
}
}
// handleHostRecoveryCredentialPut vaults a host's break-glass root@pam console credential (TASK G1). // handleHostRecoveryCredentialPut vaults a host's break-glass root@pam console credential (TASK G1).
// SELF-SCOPED (a host key writes only its own; global may write any) — day-0 posts it with the // SELF-SCOPED (a host key writes only its own; global may write any) — day-0 posts it with the
// host api_key. The secret is stored at rest and NEVER logged (only the username + a length are // host api_key. The secret is stored at rest and NEVER logged (only the username + a length are
+34 -3
View File
@@ -336,7 +336,7 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest) midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest)
sinceUTC := midnightBudapest.UTC() sinceUTC := midnightBudapest.UTC()
var backupMissed, dbdumpMissed, skipped, deferred int var backupMissed, dbdumpMissed, skipped, deferred, unbound int
for _, id := range customerIDs { for _, id := range customerIDs {
// Skip nodes that are down — they already have staleness events // Skip nodes that are down — they already have staleness events
@@ -350,6 +350,37 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
continue continue
} }
// ── R-195: a customer with NO machine EVER bound is UNKNOWN, not missed ────────────────
//
// Both verdicts below ask "did the thing we expect every day happen?". For a customer
// that has never had a machine bound, nothing has ever been expected, so the honest
// answer is UNKNOWN — the same invariant assessBackupFreshness states above, applied one
// level up, at the question of whether there is a subject at all.
//
// The discriminator is "was a host EVER bound", NOT "has a report arrived". That is the
// case this check must not break: a box that was installed, bound, and then went silent
// has a real fault and must keep alarming. It is bound, so it is judged.
//
// WHY THIS WAS REACHABLE AT ALL, measured 2026-08-04: the down-skip above is what
// protects every other silent customer, and it reads the staleness checker's state — which
// is seeded from the `reports` table (store.GetCustomers). A customer that has NEVER
// reported appears in no report row, so it gets no staleness state at all and GetState()
// returns "" rather than "down". The skip misses exactly the customer it would most
// obviously cover, and the DB-dump half below then fires every night: `david`, a
// prospective customer whose record was created 2026-08-01 with no machine ever bound,
// e-mailed an expected_dbdump_missed ERROR at 03:00 UTC on three consecutive days.
//
// Fail-open on a read error: an unreadable binding must never SUPPRESS a real alarm.
if bound, berr := s.HasEverBoundHost(id); berr != nil {
logger.Printf("[WARN] Deadline check: failed to read host binding for %s (judging anyway): %v", id, berr)
} else if !bound {
// Visible, per the v0.73.0 Part-7 precedent below: a quiet check must never be
// indistinguishable from a check that did not run. Once daily, one line per customer.
logger.Printf("[INFO] Deadline check: %s has no host EVER bound — all deadline verdicts UNKNOWN (no alarm)", id)
unbound++
continue
}
// Backup freshness from the agent's host-report (PBS snapshots + vzdump), // Backup freshness from the agent's host-report (PBS snapshots + vzdump),
// the authoritative offsite-backup signal post-slice-8C. // the authoritative offsite-backup signal post-slice-8C.
reportJSON, rerr := s.GetLatestHostReportJSON(id) reportJSON, rerr := s.GetLatestHostReportJSON(id)
@@ -423,6 +454,6 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
} }
} }
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down)", logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down), %d unknown (no host ever bound)",
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped) len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped, unbound)
} }
@@ -0,0 +1,148 @@
package monitor
import (
"io"
"log"
"path/filepath"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// R-195 — a customer with NO machine EVER bound must not alarm; a customer WITH one must.
//
// Origin, measured on the live hub 2026-08-04: `david` is a prospective customer whose record was
// created 2026-08-01 16:51:49 with no host ever bound (hosts=0, host_deletions=0, host_reports=0,
// reports=0). It e-mailed an `expected_dbdump_missed` ERROR at 03:00 UTC on 08-02, 08-03 and 08-04.
//
// The mechanism, established at source: the down-skip in CheckBackupDeadlines reads
// StalenessChecker.GetState(), whose map is seeded from store.GetCustomers() — a query over the
// `reports` table. A customer with zero reports is in no row, so it has no state, and GetState()
// returns "" rather than "down". The skip that protects every other silent customer misses the one
// that never reported at all.
//
// These tests pin BOTH halves. A suite that only proved the silence would pass against an
// implementation that never alarms, which is strictly worse than the defect it replaces.
// newUnboundStore creates a store holding ONE active customer and NO host row at all.
func newUnboundStore(t *testing.T) *store.Store {
t.Helper()
st, err := store.New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatalf("store.New: %v", err)
}
t.Cleanup(func() { st.Close() })
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ck", RetrievalPassword: "p"}); err != nil {
t.Fatalf("SaveCustomerConfig: %v", err)
}
return st
}
// ── Half 1: the silence ────────────────────────────────────────────────────────────────────────
// TestCheckBackupDeadlines_NeverBoundHost_Silent is the david case.
//
// COMPANION RED-PROOF (observed): deleting the HasEverBoundHost guard from CheckBackupDeadlines
// makes this test fail with
//
// deadline_unbound_test.go: a customer with NO host ever bound must raise NOTHING;
// got [expected_dbdump_missed]
//
// which is verbatim the event `david` e-mailed three mornings running. Restored after.
func TestCheckBackupDeadlines_NeverBoundHost_Silent(t *testing.T) {
st := newUnboundStore(t)
// No UpsertHost, no host-report, no db_dump_completed event — nothing has ever been expected.
got := runDeadline(t, st)
if len(got) != 0 {
t.Fatalf("a customer with NO host ever bound must raise NOTHING; got %v", got)
}
}
// TestHasEverBoundHost pins the predicate itself across its three inputs, because the whole
// behaviour above turns on it and a predicate that answered `false` for everything would make the
// test above pass while silencing the entire fleet.
func TestHasEverBoundHost(t *testing.T) {
st := newUnboundStore(t)
if bound, err := st.HasEverBoundHost("c1"); err != nil || bound {
t.Fatalf("no host rows → want (false,nil); got (%v,%v)", bound, err)
}
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
t.Fatalf("UpsertHost: %v", err)
}
if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
t.Fatalf("live host row → want (true,nil); got (%v,%v)", bound, err)
}
// An unknown customer is never bound — the predicate must not answer from another customer's rows.
if bound, err := st.HasEverBoundHost("nobody"); err != nil || bound {
t.Fatalf("unknown customer → want (false,nil); got (%v,%v)", bound, err)
}
}
// ── Half 2: THE RED-PROOF THAT MATTERS — a bound machine must still alarm ───────────────────────
// TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms is the case the change could break,
// and it is a real shape: a machine that was installed and bound and never phoned home. It has a
// `hosts` row and zero reports — indistinguishable from `david` on every signal EXCEPT the one the
// guard discriminates on. If the guard is ever "simplified" to key off report presence, customer
// age or a name pattern, this test goes red.
func TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms(t *testing.T) {
st := newUnboundStore(t)
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
t.Fatalf("UpsertHost: %v", err)
}
// No host-report and no db_dump_completed — the box was bound and never said anything.
got := runDeadline(t, st)
if !has(got, "expected_dbdump_missed") {
t.Fatalf("a BOUND machine that never reported is a real fault and must still alarm; got %v", got)
}
}
// TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms: the machine reported once, days ago,
// and stopped. Nothing about the guard may suppress that.
//
// The staleness checker's down-skip is nil here (runDeadline passes nil), which is deliberate: it
// isolates THIS guard. In production a genuinely down node is skipped by staleness and gets its own
// node_down event — that path is unchanged and is not what this test is about.
func TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms(t *testing.T) {
st := newUnboundStore(t)
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
t.Fatalf("UpsertHost: %v", err)
}
// One host-report whose newest backup evidence is 9 days old, and a db dump that last
// completed 5 days ago (i.e. not since midnight).
report := hostReportJSON(t, [][2]string{{rfc(-9 * 24 * time.Hour), "ok"}}, nil)
if err := st.SaveHostReport("h1", "c1", []byte(report), store.HostReportDenorm{}); err != nil {
t.Fatalf("SaveHostReport: %v", err)
}
got := runDeadline(t, st)
if !has(got, "expected_backup_missed") {
t.Fatalf("a bound machine that went quiet with stale backups must still raise expected_backup_missed; got %v", got)
}
if !has(got, "expected_dbdump_missed") {
t.Fatalf("a bound machine that went quiet must still raise expected_dbdump_missed; got %v", got)
}
}
// TestCheckBackupDeadlines_DeletedHost_StillJudged: the host row is gone but a tombstone remains
// (peti-felhom's live shape). The customer HAD a machine, so this check must not take over the
// judgement — it hands off to the staleness down-skip exactly as before the change.
func TestCheckBackupDeadlines_DeletedHost_StillJudged(t *testing.T) {
st := newUnboundStore(t)
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
t.Fatalf("UpsertHost: %v", err)
}
if err := st.DeleteHost("h1", false); err != nil {
t.Fatalf("DeleteHost: %v", err)
}
if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
t.Fatalf("a DELETED host is still a machine that was once bound → want (true,nil); got (%v,%v)", bound, err)
}
got := runDeadline(t, st)
if !has(got, "expected_dbdump_missed") {
t.Fatalf("a customer whose host was deleted was still bound and stays judged here; got %v", got)
}
}
+104 -16
View File
@@ -94,40 +94,112 @@ func (c *OffsiteDeliveryChecker) Check() {
if age < stuckAfter { if age < stuckAfter {
continue // normal convergence window continue // normal convergence window
} }
c.maybeEmitStuck(cfg.CustomerID, status, age) emitted := c.maybeEmitStuck(cfg.CustomerID, status, age)
c.maybeHeal(cfg.CustomerID, status) c.maybeHeal(cfg.CustomerID, status, emitted)
} }
} }
// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer. // deliveryShape names the two situations the ONE stuck state actually covers. They need different
func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) { // text and different advice, and conflating them is R-192's defect (a).
type deliveryShape string
const (
// shapeBurned — NO report since the consume carried an offbox target: the apply never persisted.
// Re-issue is the indicated action.
shapeBurned deliveryShape = "burned"
// shapeRegressed — reports since the consume DID carry an offbox target and the latest does not:
// the credential worked and the target was later lost (a guest rebuild does exactly this, R-193).
// Re-issue is NOT indicated; it treats a symptom whose cause is elsewhere.
shapeRegressed deliveryShape = "regressed"
)
func shapeOf(status offsite.DeliveryStatus) deliveryShape {
if status.OffsiteReportsSinceConsume == 0 {
return shapeBurned
}
return shapeRegressed
}
// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer. Returns
// whether it emitted, so the heal's refusal record rides the same cadence rather than inventing one.
//
// R-192 defect (a), fixed here: the message used to interpolate ReportsSinceConsume (the TOTAL) into
// a hardcoded phrase "report(s) since carry no offbox target", and never consulted
// OffsiteReportsSinceConsume — the field that says the opposite. On demo-hp it stated, daily, that
// 500 reports carried no offbox target when all 500 of them did, and prescribed Re-issue for a
// failure mode that had not occurred. The message now STATES WHAT WAS MEASURED and lets the operator
// read it; the recommendation follows the shape rather than being hardcoded.
//
// THE WINDOW IS NAMED ON PURPOSE. CountReportsOffsiteSince reads `ORDER BY id LIMIT 500` — the OLDEST
// 500 reports after the consume, not the newest — so on a long-lived customer these counts describe
// the beginning of the window and not the present. That is a real scoping defect (R-192's other half)
// and it stays OPEN because its correct shape depends on the recovery chain that is not yet
// assembled (R-199/R-200/R-201). Naming the window in the text is how it stays visible instead of
// being laundered into a confident sentence — an instrument that can silently mis-scope its results
// must say so where it reports them.
func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) bool {
last, err := c.store.LastEventAt(customerID, eventDeliveryStuck) last, err := c.store.LastEventAt(customerID, eventDeliveryStuck)
if err != nil { if err != nil {
c.logger.Printf("[WARN] offsite-delivery: %s: cooldown read: %v", customerID, err) c.logger.Printf("[WARN] offsite-delivery: %s: cooldown read: %v", customerID, err)
return return false
} }
if !last.IsZero() && c.now().Sub(last) < stuckCooldown { if !last.IsZero() && c.now().Sub(last) < stuckCooldown {
return return false
}
shape := shapeOf(status)
var msg string
switch shape {
case shapeBurned:
msg = fmt.Sprintf("Offsite delivery stuck (BURNED-credential shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, NONE carried an offbox target, and the latest report carries none either. The credential never reached a persisted apply. Re-issue delivers a fresh one.",
age.Round(time.Minute), status.ReportsSinceConsume)
default:
msg = fmt.Sprintf("Offsite delivery stuck (REGRESSED-apply shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, %d DID carry an offbox target — and the latest report carries none. The credential was applied and worked; the target was lost afterwards. Re-issue is NOT the indicated action: find what removed the offbox target (a guest rebuild does, R-193). Automatic restage is deliberately withheld for this shape. NOTE: the counts cover at most the first 500 reports after the consume, so on a long-lived box they describe the start of the window, not now (R-192, open).",
age.Round(time.Minute), status.ReportsSinceConsume, status.OffsiteReportsSinceConsume)
} }
msg := fmt.Sprintf("Offsite delivery stuck: one-time password consumed %s ago and %d report(s) since carry no offbox target — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one.",
age.Round(time.Minute), status.ReportsSinceConsume)
details, _ := json.Marshal(map[string]any{ details, _ := json.Marshal(map[string]any{
"state": string(status.State), "state": string(status.State),
"consumed_at": status.Since.UTC().Format(time.RFC3339), "shape": string(shape),
"reports_since_consume": status.ReportsSinceConsume, "consumed_at": status.Since.UTC().Format(time.RFC3339),
"reports_since_consume": status.ReportsSinceConsume,
"offsite_reports_since_consume": status.OffsiteReportsSinceConsume,
"count_window": "oldest 500 reports after consumed_at (R-192, open)",
}) })
c.emit(customerID, eventDeliveryStuck, "warning", msg, string(details)) c.emit(customerID, eventDeliveryStuck, "warning", msg, string(details))
return true
} }
// maybeHeal fires the R-71c self-heal when the burned-credential shape is unambiguous: // maybeHeal fires the R-71c self-heal when the burned-credential shape is unambiguous:
// consumed ≥ stuckAfter ago, ≥ healMinReports consecutive reports since with ZERO offbox evidence, // consumed ≥ stuckAfter ago, ≥ healMinReports consecutive reports since with ZERO offbox evidence,
// one heal per healCooldown — and the R-39(a) guard holds at act time. // one heal per healCooldown — and the R-39(a) guard holds at act time.
func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus) { //
// R-192 defect (b), fixed here: every refusal above the "not configured" line now leaves a RECORD.
// The regressed-shape branch used to be a bare `return`, so the operator received a daily e-mail with
// the wrong story, no heal, and nothing anywhere saying why the heal declined — "we chose not to act"
// and "the heal never ran" looked identical. `offsite_credential_restaged` has never fired for any
// customer, and until now that fact was indistinguishable from the checker being dead.
//
// The record is a notification_log row (the dispatcher's suppressed-operator-e-mail precedent, R-182:
// a decision not to act is written down on the channel it would have used). It rides `recordRefusal`
// — true only when the stuck event was emitted this pass — so it appears once per stuckCooldown
// beside the e-mail it explains, rather than once per monitor tick. The GUARD ITSELF IS UNCHANGED:
// the set of situations in which the heal fires is byte-for-byte what it was; only the silence is
// gone. The two conditions are split into separate branches solely so each refusal can name its own
// reason.
func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus, recordRefusal bool) {
if c.reissuer == nil { if c.reissuer == nil {
return // no provisioner configured: the heal does not exist on this hub, so there is nothing to explain
}
if status.OffsiteReportsSinceConsume != 0 {
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
"regressed-apply shape: %d of the first %d report(s) after the consume DID carry an offbox target, so a burned credential is ruled out — a restage would treat a symptom whose cause is elsewhere. Operator's call (R-193).",
status.OffsiteReportsSinceConsume, status.ReportsSinceConsume))
return return
} }
if status.ReportsSinceConsume < healMinReports || status.OffsiteReportsSinceConsume != 0 { if status.ReportsSinceConsume < healMinReports {
return // box not reporting enough, or offbox evidence exists (regressed-apply shape) → operator's call c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
"only %d report(s) since the consume (need %d): the box has not reported enough for the burned shape to be unambiguous.",
status.ReportsSinceConsume, healMinReports))
return
} }
last, err := c.store.LastEventAt(customerID, eventCredentialRestaged) last, err := c.store.LastEventAt(customerID, eventCredentialRestaged)
if err != nil { if err != nil {
@@ -146,8 +218,8 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
return return
} }
if info == nil || info.ConsumedAt.IsZero() { if info == nil || info.ConsumedAt.IsZero() {
c.logger.Printf("[INFO] offsite-delivery: %s: heal refused — secret row is now %s (R-39(a) guard: never restage over an unconsumed secret)", c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
customerID, secretShape(info)) "R-39(a) guard: the secret row is now %s — restaging over an unconsumed secret would clobber a password the box may be about to consume.", secretShape(info)))
return return
} }
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute) ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
@@ -167,6 +239,22 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
string(details)) string(details))
} }
// recordHealRefusal makes a decision NOT to self-heal visible. Always logs; additionally writes a
// notification_log row on the "operator" channel with status "refused" when `record` is set (the
// stuck event was emitted this pass), so the refusal sits next to the e-mail that prompted the
// question. A LogNotification failure is logged, never swallowed, and never blocks the refusal — the
// refusal is the primary effect.
func (c *OffsiteDeliveryChecker) recordHealRefusal(customerID string, record bool, reason string) {
c.logger.Printf("[INFO] offsite-delivery: %s: self-heal REFUSED — %s", customerID, reason)
if !record {
return
}
if err := c.store.LogNotification(customerID, eventCredentialRestaged, "warning",
"Automatic offsite credential restage was NOT performed.", "refused", reason, "operator"); err != nil {
c.logger.Printf("[WARN] offsite-delivery: %s: could not record the heal refusal: %v", customerID, err)
}
}
func secretShape(info *store.OneTimeSecretInfo) string { func secretShape(info *store.OneTimeSecretInfo) string {
if info == nil { if info == nil {
return "absent" return "absent"
@@ -0,0 +1,223 @@
package monitor
import (
"encoding/json"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// R-192 (v0.93.0) — the alert must describe what was MEASURED, and a refusal to self-heal must leave a
// trace.
//
// What was wrong, quoted from the live e-mail demo-hp sent daily from 2026-08-03: "one-time password
// consumed 284h19m0s ago and 500 report(s) since carry no offbox target — the credential is likely
// burned … Re-issue delivers a fresh one." OffsiteReportsSinceConsume for that same call was 500:
// every one of those reports carried a target. The message stated the precise negation of its own
// measurement and prescribed a remedy for a failure mode that had not occurred — while the self-heal
// correctly declined, in complete silence.
// regressedFixture puts c1 into demo-hp's real shape: consumed >1h ago, reports WITH an offbox target
// after the consume, then reports WITHOUT — the latest carrying none.
//
// The report timestamps get jitter deliberately: a perfectly regular series lands exactly on
// boundaries and survives mutations that a realistic one catches.
func (h dtHarness) regressedFixture(t *testing.T, withOffsite, without int) {
t.Helper()
if err := h.st.SaveOneTimeSecret("c1", "x"); err != nil {
t.Fatal(err)
}
consumed := time.Now().UTC().Add(-30 * time.Hour).Format("2006-01-02 15:04:05")
staged := time.Now().UTC().Add(-31 * time.Hour).Format("2006-01-02 15:04:05")
if err := h.st.SetOneTimeSecretTimesForTest("c1", staged, consumed); err != nil {
t.Fatal(err)
}
jitter := []int{0, 3, 1, 7, 2, 5, 11, 4}
for i := 0; i < withOffsite; i++ {
if err := h.st.SaveReport("c1", []byte(dtReportWithOffsite)); err != nil {
t.Fatal(err)
}
time.Sleep(time.Duration(jitter[i%len(jitter)]) * time.Millisecond)
}
for i := 0; i < without; i++ {
if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil {
t.Fatal(err)
}
time.Sleep(time.Duration(jitter[(i+3)%len(jitter)]) * time.Millisecond)
}
}
func (h dtHarness) refusalRows(t *testing.T) []store.NotificationLogEntry {
t.Helper()
all, err := h.st.GetRecentNotifications("c1", 50)
if err != nil {
t.Fatal(err)
}
var out []store.NotificationLogEntry
for _, n := range all {
if n.Status == "refused" {
out = append(out, n)
}
}
return out
}
// Scenario F — the message states the measured state, not a denied cause.
// RED-PROOF: restore the old single hardcoded sentence (interpolate ReportsSinceConsume into
// "report(s) since carry no offbox target") → the regressed assertions FAIL.
func TestDeliveryChecker_RegressedShape_MessageMatchesMeasurement(t *testing.T) {
h := newDTHarness(t, true)
h.regressedFixture(t, 6, 2)
h.checker.Check()
saved := h.savedEvents(t, "offsite_delivery_stuck")
if len(saved) != 1 {
t.Fatalf("stuck events = %d, want 1", len(saved))
}
msg := saved[0].Message
// It must NOT assert the thing the data contradicts.
if strings.Contains(msg, "carry no offbox target") || strings.Contains(msg, "likely burned") {
t.Fatalf("R-192(a): the message still asserts a burned credential on a REGRESSED shape — this is "+
"the sentence that stated the negation of its own measurement:\n%s", msg)
}
// It must name the shape it detected and the measurement behind it.
if !strings.Contains(msg, "REGRESSED") {
t.Errorf("the message must name the detected shape, got:\n%s", msg)
}
if !strings.Contains(msg, "6 DID carry an offbox target") {
t.Errorf("the message must state the measured offbox-bearing count (6), got:\n%s", msg)
}
// It must not prescribe the action for a failure mode that did not occur.
if !strings.Contains(msg, "Re-issue is NOT the indicated action") {
t.Errorf("the message must withdraw the Re-issue recommendation for this shape, got:\n%s", msg)
}
// The counting window is named, because CountReportsOffsiteSince reads the OLDEST 500 — an
// instrument that can silently mis-scope must say so where it reports (R-192's other half, open).
if !strings.Contains(msg, "500") {
t.Errorf("the message must name its counting window, got:\n%s", msg)
}
var details map[string]any
if err := json.Unmarshal([]byte(saved[0].DetailsJSON), &details); err != nil {
t.Fatal(err)
}
if details["shape"] != "regressed" {
t.Errorf("details.shape = %v, want regressed", details["shape"])
}
// The field whose absence was defect (a) is now carried.
if got, ok := details["offsite_reports_since_consume"].(float64); !ok || int(got) != 6 {
t.Errorf("details must carry offsite_reports_since_consume=6, got %v", details["offsite_reports_since_consume"])
}
}
// The burned shape keeps its own (correct) text and its recommendation — the fix must not blunt the
// case that was always right.
func TestDeliveryChecker_BurnedShape_KeepsItsRecommendation(t *testing.T) {
h := newDTHarness(t, false)
h.burnedFixture(t, 5)
h.checker.Check()
saved := h.savedEvents(t, "offsite_delivery_stuck")
if len(saved) != 1 {
t.Fatalf("stuck events = %d, want 1", len(saved))
}
msg := saved[0].Message
if !strings.Contains(msg, "BURNED") || !strings.Contains(msg, "NONE carried an offbox target") {
t.Errorf("the burned shape must state its own measurement, got:\n%s", msg)
}
if !strings.Contains(msg, "Re-issue delivers a fresh one") {
t.Errorf("the burned shape must keep its recommendation, got:\n%s", msg)
}
var details map[string]any
_ = json.Unmarshal([]byte(saved[0].DetailsJSON), &details)
if details["shape"] != "burned" {
t.Errorf("details.shape = %v, want burned", details["shape"])
}
}
// Scenario G — a refusal to self-heal leaves a durable record naming its reason.
// RED-PROOF: replace the recordHealRefusal call in the regressed branch with a bare `return`
// (production behaviour up to v0.92.0) → no row → this FAILS.
func TestDeliveryChecker_HealRefusal_LeavesARecord(t *testing.T) {
h := newDTHarness(t, true)
h.regressedFixture(t, 6, 2)
h.checker.Check()
if h.reissuer.count() != 0 {
t.Fatal("the heal must NOT fire on a regressed shape — the guard is unchanged by this fix")
}
rows := h.refusalRows(t)
if len(rows) != 1 {
t.Fatalf("R-192(b): the heal declined and left %d record(s), want 1 — 'we chose not to act' and "+
"'the heal never ran' must not look identical", len(rows))
}
r := rows[0]
if r.Channel != "operator" {
t.Errorf("refusal channel = %q, want operator (the channel the heal's own event would use)", r.Channel)
}
if !strings.Contains(r.ErrorMessage, "regressed-apply shape") {
t.Errorf("the record must name WHY, got %q", r.ErrorMessage)
}
if !strings.Contains(r.ErrorMessage, "6 of the first 8") {
t.Errorf("the record must carry the measurement it refused on, got %q", r.ErrorMessage)
}
}
// The R-39(a) guard's refusal is recorded too — it was the one refusal that already logged, and it
// still must not be the only one that does.
//
// Reaching it needs the real TOCTOU, not a pre-staged row: staging BEFORE Check() makes the derive
// itself return staged_awaiting_consume and the pass never reaches the heal at all. Same technique as
// TestDeliveryChecker_R39aGuard_NeverRestagesOverUnconsumed — the operator stages mid-tick, on the
// stuck event's dispatch, which runs after the derive and before maybeHeal.
func TestDeliveryChecker_R39aRefusal_LeavesARecord(t *testing.T) {
h := newDTHarness(t, true)
h.burnedFixture(t, 5)
base := h.checker.onEvent
h.checker.onEvent = func(cid, et, sev, msg, det, src string) {
if et == "offsite_delivery_stuck" {
if err := h.st.SaveOneTimeSecret("c1", "operator-fresh"); err != nil {
t.Errorf("mid-tick stage: %v", err)
}
if err := h.st.SetOneTimeSecretTimesForTest("c1", "2026-07-23 12:00:00", ""); err != nil {
t.Errorf("mid-tick stamp: %v", err)
}
}
base(cid, et, sev, msg, det, src)
}
h.checker.Check()
if h.reissuer.count() != 0 {
t.Fatal("R-39(a): the heal must never restage over an unconsumed secret")
}
rows := h.refusalRows(t)
if len(rows) != 1 {
t.Fatalf("the R-39(a) refusal left %d record(s), want 1", len(rows))
}
if !strings.Contains(rows[0].ErrorMessage, "R-39(a)") {
t.Errorf("the record must name the guard, got %q", rows[0].ErrorMessage)
}
}
// The refusal record rides the stuck event's 24h cadence: it explains the e-mail the operator just
// received, and must not accumulate one row per monitor tick.
func TestDeliveryChecker_HealRefusal_RidesTheStuckCadence(t *testing.T) {
h := newDTHarness(t, true)
h.regressedFixture(t, 6, 2)
h.checker.Check()
h.checker.Check() // the stuck event is swallowed by its cooldown; so is the refusal record
h.checker.Check()
if n := len(h.refusalRows(t)); n != 1 {
t.Fatalf("refusal records = %d after three passes, want 1 (one per stuck e-mail, not per tick)", n)
}
}
@@ -102,7 +102,7 @@ func TestOffsiteStale_NeverRanEscrowAnchorWins(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
// escrow blob stored now (SaveHostEscrow stamps updated_at with datetime('now')) // escrow blob stored now (SaveHostEscrow stamps updated_at with datetime('now'))
if _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil { if _, _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil {
t.Fatal(err) t.Fatal(err)
} }
saveOffsiteReport(t, st, "c1", offsiteJSON(true, "escrowed", "", "", 0, 50)) saveOffsiteReport(t, st, "c1", offsiteJSON(true, "escrowed", "", "", 0, 50))
+23
View File
@@ -421,6 +421,29 @@ var operatorOnlyEvents = map[string]bool{
// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped // block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
// once (v0.78.0) and the comment above records it. // once (v0.78.0) and the comment above records it.
"backup_run_failures": true, "backup_run_failures": true,
// R-197 (v0.93.0). "The sealed offsite repository key changed" is a custody fact about escrow
// blobs. A customer can take no action on it — the remedy is the operator's inspection of the
// off-site tier — and the text is operator-grade English naming host ids and retained-blob
// counts. Listed here in the SAME commit that mints the type: an operator-tier type that is not
// registered here reaches customers as raw English, because a missing customerMessages entry is
// NOT a block (the v0.78.0 defect recorded above).
"offsite_repo_key_changed": true,
// R-192 (v0.93.0). These two predate the register and were never added to it — a real gap, not a
// tidy-up. `offsite_delivery_stuck` is severity warning, has no customerMessages entry, and
// therefore fell through to FormatCustomerEmail's raw-English fallback: a customer whose box hit
// the stuck shape was in line for an English e-mail about one-time passwords being "likely
// burned". Measured on the live hub: notification_log holds operator rows for demo-hp and no
// customer rows — which is NOT evidence the leg is blocked (it is equally consistent with no
// configured recipient), so the register makes it structural instead of incidental. Narrowing
// only: the operator channel is untouched.
"offsite_delivery_stuck": true,
"offsite_credential_restaged": true,
// R-199 (v0.94.0). A host retrieved its own sealed recovery blob. Operator-tier by construction:
// it names host ids and opaque byte counts, the customer can take no action on it, and its whole
// purpose is that the operator sees a capability being used. Registered in the same commit that
// mints the type — an operator-tier type absent from this register reaches customers as raw
// English (the v0.78.0 defect recorded above).
"escrow_blob_served": true,
} }
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the // IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
+49 -24
View File
@@ -195,35 +195,60 @@ func (p *Provisioner) ReissueCredentials(ctx context.Context, customerID, typ st
return fmt.Errorf("offsite: store re-issued one-time password: %w", err) return fmt.Errorf("offsite: store re-issued one-time password: %w", err)
} }
// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing // v0.57.0 (2.3, the escrow-honesty fix). ⚠ ITS STATED REASON WAS FALSE AND IS CORRECTED HERE
// key-escrow blob — which sealed the OLD password — is now STALE. A recovery code minted against // (R-196, 2026-08-04). The comment used to read "the restic repo password just changed" — this
// it would decrypt a password that no longer opens the repo. Mark the escrow stale so the hub // function does NOT change it and CANNOT: the repository password is generated on the box
// stops advertising "ceremony done" and the customer's escrow wizard is offered again; a fresh // (controller `WriteOffboxSecrets`) and never leaves it except sealed inside the R-wrapped escrow.
// ceremony seals the new password and clears the flag. Every credential change also emits a // What this function changes is the PROVIDER account password (①), which is a different secret at
// visible customer event (offsite_reissued always; escrow_stale only when a blob was invalidated). // a different layer. The false premise survived because the EFFECT (a stale escrow) is real, so
// Best-effort: the password reset already succeeded — a bookkeeping failure here must not fail it. // nobody checked the CAUSE — and it sent two separate investigations the wrong way in one day
escrowStaled := false // (audits/SPIKE-offsite-credential-recovery-2026-08-04.md Q4).
if host, herr := p.Store.GetHostByCustomer(customerID); herr == nil && host != nil { //
if esc, eerr := p.Store.GetHostEscrow(host.HostID); eerr == nil && esc != nil { // ⚠ THE MARK IS GONE (R-196 / R-204 item 2, hub v0.95.0, 2026-08-05). What used to stand here —
if serr := p.Store.MarkEscrowStale(host.HostID); serr != nil { // a `MarkEscrowStale` on every re-issue that found an escrow row, plus an `escrow_stale` customer
p.logf("[offsite] WARN mark-escrow-stale for %s: %v", customerID, serr) // event — was PRECAUTIONARY, not evidential: it guessed that the box's re-apply MIGHT mint a fresh
} else { // repository password. On the ordinary re-issue shape (a box that still holds its
escrowStaled = true // `<DataDir>/offbox/repo_password`) the password does not change, so it marked a HEALTHY escrow
} // stale. The 2026-08-04 recovery drill (R-201) is what promoted this from a nit to a blocker.
} //
} // WHAT THE MARK ACTUALLY DID, mechanically, because "it asked for an unnecessary ceremony"
// understates it by a lot:
// 1. `stale_at` set → `GetEscrowStatusForCustomer` WITHHOLDS `restic_pw_sha256` from the report
// ACK (store.go, the v0.57.0 rule).
// 2. With no hash, the controller's SLICE-3 auto-confirm cannot flip pending→escrowed
// (report.EscrowAutoConfirmer.Reconcile returns early on an empty hash).
// 3. `OffboxRunnable() = OffboxConfigured() && EscrowState=="escrowed"` → EVERY off-site backup
// is refused, indefinitely, on a box whose key was never in doubt.
// 4. The customer is told to re-run the recovery ceremony — which mints a NEW recovery code and
// supersedes the sealed blob. During a recovery that is the one act that would have destroyed
// the key just recovered.
// A precautionary flag that stops the data-protection it is guarding is not conservative.
//
// WHY REMOVING IT LEAVES NO GAP — the case it guessed at is MEASURED elsewhere, and better:
// • Continuous, box-side: the controller compares the ACK's sealed hash against its CURRENT
// local repo password on EVERY report ACK (`reconcileEscrowed`, the Scenario-F re-check). In
// the guest-rebuild shape — the only shape where a re-issue is followed by a fresh repository
// password — that comparison mismatches within one report cycle and raises the stale card plus
// the „create a new recovery code" CTA. It is a measurement, not a guess.
// AND THE MARK WAS BLINDING IT: by emptying the hash (step 1 above) it removed the very value
// that comparison needs, so the box could only report the hash-LESS reason, which is false.
// • Edge-triggered, hub-side: R-197's `offsite_repo_key_changed` fires on a proven hash
// difference across a supersession (api.maybeEmitRepoKeyChanged) and pages the operator.
//
// DISAGREEMENT RECORDED, per the R-96 standing rule: the task's Scenario D asks that a real key
// change "marks the escrow stale". It must NOT, and nothing here was changed to make it: the hub
// learns of a real change at the moment a supersession SEALS THE NEW PASSWORD, i.e. when the escrow
// is freshest. Marking it stale there would ask for a ceremony to fix the ceremony that just ran.
// The correct consequence at that instant is the operator alarm, which is what R-197 does.
//
// `offsite_reissued` is UNCHANGED and still always fires — the customer must still learn that the
// credential moved. Best-effort: the password reset already succeeded, so a bookkeeping failure
// here must not fail it.
if _, serr := p.Store.SaveEvent(customerID, "offsite_reissued", "info", if _, serr := p.Store.SaveEvent(customerID, "offsite_reissued", "info",
"Az offsite (házon kívüli) mentési hozzáférést újra kiadtuk — az új egyszeri jelszót a vezérlő a következő frissítéskor átveszi.", "Az offsite (házon kívüli) mentési hozzáférést újra kiadtuk — az új egyszeri jelszót a vezérlő a következő frissítéskor átveszi.",
"", "hub"); serr != nil { "", "hub"); serr != nil {
p.logf("[offsite] WARN save offsite_reissued event for %s: %v", customerID, serr) p.logf("[offsite] WARN save offsite_reissued event for %s: %v", customerID, serr)
} }
if escrowStaled {
if _, serr := p.Store.SaveEvent(customerID, "escrow_stale", "warning",
"A helyreállítási kulcs-letét elavult az offsite jelszó cseréje miatt — futtasd le újra a helyreállítási szertartást (Biztonsági mentés → Helyreállítás).",
"", "hub"); serr != nil {
p.logf("[offsite] WARN save escrow_stale event for %s: %v", customerID, serr)
}
}
return nil return nil
} }
+60 -15
View File
@@ -211,12 +211,27 @@ func TestReissue_RefusesAmbiguousLookup(t *testing.T) {
} }
} }
// v0.57.0 (2.3, escrow honesty) — re-issuing offsite credentials INVALIDATES the key-escrow blob: // SCENARIO C (R-196 / R-204 item 2, hub v0.95.0) — re-issuing the offsite PROVIDER credential must
// the blob sealed the OLD repo password, so a recovery code minted against it would decrypt a // LEAVE A HEALTHY ESCROW ALONE.
// password that no longer opens the repo. RED-PROOF (Scenario C): on pre-fix code (no MarkEscrowStale //
// in ReissueCredentials + no stale plumbing) the hub keeps advertising the escrow as current after a // THIS TEST REPLACES `TestReissue_InvalidatesEscrow`, which asserted the opposite and was wrong on a
// re-issue and keeps serving its restic-hash for auto-confirm — this test asserts it does NEITHER. // false premise: a re-issue resets the storage-provider account password and cannot touch the restic
func TestReissue_InvalidatesEscrow(t *testing.T) { // REPOSITORY password, which is generated on the box and never leaves it except sealed under R. The
// old assertion looked right because the EFFECT it checked (a stale escrow) was real — nobody checked
// the CAUSE. Kept as one test rather than deleted so the inversion is visible in `git log -L`.
//
// WHAT IT ASSERTS IS THE CONSEQUENCE, not the mechanism: not "MarkEscrowStale was not called" but
// "the ACK still carries the hash", because the hash is what the controller's auto-confirm needs and
// its absence is what blocked every off-site backup during the 2026-08-04 drill.
//
// RED-PROOF: restore the marking — put back
//
// if host, _ := p.Store.GetHostByCustomer(customerID); host != nil {
// if esc, _ := p.Store.GetHostEscrow(host.HostID); esc != nil { p.Store.MarkEscrowStale(host.HostID) }
// }
//
// in ReissueCredentials → es.Stale becomes true, the hash is withheld, and this test fails on both.
func TestReissue_DoesNotMarkAHealthyEscrowStale(t *testing.T) {
p, _, st := newTestProvisioner(t) p, _, st := newTestProvisioner(t)
const cust = "cust-esc" const cust = "cust-esc"
if _, err := p.ProvisionOffsite(context.Background(), cust, Input{Enabled: true, Type: "shared", QuotaGB: 10}); err != nil { if _, err := p.ProvisionOffsite(context.Background(), cust, Input{Enabled: true, Type: "shared", QuotaGB: 10}); err != nil {
@@ -226,10 +241,9 @@ func TestReissue_InvalidatesEscrow(t *testing.T) {
if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil { if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil { if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
t.Fatal(err) t.Fatal(err)
} }
// Before re-issue: current escrow — the hub serves the sealed hash and is NOT stale.
es, err := st.GetEscrowStatusForCustomer(cust) es, err := st.GetEscrowStatusForCustomer(cust)
if err != nil || es == nil { if err != nil || es == nil {
t.Fatalf("escrow status (before): %v", err) t.Fatalf("escrow status (before): %v", err)
@@ -238,25 +252,56 @@ func TestReissue_InvalidatesEscrow(t *testing.T) {
t.Fatalf("pre-reissue escrow must be current: stale=%v hash=%q", es.Stale, es.ResticPwSHA256) t.Fatalf("pre-reissue escrow must be current: stale=%v hash=%q", es.Stale, es.ResticPwSHA256)
} }
// Re-issue the offsite credential — the repo password just changed under the sealed blob. // Re-issue the offsite PROVIDER credential. The box still holds its repo_password file, so the
// repository password does NOT change — this is the ordinary re-issue shape.
if err := p.ReissueCredentials(context.Background(), cust, "shared"); err != nil { if err := p.ReissueCredentials(context.Background(), cust, "shared"); err != nil {
t.Fatalf("reissue: %v", err) t.Fatalf("reissue: %v", err)
} }
// After: the escrow is STALE and the restic-hash is WITHHELD (no auto-confirm against a dead key).
es, err = st.GetEscrowStatusForCustomer(cust) es, err = st.GetEscrowStatusForCustomer(cust)
if err != nil || es == nil { if err != nil || es == nil {
t.Fatalf("escrow status (after): %v", err) t.Fatalf("escrow status (after): %v", err)
} }
if !es.Stale { if es.Stale {
t.Fatal("RED-PROOF: escrow must be STALE after an offsite re-issue (the hub was advertising ceremony-done against a key the repo no longer accepts)") t.Fatal("a re-issue marked a HEALTHY escrow stale — this blocks every off-site backup and asks the customer for a ceremony that would supersede a good key (R-196)")
} }
if es.ResticPwSHA256 != "" { if es.ResticPwSHA256 != "OLDHASH" {
t.Fatalf("a stale escrow must WITHHOLD the restic hash to inhibit auto-confirm, got %q", es.ResticPwSHA256) t.Fatalf("the sealed hash must keep flowing to the ACK so the controller can auto-confirm; got %q", es.ResticPwSHA256)
} }
// The customer is still TOLD the credential moved — removing the false alarm must not remove the
// true notice. `offsite_reissued` is untouched by this change.
ev, eerr := st.GetLatestEventByType(cust, "offsite_reissued")
if eerr != nil || ev == nil {
t.Fatalf("offsite_reissued must still fire on every re-issue (err=%v ev=%+v)", eerr, ev)
}
// …and the false alarm is gone.
if bad, _ := st.GetLatestEventByType(cust, "escrow_stale"); bad != nil {
t.Fatalf("a re-issue must not raise escrow_stale on a healthy escrow: %+v", bad)
}
}
// The other half of Scenario C: the stale MECHANISM is intact and still does its job when something
// legitimately sets it. Without this, "nothing marks stale any more" could decay into "stale no longer
// works", and the next evidential caller would ship inert (the seam-built-but-never-wired shape).
func TestEscrowStaleMechanism_StillWithholdsAndClears(t *testing.T) {
_, _, st := newTestProvisioner(t)
const cust = "cust-stale-mech"
if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
t.Fatal(err)
}
if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
t.Fatal(err)
}
if err := st.MarkEscrowStale(cust + "-01"); err != nil {
t.Fatalf("MarkEscrowStale: %v", err)
}
es, _ := st.GetEscrowStatusForCustomer(cust)
if es == nil || !es.Stale || es.ResticPwSHA256 != "" {
t.Fatalf("a marked escrow must read stale AND withhold the hash: %+v", es)
}
// A fresh ceremony (new blob sealing the new password) clears stale + serves the new hash. // A fresh ceremony (new blob sealing the new password) clears stale + serves the new hash.
if _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob-2"), "SHA256:fp", "zero_knowledge", "2026-07-16T01:00:00Z", "NEWHASH"); err != nil { if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("blob-2"), "SHA256:fp", "zero_knowledge", "2026-07-16T01:00:00Z", "NEWHASH"); err != nil {
t.Fatal(err) t.Fatal(err)
} }
es, _ = st.GetEscrowStatusForCustomer(cust) es, _ = st.GetEscrowStatusForCustomer(cust)
+2 -2
View File
@@ -25,10 +25,10 @@ func seedRetainedBlob(t *testing.T, st *Store, customerID string) {
if err := st.UpsertHost(&Host{HostID: hostID, CustomerID: customerID, APIKey: "h"}); err != nil { if err := st.UpsertHost(&Host{HostID: hostID, CustomerID: customerID, APIKey: "h"}); err != nil {
t.Fatalf("upsert host: %v", err) t.Fatalf("upsert host: %v", err)
} }
if _, err := st.SaveHostEscrow(hostID, []byte("A"), "fpA", "p", "2026-01-01T00:00:00Z", "shaA"); err != nil { if _, _, err := st.SaveHostEscrow(hostID, []byte("A"), "fpA", "p", "2026-01-01T00:00:00Z", "shaA"); err != nil {
t.Fatalf("escrow A: %v", err) t.Fatalf("escrow A: %v", err)
} }
if _, err := st.SaveHostEscrow(hostID, []byte("B"), "fpB", "p", "2026-01-02T00:00:00Z", "shaB"); err != nil { if _, _, err := st.SaveHostEscrow(hostID, []byte("B"), "fpB", "p", "2026-01-02T00:00:00Z", "shaB"); err != nil {
t.Fatalf("escrow B: %v", err) t.Fatalf("escrow B: %v", err)
} }
if err := st.DeleteHost(hostID, true); err != nil { if err := st.DeleteHost(hostID, true); err != nil {
@@ -0,0 +1,168 @@
package store
import (
"bytes"
"testing"
)
// R-198 (v0.93.0) — the retention must keep the OFF-SITE data key, not only the PBS key.
//
// WHAT WAS BROKEN, and why these tests are the ones that would have caught it: host_escrow_superseded
// shipped with `blob` (the K-escrow / PBS datastore key) and `identity_blob` was added to host_escrow
// by a later ALTER and never to the retained table. The restic REPOSITORY password lives inside
// identity_blob. So every supersession retained the whole-guest key and destroyed the off-site data
// key — the exact secret the retention exists to preserve — and the destroying act is the escrow
// ceremony a rebuilt box asks its customer to run.
//
// The pre-existing TestSaveHostEscrow_RetainsSuperseded was GREEN throughout, because it asserts the
// MECHANISM (a retained row exists, with the old K-blob) and not the CONSEQUENCE (the retained row can
// still yield a repository password). These assert the consequence.
// Scenario A — a re-escrow retains BOTH sealed keys.
// RED-PROOF: drop `identity_blob` from demoteCurrentEscrowTx's INSERT/SELECT (production behaviour up
// to v0.92.0) → the retained row's identity blob is nil → this FAILS.
func TestSaveHostEscrow_RetainsIdentityBlob(t *testing.T) {
st := newTestStore(t)
const h = "hid-1"
oldIdentity := []byte("age-blob-sealing-REPO-PASSWORD-OLD")
newIdentity := []byte("age-blob-sealing-REPO-PASSWORD-NEW")
// Generation 1: the K-escrow, then the identity blob — the real order the escrow PUT uses
// (SaveHostEscrow, then SaveHostDRBundle).
if _, _, err := st.SaveHostEscrow(h, []byte("k-blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
t.Fatal(err)
}
if err := st.SaveHostDRBundle(h, oldIdentity, `{"gen":1}`); err != nil {
t.Fatal(err)
}
// Generation 2 with a DIFFERENT sealed repo password → supersede.
sup, prev, err := st.SaveHostEscrow(h, []byte("k-blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
if err != nil {
t.Fatal(err)
}
if !sup {
t.Fatal("a different-passphrase upload must supersede")
}
if prev != "SHA_OLD" {
t.Fatalf("prevResticPwSHA256 = %q, want SHA_OLD (R-197 needs the replaced hash)", prev)
}
if err := st.SaveHostDRBundle(h, newIdentity, `{"gen":2}`); err != nil {
t.Fatal(err)
}
retained, err := st.ListSupersededEscrow(h)
if err != nil {
t.Fatal(err)
}
if len(retained) != 1 {
t.Fatalf("retained rows = %d, want 1", len(retained))
}
// THE ASSERTION THIS WHOLE ITEM IS ABOUT: the off-site data key survived the supersession.
if retained[0].IdentityBlob == nil {
t.Fatal("R-198: the retained row carries NO identity blob — the off-site repository password " +
"was destroyed by the ceremony that was supposed to preserve it")
}
// And it is the PREVIOUS generation's blob, not the one that replaced it. This pins the ordering
// dependency named on demoteCurrentEscrowTx: the identity blob is written AFTER SaveHostEscrow, so
// the demote sees the old one. If that order ever inverts, the retained bytes would silently be
// the new blob filed under the old hash — recoverable-looking and wrong.
if !bytes.Equal(retained[0].IdentityBlob, oldIdentity) {
t.Fatalf("retained identity blob is not the PREVIOUS generation (got %q) — the demote ran after the overwrite",
retained[0].IdentityBlob)
}
if retained[0].ResticPwSHA256 != "SHA_OLD" || string(retained[0].Blob) != "k-blob-old" {
t.Fatalf("retained row is not the old generation: %+v", retained[0])
}
// Current row unchanged in behaviour: the NEW generation, both blobs.
if bundle, berr := st.GetHostDRBundle(h); berr != nil || bundle == nil || !bytes.Equal(bundle.IdentityBlob, newIdentity) {
t.Fatalf("current identity blob is not the new one: %+v (%v)", bundle, berr)
}
}
// Scenario B — deleting a host demotes BOTH blobs too. demoteCurrentEscrowTx is shared by the
// re-escrow path and the host-delete path; a shared routine proven through one caller is how a fix
// gets believed on a path nobody exercised.
// RED-PROOF: fix only the re-escrow caller (e.g. carry the column in SaveHostEscrow's own SQL instead
// of in the shared routine) → this FAILS while Scenario A passes.
func TestDeleteHost_DemotesIdentityBlob(t *testing.T) {
s := newTestStore(t)
const hostID, cust = "hid-del", "cust-del"
identity := []byte("age-blob-sealing-REPO-PASSWORD")
if err := s.UpsertHost(&Host{HostID: hostID, CustomerID: cust, APIKey: "k"}); err != nil {
t.Fatal(err)
}
if _, _, err := s.SaveHostEscrow(hostID, []byte("k-blob"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
t.Fatal(err)
}
if err := s.SaveHostDRBundle(hostID, identity, `{}`); err != nil {
t.Fatal(err)
}
if err := s.DeleteHost(hostID, true); err != nil {
t.Fatalf("DeleteHost: %v", err)
}
retained, err := s.ListSupersededEscrow(hostID)
if err != nil {
t.Fatal(err)
}
if len(retained) != 1 {
t.Fatalf("demoted rows = %d, want 1", len(retained))
}
if !bytes.Equal(retained[0].IdentityBlob, identity) {
t.Fatalf("R-198: host delete demoted custody WITHOUT the identity blob (got %q) — the off-site "+
"repository password was destroyed by a host delete", retained[0].IdentityBlob)
}
}
// Scenario C — a legacy host whose current escrow has NO identity blob supersedes cleanly. The column
// is nullable on purpose: a NOT NULL constraint here would make the fix block a ceremony, which is a
// worse failure than the one it repairs.
func TestSaveHostEscrow_SupersedesWithoutIdentityBlob(t *testing.T) {
st := newTestStore(t)
const h = "hid-legacy"
if _, _, err := st.SaveHostEscrow(h, []byte("k-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
t.Fatal(err)
}
// No SaveHostDRBundle — a slice-7-era upload.
sup, prev, err := st.SaveHostEscrow(h, []byte("k-new"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
if err != nil {
t.Fatalf("a supersession of an identity-less escrow must not fail: %v", err)
}
if !sup || prev != "SHA_OLD" {
t.Fatalf("superseded=%v prev=%q, want true/SHA_OLD", sup, prev)
}
retained, err := st.ListSupersededEscrow(h)
if err != nil {
t.Fatal(err)
}
if len(retained) != 1 {
t.Fatalf("retained rows = %d, want 1", len(retained))
}
if retained[0].IdentityBlob != nil {
t.Fatalf("a legacy row must retain a NULL identity blob, got %q", retained[0].IdentityBlob)
}
}
// CountCurrentEscrowWithIdentity is the census §8.1 asks for: which hosts hold an identity blob today
// and are therefore protected from the next ceremony by this fix. Asserted rather than eyeballed,
// because the report quotes its numbers.
func TestCountCurrentEscrowWithIdentity(t *testing.T) {
st := newTestStore(t)
if _, _, err := st.SaveHostEscrow("with-id", []byte("k"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA1"); err != nil {
t.Fatal(err)
}
if err := st.SaveHostDRBundle("with-id", []byte("age-blob"), `{}`); err != nil {
t.Fatal(err)
}
if _, _, err := st.SaveHostEscrow("without-id", []byte("k"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA2"); err != nil {
t.Fatal(err)
}
total, withIdentity, err := st.CountCurrentEscrowWithIdentity()
if err != nil {
t.Fatal(err)
}
if total != 2 || withIdentity != 1 {
t.Fatalf("census = %d/%d, want 2 total / 1 with identity", withIdentity, total)
}
}
+3 -3
View File
@@ -11,7 +11,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
const h = "h1" const h = "h1"
// 1st upload (P_old) — nothing to supersede. // 1st upload (P_old) — nothing to supersede.
sup, err := st.SaveHostEscrow(h, []byte("blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD") sup, _, err := st.SaveHostEscrow(h, []byte("blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -20,7 +20,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
} }
// 2nd upload (P_new, DIFFERENT sha) — must supersede + retain the old. // 2nd upload (P_new, DIFFERENT sha) — must supersede + retain the old.
sup, err = st.SaveHostEscrow(h, []byte("blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW") sup, _, err = st.SaveHostEscrow(h, []byte("blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -45,7 +45,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
} }
// 3rd upload, SAME sha as current — idempotent (re-ceremony of the same password): NO supersede row. // 3rd upload, SAME sha as current — idempotent (re-ceremony of the same password): NO supersede row.
sup, err = st.SaveHostEscrow(h, []byte("blob-new-2"), "fp-new", "zk", "2026-07-16T01:00:00Z", "SHA_NEW") sup, _, err = st.SaveHostEscrow(h, []byte("blob-new-2"), "fp-new", "zk", "2026-07-16T01:00:00Z", "SHA_NEW")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -15,10 +15,10 @@ func TestDeleteHost_DemotesEscrowNeverDestroys(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
// current escrow = SHA_A, one superseded = SHA_OLD (two uploads with different passphrases). // current escrow = SHA_A, one superseded = SHA_OLD (two uploads with different passphrases).
if _, err := s.SaveHostEscrow(hostID, []byte("blob-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil { if _, _, err := s.SaveHostEscrow(hostID, []byte("blob-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, err := s.SaveHostEscrow(hostID, []byte("blob-A"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil { if _, _, err := s.SaveHostEscrow(hostID, []byte("blob-A"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if n, _ := s.CountSupersededEscrow(hostID); n != 1 { if n, _ := s.CountSupersededEscrow(hostID); n != 1 {
+1 -1
View File
@@ -57,7 +57,7 @@ func seedHostWithArtifacts(t *testing.T, s *Store, hostID, customerID string) {
"PK-"+hostID, "ip-"+hostID, hostID); err != nil { "PK-"+hostID, "ip-"+hostID, hostID); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, err := s.SaveHostEscrow(hostID, []byte("opaque-escrow"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil { if _, _, err := s.SaveHostEscrow(hostID, []byte("opaque-escrow"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
t.Fatal(err) t.Fatal(err)
} }
} }
+126 -20
View File
@@ -374,6 +374,22 @@ func (s *Store) migrate() error {
-- "history destroyed" into "history recoverable with the recovery code"). Append-only; the hub -- "history destroyed" into "history recoverable with the recovery code"). Append-only; the hub
-- never decrypts; NO pruning (the blobs are tiny + R-encrypted; custody unchanged). The ACK and -- never decrypts; NO pruning (the blobs are tiny + R-encrypted; custody unchanged). The ACK and
-- restore-serving read host_escrow (the CURRENT row) never this table. -- restore-serving read host_escrow (the CURRENT row) never this table.
--
-- THE RULING ABOVE WAS NOT MET FOR TWO MONTHS, AND THIS IS THE RECORD OF IT (R-198, fixed
-- v0.93.0). This table shipped with the blob column the K-escrow, i.e. the PBS datastore key
-- and identity_blob was added to host_escrow LATER (the slice-10D ALTER below), never here.
-- The offsite restic REPOSITORY password lives in identity_blob, not in blob. So the retention
-- preserved the whole-guest key and silently dropped the off-site data key: precisely the
-- secret the reinstall-orphan incident was about. Worse, the copy happens as the new blob
-- overwrites the old, so the destroying act was the ESCROW CEREMONY the exact thing a
-- rebuilt box asks its customer to run, on a card promising the old backups stay recoverable.
-- Both demo boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08) and their previous
-- repository passwords are unrecoverable, recovery code or not.
-- identity_blob is now carried (see demoteCurrentEscrowTx, which is still THE ONE row-copy
-- routine). Pinned by TestSaveHostEscrow_RetainsIdentityBlob and
-- TestDeleteHost_DemotesIdentityBlob the routine is proven through BOTH of its callers,
-- because a shared routine tested through one caller is how a fix gets believed on a path
-- nobody exercised. Evidence: audits/RECON-offsite-dr-chain-2026-08-04.md §7.
CREATE TABLE IF NOT EXISTS host_escrow_superseded ( CREATE TABLE IF NOT EXISTS host_escrow_superseded (
id INTEGER PRIMARY KEY AUTOINCREMENT, id INTEGER PRIMARY KEY AUTOINCREMENT,
host_id TEXT NOT NULL, host_id TEXT NOT NULL,
@@ -382,7 +398,8 @@ func (s *Store) migrate() error {
posture TEXT NOT NULL DEFAULT '', posture TEXT NOT NULL DEFAULT '',
created_at DATETIME NOT NULL, created_at DATETIME NOT NULL,
restic_pw_sha256 TEXT NOT NULL DEFAULT '', restic_pw_sha256 TEXT NOT NULL DEFAULT '',
superseded_at DATETIME NOT NULL DEFAULT (datetime('now')) superseded_at DATETIME NOT NULL DEFAULT (datetime('now')),
identity_blob BLOB
); );
CREATE INDEX IF NOT EXISTS idx_host_escrow_superseded_host ON host_escrow_superseded(host_id); CREATE INDEX IF NOT EXISTS idx_host_escrow_superseded_host ON host_escrow_superseded(host_id);
@@ -414,16 +431,28 @@ func (s *Store) migrate() error {
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`) s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`)
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`) s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`)
// R-198 (v0.93.0) — the SAME column on the RETAINED table. It was added above and not here, and
// that omission is what made the retention keep the wrong key for two months (see the comment on
// host_escrow_superseded). Additive and tolerated on re-run, exactly like the lines above; it
// changes no existing row. Rows superseded BEFORE this ships were written without the identity
// blob and their source rows are already overwritten — there is nothing to backfill, and the
// v0.93.0 report records that as a looked-at fact rather than a deduction.
s.db.Exec(`ALTER TABLE host_escrow_superseded ADD COLUMN identity_blob BLOB`)
// SLICE 3 (escrow auto-confirm) — sha256 hex of the offsite restic repo password sealed in the // SLICE 3 (escrow auto-confirm) — sha256 hex of the offsite restic repo password sealed in the
// identity blob. The hash of a 256-bit random secret is non-reversible/non-brute-forceable — safe to // identity blob. The hash of a 256-bit random secret is non-reversible/non-brute-forceable — safe to
// store and serve; it lets the controller VERIFY "the escrow covers the CURRENT repo password" // store and serve; it lets the controller VERIFY "the escrow covers the CURRENT repo password"
// instead of trusting blob-presence. NULL/'' = a legacy or password-less blob (never auto-confirms). // instead of trusting blob-presence. NULL/'' = a legacy or password-less blob (never auto-confirms).
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`) s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`)
// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at is set when the offsite repo // v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at marks an escrow blob that may no
// password is re-issued: the blob then seals a password that no longer opens the repo, so the // longer cover the box's live repository password. While set, the hub stops advertising "ceremony
// hub must stop advertising "ceremony done" and withhold the (now non-matching) restic_pw_sha256 // done" and withholds restic_pw_sha256 from the auto-confirm ACK. NULL = current; a fresh ceremony
// from the auto-confirm ACK. NULL = current; a fresh ceremony (SaveHostEscrow) clears it. // (SaveHostEscrow) clears it.
// ⚠ NOTHING SETS IT as of v0.95.0 (R-196 / R-204 item 2). The only writer was the PRECAUTIONARY
// mark on offsite re-issue, which guessed rather than measured and so blocked off-site backups on
// boxes whose key had not changed. The column and its readers stay; see MarkEscrowStale for what a
// legitimate future writer would have to prove first.
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN stale_at DATETIME`) s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN stale_at DATETIME`)
// dr_recipe (SPIKE-dr-recipe-2026-06-16): the secret-free DR reconstruction recipe, stored // dr_recipe (SPIKE-dr-recipe-2026-06-16): the secret-free DR reconstruction recipe, stored
@@ -2262,6 +2291,28 @@ func (s *Store) GetHostByCustomer(customerID string) (*Host, error) {
return h, err return h, err
} }
// HasEverBoundHost reports whether a machine was EVER bound to this customer — a live row in
// `hosts` OR a tombstone in `host_deletions`. It answers "was anything ever expected of this
// customer", which is the question the deadline verdicts actually need (R-195).
//
// It is deliberately NOT "has a report arrived", and the distinction is the whole point: a box
// that was installed, bound, and then went silent IS bound, and its silence is a real fault that
// must keep alarming. Only a customer that never had a machine at all is UNKNOWN.
//
// `host_deletions` is included because a customer whose host was removed HAD one — the deadline
// caller reaches its down-skip for that shape, and this predicate must not quietly take over a
// judgement the staleness checker owns.
func (s *Store) HasEverBoundHost(customerID string) (bool, error) {
var n int
if err := s.db.QueryRow(
`SELECT EXISTS(SELECT 1 FROM hosts WHERE customer_id = ?)
OR EXISTS(SELECT 1 FROM host_deletions WHERE customer_id = ?)`,
customerID, customerID).Scan(&n); err != nil {
return false, err
}
return n != 0, nil
}
// ListHostsByCustomer returns the customer's hosts ordered by host_id (v0.47.0 — the // ListHostsByCustomer returns the customer's hosts ordered by host_id (v0.47.0 — the
// customer page's Host tab is a LIST by design: 1 host today, N for a later HA cluster). // customer page's Host tab is a LIST by design: 1 host today, N for a later HA cluster).
// Uses the idx_hosts_customer index. // Uses the idx_hosts_customer index.
@@ -2507,6 +2558,11 @@ type HostEscrow struct {
// StaleAt (v0.57.0, 2.3) — non-empty when the offsite password was re-issued after this blob was // StaleAt (v0.57.0, 2.3) — non-empty when the offsite password was re-issued after this blob was
// sealed: the blob is stale (seals a password that no longer opens the repo). Cleared by a fresh ceremony. // sealed: the blob is stale (seals a password that no longer opens the repo). Cleared by a fresh ceremony.
StaleAt string StaleAt string
// IdentityBlob (R-198, v0.93.0) — the age-wrapped identity bundle, which is where the offsite
// restic REPOSITORY password lives. Populated by ListSupersededEscrow so a retained blob is
// reachable from Go at all; nil for pre-v0.93.0 retained rows and for hosts that never uploaded
// one. Opaque: useless without the customer's recovery code, which the hub never holds.
IdentityBlob []byte
} }
// SaveHostEscrow stores (last-write-wins) the OPAQUE escrow blob for a host. The hub keeps the // SaveHostEscrow stores (last-write-wins) the OPAQUE escrow blob for a host. The hub keeps the
@@ -2516,16 +2572,35 @@ type HostEscrow struct {
// host_escrow_superseded before overwriting the current row (Part B, v0.60.0). A same-sha re-upload // host_escrow_superseded before overwriting the current row (Part B, v0.60.0). A same-sha re-upload
// (idempotent re-ceremony of the same password) refreshes the current row and does NOT create a // (idempotent re-ceremony of the same password) refreshes the current row and does NOT create a
// superseded row. // superseded row.
//
// R-197 (v0.93.0): it also returns prevResticPwSHA256 — the hash the row being replaced sealed ("" when
// no row existed, or when it was a legacy hash-less blob). Both halves of "did the box's offsite DATA
// key change?" have been in this database since SLICE 3 and NOTHING compared them; demo-felhom's key
// changed on 2026-08-03 and no signal of any kind fired for thirteen hours. The comparison is the
// caller's (api.handleHostEscrowPut) because the event needs the customer id; returning the value is
// this function's part. The VALUE is a non-reversible hash of a 256-bit random secret and is never
// logged, mailed or written to a report — see the caller.
// demoteCurrentEscrowTx copies the host's CURRENT host_escrow row (if any) into // demoteCurrentEscrowTx copies the host's CURRENT host_escrow row (if any) into
// host_escrow_superseded as a retained blob, inside the given tx. This is THE ONE escrow row-copy // host_escrow_superseded as a retained blob, inside the given tx. This is THE ONE escrow row-copy
// routine (v0.60.0): SaveHostEscrow uses it to retain a superseded different-passphrase blob before // routine (v0.60.0): SaveHostEscrow uses it to retain a superseded different-passphrase blob before
// overwriting, and DeleteHost (v0.60.1) uses it to DEMOTE the current blob to retained custody // overwriting, and DeleteHost (v0.60.1) uses it to DEMOTE the current blob to retained custody
// instead of destroying it. Returns the number of rows copied (0 when the host has no current row). // instead of destroying it. Returns the number of rows copied (0 when the host has no current row).
// The hub never decrypts; custody is unchanged. // The hub never decrypts; custody is unchanged.
//
// R-198 (v0.93.0): `identity_blob` is copied too. It was omitted from this SELECT for two months, so
// every supersession retained the PBS datastore key and destroyed the offsite restic repository
// password — the one secret the retention exists to preserve. Copying more opaque bytes gains the hub
// NO knowledge: it still has no recovery code and no decrypt path.
//
// ORDERING THIS DEPENDS ON, stated because it is load-bearing and invisible from here: the identity
// blob is written by SaveHostDRBundle AFTER SaveHostEscrow returns (api/handler.go, the escrow PUT),
// so at demote time host_escrow still holds the OLD identity blob. If that order ever changes, this
// routine silently retains the NEW blob under the OLD blob's hash — pinned by
// TestSaveHostEscrow_RetainsIdentityBlob, which asserts the retained bytes are the previous ones.
func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) { func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
res, err := tx.Exec(` res, err := tx.Exec(`
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at) INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob)
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now') SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now'), identity_blob
FROM host_escrow WHERE host_id = ?`, hostID) FROM host_escrow WHERE host_id = ?`, hostID)
if err != nil { if err != nil {
return 0, err return 0, err
@@ -2533,10 +2608,10 @@ func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
return res.RowsAffected() return res.RowsAffected()
} }
func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, err error) { func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, prevResticPwSHA256 string, err error) {
tx, err := s.db.Begin() tx, err := s.db.Begin()
if err != nil { if err != nil {
return false, err return false, "", err
} }
defer func() { defer func() {
if err != nil { if err != nil {
@@ -2555,11 +2630,14 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
exists = false exists = false
default: default:
err = scanErr err = scanErr
return false, err return false, "", err
}
if exists {
prevResticPwSHA256 = curSHA // R-197: the caller compares; "" = no row or a legacy hash-less blob
} }
if exists && curSHA != resticPwSHA256 { if exists && curSHA != resticPwSHA256 {
if _, err = demoteCurrentEscrowTx(tx, hostID); err != nil { if _, err = demoteCurrentEscrowTx(tx, hostID); err != nil {
return false, err return false, prevResticPwSHA256, err
} }
superseded = true superseded = true
} }
@@ -2576,10 +2654,10 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
stale_at = NULL, stale_at = NULL,
updated_at = datetime('now')`, updated_at = datetime('now')`,
hostID, blob, keyFingerprint, posture, createdAt, resticPwSHA256); err != nil { hostID, blob, keyFingerprint, posture, createdAt, resticPwSHA256); err != nil {
return false, err return false, prevResticPwSHA256, err
} }
err = tx.Commit() err = tx.Commit()
return superseded, err return superseded, prevResticPwSHA256, err
} }
// CountSupersededEscrow returns how many retained (superseded) escrow blobs the hub holds for a host // CountSupersededEscrow returns how many retained (superseded) escrow blobs the hub holds for a host
@@ -2594,7 +2672,7 @@ func (s *Store) CountSupersededEscrow(hostID string) (int, error) {
// first. Opaque bytes — the hub never decrypts. Seeds the future guided-recovery flow (R-26). // first. Opaque bytes — the hub never decrypts. Seeds the future guided-recovery flow (R-26).
func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) { func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
rows, err := s.db.Query(` rows, err := s.db.Query(`
SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob
FROM host_escrow_superseded WHERE host_id = ? ORDER BY id DESC`, hostID) FROM host_escrow_superseded WHERE host_id = ? ORDER BY id DESC`, hostID)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -2603,7 +2681,7 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
var out []HostEscrow var out []HostEscrow
for rows.Next() { for rows.Next() {
var e HostEscrow var e HostEscrow
if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt); err != nil { if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt, &e.IdentityBlob); err != nil {
return nil, err return nil, err
} }
out = append(out, e) out = append(out, e)
@@ -2611,9 +2689,33 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
return out, rows.Err() return out, rows.Err()
} }
// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3) — called when the offsite repo // CountCurrentEscrowWithIdentity returns how many hosts hold a CURRENT escrow row, and how many of
// password is re-issued, because the blob then seals a password that no longer opens the repo. No-op // those carry an identity blob — i.e. the population whose offsite repository password R-198's fix
// when no escrow row exists; idempotent (only stamps the first re-issue since the last ceremony; a // now protects from the next ceremony, and the remainder for whom there is nothing to protect
// because no identity blob was ever uploaded. Read-only; no blob or hash leaves this call.
func (s *Store) CountCurrentEscrowWithIdentity() (total, withIdentity int, err error) {
err = s.db.QueryRow(`
SELECT COUNT(*), COALESCE(SUM(identity_blob IS NOT NULL), 0) FROM host_escrow`).Scan(&total, &withIdentity)
return total, withIdentity, err
}
// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3).
//
// ⚠ IT HAS NO CALLER as of hub v0.95.0 (R-196 / R-204 item 2), and that is deliberate, not an
// oversight. Its ONE caller was `offsite.ReissueCredentials`, which called it on every re-issue on
// the PRECAUTIONARY grounds that the box's re-apply might mint a fresh repository password. It
// usually does not, so the call marked healthy escrows stale — and because a stale flag WITHHOLDS
// restic_pw_sha256 from the ACK (GetEscrowStatusForCustomer, below), it blocked every off-site backup
// on those boxes and asked the customer for a ceremony that would supersede a perfectly good key. The
// full reasoning, and the two measured signals that cover the real case, are at
// offsite.ReissueCredentials.
//
// KEPT, not deleted, because the FLAG is still live and correct — `stale_at` is read by the ACK, the
// operator card and the PBS-DR view, and a future EVIDENTIAL caller (one that has measured a key
// change rather than guessed at one) is the right way to set it. Pinned by
// TestReissue_DoesNotMarkAHealthyEscrowStale: if a caller reappears without that evidence, it fails.
//
// No-op when no escrow row exists; idempotent (only stamps the first mark since the last ceremony; a
// fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT). // fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT).
func (s *Store) MarkEscrowStale(hostID string) error { func (s *Store) MarkEscrowStale(hostID string) error {
_, err := s.db.Exec(`UPDATE host_escrow SET stale_at = datetime('now') WHERE host_id = ? AND stale_at IS NULL`, hostID) _, err := s.db.Exec(`UPDATE host_escrow SET stale_at = datetime('now') WHERE host_id = ? AND stale_at IS NULL`, hostID)
@@ -2643,9 +2745,13 @@ type EscrowStatus struct {
IdentityBlobPresent bool `json:"identity_blob_present"` IdentityBlobPresent bool `json:"identity_blob_present"`
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"` ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
CreatedAt string `json:"created_at,omitempty"` CreatedAt string `json:"created_at,omitempty"`
// Stale (v0.57.0, 2.3) — true when the offsite password was re-issued after the blob was sealed. // Stale (v0.57.0, 2.3) — true when `stale_at` is stamped on the host's escrow row. ⚠ SINCE hub
// v0.95.0 (R-196 / R-204 item 2) NOTHING STAMPS IT: the one caller was the precautionary
// re-issue mark, and it was removed for marking healthy escrows stale. See MarkEscrowStale.
// When stale the ResticPwSHA256 is WITHHELD (emptied) so the controller cannot auto-confirm against // When stale the ResticPwSHA256 is WITHHELD (emptied) so the controller cannot auto-confirm against
// a hash that no longer matches the live repo password — the ceremony must run again. // a hash that may no longer match the live repo password — the ceremony must run again. That
// withholding is exactly why the precautionary caller had to go: it BLINDED the controller's own
// hash comparison, which is the measurement that actually detects a changed repository password.
Stale bool `json:"escrow_stale,omitempty"` Stale bool `json:"escrow_stale,omitempty"`
} }
+2 -2
View File
@@ -74,11 +74,11 @@ func seedDeletable(t *testing.T, st *store.Store, customerID string) string {
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi-" + hostID, LastReportAt: &long}); err != nil { if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi-" + hostID, LastReportAt: &long}); err != nil {
t.Fatalf("seed host: %v", err) t.Fatalf("seed host: %v", err)
} }
if _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil { if _, _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
t.Fatalf("seed escrow A: %v", err) t.Fatalf("seed escrow A: %v", err)
} }
// A second save supersedes A → one RETAINED blob + one CURRENT blob before the cascade runs. // A second save supersedes A → one RETAINED blob + one CURRENT blob before the cascade runs.
if _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil { if _, _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
t.Fatalf("seed escrow B: %v", err) t.Fatalf("seed escrow B: %v", err)
} }
if err := st.SaveOneTimeSecret(customerID, "one-time-pw"); err != nil { if err := st.SaveOneTimeSecret(customerID, "one-time-pw"); err != nil {
+2 -2
View File
@@ -37,10 +37,10 @@ func seedResettable(t *testing.T, st *store.Store, customerID string) {
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi"}); err != nil { if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi"}); err != nil {
t.Fatalf("seed host: %v", err) t.Fatalf("seed host: %v", err)
} }
if _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil { if _, _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
t.Fatalf("seed escrow A: %v", err) t.Fatalf("seed escrow A: %v", err)
} }
if _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil { if _, _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
t.Fatalf("seed escrow B: %v", err) t.Fatalf("seed escrow B: %v", err)
} }
if err := st.DeleteHost(hostID, true); err != nil { // demotes current → retained; records host_deletions if err := st.DeleteHost(hostID, true); err != nil { // demotes current → retained; records host_deletions
+3 -3
View File
@@ -64,7 +64,7 @@ func TestHostDelete_EscrowAckRequired(t *testing.T) {
if err := st.UpsertHost(&store.Host{HostID: "esc-host", CustomerID: "c2", APIKey: "k"}); err != nil { if err := st.UpsertHost(&store.Host{HostID: "esc-host", CustomerID: "c2", APIKey: "k"}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, err := st.SaveHostEscrow("esc-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil { if _, _, err := st.SaveHostEscrow("esc-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -110,7 +110,7 @@ func TestHostDelete_ImpactJSON(t *testing.T) {
if err := st.UpsertHost(&store.Host{HostID: "imp-host", CustomerID: "c4", APIKey: "SECRET-KEY"}); err != nil { if err := st.UpsertHost(&store.Host{HostID: "imp-host", CustomerID: "c4", APIKey: "SECRET-KEY"}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, err := st.SaveHostEscrow("imp-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil { if _, _, err := st.SaveHostEscrow("imp-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := st.UpsertGuestFromReport(&store.Guest{GuestID: store.GuestID("imp-host", 100), if err := st.UpsertGuestFromReport(&store.Guest{GuestID: store.GuestID("imp-host", 100),
@@ -159,7 +159,7 @@ func TestHostDelete_HappyPath(t *testing.T) {
if err := st.UpsertHost(&store.Host{HostID: "dr-drill", CustomerID: "c5", APIKey: "k"}); err != nil { if err := st.UpsertHost(&store.Host{HostID: "dr-drill", CustomerID: "c5", APIKey: "k"}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, err := st.SaveHostEscrow("dr-drill", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil { if _, _, err := st.SaveHostEscrow("dr-drill", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
t.Fatal(err) t.Fatal(err)
} }
rr := postHostDelete(t, s, "dr-drill", url.Values{ rr := postHostDelete(t, s, "dr-drill", url.Values{
+1 -1
View File
@@ -130,7 +130,7 @@ func TestHandleHostDetail(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
// DR + escrow present (escrow row must exist before the DR bundle UPDATE). // DR + escrow present (escrow row must exist before the DR bundle UPDATE).
if _, err := st.SaveHostEscrow("demo-felhom-01", []byte("opaque-escrow"), "fp", "posture", "2026-06-01T00:00:00Z", ""); err != nil { if _, _, err := st.SaveHostEscrow("demo-felhom-01", []byte("opaque-escrow"), "fp", "posture", "2026-06-01T00:00:00Z", ""); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := st.SaveHostDRBundle("demo-felhom-01", []byte("opaque-identity"), `{"v":1}`); err != nil { if err := st.SaveHostDRBundle("demo-felhom-01", []byte("opaque-identity"), `{"v":1}`); err != nil {
+1 -1
View File
@@ -404,7 +404,7 @@ func TestPBSDR_F14AutoReissueOnAckedDeletion(t *testing.T) {
if err := st.UpsertHost(&store.Host{HostID: "peti-00-dead", CustomerID: "peti", APIKey: "oldkey"}); err != nil { if err := st.UpsertHost(&store.Host{HostID: "peti-00-dead", CustomerID: "peti", APIKey: "oldkey"}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, err := st.SaveHostEscrow("peti-00-dead", []byte("opaque"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil { if _, _, err := st.SaveHostEscrow("peti-00-dead", []byte("opaque"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := st.DeleteHost("peti-00-dead", true); err != nil { if err := st.DeleteHost("peti-00-dead", true); err != nil {
+1 -1
View File
@@ -125,7 +125,7 @@ spec:
spec: spec:
containers: containers:
- name: hub - name: hub
image: gitea.dooplex.hu/admin/felhom-hub:0.91.1 image: gitea.dooplex.hu/admin/felhom-hub:0.95.0
ports: ports:
- containerPort: 8080 - containerPort: 8080
name: http name: http
+2 -2
View File
@@ -324,7 +324,7 @@ spec:
image: registry.k8s.io/git-sync/git-sync:v4.4.0 image: registry.k8s.io/git-sync/git-sync:v4.4.0
args: args:
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git - --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
- --ref=installer-v1.23.0 - --ref=installer-v1.25.0
- --root=/git-scripts - --root=/git-scripts
- --link=current - --link=current
- --period=30s - --period=30s
@@ -369,7 +369,7 @@ spec:
image: registry.k8s.io/git-sync/git-sync:v4.4.0 image: registry.k8s.io/git-sync/git-sync:v4.4.0
args: args:
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git - --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
- --ref=installer-v1.23.0 - --ref=installer-v1.25.0
- --root=/git-scripts - --root=/git-scripts
- --link=current - --link=current
- --one-time - --one-time
+23
View File
@@ -1,3 +1,26 @@
## 1.25.0 — the off-site tier stops asking to prune (2026-08-04, R-191)
**A backup that worked must not report failure.** The off-site tier was written with `keep_last: 2`,
so every weekly run uploaded its snapshot successfully and then failed the whole job on a prune the
box's token is deliberately refused: `prune 'ct/9201': permission check failed - missing
Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the
operator's inbox. Every week, on both boxes, about a backup that had already succeeded.
**R-89 moved off-site pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay
write-only, so a box can never delete its own off-site history. The 2026-07-26 "two weeks" ruling was
not reversed; where it is ENFORCED moved, and this value did not follow. The tier now writes
`keep_last: 0`, which the agent's existing guard (`allowPBSPrune = !primary && keep_last > 0`) already
reads as "never prune from the box" — no agent change needed.
**VERIFIED BEFORE CHANGING IT** (read-only on ep0, 2026-08-04): prune jobs `prune-demo-felhom` and
`prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
and have run **every day since 2026-07-27 — 18 tasks, all `status=OK`**, the newest showing
`retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Disabling
the client-side prune without that check would have traded a weekly false alarm for unbounded growth.
A gate now asserts the off-site tier carries no client-side prune, so the value cannot drift back
quietly. The local tier's retention is untouched.
## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185) ## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
**`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the **`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
+26 -4
View File
@@ -184,7 +184,7 @@
set -euo pipefail set -euo pipefail
SCRIPT_VERSION="1.24.0" # the SINGLE version source (F-1): -h and the run banners follow it. SCRIPT_VERSION="1.25.0" # the SINGLE version source (F-1): -h and the run banners follow it.
# The hub used to carry a copy for its Setup tab; R-94 DELETED it # The hub used to carry a copy for its Setup tab; R-94 DELETED it
# (2026-08-02) because the hub cannot know which version a box runs — # (2026-08-02) because the hub cannot know which version a box runs —
# the Setup command fetches this script at run time. scripts/ # the Setup command fetches this script at run time. scripts/
@@ -2479,8 +2479,30 @@ for _k,_v in {"unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agen
base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20}) base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20})
# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions — # R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions —
# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob # keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob
# would guarantee that one of them is wrong. keep_last=2 on the offsite tier = two weeks (operator # would guarantee that one of them is wrong.
# ruling 2026-07-26). #
# THE OFFSITE TIER CARRIES NO CLIENT-SIDE RETENTION, AND THAT IS THE POINT (R-191, 2026-08-04).
# It used to be written `keep_last: 2` ("two weeks", operator ruling 2026-07-26). **R-89 then moved
# offsite pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay write-only,
# deliberately, so that a box can never delete its own offsite history. The 2026-07-26 ruling was not
# reversed; where it is ENFORCED moved, and this value did not follow.
#
# The consequence was weekly and wrong in the worst direction: vzdump UPLOADED the snapshot fine and
# then failed the whole job on the prune the token is refused —
# `prune 'ct/9201': permission check failed - missing Datastore.Modify|Datastore.Prune`
# → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the operator's inbox. Every week, on
# both boxes, about a backup that had already succeeded. A tier that cries wolf weekly is a tier
# whose real failure nobody will see.
#
# VERIFIED BEFORE CHANGING IT (2026-08-04, read-only on ep0): prune jobs `prune-demo-felhom` and
# `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
# and have run EVERY DAY since 2026-07-27 — 18 tasks, all `status=OK`, the newest showing
# `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Retention
# happens; it happens THERE. **If that ever stops being true, this zero is unbounded growth** — check
# ep0's prune jobs before assuming the offsite tier is retained.
#
# `keep_last: 0` means "never prune from the box" and is the value the agent's own guard reads
# (allowPBSPrune = !primary && keep_last > 0), so no agent change is needed to honour it.
# #
# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only # The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only
# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent # when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent
@@ -2491,7 +2513,7 @@ base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_sec
# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here — # setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here —
# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start # they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start
# writing to an offsite datastore. # writing to an offsite datastore.
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]}) base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":0}]})
base.setdefault('local_api', {}) base.setdefault('local_api', {})
base['local_api'].setdefault('enable', True) base['local_api'].setdefault('enable', True)
base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR'] base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']
+26
View File
@@ -233,6 +233,32 @@ else:
"invisible to the agent, it is never restore-tested, and an empty listing looks exactly " "invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
"like a brand-new tier." % (resolutions, grants)) "like a brand-new tier." % (resolutions, grants))
# ── R-191: the OFFSITE tier must not arm a client-side prune ─────────────────────────────────
#
# R-89 moved offsite pruning SERVER-SIDE — ep0 runs a per-namespace prune job and box tokens stay
# write-only, so the box is REFUSED if it asks. When this default was `keep_last: 2` the effect was a
# weekly lie: vzdump uploaded the snapshot, then failed the whole job on the prune, and the operator
# was told the offsite backup had failed when it had succeeded.
#
# The assertion is on the OFFSITE entry only. The local tier's `local_backup_retention` is untouched
# and must stay untouched — it prunes correctly and is allowed to.
m = re.search(r'"backup_targets":\s*\[(.*?)\]', src, re.S)
if not m:
fail("cannot find backup_targets in the rendered agent.json defaults — the offsite-retention "
"assertion cannot run, and a check that cannot run must never report OK (R-191)")
else:
targets = m.group(1)
kl = re.search(r'"keep_last"\s*:\s*(\d+)', targets)
if not kl:
fail("the offsite backup_target carries no keep_last at all — expected an explicit 0 "
"(R-191: 0 means 'never prune from the box'; absent is not the same statement)")
elif kl.group(1) != "0":
fail("the offsite backup_target arms a CLIENT-SIDE prune (keep_last=%s). R-89 moved offsite "
"pruning server-side to ep0 and box tokens are write-only, so every weekly run will "
"upload successfully and then FAIL the job on a refused prune (R-191)." % kl.group(1))
else:
ok("the offsite tier arms no client-side prune (keep_last=0; retention is ep0's prune jobs)")
print() print()
if fails: if fails:
print("hostinstall gates: %d FAILURE(S)" % len(fails)) print("hostinstall gates: %d FAILURE(S)" % len(fails))