R-181 CLOSED (controller v0.193.0 + v0.193.1) and proven live on demo-hp for
BOTH reserve terms. The reserve is now a per-app, per-run ADMISSION decision
taken before the app's first write and covering all three write legs, and it
gained a size term. The refusal's wording was not weakened; the behaviour moved
so it became true, verified by sha256 tree fingerprint.
R-156 CLOSED — papra's template mounts the app's own data root. Precondition
re-measured rather than inherited (both boxes were wiped today).
Part 4, documentation only, nothing built:
- R-110 WAITING-ON-OPERATOR -> READY. Ruling: option (b), the installer's
publish channel moves to a TAG. Recorded with the condition that decides
whether it works at all — it must cover BOTH the /scripts/ git-sync and the
nine files the installer fetches from raw/branch/main.
- R-115 WAITING-ON-OPERATOR -> READY. Ruling: mechanism (b), a build-side gate
refusing to deploy or vouch an unpublished version. The third instance (agent
v0.120.0) would have silently downgraded both demo boxes while succeeding.
R-182 NEW: the periodic status refresh has no admission scope, so a refused app
re-alerts on every poll (measured: a second alert pair 13s after the run's).
Pre-existing in v0.192.0; deliberately not fixed in the R-181 task.
capability map: the local-backup row moves to PROVEN-LIVE in BOTH halves.
ROADMAP: R-165 collapses to CLOSED; R-181 collapsed into it.
07-backup-architecture.md: the reserve's contract stated as what the code
provides (S-1 — an architectural contract changed in the same session).
STATUS.md trimmed 150 -> 111 lines, "What's broken" no longer holds shipped
work, and the stale "After:" line (pointing at work that shipped on 2 August)
is fixed.
R-167/R-158 shipped and proven live (controller v0.191.x, hub v0.89.0):
two new capability-map rows PROVEN-LIVE with live citations, and
07-backup-architecture.md §7.5's closing claim "nothing warns when an app
crosses the line" is now false and rewritten (S-1: an architectural
contract changed in the same session). §7.5 also gains the caveat that its
size bound is ONE BOX'S, not the fleet's.
Part 3 SPIKE (audits/SPIKE-r165-mp1-merge-2026-08-02.md): M1-M5 measured,
NO layout touched. Three findings the merge session must not re-derive:
"the layout" is not one thing (200G/50G vs 50G/20G vs 16G/8G); mp1 is a
BULKHEAD and not only a ceiling, so after the merge an overflow reaches
/var/lib/docker; the golden fails closed on the split in four places.
D-a's condition (1) is currently SATISFIED — no external box is in the
hub's register, and both demo boxes are Tier 0 and reinstallable.
Recommendation given, choice NOT made — it ends at the operator's ruling.
CONTEXT.md S-11 (D-c's routing, and why R-158's own backup_failed proposal
was overruled) and S-12 (the monitoring landed BEFORE the merge).
STATUS.md gains the plain-language section and the merge decision, with two
older entries trimmed so the page did not grow.
New rows R-174 (closed same session), R-175, R-176, R-177; each ID grepped
free before minting.
Controller v0.190.0. Docs only here; no hub change, no hub version bump.
- audits/DIAG-bootrecon-drive-absent-2026-08-02.md — NEW. The Part 0 diagnosis,
including the run that produced a FALSE NEGATIVE and the mechanism behind it
(the agent re-binds an unmounted drive within ~60s, so the drive gate's startup
reconcile restarted the apps one second before the sweep looked). Records that
the write hazard was blocked only by an ACCIDENTAL filesystem permission that no
code owns and no test pins.
- architecture/02 §0a — the boot-recovery contract (S-1): both gates read desired
state; the sweep observes a SETTLED fleet and each sample must refresh first;
nothing is started without asking, fail-safe. Plus the durable warning:
Manager.StartStack has no gate of its own.
- 00-capability-map — the boot-recovery row, with the repeat count cited per N.5
(6 of 6 hard resets) rather than a bare PROVEN-LIVE.
- OPEN-ITEMS / ROADMAP — R-157 CLOSED (both mechanisms), R-170 CLOSED, R-171 NEW
and closed the same session, marked a regression from v0.189.0.
- STATUS.md — the power-cut line moved from "What's broken" to "What works right
now" with its repeat count; one dated bullet in the change log.
- CONTEXT.md S-13 — the lessons worth carrying: "it didn't happen this time" is
not a disproof; widening a window makes previously-unreachable overlaps
reachable; and a settle detector is only as good as the freshness of what it
samples — the fix's own defect, found live rather than by review.
Controller v0.189.0 implements operator decision D-b. Docs only here; no hub
change and no hub version bump.
- architecture/02-controller-module-map.md §0a — NEW, and it is the S-1 contract:
desired (app.yaml) / in-flight (own marker file) / observed (not persisted),
with the rule that ties them — never derive one from another. Absent desired
state means UNKNOWN, never "running". One file, one writer. D-b's binding
safety rule quoted verbatim.
- 00-capability-map.md — the boot-recovery row now rests on a recorded signal,
with the three live flows from 9201. The interrupted-operation half is marked
IMPLEMENTED, not PROVEN-LIVE: nobody killed the controller mid-backup on metal.
- OPEN-ITEMS/ROADMAP — R-166 SHIPPED with both blocking facts and their answers;
R-157 mechanism B CLOSED and A restated as the whole item; R-170 NEW (the
drive-backed boot gate still infers a Stop from a container count).
- STATUS.md — the "an app can stay switched off and nothing says so" line
rewritten to what is actually left: timing.
- CLAUDE.md — end-of-session checklist gains: confirm your own last push's CI run
went green, BY RUN ID. The failure email is a push signal; this is the pull check.
- CONTEXT.md S-12 — the rulings, and the two lessons worth carrying: a test that
constructs the thing it should prove the caller constructs is hollow (its
red-proof will say so), and a field-by-field struct rebuild in a save path is a
defect on sight.
SPIKE-ci-runner-2026-08-02.md: all six probes with method, measurement and ruling; none
STOPped. P2 (stock image has git but no python3) and P6 (a runner that loses its state
re-registers and orphans the old record) changed the design; P5 (a failed run signals
NOTHING) is why the alarm exists at all.
R-168 SHIPPED with its evidence. R-29 CLOSED — on the demonstrated alarm and not on a green
run, as required: the class it opened is answered at both ends, the hook refusing locally and
CI catching a --no-verify bypass and emailing. R-161 noted: its automatic half now exists for
the STATIC gate, while its original scope, the runtime gate, is deliberately still not
automatic and should stay that way.
NEW R-169 (grep established R-168 was the highest in use): CI can only report, because there
is no gate in the road. Making it blocking needs branch protection plus a PR workflow, both
of which change how the operator works — so it is theirs to decide, and the row states the
cost honestly rather than recommending it.
CONTEXT gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector
that tells no one is not finished, plus the curl and Cloudflare-1010 traps), S-10 (the runner
is unprivileged because DooPlex is Tier 2), S-11 (CI reproduces the sibling layout).
CLAUDE.md gains the rule earned by red-proofing: a go test -run pattern that matches no test
prints ok and exits 0, and an instrument that can silently drop results is not a measurement.
hub/CHANGELOG v0.87.0 + scripts/CHANGELOG gate-enforcement entry. CONTEXT gains S-6 (the
hub renders no host-install version and the gate pins its absence) and S-7 (gates run from
one entry point per repo; reuse_refs_check was fixed rather than the REUSE.md convention,
with both rejected alternatives recorded).
OPEN-ITEMS: R-94 CLOSED all three legs, leg (a) by DELETION with its reason; R-29 leg (a)
CLOSED and leg (b) HALF-SHIPPED with the census result written into the row (13 gates; every
gate a CLAUDE.md names was green, two of the four unnamed were red); R-161 gains its
successor pointer. NEW R-168 (grep established R-167 was the highest in use): Gitea Actions
runner — measured 2026-08-02 as Gitea 1.26.2, Actions enabled on all four repos, 0 runners,
0 workflow runs, 0 branch protections, and the consequence that trunk-based direct-to-main
pushes leave no merge for a status check to gate, so CI here can detect but not block.
BLOCKED on a spike over host-mode vs privileged DinD on DooPlex and whether the workflow can
avoid JavaScript actions.
ROADMAP: R-94 collapsed to its one-liner, R-29 updated, R-168 added.
The operator ruled on R-161 and the runner shipped in app-catalog-felhom.eu
(fd7747d), so the row moves from BLOCKED-needs-a-ruling to REDUCED SCOPE - open.
Both obvious enforcement points were rejected for measured reasons, and the row
now records them rather than leaving the rejection implicit. Controller-side at
template load: rejected because such a check can only read the file, and a static
audit of all 53 templates reports the catalog clean INCLUDING papra - it would
pass on the exact defect it exists to catch, the property being decidable only at
runtime. CI: rejected for now, neither repo has any and there are no users yet.
Shipped instead: scripts/catalog_gates.py, one entry point over all three gates,
non-zero exit on any failure, mandated in the catalog's CLAUDE.md the way
site_gates.py is. The rationale is recorded because it is the transferable part -
of this project's gates, the only ones that ever get run are those with a single
entry point named in a CLAUDE.md; site_gates.py is run and R-29's three orphans
are named nowhere and have stopped nothing.
What stays open is only the automatic half, which is sufficient while ONE person
touches templates - revisit when a second does.
Re-ranked accordingly: R-161 drops from 2nd to 7th, and R-156 is promoted to 2nd,
since R-161 was ranked high precisely because nothing ran the gate and that is no
longer true. The de-ranking is recorded inline with its reason, matching how R-94's
de-ranking is recorded, so a later reader sees a decision rather than drift.
Nine rows into OPEN-ITEMS.md and ROADMAP.md, matching each file's column shape.
R-156 and R-157 had lived only in audit documents - the identical "minted in a
spike doc and never carried across" failure the register already records for
R-153/R-154/R-155, caught by the catalog sweep's own section 8.0 while it was
happening. R-158 was minted by a second session the same day for an unrelated
finding, which is why the sweep's proposals were renumbered R-159..R-162 at filing
time. All nine IDs verified free in BOTH backlog files before use.
Part 0 settled the question the sizing item depended on, by reading:
mp1 is RETENTION, not staging, and neither of the two framings was right. A unit
is the KEPT copy on the app's OWN drive (backup.go:245-255); for an app with no
HDD_PATH the namespace falls back to the system SSD - "the SSD-only system-data
fallback" (appbackup/paths.go:26-27). There is no post-copy deletion: the only
prune is F5 residue-on-old-drives when an app MOVES (backup.go:1053-1112). So mp1
retains the units of driveless apps only - not every app, but not transient
either. Confirmed against the spike: sys_drive held exactly the four driveless
apps and not calibre-web, which had a drive and was still backed up.
A unit is volume tars + DB dumps only, never mp8 userdata
(recovery_unit.go:20-25), so a 1 TB photo library can never overflow one. And mp1
gates the WHOLE chain, not just Tier 1: Tier-2 mirrors the unit "(always)" from
RecoveryUnitPath (tier2.go:302,368) and Tier-3 carries it, so a unit that cannot
be written leaves both with nothing to copy.
Part 2 fired on both triggers - retention, and the fallback undocumented - so
07-backup-architecture.md gains section 7.5. Section 6.1 said a unit lives "on the
app's own drive", which is true and was the whole story only for drive-resident
apps; the no-drive case was undocumented, as was the sizing constraint. 7.5
records the mp0-50G-vs-mp1-20G mismatch, the measured ratios (DB app up to ~2x,
21.1GB -> 40.2GB; file-only 1.00x), and the bound this puts on D5's Lane-1
independence: restorable from the drive alone only while the unit still fits -
about 19 GB file-only, about 10 GB DB-backed. No number proposed; the ratio is the
operator's ruling (R-163).
R-159/R-160 marked SHIPPED only after verifying the template changes are in
app-catalog origin/main, and R-156's gate likewise (check-volume-persistence.py
present). papra is NOT fixed - referred - so R-156 stays open on that one app.
Ranked, with one line of reasoning each: R-157 first (an app can stay down
indefinitely with mechanism B silent on every channel), then R-161 (the gate
exists and nothing runs it, which is why R-156's class recurs - R-29's record is
three orphaned gates and one enforced), R-156, R-163, R-158, R-164, R-162.
PART 1 — the release gate.
G7 required the packaged .deb to sha256-match the one built from committed source. That is
unsatisfiable BY CONSTRUCTION: dpkg-deb stamps the build time into every archive, so two builds of
byte-identical source differ. It was already failing when the 1.26.1 release ran it. A criterion
nobody can satisfy gets waived once and read as advisory ever after — which is how R-29's shelf of
never-run gates was built. Sub-clause dropped, reason recorded in G7's own note the way G6's
amendment was, so a future reader can restore it if SOURCE_DATE_EPOCH ever makes it meaningful.
RULING ASKED FOR — is payload integrity covered by G9 alone? NO, and G9 is widened rather than a new
criterion invented. The package ships TWO payload files (build-deb.sh:54-55); G9 checked only the
script. The systemd UNIT was covered by nothing: G7 covered the container, G8 covers the postinst
behaviourally, G13 covers directory presence. The unit is not incidental — its After=, its
ConditionPathExists= and its Restart= decide WHEN AND WHETHER day-0 runs at all, so a drifted unit
would have shipped silently. Same shape as the /etc/felhom miss that G13 exists to prevent: a check
that proved the thing present and said nothing about what it depended on. The check passes today.
G13 moved to sit after G12 — it was minted late and left between G10 and G11.
PART 2 — register dispositions. BASELINE DISCREPANCY, reported rather than worked around: only R-128
had a row. R-154 and R-155 had NO row in either file — minted in a spike document and never carried
across, which is R-123's class, not the drift the task described. Rows created, closed, with the
reasoning, because in all three cases the reasoning is the durable part:
R-128 closed by CORRECTING a false claim, not by making the assertion real — the coupling does not
exist and asserting it would invent a constraint. Flagged so nobody 'restores' it.
R-154 closed with the measurement and where it now lives in pushed source.
R-155 NARROWED, not deleted — unchanged for FELHOM_MENU=single, inapplicable to release. Flagged so
the guard is not later removed wholesale on the strength of 'R-155 closed it'.
Documentation only: no code, no build, no ISO, no upload, no box touched.
The credential existed and was not reachable when it was wanted. Every box has
had a strong random root@pam password since TASK G1, vaulted in the hub at day 0
and used for real during the sshd incident — but the only way to read it back was
a hand-written curl carrying the global operator key, a secret kept out-of-band.
In practice the PVE web console on a demo box felt locked.
The host page grows a Console access card: presence + username + set_at by
default, Reveal fetches the plaintext on demand for 60 s with a Copy button.
Masking clears the JS variable, and also fires on a second click and on
visibilitychange. A host with nothing vaulted says so, and says why.
The secret is NEVER rendered into the page, and that constraint shapes the
change. The render path uses a new store.GetHostRecoveryMeta whose struct and
SELECT both omit the secret column, so it is structurally incapable of carrying
one. The plaintext crosses the wire only in the response to POST
/hosts/{id}/reveal-recovery-credential (Cache-Control: no-store, CSRF-gated at
the ServeHTTP level; POST precisely so that gate applies and so no secret is
retrievable by URL alone). Deliberately NOT the customer page's data-secret
widget, which embeds the plaintext on every load.
A delivered reveal writes one recovery_credential_revealed event on the host's
customer timeline (info, source hub, Hungarian) via SaveEvent alone — no
dispatcher, nobody emailed, the log_tail_requested shape. Two reveals write two
events: the register records accesses, not states. A 404 is not an access. An
unbound host reveals fine and writes no event; the [INFO] hub line, carrying the
username and a length only, is then the record.
The global-key API path is untouched by design — it is the route for when the
hub UI itself is broken, and coupling it to the session layer would delete the
independence that makes it a fallback.
Recorded as a real trade: the hub session password alone now unlocks console root
fleet-wide, where retrieval previously also needed the global key. Accepted for a
single-operator, HU-geo-fenced hub that already stores these passwords in
plaintext at rest (CONTEXT.md ruling S-4). The plaintext-at-rest half is filed as
R-133 — every hub DB backup is a fleet-wide console-credential dump.
Tests 550 -> 559; four red-proofs (page leak, audit event, CSRF gate, route
order) each run, observed failing, and reverted. The route-order proof is a seam
test driving ServeHTTP: a handler-level test cannot see that defect, because the
handler is correct and simply never runs.
Records controller v0.188.0 across the four coupled artifacts.
07-backup-architecture.md is the owning doc:
- new 7.4 = the recovery chain AFTER D5 (7.1 leg 1 superseded; leg 2,
the living-app dependency, explicitly unchanged so this is not read
as more than it is)
- 7.3 collapsed to history, with the correction that the target as
written (data_key-only) was tested in Part 0 and rejected
- 3 records that the two-lane split is now real, not just intended
- matrix rows 3 / 3c (new) / 13; 10.1 D5 itself shipped
Also: new capability-map row, D5 collapsed in ROADMAP + OPEN-ITEMS,
and R-127 filed in both (data_key flag unreliable; O4 can regenerate a
DB password that no longer matches the restored data directory).
The audit is named D5-drive-alone-restore rather than "...secrets..."
because .gitignore blocks *secret* -- a guard worth respecting, not
forcing past.
Four-artifact update per the coupling rule, plus the audit.
07-backup-architecture.md: §10.1 retitled CLOSED with the ruling and the D5
sentence; the FileBrowser network-share row flipped YES->NO, closed at the
PLACEMENT rather than at the bind; the exposure chain annotated with the fifth
surface (decommission-with-migrate guarded only its source) and the correction
that the boundary is the deploy POST, not the dropdown; §7.3 retitled UNBLOCKED;
register row collapsed; open question F answered.
00-capability-map.md: new §D row PROVEN-LIVE, with the un-exercised legs named —
the deploy-POST and decommission refusals are unit-tested, not live-fired.
OPEN-ITEMS.md: R-108 dispositioned; D5 given its OWN row as READY/UNBLOCKED (it
had existed only inside other rows' prose — the R-123 thread-loss pattern);
R-126 registered.
ROADMAP.md: R-108 collapsed to a shipped one-liner; R-126 added.
R-126 filed not fixed: a .fab bundle (plaintext secrets, optional password) can
be exported ONTO a NAS. Split out of R-108 rather than folded in — it is an
explicit customer-chosen export destination, not a browsing surface reaching a
backup tree, so it was never part of D5's precondition.
Live evidence: same-box before/after on demo-felhom through the real authenticated
endpoint, the network-specific refusal on demo-hp, non-effect verified in the
registry, and R-67's share-root bind diffed byte-identical across the deploy.
Adds the full audit: Part 0's three answers, the pre/post recipe for both boxes,
the on-disk proof that `local` froze at the 2026-07-28 target move while
felhom-backup kept running, all seven red-proofs, and the three publish
observables.
R-125 filed: v0.118.0's R-106 half shipped INERT. Two tests ran the real
Collector.Collect() but both injected a fakeObserver, and the break was one layer
below in mergeConfig, which dropped the pbs namespace. The recipe still said
"root" — now with namespace_state "resolved" beside it, confident and wrong.
Caught by live validation, not by the green suite. Fixed in 0.118.1; filed for
the doctrine point that a production-path claim must name the seam it injects at.
AssembleDRRecipe's hostHalfShape/appHalfShape are ALLOW-LISTS, not the
forward-compat their comment advertised: a section an emitter adds is silently
discarded until it is named in both the shape struct and AssembledRecipe. No
error, no log, no failing test.
R-122 (found this session): that already happened and shipped. The controller
has emitted offsite_restic since fork-4 — the offsite recovery LOCATION — the
hub stored it for all three real customers, and appHalfShape never listed the
key, so no delivered recipe has ever contained it. It stayed green because the
fixture drAppHalf is hand-written and omits the field.
R-109: the agent's new backup_target is a new top-level host-half section and
would have been dropped identically, making the fix read as shipped while
changing nothing an operator can see.
3 tests built on halves read verbatim out of the live dr_recipe table, plus
2 red-proofs (each mutation asserted to have landed). vet rc=0, suite rc=0, 17 ok.
Registers: R-106 + R-109 dispositioned; R-105/R-106 were READY in ROADMAP with
no OPEN-ITEMS row (→ R-123, registered); R-124 filed on the "root" spelling.
The golden's version IS the controller it bakes (build-golden.sh:345 defaults
GOLDEN_VERSION to the controller tag), so a golden behind the newest deployed
controller means every FRESH install lands on stale application code. On the R-120
occurrence that stale code shipped a customer-facing falsehood: a box from the
0.185.1 golden told a customer whose backup drive had fallen out that the backup was
on the same disk as the system -- false, the drive was gone -- and offered a
different drive as the remedy.
WHY A GATE, NOT A REMINDER. The gap has opened three times: R-111 (golden's agent 17
releases behind), R-115 (agent built and deployed, never published), R-120 (this).
The first two were closed by re-baking and remembering; remembering then failed
again. R-29 is the standing proof that a check nobody runs is worse than none because
it reads as coverage -- hostinstall_gates.py sat RED and uninvoked across three
version bumps and hub_confirm_gate.py has never run at all. So the property that
matters is not whether a check exists but whether it BLOCKS.
- Wired into handleSetArtifacts (internal/web/configs.go), immediately before the
only write, on the sole UI path to SetArtifactManifest -- it runs on every vouch
without anyone choosing to. A script in scripts/ would have been a fourth orphan.
- It REFUSES (operator ruling, 2026-07-30), with a flash naming the remedy.
- Signal: store.NewestReportedControllerVersion() over reports.controller_version,
SEMVER-compared in Go -- MAX() in SQL ranks 0.99.0 above 0.186.0, a pair this
fleet has shipped. No outbound call, no new credential.
- Fail-open in exactly two deliberate cases: an empty golden field (clearing the
manifest is legitimate) and an unknown fleet version (a new hub must vouch its
first golden).
NEAR-MISS RECORDED: the first draft read guests.controller_version, a column that
exists in the schema and that NOTHING writes -- it would always have seen "" and
failed open, i.e. inert, this gate's own failure shape. Caught by grepping for a
writer before trusting the column.
Blind spot stated rather than papered over: a controller no box has ever run is
invisible to this signal. Not the failure that has bitten -- all three instances were
deployed-newer-than-baked.
4 tests through the PRODUCTION handler over httptest, never an injected seam. The
refusal asserts both the flash and that the manifest was NOT written, because a gate
that redirects and saves anyway reads as enforcement while providing none. Red-proof:
deleting the block makes the stale golden vouchable and both assertions fail.
ROADMAP R-29's audit list now records this as the FIRST enforced gate, so the
contrast with its three orphans is kept rather than lost. The orphans are unchanged.
Suite rc=0 read separately from this commit.
Full ISO/PAIRING run on a fresh nested box. Agent 0.114.0 came from the Day-0
manifest -- the SHIPPED binary -- so C5 tested the real artifact. Controller
0.186.0 hand-deployed after install per the §3.1 ruling; the vouched golden
bakes 0.185.1, so C3/C4 prove the code not the shipped golden, and that lag is
filed against R-115 rather than a new ID.
R-113 PROVEN: detach 18:43:50, gate fired 18:43:54 -- four seconds, where E-2d
measured zero over 4.5 minutes -- and SetDisconnected was reached. It fired on
exactly the shape that defeated it: raw /mnt/mentes NOT mounted while the bind
/mnt/felhom-drives/mentes still read /dev/sdb[/felhom-data].
R-114 PROVEN: with the target absent the page rendered the absent copy, the
system-disk copy 0 and the offer block 0. Both of E-2d's falsehoods are gone.
R-112 PROVEN: the banner reached a customer's page for the first time. Healthy
renders nothing, proven POSITIVELY -- idle delta 0 /backup/tiers calls, page
load delta +1, single caller, so the seam ran and chose silence.
C5 FAILED on a fourth, separate defect. The alarm fires but as the GENERIC
storage_disconnected, while the recovery is the SPECIFIC backup_target_restored
-- a pair an operator cannot match, which is what notifyDriveReturned's own
comment forbids. backup_target_absent count 0 across the run. Root cause: the
drive is TWO /disks rows and BackupTarget and GuestPath sit on different ones;
absent they separate, on return they rejoin. v0.184.1 fixed the keying, not
this. Only reachable because R-113 made the gate fire at all. Filed as R-116.
Mirror + over-correction guard PASS: non-target drive -> storage_disconnected,
backup_target_absent 0; both drives present -> 0 ABSENT lines and the target
stayed healthy. Caveat recorded: the mirror passes trivially because the target
also produced the generic event.
E-2 and E-2d CLOSED as partially proven with R-116 the one named open leg, per
the runbook's §9 rule decided in advance rather than mid-run.
Capability map NOT touched: it has no E-2 rows at all, so nothing could move to
PROVEN-LIVE. Creating them is a design act, not a validation act.
Teardown clean: pvesm status after == before (local-lvm 38.78%), guest 9201 and
drill-r50 untouched. Customer delete attempted and correctly refused while the
host still reads ONLINE; command recorded for once it ages to DOWN.
PART 1 — Session C unblocked.
Agent 0.114.0 (the R-113 fix) was built, pushed and deployed but never
published, so a fresh drill box would have installed 0.113.0 and proven the bug
rather than the fix. Published from the clean tree at b58d7bc via
scripts/publish-agent.sh; sha 5e4c15ebee2d7583d57301d1f7c9cc7d4276262966bf738b05e34653bfd18c31,
verified by an INDEPENDENT round-trip GET (http=200, sha match, binary
self-reports 0.114.0), and the hub manifest read back after the write.
Deliberately NOT done, each with a reason:
- No golden bake. The golden bakes the CONTROLLER, not the agent, and
host-install fetches them as separate generic packages (:1945 / :2573). Golden
0.185.1 is current, so there is no new-agent-against-old-golden risk.
- min_agent NOT raised, stays 0.113.0. It expresses what the CONTROLLER requires
of the agent, and controller v0.185.0 declares MinAgent 0.113.0 — which
0.114.0 already satisfies. Raising it to 0.114.0 would have been a false claim
AND would have held demo-hp and drill-r50. No box is held; no §3 STOP fired.
- Global controller floor NOT raised (v0.156.0), per R-111's reasoning.
- wrapper_sha256 preserved verbatim; re-checked against configs/felhom-pbs-apply
before and after — no drift both times.
demo-hp RULING: left on 0.113.0. The R-113 fix is not live-validated, so putting
it on a second box widens exposure for no proof, and Session C's nested box
takes its agent from the manifest, not from demo-hp's host agent. Move the fleet
once, after Session C.
PART 2 — R-115 opened (WAITING-ON-OPERATOR).
The finding is the RECURRENCE, not either instance: publishing is a remembered
step, and it was forgotten within eight hours of R-111 documenting it as
forgettable. Filed as a new ID with a back-pointer rather than reopening R-111,
because R-111's finding (the channel WAS stale) is closed and verified
end-to-end, while the process defect that caused it is a distinct problem with a
distinct fix and owner. Class cross-linked to R-29 (a control that exists and is
never walked) WITHOUT minting a second ID for it. Options are stated as the
operator's decision, with mechanisms (build-step, deploy gate) separated from
reminders (checklist, manual) — R-29's whole finding being that reminders do not
hold. No code written, by design.
R-111 gains a deferred-leg-recurred line; its shipped evidence is untouched and
it is NOT reopened. R-113 records that Session C is now unblocked.
Full ISO/PAIRING route on a nested PVE VM on demo-hp, after R-111 was fixed
earlier in the session. Bind -> running controller in 3m35s. The install fetched
the artifacts published an hour before and restored the golden baked 20 minutes
before, so the publish train is proven end to end on a real install.
C1 PROVEN: "felhom-host-install v1.22.0", "Day-0 provision SUCCESS", guest 9201
running, bootstrap unit wrote its done-flag and self-disabled. This retires E-2's
"installer-logic-tested, not install-tested".
C2 PROVEN: both DEGRADED warning lines verbatim, backup.local_backup_target=local,
no felhom-backup storage created, and the install did not abort.
C3/C4 PARTIAL and C5 FAILED — three findings, none fixed:
R-112 (P1): E-2's degraded banner and offer have NO UI CONSUMER. The endpoint
returns byte-exact copy; grep 'backup-target' across every html/js/css is 0 hits
and no page handler injects the state. Templates fetch 18 distinct /api/storage/*
endpoints; these two are the only ones with zero references. v0.185.1 fixed the
router mount and stopped one layer short of the render. Fifth instance of
seam-built-but-never-wired.
R-113 (P1): the drive-absent gate CANNOT FIRE on device loss. planDriveGates
reads presence from BoundUnderParent = "is this path in the guest's mountinfo".
The raw mount is a device-bound systemd unit and dies with the device; the
agent's own bind is not device-bound and outlives it, so the gate sees "present"
forever. Live: agent reported the drive absent every 20s for 4.5 minutes, the
controller logged 0 [gate] lines, the hub received zero events -- neither
backup_target_absent nor the generic storage_disconnected. Sixth instance of the
class: E-2b wired the seam to a condition that cannot occur.
R-114: on target-drive loss the message claims the backup is on the system disk
(false) and offers the drive that just vanished. Invisible only because of R-112,
so it must be fixed BEFORE R-112 is wired.
Also filed as a second instance under R-110 rather than a new ID: host-install
fetches nine files from raw/branch/main and the hub vouches a sha for one;
E-2a's wrapper is installed 0755 to /usr/local/sbin, root-fenced in sudoers,
validated only by bash -n.
C4 is fully proven at API level: decline path (registration confers no role),
restart_required:true, agent did NOT self-restart (in-flight check performed and
recorded first), E-2a wrapper created the storage at the drive's own mountpoint,
and healthy renders nothing.
Teardown: VM destroyed, scratch storage removed, pvesm status after == before
(local-lvm 38.77%), guest 9201 and drill-r50 untouched. Hub records for e2d-fresh
remain -- delete correctly refused at four gates, finally "host is ONLINE";
deletable once it ages to DOWN. Command recorded in OPEN-ITEMS.md.
capability-map NOT touched: the customer-facing legs are broken rather than
proven, and the map has no E-2 rows at all.
No VM created, no install run, no box touched. The run stopped at the Phase 0
gate per runbook §3, before provisioning.
felhom-host-install.sh does not install what is on main. resolve_artifacts()
(:423-436) reads the hub-vouched manifest and fetches Gitea GENERIC PACKAGES
(agent :1945, golden :2573). Gitea's newest are agent 0.96.0 and golden 0.161.0;
the hub manifest selects exactly those; the global floor v0.156.0 is below the
golden's 0.161.0 so nothing self-updates. A fresh box therefore lands on agent
0.96.0 + controller 0.161.0 against main's 0.113.0 / 0.185.1. Agent 0.113.0
reached both demo boxes by direct deploy and is not in the channel at all.
Claim impact, each pinned to its introducing commit:
- C1 (real rc=0 1.22.0 install) and C2 (Case B natural) — ACHIEVABLE, not run;
both are installer-side and host-install is served at 1.22.0.
- C3 — BLOCKED: banner + GET /api/storage/backup-target are controller v0.185.1
(cdaeb36), copy v0.185.0 (3f7cf2a). Unblocks cheaply by raising the hub floor
to >=0.185.0; measured fleet impact nil (both demo boxes already 0.185.1).
- C4 — BLOCKED: needs controller v0.185.1 + agent v0.113.0 (58b598b).
- C5 — BLOCKED: needs controller v0.184.0 (c1a63de) + agent v0.112.0.
Filed R-111 (P1): 17 unpublished agent releases (v0.97.0-v0.113.0) strand the
entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT, so a new customer's
box installs without them. Mirror of R-110, not a duplicate.
- audits/E2D-fresh-vm-2026-07-29.md — all four Phase 0 answers recorded so a
resumed run does not re-derive them (cadence 30s; hot-detach available; ISO
present; local-lvm fence re-measured at 38.77%, unchanged).
- OPEN-ITEMS.md — R-111 opened; E-2d re-stated, NOT closed.
- ROADMAP.md — R-111 under P1.
- capability map NOT touched: nothing was proven live.
The §5.1a operator STOP is retired — HUB_PW is in ~/.config/credentials and hub
auth was verified, so CC can bind on a resumed run.
d4c07873 filed "hostinstall_gates.py is invoked by nothing" as a novel
observation. It is not novel — R-29 already names the class (green gates are
enforced nowhere; one sat RED for 16 releases while every REPORT said green),
and R-29 was missing from OPEN-ITEMS.md entirely, having never been carried
across the 2026-07-27 register rebuild. An open item about work not getting
done was absent from the page that decides what gets done.
Ruling on whether R-29 is the right home for a non-design-v2 gate: YES. Its
title says design-v2, but its own audit list already spans mount-safety,
secrets and dedup gates across four repos, and its part (b) — "the systemic
half is the real item" — is about the enforcement mechanism, which is
gate-agnostic. hub_confirm_gate.py is already on its list and sits in the same
scripts/ directory. No new ID minted; R-29's own text forbids it, and this is
the third re-raise it has absorbed.
- OPEN-ITEMS.md: open R-29 (READY, S(a)/M(b)), with the orphan evidence and
the two separable parts R-29 already defines.
- OPEN-ITEMS.md: R-94 leg (b) now points at R-29 as its class.
- ROADMAP.md:158: audit list extended with hostinstall_gates.py (RED today,
1.19.0 != 1.22.0) + hub_confirm_gate.py verified orphan. Entry not rewritten.
- ROADMAP.md:147: cited a non-existent R-164 — it means controller v0.164.0.
- CONTEXT.md:564: asserted in the present tense that the version cross-check is
"gated by scripts/hostinstall_gates.py". It exists, is red, and runs nowhere.
- OPEN-ITEMS.md: READY #1/#3/#4 markers dropped — they duplicated ranked-list
positions and the gap was left by the row merged in d4c07873.
- OPEN-ITEMS.md: E-2d citation :322-341 widened to :322-343; the invocation it
describes is at :343, two lines outside the old range.
- backlog/README.md: two-line lead naming OPEN-ITEMS.md and ROADMAP.md.
- REPORT-record-correction-2026-07-29.md: the report CLAUDE.md:82-87 requires
for both commits. Root REPORT.md (E-2 increment 1) untouched.
No gate wired, fixed, run or deleted — that is R-29 part (b), its own task.
Documentation only. No version bump, no CHANGELOG entry, no box touched.
The 2026-07-29 R-94/E-2d finding was written from an unverified claim and was
false. `felhom-bootstrap.sh:96` fetches the installer from the WEBSITE, not the
hub; the website git-syncs /scripts/ from main on a 30s period; every install
since 1.22.0 hit main this morning already runs 1.22.0. Confirmed by live fetch.
- OPEN-ITEMS.md: merge the two duplicate R-94 rows into one, retract the false
framing, re-scope to what it actually is (a drifting hand-synced constant plus
two pieces of dead safety equipment), unblock it from E-2d.
- OPEN-ITEMS.md: de-rank R-94 in the ranked list — the "high-consequence" reason
was the false claim in its most load-bearing form.
- OPEN-ITEMS.md: E-2d — the ISO is the STRONGER proof route, not an obstacle.
Phase 0 question answered at source: PAIRING falls through to run_direct in
the same invocation (:495-499), so it reaches the identical installer call.
- ROADMAP.md:149: same retraction; the original diagnosis (a hand-synced
constant in a second repo drifts every time the first ships) survives.
- ROADMAP.md + OPEN-ITEMS.md: new R-110 — main is the installer's publish
channel and there is no staging, tag, pinned path or rollback, for the one
artifact that runs as root on a virgin box. Operator ruling, not a defect.
- day0-install.md C.1: one sentence recording the same about the fetch URL.
Documentation only. No version bump, no CHANGELOG entry, no code, no box touched.
Supervised operational run. No code, no version bump, nothing deleted.
Primary backup tier on both demo boxes moved from `local` (a dir storage on
/var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir
storage on each box's secondary drive:
demo-hp /mnt/nvme-1tb uuid:91d2dc2d-... archive 2,256,044,492 B
demo-felhom /mnt/hdd_1 uuid:47a3361a-... archive 5,957,878,962 B
Both proven end to end via the real UI path: archive lands on the secondary
drive (df delta matches the archive byte-for-byte), restore-test auto-selects it
and passes with mount_parity: ok, and freshness survives an agent restart with
an empty in-memory store -- so the age can only have come from the new storage.
Phase 0: the target is CONFIGURATION, not converged (the sole writer of
agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so
the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path.
Findings:
- F-1 the storage path must BE the mountpoint; a subdirectory fails exactMount
and the target reports disconnected permanently (observe.go:321)
- F-2 --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a
non-mounted path reports active with the ROOT filesystem's free space and
had already created dump/ on pve-root -- a silent retarget onto the very
device this change escapes
- F-3 FelhomAgentStore is granted per storage path; without it every backup
403s. felhom-host-install.sh must issue it for new installs
- R-109 (new) the DR recipe records no backup target, and each box now carries
two content=backup dir storages, one live and one frozen
- R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the
enrolled drives were never PVE storages, so isUserDataDrive never saw
them. Both boxes now populate drives; SMART on the backup drives too
Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half
live-proven; agent half source-traced). That is NOT the intended fall-back-and-
alarm design -- filed as E-2 with the honest single-drive label.
Reported in full in the record: the agent was restarted with a felhom-pbs backup
in flight, producing a spurious tier failure. The backup had in fact succeeded
(PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no
channel -- R-84 ground truth superseded it.
Outstanding: full drive-loss recovery (needs physical access) and the agent half
of the absent-drive behaviour.
Replaces the 2026-07-14 DRAFT entirely - it was verified against controller v0.132.0,
51 versions stale, while being cited as authoritative. Now verified against controller
0.183.0 / agent 0.110.0 / hub 0.80.0 / catalog 4252121. NOT ratified.
Every statement is labelled [DESIGN] (a decision from the architecture discussion) or
[FACT] (file:line, live output, or a citation to _recovery-inventory-2026-07-28.md).
Phase 0 verified D5's premise and it FAILED. Ten read surfaces were checked; nine are
clean (SMB under both namespace shapes, FileBrowser for local drives, .fab import and
download, /api/debug/*, all three ServeFile sites, registering the backup dir as a
drive). One is not: FileBrowser binds a NETWORK share at its ROOT while local drives
are userdata-scoped, and no IsNetwork() filter guards the deploy dropdown, the per-app
migrate targets, or handleStorageMigrateApp. So an app namespace - and under D5 its
plaintext secrets - can land somewhere a browsing surface serves with download:true.
Not a leak today (the unit's app.yaml is secret-stripped); it becomes one under D5.
D5 is therefore written as a BLOCKED target, not as settled. -> R-108
Core artifact is the failure -> recovery matrix (15 rows). RTO cells carry only
measured durations; 12 are blank and each blank is listed explicitly as a finding.
RPO cells carry the configured cadence that bounds RPO, labelled as such - no RPO has
ever been measured from an incident and none is invented.
The 7/53 vs 9/43/1 Tier-2 coverage counts are both recorded, unresolved; neither adopted.
Coupling (S-1): capability map section C now cites the matrix instead of restating
routes; ROADMAP gains R-102..R-108; OPEN-ITEMS gains only the in-flight R-108 plus
C9-F1b/F3/F4 -> R-n cross-references; CONTEXT records D1-D6 as standing ruling S-3.
No code, no version bump, no deploy, no box touched.
R-97 collapsed to its shipped one-liner in ROADMAP and closed in OPEN-ITEMS.
PROMPT-TEMPLATE N.5 now names FOUR coupled artifacts instead of two: the
capability map, ROADMAP, the owning architecture doc (ruled as S-1 in CONTEXT.md
but never reflected in the template CC actually reads, so it bound nobody), and
OPEN-ITEMS.md. Tasks must now report which register rows they opened, closed or
re-ranked.
Ops: R-90 swap done (interim; CX33 still blocked), R-95 mitigation armed but zero
snapshots taken so it moves to WATCHING rather than closed, R-91 gate still not
satisfied. CONTEXT.md datastore path corrected to /mnt/pbs-datastore.
R-88 split: Part 1 (the failure breaker) SHIPPED in controller v0.176.0 and live
on both boxes; Part 2 (unknown != never) stays OPEN and is agent-side.
Phase 0 established the root cause at source: newestArchiveOn's (time.Time, bool)
signature cannot represent 'unknown', so a storage read ERROR collapses into a
positive 'no successful backup recorded yet'. The errored and genuine-never paths
are byte-identical on the wire, which is why Part 2 cannot be done controller-side.
R-97: the whole-guest backup tier has no failure signal to the hub at all —
internal/quiesce never imports internal/notify, so three failed backups and three
app-stack outages produced zero backup_failed events. Its only trace was a
customer-tier Hungarian app_start_failed for an app the backup itself had stopped.
Read-only triage found work that was agreed or discovered but never given an id:
R-95 restic offsite credential CAN delete — answers the parallel question R-89
raised and left open. Per-customer subaccounts report readonly=False, the
controller runs forget --prune from the box, and the sftp: backend cannot
express append-only. Storage Box snapshots (snapshot_limit=10, plan=null,
0 used) are server-side and SFTP cannot delete them — an unused zero-code
mitigation.
R-94 hub pins hostInstallVersion 1.19.0 while host-install ships 1.20.0, so a
hub-driven install still gets the pre-R-82 backup default.
R-90 ep0 has no swap at all and OOM'd today; gates R-86.
R-91 the pre-migration 13 GB datastore copy still occupies ep0's root disk.
R-92 PBS-DR gauge granularity. R-93 drill-r50 fixture tension.
R-96 two standing rules agreed in chat and never committed (the third, N.5's
third leg, IS committed at CONTEXT.md:8).
Supervised runbook execution. No code, no version bump.
The felhom-pbs tier had reported `job errors` on EVERY demo-hp backup
since the tier was created on 07-26, while the data landed correctly
every time: `DatastoreBackup` grants Datastore.Backup but not
Datastore.Prune, so the box's keep_last=2 prune was denied.
Operator ruling: retention is a COMMERCIAL attribute owned by the hub;
ep0 executes. Box tokens therefore stay write-only - a compromised box
must not be able to delete its own offsite backups. No grant was widened
and felhom-tenantsync.sh is unchanged (the ruling makes it correct).
Increment 1:
- boxes stop attempting prune. allowPBSPrune is DERIVED
(`!t.Primary && t.KeepLast > 0`), so keep_last: 0 on the PBS tier
disables both the --prune-backups value and the gate in one config
edit, and the tier stays armed. Verified prune_pbs_allowed=false on
both boxes with no tier REJECTED line.
- per-namespace prune jobs on ep0, keep-last 2, daily 03:30 UTC
(05:30 CEST), dry-run gated. demo-hp 3->2, demo-felhom untouched,
chunk count unchanged (prune removes indexes, not chunks).
Write proof CLOSED: 08:25:47 job errors -> 09:37:29 TASK OK, snapshot
2026-07-27T09:37:29Z, chunks 9787->9813, prune step absent entirely.
Driven through POST /api/guest-backup/trigger (the UI path), not
--selftest and not raw vzdump. Hub gauge evidence explicitly NOT
satisfied - the delta is below its 0.1 GB display granularity.
GC scheduled sun 04:30 UTC and deliberately NOT run: every chunk still
carries a fresh atime from the migration copy, so a run today would
reclaim nothing. verify-new enabled per operator ruling, turning an
inert hub alarm live.
Legacy demo-felhom-01 namespace deleted with its two ACL entries and its
token (operator ruling, confirmed twice) so nothing dangles.
R-89 records the target architecture and carries the unanswered parallel
question: does the restic key on storage-box-pool-1 have DELETE rights?
If so the daily app-data tier has the identical exposure and append-only
is the equivalent answer.
ep0 is Etc/UTC, not CEST - corrected in the record.
Corrects two wrong severity readings with evidence from the box and the code.
The PBS outage was ~15 min (07:00-07:18 UTC), caused by a global OOM at 06:58:12:
proxmox-backup-proxy peaked at 3.2G on a 3.8G box and a concurrent 1.9G rsync
tipped it over. Root SSH to that box works from DooPlex via the public IP, not
from felhom-pve via the tunnel IP — the documented path I failed to try first.
R-88: internal/quiesce has NO failure limiter, backoff or breaker; the loop
stopped after three cycles only because PBS recovered. Verified additionally that
scheduledRunAllowed (quiesce.go:476-478) returns true whenever lastAgeSecs is nil,
so the same missing value that makes every poll due also bypasses the time-of-day
gate — the cycles ran outside the [04:30,08:30) window. Fixing the due-verdict
without fixing the nil-age bypass would leave the hole open.
Measured on demo-felhom while the offsite PBS service was down: the controller
re-polls /backup/due every ~5 min, still gets 'due' (storage unreachable + cold
store), and runs the FULL quiesce cycle each time — all four customer app stacks
stopped and restarted for a backup that cannot succeed. ~19 s of app downtime per
cycle, unbounded. The first entry called this bounded and event-only; it is an
availability fault.
Observed live on demo-felhom 2026-07-27 07:02:57 UTC: an agent restart while the
offsite PBS service was down produced a doomed vzdump at that tier. R-84's
read-error fallback to the in-memory record is correct alone but empty after a
restart, so "cannot read the storage" resolved to "no backup has ever been
taken" = due. Same class R-81 fixed in the hub, one layer down in the agent:
unreachable must be UNKNOWN, not resolved.
R-86 (NEXT, operator ruling 2026-07-27): backup-ALIGNED restore-test scheduling
— test a tier ~1 day after ITS OWN backup. R-85 schedules on a free-running
interval, which cannot express 'the day after the PBS backup': any fixed offset
drifts, so alignment would be luck. Shape: trigger from the tier's own last
successful backup rather than a clock. Interim in force: 302400s (3.5d), which
lands each tier ~weekly — the cadence half of the ruling, not the alignment half.
R-87: the restic app-data offsite tier is NEVER restore-tested. R-85 covers
whole-guest vzdump tiers only; the agent has no restic surface. That is arguably
the tier that matters most — the only one that survives losing the box AND
carries the customer's app data, since the whole-guest snapshot excludes the
bind-mounted drives. Exactly the state PBS was in before R-85.
REPORT.md: the full R-80 -> R-85 arc, including a section on the seven mistakes
I made and the two recurring shapes behind them (inferring behaviour from an
artifact instead of the code that consumes it; reading a result without its exit
code). Records demo-felhom's restore-test as IN FLIGHT at close, with the manual
recovery step if the deferred restart watcher does not complete.
Hub gate green (17 packages, rc=0).
- ROADMAP: R-85 row. Code SHIPPED; rotation NOT YET OBSERVED LIVE, stated as
such rather than written as done.
- Capability map: a new row for UNATTENDED restore-proof, IMPLEMENTED not
PROVEN-LIVE, kept distinct from the R-82 row that a MANUAL selftest earned.
That distinction is the same one the activation-vs-arrival split made.
- 03-host-agent §8: the scheduler covers every tier, oldest-proven first; the
spec is per-run; a restore-test joins the one-heavy-op gate. The safety
properties that must not be re-derived are listed.
- 07: restore-proof recorded as a per-tier property. Doc still NOT ratified.
- 06: corrects S4.1's 'the offsite restore-test now runs unattended' — it
silently stopped being true when local_backup_target was retargeted to 'local',
the SECOND time in that doc that a correct mechanism was broken by its input
changing underneath it.
- CONTEXT + REUSE.
Hub gate green (17 packages, rc=0).
R-84 resolved by asking the STORAGE rather than persisting the store: ground
truth, so a pruned archive correctly stops counting where a persisted record
would keep claiming a backup that no longer exists. Proven live on both boxes
with the in-memory store cold.
demo-hp's FIRST EVER offsite backup landed (4.25 GB) — the R-82 finding closed
on the box where it was worst. Controller v0.175.0 deployed to both boxes.
Slice D.1 — host-install 1.20.0: a FRESH box defaults to local-daily +
offsite-weekly (felhom-pbs, 604800s, keep_last=2). setdefault semantics proven
both ways: fresh gets the tier, an UPGRADE preserves the existing backup block
verbatim — so an in-place upgrade can never silently start writing to an
offsite datastore. Existing boxes are migrated explicitly.
Slice E:
- 07-backup-architecture.md: honest status header per CONTEXT ruling S-2, with
an explicit STALE-outside-the-PBS-tier verdict (the controller tiers were last
verified 41 controller versions ago). The PBS row claimed 'PBS on DooPlex'
(the retired spike store) with no cadence; it now names felhom-pbs ->
felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2. NOT marked
ratified — that is Viktor's review of the section 10 list. Discharges R-83.
- 06-offsite-connectivity.md: the target-split remaining-work note collapsed
(shipped), and records HOW S4.1's tier-aware timeout silently regressed — the
mechanism was never removed, its INPUT changed when local_backup_target was
retargeted to 'local'. Also notes S4.1 already diagnosed the teardown 403 as a
phantom (a timeout consequence, not an ACL gap).
- capability map: new row for recurring offsite backups actually LANDING, as
distinct from the existing row proving ACTIVATION. IMPLEMENTED, not
PROVEN-LIVE — the restore round-trip has not completed under the fixed code.
- ROADMAP: R-82 SHIPPED with its remaining gate named, R-83 DISCHARGED, R-84
left open.
- CONTEXT + REPORT: the arc, including the mid-arc correction I had to make.
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class.
On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp
and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel
claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump
archives were on disk. Cause: the agent backup store is in-memory, so the
R-50 fleet restart emptied `backups` until the next run, and the hub read
empty as "no backup exists".
- assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`;
absence is UNKNOWN until it outlives an anchored window. Still pure.
- store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own
retained history (bounded 7-day lookback, early-exit on fresh evidence) —
"when did I last SEE evidence of a backup?" The anchor was free: the hub
already retains 90 days. No agent change, no new persisted state.
- store.GetFirstHostReportAt anchors absence at first contact, reusing the
existing 26h threshold as the grace (no new knob, the v0.73.0 shape).
- Deferrals logged + counted; reason strings kept distinct.
- backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would
alarm six days in seven) and owned by R-82.
Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live
message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all
three now silent.
Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
The allowlist entry is REQUIRED, not cosmetic: handleEvent 400s an unknown
event_type, so controller v0.173.0's new drift alert would be silently inert
without it. Shipped with the controller that emits it.
Docs:
- RUNBOOK-local-api-endpoint-drift.md — how to repair a drift, including the
step everyone will want to skip (establish which value is CORRECT from what
the agent is actually bound to, rather than assuming bootstrap.json wins) and
what success looks like (SILENCE, not a "recovered" line, because a fresh
controller's healthy first observation is not logged). Records both
2026-07-26 repairs.
- ROADMAP: R-77 shipped; R-78 the local_api authority ruling, with the
clobber-a-working-channel risk spelled out in BOTH directions so it is not
resolved opportunistically; R-79 the whole-surface English-strings sweep;
R-80 expected_backup_missed, flagged as likely outranking R-77 because 7.3
days of stale backup materially exceeds the ~1.5-day channel outage, so the
causal link the DIAG hedged on cannot be the whole story.
- Capability map: note against the drive-wizard row (every agent-backed
capability rides this channel) that a silent drift class is now detected.
NO row status flips — detection is not prevention.
New documentation/controller/import-and-data-paths.md: the canonical import root
(and why it is NOT a registered StoragePath), the three data_paths roles, the
Fork-3 validation asymmetry, the class-driven copy rule, and the seven
invariants a future change must not break.
Capability map "File access via browser" — status DELIBERATELY UNCHANGED. The
drop-zone now has its own FileBrowser source and the app page carries a deep
link, both verified live, but nothing drove the FileBrowser HTTP UI (no browser
on DooPlex), so the row's standing "browse is exercised in no doc" caveat still
holds and PROVEN-LIVE remains unearned.
R-75 collapsed to its shipped one-liner. R-76 left open — this task does not fix
it, and nothing built here assumes an import/* directory stays 2775.
R-75 (spiked, GO) names the capability-map row it would flip: "File access via
browser" (00-capability-map.md line 96), currently IMPLEMENTED with the caveat
that browse/download through FileBrowser is exercised in no doc. Carries the
mandatory determinism constraint from P6 (sort + red-proof, or FileBrowser
force-recreates on every sync pass), the zero-removals invariant for
`documents`, the url.PathEscape-not-QueryEscape trap, and the four design forks
with evidence + recommendation, all awaiting operator ruling.
R-76 is minted for the two PRE-EXISTING defects the spike surfaced and
deliberately did not fix: FileBrowser Quantum creating 0644/0755 without
propagating setgid (breaking the shared-group chain one level below any
customer-created folder -- latent only because every userdata-touching app runs
uid 1000), and import/calibre living at 755 on demo-felhom where its same-app
sibling media/books is 2775.
Source: audits/SPIKE-catalog-data-paths-2026-07-26.md
- B2 demo-hp + B3 demo-felhom migrated to the island (agent 0.96.0), apps
served throughout (0 container restarts), island /storage 200, LAN DNS pinned
to the LAN IP, hub reports 0.96.0. No rollback.
- capability-map 'site/network change' row PARTIAL -> PROVEN-LIVE
- ROADMAP R-50 -> SHIPPED (fleet-migrated); add R-74 (island on Peti's cluster)
- nodes.md: both boxes island-bound, agent 0.96.0
Provisioned nested-PVE drill 'drill-r50' (qm300 on demo-hp) via the v1.25.0
nested-vm ISO through the real day-0, then ran the R-50 empirical spike:
- vmbr9 portless island bridge + guest island NIC hot-add (LAN undisturbed)
- F1 replay money shot: LAN move survives on the island; LAN-literal bind
reproduces the 2026-07-20 daemon-exit bug verbatim
- dnsmasq trap confirmed live + lan_resolver.host_ip fix proven
- pin address-independent (leaf SHA-256 unchanged, HTTP 200 over island)
- survival matrix: agent/guest/host-cold-reboot all return on the island
Docs: SPIKE verdict BLOCKED->GO, ROADMAP R-50 SPIKED->GO, nodes.md drill VM,
REPORT overwrite.
New capability-map row (IMPLEMENTED; §13 endpoint-level live on 9201). Records the
ruling: member accounts are superseded by the capability-URL guest share for
launcher sharing; per-member tile visibility parked under the SSO/members arc (R-15).
Updated the launcher row's member-coupling note and R-15 accordingly.
admin
Claude Code