5ca5082e7caa4cbba654a2decec23ba34c866f4e
127 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 admin
|
f45b1f6761 |
docs: R-193 CLOSED (the recovery screen); R-213 minted for the put-back
gates / gates (push) Successful in 7s
- OPEN-ITEMS: R-193 CLOSED with both 2026-08-05 rulings (unlocking and restoring are separate; 'I do not want the old data' moves the store aside after a double confirmation), and the shape-(b) reasoning — WriteOffboxSecrets auto-generates a repository password on re-apply, so the literal 'fresh data area' trigger would have opened a window that closes by itself. - R-213 MINTED (R-212 was and still is the highest, re-checked for the second writer): putting files back in place, with the live-versus-backup comparison named as its requirement. Not started, deliberately. - capability map: the 'needs someone who knows to look' qualifier is GONE; what remains is stated narrowly — no correct-code run through the page, the put-back is out of scope, and the journey has not been re-walked end to end. - 07-backup-architecture 7.0: a fifth row, and where the screen deliberately stops. - CONTEXT: standing ruling S-34. - STATUS: the headline change and the two things still owed as proof. No hub change and no hub bump. |
||
|
admin
|
4faebe2926 |
docs: R-204 ALL FOUR items closed; R-193 credential half; R-192 by replacement; R-212 filed
gates / gates (push) Successful in 8s
- OPEN-ITEMS: R-204 all four CLOSED with both 2026-08-05 rulings recorded (the declared-state trigger and its four-meanings-of-absence reasoning; the recovery preview's dashboard-password exposure accepted as metadata, not content). R-193's credential half CLOSED, screen + deletion still open. R-192 CLOSED by REPLACEMENT. R-202 untouched. - R-212 MINTED (R-211 was the highest, grepped): the orphaned-ciphertext deletion HALTED at its STOP because the measured paths do not match the register — three set-aside stores totalling ~1.45 GB, and the thing that is exactly 1.2 GB is demo-felhom's LIVE repo. Nothing was deleted. - capability map: all four interventions closed; the row KEEPS a qualifier for a new reason — no step needs an operator, but there is no customer-facing recovery screen, and the journey has not been re-walked end to end. - 07-backup-architecture 7.0: the four-step table updated; the declaration-vs- inference reasoning and the credential-automatic/key-customer-present split. - CONTEXT: standing ruling S-33. - STATUS: the headline change and the deletion STOP. - REPORT-r204-item4.md rather than REPORT.md: a parallel session is active in this shared clone. |
||
|
admin
|
0dbd954fec |
docs: R-196 closed, R-204 items 1-3 closed, item 4 open (R-193)
gates / gates (push) Successful in 7s
- OPEN-ITEMS: R-196 CLOSED; R-204 items 1-3 CLOSED with item 4 named and its dependency stated. Header restates that R-202, the 1.2 GB ciphertext deletion and R-198's still-unit-proven retention all REMAIN OPEN. - capability map: the recovery row keeps its 'with a person present' qualifier, names which crutch remains, and cites the three now gone. - 07-backup-architecture: new 7.0 - what a customer can and cannot do ALONE, the four steps in a table with status. This is the section a future reader will use to answer that question. - CONTEXT: standing ruling S-32, superseding S-31 steps 2-5. - STATUS: rewritten to one screen per its own header; removes a corrupted half-overwritten section left from the drill session. - ROADMAP: R-196 and R-204 collapsed. |
||
|
admin
|
2a7ac03c47 |
R-201 PASSED: a customer's file survived a machine rebuild and came back byte-identical
gates / gates (push) Successful in 7s
|
||
|
admin
|
b228fd102d |
R-201 night run: the off-site key IS recoverable after a real rebuild (proven); the verdict is blocked by R-204
gates / gates (push) Successful in 6s
|
||
|
admin
|
73fb595e38 |
R-203 shipped: the app and its backup agree, and 'ok' means it — R-201 unblocked
gates / gates (push) Successful in 7s
|
||
|
admin
|
a0c4b607a6 |
drill R-201: prepared and HALTED before the wipe — a mandatory data directory was missing from the off-site snapshot while the run said ok (R-203)
gates / gates (push) Successful in 8s
|
||
|
admin
|
846253baa8 |
docs: the offsite key is proven recoverable (R-199 closed, R-200 half, R-201 scoped)
gates / gates (push) Successful in 8s
|
||
|
admin
|
91cabdde1b |
hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196)
gates / gates (push) Successful in 7s
R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.
- identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
- ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
- CountCurrentEscrowWithIdentity is the census of who the fix protects.
- Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
- Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
the pre-existing retention test stayed green for two months asserting the mechanism.
R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.
R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.
R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.
Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
|
||
|
admin
|
b7c5cfc061 |
docs: R-193 spike report, operator STATUS, and the ROADMAP R-1 scope correction (F3 covers HOST re-enroll, not a guest rebuild)
gates / gates (push) Successful in 8s
|
||
|
admin
|
c102832892 |
docs: R-86 closed and proven live; ep0 recorded as protected; R-185/186/187 filed
gates / gates (push) Successful in 8s
- OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal reading is never true on a daily tier); R-87 re-ranked UP because R-86 built most of what it waited for; R-185 (the agent cannot list demo-felhom's host backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's sha is not reproducible from its tag), R-187 (R-115's publish leg had never actually run) filed. R-184 was the highest ID in use. - ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row itself proposed — which would have been the never-fires version. - 07-backup-architecture: new contract section — restore-testing is per ARCHIVE GENERATION, with the trap and what did not change (S-1). - 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on the 635 s due-triggered offsite run, with the restart and teardown evidence. - CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines). Numbered 17/18 because S-14 and S-15 were already duplicated in the file. - STATUS: rewritten for the operator, trimmed back to one screen. |
||
|
admin
|
e34b614e5b |
docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected
gates / gates (push) Successful in 7s
R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The hub's notification_log for the run reads: two per-app failures RECORDED, one digest SENT naming both, and the customer channel SKIPPED with operator_only. Against the measured previous behaviour — two failures, one email naming one app, one leaving no trace anywhere. Scenario D proved itself on an event I had not planned: disk_critical alarmed on two filesystems, the second was collapsed by the cooldown, and that collapse is now visible WITH ITS KEY. Yesterday it would have left nothing at all. A gap the spec did not anticipate is recorded with its fix: the per-app event also fires from the periodic sweep, outside any run, so making it record-only would have created a NEW silence. The sweep emits a digest too, with no run_id, so it stays under the ordinary hourly cooldown. ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile SURVIVED the resize and is active (checked, because a resize is a stop/start). The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere. Five documents corrected — three of which the task's list did not name, found by searching. Two audit/evidence documents ANNOTATED, body untouched: they record what was true when written and that is their value. R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable, not unbounded — the original OOM was a 14.46 GB restore — so the restore-test cadence should still be paced, just not by fear of the endpoint. target-selection.md's "D-d did not name ep0 either way" is deliberately left standing. It is the operator's question, not CC's. STATUS.md 127 -> 83 lines, items rather than sentences. |
||
|
admin
|
b0b269b28d |
docs: R-110 + R-115 closed, R-182 re-scoped by measurement, R-183/R-184 filed
gates / gates (push) Successful in 7s
R-182's direction REVERSED by Part 0's measurement. Filed yesterday as "the reserve re-alerts on every status refresh" — too many alerts, seen at the sending end. Measured at the receiving end: 9 events received today, 2 operator emails sent. When two apps are refused in the same second the operator is told about ONE; the other is dropped before LogNotification, so it leaves no row on any channel and cannot be audited. The operator cooldown key is customerID:eventType(+tier) and the capture-failed event carries `app` but no `tier`, so the key has no app identifier. Same failure mode as R-97a, in a second event type that never opted into the narrow fix. Nothing changed — Part 0 was investigation only. Correction owed: yesterday's report said "one recovery_unit_capture_failed per app, HTTP 200". True of what the CONTROLLER pushed; a reader would take it as "the operator was told about each app", which is false. R-110 CLOSED (installer v1.23.0). Both channels moved. The spec's mechanism for channel 2 rested on a factual error — the run-time fetches are sixteen, not nine, and come from felhom-agent, not this repo — so no tag here could cover them; pinned to the agent version being installed instead, on the operator's ruling. Channel 3 needed no change: the URL never carried a ref, so no hub change and no hub bump. R-115 CLOSED. release-agent.sh builds, tags, publishes and verifies by an independent download; check-published-versions.py refuses a tag with no package; CI now runs the full gate set so it actually runs. R-183 NEW+CLOSED: a fresh install fetched the vouched agent binary and its sixteen config files from two different refs, and nothing compared them. R-184 NEW: nothing stops the hub vouching a version that was never released. The R-115 gate cannot see it — measured, the hub manifest and Gitea's package listing are both 401 anonymously. capability map: new PROVEN-LIVE row for the published installer channel. STATUS.md 138 -> 127 lines. |
||
|
admin
|
fb652024ea |
docs: R-181 closed, R-156 closed, R-110 + R-115 rulings recorded, R-182 filed
gates / gates (push) Successful in 7s
R-181 CLOSED (controller v0.193.0 + v0.193.1) and proven live on demo-hp for BOTH reserve terms. The reserve is now a per-app, per-run ADMISSION decision taken before the app's first write and covering all three write legs, and it gained a size term. The refusal's wording was not weakened; the behaviour moved so it became true, verified by sha256 tree fingerprint. R-156 CLOSED — papra's template mounts the app's own data root. Precondition re-measured rather than inherited (both boxes were wiped today). Part 4, documentation only, nothing built: - R-110 WAITING-ON-OPERATOR -> READY. Ruling: option (b), the installer's publish channel moves to a TAG. Recorded with the condition that decides whether it works at all — it must cover BOTH the /scripts/ git-sync and the nine files the installer fetches from raw/branch/main. - R-115 WAITING-ON-OPERATOR -> READY. Ruling: mechanism (b), a build-side gate refusing to deploy or vouch an unpublished version. The third instance (agent v0.120.0) would have silently downgraded both demo boxes while succeeding. R-182 NEW: the periodic status refresh has no admission scope, so a refused app re-alerts on every poll (measured: a second alert pair 13s after the run's). Pre-existing in v0.192.0; deliberately not fixed in the R-181 task. capability map: the local-backup row moves to PROVEN-LIVE in BOTH halves. ROADMAP: R-165 collapses to CLOSED; R-181 collapsed into it. 07-backup-architecture.md: the reserve's contract stated as what the code provides (S-1 — an architectural contract changed in the same session). STATUS.md trimmed 150 -> 111 lines, "What's broken" no longer holds shipped work, and the stale "After:" line (pointing at work that shipped on 2 August) is fixed. |
||
|
admin
|
41dbecb264 |
docs: R-167 + R-158 CLOSED, R-165 SPIKED, R-174..R-177 filed
gates / gates (push) Successful in 8s
R-167/R-158 shipped and proven live (controller v0.191.x, hub v0.89.0): two new capability-map rows PROVEN-LIVE with live citations, and 07-backup-architecture.md §7.5's closing claim "nothing warns when an app crosses the line" is now false and rewritten (S-1: an architectural contract changed in the same session). §7.5 also gains the caveat that its size bound is ONE BOX'S, not the fleet's. Part 3 SPIKE (audits/SPIKE-r165-mp1-merge-2026-08-02.md): M1-M5 measured, NO layout touched. Three findings the merge session must not re-derive: "the layout" is not one thing (200G/50G vs 50G/20G vs 16G/8G); mp1 is a BULKHEAD and not only a ceiling, so after the merge an overflow reaches /var/lib/docker; the golden fails closed on the split in four places. D-a's condition (1) is currently SATISFIED — no external box is in the hub's register, and both demo boxes are Tier 0 and reinstallable. Recommendation given, choice NOT made — it ends at the operator's ruling. CONTEXT.md S-11 (D-c's routing, and why R-158's own backup_failed proposal was overruled) and S-12 (the monitoring landed BEFORE the merge). STATUS.md gains the plain-language section and the merge decision, with two older entries trimmed so the page did not grow. New rows R-174 (closed same session), R-175, R-176, R-177; each ID grepped free before minting. |
||
|
admin
|
ad28699761 |
docs: R-157 A / R-170 / R-171 closed — boot recovery finished
gates / gates (push) Successful in 7s
Controller v0.190.0. Docs only here; no hub change, no hub version bump. - audits/DIAG-bootrecon-drive-absent-2026-08-02.md — NEW. The Part 0 diagnosis, including the run that produced a FALSE NEGATIVE and the mechanism behind it (the agent re-binds an unmounted drive within ~60s, so the drive gate's startup reconcile restarted the apps one second before the sweep looked). Records that the write hazard was blocked only by an ACCIDENTAL filesystem permission that no code owns and no test pins. - architecture/02 §0a — the boot-recovery contract (S-1): both gates read desired state; the sweep observes a SETTLED fleet and each sample must refresh first; nothing is started without asking, fail-safe. Plus the durable warning: Manager.StartStack has no gate of its own. - 00-capability-map — the boot-recovery row, with the repeat count cited per N.5 (6 of 6 hard resets) rather than a bare PROVEN-LIVE. - OPEN-ITEMS / ROADMAP — R-157 CLOSED (both mechanisms), R-170 CLOSED, R-171 NEW and closed the same session, marked a regression from v0.189.0. - STATUS.md — the power-cut line moved from "What's broken" to "What works right now" with its repeat count; one dated bullet in the change log. - CONTEXT.md S-13 — the lessons worth carrying: "it didn't happen this time" is not a disproof; widening a window makes previously-unreachable overlaps reachable; and a settle detector is only as good as the freshness of what it samples — the fix's own defect, found live rather than by review. |
||
|
admin
|
5c97fbc397 |
docs: R-166 SHIPPED — the desired/in-flight/observed split (D-b)
gates / gates (push) Successful in 8s
Controller v0.189.0 implements operator decision D-b. Docs only here; no hub change and no hub version bump. - architecture/02-controller-module-map.md §0a — NEW, and it is the S-1 contract: desired (app.yaml) / in-flight (own marker file) / observed (not persisted), with the rule that ties them — never derive one from another. Absent desired state means UNKNOWN, never "running". One file, one writer. D-b's binding safety rule quoted verbatim. - 00-capability-map.md — the boot-recovery row now rests on a recorded signal, with the three live flows from 9201. The interrupted-operation half is marked IMPLEMENTED, not PROVEN-LIVE: nobody killed the controller mid-backup on metal. - OPEN-ITEMS/ROADMAP — R-166 SHIPPED with both blocking facts and their answers; R-157 mechanism B CLOSED and A restated as the whole item; R-170 NEW (the drive-backed boot gate still infers a Stop from a container count). - STATUS.md — the "an app can stay switched off and nothing says so" line rewritten to what is actually left: timing. - CLAUDE.md — end-of-session checklist gains: confirm your own last push's CI run went green, BY RUN ID. The failure email is a push signal; this is the pull check. - CONTEXT.md S-12 — the rulings, and the two lessons worth carrying: a test that constructs the thing it should prove the caller constructs is hollow (its red-proof will say so), and a field-by-field struct rebuild in a save path is a defect on sight. |
||
|
admin
|
c718aad1bc |
docs: R-168 SHIPPED, R-29 CLOSED on the demonstrated alarm, R-169 minted
gates / gates (push) Successful in 7s
SPIKE-ci-runner-2026-08-02.md: all six probes with method, measurement and ruling; none STOPped. P2 (stock image has git but no python3) and P6 (a runner that loses its state re-registers and orphans the old record) changed the design; P5 (a failed run signals NOTHING) is why the alarm exists at all. R-168 SHIPPED with its evidence. R-29 CLOSED — on the demonstrated alarm and not on a green run, as required: the class it opened is answered at both ends, the hook refusing locally and CI catching a --no-verify bypass and emailing. R-161 noted: its automatic half now exists for the STATIC gate, while its original scope, the runtime gate, is deliberately still not automatic and should stay that way. NEW R-169 (grep established R-168 was the highest in use): CI can only report, because there is no gate in the road. Making it blocking needs branch protection plus a PR workflow, both of which change how the operator works — so it is theirs to decide, and the row states the cost honestly rather than recommending it. CONTEXT gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector that tells no one is not finished, plus the curl and Cloudflare-1010 traps), S-10 (the runner is unprivileged because DooPlex is Tier 2), S-11 (CI reproduces the sibling layout). CLAUDE.md gains the rule earned by red-proofing: a go test -run pattern that matches no test prints ok and exits 0, and an instrument that can silently drop results is not a measurement. |
||
|
admin
|
4707be755c |
docs: R-94 closed, R-29 leg (a) closed + leg (b) half, R-168 minted
hub/CHANGELOG v0.87.0 + scripts/CHANGELOG gate-enforcement entry. CONTEXT gains S-6 (the hub renders no host-install version and the gate pins its absence) and S-7 (gates run from one entry point per repo; reuse_refs_check was fixed rather than the REUSE.md convention, with both rejected alternatives recorded). OPEN-ITEMS: R-94 CLOSED all three legs, leg (a) by DELETION with its reason; R-29 leg (a) CLOSED and leg (b) HALF-SHIPPED with the census result written into the row (13 gates; every gate a CLAUDE.md names was green, two of the four unnamed were red); R-161 gains its successor pointer. NEW R-168 (grep established R-167 was the highest in use): Gitea Actions runner — measured 2026-08-02 as Gitea 1.26.2, Actions enabled on all four repos, 0 runners, 0 workflow runs, 0 branch protections, and the consequence that trunk-based direct-to-main pushes leave no merge for a status check to gate, so CI here can detect but not block. BLOCKED on a spike over host-mode vs privileged DinD on DooPlex and whether the workflow can avoid JavaScript actions. ROADMAP: R-94 collapsed to its one-liner, R-29 updated, R-168 added. |
||
|
admin
|
260a8f6e58 |
register: R-161 ruled and shipped at reduced scope; re-ranked
The operator ruled on R-161 and the runner shipped in app-catalog-felhom.eu (fd7747d), so the row moves from BLOCKED-needs-a-ruling to REDUCED SCOPE - open. Both obvious enforcement points were rejected for measured reasons, and the row now records them rather than leaving the rejection implicit. Controller-side at template load: rejected because such a check can only read the file, and a static audit of all 53 templates reports the catalog clean INCLUDING papra - it would pass on the exact defect it exists to catch, the property being decidable only at runtime. CI: rejected for now, neither repo has any and there are no users yet. Shipped instead: scripts/catalog_gates.py, one entry point over all three gates, non-zero exit on any failure, mandated in the catalog's CLAUDE.md the way site_gates.py is. The rationale is recorded because it is the transferable part - of this project's gates, the only ones that ever get run are those with a single entry point named in a CLAUDE.md; site_gates.py is run and R-29's three orphans are named nowhere and have stopped nothing. What stays open is only the automatic half, which is sufficient while ONE person touches templates - revisit when a second does. Re-ranked accordingly: R-161 drops from 2nd to 7th, and R-156 is promoted to 2nd, since R-161 was ranked high precisely because nothing ran the gate and that is no longer true. The de-ranking is recorded inline with its reason, matching how R-94's de-ranking is recorded, so a later reader sees a decision rather than drift. |
||
|
admin
|
b06ea9c877 |
register: file R-156..R-164 in one pass, ranked; and record what mp1 is actually for
Nine rows into OPEN-ITEMS.md and ROADMAP.md, matching each file's column shape. R-156 and R-157 had lived only in audit documents - the identical "minted in a spike doc and never carried across" failure the register already records for R-153/R-154/R-155, caught by the catalog sweep's own section 8.0 while it was happening. R-158 was minted by a second session the same day for an unrelated finding, which is why the sweep's proposals were renumbered R-159..R-162 at filing time. All nine IDs verified free in BOTH backlog files before use. Part 0 settled the question the sizing item depended on, by reading: mp1 is RETENTION, not staging, and neither of the two framings was right. A unit is the KEPT copy on the app's OWN drive (backup.go:245-255); for an app with no HDD_PATH the namespace falls back to the system SSD - "the SSD-only system-data fallback" (appbackup/paths.go:26-27). There is no post-copy deletion: the only prune is F5 residue-on-old-drives when an app MOVES (backup.go:1053-1112). So mp1 retains the units of driveless apps only - not every app, but not transient either. Confirmed against the spike: sys_drive held exactly the four driveless apps and not calibre-web, which had a drive and was still backed up. A unit is volume tars + DB dumps only, never mp8 userdata (recovery_unit.go:20-25), so a 1 TB photo library can never overflow one. And mp1 gates the WHOLE chain, not just Tier 1: Tier-2 mirrors the unit "(always)" from RecoveryUnitPath (tier2.go:302,368) and Tier-3 carries it, so a unit that cannot be written leaves both with nothing to copy. Part 2 fired on both triggers - retention, and the fallback undocumented - so 07-backup-architecture.md gains section 7.5. Section 6.1 said a unit lives "on the app's own drive", which is true and was the whole story only for drive-resident apps; the no-drive case was undocumented, as was the sizing constraint. 7.5 records the mp0-50G-vs-mp1-20G mismatch, the measured ratios (DB app up to ~2x, 21.1GB -> 40.2GB; file-only 1.00x), and the bound this puts on D5's Lane-1 independence: restorable from the drive alone only while the unit still fits - about 19 GB file-only, about 10 GB DB-backed. No number proposed; the ratio is the operator's ruling (R-163). R-159/R-160 marked SHIPPED only after verifying the template changes are in app-catalog origin/main, and R-156's gate likewise (check-volume-persistence.py present). papra is NOT fixed - referred - so R-156 stays open on that one app. Ranked, with one line of reasoning each: R-157 first (an app can stay down indefinitely with mechanism B silent on every channel), then R-161 (the gate exists and nothing runs it, which is why R-156's class recurs - R-29's record is three orphaned gates and one enforced), R-156, R-163, R-158, R-164, R-162. |
||
|
admin
|
e9a74a0019 |
docs: remove a gate criterion that could never pass, and close three register rows
PART 1 — the release gate.
G7 required the packaged .deb to sha256-match the one built from committed source. That is
unsatisfiable BY CONSTRUCTION: dpkg-deb stamps the build time into every archive, so two builds of
byte-identical source differ. It was already failing when the 1.26.1 release ran it. A criterion
nobody can satisfy gets waived once and read as advisory ever after — which is how R-29's shelf of
never-run gates was built. Sub-clause dropped, reason recorded in G7's own note the way G6's
amendment was, so a future reader can restore it if SOURCE_DATE_EPOCH ever makes it meaningful.
RULING ASKED FOR — is payload integrity covered by G9 alone? NO, and G9 is widened rather than a new
criterion invented. The package ships TWO payload files (build-deb.sh:54-55); G9 checked only the
script. The systemd UNIT was covered by nothing: G7 covered the container, G8 covers the postinst
behaviourally, G13 covers directory presence. The unit is not incidental — its After=, its
ConditionPathExists= and its Restart= decide WHEN AND WHETHER day-0 runs at all, so a drifted unit
would have shipped silently. Same shape as the /etc/felhom miss that G13 exists to prevent: a check
that proved the thing present and said nothing about what it depended on. The check passes today.
G13 moved to sit after G12 — it was minted late and left between G10 and G11.
PART 2 — register dispositions. BASELINE DISCREPANCY, reported rather than worked around: only R-128
had a row. R-154 and R-155 had NO row in either file — minted in a spike document and never carried
across, which is R-123's class, not the drift the task described. Rows created, closed, with the
reasoning, because in all three cases the reasoning is the durable part:
R-128 closed by CORRECTING a false claim, not by making the assertion real — the coupling does not
exist and asserting it would invent a constraint. Flagged so nobody 'restores' it.
R-154 closed with the measurement and where it now lives in pushed source.
R-155 NARROWED, not deleted — unchanged for FELHOM_MENU=single, inapplicable to release. Flagged so
the guard is not later removed wholesale on the strength of 'R-155 closed it'.
Documentation only: no code, no build, no ISO, no upload, no box touched.
|
||
|
admin
|
1956e5d390 |
hub v0.84.0 — break-glass console credential on the host page
The credential existed and was not reachable when it was wanted. Every box has
had a strong random root@pam password since TASK G1, vaulted in the hub at day 0
and used for real during the sshd incident — but the only way to read it back was
a hand-written curl carrying the global operator key, a secret kept out-of-band.
In practice the PVE web console on a demo box felt locked.
The host page grows a Console access card: presence + username + set_at by
default, Reveal fetches the plaintext on demand for 60 s with a Copy button.
Masking clears the JS variable, and also fires on a second click and on
visibilitychange. A host with nothing vaulted says so, and says why.
The secret is NEVER rendered into the page, and that constraint shapes the
change. The render path uses a new store.GetHostRecoveryMeta whose struct and
SELECT both omit the secret column, so it is structurally incapable of carrying
one. The plaintext crosses the wire only in the response to POST
/hosts/{id}/reveal-recovery-credential (Cache-Control: no-store, CSRF-gated at
the ServeHTTP level; POST precisely so that gate applies and so no secret is
retrievable by URL alone). Deliberately NOT the customer page's data-secret
widget, which embeds the plaintext on every load.
A delivered reveal writes one recovery_credential_revealed event on the host's
customer timeline (info, source hub, Hungarian) via SaveEvent alone — no
dispatcher, nobody emailed, the log_tail_requested shape. Two reveals write two
events: the register records accesses, not states. A 404 is not an access. An
unbound host reveals fine and writes no event; the [INFO] hub line, carrying the
username and a length only, is then the record.
The global-key API path is untouched by design — it is the route for when the
hub UI itself is broken, and coupling it to the session layer would delete the
independence that makes it a fallback.
Recorded as a real trade: the hub session password alone now unlocks console root
fleet-wide, where retrieval previously also needed the global key. Accepted for a
single-operator, HU-geo-fenced hub that already stores these passwords in
plaintext at rest (CONTEXT.md ruling S-4). The plaintext-at-rest half is filed as
R-133 — every hub DB backup is a fleet-wide console-credential dump.
Tests 550 -> 559; four red-proofs (page leak, audit event, CSRF gate, route
order) each run, observed failing, and reverted. The route-order proof is a seam
test driving ServeHTTP: a handler-level test cannot see that defect, because the
handler is correct and simply never runs.
|
||
|
admin
|
0a9bd3829d |
D5 SHIPPED: Tier-1/2 restore no longer depends on the whole-guest tier
Records controller v0.188.0 across the four coupled artifacts. 07-backup-architecture.md is the owning doc: - new 7.4 = the recovery chain AFTER D5 (7.1 leg 1 superseded; leg 2, the living-app dependency, explicitly unchanged so this is not read as more than it is) - 7.3 collapsed to history, with the correction that the target as written (data_key-only) was tested in Part 0 and rejected - 3 records that the two-lane split is now real, not just intended - matrix rows 3 / 3c (new) / 13; 10.1 D5 itself shipped Also: new capability-map row, D5 collapsed in ROADMAP + OPEN-ITEMS, and R-127 filed in both (data_key flag unreliable; O4 can regenerate a DB password that no longer matches the restored data directory). The audit is named D5-drive-alone-restore rather than "...secrets..." because .gitignore blocks *secret* -- a guard worth respecting, not forcing past. |
||
|
admin
|
d42d90fed7 |
R-108 CLOSED — D5's precondition is met (controller v0.187.0)
Four-artifact update per the coupling rule, plus the audit. 07-backup-architecture.md: §10.1 retitled CLOSED with the ruling and the D5 sentence; the FileBrowser network-share row flipped YES->NO, closed at the PLACEMENT rather than at the bind; the exposure chain annotated with the fifth surface (decommission-with-migrate guarded only its source) and the correction that the boundary is the deploy POST, not the dropdown; §7.3 retitled UNBLOCKED; register row collapsed; open question F answered. 00-capability-map.md: new §D row PROVEN-LIVE, with the un-exercised legs named — the deploy-POST and decommission refusals are unit-tested, not live-fired. OPEN-ITEMS.md: R-108 dispositioned; D5 given its OWN row as READY/UNBLOCKED (it had existed only inside other rows' prose — the R-123 thread-loss pattern); R-126 registered. ROADMAP.md: R-108 collapsed to a shipped one-liner; R-126 added. R-126 filed not fixed: a .fab bundle (plaintext secrets, optional password) can be exported ONTO a NAS. Split out of R-108 rather than folded in — it is an explicit customer-chosen export destination, not a browsing surface reaching a backup tree, so it was never part of D5's precondition. Live evidence: same-box before/after on demo-felhom through the real authenticated endpoint, the network-specific refusal on demo-hp, non-effect verified in the registry, and R-67's share-root bind diffed byte-identical across the deploy. |
||
|
admin
|
70f84941d4 |
R-106/R-109 audit + registers: shipped at agent 0.118.1, plus R-125
Adds the full audit: Part 0's three answers, the pre/post recipe for both boxes, the on-disk proof that `local` froze at the 2026-07-28 target move while felhom-backup kept running, all seven red-proofs, and the three publish observables. R-125 filed: v0.118.0's R-106 half shipped INERT. Two tests ran the real Collector.Collect() but both injected a fakeObserver, and the break was one layer below in mergeConfig, which dropped the pbs namespace. The recipe still said "root" — now with namespace_state "resolved" beside it, confident and wrong. Caught by live validation, not by the green suite. Fixed in 0.118.1; filed for the doctrine point that a production-path claim must name the seam it injects at. |
||
|
admin
|
acfc2b7e95 |
R-109 + R-122: the recipe assembly stops dropping sections (hub v0.83.0)
AssembleDRRecipe's hostHalfShape/appHalfShape are ALLOW-LISTS, not the forward-compat their comment advertised: a section an emitter adds is silently discarded until it is named in both the shape struct and AssembledRecipe. No error, no log, no failing test. R-122 (found this session): that already happened and shipped. The controller has emitted offsite_restic since fork-4 — the offsite recovery LOCATION — the hub stored it for all three real customers, and appHalfShape never listed the key, so no delivered recipe has ever contained it. It stayed green because the fixture drAppHalf is hand-written and omits the field. R-109: the agent's new backup_target is a new top-level host-half section and would have been dropped identically, making the fix read as shipped while changing nothing an operator can see. 3 tests built on halves read verbatim out of the live dr_recipe table, plus 2 red-proofs (each mutation asserted to have landed). vet rc=0, suite rc=0, 17 ok. Registers: R-106 + R-109 dispositioned; R-105/R-106 were READY in ROADMAP with no OPEN-ITEMS row (→ R-123, registered); R-124 filed on the "root" spelling. |
||
|
admin
|
1a68b53b06 |
hub v0.82.0 (R-120): the vouch path REFUSES a golden the fleet has already outrun
The golden's version IS the controller it bakes (build-golden.sh:345 defaults GOLDEN_VERSION to the controller tag), so a golden behind the newest deployed controller means every FRESH install lands on stale application code. On the R-120 occurrence that stale code shipped a customer-facing falsehood: a box from the 0.185.1 golden told a customer whose backup drive had fallen out that the backup was on the same disk as the system -- false, the drive was gone -- and offered a different drive as the remedy. WHY A GATE, NOT A REMINDER. The gap has opened three times: R-111 (golden's agent 17 releases behind), R-115 (agent built and deployed, never published), R-120 (this). The first two were closed by re-baking and remembering; remembering then failed again. R-29 is the standing proof that a check nobody runs is worse than none because it reads as coverage -- hostinstall_gates.py sat RED and uninvoked across three version bumps and hub_confirm_gate.py has never run at all. So the property that matters is not whether a check exists but whether it BLOCKS. - Wired into handleSetArtifacts (internal/web/configs.go), immediately before the only write, on the sole UI path to SetArtifactManifest -- it runs on every vouch without anyone choosing to. A script in scripts/ would have been a fourth orphan. - It REFUSES (operator ruling, 2026-07-30), with a flash naming the remedy. - Signal: store.NewestReportedControllerVersion() over reports.controller_version, SEMVER-compared in Go -- MAX() in SQL ranks 0.99.0 above 0.186.0, a pair this fleet has shipped. No outbound call, no new credential. - Fail-open in exactly two deliberate cases: an empty golden field (clearing the manifest is legitimate) and an unknown fleet version (a new hub must vouch its first golden). NEAR-MISS RECORDED: the first draft read guests.controller_version, a column that exists in the schema and that NOTHING writes -- it would always have seen "" and failed open, i.e. inert, this gate's own failure shape. Caught by grepping for a writer before trusting the column. Blind spot stated rather than papered over: a controller no box has ever run is invisible to this signal. Not the failure that has bitten -- all three instances were deployed-newer-than-baked. 4 tests through the PRODUCTION handler over httptest, never an injected seam. The refusal asserts both the flash and that the manifest was NOT written, because a gate that redirects and saves anyway reads as enforcement while providing none. Red-proof: deleting the block makes the stale golden vouchable and both assertions fail. ROADMAP R-29's audit list now records this as the FIRST enforced gate, so the contrast with its three orphans is kept rather than lost. The orphans are unchanged. Suite rc=0 read separately from this commit. |
||
|
admin
|
06d7788392 |
Session C: R-113/R-114/R-112 PROVEN LIVE; C5 fails on a new defect (R-116)
Full ISO/PAIRING run on a fresh nested box. Agent 0.114.0 came from the Day-0 manifest -- the SHIPPED binary -- so C5 tested the real artifact. Controller 0.186.0 hand-deployed after install per the §3.1 ruling; the vouched golden bakes 0.185.1, so C3/C4 prove the code not the shipped golden, and that lag is filed against R-115 rather than a new ID. R-113 PROVEN: detach 18:43:50, gate fired 18:43:54 -- four seconds, where E-2d measured zero over 4.5 minutes -- and SetDisconnected was reached. It fired on exactly the shape that defeated it: raw /mnt/mentes NOT mounted while the bind /mnt/felhom-drives/mentes still read /dev/sdb[/felhom-data]. R-114 PROVEN: with the target absent the page rendered the absent copy, the system-disk copy 0 and the offer block 0. Both of E-2d's falsehoods are gone. R-112 PROVEN: the banner reached a customer's page for the first time. Healthy renders nothing, proven POSITIVELY -- idle delta 0 /backup/tiers calls, page load delta +1, single caller, so the seam ran and chose silence. C5 FAILED on a fourth, separate defect. The alarm fires but as the GENERIC storage_disconnected, while the recovery is the SPECIFIC backup_target_restored -- a pair an operator cannot match, which is what notifyDriveReturned's own comment forbids. backup_target_absent count 0 across the run. Root cause: the drive is TWO /disks rows and BackupTarget and GuestPath sit on different ones; absent they separate, on return they rejoin. v0.184.1 fixed the keying, not this. Only reachable because R-113 made the gate fire at all. Filed as R-116. Mirror + over-correction guard PASS: non-target drive -> storage_disconnected, backup_target_absent 0; both drives present -> 0 ABSENT lines and the target stayed healthy. Caveat recorded: the mirror passes trivially because the target also produced the generic event. E-2 and E-2d CLOSED as partially proven with R-116 the one named open leg, per the runbook's §9 rule decided in advance rather than mid-run. Capability map NOT touched: it has no E-2 rows at all, so nothing could move to PROVEN-LIVE. Creating them is a design act, not a validation act. Teardown clean: pvesm status after == before (local-lvm 38.78%), guest 9201 and drill-r50 untouched. Customer delete attempted and correctly refused while the host still reads ONLINE; command recorded for once it ages to DOWN. |
||
|
admin
|
338b2ccf86 |
agent 0.114.0 published + vouched; R-115 files the recurring publish gap
PART 1 — Session C unblocked. Agent 0.114.0 (the R-113 fix) was built, pushed and deployed but never published, so a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix. Published from the clean tree at b58d7bc via scripts/publish-agent.sh; sha 5e4c15ebee2d7583d57301d1f7c9cc7d4276262966bf738b05e34653bfd18c31, verified by an INDEPENDENT round-trip GET (http=200, sha match, binary self-reports 0.114.0), and the hub manifest read back after the write. Deliberately NOT done, each with a reason: - No golden bake. The golden bakes the CONTROLLER, not the agent, and host-install fetches them as separate generic packages (:1945 / :2573). Golden 0.185.1 is current, so there is no new-agent-against-old-golden risk. - min_agent NOT raised, stays 0.113.0. It expresses what the CONTROLLER requires of the agent, and controller v0.185.0 declares MinAgent 0.113.0 — which 0.114.0 already satisfies. Raising it to 0.114.0 would have been a false claim AND would have held demo-hp and drill-r50. No box is held; no §3 STOP fired. - Global controller floor NOT raised (v0.156.0), per R-111's reasoning. - wrapper_sha256 preserved verbatim; re-checked against configs/felhom-pbs-apply before and after — no drift both times. demo-hp RULING: left on 0.113.0. The R-113 fix is not live-validated, so putting it on a second box widens exposure for no proof, and Session C's nested box takes its agent from the manifest, not from demo-hp's host agent. Move the fleet once, after Session C. PART 2 — R-115 opened (WAITING-ON-OPERATOR). The finding is the RECURRENCE, not either instance: publishing is a remembered step, and it was forgotten within eight hours of R-111 documenting it as forgettable. Filed as a new ID with a back-pointer rather than reopening R-111, because R-111's finding (the channel WAS stale) is closed and verified end-to-end, while the process defect that caused it is a distinct problem with a distinct fix and owner. Class cross-linked to R-29 (a control that exists and is never walked) WITHOUT minting a second ID for it. Options are stated as the operator's decision, with mechanisms (build-step, deploy gate) separated from reminders (checklist, manual) — R-29's whole finding being that reminders do not hold. No code written, by design. R-111 gains a deferred-leg-recurred line; its shipped evidence is untouched and it is NOT reopened. R-113 records that Session C is now unblocked. |
||
|
admin
|
f3975cf5bc |
E-2d executed on a fresh box: C1/C2 proven, C3/C4 partial, C5 FAILS — R-112/113/114
Full ISO/PAIRING route on a nested PVE VM on demo-hp, after R-111 was fixed earlier in the session. Bind -> running controller in 3m35s. The install fetched the artifacts published an hour before and restored the golden baked 20 minutes before, so the publish train is proven end to end on a real install. C1 PROVEN: "felhom-host-install v1.22.0", "Day-0 provision SUCCESS", guest 9201 running, bootstrap unit wrote its done-flag and self-disabled. This retires E-2's "installer-logic-tested, not install-tested". C2 PROVEN: both DEGRADED warning lines verbatim, backup.local_backup_target=local, no felhom-backup storage created, and the install did not abort. C3/C4 PARTIAL and C5 FAILED — three findings, none fixed: R-112 (P1): E-2's degraded banner and offer have NO UI CONSUMER. The endpoint returns byte-exact copy; grep 'backup-target' across every html/js/css is 0 hits and no page handler injects the state. Templates fetch 18 distinct /api/storage/* endpoints; these two are the only ones with zero references. v0.185.1 fixed the router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired. R-113 (P1): the drive-absent gate CANNOT FIRE on device loss. planDriveGates reads presence from BoundUnderParent = "is this path in the guest's mountinfo". The raw mount is a device-bound systemd unit and dies with the device; the agent's own bind is not device-bound and outlives it, so the gate sees "present" forever. Live: agent reported the drive absent every 20s for 4.5 minutes, the controller logged 0 [gate] lines, the hub received zero events -- neither backup_target_absent nor the generic storage_disconnected. Sixth instance of the class: E-2b wired the seam to a condition that cannot occur. R-114: on target-drive loss the message claims the backup is on the system disk (false) and offers the drive that just vanished. Invisible only because of R-112, so it must be fixed BEFORE R-112 is wired. Also filed as a second instance under R-110 rather than a new ID: host-install fetches nine files from raw/branch/main and the hub vouches a sha for one; E-2a's wrapper is installed 0755 to /usr/local/sbin, root-fenced in sudoers, validated only by bash -n. C4 is fully proven at API level: decline path (registration confers no role), restart_required:true, agent did NOT self-restart (in-flight check performed and recorded first), E-2a wrapper created the storage at the drive's own mountpoint, and healthy renders nothing. Teardown: VM destroyed, scratch storage removed, pvesm status after == before (local-lvm 38.77%), guest 9201 and drill-r50 untouched. Hub records for e2d-fresh remain -- delete correctly refused at four gates, finally "host is ONLINE"; deletable once it ages to DOWN. Command recorded in OPEN-ITEMS.md. capability-map NOT touched: the customer-facing legs are broken rather than proven, and the map has no E-2 rows at all. |
||
|
admin
|
f3f0d58844 |
E-2d: Phase 0 STOP — the Day-0 artifact channel cannot deliver the code under test
No VM created, no install run, no box touched. The run stopped at the Phase 0 gate per runbook §3, before provisioning. felhom-host-install.sh does not install what is on main. resolve_artifacts() (:423-436) reads the hub-vouched manifest and fetches Gitea GENERIC PACKAGES (agent :1945, golden :2573). Gitea's newest are agent 0.96.0 and golden 0.161.0; the hub manifest selects exactly those; the global floor v0.156.0 is below the golden's 0.161.0 so nothing self-updates. A fresh box therefore lands on agent 0.96.0 + controller 0.161.0 against main's 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by direct deploy and is not in the channel at all. Claim impact, each pinned to its introducing commit: - C1 (real rc=0 1.22.0 install) and C2 (Case B natural) — ACHIEVABLE, not run; both are installer-side and host-install is served at 1.22.0. - C3 — BLOCKED: banner + GET /api/storage/backup-target are controller v0.185.1 (cdaeb36), copy v0.185.0 (3f7cf2a). Unblocks cheaply by raising the hub floor to >=0.185.0; measured fleet impact nil (both demo boxes already 0.185.1). - C4 — BLOCKED: needs controller v0.185.1 + agent v0.113.0 (58b598b). - C5 — BLOCKED: needs controller v0.184.0 (c1a63de) + agent v0.112.0. Filed R-111 (P1): 17 unpublished agent releases (v0.97.0-v0.113.0) strand the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT, so a new customer's box installs without them. Mirror of R-110, not a duplicate. - audits/E2D-fresh-vm-2026-07-29.md — all four Phase 0 answers recorded so a resumed run does not re-derive them (cadence 30s; hot-detach available; ISO present; local-lvm fence re-measured at 38.77%, unchanged). - OPEN-ITEMS.md — R-111 opened; E-2d re-stated, NOT closed. - ROADMAP.md — R-111 under P1. - capability map NOT touched: nothing was proven live. The §5.1a operator STOP is retired — HUB_PW is in ~/.config/credentials and hub auth was verified, so CC can bind on a resumed run. |
||
|
admin
|
7383400a23 |
docs: file R-29 to the register; attach the gate-orphan instance to its class
|
||
|
admin
|
d4c07873ca |
docs: correct the installer-channel record — R-94 retracted and re-scoped, R-110 opened
The 2026-07-29 R-94/E-2d finding was written from an unverified claim and was false. `felhom-bootstrap.sh:96` fetches the installer from the WEBSITE, not the hub; the website git-syncs /scripts/ from main on a 30s period; every install since 1.22.0 hit main this morning already runs 1.22.0. Confirmed by live fetch. - OPEN-ITEMS.md: merge the two duplicate R-94 rows into one, retract the false framing, re-scope to what it actually is (a drifting hand-synced constant plus two pieces of dead safety equipment), unblock it from E-2d. - OPEN-ITEMS.md: de-rank R-94 in the ranked list — the "high-consequence" reason was the false claim in its most load-bearing form. - OPEN-ITEMS.md: E-2d — the ISO is the STRONGER proof route, not an obstacle. Phase 0 question answered at source: PAIRING falls through to run_direct in the same invocation (:495-499), so it reaches the identical installer call. - ROADMAP.md:149: same retraction; the original diagnosis (a hand-synced constant in a second repo drifts every time the first ships) survives. - ROADMAP.md + OPEN-ITEMS.md: new R-110 — main is the installer's publish channel and there is no staging, tag, pinned path or rollback, for the one artifact that runs as root on a virgin box. Operator ruling, not a defect. - day0-install.md C.1: one sentence recording the same about the fetch URL. Documentation only. No version bump, no CHANGELOG entry, no code, no box touched. |
||
|
admin
|
b5a73e050b |
Move the local whole-guest backup off the guest's own device (demo-hp + demo-felhom)
Supervised operational run. No code, no version bump, nothing deleted.
Primary backup tier on both demo boxes moved from `local` (a dir storage on
/var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir
storage on each box's secondary drive:
demo-hp /mnt/nvme-1tb uuid:91d2dc2d-... archive 2,256,044,492 B
demo-felhom /mnt/hdd_1 uuid:47a3361a-... archive 5,957,878,962 B
Both proven end to end via the real UI path: archive lands on the secondary
drive (df delta matches the archive byte-for-byte), restore-test auto-selects it
and passes with mount_parity: ok, and freshness survives an agent restart with
an empty in-memory store -- so the age can only have come from the new storage.
Phase 0: the target is CONFIGURATION, not converged (the sole writer of
agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so
the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path.
Findings:
- F-1 the storage path must BE the mountpoint; a subdirectory fails exactMount
and the target reports disconnected permanently (observe.go:321)
- F-2 --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a
non-mounted path reports active with the ROOT filesystem's free space and
had already created dump/ on pve-root -- a silent retarget onto the very
device this change escapes
- F-3 FelhomAgentStore is granted per storage path; without it every backup
403s. felhom-host-install.sh must issue it for new installs
- R-109 (new) the DR recipe records no backup target, and each box now carries
two content=backup dir storages, one live and one frozen
- R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the
enrolled drives were never PVE storages, so isUserDataDrive never saw
them. Both boxes now populate drives; SMART on the backup drives too
Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half
live-proven; agent half source-traced). That is NOT the intended fall-back-and-
alarm design -- filed as E-2 with the honest single-drive label.
Reported in full in the record: the agent was restarted with a felhom-pbs backup
in flight, producing a spurious tier failure. The backup had in fact succeeded
(PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no
channel -- R-84 ground truth superseded it.
Outstanding: full drive-loss recovery (needs physical access) and the agent half
of the absent-drive behaviour.
|
||
|
admin
|
070b0ce2dc |
07-backup-architecture: rewrite as the recovery model (D1-D6); D5 BLOCKED by R-108
Replaces the 2026-07-14 DRAFT entirely - it was verified against controller v0.132.0, 51 versions stale, while being cited as authoritative. Now verified against controller 0.183.0 / agent 0.110.0 / hub 0.80.0 / catalog 4252121. NOT ratified. Every statement is labelled [DESIGN] (a decision from the architecture discussion) or [FACT] (file:line, live output, or a citation to _recovery-inventory-2026-07-28.md). Phase 0 verified D5's premise and it FAILED. Ten read surfaces were checked; nine are clean (SMB under both namespace shapes, FileBrowser for local drives, .fab import and download, /api/debug/*, all three ServeFile sites, registering the backup dir as a drive). One is not: FileBrowser binds a NETWORK share at its ROOT while local drives are userdata-scoped, and no IsNetwork() filter guards the deploy dropdown, the per-app migrate targets, or handleStorageMigrateApp. So an app namespace - and under D5 its plaintext secrets - can land somewhere a browsing surface serves with download:true. Not a leak today (the unit's app.yaml is secret-stripped); it becomes one under D5. D5 is therefore written as a BLOCKED target, not as settled. -> R-108 Core artifact is the failure -> recovery matrix (15 rows). RTO cells carry only measured durations; 12 are blank and each blank is listed explicitly as a finding. RPO cells carry the configured cadence that bounds RPO, labelled as such - no RPO has ever been measured from an incident and none is invented. The 7/53 vs 9/43/1 Tier-2 coverage counts are both recorded, unresolved; neither adopted. Coupling (S-1): capability map section C now cites the matrix instead of restating routes; ROADMAP gains R-102..R-108; OPEN-ITEMS gains only the in-flight R-108 plus C9-F1b/F3/F4 -> R-n cross-references; CONTEXT records D1-D6 as standing ruling S-3. No code, no version bump, no deploy, no box touched. |
||
|
admin
|
655b69f37d |
docs: R-97 shipped; N.5 gains its four legs; ops results recorded
R-97 collapsed to its shipped one-liner in ROADMAP and closed in OPEN-ITEMS. PROMPT-TEMPLATE N.5 now names FOUR coupled artifacts instead of two: the capability map, ROADMAP, the owning architecture doc (ruled as S-1 in CONTEXT.md but never reflected in the template CC actually reads, so it bound nobody), and OPEN-ITEMS.md. Tasks must now report which register rows they opened, closed or re-ranked. Ops: R-90 swap done (interim; CX33 still blocked), R-95 mitigation armed but zero snapshots taken so it moves to WATCHING rather than closed, R-91 gate still not satisfied. CONTEXT.md datastore path corrected to /mnt/pbs-datastore. |
||
|
admin
|
65409aecd1 |
docs: R-88 Part 1 shipped; Phase 0 root cause; R-97 minted
R-88 split: Part 1 (the failure breaker) SHIPPED in controller v0.176.0 and live on both boxes; Part 2 (unknown != never) stays OPEN and is agent-side. Phase 0 established the root cause at source: newestArchiveOn's (time.Time, bool) signature cannot represent 'unknown', so a storage read ERROR collapses into a positive 'no successful backup recorded yet'. The errored and genuine-never paths are byte-identical on the wire, which is why Part 2 cannot be done controller-side. R-97: the whole-guest backup tier has no failure signal to the hub at all — internal/quiesce never imports internal/notify, so three failed backups and three app-stack outages produced zero backup_failed events. Its only trace was a customer-tier Hungarian app_start_failed for an app the backup itself had stopped. |
||
|
admin
|
7826bd7179 |
docs(roadmap): reconcile against reality — seven previously unfiled items
Read-only triage found work that was agreed or discovered but never given an id:
R-95 restic offsite credential CAN delete — answers the parallel question R-89
raised and left open. Per-customer subaccounts report readonly=False, the
controller runs forget --prune from the box, and the sftp: backend cannot
express append-only. Storage Box snapshots (snapshot_limit=10, plan=null,
0 used) are server-side and SFTP cannot delete them — an unused zero-code
mitigation.
R-94 hub pins hostInstallVersion 1.19.0 while host-install ships 1.20.0, so a
hub-driven install still gets the pre-R-82 backup default.
R-90 ep0 has no swap at all and OOM'd today; gates R-86.
R-91 the pre-migration 13 GB datastore copy still occupies ep0's root disk.
R-92 PBS-DR gauge granularity. R-93 drill-r50 fixture tension.
R-96 two standing rules agreed in chat and never committed (the third, N.5's
third leg, IS committed at CONTEXT.md:8).
|
||
|
admin
|
a31872ea24 |
docs(pbs): move PBS prune server-side, close the write proof, schedule GC
Supervised runbook execution. No code, no version bump. The felhom-pbs tier had reported `job errors` on EVERY demo-hp backup since the tier was created on 07-26, while the data landed correctly every time: `DatastoreBackup` grants Datastore.Backup but not Datastore.Prune, so the box's keep_last=2 prune was denied. Operator ruling: retention is a COMMERCIAL attribute owned by the hub; ep0 executes. Box tokens therefore stay write-only - a compromised box must not be able to delete its own offsite backups. No grant was widened and felhom-tenantsync.sh is unchanged (the ruling makes it correct). Increment 1: - boxes stop attempting prune. allowPBSPrune is DERIVED (`!t.Primary && t.KeepLast > 0`), so keep_last: 0 on the PBS tier disables both the --prune-backups value and the gate in one config edit, and the tier stays armed. Verified prune_pbs_allowed=false on both boxes with no tier REJECTED line. - per-namespace prune jobs on ep0, keep-last 2, daily 03:30 UTC (05:30 CEST), dry-run gated. demo-hp 3->2, demo-felhom untouched, chunk count unchanged (prune removes indexes, not chunks). Write proof CLOSED: 08:25:47 job errors -> 09:37:29 TASK OK, snapshot 2026-07-27T09:37:29Z, chunks 9787->9813, prune step absent entirely. Driven through POST /api/guest-backup/trigger (the UI path), not --selftest and not raw vzdump. Hub gauge evidence explicitly NOT satisfied - the delta is below its 0.1 GB display granularity. GC scheduled sun 04:30 UTC and deliberately NOT run: every chunk still carries a fresh atime from the migration copy, so a run today would reclaim nothing. verify-new enabled per operator ruling, turning an inert hub alarm live. Legacy demo-felhom-01 namespace deleted with its two ACL entries and its token (operator ruling, confirmed twice) so nothing dangles. R-89 records the target architecture and carries the unanswered parallel question: does the restic key on storage-box-pool-1 have DELETE rights? If so the daily app-data tier has the identical exposure and append-only is the equivalent answer. ep0 is Etc/UTC, not CEST - corrected in the record. |
||
|
admin
|
a16896af86 |
docs: R-88 root cause established — no limiter, and the nil age bypasses the window gate
Corrects two wrong severity readings with evidence from the box and the code. The PBS outage was ~15 min (07:00-07:18 UTC), caused by a global OOM at 06:58:12: proxmox-backup-proxy peaked at 3.2G on a 3.8G box and a concurrent 1.9G rsync tipped it over. Root SSH to that box works from DooPlex via the public IP, not from felhom-pve via the tunnel IP — the documented path I failed to try first. R-88: internal/quiesce has NO failure limiter, backoff or breaker; the loop stopped after three cycles only because PBS recovered. Verified additionally that scheduledRunAllowed (quiesce.go:476-478) returns true whenever lastAgeSecs is nil, so the same missing value that makes every poll due also bypasses the time-of-day gate — the cycles ran outside the [04:30,08:30) window. Fixing the due-verdict without fixing the nil-age bypass would leave the hole open. |
||
|
admin
|
5aca709ca1 |
docs(roadmap): correct R-88 severity — it is a repeating quiesce loop, not one event
Measured on demo-felhom while the offsite PBS service was down: the controller re-polls /backup/due every ~5 min, still gets 'due' (storage unreachable + cold store), and runs the FULL quiesce cycle each time — all four customer app stacks stopped and restarted for a backup that cannot succeed. ~19 s of app downtime per cycle, unbounded. The first entry called this bounded and event-only; it is an availability fault. |
||
|
admin
|
eb3f0b8d3f |
docs(roadmap): R-88 — an unreachable backup target reads as "no backup exists"
Observed live on demo-felhom 2026-07-27 07:02:57 UTC: an agent restart while the offsite PBS service was down produced a doomed vzdump at that tier. R-84's read-error fallback to the in-memory record is correct alone but empty after a restart, so "cannot read the storage" resolved to "no backup has ever been taken" = due. Same class R-81 fixed in the hub, one layer down in the agent: unreachable must be UNKNOWN, not resolved. |
||
 Claude Code
|
6d11c54a50 |
R-85 close-out: session REPORT + R-86/R-87 queued
R-86 (NEXT, operator ruling 2026-07-27): backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup. R-85 schedules on a free-running interval, which cannot express 'the day after the PBS backup': any fixed offset drifts, so alignment would be luck. Shape: trigger from the tier's own last successful backup rather than a clock. Interim in force: 302400s (3.5d), which lands each tier ~weekly — the cadence half of the ruling, not the alignment half. R-87: the restic app-data offsite tier is NEVER restore-tested. R-85 covers whole-guest vzdump tiers only; the agent has no restic surface. That is arguably the tier that matters most — the only one that survives losing the box AND carries the customer's app data, since the whole-guest snapshot excludes the bind-mounted drives. Exactly the state PBS was in before R-85. REPORT.md: the full R-80 -> R-85 arc, including a section on the seven mistakes I made and the two recurring shapes behind them (inferring behaviour from an artifact instead of the code that consumes it; reading a result without its exit code). Records demo-felhom's restore-test as IN FLIGHT at close, with the manual recovery step if the deferred restart watcher does not complete. Hub gate green (17 packages, rc=0). |
||
|
Claude Code
|
57ba3c7c8c |
R-85 Phase 4: docs — the UNATTENDED path is documented as unproven
- ROADMAP: R-85 row. Code SHIPPED; rotation NOT YET OBSERVED LIVE, stated as such rather than written as done. - Capability map: a new row for UNATTENDED restore-proof, IMPLEMENTED not PROVEN-LIVE, kept distinct from the R-82 row that a MANUAL selftest earned. That distinction is the same one the activation-vs-arrival split made. - 03-host-agent §8: the scheduler covers every tier, oldest-proven first; the spec is per-run; a restore-test joins the one-heavy-op gate. The safety properties that must not be re-derived are listed. - 07: restore-proof recorded as a per-tier property. Doc still NOT ratified. - 06: corrects S4.1's 'the offsite restore-test now runs unattended' — it silently stopped being true when local_backup_target was retargeted to 'local', the SECOND time in that doc that a correct mechanism was broken by its input changing underneath it. - CONTEXT + REUSE. Hub gate green (17 packages, rc=0). |
||
|
Claude Code
|
a0bdd8eed6 |
docs: R-84 SHIPPED (agent v0.103.0) + demo-hp's first offsite backup
R-84 resolved by asking the STORAGE rather than persisting the store: ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists. Proven live on both boxes with the in-memory store cold. demo-hp's FIRST EVER offsite backup landed (4.25 GB) — the R-82 finding closed on the box where it was worst. Controller v0.175.0 deployed to both boxes. |
||
|
Claude Code
|
adf1d1e619 |
R-82 Slice D/E: installer default 1.20.0 + architecture docs brought current
Slice D.1 — host-install 1.20.0: a FRESH box defaults to local-daily + offsite-weekly (felhom-pbs, 604800s, keep_last=2). setdefault semantics proven both ways: fresh gets the tier, an UPGRADE preserves the existing backup block verbatim — so an in-place upgrade can never silently start writing to an offsite datastore. Existing boxes are migrated explicitly. Slice E: - 07-backup-architecture.md: honest status header per CONTEXT ruling S-2, with an explicit STALE-outside-the-PBS-tier verdict (the controller tiers were last verified 41 controller versions ago). The PBS row claimed 'PBS on DooPlex' (the retired spike store) with no cadence; it now names felhom-pbs -> felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2. NOT marked ratified — that is Viktor's review of the section 10 list. Discharges R-83. - 06-offsite-connectivity.md: the target-split remaining-work note collapsed (shipped), and records HOW S4.1's tier-aware timeout silently regressed — the mechanism was never removed, its INPUT changed when local_backup_target was retargeted to 'local'. Also notes S4.1 already diagnosed the teardown 403 as a phantom (a timeout consequence, not an ACL gap). - capability map: new row for recurring offsite backups actually LANDING, as distinct from the existing row proving ACTIVATION. IMPLEMENTED, not PROVEN-LIVE — the restore round-trip has not completed under the fixed code. - ROADMAP: R-82 SHIPPED with its remaining gate named, R-83 DISCHARGED, R-84 left open. - CONTEXT + REPORT: the arc, including the mid-arc correction I had to make. |
||
|
Claude Code
|
f5a5e2b911 |
hub v0.75.0: R-81 — "no signal" is not "bad signal" (anchor the backup deadline check)
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class. On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump archives were on disk. Cause: the agent backup store is in-memory, so the R-50 fleet restart emptied `backups` until the next run, and the hub read empty as "no backup exists". - assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored window. Still pure. - store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own retained history (bounded 7-day lookback, early-exit on fresh evidence) — "when did I last SEE evidence of a backup?" The anchor was free: the hub already retains 90 days. No agent change, no new persisted state. - store.GetFirstHostReportAt anchors absence at first contact, reusing the existing 26h threshold as the grace (no new knob, the v0.73.0 shape). - Deferrals logged + counted; reason strings kept distinct. - backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would alarm six days in seven) and owned by R-82. Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all three now silent. Source: documentation/audits/DIAG-backup-missed-2026-07-26.md |
||
|
admin
|
9cfa619ec3 |
hub v0.74.0: allow local_api_endpoint_drift; R-77 docs + R-78/79/80
The allowlist entry is REQUIRED, not cosmetic: handleEvent 400s an unknown event_type, so controller v0.173.0's new drift alert would be silently inert without it. Shipped with the controller that emits it. Docs: - RUNBOOK-local-api-endpoint-drift.md — how to repair a drift, including the step everyone will want to skip (establish which value is CORRECT from what the agent is actually bound to, rather than assuming bootstrap.json wins) and what success looks like (SILENCE, not a "recovered" line, because a fresh controller's healthy first observation is not logged). Records both 2026-07-26 repairs. - ROADMAP: R-77 shipped; R-78 the local_api authority ruling, with the clobber-a-working-channel risk spelled out in BOTH directions so it is not resolved opportunistically; R-79 the whole-surface English-strings sweep; R-80 expected_backup_missed, flagged as likely outranking R-77 because 7.3 days of stale backup materially exceeds the ~1.5-day channel outage, so the causal link the DIAG hedged on cannot be the whole story. - Capability map: note against the drive-wizard row (every agent-backed capability rides this channel) that a silent drift class is now detected. NO row status flips — detection is not prevention. |
||
|
admin
|
89d64f64b3 |
R-75 SHIPPED (controller v0.172.0): docs, capability-map note, ROADMAP collapse
New documentation/controller/import-and-data-paths.md: the canonical import root (and why it is NOT a registered StoragePath), the three data_paths roles, the Fork-3 validation asymmetry, the class-driven copy rule, and the seven invariants a future change must not break. Capability map "File access via browser" — status DELIBERATELY UNCHANGED. The drop-zone now has its own FileBrowser source and the app page carries a deep link, both verified live, but nothing drove the FileBrowser HTTP UI (no browser on DooPlex), so the row's standing "browse is exercised in no doc" caveat still holds and PROVEN-LIVE remains unearned. R-75 collapsed to its shipped one-liner. R-76 left open — this task does not fix it, and nothing built here assumes an import/* directory stays 2775. |