487 Commits

Author	SHA1	Message	Date
admin	c917251eeb	docs: P6 EXECUTED — containerd store moved to SSD2; reboot deferred (R-209, R-209a) ... Operator ruled "proceed" after reading the pre-analysis; the storageReserved condition CC made the move conditional on was applied with it. The pre-analysis is retained verbatim as §11-bis so the ruling can be audited against what it was given. MOVED WITH ZERO LOSS, verified BEFORE the original was touched on four independent observables: 550,891 = 550,891 filesystem entries; 448 = 448 trusted.overlay xattrs (load-bearing — overlayfs stacking rides them, which is why rsync -X is not optional); 37,243 = 37,243 hardlinks; byte-identical meta.db sha256. Verified again after: identical image/tag/volume ID SETS, cache 2.782 GB / 38 records, ceiling still 30 GiB, pg 4 DBs / 31 tables / 175,135,767 B, redis DBSIZE 2437. End-to-end proof was a real build on the relocated store, rc=0 — not an inventory count. k3s was never at risk, and that was established BEFORE stopping anything: it runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and ~160 pods stayed up. Only the two jarr-* dev containers were affected. TRAP FOUND WHILE PROVING THE GUARD, and it is the reusable part: RequiresMountsFor on a path with NO mount unit is a SILENT NO-OP — containerd started normally against an absent-but-unmounted path. A typo'd guard buys nothing and says nothing: the built-but-never-wired shape again. The guard was therefore verified positively at the unit level (Requires= AND After=mnt-ssd_2.mount on both units), and refusal proven with a genuinely absent DEVICE via a temporary synthetic .mount unit — /mnt/ssd_2 hosts 12 live Longhorn replicas and must never be unmounted, and editing fstab on a production host risks emergency mode at boot. Result: "Job containerd.service/start failed with result 'dependency'", is-active inactive. It refused rather than creating an empty store on the root disk. storageReserved on SSD2 0 -> 80 GB; still Schedulable=True at 76.34%. NOT PROVEN, AND NOT GLOSSED (R-209a): the operator ruled DO NOT REBOOT, so the move has not survived a reboot — which is this task's own standard for validated. The MECHANISM is proven; the CONSEQUENCE (that a real boot mounts /mnt/ssd_2 before containerd starts, in this host's actual ordering) is not, and mount-ordering reasoning is exactly the class this project has been burned by. So: the pre-move tree stays MOVED ASIDE, not deleted (34.3 GB, which is why / sits at 54% and not lower), and felhom-store-postboot-check.service — oneshot, enabled, dry-run PASS at install — validates it automatically at the next boot, asserting positively that /var/lib/containerd does NOT exist. Read /var/log/felhom-store-postboot-check.log after the next boot; on PASS the rollback copy can go and ~34 GB returns. Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).	2026-08-05 10:27:40 +02:00
admin	ee9d9bf203	docs: SPIKE — DooPlex build-cache containment; R-205..R-211 (2026-08-05) ... Spike output only; no production Go code. The one shipped change rides in homelab-manifests 6808a4b (R-205, the monitoring rule). VERDICT: mechanism confirmed, with one correction and one refutation. - CONFIRMED: builder.gc IS honoured under the containerd worker and DOES evict. Proven by naming a 440 MB `go mod download` record present at build N and absent by N+2 — not by absence of an error. - CORRECTED: honoured ONLY in the `policy` array form. The flat form is SILENTLY ignored — daemon starts, logs nothing, keeps its defaults. `dockerd --validate` returned "configuration OK" for a bogus key AND for a config that then crashed the daemon. The oracle is `docker buildx inspect`. - REFUTED: Docker's `data-root` would NOT move the cache — it moves 0.62 GB. The 181.4 GB belongs to the system containerd (`root` in /etc/containerd/config.toml). P3 (operator-approved) executed: prune claimed 156.9 GB, the filesystem returned 150.35 GB (the 6.5 GB gap is layers shared with images), SYNCHRONOUSLY — / went 86% -> 53% used, and Longhorn's default disk went Schedulable=False (DiskPressure) -> Schedulable=True (18.85% -> 50.32%). P7 root-caused the largest item and it is NOT the cap: all 208 `go mod download` records had Usage count 1. Isolated by controlled builds — same VERSION build-arg -> CACHED, new VERSION -> executed, byte-identical tree. `ARG VERSION`/`ARG GIT_COMMIT` sit ABOVE the module-download step, and a RUN's cache key includes the stage environment. Both Dockerfiles have it. One line each to fix -> R-208. P6 NOT EXECUTED — stops at the operator, as specified. Pre-analysis: the move is safe as measured (+38.8 pp above the 25% floor) but SSD2 is the only Longhorn disk with storageReserved=0 and is overcommitted 6.9x; at full inflation the move lands 12 pp BELOW the floor. The prune removed the move's urgency, so CC recommends against it unless ~80 GB is reserved on SSD2 -> R-209. Register: R-205 (CLOSED, shipped), R-206 (Ansible: cap + prune + narrowed Docker ban), R-207 (DRY_RUN guard), R-208 (ARG ordering), R-209/R-210 (operator), R-211 (Prometheus has no config-reloader — rules changes have never applied until something restarted the pod; found while verifying R-205). Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).	2026-08-05 10:02:46 +02:00
admin	0dbd954fec	docs: R-196 closed, R-204 items 1-3 closed, item 4 open (R-193) ... - OPEN-ITEMS: R-196 CLOSED; R-204 items 1-3 CLOSED with item 4 named and its dependency stated. Header restates that R-202, the 1.2 GB ciphertext deletion and R-198's still-unit-proven retention all REMAIN OPEN. - capability map: the recovery row keeps its 'with a person present' qualifier, names which crutch remains, and cites the three now gone. - 07-backup-architecture: new 7.0 - what a customer can and cannot do ALONE, the four steps in a table with status. This is the section a future reader will use to answer that question. - CONTEXT: standing ruling S-32, superseding S-31 steps 2-5. - STATUS: rewritten to one screen per its own header; removes a corrupted half-overwritten section left from the drill session. - ROADMAP: R-196 and R-204 collapsed.	2026-08-05 07:37:35 +02:00
admin	2a7ac03c47	R-201 PASSED: a customer's file survived a machine rebuild and came back byte-identical	2026-08-04 23:18:47 +02:00
admin	b228fd102d	R-201 night run: the off-site key IS recoverable after a real rebuild (proven); the verdict is blocked by R-204	2026-08-04 22:33:01 +02:00
admin	73fb595e38	R-203 shipped: the app and its backup agree, and 'ok' means it — R-201 unblocked	2026-08-04 18:52:14 +02:00
admin	a0c4b607a6	drill R-201: prepared and HALTED before the wipe — a mandatory data directory was missing from the off-site snapshot while the run said ok (R-203)	2026-08-04 15:00:06 +02:00
admin	846253baa8	docs: the offsite key is proven recoverable (R-199 closed, R-200 half, R-201 scoped)	2026-08-04 13:55:25 +02:00
admin	91cabdde1b	hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196) ... R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site data key, which is the secret the retention was built to preserve. And because the copy happens as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups stay recoverable. Both demo boxes crossed that line on 2026-08-04. - identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion. - ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob. - CountCurrentEscrowWithIdentity is the census of who the fix protects. - Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone. - Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why the pre-existing retention test stayed green for two months asserting the mechanism. R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity blob retained, a changed key is "this history now depends on an older recovery code", not a loss. R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and regressed, each stating its own measurement; the regressed text withdraws the Re-issue recommendation. Every self-heal refusal leaves a notification_log row with its reason. The guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in the alert text so the limitation travels with the number. offsite_delivery_stuck and offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither has a customerMessages entry, which is not a block. R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the restic repo password. It resets the PROVIDER password and cannot touch the repo password, which is generated on the box. All five corrected; the staleness mark documented as precautionary. The BEHAVIOUR stays open. Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).	2026-08-04 12:56:58 +02:00
admin	3f2b7bc023	RECON: trace the offsite DR chain link by link — it does not join up (R-198..R-201) ... Read-only recon of the escrow -> recovery chain, from a dead node to an open repository. No production code, no build, no version bump. Headline: the hub's superseded-escrow retention does NOT retain the offsite repository password. host_escrow_superseded has no identity_blob column and demoteCurrentEscrowTx copies only the K-escrow blob, so what survives a supersession is the PBS datastore key, not the restic repo password. The next escrow ceremony -- which the system tells a rebuilt box's customer to run -- destroys the last copy. Both demo boxes crossed that line on 2026-08-04. Also established: - the hub's blob-serving endpoints (re-enroll / restore-directive) have zero callers anywhere: agent, hub UI, scripts, runbooks (R-199) - POST /backup/offbox/inject-password is routed and handled but no template contains the form (R-200) - nothing in the recovery path has ever been exercised; the one live round-trip proof (2026-06-10) predates the ResticRepoPassword field (R-201) - a fail-closed mint refusal IS implementable: the report ACK already carries escrow{identity_blob_present, restic_pw_sha256} and the controller discards it whenever no offbox target exists Corrections: yesterday's spike annotated (candidate (b) overturned in part -- unattended recovery is impossible, customer-present is not); capability-map retention claim struck through and replaced with what the code does. Deliverable: documentation/audits/RECON-offsite-dr-chain-2026-08-04.md Register: new R-198..R-201; R-193 and R-192 updated; STATUS.md refreshed.	2026-08-04 12:16:04 +02:00
admin	b7c5cfc061	docs: R-193 spike report, operator STATUS, and the ROADMAP R-1 scope correction (F3 covers HOST re-enroll, not a guest rebuild)	2026-08-04 11:12:08 +02:00
admin	7fff45d688	R-195: a customer with no machine ever bound does not alarm (hub v0.92.0) + R-193/R-192 spike ... Part 4 (ships): `david` — a prospective customer with hosts=0, host_deletions=0, reports=0 — e-mailed an expected_dbdump_missed ERROR at 03:00 UTC three mornings running. The existing down-skip could never cover it: it reads the staleness checker's state, which is seeded from a query over the `reports` table, so a customer that never reported has no state at all and GetState() returns "" rather than "down". store.HasEverBoundHost (hosts row OR host_deletions tombstone) is consulted once per customer at the top of the deadline loop. The discriminator is "was a host EVER bound", never "has a report arrived" — a box installed and never heard from is a real fault and keeps alarming. Fail-OPEN on a read error. Red-proof observed: removing the guard fails with `got [expected_dbdump_missed]`, verbatim the event david sent. Parts 0-3 (spike, NO production code for R-193/R-192): audits/SPIKE-offsite-credential-recovery-2026-08-04.md establishes that the one-shot provider password is the RECOVERABLE secret and the restic repository password is the irreplaceable one — and that a guest rebuild mints a fresh one, orphaning the previous off-site history. Measured without touching a box, by comparing host_escrow.restic_pw_sha256 against host_escrow_superseded: BOTH demo boxes changed (demo-hp 15 snapshots / 40.9 MB, demo-felhom 36 snapshots / 1.14 GB). demo-felhom's "lucky" 76-second recovery restored delivery and not the repository, silently, for 13h. ReissueCredentials does NOT rotate the restic password (R-39's record and two hub comments are wrong -> R-196); candidate (b) is not implementable against a zero-knowledge escrow; candidate (a) already exists as F3 and is wired to the wrong event. Ends in ranked options and an unanswered question for the operator. R-195 SHIPPED; R-196 + R-197 filed; R-192 + R-193 updated, neither closed.	2026-08-04 11:04:39 +02:00
admin	f456835bbc	REPORT/R-190: the alert pair closed — one loss, one email, one recovery ... The 20-minute latch expired at 10:20:29 and the hub logged degraded -> ok (agent_capability_recovered) at 10:30:40. Final state on both boxes: agent 0.124.1, two ACL rows on /storage/felhom-backup.	2026-08-04 10:31:46 +02:00
admin	e06920d930	R-188: the reorder moved the race rather than removing it (observed on the v0.124.0 release) ... CI run 28 failed and run 29 passed on the same sha. The tag was created locally at 07:38:36Z; run 28 started 07:38:31Z (commit push) and run 29 at 07:38:41Z (tag push). The old race was 'tag before package'; the new one is 'package before its tag is pushed' — the window the converse probe exists to detect, and it did. Not confirmed from the run log (the endpoint 404s), so recorded as the timing-consistent explanation rather than a proven one. The gate passes locally with all 7 versions installable and none untagged. Both obvious closures are the 'make the gate blind for a window' shape this row refused, so the options are to accept a seconds-wide window or to make releases quiet to CI another way.	2026-08-04 10:08:10 +02:00
admin	a684a81796	docs: R-191 closed, R-190 mitigation shipped (mechanism open), R-194 filed ... R-191: the gate was run first and it mattered — ep0's prune jobs have run daily since 2026-07-27 (18 tasks, all OK). THREE of my own queries said the opposite and all three were broken instruments (worker-type vs worker_type, prune vs prunejob, proxmox-backup vs proxmox-backup-proxy); acting on any would have disabled the only pruning attempt while reporting that nothing prunes. R-190: the row now reflects the MITIGATION, not the cause, and says so. Includes that v0.124.0's transition record failed in production while every unit test passed, and Part 3's single pass — the token-rotation class is real and documented in our own installer but does not fit this box (same token still authenticates, the other three grants survived, no installer run evidenced); ruled out on this box, not fleet-wide. R-194: PVE caches permissions (~40s and ~16min measured), so a single permission read is a lagging indicator and may explain when R-190's refusal surfaced. 07-backup-architecture: the offsite retention row was RIGHT and the configuration disagreed with it — noted, since a doc that states a contract does not enforce it.	2026-08-04 10:04:25 +02:00
admin	33f0ab2f33	R-193: demo-hp offsite restored on the box; escrow correction recorded ... Re-issue through the designed endpoint: fresh password staged 07:11:51, settle-gate GO, consumed 15s later at 07:12:06, offsite configured at 07:12:09 against the SAME sub-account (275124) and repo path. Both boxes now escrowed. CORRECTION to this session's own reading: the escrow did NOT recover by itself. escrow_state went pending -> escrowed 15s after the apply and I inferred automation; the operator had run the ceremony. It needed a human on both boxes — demo-hp at 07:16:02, demo-felhom at 07:20:28 (its escrow had been pending since its own rebuild the day before). A 15-second state change is not evidence of automation, which is the same class as reading an absent log line as success. Snapshot continuity is NOT established: both boxes report snapshot_count 0, but the run-history keys are ABSENT rather than zeroed — a controller that has never run an offbox backup, not an empty repo. The 04:15 run decides it.	2026-08-04 09:23:08 +02:00
admin	2584dfb938	R-193: a guest rebuild silently drops the offsite tier; R-192 cause established ... Operator confirms no hub-side offsite config change, so the regression was not an action. Evidence: demo-hp's controller went 0.187.0 -> 0.192.0 at 06:12:18 with a new config hash and the agent re-keyed its leaf three minutes earlier — a guest rebuild. The last pre-rebuild report shows the tier fully healthy: escrowed, last success 02:16:39Z, 15 snapshots, 40.9 MB. No offsite object in the 108 reports since. Mechanism: the restic credential is delivered once. demo-hp consumed its secret on 2026-07-23; the rebuilt controller has no copy and no way to request another. demo-felhom survived the SAME rebuild only because its secret was still unconsumed — it consumed it four seconds after its config hash changed and was reporting offsite again 76 seconds later. That difference was luck, not design. Also sharpens R-192: the self-heal's guard refuses when any report since the consume carried an offbox target, but that query reads the OLDEST 500 reports — all of which predate the rebuild. Healthy history before a rebuild is not evidence the credential still works, which is why the automation that exists for this case declined to act.	2026-08-04 09:05:53 +02:00
admin	967ddf2f44	R-192: offsite_delivery_stuck states the opposite of what it measured ... demo-hp has been emailing daily since 2026-08-03 06:12 UTC that '500 reports since carry no offbox target — the credential is likely burned'. Measured against the hub's own data: all 500 of those reports DO carry one. The credential was consumed 2026-07-23 and applied successfully; the controller reported an offsite object continuously until 2026-08-03 05:59:21, then it vanished at 06:12:19 and has been absent for 108 reports. A regressed apply, not a burn. Two defects. maybeEmitStuck builds its message from ReportsSinceConsume while hardcoding 'carry no offbox target' and never consults OffsiteReportsSinceConsume, so the recommended action targets a failure that did not occur. And maybeHeal refuses SILENTLY on exactly that field ('regressed-apply shape -> operator's call', a bare return), so there is a daily email with the wrong story, no heal, and nothing saying why the heal declined. The underlying condition is real: that box currently has no off-site copy of its customer app data, since 08:12 CEST on 2026-08-03. What removed it is not established.	2026-08-04 08:57:53 +02:00
admin	d692c46db0	RUNBOOK: the host tier is restore-PROVEN unattended on both demo boxes ... Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite 540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest, booted, verified and destroyed itself; zero 990000 guests or volumes afterwards and both local-lvm figures returned to their pre-run values. Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for the first time: never-proven sorted first, each box took its HOST tier, deferred the offsite one, and picked it up on the next evaluation six hours later. The host-tier proofs reached the hub through R-189's merge — demo-felhom's report carries two tiers, and the local one can only have come from disk. The capability map's optimistic half is cashed, with its scope stated: these two boxes, not the fleet. Surfaced and filed rather than fixed: - R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with a reinstall, any logged pveum activity and any cluster-log entry ruled out. - R-191: every weekly offsite backup uploads successfully and then fails the job on a prune the box is deliberately not allowed to do (R-89 moved it server-side; both boxes still arm keep_last=2). Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the write path (six, with the hub raising whole_guest_backup_failed at the first), and my earlier "no restore_test_* events" was produced by grepping a 404 page.	2026-08-04 08:13:43 +02:00
admin	b93ee06abc	R-190 filed; two corrections to yesterday's R-185 record ... CORRECTION 1 — the runbook annotation and the R-185 row both said the drift did not surface as a 403 because writes go through a root path. That is WRONG. demo-felhom's local-api backup jobs 403'd six times between 09:24 and 17:34 CEST on exactly that storage and privilege, and the hub raised whole_guest_backup_failed at the first with edge-triggering suppressing the rest. The impact was not only an unreadable tier: the agent's own whole-guest backups to it were failing. CORRECTION 2 — on this box the grant was LOST, not never issued. A vzdump by the agent's token to that storage completed OK at 04:44:50 the same morning; the first 403 is 09:24:56. Ruled out by measurement: a host reinstall (uptime 12 days), any pveum/ACL/user.cfg activity in syslog 04:00-10:00, any ACL entry in the cluster log. Correlated but not established: guest 9201 was reprovisioned nine minutes before the first failure. R-190 files the unexplained disappearance, and notes that the new store-grant probe detects the STATE but says nothing about the TRANSITION.	2026-08-03 20:11:09 +02:00
admin	e3187c86d5	docs: R-185 closed — the silence as well as the grant ... - OPEN-ITEMS: R-185 closed with the measurement, the corrected root cause (the installer's Scenario-F reuse arm, not PVE_STORAGES), and the live sequence. Records that demo-hp carried the same drift and was fixed too. - capability map: the whole-guest row's HOST-tier half was OPTIMISTIC and now says so — that tier was not merely unproven, it was unprovable on both demo boxes, and every live proof cited was on the offsite tier. - vzdump-target-move runbook: its item 5 predicted this; annotated (not rewritten) with what actually happened — the create arm did grant, the reuse arm did not, and it surfaced as a silent unreadable tier rather than the 403 the item expected, because vzdump writes through a root path. - CONTEXT: S-21 (an empty listing cannot distinguish forbidden from newborn; the measured trap that an ungranted path answers with INHERITED privileges) and S-22 (the Scenario-F arm must finish the job). - STATUS: rewritten for the operator, back to one screen.	2026-08-03 19:02:25 +02:00
admin	81d4438630	docs: R-189, R-188 and R-186 closed (agent v0.122.0); R-185 stays open ... - OPEN-ITEMS: all three rows closed with their measurements. R-189 records the migration consequence seen live (a pre-R-189 record has no tier and is not reportable, so upgrading does not retroactively make an old proof visible). R-188 records both directions demonstrated and the fixture cleaned up. R-186 carries the before/after sha values including the CGO_ENABLED discrepancy. The priority list states explicitly that R-185 is the one left open from that group and that this session did not touch it. - capability map: the restore-proof row now says the evidence path itself had a gap — the proof reached the hub only because no restart intervened — and what closed it. - CONTEXT: S-19 (a proof is durable and reportable, a failure is neither, and why) and S-20 (the release ordering, what each step protects, and how a binary is verified). - STATUS: rewritten for the operator, back to one screen.	2026-08-03 16:46:49 +02:00
admin	c9a3e48b21	R-189: a passing restore-test can be invisible to the hub — observed live ... The agent reports restore_tests[] from an IN-MEMORY store ('lost on restart; the cadence re-populates'), and the hub report interval is 900s. A restart inside that window drops the result. That used to self-heal within 24h because the next cadence re-tested the tier; under per-archive due-ness the agent will NOT re-test a proven archive, so the hub can stay ignorant until the next archive generation — a week on the offsite tier. Observed today: the 15:25:14 PASS on felhom-pbs (a real 14.5 GB offsite restore) reached no host-report at all, because the agent was restarted 2m43s later for the v0.121.1 deploy. The persisted proof already exists — RestoreTestState.Snapshot() even carries the comment 'for the host-report gauge' and has NO production caller: a seam built and never wired, and an invariant asserted in a comment with no test, in one method.	2026-08-03 15:35:17 +02:00
admin	0f8f084817	R-188: a correct agent release emails a CI failure ~50% of the time ... on: [push] fires the gates workflow on the TAG push too, and release-agent.sh pushes the tag before publishing (deliberately). CI can therefore run the published-versions gate inside the window where the tag exists and the package does not, and correctly reports 'every released agent version must be INSTALLABLE' for a release that completes seconds later. Measured across two releases in one session: v0.121.0 -> runs 12/13 (success, failure) on the same sha; v0.121.1 -> runs 17/18 (failure, success) on the same sha; and one pair both green — a race, not a rule. It matters because R-168 made CI email on failure so a red gate cannot be missed. A signal that cries wolf on every second correct release is how that mail becomes something you archive unread.	2026-08-03 15:29:59 +02:00
admin	c102832892	docs: R-86 closed and proven live; ep0 recorded as protected; R-185/186/187 filed ... - OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal reading is never true on a daily tier); R-87 re-ranked UP because R-86 built most of what it waited for; R-185 (the agent cannot list demo-felhom's host backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's sha is not reproducible from its tag), R-187 (R-115's publish leg had never actually run) filed. R-184 was the highest ID in use. - ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row itself proposed — which would have been the never-fires version. - 07-backup-architecture: new contract section — restore-testing is per ARCHIVE GENERATION, with the trap and what did not change (S-1). - 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on the 635 s due-triggered offsite run, with the restart and teardown evidence. - CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines). Numbered 17/18 because S-14 and S-15 were already duplicated in the file. - STATUS: rewritten for the operator, trimmed back to one screen.	2026-08-03 15:28:32 +02:00
admin	323f45a5ef	hub v0.91.0 — the staleness window learns each tier's own rhythm (R-86 Part 2) ... Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86 removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before its next proof: it sat ON the line, so any ordinary delay tipped it into a nightly alarm about a working system. restoreProvenWindow(tier, observed, ok): - the tier's own archive interval, OBSERVED from reports the hub already holds (pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4) - x4 generations = the same tolerance the flat constant expressed - floored at 7d (never tighter than before), capped at 12d (strictly inside the 2-week offsite retention) - falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the backup-freshness checker already uses) when history is too short to observe one; falling back to the FLOOR would recreate the false alarm on a fresh box Kept: absence is UNKNOWN until the anchored window passes; the signal stays edge-triggered; failed and stale remain distinct events. Every reason string now states the window it was judged against (R-100's corollary). Also backfills the missing v0.90.1 CHANGELOG entry (deployed since f21e7ca), and records the operator's 2026-08-03 ruling that ep0 is Tier 2 / protected.	2026-08-03 15:03:35 +02:00
admin	e34b614e5b	docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected ... R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The hub's notification_log for the run reads: two per-app failures RECORDED, one digest SENT naming both, and the customer channel SKIPPED with operator_only. Against the measured previous behaviour — two failures, one email naming one app, one leaving no trace anywhere. Scenario D proved itself on an event I had not planned: disk_critical alarmed on two filesystems, the second was collapsed by the cooldown, and that collapse is now visible WITH ITS KEY. Yesterday it would have left nothing at all. A gap the spec did not anticipate is recorded with its fix: the per-app event also fires from the periodic sweep, outside any run, so making it record-only would have created a NEW silence. The sweep emits a digest too, with no run_id, so it stays under the ordinary hourly cooldown. ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile SURVIVED the resize and is active (checked, because a resize is a stop/start). The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere. Five documents corrected — three of which the task's list did not name, found by searching. Two audit/evidence documents ANNOTATED, body untouched: they record what was true when written and that is their value. R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable, not unbounded — the original OOM was a 14.46 GB restore — so the restore-test cadence should still be paced, just not by fear of the endpoint. target-selection.md's "D-d did not name ep0 either way" is deliberately left standing. It is the operator's question, not CC's. STATUS.md 127 -> 83 lines, items rather than sentences.	2026-08-03 14:00:54 +02:00
admin	b0b269b28d	docs: R-110 + R-115 closed, R-182 re-scoped by measurement, R-183/R-184 filed ... R-182's direction REVERSED by Part 0's measurement. Filed yesterday as "the reserve re-alerts on every status refresh" — too many alerts, seen at the sending end. Measured at the receiving end: 9 events received today, 2 operator emails sent. When two apps are refused in the same second the operator is told about ONE; the other is dropped before LogNotification, so it leaves no row on any channel and cannot be audited. The operator cooldown key is customerID:eventType(+tier) and the capture-failed event carries `app` but no `tier`, so the key has no app identifier. Same failure mode as R-97a, in a second event type that never opted into the narrow fix. Nothing changed — Part 0 was investigation only. Correction owed: yesterday's report said "one recovery_unit_capture_failed per app, HTTP 200". True of what the CONTROLLER pushed; a reader would take it as "the operator was told about each app", which is false. R-110 CLOSED (installer v1.23.0). Both channels moved. The spec's mechanism for channel 2 rested on a factual error — the run-time fetches are sixteen, not nine, and come from felhom-agent, not this repo — so no tag here could cover them; pinned to the agent version being installed instead, on the operator's ruling. Channel 3 needed no change: the URL never carried a ref, so no hub change and no hub bump. R-115 CLOSED. release-agent.sh builds, tags, publishes and verifies by an independent download; check-published-versions.py refuses a tag with no package; CI now runs the full gate set so it actually runs. R-183 NEW+CLOSED: a fresh install fetched the vouched agent binary and its sixteen config files from two different refs, and nothing compared them. R-184 NEW: nothing stops the hub vouching a version that was never released. The R-115 gate cannot see it — measured, the hub manifest and Gitea's package listing are both 401 anonymously. capability map: new PROVEN-LIVE row for the published installer channel. STATUS.md 138 -> 127 lines.	2026-08-03 12:44:08 +02:00
admin	8360f940bf	docs: seventh row in the shipped-guarantees table (R-181), versioned workspace CLAUDE.md ... Syncs documentation/runbooks/workspace-CLAUDE.md with the workspace root file. Row 7: the B2 refusal claimed the previous unit was untouched; nothing-deleted held, untouched was measured false. Also records WHY it survived review: it passed a full green suite AND three of its own red-proofs, because every one of them asserted the mechanism inside captureAllRecoveryUnits and none asserted the consequence across the whole backup run. The test that would have caught it is the one the fix ships — fingerprint the tree before and after, and compare.	2026-08-03 11:36:47 +02:00
admin	fb652024ea	docs: R-181 closed, R-156 closed, R-110 + R-115 rulings recorded, R-182 filed ... R-181 CLOSED (controller v0.193.0 + v0.193.1) and proven live on demo-hp for BOTH reserve terms. The reserve is now a per-app, per-run ADMISSION decision taken before the app's first write and covering all three write legs, and it gained a size term. The refusal's wording was not weakened; the behaviour moved so it became true, verified by sha256 tree fingerprint. R-156 CLOSED — papra's template mounts the app's own data root. Precondition re-measured rather than inherited (both boxes were wiped today). Part 4, documentation only, nothing built: - R-110 WAITING-ON-OPERATOR -> READY. Ruling: option (b), the installer's publish channel moves to a TAG. Recorded with the condition that decides whether it works at all — it must cover BOTH the /scripts/ git-sync and the nine files the installer fetches from raw/branch/main. - R-115 WAITING-ON-OPERATOR -> READY. Ruling: mechanism (b), a build-side gate refusing to deploy or vouch an unpublished version. The third instance (agent v0.120.0) would have silently downgraded both demo boxes while succeeding. R-182 NEW: the periodic status refresh has no admission scope, so a refused app re-alerts on every poll (measured: a second alert pair 13s after the run's). Pre-existing in v0.192.0; deliberately not fixed in the R-181 task. capability map: the local-backup row moves to PROVEN-LIVE in BOTH halves. ROADMAP: R-165 collapses to CLOSED; R-181 collapsed into it. 07-backup-architecture.md: the reserve's contract stated as what the code provides (S-1 — an architectural contract changed in the same session). STATUS.md trimmed 150 -> 111 lines, "What's broken" no longer holds shipped work, and the stale "After:" line (pointing at work that shipped on 2 August) is fixed.	2026-08-03 11:36:16 +02:00
admin	aa62449694	R-178 CLOSED: both demo boxes reinstalled from the merged golden and proven ... Two boxes, two DIFFERENT supply paths, so the session proved the disk shape and the delivery route rather than one of them twice. demo-hp (layout proof, --golden <local volid>): mp0 at /var/lib/felhom, backup=1, 70G, no mp1; /var/lib/docker and /mnt/sys_drive both real mounts of its subdirectories via fstab; one df figure and one device id (64519) on all three paths; reboots 3/3 with the binds surviving each. demo-felhom (pipeline proof, --force-gitea-golden): fetch_verify succeeding against the vouched manifest for BOTH artifacts -- 'verified sha256 54e2a4c431daf580... matches the hub manifest' for the golden, a7763d31... for the agent. 250G single volume, grep -c '^mp1:' = 0, reboots 3/3. Journey proven on both, endpoint-level: claim -> deploy -> back up -> restore, with a planted marker returning byte-identical on each box. Ceiling measured gone: 65 GiB and 233 GiB available to a recovery unit, against 19 and 45. R-165 -> IMPLEMENTED, not PROVEN-LIVE, on the operator's ruling. B2, which that row records as the bulkhead's replacement, fired live for the first time and does refuse per app, delete nothing and alert -- but it is checked only in captureAllRecoveryUnits while runVolumeDumps writes the bulk unguarded, and its 'the previous unit is untouched' claim was measured false (182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). -> R-181. New: R-179 (uninstall leaves NAS network-storage units), R-180 (--archive-storage not cross-checked against the ACL grant; 403 at step 8/8 after root@pam is rotated), R-181. Third instance of R-115 recorded (agent 0.120.0 unpublished). No code written, no version bumps -- this was a runbook.	2026-08-03 09:34:15 +02:00
admin	14d8c00781	docs: R-165 merge built and proven at the bake; R-163 + R-175 closed, R-178 filed ... 07-backup-architecture.md gains §7.5.1 (S-1: the contract changed in the same session): the ceiling §7.5 describes no longer exists for a box built from golden >= 0.192.0, the bulkhead's replacement is recorded, and R-175 is FIXED here rather than left standing — the bound is restated as a function of mp1 and scoped to split-layout boxes, naming all three real shapes. Capability map: new row as IMPLEMENTED, deliberately NOT proven-live, with the missing leg named — no box has been reinstalled from the golden, and "the golden baked" is not "a box built from it works". R-163 CLOSED: the ceiling it recorded stops existing. R-176(a) answered by P1; (b) WITHDRAWN, since every node is reinstalled rather than migrated. R-178 filed for the reinstalls, which were not done this session. CONTEXT S-13 (the variant chosen on measurement; pruning rejected with its reason) and S-14 (prove first, then vouch — the golden is published but deliberately unvouched, because vouching is what makes a fresh install pick up a layout no box has been proven from). STATUS: plain-language section; both operator questions now answered, so the waiting-on-you item is cleared. Two older entries trimmed so the page did not grow.	2026-08-03 07:16:15 +02:00
admin	7406ac7bbf	audits: R-165 Phase 0 — P1 and P2 measured, nothing changed ... P1 PASS: a pre-merge archive (mp0+mp1, confirmed from its own vzdump log) restore-tests clean on demo-hp with mount_parity ok in 84s. mountParity was not touched. Limit stated: run with the current agent because the merged one does not exist until after the STOP; the comparison is archive-vs-its-own- restore and never consults the host layout, so it carries provided Part 2 honours its constraint not to touch the restore path. Re-run after Part 2. P2: all three probed variants are mechanically clean — both paths writable, ONE df figure, dockerd 3/3 reboots, /mnt propagation, and a container's statfs(/) reporting the merged volume. The task's flagged ordering risk for V-b did not materialise. They are separated by SCOPING instead: V-a container sees /mnt = 8.0K but customer data sits inside Docker's data-root, so clearing /var/lib/docker destroys every local unit V-b container sees Docker's ENTIRE data-root under /mnt (17.9M on an empty box), making the bootstrap's own scoping comment false V-c neutral mount at /var/lib/felhom, both paths binds — breaks neither V-c was probed because the measurements showed each named variant violates a different documented invariant. It is offered as a measured option for the operator, NOT adopted. Teardown all three layers: 9401/9402/9403 destroyed, 5.19 GB returned, and the hub registers verified unchanged (5 customers, 4 hosts).	2026-08-03 06:34:26 +02:00
admin	41dbecb264	docs: R-167 + R-158 CLOSED, R-165 SPIKED, R-174..R-177 filed ... R-167/R-158 shipped and proven live (controller v0.191.x, hub v0.89.0): two new capability-map rows PROVEN-LIVE with live citations, and 07-backup-architecture.md §7.5's closing claim "nothing warns when an app crosses the line" is now false and rewritten (S-1: an architectural contract changed in the same session). §7.5 also gains the caveat that its size bound is ONE BOX'S, not the fleet's. Part 3 SPIKE (audits/SPIKE-r165-mp1-merge-2026-08-02.md): M1-M5 measured, NO layout touched. Three findings the merge session must not re-derive: "the layout" is not one thing (200G/50G vs 50G/20G vs 16G/8G); mp1 is a BULKHEAD and not only a ceiling, so after the merge an overflow reaches /var/lib/docker; the golden fails closed on the split in four places. D-a's condition (1) is currently SATISFIED — no external box is in the hub's register, and both demo boxes are Tier 0 and reinstallable. Recommendation given, choice NOT made — it ends at the operator's ruling. CONTEXT.md S-11 (D-c's routing, and why R-158's own backup_failed proposal was overruled) and S-12 (the monitoring landed BEFORE the merge). STATUS.md gains the plain-language section and the merge decision, with two older entries trimmed so the page did not grow. New rows R-174 (closed same session), R-175, R-176, R-177; each ID grepped free before minting.	2026-08-02 23:56:16 +02:00
admin	8ef92a3fa7	docs: R-172 CLOSED (hub v0.88.0), R-173 filed, session report ... R-172's root cause was not tuning — the WAL/busy_timeout pragmas had never been applied, because the DSN used mattn/go-sqlite3 syntax against modernc.org/sqlite, which ignores unknown parameters without an error. Recorded that way so nobody re-reads it as "SQLite was slow". R-173 NEW: while establishing who copies hub.db for the WAL change, found pvc/hub-data labelled recurring-job-group.longhorn.io/default: disabled, with backup-daily and backup-weekly the only recurring jobs and both on the default group — so the hub database has no volume-level backup, and it holds every box's break-glass root password plus the escrow custody records. Filed, not fixed: whether the exclusion is deliberate is an operator question. The session report is REPORT-r172-hub-wal.md, not REPORT.md, per the parallel-session rule — REPORT.md belongs to the controller session that ran immediately before this one. It also records, plainly, that a 60-concurrent load test I ran OOM-killed the hub pod three times against a 256Mi limit. Not the WAL change, and not a test I should have run against a Tier-2 box; the unit tests already proved the property.	2026-08-02 21:22:14 +02:00
admin	0fc54e0122	hub v0.88.0 — the WAL that never was (R-172) ... store.New opened the DB with `?_journal_mode=WAL&_busy_timeout=5000`, which is mattn/go-sqlite3 syntax. The driver is modernc.org/sqlite, whose applyQueryParams reads only _pragma/_time_format/_time_integer_format/_txlock/_inttotime and IGNORES anything else WITHOUT AN ERROR. So the hub ran in rollback-journal mode with busy_timeout=0 for its entire life while its own source said otherwise. Surfaced as a false HOST STALE banner: in rollback-journal mode a reader excludes a writer, so rendering an operator page blocks a host report; the hub 500s, the agent waits its full 15-minute interval without retrying, and staleness fires at 30 minutes — two collisions is a false alarm plus an operator email. 13 collisions in one pod lifetime; the alarm fired twice on 2026-08-02 for a host that was up two days and reconciling throughout. The observable that proved it: a 128 MB /data/hub.db with no -wal/-shm beside it while the DB was open. Fix: ?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate. _txlock=immediate is not optional — database/sql's Begin() is DEFERRED, so a read-then-write tx must upgrade its lock and a failed upgrade is SQLITE_BUSY_SNAPSHOT, which busy_timeout does NOT retry; this store has 10+ db.Begin() sites and they are all write paths. Every test asserts what the DATABASE reports, never the DSN string — a string test would have passed for the whole life of the bug. Red-proof: restoring the shipped DSN reproduces journal_mode="delete", the missing -wal, and the live "database is locked (5) (SQLITE_BUSY)". Operational consequence handled: a WAL DB cannot be copied by taking hub.db alone — a bare `cat` opens cleanly and silently omits the newest writes. The break-glass retrieval in operations/nodes.md used exactly that; it and the recovery-inventory note are now WAL-aware.	2026-08-02 21:06:29 +02:00
admin	2c35c4204a	OPEN-ITEMS: R-172 — false host_stale when SQLite refuses two consecutive host reports ... The hub's /data/hub.db (128 MB) is in rollback-journal mode, not WAL, so a UI render can block a report write; the hub returns 500 on SQLITE_BUSY without retrying, and the agent waits its full 15-minute interval rather than retrying. Staleness fires at 30 minutes, so two consecutive collisions produce a false host_stale and an operator email for a healthy host. Observed twice on 2026-08-02 while the agent was up 2 days and reconciling throughout. Pre-existing: 13 collisions in one pod lifetime, first ~3h before that day's controller work, though a burst of restarts amplifies it.	2026-08-02 20:48:04 +02:00
admin	ad28699761	docs: R-157 A / R-170 / R-171 closed — boot recovery finished ... Controller v0.190.0. Docs only here; no hub change, no hub version bump. - audits/DIAG-bootrecon-drive-absent-2026-08-02.md — NEW. The Part 0 diagnosis, including the run that produced a FALSE NEGATIVE and the mechanism behind it (the agent re-binds an unmounted drive within ~60s, so the drive gate's startup reconcile restarted the apps one second before the sweep looked). Records that the write hazard was blocked only by an ACCIDENTAL filesystem permission that no code owns and no test pins. - architecture/02 §0a — the boot-recovery contract (S-1): both gates read desired state; the sweep observes a SETTLED fleet and each sample must refresh first; nothing is started without asking, fail-safe. Plus the durable warning: Manager.StartStack has no gate of its own. - 00-capability-map — the boot-recovery row, with the repeat count cited per N.5 (6 of 6 hard resets) rather than a bare PROVEN-LIVE. - OPEN-ITEMS / ROADMAP — R-157 CLOSED (both mechanisms), R-170 CLOSED, R-171 NEW and closed the same session, marked a regression from v0.189.0. - STATUS.md — the power-cut line moved from "What's broken" to "What works right now" with its repeat count; one dated bullet in the change log. - CONTEXT.md S-13 — the lessons worth carrying: "it didn't happen this time" is not a disproof; widening a window makes previously-unreachable overlaps reachable; and a settle detector is only as good as the freshness of what it samples — the fix's own defect, found live rather than by review.	2026-08-02 20:38:21 +02:00
admin	5c97fbc397	docs: R-166 SHIPPED — the desired/in-flight/observed split (D-b) ... Controller v0.189.0 implements operator decision D-b. Docs only here; no hub change and no hub version bump. - architecture/02-controller-module-map.md §0a — NEW, and it is the S-1 contract: desired (app.yaml) / in-flight (own marker file) / observed (not persisted), with the rule that ties them — never derive one from another. Absent desired state means UNKNOWN, never "running". One file, one writer. D-b's binding safety rule quoted verbatim. - 00-capability-map.md — the boot-recovery row now rests on a recorded signal, with the three live flows from 9201. The interrupted-operation half is marked IMPLEMENTED, not PROVEN-LIVE: nobody killed the controller mid-backup on metal. - OPEN-ITEMS/ROADMAP — R-166 SHIPPED with both blocking facts and their answers; R-157 mechanism B CLOSED and A restated as the whole item; R-170 NEW (the drive-backed boot gate still infers a Stop from a container count). - STATUS.md — the "an app can stay switched off and nothing says so" line rewritten to what is actually left: timing. - CLAUDE.md — end-of-session checklist gains: confirm your own last push's CI run went green, BY RUN ID. The failure email is a push signal; this is the pull check. - CONTEXT.md S-12 — the rulings, and the two lessons worth carrying: a test that constructs the thing it should prove the caller constructs is hollow (its red-proof will say so), and a field-by-field struct rebuild in a save path is a defect on sight.	2026-08-02 18:58:27 +02:00
admin	c718aad1bc	docs: R-168 SHIPPED, R-29 CLOSED on the demonstrated alarm, R-169 minted ... SPIKE-ci-runner-2026-08-02.md: all six probes with method, measurement and ruling; none STOPped. P2 (stock image has git but no python3) and P6 (a runner that loses its state re-registers and orphans the old record) changed the design; P5 (a failed run signals NOTHING) is why the alarm exists at all. R-168 SHIPPED with its evidence. R-29 CLOSED — on the demonstrated alarm and not on a green run, as required: the class it opened is answered at both ends, the hook refusing locally and CI catching a --no-verify bypass and emailing. R-161 noted: its automatic half now exists for the STATIC gate, while its original scope, the runtime gate, is deliberately still not automatic and should stay that way. NEW R-169 (grep established R-168 was the highest in use): CI can only report, because there is no gate in the road. Making it blocking needs branch protection plus a PR workflow, both of which change how the operator works — so it is theirs to decide, and the row states the cost honestly rather than recommending it. CONTEXT gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector that tells no one is not finished, plus the curl and Cloudflare-1010 traps), S-10 (the runner is unprivileged because DooPlex is Tier 2), S-11 (CI reproduces the sibling layout). CLAUDE.md gains the rule earned by red-proofing: a go test -run pattern that matches no test prints ok and exits 0, and an instrument that can silently drop results is not a measurement.	2026-08-02 16:35:34 +02:00
admin	4707be755c	docs: R-94 closed, R-29 leg (a) closed + leg (b) half, R-168 minted ... hub/CHANGELOG v0.87.0 + scripts/CHANGELOG gate-enforcement entry. CONTEXT gains S-6 (the hub renders no host-install version and the gate pins its absence) and S-7 (gates run from one entry point per repo; reuse_refs_check was fixed rather than the REUSE.md convention, with both rejected alternatives recorded). OPEN-ITEMS: R-94 CLOSED all three legs, leg (a) by DELETION with its reason; R-29 leg (a) CLOSED and leg (b) HALF-SHIPPED with the census result written into the row (13 gates; every gate a CLAUDE.md names was green, two of the four unnamed were red); R-161 gains its successor pointer. NEW R-168 (grep established R-167 was the highest in use): Gitea Actions runner — measured 2026-08-02 as Gitea 1.26.2, Actions enabled on all four repos, 0 runners, 0 workflow runs, 0 branch protections, and the consequence that trunk-based direct-to-main pushes leave no merge for a status check to gate, so CI here can detect but not block. BLOCKED on a spike over host-mode vs privileged DinD on DooPlex and whether the workflow can avoid JavaScript actions. ROADMAP: R-94 collapsed to its one-liner, R-29 updated, R-168 added.	2026-08-02 15:28:31 +02:00
admin	e994bf35d2	STATUS.md: a plain-language operator page, and today's four decisions recorded ... Documentation only — no code, no box, no build. STATUS.md (repo root, 652 words / 67 lines): what works · what's broken · what we're working on · waiting on you · changed since. A VIEW of OPEN-ITEMS.md, holding nothing of its own; not CONTEXT.md, and both files now say why they stay separate. No R-n is the subject of a sentence — identifiers are bracketed pointers only. CONTEXT.md S-5 records the four operator decisions taken 2026-08-02 (D-a … D-d), none of them implemented: D-a merge mp1 into mp0 rather than resize it — before any external install, and D-c ships in the same step → R-165 D-b desired/observed app state in its own store, with the state-store safety rule verbatim → R-166 (BLOCKED) D-c customer fill warning + operator backup-failure alert → R-167 D-d only DooPlex and Peti's box are protected → target-selection.md R-163 RE-FRAMED, not closed: the sizing question is withdrawn rather than answered; the row survives as the record of the constraint until R-165 lands. R-156's papra referral RESOLVED — deployed nowhere, so the template fix strands nothing; the docker ps evidence is recorded with its provenance and its scope limit. target-selection.md: two protected machines, everything else disposable. ep0 is no longer Tier 2 but is not scratch (it holds the only off-premises copy of real customer data) — flagged for explicit operator confirmation. The demo-box backup-target fence drops from prohibition to stated cost, because D-d spends that reference anyway. CLAUDE.md gains an End-of-session checklist carrying the STATUS.md maintenance rule and "a finding goes in OPEN-ITEMS.md first".	2026-08-02 14:20:29 +02:00
admin	260a8f6e58	register: R-161 ruled and shipped at reduced scope; re-ranked ... The operator ruled on R-161 and the runner shipped in app-catalog-felhom.eu (fd7747d), so the row moves from BLOCKED-needs-a-ruling to REDUCED SCOPE - open. Both obvious enforcement points were rejected for measured reasons, and the row now records them rather than leaving the rejection implicit. Controller-side at template load: rejected because such a check can only read the file, and a static audit of all 53 templates reports the catalog clean INCLUDING papra - it would pass on the exact defect it exists to catch, the property being decidable only at runtime. CI: rejected for now, neither repo has any and there are no users yet. Shipped instead: scripts/catalog_gates.py, one entry point over all three gates, non-zero exit on any failure, mandated in the catalog's CLAUDE.md the way site_gates.py is. The rationale is recorded because it is the transferable part - of this project's gates, the only ones that ever get run are those with a single entry point named in a CLAUDE.md; site_gates.py is run and R-29's three orphans are named nowhere and have stopped nothing. What stays open is only the automatic half, which is sufficient while ONE person touches templates - revisit when a second does. Re-ranked accordingly: R-161 drops from 2nd to 7th, and R-156 is promoted to 2nd, since R-161 was ranked high precisely because nothing ran the gate and that is no longer true. The de-ranking is recorded inline with its reason, matching how R-94's de-ranking is recorded, so a later reader sees a decision rather than drift.	2026-08-02 14:05:13 +02:00
admin	b06ea9c877	register: file R-156..R-164 in one pass, ranked; and record what mp1 is actually for ... Nine rows into OPEN-ITEMS.md and ROADMAP.md, matching each file's column shape. R-156 and R-157 had lived only in audit documents - the identical "minted in a spike doc and never carried across" failure the register already records for R-153/R-154/R-155, caught by the catalog sweep's own section 8.0 while it was happening. R-158 was minted by a second session the same day for an unrelated finding, which is why the sweep's proposals were renumbered R-159..R-162 at filing time. All nine IDs verified free in BOTH backlog files before use. Part 0 settled the question the sizing item depended on, by reading: mp1 is RETENTION, not staging, and neither of the two framings was right. A unit is the KEPT copy on the app's OWN drive (backup.go:245-255); for an app with no HDD_PATH the namespace falls back to the system SSD - "the SSD-only system-data fallback" (appbackup/paths.go:26-27). There is no post-copy deletion: the only prune is F5 residue-on-old-drives when an app MOVES (backup.go:1053-1112). So mp1 retains the units of driveless apps only - not every app, but not transient either. Confirmed against the spike: sys_drive held exactly the four driveless apps and not calibre-web, which had a drive and was still backed up. A unit is volume tars + DB dumps only, never mp8 userdata (recovery_unit.go:20-25), so a 1 TB photo library can never overflow one. And mp1 gates the WHOLE chain, not just Tier 1: Tier-2 mirrors the unit "(always)" from RecoveryUnitPath (tier2.go:302,368) and Tier-3 carries it, so a unit that cannot be written leaves both with nothing to copy. Part 2 fired on both triggers - retention, and the fallback undocumented - so 07-backup-architecture.md gains section 7.5. Section 6.1 said a unit lives "on the app's own drive", which is true and was the whole story only for drive-resident apps; the no-drive case was undocumented, as was the sizing constraint. 7.5 records the mp0-50G-vs-mp1-20G mismatch, the measured ratios (DB app up to ~2x, 21.1GB -> 40.2GB; file-only 1.00x), and the bound this puts on D5's Lane-1 independence: restorable from the drive alone only while the unit still fits - about 19 GB file-only, about 10 GB DB-backed. No number proposed; the ratio is the operator's ruling (R-163). R-159/R-160 marked SHIPPED only after verifying the template changes are in app-catalog origin/main, and R-156's gate likewise (check-volume-persistence.py present). papra is NOT fixed - referred - so R-156 stays open on that one app. Ranked, with one line of reasoning each: R-157 first (an app can stay down indefinitely with mechanism B silent on every channel), then R-161 (the gate exists and nothing runs it, which is why R-156's class recurs - R-29's record is three orphaned gates and one enforced), R-156, R-163, R-158, R-164, R-162.	2026-08-02 12:33:12 +02:00
admin	482af37b7d	Campaign 10 closeout Part 2: teardown complete — five layers, each verified gone ... Evidence-survival check FIRST: HEAD == origin/main == 7efb7a5, and every artefact confirmed present in origin/main with git cat-file, including all 27 evidence files. Nothing died with the rig. 1. VM 311 destroyed by exact VMID. qm list now shows only drill-r50; 311.conf does not exist; images/311 gone. 126 GB returned - nvme 133G used / 758G avail -> 6.7G / 884G. A first attempt stopped the VM but its destroy never ran, killed by an unescaped paren in my own echo; caught by checking qm list rather than the exit code. 2. c10-scratch removed by name. storage.cfg 5 -> 4 entries. felhom-backup shares the same path and is intact, config unchanged. 3. PBS on DooPlex (Tier 2). Identity confirmed first - the target carried the comment this campaign set at creation. Datastore felhom-c10, user c10@pbs, token !box, both ACLs and the on-disk chunkstore all gone; felhom-spike, felhom@pbs, felhom@pbs!n100 and /mnt/5_hdd/backup/demo-felhom all intact. Only the campaign's entries disappeared. "user delete" is not a PBS subcommand - it printed usage and the user survived rc=0; caught by re-reading the user list. 4. Restic subaccount 281530 deleted after asserting home==felhom-campaign10 and the description names Campaign 10 - the script aborts otherwise. 4 -> 3 subaccounts; demo-felhom, peti-felhom and demo-hp all present. 5. Hub customer c10-soak. The ONLINE gate REFUSED with HTTP 409 for 18 consecutive attempts, exactly as anticipated (customer_delete.go:98-102, :146). Predicted deletable at last-report 09:38:19 + 30 min = 10:08:19; measured 303 at 10:08:23. Verified positively: 0 customer rows, 0 host/appliance rows, GET /customers/c10-soak -> 404, and the hub logged the DELETE cascade COMPLETE with residue purged. No /appliances/discard was needed - the appliance record was bound and went with the cascade. drill-r50 verified INTACT (200, VM 300 present). All 12 credential files shredded and the directory removed. Register rows NOT written: app-catalog-felhom.eu has an uncommitted working tree including scripts/check-volume-persistence.py, so the parallel catalog session is still open and owns the register. R-156/R-157/R-158 rows and rankings remain owed, itemised in the doc - and that session's volume-persistence checker looks like R-156's fix shape, so the two should be reconciled before filing.	2026-08-02 10:11:21 +02:00
admin	7efb7a53d3	Campaign 10 closeout Part 1: Q1 lowers R-158's rank; Q2 clears ValidateDump and kills C2's gate ... Q1 - what the customer sees when a backup refuses for lack of space. The failure IS customer-visible: /backups renders "Adatmentés sikertelen" with a cross mark. It is absent from the dashboard, the launcher, the app detail page, and - the one worth fixing - from /backups/apps, the per-app page where you would naturally ask whether a given app is backed up. Point 5 measured across three runs: it retries, stays failed while constrained (marker persists, unit mtime unchanged at 07:30:50), and clears on recovery with a fresh unit at 07:37:38. /backups/apps reading "Utolsó: 3 perce" tracks the unit's REAL mtime, not the failed run, so it is honest about the age of the last good unit rather than claiming a fresh one. Explicitly NOT the R-156 family. So R-158 is a NOTIFICATION GAP, not a silent-failure defect, and ranks BELOW R-157 - whose mechanism B leaves a deployed app not running while deadapp reports "0 currently down", silent on every channel. Q2 - ValidateDump was right and no bad dumps are shipping. The live DB genuinely had zero accounts (only _prisma_migrations 129, cc_proof 82, instance_settings 1). An empty table proves nothing, so an account was SEEDED as the task required: the warning then stopped entirely and the dump provably contained the rows (c10acct 1, c10user 2; 102766 -> 103029 bytes). But that kills C2's proposed ordering. A fresh appliance legitimately has zero accounts, so gating on "accounts has rows" would block the backups of every new customer until someone registers. The validator's fact is right; its inference ("may predate the customer's data") is wrong - there was no data to predate. The chain is therefore longer: a sound predicate first (compare the dump against the LIVE db, per-table counts, not an absolute expectation), then warn->gate, then the tar-drop. Until then the DB volume tar stays load-bearing - not because dumps are bad, but because nothing can yet prove one is good. No new R-n; register grepped. Nothing fixed. Part 2 (teardown) follows.	2026-08-02 09:39:27 +02:00
admin	0afadbdeff	SPIKE: recovery-unit space — the ceiling is real on mp1, overflow is clean, but silent (R-158) ... Three headline answers. 1. The ceiling is REAL and on mp1 (/mnt/sys_drive), but its shape is a MISMATCH rather than a single number. A1: docker's data-root is a SEPARATE 50G volume (mp0) and every app volume resolves there, so app DBs are NOT on sys_drive - build-golden.sh:68's "like the Docker-data" reading is correct. A2: the recovery units ARE on sys_drive, which the golden ships at 20G. So a box permits 50 GB of live app data while capping local backup at 20 GB, and crossing that line is invisible until a backup fails. A3 rules out the lab-default explanation: --sysdata-grow defaults to 0 (main.go:178) and is not computed from the drive. demo-hp's REAL guest 9201 runs a bare --config ExecStart and shows mp0 50G / mp1 20G; agent.json has no sizing keys at all. A4, measured not read: restore extracts IN PLACE on the docker volume - sys_drive avail was 799.2M before and after a restore run under constraint - so the constrained mount is written only during backup. 2. Overflow behaves WELL. With sys_drive ballasted to 799 MB, backup refused per-app ("No space left on device"), other apps continued, status reported success=false, and the "last good dump preserved" claim VERIFIED byte-for-byte: size and md5 unchanged, tar valid end-to-end, no .tmp residue. Restoring that preserved unit under the same constraint returned correct data and claimed success honestly. Explicitly NOT the R-156 family. 3. But it is SILENT - R-158, filed. Zero events reached the hub. NotifyBackupFailed exists and the hub allowlists backup_failed, but the only production caller is the off-box/NAS leg (main.go:659); the backup manager has tier2/offbox/offbox-enlarge notify seams and none for the local recovery-unit capture. This is R-97's shipped defect exactly one tier over, and the fifth instance of "seam built but never wired" - a pattern the codebase names in its own R-97 wiring test. Sizing rule corrected: unit ~= volume-tar bytes + logical dump bytes, not a constant 1.90x. Measured C1: file-only apps are 1.00x (homebox 2305->2305 MB, no db-dumps dir at all), and the SAME DB app with an empty DB is also 1.00x. So a 20G sys_drive holds ~19 GB file-only or ~10 GB DB-backed. That is the bound on D5's Lane-1 independence. C2: both representations are used for a reason stated in code (F17 - the dump is authoritative and WINS over the tar; R-47 - replayed with only the DB service up). The dump is single-database pg_dump --no-owner, so a fresh initdb plus the dump is logically sufficient and the tar is a PHYSICAL FALLBACK. Dropping it would halve DB-app units and also close the D5/R-127(b) password trap (restored PGDATA makes postgres skip initdb and ignore POSTGRES_PASSWORD) - but only after ValidateDump is promoted from a warning to a gate, since it currently WARNS on a dump whose accounts table has no rows. In its present form the tar is load-bearing. No production code, no template change. Teardown still owed and itemised.	2026-08-02 09:07:09 +02:00
admin	5f35aa0346	Campaign 10: M-band RTO measured — RTO ~= 40s + 26.9s/GB, and a capacity ceiling that matters more ... The S figures (66 MB -> 42.0s, two passes agreeing to 0.6s) had a spread tight enough to prove fixed work dominates, which is exactly why they said nothing about M. Second point taken 327x larger, same app, same method: clock from restore request to the app serving the correct discriminator. rallly's postgres volume grown 66 MB -> 21.1 GB (200k rows, STORAGE EXTERNAL so TOAST cannot compress it into a fake number). Two reps: rep 1 backup 406.4s unit 41149 MB RTO 624.5s discriminator correct rep 2 backup 387.2s unit 41133 MB RTO 591.8s discriminator correct 327x the data cost 14.5x the time - strongly sub-linear: RTO ~= 40s + 26.9 s/GB backup ~= 29s + 17.4 s/GB 10 GB -> 5.2 min 20 GB -> 9.6 min (measured 10.1) 100 GB -> 46 min The fixed ~40s dominates below ~1.5 GB, which IS the S band and explains its tight clustering. The more consequential result is capacity. A DB-backed app's recovery unit is 1.90x its data (volume tar PLUS SQL dump): 21.1 GB produced a 40.2 GB unit. The default appliance ships /mnt/sys_drive at 20 GB, so the largest app that can hold a local Tier-1/2 recovery unit on a default box is about 10 GB - and that fills the volume. The M band does not fit on a default box at all; this test only reached 21 GB because sys_drive was first grown 20G -> 70G with the same operation the product performs via SysDataGrowGB. A tier-sizing decision, not a defect, but it is invisible until an app crosses it. Caveats stated in the doc: two points define a line but do not test linearity; the 1.90x is DB-app-specific and a file-only app should be nearer 1.0x (inferred, not measured); synthetic incompressible data; one app, one box.	2026-08-02 08:31:51 +02:00
admin	7ba7c2a271	Campaign 10: final results — 39 cycles, full atom set, R-156 + R-157, no leaks ... Phase B completed in three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, stopped deliberately - two violations were harness defects), run 2b (39 cycles, 12 of the brief's ~13 atom families). 1461 invariant checks. Depth reached 39 consecutive cycles, past the brief's "drift at the thirty-eighth", with c34-c39 clean on every invariant. I7 headline: 66 restores across both passes, 66 correct discriminators - never stale, never empty. I2/I3/I4/I5/I6/I10/I11 zero violations in either pass. I1-under-load 5/5: the target pulled WHILE a backup ran still produced backup_target_absent and a clean recovery. R-117's Q7 case holds - a filesystem aborted in place surfaces and the gate stops the app on the dead namespace. RTO Tier-1 rallly 66MB: run 1 median 42.0s, run 2b median 41.4s over 38 restores - two independent passes agreeing to 0.6s. S band's lower end only; nothing extrapolates to M or L. RPO not measured. Monotonic growth, 9457 samples of 19 metrics over 13.5h: NO leak. Controller and agent RSS flat, fds flat, no orphaned volumes/images/containers despite dozens of redeploys, kills, reboots and hard resets. Only curve with real slope is the agent journal at ~20MB/h, bounded by journald. Findings: R-156 (papra's data neither persisted nor backed up, reports healthy) and R-157 (bootrecon's start-once sweep, two mechanisms - the zero-container one is silent on every channel). Four suspicions investigated and DISPROVED, each recorded with what settled it.	2026-08-02 07:13:04 +02:00
admin	405a795e32	Campaign 10: RESOLVED — the backup_target_* silence was transient and self-recovered; SQLITE_BUSY drops are absorbed by retry ... Both halves of the disposition were run and neither survived as a finding. The I1/I1-pair violations cluster at cycles 31-33 and nowhere else across 39 cycles; c34-c39 are clean, so it recovered with no intervention. Final tally I1 37 PASS / 2 VIOLATION, I1-pair 36 PASS / 3 VIOLATION. On the quiesced box one slow detach with 4 minutes either side produced a perfect pair. And the alarming false-healthy (mentes bound=False while degraded=false) does not survive quiescence - I had been reading the two halves at different instants of a detach. No R-n. Separately cleared: the hub's SQLITE_BUSY event drops. 7 in 24h including one for the real customer demo-felhom, and the hub does return 500 with notification dispatch only after a successful save - so a lost event would be a lost alarm. But the controller retries 3 times and ZERO events exhausted their attempts; the 07:04:39 drop landed at 07:04:42. Nothing lost. Only cosmetic note: the ERROR line reads like data loss and is not.	2026-08-02 07:10:49 +02:00