287 Commits

Author	SHA1	Message	Date
admin	417df06f35	slice 3 docs: the ruling, the shipped mechanism, and four rows closed ... 09-update-architecture.md gains the fourth dated operator ruling (2026-09-06, Option 1) and its section 5 is rewritten from a proposed shape into the shipped one: the pin, the stored definition, the render table, the four writers, the startup ordering, and the trap this slice set for slice 2 - the live compose file is now the frozen one, so a badge comparing against it would answer Naprakesz on exactly the apps that are behind. 02-controller-module-map.md said 'copy compose + .felhom.yml'. That stopped being true today, so it is corrected, and the two sections describing the old seam now carry a banner saying they describe v0.234.0 and below - kept because every box under v0.235.0 still behaves that way and because they are the measured account of why it changed. R-447, R-441, R-438 and R-455 closed and compressed into CLOSED-ITEMS; R-458 opened for the .felhom.yml asymmetry, with what would settle it by measurement. Live evidence: two real catalog pushes travelling the real 15-minute cycle, both reverted, the tree byte-identical afterwards. The restart that used to take 18.3 seconds and pull a new image now takes 0.1 seconds and pulls nothing.	2026-09-06 10:37:33 +02:00
admin	bc47dd4ef9	v0.234.0: a known limitation written on 2026-09-02 was a defect by the next morning ... The operator looked at demo-felhom and found OpenGist - up 15 hours, running exactly the catalog pin, showing no badge at all. 09-update-architecture.md had recorded that as an accepted limitation the day before: 'the fleet view fills in gradually'. On a quiet box gradually means never, and a feature that fills itself in on an event nobody triggers is, on the quiet installations, not shipped. That limitation row is now struck with the reason kept. The living document gains slice 1b, the two admission rules of the backfill (it never overwrites, and it refuses to seed a partial observation because the badge reads a service-count mismatch as BEHIND), and the note that the same field having two writers with two different admission rules is deliberate. Live evidence added: all nine apps already had records by the time 0.234.0 was ready, so the natural fleet state could no longer exercise the new code - said plainly rather than papered over. The pre-0.233.0 shape was recreated on demo-hp by stripping two records; the backfill re-seeded exactly those two with digests matching independently-read ground truth and left the other seven alone. The refusal half was deliberately NOT staged live: it needs a degraded app, and manufacturing one risks the false-customer-email class that already cost 61 mails (R-330). Unit-tested with a red-proof, and recorded as unproven-live. R-457: a test that hardcodes a date and asserts an age derived from it is green only on the day it is written. Mine was, and it went red overnight. Six other files carry both a date literal and time.Now() - named as candidates, not accused.	2026-09-03 12:01:44 +02:00
admin	7941b0c159	R-455: the mirror base images are MEASURED byte-identical to Docker Hub ... The row carried 'KNOWN, not MEASURED' because the throttle was still in force. It cleared 40 minutes later and the check was run: docker pull from Hub answered 'Image is up to date' for both bases - Hub's own manifest resolved to the images already local, the ones v0.233.0 was built from - and the manifest bodies are identical between registries. This matters for the row's own decision: the mirror being a sound source is what makes 'sanction the mirror in build.sh' a real option next to 'get a Docker Hub login', rather than a hope.	2026-09-02 21:03:40 +02:00
admin	0705942783	the badge IS proven live, and the 'stale password' finding was mine, not the box's ... I reported that the vaulted dashboard password no longer worked on either demo box, and quoted the controller's own 'Failed login' as the discriminator. The password was fine. ~/.config/credentials quotes its values with SINGLE quotes and my sed stripped only double quotes, so the quote characters went out as part of the password. The operator corrected it in one line; one retry returned 302. The instrumentation lesson is the finding and R-453 now carries it: 'Failed login' separates wrong-password from wrong-Host-header, and that is ALL it separates. It cannot tell a wrong password from wrong password HANDLING, and I read it as if it could. This is the second time this file's quoting has produced a confident wrong verdict, so the fix is one shared extraction helper, not a resolution to be careful. With the session recovered, the badge is validated on live pages: Naprakesz twice on /stacks and on /apps/bookstack; NO badge at all on /apps/docmost (a deployed app with no record - absent is UNKNOWN, not current); and 'Frissites elerheto - 52 napja' on both surfaces, the age being real arithmetic on bentopdf's catalog_since. The behind state was staged by editing one compose tag, with no restart and no up -d, and reverted byte-identically (sha256 equal, diff empty, container never touched). Capability-map row upgraded to PROVEN-LIVE with the one unexercised badge state named. STATUS item 9 now needs nothing from the operator.	2026-09-02 20:47:53 +02:00
admin	e86cf42e0b	three more register rows: the observations gate refused a report that filed none ... R-454 five gofmt-unclean internal/web test files at the BASELINE, with no gate that would ever notice; R-455 DooPlex has no Docker Hub login and the unauthenticated ceiling now blocks a BUILD, not just the catalog's resolvability gate; R-456 a partly-dead stack is not a boot orphan and that rule exists in no document, so this session re-derived it by watching a repair not happen. All three were observations in felhom-controller/REPORT.md, which is overwritten every session. The controller's observations gate refused the push until each one either named a row or declared itself not a finding - which is exactly its job.	2026-09-02 20:36:52 +02:00
admin	6035dfcc3a	09-update-architecture.md: the update path finally has a document, and it is a living one ... R-438's document half. It records how an update works AS MEASURED, quotes the RestartStack comment that proves the restart half was CHOSEN (a design decision is not a defect), carries the three operator rulings of 2026-09-02, strikes the word 'rollback' (once a migration has run the old image will not start), states the target shape, and lists the seven slices with a status each. R-438 and R-440 amended and BOTH STAY OPEN: the mechanism is documented, not changed. Nothing closed, so CLOSED-ITEMS.md is untouched. Eight new register rows, 194 -> 202: R-446 (Naprakesz can be false for the 23 floating pins), R-447..R-451 (one per remaining slice, with a rank and an owner), R-452 (no gate enforces catalog_since - the runner fetches at --depth 1), and R-453 (the vaulted dashboard password is stale on BOTH demo boxes, which is what stopped the badge render from being validated live). Live evidence for slices 1 and 2 in documentation/tests/. The record is PROVEN LIVE through the boot reconciler on demo-hp - one entry per compose service, digests matching ground truth read independently. The badge RENDER is not, and the five attempts are listed rather than summarised.	2026-09-02 20:32:30 +02:00
admin	56c7e373a3	SPIKE: what an app update actually does, and which other paths do it too ... THE GATE IS ANSWERED: YES. compose up -d upgrades an app whose compose file has already moved, and the Restart button does it — 18.3s with a network pull when the target image is absent, 0.5s when present, against a negative control that did not even recreate the container. The boot reconciler does the same thing unattended when an app fails to come back (bootrecon.go:269 -> StartStack). AND ONE FEAR IS SMALLER THAN THE BRIEF CLAIMED: a plain power cut upgrades nothing. Docker restores the old containers and the reconciler logs 'no boot-orphaned apps (nothing to start)'. AND ONE IS BIGGER: app data CANNOT be rolled back. Once a migration has run, the old image refuses to start — Nextcloud: 'the version of the data (32.0.9.2) is higher than the docker image version (31.0.14.1) and downgrading is not supported'. 'Rollback' is the wrong word for this arc and is struck. Phases 0-6 all run, on demo-hp (Tier 0, disposable). Phase 6 run on operator confirmation. Peti's box was never contacted. No production code written in any repo: felhom-controller is at 960d29b0612c before and after, tree clean, and build/vet/test are green — run at the end to prove exactly that. Register: R-438/439/440 updated with live evidence; R-441..R-445 opened (restore-vs-sync conflict; remove_hdd_data inert with no paths.hdd_path, 128 MB left behind; update reports success over a broken app; no fleet fstrim; hub telemetry outlives the app). Capability map gains three measured rows. The mechanism is written down in 02-controller-module-map.md as MEASURED BEHAVIOUR, not [DESIGN] — the operator has not ruled. STATUS.md carries the one decision. Five claims in the brief are named as wrong, including two of my own method.	2026-09-01 21:35:32 +02:00
admin	f380c6d43c	REPORT: correct the register line count (620 -> 688, one measure) and record the CI run ids ... Both earlier commit messages carried a wrong line count - 621->688 and 688->700 - because I mixed wc -l with a Python line split and then carried the error forward. Corrected in the report rather than by rewriting history, and named there. CI verified by run id against head_sha: 500, 501, 502 all success. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM	2026-09-01 18:43:20 +02:00
admin	1a1b32b3dd	REPORT + R-437: the beta stopping line recorded, and the live trigger declined with its reason ... REPORT.md carries the deployed sentence quoted from the RUNNING binary (kubectl cp + byte grep, both controls), the stopping line as it reads in all three places, the enumerated deferred set, the two provider questions, the register census, and the ArgoCD verification. R-437 filed: the register compression sweep is OWED and was deliberately not run here. Measured first — 12 of 181 rows / ~25 KB of 316 KB (about 7%) carry a closed leading verdict — so it buys little and touches everything, and it is the exact operation that misfiled seven rows in August (R-378; the seventh, R-87, sat wrong for nine days, R-405). The row carries the scope so it can be picked up cold. The live alarm trigger was NOT run and the report says so in its own section rather than substituting quietly: this alarm only fires on a real fall in a real customer's snapshot count, so firing it means either deleting real backups or POSTing a falsified report claiming demo-hp lost its own. That would write a fabricated point into a customer's report history, move its latch and baseline, and mail the operator a second alarm about a real box hours after the first one already confused him. Covered instead by the deployed-bytes proof plus three red-proofed tests driving saveReport -> Check -> notify. What remains unproven is named: that the dispatcher delivers THIS wording to a mailbox. Register 688 -> 700 lines; 182 rows; open-state 170. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM	2026-09-01 18:41:50 +02:00
admin	10c223bdfe	DRILL R-95: the recovery route does not exist — stopped before the destructive phase ... The drill was to delete demo-hp's off-site history and get it back out of a Storage Box snapshot, filling row 10's blank RTO. Phase 1 found there is nothing to get it back from: no snapshot is reachable from a sub-account BY ANY NAME. Measured, read-only, no delete verb issued against any live store: - 777,600 exact names in the vendor form YYYY-MM-DDTHH-MM-SS, nine full days at second granularity, plus 126 alternative shapes -> ZERO hits. - The control is what makes that mean anything: the identical 600-name batch shape with one real path appended returned it, 6 of 6. - Structural cause: /home (u629488-sub3) is st_dev 0,82; /.zfs/snapshot is st_dev 0,276; /home/.zfs does not exist. A snapshot under /.zfs/snapshot belongs to a different dataset than the one holding felhom-repo. - Three tools agree with controls in the same run: SFTP, the port-23 shell, rsync --list-only. So yesterday's re-scope splits: clause (a) "the box cannot write into the snapshot area" STANDS and is re-confirmed; clause (b) "the rest is recoverable file by file" is NOT SUPPORTED. STOPPED before Phase 2 on the operator's ruling — with no recovery leg the deletion would have destroyed real history to buy only an alarm test that could not fire at the specified size. Store verified untouched at 69 snapshots. R-432 ANSWERED (negatively; its panel-read next step withdrawn as unnecessary). R-433 no snapshot reachable by any name — decides R-95's remedy and its rank. R-434 the drop alarm's text promises a file-by-file recovery that cannot be performed. R-435 the drop detector is blind to a single-app deletion (>50% of 69 needed, ~9 given). R-436 LEAD: the provider offers `rclone serve restic --stdio` and restic 0.14.0 speaks `rclone:` (measured, controlled) — real prevention may need no new machine, IF the vendor pins --append-only. Ask before building. 07 §8 row 10: text corrected, status NOT moved, RTO still blank. No code, no version bump, no image, no golden. REPORT.md's only copy of the R-331 report preserved as REPORT-r331-backup-card.md before overwrite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM	2026-09-01 16:55:50 +02:00
admin	ac6ac037bc	R-331 live: the Backup card now reads 67 snapshots, not 0 ... Hub 0.109.0 + controller 0.225.0 deployed and verified from the live objects, not from a rollout message (an ArgoCD "rolled out" can name the old image): argocd sync=Synced rev==HEAD, deploy and pod both on felhom-hub:0.109.0, both boxes on felhom-controller:0.225.0 (healthy). Fetched from the live hub at the exact URL the operator's browser requests: demo-hp 67 snapshots / 134.3 MB / last success 15h ago / 50 GB quota demo-felhom 10 snapshots / 132.5 KB / last success 15h ago / 50 GB quota Both read "Snapshots 0 / Repo Size 0 MB / Integrity Unknown" before this change. Integrity row grep count is 0 on both pages. Cross-checked against the SOURCE rather than against the card itself: the boxes' own settings.json hold snapshot_count 67 / 10 and repo_size_bytes 140829678 / 135635, and 135635/1024 = 132.5 KB, matching the rendered value. Stated rather than implied: the "never measured" branch was NOT verified live. Both boxes report stats_known:true, so exercising it would have meant falsifying a box's state. It is covered at render level by TestBackupCard_ThreeWayRuling and TestBackupCard_OldControllerDegradesToUnknownNotEmpty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM	2026-08-30 18:53:10 +02:00
admin	36f8630020	R-341 check taken, and the golden-currency bypass declared ... Two gates blocked the R-331 hub push. One is FIXED, one is BYPASSED, and the difference is stated rather than blurred. FIXED -- due-checks (R-341, 5 days overdue). The +7d measurement was TAKEN on ep0 rather than deferred again. Precondition passed: proxy still MainPID 551655, ps -o lstart= still 2026-08-18 09:51:04, NRestarts=0, so this is the same proxy generation as t0 (anchor is ps, not ActiveEnterTimestamp, which reads 03:54:54Z here -- R-346's trap). Result: fd = 17. Not 17 more -- seventeen TOTAL, exactly the documented baseline, against 405 at the first check. Socket histogram: one LISTEN, ESTAB 0, CLOSE-WAIT 0. The verdict is UNANSWERABLE, not "the upgrade fixed it". R-341 asks whether the PBS 4.2.5-1 upgrade changed the fd slope; inside this interval we removed the leak OURSELVES (R-344, agent 0.130.0, now live on both boxes). A slope of ~0 measures our fix, not the upgrade, and reading it the other way would credit a changelog that was read in advance and found to contain no such mechanism. The perturbation pre-registered for this window was Phase C at ~3%; the actual perturbation was the removal of the entire phenomenon. Row closed as moot. What it DOES establish is worth more than the original question: twelve days after the R-344 fix, same proxy generation, no restart to hide behind, ep0 sits at baseline with zero established connections. R-336's ~323-day runway concern retires with it. BYPASSED -- golden-currency. Controller v0.224.0 and v0.225.0 are released and the newest golden bake carries 0.223.0, so a machine installed right now gets neither. The gate is RIGHT. This push therefore uses `git push --no-verify`, declared here, in hub/CHANGELOG.md, in REPORT.md and on R-242. A BYPASS, not a waiver: the gate offers a waiver only for a release that DELIBERATELY needs no golden, and these need one. The operator was asked and ruled bypass-now-bake-later, on the ground that neither fix bites a day-0 box -- R-330 is a nightly false alarm about apps a new box has not installed yet, R-331 is a hub display over backups a new box has not taken yet -- and both arrive by self-update. That ground is recorded because it is what to re-check: it does NOT extend to a release changing first-boot behaviour. OWED: bake a golden carrying 0.225.0 and vouch it (RUNBOOK-manual-build.md 4.1, three-field change, MinAgent 0.129.0). Fourth bypass of this gate, and the gap is now two releases wide rather than one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM	2026-08-30 18:49:53 +02:00
admin	f5c9411e5e	R-331 (hub half): the Backup card reads offsite, not the dead backup fields (v0.109.0) ... The customer page's Backup card read `Snapshots 0 / Repo Size 0 MB / Integrity Unknown` for EVERY customer, indefinitely. Measured on demo-hp 2026-08-30 while that night's controller log said `[offbox] backup OK: 8 app(s) backed up, 67 snapshot(s), 2m14s` and the box held snapshot_count:67, repo_size_bytes: 140829678, stats_known:true. A card reading "no backups" over a working backup is worse than no card -- the R-88 direction of failure (degrade to NO BACKUP rather than to UNKNOWN) on the one screen that answers "is this customer protected?". The data was never missing. The card rendered the report's `backup` object, whose snapshot/size/integrity fields have had no producer since slice 8C. The live numbers are in the `offsite` object, which THIS PACKAGE already reads for the Offsite page and which monitor.OffsiteChecker already alarms from. Proof the bytes were arriving: the Offsite page rendered demo-hp's usage as 0.1 GB from that very object while the Backup card said 0 MB. So this is a render fix over an existing feed, not a new pipeline. Not a one-line swap, because snapshot_count:0 means two opposite things -- "holds nothing" and "never measured". R-225 measured that confusion one layer down. backup_card.go resolves a three-way ruling in Go (a {{if}} chain over map[string]interface{} float64s cannot keep the absent/zero distinction the card is entirely about): no offsite object -> "No off-site data reported", and says explicitly that this is NOT the same as "no backups" disabled + state -> names the blocker (needs_credential) stats_known:false -> em-dash + "never been measured". NEVER 0 stats_known:true -> the real numbers, INCLUDING a real 0 A pre-v0.225.0 controller sends no stats_known -> false -> "unknown". That direction is pinned: upgrading the hub ahead of the fleet must not report every un-upgraded customer as having zero backups. The Integrity row is DELETED, not re-sourced: nothing produces it, the controller runs no integrity check, and NotifyIntegrityOK/Failed are called from nowhere. RED-PROOF: restore the pre-fix card markup -> all four tests fail, reporting 67 and 134.3 MB absent from the rendered page and the Integrity row present. The tests drive handleCustomerUnified and grep the HTML on purpose: the defect was the template's choice of source object, so a test one layer below it would have been green against the shipped bug. Green gate clean: 18 packages, rc 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM	2026-08-30 18:39:15 +02:00
admin	c2c1fb48dc	REPORT: record the pushed commit hash	2026-08-25 09:36:29 +02:00
admin	c30430c530	skills: five process-domain skills + check_skills.py ... The four existing skills cover the product; nothing covered how work is reported. Two rules this project has paid for — check the artifact rather than the report, and do not state a claim more firmly than the evidence allows — lived only in the operator's head and in chat, where Claude Code never read them. - felhom-evidence five confidence tiers, artifact-over-report - felhom-diagnosis no hypothesis until a command has been seen red - felhom-plain-language ASD-STE100, two options, the re-pitch - felhom-handoff the note goes to a FILE, not the conversation - felhom-doc-authoring the pointer decides whether material is reached scripts/check_skills.py asserts what decides whether a skill is EVER reached: frontmatter parses, name == directory, description and body non-empty, under 150 lines, installed copy still samefile()s into the repo. install_skills.py globs and never reads the file, so a missing description installs perfectly and then silently never loads. It convicted on its first run: felhom-build-deploy is 179 lines. NOT trimmed here (pre-existing skills are out of scope, and trimming a deploy skill without exercising its commands is how a wrong command reaches a live host) — a named single-entry GRANDFATHERED exception, WARNed every run, R-394. A new skill over the limit is convicted. Red-proof run and seen failing: description removed from felhom-evidence -> exit 1, "frontmatter field 'description' is missing or empty". Restored, tree clean. skills/SOURCES.md records both MIT upstreams, that these are adaptations not copies, and the six pieces deliberately EXCLUDED with reasons. Register: R-392 (no architecture doc covers the two-AI workflow), R-393 (decision-log skill deferred, with the reason), R-394.	2026-08-25 09:36:20 +02:00
admin	ebdc04601d	docs(hub v0.108.0): the delivery grain, the cooldown ruling, and gate 11's first subject ... The alarm ladder gains §6.2 - which events are per-app, per-run, per-tier or coarse, and why the default is coarse. CONTEXT records two rulings: the grain is allow-listed rather than inferred from the payload, with crossdrive_failed as the proof that a payload rule would have been wrong; and a finding recorded only in REPORT.md has a lifetime of one session. R-389 closed and compressed, keeping its rules and naming the commit whose git show returns the full text. R-390 and R-391 left open. REPORT.md is gate 11's first real subject and passes: six observations, two FILED, four NOT-A-FINDING with their reasons. Three of those declarations are things a tidier report would have omitted - the gate's own spec would have passed the item it was built to catch, the burst has no ceiling, and ArgoCD said "successfully rolled out" while still running the old image. STATUS carries forward the one thing outstanding: the controller floor still reads 0.222.0 while the golden reads 0.223.0.	2026-08-23 14:12:23 +02:00
admin	2fc4a15fa3	R-389: key the operator cooldown per app for app_start_failed; gate 11 makes an unfiled observation refuse the push ... The cooldown key was customerID:eventType plus the tier and run suffixes, and none of them names an app, so every app going down inside the same hour collapsed onto one key and only the first was mailed. Measured on demo-hp: bookstack sent 09:27:51, privatebin suppressed 09:31:51 under key=demo-hp:app_start_failed. cooldownStackSuffix is the third sibling of cooldownTierSuffix and cooldownRunSuffix, and separate for the reason the second one's docstring already gives: the existing two keep byte-identical semantics for every type that uses them. It is ALLOW-LISTED to app_start_failed and takes the event type as well as the details, unlike its siblings, and that asymmetry is the safety property. The backup family's cooldown is coarse ON PURPOSE (R-97a, R-182) so one full disk sends one digest rather than one mail per app - and crossdrive_failed is severity error, reaches the operator leg, and carries stack_name through a DIFFERENT struct, so a payload-shape rule would have split it silently. The hour itself does not change. Gate 11 refuses a push whose REPORT.md carries an observation with neither `FILED: R-NNN` nor `NOT-A-FINDING: <reason>`. It deliberately does NOT accept a passing mention of some other R-number: the lost item cited R-182 as an analogy, so "cites a register row" would have passed the very item the gate exists to catch. That discrepancy with the spec is recorded in the gate's docstring. Registered here and in the controller and agent runners. NOT in the catalog runner - it has no shared-gate mechanism and appends --all to every gate; filed as R-391 rather than left as a sentence, which is this session's lesson. PROMPT-TEMPLATE.md §15.9 corrected: "documented, NOT acted on" was the wording that invited the gap, and it now names the markers and points at the gate. R-390 filed for the golden-bake runbook's missing `pveam update`. Hub tests 709 -> 716.	2026-08-23 13:53:01 +02:00
admin	2f7c9a6ce5	docs(R-329/R-386/R-387): the severity contract, the intent ruling, and Part 5 recorded ... The alarm ladder gains the severity contract (the hub's vocabulary is exact, it coerces silently, and three things now hold it) and the intent test with its three-way ruling on unknown. Both marked [DESIGN] with the live measurements. Part 5 is RECORDED AND NOT IMPLEMENTED: the operator's notification philosophy, verbatim, marked plainly as direction rather than current behaviour, with the 12 -> 15 toggle growth as the argument. Filed as R-388, a product decision. R-329 and R-386 compressed into CLOSED-ITEMS with their rules kept and the full-text commit named. R-387 filed closed - including WHY the dispatcher branch was kept rather than deleted, which is evidence (three monitor checkers call ProcessEvent directly) and not caution. The drill record names three things that had to be re-run: an inert red-proof mutation, Scenario G refused twice behind an HTTP 200, and the live Scenario A NOT proving the customer gate because demo-hp has no prefs row at all. Register: OPEN 328325 -> 328132 B, CLOSED 71441 -> 74642 B.	2026-08-23 12:03:49 +02:00
admin	55274d5ef3	R-385: make an UNRECORDED golden fail the currency gate; file R-386; own the alarm ladder ... The gate failed only on `released > baked`, so it could catch a forgotten bake and nothing else. A golden AHEAD of the record passed silently - and that is how controller 0.221.1 was built, baked AND vouched while the newest CHANGELOG heading still read v0.221.0, with every gate green. Reproduced on the real history: newest released 0.221.0 / newest golden baked 0.221.1 -> exit 0. The gate now asks whether the version being shipped is WRITTEN DOWN: the baked version must have its own `## vX.Y.Z` heading anywhere in the CHANGELOG. Membership rather than `baked > released` deliberately - a comparison against the newest heading alone goes green the moment any later entry is written, leaving the unrecorded version permanently unrecorded. INCONCLUSIVE (exit 2) preserved; every refusal names a reason and a route. Red-proofed both directions: old gate/old record exit 0, new gate/old record exit 1, new gate/fixed record exit 0, absent clone exit 2, post-bake exit 0. 08-alarm-ladder.md is new, and its absence was itself the finding: no document owned "when does a broken app raise an alarm?". The rules lived as comments in four packages, each locally correct, with the ordering between them legible only by reading one function top to bottom - which is how R-384 survived review. R-383 and R-384 closed into CLOSED-ITEMS with their rules kept. R-385 filed closed. R-386 filed OPEN: a single-container app stopped out of band raises no alarm, and a comment claims the opposite - measured live, 9 scans, 0 events, against a positive control from the same box 17 minutes earlier. Not fixed here. Golden 0.222.0 baked and published; vouching is the operator's act.	2026-08-23 07:59:52 +02:00
admin	1eb64bec51	R-361 docs: the [FACT], the negative that cancelled Part 2, R-383/R-384, golden 0.221.1 ... 07-backup-architecture.md gains a dated [FACT] on R-361 - a comment asserting an invariant the code did not have, for four months - and a [DESIGN] on the db_dumps decision INCLUDING the trap it created: a stable list lets the already-current early return fire, so per-capture housekeeping must sit above it. 00-capability-map.md records the NEGATIVE from Part 3 so it is not re-derived: a held app does NOT raise the dead-app alarm. It aggregates to unhealthy, which IsDownState excludes. Measured on the shipped build with the scans demonstrably running over it. No suppression was built and no row opened. R-383: the double-failure message names an undo copy that is not there - R-361's own class, one surface over, observed on both 0.220.2 and 0.221.1. R-384: an app whose database has died reads unhealthy and raises no alarm. R-361 closed and compressed. OPEN-ITEMS 325236 -> 327266 bytes. Golden 0.221.1 baked, published and round-trip verified. The golden-currency gate blocked this push and that block is not circular, so it was satisfied rather than bypassed - no --no-verify anywhere in this session.	2026-08-23 00:33:12 +02:00
admin	a8caa0fdde	R-379/R-380 docs: the failure ladder, the drill record, register housekeeping ... 07-backup-architecture.md 6.3 gains a dated [DESIGN] paragraph on replay -> rollback -> hold, including why no engine flag closes it: --single-transaction makes Postgres atomic, MariaDB DDL is not transactional, so the rollback is the fix and the flag is a belt. Drill record for the live walk, including the TWO defects the walk found in the fix itself (a rollback into a re-created container; an operator route that cleared the file while the running controller kept refusing) and the ONE red-proof that PASSED, which is reported rather than omitted. R-379..R-382 compressed into CLOSED-ITEMS.md. OPEN-ITEMS 330683 -> 325236 bytes. STATUS.md restates the outcome and names the next operator step.	2026-08-22 18:40:22 +02:00
admin	4e488321bf	DRILL R-356b: the off-site restore for a driveless app that HAS a database ... A drill, not an implementation. No code, no version bump, no CHANGELOG entry. Ten of the forty driveless apps carry a database; I re-measured that count and got 10. For those ten the restore is a five-leg operation that never ran at all until this week, because R-356 refused before any of it started. Walked end to end on demo-hp for both engines - docmost (Postgres 16) and bookstack (MariaDB 12.3) - each deployed for the drill, planted through the app's own interface, destroyed for real, restored through the endpoint the UI posts to. Q1 does it complete: YES. All five legs ran and succeeded, 32s / 25s. Accented names byte-identical both directions. Q2 which leg won: the SQL DUMP. Three-way discriminator returned the altered dump's value. This confirms R-164's F17 ordering on the OFF-SITE path; R-164 only ever cited the local one. Scratch-only mutation; store proved unmutated. Q3 does a failure tell the truth: partly, and two defects. Filed R-379 (HIGH, the undo copy is valid, named, and unappliable by any product action - proven by applying it by hand on both engines), R-380 (HIGH, a failed MariaDB replay leaves a partial database behind an app reporting healthy, where Postgres crash-loops visibly), R-381 (MEDIUM, the failure message pastes engine stderr including customer table rows into the Hungarian surface), R-382 (LOW, the summary log omits the volume count it already has). H1, H2 and H4 did NOT fire and that is recorded. H3 fired in a shape nobody predicted: not a quiet success, but a loud error over a silent inconsistency. R-361 reproduced independently on a second app. restic check: no errors, 29 snapshots. A flaw in the drill's own planting - a double-escaped accented title - was caught by reading stored bytes as hex, recorded, and re-measured in Phase 1b. Register 325236 -> 330683 bytes. Nothing dropped. Teardown: two apps retained with reason, no pvesm before-snapshot taken (said plainly), no hub-side record created.	2026-08-22 16:33:45 +02:00
admin	c297b9f85e	R-356 docs: correct R-107 in the architecture, record the design, refresh STATUS, compress the register ... 07-backup-architecture.md: three places said no offsite action unpacks the named-volume tars. R-107 closed in controller v0.218.0; all three corrected with a dated [FACT], the old sentence kept in the past tense. R-102 is NOT closed and the correction says so explicitly. New [DESIGN] paragraph in 6.3: the restore destination is resolved by the same rule as the capture destination, and the wrong-disk refusal applies to apps that have a drive to get wrong. Carries the 13/40 measurement. STATUS.md was internally contradictory - nothing waiting, and one decision waiting, for something the same page recorded as shipped. 218 -> 102 lines; the deciding section now says what happens if nothing is done. R-356 compressed into CLOSED-ITEMS.md; OPEN-ITEMS 327109 -> 325236 bytes. Drill record and 16 evidence files for the live walk on demo-hp.	2026-08-22 13:26:10 +02:00
admin	f2edf7e545	REPORT: record CI run 384/252 by id	2026-08-22 12:14:32 +02:00
admin	ef6ac6fe74	One register, enforced by a gate; closed work compressed into siblings (R-376..R-378) ... Records and process only. No machine contacted. ONE REGISTER (operator ruling). 17 roadmap rows moved into OPEN-ITEMS.md keeping their identifiers, evidence and original filing dates - the oldest R-10, filed 2026-07-15, 38 days. 15 ideas stay in ROADMAP.md, which is their home; the gate exempts them by their own state word. 59 already-closed rows stay as history. Sorting rule recorded in the roadmap header: does the item assert something about the shipped product a reader could check and find false? scripts/one_register_gate.py, wired as the 11th gate. Control run: baseline passes, a planted open roadmap-only row is convicted by name, removing it passes with the file byte-identical, and a planted `idea` row is correctly exempt. Its four residual holes are in its docstring. The gate earned its keep immediately: it caught R-103, a READY finding my hand-sort mis-read as done because my regex matched the whole row where the body contains "shipped" - the gate matches the state cell. It also caught R-203 and R-163, recorded closed in the register and still open in the roadmap; the roadmap copies are marked SUPERSEDED with the register's verdict. HOUSEKEEPING. OPEN-ITEMS 672,376 -> 327,109 bytes (-51%); ROADMAP 239,306 -> 78,110 (-67%). Closed work compressed to 17% into CLOSED-ITEMS.md and ROADMAP-HISTORY.md; every entry names the commit whose git show returns the full original text. Rule-sentences are kept verbatim under "Reasoning kept" rather than judged entry by entry - 25 carry one. CONTEXT.md deliberately NOT compressed and the disagreement is argued in the report: 86% of it is standing rulings still in force, this prompt's own 3.4 says the log is never edited, and it has no per-ruling delimiter. Filed as R-377 - the problem is navigational, not volumetric. The hot/bulk placement decision was NEVER recorded as a decision anywhere - established, not assumed. Now marked [DESIGN] with a pointer honest about having no original date, given a decision-log entry that records what was rejected, and the [DESIGN]/[FACT] legend carried from 1 of 8 architecture documents to 8 of 8. Existing statements deliberately left unmarked (R-376). PROMPT-TEMPLATE gains N.7: compress what you closed, rehome live reasoning before it goes, state the register's size before and after. Ceiling R-375 -> R-378.	2026-08-22 12:13:54 +02:00
admin	fddfe00ce2	REPORT: record CI run 382/250 by id	2026-08-22 11:18:46 +02:00
admin	091a4b7444	Correct the placement mis-framing, and file what we wrote down and never filed (R-368..R-375) ... Documentation and survey only. No code, no machine contacted. THE CORRECTION. The 40 catalogue templates without a configurable path are not missing a choice: 01-topology-and-trust.md:150-152 classes each volume hot (DB/config/cache -> fast storage, ENFORCED) or bulk (media/files), and the 40 are all-hot apps. The deploy page has been saying so to the customer all along (deploy.html:624-625). SPEC-app-data-placement and R-352 are corrected in place with the framing MARKED, not deleted; every measurement stands. R-356 was re-checked and survives, strengthened - an absent HDD_PATH is the normal state, so reading it as "not installed" misreads a correct configuration. The disk claim, precisely: since R-165 there is ONE guest data volume with two binds, not two volumes (build-golden.sh:29-40, 99). A physical-disk failure losing data and first-tier copy together is REAL and is what the other tiers exist for. A full data volume stopping the OS is NOT real and was the overstated one. THE SWEEP. 113 survey-class documents examined, 14 statements of "not filed", 2 already filed. Its positive control convicted the sweep itself twice before it convicted the corpus - markdown bold broke the strongest pattern, and the reporter re-searched a truncated line - both false zeros of the exact class being hunted, and together worth 2 of the 14. THE HEADLINE. The gap the 2026-08-21 drill rediscovered WAS filed - as R-107, ROADMAP.md:122, M/READY, 2026-07-28 - and is absent from OPEN-ITEMS.md, which calls itself the single source of truth. OPEN-ITEMS and that rule both landed 2026-07-27; R-107 went to ROADMAP alone the day after. 72 ids live only in ROADMAP, 29 not done, some of them findings. Filed as R-369 (HIGH). Five more still-open gaps filed with their ages: R-371 (17d), R-372 (38d, the oldest), R-373 (20d), R-374 (14d), R-375 (4d). R-368 corrects Part 4: the storage default IS applied at deploy time via deploy.html:612 - the earlier "the deploy route never reads it" came from grepping Go and never the templates. R-370 records the process failure and is closed by the template change. PROMPT-TEMPLATE gains the two rules it lacked: name the architecture document for the area and say what it says (with a file->area map and the test "is this something we chose?"), and an enumerated gap becomes a register row in the same session - a ROADMAP row alone does not count. Ceiling R-367 -> R-375.	2026-08-22 11:18:08 +02:00
admin	5a7502b6d5	REPORT: record the where-felhom-stands refresh, the three downgrades and the renderer defect ... NOT WALKED moved 32 -> 35 of 55, which the repo checklist asks be stated. Also records the hardcoded-sha defect found in render_stands.py and the positive control run on check_stands.py.	2026-08-22 10:37:36 +02:00
admin	877fcd2a38	R-354 + R-355 CLOSED, proven live; golden 0.218.0 baked; R-367 filed ... Both of the drill's HIGH findings are fixed in controller v0.218.0 and confirmed on demo-hp with a negative control first — the same planted, hash-recorded fixture run through the same steps on both builds. R-355: paperless-ngx's PostgreSQL was dumped into a directory for a stack that does not exist, so it never entered the recovery unit, the off-site copy or the restore; and because the same wrong name reached writeSafetyDump, a destructive restore took no undo copy and the fail-closed refusal was never reached. Fixed by reading the compose project label. Sweep proven able to convict before its count was trusted: one affected app of 53. R-354: the off-site restore had no named-volume leg. Now it replays them from the scratch unit, before the database and inside the stopped window, and VolumesReplayed reaches the sentence. The half-false comment beside the skip is corrected and the half that still holds is named. Golden 0.218.0 baked and published, sha 8e427869d13eafb71562b77d1535eef6c7f32b4db24f659b988ec6d7db8f478b, verified by round trip on the downloaded bytes. NOT vouched and the floor NOT raised — both are the operator's decision, and raising the floor is what puts this on demo-felhom, which is still on 0.217.0 and still has both defects. R-367 filed: the dumps already written under the wrong name are stranded. Nothing deletes them (an existing guard), they are adoptable by hand, and doing it automatically would be a migration. Ceiling R-366 -> R-367.	2026-08-22 10:11:46 +02:00
admin	7064596c2e	DRILL closeout: the scheduled cycle agrees, the abandonment sweep watched firing, R-366 ... The box's own 02:30 / 03:30 / 04:15 cycle ran unattended and agrees with the manual one on every measure. The one that matters: R-355 is not an artefact of manual triggering — the scheduled run again wrote paperless-ngx's PostgreSQL dump into a directory for a stack that does not exist, and again left the app's own unit recording db_dumps: null. Part 4.2's terminal deletion has now been observed. At 05:10 the sweep removed exactly the recorded set-aside store and left the live repository untouched; at 05:13 the hub dropped the sealed package that protected it and said so (event 3025). Both halves went together, three minutes apart, and the controller cleared its own state. demo-felhom's two preserved fixtures were verified untouched throughout. R-366 (HIGH) filed, found incidentally: the 21 August reinstall orphaned demo-hp's PBS whole-guest archives as well as its restic repo, so a rebuilt box loses BOTH off-premises tiers at once. The restore-test caught it and named the key mismatch precisely; it is merely called "a failed restore test" rather than "your older backups are unreadable". demo-hp is left HEALTHY, not broken. Ceiling R-353 -> R-366.	2026-08-22 05:25:30 +02:00
admin	d895d9f7dd	STATUS + capability map: narrow the end-to-end off-site claim to the leg it was proven on ... The 2026-08-04 row claimed "a customer's file survives a machine rebuild and comes back — the whole off-site story, end to end". Tonight's drill shows that holds for the declared-userdata leg of a drive-declaring app and for nothing else: the off-site restore has no named-volume leg (R-354), and refuses outright for the 40 apps that declare no data drive (R-356). Since that class keeps ALL its data in named volumes, the end-to-end story is unproven there and disproven for the volume leg generally. The escrow/key half of the row is untouched and still stands. STATUS.md also corrects the fleet pair it still named (0.214.0/0.129.0 -> 0.217.0/0.130.0) and records that demo-hp's off-site had been silent since 9 August.	2026-08-21 23:34:21 +02:00
admin	f5a4fceeeb	DRILL 2026-08-21: the off-site restore never replays named volumes (R-354..R-365) ... Diagnostic only — no code changed, no version bumped, nothing deployed. The verdict is a mixture. The unit and the off-site snapshot HOLD the data, proven by identity in both storage classes including two Hungarian accented filenames. The loss is in the last leg: ReconstituteFromOffsite skips every isUnit placement and the volume tars live inside the unit, so the off-site full restore has no named-volume leg at all — while the local restore-from-unit does, and returned the same tar byte-identical minutes later. Twelve rows opened, ceiling R-353 -> R-365. Three HIGH: R-354 off-site restore never replays volume dumps R-355 paperless-ngx's Postgres is dumped under a non-existent stack, so its unit has no DB dump, no safety dump is taken, and the customer is told it has none R-356 the off-site restore refuses for all 40 no-drive apps saying the running app "is not installed", with a remedy those apps make impossible R-353's instruction (2) is satisfied and annotated: the 40-class DOES reach the off-site tier. Its instruction (1) stands and is now larger. R-329 confirmed still live and now the only bad-severity emit fleet-wide. Evidence: documentation/audits/DRILL-backup-truth-2026-08-21/evidence/	2026-08-21 23:30:27 +02:00
admin	1b4d005f80	REPORT: CI run 351 by id (inherited golden-currency), unproven unchanged	2026-08-18 12:27:36 +02:00
admin	3e50902a98	RUNBOOK ep0: PBS 4.2.2-1 -> 4.2.5-1, slope unchanged as predicted (R-341) ... Both STOPs cleared by the operator. No code changed; documentation only. STEP 3 (the run's primary deliverable): the full changelog range 4.2.2-1 -> 4.2.5-1 was read (128 lines, all three entries) and swept for connection-handling vocabulary. Exactly one keyword hit, a false positive ("S3 ... honor the node's proxy settings" = HTTP proxy config for S3, not the PBS proxy daemon). 4.2.5-1 is a manifest-hardening security release; 4.2.4-1 is S3 rate limits and a locking cache; 4.2.3-1 is UI/LDAP/tape. NOTHING addresses descriptor lifetime or connection reaping. Recommendation was: do not upgrade for this reason. STOP 1: operator ruled to upgrade anyway for rehearsal value. Recorded as a practice run, not a fix -- and the interpretation was fixed IN WRITING BEFORE any numbers existed (stop1-ruling.txt): unchanged = expected; changed = surprise. Neither outcome could then be rationalised into a success. STOP 2: Hetzner snapshot 421440873, Available. Documented that it covers /dev/sda ONLY -- /mnt/pbs-datastore is a separate Volume and is NOT in it, so it is a software rollback and not a backup of the backup data. UPGRADE: simulated first (0 to remove), then installed 09:51:00->09:51:06Z, exit 0. Verified: 4.2.5-1 installed, both daemons active, effective open files still 65536 (the drop-in survived the new package), Recv-Q 0, loopback 200, 200 from BOTH boxes over the tunnel with felhom-pbs active, and the hub gauge refreshed post-upgrade at 11:59:31. SLOPE: before +4 fd/1885 s = 183/day; after +5 fd/1919 s = 225/day. NOT distinguishable -- one descriptor apart, Poisson +/-2 on such counts. The higher after-figure is noise, not a regression and not an improvement. 30 minutes cannot settle it; R-341 files the +24 h and +7 d checks. CORRECTIONS to this morning's own report, both published rather than quietly fixed: - the "~85/day, ~2 years of runway" figures were WRONG. They came from a single 17-minute window with a delta of ONE descriptor. Real rate is 183-200/day over two independent windows; runway ~357 days, not 2 years. - the leak was attributed to CLOSE-WAIT. It is mostly ESTAB: CLOSE-WAIT held flat at 1 while ESTAB grew 45->49, and at the wedge it was 1011 ESTAB vs 543 CLOSE-WAIT. R-336's fix must target unreaped connections. - "proxmox-backup-api" reported inactive during verification; that unit does not exist. Bad query, not a fault, written down because it looked like one. R-336 stays open: even a fixed leak would not make ~85k requests/day to a weekly-write DR endpoint correct. golden-currency still convicts (inherited R-334, controller 0.216.0 vs golden 0.214.0, untouched by this run), so this push is --no-verify per .claude/rules/gates.md.	2026-08-18 12:26:53 +02:00
admin	172584c26f	REPORT: record CI run 348 by id, and what could not be read ... Checklist item "confirm your own push's CI run by run ID". Run 348, head_sha ebfd0967c, conclusion failure, elapsed 13 s -- the inherited golden-currency conviction (R-334), not a new fault: CI's only step is the same repo_gates.py --fast entry point, and 13 s is the honest-failure band rather than R-265's reap band. Stated plainly that the run LOG could not be read (runs/348/logs and tasks/348/logs both 404 authenticated as admin, runs/348/jobs empty, web endpoint 302), so naming the gate is an inference from the local run plus gates.yml -- not CI's own words. Standing rule 2: a "no access" claim names what was tried. Also notes the [felhom CI] gates FAILED mail run 348 will send, so it is not read as a second incident alongside this morning's backup alerts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016p1PTCzb8rF5G9Aa1qhpBN	2026-08-18 06:11:29 +02:00
admin	ebfd0967c1	INCIDENT + registers: ep0's PBS proxy served nobody for 9.5h (R-336..R-338) ... Two whole_guest_backup_failed alerts at 04:30 and 04:32 CEST were one incident, and not on either customer box: ep0's proxmox-backup-proxy was active, holding its listening socket, and accepting nothing. Root cause: accept() returning EMFILE. The process held exactly 1024 fds -- its systemd-default soft RLIMIT_NOFILE -- of which 1016 were sockets and 547 connections sat in CLOSE-WAIT. The 1024-deep accept backlog had overflowed (Recv-Q 1025), so every client timed out. It was wedged from its own loopback too, which is what moved this from a network problem to a process problem. Fed by ~85k requests/day (a flat 3,538/hour) against an endpoint written to weekly, that leak reached the ceiling in 14 days of uptime. Fix: LimitNOFILE=65536 drop-ins for both PBS units, restart, verified from both boxes (200 in ~0.1s, felhom-pbs active), then re-drove the missed backups through the product path -- POST /backup?target=felhom-pbs on each agent's local API, not a hand-run vzdump. demo-felhom ct/9201/2026-08-18T03:57:43Z 4.10 GB 36.4s demo-hp ct/9201/2026-08-18T03:58:43Z 4.29 GB 41.5s Both host reports now carry felhom-pbs success=true, so the hub is green on the evidence rather than on a restart having been performed. No data lost, no backup skipped: the daily local tier was never affected and the PBS tier is weekly, so the window cost exactly one attempt. Evidence copied off ep0 BEFORE the restart, per standing rule 5. Filed: R-336 (the ~1 req/s poll rate is the real defect; the raised ceiling is mitigation, not a cure), R-337 (a status endpoint that trailed its own artifact by minutes then caught up -- WATCHING, downgraded from the defect I first wrote, because it self-corrected), R-338 (demo-hp is not on the R-50 island at all and nodes.md says it is; its local API is bound to the customer LAN). R-334 updated: still open, now one version wider (controller 0.216.0 vs golden 0.214.0). golden-currency is the only failing gate and is inherited -- it reads files this session did not touch -- so this push used --no-verify, stated per .claude/rules/gates.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016p1PTCzb8rF5G9Aa1qhpBN	2026-08-18 06:09:38 +02:00
admin	e0b56c976f	REPORT + CONTEXT: the third name, the second door, and a number that answered a different question ... Three rules carried forward. A name must separate on the STEM, not the noun — naming this secret after the act it is used in would have recreated the trap, because the other factor on the same page is the „Párosító kód". A guard is worth what its positive control is worth: this one's selftest convicted its own step-3 case and found a defect in the guard itself. And a suppression must rest on the machine's own declaration, then be checked for the SECOND door — recording the disabled state rather than deleting it is what let the deadline check skip it too. Yesterday's report is preserved to audits/ because it carries the only record of the self-heal verdict (Part C was dropped, so that reasoning is in no register row) — the rule written last night, applied to itself the first time it mattered.	2026-08-13 16:01:45 +02:00
admin	955a4f07b7	REPORT + CONTEXT: A4 reverses its own premise, the naming half, and the first R-264 reader ... A4's evidence contradicts the task's framing and the register now says so: peti-felhom is a real machine with 482 reports and a real person behind it; david -> tester-1 is a record that has never had a host, an escrow or a report. The risk is real; only the word that named it was wrong. CONTEXT gains three standing rulings: read the fact that carries the RISK (heals_last_hour, not state) and decode the one that decides the question (heal_succeeded — R-260's lesson); one secret in two situations keeps its name and changes its sentence, and the mail must name the page the machine actually shows; and evidence dies in the INTERMEDIATE revert — with the corollary that a durable citation may never point at a file whose contract is to be overwritten, which is why the R-316 report was moved to audits/ before this one was written.	2026-08-13 10:59:55 +02:00
admin	2d05b29b82	REPORT: installer v1.28.0 published and verified live; evidence, and the Part 1 logs I lost again	2026-08-13 08:19:41 +02:00
admin	4906aeb3f9	R-311 proven live (HTTP 422 on hardware); R-308 WITHDRAWN — my quoting bug, not a stale credential ... The live test read as a FAILURE for twenty minutes because I stripped only double quotes from a credentials value wrapped in SINGLE ones, sending a literal ' as part of the recovery code. Correctly unquoted: the old code returns 422 with opens_retained=true and the supersession date; a wrong code still returns 400. The same bug produced the R-308 finding in the previous report. The dashboard password is fine - HTTP 302 with a session cookie on the first try. Third time this project has produced a wrong 'the credential is stale' verdict from that one trap.	2026-08-12 19:05:58 +02:00
admin	8b188bea68	hub v0.103.0 — a host can read the packages we kept for it (R-311) ... ListSupersededEscrow had zero production callers for nineteen days. It is the only reader of a retained identity_blob, so the retention shipped in v0.93.0 was material the product could not reach - proven on the fixture 2026-08-12, where a code that opens a retained package was answered as a code that opened nothing. New GET /api/v1/hosts/<id>/escrow/retained: self-scoped exactly as the current-row GET, same recovery-mode gate, same audit event written BEFORE the bytes leave, capped at 16. Rows with a NULL identity_blob are WITHHELD and returned as unopenable_count - they retain the PBS key, not the repository password, so they can never open what the caller is asking about, and serving them would let the screen claim an earlier package is openable on exactly the boxes the original defect hurt. The count is returned because their existence is load-bearing and underivable by the caller. The trade, stated rather than waved through: the hub still cannot read any of it - sealed bytes in, sealed bytes out, no decrypt path, no recovery code ever held. What widens is volume, bounded by self-scope, the recovery-mode gate and the cap. The response is a NAMED TYPE, not a map, so the wire-contract gate can resolve it; the wire is declared as a fourth ROOT and the gate now checks 182 tags rather than 174. A positive control shows that check is name-presence, not decodability - filed as R-315 rather than reported as coverage. Also: golden 0.214.0 baked, published and round-trip verified; the countdown on demo-felhom cancelled on the operator's ruling (R-307); the spike that halted Part 3 recorded as R-312; the set-aside store found unrecoverable as R-313. Six hub tests through the real endpoint; four red-proofs asserted applied.	2026-08-12 18:56:44 +02:00
admin	1d4985d87c	REPORT: the same wall-clock correction	2026-08-12 17:45:39 +02:00
admin	1d5f2b8bb6	DRILL: the retained key works, and the customer cannot reach it ... Three verdicts, kept separate because collapsing them is how this assumption survived a week. (a) The material IS retained. host_escrow_superseded id 11 is the first retained row in fleet history to carry identity_blob (572 B), byte-identical to the pre-supersession row (sha256 a10032341c8584ed...). (b) The retained material DOES open the old store. Unsealed with the old recovery code it yielded a password byte-identical to the pre-change one, and restored three planted files byte-identical from a store the box itself could no longer open - including a Hungarian accented filename verified as raw bytes. Negative control ran first and failed closed. (c) The customer has NO route, and is misinformed. ListSupersededEscrow has zero production callers; the recovery path selects FROM host_escrow. Asked with the code that had just worked by hand, the product answered "the recovery code did not open the sealed bundle". A valid code for retained history is reported as a bad code - the R-224 class again. R-304, rank 1. Both installer faults were watched happening first, so installer-v1.27.0 is now published (tag + both webpage.yaml refs). Pre-fix: the box came up on controller 0.98.3 against a vouched 0.213.0, below the floor and below the version carrying the recovery screen; and our own uninstall left dnsmasq on 0.0.0.0:53 so our own next install refused. R-297 and R-300 CLOSED. Also filed R-305 (the dnsmasq fix fires once per machine - the leftover returns on the second reinstall, proven), R-306 (--preflight-only writes state it says it does not), R-307 (a live abandon countdown on demo-felhom, firing 2026-08-24 - operator decision), R-308 (stored controller password stale), R-309 (the day-0 runbook's publication claim has been false since R-110), R-310 (two edges). Ceiling R-303 -> R-310. Capability map moved: the retention claim is now marked operator-only. Phase A logs did not survive the intermediate revert; recorded.	2026-08-12 17:41:56 +02:00
admin	e1b679782a	REPORT: record corrected, second promise closed, removal chain code-only	2026-08-12 14:05:15 +02:00
admin	2ccbbb8da1	REPORT: installer golden gate and the 0.211.0 bake	2026-08-10 14:20:47 +02:00
admin	c04f933d0b	The census answers no, three receipts found, and the prune was on file all along ... CENSUS (read-only, hub store, tester's machine not contacted): no machine that is not ours can be in the state that cost demo-felhom its history. The hub holds escrow for three hosts; both demo boxes lost their pre-fix key in the same four hours on 2026-08-04; peti-felhom and david have no host row and no escrow at all. A control ran FIRST and had to pass -- the query returned "present (572 bytes)" for a host known to have material and "absent (NULL)" for one known not to. Corrected my own instrument on the way: a date-only comparison mislabelled both losses as after the fix, so the in-force moment is now pinned from the hub's first post-fix escrow row (11:11:37Z), which independently agrees with the register. PART 1 ESTABLISHED. The prune is recorded inside R-267 -- the row about the Configuration page being slow -- because pruning artifacts is what made that page fast. Arithmetic checks (23+7=30, plus three versions that only surfaced after the first thirty moved them onto page one = 33) and the PAGINATED listing shows both generics at exactly ten. R-291's blocking condition is released: the operator was being asked to establish something already written down. And my counter-argument yesterday was wrong in exactly the way R-267 warns about -- "containers hold 19" came from an unpaginated query; paginated they hold 270 and 169. RECEIPTS: three restored (drives.enrol, backup.tier1, fail.lost-recovery-code), each citing the document that walked it; the map already read PROVEN-LIVE for all three, so this follows the map rather than raising a status in the view. NINE HONEST GREYS. fault.selfheal's best hit argues against it -- an incident recording self-heal's absence through a 1h15m outage. THE DECAY RULE FIRED FOR THE FIRST TIME. backup.restore-proof has a receipt from 28 July and is superseded anyway: demo-hp's restore-test failed 5 August and the box has since been rebuilt. A claim about a continuing behaviour cannot rest on an old observation. The capability map still reads PROVEN-LIVE and is now the thing out of step -- recorded, not silently rewritten. PART 4 specified, not implemented. The orphan card promises restorability the box rendering it cannot evaluate: the discriminator is on the hub and no wire field carries it. A conditional promise the system cannot evaluate is the same defect as an unconditional false one, so the copy stops promising, says what happens, and names a route. Ships with the next controller change so one bake covers both.	2026-08-10 11:15:37 +02:00
admin	985f0ba63c	Record the guards, the narrowing, and the two things I could not do ... R-273's owed guards are both built and closed. R-291 records what CI stopped covering and why, so it can be widened deliberately rather than discovered. R-292 is new and was found by a test failing for the wrong reason: artifact_sha_invalid conflates "version missing", "registry unreachable" and "bad sha" into one message. v0.102.0 works around it by ORDERING -- the probes run first, so an unreachable registry is reported as unreachable -- but the message itself is untouched. CONTEXT gains the rule this session is about: a check and the policy it enforces must read the same number from the same place, or they drift and the drift looks like a defect in something else. Two corollaries, both of which cost something: a bounded check must print what it stopped covering on every run, and an unreadable policy is INCONCLUSIVE rather than unbounded. Stated in the report rather than glossed: Part 4 (finding receipts for the twelve downgraded claims) was NOT done and is a shortfall, not a decision -- splitting it would have produced exactly the half-checked green the exercise exists to prevent. Part 5 was droppable and dropped. The tag-push green is not re-proved tonight and is not claimed; the evidence offered is runs 190 and 216.	2026-08-09 19:28:45 +02:00
admin	6088afcbed	Verify the standing picture against source: 12 downgrades, and the decay ran both ways ... 55 claims verified. Twelve moved, all downwards: walked 32 -> 20, built 5 -> 17. Register ceiling R-284 -> R-290. THE RULE DID NOT FIRE THE WAY IT WAS EXPECTED TO. Not one downgrade came from code moving under an old proof. All twelve came from step 1 of the same rule -- the cited evidence does not exist. Measured: of the 28 capability-map rows behind the page's claims, 8 carry a tests/ or audits/ path and 20 carry prose only. The green dots were drawn from rows that cite an argument, not a walk (R-290). The map, not the dataset, is what needs fixing -- it still says PROVEN-LIVE for all twelve. And once it ran backwards: fault.operator-email looked contradicted by R-182, but live source shows the backup_run_failures digest allowlisted, operator-only and templated, with recovery_unit_capture_failed now record-only. The claim is right and the REGISTER ROW is stale (R-289). The session went looking for stale proofs and found a stale defect. R-281 WITHDRAWN -- wrong in both directions, settled by the operator's mailbox. The tripwire DID fire (escrow_blob_served 10:19:41Z = 12:19 CEST) and false error-severity alarms fired too, for deliberate attended work (R-285). The measurement's cause is ESTABLISHED: the P7 query copied hub.db without hub.db-wal, and the signature is exact -- it reported "2 events all day, newest 00:30:07", and the rows at or before 00:30:07 number exactly 2. Timezone and wrong-key were tested and refuted. The control had been drawn from the same stale snapshot as the measurement, which is why it agreed (R-286). Part 4: NO WORKFLOW CHANGED, deliberately. The gate is not ref-sensitive -- it enumerates from the Gitea tags API, and both previous tag pushes passed. The red is TRUE: run 267 saw v0.120.0 downloadable, run 284 on the same commit saw 404. Who deleted the package is NOT established and is not guessed (R-287). The page is now generated from where-felhom-stands.yaml by scripts/render_stands.py: static, zero script tags, every moved status carrying a visible "changed, was X" chip. The React bundle -- whose content was gzip+base64 inside a JS module map -- is kept as a dated snapshot. scripts/check_stands.py gates the data and convicted 51 problems in my own first draft before the staged positive control ever ran.	2026-08-09 18:40:49 +02:00
admin	9e12116903	REPORT: R-221's live sequence RAN and PASSED — the preflight before/after, quoted ... Operator-present drill on demo-felhom guest 9201. Preflight refused with the key removed, went green after one 60s tick with the SAME MainPID (1993397 both sides, so no restart), and all 45 config keys came back identical. Positive control run before the change so the green afterwards is a measurement, not an artefact of the probe. Marker sha unchanged throughout.	2026-08-08 20:13:46 +02:00
admin	4a4a1e245a	R-265 CI timeout + golden 0.210.0 baked; R-221/R-259/R-258 closed, R-266 minted, G-3 unblocked ... Four defects of one family, all shipped today: something the box already knows, thrown away or drawn as its opposite. Agent v0.128.0, controller v0.210.0. NO HUB CODE, no hub bump, no ArgoCD sync. R-265 (this repo). timeout-minutes: 5 on the gates job — every honest run in the observed session finished in 18-34s, so this is ~9x the slowest and far under whatever reaped run 264 at 834s with no log. The alarm mail now carries Elapsed (start stamp via $GITHUB_ENV; an absent stamp prints "unknown (no start stamp)", never a bogus 1.7-billion-second figure) and its "names itself in the run log" sentence is qualified so it cannot mislead when there is no log. ⚠ THE UNKNOWN IS NOT CLOSED. Whether the if: failure() alarm fires for a REAPED job is still unverified. The timeout makes the reap unreachable in practice; it does not answer what happens in one. Demonstrating it means deliberately hanging a run on main, which would leave the branch red for a parallel session. Said in the workflow comment, the changelog, R-265 and the report — none of them claiming it is answered. GOLDEN 0.210.0 baked, published, round-trip verified, NOT VOUCHED. The currency gate went red the moment the controller was bumped — correct — and is closed by the bake, never --no-verify. No --no-verify anywhere this session. ⚠ THE AGENT WAS NOT PUBLISHED UNTIL THIS SESSION CHECKED, AND IT MATTERED. R-221's fix is in the AGENT, and a fresh install takes its agent from the Day-0 manifest. The binary had been hand-deployed to felhom-pve and never published, so agent_version 0.128.0 was not selectable and a fresh install would have received 0.127.0 — the golden would have carried the controller fixes and NOT the one the headline defect needed. Caught by checking each Day-0 value was FETCHABLE rather than assuming. Published from the live-deployed bytes, sha-verified across the hop first. Registers. R-221, R-259, R-258, R-265 CLOSED. R-266 MINTED (READY): the failed root statfs still travels to the hub as a 0-of-0 disk; ranked LOW because it is the quiet direction — it can only miss a true alarm, never raise a false one — and it is now a two-repo wire change governed by G-1's gate. Highest ID moved R-265 -> R-266. CONTEXT S-39 rules the convention this project was missing: "we do not know" is never drawn as "fine", and the codebase has ONE way of saying it — an explicit ...Known bool companion checked in the template. ROADMAP G-3 was explicitly blocked on that decision and is unblocked; what remains there is a survey-and-convert of existing sites, not the gate. Capability map row 93 CHECKED and it was NOT claiming something untrue — it is about the operator notification path. But its narrative ("the page you open to ask whether ONE app is backed up") invites the wrong reading, and the adjacent thing WAS false until v0.210.0, so the row now records that the two halves disagreed and only the operator half was true. Six red-proofs across the two code repos, each with the mutation asserted applied. The one that matters: Part 1 Scenario A FAILED against today's tree, with the intended message. Part 1's operator-present live validation is OWED and is the session's STOP. repo_gates --fast: all 8 OK.	2026-08-08 16:52:39 +02:00