111 Commits

Author	SHA1	Message	Date
admin	0476a8d8e6	SPIKE R-95: the safety net cannot be seen from the box - and that RAISES the urgency ... READ-ONLY STUDY. No code, no version, no image, no golden. No delete verb was issued against any live store. ep0, DooPlex and Peti's box were not touched at all. Q1 FIRST, AND IT DID NOT GO THE EXPECTED WAY. The brief supposed a working seven-day snapshot net might bound the worst case. Measured on BOTH boxes over their own SFTP credential, with a positive and a negative control on each: NO .snapshots is visible to either sub-account - not in the account home, not inside the repo - and the account is jailed at /. Either none exist or a sub-account cannot see them, and the second is not a reprieve: a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel. The register's claim rests on nothing that was checked. The row has NO R-number, so nothing can cite it; its "confirm tomorrow" was 2026-07-27, 36 days ago; and the DUE-CHECKS block built for exactly this (R-341) is EMPTY. R-95's word "ARMED" is withdrawn pending R-429. The confirming field is a Hetzner API field, so this spike STOPPED at the section 11-D fence and left it for Viktor - ten minutes in the panel, and it re-ranks everything. Q2: TEN verbs, not nine. `check` was missing from the brief's list; `dump` is not a verb (it is a progress phase constant) and was withdrawn. There are TWO `forget --prune` sites - offbox.go:1388 AND offbox.go:1759 - and disarming one without the other reproduces R-191 exactly. Q3 (documented, from our own API mirror): AccessSettings has five booleans and `readonly` is the only permission axis. No append-only. So the PBS shape does NOT transfer - PBS is a server that can refuse; a Storage Box is a filesystem that runs nothing. Q5 (measured, faithful append-only model, both controls passed first): withdrawing delete does NOT wedge the store - restic treats a dead owner's lock as stale and proceeds. The constraint everyone feared is not the blocker. But `unlock --remove-all` printed "successfully removed locks" while the lock was still there, and resticStep's crash-lock self-heal is built on that call - R-430. Q6 (measured, with a control): restic 0.14.0 DOES speak rest:. Append-only is a rest-server flag, not a restic one. Q7 (measured): detection is nearly free. snapshot_count already reaches the hub and the hub APPENDS reports, so the history to compare against is already on disk. RECOMMENDATION: answer Q1 today (Viktor, ten minutes), then build detection, then move retention off the box. Defer the transport change until Q1 is answered. Register: OPEN 179 -> 181. Filed R-429, R-430; R-95 updated and kept OPEN. 07 row 10's status is deliberately UNCHANGED.	2026-09-01 13:55:35 +02:00
admin	1e6c387a0b	R-404 CLOSED with the ruling; R-417 CLOSED by cause removal; R-418/419/420 filed ... THE RULING WAS NEITHER OPTION AS FRAMED. Both offered answers - narrow the gate, or leave it and write waivers - argued about the gate, and the gate was never the problem. DIAGNOSIS, from live source: golden_currency_gate.py never looks at the push. It compares the controller's newest CHANGELOG heading against this repo's bake evidence and returns the same verdict whatever you are pushing - correct for a standing invariant, wrong as a push gate. And controller_gates.py had NO golden-currency entry at all. So the repo where a release happens never checked, and the repo that cannot create the debt was refused on every push. 18 of the last 24 pushes here touched no code - measured, and the new classifier agrees EXACTLY - most of them by construction, because the controller's code is in one repo and its register lives in this one. SIX of those 18 were bake records, so the push that PAYS the debt is itself documents-only: the gate was blocking its own cure. Not the waiver its docstring prescribes: that clause was written for a release nobody wants a golden for. R-417 was a release we DID want a golden for, on a night the runbook forbade baking. A waiver would have recorded a lie. RULING: block the push that can create the debt, notify the push that cannot. The gate's logic, exit codes and wording are BYTE-IDENTICAL. Only the consequence changed, for one gate, on one kind of push, with a loud ADVISORY block so nothing goes quiet. R-242's vouch half is amended in place to say it is UNTOUCHED and still open - a baked-but-unvouched golden still passes both the gate and the new notice. Do not read R-404's closure as closing it. FILED: R-418 - this runner's docstring listed ELEVEN gates while THIRTEEN were registered; one-register and closed-register ran undocumented since 2026-08-24. Enumeration fixed here, the correspondence is still unenforced. R-419 - observations_gate.py accepts an item whose body merely CONTAINS "NOT-A-FINDING", even in prose disclaiming it; found by accident when a planted test observation passed and my live validation proved nothing. R-420 - controller_gates.py could not express a non-blocking gate at all before today. Register: OPEN 171 -> 172, CLOSED 158 -> 160.	2026-09-01 12:01:10 +02:00
admin	63eff21a5c	golden 0.232.0 baked, vouched, floor raised — and it carries TWO releases ... GOLDEN_SHA256 5f8a53ed5b19a6cb2006298ce6239f6fca2b990cc3ef6eada89f602801ca91b8, 657 494 489 B. 0.231.0 was never baked, so the fleet went 0.230.0 -> 0.232.0. THE CHECK THE 0.230.0 BAKE SKIPPED, AND THIS ONE DID NOT: the bake script's fingerprint was compared ACROSS THE HOP - 7b0fb5cf...73b6a1 on DooPlex and inside the VM. The previous bake recorded only the DooPlex-side hash and said so; this one is a measurement. Three independent readers agreed before anything was vouched: the bake's own print, the round trip of the published bytes (HTTP 200, 657494489 B, same sha, hashed from what was downloaded), and the hub's Day-0 dropdown reading Gitea on a different code path. The delivered artifact names its own controller - ./etc/felhom-controller-image reads felhom-controller:0.232.0 - with 19382 entries under var/lib/felhom/docker/. Both pre-gates were shown able to see something before their zeroes were believed, and the manifest was RE-READ after vouching rather than trusted from the 303 flash. DELIVERY WAS ACTUALLY EXERCISED. Both boxes had been hand-deployed during validation, so the floor had nothing to move. Rather than report delivery untested, demo-felhom was rolled back to 0.231.0 and the chain run for real - it moved itself in ~20s: 10:47:16 controller-swap: image file written, restarting bootstrap target=...0.232.0 10:47:26 controller-swap: new controller healthy target=...0.232.0 And this is the first bake golden_currency_gate.py actually gates: it now reads the GOLDEN_SHA256 line out of the bake log rather than matching a directory name (R-410, shipped hours earlier the same day). All 13 felhom.eu gates are green, golden-currency included, for the first time since v0.230.0 was released.	2026-09-01 10:50:54 +02:00
admin	f8f9ffdf2b	STATUS: R-414 is the morning's first item - the new nightly check cannot run on demo-felhom	2026-09-01 06:07:52 +02:00
admin	7ee25925f9	R-87 CLOSED: live evidence, capability row, architecture verdict, registers ... Controller v0.231.0 + hub v0.110.0, both deployed and verified on demo-hp. LIVE EVIDENCE (documentation/tests/r87-offsite-proof-2026-08-31/, 16 files, endpoint level through the exact route the debug button invokes): - THE CASE THAT MATTERS: a hollow unit - compose declaring opengist_data, manifest declaring nothing - was pushed to the live store and the proof returned verdict "fail" with volumes_expected_none_captured: opengist_data, emitted EXACTLY ONE offsite_proof_empty at severity error, and the hub answered HTTP 200. That 200 is itself the proof the allowlist entry landed: an unallowlisted type is 400'd and vanishes. - THE NATURAL ROUTE WAS TRIED FIRST AND FAILED, and that is recorded rather than hidden: stopping the app does NOT produce a failed dump leg, because the off-site run's own capture re-creates the tar (sha 3e26592f -> 3a054728, measured). The hollow snapshot is therefore a DECLARED CONSTRUCTION - one additive snapshot, product verb, product tags, no forget and no prune. State restored: the product's own run made a healthy snapshot the newest again and the proof then passed opengist. - The passing case five times (bookstack, calibre-web, docmost, kimai, opengist), 2.2-4.0s each, matching the spike's measured band. - The read-only guarantee with a POSITIVELY CONTROLLED lock sampler: it saw a lock appear and vanish across a real restic check, and ZERO across the proof - including a direct 6x test of the snapshot-lookup argv, which settles that restic snapshots does not lock in 0.14.0 either. - Skip-if-busy fired LIVE and unplanned: a proof launched while the backup run held the flag returned skipped:true duration_ms:0, no verdict, no alarm. - The customer's own verification copies were untouched throughout, which is the safety property the separate proof root exists for. ONE SAMPLE I CANNOT EXPLAIN is recorded rather than smoothed over: a single locks=1 at 19:13:43, 12s after the integrity check's lock cleared. Two independent tests exclude the proof; I did not establish what it was. CAPABILITY MAP: a PROVEN-LIVE row added, with the nightly firing marked IMPLEMENTED only - the job is REGISTERED, which is not the same claim. 07 section 8 MATRIX ROW 4 WAS NOT MOVED, deliberately, and section 10.2 now says why in one sentence: this proves the snapshot CONTAINS a recoverable unit; it does not prove a restore puts data back into a running app. Without that sentence the new green tick reads as covering the drill. REGISTER: R-87 CLOSED and compressed into CLOSED-ITEMS.md. OPEN 172 -> 171, CLOSED 151 -> 152. No new rows minted. R-408 and R-409 stay open and are referenced by this work. golden-currency is RED and it is a DECLARED, EXPECTED debt: v0.231.0 is released and the newest golden carries 0.230.0. The fleet is on 0.230.0; demo-felhom does not have this job. A golden carrying 0.231.0 is OWED and it is Viktor's call (R-242). This push uses --no-verify for that reason - bypass #8.	2026-08-31 21:32:06 +02:00
admin	2263245cf2	golden 0.230.0 baked, vouched, floor raised - demo-felhom moved itself off the R-403 build (R-410 filed) ... GOLDEN_SHA256 9287f7cef5f13166276e8406005e3f28004004510c5184f1c1c7377f7aafad2e, 657 873 700 B. Evidence documentation/tests/golden-0.230.0-2026-08-31/. WHY IT WAS OWED: the newest golden was 0.229.0, which IS the build R-403 says deletes a good copy. Every fresh install and the whole fleet floor still carried it. golden_currency_gate.py had been red across dddcc80, 6e550ae, 130f7a6 and 32a4c35. THREE INDEPENDENT READERS agreed before anything was vouched: the bake's own print, the round trip of the PUBLISHED bytes (HTTP 200, 657873700 B, same sha), and the hub's Day-0 dropdown reading Gitea on a different code path. And the delivered artifact names the controller it will start - ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.230.0, with 19382 entries under var/lib/felhom/docker/. BOTH PRE-GATES were shown able to see something before their zeroes were believed: the 404 pre-gate, and the token-leak grep which returns 0 on the committed log and 1 on a seeded throwaway copy. The transient unit's own properties were grepped for the token too - 0, with the same seeded positive control returning 1. Acceptance markers counted on the COMMITTED log: 1/1/1/1 present, 0/0 absent, and the zeroes are believable because the same including-mount-point pattern returns two real lines on that file. THE VOUCH IS A THREE-FIELD CHANGE and only one field moved, which is stated rather than left to look careless: golden_version 0.229.0 -> 0.230.0; agent_version 0.130.0 and min_agent 0.129.0 UNCHANGED because v0.230.0's CHANGELOG header says MinAgent 0.129.0 and 0.129.0 <= 0.130.0, so this is not the R-216 shape. The 303 flash was not treated as proof - the page was re-read and golden_behind_fleet confirmed absent. THE FLOOR is a separate setting and was raised on the operator's explicit answer: min_controller_version 0.229.0 -> 0.230.0. THE POSITIVE OBSERVABLE, from the agent's own journal on demo-felhom, which was still running the defective 0.229.0: 16:21:30 controller-swap: image file written, restarting bootstrap target=...0.230.0 16:21:40 controller-swap: new controller healthy target=...0.230.0 Both boxes now 0.230.0 healthy. Honest note: the polling loop's first read already said 0.230.0, so the transition was not seen by the loop - the journal is the evidence. R-410 FILED, found while the gate went green: golden_currency_gate.py is satisfied by a DIRECTORY NAME (EVIDENCE_RE against os.listdir, :89,:123). I created the evidence directory before the bake finished and the gate would have passed at that moment. It already declares that it does not check the vouch; it does not declare that the bake check is a filename check. Fix: read the GOLDEN_SHA256= line out of the directory's bake.log, with a red-proof on an empty directory. R-242 updated - seventh debt, paid the same day, twice in one day. Teardown: pct destroy 9100 --purge, shred -u AFTER the log was copied out, poweroff, qemu confirmed exited with ps -eo comm (not pgrep -f, which self-matches), disk reverted to virgin. All 13 gates green - the first push this session that needed no --no-verify. Ceiling R-409 -> R-410.	2026-08-31 16:25:35 +02:00
admin	130f7a6eba	R-87 SPIKE: measured, do not build it as written (R-407..R-409 filed) ... Spike. NO production code. No version bump, no build, no deploy, no golden. felhom-controller and felhom-agent were READ ONLY. The fleet stays on v0.230.0. Q1 restic is 0.14.0 (go1.19.8, bookworm 12.15) - the four source comments asserting it are CONFIRMED, not corrected. Q2 --verify DOES exist and is NOT a content check. Red-proof: one byte changed in a restored 160 MB tar with size and mtime preserved passed clean, rc=0. Verify took 131 ms on a 213 MB / 7-file tree, which cannot be hashing. A size or mtime mismatch causes a silent re-download, not a failure. Controls: --target 1 hit, four post-0.14 flags and a nonsense string 0 hits each. Neither --verify nor --no-lock appears anywhere in the controller source. Q3 no reference for "correct" exists. restic ls --json carries no content hash in 0.14.0, and the unit manifest hashes 4918 B of a 213231242 B unit - 0.0023 percent, the config files and not the dumps or the tars. R-409. Q4 it is CHEAP. All 8 apps / 774378123 B logical restored back to back in 25 s, against 40257 ms for the weekly 100 percent check beside it. Individual restores 2253-3978 ms regardless of size: cost is per-snapshot round-trip plus ~1 s per 200 MB. Peak scratch is the app's full logical size. The 1.1 MB restic cache is index only and hides nothing (--no-cache 5423 ms vs cached 3198 ms, trees byte-identical). Q5 skip-if-busy stays right at 25 s against a 2m52s nightly backup. But RestoreOffboxScratch takes NO acquireRunning, while offbox_integrity.go:28 asserts every off-site operation does. R-408. Q6 observed with a positively-controlled lock sampler: restic restore takes NO lock; restic check DOES (locks 0 -> 1 for nine samples -> 0 across the check, zero across two restores). The product writes anyway - unlockStale runs `restic unlock`, a delete verb, before every restore (offbox_restore.go:289). The task's lead was right in direction and wrong in mechanism. R-95's constraint IS satisfiable: --no-lock plus skipping unlockStale writes nothing, and both mechanisms exist unused. Neither was fixed - the task forbids it. offbox_integrity.go:255's "It NEVER writes to the repository" is R-407. Q7 THE DECIDING ONE: of R-353/354/356/358/403 an unattended scratch-restore would have caught ONE (R-356). The value is elsewhere, and the weekly check structurally cannot reach it: `check` proves the stored bytes are the stored bytes, never that we stored the RIGHT thing. A hollow unit backs up, checks at 100 percent and restores cleanly and recovers nothing - R-403, measured in bytes on 31 August. RECOMMENDATION: option C, the NARROW test - one app a night, restored to scratch, checked against its own manifest.json through the existing unitCarriesData, scratch deleted, the SNAPSHOT recorded as the proof. Options A (do not build) and B (scheduled attended drill) considered explicitly; B is weakest because it is what already happens. R-87 should be RE-SCOPED, not built as written, and that is Viktor's call - the row stays open carrying the verdict and STATUS.md item 4 asks it in plain words. Also corrected in 07-backup-architecture.md: matrix rows 4 and 10 both said "the depth that ships ON does not re-read pack contents (R-399)". R-399 CLOSED in v0.228.0 and the depth is 100 percent. Two stale cells, fixed, and the spike verdict added beside them. Row 4's verdict is UNCHANGED by the spike and now says so. Teardown: all three layers, none of them "nothing was created" - 6 files on the PVE host, 9 in the guest, 5 plus 2 run-flags in the container, all removed and verified empty. The four scratch directories this session's restores created were removed; three that pre-date the session were left alone. Two state changes recorded rather than hidden: the control integrity run recorded its verdict (depth structure -> 100%, due-ness +7 days), and four restores appear in the controller log. Nothing was written to the off-site repository by hand. Evidence: documentation/audits/evidence-spike-restic-restore-2026-08-31/ - 31 files, every one pulled off the box BEFORE teardown (R-320). golden-currency is RED at this commit and was already red at dddcc80. Pre-existing, not this session's debt. Second --no-verify push of the day for that reason; R-404's count goes six -> seven and its row says so. Ceiling R-406 -> R-409.	2026-08-31 15:59:09 +02:00
admin	dddcc808be	R-403 CLOSED (controller v0.230.0), R-404 filed as a decision for Viktor ... 07-backup-architecture gains section 8.2, placed beside row 5 on purpose: the derived-copy rebuild rule is UNCHANGED and section 8.2 names the single exception, so a future reader who finds RunTier2 skipping a leg does not fix it back. It carries the measurement (120 082 104 B -> 7 036 B on the shipped v0.229.0), the four-case table, why hollow is a manifest question and not a size question, why the data legs are deliberately not guarded, and why the capture job is not guarded either. 00-capability-map: the Tier-2 row's status does NOT move, stated explicitly rather than left ambiguous. R-403 removes a way the route could be DESTROYED between uses; it does not change what the route can be relied on for. Register: R-403 CLOSED and compressed into CLOSED-ITEMS (594 -> 593 open lines). R-404 FILED as a DECISION and deliberately not acted on - should a documents-only push be subject to the golden-currency gate, now that it has been correctly bypassed six times? Both sides stated, plus what happens if Viktor does nothing. The gate was NOT changed. R-242: seventh conviction, and the FIRST where the day-0 ground does not apply - R-403 is a defect in the nightly Tier-2 copy, which a day-0 box starts running on its first night. This push uses git push --no-verify, declared here and in felhom-controller/REPORT.md. A golden carrying 0.230.0 is owed and is more urgent than the previous six. STATUS: the R-403 item moves out of 'Broken' into what works, in plain words; the delivery item now says a golden is owed and that the fleet carries the defect; R-404 goes into the decide section with its do-nothing outcome. Drill evidence: nine phase logs, including the two things that went wrong (a repair whose rsync was not installed in the guest and silently did nothing, and a session that expired mid-run so a POST did nothing).	2026-08-31 14:39:29 +02:00
admin	83ff9e8e38	golden 0.229.0 baked, vouched, floor raised — R-242's sixth debt PAID the same day ... GOLDEN_SHA256 39aa886df77b21757aef3b298a389343dc0df5134bb0f14e8f92a451d7bdae87, 656 864 331 B. The evidence is the ROUND TRIP, not the build log: the published bytes were downloaded back and match the bake on both size and sha, and ./etc/felhom-controller-image read OUT of the downloaded archive says felhom-controller:0.229.0 - the delivered artifact naming the controller it will start. A THIRD independent reader agreed before anything was vouched: the hub's own Day-0 dropdown read the same sha straight from Gitea, a different code path. Both pre-gates were proven able to see something before their negative results were believed - the 404 pre-gate against a 200 from 0.228.0, and the token-leak grep against a seeded throwaway copy. Acceptance markers counted on the COMMITTED log: 1/1/1/1 present, 0/0/0 absent. The vouch is a three-field change, checked rather than assumed: MinAgent 0.129.0 read from the golden's controller CHANGELOG header, agent_version 0.130.0 >= min_agent 0.129.0 (not the R-216 shape), agent_sha256 and wrapper_sha256 carried through explicitly because the handler clears a field it is not sent. Verified by re-reading the manifest, never by trusting the flash. The R-120 gate PASSED rather than being bypassed - fleet newest 0.229.0, golden 0.229.0. The floor is proven ACTING, not merely set: demo-felhom self-updated 0.228.0 -> 0.229.0 and logged settle-gate GO at/above floor 0.229.0. Nobody deployed to that box. Both demo machines now carry the Tier-2 unit restore. golden_currency_gate.py went red -> green; the --no-verify bypass declared on c2de785 is now historical. R-242's other half is untouched and still open: nothing gates the VOUCH itself. Teardown: build guest 9100 destroyed --purge, token/runner/script/log shredded AFTER the log was copied out, VM powered off, qemu confirmed gone from ps -eo comm, disk reverted to virgin.	2026-08-31 12:38:01 +02:00
admin	c2de785bf2	R-102 + R-103 CLOSED (controller v0.229.0) — architecture, register, STATUS, drill evidence ... 07-backup-architecture: 6.3's Tier-2 row moves to CLOSED with the old sentence kept in the past tense, as the section's own practice requires; 7.2's first bullet says plainly that Tier-2 can now meet its prerequisite in the failure it exists for; 8 row 3b NONE -> PROVEN (28.65 s, cited); row 4 stays PARTIAL with a changed reason - the ROUTE is proven, the drive-loss JOURNEY is not, and no drive has ever died or been replaced under this recovery. 8.1's blanks updated per row. 6.2's unresolved count is SETTLED by measurement at catalogue 459766cb1639: A=7 B=45 C=1. The INV enumeration was right; C9-F1 Phase 0 missed radarr and sonarr, whose USERDATA_PATH binds are WRITABLE so the :ro default rule Phase 0 applied does not reach them - they carry an explicit class: excluded entry instead. Class C is bentopdf. No catalogue file was changed. 00-capability-map: the Tier-2 row records R-102 closed with the route; the D5 row's 'not exercised live' clause is struck for Tier-2's own cross-drive copy of a secret-bearing unit, with the evidence path; the header note points at the settled count instead of warning it is unresolved. Register: R-102, R-103 and their C9-F4 / C9-F1b aliases closed and compressed into CLOSED-ITEMS (596 -> 593 lines, each naming git show 1623a4d5b5 for the original). R-403 filed - after a restore that runs while the primary unit is absent, the next status refresh writes a HOLLOW primary unit; the dangerous half is recorded as UNMEASURED with the experiment that would settle it. R-242: sixth conviction of golden_currency_gate. THIS PUSH USES git push --no-verify, declared here and in felhom-controller/REPORT.md - a BYPASS, not a waiver. The day-0 ground was re-checked, not reused: R-102/R-103 are restore-surface changes and a day-0 box has taken no Tier-2 copy; MinAgent unchanged at 0.129.0. OWED: bake a golden carrying 0.229.0, vouch it, raise the floor. Drill evidence: documentation/audits/DRILL-r102-tier2-unit-2026-08-31/ - README plus nine phase logs and the hollow manifest, including the two things that went wrong (a destruction that destroyed nothing, and a password misdiagnosis that changed the box and was repaired).	2026-08-31 12:21:52 +02:00
admin	1623a4d5b5	golden 0.228.0 — baked, published, round-trip verified, vouched, floor raised ... Bake: build-golden.sh v3.0.0 in the drill VM, reverted to virgin and cold-booted. GOLDEN_SHA256 76a3a98b9e7cc23bf8ae51b38a6272f576df285cb34cd22235ac3f06a31e53ec, 658 079 744 B. All four acceptance markers counted 1; excluding/FATAL/mp1 counted 0. The 404 pre-gate was proven to work before its 404 was believed — the target URL 404'd while the existing 0.227.1 package 200'd on the same command. The evidence is the ROUND TRIP: the downloaded bytes match the bake's size and sha, and ./etc/felhom-controller-image read OUT of the downloaded archive says gitea.dooplex.hu/admin/felhom-controller:0.228.0. Vouch: three fields together — golden_version 0.228.0, agent_version 0.130.0, min_agent 0.129.0 (read from the controller CHANGELOG header, not assumed); wrapper_sha256 carried through explicitly. agent >= min_agent, so not the R-216 shape. Verified by RE-READING the manifest, never the flash. R-120 gate passed. Floor raised 0.227.1 -> 0.228.0 (impact preview {"below":3,"valid":true}). The floor is ACTING: demo-felhom self-updated 0.227.1 -> 0.228.0 in under a minute and re-registered offsite-integrity by itself. Both demo boxes now re-read their whole off-site store on the weekly check. Token hygiene: file->file scp, runner script inside the VM, unit properties grepped 0. The leak grep on the committed log was proven with a planted copy (1) before its 0 was believed. Teardown: guest 9100 purged, secrets shredded after the log was copied out, VM off, disk reverted to virgin. golden_currency_gate.py red -> green. All 12 felhom.eu gates OK.	2026-08-31 10:53:43 +02:00
admin	77a5a1154b	docs: controller v0.228.0 — R-399/R-400 closed, R-401/R-402 filed ... STATUS.md: header said 2026-08-23 over a 2026-08-30 body, and two "Waiting on you" items were both numbered 4 — both fixed. R-399 leaves that section (decided and shipped); the depth change is stated in plain words and the remaining items each say what happens if Viktor does nothing. 00-capability-map.md: the off-site verification row now carries its DEPTH, and its live citation is the 2026-08-31 run at 100%. The weekly firing at the new depth stays IMPLEMENTED, not PROVEN-LIVE. 07-backup-architecture.md §10.2: R-399 recorded closed, with the one sentence that stops it being turned back down — the structure check PASSED a size-preserving pack corruption. R-87 untouched and still OPEN. Register: R-399 and R-400 compressed into CLOSED-ITEMS.md with their reasoning kept and 300d7e8 named as the commit holding the originals. R-401 filed with a TRIGGER (the slow-check WARN firing) rather than a date. R-402 filed: the integrity verdict and its depth are on the wire and no hub surface reads either. OPEN 166 -> 165, CLOSED 148 -> 150. wire_contract_gate.py: offsite.last_integrity_depth allowlisted WITH ITS REASON beside its sibling last_integrity_ok, both to be deleted together when a hub surface is built (R-402).	2026-08-31 10:39:05 +02:00
admin	db0812b6f2	Golden 0.227.1 baked, vouched, floor raised — and the floor delivered the new job by itself ... Second full delivery of the day. golden_currency_gate.py went red -> green on the same command, so the --no-verify bypass declared on the previous push is now historical rather than standing. GOLDEN_VERSION 0.227.1 GOLDEN_SHA256 66754491dc9bd0130ef8ded9562f63c53a5ffdcfd91baa551141e55fa083ea32 size 657 403 203 B baked gitea.dooplex.hu/admin/felhom-controller:0.227.1 MinAgent 0.129.0 (read from the controller CHANGELOG header, not assumed) THE EVIDENCE IS THE ROUND TRIP. The published bytes were downloaded back -- size and sha256 identical to what the bake reported -- and ./etc/felhom-controller- image was read OUT of the downloaded archive: felhom-controller:0.227.1. That is the delivered artifact naming the controller it will start, from the bytes a customer's box would actually fetch. Markers counted: docker OK (overlay2 = 1, mount point rootfs = 1, mp0 = 1, upload OK (HTTP 201) = 1; excluding = 0, FATAL = 0, mp1 = 0. 404 pre-gate passed before the run and the script's own pre-delete agreed, so nothing was overwritten. Three-field vouch, all three checked: agent_version 0.130.0 >= min_agent 0.129.0 (NOT the R-216 shape), wrapper_sha256 carried through explicitly because the handler clears it when omitted. Verified by RE-READING the manifest rather than trusting the flash. The R-120 gate on that POST passed on its own terms rather than being worked around. AND THE LINE WORTH KEEPING. demo-felhom self-updated 0.226.1 -> 0.227.1 in under 30 seconds and then logged: [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-08-31 06:00 CEST A box nobody deployed to now runs today's off-site integrity check on its own schedule. That is a floor DELIVERING rather than merely recording, observed instead of assumed -- and it is the strongest evidence R-242 has carried. Token hygiene: file->file, read inside the VM by a runner script, never on a command line (systemctl show ... grep -c -F token = 0). The leak grep on the committed log was PROVEN TO WORK before its 0 was believed. Teardown: guest 9100 destroyed --purge, secrets shredded AFTER the log was copied out, VM powered off, disk reverted to virgin. R-242 now records the cadence as MEASURED: five convictions and two full bakes in one day. Every bypass declared, every debt paid -- and the pattern the row exists to name is exactly that a release and its delivery are separate acts. Its other half stays open: nothing gates the VOUCH itself.	2026-08-30 21:48:15 +02:00
admin	99af997ab9	R-359/R-397 closed, R-398 corrected, R-399/R-400 filed with measured numbers ... THE MEASUREMENT IS THE STORY, and it re-frames the row it was filed under. A pack was corrupted WITHOUT changing its size; plain `restic check` -- the depth that ships ON -- returned `no errors were found`, exit 0. Only --read-data caught it. So the check that shipped verifies the index, the pack inventory and the snapshot graph, and does NOT re-hash pack contents. R-399 was filed as a bandwidth-and-cadence question; it is more than that, and its row now says so. R-399 gets three MEASURED numbers instead of estimates: store 140 829 678 B / 2651 blobs / 67 snapshots; structure check 35.0 s; curve 10% 35.9 s, 50% 37.3 s, 100% 39.2 s. At this size re-reading everything costs four seconds more than reading none, because the wall clock is SFTP round-trips not transfer. The row states the limit too: these do NOT extrapolate. R-400: the sweep the task asked for found EIGHT dead debug buttons, not one. 24 endpoints referenced in debug.html, 17 dispatched. Single dispatcher, exact match, default NotFound -- so they 404. A third of a debug page does nothing, on the surface an operator reaches for when something is already wrong. R-398 is CORRECTED AND LEFT OPEN, not closed. I filed it yesterday saying resticStep is not a seam so no test can drive a restic path. The layer below it has been injectable since the off-site tier shipped. The row survives as the record that the seam EXISTS so nobody re-files it. 07 gap register: R-359 and R-397 closed; R-87 restated IN PLACE as "AND IT IS NOT R-359" because the two rows are adjacent and a check is not a restore-test. 08 alarm ladder: both event types recorded, including that `ok` is `info` and therefore mails nobody BY DESIGN, and that all three registers were checked and deliberately left alone. 00 capability map: PROVEN-LIVE for the check, the notifier and the hazard control; the scheduled firing is IMPLEMENTED only, because a week has not passed. wire_contract_gate: `offsite.last_integrity_ok` allowlisted WITH A REASON. The gate was right -- the controller emits a field no hub struct can decode. Building the display is a hub change and R-331 ruled that class the operator's decision; the entry says to delete it when a surface exists. This push used `git push --no-verify`. golden-currency is CONVICTED and right: 0.227.1 is released and the golden carries 0.226.1. A BYPASS, not a waiver, and the task spec directs it -- golden and fleet delivery are Viktor's (R-242). It is item 3 under "Waiting on you". Register 163 -> 165 -> 163.	2026-08-30 21:29:37 +02:00
admin	4f875174fe	Golden 0.226.1 baked, vouched, and the fleet floor raised — the debt is paid ... golden_currency_gate.py had been CONVICTED four times today across three controller releases. One bake covers all three, and the gate went red -> green on the same command, which is its proof that it measures something real. THE THREE DECLARED BYPASSES ARE NOW HISTORICAL RATHER THAN STANDING. GOLDEN_VERSION 0.226.1 GOLDEN_SHA256 70ed8e9377dec22a9b493e55f222b0e25a49d7f3caec8c506e0412fd6baefe69 size 657 197 592 B baked gitea.dooplex.hu/admin/felhom-controller:0.226.1 MinAgent 0.129.0 (read from the controller CHANGELOG header, not assumed) THE EVIDENCE IS THE ROUND TRIP, NOT THE BUILD LOG. The published bytes were downloaded back -- size and sha256 both identical to what the bake reported -- and ./etc/felhom-controller-image was read OUT of the downloaded archive: `felhom-controller:0.226.1`. That is the delivered artifact naming the controller it will start, from the bytes a customer's box would actually fetch. Acceptance markers counted, not eyeballed, each string captured from this run's own log rather than paraphrased from the runbook (two of the three the runbook named until R-233 could not match anything the script prints): docker OK (overlay2 = 1, including mount point rootfs = 1, mp0 = 1, upload OK (HTTP 201) = 1; excluding = 0, FATAL = 0, mp1 = 0. The 404 pre-gate passed before the run, so nothing was overwritten. THE VOUCH IS A THREE-FIELD CHANGE AND ALL THREE WERE CHECKED: agent_version 0.130.0 >= min_agent 0.129.0, so NOT the R-216 shape; wrapper_sha256 carried through explicitly because the handler clears it when omitted. Verified by RE-READING the manifest rather than trusting the flash -- golden option 0.226.1 SELECTED, all four shas matching. THE FLOOR IS PROVEN ACTING, NOT MERELY SET. demo-felhom self-updated within 30 seconds: "[selfupdate] Post-update startup: update successful (0.225.0 -> 0.226.1)". Both demo machines now run 0.226.1 and only one of them was deployed to by hand. Token hygiene: copied file->file, read inside the VM by a runner script, never on a command line (systemctl show ... | grep -c -F token = 0). THE LEAK GREP ON THE COMMITTED LOG WAS PROVEN TO WORK BEFORE ITS 0 WAS BELIEVED -- a throwaway copy with the token appended grepped 1, was shredded, and only then was the real log's 0 taken as evidence. Teardown: build guest 9100 destroyed --purge, secrets shredded AFTER the log was copied out (standing rule 5), VM powered off, disk reverted to virgin. R-242's OTHER half is untouched and still open: nothing gates the VOUCH itself.	2026-08-30 20:23:52 +02:00
admin	e027b5d999	Register + architecture for controller v0.226.0 (R-353/357/358/360/396), and R-395 fixed ... Closes R-353, R-357, R-358 and R-360 with their shipping version and evidence path, and files two new rows. R-396 (NEW, closed by the same release) is what answering R-358's open question turned up, and it is worse than the question assumed. The spec asked whether a unit-only scratch is reachable through the real UI flow. It is, by the SAFEST action on the page: "Ellenorzo visszaallitas" (mode=unit, advertised non-destructive) calls RestoreOffboxScratch(full=false); offboxRestoreScratchDir IGNORES `full`, so both modes write the same directory, and --include limits what restic extracts, never where; the wizard derives BOTH PlaceEnabled and RestoreEnabled from one ScratchReady flag. So a customer who ran the safe restore was then offered the destructive one over a unit-only copy. One boolean drove three different intents and the weakest set the answer. R-395 (filed by the spec) is fixed in this commit, not just recorded. STATUS.md said golden 0.223.0 / floor 0.222.0 in one block and demo-hp 0.219.0 / floor 0.218.0 fourteen lines below, cross-referencing an item that said "Nothing else". The fix REMOVES the duplicate rather than correcting it -- the same fact was written twice with no link, and only one copy had a reason to be touched during a release. "What works" now points at the item above instead of restating a version. 07-backup-architecture: four rows added to the 10.2 gap register plus R-396. Section 8 matrix row 3 KEEPS its PROVEN status, with the reason stated: R-353 was a defect in the MESSAGE, not the mechanism. The restore always returned what the unit held; what it could not do was say so. A status that measures whether data comes back must not move because a status line was wrong. 00-capability-map: one new row, and it splits what is claimed. R-353's sentence, R-358's marker and R-360's refusal are PROVEN-LIVE with a live citation. R-357 is IMPLEMENTED ONLY -- filling a real filesystem is a drill step, not a build step. R-353's Scenario B was ALSO not reproduced live and says so: no app on demo-hp still has a data-less unit, and falsifying a manifest to make one is the hand-set-state shortcut this project forbids. This push used `git push --no-verify`. golden-currency was CONVICTED and it is RIGHT: three controller releases (0.224.0, 0.225.0, 0.226.0) and the golden still carries 0.223.0. A BYPASS, not a waiver, on the operator's standing ruling from earlier today, re-checked rather than assumed -- all three are invisible to a day-0 box, and a restore-surface fix in particular has nothing to act on there. The ground expires the moment a release changes first-boot behaviour. Tracked on R-242; ONE bake carrying 0.226.0 covers all three.	2026-08-30 19:39:41 +02:00
admin	ebdc04601d	docs(hub v0.108.0): the delivery grain, the cooldown ruling, and gate 11's first subject ... The alarm ladder gains §6.2 - which events are per-app, per-run, per-tier or coarse, and why the default is coarse. CONTEXT records two rulings: the grain is allow-listed rather than inferred from the payload, with crossdrive_failed as the proof that a payload rule would have been wrong; and a finding recorded only in REPORT.md has a lifetime of one session. R-389 closed and compressed, keeping its rules and naming the commit whose git show returns the full text. R-390 and R-391 left open. REPORT.md is gate 11's first real subject and passes: six observations, two FILED, four NOT-A-FINDING with their reasons. Three of those declarations are things a tidier report would have omitted - the gate's own spec would have passed the item it was built to catch, the burst has no ceiling, and ArgoCD said "successfully rolled out" while still running the old image. STATUS carries forward the one thing outstanding: the controller floor still reads 0.222.0 while the golden reads 0.223.0.	2026-08-23 14:12:23 +02:00
admin	2f7c9a6ce5	docs(R-329/R-386/R-387): the severity contract, the intent ruling, and Part 5 recorded ... The alarm ladder gains the severity contract (the hub's vocabulary is exact, it coerces silently, and three things now hold it) and the intent test with its three-way ruling on unknown. Both marked [DESIGN] with the live measurements. Part 5 is RECORDED AND NOT IMPLEMENTED: the operator's notification philosophy, verbatim, marked plainly as direction rather than current behaviour, with the 12 -> 15 toggle growth as the argument. Filed as R-388, a product decision. R-329 and R-386 compressed into CLOSED-ITEMS with their rules kept and the full-text commit named. R-387 filed closed - including WHY the dispatcher branch was kept rather than deleted, which is evidence (three monitor checkers call ProcessEvent directly) and not caution. The drill record names three things that had to be re-run: an inert red-proof mutation, Scenario G refused twice behind an HTTP 200, and the live Scenario A NOT proving the customer gate because demo-hp has no prefs row at all. Register: OPEN 328325 -> 328132 B, CLOSED 71441 -> 74642 B.	2026-08-23 12:03:49 +02:00
admin	55274d5ef3	R-385: make an UNRECORDED golden fail the currency gate; file R-386; own the alarm ladder ... The gate failed only on `released > baked`, so it could catch a forgotten bake and nothing else. A golden AHEAD of the record passed silently - and that is how controller 0.221.1 was built, baked AND vouched while the newest CHANGELOG heading still read v0.221.0, with every gate green. Reproduced on the real history: newest released 0.221.0 / newest golden baked 0.221.1 -> exit 0. The gate now asks whether the version being shipped is WRITTEN DOWN: the baked version must have its own `## vX.Y.Z` heading anywhere in the CHANGELOG. Membership rather than `baked > released` deliberately - a comparison against the newest heading alone goes green the moment any later entry is written, leaving the unrecorded version permanently unrecorded. INCONCLUSIVE (exit 2) preserved; every refusal names a reason and a route. Red-proofed both directions: old gate/old record exit 0, new gate/old record exit 1, new gate/fixed record exit 0, absent clone exit 2, post-bake exit 0. 08-alarm-ladder.md is new, and its absence was itself the finding: no document owned "when does a broken app raise an alarm?". The rules lived as comments in four packages, each locally correct, with the ordering between them legible only by reading one function top to bottom - which is how R-384 survived review. R-383 and R-384 closed into CLOSED-ITEMS with their rules kept. R-385 filed closed. R-386 filed OPEN: a single-container app stopped out of band raises no alarm, and a comment claims the opposite - measured live, 9 scans, 0 events, against a positive control from the same box 17 minutes earlier. Not fixed here. Golden 0.222.0 baked and published; vouching is the operator's act.	2026-08-23 07:59:52 +02:00
admin	1eb64bec51	R-361 docs: the [FACT], the negative that cancelled Part 2, R-383/R-384, golden 0.221.1 ... 07-backup-architecture.md gains a dated [FACT] on R-361 - a comment asserting an invariant the code did not have, for four months - and a [DESIGN] on the db_dumps decision INCLUDING the trap it created: a stable list lets the already-current early return fire, so per-capture housekeeping must sit above it. 00-capability-map.md records the NEGATIVE from Part 3 so it is not re-derived: a held app does NOT raise the dead-app alarm. It aggregates to unhealthy, which IsDownState excludes. Measured on the shipped build with the scans demonstrably running over it. No suppression was built and no row opened. R-383: the double-failure message names an undo copy that is not there - R-361's own class, one surface over, observed on both 0.220.2 and 0.221.1. R-384: an app whose database has died reads unhealthy and raises no alarm. R-361 closed and compressed. OPEN-ITEMS 325236 -> 327266 bytes. Golden 0.221.1 baked, published and round-trip verified. The golden-currency gate blocked this push and that block is not circular, so it was satisfied rather than bypassed - no --no-verify anywhere in this session.	2026-08-23 00:33:12 +02:00
admin	a8caa0fdde	R-379/R-380 docs: the failure ladder, the drill record, register housekeeping ... 07-backup-architecture.md 6.3 gains a dated [DESIGN] paragraph on replay -> rollback -> hold, including why no engine flag closes it: --single-transaction makes Postgres atomic, MariaDB DDL is not transactional, so the rollback is the fix and the flag is a belt. Drill record for the live walk, including the TWO defects the walk found in the fix itself (a rollback into a re-created container; an operator route that cleared the file while the running controller kept refusing) and the ONE red-proof that PASSED, which is reported rather than omitted. R-379..R-382 compressed into CLOSED-ITEMS.md. OPEN-ITEMS 330683 -> 325236 bytes. STATUS.md restates the outcome and names the next operator step.	2026-08-22 18:40:22 +02:00
admin	8c9f1b798b	golden 0.219.0 baked, published and round-trip verified (NOT vouched) ... Baked in the drill VM per RUNBOOK-manual-build.md 4.0/4.1, carrying controller v0.219.0 (R-356). GOLDEN_VERSION 0.219.0 GOLDEN_SHA256 67b46f78f8ed9c7b1876265ab1bde9ec6798897898b1836acece9f3864a2aeb6 656832571 bytes All five pass markers matched, both negative controls at 0. Verified by ROUND TRIP - the published object downloaded again and its sha recomputed - not by the number the script printed. Both token-leak greps were proved able to convict before their zeros were believed: planted copy grepped 1, shredded, then the 0 accepted. Teardown complete: guest 9100 purged, four secret/script files shredded after the log was copied out, qemu exited, disk reverted to virgin. The revert first refused while qemu held the image, which is the runbook's own no-holder proof. NOT vouched - that is a three-field operator save (golden_version 0.219.0, agent_version 0.130.0, min_agent 0.129.0).	2026-08-22 13:56:33 +02:00
admin	c297b9f85e	R-356 docs: correct R-107 in the architecture, record the design, refresh STATUS, compress the register ... 07-backup-architecture.md: three places said no offsite action unpacks the named-volume tars. R-107 closed in controller v0.218.0; all three corrected with a dated [FACT], the old sentence kept in the past tense. R-102 is NOT closed and the correction says so explicitly. New [DESIGN] paragraph in 6.3: the restore destination is resolved by the same rule as the capture destination, and the wrong-disk refusal applies to apps that have a drive to get wrong. Carries the 13/40 measurement. STATUS.md was internally contradictory - nothing waiting, and one decision waiting, for something the same page recorded as shipped. 218 -> 102 lines; the deciding section now says what happens if nothing is done. R-356 compressed into CLOSED-ITEMS.md; OPEN-ITEMS 327109 -> 325236 bytes. Drill record and 16 evidence files for the live walk on demo-hp.	2026-08-22 13:26:10 +02:00
admin	877fcd2a38	R-354 + R-355 CLOSED, proven live; golden 0.218.0 baked; R-367 filed ... Both of the drill's HIGH findings are fixed in controller v0.218.0 and confirmed on demo-hp with a negative control first — the same planted, hash-recorded fixture run through the same steps on both builds. R-355: paperless-ngx's PostgreSQL was dumped into a directory for a stack that does not exist, so it never entered the recovery unit, the off-site copy or the restore; and because the same wrong name reached writeSafetyDump, a destructive restore took no undo copy and the fail-closed refusal was never reached. Fixed by reading the compose project label. Sweep proven able to convict before its count was trusted: one affected app of 53. R-354: the off-site restore had no named-volume leg. Now it replays them from the scratch unit, before the database and inside the stopped window, and VolumesReplayed reaches the sentence. The half-false comment beside the skip is corrected and the half that still holds is named. Golden 0.218.0 baked and published, sha 8e427869d13eafb71562b77d1535eef6c7f32b4db24f659b988ec6d7db8f478b, verified by round trip on the downloaded bytes. NOT vouched and the floor NOT raised — both are the operator's decision, and raising the floor is what puts this on demo-felhom, which is still on 0.217.0 and still has both defects. R-367 filed: the dumps already written under the wrong name are stranded. Nothing deletes them (an existing guard), they are adoptable by hand, and doing it automatically would be a migration. Ceiling R-366 -> R-367.	2026-08-22 10:11:46 +02:00
admin	d895d9f7dd	STATUS + capability map: narrow the end-to-end off-site claim to the leg it was proven on ... The 2026-08-04 row claimed "a customer's file survives a machine rebuild and comes back — the whole off-site story, end to end". Tonight's drill shows that holds for the declared-userdata leg of a drive-declaring app and for nothing else: the off-site restore has no named-volume leg (R-354), and refuses outright for the 40 apps that declare no data drive (R-356). Since that class keeps ALL its data in named volumes, the end-to-end story is unproven there and disproven for the volume leg generally. The escrow/key half of the row is untouched and still stands. STATUS.md also corrects the fleet pair it still named (0.214.0/0.129.0 -> 0.217.0/0.130.0) and records that demo-hp's off-site had been silent since 9 August.	2026-08-21 23:34:21 +02:00
admin	910fd91124	agent 0.130.0 published and vouched; R-347 closed, R-349 + R-350 filed ... Released via scripts/release-agent.sh: tag v0.130.0 at 7569f34, sha256 a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3, verified by independent download and reproducible byte for byte with -trimpath -buildvcs=false. Vouched agent 0.129.0 -> 0.130.0 in the Day-0 manifest. Only the agent fields changed: min_agent stays 0.129.0 because it states what the GOLDEN CONTROLLER requires, and raising it would have HELD the floor for every box below 0.130.0. Global floor untouched at 0.216.0 -- and on hub v0.106.0 it is a separate form with its own action, so publish-train rule 2's hazard no longer exists in the shape its incident describes. No --no-verify: the CHANGELOG heading was flipped only after the tag and package existed, so release-complete passes on the real artifact. R-349: the fleet was running a DIFFERENT binary under the same version name -- the proof deploy was a hand build, the release is -trimpath. Self-update could never have corrected it, because every version check compares the string. Both boxes reinstalled from the downloaded package. The proper fix exists in miniature as wrapper_sha256 and was never extended to the agent's own binary. R-350: I printed the hub password into the session transcript via curl -w '%{redirect_url}' -- the hub answers 303 and curl re-attaches the credential. Not in git, not in any committed file, not in the evidence directory. Rotation is the operator's call. ep0 closes at fd 17, ESTAB 0, CLOSE-WAIT 0 -- its t0 baseline -- and was read-only for this entire arc.	2026-08-20 12:54:54 +02:00
admin	57dd62b097	R-344 fixed and proven on both boxes: ep0 is back to fd 17 from 415 ... P1, outcome (i) in one second: replacing the agent on demo-hp released exactly its 199 established connections (ep0 fd 415 -> 216). CLOSE-WAIT stayed 0, so outcome (ii) does not exist and gets no row -- ep0 reaps on peer FIN correctly, and the 543 CLOSE-WAIT at the 08-18 wedge has another explanation. P2, 1.03 h (operator closed the >=4 h window early, so no daily rate is extrapolated): control +4, fixed +0, with each box making exactly 4 /snapshots and 4 /version calls. Same cadence, same work: 4 cycles -> 4 leaks vs 4 cycles -> 0. The fixed box's cycles are in ep0's log, so the zero is the fix and not a stopped agent. P3: the second box took ep0 from 220 to 17 fd in under two seconds. 17 is precisely the t0 baseline of 2026-08-18 09:51:22Z. Corrects a claim this session made earlier the same day: the accumulated descriptors did NOT need an ep0 proxy restart. They were held on both sides. ep0 was read-only throughout; its PID never changed. R-344 updated and left OPEN (unpublished is not delivered). R-336 re-scoped -- its old next-step would have fixed nothing while looking like a failed fix, and it is now a scaling row (~25 req/s at fifty customers). R-347 filed for the delivery gap (Viktor decides). R-348 filed: an agent restart blanks the reported backup list for ~18 h and the Store comment calls it unaffected -- blinds no alarm, checked not assumed.	2026-08-20 12:39:32 +02:00
admin	19672e685e	SPIKE ep0 connections: the leak is felhom-agent's, not the poll rate ... R-341's first dated check, taken at +46.2 h: fd 17 -> 405 over 166,251 s = 201.6/day. Pre-registered range was 370-450; observed 388. UNCHANGED, as predicted. CLOSE-WAIT is 0 -- absent entirely, not merely flat. Q1: exactly two peers, 194 each, no third party. Q2: outcome (a). ep0 388 = 194 + 194 on the boxes, twice, and the four new sockets carry the same source ports on both sides. 0 closed in 31 min. The finding: all 388 are held by felhom-agent. pvestatd and proxmox-backup-client made 162,404 requests and leaked zero. Mechanism is a per-cycle http.Transport with a zero-value IdleConnTimeout that nothing ever closes (internal/pbs/client.go:56, main.go:1486). R-336's premise does not survive this -- cutting the poll rate would have fixed nothing. Q3 NOT measured: Phase C held at STOP 1, prediction pre-registered first. Part 0 captures the due-checks gate's first conviction on a real overdue date (rc=1, names R-341, sole failure among 10 gates). Row cleared at Part 4, after the result was recorded in R-341, not to make a push work. New: R-344 (the transport leak), R-345 (hub/Makefile pushes :latest), R-346 (ActiveEnterTimestamp reads 5h56m early -- NRestarts is still 0).	2026-08-20 10:41:37 +02:00
admin	ab2262c91c	hub v0.106.0: report loss of visibility into the off-site stores (R-339) ... THE GAP, measured not supposed. On 2026-08-18 ep0's PBS proxy was wedged for 9 h 37 m and the hub emitted NOTHING on the operator channel. Both box checkers hold their last snapshot and return silently on a failed fetch -- correct for a FILL signal, since a missing reading must never be read as 0%, but it makes a dead off-site endpoint and a healthy one indistinguishable. The only mails that morning came from the boxes' own backup failures, and only because the WEEKLY offsite run happened to land inside the window. Two days earlier nothing would have fired at all. REACHABILITY is now a second, independent signal on both checkers: consecutive failed fetch windows, reported past a default 3 windows (~30-45 min) as pbsdr_box_unreachable / offsite_box_unreachable (warning) on the customer-less pbsdr-box / pool-box scopes, each with a paired *_recovered all-clear. Tunable via alerting.box_unreachable_windows (0/invalid -> 3). THE FILL LOGIC IS UNTOUCHED. No threshold, throttle, band or escalate-once behaviour changed; a degraded read still drives no transition. Three decisions a later reader would otherwise "fix" back, so each is argued in-code: - the unreachable event REPEATS rather than escalating once. The band shape would give exactly ONE mail at ~minute 30 of a nine-hour outage, and one mail is missable. It leans on the dispatcher's 1 h operator cooldown to become an hourly "still blind" heartbeat. - ErrUsageUnsupported is NOT blindness: an old ep0 answers "no such op", which means we reached it. Counting it would alert for days on a healthy pre-update endpoint. - born-blind is reported: the counter is not gated on having a snapshot, so a hub restarted INTO an outage still speaks. last_ok is OMITTED rather than zero-valued -- a fabricated timestamp reads as "it was fine until then". Both recoveries are severity "info" and severityNotifies drops "info", so they are registered in recoveredPairedDownTypes or the operator hears that the tier broke and never that it healed. A cross-package test drives ProcessEvent and asserts an actual operator MAIL, not a map entry -- a green checker test proves nothing about the seam (agent v0.91.0 shipped fully green with SetAuthSink never called). Tests: box_reachability_test.go (Scenarios A-F) + dispatcher_box_reachability _test.go (wiring). Three red-proofs run and reverted, each seen failing with a message naming the right cause: threshold 3->1, the sentinel counter guard, the pairing entry. Register: R-339 filed and marked SHIPPED (PROVEN-LIVE still owed -- no real or constructed outage has exercised the emit path, and one cannot be manufactured against Tier-2 ep0). R-340 filed: the reachability read rides ep0's LOCAL API daemon, which the incident explicitly cleared, so this check would have shown GREEN for all 9 h 37 m -- the honest boundary, recorded rather than glossed. R-336's next-step corrected: pvestatd's interval is NOT tunable (Proxmox staff have said so); the only lever is disabling the storage entry, which collides with the agent's consume-the-one-time-secret path. Doc-only, no agent code touched.	2026-08-18 19:27:33 +02:00
admin	0a5e9b14dc	due-checks gate (R-341), floor raise recorded (R-343), snapshot coverage (R-342) ... PART 1+2 — dated checks stop being wishes. R-341 booked two measurements as prose in a register row; nothing read those dates and nothing would have objected when they passed. The dates now live in a DUE-CHECKS block INSIDE OPEN-ITEMS.md (inside, so no sidecar can drift from it) and a new gate reads them. Registered as #10 in repo_gates.py, --fast, so it runs in BOTH the pre-push hook and CI. exit 0 nothing due (prints pending count + nearest date; empty block too) exit 1 a row is due/overdue (due <= today, UTC -- due TODAY counts), or a row names an item with no R-row exit 2 block absent/duplicated/unparseable -- INCONCLUSIVE, never 0 It REFUSES rather than warns, and its docstring states the limitation: it is NOT a scheduler, it fires on the next push, not on the date. 37 tests. BOTH red-proofs run and reverted -- and the first one earned its keep by catching a hollow assertion of MINE rather than confirming the gate: flipping <= to < left a due-today row in neither bucket, min() raised on an empty list, and the TRACEBACK exited 1, so "rc == 1" passed while the boundary was wrong. An exit code cannot tell a verdict from a crash. The test now asserts the conviction banner and the absence of a traceback, and the gate returns 2 rather than crashing if that partition breaks again. PART 3 — the floor raise, and the premise was WRONG. Read back from the store (not the form): min_controller_version = 0.216.0 @ 12:36:58Z, zero per-customer overrides, no "managed floor HELD" line. But read 5 shows the raise was NOT a no-op: demo-felhom had been on 0.214.0 since 12 Aug and auto-updated 0.214.0 -> 0.216.0 at 12:37:07Z -- NINE SECONDS after the save, exactly the immediate action publish-train rule 2 documents. No error events followed; it restarted clean. R-343 is therefore filed OPEN, not CLOSED: the closing condition was all five reads clean and no directive served. It went well, but a record calling it inert when it moved a customer box is what misleads the next reader. The row also states why the floor was behind -- rule 2 policy, not drift, earned by the 2026-07-11 skew onto Peti's box -- and cites ResolveManagedFloor (store.go:2068) plus the two build-felhom-iso.sh facts (build-time at :267, fails open at :78-82) rather than asserting them. Two boxes are below the floor and neither reports: drill-r50 (blocked, powered off) and peti-felhom (host row deleted). peti-felhom was NOT contacted -- its row records that a report from a deleted host 401s and is not persisted, so the raise cannot reach it. PART 4 — R-342 filed READY, quoting stop2-snapshot.txt verbatim: Hetzner server snapshot 421440873 covers /dev/sda only; /mnt/pbs-datastore is a separate Volume that snapshots exclude, so a rollback restores software state and NOT the datastore. Fine for that upgrade; the safeguard for any future procedure that could touch the datastore does not exist and is Viktor's call. Also: CLAUDE.md's gate list named 6 of 10 registered gates -- completed rather than appending a 7th to a wrong list (124 -> 128 effective, ceiling 200). Capability map deliberately unchanged; no row cites a floor or golden version. repo_gates.py fully green, 10/10.	2026-08-18 15:16:51 +02:00
admin	7d81681d6e	golden 0.216.0: baked, published, vouched — gates green again ... Closes the two-release day-0 gap that has been convicting CI since 2026-08-14. Run against RUNBOOK-manual-build.md 4.0 + 4.1. GOLDEN_VERSION = 0.216.0 GOLDEN_SHA256 = ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b archive = 656,970,239 bytes, controller image 0.216.0 template = debian-13-standard_13.6-1_amd64.tar.zst (listed live, not reused) Baselines re-read on the machine and all four matched the sheet: controller v0.216.0, its MinAgent 0.129.0, agent v0.129.0, previous golden 0.214.0. The published agent artifact for the vouched agent_version was confirmed present in the package registry rather than inferred from a CHANGELOG, and the R-216 check passed on the machine: MinAgent is EQUAL to, not above, the newest published agent. Verified beyond the script's own claim: the artifact was downloaded back out of Gitea and hashed, and it matches GOLDEN_SHA256 exactly. A script printing a digest and the registry serving those bytes are two different claims. Pass markers (corrected post-R-233 list) all present, quoted with line numbers in pass-markers.txt; excluding/FATAL absent; there is no mp1. Token never reached a command line: copied file->file, read inside the VM by the runner. systemctl show grep = 0. Token-leak grep on the COMMITTED log run with its positive control FIRST -- seeded copy 1, real log 0 -- because a grep -c that matches nothing also returns 0. Teardown: guest destroyed and purged, token/runner/script/log shredded AFTER the log was copied out, qemu exit confirmed with ps -eo comm (not pgrep -f), disk reverted to virgin. Vouched by the operator; verified by reading the hub's own store: golden 0.216.0 / agent 0.129.0 / min_agent 0.129.0, and the hub's recorded sha256 matches the independently downloaded artifact. That check was necessary because golden_currency_gate.py says of itself that it checks the BAKE, not the vouch. repo_gates.py --fast now rc=0, all nine gates OK -- first fully green run since 2026-08-14. Capability map deliberately NOT changed: the day-0 row cites drill documents, and the map's only golden literal is a dated historical citation on the recovery-journey row which bumping would falsify. R-334 is closed in a follow-up commit quoting this push's CI run id, since closing it without one would leave the ambiguity a third time.	2026-08-18 13:04:37 +02:00
admin	3e50902a98	RUNBOOK ep0: PBS 4.2.2-1 -> 4.2.5-1, slope unchanged as predicted (R-341) ... Both STOPs cleared by the operator. No code changed; documentation only. STEP 3 (the run's primary deliverable): the full changelog range 4.2.2-1 -> 4.2.5-1 was read (128 lines, all three entries) and swept for connection-handling vocabulary. Exactly one keyword hit, a false positive ("S3 ... honor the node's proxy settings" = HTTP proxy config for S3, not the PBS proxy daemon). 4.2.5-1 is a manifest-hardening security release; 4.2.4-1 is S3 rate limits and a locking cache; 4.2.3-1 is UI/LDAP/tape. NOTHING addresses descriptor lifetime or connection reaping. Recommendation was: do not upgrade for this reason. STOP 1: operator ruled to upgrade anyway for rehearsal value. Recorded as a practice run, not a fix -- and the interpretation was fixed IN WRITING BEFORE any numbers existed (stop1-ruling.txt): unchanged = expected; changed = surprise. Neither outcome could then be rationalised into a success. STOP 2: Hetzner snapshot 421440873, Available. Documented that it covers /dev/sda ONLY -- /mnt/pbs-datastore is a separate Volume and is NOT in it, so it is a software rollback and not a backup of the backup data. UPGRADE: simulated first (0 to remove), then installed 09:51:00->09:51:06Z, exit 0. Verified: 4.2.5-1 installed, both daemons active, effective open files still 65536 (the drop-in survived the new package), Recv-Q 0, loopback 200, 200 from BOTH boxes over the tunnel with felhom-pbs active, and the hub gauge refreshed post-upgrade at 11:59:31. SLOPE: before +4 fd/1885 s = 183/day; after +5 fd/1919 s = 225/day. NOT distinguishable -- one descriptor apart, Poisson +/-2 on such counts. The higher after-figure is noise, not a regression and not an improvement. 30 minutes cannot settle it; R-341 files the +24 h and +7 d checks. CORRECTIONS to this morning's own report, both published rather than quietly fixed: - the "~85/day, ~2 years of runway" figures were WRONG. They came from a single 17-minute window with a delta of ONE descriptor. Real rate is 183-200/day over two independent windows; runway ~357 days, not 2 years. - the leak was attributed to CLOSE-WAIT. It is mostly ESTAB: CLOSE-WAIT held flat at 1 while ESTAB grew 45->49, and at the wedge it was 1011 ESTAB vs 543 CLOSE-WAIT. R-336's fix must target unreaped connections. - "proxmox-backup-api" reported inactive during verification; that unit does not exist. Bad query, not a fault, written down because it looked like one. R-336 stays open: even a fixed leak would not make ~85k requests/day to a weekly-write DR endpoint correct. golden-currency still convicts (inherited R-334, controller 0.216.0 vs golden 0.214.0, untouched by this run), so this push is --no-verify per .claude/rules/gates.md.	2026-08-18 12:26:53 +02:00
admin	ebfd0967c1	INCIDENT + registers: ep0's PBS proxy served nobody for 9.5h (R-336..R-338) ... Two whole_guest_backup_failed alerts at 04:30 and 04:32 CEST were one incident, and not on either customer box: ep0's proxmox-backup-proxy was active, holding its listening socket, and accepting nothing. Root cause: accept() returning EMFILE. The process held exactly 1024 fds -- its systemd-default soft RLIMIT_NOFILE -- of which 1016 were sockets and 547 connections sat in CLOSE-WAIT. The 1024-deep accept backlog had overflowed (Recv-Q 1025), so every client timed out. It was wedged from its own loopback too, which is what moved this from a network problem to a process problem. Fed by ~85k requests/day (a flat 3,538/hour) against an endpoint written to weekly, that leak reached the ceiling in 14 days of uptime. Fix: LimitNOFILE=65536 drop-ins for both PBS units, restart, verified from both boxes (200 in ~0.1s, felhom-pbs active), then re-drove the missed backups through the product path -- POST /backup?target=felhom-pbs on each agent's local API, not a hand-run vzdump. demo-felhom ct/9201/2026-08-18T03:57:43Z 4.10 GB 36.4s demo-hp ct/9201/2026-08-18T03:58:43Z 4.29 GB 41.5s Both host reports now carry felhom-pbs success=true, so the hub is green on the evidence rather than on a restart having been performed. No data lost, no backup skipped: the daily local tier was never affected and the PBS tier is weekly, so the window cost exactly one attempt. Evidence copied off ep0 BEFORE the restart, per standing rule 5. Filed: R-336 (the ~1 req/s poll rate is the real defect; the raised ceiling is mitigation, not a cure), R-337 (a status endpoint that trailed its own artifact by minutes then caught up -- WATCHING, downgraded from the defect I first wrote, because it self-corrected), R-338 (demo-hp is not on the R-50 island at all and nodes.md says it is; its local API is bound to the customer LAN). R-334 updated: still open, now one version wider (controller 0.216.0 vs golden 0.214.0). golden-currency is the only failing gate and is inherited -- it reads files this session did not touch -- so this push used --no-verify, stated per .claude/rules/gates.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016p1PTCzb8rF5G9Aa1qhpBN	2026-08-18 06:09:38 +02:00
admin	b03a105375	hub v0.105.0: the third name, a machine told to be quiet, and a guard for the hub's own words ... Hub only. No controller change, no agent change, no wire change — nothing to bake. demo-hp untouched: the operator is re-deploying it this evening. R-323 — the five-word phrase is „Tulajdonosi jelmondat". It was „Visszaállító jelszó": one word from the name retired last week, and false besides — it restores nothing, it proves the account owns the box being bound. Five sites, all in the hub; felhom-controller and felhom-agent carry the name nowhere, so no halt and no bake. Both suggested names were rejected with reasons: „Fiókjelszó" would collide with the dashboard login (a DIFFERENT real secret), and „Összekötési jelszó" would leave the two factors on this page separated only by kód-versus-jelszó — the exact shape being removed, since the other factor is the „Párosító kód". The chosen name differs on both axes, stem and noun. Naming only; the acceptance pin drives the real handler. R-324 — the hub's customer copy is under a guard for the first time. Retired names banned across all 95 hub files; retrieval stems registered in four declared customer surfaces. The selftest found a defect in its own instrument on the first run. One shared vocabulary in scripts/, drift-checked into the controller gate rather than copied (R-325 removes the scaffold). R-321 — a machine we told to be quiet is no longer reported as dead, and it was two doors, not one: because the state is RECORDED rather than deleted, the morning deadline check can skip it too. A deleted state returns "", which is not "down" — R-195's shape returning through a second door. The clock runs from the report the hub can see, so re-enabling starts it there and emits no recovery for an outage that never happened. Three red-proofs; the one that matters showed a genuinely dead machine sitting at "disabled" when the suppression was made unconditional. R-326 — "which claims are unproven" is answerable by a command now. The nine I have been repeating was the count of claims the 9 August pass DOWNGRADED, not the count of unproven ones. The real figures: 55 claims, 23 walked, 32 not — and only 6 of those 32 cite evidence. Its first run found a stale claim (R-327).	2026-08-13 15:50:32 +02:00
admin	4d6ec7c7bb	hub v0.104.0: the guest network gets a reader (R-319), and the hub half of the naming (R-295) ... Four paper debts and one fact given a reader. Hub-only — nothing to bake. A4 — the entry about "the tester's machine" named a risk correctly and labelled it in a way that invited deleting it. Established from the hub's own store: `peti-felhom` is a REAL machine (482 reports, 2026-02-27 → 2026-07-15, a named person's own box) and the 3.6 GB with no key and no backup is real. `david` → `tester-1` is a DIFFERENT record with no host, no escrow and no report, ever — deleted 07:55:49 and re-created 07:56:47 this morning. The prompt's premise conflated the two; the register now says which is which. A1 — R-312/R-313/R-303 recorded as DECIDED with their re-open triggers, and moved out of STATUS's "Waiting on you", which is now empty. A3 — day0-install §C.1 said pushing the installer publishes it. It has not since R-110. Corrected, with the two manifest pins named and an outside-verification command; the one copy that repeated it (a dated audit, true when written) carries a superseded note. A5 — standing rule 5: evidence comes off the machine at the end of the phase that produced it, before any revert. Earned twice in three days on the same box at the same point (R-320). Four homes, plus what to do when it is already gone. R-295 hub half — „Beállító kód" everywhere; „Visszaállító kód" retired. New `reenroll` mail kind so the mail names the page a REBUILT box actually shows („A szerver beállítása"), not the „Elfelejtett jelszó" page it has no login screen to reach. Naming only; the acceptance pin proves the secret is untouched. R-319 — the hub models `guest_net` after 23 days of receiving and discarding it. The signal is `heals_last_hour`, not `state`: a guest the watchdog keeps repairing reads healthy between repairs. `heal_succeeded` decoded too (R-260's lesson). Unknown is never drawn as healthy — three absences, three sentences. No alarm, deliberately. Three red-proofs, mutations asserted applied. Wire-gate checked tags 182 → 190. B1 — the operator's 2026-08-12 dispositions were NOT in the register; they are now. Third allowlist kind for the five ruled "no reader wanted"; `reporting_disabled` reclassified redundant. 8 read · 5 deliberately unread · 1 redundant · 6 still owed. Also filed: R-321 (a deliberately-silent box still alarms stale/down — the checker is age-only, and decoding the flag would not have fixed it), R-322 (the claim guard has never scanned the hub; a hand scan returns zero, so it is a scope gap, not a defect).	2026-08-13 10:50:12 +02:00
admin	fc737b0fc0	installer v1.28.0: the removal genuinely reverses the installation (R-316) ... v1.27.0's fix worked exactly once per machine. Measured on drill-r50 from virgin, on the PUBLISHED v1.27.0, before anything was changed: cycle 1 recorded 'no' and freed :53; cycle 2 recorded 'yes' and left dnsmasq running on 0.0.0.0:53; cycle 3 refused, exit 1. Every box already in the field is at cycle 2, and a reinstall onto a machine that has had Felhom is cycle 2 by definition. Why cycle 2 says yes: the preflight's ownership question is dpkg-query package presence and nothing else - not the absence of a record. Stopping the unit and leaving the package made our own package read as the household's one cycle later. Now the uninstall removes the package when the record says we installed it. Order unchanged and load-bearing: read the record, act, then delete the state file that holds it. TWO packages are recorded, because dnsmasq ships the unit and dnsmasq-base ships /usr/sbin/dnsmasq, and each is taken back only if we added it. The dependency check is a SIMULATION, not a guess: apt-get -s purge is asked what it would remove and the purge proceeds only if that set is a subset of ours; otherwise stop+disable, naming the package that blocked it. Never interactive, never fatal, and the success is re-queried rather than read off an exit code. Watched: three fixed cycles -> install 3 PASSES; a household resolver untouched; a dependent package not purged and named; no record -> untouched with the command named. Red-proofs with the mutation asserted applied: remove the purge -> cycle 3 refuses in those exact words; remove the ownership check -> a household resolver is purged; infer ownership -> the guess is taken. Also: R-317 (the agent stats a path dnsmasq-base owns to decide whether to install dnsmasq - pre-existing, now reachable), R-318 (no honest ownership marker exists for existing boxes; the preflight message is the mechanism), and the status page's decisions section rewritten to say what each decision costs and what doing nothing selects.	2026-08-13 08:16:04 +02:00
admin	d102ca5767	Vouch delivered fleet-wide; the MinAgent hold observed firing and releasing for the first time ... Operator vouched golden 0.214.0 / agent 0.129.0 / min agent 0.129.0 and raised the floor to 0.214.0. Both artifact shas in hub_settings match the bake and the release byte-for-byte. With the floor at 0.214.0 and demo-hp still on agent 0.128.0, the hub HELD the controller floor - 'agent 0.128.0 < MinAgent 0.129.0 (controller floor withheld)' - and released it 6 seconds after the agent was brought up. That is the R-216 guard doing exactly what it exists for, seen firing for the first time, and it is the argument for declaring MinAgent in the CHANGELOG header. Both demo boxes verified on the boxes: controller 0.214.0, agent 0.129.0, healthy.	2026-08-12 21:14:38 +02:00
admin	6362bb6cb6	hub v0.103.0 — a host can read the packages we kept for it (R-311) ... ListSupersededEscrow had zero production callers for nineteen days. It is the only reader of a retained identity_blob, so the retention shipped in v0.93.0 was material the product could not reach - proven on the fixture 2026-08-12, where a code that opens a retained package was answered as a code that opened nothing. New GET /api/v1/hosts/<id>/escrow/retained: self-scoped exactly as the current-row GET, same recovery-mode gate, same audit event written BEFORE the bytes leave, capped at 16. Rows with a NULL identity_blob are WITHHELD and returned as unopenable_count. They retain the PBS key, not the repository password, so they can never open what the caller is asking about; serving them would have the agent try packages that cannot succeed and would let the screen claim an earlier package is openable on exactly the boxes the original defect hurt. The count is returned because their existence is load-bearing and underivable. The trade, stated rather than waved through: the hub still cannot read any of it - sealed bytes in, sealed bytes out, no decrypt path, no recovery code ever held. What widens is volume, bounded by self-scope, the recovery-mode gate and the cap. The response is a NAMED TYPE, not a map, so the wire-contract gate can resolve it; the wire is declared as a fourth ROOT and the gate now checks 182 tags rather than 174. A positive control shows that check is name-presence, not decodability - filed as R-315 rather than reported as coverage. Six tests through the real endpoint; four red-proofs asserted applied.	2026-08-12 18:43:07 +02:00
admin	1d5f2b8bb6	DRILL: the retained key works, and the customer cannot reach it ... Three verdicts, kept separate because collapsing them is how this assumption survived a week. (a) The material IS retained. host_escrow_superseded id 11 is the first retained row in fleet history to carry identity_blob (572 B), byte-identical to the pre-supersession row (sha256 a10032341c8584ed...). (b) The retained material DOES open the old store. Unsealed with the old recovery code it yielded a password byte-identical to the pre-change one, and restored three planted files byte-identical from a store the box itself could no longer open - including a Hungarian accented filename verified as raw bytes. Negative control ran first and failed closed. (c) The customer has NO route, and is misinformed. ListSupersededEscrow has zero production callers; the recovery path selects FROM host_escrow. Asked with the code that had just worked by hand, the product answered "the recovery code did not open the sealed bundle". A valid code for retained history is reported as a bad code - the R-224 class again. R-304, rank 1. Both installer faults were watched happening first, so installer-v1.27.0 is now published (tag + both webpage.yaml refs). Pre-fix: the box came up on controller 0.98.3 against a vouched 0.213.0, below the floor and below the version carrying the recovery screen; and our own uninstall left dnsmasq on 0.0.0.0:53 so our own next install refused. R-297 and R-300 CLOSED. Also filed R-305 (the dnsmasq fix fires once per machine - the leftover returns on the second reinstall, proven), R-306 (--preflight-only writes state it says it does not), R-307 (a live abandon countdown on demo-felhom, firing 2026-08-24 - operator decision), R-308 (stored controller password stale), R-309 (the day-0 runbook's publication claim has been false since R-110), R-310 (two edges). Ceiling R-303 -> R-310. Capability map moved: the retention claim is now marked operator-only. Phase A logs did not survive the intermediate revert; recorded.	2026-08-12 17:41:56 +02:00
admin	fbe1155fbb	R-302 docs: register rows, the two rules earned twice, STATUS ... Closes R-296 (verified: shipped in v0.212.0) and R-301 (premise confirmed, fixed in v0.213.0). Files R-302 with WHY the obvious condition was rejected, and R-303 for the missing markOrphaned guard - the co-render is now harmless, not impossible. Bake evidence for golden 0.213.0.	2026-08-12 15:38:11 +02:00
admin	890a474ff2	STATUS back to one screen; close R-280 and R-294 ... 211 lines -> one screen. Moves closed items out, corrects the tester paragraph, states the floor situation as the operator's one-field call, and stops asking him to decide something that shipped.	2026-08-12 14:04:35 +02:00
admin	f76cbf0ec7	Part 0: correct the PETI record - the mitigation it named does not exist ... The row said a drive failure there means offsite-only recovery. Re-read from the hub's own store: no host row (deleted 2026-07-15 08:56:22, escrow_acked=0), no escrow of any kind, and offsite backup never ran once (escrow_state pending, snapshot_count 0 - the fork-4 guard working, not a fault). The local app-data repo was empty too and the whole-guest vzdump shares the failing device. If that drive fails today, everything on it is lost. Records the fact and leaves the parked/not-parked ruling open - that is the operator's call and does not need restating to be true. STATUS.md no longer reads the absence of a hub record as reassuring.	2026-08-12 13:43:07 +02:00
admin	aaae9461c2	STATUS: 0.211.0 delivered to fresh installs; floor raise is the operator's switch	2026-08-10 14:29:54 +02:00
admin	c04f933d0b	The census answers no, three receipts found, and the prune was on file all along ... CENSUS (read-only, hub store, tester's machine not contacted): no machine that is not ours can be in the state that cost demo-felhom its history. The hub holds escrow for three hosts; both demo boxes lost their pre-fix key in the same four hours on 2026-08-04; peti-felhom and david have no host row and no escrow at all. A control ran FIRST and had to pass -- the query returned "present (572 bytes)" for a host known to have material and "absent (NULL)" for one known not to. Corrected my own instrument on the way: a date-only comparison mislabelled both losses as after the fix, so the in-force moment is now pinned from the hub's first post-fix escrow row (11:11:37Z), which independently agrees with the register. PART 1 ESTABLISHED. The prune is recorded inside R-267 -- the row about the Configuration page being slow -- because pruning artifacts is what made that page fast. Arithmetic checks (23+7=30, plus three versions that only surfaced after the first thirty moved them onto page one = 33) and the PAGINATED listing shows both generics at exactly ten. R-291's blocking condition is released: the operator was being asked to establish something already written down. And my counter-argument yesterday was wrong in exactly the way R-267 warns about -- "containers hold 19" came from an unpaginated query; paginated they hold 270 and 169. RECEIPTS: three restored (drives.enrol, backup.tier1, fail.lost-recovery-code), each citing the document that walked it; the map already read PROVEN-LIVE for all three, so this follows the map rather than raising a status in the view. NINE HONEST GREYS. fault.selfheal's best hit argues against it -- an incident recording self-heal's absence through a 1h15m outage. THE DECAY RULE FIRED FOR THE FIRST TIME. backup.restore-proof has a receipt from 28 July and is superseded anyway: demo-hp's restore-test failed 5 August and the box has since been rebuilt. A claim about a continuing behaviour cannot rest on an old observation. The capability map still reads PROVEN-LIVE and is now the thing out of step -- recorded, not silently rewritten. PART 4 specified, not implemented. The orphan card promises restorability the box rendering it cannot evaluate: the discriminator is on the hub and no wire field carries it. A conditional promise the system cannot evaluate is the same defect as an unconditional false one, so the copy stops promising, says what happens, and names a route. Ships with the next controller change so one bake covers both.	2026-08-10 11:15:37 +02:00
admin	67eced8fbf	demo-felhom is protected again, and the authorised recovery could never have worked ... Checked before acting, and the check is the finding. The box's local key and the hub's sealed escrow key hash to the SAME value (c60c8bc737a6b7c6...), and that key answers "wrong password or no key found" against its own repository. Running the recovery would have returned a key the box already held and which was already proven not to open the store. The store was written under 48741892f0ef4d59... -- host_escrow_superseded id=4, superseded 2026-08-04 07:20:08, identity_blob NULL. The restic password lives only in the identity bundle (escrow/identity.go:39, read by recover.go:91), so it is unrecoverable by construction; the surviving K-escrow payload is 64 bytes, a wrapped key, far too small to carry it. Same shape the register already records for demo-hp, four hours the wrong side of the retention fix. Took the operator's stated fallback instead: the orphan reset through the customer's own card. Old store moved aside, never deleted, to /home/felhom-repo.orphaned-20260810 (1.2 GB); fresh repository under the current key; offbox_repo_reset audited hub-side. Then PROVEN rather than assumed -- last_status ok, 10s, and the snapshot's CONTENTS listed: opengist compose files, manifest.json and volume-dumps/opengist_opengist_data.tar. Not an empty backup calling itself successful. R-202 gains hard evidence: the orphan card promises those set-aside backups may be restorable later with their recovery code. For these 1.2 GB that is false and unfixable, and it is said to the customers most likely to read it.	2026-08-10 10:10:43 +02:00
admin	ed447f940e	Both demo machines are home, unmuted and healthy; one true alarm stands ... Powered up 2026-08-10 ~09:26 CEST. Both unblocked on the hub, both OK on the approved pair (agent 0.128.0, controller 0.210.0). No false alarm on power-up -- the mute did its job and was removed as the banner said it must be. The hub briefly read "Guests 0/2": the agent's first post-boot report genuinely said stopped, because it caught the guests mid-start. Both corrected to running on the next cycle (07:40:52Z, 07:43:39Z). Transient, not a defect -- confirmed by waiting for the observable rather than assuming it. demo-hp's off-site repository still opens with the box's own credential, 18 snapshots intact including yesterday's rehearsal files, so tonight's 04:15 run has what it needs. demo-felhom's is ORPHANED with no successful run ever; the offsite_stale mail it sent this morning is a TRUE alarm and the remedy is the customer-present recovery ceremony, which needs the operator.	2026-08-10 09:44:42 +02:00
admin	985f0ba63c	Record the guards, the narrowing, and the two things I could not do ... R-273's owed guards are both built and closed. R-291 records what CI stopped covering and why, so it can be widened deliberately rather than discovered. R-292 is new and was found by a test failing for the wrong reason: artifact_sha_invalid conflates "version missing", "registry unreachable" and "bad sha" into one message. v0.102.0 works around it by ORDERING -- the probes run first, so an unreachable registry is reported as unreachable -- but the message itself is untouched. CONTEXT gains the rule this session is about: a check and the policy it enforces must read the same number from the same place, or they drift and the drift looks like a defect in something else. Two corollaries, both of which cost something: a bounded check must print what it stopped covering on every run, and an unreadable policy is INCONCLUSIVE rather than unbounded. Stated in the report rather than glossed: Part 4 (finding receipts for the twelve downgraded claims) was NOT done and is a shortfall, not a decision -- splitting it would have produced exactly the half-checked green the exercise exists to prevent. Part 5 was droppable and dropped. The tag-push green is not re-proved tonight and is not claimed; the evidence offered is runs 190 and 216.	2026-08-09 19:28:45 +02:00
admin	d7d2f84257	Both demo machines powered down for the move home, and MUTED on the hub ... Operator is moving back from the vacation home. Guests shut down cleanly first (checked: no running PVE task, no guest lock, no vzdump/restic in flight — the "reboot mid-vzdump leaves a stale lock and the guest won't autostart" trap), then the hosts. Confirmed off at the FABRIC rather than inferred from silence: the tailnet is healthy, dooplex is up, and both peers report "offline, last seen 1m ago". An absent ping is not evidence; an offline peer on a working tailnet is. Both customers blocked on the hub so the 30-minute host_stale and 60-minute host_down transitions do not fire four false alarms during the drive. IsCustomerBlocked gates every monitor plus the notification intake; it does NOT gate config pull or report intake, so the boxes return normally on power-up. STATUS.md carries a banner at the very top because the mute has a tail: while blocked, a box that FAILS to come back is equally silent. Unblocking on arrival is now the first thing anyone reads.	2026-08-09 14:11:29 +02:00
admin	1d6f1c522d	Rehearsal 2026-08-09 COMPLETE: data BYTE-IDENTICAL, journey needs a shell twice ... The walk finished. All four planted files came back byte-identical out of snapshot 41c830db, including two Hungarian accented filenames verified as RAW NAME BYTES (NFC preserved) — the discriminator the Gate 0 positive control was built for, having been watched failing on an NFC->NFD rename that renders the same. Unlock 21s, restore 13.2s. It finished only because a terminal was available twice: - R-273 CLOSED. v0.128.0 was published as a package and never git-tagged, so every install died at 5/8. Tag pushed on operator instruction after an INDEPENDENT download proved the package sha equalled the vouched value; --resume then reached Day-0 SUCCESS in 3m49s on controller 0.210.0. The two guards that would stop the class recurring are still owed. - R-280 NEW, rank 1. A reinstalled box cannot re-attach its own data drive by any dashboard route: /api/disks/candidates returns empty because both lists are built from the UNCLAIMED-disk scan, and the drive is claimed precisely because it is also the backup target. Correct for "initialise", over-broad for "attach", which is non-destructive by definition. The restore page meanwhile says "Ez ket kattintas" and points at that empty list. Cleared by POSTing /mnt/sys_drive — an internal path no household could produce. Also new: R-281 the hub said NOTHING through the entire reinstall and the sealed-backup tripwire did not fire on a real unseal (positive control: 2 events all day fleet-wide); R-282 one code with three names and a mail pointing at a page the box does not show; R-283 hub reads "Claimed 18d ago" while the box serves its setup page; R-284 "almost full" over a 93%-free store. R-274 NARROWED by measurement rather than left as written: the resume path fetched the vouched golden correctly, because --resume skips the preflight that does local discovery. What survives is real — discovery is sort|tail -1 with no manifest comparison — but a FRESH install taking a stale golden is still not observed, and the row says so. Two of my own claims were refuted by test and are recorded as refuted, not quietly dropped: the leftover sudoers file is inert (sudo skips dotted names), and demo-hp's off-site tier was healthy all along.	2026-08-09 12:35:26 +02:00
admin	b1afbb8a4d	Rehearsal 2026-08-09: the walk stops at P3 — R-273 blocks every install fleet-wide ... P1 uninstall, P2 preflight, P3 install. The install FAILED at step 5/8 in 44s, and the two rank-1 findings are both on the setting-up path a tester's visit is made of. Eleven register rows minted (R-269..R-279); ceiling moves 268 -> 279. R-273 (RANK 1) — the hub vouches agent 0.128.0; that version was published as a Gitea PACKAGE but never git-tagged. Since R-183 the installer correctly pins its config fetches to raw/tag/v<vouched>, so every fresh install and every reinstall now 404s as root, mid-install. Measured: main 200, v0.127.0 200, v0.128.0 404. This is R-184 arriving; release-agent.sh:23 already documents the exact hazard. Existing boxes are fine (self-update takes the binary from the registry). NOT fixed here — publishing a release tag is outward-facing and the runbook says stop and report. One command unblocks it; it is in STATUS.md. R-272 (RANK 1) — Felhom's own uninstall leaves the condition that makes Felhom's own reinstall refuse. It installs dnsmasq at day-0, then on teardown removes the snippet and RESTARTS the daemon unconstrained (process start time lands inside the uninstall window), which grabs 0.0.0.0:53; the next preflight then refuses, and the message reads as though the owner's LAN DNS is at fault. R-274 — a local golden is adopted with no version and no sha check; the manifest vouch is consulted only on the fetch path. demo-hp's local copy is controller 0.192.0 against a vouched 0.210.0, and below the 0.200.0 where the recovery screen shipped. Not yet observed end-to-end (R-273 killed step 5 first). Also: R-275 orphaned credential backups + uid reuse, R-276 the wg tunnel outlives the uninstall, R-269/270/271 from the token rotation, R-277 three hub surfaces misreport a healthy off-site tier, R-278 demo-felhom six days unprotected, R-279 no operator-triggerable off-site run. Two hypotheses of mine were tested and REFUTED rather than shipped as findings: the leftover sudoers file is inert (sudo skips dotted filenames), and demo-hp's off-site tier was healthy all along - I had misread the hub and said so. STATUS.md records the three rulings §8.3 asked for, with the floor CORRECTED to its live value 0.200.0 and the count corrected to twenty.	2026-08-09 10:59:44 +02:00