07-backup-architecture.md 6.3 gains a dated [DESIGN] paragraph on replay ->
rollback -> hold, including why no engine flag closes it: --single-transaction
makes Postgres atomic, MariaDB DDL is not transactional, so the rollback is the
fix and the flag is a belt.
Drill record for the live walk, including the TWO defects the walk found in the
fix itself (a rollback into a re-created container; an operator route that
cleared the file while the running controller kept refusing) and the ONE
red-proof that PASSED, which is reported rather than omitted.
R-379..R-382 compressed into CLOSED-ITEMS.md. OPEN-ITEMS 330683 -> 325236 bytes.
STATUS.md restates the outcome and names the next operator step.
A drill, not an implementation. No code, no version bump, no CHANGELOG entry.
Ten of the forty driveless apps carry a database; I re-measured that count and
got 10. For those ten the restore is a five-leg operation that never ran at all
until this week, because R-356 refused before any of it started.
Walked end to end on demo-hp for both engines - docmost (Postgres 16) and
bookstack (MariaDB 12.3) - each deployed for the drill, planted through the
app's own interface, destroyed for real, restored through the endpoint the UI
posts to.
Q1 does it complete: YES. All five legs ran and succeeded, 32s / 25s. Accented
names byte-identical both directions.
Q2 which leg won: the SQL DUMP. Three-way discriminator returned the altered
dump's value. This confirms R-164's F17 ordering on the OFF-SITE path; R-164
only ever cited the local one. Scratch-only mutation; store proved unmutated.
Q3 does a failure tell the truth: partly, and two defects.
Filed R-379 (HIGH, the undo copy is valid, named, and unappliable by any product
action - proven by applying it by hand on both engines), R-380 (HIGH, a failed
MariaDB replay leaves a partial database behind an app reporting healthy, where
Postgres crash-loops visibly), R-381 (MEDIUM, the failure message pastes engine
stderr including customer table rows into the Hungarian surface), R-382 (LOW,
the summary log omits the volume count it already has).
H1, H2 and H4 did NOT fire and that is recorded. H3 fired in a shape nobody
predicted: not a quiet success, but a loud error over a silent inconsistency.
R-361 reproduced independently on a second app. restic check: no errors, 29
snapshots. A flaw in the drill's own planting - a double-escaped accented title -
was caught by reading stored bytes as hex, recorded, and re-measured in Phase 1b.
Register 325236 -> 330683 bytes. Nothing dropped. Teardown: two apps retained
with reason, no pvesm before-snapshot taken (said plainly), no hub-side record
created.
Baked in the drill VM per RUNBOOK-manual-build.md 4.0/4.1, carrying controller
v0.219.0 (R-356).
GOLDEN_VERSION 0.219.0
GOLDEN_SHA256 67b46f78f8ed9c7b1876265ab1bde9ec6798897898b1836acece9f3864a2aeb6
656832571 bytes
All five pass markers matched, both negative controls at 0. Verified by ROUND
TRIP - the published object downloaded again and its sha recomputed - not by the
number the script printed.
Both token-leak greps were proved able to convict before their zeros were
believed: planted copy grepped 1, shredded, then the 0 accepted.
Teardown complete: guest 9100 purged, four secret/script files shredded after the
log was copied out, qemu exited, disk reverted to virgin. The revert first
refused while qemu held the image, which is the runbook's own no-holder proof.
NOT vouched - that is a three-field operator save (golden_version 0.219.0,
agent_version 0.130.0, min_agent 0.129.0).
07-backup-architecture.md: three places said no offsite action unpacks the
named-volume tars. R-107 closed in controller v0.218.0; all three corrected with
a dated [FACT], the old sentence kept in the past tense. R-102 is NOT closed and
the correction says so explicitly.
New [DESIGN] paragraph in 6.3: the restore destination is resolved by the same
rule as the capture destination, and the wrong-disk refusal applies to apps that
have a drive to get wrong. Carries the 13/40 measurement.
STATUS.md was internally contradictory - nothing waiting, and one decision
waiting, for something the same page recorded as shipped. 218 -> 102 lines; the
deciding section now says what happens if nothing is done.
R-356 compressed into CLOSED-ITEMS.md; OPEN-ITEMS 327109 -> 325236 bytes.
Drill record and 16 evidence files for the live walk on demo-hp.
Records and process only. No machine contacted.
ONE REGISTER (operator ruling). 17 roadmap rows moved into OPEN-ITEMS.md keeping their
identifiers, evidence and original filing dates - the oldest R-10, filed 2026-07-15, 38 days.
15 ideas stay in ROADMAP.md, which is their home; the gate exempts them by their own state
word. 59 already-closed rows stay as history. Sorting rule recorded in the roadmap header:
does the item assert something about the shipped product a reader could check and find false?
scripts/one_register_gate.py, wired as the 11th gate. Control run: baseline passes, a planted
open roadmap-only row is convicted by name, removing it passes with the file byte-identical,
and a planted `idea` row is correctly exempt. Its four residual holes are in its docstring.
The gate earned its keep immediately: it caught R-103, a READY finding my hand-sort mis-read as
done because my regex matched the whole row where the body contains "shipped" - the gate matches
the state cell. It also caught R-203 and R-163, recorded closed in the register and still open in
the roadmap; the roadmap copies are marked SUPERSEDED with the register's verdict.
HOUSEKEEPING. OPEN-ITEMS 672,376 -> 327,109 bytes (-51%); ROADMAP 239,306 -> 78,110 (-67%).
Closed work compressed to 17% into CLOSED-ITEMS.md and ROADMAP-HISTORY.md; every entry names the
commit whose git show returns the full original text. Rule-sentences are kept verbatim under
"Reasoning kept" rather than judged entry by entry - 25 carry one.
CONTEXT.md deliberately NOT compressed and the disagreement is argued in the report: 86% of it is
standing rulings still in force, this prompt's own 3.4 says the log is never edited, and it has no
per-ruling delimiter. Filed as R-377 - the problem is navigational, not volumetric.
The hot/bulk placement decision was NEVER recorded as a decision anywhere - established, not
assumed. Now marked [DESIGN] with a pointer honest about having no original date, given a
decision-log entry that records what was rejected, and the [DESIGN]/[FACT] legend carried from 1
of 8 architecture documents to 8 of 8. Existing statements deliberately left unmarked (R-376).
PROMPT-TEMPLATE gains N.7: compress what you closed, rehome live reasoning before it goes, state
the register's size before and after.
Ceiling R-375 -> R-378.
Documentation and survey only. No code, no machine contacted.
THE CORRECTION. The 40 catalogue templates without a configurable path are not missing a
choice: 01-topology-and-trust.md:150-152 classes each volume hot (DB/config/cache -> fast
storage, ENFORCED) or bulk (media/files), and the 40 are all-hot apps. The deploy page has
been saying so to the customer all along (deploy.html:624-625). SPEC-app-data-placement and
R-352 are corrected in place with the framing MARKED, not deleted; every measurement stands.
R-356 was re-checked and survives, strengthened - an absent HDD_PATH is the normal state, so
reading it as "not installed" misreads a correct configuration.
The disk claim, precisely: since R-165 there is ONE guest data volume with two binds, not two
volumes (build-golden.sh:29-40, 99). A physical-disk failure losing data and first-tier copy
together is REAL and is what the other tiers exist for. A full data volume stopping the OS is
NOT real and was the overstated one.
THE SWEEP. 113 survey-class documents examined, 14 statements of "not filed", 2 already filed.
Its positive control convicted the sweep itself twice before it convicted the corpus - markdown
bold broke the strongest pattern, and the reporter re-searched a truncated line - both false
zeros of the exact class being hunted, and together worth 2 of the 14.
THE HEADLINE. The gap the 2026-08-21 drill rediscovered WAS filed - as R-107, ROADMAP.md:122,
M/READY, 2026-07-28 - and is absent from OPEN-ITEMS.md, which calls itself the single source of
truth. OPEN-ITEMS and that rule both landed 2026-07-27; R-107 went to ROADMAP alone the day
after. 72 ids live only in ROADMAP, 29 not done, some of them findings. Filed as R-369 (HIGH).
Five more still-open gaps filed with their ages: R-371 (17d), R-372 (38d, the oldest), R-373
(20d), R-374 (14d), R-375 (4d). R-368 corrects Part 4: the storage default IS applied at deploy
time via deploy.html:612 - the earlier "the deploy route never reads it" came from grepping Go
and never the templates. R-370 records the process failure and is closed by the template change.
PROMPT-TEMPLATE gains the two rules it lacked: name the architecture document for the area and
say what it says (with a file->area map and the test "is this something we chose?"), and an
enumerated gap becomes a register row in the same session - a ROADMAP row alone does not count.
Ceiling R-367 -> R-375.
Eight claims re-checked against the drill and the v0.218.0 fixes; three moved, all downward.
backup.offsite walked -> partial. "18 snapshots, daily, unbroken" was true on
2026-08-09 and false by 2026-08-21: the next snapshot after that date
was put there by hand, twelve days later. The rebuild lost the target
and the per-app switches came back off, so a run reported "backup OK:
0 app(s) backed up".
fail.wiped-reinstalled.data walked -> partial. A real reinstall orphans BOTH off-premises tiers:
restic silently for 12 days (R-193), and the PBS archives from before
the reinstall cannot be opened by the rebuilt box at all (R-366).
backup.fill-warning walked -> partial. The warning fires correctly, but the watcher runs
once a day, so a filesystem that fills at 03:31 goes unannounced for
~24 h. Watched silent while a volume sat at 99%.
Five re-checked and held: backup.tier1 and recover.byte-identical carry the R-355/R-354 story and
their fixes; backup.restore-proof stays grey for a sharper reason (orphaned archives, not an
untested tier); backup.sikeres gains two fresh instances; fail.customer-self-restore records that
R-356 now blocks 40 of 53 apps regardless of who is driving.
render_stands.py: the header's commit shas were hardcoded, so the page cited the August 9th commits
while the YAML said otherwise - the stale-build-product failure the renderer exists to prevent. They
are parsed now. The count beside them said "15 status(es) moved in that pass" when 15 was every
recorded move ever; it now separates the two numbers.
check_stands passes, and was itself proven able to convict first: a claim marked `missing` flipped to
`walked` in a scratch copy fired rule 5 by name (use.dlna), and the real file still passes.
Both of the drill's HIGH findings are fixed in controller v0.218.0 and confirmed on demo-hp with
a negative control first — the same planted, hash-recorded fixture run through the same steps on
both builds.
R-355: paperless-ngx's PostgreSQL was dumped into a directory for a stack that does not exist, so
it never entered the recovery unit, the off-site copy or the restore; and because the same wrong
name reached writeSafetyDump, a destructive restore took no undo copy and the fail-closed refusal
was never reached. Fixed by reading the compose project label. Sweep proven able to convict
before its count was trusted: one affected app of 53.
R-354: the off-site restore had no named-volume leg. Now it replays them from the scratch unit,
before the database and inside the stopped window, and VolumesReplayed reaches the sentence.
The half-false comment beside the skip is corrected and the half that still holds is named.
Golden 0.218.0 baked and published, sha 8e427869d13eafb71562b77d1535eef6c7f32b4db24f659b988ec6d7db8f478b,
verified by round trip on the downloaded bytes. NOT vouched and the floor NOT raised — both are
the operator's decision, and raising the floor is what puts this on demo-felhom, which is still
on 0.217.0 and still has both defects.
R-367 filed: the dumps already written under the wrong name are stranded. Nothing deletes them
(an existing guard), they are adoptable by hand, and doing it automatically would be a migration.
Ceiling R-366 -> R-367.
The box's own 02:30 / 03:30 / 04:15 cycle ran unattended and agrees with the manual one
on every measure. The one that matters: R-355 is not an artefact of manual triggering —
the scheduled run again wrote paperless-ngx's PostgreSQL dump into a directory for a
stack that does not exist, and again left the app's own unit recording db_dumps: null.
Part 4.2's terminal deletion has now been observed. At 05:10 the sweep removed exactly
the recorded set-aside store and left the live repository untouched; at 05:13 the hub
dropped the sealed package that protected it and said so (event 3025). Both halves went
together, three minutes apart, and the controller cleared its own state. demo-felhom's
two preserved fixtures were verified untouched throughout.
R-366 (HIGH) filed, found incidentally: the 21 August reinstall orphaned demo-hp's PBS
whole-guest archives as well as its restic repo, so a rebuilt box loses BOTH off-premises
tiers at once. The restore-test caught it and named the key mismatch precisely; it is
merely called "a failed restore test" rather than "your older backups are unreadable".
demo-hp is left HEALTHY, not broken. Ceiling R-353 -> R-366.
The 2026-08-04 row claimed "a customer's file survives a machine rebuild and comes back
— the whole off-site story, end to end". Tonight's drill shows that holds for the
declared-userdata leg of a drive-declaring app and for nothing else: the off-site restore
has no named-volume leg (R-354), and refuses outright for the 40 apps that declare no data
drive (R-356). Since that class keeps ALL its data in named volumes, the end-to-end story
is unproven there and disproven for the volume leg generally. The escrow/key half of the
row is untouched and still stands.
STATUS.md also corrects the fleet pair it still named (0.214.0/0.129.0 -> 0.217.0/0.130.0)
and records that demo-hp's off-site had been silent since 9 August.
Diagnostic only — no code changed, no version bumped, nothing deployed.
The verdict is a mixture. The unit and the off-site snapshot HOLD the data, proven
by identity in both storage classes including two Hungarian accented filenames. The
loss is in the last leg: ReconstituteFromOffsite skips every isUnit placement and the
volume tars live inside the unit, so the off-site full restore has no named-volume
leg at all — while the local restore-from-unit does, and returned the same tar
byte-identical minutes later.
Twelve rows opened, ceiling R-353 -> R-365. Three HIGH:
R-354 off-site restore never replays volume dumps
R-355 paperless-ngx's Postgres is dumped under a non-existent stack, so its unit
has no DB dump, no safety dump is taken, and the customer is told it has none
R-356 the off-site restore refuses for all 40 no-drive apps saying the running app
"is not installed", with a remedy those apps make impossible
R-353's instruction (2) is satisfied and annotated: the 40-class DOES reach the
off-site tier. Its instruction (1) stands and is now larger. R-329 confirmed still
live and now the only bad-severity emit fleet-wide.
Evidence: documentation/audits/DRILL-backup-truth-2026-08-21/evidence/
GOLDEN_VERSION = 0.217.0
GOLDEN_SHA256 = 0276c5f638d140a861daba4ef25129896e259937eec0ae5cba7af42391315ad0
archive volid = local:backup/vzdump-lxc-9100-2026_08_21-21_40_05.tar.zst
MinAgent = 0.129.0 (unchanged from 0.216.0)
Acceptance markers counted in this run's own bake.log, not paraphrased:
docker OK (overlay2 1
including mount point 2 (rootfs and mp0 — there is no mp1)
upload OK (HTTP 201) 1
excluding 0
FATAL 0
Published package fetched back over HTTPS: HTTP 200.
Token handling: copied file->file, read by a runner script inside the VM, never on a command
line. systemctl show of the live unit contained it 0 times. The committed bake.log greps 0 for
the literal token AND the grep was first PROVEN to work on that same file by appending the token
to a throwaway copy (grep = 1) then shredding it — a 0 from an untested grep is not evidence.
Evidence copied off the VM BEFORE teardown. Then destroy 9100 --purge, shred token+runner+script
+log inside the VM (0 left), poweroff, waited for qemu using `ps -eo comm` (never `pgrep -f`,
which self-matches), and reverted the drill VM to `virgin`.
This unblocks the golden-currency gate, which correctly refused the previous push of the register
rows: "controller v0.217.0 is released and NO golden carries it". No --no-verify was used.
NOT DONE: the vouch. It is operator-gated and is a THREE-field change; vouching golden_version
alone would ship this controller onto an agent older than it declares it needs.
R-351 - the restore never read back where the backup said the data lived, and a second press
started a second restore. Both shipped in controller v0.217.0.
R-352 - four measured untruths about where an app's data goes:
1. 40 of 53 catalogue templates declare no data path (13 declare env_var: HDD_PATH)
2. GetDefaultStoragePath() has three non-test callers and NONE of them places data; its
field comment "new apps use this by default" has never been true
3. the first-tier backup follows the data onto the same disk (GetAppDrivePath ->
systemDataPath) - the posture Tier 2 refuses outright at tier2.go:329
4. "1 alkalmazas hasznalja" counts only Env["HDD_PATH"] == path, so it can never include
the 40-class; it means "1 of the apps that CAN use a drive does"
Visibility shipped tonight; PLACEMENT IS OPEN and is the operator's ruling. An earlier
recommendation to refuse deployment until a drive is registered was WITHDRAWN - it assumed
the customer had failed to choose, and they had no choice to make.
R-353 - a restore reported success having returned configuration and no data. OpenGist's unit
holds manifest.json + compose/ and nothing else (volume_dumps: None, db_dumps: None); the
off-site snapshot was 182.3 KB; the outcome said only "completed in 8.666896042s". Ranked as
the NEXT SESSION'S FIRST ITEM. Compounding and recorded as UNKNOWN rather than fine: whether
the 40-class reaches the off-site tier at all has not been observed - runVolumeDumps covers
them on paper, but no nightly dump run had happened on a one-hour-old box.
New: documentation/backlog/SPEC-app-data-placement-2026-08-21.md - specification only, nothing
implemented, listing the five points a placement ruling must settle. Records that the OpenGist
instance meant to be left as evidence was removed by someone between 16:57 and 17:02 UTC (not
by this session); its unit and manifest survive, and privatebin is now a live specimen.
Ceiling moved R-350 -> R-353.
0.130.0 is tagged, published and vouched, so a fresh install gets the
fixed agent. Both boxes reinstalled from the downloaded artifact.
Closing evidence is the positive observable: ep0 at fd 17 / ESTAB 0 /
CLOSE-WAIT 0, its t0 baseline, returning there between poll cycles with
both agents still demonstrably polling. ep0 was read-only throughout and
its proxy PID never changed.
Released via scripts/release-agent.sh: tag v0.130.0 at 7569f34, sha256
a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3,
verified by independent download and reproducible byte for byte with
-trimpath -buildvcs=false.
Vouched agent 0.129.0 -> 0.130.0 in the Day-0 manifest. Only the agent
fields changed: min_agent stays 0.129.0 because it states what the GOLDEN
CONTROLLER requires, and raising it would have HELD the floor for every
box below 0.130.0. Global floor untouched at 0.216.0 -- and on hub
v0.106.0 it is a separate form with its own action, so publish-train
rule 2's hazard no longer exists in the shape its incident describes.
No --no-verify: the CHANGELOG heading was flipped only after the tag and
package existed, so release-complete passes on the real artifact.
R-349: the fleet was running a DIFFERENT binary under the same version
name -- the proof deploy was a hand build, the release is -trimpath.
Self-update could never have corrected it, because every version check
compares the string. Both boxes reinstalled from the downloaded package.
The proper fix exists in miniature as wrapper_sha256 and was never
extended to the agent's own binary.
R-350: I printed the hub password into the session transcript via
curl -w '%{redirect_url}' -- the hub answers 303 and curl re-attaches the
credential. Not in git, not in any committed file, not in the evidence
directory. Rotation is the operator's call.
ep0 closes at fd 17, ESTAB 0, CLOSE-WAIT 0 -- its t0 baseline -- and was
read-only for this entire arc.
P1, outcome (i) in one second: replacing the agent on demo-hp released
exactly its 199 established connections (ep0 fd 415 -> 216). CLOSE-WAIT
stayed 0, so outcome (ii) does not exist and gets no row -- ep0 reaps on
peer FIN correctly, and the 543 CLOSE-WAIT at the 08-18 wedge has another
explanation.
P2, 1.03 h (operator closed the >=4 h window early, so no daily rate is
extrapolated): control +4, fixed +0, with each box making exactly 4
/snapshots and 4 /version calls. Same cadence, same work: 4 cycles -> 4
leaks vs 4 cycles -> 0. The fixed box's cycles are in ep0's log, so the
zero is the fix and not a stopped agent.
P3: the second box took ep0 from 220 to 17 fd in under two seconds. 17 is
precisely the t0 baseline of 2026-08-18 09:51:22Z.
Corrects a claim this session made earlier the same day: the accumulated
descriptors did NOT need an ep0 proxy restart. They were held on both
sides. ep0 was read-only throughout; its PID never changed.
R-344 updated and left OPEN (unpublished is not delivered). R-336
re-scoped -- its old next-step would have fixed nothing while looking like
a failed fix, and it is now a scaling row (~25 req/s at fifty customers).
R-347 filed for the delivery gap (Viktor decides). R-348 filed: an agent
restart blanks the reported backup list for ~18 h and the Store comment
calls it unaffected -- blinds no alarm, checked not assumed.
R-341's first dated check, taken at +46.2 h: fd 17 -> 405 over 166,251 s
= 201.6/day. Pre-registered range was 370-450; observed 388. UNCHANGED,
as predicted. CLOSE-WAIT is 0 -- absent entirely, not merely flat.
Q1: exactly two peers, 194 each, no third party.
Q2: outcome (a). ep0 388 = 194 + 194 on the boxes, twice, and the four
new sockets carry the same source ports on both sides. 0 closed in 31 min.
The finding: all 388 are held by felhom-agent. pvestatd and
proxmox-backup-client made 162,404 requests and leaked zero. Mechanism is
a per-cycle http.Transport with a zero-value IdleConnTimeout that nothing
ever closes (internal/pbs/client.go:56, main.go:1486). R-336's premise
does not survive this -- cutting the poll rate would have fixed nothing.
Q3 NOT measured: Phase C held at STOP 1, prediction pre-registered first.
Part 0 captures the due-checks gate's first conviction on a real overdue
date (rc=1, names R-341, sole failure among 10 gates). Row cleared at
Part 4, after the result was recorded in R-341, not to make a push work.
New: R-344 (the transport leak), R-345 (hub/Makefile pushes :latest),
R-346 (ActiveEnterTimestamp reads 5h56m early -- NRestarts is still 0).
THE GAP, measured not supposed. On 2026-08-18 ep0's PBS proxy was wedged for
9 h 37 m and the hub emitted NOTHING on the operator channel. Both box
checkers hold their last snapshot and return silently on a failed fetch --
correct for a FILL signal, since a missing reading must never be read as 0%,
but it makes a dead off-site endpoint and a healthy one indistinguishable.
The only mails that morning came from the boxes' own backup failures, and
only because the WEEKLY offsite run happened to land inside the window. Two
days earlier nothing would have fired at all.
REACHABILITY is now a second, independent signal on both checkers:
consecutive failed fetch windows, reported past a default 3 windows
(~30-45 min) as pbsdr_box_unreachable / offsite_box_unreachable (warning) on
the customer-less pbsdr-box / pool-box scopes, each with a paired *_recovered
all-clear. Tunable via alerting.box_unreachable_windows (0/invalid -> 3).
THE FILL LOGIC IS UNTOUCHED. No threshold, throttle, band or escalate-once
behaviour changed; a degraded read still drives no transition.
Three decisions a later reader would otherwise "fix" back, so each is
argued in-code:
- the unreachable event REPEATS rather than escalating once. The band shape
would give exactly ONE mail at ~minute 30 of a nine-hour outage, and one
mail is missable. It leans on the dispatcher's 1 h operator cooldown to
become an hourly "still blind" heartbeat.
- ErrUsageUnsupported is NOT blindness: an old ep0 answers "no such op",
which means we reached it. Counting it would alert for days on a healthy
pre-update endpoint.
- born-blind is reported: the counter is not gated on having a snapshot, so
a hub restarted INTO an outage still speaks. last_ok is OMITTED rather
than zero-valued -- a fabricated timestamp reads as "it was fine until
then".
Both recoveries are severity "info" and severityNotifies drops "info", so
they are registered in recoveredPairedDownTypes or the operator hears that
the tier broke and never that it healed. A cross-package test drives
ProcessEvent and asserts an actual operator MAIL, not a map entry -- a green
checker test proves nothing about the seam (agent v0.91.0 shipped fully green
with SetAuthSink never called).
Tests: box_reachability_test.go (Scenarios A-F) + dispatcher_box_reachability
_test.go (wiring). Three red-proofs run and reverted, each seen failing with a
message naming the right cause: threshold 3->1, the sentinel counter guard,
the pairing entry.
Register: R-339 filed and marked SHIPPED (PROVEN-LIVE still owed -- no real or
constructed outage has exercised the emit path, and one cannot be manufactured
against Tier-2 ep0). R-340 filed: the reachability read rides ep0's LOCAL API
daemon, which the incident explicitly cleared, so this check would have shown
GREEN for all 9 h 37 m -- the honest boundary, recorded rather than glossed.
R-336's next-step corrected: pvestatd's interval is NOT tunable (Proxmox staff
have said so); the only lever is disabling the storage entry, which collides
with the agent's consume-the-one-time-secret path. Doc-only, no agent code
touched.
PART 1+2 — dated checks stop being wishes. R-341 booked two measurements as
prose in a register row; nothing read those dates and nothing would have
objected when they passed. The dates now live in a DUE-CHECKS block INSIDE
OPEN-ITEMS.md (inside, so no sidecar can drift from it) and a new gate reads
them. Registered as #10 in repo_gates.py, --fast, so it runs in BOTH the
pre-push hook and CI.
exit 0 nothing due (prints pending count + nearest date; empty block too)
exit 1 a row is due/overdue (due <= today, UTC -- due TODAY counts), or a
row names an item with no R-row
exit 2 block absent/duplicated/unparseable -- INCONCLUSIVE, never 0
It REFUSES rather than warns, and its docstring states the limitation: it is
NOT a scheduler, it fires on the next push, not on the date.
37 tests. BOTH red-proofs run and reverted -- and the first one earned its
keep by catching a hollow assertion of MINE rather than confirming the gate:
flipping <= to < left a due-today row in neither bucket, min() raised on an
empty list, and the TRACEBACK exited 1, so "rc == 1" passed while the
boundary was wrong. An exit code cannot tell a verdict from a crash. The test
now asserts the conviction banner and the absence of a traceback, and the gate
returns 2 rather than crashing if that partition breaks again.
PART 3 — the floor raise, and the premise was WRONG. Read back from the store
(not the form): min_controller_version = 0.216.0 @ 12:36:58Z, zero
per-customer overrides, no "managed floor HELD" line. But read 5 shows the
raise was NOT a no-op: demo-felhom had been on 0.214.0 since 12 Aug and
auto-updated 0.214.0 -> 0.216.0 at 12:37:07Z -- NINE SECONDS after the save,
exactly the immediate action publish-train rule 2 documents. No error events
followed; it restarted clean.
R-343 is therefore filed OPEN, not CLOSED: the closing condition was all five
reads clean and no directive served. It went well, but a record calling it
inert when it moved a customer box is what misleads the next reader. The row
also states why the floor was behind -- rule 2 policy, not drift, earned by
the 2026-07-11 skew onto Peti's box -- and cites ResolveManagedFloor
(store.go:2068) plus the two build-felhom-iso.sh facts (build-time at :267,
fails open at :78-82) rather than asserting them.
Two boxes are below the floor and neither reports: drill-r50 (blocked,
powered off) and peti-felhom (host row deleted). peti-felhom was NOT
contacted -- its row records that a report from a deleted host 401s and is
not persisted, so the raise cannot reach it.
PART 4 — R-342 filed READY, quoting stop2-snapshot.txt verbatim: Hetzner
server snapshot 421440873 covers /dev/sda only; /mnt/pbs-datastore is a
separate Volume that snapshots exclude, so a rollback restores software state
and NOT the datastore. Fine for that upgrade; the safeguard for any future
procedure that could touch the datastore does not exist and is Viktor's call.
Also: CLAUDE.md's gate list named 6 of 10 registered gates -- completed
rather than appending a 7th to a wrong list (124 -> 128 effective, ceiling
200). Capability map deliberately unchanged; no row cites a floor or golden
version. repo_gates.py fully green, 10/10.
Golden 0.216.0 baked, published and vouched; the row is closed with CI run
353 (head_sha 7d81681d6, conclusion success) named in the closing text.
Quoting the run id is deliberate rather than decorative: this row was already
re-confirmed once and widened once (0.215.0 -> two releases behind at
0.216.0), and closing it on a local green a third time would have left the
same ambiguity the row keeps being reopened for. Runs 351 and 352, earlier
the same afternoon, were red on exactly this gate -- that contrast is the
evidence, not the assertion.
Also records what was verified rather than assumed: the vouch was read back
out of the hub's own store (golden 0.216.0 / agent 0.129.0 / min_agent
0.129.0) and the hub's recorded sha256 matches the artifact downloaded
independently from Gitea. golden_currency_gate.py states of itself that it
checks the BAKE and not the vouch, so the gate alone could not have closed
this.
Fixes a column-count slip in the same row: the closing text initially
replaced two cells with one, leaving R-334 at 6 pipes against its
neighbours' 7. The deps cell is restored.
Closes the two-release day-0 gap that has been convicting CI since
2026-08-14. Run against RUNBOOK-manual-build.md 4.0 + 4.1.
GOLDEN_VERSION = 0.216.0
GOLDEN_SHA256 = ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b
archive = 656,970,239 bytes, controller image 0.216.0
template = debian-13-standard_13.6-1_amd64.tar.zst (listed live, not reused)
Baselines re-read on the machine and all four matched the sheet: controller
v0.216.0, its MinAgent 0.129.0, agent v0.129.0, previous golden 0.214.0. The
published agent artifact for the vouched agent_version was confirmed present
in the package registry rather than inferred from a CHANGELOG, and the R-216
check passed on the machine: MinAgent is EQUAL to, not above, the newest
published agent.
Verified beyond the script's own claim: the artifact was downloaded back out
of Gitea and hashed, and it matches GOLDEN_SHA256 exactly. A script printing
a digest and the registry serving those bytes are two different claims.
Pass markers (corrected post-R-233 list) all present, quoted with line
numbers in pass-markers.txt; excluding/FATAL absent; there is no mp1.
Token never reached a command line: copied file->file, read inside the VM by
the runner. systemctl show grep = 0. Token-leak grep on the COMMITTED log run
with its positive control FIRST -- seeded copy 1, real log 0 -- because a
grep -c that matches nothing also returns 0.
Teardown: guest destroyed and purged, token/runner/script/log shredded AFTER
the log was copied out, qemu exit confirmed with ps -eo comm (not pgrep -f),
disk reverted to virgin.
Vouched by the operator; verified by reading the hub's own store: golden
0.216.0 / agent 0.129.0 / min_agent 0.129.0, and the hub's recorded sha256
matches the independently downloaded artifact. That check was necessary
because golden_currency_gate.py says of itself that it checks the BAKE, not
the vouch.
repo_gates.py --fast now rc=0, all nine gates OK -- first fully green run
since 2026-08-14.
Capability map deliberately NOT changed: the day-0 row cites drill documents,
and the map's only golden literal is a dated historical citation on the
recovery-journey row which bumping would falsify.
R-334 is closed in a follow-up commit quoting this push's CI run id, since
closing it without one would leave the ambiguity a third time.
Both STOPs cleared by the operator. No code changed; documentation only.
STEP 3 (the run's primary deliverable): the full changelog range 4.2.2-1 ->
4.2.5-1 was read (128 lines, all three entries) and swept for
connection-handling vocabulary. Exactly one keyword hit, a false positive
("S3 ... honor the node's proxy settings" = HTTP proxy config for S3, not the
PBS proxy daemon). 4.2.5-1 is a manifest-hardening security release; 4.2.4-1
is S3 rate limits and a locking cache; 4.2.3-1 is UI/LDAP/tape. NOTHING
addresses descriptor lifetime or connection reaping. Recommendation was: do
not upgrade for this reason.
STOP 1: operator ruled to upgrade anyway for rehearsal value. Recorded as a
practice run, not a fix -- and the interpretation was fixed IN WRITING BEFORE
any numbers existed (stop1-ruling.txt): unchanged = expected; changed =
surprise. Neither outcome could then be rationalised into a success.
STOP 2: Hetzner snapshot 421440873, Available. Documented that it covers
/dev/sda ONLY -- /mnt/pbs-datastore is a separate Volume and is NOT in it, so
it is a software rollback and not a backup of the backup data.
UPGRADE: simulated first (0 to remove), then installed 09:51:00->09:51:06Z,
exit 0. Verified: 4.2.5-1 installed, both daemons active, effective open
files still 65536 (the drop-in survived the new package), Recv-Q 0, loopback
200, 200 from BOTH boxes over the tunnel with felhom-pbs active, and the hub
gauge refreshed post-upgrade at 11:59:31.
SLOPE: before +4 fd/1885 s = 183/day; after +5 fd/1919 s = 225/day. NOT
distinguishable -- one descriptor apart, Poisson +/-2 on such counts. The
higher after-figure is noise, not a regression and not an improvement. 30
minutes cannot settle it; R-341 files the +24 h and +7 d checks.
CORRECTIONS to this morning's own report, both published rather than quietly
fixed:
- the "~85/day, ~2 years of runway" figures were WRONG. They came from a
single 17-minute window with a delta of ONE descriptor. Real rate is
183-200/day over two independent windows; runway ~357 days, not 2 years.
- the leak was attributed to CLOSE-WAIT. It is mostly ESTAB: CLOSE-WAIT held
flat at 1 while ESTAB grew 45->49, and at the wedge it was 1011 ESTAB vs
543 CLOSE-WAIT. R-336's fix must target unreaped connections.
- "proxmox-backup-api" reported inactive during verification; that unit does
not exist. Bad query, not a fault, written down because it looked like one.
R-336 stays open: even a fixed leak would not make ~85k requests/day to a
weekly-write DR endpoint correct.
golden-currency still convicts (inherited R-334, controller 0.216.0 vs golden
0.214.0, untouched by this run), so this push is --no-verify per
.claude/rules/gates.md.
Post-restart baseline on ep0: 19 fds at 16m43s (from 18), 1 CLOSE-WAIT,
Recv-Q 0. One descriptor per ~17 min is ~85/day, which agrees with the
~73/day implied independently by the failure itself (1016 sockets over 14
days of uptime).
Two estimates of the same slope agreeing turns "the ceiling raise is
mitigation, not a cure" from a plausible claim into a measured one, and
puts the next ceiling at ~2 years instead of a fortnight. Recorded because
standing rule 3 asks for a positive observable: this is it, and it fired.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016p1PTCzb8rF5G9Aa1qhpBN
Two whole_guest_backup_failed alerts at 04:30 and 04:32 CEST were one
incident, and not on either customer box: ep0's proxmox-backup-proxy was
active, holding its listening socket, and accepting nothing.
Root cause: accept() returning EMFILE. The process held exactly 1024 fds
-- its systemd-default soft RLIMIT_NOFILE -- of which 1016 were sockets
and 547 connections sat in CLOSE-WAIT. The 1024-deep accept backlog had
overflowed (Recv-Q 1025), so every client timed out. It was wedged from
its own loopback too, which is what moved this from a network problem to
a process problem.
Fed by ~85k requests/day (a flat 3,538/hour) against an endpoint written
to weekly, that leak reached the ceiling in 14 days of uptime.
Fix: LimitNOFILE=65536 drop-ins for both PBS units, restart, verified from
both boxes (200 in ~0.1s, felhom-pbs active), then re-drove the missed
backups through the product path -- POST /backup?target=felhom-pbs on each
agent's local API, not a hand-run vzdump.
demo-felhom ct/9201/2026-08-18T03:57:43Z 4.10 GB 36.4s
demo-hp ct/9201/2026-08-18T03:58:43Z 4.29 GB 41.5s
Both host reports now carry felhom-pbs success=true, so the hub is green on
the evidence rather than on a restart having been performed. No data lost,
no backup skipped: the daily local tier was never affected and the PBS tier
is weekly, so the window cost exactly one attempt.
Evidence copied off ep0 BEFORE the restart, per standing rule 5.
Filed: R-336 (the ~1 req/s poll rate is the real defect; the raised ceiling
is mitigation, not a cure), R-337 (a status endpoint that trailed its own
artifact by minutes then caught up -- WATCHING, downgraded from the defect
I first wrote, because it self-corrected), R-338 (demo-hp is not on the
R-50 island at all and nodes.md says it is; its local API is bound to the
customer LAN).
R-334 updated: still open, now one version wider (controller 0.216.0 vs
golden 0.214.0). golden-currency is the only failing gate and is inherited
-- it reads files this session did not touch -- so this push used
--no-verify, stated per .claude/rules/gates.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016p1PTCzb8rF5G9Aa1qhpBN
The 0.215.0->0.216.0 redeploy replaced the container and the state file came
back with the previous version's changed_at, so the new container loaded the
pre-restart record rather than re-baselining. The verdict path itself, and the
already-alerted-disk restart case, remain unproven.
- capability-map: the disk-failure scenario no longer says a failing disk has
never been seen. Healthy path + delivery + the severity wire stay PROVEN-LIVE;
the new Hiba-from-counters path is IMPLEMENTED and explicitly NOT proven-live
(R-332), because it has only ever run against the fixture's values.
- ROADMAP R-73: phase 1 shipped; the genuinely hub-side half splits into R-330
(phase 2, a declared wire change under G-1) and R-331 (phase 3, growth-rate
detection and retiring the static 64). Its premise 'no demo hardware exposes
real SMART' is retired — a real failing drive is now committed as a fixture.
- register: R-328 (the severity drop, CLOSED and proven live side by side),
R-329 (app_start_failed has the same defect, needs a decision first),
R-330/R-331 (phases 2 and 3), R-332 (the Fail path has never fired on real
hardware, WATCHING), R-333 (NVMe temperature bands measured 2 degrees from
tripping on a healthy drive; and the agent's smartctl has no -n standby).
Commit the raw evidence from ST3000VX010 S/N Z6A07P2G (/dev/sdg on DooPlex),
which went 8 -> 352 unreadable sectors 11-13 Aug while smart_status.passed
stayed true throughout.
- fixtures/smart-ST3000VX010-failing-2026-08-14.json: raw smartctl -a -j, verbatim
- fixtures/smartd-history-sdg-2026-08-14.txt: 406 smartd journal lines, 11-14 Aug
- DIAG-smart-passed-trap-2026-08-14.md: the mechanism (attrs 187/197/198 all carry
thresh 0, so a normalized value that floors at 1 can never fail the overall
verdict on unreadable sectors), the non-monotonic timeline, the three controller
defects with locators, and the counterfactual: zero emails would have been sent.
Three rules carried forward. A name must separate on the STEM, not the noun — naming
this secret after the act it is used in would have recreated the trap, because the
other factor on the same page is the „Párosító kód". A guard is worth what its positive
control is worth: this one's selftest convicted its own step-3 case and found a defect
in the guard itself. And a suppression must rest on the machine's own declaration, then
be checked for the SECOND door — recording the disabled state rather than deleting it
is what let the deadline check skip it too.
Yesterday's report is preserved to audits/ because it carries the only record of the
self-heal verdict (Part C was dropped, so that reasoning is in no register row) — the
rule written last night, applied to itself the first time it mattered.
Hub only. No controller change, no agent change, no wire change — nothing to bake.
demo-hp untouched: the operator is re-deploying it this evening.
R-323 — the five-word phrase is „Tulajdonosi jelmondat". It was „Visszaállító
jelszó": one word from the name retired last week, and false besides — it restores
nothing, it proves the account owns the box being bound. Five sites, all in the hub;
felhom-controller and felhom-agent carry the name nowhere, so no halt and no bake.
Both suggested names were rejected with reasons: „Fiókjelszó" would collide with the
dashboard login (a DIFFERENT real secret), and „Összekötési jelszó" would leave the
two factors on this page separated only by kód-versus-jelszó — the exact shape being
removed, since the other factor is the „Párosító kód". The chosen name differs on
both axes, stem and noun. Naming only; the acceptance pin drives the real handler.
R-324 — the hub's customer copy is under a guard for the first time. Retired names
banned across all 95 hub files; retrieval stems registered in four declared customer
surfaces. The selftest found a defect in its own instrument on the first run. One
shared vocabulary in scripts/, drift-checked into the controller gate rather than
copied (R-325 removes the scaffold).
R-321 — a machine we told to be quiet is no longer reported as dead, and it was two
doors, not one: because the state is RECORDED rather than deleted, the morning
deadline check can skip it too. A deleted state returns "", which is not "down" —
R-195's shape returning through a second door. The clock runs from the report the hub
can see, so re-enabling starts it there and emits no recovery for an outage that never
happened. Three red-proofs; the one that matters showed a genuinely dead machine
sitting at "disabled" when the suppression was made unconditional.
R-326 — "which claims are unproven" is answerable by a command now. The nine I have
been repeating was the count of claims the 9 August pass DOWNGRADED, not the count of
unproven ones. The real figures: 55 claims, 23 walked, 32 not — and only 6 of those 32
cite evidence. Its first run found a stale claim (R-327).
Four paper debts and one fact given a reader. Hub-only — nothing to bake.
A4 — the entry about "the tester's machine" named a risk correctly and labelled it
in a way that invited deleting it. Established from the hub's own store: `peti-felhom`
is a REAL machine (482 reports, 2026-02-27 → 2026-07-15, a named person's own box) and
the 3.6 GB with no key and no backup is real. `david` → `tester-1` is a DIFFERENT record
with no host, no escrow and no report, ever — deleted 07:55:49 and re-created 07:56:47
this morning. The prompt's premise conflated the two; the register now says which is which.
A1 — R-312/R-313/R-303 recorded as DECIDED with their re-open triggers, and moved out
of STATUS's "Waiting on you", which is now empty.
A3 — day0-install §C.1 said pushing the installer publishes it. It has not since
R-110. Corrected, with the two manifest pins named and an outside-verification command;
the one copy that repeated it (a dated audit, true when written) carries a superseded note.
A5 — standing rule 5: evidence comes off the machine at the end of the phase that
produced it, before any revert. Earned twice in three days on the same box at the same
point (R-320). Four homes, plus what to do when it is already gone.
R-295 hub half — „Beállító kód" everywhere; „Visszaállító kód" retired. New `reenroll`
mail kind so the mail names the page a REBUILT box actually shows („A szerver
beállítása"), not the „Elfelejtett jelszó" page it has no login screen to reach.
Naming only; the acceptance pin proves the secret is untouched.
R-319 — the hub models `guest_net` after 23 days of receiving and discarding it. The
signal is `heals_last_hour`, not `state`: a guest the watchdog keeps repairing reads
healthy between repairs. `heal_succeeded` decoded too (R-260's lesson). Unknown is never
drawn as healthy — three absences, three sentences. No alarm, deliberately.
Three red-proofs, mutations asserted applied. Wire-gate checked tags 182 → 190.
B1 — the operator's 2026-08-12 dispositions were NOT in the register; they are now.
Third allowlist kind for the five ruled "no reader wanted"; `reporting_disabled`
reclassified redundant. 8 read · 5 deliberately unread · 1 redundant · 6 still owed.
Also filed: R-321 (a deliberately-silent box still alarms stale/down — the checker is
age-only, and decoding the flag would not have fixed it), R-322 (the claim guard has
never scanned the hub; a hand scan returns zero, so it is a scope gap, not a defect).
v1.27.0's fix worked exactly once per machine. Measured on drill-r50 from virgin,
on the PUBLISHED v1.27.0, before anything was changed: cycle 1 recorded 'no' and
freed :53; cycle 2 recorded 'yes' and left dnsmasq running on 0.0.0.0:53; cycle 3
refused, exit 1. Every box already in the field is at cycle 2, and a reinstall
onto a machine that has had Felhom is cycle 2 by definition.
Why cycle 2 says yes: the preflight's ownership question is dpkg-query package
presence and nothing else - not the absence of a record. Stopping the unit and
leaving the package made our own package read as the household's one cycle later.
Now the uninstall removes the package when the record says we installed it. Order
unchanged and load-bearing: read the record, act, then delete the state file that
holds it. TWO packages are recorded, because dnsmasq ships the unit and
dnsmasq-base ships /usr/sbin/dnsmasq, and each is taken back only if we added it.
The dependency check is a SIMULATION, not a guess: apt-get -s purge is asked what
it would remove and the purge proceeds only if that set is a subset of ours;
otherwise stop+disable, naming the package that blocked it. Never interactive,
never fatal, and the success is re-queried rather than read off an exit code.
Watched: three fixed cycles -> install 3 PASSES; a household resolver untouched; a
dependent package not purged and named; no record -> untouched with the command
named. Red-proofs with the mutation asserted applied: remove the purge -> cycle 3
refuses in those exact words; remove the ownership check -> a household resolver is
purged; infer ownership -> the guess is taken.
Also: R-317 (the agent stats a path dnsmasq-base owns to decide whether to install
dnsmasq - pre-existing, now reachable), R-318 (no honest ownership marker exists
for existing boxes; the preflight message is the mechanism), and the status page's
decisions section rewritten to say what each decision costs and what doing nothing
selects.
Operator vouched golden 0.214.0 / agent 0.129.0 / min agent 0.129.0 and raised the
floor to 0.214.0. Both artifact shas in hub_settings match the bake and the release
byte-for-byte.
With the floor at 0.214.0 and demo-hp still on agent 0.128.0, the hub HELD the
controller floor - 'agent 0.128.0 < MinAgent 0.129.0 (controller floor withheld)' -
and released it 6 seconds after the agent was brought up. That is the R-216 guard
doing exactly what it exists for, seen firing for the first time, and it is the
argument for declaring MinAgent in the CHANGELOG header.
Both demo boxes verified on the boxes: controller 0.214.0, agent 0.129.0, healthy.
The live test read as a FAILURE for twenty minutes because I stripped only double
quotes from a credentials value wrapped in SINGLE ones, sending a literal ' as part
of the recovery code. Correctly unquoted: the old code returns 422 with
opens_retained=true and the supersession date; a wrong code still returns 400.
The same bug produced the R-308 finding in the previous report. The dashboard
password is fine - HTTP 302 with a session cookie on the first try. Third time this
project has produced a wrong 'the credential is stale' verdict from that one trap.
ListSupersededEscrow had zero production callers for nineteen days. It is the only
reader of a retained identity_blob, so the retention shipped in v0.93.0 was
material the product could not reach - proven on the fixture 2026-08-12, where a
code that opens a retained package was answered as a code that opened nothing.
New GET /api/v1/hosts/<id>/escrow/retained: self-scoped exactly as the current-row
GET, same recovery-mode gate, same audit event written BEFORE the bytes leave,
capped at 16. Rows with a NULL identity_blob are WITHHELD and returned as
unopenable_count - they retain the PBS key, not the repository password, so they
can never open what the caller is asking about, and serving them would let the
screen claim an earlier package is openable on exactly the boxes the original
defect hurt. The count is returned because their existence is load-bearing and
underivable by the caller.
The trade, stated rather than waved through: the hub still cannot read any of it -
sealed bytes in, sealed bytes out, no decrypt path, no recovery code ever held.
What widens is volume, bounded by self-scope, the recovery-mode gate and the cap.
The response is a NAMED TYPE, not a map, so the wire-contract gate can resolve it;
the wire is declared as a fourth ROOT and the gate now checks 182 tags rather than
174. A positive control shows that check is name-presence, not decodability -
filed as R-315 rather than reported as coverage.
Also: golden 0.214.0 baked, published and round-trip verified; the countdown on
demo-felhom cancelled on the operator's ruling (R-307); the spike that halted
Part 3 recorded as R-312; the set-aside store found unrecoverable as R-313.
Six hub tests through the real endpoint; four red-proofs asserted applied.
ListSupersededEscrow had zero production callers for nineteen days. It is the only
reader of a retained identity_blob, so the retention shipped in v0.93.0 was
material the product could not reach - proven on the fixture 2026-08-12, where a
code that opens a retained package was answered as a code that opened nothing.
New GET /api/v1/hosts/<id>/escrow/retained: self-scoped exactly as the current-row
GET, same recovery-mode gate, same audit event written BEFORE the bytes leave,
capped at 16.
Rows with a NULL identity_blob are WITHHELD and returned as unopenable_count.
They retain the PBS key, not the repository password, so they can never open what
the caller is asking about; serving them would have the agent try packages that
cannot succeed and would let the screen claim an earlier package is openable on
exactly the boxes the original defect hurt. The count is returned because their
existence is load-bearing and underivable.
The trade, stated rather than waved through: the hub still cannot read any of it -
sealed bytes in, sealed bytes out, no decrypt path, no recovery code ever held.
What widens is volume, bounded by self-scope, the recovery-mode gate and the cap.
The response is a NAMED TYPE, not a map, so the wire-contract gate can resolve it;
the wire is declared as a fourth ROOT and the gate now checks 182 tags rather than
174. A positive control shows that check is name-presence, not decodability -
filed as R-315 rather than reported as coverage.
Six tests through the real endpoint; four red-proofs asserted applied.
Three verdicts, kept separate because collapsing them is how this assumption
survived a week.
(a) The material IS retained. host_escrow_superseded id 11 is the first retained
row in fleet history to carry identity_blob (572 B), byte-identical to the
pre-supersession row (sha256 a10032341c8584ed...).
(b) The retained material DOES open the old store. Unsealed with the old recovery
code it yielded a password byte-identical to the pre-change one, and restored
three planted files byte-identical from a store the box itself could no longer
open - including a Hungarian accented filename verified as raw bytes. Negative
control ran first and failed closed.
(c) The customer has NO route, and is misinformed. ListSupersededEscrow has zero
production callers; the recovery path selects FROM host_escrow. Asked with the
code that had just worked by hand, the product answered "the recovery code did
not open the sealed bundle". A valid code for retained history is reported as a
bad code - the R-224 class again. R-304, rank 1.
Both installer faults were watched happening first, so installer-v1.27.0 is now
published (tag + both webpage.yaml refs). Pre-fix: the box came up on controller
0.98.3 against a vouched 0.213.0, below the floor and below the version carrying
the recovery screen; and our own uninstall left dnsmasq on 0.0.0.0:53 so our own
next install refused. R-297 and R-300 CLOSED.
Also filed R-305 (the dnsmasq fix fires once per machine - the leftover returns
on the second reinstall, proven), R-306 (--preflight-only writes state it says it
does not), R-307 (a live abandon countdown on demo-felhom, firing 2026-08-24 -
operator decision), R-308 (stored controller password stale), R-309 (the day-0
runbook's publication claim has been false since R-110), R-310 (two edges).
Ceiling R-303 -> R-310. Capability map moved: the retention claim is now marked
operator-only. Phase A logs did not survive the intermediate revert; recorded.
Closes R-296 (verified: shipped in v0.212.0) and R-301 (premise confirmed, fixed
in v0.213.0). Files R-302 with WHY the obvious condition was rejected, and R-303
for the missing markOrphaned guard - the co-render is now harmless, not
impossible. Bake evidence for golden 0.213.0.
211 lines -> one screen. Moves closed items out, corrects the tester paragraph,
states the floor situation as the operator's one-field call, and stops asking
him to decide something that shipped.
Removing the snippet and restarting left dnsmasq enabled and unconstrained on
0.0.0.0:53, so the next byo install's preflight refused and the customer went
debugging a home network that was never at fault.
Ownership is recorded at preflight (the only moment it is a fact - the package
is installed by the agent, not this script) and honoured at removal. Boxes
already in the field carry no record and fail safe to restart-only, with the
reason and the command logged; the preflight message covers them instead.
Not observed live - no installer-v1.27.0 tag is cut. Files R-299..R-301.
The spec listed backups_remote.html:98 as 'Accurate; keep'. It ended with the
same unevaluable promise as surface 2, in the plural - and because the spec's
own guard was written against the singular form, it could not catch it either.
Both corrected; implemented in controller v0.212.0 (R-299).
The row said a drive failure there means offsite-only recovery. Re-read from the
hub's own store: no host row (deleted 2026-07-15 08:56:22, escrow_acked=0), no
escrow of any kind, and offsite backup never ran once (escrow_state pending,
snapshot_count 0 - the fork-4 guard working, not a fault). The local app-data
repo was empty too and the whole-guest vzdump shares the failing device. If that
drive fails today, everything on it is lost.
Records the fact and leaves the parked/not-parked ruling open - that is the
operator's call and does not need restating to be true.
STATUS.md no longer reads the absence of a hub record as reassuring.
GOLDEN_SHA256=8593516889eb93fe1691410d7306be8cb87ee835b8d2378740eb34022272f849
Round-trip verified on the served bytes. Not vouched - that is the operator's.
Step 7 short-circuited on any local golden archive with no version compare, no
digest and no warning, so the manifest sha256 was consulted only on the fetch
path. Local discovery is newest-by-filename: correct by recency, never by
verification. A box could reinstall from a stale archive and come back below the
version where the offsite recovery screen exists.
Digest first, then the baked controller tag. An auto-discovered mismatch
re-fetches the vouched golden; an operator-named mismatch refuses. An unreadable
manifest refuses rather than passing.
Not published: installer-v1.26.0 is deliberately not cut until a fresh install
has been observed taking a stale local golden on drill-r50.
Also files R-295..R-298.
felhom.eu run 293 (67eced8, the demo-felhom recovery commit) FAILED, and
/actions/jobs/293/logs returns HTTP 500 while runs 292 and 294 both return 200
with 12290 bytes. The failure is real and its reason is not establishable --
exactly what this row was filed for, now seen a second time.
What can be said: the same tree passes the FULL gate set locally, all eight OK,
and run 294 on the next commit passed in CI, so nothing persisted. What cannot:
why 293 failed.
The process failure is mine. The checklist says confirm your own push's CI run by
ID; I pushed 67eced8 without doing so, and it surfaced only because the next
session happened to list recent runs. A pull check that is skipped is not a check.
admin