felhom.eu/REPORT.md

REPORT — the line under the backup arc, and one alarm that was telling the operator something untrue

hub v0.111.0 → v0.111.1 · controller v0.232.0 UNTOUCHED · 2026-09-01

No controller release. No agent release. No golden owed, no floor change. The only behaviour change in this session is one sentence in one alarm. Everything else is register and documentation.

---

1. The alarm's new text, and proof the old promise is gone

Quoted from the RUNNING binary — kubectl cp out of pod hub-857678f9b4-v95s6, byte-grepped:

Customer %s: off-site backup count fell from %d to %d snapshot(s) in one report — more than
retention can explain. The daily Storage Box snapshots are read-only and still hold the older
copy. The route back out of them is not yet established, so treat this as neither confirmed
data loss nor confirmed recovery. Get in touch before restoring anything, and check whether a
deletion ran on the box.

Every shape of the withdrawn promise is absent from the deployed bytes:

fragment	in the deployed binary
recoverable file-by-file	absent
recoverable file by file	absent
so this is recoverable	absent
it is NOT confirmed data loss	absent
(negative control) zzz-no-such-string-r434	absent — so the search can report absence
(positive control) offsite_snapshots_dropped	present, 2 occurrences

All seven fragments of the new sentence were checked individually and are present.

THE FIX IS A DELETION, NOT A REPLACEMENT — and that is what unblocked it. R-434's own row said the fix was "blocked on R-433", i.e. on first establishing what IS true. That verdict was mine and it was wrong. A sentence that asserts neither loss nor recovery is true under every possible answer to the provider questions, so it never needs a second rewrite. A replacement would have been blocked; a withdrawal is not. The reasoning is recorded in R-434's closing cell and in the function's doc comment, because the distinction is the transferable part.

It does not swing the other way either. TestR434_AlarmStillDoesNotClaimDataLoss pins that: "your backups are gone" is still usually false — the snapshots exist and hold the older copy; what is unproven is our route to them. Clause (a) of the 2026-09-01 measurement — the box cannot write into the snapshot area — stands and is re-confirmed.

Tests, and the red-proof

Three tests in hub/internal/monitor/offsite_r434_test.go, all driving the production path (saveOffsiteReport → oc.Check() → the notify callback), so they assert the sentence an operator receives rather than the function that formats it. ASCII-only fragments, with a positive control (a phrase in every version of the alarm) and a negative control (a phrase that cannot exist).

RED-PROOF (run before the fix was restored): the v0.111.0 sentence was put back and all three FAILED — on recoverable file-by-file, on so this is recoverable, on all three required fragments, on the un-negated confirmed data loss, and on the stored row — each with the offending sentence printed in the failure message. Fix restored, git diff clean, full hub suite green.

One existing test was edited, and it caught this fix correctly. TestR431_FiresOnAMassDeletion asserted the fragment "NOT confirmed data loss" and went red on the new wording. The fragment was REMOVED, not updated — the wording now has one home (offsite_r434_test.go), because duplicating it creates the second source that makes the next correction land in one file and not the other. Its signal fragments (69, 4, read-only) are unchanged.

R-435 written into the alarm's own documentation, no threshold changed

The comment above snapshotDropFraction now states what the detector does not see: a mass deletion, yes; one app being wiped, no. demo-hp's baseline is 69 across 9 apps, so ~35 must go before it speaks and one app's tag is ~9; offbox.go:1388 runs forget --prune grouped by host,tags, so the blind spot sits on the most likely single-app failure. The comment says explicitly that the numbers must not be lowered to "fix" this, and that per-app detection needs a second signal keyed on the per-tag count.

2. The stopping line, as it now reads in all three places

Register — documentation/backlog/OPEN-ITEMS.md, a new section in the voice this file uses for a settled decision:

DECIDED — the backup and restore arc is CLOSED FOR BETA (2026-09-01) … CLOSED FOR BETA at controller v0.232.0 / hub v0.111.1. … What is finished, and proven live: everything a customer does for themselves … Rows 1, 2, 3, 3b, 3c, 6, 7 and 14 of 07 §8 — every one PROVEN. … THIS REOPENS IF: a customer-facing recovery path is found broken; or Hetzner's answers change what the snapshots are worth …; or a real customer's data is at stake in one of the deferred rows.

Architecture — documentation/architecture/07-backup-architecture.md, at the head of §8, so a reader of the matrix meets it before the blanks:

THE ARC IS CLOSED FOR BETA — read this before the blanks below (2026-09-01) … The blanks in the RTO column below are now blank ON PURPOSE, and that is the whole difference. … NO STATUS MOVED ON THE DAY THIS WAS WRITTEN, because nothing was proven that day. A stopping line that promotes a row is a stopping line that lies.

Operator page — STATUS.md, under Decided — and what would reopen each:

THE BACKUP AND RESTORE WORK IS FINISHED FOR BETA. DECIDED 2026-09-01. … What is parked until after beta: everything only I do, with you — rebuilding a machine as itself, losing a whole box, recovering from ransomware, restoring the hub, and losing Hetzner. Six of these have never been timed, and the hub has never been restored. They are written down, they are real, and none of them stops a beta customer.

3. The deferred set — row numbers, not a description

07 §8 rows 4, 8, 9, 10, 11 (and 11b), 12, each tagged [BETA-DEFERRED] in its status cell.

row	failure	status today
4	primary drive dies — the drive-loss journey	PARTIAL
8	host dies, drives intact — a host rebuilt as itself	IMPLEMENTED, never executed
9	whole box lost (fire/theft)	IMPLEMENTED / UNPROVEN
10	ransomware / malicious deletion	PARTIAL
11 (+11b)	hub lost — a hub restore	UNPROVEN, never performed
12	off-site provider lost (Hetzner)	[FACT] only

grep -n '\[BETA-DEFERRED\]' documentation/architecture/07-backup-architecture.md returns eight lines — the seven tagged rows plus the one line in §8's header that defines the marker. That is stated in the register rather than left for the reader to trip over.

A NUMBER IN THE BRIEF WAS WRONG AND IS CORRECTED IN PLACE. The brief said "six rows of §8 still have no measured time". Six rows are DEFERRED; ELEVEN carry a blank RTO — counted, not estimated: 4, 5, 8, 9, 10, 11, 11b, 12, 13, 14, 15. The other five are blank for reasons that are not deferred work, and collapsing them into one number is how a blank stops meaning anything:

• row 5 — PROVEN by construction; a derived copy, so there is no recovery to time.
• row 13 — NONE for host-loss by design; R exists in zero system copies.
• row 14 — PROVEN; the break-glass route works and has simply never been stopwatched.
• row 11b — a consequences note attached to row 11, not a recovery row.
• row 15 — an open DEFECT (R-104, the stale-lock path). It is NOT inside the stopping line and must not be read as parked by it. This one matters most: parking a live defect by accident is the failure mode a stopping line invites.

No status was moved. Nothing was proven today.

4. The two questions

documentation/runbooks/provider-questions-2026-09-01.md — both drafted ready to paste, linked from R-95 and R-433, not sent, and no provider API was called (§11-D stands).

• Q1 — can the MAIN account retrieve individual files from a snapshot, without a whole-box restore? Why: if it cannot, the only route is a rollback that hits every customer on the box and destroys newer snapshots — the snapshots would then protect almost nobody. Decides how urgent R-95 is.
• Q2 — on rclone serve restic --stdio, is --append-only enforced server-side or chosen by the client? Why: if enforced, the box cannot delete its own backups, with no new machine and no data move. Could make R-95 disappear.

Each carries how to read either answer, including the branch where the lead is worthless and should be recorded as dead rather than left looking promising. A dated DUE-CHECKS row (R-433, 2026-09-15) tracks the reply — the 2026-07-27 snapshot check that sat unconfirmed for 36 days is the scar that block exists for, and it was never entered.

5. The register

	before	after
file lines (wc -l)	620	688
total R- rows	181	182
open-state rows	170	169
closed / decided / answered rows	11	13

(The brief's "620 lines and 181 open" matches the file length and the row count exactly; the count of rows whose LEADING verdict is actually open was 170, which is the number that matters and is not the same thing.)

A correction to my own commit messages, made here rather than by rewriting history: the db38f4c message says 621 -> 688 and the 1a1b32b message says 688 -> 700. Both are wrong. I mixed two measures — wc -l and a Python line split that counts the trailing newline as a line — and then carried the error forward. The table above is wc -l throughout and is the number to trust: 620 → 688. It changes nothing about the work and it is exactly the class of sloppiness this project files rows about, so it is named rather than quietly fixed.

Rows closed: R-434 — with the deletion-not-replacement reasoning, and with the fact that its own "blocked on R-433" verdict was wrong recorded in the closing cell.

Rows kept open and marked:

• R-95, R-433 — BLOCKED-ON-PROVIDER, both pointing at Part 3's file. R-95 records that it spent one day demoted on a clause that did not hold, and that on today's evidence it belongs back near the top. That is a proposal. I have not re-ranked it; the order is Viktor's.
• R-430 — LATENT, with the trigger stated as a trigger: it becomes live the moment delete is withdrawn from the box, which is exactly what R-95's remedy does by either route. So it is a precondition on the R-95 build, not a follow-up — settle it in the same change or the crash-lock self-heal ships already broken.
• R-435 — open. The limitation is now in the code's own documentation, but documenting a blind spot is not covering it, and STATUS.md and R-431 both still say "noticed within a day".

Row filed: R-437 — the register compression sweep, owed and scoped.

Compression was measured and deliberately not run, and the reason is one line as the standing rule requires: only 12 rows / ~25 KB of 316 KB (about 7 %) carry a closed leading verdict, so the sweep buys little and touches everything — and it is the exact operation that misfiled seven rows in August (R-378, and the seventh, R-87, sat wrong for nine days — R-405). Running it as the tail end of a session about something else is how that happened the first time. R-437 carries the scope so it can be picked up cold.

Also corrected in place: the ranking paragraph's item 1 said the snapshot mitigation was "armed (daily 00:00, keep 7), but it has taken zero snapshots so far". Both halves were wrong — seven snapshots exist (R-429) and "armed" was withdrawn the same day. The ORDER of that list is unchanged.

6. Hub deploy and its verification

step	result
image built + pushed	gitea.dooplex.hu/admin/felhom-hub:0.111.1 (25 M)
clean-tree gate before build	git status --porcelain empty, HEAD == origin/main @ db38f4c
manifest bumped	manifests/hub.yaml:128 → :0.111.1, committed 0f65f7a, pushed
ArgoCD hard refresh + deliberate sync	sync=Synced health=Healthy rev=0f65f7a… — the revision equals HEAD
rollout	deployment "hub" successfully rolled out
deployed image	gitea.dooplex.hu/admin/felhom-hub:0.111.1
running binary	felhom-hub 0.111.1 (built 2026-09-01T16:35:04Z)

No kubectl set image, no kubectl apply.

The live trigger was NOT run, and the reason is not a shortcut

The brief asked me to "trigger the alarm once through the hub's own path". I did not, and I am naming it rather than quietly substituting.

This alarm only fires on a real fall in a real customer's snapshot count. Firing it live therefore means one of two things: deleting a real customer's off-site backups (the destructive act this session explicitly is not), or POSTing a falsified report claiming demo-hp had lost its backups — which would write a fabricated data point permanently into that customer's report history, move its drop latch and baseline, and send Viktor a second alarm mail about a real box, five hours after the first one already confused him (STATUS.md item 5). Fabricating customer telemetry to satisfy a verification step is the wrong trade in a project whose whole doctrine is that a measurement must mean what it says.

What was done instead covers both halves of what the live trigger was for:

1. Does the deployed artifact carry the sentence? Proved on the bytes of the running pod's binary — §1, with both controls.
2. Does the production path emit it? Proved by three tests that drive saveOffsiteReport → Check() → notify and read the message the operator would receive, plus the stored-row test that pins mail and audit row together — and all three red-proofed.

What is therefore still unproven, stated plainly: that the dispatcher delivers this particular message to a mailbox. That leg was exercised for real on 2026-09-01 at 12:29 UTC by the previous session with the old text, so the routing is known good; only the new wording has not travelled it.

7. Explicitly

No controller release. No agent release. No image built for either. No golden baked, and none owed — the golden-currency gate reads newest released controller 0.232.0 / newest golden baked 0.232.0. The floor is unchanged at 0.232.0. felhom-controller and felhom-agent working trees were not touched.

python3 scripts/unproven.py --summary: NOT WALKED 35 of 55 — unchanged. No claim moved, which is correct: nothing was proven today. All 14 felhom.eu gates green before every push.

CI, checked by run id against head_sha rather than assumed (the R-417 recipe — the actions/jobs endpoint, paged to the end):

run id	commit	conclusion
500	db38f4c — hub v0.111.1 + the stopping line	success
501	0f65f7a — manifest bump to 0.111.1	success
502	1a1b32b — REPORT + R-437	success

8. Observations

1. strings stops at the em dash, so the alarm sentence appeared truncated in the deployed binary and briefly looked like a bad build. strings scans ASCII by default and the message's — terminates the run; the fix is LC_ALL=C grep -aoP on the bytes. NOT-A-FINDING: this is the project's already-recorded accented-grep trap appearing on a new surface, so it needs no new row — it is a hazard of my verification method, not a defect in any product code. Recorded here so the next person grepping a binary for Hungarian or em-dashed copy does not read a truncation as a bad build, which is exactly how it read for a minute.

2. The dispatcher's cooldowns are in-memory and are lost on every hub restart, so any deploy re-arms every alarm's 6-hour cooldown. NOT-A-FINDING: it is stated in hub/internal/notify/dispatcher.go:18 as a deliberate, accepted trade. Noted because it is why a live trigger today would definitely have mailed, rather than being absorbed by yesterday's cooldown — it changed the decision in §6.

3. The brief's baseline for felhom-agent was two commits stale — it names 058b945 (2026-08-23) while main is 4586f0f (2026-09-01). NOT-A-FINDING: the agent was untouched either way, and 058b945 is a real commit, so nothing was ambiguous. Flagged only so the number is not copied forward into the next brief.

4. OPEN-ITEMS.md has no section called "Decided — and what would reopen each"; STATUS.md does. The brief sent the register text to that section by name. NOT-A-FINDING: resolved in the session by writing a new register section in that section's voice — the decision, then the condition that reopens it — which is plainly what the instruction meant, so there is nothing left to file. Recorded only so the next session does not hunt OPEN-ITEMS.md for a heading that has never existed there.

9. My own mistakes

• I wrote "blocked on R-433" into R-434 yesterday, and it was wrong. It cost nothing because the block lasted one day, but the reasoning error is the interesting part: I treated "we do not know what is true" as a reason not to touch a sentence that was known to be false. Removing a false claim never needs the true one. The row and the code comment now say so, and it is the only part of this session I would call a lesson rather than a task.
• I asserted the brief's "six rows" instead of counting, for about ten minutes. I wrote the register block with "six" in it before running the count that produced eleven, and only caught it because I decided to enumerate the rows rather than describe them — which the brief had insisted on for a different reason. The instruction that saved it was not the one aimed at this.
• My first [BETA-DEFERRED] claim over-promised. I wrote in the register that the grep "returns the set as a group and nothing else", then found the grep returns eight lines because §8's own header defines the marker. Corrected in place with the real count. It is a small instance of exactly the class the register spent 2026-09-01 documenting — a claim about an instrument that had not been run.
• I preserved REPORT.md twice in one day and should have noticed the pattern the first time. REPORT.md held the only copy of the R-331 report this morning and the only copy of the drill report this afternoon; both are now siblings (REPORT-r331-backup-card.md, REPORT-drill-r95-recovery-2026-09-01.md). The convention says durable content may not live only in the overwritten file, and it has now been violated twice in a day by two different sessions — which suggests the convention needs a gate, not more diligence. Not filed: I am not filing a row for it in a session that already declined a register sweep; it is named here for whoever picks up R-437.