Audit doc, STATUS one sentence, capability-map first-hour row (walk re-proven, day-one
off-site sentence narrowed: R-720/R-726/R-727), teardown in four layers, R-600 measured again.
Secret scan over all audits: 0 hits, positive control 1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
- 09 §6.1 phase table (copying, undoing, undone), §6.1a SHIPPED with the two
live-only defects, §6.4 part 1 SHIPPED.
- Capability map: a failed update is undone by the box - PROVEN-LIVE.
- Live evidence on 9202: three apps undone by the product with seeds before
the backup, after it and seconds before the press read back; cut-off copy
held honestly; power cut during the undo resumed; manual press after undo.
- Register: R-637, R-639, R-641, R-642 closed; R-638, R-640 narrowed; R-643
ruled; R-646 opened. STATUS asks the floor question.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
- 09 §3: decision 19 (the copy method is chosen by a bake-off) and 20 (the
full-system backup waits for the update leg, inside its window; built later).
- Bake-off on 9202, docmost / romm / vikunja: both methods pass every case;
the folder copy wins because an app with no database server gets no dump,
so dump-and-load would need the folder copy anyway. 1-5 s extra downtime,
~420 MB/s, disk = the volumes.
- R-645 filed: lifting an update hold by hand lets the recovery unit be
re-captured with the failed definition within seconds.
Documents and evidence only; product code follows in the controller.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
- Part 1 (09 §6.1a, audit): the undo performed by hand on 9202 for docmost
(PostgreSQL), romm (MariaDB) and vikunja (SQLite volume) - all three came
back with data written before AND after the backup. The product's loader
cannot do it: over a migrated PG database it fails on the new tables'
foreign keys; over MariaDB it leaves them behind. A truncated PG copy loads
rc 0 into an empty database. No-DB apps have no last-second copy.
- Part 2: one press jumps A -> C; the box's catalog clone is depth 1.
Ladder format recommended: update_ladder in .felhom.yml, not git history.
- Part 3: memory watch red-proof results (harness change in the catalog repo).
- Part 4 (09 §6.4): ten parts, ~22 evenings; one open point (R-643).
- Rows R-637..R-644 opened; R-446/450/451/462/463 updated. STATUS, CONTEXT.
No product code. Live catalog untouched; 9202 back on it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
- 09 §3: decisions 11 (one window = a leg of the backup chain), 12 (automatic,
per-box switch on by default), 13 (the test decides, not the tag - replaces
decision 3's "never across a major"), 14 (the ladder), 15 (the box undoes a
failed update - replaces §6.1's no-auto-undo), 16 (Postgres majors converted
by the box), 17 (digests), 18 (fleet view, later).
- 09 §3b marked ANSWERED with a pointer per question; kept as the reasoning.
- 09 §6.2 rewritten to the ruled shape; §6.1 abort paragraph and §4 point at 15.
- Register: R-450, R-451, R-446, R-463 cite the decisions.
- STATUS: the seven questions no longer wait on the operator.
Documents only. No product code.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
A (R-630, the P1): paperless-ngx, same app same button - failed at +313.0s with the app stopped
under v0.261.0, done at +53.4s now, with no "no probe container" warning because the explicit
healthcheck.container resolved the target.
B (R-633): the busy guard fires - RemoveStack REFUSED (busy): a backup or restore is running - and
the live proof caught it answering HTTP 500, because router.go maps remove errors by grepping the
error TEXT. v0.262.1 makes it a typed error and a 409; re-proven live.
C (R-634 half): a half-state with app.yaml on disk and deployed=false answered 200, leftovers NONE.
Under v0.261.0 the same call said "not deployed".
F (R-614): phase done before the remove, no phase at all after redeploying the same name.
Also: the first B run proved NOTHING and nearly went down as a pass - the refusal came from the
pre-existing "still running" check, not the new guard. Recorded.
09 6.1 and 8.8, the capability map, and STATUS updated. R-625 and R-634's mechanism are named as
owed, not half-done. Register 325.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
All 28 walked on scratch guest 9202 against the private drill catalog. 26 deployed, 6 proven,
5 inconclusive, 14 with no upstream edge, 1 failed honestly (outline 1.9.1->1.10.1, HELD with the
right sentence), 2 undeployable - one (plant-it) by design, refused by the lifecycle gate, proven
live for the first time. Each app also got the half the update night skipped: a restore from its own
copy with the seed read back again - 21 restored, 2 correctly REFUSED per 07 6.2.
R-630 RAISED TO P1 by measurement: a stack with NO probe container does not skip verifying - it
waits out the full health timeout and HOLDS, stopping an app whose three containers read healthy.
The controller's own words: "not healthy within 5m0s (last: no probe container)".
R-633 opened: a remove sent during a restore reports success and leaves a container restarting with
a live public route. The product already refuses that clash for update and for restore, naming the
blocker; remove has no such guard.
R-634 opened: an app can be running, healthy and serving while recorded as deployed=false, and is
then unremovable. Reproducible alone on sparkyfitness; concurrency-linked on two others.
R-631 and R-632 CLOSED. Register 321 -> 323. Seven interventions, six of them my own harness -
named, with what each cost. No product code. The live catalog's image: lines are byte-identical to
the start of the night.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Part 1: tandoor/zipline/wger probes corrected in the catalog and red-proofed live on 9202 in both
directions - "Nem egeszseges" with the front door serving 200, then "Fut" after the real sync with
no redeploy. tandoor's failed edge re-walked: done at +41.1s where it was failed at +361.9s.
Part 2: fifteen proven versions on the live catalog, one commit per app; the guarded Update pressed
on four apps on demo-hp, all four done.
Opened: R-630 (paperless-ngx's probe has never run on any box - a silent absence, worse than the
wrong probe that was found in one night), R-631 (five templates no static rule can judge),
R-632 (28 of 53 templates never deployed by any drill). Closed: R-618.
Register 318 -> 321. No product code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
THE REPAIR. Last night's append regex ate the state cells of R-446 and R-458, left them as a stray
fourth cell on duplicated copies of R-626 and R-625, and split the table with blank lines. The
register read 317 rows for 315 findings. Both cells restored from the cells that carried them, the
two duplicates deleted, and 15 blank lines that split the register into 12 separate markdown tables
removed. Every row's text is byte-identical afterwards, proven by diff; no row added or removed.
THE RED-PROOF FOUND AN OLDER INSTANCE: R-254 lost its state cell on 2026-08-08 (59527d0) and had
rendered without a State column for 45 days. Its own verdict sentence was the cell; it has it back.
THE GATE. scripts/register_shape_gate.py, gate 14 in repo_gates.py and reached by the pre-push
hook: a row that does not end with `|` (an eaten state cell), a duplicated id, or a blank line
splitting the table. Four decoys in test_gate_decoys.py — three convicting on the exact damage
shapes, one asserting a healthy register still passes. The red-proof corrected the gate twice: a
first draft counted CELLS and convicted 125 innocent rows (register cells carry literal `|` in
prose and shell snippets, so a row cannot be split on `|`), and it skipped malformed rows before
counting ids, reporting 5 duplicates where there were 2. It then immediately caught a blank line
left by this session's own next insert.
R-618's rank now reads P1-HIGH in both its title and its state cell.
HETZNER ANSWERED, AND I FIRST SAID THEY HAD NOT. The operator supplied ticket #2026090103040671.
Q1: with the MAIN account, files and directories can be downloaded from a snapshot (Storage Box
docs govern, not the Storage Share FAQ); a restore reverts the whole box. Q2: --append-only is
enforced by pinning `command="rclone serve restic --stdio --append-only path/to/repo"` to the key
in authorized_keys — so append-only IS expressible on a Storage Box, which is what R-95 was blocked
on. Neither is measured; R-436's due check now asks for the measurement, not the question.
My error is filed as R-628 and is precise: the query was fine — a control returns 201 threads and
reaches SENT and TRASH — and the thread genuinely is not in this mailbox. What was invented was the
step from "absent here" to "Hetzner has not answered". An absent record in one place cannot answer
what someone else did. The rule is now in the Gmail-access memory.
R-629: the drill repo inherited has_actions from the migrate call and mailed the operator 47 CI
failures overnight, into the mailbox that was carrying real off-site alarms. Actions disabled and
verified; 09 §6.5 now makes it a step of creating a drill repo. The 47 mails are the operator's to
clear: subject:"gates FAILED in admin/app-catalog-drill".
Gates: repo_gates.py --fast — all 16 OK, including the new one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The drill is complete. Teardown done in three layers plus Gitea; the live catalog's every `image:`
line is proven identical to before.
WHAT WAS MEASURED. 21 edges across 19 apps, on scratch guest 9202 through the product's own
guarded Update, against a PRIVATE DRILL CATALOG so the live catalog carried no test reference at
any point: 14 proven, 3 failed, 4 inconclusive. Each app seeded and read back through its OWN
front door, with a negative control on every readback. Ten of the fourteen printed a verbatim
migration line. Up from the three apps this project had ever measured.
THE RESULT THAT MATTERS. R-618, P1: three of the 53 templates name a health probe the app does not
answer, and because the guarded update WAITS on that same probe, a SUCCESSFUL update ends by
STOPPING a working app. tandoor was measured serving HTTP 200 on the new version at four samples
across five minutes, docker's own healthcheck green, and was then stopped and the household sent
to a restore they did not need. zipline and wger are the same defect, both confirmed live. The
gate that catches all three is static and cheap: both health checks already sit in the same file.
WHAT THE NIGHT ANSWERED that was open. The UNATTENDED HOLD (312.9 s, pressed once, never again) —
which needed a purpose-built image store, because the rule that makes automatic updates safe is
the same rule that refuses the obvious way to break one. MariaDB across a major through the real
button, all four observables, first time. PostgreSQL across a major, refusing exactly as predicted,
with the conversion costed at ~9 s of engine work. There is NO single-flight: five updates ran at
once and all ended honest. And the two EARLY power-cut phases nobody had cut in.
TWELVE NEW ROWS (R-615..R-626), register 303 -> 315, and eight existing rows updated with what was
measured — including two CORRECTIONS: R-606 records the pre-flight refusals as reaching an English
household in English and they do not, and R-446/R-458 are both narrower than their rows state.
Two instrument fixes were needed before anything could be trusted: the unattended caller turned
every success into a timeout (R-623), and one of my own reproductions was wrong and is kept
labelled with what it actually measured.
Interventions: zero. No controller, agent or hub code written. The hub was never touched beyond
the floor the operator asked for.
Gates: repo_gates.py --fast, all 15 OK.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Part 0 — floor raised to 0.260.0, MinAgent 0.131.0 declared. 3 boxes below, all
down or blocked; both demo boxes SERVED.
Part 1 (R-610) — the DANGEROUS power cut, measured three times with three apps and
two cut mechanisms. All ended honest: resumed, completed, and pinned/installed/live
compose/docker inspect all agreed. vikunja's 2.6.0 migration had ALREADY run 0.64 s
after the cut decision and the seeded data read back intact — so the branch that is
one step from old-binary-on-migrated-database is now evidence, not argument.
Instrument limit stated: `starting` lasts under a second; all three landed in
`verifying`, which RecoverUpdates handles in the same branch.
Part 3 (R-611) — the night the previous session skipped without saying so. An app
updated with nobody pressing anything; a terminally-refused app was pressed exactly
once and never again over three passes. The unattended HOLD was NOT produced: the
within-a-major rule correctly refused the broken edge before it was attempted, so
Q4 still rests on the attended hold from slice 4. Said plainly rather than implied.
Rows: closed R-608/609/610/611; opened R-612 (P1 wishlist unusable on a fresh
install, and its error is a lie), R-613 (uptime-kuma healthy on its setup wizard),
R-614 (stale update phase survives a redeploy). R-520's pointer corrected.
Catalog: two drill pairs, both reverted; every image line byte-identical to
ff9717d3. The alpine:3.20 negative control a security review flagged is cleared.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Phase 0 — measured, never estimated:
- both demo boxes: 10 apps, 0 behind, 0 unknown
- 46 of 58 exact catalog pins are behind upstream; 39 within a major, 7 across
- 6 of 7 measurable floating pins have been repushed since the catalog set them
(R-446 is no longer theoretical)
- the "23 of 66 floating pins" figure repeated in four places was STALE; recounted
to 10, with the definition written down beside it
Three claims in the brief corrected, named first:
- R-589 was NOT open — it shipped in v0.258.0; only the row was stale
- the chaos-night canary is NOT a defect — both gates refused to certify by design
- the hub half of the report confirmed, with the nuance that the raw payload is
stored whole, so Slice 7 is cheaper than the row implies
Closed: R-524 (controller v0.260.0, proven live in both languages), R-520 (power cut
during a REAL version change — the pin goes back, the app runs, the page says so),
R-589, R-469 (MariaDB half). Filed: R-605, R-606. R-462's stale scope corrected.
09 gains §3 decision 10 (decided by CC unattended — operator may reverse), §3b with
the seven questions in the decision shape, §6.2/6.3 the two open slices, and §6.4 an
update night costed from R-462's real numbers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The three blockers yesterday's English walk found are closed and each proven on
a live system. The verdict is deliberately 'nothing known now stands in their
way' rather than 'the walk passed': fixes are not a journey, and the hour has
not been re-walked by a stranger on a fresh install.
Also filed: the HP demo box answers on no route this session has (R-601), the
cookie-vs-session language instrument trap that would have had me fix R-598
twice (R-602), and the apostrophe that silently never matches a rendered page
(R-603).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The setup code and the owner passphrase now follow the household's language,
one word longer in English so the entropy never drops (setup 3 hu / 4 en,
passphrase 5 hu / 6 en). List and count are chosen together so a caller cannot
pair an English list with a Hungarian count. Hungarian is byte-unchanged.
Three claims in the row were wrong and are recorded as such:
- the RECOVERY CODE is minted by felhom-agent from the EFF list and has
always been English; the hub does not own it and no row was added.
- no claim mail states a word count; the only count wording was the bind
page's passphrase hint, whose English half is now count-free.
- the proposed phone-safe filter removes 68% of the list (5270 of 7772
words) and was measured, then declined, with the reason in source.
Also: guide_quote_gate binds the English volunteer guide's three quoted
messages to the controller's English bundle — nothing did, so the guide would
have gone on quoting Hungarian after the fix. Seven decoys, all convicting,
including the name-for-fact one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Part 0 shipped as controller v0.258.0 (its own commit). Part A is the guide's
English twin. Part B is the walk: a box installed from scratch that day, used by an
English speaker following only the English guide and the screens.
VERDICT: not yet ready for an English-speaking tester, because the claim page — the
one screen between them and their box — is English chrome with Hungarian messages
(R-596, P1). Everything else held: the download page, the bilingual console, all
three customer mails, the bind page and its refusal, the dashboard's first language
from customer.language alone, both app pages, the whole catalog, the language switch
both ways — every one of them with zero Hungarian lines.
One intervention (I1 = R-494, filed 2026-09-14); the stop rule was not reached. The
walk exercised what 2026-09-14 could not: the graphical installer, the auto-reboot,
and the mailed link and self-bind page end to end — that walk's H1 is closed, because
this session had a mailbox.
R-214 CLOSED as a side effect and seen rather than reasoned about: the console's last
paint is now the bilingual "the box is linked" banner.
R-516 does NOT close, and the item-by-item note says why: more than half its twelve
items are about what a HUNGARIAN household reads, and an English walk cannot see
them. It now waits on a Hungarian walk with a second drive.
Rows opened: R-596 (P1, the claim page), R-597 (the setup code is three Hungarian
words), R-598 (the Backup page's protection warnings), R-599 (a drill's teardown is
blocked 30 minutes by report staleness and the 409 does not say so).
Golden 0.258.0 baked, published, vouched, with its record. The waiver was NOT retired
and the record says why in one line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Part C shipped the same day the pilot was read: fifty apps in three pushes, 1 031 of
1 032 strings. The English Apps list shows ZERO Hungarian app descriptions across all
53 apps — the only Hungarian left on it is the "Naprakész" badge (R-589) and the
language picker naming itself, which is correct.
The Hungarian Apps list is identical to the pre-slice capture once the per-session
CSRF token AND Docker's own "Up N hours" container string are normalised. Both
normalisations are stated in the evidence rather than applied quietly — the second
one moved because two hours of wall clock passed between captures, not because any
copy changed.
Fleet floor raised to 0.257.0 with the declared MinAgent 0.131.0, above the vouched
golden so the declaration carries it. demo-felhom went 0.255.0 -> 0.257.0 by itself
in under 12 seconds and THEN rendered the English tagline: the floor delivered the
feature, not a version string.
Rows: R-593 (papra describes a session-signing key as "the app's subdomain" — the one
string left untranslated) and R-594 (the catalog gate can convict a retrieval promise
but has no way to REGISTER a true one, which the shared vocabulary's design calls
for). R-560 closed. 281 -> 287 rows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
10-localisation.md §7 goes from [DESIGN, proposed] to [FACT], with the numbers it was
specced against corrected — and one correction chose an instrument rather than a
footnote. The catalog has 1 032 copy strings, not 835. 832 carry a Hungarian letter,
which was right. But the ASCII-ONLY Hungarian is ~120 strings, not three: „Aldomain"
appears 53 times and „A szerver domain neve" 53 times, and the three the plan named
(„Igen"/„Nem"/„Nincs") do not occur in this catalog at all. An accent-only gate passes
every one of them inside an English block — R-565's blind spot arriving again in a
different repo.
New §10.6 records what was proven live rather than reasoned about: the English pages
show English; the seven Hungarian pages are byte-identical before and after the push
apart from the per-session CSRF token; and a 0.255.0 box with the block synced onto it
renders identically and logs no warning, in a 93-line window that contains the sync's
own lines, so the absence is evidence and not a dead log.
Rows R-589 (the update badge is Hungarian on an English page), R-590 (the data-folder
card's backup promise, likewise, and it is a promise about the customer's files),
R-591 (Stack.Copy() deep-copies five Meta fields and not the new I18n map — safe
today, which is precisely why it is a row), R-592 (three defects inside the new
catalog gate, closed the same session, each found by its own decoy). R-560 updated:
Parts A and B done, Part C waiting on the operator's read of the pilot.
STATUS asks for that read, and for the floor to 0.257.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The image is BUILT but NOT PUBLISHED, and that is deliberate: publishing to
iso.felhom.eu is public and irreversible, and the runbook needs a proof install
on BOTH menu entries plus a reboot against the uploaded bytes. That is
supervised, so I stopped there. felhom-installer-1.29.0-pve9.2-1.iso, sha256
c67ceaa3…fb02, with every mechanically checkable criterion passing (G1, G2, G5,
G6, G7, G9, G16 — including both payload files byte-identical to repo HEAD).
The download pages still name 1.28.0, the image that IS published. Pointing
them at a file that is not there would hand every reader a 404. A new site gate
refuses the two pages naming different installer files or checksums, so
whoever publishes 1.29.0 cannot update one and forget the other.
Measured rather than read: the pairing banner is 24 rows on a 25-row console.
One row of margin — so the height is now pinned, because two more lines push
the HUNGARIAN code at row 5 off the top, and a banner whose code has scrolled
away is furniture.
R-587: two root-password files from July sit in the directory the public ISO is
published from. Both 404 on the bucket (against a 200 control), so nothing
leaked — but the only thing keeping them off is an --include pattern they miss
by an accident of naming. A pattern that protects by coincidence is not a
control.
R-588: release records live in two different places, which made me wrongly
conclude 1.28.0's gate had never been run. It had.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Part B's live proof is a LOG LINE rather than an e-mail, and the evidence page
says why: severityNotifies drops `info`, so every safe converted producer mails
nobody, and every converted producer above `info` describes something bad that
is not true. Triggering one would mean a false record on a real box's timeline
or a state change the fences forbid. The log line was built in v0.256.1 for
exactly this, after finding there was nothing to look at on either side of the
wire.
15:07:28 controller_started [hu-only] — Controller elindult (0.256.1)
15:08:25 controller_started [+household(en)] — Controller elindult (0.256.1)
The Hungarian sentence is identical in both, and the hub stored the Hungarian
in every case including the English-household push.
R-558 CLOSED, R-555 closed with it. R-585 filed: six producers still send
Hungarian only because their sentence arrives already finished from another
package — `offbox_enlarge_blocked` matters most, since it has no hub entry so
its raw sentence IS the household's whole mail.
10-localisation.md gains 10.4; STATUS rewritten for the operator with the two
decisions left (raise the floor to 0.256.1; whether to rotate the demo
password after R-584).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
admin