Rehearsal 2026-08-09: the walk stops at P3 — R-273 blocks every install fleet-wide
gates / gates (push) Successful in 24s

P1 uninstall, P2 preflight, P3 install. The install FAILED at step 5/8 in 44s,
and the two rank-1 findings are both on the setting-up path a tester's visit is
made of. Eleven register rows minted (R-269..R-279); ceiling moves 268 -> 279.

R-273 (RANK 1) — the hub vouches agent 0.128.0; that version was published as a
Gitea PACKAGE but never git-tagged. Since R-183 the installer correctly pins its
config fetches to raw/tag/v<vouched>, so every fresh install and every reinstall
now 404s as root, mid-install. Measured: main 200, v0.127.0 200, v0.128.0 404.
This is R-184 arriving; release-agent.sh:23 already documents the exact hazard.
Existing boxes are fine (self-update takes the binary from the registry).
NOT fixed here — publishing a release tag is outward-facing and the runbook says
stop and report. One command unblocks it; it is in STATUS.md.

R-272 (RANK 1) — Felhom's own uninstall leaves the condition that makes Felhom's
own reinstall refuse. It installs dnsmasq at day-0, then on teardown removes the
snippet and RESTARTS the daemon unconstrained (process start time lands inside
the uninstall window), which grabs 0.0.0.0:53; the next preflight then refuses,
and the message reads as though the owner's LAN DNS is at fault.

R-274 — a local golden is adopted with no version and no sha check; the manifest
vouch is consulted only on the fetch path. demo-hp's local copy is controller
0.192.0 against a vouched 0.210.0, and below the 0.200.0 where the recovery
screen shipped. Not yet observed end-to-end (R-273 killed step 5 first).

Also: R-275 orphaned credential backups + uid reuse, R-276 the wg tunnel outlives
the uninstall, R-269/270/271 from the token rotation, R-277 three hub surfaces
misreport a healthy off-site tier, R-278 demo-felhom six days unprotected,
R-279 no operator-triggerable off-site run.

Two hypotheses of mine were tested and REFUTED rather than shipped as findings:
the leftover sudoers file is inert (sudo skips dotted filenames), and demo-hp's
off-site tier was healthy all along - I had misread the hub and said so.

STATUS.md records the three rulings §8.3 asked for, with the floor CORRECTED to
its live value 0.200.0 and the count corrected to twenty.
This commit is contained in:
2026-08-09 10:59:44 +02:00
parent 34646295dc
commit b1afbb8a4d
9 changed files with 2280 additions and 41 deletions
+79 -37
View File
@@ -1,6 +1,6 @@
# STATUS — what works, what's broken, what's next
**Updated 2026-08-08.**
**Updated 2026-08-09.**
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
> part of it in plain words, and **nothing may exist only here**. Not `CONTEXT.md`, which is technical
@@ -23,9 +23,25 @@ destroyed on purpose and its files came back byte for byte identical — four ti
their recovery code got everything back with **no command line inside the machine at any point**,
in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252, R-253 — closed.)*
## Broken right now — and it stops every new machine
- **No machine can be set up from scratch today.** The install dies four steps in, on every host, and
it needs one command from you to unblock (below). Nothing already running is affected — the two demo
machines and the tester's box are fine. It is only the *setting-up* path, which is the one his visit
is made of. *(R-273)*
- **A rebuilt machine may come back on software from last week** — the install will happily reuse an
old copy it finds lying on the disk, without checking it against the version you approved. The copy
on the demo machine is old enough to predate the recovery screen. *(R-274)*
- **Taking Felhom off a machine leaves the one thing that stops it going back on.** We install a small
network service at setup; removing Felhom restarts it without its settings, it takes over the port
the next install needs, and the next install then refuses — blaming the owner's network. *(R-272)*
- **A machine we removed keeps its private line to us open.** *(R-276)*
- **demo-felhom has not had an off-site backup in six days** and is waiting for a recovery code nobody
has entered. demo-hp, rebuilt the same day, recovered by itself. *(R-278)*
## What's broken
- **Nothing new is broken.** The *check* against a fourth secret-in-a-page covers 4 pages of 27, and
- **Nothing else new is broken.** The *check* against a fourth secret-in-a-page covers 4 pages of 27, and
the cheap one covering all of them is blind to the shape that shipped. *(R-255)*
- **An already-paired box is still told to pair itself**, 25 minutes on. *(R-214, R-235)*
- **A backup that covered nothing still calls itself „Sikeres".** *(R-240)*
@@ -35,48 +51,77 @@ in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252
accumulates. *(R-244)*
- **Putting restored files back where they belong is still manual.** *(R-213)*
## Fixed today — four things the machine knew and did not say
## The reinstall rehearsal — what one afternoon on one machine found
All one family: something the box already knows, thrown away or drawn as its opposite.
**We took a working demo machine, removed Felhom, and tried to put it back the way a person with their
own hardware would.** It got three steps in. That is the finding: **nine defects, two of which stop a
visit dead**, and none of them were visible from reading the code — the last three sessions of review
did not see any of them.
- **A rebuilt machine can set up its own recovery again.** The one fact the setup needs was written
only the first time, and a rebuild replaced the configuration while leaving the note saying
"already done". It is now checked and re-written every minute instead of remembered once, so a
hand-edited or restored configuration heals too. **This was the last item blocking a customer from
something we promise them.** *(R-221 — agent 0.128.0.)*
- **A disk we failed to read is no longer drawn as a healthy empty one.** No figures, no bar, and it
says so: „A tárhely mérete most nem olvasható ki." *(R-259 — controller 0.210.0.)*
- **A backup tick now answers about that app.** It went green because *some* backup file existed and
*some other* app's database dump had succeeded most recently. Now: that app's own result, and
**no mark at all** when we have none. *(R-258 — controller 0.210.0.)*
- **Our own alarm no longer points at a page that may not exist.** A check run now gives up after
five minutes rather than hanging until something else kills it, and the mail says how long it ran.
*(R-265.)*
- It stopped at the install, on a missing version label *(R-273)* — and the step before that had
already refused, because removing Felhom leaves behind the thing that blocks reinstalling it
*(R-272)*.
- Removing Felhom also leaves five files holding old keys *(R-275)* and a live private line to us
*(R-276)*.
- Along the way: rotating a leaked key does not actually revoke the old one until the service is
restarted *(R-269)*, the written-down recipe for that rotation is missing a step *(R-270)*, and the
alarm it raises can never be closed because the fix it recommends is what silences the all-clear
*(R-271)*.
- **It also caught me being wrong.** Three separate hub screens agreed that demo-hp had no off-site
backup. It had eighteen. I told you the fleet's backups were down and had to take it back an hour
later — the screens were misleading, but I should have asked the machine before telling you. *(R-277)*
**Not fixed, and said rather than glossed:** that failed disk reading still reaches us as "0 of
0 GB". It is the quiet direction — it can only miss a true alarm, never raise a false one. *(R-266)*
**The data was never at risk** and is not now: the test files sit in an off-site snapshot with their
fingerprints recorded, plus two local copies. Full account:
`documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`.
**And R-221 is now proven on hardware, not just in tests** — on a demo machine we removed the one
line, watched the setup screen refuse, waited one minute, and watched it go green by itself with
nothing restarted. Every other line of that file came back identical.
## Three rulings, written down so they stop living in a conversation
- **The managed-update floor.** It is deliberately parked, and the trigger to raise it is **the first
machine that is not ours**; after that it moves with the publish train. Worth knowing alongside it:
the updater always aims at the floor, never at the newest, so a machine at or above the floor
updates to nothing. **Correction to the number that was going round: the floor is live at 0.200.0,
not 0.156.0** — checked twice today, on the hub page and in both machines' own logs.
- **R-264 is decided.** Build a reader for guest-network health, the staged-update pair, the
restore-test depth pair, and the two backup-integrity timestamps. Record a stated "no reader
wanted" for the repaired-recently flag, the tier-applied timestamp, the config fingerprint, the
per-stack object and the drive-migration marker. Decide the reporting-disabled flag on its own
merits — if that is a state we support, it must be visible or a staleness alarm will one day fire
on a machine that is fine. A "no" ends by changing the allowlist reason from *arguably owed* to
*deliberately not consumed* — not by ripping out an emitter, which is a two-repo change that also
breaks a shared fixture. **The implementation is its own session.** And **it is twenty facts, not
twenty-one**.
- **R-268 is closed** — the leaked key is rotated, and the rotation is proved in both directions
rather than assumed.
## Fixed 2026-08-08 — four things the machine knew and did not say
A rebuilt machine can set up its own recovery again *(R-221, agent 0.128.0 — proved on hardware)*; an
unreadable disk is no longer drawn as a healthy empty one *(R-259)*; a backup tick now answers about
*that* app *(R-258)*; our own alarm no longer points at a log that may not exist *(R-265)*. Still true
and not glossed: a failed disk reading still reaches us as "0 of 0 GB" — the quiet direction, it can
only miss a true alarm, never raise a false one *(R-266)*.
## What we're working on
- **Widening the check** so a fourth secret-in-a-page is caught by a machine. *(R-255)* · **Deciding
the twenty-one** — each gets a reader, or stops being sent. *(R-264)*
- **Widening the check** so a fourth secret-in-a-page is caught by a machine. *(R-255)* · **R-264 is
now decided** (above); building the readers is a session of its own.
- **Proving the hub really keeps the old sealed key** when a machine re-seals. *(R-198)* · Still open,
none urgent: *(R-256, R-257, R-261…R-263, R-266)*
## Waiting on you
- **A watching moment, five minutes.** Today's recovery fix is proved by removing one line from a
demo machine's config — backed up first, disposable machine, no customer data near it — and
watching the setup screen go green on its own. Nothing is destroyed. Say when.
- **One approval, three values this time.** Hub → Configuration → Day-0 artifacts: Golden
**0.210.0**, Agent **0.128.0**, minimum agent **0.127.0** (unchanged) → Save. Each was checked to
be downloadable and selectable before being written here. **Agent 0.128.0 is the one that carries
today's recovery fix**, so a new machine needs both, not just the image. It supersedes the 0.209.0
approval you already gave, and it is reversible. *(R-242)*
- **One command, and it unblocks every install.** The version you approved yesterday was published as
a download but never given its version label in the source repository, and the installer looks it up
by that label. The code is right and already shipped; only the label is missing.
`cd felhom-agent && git tag v0.128.0 28ba8593b8 && git push origin v0.128.0`. I did not do it
myself: publishing a release label is not a thing to do quietly in the middle of a measurement.
*(R-273)*
- **Then say go, and the rehearsal resumes** where it stopped. Nothing has to be redone — the test
files are already in an off-site backup and their fingerprints are recorded.
*(Both items previously here are done: the approval is saved — golden 0.210.0, agent 0.128.0, minimum
0.127.0 — and the recovery fix was proved on hardware.)*
## DooPlex infrastructure — separate from the product
@@ -97,11 +142,8 @@ being readable.*
connections are kept, the answer is held for a minute, and the old artifacts are gone. Worst case
is 5 s, once a minute at most. **Your instinct to prune was right and my measurement said
otherwise** — trimming to ten of each halved the slow path. *(R-267 — closed.)*
- **I printed a live access token into a session log** while setting up today's drill, and I am
telling you rather than quietly rotating it. It only opens the agent's private channel to one
demo guest, on a wire that exists solely between that host and that guest — not reachable from
your network or the internet, on a disposable machine with no customer data. Rotating it also
means updating the guest, so it is a deliberate act, not a background one. *(R-268)*
- **The access token I printed into a log yesterday is rotated**, and I checked it both ways: the old
one is refused, the new one works, and the machine's own channel is back up. *(R-268 — closed.)*
- **Our build-check alarm has one gap left.** A run that hangs is now cut off after five minutes and
the mail says how long it took — but **whether the alarm fires at all when the machinery kills a
run outright is still unverified**, and we have not claimed otherwise. *(R-265)*