Rehearsal 2026-08-09: the walk stops at P3 — R-273 blocks every install fleet-wide
gates / gates (push) Successful in 24s
gates / gates (push) Successful in 24s
P1 uninstall, P2 preflight, P3 install. The install FAILED at step 5/8 in 44s, and the two rank-1 findings are both on the setting-up path a tester's visit is made of. Eleven register rows minted (R-269..R-279); ceiling moves 268 -> 279. R-273 (RANK 1) — the hub vouches agent 0.128.0; that version was published as a Gitea PACKAGE but never git-tagged. Since R-183 the installer correctly pins its config fetches to raw/tag/v<vouched>, so every fresh install and every reinstall now 404s as root, mid-install. Measured: main 200, v0.127.0 200, v0.128.0 404. This is R-184 arriving; release-agent.sh:23 already documents the exact hazard. Existing boxes are fine (self-update takes the binary from the registry). NOT fixed here — publishing a release tag is outward-facing and the runbook says stop and report. One command unblocks it; it is in STATUS.md. R-272 (RANK 1) — Felhom's own uninstall leaves the condition that makes Felhom's own reinstall refuse. It installs dnsmasq at day-0, then on teardown removes the snippet and RESTARTS the daemon unconstrained (process start time lands inside the uninstall window), which grabs 0.0.0.0:53; the next preflight then refuses, and the message reads as though the owner's LAN DNS is at fault. R-274 — a local golden is adopted with no version and no sha check; the manifest vouch is consulted only on the fetch path. demo-hp's local copy is controller 0.192.0 against a vouched 0.210.0, and below the 0.200.0 where the recovery screen shipped. Not yet observed end-to-end (R-273 killed step 5 first). Also: R-275 orphaned credential backups + uid reuse, R-276 the wg tunnel outlives the uninstall, R-269/270/271 from the token rotation, R-277 three hub surfaces misreport a healthy off-site tier, R-278 demo-felhom six days unprotected, R-279 no operator-triggerable off-site run. Two hypotheses of mine were tested and REFUTED rather than shipped as findings: the leftover sudoers file is inert (sudo skips dotted filenames), and demo-hp's off-site tier was healthy all along - I had misread the hub and said so. STATUS.md records the three rulings §8.3 asked for, with the floor CORRECTED to its live value 0.200.0 and the count corrected to twenty.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# STATUS — what works, what's broken, what's next
|
||||
|
||||
**Updated 2026-08-08.**
|
||||
**Updated 2026-08-09.**
|
||||
|
||||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
|
||||
> part of it in plain words, and **nothing may exist only here**. Not `CONTEXT.md`, which is technical
|
||||
@@ -23,9 +23,25 @@ destroyed on purpose and its files came back byte for byte identical — four ti
|
||||
their recovery code got everything back with **no command line inside the machine at any point**,
|
||||
in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252, R-253 — closed.)*
|
||||
|
||||
## Broken right now — and it stops every new machine
|
||||
|
||||
- **No machine can be set up from scratch today.** The install dies four steps in, on every host, and
|
||||
it needs one command from you to unblock (below). Nothing already running is affected — the two demo
|
||||
machines and the tester's box are fine. It is only the *setting-up* path, which is the one his visit
|
||||
is made of. *(R-273)*
|
||||
- **A rebuilt machine may come back on software from last week** — the install will happily reuse an
|
||||
old copy it finds lying on the disk, without checking it against the version you approved. The copy
|
||||
on the demo machine is old enough to predate the recovery screen. *(R-274)*
|
||||
- **Taking Felhom off a machine leaves the one thing that stops it going back on.** We install a small
|
||||
network service at setup; removing Felhom restarts it without its settings, it takes over the port
|
||||
the next install needs, and the next install then refuses — blaming the owner's network. *(R-272)*
|
||||
- **A machine we removed keeps its private line to us open.** *(R-276)*
|
||||
- **demo-felhom has not had an off-site backup in six days** and is waiting for a recovery code nobody
|
||||
has entered. demo-hp, rebuilt the same day, recovered by itself. *(R-278)*
|
||||
|
||||
## What's broken
|
||||
|
||||
- **Nothing new is broken.** The *check* against a fourth secret-in-a-page covers 4 pages of 27, and
|
||||
- **Nothing else new is broken.** The *check* against a fourth secret-in-a-page covers 4 pages of 27, and
|
||||
the cheap one covering all of them is blind to the shape that shipped. *(R-255)*
|
||||
- **An already-paired box is still told to pair itself**, 25 minutes on. *(R-214, R-235)*
|
||||
- **A backup that covered nothing still calls itself „Sikeres".** *(R-240)*
|
||||
@@ -35,48 +51,77 @@ in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252
|
||||
accumulates. *(R-244)*
|
||||
- **Putting restored files back where they belong is still manual.** *(R-213)*
|
||||
|
||||
## Fixed today — four things the machine knew and did not say
|
||||
## The reinstall rehearsal — what one afternoon on one machine found
|
||||
|
||||
All one family: something the box already knows, thrown away or drawn as its opposite.
|
||||
**We took a working demo machine, removed Felhom, and tried to put it back the way a person with their
|
||||
own hardware would.** It got three steps in. That is the finding: **nine defects, two of which stop a
|
||||
visit dead**, and none of them were visible from reading the code — the last three sessions of review
|
||||
did not see any of them.
|
||||
|
||||
- **A rebuilt machine can set up its own recovery again.** The one fact the setup needs was written
|
||||
only the first time, and a rebuild replaced the configuration while leaving the note saying
|
||||
"already done". It is now checked and re-written every minute instead of remembered once, so a
|
||||
hand-edited or restored configuration heals too. **This was the last item blocking a customer from
|
||||
something we promise them.** *(R-221 — agent 0.128.0.)*
|
||||
- **A disk we failed to read is no longer drawn as a healthy empty one.** No figures, no bar, and it
|
||||
says so: „A tárhely mérete most nem olvasható ki." *(R-259 — controller 0.210.0.)*
|
||||
- **A backup tick now answers about that app.** It went green because *some* backup file existed and
|
||||
*some other* app's database dump had succeeded most recently. Now: that app's own result, and
|
||||
**no mark at all** when we have none. *(R-258 — controller 0.210.0.)*
|
||||
- **Our own alarm no longer points at a page that may not exist.** A check run now gives up after
|
||||
five minutes rather than hanging until something else kills it, and the mail says how long it ran.
|
||||
*(R-265.)*
|
||||
- It stopped at the install, on a missing version label *(R-273)* — and the step before that had
|
||||
already refused, because removing Felhom leaves behind the thing that blocks reinstalling it
|
||||
*(R-272)*.
|
||||
- Removing Felhom also leaves five files holding old keys *(R-275)* and a live private line to us
|
||||
*(R-276)*.
|
||||
- Along the way: rotating a leaked key does not actually revoke the old one until the service is
|
||||
restarted *(R-269)*, the written-down recipe for that rotation is missing a step *(R-270)*, and the
|
||||
alarm it raises can never be closed because the fix it recommends is what silences the all-clear
|
||||
*(R-271)*.
|
||||
- **It also caught me being wrong.** Three separate hub screens agreed that demo-hp had no off-site
|
||||
backup. It had eighteen. I told you the fleet's backups were down and had to take it back an hour
|
||||
later — the screens were misleading, but I should have asked the machine before telling you. *(R-277)*
|
||||
|
||||
**Not fixed, and said rather than glossed:** that failed disk reading still reaches us as "0 of
|
||||
0 GB". It is the quiet direction — it can only miss a true alarm, never raise a false one. *(R-266)*
|
||||
**The data was never at risk** and is not now: the test files sit in an off-site snapshot with their
|
||||
fingerprints recorded, plus two local copies. Full account:
|
||||
`documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`.
|
||||
|
||||
**And R-221 is now proven on hardware, not just in tests** — on a demo machine we removed the one
|
||||
line, watched the setup screen refuse, waited one minute, and watched it go green by itself with
|
||||
nothing restarted. Every other line of that file came back identical.
|
||||
## Three rulings, written down so they stop living in a conversation
|
||||
|
||||
- **The managed-update floor.** It is deliberately parked, and the trigger to raise it is **the first
|
||||
machine that is not ours**; after that it moves with the publish train. Worth knowing alongside it:
|
||||
the updater always aims at the floor, never at the newest, so a machine at or above the floor
|
||||
updates to nothing. **Correction to the number that was going round: the floor is live at 0.200.0,
|
||||
not 0.156.0** — checked twice today, on the hub page and in both machines' own logs.
|
||||
- **R-264 is decided.** Build a reader for guest-network health, the staged-update pair, the
|
||||
restore-test depth pair, and the two backup-integrity timestamps. Record a stated "no reader
|
||||
wanted" for the repaired-recently flag, the tier-applied timestamp, the config fingerprint, the
|
||||
per-stack object and the drive-migration marker. Decide the reporting-disabled flag on its own
|
||||
merits — if that is a state we support, it must be visible or a staleness alarm will one day fire
|
||||
on a machine that is fine. A "no" ends by changing the allowlist reason from *arguably owed* to
|
||||
*deliberately not consumed* — not by ripping out an emitter, which is a two-repo change that also
|
||||
breaks a shared fixture. **The implementation is its own session.** And **it is twenty facts, not
|
||||
twenty-one**.
|
||||
- **R-268 is closed** — the leaked key is rotated, and the rotation is proved in both directions
|
||||
rather than assumed.
|
||||
|
||||
## Fixed 2026-08-08 — four things the machine knew and did not say
|
||||
|
||||
A rebuilt machine can set up its own recovery again *(R-221, agent 0.128.0 — proved on hardware)*; an
|
||||
unreadable disk is no longer drawn as a healthy empty one *(R-259)*; a backup tick now answers about
|
||||
*that* app *(R-258)*; our own alarm no longer points at a log that may not exist *(R-265)*. Still true
|
||||
and not glossed: a failed disk reading still reaches us as "0 of 0 GB" — the quiet direction, it can
|
||||
only miss a true alarm, never raise a false one *(R-266)*.
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Widening the check** so a fourth secret-in-a-page is caught by a machine. *(R-255)* · **Deciding
|
||||
the twenty-one** — each gets a reader, or stops being sent. *(R-264)*
|
||||
- **Widening the check** so a fourth secret-in-a-page is caught by a machine. *(R-255)* · **R-264 is
|
||||
now decided** (above); building the readers is a session of its own.
|
||||
- **Proving the hub really keeps the old sealed key** when a machine re-seals. *(R-198)* · Still open,
|
||||
none urgent: *(R-256, R-257, R-261…R-263, R-266)*
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **A watching moment, five minutes.** Today's recovery fix is proved by removing one line from a
|
||||
demo machine's config — backed up first, disposable machine, no customer data near it — and
|
||||
watching the setup screen go green on its own. Nothing is destroyed. Say when.
|
||||
- **One approval, three values this time.** Hub → Configuration → Day-0 artifacts: Golden
|
||||
**0.210.0**, Agent **0.128.0**, minimum agent **0.127.0** (unchanged) → Save. Each was checked to
|
||||
be downloadable and selectable before being written here. **Agent 0.128.0 is the one that carries
|
||||
today's recovery fix**, so a new machine needs both, not just the image. It supersedes the 0.209.0
|
||||
approval you already gave, and it is reversible. *(R-242)*
|
||||
- **One command, and it unblocks every install.** The version you approved yesterday was published as
|
||||
a download but never given its version label in the source repository, and the installer looks it up
|
||||
by that label. The code is right and already shipped; only the label is missing.
|
||||
`cd felhom-agent && git tag v0.128.0 28ba8593b8 && git push origin v0.128.0`. I did not do it
|
||||
myself: publishing a release label is not a thing to do quietly in the middle of a measurement.
|
||||
*(R-273)*
|
||||
- **Then say go, and the rehearsal resumes** where it stopped. Nothing has to be redone — the test
|
||||
files are already in an off-site backup and their fingerprints are recorded.
|
||||
|
||||
*(Both items previously here are done: the approval is saved — golden 0.210.0, agent 0.128.0, minimum
|
||||
0.127.0 — and the recovery fix was proved on hardware.)*
|
||||
|
||||
## DooPlex infrastructure — separate from the product
|
||||
|
||||
@@ -97,11 +142,8 @@ being readable.*
|
||||
connections are kept, the answer is held for a minute, and the old artifacts are gone. Worst case
|
||||
is 5 s, once a minute at most. **Your instinct to prune was right and my measurement said
|
||||
otherwise** — trimming to ten of each halved the slow path. *(R-267 — closed.)*
|
||||
- **I printed a live access token into a session log** while setting up today's drill, and I am
|
||||
telling you rather than quietly rotating it. It only opens the agent's private channel to one
|
||||
demo guest, on a wire that exists solely between that host and that guest — not reachable from
|
||||
your network or the internet, on a disposable machine with no customer data. Rotating it also
|
||||
means updating the guest, so it is a deliberate act, not a background one. *(R-268)*
|
||||
- **The access token I printed into a log yesterday is rotated**, and I checked it both ways: the old
|
||||
one is refused, the new one works, and the machine's own channel is back up. *(R-268 — closed.)*
|
||||
- **Our build-check alarm has one gap left.** A run that hangs is now cut off after five minutes and
|
||||
the mail says how long it took — but **whether the alarm fires at all when the machinery kills a
|
||||
run outright is still unverified**, and we have not claimed otherwise. *(R-265)*
|
||||
|
||||
Reference in New Issue
Block a user