d7d2f84257
gates / gates (push) Failing after 14m48s
Operator is moving back from the vacation home. Guests shut down cleanly first (checked: no running PVE task, no guest lock, no vzdump/restic in flight — the "reboot mid-vzdump leaves a stale lock and the guest won't autostart" trap), then the hosts. Confirmed off at the FABRIC rather than inferred from silence: the tailnet is healthy, dooplex is up, and both peers report "offline, last seen 1m ago". An absent ping is not evidence; an offline peer on a working tailnet is. Both customers blocked on the hub so the 30-minute host_stale and 60-minute host_down transitions do not fire four false alarms during the drive. IsCustomerBlocked gates every monitor plus the notification intake; it does NOT gate config pull or report intake, so the boxes return normally on power-up. STATUS.md carries a banner at the very top because the mute has a tail: while blocked, a box that FAILS to come back is equally silent. Unblocking on arrival is now the first thing anyone reads.
184 lines
12 KiB
Markdown
184 lines
12 KiB
Markdown
# STATUS — what works, what's broken, what's next
|
|
|
|
**Updated 2026-08-09.**
|
|
|
|
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
|
|
> part of it in plain words, and **nothing may exist only here**. Not `CONTEXT.md`, which is technical
|
|
> state written for Claude Code. **Items, not paragraphs. One screen.** If it does not fit, something
|
|
> belongs in the register instead.
|
|
>
|
|
> *Rebuilt from the register on 2026-08-07, from 258 lines. The old "what shipped recently" log is what
|
|
> the per-repo `CHANGELOG.md` files and the register are for, and is not restated here.*
|
|
|
|
## ⚠ BOTH DEMO MACHINES ARE OFF AND MUTED — unmute them when they are home
|
|
|
|
**Powered down 2026-08-09 14:08 CEST** for the move back from the vacation home. Guests stopped
|
|
cleanly first (no vzdump was running, no locks), then the hosts. Confirmed off at the fabric, not
|
|
merely unreachable: the tailnet is healthy and both peers report *"offline, last seen 1m ago"*.
|
|
|
|
**Both customers are BLOCKED on the hub, deliberately, to stop four false alarms an hour into the
|
|
drive.** Blocking gates every monitor and the notification intake; it does **not** gate config pull or
|
|
report intake, so the boxes come back normally on power-up.
|
|
|
|
> **THE TAIL, and it is the reason this banner exists: while they are blocked, a box that FAILS to
|
|
> come back up is also silent.** When the machines are home and powered on, unblock them and confirm
|
|
> both report:
|
|
>
|
|
> Hub → Customers → **demo-hp** → Unblock, and **demo-felhom** → Unblock.
|
|
>
|
|
> Then check both read ONLINE on Hosts. **Until that is done, the hub cannot tell you either box is
|
|
> in trouble.**
|
|
|
|
## What works
|
|
|
|
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer, who
|
|
sets their own password. They install apps from a catalogue of fifty-three, share files over the home
|
|
network, and open apps from a launcher or a shared link. Backups run on their own to three places — the
|
|
machine's drive, a second drive, and an encrypted off-site copy.
|
|
|
|
**The backup promise is proved, and so is getting the data back yourself.** A machine has been
|
|
destroyed on purpose and its files came back byte for byte identical — four times now. On
|
|
**2026-08-07 the household's own journey passed for the first time**: someone with a browser and
|
|
their recovery code got everything back with **no command line inside the machine at any point**,
|
|
in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252, R-253 — closed.)*
|
|
|
|
## The rehearsal finished. The data came back byte for byte; the journey did not.
|
|
|
|
**We wiped a working demo machine and put it back. All four test files returned identical — including
|
|
the two with Hungarian accents, checked as raw bytes, not as text on screen.** The unlock took 21
|
|
seconds and the restore 13. **But it only finished because I could open a terminal twice.** A
|
|
household would have stopped, twice, and the second time the screen would have told them it was easy.
|
|
|
|
**The two walls, both fixed-or-fixable, neither about the data:**
|
|
|
|
- **The install died four steps in** — the agent version you approved had been published as a download
|
|
but never given its version label, and the installer looks it up by that label. **Now unblocked** —
|
|
I pushed the label after checking the published file matched what you vouched. *(R-273 — closed. The
|
|
two guards that would stop it recurring are still owed.)*
|
|
- **A reinstalled machine cannot re-attach its own data drive.** Every route is a dead end, and the
|
|
restore page cheerfully says „**Ez két kattintás**" while pointing at an empty list. The drive is
|
|
fine and the machine can see it — it just is not offered, because the same drive is also the backup
|
|
target. I got past it by typing an internal path no customer could know. **This is the one to fix
|
|
before the tester's visit.** *(R-280)*
|
|
|
|
**Also broken, found on the way:**
|
|
|
|
- **Taking Felhom off a machine leaves the one thing that stops it going back on.** We install a small
|
|
network service at setup; removing Felhom restarts it without its settings, it seizes the port the
|
|
next install needs, and the next install then refuses — appearing to blame the owner's network.
|
|
*(R-272)*
|
|
- **A machine we removed keeps its private line to us open.** *(R-276)*
|
|
- **The hub said nothing at all** while a machine was wiped, rebuilt, re-claimed and had its sealed
|
|
backups opened. No false alarm — but also no word, and the alarm that exists for "someone is opening
|
|
this customer's backups" stayed silent through a real one. *(R-281)*
|
|
- **A rebuilt machine may still come back on software from last week** — narrower than I first wrote:
|
|
the resumed install fetched the right version, but a fresh one takes whatever copy is newest on the
|
|
disk without checking it against what you approved. *(R-274)*
|
|
- **demo-felhom has not had an off-site backup in six days** and is waiting for a recovery code nobody
|
|
has entered. demo-hp, rebuilt the same day, recovered by itself. *(R-278)*
|
|
- **One code, three different names**, and the email points at a page the machine is not showing —
|
|
this cost us a wasted code today. *(R-282, R-283)*
|
|
|
|
## What's broken
|
|
|
|
- **Nothing else new is broken.** The *check* against a fourth secret-in-a-page covers 4 pages of 27, and
|
|
the cheap one covering all of them is blind to the shape that shipped. *(R-255)*
|
|
- **An already-paired box is still told to pair itself**, 25 minutes on. *(R-214, R-235)*
|
|
- **A backup that covered nothing still calls itself „Sikeres".** *(R-240)*
|
|
- **A machine waiting for its recovery code can stop backing up off-site without alarming us.** *(R-243)*
|
|
- **The card offering to reopen set-aside backups promises more than we can deliver.** *(R-202)*
|
|
- **Deleting a customer leaves rows behind** while reporting a clean teardown — no secrets, but it
|
|
accumulates. *(R-244)*
|
|
- **Putting restored files back where they belong is still manual.** *(R-213)*
|
|
|
|
## The rest of what the rehearsal found
|
|
|
|
Sixteen findings in one afternoon, **none of them visible from reading the code** — three sessions of
|
|
review had not seen any.
|
|
|
|
- Removing Felhom leaves five files holding old keys *(R-275)*, and rotating a leaked key does not
|
|
revoke the old one until the service restarts *(R-269)* — the written recipe for it is a step short
|
|
*(R-270)*, and the alarm it raises can never be closed because the fix it recommends is what
|
|
silences the all-clear *(R-271)*.
|
|
- **It caught me being wrong twice, and that matters more than the count.** I told you the fleet's
|
|
off-site backups were down; demo-hp had eighteen snapshots and I had read three misleading screens
|
|
instead of asking the machine *(R-277)*. And I raised a leftover permissions file as a security
|
|
hole, then tested it and refuted myself — it is inert.
|
|
|
|
**What worked, and should not be lost in the count:** the machine came up on its own at the approved
|
|
version; the setup page appeared unprompted, in Hungarian, naming the customer; **the recovery screen
|
|
appeared without being looked for** and said plainly that unlocking changes nothing; the restore told
|
|
the truth about putting files in a checking folder rather than back in place; and no false alarm fired.
|
|
|
|
Full account: `documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`.
|
|
|
|
## Three rulings, written down so they stop living in a conversation
|
|
|
|
- **The managed-update floor.** It is deliberately parked, and the trigger to raise it is **the first
|
|
machine that is not ours**; after that it moves with the publish train. Worth knowing alongside it:
|
|
the updater always aims at the floor, never at the newest, so a machine at or above the floor
|
|
updates to nothing. **Correction to the number that was going round: the floor is live at 0.200.0,
|
|
not 0.156.0** — checked twice today, on the hub page and in both machines' own logs.
|
|
- **R-264 is decided.** Build a reader for guest-network health, the staged-update pair, the
|
|
restore-test depth pair, and the two backup-integrity timestamps. Record a stated "no reader
|
|
wanted" for the repaired-recently flag, the tier-applied timestamp, the config fingerprint, the
|
|
per-stack object and the drive-migration marker. Decide the reporting-disabled flag on its own
|
|
merits — if that is a state we support, it must be visible or a staleness alarm will one day fire
|
|
on a machine that is fine. A "no" ends by changing the allowlist reason from *arguably owed* to
|
|
*deliberately not consumed* — not by ripping out an emitter, which is a two-repo change that also
|
|
breaks a shared fixture. **The implementation is its own session.** And **it is twenty facts, not
|
|
twenty-one**.
|
|
- **R-268 is closed** — the leaked key is rotated, and the rotation is proved in both directions
|
|
rather than assumed.
|
|
|
|
## Fixed 2026-08-08 — four things the machine knew and did not say
|
|
|
|
A rebuilt machine can set up its own recovery again *(R-221, agent 0.128.0 — proved on hardware)*; an
|
|
unreadable disk is no longer drawn as a healthy empty one *(R-259)*; a backup tick now answers about
|
|
*that* app *(R-258)*; our own alarm no longer points at a log that may not exist *(R-265)*. Still true
|
|
and not glossed: a failed disk reading still reaches us as "0 of 0 GB" — the quiet direction, it can
|
|
only miss a true alarm, never raise a false one *(R-266)*.
|
|
|
|
## What we're working on
|
|
|
|
- **Widening the check** so a fourth secret-in-a-page is caught by a machine. *(R-255)* · **R-264 is
|
|
now decided** (above); building the readers is a session of its own.
|
|
- **Proving the hub really keeps the old sealed key** when a machine re-seals. *(R-198)* · Still open,
|
|
none urgent: *(R-256, R-257, R-261…R-263, R-266)*
|
|
|
|
## Waiting on you
|
|
|
|
- **Nothing blocking.** The rehearsal is finished and demo-hp is back in service: agent 0.128.0,
|
|
controller 0.210.0, claimed, off-site backups unlocked and intact.
|
|
- **One decision worth taking before the tester comes:** whether to fix the drive wall *(R-280)* now.
|
|
It is the only finding that would stop his visit outright, and it is the difference between "his
|
|
data comes back" and "his data comes back if someone types a path for him."
|
|
- **Two guards are still owed** so the install cannot break the same way twice: refuse to vouch a
|
|
version whose label does not resolve, and check that a published version and its label ship
|
|
together. *(R-273's tail.)*
|
|
|
|
## DooPlex infrastructure — separate from the product
|
|
|
|
*Kept under its own heading rather than dropped: these are real asks that need you, but they concern
|
|
the machine all this is built on, not what a customer receives. Mixing them in is why the page stopped
|
|
being readable.*
|
|
|
|
- **DooPlex's own backup keeps every copy inside the same box, and is silent when it fails.** *(R-232)*
|
|
- **193 old images exist only on this machine**, ~27 GB — clutter, not space. *(R-210)*
|
|
- **The hub password needs rotating** — a diagnostic printed it into a session log; nothing suggests
|
|
anyone else saw it. *(R-132)*
|
|
- **After DooPlex next restarts**, read `/var/log/felhom-store-postboot-check.log` — the second-SSD
|
|
move has never survived a reboot; on PASS, 34 GB comes back. *(R-209a)*
|
|
- **Backup scripts on DooPlex are unversioned host state** *(R-231)*, and the instruction-file
|
|
follow-ups each need a decision rather than an edit *(R-229, R-230)*.
|
|
- **The Configuration page is fixed: 26 s → 0.14 s.** It was never hashing anything — the hashes
|
|
are already stored and simply read. It was making 42 calls one after another. Now they overlap,
|
|
connections are kept, the answer is held for a minute, and the old artifacts are gone. Worst case
|
|
is 5 s, once a minute at most. **Your instinct to prune was right and my measurement said
|
|
otherwise** — trimming to ten of each halved the slow path. *(R-267 — closed.)*
|
|
- **The access token I printed into a log yesterday is rotated**, and I checked it both ways: the old
|
|
one is refused, the new one works, and the machine's own channel is back up. *(R-268 — closed.)*
|
|
- **Our build-check alarm has one gap left.** A run that hangs is now cut off after five minutes and
|
|
the mail says how long it took — but **whether the alarm fires at all when the machinery kills a
|
|
run outright is still unverified**, and we have not claimed otherwise. *(R-265)*
|