Files

T

gates / gates (push) Successful in 7s

Details

docs: R-172 CLOSED (hub v0.88.0), R-173 filed, session report

R-172's root cause was not tuning — the WAL/busy_timeout pragmas had never been
applied, because the DSN used mattn/go-sqlite3 syntax against modernc.org/sqlite,
which ignores unknown parameters without an error. Recorded that way so nobody
re-reads it as "SQLite was slow".

R-173 NEW: while establishing who copies hub.db for the WAL change, found
pvc/hub-data labelled recurring-job-group.longhorn.io/default: disabled, with
backup-daily and backup-weekly the only recurring jobs and both on the default
group — so the hub database has no volume-level backup, and it holds every box's
break-glass root password plus the escrow custody records. Filed, not fixed:
whether the exclusion is deliberate is an operator question.

The session report is REPORT-r172-hub-wal.md, not REPORT.md, per the
parallel-session rule — REPORT.md belongs to the controller session that ran
immediately before this one.

It also records, plainly, that a 60-concurrent load test I ran OOM-killed the hub
pod three times against a 256Mi limit. Not the WAL change, and not a test I
should have run against a Tier-2 box; the unit tests already proved the property.

2026-08-02 21:22:14 +02:00

6.6 KiB

Raw Blame History

STATUS — what works, what's broken, what's next

Updated 2026-08-02.

A view, not a source. documentation/backlog/OPEN-ITEMS.md is the authority on open work; this page restates part of it in plain words, and nothing may exist only here. Not CONTEXT.md, which is technical state written for Claude Code — keep the two separate. Maintenance: update at the end of every session in which something shipped, broke, or was decided. One screen; cut items rather than extend it.

What works right now

A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer, who sets their own password. They install apps from a catalogue of fifty-three, share files over the home network, and open apps from a launcher or a shared link. Backups run on their own to three places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can restore files and app data from the drive alone. Proven end to end on real hardware.

Apps come back after a power cut. The machine tells an app the customer switched off from one that simply did not come back, and waits for the system to finish starting before deciding instead of glancing once, five seconds in. Hard-reset the demo box six times in a row: everything came back every time, and an app switched off deliberately stayed off every time.

What's broken

The off-site copy can be erased by the machine that made it — the credential that writes it can also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy. (R-95, R-87)

Three apps out of fifty-three kept their data where backups never looked. They reported healthy; the data would vanish on the next update. Two are fixed, the third is now clear to fix because it is installed nowhere. (R-156)

Local backups get 20 GB while apps get 50 GB. An app that outgrows the smaller space stops being backed up locally — and the off-site copy is made from the local one, so that stops too. Nothing is lost: the last good copy is kept intact. (R-163)

When that happens, only one page says so — no email, no alert. The page that answers "is this app backed up?" is the one that stays silent. (R-158)

The checks now have two nets, and the second one emails you. Every repository has one command that runs all of its checks; it runs by itself before every push and refuses a push that fails. That one lives on the workstation and can be skipped. So the build server now runs the same checks again, on a machine that does not care who pushed or what they typed — and when they fail it sends you an email, because a red mark on a page nobody watches is not a warning. Proven with a real broken change, not assumed. The one thing it still cannot do is stop the change: every change here goes straight to the main copy with no review step, so there is no point in the road for it to stand at. It notices, quickly, and tells you. (R-29, R-161, R-168, R-169)

What we're working on

Now: the last app whose data was never saved; today's decisions written down.
Next: merging the small backup partition into the large one, with the drive-filling warning and the backup-failure alert in the same step.
After: rebuilding how the machine records whether an app is meant to be running.

Waiting on you

How a new version reaches a machine. Pushing the installer publishes it — half a minute later every new machine downloads it, with no staging and no way back but another push. And publishing is a step we remember rather than one the release performs, forgotten twice: a fix can be live here and still not reach a new machine. Nothing is installing today, so this is the cheapest moment to settle both. (R-110, R-115)
A job, not a decision: the hub password needs changing. A diagnostic command printed it into a session log; nothing suggests anyone else saw it. (R-132)

Changed since last update

2026-08-02 — The false "host offline" warning is fixed, and the cause was not what it looked like. The hub's database was supposed to be in a mode where reading a page cannot block a machine's status update — the code said so, but a one-word syntax difference meant the setting had never taken effect, for the hub's whole life. So opening an operator page could make a machine's report fail; two failures in a row crossed the half-hour threshold and sent you an alert about a machine that was up and healthy. It had already done that twice that day. Now genuinely fixed and verified live. Also found while checking it: the hub's own database is not in any automatic backup — it holds every machine's emergency password and the escrow records. Filed, not yet fixed.
2026-08-02 — Boot recovery finished. Both halves of the power-cut problem are closed: the machine records what the customer asked for, and it now waits for the system to finish starting before deciding what is missing. Six hard resets in a row, everything back every time. A hole the previous day's change had opened — the machine would have tried to start an app whose external drive was missing — was found by reading the code, reproduced on the demo box before anything was changed, and fixed the same day. No data ever reached the wrong disk, but only because of a filesystem accident nothing was relying on deliberately, which is why it was fixed rather than noted.
2026-08-02 — Fixed and then doubled: thirteen mechanical checks had built up across the four repositories and nothing ran most of them; two were failing quietly, one since 14 July. Both fixed. Every repository now has one command that runs all its checks, it runs automatically before every push, and the build server re-runs it after every push and emails you when it fails — proven with a real broken change. One of the two failures was the setup page claiming the installer was version 1.19.0 while machines got 1.22.0; that number is now gone rather than corrected, because the page cannot know it — the installer is fetched fresh each run, so any number there is a guess. A check refuses to let one be added back.
2026-08-02 — Decided: the 20 GB backup partition goes away and shares space with app data. That changes the disk layout, so it happens before any machine is installed outside the house.
2026-08-02 — Decided: only this machine and the tester's box are protected; every other box, demo boxes included, may be broken or reinstalled freely. Two of the three apps that never saved their data are fixed; this page created.

6.6 KiB Raw Blame History