admin
8ef92a3fa7
gates / gates (push) Successful in 7s
R-172's root cause was not tuning — the WAL/busy_timeout pragmas had never been applied, because the DSN used mattn/go-sqlite3 syntax against modernc.org/sqlite, which ignores unknown parameters without an error. Recorded that way so nobody re-reads it as "SQLite was slow". R-173 NEW: while establishing who copies hub.db for the WAL change, found pvc/hub-data labelled recurring-job-group.longhorn.io/default: disabled, with backup-daily and backup-weekly the only recurring jobs and both on the default group — so the hub database has no volume-level backup, and it holds every box's break-glass root password plus the escrow custody records. Filed, not fixed: whether the exclusion is deliberate is an operator question. The session report is REPORT-r172-hub-wal.md, not REPORT.md, per the parallel-session rule — REPORT.md belongs to the controller session that ran immediately before this one. It also records, plainly, that a 60-concurrent load test I ran OOM-killed the hub pod three times against a 256Mi limit. Not the WAL change, and not a test I should have run against a Tier-2 box; the unit tests already proved the property.
100 lines
6.6 KiB
Markdown
100 lines
6.6 KiB
Markdown
# STATUS — what works, what's broken, what's next
|
|
|
|
**Updated 2026-08-02.**
|
|
|
|
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
|
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
|
> which is technical state written for Claude Code — keep the two separate. **Maintenance:** update
|
|
> at the end of every session in which something shipped, broke, or was decided. One screen; cut
|
|
> items rather than extend it.
|
|
|
|
## What works right now
|
|
|
|
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
|
|
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
|
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
|
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
|
restore files and app data from the drive alone. Proven end to end on real hardware.
|
|
|
|
**Apps come back after a power cut.** The machine tells an app the customer switched off from one
|
|
that simply did not come back, and waits for the system to finish starting before deciding instead of
|
|
glancing once, five seconds in. Hard-reset the demo box six times in a row: everything came back every
|
|
time, and an app switched off deliberately stayed off every time.
|
|
|
|
## What's broken
|
|
|
|
**The off-site copy can be erased by the machine that made it** — the credential that writes it can
|
|
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
|
*(R-95, R-87)*
|
|
|
|
**Three apps out of fifty-three kept their data where backups never looked.** They reported healthy;
|
|
the data would vanish on the next update. Two are fixed, the third is now clear to fix because it is
|
|
installed nowhere. *(R-156)*
|
|
|
|
**Local backups get 20 GB while apps get 50 GB.** An app that outgrows the smaller space stops being
|
|
backed up locally — and the off-site copy is made from the local one, so that stops too. Nothing is
|
|
lost: the last good copy is kept intact. *(R-163)*
|
|
|
|
**When that happens, only one page says so** — no email, no alert. The page that answers "is this app
|
|
backed up?" is the one that stays silent. *(R-158)*
|
|
|
|
**The checks now have two nets, and the second one emails you.** Every repository has one command
|
|
that runs all of its checks; it runs by itself before every push and refuses a push that fails. That
|
|
one lives on the workstation and can be skipped. So the build server now runs the same checks again,
|
|
on a machine that does not care who pushed or what they typed — and **when they fail it sends you an
|
|
email**, because a red mark on a page nobody watches is not a warning. Proven with a real broken
|
|
change, not assumed. The one thing it still cannot do is *stop* the change: every change here goes
|
|
straight to the main copy with no review step, so there is no point in the road for it to stand at.
|
|
It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)*
|
|
|
|
## What we're working on
|
|
|
|
- **Now:** the last app whose data was never saved; today's decisions written down.
|
|
- **Next:** merging the small backup partition into the large one, with the drive-filling warning and
|
|
the backup-failure alert in the same step.
|
|
- **After:** rebuilding how the machine records whether an app is meant to be running.
|
|
|
|
## Waiting on you
|
|
|
|
- **How a new version reaches a machine.** Pushing the installer publishes it — half a minute later
|
|
every new machine downloads it, with no staging and no way back but another push. And publishing is
|
|
a step we remember rather than one the release performs, forgotten twice: a fix can be live here
|
|
and still not reach a new machine. Nothing is installing today, so this is the cheapest moment to
|
|
settle both. *(R-110, R-115)*
|
|
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
|
session log; nothing suggests anyone else saw it. *(R-132)*
|
|
|
|
## Changed since last update
|
|
|
|
- **2026-08-02** — The false "host offline" warning is fixed, and the cause was not what it looked
|
|
like. The hub's database was supposed to be in a mode where reading a page cannot block a machine's
|
|
status update — the code said so, but a one-word syntax difference meant the setting had **never
|
|
taken effect**, for the hub's whole life. So opening an operator page could make a machine's report
|
|
fail; two failures in a row crossed the half-hour threshold and sent you an alert about a machine
|
|
that was up and healthy. It had already done that twice that day. Now genuinely fixed and verified
|
|
live. **Also found while checking it: the hub's own database is not in any automatic backup** — it
|
|
holds every machine's emergency password and the escrow records. Filed, not yet fixed.
|
|
|
|
- **2026-08-02** — Boot recovery finished. Both halves of the power-cut problem are closed: the
|
|
machine records what the customer asked for, and it now waits for the system to finish starting
|
|
before deciding what is missing. Six hard resets in a row, everything back every time. A hole the
|
|
previous day's change had opened — the machine would have tried to start an app whose external
|
|
drive was missing — was found by reading the code, reproduced on the demo box **before** anything
|
|
was changed, and fixed the same day. No data ever reached the wrong disk, but only because of a
|
|
filesystem accident nothing was relying on deliberately, which is why it was fixed rather than
|
|
noted.
|
|
|
|
- **2026-08-02** — Fixed and then doubled: thirteen mechanical checks had built up across the four
|
|
repositories and nothing ran most of them; two were failing quietly, one since 14 July. Both fixed.
|
|
Every repository now has one command that runs all its checks, it runs automatically before every
|
|
push, and the build server re-runs it after every push and **emails you when it fails** — proven
|
|
with a real broken change. One of the two failures was the setup page claiming the installer was
|
|
version 1.19.0 while machines got 1.22.0; that number is now **gone** rather than corrected,
|
|
because the page cannot know it — the installer is fetched fresh each run, so any number there is a
|
|
guess. A check refuses to let one be added back.
|
|
- **2026-08-02** — Decided: the 20 GB backup partition goes away and shares space with app data. That
|
|
changes the disk layout, so it happens before any machine is installed outside the house.
|
|
- **2026-08-02** — Decided: only this machine and the tester's box are protected; every other box,
|
|
demo boxes included, may be broken or reinstalled freely. Two of the three apps that never saved
|
|
their data are fixed; this page created.
|