admin
c917251eeb
gates / gates (push) Successful in 7s
Operator ruled "proceed" after reading the pre-analysis; the storageReserved condition CC made the move conditional on was applied with it. The pre-analysis is retained verbatim as §11-bis so the ruling can be audited against what it was given. MOVED WITH ZERO LOSS, verified BEFORE the original was touched on four independent observables: 550,891 = 550,891 filesystem entries; 448 = 448 trusted.overlay xattrs (load-bearing — overlayfs stacking rides them, which is why rsync -X is not optional); 37,243 = 37,243 hardlinks; byte-identical meta.db sha256. Verified again after: identical image/tag/volume ID SETS, cache 2.782 GB / 38 records, ceiling still 30 GiB, pg 4 DBs / 31 tables / 175,135,767 B, redis DBSIZE 2437. End-to-end proof was a real build on the relocated store, rc=0 — not an inventory count. k3s was never at risk, and that was established BEFORE stopping anything: it runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and ~160 pods stayed up. Only the two jarr-* dev containers were affected. TRAP FOUND WHILE PROVING THE GUARD, and it is the reusable part: RequiresMountsFor on a path with NO mount unit is a SILENT NO-OP — containerd started normally against an absent-but-unmounted path. A typo'd guard buys nothing and says nothing: the built-but-never-wired shape again. The guard was therefore verified positively at the unit level (Requires= AND After=mnt-ssd_2.mount on both units), and refusal proven with a genuinely absent DEVICE via a temporary synthetic .mount unit — /mnt/ssd_2 hosts 12 live Longhorn replicas and must never be unmounted, and editing fstab on a production host risks emergency mode at boot. Result: "Job containerd.service/start failed with result 'dependency'", is-active inactive. It refused rather than creating an empty store on the root disk. storageReserved on SSD2 0 -> 80 GB; still Schedulable=True at 76.34%. NOT PROVEN, AND NOT GLOSSED (R-209a): the operator ruled DO NOT REBOOT, so the move has not survived a reboot — which is this task's own standard for validated. The MECHANISM is proven; the CONSEQUENCE (that a real boot mounts /mnt/ssd_2 before containerd starts, in this host's actual ordering) is not, and mount-ordering reasoning is exactly the class this project has been burned by. So: the pre-move tree stays MOVED ASIDE, not deleted (34.3 GB, which is why / sits at 54% and not lower), and felhom-store-postboot-check.service — oneshot, enabled, dry-run PASS at install — validates it automatically at the next boot, asserting positively that /var/lib/containerd does NOT exist. Read /var/log/felhom-store-postboot-check.log after the next boot; on PASS the rollback copy can go and ~34 GB returns. Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).
120 lines
9.0 KiB
Markdown
120 lines
9.0 KiB
Markdown
# STATUS — what works, what's broken, what's next
|
||
|
||
**Updated 2026-08-05.**
|
||
|
||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
||
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
||
> which is technical state written for Claude Code — keep the two separate. **Maintenance:** update
|
||
> at the end of every session in which something shipped, broke, or was decided. One screen; cut
|
||
> items rather than extend it.
|
||
|
||
## What works right now
|
||
|
||
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
|
||
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
||
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
||
places — the machine's drive, a second drive, and an encrypted off-site copy.
|
||
|
||
**And the whole backup promise is now proved.** On 4 August we destroyed a machine on purpose and
|
||
deleted a marked file from its disk. Using the recovery code you saved: the backup key came back
|
||
**identical, character for character**; the existing off-site store **opened** rather than starting
|
||
over; and the file was restored **byte for byte identical**. *(R-201)*
|
||
|
||
## What's broken
|
||
|
||
- **A customer still cannot do that recovery alone — but only one step is left.** Getting from "the
|
||
key is recoverable" to "the file is back" took four steps that appeared in no instructions. **Three
|
||
are fixed today** *(R-204)*: the local reset-code tool works on the first try instead of needing the
|
||
controller restarted; re-issuing the storage credential no longer falsely marks the recovery key
|
||
"stale" (which used to stop every off-site backup and invite the one act that would have destroyed
|
||
the recovered key); and the everyday restore now **says in plain Hungarian that it returned the
|
||
app's settings and database and not your documents**, and names the button that does. **The step
|
||
that remains is the first one:** a rebuilt machine cannot get a storage credential by itself,
|
||
because the one-time password was used up by its predecessor — so you still have to press
|
||
Re-issue. That is a design decision waiting on you, below. *(R-193)*
|
||
- **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key
|
||
that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one. Both demo
|
||
machines did this on 3–4 August — 51 backups (~1.2 GB) between them. The old key now survives the
|
||
recovery ceremony, and a changed key now raises an alarm the same day, but **the rebuild itself
|
||
still starts a fresh history**. *(R-193)*
|
||
- **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
|
||
says the old backups may later be restorable with the matching recovery code. That is true for
|
||
machines that re-seal from now on and **false for anything already orphaned** — and the machine
|
||
cannot tell which case it is in. We deliberately did **not** patch the sentence: a conditional
|
||
promise that can still be wrong is worse there than a vague one. *(R-202)*
|
||
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
||
also delete it. A daily snapshot is armed as a stopgap. *(R-95, R-87)*
|
||
|
||
## What shipped recently
|
||
|
||
- **2026-08-05** — **The DooPlex server's disk is out of danger: 86% full → 54%, and the storage layer
|
||
is unstuck.** The cause was leftover working data from building our own software — 157 GB of it,
|
||
growing about 5 GB a day, which nothing was allowed to delete. **148 GB came back in 86 seconds.**
|
||
The storage layer had already stopped accepting new copies of any volume onto that disk; that is
|
||
fixed the same day. A **30 GB ceiling** is now in place and was **proved to work by deliberately
|
||
overfilling it and watching it evict** — not by assuming the setting took. Two things that failed
|
||
quietly around it: the warning meant to catch exactly this **could never fire** (fixed and proved),
|
||
and the weekly cleanup is still forbidden from touching the thing that grows (next session).
|
||
*(R-205 … R-211)*
|
||
- **2026-08-05** — **The build data now lives on the second SSD, moved with nothing lost.** All 345
|
||
images, every saved volume and both development databases came through identical — checked before
|
||
the original was touched and again afterwards, and confirmed by running a real build on the moved
|
||
copy. **The server's own services never went down:** Gitea, the registry, the hub and the backup
|
||
system run on a separate system and stayed up throughout. The second SSD now also **reserves 80 GB**
|
||
for this, so the storage layer can no longer quietly claim the space and repeat what happened to the
|
||
first disk. The old copy is kept as the way back until the machine next restarts. *(R-209)*
|
||
- **2026-08-05** — **Three of the four recovery crutches removed.** The reset code works first time;
|
||
a credential re-issue no longer blocks off-site backups on a healthy machine; and the default
|
||
restore no longer quietly returns the wrong thing. *(R-204 items 1–3, R-196)*
|
||
- **2026-08-04 (night)** — **The drill PASSED**, end to end, on real hardware. *(R-201)*
|
||
- **2026-08-04** — The folder-left-out-of-the-backup problem fixed both halves: the app and its backup
|
||
look in the same directory, and a backup that misses a folder marked essential reports *incomplete*
|
||
instead of success. *(R-203)*
|
||
- **2026-08-04** — The hub now keeps the off-site backup key when a machine re-seals, instead of only
|
||
the whole-machine one, and a machine can fetch its own sealed package back. *(R-198, R-199)*
|
||
- **2026-08-04** — The daily false alarm about David is gone; a vanished permission now repairs itself
|
||
**and says it had to**; the weekly off-site backup stopped reporting failure after a successful
|
||
upload. *(R-195, R-190, R-191)*
|
||
|
||
## What we're working on
|
||
|
||
- **Next: the retention proof.** The hub keeping the old sealed key when a machine re-seals is the one
|
||
remaining link that has **never run outside a test**. Proving it needs a second deliberate wipe on
|
||
the spare demo machine, and it is its own procedure. *(R-198)*
|
||
- **Then:** the orphaned-backup deletion you asked for (below), and the off-site copy the machine can
|
||
still erase. *(R-193, R-95)*
|
||
|
||
## Waiting on you
|
||
|
||
- **One thing to read after the machine next restarts — and nothing to do until then.** You told me
|
||
not to restart DooPlex, so I did not, and the move to the second SSD has therefore never been
|
||
through a restart. It works right now and nothing was lost, but a restart is the one test that
|
||
matters for this kind of change, and it has not happened. **I made it check itself:** whenever the
|
||
machine next starts, for any reason, it writes a plain PASS or FAIL line to
|
||
`/var/log/felhom-store-postboot-check.log`. **If it says PASS, the old copy can be deleted and 34 GB
|
||
comes back.** Until then I have deliberately kept that old copy, which is the only quick way back —
|
||
it is why the disk sits at 54% rather than lower. *(R-209a)*
|
||
- **One list to rule on: 193 old images that exist only on this machine.** 131 controller versions and
|
||
62 hub versions are not in the registry, so they cannot be re-downloaded — all of them old
|
||
(controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing
|
||
was deleted.** Worth knowing before you spend time on it: they only account for about 27 GB against
|
||
199 GB now free, so this is about clutter, not space. *(R-210)*
|
||
- **The one-shot credential decision — this is now the last thing between a customer and an unaided
|
||
recovery.** A rebuilt machine has no storage credential of its own, so an operator must press
|
||
Re-issue. Everything after that point is now self-service. Deciding how a rebuilt machine should get
|
||
a credential is the remaining design question. *(R-193, R-204 item 4)*
|
||
- **The recovery screen you described has been priced, and it can be built.** A freshly installed
|
||
machine that finds a sealed package waiting should say so, offer a box for the recovery code, and
|
||
show what would come back before doing anything. One thing to weigh, deliberately not decided: that
|
||
screen is reachable by anyone with the household's dashboard password, and the preview reveals
|
||
backup dates and app names. *(R-193)*
|
||
- **The orphaned backups on the storage box — you said delete, and it is still owed.** About 1.2 GB
|
||
across the two demo machines, in set-aside stores nobody can open and nothing prunes. It wants its
|
||
own session rather than riding along with other work. *(R-193)*
|
||
- *(decided 4 Aug)* You chose **not** to keep a copy of the backup key on the Proxmox host, which
|
||
makes the customer's own recovery code the only route back from a rebuild. *(R-193)*
|
||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||
- **One small question, not urgent.** The automatic version check cannot see which version you have
|
||
told machines to install, only which ones exist. *(R-184)*
|