Compare commits
216 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 877e89ee2a | |||
| ff1db11db4 | |||
| 6b820143f8 | |||
| 242f0a15d5 | |||
| 9291ae255a | |||
| 21986d03b5 | |||
| 208d21d18f | |||
| c9052264f7 | |||
| 0c55336fba | |||
| 245886ee92 | |||
| 30cfdc1dc5 | |||
| 3de64b138d | |||
| b1b5c7e7e1 | |||
| 175ecfcdd2 | |||
| bee277ffad | |||
| 5efe6daec3 | |||
| f93738dc49 | |||
| 31bdb4b549 | |||
| 0e970ba384 | |||
| a73f728a55 | |||
| 771e12445c | |||
| fd1f98547e | |||
| 693f3b45aa | |||
| 88b0a2e761 | |||
| 0ed2e8de4c | |||
| e78726314e | |||
| e55b2ceb63 | |||
| eaae94a4b8 | |||
| 0e22a0ecc7 | |||
| 44992360a4 | |||
| 66b7540b7a | |||
| a23ae9e3dc | |||
| c5f91174f6 | |||
| 6ed79cd2e9 | |||
| 1b74ddc0c9 | |||
| 85e3e05e10 | |||
| 55f7621c90 | |||
| 3885640f66 | |||
| d07a1a904c | |||
| 7a0d0c027d | |||
| 5eee18cf16 | |||
| d3c50b50f6 | |||
| 697c2a7b10 | |||
| 710a2505f9 | |||
| 2344589a5e | |||
| 207ad19746 | |||
| cdfcc47b15 | |||
| f417cdede1 | |||
| 5188dbdb44 | |||
| 9268d9933b | |||
| f4c5466c39 | |||
| 33bbf8e94a | |||
| 9f77865c2a | |||
| 71b8c8c62b | |||
| 9e2786c907 | |||
| 93052884b2 | |||
| 5ed44ada6e | |||
| 2633dc1654 | |||
| e6d1ebd152 | |||
| 68c22907e5 | |||
| 344081f064 | |||
| e215351fe2 | |||
| cba2badade | |||
| 27f4e7d976 | |||
| 09ae93db05 | |||
| b50289074d | |||
| 7c50dba454 | |||
| 88810ad09f | |||
| bc9c7b4934 | |||
| 8e31b45006 | |||
| ce2a10e995 | |||
| 0d4600741a | |||
| 40f07429ff | |||
| 2d69c61556 | |||
| b63654a299 | |||
| 11a673d597 | |||
| 8dab40c7a7 | |||
| 92a60c62bd | |||
| 939553c82f | |||
| a6a9f0b245 | |||
| daacf84e30 | |||
| b2f6c1626c | |||
| c12fd46f54 | |||
| 8699dba53d | |||
| 915b2292c7 | |||
| 2076bf9b36 | |||
| 3159892fab | |||
| 1de0f7c294 | |||
| 42bf40bd2c | |||
| 2866f6a318 | |||
| 25cb3eb9c1 | |||
| 2de56ce16b | |||
| 11591f3a93 | |||
| 7e50098088 | |||
| 80aeac71f6 | |||
| 27641ce9e7 | |||
| 4951fc2283 | |||
| cdf974d186 | |||
| be8f79e2f9 | |||
| 178be6de51 | |||
| 78126575cb | |||
| 1ec1819abc | |||
| 2bdda1161e | |||
| 8cadacb553 | |||
| 5e97c44401 | |||
| 1749e73e50 | |||
| fd9c77b92e | |||
| e649d46fea | |||
| b0f8ffb801 | |||
| 14ae67f0b8 | |||
| 2b1cb246cf | |||
| 58226e9abb | |||
| adb8d904ca | |||
| 56d6de9808 | |||
| aedaab8944 | |||
| e76c3af418 | |||
| 2cc7495b58 | |||
| 2af6896348 | |||
| e129476750 | |||
| b9af5b3913 | |||
| 648f200cdf | |||
| 5340aec56b | |||
| a8f790c0c6 | |||
| 03df4c8220 | |||
| 4c31536909 | |||
| 06744dbacc | |||
| b52cb6ec94 | |||
| 7696836f84 | |||
| bff98a81a2 | |||
| af6db38514 | |||
| b280bb5d59 | |||
| 82eef7b7cf | |||
| 6576f28cc0 | |||
| 305f79c40f | |||
| 90f1f26d20 | |||
| 302f24e47c | |||
| f12f8609fc | |||
| 9141817867 | |||
| ccd915ff34 | |||
| 3386041e62 | |||
| 7195a6342e | |||
| eb1c56a981 | |||
| 6b2176e480 | |||
| 593312daf4 | |||
| a7971c4822 | |||
| 75ff26408a | |||
| d53bd08442 | |||
| 54bff69f88 | |||
| 068e06537a | |||
| f5e11277f6 | |||
| 2d24931597 | |||
| 1a72fef87f | |||
| e924a17632 | |||
| f2234c0fdd | |||
| 83e436f609 | |||
| 4502af6bb1 | |||
| 3208cb2083 | |||
| 6ebe95c13e | |||
| 1cedf94d6b | |||
| b33a33130e | |||
| f853f43b67 | |||
| 99c0709cbe | |||
| 86c4b9a0d8 | |||
| 500488cad6 | |||
| 68bb1ea394 | |||
| e4d45a8f72 | |||
| 3e58c184f6 | |||
| 77335625fe | |||
| cf8dec8480 | |||
| 11e37ee807 | |||
| 3898580fa8 | |||
| 0030cbd1de | |||
| e443841b75 | |||
| 899ae18b95 | |||
| 210394ec6e | |||
| 3add9fa678 | |||
| 511969928d | |||
| 2644f9c5e7 | |||
| 21f17ed32b | |||
| 7caa24ffe1 | |||
| d3b284863e | |||
| 05ea21e918 | |||
| 5a349d9884 | |||
| 4c92beab8f | |||
| 805ad1e962 | |||
| 267dcad01b | |||
| 22439b0e43 | |||
| 1de6aaf904 | |||
| 737694c603 | |||
| 8efd2d00df | |||
| 186546d562 | |||
| a975cfde5b | |||
| 462ab4a5ff | |||
| d27663dd91 | |||
| 1ee14ce166 | |||
| 8d786f7940 | |||
| 9c69b3ff07 | |||
| da20722e76 | |||
| c85262111c | |||
| d19f07ea04 | |||
| 0c263c77f2 | |||
| bcdd5b2058 | |||
| 11aaeaee8a | |||
| fcdc948909 | |||
| 83b558b3ea | |||
| e02bc03819 | |||
| a499327236 | |||
| 732e9b9e0c | |||
| 75bc699f78 | |||
| 91f047dfc0 | |||
| cf09c78743 | |||
| 8e9401c6bf | |||
| f538a03bd5 | |||
| 31eeb36e88 | |||
| 8d539f971c | |||
| 5a654ebc9a |
@@ -20,6 +20,8 @@ repo. Sibling repos point here; this is where the pointed-at thing must actually
|
||||
| logging levels and phrasing | `runbooks/logging-conventions.md` |
|
||||
| a spike or campaign result | `audits/` |
|
||||
| every open finding | `backlog/OPEN-ITEMS.md` |
|
||||
| every finished finding, compressed | `backlog/CLOSED-ITEMS.md` |
|
||||
| finished notes and history moved out of a register | `archive/` |
|
||||
| the operator's one-screen view | root `STATUS.md` |
|
||||
|
||||
**Do not restate a fact that has a home** — point at it. Re-check an address rather than trusting one
|
||||
|
||||
@@ -20,7 +20,7 @@ unconditional: true
|
||||
**Not yours, ever, without a task file or an operator word:** money; anything that changes risk to
|
||||
customer data; anything that changes a promise the product makes to a customer; anything that
|
||||
reverses a documented design decision (`documentation/architecture/` — a design decision is not a
|
||||
defect, R-370); anything on DooPlex, Peti's box or ep0; baking or vouching a golden; promoting a
|
||||
defect, R-370); anything on DooPlex or ep0; baking or vouching a golden; promoting a
|
||||
catalog version; a new external dependency.
|
||||
|
||||
## 2. When you may decide instead of ask (operator grant, 2026-09-14)
|
||||
|
||||
@@ -131,7 +131,10 @@ something, not only sessions that touch `documentation/` — which is why it is
|
||||
- **`CHANGELOG.md` + `REPORT.md`** in every repo touched (see the workspace root for the rule, and
|
||||
the parallel-session caveat above).
|
||||
- **`REUSE.md`**, if a shared helper or pattern moved (same commit).
|
||||
- **`OPEN-ITEMS.md`** — every finding, with a number.
|
||||
- **`OPEN-ITEMS.md`** — every finding, with a number. **A row you close moves to `CLOSED-ITEMS.md` in the
|
||||
same commit** — `closed_register_gate.py` RULE 3 refuses a finished row left in the open register. A
|
||||
**new row** goes into its category's section with one **Category** and one **Sev** (P1–P4); the scale
|
||||
and the eleven names head `OPEN-ITEMS.md`, and `register_shape_gate.py` refuses anything else.
|
||||
- **Root `STATUS.md`** — at the end of every session in which something shipped, broke or was
|
||||
decided. It is a **view** of `OPEN-ITEMS.md`; nothing may exist only there. One screen, written for
|
||||
the operator in plain language, and deliberately **not** `CONTEXT.md`.
|
||||
@@ -161,8 +164,18 @@ something, not only sessions that touch `documentation/` — which is why it is
|
||||
|
||||
```bash
|
||||
T=$(curl -s -u "$U:$P" ".../actions/jobs?limit=1" | python3 -c 'import json,sys;print(json.load(sys.stdin)["total_count"])')
|
||||
curl -s -u "$U:$P" ".../actions/jobs?limit=50&page=$(( T/50 + 1 ))" # then grep your own head_sha
|
||||
for pg in $(seq 1 $(( (T+49)/50 ))); do curl -s -u "$U:$P" ".../actions/jobs?limit=50&page=$pg"; done
|
||||
# then match your own head_sha across ALL of them
|
||||
```
|
||||
|
||||
**Two things this recipe got wrong until 2026-09-20, both measured the hard way:**
|
||||
1. **The response key is `jobs`, not `workflow_runs`.** A parser reading `workflow_runs` gets an
|
||||
empty list and prints nothing — which reads exactly like "no CI run for this commit" and is not.
|
||||
That is R-417's own shape (a field the API never populates) committed while checking R-417.
|
||||
2. **Job ids are NOT ordered within a page**, so `page = total/50 + 1` does not hold the newest
|
||||
rows — a run can sit several pages earlier. **Scan every page** and match on `head_sha`; with a
|
||||
few hundred jobs that is a handful of requests. Guessing the page produced three consecutive
|
||||
false "no CI job for this commit" readings in one session.
|
||||
|
||||
An unchecked green is an assumption, not an observation — and so is a green read off a field the
|
||||
API never populates.
|
||||
|
||||
+601
@@ -14,6 +14,607 @@
|
||||
> language, one screen, no identifiers in the prose. Same subjects, different readers; merging them
|
||||
> would make one of the two audiences stop reading. `STATUS.md` is also a **view of `OPEN-ITEMS.md`**
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-04 (~18:49) — rulings 96–99 (the R-840 / Tester 2 brief), recorded before the work.** `09` §3: **96** build
|
||||
> R-840 now (option A, a signed config bundle; replaces 82); **97** Tester 2 may receive only the signed agent update,
|
||||
> the bundle by the route and the one-time live-restore reload (~19:05: the operator will install the bootstrap file by
|
||||
> hand through his tunnel — the route cannot reach a root side that predates it); **98** drill-r50 removed from the hub,
|
||||
> ep0 touch allowed (~19:05); tester-1 stays; **99** the operator's cause with the reviewer's correction.
|
||||
|
||||
> **2026-10-04 (~18:00) — R-858 incident + ruling 95.** The operator saw demo-felhom DOWN: the 14:13 UTC Docker step
|
||||
> left `felhom-controller` and `traefik` on the OLD socket (live-restore kept them running; their bind-mounted socket
|
||||
> file was recreated). Repaired by restarting the two (15:57 UTC). `09` §3 decision **95**: the wrapper restarts the
|
||||
> socket users after a step + the health rule checks the controller reaches Docker (agent v0.142.1). Demo ring-0 Docker
|
||||
> marks OFF until released.
|
||||
|
||||
> **2026-10-04 (evening) — the System page, the Docker engine slow lane and the crash guard BUILT.** Agent v0.142.0,
|
||||
> hub v0.132.0, installer 1.30.0, `build-golden.sh` 3.1.0. **Decided by CC unattended — operator may reverse:** `09` §3
|
||||
> decisions **90** (crash signal = a clean-shutdown marker; pstore saved nothing), **91** (`panic_on_oops` stays 0; an
|
||||
> oops is reported), **92** (the 3rd unclean stop in 60 min leaves the box off; 10 s / 24 h), **93** (the wrapper
|
||||
> verifies Docker authority against root-owned files), **94** (page colours = alarm thresholds). Live: live-restore on by
|
||||
> reload with the same ids (9202 6/6, demo-hp 24/24, demo-felhom 5/5); Docker 29.7 → 29.8.2 on both demo boxes, ids kept;
|
||||
> the operator's button approved `os-docker-20261004-142842` (TEST 0-night wait, reverted); a signed undo to 29.7.2 on
|
||||
> demo-hp and back; demo-felhom as ring 1 by a signed job; a replay refused; crash 1–2 restarted in 54/53 s, the guard
|
||||
> tripped, crash 3 stayed off, the hub mailed the trip, re-armed. `REPORT-os-docker-crash-2026-10-04.md`.
|
||||
|
||||
> **Rulings 2026-10-04 (~15:17) — recorded before the work (System page + Docker lane + crash restart brief).** `09` §3
|
||||
> decisions **87** (Docker `live-restore` ON fleet-wide, never off by a plain restart), **88** (a crashed host restarts by
|
||||
> itself, with a limit — R-851) and **89** (the operator sees the boxes' OS / Proxmox versions in the hub — R-852).
|
||||
|
||||
> **2026-10-04 (late afternoon) — OS updates: host fast lane + fleet view + alarms BUILT (`11` §8.2–§8.3); the tunnel
|
||||
> status is true (R-841).** Agent v0.141.0 → v0.141.1, hub v0.131.0 → v0.131.1, controller v0.292.0 (cloudflared
|
||||
> readiness health check). **Decided by CC unattended — operator may reverse:** `09` §3 decisions **84** (appliance proof
|
||||
> = the root-owned install record), **85** (alarm numbers 7/14/7/14 days, configuration), **86** (a same-session patch
|
||||
> release of agent and hub for the host "reboot needed" defect, R-846). Live: `tunnel_down`/`tunnel_recovered` on
|
||||
> demo-hp; ring 0 host pass on demo-felhom (108 Debian packages); a 605-package host release; ring 1 exact install;
|
||||
> by-hand host undo proved; leg 23–32 s with nothing to install. Kernel spike (R-836, narrowed): GRUB's one-shot is not
|
||||
> a one-shot on LVM `/boot`; Secure Boot fine; `kernel.panic = 0` (R-851); `sp5100_tco` answers. demo-hp now runs and
|
||||
> defaults to kernel 7.0.14-20. Docker slow lane designed (`11` §5.8). Golden 0.292.0 baked + vouched with agent 0.141.1 (min_agent
|
||||
> 0.131.0); the golden waiver deleted (not needed). `REPORT-os-host-lane-2026-10-04.md`.
|
||||
|
||||
> **Rulings 2026-10-04 (~12:20) — recorded before the work (host fast lane brief).** `09` §3 decisions **81** (R-842 A:
|
||||
> the undo is the whole-guest backup by hand; R-842 closed), **82** (R-840 not built now — "There are no older boxes";
|
||||
> row kept open with the reviewer's note) and **83** (next: R-841, the host fast lane, OS-update improvements).
|
||||
|
||||
> **2026-10-04 (afternoon) — OS updates, guest fast lane BUILT (`11` §8.1).** Agent v0.140.0: `configs/felhom-os-apply`
|
||||
> (R1–R13, repair first, snapshot.debian.org fallback; FELHOM_OSAPPLY sudoers), `internal/osupdate` (the leg after a
|
||||
> successful primary backup, under the heavy-op gate; health baseline = start of the leg), `--selftest=os-update`.
|
||||
> Hub v0.130.0: `internal/osupdates` (rings, switch default ON, candidate = what ALL ring-0 boxes run, approval 24 h +
|
||||
> 1 night via `OS_APPROVE_AFTER` / `OS_APPROVE_NIGHTS`, `os_update` desired block, `POST /hosts/{id}/os-report`,
|
||||
> operator `/os/*`). Controller v0.291.0: decision 78 (R-726), infra pins raised (R-838) + `scripts/check-infra-pins.py`.
|
||||
> Installer 1.29.0 installs the wrapper. **No automatic undo** (R-837 measured → R-842). **Existing boxes need the
|
||||
> wrapper + sudoers by hand** (R-840 — the demo boxes got them by hand). `REPORT-os-guest-lane-2026-10-04.md`.
|
||||
|
||||
> **Rulings 2026-10-04 (~10:17) — recorded before the work (guest fast lane build).** `09` §3 decisions **78** (R-726
|
||||
> option A: a returning household's box sets the old off-site copy aside on night one), **79** (`snapshot.debian.org`
|
||||
> fallback for a replaced approved version — option A; the reviewer had picked B) and **80** (next build: the guest
|
||||
> fast lane, R-837 measured first, R-838 in the same session).
|
||||
|
||||
> **2026-10-04 (day) — off-site closed; OS-update spike done.** Hub v0.129.0 (R-833: operator raises ONE window's cap,
|
||||
> `POST /offsite/window-grant/<id>` `max_remove=<n>`, 1..500, consumed once, window row records its cap). Agent v0.139.0
|
||||
> (R-834: DR bring-up refuses beside a live original; restore-test measured safe), `scripts/felhom-restore-beside.sh`
|
||||
> for the hand route, `07` Lane 2 rule. R-95 dated check 2026-10-12. R-726: options in STATUS (pick A). OS spike:
|
||||
> `11` corrected (C1–C12, §5.4.1 wrapper draft, §7.1 answers); rows R-835..R-839; demo-hp host Debian-updated, next
|
||||
> boot kernel 7.0.14-20-pve. `REPORT-backup-close-os-spike-2026-10-04.md`.
|
||||
|
||||
> **Rulings 2026-10-04 (day) — recorded before the work (off-site close + OS update spike).** `09` §3 decisions **75**
|
||||
> (the off-site topic closes first, one brief), **76** (the OS fast lane follows an approved list with a 1–2 day wait;
|
||||
> `unattended-upgrades` with no wait rejected) and **77** (`architecture/11-os-updates.md` added unchanged, NOT RATIFIED,
|
||||
> then corrected by measurement).
|
||||
|
||||
> **Rulings 2026-10-04 (morning) — recorded before the work.** `09` §3 decisions **71** (ep0's DooPlex copy keeps 8 weekly copies; a third place later, R-832), **72** (tester-1's unpinned keys removed via the registrar), **73** (the transcript-exposed Hetzner storage token is not rotated now; R-831), and **74** (the set-aside deletion is the hub's, after a 7-day wait — from the brief's Part E).
|
||||
|
||||
> **Rulings 2026-10-03 (afternoon) — recorded before the work (off-site lock build).** `09` §3 decisions **68** (the box prunes only inside a weekly hub-opened window; option 2 rejected; no box-side retention in the interim), **69** (the hub is the key registrar; the box never receives the sub-account password; the hub stores it encrypted with a key outside the database; daily `authorized_keys` check) and **70** (nightly PBS pull-sync of ep0's `felhom-offsite` to DooPlex; the fence opens for that brief's Part F acts only).
|
||||
|
||||
> **Operator request 2026-10-03 — recorded before the work (backlog triage).** (1) Four roadmap items: box OS security
|
||||
> updates (R-808, finding R-812), legal pages and business papers (R-809, finding R-813), independence — "if the household
|
||||
> leaves Felhom, or Felhom stops" (R-810, spike), a second login step for the dashboard (R-811). (2) Finished rows leave
|
||||
> `OPEN-ITEMS.md` for `CLOSED-ITEMS.md` **in the same commit that closes them**, and a gate refuses a finished row left in
|
||||
> the open register. (3) Every open row carries ONE category and ONE severity, and new rows are filed into their category.
|
||||
> (4) `ROADMAP.md` is cleaned. (5) The operator then decides what is worked on next. **Reviewer's defaults, operator may
|
||||
> reverse:** the severity scale (P1 now · P2 before the first paying customer · P3 during the first customers · P4 later)
|
||||
> and the eleven categories, both written at the top of `OPEN-ITEMS.md`.
|
||||
|
||||
> **Rules carried out of rows closed 2026-10-03 (`PROMPT-TEMPLATE.md` §N.7 item 2).** The triage moved 125 finished
|
||||
> rows to `CLOSED-ITEMS.md`; these sentences stated a rule and had no other durable home. Verbatim, with their row.
|
||||
> Decisions: **[DESIGN]** R-245 — *"An automatic ending should trigger on the harm, with a dated warning, never on a
|
||||
> date alone."* R-303 — *"the wrong fix (suppressing the orphan card during a countdown) would hide a real second
|
||||
> fault."* R-312 — *"retention is an operator-only capability, and nothing anywhere may promise the customer can
|
||||
> perform it themselves"* (all three: `07` §11, "Decided, and deliberately not built"). Method and traps: R-201 —
|
||||
> the drill's pass condition is *"a byte-identical sentinel sha256, not 'the store opened'"*, and *"a good snapshot
|
||||
> is not durable against a later bad run on the same day."* R-209 — *"`RequiresMountsFor` on a path with NO mount
|
||||
> unit is a SILENT NO-OP"*; *"`-X` is load-bearing — overlayfs stacking rides `trusted.overlay.*`."* R-520 —
|
||||
> *"`pct mount` maps the guest's ROOTFS ONLY and does not apply the guest's own internal mounts"*; *"An empty
|
||||
> directory is not evidence of an absent file."* R-582 — *"a guard ported between languages must be re-derived from
|
||||
> what the claim IS in the new language, not translated word for word"*. R-583 — *"the surface you would use to
|
||||
> CHECK a feature is the one most worth checking first — a broken instrument that reports success is worse than a
|
||||
> broken feature."* R-601 — *"a 'no access' claim must list what was tried; it does not say the list makes the claim
|
||||
> true"*; *"Six failed routes to a stale address are six failures of one assumption, not six pieces of evidence."*
|
||||
> R-611 — *"the report's FIRST section is 'not done', even when empty"*, and every part of a brief skipped,
|
||||
> shortened or changed is listed there with the reason. R-623 — *"an instrument that can report a success as a
|
||||
> timeout is not a measurement"*. R-633 — *"A refusal from the wrong rule is not evidence for the new one."* R-715 —
|
||||
> a catalog gate refuses a probe without a before/after measurement in its comment. R-736 — *"the Remove button runs
|
||||
> `RemoveStack`, not `DeleteStack`; `docker image ls` without `-a` hides the untagged digest-pulled images"*. R-745 —
|
||||
> *"the agent rolls back to the RUNNING image (what `/etc/felhom-controller-image` named), never to a previous one
|
||||
> the controller hands it"* (`09` decision 56's wording may say otherwise — R-817). R-752 — a settings-only change
|
||||
> reaches the stack file at the next sync (≤15 min) and the running app only at the next `compose up -d`
|
||||
> (Restart/Start, an Update, or a backup's restart).
|
||||
|
||||
> **2026-10-02 (afternoon) — the persistence re-sweep (R-801/R-788 CLOSED), controller v0.288.0 (R-800), golden 0.288.0.**
|
||||
> Rulings `09` §3 66 (licences: Tandoor like SparkyFitness; Emby/Plex/n8n kept; lawyer review before the first paying
|
||||
> customer, R-802) and 67 (userdata stays on every remove; dialog + result name it — `07` §6.5). The gate now sends
|
||||
> requests (port from the label NAME), calls the app's own fixture seed when a volume stays empty, and reads an empty
|
||||
> declared volume as UNDETERMINED. All 58: CLEAN 39 · UNDETERMINED 19 · BROKEN 0 (no data-loss finding). papra's
|
||||
> 256M limit OOM-killed every fresh install — 768M now (R-803 closed). Open follow-ups R-804..R-807. Report:
|
||||
> `REPORT-persistence-sweep-2026-10-02.md`.
|
||||
|
||||
> **2026-10-02 — the family gate BUILT (R-780 CLOSED), controller v0.287.0 floored, golden 0.287.0 vouched; Grimmory and
|
||||
> MeTube published behind it (R-767 CLOSED, R-775 narrowed).** `internal/family` + `stacks/family_gate.go` +
|
||||
> `web/family_gate.go`; template fields `family_gate` / `family_gate_except` (literal, anchored `^/p(/|$)`) /
|
||||
> `min_controller`; catalog gate `check-family-gate.py` (refuses a family app while the newest baked golden < 0.287.0 —
|
||||
> why this golden was baked off-cadence). `01` §5 carries the design paragraph. Licence read of all 58 templates
|
||||
> (R-787 CLOSED): `documentation/audits/licences-2026-10-02/TABLE.md`; operator rows R-789..R-795; SparkyFitness request
|
||||
> drafted, not sent (R-784, decision 65). Register 422 → 436 (R-801: the volume-persistence gate never exercised an app — fixed, re-sweep owed; R-800: remove-with-data keeps userdata, operator). Report: `REPORT-family-gate-2026-10-02.md`.
|
||||
|
||||
> **Rulings 2026-10-02 (morning, operator) — `09` §3 decisions 64, 65, recorded before the work.** **64 (R-780):** the
|
||||
> family gate — GO, both sessions. **65 (R-784 B):** SparkyFitness stays offered; the operator asks the author; no written
|
||||
> permission before the first paying customer → `lifecycle: hidden`. The licence read (R-787) runs this session.
|
||||
|
||||
> **2026-10-01 (late night) — the box tells visitors apart (R-753 CLOSED), controller v0.286.1 floored, golden 0.286.1
|
||||
> vouched.** cloudflared alone on `felhom-tunnel` at `172.16.253.2` (traefik `.3`; allocation confined to `.4/30`); traefik
|
||||
> trusts forwarded headers from that address only and strips client-writable host/path/address headers
|
||||
> (`felhom-forwarded@file`); the controller reads the hop traefik saw (`clientaddr.go`). **Readers take the visitor from the
|
||||
> RIGHT** — 19 catalog apps that read the leftmost carry `<router>-xff`; BookStack `APP_PROXIES`; checklist 3.10. R-772
|
||||
> (not-checked probe) and R-773 (lock after a restore) closed. Permanent family gate SPIKED and PASSED, not built (R-780,
|
||||
> operator go/no-go, 2 sessions). SparkyFitness record added; its licence is non-commercial only (R-784, operator); a catalog
|
||||
> licence sweep is owed (R-787). `01` §7 corrected: cloudflared runs in the guest (R-754). Report:
|
||||
> `REPORT-visitors-2026-10-01.md`.
|
||||
|
||||
> **Ruling 2026-10-01 (evening, operator) — `09` §3 decision 63, recorded before the work.** Login problems are solved by
|
||||
> EXTENDING THE BOX'S OWN GATE (option A), not by an identity app on every box (option B, Authentik/Authelia — a possible
|
||||
> later step behind the same traefik `forwardAuth` hook). Order: the box tells visitors apart (R-753) → a SPIKE of a
|
||||
> permanent gate with family accounts (Grimmory, MeTube; measured, not built) → a build only after the operator's go.
|
||||
|
||||
> **2026-10-01 (night) — the first new apps through the checklist.** Published with complete records and first ladder
|
||||
> steps: Radicale (catalog `195129c`; login file written once — R-765), Karakeep (`882ac14`; web 1536M on measurements),
|
||||
> Dawarich (`72247a3`; seeded login replaced behind the install hold; SECRET_KEY_BASE data_key). Grimmory tested on both
|
||||
> venues and HELD: a stranger's 5 wrong sign-ins lock every visitor out for 15 min (R-775, operator A/B). Stopped on the fit
|
||||
> check: Grimoire (R-768), MeTube (R-767), Pinchflat (R-769); Invidious (R-770) and moonlight-web (R-771) fit only, both
|
||||
> recommended no. Box findings: R-772 (a probe that could not run records healthy), R-773 (sign-up block lost on remove +
|
||||
> restore). New-app assets reach boxes only with a hub release (R-766). Report: `REPORT-new-apps-2026-10-01.md`.
|
||||
|
||||
> **Operator decisions 2026-10-01 (evening), recorded before the work:** wger HIDDEN (`lifecycle: hidden`) until R-762 and
|
||||
> R-763 are fixed (the fix session un-hides it). The new-app list: MeTube, Pinchflat, moonlight-web, Grimoire, Grimmory,
|
||||
> Invidious, Karakeep, Dawarich, Radicale (SparkyFitness and Calibre-Web Automated were also named — both already in).
|
||||
> Reviewer's order and limits (operator may reverse): Radicale, Karakeep, Dawarich, Grimmory, Grimoire, MeTube, Pinchflat;
|
||||
> Invidious and moonlight-web fit check only. Fit table: `documentation/audits/new-apps-2026-10-01/FIT.md`.
|
||||
|
||||
> **2026-10-01 (evening) — the new-app checklist: in the catalog, a gate, piloted on wger.** Operator request (recorded
|
||||
> before the work): *before new apps are added, a checklist every new app passes — storage and database needs, whether an
|
||||
> admin can be created and its password changed, health checks tested, resource limits tested, and more — so a new app is
|
||||
> mapped and tested before it is offered.* **Reviewer's defaults (operator may reverse):** the gate binds NEW apps only;
|
||||
> the 53 existing apps get a read-only gap page, not a re-test. Built: `app-catalog-felhom.eu/NEW-APP-CHECKLIST.md` (60
|
||||
> rows, 10 groups, a `since` date per id); `onboarding/_TEMPLATE.md` (one line per id: done + evidence / n/a + reason /
|
||||
> open); gate `onboarding` (`scripts/check-onboarding.py`, `--fast`, hook + CI; the 53 exempt BY NAME, not by commit —
|
||||
> CI has no history; evidence in a sibling repo is checked where the sibling exists and listed as NOT CHECKED where it
|
||||
> does not); 16 decoys + 5 gate mutants seen red; `onboarding/EXISTING-APPS-GAPS.md` (`scripts/onboarding_gaps.py`).
|
||||
> Pilot: the draft caught R-752 and R-755 as written, missed R-737 (its "how" logged in on the web form only) and R-738
|
||||
> for a new app (no update to run at the newest tag) — rows 1.4/1.6 sharpened, 1.7/3.9 added. The pilot's rows then found
|
||||
> R-762 (wger serves no CSS/JS and no photos), R-763 (strangers sign up / guest accounts), R-764 (no mail); also R-758
|
||||
> (8 `mem_limit` ≠ sum), R-760, R-761, R-759 (wger's open record rows). Evidence `documentation/audits/new-app-checklist-2026-10-01/`.
|
||||
> Report: `REPORT-new-app-checklist-2026-10-01.md`.
|
||||
|
||||
> **2026-10-01 (late afternoon) — 61 and 62 built.** calibre-web `ADMIN_USER` (catalog `e9f50b5`; after_install renames
|
||||
> `admin` in app.db and proves it); demo-hp renamed by hand (name in the operator's credentials file). Registry prune rule in
|
||||
> `admin/misc-scripts` `c9d5ed5` (keep 20 + in use; refuses when unreadable; dry-run only). R-750, R-752 closed; R-756
|
||||
> (9202 remove-with-data 409), R-757 (InjectMissingFields invents a secret for installed apps) opened.
|
||||
> Report: `REPORT-calibre-name-and-prune-2026-10-01.md`.
|
||||
|
||||
> **Rulings 2026-10-01 (late afternoon, operator) — `09` §3 decisions 61, 62, recorded before the work.** **61 (R-752 A):**
|
||||
> calibre-web's admin login name is generated at install and shown on the app page; the lock stays. **62 (R-750 A):** the
|
||||
> registry prune keeps the newest 20 per image plus every version the vouched golden, the floor, `agent_version` and
|
||||
> `min_agent` name (and the golden's controller image); never scheduled; `--apply` is a person's act.
|
||||
|
||||
> **2026-10-01 (afternoon) — strangers and lockouts (R-752), the one address behind the tunnel (R-753), the registry (R-750).**
|
||||
> Decisions 58–60 (decided by CC unattended — operator may reverse): wger `AXES_LOCKOUT_PARAMETERS=username`,
|
||||
> `AXES_COOLOFF_TIME=5`, database handler (catalog `82fff32`); BookStack and Grafana unchanged (1 and 5 min, measured).
|
||||
> calibre-web's day-long lock waits for the operator. Behind the tunnel every visitor reaches an app as cloudflared's
|
||||
> container address; traefik drops forged `X-Forwarded-For`; `CF-Connecting-IP` is forgeable from the LAN — no box-wide
|
||||
> fix (no controller release). Registry: no Gitea cleanup rule; a manual `gitea-image-prune.sh --keep 7` run (homelab
|
||||
> HM-024, 2026-08-22/23) removed the old versions; nothing schedules it. Report: `REPORT-lockouts-2026-10-01.md`.
|
||||
|
||||
> **Operator note 2026-10-01 (afternoon):** decision 57 kept — mealie stays on the 1-hour lock; no secret login name.
|
||||
|
||||
> **2026-10-01 — the three rulings built; decision 57 decided by CC unattended (operator may reverse).** 54/55: the first
|
||||
> full monthly re-test — nextcloud `3b59dfb` and sonarr `1a37032` re-tested on both venues and written (~17 min per app);
|
||||
> `retest-floating.py` fixed for a fresh bench (R-749). 56: controller v0.285.0 (keeps running + previous controller image,
|
||||
> previous from the swap record); R-751 nil-stack crash fixed; floor 0.285.0; demo-hp 84 → 2 and N100 76 → 2 controller
|
||||
> images; golden 0.285.0 baked + vouched (gate OK). **57 (R-747):** mealie `SECURITY_USER_LOCKOUT_TIME=1` (catalog
|
||||
> `a4597cd`) — measured 120 min lock on 9202. R-746 digest resolver `804884a`; R-744 outline fixture `9fc7052`.
|
||||
> Opened R-749..R-752 (R-752: calibre-web-automated, wger, Grafana, BookStack lockable by strangers — read, not measured).
|
||||
> Report: `REPORT-rulings-2026-10-01.md`.
|
||||
|
||||
> **Rulings 2026-10-01 (operator) — `09` §3 decisions 54–56, recorded before the work.** **54 (R-743 1A):** the monthly
|
||||
> re-test is run by a CC session the operator starts with the standing brief `claude/MONTHLY-security-retest.md` (project);
|
||||
> STATUS carries "Monthly security re-test: last run <date>, next due <date>". **55 (R-743 2A):** it covers every app with a
|
||||
> proven ladder; `--engines-only` is a switch, not the default. **56 (R-745 3A):** a box keeps the controller image it runs
|
||||
> and the one before it; older controller images are deleted by decision 53's in-use rule; registry tags never.
|
||||
|
||||
> **2026-09-30 (late evening) — both rulings built.** Decision 52: catalog `6a3ead9` (re-test entries `from` == `to` +
|
||||
> `digest_from` + `box_evidence`, gated with decoys; `upgrade-test.py --retest`; the monthly command
|
||||
> `scripts/retest-floating.py`; runbook `runbooks/monthly-floating-retest.md`; proven end to end on 9202 through the leg).
|
||||
> Decision 53 + R-741: controller v0.284.2 (image retention with a one-time sweep; the install hold), floor 0.284.2,
|
||||
> golden 0.284.2 baked + vouched (gate OK). wger JWT pair `45d8482`. wanderer runs on the bench (override), no step yet.
|
||||
> Rows R-743..R-747 opened; R-736/737/740/741 closed. Report: `REPORT-night-rulings-2026-09-30.md`.
|
||||
|
||||
> **Rulings 2026-09-30 (late evening, operator) — `09` §3 decisions 52 and 53, recorded before the work.** **52 (R-740 A):**
|
||||
> a box takes a same-tag security fix at night only after the catalog re-tested that tag at the new digest on both venues
|
||||
> and wrote it as a ladder step; database and redis lines first; one command, run monthly. **53 (R-736 A):** a box keeps,
|
||||
> per app service, the running image and the one before it (the undo's); older images of the app are deleted; never an
|
||||
> image any container or installed compose names; removing an app deletes its images under the same rule; kept data
|
||||
> (decision 40) untouched.
|
||||
|
||||
> **2026-09-30 (evening) — more night-updatable apps; the same-name fix gap measured (R-740, operator decision).**
|
||||
> Catalog: first ladders for calibre-web, gitea (its installer form, R-624), wger, crafty-controller, uptime-kuma, zipline
|
||||
> (4.6.1 → 4.7.0 → 4.8.0 — 4.8.0 refuses a database that skipped 4.7.x, R-742; the box UNDID the direct jump by itself);
|
||||
> within-major steps emby, ghost, home-assistant, outline, rallly. Night (a) 30 + 2 → 34 + 3 (calibre-web conditional:
|
||||
> its library `metadata.db` changes). wger ran no migrations on update → login 500 behind a `done` Update; fixed in the
|
||||
> template (`DJANGO_PERFORM_MIGRATIONS=True`, R-738 closed). immich's step 0b82… re-proven at 768M and rewritten by the
|
||||
> new `upgrade-test.py --restep` (Part D). Harness: 5 fixtures, HTTPS backends on the bench, per-file names behind
|
||||
> `files_may_change`, bench `password:N:special` (R-735 closed). **Part C (R-740):** the night leg skips a digest-only
|
||||
> change (`unattended.go:433–435`), AND the catalog never records a same-tag re-test (writer refuses "nothing moves"), so a
|
||||
> same-name upstream fix reaches no box; a unit walk shows the leg WOULD take a same-tag re-test entry with no controller
|
||||
> change. `09` decision 30 carries a dated note. New rows R-736 (images never deleted → disk fills), R-737 (wger app-login
|
||||
> 500, no JWT key), R-739 (bench cannot run wanderer), R-741 (after_install's default-login window, measured 1–18 s).
|
||||
> Evidence: `documentation/audits/more-night-apps-2026-09-30/`. Report: `REPORT-more-night-apps-2026-09-30.md`.
|
||||
|
||||
> **2026-09-30 (late afternoon) — immich's first start (R-732 CLOSED).** Cause measured: the first-start geodata import
|
||||
> runs up to 9 concurrent 5000-row INSERTs; the database needs ~400 MB anon + ~170 MB touched shared_buffers (the image's
|
||||
> fixed 512MB) — over a 512M limit without swap (bench: 61–104 kills), survivable only by swapping (9202). An update that
|
||||
> ships a new geodata file re-runs the import. Catalog `56c4888`: immich v3.2.4 + `immich-postgres` 768M (`mem_limit`
|
||||
> 4480M), proven with swap off on both venues. Also: the golden needs no bake (0.283.1 = newest release; STATUS corrected);
|
||||
> the ISO build refuses a dirty/unpushed tree (R-730 closed); gitea 28.0.0 is GA, mariadb 13.0 a short-term line (R-731).
|
||||
> New rows R-733 (bench no swap vs boxes 512 MiB), R-734 (immich's `.immich` markers mark `files_may_change`). Report:
|
||||
> `REPORT-immich-first-start-2026-09-30.md`.
|
||||
|
||||
> **2026-09-30 (day) — operator notes, and the last six PostgreSQL apps.** Notes recorded first (`09` §3): **the brief
|
||||
> ran by day** — every backup and automatic-update test started by hand (the night chain's debug action; demo-hp's
|
||||
> window moved for the test and put back), nothing waited for a night; **Tester-2's pre-checks** (the Cloudflare tunnel
|
||||
> route, the two connect mails) were confirmed done by the operator. Decision 52 (seed through the database) was offered
|
||||
> by the reviewer and **NOT used** — all three hard apps have a front-door seed (outline `installation.create`, rallly
|
||||
> sign-up with its own e-mail code read from its table, sparkyfitness better-auth), so it is not recorded. **Outcome:**
|
||||
> catalog `25ffd89` rallly, `aeb0cd6` outline, `1666572` sparkyfitness (15) → PostgreSQL 18, each bench + box 9202 + one
|
||||
> undo case; zipline, adventurelog, immich STAY (upstream runs the same or an older major) — **R-463 CLOSED, 8 of 11
|
||||
> moved**. Part F (both venues, one commit each): bookstack 26.09.1 `54ca055`, kimai 2.67.0 `d75d1fd` (the plain tag = the
|
||||
> Apache image by digest), audiobookshelf 2.37.1, n8n 2.42.1, navidrome 0.64.2, grafana 13.2.3, komga 1.28.0; immich v3.2.4
|
||||
> NOT moved — its first start at today's pin OOM-killed its database on the bench (R-732). Rows: R-730, R-731, R-732.
|
||||
> **R-687 item 4 proven live on demo-hp** (two deferrals, the backup after the leg, success). Currency audit:
|
||||
> `audits/catalog-currency-2026-09-30.md` (31 apps can update at night after the session). No controller/agent/hub
|
||||
> release. Report: `REPORT-pg-last-six-2026-09-30.md`.
|
||||
|
||||
> **2026-09-30 — operator rulings 50 (apps go off-site by themselves, A) and 51 (ep0 drill archives removed; the
|
||||
> restore test takes only the box's own).** `09` §3. 50: when the customer has off-site, every newly installed app is
|
||||
> in the off-site copy; over quota the page names the largest and the household chooses; history is never deleted to
|
||||
> make room without that choice. 51: the three drill archives in `tester-1`'s ep0 namespace go, nothing else. Also: the
|
||||
> first real tester is a NEW record `Tester-2`; the volunteer guide is rewritten as measured (R-722). **Outcome (same day):** controller v0.283.0/v0.283.1 (apps off-site by default + one press + size card; a Stop holds
|
||||
> at the quiesce resume, the volume dump, the update leg and the crash recovery — v0.283.0 missed the dump's production
|
||||
> adapter, caught live), agent v0.138.0 (the restore test skips an archive written with another key; signed delivery to
|
||||
> both demo boxes), hub v0.126.0 („Új linket kérek" on an expired/used bind link; no day-one false mails), ep0's three
|
||||
> drill archives removed. Floor 0.283.1. Report: `REPORT-fixes-first-tester-2026-09-30.md`.
|
||||
|
||||
> **2026-09-29 evening — operator rulings 48 (wanderer stays, A) and 49 ("close sign-up now", A).** Controller
|
||||
> **v0.282.0**: `after_setup:` (the app's own sign-up switch, env merged + one `compose up -d`, when the gate opens or on
|
||||
> the press; the window lifts it, the loop restores it), `POST /apps/<slug>/close-signup` (lock record `opened_by:
|
||||
> close-signup`, never a gate), probes read list indexes + `done_status:`. Floor 0.282.0. Catalog `6446197`: 9 own switches
|
||||
> (`SIGNUP_CLOSED`/`SIGNUP_OPEN`, compose default open), every `signup_block` `PathRegexp((?i)…)`, wanderer closable
|
||||
> (decision 48 outcome), ghost/HA/gramps probes, new gate `check-probe-measured.py`. Pressed on demo-hp adventurelog +
|
||||
> opengist, demo-felhom opengist. Open: R-717, R-718. Report: `REPORT-signup-lock-2026-09-29.md`.
|
||||
|
||||
> **2026-09-29 afternoon — operator ruling decision 47 (R-711 option A) + one CC-unattended decision (operator may
|
||||
> reverse):** sign-up is closed by a box-side block of the app's own sign-up address once the gate opens, with a
|
||||
> household 15-minute window (`internal/stacks/signup_block.go`, `.felhom.yml` `signup_block:` + `app_info.add_people`)
|
||||
> — per-app switches were measured unusable. Controller **v0.281.0** (the press asks the probe first; sign-up block;
|
||||
> R-713 `${NAME|base64}`), floor 0.281.0. Catalog `6faf432`: 32 of 34 class-4 apps gated (9 by probe), 11 sign-up
|
||||
> blocks, claper base64. Not gated: wanderer (R-714). Open: R-715 (probe shapes), R-716 (installed apps keep open
|
||||
> sign-up — operator). Report: `REPORT-gate-rollout-2026-09-29.md`.
|
||||
|
||||
> **2026-09-29 — operator rulings:** (1) CC changed the admin passwords of demo-hp's installed bookstack and calibre-web
|
||||
> (stored in the operator's credentials file as `DEMO_HP_BOOKSTACK_*` / `DEMO_HP_CALIBRE_*`, values never in a repo);
|
||||
> (2) decision 46 — the setup gate is SPIKED first. **The spike PASSED** (`audits/login-gate-2026-09-29/B/B-VERDICT.md`);
|
||||
> controller **v0.280.0** built it (`internal/stacks/setup_gate.go`, `internal/web/setup_gate.go`: forwardAuth →
|
||||
> `/__felhom_gate/auth`, handshake on `felhom.<domain>/__gate/start`, host-only gate cookie, key
|
||||
> `<data>/setup-gate.key`; the gate file is written BEFORE the first start; open = record, then remove the file) plus
|
||||
> R-710 (absent after_install record = not-run-yet for 30 min only; "I changed it"), R-709 (password fields off the
|
||||
> page), `generate: password:N:special`. Catalog `d0e7e2e`: gate on immich/n8n/audiobookshelf/uptime-kuma; mealie/wger/
|
||||
> calibre-web after_install (argv, never in code); romm/zipline notes; grafana R-708; wger R-712. Floor 0.280.0; both
|
||||
> demo boxes on it. R-707: 30 class-4 apps left. Report: `REPORT-login-gate-2026-09-29.md`.
|
||||
|
||||
> **2026-09-28 evening — operator rulings (decisions 44, 45):** 44 — demo-hp's restore test uses `nvme-scratch` (the
|
||||
> agent's storage role was granted there with the operator's word; proven). 45 — no app is published with a login a
|
||||
> stranger knows: generated first password via `after_install:` where the app's own CLI can set it, else the page
|
||||
> says the default. Controller v0.279.0 (after_install, known-login page rule, Part D empty-backup alarm, night chain,
|
||||
> R-706). claper + bookstack fixed; 37 apps remain (R-707, several sessions, operator's choice). Report:
|
||||
> `REPORT-logins-nvme-2026-09-28.md`.
|
||||
|
||||
> **2026-09-28 — decided by CC unattended (operator may reverse): `09` §3 decision 43 — claper → PostgreSQL 17, calcom → 18** (by
|
||||
> each upstream; decision 42's rule; calcom's memory 768M → 1536M first, R-703 closed). Also that day: golden 0.276.0 baked + vouched with agent 0.137.0; controller
|
||||
> v0.277.0 (kept data loads from the off-site copy, R-691 (2)); floor 0.277.0; R-702 (claper default admin, P1) and
|
||||
> R-703 (calcom OOM at 768 MB, closed by `9555e73`) filed. **Afternoon (operator: finish today):** controller v0.278.0
|
||||
> (R-704 — a hold from an earlier install no longer holds a new one; floor 0.278.0); R-691 (2) proven LIVE on demo-hp
|
||||
> (full off-site restore + "use my kept data" from the off-site copy); night 27/28 read instead of a new night; R-705,
|
||||
> R-706 filed. Report: `REPORT-golden-kept-offsite-2026-09-28.md`.
|
||||
|
||||
## 2026-09-27 evening — a restore and a drive move keep the app's records (controller v0.276.0)
|
||||
|
||||
- **Controller v0.276.0** (floor 0.276.0, MinAgent 0.131.0): R-697 closed (the restore's write carries the life records; `earlier_conversion_copies`); R-700 filed + fixed (a drive move dropped the pin → the next start took the catalog's newest version) — live proof open, no Tier-0 guest has two drives. R-691 (2) deliberately not built (no off-site target on a box CC may touch). Rule added to `07` §6.6. Evidence `audits/records-carried-2026-09-27/`.
|
||||
|
||||
## 2026-09-26/27 — a backup's data and its version travel together (controller v0.275.0), agent v0.135.0, catalog f1a7d6c
|
||||
|
||||
- **Rulings recorded (`09` §3):** 40 (D3 A — the box never deletes kept data), 41 (adventurelog keeps its world-data
|
||||
download). **D4 open, option A built and in force** (STATUS): a restore brings the app back at the version its data
|
||||
belongs to; the guarded update climbs from there. Rule: `07` §6.6.
|
||||
- **Controller v0.275.0** (floor 0.275.0, MinAgent 0.131.0): data stamps (`data-stamps.json`, manifest `data`), the unit
|
||||
keeps its data's definition, `ErrUnitVersionMismatch`, the off-site restore writes the snapshot's definition, data
|
||||
times on all tiers (`settings.offsite_data_at` for Tier 3), the conversion-copy release needs a dump on the new major,
|
||||
R-699 (a definition-only unit is never a precondition copy), R-695, R-691 (1), R-694 (`restored_logins`,
|
||||
`loginAppliedEveryStart` register).
|
||||
- **Decisions taken by CC unattended, operator may reverse:** `09` §3 decision 42 (paperless → 18, tandoor → 17);
|
||||
`07` §6.5 (the kept view joins the folder's owning group).
|
||||
- **Catalog:** paperless-ngx → PostgreSQL 18 (`46affe0`), tandoor → 17 (`53b84ca`), adventurelog v0.13.0 with its
|
||||
health and world-data fixes (`06ea7da`). **Agent v0.135.0** (R-689) released, signed-delivered to both demo hosts, NOT
|
||||
vouched.
|
||||
- **Open from this session:** R-697 (a restore drops `conversion_copy`), R-698 (images not stored), R-691 (2).
|
||||
- Record: `documentation/audits/DRILL-version-travel-2026-09-26.md`.
|
||||
|
||||
## 2026-09-25 (evening) — PostgreSQL conversion (controller v0.273.0), kept data (v0.274.0), hub v0.125.0
|
||||
|
||||
- **Controller v0.273.0** = `09` §6.4 part 10 for docmost: `stacks/pgconvert.go`, only on a ladder step marked
|
||||
`engine_conversion`; unmarked PostgreSQL major moves refused (preflight + job). **v0.274.0** = decision 36 (kept data:
|
||||
install choice, `/kept-data`, read-only file-browser source, `<drive>/kept` protected and never backed up) + R-690 +
|
||||
R-692. Floor 0.274.0 (MinAgent 0.131.0). 9202 runs 0.274.0 on the DRILL catalog (reset to live at teardown).
|
||||
- **Catalog:** harness v4; the engine gate's proof clause; **docmost `postgres:16` → `18` moved** (`afd3a60`, mount
|
||||
`/var/lib/postgresql`, docmost limit 512M); nextcloud `after_load`.
|
||||
- **Decisions taken by CC unattended, operator may reverse (`09` §3):** 37 (target 18, not 17), 38 (`pg_dumpall` from
|
||||
the old engine, zero tolerated load errors), 39 (docmost 384M → 512M).
|
||||
- **Hub v0.125.0:** the customer delete lists the Cloudflare items to remove by hand (R-688 half).
|
||||
- **Night 26:** demo-hp's automatic leg converted docmost 16 → 18 (54 s, content equal). Found R-696: Tier 1's time is the
|
||||
unit manifest's refresh, not its dump — the kept 16 copy was released on a 16 dump.
|
||||
- Record: `documentation/audits/DRILL-night-2026-09-26.md`.
|
||||
|
||||
## Rulings 2026-09-25 (evening, operator)
|
||||
|
||||
- **`09` §3 decision 35 — part 10 is built now, docmost first** (D1 option A). The box converts a PostgreSQL
|
||||
major as a guarded-update step; each other app needs its own two-venue proof; the engine-major gate stays for
|
||||
every app without it.
|
||||
- **`09` §3 decision 36 — a reinstall over kept data offers "use my kept data" / "start fresh"** (D2 option A),
|
||||
and kept data is never a dead end: visible (read only), loadable, deletable by the household.
|
||||
- **D3 is OPEN** (in `STATUS.md`): does the box ever delete kept data by itself? Until ruled, nothing does.
|
||||
|
||||
## 2026-09-25 (midday) — Peti retired in fact; controller v0.272.0
|
||||
|
||||
- Peti's hub customer deleted through the cascade (journal #20); Storage Box sub-account 269130 (`u629488-sub2`)
|
||||
deleted — it held one 81-byte `authorized_keys`, never a repo; ep0 held nothing of it. Record:
|
||||
`documentation/audits/RETIRE-peti-2026-09-25.md`. Cloudflare side not covered by the cascade: R-688.
|
||||
- Controller **v0.272.0**, floor 0.272.0 (MinAgent 0.131.0): backup-page „nem fér el" sentence (R-685 page half),
|
||||
R-671, R-670, R-677 — three proven live on 9202; the page sentence render-tested only (no box is short of space).
|
||||
|
||||
## Rulings 2026-09-25 (operator)
|
||||
|
||||
- **Peti's box (`peti-felhom`, host `peti-felhom-86d37d`) is RETIRED.** It will not return — the tester wiped his
|
||||
server. Its off-site backup holds no user data (operator's words; measured: all 482 of its reports show 0 off-site
|
||||
snapshots, 0 bytes) and is deleted. It leaves the protected list; DooPlex and ep0 stay Tier 2. Record:
|
||||
`documentation/audits/RETIRE-peti-2026-09-25.md`.
|
||||
- **A controller restart during the night's update leg does not resume the leg** (R-686, option B) — `09` §3
|
||||
decision 34. Nothing is built.
|
||||
|
||||
## 2026-09-25 (night shift) — agents 0.133.0 + 0.134.0 delivered, demo-hp backs up, automatic updates shipped (controller v0.271.0)
|
||||
|
||||
- **Agent v0.133.0 and v0.134.0 delivered to both demo boxes** by CC-signed `agent_update` jobs (ruling 1); restore
|
||||
test back ON. v0.134.0 = R-685 agent half (backup space preflight; free space from `GET /nodes/<n>/storage` —
|
||||
`GET /storage` has no usage; a build that read it started a real vzdump on demo-hp in its live test, aborted).
|
||||
- **demo-hp: operator option A applied** — `local_backup_retention: 1`, two old archives pruned by PVE's prune, one
|
||||
product-triggered backup fitted (8.18 GB). R-684 closed.
|
||||
- **Controller v0.271.0 = `09` §6.4 part 7**, floor 0.271.0 (MinAgent 0.131.0). Decisions taken by CC unattended,
|
||||
operator may reverse (`09` §3): **31** after W+5h the full-system gate waits only for a step already in flight, cap
|
||||
W+5h30m; **32** the controller's self-update waits for the whole leg; **33** one step per app per night (the brief's
|
||||
own words). The demo boxes' first real night: demo-felhom opengist 1.13 → 1.15 done in 20 s, demo-hp nothing to do;
|
||||
both summaries in the hub's stored reports (`update_leg`, allowlisted in `wire_contract_gate.py` until a hub surface
|
||||
exists).
|
||||
- **Catalog:** n8n 2.41.2 and mealie v3.28.0 moved (bench + box proven).
|
||||
- **Open for the operator:** Peti's box before it returns (STATUS); R-686 (resume the leg after a restart?).
|
||||
Record: `documentation/audits/DRILL-night-2026-09-25.md`.
|
||||
|
||||
## 2026-09-24 (evening) — the restore test off on the demo hosts, 9201 repaired, agent v0.133.0 + hub v0.124.0 (R-672, R-673)
|
||||
|
||||
**Operator rulings (recorded in `03` §8).** (1) The scheduled restore-test is OFF on both demo hosts until an
|
||||
agent with the R-672 fix is delivered there; then it goes back on. Done with
|
||||
`backup.restore_test_eval_interval_seconds: -1` — **0 would NOT disable it (0 = the 6 h default)**; saved configs
|
||||
`/etc/felhom-agent/agent.json.pre-r672`. (2) This session could stop, check and start 9201 on demo-hp — done.
|
||||
|
||||
**Shipped.** Agent **v0.133.0** released (tag `9bdb4da`, package verified by download), **not delivered** — the
|
||||
operator signs `agent_update` per box (R-530). Hub **v0.124.0** live (thin pool critical at 90 %, per pool per 6 h).
|
||||
Evidence: `audits/r672-2026-09-24/`. R-684 opened (demo-hp's root backup storage too small). Part D (controller
|
||||
v0.270.0) — see REPORT-r672-2026-09-24.md.
|
||||
|
||||
## 2026-09-24 (night shift, run in daytime) — the second drive brings a file app back whole; the box stops a crash loop; digests on the box; the update leg spiked
|
||||
|
||||
**Operator rulings recorded (`09` §3 decisions 26–28).** 26 — build one action that brings a file app back WHOLE
|
||||
from the second drive. 27 — while a held app's page says support is informed, Remove offers only "keep my data".
|
||||
28 — a crash loop or an OOM storm is STOPPED by the box; household + operator told; Start = one more try.
|
||||
|
||||
**Decisions taken by CC unattended — operator may reverse (`09` §3 decisions 29, 30).** 29 — the crash loop is
|
||||
≥ 6 restarts in 10 min from `RestartCount` (the brief's 10/10 misses a steady loop: Docker caps it at ~1/min).
|
||||
30 — an INSTALLED app keeps its image digest until a guarded Update moves it (`stacks.CarryDigests`); shipped as a
|
||||
SECOND controller release (v0.269.1) instead of taking v0.269.0's sync behaviour to the fleet.
|
||||
|
||||
**Shipped.** Controller **v0.269.0** (`3c6b49b`): `backup.RestoreTier2Whole` (R-661), fail-closed `sameDevice`
|
||||
(R-668), keep-data-only remove (R-666), `stacks.ObserveUnhealthy` + `unhealthy_stop` hold (R-667), step
|
||||
`.felhom.yml` for probe/memory/applied (R-664/R-665), R-662 removed, digests rendered + badge from the tested digest
|
||||
(§6.4 part 6 box half). Controller **v0.269.1** (`5b1b191`): `CarryDigests`. Hub **v0.123.0**:
|
||||
`app_stopped_unhealthy` (household + operator). Catalog **`cf7cf84`**: step `.felhom.yml` files, gate rule 4b.
|
||||
**Floor 0.269.1** (MinAgent 0.131.0): N100 arrived; **demo-hp 9201 did not** (R-672).
|
||||
|
||||
**Spiked, not built.** `09` §6.4 part 7 is now a measured build brief (§6.4.2): chain the update leg from the
|
||||
off-site job on EVERY exit path (no reliable "finished" signal exists — `offbox.go:863-897` early returns write
|
||||
nothing), gate interlock beside `WindowStartFn`, per-box switch default ON; gaps R-678/R-679/R-680.
|
||||
|
||||
**Found.** R-672 (P1, agent): the scheduled restore-test filled demo-hp's thin pool; 9201's disks are read-only
|
||||
— **operator: stop + fsck + start 9201**. R-669 (a hold + restore leaves the failed step's probe as applied;
|
||||
proven to cause a false hold), R-670, R-671, R-673–R-677, R-681 (a lost install is silent), R-682, R-683.
|
||||
**Not done:** Part D (no bench — the session could not destroy one). Evidence: `audits/DRILL-night-2026-09-24.md`.
|
||||
|
||||
## 2026-09-24 (morning) — the fleet on 0.267.0 then 0.268.0; the undo after a restore; option A; the ladder on the box
|
||||
|
||||
**Operator rulings (`09` §3 decisions 24–25).** 24 — the fleet takes 0.267.0 although the chaos hour's stop rule
|
||||
fired (both faults predate it). 25 — R-659 option A: a held app's page names only a copy that brings the app back
|
||||
WHOLE; with none, it says so + support is informed + `app_hold_no_whole_copy` (critical, operator-only); option B
|
||||
(a database-only restore under existing files) is NOT built.
|
||||
|
||||
**Shipped.** Controller **v0.268.0** (`206b035`, MinAgent 0.131.0): R-658 (undo selects volumes from the compose
|
||||
definition; the restore labels what it creates; the remove counts unlabelled volumes), R-659 (`backup.WholeOnTier`
|
||||
= the restores' own refusals; `HoldAfterFailedUpdateWhole`), R-660 (fourth suppression, `UpdateHeldStacks`), R-651,
|
||||
and `09` §6.4 **part 5** (one press = one tested step; `stacks.StepKey` = catalog `ladder.step_key`). Hub
|
||||
**v0.122.0** (`app_hold_no_whole_copy` allow-listed + operator-only + per-app cooldown). Catalog **`5ed599c`**:
|
||||
`steps/<key>.yml` for every intermediate step (8 backfilled from the NEWEST commit naming the step's images),
|
||||
gate rule 4 with decoys, the writer keeps the superseded step, R-653/R-656 in the bench, two suites un-drifted (R-663).
|
||||
Floors: 0.267.0 at 05:12Z, **0.268.0 at 06:47Z**, both with declared MinAgent 0.131.0; both demo boxes arrived
|
||||
healthy within ~15 s each time. Evidence and the claims the brief got wrong: `documentation/audits/ladder-2026-09-24/README.md`.
|
||||
|
||||
**Measured, and now the design:** for an app with declared drive files, neither the own-unit nor the second-drive
|
||||
unit restore brings it back (R-538's guard is in the shared function) — only off-site does (`07` §6 table; R-661 is
|
||||
the second-drive gap). The box's `--depth 1` clone carries a template's `steps/` folder (whole tree, one commit).
|
||||
|
||||
## 2026-09-24 (night shift 2026-09-23) — tests off DooPlex's Docker, the test record, 12 published steps, the chaos hour
|
||||
|
||||
**Operator word, `09` §3 decision 21:** tonight the catalog may move every app whose within-a-major edge is
|
||||
proven on both venues, each with its test record; nothing else moves. **Decided by CC unattended — operator may
|
||||
reverse** (both recorded in `09` §3): **22** — the memory watch's `memory_tight` reads the app's own memory
|
||||
(`anon`, sampled), the cgroup peak kept beside it (the file cache read 100 % with 0 kills); **23** — the move
|
||||
gate asks the registry, for moved refs only (INCONCLUSIVE when it cannot → the push waits).
|
||||
- controller **v0.267.0** (`80e6ad8c4772`): `internal/dockerexec` (R-650), `CheckDumpComplete` before any
|
||||
restore mutation (R-640, proven live), R-499's four-branch sentence, R-518's measured stop. **Deployed to 9202
|
||||
only; the floor stays 0.266.0** — the brief raises it only if every Part A/B/D proof passed, and Part D found
|
||||
R-658/R-659 (both pre-existing). Operator question in STATUS.
|
||||
- catalog: `update_ladder:` (one JSON entry per line; `scripts/ladder.py`), `check-test-record.py` (static, CI)
|
||||
+ `check-test-record-move.py` (history + registry), `upgrade-test.py --write-ladder` the only writer, the
|
||||
21-move backfill; harness v3 (box fixtures on the bench via `upgrade_boxport.py`, anon memory,
|
||||
`files_may_change`). 12 steps published across 11 apps (`audits/night-2026-09-23/C-moves.txt`); wishlist 512M
|
||||
(R-612) and uptime-kuma's DB wizard skipped (R-613).
|
||||
- **Found:** R-658 (P1 — a restore drops the compose label, the undo then copies nothing), R-659 (P1 — a held
|
||||
file-leg app is pointed at a restore that refuses it), R-655 (adventurelog v0.13.0), R-657 (nextcloud reinstall
|
||||
over kept data), R-651–R-654, R-656, R-660. Closed R-650, R-640, R-499, R-626. Register 330 → 336.
|
||||
- Record: `documentation/audits/DRILL-night-2026-09-23.md`.
|
||||
|
||||
## Ruling 2026-09-23 (night) — a failed install removes what it started (R-649, controller v0.266.0)
|
||||
|
||||
Operator: "On failed install, the controller should clean up the containers." `runComposeDeploy`'s failure
|
||||
branch runs `compose down` WITHOUT `-v` (named volumes kept — a reinstall after „keep my data" finds it)
|
||||
before recording not-deployed; a failed `down` is logged and Remove clears the rest. „Failed" now means
|
||||
nothing runs. Proven live on 9202 (`audits/r649-2026-09-23/`). Floor 0.266.0.
|
||||
|
||||
## 2026-09-23 (late evening) — clean-up: R-634's cause, held apps, the OOM storm (controller v0.265.0, hub v0.121.0); floor 0.265.0
|
||||
|
||||
**R-634 mechanism (reproduced on demand on 9202):** the in-memory `Deployed` flag is true from the moment a
|
||||
deploy is ACCEPTED (`stacks/deploy.go:395`), and `stackAdapter.ListDeployedStacks` filtered on it alone, so
|
||||
every backup leg saw a DEPLOYING app; the volume leg's `DumpAppVolumesSafe` ran `compose down` and a second
|
||||
`compose up -d` in the middle of the deploy's own `up`; both failed and `runComposeDeploy`'s failure branch
|
||||
wrote `Deployed=false` over whatever containers the race left. **Fix:** deploying apps leave the list; the
|
||||
volume leg re-asks (`deployingReporter`, optional interface — no fake churn) right before the stop and SKIPs;
|
||||
`StopStack`/`StartStack` return `ErrStackDeploying` for every caller. **Open:** R-649 (operator question: a
|
||||
deploy that fails on its own and leaves containers).
|
||||
**R-625:** held → badge `badge.update.held` (`tag-error`), no Update button (the list already hid it).
|
||||
**R-636:** `ScanOOMKilled` reads `memory.events` `oom_kill` / `memory.max` / `memory.peak` by `docker exec
|
||||
cat` for FLAGGED containers only (the controller's own cgroupns cannot see others); the notifier keeps 30 min
|
||||
of samples per `container|startedAt` and sends `app_oom_storm` (error) once at ≥20; hub: allow-list +
|
||||
`operatorOnlyEvents` + `perAppCooldownEvents`. `08` §6.2 has the rung.
|
||||
**R-647:** web `readerFuncs(lang, prev)` apply to EVERY language (holdText / updateErrorText / held badge
|
||||
title) — same bytes on a Hungarian box; held error key `stacks.UpdateErrorKeyHeld`; API uses
|
||||
`Manager.UpdateErrorFor` / `HoldReasonFor` (guards' optional `HoldForLang`).
|
||||
**R-648:** there is NO per-app backup endpoint; drills press nothing and rely on the update's `backing-up`.
|
||||
**R-650:** a test that reaches real docker acts on DooPlex — measured once (my draft), unguarded.
|
||||
Evidence: `audits/cleanup-2026-09-23/README.md`.
|
||||
|
||||
## 2026-09-23 (evening) — the household is told: controller v0.264.0 + hub v0.120.0 (`09` §6.4 parts 2–3); floor 0.264.0
|
||||
|
||||
**Events.** `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, sent by the
|
||||
controller through `Manager.SetUpdateEventSink` (wired in main.go). Details `{app, stack_name, from, to, at,
|
||||
copy_tier, copy_date, copy_holds}`. The held mail's message line is the hold sentence rendered in the
|
||||
household's language (`pushEventBoth`); the operator gets the Hungarian wire text.
|
||||
**Hub.** Allow-listed, NOT operator-only, in `defaultSeedEvents`; `mail.event.*` hu+en name the app via the
|
||||
new register `appNamedMailEvents` (reads `details.stack_name`). **Existing households:** a one-time add-only
|
||||
migration `Store.SeedEventTypesOnce` guarded by `hub_settings.seed_app_update_events_v1` — it changed all
|
||||
three rows (`demo-felhom`, `demo-hp`, `_resend-rotation-test`; before/after in
|
||||
`audits/undo-fleet-2026-09-23/10-*`, `11-*`). **Cooldown:** per app on both legs — operator via R-389's
|
||||
`perAppCooldownEvents` (widened on purpose), household via a NEW sibling `perAppCustomerCooldownEvents`
|
||||
(the customer key was `customer:type`). `app_start_failed`'s household grain is unchanged (pinned).
|
||||
**Why the box also seeds:** the controller pushes its own `EnabledEvents` to the hub on every notifications
|
||||
page save, so a hub-only migration would be undone by the next save; the box seeds once
|
||||
(`app_update_events_seeded`) and the page has the two toggles.
|
||||
**R-606:** update sentences are keys + args, rendered per reader; leftovers R-647. **R-620:** a disabled
|
||||
notifier WARNs once per type. **R-646:** startup `BackfillAppliedMeta` for apps current with the catalog.
|
||||
**Voice:** the undone line ends „nincs teendőd" (the product's „te"), not the brief's „teendője nincs".
|
||||
**Harness lesson (R-648):** the drill's „Mentés most" is whole-box and restarts standing apps.
|
||||
Evidence: `audits/undo-fleet-2026-09-23/README.md`.
|
||||
|
||||
## 2026-09-23 (afternoon) — the undo is built: controller v0.263.2 (decisions 15, 19, 20)
|
||||
|
||||
**Operator rulings 19 and 20** recorded in `09` §3: the copy method is chosen by a bake-off; on update
|
||||
nights the full-system backup waits for the update leg inside its window (leg stops at W+5h; built with
|
||||
§6.4 part 7). **Bake-off** (`audits/undo-bakeoff-2026-09-23/`): both methods passed; the folder copy
|
||||
won because an app with no database server gets no dump. **Built** in controller v0.263.0–v0.263.2:
|
||||
phase `copying` after the pull (named volumes `cp -a` into `<vol>.pre-update-<stamp>`, marker last),
|
||||
`undoing`/`undone` on a failed check, a HOLD only when the undo fails (prefix + data state), journal
|
||||
recovery for both new phases, `last_update_undone` + a page line, R-642's honest Start answer.
|
||||
**Proven live on 9202** (`audits/undo-live-2026-09-23/`). **Two live-only defects, fixed:** the
|
||||
undo's probe must run on an app the current probe marks `unhealthy`; and the pinned version's
|
||||
`.felhom.yml` must be recorded AT PIN TIME (`applied-meta/`), because `.felhom.yml` flows in on every
|
||||
catalog sync. **Residual:** R-646 (apps pinned before v0.263.2), R-638/R-640 narrowed to the restore
|
||||
paths, R-645. **The floor is NOT raised — that is the operator's question.**
|
||||
|
||||
## 2026-09-23 — the operator rules on automatic updates (`09` §3 decisions 11–18); two spikes say what the build needs
|
||||
|
||||
**Operator rulings, not CC decisions.** One window = a leg of the backup chain (11); automatic, per-box
|
||||
switch on by default (12); **the test decides, not the tag** (13, replaces decision 3's "never across
|
||||
a major"); the ladder, one tested step at a time (14); **the box undoes a failed update itself** (15,
|
||||
replaces §6.1's no-auto-undo); PostgreSQL majors converted by the box (16); digests recorded by the
|
||||
catalog and pulled exactly (17); fleet view later (18). §3b is kept, marked ANSWERED.
|
||||
|
||||
**The two mechanisms were spiked the same day, by hand, on 9202** (`audits/update-rulings-2026-09-23/`):
|
||||
- **The undo works** — docmost and romm, whose OLD versions refuse migrated data, came back with data
|
||||
written before AND after the backup, ≈16 s and ≈38 s. **But not with the loader the product has:**
|
||||
`ImportDump` over a migrated PostgreSQL database FAILS on the new tables' foreign keys, and over
|
||||
MariaDB leaves the new tables behind (R-638). A truncated PostgreSQL copy loads rc 0 into an EMPTY
|
||||
database and `ValidateDump` would accept it (R-640). No-DB apps have no last-second copy (R-641).
|
||||
Build list: R-637.
|
||||
- **The ladder is absent**: one press jumped 2.3.0 → 2.5.0; the box's catalog clone is `--depth 1`
|
||||
and cannot see steps. Recommended format: `update_ladder:` in `.felhom.yml`, each intermediate step
|
||||
carrying its own definition — not the git history.
|
||||
- **The harness watches memory** (`upgrade-test.py` v2): the RomM template as promoted fails at +76 s.
|
||||
|
||||
**Build order and costs:** `09` §6.4 (≈22 evenings). **One open point for the operator:** the chain as
|
||||
ruled leaves the update leg ≤15 min a night (R-643).
|
||||
|
||||
## 2026-09-20 — localisation slice 5: the app catalog's copy model is MEASURED, not proposed (R-560)
|
||||
|
||||
`architecture/10-localisation.md` §7 was a proposal; it is now a fact, with the numbers it was
|
||||
specced against corrected. **Controller v0.257.0** reads an `i18n: {en: …}` sibling block inside
|
||||
`.felhom.yml`; **the catalog** carries the Hungarian freeze, the `copy-i18n` gate and a three-app
|
||||
pilot (privatebin, paperless-ngx, romm — 89 of 1 032 strings).
|
||||
|
||||
**The two numbers that were wrong, and one of them chose an instrument.** The catalog has **1 032**
|
||||
copy strings, not 835 (the inventory omitted 13 `placeholder` values and every ASCII-only label);
|
||||
**832** of them carry a Hungarian letter, which was right. But the ASCII-ONLY Hungarian is **~120
|
||||
strings, not three** — „Aldomain" 53×, „A szerver domain neve" 53×, „Jelentkezz be: …" 5×, „Magyar",
|
||||
„Angol", „Titkos kulcs" — and „Igen"/„Nem"/„Nincs" do not occur in this catalog at all. **An
|
||||
accent-only scan passes every one of them inside an English block**, which is R-565's blind spot
|
||||
arriving again in a different repo. The catalog gate folds and matches WORD-BOUNDED stems, with a
|
||||
positive and two negative controls printed every run.
|
||||
|
||||
**Part C shipped the same day.** The operator read the pilot and said go; the other fifty followed in
|
||||
three pushes (17 / 17 / 16 apps). **1 031 of 1 032 strings are English.** The one that is not is
|
||||
papra's `AUTH_SECRET` description — a Hungarian defect (R-593) left to fall back rather than
|
||||
translated wrongly, and **that is why `EN_MISSING_CEILING`'s floor is 1 rather than 0**, stated in the
|
||||
ceiling's own comment so a later session cannot "reach zero" by editing Hungarian.
|
||||
|
||||
**[DESIGN, decided by CC] The English blocks are GENERATED from a flat `{path: english}` map.** A
|
||||
mistyped `env_var` in a hand-written block is INERT on the box rather than an error; the generator can
|
||||
only write paths that exist on the Hungarian side.
|
||||
|
||||
**Measured: the English Apps list shows ZERO Hungarian app descriptions across all 53 apps.** The only
|
||||
Hungarian on it is the „Naprakész" badge (R-589) and the language picker naming itself. The Hungarian
|
||||
Apps list is identical to the pre-slice capture once the CSRF token AND Docker's own „Up N hours"
|
||||
string are normalised — both normalisations stated, not applied quietly.
|
||||
|
||||
**FLEET FLOOR RAISED to 0.257.0** (operator asked), `min_agent` 0.131.0 declared, above the vouched
|
||||
golden 0.246.0 so the declaration carries it (R-472). Hub `managed floor SERVED … from declared`;
|
||||
demo-felhom went **0.255.0 → 0.257.0 by itself in under 12 s**, healthy, `settle-gate: GO`, **and then
|
||||
rendered the English tagline** — the floor delivered the FEATURE, not a version string. Peti's box and
|
||||
tester-1 are DOWN and take it unattended when they return.
|
||||
|
||||
**Gaps filed:** R-589, R-590, R-591, R-592 (closed same session), R-593, R-594. **R-560 CLOSED.**
|
||||
Evidence: `documentation/audits/i18n-slice5-2026-09-20/`.
|
||||
|
||||
## Decisions 2026-09-17 (afternoon) — the product in more than one language (localisation starter)
|
||||
|
||||
**Operator rulings on the starter** (`architecture/10-localisation.md` §11): (1) scope = what the
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
# REPORT — backlog triage, 2026-10-03
|
||||
|
||||
Paperwork session. **No machine touched. No release. No golden.** Other repos read only.
|
||||
Baseline: felhom.eu `main` `9305288` (verified). Commits: A `9e2786c` · B `71b8c8c` · C `9f77865` · D `33bbf8e` ·
|
||||
E (this commit).
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | Done? | Changed from the brief, and why |
|
||||
|---|---|---|
|
||||
| **A** — four roadmap items | **Done.** R-808 (box OS security updates), R-809 (legal + business papers), R-810 (independence, spike), R-811 (second login step). Findings filed beside two: **R-812** (no box receives OS security updates), **R-813** (website: no privacy notice, terms or imprint). `00` gained three §E/§G gap rows and a new §H "Business & legal". CONTEXT records the request first. | R-810 and R-811 got no register finding: nothing about them is false today (one password is a stated limitation, `00` §E). |
|
||||
| **B** — finished rows out + gate | **Done.** 125 rows with an id and 20 without moved to `CLOSED-ITEMS.md` (one dated section, each naming `git show 9e2786c:…`). Open rows normalised to one shape. Narratives (campaign write-ups, rulings, old ranking paragraphs) moved word for word to `documentation/archive/OPEN-ITEMS-narratives-2026-10-03.md`. `closed_register_gate.py` **RULE 3**. Workflow text in `PROMPT-TEMPLATE.md` §N.7/§N.5 and `CLAUDE.md`. Loose notes: verdict per file in `backlog/README.md`. | 14 rows with an OPEN verdict were found finished and moved — each checked by me against live source (list below). 11 rows with a finished verdict **stayed open**, narrowed, because they name work no other row carries (R-451, R-579, R-610, R-618, R-621, R-635, R-691, R-707, R-719, R-723, R-738). Two loose notes stay in place (other repos link to their path). There is no "Deliverables" line in the template; §N.5's "report which rows" line gained "closed (and moved), narrowed". |
|
||||
| **C** — category + severity | **Done.** Columns `| ID | Category | Sev | What | State | Blocked on | Next action | Owner |`; one section per category, severity order inside. `register_shape_gate.py` RULES 5–8. Duplicates folded: R-248 → R-246, R-580 → R-132. | A **column**, not a title tag: the gates read cells by the header's column NAME (new helper `register_table.py`), which a tag in prose cannot give reliably. Not folded (judged not duplicates): R-287/R-291, R-450/R-469, R-123/R-369 (R-123 closed anyway). Operator-owned rows: listed in `RECOMMENDATION.md`, with one line and the count in STATUS — STATUS is one screen and holds no ids. |
|
||||
| **D** — clean ROADMAP | **Done.** 161 → 124 lines, 81 → 42 KB. Intentions re-sorted P2/P3/P4, each names its `00` row. 33 items + the pre-invite checklist → `ROADMAP-HISTORY.md`. UPDATE-ARC collapsed. Pre-invite list → pointer to STATUS. | R-48 was SHIPPED (controller v0.154.0) though the roadmap still listed it as an idea. Fixing `one_register_gate.py` to read suffix ids found R-50b — a finding that lived only in the roadmap; moved to the register. |
|
||||
| **E** — ranking + recommendation | **Done.** `documentation/audits/backlog-triage-2026-10-03/RECOMMENDATION.md` and one decision in STATUS. | The top list is 29 rows: every P2. No P1 exists, so severity alone fills the 20–30. |
|
||||
|
||||
## Headline numbers
|
||||
|
||||
- `OPEN-ITEMS.md`: **442 rows with an id + 22 without, 824 KB, 932 lines → 326 rows, all with an id, ≈540 KB.**
|
||||
- Moved to `CLOSED-ITEMS.md`: **125 + 20**. Marked VERIFY: **11**. New rows: **11** (R-812..R-819, R-50b moved in).
|
||||
- Category × severity (P2/P3/P4; **no P1**): Install 2/11/5 · Apps 0/15/21 · App updates 0/12/7 · Backup 12/24/20 ·
|
||||
Storage 0/7/5 · Security 3/23/3 · Box system 3/11/3 · Monitoring 2/16/6 · Hub 1/7/13 · Business 4/0/3 ·
|
||||
Process 0/4/83 — totals **27 / 130 / 169**.
|
||||
|
||||
## Claims in the brief that turned out wrong
|
||||
|
||||
1. **The row counts.** Measured at `9305288` with a split that skips pipes inside backticks: **442** id rows (not
|
||||
444) plus **22 rows with no id** the count missed. By leading verdict: **113** finished (107 CLOSED + SHIPPED 2,
|
||||
FIXED 1, RULED 1, EXECUTED 1, ANSWERED 1), plus 4 DECIDED and 1 "✅ CLOSED" — 118 for the new gate; **195**
|
||||
READY (not ~129); OPEN 74 + NARROWED 10; WAITING-ON-OPERATOR 15; WATCHING 15. **"~125 unreadable" was wrong:**
|
||||
in those six-column tables the state is the THIRD column — the script read the last one, which is the owner. Only
|
||||
**18** rows needed a person: 5 because of a pipe in prose, 13 because their state word was undefined.
|
||||
2. **"Nothing updates the host or guest OS" — TRUE for boxes**, with two nuances: the installer points the host at
|
||||
the no-subscription repository "so the box can pull security updates" and then says "No upgrades are run"; the
|
||||
guest's Docker engine is current only on the day the golden is baked. The one `apt full-upgrade` in the project is
|
||||
a by-hand step for the off-site endpoint ep0.
|
||||
3. **"The website has no legal pages" — TRUE.** Worse than stated: the contact form asks for data-processing
|
||||
consent and links to no notice.
|
||||
4. **"No document answers the independence question" — PARTLY WRONG.** The LOST-hub half is answered
|
||||
(`architecture/_recovery-inventory-2026-07-28.md` §D2.4; `07` §8 row 11b), and `01` §7 rules that the customer
|
||||
owns the domain. Leaving, export and hand-over are answered nowhere — R-810 keeps those.
|
||||
5. **"No gate refuses a closed row in OPEN-ITEMS" — TRUE.** One more gate had the same blind spot in another shape:
|
||||
`instructions_gate.py` read the state by position and would have misread the new layout; fixed.
|
||||
|
||||
## Rows found finished and moved (verified by me, evidence in each CLOSED entry)
|
||||
|
||||
R-123, R-131, R-202, R-229, R-272, R-295, R-343, R-369, R-398, R-500, R-506, R-572, R-590, R-800.
|
||||
|
||||
## Gates — new rules and decoys, each seen red
|
||||
|
||||
- `closed_register_gate.py` RULE 3 — red on the real register before the move (118 convicted). Decoy
|
||||
`finished-row-in-open` **passed the old gate**; convicts now. Genuine article (READY, prose says "closed") passes.
|
||||
- `register_shape_gate.py` RULES 5–8 — five decoys (old-shape row, near-miss category, `P3-LOW` as Sev, undefined
|
||||
state word, pipe outside backticks): **all five passed the old gate**; all convict now.
|
||||
- `one_register_gate.py` — suffix ids and backtick-aware split; decoy `suffix-id-row` **passed the old gate**.
|
||||
- Suite: `test_gate_decoys.py` 29/29; `test_instructions_gate.py` 73/73; `repo_gates.py --fast` OK at every commit.
|
||||
|
||||
## Rules carried out of closed rows
|
||||
|
||||
21 sentences that stated a rule and had no other home → `CONTEXT.md` ("Rules carried out of rows closed
|
||||
2026-10-03"); the three decisions among them (R-245, R-303, R-312) → `07` §11 as `[DESIGN]`.
|
||||
|
||||
## Observations
|
||||
|
||||
1. `scripts/check_stands.py` is red and runs in no runner (2 dangling ids before today, 3 more after rows closed).
|
||||
FILED: R-819.
|
||||
2. `09` decision 56 and R-745 disagree about the controller self-update's roll-back target. FILED: R-817.
|
||||
3. Two changelogs cite R-330/R-331 for other findings. FILED: R-818.
|
||||
4. F-DIAG's six off-site failure messages have never been seen on a real failure. FILED: R-816.
|
||||
5. Two July watch rows had no id and no recorded outcome (a Storage Box deletion; the first GC on the off-site
|
||||
datastore). FILED: R-814, R-815.
|
||||
6. One stray duplicate owner word ("operator") in R-209a's broken extra cell was dropped in normalising; every other
|
||||
word of every open row is kept (checked by a token diff). NOT-A-FINDING: a duplicated cell, not content.
|
||||
|
||||
## Teardown
|
||||
|
||||
Provisioned nothing.
|
||||
@@ -0,0 +1,57 @@
|
||||
# REPORT — off-site topic closed; operating-system update spike — 2026-10-04 (day)
|
||||
|
||||
Architecture read: `07-backup-architecture.md` (Lane 2, §6.1), `_recovery-inventory-2026-07-28.md`, `03-host-agent.md`,
|
||||
`09` §3/§4, `11-os-updates.md`. Baselines (re-verified): felhom.eu `d07a1a904cbf` (hub 0.128.0), controller
|
||||
`99a149756070` (0.290.0), agent `d766666ff8cf` (0.138.0), catalog `917a779cca67`. Register 328, highest R-834.
|
||||
Rulings recorded first as `09` §3 decisions 75–77 and `11` committed verbatim (`3885640`). Evidence:
|
||||
`documentation/audits/backup-close-2026-10-04/` and `documentation/audits/os-updates-spike-2026-10-04/`.
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | Result | Notes |
|
||||
|---|---|---|
|
||||
| A — restored guest safe by default (R-834) | **done** | Routes: restore-test (MEASURED safe: onboot 0 and throwaway mp8/mp9 on every poll), DR bring-up (**fixed**: refuses beside a live original — agent v0.139.0, refused live on demo-hp, nothing created), hand route (**new** `scripts/felhom-restore-beside.sh`, proven live on 9298 then destroyed), provisioning (golden, no binds). 4 tests, 2 red-proofs + 1 built-in. **Changed:** no sudoers line — the restore-test sets onboot 0 through the API, and DR refuses rather than degrades. |
|
||||
| B — clean-up cannot wedge (R-833) | **done** | Hub v0.129.0 deployed. 4 red-proofs. Lab proof on a real restic 0.14.0 repo: 98 → 13 under a raised cap, default refused before, normal after. Live: 4 bad grants refused (400). **Changed:** no valid grant placed on a real customer — a demo box would consume it (brief: lab repo only). No controller change needed. |
|
||||
| C — returning household (R-726) | **done** | Two options in STATUS; pick A. Nothing built. |
|
||||
| D — dated check (R-95) | **done** | Due 2026-10-12, four checks named in R-95. |
|
||||
| E — where we stand | **done** | 5 systems surveyed read-only (throwaway apt indexes). |
|
||||
| F — exact version later | **done** | madison host + guest; DSA history 3 months; snapshot.debian.org from a throwaway container on 9202. |
|
||||
| G — guest update on 9202 | **done, one deviation** | **Changed:** 9202 is on `dir` storage and cannot snapshot, so the undo was a backup + restore (73 s); the snapshot rollback is unmeasured (R-837). G3 interrupted the update straight after the undo (the "apply again" happened as G3's repair) — the same update could not be interrupted once applied. |
|
||||
| H — host update on demo-hp | **done** | Debian lane 108 packages, 60 s, guests up. One-package undo: rsync gone, libpng worked. Kernel: two reboots on the operator's word — new kernel, then fallback to old. Proxmox simulated only. |
|
||||
| I — design record | **done** | `11` corrected (C1–C12), wrapper draft §5.4.1, answers §7.1, sample list (157 packages) simulated on demo-felhom, Q10 price recorded. Two STATUS decisions. |
|
||||
|
||||
## Claims that turned out wrong (named)
|
||||
|
||||
1. **"Debian's archives keep only the newest version"** (`11` §5.3) — they keep two: the point-release one and the newest security one; intermediates are gone (C2).
|
||||
2. **"Proxmox and Docker keep older ones"** — true, measured: 30–66 and 18–46 versions.
|
||||
3. **"`--next-boot` falls back by itself"** (`11` §5.6) — only after a boot that reaches userspace; on GRUB it is an ordinary default; a hang keeps the new kernel (code-read). And installing a kernel alone makes it the default (C4).
|
||||
4. **"The guest has no `live-restore`"** — true. But `live-restore` is the answer to Q3, and switching it off again is a trap (C5, R-835).
|
||||
5. **"The agent may not run `apt` except for `dnsmasq`"** — it may also install `wireguard-tools` (C1).
|
||||
6. **"The restore-test guest is safe today"** — TRUE, measured (onboot 0, no host bind). The unsafe routes were the DR bring-up and the hand route.
|
||||
7. Also wrong in `11`: the slow-lane list by name (40 Proxmox packages have plain names, C3); `cloudflared` "on the host" (it is a guest container, C8); approving what ring 0 installed (C9); a fast-lane run is "a service restart at most" — libc leaves PID 1 and `lxc-start` on the old library (C11).
|
||||
|
||||
## Found and handled in-session
|
||||
|
||||
- My own output filter dropped every line containing "perl" — including "paperless". A false "the app vanished" was caught before acting on it; evidence files were saved unfiltered.
|
||||
- `pkill -f dpkg-deb` killed my own shell during G3 (the known trap); the kill itself had landed, and the state was read in a fresh command.
|
||||
- The first Docker probe counted 302/404 answers as down; re-counted from the raw probe files with "no answer" as down.
|
||||
- A `pgrep` waiter matched itself and never ended (the known trap); it was harmless and killed by its timeout.
|
||||
|
||||
## Rows
|
||||
|
||||
Closed: **R-833, R-834**. Opened: **R-835** (live-restore off trap), **R-836** (kernel hang keeps new kernel), **R-837**
|
||||
(snapshot undo unmeasured), **R-838** (cloudflared pinned since June, P2), **R-839** (boot sweep held an app whose
|
||||
`HDD_PATH` names its folder). Narrowed: **R-812** (spike done), **R-95** (dated check), **R-726** (waiting on the
|
||||
operator). Register **328 → 331**.
|
||||
|
||||
## Teardown, three layers
|
||||
|
||||
- **Machines:** scratch VMIDs 990000 (restore-test, torn down by the agent) and 9298 (destroyed); no 9297 was created.
|
||||
9202: Debian fully updated, Docker 29.8.2 / containerd 2.3.6 (golden 0.290.0's), `daemon.json` byte-identical to the
|
||||
baked one, all apps healthy; its backup deleted; the `debian:trixie` probe image removed. **demo-hp host:** 108 Debian
|
||||
packages + kernel `7.0.14-20-pve` installed (110 changes, `partH/H-final-host-packages-after.tsv`); **running
|
||||
`7.0.2-6-pve`, next boot `7.0.14-20-pve`, no pins**; 78 Proxmox packages still pending. demo-felhom: read only (its
|
||||
agent updated to 0.139.0 by signed job). Helper files removed from every host and guest.
|
||||
- **Host (DooPlex):** the lab restic repo removed; scratch copies of the hub password and the DSA list shredded. Agent
|
||||
0.139.0 released (tag + package, verified by download), not vouched.
|
||||
- **Hub:** v0.129.0 deployed; no grant left pending; weekly windows unchanged (ON).
|
||||
@@ -0,0 +1,74 @@
|
||||
# REPORT — rulings 61 (calibre-web's generated login name) and 62 (the registry prune rule) — 2026-10-01 late afternoon
|
||||
|
||||
Evidence: `documentation/audits/calibre-name-and-prune-2026-10-01/` (A, B, T); tools in `audits/lockouts-2026-10-01/tools/`
|
||||
(`a_calibre_name.py`, `lk.py`, `walk.py`, `repoint.py`).
|
||||
Read: `09` §3 decisions 45, 57–60; `FIRST-ADMIN.md`; rows R-752, R-750, R-753; `audits/lockouts-2026-10-01/` B1, C1;
|
||||
homelab-manifests HM-024. Baselines (~12:55 CEST): controller `c1b123c64955`, felhom.eu `8dab40c7a786`, catalog
|
||||
`ed6df4b46b93` — matched. Register 390; highest R-755; last decision 60 → the rulings are **61 and 62**.
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | done / not done / changed | why |
|
||||
|---|---|---|
|
||||
| Rulings 61, 62 | **done** — recorded first (`09` §3, CONTEXT) | numbered 61/62: 58–60 were taken by the lockouts session |
|
||||
| **A1 measure** | **done** — `hex:N` + `type: secret` already exist; calibre-web has no rename command | — |
|
||||
| **A2 build** | **done** — catalog `e9f50b5` (template, hu + en copy, the freeze for those 5 strings, FIRST-ADMIN) | no controller change |
|
||||
| **A3 proof on 9202** | **done** | — |
|
||||
| **A4 installed apps** | **done — and a defect found** (R-757); demo-hp renamed TWICE | the box invented a name for the installed app |
|
||||
| **B prune rule** | **done** — `admin/misc-scripts` `c9d5ed5`; test red-proofed; live dry-run | the running hub added to "in use" (a version in use the ruling did not name) |
|
||||
| B4 runbook line | **done** — `RUNBOOK-manual-build.md` §4.1a | HM-024 lives in homelab-manifests (outside the felhom fence) |
|
||||
| **C release / golden** | **not needed** | A1 needed no controller change |
|
||||
|
||||
## Claims in the brief that turned out wrong (or right)
|
||||
|
||||
1. **"The controller can generate a login name"** — right: `generate: "hex:N"` (deploy.go:1187) gives lowercase a–f and
|
||||
digits; `type: secret` is filled when empty and shown behind „Megjelenítés".
|
||||
2. **"calibre-web can rename a user"** — **no command does**: `cps/cli.py` offers only `-s user:password`
|
||||
(`ub.py:1350 password_change`). Its admin page renames by setting `user.name` (`admin.py:2789`, column `ub.py:264`,
|
||||
unique). So `after_install` updates that column itself, then uses Calibre-Web's own `-s` for the password.
|
||||
3. **"The OPDS door uses the same name"** — right: OPDS is limited per name (`cps/main.py:75`, `request_username`); a
|
||||
stranger's tries on `admin` never touch the real name (measured: OPDS with the real name ok after 40 tries on `admin`).
|
||||
4. **Where the prune script lives** — in a repo already: Gitea `admin/misc-scripts` (`~/git/misc-scripts`). The August
|
||||
run is in its own log: `2026-08-22T16:02:20Z RUN action=prune … apply=true keep='7'`.
|
||||
5. **"A template change reaches an installed calibre-web only through an Update"** — wrong in a way that matters: the
|
||||
template reached demo-hp at the next sync (images equal), and the box then INVENTED the new field's value
|
||||
(`InjectMissingFields`, R-757). My own first CHANGELOG line said "frozen until an Update" — also wrong.
|
||||
|
||||
## Part A — calibre-web
|
||||
|
||||
**9202 (drill catalog `4e18b3a`, identical to live `e9f50b5`)** — `A/A1-9202-calibre-generated-name.txt`:
|
||||
install hold before the first start, opened by `after_install` at 11:02:17; a stranger polling `admin/admin123` from the
|
||||
deploy press got in **0 of 31** times; `after_install` record `ok: true`; the name 10 lowercase hex characters (read
|
||||
through the page's reveal); app.db: 2 users, 0 named `admin`; name + password: form ok, OPDS ok; `admin` + the right
|
||||
password refused; **40 wrong tries on `admin` at 3/min (11:02–11:16) → the household at once: form ok, OPDS ok**; a wrong
|
||||
password on the real name refused. Removed (drive data kept: R-756).
|
||||
|
||||
**demo-hp** — `A/A2-demo-hp-rename.txt`: renamed by the same method (values through stdin, never printed); a real login
|
||||
over its traefik: name ok (form, OPDS), `admin` wrong. Then the box's sync injected a DIFFERENT `ADMIN_USER` into its
|
||||
app.yaml (R-757, `A/A3…`); renamed again to the box's recorded value; verified (the earlier name and `admin` refused).
|
||||
**The name is in `~/.config/credentials` as `DEMO_HP_CALIBRE_USER`** (backup `credentials.bak-20261001-calibre`); never in a repo.
|
||||
|
||||
**What any other installed calibre-web gets, and when:** at the next catalog sync (≤ 15 min) its `.felhom.yml` gains the
|
||||
field and the box invents an `ADMIN_USER` for it; its login stays `admin` (after_install runs only after a fresh install).
|
||||
No other box has calibre-web today (the N100 does not; Tester-2 has not registered).
|
||||
|
||||
## Part B — the prune rule
|
||||
|
||||
`tests/test-prune-plan.sh`: 7 checks pass (an in-use version older than the newest 20 is kept, with its reason; `--keep`
|
||||
defaults to 20; dry-run; an unreadable in-use list → exit 3). Red-proofs: the same plan with an empty in-use list deletes
|
||||
0.262.0; the in-use check removed from `is_protected` → 3 checks fail (`B/B1-test-and-red-proof.txt`).
|
||||
Live dry-run (`B/B2-live-dry-run.txt`): in use — controller 0.285.0 (floor, golden's, baked), golden 0.285.0, agent
|
||||
0.138.0 and 0.131.0, hub 0.126.0, felhom-samba 1.1.0. Would delete: felhom-controller 70, felhom-hub 8; every other
|
||||
package nothing. **No `--apply`.** No token or password in any output (grepped for each value).
|
||||
|
||||
## Rows
|
||||
|
||||
**390 → 392.** Closed R-750, R-752. Opened R-756 (9202 remove-with-data refused), R-757 (the box invents a new secret
|
||||
field's value for installed apps).
|
||||
|
||||
## Teardown
|
||||
|
||||
- **Machine:** 9202 back on the live catalog (`repo_url` read back), the same six containers; calibre-web removed through
|
||||
the product (drive data kept, R-756). demo-hp: calibre-web's user renamed (the only change there).
|
||||
- **Host:** nothing. **Hub:** read only (the Configuration page, for the dry-run). **Gitea:** read only; one repo push
|
||||
(`misc-scripts`). Drill catalog reset to live (`e9f50b5`).
|
||||
@@ -0,0 +1,20 @@
|
||||
# REPORT — 2026-09-29/30: DRILL — a new household's first day on golden 0.282.0
|
||||
|
||||
The full record is `documentation/audits/DRILL-new-household-2026-09-30.md` (evidence beside it). This file is the
|
||||
session summary only.
|
||||
|
||||
- **Interventions: 0.** Ready for a first real tester: **yes, on a new customer record** — after the guide fix
|
||||
(R-722) and with the off-site-per-app decision (R-720) in view.
|
||||
- **Golden 0.282.0** baked, round-trip verified, vouched (agent 0.137.0, min_agent 0.131.0); R-120 negative control
|
||||
refused 0.276.0. Record `documentation/tests/golden-0.282.0-2026-09-29/`.
|
||||
- **Operator ruling during the run:** customer `tester-1` instead of a new drill record → no Cloudflare items were
|
||||
created; the customer is kept; only the drill host was deleted.
|
||||
- **Rows:** opened R-719 … R-727 (six P2, three P3); closed R-505; measured again R-718, R-600. Register **350 → 359**.
|
||||
- **Night one:** database dump ran; second-drive copy not configured; off-site copy skipped (orphaned repository of
|
||||
an earlier box, R-726; and apps are off-site OFF by default, R-720); whole-guest tiers not due; restore test failed
|
||||
on an earlier box's archive (R-727).
|
||||
- **Teardown:** VM 340 destroyed, ISO removed, storages back to their starting sizes; host `tester-1-693e79` deleted
|
||||
with the escrow acknowledgement; ep0's WireGuard peer gone 48 s later by itself; three tester-1 whole-guest
|
||||
archives on ep0 listed and left for the operator. Demo boxes' guests, `drill-r50`, DooPlex beyond bake/vouch/hub
|
||||
pages: untouched.
|
||||
- **No product code changed. No `--no-verify`.**
|
||||
@@ -0,0 +1,77 @@
|
||||
# REPORT — the family gate built; Grimmory and MeTube published; every app's licence read (2026-10-02)
|
||||
|
||||
Brief: "the family gate, built (controller + catalog), Grimmory and MeTube published behind it; every app's licence read;
|
||||
SparkyFitness's licence ruling recorded; one controller release and a golden bake". Architecture read:
|
||||
`01-topology-and-trust.md` §5 (the family-gate paragraph is new), `09` §3 decisions 46, 47, 63, 64, 65.
|
||||
Evidence root: `documentation/audits/family-gate-2026-10-02/`.
|
||||
|
||||
| Part | State | One line |
|
||||
|---|---|---|
|
||||
| **A** — the controller | **DONE** | v0.287.0: family members with own logins, a door per family app, anchored exceptions, the Család card. Exit items 1–5 live on 9202. |
|
||||
| **B** — the catalog | **DONE** | Format + gate `family-gate`; Grimmory (4 e-reader exceptions) and MeTube (none) published (catalog `96829d0`) with complete records; on 9202 and demo-hp through the product. |
|
||||
| **C** — licences | **DONE** | 58 templates, every image, one table (`audits/licences-2026-10-02/TABLE.md`); short list in STATUS; nothing hidden or changed. |
|
||||
| **D** — SparkyFitness | **DONE (not sent)** | Request drafted (`audits/licences-2026-10-02/EMAIL-DRAFT-sparkyfitness.md`); decision 65 + trigger in STATUS. |
|
||||
| **E** — release | **DONE** | Floor 0.287.0 (min_agent 0.131.0), both demo boxes in ~25 s; golden 0.287.0 baked, round-tripped, vouched; currency gate OK. Phone test SKIPPED (operator not present). |
|
||||
|
||||
Not stopped between sessions: Part A was proven live on 9202 before Part B started (`A/items.txt`).
|
||||
|
||||
## Claims in the brief, checked
|
||||
|
||||
- **"The family session is separable with the existing cookie scheme"** — right, with one addition: a separate cookie
|
||||
name (`felhom_family`) scoped to `Path=/__family` and a separate store; RequireAuth never reads it (test
|
||||
`TestFamilyGate_MemberPassesButNeverTheDashboard`, red-proof RP-F1; live: the member's cookies at `/launcher` → the
|
||||
dashboard login, also through the real internet).
|
||||
- **"Grimmory's e-reader paths keep their own auth behind an exception"** — right, measured per path as a stranger
|
||||
through the simulated tunnel: OPDS 401 (Grimmory's), Kobo made-up token 401, KOReader wrong key 401, Komga API 401;
|
||||
the right credentials 200; look-alikes (`/api/v1/opdsx`, `/api/koreaderx`, `../`) → the gate.
|
||||
- **"MeTube's websocket passes forwardAuth"** — right: a member's upgrade → 101; a stranger's websocket and polling → 401.
|
||||
- **"Whether n8n's licence limits paid services"** — partly: own internal or personal use is allowed; providing it to
|
||||
others is allowed only free of charge for non-commercial purposes. Pulling the unchanged image onto the household's
|
||||
box is arguably not that — a grey zone, now an operator row (R-791).
|
||||
|
||||
## Proof
|
||||
|
||||
- **Controller:** suites green; red-proofs RP-F1..F7 (`A/RP-F-family-gate-mutants.txt`). Live (`A/items.txt`): a stranger
|
||||
0 app answers of 36; members in; reset/remove/logout end access at the next request; a stranger's 7 guesses lock only
|
||||
the stranger; controller down → 500, never the app; the gate costs 0.49 ms. **Through the real internet on demo-hp**
|
||||
(`B4/member-internet.txt`): a member's browser → the family sign-in → MeTube; logout → refused; a stranger 401.
|
||||
- **Catalog:** `catalog_gates.py grimmory` and `metube` exit 0 on the bench, every gate OK (`B/catalog-gates-*.txt`).
|
||||
Decoys of the new gate seen red (`B/family-gate-decoys-red.txt`).
|
||||
- **Stranger per exception (Grimmory):** `A/items.txt` item 4; R-775 re-measured: 6 tries → the gate's 401, then the
|
||||
household signs in 200 — the 15-minute lock can no longer be aimed from outside.
|
||||
- **9202 lifecycle** (`B/box/`): Grimmory step 3.4.1 → 3.5.0 behind the gate (58.5 s, book read back); MeTube fresh
|
||||
install (a stranger's 63 polls: 404 → 401, never the app), step .28 → .29 (34.9 s), night backup, remove keeping data,
|
||||
restore (64.8 s / 38.5 s), seeds read back, the gate files and the stranger's refusal back by themselves.
|
||||
- **demo-hp, live catalog** (`B4/`): both installed through the product, strangers 401 through the real internet and
|
||||
the LAN, removed with data; the apps were removed after (STATUS asks whether to keep them).
|
||||
|
||||
## Found on the way (all register rows)
|
||||
|
||||
- **R-801 (P2, fixed in the catalog):** the volume-persistence gate never sent a request to ANY app — it read the port
|
||||
from label values, the port is in the label name. Fixed (`routed_ports()`), red-proofed (`B/RP-R801-routed-ports.txt`).
|
||||
**A re-sweep of all 58 templates is owed.** Before the fix MeTube answered UNDETERMINED; after, CLEAN with its own
|
||||
download (`B/volume-persistence-after-R801.txt`).
|
||||
- **R-800 (P2, operator):** "remove with data" keeps an app's files in userdata, and the dialog does not say so.
|
||||
- R-788 (`APP_EXERCISE`), R-796 (MeTube's send-to helpers cannot pass the gate), R-797 (rule 3 not checkable in CI),
|
||||
R-798 (Grimmory's dead env line), R-799 (fixture field). Licence rows R-789..R-795.
|
||||
- Harness only (no row): box_walk cached "not gated" while an app had no router (fixed); Cloudflare refuses Python's
|
||||
user agent (error 1010) — a test client detail, the box was never reached.
|
||||
|
||||
## Rows and gates
|
||||
|
||||
- **Register 422 → 436:** closed R-767, R-780, R-787; updated R-775 (narrowed), R-784 (decided B, draft ready),
|
||||
R-788; opened R-788..R-801 (14).
|
||||
- **Releases:** controller v0.287.0 (one release). Catalog `96829d0` (+ the record amendment). Golden 0.287.0.
|
||||
- **CI by head_sha:** catalog 96829d0 → job 1201 success; felhom.eu e6d1ebd → 1200 success; controller 9821690 → 1199
|
||||
success (all pushes of the session green).
|
||||
- `unproven.py --summary`: unchanged (not walked 35 of 55).
|
||||
|
||||
## Teardown
|
||||
|
||||
- **9202:** both apps removed (keep-data remove; the harness removed its own test folders by hand — R-442 refuses
|
||||
with-data there); back on the live catalog (`A/repoint-drill.txt` controls); family members anna/bela remain on 9202's
|
||||
card (scratch box, harmless).
|
||||
- **Bench 9401:** rebuilt and destroyed four times; `pct list` 9201 + 9202 only; nvme-scratch back to its baseline.
|
||||
- **demo-hp 9201:** apps removed, test files removed by hand, the test member removed (`B4/teardown.txt`).
|
||||
- **Hub:** the floor and the artifact manifest changed on purpose; nothing else. **Secrets:** scratch files shredded at
|
||||
the end; evidence scanned for every secret value used (0 hits, positive control 1).
|
||||
@@ -0,0 +1,79 @@
|
||||
# REPORT — 2026-09-30: the fixes before the first real tester
|
||||
|
||||
Architecture read first: `07` §6 (the 2026-09-16 ruling), `03` (the restore test's selection), `05` (self-bind),
|
||||
`04` §3.1 (signed delivery), `09` §3 decisions 45–49; rows R-719 … R-727, R-95, R-240, R-494, R-600, R-688.
|
||||
Releases: **controller v0.283.0 + v0.283.1**, **agent v0.138.0**, **hub v0.126.0**. Evidence:
|
||||
`documentation/audits/evidence-fixes-first-tester-2026-09-30/` (part0, partA, partC, partD, partE, release).
|
||||
|
||||
## Tester-2 — read-only checklist (nothing on Tester-2, Cloudflare, ep0 or the Storage Box was changed)
|
||||
|
||||
| # | item | state | where to fix |
|
||||
|---|---|---|---|
|
||||
| 1 | Tunnel token pasted | **done** — tunnel `3ce0eccd…` (Peti's original, reused) | — |
|
||||
| 2 | DNS of `sajatfelhom.hu` | **done** — ONE record `*.sajatfelhom.hu` → that same tunnel; no leftover pointing elsewhere; today 530 (no connector, right with no box) | — |
|
||||
| 3 | The tunnel's route `*.sajatfelhom.hu` → `https://traefik`, No TLS Verify | **UNKNOWN** — the record's Cloudflare key reads DNS only (`Authentication error` on the tunnel; stopped there) | Cloudflare → Zero Trust → Networks → Tunnels → this tunnel → Published application routes |
|
||||
| 4 | Off-site | **done** — shared 100 GB, new sub-account 322460 (username `sub2` reused; Peti's was emptied and deleted 2026-09-25) | — |
|
||||
| 5 | DR tier (ep0) | **done** — ON; nothing of Tester-2 or Peti on ep0 yet (made at the first connection) | — |
|
||||
| 6 | The connect e-mail | **sent twice at 07:36 UTC** (the customer was created twice, R-728) — **only one of the two links works**; valid until 2026-10-07 07:36 UTC | Tell your friend: if one link says „expired", use the other; or press „Send self-bind link" once just before the install |
|
||||
| 7 | E-mail language | **English** — mail, bind page and the box start in English | Hub → Tester-2 → Edit, if Hungarian is wanted |
|
||||
| 8 | Owner passphrase | yours to hand over | in person / by phone |
|
||||
| 9 | Customer id `Tester-2` has a capital letter | never walked before; no known break | note only |
|
||||
|
||||
## The Parts
|
||||
|
||||
| Part | state | note |
|
||||
|---|---|---|
|
||||
| 0 Tester-2 read-only | **done** | checklist above; items 3 and 6 need you |
|
||||
| A1 measure the over-quota path | **done** | it deletes nothing extra: refuses new pushes, runs only the ruled retention; now pinned |
|
||||
| A2 apps off-site by default + one press for older apps | **done** | controller v0.283.0; live on 9202 |
|
||||
| A3 size warning | **done** | page card; unit + parity proven (a household NAS target has no quota to test live) |
|
||||
| B guide + slips | **done / narrowed** | six stale lines rewritten (the sixth found on the way: auto-reboot); R-724/R-725 partly, the rest narrowed |
|
||||
| C1 ep0 cleanup | **done** | three archives removed; every other namespace byte-identical |
|
||||
| C2 agent v0.138.0 | **done** | signed delivery to both demo boxes (340 s); due-check normal on both |
|
||||
| D fresh link for a returning customer | **changed** | the brief's trigger cannot be built (a registering box is unclaimed); built „Új linket kérek" on the expired/used page |
|
||||
| E Stop holds | **done, after a live failure** | v0.283.0 was wrong in production (adapter); v0.283.1 fixed and proven live |
|
||||
| F day-one mails | **done** | unit + red-proof; live proof at Tester-2's first hour |
|
||||
|
||||
## Claims in the brief that turned out wrong, named
|
||||
|
||||
- **"The over-quota path prunes history"** — **wrong.** Over the quota the box refuses new pushes and runs the SAME
|
||||
retention as every night; with no new snapshots nothing extra ages out. No P1.
|
||||
- **"Peti's Cloudflare records for `sajatfelhom.hu` may still be there"** — **there is exactly one record, and it
|
||||
points at the tunnel Tester-2 now carries** (Peti's tunnel, reused). Nothing stale to remove.
|
||||
- **"A PBS archive carries its box's key fingerprint or host id"** — **half right:** the key fingerprint yes (PVE
|
||||
content `encrypted`), a host id no (the comment is only „felhom local-api", the owner is the customer's token).
|
||||
- **"The hub sends no link at registration"** — **right, and it cannot:** the registration carries nothing of a
|
||||
customer. The fix was changed to a button on the old link's page.
|
||||
- **"Stop is lost at the backup's resume only"** — **wrong:** also at the nightly volume dump, the update leg and the
|
||||
startup crash recovery; all four fixed.
|
||||
- **Mine, from 2026-09-30:** "the ✗ names the wrong tier" — misread; the local-tier heading was the NEXT section.
|
||||
|
||||
## Red-proofs (each seen failing on its assertion, then restored)
|
||||
|
||||
RP31 new app not ON · RP32 earlier OFF overridden · RP33 hook not wired · RP34 over-quota forget differs · RP35 exact
|
||||
quota "does not fit" · RP36 quiesce restarts a stopped app · RP37 dump restarts it · RP38 update leg presses it ·
|
||||
RP39 (agent) an earlier box's archive picked · RP40 new box "recovered" · RP41 first-hour skip mailed · RP42 no fresh
|
||||
link · RP43 the production adapter does not answer · RP44 crash recovery restarts it. Outputs:
|
||||
`evidence-fixes-first-tester-2026-09-30/` and the scratchpad `rp/` copies.
|
||||
|
||||
## Slips of mine, said
|
||||
|
||||
- The first hub/evidence commit went out after my secret-scan script crashed (it looked for last night's shredded
|
||||
files). Scanned right after: 5 125 files, 7 secrets, **0 hits**, control 1. Nothing leaked.
|
||||
- v0.283.0 shipped the Stop fix un-wired in production; the live test caught it; v0.283.1 is the second controller
|
||||
release this session (the one-release rule bent, reason in its CHANGELOG).
|
||||
|
||||
## Rows
|
||||
|
||||
Closed **R-719, R-720, R-721, R-722, R-727**. Fixed pending live proof **R-723**. Narrowed **R-724, R-725**. Opened
|
||||
**R-728** (customer created twice), **R-729** (no way to remove an off-site target). **Register 359 → 361 rows.**
|
||||
R-726 (a returning customer's orphaned repository) stays open — a NEW record does not meet it.
|
||||
|
||||
## Teardown
|
||||
|
||||
- **9202:** controller left on 0.283.1 (scratch; the floor does not reach it); the throwaway NAS target switched off
|
||||
through the form and then removed from `settings.json` with the controller stopped (R-729: no product path);
|
||||
`glance` removed with its data; paperless-ngx running; the off-site switches back OFF.
|
||||
- **Demo boxes:** controller 0.283.1 by the floor, agent 0.138.0 by signed jobs — nothing else.
|
||||
- **ep0:** only the three archives of decision 51. **Hub:** floor 0.283.1 (MinAgent 0.131.0 declared); hub 0.126.0.
|
||||
- **Tester-2 / Cloudflare:** read only. DooPlex: pushes, builds, the hub deploy, signing.
|
||||
@@ -0,0 +1,90 @@
|
||||
# REPORT — 2026-09-29 afternoon: the setup gate on the other 30 apps; "Done" asks the app first; open sign-up closed after the first admin; claper's password never in code
|
||||
|
||||
Architecture read first: `09` §3 decisions 45–47, `01-topology-and-trust.md` §5, `audits/login-gate-2026-09-29/B/B-VERDICT.md`,
|
||||
`app-catalog-felhom.eu/FIRST-ADMIN.md`, rows R-707, R-711, R-713. Controller **v0.281.0** (one release). Floor 0.281.0;
|
||||
both demo boxes run it. Catalog `6faf432`. Evidence: `documentation/audits/gate-rollout-2026-09-29/`
|
||||
(0 = Part 0, A, B = per-app, C = sign-up, D = claper, redproofs).
|
||||
|
||||
## The Parts
|
||||
|
||||
| Part | Step | State | Note |
|
||||
|---|---|---|---|
|
||||
| — | decision 47 recorded first | done | `09` §3 + `01` §5 before any work |
|
||||
| 0 | installed app never gated by a catalog change | done — **holds** | 9202: vikunja installed ungated and set up, then the drill catalog added `setup_gate: true`, synced, two loop ticks, a controller restart: still answered strangers, no gate file, no record (`0/P0-1`, `P0-2`). Code: the gate is set only in `DeployStack`. Pinned by `TestSignupBlock_NeverOnAnAppThisBoxDidNotGate` (RP23). After the live push, the demo boxes' installed class-4 apps (adventurelog, docmost, opengist, romm; opengist) got no gate and no block (`0/P0-3`) |
|
||||
| A1 | "Done" asks the probe first | done | refuses while the app says not done and when it cannot be read (409 + the sentence). Live: zipline pressed before its setup → 409, twice (`A/`, `B/B2-zipline-probe-before.txt`) |
|
||||
| A2 | confirm for an app without a probe | done | in-page `felhomConfirm` with the sentence (native confirm is banned by a gate) |
|
||||
| A3 | tests + red-proofs | done | RP17, RP18 |
|
||||
| B | the gate on the other 30 | done — **28 gated (25 fully proven, 3 opening not provable here), 2 not gated** | per-app table below |
|
||||
| C1 | spike: how sign-up can be closed | done, **mechanism changed** | opengist, wishlist: only in their own admin settings; vikunja: an env, but then no way to add a user but its CLI. Chosen: a box-side block of the app's own sign-up address + a household 15-minute window (CC-unattended decision, `09` §3 decision 47) |
|
||||
| C2 | per app | done | 11 apps let a stranger sign up after the setup → blocked and refused; 11 more refuse by themselves; window proven (gitea, calcom, vikunja) and closes again (gitea, calcom) |
|
||||
| C3 | apps that cannot close sign-up | **1: wanderer** | not gated at all (R-714) — STATUS asks |
|
||||
| D | R-713 | done | code-bound values refused; `${NAME|base64}`; claper proven live with a typed password holding `"` and `#{` (default refused, typed signs in). RP24 |
|
||||
|
||||
**Recommendation not followed, one line why (standing rule 4):** the ruling says "only the admin adds people, from the
|
||||
app's own user page"; for apps with no such page (opengist, vikunja, termix, sparkyfitness, adventurelog) the page
|
||||
says to open sign-up for 15 minutes instead — the only way a family member can join those apps at all.
|
||||
|
||||
### Part B / C — one row per app (9202, controller 0.281.0, drill catalog)
|
||||
|
||||
| app | gate proven (stranger refused · household reached setup · opened · answered after) | opened by | sign-up after the setup |
|
||||
|---|---|---|---|
|
||||
| actualbudget | yes | probe `data.bootstrapped` (M) | refused by the app |
|
||||
| komga | yes | probe `isClaimed` (M) | refused by the app |
|
||||
| jellyfin | yes | probe `StartupWizardCompleted` (M) | refused by the app |
|
||||
| romm | yes | probe `SYSTEM.SHOW_SETUP_WIZARD` (M) | refused by the app |
|
||||
| zipline | yes | probe `/api/server/public firstSetup` (M) | refused by the app (`userRegistration: false`) |
|
||||
| termix | yes | probe `setup_required` (M) | **was open → blocked** |
|
||||
| emby | yes | button | refused by the app |
|
||||
| navidrome | yes | button (no JSON status) | refused by the app |
|
||||
| ghost | yes | button (status is a list — R-715) | refused by the app |
|
||||
| home-assistant | yes | button (status is a list — R-715) | refused by the app |
|
||||
| gitea | yes | button | **was open → blocked** |
|
||||
| docmost | yes | button | no public sign-up |
|
||||
| calcom | yes | button | **was open (a stranger's account was created) → blocked** |
|
||||
| tandoor | yes | button | "Sign Up Closed" by the app |
|
||||
| gramps-web | yes | button (its status answers 405 — R-715) | answered 500 → blocked anyway |
|
||||
| adventurelog | yes | button | **was open → blocked** |
|
||||
| homebox | yes | button | **was open → blocked** |
|
||||
| papra | yes | button | **was open → blocked** |
|
||||
| sparkyfitness | yes | button | **was open → blocked** |
|
||||
| vikunja | yes | button | **was open → blocked**; window let a family member in |
|
||||
| opengist | yes | button | **was open → blocked** |
|
||||
| wishlist | yes | button | **was open → blocked** |
|
||||
| radarr, sonarr | yes | button | single user; their API key was public BEFORE the setup — the gate hides it |
|
||||
| recipe-importer | yes | button | our own app, open until a password is set — the confirm says so |
|
||||
| seerr | stranger refused, household reached setup | **opening not proven** (needs a media server) | not measured |
|
||||
| outline | stranger refused, household reached setup | **opening not proven** (needs e-mail / SSO) | not measured |
|
||||
| rallly | stranger refused, household reached setup | **opening not proven** (e-mail) | not measured |
|
||||
| wanderer | **not gated** | — | open (R-714) |
|
||||
| plant-it | not installable (`lifecycle: abandoned`); the template carries the gate | — | — |
|
||||
|
||||
## Claims in the brief that turned out wrong (or right), named
|
||||
|
||||
- **"A catalog change never gates an installed app"** — **right** (measured on 9202 and read on both demo boxes).
|
||||
- **"14 of the 34 have a probe"** — **wrong**: 9 have a probe that works (3 from the morning, 6 today); 3 more have a
|
||||
status the box cannot read (ghost, home-assistant — lists; gramps-web — 405, R-715); zipline's upstream route was
|
||||
wrong (`/api/setup` answers 403 after the setup; `/api/server/public` works).
|
||||
- **"The first admin is the first registered user on opengist and wishlist"** — **right** (measured: the household's
|
||||
sign-up became the admin; after it, a stranger could still sign up).
|
||||
- **"Each app in R-711's list can close sign-up"** — **wrong** as a per-app switch: opengist and wishlist keep it only
|
||||
in their admin settings, vikunja only as a start-up env; and wanderer cannot be closed at all today. The box-side
|
||||
block closes all but wanderer.
|
||||
- **"An env switch needs a restart"** — **right** for vikunja (read at start); not used — the block needs no restart.
|
||||
- **"Register 346 rows"** — was **350** at the start of this session (the morning session ended at 350).
|
||||
|
||||
## Also found
|
||||
|
||||
- A probe that never flips blocks the household's press (fail closed; measured on gramps-web) → R-715.
|
||||
- calcom created a stranger's account after the setup (measured) — closed.
|
||||
- Apps installed before today keep their open sign-up (demo boxes only) → R-716, needs an operator word.
|
||||
|
||||
## Rows
|
||||
|
||||
Closed: R-707, R-711, R-713. Opened: R-714 (wanderer), R-715 (probe shapes), R-716 (installed apps' sign-up, operator).
|
||||
**Register 350 → 353 rows.**
|
||||
|
||||
## Teardown
|
||||
|
||||
Machines: 9202 — every test app removed through the product (the scratch drive keeps some app folders, R-442's
|
||||
refusal as before); no gate or block file left; back on the live catalog; the drill catalog reset to live `main`.
|
||||
Demo boxes — read only (plus the floor). Host: nothing. Hub: floor 0.281.0. ep0: untouched.
|
||||
@@ -0,0 +1,61 @@
|
||||
# REPORT — 2026-09-28: weekly golden + Day-0 vouch, kept data from the off-site copy, claper/calcom, demo-hp space, night read, first live off-site restores
|
||||
|
||||
Brief: "the weekly golden, the Day-0 vouch, use my kept data from the off-site copy, two more PostgreSQL fixtures,
|
||||
demo-hp's restore-test space, the night watch, and the first live off-site restore" (revised 2026-09-28).
|
||||
Architecture read before any claim: `07-backup-architecture.md` §6.5, §6.6; `06-offsite-connectivity.md`;
|
||||
`03-host-agent.md` (restore-test storage); `09-update-architecture.md` §3 decisions 35–42.
|
||||
**Operator change mid-session (15:14):** "finish today" → Part E and D2 were done in the day, not overnight (below).
|
||||
|
||||
## The Parts
|
||||
|
||||
| Part | Step | State | Note |
|
||||
|---|---|---|---|
|
||||
| A | 1 read the manifest (rollback) | done | agent 0.132.0, golden 0.258.0, min_agent 0.131.0 |
|
||||
| A | 2 bake golden 0.276.0 | done, **with a deviation** | attempt 1 picked the `arm64` template; stopped by CC before `pct create` finished, nothing published. Attempt 2: all markers, leak grep 0 (control 1), 404 → 200, anonymous download sha equal, VM back to `virgin` |
|
||||
| A | 3 vouch (3 fields) | done | R-120 gate refused golden 0.258.0 first (negative control); `agent=0.137.0 golden=0.276.0 min_agent="0.131.0"` read back |
|
||||
| A | 4 Day-0 test install | done | installer 1.28.0, agent + golden fetched through the manifest and sha-verified, controller 0.276.0 healthy, claim gate armed; not claimed; scratch customer deleted by the hub's own cascade |
|
||||
| A | 5 golden-currency gate | done | green, not waived (record `documentation/tests/golden-0.276.0-2026-09-28/`); waiver file left as it is (to 2026-10-04) |
|
||||
| A | 6 agent CHANGELOG line | done | felhom-agent `5c68c86`; no floor move in Part A |
|
||||
| B | 1–3 code, tests, red-proofs | done | controller **v0.277.0**; RP1–RP3 |
|
||||
| B | 4 live Tier 1/Tier 2 regression on 9202 | Tier 1 done; **Tier 2 not runnable** | 9202 has one drive |
|
||||
| B | 5 release + floor | done | floor 0.277.0 at 10:11 (before 01:30), both boxes healthy |
|
||||
| B | — | **changed: a second release, v0.278.0** | R-704 blocked Part E; floor 0.278.0 at 15:36 |
|
||||
| C | calcom | done | memory fix first (R-703, 768M OOM at every start → 1536M); PG 16 → **18** proven on bench + box; catalog `037f956` |
|
||||
| C | claper | done | PG 16 → **17** proven on bench + box; catalog `4a249b9`; found R-702 (default admin) |
|
||||
| D | 1 R-701 (a) | done — **not enough** | 20.3 → 26.6 GiB free; 31 needed; 14:13 cycle refused again |
|
||||
| D | 2 night read | **changed** | the night 27/28 was read (logs, hub, agent journals), not tonight's; demo-felhom's off-site/update legs not readable |
|
||||
| E | 1 setup | done | nextcloud on demo-hp, seeded, joined the off-site copy |
|
||||
| E | 2 (a) off-site restore | done — **changed timing** | the snapshot came from the page's "run now" (15:40), not the night; seed back, marker gone |
|
||||
| E | 2 (b) kept data from off-site | done | choice named "távoli mentés, 2026-09-28 15:40"; seed + files back |
|
||||
| E | 3 teardown | done | app removed with its data; app list equal to before; verification copy deleted by the product |
|
||||
|
||||
## Claims in the brief that turned out wrong (or right)
|
||||
|
||||
- **"0.276.0's MinAgent is 0.131.0"** — right.
|
||||
- **"The R-120 gate accepts golden 0.276.0"** — right (it refused 0.258.0 and accepted 0.276.0).
|
||||
- **"The Day-0 test install leaves no hub record"** — **wrong.** It left a host, a vaulted break-glass credential, a claim code, a WireGuard peer (10.77.0.5, synced toward ep0) and reports. The customer DELETE cascade removed the hub side; the ep0 peer removal was **not observed** (ep0 untouched by rule).
|
||||
- **"The pairing code is shown"** — **wrong for this path.** The one-liner install shows no pairing code (that is the ISO path); what shows is the controller's claim gate (`dashboard not yet claimed`).
|
||||
- **"`pct fstrim` frees enough for R-701"** — **wrong.** It freed 6.3 GiB; the pool is 53.9 GiB and the guest holds ~26 GiB, so 31 GiB free cannot be reached by trimming.
|
||||
- **"paperless-ngx runs on demo-hp"** — right; it converted to PostgreSQL 18 in the night 27/28 (04:22 CEST, rows equal, 0 documents).
|
||||
- **"An app joins the off-site copy by a per-app switch"** — right (`POST /backup/offbox/toggle`, `app_backup.<app>.offbox`).
|
||||
|
||||
## Evidence
|
||||
|
||||
- Golden, vouch, Day-0, D1, D2: `documentation/audits/evidence-golden-0276-2026-09-28/`
|
||||
- Part B + E: `documentation/audits/kept-offsite-2026-09-28/` (redproofs/, live/, E/)
|
||||
- Part C: `documentation/audits/pg-calcom-claper-2026-09-28/`
|
||||
|
||||
## Rows
|
||||
|
||||
Opened: R-702 (claper default admin, P1), R-703 (calcom OOM — closed the same day), R-704 (leftover holds — fixed in
|
||||
0.278.0, WATCHING), R-705 (no "run the night now"), R-706 (verification copy survives removal). Closed: R-691, R-703.
|
||||
Updated: R-463, R-687, R-701. Register 337 → 342 rows. `unproven.py --summary`: not walked 35 of 55 (unchanged; the
|
||||
capability map was not edited).
|
||||
|
||||
## Teardown, three layers
|
||||
|
||||
- **Machines:** drill VM reverted to `virgin` (qemu gone); bench LXC 9401 created and destroyed twice; nextcloud,
|
||||
calcom, claper removed through the product on 9201/9202; 9202 pointed back to the live catalog; drill catalog = live.
|
||||
- **Hosts:** demo-hp `local-lvm` trimmed (kept); template cache files removed; no storage added.
|
||||
- **Hub:** scratch customer `drill-g0276` deleted by its cascade; manifest now golden 0.276.0 / agent 0.137.0; floor
|
||||
0.278.0. nextcloud's off-site snapshots stay in demo-hp's repository (removal never touches off-site history, R-474).
|
||||
@@ -0,0 +1,119 @@
|
||||
# REPORT — the last three things between an English household and their box
|
||||
|
||||
**R-596, R-598 (controller v0.259.0) · R-597 (hub v0.119.0).** 2026-09-21.
|
||||
Written as `REPORT-<topic>.md` because `REPORT.md` is shared in this repo.
|
||||
|
||||
---
|
||||
|
||||
## 1. Claims in the task that turned out wrong — named first
|
||||
|
||||
| the claim | what is true |
|
||||
|---|---|
|
||||
| "**Sixteen** Hungarian literals reach the claim page" | **Fifteen** sites, **nine** distinct messages (four repeat). One of the fifteen, `data["Title"]`, is **DEAD** — `claim.html` is standalone with its own bundle-backed `<title>`, and `.Title` is read only by `layout.html`. Deleted, not translated. L523 (operator stdout) and L563 (the `claim_lockout` event, whose customer copy the hub already localises) are wire copy and correctly untouched. **Fourteen live sites converted.** |
|
||||
| "`backup_handlers.go` (**12** Hungarian literals)" | **Nine** are code; three are Hungarian inside comments. `backup_target_offer.go`'s ten is right. |
|
||||
| "the recovery code (10 words — **find its caller**)" in the hub | **The hub does not mint it.** `felhom-agent`'s `internal/escrow` does, from the **EFF large wordlist** — so the recovery code **has always been English**, ten words, ≈129 bits. No work needed, none done, and **no row opened**: a second definition of that secret here is exactly the cross-repo drift `backupTargetAbsentText` already demonstrates. |
|
||||
| "the mail says 'three words' … `strings.Count(code,"-")+1`" | **No claim mail states a count.** They say `Setup code: %s`. The only count wording in the product was the **bind page's** passphrase hint ("five words"); its English half is now count-free, Hungarian unchanged. |
|
||||
| "§8's phone-safe filter: no two words differing by one letter in the first six" | **Measured, then declined.** It removes **5270 of 7772** words — 68%, 12.92 → 11.29 bits/word — and would make this list stricter than the one the product already uses for the code a household writes on paper during a disaster. Reason and measurement recorded in source; **operator may reverse.** Replaced by an assertion: every word is 3–9 lower-case ASCII letters, no digit, no separator. |
|
||||
| "request a reset code for **the demo customer (`en`)**" | **There is no English customer on this hub.** All five are `hu`. A scratch customer was created, proven, and deleted. |
|
||||
| "demo-hp guest 9201" | **Guest 9201 is on `felhom-pve`.** I also claimed demo-hp was offline — **that was MY error, withdrawn the same day (R-601)**: the box had been up four and a half weeks and reporting; both of my SSH routes pointed at stale addresses. |
|
||||
| "`customer.language` reaches the anonymous claim page" | **TRUE**, verified at source before any edit and now **pinned by a test** rather than assumed. |
|
||||
| "the box checks a hash and needs no change" | **TRUE**, and pinned by `TestClaimAcceptsAnEnglishWordCode`. |
|
||||
| "29 633 words"; the line numbers | **Right.** (29 634 lines, 29 609 after dedup.) Every cited line number was accurate. |
|
||||
|
||||
---
|
||||
|
||||
## 2. What shipped
|
||||
|
||||
**Controller 0.259.0** — the claim page's fourteen sites through `s.msg`; the backup page's three
|
||||
protection constants become KEYS, with `degradedMessageFor` returning the key so the decision stays
|
||||
language-free and in one place; `buildTierViews` / `backupTargetLabel` / `loadGuestBackup` take the
|
||||
reader's language. 23 new keys in both bundles, all listed for the Go-parity gate.
|
||||
|
||||
**Hub 0.119.0** — `english.txt` (EFF large, CC BY 3.0 US, provenance in source);
|
||||
`RandomPassphraseFor(lang, use)` choosing list **and** count together; all four callers pass a
|
||||
language; the English bind hint is count-free.
|
||||
|
||||
**felhom.eu** — the guide's three quoted messages corrected; **`guide_quote_gate.py`** binds them to
|
||||
the controller's English bundle (nothing did, so the guide would have gone on quoting Hungarian
|
||||
after the fix), with seven decoys; `05-hub-architecture.md` §15.6; `10-localisation.md` §10.6c.
|
||||
|
||||
---
|
||||
|
||||
## 3. Evidence
|
||||
|
||||
| check | result |
|
||||
|---|---|
|
||||
| controller: build / vet / full suite | green |
|
||||
| hub: build / vet / full suite | green |
|
||||
| `controller_gates.py --fast` (17) | all OK |
|
||||
| `repo_gates.py --fast` (15, incl. the new `guide-quote`) | all OK |
|
||||
| `i18n_go_parity.py` | OK — 718 keys byte-for-byte against the frozen base |
|
||||
| `i18n_missing_gate.py` | English missing **0** (ceiling 0); Hungarian formal 18 (ceiling 18) |
|
||||
| decoys: felhom.eu 16/16, controller 23/23 | all convict |
|
||||
| `unproven.py --summary` | **no number moved** — still 35 of 55 not-walked |
|
||||
|
||||
**Four red-proofs, each seen failing:**
|
||||
1. One added full stop in `hu.json` → the go-parity gate named both sides.
|
||||
2. The wrong-code Hungarian literal restored → the English test convicted **twice** (English absent AND Hungarian present).
|
||||
3. The English setup code set to 3 words → the entropy test named the 38.77-vs-44.56 gap.
|
||||
4. The engine reverted to `RandomPassphrase(3)` → the wiring test convicted on the word count **and** on the non-ASCII code.
|
||||
|
||||
**Live, on real systems:**
|
||||
- Claim page, guest 9201, through the **`felhom_lang` cookie** — `en`: **"Wrong or expired code"** (the drill's own screen), "Invalid form — reload the page.", "Too many attempts — try again in 15 minutes."; `hu`: the byte-identical Hungarian for each.
|
||||
- The **lockout proved itself unasked**: Hungarian attempts locked out the English request from the same source, demonstrating live that the counter is per source, not per language.
|
||||
- Backups page: `Local storage (felhom-backup)` / `Backup server – separate hardware (PBS)` against the Hungarian.
|
||||
- **The setup mail, one day apart in the same inbox**: 2026-09-20 `képző-szkítia-ásatás` → 2026-09-21 four plain-ASCII English words.
|
||||
- Owner passphrase from the hub's own store: `en` **6 ASCII words**, `hu` **5 accented** — shape only, values never read out.
|
||||
|
||||
---
|
||||
|
||||
## 4. What I did NOT do, and why
|
||||
|
||||
- **I did not complete a password reset on guest 9201.** The task asked for it. To get an *English*
|
||||
code for that box I would have had to change the **box's own** language setting, because
|
||||
`CustomerLanguage` prefers the **reported** language over the config's — so flipping the hub's field
|
||||
alone would have produced a Hungarian code and proved nothing. Changing a live box's household
|
||||
setting to stage a test, and rewriting its password hash (this repo records a session that did
|
||||
exactly that and lost the original bytes), buys little: the acceptance path is untouched by this
|
||||
release and is pinned by `TestClaimAcceptsAnEnglishWordCode`. The refusals — which is what R-596
|
||||
was about — were walked live in both languages, including the wrong-code answer that stopped the drill.
|
||||
- **The two Backup-page warnings were not walked live.** Guest 9201 is healthy and a healthy box
|
||||
renders none, by design. Producing either state means un-assigning a live backup target. They are
|
||||
covered by render tests through the real handler.
|
||||
|
||||
---
|
||||
|
||||
## 5. Rows
|
||||
|
||||
**Closed:** R-596, R-597, R-598 — each with what it actually turned out to be, not just "fixed".
|
||||
**Opened:** R-602 (a live probe that uses a cookie on a signed-in page reports a fixed defect as
|
||||
unfixed), R-603 (an English string with an apostrophe silently never matches a rendered page),
|
||||
**R-604 (a per-customer floor override silently excludes a box from every global raise — demo-hp had
|
||||
missed four)**.
|
||||
**Withdrawn as false the same day:** R-601 ("demo-hp is unreachable"). The operator looked at the hub
|
||||
and said it was online; it was, and had been for four and a half weeks. Both of my routes pointed at
|
||||
stale addresses — one at a tailnet peer for a box with no tailscale installed, one at an address the
|
||||
box left behind at a reprovision. **The hub had carried the right address in every report.** The
|
||||
lesson kept in the row: the standing rule says a "no access" claim must list what was tried; it does
|
||||
not say the list makes the claim true. Six failures against one wrong assumption is one failure.
|
||||
|
||||
---
|
||||
|
||||
## 6. The verdict
|
||||
|
||||
**Nothing known now stands between an English-speaking tester and their box.**
|
||||
|
||||
That is deliberately not the same sentence as *"the walk passed"*. The three blockers the drill found
|
||||
are closed and each is proven on a live system — but **the hour has not been re-walked end to end by
|
||||
a stranger on a fresh install**, and this project's own rule, written into the recovery-journey row,
|
||||
is that **fixes are not a journey**. The next English walk is what turns this into a green row; it is
|
||||
also the walk that would exercise the two backup warnings, and it wants a one-drive machine.
|
||||
|
||||
**The fleet floor is raised to 0.259.0** (operator asked, same session), `min_agent` 0.131.0
|
||||
declared — above the vouched golden 0.258.0, so the declaration carries it (R-472). **Both live boxes
|
||||
run 0.259.0.** demo-hp took it **by itself in under four minutes** once its stale per-customer
|
||||
override was cleared, and its claim page then answered **"Wrong or expired code"** in English — the
|
||||
floor delivered the FIX to a box nobody hand-deployed, which is the only thing that shows a raise
|
||||
worked. Evidence: `audits/i18n-closing-2026-09-21/floor-raise-0.259.0.md`.
|
||||
|
||||
**Needs the operator: nothing from this session.**
|
||||
@@ -0,0 +1,47 @@
|
||||
# REPORT — 2026-09-30 (late afternoon): immich's first start, the cause and the fix; STATUS golden line; R-730, R-731
|
||||
|
||||
| part | outcome | why / where |
|
||||
|---|---|---|
|
||||
| A cause | **done** — up to 9 concurrent geodata INSERTs need ~400 MB anon + ~170 MB touched shared_buffers; 512M fits only with swap. Control pair: swap alone → pass, limit alone → pass, `shared_buffers` alone → still killed | `audits/immich-first-start-2026-09-30/A-cause.md` |
|
||||
| B fix + proof | **done** — catalog `56c4888`: v3.2.4 + `immich-postgres` 768M, `mem_limit` 4480M. Fresh installs, swap OFF: bench ×2 and 9202, 0 kills, anon ≤ 54 %. Step: bench proven (10-min watch, 0 kills), box done 58.5 s, read back, running limit 768M | `bench/`, `box/` |
|
||||
| C STATUS + register | **done** — the golden line corrected; R-732 closed | `STATUS.md` |
|
||||
| D1 R-730 | **done** — the ISO build refuses a dirty/unpushed tree; red-proof run; `iso-v<version>` | `scripts/iso/test/clean-tree.sh` |
|
||||
| D2 R-731 | **done (narrowed)** — gitea 28.0.0 is GA; mariadb 13.0 is a short-term line; the standing shape-switch control NOT built | `D/D2-release-checks.txt` |
|
||||
|
||||
No controller, agent or hub release. No bake (none is due).
|
||||
|
||||
## Claims in the brief, checked
|
||||
|
||||
- **"the limit is 512M and the header says 256M"** — right (and `mem_limit` 4096M was already 128 MB under the sum of the four limits).
|
||||
- **"no `shm_size`"** — right; `/dev/shm` 64M, 1.1M used — not involved, so none was added.
|
||||
- **"the image sizes memory from host RAM"** — **wrong**: `shared_buffers` 512MB and `work_mem` 16MB are FIXED in the image's own
|
||||
`postgresql.conf`; the rest are PostgreSQL defaults.
|
||||
- **"bench and box differ by host RAM"** — **wrong**: both on demo-hp. They differ by **swap** (box 512 MiB, bench 0), proven by giving
|
||||
the bench swap alone.
|
||||
- **"no bake is due"** — right (the gate: `newest golden baked 0.283.1`, OK).
|
||||
- **"the fix reaches installed apps only through the v3.2.4 step"** — right, and more: the step itself RE-RUNS the geodata import
|
||||
(228 294 → 228 571 places), so publishing v3.2.4 without the fix would have killed the database during the update.
|
||||
|
||||
## Per-box cost
|
||||
|
||||
**+256 MB** on immich's database limit (512M → 768M); the declared `mem_limit` goes 4096M → 4480M (+384, of which 128 corrects an
|
||||
old undercount). Only boxes with immich.
|
||||
|
||||
## What an installed immich gets, and when
|
||||
|
||||
An immich on 3.2.2 keeps 512M until its next guarded Update, which moves it to v3.2.4 with 768M in one step (measured on 9202:
|
||||
running limit 805306368 after). The night leg takes that step only when a fresh whole copy exists (the step carries
|
||||
`files_may_change` — R-734); otherwise the household's button does. A 3.2.2 immich's own first start is already behind it.
|
||||
|
||||
## Rows
|
||||
|
||||
Register **364 → 366**. Closed: R-732, R-730. Narrowed: R-731. Notes: R-676. Opened: **R-733** (the bench has no swap, the boxes
|
||||
do; a customer guest's swap is not recorded), **R-734** (immich's `.immich` markers set `files_may_change`).
|
||||
|
||||
## Teardown (three layers)
|
||||
|
||||
- **Machine:** 9202 — immich removed through the product (no volume left; its drive folder kept by R-442's refusal, as before),
|
||||
`controller.yaml` restored (live catalog, read back), swap back at 512 MiB (it was 0 for the one fresh-install proof).
|
||||
- **Host:** bench LXC 9401 destroyed, template removed, host temp files removed; drill repo reset to the live `main` (`56c4888`),
|
||||
image lines identical.
|
||||
- **Hub:** not touched.
|
||||
@@ -0,0 +1,79 @@
|
||||
# REPORT — strangers and lockouts (R-752), the one address behind the tunnel (R-753), the registry (R-750) — 2026-10-01 afternoon
|
||||
|
||||
Evidence: `documentation/audits/lockouts-2026-10-01/` (A, B, C, T, tools).
|
||||
Architecture read: `01-topology-and-trust.md` §5, §7; `09` §3 decisions 45–47, 57; `06` (the tunnel is not described
|
||||
there). Baselines (live Gitea ~10:55 CEST): controller `c1b123c64955`, agent `d766666ff8cf`, felhom.eu `a6a9f0b2458e`,
|
||||
catalog `83636352ea10` — all matched. Register 387 rows; highest R-752; last decision 57.
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | done / not done / changed | why |
|
||||
|---|---|---|
|
||||
| Operator note (decision 57 kept) | **done** — `09` §3 + CONTEXT | first |
|
||||
| **A — the client address** | **done — measured; no box-wide fix** (R-753) | trusting cloudflared would pass a client-written leftmost address; a single-address rewrite needs a plugin |
|
||||
| A1 two outside addresses | **changed — one** (DooPlex 37.191.56.193; no IPv6 here) | the "same address for everyone" result does not depend on a second one |
|
||||
| A1 demo-hp | **done, read only** — two GETs of a 404 path, then the logs | — |
|
||||
| **B1 calibre-web** | **measured; not fixed — operator decision** (STATUS) | no knob for the daily lock; both fixes cost the household |
|
||||
| **B2 wger** | **done — decision 58**, catalog `82fff32`; control + two fix runs on 9202 | the first fix (15 min) proved every try during a lock restarts it; changed to 5 min |
|
||||
| **B3 Grafana** | **done — decision 60: no change** (5.0 min measured; trickle measured) | already short |
|
||||
| **B4 BookStack** | **done — decision 59: no change** (1.0 min measured) | already short; `APP_PROXIES` would not help through the tunnel |
|
||||
| B installed apps | **done** — measured on 9202 | see below |
|
||||
| **C — the registry** | **done, read only** — cause found (R-750 answered) | — |
|
||||
| **D — release / golden** | **not done — not needed** | Part A built nothing |
|
||||
|
||||
## Claims in the brief that turned out wrong (or right)
|
||||
|
||||
1. **"Apps see traefik's address for every client"** — half right. The app's TCP peer is traefik, but `X-Forwarded-For`
|
||||
carries cloudflared's container address through the tunnel (the same for everyone) and the REAL address from the LAN.
|
||||
Apps that read it (calibre-web's ProxyFix, `TRUSTED_PROXY_COUNT` 1) still see one address for every tunnel visitor.
|
||||
2. **"calibre-web has no env switch"** — right for the limiter (a database setting, `config_ratelimiter`); it has an env
|
||||
`TRUSTED_PROXY_COUNT`, irrelevant here (the login limit is keyed on the user name).
|
||||
3. **"BookStack's 60 s is hard-coded"** — right (`ThrottlesLogins.php:82` 5 tries, `:90` 1 minute).
|
||||
4. **"A Gitea cleanup rule removed the old versions"** — wrong. No rule exists; a manual prune script did (HM-024).
|
||||
5. R-752's own claims: calibre-web "up to a day" — **right** (measured: still locked 2 min after the minute window; only a
|
||||
restart cleared it). My own earlier guess that calibre-web's OPDS door had no limit — **wrong**: 3/minute per name.
|
||||
Grafana "a slow trickle keeps it closed indefinitely" — **not as measured**: the household got in once the burst aged
|
||||
out, and a success resets the count. wger "everyone at once" — **right** (measured).
|
||||
6. `01` §7 "cloudflared runs on the host" — **the build differs**: it runs in the guest (R-754).
|
||||
|
||||
## Part A — the answer
|
||||
|
||||
| path | the app's TCP peer | X-Forwarded-For / X-Real-Ip | the real client is in | forgeable? |
|
||||
|---|---|---|---|---|
|
||||
| tunnel | traefik | cloudflared's container — same for every visitor | `CF-Connecting-IP` only | XFF no (traefik drops it); `CF-Connecting-IP` not through the tunnel, **yes from the LAN** |
|
||||
| LAN | traefik | the real LAN address | XFF / X-Real-Ip | no |
|
||||
|
||||
## Part B — per app (9202, the public name, a stranger through traefik)
|
||||
|
||||
| app | setting (pinned tag) | measured before | fix | after |
|
||||
|---|---|---|---|---|
|
||||
| wger 2.7 | `settings/main.py:268-272` (`AXES_*` env), `settings_global.py:485` reset-on-failure True | 10 wrong → the second member locked too | username, 5 min, DB handler (decision 58) | other member fine; admin in at 7.5 min with one retry; wrong still refused |
|
||||
| BookStack 26.09.1 | `ThrottlesLogins.php:66,82,90` | locked 1.0 min | none (59) | — |
|
||||
| Grafana 13.2.3 | `login_attempt.go:14,65-85`, `defaults.ini:498-507` | locked 5.0 min; trickle: in after the burst aged | none (60) | — |
|
||||
| calibre-web-automated v4.0.8 | `cps/web.py:2218-2219` (3/min, 40/day per name), `cps/main.py:75` (OPDS 3/min) | form 1.2 min; 40 wrong in 14 min → refused 2+ min later; restart cleared | **operator** | — |
|
||||
|
||||
**What an installed app gets, and when (measured with wger):** a settings-only change reaches the app's stack file at the
|
||||
next catalog sync (when its images equal the catalog's; ≤ 15 min); the RUNNING app keeps the old value until the next
|
||||
`compose up -d` — the app page's Restart or Start (measured: the env changed exactly at Restart), an Update, or a
|
||||
backup's restart of the app (`backup.go:972`, read, not measured). An app pinned to an older version than the catalog
|
||||
gets nothing until its Update (the frozen render, `09` §5.4).
|
||||
|
||||
## Part C — the registry (read only)
|
||||
|
||||
`package_cleanup_rule` empty; Gitea logs only to the console and the pod started 2026-08-23, so August logs are gone. The
|
||||
cause is recorded in homelab-manifests HM-024: `gitea-image-prune.sh --all --keep 7 --apply --reclaim` the night of
|
||||
2026-08-22/23 (the Gitea volume was full). Nothing schedules it. STATUS carries the decision (keep / a written rule, pick: a rule).
|
||||
|
||||
## Rows
|
||||
|
||||
**387 → 390.** Opened R-753 (one address behind the tunnel), R-754 (`01` §7 vs the build), R-755 (wger on runserver).
|
||||
Narrowed R-752. Answered R-750 (waiting on the operator). Closed none.
|
||||
|
||||
## Teardown
|
||||
|
||||
- **Machine:** 9202 back on the live catalog (`repo_url` read back), the same six containers as at the start; the apps
|
||||
this session installed (bookstack, grafana, calibre-web, wger ×5) removed through the product — calibre-web's drive
|
||||
data kept because the remove refused the drive path (R-442's fail-closed rule; its folder predates today); the echo
|
||||
container and both probe images removed. Drill catalog reset to live (`82fff32`).
|
||||
- **Host:** demo-hp untouched except two read-only GETs through its tunnel and log reads; `pct list` unchanged.
|
||||
- **Hub:** nothing. **Gitea / DooPlex:** read only (one READ ONLY database transaction, config and log reads).
|
||||
@@ -0,0 +1,70 @@
|
||||
# REPORT — 2026-09-29: demo-hp's two open default logins closed; the setup gate SPIKED, PASSED and BUILT; every hard-coded default replaced
|
||||
|
||||
Architecture read first: `01-topology-and-trust.md` §5 (trust boundaries — it had no statement of who may reach an app;
|
||||
now it does), `04-control-plane-authorization.md` (control plane only — nothing about app reachability), `09` §3
|
||||
decisions 45–46, `app-catalog-felhom.eu/FIRST-ADMIN.md`. Controller **v0.280.0** (one release). Floor 0.280.0; both demo
|
||||
boxes run it. Evidence: `documentation/audits/login-gate-2026-09-29/` (A, B, C, D).
|
||||
|
||||
## The Parts
|
||||
|
||||
| Part | Step | State | Note |
|
||||
|---|---|---|---|
|
||||
| — | rulings recorded first | done | decision 46 + the demo-login ruling in `09` §3 before any work |
|
||||
| A1 | read-only: defaults still work | done | bookstack and calibre-web: default signed in (302 → /), wrong refused (A1) |
|
||||
| A2 | change through the app's own route | done | bookstack `artisan bookstack:create-admin --initial`; calibre-web `cps.py -s` with a special-character password. Default refused, new signs in, wrong refused, through each app's own login form |
|
||||
| A3 | save in `~/.config/credentials` | done | four lines appended in the file's format (`DEMO_HP_BOOKSTACK_USER/_PW`, `DEMO_HP_CALIBRE_USER/_PW`), file 0600, read back equal; values in no evidence, commit or log (value scan before every evidence commit) |
|
||||
| A4 | pages stop warning | done, **changed** | `known_login.go` could NOT learn of a manual change, and bookstack's page had ALREADY stopped warning while its default worked (R-710). Built "I changed it" in v0.280.0 (an honest household/operator record, not a faked after_install result) and pressed it on demo-hp: both sentences gone (A3) |
|
||||
| B1 | dashboard session on the app's address | done | the cookie is host-only — it cannot be seen there; a redirect handshake instead, nothing widened |
|
||||
| B2 | stranger / household in a browser | done | stranger: gate page or 401, never the app; household: 0.2 s, no extra step; one sign-in on a phone not signed in |
|
||||
| B3 | "setup done" probes | done | n8n and immich measured flipping; 14 of 34 have a probe (then 2 measured, 12 upstream), 20 need the button |
|
||||
| B4 | after the gate opens | done | immich's phone-app API (Bearer) and n8n's API unchanged; the gate stopped and removed |
|
||||
| B5 | cost | done | ~2 ms; controller down → gated apps 500 (closed); phone app first → 401 until the web setup |
|
||||
| B6 | exit test | **PASSED** | written before any build: `B/B-VERDICT.md` |
|
||||
| C1 | controller v0.280.0: the gate | done | written before the first start (a failed write refuses the install); probe loop 20 s; button; restart keeps it; restore keeps the record; kept data never gates. hu + en copy, informal, no "please"; parity green |
|
||||
| C2 | tests + red-proofs | done | `TestSetupGate_*` (stacks 9, web 4 + page 2), all green; RP1–RP12 each seen failing on an assertion |
|
||||
| C3 | live on 3–5 class-4 apps | done (4) | immich (phone app), n8n, audiobookshelf (probe), uptime-kuma (button): ~530 stranger polls during the installs, 0 app answers before each gate opened; the probes opened 3 gates ≤ 27 s after setup; the press opened the 4th; a controller restart kept the 4th closed and the household's pass valid |
|
||||
| C3 | restore does not re-gate | **not live** | proven by test only (`TestSetupGate_ARestoreKeepsTheRecord`): a per-app backup on 9202 needs a whole-box backup, which drills must not run (R-648) |
|
||||
| C4 | design record + decision 46 outcome | done | `01` §5 "who may reach an app, and through what"; `09` §3 decision 46 outcome |
|
||||
| C4 | floor | done | 0.280.0 after every live proof passed; both demo boxes delivered |
|
||||
| D1 | mealie, wger | done | `after_install`, password as `sys.argv[1]`; fresh-install proof: default refused, generated signs in, wrong refused. Found and fixed R-712 (wger refused every browser sign-in: CSRF) |
|
||||
| D2 | calibre-web | done | `generate: password:24:special` (server + install page; 2000 JS runs in node, 0 bad); `cps.py -s` as `abc`; fresh-install proof as D1 |
|
||||
| D3 | romm, zipline notes | done | removed (hu + en); first steps say create the admin; romm's page on demo-hp no longer warns |
|
||||
| D4 | R-708, R-709 | done | grafana `${…:?…}` (compose refuses empty/unset); password fields off the page with a reveal eye (live: the value is not in the HTML) |
|
||||
| D5 | tests + red-proofs + live | done | RP13–RP16; live on 9202 |
|
||||
|
||||
## Claims in the brief that turned out wrong (or right), named
|
||||
|
||||
- **"No class-4 app is reachable except through traefik"** — **right** (read: only crafty-controller publishes ports, and
|
||||
it is class 1). Nuance: wanderer publishes a SECOND host (its database admin) — a gate must cover every host an app's
|
||||
labels publish; the built gate does.
|
||||
- **"Most class-4 apps expose a setup-done status"** — **wrong**: 14 of 34 (now 3 measured, 11 upstream); 20 need the
|
||||
household's button.
|
||||
- **"The dashboard session can be checked on an app's subdomain without widening it"** — **wrong as stated**: the
|
||||
cookie is host-only and never reaches an app host. A redirect handshake (a 60-second, one-use, host-bound token
|
||||
minted on the dashboard's own host) does the check instead — and nothing is widened.
|
||||
- **"immich's phone app works unchanged after the gate opens"** — **right**, measured through its API (login → Bearer →
|
||||
`/users/me`, `/server/ping`, `/server/version`, all 200); the real phone app was not run.
|
||||
- **"`known_login.go` can learn of a manual password change"** — **wrong**: it knew only `after_install` records. And
|
||||
worse than the brief assumed: bookstack's page on demo-hp had already stopped warning while its default still worked
|
||||
(an absent record read as "not run yet" for ever — R-710). Fixed; proven live.
|
||||
|
||||
## Also found
|
||||
|
||||
- **The household's "Done" press trusts the household.** The uptime-kuma proof pressed it without doing the setup, and
|
||||
the app then answered anyone. The page tells the household to press after the setup; nothing checks it (no probe
|
||||
exists for uptime-kuma over HTTP). Recorded in decision 46's outcome.
|
||||
- **A security review of the drill commit** flagged mealie's and wger's commands (the password pasted into Python
|
||||
code). Fixed before the live catalog (argv). claper's Elixir command has the same shape → R-713.
|
||||
|
||||
## Rows
|
||||
|
||||
Opened: R-710 (closed the same day), R-711, R-712 (closed), R-713. Closed: R-708, R-709, R-710, R-712. Narrowed: R-707
|
||||
(30 of 37 left). **Register 346 → 350 rows.**
|
||||
|
||||
## Teardown
|
||||
|
||||
Machines: 9202 — the spike's container, file and two apps removed; the seven Part C/D apps removed through the product
|
||||
(immich, audiobookshelf and calibre-web kept their scratch-drive folders — R-442's refusal, as in earlier sessions);
|
||||
no gate file left; the spike's python and node images removed; back on the live catalog; the drill catalog reset to
|
||||
live `main`. demo-hp 9201 — the two admin passwords changed and the two "I changed it" records (the operator's
|
||||
ruling); nothing else. demo-felhom — nothing. Host: nothing. Hub: floor 0.280.0. ep0: untouched.
|
||||
@@ -0,0 +1,46 @@
|
||||
# REPORT — 2026-09-28 evening: no app goes live with a login a stranger knows; demo-hp's restore test on the NVMe; an empty backup is an alarm; ep0; small leftovers
|
||||
|
||||
Architecture read first: `09` §3 (decisions 11–43, and the new 44–45), `01` §5, `03` (restore storage), `07` §6,
|
||||
`06` + R-600. Controller **v0.279.0** (one release). Scope ruled mid-session by the operator: the brief as written for
|
||||
all 40 apps, across several sessions; this session starts it and hands over.
|
||||
|
||||
## The Parts
|
||||
|
||||
| Part | Step | State | Note |
|
||||
|---|---|---|---|
|
||||
| A | audit of 53 apps | done | `app-catalog-felhom.eu/FIRST-ADMIN.md`; linked from `01` §5 and `09` decision 45. Measured: claper, bookstack, calibre-web, calcom (9202); bookstack, calibre-web, romm (demo-hp). The rest is marked "read" |
|
||||
| A | measure every class-3/4 app on 9202 | **partly** | 4 of 39 measured today; the rest is R-707 |
|
||||
| B1 | route (a)/(b) per app | **2 done** | claper, bookstack: fresh-install proof (default fails, generated works, wrong fails); claper after a restore too. calibre-web: route found, blocked by our generator (no special character) |
|
||||
| B2 | route (c) sentence | done (mechanism) | shown for every template with `default_creds` and no working `after_install` — today bookstack(installed)/calibre-web/mealie/romm/wger/zipline pages; class-4 apps have no default to name |
|
||||
| B3 | demo boxes read-only | done | demo-hp: bookstack + calibre-web defaults still log in (page now warns); romm's note is stale (401). Nothing changed |
|
||||
| B4 | box-side mechanism | done | `after_install:` in v0.279.0, tests + red-proofs; failures recorded and shown, app stays running |
|
||||
| C | demo-hp restore test on NVMe | done, **changed** | "set restore_storage, nothing else" did not work: 403 without a grant; operator approved the grant; one test passed in 8m46s; `local-lvm` unchanged. Next scheduled cycle: see below |
|
||||
| D | empty-backup alarm | done | measured before: a WARN line only (demo-hp, yesterday). Built: digest once/app/tier/day + page sentence; tests + red-proofs; live negative on 9202 (no false alarm). Live positive not reproducible (its known cause, R-704, is fixed) |
|
||||
| E | ep0 peer | done — **nothing to remove** | the peer was already gone (the hub's sync) |
|
||||
| F1 | R-706 | done | v0.279.0, red-proofed; not seen live |
|
||||
| F2 | R-705 controller half | done | proven on 9202 |
|
||||
| G | night watch | not done | optional; not run (see STATUS) |
|
||||
|
||||
## Claims in the brief that turned out wrong
|
||||
|
||||
- **"Six apps ship a hard-coded default"** — wrong: 5 (bookstack, calibre-web, claper, mealie, wger); romm's and
|
||||
zipline's notes are stale (romm's does not log in). And **34 more** have an open first-run screen.
|
||||
- **"The box publishes every app on the internet at install"** — right (the tunnel's `*.domain` route).
|
||||
- **"No architecture document covers default logins"** — right (only `10-localisation.md` mentions `default_creds`
|
||||
for translation); the audit's home is now `FIRST-ADMIN.md`, linked from `01` and `09`.
|
||||
- **"`nvme-scratch` can hold a restored guest"** — the disk and content types could; the agent could not use it
|
||||
without a storage grant (403). Granted with the operator's word.
|
||||
- **"The test-install peer is still on ep0"** — wrong: gone.
|
||||
- **"Nothing flags a running app with an empty backup today"** — right for yesterday's controller (a WARN line only).
|
||||
|
||||
## Rows
|
||||
|
||||
Opened: R-707 (37 apps), R-708 (grafana `admin` fallback), R-709 (password fields in the page HTML). Closed: R-701,
|
||||
R-702. Narrowed: R-705 (agent half left). Watching: R-706. R-600 annotated. Register 342 → 345 rows.
|
||||
|
||||
## Teardown
|
||||
|
||||
Machines: 9202 — claper, bookstack, calibre-web installed and removed through the product; back on the live catalog;
|
||||
drill catalog = live. demo-hp 9201 — nothing changed (read-only logins). Host: demo-hp agent config
|
||||
(`restore_storage`, saved `agent.json.pre-d44`) and one ACL grant — both kept (decision 44). Hub: floor 0.279.0. ep0:
|
||||
read only.
|
||||
@@ -0,0 +1,56 @@
|
||||
# REPORT — more apps that update themselves (2026-09-30, evening, by day)
|
||||
|
||||
Evidence: `documentation/audits/more-night-apps-2026-09-30/` (README there has the per-app table). Architecture read:
|
||||
`architecture/09-update-architecture.md` §3 decisions 6, 13, 14, 17, 22, 30, §6.4 parts 4–7, §6.5.
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | done / not done / changed | why |
|
||||
|---|---|---|
|
||||
| **A1 calibre-web** | **done** — v4.0.6 → v4.0.8, first ladder `53a4a1d` | front door = the Upload button (HTTP form), read back through OPDS + the served EPUB. The ingest folder was NOT used (it would be a file planted in a mount). `files_may_change`: the library DB `metadata.db` (+`-shm`, `-wal`) in the books folder; the book file did not change |
|
||||
| **A2 gitea** | **done** — 1.27.0 → 1.27.3, first ladder `d7ba60c` | the installer-form POST (no CSRF on that form), through the setup gate on the box. 28.0.0 not touched |
|
||||
| **B1 emby, ghost, home-assistant** | **done** — `3e4aa77`, `9a78e3b`, `15f4d59` | each had a newer tag inside its line (re-checked) |
|
||||
| **B2 uptime-kuma, wger, crafty-controller** | **done** — `35dd5cf`, `4ad32aa` (+ fix `7a4ff48`), `e1f0179` | new fixtures; wger needed a template fix first (R-738) |
|
||||
| **B2 wanderer** | **not done** | the bench cannot run it at all: its web server calls the DB at a public https name (R-739). The meilisearch question is NOT measured |
|
||||
| **B3 time left** | **done** — outline `b0b2514`, rallly `8d3a35a`, zipline 4.6.1 → 4.7.0 `a9700e2` → 4.8.0 `fb87030` | zipline 4.8.0 refuses a database that skipped 4.7.x (R-742); the box undid the direct jump itself |
|
||||
| **C the same-name fix gap** | **done** — row R-740, STATUS decision, `09` decision 30 dated note | measured from source, a unit walk, the registry and demo-hp; **nothing built** |
|
||||
| **D immich's older step** | **done (changed)** — re-proven at 768M, step files rewritten `48440ce` | the brief's route needed a writer mode that did not exist → `upgrade-test.py --restep` `63a96b0`, with tests |
|
||||
|
||||
**Night-updatable apps (the audit's method (a)): 30 + 2 conditional at the start → 35 + 3 conditional** (38 carry a proven step; 15 have no ladder).
|
||||
|
||||
## Claims in the brief that turned out wrong
|
||||
|
||||
1. **"Night-updatable: 31 (+ nextcloud conditional)"** — at the session's start it was **30 + 2 conditional**: the
|
||||
afternoon's immich step carries `files_may_change` (R-734), so immich counts as conditional.
|
||||
2. **"Register 366 rows"** — **369**: the afternoon session added R-732..R-734 after STATUS said 366.
|
||||
3. **"calibre-web and gitea have no ladder"** — true.
|
||||
4. **"gitea's fixture fails at the installer"** — true (`MustInstalled`); the installer form works headless.
|
||||
5. **"The leg never presses a digest-only change"** — true (`unattended.go:433–435`). But the larger fact is that the
|
||||
catalog never records a same-tag re-test at all, so the Update button does not deliver it either (R-740).
|
||||
6. **"Decision 30's text says the leg would"** — true: „until someone (or the automatic leg) presses Update".
|
||||
7. **"immich's 0b82… step pins 512M"** — true. **"The writer rewrites the step file"** — it could not (it writes a
|
||||
step file only once, when a step is superseded); `--restep` was added.
|
||||
8. **"No real box runs immich v3.0.3"** — true and stronger: no box reporting to the hub runs immich at all
|
||||
(hub `/apps`, positive control bookstack = 2 deployments).
|
||||
9. **"wger: behind inside the major"** and **"crafty/uptime-kuma/wanderer need a new fixture"** — true; wanderer
|
||||
cannot be run on the bench at all.
|
||||
|
||||
## Rows
|
||||
|
||||
Before **369**, after **377**. Opened: R-735 (bench password shape — closed), R-736 (old images never deleted),
|
||||
R-737 (wger app-login 500), R-738 (wger migrations — closed), R-739 (bench cannot run wanderer), R-740 (same-name fix —
|
||||
decision), R-741 (after_install default-login window), R-742 (zipline needs 4.7.x first — closed). Narrowed: R-462, R-624, R-446,
|
||||
R-440, R-734, R-732 (note). Closed: R-735, R-738, R-742.
|
||||
|
||||
## Teardown
|
||||
|
||||
Three layers:
|
||||
- **Machine.** Scratch guest 9202: every app this session installed removed through the product (calibre-web twice,
|
||||
gitea, emby, ghost, home-assistant, wger ×3, crafty-controller, uptime-kuma, outline, rallly, zipline ×3); their images
|
||||
and older drill leftovers removed **by name** (53 GB freed at 17:43 when the disk refused an install — R-736); pointed
|
||||
back at the live catalog and its `update:` override removed (`box/T6`, `repo_url` read back); the same six containers
|
||||
run as at the start. Bench LXC 9401: evidence copied off after every run, its images removed by name once (disk full),
|
||||
then destroyed with its template (`bench/B9`). Drill catalog: force-reset to live `main` `fb87030` (operator-allowed),
|
||||
image lines identical (`box/T7`).
|
||||
- **Host.** demo-hp: `pct list` shows 9201 and 9202 only, as at the start.
|
||||
- **Hub.** Read only (`/hosts`, `/apps`); nothing provisioned.
|
||||
@@ -0,0 +1,60 @@
|
||||
# REPORT — the new-app checklist: in the catalog, a gate, piloted on wger (2026-10-01, evening)
|
||||
|
||||
Architecture read: `documentation/architecture/09-update-architecture.md` §3 (decisions 13, 22, 37, 42, 45–50, 61) and
|
||||
§6.5. Evidence: `documentation/audits/new-app-checklist-2026-10-01/README.md` (the full write-up).
|
||||
|
||||
## The Part table
|
||||
|
||||
| part | result | changed from the brief, and why |
|
||||
|---|---|---|
|
||||
| A — checklist in the catalog | **done** — `NEW-APP-CHECKLIST.md` 60 rows / 10 groups; `onboarding/_TEMPLATE.md`; CLAUDE.md, REUSE.md §5, README point to it | 7 rows added, 16 sharpened, 9 wrong claims fixed (below). A `since` column per id (B's date rule) |
|
||||
| B — the gate | **done** — `scripts/check-onboarding.py`, gate `onboarding` in `--fast` (hook + CI); 16 decoys, all judged right; 5 gate mutants each turn the suite red | the 53 are exempt **by name**, not "new since commit X": CI fetches at depth 1 and cannot diff (R-452). Evidence in `felhom.eu/` is checked where that repo sits beside the catalog (the hook) and printed as NOT CHECKED where it does not (CI). Rows inside an HTML comment do not count; an empty evidence directory does not count |
|
||||
| C — wger pilot | **done** — both records filled; table below | the restore round trip (2.5) could not be measured: no per-app backup press exists outside an Update (R-648) — open, R-759 |
|
||||
| D — gap page | **done** — `onboarding/EXISTING-APPS-GAPS.md` from `scripts/onboarding_gaps.py` | none |
|
||||
|
||||
**The date rule (B.1):** each checklist id carries `since`; a record answers every id with `since` ≤ its `opened:`.
|
||||
`opened:` must be on or after 2026-10-01 and not in the future, so a later id binds only apps opened after it. Residual:
|
||||
an author can date `opened:` back to the cut-off to skip ids added since — the gate cannot see that; review can.
|
||||
|
||||
## Claims in the draft that were wrong
|
||||
|
||||
3.3 "32 apps" (33 today) · 5.2 "romm OOM at +76 s (decision 22)" (no such figure anywhere; R-635) · 5.3 "gate" (no gate;
|
||||
8 templates differ, R-758) · 6.2 "35 of 53 update at night" (not reproducible; 38 carry a ladder) · 8.3 logo address
|
||||
(`.webp` vs the controller's `.svg`/`.png`, R-761) · 1.6's how could not show R-737 · 1.4's how has nothing to run for
|
||||
an app at its newest tag · 0.4's packet capture is not in our kit · 2.6's R-756 is an unexplained venue case (R-442
|
||||
added). Duplicates of gates now name the gate (1.1, 2.1, 3.3, 4.2, 6.2, 8.1, 9.4).
|
||||
|
||||
## The pilot — the four problems
|
||||
|
||||
| problem | caught by | the draft's how? |
|
||||
|---|---|---|
|
||||
| R-737 JWT key | 1.6, 3.9 (new), 0.7 | **missed** — web login worked |
|
||||
| R-738 no migration | 1.4, 1.7 (new), 6.1 | **only if an update existed** — not for a new app |
|
||||
| R-752 lock-out → everyone | 3.6 | **caught** |
|
||||
| R-755 dev server | 1.5, 1.7 | **caught** |
|
||||
|
||||
**And three more, found by the new rows on the LIVE wger template (9202, drill catalog):** R-762 no CSS/JS and no
|
||||
uploaded photo is ever served (404); R-763 a stranger signs up after the setup, and every anonymous dashboard visit
|
||||
creates a guest account; R-764 mail goes to the console. Not fixed: this task changes no template.
|
||||
|
||||
## Gap page headline (of 53)
|
||||
|
||||
Fit 52 · images/DB 53 · storage 38 · accounts 39 · health 49 · resources 24 · updates 26 · mail — (6 mapped) · text 52.
|
||||
|
||||
## Rows
|
||||
|
||||
Opened **R-758** (8 `mem_limit` ≠ sum), **R-759** (wger's open record rows), **R-760** (vikunja healthcheck),
|
||||
**R-761** (logo comment), **R-762** (P2, wger static + media), **R-763** (P2, wger strangers + guests), **R-764**
|
||||
(wger mail). Narrowed: none. Note added to R-755 (same server question as R-762). Closed: none.
|
||||
**Register 392 → 399.** STATUS updated.
|
||||
|
||||
## Live work and teardown
|
||||
|
||||
9202 only, drill catalog `e9f50b5` (repointed, then restored to live `6d72c09` — three controls each way). wger
|
||||
installed and removed twice through the product. **Machine:** no wger container, volume or image left; sampler files
|
||||
removed. **Host:** nothing. **Hub:** untouched. Secrets never printed; evidence scanned for their values.
|
||||
|
||||
## Gates
|
||||
|
||||
Catalog: `catalog_gates.py --fast` all OK (11 gates); `test_gate_decoys.py` 121 cases OK; `test_catalog_gates.py` OK;
|
||||
`decoy_coverage_gate.py` 0 unaccounted. felhom.eu: `repo_gates.py --fast` — see the commit. No `--no-verify`.
|
||||
@@ -0,0 +1,65 @@
|
||||
# REPORT — wger hidden; the first new apps through the checklist (2026-10-01, night)
|
||||
|
||||
Architecture read: `09-update-architecture.md` §3 (13, 22, 42, 45–50, 57, 61) and §6.5; `01-topology-and-trust.md` (the
|
||||
HTTP-only tunnel; §5 the setup gate); `07-backup-architecture.md` (classes). Evidence:
|
||||
`documentation/audits/new-apps-2026-10-01/` (README, FIT.md, A/ B/ S/ G/ bench/ box/ undo/ shots/ wip/ tools/).
|
||||
|
||||
## The Part table
|
||||
|
||||
| part | result | changed from the brief, and why |
|
||||
|---|---|---|
|
||||
| 0 — wger hidden | **done** — catalog `55b8c8a`; on 9202 after a sync wger is not on the app list (control: mealie is, plant-it is not) | none |
|
||||
| A — fit table | **done** — `FIT.md`, nine apps; verdicts in STATUS | three research passes in parallel, facts kept with their sources |
|
||||
| B — build in order | **Radicale, Karakeep, Dawarich published; Grimmory tested and held (R-775); Grimoire, MeTube, Pinchflat stopped on the fit verdict** | no app was "not reached": every app in the order got its verdict before 23:00 |
|
||||
|
||||
## The fit table (one line each — the full table is FIT.md)
|
||||
|
||||
Radicale build · Karakeep build with a note · Dawarich build with a note · Grimmory build with a note · Grimoire **stop**
|
||||
(upstream: public exposure unsupported; no 1.x image) · MeTube **stop** (no login by design) · Pinchflat **stop** (paused
|
||||
upstream; no tag for its last release) · Invidious **stop recommended** (companion-only playback, YouTube blocks the
|
||||
household's IP) · moonlight-web **stop** (UDP / gaming PC).
|
||||
|
||||
## Claims in the brief that turned out wrong
|
||||
|
||||
1. **Grimmory / BookLore** — "the original was abandoned" is half right: the developer deleted it 2026-03-23, it came back
|
||||
~04-30 in maintenance mode pointing at BookOrbit; `grimmory-tools/grimmory` is the right fork (an organisation, 24
|
||||
releases in 2026).
|
||||
2. **Invidious** — the "signature helper" is gone; `invidious-companion` replaces it and the po_token generator.
|
||||
3. **moonlight-web** — two unrelated projects; both have a WebSocket fallback (the brief implied UDP only).
|
||||
4. **Karakeep's AI default** — off unless a key is set (as the brief hoped); not in the brief: its phone app sends crash
|
||||
reports to its makers.
|
||||
5. **Grimoire** is not a lighter Karakeep we can publish — its makers rule out internet exposure.
|
||||
6. **SparkyFitness and Calibre-Web Automated** — confirmed already in the catalog.
|
||||
7. **Radicale** — the brief offered `tomsquest/docker-radicale` or upstream: upstream's own `ghcr.io/kozea/radicale`
|
||||
chosen (same cadence, multi-arch, the project's own).
|
||||
|
||||
## Per published app
|
||||
|
||||
| app | commit | record | bench (swap 0) | 9202 | memory peaks (anon) | ladder |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Radicale | `195129c` | 60/60 done or n/a | proven | step done 14 s; undo on a forced failure; remove + restore read back | 26 MiB / 128M | 3.8.0 → 3.8.1 |
|
||||
| Karakeep | `882ac14` | 60/60 | proven (768M: 79 %; 1024M: 51 %) | step done 95 s; restore read back; 10-page burst at 1536M | web 832 / 1536M, chrome 218 / 768, meili 84 / 512 | 0.33.1 → 0.33.2 |
|
||||
| Dawarich | `72247a3` | 60/60 | proven | step done 158 s; restore read back + own password signs in | app 460–505 / 1024, sidekiq 270–314 / 1024, db 148 / 512 | 1.15.2 → 1.15.3 |
|
||||
|
||||
**Grimmory** — bench proven (v3.4.1 → v3.5.0, app 522 / 1024M, db 109 / 384M); on 9202 the update step done in 78 s, the
|
||||
gate opened by its own probe (`data` false → true), OPDS through traefik right 200 / wrong 401, and a stranger's 5 wrong
|
||||
sign-ins locked every visitor out (429) for 15 min (Caffeine `expireAfterWrite(15 min)`, keys by address and by name,
|
||||
hard-coded) — while the OPDS feed kept working. Held: R-775 (operator: A publish with a page sentence / B wait for R-753).
|
||||
|
||||
## Rows
|
||||
|
||||
Opened **R-765** (Radicale restore — closed the same session), **R-766** (assets reach boxes only with a hub release),
|
||||
**R-767** MeTube, **R-768** Grimoire, **R-769** Pinchflat, **R-770** Invidious, **R-771** moonlight-web, **R-772** a probe
|
||||
that cannot run records healthy, **R-773** sign-up block lost after remove + restore, **R-774** Karakeep/Dawarich mail ON +
|
||||
Karakeep's phone-app crash reports, **R-775** Grimmory lock-out. Notes on R-762/R-763 (wger hidden). **Register 399 → 410.**
|
||||
|
||||
## Teardown
|
||||
|
||||
- **Machine (9202):** every app installed tonight removed through the product (Radicale, Karakeep, Dawarich with data;
|
||||
Grimmory keeping drive data because of R-756's 409, then its harness-made test folders deleted by name, two empty
|
||||
folders removed only after checking they were empty); no container or volume of tonight's apps left; temp files
|
||||
removed; back on the LIVE catalog (`B/B9-restore-live.txt`: clone = live `72247a3`, standing apps healthy).
|
||||
- **Host (demo-hp):** bench LXC 9401 destroyed (`pct destroy --purge`); the drill repo reset to live `main`.
|
||||
- **Hub:** untouched (9202 is unenrolled; no asset reseed — R-766).
|
||||
- Secrets: the 9202 dashboard password and the generated app passwords lived in a 0600 scratch directory, never printed;
|
||||
the evidence was scanned for their values (none); the scratch directory is shredded at the end.
|
||||
@@ -0,0 +1,21 @@
|
||||
# REPORT — night shift 2026-09-24 (run in daytime from 11:07 CEST)
|
||||
|
||||
Full record: `documentation/audits/DRILL-night-2026-09-24.md`; step log `documentation/audits/night-2026-09-24/PROGRESS.md`.
|
||||
|
||||
- **Hub v0.123.0** (`99c0709`, manifest `b33a331`, live): `app_stopped_unhealthy` allow-listed, per-app cooldowns
|
||||
on both legs, app named in the mail subject, seeded once into every household (3 seeded); hu/en mail goldens;
|
||||
red-proofed (`night-2026-09-24/redproofs/A3-hub.txt`).
|
||||
- **Docs:** `09` §3 decisions 26–28 (operator) and 29–30 (CC unattended); §6.4 part 6 marked shipped (both
|
||||
halves); part 7 rewritten as the measured build brief §6.4.2. `07` whole-copy table (the second drive is whole
|
||||
for a file app since v0.269.0). `08` §6.2 the stop rule. Capability map: five rows. CONTEXT and STATUS.
|
||||
- **Register:** 335 → 344 rows (679,393 → 685,662 B); opened R-668…R-683, closed R-661, R-662, R-664–R-668.
|
||||
- **Floor 0.269.1** (MinAgent 0.131.0), read back; N100 arrived; demo-hp 9201 did not (R-672).
|
||||
- **Interventions: 1** — an agent restart on demo-hp so its own Recover removed a leaked restore-test guest that had
|
||||
filled the pool. The session's permission check refused a direct `pct destroy` and a later read of vzdump task
|
||||
logs; both said so in the findings doc.
|
||||
- **Slip:** `4502af6` committed three drill test-account passwords (apps since removed, accounts gone);
|
||||
deleted + ignored in `f2234c0`, history not rewritten.
|
||||
- `unproven.py --summary`: unchanged — 35 of 55 not walked.
|
||||
- Teardown, three layers: machine (9202 back to live catalog + saved config, tonight's apps removed through the
|
||||
product, kept drive folders removed by name, backup window 02:30), host (demo-hp `pct list` 9201 + 9202, no
|
||||
bench), hub (floor; drill repo reset to live `c8025093d23c`, `has_actions: false`).
|
||||
@@ -0,0 +1,54 @@
|
||||
# REPORT — the operator's two rulings built (2026-09-30, late evening)
|
||||
|
||||
Evidence: `documentation/audits/night-rulings-2026-09-30/` · golden: `documentation/tests/golden-0.284.2-2026-09-30/`.
|
||||
Architecture read: `09` §3 decisions 13, 14, 17, 30, 40, 45–47, §5.3, §6.4 parts 4–7, §6.5; `07` (R-698); `01` §5.
|
||||
Baselines (live Gitea 21:48): controller `d48da6c` (0.283.1), agent `d766666` (0.138.0), felhom.eu `42bf40b`, catalog
|
||||
`d181165`. Register 377 rows by the method "every table row that starts with an R-id, bold or not, unique ids" (the
|
||||
reviewer's regex and mine agree on 377 today).
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | done / not done / changed | why |
|
||||
|---|---|---|
|
||||
| **Rulings 52, 53** | **done** — recorded first (`09` §3, `07`, CONTEXT; decision 30 note) `1de0f7c` | before any work |
|
||||
| **A — same-tag re-tests** | **done** — catalog `6a3ead9`: the entry shape, the gates (+8 decoys, seen red), `upgrade-test.py --retest`, the ONE command `retest-floating.py` | spike note `A/A0-spike.md` first: the move gate never looked at a re-test (it changes `.felhom.yml` only) |
|
||||
| A4 end to end on 9202 | **done** — docmost at an older `redis:7-alpine` digest, "run tonight's chain now", the leg's `step pressed … step ended done after 95.0 s`, new digest running, read back, badge current | the first attempt (outline) stopped at outline's fixture on both venues (R-744) |
|
||||
| A5 the real run | **done — nothing to re-test on the engine lines**; two EXACT tags were rebuilt upstream (R-743) | `--engines-only` is decision 52's start |
|
||||
| A6 scheduling | **changed: a runbook, not a cron job** — `runbooks/monthly-floating-retest.md`; a standing monthly step in STATUS | it needs a fresh bench, 9202 on the drill, a drill force-reset, pushes to the live catalog |
|
||||
| **B — image retention** | **done — controller v0.284.2** (0.284.0 and 0.284.1 never floored) | two faults found LIVE on 9202: the Remove button runs `RemoveStack` (only `DeleteStack` was wired); `docker image ls` without `-a` hides the untagged digest-pulled app images |
|
||||
| B3 one-time sweep | **done** — 9202 26.6 → 5.7 GB (26 images), demo-hp 24.3 → 13.5 GB (24), the N100 6.15 → 6.07 GB (1); every app healthy | old controller images stay by design (R-745) |
|
||||
| B4 red-proofs | **done at unit level** (shared image, undo's image, stopped app's compose, update in flight, unreadable keep set, both wirings, untagged images); **the live "update → failure → undo with no pull" was NOT run** | no failing edge was built tonight; the previous image is in the keep set (red-proofed) |
|
||||
| **C — the install hold** | **done — controller v0.284.x** — the setup gate's door in front of an `after_install` app until the login is replaced | the brief expected 404 until the change; it is the gate's 401 (the household still passes) |
|
||||
| C3 live on 9202 | **done** — calibre-web 0 of 192 stranger tries with the default login got in; mealie 0 of 97 | the positive control with the generated password was NOT obtained (a backup stopped calibre-web at that moment; mealie locked itself — R-747) |
|
||||
| **D1 wger's key** | **done** — catalog `45d8482`; bench: right password 200 with a token that reads the API; wrong password **400** (not 401) | not proven on a box |
|
||||
| **D2 wanderer** | **changed: the bench runs it; no step** — web/sign-up/login 200 with a bench-only override; meilisearch v1.54 needs `MEILI_UPGRADE_DB=true` (then the indexes survive) | a list could not be created (PocketBase refused) — no fixture, so no step (R-739) |
|
||||
| **E — release, floor, golden** | **done** — floor **0.284.2** (MinAgent 0.131.0 declared), both demo boxes on 0.284.2 within a minute; golden **0.284.2** baked, round-trip identical, vouched (agent 0.138.0, min_agent 0.131.0); the gate prints **OK** (not WAIVED) | no agent release: MinAgent unchanged |
|
||||
|
||||
## Claims in the brief that turned out wrong (or right)
|
||||
|
||||
1. **"The box needs no change to press a same-tag re-test"** — **right** (unit walk; the e2e leg on 9202).
|
||||
2. **"No tested digest differs from the registry today"** — right for the database/redis lines; **wrong for two exact
|
||||
tags**: `nextcloud:34.0.4-apache`, linuxserver `sonarr:4.0.20` (R-743).
|
||||
3. **"`--rmi local` never removes a registry-tagged image"** — right, and **the Remove button does not even use it**:
|
||||
`RemoveStack` runs `down --volumes`; only the older `DeleteStack` had `--rmi local`.
|
||||
4. **"Images are shared between apps"** — right: `postgres:18-alpine` and `redis:7-alpine` served docmost and paperless;
|
||||
a remove of docmost kept both.
|
||||
5. **"The route is published before `after_install` runs"** — right (measured earlier; now held).
|
||||
6. **"Wanderer's web server needs the public DB name"** — right: `PUBLIC_POCKETBASE_URL` is the only address both images read.
|
||||
7. Also: the brief expected 404 during the hold (it is the gate's 401) and 401 for wger's wrong password (it is 400);
|
||||
"hub — read only" and Part E's floor + vouch conflict — the two form saves were done, nothing else on the hub.
|
||||
|
||||
## Rows
|
||||
|
||||
**377 → 383** (every `| **R-<n>[letter]** |` row; the register-shape gate skipped the 3 lettered ids until tonight — R-748). Opened R-743 (exact tags rebuilt), R-744 (outline fixture at 1.10.1), R-745 (old controller images),
|
||||
R-746 (`image_digest.resolve` ignores a digest), R-747 (mealie lockout by strangers), R-748 (the gate's lettered-id blind spot — fixed). Closed R-736, R-737, R-740, R-741, R-748.
|
||||
Narrowed R-739, R-698, R-446.
|
||||
|
||||
## Teardown
|
||||
|
||||
- **Machine:** 9202 back on the live catalog (`repo_url` read back), its drill `update:` override removed, the same
|
||||
containers as at the start, controller 0.284.2; the apps this run installed removed through the product. Bench LXC 9401
|
||||
destroyed with its template. Drill VM: CT 9100 destroyed, secrets shredded, qemu exited, `drill.qcow2` reverted to
|
||||
`virgin`. The drill catalog reset to live (`6a3ead9`), image lines identical.
|
||||
- **Host:** demo-hp `pct list` = 9201, 9202 (as at the start); the N100 untouched except the floor's controller update.
|
||||
- **Hub:** two form saves only — the floor (0.284.2, MinAgent 0.131.0) and the vouch (golden 0.284.2).
|
||||
@@ -0,0 +1,85 @@
|
||||
# REPORT — off-site backup safety, step 1: the append-only lock measured on the provider (2026-10-03)
|
||||
|
||||
A spike. No product code changed, no release. Evidence, exit test and design:
|
||||
`documentation/audits/offsite-append-only-2026-10-03/`. Architecture read: `07-backup-architecture.md`
|
||||
§8a, threat row 10, §D; `06-offsite-connectivity.md` (PBS/tunnel only — it does not describe the restic
|
||||
tier, so the facts went to `07` §D). Baselines (re-verified): felhom.eu `f4c5466`, controller `0945332`
|
||||
(v0.288.0), register 326 rows, highest id R-819.
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | done / not done / changed | why |
|
||||
|---|---|---|
|
||||
| 0 — venue | **changed** — `u629488-sub4` (tester-1) instead of a new scratch customer | operator ruled "use Tester1" in-session. tester-1's box was deleted 2026-09-30; nothing writes there. Credential: the hub's stored tester-1 value, read from a copy of the hub DB on a second operator ruling (copy deleted, value never printed or written to a committed file). A new repo dir `spike-r436` only; `felhom-repo` never read or written |
|
||||
| A — the lock | **done**, exit test written first (`EXIT-TEST.md`) | E1–E8 and C1–C2 as stated; locks measured |
|
||||
| B — the attacker | **done**, one item lab-only | raw-HTTP path escape through the pinned server measured in the lab only — a live HTTP/2 bridge over the forced ssh could not be made to work in the time box |
|
||||
| C — design | **done** — `DESIGN.md`, STATUS decision 0 | |
|
||||
| D — ep0 | **done** — `PART-D-ep0-safeguard.md`, STATUS decision 0b | read only; ep0 not touched |
|
||||
| E — records | **done** | below |
|
||||
|
||||
## Claims in the brief (and the register) that turned out wrong
|
||||
|
||||
1. **"The box holds no sub-account password"** — it does not STORE one, but it can **obtain it at will**:
|
||||
declare `needs_credential` twice → the hub re-arms the stored value → the box consumes it (R-820).
|
||||
2. **"A forced command cannot be bypassed by the sub-account itself"** — the pinned key cannot; the
|
||||
**password can** (logs in on ports 22 and 23, rewrote `authorized_keys` this session).
|
||||
3. **"The hub cannot prune because of custody"** — true for *pruning*; but the hub can **delete**: it
|
||||
holds every sub-account password in the clear (R-821).
|
||||
4. **"Both `forget` sites must change together"** — there are **four** deleting features on the box:
|
||||
both `forget` sites, the orphan move-aside (`mv`) and the abandonment (`rm -rf`).
|
||||
5. **rclone in the image** (R-436 row: "rclone is not in the controller image today", implying it is
|
||||
needed) — **not needed**; restic 0.14.0 with `-o rclone.program="ssh … rclone"` is enough.
|
||||
6. **R-342's first candidate, a Hetzner Volume snapshot** — does not exist.
|
||||
7. **R-430's model** (a locks dir where deletion is refused) — does not describe this transport; the
|
||||
append-only server allows lock deletion and `unlock --remove-all` works.
|
||||
8. **The vendor's cited blog** (`fluix.one`) shows the line WITHOUT `--append-only`; only Hetzner's
|
||||
ticket reply adds it. Copying the blog would give a deleting key.
|
||||
9. Held: restic is **0.14.0** (`0.14.0-1+b5`); **restore works through the add-only key**.
|
||||
|
||||
## Part A — results (verbatim refusal)
|
||||
|
||||
`blob not removed, server response: 403 Forbidden (403)` for `forget d807418c --prune`,
|
||||
`forget --keep-last 1` and a real `prune` (each ~45–48 s of retries, rc=1); snapshot count unchanged;
|
||||
control key: `1 / 1 files deleted`. Crash lock: blocks `check`, not `backup`; plain `unlock` prints
|
||||
success and removes nothing; `--remove-all` removes it. Files: `live/E1-E3…`, `live/E4-E6…`, `live/C2-A5…`.
|
||||
|
||||
## Part B — the attacker table
|
||||
|
||||
| Route | Tried how | Result | What closes it |
|
||||
|---|---|---|---|
|
||||
| Password, port 23 | `sshpass ssh -p 23` | **logs in**; `authorized_keys` read and **rewritten** | box never receives it (hub = key registrar) |
|
||||
| Password, port 22 | `sshpass sftp -P 22` | **logs in** (SFTP), `.ssh` listed | same |
|
||||
| Box obtains the password | source read | **yes, at will** (self-heal re-arm + consume) | same — R-820 |
|
||||
| Pinned key: shell / `rm -rf` | `ssh … 'ls'`, `'rm -rf spike-r436'` | runs the forced rclone; repo intact | — (holds) |
|
||||
| Pinned key: sftp / scp / rsync | each | refused / protocol error | — (holds) |
|
||||
| Pinned key: port forward | `-L`, then connect | `administratively prohibited` | — (holds) |
|
||||
| Pinned key: other path, no flag | `rclone serve restic --stdio felhom-repo` | pinned dir served, append-only | — (holds) |
|
||||
| Pinned key: `../` escapes, overwrite | raw HTTP (lab) | 400 / 403 | — (holds; lab rclone) |
|
||||
| Pinned key: add junk / new `keys/` | raw HTTP (lab) | allowed | quota fills — R-431/quota alarms |
|
||||
| Pinned key: future-dated snapshots | restic (lab) | allowed → retention erases real history | poisoning guard — R-822 |
|
||||
| Any key on port 22 | both test keys | refused (port 22 takes no OpenSSH key) | — |
|
||||
| Hetzner API / panel | box code read | nothing on the box reaches either | — |
|
||||
| Hub DB | operator-tier | every sub-account password in clear | R-821 |
|
||||
|
||||
**A route defeats the lock: the password (R-820).** The lock alone is not protection until it is closed.
|
||||
|
||||
## Records
|
||||
|
||||
- **Closed:** R-436 (measured; the 2026-10-06 due-check is cleared — the block is now empty), R-430.
|
||||
- **Opened:** R-820 (P2, Security), R-821 (P2, Security), R-822 (P2, Backup). None is P1 by the
|
||||
scale: today the box's own key can already delete (R-95), so none adds harm *today*.
|
||||
- **Updated:** R-95 (the measurement, the four sites, the proposal; rank untouched), R-342 (options costed).
|
||||
- **Register: 326 → 327** (`register_shape_gate`). All felhom.eu gates green.
|
||||
- `07` §D: one `[FACT]` block. STATUS: two decisions in the operator's format.
|
||||
- `unproven.py --summary`: NOT WALKED 35 of 55 — unchanged.
|
||||
|
||||
## Teardown
|
||||
|
||||
- **Provider:** `authorized_keys` restored — sha256 `795e7153…` before and after, identical; `spike-r436`
|
||||
removed; `~/.config/rclone/` (created by the provider's rclone during the test) removed; home is back to
|
||||
`.ssh`, `felhom-repo`. Both test keys refused afterwards. (`live/TEARDOWN.txt`)
|
||||
- **DooPlex:** lab container, network and image removed; test keys, the password file, the hub DB copy
|
||||
and hub page copies deleted from the scratchpad.
|
||||
- **Hub:** nothing changed (two reads).
|
||||
- **Left as is, on purpose:** the tester-1 sub-account password was NOT rotated — the next tester-1 install
|
||||
needs the stored value. R-821 covers why that is itself a risk.
|
||||
@@ -0,0 +1,36 @@
|
||||
# REPORT — off-site safety finished (decisions 71–74) — 2026-10-04
|
||||
|
||||
Architecture: `07-backup-architecture.md` (custody block, threat rows 9/10), `06` §3.6, `09` §3 decisions 68–74.
|
||||
Baselines (re-verified): controller `c4bf7306371a` (0.289.1), agent `d766666ff8cf`, felhom.eu `710a2505f9b4` (hub
|
||||
0.127.0). Register 330, highest R-830. Rulings recorded first (decisions 71–73, R-831, R-832: `697c2a7`).
|
||||
Evidence: `documentation/audits/offsite-finish-2026-10-04/`.
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | Result | Notes |
|
||||
|---|---|---|
|
||||
| A — guard fixed, one real window | **done; a window that removes something NOT yet observed** | Controller v0.290.0: a young snapshot superseded the same day is excluded instead of refusing; future-dated / newer-than-hub / above-the-week's-cap still refuse. 3 red-proofs (the demo-hp shape runs; the 13 future fakes refused; above-cap refused). Live: window 2 on demo-hp opened, guard ran with no refusal, closed in 3 s, **127→127 removed nothing**, because every candidate was a young same-day copy. The key file is clean and the hub's before/after check is quiet. Weekly windows **ON** (fleet switch). Next scheduled windows: demo-felhom at its next night run (never had one); demo-hp at the first night run after 2026-10-10 17:36 UTC. The first real removals are expected around 2026-10-11. |
|
||||
| B — copy keeps 8 weekly | **done** | `prune-ep0-copy` keep-weekly 8, all namespaces, daily 07:30 (sync 05:00 — ran OK today); GC Sundays 08:30; `remove-vanished` false. Dry-run **by reasoning**, because PBS has no CLI dry-run for a prune job: nothing to remove (2 snapshots per group, 2 different weeks). 12 GB used. |
|
||||
| C — tester-1's keys | **done** | Through the hub (`POST /offsite/remove-unpinned/tester-1` → `changed: 3`); the check reads 0 lines and raises no alarm. |
|
||||
| D — restore from the copy | **done** | demo-hp, scratch VMID 9299 on `nvme-scratch`: list 2 s, restore **186 s** (15 GB logical, 14 GB on disk), data read by `pct mount` (not started — starting it would run a second demo-hp controller against the hub). Torn down: VMID, storage entry, DooPlex temporary token + ACL. **Trap found → R-834.** |
|
||||
| E — set-aside deletion via the hub | **done** | Hub v0.128.0 + controller v0.290.0. Red-proofs: no deletion before the delay; a cancelled request deletes nothing; a recovery that does not cancel at the hub fails its test. Live on tester-1: naming the live repo was refused; a planted set-aside dir was deleted after the delay and read back absent. The delay was shortened to 3 min for that test only, by manifest config, logged at start-up, and reverted (the new pod logs no override). |
|
||||
| F — releases, floor, golden | **done** | Hub v0.128.0 deployed. Controller v0.290.0 on both demo boxes. Golden 0.290.0 baked (subagent), round-trip sha matches, leak grep 0 with a control, vouched (agent 0.138.0, min_agent 0.131.0). Floor 0.290.0 SERVED. Golden gate OK. |
|
||||
|
||||
## Claims in the brief that turned out wrong
|
||||
|
||||
1. **"The young superseded copies are the only cause of the refusal"** — right for 2026-10-03. But the brief's own "refuse above the weekly cap" would also have refused every honest window: the hub's cap was 40% and an honest week removes about 41%. I raised the hub cap to half (red-proved), and the cap refusal can still wedge after a long gap (R-833).
|
||||
2. **"PBS can prune `ep0-copy` without touching the sync"** — true. They are separate jobs at separate times. PBS has no dry-run for a prune JOB, so the dry-run was done by reasoning.
|
||||
3. **"A demo box's whole-guest backup is in the copy and restorable with its own key"** — true (demo-hp's own `felhom-pbs.enc`). Two things the brief did not expect: `pvesm add pbs` without `--password` fails, and on failure it **deletes** the key files you placed; and the restored config is the production one (`onboot: 1`, the real drive binds) — R-834.
|
||||
4. **"The household's page still names a deletion date"** — it did, during the countdown. After the date (since v0.289) the page showed nothing while nothing was deleted. It now shows the hub's date, and that date is true.
|
||||
5. A live window that removes snapshots could not be shown today. Nothing was old enough. I did not fake the history.
|
||||
|
||||
## Rows
|
||||
|
||||
Closed: **R-823, R-824, R-826, R-827, R-828, R-830**. Opened: **R-831** (the token, waiting on the operator), **R-832**
|
||||
(roadmap P4), **R-833**, **R-834**. R-95 narrowed further. Register **330 → 328**.
|
||||
|
||||
## Teardown, three layers
|
||||
|
||||
- **Machines:** demo-hp has no VMID 9299, no `tmp-dooplex-copy` entry, and only its own `felhom-pbs.*` priv files. tester-1's sub-account holds `.ssh` (an empty key file) and `felhom-repo`; the planted dir was deleted by the hub. Helper scripts were removed from demo-hp. The drill VM is back on `virgin`.
|
||||
- **Host (DooPlex):** **kept on purpose:** the prune job and GC schedule (Part B). Removed: the temporary restore token and its ACL. Shredded in the scratchpad: tester-1's password, its API key, the seal key copy, the restore token.
|
||||
- **Hub:** v0.128.0 at the 7-day delay (the test override was reverted). Weekly windows ON. Floor 0.290.0, golden 0.290.0 vouched.
|
||||
@@ -0,0 +1,56 @@
|
||||
# REPORT — off-site backups a box cannot delete: built and live (decisions 68–70) — 2026-10-03 (evening)
|
||||
|
||||
Architecture read: `07-backup-architecture.md` (custody, threat rows 9–12, §D), `06-offsite-connectivity.md` §3/§5,
|
||||
`09` §3. Baselines (re-verified): controller `09453325d1b2` (0.288.0), agent `d766666ff8cf` (0.138.0), felhom.eu
|
||||
`9268d9933b8f` (hub 0.126.0 deployed), catalog `917a779cca67`. Register 327 rows, highest R-822. Rulings recorded
|
||||
first as decisions 68–70 (`5188dbd`). Evidence: `documentation/audits/offsite-lock-build-2026-10-03/`.
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | Result | Notes |
|
||||
|---|---|---|
|
||||
| A — migration spike | **done, passed** | An sftp-written repo is listed, extended, restored from (bytes identical), `check`ed and `check --read-data`ed through the pinned `rclone:` key with restic 0.14.0; a delete is refused (403). The same measurements also settled several facts: one key on two lines → the **first** line wins (so the window = prepend a deleting line); an absolute pinned path works; a probe signal (exit 0 + rclone output = pinned; exit 8 = unpinned); the restricted shell's `dd`/`mv`/`cat` (no `test`). |
|
||||
| B — hub registrar + sealed password | **done** — hub v0.127.0 deployed | `internal/offsitekeys`; `consume-password` → 410; password AES-256-GCM at rest (4 live rows sealed, read back as `enc:v1:`); daily key check 07:10 + on demand. 3 red-proofs. |
|
||||
| C — box on the locked key | **done** — controller v0.289.0, then **v0.289.1** | registrar client, pinned probe, `rclone:` transport (hub tier only — the household NAS stays sftp), all four deleting features off the box, window client + fake-snapshot guard. 4 red-proofs. **v0.289.1 fixes a defect v0.289.0 put live (below).** |
|
||||
| D — live, both demo boxes | **done** | demo-felhom 11→13, demo-hp 91→100 (history kept); a delete from each box refused (403), count unchanged; one-file restore and `check` through the pinned key on each; the old endpoint answers 410 to demo-hp's own key. The hand-run key check is clean for both. **Changed:** the "unprefixed test line on a demo sub-account" decoy ran on tester-1's account instead, which already held 3 unpinned lines. The demo passwords are now sealed in the hub, and the hub DB is the only route to them. |
|
||||
| — stop point | **passed** | |
|
||||
| E — window + guard | **mechanics done; a real prune NOT done** | Window 1 on demo-hp ran live: the hub opened it (deleting line first), the guard refused, the window closed in 3 s, the operator was mailed, and the key file read back clean. The refusal is a **design defect** (R-824): any manual run makes the plan remove a same-day snapshot younger than 8 days. Weekly windows stay OFF. |
|
||||
| F — ep0 → DooPlex copy | **done** | ep0: one read-only token (the only change there). DooPlex: an SSH forward (`felhom-ep0-pbs-tunnel.service` — **operator ruling in-session**, because ep0's PBS listens on `wg0` only), PBS remote, datastore `ep0-copy`, nightly pull 05:00 with `remove-vanished false`, Saturday verify, failures to admin@ via Resend (the test mail arrived). First pull: 201 s, 12 GB, 4 of 4 snapshots, matching ep0. Runbook: `runbooks/ep0-datastore-copy.md`. |
|
||||
| Golden + floor | **done** | Golden 0.289.1 baked (subagent, runbook §4.1, token-leak grep 0 with a positive control), round-trip sha matches, vouched (agent 0.138.0, min_agent 0.131.0), floor 0.289.1 SERVED to both boxes. |
|
||||
|
||||
## Claims in the brief that turned out wrong
|
||||
|
||||
1. **"An sftp-written repo reads through rclone"** — confirmed: it was expected, and now it is measured.
|
||||
2. **"The box stores no password today"** — true on disk (it was only in an env var during install), but the box could fetch the password at will; that route is closed now.
|
||||
3. **"One authorized_keys can hold two lines for the same key"** — it can, but only the **first** line counts. That is what makes the window possible with a single key.
|
||||
4. **"The integrity check works through the forced key"** — confirmed (`check`, `check --read-data`, the exclusive lock).
|
||||
5. **"DooPlex has room and a PBS that can pull from ep0"** — it has the room (5.5 TB) and a PBS, but it **cannot reach** ep0's PBS (open on `wg0` only). An SSH forward was added on an operator ruling.
|
||||
6. **"Every deleting feature leaves the box"** — only on the hub tier. The same code serves the household's own SFTP NAS, which keeps box-side retention. A `Transport` flag separates the two.
|
||||
7. **"Abort when the plan exceeds a week's removal"** — that would never prune after the interim. The box takes the oldest snapshots up to the cap instead (disagreement recorded in the code and the CHANGELOG).
|
||||
8. **The guard as written is too strict** (R-824). It also cannot see past-dated poisoning (R-822, residual).
|
||||
|
||||
## Found and fixed in-session
|
||||
|
||||
- **R-825 (v0.289.1):** the provider's rclone prints a NOTICE line on every connection, and restic forwards it into the output. Every `--json` parse failed, so demo-felhom recorded **0 snapshots as measured**, and the hub mailed a **false** `offsite_snapshots_dropped` (11→0) at 17:17. The fix went live 15 min later. It strips the notice, and an unreadable count is never a measured zero. Red-proved.
|
||||
- A shadowed `newPath` in the NAS move-aside path was caught by the existing suite before release.
|
||||
- Stale **unpinned keys** sat in the sub-accounts: 4 on demo-felhom's and 5 on demo-hp's (every reinstall added one). The registrar removed them. tester-1's 3 remain, and the daily check alarms on them (R-826).
|
||||
|
||||
## Deviations, stated
|
||||
|
||||
- **Two controller releases**, against the one-release rule: v0.289.1 fixes a false zero that v0.289.0 put live.
|
||||
- The hub has a **test-only commit after the release** (window-sweep test + a test helper in the store). The deployed v0.127.0 image does not contain it; behaviour is unchanged.
|
||||
- I read tester-1's sealed-era password from the hub DB again for Part A (operator ruling from the morning session; the copy was deleted, the value never printed).
|
||||
- A Hetzner storage API token was printed into this session's transcript while I read `manifests/storagebox.secret.yaml` (a gitignored file; the redaction regex missed the quoted value). **Rotate `HETZNER_TOKEN`** — it is in Secret/storagebox.
|
||||
- The window's red-proofs ran in unit tests and on one live window. A live real prune did not happen (R-824).
|
||||
|
||||
## Records
|
||||
|
||||
- Closed: **R-820, R-821, R-342** (+ **R-825** opened and closed). Narrowed: **R-95, R-822**. Opened: **R-823, R-824, R-826, R-827, R-828, R-830**. Register **327 → 330**.
|
||||
- Decisions 68–70 in `09` §3, CONTEXT, `07`, `06`. `07` threat rows 9/10/12 and `06` §3.6 carry `[FACT]` lines.
|
||||
- Runbooks: `ep0-datastore-copy.md` (new), `secrets.md` (offsite key, DooPlex PBS secrets), `RUNBOOK-manual-build.md` (`pveam update`).
|
||||
|
||||
## Teardown, three layers
|
||||
|
||||
- **Machines:** helper scripts were removed from both demo guests, their containers and hosts (0 left). tester-1's sub-account is back to `.ssh`, `felhom-repo`, with `authorized_keys` byte-identical to the start (sha256 `795e7153…`). The scratch dirs `spike-r436`, `spike-migrate` and the rclone `.config` were removed. The drill VM is reverted to `virgin`, build guest 9100 destroyed, the bake token shredded.
|
||||
- **Host (DooPlex):** **kept on purpose:** `felhom-ep0-pbs-tunnel.service`, the PBS remote/datastore/jobs/notification target (Part F). The scratchpad secret files (sub4 password, ep0 token, Resend key copy) were shredded.
|
||||
- **Hub:** **kept:** v0.127.0, Secret/offsite-secret-key, floor 0.289.1, golden 0.289.1 vouched. Weekly windows OFF. The one-shot grant for demo-hp was consumed.
|
||||
@@ -0,0 +1,99 @@
|
||||
# REPORT — System page, Docker slow lane, crash restart (2026-10-04, late afternoon–evening)
|
||||
|
||||
Brief: "OS updates — a System page …; the Docker engine slow lane built (live-restore ON, decision); a crashed host
|
||||
restarts by itself, with a limit (decision)". Architecture read first: `documentation/architecture/11-os-updates.md`
|
||||
(owner; §5.6, §5.8, §8.1–8.3), `05-hub-architecture.md`, `03-host-agent.md`, `08-alarm-ladder.md`, `07` §6.1. Rulings
|
||||
recorded before the work: `09` §3 decisions 87–89. Evidence for every claim: `documentation/audits/os-docker-crash-2026-10-04/`.
|
||||
|
||||
## Part table
|
||||
|
||||
| Part | What | Result | Evidence |
|
||||
|---|---|---|---|
|
||||
| A | System page + version report (R-852) | **DONE.** The box reports Proxmox + kernel (API) and the wrapper's read-only facts (host Debian, next-boot kernel, held packages, taint, `kernel.panic`, crash guard; guest Debian, Docker, containerd, live-restore); unreadable = `unknown`. Hub: **System** tab on every page — per box ring + switch with buttons, tunnel, host, guest, Docker, last leg; releases per layer; what ring 0 runs; "Approve now" (confirm) and "Approve Docker set" (only when allowed). Hosts gets a Proxmox / kernel column. R-849: guest scanned every pass. Rendered with real data from both demo boxes. | `partA/` (`live/system-page.html`, `hosts-page.html`, facts) |
|
||||
| B | Docker engine slow lane (`11` §5.8) | **DONE.** live-restore on by RELOAD: same ids on 9202 (6/6, by hand — R10 refuses a scratch guest by design), demo-hp (24/24, 7.5 s), demo-felhom (5/5). Ring 0 on both: 29.7.x → **29.8.2**, every id kept (122 s / 99 s whole pass). Approval with the page button under a TEST 0-night wait (logged, reverted): `os-docker-20261004-142842`. **Signed undo** on demo-hp → 29.7.2 (wrapper `authority=signed UNDO`, 24/24 ids, 41 s) and back to 29.8.2 by its ring-0 pass. **demo-felhom as ring 1**: its night pass skips Docker; a signed job with the approved set verified by the wrapper (already current → nothing); the **replayed** job refused ("nonce already seen"); back to ring 0. | `partB/` |
|
||||
| C | Crash restart with a limit (R-851) | **DONE.** Spike (your word before each crash): `kernel.panic=10` + `echo c` → back by itself in 54 s, same kernel; pstore saved nothing; no oops history. Guard built (decision 88, 90–92). Live: crash 1 → 54 s, crash 2 → 53 s and the guard **tripped**, crash 3 → **stayed off** (184 s watched) until you switched it on; the hub mailed `host_crash_guard_tripped` (+3 restart events, 3 household lines); System page red; re-armed by `felhom-crash-guard rearm`. | `partC/` |
|
||||
| D | Releases, golden, records | Agent **v0.142.0** (signed to both demo boxes), hub **v0.132.0**, installer **1.30.0** (tagged, public, verified), `build-golden.sh` 3.1.0. Golden: see below. Records: `11` §5.7/§5.8/§5.9, `00`, `03`, `07`, `08`, decisions 90–94, two runbooks. | `partD/`, git |
|
||||
|
||||
## Claims in the brief that turned out wrong (or half right)
|
||||
|
||||
- **"No box reports its versions"** — half right: every box already sent `pveversion` and `kversion` inside the Proxmox
|
||||
API answer the agent reads each report (`NodeStatus`); nothing stored or showed them. Debian, the next-boot kernel and
|
||||
everything Docker were truly not reported.
|
||||
- **"A reload turns live-restore on with no container restart, on the demo boxes too"** — TRUE, measured: 24/24 and 5/5
|
||||
ids kept. But "prove on 9202 first" could not use the product path: 9202 binds a scratch folder, not the drives, so the
|
||||
wrapper refuses it (R10, by design); proved there by hand with the same two steps.
|
||||
- **"A crash boot can be told apart from a clean one"** — only from a CLEAN one. Not from a power cut or a hard reset:
|
||||
`efi_pstore` is on, yet a real panic saved nothing on demo-hp. The guard counts every unclean stop (decision 90).
|
||||
- **"`echo c` crashes the host and `kernel.panic` brings it back"** — TRUE (54 s, 53 s). But `sysctl -w` does not survive
|
||||
the restart (back to 0), so it must be set at every boot — the guard does that.
|
||||
- **The guard's count** — the brief said both "at 3 … the next crash leaves the box off" (the 4th) and "crashes 3 times
|
||||
within one hour, it stays off" (the 3rd). Built per your words: the 3rd (decision 92).
|
||||
|
||||
## Decisions taken by CC unattended (operator may reverse) — `09` §3
|
||||
|
||||
90 crash signal = clean-stop marker · 91 `panic_on_oops` stays 0, an oops is mailed · 92 the 3rd unclean stop in 60 min
|
||||
stays off; 10 s; 24 h · 93 the wrapper verifies Docker authority against root-owned files · 94 page colours = alarm
|
||||
thresholds.
|
||||
|
||||
## Releases and what was copied by hand
|
||||
|
||||
- **Agent v0.142.0** (`b1746c2`, sha256 `7beb3222…de6`): signed `agent_update` to both demo boxes, both COMPLETED.
|
||||
- **Hub v0.132.0** (`175ecfc`): image tag verified on the pod; ArgoCD Synced/Healthy.
|
||||
- **Installer 1.30.0**: tag `installer-v1.30.0`, both git-sync refs; `https://felhom.eu/scripts/felhom-host-install.sh`
|
||||
serves `SCRIPT_VERSION="1.30.0"`.
|
||||
- **By hand on both demo hosts** (R-840; `partD/copied-by-hand-*.txt`, hashes equal to tag v0.142.0):
|
||||
`/usr/local/sbin/felhom-os-apply`, `/usr/local/sbin/felhom-crash-guard`, `/etc/systemd/system/felhom-crash-guard.service`,
|
||||
`…/felhom-crash-guard-check.service`, `…/felhom-crash-guard-check.timer`, `/etc/felhom/crash-guard.conf`,
|
||||
`/etc/felhom/operator-signers`, `/etc/felhom/os-trust.json` (with `ring0_slow_lane: true` — the demo boxes only);
|
||||
units enabled. Previous wrapper kept as `/root/felhom-os-apply.bak-0.141.1`. A test binary (`felhom-agent-0.142.0-rc1`)
|
||||
ran the debug actions before the release and was removed after.
|
||||
|
||||
## Golden
|
||||
|
||||
- **Re-baked golden 0.292.0** with `build-golden.sh` 3.1.0 (by a helper agent, RUNBOOK-manual-build §4.0/§4.1; the
|
||||
documented publish replaces the same version: pre-delete 204, upload 201). The log shows "Docker engine set PINNED",
|
||||
the six approved versions (docker-ce 5:29.8.2, containerd.io 2.3.6, buildx 0.37.1, compose 5.6.0, …) and
|
||||
"live-restore: on". New sha256 `79a1dce3…d43a`, re-hashed by download in the main session: match. Drill VM destroyed,
|
||||
drill disk back to `virgin`. Evidence `documentation/tests/golden-0.292.0-2026-10-04-rebake/` (commit `208d21d`).
|
||||
- **Re-vouched:** agent 0.142.0 + golden 0.292.0 (new sha), `min_agent` 0.131.0 → `artifacts_set` (17:48). Between the
|
||||
re-bake upload and the re-vouch the hub vouched the old sha — a fresh install would have failed closed; none ran (R-857).
|
||||
- The crash guard is NOT in the golden (it is a host program): the installer 1.30.0 installs it.
|
||||
- The golden waiver stays deleted: the newest controller (0.292.0) has its golden.
|
||||
|
||||
## Register
|
||||
|
||||
Open rows **334 → 333** (333 at the start + R-852 filed first). Closed: **R-852, R-835, R-848, R-849, R-851**; filed and
|
||||
closed: **R-854**. Opened: **R-853** (facts reach the hub ~15 min late after a boot), **R-855** (cosmetic TEST log line),
|
||||
**R-856** (after a crash the household also gets app mails — your choice later), **R-857** (a same-version golden
|
||||
re-bake: the gate shows the first sha; a window until the re-vouch). Narrowed: **R-812** (Docker lane built;
|
||||
kernel left), **R-840** (by hand again). `unproven.py`: unchanged (35 of 55 not walked).
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
- **Machine:** demo-hp and demo-felhom customer guests running, live-restore on, Docker 29.8.2, all apps up. 9202
|
||||
running, live-restore on (its old daemon.json kept as `/root/daemon.json.bak-2026-10-04` in the guest).
|
||||
- **Host:** both hosts run agent 0.142.0, the crash guard ARMED (`kernel.panic = 10`), ring 0, switch ON; the test
|
||||
binary and the id lists removed. demo-hp booted 3 extra times today (the crash test); kernel unchanged (7.0.14-20).
|
||||
- **Hub:** the TEST Docker wait reverted (log: 2 nights); the approval `os-docker-20261004-142842` stays (a real
|
||||
approval of what ring 0 runs). The crash and trip mails for demo-hp were sent on purpose. The hub password copy in
|
||||
the scratchpad shredded at the end. The hub announced the re-arm (`host_crash_guard_rearmed`, 17:47).
|
||||
|
||||
## CI
|
||||
|
||||
Checked by `head_sha` over every page of the Gitea `jobs` endpoint: felhom.eu — all 10 commits from `bee277f` (rulings)
|
||||
to `21986d0` (re-vouch records) **success** (incl. hub `175ecfc` 1265→, installer, manifests, docs); felhom-agent —
|
||||
`b1746c2` (1273, 1274), `f24dce5` (1275), `42af3ab` (1281) **success**. This report's own commit: checked after the push
|
||||
(see the session's final message).
|
||||
|
||||
## Addendum (~18:30) — R-858, found by the operator
|
||||
|
||||
The N100 showed DOWN from 14:18 UTC. Cause: v0.142.0's Docker step restarted dockerd (14:13); live-restore kept the
|
||||
containers running, but `felhom-controller` and `traefik` bind-mount the socket FILE and kept the deleted inode, so the
|
||||
controller could not reach Docker. My Docker health rule passed it (the controller's own check said healthy) — the rule
|
||||
checked the mechanism, not the consequence. Repaired by restarting the two containers (15:57 UTC). Ruling 95: agent
|
||||
**v0.142.1** (`4950030`, sha256 `003f882a…62bd`) restarts only the socket users after a step and fails health when the
|
||||
controller cannot reach Docker. Proven live on demo-hp before the release (signed undo, then forward): `applied, healthy`,
|
||||
guest / controller / traefik on the same socket inode both times. Ring-0 marks were OFF during the fix, back ON after.
|
||||
Both boxes on 0.142.1; vouched for new installs (golden unchanged). Red-proofs 4/4. Register: R-858 opened and closed
|
||||
(still **333** open). Evidence `partE-incident/`. Not touched: Tester 1 shows DOWN for 4 days on the dashboard — a
|
||||
fenced tester box, outside this brief.
|
||||
@@ -0,0 +1,60 @@
|
||||
# REPORT — OS updates build step 1: the guest's Debian fast lane; decision 78; the infrastructure images — 2026-10-04
|
||||
|
||||
Architecture read: `11-os-updates.md` (with C1–C12, §5.4.1, §7.1 — the design; it won wherever it differed from the
|
||||
brief, see below), `03-host-agent.md`, `07` §6.1, `09` §3 decisions 11/12/15/18, `08`. Baselines (re-verified):
|
||||
felhom.eu `1b74ddc0c9` (hub 0.129.0), agent `596238cc2e` (0.139.0), controller `99a1497560` (0.290.0), catalog
|
||||
`917a779cca`. Register 331, highest R-839. Rulings recorded first: `09` §3 decisions 78–80 (`6ed79cd`). Evidence: `documentation/audits/os-guest-lane-2026-10-04/` (parts A–G).
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | Result | Notes |
|
||||
|---|---|---|
|
||||
| A — the snapshot undo first (R-837) | **done — and it FAILED: no snapshot is possible** | PVE refuses any snapshot not named `vzdump` of a guest with host-path binds (mp8/mp9), as the agent's token (which has `VM.Snapshot` + `VM.Snapshot.Rollback`) and as root. By the brief's rule: **no automatic undo built**; the decision is in STATUS (R-842). Steps 2–5 (apply, roll back, re-apply) had nothing to roll back to; 9201 was brought current by the product's own leg in Part G. Thin pool unchanged. |
|
||||
| B — the wrapper | **done** | `felhom-os-apply` (Python 3 stdlib), R1–R13, repair first, snapshot.debian.org fallback, log lines; host layer and slow lane refused. 35 tests; **every refusal red-proved** (13/13). `visudo -cf` OK. **Changed:** Python not shell (a JSON plan cannot be parsed safely in sh — so "shellcheck clean" became `ast`/compile-checked + the suite); one sudoers entry with a plan `mode` instead of a separate `--repair-only`. **The route for existing boxes: none exists** (R-840, with a proposal). |
|
||||
| C — the agent's leg | **done** | After a successful primary whole-guest backup, under the heavy-op gate (red-proved: the gate is held), once per 20 h, 90 s settle. Health rule written and pinned (`HealthVerdict`). Report: full installed set with origins, pending, not covered, restart-needed. Debug action `--selftest=os-update`. 7 leg red-proofs + 2 hook red-proofs. |
|
||||
| D — the hub | **done** — hub v0.130.0 | Rings, per-box switch (default ON), the candidate/approval rule (24 h + 1 night, config), approve-now, events, fleet JSON. 5 approval red-proofs; the `os_update` wire golden byte-identical in both repos. |
|
||||
| E — household line + decision 78 | **done** | Line = hub customer event `os_update_applied` (info: on the household's timeline, not mailed; hu/en in the bundle). **Changed:** there is no box-side event surface, so the hub event is it (R-844). Decision 78 built in controller v0.291.0, red-proved both ways. |
|
||||
| F — infrastructure images (R-838) | **done** | traefik v3.7.13, cloudflared 2026.9.3, filebrowser 1.5.6-stable; breaking changes named (none we use). A release moves all three (9202: ≤ 1.9 s / ≤ 1.5 s; demo boxes: public gap ≤ 19.6 s / ≤ 14.7 s incl. the controller restart). `scripts/check-infra-pins.py` + runbook section. **Changed:** the standing brief `claude/MONTHLY-security-retest.md` lives in the claude.ai project, not the repo — the repo half is the runbook; the project file is the operator's to update. `03` corrected (3 lines). |
|
||||
| G — live proof | **done, one part changed** | Ring 0 on both boxes (53 packages each, healthy); approval with a 2-minute TEST wait (272 packages, auto), then the ruled values back; ring 1 on demo-felhom (exactly the 3 approved versions, nothing newer); a failed health check → `health_failed`, operator mail, household line. **Changed:** "show the rollback" — there is none (Part A). Teardown: no snapshot, no plan files, test config gone, demo-felhom back to ring 0. |
|
||||
| H — release, golden, records | **done** (see Teardown for the golden) | Agent 0.140.0 (signed per box, both demo boxes on it), hub 0.130.0, controller 0.291.0 (floor 0.291.0, MinAgent 0.131.0 declared), installer 1.29.0. `11` §8.1, `00`, `07` §6.1, `03` updated. |
|
||||
|
||||
## Claims in the brief that turned out wrong (named)
|
||||
|
||||
1. **"The agent's token can snapshot and roll back"** — it HAS the rights, but no snapshot of a customer guest is
|
||||
possible at all (bind mounts). Neither the token nor root can.
|
||||
2. **"A snapshot rollback leaves the thin pool clean"** — unmeasurable: there was no snapshot.
|
||||
3. **"A new sudoers line can reach an installed box through the product"** — false. Only the installer writes it;
|
||||
the signed agent update replaces the binary only (R-840). The demo boxes got the wrapper + sudoers BY HAND.
|
||||
4. **"A controller release moves the infrastructure containers"** — TRUE for all three. (I first wrote the opposite
|
||||
for the file browser and corrected it the same hour: its start-up mount sync renders the new image.)
|
||||
5. **"An agent event can reach the household's timeline"** — only through the hub (a hub customer event); the box has
|
||||
no timeline of its own (R-844).
|
||||
6. **"Before each guest update, the box takes a snapshot"** (the one-page summary) — impossible (Part A).
|
||||
7. `11` vs the brief: `11` §5.4.1's `--repair-only` flag was folded into the plan; `11`'s "a missed night waits" holds.
|
||||
|
||||
## Found and fixed live (before the release)
|
||||
|
||||
- `--selftest=os-update` was refused by the flag's allow-list — and so was `--selftest=wgtunnel`, since S3 (R-843,
|
||||
opened and closed; a new test pins every dispatched mode).
|
||||
- The wrapper logged an UPDATED conffile as "kept" (dpkg's two message shapes; fixed + tested).
|
||||
- An app stopped between the inventory and the apply escaped the health check; the baseline is now the start of the
|
||||
leg (fixed + red-proved).
|
||||
- My stopped-app test also made the box mail one `app_start_failed` (a second one was held by the cooldown).
|
||||
|
||||
## Rows
|
||||
|
||||
Closed: **R-837** (measured), **R-838**, **R-726**, **R-843** (opened and closed). Opened: **R-840** (no product route
|
||||
to installed boxes, P2), **R-841** (the agent's cloudflared probe reads a host unit that does not exist, P3), **R-842**
|
||||
(the undo decision, waiting on the operator), **R-844** (household line only on the hub, P4), **R-845** (a pass takes
|
||||
3–4 min, P4). Narrowed: **R-812**. Register **331 → 333**.
|
||||
|
||||
## Teardown, three layers
|
||||
|
||||
- **Machines:** no snapshot on either 9201; no plan files; privatebin restarted and healthy; demo-felhom back to ring 0;
|
||||
both 9201s fully Debian-current (openssl at the approved u3). 9202 runs controller 0.291.0 (from Part F).
|
||||
**Kept on purpose:** the wrapper + sudoers on both demo hosts (installed by hand; the old sudoers saved as
|
||||
`/root/felhom-agent.sudoers.bak-pre-osapply`); agent 0.140.0 (signed update).
|
||||
- **Host (DooPlex):** helper scripts in the scratchpad only; the hub password copy shredded at the end.
|
||||
- **Hub:** v0.130.0 at the ruled 24 h + 1 night (the TEST override reverted and the start log shows no override);
|
||||
both demo boxes ring 0, ON; release `os-20261004-091417` approved (it was approved under the TEST wait — ring 1 boxes
|
||||
will install it; every version in it already runs on both demo boxes). Floor 0.291.0.
|
||||
@@ -0,0 +1,90 @@
|
||||
# REPORT — OS updates, build steps 3 + 4 (2026-10-04, afternoon–evening)
|
||||
|
||||
Brief: "OS updates, build step 3 + 4" (Parts A–G). Architecture read first: `documentation/architecture/11-os-updates.md`
|
||||
(owner), `08-alarm-ladder.md`, `03-host-agent.md`, `07-backup-architecture.md` §6.1. Evidence for every claim:
|
||||
`documentation/audits/os-host-lane-2026-10-04/` (partA … partG).
|
||||
|
||||
## Part table
|
||||
|
||||
| Part | What | Result | Evidence |
|
||||
|---|---|---|---|
|
||||
| A | The tunnel status is true (R-841) | **DONE.** Three states from the guest container + controller v0.292.0's readiness check; `unknown` never alarms; `tunnel_down` after two `not_running` reports. Live on demo-hp: port 7844 blocked → `tunnel_down` mailed 11:52 UTC (2nd report); unblocked → `tunnel_recovered` 12:07 UTC. No new sudoers line. | `partA/` (red-proofs agent/hub/controller, `live/`) |
|
||||
| B | Host fast lane (`11` §8 step 3) | **DONE.** R12 lifted for lane fast / layer host on an appliance only (root-owned install record); R14 refuses kernel/boot/firmware; host step after a healthy guest step; host health rule in `11` §8.2; reboot-needed with first date, never reboots; separate per-layer approved sets. Live: demo-felhom ring 0 → 108 Debian host packages (all Debian origin, checked against apt), healthy; demo-hp ring 0 (nothing pending); 605-package host release approved under a TEST wait (2 m / 0 nights, logged, reverted); demo-felhom as ring 1 installed exactly the 1 version it lacked; back to ring 0 and the ruled 24 h + 1 night. Host undo runbook proved on demo-hp (`tzdata`). | `partB/` (`live/`, `ring1/`, `undo/`, red-proofs) |
|
||||
| C | Fleet view + four alarms (`11` §8 step 4) | **DONE.** `GET /os/fleet`: one line per box, ring, switch, tunnel, per layer release/pending/not-covered/last good leg/reboot-since. Alarms `os_update_stale`, `os_reboot_needed`, `os_ring0_stalled`, `os_not_covered`, hourly, operator-only, each red-proved (9 mutations caught). Numbers in `11` §8.3 / `09` decision 85 as CC-decided. | `partC/` |
|
||||
| D | The leg is fast (R-845) | **DONE.** Nothing to install, both layers: **23.3 s** (demo-felhom), **31.5 s** (demo-hp). Before (agent 0.140.0, guest only, nothing to install): 14.0 s. A 108-package host pass: 70 s. | `partD/`, `partB/live/` |
|
||||
| E | Kernel one-shot spike on demo-hp (R-836), operator's word before each reboot | **DONE (measured).** Secure Boot ON boots the new kernel fine; **`grub-reboot` is NOT a one-shot here** — `/boot` on LVM, GRUB cannot clear `next_entry`, reboot 2 (no command) came back on the NEW kernel. `kernel.panic = 0`. `sp5100_tco` loads and answers (sysfs only, never armed, unloaded). No kernel installed (7.0.14-20 was already there). Left: runs 7.0.14-20, saved default 7.0.14-20, both kernels installed, `GRUB_DEFAULT=saved`. | `partE/` |
|
||||
| F | Docker slow lane design | **DONE (design only).** `11` §5.8. One STATUS decision: `live-restore` on, fleet-wide. | `11` §5.8, STATUS |
|
||||
| G | Releases, golden, records | See below. Agent **v0.141.0 + v0.141.1**, hub **v0.131.0 + v0.131.1** (decision 86 — a second release in each, for a live-found defect), controller **v0.292.0**. Installer: **no new release** (see below). | `partG/` |
|
||||
|
||||
## Claims in the brief that turned out wrong (or only half right)
|
||||
|
||||
- **cloudflared readiness** — it EXISTS (`/ready`, 200 only with a connection), but nothing exposed it: the metrics
|
||||
port was random and there was no health check. A container state alone lies (wrong token: `running`, `/ready` 503).
|
||||
Controller v0.292.0 adds the fixed port + Docker health check; the agent reads it with its existing sudoers line.
|
||||
- **"Find where the install mode is known"** — known in TWO places, and only one is trustworthy: `agent.json`
|
||||
`deployment_mode` (the agent can write it) and the installer's root-owned `state.json` `mode`. The wrapper trusts
|
||||
only the second (decision 84).
|
||||
- **`grub-reboot` with Secure Boot** — Secure Boot was not the problem (it booted fine). The one-shot itself fails on
|
||||
these boxes because GRUB cannot write its state on LVM `/boot`.
|
||||
- **Panic auto-restart** — there is none: `kernel.panic = 0`; a panicked host stays down (R-851).
|
||||
- **Under 60 s** — met (23–32 s). But the leg was ALREADY under 60 s with nothing to install (14 s, guest only); the
|
||||
3–4-minute passes of R-845 were passes that installed something.
|
||||
|
||||
## Releases
|
||||
|
||||
- **Agent v0.141.0** (`cfba0d0`, sha256 `6eaad980…`) and **v0.141.1** (`a6bc3f1`, sha256 `b712f577…`) — signed
|
||||
`agent_update` jobs to both demo boxes (both COMPLETED). v0.141.1 fixes R-846 (host "reboot needed" hid `lxc-start`,
|
||||
and a reboot never cleared it).
|
||||
- **Hub v0.131.0** (`88b0a2e`) and **v0.131.1** (`fd1f985`) — deployed via ArgoCD; image tag verified on the pod.
|
||||
- **Controller v0.292.0** (`09e634d`) — floor raised (`min_controller_version` 0.292.0, `min_agent` 0.131.0); both demo
|
||||
boxes run it; cloudflared recreated once, `healthy`.
|
||||
- **Installer — no new release, deliberately** (recommendation not followed, one line why): the installer fetches
|
||||
`configs/felhom-os-apply` from the VOUCHED agent's tag, the file name did not change and the sudoers line did not
|
||||
change, so vouching agent 0.141.1 delivers the new wrapper to every fresh install with no installer change.
|
||||
- **Copied by hand to both demo hosts** (`partG/wrapper-copied-by-hand.txt`): `/usr/local/sbin/felhom-os-apply` only —
|
||||
first from v0.141.0 (sha `4729769c…`), then from v0.141.1 (sha `51e100ad…`), installed `0755 root:root`, the old
|
||||
file kept as `/root/felhom-os-apply.bak-0.140.0`. `/etc/sudoers.d/felhom-agent` was NOT copied: it already equals the
|
||||
repo's (`02df92d7…` on both).
|
||||
|
||||
## Golden
|
||||
|
||||
- **Golden 0.292.0 baked** (by a helper agent, RUNBOOK-manual-build §4.0/§4.1, same `build-golden.sh` bytes as
|
||||
0.291.0; drill VM 9100 destroyed, drill disk back to `virgin`): sha256 `d6cf8b33…16671`, re-hashed by download in the
|
||||
main session — match. Evidence `documentation/tests/golden-0.292.0-2026-10-04/` (commit `f93738d`).
|
||||
- **Vouched:** agent 0.141.1 + golden 0.292.0, `min_agent` 0.131.0 → `artifacts_set` (`05-vouch.txt`). Fresh installs
|
||||
now get agent 0.141.1, its wrapper, and controller 0.292.0.
|
||||
- **Waiver REMOVED** (`documentation/tests/golden-waiver.yml` deleted): the newest controller (0.292.0) now has its
|
||||
golden, so `golden_currency_gate.py` passes without it. Golden 0.291.0 alone could NOT have retired it — this session
|
||||
released controller 0.292.0, which put 0.291.0 behind.
|
||||
|
||||
## Decisions taken by CC unattended (operator may reverse) — `09` §3
|
||||
|
||||
- **84** — the appliance proof is the root-owned install record.
|
||||
- **85** — alarm numbers 7 / 14 / 7 / 14 days, all configuration.
|
||||
- **86** — a second same-session release of agent and hub (R-846).
|
||||
|
||||
## Register
|
||||
|
||||
Open rows **332 → 333**. Closed: **R-841**, **R-845**; filed and closed the same day: **R-846**, **R-850**. Opened:
|
||||
**R-848** (a held host package is invisible to the hub), **R-849** (the guest "reboot needed since" never clears),
|
||||
**R-851** (a panicked host stays down — operator). Narrowed: **R-836** (GRUB one-shot measured: not a one-shot),
|
||||
**R-812** (host lane built). `unproven.py`: unchanged (35 of 55 not walked).
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
- **Machine:** demo-hp guest 9201 — the port-7844 block removed (`DOCKER-USER` empty, verified); cloudflared
|
||||
`healthy`. demo-hp host — `tzdata` back on 2026c, no holds, no snapshot source left; `sp5100_tco` unloaded; GRUB:
|
||||
`GRUB_DEFAULT=saved`, saved default 7.0.14-20, `next_entry` cleared, `/etc/default/grub` backup at
|
||||
`/root/grub.default.bak-2026-10-04`. demo-felhom host — `tzdata` 2026c (re-installed by its ring-1 run), no snapshot
|
||||
source left.
|
||||
- **Host:** both demo hosts run agent 0.141.1 + wrapper `51e100ad…`; both ring 0, switch ON.
|
||||
- **Hub:** the TEST approval wait reverted (log: 24 h and 1 night); the TEST releases `os-guest-20261004-123933` and
|
||||
`os-host-20261004-124034` stay (they are real approvals of what ring 0 runs). The operator mails `tunnel_down` /
|
||||
`tunnel_recovered` for demo-hp were sent on purpose (Part A). The hub password copy in the scratchpad was shredded.
|
||||
|
||||
## CI
|
||||
|
||||
Checked by `head_sha` over every page of the Gitea `jobs` endpoint (felhom.eu `CLAUDE.md` recipe):
|
||||
agent `cfba0d0` (jobs 1259, 1260), `3bf77c3` (1261), `a6bc3f1` (1262, 1263), `2e2e8f5` (1264) — all **success**;
|
||||
controller `09e634d` (1258) **success**; felhom.eu `88b0a2e` (1256), `fd1f985` (1265) **success**. The last docs pushes
|
||||
(`31bdb4b` and this report's commit): see the session's final message — checked after the push.
|
||||
@@ -0,0 +1,148 @@
|
||||
# REPORT — every app checked again with the fixed persistence check; the remove dialog tells the truth; licence rulings; v0.288.0 + golden (2026-10-02, afternoon)
|
||||
|
||||
Architecture read: `07-backup-architecture.md` §6 (tiers) and §6.5 (kept data — now with decision 67), `09` §3 decisions
|
||||
36, 65, 66, 67. Evidence root: `documentation/audits/persistence-sweep-2026-10-02/`.
|
||||
|
||||
| Part | State | One line |
|
||||
|---|---|---|
|
||||
| **A** — the re-sweep (R-801) | **DONE** | R-788 rule + the app's own fixture seed in the gate; all 58 on the bench; **0 BROKEN**; papra's start-up defect found and fixed (R-803). |
|
||||
| **B** — the remove dialog (R-800) | **DONE** | v0.288.0: `userdata_kept` in the dialog data and both results; the sentence in both languages; red-proofed; live on 9202. |
|
||||
| **C** — release, floor, golden | **DONE** | Floor 0.288.0 (min_agent 0.131.0), both demo boxes in 20 s; golden 0.288.0 baked, round-tripped, vouched; currency gate OK. |
|
||||
| Rulings | **RECORDED** | `09` §3 66 (licences) and 67 (userdata stays); `07` §6.5; STATUS "Before the first paying customer". |
|
||||
|
||||
**Sweep headline (58 templates):** before (2026-08-02, 53 apps, + the new apps' records) CLEAN 38 · UNDETERMINED 11 ·
|
||||
BROKEN 4 — **every one of those decided by start-time writes alone** → now **CLEAN 40 · UNDETERMINED 18 · BROKEN 0 ·
|
||||
INCONCLUSIVE 0** (39 · 19 in the sweep; papra CLEAN after its fix). The August BROKEN four (gramps-web, papra, privatebin,
|
||||
wishlist) were fixed in Campaign 10; none is BROKEN now. **No data-loss finding: no app writes outside a preserved folder.**
|
||||
|
||||
## Claims in the brief, checked
|
||||
|
||||
- **"Every stored verdict came only from start-time writes"** — RIGHT for the August sweep: 0 of 53 probes had sent a
|
||||
request (`ports: []`, `exercise: []` in every `probe.json`). Not quite for today's morning runs of MeTube and
|
||||
Grimmory, which ran after the R-801 fix.
|
||||
- **"The upgrade fixtures can serve as exercisers"** — PARTLY. 51 of 58 apps have one; the gate called it for 16 apps;
|
||||
it decided 2 (metube, privatebin). 13 seeded fine and stayed UNDETERMINED, because the seeds write only to the
|
||||
database — upload / media / cache / redis volumes stay empty (R-807). plex's seed cannot run (a plex.tv claim token).
|
||||
- **"Userdata is never touched by a remove"** — RIGHT, and it was already true before this session: the remove reads
|
||||
`${HDD_PATH}` binds only, pinned since R-442 (`TestRemoveStack_R442_UserdataConventionFromPerAppPath`). What was
|
||||
wrong was only that nothing SAID so. Measured again on 9202 (the video stayed).
|
||||
- **The R-788 decoy as written ("an app that writes nothing → exit 2, red on the old rule")** — the old rule already gave
|
||||
exit 2 for an app that writes nothing. The real gap was an app that writes SOMEWHERE while a declared volume stays
|
||||
empty (old: CLEAN). The decoy was built for that shape and seen red (`A/RP-R788-empty-volume.txt`).
|
||||
|
||||
## Part A — the sweep
|
||||
|
||||
Gate changes (catalog): an empty declared volume after the exercise is UNDETERMINED (R-788, red-proofed; three old tests
|
||||
pinned the old rule and changed with it); when a volume stays empty the gate calls the app's own upgrade-fixture seed
|
||||
(one call through `upgrade_fixtures` / `upgrade_boxport`, as `upgrade-test.py` does — it replaces the morning's
|
||||
MeTube-only exerciser). Bench 9401 on demo-hp, 200G, swap 0, Docker Hub logged in, 8 batches; full table with reasons:
|
||||
`A/sweep/TABLE.md`; verdict lines in the catalog: `audits/persistence-sweep-2026-10-02/verdicts.txt`.
|
||||
|
||||
| app | before | now | what made it write | notes |
|
||||
|---|---|---|---|---|
|
||||
| actualbudget | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| adventurelog | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| audiobookshelf | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| bentopdf | UNDETERMINED (08-02, start writes) | UNDETERMINED | deep pass | nothing was written to any mount and nothing data-classified in any writable layer — the app produced no data |
|
||||
| bookstack | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| calcom | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| calibre-web | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | calibre-web: /app/calibre-web-automated/empty_library — database file(s) touched but byte-identical to the ima |
|
||||
| claper | UNDETERMINED (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture Claper: seed OK — claper: see | claper: declared volume /app/priv/static/uploads is EMPTY after the exercise — the app was not shown to write |
|
||||
| code-server | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| crafty-controller | CLEAN (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture Crafty: seed OK — crafty: POS | crafty-controller: declared volume /crafty/servers is EMPTY after the exercise — the app was not shown to writ |
|
||||
| dawarich | CLEAN (10-01, start writes) | UNDETERMINED | fixture seed OK — `fixture Dawarich: seed OK — dawarich: | dawarich-redis: declared volume /data is EMPTY after the exercise — the app was not shown to write where the t |
|
||||
| docmost | UNDETERMINED (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture Docmost: seed OK — docmost: / | docmost: declared volume /app/data/storage is EMPTY after the exercise — the app was not shown to write where |
|
||||
| emby | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| ghost | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| gitea | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| glance | UNDETERMINED (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| gokapi | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| grafana | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| gramps-web | BROKEN (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture GrampsWeb: seed OK — gramps-w | gramps-web: declared volume /app/secret is EMPTY after the exercise — the app was not shown to write where the |
|
||||
| grimmory | CLEAN (10-02, after R-801) | CLEAN | GET exercise (or start) | grimmory: this container's mounts are all empty while it created entries in ['/', '/app', '/etc'] — benign whe |
|
||||
| home-assistant | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| homebox | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| homepage | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| immich | UNDETERMINED (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture Immich: seed OK — immich: see | immich-machine-learning: declared volume /cache is EMPTY after the exercise — the app was not shown to write w |
|
||||
| jellyfin | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| karakeep | CLEAN (10-01, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| kimai | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| komga | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| mealie | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| metube | CLEAN (10-02, after R-801) | CLEAN | fixture seed OK — `fixture MeTube: seed OK — metube: the | - |
|
||||
| n8n | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| navidrome | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| nextcloud | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| onlyoffice | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | onlyoffice: writable-layer writes at /var/www/onlyoffice/documentserver/sdkjs-plugins/{07FD8DFA-DFE0-4089-AL24 |
|
||||
| opengist | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| outline | CLEAN (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture Outline: seed OK — outline: s | outline: declared volume /var/lib/outline/data is EMPTY after the exercise — the app was not shown to write wh |
|
||||
| paperless-ngx | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| papra | BROKEN (08-02, start writes) | CLEAN after the fix (768M); UNDETERMINED at 256M (OOM) | nothing answered HTTP | R-803: OOM-killed at 256M; fixed |
|
||||
| plant-it | UNDETERMINED (08-02, start writes) | UNDETERMINED | start only (no routed port, no HTTP exercise) | no containers created (compose up rc=18: se/plant-it, repository does not exist or may require 'docker login': |
|
||||
| plex | CLEAN (08-02, start writes) | UNDETERMINED | fixture seed FAILED — `fixture NoRoute: seed returned nothin | plex: declared volume /transcode is EMPTY after the exercise — the app was not shown to write where the templa |
|
||||
| privatebin | BROKEN (08-02, start writes) | CLEAN | fixture seed OK — `fixture PrivateBin: seed OK — private | - |
|
||||
| radarr | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | radarr: writable-layer writes at /media (path suggests state, no database signature — judgement needed): ['mov |
|
||||
| radicale | CLEAN (10-01, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| rallly | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| recipe-importer | UNDETERMINED (08-02, start writes) | UNDETERMINED | deep pass | NOTHING this app wrote landed in ANY folder the template preserves: all 1 mount(s) across 1 container(s) are e |
|
||||
| romm | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| seerr | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| sonarr | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | sonarr: writable-layer writes at /media (path suggests state, no database signature — judgement needed): ['tv' |
|
||||
| sparkyfitness | UNDETERMINED (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture Sparkyfitness: seed OK — spar | sparkyfitness-server: declared volume /app/SparkyFitnessServer/backup is EMPTY after the exercise — the app wa |
|
||||
| tandoor | CLEAN (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture Django: seed OK — tandoor: se | tandoor: declared volume /opt/recipes/mediafiles is EMPTY after the exercise — the app was not shown to write |
|
||||
| termix | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| uptime-kuma | UNDETERMINED (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| vaultwarden | CLEAN (08-02, start writes) | CLEAN | GET exercise (or start) | - |
|
||||
| vikunja | CLEAN (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture Vikunja: seed OK — vikunja: c | vikunja: declared volume /app/vikunja/files is EMPTY after the exercise — the app was not shown to write where |
|
||||
| wanderer | UNDETERMINED (08-02, start writes) | UNDETERMINED | GET exercise (or start) | wanderer: unhealthy; wanderer: declared volume /app/uploads is EMPTY after the exercise — the app was not show |
|
||||
| wger | CLEAN (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture Wger: seed OK — wger: POST /a | wger: declared volume /home/wger/media is EMPTY after the exercise — the app was not shown to write where the |
|
||||
| wishlist | BROKEN (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture Wishlist: seed OK — wishlist: | wishlist: declared volume /usr/src/app/uploads is EMPTY after the exercise — the app was not shown to write wh |
|
||||
| zipline | UNDETERMINED (08-02, start writes) | UNDETERMINED | fixture seed OK — `fixture Zipline: seed OK — zipline: / | zipline: declared volume /zipline/uploads is EMPTY after the exercise — the app was not shown to write where t |
|
||||
|
||||
**REFUSED (BROKEN): none** — so no P1 data-loss row, no stop, no box to name.
|
||||
|
||||
**UNDETERMINED, why:** 13 apps — a second volume (uploads / media / cache / redis / backup) stays empty after a
|
||||
successful seed that writes only the database (claper, crafty-controller, dawarich, docmost, gramps-web, immich, outline,
|
||||
sparkyfitness, tandoor, vikunja, wger, wishlist, zipline — R-807). plex — no seed route (a plex.tv claim token).
|
||||
bentopdf — stateless by design (no volume; a browser-side PDF tool). recipe-importer — stateless converter (its /data is
|
||||
never written). wanderer — unhealthy under the gate, no fixture. plant-it — its image is gone from Docker Hub (R-804).
|
||||
|
||||
**Found and fixed on the way: papra (R-803, P1).** At its 256M limit every fresh install was OOM-killed in its
|
||||
migration and crash-looped (its 26.6.2 step was proven by harness v1, without a memory watch). Measured from birth: peak
|
||||
anon 405 MiB, cgroup peak 483 MiB, ~255 MiB idle. Now 768M (`mem_request` 256M): healthy in 31 s, 0 kills, the seed
|
||||
read back, persistence CLEAN (`A/papra/`). No box ran papra.
|
||||
|
||||
`EXISTING-APPS-GAPS.md` regenerated from the new verdicts (storage 38 → 36 of 53); records 2.1 updated (radicale,
|
||||
karakeep, grimmory, metube CLEAN; dawarich done with the redis volume named; sparkyfitness and wger open).
|
||||
|
||||
## Part B — the remove dialog (controller v0.288.0)
|
||||
|
||||
`GET /api/stacks/{name}/hdd-data`, `POST …/remove`, `DELETE /api/stacks/{name}` carry `userdata_kept` (always a
|
||||
list). Both dialogs and both results: „A fájljaid ezekben a mappákban megmaradnak — a fájlböngészőben látod őket:" /
|
||||
"Your files in these folders stay — you see them in the file browser:" + the folders. The false "no data on a drive" note
|
||||
is gone when userdata exists. Tests + red-proof (two mutants, `B/RP-R800-mutants.txt`); parity fixtures regenerated —
|
||||
additions only, 0 deletions, 109 pages. **Live on 9202** (`B/r800-9202.txt`, endpoint-level, no browser): MeTube from the
|
||||
live catalog, one download; the dialog's data names the folder; the page carries the line in hu and en (with a negative
|
||||
control); remove with data → 409 on 9202 (no host agent, R-442 — the with-data result is pinned by the unit test);
|
||||
remove keeping data → the result names the folder; the video still there.
|
||||
|
||||
## Part C — release
|
||||
|
||||
Controller v0.288.0 (`580b656`), image built from the pushed tree. Floor 0.288.0 + min_agent 0.131.0 → demo-hp and
|
||||
demo-felhom on 0.288.0 in 20 s (`C/floor.txt`). Golden 0.288.0: sha `436fdfa1…`, round trip equal, vouched (golden 0.288.0
|
||||
/ agent 0.138.0 / min_agent 0.131.0), R-120 refused 0.287.0 after; `golden_currency_gate.py`: "OK — the newest released
|
||||
controller has a golden" (`documentation/tests/golden-0.288.0-2026-10-02/`).
|
||||
|
||||
## Rows
|
||||
|
||||
Register **436 → 442**. Closed: R-788, R-790, R-791, R-792, R-795, R-801, R-803. Updated: R-789 (Tandoor — trigger),
|
||||
R-800 (ruled A, built). Opened: R-802 (lawyer review), R-804 (plant-it image gone), R-805 (empty binds not judged),
|
||||
R-806 (https backends / gramps-web first GET), R-807 (seeds that write only the database).
|
||||
|
||||
## Teardown
|
||||
|
||||
- **Bench 9401:** built and destroyed twice; `pct list` 9201 + 9202 only; nvme-scratch back to its baseline (+120 KiB).
|
||||
Docker Hub credentials shredded on DooPlex, demo-hp and the bench; `docker logout` before the destroy.
|
||||
- **9202:** MeTube removed (keeping data — with data refused by R-442), the harness's own test folder removed by hand;
|
||||
controller 0.288.0; live catalog.
|
||||
- **Hub:** the floor and the artifact manifest changed on purpose; nothing else. Tester-2: not touched.
|
||||
@@ -0,0 +1,74 @@
|
||||
# REPORT — 2026-09-30 (day): the last six PostgreSQL apps, the demo boxes by day, the gate's wait seen live, catalog currency, stale rows
|
||||
|
||||
**Tester-2 (read only, hub `GET /configs` + `/hosts`, 12:03 CEST):**
|
||||
- The customer record `Tester-2` (sajatfelhom.hu) exists; config `v0.283.1 MANAGED`.
|
||||
- **No box has registered** — status and version read `—`, no host in `/hosts`.
|
||||
- So no installed apps and no `app_update.unattended` value exist to read yet.
|
||||
|
||||
## The Part table
|
||||
|
||||
| part | outcome | why / where |
|
||||
|---|---|---|
|
||||
| 0 Tester-2 | **done** (3 lines above) | `audits/pg-last-six-2026-09-30/P0/` |
|
||||
| A1 upstream table | **done** — 3 target 18, 3 stay | `audits/pg-last-six-2026-09-30/README.md`, `A1/upstream-notes.md` |
|
||||
| A2 special images | **done, no build** — immich and adventurelog both stay by the rule; immich's 18 images would also swap an extension | same |
|
||||
| A3 fixtures | **done** — sparkyfitness, rallly, outline all through the front door; decision 52 NOT needed | catalog `e6f3ec2` |
|
||||
| A4 two venues + undo | **done** — rallly `25ffd89`, outline `aeb0cd6`, sparkyfitness `1666572` → 18; one undo case each; none `memory_tight` | `bench/`, `box/` |
|
||||
| B demo boxes by day | **done / changed** — no demo box has any of the three moved PostgreSQL apps (nothing installed for it); the Part F apps bookstack + kimai on demo-hp were stepped by the chain press of Part C and read „Naprakész" after | `C/C1-*`, `C/C9-after.txt` |
|
||||
| C gate waits for the leg | **done — PROVEN LIVE** on demo-hp: two deferral lines while the leg stepped two apps, the backup started on the first poll after the leg ended, and succeeded (9.98 GB); config + window put back and read back | `C/README.md` |
|
||||
| D catalog currency | **done** — 25 of 53 behind inside a major, 19 across; night-updatable **28 (+1) → 31 (+1)** after the session | `audits/catalog-currency-2026-09-30.md` |
|
||||
| E stale rows | **done** — R-463 closed, R-446 + R-440 narrowed, STATUS; **E4 changed:** no tag — the ISO's build commit cannot be proven and `installer-v*` is the wrong line → R-730 | `E/` |
|
||||
| F more self-updating apps | **7 of 8 done** — bookstack, kimai, audiobookshelf, n8n, navidrome, grafana, komga; **immich not moved** (first start OOM on the bench, twice → R-732) | `F/README.md` |
|
||||
|
||||
No controller, agent or hub release. The golden is not behind anything new; the waiver runs to 2026-10-04 (a weekly bake is due
|
||||
before then — unchanged by this session).
|
||||
|
||||
## Claims in the brief that turned out wrong (or right), named
|
||||
|
||||
- **Six apps and images as listed** — right.
|
||||
- **"outline and rallly have no front-door seed route"** — WRONG. outline: `POST /api/installation.create` (its self-hosted
|
||||
first run). rallly: its own sign-up, with the e-mail code read from its own `verifications` row in place of a mailbox.
|
||||
- **zipline "closed sign-up, no route" (R-624)** — WRONG for a fresh install: `POST /api/setup` is its first-run route (the old
|
||||
fixture called two other paths). zipline stays on 16 anyway.
|
||||
- **"immich upstream still runs an older major than ours"** — right: upstream runs 14 (ours 16).
|
||||
- **"the PostGIS target carries the same PostGIS major"** — moot (adventurelog stays on 16); every candidate tag is PostGIS 3.x
|
||||
and a dump/load needs no `postgis_extensions_upgrade()`.
|
||||
- **"the night-chain action does not include the whole-guest backup"** — right (its header, and live: the backup came from the
|
||||
quiesce loop's own poll, not the chain).
|
||||
- **"a manual leg sets the flag the gate reads"** — right, and now proven live (the deferral fired during a manual chain).
|
||||
- **"R-446 is stale"** — right (it said READY TO BUILD; the box half shipped in v0.269.x) → narrowed.
|
||||
- **"the installer 1.29.0 is untagged"** — right, but **the fix named was wrong**: `installer-v*` is the host-install script's line
|
||||
(1.28.0 on main), and 1.29.0 is the ISO's; and the ISO was built from an uncommitted tree → R-730, no tag.
|
||||
- **Part C "if the tier is not due, shorten its cadence"** — not needed: demo-hp's local tier WAS due, because the space
|
||||
preflight had refused it 10 times since 2026-09-27 (R-548 note). No cadence was changed.
|
||||
- **Part C "set the backup window"** — the window lives in the product's own setting (`settings.json`, via `POST /backups/window`),
|
||||
not in `controller.yaml`; it was put back by clearing that key with the controller stopped (it had never been set).
|
||||
- **Decision 52 "outline, rallly and zipline never move without it"** — two of them moved without it; the third stays by the
|
||||
upstream rule, not for want of a seed.
|
||||
|
||||
## Decisions I took
|
||||
|
||||
None. Decision 52 was offered and was not needed, so it is not recorded (`09` §3, 2026-09-30 note).
|
||||
|
||||
## Rows
|
||||
|
||||
Register **361 → 364**. Closed: R-463. Narrowed: R-446, R-440. Opened: **R-730** (the ISO build names a commit that is not the
|
||||
image), **R-731** (tag-shape switches + two release checks), **R-732** (immich's first start OOM-killed its database on the bench).
|
||||
Notes added: R-462, R-548, R-624, R-687 (item 4 proven; the manual-chain deferral text).
|
||||
|
||||
## Teardown (three layers)
|
||||
|
||||
- **Machine:** 9202 — every test app removed through the product (volumes none left; four drive folders kept by R-442's refusal,
|
||||
as found before today), `controller.yaml` restored (`git.repo_url` = the live catalog, read back). demo-hp 9201 — `controller.yaml`
|
||||
IDENTICAL to `controller.yaml.pre-gate-day`, `backup_window_start` cleared, the page reads 02:30 / 03:30 / 04:15 / 04:30–08:30,
|
||||
poll 5m; bookstack and kimai stepped (intended: they were moved in the live catalog today and would have stepped tonight).
|
||||
demo-felhom — untouched (it has none of the moved apps).
|
||||
- **Host:** bench LXC 9401 destroyed, its template removed, host temp files removed; drill repo reset to the live `main`
|
||||
(`403a8f5`), image lines identical, `has_actions: False`.
|
||||
- **Hub:** read only; nothing provisioned.
|
||||
- **Consequence stated:** demo-hp's local whole-guest tier ran at 13:37 today, so it next runs at the 2026-10-02 04:30 window.
|
||||
|
||||
## Numbers moved
|
||||
|
||||
`unproven.py --summary`: unchanged — walked 20, partial 17, built 14, missing 4, **not walked 35 of 55**. Evidence left the machines at the end of each phase (every bench edge was copied
|
||||
off before the next started; the box logs are written on DooPlex by the walk tools).
|
||||
@@ -0,0 +1,44 @@
|
||||
# REPORT — probe fix, gate, promotion train, 2026-09-22
|
||||
|
||||
**The full record is `documentation/audits/PROBE-FIX-2026-09-22.md`.** This file is the session
|
||||
report. The shared `REPORT.md` is deliberately not touched (two sessions in this repo clobber it).
|
||||
|
||||
## Not done, or changed from the brief
|
||||
|
||||
1. **FIFTEEN versions moved, not fourteen** — tandoor was re-walked today and became the fifteenth.
|
||||
2. **I nearly dropped `nextcloud`'s MariaDB engine move on a wrong assumption.** Running
|
||||
`check-engine-major.py` against that exact commit ALLOWED it by name under R-469. It moved.
|
||||
Recorded as a decision the operator may reverse.
|
||||
3. **The first push of the moves FAILED CI** (job 877, `15d7c2b`): no PyYAML on the runner, the new
|
||||
gate answered INCONCLUSIVE. Fixed with a degraded mode + five decoys; job **878** = success.
|
||||
4. **bookstack's phase trace on demo-hp is incomplete** — my own 115 s probe run pressed the Update
|
||||
and timed out mid-flight. Said plainly rather than presented as a full trace.
|
||||
5. **The four apps were pressed on ONE demo box, not two.** `demo-felhom` has only `opengist`
|
||||
installed. I did not install four apps on it to satisfy the instruction.
|
||||
|
||||
**No brief claim turned out wrong.** All three probe faults were verified against the files before
|
||||
any edit and all three were exactly as stated.
|
||||
|
||||
## What ran
|
||||
|
||||
- **Part 1** — the three probes fixed in one commit; red-proofed live on 9202 in **both**
|
||||
directions through the product; tandoor's failed edge re-walked and now `done` at +41.1 s; a new
|
||||
`--fast` catalog gate with four red-proofs and ten decoys; the never-judged templates counted.
|
||||
- **Part 2** — fifteen moves, one commit per app, `catalog_since` today, all gates green, CI green
|
||||
by job id; the guarded Update pressed on four apps on demo-hp, all four `done`.
|
||||
|
||||
## What shipped
|
||||
|
||||
- `app-catalog-felhom.eu` **@1ad1f34** — three probe corrections, fifteen version moves, one new
|
||||
gate, `test_gate_decoys.py` 51 → 56 cases. CI job **878 = success**.
|
||||
- `felhom.eu` — this report, the audit, the evidence, R-630/R-631/R-632, R-618 CLOSED, `09` §8
|
||||
limitation 8, the capability map's guarded-update row, and `STATUS.md`.
|
||||
- **No controller, agent or hub code.** The brief forbade it and none was needed.
|
||||
|
||||
## What is owed
|
||||
|
||||
- **What `verifying` does for a stack with no probe target** (R-630) — answerable on demo-hp, where
|
||||
`paperless-ngx` is installed. Not run.
|
||||
- **A live probe reading for five templates** no static rule can judge (R-631).
|
||||
- **28 templates never deployed by any drill** (R-632) — the rotation's queue.
|
||||
- **`vikunja` has no compose healthcheck at all**, so its probe has no oracle in either direction.
|
||||
@@ -0,0 +1,13 @@
|
||||
# REPORT — R-672/R-673: restore test off, 9201 repaired, agent v0.133.0 + hub v0.124.0, controller v0.270.0
|
||||
|
||||
Full record: `documentation/audits/r672-2026-09-24/README.md` (opens with "not done, or changed" and the brief's
|
||||
wrong claims).
|
||||
|
||||
- **Hub v0.124.0** (`2d24931`, manifest `068e065`, live): a thin pool is judged on the worse of data and metadata,
|
||||
critical at 90 %; `storage_fill_*` cooldown per pool (host/storage), 6 h. Two red-proofs; suite green.
|
||||
- **Agent v0.133.0** released (tag `9bdb4da`), not delivered; **controller v0.270.0**, floor 0.270.0, both demo boxes
|
||||
arrived.
|
||||
- **Docs:** `03` §8 (the preflight, the operator rulings, the -1 trap), `08` §6.2 (thin pool at 90 %), capability map
|
||||
row, register (R-684 opened; R-669/R-674/R-679/R-681 closed; R-672/R-673 updated), CONTEXT, STATUS.
|
||||
- **Register:** 344 → 341 rows (685,662 → 685,148 B).
|
||||
- `unproven.py --summary`: see the session's final message (unchanged — 35 of 55 not walked).
|
||||
@@ -0,0 +1,90 @@
|
||||
# REPORT — the operator's three rulings of 2026-10-01 built (day session)
|
||||
|
||||
Evidence: `documentation/audits/rulings-2026-10-01/` (A–D, T, tools) · `documentation/audits/retest-2026-10/` (the monthly
|
||||
run) · golden: `documentation/tests/golden-0.285.0-2026-10-01/`.
|
||||
Architecture read: `09` §3 decisions 30, 45, 52, 53 and §6.5; `03-host-agent.md` (the controller swap) with agent
|
||||
`internal/localapi/controllerswap.go`; `07` (R-698); `runbooks/monthly-floating-retest.md`; `audits/night-rulings-2026-09-30/`.
|
||||
Baselines (live Gitea ~07:10 CEST): controller `a70c398` (0.284.2), agent `d766666` (0.138.0), felhom.eu `3159892`, catalog
|
||||
`efd492d`. Register 383 rows by `register_shape_gate`'s method; highest id R-748; last decision 53.
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | done / not done / changed | why |
|
||||
|---|---|---|
|
||||
| **Rulings 54–56** | **done** — recorded first in `09` §3 and CONTEXT (`2076bf9`) | before any work |
|
||||
| **A — the full re-test** | **done** — nextcloud `3b59dfb` and sonarr `1a37032` re-tested on both venues and written, pushed through the pre-push gates | first start refused in one minute: the script checked the bench for its own files before copying them (R-749, fixed `9e53205`) |
|
||||
| A1 runbook | **done** — every app by default, `--engines-only` the switch, the standing brief, cost | — |
|
||||
| A4 linuxserver cost | **done** — see below | — |
|
||||
| A5 STATUS standing line | **done** — "last run 2026-10-01, next due ~2026-11-01" | — |
|
||||
| **B — two controller versions** | **done — controller v0.285.0**, floor 0.285.0 (MinAgent 0.131.0 declared) | measured first: the agent rolls back to the RUNNING image |
|
||||
| B3 live | **done** on 9202 (version-order fallback) and both demo boxes (the swap record) | 9202's self-update is off (no hub), so the record path was shown on the demo boxes |
|
||||
| — R-751 | **added** — a nil-stack panic in the update clean-up, found by the full suite, fixed in the same release | a panic in a goroutine ends the controller |
|
||||
| **C — mealie** | **done — decided by CC unattended (decision 57):** `SECURITY_USER_LOCKOUT_TIME=1`, catalog `a4597cd`; 9202 proof 120 min | no fix stops an hourly renewal without changing the login name (option d, left open) |
|
||||
| C other apps | **done (read, not measured)** — four more lockable: R-752 | source reading by a sub-agent at each pinned tag |
|
||||
| **D1 — R-746** | **done** `804884a`, red-proofed (unit + live registry) | — |
|
||||
| **D2 — R-744** | **done** `9fc7052`, proven on 9202 at 1.10.1 | outline has no newer release, so no step |
|
||||
| **E — release, floor, golden** | **done** — demo boxes on 0.285.0 within ~6 s; golden 0.285.0 baked, round-trip identical, vouched; the gate prints **OK** | — |
|
||||
|
||||
## Claims in the brief that turned out wrong (or right)
|
||||
|
||||
1. **"`previousImage` … handed to the agent's `SwapController`"** — **wrong.** `SwapController(ctx, target)` carries the
|
||||
target only; `previousImage` goes into `update-state.json`. The agent reads `/etc/felhom-controller-image` when the swap
|
||||
begins and rolls back to THAT — the running image. On all three boxes it equals `<image>:<current version>`, so the
|
||||
practical outcome matches; the mechanism does not.
|
||||
2. **"The sweep skips every controller image today"** — **right**, twice over: the sweep's repositories never include the
|
||||
controller's, and it skips any repo containing `felhom-controller`.
|
||||
3. **"nextcloud and sonarr still differ"** — **right**, and only those two (bookstack, radarr, code-server did not differ today).
|
||||
4. **"mealie's lock is per account"** — **right** (`login_attemps` / `locked_at` on the user). Also: the lock outlives its
|
||||
hours until mealie's HOURLY job resets the counter, so `1` means 1–2 h (measured 120 min).
|
||||
5. **"The golden's image is not needed after first boot"** — right **after the first swap**; until then it IS the running
|
||||
image (kept as such). A whole-guest restore brings its own Docker store (`mp0 backup=1`).
|
||||
6. **"Is an old controller tag still in the registry?"** — **no, below 0.213.0** (2026-08-12): `0.201.0` answers 404. Nobody
|
||||
recorded what removed them (R-750). This session deleted no registry tag.
|
||||
7. "Live on 9202 … after the release" for the record path — 9202 has self-update off; the record path ran on the demo boxes.
|
||||
|
||||
## Part A — the run
|
||||
|
||||
| app | result | bench | box |
|
||||
|---|---|---|---|
|
||||
| nextcloud `34.0.4-apache` | **DONE** — written `3b59dfb` | 05:19–05:32 UTC (proven, peak 23.7 %) | ~5 min: old digest installed, seeded, the guarded Update ran the re-test step, new digest running, read back, badge "Naprakész" |
|
||||
| sonarr `4.0.20` (linuxserver) | **DONE** — written `1a37032` | 05:38–05:50 (proven, peak 13.9 %) | ~3 min, same steps |
|
||||
|
||||
**Monthly cost.** ~17 min per app (bench ~13 incl. the 10-minute watch, box ~4) plus ~15 min to set up and tear down. The
|
||||
four linuxserver apps with ladders (bookstack, radarr, sonarr, code-server) are rebuilt weekly, but the monthly run tests
|
||||
only the day's digest — **at most 4 re-tests a month from them (~70 min of bench, ~16 min of 9202), not 16.** Acceptable.
|
||||
|
||||
## Part B — images before → after
|
||||
|
||||
| box | controller images | Docker images (`system df`) | `/var/lib/docker` used |
|
||||
|---|---|---|---|
|
||||
| 9202 | 5 → 2 | 3.17 → 3.10 GB (layers are shared) | 3.5 → 3.4 GB |
|
||||
| demo-hp 9201 | 84 → 2 (82 deleted) | 15.2 → 11.65 GB | 18 → 15 GB |
|
||||
| N100 9201 | 76 → 2 (74 deleted) | 6.07 → 1.11 GB | 6.0 → 1.2 GB |
|
||||
|
||||
Each kept 0.285.0 (running) and 0.284.2 (previous). Red-proofs (`B/B1-red-proofs.txt`): previous dropped → 0.284.2
|
||||
deleted; record ignored → 0.283.1 deleted; no swap check → deleted while swapping; no in-use check → a used image deleted;
|
||||
no success check → a failed swap's image named previous; no nil-stack check → panic. The first in-use red attempt
|
||||
removed a line and did not build; redone with the condition disabled (recorded).
|
||||
|
||||
## Part C — mealie
|
||||
|
||||
Settings (v3.28.0 source): `SECURITY_MAX_LOGIN_ATTEMPTS` 5, `SECURITY_USER_LOCKOUT_TIME` 24 (hours), per account, lifted
|
||||
by an hourly job; `POST /api/admin/users/unlock` exists but the only admin is the locked account. Fix and 9202 proof: see
|
||||
decision 57 (`C/C1-mealie-lockout-1h.txt`: right password 200 → 5 wrong 401 → 423 → right password 423 for 120 min → 200;
|
||||
a wrong one 401 after). Other apps (R-752): calibre-web-automated (per username, 3/min and 40/day), wger (per IP = traefik's,
|
||||
30 min, everyone), Grafana (per account, 5 min), BookStack (e-mail|IP, 60 s); gokapi and claper cannot.
|
||||
|
||||
## Rows
|
||||
|
||||
**383 → 387.** Opened R-749 (re-test start), R-750 (old registry tags gone), R-751 (update clean-up crash), R-752 (four
|
||||
lockable apps). Closed R-743, R-744, R-745, R-746, R-749, R-751. Narrowed R-747.
|
||||
|
||||
## Teardown
|
||||
|
||||
- **Machine:** 9202 back on the live catalog (`repo_url` read back), the same six containers as at the start, controller
|
||||
0.285.0 (the release); the apps this session installed (nextcloud, sonarr, outline, mealie) removed through the product.
|
||||
Bench 9401 destroyed with its template. Drill VM: CT 9100 destroyed, token/runner/log shredded, qemu exited, reverted to
|
||||
`virgin`. Drill catalog reset to live (`a4597cd`), image lines identical.
|
||||
- **Host:** demo-hp `pct list` = 9201, 9202 (as at the start); the N100 untouched except the floor's controller update.
|
||||
- **Hub:** two form saves — the floor (0.285.0, MinAgent 0.131.0) and the vouch (golden 0.285.0); one floor attempt without
|
||||
the credential answered 302 `/login` and stored nothing.
|
||||
@@ -0,0 +1,73 @@
|
||||
# REPORT — 2026-09-29 evening: sign-up locked twice; "close sign-up now"; wanderer closable; three more probes
|
||||
|
||||
Architecture read first: `09` §3 decisions 45–49, `01-topology-and-trust.md` §5, `audits/gate-rollout-2026-09-29/`,
|
||||
rows R-714, R-715, R-716. Controller **v0.282.0** (one release), floor 0.282.0, both demo boxes on it. Catalog
|
||||
`6446197`. Evidence: `documentation/audits/signup-lock-2026-09-29/` (A own switch, B tricks, C demo boxes, D wanderer,
|
||||
E probes, redproofs).
|
||||
|
||||
## The Parts
|
||||
|
||||
| Part | Step | State | Note |
|
||||
|---|---|---|---|
|
||||
| — | decisions 48, 49 recorded first | done | `09` §3 |
|
||||
| A1 | own switch per app (spike) | done | 9 of 11 have an env switch (below); opengist, wishlist only in their database (R-717) |
|
||||
| A1 | "can be set only after the first admin" | measured | 6 of 9 refuse the household's own first account while on; 3 (calcom, gitea, gramps-web) do not |
|
||||
| A2 | `after_setup:` (controller) | done | env merged + one `compose up -d` when the gate opens; command form with after_install's argv rules; an old compose reported, not faked; retries every 30 min at most |
|
||||
| A2 | live | done | all 9 apps: record `ok`, running container carries the switch, a stranger straight at the app refused |
|
||||
| A3 | the window with an own switch | done — **lift and restore** | homebox: the window turned the switch off (one restart), a family member joined; after 15 min both locks back, a stranger refused through the web and straight at the app |
|
||||
| B | trick table | done, **changed** | termix's router ignores case: `/users/CREATE` got past the old prefix block (its own switch refused it). All blocks now `PathRegexp((?i)…)`, slash-tolerant. Final: 113 tries on 11 apps, 0 got in, 106 refused by the block, 7 were vikunja's plain HTML page (its API blocked) |
|
||||
| B | login / reset / sharing | done | household sign-in on 7 apps; password reset and share routes reach the apps, not the block |
|
||||
| C | "close sign-up now" (controller) | done | lock record `opened_by: close-signup`, block, own switch; offered once; never a gate |
|
||||
| C3 | demo boxes | done, **changed** | pressed on demo-hp adventurelog + opengist, demo-felhom opengist. "Before" proven WITHOUT making an account (the app answered an invalid sign-up with its own validation) — the apps' admin passwords are the operator's, so a test account could not be deleted through their admin pages. After: refused; login pages answer. adventurelog's backend restarted once (~30 s) for its own switch, same images (R-718: the card does not say so) |
|
||||
| D | wanderer | done, **changed** | no gate: one DB URL for server and browser (measured), and no first-admin screen for a stranger (PocketBase's installer needs the log link). Closed by "close sign-up now": two holes found and closed (collection id `_pb_users_auth_`, collection name in capitals); proven on 9202 |
|
||||
| E1 | probe reads lists / a done status | done | RP29, RP30 |
|
||||
| E2 | ghost, home-assistant, gramps-web | done | before/after measured on fresh installs; each gate opened by itself within seconds of the setup; a press before the setup refused on all three |
|
||||
| E3 | catalog gate `probe-measured` | done | 5 decoys seen failing; it caught immich/n8n/audiobookshelf (their note sat one block too high) |
|
||||
|
||||
### Part A / B — one row per app
|
||||
|
||||
| app | own switch | blocks the household's first account? | own switch live after the gate | tricks (8 shapes per route) |
|
||||
|---|---|---|---|---|
|
||||
| adventurelog | `DISABLE_REGISTRATION` | yes | on; `is_disabled: true` | 0 in |
|
||||
| calcom | `NEXT_PUBLIC_DISABLE_SIGNUP` | no | on; "Signup is disabled" | 0 in |
|
||||
| gitea | `GITEA__service__DISABLE_REGISTRATION` | no | on; "Registration is disabled" | 0 in |
|
||||
| gramps-web | `GRAMPSWEB_REGISTRATION_DISABLED` | no | on; 405 "Registration is disabled" | 0 in |
|
||||
| homebox | `HBOX_OPTIONS_ALLOW_REGISTRATION` | yes | on; "user registration disabled" | 0 in |
|
||||
| papra | `AUTH_IS_REGISTRATION_ENABLED` | yes | on | 0 in |
|
||||
| sparkyfitness | `SPARKY_FITNESS_DISABLE_SIGNUP` | yes | on | 0 in |
|
||||
| termix | `ALLOW_REGISTRATION` | yes | on — and it caught the case hole | 0 in |
|
||||
| vikunja | `VIKUNJA_SERVICE_ENABLEREGISTRATION` | yes | on | 0 in |
|
||||
| opengist | none reachable (DB) | — | block only | 0 in |
|
||||
| wishlist | none reachable (DB) | — | block only | 0 in |
|
||||
| wanderer | `PUBLIC_DISABLE_SIGNUP` (web only) | — | on after the press | 0 in after the fix (2 holes before) |
|
||||
|
||||
## Claims in the brief that turned out wrong (or right), named
|
||||
|
||||
- **The three env names given from memory** — **right**: gitea `GITEA__service__DISABLE_REGISTRATION` (through its
|
||||
env-to-ini), homebox `HBOX_OPTIONS_ALLOW_REGISTRATION`, papra `AUTH_IS_REGISTRATION_ENABLED` — each measured working.
|
||||
- **"A native setting can be set only after the first admin exists"** — **true for 6 of 9**, **wrong for 3** (calcom,
|
||||
gitea, gramps-web make their first admin by another route).
|
||||
- **"The address block can be passed by case or encoding tricks"** — **right for case, wrong for encoding**: termix
|
||||
(router ignores case) and PocketBase (collection name in any case, and by id) got past the old blocks; percent-encoding,
|
||||
double slashes, trailing slashes and query strings never did (traefik decodes and cleans before matching).
|
||||
- **"wanderer separates its internal and public database URL"** — **wrong**: one `PUBLIC_POCKETBASE_URL` for both.
|
||||
- **"gramps-web answers 405 only after its setup"** — **right**: 200 with an owner token before, 405 after.
|
||||
|
||||
## Also found
|
||||
|
||||
- The scratch box's Docker disk was full of old images (1 GB free) — the install check refused correctly; 44 unused
|
||||
images removed by name (no prune).
|
||||
- `test_gate_decoys.py` had stopped running any case after docmost moved to PostgreSQL 18 (a typed "16"); fixed to read it.
|
||||
- **My own slip:** I restarted the scratch box's controller while a removal job was running; one removal was cut off
|
||||
(502). Re-run; nothing left behind.
|
||||
|
||||
## Rows
|
||||
|
||||
Closed: R-714, R-715, R-716. Opened: R-717 (opengist/wishlist own switch in their DB), R-718 (the close card should say
|
||||
the app restarts). **Register 353 → 355 rows.**
|
||||
|
||||
## Teardown
|
||||
|
||||
Machines: 9202 — every test app removed through the product; no gate or block file left; back on the live catalog; the
|
||||
drill catalog reset. Demo boxes — the floor, and the three "close sign-up now" presses (ruled). Host: nothing. Hub: floor
|
||||
0.282.0. ep0: untouched.
|
||||
@@ -0,0 +1,52 @@
|
||||
# REPORT — THE TWENTY-EIGHT, 2026-09-22
|
||||
|
||||
**The full record is `documentation/audits/DRILL-the-28-2026-09-22.md`.** The shared `REPORT.md` is
|
||||
deliberately untouched (two sessions in this repo clobber it).
|
||||
|
||||
## Not done, or changed from the brief
|
||||
|
||||
1. **Interventions: SEVEN, over the brief's limit of five** — and six of the seven were my own
|
||||
harness, not the product. Three driver bugs fixed mid-run, two deliberate method changes, one
|
||||
deadlocked waiter. The seventh was the product's: three leftovers it could not clear.
|
||||
2. **There is no `requires:` key in `.felhom.yml`.** The constraints live under `resources:`
|
||||
(`needs_hdd`, `pi_compatible`), and 9202 met all of them — nothing was skipped for a resource
|
||||
reason.
|
||||
3. **The brief's file-leg list is wrong.** Read from `07` §6.2 as the brief itself instructs, only
|
||||
four of the 28 are class A: `calibre-web`, `immich`, `komga`, `paperless-ngx`. `jellyfin`, `plex`
|
||||
and `emby` are class B because their only bind is a `:ro` media mount.
|
||||
4. **The brief's database list is incomplete** — `immich` also carries PostgreSQL and redis, and
|
||||
`wanderer` carries meilisearch.
|
||||
5. **`plant-it` cannot be installed at all, by design** (`lifecycle: abandoned`), refused by the
|
||||
product's lifecycle gate — the only such template in the catalog, proven live for the first time.
|
||||
6. **A REFUSED restore is recorded as its own verdict, not as a failure.** The first version of the
|
||||
harness collapsed them and mislabelled `calibre-web`, where the product had done the right thing.
|
||||
7. **Verified true by looking, not assumed:** the drill repo's Actions are off (**47 CI jobs before
|
||||
the first push, 47 after, all night**); `repoint_drill.py` still works; 9202 had the capacity.
|
||||
|
||||
## What ran
|
||||
|
||||
All 28 walked: deploy at the live pin → seed through the app's own front door → read back → backup →
|
||||
the guarded Update where a real within-a-major edge exists → **restore and read back again** → remove
|
||||
and a 60-second check. Plus both side jobs.
|
||||
|
||||
**26 of 28 deployed · 6 proven · 5 inconclusive · 14 no upstream edge · 1 failed honestly ·
|
||||
2 could not deploy · 21 restored · 2 correctly refused a restore.**
|
||||
|
||||
## What shipped
|
||||
|
||||
- `felhom.eu` — this report, the audit, the evidence, R-633 and R-634 opened, R-630 **raised to P1**
|
||||
by measurement, R-631 and R-632 **closed**, `09` §6.4 leg F and §8.8, the capability map, the
|
||||
rotation file (28 lines rewritten + tandoor corrected), and `STATUS.md`.
|
||||
- `app-catalog-felhom.eu` — **nothing.** No fixture was ready to ship tonight and no template moved.
|
||||
- **No controller, agent or hub code.**
|
||||
|
||||
## What is owed
|
||||
|
||||
- **R-630's fix**: a `container_name` on paperless-ngx's webserver, or a `verifying` phase that
|
||||
treats "no probe target" as something other than a failure. Both are decisions, not clean-ups.
|
||||
- **R-634's mechanism** — `runComposeDeploy`'s pin write was not read; the brief forbade product code.
|
||||
- **Fixtures for 20 of the 28** that have no non-browser route yet, and a second look at `kimai`
|
||||
(its own `user:create` succeeded but `user:list` did not show the user) and `jellyfin` (the
|
||||
`/Startup/User` wizard route that worked for `emby` did not).
|
||||
- **The six edges that reached `done` with no data proof** — `code-server`, `crafty-controller`,
|
||||
`komga`, `plex`, `rallly` — need a fixture before they can be promoted.
|
||||
@@ -0,0 +1,60 @@
|
||||
# REPORT — UPDATE NIGHT, 2026-09-21
|
||||
|
||||
**The full record is `documentation/audits/DRILL-update-night-2026-09-21.md`.** This file is the
|
||||
session report: what ran, what shipped, what is owed.
|
||||
|
||||
|
||||
## Not done, or changed from the brief
|
||||
|
||||
**Nothing in the brief was skipped.** Five things were changed, re-run or measured on a different
|
||||
venue, each named with its reason in the audit's own first section. In short: the PostgreSQL
|
||||
rehearsal ran on guest 9202 rather than a separate harness LXC; the `pg_upgrade` route was not run
|
||||
(it needs an image that does not exist here); B5's `safety-dump` cut MISSED first and was recorded
|
||||
as a miss before being retried and hit; B8 and the rehearsal were re-run after B1's own precondition
|
||||
swept the app they needed; and the harness RUNS of the new catalog edges are owed although the code
|
||||
is shipped.
|
||||
|
||||
**One thing the brief asked for that this venue cannot produce at all:** every event and every
|
||||
customer mail. Guest 9202 runs `hub.enabled: false` and the notifier returns before it logs
|
||||
(**R-620**). Stated on every row of the alarm truth table rather than left blank.
|
||||
|
||||
## What ran
|
||||
|
||||
- **Phase 0** — the fleet floor to **0.261.0** (both demo boxes in **13 s**, hub `SERVED … from
|
||||
declared`); a private **drill catalog** with a positive and two negative controls; a throwaway
|
||||
**image store** on the scratch guest; capacity measured; the upstream drift re-run.
|
||||
- **Phase 1** — 21 edges across 19 apps walked on guest 9202 through the product's
|
||||
own guarded Update, each seeded and read back through the app's own front door.
|
||||
- **Phase 2** — the two database engines across a major, through the real Update button.
|
||||
- **Phase 3** — the bad days, B1–B9.
|
||||
- **Phase 4** — the morning after.
|
||||
- **Phase 5** — teardown, three layers, plus Gitea.
|
||||
|
||||
## What shipped
|
||||
|
||||
- `app-catalog-felhom.eu` **@4463243f2e09** — TEST CODE ONLY: four new harness fixtures and seven
|
||||
new edges (U1–U7). No template changed; no `image:` line moved. Gates green; CI job **830 =
|
||||
success**.
|
||||
- `felhom.eu` — this report, the audit, the evidence, the register rows, the architecture updates,
|
||||
and one correction to `update-arc-gaps-2026-09-21/00-api-recipe.md` (the app page is `/apps/<n>`,
|
||||
not `/app/<n>`) and one FIX to `unattended-caller.py` (R-623).
|
||||
- **No controller, agent or hub code was written.** The brief forbade it and none was needed.
|
||||
|
||||
## What is owed
|
||||
|
||||
- **The harness RUNS of edges U1–U7.** The code is in the catalog repo and the gates are green; the
|
||||
runs, and with them the per-app ABORT answers, have not been performed.
|
||||
- **A cut inside `starting` itself.** Both EARLY phases were cut tonight; `starting` lasts well under
|
||||
a second and still needs an in-process fault injector rather than a faster shell.
|
||||
- **The mail half of Q4**, and every event: structurally unmeasurable on this venue (R-620).
|
||||
- **Fixtures for the four inconclusive apps** — and for two of them (vaultwarden, zipline) the honest
|
||||
maximum is `inconclusive` while the catalog rightly closes their sign-up (R-624).
|
||||
- **What re-created the removed `navidrome` container** (R-626): observed, not diagnosed, because the
|
||||
controller had restarted and its log no longer reached that moment.
|
||||
- **`wger`'s own edge** — it was deployed only to measure its probe and was then removed.
|
||||
|
||||
## The live catalog
|
||||
|
||||
**Never touched with a broken, dummy, cross-repo or engine-major reference — not once, not for
|
||||
thirteen minutes.** Its `main` moved only for the harness commit above, which changes `scripts/`
|
||||
and zero `image:` lines; the teardown diff proves every `image:` line identical to the drill copy.
|
||||
@@ -0,0 +1,139 @@
|
||||
# REPORT — the box tells visitors apart; the permanent-gate spike; SparkyFitness; R-772/R-773; release + golden (2026-10-01 night)
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | Result | Notes |
|
||||
|---|---|---|
|
||||
| **A** — visitors apart (build) | **DONE, with one change to the brief's sketch and one leg unmeasured** | The sketch plus 2 costs paid in the same rollout: a header clean-up at traefik, and 19 catalog router resets for the apps that read the LEFTMOST address. The brief named neither. ep0's one request was refused by Cloudflare before the box (R-779). |
|
||||
| **B** — permanent family gate (spike) | **DONE — PASS** on exit items 1–5; costs measured | Exit test committed before the run. Build plan: 2 sessions. Waits for the operator (R-780). |
|
||||
| **C** — SparkyFitness record | **DONE, with 7 rows open** | Bench (9401 rebuilt and destroyed) and 9202 measured. **Its licence forbids commercial use** (R-784, operator). |
|
||||
| **D** — R-772, R-773 | **DONE** | Both red-proven and live on 9202. R-772 needed a second fix found live (0.286.0 → 0.286.1). |
|
||||
| **E** — release, floor, golden | **DONE** | Controller 0.286.1 floored (min_agent 0.131.0); both demo boxes took it and reconciled traefik/cloudflared by themselves. Golden 0.286.1 baked, round-tripped and vouched; the currency gate is OK, not waived. |
|
||||
|
||||
Baselines (re-verified): controller `c1b123c64955`, agent `d766666ff8cf`, felhom.eu `bc9c7b49346b`, catalog `94477cba435a`.
|
||||
Ends at: controller `a4a753b` (v0.286.1), catalog `de0a9bd`, felhom.eu (this commit). Agent untouched.
|
||||
Architecture read: `01-topology-and-trust.md` §5 and §7 (both corrected), `09` §3 decisions 45–49, 57–62; decision 63 added
|
||||
first (operator ruling).
|
||||
|
||||
## Claims in the brief that turned out wrong (or right), named
|
||||
|
||||
- *`clientIP` takes the leftmost hop.* **Right** (`claim.go:232`, pre-0.286).
|
||||
- *A stranger can lock the household out of the dashboard.* **Right, through the tunnel only.** Every tunnel visitor shared
|
||||
cloudflared's address as the key; a LAN visitor always had its own key.
|
||||
- *Cloudflare appends to a client-sent `X-Forwarded-For`.* **Right, measured:** `6.6.6.6,37.191.56.193` (M2). Also
|
||||
measured, not in the brief:
|
||||
- Cloudflare passes a client's `X-Forwarded-Host` and `X-Forwarded-Port` unchanged.
|
||||
- It strips a client's `X-Real-IP`.
|
||||
- It refuses a client-sent `CF-Connecting-IP` at the edge (403).
|
||||
- *cloudflared's address is docker-assigned.* **Right** (`172.18.0.5`, no `ipv4_address`).
|
||||
- *The setup gate's token is minted only from a dashboard session.* **Right** (`ServeGateStart`, one caller). But **"the
|
||||
setup gate uses [the visitor address]" was wrong:** the gate read no client address at all, so there was nothing to
|
||||
switch over. It now logs the visitor.
|
||||
- *"Any reader takes the address at a fixed distance from the RIGHT."* **Half wrong.** The distance differs by path:
|
||||
second from the right through the tunnel, first on the LAN. So a fixed count is wrong for one of the two paths, and
|
||||
readers must skip trusted proxies from the right. Count readers (calibre-web, tandoor, wger) were left alone for that
|
||||
reason.
|
||||
- *The sketch (traefik trusts cloudflared) is enough.* **Not on its own.** Trusting cloudflared makes the leftmost entry
|
||||
— written by a stranger — reach every app. 19 apps read exactly that one (several for rate limits: audiobookshelf and
|
||||
docmost by address only, papra skips its limit on a junk value). It also lets a stranger's `X-Forwarded-Host` through.
|
||||
Both were closed in the same rollout.
|
||||
- *calibre-web `TRUSTED_PROXY_COUNT`, wger `AXES_*` proxy count.* **Left as they are.** Both are count readers, and both
|
||||
lock per NAME (decisions 58, 61). A count of 2 would break the LAN path, and for calibre-web also its proto/host.
|
||||
- *Grimmory improves.* **Right as read in source; not measured in this session** (R-775 updated).
|
||||
|
||||
## Part A — the box tells visitors apart
|
||||
|
||||
- **Paths**: `audits/visitors-2026-10-01/A/` — M1–M4 before, M5 after, DESIGN.md (the path table, the options, the choice,
|
||||
the docs quoted).
|
||||
- **Built** (controller v0.286.0/.1):
|
||||
- A `felhom-tunnel` network `172.16.253.0/29` with docker's allocation confined to `.4/30`. The hand prototype found
|
||||
traefik grabbing `.2`, so cloudflared now sits at a fixed `.2` and traefik at a fixed `.3`.
|
||||
- traefik trusts `172.16.253.2/32` only, and runs `felhom-forwarded@file`.
|
||||
- `EnsureBaseStack` reconciles a running traefik or cloudflared.
|
||||
- `clientaddr.go`, plus IPv6 counted per /64.
|
||||
- The dashboard login messages are informal, in both languages.
|
||||
- **Red-proofs**: RP-A1, RP-A2 and RP-A3 each fail on their mutant.
|
||||
- **Live**:
|
||||
- **9202, simulated tunnel:** a stranger rotating a forged leftmost address was locked after 5 tries; the household
|
||||
from another address got in at once; LAN and impostor forgeries were counted as themselves (L1).
|
||||
- **demo-hp, real tunnel:** an app sees `6.6.6.6,37.191.56.193, 172.16.253.2`. The dashboard counter was keyed on
|
||||
`37.191.56.193` and locked after 5. Every app answered (L2, M5, E/).
|
||||
- **ep0's one request:** refused 403 by Cloudflare's edge, with no line in the box's log, so the second outside
|
||||
address is unmeasured on the real tunnel → **R-779**.
|
||||
- **Apps** (sweep of all 56 apps plus Grimmory and MeTube, read in source: `A/sweep/`):
|
||||
- **19 router resets**, one commit each, shipped before the release. docmost on the real tunnel still throttles a
|
||||
rotating forger.
|
||||
- **BookStack `APP_PROXIES`**, with 3.6 re-measured before and after: the household is no longer throttled by the
|
||||
stranger.
|
||||
- **Gains with no change:** Home Assistant (it treated every internet visitor as "local"), actualbudget, immich,
|
||||
dawarich, claper, termix, Grimmory.
|
||||
- **Open:** kimai, zipline, vikunja, nextcloud (R-776); Emby and Jellyfin LAN rights (R-777).
|
||||
- **Checklist 3.10** added.
|
||||
|
||||
## Part B — the permanent gate (spike)
|
||||
|
||||
`audits/permanent-gate-2026-10-01/VERDICT.md`. **PASS** on items 1–5.
|
||||
- 36 stranger requests, 0 reached an app.
|
||||
- Family members got their own 30-day logins, websockets worked, and logout worked.
|
||||
- A stranger's guesses locked only him.
|
||||
- OPDS, Kobo and KOReader worked through anchored exceptions, with the app's own login still in force.
|
||||
- The family login never opened the dashboard.
|
||||
|
||||
Costs: +0.4 ms per request; with the gate down, apps answer 500 (closed); the build is 2 sessions. One build
|
||||
requirement: anchor every exception (`/api/v1/opdsx` slipped past an unanchored prefix; Grimmory's own login still
|
||||
refused it). Fully torn down.
|
||||
|
||||
## Part C — SparkyFitness
|
||||
|
||||
`app-catalog-felhom.eu/onboarding/sparkyfitness.md`.
|
||||
- **Bench:** 5.1 server 33 %; 5.2 server 38 % and db 53 % under a heavy burst, 0 kills; 1.4 v0.17.2 → v0.17.3 proven;
|
||||
2.1 CLEAN; 9.1 green.
|
||||
- **9202:**
|
||||
- A stranger got nothing during install.
|
||||
- Sign-up was blocked in 3 spellings, and blocked again after remove + restore.
|
||||
- Sign-in limit (3.6): ~10 s for everyone (R-783).
|
||||
- 4.4: the state read degraded within 10 s.
|
||||
- Backup → remove → restore read the data back.
|
||||
- **Open:** 0.1 licence (**R-784 — non-commercial only**), 0.5, 0.7, 1.6, 1.7, 5.4, 8.3 (R-786). The pin is 11 releases
|
||||
and a major behind (R-785). A licence sweep of the whole catalog is owed (R-787).
|
||||
|
||||
## Part D — R-772, R-773
|
||||
|
||||
- **R-772:** not-run → `healthy:false, not_checked:true`, plus a line on the app page. 0.286.0 still waited for the
|
||||
last healthy record's 5-minute interval; 0.286.1 sees it on the next tick. **Live:** not_checked 14 s after the stop,
|
||||
healthy 42 s after the start. Red-proofs RP-D1 and RP-D1b. Readers of `health_probe`: the state override and the
|
||||
API/page only; the hub never sees it.
|
||||
- **R-773:** a restore of a removed app writes the lock record (`opened_by: restore`) and the block before the start.
|
||||
**Live** on Karakeep and SparkyFitness: a stranger got 403 after remove + restore. Red-proof RP-D2.
|
||||
|
||||
## Part E — release, floor, golden
|
||||
|
||||
- **Release:** controller 0.286.1. Full suite rc 0, gates green. 0.286.0 ran on 9202 only.
|
||||
- **Floor:** `0.286.1` + `min_agent 0.131.0` → demo-hp and demo-felhom on 0.286.1 within ~60 s. Each created the
|
||||
network, recreated traefik and moved cloudflared by itself (`E/rollout-*.txt`).
|
||||
- **Golden 0.286.1** (`documentation/tests/golden-0.286.1-2026-10-01/`):
|
||||
- sha `1dec051d…a89`; the round trip is equal; token leaks 0, with the positive control at 1.
|
||||
- Vouched: golden 0.286.1 / agent 0.138.0 / min_agent 0.131.0. The hub then refused golden 0.285.0.
|
||||
- `golden_currency_gate`: OK (not waived). The waiver was not renewed — the bake ran.
|
||||
|
||||
## Rows
|
||||
|
||||
- **Closed:** R-753, R-754, R-772, R-773.
|
||||
- **Narrowed:** R-775.
|
||||
- **Opened:** R-776..R-787 (12).
|
||||
- **Register:** 410 → 422.
|
||||
- **Capability map:** two rows added (PARTIAL: visitors apart; MISSING/spiked: the family gate).
|
||||
- **`unproven.py`:** unchanged — 35 of 55 not walked.
|
||||
|
||||
## Teardown (three layers)
|
||||
|
||||
- **Machine:**
|
||||
- 9202: karakeep, bookstack and sparkyfitness removed with their data and backups through the product; the echo
|
||||
containers, the spike (gate, Grimmory, MariaDB, MeTube, volumes, network, dynamic file), test images and helper
|
||||
files removed. 9202 keeps controller 0.286.1, its new traefik config and `felhom-tunnel` — the product's own.
|
||||
- demo-hp 9201: `a1-echo` and its image removed; its traefik config was restored after the M2 minute, and is now the
|
||||
release's.
|
||||
- **Host:** the bench 9401 was rebuilt and destroyed (`pvesm` before/after in `C/bench/C0`, `C9`). The drill VM is
|
||||
powered off and reverted to `virgin`; CT 9100 destroyed.
|
||||
- **Hub:** the floor and the artifact manifest changed on purpose (above); no customer or appliance records created.
|
||||
- **ep0:** one HTTP request, nothing else.
|
||||
@@ -1,99 +1,90 @@
|
||||
# REPORT — localisation slice 3 Part A: the hub's e-mails follow the household's language
|
||||
# REPORT — the operator's three rulings of 2026-10-01 built (day session)
|
||||
|
||||
**hub v0.118.0** · felhom.eu base `20aafc3dec20` · 2026-09-18 · R-558 (Part A), R-555 closed
|
||||
Evidence: `documentation/audits/rulings-2026-10-01/` (A–D, T, tools) · `documentation/audits/retest-2026-10/` (the monthly
|
||||
run) · golden: `documentation/tests/golden-0.285.0-2026-10-01/`.
|
||||
Architecture read: `09` §3 decisions 30, 45, 52, 53 and §6.5; `03-host-agent.md` (the controller swap) with agent
|
||||
`internal/localapi/controllerswap.go`; `07` (R-698); `runbooks/monthly-floating-retest.md`; `audits/night-rulings-2026-09-30/`.
|
||||
Baselines (live Gitea ~07:10 CEST): controller `a70c398` (0.284.2), agent `d766666` (0.138.0), felhom.eu `3159892`, catalog
|
||||
`efd492d`. Register 383 rows by `register_shape_gate`'s method; highest id R-748; last decision 53.
|
||||
|
||||
## Claims in the task that turned out wrong, named first
|
||||
## The Part table
|
||||
|
||||
1. **"`customerMessages` 40 entries (L70)"** — **39 entries, at L69.**
|
||||
2. **"`customers.language`, `appliances.reported_language`"** — **neither table exists.** There is no
|
||||
`customers` table and no `appliances` table. The real ones are **`customer_configs`** (the customer
|
||||
record) and **`reports`** (the box's heartbeat, one row per report). The columns added are
|
||||
`customer_configs.language` and `reports.language`.
|
||||
3. **"no `ALTER TABLE customers ADD` found by grep — the migration pattern is something else; if it
|
||||
is not obvious, stop and report."** The pattern is completely obvious and there are ~20 instances:
|
||||
`s.db.Exec("ALTER TABLE <t> ADD COLUMN <c> <type> NOT NULL DEFAULT <v>")`, error deliberately
|
||||
ignored so it is idempotent. The grep failed only because it named a table that does not exist.
|
||||
4. **"`renderBackupRunFailures` gains a `lang` parameter"** — it is **operator-only and already
|
||||
English**. Its single caller is `FormatOperatorEmail`. Untouched.
|
||||
5. **"`message_customer`'s length is capped like `message`"** — **`message` has no length cap.** Both
|
||||
are bounded only by the 1 MB `LimitReader` on the request body. I wrote a cap, then removed it: a
|
||||
byte-count truncation would also cut a UTF-8 sequence in half.
|
||||
6. **"`python3 scripts/hub_gates.py` (or the runner the hub uses — name it)"** — there is no
|
||||
`hub_gates.py`. The hub's gates run from **`scripts/repo_gates.py`**, this repo's single runner.
|
||||
7. **Line numbers** were mostly off by one or two (`customerMessages` 69 not 70, `severityLabels` 158
|
||||
not 159, `FormatCustomerEmail` 166 not 167).
|
||||
| Part | done / not done / changed | why |
|
||||
|---|---|---|
|
||||
| **Rulings 54–56** | **done** — recorded first in `09` §3 and CONTEXT (`2076bf9`) | before any work |
|
||||
| **A — the full re-test** | **done** — nextcloud `3b59dfb` and sonarr `1a37032` re-tested on both venues and written, pushed through the pre-push gates | first start refused in one minute: the script checked the bench for its own files before copying them (R-749, fixed `9e53205`) |
|
||||
| A1 runbook | **done** — every app by default, `--engines-only` the switch, the standing brief, cost | — |
|
||||
| A4 linuxserver cost | **done** — see below | — |
|
||||
| A5 STATUS standing line | **done** — "last run 2026-10-01, next due ~2026-11-01" | — |
|
||||
| **B — two controller versions** | **done — controller v0.285.0**, floor 0.285.0 (MinAgent 0.131.0 declared) | measured first: the agent rolls back to the RUNNING image |
|
||||
| B3 live | **done** on 9202 (version-order fallback) and both demo boxes (the swap record) | 9202's self-update is off (no hub), so the record path was shown on the demo boxes |
|
||||
| — R-751 | **added** — a nil-stack panic in the update clean-up, found by the full suite, fixed in the same release | a panic in a goroutine ends the controller |
|
||||
| **C — mealie** | **done — decided by CC unattended (decision 57):** `SECURITY_USER_LOCKOUT_TIME=1`, catalog `a4597cd`; 9202 proof 120 min | no fix stops an hourly renewal without changing the login name (option d, left open) |
|
||||
| C other apps | **done (read, not measured)** — four more lockable: R-752 | source reading by a sub-agent at each pinned tag |
|
||||
| **D1 — R-746** | **done** `804884a`, red-proofed (unit + live registry) | — |
|
||||
| **D2 — R-744** | **done** `9fc7052`, proven on 9202 at 1.10.1 | outline has no newer release, so no step |
|
||||
| **E — release, floor, golden** | **done** — demo boxes on 0.285.0 within ~6 s; golden 0.285.0 baked, round-trip identical, vouched; the gate prints **OK** | — |
|
||||
|
||||
Claims that were **right**: no mail goldens existed; the hub read no `language` anywhere outside a
|
||||
comment; `05-hub-architecture.md` had no customer-mail section; the box has published the field since
|
||||
v0.247.0. I briefly believed `Report.Language` was never assigned and was wrong — `cmd/` is
|
||||
gitignored, so `rg` skips `main.go`, where all four assignments live.
|
||||
## Claims in the brief that turned out wrong (or right)
|
||||
|
||||
## What shipped
|
||||
1. **"`previousImage` … handed to the agent's `SwapController`"** — **wrong.** `SwapController(ctx, target)` carries the
|
||||
target only; `previousImage` goes into `update-state.json`. The agent reads `/etc/felhom-controller-image` when the swap
|
||||
begins and rolls back to THAT — the running image. On all three boxes it equals `<image>:<current version>`, so the
|
||||
practical outcome matches; the mechanism does not.
|
||||
2. **"The sweep skips every controller image today"** — **right**, twice over: the sweep's repositories never include the
|
||||
controller's, and it skips any repo containing `felhom-controller`.
|
||||
3. **"nextcloud and sonarr still differ"** — **right**, and only those two (bookstack, radarr, code-server did not differ today).
|
||||
4. **"mealie's lock is per account"** — **right** (`login_attemps` / `locked_at` on the user). Also: the lock outlives its
|
||||
hours until mealie's HOURLY job resets the counter, so `1` means 1–2 h (measured 120 min).
|
||||
5. **"The golden's image is not needed after first boot"** — right **after the first swap**; until then it IS the running
|
||||
image (kept as such). A whole-guest restore brings its own Docker store (`mp0 backup=1`).
|
||||
6. **"Is an old controller tag still in the registry?"** — **no, below 0.213.0** (2026-08-12): `0.201.0` answers 404. Nobody
|
||||
recorded what removed them (R-750). This session deleted no registry tag.
|
||||
7. "Live on 9202 … after the release" for the record path — 9202 has self-update off; the record path ran on the demo boxes.
|
||||
|
||||
The hub reads the language the box reports and writes the household's e-mail in it. **Nothing an
|
||||
operator reads changed.** Hungarian is byte-identical, measured.
|
||||
## Part A — the run
|
||||
|
||||
- **`internal/i18n`** — flat bundle, 79 keys, hu authoritative + hu fallback, missing ceiling 0.
|
||||
- **56 mail goldens per language**, captured from v0.117.0 **before** any string moved. All 56
|
||||
Hungarian ones pass unchanged after the rewrite. A `nowFn` seam makes them byte-stable.
|
||||
- **`customerMessages`/`severityLabels` are derived from the bundle** — one place per sentence, and
|
||||
all 40+ existing tests and comments that read them still work.
|
||||
- **Order: last reported → created-with → `hu`.** `reports.language` defaults to **empty**, never
|
||||
`hu`: "never told us" is not "chose Hungarian".
|
||||
- **`message_customer`** accepted on `POST /api/v1/event` (additive, optional forever).
|
||||
- **Per-language bind page**, built the controller's way (substitute markers, then parse).
|
||||
- Customer form `language` select; `customer.language` in `controller.yaml` (diff: one line).
|
||||
| app | result | bench | box |
|
||||
|---|---|---|---|
|
||||
| nextcloud `34.0.4-apache` | **DONE** — written `3b59dfb` | 05:19–05:32 UTC (proven, peak 23.7 %) | ~5 min: old digest installed, seeded, the guarded Update ran the re-test step, new digest running, read back, badge "Naprakész" |
|
||||
| sonarr `4.0.20` (linuxserver) | **DONE** — written `1a37032` | 05:38–05:50 (proven, peak 13.9 %) | ~3 min, same steps |
|
||||
|
||||
## Live evidence, read off the running hub
|
||||
**Monthly cost.** ~17 min per app (bench ~13 incl. the 10-minute watch, box ~4) plus ~15 min to set up and tear down. The
|
||||
four linuxserver apps with ladders (bookstack, radarr, sonarr, code-server) are rebuilt weekly, but the monthly run tests
|
||||
only the day's digest — **at most 4 re-tests a month from them (~70 min of bench, ~16 min of 9202), not 16.** Acceptable.
|
||||
|
||||
The wire already worked and I measured it before writing anything (read-only copy of `hub.db` + WAL):
|
||||
## Part B — images before → after
|
||||
|
||||
```
|
||||
demo-felhom 2026-09-18 13:42:56 language='hu' controller 0.255.0
|
||||
demo-hp 2026-09-18 13:45:24 language='en' controller 0.255.0
|
||||
drill-r50 / peti-felhom / tester-1 language=<ABSENT> (0.213.0 / 0.115.0 / 0.245.0)
|
||||
```
|
||||
| box | controller images | Docker images (`system df`) | `/var/lib/docker` used |
|
||||
|---|---|---|---|
|
||||
| 9202 | 5 → 2 | 3.17 → 3.10 GB (layers are shared) | 3.5 → 3.4 GB |
|
||||
| demo-hp 9201 | 84 → 2 (82 deleted) | 15.2 → 11.65 GB | 18 → 15 GB |
|
||||
| N100 9201 | 76 → 2 (74 deleted) | 6.07 → 1.11 GB | 6.0 → 1.2 GB |
|
||||
|
||||
That is the positive and the negative control in one read: boxes ≥ 0.247.0 report the field, older
|
||||
ones send nothing at all — which is exactly the case the empty default exists for.
|
||||
Each kept 0.285.0 (running) and 0.284.2 (previous). Red-proofs (`B/B1-red-proofs.txt`): previous dropped → 0.284.2
|
||||
deleted; record ignored → 0.283.1 deleted; no swap check → deleted while swapping; no in-use check → a used image deleted;
|
||||
no success check → a failed swap's image named previous; no nil-stack check → panic. The first in-use red attempt
|
||||
removed a line and did not build; redone with the condition disabled (recorded).
|
||||
|
||||
## Two defects found inside this release
|
||||
## Part C — mealie
|
||||
|
||||
- **R-581 (P2, partly still open).** `CustomerLanguage` picked the newest report by `received_at`,
|
||||
which has **second** granularity — same-second reports tie and the winner is arbitrary. A household
|
||||
that had just switched would get the old language back at random. Fixed by ordering on the
|
||||
autoincrement `id`; caught by a test that failed on the first version. **Still open:
|
||||
`GetCustomers()` has the same shape** and feeds the whole operator dashboard.
|
||||
- **R-582 (closed, kept for the lesson).** The English copy-guard stems, ported word-for-word from the
|
||||
Hungarian, convicted **141 honest sentences**. In Hungarian the stem *is* the claim
|
||||
(`visszaállíthat` = *can restore*); English splits the modal from the verb, so the claim is a
|
||||
phrase. Then the decoy suite caught the fix being too narrow — "can **still** be restored" walked
|
||||
through a pattern written for "can be restored".
|
||||
Settings (v3.28.0 source): `SECURITY_MAX_LOGIN_ATTEMPTS` 5, `SECURITY_USER_LOCKOUT_TIME` 24 (hours), per account, lifted
|
||||
by an hourly job; `POST /api/admin/users/unlock` exists but the only admin is the locked account. Fix and 9202 proof: see
|
||||
decision 57 (`C/C1-mealie-lockout-1h.txt`: right password 200 → 5 wrong 401 → 423 → right password 423 for 120 min → 200;
|
||||
a wrong one 401 after). Other apps (R-752): calibre-web-automated (per username, 3/min and 40/day), wger (per IP = traefik's,
|
||||
30 min, everyone), Grafana (per account, 5 min), BookStack (e-mail|IP, 60 s); gokapi and claper cannot.
|
||||
|
||||
## Gate work
|
||||
## Rows
|
||||
|
||||
`wire_contract_gate.py`: the R-555 `language` allowlist entry **deleted** — the field is genuinely
|
||||
decoded now and the gate checks it (202 tags, was 201). `hub_copy_gate.py`: **the sentences moved into
|
||||
the bundle**, so `CUSTOMER_SURFACES` had to move with them — without that the four declared Go files
|
||||
would all still exist, the gate would still report success, and it would be scanning nothing. It
|
||||
learned English, and found one real occurrence (operator-tier, registered with its reason). Three new
|
||||
decoys including an **innocent control**; the `hub-copy` exemption is removed from
|
||||
`decoy_coverage_gate.py`.
|
||||
**383 → 387.** Opened R-749 (re-test start), R-750 (old registry tags gone), R-751 (update clean-up crash), R-752 (four
|
||||
lockable apps). Closed R-743, R-744, R-745, R-746, R-749, R-751. Narrowed R-747.
|
||||
|
||||
## The mails
|
||||
## Teardown
|
||||
|
||||
| Group | Count | Golden | English | Live-tested |
|
||||
|---|---|---|---|---|
|
||||
| Event types (`mail.event.*`) | 39 | ✅ hu + en | ✅ | pending (§13.1) |
|
||||
| Severity labels | 4 | ✅ | ✅ | — |
|
||||
| Customer wrapper (subject, body, 2 lines, sign-off) | 5 | ✅ | ✅ | — |
|
||||
| Claim arc (claim / reset / reenroll / claimed) | 4×2 | ✅ | ✅ | pending (§13.2) |
|
||||
| Self-bind mail | 2 | ✅ | ✅ | pending |
|
||||
| Bind page | 21 | — (render tests) | ✅ | pending |
|
||||
| Operator mails | 3 | ✅ (unchanged) | n/a — never localised | — |
|
||||
|
||||
## Green
|
||||
|
||||
`go build` / `go vet` / `go test ./...` clean. All 14 felhom.eu gates OK. 15/15 decoys behave.
|
||||
|
||||
**Not yet done:** deploy + live proof (§13), and Part B (controller v0.256.0, the box's own sentence).
|
||||
- **Machine:** 9202 back on the live catalog (`repo_url` read back), the same six containers as at the start, controller
|
||||
0.285.0 (the release); the apps this session installed (nextcloud, sonarr, outline, mealie) removed through the product.
|
||||
Bench 9401 destroyed with its template. Drill VM: CT 9100 destroyed, token/runner/log shredded, qemu exited, reverted to
|
||||
`virgin`. Drill catalog reset to live (`a4597cd`), image lines identical.
|
||||
- **Host:** demo-hp `pct list` = 9201, 9202 (as at the start); the N100 untouched except the floor's controller update.
|
||||
- **Hub:** two form saves — the floor (0.285.0, MinAgent 0.131.0) and the vouch (golden 0.285.0); one floor attempt without
|
||||
the credential answered 302 `/login` and stored nothing.
|
||||
|
||||
@@ -67,6 +67,14 @@
|
||||
| `compareVersions` | hub/internal/web/server.go (~L571) | `(a, b string) int` | X.Y.Z comparisons in web (floor checks, update-available) | Returns 0 on parse error — unparseable compares as "equal" (see §3). |
|
||||
| `reportBackupCard` + `backupCardView` + `fmtBytesAuto` (R-331, v0.109.0) | hub/internal/web/backup_card.go | `(reportJSON string) backupCardView` / `(int64) string` | THE customer-page Backup card — everything it shows about a customer's backups | **Reads the report's `offsite` object, NEVER `backup`.** The `backup` object's `snapshot_count`/`repo_size_mb`/`integrity_ok` have had no producer since slice 8C and rendering them told every operator every customer had `Snapshots 0` (measured on demo-hp over a repo holding 67). **Always consult `StatsKnown` before believing a zero** — absent/false means "never measured", NOT "empty", and those are opposite news (R-225 measured the same confusion one layer down). Resolved in Go, not the template, because a `{{if}}` chain over `Report`'s `map[string]interface{}` float64s cannot keep the absent/zero distinction the card is entirely about. `fmtBytesAuto` scales MB/GB/TB — do NOT swap in `fmtBytesGB`, which renders demo-hp's real 140 829 678 B as `0.1 GB`. |
|
||||
|
||||
### Off-site key registrar (v0.127.0, decisions 68–69, hub/internal/offsitekeys + store)
|
||||
|
||||
| Symbol | File | Short signature | Use for | Gotchas |
|
||||
|---|---|---|---|---|
|
||||
| `offsitekeys.Registrar` (`Install` / `Confirm` / `Audit` / `OpenWindow` / `CloseWindow` / `MoveAside`) | hub/internal/offsitekeys/offsitekeys.go | `(ctx, Target, password, …)` | EVERY write to a sub-account's `.ssh/authorized_keys` and every repo move-aside | **The only writer of that file, and the only deleter on a sub-account (`DeleteSetAside`: `<repo>.orphaned-*` only, decision 74).** `read()` is read-only (R-827). Uses the provider's port-23 restricted shell (`dd of=` takes stdin, `mv` overwrites, `test` does NOT exist — measured); an unpinned line is a deletion route and is dropped on every install; the window line goes FIRST (first match wins). Never `rm`. |
|
||||
| `offsitekeys.Service` (`RegisterKey`, `ConfirmKey`, `AuditAll`, `OpenWindowFor`, `CloseWindowFor`, `SweepExpiredWindows`) | hub/internal/offsitekeys/service.go | — | Binding the registrar to the store, descriptor and operator events | The box-facing API (`/api/v1/offsite/register-key…`) answers with NO credential — pinned by `TestOffsiteKeyEndpoints_AuthAndNoPasswordInAnyResponse`. |
|
||||
| `(*Store).SaveOneTimeSecret` / `OffsitePassword` / `SealLegacyOffsiteSecrets` | hub/internal/store/offsite_seal.go | — | Storing / reading the sub-account password | **Sealed AES-256-GCM; no key → refused (fail-closed).** Under `go test` every store gets a fixed key (`testing.Testing()`); production needs `OFFSITE_SECRET_KEY`. Never serve the value to a box. |
|
||||
|
||||
### Host views & lifecycle / offsite endpoints (v0.47.0, hub/internal/web + store)
|
||||
|
||||
| Symbol | File | Short signature | Use for | Gotchas |
|
||||
|
||||
@@ -371,7 +371,9 @@ or **what is open changed**), update **all four** in the SAME session:
|
||||
that changes an **architectural contract** (tiers, targets, cadences, trust boundaries) updates the
|
||||
owning design doc in the same session. It was ruled but never written here, in the template CC
|
||||
actually reads — so it bound nobody. Now it does.
|
||||
- **`backlog/OPEN-ITEMS.md`** — the register, and the single source of truth for open work. A task
|
||||
- **`backlog/OPEN-ITEMS.md`** — the register, and the single source of truth for open work. **A new row
|
||||
is filed into its category's section with one Category and one Sev (P1–P4)** — the scale and the eleven
|
||||
categories head the file, and `register_shape_gate.py` refuses a row without them (2026-10-03). A task
|
||||
that changes what is open without touching it re-creates exactly the thread-loss the register was
|
||||
built to solve: `REPORT.md` is overwritten every session, so nothing durable may live only there.
|
||||
|
||||
@@ -394,8 +396,9 @@ or **what is open changed**), update **all four** in the SAME session:
|
||||
> later by an overnight drill that planted files and watched them not come back**, and shipped as
|
||||
> R-354. The work was right the first time; only the filing was missing.
|
||||
|
||||
**Report which `OPEN-ITEMS.md` rows the task opened, closed or re-ranked** (§15). Every row carries
|
||||
an owner — a row nobody owns is how items got lost in the first place.
|
||||
**Report which `OPEN-ITEMS.md` rows the task opened, closed (and moved to `CLOSED-ITEMS.md`), narrowed
|
||||
or re-ranked** (§15). Every row carries an owner — a row nobody owns is how items got lost in the first
|
||||
place.
|
||||
|
||||
### N.6 Website version bump (if controller/hub version is shown on the site).
|
||||
|
||||
@@ -406,10 +409,14 @@ over half of it finished work, one entry at 16 KB. **A file that cannot be read
|
||||
be checked**, and this project has paid for that twice — a record nobody could find because it sat
|
||||
inside an entry about something else, and a finding rediscovered because nobody could see it.
|
||||
|
||||
1. **Compress what this session closed.** A closed row keeps its title, the version it shipped in, its
|
||||
evidence paths, and any sentence stating a rule. Everything else goes, and it moves to
|
||||
`backlog/CLOSED-ITEMS.md`. **Nothing is deleted:** the compressed entry names the commit whose
|
||||
1. **Move what this session closed — IN THE SAME COMMIT THAT CLOSES IT.** A closed row keeps its title, the
|
||||
version it shipped in, its evidence paths, and any sentence stating a rule. Everything else goes, and it
|
||||
moves to `backlog/CLOSED-ITEMS.md`. **Nothing is deleted:** the compressed entry names the commit whose
|
||||
`git show` returns the full original text. **Open rows are not touched — their detail is doing a job.**
|
||||
**This is a gate, not a habit (2026-10-03):** `scripts/closed_register_gate.py` RULE 3 refuses a push
|
||||
whose `OPEN-ITEMS.md` holds a row LEADING with a finished word (CLOSED, SHIPPED, FIXED, DECIDED, …).
|
||||
The rule existed from 2026-08-22 without a gate and was followed only sometimes: on 2026-10-03 the open
|
||||
register held 113 finished rows, a quarter of the file.
|
||||
2. **Rehome live reasoning before compressing it away.** If a closed entry carries the reason a rule
|
||||
exists or a fence sits where it does, that reasoning moves — to `CONTEXT.md` if it is a decision,
|
||||
to the owning `architecture/*.md` if it is a shape. **Where it is a decision, mark the resulting
|
||||
|
||||
@@ -30,6 +30,7 @@ The operator-tier agent and the Proxmox platform.
|
||||
- [`architecture/04-control-plane-authorization.md`](architecture/04-control-plane-authorization.md) — signing, escrow, authz
|
||||
- [`architecture/02-controller-module-map.md`](architecture/02-controller-module-map.md) — **historical** v0.33 planning map; the live map is [`controller/module-map.md`](controller/module-map.md)
|
||||
- [`proxmox-platform.md`](proxmox-platform.md) — Proxmox platform reference
|
||||
- [`architecture/11-os-updates.md`](architecture/11-os-updates.md) — operating-system updates: host, guest, Docker engine (**NOT RATIFIED**, 2026-10-04)
|
||||
|
||||
### Hub (operator backend) — `architecture/05`
|
||||
- [`architecture/05-hub-architecture.md`](architecture/05-hub-architecture.md) — hub architecture (v0.11.0)
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -119,6 +119,62 @@ credentials.
|
||||
| guest ↔ Proxmox host | **(none direct)** | the guest holds no Proxmox creds; all via the agent | — |
|
||||
| hub ↔ Cloudflare API | geo-restriction WAF (enforcement) | the **hub** holds the CF API token; reconciles geo desired-state → WAF | the customer's zone/WAF |
|
||||
|
||||
**Every app is on the internet from its first minute** (`*.domain` through the tunnel), so an app's FIRST admin login is
|
||||
a trust boundary too: a default password, or a "first visitor creates the admin" screen, is open to a stranger until the
|
||||
household acts. Rule and per-app status: `09` §3 decision 45 and `app-catalog-felhom.eu/FIRST-ADMIN.md` (the audit
|
||||
of all 53 apps, 2026-09-28).
|
||||
|
||||
**Who is the visitor — the address (recorded 2026-10-01, R-753, `09` §3 decision 63, controller ≥ 0.286.0).** Rule:
|
||||
*never believe an address a client can write.* Through the tunnel every visitor used to reach traefik as cloudflared's
|
||||
one docker-assigned address, so every per-address guard (the dashboard's login counter, an app's lock) was an
|
||||
"everyone" guard a stranger could aim at the household. Now cloudflared has a fixed address and traefik trusts forwarded
|
||||
headers from that one address only: an app receives `X-Forwarded-For: <client-written…>, <real visitor>, 172.16.253.2`
|
||||
(Cloudflare APPENDS to a client's own chain — measured), a LAN visitor arrives as itself, and every other peer's chain is
|
||||
dropped. traefik's entrypoint middleware `felhom-forwarded` removes every header a client could write a host, a path or
|
||||
an address into (`X-Forwarded-Host`, `Forwarded`, `True-Client-Ip`, …) and fixes `X-Forwarded-Port: 443`. **Readers take
|
||||
the visitor from the RIGHT.** The controller (`clientaddr.go`) believes the chain only from traefik, takes the hop
|
||||
traefik saw, and for the tunnel's hop reads `CF-Connecting-IP` (the edge refuses a client-sent one). Catalog apps that
|
||||
read the LEFTMOST entry have the chain removed on their router. Design and measurements:
|
||||
`audits/visitors-2026-10-01/A/DESIGN.md`. A permanent household gate with family accounts in front of an app (Grimmory,
|
||||
MeTube) was SPIKED on this base and passed (`audits/permanent-gate-2026-10-01/VERDICT.md`) and is BUILT since
|
||||
controller 0.287.0 (`09` §3 decision 64) — the family gate, at the end of this section.
|
||||
|
||||
**Who may reach an app, and through what (recorded 2026-09-29 — no document said it before; spike finding F1).**
|
||||
Every app is reached only through the box's traefik (no catalog app publishes a host port except crafty-controller's
|
||||
game ports; none uses host networking — read from the catalog 2026-09-29). traefik routes by host name: the tunnel's
|
||||
`*.domain` and the LAN both land there. The dashboard (`felhom.<domain>`) has its own password; its session cookie is
|
||||
**host-only** and never reaches an app host. An app answers anyone who reaches its host, with the app's own login —
|
||||
**except while its setup gate is closed** (`09` §3 decision 46, controller ≥ 0.280.0): then traefik asks the
|
||||
controller first (`forwardAuth`), and only a browser holding a gate cookie for that one host gets through. The cookie
|
||||
is minted after a valid dashboard session vouched for the browser (a 60-second, one-use token bound to the host, on
|
||||
the dashboard's own `/__gate/start`). The controller is in an app's request path ONLY while its gate is closed; once
|
||||
open, the gate's traefik router is removed and the app is reached exactly as without it. The gate decides who creates
|
||||
the first admin. **Who may sign up afterwards** is `09` §3 decision 47 (operator ruling 2026-09-29): once the first admin
|
||||
exists, open sign-up is closed; only the admin adds people, from the app's own user page. An app that cannot close it
|
||||
says so on its page. Mechanism (controller ≥ 0.281.0): the box keeps a small traefik router on the app's own sign-up
|
||||
address once the gate opens, answered "sign-up is closed" by the controller; the household opens it for 15 minutes
|
||||
from the app page to let a family member in. The controller is in THAT address's path only.
|
||||
Since controller 0.282.0 there are two locks where the app has its own switch: the box also sets the app's own
|
||||
"no sign-up" setting (`after_setup`), and the block matches any letter case and extra slashes. An app installed before
|
||||
decision 47 gets both only when the household presses "Close sign-up now" (decision 49); wanderer, which is not gated,
|
||||
the same way.
|
||||
|
||||
**The family gate (`09` §3 decisions 63/64, controller ≥ 0.287.0).** An app whose template says `family_gate: true` is
|
||||
PERMANENTLY behind a forwardAuth door, the setup gate's mechanism with three differences: it never opens; its priority is
|
||||
below the install hold, the setup gate and the sign-up block; and the cookie it accepts is a FAMILY cookie. Each family
|
||||
member has their own name and password (the dashboard's Biztonság → Család card adds, resets and removes them; the
|
||||
password is shown once, stored as a bcrypt hash in the controller's data). A member signs in on the dashboard host's
|
||||
`/__family/login`; that sets a family session cookie on `Path=/__family` only — **never accepted by the dashboard's own
|
||||
RequireAuth** — and each app host gets a host-only cookie through a 60-second one-use token, as the setup gate does. The
|
||||
session store is asked on EVERY request, so a reset, a removal or a logout ends access at the next request. The family
|
||||
sign-in locks per real visitor (5 a minute) and per name (10 in 10 minutes) — never "everyone" (it reads the visitor
|
||||
from the right, as above). `family_gate_except:` lists LITERAL path prefixes a phone or e-reader app calls (Grimmory's
|
||||
OPDS, Kobo, KOReader, Komga API); each becomes a router without the door, anchored at a segment boundary
|
||||
(`^/prefix(/|$)`), so the app's OWN login decides there and a look-alike (`/api/v1/opdsx`) stays behind the gate. The
|
||||
gate is recorded at install (and at a removed app's restore), so a catalog change never gates or un-gates an installed
|
||||
app. Controller down → the gated app answers an error, never the app. Measured on 9202:
|
||||
`audits/family-gate-2026-10-02/A/items.txt`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Enrollment & identity
|
||||
@@ -157,15 +213,24 @@ credentials.
|
||||
on Cloudflare) is register row R-494, P3, not blocking. Measured reason this was ruled now: the
|
||||
2026-09-14 first-hour drill used a `*.felhom.eu` customer domain with no tunnel, and the dashboard link
|
||||
in the setup-code mail did not resolve.
|
||||
- **Tunnel placement: host** (resolved, Part 3 §3/§5). `cloudflared` runs on the Proxmox host
|
||||
as its own **agent-managed systemd service** — not inside the guest — so the data path
|
||||
survives control-plane death by construction. Geo-restriction WAF is **hub-enforced** (the
|
||||
hub holds the CF API token; the controller only reports geo desired-state).
|
||||
- **Tunnel placement: INSIDE the guest** (corrected 2026-10-01, R-754 — the operator's brief of that evening: the build is
|
||||
right, correct the document). `cloudflared` is a container the CONTROLLER renders and keeps up (`internal/infra`,
|
||||
`EnsureBaseStack`, a protected stack), with the tunnel token from `controller.yaml`. *This page used to say it ran on
|
||||
the Proxmox host as an agent-managed systemd service; no box has ever been built that way* (read: the controller's
|
||||
template; seen running in demo-hp's guest 9201 and in R-505's VM 331). Consequence, stated: the data path is NOT
|
||||
independent of the guest — a dead guest is an unreachable box, and the controller (not the agent) restarts the
|
||||
tunnel. Since controller v0.286.0 it sits ALONE on the `felhom-tunnel` network at the fixed address `172.16.253.2`
|
||||
(traefik at `.3`), so traefik can believe forwarded headers from it and from nothing else (§5). Geo-restriction WAF
|
||||
is **hub-enforced** (the hub holds the CF API token; the controller only reports geo desired-state).
|
||||
|
||||
---
|
||||
|
||||
## 8. Storage & backup
|
||||
|
||||
> **2026-09-30 (operator, `09` §3 decisions 50–51):** a new box's apps are in the off-site copy by default (07 §6);
|
||||
> the whole-guest restore test takes only the box's OWN archives — an earlier box's archive left in the customer's
|
||||
> ep0 namespace is never this box's proof, and the three drill archives found there are removed.
|
||||
|
||||
**Tiers** (escalating failure scope):
|
||||
|
||||
| Layer | Mechanism | Survives | Note |
|
||||
|
||||
@@ -47,7 +47,13 @@ Owns:
|
||||
1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable.
|
||||
2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests.
|
||||
3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**.
|
||||
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub.
|
||||
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Three states: `running` (healthy), `not_running` (stopped, absent, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
|
||||
Proxmox API, plus the OS wrapper's read-only `facts` (host Debian, next-boot kernel, held packages, taint, the crash
|
||||
guard; guest Debian, Docker engine, containerd, live-restore), at most every 10 min. **Still no new sudoers line** — the
|
||||
facts, `live-restore-on` and the Docker layer all ride `FELHOM_OSAPPLY`. New ROOT-owned files instead (installer 1.30.0;
|
||||
by hand on installed boxes, R-840): `/etc/felhom/os-trust.json`, `/etc/felhom/operator-signers` (the wrapper verifies
|
||||
Docker authority against them, never against the agent-writable config — `09` decision 93), and the crash guard
|
||||
(`/usr/local/sbin/felhom-crash-guard`, its units, `/etc/felhom/crash-guard.conf`).
|
||||
5. **Provisioning** — provision a guest **by restoring the golden base image** (§9), deploy the controller into it, hand it its bootstrap config; also **build and refresh the golden base image** itself.
|
||||
6. **Hub control loop** — poll for desired state + signed jobs, reconcile, execute, report, heartbeat.
|
||||
7. **Local API** — the per-guest authorization gate the controller calls.
|
||||
@@ -64,7 +70,7 @@ Explicitly does **not**:
|
||||
|
||||
- **Native Go binary, systemd service** on the host: boot-start, `Restart=always`, systemd watchdog (kill+restart on hang), journald logging, resource limits.
|
||||
- **Root-minimized (boundary settled — Phase 3 B3).** The agent runs as a **non-root** service user with the scoped `FelhomAgent` token for all API-covered work + a **narrow `sudoers` allowlist** for true host ops. Per Phase 3 (B3) the boundary is settled: the entire per-customer guest lifecycle — provision (by restore, §9), config, start/stop, snapshot, backup, **restore**, destroy — is token-covered. Genuine OS-root is confined to: (1) building/refreshing the **golden base image** (`keyctl` create is `root@pam`-only — one-time at enrollment + a maintenance cadence, §9); (2) **host mounts** (USB mount-by-UUID, systemd mount units / fstab); (3) **SMART / hardware sensors**. Root therefore never sits on the per-customer path. See `proxmox-platform.md` §3.6 for the role + boundary table.
|
||||
- **`cloudflared` is a separate systemd service**, not embedded in the agent. This is what makes the data path survive control-plane death by construction. The agent **manages and health-watches** it (see §5) but the tunnel does not live or die with the agent process.
|
||||
- ~~**`cloudflared` is a separate systemd service**, not embedded in the agent. … The agent **manages and health-watches** it (see §5) but the tunnel does not live or die with the agent process.~~ **[FACT, corrected 2026-10-04 — `11-os-updates.md` C8, R-838]** `cloudflared` is a **container in the customer guest** (`cloudflare/cloudflared:<pin>`), rendered and kept up by the in-guest controller (`felhom-controller/controller/internal/infra/infra.go`, `internal/stacks/infra.go`) and baked into the golden; there is no host systemd unit. It is still NOT embedded in the agent, so the data path survives the agent's death — and the agent does not manage it; it READS its health (R-841, agent v0.141.0). Its version moves only by a controller release (the pin), on the monthly re-test (`runbooks/monthly-floating-retest.md` "Infrastructure pins").
|
||||
|
||||
## 4. Control model — reconcile + signed destructive ops
|
||||
|
||||
@@ -143,7 +149,7 @@ notification) is the control.
|
||||
|
||||
**Box-initiated poll.** The hub never connects inbound. Each poll cycle exchanges:
|
||||
|
||||
- **Up:** heartbeat + a host-domain state report — host CPU/RAM/disk, per-guest up/down + spec, storage-target status (USB connected? NFS/CIFS reachable? PBS reachable?), last backup per target, last restore-test result, `cloudflared` health, agent + controller versions, audit-log tail.
|
||||
- **Up:** heartbeat + a host-domain state report — host CPU/RAM/disk, per-guest up/down + spec, storage-target status (USB connected? NFS/CIFS reachable? PBS reachable?), last backup per target, last restore-test result, `cloudflared` health (`running` / `not_running` / `unknown` from the guest container's readiness check — R-841), agent + controller versions, audit-log tail.
|
||||
- **Down:** the current desired state, any pending signed one-shot jobs, and config (poll interval, update window, policy changes).
|
||||
|
||||
**Dead-man's-switch (essential, not optional).** In a box-initiated model the heartbeat
|
||||
@@ -308,6 +314,42 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
|
||||
> Unchanged and load-bearing: `onboot=0` on the scratch at restore time, **every NIC link-down before
|
||||
> boot**, journal-before-mutate, guaranteed teardown, and the per-tier restore-task timeout.
|
||||
|
||||
> **A restore-test can never fill a box's disk (R-672, agent v0.133.0 + hub v0.124.0).** Measured
|
||||
> 2026-09-24 on demo-hp: the scheduled restore-test restored 9201's archive into `local-lvm` — the pool
|
||||
> holding 9201 — with no space check; the pool reached 100 % and 9201's disks remounted read-only.
|
||||
> Since v0.133.0, before anything is journaled or created:
|
||||
> - **Space first.** Free data ≥ restored × 1.2 + 5 GiB (`backup.restore_test_space_factor`,
|
||||
> `…_reserve_gib`) and room in a thin pool's metadata. `restored` is the **uncompressed** size — the
|
||||
> vzdump log's "Total bytes written" or the PBS snapshot size. **Never the archive file:** 9201's file
|
||||
> was 6.9 GB and its restore wrote 22.6 GB, so "file × 1.2 + 5 GiB" would have let that test run.
|
||||
> - **Off the tested guest's pool** when another storage is eligible (active, `rootdir`, and the agent
|
||||
> holds `Datastore.AllocateSpace` there) and fits. demo-hp had none until 2026-09-28: `nvme-scratch` carried no grant. **Since then (decision 44):** demo-hp's `restore_storage` is `nvme-scratch`, with the agent's `FelhomAgentStore` role granted there (user + token) — one restore test passed in 8m46s, `local-lvm` untouched (`audits/logins-nvme-2026-09-28/C/`). Saved config: `/etc/felhom-agent/agent.json.pre-d44`.
|
||||
> - **Unknown refuses.** A refusal is the test's result — `pass=false`, `skipped`, "skipped: not enough
|
||||
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
|
||||
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
|
||||
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
|
||||
> teardown tries the operator is told.
|
||||
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
|
||||
> and metadata, critical at 90 %, one alarm per pool per 6 hours (`08` §6.2).
|
||||
>
|
||||
> **Operator ruling 2026-09-24 (evening):** the scheduled restore-test is **OFF on both demo hosts**
|
||||
> (`backup.restore_test_eval_interval_seconds: -1` — **0 does not disable, it means the 6-hour default**)
|
||||
> until v0.133.0 is delivered there; then it is switched back on. Saved configs:
|
||||
> `/etc/felhom-agent/agent.json.pre-r672`. On demo-hp, a full restore-test of 9201 does not fit today
|
||||
> (21.1 GiB restored needs 30.3 GiB; 22.1 GiB free) — the preflight will refuse it, correctly, until the
|
||||
> pool has room.
|
||||
>
|
||||
> **Delivered 2026-09-24 night** to demo-felhom and demo-hp by CC-signed `agent_update` jobs; the restore test is
|
||||
> back ON on both (the `-1` config kept as `agent.json.night-0925-off`). Peti's box never received it — it was RETIRED 2026-09-25 and will not return.
|
||||
|
||||
> **A whole-box backup that cannot fit is skipped with a reason, before anything starts (R-685, agent v0.134.0).**
|
||||
> Before a vzdump to a LOCAL (non-PBS) target: free ≥ the guest's newest archive on that target × 1.25 + 1 GiB,
|
||||
> because PVE prunes old archives only AFTER a successful backup. A shortfall is a named skip (`skipped: not enough
|
||||
> space: <target> has X GiB free; the last archive … needs about Z GiB`) that reaches the tier view and the
|
||||
> operator; it FAILS OPEN on a PBS target, a first backup or unreadable usage. **Free space comes from
|
||||
> `GET /nodes/<node>/storage` — `GET /storage` is the definitions and has NO usage** (a build that read it let a
|
||||
> real vzdump start in its own live test, 2026-09-24 night — aborted, no archive left).
|
||||
|
||||
- **Quiescing (controller-driven for app-consistency) — implemented (slice 8B):** an LXC has no
|
||||
fsfreeze (`proxmox-platform.md` §4.2), so app-consistency is the controller's job: it learns a
|
||||
backup is due (`GET /backup/due`, §6) → **quiesces** (stops its app stacks) → `POST /backup` →
|
||||
@@ -597,6 +639,13 @@ buildable until then; recorded here so the front-half built in slice 7 lands rea
|
||||
never — the binary is what flips. Report field `selfupdate_pending` surfaces a runs-but-never-commits
|
||||
binary. v1 scope-outs: no hub-floor auto-update, no failed-update auto-retry (the operator re-signs),
|
||||
no pending-timeout auto-rollback.
|
||||
- **OS updates, guest fast lane (agent v0.140.0, `11-os-updates.md` §8.1).** A second root-owned wrapper,
|
||||
`/usr/local/sbin/felhom-os-apply`, behind ONE sudoers entry (`FELHOM_OSAPPLY`: `felhom-os-apply --plan
|
||||
/var/lib/felhom-agent/os/plan-*.json`). The agent has no `apt` grant of its own for this; every rule (no removal,
|
||||
no downgrade, no new or unlisted package, Debian origin only, the box's own customer guest only) is in the wrapper,
|
||||
red-proved per rule. The leg runs after a successful primary whole-guest backup, under the heavy-op gate.
|
||||
**[FACT] The signed agent update does NOT carry the wrapper or the sudoers line** — only the installer installs
|
||||
them (R-840).
|
||||
- **Controller (the easy case — it's a guest).** The agent owns the controller's lifecycle,
|
||||
so the **agent updates the controller**: snapshot-before-update (free rollback, because the
|
||||
controller *is* a snapshottable guest) → pull new image → redeploy → health-check → rollback
|
||||
|
||||
@@ -356,3 +356,50 @@ token is real. Pinned by `TestBindExpiredIsAlwaysDefaultLanguage`.
|
||||
56 goldens captured from v0.117.0 before any string moved, in
|
||||
`hub/internal/notify/testdata/mail_goldens/hu/`, with the English set beside them. The claim is a
|
||||
diff, not a reading. A golden is never regenerated to make a change pass.
|
||||
|
||||
### 15.6 The CODES a household types, per language [hub v0.119.0, R-597]
|
||||
|
||||
A mail in English that carries three Hungarian words with accents is not an English mail. The 2026-09-20
|
||||
drill received exactly that, and could paste the code but not read it to anyone.
|
||||
|
||||
**Two secrets this repo mints follow the household's language**, list and word count chosen together
|
||||
by `configgen.RandomPassphraseFor(lang, use)`:
|
||||
|
||||
| Secret | hu | en | who calls it |
|
||||
|---|---|---|---|
|
||||
| Setup / reset code | 3 words, 44.6 bits | **4 words, 51.7 bits** | `claim.Engine` via `CustomerLanguage` |
|
||||
| Owner passphrase | 5 words, 74.3 bits | **6 words, 77.5 bits** | the three `configs.go` sites |
|
||||
|
||||
**The rule is per use: English ≥ Hungarian, in bits.** The English list (EFF large, 7772 words after
|
||||
filtering) carries 12.92 bits/word against the Hungarian list's 14.85, so English takes one more
|
||||
word. The test computes both sides from the embedded lists rather than comparing a constant with
|
||||
itself, so shrinking a list or lowering a count fails.
|
||||
|
||||
**A third secret is NOT minted here and must not be added.** The customer **recovery code** is minted
|
||||
by `felhom-agent` (`internal/escrow`) from the same EFF list, ten words, ≈129 bits — it has been
|
||||
English since it was written. R-597's row listed it here; that was wrong. Writing a row for it in
|
||||
this table would create a second definition of a secret the hub does not own, which is the drift
|
||||
`backupTargetAbsentText` already demonstrates across two repos.
|
||||
|
||||
**Which language, and when.** The setup code follows `Store.CustomerLanguage` — the same order the
|
||||
mail carrying it follows (reported → created-with → `hu`), so a code and its e-mail can never
|
||||
disagree. Two consequences, both deliberate:
|
||||
|
||||
- **A household created as `hu` whose box later reports `en`** keeps every code already issued
|
||||
exactly as it was; the **next** code issued is English. A code is a hash on the box, never
|
||||
retranslated.
|
||||
- **At customer creation there is no stored customer yet**, so the Owner passphrase generated on that
|
||||
form reads the language from **the form field**, not from `CustomerLanguage` — which would answer
|
||||
Hungarian for every English household created. The store's `createdLanguage` applies the same
|
||||
default to an absent value, so the two agree.
|
||||
|
||||
**The box needs no change for any of this.** It stores and compares a bcrypt hash of whatever was
|
||||
minted and has no notion of which list the words came from — pinned by the controller's
|
||||
`TestClaimAcceptsAnEnglishWordCode`. The TTL, the single-use generation and the five-attempt lockout
|
||||
are untouched and language-blind.
|
||||
|
||||
**Count wording.** No claim mail states a word count; they say `Setup code: %s`. The only place a
|
||||
count appeared was the bind page's passphrase hint, and its English half is now count-free ("The word
|
||||
phrase you received from your operator during setup") — because "five words" stops being true for an
|
||||
English household, and was already wrong for one whose passphrase predates this release. The
|
||||
Hungarian „öt szó" is correct and unchanged.
|
||||
|
||||
@@ -180,6 +180,10 @@ production endpoint exists.
|
||||
|
||||
---
|
||||
|
||||
### 3.6 The ep0 datastore has a second copy — decision 70 (2026-10-03)
|
||||
|
||||
**[DESIGN]** A nightly PBS pull-sync copies `felhom-offsite` from ep0 to DooPlex's PBS over a read-only token. ep0's server snapshot never covered this volume and Hetzner has no volume snapshots (R-342). The copy is ciphertext per customer. **`[FACT]` BUILT 2026-10-03:** ep0 token `root@pam!dooplex-sync` (`DatastoreReader` only — the one change on ep0); ep0's PBS listens on `wg0` only, so DooPlex reaches it through an SSH forward (`felhom-ep0-pbs-tunnel.service`, operator ruling the same day); DooPlex datastore `ep0-copy`, sync daily 05:00 with `remove-vanished false`, verify Saturdays, failures mailed via Resend. First pull 201 s / 12 GB / 4 of 4 snapshots. Restore route: `runbooks/ep0-datastore-copy.md`. Evidence `audits/offsite-lock-build-2026-10-03/partF/`.
|
||||
|
||||
## 4. Robustness (production details beyond the spike)
|
||||
|
||||
- **4.1 Customer IP change = free, and explicitly NOT a DynDNS dependency.** The box dials out;
|
||||
@@ -314,7 +318,8 @@ caveats, recorded not papered over: **(1)** this SIM was handed a **public mobil
|
||||
stays a *retest-on-a-CGNAT-SIM-when-available* follow-up (low risk: mapping-hold is NAT-tier-agnostic
|
||||
by mechanism). **(2)** a dual-stack mobile uplink made `wg-quick` prefer the endpoint **AAAA and ride
|
||||
un-NATed IPv6** until v4 was forced — functionally fine, but see §4.2. The deferred second-ISP
|
||||
vantage (Peti VM 110) remains the thorough confirmation but no longer gates anything. Runbook:
|
||||
vantage (Peti VM 110) is gone — Peti's box was RETIRED 2026-09-25 — so a second-ISP confirmation needs another
|
||||
venue; it gates nothing. Runbook:
|
||||
`RUNBOOK-s3-cgnat-smoke`.
|
||||
|
||||
**Open sub-decisions (deferred by design):**
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -80,7 +80,7 @@ is allowed; every down member then reads as supervised.
|
||||
| `stopped`, `exited` | **yes** | not running, will not recover alone |
|
||||
| `degraded` | **yes** | [DESIGN, R-51] a dead supervised member is as unreachable as a single app that exited — immich-server sat Exited 18 h with the app 100 % dead and no alert |
|
||||
| `unhealthy` | **NO** | [DESIGN] a running container whose healthcheck is failing. Folding it in reintroduces the flapping fix-3 was written to stop. **R-384 did not change this** — it asks a prior question instead |
|
||||
| `restarting` | **NO**, until sustained | [DESIGN, C9-F2] `restarting` is on the normal deploy path, so folding it in would alarm fleet-wide on every update. Becomes down after **5 min** (`crashLoopAfter`) |
|
||||
| `restarting` | **NO**, until sustained | [DESIGN, C9-F2] `restarting` is on the normal deploy path, so folding it in would alarm fleet-wide on every update. Becomes down after **5 min** (`crashLoopAfter`) of UNINTERRUPTED `restarting` — **measured 2026-09-24: a container that reads `running` for a moment between restarts resets the clock and never gets there (gokapi, 385 restarts, „0 currently down"; R-667)** **Since controller v0.269.0 the box counts `RestartCount` instead and STOPS the app (§6.2, decision 28).** |
|
||||
| `starting`, `deploying` | no | mid-start |
|
||||
| `paused` | no | a deliberate user action |
|
||||
| `unknown` | no | [DESIGN] fail-OPEN — never manufacture a dead-app alert from an inconclusive read |
|
||||
@@ -91,13 +91,14 @@ member is dead; only the excuse is missing). Both are recorded at their sites.
|
||||
|
||||
---
|
||||
|
||||
## 5. The three suppressions, all at `classifyRunStates`
|
||||
## 5. The four suppressions, all at `classifyRunStates`
|
||||
|
||||
| Suppression | Rule | Expires? |
|
||||
|---|---|---|
|
||||
| **deliberate user stop** | `StateStopped` is not down **unless** the quiesce loop reports it failed to restart that stack | n/a — lifted by `failedRestart` |
|
||||
| **quiesce cycle** | a stack this backup cycle stopped is exempt | **yes**, 180 s after unquiescing |
|
||||
| **boot grace** | no evaluation for 90 s after controller start | **yes** |
|
||||
| **update hold** (v0.268.0, R-660) | an app HELD after a failed update is stopped by the product and has its own event (`app_update_held`, and `app_hold_no_whole_copy` when no copy brings it back whole); it is not "down" | **yes** — lifted with the hold (a restore, or the operator). A RESTORE hold (R-379) is deliberately not in the set: it has no event of its own |
|
||||
|
||||
**None of them latch.** [DESIGN, R-97b + R-88 Scenario D] A permanent suppression trades a loud false
|
||||
alarm for a silent real one, which is the same error as an over-eager alarm, in the opposite
|
||||
@@ -211,6 +212,58 @@ minutes later (90 m instead of 60). **One value, everywhere:** both staleness ch
|
||||
hardcoded 30 m / 1 h until then; pinned by `TestControllerStatus_FollowsConfiguredThreshold`). A running
|
||||
hub prints it at startup (`node_stale after 45m0s, node_down after 1h30m0s`).
|
||||
|
||||
**An out-of-memory storm gets its own, louder rung (R-636, controller v0.265.0 / hub v0.121.0, 2026-09-23).**
|
||||
`app_oom` stays exactly as it was: `warning`, operator-only, ONCE per container run (R-514) — the once
|
||||
is what stops a crash loop from mailing thousands of times (R-629). Beside it, `app_oom_storm`:
|
||||
|
||||
| event | severity | who | minted by | why that audience |
|
||||
|---|---|---|---|---|
|
||||
| `app_oom_storm` | **error** | **operator only** | controller v0.265.0, when the kernel's `oom_kill` counter of the SAME container run rises by **≥ 20 within 30 min**; once per run | raw container names and memory figures; the household's side is the dashboard tag |
|
||||
|
||||
**Why a counter and not the flag:** Docker's `OOMKilled` is sticky — true for the whole run after ONE
|
||||
kill — so "the key re-fired N times" measures only how long ago the first kill was. The kernel's
|
||||
`memory.events` `oom_kill` counts kills. **Why 20 in 30 minutes:** RomM's measured rate on 2026-09-22
|
||||
was 4,530 kills in six hours ≈ 375 per 30 min; a hiccup is 1–3. Live on 9202 (2026-09-23): RomM at 320M
|
||||
reached 21 kills 2.5 min after start and sent ONE storm; at 49 kills still one. **Not an app-down state:**
|
||||
the app still reads `running` and `IsDownState` is unchanged (§4). **Limit:** a container whose
|
||||
`OOMKilled` flag stays false in an LXC guest (R-528) is never read, so it never storms.
|
||||
|
||||
**The box STOPS a crash loop or an out-of-memory storm (decision 28 of `09` §3, R-667, controller v0.269.0 /
|
||||
hub v0.123.0, 2026-09-24).** Alarming was not enough: gokapi crash-looped for hours at 385 → 546 restarts
|
||||
while the ladder said „0 currently down" (the `restarting` clock resets, §4). Now, besides the alarms above:
|
||||
|
||||
| rule | threshold | what the box does | event |
|
||||
|---|---|---|---|
|
||||
| crash loop | **≥ 6 restarts within 10 min**, from Docker's `RestartCount` summed per app, sampled every scan (a drop = the container was recreated → the window starts again) | stops the app, records an `unhealthy_stop` hold (the same store as every hold, so no start path revives it), the page says so with a Start button | `app_stopped_unhealthy`, **warning**, **the household AND the operator** (default on, seeded add-only) |
|
||||
| out-of-memory storm | the `app_oom_storm` rule above (≥ 20 kernel kills within 30 min) | the same stop and hold | `app_oom_storm` (operator) + `app_stopped_unhealthy` |
|
||||
| Start pressed | — | lifts the hold: **one more try** | — |
|
||||
| stopped again within 24 h | — | stops it again; the sentence now says Felhom support is informed | `app_stopped_unhealthy` (repeat wording) |
|
||||
|
||||
**Why 6 in 10 minutes and not 10:** Docker's restart back-off caps a steady loop at about ONE restart a
|
||||
minute (gokapi: 539 → 546 in 7 min), so 10 in 10 sits on the edge and misses a steady loop; a FRESH
|
||||
container restarts fast (9 in 32 s after a Start, measured). No healthy app in any drill evidence (1,831
|
||||
harness samples, 40 live containers) restarted more than once on a first start; the one borderline shape
|
||||
is immich's first-start import (12 restarts, 2026-09-17 — broken that night; R-676 watches it).
|
||||
**Never judged:** an app that is deploying, updating, held, or stopped by a backup or a quiesce.
|
||||
**Precisely (read from source, controller v0.271.0, 2026-09-25):** "updating" covers an automatic update's
|
||||
whole step — pull, start, verify AND its undo — because `Updating` stays true until the step ends
|
||||
(`TestD28_NoCrashLoopStopDuringAnAutomaticStep`; the leg's own pages are therefore never stopped mid-step).
|
||||
"Deploying" does **not** cover a deploy's FIRST START: the flag clears when `compose up -d` returns, and the
|
||||
app is sampled from then on — a first start that restarts ≥ 6 times in 10 min is stopped (R-676). An automatic
|
||||
update is a product stop in the §5 sense: a step's restarts are the product's own, never an alarm.
|
||||
**Measured in the chaos hour (night 2026-09-24):** stopped at +185 s after a power cut mid-storm (the
|
||||
counting starts again after the boot), +116 s for a crash loop under a backup run, +102 s for a storm with
|
||||
the disk 1 GB above the floor.
|
||||
|
||||
**A thin pool is critical at 90 % (R-672, hub v0.124.0 + agent v0.133.0).** `storage_fill_*` judges an
|
||||
`lvmthin` target on the worse of data and metadata fill, warning 85 %, **critical 90 %** (other storages keep
|
||||
90/95 %). A full thin pool does not only refuse writes: every guest on it remounts read-only (demo-hp
|
||||
2026-09-24 — 95 % → 100 % in about a minute, 9201 read-only five minutes later). The agent requests an
|
||||
immediate host report when a pool crosses 90 %, so the alarm fires in seconds, not at the next 15-minute
|
||||
report. Grain: **per pool, 6 hours** (`customer:type:host/storage`) — the old `customer:type`, 1 hour let one
|
||||
pool silence another. The hub HAD alarmed on 2026-09-24 (operator mail at 100 %), but late and at the generic
|
||||
bands.
|
||||
|
||||
**Two event types added 2026-09-17, with who receives them:**
|
||||
|
||||
| event | severity | who | minted by | why that audience |
|
||||
@@ -221,6 +274,8 @@ hub prints it at startup (`node_stale after 45m0s, node_down after 1h30m0s`).
|
||||
| Family | Grain | Key carries | Why |
|
||||
|---|---|---|---|
|
||||
| app down (`app_start_failed`) | **per APP** | `…:<stack_name>` | no digest exists for it — see below |
|
||||
| update outcome (`app_update_undone`, `app_update_held`) | **per APP**, both legs | `…:<stack_name>` | hub v0.120.0 — one mail per app per outcome; the household leg has its own register (`perAppCustomerCooldownEvents`) |
|
||||
| OOM storm (`app_oom_storm`) | **per APP**, operator | `…:<stack_name>` | hub v0.121.0 — no digest; the controller already sends it at most once per container run |
|
||||
| backup run (`backup_run_failures`) | per RUN | `…:<run_id>` | a digest already lists every failing app; one per run |
|
||||
| tiered backup (`whole_guest_backup_failed`, …) | per TIER | `…:<tier>` | the tiers fail independently and mean different things |
|
||||
| everything else, incl. `crossdrive_failed` and `backup_integrity_failed` | per TYPE, per hour | — | coarse **on purpose** |
|
||||
@@ -255,6 +310,36 @@ message, not a wider cooldown.
|
||||
|
||||
---
|
||||
|
||||
## 6.3 Box alarms outside the app ladder: the tunnel and OS updates [DESIGN, hub v0.131.0, 2026-10-04]
|
||||
|
||||
These are **operator-only** (the household can act on none of them — except `host_restarted_after_crash`, the household's
|
||||
one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
|
||||
`TestOSUpdateEvents_OperatorOnlyExceptApplied`). They go through the same dispatcher and severity contract (§6.1):
|
||||
`info` is recorded and never mailed; `warning` and `error` are mailed.
|
||||
|
||||
| Event | Severity | Raised when | Cleared | Pinned by |
|
||||
|---|---|---|---|---|
|
||||
| `tunnel_down` | error | the box's two newest host reports say the tunnel is `not_running` (more than one report cycle, 15 min) and the one before did not | the first `running` after it → `tunnel_recovered` (info) | `api/tunnel_test.go` |
|
||||
| `os_update_stale` | warning | no successful OS leg for **7 days** while the switch is ON (agents that can run the leg only); the mail names the likely reason (box not reporting / the last leg's failure / no good night backup) | a successful leg | `TestAlarm_StaleLeg`, `TestAlarm_StaleNamesTheReason` |
|
||||
| `os_reboot_needed` | warning | the host has needed a reboot for **14 days** (from the FIRST scanned report that said so) | a scanned pass that finds nothing (agent ≥ 0.141.1 scans the host every pass) | `TestAlarm_RebootNeeded`, `TestRebootNeeded_ClearedByAScannedPass` |
|
||||
| `os_ring0_stalled` | error | ring 0 approved nothing for **7 days** in a layer while it has pending FAST-lane updates (a pending kernel does not count) | a new release | `TestAlarm_Ring0Stalled` |
|
||||
| `os_not_covered` | warning | a ring-1 box has had fast-lane packages no approved release names for **14 days** | the packages are covered or gone | `TestAlarm_NotCovered` |
|
||||
| `host_crash_restart` | warning | the box's crash guard reports a NEW unclean boot (a crash, a power cut or a hard reset; hub v0.132.0) | — (one per boot) | `api/crash_test.go` |
|
||||
| `host_crash_guard_tripped` | error | the guard tripped: the next crash leaves the box OFF | the re-arm → `host_crash_guard_rearmed` (info) | `api/crash_test.go` |
|
||||
| `host_kernel_oops` | warning | a kernel oops this boot (taint D) — the box keeps running | — (once per boot) | `api/crash_test.go` |
|
||||
|
||||
- **`unknown` never alarms** (R-96 rule 3): a probe that could not ask is neither up nor down. An `unknown` report
|
||||
breaks a `not_running` run.
|
||||
- **A stopped cloudflared heals itself before the hub can see it** (measured 2026-10-04): the controller's
|
||||
protected-container check recreates it within 5 minutes, and the host reports every 15. So `tunnel_down` catches
|
||||
what the box cannot heal — a running container with no connection (wrong token, blocked network).
|
||||
- The OS alarms are checked **hourly**, re-sent at most **once a week** while true, and forgotten when false, so the
|
||||
next occurrence is announced again. The four numbers are configuration (`OS_ALARM_STALE_AFTER`,
|
||||
`OS_ALARM_REBOOT_AFTER`, `OS_ALARM_RING0_STALL_AFTER`, `OS_ALARM_NOT_COVERED_AFTER`) — *decided by CC unattended,
|
||||
operator may reverse* (`11` §8.3).
|
||||
|
||||
---
|
||||
|
||||
## 7. The intent test [DESIGN, R-386 — CLOSED controller v0.223.0]
|
||||
|
||||
**"The customer stopped this" is asked of the FIELD THAT RECORDS IT, never inferred from the state.**
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -208,8 +208,11 @@ vissza") — R-564. The hub copy gate and the catalog have no language concept
|
||||
|
||||
## 7. The catalog's copy model
|
||||
|
||||
**[DESIGN, proposed — slice 5 decides after seeing it work]** A sibling block per language inside
|
||||
`.felhom.yml`, not a sibling file:
|
||||
**[FACT] MEASURED 2026-09-20 (controller v0.257.0 + catalog pilot). The proposal below was tried and
|
||||
it holds; what changed under measurement is written out, because the numbers it was specced against
|
||||
were wrong in two places.**
|
||||
|
||||
A sibling block per language inside `.felhom.yml`, not a sibling file:
|
||||
|
||||
```yaml
|
||||
description: Titkosított jegyzetek…
|
||||
@@ -222,12 +225,70 @@ i18n:
|
||||
first_steps: [...]
|
||||
```
|
||||
|
||||
Why: one app's copy stays in one file a reviewer sees whole; the controller's `yaml.Unmarshal`
|
||||
ignores unknown keys (`internal/stacks/metadata.go` L317), so older controllers are unaffected; a
|
||||
missing English field falls back field by field. Cost: 835 strings, ~4 984 words; the catalog copy
|
||||
gates need an English rule. Read on privatebin, nextcloud and immich (inventory §2.5).
|
||||
Why: one app's copy stays in one file a reviewer sees whole; the controller's `yaml.Unmarshal` ignores
|
||||
unknown keys (`internal/stacks/metadata.go` L316 — **verified: this repo constructs no `yaml.Decoder`
|
||||
at all, so there is no `KnownFields` anywhere to turn strict**), so older controllers are unaffected;
|
||||
a missing English field falls back field by field.
|
||||
|
||||
---
|
||||
**[FACT] The read path** is `Metadata.For(lang)` in `controller/internal/stacks/metadata_i18n.go`,
|
||||
reached only through `LocalizeStacks` / `LocalizeStackPtr` / `Stack.MetaFor`.
|
||||
|
||||
- **`For("hu")` is the parsed struct with `I18n` cleared and nothing else**, deep-compared against all
|
||||
53 real catalog files (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
|
||||
§1 therefore holds by construction for catalog copy, and `LocalizeStacks` returns the INPUT SLICE
|
||||
for Hungarian rather than a copy, so nothing on the Hungarian path is even touched.
|
||||
- **Fallback is field by field, and blank counts as absent** (ruling 3). A half-translated app is a
|
||||
legal state — that is what makes the batches pushable one at a time.
|
||||
- **The three prose lists replace WHOLE** (`use_cases`, `first_steps`, `prerequisites`): merging item
|
||||
3 of one language with item 4 of another produces a list nobody wrote. **Every other list is
|
||||
matched by its own key** — `deploy_fields` by `env_var`, options by `value`, `optional_config`
|
||||
groups by `match_group` (the Hungarian `group` value they translate; a group has no other identity),
|
||||
its fields by `env_var`, integrations by `target`, data paths by `path`. Position matching
|
||||
mistranslates silently the first time a Hungarian field is inserted above another, and the page
|
||||
still looks right.
|
||||
- **`For` never writes through.** The metadata lives in the stack manager's cache and is shared by
|
||||
concurrent requests; an in-place merge would put one household's language on another's page.
|
||||
- **Two copy producers have no request and therefore no language** — the integration rows
|
||||
(`internal/integrations`) and the initial-credentials note (`internal/stacks/initialcreds.go`).
|
||||
Both are re-taken from the localised metadata in the handler.
|
||||
|
||||
**[FACT] The numbers, measured on `94bc5febaca2`** — the inventory's were close on one count and wrong
|
||||
on another, and both matter to the gate:
|
||||
|
||||
| | inventory / task | measured |
|
||||
|---|---|---|
|
||||
| copy strings | 835 | **1 032** (incl. 13 placeholders the table omitted, and every ASCII label) |
|
||||
| with a Hungarian letter | 832 | **832** ✔ |
|
||||
| ASCII-ONLY Hungarian | „Igen", „Nem", „Nincs" — three | **~120**. Those three do not occur; the three ASCII-only OPTION labels are „Magyar", „Angol", „Magyar + Angol" — language names, not yes/no words |
|
||||
|
||||
The ASCII-only count is the one that decides an instrument: „Aldomain" (53×), „A szerver domain neve"
|
||||
(53×), „Jelentkezz be: …" (5×), „Magyar", „Angol", „Titkos kulcs", „Oszd meg a linket". **An
|
||||
accent-only scan passes every one of them inside an English block** — the same blind spot §2.3 records
|
||||
for the English page test (R-565), arriving again in a different repo. So
|
||||
`app-catalog-felhom.eu/scripts/check-copy-i18n.py` folds and matches word-bounded stems, with a
|
||||
positive and two negative controls printed on every run.
|
||||
|
||||
**[FACT] The gate**, fifth row of `catalog_gates.py`, static, in the pre-push hook:
|
||||
|
||||
1. **FREEZE** — every Hungarian copy string equals `scripts/copy_freeze/hu.json`, captured in its own
|
||||
commit before any translation. Runs on all 53 apps whatever scope is named. A new app must be
|
||||
admitted with `--add-app NAME --reason "…"`.
|
||||
2. **STRUCTURE** — the `en` block may carry copy fields and nothing else; every key-matched entry must
|
||||
have a Hungarian twin, or it is INERT on the box and the translator never learns.
|
||||
3. **LANGUAGE** — no accented letter, no ASCII-only Hungarian, no „please"/„kindly", no English
|
||||
retrieval promise the Hungarian does not make, the app name and „Felhom" preserved.
|
||||
4. **CREDENTIALS** — the login tokens inside `default_creds` and the initial-credentials note survive
|
||||
translation verbatim, matched on WORD boundaries.
|
||||
5. **RATCHET** — `EN_MISSING_CEILING`, convicting above and below, counted over the WHOLE catalog
|
||||
whatever scope is named.
|
||||
|
||||
33 decoy cases (R-421). **Three defects in the gate were found by its own decoys and not by reading
|
||||
it** — R-592.
|
||||
|
||||
**[FACT] What a gate cannot judge**, and the pilot report lists per app: whether an app's „first
|
||||
steps" name the buttons that app's ENGLISH interface actually shows. The Hungarian was written against
|
||||
whatever interface the author had; the English UI label may differ. Those lines are marked unverified
|
||||
and belong to the slice-6 walk (R-561).
|
||||
|
||||
## 8. Dates, numbers, plurals — as found
|
||||
|
||||
@@ -281,6 +342,12 @@ ENGLISH internal words — the same shape, one language over, filed as R-569 rat
|
||||
|
||||
## 10. The plan
|
||||
|
||||
**ALL SIX SLICES ARE DONE** — slice 1 (2026-09-17, v0.250.0), slice 2 (2026-09-18, v0.254.0),
|
||||
slice 3 (2026-09-18, hub v0.118.x), slice 4 (2026-09-18, ISO 1.29.0), slice 5 (2026-09-20, v0.257.0 +
|
||||
the whole catalog), slice 6 (2026-09-20, v0.258.0 + the guide + the walk). What remains is written in
|
||||
§10.6b's residue table, and every line of it has a row.
|
||||
|
||||
|
||||
Costs are CC-hours, estimated from the spike (mechanism + three pages + layout + gates ≈ one working
|
||||
session). Each slice ends with the parity gate green for every page it touched.
|
||||
|
||||
@@ -477,10 +544,6 @@ reported 60 520 changed lines and measured nothing. A line-index compare is not
|
||||
|
||||
---
|
||||
|
||||
## 11. Operator decisions
|
||||
|
||||
These are rulings, not proposals. Anything specced against a different assumption is wrong.
|
||||
|
||||
### 10.4 Slice 3 — the hub's e-mails follow the household (hub v0.118.x, controller v0.256.x). SLICE 3 CLOSED.
|
||||
|
||||
SHIPPED 2026-09-18. Full design: `05-hub-architecture.md` §15.
|
||||
@@ -547,3 +610,216 @@ one mail that did not follow the language — and the one an operator would use
|
||||
7. **Interface nouns translate** (operator: „translate"). „Indítópult" → Launcher, „Vezérlőpult" →
|
||||
Dashboard, „Biztonsági mentés" → Backup, as the spike built. App names and „Felhom" stay as they
|
||||
are. Every slice follows this.
|
||||
|
||||
### 10.5 Slice 4 — the console banner and the download page (ISO 1.29.0 source, R-559)
|
||||
|
||||
SOURCE SHIPPED 2026-09-18. **The image is built but NOT published** — that is an operator step; see
|
||||
`documentation/audits/i18n-slice4-2026-09-18/README.md`.
|
||||
|
||||
**The box's screen is BILINGUAL, and that is a decision rather than a stage.** At the moment the
|
||||
pairing code is shown, nobody has told the box who owns it — there is no language to follow. So the
|
||||
three console texts carry the Hungarian block byte-for-byte as before, then one blank line, then an
|
||||
English block, inside the same frame: `print_pairing_banner`, `print_bound_banner` and
|
||||
`install_felhom_issue` (with the `postinst`'s byte-coupled copy of the issue text). The GRUB entries
|
||||
gain an English half. **§16 default taken: bilingual for good** — it needs no hub change, no protocol
|
||||
field, and it has no way to show the wrong language.
|
||||
|
||||
**The Hungarian is a golden, not a grep** (`scripts/iso/test/golden/*.hu.txt`, captured before any
|
||||
English existed). The harness also pins: no Hungarian letter in the English block (with the Hungarian
|
||||
block as the positive control), every line ≤ 80 columns, and **the whole paint ≤ 25 rows**. The
|
||||
pairing banner is **24 rows on a 25-row console** — one row of margin, which is why the height is
|
||||
pinned: two more lines and the Hungarian pairing code, at row 5, scrolls off the top.
|
||||
|
||||
**The download page has an English twin** at `felhom.eu/en/download`, linked both ways with
|
||||
`hreflang`. The rest of the marketing site stays Hungarian (ruling 1b). A site gate refuses the two
|
||||
pages naming different installer files or checksums.
|
||||
|
||||
**The release gate had to be amended.** G16 required every Felhom-authored string to be Hungarian and
|
||||
would have stopped this publication; ruling 1b supersedes that scope, so G16 was rewritten rather
|
||||
than waived — Hungarian FIRST, pinned by the golden.
|
||||
|
||||
**Gaps filed:** R-586 (the ISO harness had been red for two days and is in no gate), R-587
|
||||
(root-password files in the publish source directory), R-588 (release records live in two places).
|
||||
|
||||
### 10.6 Slice 5 — the app catalog's own words (controller v0.257.0 + the catalog pilot, R-560)
|
||||
|
||||
PART A + PART B SHIPPED 2026-09-20. **PART C — the other fifty apps — is deliberately not started:**
|
||||
the §16 default was to stop after the pilot for the operator's read, and nobody has read it yet.
|
||||
|
||||
**The format is no longer a proposal — §7 is now measured.** The controller reads the block; three
|
||||
apps carry one; the fleet is unaffected until the floor moves.
|
||||
|
||||
**What was proven, live, rather than reasoned about:**
|
||||
|
||||
- **The English pages show the English.** On demo-hp (0.257.0) the app pages for privatebin,
|
||||
paperless-ngx and romm, the Apps list and two deploy pages render the catalog's English text.
|
||||
- **The Hungarian did not move.** The same seven pages fetched in Hungarian before and after the
|
||||
catalog push are byte-identical apart from the per-session CSRF token — equal raw byte counts
|
||||
(43 253 / 41 532 / 41 547 / 147 121 / 74 326 / 70 290) and equal hashes once the token is
|
||||
normalised.
|
||||
- **An old controller ignores the block.** demo-felhom runs **0.255.0**. With the pilot synced onto
|
||||
that box — confirmed positively: the sync named the three apps and the block is in both the cache
|
||||
and the stack copy — its pages hash identically before and after, and its log carries no parse
|
||||
warning **in a 93-line window that includes the sync lines**, so the absence is a fact and not a
|
||||
dead log.
|
||||
|
||||
**What the English pages still carry in Hungarian, measured with an ASCII-folded scan plus an accent
|
||||
scan, both controlled:** exactly three things, and one of them is correct.
|
||||
|
||||
1. The language picker's own „Magyar" button — a language picker names each language in its own
|
||||
tongue. Not a defect.
|
||||
2. The update badge „Naprakész" and its tooltip — **R-589**. Slice 1 listed it (§2.3); slice 2 was to
|
||||
take it and closed without it.
|
||||
3. The data-folder card's consequence sentence — **R-590**, and this one makes a promise about the
|
||||
customer's files while the label above it is already English.
|
||||
|
||||
Everything else Hungarian on an English Apps LIST is simply the fifty apps nobody has translated yet.
|
||||
|
||||
**Gaps filed:** R-589, R-590, R-591 (`Stack.Copy()` has one shallow field and it is the new one),
|
||||
R-592 (three defects inside the new gate, each found by its own decoy).
|
||||
|
||||
**Evidence:** `documentation/audits/i18n-slice5-2026-09-20/`.
|
||||
|
||||
**PART C SHIPPED THE SAME DAY.** The operator read the pilot's English and said go, so the other
|
||||
fifty followed its voice in three pushes — 17, 17 and 16 apps; 319, 317 and 306 strings. **1 031 of
|
||||
the catalog's 1 032 customer-facing strings now carry an English twin.**
|
||||
|
||||
**[DESIGN, decided by CC] The blocks are GENERATED from a flat `{path: english}` map, not
|
||||
hand-written.** Fifty nested blocks whose keys must match the Hungarian side exactly is fifty
|
||||
chances to mistype an `env_var` — and **a mistyped key is INERT on the box, not an error**, so the
|
||||
translator never learns. The generator builds from the same flat paths the freeze uses, which are
|
||||
derived from the Hungarian file itself, so a key that does not exist on the Hungarian side cannot be
|
||||
written at all. The review then happens on the map, where one line is one string, instead of on
|
||||
YAML indentation.
|
||||
|
||||
**[FACT] The one string left untranslated, on purpose.** papra's
|
||||
`deploy_fields[AUTH_SECRET].description` reads „Az alkalmazás aldomainje" — the sentence that belongs
|
||||
on `SUBDOMAIN`, sitting on a session-signing key (R-593). §1 forbids changing the Hungarian in a
|
||||
localisation release, and translating a wrong sentence faithfully would ship the error in a second
|
||||
language. So it falls back. **That is why `EN_MISSING_CEILING`'s floor is 1 rather than 0, and it is
|
||||
written into the ceiling's own comment** so a later session does not "reach zero" by editing
|
||||
Hungarian.
|
||||
|
||||
**[FACT] Measured on the English Apps list, all 53 apps: ZERO Hungarian app descriptions.** The only
|
||||
Hungarian left on that page is the „Naprakész" badge (R-589, ten occurrences) and the language
|
||||
picker naming itself. The Hungarian Apps list is identical to the pre-slice capture once the
|
||||
per-session CSRF token and Docker's own „Up N hours" string are normalised.
|
||||
|
||||
**[FACT] The gate convicted two of the translator's own sentences and was HALF right.** Vaultwarden's
|
||||
invite step ended „…can open an account", and the English retrieval-promise pattern reads `can …
|
||||
open` as the claim that sealed backups can be opened. Opening an ACCOUNT is not that claim, so the
|
||||
conviction was a false positive — and the wording was also the weaker wording, so it became „can
|
||||
sign up". **But the gate has no way to REGISTER a true occurrence**, which the shared vocabulary's
|
||||
own design calls for; R-594.
|
||||
|
||||
**[FACT] The fleet floor was raised to 0.257.0** the same session (operator asked), `min_agent`
|
||||
0.131.0 declared — above the vouched golden 0.246.0, so the declaration is what carries it (R-472).
|
||||
Hub: `managed floor SERVED for demo-felhom: floor 0.257.0, agent requirement "0.131.0" from declared
|
||||
(golden 0.246.0)`. **demo-felhom went 0.255.0 → 0.257.0 by itself in under 12 seconds**, healthy,
|
||||
its own log reading `settle-gate: GO — at/above floor 0.257.0`, and it then rendered the English
|
||||
tagline — the floor delivered function, not just a version string. Peti's box and tester-1 are DOWN
|
||||
and take it unattended when they return; untested on this version.
|
||||
|
||||
|
||||
### 10.6b Slice 6 — the guide in English, and a stranger's first hour (controller v0.258.0, R-561)
|
||||
|
||||
SHIPPED 2026-09-20. **All six slices are now done.** The slice had three parts and all three landed:
|
||||
the four leftovers slice 5 found live (v0.258.0), the guide's English twin, and the walk.
|
||||
|
||||
**Part 0 — the four leftovers.** R-589 (the update badge) and R-590 (the data-folder card's backup
|
||||
promise) went through `localeFuncs` and `s.msgLang`; R-573 (the agent-channel and endpoint-drift
|
||||
banners) is now keyed by the checker's own CLASSIFICATION, with the composed sentence kept as a
|
||||
fail-open fallback. **R-572 was not what its row said** — measured, no template and no Go file called
|
||||
`pruneLabel`/`nextPruneLabel`; they were dead func-map entries returning Hungarian, so they were
|
||||
**deleted**, and the deletion is fail-loud (a template naming a removed func panics `loadTemplates`
|
||||
at startup, proven).
|
||||
|
||||
**Part A — the guide.** `runbooks/VOLUNTEER-first-hour.en.md`, a twin: 16 sections in the same order,
|
||||
identical step counts, table rows and warning blocks per section (0 sections differing in structure).
|
||||
**Word counts are NOT a twin** — English runs 19 % longer overall and up to 42 % on the short
|
||||
sections, because Hungarian is agglutinative; the ±15 % criterion the task set does not survive
|
||||
contact with this language pair, and structure was measured instead. **The walk then corrected the
|
||||
guide in five places** — it had been written from the bundle, and the bundle is not the screen.
|
||||
|
||||
**Part B — the walk.** `audits/DRILL-first-hour-en-0258-2026-09-20.md`. The verdict:
|
||||
**not yet ready for an English-speaking tester, because the claim page answers in Hungarian (R-596)** —
|
||||
everything else held.
|
||||
|
||||
**[FACT] What stays Hungarian for an English household — built ONLY from what the stranger saw:**
|
||||
|
||||
| what | why | row |
|
||||
|---|---|---|
|
||||
| ~~the **claim page's messages**~~ | ~~composed sentences passed into page data~~ | **R-596 CLOSED**, controller 0.259.0 |
|
||||
| ~~the **setup code** and the **owner passphrase**~~ | ~~one Hungarian wordlist~~ | **R-597 CLOSED**, hub 0.119.0 |
|
||||
| ~~the **Backup page's two protection warnings** and its two target names~~ | ~~composed sentences in Go~~ | **R-598 CLOSED**, controller 0.259.0 |
|
||||
| the menu word **"Debug"** | it is already English; the complaint was the Hungarian household's | R-516 item 1 |
|
||||
| the **operator's copy** of every event | operator-tier is Hungarian **by design** (ruling 1) | — |
|
||||
| the **18 formal „ön" forms** | a localisation release may not change Hungarian bytes (§1); counted, ratcheted | R-516 |
|
||||
| the **apps' own English UIs** | not Felhom's copy — and for this reader an advantage | — |
|
||||
|
||||
**That is the honest residue.** Everything not in this table — the download page, the console, all
|
||||
three customer mails, the bind page and its refusal, the dashboard, Apps, Storage, Monitoring, the
|
||||
Launcher, both app pages, the whole catalog, the language switch — was **English with zero Hungarian
|
||||
lines** on a box installed from scratch that day.
|
||||
|
||||
**R-516 does NOT close**, and `audits/i18n-slice6-2026-09-20/R-516-item-by-item.md` says why item by
|
||||
item: more than half its twelve items are about what a **Hungarian** household reads, and an English
|
||||
walk cannot see them.
|
||||
|
||||
**R-214 closed as a side effect** — the console's last paint is now the bilingual "the box is linked"
|
||||
banner. The 2026-09-14 walk recorded it as still reproducing.
|
||||
|
||||
### 10.6c Slice 6's residue, closed (controller v0.259.0 + hub v0.119.0, 2026-09-21)
|
||||
|
||||
The three rows above are closed. What is worth keeping is not that they closed but **what each one
|
||||
turned out to be**, because two of the three were not what the row said.
|
||||
|
||||
**R-596 — the claim page.** Fourteen live call sites carrying **nine** distinct messages, not the
|
||||
sixteen literals the row counted; one of the sixteen (`data["Title"]`) was **dead** — `claim.html` is
|
||||
standalone and `.Title` belongs to `layout.html` — and was deleted rather than translated. The
|
||||
anonymous, cookie-less page takes its language from `customer.language` in `controller.yaml`
|
||||
(`langFor` → `settings.GetLanguage` → `configLanguage`); that chain was an unpinned assumption and is
|
||||
now a test.
|
||||
|
||||
**R-598 — the backup warnings.** `degradedMessageFor` now returns a **key**, so the decision stays
|
||||
language-free and in one place while the words are chosen by whoever knows the reader. The English
|
||||
is asserted to carry the same NEGATION the Hungarian does — *protects against corrupted files, but
|
||||
**not** against a disk failure*. An English sentence that promised disk-failure protection would be
|
||||
worse than leaving it Hungarian.
|
||||
|
||||
**R-597 — the codes.** Two of the row's three secrets were mis-attributed:
|
||||
|
||||
- **The recovery code was never Hungarian.** `felhom-agent` mints it from the **EFF large wordlist**
|
||||
and always has — ten English words, ≈129 bits. The hub does not own it, and no row was added for
|
||||
it: a second definition of that secret is exactly the drift this section exists to prevent.
|
||||
- **No claim mail states a word count.** The only count wording in the product is the bind page's
|
||||
passphrase hint, and its English half is now count-free.
|
||||
- The setup code and the owner passphrase now follow the household's language, one word longer in
|
||||
English so the entropy **never drops**: setup 3 hu (44.6 bits) → 4 en (51.7); passphrase 5 hu
|
||||
(74.3) → 6 en (77.5). The floor is computed from the embedded lists at test time, not asserted
|
||||
against a constant.
|
||||
|
||||
**[FACT] The defect class is now four instances deep** — R-573, R-590, R-596, R-598: *a composed
|
||||
sentence handed to a renderer as page data*. Nothing structural sees it. A template-parity fixture
|
||||
renders the field faithfully; `TestI18nEnglishPages` reads a template, not a struct; the Go-parity
|
||||
gate proves the Hungarian is unchanged and says nothing about which language reached the page. **The
|
||||
only instruments that find it are a live English page and a handler-level render test**, and this
|
||||
release added the second for both surfaces.
|
||||
|
||||
**[FACT] A method finding worth more than the result.** The first live check of the backup page used
|
||||
the `felhom_lang` cookie and got the **Hungarian** page for `en`. That is correct: `langFor` step 2
|
||||
says a request carrying a **session** reads the household's saved setting and deliberately ignores
|
||||
the visitor cookie. The cookie is the right instrument for the anonymous claim page and the **wrong**
|
||||
one for any signed-in page — where `?lang=` is. A session that had run only the cookie probe would
|
||||
have concluded R-598 was unfixed and fixed it again. Recorded in
|
||||
`audits/i18n-closing-2026-09-21/live/backups-page.md`.
|
||||
|
||||
**What did NOT get a live walk:** the degraded and absent-drive warnings themselves. Guest 9201 has a
|
||||
real backup drive, so it is healthy and renders nothing — by design (E-2 Scenario E) — and producing
|
||||
either state would mean un-assigning a live box's backup target. They are covered by render tests
|
||||
through the real handler. **The next English walk on a one-drive machine is what actually closes
|
||||
that**, and it is the same walk R-516 is waiting for in the other language.
|
||||
|
||||
## 11. Operator decisions
|
||||
|
||||
These are rulings, not proposals. Anything specced against a different assumption is wrong.
|
||||
|
||||
@@ -0,0 +1,567 @@
|
||||
# 11 — Operating-system updates: the host, the guest and the Docker engine
|
||||
|
||||
> | | |
|
||||
> |---|---|
|
||||
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1).** Ratification is Viktor's review, not an editor's. |
|
||||
> | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. |
|
||||
> | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 |
|
||||
> | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. |
|
||||
>
|
||||
> **How to read this document.** Each statement has a label:
|
||||
>
|
||||
> - **[FACT]**: an observed property, with a `file:line`, a register row or an audit path.
|
||||
> - **[RULED]**: an operator decision, with its date.
|
||||
> - **[PROPOSAL]**: the reviewer's design. It is not decided and not built.
|
||||
> - **OPEN**: a question that nobody has answered yet. The spike measures it. Nobody guesses it.
|
||||
>
|
||||
> **Why this file exists.** No architecture document covered operating-system updates. `00` §G marks the
|
||||
> capability MISSING (added 2026-10-03). The finding is **R-812**. The roadmap intention is **R-808**.
|
||||
> **The register carries the work. This file carries the reasoning. The source is the truth.**
|
||||
>
|
||||
> **Spike corrections, 2026-10-04** (`audits/os-updates-spike-2026-10-04/`). Each is made in place: the reviewer's
|
||||
> text is struck (~~like this~~) and the measured text follows, labelled `[FACT]`, with its id.
|
||||
>
|
||||
> | Id | Where | What the spike changed |
|
||||
> |---|---|---|
|
||||
> | C1 | §2 | The agent may run `apt-get install` for TWO packages, not one (dnsmasq and wireguard-tools). |
|
||||
> | C2 | §5.3 | Debian keeps two versions, not one; for box packages no fix was replaced within 14 days in 3 months; `snapshot.debian.org` works from a box in seconds. |
|
||||
> | C3 | §5.2 | The lanes must follow the package's ORIGIN, not its name: 40 Proxmox-repository packages have ordinary names (ZFS, the Secure Boot shim, Ceph, corosync, chrony, CPU microcode). |
|
||||
> | C4 | §5.6 | `--next-boot` is NOT a one-shot on these GRUB hosts. The fallback works only after a boot that reaches userspace. Installing a kernel alone makes it the default. Only a software watchdog runs. |
|
||||
> | C5 | §5.6, §6 row 4 | Docker's `live-restore` keeps every container running across an engine update (measured). Turning it OFF again stops every container and starts none. |
|
||||
> | C6 | §5.6 | A guest snapshot works on LVM-thin (customer guests) but not on `dir` storage; the snapshot rollback itself is unmeasured; a backup-restore undo took 73 s. |
|
||||
> | C7 | §6 row 2 | A killed `apt` run does not recover by itself; the repair took ~5 s. |
|
||||
> | C8 | §1, §6 row 14 | `cloudflared` is not a host package: it is a container in the guest, pinned by the controller since June. It belongs with the controller's infrastructure pins, not this file's lanes. |
|
||||
> | C9 | §5.3 | The approved list must record what ring 0 RUNS healthy, not only what it installed that night. |
|
||||
> | C10 | §5.5 | Restore-tests and agent updates are not windowed; the host's `apt` timers install nothing today. |
|
||||
> | C11 | §5.2 | A Debian (fast-lane) update leaves PID 1, `lxc-start`, the Proxmox daemons and dockerd on the old library: its full effect needs a restart the fast lane does not do. |
|
||||
> | C12 | §6 row 9 | The two demo hosts differ by 7 packages, including the CPU microcode (AMD vs Intel) and Secure Boot (on vs off). |
|
||||
|
||||
---
|
||||
|
||||
## 0. In plain language
|
||||
|
||||
A box runs three layers that we install and never update: the Proxmox host, the small Debian system
|
||||
inside the guest, and the Docker engine. App images are updated (see `09`). The layer under them is not.
|
||||
A box lives in a home for years, so this is a security gap.
|
||||
|
||||
The proposal has two lanes. **The fast lane** applies Debian's security fixes automatically, but only
|
||||
the exact versions that already ran well on the demo boxes for 1–2 days. **The slow lane** covers the
|
||||
kernel, Proxmox and Docker. These cause restarts or big changes, so a person approves them one version at
|
||||
a time, and they run only at night. The tested versions are recorded automatically from what the demo
|
||||
boxes installed. Nobody keeps a hand-written list.
|
||||
|
||||
---
|
||||
|
||||
## 1. Scope
|
||||
|
||||
**In scope.**
|
||||
- The Proxmox host on an **appliance** install: the Debian 13 base, the Proxmox packages, and the kernel.
|
||||
- The guest (the customer LXC): its Debian 13 packages.
|
||||
- The Docker engine inside the guest (`docker-ce`, `docker-ce-cli`, `containerd.io`).
|
||||
- How the household and the operator are told, and how a failed update is undone.
|
||||
|
||||
**Out of scope.**
|
||||
- **App images.** `09` covers them (the ladder, the monthly same-tag re-test).
|
||||
- **The controller and agent binaries.** Their own self-update covers them (`03`, the self-update section; `09` R-608).
|
||||
- **DooPlex and ep0.** The operator updates them by hand (`runbooks/offsite-endpoint.md`).
|
||||
- **A BYO host** (the owner brought their own Proxmox). `[FACT]` The installer leaves its repositories
|
||||
alone: *"apt repo alignment skipped (byo — the owner manages repos)"* (`scripts/felhom-host-install.sh`,
|
||||
`align_apt_repos`). `[PROPOSAL]` On a BYO box we update the guest and Docker only, never the host.
|
||||
- **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).
|
||||
- **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in
|
||||
the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since
|
||||
2026-06-11) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
|
||||
— the app-image question (`09`), not an OS package. The gap is R-838.
|
||||
|
||||
---
|
||||
|
||||
## 2. What a box runs, as measured
|
||||
|
||||
| Layer | What it is | Source of packages | Evidence |
|
||||
|---|---|---|---|
|
||||
| Host | Proxmox VE 9.2 on Debian 13, LVM-thin | Debian mirrors + `pve-no-subscription` (enterprise repo switched off on appliance installs) | `[FACT]` `01` §2; `felhom-host-install.sh` `align_apt_repos` (~L2130–2136) |
|
||||
| Host kernel | The only kernel on the box. The guest has none. | `pve-no-subscription` (`proxmox-kernel-*`) | `[FACT]` LXC shares the host kernel (`01` §2: "one LXC/kernel/Docker daemon") |
|
||||
| Guest | Unprivileged LXC, `nesting=1,keyctl=1`, from `debian-13-standard_13.1-2` | Debian mirrors | `[FACT]` `felhom-agent/configs/build-golden.sh:67,101-103` |
|
||||
| Docker engine | `docker-ce`, `docker-ce-cli`, `containerd.io`, installed at golden BAKE time | `download.docker.com/linux/debian trixie stable` | `[FACT]` `build-golden.sh:108-125` |
|
||||
| Docker settings | `containerd-snapshotter: false`, json-file log caps. **No `live-restore`.** | baked `daemon.json` | `[FACT]` `build-golden.sh:138-144` |
|
||||
| Controller | A container in the guest. A Docker engine restart restarts it too. | Felhom registry | `[FACT]` `03` §1 |
|
||||
|
||||
**[FACT] Nothing updates any of these layers today** (R-812, searched 2026-10-03). The installer
|
||||
says *"No upgrades are run — repo alignment only"* (`felhom-host-install.sh` ~L2135). A fresh install
|
||||
gets the Docker engine that was current when its golden was baked, and keeps it.
|
||||
|
||||
~~**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist
|
||||
(`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.~~
|
||||
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:58`
|
||||
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:107`) and `:194`
|
||||
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:628`). Nothing else.
|
||||
`apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4
|
||||
proposes how to avoid that.
|
||||
|
||||
---
|
||||
|
||||
## 3. Operator rulings
|
||||
|
||||
**2026-10-03.** R-808 is on the roadmap at P2: *"every box receives operating-system security
|
||||
patches on a schedule, and a failed update is undone."*
|
||||
|
||||
**[RULED] 2026-10-04: the fast lane follows an approved list, with a 1–2 day wait.** Every update
|
||||
runs on the demo boxes first. The other boxes install only the exact versions that the demo boxes ran
|
||||
without trouble. The exact wait is set when the feature is built. **Rejected:** Debian's own
|
||||
`unattended-upgrades` with no wait. It is simpler and common, but a bad update would reach every
|
||||
customer at the same time.
|
||||
|
||||
**[RULED] 2026-10-04: the off-site backup topic is closed first.** The same brief carries both
|
||||
topics, and the backup part runs first.
|
||||
|
||||
The two-lane split (§5.2) is the reviewer's proposal. The operator's ruling above assumes it, but he
|
||||
has not ruled on it as such.
|
||||
|
||||
---
|
||||
|
||||
## 4. The constraints that shape the design
|
||||
|
||||
1. **Unattended.** Nobody is at the box. Nobody answers a question that `apt` asks.
|
||||
2. **No screen.** If the box does not boot, the household sees only that nothing works.
|
||||
3. **One kernel for everything.** A host kernel update needs a host reboot. A reboot stops every app.
|
||||
4. **The controller lives in Docker.** A Docker engine update restarts the controller in the middle of
|
||||
its own work. So the controller cannot drive a Docker update. The agent, on the host, must.
|
||||
5. **The host has no whole-system backup.** The guest has three tiers (`07`). The host has none.
|
||||
A host update can only be undone by installing the previous version again.
|
||||
6. **Every update must already have run on a box we own.** This is the lesson of the update arc
|
||||
(`09` §3 decision 13: "the test decides").
|
||||
7. **Few packages.** Each extra package is one more thing to update and break. The guest is the
|
||||
Debian standard template plus Docker. Keep it that way.
|
||||
|
||||
---
|
||||
|
||||
## 5. The proposed shape `[PROPOSAL]`
|
||||
|
||||
### 5.1 Two rings
|
||||
|
||||
- **Ring 0:** demo-felhom (N100) and demo-hp. They are disposable (`runbooks/target-selection.md`), and
|
||||
they have different hardware. They take every update first.
|
||||
- **Ring 1:** every other box. It takes only what ring 0 approved.
|
||||
|
||||
### 5.2 Two lanes
|
||||
|
||||
| | Fast lane | Slow lane |
|
||||
|---|---|---|
|
||||
| What | Debian packages on host and guest, from `trixie-security` and the stable point releases, **except** the slow-lane list | Kernel (`proxmox-kernel-*`), Proxmox (`pve-*`, `proxmox-*`, `lxc-pve`, `qemu-server`, …), Docker (`docker-ce*`, `containerd.io`) |
|
||||
| Restart | A service restart at most. No reboot. | Kernel: a host reboot. Docker: every container restarts. Proxmox: its services restart. |
|
||||
| Approval | Automatic. A version is approved when ring 0 ran it and stayed healthy for the wait (1–2 days). | A person approves one version at a time, like a controller floor. |
|
||||
| Urgent fix | The operator can approve a version on the same day once ring 0 has run it. | The same. |
|
||||
| When | The night window (§5.5) | The night window. A kernel reboot only on a night the operator scheduled. |
|
||||
|
||||
~~Which packages count as slow lane is a list the spike checks (OPEN Q7). A Debian package that restarts
|
||||
something big (for example `systemd`, `libc6`, `openssh-server`) may belong in the slow lane too.~~
|
||||
|
||||
**[FACT] (C3) The lane must be decided by ORIGIN, not by name.** On demo-hp, 40 pending packages come from the
|
||||
Proxmox repository under ordinary names: `zfsutils-linux`, `zfs-zed`, `libzfs7linux`…, **`shim-signed` and friends (the
|
||||
Secure Boot loader)**, `ceph-common`/`librados2`…, `corosync`, `chrony`, `frr`, `amd64-microcode`
|
||||
(`partH/H2-proxmox-origin-debian-names.txt`). A name rule (`pve-*`, `proxmox-*`) would have put them in the fast lane.
|
||||
The fast lane is: origin `Debian` or `Debian-Security`, and nothing else. On demo-hp that selection was 108 packages
|
||||
and pulled in **zero** Proxmox packages.
|
||||
|
||||
**[FACT] (C11) What a fast-lane run restarts, and what it does not.** Guest (9202, 49 packages incl. libc6): the
|
||||
packages' own scripts restarted postfix, journald, networkd; **dockerd, containerd, sshd, dbus, logind, cron** kept the
|
||||
old libc; no container stopped (13 samples). Host (demo-hp, 108 packages): dnsmasq, postfix, journald restarted;
|
||||
**systemd (PID 1), `lxc-start`, pveproxy, pvedaemon, pvestatd, pvescheduler, watchdog-mux, sshd, zed, chronyd** kept the
|
||||
old libc; both guests and the agent stayed up. So the fast lane is safe to run unattended, but a libc fix is only
|
||||
fully in force after a reboot (host) or a Docker restart (guest) — which are slow-lane acts. `[PROPOSAL]` the box
|
||||
reports "restart needed" (processes on deleted libraries) and the slow lane's next reboot picks it up.
|
||||
|
||||
### 5.3 The approved list (the "tested versions" record)
|
||||
|
||||
Nobody writes the list by hand. It fills itself:
|
||||
|
||||
1. A ring-0 box updates. Afterwards it reports to the hub the exact `package=version` it installed,
|
||||
per layer (host, guest), with each package's origin (`Debian-Security`, `Debian`, `Proxmox`,
|
||||
`Docker`).
|
||||
2. The box then reports health for the wait period: the agent, the controller, every app's health,
|
||||
and the guest's network.
|
||||
3. When the wait passes with ring 0 healthy, the hub marks that set **approved**. It is one record:
|
||||
an **OS release**, with an id and a date. The hub stores it. The register does not.
|
||||
4. A ring-1 box asks the hub for the newest approved OS release. For each package it has installed,
|
||||
if the approved version is newer, it installs that **exact** version. It never installs a version
|
||||
newer than the approved one.
|
||||
5. Each box reports which OS release it runs, how many updates are waiting, whether it needs a reboot,
|
||||
and which packages it has that **no approved list covers** (see §6, edge case 9).
|
||||
|
||||
~~**OPEN Q1 is the weak point.** The design works only if a box can still download the approved version
|
||||
a week later. Debian's main and security archives keep only the newest version of each package. If
|
||||
Debian publishes a newer fix between approval and install, the approved version is gone. Options:
|
||||
the box waits for the next approval; or the box uses `snapshot.debian.org` (Debian's own dated archive,
|
||||
still signed by Debian); or Felhom runs a package cache. The spike measures how often this happens.~~
|
||||
|
||||
**[FACT] (C2) Q1 measured.** The live Debian archives keep **two** versions: the point-release one in `trixie`
|
||||
(main) and the newest in `trixie-security`; intermediate versions are gone (openssl: installed `u1`, main `u2`,
|
||||
security `u3`). Over 2026-07-04..10-04, 167 trixie security advisories; for the 517 source packages installed on a box,
|
||||
**no package got a second advisory within 2, 7 or 14 days** (the 3 within 2 days were chromium and webkit2gtk, not on a
|
||||
box). `snapshot.debian.org` answers a box: a dated index in **2.3–3.0 s**, a gone exact version
|
||||
(`openssl 3.5.6-1~deb13u1`) downloaded in **2.0 s**, Debian-signed. Proxmox and Docker keep many old versions
|
||||
(pve-manager 66, docker-ce 46). So with a 1–2 day wait the approved version is almost always still live; the rare
|
||||
miss is fetched from the snapshot taken at approval time. `[PROPOSAL]` each OS release records its approval
|
||||
timestamp; a box installs from its own sources, and only for a Debian package that is no longer there, from
|
||||
`snapshot.debian.org/archive/<debian|debian-security>/<timestamp>`. This is the operator decision in STATUS.
|
||||
|
||||
**[FACT] (C9) Approve what ring 0 RUNS, not what it installed.** In the simulation on demo-felhom
|
||||
(`partI/demo-felhom-simulation.txt`), every one of the 108 host and 49 guest approved versions was installable and
|
||||
downloadable — but `curl`, `libcurl*` and `libssh2` were "not covered" in the guest, only because scratch guest 9202
|
||||
ALREADY ran the newer version and so installed nothing. `[PROPOSAL]` step 1 reports the full installed
|
||||
`package=version` set after the run, and approval covers every version ring 0 runs healthy.
|
||||
|
||||
### 5.4 Who runs it, and with what permission
|
||||
|
||||
- **The agent runs every OS update**, for the host and for the guest (`pct exec`). The controller does
|
||||
not, because of constraint 4.
|
||||
- **The agent gets no general `apt` permission.** Like `felhom-selfupdate-guarded` (`03`, the self-update section),
|
||||
a small root-owned wrapper does the work. It accepts only an approved list and refuses everything else:
|
||||
- no package removal;
|
||||
- no downgrade, except the undo of §5.6, which an operator job signs;
|
||||
- no package that the box does not already have, unless the approved list records it as a dependency
|
||||
that the same update pulled in on ring 0 (a kernel update installs a NEW package name each time,
|
||||
for example `proxmox-kernel-6.x.y-z-pve-signed`, pulled by `proxmox-default-kernel`);
|
||||
- no package source other than the ones the installer set up;
|
||||
- non-interactive, and it always keeps the existing config file (`--force-confold`) and reports the
|
||||
conflict.
|
||||
- **Package signatures stay the publishers'** (Debian, Proxmox, Docker). The hub sends only names and
|
||||
versions. A broken-into hub can choose an older version or no version. It cannot make a box install a
|
||||
package that the publisher did not sign.
|
||||
|
||||
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]`
|
||||
|
||||
`felhom-os-apply` — root-owned (`0755 root:root`), installed by the installer beside `felhom-selfupdate-guarded`;
|
||||
the agent's sudoers gets exactly `/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json` and
|
||||
`… --repair-only`. For the guest it runs on the host and enters the guest with `pct exec <vmid> --` itself, so the
|
||||
agent needs no `pct exec … apt` line. Built from what Parts G and H measured.
|
||||
|
||||
**Input — one JSON plan file** (written by the agent, from the hub's approved OS release):
|
||||
|
||||
```json
|
||||
{
|
||||
"release_id": "os-2026-10-04-1", "approved_at": "2026-10-04T08:00:00Z",
|
||||
"layer": "host", // "host" | "guest"
|
||||
"vmid": 9201, // guest only
|
||||
"snapshot": "20261004T080000Z", // the snapshot.debian.org timestamp of approval (C2)
|
||||
"packages": [ {"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"} ],
|
||||
"allow_new": ["proxmox-kernel-7.0.14-20-pve-signed"], // slow lane only, signed operator job
|
||||
"lane": "fast" // "fast" | "slow"
|
||||
}
|
||||
```
|
||||
|
||||
**Order of work:** (1) refuse checks below; (2) **repair first** — `dpkg --configure -a` then `apt-get -f install`,
|
||||
logging what it repaired (C7); (3) `apt-get -s install` of exactly `name=version` for every package that is
|
||||
installed AND older; (4) refuse if the simulation would remove, downgrade, add an unlisted package, or touch a package
|
||||
whose candidate origin is not the plan's; (5) download — from the box's own sources, or for a Debian version no
|
||||
longer there, from `snapshot.debian.org/archive/<archive>/<snapshot>` with a temporary sources list it deletes after
|
||||
(C2); (6) install with `DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=none -o Dpkg::Options::=--force-confold
|
||||
-o Dpkg::Options::=--force-confdef`; (7) `apt-get clean`; (8) report.
|
||||
|
||||
**Refusals** (each exits non-zero with one line `os-apply: REFUSED: <reason>` and changes nothing):
|
||||
|
||||
| # | Refuses when |
|
||||
|---|---|
|
||||
| R1 | the plan file is not under `/var/lib/felhom-agent/os/`, not owned by the agent, or not valid JSON |
|
||||
| R2 | `lane` is `fast` and any package's origin is not `Debian` / `Debian-Security` (C3) |
|
||||
| R3 | `lane` is `slow` and the plan is not carried by a verified signed operator job (R-530's mechanism) |
|
||||
| R4 | the simulation removes any package |
|
||||
| R5 | the simulation downgrades any package (the operator undo is a separate signed op, §5.6) |
|
||||
| R6 | the simulation installs a package that is neither installed nor in `allow_new` |
|
||||
| R7 | a listed version is not downloadable from the sources the installer set up or the named snapshot |
|
||||
| R8 | free space on `/` (or the guest's rootfs) is below 3× the download size, minimum 500 MB (edge case 8) |
|
||||
| R9 | another apt/dpkg holds the lock, or the per-guest lane lock is held (a backup, a restore-test, C10) |
|
||||
| R10 | `layer` is `guest` and the vmid is not the box's own customer guest |
|
||||
| R11 | the plan names a package twice, or a version that is not a Debian version string |
|
||||
|
||||
**Log lines** (to the journal, tag `felhom-os-apply`, and echoed for the agent to forward to the hub):
|
||||
|
||||
```
|
||||
os-apply: START release=<id> layer=<host|guest:vmid> lane=<fast|slow> packages=<n>
|
||||
os-apply: REPAIR configured=<n> fixed=<n> (always printed; 0 0 when nothing was half-done)
|
||||
os-apply: PLAN upgrade=<n> already=<n> not-installed=<n> from-snapshot=<n> download=<bytes>
|
||||
os-apply: REFUSED: <R-number> <reason>
|
||||
os-apply: CONFFILE kept <path> (new version saved as <path>.dpkg-dist)
|
||||
os-apply: DONE rc=0 seconds=<s> upgraded=<n> restarted=<unit,…> restart-needed=<process,…> reboot-needed=<yes|no>
|
||||
os-apply: FAILED rc=<n> step=<download|install> — dpkg state: <dpkg --audit first line>
|
||||
```
|
||||
|
||||
`restart-needed` lists processes still mapping deleted libraries (C11); `reboot-needed` is yes when that list holds
|
||||
PID 1 or `lxc-start`, or a kernel was installed.
|
||||
|
||||
### 5.5 When
|
||||
|
||||
Inside the household's night window, after the backups:
|
||||
|
||||
```
|
||||
W DB dump
|
||||
W+60m Tier 2
|
||||
W+105m off-site → app updates (until W+5h at most)
|
||||
[W+2h, W+6h) whole-guest backup (agent)
|
||||
after it OS updates — guest first, then host
|
||||
```
|
||||
|
||||
The OS leg starts **after the whole-guest backup has finished**, so the guest's newest full copy is
|
||||
minutes old. This is the opposite order to app updates, which run before the whole-guest backup
|
||||
(`07` §6.1, `09` decision 11). The reason: the whole-guest backup is the guest's undo.
|
||||
|
||||
**[FACT] (C10) Q8 measured** (guest UTC; demo-hp W = 02:30): db-dump 02:30, tier-2 03:30, off-site ~04:15,
|
||||
whole-guest gate [04:30, 08:30), controller self-update 04:30, offsite-integrity 06:00. Host: `apt-daily` and
|
||||
`apt-daily-upgrade` run daily but install nothing (no `unattended-upgrades`, no `APT::Periodic`); `pve-daily-update`
|
||||
refreshes the lists daily. **Restore-tests are NOT windowed** — they run on a cadence at any hour (demo-felhom 10:38
|
||||
daily, demo-hp 16:43 and 22:46); agent updates arrive by signed job at any hour. `[PROPOSAL]` the OS leg takes the same
|
||||
per-guest lane lock the restore-test and the whole-guest backup take, rather than a clock slot.
|
||||
|
||||
~~**OPEN Q8:** what else runs then. The controller's self-update (default 04:30, and after any hub report
|
||||
when a floor is above it, R-608), the agent's self-update, the restore-tests, and PBS jobs. The OS leg
|
||||
must never overlap a backup, a restore-test or a self-update.~~
|
||||
|
||||
### 5.6 How a failed update is undone
|
||||
|
||||
"Rollback" is not used (`09` §4). The shapes:
|
||||
|
||||
| Layer | Undo | Limit |
|
||||
|---|---|---|
|
||||
| Guest packages | Restore the guest snapshot taken just before the update | It also undoes app data written after the snapshot. Use it only inside the health window, before apps have written much. After that, install the previous version (needs OPEN Q1). **[FACT] (C6)** Customer guests are on LVM-thin and can snapshot; scratch 9202 (`dir` storage) cannot (`snapshot feature is not available`), so the measured undo was a whole-guest backup + restore: **73 s down**, all apps healthy, libc back. The snapshot rollback itself is unmeasured (R-837). |
|
||||
| Docker engine | Install the previous version (Docker's repository keeps old versions — OPEN Q2) | Every container restarts again. **[FACT] (C5)** Docker keeps 46 `docker-ce` versions. A step (either way) without `live-restore`: 6 of 6 containers restart, the app silent **26.5–30 s**, healthy at +42–45 s. With `live-restore` on: **0 restarts, no gap**, also across a containerd step. But a restart that turns `live-restore` OFF stops every container and starts NONE (`unless-stopped` ignored) — on 9202 they stayed down until restarted by hand; `systemctl reload` turns it on but not off. |
|
||||
| Host packages | Install the previous version | Only if the source still has it (OPEN Q1/Q2). **[FACT]** `rsync` back to its pre-update version: `not found`; `libpng16-16t64` back to the point-release version: worked. A Debian undo needs the snapshot archive (C2). **[FACT, 2026-10-04]** Proved by hand: `runbooks/os-updates-host-undo.md` (demo-hp, `tzdata` back one version from `snapshot.debian.org`, held, released). Use the NEW version's `first_seen` as the timestamp when there is no previous host release; a package that pins its siblings (`eject` → `libmount1 =`) goes back only with them. |
|
||||
| Host kernel | ~~Boot the previous kernel. Proxmox can boot a new kernel **once** (`proxmox-boot-tool kernel pin <ver> --next-boot`). If that boot fails, the next boot uses the old kernel again. Make the new kernel permanent only after a healthy boot.~~ **[FACT] (C4)** Both demo hosts boot UEFI + GRUB (no proxmox-boot-tool ESPs). **Installing a kernel makes it the GRUB default at once.** `--next-boot` on GRUB writes an ordinary `GRUB_DEFAULT` + `update-grub`; `proxmox-boot-cleanup.service` clears it only once a boot reaches userspace. Measured on demo-hp (old kernel pinned permanently FIRST, new pinned for next boot): boot 1 → `7.0.14-20-pve`, healthy, 60 s; boot 2 → back on `7.0.2-6-pve`, healthy, 76 s. **So the fallback works after a boot that succeeds; read from the code, a kernel that hangs before userspace stays the default on every power cycle.** ~~`[PROPOSAL]` use GRUB's own one-shot (`GRUB_DEFAULT=saved` + `grub-reboot`) with the old kernel as the saved default — unmeasured (R-836).~~ **[FACT, 2026-10-04, demo-hp, operator's word before each reboot] GRUB's one-shot is NOT a one-shot here either.** With `GRUB_DEFAULT=saved` (old 7.0.2-6 saved) and `grub-reboot` 7.0.14-20: boot 1 → 7.0.14-20, **Secure Boot ON, booted fine** (signed kernel, shim → GRUB); but `/boot` is ext4 on LVM, GRUB cannot write its environment block there (`grub-reboot` warns so itself), `next_entry` was never cleared, and boot 2 with no command → **7.0.14-20 again**. A kernel lane needs a writable env block (the ESP) or a userspace "boot good" step — R-836. demo-hp left on 7.0.14-20, saved default 7.0.14-20, both kernels installed (`audits/os-host-lane-2026-10-04/partE/`). | ~~If the new kernel hangs, someone must switch the box off and on. The spike checks whether a hardware watchdog can do that (OPEN Q4).~~ **[FACT]** Only `softdog` runs (loaded by `watchdog-mux`); it cannot rescue a kernel that never boots. demo-hp has an AMD FCH whose `sp5100_tco` driver ships but is not loaded; untested. A hang still needs a person. **[FACT, 2026-10-04]** `sp5100_tco` is blacklisted by the Proxmox kernel package; loaded by hand it answers (`SP5100 TCO timer`, 60 s, inactive, nowayout 0 — read from sysfs, never opened, so never armed; unloaded). `kernel.panic = 0`: a panic leaves the host stopped (R-851). |
|
||||
|
||||
### 5.7 Telling people
|
||||
|
||||
- **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release,
|
||||
how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest
|
||||
approved release raises an alarm (`08`).
|
||||
- **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with
|
||||
buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held /
|
||||
reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed,
|
||||
Docker engine / containerd / live-restore / release, the last leg — and above it the releases, what ring 0 runs, "Approve
|
||||
now" and "Approve Docker set". Colours are the alarm thresholds (decision 94). Hosts shows Proxmox / kernel too.
|
||||
- **Household:** one line on the timeline in both languages, informal voice: what was updated and
|
||||
whether the box restarted. Telling households in advance that the box may restart at night is a
|
||||
**promise to users**. That is the operator's decision when the slow lane is built.
|
||||
|
||||
### 5.8 The Docker engine slow lane — BUILT 2026-10-04 (agent v0.142.0, hub v0.132.0) `[FACT]`
|
||||
|
||||
**As built** (evidence `audits/os-docker-crash-2026-10-04/partB/`; decisions 87, 93):
|
||||
- **live-restore ON**: the golden bakes it (`build-golden.sh` 3.1.0, fail-closed assertion); an installed box gets it once
|
||||
by the wrapper's `live-restore-on` (merge into daemon.json + `systemctl reload docker`). Measured: the same container ids
|
||||
after (9202 6/6 by hand — R10 refuses a scratch guest by design —, demo-hp 24/24, demo-felhom 5/5).
|
||||
- **The step** runs in the night leg after a healthy guest and host step, **ring 0 only**, `select pending-docker`,
|
||||
allowed by the wrapper only with the root-owned `ring0_slow_lane` mark. **Ring 1 and every undo** only through a signed
|
||||
`os_docker_step` the wrapper re-verifies itself (decision 93). Measured: 29.7.x → 29.8.2 on both demo boxes, every id
|
||||
kept; a signed undo to 29.7.2 on demo-hp (41 s, ids kept) and back; demo-felhom as ring 1 by a signed job; a replayed
|
||||
job refused by the agent (nonce).
|
||||
- **Approval**: the hub never approves a Docker set automatically; the System page's button works after every ring-0 box
|
||||
ran the set in 2 healthy night Docker steps (`OS_DOCKER_APPROVE_NIGHTS` TEST override, logged). An approval nudges no
|
||||
box. Undo: `runbooks/os-updates-docker-undo.md`.
|
||||
|
||||
**The design as written before the build:**
|
||||
|
||||
Built from C5 (measured) and R-835. **The one decision it needs is in STATUS: `live-restore` on, fleet-wide.**
|
||||
|
||||
- **What moves:** `docker-ce`, `docker-ce-cli`, `containerd.io`, `docker-buildx-plugin`, `docker-compose-plugin`,
|
||||
`docker-ce-rootless-extras` in the customer guest — today the six "not covered" packages on every box. One
|
||||
approved **engine set** at a time, like a controller floor: the operator approves it (slow lane, §5.2), after ring 0
|
||||
has run it for at least 2 nights healthy. Never two steps in one night.
|
||||
- **Precondition: `live-restore` ON.** Without it an engine step restarts every container — 26.5–30 s of silence,
|
||||
healthy at +42–45 s (C5). With it: 0 restarts, no gap, also across a containerd step. Turning it ON is safe
|
||||
(`systemctl reload docker` applies it without a restart, C5); turning it OFF later by a plain restart stops every
|
||||
container and starts none (R-835) — so it is turned on once, by the golden and by a one-time fleet step, and never
|
||||
turned off by the lane.
|
||||
- **Who and when:** the agent, through the same wrapper (`lane: slow`, refusal R3: only inside a verified signed
|
||||
operator job, R-530's mechanism), in the guest, after the guest and host fast-lane steps, under the same heavy-op
|
||||
gate, on a night the operator scheduled. Debian origin rule replaced by "origin `Docker CE`, exactly these names".
|
||||
- **Health:** the guest rule (§8.1) plus `docker version` reports the approved engine, and every container running
|
||||
at the start is running with the SAME container id (proof that `live-restore` held). A changed id is
|
||||
`health_failed` even if the app is healthy — it means the households' apps restarted when they should not have.
|
||||
- **Undo:** install the previous engine set (Docker's repository keeps 46 versions, C2) — by an operator job, with
|
||||
`live-restore` still on, so the undo is also restart-free.
|
||||
- **Not covered here:** the golden's own engine (baked weekly; a new golden carries the approved set), and BYO hosts
|
||||
(the guest is ours on both, so the lane applies there too).
|
||||
|
||||
### 5.9 A crashed host restarts, with a limit — BUILT 2026-10-04 (agent v0.142.0, installer 1.30.0) `[FACT]`
|
||||
|
||||
Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-docker-crash-2026-10-04/partC/`.
|
||||
- **Measured first (demo-hp, the operator's word before each crash):** `kernel.panic = 10` + `echo c >
|
||||
/proc/sysrq-trigger` → the box restarted by itself in 54 s, on the same kernel (the saved default), the agent up 18 s
|
||||
after the boot. `kernel.panic` set by `sysctl -w` is gone after the restart (back to 0) — it must be set at every boot.
|
||||
- **The crash signal** (decision 90): `efi_pstore` is on, yet a real panic saved NOTHING; the journal and `last` show
|
||||
only "no shutdown". The guard uses a **clean-stop marker** (the unit's ExecStop at every orderly shutdown); a boot
|
||||
without it followed a crash, a power cut or a hard reset — counted alike.
|
||||
- **The guard** (`felhom-crash-guard`, early boot unit + hourly re-arm timer): armed → `kernel.panic = 10`; the 2nd
|
||||
unclean boot within 60 minutes TRIPS it (`kernel.panic = 0`), so **the 3rd crash within the hour leaves the box off**
|
||||
(decision 92 — the operator's words); re-arms after 24 h of normal running or `felhom-crash-guard rearm`. A crash before
|
||||
the unit runs (very early boot) leaves the box off: the safe side. `panic_on_oops` stays 0; an oops is reported
|
||||
(decision 91).
|
||||
- **Telling people:** each unclean boot = an operator mail (`host_crash_restart`) + the household's line; the trip = an
|
||||
alarm (`host_crash_guard_tripped`); re-arm and oops announced once (`08` §6.3). The System page shows the guard.
|
||||
- **Measured live:** crash 1 → back in 54 s; crash 2 → back in 53 s, guard tripped; crash 3 → **stayed off** until the
|
||||
operator switched it on; the hub mailed the trip (15 min after the boot — R-853); re-armed by hand.
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks and edge cases
|
||||
|
||||
| # | What can go wrong | What the design does |
|
||||
|---|---|---|
|
||||
| 1 | A new kernel does not boot | Boot it once (`--next-boot`); the old kernel stays the default. A hard hang needs a power cycle (OPEN Q4). |
|
||||
| 2 | Power cut during an update: `dpkg` is half done | The wrapper runs `dpkg --configure -a` and `apt-get -f install` first, every time, and reports what it repaired (spike measures, Q5). **[FACT] (C7)** Killed after 15 unpacks: 5 packages `iU`, 4 triggers pending; the next ordinary `apt-get install` REFUSES (`Unmet dependencies`) — nothing repairs it by itself. The two commands repaired it in 1.4 s + 3.9 s; apps stayed up. |
|
||||
| 3 | An update asks a question (changed config file, service restart prompt) | Non-interactive, keep the old config, report the conflict. |
|
||||
| 4 | A Docker update stops every app, and the controller | Stop the apps cleanly first, like before a backup. Measure whether `live-restore` keeps containers running (Q3). The agent drives it. **[FACT] (C5)** `live-restore` keeps them running (0 restarts). The controller keeps running too; its `docker` calls fail for the seconds dockerd is down (logged errors, no app event). Turning `live-restore` on is a golden + fleet change — and turning it off later must not be a plain restart (R-835). |
|
||||
| 5 | The approved version is no longer downloadable | OPEN Q1. Until it is answered, the box waits for the next approval and reports it. |
|
||||
| 6 | An urgent security hole | The operator approves the same day once ring 0 has run it. |
|
||||
| 7 | A box was off for months | It catches up through the newest approved release. It does not install every release in between. Packages are not like app data; one `apt` step is enough. Kernel and Docker still go one approved step at a time. |
|
||||
| 8 | The system disk is full | Check free space before downloading; clean the package cache after; refuse and report. |
|
||||
| 9 | A customer box has a package that ring 0 does not have (other hardware: firmware, CPU microcode, NIC drivers) | It is never approved, so it is never updated. The box reports it as "not covered", and the hub raises it. Fix: add matching hardware to ring 0, or approve it by hand. **[FACT] (C12)** The two demo hosts differ by 7 packages: `amd64-microcode`, `proxmox-secure-boot-support`, `felhom-bootstrap` (demo-hp) vs `intel-microcode`, `proxmox-first-boot`, `tailscale`, `tailscale-archive-keyring` (demo-felhom); Secure Boot is ON on demo-hp, OFF on demo-felhom. Ring 0 covers both CPU vendors today. |
|
||||
| 10 | The package source is down, or its signing key changes (Docker has done this) | The update fails cleanly and the box reports it. A key change is a slow-lane act for a person. |
|
||||
| 11 | A BYO host | Only the guest and Docker are updated (§1). |
|
||||
| 12 | Two boxes on ring 0 is a small sample | Accepted for now. When there are customers, the first tester boxes can become a second ring. |
|
||||
| 13 | A broken-into hub sends a harmful list | The wrapper refuses removals, downgrades and new packages, and the publishers' signatures still apply (§5.4). |
|
||||
| 14 | `cloudflared` on the host | ~~Not covered by this file yet. How it is installed and updated is OPEN Q9. It faces the internet, so it matters.~~ **[FACT] (C8)** Not on the host: a pinned container in the guest (§1). Four months behind upstream on 2026-10-04 (R-838). |
|
||||
| 15 | The no-subscription Proxmox repository gets less testing than the enterprise one | Ring 0 is our test. The enterprise repository costs a yearly fee per box. That is a **money** decision for the operator, later (Q10). |
|
||||
|
||||
---
|
||||
|
||||
## 7. Open questions the spike must answer
|
||||
|
||||
| Q | Question | How to answer |
|
||||
|---|---|---|
|
||||
| Q1 | Can a box install an exact Debian version one week after approval? How often is it already superseded? | `apt-cache madison` on host and guest for security packages. Debian's security announcement history. Whether `snapshot.debian.org` is reachable and fast enough. |
|
||||
| Q2 | Do the Proxmox and Docker sources keep older versions? | `apt-cache madison` for `pve-manager`, `proxmox-kernel-*`, `docker-ce`, `containerd.io`. |
|
||||
| Q3 | What does a Docker engine update do to running containers, with and without `live-restore`? | On scratch guest 9202: step from one pinned version to the next; time the downtime. |
|
||||
| Q4 | Does `proxmox-boot-tool kernel pin --next-boot` work on both demo hosts' boot setups (UEFI or legacy, GRUB or systemd-boot)? Does either box have a hardware watchdog? | On demo-hp, with the operator's go before each reboot. |
|
||||
| Q5 | What does an interrupted `apt` run leave, and does the repair recover it? | Kill a run on 9202, after a snapshot. |
|
||||
| Q6 | How far behind are the demo boxes today, per layer and per source? How long does catching up take, and what restarts? | `apt-get -s upgrade` (read only) first; then a real run on 9202 and on one demo host. |
|
||||
| Q7 | Which Debian packages restart something big? | `needrestart` in list mode after a run. |
|
||||
| Q8 | What else runs in the night window, and where does the OS leg fit? | Read the timers on host and guest. |
|
||||
| Q9 | How is `cloudflared` installed and updated on the host? | Read the installer and the agent. |
|
||||
| Q10 | What does the Proxmox enterprise repository cost per box per year, and what does it add? | The publisher's price page. Record only. The operator decides later. |
|
||||
|
||||
---
|
||||
|
||||
### 7.1 Answers — the 2026-10-04 spike `[FACT]`
|
||||
|
||||
All evidence: `audits/os-updates-spike-2026-10-04/` (its `README.md` carries every number).
|
||||
|
||||
| Q | Answer in one line | Detail |
|
||||
|---|---|---|
|
||||
| Q1 | Usually yes: Debian keeps 2 versions; no box package was re-fixed within 14 days in 3 months; the snapshot archive serves a gone version in 2 s. | C2 |
|
||||
| Q2 | Yes for Proxmox (30–66 versions) and Docker (18–46); Debian only the point-release version. | C2, §5.6 |
|
||||
| Q3 | Without `live-restore`: every container restarts, ~30 s of silence. With it: none. Switching it off is a trap. | C5 |
|
||||
| Q4 | `--next-boot` falls back only after a boot that succeeds (measured); a hang keeps the new kernel (code). Software watchdog only. | C4 |
|
||||
| Q5 | Not by itself; `dpkg --configure -a` + `apt-get -f install` repair it in ~5 s. | C7 |
|
||||
| Q6 | Hosts 188 pending each, guests 54–59; guest Debian 24 s, host Debian 60 s, kernel 47 s. Three guests, three Docker versions. | audit README |
|
||||
| Q7 | Few restarts by script; libc leaves PID 1, `lxc-start`, Proxmox daemons and dockerd on the old library. Proxmox packages restart their own daemons. | C11 |
|
||||
| Q8 | The backup legs are windowed; restore-tests and agent updates are not; host apt timers are inert. | C10 |
|
||||
| Q9 | Not a host package: a pinned guest container, 4 months behind. | C8 |
|
||||
| Q10 | €120 (Community) to €1,100 (Premium) per CPU socket per year, net; every tier includes the Enterprise Repository. Money — the operator's. | audit README |
|
||||
|
||||
**Sample approved list** built from what ring 0 installed (`partI/sample-approved-list.tsv`: 108 host + 49 guest
|
||||
packages, with origin) and simulated read-only on demo-felhom: **all 157 would install, exact version, downloadable**;
|
||||
not covered: 79 Proxmox + 1 Tailscale on the host (slow lane / not ours), 6 Docker + 4 Debian in the guest (C9).
|
||||
|
||||
## 8. Build order `[PROPOSAL]`
|
||||
|
||||
Each step returns to the operator for go or no-go.
|
||||
|
||||
1. **Spike** (measure Q1–Q10; no product code).
|
||||
2. **Guest Debian, fast lane.** ~~Lowest risk: a snapshot undo exists.~~ **BUILT 2026-10-04** — agent v0.140.0, hub
|
||||
v0.130.0, installer 1.29.0; §8.1. **There is no snapshot undo** (R-837, measured).
|
||||
3. **Host Debian, fast lane** (no kernel, no Proxmox packages). **BUILT 2026-10-04** — agent v0.141.1, hub v0.131.1;
|
||||
§8.2.
|
||||
4. **Fleet view and alarms** (§5.7). **BUILT 2026-10-04** — hub v0.131.0/v0.131.1; §8.3.
|
||||
5. **Slow lane: Docker engine.** **BUILT 2026-10-04** — agent v0.142.0, hub v0.132.0; §5.8.
|
||||
6. **Slow lane: host kernel and Proxmox packages, with the reboot.**
|
||||
7. **Later:** the Proxmox major upgrade (PVE 9 → 10), drilled on ring 0 first.
|
||||
|
||||
---
|
||||
|
||||
### 8.1 Step 2 as BUILT (2026-10-04) `[FACT]`
|
||||
|
||||
Evidence: `audits/os-guest-lane-2026-10-04/` (parts A–G). Brief: guest fast lane, decisions 78–80 (`09` §3).
|
||||
|
||||
- **The undo (R-837, measured first): none automatic.** PVE refuses ANY snapshot of a customer guest —
|
||||
`PVE/AbstractConfig.pm:755-757` skips non-snapshot mounts only for a snapshot named `vzdump`, and every customer
|
||||
guest carries the host-path binds mp8/mp9. As the agent's token and as root: `snapshot feature is not available`.
|
||||
The token's role HAS `VM.Snapshot` / `VM.Snapshot.Rollback`. So §5.6's first row does not exist: a failed health
|
||||
check stops, reports `health_failed`, and the hub mails the operator; the whole-guest backup taken minutes earlier
|
||||
is the undo, by hand. The choice of a real undo is in STATUS (R-842).
|
||||
- **The wrapper** `felhom-os-apply` (agent repo `configs/`), Python 3 stdlib, per §5.4.1 with refusals R1–R13 (R12:
|
||||
the host layer; R13: dpkg still broken after the repair). **Changed from the draft** *(decided by CC unattended —
|
||||
operator may reverse)*: one sudoers entry (`--plan <file>`); the plan's `mode` field (`inventory` / `apply` /
|
||||
`health`) replaces a separate `--repair-only` (the repair runs first on every apply); Python, because a JSON plan
|
||||
cannot be parsed safely in sh. The box's own customer guest is "the guest that binds `/mnt/felhom-drives`" (R10).
|
||||
- **The leg** (agent `internal/osupdate`): after a SUCCESSFUL primary whole-guest backup, inside the backup's
|
||||
goroutine before the host-wide heavy-op gate is released (so never beside a backup or a restore-test, C10), 90 s
|
||||
after the backup, at most once per 20 h. *Decided by CC unattended — operator may reverse:* the 90 s settle and the
|
||||
20 h gap; the controller's own self-update (04:30) is not detected — the 5-minute health wait absorbs a restart.
|
||||
- **The health rule** (`HealthVerdict`, pinned): docker answers, the guest resolves `deb.debian.org`, the controller's
|
||||
health check is `healthy`, and every container running at the START of the leg runs again (healthy if it was). The
|
||||
baseline merges the inventory's reading with the apply's own — found live: an app stopped between them escaped the
|
||||
first rule. Wait 5 min, poll 15 s.
|
||||
- **Rings and the switch** (hub, per box): ring 0 = demo-hp + demo-felhom, everything else ring 1; switch ON by
|
||||
default; OFF → the box reports, installs nothing. A box with no `os_update` block (older hub) = ring 1, ON, no
|
||||
release.
|
||||
- **The approval rule (the ruled "1–2 day wait")**: every Debian / Debian-Security package=version that ALL ring-0
|
||||
boxes having it agree on; approved when, since that set was first seen, **24 h** passed with every ring-0 report
|
||||
healthy and every ring-0 box completed **1** post-backup night run (`OS_APPROVE_AFTER`, `OS_APPROVE_NIGHTS`; an
|
||||
override is logged as a TEST configuration). The approval time is the snapshot.debian.org timestamp (decision 79).
|
||||
"Approve now" is an operator event. *The 24 h / 1 night numbers: decided by CC unattended within the ruled 1–2 days.*
|
||||
- **The household's line**: hub event `os_update_applied` (info: on the household's hub timeline, never mailed;
|
||||
hu/en in the bundle). There is no surface on the box itself (R-844).
|
||||
- **Measured live**: ring 0 — 53 Debian packages on each demo box (18.7–31.7 s inside the wrapper; 174–226 s for the
|
||||
whole leg incl. the inventory), healthy, 6 Docker updates "not covered"; approval — with a 2-minute TEST wait, a
|
||||
272-package release approved automatically, then the ruled values restored; ring 1 — demo-felhom installed exactly
|
||||
the 3 approved versions it lacked (269 already current) and left the newer Docker packages alone; a deliberately
|
||||
stopped app → `health_failed` after 5 min, the operator mailed, the household line recorded.
|
||||
- **Not delivered to existing boxes by the product**: the wrapper and the sudoers line reach a box only through the
|
||||
installer; the signed agent update replaces the binary only (R-840). The demo boxes got them BY HAND.
|
||||
|
||||
### 8.2 Step 3 as BUILT (2026-10-04) `[FACT]`
|
||||
|
||||
Evidence: `audits/os-host-lane-2026-10-04/` (parts A–G).
|
||||
|
||||
- **Where it runs:** appliances only. The proof is the ROOT-owned install record `/var/lib/felhom-install/state.json`
|
||||
`mode: appliance` (written by the installer as root); the agent-writable `agent.json` `deployment_mode` is not
|
||||
trusted for this. A BYO host gets no host step (wrapper refusal **R12**, now lifted only for lane fast / layer host
|
||||
on an appliance). *Decided by CC unattended — operator may reverse.*
|
||||
- **What:** origin `Debian` / `Debian-Security` only, and never a kernel, boot or firmware package (name pattern
|
||||
`HOST_SLOW_RE` — `linux-*`, `proxmox-kernel*`, `pve-kernel*`, `pve-firmware`, `firmware-*`, `grub*`, `shim*`,
|
||||
`systemd-boot*`, `*-microcode`, `efibootmgr`; refusal **R14**). The hub leaves the same names out of the host
|
||||
candidate.
|
||||
- **When:** in the same leg, after the guest step, under the same heavy-op gate. A failed or unhealthy guest step
|
||||
skips the host step.
|
||||
- **The host health rule** (`HostHealthVerdict`, pinned in `internal/osupdate`): `felhom-agent`, `pveproxy`,
|
||||
`pvedaemon`, `pvestatd` and `pve-cluster` are `active`; the customer guest runs; the guest health rule (§8.1)
|
||||
passes; the tunnel is `running` — an `unknown` tunnel does not fail it, `not_running` does. Same 5-minute wait.
|
||||
- **Separate approved sets:** host and guest releases are separate (`os-host-…`, `os-guest-…`), each by the same rule
|
||||
(24 h, 1 night of THAT layer, every ring-0 box). Ring 1 receives `host_release` beside `release`.
|
||||
- **Reboot needed:** reported when PID 1 or `lxc-start` maps a replaced file, with the date of the first scanned
|
||||
report that said so. **Never reboots.** The host is scanned on every pass, so a reboot clears it (v0.141.1 — v0.141.0
|
||||
hid `lxc-start` and never cleared, R-846).
|
||||
- **Undo:** by hand, `runbooks/os-updates-host-undo.md` (proved). No automatic undo.
|
||||
- **Speed (R-845):** one call per layer; both steps with nothing to install 23–32 s; a 108-package host pass 70 s.
|
||||
- **Measured live:** demo-felhom ring 0 installed 108 Debian host packages, all Debian origin (checked against apt),
|
||||
healthy; a 605-package host release approved (TEST wait 2 min / 0 nights, logged, reverted to 24 h + 1 night);
|
||||
demo-felhom as ring 1 installed exactly the one version it lacked (604 already current).
|
||||
|
||||
### 8.3 Step 4 as BUILT (2026-10-04) `[FACT]`
|
||||
|
||||
- **The fleet view** (`GET /os/fleet`, operator): one line per box — ring, switch, the tunnel, and per layer: the
|
||||
release, the last outcome, the last successful leg, pending, not covered, restart needed, reboot needed since, the
|
||||
wrapper's own seconds.
|
||||
- **The four alarms** (`08` §6.3), operator-only, hourly, at most weekly while true: no successful OS leg for
|
||||
**7 days** while the switch is ON (naming the likely reason); reboot needed for **14 days**; ring 0 approved nothing
|
||||
for **7 days** while it has pending fast-lane updates; not-covered fast-lane packages for **14 days**. The four
|
||||
numbers are configuration (`OS_ALARM_*`). *Decided by CC unattended — operator may reverse:* 7 days = a week of
|
||||
missed nights is past any normal hiccup (a box off for a weekend does not alarm); 14 days for reboot and coverage =
|
||||
two weekly golden cycles, both need a person anyway.
|
||||
- **The tunnel** (R-841): `running` / `not_running` / `unknown`; `tunnel_down` after two `not_running` reports.
|
||||
|
||||
## 9. Where the rest lives
|
||||
|
||||
- The finding: **R-812** (`backlog/OPEN-ITEMS.md`). The intention: **R-808** (`backlog/ROADMAP.md`).
|
||||
- Related: **R-604** (a per-customer floor hides a box from global raises; the same risk applies to an
|
||||
OS-release floor), **R-530** (agents update only by a signed job per box).
|
||||
- App updates: `09`. Backups and the night chain: `07` §6.1. The agent's permissions: `03` §3.
|
||||
@@ -0,0 +1,361 @@
|
||||
# OPEN-ITEMS — the narrative sections, as they stood on 2026-10-03
|
||||
|
||||
> **History, not a register.** These sections sat between the register tables of
|
||||
> `documentation/backlog/OPEN-ITEMS.md` until the 2026-10-03 triage. They are moved here word for word
|
||||
> (`git show 9e2786c:documentation/backlog/OPEN-ITEMS.md` holds them in place). Every row they discuss is in
|
||||
> `OPEN-ITEMS.md` (open) or `CLOSED-ITEMS.md` (finished). A status word below is the status ON THE DATE OF
|
||||
> ITS SECTION and may be stale — the registers are the authority. The ranking paragraphs ("Why the TOP
|
||||
> READY rows rank this way", "The 2026-08-02 intake, ranked") are superseded by
|
||||
> `documentation/audits/backlog-triage-2026-10-03/RECOMMENDATION.md`.
|
||||
|
||||
## Operator rulings — 2026-08-04
|
||||
|
||||
Recorded here because a ruling that lives only in a conversation binds nobody (the R-96 standing rule).
|
||||
|
||||
1. **Run the recovery drill, after R-198.** R-198 shipped in hub v0.93.0; **the drill is the next
|
||||
session.** Design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10 — demo-hp, ~3–4 h, a recovery
|
||||
code created and KEPT, a sentinel file, wipe, reinstall, recover, and **pass = a byte-identical
|
||||
sha256, not "the repository opened"**. → R-201
|
||||
2. **Delete the orphaned ciphertext** (~1.2 GB across the two demo boxes, in set-aside restic stores
|
||||
nothing prunes). **STILL OWED — not done in v0.93.0.** It is a destructive act on a protected
|
||||
endpoint and belongs to a session that is scoped for it, not to a release that ships a schema
|
||||
change. → R-193
|
||||
3. **Accept the risk on R-193 candidate (c)** — no repository password retained on the Proxmox host.
|
||||
**This is what makes R-198 load-bearing rather than tidy:** with no host-retained copy, the
|
||||
customer-present recovery path is the ONLY way back from a rebuild, and that path runs entirely
|
||||
through the retained identity blob. → R-193, R-199, R-200, R-201
|
||||
|
||||
~~**Still open and untouched by v0.93.0:** R-199, R-200, R-201.~~ **UPDATED 2026-08-04 (evening), after
|
||||
hub v0.94.0 + agent v0.125.0 + controller v0.195.0:**
|
||||
|
||||
- **R-199 — CLOSED, proven on hardware.** Chain links 6–8 are assembled and walked. The offsite
|
||||
repository password came back out of the sealed bundle **byte-identical** to the one on disk
|
||||
(`c60c8bc737a6…` from three independent sources: the box's file, the recovered bundle, and the hash
|
||||
the hub already stored).
|
||||
- **R-200 — the plumbing half shipped**, the customer-facing form did not, deliberately.
|
||||
- **R-201 — PASSED 2026-08-04 (night run).** A customer's file survived a machine rebuild and came
|
||||
back byte-identical, through the customer's own restore flow. **It passed only because a person was
|
||||
there:** four manual interventions stood between the recovered key and the restored file, none of
|
||||
them in any design document → R-204.
|
||||
- **R-202 — untouched.** The orphan card still promises recoverability unconditionally.
|
||||
- **The orphaned ciphertext deletion (~1.2 GB) is STILL OWED** — ruling 2, above.
|
||||
|
||||
**UPDATED 2026-08-05, after controller v0.198.0 + hub v0.95.0 (R-204):**
|
||||
|
||||
- **R-204 items 1–3 — CLOSED.** The reset code works without a restart; a re-issue no longer marks a
|
||||
healthy escrow stale (→ **R-196 CLOSED**); a unit restore states what it did NOT restore.
|
||||
- **R-204 item 4 — OPEN and unstarted:** a rebuilt box cannot obtain an off-site credential unaided.
|
||||
It needs an operator ruling on the one-shot credential design → **R-193**.
|
||||
- **Still open and untouched by this session, stated so nothing is presumed closed by association:**
|
||||
**R-202** (the orphan card's unconditional promise), **the ~1.2 GB orphaned-ciphertext deletion**
|
||||
(ruling 2 — still owed, still needs its own scoped session), and **R-198's retention, which remains
|
||||
UNIT-PROVEN ONLY** — nothing has superseded a key in production, and proving it needs a SECOND
|
||||
deliberate wipe. That retention drill is the next item, and it is not this session's.
|
||||
|
||||
v0.93.0 made the key survive. v0.94.0/v0.125.0/v0.195.0 make it come back. v0.197.0 got the file into
|
||||
the snapshot and the drill got it out again. **v0.198.0/v0.95.0 remove three of the four crutches the
|
||||
drill needed — the fourth is R-193, and until it goes the recovery is still operator-assisted.**
|
||||
|
||||
## R-201 — THE RE-WALK, 2026-08-06 (attended)
|
||||
|
||||
**The question was asked a second time, on the fixed build, on a brand-new appliance built from the
|
||||
published ISO. The answer is still no — but it is a nearer no.**
|
||||
|
||||
| half | verdict |
|
||||
|---|---|
|
||||
| **the data** | **PASS** — all three sentinels byte-identical, including a 12 MB binary and an accented Hungarian filename **whose NAME BYTES are also byte-identical** (verified as hex, not as rendered text). Restored in **23 s** out of the pre-destruction snapshot `a7bc23bd`, through the customer's own two-step full-restore flow |
|
||||
| **the journey** | **FAIL — two dead ends**, against Phase 1's four. One needed a command line **inside the guest**, one a **Proxmox-host** action |
|
||||
|
||||
**RTO: the unaided figure is STILL UNDEFINED**, because the unaided journey still does not complete.
|
||||
Attended: login 11:42:22 → key placed **+45 s** → tier up **+24 m 12 s** (after intervention 1) →
|
||||
all three sentinels restored and verified **+30 m 13 s** (after intervention 2). **The 30 m figure must
|
||||
not be quoted as the customer number.** The only segment that reflects the product working alone is
|
||||
**23 seconds** to pull 12.8 MB back once everything was in place.
|
||||
|
||||
**The two dead ends:** **R-218's consume half** (row corrected above) and **R-220** (drives
|
||||
unenrollable after a rebuild — reproduced and red-proved again; without it no app can be redeployed,
|
||||
and without a redeployed app the restore page is empty, which is R-213's territory and follows from
|
||||
R-220 rather than being separate).
|
||||
|
||||
**What PASSED and is worth keeping:** the recovery screen **appeared without being sought**
|
||||
(`/` → `/launcher` → `/recovery`); it answered all three of its questions and its seal date matched the
|
||||
hub's `created_at` exactly; the emailed reset code worked **first try**; the unlock took **1.528 s** —
|
||||
a real unseal — and placed the key; and **R-225's fix was seen working in the wild** (the store read
|
||||
„a pillanatképek száma még ismeretlen" rather than a false zero).
|
||||
|
||||
**R-216 part 4 reproduced live:** the reinstall **downgraded the agent 0.126.0 → 0.125.0**, back to the
|
||||
vouched version — an operator's hand-fix undone by the very event that makes recovery necessary.
|
||||
|
||||
⚠ **THE DELIVERY GAP, and it is owed.** A fresh install landed on controller **0.201.0** / agent
|
||||
**0.125.0** — the vouched versions, **neither carrying the fixes**. They were installed **by hand**.
|
||||
Fleet delivery needs a golden carrying 0.202.0 **and** a vouched agent 0.126.0. **Nothing was vouched;
|
||||
that is the operator's act.** **This re-walk proves the JOURNEY on the fixed build; it does NOT prove a
|
||||
customer would receive that build.**
|
||||
|
||||
Evidence: `tests/rewalk-r201-2026-08-06/journal.md`.
|
||||
|
||||
## CAMPAIGN 11 — the recovery journey, 2026-08-05
|
||||
|
||||
**The whole journey was walked end to end for the first time, on a throwaway appliance built from the
|
||||
published ISO. The data came back byte-identical; the journey did not exist.** **Ten findings from
|
||||
Phases 1 and 3, R-214 … R-223** (seven fixed in controller **v0.201.0** + hub **v0.97.0/0.97.1**;
|
||||
**three deliberately still open**, each blocking a real flow), **plus five from Phase 2's injected
|
||||
faults, R-224 … R-228.** Evidence: `tests/campaign11-evidence-2026-08-05/journal.md` (Phases 0/1/3)
|
||||
and `journal-phase24.md` (Phases 2/4). Campaign document:
|
||||
`audits/CAMPAIGN-11-recovery-journey-2026-08-05.md`.
|
||||
|
||||
> **Phase 2's verdict in one line.** The cryptography, the retention and the transport all work and
|
||||
> are now proven live. **What fails is being told the truth:** a mistyped code, a hub outage, a
|
||||
> stopped agent and a correct code for a retained earlier package all produce **one** message, and
|
||||
> three of the four are wrong.
|
||||
|
||||
### Phase 2 — the injected faults, 2026-08-05/06 (unattended)
|
||||
|
||||
> **ALL FIVE CLOSED 2026-08-06** in controller **v0.202.0** + agent **v0.126.0**. The rule they now
|
||||
> enforce, stated so it outlives them: **on the unlock path the customer is blamed only after a real
|
||||
> attempt REFUSED their code; every other outcome, including an unclassifiable one, says something
|
||||
> else.**
|
||||
>
|
||||
> **STILL OPEN AND DELIBERATELY UNTOUCHED BY THAT WORK — said explicitly rather than left to
|
||||
> inference:** **R-214** (the console never stops showing a stale pairing code), **R-220** (a rebuilt
|
||||
> box's drives cannot be re-enrolled — *currently worked around BY HAND on the campaign venue*, which
|
||||
> is the only reason an app could be deployed there at all), **R-221** (a rebuilt box cannot run the
|
||||
> escrow ceremony), **R-213** (putting files back), **R-202** (the orphan card's unconditional
|
||||
> promise — now the **last** place on that surface still promising recoverability, two doors from
|
||||
> where R-228 removed the same promise).
|
||||
|
||||
Eleven faults, each judged on **the message, not the outcome**, each with a positive control proving
|
||||
the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/journal-phase24.md`.
|
||||
|
||||
## Instruction files — deferred half, 2026-08-06
|
||||
|
||||
**Recorded against existing rows by Phase 2:**
|
||||
|
||||
- **R-216 — §4.1 is now MEASURED, not deduced.** The previous session could only offer two absences.
|
||||
The box's own `/settings` renders „Minimális verzió (üzemeltető) **0.200.0**" (`GetFloor()`, whose
|
||||
only writer is the report-ACK handler; **both hold branches serve `Floor=""`**, pinned by
|
||||
`managed_floor_test.go:94`), and a **cold-started** controller logs
|
||||
`settle-gate: GO — at/above floor 0.200.0 (we are 0.201.0)` against the same line reading
|
||||
`floor still unknown after 1m30s` while the hold was in force. The hub's HELD lines ran every
|
||||
15 min to 22:12:06 and stopped, with a liveness control proving the hub kept logging. **The floor
|
||||
is served.**
|
||||
- **⚠ A correction to how that positive was to be taken.** `SetFloor`'s line is `u.dbg(...)`, gated on
|
||||
`cfg.Logging.Level == "debug"` and written to the **logger** — it can **never** reach the logx debug
|
||||
ring, so it cannot appear in `/api/debug/logs` at any level. A controller restart alone would not
|
||||
have produced it. Confirmed with a level census on the ring first (1196 DEBUG / 2802 INFO / 2 WARN),
|
||||
so the absence was known to be structural rather than evidential.
|
||||
- **R-218 — the live half is STILL NOT MEASURED, deliberately.** The fix is present and correct
|
||||
(`needsOffsiteCredential` now retires on the **target**, not the key), but the venue **has** a
|
||||
target, so the box correctly does not declare; declaring here would be the bug. The state that
|
||||
exercises it is shape (a), which the venue no longer holds. **Recorded as not measured rather than
|
||||
inferred from the unit test.**
|
||||
- **R-217 — its fix HELD under exactly its fault** (F5): with the store blocked after a successful
|
||||
unlock, the page rendered M3 and **no listing block at all**; the three false-claim strings are
|
||||
absent, verified in UTF-8 with accented positive controls present.
|
||||
- **R-215 — its fix is present** and the `GET /recovery` gate consults the same predicate as the POST
|
||||
sibling.
|
||||
- **R-199's back-pointer in `architecture/00-capability-map.md` was already added** — the brief lists
|
||||
it as owed; it is present on the escrow-recovery row, explicitly labelled as the omitted
|
||||
back-pointer. **No action taken; the brief's assumption was stale.**
|
||||
|
||||
**Untouched by this session, stated so nothing is presumed closed by association:** **R-213** (putting
|
||||
files back — the half the recovery screen deliberately does not do) and **R-202** (the orphan card's
|
||||
unconditional promise, which CAMPAIGN-11 §7 step 7 measured the customer-facing cost of).
|
||||
|
||||
## CAMPAIGN 12 — the class sweep, 2026-08-08 (unattended)
|
||||
|
||||
Eight rows, **grouped by class so the classes are visible as classes**. Full method, controls, blind
|
||||
spots and the Part-4 gating ranking: `audits/CAMPAIGN-12-class-sweep-2026-08-08.md`. Gating candidates
|
||||
are in `ROADMAP.md`, not here. **C1 produced no new instance** and has no row, deliberately.
|
||||
|
||||
**⚠ A correction the campaign owed to its own brief:** the task described C5's `escrow_stale` instance
|
||||
as "closed individually". **It is not — it is R-247, `READY`.** The live repo is the source.
|
||||
|
||||
## CAMPAIGN 12 follow-through — G-1 built, R-260 and R-247 closed, 2026-08-08
|
||||
|
||||
**The gate was built BEFORE the fixes and was seen failing on 40 fields**, captured verbatim in
|
||||
`documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md`. That order was the method, not
|
||||
bureaucracy: the night before, `deadcode` was rejected for class C6 precisely because it was made to
|
||||
prove itself first and found neither of the two defects it was meant for.
|
||||
|
||||
**⚠ A COUNT THIS SESSION'S OWN PROMPT GOT WRONG, corrected against the repo rather than quoted.** The
|
||||
prompt said *"465 emitted tags, eight unreachable"*. R-260's wording was "at least eight
|
||||
DECISION-BEARING facts", never eight tags in total, and its own census already listed more. Measured
|
||||
on the three declared wires: **210 tags checked, 51 skipped, 40 convicted.** Two prompt claims were
|
||||
wrong this week and both were caught the same way.
|
||||
|
||||
**Two things the gate's CONTROL caught before it was trusted**, each a defect in the instrument:
|
||||
|
||||
1. **A substring false negative.** `grep -F healed_at` also matches `privsep_healed_at`, so a
|
||||
genuinely dropped field read as received. R-260 named `healed_at`, so its absence from the output
|
||||
was the tell. Now a whole-token regex.
|
||||
2. **`dr_recipe` is not wholly opaque.** The hub stores each half as `json.RawMessage` and re-emits
|
||||
nested shapes verbatim — but the TOP-LEVEL section keys are decoded by `hostHalfShape` /
|
||||
`appHalfShape`, and **those are allow-lists**: a section an emitter adds is silently dropped until
|
||||
named in both. That already cost `offsite_restic` (R-122). The gate is therefore opaque BELOW
|
||||
depth 1, so the sections are checked; treating the whole subtree as opaque would have put R-122's
|
||||
shape back outside its reach.
|
||||
|
||||
**THE FORTY, BY DISPOSITION.** Full per-field reasons live in the gate's own `ALLOWLIST`, where each
|
||||
entry is a claim someone can re-check.
|
||||
|
||||
| # | field(s) | direction | decision | what changed |
|
||||
|---|---|---|---|---|
|
||||
| 1 | `oob.operator_key_configured` | agent → hub | **RECEIVE AND ACT ON IT** | decoded as a pointer; `oobDegraded` now fails when the key is absent, and the alert NAMES it. Hub v0.99.0 |
|
||||
| 2 | `oob.wg_handshake_age_s`, `oob.healed_at` | agent → hub | **RECEIVE, for the message only** | decoded into `HostOOBRow` and put in the event payload; deliberately NOT in the predicate — widening the check beyond the fact that is now arriving is how a check stops being read |
|
||||
| 3 | `escrow_stale` | hub → controller | **RECEIVE AND ACT ON IT** | `report.EscrowStatus.Stale`; the box tells a withheld hash from a hash-less one. Controller v0.209.0. **This is R-247** |
|
||||
| 4 | `host.{cpu_temp_c,loadavg,memory_total_bytes,memory_used_bytes,uptime_seconds}`, `system.{load_avg_1,5,15, memory_total_mb, memory_used_mb, temperature_celsius, uptime_seconds}` | agent/controller → hub | **NO CONSUMER WANTED — redundant** | allowlisted: the hub decodes `cpu_percent` / `memory_percent` / `disk_percent` from the same stanzas and every threshold is expressed on those |
|
||||
| 5 | `guests.spec.{disk_bytes,memory_bytes}` | agent → hub | **NO CONSUMER WANTED — redundant** | allowlisted: guest sizing is hub-owned INTENT (the manifest), not mirrored reality |
|
||||
| 6 | `storage_targets.smart.model_name` | agent → hub | **NO CONSUMER WANTED — redundant** | allowlisted: a display label with no threshold on it; `smart.health` and every counter the hub bands on ARE decoded |
|
||||
| 7 | `wireguard.last_handshake_age_s` | agent → hub | **NO CONSUMER WANTED — redundant** | allowlisted: wgsync reconciles from its own state, and the OOB path's own handshake age is now decoded |
|
||||
| 8 | `guest_net` + its 7 children, `selfupdate_pending(_version)`, `mgmt_plane.healed_recently`, `pbs_dr.applied_at`, `restore_tests.mount_{parity,inventory}`, `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check` | agent/controller → hub | **NO CONSUMER TODAY, AND ONE IS ARGUABLY OWED** | allowlisted **against R-264, which stays OPEN**. Allowlisting is not deciding, and the entries say so |
|
||||
|
||||
**A C7 instance found while doing it, and corrected.** `HostReport.SelfUpdatePending`'s own comment
|
||||
claimed *"The hub reads an absent field as pending=false, the correct default."* The hub has no field
|
||||
for it and reads nothing either way. Comment corrected in the agent (no version bump — comment only).
|
||||
|
||||
**The hub's OOB test fixture was part of why this survived.** `oobReport()` omitted
|
||||
`operator_key_configured` entirely, so every pre-existing scenario ran against a report shape **no
|
||||
released agent produces**. Fixed, and the new tests drive the raw JSON decode boundary — a test that
|
||||
builds the receiving struct by hand cannot see a field that never decodes, which is the whole class.
|
||||
|
||||
**Explicitly still open, untouched by this session:** R-246 (the wrong stale flag on `demo-hp` —
|
||||
clearing it is an operator act hub-side), R-255, R-256, R-257, R-258, R-259, R-261, R-262, R-263, and
|
||||
**C7's test-comment half**, which Campaign 12 recorded as *owed, not done* (60 of 2652 production
|
||||
invariant comments sampled; none of the 1440 test comments).
|
||||
|
||||
## The seed that never ran twice, and three pictures that were not true — 2026-08-08
|
||||
|
||||
Four defects of one family: something the box already knows, either thrown away or drawn as its
|
||||
opposite. Agent **v0.128.0**, controller **v0.210.0**, `gates.yml` (no hub change, no hub bump).
|
||||
|
||||
**R-221's writer, ESTABLISHED at `file:line` rather than assumed** — the prompt asked for this and it
|
||||
was owed. `step_agent_config` (`felhom.eu/scripts/felhom-host-install.sh:2396`) renders `agent.json`
|
||||
from `base = {}` unless an explicit `--preserve-from` is passed (flag `:1246`, defaulting empty at
|
||||
`:256`), and writes it with `O_TRUNC` (`:2579`). **The render never writes an `escrow` section at
|
||||
all** — grep over the whole heredoc returns zero hits. The pbsdr marker lives host-side
|
||||
(`<agent-state>/pbsdr/marker.json`) and survives. So a rebuild keeps the marker and takes the key:
|
||||
same descriptor, same hash, early return, seed never re-runs. **The attribution in R-221 was
|
||||
correct.** A rebuild is nonetheless only the case that was measured — the same hole opens for a
|
||||
hand-edited or restored config, which is the honest reason the fix is at the seam and not in the
|
||||
installer.
|
||||
|
||||
**The idempotent early return was KEPT**, and that is load-bearing: it stops a converged box
|
||||
re-running Proxmox operations every 60 s.
|
||||
`TestSeedReasserted_OnConvergedTick_WithZeroProxmoxCalls` asserts **zero** recorded runner calls on
|
||||
that tick, so a "fix" that simply deleted the return fails the test. Verified by mutation.
|
||||
|
||||
**The §7.3 truth table as implemented** (R-258):
|
||||
|
||||
| this app's own most recent dump result | restore point | verdict |
|
||||
| any of its databases failed | yes | `error` — cross |
|
||||
| all clean | yes | `ok` — tick |
|
||||
| none recorded (no database / no run yet) | yes | **no icon**, time only, title „Erről a mentésről nincs eredményünk." |
|
||||
| any | no | no tier-1 row at all, unchanged |
|
||||
|
||||
**An existing test was asserting the defect and was corrected, not deleted.**
|
||||
`TestBuildAppBackupRows_Tier1FromRestorePoints` expected `"ok"` for a `FullBackupStatus` with **no
|
||||
`LastDBDump` at all** — a green tick derived from nothing but a file's existence, i.e. Scenario G.
|
||||
Its real subject, the `Tier1LastRun` time, is unchanged.
|
||||
|
||||
**The convention is now ruled** (§7.2, `CONTEXT.md` S-39): a `…Known bool` companion beside the
|
||||
figures. `ROADMAP.md` G-3 was blocked on that decision and is unblocked.
|
||||
|
||||
**Six red-proofs, every one demonstrated failing and restored, each with the mutation asserted
|
||||
applied.** The one that matters: Scenario A **fails against today's tree** with the intended message
|
||||
— so the test tests the defect.
|
||||
|
||||
**Explicitly still open, untouched by this session:** R-246, R-255, R-256, R-257, R-261, R-262,
|
||||
R-263, **R-264** (the twenty-one undecided facts — a design session of its own), R-240, R-243,
|
||||
R-202, R-213, R-244, R-214/R-235, and **C7's test-comment half**, which Campaign 12 recorded as
|
||||
*owed, not done*. **G-8's other half** (a hub-side check that notices a *vouch* has been forgotten)
|
||||
was deliberately not built: it is hub work whose payoff is a daily email, and this session already
|
||||
ends with a bake-and-vouch cycle in front of the operator.
|
||||
|
||||
## Why the TOP READY rows rank this way
|
||||
|
||||
This covers the next few only — it is deliberately **not** a full ordering of the table above, so that
|
||||
there is one ranking to maintain rather than two.
|
||||
|
||||
1. **R-95** — the largest *data* exposure: the tier holding the customer's documents and photos is the
|
||||
one whose credential can delete. **THIS PARAGRAPH WAS STALE UNTIL 2026-09-01 AND ITS OLD TEXT IS
|
||||
NAMED SO THE CORRECTION IS NOT MISTAKEN FOR A RE-RANK.** It said the snapshot mitigation was
|
||||
*"armed (daily 00:00, keep 7), but it has taken zero snapshots so far"*. **Both halves were wrong:**
|
||||
seven daily snapshots do exist (R-429), and the word "armed" was withdrawn by the spike that same
|
||||
day. **What is true now:** the snapshots exist and the box cannot write into them, but **no account
|
||||
we hold can read anything out of one** (R-433, measured — 777,600 names, zero hits, controlled), so
|
||||
they do not yet bound this exposure. The root cause is untouched either way — the box can still
|
||||
`forget --prune` its own repo, from two call sites. **The ORDER of this list is unchanged and is
|
||||
Viktor's**; only the facts under item 1 were corrected.
|
||||
2. **R-94** — **de-ranked 2026-07-29.** The prior rationale ("until it moves every hub-driven
|
||||
install gets the pre-R-82 default") was false: the constant selects no script and every install
|
||||
already fetches 1.22.0. What remains is a wrong label plus two pieces of dead safety equipment —
|
||||
a drift gate nobody runs and a test that compares a constant to itself. Cheap and worth doing;
|
||||
not high-consequence, and it blocks nothing.
|
||||
3. ~~**R-86**~~ — **CLOSED 2026-08-03**, agent v0.121.0 + hub v0.91.0, proven live on demo-felhom.
|
||||
4. **R-87** — **SPIKED 2026-08-31 and now a DECISION, not work.** It also spent 2026-08-22..31 in
|
||||
`CLOSED-ITEMS.md` by mistake while this paragraph ranked it fourth and pointed at nothing
|
||||
(R-405). The spike measured it rather than designing it: a scratch restore of all 8 apps costs
|
||||
25 s against the 40.3 s weekly check, but it would have caught ONE of the five drill-found
|
||||
restore defects. **Recommendation: build the NARROW version — prove the snapshot still
|
||||
CONTAINS a recoverable unit — or close the row.** Viktor's call; see
|
||||
`audits/SPIKE-restic-restore-test-2026-08-31.md`. *The 2026-08-03 rationale, kept:*
|
||||
R-86 built most of what it was waiting for (per-archive due-ness, a
|
||||
proof that names its archive, and a staleness window that learns a tier's rhythm). What is left is
|
||||
restic-specific — there is no scratch-guest analogue — so it still needs its own design, but it is
|
||||
no longer waiting on a scheduling model that did not exist.
|
||||
5. ~~**R-185**~~ — **CLOSED 2026-08-03**, agent v0.123.0 + installer 1.24.0, proven live on both demo
|
||||
boxes. The silence was fixed as well as the grant: the box now asks whether it may READ each tier
|
||||
it depends on, because an empty listing cannot distinguish forbidden from newborn.
|
||||
6. ~~**R-189**~~ — **CLOSED 2026-08-03** with **R-188** and **R-186**, agent v0.122.0. The three
|
||||
reporting/release signals that misreported their own work are fixed; **R-185 is the one that
|
||||
remains open from that group** and is untouched by this — it is a missing storage ACL on
|
||||
demo-felhom, not a reporting defect.
|
||||
7. **R-110** — last **because it is not a READY row**: the ruling is the operator's, not CC's, and
|
||||
there is nothing for CC to build until it lands. Ranked here rather than omitted because it is the
|
||||
only item on this page about the *publish channel* of the most privileged artifact Felhom ships,
|
||||
and today's exposure is zero — which makes now the cheapest moment it will ever be to decide.
|
||||
|
||||
### The 2026-08-02 intake (R-156 … R-164), ranked
|
||||
|
||||
Filed in one pass from Campaign 10, its two spikes, and the 53-template catalog persistence sweep.
|
||||
**R-156 and R-157 had lived only in audit documents** — the identical *"minted in a spike doc and
|
||||
never carried across"* failure the register already records for R-153/R-154/R-155, caught by the
|
||||
sweep's own §8.0 while it was happening. R-158 was minted by a second session on the same day for an
|
||||
unrelated finding, which is why the sweep's proposals were renumbered to R-159…R-162 at filing time.
|
||||
|
||||
1. **R-157** — highest: a `deployed: true` app can stay **down indefinitely** after a power cut or
|
||||
hard reset, and in mechanism **B** nothing reports it on any channel (`0 currently down`). It is the
|
||||
only row here where the customer loses service and has no signal at all.
|
||||
2. **R-156** — the class is now detectable and two of three apps are fixed; what remains is **papra's
|
||||
referral**, one app, well understood. *(Promoted 2026-08-02: R-161 was ranked here because nothing
|
||||
ran the gate; it now has a mandated entry point, so R-156's residue is the larger remaining item.)*
|
||||
4. **R-163** — a real ceiling that silently caps local backup once an app outgrows `mp1`, and it gates
|
||||
Tier-2 and Tier-3 as well. Ranked below the above only because **overflow itself is safe today** —
|
||||
it refuses per app and preserves the last good unit byte-identical. **RE-FRAMED 2026-08-02:** no
|
||||
longer waiting on a ratio — decision **D-a** merges `mp1` away, so the row is now the record of the
|
||||
constraint and the work moves to **R-165** (with **R-167** shipping in the same step). R-165 inherits
|
||||
this rank; it is the highest-ranked item that must land **before any external install**.
|
||||
5. **R-158** — the gap that makes R-163 dangerous: cross the size line and **one page** tells you. On
|
||||
its own it is a notification gap, not a silent failure, which is why it sits here and not higher.
|
||||
6. **R-164** — blocked on a predicate, no customer impact today; it only becomes urgent if the unit
|
||||
size in R-163 is judged unacceptable, since the tar-drop is the cheapest way to halve it.
|
||||
7. **R-161** — **de-ranked 2026-08-02, ruled and shipped at reduced scope.** The gate now has one
|
||||
mandated entry point (`catalog_gates.py`), which is the shape that actually gets run here. What is
|
||||
left is the automatic half, and that is sufficient while **one** person touches templates — so it
|
||||
ranks low by design, not by neglect. Revisit when a second does.
|
||||
8. **R-162** — `WATCHING` only. A limitation that fails closed; revisit if a non-overlay driver ships.
|
||||
|
||||
**R-159 and R-160 are SHIPPED** and are not ranked; they are filed to record the class, and R-159's
|
||||
class (an image `VOLUME` at an unmounted path) is still live — `immich-server` has one today.
|
||||
|
||||
<!-- RESTORED 2026-08-22: these rows are NOT fully closed and were moved to CLOSED-ITEMS.md in
|
||||
error by this session's compressor, whose status regex matched the word CLOSED inside
|
||||
'PARTLY CLOSED' and the word FIXED inside 'OPEN - NOT FIXED'. They are restored VERBATIM
|
||||
from commit fddfe00ce268, not from the compressed form: an open row keeps its detail. -->
|
||||
|
||||
<!-- ── MIGRATED FROM ROADMAP.md, 2026-08-22 (R-369 ruling: ONE register) ──────────────────
|
||||
16 rows moved: every ROADMAP row that asserted something checkable about the shipped
|
||||
product, plus one owed operator decision. Ideas and proposals stayed in ROADMAP, which
|
||||
is their home. Each row below keeps its ORIGINAL identifier and filing date — the age is
|
||||
the point. The roadmap keeps its copy as history, marked moved, with a pointer here. -->
|
||||
@@ -0,0 +1,111 @@
|
||||
# DRILL — an English speaker's first hour on 0.258.0 (2026-09-20)
|
||||
|
||||
**Interventions a volunteer could not have made: 1** (I1 — reaching the dashboard at all, R-494,
|
||||
filed 2026-09-14, long before this walk).
|
||||
**Felhom-owned Hungarian things the stranger met: 4 kinds** — the claim page's messages, the setup
|
||||
code itself, the Backup page's two protection warnings and its two target names, and the menu word
|
||||
"Debug" (which is correct English for this reader).
|
||||
|
||||
**Ready for an English-speaking tester: NOT YET — because the one screen that stands between them
|
||||
and their box answers them in Hungarian.** Everything else held: the mails, the console, the bind
|
||||
page, the dashboard, the apps, the catalog, the language switch. The blocker is R-596, and it is a
|
||||
day's work, not a redesign.
|
||||
|
||||
Evidence: `i18n-slice6-2026-09-20/drill/` — `journal.md` (every observable, in order),
|
||||
`screens/` (22 console and installer captures), `pages/` (the dashboard as the household saw it),
|
||||
`box-logs/`. Golden: `../tests/golden-0.258.0-2026-09-20/`. R-516: `../audits/i18n-slice6-2026-09-20/R-516-item-by-item.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1. What held, in one table
|
||||
|
||||
| # | step | result | time |
|
||||
|---|---|---|---|
|
||||
| 0.1 | golden **0.258.0** baked, published, vouched — one run, no aborted attempts | **PASS** | bake ≈8 min |
|
||||
| 0.2 | customer created with `language=en`; bind mail auto-sent on creation | **PASS** | — |
|
||||
| 1 | download page `felhom.eu/en/download` | **PASS** — 200, **0 Hungarian**, names file + checksum; the ISO the VM booted hashes to the published checksum **exactly** | 0.12 s |
|
||||
| 2 | install — the **GRAPHICAL** installer, which 2026-09-14 did NOT exercise | **PASS**, with **F1** (the guide told an English reader to keep a Hungarian keyboard) and **F2** (it describes a reboot prompt the defaults never show) | install → console banner ≈11 min |
|
||||
| 3 | first screen, pairing | **PASS** — bilingual banner, code `P3G-JGW` readable in both halves; `/etc/issue` in English | power-on → "waiting to be paired" ≈11 min |
|
||||
| 4 | **self-bind through the mailed link** | **PASS — and this is what 2026-09-14 could not do at all** (its H1). Page, refusal and success all English, 0 Hungarian lines | bind at 16:54:42Z |
|
||||
| A2 | typo in the pairing code | **PASS** — *"Those details are not right…"*, in English, does not say which half was wrong; hub logged `attempt 1/5` | — |
|
||||
| 5 | setup code → claim | **PASS**, with **F3** (the page's messages are Hungarian) and **F4** (the code is three accented Hungarian words) | power-on → claimed ≈23 min |
|
||||
| 4b | the dashboard's **first** language | **PASS** — English from `customer.language` alone, nobody touched a switch | — |
|
||||
| 6 | the two apps the guide names | **PASS** — installed with the default subdomains; API answered in English; both app pages **0 Hungarian** | PrivateBin ≈40 s, BookStack ≈50 s |
|
||||
| 7 | the Backup page as a first-timer | **F5** — 4 Hungarian strings in 73 lines, and two of them are the protection warnings | — |
|
||||
| 9 | version labels | **PASS** — "Up to date" in English (R-589, fixed this morning, proven on a fresh box) | — |
|
||||
| 13 | the language switch, both ways | **PASS** — the globe moves the household; a visitor's switch on the sign-in page sets only their own cookie | — |
|
||||
| 14 | the mailbox | **PASS** — **all three customer mails in English**; the operator's copy Hungarian by design; `[+household(en)]` in the box's own log | — |
|
||||
| — | **R-214** (console never stops asking to be paired) | **CLOSED — it no longer reproduces.** The last paint is the bilingual "the box is linked" banner | — |
|
||||
|
||||
**Not walked, and saying so rather than implying coverage:** steps 8 (Back up now), 10 (remove with
|
||||
"delete my data too"), 11 (restore), 12 (the launcher a week later) and A1 (power cut). All five were
|
||||
proven on 2026-09-14 and none is language-bearing except through pages this walk already read in both
|
||||
languages. **Step 7's recovery-code section could not be walked at all** — the off-site tier needs
|
||||
ep0, which this task fenced, exactly as on 2026-09-14.
|
||||
|
||||
## 2. The intervention
|
||||
|
||||
**I1 — the dashboard has no reachable address (R-494).** The setup mail names
|
||||
`https://felhom.drill-en-0920.felhom.eu`; `getent hosts` finds nothing, because no Cloudflare tunnel
|
||||
was created and the token is an optional field on the customer page. The walk reached the dashboard
|
||||
by breaking glass into the box (the hub-vaulted `host_recovery` credential) and driving the
|
||||
controller from inside the guest. **R-494 was filed on 2026-09-14 and narrowed to P3 by an operator
|
||||
ruling, so it was filed long before it was performed.** It was performed once and covered every later
|
||||
dashboard and app request. **The stop rule (four) was not reached.**
|
||||
|
||||
## 3. Harness substitutions — not interventions, and why
|
||||
|
||||
| | what | consequence for the claim |
|
||||
|---|---|---|
|
||||
| **H1 (GONE)** | 2026-09-14 had **no mailbox** and could not exercise the mailed link or the self-bind page. **This walk had one** (the `@felhom.eu` catch-all), so the bind mail, the setup-code mail and the confirmation mail were all read as the customer, and the self-bind page was driven end to end. | the biggest gap in the previous walk is closed |
|
||||
| **H2 (NOT a substitution this time)** | The keyboard layout was set to U.S. English. On 2026-09-14 that was a harness need (`qm sendkey` speaks US scancodes). **Here it is what the persona would genuinely choose** — and the fact that the guide said otherwise is F1. | — |
|
||||
| H3 (GONE) | auto-reboot was left **ticked**, so the install rebooted itself — the path 2026-09-14 avoided. | the "stick still in" reboot is now exercised |
|
||||
| H4 (GONE) | the **graphical** installer was used (it is the default), not the Terminal UI. | the graphical path is now exercised |
|
||||
| H5 | no browser: every page was fetched through the endpoint it renders from, inside the guest. | script-rendered state was not observed |
|
||||
|
||||
**Harness slips, recorded:** the owner passphrase was printed to stdout once by a probe, against the
|
||||
project's own file→file rule — it belongs to a throwaway customer and the teardown invalidates it;
|
||||
every later use read a 0600 file. The first claim POST used the wrong field names (`password` instead
|
||||
of `new_password`) and was answered 200 — a refusal, not a success (the "HTTP 200 can be a refusal"
|
||||
trap, met and caught). Both are in `journal.md`.
|
||||
|
||||
## 4. Findings
|
||||
|
||||
| row | rank | one line |
|
||||
|---|---|---|
|
||||
| **R-596** | **P1** | The claim page — the first screen they touch — is English chrome with **16 Hungarian messages**, two of which the guide quotes in English |
|
||||
| **R-597** | P2 | The setup code is three Hungarian words with five accented characters, inside a fully English mail |
|
||||
| **R-598** | P2 | The Backup page's two **protection warnings** and its two target names are Hungarian on an English dashboard |
|
||||
| **R-599** | P3 | A drill's teardown is blocked for 30 minutes by design (report staleness), and the 409 does not say so |
|
||||
|
||||
**Classification of every Hungarian thing met:** (F) R-596, R-597, R-598 · (A) the apps' own English
|
||||
UIs, which for this reader are an advantage · (K) "Debug" in the menu (R-516 item 1, correct English
|
||||
here), the operator-tier event copies (Hungarian by design), the 18 counted formal „ön" forms
|
||||
(R-516, unchanged by rule).
|
||||
|
||||
## 5. Scope and what this walk does NOT claim
|
||||
|
||||
- **Not the recovery-code journey.** Off-site needs ep0, which was fenced. Guide §7 — its longest and
|
||||
most emphatic section — is unproven by this walk, exactly as on 2026-09-14.
|
||||
- **Not a backup/restore/remove/power-cut walk.** See the "not walked" note above.
|
||||
- **Not a Hungarian walk**, which is why **R-516 does not close** — see `R-516-item-by-item.md`.
|
||||
- **Not a browser.** Endpoint-level throughout, stated at every step.
|
||||
|
||||
## 6. Teardown — three layers
|
||||
|
||||
- **Machine:** VM 9301 stopped and `qm destroy --purge`d; `/mnt/hdd_1/images/` holds only 9202's disk.
|
||||
- **Hub:** customer `drill-en-0920` deleted through the guarded path (`confirm_id` + three
|
||||
acknowledgements + `expect_hosts`), which cascades the host, the claim, the DR recipe and 17
|
||||
residue rows. **The hub refused until the host went stale — 45 minutes after its last report
|
||||
(`alerting.stale_threshold` in the deployed manifest, not the 30m literal in the checker's source;
|
||||
R-599).**
|
||||
- **ep0:** the WireGuard peer `10.77.0.5` the enrolment registered. **Checked on ep0 itself, not
|
||||
inferred from the hub** — and it was still there 3 minutes after the cascade logged *full
|
||||
teardown*. Watched until it went: **gone by 17:47:46Z**, about 6 minutes after the delete
|
||||
(`wg show wg0 peers` 5 → 4). The mechanism works; the log line is premature about this layer.
|
||||
Filed as **R-600**. The 2026-09-14 findings say the host delete removes it; measured, the host
|
||||
delete removes the hub's RECORD and `wgsync` removes the peer on its next push.
|
||||
|
||||
**Untouched, and checked:** demo-hp guests **9201** and **9202** (running throughout, 24 containers
|
||||
before and after), demo-felhom, DooPlex beyond the golden bake's own VM (reverted to `virgin`),
|
||||
Peti's box, and ep0 beyond the product's own peer.
|
||||
@@ -5,7 +5,7 @@
|
||||
Source findings: `DRILL-day0-cleanroom-2026-07-03.md` §9 **B5** (golden bakes a pre-floor controller →
|
||||
mandatory manual D.1b on every fresh install) and **B1** (controller-bootstrap only fires at boot; the
|
||||
hot-plugged config mount needed the installer's reboot crutch). Also closes the stale backlog note
|
||||
`documentation/backlog/FOLLOWUP-golden-default-controller-tag.md`.
|
||||
`documentation/archive/FOLLOWUP-golden-default-controller-tag.md` (moved from `backlog/` 2026-10-03).
|
||||
|
||||
**Verdict (short):** both findings are FIXED in the product. A fresh Day-0 install now lands controller
|
||||
**0.98.3** on first boot and self-manages from there (no D.1b), and the baked
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
# DRILL — a new household's first day on golden 0.282.0 (2026-09-29/30)
|
||||
|
||||
**Interventions a volunteer could not have made: 0.**
|
||||
**Ready for a first real tester: YES, on a new customer record — nothing stopped the walk, from the download to a
|
||||
restored app; but send the guide only after its five stale lines are fixed (R-722), and know that their apps get NO
|
||||
off-site copy until someone switches each one on (R-720).**
|
||||
|
||||
Evidence: `evidence-drill-new-household-2026-09-30/` — `journal.md` (every observable, in order), `golden/`,
|
||||
`phase0/`, `phase1/`, `phase2/`, `screens/` (codes blacked out), `box-logs-phase1/`, `box-logs-final/`, `teardown/`.
|
||||
Golden record: `../tests/golden-0.282.0-2026-09-29/`. Architecture read first: `01` §5 (who may reach an app), `07`
|
||||
§6.5–§6.6, `09` §3 decisions 11–49, `FIRST-ADMIN.md`, the first-hour row of `00-capability-map.md`.
|
||||
|
||||
---
|
||||
|
||||
## Claims in the brief that turned out wrong (or right), named first
|
||||
|
||||
| claim | verdict |
|
||||
|---|---|
|
||||
| The box lands on 0.282.0 from the golden | **RIGHT.** Controller 0.282.0 + agent 0.137.0, the vouched set; `selfupdate: Current version 0.282.0 is up to date`; no self-update. |
|
||||
| A one-level-deeper app address (`<app>.<sub>.felhom.eu`) fails TLS at the edge | **CONSISTENT, NOT MEASURED END TO END.** Every working customer domain is its own zone and the edge serves `*.enkisfelhom.hu, enkisfelhom.hu` only; `felhom.eu`'s own names are DNS-only. A proxied deeper name would get `*.felhom.eu`, which cannot match. No record was made to prove it (no Cloudflare key; the operator ruled the drill onto an existing zone). |
|
||||
| The gate never re-gates a set-up app after a kept-data reinstall | **THE PREMISE DID NOT HOLD for the app walked.** A volume-only app has no „A megőrzött adataimat használom / Tiszta lappal kezdem" choice — the volume is always deleted and only the backup is kept. The reinstall came back **empty and gated** (correct); after the household restored its backup the data returned and **the gate opened by itself 10 s later**. The household never sees a set-up app gated over its own data. The drive-kept path was not walked (no data drive was set up). |
|
||||
| A fresh customer's box runs an off-site leg on night one | **WRONG, twice.** The leg ran and **copied nothing it could keep**: (1) apps start with off-site OFF and nothing tells the household to switch them on (**R-720**); (2) on this customer, who had a box before, the repository was found **orphaned** and every run skips until the household presses a reset nobody pointed them to (**R-726**). |
|
||||
| The claim code is one-shot | **RIGHT.** The spent setup code → „Hibás vagy lejárt kód"; the bind link survived one typo and then bound. |
|
||||
|
||||
## 1. What held, in one table
|
||||
|
||||
| # | step | result | time |
|
||||
|---|---|---|---|
|
||||
| 0.1 | golden 0.282.0 baked, round-trip verified, vouched; R-120 refused 0.276.0 after | **PASS** | bake 7 min |
|
||||
| 0.2 | operator's part | customer `tester-1` (operator ruling); tunnel already on the record; passphrase from the page; **„Send self-bind link" pressed** (R-719) | — |
|
||||
| 1 | download from `felhom.eu/letoltes` | **PASS** — 1 705 324 544 B, sha == page == release record | 46 s |
|
||||
| 2 | install (guide answers, **Hungarian keyboard kept**) | **PASS** — two screens differ from the guide (pre-selected disk; focus on „Previous") | ≤ 7 m 27 s copying |
|
||||
| 3 | first boot → **the mailed link** → self-bind page | **PASS** — registered 31 s after power-on; one typo refused; bound; console turns to „a doboz össze van kötve" | bind → controller 2 m 51 s |
|
||||
| 4 | „ready", versions | **PASS** — 0.282.0 / 0.137.0 from the golden | — |
|
||||
| 5 | dashboard **through the public tunnel**; recovery code | **PASS** — no LAN detour (R-505 closed); code ready 12 min after the claim, made in 5 s, shown once, 410 after | power-on → claimed 27 m 51 s |
|
||||
| 6 | three apps: random first password / gated with a probe / gated with a sign-up lock | **PASS** — 65 s / 23 s / 12 s; the stranger saw only the gate page; „Kész" before the setup refused 409; gates opened (probe ~12 s; press) | — |
|
||||
| 7 | use for ~10 min; a family member through the 15-minute window | **PASS** — BookStack page + attachment, Vikunja project + tasks + file (sha equal); stranger 403 before and after the window, family member in | — |
|
||||
| 8 | backups page, „Mentés most" | **PASS** — true dates on both pages, no customer alarm; every app reads „3. mentés Kikapcsolva" (R-720) | 19 s |
|
||||
| 9 | „Naprakész" | **TRUE** — installed == catalog; no newer tested step offered | — |
|
||||
| 10 | remove keeping data, reinstall | **PASS with a defect** — a Stop during the first whole-guest backup was undone by that backup (**R-721**); no kept-data choice for a volume app; restore brought the data back and the gate lifted itself | — |
|
||||
| 11 | restore the password app | **PASS** — page + attachment byte-identical, the shown password works, same version | 26 s |
|
||||
| 12 | remove with „delete my data too" | **PASS** — volume, backup, restore points gone; front door 404; „Nincs megőrzött adat" | — |
|
||||
| 13 | status pages as a week-old household | small disagreements (**R-724**) | — |
|
||||
| A1 | power cut (60 s dark) | **PASS** — same versions, gates kept their state, data equal, **no alarm**; one re-login | +123 s to dashboard |
|
||||
| A2 | typos: pairing code on the bind page; the setup code on „Elfelejtett jelszó" | **PASS** — both refused in Hungarian, the right code accepted right after; spent code refused | — |
|
||||
| A3 | a phone opens a gated app first | **PASS** — Hungarian gate page → „Bejelentkezés" → dashboard login → straight back into the app; a native app's call gets English JSON (R-725) | — |
|
||||
|
||||
## 2. The operator's part — every step the volunteer depends on
|
||||
|
||||
1. **The customer:** `tester-1` (operator ruling during the run, instead of a new drill record) — domain
|
||||
`enkicsifelhom.hu`, `tester1@felhom.eu`, off-site shared 100 GB ON, DR tier ON.
|
||||
2. **The domain and tunnel:** already on the record (R-505, 2026-09-14). No Cloudflare API token exists in the
|
||||
operator's credentials file; nothing in Cloudflare was created or changed.
|
||||
3. **The owner passphrase:** read from the customer page's reveal field (5 words), handed over out of band.
|
||||
4. **„Send self-bind link" — pressed at 18:54:51 UTC.** The guide says no press is needed; for a customer whose last
|
||||
link expired (2026-09-24) nothing re-sends one when a new box registers (**R-719**).
|
||||
|
||||
Everything after that was the volunteer's own: both mails arrived in the tester's mailbox and were followed.
|
||||
|
||||
## 3. The first night, leg by leg (box time CEST; UTC in brackets)
|
||||
|
||||
| leg | what happened | why |
|
||||
|---|---|---|
|
||||
| database dump 02:30 (00:30) | **RAN, OK** — 1 database, 3 volume dumps, 15 s; recovery units for all three apps | — |
|
||||
| second-drive copy 03:30 (01:30) | **not run** — „nincs másik fizikai meghajtó — a 2. mentéshez 2. meghajtó szükséges" | not configured: no second drive assigned |
|
||||
| off-site copy 04:15 (02:15) | **SKIPPED** — `offsite repo ORPHANED … runs will skip until reset`; household timeline + operator mail | the customer's previous box left its repository under a destroyed key (**R-726**); and only BookStack was switched on at all (**R-720**) |
|
||||
| app update leg (after off-site) | **RAN** — `done=0 undone=0 held=0 failed=0` | nothing newer in the catalog |
|
||||
| whole-guest backup (agent) | **not run tonight** — no new archive on either storage | not due: local tier ran 21:27 CEST (24 h cadence), off-site PBS tier 21:37 CEST (7-day cadence) |
|
||||
| restore test (agent) 03:52 (01:52) | **FAILED** — picked a 2026-09-16 archive of an earlier drill box in the same namespace, `wrong key`; operator mail; the household sees „✗ Visszaállítás ellenőrizve — Helyi tároló (local)" | **R-727** |
|
||||
| off-site proof / integrity 05:30 / 06:00 | checked nothing — the repository is orphaned (said so, not called a failure) | follows R-726 |
|
||||
|
||||
**So a new household's box does not fully protect itself on night one.** The local layers did (database, volumes,
|
||||
whole-guest on the evening before); the off-site app copy did not, for two separate reasons, and the restore test
|
||||
proved nothing.
|
||||
|
||||
## 4. Findings
|
||||
|
||||
| row | rank | one line |
|
||||
|---|---|---|
|
||||
| **R-719** | P2 | An existing customer's new box gets no fresh connect link — the operator must press „Send self-bind link" |
|
||||
| **R-720** | P2 | Apps start with off-site OFF and nothing tells the household to switch them on — night one copies nothing off-site |
|
||||
| **R-721** | P2 | A household's Stop during a whole-guest backup is undone by that backup's resume |
|
||||
| **R-722** | P2 | The volunteer guide is stale in five places (BookStack's old default login is now refused, among them) |
|
||||
| **R-726** | P2 | A customer with an earlier box: the off-site repository is orphaned on night one; the fix is an unannounced button |
|
||||
| **R-727** | P2 | The restore test picks a previous box's archive, fails on its key; the page blames the local tier |
|
||||
| R-723 | P3 | Two operator mails on day one that describe nothing wrong (`node_recovered`, `backup_tier_skipped`) |
|
||||
| R-724 | P3 | Status pages disagree (schedule, raw timestamp, LAN unknown, „0 órája", R-500's UTC) |
|
||||
| R-725 | P3 | Copy slips (bind page „a beállításkor", formal wizard, stray console glyph, English JSON to phone apps) |
|
||||
| R-718 | P3 | measured again: the gate-open press restarts an app with its own switch (2 s 404) unannounced |
|
||||
|
||||
**Closed by this walk:** **R-505** — the box-side tunnel hop, proven end to end on a fresh box. **Held:** R-214
|
||||
(console after bind), R-496/R-497 (console text). **Still reproducing:** R-498 (`wiki.DOMAIN`), R-500 (UTC).
|
||||
**Register 350 → 359 rows.**
|
||||
|
||||
**No P1.** Nothing stopped a volunteer; every stop above is either an operator press (R-719) or a backup promise
|
||||
not yet kept (R-720, R-726, R-727).
|
||||
|
||||
## 5. Harness substitutions — not interventions, and what each hides
|
||||
|
||||
| | what | consequence |
|
||||
|---|---|---|
|
||||
| H3 | auto-reboot unticked, ISO detached by `qm set` | the "stick still in" reboot not exercised |
|
||||
| H4 | the Terminal-UI entry chosen over the graphical default | the graphical installer not exercised |
|
||||
| H5 | a nested VM with three virtual disks | disk choice and drive health not realistic („0 °C", „Nincs adat") |
|
||||
| H6 | no browser: every screen driven through the endpoint the page calls | script-rendered state not observed |
|
||||
|
||||
**Gone since 2026-09-14:** H1 (no mailbox — the tester1@ mailbox was read through the Gmail connector, both customer
|
||||
mails followed) and H2 (US keyboard — keystrokes sent as Hungarian-layout scancodes, `@` as AltGr+V).
|
||||
|
||||
**Harness slips, recorded:** two joined form tokens on the first claim („Érvénytelen űrlap", no code judged); a POST
|
||||
without `Origin` and a `-X POST` carried across a redirect on the phone login (two 403s, both the harness); a hub-page
|
||||
search for the MAC that could never match; `pkill -f` ended my own shell once (redone).
|
||||
|
||||
## 6. Teardown — four layers
|
||||
|
||||
See `evidence-drill-new-household-2026-09-30/teardown/`.
|
||||
|
||||
1. **Machine:** VM 340 destroyed `--purge` 06:48 UTC; `qm list` → only 9202 (and 9201 as a CT) remained.
|
||||
2. **Host (demo-hp):** the drill ISO removed (the three older ISOs were there before and stay); `nvme-scratch`
|
||||
back to 77.0 GB used, `local` to 24.07 GB; guests 9201 and 9202 running throughout; `local-lvm` never used.
|
||||
3. **Hub:** host `tester-1-693e79` deleted with the escrow acknowledgement (key → retained custody). **Customer
|
||||
`tester-1` KEPT, not reset** (operator ruling); its events are append-only and stay.
|
||||
4. **Outside the hub:** Cloudflare — nothing created, nothing removed (the tunnel is tester-1's own). **ep0
|
||||
(read-only):** see §6b.
|
||||
|
||||
Untouched, and said: both demo boxes' guests, `drill-r50`, DooPlex beyond the bake / vouch / hub pages.
|
||||
|
||||
### 6a. Teardown, measured
|
||||
|
||||
| layer | UTC | result |
|
||||
|---|---|---|
|
||||
| **1. machine** | 06:48:25 | `qm destroy 340 --purge` → `qm list` shows only 9202; `/mnt/hdd_1/images/340` gone |
|
||||
| **2. host** | 06:48:3x | drill ISO removed; `nvme-scratch` used 89.3 → **77.0 GB** (76.98 GB before the VM existed); `local` 25.7 → **24.07 GB** (24.06 before); 9201 and 9202 running before and after |
|
||||
| **3. hub** | 07:23:03 | the host was „ONLINE" to the hub until 45 min after its last report (the configured stale threshold, R-549) — 26 refusals `409 Host is ONLINE`, none changing anything; then `host deleted: tester-1-693e79` with the escrow acknowledgement (key demoted to retained custody — the log's „escrow deleted: true" wording is R-544). Host page 404; hosts list back to the three it had before. **Customer `tester-1` KEPT (200).** The delete sent the connect mail by itself (`self-bind link auto-minted … on host delete`) — **R-509's trigger proven again**; a valid 7-day link now sits in the tester mailbox with no box behind it |
|
||||
| **4. outside the hub** | 07:23:51 | **ep0, read-only:** `wgsync: pushed 4 peers` → `10.77.0.5/32` gone, 48 s after the delete; peers 5 → 4; the four others unchanged. **No manual step was needed (R-600 did not reproduce on a host delete).** Cloudflare: nothing created, nothing removed |
|
||||
|
||||
### 6b. Left on ep0 — listed, not touched, operator asked
|
||||
|
||||
`/mnt/pbs-datastore/ns/tester-1/ct/9201/`: `2026-09-16T17:27:32Z`, `2026-09-16T21:59:54Z` (earlier drill boxes, their
|
||||
keys destroyed on acknowledged deletes) and `2026-09-29T19:37:07Z` (this drill's box, key now in retained custody).
|
||||
The older two are what the restore test tripped on (R-727). **Per the brief, these are listed and left; STATUS asks
|
||||
the operator.** The customer's Storage Box sub-account (311327) holds the orphaned restic repository of an earlier
|
||||
box; this drill's box wrote nothing to it (every run skipped) — also left.
|
||||
|
||||
### 6c. Local secrets
|
||||
|
||||
The session's 0600 files (passphrase, pairing and setup codes, dashboard and app passwords, recovery code, the
|
||||
vaulted break-glass credential, the bind token) are shredded at the end of this session. The break-glass credential
|
||||
belonged to the deleted host.
|
||||
@@ -0,0 +1,289 @@
|
||||
# DRILL — night shift 2026-09-23: DooPlex guarded from its own tests, the test record, the proven moves, the chaos hour
|
||||
|
||||
Evidence: `audits/night-2026-09-23/` (`PROGRESS.md` is the step log). Architecture read first and named:
|
||||
`architecture/09-update-architecture.md` (§3 decisions 11–20, §6.1a, §6.4 parts 4–6, the verdict record),
|
||||
`07-backup-architecture.md` §6, `08-alarm-ladder.md`. **Method: endpoint-level** — every product act is
|
||||
the endpoint the UI invokes; pages are fetched as HTML in both languages; no browser.
|
||||
|
||||
## Not done, or changed
|
||||
|
||||
- **The floor stays at 0.266.0.** The brief raises it only if every live proof of Parts A, B and D passed; Part D
|
||||
found two P1s (R-658, R-659). Both exist in 0.266.0 as well — v0.267.0 does not cause them and adds R-640's
|
||||
protection to the same restore path — so the recommendation is to raise it; that is the operator's one
|
||||
question this morning.
|
||||
- **Adventurelog v0.13.0 was not moved** (R-655): our own template's healthcheck points at a node path the new
|
||||
image moved, and the new backend downloads world data from the internet at every start and crash-loops when
|
||||
the download is cut short.
|
||||
- **The stop rule fired in round 11** (R-659): all twelve rounds had already run; nothing further was injected.
|
||||
- **Rounds 2–5 did not land their accidents inside the action** (the instrument, stated in Part D); round 9 is
|
||||
the clean mid-update kill, round 6 the mid-verify memory hog, rounds 8 and 11 the mid-verify backup and the
|
||||
concurrent second update.
|
||||
- **Decisions taken by CC unattended** (operator may reverse, `09` §3): 22 — the memory mark reads the app's own
|
||||
memory, not the file cache; 23 — the push-time gate asks the registry for moved refs only.
|
||||
- R-518's per-tier quiesce, R-612's silent-seed half and R-613's template sweep stay open, narrowed.
|
||||
|
||||
## Three lines
|
||||
|
||||
|
||||
- **Interventions:** none by the operator; 5 by me on my own instruments (a stale-template wait, a dropped
|
||||
event ring, a killed runner of my own, a broken local edit reverted before any commit, the stop rule on a held
|
||||
app) — each stated where it happened. No product state was set by hand.
|
||||
- **Apps moved tonight: 11 of 14 tried (12 steps of 15 edges)**, each with its test record; the 21 moves of
|
||||
2026-09-22 backfilled.
|
||||
- **The result that matters most:** after a restore, an app's volumes lose the label the undo finds them by —
|
||||
so the next failed update is "undone" with **nothing** put back (R-658, P1); and a held file-leg app can be
|
||||
pointed at a restore that refuses it (R-659, P1).
|
||||
|
||||
---
|
||||
|
||||
## Part A — controller v0.267.0 (`80e6ad8c4772`, CI job 919 success)
|
||||
|
||||
| row | what shipped | red-proof (seen failing, then restored) | evidence |
|
||||
|---|---|---|---|
|
||||
| **R-650** | `internal/dockerexec`: every docker exec (77 sites) goes through it; under `go test` a real docker is refused with an error naming the command, unless `FELHOM_TEST_REAL_DOCKER=1` or the binary is a stub under the temp dir. `TestR650_NoBareDockerExec` sweeps the tree. | guard disabled → the decoy's `docker ps -a` runs (`got <nil>`); one bare call put back → the sweep names `dbdump.go:140` | `A1-r650-*.txt` |
|
||||
| R-650 sweep | **8 `api` tests FAILED on the refusal** — they built a real `stacks.Manager` and ran `docker ps` on DooPlex; `web` ran 67× `docker compose version`, 19× `docker ps`, 19× `docker info`, 10× `docker inspect felhom-samba`, 2× `docker exec felhom-samba`; `stacks` ran **1× `docker-compose down`**, 11× `compose ps`; `backup` 46× `docker ps`; `appexport` 7×; `system` 1×. `api`/`stacks`/`web` now run under `RunWithStub` (TestMain); the other three pass on the refusal. | — | `A1-r650-sweep.txt` |
|
||||
| **R-640** | `appbackup.CheckDumpComplete` (the engine's end marker, last 4 KiB, gzip-aware). The unit restore and the off-site restore refuse a cut-off copy **before the first mutation**; every replay checks again before any load, whatever the import seam is. | replay check removed → `a cut-off copy was LOADED`; unit gate removed → `stopped=true calls=[stop recreate startsvc:immich-postgres start]`; off-site gate removed → only the replay check caught it, after a stop and a rollback | `A2-r640-redproof.txt` |
|
||||
| **R-626** | **measured, not reproduced** on v0.266.0: navidrome deployed and removed through the product; `docker events` 390 s — the remove's kill/stop/die/destroy seen (the positive observable), **no create**; controller restarted at +150 s; guest rebooted after → 0 containers, 0 volumes at every check. My script first said `came_back: true` — it had counted the catalog-mirror directory every catalog app has; corrected in the verdict file, both values kept. Leftover found: `applied-compose.yml` + `applied-meta/` stay after a remove (R-651). | — | `A3-r626-*` |
|
||||
| **R-499** | the Tier-2 page's system-disk sentence has four branches (own drive / same disk / drive gone / cannot ask); „(PBS)" and „nincs külön teendő" only where true. Live on 9202 (no agent): the `unknown` branch in hu + en, matching `/api/storage/backup-target` `known:false`. | handler not passing the fact → 3 branches missing; the old sentence in the same-disk branch → `promises the backup protects this app` | `A4-*`, `A7-*` |
|
||||
| **R-518** | the backup button states the measured stop (≈ 8 min on a 12-app box), both languages, page + confirm. **The brief's „csak néhány másodpercre" was already gone** (v0.243.0); the vague „általában néhány perc" is replaced. | bundles back to v0.243.0 → measured downtime appears 0 times | `A5-*` |
|
||||
|
||||
Gates: `go build/vet` rc 0; `go test -count=1 ./...` rc 0 (31 packages); `controller_gates.py` all OK.
|
||||
Deployed to **9202 only** (`A6-*`); the floor waits for Part E.
|
||||
|
||||
## Part B — the test record and its gate (catalog `6db08a5`, CI job 920 success)
|
||||
|
||||
**Spike first (15 min):** a drill navidrome carrying a top-level `update_ladder:` (one JSON entry per
|
||||
line) on 9202 at **v0.266.0 and v0.267.0**: synced, deploy-fields read, deployed `running`, probe
|
||||
`API GET :4533/ping → 200`, badge „Naprakész" / "Up to date", no YAML error. Source agrees (no
|
||||
`KnownFields` anywhere). The brief's claim holds. `B1-spike-*`.
|
||||
|
||||
**The record** (`scripts/ladder.py`): `from`/`to` per service, `digest` per `to` ref (sha256 from
|
||||
`scripts/image_digest.py` — stdlib, equals Docker's `RepoDigests` for `privatebin/pdo:2.0.6` on 9202),
|
||||
`verdict` (proven | unrecorded), `tested_at`, `harness_version`, `evidence`, `box_evidence`,
|
||||
`memory_peak_pct` (+ `memory_basis`, `memory_cgroup_peak_pct`), `marks` {files_may_change, needs_person,
|
||||
memory_tight}. Written ONLY by `upgrade-test.py --write-ladder` (bench AND box `proven`, template at
|
||||
FROM, digests resolved) — `test_ladder_writer.py` 5/5.
|
||||
|
||||
**The gate** — two rows of `catalog_gates.py`, both `--fast`:
|
||||
`check-test-record.py` (static; runs in CI too: a ladder well-formed, continuous, and its newest step IS
|
||||
the compose) and `check-test-record-move.py` (history + the registry for MOVED refs only: a move adds a
|
||||
PROVEN, non-backfilled entry whose `from`/`to` are the compose before/after and whose digests the
|
||||
registry still serves; `memory_tight` needs the limit raised in the same range). 16 decoy cases, both
|
||||
directions. **Red-proofs:** the no-entry refusal removed → *a bare image move* and *the entry only in
|
||||
README* both pass (rc 0, expected 1); `failed` allowed → both failed-verdict cases pass; the digest
|
||||
comparison removed → *the registry now serves another digest* passes. `B2-gate-redproofs.txt`.
|
||||
Live, on the first real move (romm): the gate judged it against the real registry (rc 0), and with a
|
||||
wrong digest table it REFUSED all three services.
|
||||
|
||||
**Backfill** — the 21 moves of 2026-09-22, each from the record its commit cited:
|
||||
|
||||
| app | step (the service that moved) | verdict | evidence | note |
|
||||
|---|---|---|---|---|
|
||||
| actualbudget | actualbudget: actual-server:26.7.0 → actual-server:26.9.0 | proven | `audits/update-night-2026-09-21/apps/actualbudget/verdict.json` | — |
|
||||
| audiobookshelf | audiobookshelf: audiobookshelf:2.35.1 → audiobookshelf:2.36.1 | proven | `audits/update-night-2026-09-21/apps/audiobookshelf/verdict.json` | — |
|
||||
| bookstack | bookstack: bookstack:26.05.2 → bookstack:26.05.5 | proven | `audits/update-night-2026-09-21/apps/bookstack/verdict.json` | — |
|
||||
| docmost | docmost: docmost:0.95.0 → docmost:0.96.0 | proven | `audits/update-night-2026-09-21/apps/docmost/verdict.json` | — |
|
||||
| emby | emby: embyserver:4.10.0.20 → embyserver:4.11.0.1 | proven | `audits/the-28-2026-09-22/apps/emby/verdict.json` | — |
|
||||
| ghost | ghost: ghost:6.53.0-alpine → ghost:6.64.0-alpine | proven | `audits/the-28-2026-09-22/apps/ghost/verdict.json` | — |
|
||||
| grafana | grafana: grafana:13.1.0 → grafana:13.2.2 | proven | `audits/update-night-2026-09-21/apps/grafana/verdict.json` | — |
|
||||
| home-assistant | home-assistant: home-assistant:2026.7.2 → home-assistant:2026.9.3 | proven | `audits/update-night-2026-09-21/apps/home-assistant/verdict.json` | — |
|
||||
| immich | immich-server: immich-server:v3.0.3 → immich-server:v3.2.2 | proven | `audits/the-28-2026-09-22/apps/immich/verdict.json` | — |
|
||||
| mealie | mealie: mealie:v3.20.1 → mealie:v3.27.0 | proven | `audits/update-night-2026-09-21/apps/mealie/verdict.json` | — |
|
||||
| n8n | n8n: n8n:2.31.3 → n8n:2.40.5 | proven | `audits/update-night-2026-09-21/apps/n8n/verdict.json` | — |
|
||||
| navidrome | navidrome: navidrome:0.63.2 → navidrome:0.64.0 | proven | `audits/update-night-2026-09-21/apps/navidrome/verdict.json` | — |
|
||||
| nextcloud | nextcloud-db: mariadb:11.6 → mariadb:12.3 | proven | `audits/update-night-2026-09-21/apps/nextcloud-engine-mariadb/verdict.json` | commit cited no record; this one named |
|
||||
| papra | papra: papra:26.6.1-rootless → papra:26.6.2-rootless | proven | `audits/update-night-2026-09-21/apps/papra/verdict.json` | — |
|
||||
| privatebin | privatebin: pdo:2.0.5 → pdo:2.0.6 | proven | `audits/update-night-2026-09-21/apps/privatebin/verdict.json` | — |
|
||||
| radarr | radarr: radarr:6.3.0 → radarr:6.4.4 | proven | `audits/the-28-2026-09-22/apps/radarr/verdict.json` | — |
|
||||
| romm | romm: romm:5.0.0 → romm:5.3.0 | proven | `audits/update-night-2026-09-21/apps/romm/verdict.json` | memory watch 80.9 % (M1) |
|
||||
| sonarr | sonarr: sonarr:4.0.19 → sonarr:4.0.20 | proven | `audits/the-28-2026-09-22/apps/sonarr/verdict.json` | — |
|
||||
| tandoor | tandoor: recipes:2.6.13 → recipes:2.6.15 | proven | `audits/update-night-2026-09-21/apps/tandoor/verdict.json` | — |
|
||||
| termix | termix: termix:2.5.0 → termix:2.8.0 | proven | `audits/the-28-2026-09-22/apps/termix/verdict.json` | — |
|
||||
| vikunja | vikunja: vikunja:2.3.0 → vikunja:2.6.0 | proven | `audits/update-night-2026-09-21/apps/vikunja/verdict.json` | — |
|
||||
|
||||
**21 of 21 `proven`, 0 `unrecorded`.** Every entry is marked `backfilled`, carries harness v1 (box walk only, no memory watch — except romm's M1 figure), and a digest resolved TONIGHT, which each entry's `note` says. `scripts/ladder_backfill.py`.
|
||||
|
||||
## Part C — move every app that proves itself
|
||||
|
||||
**Venues:** the bench = throwaway LXC **9401** on demo-hp (Debian 13, docker 26.1.5, Docker Hub login),
|
||||
created tonight and destroyed at teardown; the box = guest **9202**, v0.267.0, drill catalog. **Negative
|
||||
control first:** C3 (`alpine:3.20` as the TO image) → `failed` — the bench measures (`bench-C3/`).
|
||||
Box walks by `run_edge.py`; bench edges by `upgrade-test.py --move`; each edge's bench evidence copied off
|
||||
the bench when it ended (`benchq.py`). Evidence per app: `night-2026-09-23/apps/<app>/`.
|
||||
|
||||
| app | from → to | class | bench verdict | box verdict | memory: own (cgroup) peak % | migration line | moved? |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| nextcloud | nextcloud 34.0.1 → 34.0.4 | MariaDB + file-leg | proven | proven (done) | 24.2 (100) | nextcloud / Upgrading nextcloud from 34.0.1.2 ... | moved `5d49a11` (files_may_change) |
|
||||
| romm | romm 5.3.0 → 5.3.1 | MariaDB | proven | proven (done) | n/a — measured before the own-memory sample existed (cgroup 79.5) | romm / INFO: [RomM][init][2026-09-23 21:12:26][0 | moved `7ff4e68` |
|
||||
| romm-engine | romm-db mariadb 11.4 → 11.8 (own edge) | MariaDB engine | proven | proven (done) | 78.0 (84) | romm / INFO: [RomM][init][2026-09-23 21:47:51][0 | moved `431cdec` |
|
||||
| kimai | kimai-db mariadb 11.6 → 11.8 (own edge) | MariaDB engine | proven | proven (done) | 37.9 (100) | kimai / [OK] Already at the latest version ("DoctrineMigrations\Version2 | moved `376b2c3` |
|
||||
| adventurelog | adventurelog v0.12.1 → v0.13.0 (both images) | PostgreSQL (PostGIS) | failed ×2 (frontend never healthy; then backend crash-loop on `download-countries`) | failed (undone at +196 s, drill timeout 90 s) | — | adventurelog / Apply all migrations: account, admin, adventures, aut | **NOT moved** — our template's frontend healthcheck points at `/nodejs/bin/node`, gone in v0.13.0; with it dropped, the backend's start-time internet download crash-looped (R-655) |
|
||||
| immich | immich-machine-learning v3.0.3 → v3.2.2 | PostgreSQL (server unchanged) | proven | proven (done) | 51.4 (100) | immich-server / [Nest] 7 - 09/23/2026, 9:11:07 PM LO | moved `b82b7c6` |
|
||||
| n8n | n8n 2.40.5 → 2.41.1 | SQLite | proven | proven (done) | 22.7 (47) | n8n / Migrations in progress, please do NOT stop the process. | moved `04b63db` |
|
||||
| ghost | ghost 6.64.0 → 6.65.0 | SQLite | proven | proven (done) | 24.9 (43) | ghost / [2026-09-23 21:21:22] WARN Database state requires migration | moved `e6afab9` (first watch had no load — re-run) |
|
||||
| komga | komga 1.25.0 → 1.27.1 | embedded | proven | proven (done) | 60.0 (77) | komga / 2026-09-23T21:35:41.616+02:00 INFO 1 --- [ main] o.f.core | moved `fc1becc` with 768M (memory_tight at 512M) |
|
||||
| opengist | opengist 1.13 → 1.15 | SQLite | proven | proven (done) | 77.8 (100) | — | moved `657c83f` (pages under `/-/`, R-654) |
|
||||
| wishlist | wishlist v0.66.0 → v0.67.1 | SQLite | proven | proven (done) | 35.9 (53) | wishlist / Prisma schema loaded from prisma/schema.prisma. | moved `1545c5f` (after R-612's 512M) |
|
||||
| navidrome | navidrome 0.64.0 → 0.64.1 | SQLite | proven | proven (done) | 6.3 (9) | navidrome / time="2026-09-23T19:31:04Z" level=info msg="goose: no migrations | moved `76cc1f5` (second ladder step) |
|
||||
| emby | emby 4.11.0.1 → 4.11.0.3 | embedded | proven | proven (done) | 4.5 (8) | emby / Info SqliteUserRepository: Sqlite compiler options: ATOMIC_INTRINSICS | moved `8d573ad` |
|
||||
| gitea | gitea 1.27.0 → 1.27.3 | SQLite | inconclusive | inconclusive () | — | — | NOT moved — inconclusive on both: its web installer blocks a headless first account (R-624) |
|
||||
|
||||
**12 steps across 11 apps published**, each written by `upgrade-test.py --write-ladder`, each its own
|
||||
commit, every push through the hook (the new gate judged each against the registry), CI green (jobs
|
||||
922–937, matched on head_sha; two commits rode inside a later push and were judged there). Tried: 14
|
||||
apps, 15 edges. `night-2026-09-23/C-moves.txt` lists the commits.
|
||||
|
||||
**Listed, not moved:**
|
||||
- *across a major* (11): sparkyfitness v0.17.3→v1.7.2 (both images), gokapi v1.9.6→v2.2.4, claper 2.5→3.0,
|
||||
homepage v1.13.2→v2.4.0, paperless-ngx 2.20.15→3.2.1, mariadb →13.0 (romm, kimai, bookstack,
|
||||
nextcloud), nextcloud 34→35, plex 1.41→1.43.
|
||||
- *no front-door seed route* (R-624): code-server, outline, rallly; *sign-up closed by design*: vaultwarden,
|
||||
zipline; *installer*: gitea (tried — inconclusive on both venues).
|
||||
- *no fixture tonight*: bentopdf, glance, crafty-controller, wger (2.6→2.7), wanderer's meilisearch
|
||||
(v1.36→v1.54), uptime-kuma (2.4.0→2.5.5 — its first account exists only over socket.io).
|
||||
- *tried and failed*: adventurelog (R-655, two causes measured).
|
||||
|
||||
**What the night added to the bench:** the box walk's fixtures run on the bench (`upgrade_boxport.py`); a
|
||||
wishlist fixture; opengist, komga and nextcloud fixtures fixed; the memory watch's load no longer errors on
|
||||
box-fixture apps (R-653) and samples the app's own memory (decision 22, R-652); `files_may_change` from a
|
||||
bind-mount tree hash (nextcloud's is the first entry carrying it).
|
||||
|
||||
**R-612 / R-613 — both fixed** (catalog `a5a729a`): wishlist 512M — 128M kills the first-boot seed every
|
||||
run, sometimes after the Role/Group rows (so the sign-up lie is timing-dependent; it did NOT reproduce on
|
||||
9202 tonight); 512M completes it (peak 312M). uptime-kuma `UPTIME_KUMA_DB_TYPE=sqlite` — before, the box
|
||||
read `running` over `setup-database`; after, the real server (`/metrics` 401) and the database in the
|
||||
volume. `apps/wishlist/r612-*`, `apps/uptime-kuma/r613-*`.
|
||||
|
||||
**A consequence of publishing tonight, seen on demo-hp:** romm now has three steps (the backfilled
|
||||
5.0.0→5.3.0, 5.3.0→5.3.1, mariadb 11.4→11.8). The box cannot climb one step at a time yet (part 5), so the
|
||||
press on 9201 took BOTH new steps at once — each tested alone, never together. It ended `done`, front door
|
||||
200, 0 OOM kills after (`press9201/`).
|
||||
|
||||
## Part D — the chaos hour
|
||||
|
||||
**Written BEFORE round 1 (committed with this section).** Guest 9202, controller v0.267.0, drill catalog,
|
||||
`update.health_timeout: 90s` (the drill setting — the product default is 5 min). Schedule drawn once by
|
||||
`night-2026-09-23/chaos_schedule.py` from seed **20260923** (constraints in its docstring):
|
||||
|
||||
seed 20260923, drawn in 3 attempt(s)
|
||||
|
||||
| round | action | accident | at +s |
|
||||
|---|---|---|---|
|
||||
| 1 | install | none | 11 |
|
||||
| 2 | good_update | box_backup | 46 |
|
||||
| 3 | restore | controller_kill | 28 |
|
||||
| 4 | fail_health | memory_hog | 43 |
|
||||
| 5 | install | disk_fill | 28 |
|
||||
| 6 | good_update | memory_hog | 46 |
|
||||
| 7 | restore | power_cut | 56 |
|
||||
| 8 | cutoff | box_backup | 23 |
|
||||
| 9 | fail_health | controller_kill | 58 |
|
||||
| 10 | remove | docker_restart | 10 |
|
||||
| 11 | cutoff | two_updates | 51 |
|
||||
| 12 | remove | none | 47 |
|
||||
|
||||
**The app each round acts on** (fixed by rule in `chaos.py`, not drawn):
|
||||
|
||||
| round | app | why this app |
|
||||
|---|---|---|
|
||||
| 1 install / 10 remove | actualbudget | a throwaway beside the six |
|
||||
| 5 install / 12 remove | papra | a second throwaway |
|
||||
| 2 good_update | docmost 0.95.0 → 0.96.0 | PostgreSQL, the BIG database (300 extra spaces) |
|
||||
| 6 good_update | romm 5.3.0 → 5.3.1 | MariaDB |
|
||||
| 3 restore | vikunja | volume-only (SQLite + files) |
|
||||
| 7 restore | docmost | the big database, under a power cut |
|
||||
| 4 fail_health | adventurelog v0.12.1 → v0.13.0, probe on a port it does not answer | the second PostgreSQL (PostGIS) |
|
||||
| 9 fail_health | vikunja 2.5.0 → 2.6.0, wrong probe | volume-only, under a controller kill |
|
||||
| 8 cutoff | navidrome 0.64.0 → 0.64.1, wrong probe, the undo copy's finished-marker removed during `verifying` | volume-only + a drive folder |
|
||||
| 11 cutoff + two_updates | nextcloud 34.0.1 → 34.0.4 (the file-leg), and romm's database-engine Update (11.4 → 11.8, a tested good step) pressed at +51 s | two updates at once. *Changed before round 1:* the first draft named adventurelog, whose v0.13.0 cannot pass health (Part C) |
|
||||
|
||||
Setup: the six at their FROM pins, each seeded through its front door (A); one whole-box backup on 9202
|
||||
(the restore rounds need a copy); seed B on docmost, romm and vikunja after it.
|
||||
|
||||
### The twelve rounds — the five things, and the data read back
|
||||
|
||||
Evidence per round: `night-2026-09-23/chaos/round-NN.{json,log}`, `round-NN-run.txt`; rounds 11–12 also
|
||||
`round-NN-controller.log`. **When the accident landed** is stated, because in rounds 2–5 it did not land
|
||||
where the schedule meant it to (below).
|
||||
|
||||
| # | action · app × accident | what the household saw (hu / en) | what the box did by itself | steady | events that fired | should have fired, did not | data read back |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 1 | install · actualbudget × none | installed; badge none | deploy 202 → running | 26.6 s | app_deploy_started, app_deployed; **app_start_failed (warning)** 6 s in — its app cannot be named (see "instrument") | — | A ✓ |
|
||||
| 2 | good update · docmost 0.95→0.96 (412 spaces) × whole-box backup | „Naprakész" / "Up to date" | safety-dump → pulling → copying (+76.6 s) → starting → verifying → **done 108 s** | 558 s (waited ~7 min for the catalog — instrument) | none | — | A ✓, B ✓ (paged re-read; the first read asked page 1 of 21) |
|
||||
| 3 | restore · vikunja × controller kill (+28 s) | „Naprakész" / "Up to date" | restore 6 s, running; the kill landed AFTER it; controller back in 0.2 s | 31.5 s | none | — | A ✓, B ✓ |
|
||||
| 4 | fail health · adventurelog × 1 GB hog | „A(z) adventurelog frissítése … nem sikerült. A doboz automatikusan visszaállította az előző változatot és az adatokat — semmi nem veszett el." / "The update of adventurelog … did not succeed. The box put back the previous version and its data automatically — nothing was lost." | verifying → **undoing +110.9 s → undone +155.1 s** (the hog had ended — instrument) | 605 s | health_change, **app_update_undone (warning)** ✓ | — | A ✓ |
|
||||
| 5 | install · papra × disk to 3 GB free | installed | deploy → running in 27 s; the fill landed just after | 26.6 s | deploy pair | a disk warning: not measurable (the fill lasted ~20 s, the health job runs every 5 min) | A ✓ |
|
||||
| 6 | good update · romm 5.3.0→5.3.1 × 1 GB hog (**during verifying**) | „Naprakész" / "Up to date" | … verifying → **done 55.8 s** | 61.7 s | none | — | A ✓ |
|
||||
| 7 | restore · docmost × **power cut** (+56 s, after the 32 s restore) | „Naprakész" / "Up to date" | restore 32 s; cut; guest back, controller back 12 s, apps up | 103.5 s | controller_started | — | A ✓, B ✓ |
|
||||
| 8 | cut-off undo copy · navidrome × whole-box backup (+23 s, during verifying) | „Megállítva — visszaállítás szükséges" + the hold sentence naming a restore / "Stopped — restore needed" + the same in English | verifying → undoing +94.9 → **failed 95.4 s**: the copy's marker missing → nothing poured back → HOLD | 101 s | **app_update_held (error)** ✓, then **app_start_failed** 11 s later | — (the second is extra: R-660) | held (A unreadable by design) → **the household's restore brought it back, A ✓** |
|
||||
| 9 | fail health · vikunja 2.5→2.6 × **controller kill in verifying** | „A(z) vikunja frissítése … nem sikerült. A doboz automatikusan visszaállította …" / "… put back the previous version and its data automatically …" | killed at +58 s → *"update recovery: vikunja was interrupted in verifying … RESUMING the health wait"* → undoing → **undone 157.7 s** — with **`the undo copy will hold 0 named volume(s)`** (R-658) | 163 s | controller_started, **app_update_undone** ✓ | — | A ✓, B ✓ (the old version happens to run on migrated data) |
|
||||
| 10 | remove · actualbudget × docker restart (+10 s) | the remove press answered **502** | stop done; remove lost to the restart; app stays installed and stopped | cap 1500 s | controller_started; app_start_failed + health_change every 5 min | — | app stopped (A not readable); the household must press again |
|
||||
| 11 | cut-off undo copy · nextcloud × **a second Update (romm's engine step) at the same moment** | hold as round 8 (both languages); romm „Naprakész" | nextcloud → **failed/HOLD 112.6 s**; romm ran at the same time → **done** (no single-flight, §3b Q4) | 143 s | **app_update_held** ✓, app_start_failed 13 s later | — | romm A ✓; nextcloud held → **the named restore REFUSED → R-659, the stop rule** |
|
||||
| 12 | remove · papra × none | not installed | stop → remove 200 → verified | 40.9 s | app_removed; health_change (the box-wide job: a held app + a stopped one — true) | — | removed clean (0 containers, 0 volumes) |
|
||||
|
||||
**The instrument, stated rather than hidden.** (1) Rounds 2 and 4 waited ~7 min for the drill commit on
|
||||
the wrong file (an installed app's stack dir holds the APPLIED compose), so their accidents fired before the
|
||||
update; round 3's and 5's actions ended before theirs. From round 5 the accident is armed at the press, and
|
||||
from round 6 the wait reads the badge's own `catalog_images` (seconds). (2) The event column first came from
|
||||
the controller's debug ring, which overflowed in the long rounds and MISSED both `app_update_held` events; it
|
||||
was rebuilt from the controller's full log (`E8-events-*`) for rounds 7–12. Rounds 1–6 keep the ring's view,
|
||||
and round 1's `app_start_failed` cannot be tied to an app: round 7's power cut took the container's earlier
|
||||
log. From round 11 each round saves the whole controller log. (3) My stop rule first fired on round 8 because
|
||||
a held app cannot answer; a hold is by design, so recovery — not the readback — is the test.
|
||||
|
||||
### The truth table (`08-alarm-ladder.md`)
|
||||
|
||||
| situation | the ladder says | fired? |
|
||||
|---|---|---|
|
||||
| an update undone (rounds 4, 9) | `app_update_undone`, warning, household + operator, one per app per outcome | ✓ both |
|
||||
| an update held (rounds 8, 11) | `app_update_held`, error | ✓ both — and each was followed by an `app_start_failed` for the same moment (R-660: a held app is stopped by the product and should not ALSO read as down) |
|
||||
| a controller kill / power cut / docker restart (3, 7, 9, 10) | `controller_started`, info; the boot grace suppresses app alarms for 90 s | ✓ controller_started each time; no app alarm inside the grace |
|
||||
| a restore that finished (3, 7) | nothing (a refused or interrupted one: `restore_interrupted`) | ✓ nothing |
|
||||
| a remove (12, teardown) | `app_removed`, info | ✓ |
|
||||
| a disk filled to 1 GB above the floor (5) | a disk warning from the health job | not measurable — the fill lasted ~20 s |
|
||||
| the box-wide health job with a held + a stopped app | `health_change`, warning | ✓ every 5 min |
|
||||
|
||||
### demo-hp 9201 — the standing apps Part C moved, pressed through the product
|
||||
|
||||
`night-2026-09-23/press9201/` (live catalog, a hub attached, controller v0.266.0; no whole-box backup pressed).
|
||||
|
||||
| app | from → to | phases | end | front door after | memory after |
|
||||
|---|---|---|---|---|---|
|
||||
| opengist | 1.13 → 1.15 | backing-up → … → done | **done in ~16 s** | `/healthcheck` 200 | — |
|
||||
| kimai | mariadb 11.6 → 11.8 (engine alone) | … → done | **done in ~69 s** | `/en/login` 200 | — |
|
||||
| romm | 5.3.0 → 5.3.1 **and** mariadb 11.4 → 11.8 in ONE press (no ladder on the box yet) | … → done | **done in ~90 s** | `/api/heartbeat` 200 | 600 MiB / 768, `oom_kill` 0, 0 restarts (checked 10 min later) |
|
||||
|
||||
Every standing app afterwards: 24 containers before, 24 after, all healthy (`E7-9201-after.txt`). Nothing
|
||||
else on 9201 was touched.
|
||||
|
||||
## Teardown
|
||||
|
||||
**Machine — 9202:** every app of the night removed THROUGH THE PRODUCT (docmost, navidrome, vikunja, romm,
|
||||
nextcloud, adventurelog, actualbudget; papra in round 12): 0 containers, 0 volumes, 0 undo copies left
|
||||
(`E2-teardown-removes.*`). For the three RESTORED apps the remove answered `volumes_removed: []` while their
|
||||
volumes did go (compose's `down --volumes` removes them by name) — the report is wrong, not the act (R-658).
|
||||
The drive folders the product kept (R-442 on 9202, as every night) removed by name: navidrome, nextcloud, romm
|
||||
(`E3-*`); older folders from earlier nights left as found. 54 test images removed by name before Part D (no
|
||||
prune): `/var/lib/docker` 86 % → 24 %. `controller.yaml` restored and read back **identical** to
|
||||
`.pre-night0923`; the catalog cache on 9202 reads live `5d49a11` (`E4-*`, `E5-*`). **9202 stays on controller
|
||||
v0.267.0** (the scratch guest; self-update off).
|
||||
**Machine — the bench:** LXC 9401 destroyed (`pct destroy --purge`, its hostname checked first); the Debian
|
||||
template downloaded for it removed (`E6-*`).
|
||||
**Machine — 9201:** only the three guarded Updates above; 24 containers before and after, all healthy.
|
||||
**Host — demo-hp:** `pct list` = 9201, 9202 (as before); `local` 84.49 % before and after; `nvme-scratch`
|
||||
7.45 % → 7.59 % and `local-lvm` 56.3 % → 57.7 % (the two guests' pulled images) (`00-*`, `E6-*`).
|
||||
**Drill repo:** reset to live `main` `5d49a11b31fe`, `has_actions: false`, read back (`E5-*`).
|
||||
**Hub:** not touched. **The floor was NOT raised** — see the three lines.
|
||||
|
||||
## Claims in the brief that turned out wrong, or held
|
||||
|
||||
| the brief's claim | what the night found |
|
||||
|---|---|
|
||||
| the controller ignores an unknown `.felhom.yml` key (Part B spike) | **held** — v0.266.0 and v0.267.0 synced, deployed, probed and badged navidrome with `update_ladder:` exactly as without it |
|
||||
| R-633's window closes R-626 | **held, as far as one measurement goes** — 390 s of events + a restart + a reboot: nothing came back. Closed by measurement, not by a named creator |
|
||||
| 21 backfill records exist | **held** — 21 of 21; one commit (nextcloud's engine move) cited none and its record was found by name |
|
||||
| wishlist's fault is memory (read from R-612) | **held, and sharpened** — 128M kills the seed every run; whether the sign-up then fails depends on WHEN the kill lands (it did not fail on 9202 tonight) |
|
||||
| R-518: the page promises „csak néhány másodpercre" | **wrong** — v0.243.0 had already changed it to „általában néhány perc"; tonight states the measured ≈ 8 min |
|
||||
| `check-image-resolvable.py` already resolves digests | **half** — it asks whether a ref EXISTS and throws the digest away; `image_digest.py` was written |
|
||||
| one chaos round ≈ one accident inside one action | **not as drawn** — actions of 6–30 s ended before accidents at +28–56 s; four rounds' accidents landed outside the action (stated per round) |
|
||||
| the chaos schedule's round-11 second update (first draft: adventurelog) | changed BEFORE round 1 to romm's engine step — adventurelog v0.13.0 cannot pass health |
|
||||
@@ -0,0 +1,201 @@
|
||||
# DRILL — night 2026-09-24 (run 2026-09-24 from 11:07 CEST): the second drive brings a file app back whole; a crash loop is stopped; exact image fingerprints; automatic updates spiked; a chaos hour
|
||||
|
||||
Run 2026-09-24 from 11:07 CEST (daytime; the brief's clock times were applied relative to the start).
|
||||
Evidence: `night-2026-09-24/` (PROGRESS.md is the step log). Controller **v0.269.0 → v0.269.1**, hub
|
||||
**v0.123.0**, catalog **cf7cf84**, floor **0.269.1** (MinAgent 0.131.0). Architecture read: `09` (§3
|
||||
decisions 11–28, §6.4), `07` (the whole-copy table), `08` (the alarm ladder).
|
||||
|
||||
## Not done, or changed
|
||||
|
||||
- **Part D (move more apps) was NOT run.** Its bar needs a bench machine on demo-hp that is created and
|
||||
destroyed tonight; this session's permission check refused host-level destroys (it refused a
|
||||
`pct destroy`), so a bench could not be torn down — a half-state the rules forbid. Part D is first in
|
||||
the brief's order of sacrifice. **Apps moved: 0 of 0 tried.** Part E's last step (Update on demo-hp
|
||||
9201 the standing apps Part D moved) therefore had nothing to press.
|
||||
- **Two controller releases, not one** — v0.269.1 fixes a defect v0.269.0's own live proof found (the
|
||||
sync moved an installed app's image digest, so a restart would have changed the image unguarded).
|
||||
*Decided by CC unattended — operator may reverse.* The floor went to 0.269.1.
|
||||
- **The crash-loop threshold is 6 restarts in 10 minutes, not the brief's 10** — measured: Docker's
|
||||
back-off caps a steady loop at ~1 restart/min. *Decided by CC unattended — operator may reverse.*
|
||||
- **Part E's "whole-box backup" accident was the box's own full backup run**, not a vzdump: demo-hp's
|
||||
backup storage had ~4 GB free and its root was at 90 %.
|
||||
- **Part C step 1 was measured on demo-hp 9201's real night, not on 9202**: 9202 has no off-site target,
|
||||
so its off-site leg never runs. 9202 gave the db-dump leg and the window move.
|
||||
- **demo-hp is damaged and needs the operator (R-672, P1):** before tonight's run the agent's scheduled
|
||||
restore-test filled the host's thin pool; 9201's disks went READ-ONLY. The pool was freed (intervention
|
||||
1); 9201 still needs stop + fsck + start, and did not take the 0.269.1 floor.
|
||||
- Round 2's install and round 9's remove were broken by their accidents (R-681, R-682) — recorded, not fixed.
|
||||
|
||||
- **A slip of mine:** commit `4502af6` carried `E/chaos-state.json` with the passwords of three drill test
|
||||
accounts (wishlist, navidrome, n8n on scratch guest 9202). All three apps and their data were removed at
|
||||
teardown, so the accounts no longer exist; the file was deleted and ignored in `f2234c0`. History keeps it —
|
||||
not rewritten (a shared repo). The runner now keeps its state outside the evidence tree.
|
||||
|
||||
**Interventions: 1** (an agent restart on demo-hp so its own Recover tore down the leaked restore-test
|
||||
guest). **Apps moved: 0 of 0 tried.** **The result that matters most:** a file app now comes back WHOLE
|
||||
from the second drive — proven four times, twice under a power cut or a nearly full disk — and nothing the
|
||||
household had was deleted or overwritten.
|
||||
|
||||
## A1 — the whole restore from the second drive (decision 26, R-661) — PROVEN LIVE
|
||||
|
||||
- **Spike** (`A1/00-spike*`): the Tier-2 mirror keeps a file app's drive files as plain files under
|
||||
`hdd/<rel>` and `userdata/<rel>` — readable file by file. **R-668 found on the way:** the Tier-2 target
|
||||
was a removed folder on the app's own disk (the same-disk check failed open) — fixed, fails closed.
|
||||
- **File rules, tested** (`tier2_whole_test.go`): the whole tree fingerprinted before and after; never
|
||||
delete, never overwrite a newer live file, bring back the missing, keep an older replaced file beside.
|
||||
Red-proofs: `redproofs/A1-file-rules.txt`, `A1-r668.txt`.
|
||||
- **Live, 9202** (`A1/21-readback.json`): restored 1, replaced 0, kept-newer 1, unchanged 137; 3/3 volumes,
|
||||
1/1 database; 38 s; the account, the deleted file (back), the edited file (the household's newer copy kept)
|
||||
and an untouched file read back. **From a hold** (`A1/30-*`): the page named „második meghajtó … a
|
||||
beállításokat, az adatbázist és a fájlokat" / "second drive … the settings, the database and the files";
|
||||
its restore cleared the hold in 33.8 s. **Chaos rounds 3 and 11** repeated it under a power cut and a disk
|
||||
1 GB above the floor.
|
||||
|
||||
## A2 — keep-data-only Remove while support is informed (decision 27, R-666) — PROVEN LIVE
|
||||
|
||||
One-drive hold (the second-drive mirror moved aside): page hu „…A Felhom ügyfélszolgálatát értesítettük —
|
||||
addig ne indítsd újra, és ne töröld az adatait." / en "…Felhom support has been told — until then, do not
|
||||
restart the app or delete its data."; `keep_data_only: true`; Remove with data or with backups → **409**
|
||||
hu „Amíg az ügyfélszolgálat foglalkozik az alkalmazással, az adatai nem törölhetők — az alkalmazást az
|
||||
adatai megtartásával eltávolíthatod." / en "While support is working on this app, its data cannot be deleted
|
||||
— you can remove the app and keep its data."; the app stayed. Positive control: a hold WITH a whole copy
|
||||
→ `keep_data_only` absent (`A2/10-*`). **Found (R-669, proven):** that hold was reached because the undo
|
||||
judged the right old version with the FAILED step's probe, left by the previous hold + restore.
|
||||
|
||||
## A3 — the box stops a crash loop / OOM storm (decision 28, R-667) — PROVEN LIVE
|
||||
|
||||
gokapi (a real crash loop): stopped at „crash_loop — 6 in 10m0s", trip 1, page hu/en + Start, no
|
||||
„restore needed" badge (`A3/21`); **Start** lifted it, the fresh container restarted 9 times in 32 s, stopped
|
||||
again: „Az alkalmazást ismét leállítottuk … A Felhom ügyfélszolgálatát értesítettük." / "We stopped the app
|
||||
again … Felhom support has been told." (`A3/30`). Detector red-proofed (`redproofs/A3-r667-detector.txt`).
|
||||
Hub v0.123.0 routes `app_stopped_unhealthy` to the household and the operator (red-proofed, `A3-hub.txt`).
|
||||
|
||||
## A4 — each step judged by its own file (R-664, R-665); R-662 removed — shipped, red-proofed
|
||||
|
||||
`redproofs/A4-*`. Live through Part C (three apps climbed with their own step files).
|
||||
|
||||
## Part B — exact image fingerprints on the box — PROVEN LIVE (on v0.269.1)
|
||||
|
||||
- Compose accepts `tag@digest` for **every** template with a tested digest: 25 apps, 37 digests, 0 missing,
|
||||
0 parse differences (`B/01`).
|
||||
- A floating tag (redis:7-alpine) re-tested at a new digest: badge „Frissítés elérhető" (hu/en), Update
|
||||
pulled exactly that digest, badge „Naprakész", records digest-free (`B/10`, `B/21`).
|
||||
- **Found and fixed:** v0.269.0's sync wrote the NEW digest into the RUNNING app's compose before Update —
|
||||
a restart would have changed the image with no backup and no undo. v0.269.1 `CarryDigests` keeps an
|
||||
installed app's digest; red-proofed (`redproofs/B-sync-carry.txt`); re-proven live (`B/21`: "the sync kept
|
||||
the running digest: True").
|
||||
|
||||
## Part C — automatic updates, SPIKE — the build brief is `09` §6.4.2
|
||||
|
||||
Chain today: legs clock-scheduled, nothing waits. 9201's real night: off-site 04:15:05 → 04:18:33 (3 m 28 s),
|
||||
gate opens 04:30 → ~11 min for an update leg (decision 20's wait is required). "Off-site finished" signal:
|
||||
only the persisted `offbox.LastStatus`+`LastRun` (`offbox.go:1003-1125`), not written on five early returns
|
||||
— **build: chain, do not poll.** Gate interlock: an `Options` func beside `WindowStartFn` in `quiesce.runOnce`.
|
||||
Simulated night on 9202 (`C/31`): romm 3 steps 77/71/94 s; wishlist 56 s; navidrome 11/9 s; vikunja's failing
|
||||
step undone in 104 s (90 s drill timeout; ≈ 6–11 min at the 5 min default) and set aside; total 7 m 18 s.
|
||||
Found: stale steps-left after `done` (R-678), Update on a current app runs a full update (R-679), the box
|
||||
does not remember a failed step (R-680). Default of the per-box switch: **ON** (decision 12).
|
||||
|
||||
## Part D — not run (see "Not done")
|
||||
|
||||
## Part E — the schedule (committed before round 1)
|
||||
|
||||
Drawn by `night-2026-09-24/tools/chaos_schedule.py`, seed **20260924** (constraints: every action at least
|
||||
once; controller killed at most 3 times, ≥ 20 min apart; Start-after-unhealthy after a stop; Remove-on-held
|
||||
after a hold). Guest **9202** (scratch), controller v0.269.1, drill catalog.
|
||||
|
||||
seed 20260924, 12 rounds
|
||||
| round | action | accident |
|
||||
|---|---|---|
|
||||
| 1 | oom_app | power_cut |
|
||||
| 2 | install | docker_restart |
|
||||
| 3 | cutoff_undo_file_app | power_cut |
|
||||
| 4 | failing_step | controller_killed |
|
||||
| 5 | remove_held | none |
|
||||
| 6 | failing_step | whole_box_backup |
|
||||
| 7 | crash_loop | whole_box_backup |
|
||||
| 8 | oom_app | disk_to_floor_plus_1g |
|
||||
| 9 | remove_held | controller_killed |
|
||||
| 10 | ladder_step | docker_restart |
|
||||
| 11 | cutoff_undo_file_app | disk_to_floor_plus_1g |
|
||||
| 12 | start_after_unhealthy | none |
|
||||
|
||||
**The app each round acts on — fixed here, not drawn** (an app's state after round N decides what N+1 can do):
|
||||
|
||||
| round | app | why |
|
||||
|---|---|---|
|
||||
| 1 oom_app | `chaosoom` (drill-only: alpine, 32M cap, a 200 MB eater in a loop) | out-of-memory storm → the box must stop it |
|
||||
| 2 install | n8n at its ladder's FIRST `from` (2 steps behind), seeded through its front door | gives round 10 a step |
|
||||
| 3 cutoff_undo_file_app | nextcloud (the file app), a failing step (redis 7-alpine → 7.4.0-alpine, probe 8999), the NEWEST undo copy's marker cut at verifying → the hold must name the second drive → its whole restore, then the seeds + files read back | decision 26 under a power cut |
|
||||
| 4 failing_step | wishlist v0.67.1 → `:latest` with probe 8999 → undo | the undo under a controller kill |
|
||||
| 5 remove_held | gokapi (held since A3: the box stopped its crash loop) — Remove with data | Remove on a held app |
|
||||
| 6 failing_step | navidrome 0.64.1 → `:latest` with probe 8999 → undo | the undo under a backup run |
|
||||
| 7 crash_loop | `chaoscrash` (drill-only: alpine that exits after 3 s) | the box must stop it |
|
||||
| 8 oom_app | `chaosoomb` (a second eater) | the OOM stop under a full disk |
|
||||
| 9 remove_held | `chaoscrash` (held since round 7) | Remove on a held app under a controller kill |
|
||||
| 10 ladder_step | n8n, one step (2 → 1) | a good step under a docker restart |
|
||||
| 11 cutoff_undo_file_app | nextcloud again | decision 26 with the disk 1 GB above the floor (a `disk` refusal is a valid outcome) |
|
||||
| 12 start_after_unhealthy | `chaosoomb` (held since round 8) — Start → one more try → the repeat stop says support is told | decision 28's second half |
|
||||
|
||||
**Changed before round 1:** the accident "a whole-box backup started on 9202" is the box's own full backup run
|
||||
(`POST /api/backup/run`, as on 2026-09-23), not a vzdump: demo-hp's backup storage had ~4 GB free and the host
|
||||
root was at 90 %, so a vzdump of 9202 could have filled the HOST (demo-hp also had R-672's full pool today).
|
||||
**How accidents are fired:** the runner arms the round and waits; the session fires each host-level accident
|
||||
as its own visible command (power cut = `pct stop 9202; pct start 9202`, kill -9 of the controller PID,
|
||||
`systemctl restart docker` in the guest, a `fallocate` to leave 3 GB free on `/var/lib/docker`).
|
||||
|
||||
|
||||
### The rounds
|
||||
|
||||
| # | action × accident | app | steady | end state | data read back | events (from the box's own log) |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | oom_app × power cut | chaosoom | 185 s | stopped, `unhealthy_stop` | — | app_oom (w), app_oom_storm (e), app_stopped_unhealthy (w) |
|
||||
| 2 | install × docker restart | n8n | — | **not installed, silently** (R-681); reinstalled after, 211 s | — | app_deploy_started only |
|
||||
| 3 | cutoff file app × power cut at verifying | nextcloud | 204 s | the update RESUMED after boot → held (second drive) → whole restore → running | account ✓ files 2/2 ✓ | app_update_held (e) |
|
||||
| 4 | failing step × controller kill | wishlist | 148 s | resumed → undone | ✓ | app_update_undone (w) |
|
||||
| 5 | remove held × none | gokapi | 27 s | removed, 0 leftovers | — | app_removed (i) |
|
||||
| 6 | failing step × backup run | navidrome | 110 s | undone (the backup skipped the updating app) | ✓ | app_update_undone (w) |
|
||||
| 7 | crash loop × backup run | chaoscrash | 116 s | stopped, `unhealthy_stop` | — | app_stopped_unhealthy (w) |
|
||||
| 8 | oom_app × disk to floor + 1 GB | chaosoomb | 102 s | stopped, `unhealthy_stop` | — | app_oom (w), app_stopped_unhealthy (w) |
|
||||
| 9 | remove held × controller kill | chaoscrash | — | **half-removed**: 502 to the household, containers gone, still listed (R-682); a second Remove finished it | — | — |
|
||||
| 10 | ladder step × docker restart | n8n | 155 s | resumed → done, 2.31.3 → 2.40.5, steps 2 → 1 | ✓ | — |
|
||||
| 11 | cutoff file app × disk to floor + 1 GB | nextcloud | 194 s | held (fresh 14:34 second-drive copy) → whole restore → running | account ✓ files 2/2 ✓ | app_update_held (e) |
|
||||
| 12 | Start after unhealthy × none | chaosoomb | 125 s | one more try → stopped again, „support is told" hu/en | — | app_oom (w), app_oom_storm (e), app_stopped_unhealthy (w) |
|
||||
|
||||
Controller kills: 2 of max 3, 20 min 32 s apart. **Against `08`:** every stop sent `app_stopped_unhealthy`;
|
||||
every undo `app_update_undone`, every hold `app_update_held`. **What should have fired and did not:** round 2
|
||||
(an install lost — nothing) and round 9 (a remove half-done — nothing); round 3's hold named an hour-old copy
|
||||
(R-683, watch). No user file was deleted or overwritten by any restore; no data was lost.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
- **Machine (9202):** tonight's 8 apps removed through the product (0 containers, 0 volumes, 0 undo copies);
|
||||
the three drive folders the product kept (R-442 on a box without drive access) removed by name; backup
|
||||
window back to 02:30 through the product's form; `controller.yaml` restored from `.pre-night0924` (live
|
||||
catalog, no drill `update:` settings); controller stays on 0.269.1 (the floor). Standing before tonight and
|
||||
kept: privatebin, paperless-ngx, filebrowser. gokapi removed in round 5 (R-644's crash loop). `F/10-12`.
|
||||
- **Host (demo-hp):** `pct list` 9201 + 9202 only (as at the start, minus R-672's leaked 990000); no bench
|
||||
was created; storage unchanged; the thin pool at 58.9 % after intervention 1. `F/20`.
|
||||
- **Hub:** floor 0.269.1 with MinAgent 0.131.0, read back; N100 9201 arrived in ~16 s (`managed floor SERVED
|
||||
… from declared`); **demo-hp 9201 did NOT arrive** (read-only disks, R-672). Drill repo reset to live
|
||||
`cf7cf8456f00`, `has_actions: false`. `F/13`, `F/30-31`.
|
||||
|
||||
## Claims in the brief that turned out wrong (or right), named
|
||||
|
||||
1. *The Tier-2 mirror holds a file app's files in a form a restore can read file by file* — **TRUE**
|
||||
(plain files under `hdd/` and `userdata/`; the merge walked 139).
|
||||
2. *`RestartCount` survives the scans that reset `restarting_since`* — **TRUE per container run**; it resets
|
||||
when the container is recreated (a Start after the box's stop), so the detector treats a drop as a new window.
|
||||
3. *10 restarts in 10 minutes separates a crash loop from a slow start* — **WRONG**: a steady loop runs at
|
||||
~1 restart/min (gokapi 7 in 7 min) and would never reach 10; 6 chosen. No healthy first start in the
|
||||
drill evidence reached 6 (immich's broken first start reached 12 — R-676).
|
||||
4. *Docker Compose accepts `tag@digest` in every template's image line* — **TRUE** (25/25). What the brief
|
||||
did not foresee: rendering digests in the SYNC moved a running app's image (fixed v0.269.1).
|
||||
5. *The chain signal can be measured on 9202* — **WRONG**: 9202 has no off-site target.
|
||||
6. *One release per repo* could not hold (see "Not done").
|
||||
|
||||
## Register
|
||||
|
||||
Before **335 rows / 679,393 B**; after **344 rows / 685,662 B**. Opened R-668…R-683 (16); closed R-661,
|
||||
R-662, R-664, R-665, R-666, R-667, R-668 (7, compressed into CLOSED-ITEMS).
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
# DRILL — night 2026-09-25: the fixed agent delivered, the HP box backs up again, automatic updates built, proven, and watched through the demo boxes' first real night
|
||||
|
||||
Brief: "NIGHT SHIFT 2026-09-24/25". Architecture read before any claim: `09-update-architecture.md` §3 decisions
|
||||
11–30 and §6.4.2 (the spec for Part B), `03-host-agent.md`, `07-backup-architecture.md`, `08-alarm-ladder.md`,
|
||||
`04-control-plane-authorization.md` §2–§3.1. Evidence: `audits/night-2026-09-25/` (A, B, C, D, E, F, G, tools).
|
||||
|
||||
## Not done, or changed
|
||||
|
||||
- **An unplanned whole-box backup of demo-hp 9201 ran for 5 min 16 s (22:40:21–22:45:37 CEST) and was aborted by
|
||||
me.** It came from my own live test of agent v0.134.0 (R-685), which was meant to prove a REFUSAL: the first build
|
||||
read free space from `GET /storage`, which carries no usage, so the check failed open and a real vzdump started.
|
||||
I stopped the task; vzdump removed its temporary files; no archive, no lock; root disk back to 60 %; the guest was
|
||||
suspended 2 s (snapshot mode), its apps kept running. This breached the brief's fence "no whole-box backup on a
|
||||
demo box except Part A4". Fixed before release (free space from `GET /nodes/<node>/storage`), red-proofed twice,
|
||||
and re-proven with safe builds that stop before any vzdump (`F/F1-*`, `F/F2-*`).
|
||||
- **The floor was raised at 22:59 CEST, not "around 02:00".** Part C had passed; raising early gave 3.5 hours to
|
||||
see both boxes arrive and read their switch and window before the night. Both arrived in ~15 s.
|
||||
- **Part F ran BEFORE Part D, and its controller half is not done.** Agent v0.134.0 (the space check) was released,
|
||||
signed and delivered to both demo boxes (Part F's live proof passed). The controller's backup-page line for a space
|
||||
skip needs a controller release, and v0.271.0 was already the night's one — R-685 stays open for that half.
|
||||
**R-671, R-670, R-677 were not started** — each needs a controller release (one per repo per night).
|
||||
- **Both agent_update jobs of 0.134.0 were queued within a minute**, not one box after the other's commit (each is
|
||||
per-box by signature; the 0.132.0 delivery did the same). 0.133.0 went strictly one after the other.
|
||||
- **Part C's accidents: two ran live, two could not** on a scratch box without an off-site tier and an agent — the
|
||||
off-site leg FAILING and W+5h reached with steps left are unit-proven only (R-687). "The controller killed BETWEEN
|
||||
two apps" landed INSIDE the next app's step: the leg presses the next app in the same second (see Part C).
|
||||
- **Part D: no whole-box backup followed the leg tonight, so the backup gate never had to wait.** demo-hp's last
|
||||
whole-box backup was Part A4's (21:59), so it is not due until this evening; demo-felhom's is due ~07:27, after the
|
||||
session. Both legs ended by 04:19, before the gate opened at 04:30. The gate's wait is unit-proven and red-proofed
|
||||
only (R-687 updated).
|
||||
- **One step per app per night** (decision 33) follows the brief's own words; §6.4.2 (a)'s "rescan between steps"
|
||||
read as climbing several — recorded as a decision the operator may widen.
|
||||
|
||||
**Interventions: 0** on product behaviour during the real night (nothing was touched on the demo boxes between the
|
||||
floor raise and the end of the night except to read). **The demo boxes' real night: demo-felhom 1 step done
|
||||
(opengist 1.13 → 1.15, 20 s), demo-hp nothing to do (every app current); 0 undone, 0 held, 0 skipped.** **The one
|
||||
result that matters most:** the automatic leg ran unattended on both demo boxes exactly where the chain puts it —
|
||||
after the off-site copy — pressed only the one tested step waiting, and reported itself to the hub.
|
||||
|
||||
|
||||
## Part B — controller v0.271.0 (`9cf13a3`, CI job 974 success)
|
||||
|
||||
Built exactly §6.4.2 point 6 (a)–(d) plus R-680, R-678, decision 13's marks, the older-than-ladder rule, the switch
|
||||
and the events rule. As built: `stacks/unattended.go` (`RunUpdateLeg`), `chainUpdateLeg` in `cmd/controller/main.go`
|
||||
(the `offbox-backup` job), `quiesce.Options.UpdateLegFn`, `backupwindow.UpdateLegStopOffsetMin`, `settings.json`
|
||||
`app_update.unattended` (absent = ON) with its card on /settings, app.yaml `failed_update_step` and
|
||||
`last_auto_update`, the report's `update_leg`, `backup.FreshWholeCopy`; `stacks.update_window` removed. Decisions
|
||||
31–33 taken unattended (`09` §3). **Red-proofs — 11, each seen failing, tree restored after each**
|
||||
(`night-2026-09-25/B/redproofs/`): the leg on every off-site path (a deferred call: error and panic cases fail without
|
||||
it); the gate deferring while the leg runs and not at W+5h; the leg starting nothing at W+5h; one step per app a
|
||||
night; the failed step skipped; `needs_person` skipped; `files_may_change` without a whole copy skipped; no crash-loop
|
||||
verdict during a step (the first version of this test passed under the mutation — its container was set before the
|
||||
leg's rescan rebuilt the stack; fixed and re-seen red); the steps-left count fresh at `done`; the switch ON by
|
||||
default; the leg wired at startup. Gates green; one parity snapshot regenerated for the new card (diff: additions only).
|
||||
|
||||
## Part A — the agent, the restore test, the HP backup
|
||||
|
||||
**A1 — agent v0.133.0 signed and delivered, one box after the other** (ruling 1, 2026-09-16). Package sha256
|
||||
`3aa30345…e69b6` verified against the published package before signing (`A/A1-*.txt`).
|
||||
|
||||
| box | signed (UTC) | box took it | AUTHORIZED → COMPLETED | committed | hub reads |
|
||||
|---|---|---|---|---|---|
|
||||
| demo-felhom-8363b5 | 19:25:30 | 21:34:05 CEST | job `c4b592ff7c9ff307` | 21:35:09 | 0.133.0 |
|
||||
| demo-hp-bb76ea | 19:35:55 | 21:48:58 CEST | job `36aab1ed8ad00979` | 21:50:02 | 0.133.0 |
|
||||
|
||||
Peti's box: never signed for.
|
||||
|
||||
**A2 — the restore test back on, both boxes.** `agent.json.pre-r672` copied back; the diff was only
|
||||
`restore_test_eval_interval_seconds: -1` (and the trailing newline). The `-1` config is kept as
|
||||
`agent.json.night-0925-off`. Start-up line on both: `restore-test scheduler starting (per-archive due-check)
|
||||
eval_interval=6h0m0s settle=24h0m0s`.
|
||||
|
||||
**A3 — one on-demand restore test.** demo-felhom: **PASS** — scratch 990000 restored, booted, verified,
|
||||
torn down in 1 min 25 s; pool 3.13 % → peak 5.68 % → 3.13 %; preflight "required 12.8 GB, avail 362.8 GB".
|
||||
demo-hp: **REFUSED, correctly** — "restoring 21.1 GiB (vzdump log: total bytes written) needs 30.3 GiB free, has
|
||||
22.1 GiB", exit 4, pool 59.01 % before and after, nothing created.
|
||||
|
||||
**A4 — HP whole-box backups, option A.** Before: root disk (`local`) 89 % used, 4.3 GB free, three 9201 archives
|
||||
(6.2, 6.3, 7.4 GB), retention 3. Changed: `local_backup_retention` 3 → 1 (saved copy
|
||||
`agent.json.pre-a4-retention`), agent restarted, the two oldest archives removed with PVE's own
|
||||
`pvesm prune-backups local --vmid 9201 --keep-last 1` (dry-run shown first) → 58 % used, 16.8 GB free. **Why the
|
||||
prune by hand:** PVE prunes AFTER a successful backup, so lowering the retention alone would not have let the next
|
||||
backup fit. One whole-box backup by the product's own button (`POST /api/guest-backup/trigger`, the Mentések
|
||||
page) — 9201's apps were stopped for it (snapshot mode, resumed at "snapshotted"): local archive **8.18 GB** in
|
||||
7 min, then the PBS tier (the button covers every tier) 23.2 GB in 4.5 min; the new prune kept 1 archive; root
|
||||
60 %; all 20 app containers healthy. The product row is **R-685**.
|
||||
## Part C — the proof on 9202: six simulated nights
|
||||
|
||||
Controller v0.271.0 on 9202 only (`C/C0-deploy-9202.txt`), drill catalog (`C/C0-repoint-drill.txt`, health timeout
|
||||
90 s). Four apps installed through the product at the ladder's first `from`, then the drill put back at the head:
|
||||
wishlist 1 step behind, navidrome 2, romm 3, vikunja 1 with a failing step (its head probe on port 8999). Marks
|
||||
set in the drill: navidrome step 1 `needs_person`, wishlist's step `files_may_change`. Data seeded through each
|
||||
app's own front door and read back after EVERY night (`C/data-readback.txt`: 8 of 8 rounds, 4 of 4 apps, each
|
||||
fixture with its negative control). The window was moved through the product's own form (`POST /backups/window`)
|
||||
so the off-site job — and the leg chained to it — fired two minutes later.
|
||||
|
||||
| night | what it proved | leg summary (the box's own line) | pages (hu + en) |
|
||||
|---|---|---|---|
|
||||
| 1 | the chain on a box with NO off-site target ("the off-site leg does nothing; the update leg runs now"); one step per app; `needs_person` skipped; `files_may_change` taken WITH a whole copy (wishlist); a failing step undone | `done=2 undone=1 held=0 failed=0 skipped=1 in 3m6s [skipped: navidrome=needs_person]` — romm 60.1 s, vikunja undone 105.2 s, wishlist 20.1 s | „Automatikus frissítés 2026-09-24 22:31-kor — sikeres." / "Automatic update at … — done."; vikunja's undone sentence in both |
|
||||
| 2 | R-680 live: the undone step is NOT pressed again; navidrome's `files_may_change` step taken (its music folder is `excluded`, so its own unit is whole) | `done=2 undone=0 … skipped=1 in 1m10s [skipped: vikunja=failed_before]` | badges current / behind as expected |
|
||||
| 3 — accident | controller killed (kill -9) the moment navidrome's step ended | the leg had pressed romm IN THE SAME SECOND; the kill landed before romm's `up` → pin and definition put back, romm runs its previous version, page: „A frissítés megszakadt, mert a vezérlő újraindult…" / "The update was interrupted…"; the leg was not resumed (R-686) | all data read back |
|
||||
| 4 — accident | power cut (`pct stop`/`pct start` of 9202) while romm's step was `verifying` | box up in 13 s; "resumed 1 interrupted update(s)"; romm healthy after 30 s, DONE in 1 m 32 s; no automatic-update line on the page (the leg died — R-686) | all data read back |
|
||||
| 5 | the switch OFF (`POST /settings/app-update`, page reads unchecked) | `done=0 … stopped=switched_off in 0s` — nothing pressed | — |
|
||||
| 6 | the switch ON again; the catalog RE-TESTED vikunja's step (probe fixed, new `tested_at`) → the ladder print changed → R-680's record no longer binds | `done=1 … in 10s` — vikunja done | „Automatikus frissítés … 22:58-kor — sikeres." |
|
||||
|
||||
**Not run live, and why (R-687):** W+5h reached with steps left (the leg starts at W+105m; needs a 3-hour leg — unit
|
||||
test `TestLeg_NoStepAtOrAfterW5h`); the off-site leg FAILING (9202 has no off-site tier — unit test
|
||||
`TestChainUpdateLeg_EveryPath` covers error and panic); a `files_may_change` step WITHOUT a whole copy (every app
|
||||
given the mark was whole on 9202); the full-system gate's deferral (9202 has no agent, so no quiesce loop).
|
||||
|
||||
**Verdict: Part C passed** — every accident that could run on 9202 ended with the app healthy, the data read back,
|
||||
and a true sentence on its page; the four not-runnable cases are named, unit-proven and in the register.
|
||||
|
||||
## Part D — the demo boxes' first real automatic night
|
||||
|
||||
Floor 0.271.0 (declared MinAgent 0.131.0) saved 20:59:04Z, read back; both boxes on 0.271.0 ~15 s later
|
||||
(`D/D1`, `D/D2`). Before the night, per box (`D/D3`, `D/D0`): switch ON (key absent = the default; demo-hp's
|
||||
settings page reads it checked in both languages), window 02:30 (off-site + update leg at 04:15, gate [04:30, 08:30)).
|
||||
Tested steps waiting: **demo-felhom — opengist 1.13 → 1.15; demo-hp — none** (all 10 apps at the catalog head).
|
||||
Agents: 0.134.0 on both (Part F). Nothing was touched on either box during the night except to read.
|
||||
|
||||
**demo-felhom (N100), leg by leg** (`D/D6-demo-felhom-night-full.log`): db-dump 02:30:00 (0.8 s) → Tier 2 03:30:00
|
||||
→ off-site 04:15:00–04:15:46 (1 app, 11 snapshots, 42 s) → **update leg 04:15:46–04:16:06: opengist pressed; safety
|
||||
dump, undo copy (0.2 MiB), pin, pull, start, verify healthy after 10 s, DONE in 17 s; `done=1 undone=0 held=0 failed=0
|
||||
skipped=0 in 20s`**. After: container `opengist:1.15` healthy; app.yaml `last_auto_update: done 1.13 → 1.15` (the page
|
||||
line). One WARN worth keeping: the volume "carries no compose label (recreated by a restore before v0.268.0 — R-658)
|
||||
— copied by name" — the R-658 fallback working. No mail (a successful step sends none).
|
||||
|
||||
**demo-hp** (`D/D6-demo-hp-night-full.log`): db-dump 02:30:00–02:31:59 → Tier 2 03:30:00–03:30:24 → off-site
|
||||
04:15:00–04:18:41 (9 apps, 90 snapshots, 3 min 35 s, OK) → **update leg 04:18:41: nothing to press, `done=0 … in
|
||||
0s`**.
|
||||
|
||||
**The hub** holds both summaries in its stored reports (received 02:44Z; `D/D7-hub-report-update-leg.txt`, read from a
|
||||
copy of the hub DB taken with its -wal, then deleted). **Whole-box backups:** none ran tonight on either box — not
|
||||
due (see "Not done"); both agents alive (627 / 1,069 journal lines since 02:25, the 10-minute janitor on schedule).
|
||||
**No app ended held or stopped.**
|
||||
|
||||
## Part E — two apps moved
|
||||
|
||||
Venue: bench LXC **9401** on demo-hp (created and destroyed tonight — no permission check refused either; the
|
||||
Debian template it needed was removed again), harness v3; box walk on 9202 through the real guarded Update.
|
||||
**Negative control: `n8n=alpine:3.20` → `failed`.** A bench setup miss is recorded: the first queue ran without
|
||||
the `traefik-public` network and returned `inconclusive — FROM deploy failed` (kept as `*-noNetwork`); re-run.
|
||||
|
||||
| app | step | bench | box | memory peak (own) | catalog commit (CI) |
|
||||
|---|---|---|---|---|---|
|
||||
| n8n | 2.41.1 → 2.41.2 | proven | proven | 22.4 % | `f14a608` (job 980) |
|
||||
| mealie | v3.27.0 → v3.28.0 | proven | proven | 23.1 % | `b996218` (job 981) |
|
||||
|
||||
Published at 02:31–02:32, after the demo boxes' night began and on neither box's app list. Not moved, with the
|
||||
reason for each: catalog `REPORT.md` (drift `E/E0-drift.log`).
|
||||
|
||||
## Part F — agent v0.134.0 (R-685 agent half)
|
||||
|
||||
Released by `scripts/release-agent.sh` (tag `v0.134.0` at `0722b2c`, sha256 `7593bebe…81c72d`, verified by download;
|
||||
CI jobs 975–977). A vzdump to a local target needs free ≥ newest archive × 1.25 + 1 GiB; a shortfall is a named skip
|
||||
before anything starts; fail-open on PBS / first backup / unknown usage. Live, safe builds (a hard stop before vzdump):
|
||||
×10 → "local has 14.9 GiB free; … needs about 77.2 GiB" refused; ×1.25 → "space preflight passed" need 11.3 GB,
|
||||
avail 16.0 GB. Signed to both demo boxes (`F/F3-*`): demo-felhom committed 22:52:45, demo-hp 22:58:53; hub reads 0.134.0.
|
||||
The incident above belongs to this part.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
- **Machine:** 9202 back on the live catalog (`catalog-cache` at `c802509` at the time; the controller follows `main`),
|
||||
window 02:30 again, the six drill apps removed through the product (0 containers, 0 volumes, 0 undo copies); left by
|
||||
name on the scratch drive: `userdata/navidrome`, `userdata/romm` (the remove kept drive data — R-442's fail-closed
|
||||
path on 9202). The switch key on 9202 is now explicit `true` (was absent = ON). 9202 stays on controller 0.271.0 (the
|
||||
floor). Bench 9401 destroyed (hostname checked first), template removed.
|
||||
- **Host:** demo-hp `pct list` 9201 + 9202, local-lvm 60.05 %, root 60 %; demo-felhom 9201 only, local-lvm 3.16 %.
|
||||
Config changes kept on purpose: restore test ON (both), `local_backup_retention: 1` (demo-hp). No test binaries left
|
||||
in /tmp.
|
||||
- **Hub:** global floor 0.271.0 (MinAgent 0.131.0); both demo boxes 0.271.0 / agent 0.134.0. Drill repo reset to live
|
||||
`main` (`dd9c6ad`), private, Actions off.
|
||||
|
||||
## Claims in the brief that turned out wrong (or right)
|
||||
|
||||
1. *The off-site job function is the single place the leg can be chained from on every path* — **TRUE for every path
|
||||
of the job, with one exception outside it:** a box with backups disabled registers no off-site job at all, so no
|
||||
leg (the update itself would refuse `no_backup` there anyway). The job's own early returns and the scheduler's "not
|
||||
configured" skip all return into `chainUpdateLeg`. A controller restart after the job fired loses the rest of the
|
||||
night (R-686).
|
||||
2. *A demo box has a tested step waiting tonight* — **TRUE for demo-felhom (opengist), FALSE for demo-hp.**
|
||||
3. *The whole-box backup lands on demo-hp's root disk* — **TRUE** (`local` = `/var/lib/vz`, root LV).
|
||||
4. *Decision 28's suppressions cover a deploy's first start* — **WRONG.** `Deploying` clears when `compose up -d`
|
||||
returns; a first start that restarts ≥ 6 times in 10 min is stopped (R-676 updated). An automatic step, its verify
|
||||
and its undo ARE covered (pinned by a test).
|
||||
|
||||
## Register
|
||||
|
||||
Before: **341 rows / 685,148 B**. After: **338 rows / 682,053 B**. Opened R-685 (backup-fit warning; agent half
|
||||
shipped, controller page half open), R-686 (no resume of the leg after a restart), R-687 (live-proof gaps). Closed
|
||||
R-672, R-673 (delivered), R-684 (option A), R-680, R-678, R-643. Updated R-676 (deploy first start), R-450 (narrowed
|
||||
to part 10).
|
||||
@@ -0,0 +1,129 @@
|
||||
# DRILL — 2026-09-25 evening → 26 night: the box converts a PostgreSQL major (docmost first); kept data; the night watch
|
||||
|
||||
Brief: "the box converts a PostgreSQL major for its first app (docmost); a reinstall over kept data asks the household…".
|
||||
Architecture read before any claim: `09-update-architecture.md` §3 decisions 13–36, §3b Q5, §6.1a, §6.4 part 10;
|
||||
`07-backup-architecture.md` §6; `08-alarm-ladder.md`. Evidence: `audits/night-2026-09-26/` (part0, A, B, C, D, E, F, G, tools).
|
||||
|
||||
## Not done, or changed
|
||||
|
||||
- **Two controller releases, not one** — as the brief allowed: **v0.273.0** (the conversion; floor raised at 11:32Z so
|
||||
the live catalog could move docmost before the night) and **v0.274.0** (kept data; 12:30Z).
|
||||
- **Part E was built by a parallel helper session** in its own git worktree (the brief's parts B–D and E touch
|
||||
different files); I reviewed it, found one defect live (R-692) and fixed it before the release, and ran the whole
|
||||
E5 proof myself.
|
||||
- **The live proofs of Part D ran on `0.273.0-rc1`**, not the released image: the release adds only the recovery line's
|
||||
wording (found live in case (c)) and R-687's two small items.
|
||||
- **docmost moved with its own memory limit raised 384M → 512M** (decision 39): the bench marked the step
|
||||
`memory_tight`, and the move gate refuses a tight step whose limit does not move. The mark stays at 512M (80.4 %) —
|
||||
Node sizes its heap from the limit (R-693).
|
||||
- **The bench was a throwaway LXC 9401 on demo-hp**, created and destroyed tonight (the first create picked an ARM
|
||||
template and failed to start; destroyed, name checked first, recreated with amd64). 9202 could not be the bench:
|
||||
the harness's container names collide with the installed docmost and it ends with `down -v`.
|
||||
- **Part F3 (adventurelog, R-655): measured, not moved.** The answer to "skip or pre-seed download-countries" is in the
|
||||
row; the move needs a choice between two options with different costs.
|
||||
- **My own tool error in Part A4:** a bare `docker compose up -d` in docmost's stack dir (no `.env` there — the controller
|
||||
passes the env) started docmost without its secrets; it crash-looped and **the box stopped it after 7 restarts
|
||||
(decision 28 working)**. Started again with the product's Start; the seed read back.
|
||||
- **The kept pre-conversion copy was released too early on demo-hp (R-696, found in Part G):** the release trusted the
|
||||
unit's refresh time, and the unit's dump was from PostgreSQL 16. Low harm tonight; not fixed (a third release).
|
||||
|
||||
## Claims in the brief, judged
|
||||
|
||||
1. *`postgres:18` moved its datadir and refuses or warns at `/var/lib/postgresql/data`* — **TRUE, and stronger:** it
|
||||
REFUSES (exit 1) even an EMPTY volume there (`A/A1-images.txt`).
|
||||
2. *The safety dump is `pg_dump` per database with `--no-owner --no-privileges`, not `pg_dumpall`* — **TRUE**. For docmost
|
||||
both routes rebuild an identical database (one role, one database); the box uses `pg_dumpall` (decision 38).
|
||||
3. *`Restore` brings the old datadir back so the old engine starts* — **TRUE**, by hand (A4) and by the product three
|
||||
times (Part D cases a–c).
|
||||
4. *The R-487 restore of a removed app picks up the kept drive files* — **FALSE on 0.272.0**: nextcloud came back with no
|
||||
env, no database, its files mounted on the guest's root disk — the unit on the DATA drive was never found (R-690,
|
||||
fixed in v0.274.0, proven live).
|
||||
5. *nextcloud needs a rescan for files newer than its backup* — **TRUE** (`occ files:scan --all`, 0.6 s); now the
|
||||
template's `after_load:`.
|
||||
6. *Only demo-hp runs docmost* — **TRUE**.
|
||||
- Also measured: R-657's install LOOP did not recur on 0.272.0 — the install succeeded and the old files were silently
|
||||
unreachable; the choice replaces both shapes.
|
||||
|
||||
## Part 0 — `part0/README.md`
|
||||
|
||||
Decisions 35, 36 recorded; D3 in STATUS. Last night: demo-felhom's leg did opengist in 20 s at 04:15; its whole-box
|
||||
backup ran at 07:29, three hours after the leg — nothing waited; demo-hp's leg had nothing and reported
|
||||
`"steps": null` (fixed); no whole-box backup was due there. **Found: R-689** — demo-hp's restore test picks the golden
|
||||
template as "the newest settled archive" and fails every 6 h.
|
||||
|
||||
## Part A — the spike (`A/README.md`)
|
||||
|
||||
Target **18** (decision 37); load from **`pg_dumpall`** with zero tolerated errors (decision 38); the check (owners,
|
||||
encodings, roles, extensions, per-table rows) in 0.5 s; the undo after emptying proven by hand; space: the dump is
|
||||
0.2 % of the datadir — the box's bound is the volume's size × 1.25 + the 2 GB floor.
|
||||
|
||||
## Part B — controller v0.273.0 (`B/`)
|
||||
|
||||
`internal/stacks/pgconvert.go`; 17 tests; **9 red-proofs** each seen failing (mark missing → refused; marker re-checked
|
||||
before emptying; counts compared; `PG_VERSION` checked; a cut-off dump; a restart during `converting` undone; the old
|
||||
copy kept; the release wired; the space check).
|
||||
|
||||
## Part C — the catalog (`C/`)
|
||||
|
||||
Harness v4 (converts on the bench, moves an 18 mount, writes the mark only when both venues converted); the engine
|
||||
gate's proof clause with 5 decoys + a red-proof; the postgis family judged. Bench: negative control C3 `failed`;
|
||||
docmost run 1 (384M) converted in 25.0 s, proven, `memory_tight` 90.9 %; run 2 (512M) converted in 11.0 s, proven,
|
||||
80.4 %; abort (16 images on the 18 datadir) refuses, as expected — the product's route back is the undo, not that.
|
||||
|
||||
## Part D — the live proof, the floor, the move (`D/`)
|
||||
|
||||
| case (9202, drill catalog, `0.273.0-rc1`) | phases | end | PG_VERSION after | seed |
|
||||
|---|---|---|---|---|
|
||||
| happy path | safety-dump → pulling → copying → **converting 10 s** → starting → verifying | **done, 42.1 s** | 18 | read back |
|
||||
| (a) the load fails (`adminpack`, gone in 17+) | … → converting → undoing | **undone, 43.6 s** | 16 | read back |
|
||||
| (b) unhealthy on 18 (drill probe :3999) | … converting → starting → verifying (90 s) → undoing | **undone, 148.1 s** | 16 | read back |
|
||||
| (c) SIGKILL 1 s after the volume was emptied | … converting → (restart) → undoing | **undone, 40 s after the restart** | 16 | read back |
|
||||
|
||||
Floor 0.273.0 (then 0.274.0) read back from the hub; both demo boxes healthy within ~25 s. **docmost moved in the live
|
||||
catalog at ~14:40 CEST** (`afd3a60`): the engine gate printed `ALLOWED … proven on both venues and carries the
|
||||
conversion mark`; the Hungarian copy gate caught the RAM line's change on the first push (freeze updated on purpose).
|
||||
|
||||
## Part E — kept data (`E/`)
|
||||
|
||||
E1 answers above (claims 4, 5). Built: the install choice (409 `kept_data_choice`), the „Megőrzött adatok" page, the
|
||||
read-only view, the drive-full naming, `after_load:`. **E5 on 9202 (`0.274.0-rc2`), both languages, all passed:** ask
|
||||
(use off without a database copy) → start fresh (126 MB renamed to `kept/nextcloud/<date>/`) → seed, a file, a unit, a
|
||||
file after it → remove keeping data → **use** (loaded from the unit on the DATA drive; rescan ran; account + both
|
||||
files back; a never-written file stays invisible) → start fresh again → Load refused while a leftover occupies the
|
||||
folder → Delete refused for a wrong name and for an unlisted path → Delete → **Load** (files moved back, database
|
||||
loaded, rescan; account + both files back) → a write into the view: `Read-only file system`. **R-692 found and fixed
|
||||
live** (two leftovers named „Filebrowser").
|
||||
|
||||
## Part F
|
||||
|
||||
F1 (R-687 `[]` + the taken step's log): v0.273.0, 2 red-proofs. F2 (R-688): hub v0.125.0 — the delete dialog lists the
|
||||
Cloudflare items to remove by hand; live preview read for demo-hp. F3 (R-655): measured, not moved (see the row).
|
||||
|
||||
## Part G — the night watch
|
||||
|
||||
Read-only on both demo boxes (`G/`). **demo-hp converted docmost BY ITSELF in the automatic leg:** off-site copy
|
||||
04:14–04:18 → leg 04:18:25–04:19:20 CEST → `CONVERTED 16 → 18 in 9.898s — the check is equal (2 database(s), 50
|
||||
table(s), 73 row(s)), PG_VERSION 18` → `DONE in 54s`; hub summary `done=1`, `steps: [docmost 55.1 s]`. Content read back
|
||||
(read-only SELECTs, before 12:41Z / after 02:40Z): users 1 = 1, workspaces 1 = 1, spaces 1 = 1, pages 4 = 4;
|
||||
`PG_VERSION` 16 → 18; front door 200; all three containers healthy. **The whole-box backup** was due since the
|
||||
evening and held for its window [04:30, 08:30); it started at 04:30:45, eleven minutes after the leg ended — nothing had
|
||||
to wait (R-687 item 4 still unobserved). demo-felhom: db-dump 02:30, Tier 2 03:30, off-site 04:15, leg nothing to do —
|
||||
hub reads `"steps": []` (**R-687's fix, live**). **9202's release (B5) worked:** 13:30:39Z, one hour after a backup whose
|
||||
dump (11:57) came after the conversion (11:13). **demo-hp's release was premature (R-696):** at 02:30:16Z it removed the
|
||||
16 copy citing "Tier 1 at 02:20:16Z" — the unit's manifest refresh; its `docmost-postgres.sql` is 02:15:01Z and says
|
||||
`Dumped from database version 16.15`.
|
||||
|
||||
## Teardown
|
||||
|
||||
- **Machine:** 9202 — docmost and nextcloud removed through the product; this session's kept folders deleted through
|
||||
the Kept-data page (the older romm/paperless leftovers stay as found); back on the LIVE catalog (`controller.yaml`
|
||||
from the saved copy), controller 0.274.0; one empty folder recreated by the file browser's stale bind removed by hand
|
||||
after the loop was understood (R-695). The drill repo reset to live `main`. demo boxes: nothing touched but the two
|
||||
floors (read-only otherwise); docmost on demo-hp now runs PostgreSQL 18 — the intended outcome.
|
||||
- **Host:** bench LXC 9401 destroyed (name checked), its template removed; demo-hp `pct list` 9201 + 9202 only.
|
||||
- **Hub:** global floor 0.274.0 (MinAgent 0.131.0); hub 0.125.0; no customer or appliance created.
|
||||
|
||||
## Register
|
||||
|
||||
Before: **334 rows / 674,532 B**. Opened R-689, R-691 (helper), R-693, R-694; opened and closed R-690, R-692; closed
|
||||
R-657; opened R-695 (teardown finding) and R-696 (the night watch); narrowed R-450, R-463, R-469, R-687, R-688, R-691; R-655 measured. After: **339 rows / 682,159 B**.
|
||||
@@ -0,0 +1,94 @@
|
||||
# THE TWENTY-EIGHT — every app no update drill had ever touched, 2026-09-22
|
||||
|
||||
Evidence: `the-28-2026-09-22/`. What came before: `DRILL-update-night-2026-09-21.md` (21 edges,
|
||||
19 apps) and `PROBE-FIX-2026-09-22.md` (the probe fix and the fifteen moves).
|
||||
|
||||
---
|
||||
|
||||
## Not done, or changed from the brief
|
||||
|
||||
**Read this first.**
|
||||
|
||||
### Claims in the brief that turned out wrong
|
||||
|
||||
1. **There is no `requires:` key in `.felhom.yml`.** The brief said *"A `.felhom.yml` `requires:`
|
||||
(HDD, x86) that 9202 cannot meet → recorded, not forced."* No template has such a key. The
|
||||
constraints live under **`resources:`** as `needs_hdd` and `pi_compatible`. **And none of them
|
||||
excluded anything:** 9202 is `x86_64` with `/mnt/felhom-drives/scratch_hdd` mounted, so every one
|
||||
of the 28 was installable on those grounds. Nothing was skipped for a resource reason.
|
||||
|
||||
2. **The brief's file-leg list is wrong, and it told me where to check.** It grouped *"immich,
|
||||
jellyfin, plex, emby, komga, calibre-web, gokapi, homebox, gramps-web"* as file-leg apps. Read
|
||||
from `07-backup-architecture.md` §6.2 — which the brief itself says to read rather than re-derive
|
||||
— only **four of the 28 are class A** (at least one readable file leg): `calibre-web`, `immich`,
|
||||
`komga`, `paperless-ngx`. **`jellyfin`, `plex` and `emby` are class B precisely because their only
|
||||
bind is a `:ro` media mount**, which `ClassifyBinds` excludes; `gokapi`, `homebox` and
|
||||
`gramps-web` keep everything in named volumes. The table below uses `07` §6.2's classes, not the
|
||||
brief's.
|
||||
|
||||
3. **The brief's database list is incomplete.** It named calcom, claper, outline, paperless-ngx,
|
||||
rallly and sparkyfitness for PostgreSQL and kimai for MariaDB. **`immich` also carries PostgreSQL
|
||||
(a vectorchord variant) and redis, and `wanderer` carries meilisearch** — two apps the brief put
|
||||
in the "file-leg" group actually run their own datastore.
|
||||
|
||||
4. **One of the 28 cannot be installed at all, by design.** `plant-it` is `lifecycle: "abandoned"`,
|
||||
and the product's lifecycle gate refused the deploy with **409 „Ez az alkalmazás jelenleg nem
|
||||
telepíthető."** That is correct behaviour, measured live for the first time. It is the only
|
||||
lifecycle-gated template in the whole catalog of 53.
|
||||
|
||||
### Claims in the brief that were verified true, by looking
|
||||
|
||||
- **The drill repo's Actions are off (R-629).** Read from the API: `has_actions: false,
|
||||
private: true`. And measured rather than trusted: **47 CI jobs before the reset push, 47 after** —
|
||||
the push produced no run and no mail.
|
||||
- **`repoint_drill.py` still works after the catalog moved.** 9202's cache now reads
|
||||
`origin …/app-catalog-drill.git` at `1ad1f34`.
|
||||
- **9202 has the capacity.** 25.9 GB RAM (23.5 free), 28 GB free on `/`, 842 GB on the scratch
|
||||
drive. The root disk is the binding constraint, so **each app's images are removed by name after
|
||||
its verdict** — never `prune` (rule 3). Disk held at 1.9 GB used throughout.
|
||||
|
||||
### Changed method, named
|
||||
|
||||
5. **A restore that is REFUSED is recorded as `refused-with-a-sentence`, not as a failed restore.**
|
||||
The first version of the harness collapsed the two and mislabelled `calibre-web`. The product had
|
||||
in fact done the right thing — see the finding below — and a harness that calls a correct refusal
|
||||
a failure would have buried it.
|
||||
|
||||
6. **The harness now waits for a restore to settle before removing.** It did not at first, and that
|
||||
race produced **R-633**, a real defect. The race was left in the record for `gokapi` and fenced
|
||||
out afterwards, so the remaining apps measure the product rather than the harness.
|
||||
|
||||
|
||||
7. **Three bugs in tonight's own harness, each named with what it cost.** A missing `import re` in
|
||||
the restore step killed the restore half for `wanderer`, `claper`, `sparkyfitness` and `calcom`.
|
||||
A variable named `m` shadowed the app's metadata and broke `paperless-ngx`'s teardown. An empty
|
||||
phase list crashed on `[-1]` when the Update was **refused** before any phase existed, which cost
|
||||
`rallly` its whole walk. **All four apps, and five more, were re-walked serially afterwards** —
|
||||
and that re-walk is what corrected R-634 and produced `ghost`'s proof. An instrument that can
|
||||
drop results silently is not a measurement; these dropped them loudly and were re-run.
|
||||
|
||||
8. **Concurrency is part of the method and it changed two results.** Three walks ran at once to fit
|
||||
28 apps in one night. `POST /api/backup/run` is **box-wide**, so a second caller gets
|
||||
`409 „Mentés már folyamatban"`, and the Update refuses while a backup or restore is in flight
|
||||
(`409 busy`). **Both refusals are the product being right** and both are quoted below. But they
|
||||
cost `ghost` and `rallly` their edge on the first pass, and they are implicated in two of R-634's
|
||||
three instances. The nine re-walks were serial for exactly this reason.
|
||||
|
||||
|
||||
### Corrected 2026-09-22 (evening), from the records rather than by re-running
|
||||
|
||||
9. **`calibre-web`'s restore was `refused-with-a-sentence`, not `failed`.** The refusal is in this
|
||||
app's own `log.txt` line 12 as a `flash_error` on the redirect, and its state stayed `running`
|
||||
throughout. It was recorded `failed` because that walk ran **before** the refusal-capture code was
|
||||
added later the same night — the document's own section *"A restore that is REFUSED"* already said
|
||||
so while the table and the record disagreed with it.
|
||||
|
||||
10. **`calcom`'s restore was `inconclusive`, not `failed` — and that was my harness, not the
|
||||
product.** The restore was accepted (`flash=flash.restore.started`, no `flash_error`), the app
|
||||
read `running` at +45 s with no hold and no phase, and the classifier looked once more and saw
|
||||
**`starting`** — a settling state it did not list beside `running`/`unhealthy`, so it fell
|
||||
through to `failed`. **The task brief supposed a different cause** — "a read-back of data that
|
||||
was never seeded" — and that is wrong: the seed half is recorded separately and was already
|
||||
`no route`. The record's own `restore_state_seen: "starting"` is the evidence.
|
||||
|
||||
**Totals move with it:** restores correctly REFUSED go from 2 to **3**.
|
||||
@@ -0,0 +1,369 @@
|
||||
# THE TWENTY-EIGHT — every app no update drill had ever touched, 2026-09-22
|
||||
|
||||
Evidence: `the-28-2026-09-22/`. What came before: `DRILL-update-night-2026-09-21.md` (21 edges,
|
||||
19 apps) and `PROBE-FIX-2026-09-22.md` (the probe fix and the fifteen moves).
|
||||
|
||||
---
|
||||
|
||||
## Not done, or changed from the brief
|
||||
|
||||
**Read this first.**
|
||||
|
||||
### Claims in the brief that turned out wrong
|
||||
|
||||
1. **There is no `requires:` key in `.felhom.yml`.** The brief said *"A `.felhom.yml` `requires:`
|
||||
(HDD, x86) that 9202 cannot meet → recorded, not forced."* No template has such a key. The
|
||||
constraints live under **`resources:`** as `needs_hdd` and `pi_compatible`. **And none of them
|
||||
excluded anything:** 9202 is `x86_64` with `/mnt/felhom-drives/scratch_hdd` mounted, so every one
|
||||
of the 28 was installable on those grounds. Nothing was skipped for a resource reason.
|
||||
|
||||
2. **The brief's file-leg list is wrong, and it told me where to check.** It grouped *"immich,
|
||||
jellyfin, plex, emby, komga, calibre-web, gokapi, homebox, gramps-web"* as file-leg apps. Read
|
||||
from `07-backup-architecture.md` §6.2 — which the brief itself says to read rather than re-derive
|
||||
— only **four of the 28 are class A** (at least one readable file leg): `calibre-web`, `immich`,
|
||||
`komga`, `paperless-ngx`. **`jellyfin`, `plex` and `emby` are class B precisely because their only
|
||||
bind is a `:ro` media mount**, which `ClassifyBinds` excludes; `gokapi`, `homebox` and
|
||||
`gramps-web` keep everything in named volumes. The table below uses `07` §6.2's classes, not the
|
||||
brief's.
|
||||
|
||||
3. **The brief's database list is incomplete.** It named calcom, claper, outline, paperless-ngx,
|
||||
rallly and sparkyfitness for PostgreSQL and kimai for MariaDB. **`immich` also carries PostgreSQL
|
||||
(a vectorchord variant) and redis, and `wanderer` carries meilisearch** — two apps the brief put
|
||||
in the "file-leg" group actually run their own datastore.
|
||||
|
||||
4. **One of the 28 cannot be installed at all, by design.** `plant-it` is `lifecycle: "abandoned"`,
|
||||
and the product's lifecycle gate refused the deploy with **409 „Ez az alkalmazás jelenleg nem
|
||||
telepíthető."** That is correct behaviour, measured live for the first time. It is the only
|
||||
lifecycle-gated template in the whole catalog of 53.
|
||||
|
||||
### Claims in the brief that were verified true, by looking
|
||||
|
||||
- **The drill repo's Actions are off (R-629).** Read from the API: `has_actions: false,
|
||||
private: true`. And measured rather than trusted: **47 CI jobs before the reset push, 47 after** —
|
||||
the push produced no run and no mail.
|
||||
- **`repoint_drill.py` still works after the catalog moved.** 9202's cache now reads
|
||||
`origin …/app-catalog-drill.git` at `1ad1f34`.
|
||||
- **9202 has the capacity.** 25.9 GB RAM (23.5 free), 28 GB free on `/`, 842 GB on the scratch
|
||||
drive. The root disk is the binding constraint, so **each app's images are removed by name after
|
||||
its verdict** — never `prune` (rule 3). Disk held at 1.9 GB used throughout.
|
||||
|
||||
### Changed method, named
|
||||
|
||||
5. **A restore that is REFUSED is recorded as `refused-with-a-sentence`, not as a failed restore.**
|
||||
The first version of the harness collapsed the two and mislabelled `calibre-web`. The product had
|
||||
in fact done the right thing — see the finding below — and a harness that calls a correct refusal
|
||||
a failure would have buried it.
|
||||
|
||||
6. **The harness now waits for a restore to settle before removing.** It did not at first, and that
|
||||
race produced **R-633**, a real defect. The race was left in the record for `gokapi` and fenced
|
||||
out afterwards, so the remaining apps measure the product rather than the harness.
|
||||
|
||||
|
||||
7. **Three bugs in tonight's own harness, each named with what it cost.** A missing `import re` in
|
||||
the restore step killed the restore half for `wanderer`, `claper`, `sparkyfitness` and `calcom`.
|
||||
A variable named `m` shadowed the app's metadata and broke `paperless-ngx`'s teardown. An empty
|
||||
phase list crashed on `[-1]` when the Update was **refused** before any phase existed, which cost
|
||||
`rallly` its whole walk. **All four apps, and five more, were re-walked serially afterwards** —
|
||||
and that re-walk is what corrected R-634 and produced `ghost`'s proof. An instrument that can
|
||||
drop results silently is not a measurement; these dropped them loudly and were re-run.
|
||||
|
||||
8. **Concurrency is part of the method and it changed two results.** Three walks ran at once to fit
|
||||
28 apps in one night. `POST /api/backup/run` is **box-wide**, so a second caller gets
|
||||
`409 „Mentés már folyamatban"`, and the Update refuses while a backup or restore is in flight
|
||||
(`409 busy`). **Both refusals are the product being right** and both are quoted below. But they
|
||||
cost `ghost` and `rallly` their edge on the first pass, and they are implicated in two of R-634's
|
||||
three instances. The nine re-walks were serial for exactly this reason.
|
||||
|
||||
|
||||
### Corrected 2026-09-22 (evening), from the records rather than by re-running
|
||||
|
||||
9. **`calibre-web`'s restore was `refused-with-a-sentence`, not `failed`.** The refusal is in this
|
||||
app's own `log.txt` line 12 as a `flash_error` on the redirect, and its state stayed `running`
|
||||
throughout. It was recorded `failed` because that walk ran **before** the refusal-capture code was
|
||||
added later the same night — the document's own section *"A restore that is REFUSED"* already said
|
||||
so while the table and the record disagreed with it.
|
||||
|
||||
10. **`calcom`'s restore was `inconclusive`, not `failed` — and that was my harness, not the
|
||||
product.** The restore was accepted (`flash=flash.restore.started`, no `flash_error`), the app
|
||||
read `running` at +45 s with no hold and no phase, and the classifier looked once more and saw
|
||||
**`starting`** — a settling state it did not list beside `running`/`unhealthy`, so it fell
|
||||
through to `failed`. **The task brief supposed a different cause** — "a read-back of data that
|
||||
was never seeded" — and that is wrong: the seed half is recorded separately and was already
|
||||
`no route`. The record's own `restore_state_seen: "starting"` is the evidence.
|
||||
|
||||
**Totals move with it:** restores correctly REFUSED go from 2 to **3**.
|
||||
|
||||
---
|
||||
|
||||
## What this night is, in three lines
|
||||
|
||||
- **Interventions: SEVEN — over the brief's limit of five, and six of the seven were my own harness,
|
||||
not the product.** Three were bugs in tonight's driver that cost apps their walk and were fixed
|
||||
mid-run (a missing `import re`; a variable that shadowed the app's metadata; a crash when the
|
||||
Update was refused before any phase existed). Two were deliberate method changes (recording a
|
||||
REFUSED restore as its own verdict; making the harness wait for a restore to settle before
|
||||
removing). One was a waiter that deadlocked on its own command line. **The seventh was the
|
||||
product's:** three leftovers it could not clear, which a shell had to.
|
||||
- **26 of 28 deployed; 6 proven; 5 inconclusive; 14 with no upstream edge; 1 failed honestly;
|
||||
2 that could not be deployed** — one of those by design.
|
||||
- **The one result that matters most:** an app with **no health probe at all** has its working
|
||||
installation **stopped by a successful update**. `paperless-ngx` was healthy on all three
|
||||
containers; the Update ran the full five-minute health wait and then held the app, and the
|
||||
controller named the reason itself — **`no probe container`**. R-630 is raised to P1.
|
||||
|
||||
---
|
||||
|
||||
## The two findings that are not about any single app
|
||||
|
||||
### R-633 — a remove sent while a restore is still running reports success and leaves an orphan
|
||||
|
||||
`gokapi` was restored from its own local copy at **11:34:07** and removed at **11:34:22**.
|
||||
`POST /backup/restore` answers **302 and does its work in the background**; the remove tore down
|
||||
what existed, and the restore's own `compose up` then **re-created the container at 11:34:24**.
|
||||
Both calls returned success.
|
||||
|
||||
Twenty-five minutes later, `GET /api/stacks/gokapi` reads **`deployed: false`** while `docker ps -a`
|
||||
shows `gokapi` **`Restarting (1)`** with its full Traefik label set still attached — including
|
||||
`traefik.http.routers.gokapi.rule: Host(`.enkisfelhom.hu`)`, **a rule with an empty subdomain**,
|
||||
because the deploy values that filled it were deleted with the app. Its own log loops
|
||||
*„Salt for admin password invalid… password does not appear to be a SHA-1 hash"* — the volume
|
||||
holding its config was removed correctly, so the binary can never start.
|
||||
|
||||
**A household can press exactly those two buttons in that order.** The product accepted both and
|
||||
**the remove reported success while leaving the orphan**; nothing in the alarm ladder can fire,
|
||||
because `08` §4 keys on stacks the controller still knows about. This is **R-626's class with the
|
||||
mechanism finally visible** — that row saw a removed `navidrome` come back and could not diagnose
|
||||
it, because the controller had restarted and its log no longer reached the moment. Here the window
|
||||
is **seventeen seconds** and both halves are in the evidence.
|
||||
|
||||
The harness was then fenced against its own race, so every app after `gokapi` measures the product.
|
||||
Evidence: `apps/gokapi/came-back-evidence.txt`.
|
||||
|
||||
### A restore that is REFUSED is the product being right, and nearly went down as a failure
|
||||
|
||||
`calibre-web` is a class-A app (`07` §6.2): it has a readable file leg, and the local Tier-1 copy
|
||||
does not hold it. The restore was **refused**, with this sentence:
|
||||
|
||||
> „Ez a mentés nem tartalmazza az alkalmazás fájljait, ezért nem állítjuk vissza az adatbázist
|
||||
> föléjük — a fájlok így a helyükön maradnak. A fájlok a távoli másolatból állíthatók vissza:
|
||||
> Biztonsági mentés → Visszaállítás, „Teljes visszaállítás (fájlok + adatbázis)"."
|
||||
|
||||
That is exactly what `07` §6.2 predicts, it names the action that does work, and it refuses
|
||||
**before** touching anything. The first version of tonight's harness recorded it as a failed
|
||||
restore. **A harness that calls a correct refusal a failure buries the best result of the night**,
|
||||
so refusals are now recorded as their own verdict and the sentence is quoted.
|
||||
|
||||
|
||||
### One real upstream edge HELD honestly — `outline 1.9.1 → 1.10.1`
|
||||
|
||||
The most valuable single result after R-630, because it is the guarded update's own promise
|
||||
exercised on a real upstream version rather than a staged one.
|
||||
|
||||
| phase | at |
|
||||
|---|---|
|
||||
| `safety-dump` | 0.0 s |
|
||||
| `pulling` | +1.1 s |
|
||||
| `starting` | +64.8 s |
|
||||
| `verifying` | +65.8 s |
|
||||
| **`failed`** | **+368.6 s** |
|
||||
|
||||
The app was stopped and held, and the sentence the household reads names the tier, the date **and
|
||||
what the copy contains**:
|
||||
|
||||
> „A(z) outline frissítése 2026-09-22 14:50-kor nem sikerült, és az alkalmazás nem indult el az új
|
||||
> verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek.
|
||||
> Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-22 14:44
|
||||
> — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza."
|
||||
|
||||
The restore named in that sentence was then walked and the app came back. **`outline` must not be
|
||||
promoted.**
|
||||
|
||||
### Two refusals that are the product guarding itself, and both name the blocker
|
||||
|
||||
- **The Update refuses while a backup or restore runs:** „A frissítés most nem indítható:
|
||||
mentés/visszaállítás folyamatban. Próbáld újra, ha befejeződött."
|
||||
- **A second restore refuses and NAMES the app that is blocking it:** „Egy visszaállítási művelet
|
||||
**(jellyfin)** már fut, ezért most nem indítható újabb."
|
||||
|
||||
**That second guard is exactly the fence R-633 is missing.** The product already knows how to refuse
|
||||
a conflicting operation and how to say which one — for `update` and for `restore`. **`remove` has no
|
||||
such guard**, which is why a remove sent during a restore reports success and leaves an orphan.
|
||||
|
||||
---
|
||||
|
||||
## The table — all twenty-eight
|
||||
|
||||
Classes are `07-backup-architecture.md` §6.2's, not re-derived.
|
||||
|
||||
| app | class | deployed | seeded | backup | edge | update | restore | removed clean | s | evidence |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| `calcom` | B volumes-only + postgres | yes | no route | 1 copy | none upstream | — | inconclusive | yes | 675.8 | `apps/calcom/` |
|
||||
| `calibre-web` | A file-leg | yes | no route | 1 copy | none upstream | — | refused-with-a-sentence | yes | 187.5 | `apps/calibre-web/` |
|
||||
| `claper` | B volumes-only + postgres | yes | no route | 1 copy | none upstream | — | ok | yes | 255.3 | `apps/claper/` |
|
||||
| `code-server` | B volumes-only | yes | route failed | 1 copy | `4.129.0` → `4.138.0` | done | ok | yes | 299.9 | `apps/code-server/` |
|
||||
| `crafty-controller` | B volumes-only | yes | no route | 1 copy | `4.10.7` → `4.11.0` | done | ok | yes | 288.5 | `apps/crafty-controller/` |
|
||||
| `emby` | B volumes-only | yes | yes | 1 copy | `4.10.0.20` → `4.11.0.1` | done | ok | yes | 198.7 | `apps/emby/` |
|
||||
| `ghost` | B volumes-only | yes | yes | 1 copy | `6.53.0-alpine` → `6.64.0-alpine` | done | ok | yes | 289.6 | `apps/ghost/` |
|
||||
| `gokapi` | B volumes-only | yes | no route | 1 copy | none upstream | — | ok | **no** | 132.3 | `apps/gokapi/` |
|
||||
| `gramps-web` | B volumes-only | yes | route failed | 1 copy | none upstream | — | ok | yes | 334.1 | `apps/gramps-web/` |
|
||||
| `homebox` | B volumes-only | yes | route failed | 1 copy | none upstream | — | ok | yes | 127.9 | `apps/homebox/` |
|
||||
| `homepage` | B volumes-only | yes | no route | 1 copy | none upstream | — | ok | yes | 137.1 | `apps/homepage/` |
|
||||
| `immich` | A file-leg + postgres+redis | yes | yes | 1 copy | `v3.0.3` → `v3.2.2` | done | refused-with-a-sentence | yes | 375.0 | `apps/immich/` |
|
||||
| `jellyfin` | B volumes-only | yes | route failed | 1 copy | none upstream | — | ok | yes | 220.7 | `apps/jellyfin/` |
|
||||
| `kimai` | B volumes-only + mariadb | yes | route failed | 1 copy | none upstream | — | ok | yes | 311.3 | `apps/kimai/` |
|
||||
| `komga` | A file-leg | yes | route failed | 1 copy | `1.25.0` → `1.27.1` | done | ok | yes | 251.9 | `apps/komga/` |
|
||||
| `onlyoffice` | B volumes-only | yes | route failed | 1 copy | none upstream | — | ok | yes | 227.3 | `apps/onlyoffice/` |
|
||||
| `outline` | B volumes-only + postgres+redis | yes | route failed | 1 copy | `1.9.1` → `1.10.1` | failed | failed | yes | 1389.6 | `apps/outline/` |
|
||||
| `paperless-ngx` | A file-leg + postgres+redis | yes | no route | 1 copy | none upstream | — | refused-with-a-sentence | yes | 253.2 | `apps/paperless-ngx/` |
|
||||
| `plant-it` | B volumes-only | **no** | no route | — | none upstream | — | not-attempted | yes | 78.2 | `apps/plant-it/` |
|
||||
| `plex` | B volumes-only | yes | route failed | 1 copy | `1.41.4.9463-630c9f557` → `1.43.4.10903-e5521bd8c` | done | ok | yes | 262.1 | `apps/plex/` |
|
||||
| `radarr` | B volumes-only | yes | yes | 1 copy | `6.3.0` → `6.4.4` | done | ok | yes | 227.8 | `apps/radarr/` |
|
||||
| `rallly` | B volumes-only + postgres | yes | route failed | 1 copy | `4.11.1` → `4.15.2` | done | ok | yes | 230.1 | `apps/rallly/` |
|
||||
| `recipe-importer` | B volumes-only | yes | no route | 1 copy | none upstream | — | ok | yes | 123.0 | `apps/recipe-importer/` |
|
||||
| `seerr` | B volumes-only | yes | no route | 1 copy | none upstream | — | ok | yes | 218.8 | `apps/seerr/` |
|
||||
| `sonarr` | B volumes-only | yes | yes | 1 copy | `4.0.19` → `4.0.20` | done | ok | yes | 216.6 | `apps/sonarr/` |
|
||||
| `sparkyfitness` | B volumes-only + postgres | **no** | no route | — | none upstream | — | not-attempted | **no** | 534.0 | `apps/sparkyfitness/` |
|
||||
| `termix` | B volumes-only | yes | yes | 1 copy | `2.5.0` → `2.8.0` | done | ok | **no** | 232.5 | `apps/termix/` |
|
||||
| `wanderer` | B volumes-only + meilisearch | yes | no route | 1 copy | none upstream | — | ok | yes | 400.5 | `apps/wanderer/` |
|
||||
|
||||
**Totals:** **14** no-edge · **6** proven · **5** inconclusive · **2** could-not-deploy · **1** failed — 28 of 28 recorded.
|
||||
|
||||
**Read across the walk rather than down one column:** **26 of 28 deployed**, **6** had a non-browser route that seeded AND read back, **21** restored from their own copy, **3** were correctly REFUSED a restore, **6 proven / 1 failed** on the apps that had an upstream edge, and **2** left a container behind (R-633).
|
||||
|
||||
---
|
||||
|
||||
## 5.2 — the templates the static gate cannot judge (R-631)
|
||||
|
||||
The probe gate's oracle is the probed service's own compose healthcheck. For five templates there is
|
||||
no such oracle, or the paths differ on a check that cannot fail. A static rule cannot settle any of
|
||||
them; asking the running container can. Each was deployed on 9202, its listening sockets read from
|
||||
inside the container, and the probe's own target dialled **on the compose network** — the same call
|
||||
the controller makes.
|
||||
|
||||
| app | probe | what it listens on | the probe's own dial | verdict |
|
||||
|---|---|---|---|---|
|
||||
| `mealie` | `tcp` 9000 | `0.0.0.0:9000` | 200 | **correct** |
|
||||
| `uptime-kuma` | `http` 3001 | `*:3001` | 302 | **correct** — `http` calls any response healthy, and 302 proves something answers |
|
||||
| `vikunja` | `api` 3456 `/api/v1/info` **expect 200** | (busybox: no `ss`, no `netstat`) | **200** | **correct** — and this is the one that could have failed, because its `expect` block compares the code |
|
||||
| `home-assistant` | `api` 8123 `/api/` **no expect** | `0.0.0.0:8123` | **401** | **correct today, and the 401 is the measurement that proves the warning** |
|
||||
| `crafty-controller` | `tcp` 8443 | (not read — the app never reached `deployed`; see R-634) | **ok (1 ms)** | **correct** |
|
||||
|
||||
**home-assistant is the one to carry forward.** Its probe dials `/api/` and gets **401** — not 200.
|
||||
It reads healthy only because `probeHTTP` treats any response as healthy when the type is `api` with
|
||||
no `expect` block (`healthprobe.go:253-262`). **Add `expect: {status: 200}` to that template — a
|
||||
change that looks like a tightening — and home-assistant goes permanently unhealthy, and every
|
||||
successful update of it starts stopping it.** That is R-618's failure exactly, one edit away, and it
|
||||
is now a measured number rather than a caution.
|
||||
|
||||
**All five are settled.** crafty-controller's reading came from its own walk rather than the side job: the controller's log shows `Health probe crafty-controller: TCP :8443 -> ok (1ms)` twice, six minutes apart, while the app was running. The gate's WARN list is therefore not a backlog of suspects: it is
|
||||
four correct templates the gate honestly cannot prove, and one that is correct by accident.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 5.1 — what the guarded Update does when NO probe exists (R-630)
|
||||
|
||||
`paperless-ngx` has no container whose name equals or begins with its stack name, so
|
||||
`findProbeContainer` returns `""` and `RunHealthProbes` skips the stack silently. **Its probe has
|
||||
never run on any box.** The open question was what `verifying` — which waits on that same probe —
|
||||
does when there is nothing to wait on: pass at once, wait out the timeout, or hold.
|
||||
|
||||
**It waits out the full timeout and then HOLDS, stopping a working app.**
|
||||
|
||||
Deployed on 9202, all three containers reported **`healthy`**, the controller read **`running`**, the
|
||||
front door answered **302**. No upstream edge exists for paperless-ngx tonight, so the Update was
|
||||
pressed on the **same version** — which is what a household does on an up-to-date app, and it still
|
||||
walks the whole phase machine. That difference is stated, not glossed.
|
||||
|
||||
| phase | at |
|
||||
|---|---|
|
||||
| `checking` → `safety-dump` → `pinning` → `pulling` | 0.0–1.1 s |
|
||||
| `starting` | +2.1 s |
|
||||
| `verifying` | +3.1 s |
|
||||
| **`failed`** | **+313.0 s — the app STOPPED** |
|
||||
|
||||
Afterwards: controller state **`stopped`**, front door **404**.
|
||||
|
||||
**The controller names the cause itself, so no inference was needed:**
|
||||
|
||||
> `update paperless-ngx FAILED after the new version was started: not healthy: not healthy within
|
||||
> 5m0s (last: no probe container) — stopping and HOLDING the app; the pin stays on the new version
|
||||
> (its migration may have run)`
|
||||
|
||||
**`no probe container`.** And the hold sentence is correct about the route back — for this class-A
|
||||
app it warns *„csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem"*.
|
||||
|
||||
**This is R-618's outcome reached by the opposite road.** There a probe named a port the app does not
|
||||
answer; here no probe exists at all — and the static gate cannot see it, because there is nothing to
|
||||
compare. The gate does print it as a WARNING on every push, which is how it was found. **R-630 is
|
||||
raised P2 → P1.**
|
||||
|
||||
|
||||
## The second promotion list — for the operator, not for me
|
||||
|
||||
**CC promotes nothing.** These are proposals with the evidence beside them.
|
||||
|
||||
### Proposed to move (6)
|
||||
|
||||
| app | move | update took | its own migration line |
|
||||
|---|---|---|---|
|
||||
| `emby` | `4.10.0.20` → `4.11.0.1` | 198.7 s | yes — `emby \| Info SqliteUserRepository: Sqlite compiler options: ATOMIC_INTRINSICS=1,COMPILER=g` |
|
||||
| `ghost` | `6.53.0-alpine` → `6.64.0-alpine` | 289.6 s | yes — `ghost \| [2026-09-22 15:13:37] [36mINFO[39m Stripe members-migrations skipped because it` |
|
||||
| `immich` | `v3.0.3` → `v3.2.2` | 375.0 s | none printed |
|
||||
| `radarr` | `6.3.0` → `6.4.4` | 227.8 s | yes — `radarr \| [migrations] started` |
|
||||
| `sonarr` | `4.0.19` → `4.0.20` | 216.6 s | yes — `sonarr \| [migrations] started` |
|
||||
| `termix` | `2.5.0` → `2.8.0` | 232.5 s | yes — `termix \| [1:37:29 PM] [INFO] [🗄️] Database layer pre-upgrade backup created [op:database_` |
|
||||
|
||||
### Must NOT move, with why (6)
|
||||
|
||||
| app | edge | why not |
|
||||
|---|---|---|
|
||||
| `code-server` | `4.129.0` → `4.138.0` | the update reached `done`, but this app has no non-browser data route, so nothing proves the household's data survived it |
|
||||
| `crafty-controller` | `4.10.7` → `4.11.0` | the update reached `done`, but this app has no non-browser data route, so nothing proves the household's data survived it |
|
||||
| `komga` | `1.25.0` → `1.27.1` | the update reached `done`, but this app has no non-browser data route, so nothing proves the household's data survived it |
|
||||
| `outline` | `1.9.1` → `1.10.1` | the update ended `failed` — fixture ran and found no non-browser seed route: sign-in requires an external identity provider (OIDC/Slack/Google); no local sign-up route exists |
|
||||
| `plex` | `1.41.4.9463-630c9f557` → `1.43.4.10903-e5521bd8c` | the update reached `done`, but this app has no non-browser data route, so nothing proves the household's data survived it |
|
||||
| `rallly` | `4.11.1` → `4.15.2` | the update reached `done`, but this app has no non-browser data route, so nothing proves the household's data survived it |
|
||||
|
||||
**No upstream edge tonight, so nothing to propose (16):** `calcom`, `calibre-web`, `claper`, `gokapi`, `gramps-web`, `homebox`, `homepage`, `jellyfin`, `kimai`, `onlyoffice`, `paperless-ngx`, `plant-it`, `recipe-importer`, `seerr`, `sparkyfitness`, `wanderer`.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Teardown — three layers plus Gitea, every claim READ BACK
|
||||
|
||||
**The machine (9202).** `controller.yaml` restored from `controller.yaml.pre-28`; `git.repo_url`
|
||||
reads back as the **live** catalog with an empty token; and — the one that actually decides which
|
||||
remote is followed (R-615) — the **cache** reads
|
||||
`origin https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git` at **`1ad1f34`**. No drill
|
||||
images. Disk 1.9 GB used of 32 GB, unchanged from the start.
|
||||
|
||||
**Three things the product could NOT clear, and a shell had to.** This is not tidy-up, it is the
|
||||
finding: the `termix` and `gokapi` containers left by R-633, and `sparkyfitness`'s `app.yaml` left by
|
||||
R-634. `gokapi` was still `Restarting` two hours later. They were removed **by name**
|
||||
(`docker rm -f termix gokapi`, `rm .../sparkyfitness/app.yaml`) — never a `prune`. Afterwards 9202
|
||||
runs exactly `felhom-controller`, `filebrowser`, `traefik`, and no `app.yaml` exists anywhere.
|
||||
**A household has no shell.** Evidence: `teardown/manual-cleanup.txt`.
|
||||
|
||||
**The host (demo-hp).** `pct list` before and after: `9201 demo-hp` and `9202 demo-hp-scratch`, both
|
||||
running, unchanged. `pvesm status` unchanged but for expected scratch growth (`nvme-scratch`
|
||||
5.19% → 6.95%). **Guest 9201 was never touched.**
|
||||
|
||||
**The hub.** Nothing provisioned, nothing changed. 9202 runs `hub.enabled: false` (R-620).
|
||||
|
||||
**Gitea.** The drill repo is reset to the live `main` (`1ad1f34b6e51`). **`git diff` of the live
|
||||
catalog's `templates/` against the night's baseline: 0 lines, and `image:` lines changed: NONE.**
|
||||
|
||||
**The fences, each read back rather than asserted:**
|
||||
|
||||
| fence | at the start | at the end |
|
||||
|---|---|---|
|
||||
| live catalog `origin/main` | `1ad1f34b6e51` | **`1ad1f34b6e51`** |
|
||||
| demo-hp guest 9201 catalog cache | `1ad1f34`, live remote | **`1ad1f34`, live remote** |
|
||||
| demo-felhom guest 9201 catalog cache | `1ad1f34`, live remote | **`1ad1f34`, live remote** |
|
||||
| drill repo CI jobs | **47** | **47** — no run, no mail, all night (R-629 holds) |
|
||||
|
||||
**Peti's box is parked and received nothing.** Nothing ran on DooPlex beyond ordinary pushes, and
|
||||
nothing on `ep0`. `felhom-controller`, `felhom-agent` and the hub were read only — **no product code
|
||||
was written.** No golden, no bake, no vouch, no `--no-verify`, no branch. `local-lvm` untouched, no
|
||||
`prune`, `tester-1` never reset, `drill-r50` untouched.
|
||||
@@ -0,0 +1,498 @@
|
||||
# DRILL — UPDATE NIGHT, 2026-09-21
|
||||
|
||||
**Venue:** scratch guest **9202 `demo-hp-scratch`** on host `demo-hp`, controller **v0.261.0**.
|
||||
**Catalog:** a private drill copy, `admin/app-catalog-drill`. **The live catalog was not touched.**
|
||||
**Evidence:** `audits/update-night-2026-09-21/` — `PROGRESS.md` is the step log, `apps/<n>/verdict.json`
|
||||
the per-edge records, `bad-days/<leg>/` the Phase-3 legs.
|
||||
|
||||
---
|
||||
|
||||
## Not done, or changed from the brief
|
||||
|
||||
*(Filled at the end of the run. Every phase and every B-leg is listed here if it was skipped,
|
||||
shortened or altered, with the reason. Empty only if true — R-611.)*
|
||||
|
||||
**Nothing in the brief was skipped. Four things were CHANGED or RE-RUN, and one was measured on a
|
||||
different venue than the brief named — each with its reason.**
|
||||
|
||||
| what | what happened | why |
|
||||
|---|---|---|
|
||||
| **Phase 2.3, the PostgreSQL rehearsal** | run on **guest 9202 itself**, with plain `docker` beside the product, not on a separate harness LXC | the rehearsal needed the SAME app the 5.2 leg had on a real seeded 16 datadir. No harness LXC was created tonight, so none was destroyed — stated again in the teardown |
|
||||
| **Phase 2.3, the `pg_upgrade` route** | **NOT run.** The logical dump-and-restore route was run end to end and costed | `pg_upgrade` needs both majors' binaries in one image and no such image exists in this project. Building it is the work Q5's first option is really asking for; naming it costs nothing, and the logical route may make it unnecessary at this size |
|
||||
| **B5's `safety-dump` cut** | **MISSED on the first attempt and recorded as a MISS**, then retried with a real pending edge and HIT | the first attempt's app was level with the catalog, so the update failed in 0.473 s and `safety-dump` was never observed. A miss recorded as a miss, then fixed |
|
||||
| **B8, and Phase 2.3's first attempt** | **re-run** after B1's own precondition swept the app they needed | B1 removes every other behind-app so the unattended caller has exactly one thing to react to. That is correct and is recorded; it also removed `docmost`. Re-run in `phase2_redo.sh` |
|
||||
| **The harness runs of the new catalog EDGES (U1–U7)** | **code shipped, runs OWED** | the box-side result for each of those edges exists; the harness adds the ABORT step, and setting up `/opt/upg` was not worth the last hour against the teardown |
|
||||
|
||||
**And one thing the brief asked for that this VENUE cannot produce at all, named rather than left
|
||||
blank: every event and every customer mail.** Guest 9202 runs `hub.enabled: false` and every
|
||||
notifier entry point returns before it logs anything (R-620). The hub was not enabled to get around
|
||||
it — that would register an unclaimed host at the live hub and could mail a real address, and the
|
||||
brief fences the hub. The alarm truth table below says so on every row.
|
||||
|
||||
---
|
||||
|
||||
## The three lines
|
||||
|
||||
**Interventions: ZERO.** Nothing tonight needed an act a household could not perform from the
|
||||
screens. Every app was deployed, seeded, updated, held, restored and removed through the product's
|
||||
own endpoints; the only non-product commands were the power cuts (`pct stop`, which IS the fault
|
||||
being tested) and the reproduction of a refusal the product had destroyed.
|
||||
|
||||
**21 edges attempted: 14 proven, 3 failed, 4 inconclusive.** Up from the **three** apps this project
|
||||
had ever measured. Ten of the fourteen printed a verbatim migration line, so the database really was
|
||||
rewritten and the data still read back.
|
||||
|
||||
**The one result that matters most: three of the 53 templates name a health probe the app does not
|
||||
answer — and because the guarded Update waits on that same probe, a SUCCESSFUL update ends by
|
||||
STOPPING a working app.** `tandoor` was measured serving HTTP 200 on the new version at four samples
|
||||
across five minutes, with docker's own healthcheck green, and was then stopped by `failAndHold` and
|
||||
the household sent to a restore they did not need. `zipline` and `wger` are the same defect. **R-618,
|
||||
P1.** No data is lost and the restore works — but "the update is guarded" must not be read as "the
|
||||
guard is right about whether the app came up".
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — the two mechanisms, each with its controls
|
||||
|
||||
### 0.1 The floor to 0.261.0 — PROVEN, 13 seconds
|
||||
|
||||
Saved as `min_controller_version=0.261.0` with `min_agent=0.131.0` (read from the release's own
|
||||
CHANGELOG header, as publish-train rule 1 requires). The hub answered `303 …?flash=floor_set` — not
|
||||
`floor_needs_min_agent` — and said so itself, twice:
|
||||
|
||||
```
|
||||
2026/09/21 20:07:07 [INFO] Global controller-version floor set to "0.261.0" (declared MinAgent "0.131.0")
|
||||
2026/09/21 20:07:09 [INFO] managed floor SERVED for demo-felhom: floor 0.261.0, agent requirement "0.131.0" from declared (golden 0.258.0)
|
||||
2026/09/21 20:07:10 [INFO] managed floor SERVED for demo-hp: floor 0.261.0, agent requirement "0.131.0" from declared (golden 0.258.0)
|
||||
```
|
||||
|
||||
Both demo boxes were running `0.261.0` **13 seconds after the save** (`20:07:07` → `20:07:20`), each
|
||||
container `Up … (healthy)`. The vouched golden is `0.258.0`, so this is the declared-MinAgent path of
|
||||
§3 decision 7 working exactly as R-472 closed it.
|
||||
|
||||
**Which other boxes it reaches:** none. The hub lists three hosts; the third, `drill-r50-0a4f9a`, is
|
||||
`DOWN` and its agent is `0.129.0`, below the declared `0.131.0`, so the floor is held for it — by
|
||||
design, and it was left alone.
|
||||
|
||||
Evidence: `01-floor-pre.txt`, `02-floor-save.txt`.
|
||||
|
||||
### 0.2 The drill catalog — PROVEN, with all three controls
|
||||
|
||||
`admin/app-catalog-drill`, private, created from the live catalog's `main` (`f5f6a152b513`).
|
||||
|
||||
**One claim in the brief turned out wrong before a single command was run, and it is the reason the
|
||||
leg worked at all.** The brief assumed a box follows a second catalog once `git.repo_url` changes.
|
||||
It does not. `Syncer.gitCloneOrPull` clones **only when `.git` is absent**; otherwise it fetches from
|
||||
the remote stored in the clone. After the repoint, `git remote -v` in the box's cache still read
|
||||
`app-catalog-felhom.eu`. The cache directory had to be removed as well. Filed as **R-615**.
|
||||
|
||||
- **Positive control.** A one-step bump committed to the DRILL repo (`uptime-kuma 2.4.0 → 2.5.5`,
|
||||
a real upstream edge) appeared on 9202 as „**Frissítés elérhető — ma**" / "**Update available —
|
||||
today**", both languages, with the matching title text.
|
||||
- **Negative control 1.** The live catalog's `main` is still `f5f6a152b513` and its `uptime-kuma`
|
||||
pin is still `2.4.0`.
|
||||
- **Negative control 2.** Both real boxes' catalog caches are still at `f5f6a15` — neither followed
|
||||
anything.
|
||||
- **R-607 fired again**, exactly as its row predicts: `POST /api/sync` answered
|
||||
„Sablonok naprakészek — nincs változás" while the catalog had in fact moved; only
|
||||
`POST /api/stacks/rescan` made `catalog_images` current. Tonight's line is added to that row.
|
||||
|
||||
**A second brief-claim corrected:** the app page is at **`/apps/<name>`**, not `/app/<name>` as
|
||||
`update-arc-gaps-2026-09-21/00-api-recipe.md` says. That recipe line is fixed.
|
||||
|
||||
Evidence: `03-drill-repo.txt`, `04-9202-config-pre.txt`, `05-9202-follows-drill.txt`,
|
||||
`07-positive-control.txt`.
|
||||
|
||||
### 0.3 The throwaway image store — PROVEN, and the comparator claim RUN rather than read
|
||||
|
||||
`registry:2` on 9202 at `127.0.0.1:5000`. Never DooPlex's registry; no real box can reach it.
|
||||
|
||||
| tag | what it is |
|
||||
|---|---|
|
||||
| `localhost:5000/drill/glance:1.0.0` | the real `glanceapp/glance:v0.8.6`, retagged — it serves |
|
||||
| `localhost:5000/drill/glance:1.0.1` | a built image that starts, stays up and **never serves** |
|
||||
| `localhost:5000/drill/glance:1.0.2` | the FIXED next version, for B6 |
|
||||
| `localhost:5000/drill/pdf:1.0.0` | the real bentopdf, retagged |
|
||||
| `localhost:5000/drill/pdf:1.0.1` | **absent from the store** — 404, for the pull-fail leg |
|
||||
|
||||
**The brief's worry about `host:port/` was unfounded, and it was settled by running the comparator,
|
||||
not by reading it.** `splitImageRef` takes the LAST colon and rejects it only when a `/` follows, so
|
||||
a registry port is never mistaken for a tag. Four positive cases and one negative control, in the
|
||||
controller's own package:
|
||||
|
||||
```
|
||||
OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","localhost:5000/drill/glance:1.0.1") = (-1,true)
|
||||
OK CompareImageRefs("localhost:5000/drill/glance:1.0.1","localhost:5000/drill/glance:1.0.0") = ( 1,true)
|
||||
OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","localhost:5000/drill/glance:1.0.0") = ( 0,true)
|
||||
OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","localhost:5000/drill/glance:1.0.2") = (-1,true)
|
||||
OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","glanceapp/glance:1.0.1") = ( 0,false) <- negative control
|
||||
```
|
||||
|
||||
The temporary test file was deleted and `git status --porcelain` is empty again.
|
||||
|
||||
Evidence: `09-image-store.txt`.
|
||||
|
||||
### 0.4 Capacity — the brief's claim HOLDS
|
||||
|
||||
9202: **26 GB RAM** (22 free), docker root on the `mp0` volume with **56 GB free**, a 938 GB scratch
|
||||
drive at 3%. `demo-hp`'s own `/` is at 86% but holds neither the rootfs nor the docker root — both
|
||||
live on `nvme-scratch`, which is at 2%. Evidence: `08-capacity.txt`.
|
||||
|
||||
### 0.5 The drift re-run — the brief's numbers HOLD EXACTLY
|
||||
|
||||
Re-measured against the live registries at 20:07, catalog `f5f6a152b513`: 53 apps, 66 unique pins,
|
||||
**46 behind upstream, 39 within a major, 7 across a major** — the same 39 and 7 the brief names.
|
||||
One pin is unmeasurable tonight (`msdeluise/plant-it` — Docker Hub answered 401 on its tag list) and
|
||||
one is internal. Evidence: `06-drift-rerun.txt`.
|
||||
|
||||
---
|
||||
|
||||
## The verdict table
|
||||
|
||||
One row per edge attempted tonight. `inconclusive` means *we could not measure it*, which
|
||||
is a different fact from *it does not work* — and only one of them is about the app.
|
||||
|
||||
| app | from → to | class | box verdict | seed before → after | secs | migration line seen | evidence |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| `actualbudget` | actual-server:26.7.0 → actual-server:26.9.0 | other | **proven** | True → True | 19.5 | yes | `apps/actualbudget/` |
|
||||
| `audiobookshelf` | audiobookshelf:2.35.1 → audiobookshelf:2.36.1 | file-leg | **proven** | True → True | 23.6 | yes | `apps/audiobookshelf/` |
|
||||
| `bookstack` | bookstack:26.05.2, mariadb:12.3 → bookstack:26.05.5, mariadb:12.3 | db-mariadb | **proven** | True → True | 45.1 | none printed | `apps/bookstack/` |
|
||||
| `docmost` | docmost:0.95.0, postgres:16-alpine, redis:7-alpine → docmost:0.96.0, postgres:16-alpine, redis:7-alpine | db-postgres | **proven** | True → True | 103.6 | yes | `apps/docmost/` |
|
||||
| `grafana` | grafana:13.1.0 → grafana:13.2.2 | other | **proven** | True → True | 26.7 | yes | `apps/grafana/` |
|
||||
| `home-assistant` | home-assistant:2026.7.2 → home-assistant:2026.9.3 | other | **proven** | True → True | 103.6 | none printed | `apps/home-assistant/` |
|
||||
| `mealie` | mealie:v3.20.1 → mealie:v3.27.0 | db-postgres | **proven** | True → True | 18.5 | yes | `apps/mealie/` |
|
||||
| `n8n` | n8n:2.31.3 → n8n:2.40.5 | db-postgres | **proven** | True → True | 117.9 | yes | `apps/n8n/` |
|
||||
| `navidrome` | navidrome:0.63.2 → navidrome:0.64.0 | file-leg | **proven** | True → True | 11.3 | yes | `apps/navidrome/` |
|
||||
| `nextcloud` | mariadb:11.6 → mariadb:12.3 | engine-major-mariadb | **proven** | True → True | 217.4 | none printed | `apps/nextcloud-engine-mariadb/` |
|
||||
| `papra` | papra:26.6.1-rootless → papra:26.6.2-rootless | other | **proven** | True → True | 60.5 | yes | `apps/papra/` |
|
||||
| `privatebin` | pdo:2.0.5 → pdo:2.0.6 | file-leg | **proven** | True → True | 15.4 | none printed | `apps/privatebin/` |
|
||||
| `romm` | mariadb:11.4, redis:7-alpine, romm:5.0.0 → mariadb:11.4, redis:7-alpine, romm:5.3.0 | db-mariadb | **proven** | True → True | 60.6 | yes | `apps/romm/` |
|
||||
| `vikunja` | vikunja:2.3.0 → vikunja:2.6.0 | other | **proven** | True → True | 24.6 | yes | `apps/vikunja/` |
|
||||
| `adventurelog` | adventurelog-backend:v0.12.1, adventurelog-frontend:v0.12.1, postgis:16-3.5-alpine → adventurelog-backend:v0.13.0, adventurelog-frontend:v0.13.0, postgis:16-3.5-alpine | db-postgis | **failed** | True → False | 346.6 | — | `apps/adventurelog/` |
|
||||
| `docmost` | postgres:16-alpine → postgres:17-alpine | engine-major-postgres | **failed** | True → False | 254.5 | — | `apps/docmost-engine-postgres/` |
|
||||
| `gitea` | gitea:1.27.0 → — | other | **inconclusive** | False → False | 20.1 | — | `apps/gitea/` |
|
||||
| `opengist` | opengist:1.13 → opengist:1.15 | other | **inconclusive** | True → False | 14.4 | — | `apps/opengist/` |
|
||||
| `tandoor` | postgres:16-alpine, recipes:2.6.13 → postgres:16-alpine, recipes:2.6.15 | db-postgres | **failed** | True → False | 361.9 | — | `apps/tandoor/` |
|
||||
| `vaultwarden` | server:1.36.0-alpine → — | other | **inconclusive** | False → False | 17.9 | — | `apps/vaultwarden/` |
|
||||
| `zipline` | postgres:16-alpine, zipline:4.6.1 → — | db-postgres | **inconclusive** | False → False | 73.3 | — | `apps/zipline/` |
|
||||
|
||||
**14 proven · 3 failed · 4 inconclusive — out of 21 attempted.**
|
||||
|
||||
### Why each inconclusive edge could not be judged
|
||||
|
||||
- **`gitea`** — INCONCLUSIVE: the template sets no `INSTALL_LOCK`, so a fresh Gitea starts in its web-installer state and `gitea admin user create` refuses with `MustInstalled() [F] Unable to load config file for a installed Gitea instance`. The route that would work is POSTing the installer form first; that was not written tonight and is listed as owed rather than faked.
|
||||
- **`opengist`** — CORRECTED from `failed` to `inconclusive` the same night, deliberately. The UPDATE itself SUCCEEDED: phase `done` in 14.4 s, and all four version observables agree on `ghcr.io/thomiceli/opengist:1.15` with the container running and zero restarts. What failed was the READBACK: it was attempted immediately after `done` and the sign-in form was not yet being served, so the fixture got no `_csrf` and returned `http=None`. Whether the seeded account survived was therefore NOT ESTABLISHED. Recording that as `failed` would have blamed the app for the harness's impatience — `inconclusive` is the honest verdict and it is never collapsed into `failed`. The fixture now waits for the LOGIN FORM rather than for the root page.
|
||||
- **`vaultwarden`** — INCONCLUSIVE BY DESIGN, not by a gap in the harness: the catalog CLOSES self-registration on purpose (`SIGNUPS_ALLOWED=false`, R-512 — *a stranger who guesses vault.<domain> must not be able to register*), so `/api/accounts/register` answers 404 and there is NO account-creating route without the admin secret. Vaultwarden also ships no CLI. Tried: `POST /api/accounts/register` with a valid KDF envelope. This app cannot be seeded headlessly while that setting stands, and the setting is right.
|
||||
- **`zipline`** — INCONCLUSIVE BY DESIGN: the deploy answers `E1037: User registration is disabled`, so no first account can be created from outside. Tried: `POST /api/auth/register` and `POST /api/auth/setup`. SEPARATELY, zipline is one of R-618's two confirmed victims — its `.felhom.yml` probe expects 200 on `/api/health`, which the app answers 404 — so even with a seed its update would have been HELD by a wrong probe rather than by anything about the edge.
|
||||
|
||||
### The edges that failed — the most valuable results of the night
|
||||
|
||||
- **`adventurelog`** — final phase `failed`, hold `A(z) adventurelog frissítése 2026-09-21 20:53-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.`, error `A(z) adventurelog frissítése 2026-09-21 20:53-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.`. the edge ended HELD or failed — this is a RESULT, not an error of the run
|
||||
- **`docmost`** — final phase `failed`, hold `A(z) docmost frissítése 2026-09-21 21:44-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:44 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.`, error `A(z) docmost frissítése 2026-09-21 21:44-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:44 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.`. ended HELD or failed — a RESULT, not an error of the run
|
||||
- **`tandoor`** — final phase `failed`, hold `A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.`, error `A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.`. the edge ended HELD or failed — this is a RESULT, not an error of the run
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — the two database engines
|
||||
|
||||
### 2.1 MariaDB across a major, through the REAL Update button — PROVEN, and a first
|
||||
|
||||
`nextcloud`, app image held constant, `mariadb: 11.6 → 12.3`. Seeded and read back through
|
||||
`occ user:add` / `occ user:info`, with the fixture's own negative control on every readback.
|
||||
|
||||
**The four observables of `SPIKE-r459`, before → after:**
|
||||
|
||||
| # | observable | before | after |
|
||||
|---|---|---|---|
|
||||
| 1 | `mariadb_upgrade_info` | `11.6.2-MariaDB` | **`12.3.3-MariaDB`** |
|
||||
| 2 | the engine's own check (R-464 — never the log line) | *not measured: the probe was unauthenticated, see below* | **„This installation of MariaDB is already upgraded to 12.3.3-MariaDB. There is no need to run mariadb-upgrade again."** |
|
||||
| 3 | the entrypoint | — | **„Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required!"** → „Starting mariadb-upgrade" → **„Finished mariadb-upgrade"** |
|
||||
| 4 | the engine's own pre-upgrade backup | absent | **`system_mysql_backup_11.6.2-MariaDB.sql.zst`, 631 905 bytes** |
|
||||
|
||||
**Observable 3 is the one that matters**, because R-459's whole finding was that MariaDB can apply a
|
||||
major and *skip* the conversion quietly, printing `skipped due to $MARIADB_AUTO_UPGRADE`. **That line
|
||||
is absent**; the conversion was detected, started and finished. The seeded account read back and all
|
||||
four version observables agree.
|
||||
|
||||
**Two honest notes on the instrument.** The BEFORE capture of observable 2 asked the engine without
|
||||
credentials and got `ERROR 1045 Access denied`; the probe was corrected and the AFTER capture
|
||||
retaken with it, so the before value is **not measured** and is stated as such rather than inferred.
|
||||
And the `ls` in observable 4 printed a "(no pre-upgrade backup file present)" fallback *after*
|
||||
listing the file, because it globs two patterns and one did not match — the file is there.
|
||||
|
||||
Full record: `16-phase2.1-mariadb-major.md`, `apps/nextcloud-engine-mariadb/`.
|
||||
|
||||
### 2.2 PostgreSQL across a major — what a household would see TODAY
|
||||
|
||||
**Exactly what R-463 predicted, and nobody had measured.** `docmost`, engine only, `16 → 17`:
|
||||
the update ended **`failed` in 5.1 s**, the app was stopped and held, **the pin named
|
||||
`postgres:17-alpine` while `installed_images` still said 16 and nothing was running**, and the data
|
||||
was intact. The restore the hold sentence names brought it back in **29.1 s**, hold cleared, health
|
||||
probe 200.
|
||||
|
||||
**The engine's refusal line had to be REPRODUCED**, because `failAndHold` removed the container
|
||||
before any probe could read it (**R-621**) and the controller log does not carry it either. Done
|
||||
independently with a control on every step — source proven 16, copy proven 16, 49 MB:
|
||||
|
||||
```
|
||||
FATAL: database files are incompatible with server
|
||||
DETAIL: The data directory was initialized by PostgreSQL version 16,
|
||||
which is not compatible with this version 17.11.
|
||||
```
|
||||
|
||||
**The datadir was still `16` afterwards** — nothing migrated, nothing damaged — and the positive
|
||||
control (the same copy under `postgres:16-alpine`) started and held **48 tables**.
|
||||
|
||||
**My own first reproduction was WRONG and is kept, labelled.** The volume lookup returned empty, so
|
||||
the copy was empty, so 17 initialised a fresh datadir and started happily — and the run reported
|
||||
`running=true` as though no refusal had happened. A blank `PG_VERSION` one line earlier should have
|
||||
stopped the step and did not. It is kept because it accidentally measured the MIRROR case (16
|
||||
refusing a 17 datadir, verbatim), and relabelled so nobody reads it as the main result.
|
||||
`17-postgres-refusal-reproduced.txt` (wrong) and `18-postgres-refusal-reproduced.txt` (right).
|
||||
|
||||
### 2.3 The conversion rehearsal, COSTED — the answer Q5 was asking for
|
||||
|
||||
Logical dump and restore, on a fresh seeded `docmost`: **49.0 MB datadir, 48 tables.**
|
||||
|
||||
| step | time | what it produced |
|
||||
|---|---|---|
|
||||
| dump with 16 (`pg_dumpall`) | **2.6 s** | **132 201 bytes**, 48 `CREATE TABLE` statements |
|
||||
| fresh 17 datadir + restore | **6.5 s** | `PG_VERSION` 17, **48 tables restored**, 2 benign ERROR lines |
|
||||
| point the app at 17 and start it | 124.8 s | the app's own words: *„Database connection successful"* |
|
||||
| **the seed read back on 17** | — | **TRUE**, through the app's own login |
|
||||
| **total** | **155.9 s** | of which **~9 s is engine work** |
|
||||
|
||||
Full paragraph for Q5, including what could lose data and why `pg_upgrade` was not run:
|
||||
`24-Q5-postgres-conversion-costed.md`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — the bad days
|
||||
|
||||
Every leg records the same five things. **The event/mail column is empty on every row for the same
|
||||
structural reason — see the alarm truth table.**
|
||||
|
||||
| leg | what the household saw | what the box did by itself | time to steady | the alarm |
|
||||
|---|---|---|---|---|
|
||||
| **B1 unattended HOLD** | „Frissítés elérhető" → app **Leállítva**, the hold sentence naming tier, date and what the copy holds; the banner „Telepített alkalmazás nem fut" on **every** page | pressed **once**, held after **312.9 s**, and **never pressed again** across two further passes | 312.9 s | unmeasurable (R-620) |
|
||||
| **B2 pull fails** | „Az új verzió letöltése nem sikerült, ezért a frissítés elmaradt. Az alkalmazás a korábbi verzióval fut tovább." | pin **and** definition put back in **1.0 s**, `hold=None`, old version still serving | 1.0 s | none, correctly — nothing is down |
|
||||
| **B3 busy** | „A frissítés most nem indítható: mentés/visszaállítás folyamatban." | refused `409 reason='busy'` on six consecutive presses; the transient reason a caller needs | — | n/a |
|
||||
| **B4 concurrency** | nothing — all succeeded | **NO single-flight.** 2 of 2, then **5 of 5**, ran at once; all ended `done`, every pin advanced | ~30 s for five | none |
|
||||
| **B5 cut in `backing-up`** | „A frissítés megszakadt, mert a vezérlő újraindult…" | the box said so itself at boot (three positive observables), **pin unmoved**, data intact | one boot | none |
|
||||
| **B5 cut in `safety-dump`** | nothing — the ordinary badge, **no interrupted sentence** | no recovery line, no journal, **pin unmoved**, data intact | one boot | none |
|
||||
| **B6 way out forwards** | badge still says „Frissítés elérhető" and offers the button | the button **refuses `409 reason='held'`** | — | n/a |
|
||||
| **B7 disk floor** | „Nincs elég szabad hely a frissítéshez: 1.4 GB szabad… legalább 2 GB szükséges." | refused before anything moved | instant | n/a |
|
||||
| **B8 floating pin** | „Naprakész" | and it is **TRUE on this box** — both floating digests match upstream exactly | — | n/a |
|
||||
| **B9 frozen app, newer `.felhom.yml`** | nothing — 10 samples, all `running`/200 | the newer `.felhom.yml` reached the frozen app; the compose stayed frozen | — | none |
|
||||
|
||||
**The three that changed what is known:**
|
||||
|
||||
1. **B1 produced the unattended HOLD** this project has never had — see `19-Q4-the-unattended-hold.md`.
|
||||
2. **B4 answered the single-flight question: there is none.** Five updates ran together and all
|
||||
ended honest. Slice 6 must decide whether that is what it wants.
|
||||
3. **B6 found an inconsistency R-524 already removed for the other case:** a held app keeps inviting
|
||||
the household to update and the button refuses. **R-625.**
|
||||
|
||||
Also proven for free, across a genuine power cut: the boot sweep met a held app after an unclean
|
||||
shutdown and deliberately left it alone — *„whatever is holding it owns its recovery"*.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — the morning after
|
||||
|
||||
**B1's held app, as a household would find it at breakfast.** The app page, the dashboard, the
|
||||
launcher and both backups pages were read in **both languages** and are saved as HTML in
|
||||
`bad-days/P4-morning-after/`.
|
||||
|
||||
**Is there ONE sentence that says what happened, since when, which copy holds what, and what to
|
||||
press?** Scored against Q4's recommended option:
|
||||
|
||||
| Q4 promises the household are told… | measured |
|
||||
|---|---|
|
||||
| **what happened** | ✔ „…frissítése 2026-09-21 21:53-kor nem sikerült, és az alkalmazás nem indult el az új verzióval." |
|
||||
| **since when** | ✔ the time is in the sentence |
|
||||
| **which copy holds what** | ✔ „saját meghajtó, 2026-09-21 21:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza." |
|
||||
| **what to press** | ✔ „Visszaállítható a Mentések oldalon…" |
|
||||
| **the same in English** | ✘ **the sentence is Hungarian on the English page** (R-606, confirmed on the hold sentence itself, with positive and negative controls) |
|
||||
| by mail | **unmeasurable on this venue** (R-620) |
|
||||
|
||||
**The app is surfaced everywhere, not only on its own page** — the banner „Telepített alkalmazás nem
|
||||
fut: …" / „An installed app is not running: …" appeared at the top of every authenticated page, and
|
||||
carried both held apps at once when there were two.
|
||||
|
||||
**Every app still on the box, and every badge, after a rescan.** Ten deployed apps: every badge is
|
||||
**TRUE** — references equal ⇔ „Naprakész", references differ ⇔ „Frissítés elérhető". `zipline` shows
|
||||
the household „Nem egészséges — URL nem elérhető" while it is in fact serving, which is R-618 in the
|
||||
household's own words.
|
||||
|
||||
---
|
||||
|
||||
## The alarm truth table
|
||||
|
||||
**The event-and-mail half of this night could not be measured, and that is a property of the venue,
|
||||
not an omission.** Guest 9202 has `hub.enabled: false`; every notifier entry point returns before it
|
||||
logs anything (`notify/notifier.go:269, :359, :917, :959, :1058`), so no hub event and no customer
|
||||
mail can be produced or observed there. The hub was **not** enabled on 9202 to get around this: that
|
||||
would register an unclaimed host at the live hub and could mail a real address, and the brief fences
|
||||
the hub. Filed as **R-620** (a disabled notifier should at least say which event it dropped).
|
||||
|
||||
So the table below scores the surfaces that DO exist on this box — the app page in both languages,
|
||||
the dashboard, and the controller's own log — against `08-alarm-ladder.md`.
|
||||
|
||||
| # | what happened | should it alarm, per `08` | what the box did | what the household could READ | verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | `adventurelog` held after a real failed edge — app **stopped** | **YES** — `stopped` is in `IsDownState` | classified `stopped`; the boot sweep refused to restart it | the hold sentence on the app page **and** a banner on every page | **correct** — but the SEND is unmeasurable here |
|
||||
| 2 | `tandoor` held the same way | **YES** | same | same, both apps in one banner | **correct**, same caveat |
|
||||
| 3 | `glance` held by the **unattended** update | **YES** | same, and honoured across a **power cut** | same | **correct**, same caveat |
|
||||
| 4 | `tandoor` and `zipline` reading `unhealthy` for hours while SERVING | **NO** — `08` §4 puts `unhealthy` deliberately in the not-down set | did not alarm | „Nem egészséges — URL nem elérhető" on the dashboard | **the ladder is right and the outcome is still wrong** — see below |
|
||||
| 5 | pull failure (B2) — app kept running the old version | **NO** — nothing is down | did not alarm | one sentence on the card | **correct** |
|
||||
| 6 | five updates at once (B4) | **NO** | did not alarm | nothing | **correct** |
|
||||
| 7 | two power cuts (B5) | `restarting` is not down until sustained | recovered; nothing alarmed | one interrupted sentence in one case, nothing in the other | **correct** |
|
||||
| 8 | disk under the 2 GB floor (B7) | not an app-down state | refused the update; no alarm | the refusal sentence | **correct** — though a box at 1.4 GB free is arguably worth telling someone about, and nothing does |
|
||||
|
||||
**Which alarm fired and was it true:** none fired, and none could — see the venue limit above. Every
|
||||
classification the box made was correct against `08`.
|
||||
|
||||
**Which should have fired and did not:** on this evidence, none. Row 8 is the only candidate and it
|
||||
is a design question rather than a defect: `08` is an *app-down* ladder and a full disk is not an app
|
||||
being down.
|
||||
|
||||
**The one that matters, and it is row 4.** `08` §4 deliberately excludes `unhealthy` — *"folding it
|
||||
in reintroduces the flapping fix-3 was written to stop"* — and that ruling is right. **But the same
|
||||
probe result the alarm ladder correctly ignores is NOT ignored by the guarded update's `verifying`
|
||||
phase, which waits on it and then stops the app.** One probe, two consumers, opposite tolerances,
|
||||
and neither document says so. That asymmetry is the whole of R-618's severity.
|
||||
|
||||
---
|
||||
|
||||
## The promotion list for the operator
|
||||
|
||||
**CC promotes nothing.** These are the real, within-a-major edges that ended `proven` on
|
||||
the box tonight, with the data read back through the app's own front door both before and
|
||||
after. Moving each of them on the LIVE catalog is the operator's call.
|
||||
|
||||
| app | the move | what it would mean for a box in the field |
|
||||
|---|---|---|
|
||||
| `actualbudget` | actual-server:26.7.0 → actual-server:26.9.0 | the app runs its own schema migration on the way — proven here, and the update takes a backup first |
|
||||
| `audiobookshelf` | audiobookshelf:2.35.1 → audiobookshelf:2.36.1 | the app runs its own schema migration on the way — proven here, and the update takes a backup first |
|
||||
| `bookstack` | bookstack:26.05.2, mariadb:12.3 → bookstack:26.05.5, mariadb:12.3 | no migration line printed; the app came up on the new version with its data intact |
|
||||
| `docmost` | docmost:0.95.0, postgres:16-alpine, redis:7-alpine → docmost:0.96.0, postgres:16-alpine, redis:7-alpine | the app runs its own schema migration on the way — proven here, and the update takes a backup first |
|
||||
| `grafana` | grafana:13.1.0 → grafana:13.2.2 | the app runs its own schema migration on the way — proven here, and the update takes a backup first |
|
||||
| `home-assistant` | home-assistant:2026.7.2 → home-assistant:2026.9.3 | no migration line printed; the app came up on the new version with its data intact |
|
||||
| `mealie` | mealie:v3.20.1 → mealie:v3.27.0 | the app runs its own schema migration on the way — proven here, and the update takes a backup first |
|
||||
| `n8n` | n8n:2.31.3 → n8n:2.40.5 | the app runs its own schema migration on the way — proven here, and the update takes a backup first |
|
||||
| `navidrome` | navidrome:0.63.2 → navidrome:0.64.0 | the app runs its own schema migration on the way — proven here, and the update takes a backup first |
|
||||
| `nextcloud` | mariadb:11.6 → mariadb:12.3 | no migration line printed; the app came up on the new version with its data intact |
|
||||
| `papra` | papra:26.6.1-rootless → papra:26.6.2-rootless | the app runs its own schema migration on the way — proven here, and the update takes a backup first |
|
||||
| `privatebin` | pdo:2.0.5 → pdo:2.0.6 | no migration line printed; the app came up on the new version with its data intact |
|
||||
| `romm` | mariadb:11.4, redis:7-alpine, romm:5.0.0 → mariadb:11.4, redis:7-alpine, romm:5.3.0 | the app runs its own schema migration on the way — proven here, and the update takes a backup first |
|
||||
| `vikunja` | vikunja:2.3.0 → vikunja:2.6.0 | the app runs its own schema migration on the way — proven here, and the update takes a backup first |
|
||||
|
||||
**And the apps that must NOT be promoted, which is the other half of the list:**
|
||||
|
||||
| app | the move | why not |
|
||||
|---|---|---|
|
||||
| `adventurelog` | `v0.12.1 → v0.13.0` | applies **nine database migrations successfully** and then never binds its port. Held after the full health wait; restored in 75 s. **R-622** |
|
||||
| `tandoor` | `2.6.13 → 2.6.15` | the update SUCCEEDS — the app served HTTP 200 on the new version for five minutes — and is then stopped by the template's own wrong health port. Fix **R-618** first; the edge itself is probably fine |
|
||||
| `postgres:16-alpine → 17-alpine`, anywhere | the engine | refuses to start on a 16 datadir, verbatim. The engine-major gate stays until Q5's conversion exists |
|
||||
|
||||
**`nextcloud`'s MariaDB `11.6 → 12.3` is proven and is a different kind of entry**: it is not an app
|
||||
version but an engine major, and §3 decision 5 plus R-469 already permit it. It is listed here
|
||||
because tonight is the first time it has been pressed through the button a household presses.
|
||||
|
||||
---
|
||||
|
||||
## Teardown — three layers, plus Gitea
|
||||
|
||||
### Machine — guest 9202
|
||||
|
||||
Every throwaway app removed **through the product**, with its data where the product allowed it.
|
||||
Three apps carrying an `HDD_PATH` were REFUSED at „remove with data" — `/api/disks` answers
|
||||
`agent not configured` on this guest, so the drive path cannot be resolved and R-442's fail-closed
|
||||
guard keeps the app rather than half-deleting it. Each was then removed with the data KEPT, which
|
||||
the product does accept, and the harness's own directories were removed by name afterwards.
|
||||
|
||||
```
|
||||
containers now: felhom-controller · filebrowser · traefik <- the three protected only
|
||||
drill images: (none)
|
||||
drill volumes: (none)
|
||||
registry:2: removed, with its volume
|
||||
scratch drive: documents · downloads · media · roms <- the drive's own folders
|
||||
free space: 33 G on the docker root
|
||||
```
|
||||
|
||||
**`controller.yaml` restored** from `controller.yaml.pre-update-night`, the controller restarted,
|
||||
and the cache's origin **read back and quoted** — which is the point of the exercise:
|
||||
|
||||
```
|
||||
origin https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (fetch)
|
||||
origin https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (push)
|
||||
4463243 Upgrade harness: four fixtures and seven real upstream edges from the update night (R-462)
|
||||
```
|
||||
|
||||
**The teardown found the night's last defect**, which is the argument for doing it properly: a
|
||||
`navidrome` container that the product's own removal had left behind and Docker's restart policy had
|
||||
resurrected, invisible to every sweep that keys on `deployed`. Removed by name. **R-626.**
|
||||
|
||||
### Host — demo-hp
|
||||
|
||||
**No harness LXC was created tonight, so none was destroyed** — the PostgreSQL rehearsal ran on 9202
|
||||
itself with plain `docker` beside the product. `pct list` and `pvesm status` before and after are in
|
||||
`teardown/00-host-before.txt` and `04-host-after.txt`. **Guest 9201 untouched** — container count
|
||||
unchanged, and it was never addressed except to read its catalog cache for the negative control.
|
||||
|
||||
### Hub
|
||||
|
||||
**Nothing provisioned: no customer, no config, no appliance, no binding.** The only hub act of the
|
||||
whole night was the floor save in Phase 0.1. Final state: floor `0.261.0`, declared MinAgent
|
||||
`0.131.0`, three host rows — unchanged from the start except the floor the operator asked for.
|
||||
|
||||
### Gitea
|
||||
|
||||
```
|
||||
live catalog origin/main : 4463243f2e09
|
||||
drill repo HEAD after reset : 4463243f2e09
|
||||
diff of every `image:` line, live vs drill:
|
||||
IDENTICAL — every image: line matches the live catalog
|
||||
```
|
||||
|
||||
**One later commit, after the teardown was taken:** the vikunja fixture's create verb was corrected
|
||||
from `POST` to `PUT` (test code only), so the live catalog's `main` now reads `d4392e2a10f4`. The
|
||||
check that matters is unchanged and was re-run afterwards — `git diff f5f6a152b513 origin/main --
|
||||
templates/` is **empty**: **not one `image:` line moved on the live catalog at any point in the
|
||||
night.** CI green by id for both catalog pushes (jobs 830 and 872) and for felhom.eu (job 870).
|
||||
|
||||
The drill repo is **KEPT**, private, and reset to the live catalog's `main`, so the next drill starts
|
||||
clean. **The live catalog's `main` moved once tonight** — from `f5f6a152b513` to `4463243f2e09` — and
|
||||
that commit changes `scripts/` only: four harness fixtures and seven edge definitions. **Zero
|
||||
`image:` lines moved on the live catalog at any point in the night**, which the diff above proves
|
||||
rather than asserts.
|
||||
|
||||
---
|
||||
|
||||
## Claims in the brief that turned out wrong
|
||||
|
||||
The brief asked for this explicitly. Each claim, and what was measured.
|
||||
|
||||
| the brief said | measured |
|
||||
|---|---|
|
||||
| **a box will follow a second catalog by `git.repo_url` alone** (read from config source, never run) | **WRONG.** `Syncer.gitCloneOrPull` clones only when `.git` is absent; otherwise it fetches from the remote the clone already stores. The cache directory had to be removed too. **R-615** |
|
||||
| **`CompareImageRefs` may not order references carrying a `host:port/` prefix** (read, not run) | **The worry was unfounded.** `splitImageRef` takes the LAST colon and rejects it only when a `/` follows, so a registry port is never read as a tag. Proven by RUNNING it: four positive cases and a negative control |
|
||||
| **PostgreSQL 17 refuses a 16 datadir and the update ends HELD with data intact** (R-463 and source, not measured) | **RIGHT, and now measured** — 5.1 s to held, pin on 17 with nothing running, data intact, restore back in 29.1 s. The refusal line itself had to be reproduced because the product destroyed it (**R-621**) |
|
||||
| **a MariaDB sidecar major through the BUTTON behaves as it did on the harness** | **RIGHT.** All four observables, including the conversion actually running rather than being skipped, and the engine's own pre-upgrade backup |
|
||||
| **9202 has the capacity for this** | **RIGHT.** 26 GB RAM, 56 GB free on the docker root at the start, a 938 GB scratch drive. Peak usage never threatened it; images were reclaimed BY NAME twice, never pruned |
|
||||
| **39 within-a-major edges still exist upstream tonight** | **RIGHT, exactly.** The drift script re-run at 20:07 returned 66 pins, 46 behind, **39 within a major and 7 across** — the same numbers |
|
||||
|
||||
**And two more the brief did not name, found the same way:**
|
||||
|
||||
- `update-arc-gaps-2026-09-21/00-api-recipe.md` said the app page is `/app/<n>`. **It is `/apps/<n>`**,
|
||||
and every call that recipe described 404s. Corrected in that file.
|
||||
- `unattended-caller.py`'s `follow()` read the API **envelope**, so every update it followed would
|
||||
have run to a 900 s timeout and been recorded `timeout` rather than `held`. **R-623**, fixed before
|
||||
B1 relied on it — and B1's log is what the fixed version produces.
|
||||
|
||||
**One correction to a register row, which is the same class of error one layer up.** R-606 records
|
||||
controller v0.260.0 as having made the pre-flight refusals reach an English household in English.
|
||||
**Measured: it did not.** `held`, `not_deployed` and `disk` all come back identical Hungarian with
|
||||
`?lang=en`, because the routing exists and the sentences are frozen string constants that never
|
||||
entered it. A row that records something as fixed when it is not is worse than an open row.
|
||||
@@ -0,0 +1,69 @@
|
||||
# DRILL — 2026-09-26/27: a backup's data and its version travel together; the next PostgreSQL apps; adventurelog; leftovers
|
||||
|
||||
Brief: "a backup's data and its version always travel together; the next PostgreSQL apps; adventurelog; small leftovers".
|
||||
Architecture read before any claim: `07-backup-architecture.md` §6 (now §6.6 added), `09-update-architecture.md` §3
|
||||
decisions 13–42, §5.3, §6.4 parts 4, 5, 10. Evidence: `audits/version-travel-2026-09-26/`.
|
||||
|
||||
## Not done, or changed
|
||||
|
||||
- **One controller release (v0.275.0), not two**, with Parts A, D2–D4 and **R-699** (found live during Part B, filed and
|
||||
fixed). Live proofs ran on `0.275.0-rc1` (identical except R-699); 0.275.0 itself ran the adventurelog box proof.
|
||||
- **The session paused ~17 h** (usage limit) between the build and the live proof. In that gap the box's own night ran the
|
||||
engine step on docmost, which became the shape-2 live proof — observed, not pressed.
|
||||
- **R-691 (2) NOT built** — Load does not look at the off-site copy. (1) built and red-proofed, **not live-proven** (no
|
||||
nextcloud kept folder existed on 9202).
|
||||
- **Part B moved 2 apps** (paperless-ngx, tandoor). calcom and claper were not attempted (no seed route in the fixtures).
|
||||
- **adventurelog's health fix is a DROP of the override, not `/usr/bin/node`**: no single node path works on both versions
|
||||
(measured). It rides the image move in the same commit, as asked.
|
||||
- **Off-site restore NOT exercised live** (9202 has no off-site target; the boxes that do are fenced). Unit-tested.
|
||||
- **The conversion-copy RELEASE on a real 18 dump not observed** — due on 9202 after tonight's 00:30 data run.
|
||||
- **D1 took THREE agent releases.** Instead of waiting 6 h for the scheduled cycle, the scheduler's own read-only verdict
|
||||
(`-selftest=restore-test-due`) was read on demo-hp: the golden was skipped, but the pick fell to a leftover archive of
|
||||
guest 9100, deleted in August. Fixed in **v0.136.0** (the guest must exist) — whose lookup then met PVE's 403 for a guest outside the agent's pool and made the local
|
||||
tier UNKNOWN (my regression, read the same way); fixed in **v0.137.0**, verified read-only on demo-hp with the pre-release
|
||||
binary before the release. Each red-proofed and signed-delivered to both hosts.
|
||||
- **Part E (night watch) not done.**
|
||||
- D4 is **not ruled**: option A is built and in force (STATUS).
|
||||
|
||||
## Claims in the brief, judged
|
||||
|
||||
1. *`CaptureRecoveryUnit` rewrites `compose/` and `image_pins` before the data is refreshed* — **TRUE** (measured: the
|
||||
refresh 2 min after the update rewrote both; the dumps stayed).
|
||||
2. *A restore replays the database's raw volume tar before the SQL dump* — **TRUE** (`restore_unit.go`; live: the 16 datadir
|
||||
tar replaced the volume first).
|
||||
3. *The 18 definition over a 16 tar does X* — **REFUSED by the engine, and the app was left DOWN**: `postgres:18` refused the
|
||||
old layout, the replay timed out, the start failed. Not "a fresh cluster with the SQL".
|
||||
4. *A restored version older than the ladder would jump* — **TRUE for a person's press, FALSE for the automatic leg**
|
||||
(`LegSkipOlderThanLadder` already refuses). The page now says the box will not update it by itself.
|
||||
5. *`07` says nothing on which version a restore brings back* — **TRUE** for `07` (only `09` §5.3 said the restore pins to
|
||||
the unit's compose). `07` §6.6 now says it.
|
||||
|
||||
## A — the data and its version (R-696)
|
||||
|
||||
A1 spike (`A1/README.md`) → A2–A4 built (stamps, the definition kept with its data, restore guard, off-site definition,
|
||||
data times on all three tiers, release needs a dump on the new major) → red-proofs RP1–RP6 (`A5-redproofs/`) → live
|
||||
(`A5-live/README.md`): shape 2 in the box's own night (restore offered 00:30 = the data; restore back whole at 16 with the
|
||||
sentence „A(z) docmost visszaállt a(z) 2026-09-27 02:30-i mentésből, a(z) docmost:0.96.0, postgres:16-alpine,
|
||||
redis:7-alpine verzióra. Elérhető frissítés: a doboz lépésenként hozza naprakészre."; climb again 55 s); shape 1 (back
|
||||
whole at 0.95.0, "No pending database migrations"; climb 46 s). A7: 42 of 42 ladder digests resolve; filed R-698.
|
||||
|
||||
## B, C, D
|
||||
|
||||
B: `B/README.md` — paperless-ngx → 18 and tandoor → 17, both venues, engine gate ALLOWED, catalog `46affe0`, `53b84ca`.
|
||||
C: `C/README.md` — adventurelog v0.13.0, catalog `06ea7da`. D1 agent v0.135.0 → v0.137.0 (signed delivery to both hosts); D2 R-695;
|
||||
D3 R-691 (1); D4 R-694 (per-app measurement, 6 of 7 keep the login in their data).
|
||||
|
||||
## Teardown
|
||||
|
||||
- **Machine:** bench LXC 9401 created and destroyed (hostname checked), template removed. 9202: docmost, tandoor and
|
||||
adventurelog removed through the product; paperless-ngx (standing) now on PostgreSQL 18 like the live catalog, its kept
|
||||
16 copy waiting for the release; back on the LIVE catalog (`controller.yaml.pre-vt0926`), controller 0.275.0. Drill repo
|
||||
reset to live `main` (`f1a7d6c`).
|
||||
- **Host:** demo-hp `pct list` 9201 + 9202 only; `local` and `nvme-scratch` as before the bench.
|
||||
- **Hub:** global floor 0.275.0 (declared MinAgent 0.131.0); six signed `agent_update` jobs (0.135.0, 0.136.0, 0.137.0 × 2 hosts); no customer or
|
||||
appliance created.
|
||||
|
||||
## Register
|
||||
|
||||
Before: **339 rows / 682,159 B** (the brief's count). Opened R-697, R-698, R-699; closed R-655, R-689, R-694, R-695, R-696,
|
||||
R-699 (moved to `CLOSED-ITEMS.md`); narrowed R-691 (to its (2)) and R-463 (3 of 11 moved). After: **336 rows / 678,058 B**.
|
||||
@@ -0,0 +1,247 @@
|
||||
# THE MORNING AFTER — probe fix, gate, and the promotion train, 2026-09-22
|
||||
|
||||
Evidence: `probe-fix-2026-09-22/`. The night this follows: `DRILL-update-night-2026-09-21.md`.
|
||||
|
||||
---
|
||||
|
||||
## Not done, or changed from the brief
|
||||
|
||||
**Read this first. Everything else in this document is a claim; this section is where the claims
|
||||
are bounded.**
|
||||
|
||||
1. **The brief said "the fourteen proven versions". FIFTEEN moved.** The night's fourteen included
|
||||
`nextcloud`'s MariaDB engine move; tandoor was re-walked today and became the fifteenth. Both are
|
||||
named below with why.
|
||||
|
||||
2. **I nearly dropped `nextcloud`'s engine move on a wrong assumption, and the gate corrected me.**
|
||||
I assumed `check-engine-major.py` would refuse `mariadb:11.6 -> 12.3` and had written it up as
|
||||
"dropped, named". Running the gate against that exact commit instead of assuming it ALLOWED the
|
||||
move by name, citing **R-469** (Slice 4 shipped; `MARIADB_AUTO_UPGRADE=1` is already in that
|
||||
template, verified by reading it). The brief forbade **PostgreSQL** engine moves; this is MariaDB
|
||||
and it was proven end to end in 217.4 s. It moved. **This is the second time in two days that
|
||||
running the instrument beat reasoning about it** — see R-628.
|
||||
|
||||
3. **The first push of the fifteen moves FAILED CI, and I found it by checking rather than by being
|
||||
told.** Job **877** on `15d7c2b` read `conclusion: failure`: the CI runner has no PyYAML and the
|
||||
new gate answered INCONCLUSIVE, which the runner rightly refuses to call a pass. Fixed with a
|
||||
degraded line-reader mode and five more decoy cases; job **878** on `1ad1f34` is `success`.
|
||||
**The moves themselves were never in doubt — the gate that shipped beside them was.**
|
||||
|
||||
4. **bookstack's phase trace on demo-hp is INCOMPLETE and the reason is my own instrument.** A
|
||||
115-second probe run, meant only to list which apps were installed, pressed the Update as
|
||||
designed and then hit its timeout mid-update. Its phases are recorded to `+21.6 s starting`; the
|
||||
rest was read off the box afterwards (`done`, pin and installed both at `26.05.5`). The second,
|
||||
full run then pressed Update on an already-current app and completed in **3.1 s**, moving
|
||||
nothing. That second trace is real and is reported, but it is **not** a `26.05.2 -> 26.05.5`
|
||||
trace. Said here rather than presented as one.
|
||||
|
||||
5. **The brief asked me to "list templates the night's sweep could not judge". The honest answer
|
||||
needed three categories, not one** — 20 with a verdict record, 4 deployed as props with no edge
|
||||
walked, 1 deployed only to measure its probe, and **28 never deployed at all**. Filed as R-632
|
||||
with the machine-readable list.
|
||||
|
||||
6. **Two things the brief could not have known, both found by the new gate and both left OPEN:**
|
||||
`paperless-ngx`'s health probe has never run on any box (R-630), and five templates cannot be
|
||||
judged by any static rule (R-631). Neither is fixed here: R-630's fix is product code or a
|
||||
container rename on live boxes, and the brief forbade product code.
|
||||
|
||||
**One brief claim that turned out wrong:** none. All three probe faults the brief named were
|
||||
verified against the files before any edit and all three were exactly as stated — tandoor's compose
|
||||
line 42 dials `127.0.0.1:80/accounts/login/`, zipline's line 38 dials `/api/healthcheck`, wger's
|
||||
line 42 dials `127.0.0.1:8000`.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — the probes
|
||||
|
||||
### 1.1 The fix
|
||||
|
||||
One commit, `app-catalog-felhom.eu@793c4fb`. No `image:` line moved, so no `catalog_since` moved.
|
||||
|
||||
| app | was | is | the oracle that was already in the file |
|
||||
|---|---|---|---|
|
||||
| tandoor | port `8080` | port `80` | `wget --spider -q http://127.0.0.1:80/accounts/login/` |
|
||||
| zipline | path `/api/health` | path `/api/healthcheck` | `http.get('http://127.0.0.1:3000/api/healthcheck', …)` |
|
||||
| wger | port `80` | port `8000` | `wget --spider -q http://127.0.0.1:8000` |
|
||||
|
||||
### 1.2 Red-proofed live on 9202, through the product, in both directions
|
||||
|
||||
Deployed at the **live (unfixed) pin** first. Evidence `probe-fix-2026-09-22/probe-before-observe.json`,
|
||||
`front-door-before.json`, `probe-after.json`.
|
||||
|
||||
| | tandoor | zipline | wger |
|
||||
|---|---|---|---|
|
||||
| **BEFORE — controller state** | `unhealthy` | `unhealthy` | `unhealthy` |
|
||||
| **BEFORE — the app page, HU** | `Nem egészséges` | `Nem egészséges` | `Nem egészséges` |
|
||||
| **BEFORE — the app page, EN** | `Not healthy` | `Not healthy` | `Not healthy` |
|
||||
| **BEFORE — docker's own healthcheck** | `healthy, healthy` | `healthy, healthy` | `healthy` |
|
||||
| **BEFORE — the front door, followed** | **200** `Login / Sign In` | **200** `Zipline` | **200** `wger Workout Manager` |
|
||||
| **AFTER — controller state** | `running` | `running` | `running` |
|
||||
| **AFTER — the app page, HU / EN** | `Fut` / `Running` | `Fut` / `Running` | `Fut` / `Running` |
|
||||
|
||||
The fix arrived through the **real sync**: `POST /api/sync` answered
|
||||
*„Sablonok frissítve — frissítve: tandoor, wger, zipline"*, and all three read `running` at the next
|
||||
poll — **no redeploy, no container restart**.
|
||||
|
||||
**The badge I quote is the HEALTH word, not the version badge.** The version badge reads
|
||||
`Naprakész` / `Up to date` in both states and would have proved nothing; a first pass captured it
|
||||
and it was corrected. The scan strips `<script>` and `<style>` first (the same page carries those
|
||||
words inside a script block).
|
||||
|
||||
### 1.2b tandoor's edge, re-walked
|
||||
|
||||
Via the drill catalog (`09` §6.5 — `git.repo_url` **and** `rm -rf <data>/catalog-cache`, R-615).
|
||||
Evidence `probe-fix-2026-09-22/tandoor-rerun/`.
|
||||
|
||||
| | 2026-09-21, before the fix | 2026-09-22, after |
|
||||
|---|---|---|
|
||||
| entered `verifying` | +58.4 s | **+5.2 s** |
|
||||
| outcome | **`failed` at +361.9 s**, app STOPPED | **`done` at +41.1 s** |
|
||||
| seed read back after | **UNUSABLE** — `No such container: tandoor` | **found = True** |
|
||||
| `installed_images` | rolled back to `2.6.13` | `2.6.15` |
|
||||
| containers | none | both `running`, `restarts=0` |
|
||||
|
||||
**Same app, same versions, same button. The only change is one port number.** tandoor's verdict
|
||||
moves `failed -> proven` and it is on the live catalog.
|
||||
|
||||
**Teardown, stated:** `controller.yaml` restored, `git.repo_url` read back as the live catalog,
|
||||
`<data>/data/catalog-cache` re-cloned from the live remote and quoted
|
||||
(`probe-fix-2026-09-22/teardown-catalog-pointer.txt`), drill repo reset to the live `main`.
|
||||
|
||||
### 1.3 The gate
|
||||
|
||||
`app-catalog-felhom.eu/scripts/check-probe-matches-compose.py`, registered in `catalog_gates.py` as
|
||||
a **`--fast`** row, so it runs in the pre-push hook and in CI.
|
||||
|
||||
**The oracle was already in every template.** The probed service's own compose `healthcheck.test`
|
||||
dials the app on loopback. The gate resolves the probed service exactly as `findProbeContainer`
|
||||
does (`healthprobe.go:297`) and compares.
|
||||
|
||||
**Two verdicts, and the narrower one is a measurement.** A wrong PORT refuses for every check type —
|
||||
nothing listens, every dial is refused. A wrong PATH refuses **only** where the probe can fail on
|
||||
it: `probeHTTP` calls any response healthy for `type: http`, and for `type: api` with no `expect`
|
||||
block (`healthprobe.go:253-262`). So the path rule convicts `zipline` and merely warns about
|
||||
`home-assistant`, whose badge is right today and breaks the day someone adds an `expect`.
|
||||
|
||||
**Red-proofs, four:** on the unfixed tree (`d4392e2a10f4` — the hash the brief names, and the repo's
|
||||
HEAD when I started) it refuses all three; on the fixed tree it passes; a clean app given a wrong
|
||||
port is refused; an `api`+`expect` probe given a wrong path is refused. Each of the three real
|
||||
faults is **re-introduced by the test** rather than read off the tree, because reading them off the
|
||||
tree would have gone green for the wrong reason the moment R-618 was fixed.
|
||||
|
||||
**Decoys, both ways, ten in total** (suite now **56 cases**): the port moved in a YAML comment;
|
||||
Traefik's `loadbalancer.server.port`; a published `ports:` mapping; a NON-probed sidecar's own
|
||||
healthcheck; a path mismatch on a probe that cannot fail on it; and five more with **PyYAML shadowed
|
||||
out** — the mode CI actually runs — where the three faults must still be refused.
|
||||
|
||||
**`--root=<dir>`** exists so the suite judges its own clone. Without it every case would read
|
||||
identical bytes and pass for nothing: the constant-for-measurement shape this repo's suite already
|
||||
warns about.
|
||||
|
||||
**Run over all 53 — six WARNINGS, none a conviction and none a pass:**
|
||||
|
||||
| app | why no verdict |
|
||||
|---|---|
|
||||
| `paperless-ngx` | no container name matches the stack name — **the probe has never run** (R-630) |
|
||||
| `vikunja` | the probed service has no compose healthcheck — no oracle |
|
||||
| `crafty-controller`, `mealie`, `uptime-kuma` | the healthcheck runs through a python/script helper — no loopback URL to read |
|
||||
| `home-assistant` | path mismatch on a probe that cannot fail on it — right today, fragile (R-631) |
|
||||
|
||||
### 1.4 What the night's sweep could not judge
|
||||
|
||||
**28 of 53 templates have never been deployed by any drill** (R-632). Full list in the register and
|
||||
in `probe-fix-2026-09-22/not-judged.json`. Also: 4 were deployed as props in the bad-days legs with
|
||||
no edge walked (`bentopdf`, `glance`, `uptime-kuma`, `wishlist`), and 1 only to measure its probe
|
||||
(`wger`).
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — the promotion train
|
||||
|
||||
### The moves
|
||||
|
||||
Fifteen commits, one per app, `catalog_since: "2026-09-22"` on each. Every gate run before each commit; nothing forced, nothing dropped. `check-image-resolvable.py` confirms all 18 unique pins still exist upstream.
|
||||
|
||||
| app | move | commit |
|
||||
|---|---|---|
|
||||
| `actualbudget` | 26.7.0 -> 26.9.0 | `2060032` |
|
||||
| `audiobookshelf` | 2.35.1 -> 2.36.1 | `6525b8e` |
|
||||
| `bookstack` | 26.05.2 -> 26.05.5 | `ac4828f` |
|
||||
| `docmost` | 0.95.0 -> 0.96.0 | `6d8cd87` |
|
||||
| `grafana` | 13.1.0 -> 13.2.2 | `068f445` |
|
||||
| `home-assistant` | 2026.7.2 -> 2026.9.3 | `4405f12` |
|
||||
| `mealie` | v3.20.1 -> v3.27.0 | `008348b` |
|
||||
| `n8n` | 2.31.3 -> 2.40.5 | `4132e35` |
|
||||
| `navidrome` | 0.63.2 -> 0.64.0 | `7708a04` |
|
||||
| `papra` | 26.6.1 -> 26.6.2 | `e96887e` |
|
||||
| `privatebin` | 2.0.5 -> 2.0.6 | `f547f16` |
|
||||
| `romm` | 5.0.0 -> 5.3.0 | `15f9ebf` |
|
||||
| `tandoor` | 2.6.13 -> 2.6.15 | `c3807c7` |
|
||||
| `vikunja` | 2.3.0 -> 2.6.0 | `22f598b` |
|
||||
| `nextcloud` | **ENGINE** `mariadb:11.6` -> `mariadb:12.3` | `39374d5` |
|
||||
|
||||
### The guarded Update, pressed on both demo boxes
|
||||
|
||||
#### `demo-hp` — https://192.168.0.138
|
||||
|
||||
| app | badge before (HU / EN) | phases | pin after | installed after | front-door state |
|
||||
|---|---|---|---|---|---|
|
||||
| `bookstack` | `Naprakész` / `Up to date` | `safety-dump` +0.0s → `pulling` +1.1s → `starting` +2.1s → `done` +3.1s | `lscr.io/linuxserver/bookstack:26.05.5` | `lscr.io/linuxserver/bookstack:26.05.5` | `running` |
|
||||
| `docmost` | `Frissítés elérhető — ma` / `Update available — today` | `safety-dump` +0.0s → `pulling` +1.0s → `starting` +79.0s → `verifying` +91.3s → `done` +106.6s | `docmost/docmost:0.96.0` | `docmost/docmost:0.96.0` | `running` |
|
||||
| `privatebin` | `Frissítés elérhető — ma` / `Update available — today` | `safety-dump` +0.0s → `pulling` +1.1s → `starting` +4.1s → `verifying` +5.2s → `done` +15.4s | `privatebin/pdo:2.0.6` | `privatebin/pdo:2.0.6` | `running` |
|
||||
| `romm` | `Frissítés elérhető — ma` / `Update available — today` | `safety-dump` +0.0s → `pulling` +1.1s → `starting` +13.3s → `verifying` +40.0s → `done` +74.8s | `rommapp/romm:5.3.0` | `rommapp/romm:5.3.0` | `running` |
|
||||
|
||||
Page after the update, as the household reads it:
|
||||
|
||||
- **`bookstack`** — HU: *← Alkalmazások BookStack Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Egyszerű wiki platform - polcok, könyvek, fejezetek és oldalak ~150M RAM productivity Pi k*
|
||||
EN: *← Apps BookStack Running Up to date Open ↗ Log Export Settings A simple wiki platform - shelves, books, chapters and pages ~150M RAM productivity Runs on Pi Move to anoth*
|
||||
- **`docmost`** — HU: *← Alkalmazások Docmost Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Modern wiki és dokumentáció platform Notion-szerű szerkesztővel ~200M RAM productivity Csak *
|
||||
EN: *← Apps Docmost Running Up to date Open ↗ Log Export Settings A modern wiki and documentation platform with a Notion-like editor ~200M RAM productivity x86 only Move to an*
|
||||
- **`privatebin`** — HU: *← Alkalmazások PrivateBin Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Titkosított szöveg megosztás - a szerver nem látja a tartalmat ~30M RAM security Pi kompa*
|
||||
EN: *← Apps PrivateBin Running Up to date Open ↗ Log Export Settings Encrypted text sharing - the server never sees the content ~30M RAM security Runs on Pi Move to another st*
|
||||
- **`romm`** — HU: *← Alkalmazások RomM Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Retró játékgyűjtemény kezelő, böngésző és lejátszó ~300M RAM media HDD szükséges Csak x86 Hova *
|
||||
EN: *← Apps RomM Running Up to date Open ↗ Log Export Settings Retro game collection manager, browser and player ~300M RAM media Needs a hard drive x86 only Where do I put my *
|
||||
|
||||
#### `demo-felhom` — https://192.168.0.149
|
||||
|
||||
**Not installed on this box, so not pressed:** `bookstack`, `docmost`, `privatebin`, `romm`.
|
||||
|
||||
| app | badge before (HU / EN) | phases | pin after | installed after | front-door state |
|
||||
|---|---|---|---|---|---|
|
||||
|
||||
Page after the update, as the household reads it:
|
||||
|
||||
|
||||
|
||||
**The second box, honestly:** the four moved apps were pressed on **one** demo box, not two, because
|
||||
they exist on only one. Both boxes are on catalog `1ad1f34`.
|
||||
|
||||
---
|
||||
|
||||
## What this leaves open
|
||||
|
||||
| row | what |
|
||||
|---|---|
|
||||
| **R-630** (new, P2) | `paperless-ngx`'s probe has never run on any box. Not fixed: both candidate fixes are product code or rename a running container on live boxes. **Not measured, and said so:** what `verifying` does for a stack with no probe target. |
|
||||
| **R-631** (new, P3) | five templates no static rule can judge; `home-assistant` is right only because its check type cannot fail. |
|
||||
| **R-632** (new, P3) | 28 of 53 templates have never been deployed by any drill. This is the rotation's queue. |
|
||||
| **R-618** | **CLOSED** — three probes fixed, red-proofed live both ways, gate shipped with ten decoys, tandoor re-walked `failed -> proven`. |
|
||||
| **R-462** | widened: the night's `14 proven / 3 failed / 4 inconclusive` becomes **15 proven / 2 failed / 4 inconclusive**, and all fifteen are now on the live catalog. |
|
||||
|
||||
**`unproven.py --summary` did not move:** 55 claims, 20 walked, 35 not walked — the same as
|
||||
2026-08-22. Nothing here changed a claim's status; it changed the evidence behind one row of the
|
||||
capability map.
|
||||
|
||||
## Fences
|
||||
|
||||
**`Peti's box` is parked and was not touched.** It receives a badge if it ever syncs and nothing
|
||||
else. Nothing ran on DooPlex beyond ordinary pushes, nothing on `ep0`. `felhom-controller`,
|
||||
`felhom-agent` and the hub were read only — **no product code was written.** No golden, no bake, no
|
||||
vouch, no `--no-verify`, no branch. `local-lvm` untouched, no `prune`, `tester-1` never reset.
|
||||
|
||||
## Repository state
|
||||
|
||||
| repo | head | CI |
|
||||
|---|---|---|
|
||||
| `app-catalog-felhom.eu` | `1ad1f34` | job **878** `success` (job **877** on `15d7c2b` was `failure` — see "Not done" §3) |
|
||||
| `felhom.eu` | this commit | all 16 gates green locally |
|
||||
@@ -0,0 +1,101 @@
|
||||
# RETIRE — Peti's box (`peti-felhom`), 2026-09-25
|
||||
|
||||
Operator ruling 2026-09-25: Peti's box is retired (the tester wiped his server; it will not return); its off-site
|
||||
backup holds no user data and is to be deleted; it leaves the protected list. Architecture read first:
|
||||
`runbooks/target-selection.md`, `05-hub-architecture.md` §14 (what a customer delete deprovisions),
|
||||
`06-offsite-connectivity.md`, `04-control-plane-authorization.md`. Evidence: `audits/retire-peti-2026-09-25/`.
|
||||
|
||||
## Not done, or changed
|
||||
|
||||
- **Nothing of Peti's was on ep0.** No PBS namespace (`ns`: demo-felhom, demo-hp, tester-1), no WireGuard peer in
|
||||
the live `wg show`, no config naming him. His only off-site item was a Storage Box **sub-account**
|
||||
(`u629488-sub2`, id 269130, label `felhom-customer=peti-felhom`, home `felhom-peti-felhom`) on the pool box.
|
||||
- **The "backup" was never a backup.** The sub-account's folder held ONE file: `.ssh/authorized_keys`, 81 bytes. No
|
||||
restic repository was ever created — all 482 of his reports (2026-07-10 … 07-15) show 0 off-site snapshots and
|
||||
0 bytes; his escrow stayed pending. The operator's "no user data" is confirmed by the listing, not only by word.
|
||||
- **To list and empty the folder I reset the sub-account's password through the Hetzner API** (the product's own
|
||||
`reset_subaccount_password` action, scoped to id 269130 after re-reading its label). The key file was removed from
|
||||
inside the folder, then the hub deleted the sub-account. Whether Hetzner deletes a sub-account's folder with it
|
||||
is NOT measured here — so the folder was emptied first, and nothing of his could remain either way.
|
||||
- **Cloudflare:** his customer config carried a Cloudflare tunnel token and an API token (for `sajatfelhom.hu`).
|
||||
They were purged with the customer record. The hub's delete dialog says "tunnel/zone removed", but no leg of the
|
||||
cascade calls Cloudflare — the tunnel and DNS records on Cloudflare's side, if any remain, were NOT inventoried or
|
||||
removed (a row: R-688).
|
||||
- **A second, older Storage Box** (`PBS-storage-1`, u629193, box 611421) once held a folder `felhom-peti-spike/`
|
||||
(spike leftovers, per `RUNBOOK-ep0-datastore-volume-2026-07-27.md`). Its mount on ep0 is gone, and the box is
|
||||
not visible to either API token (404). The register row "delete the box" is the operator's. Not touched.
|
||||
- **Code comments that name Peti were left** (hub `rollup.go`, `appliances.go`, `notify/templates.go`; agent pbsdr
|
||||
tests). They explain why code is shaped as it is; no behaviour depends on Peti.
|
||||
|
||||
## A1 — inventory (before anything was removed)
|
||||
|
||||
| where | item | how found |
|
||||
|---|---|---|
|
||||
| hub | customer `peti-felhom` („Peti Proxmox", domain `sajatfelhom.hu`, dr_tier 0) with 482 reports, 123 events, 1,036 telemetry rows, DR recipe, one-time secret, claim, 1 log tail | copy of the hub DB (with its -wal), every table's `customer_id` / `host_id` |
|
||||
| hub | host `peti-felhom-86d37d` — already DELETED 2026-07-15 08:56:22 (`host_deletions`, escrow_acked 0); no host, guest, escrow, recovery or PBS-secret rows | same |
|
||||
| hub | WireGuard peer — none in `wg_peers` | same |
|
||||
| Storage Box | sub-account id 269130 `u629488-sub2`, home `felhom-peti-felhom`, label `felhom-customer=peti-felhom`, created 2026-07-10 | Hetzner API with the hub's own token, matched on the LABEL |
|
||||
| ep0 | nothing (namespaces, peers, `/etc`, `/root`, `/srv` grep — one false hit, "competing" in `lvm.conf`) | ssh read-only |
|
||||
| documents | 48 non-history files + workspace rules + 21 memory notes (whole-word pattern; positive control target-selection.md 7 hits, negative control catalog templates 0) | `A1-documents.txt` |
|
||||
|
||||
**Control, before:** pool box `size_data 3,120,562,176`, sub-accounts sub1 (demo-felhom), sub3 (demo-hp), sub4
|
||||
(tester-1); ep0 namespaces demo-felhom 2 snapshots / 220K, demo-hp 2 / 540K, tester-1 2 / 368K; demo boxes'
|
||||
off-site `last_status ok` (11 and 90 snapshots, runs 02:15:46Z / 02:18:41Z).
|
||||
|
||||
## A3 — removal
|
||||
|
||||
1. `POST …/subaccounts/269130/actions/reset_subaccount_password` → success (password kept 0600 in the scratchpad,
|
||||
deleted afterwards).
|
||||
2. As `u629488-sub2`: `rm .ssh/authorized_keys`, `rmdir .ssh` → `ls -la` empty, `du -s .` 1.
|
||||
3. Hub preview `GET /configs/peti-felhom/delete` — 0 hosts, off-site `u629488-sub2`, residue 1,519. Then
|
||||
`POST /configs/peti-felhom/delete` (ack_hosts, ack_reset, ack_purge, confirm_id, expect_hosts=0) → 303. The hub's
|
||||
log: sub-account 269130 deprovisioned; PBS tenancy `existed=false`; claim reset; residue purged (reports 482,
|
||||
telemetry 1,036, log tails 1); cascade COMPLETE (journal #20).
|
||||
|
||||
## A4 — nothing else moved
|
||||
|
||||
| item | before | after |
|
||||
|---|---|---|
|
||||
| sub-accounts | sub1, sub2 (Peti), sub3, sub4 | sub1, sub3, sub4 — labels and homes unchanged |
|
||||
| pool box `size_data` | 3,120,562,176 | 3,120,562,176 |
|
||||
| login as `u629488-sub2` | worked | "Permission denied" |
|
||||
| ep0 namespaces (snapshots / du) | demo-felhom 2/220K, demo-hp 2/540K, tester-1 2/368K | identical |
|
||||
| ep0 WireGuard peers | .2 .3 .4 .250 | identical |
|
||||
| hub hosts | demo-felhom, demo-hp, drill-r50 | identical |
|
||||
| hub customers | demo-felhom, demo-hp, drill-r50, peti-felhom, tester-1 | without peti-felhom |
|
||||
| hub rows naming peti | — | events 124, notification_log 80, host_deletions 1, customer_resets 1 (audit, by design), app_log_issues 8 (R-244) |
|
||||
|
||||
## A5 — documents changed
|
||||
|
||||
`runbooks/target-selection.md` (Tier 2 = DooPlex + ep0, ep0's reason rewritten, Peti's section retired); the
|
||||
unprompted-work rule (4 identical copies); `03`, `06`, `00-capability-map`; runbooks `TASK-identity-only-escrow`
|
||||
(obsolete note), `RUNBOOK-vzdump-target-move` (row 9), `RUNBOOK-island-migration`; retired banners on
|
||||
`pilot/PETI-tester-agreement.md`, `RUNBOOK-peti-return`, `RUNBOOK-peti-pbsdr`; register (PETI closed as retired,
|
||||
R-530, R-244, R-600 annotated); CONTEXT; STATUS; memory notes. Historic audits, tests and CHANGELOGs keep their text.
|
||||
|
||||
## Claims in the brief
|
||||
|
||||
1. *Peti's backup is on ep0* — **WRONG.** ep0 held nothing of his; the item was a Storage Box sub-account.
|
||||
2. *The hub's host delete removes the WireGuard peer* — **not testable here:** the host was deleted in July and
|
||||
ep0 carries no peer for it now; whether that delete removed one is not recorded (R-600 annotated).
|
||||
3. *The backup holds no user data* — **TRUE, and stronger:** it held no backup at all (one 81-byte key file).
|
||||
|
||||
## Part B — controller v0.272.0 (`44ae4de`), floor 0.272.0
|
||||
|
||||
Red-proofs, each seen failing (`retire-peti-2026-09-25/B/redproof-*.txt`): R-670 (undo read with the validating
|
||||
loader → "the undo logged a false backup-block ERROR"), R-671 (remover call dropped → "copies were left behind"; wiring
|
||||
dropped → "main.go never calls SetUndoCopyRemover"), R-677 (always catalog_since → "got 3 days, want 0"), R-685 page
|
||||
(line dropped → "a space skip must be said on the page"). Full suite + controller gates green; parity unchanged.
|
||||
|
||||
Live on 9202, drill catalog (`B/live/`): navidrome installed at 0.64.0 with a wrong probe; the step to 0.64.1 failed and
|
||||
its undo failed → HOLD, 1 undo copy kept; **R-670:** 0 "backup block rejected" lines while the undo ran (my first count
|
||||
of 3 "unreadable" was disk-watch lines — corrected in `r670-verdict.txt`); **R-671:** the backup page's restore →
|
||||
"update hold … CLEARED", "removed 1 undo cop(y/ies)", 0 copies left, the seeded user read back; **R-677:** the head
|
||||
re-tested at a new digest → „Frissítés elérhető — ma" / "Update available — today" (catalog_since two days old).
|
||||
**R-685 page line: not live** — 9202 has no agent, and no demo box is short of space. Teardown: navidrome removed
|
||||
through the product (0 volumes, 0 copies), 9202 back on the live catalog, drill reset to live `main`.
|
||||
Floor saved 09:31:31Z; both demo boxes on 0.272.0 at 09:31:55Z; demo-hp's backup page renders in both languages.
|
||||
|
||||
## Part C — the night watch: NOT REACHED
|
||||
|
||||
The session ended in the daytime (≈11:40 CEST).
|
||||
@@ -0,0 +1,357 @@
|
||||
# UPDATE ARC — the state, measured (2026-09-21)
|
||||
|
||||
Phase 0 of the resumed update arc. **Nothing here is an estimate.** Every number was read from a
|
||||
live box, a live registry or live source on 2026-09-21. Raw captures:
|
||||
`audits/update-arc-2026-09-21/`.
|
||||
|
||||
Baselines at the start: controller `19ef0329ab66` v0.259.0 · agent `d9864a94bf62` v0.132.0 ·
|
||||
felhom.eu `bcdd5b205875` hub v0.119.0 · catalog `18a6d2d8243e`. All four trees clean and level with
|
||||
`origin/main`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Three claims in the brief that turned out wrong — named first
|
||||
|
||||
**1.1 R-589 is NOT open. It shipped in v0.258.0.** The brief said, reviewer-verified, that
|
||||
`updatebadge.go` L82–L99 builds the badge from four raw Hungarian literals and that R-589 is
|
||||
therefore open. **The literals are real and the conclusion does not follow.** Those literals are the
|
||||
Hungarian form, and they are deliberately frozen — that IS the localisation parity guarantee. The
|
||||
ENGLISH form has been rebuilt from the bundle in `web.localeFuncs` (`internal/web/i18n_web.go`,
|
||||
the `"updateBadge"` entry) since **v0.258.0**, with `badge.update.*` in both `hu.json` and `en.json`,
|
||||
pinned by `TestUpdateBadgeFollowsTheLanguage`, and **proven live on a fresh box the same morning** —
|
||||
`DRILL-first-hour-en-0258-2026-09-20.md` item 9 reads *"PASS — 'Up to date' in English (R-589, fixed
|
||||
this morning, proven on a fresh box)"*, and R-561's own row records R-589 among the three defects
|
||||
Part 0 of v0.258.0 fixed.
|
||||
|
||||
**The register row is stale, not the code.** It is closed in this session with that citation. The
|
||||
lesson is the general one: *a reviewer who reads one producer cannot see a second producer that
|
||||
overrides it.* Reading `updatebadge.go` alone gives exactly the wrong answer, and the file now says
|
||||
so in its own comment.
|
||||
|
||||
**1.2 The chaos-night canary is NOT a defect to fix.** The brief asked why
|
||||
`check-image-resolvable` and `check-volume-persistence` returned INCONCLUSIVE on 2026-09-17 and
|
||||
whether the fix is bounded. **They behaved exactly as designed.** Both scripts' own headers state the
|
||||
rule: a detector that cannot prove itself must refuse to report rather than guess —
|
||||
`check-image-resolvable.py` cites the 2026-07-21 incident where treating a Docker Hub throttle as
|
||||
failure swept 24 of 65 pins as falsely dead, and the phrase in the drill doc, *"its own canary
|
||||
failed"*, is `check-volume-persistence.py`'s own literal error text (its `self_test`, which needs to
|
||||
`docker build` a canary image and therefore needs registry access). The most-supported reading is
|
||||
that registry access was constrained at that moment; **that is inferred from the code plus the
|
||||
documented throttle precedent, not observed** — no raw stdout of the gate run survives in either
|
||||
evidence directory. Round 7's response — substitute `use` for `update`, log the deviation, leave the
|
||||
tree clean — is what the scripts' own documentation asks for.
|
||||
|
||||
**No register row named it, and there is one real gap**, which is diagnostic rather than behavioural:
|
||||
`catalog_gates.py` collapses *"the harness refused to run at all"* and *"a per-app result is
|
||||
genuinely undetermined"* into one `INCONCLUSIVE` label, so a reader cannot tell which happened
|
||||
without re-running with output captured. Filed as **R-605**.
|
||||
|
||||
**1.3 "The hub report carries no image field" — CONFIRMED, on both sides, with one useful nuance.**
|
||||
The brief confirmed the controller half (`internal/report/types.go` L98–103: name, state, CPU,
|
||||
memory). The hub half was this session's to check. `Store.SaveReport`
|
||||
(`hub/internal/store/store.go:965`) **stores the raw report JSON whole** and denormalises only a
|
||||
fixed list out of it — controller version, URL, language, CPU/memory percent, container **total and
|
||||
running counts**, last snapshot, health status. `Containers` is two integers; there is no per-app
|
||||
structure anywhere, and no hub code reads an image tag.
|
||||
|
||||
**The nuance matters for costing Slice 7:** because the raw payload is kept whole, a new
|
||||
controller-side field would already be *stored* the day the controller sends it — what is missing is
|
||||
the denormalisation and the page, not the transport. That makes Slice 7 additive on both sides, as
|
||||
`09` §8.9 says, and slightly cheaper than "a hub-side change" suggests.
|
||||
|
||||
---
|
||||
|
||||
## 2. How far behind are the two demo boxes? Not at all.
|
||||
|
||||
Read on-disk on both boxes (`app.yaml` `installed_images` against the syncer's own catalog clone at
|
||||
`<data>/catalog-cache/templates/<app>/docker-compose.yml` — the exact two inputs
|
||||
`stacks.CatalogOrder` reads), controller v0.259.0 on both, catalog cache at `18a6d2d8` on both.
|
||||
|
||||
| box | deployed apps | behind | unknown (no record) | oldest `catalog_since` |
|
||||
|---|---|---|---|---|
|
||||
| N100 `demo-felhom-8363b5` | 1 (opengist) | **0** | **0** | 2026-07-18 |
|
||||
| HP t740 `demo-hp-bb76ea` | 9 | **0** | **0** | 2026-07-12 |
|
||||
|
||||
**Every deployed app on the fleet reads „Naprakész" today, and every one of them has a record.** The
|
||||
v0.234.0 backfill has done its job: the "unknown on a quiet box" residue of §8.3 is zero here.
|
||||
|
||||
**This is the honest caveat, and it is the whole of §8.1:** „Naprakész" on these boxes means *the
|
||||
reference matches*, not *the image has not moved*. Section 3 measures exactly how false that can be.
|
||||
|
||||
---
|
||||
|
||||
## 3. Upstream drift — the number that decides how urgent Slice 6 is
|
||||
|
||||
53 apps · 79 `image:` lines · **66 unique pins**. Measured from DooPlex
|
||||
against the public registries, never from a box (§8.1's rule applies to the product, not to an
|
||||
audit). The dead-image sweep used the catalog's own `check-image-resolvable.py --all`, unmodified.
|
||||
|
||||
### 3.1 The exact pins
|
||||
|
||||
| | count |
|
||||
|---|---|
|
||||
| exact pins evaluated | 58 |
|
||||
| **behind upstream today** | **46** |
|
||||
| …of those, a MAJOR step | **7** |
|
||||
| …of those, a minor/patch step | **39** |
|
||||
| already newest | 10 |
|
||||
| Felhom's own internal image (not upstream) | 1 |
|
||||
| inconclusive | 1 |
|
||||
|
||||
**39 of 46 are within a major — that is the population Slice 6's automatic update would serve.**
|
||||
Seven cross a major and are human work by the 2026-09-02 ruling: nextcloud 34→35, paperless-ngx
|
||||
2.20→3.2, claper 2.5→3.0, gokapi 1.9→2.2, homepage 1.13→2.4, sparkyfitness 0.17→1.7 (two images).
|
||||
**All seven were pinned 2026-07-18 — 65 days ago.**
|
||||
|
||||
### 3.2 By class
|
||||
|
||||
| class | apps | pins | floating | behind (of exact) | major | minor/patch |
|
||||
|---|---|---|---|---|---|---|
|
||||
| database | 14 | 23 | 7 | 14 of 16 | 5 | 9 |
|
||||
| file-leg | 9 | 9 | 0 | 7 of 9 | 0 | 7 |
|
||||
| other | 30 | 34 | 1 | 25 of 33 | 2 | 23 |
|
||||
|
||||
*Classification caveat, stated rather than folded in:* `adventurelog` runs a `postgis/postgis`
|
||||
sidecar, which is PostgreSQL, but the literal substring rule puts it in `other`. It is the one app
|
||||
that moves if postgis counts as a database (`other` 30→29, `database` 14→15). **v0.240.0's R-484
|
||||
already treats PostGIS/pgvector/TimescaleDB as Postgres for the logical dump**, so the product is
|
||||
right and only this table's rule is literal.
|
||||
|
||||
### 3.3 The floating pins — §8.1 is now MEASURED, and its COUNT was stale
|
||||
|
||||
**First, the count itself.** `09` §8.1, `REUSE.md`, the badge's own comment and R-446 all said
|
||||
*"23 of the catalog's 66 distinct pins float"*. **That number matches no definition the catalog
|
||||
supports today**, and it has been corrected in all four places. Recounted at catalog `18a6d2d8`,
|
||||
with the definition written down so it can be rechecked: **a pin FLOATS when its tag names a version
|
||||
LINE rather than an exact release.**
|
||||
|
||||
| shape | count | examples |
|
||||
|---|---|---|
|
||||
| full `X.Y.Z` | 48 | `privatebin/pdo:2.0.5` |
|
||||
| **two-part line** | **6** | `mariadb:11.4`/`11.6`/`12.3`, `claper:2.5`, `opengist:1.13`, `wger/server:2.6` |
|
||||
| **major line** | **4** | `postgres:15-alpine`, `postgres:16-alpine`, `redis:7-alpine`, `postgis:16-3.5-alpine` |
|
||||
| exact version + variant suffix | 8 | `ghost:6.53.0-alpine`, `nextcloud:34.0.1-apache`, `kimai/kimai2:apache-2.57.0` |
|
||||
|
||||
**10 of 66 float.** The last row does not, by this definition — those name an exact release and only
|
||||
wear a flavour.
|
||||
|
||||
**Then, the movement.** The sweep measured the 8 database and cache engine pins, of which 7 were
|
||||
measurable:
|
||||
|
||||
For each floating tag, the registry's own last-push date against the date the catalog set the pin:
|
||||
|
||||
| pin | apps | pinned | last repushed upstream | moved? |
|
||||
|---|---|---|---|---|
|
||||
| `postgres:16-alpine` | 8 apps | 2026-07-18 | 2026-09-21 | **yes** |
|
||||
| `postgres:15-alpine` | sparkyfitness | 2026-07-18 | 2026-09-21 | **yes** |
|
||||
| `redis:7-alpine` | 6 apps | 2026-07-18 | 2026-09-18 | **yes** |
|
||||
| `mariadb:11.4` | romm | 2026-07-18 | 2026-09-18 | **yes** |
|
||||
| `mariadb:12.3` | bookstack | 2026-07-18 | 2026-09-18 | **yes** |
|
||||
| `postgis/postgis:16-3.5-alpine` | adventurelog | 2026-07-18 | 2026-08-31 | **yes** |
|
||||
| `mariadb:11.6` | kimai, nextcloud | 2026-07-18 | 2025-02-04 | no |
|
||||
| `ghcr.io/immich-app/postgres:16-vectorchord…` | immich | 2026-07-18 | — | **UNMEASURED** |
|
||||
|
||||
**Six of the seven measurable floating tags have been repushed since the catalog pinned them.** So on
|
||||
demo-hp today, `adventurelog`, `bookstack`, `docmost` and `romm` read „Naprakész" over a database
|
||||
engine image that has demonstrably moved. The badge is not lying — it answers the question it was
|
||||
built to answer — but the question a household hears is the other one. **R-446 is no longer a
|
||||
theoretical blind spot: it is six pins out of seven.** The eighth (immich's own ghcr build) could not
|
||||
be measured: ghcr's anonymous API exposes no last-modified timestamp and no authenticated token was
|
||||
used. Stated unmeasured rather than guessed.
|
||||
|
||||
### 3.4 One image is gone upstream
|
||||
|
||||
`msdeluise/plant-it:0.10.0` — the gate says INCONCLUSIVE (a `denied`/`unauthorized` stderr, which it
|
||||
refuses to read as "dead"), and two independent signals say it really is gone: Docker Hub's catalog
|
||||
API answers `object not found` for the whole repository, and the upstream GitHub repo 404s. The app
|
||||
is already `lifecycle: abandoned`, so this is the expected end state, not an incident. **A deployed
|
||||
plant-it would survive — nothing re-pulls a running image — but it can never be redeployed.**
|
||||
|
||||
---
|
||||
|
||||
## 4. What this session then did
|
||||
|
||||
- **controller v0.260.0** — R-524 closed: a box AHEAD of the catalog reads „Naprakész", and the
|
||||
guarded Update refuses to move a pin backwards (`downgrade`, 409). The comparison moved to
|
||||
`stacks.CatalogOrder` so the badge and the refusal cannot drift. Three red-proofs, each seen to
|
||||
fail. See the repo's CHANGELOG.
|
||||
- **catalog** — R-469: the MariaDB half of the engine-major rule lifted, with R-450's own-edge clause
|
||||
enforced in its place; PostgreSQL and MySQL stay refused, now citing R-463 rather than the shipped
|
||||
R-448. Two new decoys, two red-proofs.
|
||||
- **R-589** closed as already-shipped (§1.1). **R-605** filed (§1.2).
|
||||
|
||||
Live proof of v0.260.0 and the R-520 power cut: `audits/update-arc-2026-09-21/`.
|
||||
|
||||
---
|
||||
|
||||
## 4a. R-524 — PROVEN LIVE, both languages, both surfaces
|
||||
|
||||
The state was built the honest way, not by editing a file on the box: a throwaway `uptime-kuma` on
|
||||
the scratch guest, a real catalog move **2.4.0 → 2.5.0**, the guarded Update pressed and run to
|
||||
completion, then the catalog **reverted to 2.4.0** — exactly the BIGNIGHT Phase 6 shape that R-524
|
||||
recorded. The box then runs 2.5.0 against a 2.4.0 catalog.
|
||||
|
||||
**The badge — app page AND apps list, both languages:**
|
||||
|
||||
```
|
||||
[hu] <span class="tag tag-ok" title="Ez az alkalmazás a katalógusnál újabb változatot futtat,
|
||||
ezért nincs teendőd.">Naprakész</span>
|
||||
[en] <span class="tag tag-ok" title="This app is running a version newer than the catalog offers,
|
||||
so there is nothing for you to do.">Up to date</span>
|
||||
```
|
||||
|
||||
`tag-warn` is **absent** on all four captures (negative control), and the AHEAD title is a *different
|
||||
sentence* from the ordinary up-to-date title captured at baseline — which is the v0.260.0 behaviour,
|
||||
not a coincidence of wording.
|
||||
|
||||
**The refusal — the endpoint the button invokes:**
|
||||
|
||||
| request | code | body |
|
||||
|---|---|---|
|
||||
| `POST /api/stacks/uptime-kuma/update` | **409** | „Ez a változat újabb a katalógusban lévőnél — visszalépés csak az üzemeltető kérésére." |
|
||||
| the same with `?lang=en` | **409** | "This version is newer than the one in the catalog — moving back needs the operator." |
|
||||
| the same with `?lang=hu` (control) | **409** | the Hungarian sentence |
|
||||
|
||||
**The English one is the half that was nearly missed.** The new refusal was born as a bundle key, but
|
||||
`api.Router` rendered update refusals from `ref.Message` — the Hungarian fallback — so without the
|
||||
one-line change to `errText` in the same release the key would have been a seam built and never
|
||||
wired. The `?lang=en` row above is the proof that it is wired.
|
||||
|
||||
**The log names both maps, so the verdict can be audited without guessing:**
|
||||
|
||||
```
|
||||
[ERROR] [stacks] update uptime-kuma REFUSED (downgrade): installed is provably NEWER than the
|
||||
catalog on every differing service (installed=map[uptime-kuma:{louislam/uptime-kuma:2.5.0 …}]
|
||||
catalog=map[uptime-kuma:louislam/uptime-kuma:2.4.0])
|
||||
```
|
||||
|
||||
**And NOTHING MOVED — which is the property that makes the refusal safe to add.** After three refused
|
||||
POSTs the app is still up and healthy on 2.5.0, the live compose line still reads 2.5.0, and **no
|
||||
update journal exists** (with a positive control proving the directory searched was the real one).
|
||||
That is §6.1's phase-0 contract intact: a refusal taken *before* the intent is recorded leaves no
|
||||
trace, so a household that presses a refused button has changed nothing.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 4b. R-520 — a power cut during a REAL version change. CLOSED, and it behaved.
|
||||
|
||||
**What R-520 asked for**, and why the 2026-09-14 attempt could not answer it: that night the only
|
||||
Update the catalog allowed was a **same-version** one, so nothing could have gone wrong and nothing
|
||||
did. The row asked for the same cut on a **real bump**.
|
||||
|
||||
**The measurement**, on the scratch guest 9202 (demo-hp), controller v0.260.0, with a throwaway
|
||||
`uptime-kuma` and a real one-step catalog move 2.4.0 → 2.5.0 (reverted in the same session):
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| 11:04:54.436Z | poller sees `updating: true`, `update_phase: pulling` |
|
||||
| 11:04:54.439Z | `pct stop 9202` — the cut |
|
||||
| 11:04:58.207Z | stopped |
|
||||
|
||||
*Honest limit on the instrument, recorded rather than smoothed:* `pct stop` took 3.8 s to return, so
|
||||
the phase **at the moment of the decision** is observed and the phase **at the moment the kernel
|
||||
froze** is inferred.
|
||||
|
||||
**After `pct start`, the box said so itself** — the POSITIVE observable, not an absence:
|
||||
|
||||
```
|
||||
update recovery: uptime-kuma was interrupted in pulling (started 2026-09-21T11:04:52Z)
|
||||
— nothing had run; putting the pin back
|
||||
pin uptime-kuma: uptime-kuma=louislam/uptime-kuma:2.4.0
|
||||
update uptime-kuma: pin and definition PUT BACK to the pre-update version
|
||||
```
|
||||
|
||||
- `pinned_images` **and** `installed_images` both 2.4.0; the live compose line back to 2.4.0.
|
||||
- No hold (correctly — nothing ran), `update_phase: failed`.
|
||||
- The app is **running and healthy on 2.4.0**, 31 s after boot.
|
||||
- The household is told, and the sentence is true: *„A frissítés megszakadt, mert a vezérlő
|
||||
újraindult, mielőtt az új verzió elindult volna. Az alkalmazás a korábbi verzióval fut tovább."*
|
||||
|
||||
**R-520 closes.** A power cut in `pulling` during a real version change is recovered honestly: the
|
||||
pin goes back, the old version runs, and the page says what happened.
|
||||
|
||||
### The instrument trap this run found, and it is worth more than the result
|
||||
|
||||
The first post-crash read, taken from the host against the **stopped** guest, reported the update
|
||||
journal **ABSENT** — which would have made this a defect finding. It was a **false negative**.
|
||||
|
||||
**`pct mount` maps the guest's ROOTFS ONLY and does not apply the guest's own internal mounts.**
|
||||
Guest 9202 keeps `/var/lib/docker` on its own ext4 mount under the `mp0` volume; from the host that
|
||||
path exists and is an **empty stub**. The journal was at
|
||||
`/var/lib/lxc/9202/rootfs/var/lib/felhom/docker/volumes/felhom-controller-data/_data/data/`, and the
|
||||
box's own recovery read it at the next boot — which is the proof that it was there all along.
|
||||
|
||||
**It was caught by running `find` over the whole rootfs instead of trusting one constructed path**,
|
||||
and by demanding a positive control for the directory being searched. Read from the right place, the
|
||||
journal held exactly what §6.1 promises: `phase: "pulling"`, `prev_pin: …2.4.0`. **An empty directory
|
||||
is not evidence of an absent file** — it is equally consistent with "you are looking at an unmounted
|
||||
stub". That is R-96 rule 3 in a new place.
|
||||
|
||||
### A second instrument trap in the same run — and it nearly bought a false proof
|
||||
|
||||
`POST /api/sync` answered **„nincs változás"** while the box's catalog cache file HAD moved to the new
|
||||
tag, and the API's `catalog_images` stayed at the old value until a separate `POST /api/stacks/rescan`.
|
||||
Since `CatalogImages` is the one input `stacks.CatalogOrder` compares against, **the badge answers
|
||||
from a stale catalog for that window** — and the session nearly recorded a `tag-ok` badge as proof of
|
||||
the R-524 ahead arm when the badge was merely stale. The honest reading came only after the rescan.
|
||||
**An instrument that can report an old value as a current one is not a measurement.** Filed as
|
||||
**R-607**; neither half of it was isolated, so the row records the observation, not a diagnosis.
|
||||
|
||||
---
|
||||
|
||||
## 4c. A defect the live run found — filed, not fixed today
|
||||
|
||||
**On the English app page, the badge is English and the sentence under it is Hungarian.** Captured
|
||||
verbatim at step 5, directly above one another:
|
||||
|
||||
```
|
||||
[en] <span class="tag tag-warn" title="A newer version of this app is available…">Update available — today</span>
|
||||
A frissítés megszakadt, mert a vezérlő újraindult, mielőtt az új verzió elindult volna.
|
||||
Az alkalmazás a korábbi verzióval fut tovább.
|
||||
```
|
||||
|
||||
`Stack.UpdateError` is a finished Hungarian STRING, not a key. `Manager.finishUpdate` stores it from
|
||||
eight raw literals in `update.go`, and **both** `app_info.html` and `stacks.html` render it verbatim.
|
||||
The same path carries `UpdatePhaseLabel` and the HOLD sentence — including
|
||||
`backup.UpdateCopyHolds`'s *„csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem"*,
|
||||
**which is a promise about whether the customer's files come back**.
|
||||
|
||||
**v0.260.0 did not close this.** That release routed the 409 REFUSAL through `errText`, and a refusal
|
||||
is the path where nothing happened; these are the sentences for when something did. **Filed as
|
||||
R-606 (P2), not fixed** — one release per repo per session, and today's had already shipped.
|
||||
|
||||
---
|
||||
|
||||
## 4d. Teardown — three layers, stated
|
||||
|
||||
| layer | state |
|
||||
|---|---|
|
||||
| **machine** (guest 9202) | throwaway `uptime-kuma` removed **through the product** — the API refused the remove while it ran (409, *"stop it first"*), so it was stopped through the product and then removed, taking its volume and its 24K backup. Only the three protected infra containers remain, as at baseline. No update journal. |
|
||||
| **host** (demo-hp) | guest 9202 left running, as it was found. **Guest 9201 untouched** — 24 containers before and after. `felhom-pve` not involved. |
|
||||
| **hub** | nothing provisioned, nothing enrolled, nothing discarded. Guest 9202 reports to no hub by design. |
|
||||
|
||||
**The catalog is back where it started:** `louislam/uptime-kuma:2.4.0`, tree clean and level with
|
||||
`origin/main`. `catalog_since` reads 2026-09-21 rather than 2026-07-18 — the gate requires an image
|
||||
move to carry the day's date, and the bump and its revert are two moves that net to zero. No box is
|
||||
behind on that app, so no age count is affected.
|
||||
|
||||
---
|
||||
|
||||
## 5. What the numbers say about the two open slices
|
||||
|
||||
**Slice 6 (automatic within a major) is worth more than its P2 suggests, and the reason is §3.1:**
|
||||
39 of 58 exact pins are one minor step behind, every one of them a change the 2026-09-02 ruling
|
||||
already says may happen without a human. Nobody presses 39 buttons. The support window (§3 decision
|
||||
2) runs on how far behind the catalog a box is, so today every box is inside it only because the
|
||||
catalog has not moved since 2026-09-15 — the moment the catalog catches up with upstream, every box
|
||||
on the fleet is 46 pins behind at once.
|
||||
|
||||
**Slice 7 (the fleet view) is still correctly P3-LOW** at two enrolled boxes — but §2 shows why it
|
||||
will not stay cheap: the only reason a person could answer "is the fleet current?" today is that a
|
||||
session read both boxes' files by hand.
|
||||
|
||||
**R-446 (digests) is the cheapest real improvement on this list.** §3.3 measured the blind spot at
|
||||
six pins of seven, and the catalog's `check-image-resolvable.py` already resolves a digest for every
|
||||
pin at push time. Recording it costs one field and closes the gap without the box ever reaching a
|
||||
registry.
|
||||
@@ -0,0 +1,115 @@
|
||||
# What to work on next — backlog triage, 2026-10-03
|
||||
|
||||
## In one screen
|
||||
|
||||
- **The open list is now readable.** It went from 442 rows to 326. It went from 824 KB to about 540 KB.
|
||||
- **Every open row has one category and one severity.** Two gates refuse a row without them, and refuse a
|
||||
finished row left in the open list.
|
||||
- **No row is P1.** Nothing in the list is harm that happens today on shipped code. 27 rows are P2 (must be done
|
||||
before the first paying customer).
|
||||
- **You decide one thing:** which theme starts next. Recommendation: **off-site backup safety**. The other
|
||||
option: **box system security updates**. Details are in "The decision" below.
|
||||
|
||||
## Headline numbers (before → after)
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| rows in the open list (`OPEN-ITEMS.md`) | 442 with an id + 22 without | 326, all with an id |
|
||||
| size of the open list | 824 KB, 932 lines | ≈540 KB |
|
||||
| finished rows moved to `CLOSED-ITEMS.md` | — | 125 with an id + 20 without |
|
||||
| rows marked VERIFY (may be finished; not proven) | — | 11 |
|
||||
| rows with no severity | the reviewer counted ~182 | 0 |
|
||||
| new rows filed this session | — | 11: R-812, R-813 (findings beside two roadmap items), R-814, R-815 (old July watches that had no id), R-816 to R-819 (gaps found while sorting), R-50b (a finding that lived only in the roadmap) |
|
||||
| roadmap (`ROADMAP.md`) | 161 lines, 81 KB | 124 lines, 42 KB |
|
||||
|
||||
**Open rows, category × severity** (no P1 anywhere):
|
||||
|
||||
| Category | P2 | P3 | P4 | total |
|
||||
|---|---|---|---|---|
|
||||
| Install & onboarding | 2 | 11 | 5 | 18 |
|
||||
| Apps & catalog | 0 | 15 | 21 | 36 |
|
||||
| App updates | 0 | 12 | 7 | 19 |
|
||||
| Backup & restore | 12 | 24 | 20 | 56 |
|
||||
| Storage & devices | 0 | 7 | 5 | 12 |
|
||||
| Security & access | 3 | 23 | 3 | 29 |
|
||||
| Box system & updates | 3 | 11 | 3 | 17 |
|
||||
| Monitoring & notifications | 2 | 16 | 6 | 24 |
|
||||
| Hub & operator | 1 | 7 | 13 | 21 |
|
||||
| Business & legal | 4 | 0 | 3 | 7 |
|
||||
| Process & tooling | 0 | 4 | 83 | 87 |
|
||||
| **total** | **27** | **130** | **169** | **326** |
|
||||
|
||||
## The top 29 — every P2, open list and roadmap together
|
||||
|
||||
Order: severity first, then "a household meets it", then cost (small first). Size: XS < S < M < L.
|
||||
|
||||
| # | ID | Category | Size | Household meets it? | What is wrong or missing |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | R-508 | Install & onboarding | XS | yes | The hub does not warn when a customer who waits for a box has no e-mail address. |
|
||||
| 2 | R-509 | Install & onboarding | XS | yes | A new box for an existing customer may not send the connect e-mail. Fix shipped; never seen live (VERIFY). |
|
||||
| 3 | R-604 | Box system & updates | S | yes | A per-customer version floor can keep one box out of every fleet update, silently. |
|
||||
| 4 | R-784 | Business & legal | S | yes | SparkyFitness forbids commercial use; the author's written permission is needed. |
|
||||
| 5 | R-789 | Business & legal | S | yes | Tandoor's licence may forbid selling a service built on it; the authors' permission is needed. |
|
||||
| 6 | R-342 | Backup & restore | S | yes | Nothing protects the off-site backup data if maintenance on that server goes wrong. |
|
||||
| 7 | R-813 | Business & legal | S | yes | The website collects personal data but has no privacy notice, no terms and no imprint. |
|
||||
| 8 | R-243 | Monitoring & notifications | M | yes | A box that waits for its recovery key stops off-site backups, and nobody is told. |
|
||||
| 9 | R-95 | Backup & restore | M | yes | A box can delete its own off-site backups, and no older copy can be read back today. |
|
||||
| 10 | R-436 | Backup & restore | M | yes | The provider's "append-only" lock (a box may add but not delete) is untested. **Its check is due 2026-10-06.** |
|
||||
| 11 | R-726 | Backup & restore | M | yes | A returning customer's new box makes no off-site backup until someone presses a hidden reset. |
|
||||
| 12 | R-366 | Backup & restore | M | yes | After a reinstall the box cannot read its older whole-machine backups, and it reads as a test failure. |
|
||||
| 13 | R-32 | Backup & restore | M | yes | Resetting a customer leaves their old encrypted off-site data behind, unseen and uncounted. |
|
||||
| 14 | R-105 | Backup & restore | M | yes | Two records a whole-box recovery needs may still be empty on the hub; not checked again. |
|
||||
| 15 | R-518 | Backup & restore | M | yes | "Back up now" stops every app for minutes while a slow second backup runs. |
|
||||
| 16 | R-519 | Backup & restore | M | yes | After an interrupted backup, a restore point shows a newer time than its files. |
|
||||
| 17 | R-638 | Backup & restore | M | yes | Restoring a database copy from before an update can fail on top of the newer database. |
|
||||
| 18 | R-528 | Monitoring & notifications | M | yes | An app killed for lack of memory is often not reported. |
|
||||
| 19 | R-777 | Security & access | M | yes | Emby and Jellyfin treat internet visitors as home-network visitors and skip their remote limits. |
|
||||
| 20 | R-304 | Backup & restore | L | yes | A household's correct recovery code for old backups is rejected as wrong. |
|
||||
| 21 | R-812 / R-808 | Box system & updates | L | yes | Nothing ever installs operating-system security updates on a box (host, guest, Docker engine). |
|
||||
| 22 | R-809 | Business & legal | M | yes | Contract, data-processing agreement, billing and invoicing do not exist yet. |
|
||||
| 23 | R-135 | Security & access | S | no | The hub skips its forged-request check when no session cookie is sent. |
|
||||
| 24 | R-802 | Business & legal | M | no | A lawyer must review the list of non-open-source licences. |
|
||||
| 25 | R-133 | Security & access | M | no | Every copy of the hub database holds every box's root console password in readable form. |
|
||||
| 26 | R-173 | Hub & operator | M | no | The hub database, which holds recovery secrets for every box, has no scheduled backup. |
|
||||
| 27 | R-530 | Box system & updates | M | no | Host agents update only by a signed job per box, and nothing lists which boxes are behind. |
|
||||
| 28 | R-232 | Backup & restore | L | no | The server that runs the hub keeps all its backups on itself, nothing off-site, and a failure alerts nobody. |
|
||||
|
||||
(R-808 and R-812 are one item: the roadmap intention and its finding. 28 lines, 29 rows.)
|
||||
|
||||
## Five work themes (about one session each)
|
||||
|
||||
1. **Off-site backup safety** — R-436, R-95, R-342, R-243, R-726, R-366, R-32, R-105, R-304. The household's
|
||||
files and photos sit in the tier whose credential can delete. Start with R-436's measurement (due 2026-10-06),
|
||||
then the cheapest step that stops a box deleting its own history.
|
||||
2. **Box system security updates** — R-812 / R-808, R-604, R-530. A spike first (what the agent may run, what a
|
||||
half-done update leaves), on a throwaway box. Nothing exists today.
|
||||
3. **The hub's own safety** — R-133, R-135, R-173, R-232. Operator-side; no household meets it directly, but a hub
|
||||
loss or leak touches every box. R-232 is on DooPlex, which needs an operator word before anyone touches it.
|
||||
4. **Honest backup and app behaviour** — R-518, R-519, R-638, R-528, R-777, R-508, R-509. Smaller items a
|
||||
household meets; good for a session after theme 1.
|
||||
5. **Business and legal** — R-809, R-813, R-784, R-789, R-802. **Yours**, not a CC session. CC drafts a text when
|
||||
you ask. It can run beside any other theme.
|
||||
|
||||
## The decision
|
||||
|
||||
**Which theme does the next session start with?**
|
||||
|
||||
| | A — Off-site backup safety (recommended) | B — Box system security updates |
|
||||
|---|---|---|
|
||||
| What it does | Measures the provider's append-only lock (R-436), then closes the biggest data risk: a box that can delete its own off-site history (R-95). | A spike on a throwaway box: what update path is safe for host, guest and Docker engine. Then a design. |
|
||||
| What it costs | One session. Touches the off-site test account, never a customer's data. | One spike session plus a later build session. Touches only a Tier-0 box. |
|
||||
| Why | The largest data risk the list has ranked first since July. The household's own files sit there. | Boxes stay in homes for years. Today they never receive a security patch. |
|
||||
| If you decide nothing | On 2026-10-06 the dated check turns red and refuses every push to this repo until the measurement is done or the date is moved. | Boxes keep the packages they were installed with. Nothing breaks this week; the risk grows every month. |
|
||||
|
||||
**My pick: A.** It has a hard date in three days, it guards the household's data, and it needs no new mechanism.
|
||||
B is next.
|
||||
|
||||
## Rows that wait on you (18)
|
||||
|
||||
R-132 (change the hub password), R-169, R-210, R-503, R-504, R-508, R-526, R-530, R-719, R-769, R-770, R-771, R-779,
|
||||
R-784, R-789, R-802, R-804, R-813. The P2 ones are in the top list above. Each row says what it needs from you.
|
||||
|
||||
## Rows marked VERIFY (11) — may already be finished; nobody proved it
|
||||
|
||||
R-274, R-282, R-284, R-287, R-509, R-527, R-602, R-621, R-814, R-815, R-817. Each carries, at the start of its state,
|
||||
what suggests it is finished and the evidence pointer. Close one only after checking it against live source.
|
||||
@@ -0,0 +1,14 @@
|
||||
arch: amd64
|
||||
cores: 7
|
||||
features: nesting=1,keyctl=1
|
||||
hookscript: local:snippets/felhom-guest-hook.sh
|
||||
hostname: demo-hp
|
||||
memory: 25898
|
||||
mp0: nvme-scratch:9298/vm-9298-disk-1.raw,mp=/var/lib/felhom,backup=1,size=70G
|
||||
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:0F:7E:C5,ip=dhcp,link_down=1,type=veth
|
||||
net1: name=eth1,bridge=vmbr9,hwaddr=BC:24:11:28:B4:F5,ip=169.254.253.2/30,link_down=1,type=veth
|
||||
onboot: 0
|
||||
ostype: debian
|
||||
rootfs: nvme-scratch:9298/vm-9298-disk-0.raw,size=32G
|
||||
swap: 512
|
||||
unprivileged: 1
|
||||
@@ -0,0 +1,25 @@
|
||||
restore-beside: restoring local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst -> 9298 on nvme-scratch with onboot 0 (never started)
|
||||
recovering backed-up configuration from 'local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst'
|
||||
Formatting '/mnt/hdd_1/images/9298/vm-9298-disk-0.raw', fmt=raw size=34359738368 preallocation=off
|
||||
Creating filesystem with 8388608 4k blocks and 2097152 inodes
|
||||
Filesystem UUID: 7ddc4545-38c5-4362-8e3d-31a715b29c1b
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
4096000, 7962624
|
||||
Formatting '/mnt/hdd_1/images/9298/vm-9298-disk-1.raw', fmt=raw size=75161927680 preallocation=off
|
||||
Creating filesystem with 18350080 4k blocks and 4587520 inodes
|
||||
Filesystem UUID: 28215aa3-d7b4-484b-b2dc-9f85265e80a5
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
4096000, 7962624, 11239424
|
||||
restoring 'local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst' now..
|
||||
extracting archive '/var/lib/vz/dump/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst'
|
||||
tar: ./var/lib/felhom/docker/volumes/kimai_kimai_var/_data/cache/prod/pools/app/vl6nzGcpxe/7/D/WpVn7g-RmFPG2mIs6Gaw: time stamp 2026-10-05 04:16:20 is 69996.875442074 s in the future
|
||||
Total bytes read: 15959541760 (15GiB, 276MiB/s)
|
||||
merging backed-up and given configuration..
|
||||
restore-beside: removing host-path binds: mp8,mp9
|
||||
restore-beside: link down: net0
|
||||
restore-beside: link down: net1
|
||||
restore-beside: OK: 9298 has onboot 0, no host-path binds, every NIC link_down; it was not started
|
||||
restore-beside: read it with: pct mount 9298 (then pct unmount 9298; pct destroy 9298 --purge when done)
|
||||
rc=0
|
||||
@@ -0,0 +1,19 @@
|
||||
=== felhom-agent 0.139.0 selftest=bring-up (mode=dr vmid=9297) ===
|
||||
--- recover: reaping any half-built guest from a prior crashed bring-up ---
|
||||
recover: examined=0 bring_up_rolled_back=0 bring_up_clean=0 scratch_destroyed=0
|
||||
bringing up local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst → vmid 9297 on nvme-scratch …
|
||||
--- bring-up record ---
|
||||
{
|
||||
"VMID": 9297,
|
||||
"AssignedMAC": "",
|
||||
"Hostname": "",
|
||||
"Pass": false,
|
||||
"Verified": "",
|
||||
"Err": {},
|
||||
"StartedAt": "2026-10-04T06:59:11.397594956Z",
|
||||
"Duration": 118096279,
|
||||
"StartWarnings": null,
|
||||
"WarningsRecognized": false
|
||||
}
|
||||
[FAIL] bring-up (vmid 9297): reconcile: dr bring-up refused: the archive's source guest 9201 still exists on this host (status running) — a DR restore beside a live original would run two boxes on the same drives (R-834)
|
||||
rc=1
|
||||
@@ -0,0 +1,54 @@
|
||||
=== 2026-10-04T06:44:15Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:44:19Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:44:23Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:44:27Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:44:31Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:44:36Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:44:40Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:44:44Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:44:48Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:44:52Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:44:56Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:45:00Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:45:04Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:45:08Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:45:12Z vmid 990000 status=status: stopped
|
||||
lock: create
|
||||
=== 2026-10-04T06:45:17Z vmid 990000 status=status: stopped
|
||||
hostname: demo-hp
|
||||
mp0: nvme-scratch:990000/vm-990000-disk-1.raw,mp=/var/lib/felhom,backup=1,size=70G
|
||||
mp8: nvme-scratch:990000/vm-990000-disk-2.raw,mp=/mnt/felhom-drives,backup=0,size=1G
|
||||
mp9: nvme-scratch:990000/vm-990000-disk-3.raw,mp=/etc/felhom-bootstrap,backup=0,size=1G
|
||||
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:0F:7E:C5,ip=dhcp,type=veth
|
||||
net1: name=eth1,bridge=vmbr9,hwaddr=BC:24:11:28:B4:F5,ip=169.254.253.2/30,type=veth
|
||||
onboot: 0
|
||||
=== 2026-10-04T06:45:20Z vmid 990000 status=status: running
|
||||
hostname: demo-hp
|
||||
mp0: nvme-scratch:990000/vm-990000-disk-1.raw,mp=/var/lib/felhom,backup=1,size=70G
|
||||
mp8: nvme-scratch:990000/vm-990000-disk-2.raw,mp=/mnt/felhom-drives,backup=0,size=1G
|
||||
mp9: nvme-scratch:990000/vm-990000-disk-3.raw,mp=/etc/felhom-bootstrap,backup=0,size=1G
|
||||
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:0F:7E:C5,ip=dhcp,link_down=1,type=veth
|
||||
net1: name=eth1,bridge=vmbr9,hwaddr=BC:24:11:28:B4:F5,ip=169.254.253.2/30,link_down=1,type=veth
|
||||
onboot: 0
|
||||
=== 2026-10-04T06:45:25Z vmid 990000 status=status: running
|
||||
hostname: demo-hp
|
||||
mp0: nvme-scratch:990000/vm-990000-disk-1.raw,mp=/var/lib/felhom,backup=1,size=70G
|
||||
mp8: nvme-scratch:990000/vm-990000-disk-2.raw,mp=/mnt/felhom-drives,backup=0,size=1G
|
||||
mp9: nvme-scratch:990000/vm-990000-disk-3.raw,mp=/etc/felhom-bootstrap,backup=0,size=1G
|
||||
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:0F:7E:C5,ip=dhcp,link_down=1,type=veth
|
||||
net1: name=eth1,bridge=vmbr9,hwaddr=BC:24:11:28:B4:F5,ip=169.254.253.2/30,link_down=1,type=veth
|
||||
onboot: 0
|
||||
@@ -0,0 +1,27 @@
|
||||
=== felhom-agent 0.138.0 selftest=restore-test ===
|
||||
--- recover: reaping any leaked scratch from a prior crashed test ---
|
||||
recover: examined=0 scratch_destroyed=0 scratch_clean=0
|
||||
restoring local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst into scratch band [990000,990009] on nvme-scratch …
|
||||
time=2026-10-04T08:44:13.035+02:00 level=INFO msg="restore-test: space preflight passed" storage=nvme-scratch required_bytes=24520159232 avail_bytes=876487208960
|
||||
time=2026-10-04T08:44:13.137+02:00 level=INFO msg="restore-test: full-fidelity restore params derived from the archive config" scratch=990000 params=4
|
||||
time=2026-10-04T08:45:25.769+02:00 level=INFO msg="audit: gate decision" class=guest_destroy host=demo-hp-bb76ea guest=990000 source=one_shot_job disposition=benign allowed=true reason=benign key_id="" nonce="" durable_id=""
|
||||
time=2026-10-04T08:45:25.769+02:00 level=INFO msg="gate decision" class=guest_destroy guest=990000 source=one_shot_job disposition=benign allowed=true reason=benign
|
||||
time=2026-10-04T08:45:31.852+02:00 level=INFO msg="restore-test: scratch guest torn down" vmid=990000
|
||||
--- restore-test record ---
|
||||
{
|
||||
"source_archive": "local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst",
|
||||
"source_tier": "local",
|
||||
"scratch_vmid": 990000,
|
||||
"pass": true,
|
||||
"verified": "boot+running",
|
||||
"tested_at": "2026-10-04T06:45:31Z",
|
||||
"duration_seconds": 79.434773547,
|
||||
"mount_parity": "ok",
|
||||
"mount_inventory": [
|
||||
"mp0=/var/lib/felhom (70G)",
|
||||
"mp8=/mnt/felhom-drives (throwaway for the archived bind)",
|
||||
"mp9=/etc/felhom-bootstrap (throwaway for the archived bind)"
|
||||
]
|
||||
}
|
||||
space preflight passed: storage=nvme-scratch required=24520159232 avail=876487208960
|
||||
=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m19s) ===
|
||||
@@ -0,0 +1,17 @@
|
||||
# R-833 hub red-proofs, 2026-10-04 (hub tree before v0.129.0 release). Each mutation applied, test run, reverted.
|
||||
RP1 raise ignored (OpenWindowFor keeps the default cap):
|
||||
--- FAIL: TestWindow_RaisedCapIsOneWindowOnly — raised grant: {Granted:true WindowID:2 ... MaxRemove:20} — want MaxRemove 30
|
||||
RP2 grant not consumed (TakeOffsiteWindowGrant does not clear the setting):
|
||||
--- FAIL: TestWindow_RaisedCapIsOneWindowOnly — the raised grant was not consumed
|
||||
RP3 close check uses the default cap instead of the window's own:
|
||||
--- FAIL: TestWindow_RaisedCapIsOneWindowOnly — a drop of 28 under a raised cap of 30 alarmed — the close check ignored the window's own cap
|
||||
RP4 the box API window-open sets a grant from a max_remove in its body:
|
||||
--- FAIL: TestOffsiteWindow_BoxCannotGrantItself — a box call left a grant (ok=true max=400)
|
||||
After revert: ok internal/offsitekeys, ok internal/api
|
||||
|
||||
# Live on hub v0.129.0 (2026-10-04), operator Basic auth, POST /offsite/window-grant/<id> — every bad grant refused, nothing stored:
|
||||
demo-hp-bb76ea max_remove=abc -> 400 max_remove must be a number
|
||||
demo-hp-bb76ea max_remove=0 -> 400 offsitekeys: max_remove 0 is outside 1..500
|
||||
demo-hp-bb76ea max_remove=501 -> 400 offsitekeys: max_remove 501 is outside 1..500
|
||||
no-such-customer max_remove=10 -> 400 offsitekeys: no customer "no-such-customer"
|
||||
A valid grant was NOT placed on a real customer: it would be consumed by that demo box's next window (the brief: lab repo only).
|
||||
@@ -0,0 +1,20 @@
|
||||
# R-833 lab proof — 2026-10-04 (a lab repo on DooPlex scratch, not a demo box's)
|
||||
|
||||
restic 0.14.0 from the controller image `felhom-controller:0.290.0` (`docker run --rm --entrypoint sh`), a local repo,
|
||||
password `lab-only-password` (a lab value). 98 daily snapshots, `--host demo-lab --tag felhom-offsite`, dated 97..0 days
|
||||
back — the shape of a box whose windows were off for three months.
|
||||
|
||||
The box's own guard (`offsiteGuard`, controller v0.290.0 source) was run on the repo's REAL `snapshots --json` and the
|
||||
policy's REAL `forget --dry-run --json` by `offbox_window_lab_test.go.txt` (copy it into
|
||||
`controller/internal/backup/` and set `OFFSITE_LAB_DIR` to rerun; it skips without it, so it is not committed).
|
||||
|
||||
| Step | Result |
|
||||
|---|---|
|
||||
| Honest plan | 98 snapshots, the policy removes **85** |
|
||||
| Default cap (hub `MaxRemove` = half) | **49 → REFUSED**: `the plan would remove 85 snapshots, more than one week's retention may (49)` |
|
||||
| Operator-raised cap 90 (one window) | **85 ids, no refusal** |
|
||||
| `restic forget <those 85 ids> --prune` | rc 0; **98 → 13**; `restic check` rc 0 |
|
||||
| Next window, default cap again (6) | plan 0, **no refusal** — the cap only needed raising once |
|
||||
|
||||
The hub half (the grant is one-shot, the next window has the default cap, the close check uses the window's own cap, a
|
||||
box cannot grant itself) is in `hub-redproofs.txt` and the hub suite.
|
||||
@@ -0,0 +1,64 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-833 LAB (not run in CI: needs OFFSITE_LAB_DIR from the lab script). Reads a REAL restic 0.14.0
|
||||
// repo's `snapshots --json` and the policy's `forget --dry-run --json`, and runs the box's own guard
|
||||
// with the hub's default cap and with an operator-raised cap. Writes the ids the guard would remove to
|
||||
// $OFFSITE_LAB_DIR/ids.txt for the lab script to prune. Evidence: audits/backup-close-2026-10-04/partB/.
|
||||
func TestOffsiteGuard_R833_LabRepo(t *testing.T) {
|
||||
dir := os.Getenv("OFFSITE_LAB_DIR")
|
||||
if dir == "" {
|
||||
t.Skip("lab only: set OFFSITE_LAB_DIR")
|
||||
}
|
||||
var all []guardSnap
|
||||
b, err := os.ReadFile(filepath.Join(dir, "snaps.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := json.Unmarshal(b, &all); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, err = os.ReadFile(filepath.Join(dir, "plan.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var groups []struct {
|
||||
Remove []guardSnap `json:"remove"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &groups); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var plan []guardSnap
|
||||
for _, g := range groups {
|
||||
plan = append(plan, g.Remove...)
|
||||
}
|
||||
now := time.Now()
|
||||
def := len(all) / 2 // the hub's MaxRemove(count): half, minimum 5
|
||||
if def < 5 {
|
||||
def = 5
|
||||
}
|
||||
_, refuse := offsiteGuard(all, plan, now, now, def)
|
||||
t.Logf("snapshots=%d plan=%d default cap=%d → refusal: %q", len(all), len(plan), def, refuse)
|
||||
raised := 0
|
||||
if v := os.Getenv("OFFSITE_LAB_RAISED"); v != "" {
|
||||
for _, c := range v {
|
||||
raised = raised*10 + int(c-'0')
|
||||
}
|
||||
}
|
||||
if raised == 0 {
|
||||
return
|
||||
}
|
||||
ids, refuse2 := offsiteGuard(all, plan, now, now, raised)
|
||||
t.Logf("raised cap=%d → %d id(s), refusal: %q", raised, len(ids), refuse2)
|
||||
if err := os.WriteFile(filepath.Join(dir, "ids.txt"), []byte(strings.Join(ids, "\n")), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
11:01:30 the box's template copy names ADMIN_USER: 4 times
|
||||
13:01:33 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/calibre-web']
|
||||
13:01:33 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH', 'ADMIN_PASSWORD']
|
||||
13:01:34 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
13:02:19 [1] deployed, controller state=running, pinned={'calibre-web': 'crocodilestick/calibre-web-automated:v4.0.8'}
|
||||
11:02:19 deploy: True
|
||||
11:02:34 stranger, default login admin/admin123 -> other:404
|
||||
11:02:34 stranger, default login admin/admin123 -> other:401
|
||||
11:02:34 stranger, default login admin/admin123 -> wrong
|
||||
11:02:34 stranger, default login admin/admin123 -> locked
|
||||
11:02:34 stranger got in with the default login: 0 of 31
|
||||
11:02:37 2026/10/01 11:01:34 install_hold.go:106: [INFO] [stacks] calibre-web: install HOLD before the first start — only the household reaches [books.enkisfelhom.hu] until the known first login is replaced
|
||||
2026/10/01 11:02:17 install_hold.go:135: [INFO] [stacks] calibre-web: install hold OPENED by after_install — the app is reached as without a hold
|
||||
11:02:40 app.yaml after_install record: after_install: at: "2026-10-01T11:02:17Z" ok: true install_hold:
|
||||
11:02:40 the household's name (read through the page's reveal): 10 characters, lowercase hex: True
|
||||
11:02:43 users in app.db (total, named admin): (2, 0)
|
||||
11:02:44 positive — form, name + password: ok | OPDS: ok
|
||||
11:02:44 admin + the RIGHT password (the old name is gone): locked
|
||||
11:02:44 ## a stranger: 40 wrong tries on `admin`, 3 a minute
|
||||
11:02:44 wrong try 1: locked
|
||||
11:02:44 wrong try 2: locked
|
||||
11:02:44 wrong try 3: locked
|
||||
11:05:48 wrong try 10: wrong
|
||||
11:08:51 wrong try 20: wrong
|
||||
11:11:55 wrong try 30: wrong
|
||||
11:16:00 wrong try 40: wrong
|
||||
11:17:01 one more stranger try on admin: locked
|
||||
11:17:02 HOUSEHOLD at once — form, own name + password: ok | OPDS: ok
|
||||
11:17:02 a wrong password on the real name: wrong | right again: ok
|
||||
13:17:07 [X] stop -> 200 {'ok': True, 'message': 'Stack calibre-web stop completed'}
|
||||
13:17:12 [X] remove (with drive data) -> 409 {'ok': False, 'error': 'A(z) /mnt/felhom-drives/scratch_hdd/userdata/calibre-web tárhely jelenleg nem elérhető — az alkalmazás nem távolítható el, amíg a meghaj
|
||||
13:17:12 [X] refused because the drive path cannot be resolved (R-442, fail-closed and right) — removing the app and KEEPING the drive data instead
|
||||
13:17:39 [X] remove (keeping drive data) -> 200 {'ok': True, 'data': {'removed': 'calibre-web', 'volumes_removed': ['calibre-web_calibre_web_config'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'back
|
||||
13:17:47 [X] after remove: deployed=False leftovers='/opt/docker/stacks/calibre-web'
|
||||
11:17:47 removed; deployed = False
|
||||
@@ -0,0 +1,14 @@
|
||||
# demo-hp 9201 calibre-web rename 2026-10-01T11:19:40Z (values in through stdin; never printed)
|
||||
admin row found: True
|
||||
cps.py: 0 (Standard/LDAP login)
|
||||
Password for user '<name>' changed
|
||||
hash takes the password: True | admin left: 0
|
||||
demo-hp login form — new name + password: ok | OPDS: ok
|
||||
demo-hp login form — admin + the same password: wrong
|
||||
# demo-hp: rename again, to the name the BOX recorded (InjectMissingFields 11:26:10) — 2026-10-01T11:27:25Z
|
||||
the current name found: True
|
||||
cps.py rc: 0
|
||||
hash takes the password: True | old name left: 0 | admin left: 0
|
||||
demo-hp — the box's recorded name + password: ok | OPDS: ok
|
||||
demo-hp — the previous name + password: wrong | admin + password: wrong
|
||||
credentials file: holds the box's recorded name
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
page carries the ADMIN_USER field: False
|
||||
label shown: False | reveal button: False
|
||||
reveal ADMIN_USER -> ok: True | value length: 10 | error:
|
||||
the password field's description now says 'the user name above': False
|
||||
@@ -0,0 +1,22 @@
|
||||
## GREEN
|
||||
PASS --keep defaults to 20
|
||||
PASS plan: delete 4, keep 20, protect 2 (latest + the floor)
|
||||
PASS the in-use 0.262.0 is not in the delete plan
|
||||
PASS the plan says why 0.262.0 is kept
|
||||
PASS dry-run
|
||||
PASS red: without the in-use list 0.262.0 WOULD be deleted
|
||||
PASS unreadable in-use list -> refused (exit 3)
|
||||
rc=0
|
||||
1
|
||||
## RED (is_protected without the in-use check)
|
||||
PASS --keep defaults to 20
|
||||
FAIL plan counts: Would delete 5 tag(s); keep 20; protect 1:
|
||||
FAIL the in-use 0.262.0 is in the delete plan
|
||||
FAIL no reason printed for 0.262.0
|
||||
PASS dry-run
|
||||
PASS red: without the in-use list 0.262.0 WOULD be deleted
|
||||
PASS unreadable in-use list -> refused (exit 3)
|
||||
rc=1
|
||||
sed: couldn't edit /dev/null: not a regular file
|
||||
## GREEN again
|
||||
rc=0
|
||||
@@ -0,0 +1,173 @@
|
||||
################ --type container --all prune (DRY-RUN)
|
||||
|
||||
[INFO] ╔══════════════════════════════════════════╗
|
||||
[INFO] ║ Gitea package prune ║
|
||||
[INFO] ╚══════════════════════════════════════════╝
|
||||
[INFO] Server: https://gitea.dooplex.hu Owner: admin Type: container Log: /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/b4d68b9b-a6cf-4d21-8220-ece956837fc3/scratchpad/prune-dry.log
|
||||
[INFO] Auth: git credential helper (user: kisfenyo)
|
||||
[INFO] Gitea version: 1.26.2
|
||||
[STEP] Fetching container packages for owner 'admin' from https://gitea.dooplex.hu ...
|
||||
[INFO] Loaded 448 version records across 9 package(s).
|
||||
[WARN] --all covers type 'container' ONLY. Other types are untouched by this run.
|
||||
[INFO] prune: --keep defaults to 20 (decision 62)
|
||||
[INFO] In-use list (decision 62): 6 version(s) from the hub's vouch, golden 0.285.0's bake.log and the hub manifest.
|
||||
in use: felhom-agent:0.131.0 — min_agent
|
||||
in use: felhom-agent:0.138.0 — the vouched agent
|
||||
in use: felhom-controller:0.285.0 — the controller floor; the vouched golden's controller; baked into golden 0.285.0
|
||||
in use: felhom-golden:0.285.0 — the vouched golden
|
||||
in use: felhom-hub:0.126.0 — the hub the GitOps manifest runs
|
||||
in use: felhom-samba:1.1.0 — baked into golden 0.285.0
|
||||
[INFO] Action: prune Type: container Targets: felhom-act-runner felhom-controller felhom-hub felhom-samba jarr recipe-importer revfulop-calendar sparkyfitness-export wan-probe
|
||||
|
||||
== prune felhom-act-runner == mode: keep-last 20
|
||||
[INFO] Nothing to prune (1 tags: 1 kept, 0 protected).
|
||||
|
||||
== prune felhom-controller == mode: keep-last 20
|
||||
PROTECTED latest — matches --protect ^latest$
|
||||
PROTECTED 0.285.0 — the controller floor; the vouched golden's controller; baked into golden 0.285.0
|
||||
Would delete 70 tag(s); keep 20; protect 2:
|
||||
0.271.0
|
||||
0.270.0
|
||||
0.269.1
|
||||
0.269.0
|
||||
0.268.0
|
||||
0.267.0
|
||||
0.266.0
|
||||
0.265.0
|
||||
0.264.0
|
||||
0.263.2
|
||||
0.263.1
|
||||
0.263.0
|
||||
0.262.1
|
||||
0.262.0
|
||||
0.261.0
|
||||
0.260.0
|
||||
0.259.0
|
||||
0.258.0
|
||||
0.257.0
|
||||
0.256.1
|
||||
0.256.0
|
||||
0.255.0
|
||||
0.254.0
|
||||
0.253.0
|
||||
0.252.0
|
||||
0.251.0
|
||||
0.250.0
|
||||
0.249.0
|
||||
0.248.0
|
||||
0.247.0
|
||||
0.246.0
|
||||
0.245.0
|
||||
0.244.0
|
||||
0.243.0
|
||||
0.242.0
|
||||
0.241.0
|
||||
0.240.0
|
||||
0.239.0
|
||||
0.238.1
|
||||
0.238.0
|
||||
0.237.0
|
||||
0.236.0
|
||||
0.235.0
|
||||
0.234.0
|
||||
0.233.0
|
||||
0.232.0
|
||||
0.231.0
|
||||
0.230.0
|
||||
0.229.0
|
||||
0.228.0
|
||||
0.227.1
|
||||
0.227.0
|
||||
0.226.1
|
||||
0.226.0
|
||||
0.225.0
|
||||
0.224.0
|
||||
0.223.0
|
||||
0.222.0
|
||||
0.221.1
|
||||
0.221.0
|
||||
0.220.2
|
||||
0.220.1
|
||||
0.220.0
|
||||
0.219.0
|
||||
0.218.0
|
||||
0.217.0
|
||||
0.216.0
|
||||
0.215.0
|
||||
0.214.0
|
||||
0.213.0
|
||||
[WARN] DRY-RUN — nothing deleted. --apply is a person's act (decision 62); nothing schedules it.
|
||||
|
||||
== prune felhom-hub == mode: keep-last 20
|
||||
PROTECTED latest — matches --protect ^latest$
|
||||
PROTECTED 0.126.0 — the hub the GitOps manifest runs
|
||||
Would delete 8 tag(s); keep 20; protect 2:
|
||||
0.107.0
|
||||
0.106.0
|
||||
0.105.0
|
||||
0.104.0
|
||||
0.103.0
|
||||
0.102.0
|
||||
0.101.0
|
||||
0.100.2
|
||||
[WARN] DRY-RUN — nothing deleted. --apply is a person's act (decision 62); nothing schedules it.
|
||||
|
||||
== prune felhom-samba == mode: keep-last 20
|
||||
PROTECTED 1.1.0 — baked into golden 0.285.0
|
||||
[INFO] Nothing to prune (2 tags: 1 kept, 1 protected).
|
||||
|
||||
== prune jarr == mode: keep-last 20
|
||||
PROTECTED latest — matches --protect ^latest$
|
||||
[INFO] Nothing to prune (2 tags: 1 kept, 1 protected).
|
||||
|
||||
== prune recipe-importer == mode: keep-last 20
|
||||
PROTECTED latest — matches --protect ^latest$
|
||||
[INFO] Nothing to prune (8 tags: 7 kept, 1 protected).
|
||||
|
||||
== prune revfulop-calendar == mode: keep-last 20
|
||||
[INFO] Nothing to prune (7 tags: 7 kept, 0 protected).
|
||||
|
||||
== prune sparkyfitness-export == mode: keep-last 20
|
||||
[INFO] Nothing to prune (4 tags: 4 kept, 0 protected).
|
||||
|
||||
== prune wan-probe == mode: keep-last 20
|
||||
[INFO] Nothing to prune (1 tags: 1 kept, 0 protected).
|
||||
|
||||
[INFO] Prune summary: deleted=0 failed=0 (dry-run=yes)
|
||||
|
||||
[INFO] Done. Audit log: /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/b4d68b9b-a6cf-4d21-8220-ece956837fc3/scratchpad/prune-dry.log
|
||||
rc=0
|
||||
################ --type generic --all prune (DRY-RUN)
|
||||
|
||||
[INFO] ╔══════════════════════════════════════════╗
|
||||
[INFO] ║ Gitea package prune ║
|
||||
[INFO] ╚══════════════════════════════════════════╝
|
||||
[INFO] Server: https://gitea.dooplex.hu Owner: admin Type: generic Log: /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/b4d68b9b-a6cf-4d21-8220-ece956837fc3/scratchpad/prune-dry.log
|
||||
[INFO] Auth: git credential helper (user: kisfenyo)
|
||||
[INFO] Gitea version: 1.26.2
|
||||
[STEP] Fetching generic packages for owner 'admin' from https://gitea.dooplex.hu ...
|
||||
[INFO] Loaded 41 version records across 2 package(s).
|
||||
[WARN] --all covers type 'generic' ONLY. Other types are untouched by this run.
|
||||
[INFO] prune: --keep defaults to 20 (decision 62)
|
||||
[INFO] In-use list (decision 62): 6 version(s) from the hub's vouch, golden 0.285.0's bake.log and the hub manifest.
|
||||
in use: felhom-agent:0.131.0 — min_agent
|
||||
in use: felhom-agent:0.138.0 — the vouched agent
|
||||
in use: felhom-controller:0.285.0 — the controller floor; the vouched golden's controller; baked into golden 0.285.0
|
||||
in use: felhom-golden:0.285.0 — the vouched golden
|
||||
in use: felhom-hub:0.126.0 — the hub the GitOps manifest runs
|
||||
in use: felhom-samba:1.1.0 — baked into golden 0.285.0
|
||||
[INFO] Action: prune Type: generic Targets: felhom-agent felhom-golden
|
||||
|
||||
== prune felhom-agent == mode: keep-last 20
|
||||
PROTECTED 0.138.0 — the vouched agent
|
||||
PROTECTED 0.131.0 — min_agent
|
||||
[INFO] Nothing to prune (20 tags: 18 kept, 2 protected).
|
||||
|
||||
== prune felhom-golden == mode: keep-last 20
|
||||
PROTECTED 0.285.0 — the vouched golden
|
||||
[INFO] Nothing to prune (21 tags: 20 kept, 1 protected).
|
||||
|
||||
[INFO] Prune summary: deleted=0 failed=0 (dry-run=yes)
|
||||
|
||||
[INFO] Done. Audit log: /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/b4d68b9b-a6cf-4d21-8220-ece956837fc3/scratchpad/prune-dry.log
|
||||
rc=0
|
||||
@@ -0,0 +1,12 @@
|
||||
# Rulings 61 (calibre-web's generated login name) and 62 (the registry prune rule) — 2026-10-01
|
||||
|
||||
Report: `felhom.eu/REPORT-calibre-name-and-prune-2026-10-01.md`. Tools: `../lockouts-2026-10-01/tools/`.
|
||||
|
||||
| file | what |
|
||||
|---|---|
|
||||
| `A/A1-9202-calibre-generated-name.txt` | 9202: install hold, after_install, the name, 40 wrong tries on `admin`, the household in at once |
|
||||
| `A/A2-demo-hp-rename.txt` | demo-hp: the rename by hand (twice — see A3), real logins over its traefik; no value printed |
|
||||
| `A/A3-demo-hp-page-after-sync.txt` | demo-hp after the sync: the box injected its own `ADMIN_USER` (R-757) |
|
||||
| `B/B1-test-and-red-proof.txt` | `misc-scripts/tests/test-prune-plan.sh` green, red (in-use check removed), green |
|
||||
| `B/B2-live-dry-run.txt` | the live dry-run, both package types — nothing deleted |
|
||||
| `T/` | 9202 back on live, the drill reset |
|
||||
@@ -0,0 +1,15 @@
|
||||
git:
|
||||
branch: main
|
||||
repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
|
||||
sync_interval: 15m
|
||||
token: <redacted>
|
||||
username: ""
|
||||
hub:
|
||||
0
|
||||
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.285.0
|
||||
filebrowser gtstef/filebrowser:1.3.3-stable
|
||||
paperless-postgres postgres:18-alpine
|
||||
paperless-redis redis:7-alpine
|
||||
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
traefik traefik:v3.6.7
|
||||
@@ -0,0 +1,3 @@
|
||||
# drill reset 2026-10-01T11:28:12Z
|
||||
4e18b3a DRILL calibre-web: generated admin login name (decision 61 proof on 9202)
|
||||
after: e9f50b5 = live e9f50b5
|
||||
@@ -0,0 +1,413 @@
|
||||
# Catalog currency — measured 2026-09-30
|
||||
|
||||
Catalog `app-catalog-felhom.eu` main = `64461979250b` (clean tree). Controller source read at
|
||||
`felhom-controller` main = `d48da6c`. Read-only. Measured from DooPlex against the public registries
|
||||
with anonymous tokens. No box was touched. Nothing was pulled. Nothing was committed.
|
||||
|
||||
Method: the 2026-09-21 method (`UPDATE-ARC-STATE-2026-09-21.md` §3, `update-arc-2026-09-21/00-drift.py`),
|
||||
adapted in `catalog-currency-2026-09-30/00-currency.py`. What changed is written at the top of that script.
|
||||
|
||||
## 1. Headline numbers
|
||||
|
||||
**53 apps · 79 `image:` lines · 67 distinct pins.** 53 lines are exact releases, 24 are floating lines
|
||||
(12 distinct), 1 is Felhom's own image, 1 could not be measured (plant-it: the repository is gone).
|
||||
|
||||
| question | apps |
|
||||
|---|---|
|
||||
| **behind upstream inside the same major** (any image) | **25 of 53** |
|
||||
| **behind upstream across a major** (any image) | **19 of 53** |
|
||||
| behind in either way | 37 of 53 |
|
||||
| — inside the major, exact pins only | 24 |
|
||||
| — across a major, exact pins only | 8 (gitea, gokapi, gramps-web, homepage, jellyfin, nextcloud, paperless-ngx, sparkyfitness) |
|
||||
| — across a major, only because a floating engine line has a newer major | 14 (postgres → 18, redis → 8, mariadb → 13.0, postgis → 18, claper 2.5 → 3.0) |
|
||||
|
||||
**Can update itself at night today:**
|
||||
|
||||
| number | apps | what it counts |
|
||||
|---|---|---|
|
||||
| **(a)** a box at the pin just before the catalog's head gets a step tonight | **28**, plus **1** conditional (nextcloud) | the controller's own rule, catalog side (§4) |
|
||||
| **(b)** the catalog's ladder has ≥1 `proven` entry | **29** | a looser count |
|
||||
| never, today | **24** | no `update_ladder:` at all → skip reason `no_test_record` |
|
||||
|
||||
(a) and (b) differ by one app only. Every entry in all 29 ladders is `proven`, and no entry has
|
||||
`needs_person`. The one difference is nextcloud: its newest step has `files_may_change`, so the leg
|
||||
takes it only when a fresh whole copy of the app is on the box.
|
||||
|
||||
**What (a) does NOT say.** It says the catalog offers a step. It does not say how many boxes are at a
|
||||
pin that matches. Nobody measured the fleet's pins today. A box installed from today's catalog is at
|
||||
the head and has nothing to do. And a night step only goes to the catalog's head, which is itself
|
||||
behind upstream for 37 apps.
|
||||
|
||||
### 1a. After the same day's moves (added by the session, 2026-09-30 afternoon)
|
||||
|
||||
The tables below are the MORNING measurement (catalog `6446197`). The session then moved five apps on both venues:
|
||||
rallly, outline and sparkyfitness to PostgreSQL 18 (their first ladder steps), bookstack 26.05.5 → 26.09.1 and kimai
|
||||
apache-2.57.0 → 2.67.0 (+ the Part F apps named in `pg-last-six-2026-09-30/F/`). Re-running `04-analyse.py` on the
|
||||
afternoon catalog with the same registry data: **night (a) 31 + nextcloud conditional; (b) 32 carry a proven step; 21
|
||||
have no ladder.** The "behind upstream" counts are not re-measured; bookstack and kimai are now at their newest release
|
||||
inside the major.
|
||||
|
||||
### 1b. After the evening's moves (added 2026-09-30 evening, `more-night-apps-2026-09-30/`)
|
||||
|
||||
**Correction first:** the numbers above were counted before the immich step of the late afternoon; that step carries
|
||||
`files_may_change` (R-734), so at `4c552cf` — the evening session's start — `04-analyse.py` reads **night (a) 30 + 2
|
||||
conditional (nextcloud, immich); (b) 32**, not 31 + 1. The evening published first ladders for calibre-web, gitea, wger,
|
||||
crafty-controller, uptime-kuma and zipline, and within-major steps for emby, ghost, home-assistant, outline, rallly
|
||||
(and zipline's second). Re-run of `04-analyse.py` on the evening catalog, same registry data (`more-night-apps-2026-09-30/count/`):
|
||||
**night (a) 35 + 3 conditional (nextcloud, immich, calibre-web); (b) 38 carry a proven step; 15 have no ladder.**
|
||||
The "behind upstream" counts are not re-measured; each of the eleven moved apps is now at the newest release inside its
|
||||
major (re-checked the same evening, `more-night-apps-2026-09-30/00-upstream-recheck.txt`).
|
||||
|
||||
## 2. What could not be measured — stated, not guessed
|
||||
|
||||
1. **plant-it** (`msdeluise/plant-it:0.10.0`): Docker Hub gives 401 on the tag list, and its catalog
|
||||
API gives **404** for the repository. Same finding as 2026-09-21. The app is `lifecycle: abandoned`.
|
||||
2. **recipe-importer**: Felhom's own image (`gitea.dooplex.hu`), not upstream. Not compared.
|
||||
3. **Upload dates**: not measured for any image. ghcr's anonymous API has no timestamp. Not needed
|
||||
for this question, so not attempted.
|
||||
4. **Whether the newest tag is a GA release.** The filter is by tag name only (rc, beta, alpha, dev,
|
||||
nightly, …). Two results need a person to check: `mariadb:13.0` (a two-part line tag exists; GA not
|
||||
checked) and `gitea/gitea:28.0.0` (pushed to Docker Hub 2026-09-30 00:15 UTC, the same day; a
|
||||
version renumbering, not checked).
|
||||
5. **Tag shape switches.** The rule compares only tags of the same shape. A control pass
|
||||
(`09-shape-switch-check.txt`) looked for higher versions under ANY shape for every pin called
|
||||
"up to date". It found three real switches, applied by hand with their tag lists
|
||||
(`10-shape-switch-detail.txt`):
|
||||
- **gramps-web** `v25.6.0`: upstream dropped the `v`. It is at `26.9.1`. Calendar versions, so 25→26
|
||||
is counted as across a major.
|
||||
- **jellyfin** `10.11.11`: upstream 12 uses two-part tags. It is at `12.1` (12.0 had rc1..rc7).
|
||||
- **kimai** `apache-2.57.0`: upstream stopped the `apache-` tags. Plain `2.67.0` exists. Whether it
|
||||
is the same Apache variant is NOT checked.
|
||||
Two more hits were noise: sonarr (`5.14-2.0.0…-ls5`, a develop-style tag) and tandoor (a
|
||||
`dependabot-…` branch tag). The control ran only on pins called up to date, and not on floating
|
||||
lines. A switch hiding behind a pin already called "behind" would change only its "newest" value.
|
||||
6. **Floating pins — did the tag move?** 18 of 24 floating lines resolve to **exactly the digest
|
||||
the ladder recorded when the step was tested**. None has moved since its test. 6 cannot be compared,
|
||||
because the app has no ladder: outline (postgres 16, redis 7), rallly (postgres 16), sparkyfitness
|
||||
(postgres 15), zipline (postgres 16), wger (`2.6`).
|
||||
7. **Majors for 0.x and calendar versions.** "Major" is the first number of the tag, as written. For
|
||||
`v0.x` apps a minor step can be breaking. For calendar versions (bookstack 26.05, home-assistant
|
||||
2026.9, actualbudget 26.9, papra 26.6) the first number is a year. This table does not interpret it.
|
||||
8. **ghcr rate limit.** immich-server and immich-machine-learning got HTTP 429 on the first run. They
|
||||
were measured on a slower retry (`02-retry-429-run.txt`). Nothing is left at 429.
|
||||
|
||||
## 3. Per image
|
||||
|
||||
"Floating (line)" = the tag names a version LINE, not a release: fewer than three numbers in its
|
||||
first version run (`12.3`, `16-alpine`, `16-3.5-alpine`, immich's `16-vectorchord…`). For a floating
|
||||
line, "newest same major" is the newest line of the same shape in that major. "Newest at all" is the
|
||||
newest line of that shape.
|
||||
|
||||
| app | service | pinned | newest same major | newest at all | behind in major | behind across major | note |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| actualbudget | actualbudget | `26.9.0` | `26.9.0` | `26.9.0` | no | no | |
|
||||
| adventurelog | adventurelog | `v0.13.0` | `v0.13.0` | `v0.13.0` | no | no | |
|
||||
| adventurelog | adventurelog-postgres | `16-3.5-alpine` | `16-3.5-alpine` | `18-3.6-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| adventurelog | adventurelog-frontend | `v0.13.0` | `v0.13.0` | `v0.13.0` | no | no | |
|
||||
| audiobookshelf | audiobookshelf | `2.36.1` | `2.37.1` | `2.37.1` | yes | no | |
|
||||
| bentopdf | bentopdf | `v2.8.6` | `v2.8.8` | `v2.8.8` | yes | no | |
|
||||
| bookstack | bookstack | `26.05.5` | `26.09.1` | `26.09.1` | yes | no | |
|
||||
| bookstack | bookstack-db | `12.3` | `12.3` | `13.0` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| calcom | calcom | `v6.2.0` | `v6.2.0` | `v6.2.0` | no | no | |
|
||||
| calcom | calcom-postgres | `18-alpine` | `18-alpine` | `18-alpine` | no | no | floating (line); digest = ladder's tested digest |
|
||||
| calibre-web | calibre-web | `v4.0.6` | `v4.0.8` | `v4.0.8` | yes | no | |
|
||||
| claper | claper | `2.5` | `2.5` | `3.0` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| claper | claper-postgres | `17-alpine` | `17-alpine` | `18-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| code-server | code-server | `4.129.0` | `4.139.1` | `4.139.1` | yes | no | |
|
||||
| crafty-controller | crafty-controller | `4.10.7` | `4.11.0` | `4.11.0` | yes | no | |
|
||||
| docmost | docmost | `0.96.0` | `0.96.0` | `0.96.0` | no | no | |
|
||||
| docmost | docmost-postgres | `18-alpine` | `18-alpine` | `18-alpine` | no | no | floating (line); digest = ladder's tested digest |
|
||||
| docmost | docmost-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| emby | emby | `4.11.0.3` | `4.11.0.4` | `4.11.0.4` | yes | no | |
|
||||
| ghost | ghost | `6.65.0-alpine` | `6.67.0-alpine` | `6.67.0-alpine` | yes | no | |
|
||||
| gitea | gitea | `1.27.0` | `1.27.3` | `28.0.0` | yes | yes | |
|
||||
| glance | glance | `v0.8.5` | `v0.8.6` | `v0.8.6` | yes | no | |
|
||||
| gokapi | gokapi | `v1.9.6` | `v1.9.6` | `v2.2.4` | no | yes | |
|
||||
| grafana | grafana | `13.2.2` | `13.2.3` | `13.2.3` | yes | no | |
|
||||
| gramps-web | gramps-web | `v25.6.0` | `v25.6.0` | `26.9.1` | no | yes | SHAPE SWITCH: upstream dropped the `v` prefix in 2026 (26.x.y); calendar versions, so the 25→26 step is a new year, counted as across |
|
||||
| home-assistant | home-assistant | `2026.9.3` | `2026.9.4` | `2026.9.4` | yes | no | |
|
||||
| homebox | homebox | `0.26.2` | `0.26.2` | `0.26.2` | no | no | |
|
||||
| homepage | homepage | `v1.13.2` | `v1.13.2` | `v2.4.0` | no | yes | |
|
||||
| immich | immich-server | `v3.2.2` | `v3.2.4` | `v3.2.4` | yes | no | measured on retry after ghcr 429 |
|
||||
| immich | immich-machine-learning | `v3.2.2` | `v3.2.4` | `v3.2.4` | yes | no | measured on retry after ghcr 429 |
|
||||
| immich | immich-postgres | `16-vectorchord0.4.3-pgvectors0.2.0` | `16-vectorchord0.4.3-pgvectors0.3.0` | `17-vectorchord0.4.3-pgvectors0.3.0` | yes | yes | floating (line); digest = ladder's tested digest |
|
||||
| immich | immich-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| jellyfin | jellyfin | `10.11.11` | `10.11.11` | `12.1` | no | yes | SHAPE SWITCH: upstream 12.x publishes two-part tags (`12.1`); 12.0 went through rc1..rc7 |
|
||||
| kimai | kimai | `apache-2.57.0` | `2.67.0` | `2.67.0` | yes | no | SHAPE SWITCH: upstream stopped publishing `apache-` tags after 2.57.0; plain `2.67.0` exists (whether it is the same apache variant is NOT checked) |
|
||||
| kimai | kimai-db | `11.8` | `11.8` | `13.0` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| komga | komga | `1.27.1` | `1.28.0` | `1.28.0` | yes | no | |
|
||||
| mealie | mealie | `v3.28.0` | `v3.28.0` | `v3.28.0` | no | no | |
|
||||
| n8n | n8n | `2.41.2` | `2.42.1` | `2.42.1` | yes | no | |
|
||||
| navidrome | navidrome | `0.64.1` | `0.64.2` | `0.64.2` | yes | no | |
|
||||
| nextcloud | nextcloud | `34.0.4-apache` | `34.0.4-apache` | `35.0.1-apache` | no | yes | |
|
||||
| nextcloud | nextcloud-db | `12.3` | `12.3` | `13.0` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| nextcloud | nextcloud-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| onlyoffice | onlyoffice | `9.4.0` | `9.4.0` | `9.4.0` | no | no | |
|
||||
| opengist | opengist | `1.15` | `1.15` | `1.15` | no | no | floating (line); digest = ladder's tested digest |
|
||||
| outline | outline | `1.9.1` | `1.10.1` | `1.10.1` | yes | no | |
|
||||
| outline | outline-postgres | `16-alpine` | `16-alpine` | `18-alpine` | no | yes | floating (line); no ladder digest to compare |
|
||||
| outline | outline-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); no ladder digest to compare |
|
||||
| paperless-ngx | paperless-webserver | `2.20.15` | `2.20.15` | `3.2.1` | no | yes | |
|
||||
| paperless-ngx | paperless-postgres | `18-alpine` | `18-alpine` | `18-alpine` | no | no | floating (line); digest = ladder's tested digest |
|
||||
| paperless-ngx | paperless-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| papra | papra | `26.6.2-rootless` | `26.6.2-rootless` | `26.6.2-rootless` | no | no | |
|
||||
| plant-it | plant-it | `0.10.0` | ? | ? | ? | ? | NOT MEASURED: HTTPError: 401 Client Error: Unauthorized for url: https://r |
|
||||
| plex | plex | `1.41.4.9463-630c9f557` | `1.43.4.10903-e5521bd8c` | `1.43.4.10903-e5521bd8c` | yes | no | |
|
||||
| privatebin | privatebin | `2.0.6` | `2.0.6` | `2.0.6` | no | no | |
|
||||
| radarr | radarr | `6.4.4` | `6.4.4` | `6.4.4` | no | no | |
|
||||
| rallly | rallly | `4.11.1` | `4.15.3` | `4.15.3` | yes | no | |
|
||||
| rallly | rallly-postgres | `16-alpine` | `16-alpine` | `18-alpine` | no | yes | floating (line); no ladder digest to compare |
|
||||
| recipe-importer | recipe-importer | `v0.9.11` | — | — | n/a | n/a | Felhom's own image, not upstream |
|
||||
| romm | romm | `5.3.1` | `5.3.1` | `5.3.1` | no | no | |
|
||||
| romm | romm-db | `11.8` | `11.8` | `13.0` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| romm | romm-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| seerr | seerr | `2.7.3` | `2.7.3` | `2.7.3` | no | no | |
|
||||
| sonarr | sonarr | `4.0.20` | `4.0.20` | `4.0.20` | no | no | |
|
||||
| sparkyfitness | sparkyfitness-db | `15-alpine` | `15-alpine` | `18-alpine` | no | yes | floating (line); no ladder digest to compare |
|
||||
| sparkyfitness | sparkyfitness-server | `v0.17.3` | `v0.17.3` | `v1.7.3` | no | yes | |
|
||||
| sparkyfitness | sparkyfitness-frontend | `v0.17.3` | `v0.17.3` | `v1.7.3` | no | yes | |
|
||||
| tandoor | tandoor | `2.6.15` | `2.6.15` | `2.6.15` | no | no | |
|
||||
| tandoor | tandoor-postgres | `17-alpine` | `17-alpine` | `18-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| termix | termix | `2.8.0` | `2.8.0` | `2.8.0` | no | no | |
|
||||
| uptime-kuma | uptime-kuma | `2.4.0` | `2.5.5` | `2.5.5` | yes | no | |
|
||||
| vaultwarden | vaultwarden | `1.36.0-alpine` | `1.37.3-alpine` | `1.37.3-alpine` | yes | no | |
|
||||
| vikunja | vikunja | `2.6.0` | `2.6.0` | `2.6.0` | no | no | |
|
||||
| wanderer | wanderer | `v0.20.0` | `v0.21.0` | `v0.21.0` | yes | no | |
|
||||
| wanderer | wanderer-db | `v0.20.0` | `v0.21.0` | `v0.21.0` | yes | no | |
|
||||
| wanderer | wanderer-search | `v1.36.0` | `v1.54.2` | `v1.54.2` | yes | no | |
|
||||
| wger | wger | `2.6` | `2.7` | `2.7` | yes | no | floating (line); no ladder digest to compare |
|
||||
| wishlist | wishlist | `v0.67.1` | `v0.67.1` | `v0.67.1` | no | no | |
|
||||
| zipline | zipline | `4.6.1` | `4.8.0` | `4.8.0` | yes | no | |
|
||||
| zipline | zipline-postgres | `16-alpine` | `16-alpine` | `18-alpine` | no | yes | floating (line); no ladder digest to compare |
|
||||
|
||||
## 4. The night rule, from the code
|
||||
|
||||
Source: `felhom-controller/controller/internal/stacks/unattended.go` at `d48da6c`, function
|
||||
`legCandidate` (L392–463). `RunUpdateLeg` (L221) → `runUpdateLeg` (L232–347) calls it for every
|
||||
deployed, non-protected app (L276–281) and presses `StartGuardedUpdate` only when it returns no reason
|
||||
(L311–321). The ladder reader is `ladder.go` (`LoadLadder` L83, `sameRefs` L118, `nextLadderStep`
|
||||
L152, which makes the same choice, L163).
|
||||
|
||||
In order, an app is skipped when:
|
||||
|
||||
| code line | skip | condition |
|
||||
|---|---|---|
|
||||
| L397–407 | `held` | a hold on the app |
|
||||
| L400 | `stopped_by_household` | the household stopped it |
|
||||
| L408–416 | current / `ahead` / `order_unknown` | `CatalogOrder` is not Behind |
|
||||
| L417–419 | `unpinned` | app.yaml has no `pinned_images` |
|
||||
| L422–424 | `no_test_record` | the template has no `update_ladder:` |
|
||||
| L426–432 | — | picks the NEWEST entry whose `from` equals the box's pin, every service, exactly |
|
||||
| L433–435 | `no_test_record` | no match, and the pin is the head's `to` (behind only on a digest no step records) |
|
||||
| L436–438 | `older_than_ladder` | no match at all: the box is older than the ladder |
|
||||
| L441 | `not_proven` | the entry's verdict is not `proven` |
|
||||
| L444 | `needs_person` | `marks.needs_person` is set and not blank |
|
||||
| L447 | `failed_before` | the same step was undone or held on this ladder (R-680) |
|
||||
| L450–458 | `files_may_change_no_whole_copy` | `marks.files_may_change` and no fresh whole copy on the box |
|
||||
|
||||
After that, `StartGuardedUpdate` can still refuse for box reasons: `no_backup`, `engine_no_test`,
|
||||
`memory`, `disk`, `busy`… (`UpdatePreflight`, `update.go` L368 on; the checks at L431–470).
|
||||
|
||||
**Definition used for (a).** For each app, take a box whose pin is the `from` of the ladder's
|
||||
newest entry (one step behind the head). Apply the catalog-side rows above (L422–458) to the entry
|
||||
the code would pick. Count "yes" when nothing skips it, and "conditional" when only
|
||||
`files_may_change` stands in the way. The box-side rows (hold, stop, failed_before, whole copy,
|
||||
refusals) are not in this count, because they depend on the box.
|
||||
|
||||
Three things this rule means, read from the code:
|
||||
|
||||
- **A box older than the ladder is never pressed** (L436–438). For 21 apps the ladder's first entry is
|
||||
a backfill of the 2026-09-21/22 move (`backfilled`). A box installed before that move matches it and
|
||||
can climb. A box older than that `from` cannot.
|
||||
- **A digest-only change is never pressed at night.** A box at the head tag with an older digest is
|
||||
"behind" by `digestBehind` (updateorder.go L95), but no entry has that `from`, so L433–435 skips it.
|
||||
- **The 11 backfilled head entries are harness v1** (no memory watch). The box does not read
|
||||
`harness_version`, only `verdict`, so it presses them like any other: actualbudget, audiobookshelf,
|
||||
bookstack, grafana, home-assistant, papra, privatebin, radarr, sonarr, termix, vikunja.
|
||||
|
||||
## 5. Per app — the ladder
|
||||
|
||||
"Head = compose" is the gate's rule 3 (`check-test-record.py`, run today: 53 read, 29 with a ladder,
|
||||
0 convicted, `03-check-test-record.txt`). "Box one step behind, tonight" is number (a).
|
||||
|
||||
| app | entries | proven | of which backfilled | needs_person | files_may_change | engine conversion | head = compose | box one step behind, tonight | catalog_since |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| actualbudget | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| adventurelog | 1 | 1 | 0 | 0 | 0 | 0 | yes | yes | 2026-09-27 |
|
||||
| audiobookshelf | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| bentopdf | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-12 |
|
||||
| bookstack | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| calcom | 1 | 1 | 0 | 0 | 0 | 1 | yes | yes | 2026-09-28 |
|
||||
| calibre-web | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-12 |
|
||||
| claper | 1 | 1 | 0 | 0 | 0 | 1 | yes | yes | 2026-09-28 |
|
||||
| code-server | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| crafty-controller | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-06-26 |
|
||||
| docmost | 2 | 2 | 1 | 0 | 0 | 1 | yes | yes | 2026-09-25 |
|
||||
| emby | 2 | 2 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| ghost | 2 | 2 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| gitea | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| glance | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-09-21 |
|
||||
| gokapi | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| grafana | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| gramps-web | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| home-assistant | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| homebox | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-19 |
|
||||
| homepage | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| immich | 2 | 2 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| jellyfin | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| kimai | 1 | 1 | 0 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| komga | 1 | 1 | 0 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| mealie | 2 | 2 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-25 |
|
||||
| n8n | 3 | 3 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-25 |
|
||||
| navidrome | 2 | 2 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| nextcloud | 2 | 2 | 1 | 0 | 1 | 0 | yes | conditional: files_may_change (needs a fresh whole copy) | 2026-09-23 |
|
||||
| onlyoffice | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| opengist | 1 | 1 | 0 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| outline | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| paperless-ngx | 1 | 1 | 0 | 0 | 0 | 1 | yes | yes | 2026-09-27 |
|
||||
| papra | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| plant-it | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-21 |
|
||||
| plex | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-02-15 |
|
||||
| privatebin | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| radarr | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| rallly | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| recipe-importer | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-12 |
|
||||
| romm | 3 | 3 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| seerr | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| sonarr | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| sparkyfitness | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| tandoor | 2 | 2 | 1 | 0 | 0 | 1 | yes | yes | 2026-09-27 |
|
||||
| termix | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| uptime-kuma | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-09-21 |
|
||||
| vaultwarden | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| vikunja | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| wanderer | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-21 |
|
||||
| wger | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-19 |
|
||||
| wishlist | 1 | 1 | 0 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| zipline | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
|
||||
## 6. Plan for R-462 — widen the upgrade harness
|
||||
|
||||
**Order.** First: what a household loses if an update goes wrong. Then: how far behind.
|
||||
**T1** = a database engine service, or household files the backup marks `mandatory`.
|
||||
**T2** = household data in the app's own volume (SQLite or files). **T3** = little or no household data.
|
||||
|
||||
**Cost, per app.** The only measured numbers are in R-462: a proven edge takes 6.4–305 s (median 72 s)
|
||||
plus the 600 s soak, a failing edge ~556 s, and a new fixture dominates everything else (the database
|
||||
legs were costed at 21–34 CC-hours for 15 services, `09` §6.4). So: **fixture exists** = one bench run
|
||||
plus one box run. **Fixture needed** = write a seed and a read-back through the app's own interface,
|
||||
with a negative control, first. **No route** = the harness can only answer `inconclusive`.
|
||||
|
||||
### 6.1 Do next — in this order
|
||||
|
||||
1. **immich** (T1, fixture exists). Server and ML `v3.2.2 → v3.2.4`, and a newer pgvectors line inside
|
||||
PostgreSQL 16. Cheapest high-value run: the fixture exists and the step is inside the major.
|
||||
2. **bookstack** (T1, fixture exists). `26.05.5 → 26.09.1`. The fixture proves the database half only
|
||||
(R-460).
|
||||
3. **kimai** (T1, fixture exists). `2.57.0 → 2.67.0`, but the tag shape changed (`apache-` is gone).
|
||||
A person must first confirm which plain tag is the Apache image. Then one run.
|
||||
4. **calibre-web** (T1, **fixture needed**). `v4.0.6 → v4.0.8`, household books marked `mandatory`,
|
||||
and no ladder at all. The first new fixture to write.
|
||||
5. **gitea** (T2, fixture exists but fails at the web installer, R-624). `1.27.0 → 1.27.3`. Cheap:
|
||||
the installer-form POST was never written. It unblocks gitea's first ladder entry.
|
||||
6. **The T2 apps with a fixture and a step inside the major** — one run each: audiobookshelf, emby,
|
||||
ghost, grafana, home-assistant, komga, n8n, navidrome. This is the bulk and costs machine time only.
|
||||
7. **T2 apps that need a new fixture and are behind inside the major:** wanderer (3 images, incl.
|
||||
meilisearch `v1.36 → v1.54`), uptime-kuma, crafty-controller, wger.
|
||||
|
||||
### 6.2 Majors — human work by the 2026-09-02 ruling, not harness bulk
|
||||
|
||||
nextcloud 34 → 35, paperless-ngx 2 → 3, gokapi 1 → 2, homepage 1 → 2, sparkyfitness 0.17 → 1.7,
|
||||
gitea 1 → 28 (check first), jellyfin 10 → 12, gramps-web v25 → 26. Engine lines: redis 7 → 8 (6 apps),
|
||||
mariadb → 13.0 (bookstack, kimai, nextcloud, romm; GA not checked), PostgreSQL for the R-463 remainder
|
||||
(adventurelog/postgis, immich, outline, rallly, sparkyfitness on 15, zipline).
|
||||
|
||||
### 6.3 The honest ceiling — CORRECTED the same day
|
||||
|
||||
The morning version of this section named 8 apps that can only ever be `inconclusive`. **Three of them were wrong**,
|
||||
measured on the bench the same afternoon: **outline** has a front-door first-run route (`POST /api/installation.create`,
|
||||
refused once a workspace exists), **rallly** signs up through its own API with the six-digit e-mail code read from its own
|
||||
table in place of a mailbox, and **zipline** 4's first-run route is `POST /api/setup` (the old fixture called two other
|
||||
paths). outline and rallly moved on both venues the same day; zipline's fixture now tries `/api/setup` first.
|
||||
|
||||
**What remains in the class:** vaultwarden (closed sign-up by design, R-624) and code-server (a browser IDE); plex (a
|
||||
plex.tv claim token), homepage and onlyoffice (no household data) are no-route by nature. Plus plant-it (image gone) and
|
||||
recipe-importer (our own image; a fixture is needed). So the most the harness can prove is **47 of 53**.
|
||||
|
||||
### 6.4 The whole ranking
|
||||
|
||||
| rank | app | data at risk | ladder entries | images behind in major | across major | fixture |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | outline | T1: DB engine: outline-postgres | 0 | 1 | 2 | ~~no route~~ **fixture written 2026-09-30** (`installation.create`); moved to PG 18 the same day |
|
||||
| 2 | rallly | T1: DB engine: rallly-postgres | 0 | 1 | 1 | ~~no route~~ **fixture written 2026-09-30** (sign-up + its own e-mail code); moved to PG 18 the same day |
|
||||
| 3 | zipline | T1: DB engine: zipline-postgres | 0 | 1 | 1 | ~~no route~~ **first-run route `POST /api/setup` measured 2026-09-30**; fixture fixed; PG stays 16 |
|
||||
| 4 | calibre-web | T1: household files (backup class mandatory) | 0 | 1 | 0 | fixture needed |
|
||||
| 5 | sparkyfitness | T1: DB engine: sparkyfitness-db | 0 | 0 | 3 | fixture needed |
|
||||
| 6 | immich | T1: DB engine: immich-postgres; household files (backup class mandatory) | 2 | 3 | 2 | fixture exists |
|
||||
| 7 | bookstack | T1: DB engine: bookstack-db | 1 | 1 | 1 | fixture exists |
|
||||
| 8 | kimai | T1: DB engine: kimai-db | 1 | 1 | 1 | fixture exists |
|
||||
| 9 | nextcloud | T1: DB engine: nextcloud-db; household files (backup class mandatory) | 2 | 0 | 3 | fixture exists |
|
||||
| 10 | claper | T1: DB engine: claper-postgres | 1 | 0 | 2 | fixture exists |
|
||||
| 11 | paperless-ngx | T1: DB engine: paperless-postgres; household files (backup class mandatory) | 1 | 0 | 2 | fixture exists |
|
||||
| 12 | romm | T1: DB engine: romm-db | 3 | 0 | 2 | fixture exists |
|
||||
| 13 | adventurelog | T1: DB engine: adventurelog-postgres | 1 | 0 | 1 | fixture exists |
|
||||
| 14 | docmost | T1: DB engine: docmost-postgres | 2 | 0 | 1 | fixture exists |
|
||||
| 15 | tandoor | T1: DB engine: tandoor-postgres | 2 | 0 | 1 | fixture exists |
|
||||
| 16 | calcom | T1: DB engine: calcom-postgres | 1 | 0 | 0 | fixture exists |
|
||||
| 17 | wanderer | T2: household data in the app's own volume (SQLite / files) | 0 | 3 | 0 | fixture needed |
|
||||
| 18 | gitea | T2: household data in the app's own volume (SQLite / files) | 0 | 1 | 1 | fixture exists, fails at the web installer (R-624, fixable) |
|
||||
| 19 | code-server | T2: household data in the app's own volume (SQLite / files) | 0 | 1 | 0 | no route: its front door is a browser IDE behind one password |
|
||||
| 20 | crafty-controller | T2: household data in the app's own volume (SQLite / files) | 0 | 1 | 0 | fixture needed |
|
||||
| 21 | plex | T2: household data in the app's own volume (SQLite / files) | 0 | 1 | 0 | no route: the first-run claim needs a token minted at plex.tv by a real Plex acc |
|
||||
| 22 | uptime-kuma | T2: household data in the app's own volume (SQLite / files) | 0 | 1 | 0 | fixture needed |
|
||||
| 23 | vaultwarden | T2: household data in the app's own volume (SQLite / files) | 0 | 1 | 0 | no route by design: closed sign-up (R-624) |
|
||||
| 24 | wger | T2: household data in the app's own volume (SQLite / files) | 0 | 1 | 0 | fixture needed |
|
||||
| 25 | gokapi | T2: household data in the app's own volume (SQLite / files) | 0 | 0 | 1 | fixture needed |
|
||||
| 26 | gramps-web | T2: household data in the app's own volume (SQLite / files) | 0 | 0 | 1 | fixture exists |
|
||||
| 27 | jellyfin | T2: household data in the app's own volume (SQLite / files) | 0 | 0 | 1 | fixture exists |
|
||||
| 28 | homebox | T2: household data in the app's own volume (SQLite / files) | 0 | 0 | 0 | fixture exists |
|
||||
| 29 | plant-it | T2: household data in the app's own volume (SQLite / files) | 0 | 0 | 0 | fixture needed |
|
||||
| 30 | recipe-importer | T2: household data in the app's own volume (SQLite / files) | 0 | 0 | 0 | fixture needed |
|
||||
| 31 | seerr | T2: household data in the app's own volume (SQLite / files) | 0 | 0 | 0 | fixture needed |
|
||||
| 32 | audiobookshelf | T2: household data in the app's own volume (SQLite / files) | 1 | 1 | 0 | fixture exists |
|
||||
| 33 | emby | T2: household data in the app's own volume (SQLite / files) | 2 | 1 | 0 | fixture exists |
|
||||
| 34 | ghost | T2: household data in the app's own volume (SQLite / files) | 2 | 1 | 0 | fixture exists |
|
||||
| 35 | grafana | T2: household data in the app's own volume (SQLite / files) | 1 | 1 | 0 | fixture exists |
|
||||
| 36 | home-assistant | T2: household data in the app's own volume (SQLite / files) | 1 | 1 | 0 | fixture exists |
|
||||
| 37 | komga | T2: household data in the app's own volume (SQLite / files) | 1 | 1 | 0 | fixture exists |
|
||||
| 38 | n8n | T2: household data in the app's own volume (SQLite / files) | 3 | 1 | 0 | fixture exists |
|
||||
| 39 | navidrome | T2: household data in the app's own volume (SQLite / files) | 2 | 1 | 0 | fixture exists |
|
||||
| 40 | actualbudget | T2: household data in the app's own volume (SQLite / files) | 1 | 0 | 0 | fixture exists |
|
||||
| 41 | mealie | T2: household data in the app's own volume (SQLite / files) | 2 | 0 | 0 | fixture exists |
|
||||
| 42 | opengist | T2: household data in the app's own volume (SQLite / files) | 1 | 0 | 0 | fixture exists |
|
||||
| 43 | papra | T2: household data in the app's own volume (SQLite / files) | 1 | 0 | 0 | fixture exists |
|
||||
| 44 | privatebin | T2: household data in the app's own volume (SQLite / files) | 1 | 0 | 0 | fixture exists |
|
||||
| 45 | radarr | T2: household data in the app's own volume (SQLite / files) | 1 | 0 | 0 | fixture exists |
|
||||
| 46 | sonarr | T2: household data in the app's own volume (SQLite / files) | 1 | 0 | 0 | fixture exists |
|
||||
| 47 | termix | T2: household data in the app's own volume (SQLite / files) | 1 | 0 | 0 | fixture exists |
|
||||
| 48 | vikunja | T2: household data in the app's own volume (SQLite / files) | 1 | 0 | 0 | fixture exists |
|
||||
| 49 | wishlist | T2: household data in the app's own volume (SQLite / files) | 1 | 0 | 0 | fixture exists |
|
||||
| 50 | bentopdf | T3: little or no household data (stateless or config only) | 0 | 1 | 0 | fixture needed |
|
||||
| 51 | glance | T3: little or no household data (stateless or config only) | 0 | 1 | 0 | fixture needed |
|
||||
| 52 | homepage | T3: little or no household data (stateless or config only) | 0 | 0 | 1 | no route: a dashboard rendered from config files in the template |
|
||||
| 53 | onlyoffice | T3: little or no household data (stateless or config only) | 0 | 0 | 0 | no route: a stateless document server: it holds no household data of its own, so |
|
||||
|
||||
## 7. Files
|
||||
|
||||
In `catalog-currency-2026-09-30/`:
|
||||
|
||||
| file | what |
|
||||
|---|---|
|
||||
| `00-currency.py` | the measurement (registries, read-only) |
|
||||
| `01-currency-raw.json`, `01-currency-run.txt` | its raw output |
|
||||
| `02-retry-429.py`, `02-retry-429-run.txt` | the slower retry for the two ghcr 429 rows |
|
||||
| `03-check-test-record.txt` | the catalog's own ladder gate, run today |
|
||||
| `04-analyse.py`, `04-analyse-run.txt` | tables and counts (offline; the night rule is in its header) |
|
||||
| `05-summary.json` | the counts and app lists |
|
||||
| `06-`, `07-`, `08-*.md` | the three tables above |
|
||||
| `09-shape-switch-check.txt`, `10-shape-switch-detail.txt` | the shape-switch control |
|
||||
|
||||
**Findings that should become register rows** (this session did not edit the register — read-only
|
||||
brief): the three tag-shape switches (gramps-web, jellyfin, kimai), which the badge and any shape-based
|
||||
tool read as "up to date"; and the `gitea 28.0.0` / `mariadb 13.0` tags, which need a GA check before
|
||||
anyone plans on them.
|
||||
@@ -0,0 +1,234 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Catalog currency audit, 2026-09-30. Read-only, DooPlex-only, anonymous public registry API.
|
||||
|
||||
Adapted from documentation/audits/update-arc-2026-09-21/00-drift.py (same bearer-token flow and the
|
||||
same Link pagination for /v2/<repo>/tags/list; same lscr.io date-rebuild exclusion; same unstable
|
||||
markers). Changes, and why:
|
||||
- reads the pins straight from the catalog checkout (scripts/ladder.py images_in — the same reading
|
||||
the gates make), not from a hand-built pins.json;
|
||||
- one generic tag tokenizer for every pin (numbers vs the text between them; a long hex token is a
|
||||
build hash, a wildcard). Two tags are comparable only when the text between their numbers is
|
||||
identical and they have the same count of numbers. So `postgres:16-alpine` compares only with
|
||||
`<N>-alpine`, `nextcloud:34.0.4-apache` only with `<a>.<b>.<c>-apache`;
|
||||
- reports BOTH the newest within the pin's major (first number equal) and the newest at all;
|
||||
- classifies a pin as FLOATING when it names a version LINE, not an exact release: fewer than three
|
||||
version numbers in the tag's leading version run (e.g. `mariadb:12.3`, `postgres:16-alpine`,
|
||||
`redis:7-alpine`, `postgis:16-3.5-alpine`, immich's `postgres:16-vectorchord...`);
|
||||
- for floating pins, also resolves the tag's CURRENT digest (HEAD, no pull, no rate-limit count on
|
||||
Docker Hub) and compares it with the digest the app's ladder recorded when the step was tested.
|
||||
Nothing is pulled; nothing is written outside the output directory.
|
||||
|
||||
Usage: python3 00-currency.py <catalog-root> <outdir>
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
|
||||
import requests
|
||||
|
||||
CAT = sys.argv[1]
|
||||
OUT = sys.argv[2]
|
||||
sys.path.insert(0, os.path.join(CAT, "scripts"))
|
||||
import ladder # noqa: E402
|
||||
|
||||
S = requests.Session()
|
||||
S.headers.update({"User-Agent": "felhom-catalog-currency-audit/1.0 (read-only measurement)"})
|
||||
|
||||
|
||||
def bearer(www):
|
||||
parts = www[len("Bearer "):].split(",")
|
||||
d = {}
|
||||
for p in parts:
|
||||
k, v = p.split("=", 1)
|
||||
d[k.strip()] = v.strip('"')
|
||||
r = S.get(d["realm"], params={"service": d.get("service"), "scope": d.get("scope")}, timeout=20)
|
||||
r.raise_for_status()
|
||||
j = r.json()
|
||||
return j.get("token") or j.get("access_token")
|
||||
|
||||
|
||||
def get(url, headers=None, method="GET", params=None, _tok={}):
|
||||
headers = dict(headers or {})
|
||||
host = url.split("/")[2]
|
||||
if host in _tok:
|
||||
headers["Authorization"] = "Bearer " + _tok[host]
|
||||
r = S.request(method, url, headers=headers, params=params, timeout=30)
|
||||
if r.status_code == 401 and "WWW-Authenticate" in r.headers:
|
||||
tok = bearer(r.headers["WWW-Authenticate"])
|
||||
headers["Authorization"] = "Bearer " + tok
|
||||
r = S.request(method, url, headers=headers, params=params, timeout=30)
|
||||
return r
|
||||
|
||||
|
||||
def tags_all(host, repo):
|
||||
url = f"https://{host}/v2/{repo}/tags/list"
|
||||
r = get(url, params={"n": 1000})
|
||||
r.raise_for_status()
|
||||
auth = r.request.headers.get("Authorization")
|
||||
tags = list(r.json().get("tags") or [])
|
||||
link = r.headers.get("Link")
|
||||
pages = 1
|
||||
while link and pages < 1000:
|
||||
m = re.search(r'<([^>]+)>;\s*rel="next"', link)
|
||||
if not m:
|
||||
break
|
||||
nxt = m.group(1)
|
||||
if nxt.startswith("/"):
|
||||
nxt = f"https://{host}{nxt}"
|
||||
r = S.get(nxt, headers={"Authorization": auth} if auth else {}, timeout=30)
|
||||
r.raise_for_status()
|
||||
tags.extend(r.json().get("tags") or [])
|
||||
link = r.headers.get("Link")
|
||||
pages += 1
|
||||
return tags
|
||||
|
||||
|
||||
ACCEPT = ("application/vnd.docker.distribution.manifest.v2+json,"
|
||||
"application/vnd.docker.distribution.manifest.list.v2+json,"
|
||||
"application/vnd.oci.image.manifest.v1+json,application/vnd.oci.image.index.v1+json")
|
||||
|
||||
|
||||
def digest(host, repo, tag):
|
||||
r = get(f"https://{host}/v2/{repo}/manifests/{tag}", headers={"Accept": ACCEPT}, method="HEAD")
|
||||
if r.status_code != 200:
|
||||
return None, f"HTTP {r.status_code}"
|
||||
return r.headers.get("Docker-Content-Digest"), None
|
||||
|
||||
|
||||
def host_repo(ref_repo):
|
||||
for h in ("ghcr.io", "lscr.io", "registry.gitlab.com", "gitea.dooplex.hu", "quay.io"):
|
||||
if ref_repo.startswith(h + "/"):
|
||||
return h, ref_repo[len(h) + 1:]
|
||||
if "/" not in ref_repo:
|
||||
return "registry-1.docker.io", "library/" + ref_repo
|
||||
return "registry-1.docker.io", ref_repo
|
||||
|
||||
|
||||
def split_ref(ref):
|
||||
ref = ref.split("@")[0]
|
||||
slash = ref.rfind("/")
|
||||
colon = ref.rfind(":")
|
||||
if colon > slash:
|
||||
return ref[:colon], ref[colon + 1:]
|
||||
return ref, "latest"
|
||||
|
||||
|
||||
UNSTABLE = ("rc", "beta", "alpha", "dev", "nightly", "canary", "edge", "preview", "snapshot", "-pr", "test", "unstable")
|
||||
HASH_RE = re.compile(r"(?<![0-9a-z])(?=[0-9a-f]*[a-f])(?=[0-9a-f]*[0-9])[0-9a-f]{7,40}(?![0-9a-z])")
|
||||
|
||||
|
||||
def tokenize(tag):
|
||||
"""(shape, numbers) or None when the tag carries no number."""
|
||||
t = HASH_RE.sub("<hash>", tag.lower())
|
||||
parts = re.split(r"(\d+)", t)
|
||||
nums = tuple(int(p) for p in parts[1::2])
|
||||
if not nums:
|
||||
return None
|
||||
shape = tuple(parts[0::2])
|
||||
return shape, nums
|
||||
|
||||
|
||||
def lead_run_len(tag):
|
||||
m = re.search(r"\d+(?:\.\d+)*", tag)
|
||||
return len(m.group(0).split(".")) if m else 0
|
||||
|
||||
|
||||
def is_floating(tag):
|
||||
if not re.search(r"\d", tag):
|
||||
return True # `latest`-style: no version at all
|
||||
return lead_run_len(tag) < 3
|
||||
|
||||
|
||||
def unstable(tag):
|
||||
low = tag.lower()
|
||||
return any(mk in low for mk in UNSTABLE)
|
||||
|
||||
|
||||
def newest(tags, cur):
|
||||
c = tokenize(cur)
|
||||
if not c:
|
||||
return None
|
||||
shape, nums = c
|
||||
best_all = best_maj = None
|
||||
for t in tags:
|
||||
if unstable(t) and not unstable(cur):
|
||||
continue
|
||||
p = tokenize(t)
|
||||
if not p or p[0] != shape or len(p[1]) != len(nums):
|
||||
continue
|
||||
if any(n >= 20000 for n in p[1]) and not any(n >= 20000 for n in nums):
|
||||
continue # lscr.io-style date rebuild tag, not a release (see 00-drift.py)
|
||||
if best_all is None or p[1] > best_all[1]:
|
||||
best_all = (t, p[1])
|
||||
if p[1][0] == nums[0] and (best_maj is None or p[1] > best_maj[1]):
|
||||
best_maj = (t, p[1])
|
||||
return {"cur": nums, "all": best_all, "maj": best_maj}
|
||||
|
||||
|
||||
def ladder_digests(app_dir):
|
||||
fy = os.path.join(app_dir, ".felhom.yml")
|
||||
entries, _r, _e = ladder.parse(open(fy, encoding="utf-8").read())
|
||||
if not entries:
|
||||
return {}, None
|
||||
head = entries[-1]
|
||||
return head.get("digest", {}), head.get("tested_at")
|
||||
|
||||
|
||||
def main():
|
||||
tdir = os.path.join(CAT, "templates")
|
||||
rows = []
|
||||
cache = {}
|
||||
for app in sorted(os.listdir(tdir)):
|
||||
d = os.path.join(tdir, app)
|
||||
imgs = ladder.images_in(open(os.path.join(d, "docker-compose.yml"), encoding="utf-8").read())
|
||||
ldig, ltested = ladder_digests(d)
|
||||
for svc, ref in imgs.items():
|
||||
repo, tag = split_ref(ref)
|
||||
host, hrepo = host_repo(repo)
|
||||
row = {"app": app, "service": svc, "ref": ref, "repo": repo, "tag": tag, "host": host,
|
||||
"floating": is_floating(tag)}
|
||||
if host == "gitea.dooplex.hu":
|
||||
row["status"] = "internal"
|
||||
rows.append(row)
|
||||
print(f"{app}/{svc}: internal image, not upstream")
|
||||
continue
|
||||
try:
|
||||
if (host, hrepo) not in cache:
|
||||
cache[(host, hrepo)] = tags_all(host, hrepo)
|
||||
time.sleep(0.15)
|
||||
tags = cache[(host, hrepo)]
|
||||
row["n_tags"] = len(tags)
|
||||
row["pinned_tag_listed"] = tag in tags
|
||||
res = newest(tags, tag)
|
||||
if res is None:
|
||||
row["status"] = "unparsed"
|
||||
else:
|
||||
row["status"] = "ok"
|
||||
ba, bm = res["all"], res["maj"]
|
||||
row["newest_all"] = ba[0] if ba else None
|
||||
row["newest_major"] = bm[0] if bm else None
|
||||
row["behind_within_major"] = bool(bm and bm[1] > res["cur"])
|
||||
row["behind_across_major"] = bool(ba and ba[1][0] > res["cur"][0])
|
||||
row["major"] = res["cur"][0]
|
||||
if row["floating"]:
|
||||
dg, err = digest(host, hrepo, tag)
|
||||
row["current_digest"] = dg
|
||||
row["digest_error"] = err
|
||||
row["ladder_digest"] = ldig.get(svc)
|
||||
row["ladder_tested_at"] = ltested
|
||||
if dg and ldig.get(svc):
|
||||
row["moved_since_tested"] = dg != ldig.get(svc)
|
||||
except Exception as e: # recorded, never guessed
|
||||
row["status"] = "error"
|
||||
row["error"] = f"{type(e).__name__}: {e}"[:300]
|
||||
rows.append(row)
|
||||
print(f"{app}/{svc}: {tag} -> maj={row.get('newest_major')} all={row.get('newest_all')} "
|
||||
f"float={row['floating']} {row['status']} {row.get('error','')}")
|
||||
json.dump(rows, open(os.path.join(OUT, "01-currency-raw.json"), "w"), indent=1)
|
||||
print("rows", len(rows))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,80 @@
|
||||
actualbudget/actualbudget: 26.9.0 -> maj=26.9.0 all=26.9.0 float=False ok
|
||||
adventurelog/adventurelog: v0.13.0 -> maj=v0.13.0 all=v0.13.0 float=False ok
|
||||
adventurelog/adventurelog-postgres: 16-3.5-alpine -> maj=16-3.5-alpine all=18-3.6-alpine float=True ok
|
||||
adventurelog/adventurelog-frontend: v0.13.0 -> maj=v0.13.0 all=v0.13.0 float=False ok
|
||||
audiobookshelf/audiobookshelf: 2.36.1 -> maj=2.37.1 all=2.37.1 float=False ok
|
||||
bentopdf/bentopdf: v2.8.6 -> maj=v2.8.8 all=v2.8.8 float=False ok
|
||||
bookstack/bookstack: 26.05.5 -> maj=26.09.1 all=26.09.1 float=False ok
|
||||
bookstack/bookstack-db: 12.3 -> maj=12.3 all=13.0 float=True ok
|
||||
calcom/calcom: v6.2.0 -> maj=v6.2.0 all=v6.2.0 float=False ok
|
||||
calcom/calcom-postgres: 18-alpine -> maj=18-alpine all=18-alpine float=True ok
|
||||
calibre-web/calibre-web: v4.0.6 -> maj=v4.0.8 all=v4.0.8 float=False ok
|
||||
claper/claper: 2.5 -> maj=2.5 all=3.0 float=True ok
|
||||
claper/claper-postgres: 17-alpine -> maj=17-alpine all=18-alpine float=True ok
|
||||
code-server/code-server: 4.129.0 -> maj=4.139.1 all=4.139.1 float=False ok
|
||||
crafty-controller/crafty-controller: 4.10.7 -> maj=4.11.0 all=4.11.0 float=False ok
|
||||
docmost/docmost: 0.96.0 -> maj=0.96.0 all=0.96.0 float=False ok
|
||||
docmost/docmost-postgres: 18-alpine -> maj=18-alpine all=18-alpine float=True ok
|
||||
docmost/docmost-redis: 7-alpine -> maj=7-alpine all=8-alpine float=True ok
|
||||
emby/emby: 4.11.0.3 -> maj=4.11.0.4 all=4.11.0.4 float=False ok
|
||||
ghost/ghost: 6.65.0-alpine -> maj=6.67.0-alpine all=6.67.0-alpine float=False ok
|
||||
gitea/gitea: 1.27.0 -> maj=1.27.3 all=28.0.0 float=False ok
|
||||
glance/glance: v0.8.5 -> maj=v0.8.6 all=v0.8.6 float=False ok
|
||||
gokapi/gokapi: v1.9.6 -> maj=v1.9.6 all=v2.2.4 float=False ok
|
||||
grafana/grafana: 13.2.2 -> maj=13.2.3 all=13.2.3 float=False ok
|
||||
gramps-web/gramps-web: v25.6.0 -> maj=v25.6.0 all=v25.6.0 float=False ok
|
||||
home-assistant/home-assistant: 2026.9.3 -> maj=2026.9.4 all=2026.9.4 float=False ok
|
||||
homebox/homebox: 0.26.2 -> maj=0.26.2 all=0.26.2 float=False ok
|
||||
homepage/homepage: v1.13.2 -> maj=v1.13.2 all=v2.4.0 float=False ok
|
||||
immich/immich-server: v3.2.2 -> maj=None all=None float=False error HTTPError: 429 Client Error: Too Many Requests for url: https://ghcr.io/v2/immich-app/immich-server/tags/list?last=commit-43b76ddf3408f82e9ac2d0d82814c7508e962cc7&n=1000
|
||||
immich/immich-machine-learning: v3.2.2 -> maj=None all=None float=False error HTTPError: 429 Client Error: Too Many Requests for url: https://ghcr.io/v2/immich-app/immich-machine-learning/tags/list?last=pr-13094-armnn&n=1000
|
||||
immich/immich-postgres: 16-vectorchord0.4.3-pgvectors0.2.0 -> maj=16-vectorchord0.4.3-pgvectors0.3.0 all=17-vectorchord0.4.3-pgvectors0.3.0 float=True ok
|
||||
immich/immich-redis: 7-alpine -> maj=7-alpine all=8-alpine float=True ok
|
||||
jellyfin/jellyfin: 10.11.11 -> maj=10.11.11 all=10.11.11 float=False ok
|
||||
kimai/kimai: apache-2.57.0 -> maj=apache-2.57.0 all=apache-2.57.0 float=False ok
|
||||
kimai/kimai-db: 11.8 -> maj=11.8 all=13.0 float=True ok
|
||||
komga/komga: 1.27.1 -> maj=1.28.0 all=1.28.0 float=False ok
|
||||
mealie/mealie: v3.28.0 -> maj=v3.28.0 all=v3.28.0 float=False ok
|
||||
n8n/n8n: 2.41.2 -> maj=2.42.1 all=2.42.1 float=False ok
|
||||
navidrome/navidrome: 0.64.1 -> maj=0.64.2 all=0.64.2 float=False ok
|
||||
nextcloud/nextcloud: 34.0.4-apache -> maj=34.0.4-apache all=35.0.1-apache float=False ok
|
||||
nextcloud/nextcloud-db: 12.3 -> maj=12.3 all=13.0 float=True ok
|
||||
nextcloud/nextcloud-redis: 7-alpine -> maj=7-alpine all=8-alpine float=True ok
|
||||
onlyoffice/onlyoffice: 9.4.0 -> maj=9.4.0 all=9.4.0 float=False ok
|
||||
opengist/opengist: 1.15 -> maj=1.15 all=1.15 float=True ok
|
||||
outline/outline: 1.9.1 -> maj=1.10.1 all=1.10.1 float=False ok
|
||||
outline/outline-postgres: 16-alpine -> maj=16-alpine all=18-alpine float=True ok
|
||||
outline/outline-redis: 7-alpine -> maj=7-alpine all=8-alpine float=True ok
|
||||
paperless-ngx/paperless-webserver: 2.20.15 -> maj=2.20.15 all=3.2.1 float=False ok
|
||||
paperless-ngx/paperless-postgres: 18-alpine -> maj=18-alpine all=18-alpine float=True ok
|
||||
paperless-ngx/paperless-redis: 7-alpine -> maj=7-alpine all=8-alpine float=True ok
|
||||
papra/papra: 26.6.2-rootless -> maj=26.6.2-rootless all=26.6.2-rootless float=False ok
|
||||
plant-it/plant-it: 0.10.0 -> maj=None all=None float=False error HTTPError: 401 Client Error: Unauthorized for url: https://registry-1.docker.io/v2/msdeluise/plant-it/tags/list?n=1000
|
||||
plex/plex: 1.41.4.9463-630c9f557 -> maj=1.43.4.10903-e5521bd8c all=1.43.4.10903-e5521bd8c float=False ok
|
||||
privatebin/privatebin: 2.0.6 -> maj=2.0.6 all=2.0.6 float=False ok
|
||||
radarr/radarr: 6.4.4 -> maj=6.4.4 all=6.4.4 float=False ok
|
||||
rallly/rallly: 4.11.1 -> maj=4.15.3 all=4.15.3 float=False ok
|
||||
rallly/rallly-postgres: 16-alpine -> maj=16-alpine all=18-alpine float=True ok
|
||||
recipe-importer/recipe-importer: internal image, not upstream
|
||||
romm/romm: 5.3.1 -> maj=5.3.1 all=5.3.1 float=False ok
|
||||
romm/romm-db: 11.8 -> maj=11.8 all=13.0 float=True ok
|
||||
romm/romm-redis: 7-alpine -> maj=7-alpine all=8-alpine float=True ok
|
||||
seerr/seerr: 2.7.3 -> maj=2.7.3 all=2.7.3 float=False ok
|
||||
sonarr/sonarr: 4.0.20 -> maj=4.0.20 all=4.0.20 float=False ok
|
||||
sparkyfitness/sparkyfitness-db: 15-alpine -> maj=15-alpine all=18-alpine float=True ok
|
||||
sparkyfitness/sparkyfitness-server: v0.17.3 -> maj=v0.17.3 all=v1.7.3 float=False ok
|
||||
sparkyfitness/sparkyfitness-frontend: v0.17.3 -> maj=v0.17.3 all=v1.7.3 float=False ok
|
||||
tandoor/tandoor: 2.6.15 -> maj=2.6.15 all=2.6.15 float=False ok
|
||||
tandoor/tandoor-postgres: 17-alpine -> maj=17-alpine all=18-alpine float=True ok
|
||||
termix/termix: 2.8.0 -> maj=2.8.0 all=2.8.0 float=False ok
|
||||
uptime-kuma/uptime-kuma: 2.4.0 -> maj=2.5.5 all=2.5.5 float=False ok
|
||||
vaultwarden/vaultwarden: 1.36.0-alpine -> maj=1.37.3-alpine all=1.37.3-alpine float=False ok
|
||||
vikunja/vikunja: 2.6.0 -> maj=2.6.0 all=2.6.0 float=False ok
|
||||
wanderer/wanderer: v0.20.0 -> maj=v0.21.0 all=v0.21.0 float=False ok
|
||||
wanderer/wanderer-db: v0.20.0 -> maj=v0.21.0 all=v0.21.0 float=False ok
|
||||
wanderer/wanderer-search: v1.36.0 -> maj=v1.54.2 all=v1.54.2 float=False ok
|
||||
wger/wger: 2.6 -> maj=2.7 all=2.7 float=True ok
|
||||
wishlist/wishlist: v0.67.1 -> maj=v0.67.1 all=v0.67.1 float=False ok
|
||||
zipline/zipline: 4.6.1 -> maj=4.8.0 all=4.8.0 float=False ok
|
||||
zipline/zipline-postgres: 16-alpine -> maj=16-alpine all=18-alpine float=True ok
|
||||
rows 79
|
||||
@@ -0,0 +1,8 @@
|
||||
429, waiting 20 s
|
||||
429, waiting 20 s
|
||||
429, waiting 20 s
|
||||
ghcr.io/immich-app/immich-server:v3.2.2 30725 v3.2.4 v3.2.4
|
||||
429, waiting 20 s
|
||||
429, waiting 20 s
|
||||
429, waiting 20 s
|
||||
ghcr.io/immich-app/immich-machine-learning:v3.2.2 206974 v3.2.4 v3.2.4
|
||||
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Re-measure the rows 00-currency.py recorded as HTTP 429 (ghcr rate limit), slower, and merge them
|
||||
into 01-currency-raw.json. Same functions, same rules; only the pacing differs."""
|
||||
import importlib.util, json, sys, time
|
||||
sys.argv = ["x", "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu", "."]
|
||||
spec = importlib.util.spec_from_file_location("c", "00-currency.py")
|
||||
c = importlib.util.module_from_spec(spec); spec.loader.exec_module(c)
|
||||
orig = c.S.get
|
||||
def slow_get(*a, **k):
|
||||
for i in range(8):
|
||||
r = orig(*a, **k)
|
||||
if r.status_code != 429:
|
||||
time.sleep(1.0)
|
||||
return r
|
||||
wait = int(r.headers.get("Retry-After", "0") or 0) or 20 * (i + 1)
|
||||
print("429, waiting", wait, "s"); time.sleep(wait)
|
||||
return r
|
||||
c.S.get = slow_get
|
||||
rows = json.load(open("01-currency-raw.json"))
|
||||
for row in rows:
|
||||
if "429" not in row.get("error", ""):
|
||||
continue
|
||||
host, hrepo = c.host_repo(row["repo"])
|
||||
tags = c.tags_all(host, hrepo)
|
||||
res = c.newest(tags, row["tag"])
|
||||
row.pop("error"); row["status"] = "ok"; row["n_tags"] = len(tags); row["pinned_tag_listed"] = row["tag"] in tags
|
||||
ba, bm = res["all"], res["maj"]
|
||||
row["newest_all"] = ba[0] if ba else None; row["newest_major"] = bm[0] if bm else None
|
||||
row["behind_within_major"] = bool(bm and bm[1] > res["cur"]); row["behind_across_major"] = bool(ba and ba[1][0] > res["cur"][0])
|
||||
row["major"] = res["cur"][0]; row["retried_after_429"] = True
|
||||
print(row["ref"], len(tags), row["newest_major"], row["newest_all"])
|
||||
json.dump(rows, open("01-currency-raw.json", "w"), indent=1)
|
||||
@@ -0,0 +1 @@
|
||||
test-record gate — 53 template(s) read, 29 carry a ladder, 0 convicted
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"unique_pins": 67,
|
||||
"unique_floating_pins": 12,
|
||||
"images": {
|
||||
"images": 79,
|
||||
"exact": 53,
|
||||
"floating": 24,
|
||||
"internal": 1,
|
||||
"unmeasured": 1
|
||||
},
|
||||
"image_rows_behind_within_major": 29,
|
||||
"image_rows_behind_across_major": 28,
|
||||
"apps_behind_within_major_any_image": 25,
|
||||
"apps_behind_across_major_any_image": 19,
|
||||
"apps_behind_within_major_exact_pins_only": 24,
|
||||
"apps_behind_across_major_exact_pins_only": 8,
|
||||
"apps_behind_within_major_floating_only": 2,
|
||||
"apps_behind_across_major_floating_only": 14,
|
||||
"apps_behind_at_all": 37,
|
||||
"night_a_unconditional": 28,
|
||||
"night_a_conditional": 1,
|
||||
"night_b_any_proven": 29
|
||||
}
|
||||
night (a) unconditional 28 ['actualbudget', 'adventurelog', 'audiobookshelf', 'bookstack', 'calcom', 'claper', 'docmost', 'emby', 'ghost', 'grafana', 'home-assistant', 'immich', 'kimai', 'komga', 'mealie', 'n8n', 'navidrome', 'opengist', 'paperless-ngx', 'papra', 'privatebin', 'radarr', 'romm', 'sonarr', 'tandoor', 'termix', 'vikunja', 'wishlist']
|
||||
night (a) conditional 1 ['nextcloud']
|
||||
night (b) any proven 29
|
||||
@@ -0,0 +1,218 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Turn 01-currency-raw.json + the catalog's ladders + the fixture registries into the audit tables.
|
||||
Offline: reads files only. Usage: python3 04-analyse.py <catalog-root> <outdir>
|
||||
|
||||
THE NIGHT RULE, as the controller codes it (felhom-controller d48da6c, internal/stacks/unattended.go
|
||||
legCandidate L392-463, with nextLadderStep ladder.go L152-199). Catalog-side part only — a box can
|
||||
still skip for its own reasons (held, stopped, switch off, failed_before, no fresh whole copy, W+5h,
|
||||
or a guarded-update refusal: no_backup / memory / disk / busy):
|
||||
* the template has an update_ladder (else no_test_record, L423-424);
|
||||
* the NEWEST entry whose `from` equals the box's pinned images EXACTLY, every service (sameRefs,
|
||||
L426-432) — none: older_than_ladder, or no_test_record when the box is at the head (L433-439);
|
||||
* that entry's verdict is "proven" (L441), and marks.needs_person is null/blank (L444);
|
||||
* marks.files_may_change → only with a fresh whole copy on the box (L450-461) — "conditional".
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
CAT, OUT = sys.argv[1], sys.argv[2]
|
||||
sys.path.insert(0, os.path.join(CAT, "scripts"))
|
||||
import ladder # noqa: E402
|
||||
import upgrade_fixtures as fx1 # noqa: E402 (bench-native fixtures)
|
||||
|
||||
# the box fixture modules import nothing heavy; read their registries as the bench does
|
||||
import upgrade_fixtures_box as fxb # noqa: E402
|
||||
import upgrade_fixtures_box28 as fxb28 # noqa: E402
|
||||
|
||||
rows = json.load(open(os.path.join(OUT, "01-currency-raw.json")))
|
||||
|
||||
# SHAPE SWITCHES found by the control (09-shape-switch-check.txt, detail in 10-shape-switch-detail.txt):
|
||||
# the upstream now releases under a DIFFERENT tag shape, so the shape rule read "up to date". Applied
|
||||
# here by hand, each with the tag list that shows it. sonarr (a `5.14-2.0.0...-ls5` develop-style tag)
|
||||
# and tandoor (a `dependabot-...` branch tag) were flagged by the same control and are NOT releases —
|
||||
# left as measured.
|
||||
SHAPE_SWITCH = {
|
||||
"ghcr.io/gramps-project/grampsweb:v25.6.0": dict(newest_major="v25.6.0", newest_all="26.9.1",
|
||||
behind_within_major=False, behind_across_major=True,
|
||||
shape_note="upstream dropped the `v` prefix in 2026 (26.x.y); calendar versions, so the 25→26 step is a new year, counted as across"),
|
||||
"jellyfin/jellyfin:10.11.11": dict(newest_major="10.11.11", newest_all="12.1",
|
||||
behind_within_major=False, behind_across_major=True,
|
||||
shape_note="upstream 12.x publishes two-part tags (`12.1`); 12.0 went through rc1..rc7"),
|
||||
"kimai/kimai2:apache-2.57.0": dict(newest_major="2.67.0", newest_all="2.67.0",
|
||||
behind_within_major=True, behind_across_major=False,
|
||||
shape_note="upstream stopped publishing `apache-` tags after 2.57.0; plain `2.67.0` exists (whether it is the same apache variant is NOT checked)"),
|
||||
}
|
||||
for r in rows:
|
||||
if r["ref"] in SHAPE_SWITCH:
|
||||
r.update(SHAPE_SWITCH[r["ref"]])
|
||||
tdir = os.path.join(CAT, "templates")
|
||||
apps = sorted(os.listdir(tdir))
|
||||
|
||||
DB_MARKERS = ("postgres", "mariadb", "mysql", "postgis", "mongo")
|
||||
NO_SEED_BY_DESIGN = {"vaultwarden": "closed sign-up (R-624)", "zipline": "closed sign-up (R-624)"}
|
||||
|
||||
|
||||
def fixture_state(app):
|
||||
box = fxb.FIXTURES.get(app) or fxb28.FIXTURES28.get(app)
|
||||
if app in NO_SEED_BY_DESIGN:
|
||||
return "no route by design: " + NO_SEED_BY_DESIGN[app]
|
||||
if box is not None and type(box).__name__ == "NoRoute":
|
||||
return "no route: " + box.tried.split(";")[0][:70]
|
||||
if app == "gitea":
|
||||
return "fixture exists, fails at the web installer (R-624, fixable)"
|
||||
if box is not None or app in fx1.FIXTURES:
|
||||
return "fixture exists"
|
||||
return "fixture needed"
|
||||
|
||||
|
||||
def data_tier(app, imgs, fy_text):
|
||||
engines = [s for s, r in imgs.items() if any(m in r.split(":")[0].lower() for m in DB_MARKERS)]
|
||||
mandatory = "class: mandatory" in fy_text
|
||||
if engines or mandatory:
|
||||
why = []
|
||||
if engines:
|
||||
why.append("DB engine: " + ", ".join(engines))
|
||||
if mandatory:
|
||||
why.append("household files (backup class mandatory)")
|
||||
return 1, "; ".join(why)
|
||||
if app in ("bentopdf", "onlyoffice", "homepage", "glance"):
|
||||
return 3, "little or no household data (stateless or config only)"
|
||||
return 2, "household data in the app's own volume (SQLite / files)"
|
||||
|
||||
|
||||
img_by_app = {}
|
||||
for r in rows:
|
||||
img_by_app.setdefault(r["app"], []).append(r)
|
||||
|
||||
ladder_rows, plan_rows = [], []
|
||||
count_a_uncond = count_a_cond = count_b = 0
|
||||
a_apps, a_cond_apps, b_apps = [], [], []
|
||||
for app in apps:
|
||||
d = os.path.join(tdir, app)
|
||||
fy_text = open(os.path.join(d, ".felhom.yml"), encoding="utf-8").read()
|
||||
imgs = ladder.images_in(open(os.path.join(d, "docker-compose.yml"), encoding="utf-8").read())
|
||||
entries, _r, errs = ladder.parse(fy_text)
|
||||
n = len(entries)
|
||||
proven = sum(1 for e in entries if e.get("verdict") == "proven")
|
||||
backfilled = sum(1 for e in entries if e.get("backfilled"))
|
||||
needs = [e["marks"]["needs_person"] for e in entries if (e.get("marks") or {}).get("needs_person")]
|
||||
fmc = sum(1 for e in entries if (e.get("marks") or {}).get("files_may_change"))
|
||||
conv = sum(1 for e in entries if e.get("engine_conversion"))
|
||||
head_eq = (entries[-1]["to"] == imgs) if entries else None
|
||||
# the step a box AT THE PREVIOUS PIN (= head.from) would be offered tonight, catalog-side
|
||||
night = "no ladder: never (no_test_record)"
|
||||
if entries:
|
||||
pinned = entries[-1]["from"]
|
||||
idx = max(i for i, e in enumerate(entries) if e["from"] == pinned)
|
||||
e = entries[idx]
|
||||
np_ = (e.get("marks") or {}).get("needs_person")
|
||||
if e.get("verdict") != "proven":
|
||||
night = "skip: not_proven"
|
||||
elif np_ and str(np_).strip():
|
||||
night = "skip: needs_person"
|
||||
elif (e.get("marks") or {}).get("files_may_change"):
|
||||
night = "conditional: files_may_change (needs a fresh whole copy)"
|
||||
count_a_cond += 1
|
||||
a_cond_apps.append(app)
|
||||
else:
|
||||
night = "yes"
|
||||
count_a_uncond += 1
|
||||
a_apps.append(app)
|
||||
if proven:
|
||||
count_b += 1
|
||||
b_apps.append(app)
|
||||
since = ""
|
||||
for l in fy_text.splitlines():
|
||||
if l.startswith("catalog_since:"):
|
||||
since = l.split(":", 1)[1].strip().strip('"')
|
||||
ladder_rows.append((app, n, proven, backfilled, len(needs), fmc, conv, head_eq, night, since))
|
||||
|
||||
tier, why = data_tier(app, imgs, fy_text)
|
||||
ir = img_by_app.get(app, [])
|
||||
wm = sum(1 for r in ir if r.get("behind_within_major"))
|
||||
am = sum(1 for r in ir if r.get("behind_across_major"))
|
||||
plan_rows.append((tier, -(1 if n == 0 else 0), -wm, -am, app, why, n, wm, am, fixture_state(app)))
|
||||
|
||||
# ---- per-image table
|
||||
def yn(v):
|
||||
return "yes" if v else ("no" if v is False else "?")
|
||||
|
||||
lines = ["| app | service | pinned | newest same major | newest at all | behind in major | behind across major | note |",
|
||||
"|---|---|---|---|---|---|---|---|"]
|
||||
stats = {"images": 0, "exact": 0, "floating": 0, "internal": 0, "unmeasured": 0}
|
||||
apps_in, apps_across, apps_float_only = set(), set(), set()
|
||||
img_in = img_across = 0
|
||||
for r in rows:
|
||||
stats["images"] += 1
|
||||
note = []
|
||||
if r["status"] == "internal":
|
||||
stats["internal"] += 1
|
||||
lines.append(f"| {r['app']} | {r['service']} | `{r['tag']}` | — | — | n/a | n/a | Felhom's own image, not upstream |")
|
||||
continue
|
||||
if r["status"] != "ok":
|
||||
stats["unmeasured"] += 1
|
||||
lines.append(f"| {r['app']} | {r['service']} | `{r['tag']}` | ? | ? | ? | ? | NOT MEASURED: {r.get('error','')[:60]} |")
|
||||
continue
|
||||
kind = "floating (line)" if r["floating"] else "exact"
|
||||
stats["floating" if r["floating"] else "exact"] += 1
|
||||
if r["floating"]:
|
||||
note.append("floating (line)")
|
||||
if r.get("moved_since_tested") is False:
|
||||
note.append("digest = ladder's tested digest")
|
||||
elif r.get("moved_since_tested"):
|
||||
note.append("digest MOVED since tested")
|
||||
elif r.get("current_digest"):
|
||||
note.append("no ladder digest to compare")
|
||||
if r.get("shape_note"):
|
||||
note.append("SHAPE SWITCH: " + r["shape_note"])
|
||||
if r.get("retried_after_429"):
|
||||
note.append("measured on retry after ghcr 429")
|
||||
wm, am = r.get("behind_within_major"), r.get("behind_across_major")
|
||||
if wm:
|
||||
img_in += 1
|
||||
apps_in.add(r["app"])
|
||||
if am:
|
||||
img_across += 1
|
||||
apps_across.add(r["app"])
|
||||
lines.append(f"| {r['app']} | {r['service']} | `{r['tag']}` | `{r.get('newest_major') or '—'}` | `{r.get('newest_all') or '—'}` | {yn(wm)} | {yn(am)} | {'; '.join(note)} |")
|
||||
|
||||
exact_rows = [r for r in rows if r["status"] == "ok" and not r["floating"]]
|
||||
exact_in = {r["app"] for r in exact_rows if r.get("behind_within_major")}
|
||||
exact_across = {r["app"] for r in exact_rows if r.get("behind_across_major")}
|
||||
float_rows = [r for r in rows if r["status"] == "ok" and r["floating"]]
|
||||
float_in = {r["app"] for r in float_rows if r.get("behind_within_major")}
|
||||
float_across = {r["app"] for r in float_rows if r.get("behind_across_major")}
|
||||
behind_any = apps_in | apps_across
|
||||
summary = {
|
||||
"unique_pins": len({r["ref"] for r in rows}), "unique_floating_pins": sorted({r["ref"] for r in rows if r.get("floating")}),
|
||||
"images": stats, "image_rows_behind_within_major": img_in, "image_rows_behind_across_major": img_across,
|
||||
"apps_behind_within_major_any_image": sorted(apps_in),
|
||||
"apps_behind_across_major_any_image": sorted(apps_across),
|
||||
"apps_behind_within_major_exact_pins_only": sorted(exact_in),
|
||||
"apps_behind_across_major_exact_pins_only": sorted(exact_across),
|
||||
"apps_behind_within_major_floating_only": sorted(float_in),
|
||||
"apps_behind_across_major_floating_only": sorted(float_across),
|
||||
"apps_behind_at_all": sorted(behind_any),
|
||||
"night_a_unconditional": a_apps, "night_a_conditional": a_cond_apps, "night_b_any_proven": b_apps,
|
||||
}
|
||||
json.dump(summary, open(os.path.join(OUT, "05-summary.json"), "w"), indent=1)
|
||||
open(os.path.join(OUT, "06-image-table.md"), "w").write("\n".join(lines) + "\n")
|
||||
|
||||
ll = ["| app | entries | proven | of which backfilled | needs_person | files_may_change | engine conversion | head = compose | box one step behind, tonight | catalog_since |",
|
||||
"|---|---|---|---|---|---|---|---|---|---|"]
|
||||
for (app, n, p, bf, np_, fmc, conv, heq, night, since) in ladder_rows:
|
||||
ll.append(f"| {app} | {n} | {p} | {bf} | {np_} | {fmc} | {conv} | {'—' if heq is None else ('yes' if heq else 'NO')} | {night} | {since} |")
|
||||
open(os.path.join(OUT, "07-ladder-table.md"), "w").write("\n".join(ll) + "\n")
|
||||
|
||||
plan_rows.sort()
|
||||
pl = ["| rank | app | data at risk | ladder entries | images behind in major | across major | fixture |",
|
||||
"|---|---|---|---|---|---|---|"]
|
||||
for i, (tier, _nl, _wm, _am, app, why, n, wm, am, fs) in enumerate(plan_rows, 1):
|
||||
pl.append(f"| {i} | {app} | T{tier}: {why} | {n} | {wm} | {am} | {fs} |")
|
||||
open(os.path.join(OUT, "08-plan-table.md"), "w").write("\n".join(pl) + "\n")
|
||||
|
||||
print(json.dumps({k: (len(v) if isinstance(v, list) else v) for k, v in summary.items()}, indent=1))
|
||||
print("night (a) unconditional", count_a_uncond, a_apps)
|
||||
print("night (a) conditional", count_a_cond, a_cond_apps)
|
||||
print("night (b) any proven", count_b)
|
||||
@@ -0,0 +1,233 @@
|
||||
{
|
||||
"unique_pins": 67,
|
||||
"unique_floating_pins": [
|
||||
"ghcr.io/claperco/claper:2.5",
|
||||
"ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0",
|
||||
"ghcr.io/thomiceli/opengist:1.15",
|
||||
"mariadb:11.8",
|
||||
"mariadb:12.3",
|
||||
"postgis/postgis:16-3.5-alpine",
|
||||
"postgres:15-alpine",
|
||||
"postgres:16-alpine",
|
||||
"postgres:17-alpine",
|
||||
"postgres:18-alpine",
|
||||
"redis:7-alpine",
|
||||
"wger/server:2.6"
|
||||
],
|
||||
"images": {
|
||||
"images": 79,
|
||||
"exact": 53,
|
||||
"floating": 24,
|
||||
"internal": 1,
|
||||
"unmeasured": 1
|
||||
},
|
||||
"image_rows_behind_within_major": 29,
|
||||
"image_rows_behind_across_major": 28,
|
||||
"apps_behind_within_major_any_image": [
|
||||
"audiobookshelf",
|
||||
"bentopdf",
|
||||
"bookstack",
|
||||
"calibre-web",
|
||||
"code-server",
|
||||
"crafty-controller",
|
||||
"emby",
|
||||
"ghost",
|
||||
"gitea",
|
||||
"glance",
|
||||
"grafana",
|
||||
"home-assistant",
|
||||
"immich",
|
||||
"kimai",
|
||||
"komga",
|
||||
"n8n",
|
||||
"navidrome",
|
||||
"outline",
|
||||
"plex",
|
||||
"rallly",
|
||||
"uptime-kuma",
|
||||
"vaultwarden",
|
||||
"wanderer",
|
||||
"wger",
|
||||
"zipline"
|
||||
],
|
||||
"apps_behind_across_major_any_image": [
|
||||
"adventurelog",
|
||||
"bookstack",
|
||||
"claper",
|
||||
"docmost",
|
||||
"gitea",
|
||||
"gokapi",
|
||||
"gramps-web",
|
||||
"homepage",
|
||||
"immich",
|
||||
"jellyfin",
|
||||
"kimai",
|
||||
"nextcloud",
|
||||
"outline",
|
||||
"paperless-ngx",
|
||||
"rallly",
|
||||
"romm",
|
||||
"sparkyfitness",
|
||||
"tandoor",
|
||||
"zipline"
|
||||
],
|
||||
"apps_behind_within_major_exact_pins_only": [
|
||||
"audiobookshelf",
|
||||
"bentopdf",
|
||||
"bookstack",
|
||||
"calibre-web",
|
||||
"code-server",
|
||||
"crafty-controller",
|
||||
"emby",
|
||||
"ghost",
|
||||
"gitea",
|
||||
"glance",
|
||||
"grafana",
|
||||
"home-assistant",
|
||||
"immich",
|
||||
"kimai",
|
||||
"komga",
|
||||
"n8n",
|
||||
"navidrome",
|
||||
"outline",
|
||||
"plex",
|
||||
"rallly",
|
||||
"uptime-kuma",
|
||||
"vaultwarden",
|
||||
"wanderer",
|
||||
"zipline"
|
||||
],
|
||||
"apps_behind_across_major_exact_pins_only": [
|
||||
"gitea",
|
||||
"gokapi",
|
||||
"gramps-web",
|
||||
"homepage",
|
||||
"jellyfin",
|
||||
"nextcloud",
|
||||
"paperless-ngx",
|
||||
"sparkyfitness"
|
||||
],
|
||||
"apps_behind_within_major_floating_only": [
|
||||
"immich",
|
||||
"wger"
|
||||
],
|
||||
"apps_behind_across_major_floating_only": [
|
||||
"adventurelog",
|
||||
"bookstack",
|
||||
"claper",
|
||||
"docmost",
|
||||
"immich",
|
||||
"kimai",
|
||||
"nextcloud",
|
||||
"outline",
|
||||
"paperless-ngx",
|
||||
"rallly",
|
||||
"romm",
|
||||
"sparkyfitness",
|
||||
"tandoor",
|
||||
"zipline"
|
||||
],
|
||||
"apps_behind_at_all": [
|
||||
"adventurelog",
|
||||
"audiobookshelf",
|
||||
"bentopdf",
|
||||
"bookstack",
|
||||
"calibre-web",
|
||||
"claper",
|
||||
"code-server",
|
||||
"crafty-controller",
|
||||
"docmost",
|
||||
"emby",
|
||||
"ghost",
|
||||
"gitea",
|
||||
"glance",
|
||||
"gokapi",
|
||||
"grafana",
|
||||
"gramps-web",
|
||||
"home-assistant",
|
||||
"homepage",
|
||||
"immich",
|
||||
"jellyfin",
|
||||
"kimai",
|
||||
"komga",
|
||||
"n8n",
|
||||
"navidrome",
|
||||
"nextcloud",
|
||||
"outline",
|
||||
"paperless-ngx",
|
||||
"plex",
|
||||
"rallly",
|
||||
"romm",
|
||||
"sparkyfitness",
|
||||
"tandoor",
|
||||
"uptime-kuma",
|
||||
"vaultwarden",
|
||||
"wanderer",
|
||||
"wger",
|
||||
"zipline"
|
||||
],
|
||||
"night_a_unconditional": [
|
||||
"actualbudget",
|
||||
"adventurelog",
|
||||
"audiobookshelf",
|
||||
"bookstack",
|
||||
"calcom",
|
||||
"claper",
|
||||
"docmost",
|
||||
"emby",
|
||||
"ghost",
|
||||
"grafana",
|
||||
"home-assistant",
|
||||
"immich",
|
||||
"kimai",
|
||||
"komga",
|
||||
"mealie",
|
||||
"n8n",
|
||||
"navidrome",
|
||||
"opengist",
|
||||
"paperless-ngx",
|
||||
"papra",
|
||||
"privatebin",
|
||||
"radarr",
|
||||
"romm",
|
||||
"sonarr",
|
||||
"tandoor",
|
||||
"termix",
|
||||
"vikunja",
|
||||
"wishlist"
|
||||
],
|
||||
"night_a_conditional": [
|
||||
"nextcloud"
|
||||
],
|
||||
"night_b_any_proven": [
|
||||
"actualbudget",
|
||||
"adventurelog",
|
||||
"audiobookshelf",
|
||||
"bookstack",
|
||||
"calcom",
|
||||
"claper",
|
||||
"docmost",
|
||||
"emby",
|
||||
"ghost",
|
||||
"grafana",
|
||||
"home-assistant",
|
||||
"immich",
|
||||
"kimai",
|
||||
"komga",
|
||||
"mealie",
|
||||
"n8n",
|
||||
"navidrome",
|
||||
"nextcloud",
|
||||
"opengist",
|
||||
"paperless-ngx",
|
||||
"papra",
|
||||
"privatebin",
|
||||
"radarr",
|
||||
"romm",
|
||||
"sonarr",
|
||||
"tandoor",
|
||||
"termix",
|
||||
"vikunja",
|
||||
"wishlist"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
| app | service | pinned | newest same major | newest at all | behind in major | behind across major | note |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| actualbudget | actualbudget | `26.9.0` | `26.9.0` | `26.9.0` | no | no | |
|
||||
| adventurelog | adventurelog | `v0.13.0` | `v0.13.0` | `v0.13.0` | no | no | |
|
||||
| adventurelog | adventurelog-postgres | `16-3.5-alpine` | `16-3.5-alpine` | `18-3.6-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| adventurelog | adventurelog-frontend | `v0.13.0` | `v0.13.0` | `v0.13.0` | no | no | |
|
||||
| audiobookshelf | audiobookshelf | `2.36.1` | `2.37.1` | `2.37.1` | yes | no | |
|
||||
| bentopdf | bentopdf | `v2.8.6` | `v2.8.8` | `v2.8.8` | yes | no | |
|
||||
| bookstack | bookstack | `26.05.5` | `26.09.1` | `26.09.1` | yes | no | |
|
||||
| bookstack | bookstack-db | `12.3` | `12.3` | `13.0` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| calcom | calcom | `v6.2.0` | `v6.2.0` | `v6.2.0` | no | no | |
|
||||
| calcom | calcom-postgres | `18-alpine` | `18-alpine` | `18-alpine` | no | no | floating (line); digest = ladder's tested digest |
|
||||
| calibre-web | calibre-web | `v4.0.6` | `v4.0.8` | `v4.0.8` | yes | no | |
|
||||
| claper | claper | `2.5` | `2.5` | `3.0` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| claper | claper-postgres | `17-alpine` | `17-alpine` | `18-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| code-server | code-server | `4.129.0` | `4.139.1` | `4.139.1` | yes | no | |
|
||||
| crafty-controller | crafty-controller | `4.10.7` | `4.11.0` | `4.11.0` | yes | no | |
|
||||
| docmost | docmost | `0.96.0` | `0.96.0` | `0.96.0` | no | no | |
|
||||
| docmost | docmost-postgres | `18-alpine` | `18-alpine` | `18-alpine` | no | no | floating (line); digest = ladder's tested digest |
|
||||
| docmost | docmost-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| emby | emby | `4.11.0.3` | `4.11.0.4` | `4.11.0.4` | yes | no | |
|
||||
| ghost | ghost | `6.65.0-alpine` | `6.67.0-alpine` | `6.67.0-alpine` | yes | no | |
|
||||
| gitea | gitea | `1.27.0` | `1.27.3` | `28.0.0` | yes | yes | |
|
||||
| glance | glance | `v0.8.5` | `v0.8.6` | `v0.8.6` | yes | no | |
|
||||
| gokapi | gokapi | `v1.9.6` | `v1.9.6` | `v2.2.4` | no | yes | |
|
||||
| grafana | grafana | `13.2.2` | `13.2.3` | `13.2.3` | yes | no | |
|
||||
| gramps-web | gramps-web | `v25.6.0` | `v25.6.0` | `26.9.1` | no | yes | SHAPE SWITCH: upstream dropped the `v` prefix in 2026 (26.x.y); calendar versions, so the 25→26 step is a new year, counted as across |
|
||||
| home-assistant | home-assistant | `2026.9.3` | `2026.9.4` | `2026.9.4` | yes | no | |
|
||||
| homebox | homebox | `0.26.2` | `0.26.2` | `0.26.2` | no | no | |
|
||||
| homepage | homepage | `v1.13.2` | `v1.13.2` | `v2.4.0` | no | yes | |
|
||||
| immich | immich-server | `v3.2.2` | `v3.2.4` | `v3.2.4` | yes | no | measured on retry after ghcr 429 |
|
||||
| immich | immich-machine-learning | `v3.2.2` | `v3.2.4` | `v3.2.4` | yes | no | measured on retry after ghcr 429 |
|
||||
| immich | immich-postgres | `16-vectorchord0.4.3-pgvectors0.2.0` | `16-vectorchord0.4.3-pgvectors0.3.0` | `17-vectorchord0.4.3-pgvectors0.3.0` | yes | yes | floating (line); digest = ladder's tested digest |
|
||||
| immich | immich-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| jellyfin | jellyfin | `10.11.11` | `10.11.11` | `12.1` | no | yes | SHAPE SWITCH: upstream 12.x publishes two-part tags (`12.1`); 12.0 went through rc1..rc7 |
|
||||
| kimai | kimai | `apache-2.57.0` | `2.67.0` | `2.67.0` | yes | no | SHAPE SWITCH: upstream stopped publishing `apache-` tags after 2.57.0; plain `2.67.0` exists (whether it is the same apache variant is NOT checked) |
|
||||
| kimai | kimai-db | `11.8` | `11.8` | `13.0` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| komga | komga | `1.27.1` | `1.28.0` | `1.28.0` | yes | no | |
|
||||
| mealie | mealie | `v3.28.0` | `v3.28.0` | `v3.28.0` | no | no | |
|
||||
| n8n | n8n | `2.41.2` | `2.42.1` | `2.42.1` | yes | no | |
|
||||
| navidrome | navidrome | `0.64.1` | `0.64.2` | `0.64.2` | yes | no | |
|
||||
| nextcloud | nextcloud | `34.0.4-apache` | `34.0.4-apache` | `35.0.1-apache` | no | yes | |
|
||||
| nextcloud | nextcloud-db | `12.3` | `12.3` | `13.0` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| nextcloud | nextcloud-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| onlyoffice | onlyoffice | `9.4.0` | `9.4.0` | `9.4.0` | no | no | |
|
||||
| opengist | opengist | `1.15` | `1.15` | `1.15` | no | no | floating (line); digest = ladder's tested digest |
|
||||
| outline | outline | `1.9.1` | `1.10.1` | `1.10.1` | yes | no | |
|
||||
| outline | outline-postgres | `16-alpine` | `16-alpine` | `18-alpine` | no | yes | floating (line); no ladder digest to compare |
|
||||
| outline | outline-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); no ladder digest to compare |
|
||||
| paperless-ngx | paperless-webserver | `2.20.15` | `2.20.15` | `3.2.1` | no | yes | |
|
||||
| paperless-ngx | paperless-postgres | `18-alpine` | `18-alpine` | `18-alpine` | no | no | floating (line); digest = ladder's tested digest |
|
||||
| paperless-ngx | paperless-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| papra | papra | `26.6.2-rootless` | `26.6.2-rootless` | `26.6.2-rootless` | no | no | |
|
||||
| plant-it | plant-it | `0.10.0` | ? | ? | ? | ? | NOT MEASURED: HTTPError: 401 Client Error: Unauthorized for url: https://r |
|
||||
| plex | plex | `1.41.4.9463-630c9f557` | `1.43.4.10903-e5521bd8c` | `1.43.4.10903-e5521bd8c` | yes | no | |
|
||||
| privatebin | privatebin | `2.0.6` | `2.0.6` | `2.0.6` | no | no | |
|
||||
| radarr | radarr | `6.4.4` | `6.4.4` | `6.4.4` | no | no | |
|
||||
| rallly | rallly | `4.11.1` | `4.15.3` | `4.15.3` | yes | no | |
|
||||
| rallly | rallly-postgres | `16-alpine` | `16-alpine` | `18-alpine` | no | yes | floating (line); no ladder digest to compare |
|
||||
| recipe-importer | recipe-importer | `v0.9.11` | — | — | n/a | n/a | Felhom's own image, not upstream |
|
||||
| romm | romm | `5.3.1` | `5.3.1` | `5.3.1` | no | no | |
|
||||
| romm | romm-db | `11.8` | `11.8` | `13.0` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| romm | romm-redis | `7-alpine` | `7-alpine` | `8-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| seerr | seerr | `2.7.3` | `2.7.3` | `2.7.3` | no | no | |
|
||||
| sonarr | sonarr | `4.0.20` | `4.0.20` | `4.0.20` | no | no | |
|
||||
| sparkyfitness | sparkyfitness-db | `15-alpine` | `15-alpine` | `18-alpine` | no | yes | floating (line); no ladder digest to compare |
|
||||
| sparkyfitness | sparkyfitness-server | `v0.17.3` | `v0.17.3` | `v1.7.3` | no | yes | |
|
||||
| sparkyfitness | sparkyfitness-frontend | `v0.17.3` | `v0.17.3` | `v1.7.3` | no | yes | |
|
||||
| tandoor | tandoor | `2.6.15` | `2.6.15` | `2.6.15` | no | no | |
|
||||
| tandoor | tandoor-postgres | `17-alpine` | `17-alpine` | `18-alpine` | no | yes | floating (line); digest = ladder's tested digest |
|
||||
| termix | termix | `2.8.0` | `2.8.0` | `2.8.0` | no | no | |
|
||||
| uptime-kuma | uptime-kuma | `2.4.0` | `2.5.5` | `2.5.5` | yes | no | |
|
||||
| vaultwarden | vaultwarden | `1.36.0-alpine` | `1.37.3-alpine` | `1.37.3-alpine` | yes | no | |
|
||||
| vikunja | vikunja | `2.6.0` | `2.6.0` | `2.6.0` | no | no | |
|
||||
| wanderer | wanderer | `v0.20.0` | `v0.21.0` | `v0.21.0` | yes | no | |
|
||||
| wanderer | wanderer-db | `v0.20.0` | `v0.21.0` | `v0.21.0` | yes | no | |
|
||||
| wanderer | wanderer-search | `v1.36.0` | `v1.54.2` | `v1.54.2` | yes | no | |
|
||||
| wger | wger | `2.6` | `2.7` | `2.7` | yes | no | floating (line); no ladder digest to compare |
|
||||
| wishlist | wishlist | `v0.67.1` | `v0.67.1` | `v0.67.1` | no | no | |
|
||||
| zipline | zipline | `4.6.1` | `4.8.0` | `4.8.0` | yes | no | |
|
||||
| zipline | zipline-postgres | `16-alpine` | `16-alpine` | `18-alpine` | no | yes | floating (line); no ladder digest to compare |
|
||||
@@ -0,0 +1,55 @@
|
||||
| app | entries | proven | of which backfilled | needs_person | files_may_change | engine conversion | head = compose | box one step behind, tonight | catalog_since |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| actualbudget | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| adventurelog | 1 | 1 | 0 | 0 | 0 | 0 | yes | yes | 2026-09-27 |
|
||||
| audiobookshelf | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| bentopdf | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-12 |
|
||||
| bookstack | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| calcom | 1 | 1 | 0 | 0 | 0 | 1 | yes | yes | 2026-09-28 |
|
||||
| calibre-web | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-12 |
|
||||
| claper | 1 | 1 | 0 | 0 | 0 | 1 | yes | yes | 2026-09-28 |
|
||||
| code-server | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| crafty-controller | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-06-26 |
|
||||
| docmost | 2 | 2 | 1 | 0 | 0 | 1 | yes | yes | 2026-09-25 |
|
||||
| emby | 2 | 2 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| ghost | 2 | 2 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| gitea | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| glance | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-09-21 |
|
||||
| gokapi | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| grafana | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| gramps-web | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| home-assistant | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| homebox | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-19 |
|
||||
| homepage | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| immich | 2 | 2 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| jellyfin | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| kimai | 1 | 1 | 0 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| komga | 1 | 1 | 0 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| mealie | 2 | 2 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-25 |
|
||||
| n8n | 3 | 3 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-25 |
|
||||
| navidrome | 2 | 2 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| nextcloud | 2 | 2 | 1 | 0 | 1 | 0 | yes | conditional: files_may_change (needs a fresh whole copy) | 2026-09-23 |
|
||||
| onlyoffice | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| opengist | 1 | 1 | 0 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| outline | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| paperless-ngx | 1 | 1 | 0 | 0 | 0 | 1 | yes | yes | 2026-09-27 |
|
||||
| papra | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| plant-it | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-21 |
|
||||
| plex | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-02-15 |
|
||||
| privatebin | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| radarr | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| rallly | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| recipe-importer | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-12 |
|
||||
| romm | 3 | 3 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| seerr | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| sonarr | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| sparkyfitness | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| tandoor | 2 | 2 | 1 | 0 | 0 | 1 | yes | yes | 2026-09-27 |
|
||||
| termix | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| uptime-kuma | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-09-21 |
|
||||
| vaultwarden | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
| vikunja | 1 | 1 | 1 | 0 | 0 | 0 | yes | yes | 2026-09-22 |
|
||||
| wanderer | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-21 |
|
||||
| wger | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-19 |
|
||||
| wishlist | 1 | 1 | 0 | 0 | 0 | 0 | yes | yes | 2026-09-23 |
|
||||
| zipline | 0 | 0 | 0 | 0 | 0 | 0 | — | no ladder: never (no_test_record) | 2026-07-18 |
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user