Night 2026-10-28 close-out: report, STATUS morning list, CONTEXT (CC decisions), R-1256/R-1238 states, R-1339 filed, Plex box walk evidence, catalog morning patches (prepared, not pushed)
gates / gates (push) Successful in 1m50s
gates / gates (push) Successful in 1m50s
This commit is contained in:
admin committed
1 parent
7a6b9ff19d
commit
7799ef5f54
9 files changed
+450
-6
No files matched your search
+14
@@ -16,6 +16,20 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-28 night (unattended) — restore edges, escrow edges, the host OOM count on main; NOTHING released.**
|
||||
> **Decided by CC unattended — operator may reverse:** (1) an unused restore copy goes after 14 days, swept every 6 h
|
||||
> (`07` §6.3 [DESIGN] 2026-10-28; options 14 d / for ever / 3 d); (2) R-1256's H4 and H5's failed-stop branch stay on main
|
||||
> with test proofs only — the brief wants a live proof or a revert; H4 runs only on an off-site restore (9202 has no
|
||||
> target) and both branches fail safe (hold or keep, never pour); the attended live proof or a revert of those two branches
|
||||
> of controller `342a394` is owed; (3) the catalog changes (R-1338 measurement, Plex fixtures, the Plex ladder entry) are a
|
||||
> prepared patch, `audits/night-2026-10-28/catalog-morning/`, not a push. Controller `342a394` (R-1256), `e33cffa`
|
||||
> (R-1254 controller half), `eed23e7` (R-528), `20c5877` (unknown app → 404); agent `5ddd124` (R-1254), `6f933d6` (R-528,
|
||||
> `GET /guest/oom-kills`); hub `1ee6d5a5` (R-1255). Live: H5 busy volume + H8 sweep on 9202; R-528 on Tester 1 (one
|
||||
> `app_oom` „seen by docker,cgroup"); R-1238 Plex: the unclaimed route works from Plex's own container (its front door answers 401 to the
|
||||
> proxy), bench + box 9202 proven, the ladder entry in the prepared patch (`catalog_since` kept at 2026-10-23: the pin did
|
||||
> not move, the writer had set today). Claims walked: fail.customer-self-restore (route proven, stays partial:
|
||||
> no non-operator yet), fault.selfheal (built → walked). R-1339 filed (a false „not mounted" log line).
|
||||
|
||||
> **2026-10-28 (evening, attended) — Gitea 28.2.0; Wanderer's search memory measured, not changed.** Gitea 1.27.3 → 28.2.0
|
||||
> live `0baf8f9` (bench + box + undo; 1.27.3 refuses a 28-migrated database, the undo copy restores it). D57 → decision
|
||||
> 280: imports only from github.com/codeberg.org/gitlab.com (`[migrations] EGRESS_MODE=strict` + `ALLOWED_HOST_LIST`, the
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
# REPORT — night 2026-10-28 → 29 (unattended)
|
||||
|
||||
## Safest readings taken (decided by CC unattended — operator may reverse)
|
||||
|
||||
1. **An unused restore copy is deleted after 14 days** (R-1256 H8; `07` §6.3 [DESIGN] 2026-10-28; CONTEXT). Options:
|
||||
14 days, for ever (as before), or 3 days. A held or waiting copy is never touched.
|
||||
2. **H4 and H5's failed-stop branch stay on main with test proofs only.** Rule 1 asks for a live proof by morning or a
|
||||
revert.
|
||||
- H4 runs only on an off-site restore, and 9202 has no off-site target. A fault injected there races the write-back.
|
||||
It can leave an operator-only hold (R-379), which is not ours to clear.
|
||||
- H5's failed stop needs a `docker stop` that fails on demand.
|
||||
- Both fixes fail safe: they hold or keep, and never pour.
|
||||
- Reading: keep them, and say so first. The live proof is owed to an attended session. Or the operator reverts those
|
||||
two branches of controller `342a394`.
|
||||
3. **The catalog changes are two prepared patches, not a push** (the brief: drill only, no live catalog push).
|
||||
`documentation/audits/night-2026-10-28/catalog-morning/`. The catalog working tree is clean again.
|
||||
4. **Plex's `catalog_since` stays 2026-10-23** in the prepared patch. The ladder writer set it to the day it ran. But the
|
||||
date means "the pin last moved", and the pin did not move tonight (it moved on 10-23, the one `--no-verify` push).
|
||||
Put back by hand, and written in the patch README.
|
||||
5. **STATUS was at its 50-line limit.** Yesterday's one-item do-list (the homelab list, R-1328) became item 5 of the
|
||||
morning list. Its words did not change.
|
||||
|
||||
## Five lines
|
||||
|
||||
1. Restore edges: 4 of 4 fixed (H4, H5, H6, H8), each test red first. Proven live: 2 (H5's busy volume, H8's sweep).
|
||||
2. Escrow edges: R-1255 (hub: the carry, the set-aside name, the purge) and R-1254 (the ceremony half, agent +
|
||||
controller) fixed on main, tests red first. No live proof: that needs a hub deploy, and no hub deploy runs at night.
|
||||
3. OOM proven: **yes** — on Tester 1, one `app_oom` "seen by docker,cgroup" and one mail.
|
||||
4. Plex route: **works** — unclaimed, from Plex's own container. Proven on the bench and on box 9202 (1.41.4 → 1.43.5,
|
||||
the section read back with the same uuid).
|
||||
5. Claims walked: 2 walked, 2 proven, 0 failed. Not walked: 31 of 55 (was 32).
|
||||
|
||||
## Morning release list (each one is yours; nothing was released)
|
||||
|
||||
1. **Hub vX:** R-1255 (felhom.eu `1ee6d5a5`).
|
||||
2. **Agent vY + your vouch:** R-1254 (`5ddd124`) and R-528 (`6f933d6`). Then `pinned.day0_agent` (D50).
|
||||
3. **Controller vZ:** R-1256 (`342a394`), R-1254's controller half (`e33cffa`), R-528 (`eed23e7`), and an unknown app
|
||||
answers 404 (`20c5877`). MinAgent unchanged (0.158.1): an older agent answers 404, and the controller asks again
|
||||
after 10 minutes.
|
||||
4. **Catalog:** the two prepared patches. R-1338 (the memory figure; Wanderer re-measured at 27.7 %, not tight), and
|
||||
the Plex fixtures + the Plex ladder entry (R-1238, proven on both venues).
|
||||
5. Owed to an attended session: the live proof of H4 and H5's failed stop, or a revert of those two branches.
|
||||
|
||||
If nothing is done: the fixes stay on main only. No box changes.
|
||||
|
||||
## [DESIGN] lines
|
||||
|
||||
- R-1256: `documentation/architecture/07-backup-architecture.md` §6.3, "[DESIGN] 2026-10-28 night — four restore
|
||||
edges", plus two [FACT] paragraphs (H5, H8 live).
|
||||
- R-1255: `03-host-agent.md`, "[DESIGN] 2026-10-28 night — WHICH retained row is carried".
|
||||
- R-1254: `03-host-agent.md`, "[DESIGN] 2026-10-28 night — a lost or late answer".
|
||||
- R-528: `08-alarm-ladder.md`, "[DESIGN] 2026-10-28 night — the host's counter beside Docker's flag", with its [FACT]
|
||||
of the Tester 1 proof.
|
||||
|
||||
## What was proven live
|
||||
|
||||
- **H5, scratch 9202, 01:33 UTC:** Vikunja's first volume was held by a created-only container. The household's
|
||||
restore was pressed through the page. The result was "Failed to remove volume vikunja_vikunja_data before the restore
|
||||
… (not written into)". The restore failed, and the put-back poured the live copy of both volumes. The app ran, with
|
||||
no hold. Projects from before and after the restore point read back.
|
||||
- **H8, scratch 9202, 03:29 UTC:** three copies were planted by hand. The first sweep removed the two older than 14
|
||||
days and kept the young one (8 ms). Then the young one was removed by hand.
|
||||
- **R-528, Tester 1, 2026-10-28 23:27 UTC:** a Radicale capped at 128 MB. The host counter went 0 → 1, with one
|
||||
`app_oom` "seen by docker,cgroup" and one operator mail.
|
||||
- **R-1238 Plex box walk, 9202, 03:45–03:59 UTC:**
|
||||
- Deployed through the setup gate at 1.41.4.
|
||||
- A library section was seeded from Plex's own container. Its front door answers 401 to the proxy.
|
||||
- The household's Update ran in 41 s.
|
||||
- The section read back on 1.43.5 with the same uuid.
|
||||
- The 10-minute soak peaked at 10.6 %, with no kill. The verdict was `proven`.
|
||||
|
||||
## Claims walked (household first; front doors only)
|
||||
|
||||
| Claim | Result | Evidence |
|
||||
|---|---|---|
|
||||
| fail.customer-self-restore | **PROVEN-LIVE** (the route). Stays `partial`: no one other than the operator has done it yet. | `audits/night-2026-10-28/part5-claims/selfrestore-walk.log` |
|
||||
| fault.selfheal | **PROVEN-LIVE**, built → walked. A crashed redis ran again in 1 s, and the probe read the app healthy. A guest reboot brought everything back in under 2 minutes. No event, no mail. | `…/part5-claims/selfheal/WALK.md` |
|
||||
|
||||
Not walkable tonight, with the reason for each:
|
||||
|
||||
- backup.sikeres (the run half): needs an off-site target, and 9202 has none.
|
||||
- backup.fill-warning: 9202's drive is a shared 938 GB host disk, which cannot be filled safely.
|
||||
- The app-crash mail and fault.operator-email: 9202 has no hub, and a hub deploy is not a night's work.
|
||||
- backup.restore-proof: needs every tier on both machines. Its block is R-366. CC's hub token reads no event list.
|
||||
|
||||
## Commits and CI (by commit)
|
||||
|
||||
| Repo | Commit | CI run id / # | Result |
|
||||
|---|---|---|---|
|
||||
| felhom.eu (hub) | `1ee6d5a5` R-1255 | 2470 / #1331 | green |
|
||||
| felhom.eu | `8a3df79f` design notes | 2476 / #1332 | green |
|
||||
| felhom.eu | `d8752de9` R-1338 evidence | 2477 / #1333 | green |
|
||||
| felhom.eu | `0abed37f` + `6a6dd94d` H5, the self-restore walk, row states | 2478 / #1334 | green |
|
||||
| felhom.eu | `7a6b9ff1` 07 [DESIGN] + facts, fault.selfheal walked | 2482 / #1337 | green |
|
||||
| felhom.eu | close-out (this report, STATUS, CONTEXT, rows, Plex box evidence, patches) | see the end of the final report | — |
|
||||
| controller | `342a394` R-1256 | 2469 / #532 | green |
|
||||
| controller | `e33cffa` R-1254 | 2472 / #533 | green |
|
||||
| controller | `eed23e7` R-528 | 2474 / #534 | green |
|
||||
| controller | `20c5877` unknown app → 404 | 2475 / #535 | green |
|
||||
| controller | `a815b37` docs | 2483 / #536 | green |
|
||||
| agent | `5ddd124` R-1254 | 2471 / #283 | green |
|
||||
| agent | `6f933d6` R-528 | 2473 / #284 | green |
|
||||
| agent | `91bd96c` docs | 2479 / #285 | green |
|
||||
| catalog | none pushed (2 prepared patches) | — | — |
|
||||
| drill | `41b37ad`, `37736f0`, then their reverts `8aaf198` + `3b66d35` (the tree equals its start) | — | — |
|
||||
|
||||
## Teardown (three layers)
|
||||
|
||||
- **Machine — scratch 9202 (04:14 UTC):**
|
||||
- The controller is back on 0.346.1 and healthy. The test image `0.347.0-rc.n1028` is removed (a local tag, never
|
||||
pushed).
|
||||
- The live catalog is back; the config is byte-identical to the saved copy.
|
||||
- The drill mirror is back at its start (`320cff5d`).
|
||||
- No throwaway app, no planted restore copy, no volume made tonight. The containers are as found: the controller,
|
||||
the file manager, paperless (3) and traefik.
|
||||
- **Machine — Tester 1 (23:30 UTC):** controller 0.346.1, agent 0.167.0 (sha256 `28bdd282…`), the test app removed.
|
||||
- **Host — demo-hp:** the bench 9401 is stopped. No new guest, and nothing on the thin pool.
|
||||
- **Hub:** nothing provisioned.
|
||||
|
||||
## Register
|
||||
|
||||
Before: 225. After: 226. Opened: 1 (R-1339). Closed: 0.
|
||||
|
||||
## Observations
|
||||
|
||||
- FILED: R-1339 — at boot the controller logs "drive …/userdata/<app> not mounted", but the drive IS mounted. No data
|
||||
is lost; the log line misleads a diagnosis.
|
||||
- NOT-A-FINDING: the Plex bench soak got 401s. An unclaimed Plex refuses the harness's request. There were 0 kills, and
|
||||
the peak was 12 % of 2 GiB.
|
||||
- NOT-A-FINDING: Plex's front door answers 401 to the box walk. Plex does not take the proxy's address as local. The
|
||||
fixture asks from Plex's own container, which is the measured route (R-1238).
|
||||
- NOT-A-FINDING: 9202 drops `app_start_failed` and `controller_started`, because it has no hub. This is by design on a
|
||||
scratch box.
|
||||
- NOT-A-FINDING: the ladder writer sets `catalog_since` to the day it runs. It is made for a pin move, and this was the
|
||||
one entry written after a move made by hand (R-1238's history). Put back in the patch.
|
||||
- NOT-A-FINDING: the security review flagged the drill's Plex downgrade. It was on purpose: drill only, scratch 9202,
|
||||
and reverted.
|
||||
- NOT-A-FINDING: the 03:45 push was refused once. The nightly security watch pushed its own state during the gate run.
|
||||
A rebase, then the push went in.
|
||||
- NOT-A-FINDING: the security watch's daily list has a new Brotli decoder hole in many images. That is its normal job,
|
||||
with its own mail. A catalog image move would be a live catalog push, which the night fences.
|
||||
- NOT-A-FINDING: 9202 holds 359 Docker volumes, all made before 2026-10-28 16:00. None is from tonight. This is scratch
|
||||
hygiene from earlier sessions, not a product defect.
|
||||
- Fixed (small): two dead bench Wanderer tokens were redacted from committed evidence. They stay in the history of
|
||||
`d8752de9`; the throwaway app is gone.
|
||||
- Fixed (small): a stale R-356 sentence in the self-restore claim's note.
|
||||
|
||||
## Other
|
||||
|
||||
- Instruction files: none edited.
|
||||
- System poster: no listed fact changed. The restore copy is not on the poster, and nothing was released.
|
||||
- Architecture read: `07-backup-architecture.md` (restore), `03-host-agent.md` (escrow), `08-alarm-ladder.md` (OOM),
|
||||
`09-update-architecture.md` §6.4 (the ladder).
|
||||
- Secrets: the night's passwords stayed in the scratchpad. Every evidence folder was scanned; the only hits are setting
|
||||
names (`cf_tunnel_token`, `PLEX_CLAIM`, "password from settings.json"), never values.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-10-28 (evening).** On your yes, the four big updates are live with their undo tested: Wanderer v0.21.2 (offered again, D59 ended; federation from other servers stays off), Termix 26.10.5 (D58 ended), Homepage v2.7.0 and Seerr v3.5.1. Gitea 28.2.0 too (imports only from GitHub, Codeberg, GitLab); Docker Hub no longer refuses the watch in test runs — tonight's run is the check.
|
||||
**Updated 2026-10-29 (night, unattended).** Nothing released; CC decided (you may reverse) that an unused restore copy is deleted after 14 days. On main, each test seen failing first: four restore fixes, two escrow fixes, a memory kill Docker misses now alarms (proven on Tester 1), and Plex's update test works on both test machines; two promises walked, both pass, register 225 → 226 (1 opened, 0 closed).
|
||||
|
||||
## What works
|
||||
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
|
||||
@@ -48,7 +48,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. (since 10-14)
|
||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). (since 10-12)
|
||||
|
||||
**2026-10-28 day.** 1. Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes.
|
||||
**2026-10-29 morning list.** 1. Release the hub (R-1255). 2. Release the agent, then your vouch (R-1254, R-528). 3. Release the controller (R-1256, R-528). 4. Push the two prepared catalog patches (R-1338, R-1238). If not: the fixes stay unreleased. 5. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes.
|
||||
**2026-10-23 do list.**
|
||||
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
|
||||
2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms.
|
||||
|
||||
+2
-2
@@ -1,8 +1,8 @@
|
||||
From be6a62917fbd694bb5ed4d6927492b3e97c4cf45 Mon Sep 17 00:00:00 2001
|
||||
From: cc <cc@local>
|
||||
Date: Thu, 29 Oct 2026 00:12:28 +0100
|
||||
Subject: [PATCH] PREPARED (not pushed): R-1338 memory figure + Plex fixtures +
|
||||
Wanderer entry re-measure
|
||||
Subject: [PATCH 1/2] PREPARED (not pushed): R-1338 memory figure + Plex
|
||||
fixtures + Wanderer entry re-measure
|
||||
|
||||
---
|
||||
scripts/box_walk.py | 32 +++++++---
|
||||
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
From 36e58cc3afbc478fa3191b407210f23a5f0ac966 Mon Sep 17 00:00:00 2001
|
||||
From: kisfenyo <nagyfenyvesi.viktor@gmail.com>
|
||||
Date: Thu, 29 Oct 2026 05:11:58 +0100
|
||||
Subject: [PATCH 2/2] PREPARED (not pushed): Plex test record 1.41.4.9463 ->
|
||||
1.43.5.11029, bench + box 9202 proven (felhom.eu R-1238); catalog_since stays
|
||||
2026-10-23 (the pin did not move)
|
||||
|
||||
---
|
||||
templates/plex/.felhom.yml | 6 ++++++
|
||||
1 file changed, 6 insertions(+)
|
||||
|
||||
diff --git a/templates/plex/.felhom.yml b/templates/plex/.felhom.yml
|
||||
index 7379349..876c96f 100644
|
||||
--- a/templates/plex/.felhom.yml
|
||||
+++ b/templates/plex/.felhom.yml
|
||||
@@ -237,3 +237,9 @@ i18n:
|
||||
# One line per image in docker-compose.yml; the grammar is in NEW-APP-CHECKLIST.md (1.10).
|
||||
upstream:
|
||||
plexinc/pms-docker: "gap:closed source; Plex announces fixes on its forum only"
|
||||
+
|
||||
+# update_ladder — the test record: one tested step per line, oldest first (JSON flow mappings,
|
||||
+# `09-update-architecture.md` §6.4 part 4). WRITTEN BY scripts/upgrade-test.py, never by hand;
|
||||
+# gated by scripts/check-test-record.py. An image: move without a proven entry here is refused.
|
||||
+update_ladder:
|
||||
+ - {"from": {"plex": "plexinc/pms-docker:1.41.4.9463-630c9f557"}, "to": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "digest": {"plex": "sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f"}, "verdict": "proven", "tested_at": "2026-10-28T21:48:51Z", "harness_version": 5, "evidence": "documentation/audits/night-2026-10-28/bench/plex/edge", "box_evidence": "documentation/audits/night-2026-10-28/r1238-plex/box-9202", "memory_peak_pct": 10.7, "marks": {"files_may_change": false, "needs_person": null, "memory_tight": false}, "memory_basis": "anon", "memory_cgroup_peak_pct": 12.3}
|
||||
--
|
||||
2.47.3
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
# Catalog morning patches (night 2026-10-28, prepared, NOT pushed)
|
||||
|
||||
The night brief allowed the catalog as a drill only, so these two commits were never pushed.
|
||||
They apply on catalog `main` at `0baf8f9`.
|
||||
|
||||
1. `0001` — R-1338: the memory watch judges the LIVE anon figure (`memfigure.py`, `memory.stat` minus reclaimable
|
||||
parts), Plex fixtures (bench + box), Wanderer's entry re-measured (27.7 %, `memory_tight` false).
|
||||
2. `0002` — R-1238: Plex's test record 1.41.4.9463 → 1.43.5.11029, proven on the bench and on scratch 9202
|
||||
(seed read back, 10-min soak 10.6 %, no kill). `catalog_since` stays 2026-10-23: the pin did not move.
|
||||
The ladder writer had set it to the day it ran; it was put back by hand, because the date means "the pin last
|
||||
moved", and the badge would read "0 days" for a tag that is six days old.
|
||||
|
||||
Apply: `cd app-catalog-felhom.eu && git am <this folder>/000*.patch`, then `python3 scripts/catalog_gates.py`, then push.
|
||||
If `main` moved past `0baf8f9`, apply the same way; a conflict in `templates/wanderer/.felhom.yml` means the Wanderer
|
||||
entry changed again — re-run the writer rather than merge by hand.
|
||||
@@ -0,0 +1,227 @@
|
||||
{
|
||||
"app": "plex",
|
||||
"walk": "security",
|
||||
"box_walk_version": 1,
|
||||
"verdict": "proven",
|
||||
"venue": "box 9202 (drill catalog), the guarded Update",
|
||||
"from": {
|
||||
"plex": "plexinc/pms-docker:1.41.4.9463-630c9f557"
|
||||
},
|
||||
"to": {
|
||||
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
},
|
||||
"observables_before": {
|
||||
"pinned_images": {
|
||||
"plex": "plexinc/pms-docker:1.41.4.9463-630c9f557"
|
||||
},
|
||||
"installed_images": {
|
||||
"plex": "plexinc/pms-docker:1.41.4.9463-630c9f557"
|
||||
},
|
||||
"catalog_images": null,
|
||||
"live_compose_image_lines": [
|
||||
"image: plexinc/pms-docker:1.41.4.9463-630c9f557"
|
||||
],
|
||||
"docker_inspect": [
|
||||
"plex plexinc/pms-docker:1.41.4.9463-630c9f557 running=true restarts=0"
|
||||
]
|
||||
},
|
||||
"setup_gate_before": {
|
||||
"browser": {
|
||||
"status": "302",
|
||||
"redirect": "https://felhom.enkisfelhom.hu/__gate/start?…"
|
||||
},
|
||||
"script": {
|
||||
"status": "401",
|
||||
"redirect": null
|
||||
}
|
||||
},
|
||||
"seed": "read_back",
|
||||
"drill_commit": "37736f07d9ae",
|
||||
"sync_waited_s": 4.5,
|
||||
"badge_before": {
|
||||
"hu": [
|
||||
{
|
||||
"title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.",
|
||||
"text": "Frissítés elérhető — ma"
|
||||
}
|
||||
],
|
||||
"en": [
|
||||
{
|
||||
"title": "A newer version of this app is available. Select the Update button to start it.",
|
||||
"text": "Update available — today"
|
||||
}
|
||||
]
|
||||
},
|
||||
"measured_at": "2026-10-29T03:47:17Z",
|
||||
"update": {
|
||||
"accepted": true,
|
||||
"http": "202",
|
||||
"phases": [
|
||||
{
|
||||
"t": 0.0,
|
||||
"phase": "backing-up",
|
||||
"label": "Biztonsági mentés készül a frissítés előtt…",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 11.3,
|
||||
"phase": "safety-dump",
|
||||
"label": "Adatbázis pillanatkép…",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 12.3,
|
||||
"phase": "pulling",
|
||||
"label": "Új verzió letöltése…",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 21.5,
|
||||
"phase": "copying",
|
||||
"label": "Az adatok másolása a frissítés előtt…",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 35.9,
|
||||
"phase": "verifying",
|
||||
"label": "Működés ellenőrzése…",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 41.0,
|
||||
"phase": "done",
|
||||
"label": "Frissítve",
|
||||
"updating": false,
|
||||
"error": null,
|
||||
"hold": null
|
||||
}
|
||||
],
|
||||
"duration_s": 41.0,
|
||||
"final_phase": "done",
|
||||
"update_error": null,
|
||||
"hold_reason": null,
|
||||
"state": "running"
|
||||
},
|
||||
"observables_after": {
|
||||
"pinned_images": {
|
||||
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
},
|
||||
"installed_images": {
|
||||
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
},
|
||||
"catalog_images": {
|
||||
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
},
|
||||
"live_compose_image_lines": [
|
||||
"image: plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
],
|
||||
"docker_inspect": [
|
||||
"plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 running=true restarts=0"
|
||||
]
|
||||
},
|
||||
"badge_after": {
|
||||
"hu": [
|
||||
{
|
||||
"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.",
|
||||
"text": "Naprakész"
|
||||
}
|
||||
],
|
||||
"en": [
|
||||
{
|
||||
"title": "This app is running the newest version available.",
|
||||
"text": "Up to date"
|
||||
}
|
||||
]
|
||||
},
|
||||
"state_after": "running",
|
||||
"hold_after": null,
|
||||
"to_digests": {
|
||||
"plex": "sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f"
|
||||
},
|
||||
"four_observables": {
|
||||
"pinned": {
|
||||
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
},
|
||||
"installed": {
|
||||
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
},
|
||||
"catalog": {
|
||||
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
},
|
||||
"live_compose": [
|
||||
"plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
]
|
||||
},
|
||||
"seed_read_after": true,
|
||||
"setup_gate_after": {
|
||||
"browser": {
|
||||
"status": "302",
|
||||
"redirect": "https://felhom.enkisfelhom.hu/__gate/start?…"
|
||||
},
|
||||
"script": {
|
||||
"status": "401",
|
||||
"redirect": null
|
||||
}
|
||||
},
|
||||
"memory": {
|
||||
"soak_s": 600,
|
||||
"interval_s": 10,
|
||||
"samples": 60,
|
||||
"containers": {
|
||||
"plex": {
|
||||
"samples": 60,
|
||||
"anon_peak_bytes": 230629376,
|
||||
"limit_bytes": 2147483648,
|
||||
"anon_peak_pct": 10.7,
|
||||
"anon_live_peak_bytes": 228552704,
|
||||
"anon_live_peak_pct": 10.6,
|
||||
"cgroup_peak_pct": 12.3,
|
||||
"oom_kill_delta": 0,
|
||||
"oom_killed_flag": false,
|
||||
"restarts_delta": 0,
|
||||
"running_all_samples": true,
|
||||
"cgroup": "/sys/fs/cgroup/system.slice/docker-ef9b5b1114ba34f44c59d053b2651e68aff8e2e3070f8ac70397c93f7ea84eab.scope"
|
||||
}
|
||||
},
|
||||
"oom_kills": 0,
|
||||
"restarts": 0,
|
||||
"peak_pct": 10.6,
|
||||
"peak_basis": "anon_live"
|
||||
},
|
||||
"why": "the security-fix step ran on the box through the guarded Update; healthy, TO on all four observables, no undo, no hold, a clean soak, seed read back",
|
||||
"log": [
|
||||
"04:45:38 [5] the box's drill mirror now at 41b37adc",
|
||||
"04:45:42 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['PLEX_CLAIM', 'HDD_PATH']",
|
||||
"04:45:43 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}",
|
||||
"04:46:13 [1] deployed, controller state=running, pinned={'plex': 'plexinc/pms-docker:1.41.4.9463-630c9f557'}",
|
||||
"04:46:13 gate: plex is gated — passed as the household (cookie set)",
|
||||
"04:46:17 [2] setup gate as installed: {'browser': {'status': '302', 'redirect': 'https://felhom.enkisfelhom.hu/__gate/start?…'}, 'script': {'status': '401', 'redirect': None}}",
|
||||
"04:46:53 plex: POST /library/sections through the front door http=401",
|
||||
"04:46:53 plex: the front door was not taken as local (<html><head><title>Unauthorized</title></head><body><h1>401 Unauthorized</h1></body></html>) — asking from Plex's own container",
|
||||
"04:46:56 plex: POST /library/sections from its own container http=201",
|
||||
"04:46:56 plex: seeded library section felhom834284c6 uuid=fce5ae7f-2c56-4992-b730-058e892014ab (route own-container, server 1.41.4.9463-630c9f557)",
|
||||
"04:47:03 plex: section felhom834284c6 read back (server 1.41.4.9463-630c9f557, route own-container), same uuid=True, folders=['/media']",
|
||||
"04:47:04 [5] drill commit 37736f07d9ae: plex plexinc/pms-docker:1.41.4.9463-630c9f557 -> plexinc/pms-docker:1.43.5.11029-6bf8af3f0 (push rc=0)",
|
||||
"04:47:12 [5] the box's drill mirror now at 37736f07",
|
||||
"04:47:17 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}",
|
||||
"04:47:17 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None",
|
||||
"04:47:28 + 11.3s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None",
|
||||
"04:47:29 + 12.3s phase=pulling label=Új verzió letöltése… err=None hold=None",
|
||||
"04:47:39 + 21.5s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None",
|
||||
"04:47:53 + 35.9s phase=verifying label=Működés ellenőrzése… err=None hold=None",
|
||||
"04:47:58 + 41.0s phase=done label=Frissítve err=None hold=None",
|
||||
"04:48:23 plex: section felhom834284c6 read back (server 1.43.5.11029-6bf8af3f0, route own-container), same uuid=True, folders=['/media']",
|
||||
"04:48:23 [7] memory soak 600 s …",
|
||||
"04:58:30 [7] soak: peak 10.6% of the limit (anon), kills 0, restarts 0"
|
||||
]
|
||||
}
|
||||
@@ -153,6 +153,7 @@ stopping line that lies.
|
||||
| **R-1336** | Apps & catalog | P3 | **ActualBudget has no release with the HTTP2Shell fix (CVE-2026-111497): 26.9.0 carries Node.js 24.20.0 and the newest 26.10.0 still Node.js 24.21.0 (fixed: 24.22.1), read from both images.** `node app.js` serves the app; the setup gate covers it until the first account exists, then its own password. No box runs it. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Node ≥ 24.22.1 | the watch lists it nightly (R-1327); take that release as a lane step | CC |
|
||||
| **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC |
|
||||
| **R-1338** | Apps & catalog | P3 | **Wanderer's Meilisearch fills whatever memory limit it gets: 92–100 % anon of 256 MB (bench and 9202), still 88.8 % of 384 MB on the bench (2026-10-28)** — it sizes its indexing memory from the memory it sees (its default: 2/3 of the available RAM), so a higher limit alone does not end the `memory_tight` mark of Wanderer's step. No kill in any run. `audits/dockerhub-bigsteps-2026-10-28/wanderer/` | OPEN — **2026-10-28 night: MEASURED — it is the measurement, not Wanderer, and the cap is not needed** (`audits/night-2026-10-28/r1338-meili/`; `09` §3 decision 22 refined). Meilisearch frees with MADV_FREE: the given-back pages stay in `anon` but sit on the inactive FILE list (`file` ~1 MB, `inactive_file` ~290 MB) and the kernel takes them first — 83–96 % `anon` on the bench (no swap) AND on 9202, with and without the cap, 0 kills; it uses ~100 MB. The evening's 17.7 % was the container after the swap, not the cap. **Fixed in the tools:** the mark reads the live anon (catalog `scripts/memfigure.py`, the larger of two estimates, errs high; `upgrade-test.py`, `box_walk.py` soak; `test_memory_live.py` red-proved). Re-run: bench Wanderer v0.20.0 → v0.21.2 `proven`, **no marks** (search 83.3 % anon, 27.7 % live); 9202 95.2 % anon, 25.1 % live. **Prepared, NOT pushed** (the night's catalog fence): the tool change + Wanderer's entry without `memory_tight` (`audits/night-2026-10-28/catalog-morning/`; the catalog gates passed in a scratch clone). **Earlier, 2026-10-28 evening: `MEILI_MAX_INDEXING_MEMORY` = 128 MiB (a third of 384 MB) measured, NOT pushed** (`audits/gitea-wanderer-2026-10-28/wanderer-meili/`): on the bench, under the harness's load, the search container peaked at 17.7 % (84 MB; 88.8 % without the cap, same load); on 9202 the cap was in the container (read back) and it still held 95.7 % anon (363 MB steady over 6 min, no kill). The box result decides: not shown to help, so not pushed. Also: the catalog's ladder has no entry type for a re-measure of a definition change at the SAME digests (`ladder.py`: a re-test whose digest equals its digest_from tests nothing new), so the v0.21.2 entry's `memory_tight` mark cannot be replaced without an image move. | the catalog push (the morning list) | push the prepared catalog change (the live figure + Wanderer's entry re-measured, `memory_tight` off — an existing entry's memory fields edited at the SAME digests, which the gates accept); then close | CC |
|
||||
| **R-1339** | Storage & devices | P4 | **At every start the controller logs „FileBrowser: drive …/userdata/<app> not mounted — skipping userdata skeleton" for per-app folders on a drive that IS mounted** (scratch 9202, 2026-10-29 03:35 UTC: nine lines, `scratch_hdd` a live mountpoint, mp8). `skipFileBrowserPath` (`controller/internal/web/handlers.go:3766`) asks whether the folder ITSELF is a mountpoint, and a folder under a mounted drive never is. No data is lost: the file browser shows the folders through the drive's own mount (`/srv/scratch_hdd`). The cost is a false „not mounted" in the log of a box whose drive works, which misleads a diagnosis. Fix: ask about the drive the folder lives on, with a test that a sub-folder of a mounted drive is not skipped. Evidence: `audits/night-2026-10-28/part5-claims/selfheal/controller-after-reboot.log`. | OPEN | — | the fix above, with its red-first test | CC |
|
||||
|
||||
|
||||
## App updates — 10 rows (P2 1, P3 9)
|
||||
@@ -164,7 +165,7 @@ stopping line that lies.
|
||||
| **R-1319** | App updates | P3 | **Paperless-ngx 2.20.15: GHSA-2jhj (low; needs a setting we do not set — not reachable); the fix is 3.3.4, a big step** (encryption and API v1 removed, the search index rebuilt). It runs on all four test boxes. Not urgent. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step 3.3.4 when the catalog takes Paperless 3 for other reasons; read our API v1 use first | CC |
|
||||
| **R-1321** | App updates | P3 | **Tandoor 2.6.15: GHSA-5f74 (high, an XXE file read) and GHSA-69q3 (any user); the fix is 2.7.2** — 2.7.0 removes settings and an endpoint and rewrites the login pages (our probe path — not sure), and 2.7.0/2.7.1 break invite links with sign-up off, so straight to 2.7.2. No box runs it. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | a normal step to 2.7.2 with the probe path checked; bench + 9202 (+ undo if a migration); a STOP | CC |
|
||||
| **R-1324** | App updates | P3 | **RomM 5.4's first start can run a 15–20 minute migration on a big library (upstream notes) — longer than the update's check window, so a household's night step could be undone for slowness, not for a fault.** Today's walk (9202, a small seed) took the step in seconds; demo-hp's RomM takes it tonight (security-fix step, catalog `06e603e`). `audits/security-app-updates-2026-10-27/triage-notes/romm.md` | OPEN — filed 2026-10-27; WATCH demo-hp's night leg line | — | read demo-hp's night leg line for romm (done / undone, the time); if undone for time: give the step a longer check or the `needs_person` mark | CC |
|
||||
| **R-1238** | App updates | P2 | **Plex has no test record and no upgrade fixture, so the night leg never updates Plex on a household box.** Filed 2026-10-23. Catalog `6cc317d` moved Plex 1.41.4.9463 → 1.43.5.11029-6bf8af3f0 (the Plex2Shell fix) on the operator's ruling, with NO `update_ladder` entry (the move gate refused; pushed `--no-verify`, the one time). With no ladder, a person's Update press pins the current compose (`stacks/ladder.go` `nextLadderStep`), but the night leg skips the app (`unattended.go` → `LegSkipNoTestRecord`), so every future Plex fix reaches a household only by a hand press. `scripts/upgrade_fixtures.py` has no Plex fixture. A likely seed route is NOT measured: an UNCLAIMED Plex answers a private-address request with no token (`architecture/13-threat-model.md`, the R-1045 row), so the bench may create and read back a library section through Plex's own API without a plex.tv account. **Needs a spike:** prove that route on the bench (seed a section, swap the image, read it back), add the fixture, then `upgrade-test.py --write-ladder` for the current pin — the box walk on 9202 needs the same route behind the setup gate. Evidence: `audits/plex-demo-boxes-2026-10-23/`. **2026-10-23 (later): the NEXT Plex fix need not wait for this** — the security-fix lane (`09` §3 decision 255, D39 = B, controller 0.341.0) takes a box-only step with `seed: "none"` at night (proven on 9202 with Plex 1.43.4 → 1.43.5, `audits/security-lane-2026-10-23/`). The bench fixture stays the aim: it is the only check that reads Plex's data back after a step. | **OPEN** | a claimed-Plex seed route on the bench | the spike above | CC |
|
||||
| **R-1238** | App updates | P2 | **Plex has no test record and no upgrade fixture, so the night leg never updates Plex on a household box.** Filed 2026-10-23. Catalog `6cc317d` moved Plex 1.41.4.9463 → 1.43.5.11029-6bf8af3f0 (the Plex2Shell fix) on the operator's ruling, with NO `update_ladder` entry (the move gate refused; pushed `--no-verify`, the one time). With no ladder, a person's Update press pins the current compose (`stacks/ladder.go` `nextLadderStep`), but the night leg skips the app (`unattended.go` → `LegSkipNoTestRecord`), so every future Plex fix reaches a household only by a hand press. `scripts/upgrade_fixtures.py` has no Plex fixture. A likely seed route is NOT measured: an UNCLAIMED Plex answers a private-address request with no token (`architecture/13-threat-model.md`, the R-1045 row), so the bench may create and read back a library section through Plex's own API without a plex.tv account. **Needs a spike:** prove that route on the bench (seed a section, swap the image, read it back), add the fixture, then `upgrade-test.py --write-ladder` for the current pin — the box walk on 9202 needs the same route behind the setup gate. Evidence: `audits/plex-demo-boxes-2026-10-23/`. **2026-10-23 (later): the NEXT Plex fix need not wait for this** — the security-fix lane (`09` §3 decision 255, D39 = B, controller 0.341.0) takes a box-only step with `seed: "none"` at night (proven on 9202 with Plex 1.43.4 → 1.43.5, `audits/security-lane-2026-10-23/`). The bench fixture stays the aim: it is the only check that reads Plex's data back after a step. **2026-10-28 night: the route works, unclaimed.** Plex refuses the request through its front door (401: the proxy's address is not taken as local), so the fixture asks from Plex's own container (`POST /library/sections` → 201) and reads the section back by its uuid. Bench 1.41.4.9463 → 1.43.5.11029 proven (`audits/night-2026-10-28/bench/plex/edge/`); box walk on scratch 9202 through the setup gate proven 2026-10-29 (the household's Update, 41 s; the section read back on 1.43.5 with the same uuid; 10-min soak 10.6 %, no kill; `…/r1238-plex/box-9202/`). The fixtures and the ladder entry are a PREPARED patch (`…/catalog-morning/`), not pushed — the night allowed no live catalog push. | OPEN — proven on both venues, not pushed | the morning catalog push | push the two prepared catalog commits (the operator's word or an attended session) | CC |
|
||||
| **R-440** | App updates | P3 | **[P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible.** `compose pull` on a moving tag fetches whatever upstream published that day. **MEASURED 2026-09-01 over `app-catalog-felhom.eu` @ `29edad9c5bf4`: 79 `image:` lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version.** `postgres:16-alpine` (8 apps), `redis:7-alpine` (6), `mariadb:11.6` (2), plus one each of `postgres:15-alpine`, `postgis/postgis:16-3.5-alpine`, `mariadb:11.4`, `mariadb:12.3`, `ghcr.io/claperco/claper:2.5`, `ghcr.io/thomiceli/opengist:1.13`, `wger/server:2.6`. **A 24th is arguable and is recorded rather than rounded away:** `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. **Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists**, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. **MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control.** Running digests on demo-hp compared against what the registry serves for the same tag today: **`mariadb:11.4` MOVED** (`sha256:4f1d8d20...` -> `sha256:611a2fcc...`) and **`mariadb:12.3` MOVED** (`sha256:a02fe89c...` -> `sha256:dd9b303a...`), while `postgres:16-alpine`, `redis:7-alpine`, `mariadb:11.6` and `opengist:1.13` were SAME — **and both fully-pinned CONTROLS (`rommapp/romm:5.0.0`, `privatebin/pdo:2.0.5`) were SAME.** So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under `romm` and `bookstack`, with no catalog change and no record. **Compounding fact found while reading:** the recovery unit records `ImagePins` but the manifest comment says *"image NOT stored - re-pulled on restore"*, so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. **HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): `app.yaml.installed_images` now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade.** What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). **The row therefore stays OPEN and its rank is unchanged** — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. `audits/SPIKE-app-update-2026-09-01.md` **— NIGHT 2026-09-23:** every image moved tonight (and the 21 backfilled moves) carries its resolved digest in the catalog's test record; the floating pins themselves still float until the box pulls by digest (`09` §6.4 part 6). **-- MEASURED 2026-09-30 on demo-hp 9201: a fresh install or a guarded Update renders `name:tag@sha256` from the ladder (`digest.go` `RenderWithLadderDigests`) — 9 of 20 services there run such a definition (adventurelog, docmost, paperless-ngx). The other 11 run TAG-ONLY definitions: bookstack, kimai, opengist, privatebin, romm were installed before v0.269.0 and not updated since, and `CarryDigests` (`digest.go:141`) keeps only a digest the running definition already has; bentopdf and calibre-web have no ladder at all. So a re-pull of those (a restore) takes whatever the tag serves that day. `audits/pg-last-six-2026-09-30/E/E2-digests-demo-hp.txt`.** **-- 2026-09-30 (evening):** calibre-web now has a ladder entry (`53a4a1d`), so a new install or a guarded Update renders its tested digest; bentopdf still has none. `audits/more-night-apps-2026-09-30/` **Merged 2026-10-05 from R-446 (duplicate):** the household-visible consequence — for these apps the „Naprakész" badge can never turn „behind" (`felhom-controller controller/internal/stacks/updateorder.go:96` returns false with no catalog digests / no test date); R-740 (same-tag re-test) is closed, so floating-tag drift for laddered apps is handled. | **NARROWED 2026-09-30 — floating only for (1) apps with no ladder entry and (2) apps installed before v0.269.0 until their next guarded Update; owner: CC.** **Re-ranked 2026-10-03: P2→P3: narrowed to apps with no ladder entry and pre-v0.269 installs; each guarded update fixes the latter.** | — | — | CC |
|
||||
| **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s,Line truncated
|
||||
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
|
||||
@@ -186,7 +187,7 @@ stopping line that lies.
|
||||
| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. | — | — | CC |
|
||||
| **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) |
|
||||
| **R-1255** | Backup & restore | P3 | **Escrow edge cases (review round 2, HUNT-3 H3/H4):** the R-1176 carry moves the newest kept package even when it holds no repository password (`hub/internal/store/escrow_carry.go:100-110` checks `identity_blob` only), while the right one stays under the deleted host (`store.go:3533-3534`); two set-asides on the same UTC day reuse a name and the purge can delete a still-stored copy's package (`hub/internal/offsitekeys/offsitekeys.go:378-385`; `store.go:4295-4300`). Static. | OPEN — **fixed on main 2026-10-28 night, not released** (felhom.eu `1ee6d5a5`, CI 2472; store tests red first; `03` §8 [DESIGN] 2026-10-28): the carry takes the newest retained row with an identity AND a repository-password hash (the newest row only when none has a hash — a legacy package, as before); a set-aside is `<repo>.orphaned-<YYYYMMDDTHHMMSSZ>`; a retained row is purged only when it was tagged no later than its path's deletion (`repo_tagged_at`, new column). No live proof: a hub deploy is not a night's. | the hub release (attended) | release it (the morning list), read the store back after one carry and one abandonment, then close | CC |
|
||||
| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. | OPEN | — | one attended pass over the restore paths with these four as tests | CC |
|
||||
| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. **2026-10-28 night:** all four fixed, each test red first (controller `342a394`, CI 2469 green; `07` §6.3 [DESIGN] 2026-10-28 — an unused copy goes after 14 days, decided by CC unattended). Live on scratch 9202 2026-10-29: H5's busy volume (not written into, the live data put back, no hold; `audits/night-2026-10-28/r1256-restore/h5-live/`) and H8 (the 03:29 sweep removed the two old copies and kept the young one; `…/h8-live/`). H4 and H5's failed stop: tests only; H6: tests only (a live press cannot tell old from new). | OPEN — fixed on main, not released | the controller release (morning list) | the release; then an attended live proof of H4 and H5's failed stop, or revert those two branches of `342a394` | CC |
|
||||
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
|
||||
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
|
||||
| **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
|
||||
|
||||
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-10-29 (night)
|
||||
|
||||
**Updated 2026-10-28 (evening).** On your yes, the four big updates are live with their undo tested: Wanderer v0.21.2 (offered again, D59 ended; federation from other servers stays off), Termix 26.10.5 (D58 ended), Homepage v2.7.0 and Seerr v3.5.1. Gitea 28.2.0 too (imports only from GitHub, Codeberg, GitLab); Docker Hub no longer refuses the watch in test runs — tonight's run is the check.
|
||||
|
||||
## Moved from STATUS on 2026-10-28 (evening)
|
||||
|
||||
**Updated 2026-10-28 (day).** On your yes, tested security steps are live for 10 apps (Home Assistant, Immich, AdventureLog, Ghost, MeTube, Zipline, Karakeep, Tandoor, Papra, Dawarich); Wanderer is hidden. The nightly watch now reads the runtime inside every image in CI by itself. Tonight demo-hp takes the Home Assistant and AdventureLog steps; no box runs the other eight apps.
|
||||
|
||||
Reference in new issue
Block a user