Read-back done, releases delivered (hub 0.141.0, agent 0.150.0, controller 0.302.0, catalog 872039d), R-892 proven on the Tester 1 box; 7 rows closed (137 -> 130)
gates / gates (push) Successful in 2m47s

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
2026-10-07 09:30:55 +02:00
parent 9828acc646
commit 5ba1702fcf
22 changed files with 350 additions and 87 deletions
+6
View File
@@ -16,6 +16,12 @@
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
> **2026-10-07 (morning) — read-back B + D, the releases, the Tester 1 proof (`09` §3 160–161).** R-518 read back (night
> stop 91 s on demo-hp, one press 80 s; the page text holds; the two-tier night ~2026-10-08 is left). Released + delivered
> to demo-hp, demo-felhom, Tester 1: hub 0.141.0, agent 0.150.0 (+ bundle, probe 68/68), controller 0.302.0 (floors with
> declared MinAgent 0.131.0; golden 0.301.0, waiver to 2026-10-13), catalog `872039d`. R-892 proven on the Tester 1 box
> (vaultwarden 14.4 s). Rule 11 (helpers get the full fences). Register 137 → 130.
> **2026-10-06 (night) — the second burn-down night (in progress; morning note `audits/night-burndown-2026-10-06/`).**
> **Decided by CC unattended — operator may reverse:** `09` §3 decision 159 (R-805: an empty bind after the persistence
> exercise is a note in the reasons, the verdict unchanged). Agent main carries R-894 (`74b5eae`, the newest backup per
+23 -71
View File
@@ -1,81 +1,33 @@
# REPORT — night read-back brief, the night-free parts (2026-10-06 evening; Parts B and D follow after 08:30 on 2026-10-07)
# REPORT — the read-back, the releases and the Tester 1 proof (2026-10-07 morning)
The brief said „start after 08:30 on 2026-10-07". The operator then said „start A, C, E, F now". Parts B (the night
read-back) and D (one press, outside 02:00–08:30) wait for tomorrow. **Nothing was delivered to a box tonight**, so the
night of 2026-10-06→07 runs controller v0.301.0 and agent v0.149.0, and Part B reads a clean result. The hub was
released (it is not a box).
Operator instruction 2026-10-07 08:40: Parts B and D of `TASK-night-readback-offsite-gap-tester1-2026-10-07.md`, then
release hub 0.141.0 → agent 0.150.0 → controller → the held catalog branch, deliver to demo-hp, demo-felhom and the
Tester 1 box, then the Tester 1 update test with the operator present; rule 11 in the shared rule file. Recorded as
`09` §3 decisions 160–161. **The task file is not on DooPlex** (searched the disk and git); Parts B and D were taken from
the evening report and the R-518 row.
| Part | Result |
|---|---|
| **A** — the rulings | **done** — `09` §3 decisions 157 (R-528 option A + the Docker-approval check) and 158 (R-892: VM 341 on the HP box); R-892's "no Proxmox host known" corrected |
| **B** — the night read-back (R-518) | **waits for 08:30 2026-10-07** |
| **C** — demo-hp's off-site copy | **diagnosed: the premise was wrong — no 10-day gap.** Last copy 2026-10-01 20:15Z (ep0's own listing, verify ok); every box within 7 days. Two real findings: an off-site tier read DUE after an agent restart while its storage was unreachable (R-894, filed), and the alarm's operator mail failed and was never retried (**fixed**, hub v0.140.0) |
| **D** — one press, measured | **waits for after 08:30 2026-10-07** |
| **E** — the Tester 1 box in the update test (R-892) | **route built and identity matched; the live proof is BLOCKED** — DooPlex's key is not authorized on VM 341, and the permission check refused fetching its vaulted password (the brief's rule: a refusal stops that item) |
| **F** — the Docker-approval memory-kill check (R-528) | **built**: hub v0.140.0 LIVE (the approval waits for a passing check); the agent's wrapper merged, unreleased (ships as v0.150.0 after Part B); proven by hand on demo-hp's guest |
| small check — wger's 100 % peak | file cache, not a kill (kill counter 0, restarts 0, anon 50.3 %); the memory hint stays |
| **B** — the night read-back (R-518) | demo-hp (9 apps) stop **~91 s** (was 5 min 47 s); demo-felhom (1 app) ~11 s; local tier only, off-site not due; two channels each. `audits/readback-2026-10-07/RESULT-B-D.md` |
| **D** — one press, measured | demo-hp: **80 s** from the press to the last app (per app 39–79 s); the copy finished 4 min later with the apps up; only the local tier ran; the page's „kb. 1–1,5 perc" holds — no text change |
| hub 0.141.0 | built, manifest `011a481a`, Synced/Healthy at HEAD, image 0.141.0, `starting` log, healthz + System 200 at 06:57:42Z (40 s after the sync) |
| agent 0.150.0 | released (sha `a23d1c90…`, bundle `88456b38…`, tag `v0.150.0`), vouched (golden 0.301.0 and MinAgent 0.131.0 unchanged); signed `agent_update` → all three on 0.150.0 by 07:07Z; signed `agent_config_update` → `BUNDLE DONE written=1 same=24 self-check=ok`, probe 68/68 on all three by 07:21Z |
| controller 0.302.0 | MinAgent 0.131.0; floors 0.302.0 with declared MinAgent for demo-hp, demo-felhom, tester-1 → all three `0.302.0 (healthy)` by 07:10Z. Global floor and Tester 2 not touched. Golden waiver issued to 2026-10-13 (the weekly bake's date) |
| catalog | `night-held-2026-10-06` merged (`872039d`); synced on all three boxes (read in their templates) |
| **Tester 1 proof (R-892)** | vaultwarden 1.36.0 → 1.37.4 through the guarded Update: **proven, 14.4 s**, seed read back before and after; removed with its volume; app list equal; pointer restored byte-identical; drill reset. `audits/night-burndown-2026-10-06/s3/box/` |
| rule 11 | „Every helper prompt carries the brief's fences in full" — in all five copies (one md5) |
| Rows before | Rows after | Opened | Closed |
|---|---|---|---|
| **137** | **138** | **1** (R-894) | **0** (so far) |
| **137** | **130** | **0** | **7** (R-892, R-894, R-542, R-777, R-612, R-805, R-806) |
## Part C — what happened on demo-hp, with times
**Fixed without a row:** none. **One slip:** the catalog merge was pushed in the same command as its unit-test run,
before reading the result (standing rule 1). Read right after: 156 tests, the single known error (`test_pg_conversion`
is a script, not a test module — the same on `main` before the merge); `catalog_gates.py --fast` was green before the push.
- **My last report was wrong**: it said „demo-hp had no successful off-site run in 10 days". I read a log view cut by
`tail -40` that started on 2026-10-04. The agent's full log and ep0's own listing both show a completed off-site copy
on **2026-10-01 20:15Z** (15.2 GB, verify `ok`). With the 7-day cadence the next one is due ~2026-10-08.
- ep0, read only (`C/C2-ep0-snapshot-listing.txt`): demo-hp 2026-10-01T20:15Z, demo-felhom 2026-10-06T04:21Z, tester-1
2026-10-04T19:57Z, Tester-2 2026-10-04T16:31Z — every box within 7 days.
- **2026-10-05 04:25Z (06:25 local), demo-hp:** the off-site storage answered *Can't connect to 10.77.0.1:8007*; the
agent had restarted at 02:57Z (04:57 local), 1.5 hours before; its per-tier backup record is in memory only
(`internal/backup/store.go`, R-348), so the due-check fell back to an EMPTY record and read the tier DUE although it
was not (`internal/localapi/server.go` `newestArchiveOn` → unknown → in-memory). The controller asked; vzdump failed
(*could not activate storage 'felhom-pbs'*). By design the controller stops the apps before it asks; whether it did
that night is **not known** (the controller's log was lost to a later restart). → **R-894**.
- **Who was told:** the hub recorded `whole_guest_backup_failed` (error) at 04:27Z; the household was not mailed (by
design: operator only); **the operator's mail FAILED** (Resend: *context deadline exceeded*) and was never retried.
It is the only failed operator mail of 692 since 2026-02-16 (`C/C4-hub-failed-mails.txt`).
- **Fixed** (hub v0.140.0): a failed operator mail is tried again after 1, 5 and 15 minutes; each try is a row; giving
up is an ERROR line. Red-proof `C/red-operator-mail-retry.txt` („the failed mail was never sent again (tries=1)").
- Not read: the household's backups page on demo-hp (no session tonight).
**Instruction-file edits:** `.claude/rules/unprompted-work.md` §3 item 11 added in all five copies (operator ruling,
decision 160).
## Part E — the Tester 1 box
- Identity (`E1-identity-match.txt`): VM 341 `night1004-tester1` on demo-hp, at 192.168.0.154 (found by its MAC);
its certificate `CN=felhom.enkicsifelhom.hu`; the agent's own hub report for `tester-1-d70be4` says `host.node=felhom`;
its guest at 192.168.0.101 answers `felhom.enkicsifelhom.hu` with the Felhom login (200) and demo-hp's domain 404.
- Built (catalog `d63ea35`): `box_walk.py` `TARGETS` (9202, 9201, tester-1 via `-J demo-hp`), `BOX_ADMIN_SEED_GUESTS`
adds `("tester-1", "9201")`; tests BoxWalkTargets (red-proved). `operations/nodes.md` has the box.
- **Blocked:** `ssh -J demo-hp root@192.168.0.154` → `Permission denied (publickey,password)`. The VM has no guest agent;
editing its disk needs a VM stop (a reboot — not allowed). I tried to read the hub's code for revealing the vaulted
console password; **the session's permission check refused it**, and I did not try another way. R-892 now asks the
operator.
## Part F — the memory-kill check
- Hub v0.140.0 (LIVE 17:34Z): `DockerStatus` needs, per ring-0 box, a passing `oom_check` with the set; failed, errored or
missing blocks. Red-proof `F/red-hub-docker-approval.txt`; the System page hides the button and says why.
- Agent (`acccb66`, unreleased): after a Docker step the wrapper runs a throwaway container from the running
controller's image (`--pull never`, `--network none`, 64 MB cap, one 200 MB block); pass = `OOMKilled=true` AND the
`oom` event; always removed. 12 red-proofs in `F/`. The helper also found 11 wrapper tests that never ran (a
`unittest.main()` mid-file) — moved; all pass.
- **By hand on demo-hp's guest** (`F/F1`, `F/F2`): `OOMKilled=true`, exit 137, 21 → 21 containers, none left. The `oom`
event was MISSING when the events window ended in the same second as the run, and present (`create attach start
oom die`) with the window ending a second later — the wrapper waits 2 s and reads to epoch + 1.
- Until agent v0.150.0 reaches the ring-0 boxes, no Docker set can be approved (none is pending).
## Instruction-file edits
None tonight.
## CI
felhom.eu `e0bdd52` → 1448 success (and `eed1dbd` 1446, `01c4a5d` 1447); catalog `d63ea35` → 1445; agent `7e82f32` → 1449;
this commit checked after its push.
## What is left for 2026-10-07 after 08:30
Part B (read both demo boxes' night under v0.301.0), Part D (one press on a demo box, measured; the page text), the
agent v0.150.0 release + bundle and its delivery, the controller release for Part D's text, and Part E's live proof if
the operator authorizes the key. Teardown tonight: none needed (no box provisioned; the hub DB copies deleted; the
by-hand check containers removed — 0 left).
**Teardown:** Tester 1 — vaultwarden removed through the product (no container, no volume), the saved
`controller.yaml.pre-r892` shredded, pointer restored identical; demo-hp — the one press made a normal local copy (kept
by retention), nothing else; hub — nothing provisioned. The night report stays in `REPORT-night-burndown-2026-10-06.md`.
+14 -4
View File
@@ -2,10 +2,20 @@
**Ready for the first real tester (Tester-2): yes. Tester 2 (a laptop) is off; nothing was sent to it.**
**Updated 2026-10-07 05:50: hub 0.140.0; demo-hp, demo-felhom and Tester 1 run controller 0.301.0, agent 0.149.0 —
nothing was delivered tonight. The open-items list is at 137. Night: `documentation/audits/night-burndown-2026-10-06/MORNING-NOTE.md`.
Day: Parts B and D of the read-back brief after 08:30, then the releases (hub 0.141.0, agent 0.150.0, controller, the
held catalog branch).**
**Updated 2026-10-07 09:35: hub 0.141.0; demo-hp, demo-felhom and Tester 1 run controller 0.302.0, agent 0.150.0. The
open-items list is at 130. Report: `REPORT.md`.**
## Morning (2026-10-07): read back, released, delivered — and the Tester 1 test passed
- **The shorter backup stop works.** Last night the apps on demo-hp were down about 1.5 minutes (it was almost
6 minutes before). One press of „Mentés most" stopped them for about 1 minute. The page tells the truth.
- **Released and delivered to all three boxes:** hub 0.141.0, agent 0.150.0 (with its root files), controller 0.302.0,
and the night's catalog fixes. Tester 2 was not touched (it is off).
- **The update test ran on the Tester 1 box with you present:** vaultwarden updated in 14 seconds, the test account
survived, and the box is back as it was.
- **New rule:** every helper gets the full fence list, word for word.
**Needs you (none urgent):** the six one-page designs from the night (each has one question). If nothing: they wait.
## Night (2026-10-06 → 07): the second burn-down night — fixes on main, nothing delivered
@@ -56,3 +56,13 @@ no reboot.
| R-872 | **closed** — the 05:00 run judged Tester 2 on the longer lines (dump missed=1, backup missed=0 correctly) and the mail reached the operator inbox (second channel) | 10 | (this batch) |
| R-892 | 07:10 (operator: key added, „continue"): SSH works, app list read; **the walk is blocked again** — no dashboard password for this box on DooPlex. Nothing changed on the box | 6 | (this batch) |
| R-892 | 07:41 dashboard password reset through the box's own reset (operator's word; new password only in a 0600 file); the walk was then **refused by the permission check** before any change — box unchanged | 20 | (this batch) |
## Morning 2026-10-07 (operator present)
| Act | Result | Commit |
|---|---|---|
| Parts B + D | R-518 read back: night stop 91 s / 11 s; one press 80 s; page text holds | `audits/readback-2026-10-07/RESULT-B-D.md` |
| releases | hub 0.141.0, agent 0.150.0 (+ bundle), controller 0.302.0, catalog merge `872039d` — all delivered to the three boxes | `readback-2026-10-07/delivery/` |
| R-892 | **closed** — vaultwarden 1.36.0 → 1.37.4 on the Tester 1 box, proven in 14.4 s; box back as before | `s3/box/` |
| rows closed on delivery | R-894, R-542, R-777, R-612, R-805, R-806 | — |
| rule 11 | every helper prompt carries the brief's fences in full (five copies) | decision 160 |
@@ -0,0 +1,2 @@
24: repo_url: https://gitea.dooplex.hu/admin/app-catalog-drill.git
@@ -0,0 +1,24 @@
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
total 20
drwxr-xr-x 2 root root 4096 Oct 7 07:26 .
drwxr-xr-x 63 root root 4096 Oct 4 19:41 ..
-rw-r--r-- 1 root root 7661 Oct 6 11:51 .felhom.yml
-rw-r--r-- 1 root root 3447 Oct 7 07:25 docker-compose.yml
@@ -0,0 +1,4 @@
24: repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
1
RESTORED-IDENTICAL
@@ -0,0 +1 @@
drill=872039d live=872039d
@@ -0,0 +1,10 @@
bookstack Up 5 hours (healthy)
bookstack-db Up 5 hours (healthy)
cloudflared Up 47 hours (healthy)
felhom-controller Up 45 seconds (healthy)
filebrowser Up 47 hours (healthy)
paperless-postgres Up 5 hours (healthy)
paperless-redis Up 5 hours (healthy)
paperless-webserver Up 5 hours (healthy)
privatebin Up 5 hours (healthy)
traefik Up 47 hours
@@ -0,0 +1,18 @@
{
"app": "vaultwarden",
"venue": "the Tester 1 box (VM 341, guest 9201; drill catalog), the product's guarded Update; seeded through the admin invite inside the box (R-892)",
"from": {
"vaultwarden": "vaultwarden/server:1.36.0-alpine"
},
"to": {
"vaultwarden": "vaultwarden/server:1.37.4-alpine"
},
"verdict": "proven",
"seed_read_before": true,
"seed_read_after": true,
"healthy_after": true,
"duration_s": 14.4,
"final_phase": "done",
"measured_at": "2026-10-07T07:25:12Z",
"evidence": "felhom.eu/documentation/audits/night-burndown-2026-10-06/s3/box/vaultwarden/step.txt"
}
@@ -0,0 +1,39 @@
drill (old pin): 61c5082 DRILL vaultwarden: back to vaultwarden/server:1.36.0-alpine for the Tester 1 box proof (R-892)
vaultwarden: self-registration http=400 (closed by design, R-512 — 400 expected)
vaultwarden: test-box admin sign-in and invite (inside the box) -> ('200', 'yes', '200')
vaultwarden: invited registration http=200
vaultwarden: token for the seeded account http=200 ok=True
C1 seed reads back BEFORE: True
before: pinned={'vaultwarden': 'vaultwarden/server:1.36.0-alpine'}
drill: 93eaacb DRILL vaultwarden: vaultwarden/server:1.36.0-alpine -> vaultwarden/server:1.37.4-alpine (Tester 1 box proof, R-892)
badge before: {'hu': [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — 1 napja'}], 'en': [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — 1 day ago'}]}
phase +0.0s checking | err=None
phase +2.1s backing-up | err=None
phase +4.1s pulling | err=None
phase +8.2s copying | err=None
phase +9.3s verifying | err=None
phase +14.4s done | err=None
vaultwarden: token for the seeded account http=200 ok=True
2026/10/07 07:25:13 [INFO] [stacks] update vaultwarden: accepted — guarded update started
2026/10/07 07:25:13 [INFO] [stacks] update vaultwarden: phase checking
2026/10/07 07:25:13 [INFO] [stacks] update vaultwarden: ladder — the last step (1 of 1) — the catalog's current definition
2026/10/07 07:25:15 [INFO] [stacks] update vaultwarden: no usable copy on any tier — younger than 24h0m0s and not older than this install's deploy (2026-10-07T07:24:33Z) (found: none) — backing up first
2026/10/07 07:25:15 [INFO] [stacks] update vaultwarden: phase backing-up
2026/10/07 07:25:16 [INFO] [stacks] update vaultwarden: precondition met after the backup — Tier 2 (second drive) copy from 2026-10-07T07:25:15Z
2026/10/07 07:25:16 [INFO] [stacks] update vaultwarden: phase safety-dump
2026/10/07 07:25:16 [INFO] [stacks] update vaultwarden: safety dump done (0 file(s)) []
2026/10/07 07:25:16 [INFO] [stacks] update vaultwarden: the undo copy will hold 1 named volume(s), 0.3 MiB
2026/10/07 07:25:16 [INFO] [stacks] update vaultwarden: phase pinning
2026/10/07 07:25:16 [INFO] [stacks] update vaultwarden: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/vaultwarden/docker-compose.yml (vaultwarden=vaultwarden/server:1.37.4-alpine)
2026/10/07 07:25:16 [INFO] [stacks] update vaultwarden: phase pulling
2026/10/07 07:25:20 [INFO] [stacks] update vaultwarden: phase copying
2026/10/07 07:25:21 [INFO] [stacks] update vaultwarden: phase copying
2026/10/07 07:25:21 [INFO] [stacks] update vaultwarden: copied vaultwarden_vaultwarden_data → vaultwarden_vaultwarden_data.pre-update-20261007T072521Z in 387ms
2026/10/07 07:25:21 [INFO] [stacks] update vaultwarden: phase starting
2026/10/07 07:25:22 [INFO] [stacks] update vaultwarden: phase verifying
2026/10/07 07:25:27 [INFO] [stacks] update vaultwarden: healthy after 5s (the app's health check passed)
2026/10/07 07:25:27 [INFO] [stacks] update vaultwarden: DONE in 14s
badge after: {'hu': [{'title': 'Ez az alkalmazás a legfrissebb elérhető változatot futtatja.', 'text': 'Naprakész'}], 'en': [{'title': 'This app is running the newest version available.', 'text': 'Up to date'}]}
RESULT final_phase=done after={'vaultwarden': 'vaultwarden/server:1.37.4-alpine'} seed_after=True verdict=proven (14.4 s)
remove -> 200
@@ -0,0 +1,37 @@
#!/usr/bin/env python3
"""Point the box (TARGET) at the drill catalog, or put the saved controller.yaml back. `09` §6.5."""
import io, os, re, sys
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
import box_walk as w
VOL = "/var/lib/docker/volumes/felhom-controller-data/_data"
SAVE = f"{VOL}/controller.yaml.pre-r892"
DRILL = "https://gitea.dooplex.hu/admin/app-catalog-drill.git"
def creds():
for l in io.open(os.path.expanduser("~/.git-credentials")).read().split("\n"):
m = re.match(r"https://(admin):([^@]+)@gitea\.dooplex\.hu", l)
if m: return m.group(1), m.group(2)
sys.exit("no admin credential")
if sys.argv[1] == "drill":
u, t = creds()
out = w.guest(f"""set -e
test -f {SAVE} || cp -p {VOL}/controller.yaml {SAVE}
python3 - <<'PY'
import re
p = "{VOL}/controller.yaml"; s = open(p).read()
s = re.sub(r'(^\\s+repo_url: ).*$', r'\\g<1>{DRILL}', s, count=1, flags=re.M)
s = re.sub(r'(^git:(?:\\n\\s+.*)*?\\n\\s+token: ).*$', r'\\g<1>"{t}"', s, count=1, flags=re.M)
s = re.sub(r'(^git:(?:\\n\\s+.*)*?\\n\\s+username: ).*$', r'\\g<1>"{u}"', s, count=1, flags=re.M)
open(p, "w").write(s)
PY
rm -rf {VOL}/catalog-cache {VOL}/data/catalog-cache
docker restart felhom-controller >/dev/null
grep -n 'repo_url' {VOL}/controller.yaml
""")
print(out.replace(t, "<token>"))
elif sys.argv[1] == "restore":
print(w.guest(f"""set -e
cp -p {SAVE} {VOL}/controller.yaml
rm -rf {VOL}/catalog-cache {VOL}/data/catalog-cache
docker restart felhom-controller >/dev/null
grep -n 'repo_url' {VOL}/controller.yaml; grep -c 'token: ""' {VOL}/controller.yaml || true
cmp {SAVE} {VOL}/controller.yaml && echo RESTORED-IDENTICAL"""))
@@ -0,0 +1,87 @@
"""vwstep.py <to-ref> — R-892: vaultwarden's step on the Tester 1 box (TARGET=tester-1, drill catalog), ONE process, through the product.
1 install vaultwarden fresh at the live pin (this run installs it — the admin seed refuses otherwise);
2 seed through the household's door, the invite through the admin page INSIDE the box
(FELHOM_BOX_ADMIN_SEED=1, upgrade_fixtures_box.box_admin_seed_allowed); read it back (C1);
3 a DRILL-only commit moves the image and adds a ladder entry; sync, rescan;
4 the product's guarded Update; the seed read back; box verdict JSON;
5 remove through the product.
Evidence: ../box/vaultwarden/step.txt + box-verdict-vaultwarden.json. The live entry is written ONLY by
`upgrade-test.py --write-ladder` from both verdicts."""
import json, os, re, subprocess, sys, time
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
import box_walk as w
import upgrade_fixtures_box as fixtures
APP, SUB, SVC = "vaultwarden", "vault", "vaultwarden"
to = sys.argv[1]
HERE = os.path.dirname(os.path.abspath(__file__))
EVD = os.path.join(HERE, "..", "box", APP); os.makedirs(EVD, exist_ok=True)
log = open(f"{EVD}/step.txt", "a", buffering=1)
D = "/mnt/5_hdd/felhom.eu/drill/app-catalog-drill"
def say(*a):
w.say(*a); log.write(" ".join(map(str, a)) + "\n")
fx = fixtures.FIXTURES[APP]
w.login()
if w.stack(APP).get("deployed"):
sys.exit(say("vaultwarden is already installed on this box — STOP (not ours to remove)") or 1)
FROM = sys.argv[2] # the old pin to install first, e.g. vaultwarden/server:1.36.0-alpine
subprocess.run(["git", "-C", D, "pull", "-q", "--rebase", "origin", "main"], check=True)
_c = f"{D}/templates/{APP}/docker-compose.yml"; _s = open(_c).read()
_cur = re.search(r"^\s+image:\s*(\S+)", _s, re.M).group(1)
if _cur != FROM:
open(_c, "w").write(_s.replace("image: " + _cur, "image: " + FROM, 1))
subprocess.run(["git", "-C", D, "commit", "-q", "-am", f"DRILL {APP}: back to {FROM} for the Tester 1 box proof (R-892)"], check=True)
subprocess.run(["git", "-C", D, "push", "-q", "origin", "main"], check=True, capture_output=True)
say("drill (old pin):", subprocess.run(["git", "-C", D, "log", "--oneline", "-1"], capture_output=True, text=True).stdout.strip())
w.sync_rescan(APP, FROM)
if not w.deploy(APP, SUB):
sys.exit(say("RESULT the install did not complete") or 1)
tok = fx.seed(w, SUB, say)
if tok is None or not fx.verify(w, SUB, tok, say):
say(f"RESULT C1 failed: {getattr(fx, 'tried', '')}")
w.remove(APP); sys.exit(1)
say("C1 seed reads back BEFORE: True")
before = (w.stack(APP).get("app_config") or {}).get("pinned_images")
say(f"before: pinned={before}")
subprocess.run(["git", "-C", D, "pull", "-q", "--rebase", "origin", "main"], check=True)
comp, fy = f"{D}/templates/{APP}/docker-compose.yml", f"{D}/templates/{APP}/.felhom.yml"
s = open(comp).read()
frm = re.search(r"^\s+image:\s*(\S+)", s, re.M).group(1)
open(comp, "w").write(s.replace("image: " + frm, "image: " + to, 1))
entry = {"from": {SVC: frm}, "to": {SVC: to}, "verdict": "proven", "tested_at": "DRILL", "harness_version": 5,
"evidence": "DRILL (box proof in progress)", "marks": {"files_may_change": False, "needs_person": None, "memory_tight": False}}
f = open(fy).read()
if to in f and frm in f:
pass
else:
f = (f.rstrip("\n") + "\n - " + json.dumps(entry) + "\n") if "update_ladder:" in f else (f.rstrip("\n") + "\nupdate_ladder:\n - " + json.dumps(entry) + "\n")
open(fy, "w").write(f)
subprocess.run(["git", "-C", D, "commit", "-q", "-am", f"DRILL {APP}: {frm} -> {to} (Tester 1 box proof, R-892)"], check=True)
subprocess.run(["git", "-C", D, "push", "-q", "origin", "main"], check=True, capture_output=True)
say("drill:", subprocess.run(["git", "-C", D, "log", "--oneline", "-1"], capture_output=True, text=True).stdout.strip())
w.sync_rescan(APP, to)
say(f"badge before: {w.badges(APP)}")
since = w.guest("date -u +%Y-%m-%dT%H:%M:%SZ").strip()
res = w.press_update(APP, poll=1, cap_s=1800)
for p in res.get("phases", []):
log.write(f" phase +{p['t']}s {p['phase']} | err={p['error']}\n")
time.sleep(10)
read = fx.verify(w, SUB, tok, say)
lines = w.guest(f"docker logs --since {since} felhom-controller 2>&1 | grep -E 'update {APP}' | grep -v DEBUG | cut -c1-400")
log.write(lines + "\n")
st = w.stack(APP); after = (st.get("app_config") or {}).get("pinned_images")
verdict = {"app": APP, "venue": "the Tester 1 box (VM 341, guest 9201; drill catalog), the product's guarded Update; seeded through the admin invite inside the box (R-892)",
"from": before, "to": after,
"verdict": "proven" if (res.get("final_phase") == "done" and read and (after or {}).get(SVC) == to) else "failed",
"seed_read_before": True, "seed_read_after": read, "healthy_after": st.get("state") == "running",
"duration_s": res.get("duration_s"), "final_phase": res.get("final_phase"), "measured_at": since,
"evidence": "felhom.eu/documentation/audits/night-burndown-2026-10-06/s3/box/vaultwarden/step.txt"}
json.dump(verdict, open(f"{EVD}/box-verdict-{APP}.json", "w"), indent=2)
say(f"badge after: {w.badges(APP)}")
say(f"RESULT final_phase={res.get('final_phase')} after={after} seed_after={read} verdict={verdict['verdict']} ({res.get('duration_s')} s)")
say(f"remove -> {w.remove(APP)}")
@@ -0,0 +1,15 @@
== hp
Oct 07 09:21:47 demo-hp felhom-agent[4029318]: time=2026-10-07T09:21:47.848+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE WROTE /usr/local/sbin/felhom-os-apply (replaced)"
Oct 07 09:21:47 demo-hp felhom-agent[4029318]: time=2026-10-07T09:21:47.848+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.150.0 written=1 same=24 self-check=ok signers-created=False"
Oct 07 09:21:48 demo-hp felhom-agent[4029318]: time=2026-10-07T09:21:48.765+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=68 total=68 degraded=""
Oct 07 09:21:48 demo-hp felhom-agent[4029318]: time=2026-10-07T09:21:48.765+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=23d9f79da880c635 op=agent_config_update
== felhom-pve
Oct 07 09:21:41 demo-felhom felhom-agent[2298873]: time=2026-10-07T09:21:41.733+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE WROTE /usr/local/sbin/felhom-os-apply (replaced)"
Oct 07 09:21:41 demo-felhom felhom-agent[2298873]: time=2026-10-07T09:21:41.733+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.150.0 written=1 same=24 self-check=ok signers-created=False"
Oct 07 09:21:42 demo-felhom felhom-agent[2298873]: time=2026-10-07T09:21:42.308+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=68 total=68 degraded=""
Oct 07 09:21:42 demo-felhom felhom-agent[2298873]: time=2026-10-07T09:21:42.308+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=883aca6d5a4d973d op=agent_config_update
== root@192.168.0.154
Oct 07 09:21:46 felhom felhom-agent[3741283]: time=2026-10-07T09:21:46.282+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE WROTE /usr/local/sbin/felhom-os-apply (replaced)"
Oct 07 09:21:46 felhom felhom-agent[3741283]: time=2026-10-07T09:21:46.282+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.150.0 written=1 same=24 self-check=ok signers-created=False"
Oct 07 09:21:47 felhom felhom-agent[3741283]: time=2026-10-07T09:21:47.109+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=68 total=68 degraded=""
Oct 07 09:21:47 felhom felhom-agent[3741283]: time=2026-10-07T09:21:47.109+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=2aad600292ed8add op=agent_config_update
@@ -0,0 +1,4 @@
2026-10-07T07:22:13Z hp guest 9201: gitea.dooplex.hu/admin/felhom-controller:0.302.0 Up 12 minutes (healthy); agent felhom-agent 0.150.0
2026-10-07T07:22:15Z felhom-pve guest 9201: gitea.dooplex.hu/admin/felhom-controller:0.302.0 Up 12 minutes (healthy); agent felhom-agent 0.150.0
2026-10-07T07:22:16Z root@192.168.0.154 guest 9201: gitea.dooplex.hu/admin/felhom-controller:0.302.0 Up 12 minutes (healthy); agent felhom-agent 0.150.0
Tester 2: not touched (off)
@@ -0,0 +1,7 @@
== floors 2026-10-07T07:09:19Z: 0.302.0 with min_agent 0.131.0 (golden stays 0.301.0); global floor not touched; Tester 2 not touched
demo-hp: Location: /customers/demo-hp?flash=floor_set
demo-felhom: Location: /customers/demo-felhom?flash=floor_set
tester-1: Location: /customers/tester-1?flash=floor_set
2026/10/07 09:09:19 [INFO] Customer demo-hp controller-version floor override set to "0.302.0" (declared MinAgent "0.131.0")
2026/10/07 09:09:20 [INFO] Customer demo-felhom controller-version floor override set to "0.302.0" (declared MinAgent "0.131.0")
2026/10/07 09:09:20 [INFO] Customer tester-1 controller-version floor override set to "0.302.0" (declared MinAgent "0.131.0")
@@ -0,0 +1,3 @@
2026-10-07T06:57:48Z
hub 0.141.0: sync=Synced health=Healthy rev=011a481a325ab6cec3f7cfadf7d39858837067ef image=gitea.dooplex.hu/admin/felhom-hub:0.141.0
2026/10/07 08:57:09 [INFO] felhom-hub 0.141.0 starting; sync 06:57:02Z, pod ready 06:57:35Z, healthz 200 + /system 200 at 06:57:42Z
@@ -0,0 +1,13 @@
== agent_update 0.150.0 (sha a23d1c90…) signed with felhom-op-1, ttl 45m, 2026-10-07T07:00:25Z
-- demo-hp-bb76ea
signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=e32907d89af4721e514c265fd9e070f5 expires=2026-10-07T07:45:25Z
{"op_blob_b64":"eyJleHBpcmVzX2F0IjoiMjAyNi0xMC0wN1QwNzo0NToyNVoiLCJpc3N1ZWRfYXQiOiIyMDI2LTEwLTA3VDA3OjAwOjI1WiIsImtleV9pZCI6ImZlbGhvbS1vcC0xIiwibm9uY2UiOiJlMzI5MDdkODlhZjQ3MjFlNTE0YzI2NWZkOWUwNzBmNSIsIm9wIjoiYWdlbnRfdXBkYXRlIiwicGFyYW1zIjp7InNoYTI1NiI6ImEyM2QxYzkwODViYzdmZDRmYzQ4ZmUwMzI3ZjY1MDRhYTgzZTYzMzE1MTBmYjRhM2QyM2UwNDJkZGRiMjZmOWMiLCJ2ZXJzaW9uIjoiMC4xNTAuMCJ9LCJ0YXJnZXQiOnsiZ3Vlc3RfaWQiOiIiLCJob3N0X2lkIjoiZGVtby1ocC1iYjc2ZWEifX0=","sig_armored":"-----BEGIN SSH SIGNATURE-----\nU1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgvzPSoI2ADfHbHEAHRCPmujzBoW\nMZnxNr+xYHre2UvD4AAAAMZmVsaG9tLW9wLXYxAAAAAAAAAAZzaGE1MTIAAABTAAAAC3Nz\naC1lZDI1NTE5AAAAQPaXiF/S1nw49tYDUIdJmvknW/ZvWbaP8ZjBsXB40tN7TYCuDLrsCH\n4ic3xaDV4BZa09MVpYE3YnEevC3by60QM=\n-----END SSH SIGNATURE-----\n"}
uploaded signed op to the hub jobs queue
-- demo-felhom-8363b5
signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=0c838937b7174329c68effdd91d8aaaf expires=2026-10-07T07:45:25Z
{"op_blob_b64":"eyJleHBpcmVzX2F0IjoiMjAyNi0xMC0wN1QwNzo0NToyNVoiLCJpc3N1ZWRfYXQiOiIyMDI2LTEwLTA3VDA3OjAwOjI1WiIsImtleV9pZCI6ImZlbGhvbS1vcC0xIiwibm9uY2UiOiIwYzgzODkzN2I3MTc0MzI5YzY4ZWZmZGQ5MWQ4YWFhZiIsIm9wIjoiYWdlbnRfdXBkYXRlIiwicGFyYW1zIjp7InNoYTI1NiI6ImEyM2QxYzkwODViYzdmZDRmYzQ4ZmUwMzI3ZjY1MDRhYTgzZTYzMzE1MTBmYjRhM2QyM2UwNDJkZGRiMjZmOWMiLCJ2ZXJzaW9uIjoiMC4xNTAuMCJ9LCJ0YXJnZXQiOnsiZ3Vlc3RfaWQiOiIiLCJob3N0X2lkIjoiZGVtby1mZWxob20tODM2M2I1In19","sig_armored":"-----BEGIN SSH SIGNATURE-----\nU1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgvzPSoI2ADfHbHEAHRCPmujzBoW\nMZnxNr+xYHre2UvD4AAAAMZmVsaG9tLW9wLXYxAAAAAAAAAAZzaGE1MTIAAABTAAAAC3Nz\naC1lZDI1NTE5AAAAQApxXKRWy1eB8LFZgOSZNXf/1qTNV2abKQ4bS/JgTW7v4B/s+0Ythq\n6PSEuRlK1Mh/fJPua0Cq0PIXQfkfXa0Qo=\n-----END SSH SIGNATURE-----\n"}
uploaded signed op to the hub jobs queue
-- tester-1-d70be4
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=7dfc6a5ca403c26cdf9801e478348b3f expires=2026-10-07T07:45:25Z
{"op_blob_b64":"eyJleHBpcmVzX2F0IjoiMjAyNi0xMC0wN1QwNzo0NToyNVoiLCJpc3N1ZWRfYXQiOiIyMDI2LTEwLTA3VDA3OjAwOjI1WiIsImtleV9pZCI6ImZlbGhvbS1vcC0xIiwibm9uY2UiOiI3ZGZjNmE1Y2E0MDNjMjZjZGY5ODAxZTQ3ODM0OGIzZiIsIm9wIjoiYWdlbnRfdXBkYXRlIiwicGFyYW1zIjp7InNoYTI1NiI6ImEyM2QxYzkwODViYzdmZDRmYzQ4ZmUwMzI3ZjY1MDRhYTgzZTYzMzE1MTBmYjRhM2QyM2UwNDJkZGRiMjZmOWMiLCJ2ZXJzaW9uIjoiMC4xNTAuMCJ9LCJ0YXJnZXQiOnsiZ3Vlc3RfaWQiOiIiLCJob3N0X2lkIjoidGVzdGVyLTEtZDcwYmU0In19","sig_armored":"-----BEGIN SSH SIGNATURE-----\nU1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgvzPSoI2ADfHbHEAHRCPmujzBoW\nMZnxNr+xYHre2UvD4AAAAMZmVsaG9tLW9wLXYxAAAAAAAAAAZzaGE1MTIAAABTAAAAC3Nz\naC1lZDI1NTE5AAAAQM2jDDmhZ80fYh3+ye6eeaAtZkb+8UfJaBOEru4/bnUjND1bD0P6oH\n5Ngnh4N2vjj8BWDZ+JLaNvQ/RHKZRaqAU=\n-----END SSH SIGNATURE-----\n"}
uploaded signed op to the hub jobs queue
@@ -0,0 +1,10 @@
== agent_config_update 0.150.0 (bundle 88456b38…) signed with felhom-op-1, ttl 45m, 2026-10-07T07:09:11Z
-- demo-hp-bb76ea
signed: op=agent_config_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=172c387e5c67a55df5c757e40a3e2141 expires=2026-10-07T07:54:11Z
uploaded signed op to the hub jobs queue
-- demo-felhom-8363b5
signed: op=agent_config_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=4152bdbf80d35e572bbd5fe526ea273f expires=2026-10-07T07:54:11Z
uploaded signed op to the hub jobs queue
-- tester-1-d70be4
signed: op=agent_config_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=f55e84c49d56ba88dca3711e933b2527 expires=2026-10-07T07:54:11Z
uploaded signed op to the hub jobs queue
@@ -0,0 +1,4 @@
== vouch 2026-10-07T06:59:41Z: agent 0.150.0, golden 0.301.0 (unchanged), min_agent 0.131.0 (unchanged)
HTTP/1.1 303 See Other
Location: /configuration?flash=artifacts_set
2026/10/07 09:00:00 [INFO] Artifact manifest set: agent=0.150.0 golden=0.301.0 min_agent="0.131.0" wrapper_sha=false bundle_sha="88456b386d9b1027bd22861cac8c23df004bf9fd9f67644d6595bfca8c94498e"
+14
View File
@@ -26,6 +26,20 @@
---
## 2026-10-07 (morning) — the read-back, the releases, the Tester 1 proof
The full text of every row below: `git show 011a481a32:documentation/backlog/OPEN-ITEMS.md`.
| Row | What | Closed | Evidence |
|---|---|---|---|
| **R-612** | **[P1-HIGH] `wishlist` cannot be signed up to on a fresh Felhom install, the deploy reports SUCCESS, and the error the customer sees is a LIE.** (P3) | CLOSED 2026-10-07 — DELIVERED (catalog 872039d) | Wishlist's healthcheck needs the seed (bench old rc 0 / new rc 1); `cat/R-612-red.txt`. |
| **R-894** | **After an agent restart, an UNREADABLE off-site storage makes the off-site tier look DUE, so the box asks for a copy that cannot be made.** (P3) | CLOSED 2026-10-07 — DELIVERED (agent v0.150.0) | Agent 0.150.0 on demo-hp, demo-felhom and Tester 1 (signed `agent_update` 07:06Z, bundle 07:21Z, capability probe 68/68). Built `74b5eae`, four red-proofs `audits/night-burndown-2026-10-06/s4/`; `07` §6.1. Not yet seen live: a restart followed by an unreadable storage (it needs an outage). |
| **R-542** | **[P3-LOW] `/api/disks/candidates` offers a REGISTERED, in-use drive under „initialize".** (P3) | CLOSED 2026-10-07 — DELIVERED (controller v0.302.0) | `/api/disks/candidates` drops drives backing a registered path; two red-proofs `audits/night-burndown-2026-10-06/ctrl/R-542-red.txt`; 0.302.0 healthy on the three boxes. |
| **R-777** | **[P2-MEDIUM] Emby and Jellyfin treat every internet visitor as being on the LAN — users with "remote access" off can sign in from the internet, IP filters and remote limits are skipped.** (P2) | CLOSED 2026-10-07 — DELIVERED (catalog 872039d) | Measured on 9202 (remote-access-OFF user signed in from the internet: 200 → 403 after; LAN 200); Jellyfin KnownProxies + Emby LocalNetworkSubnets seeded on a fresh volume or an empty list; synced to all three boxes (KnownProxies present in their jellyfin template). `audits/night-burndown-2026-10-06/r777/RESULT.md` |
| **R-892** | **The update test's box walk cannot reach the Tester 1 box, so decision 149's admin seed there cannot be used.** (P4) | CLOSED 2026-10-07 — PROVEN LIVE | The update test ran on the Tester 1 box (TARGET=tester-1, operator present): vaultwarden 1.36.0-alpine → 1.37.4-alpine through the product's guarded Update in 14.4 s (checking → backing-up → pulling → copying → verifying → done), the seeded account read back before and after, badge „Naprakész" after; app removed with its volume; app list equal to before; catalog pointer restored byte-identical; drill reset to live. Dashboard password reset through the box's own reset on the operator's word (kept in a 0600 file on DooPlex). `audits/night-burndown-2026-10-06/s3/box/` |
| **R-805** | **[P3-LOW] The volume-persistence gate judges an empty NAMED volume (R-788) but not an empty BIND mount.** (P4) | CLOSED 2026-10-07 — DELIVERED (catalog 872039d) | An empty bind is a note in the reasons, verdict unchanged (`09` §3 decision 159, CC unattended — operator may reverse); `cat/R-805-red.txt`. |
| **R-806** | **[P3-LOW] The gate's GET exercise speaks plain http to an HTTPS backend (crafty-controller :8443), and gramps-web did not answer on :5000.** (P4) | CLOSED 2026-10-07 — DELIVERED (catalog 872039d) | gramps-web runs 2 gunicorn workers (8 filled 1 GB); synced to the three boxes; `cat/R-806-red.txt`. |
## 2026-10-06 (night) — the second burn-down night
The full text of every row below: `git show e866a66b56:documentation/backlog/OPEN-ITEMS.md`.
File diff suppressed because one or more lines are too long