Register 189 -> 191 (R-1232 a reinstalled box's first ep0 copy may come a week late — likely; R-1233 an error line naming nothing); test-night evidence: night jobs, D21 on the box, no copy container left, off-site leg/proof/integrity, controller log, agent journal (sudo noise trimmed), recorder, outside probe, loader logs, Immich per-user counts, night numbers

2026-10-23 07:13:52 +02:00
parent 5dc17cab03
commit 1cb3e57133
17 changed files with 24124 additions and 2 deletions
@@ -24,3 +24,9 @@ Test night provenance (night 2026-10-22 → 23, Part 4)
  with the household's recovery code (07-unlock.txt).
 - Recorders: rec.py on the box host (10 s) from 19:47 UTC; outside.sh on DooPlex (10 s /api/health + qemu I/O) from
  20:09 UTC.
+- LOAD CORRECTION (found 22:14 UTC): the VOID attempt's load2.py (pid 4514, „load2.py 120") was NOT stopped at 19:58 —
+  the `pkill -f load2.py` ran inside a `bash -c` whose own command line held the pattern and killed its own shell first
+  (the known pgrep-matches-itself trap). It ran from 19:55:59 to 22:14 UTC beside the planned load: 406 Paperless cycles
+  (upload / delete / empty trash), Immich and Nextcloud cycles, with the same household credentials. So the box carried
+  MORE than the planned load for ~2 h. Stopped 22:14 UTC (kill by pid), with its orphaned rclone (pid 51651). Its log:
+  the loader's /root/void-1955/proof.log (3908 DONE lines).
@@ -0,0 +1,14 @@
+/mnt/felhom-drives/belso/backups/primary/immich
+/mnt/felhom-drives/belso/backups/primary/immich/compose
+/mnt/felhom-drives/belso/backups/primary/immich/compose/.felhom.yml
+/mnt/felhom-drives/belso/backups/primary/immich/compose/app.yaml
+/mnt/felhom-drives/belso/backups/primary/immich/compose/docker-compose.yml
+/mnt/felhom-drives/belso/backups/primary/immich/manifest.json
+48K	/mnt/felhom-drives/belso/backups/primary/immich
+35:  volumes:
+36-    # R-49, operator ruling D21 = B (2026-10-22): the machine-learning model cache (~824 MB, 62 % of the off-site copy)
+37-    # is never backed up — Immich downloads the models again the first time smart search or face recognition needs
+38-    # them. Read by controller >= 0.340.0 (backup.volumes); an older controller ignores the key.
+39-    - name: immich_ml_cache
+40-      class: excluded
+41-
@@ -0,0 +1,14 @@
+/mnt/felhom-drives/belso/backups/primary/immich/db-dumps/:
+drwxr-xr-x 2 root root      4096 Oct 23 00:30 .
+drwxr-xr-x 5 root root      4096 Oct 23 00:33 ..
+-rw-r--r-- 1 root root 188841957 Oct 23 00:30 immich-postgres.sql
+
+/mnt/felhom-drives/belso/backups/primary/immich/volume-dumps/:
+drwxr-xr-x 2 root root      4096 Oct 23 00:31 .
+drwxr-xr-x 5 root root      4096 Oct 23 00:33 ..
+-rw-r--r-- 1 root root 415627264 Oct 23 00:31 immich_immich_postgres_data.tar
+577M	/mnt/felhom-drives/belso/backups/primary/immich
+2026/10/23 00:31:10 [INFO] [backup] Stopping immich for safe volume dump
+2026/10/23 00:31:13 [INFO] [backup] Volume immich/immich_immich_ml_cache is excluded by the template (a cache the app rebuilds) — not dumped (R-49)
+2026/10/23 00:31:23 [INFO] [backup] Volume dump: immich/immich_immich_postgres_data → 396.4 MB
+2026/10/23 00:31:23 [INFO] [backup] Restarting immich after volume dump
@@ -0,0 +1 @@
+copies left: 0
@@ -0,0 +1,18 @@
+2026/10/23 02:15:00 [INFO] [scheduler] Running job: offbox-backup
+2026/10/23 02:15:00 [INFO] [offbox] backup run started (8 app(s) toggled)
+2026/10/23 02:17:39 [INFO] [offbox] pre-push dump leg completed in 2m39.723s — snapshot pair is coherent
+2026/10/23 02:17:55 [INFO] [offbox] backed up paperless-ngx (/mnt/felhom-drives/belso/backups/primary/paperless-ngx, 1 mandatory path(s))
+2026/10/23 02:17:57 [INFO] [offbox] backed up radicale (/mnt/sys_drive/felhom-data/backups/primary/radicale, 0 mandatory path(s))
+2026/10/23 02:18:00 [INFO] [offbox] backed up vaultwarden (/mnt/sys_drive/felhom-data/backups/primary/vaultwarden, 0 mandatory path(s))
+2026/10/23 02:18:03 [INFO] [offbox] backed up actualbudget (/mnt/sys_drive/felhom-data/backups/primary/actualbudget, 0 mandatory path(s))
+2026/10/23 02:18:07 [INFO] [offbox] backed up bookstack (/mnt/sys_drive/felhom-data/backups/primary/bookstack, 0 mandatory path(s))
+2026/10/23 02:18:44 [INFO] [scheduler] Running job: offsite-credential-retry
+2026/10/23 02:18:44 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
+2026/10/23 02:19:49 [INFO] [offbox] backed up immich (/mnt/felhom-drives/belso/backups/primary/immich, 1 mandatory path(s))
+2026/10/23 02:19:52 [INFO] [offbox] backed up mealie (/mnt/sys_drive/felhom-data/backups/primary/mealie, 0 mandatory path(s))
+2026/10/23 02:20:13 [INFO] [offbox] backed up nextcloud (/mnt/felhom-drives/belso/backups/primary/nextcloud, 1 mandatory path(s))
+2026/10/23 02:20:15 [INFO] [offbox] retention skipped (after-run): no clean-up window now (not due (last window 2026-10-20T17:07:48Z)) — nothing deleted (decision 68)
+2026/10/23 02:20:27 [INFO] [offbox] backup OK: 8 app(s) backed up, 80 snapshot(s), 5m15s
+2026/10/23 02:20:27 [INFO] [update-leg] started (after-offsite): window 02:30, no step starts at or after 07:30
+2026/10/23 02:20:27 [INFO] [update-leg] update leg (after-offsite): done=0 undone=0 held=0 failed=0 skipped=0 in 0s [skipped: ]
+2026/10/23 02:20:27 [INFO] [scheduler] Job offbox-backup completed (took 5m27.356s)
@@ -0,0 +1,3 @@
+2026/10/23 03:30:00 [INFO] [scheduler] Running job: offsite-proof
+2026/10/23 03:30:03 [INFO] [offbox] proof: actualbudget PASSED on snapshot 47244102 in 1.948s — the backup holds what this app should have
+2026/10/23 03:30:03 [INFO] [scheduler] Job offsite-proof completed (took 3.697s)
@@ -0,0 +1,91 @@
+# 2026-10-23T05:11:55Z Immich DB, read-only SELECT (photos per user)
+tester4+odon@felhom.eu|303
+tester4+proof10@felhom.eu|303
+tester4+proof11@felhom.eu|303
+tester4+proof12@felhom.eu|303
+tester4+proof13@felhom.eu|303
+tester4+proof14@felhom.eu|303
+tester4+proof15@felhom.eu|303
+tester4+proof16@felhom.eu|303
+tester4+proof17@felhom.eu|303
+tester4+proof18@felhom.eu|303
+tester4+proof19@felhom.eu|303
+tester4+proof1@felhom.eu|303
+tester4+proof20@felhom.eu|303
+tester4+proof21@felhom.eu|303
+tester4+proof22@felhom.eu|303
+tester4+proof23@felhom.eu|303
+tester4+proof24@felhom.eu|303
+tester4+proof25@felhom.eu|303
+tester4+proof26@felhom.eu|303
+tester4+proof27@felhom.eu|303
+tester4+proof28@felhom.eu|303
+tester4+proof29@felhom.eu|303
+tester4+proof2@felhom.eu|303
+tester4+proof30@felhom.eu|303
+tester4+proof31@felhom.eu|303
+tester4+proof32@felhom.eu|303
+tester4+proof33@felhom.eu|303
+tester4+proof34@felhom.eu|303
+tester4+proof35@felhom.eu|303
+tester4+proof36@felhom.eu|303
+tester4+proof37@felhom.eu|303
+tester4+proof38@felhom.eu|303
+tester4+proof39@felhom.eu|303
+tester4+proof3@felhom.eu|303
+tester4+proof40@felhom.eu|303
+tester4+proof41@felhom.eu|303
+tester4+proof42@felhom.eu|303
+tester4+proof43@felhom.eu|303
+tester4+proof44@felhom.eu|303
+tester4+proof45@felhom.eu|303
+tester4+proof46@felhom.eu|303
+tester4+proof47@felhom.eu|303
+tester4+proof48@felhom.eu|303
+tester4+proof49@felhom.eu|303
+tester4+proof4@felhom.eu|303
+tester4+proof50@felhom.eu|303
+tester4+proof51@felhom.eu|303
+tester4+proof52@felhom.eu|303
+tester4+proof53@felhom.eu|303
+tester4+proof54@felhom.eu|303
+tester4+proof55@felhom.eu|303
+tester4+proof56@felhom.eu|303
+tester4+proof57@felhom.eu|303
+tester4+proof58@felhom.eu|303
+tester4+proof59@felhom.eu|303
+tester4+proof5@felhom.eu|303
+tester4+proof60@felhom.eu|303
+tester4+proof61@felhom.eu|303
+tester4+proof62@felhom.eu|303
+tester4+proof63@felhom.eu|303
+tester4+proof64@felhom.eu|303
+tester4+proof65@felhom.eu|303
+tester4+proof66@felhom.eu|303
+tester4+proof67@felhom.eu|303
+tester4+proof68@felhom.eu|303
+tester4+proof69@felhom.eu|303
+tester4+proof6@felhom.eu|303
+tester4+proof70@felhom.eu|303
+tester4+proof71@felhom.eu|303
+tester4+proof72@felhom.eu|303
+tester4+proof73@felhom.eu|303
+tester4+proof74@felhom.eu|303
+tester4+proof75@felhom.eu|303
+tester4+proof76@felhom.eu|303
+tester4+proof77@felhom.eu|303
+tester4+proof78@felhom.eu|303
+tester4+proof79@felhom.eu|303
+tester4+proof7@felhom.eu|303
+tester4+proof80@felhom.eu|303
+tester4+proof81@felhom.eu|303
+tester4+proof82@felhom.eu|303
+tester4+proof83@felhom.eu|303
+tester4+proof84@felhom.eu|303
+tester4+proof85@felhom.eu|303
+tester4+proof86@felhom.eu|303
+tester4+proof87@felhom.eu|303
+tester4+proof88@felhom.eu|303
+tester4+proof89@felhom.eu|303
+tester4+proof8@felhom.eu|303
+tester4+proof9@felhom.eu|303
@@ -0,0 +1,9 @@
+== load.log
+DONE lines 2 ok 358 bad 1
+== proof.log
+DONE lines 154 ok 30590 bad 89
+== void-1955/proof.log
+DONE lines 406 ok 4224 bad 17294
+19:56:00Z c1 paperless DONE ok 0 bad 53
+19:56:00Z c2 paperless DONE ok 0 bad 53
+19:56:00Z c3 paperless DONE ok 0 bad 53
@@ -0,0 +1,10 @@
+== load 19:56–23:38: rec samples 1349, health probes 1175 / failed 2
+   worst health gap between answers 23.0 s at 22:03:52; slowest answer 3.348819 s at 21:57:46
+   worst memory full avg10 host 22.74 % (2026-10-22T22:04:38Z) / guest 24.33 % (2026-10-22T22:04:38Z); samples > 20 %: 2; worst avg60 host 7.96 / guest 8.38
+   host oom_kill +0; guest memory.events: max +42, oom +0, oom_kill +0, high +758573; guest swap max 2601 MB
+   non-200: 20:16:34 502
+   non-200: 22:03:40 502
+== night 23:38–05:00: rec samples 1932, health probes 1833 / failed 0
+   worst health gap between answers 12.0 s at 23:40:18; slowest answer 0.272576 s at 01:30:13
+   worst memory full avg10 host 28.22 % (2026-10-23T00:32:39Z) / guest 30.62 % (2026-10-23T00:32:39Z); samples > 20 %: 4; worst avg60 host 9.55 / guest 11.36
+   host oom_kill +0; guest memory.events: max +77845, oom +0, oom_kill +0, high +54086; guest swap max 2211 MB
@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+# soakstats.py REC OUTSIDE — the test night's numbers, per window (UTC): load (the uploads) and night (the chain).
+import json, sys, datetime as dt
+rec, out = sys.argv[1], sys.argv[2]
+W = {"load 19:56–23:38": ("2026-10-22T19:56:00Z", "2026-10-22T23:38:00Z"),
+     "night 23:38–05:00": ("2026-10-22T23:38:00Z", "2026-10-23T05:00:00Z")}
+def T(s): return dt.datetime.strptime(s, "%Y-%m-%dT%H:%M:%SZ")
+R = [json.loads(l) for l in open(rec) if l.strip()]
+O = []
+for l in open(out):
+    p = l.split()
+    if len(p) < 3: continue
+    code = p[1].split("=")[1]; lat = float(p[2]) if code != "000" else None
+    O.append((T(p[0]), code, lat))
+for name, (a, b) in W.items():
+    a, b = T(a), T(b)
+    rs = [r for r in R if a <= T(r["t"]) < b]
+    os_ = [o for o in O if a <= o[0] < b]
+    ok = [o for o in os_ if o[1] == "200"]
+    gaps = [(ok[i][0] - ok[i-1][0]).total_seconds() for i in range(1, len(ok))]
+    wg = max(gaps) if gaps else None
+    wgat = ok[gaps.index(wg)+1][0].strftime("%H:%M:%S") if gaps else ""
+    slow = max((o for o in ok), key=lambda o: o[2]) if ok else None
+    hf = [(r["psi"]["memory"]["full"][0], r["t"]) for r in rs]
+    gf = [((r["guest"].get("psi") or {}).get("full", [0])[0] if isinstance(r["guest"].get("psi"), dict) else 0, r["t"]) for r in rs]
+    h60 = max((r["psi"]["memory"]["full"][1] for r in rs), default=0)
+    g60 = max(((r["guest"].get("psi") or {}).get("full", [0, 0])[1] if isinstance(r["guest"].get("psi"), dict) else 0) for r in rs) if rs else 0
+    over = sum(1 for (v, _), (g, _) in zip(hf, gf) if v > 20 or g > 20)
+    ev0 = rs[0]["guest"]["events"] if rs else {}; ev1 = rs[-1]["guest"]["events"] if rs else {}
+    def d(k): return int(ev1.get(k, 0)) - int(ev0.get(k, 0))
+    ook = int(rs[-1]["vm"].get("oom_kill", 0)) - int(rs[0]["vm"].get("oom_kill", 0)) if rs else 0
+    sw = max((int(r["guest"]["swap"] or 0) for r in rs), default=0) // (1024*1024)
+    mh = max(hf, default=(0, "")); mg = max(gf, default=(0, ""))
+    print(f"== {name}: rec samples {len(rs)}, health probes {len(os_)} / failed {len(os_)-len(ok)}")
+    print(f"   worst health gap between answers {wg} s at {wgat}; slowest answer {slow[2] if slow else None} s at {slow[0].strftime('%H:%M:%S') if slow else ''}")
+    print(f"   worst memory full avg10 host {mh[0]} % ({mh[1]}) / guest {mg[0]} % ({mg[1]}); samples > 20 %: {over}; worst avg60 host {h60} / guest {g60}")
+    print(f"   host oom_kill +{ook}; guest memory.events: max +{d('max')}, oom +{d('oom')}, oom_kill +{d('oom_kill')}, high +{d('high')}; guest swap max {sw} MB")
+    nf = [o for o in os_ if o[1] != "200"]
+    for o in nf[:5]: print("   non-200:", o[0].strftime("%H:%M:%S"), o[1])
@@ -153,7 +153,7 @@ stopping line that lies.
 | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
 | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |

-## Backup & restore — 46 rows (P2 10, P3 19, P4 17)
+## Backup & restore — 47 rows (P2 11, P3 19, P4 17)

 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -203,6 +203,7 @@ stopping line that lies.
 | **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | operator |
 | **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator |
 | **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC |
+| **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go` `newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC |

 ## Storage & devices — 11 rows (P3 4, P4 7)

@@ -262,7 +263,7 @@ stopping line that lies.
 | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |


-## Monitoring & notifications — 19 rows (P2 1, P3 13, P4 5)
+## Monitoring & notifications — 20 rows (P2 1, P3 13, P4 6)

 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -285,6 +286,7 @@ stopping line that lies.
 | **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC |
 | **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
 | **R-1230** | Monitoring & notifications | P3 | **A reinstalled box that waits for the household's recovery code was reported „stuck” after an hour: a false operator mail, a needless re-issue of the off-site login, and a controller restart on the box.** Measured on the test night 2026-10-23 (Tester4-16fdef, `audits/night-2026-10-23/P4-testnight/`): the box declared `offsite.state=awaiting_recovery_key`; the hub's delivery check knew only `needs_credential`, read „consumed, never applied” and at 20:16 UTC mailed `offsite_delivery_stuck` and re-staged the password (the config bump restarted the controller — one 502 on the outside probe). The controller's own comment said this branch is skipped — false. | READY — hub 0.167.0 source `14388e21`: the checker stands down for that declared state; `TestR1230_AHeldBoxIsNotStuck`, red-proved. | the hub deploy (the operator's yes) | Deploy hub 0.167.0; close then | CC |
+| **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |


 ## Hub & operator — 12 rows (P3 5, P4 7)