132 lines
11 KiB
Markdown
132 lines
11 KiB
Markdown
# REPORT — the day brief of 2026-10-21: fix the four serious problems from the rehearsal, and the Sunday clock change
|
||
|
||
Separate from the shared `REPORT.md` on purpose: another session works in this clone (repo rule: a second session writes
|
||
`REPORT-<topic>.md`). Evidence: `documentation/audits/fixes-2026-10-21/`. Design read first: `03` §8a, `07` §7.0/§6/§8.2,
|
||
`05` (scheduler, escrow store), `04` (self-scoped reads), `09` §3 decisions D19, 78, R-241.
|
||
|
||
## 1. Five lines
|
||
|
||
1. **R-1176 (a rebuilt box gets its off-site backups back) — FIXED AND LIVE.** Hub 0.163.0 (operator yes, 11:46 UTC);
|
||
proven end to end on demo-hp: no fresh key, recovery offered, wrong code refused, right code opened the history,
|
||
Nextcloud 3 031 of 3 033 files byte-identical, the next off-site runs appended (63 → 64).
|
||
2. **R-1175 (shares on two drives) — FIXED AND LIVE.** Controller 0.337.0 (global floor 15:54 UTC), D25: a gone source
|
||
keeps its copy until the household removes it.
|
||
3. **R-1191 (an imported app keeps its family lock) — FIXED AND LIVE.** Controller 0.337.0; proven on 9202: a stranger
|
||
gets 401 after export → remove → import (it was 200).
|
||
4. **R-1192 (the 8 GB freeze) — CAUSE MEASURED, FIX COMMITTED, NOT RELEASED.** The guest's swap was capped at Proxmox's
|
||
512 MB default; the agent fix (swap = RAM) is committed (`286a7e3`); its release and vouch are the operator's (D26).
|
||
Proven by applying the same setting by hand for 2 hours of the same uploads.
|
||
5. **Ready for Tester 2: NO — one step left:** the agent release with the swap fix. Everything else on the P1 list is live.
|
||
|
||
## 2. R-1192 — the mechanism, the numbers, the fix
|
||
|
||
**One sentence:** the guest's swap is capped at 512 MB (Proxmox's LXC default — the agent never set `swap`), so when
|
||
Immich loaded its analysis models (+1.15 GB) on top of 8 apps, the 4096 MB guest could evict only program code, and
|
||
~25 containers re-read their code from disk without end.
|
||
|
||
| | the freeze (13:26–13:34 UTC, cap 512 MB) | the proof (13:43–15:43 UTC, cap = RAM by hand) |
|
||
|---|---|---|
|
||
| guest private memory (anon), max | 3 585 MB | 2 782 MB |
|
||
| guest swap used, max | 511 MB of 512 (full) | 2 555 MB |
|
||
| refaults per 10 s, max | 2 741 774 | 187 462 |
|
||
| disk reads per 10 s, max | 12 GB | 0.9 GB |
|
||
| memory „full" pressure (avg10), max | 96 % | 12 % |
|
||
| IO pressure, max | 97 % | 53 % |
|
||
| load, max | 162 | 33 |
|
||
| guest „limit hit" events per 10 s, max | 3 889 | 0 |
|
||
| OOM kills | 0 | 0 |
|
||
| dashboard (DooPlex probe every 10 s) | dead 13:28:18–13:34:48 | every probe answered, slowest 1.26 s |
|
||
|
||
The experiment between them: `pct set 9201 --swap 4096` at 13:34:06, uploads still running → memory „full" 96 % → 1.7 %
|
||
within 2 minutes, the dashboard back 42 s after the change. The host had 2.2 GB RAM and 7.4 GB swap free throughout.
|
||
Load for the proof: the same uploads as fresh work, repeated — Paperless 110 cycles × 53 PDFs (emptied each time), Immich
|
||
111 new users × 306 photos (305 accepted each time; one photo is refused every cycle), two Nextcloud trees (WebDAV small
|
||
files, ~20 files/min under this load — slow, nothing lost). Method: a recorder on the box (`tools/rec.py`, 10 s, copied to
|
||
DooPlex every minute) and the second channel on DooPlex (`tools/outside.sh`: the dashboard's `/api/health` and QEMU's
|
||
disk counters from demo-hp).
|
||
|
||
**Fix:** agent `286a7e3` (commit only): `swap` = memory at bring-up, at a RAM resize, and once at agent start for every
|
||
pool guest whose swap is below its RAM (live, no restart, never lowers). Tests red-proved. The controller half (0.337.0):
|
||
„Normál használat mellett ez nem okoz problémát." is gone; the install page now says **„Az alkalmazások csúcsterhelése
|
||
meghaladhatja a rendelkezésre álló memóriát. Ilyenkor a doboz nagy feltöltések alatt lelassulhat, és akár percekig sem
|
||
válaszol. Ha ez gyakran előfordul, távolíts el egy alkalmazást."** No catalog change and no admission-ceiling change:
|
||
the measured cause is the swap cap, and the order of preference (catalog → controller → agent) gave way to the proof.
|
||
R-1193 (an earlier, plainer alarm): a design note in its row; build it only if a freeze is seen after the agent release.
|
||
|
||
## 3. R-1176 — the shape, the order check, the live proof
|
||
|
||
**Shape (operator ruling):** `store.ReattachRetainedEscrow` moves the newest retained row of the customer's newest
|
||
escrow-ACKED host deletion into `host_escrow` for its one live host, in one transaction (no other live host, no live
|
||
package, the deleted host not live again, an identity blob present). Called on the agent's host report and before the
|
||
controller ACK's escrow status. Never opened, re-sealed or logged; audit `escrow_carried` (host ids + hash); two live
|
||
hosts → `escrow_carry_refused` mailed once. Self-scope unchanged: the new host reads only rows attached to it (its
|
||
retained GET for the old host is 403).
|
||
|
||
**Order check (measured, not assumed):** the hub carried at the agent's FIRST report (12:04:21 UTC); the controller's
|
||
settle gate went GO at 12:06:25 after its first ACK; at 12:06:28 it logged „NOT minting a repository password: the hub
|
||
holds a sealed recovery package". So no controller change was needed; 0.337.0 still records the escrow before
|
||
`SetFloor` (a pinned belt for a hub that carries later).
|
||
|
||
**Live proof (demo-hp VM 9321, Tester4, public ISO 1.29.0, no operator press before the install):** bind (one OCR misread
|
||
of the pairing code cost one of five tries — harness), claim, the dashboard offered „A korábbi, házon kívüli mentéseid
|
||
visszaszerezhetők"; a wrong code: „A megadott helyreállítási kódot nem fogadtuk el … Semmi nem változott"; the right code
|
||
unlocked 9 groups; Nextcloud reinstalled („előbb telepítsd újra" is the page's own order), „Teljes visszaállítás" with
|
||
the placement confirm → „3113 fájl és 2 adatkötet és az adatbázis visszaállítva"; **sha256 through Nextcloud's own WebDAV
|
||
against the rehearsal's A5 manifest: 3 031 of 3 033 byte-identical** — the 2 missing never reached the old box (the 1.5 GB
|
||
file Nextcloud refused, a symlink rclone skips); 4 extra files are the rehearsal's later uploads. Off-site: two runs, 63 → 64
|
||
snapshots in the same 1.4 GB store, `orphaned: false`; the hub's full log has no set-aside and no key change.
|
||
|
||
## 4. Versions, commits, tests
|
||
|
||
| Component | Version | State | Commits |
|
||
|---|---|---|---|
|
||
| hub | 0.163.0 | **deployed** 11:46 UTC (operator yes), Synced/Healthy | `9688e76e`, manifest `cd3d0598` |
|
||
| controller | 0.337.0 (`sha256:264af313…`) | **released**: rc.1 on 9202; per-box floors 13:59 → three test boxes on 0.337.0 by 14:06; box walk green; **global floor 0.337.0 / MinAgent 0.158.1 at 15:54:29 UTC**, read back; per-box floors cleared | `8f5bea0`, `5929b8e`, `7848d4a` |
|
||
| agent | (0.162.0 when released) | **committed, not released** | `286a7e3` |
|
||
| catalog, installer | — | unchanged | — |
|
||
|
||
Tests (all pass; each red-proved — seen failing first): hub `TestR1183_*` (3), `TestR1176_*` (store 7, api 3),
|
||
`TestR1182_*` (store 2, api 1), `TestR1203_*`; controller `TestR1175_*` (3), `TestR1181_*` (2), `TestD25_*` (3),
|
||
`TestR1179_*`, `TestR1188c_*`, `TestR1191_*` (2), `TestR1176_EscrowIsRecordedBeforeTheFloorIsKnown`, `TestR1182_*` (2);
|
||
agent `TestR1192_*` (2) + the resize test's swap assertion. CI: green by job id for every push (see §5).
|
||
|
||
## 5. Morning list, teardown, rows, observations
|
||
|
||
**Morning list (operator):** D26 — release + vouch the agent with the swap fix; delete host **Tester4-f6b67c** (asked
|
||
today); R-1205 (the whole-box tier waits for „Re-issue PBS credentials" after a failed second install); the
|
||
`REHEARSAL_2026_10_20_*` / `FIXES_2026_10_21_*` names in sec.
|
||
|
||
**Operator acts today:** the hub-deploy yes; the root credential reveal for the freeze test (~2 min); the host delete
|
||
(done 16:49 UTC — the delete appears 6 minutes after a box goes off, R-30's rule).
|
||
|
||
**Teardown (three layers):**
|
||
- **Machine (demo-hp, Tier 0):** VM 9321 `rh1021-box2` and helper CT 9331 `rh1021-loader` destroyed with `--purge` at
|
||
16:27:40 UTC (names checked first); no image left on `nvme-scratch`; 341, 9201, 9202, 9401 untouched. 9202 moved from
|
||
`0.337.0-rc.1` to `0.337.0` by hand (it does not take the hub's floor). The public ISO 1.29.0 was only read.
|
||
- **Hub:** the operator deleted host Tester4-f6b67c (16:49:37 UTC, escrow acknowledged): no Tester4 host, no current
|
||
package, the household's package retained under Tester4-f6b67c with the same hash, `repo_tag` 'live' — a future
|
||
rebuild carries it again; the WireGuard peer gone (`P6-release/05-teardown-hub.txt`).
|
||
- **ep0 / Storage Box (read only, nothing removed):** Tester4's off-site store gained 2 snapshots (63 → 64, normal
|
||
traffic); the hub's registrar pinned the rebuilt box's key and removed the dead box's line (normal enrollment). No PBS
|
||
descriptor was made (R-1205), so nothing new on ep0's PBS.
|
||
|
||
**Rows:** before 182 (181 after a parallel session closed R-1201); after 176; opened 2 (R-1205, R-1206); closed 7 (R-1176,
|
||
R-1203, R-1175, R-1179, R-1181, R-1182, R-1191). R-1183 → VERIFY (due 2026-10-26); R-1192 → WAITING-ON-OPERATOR;
|
||
R-1188 (c) fixed. Capability map A12, A13 → PROVEN-LIVE. `unproven.py --summary`: unchanged, 35 of 55 not walked.
|
||
|
||
**Observations:**
|
||
- PBS auto re-issue refused for the rebuilt Tester4 (latest deletion un-acked) — **FILED: R-1205**.
|
||
- Future-dated snapshots from the clock test show as „Legutóbbi mentés: 2026. 10. 28." — **FILED: R-1206**.
|
||
- The pairing-code OCR misread one character — **NOT-A-FINDING:** harness (a human reads the screen); corrected against the hub's record.
|
||
- Immich's first admin needs the household's dashboard session (the setup gate) — **NOT-A-FINDING:** designed.
|
||
- The full restore asks to confirm a new place (backup on `hdd_1`, now the internal disk) — **NOT-A-FINDING:** designed (R-351).
|
||
- Nextcloud's small-file WebDAV upload ~20 files/min under heavy concurrent load — **NOT-A-FINDING:** slow, not lost; the new install-page sentence says big uploads can be slow.
|
||
- `kubectl logs -l` reads only 10 lines — **NOT-A-FINDING** (a tooling trap; saved to memory, two checks redone over the full log).
|
||
- Tester 2 last reported 2026-10-04 on 0.292.0; the floor reaches it only when it reports — **NOT-A-FINDING** (R-1139 covers the box).
|
||
|
||
**Secrets:** names only — `FIXES_2026_10_21_PAIRING_CODE`, `FIXES_2026_10_21_BIND_TOKEN`, `FIXES_2026_10_21_SETUP_CODE`
|
||
(one-time, spent), `FIXES_2026_10_21_ROOTPW` (the operator's reveal; the box is torn down). The bind link and the setup code
|
||
were read from the mailbox by the mail connector, so both one-time values entered this session's record (spent within
|
||
minutes, as in the rehearsal). A scan of all 45 evidence files for every secret used today: 0 hits. Nothing secret in any
|
||
commit.
|