felhom.eu/documentation/audits/DRILL-night-2026-09-26.md

DRILL — 2026-09-25 evening → 26 night: the box converts a PostgreSQL major (docmost first); kept data; the night watch

Brief: "the box converts a PostgreSQL major for its first app (docmost); a reinstall over kept data asks the household…". Architecture read before any claim: 09-update-architecture.md §3 decisions 13–36, §3b Q5, §6.1a, §6.4 part 10; 07-backup-architecture.md §6; 08-alarm-ladder.md. Evidence: audits/night-2026-09-26/ (part0, A, B, C, D, E, F, G, tools).

Not done, or changed

• Two controller releases, not one — as the brief allowed: v0.273.0 (the conversion; floor raised at 11:32Z so the live catalog could move docmost before the night) and v0.274.0 (kept data; 12:30Z).
• Part E was built by a parallel helper session in its own git worktree (the brief's parts B–D and E touch different files); I reviewed it, found one defect live (R-692) and fixed it before the release, and ran the whole E5 proof myself.
• The live proofs of Part D ran on 0.273.0-rc1, not the released image: the release adds only the recovery line's wording (found live in case (c)) and R-687's two small items.
• docmost moved with its own memory limit raised 384M → 512M (decision 39): the bench marked the step memory_tight, and the move gate refuses a tight step whose limit does not move. The mark stays at 512M (80.4 %) — Node sizes its heap from the limit (R-693).
• The bench was a throwaway LXC 9401 on demo-hp, created and destroyed tonight (the first create picked an ARM template and failed to start; destroyed, name checked first, recreated with amd64). 9202 could not be the bench: the harness's container names collide with the installed docmost and it ends with down -v.
• Part F3 (adventurelog, R-655): measured, not moved. The answer to "skip or pre-seed download-countries" is in the row; the move needs a choice between two options with different costs.
• My own tool error in Part A4: a bare docker compose up -d in docmost's stack dir (no .env there — the controller passes the env) started docmost without its secrets; it crash-looped and the box stopped it after 7 restarts (decision 28 working). Started again with the product's Start; the seed read back.
• The kept pre-conversion copy was released too early on demo-hp (R-696, found in Part G): the release trusted the unit's refresh time, and the unit's dump was from PostgreSQL 16. Low harm tonight; not fixed (a third release).

Claims in the brief, judged

1. postgres:18 moved its datadir and refuses or warns at /var/lib/postgresql/data — TRUE, and stronger: it REFUSES (exit 1) even an EMPTY volume there (A/A1-images.txt).
2. The safety dump is pg_dump per database with --no-owner --no-privileges, not pg_dumpall — TRUE. For docmost both routes rebuild an identical database (one role, one database); the box uses pg_dumpall (decision 38).
3. Restore brings the old datadir back so the old engine starts — TRUE, by hand (A4) and by the product three times (Part D cases a–c).
4. The R-487 restore of a removed app picks up the kept drive files — FALSE on 0.272.0: nextcloud came back with no env, no database, its files mounted on the guest's root disk — the unit on the DATA drive was never found (R-690, fixed in v0.274.0, proven live).
5. nextcloud needs a rescan for files newer than its backup — TRUE (occ files:scan --all, 0.6 s); now the template's after_load:.
6. Only demo-hp runs docmost — TRUE.

• Also measured: R-657's install LOOP did not recur on 0.272.0 — the install succeeded and the old files were silently unreachable; the choice replaces both shapes.

Part 0 — part0/README.md

Decisions 35, 36 recorded; D3 in STATUS. Last night: demo-felhom's leg did opengist in 20 s at 04:15; its whole-box backup ran at 07:29, three hours after the leg — nothing waited; demo-hp's leg had nothing and reported "steps": null (fixed); no whole-box backup was due there. Found: R-689 — demo-hp's restore test picks the golden template as "the newest settled archive" and fails every 6 h.

Part A — the spike (A/README.md)

Target 18 (decision 37); load from pg_dumpall with zero tolerated errors (decision 38); the check (owners, encodings, roles, extensions, per-table rows) in 0.5 s; the undo after emptying proven by hand; space: the dump is 0.2 % of the datadir — the box's bound is the volume's size × 1.25 + the 2 GB floor.

Part B — controller v0.273.0 (B/)

internal/stacks/pgconvert.go; 17 tests; 9 red-proofs each seen failing (mark missing → refused; marker re-checked before emptying; counts compared; PG_VERSION checked; a cut-off dump; a restart during converting undone; the old copy kept; the release wired; the space check).

Part C — the catalog (C/)

Harness v4 (converts on the bench, moves an 18 mount, writes the mark only when both venues converted); the engine gate's proof clause with 5 decoys + a red-proof; the postgis family judged. Bench: negative control C3 failed; docmost run 1 (384M) converted in 25.0 s, proven, memory_tight 90.9 %; run 2 (512M) converted in 11.0 s, proven, 80.4 %; abort (16 images on the 18 datadir) refuses, as expected — the product's route back is the undo, not that.

Part D — the live proof, the floor, the move (D/)

case (9202, drill catalog, 0.273.0-rc1)	phases	end	PG_VERSION after	seed
happy path	safety-dump → pulling → copying → converting 10 s → starting → verifying	done, 42.1 s	18	read back
(a) the load fails (adminpack, gone in 17+)	… → converting → undoing	undone, 43.6 s	16	read back
(b) unhealthy on 18 (drill probe :3999)	… converting → starting → verifying (90 s) → undoing	undone, 148.1 s	16	read back
(c) SIGKILL 1 s after the volume was emptied	… converting → (restart) → undoing	undone, 40 s after the restart	16	read back

Floor 0.273.0 (then 0.274.0) read back from the hub; both demo boxes healthy within ~25 s. docmost moved in the live catalog at ~14:40 CEST (afd3a60): the engine gate printed ALLOWED … proven on both venues and carries the conversion mark; the Hungarian copy gate caught the RAM line's change on the first push (freeze updated on purpose).

Part E — kept data (E/)

E1 answers above (claims 4, 5). Built: the install choice (409 kept_data_choice), the „Megőrzött adatok" page, the read-only view, the drive-full naming, after_load:. E5 on 9202 (0.274.0-rc2), both languages, all passed: ask (use off without a database copy) → start fresh (126 MB renamed to kept/nextcloud/<date>/) → seed, a file, a unit, a file after it → remove keeping data → use (loaded from the unit on the DATA drive; rescan ran; account + both files back; a never-written file stays invisible) → start fresh again → Load refused while a leftover occupies the folder → Delete refused for a wrong name and for an unlisted path → Delete → Load (files moved back, database loaded, rescan; account + both files back) → a write into the view: Read-only file system. R-692 found and fixed live (two leftovers named „Filebrowser").

Part F

F1 (R-687 [] + the taken step's log): v0.273.0, 2 red-proofs. F2 (R-688): hub v0.125.0 — the delete dialog lists the Cloudflare items to remove by hand; live preview read for demo-hp. F3 (R-655): measured, not moved (see the row).

Part G — the night watch

Read-only on both demo boxes (G/). demo-hp converted docmost BY ITSELF in the automatic leg: off-site copy 04:14–04:18 → leg 04:18:25–04:19:20 CEST → CONVERTED 16 → 18 in 9.898s — the check is equal (2 database(s), 50 table(s), 73 row(s)), PG_VERSION 18 → DONE in 54s; hub summary done=1, steps: [docmost 55.1 s]. Content read back (read-only SELECTs, before 12:41Z / after 02:40Z): users 1 = 1, workspaces 1 = 1, spaces 1 = 1, pages 4 = 4; PG_VERSION 16 → 18; front door 200; all three containers healthy. The whole-box backup was due since the evening and held for its window [04:30, 08:30); it started at 04:30:45, eleven minutes after the leg ended — nothing had to wait (R-687 item 4 still unobserved). demo-felhom: db-dump 02:30, Tier 2 03:30, off-site 04:15, leg nothing to do — hub reads "steps": [] (R-687's fix, live). 9202's release (B5) worked: 13:30:39Z, one hour after a backup whose dump (11:57) came after the conversion (11:13). demo-hp's release was premature (R-696): at 02:30:16Z it removed the 16 copy citing "Tier 1 at 02:20:16Z" — the unit's manifest refresh; its docmost-postgres.sql is 02:15:01Z and says Dumped from database version 16.15.

Teardown

• Machine: 9202 — docmost and nextcloud removed through the product; this session's kept folders deleted through the Kept-data page (the older romm/paperless leftovers stay as found); back on the LIVE catalog (controller.yaml from the saved copy), controller 0.274.0; one empty folder recreated by the file browser's stale bind removed by hand after the loop was understood (R-695). The drill repo reset to live main. demo boxes: nothing touched but the two floors (read-only otherwise); docmost on demo-hp now runs PostgreSQL 18 — the intended outcome.
• Host: bench LXC 9401 destroyed (name checked), its template removed; demo-hp pct list 9201 + 9202 only.
• Hub: global floor 0.274.0 (MinAgent 0.131.0); hub 0.125.0; no customer or appliance created.

Register

Before: 334 rows / 674,532 B. Opened R-689, R-691 (helper), R-693, R-694; opened and closed R-690, R-692; closed R-657; opened R-695 (teardown finding) and R-696 (the night watch); narrowed R-450, R-463, R-469, R-687, R-688, R-691; R-655 measured. After: 339 rows / 682,159 B.