docs: slice 3 Part A live evidence, and two findings (R-583, R-584)

The live proof: the box's own "send test notification" button pressed twice,
74 seconds apart, on demo-hp. Reporting `en` it produced "[Felhom] Test
notification / Dear Customer, ..."; switched to `hu` it produced "[Felhom]
Teszt értesítés / Kedves Ügyfél! ...", byte-for-byte the v0.117.0 literal. The
operator's copy is identical in both, which is the half worth stating.

R-583 (closed, hub v0.118.1): the test mail was the one customer mail that did
not follow the language, and it is the mail an operator would use to CHECK
that the language works. The surface you would use to check a feature is the
one most worth checking first.

R-584 (open, P2): five probe scripts from slice 2's releases B/C/D were still
in the guest's /tmp carrying the controller password INLINE. The rule to
delete them exists, was loaded, and was not followed three times running - so
the rule is not the mechanism. All shredded; whether to rotate the shared demo
password is the operator's call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

documentation/audits/i18n-slice3-2026-09-18/A/README.md

@@ -0,0 +1,92 @@
+# Slice 3 Part A — hub v0.118.0 / v0.118.1, live evidence (2026-09-18)
+
+**Method: endpoint-level**, plus one real e-mail read in the operator's own inbox. No browser on
+DooPlex. Nothing was installed, nothing removed; the deploy endpoint was never touched.
+
+## 1. The wire already worked — measured BEFORE writing any code
+
+Read-only copy of the live `hub.db` (+ `-wal`, or it is hours stale):
+
+```
+demo-felhom  14:12 UTC  language='hu'   controller 0.255.0
+demo-hp      14:09 UTC  language='en'   controller 0.255.0
+drill-r50 / peti-felhom / tester-1      language=<ABSENT>  (0.213.0 / 0.115.0 / 0.245.0)
+```
+
+The positive and the negative control in one read: every box on ≥ 0.247.0 publishes the field, every
+older one sends nothing at all — which is the case the empty default exists for.
+
+## 2. The migration applied
+
+```
+reports.language          present: True
+customer_configs.language present: True
+customer_configs.language = 'hu' for all five customers (the default backfill)
+```
+
+The first reports written by the NEW hub, minutes after the sync, carry the denormalised value:
+
+```
+demo-felhom  id=25717  14:22:59  language='hu'
+demo-hp      id=25718  14:23:00  language='en'
+```
+
+**Rollback, written before the sync:** revert the manifest tag to 0.117.0 and sync. The two columns
+stay and need no undo — v0.117.0 neither reads nor writes them and both carry a DEFAULT, so every
+INSERT the old binary performs still succeeds. Rolling back the image is the whole rollback.
+
+## 3. One real e-mail, both directions, 74 seconds apart
+
+The path is the product's own: sign in to the box, press **Send test notification** on the
+notifications page (`POST /settings/notifications/test`) — which is what a customer's own click does.
+The box POSTs `event_type=test` to the hub, and the hub composes the mail.
+
+**Box reporting `en`** — 14:30:43 UTC, to the household address `drill@felhom.eu`:
+
+```
+Subject: [Felhom] Test notification
+         Dear Customer,
+         This is a test notification from the Felhom monitoring system.
+         Notifications are working correctly.
+         Best regards,
+         Felhom.eu monitoring
+```
+
+**The same button, box switched to `hu`** — 14:31:57 UTC, same address:
+
+```
+Subject: [Felhom] Teszt értesítés
+         Kedves Ügyfél! Ez egy teszt értesítés a Felhom monitoring rendszerből.
+         Az értesítések megfelelően működnek. Üdvözlettel, Felhom.eu monitoring
+```
+
+The Hungarian is byte-for-byte the literal that shipped in v0.117.0 — it was extracted into the
+bundle by script, never retyped.
+
+**The operator's copy is identical in both**, to `admin@felhom.eu`:
+`[Felhom] ✅ demo-hp: teszt / operator channel OK`. That is the half worth stating: the household's
+language changed twice and nothing the operator reads moved.
+
+## 4. What this did NOT prove
+
+- **`message_customer` has no live proof yet.** No box sends it until controller v0.256.0 (Part B).
+  The hub half is covered by tests only.
+- The `test` mail bypasses `FormatCustomerEmail`, so this proof exercises the bundle and the language
+  resolution, **not** the event-mail wrapper. The wrapper's live proof waits for Part B.
+- Nobody looked at these mails in a mail CLIENT. They are plain text; the bytes above are what was
+  sent.
+
+## 5. A lapse, recorded
+
+Earlier in this same session (slice 2 releases B/C/D) I left five probe scripts in the guest's
+`/tmp` — `probeB.sh`, `probeB2.sh`, `probeB3.sh`, `probeC.sh`, `probeD.sh` — and they carried the
+controller password inline. The standing rule says a credential-bearing helper is deleted from
+`/tmp` on both host and guest when done; they sat there for hours instead. Found while cleaning up
+after this proof, by grepping my own litter for secret-shaped strings before deleting it. All are
+now shredded, along with this run's password file on the host, the guest and DooPlex. **This run
+used a file→file password (never an inline literal), which is why its own scripts were clean.**
+
+## 6. State at the end
+
+Hub **0.118.1**, Synced/Healthy, clean startup log. demo-hp back on **Hungarian**. Nothing
+installed, nothing removed, no floor change, no golden. Guest `/tmp` clean.

documentation/audits/i18n-slice3-2026-09-18/A/deploy.log

@@ -0,0 +1,25 @@
+=== BEFORE (rollback point):
+gitea.dooplex.hu/admin/felhom-hub:0.117.0
+sync=Synced health=Healthy rev=20aafc3dec2008f8f4697129bcf95bd354edef20
+ROLLBACK LINE (written BEFORE the sync):
+  git revert the manifest tag to 0.117.0, commit, push, sync.
+  The two new COLUMNS stay and need no undo: v0.117.0 neither reads nor writes
+  them, both carry a DEFAULT, so every INSERT the old binary performs still
+  succeeds. SQLite cannot drop a column before 3.35 and the data is worth
+  keeping. Rolling back the IMAGE is the whole rollback.
+=== AFTER:
+sync=Synced health=Healthy rev=9167cf53afbc9cf62fdc92ca248d37a493e915e7
+image=gitea.dooplex.hu/admin/felhom-hub:0.118.0
+=== startup log:
+2026/09/18 16:22:18 [INFO] Staleness checker initialized: 2 ok, 0 stale, 2 down (node_stale after 45m0s, node_down after 1h30m0s)
+2026/09/18 16:22:18 [INFO] Host staleness checker initialized: 2 ok, 0 stale, 0 down (stale/down left unseeded → first Check emits; host_stale after 45m0s, host_down after 1h30m0s)
+2026/09/18 16:22:20 [INFO] Host capability checker initialized: 2 ok, 0 degraded (degraded left unseeded → first Check emits)
+2026/09/18 16:22:20 [INFO] Host leaf checker initialized: 2 host fingerprint(s) seeded
+2026/09/18 16:22:20 [INFO] Host disk checker initialized: warn=90% crit=95%, 2 ok seeded, 0 already-breached left unseeded (first Check emits)
+2026/09/18 16:22:20 [INFO] Storage fill checker initialized: warn=90% crit=95%, 6 ok seeded, 0 already-breached left unseeded, 2 root-backed excluded
+2026/09/18 16:22:20 [INFO] Host mgmt-plane checker initialized: 0 host heal-state(s) seeded
+2026/09/18 16:22:20 [INFO] Host OOB checker initialized: 0 host(s) seeded degraded
+2026/09/18 16:22:20 [WARN] [offsite] tester-1: controller sends no last_success — staleness degraded to the last-ATTEMPT anchor (pre-v0.181.0 controller; a persistently failing tier will not go stale here until it upgrades)
+2026/09/18 16:22:20 [INFO] Offsite checker initialized: fill warn=90% crit=95%, stale after 48h0m0s, 4 ok-seeded
+2026/09/18 16:22:20 [INFO] Listening on :8080
+2026/09/18 16:22:20 [INFO] deadline-check: next run at 2026-09-19 05:00 CEST (in 12h37m40s)

documentation/backlog/OPEN-ITEMS.md

@@ -758,6 +758,8 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
 | **R-580** | **[P3-LOW] `curl -w '%{redirect_url}'` prints Basic-auth credentials back into the session transcript.** FOUND 2026-09-18 raising the fleet floor to 0.255.0: the hub's `/configuration/global-floor` form was driven with `--netrc-file` and `-w 'POST http=%{http_code} -> %{redirect_url}'` to read the `flash=floor_set` confirmation. curl re-injects the credentials into the redirect URL it formats, so the operator's hub password appeared in the session's own output. **Nothing was written to a file and nothing was committed** — the exposure is the transcript. **The general form, which is the part worth keeping: a curl format variable can carry a secret that never appeared in the command.** `%{redirect_url}`, `%{url_effective}` and `%{referer}` all reconstruct the request URL, and `--netrc`/`-u` credentials go back into it. **Fix shape:** print `%{http_code}` alone and read the destination from `-D-` headers, or pipe the format output through a redactor. Recorded in the `felhom-build-deploy` skill so the next hub session does not repeat it. | **READY - rank P3-LOW; owner: CC** |
 | **R-581** | **[P2-MED] `ORDER BY received_at` cannot answer "the newest report" — the column has SECOND granularity.** FOUND 2026-09-18 building hub v0.118.0 (R-558): `Store.CustomerLanguage` read the household's language from the newest report ordered by `received_at`, and `TestNewestReportedLanguageWins` failed — four reports written in the same test tick all carry the same `datetime('now')` string, so the "newest" was whichever row SQLite felt like returning. On a real box the same shape appears whenever two reports land in one second (a settle burst, a restart race), and the symptom would have been a household switching language on their dashboard and getting the old language back at random. FIXED in the same release by ordering on the autoincrement `id`, which is the real insertion order. **What is still open, and it is the reason this is a row rather than a note: `GetCustomers()` has the same shape** — `INNER JOIN (SELECT customer_id, MAX(received_at) …)` with no tie-break — and it is what the whole operator dashboard and `countBoxesBelowFloor` read. A same-second tie there picks an arbitrary report's health, version and vitals. Not observed in the wild; not looked for either. **Fix shape:** tie-break every newest-report query on `id DESC`, or give `reports` a monotonic ordering column and use it everywhere; then a test that writes two reports in one tick and asserts which one wins. | **READY - rank P2-MED; owner: CC** |
 | **R-582** | **[P3-LOW] An English copy guard written from the Hungarian one matches ordinary words instead of the claim.** FOUND 2026-09-18 extending `hub_copy_gate.py` for R-558. The Hungarian retrieval stems ARE the claim — `visszaállíthat` is one word meaning *can restore* — so the guard matches a stem. Ported to English as the bare stems `restor`/`recover`, the gate convicted **141 honest sentences** on its first run: "Disaster recovery has started", "Your server can be reached again", the name of the Restore page, and every JSON KEY containing the word. English splits the modal from the verb, so the English form of the claim is a PHRASE. FIXED in the same session: the patterns carry the modal, and the gate scans bundle VALUES only, never keys. **Then the decoy suite caught the fix being too narrow** — a planted "can STILL be restored" walked through a pattern written for "can be restored", which is R-299 again in another language; every pattern now admits an adverb. **The general form worth keeping: a guard ported between languages must be re-derived from what the claim IS in the new language, not translated word for word — and a guard that convicts 141 true sentences is worse than no guard, because it earns an allowlist entry per sentence and then nobody reads it.** | **CLOSED 2026-09-18 - hub v0.118.0 (recorded for the lesson; three decoys incl. an innocent control)** |
+| **R-583** | **[P3-LOW] The test-notification mail was the one customer mail that did not follow the language — and it is the mail an operator would use to CHECK that the language works.** FOUND 2026-09-18 during hub v0.118.0's own live proof: I went to press "send test notification" for the English demo box, read `sendTestEmail` first, and found it composes its own hardcoded Hungarian subject and body instead of going through `FormatCustomerEmail`. Every other customer mail had been localised. **The general form worth keeping: the surface you would use to CHECK a feature is the one most worth checking first — a broken instrument that reports success is worse than a broken feature.** FIXED and shipped as hub v0.118.1 in the same session: the two sentences extracted byte-for-byte into `mail.test.subject` / `mail.test.body`, red-proofed against the hardcoded version, and proven live in both languages 74 seconds apart on demo-hp. | **CLOSED 2026-09-18 - hub v0.118.1** |
+| **R-584** | **[P2-MED] Credential-bearing probe scripts were left in a live guest's `/tmp` for hours, across three releases.** FOUND 2026-09-18 while cleaning up after slice 3's live proof: `probeB.sh`, `probeB2.sh`, `probeB3.sh`, `probeC.sh` and `probeD.sh` were still in demo-hp guest 9201's `/tmp` from slice 2's releases B, C and D earlier the same session, and all five carried the controller password INLINE (`-d "password=$PW"` with the value substituted). The standing rule in `.claude/rules/ui-hungarian.md` says exactly this: *delete any credential-bearing helper from `/tmp` (host AND guest) when done*. They were not deleted at the end of each phase; they were found by grepping my own litter for secret-shaped strings before removing it, which is a check that happened only because a later phase went looking. All five are shredded. **Why it is a row and not a note: the rule existed, was loaded, and was still not followed three times running — so the rule is not the mechanism.** The password is the shared demo one (Tier-0 boxes, `~/.config/credentials`), unchanged; rotating it is cheap and is the operator's call. **Fix shape:** never inline a credential in a helper — write it to a mode-0600 file and read it with curl's `@file` form, which this run did and which is why this run's own scripts were clean; and make the last act of a phase that pushed a script to a box the `shred -u` of it, the same way R-320 made evidence-copying the last act of a phase. | **READY - rank P2-MED; owner: CC (the mechanism), operator (whether to rotate)** |
 | **R-537** | **[P1-HIGH] The app-backup page labels the tier-1 backup „DB + Konfig + Adatok" and prints the app's data-drive size next to it — but the tier-1 unit contains NO drive-side app data at all.** MEASURED 2026-09-16 on the drill box (fresh install, controller 0.243.0, one drive, tier 2 and tier 3 both „Nincs beállítva"): five photos (3 000 000 B) were uploaded into Nextcloud through its own WebDAV interface, then the customer-visible „Mentés most" was pressed (`POST /api/backup/run` → 200, the unit grew 25 337 B → 978 MB). The resulting unit's `manifest.json` lists `db-dumps` + three **docker volume** dumps and nothing else; listing the 781 MB `nextcloud_nextcloud_html.tar` (29 346 entries, positive control `version.php` = 3 hits) gives **`Fotok` = 0 and `nyaralas` = 0**, and `./data/` is the empty bind-mount point. A `find` over the whole `backups/` tree for `*appdata*` / `*Fotok*` returns nothing. The page nevertheless renders „1. mentés … DB + Konfig + Adatok" and „Nextcloud Adatlemez 65.1 MB" — a size measured on exactly the data it does not copy (`internal/web/handlers.go:1176-1178`, `BackupContents`). **This is a truth defect, not a design defect:** `07-backup-architecture.md` §6.2 places nextcloud's file leg at **Tier 2 and Tier 3 only**, and its „[FACT] What the whole-guest tiers do NOT carry" says `mp8 /mnt/felhom-drives` is out of vzdump scope (confirmed live: „excluding bind mount point mp8 … (not a volume)"). So on a one-drive box with no off-site tier — the state every fresh install starts in — the household's files are in **no backup**, while the page says „Adatok". Same family as R-517/R-518. **Fix shape:** render tier-1 contents from the capture set actually written (`ComputeCaptureSet`), so a unit with no file leg reads „DB + Konfig" and the drive size is not shown beside it; and say on the page that the app's files need tier 2 or tier 3. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-f10.txt`. **CLOSED 2026-09-16 — controller v0.244.0, proven live.** The contents label is computed PER TIER from what that tier captures: Tier 1 says „Adatok" only when the app's data really is in the volumes the unit captured, and a class-A app carries one sentence saying where its files ARE protected. Proven on demo-hp through the page the customer opens: Paperless-ngx reads „1. mentés … DB + Konfig" with „Az alkalmazás fájljait a távoli másolat (és a második meghajtó) védi …", while its „2. mentés" row still reads „DB + Konfig + Adatok". Red-proof: restoring the old app-shaped label fails `TestAppBackupRows_Tier1LabelDoesNotClaimFilesItCannotHold`. **RE-PROVEN 2026-09-16 on a FRESH box** (installed from the built ISO 1.28.0, controller 0.244.0, off-site on by default): the Nextcloud row read „1. mentés … DB + Konfig" with the new sentence, „2. mentés … Nincs 2. (off-drive) másolat", „3. mentés Sikeres restic → …your-storagebox.de"; „DB + Konfig + Adatok" appeared ZERO times while the local unit held no file leg. | **CLOSED 2026-09-16 — controller v0.244.0 (proven live on demo-hp)** |
 | **R-538** | **[P1-HIGH] A tier-1 app restore reports plain success and leaves Nextcloud listing files whose bytes were never in the backup — and it destroys the app's own trash, the customer's last copy.** MEASURED 2026-09-16 on the drill box, F10 („a child deletes the photo folder"): the five photos were deleted through Nextcloud (DELETE 204, PROPFIND 404), then restored through the page exactly as a customer would (`POST /backup/restore` `stack_name=nextcloud` `snapshot_id=helyi` → 302, finished in **35 s**, „A(z) nextcloud: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult."). Afterwards the folder is back and **lists all five photos**, and **none of them opens**: `GET nyaralas-1..5` = 404 / 503×4 with `Sabre\DAV\Exception\NotFound`, while the positive controls at the same moment pass (`status.php` 200, WebDAV PUT 201, GET 200). Cause: the replayed MariaDB dump (11:01:45Z) knows the photos, the bytes live on `mp8` and were never captured (R-537). **Worse:** the bytes were still on the drive in Nextcloud's own trash (`appdata/nextcloud/admin/files_trashbin/files/Fotok.d1789556707/nyaralas-1..5.jpg`, all five present) and the restored database no longer references them — the trash listing comes back **empty**, so „restore from trash", the one route that would have worked, is gone. The customer is left with five unopenable photos, a success message, and no warning. **Fix shape:** before replaying a database whose app has an uncaptured file leg, refuse or warn („ennek az alkalmazásnak a fájljai nincsenek ebben a mentésben — a visszaállítás után a fájlok hiányozni fognak"); and never present a DB-only restore of a class-A app as a complete one. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-f10.txt`. **CLOSED 2026-09-16 — controller v0.244.0, proven live.** A unit restore refuses before anything is touched when the unit cannot return the app's drive-side files, and names the route that can. Fired live on demo-hp: `POST /backup/restore` for paperless-ngx → 302 with „Ez a mentés nem tartalmazza az alkalmazás fájljait, ezért nem állítjuk vissza az adatbázist föléjük — a fájlok így a helyükön maradnak. A fájlok a távoli másolatból állíthatók vissza …", and the app read `running` before AND after, so nothing was stopped and no trash was made unreachable. The database-and-settings-only path exists as a separately worded second step. Red-proof: disabling the guard fails `TestUnitRestore_RefusesWhenTheUnitCannotHoldTheFiles`. **RE-PROVEN 2026-09-16 on a FRESH box, and this time the refusal had somewhere to point:** after five photos were deleted, `POST /backup/restore` was refused with „…a fájlok így a helyükön maradnak. A fájlok a távoli másolatból állíthatók vissza: … „Teljes visszaállítás (fájlok + adatbázis)"", the app read `running` before AND after, and the wastebasket was untouched. The off-site route then returned all five photos — 200 with the exact uploaded sizes and sha256 IDENTICAL to the originals, 5/5, with a negative control. Evidence: `audits/evidence-backup-promise-2026-09-16/phaseE-photos.txt`. | **CLOSED 2026-09-16 — controller v0.244.0 (proven live on demo-hp)** |
 | **R-525** | **[P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured.** Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. **What it needs:** a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | **READY — rank P3-LOW; owner: CC (spike)** |