diff --git a/documentation/audits/answers-2026-10-12/disk-Z6A07P2G.txt b/documentation/audits/answers-2026-10-12/disk-Z6A07P2G.txt
new file mode 100644
index 00000000..f36b4cfd
--- /dev/null
+++ b/documentation/audits/answers-2026-10-12/disk-Z6A07P2G.txt
@@ -0,0 +1,21 @@
+# Disk ST3000VX010 S/N Z6A07P2G on DooPlex — read 2026-10-12 ~14:30Z, then three changes after the operator's yes in chat
+alerts (Alertmanager active): DiskUncorrectableSectors and DiskOfflineUncorrectable, labels component=disk,
+  device=ata-ST3000VX010-2E3166_Z6A07P2G, severity critical, receiver email-notifications; expressions
+  max by (device)(smartctl_device_attribute{attribute_name="Current_Pending_Sector"|"Offline_Uncorrectable",raw}) > 0;
+  startsAt 2026-10-12T11:49:06Z (the Prometheus restart of the reloader change; first fired 2026-10-10 08:21Z).
+SMART history (Prometheus, 2026-10-09 → 2026-10-12, 6 h step): Current_Pending_Sector 352 flat, Offline_Uncorrectable 352 flat, Reallocated 0.
+device: /dev/disk/by-id/ata-ST3000VX010-2E3166_Z6A07P2G -> sdg; by-path pci-0000:00:1f.2-ata-6; kernel ata10; sdg2 ext4 UUID 9cf20140-…; NOT mounted.
+twin: ST3000VX010 S/N Z6A070XZ = sdd, mounted /mnt/3_hdd — in use.
+fstab line 16: "# DISABLED 2026-08-14, drive failed (… Z6A07P2G): UUID=9cf20140-… /mnt/4_hdd …" (commented out).
+k3s.service.d/wait-for-mounts.conf: RequiresMountsFor=/mnt/2_hdd /mnt/ssd_2 /mnt/1_hdd /mnt/3_hdd (no 4_hdd).
+Longhorn node dooplex, before: disk HDD4 path /mnt/4_hdd/data/longhorn allowScheduling false, 0 replicas, Ready False
+  (spec kept here for undo: {"allowScheduling":false,"diskType":"filesystem","path":"/mnt/4_hdd/data/longhorn","storageReserved":536870912000,"tags":["hdd","hdd4"]}).
+backup scripts (live and role), manifests, PVs, pod hostPaths: no path on /mnt/4_hdd (only comments).
+SMART exporter: admin-system/smartctl-exporter.yaml line 63 names the disk by id (to remove after the pull-out).
+CHANGE 1 (yes): silence 1e18c10a-9398-4a67-9769-03e5405fce1b, matcher device=~".*_Z6A07P2G", until 2026-11-11T14:32:06Z,
+  comment "disk Z6A07P2G evacuated 2026-10-10, waiting for physical removal". Read back: both alerts "suppressed".
+  Control: of the 7 disks Prometheus sees (smartctl_device_temperature), the pattern matches exactly 1.
+CHANGE 2 (yes): `kubectl patch nodes.longhorn.io dooplex` disks.HDD4=null. Read back: spec disks + status disks = Crucial-SSD-240G, HDD2,
+  default-disk; node conditions identical to before; 78 Longhorn volumes, all healthy.
+CHANGE 3 (yes, R-231): backup-manifests.sh sends the token as a header (homelab-manifests 1943a70, installed by the role, live file = git).
+  Read-only proof: git ls-remote of the private repo with the header -> HEAD d0767fe; without -> "could not read Username" (refused).
diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md
index 2df5a65b..cbcf0d1c 100644
--- a/documentation/backlog/OPEN-ITEMS.md
+++ b/documentation/backlog/OPEN-ITEMS.md
@@ -152,7 +152,7 @@ stopping line that lies.
 | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). | operator: print the sheet | Operator: print + store the sheet; then close | operator |
 | **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC |
 | **R-127** | Backup & restore | P3 | **The catalog's `data_key: true` flag is UNRELIABLE — at least four data-encrypting keys the catalog itself labels as encryption keys are unflagged; and the O4 restore path can regenerate a DB password that then does not match the restored data directory** | **READY (S/M)** **2026-10-06: leg (a) PUSHED** to the live catalog (`c265b37`). **2026-10-06 (burn-down night, later): leg (b) NEEDS A DESIGN.** `07` §7.4 sets no direction; refusing the restore without the DB password, or `ALTER USER` after it, each change restore behaviour on customer data. | — | **Found by D5's Part 0, and it is why D5's boundary is `type: secret` rather than `data_key`.** Two separable legs. **(a) The misclassification.** Only 5 fields across 4 apps set `data_key: true` (`adventurelog/SECRET_KEY`, `homebox/HBOX_AUTH_API_KEY_PEPPER`, `papra/AUTH_SECRET`, `sparkyfitness/{API_ENCRYPTION_KEY,BETTER_AUTH_SECRET}`), yet `n8n/N8N_ENCRYPTION_KEY` („Titkosítási kulcs"), `wanderer/POCKETBASE_ENCRYPTION_KEY` („Adatbázis titkosítási kulcs"), `calcom/CALENDSO_ENCRYPTION_KEY` and `bookstack/APP_KEY` are unflagged — the catalog's own Hungarian labels contradict the flag. **D5 makes this non-urgent but not harmless:** everything `type: secret` now travels, so the keys DO reach the drive; what stays wrong is the **fail-closed gate**, which only refuses for `data_key` names — so if one of these is missing from both sources the restore proceeds onto data it cannot decrypt instead of refusing. Fix = flag them (`app-catalog-felhom.eu`, a catalog-only change) + a gate/test that the flag set and the label set agree. **(b) The regenerated-DB-password trap.** `internal/backup/restore_unit.go` O4 generates a replacement for any missing non-data-key secret. Proven on `postgres:16-alpine`: with PGDATA restored from the volume tar, `POSTGRES_PASSWORD` is ignored (initdb skipped), so the app fails over the compose network while the dump replay still succeeds through the container's local **trust** socket — success reported, data unreachable. v0.188.0 corrected the WARN's false claim that *"stored data is unaffected"* and scoped it, but did **not** add a guard: D5 shrinks this to the rare case (the secret was empty at capture AND absent from the guest). Real fix = either treat a DB password as fail-closed like a data key, or `ALTER USER` to the regenerated value after the volume restore. 18 DB/root-password fields are in scope; MariaDB fails loudly instead (Access denied), which is the safer half | CC |
-| **R-231** | Backup & restore | P3 | **`/opt/backup/scripts/` on DooPlex is unversioned host state** — found 2026-08-06 while adding the auto-memory store to the backup set. No repository tracks the scripts that protect the recovery chain, so the edit made that day (`CLAUDE_MEMORY_DIR` in `backup-config.sh`, multi-path restic call in `backup-data.sh`) exists only on the box. This is the same class the part-2 session was closing, found inside the fix for it; the change is transcribed in `felhom.eu/workspace/README.md` so it is at least *recorded*. **Two related facts, both understating current safety:** the backup destination (`/mnt/5_hdd/backup`) is on the **same physical disk** as the workspace it protects, and the DooPlex backup set has **no off-site leg** (`sync-hetzner-backups.sh` is jarrs.eu and pulls *from* Hetzner *to* DooPlex). Bringing a root-owned production backup script under version control, and deciding what installs it, is its own scoped change. | **VERIFY** — **2026-10-12: in git and installed from it** (operator yes in chat, operator in the session): homelab-manifests `fdd324b`, `homelab-ansible/roles/dooplex_backup/` (11 scripts + the timer units, byte-identical to the live ones; no secret in any file; `install.sh` dry run / `--apply`). The install changed only the units' mode (744 → 644). Baseline: last night's run `Result=success`, exit 0 (`audits/answers-2026-10-12/s20-backup-scripts.txt`). **Seen by the commit's security review, NOT changed (a DooPlex behaviour change needs the operator's word):** `backup-manifests.sh` puts the Gitea token into the clone URL for the mirror run (visible in the process list for those seconds) and strips it from the stored remote afterwards; no token VALUE is in any file, and the repo is private (anonymous API read = 404). A header-based fetch (`git -c http.extraHeader=…` from the token file) would remove it — one line in the decision list. **READY** — operator ruling 2026-10-12 (S20, `09` §3 decision 215): yes — the scripts into a repo + an install script; the operator present for the install. **READY** — owner Viktor | — | 2026-10-13: read `dooplex-backup.service` Result/ExecMainStatus after the 03:10 run; success → close | CC |
+| **R-231** | Backup & restore | P3 | **`/opt/backup/scripts/` on DooPlex is unversioned host state** — found 2026-08-06 while adding the auto-memory store to the backup set. No repository tracks the scripts that protect the recovery chain, so the edit made that day (`CLAUDE_MEMORY_DIR` in `backup-config.sh`, multi-path restic call in `backup-data.sh`) exists only on the box. This is the same class the part-2 session was closing, found inside the fix for it; the change is transcribed in `felhom.eu/workspace/README.md` so it is at least *recorded*. **Two related facts, both understating current safety:** the backup destination (`/mnt/5_hdd/backup`) is on the **same physical disk** as the workspace it protects, and the DooPlex backup set has **no off-site leg** (`sync-hetzner-backups.sh` is jarrs.eu and pulls *from* Hetzner *to* DooPlex). Bringing a root-owned production backup script under version control, and deciding what installs it, is its own scoped change. | **VERIFY** — **2026-10-12 evening: the token left the clone URL** (operator yes in chat; homelab-manifests `1943a70`, installed by the role): `backup-manifests.sh` sends it as an `http.extraHeader` through `GIT_CONFIG_*` env; read-only proof: `git ls-remote` of the private repo with the header → HEAD, without → refused. The 2026-10-13 03:10 run is the test of both changes. **VERIFY** — **2026-10-12: in git and installed from it** (operator yes in chat, operator in the session): homelab-manifests `fdd324b`, `homelab-ansible/roles/dooplex_backup/` (11 scripts + the timer units, byte-identical to the live ones; no secret in any file; `install.sh` dry run / `--apply`). The install changed only the units' mode (744 → 644). Baseline: last night's run `Result=success`, exit 0 (`audits/answers-2026-10-12/s20-backup-scripts.txt`). **Seen by the commit's security review, NOT changed (a DooPlex behaviour change needs the operator's word):** `backup-manifests.sh` puts the Gitea token into the clone URL for the mirror run (visible in the process list for those seconds) and strips it from the stored remote afterwards; no token VALUE is in any file, and the repo is private (anonymous API read = 404). A header-based fetch (`git -c http.extraHeader=…` from the token file) would remove it — one line in the decision list. **READY** — operator ruling 2026-10-12 (S20, `09` §3 decision 215): yes — the scripts into a repo + an install script; the operator present for the install. **READY** — owner Viktor | — | 2026-10-13: read `dooplex-backup.service` Result/ExecMainStatus after the 03:10 run; success → close | CC |
 | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC |
 | **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC |
 | **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC |
@@ -243,13 +243,14 @@ stopping line that lies.
 | **R-920** | Business & legal | P4 | **The Messenger „Gyakori kérdések" automation does not exist for this Page, so `COPY.md` §5 — four questions with answers condensed from `gyik.html` — has nowhere to go.** FOUND 2026-10-09 (Page details task, `audits/facebook-page-details-2026-10-09/`). SEARCHED, not assumed absent: the create-automation catalogue („Az összes automatizálás") holds exactly THREE templates — Automatikus válasz, Távolléti üzenet, A megválaszolatlan üzenetek azonosítása (`B5-no-faq-template-all-three.jpg`); the template search for „kérdés" answers **„Nincs a keresésnek megfelelő automatizálási sablon."** while the POSITIVE CONTROL „üzenet" returns two, so the search works and the term genuinely misses; the existing instant-reply automation carries only channel, message and media — no FAQ and no quick replies; and the business-portfolio settings have no messaging/FAQ entry. The copy is written, sourced line by line to `gyik.html` and committed as `marketing/facebook/COPY.md` §5, ready to paste unchanged the day the feature appears. **Same shape as R-917** (§2 has nowhere to go), and the same cause: Meta removed a Page field this project had planned copy for. **Options for the operator:** (a) leave §5 unused until Meta brings the feature back; (b) fold the four answers into the Messenger welcome message (§3) — it holds 500 characters and today uses 120, so one or two would fit, not four; (c) publish them as a pinned FAQ post once the app is Live (R-915); (d) ask Meta support whether the FAQ automation still exists for this Page type. Recommended (a) with (c) later — the welcome message stays short, and the website's own `gyik.html` already answers these. | **DEFERRED — operator chose (c) on 2026-10-09:** park the text and publish it as a post once posting starts. Not a defect, and not waiting on CC | nothing — **UNBLOCKED 2026-10-09**: R-915 is closed, the app is Live, so posting can start whenever the operator wants | When posting starts: publish `COPY.md` §5 (the four Messenger FAQ answers) as one later post. If nothing is done: the text stays in `COPY.md` unused, which the operator has accepted | operator |
 | **R-930** | Business & legal | P3 | **The site's primary call-to-action fails WCAG AA: white on `--blue` #0083D8 measures 4.01:1 at 16px/600, against a 4.5:1 floor.** MEASURED 2026-10-10 in the browser, on the LIVE home page and on the redesign preview: identical, so this is **pre-existing and site-wide**, not a regression of the redesign. It is the „Jelentkezem tesztelőnek" button and every `.cta-button` on every page. `.cta-button-secondary` is fine (7.17:1). Three fixes, all needing the operator because each touches brand: (a) darken `--blue` until the ratio clears 4.5 (changes a token used across hub, controller and website); (b) make the label 700 weight at 18.66px+, which moves it into WCAG's „large text" band where 3:1 applies and 4.01 passes, at the cost of a bigger button everywhere; (c) accept it and record the decision. **Not changed unilaterally**: `felhom-web-design` §6 puts a new accent colour behind the operator's word. | **WAITING-ON-OPERATOR** — filed 2026-10-10 by the home-page redesign session; evidence `audits/redesign-2026-10-10/AUDIT.md` | operator's choice between (a), (b), (c) | pick one of the three | operator |
 
-## Process & tooling — 16 rows (P3 3, P4 13)
+## Process & tooling — 17 rows (P3 4, P4 13)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
 | **R-733** | Process & tooling | P3 | **[P3-LOW] The test bench has NO swap and the boxes have 512 MiB — so a box proof can pass on swap where the bench fails, and nobody records whether a customer guest has swap.** MEASURED 2026-09-30 (R-732): immich's first start was OOM-killed 61–104 times on the bench (swap 0) and passed on 9202 by swapping ~108 MB; the bench given 512 MiB swap passed too. demo-hp 9201, 9202 and demo-felhom 9201 all read `swap: 512`; the golden's guest config is not recorded in its bake evidence, so a customer guest's swap is NOT measured. The harness's memory watch judges `anon` against the limit and never reads `memory.swap.current`. **Needs:** the golden's `swap` read and recorded; the box walk and the harness report `memory.swap.peak` beside `anon`; a decision whether proofs run with swap off (the stricter venue, as R-732's fix was proven). | **READY — rank P3-LOW; owner: CC (harness + golden evidence)** **2026-10-06 night: NARROWED** — the harness records each container's `swap_peak` and the venue's swap (catalog branch `night-held-2026-10-06`, `3f4611c`; SwapRecorded red-proved; bench SwapTotal 0 kB, 9202 524288 kB). LEFT: the golden's swap in its bake evidence, the box walk's record, and whether proofs run with swap off. | — | — | CC |
 | **R-887** | Process & tooling | P3 | **Some CI jobs are never run, and Gitea fails them ~10–13 minutes later with no log.** Seen 2026-10-05: felhom.eu job 1361 (commit `1122b5c`) and felhom-controller job 1357 (`114ff27`): every step reads `failure`, including the first fetch, the log API answers `file does not exist`, and the runner pod's log has no `task` line for them (its task ids are job id + 1). A re-run through the API ran the controller job normally (success in 32 s) but the felhom.eu job was again never picked up and failed after ~12 min. The runner pod (`gitea-system/act-runner`, image `felhom-act-runner:0.1.0`) had restarted 5 times ~142 min earlier, around the Longhorn instance-manager restart (R-882). Suspected, NOT measured: a stale runner registration claims jobs it never runs — the session's Gitea token cannot list runners (`read:admin` scope). Consequence: a red CI verdict that is not about the code, and **no failure mail** (the alarm step never runs either), so only the pull check sees it. **CORRECTED 2026-10-05 18:21 (operator's screenshot of Gitea → Site Administration → Runners): ONE runner only — ID 2, `felhom-gates-runner`, v0.6.1, label `felhom-gates`, Idle, last online „now". There is no old registration; the stale-registration guess (this row's first text and the reviewer's) was WRONG.** **RE-DIAGNOSED 2026-10-05 (round 2), from the logs that survive:** (1) **„lost in a runner restart" does NOT fit** — the runner pod last restarted 13:24:42Z (`restartCount 5`, all around the 13:20Z Longhorn restart), the lost attempts started 1.5–2.5 h later. (2) **FOUR attempts were lost, not two:** controller job 1357 (start 15:05:41Z → failed 15:18:38Z), felhom.eu job 1359 (15:15:37 → 15:28:38 — the previous session blamed that one on the BusyBox fault; the runner never ran it), job 1361 (15:33:21 → 15:43:38) and its API re-run (15:46:48 → 15:58:38). None has a `task` line in the runner log; every one was failed at a :38-second mark on a 5-minute step, 10–13 min after it was handed out — **the shape of Gitea's periodic „zombie task" stop** (a task assigned to a runner that never reports is failed after ~10 min; no log exists because none was written). (3) The runner's task ids are NOT job id + 1 (the controller re-run was task 1363). (4) **Gitea's own log for the window is gone** — the pod log starts 16:16:32Z (rotated), so the assignment side cannot be read. **Likely mechanism, NOT proven:** the runner's fetch-task request timed out on its side after Gitea had already assigned the task, so the task was orphaned. In that same hour this session polled Gitea's jobs API hard (15 pages every 15 s per wait loop) and Gitea logged „slow" requests — a plausible load cause, and the session's own. Mitigation taken: the session's CI waiter now polls once a minute. **Nothing changed on DooPlex.** **MECHANISM SEEN 2026-10-05 17:15–17:28Z, with Gitea's own log (round 2):** catalog run 1368 (`4828dc7`) — 17:15:14 the job is marked started; 17:15:16 `router: slow POST /api/actions/runner.v1.RunnerService/FetchTask for 10.42.0.42 (the runner), elapsed 3192ms`, then `UpdateRepoRunsNumbers … context canceled` and `GetActionWorkflow: EOF` — **the runner abandoned its fetch after Gitea had assigned the task**; the runner log has no line for task 1371; 17:28:39 `actions/clear_tasks.go:174 stopTasks() [W] Cannot transfer logs of task 1371` — Gitea's zombie-task stop. **The load at that minute:** an outside crawler (216.73.216.78) walking commit pages and `archive/*.tar.gz`, and THIS session's CI waiter, whose 15-page job listings took 13–31 s each. An API re-run passed in 7 s. **Done in-session:** the waiter now asks `GET …/actions/runs?head_sha=<sha>` once a minute (1 s). **Not done (DooPlex, the operator's):** the runner's fetch timeout and Gitea's exposure to the crawler. **-- 2026-10-08 (seen):** felhom-controller job 1549 (`05e1292`, a REPORT/CONTEXT-only commit) started 08:29:46Z and ended 08:43:17Z with all three steps `failure` at the same second and no log (logs endpoint HTTP 500); the same tree passed `controller_gates.py --fast` locally; a re-run (`POST …/actions/runs/1549/rerun`) completed `success`. A lost job inside the watch window. **SEEN AGAIN 2026-10-10, and this time TWO JOBS IN TWO REPOS DIED AT THE SAME INSTANT — which is new information about the cause:** app-catalog-felhom.eu job **1631** (`b7f0f7c`, started 10:31:54Z, three steps) and felhom.eu job **1632** (`970616b7`, started 10:34:17Z, seven steps) both have EVERY step `failure` at **10:45:58Z with 0 s each** — the same wall-clock second — after ~14 and ~11.5 minutes in set-up with nothing run, and both logs answer `file does not exist`. A per-job problem cannot do that; one runner event took both. The SAME TREES passed on the next push (catalog job **1633**, `8ea72aa`, success in the normal time), and a shallow clone with no sibling — the CI shape — runs all nine applicable catalog gates green by hand. So this is the flake, not the code. **That makes the DUE-CHECK's premise false: jobs HAVE been lost since 2026-10-05 16:00Z.** **THREE jobs were lost on 2026-10-10 — 1631, 1632 and 1634 — and ALL THREE ended at :45:58 past the hour** (10:45:58Z, 10:45:58Z, 12:45:58Z) after sitting 10.5–14 minutes in set-up. A job that is simply never picked up would end at an arbitrary second; ending at the same second-of-the-hour twice two hours apart points at a periodic sweep or a fixed-deadline timer, not at the job. Between the lost ones the runner DID work: catalog job **1633** started 12:35:06Z and succeeded normally. Worth knowing for the diagnosis: the runner pod was restarted earlier the same day (`audits/release-2026-10-10/ci-runner-restart.txt`). **Next step that would settle it:** read the runner pod's own log around an HH:45:58 and see what it says about the task id (job id + 1 — 1645, 1646, 1648 here), and look for a 10-minute or hourly timer in its config. | **WATCHING** — **2026-10-12 12:50Z, my own change broke pushes, fixed at once:** the 2 r/s limit refused a hub image push with 503 (DooPlex and the home boxes reach `gitea.dooplex.hu` via the home public address 37.191.56.193). homelab-manifests `d0767fe`: that address added to the whitelist; a new Ingress `gitea-registry` serves `/v2` without any limit (boxes anywhere pull in parallel). The rebuild then pushed. The security review noted that `/v2` login attempts are not rate-limited — as before 2026-10-12, when nothing was. **WATCHING** — **2026-10-12 12:17Z: the S21 change is LIVE on DooPlex** (operator yes in chat; homelab-manifests `ad8618f`): runner `fetch_timeout` 5s → 30s (`/config/config.yaml`, read in the running process); Gitea Ingress `limit-rps 2` (burst 10), LAN not limited. Measured before: the crawler 20 546 requests in 24 h, up to 475/min; first minute after: crawler 7×503, our traffic 0×503 (`audits/answers-2026-10-12/s21-runner-and-ratelimit.txt`). Closes if no job is lost by 2026-10-19 (DUE-CHECKS). **READY** — operator ruling 2026-10-12 (S21, `09` §3 decision 216): yes — raise the act-runner fetch timeout and rate-limit the public Gitea pages the crawler walks (DooPlex, each change after the operator's yes in chat). **2026-10-12 dated check read: NOT met** — 4 jobs lost 2026-10-10/11 (controller 1639, felhom.eu 1645, 1646, 1672; each re-run once, each passed); re-dated 2026-10-19, measured over the 7 days after the S21 change. **OPEN** **DATED CHECK 2026-10-12 (DUE-CHECKS):** if no job was lost since 2026-10-05 16:00Z (no completed job whose runner log has no `task` line / whose log API answers `file does not exist`), close. **NIGHT WATCH 2026-10-05/06 (burn-down night): 2 jobs lost of ~30 runs** — felhom.eu run 1384 (`4aa4d837`, 21:23→21:33Z, no log) and felhom-controller run 1401 (`c67b26be`, 00:45→00:58Z, no log); each re-run once through the API and each passed (2 m 05 s, 57 s). The night's waiter made one filtered call a minute. So the 2026-10-12 close condition („no job lost since 2026-10-05 16:00Z") is already NOT met. `audits/night-burndown-2026-10-05/r887-lost-jobs.txt`. **NIGHT WATCH 2026-10-06/07 (second burn-down night): 0 jobs lost** — every push checked by its commit (felhom.eu, controller, agent, catalog main and the held branch), each completed `success`, one filtered call per check. | — | 2026-10-19: count lost jobs since 2026-10-12 12:17Z; none → close | CC |
 | **R-902** | Process & tooling | P3 | **The contact form's mailer has no source in any repository: the running image cannot be rebuilt or changed.** FOUND 2026-10-08 (website refresh, brief Part C5). `manifests/contact-mailer.yaml` names `contact-mailer:latest` with `imagePullPolicy: Never` (imported into k3s by hand); the binary in the pod is `/app/contact-mailer` dated 2026-02-05, Go module `github.com/felhom/contact-mailer` (devel). Searched: `git log --all` of felhom.eu (only the manifest), every repo under `/mnt/5_hdd/felhom.eu/git`, `find /mnt/5_hdd/felhom.eu ~ -maxdepth 5 -iname '*contact-mailer*'`. READ from the binary (strings, read-only `kubectl exec … cat`): ONE send per form (`sendViaResend`) to `TO_EMAIL`, `reply_to` = the visitor; nothing is sent back to the visitor; the operator's notification is Hungarian (`buildEmailHTML`, „Csatolmány … fájl"). The English form needs no mailer change (its subject labels carry „(EN)"). `audits/website-refresh-2026-10-08/mailer.md` | **OPEN — NARROWED 2026-10-08 (search done, NOT FOUND): owner: operator.** A read-only search found no source on DooPlex (55,548 `*.go`/`go.mod`/`Dockerfile*` files, every disk, media/Longhorn/containerd/PBS excluded by name), in the web editor's storage and file history, in all 10 Gitea repositories' history, or in Docker's build cache (oldest record 2026-09-28). The binary's own clues: one file `/build/main.go`, go1.23.12, no `vcs.*` stamp; image built 2026-02-05 10:40 by BuildKit. The binary is now KEPT in git (`audits/mailer-source-2026-10-08/contact-mailer.bin`), so a DooPlex rebuild no longer loses the program. A replacement plan is written (`audits/mailer-source-2026-10-08/PLAN-replacement.md`); one question left: is the February folder on the operator's Windows workstation? | — | Operator: look on the workstation; if absent, build the replacement from the plan (attended deploy) | operator |
+| **R-934** | Process & tooling | P3 | **Pull the failed disk ST3000VX010 S/N Z6A07P2G out of DooPlex; follow the checklist.** The data was moved off at the first alarm (2026-10-10); the disk is unmounted, its fstab line commented out since 2026-08-14, k3s no longer waits for it, and its empty Longhorn disk entry was removed 2026-10-12. Its two SMART alarms (352 unreadable / offline-uncorrectable sectors, flat since 2026-10-09) are silenced by serial until **2026-11-11 14:32 UTC** — if the disk is still in the machine then, they come back on purpose. **A TWIN of the same model (S/N Z6A070XZ, `/mnt/3_hdd`) is in use: read the serial before unplugging.** Checklist: `runbooks/dooplex-remove-disk-Z6A07P2G.md`. | **OPEN** — NEW 2026-10-12 | the operator (hands, a power-off window) | Follow the checklist; after the first boot remove the exporter line and the silence; tell CC | operator |
 | **R-206** | Process & tooling | P4 | **The build-cache cap and the weekly prune exist only as a hand-edited `/etc/docker/daemon.json` on DooPlex — not in Ansible, so a rebuild loses them.** The `node_housekeeping` role must also carry the prune, which today it is forbidden to run | **READY (M) — NEW 2026-08-05** | — | **The spike validated the recipe; this row builds it.** Three parts. **(a) Template `/etc/docker/daemon.json`** with the **`policy` array** form — **the flat form (`{"gc":{"reservedSpace":…}}`) is SILENTLY IGNORED**, measured: the daemon starts, logs nothing, and `docker buildx inspect` still reports the built-in defaults. **The oracle is `docker buildx inspect`, never `dockerd --validate`** — the validator returned `configuration OK` for a bogus key AND for the config that then **crashed the daemon** (`filter` takes one value per policy entry, not an array; `error initializing buildkit: filters expect only one value`). **(b) Narrow the role's Docker ban** (`node-housekeeping.sh.j2:10-14`) to permit exactly `docker builder prune -af` and nothing else — the ban's stated premise ("Docker here runs only unrelated jarr-* dev containers") is obsolete: the growth is Felhom Go build cache. **The measured prune is SYNCHRONOUS** (150.35 GB back at t+0, two consecutive polls <1 MB apart within 60 s) — **unlike containerd's image GC, so it needs no `settle_imagefs` equivalent**, but it MUST measure the filesystem rather than trust the command: `prune` claimed **156.9 GB** and the filesystem returned **150.35 GB**, the 6.5 GB gap being layers still shared with images. **(c) A restart-safety note in the role:** a bad `daemon.json` takes the daemon down AND leaves the `unless-stopped` dev containers stopped — they needed a manual `docker start` — so the role must restart-and-verify, not validate-and-assume. Recipe + every measurement: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` | CC |
 | **R-928** | Process & tooling | P4 | **Two sessions running `repo_gates.py` at the same time in the shared `felhom.eu` clone see each other's DECOY plants.** SEEN 2026-10-10 ~15:40: while another session's gate run was in its decoy suite, this session's working tree showed `website/assets/crafty-controller-logo.png` renamed to `crafty-logo.png`, `website/en/apps.html` with a leftover text language switch, then `website/en/contact.html` and `scripts/felhom-host-install.sh` modified — each a plant of `test_gate_decoys.py`, each gone when that run ended. This session's own `site_gates.py` failed three times in that window on plants, not on its change, and a `git add` of a broad path at that moment would have committed a plant. A killed run (`timeout`) leaves its plant in place (its cleanup never runs). Fix direction (a design): decoys plant into a temporary COPY of the tree (as the catalog's volume-persistence decoys already do), or a lock file that a second run waits on. | **OPEN** — filed 2026-10-10 (register-shrink session) | — | Design the copy-or-lock; meanwhile stage explicit paths only (already the rule) and re-run a red site gate once another session's run has ended | CC |
 | **R-394** | Process & tooling | P4 | **`felhom-build-deploy/SKILL.md` is 179 lines, over the 150-line limit its own repo now enforces.** Found 2026-08-25 by `scripts/check_skills.py` on its first run — the over-length was discovered BY the new checker, on the day the limit was written down, which is the checker working as intended. **It is not edited and not trimmed here:** the task that introduced the limit explicitly scoped the four pre-existing skills out, and trimming a build-and-deploy skill without exercising its commands is how a wrong command ships to a live host. **It is a named single-entry exception in `GRANDFATHERED` in `scripts/check_skills.py`, printed as a WARN on every run**, so it cannot fade; a NEW skill over the limit is convicted normally, and growing the set requires editing that file in a commit with a row to name. **The rationale for the limit** — attention thins across the excess, so the lines that matter are not the ones that survive — is in `skills/felhom-doc-authoring/SKILL.md` §5. | **OPEN — LOW** | — | Trim `felhom-build-deploy/SKILL.md` under 150 lines in a session that can VERIFY the commands it keeps, then delete its `GRANDFATHERED` entry in the same commit. The likely trim is the per-artifact command blocks moving behind a pointer to the runbooks, keeping the gotchas inline — but that is a judgement for a session with a build to run, not a line-count exercise. | CC |
diff --git a/documentation/runbooks/dooplex-remove-disk-Z6A07P2G.md b/documentation/runbooks/dooplex-remove-disk-Z6A07P2G.md
new file mode 100644
index 00000000..61938404
--- /dev/null
+++ b/documentation/runbooks/dooplex-remove-disk-Z6A07P2G.md
@@ -0,0 +1,50 @@
+# RUNBOOK — pull the failed disk Z6A07P2G out of DooPlex
+
+> Written 2026-10-12 by CC from read-only checks (`audits/answers-2026-10-12/disk-Z6A07P2G.txt`). Operator's job: the
+> power-off and the hands. Register row R-934.
+
+## The disk — by SERIAL, never by `sdX`
+
+| | |
+|---|---|
+| Model | Seagate Skyhawk **ST3000VX010-2E3166**, 3 TB |
+| Serial | **Z6A07P2G** |
+| Port (2026-10-12) | the onboard SATA controller `0000:00:1f.2`, port **ata-6** (`/dev/disk/by-path/pci-0000:00:1f.2-ata-6`); kernel `ata10` |
+| Fault | 352 unreadable (pending) + 352 offline-uncorrectable sectors, flat since 2026-10-09 |
+
+**⚠ THE TWIN.** DooPlex has a SECOND disk of the same model: **ST3000VX010, serial Z6A070XZ — IN USE as `/mnt/3_hdd`, keep it.**
+`Z6A07P2G` (pull) and `Z6A070XZ` (keep) differ only in the last four characters. **Read the serial on the disk's
+label before you unplug anything.** If you cannot read it, follow the cable to SATA port 6 on the board. Do not
+make the disk busy to find it by its LED: no reads on a failing disk.
+
+## Before power-off (already true on 2026-10-12 — re-check in one minute)
+
+```bash
+lsblk -o NAME,SERIAL,MOUNTPOINTS | grep Z6A07P2G      # no mountpoint
+grep -n 9cf20140 /etc/fstab                          # only a '# DISABLED 2026-08-14' line
+grep RequiresMountsFor /etc/systemd/system/k3s.service.d/wait-for-mounts.conf   # no /mnt/4_hdd
+sudo kubectl -n longhorn-system get nodes.longhorn.io dooplex -o jsonpath='{.spec.disks}' | grep -c 4_hdd   # 0 (HDD4 entry removed 2026-10-12)
+```
+
+Nothing else reads the disk: no backup script, PV, hostPath or running app names `/mnt/4_hdd` (2026-10-12).
+
+## Pull it
+
+1. Plan a window: DooPlex runs Gitea, the registry, the hub and Longhorn. A box keeps running while the hub is off.
+2. `sudo poweroff`. Unplug power.
+3. Find the disk with serial **Z6A07P2G** (label), on SATA port 6. Unplug its data and power cable. Remove it.
+4. Power on.
+
+## After the first boot
+
+1. All mounts back: `findmnt /mnt/1_hdd /mnt/2_hdd /mnt/3_hdd /mnt/5_hdd /mnt/ssd_2` — five lines.
+2. k3s up and Longhorn healthy: `sudo kubectl -n longhorn-system get nodes.longhorn.io` READY True; all volumes
+   `healthy` (`sudo kubectl -n longhorn-system get volumes.longhorn.io`).
+3. The hub answers (`https://hub.felhom.eu`), Gitea answers.
+4. The SMART exporter: delete the line `--smartctl.device=/dev/disk/by-id/ata-ST3000VX010-2E3166_Z6A07P2G` in
+   `homelab-manifests/admin-system/smartctl-exporter.yaml`, commit, sync — otherwise it keeps asking for a disk that
+   is gone.
+5. The next DooPlex backup (03:10) is green: `systemctl show dooplex-backup.service -p Result -p ExecMainStatus`.
+6. Delete the Alertmanager silence for `device=~".*_Z6A07P2G"` (it ends by itself on **2026-11-11 14:32 UTC**).
+   If the disk is still in the machine on that date, the two alarms come back **on purpose**.
+7. Tell CC: the row closes.