R-1007 closed: proven live on demo-hp — a real power cut mid-upload, 41 of 41 broken photos repaired by Immich itself, 120 controls untouched
gates / gates (push) Successful in 6m59s

Docs 02/05/07/13 and capability F12 (PROVEN-LIVE); the audit's C and D sections; R-1020 (the
duration reads 0) and R-1021 (the hub build builds a stale checkout) filed; STATUS one screen.
Register 133 -> 140 (this session: opened 2, closed 1; another session added 6). Both plugs ON.
Report: REPORT-immich-repair-2026-10-14.md.

Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
This commit is contained in:
2026-10-14 17:50:17 +02:00
parent 8395aee4c2
commit 170d090e7b
23 changed files with 616 additions and 8 deletions
@@ -217,6 +217,7 @@
| <a id="f7"></a>[F7](capability-map-detail/F.md#f7) | Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
| <a id="f8"></a>[F8](capability-map-detail/F.md#f8) | Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | no corpus validation doc | operator infra; no evidence document |
| <a id="f9"></a>[F9](capability-map-detail/F.md#f9) | Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | — | `audits/DIAG-backup-missed-2026-07-26.md` (2026-07-26); R-82 |
| <a id="f12"></a>[F12](capability-map-detail/F.md#f12) | After a power cut, an app rebuilds the files the cut left empty, and the household sees any lost photo on the app page | controller v0.311.0, hub v0.150.0, catalog Immich block | **PROVEN-LIVE** | demo-hp 2026-10-14: a real cut mid-upload, 41 of 41 broken photos repaired, 120 controls untouched; the lost-photo note not exercised live (0 lost; unit- and 9202-tested) | `audits/immich-repair-2026-10-14/README.md` (2026-10-14); `09` §3 decision 236; R-1007 |
| <a id="f11"></a>[F11](capability-map-detail/F.md#f11) | The household turns the whole machine off („A gép kikapcsolása"); the hub holds every silence alarm for 14 days and shows KIKAPCSOLVA | agent v0.155.0, hub v0.149.0, controller v0.310.0 | **PROVEN-LIVE** | proven on the N100 2026-10-13; fleet delivery waits on the floor | `audits/poweroff-2026-10-13/README.md` (2026-10-13); `architecture/14-host-power-off.md`; R-1000, R-1002 |
| <a id="f10"></a>[F10](capability-map-detail/F.md#f10) | A box that is not always on (off at its backup time) makes the missed night up once when it comes back; the household sees a banner; the operator gets the missed-backup alarm, down or not | controller v0.295.0, hub v0.134.0, agent v0.145.0 | **PARTIAL** | catch-up and banner live; R-872 alarm by tests; suspend not measured | `audits/catchup-2026-10-05/partA/` (2026-10-05); R-872 |
@@ -285,6 +285,17 @@ own; every caller that is not the customer must decide for itself whether the ap
| `collector.go` | **PORT** | Container metrics (`docker stats`) keep; host metrics via `system.GetInfo` (temp, physical disk) become **agent-provided or dropped**. | needs-rework |
| `sysinfo.go`/`sysinfo_other.go` | **MODIFY** | Reads `/host/etc`, `/proc/cpuinfo`, uptime — host static info; in-guest some is meaningful, hardware identity via agent. | needs-rework |
### `aftercrash/` + `stacks/after_unclean_boot.go` — KEEP (v0.311.0, R-1007, `09` §3 decision 236)
**[FACT, read from source 2026-10-14, felhom-controller `3b85a0c`]** Once per controller start that followed an UNCLEAN
host boot — `crashboot.Grace.Decided` (the host crash guard's fact via the agent; unknown = a normal boot), after
`CrashGrace` (15 min) — `aftercrash.Runner` runs each deployed app's `after_unclean_boot` template block: scan the app's
folder for zero-byte regular files (no symlinks, one filesystem), map names to the app's ids, run the app's own repair
command for those ids in batches, re-scan, retry once, run the app's lost-file check; record in `app.yaml`
(`after_unclean_boot:`), push `app_repaired_after_crash` (counts only). The commands run through a QUIET compose exec
(`quietComposeExec`: nothing logged, stderr dropped, the error is the exit code only) because their output can carry the
household's file names. The household's lost list is shown on the app page (`lostFilesNote`) until dismissed.
### `monitor/`
| File | Class | Reason | Risk |
|---|---|---|---|
@@ -382,6 +382,15 @@ answer the question the text refuses to — for every customer who is not Hungar
state therefore always renders in the default language; every other state already discloses that the
token is real. Pinned by `TestBindExpiredIsAlwaysDefaultLanguage`.
### 15.4a What the household is NOT mailed: an app's repair after a power cut [hub v0.150.0, R-1007]
**[FACT, hub `822d0393`]** `app_repaired_after_crash` (controller v0.311.0) is allow-listed AND operator-only
(`notify.operatorOnlyEvents`, registered in one commit and pinned together), and a per-app cooldown member. It carries
counts only — empty files found / repaired / still broken, originals lost, the boot, the duration, or a skip reason.
warning mails the operator (anything lost, still broken, failed or skipped); info (all repaired) is recorded, not mailed.
The operator mail renders the counts as one `Repair:` line. The household's side is NOT a mail: the lost files (name,
date taken) are shown on its own app page, on its box, until it dismisses them — file names never leave the box.
### 15.5 How "the Hungarian did not change" is known
56 goldens captured from v0.117.0 before any string moved, in
@@ -458,6 +458,16 @@ nothing holds the entry open (a killed worker leaves `tar`/`zstd` writing for mi
`audits/power-cut-fixes-2026-10-14/`. A full host root also breaks LVM's own metadata writes (C4) — the reason this matters
beyond one skipped night.
**[FACT, 2026-10-14] A power cut while an app writes leaves some of its files ZERO BYTES, and from controller v0.311.0 the
app repairs them itself.** ext4 keeps the name and loses the data written in the last seconds: the night's cut left 81 of
101 recent Immich photos with empty thumbnails and previews and one original gone (`audits/night-power-2026-10-13/` R2);
the R-1007 live proof (demo-hp 9201, 15:25:04 UTC, mid-upload) left 41 photos with both files empty. Immich never redoes
them (its „missing" run reads its database, not the disk). After an UNCLEAN boot the controller runs the app's
`after_unclean_boot` block (`02` `aftercrash/`): 41 of 41 repaired in 10 s, the 120 control files untouched (mtimes),
the lost-file check read (0 lost). A lost ORIGINAL is restored only from a backup (Tier 2, the night's DB + volumes; the
upload library is a mandatory class) or uploaded again by the household — its app page lists it
(`audits/immich-repair-2026-10-14/`).
**[FACT] What the whole-guest tiers do NOT carry.** `mp8 /mnt/felhom-drives` and
`mp9 /etc/felhom-bootstrap` are **host bind mounts** and are out of `vzdump` scope entirely (LIVE
`pct config 9201`, both hosts). So the whole-guest tiers carry the guest and **none of the customer's
@@ -60,6 +60,7 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981) |
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976) |
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978), and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
| **added (R-1007):** controller → an app's own job queue | holds — nothing new | the `after_unclean_boot` commands run inside the app's container through the docker socket the controller already holds, and read the app's database with a read-only session: no key into the household's account, nothing a compromised controller could not already do (exec in any app, read its database). The household's lost-file names stay on the box (`app.yaml`, the app page); the operator event carries counts only, and a failing command's output is never logged (security review of controller `8d862e4`, fixed in `3b85a0c`) |
| **added:** agent ↔ root on the host | holds by the sudoers | every root verb is one regex-anchored line or a root wrapper that re-checks its input (`03` §3); agent v0.156.1 adds one delete: `felhom-vzdump-partial-clean` removes only a DEAD half-written vzdump of the named guest on a `dir` storage (no vzdump on the host, nothing holding it open) — a compromised agent gains the deletion of a leftover that is not a backup |
| agent ↔ hub | holds for destructive ops | signed jobs verified in the documented order; unsigned spec has no size bounds (R-984) |
| controller ↔ hub | **did not hold until 2026-10-12** | the global key was public (R-950, rotated) and three routes needed no key (R-951, fixed); per-box keys are bound to their box (R-955) |
@@ -198,3 +198,20 @@ v0.310.0 (the card, `POST /api/server/poweroff`).
**Gap / roadmap:** the website FAQ line (R-1000); the hosts-page badge reads ONLINE for the first 45 min (R-1002).
---
## F12 — After a power cut, an app rebuilds the files the cut left empty
*Map row: [F12](../00-capability-map.md#f12).*
**Scenario:** the power goes off while photos upload; some thumbnails and previews are left zero bytes, rarely an
original is lost. After the unclean boot (and the 15-minute crash grace) the box asks Immich to rebuild exactly those
photos, runs Immich's missing-files check, shows a lost photo's name and date on the app page, and gives the operator the
counts.
**Components:** controller v0.311.0 (`internal/aftercrash`, `internal/stacks/after_unclean_boot.go`), hub v0.150.0
(`app_repaired_after_crash`, operator-only), catalog `templates/immich/.felhom.yml` `after_unclean_boot` (proven on
`immich-server:v3.2.4`). Decision 236.
**Evidence:** `audits/immich-repair-2026-10-14/` — spike A on 9202 (the five candidates), the scripts on real damage
(9202), the live cut on demo-hp 9201 (D/). **Not exercised live:** a lost original (the cut lost none); pinned by unit
tests and the 9202 run.