diff --git a/documentation/architecture/09-update-architecture.md b/documentation/architecture/09-update-architecture.md index a35bd5e5..7296c056 100644 --- a/documentation/architecture/09-update-architecture.md +++ b/documentation/architecture/09-update-architecture.md @@ -915,6 +915,17 @@ its length, and both fixes cost something the household would notice — operato 127. **The agent's three by-design abilities (`03` §3.1) stay for now**; revisited before the first paying customer. *Operator ruling 2026-10-05.* (R-861) +### 2026-10-14 — R-1007: how the box asks Immich to repair after a power cut (recorded before the build) + +236. **The box repairs Immich's broken photos through Immich's OWN job queue (option 1, spike A4)** — the controller adds + the same job Immich's API adds (`AssetGenerateThumbnails {id}` on `thumbnailGeneration`, Immich's own `bullmq` + inside `immich-server`) for the asset ids of the zero-byte thumbnail/preview files only, queues Immich's own + missing-files check, and READS its report from the database (a SELECT, never a write). Not chosen: a household-made + API key (it covers only the admin's photos — a family member's were refused, measured), a database write, the + household pressing „thumbnails: all". The job/queue names are internal to Immich: the catalog pins the Immich + versions they were proven on; any other version is skipped and reported, never guessed. Evidence: + `audits/immich-repair-2026-10-14/` Part A. *Operator ruling 2026-10-14, in chat.* + ### 2026-10-12 (evening) — the notification page's two questions answered (recorded before the work; the disk + shrink brief) *Operator answers in chat 2026-10-12 ~16:30, „as picked" (`08` §8.1).* diff --git a/documentation/audits/immich-repair-2026-10-14/C/scripts-on-9202.txt b/documentation/audits/immich-repair-2026-10-14/C/scripts-on-9202.txt new file mode 100644 index 00000000..f1c9a016 --- /dev/null +++ b/documentation/audits/immich-repair-2026-10-14/C/scripts-on-9202.txt @@ -0,0 +1,14 @@ +syntax rc=1 +uploaded admin 20 member 10 +damage: 4 empty files, 1 original deleted +== repair.js with 4 ids +FELHOM_REPAIR_OK queued=4 waited_ms=2005 +rc=0 in 2.38 s; empty now: 0 +controls rewritten: 0 of 53 +== lost.js +LOST {"name":"gen-007.jpg","taken":"2026-01-03 01:00"} +FELHOM_LOST_OK lost=1 waited_ms=2040 +rc=0 in 2.4 s +== a second run of lost.js (a stale or repeated report?) +FELHOM_LOST_OK lost=1 waited_ms=2040 + diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index f575dbae..dbc3575e 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -136,7 +136,7 @@ stopping line that lies. | **R-707** | Apps & catalog | P4 | **[P2] 37 apps still start with a login a stranger can take (`09` §3 decision 45).** Audit of all 53 apps: `app-catalog-felhom.eu/FIRST-ADMIN.md` (class, fix route, status, measured or read). Open: **3 hard-coded defaults** — calibre-web (`admin / admin123`, measured working on demo-hp and 9202; its own `cps.py -s` route needs a generated password WITH a special character — our generator is letters+digits, a controller change), mealie (`changeme@example.com / MyPassword`), wger (`admin / adminadmin`); **34 open first-run screens** (the first visitor creates the admin: actualbudget, adventurelog, audiobookshelf, calcom, docmost, emby, ghost, gitea, gramps-web, home-assistant, homebox, immich, jellyfin, komga, n8n, navidrome, opengist, outline, papra, plant-it, radarr, rallly, recipe-importer, romm, seerr, sonarr, sparkyfitness, tandoor, termix, uptime-kuma, vikunja, wanderer, wishlist, zipline). **Stale notes:** romm's `default_creds` `admin / admin` answers 401 on demo-hp (like a wrong password) — the page now warns with a login that does not exist; zipline's looks stale too. **Measured on demo-hp 2026-09-28 (read-only):** bookstack's default still logs in on the INSTALLED app (the fix is for new installs; the page now warns). Each fix: route (a) env or (b) the app's own CLI/API via `after_install:`, proven on 9202 with the default failing and the generated password working; route (c) a page sentence. Several sessions (operator, 2026-09-28). **2026-09-29 (controller v0.280.0, catalog `d0e7e2e`):** every class-3 app fixed — mealie, wger, calibre-web by `after_install` (calibre-web with the new `password:24:special`), proven on 9202 fresh installs (`audits/login-gate-2026-09-29/D/`); the setup gate (decision 46, spike PASSED) built and live on immich, n8n, audiobookshelf (probes measured) and uptime-kuma (button) (`…/C/`); romm's and zipline's stale notes removed. **Left: 30 class-4 apps** — gate each (probe measured on 9202 where one exists — 11 upstream candidates listed in `…/B/B-VERDICT.md` §3; the button otherwise). **2026-09-29 afternoon (controller v0.281.0, catalog `6faf432`):** 28 more class-4 apps gated — 32 of 34 — each proven on 9202 (`audits/gate-rollout-2026-09-29/`B): stranger → gate page / 401, household reached the first-setup screen, the gate opened (9 by a measured probe, the rest by the press), the app answered after. seerr, outline, rallly: gated, their opening needs a media server / e-mail (not proven). **Left:** wanderer (R-714); plant-it is not installable. | **NARROWED** (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — seerr, outline and rallly are gated, but the gate OPENING is not proven (needs a media server / e-mail)) — **CLOSED — 2026-09-29 (the rest → R-714)** | — | — | CC | | **R-905** | Apps & catalog | P4 | **wger's collected style files (283 MB) go into every wger backup, though the app rebuilds them at every start.** Since R-762's fix (`DJANGO_DEBUG=False` + `collectstatic`, catalog `cf1ed43`) the static files sit in a named volume that the backup legs copy like data (measured 283 MB, `audits/design-build-2026-10-06/`); `collectstatic` regenerates them, so they are rebuildable bytes in every Tier-1 unit, Tier-2 copy and off-site snapshot. wger is `lifecycle: hidden` and no box runs it (hub read 2026-10-08). Options to weigh: a non-backed-up volume class for regenerable data, or an anonymous volume. Found by the R-762 helpers 2026-10-08. | **OPEN — needs a design before wger is shown** | — | Design the „regenerable volume" exclusion (catalog + controller backup legs) | CC | | **R-995** | Apps & catalog | P4 | **Removing an app through the product can leave unnamed Docker volumes and an empty import folder behind.** MEASURED 2026-10-13 on demo-felhom after 61 removals (`documentation/audits/sizing-2026-10-13/METHOD.md` „Teardown”, `evidence/leftover-volumes-after-bench.txt`): 8 unnamed volumes (4–72 KB; immich 1, romm 2, privatebin 3, onlyoffice 2 — volumes the images declare themselves) and `userdata/import/{paperless,calibre,grimmory}` (empty, created by the deploy). The removal answered 200 and its own leftover check was clean, because it looks only at labelled volumes. No data risk; disk and tidiness only. Removed by name on the bench. | **READY** — filed 2026-10-13 | — | Make the removal account for the containers' unnamed volumes (and an empty import folder it created), with a test that deploys an image that declares a VOLUME | CC | -| **R-1007** | Apps & catalog | P3 | **After a power cut Immich shows photos as done that are broken, and nothing repairs them: of 101 photos uploaded in the last ~30 s before the cut, 81 kept their original but got an EMPTY thumbnail and preview (162 zero-byte files), and 1 — answered „201 Created" 0.3 s before the cut — has no original at all.** Immich's job queues are empty after the start, so it never redoes them (`audits/night-power-2026-10-13/` R2, `immich-check.txt`). Same ext4 effect on a plain 40-byte file. Other apps may do the same with files written just before a cut. | **OPEN** — 2026-10-14 spike (`audits/power-cut-fixes-2026-10-14/README.md` F, Immich v3.2.4): „thumbnails: missing" does not fix a zero-byte thumbnail (nor after deleting it — Immich decides from its database); „thumbnails: all" (force) fixes them; `integrity-missing-files` (`/api/admin/integrity/report?type=missing_file`) finds a lost original; nothing restores one but a backup. **Operator ruling 2026-10-14: fix only the broken photos** — after an unclean boot, find the photos with an empty thumbnail or preview, rebuild only those, run the missing-files integrity check, and tell the operator about lost originals. No „thumbnails: all". 2026-10-14 B2 spike (`audits/power-cut-fixes-2026-10-14/README.md` B2): **yes** — `POST /api/assets/jobs {assetIds, name: regenerate-thumbnail}` rebuilt exactly the 10 broken assets (1.55 s, 5.3 s CPU) and rewrote none of the other 20; an empty file maps to its asset by the first 36 characters of its name. | — | CC: build — a per-app after-unclean-boot hook that finds the zero-byte files under `thumbs/`, calls `POST /api/assets/jobs` `regenerate-thumbnail` for those asset ids only, runs `integrity-missing-files`, and tells the operator about lost originals (a task file: catalog + controller) | CC | +| **R-1007** | Apps & catalog | P3 | **After a power cut Immich shows photos as done that are broken, and nothing repairs them: of 101 photos uploaded in the last ~30 s before the cut, 81 kept their original but got an EMPTY thumbnail and preview (162 zero-byte files), and 1 — answered „201 Created" 0.3 s before the cut — has no original at all.** Immich's job queues are empty after the start, so it never redoes them (`audits/night-power-2026-10-13/` R2, `immich-check.txt`). Same ext4 effect on a plain 40-byte file. Other apps may do the same with files written just before a cut. | **OPEN** — 2026-10-14 spike (`audits/power-cut-fixes-2026-10-14/README.md` F, Immich v3.2.4): „thumbnails: missing" does not fix a zero-byte thumbnail (nor after deleting it — Immich decides from its database); „thumbnails: all" (force) fixes them; `integrity-missing-files` (`/api/admin/integrity/report?type=missing_file`) finds a lost original; nothing restores one but a backup. **Operator ruling 2026-10-14: fix only the broken photos** — after an unclean boot, find the photos with an empty thumbnail or preview, rebuild only those, run the missing-files integrity check, and tell the operator about lost originals. No „thumbnails: all". **Decision 236 (operator 2026-10-14, after spike A `audits/immich-repair-2026-10-14/`): the box asks Immich through Immich's own job queue (no key; every user's photos; the lost-file report read from the database); the catalog pins the proven Immich versions.** 2026-10-14 B2 spike (`audits/power-cut-fixes-2026-10-14/README.md` B2): **yes** — `POST /api/assets/jobs {assetIds, name: regenerate-thumbnail}` rebuilt exactly the 10 broken assets (1.55 s, 5.3 s CPU) and rewrote none of the other 20; an empty file maps to its asset by the first 36 characters of its name. | — | CC: build per decision 236 — catalog block `after_unclean_boot` (Immich), controller runner after an unclean boot, household note for lost photos, operator-only event; live proof on demo-hp 9201 (`audits/immich-repair-2026-10-14/`) | CC | ## App updates — 4 rows (P3 4) diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index a8ef341e..64f02664 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,18 @@ +## v0.150.0 — an app's repair after a power cut reaches the operator, counts only (R-1007) (2026-10-14) + +Needs **controller ≥ 0.311.0** to see anything (it sends the event). Older boxes never send it — nothing changes for them. + +- **`app_repaired_after_crash`** — what an app's `after_unclean_boot` run found and did after an unclean host boot: + empty files found / repaired / still broken, originals lost, the boot, the duration, or why it was skipped. Allow-listed + AND operator-only, registered in the same commit (`internal/api/handler.go`, `notify.operatorOnlyEvents`); a + per-app cooldown member (one app's event, no digest). The household's side is the lost-files note on its own app + page — never a mail with its file names. warning mails; info (all repaired) is recorded, not mailed. +- The operator mail carries one readable line: „Repair: Immich — after the unclean boot at …: 81 empty file(s) found, + 81 repaired, 0 still broken; originals lost: 1; 4 s". +- Tests: `internal/api/chaosnight_events_test.go` (the pair), `internal/notify/app_repaired_after_crash_test.go` (a + household that enabled the type by name gets nothing; info is not mailed; the counts line), `r389` (the cooldown + list). Red-proofed: each of the three guards removed convicted its test. + ## v0.149.0 — a box the household switched off is not an outage, for 14 days (R-998) (2026-10-13) Needs **agent ≥ 0.155.0** to see anything: the agent's new `deliberate_off.requested_at` report field (household diff --git a/hub/internal/api/chaosnight_events_test.go b/hub/internal/api/chaosnight_events_test.go index 1d582235..5a6ba7ec 100644 --- a/hub/internal/api/chaosnight_events_test.go +++ b/hub/internal/api/chaosnight_events_test.go @@ -111,3 +111,15 @@ func TestAppStoppedUnhealthyIsAHouseholdEvent(t *testing.T) { } } } + +// R-1007 (v0.150.0) — app_repaired_after_crash is allow-listed AND operator-only, pinned together. +// RED-PROOF: drop it from allowedEventTypes → "the controller's push would 400". +func TestAppRepairedAfterCrashIsAllowlistedAndOperatorOnly(t *testing.T) { + et := "app_repaired_after_crash" + if !allowedEventTypes[et] { + t.Fatalf("%s must be in allowedEventTypes — the controller's push would 400", et) + } + if !notify.IsOperatorOnly(et) { + t.Fatalf("%s must be operator-only — counts of a household's damaged files are support's", et) + } +} diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index 8fc5341a..471f5be6 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -2318,6 +2318,10 @@ var allowedEventTypes = map[string]bool{ // with NO copy on its box that brings it back whole — support must act. Operator-only // (notify.operatorOnlyEvents) — registered in BOTH in the same commit. "app_hold_no_whole_copy": true, + // R-1007 (hub v0.150.0, controller v0.311.0, `09` §3 decision 236): what an app's after_unclean_boot run found + // and repaired after an unclean host boot — COUNTS ONLY. Operator-only (notify.operatorOnlyEvents) — registered in + // BOTH in the same commit; the household's side is the app page's lost-files note on its own box. + "app_repaired_after_crash": true, // Special "test": true, } diff --git a/hub/internal/notify/app_repaired_after_crash_test.go b/hub/internal/notify/app_repaired_after_crash_test.go new file mode 100644 index 00000000..a9cc71f6 --- /dev/null +++ b/hub/internal/notify/app_repaired_after_crash_test.go @@ -0,0 +1,49 @@ +package notify + +import ( + "io" + "log" + "strings" + "testing" +) + +// R-1007 (v0.150.0, controller v0.311.0, `09` §3 decision 236) — app_repaired_after_crash reaches the OPERATOR and +// never the household, even a household that enabled it by name; warning mails, info is recorded only; the mail +// carries the counts as one line. +// RED-PROOF: drop it from operatorOnlyEvents → the customer gets a mail → fails. +func TestAppRepairedAfterCrash_OperatorOnlyEvenWhenTheHouseholdEnabledIt(t *testing.T) { + st := newDispStore(t) + if err := st.SaveNotificationPrefs("c1", "cust@example.com", []string{"app_repaired_after_crash", "node_down"}, 6); err != nil { + t.Fatal(err) + } + d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + sent := captureSeam(d) + details := `{"app":"Immich","stack_name":"immich","boot_at":"2026-10-14T12:07:26Z","found":81,"repaired":81,"still_broken":0,"lost":1,"lost_checked":true,"duration_s":4}` + d.ProcessEvent("c1", "app_repaired_after_crash", "warning", "Immich: ...", details, "controller") + if got := mailsFor(*sent, "cust@example.com"); len(got) != 0 { + t.Fatalf("the household got %d mail(s) — the counts are the operator's; its side is the app page's note", len(got)) + } + if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 { + t.Fatalf("operator mails = %d, want 1", got) + } + // info (everything repaired): recorded, not mailed. + *sent = nil + d.ProcessEvent("c1", "app_repaired_after_crash", "info", "Immich: ...", strings.Replace(details, `"lost":1`, `"lost":0`, 1), "controller") + if got := len(*sent); got != 0 { + t.Fatalf("an all-repaired (info) run mailed %d time(s)", got) + } +} + +// RED-PROOF: drop the renderAppRepair call → the body lacks the counts line → fails. +func TestAppRepairedAfterCrash_OperatorMailHasTheCountsLine(t *testing.T) { + _, body := FormatOperatorEmail("c1", "app_repaired_after_crash", "warning", "msg", + `{"app":"Immich","boot_at":"2026-10-14T12:07:26Z","found":81,"repaired":80,"still_broken":1,"lost":1,"lost_checked":true,"duration_s":4}`) + want := "Repair: Immich — after the unclean boot at 2026-10-14T12:07:26Z: 81 empty file(s) found, 80 repaired, 1 still broken; originals lost: 1; 4 s" + if !strings.Contains(body, want) { + t.Fatalf("body lacks %q:\n%s", want, body) + } + _, skipped := FormatOperatorEmail("c1", "app_repaired_after_crash", "warning", "msg", `{"app":"Immich","boot_at":"b","skipped":"not proven"}`) + if !strings.Contains(skipped, "SKIPPED after the unclean boot at b: not proven") { + t.Fatalf("skipped body:\n%s", skipped) + } +} diff --git a/hub/internal/notify/dispatcher.go b/hub/internal/notify/dispatcher.go index 1f594903..05abde5b 100644 --- a/hub/internal/notify/dispatcher.go +++ b/hub/internal/notify/dispatcher.go @@ -480,6 +480,9 @@ var perAppCooldownEvents = map[string]bool{ "app_hold_no_whole_copy": true, // v0.123.0 (decision 28): one app's stop, no digest behind it. "app_stopped_unhealthy": true, + // v0.150.0 (R-1007): one app's repair after an unclean boot, no digest behind it — two apps on one boot are two + // records, and a lost photo on the second must not be swallowed by the first's cooldown. + "app_repaired_after_crash": true, } // perAppCustomerCooldownEvents is the CUSTOMER-leg sibling of perAppCooldownEvents (v0.120.0). The @@ -857,6 +860,10 @@ var operatorOnlyEvents = map[string]bool{ // operator-grade (the copies seen, per tier, with dates), and the act it calls for is support's. // Registered in the same commit that mints it. "app_hold_no_whole_copy": true, + // R-1007 (v0.150.0, controller v0.311.0, `09` §3 decision 236). An app's repair after an unclean host boot: counts + // of empty files found / repaired / still broken / originals lost. The household learns of a lost photo on its own + // app page (name and date, on its box); THESE counts are the operator's. Registered in the same commit that mints it. + "app_repaired_after_crash": true, } // IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the diff --git a/hub/internal/notify/r389_cooldown_grain_test.go b/hub/internal/notify/r389_cooldown_grain_test.go index 289b2bd7..70f34b4e 100644 --- a/hub/internal/notify/r389_cooldown_grain_test.go +++ b/hub/internal/notify/r389_cooldown_grain_test.go @@ -71,7 +71,8 @@ func TestR389_TheAllowListHasExactlyOneMember(t *testing.T) { // v0.121.0 widened it again, on purpose (R-636): an OOM storm is one app's event with no digest. // v0.122.0 once more (R-659): a stranded held app is one app's event with no digest. // v0.123.0 once more (decision 28): an unhealthy stop is one app's event with no digest. - want := []string{"app_hold_no_whole_copy", "app_oom_storm", "app_start_failed", "app_stopped_unhealthy", "app_update_held", "app_update_undone"} + // v0.150.0 once more (R-1007): an app's repair after an unclean boot is one app's event with no digest. + want := []string{"app_hold_no_whole_copy", "app_oom_storm", "app_repaired_after_crash", "app_start_failed", "app_stopped_unhealthy", "app_update_held", "app_update_undone"} ok := len(perAppCooldownEvents) == len(want) for _, w := range want { ok = ok && perAppCooldownEvents[w] @@ -81,7 +82,7 @@ func TestR389_TheAllowListHasExactlyOneMember(t *testing.T) { for k := range perAppCooldownEvents { got = append(got, k) } - t.Fatalf("perAppCooldownEvents = %v, want exactly [app_hold_no_whole_copy app_oom_storm app_start_failed app_stopped_unhealthy app_update_held app_update_undone]. Adding a member is the "+ + t.Fatalf("perAppCooldownEvents = %v, want exactly [app_hold_no_whole_copy app_oom_storm app_repaired_after_crash app_start_failed app_stopped_unhealthy app_update_held app_update_undone]. Adding a member is the "+ "fenced act: the backup family's cooldown is coarse ON PURPOSE (R-97a, R-182) so one full "+ "disk sends one digest, not one mail per app. Read the fence before widening this.", got) } diff --git a/hub/internal/notify/templates.go b/hub/internal/notify/templates.go index bd778ce9..1ecced14 100644 --- a/hub/internal/notify/templates.go +++ b/hub/internal/notify/templates.go @@ -63,6 +63,13 @@ Message: %s`, customerID, eventType, severity, now, message) // digest that renders badly still tells the operator something; a swallowed one does not. } + // R-1007: the repair after an unclean boot, as one readable line of counts (the payload carries nothing else). + if eventType == "app_repaired_after_crash" { + if line, ok := renderAppRepair(detailsJSON); ok { + body += "\n" + line + } + } + if detailsJSON != "" && detailsJSON != "{}" { body += fmt.Sprintf("\nDetails: %s", detailsJSON) } @@ -424,3 +431,35 @@ func stackNameOf(detailsJSON string) string { } return d.StackName } + +// renderAppRepair is app_repaired_after_crash's operator line (R-1007): the counts, the boot and the duration. +func renderAppRepair(detailsJSON string) (string, bool) { + var d struct { + App string `json:"app"` + BootAt string `json:"boot_at"` + Found int `json:"found"` + Repaired int `json:"repaired"` + StillBroken int `json:"still_broken"` + Lost int `json:"lost"` + LostChecked bool `json:"lost_checked"` + DurationS int64 `json:"duration_s"` + Skipped string `json:"skipped"` + Problem string `json:"problem"` + } + if json.Unmarshal([]byte(detailsJSON), &d) != nil || d.App == "" { + return "", false + } + if d.Skipped != "" { + return fmt.Sprintf("Repair: %s — SKIPPED after the unclean boot at %s: %s", d.App, d.BootAt, d.Skipped), true + } + lost := "not checked" + if d.LostChecked { + lost = fmt.Sprintf("%d", d.Lost) + } + line := fmt.Sprintf("Repair: %s — after the unclean boot at %s: %d empty file(s) found, %d repaired, %d still broken; originals lost: %s; %d s", + d.App, d.BootAt, d.Found, d.Repaired, d.StillBroken, lost, d.DurationS) + if d.Problem != "" { + line += "\nProblem: " + d.Problem + } + return line, true +}