Night 2026-10-19 Part 10: ep0's first weekly prune matched the dry runs (only tester-1's 10-04 copy removed; every household has a rule) -> R-1162 closed, register 161 -> 160; R-895 and R-435 due dates moved 10-20 -> 10-21 with the reason; P10 evidence
@@ -488,7 +488,8 @@ data drives** — 916 GB on demo-felhom, 938 GB on demo-hp.
> (today, floodable); a per-tier count per rolling window (chosen: no ordinary day is refused); a time-bucketed prune
> instead (the durable shape, but a retention change — R-1149 for the local tier, R-973's design line for ep0). Limits:
> ep0's prune is time-bucketed since 2026-10-19: **[FACT] `prune-demo-hp` and `prune-demo-felhom` keep `keep-weekly 2`**
> (R-1162, operator decision STATUS D18, 2026-10-19 08:24; read back 07:49 UTC) — the newest copy of each of the two newest
> (R-1162, operator decision STATUS D18, 2026-10-19 08:24; read back 07:49 UTC; the first scheduled run, 2026-10-20
> 03:30 UTC, removed only tester-1's 2026-10-04 copy, as the dry runs said — `audits/night-2026-10-19/P10/`) — the newest copy of each of the two newest
> ISO weeks, so a flood inside a week replaces only that week's copy and the previous week's survives, however many runs
> get through. Before the change, PBS's own dry run and an offline model of its selection both showed the new rule
> removing nothing on either box (`audits/day-2026-10-19b/P1/`). DooPlex's copy of ep0 already prunes by time
| **R-921** | **When the off-site tier answers BUSY, the household still gets a short stop of every app — with no copy.** (P3) | CLOSED 2026-10-19 — **the measured case is now seen before the stop** (operator answer „add one field", 19:2x): the agent serves its host-wide gate as `host_busy` on `GET /backup/status` (agent `56aabc7`, committed, NOT released — it reaches a box with the next agent release, STATUS's morning question), and controller 0.335.0 (the floor since 18:39 UTC) reads it before any stop: held → no app stopped, each due tier recorded as contention (15-minute skip, one „BLOCKED" alarm after 3 h). An agent without the field keeps the old order (proven live: demo-hp's agent answers no `host_busy`). The race (free at the read, busy at the start) keeps the old short stop, resumed at once. Tests red-proved: agent `TestBackupStatus_R921_HostBusy`; controller `TestR921_NoStopWhileTheHostIsBusy` (without the branch: 3 apps stopped), `…SkipsStillAlarmAfterTheCeiling`, `…OtherAnswersKeepTheOldOrder`, `TestHostBusy_ReadsTheAgentsGate`. A live busy state was not made on 9201 (it needs the new agent there). | `audits/night-2026-10-19/P9/` (release), controller `15ea002`, agent `56aabc7` |
| **R-953** | **A full ArgoCD sync of the `felhom` app with prune would DELETE three live Secrets and so break the hub and Umami.** (P3) | CLOSED 2026-10-19 — **not reproducing: ArgoCD no longer tracks the three Secrets.** Read back 18:21 UTC (read only, no sync): app `felhom` Synced/Healthy at `a28f162c`, 25 tracked resources, none of them a Secret, none `requiresPruning`, `Deployment/umami` Synced; the three Secrets carry no labels and no annotations (no tracking id). A sync with prune touches only tracked resources, so the next deploy does nothing with them. No manifest change was possible or needed (a `Prune=false` annotation would live on the untracked live object). How the tracking marks went after 2026-10-12 is not recorded. The build-deploy skill's „sync only the hub" line stays. | `audits/night-2026-10-19/P4/r953-readback.txt` |
| **R-986** | **No dependency or secret scanner runs on any push, so a known-vulnerable library or a committed key is found only by a review like this one.** (P3) | CLOSED 2026-10-19 — **the warn-first gate runs on every push in all four repos.**`felhom.eu/scripts/security_scan_gate.py` (shared, never copied) in each runner's FAST set — pre-push hook and CI: govulncheck v1.8.0 (called vulnerabilities only) on each Go module, gitleaks 8.24.3 on the pushed commits with the default rules pinned; hits listed, exit 0. Accepted hits per repo in `.security-accepted` (gitleaks by rule + a hash of the line + file), reason per line. CI fetches the pinned gitleaks with both sha256 checked (seen in CI 2108: 42 hits, all accepted); CI has no Go, so govulncheck is NOT CHECKED there, out loud — the hook runs it. 8 decoys in scratch repos, red-proved. First scan of today's `main`: 0 called vulnerabilities, 0 real secrets, every gitleaks hit with a verdict. The govulncheck pin moved v1.1.4 → v1.8.0 (v1.1.4 cannot read Go 1.27 code). Warn → fail: R-1168. | `audits/night-2026-10-19/P7/triage.md`, `SCANNERS.md` §5 |
| **R-1162** | **ep0's whole-guest off-site prune was a count (keep-last 2), so a compromised controller could push out the pre-compromise copy, slowly.** (P3) | CLOSED 2026-10-20 — **the first scheduled run matched the dry runs** (read only on ep0, 03:45–03:47 UTC). The four household prune jobs ran at 03:30 UTC, each `TASK OK`, each `--max-depth 0 --keep-weekly 2`: demo-hp kept 2026-10-08 and 10-15; demo-felhom kept 10-06 and 10-13; Tester-2 kept its one copy (10-04); tester-1 removed 10-04 and kept 10-11 and 10-18 — the only copy removed, as predicted. Copies after: 2, 2, 1, 2. The hub's ep0 read (tenantsync v1.3.0 on ep0, the repo file's sha256) answers `unpruned_checked: true, unpruned: []` when run directly; the hub (0.161.0) logs a PBS-DR refresh every 15 minutes and no „no prune job" line since its start (2026-10-19 14:10 UTC). The operator job (`keep-daily 14 --keep-weekly 8`, 03:45) removed one hub copy (2026-10-06), its own retention. | `audits/night-2026-10-19/P10/01-before.txt`, `02-after.txt` |
## 2026-10-19 (third day brief) — the passphrase locks once the box uses its own key, every ep0 household gets a prune job, demo-hp's address (full text of each row: `git show 97f27a75:documentation/backlog/OPEN-ITEMS.md`)
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/`**LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest | — | — | operator |
| **R-893** | Backup & restore | P3 | **After a failed OFF-SITE replay, the rollback pours the NEWER pre-restore copy over the OLDER volume just put back.** Read in source 2026-10-06 (R-638 option A, not measured): `internal/backup/offbox_reconstitute.go` writes the undo copy from the live (newer) database, replaces the volumes with the snapshot's older tars, then — when the replay fails — `rollbackSafetyDump` loads that newer dump over the older database volume. The loader only drops what the dump knows, so tables the newer migration removed stay; and when the snapshot's older definition was written, the rollback branch does not put the newer definition back, so the older app starts on rolled-back data; non-database volumes stay at the snapshot's state. An order change cannot fix it (the only undo is a logical dump, and its volume was replaced). Known limit in `07` §6.3. **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-893.md` — re-verified; and the screen's „your data is back as it was" (`err.backup.db_restore_failed_rolled_back`) is false for files, other volumes and the app version. Question D8 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D8 (`09` §3 decision 192):** yes, both, in that order — first the app stays stopped for support (option C), next „put back exactly as it was" (option A). | **READY — measured 2026-10-19 on 9202 and a design note written; the decision is the operator's** (STATUS D20). A forced replay failure after the volume was replaced (Rallly + postgres 18, scratch SFTP off-site on 9202's own disk): the undo dump's newest rows on top of the backup's volume, a table only the backup had left behind (37 tables against 36), definition unchanged (same version), app held (`restore_mixed`, start → 409). Note: `audits/night-2026-10-19/P3/design-R-893.md` — A (copy the app's live volumes before the replay, put them back on failure; one copy's disk for minutes, a fit check before the stop) or B (empty the DB volume before the undo load; no disk, a new way to lose the database). Pick A. The version-change branch was not measured (it needs a drill-catalog commit). The page's false sentence: R-1169. **NARROWED — 2026-10-09: the first half DELIVERED in controller 0.304.0. Slice 0 (the live measurement) NOT run: 9202 has no off-site, and a dump cannot be cut by hand inside an encrypted off-site copy on Tester 1; it needs a scratch off-site repo — a plan, not a finding.****NARROWED 2026-10-08 — the first half (D8 option C, the app held stopped) is built on controller main and ships tomorrow; what stays open is the second half, „put back exactly as it was” (option A: a pre-restore copy of the app's volumes and placed files, a fit check, disk for one copy; the slice-0 measurement on 9202 first).****OPEN — filed 2026-10-06****2026-10-06 night: verified in source, no code** — `offbox_reconstitute.go:758` (undo dump from the live DB), `:778-860` (files and volumes from the snapshot), `:804` (the snapshot's definition is written when its version differs), `:898` (the rollback loads the newer dump over the older volume; nothing writes the live definition back). Not a reorder fix: it needs R-638 option B (a rebuilding loader) or a pre-restore volume copy (disk cost; R-685's class). Which state a household gets after a failed off-site replay is the operator's call. Next: the 9202 measurement, then the design. | a design: R-638 option B (a loader that rebuilds instead of overlays) or a pre-restore volume copy | Measure it once on 9202 (a forced replay failure after an off-site restore over a migrated app); then a design for the operator | CC |
| **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):**`_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC |
| **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):**`_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected.**2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC |
| **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). | operator: print the sheet | Operator: print + store the sheet; then close | operator |
| **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar`**823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar`**308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/`**18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged****Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC |
| **R-231** | Backup & restore | P3 | **`/opt/backup/scripts/` on DooPlex is unversioned host state** — found 2026-08-06 while adding the auto-memory store to the backup set. No repository tracks the scripts that protect the recovery chain, so the edit made that day (`CLAUDE_MEMORY_DIR` in `backup-config.sh`, multi-path restic call in `backup-data.sh`) exists only on the box. This is the same class the part-2 session was closing, found inside the fix for it; the change is transcribed in `felhom.eu/workspace/README.md` so it is at least *recorded*. **Two related facts, both understating current safety:** the backup destination (`/mnt/5_hdd/backup`) is on the **same physical disk** as the workspace it protects, and the DooPlex backup set has **no off-site leg** (`sync-hetzner-backups.sh` is jarrs.eu and pulls *from* Hetzner *to* DooPlex). Bringing a root-owned production backup script under version control, and deciding what installs it, is its own scoped change. | **VERIFY** — **MEASURED 2026-10-13 — the due check is GREEN** (read by the sizing session, two channels): `systemctl show dooplex-backup.service` → `Result=success`, `ExecMainStatus=0`, ran 03:27:39–03:33:29 CEST; its journal ends „All backup components completed successfully!”, and the manifests leg (the header-token change) logged „Successfully backed up homelab-manifests” with the plain URL `https://gitea.dooplex.hu/admin/homelab-manifests` (no token in it). Left open, not closed: the row also carries the same-disk and no-off-site facts, which closing would drop — the owner decides. **VERIFY** — **2026-10-12 evening: the token left the clone URL** (operator yes in chat; homelab-manifests `1943a70`, installed by the role): `backup-manifests.sh` sends it as an `http.extraHeader` through `GIT_CONFIG_*` env; read-only proof: `git ls-remote` of the private repo with the header → HEAD, without → refused. The 2026-10-13 03:10 run is the test of both changes. **VERIFY** — **2026-10-12: in git and installed from it** (operator yes in chat, operator in the session): homelab-manifests `fdd324b`, `homelab-ansible/roles/dooplex_backup/` (11 scripts + the timer units, byte-identical to the live ones; no secret in any file; `install.sh` dry run / `--apply`). The install changed only the units' mode (744 → 644). Baseline: last night's run `Result=success`, exit 0 (`audits/answers-2026-10-12/s20-backup-scripts.txt`). **Seen by the commit's security review, NOT changed (a DooPlex behaviour change needs the operator's word):**`backup-manifests.sh` puts the Gitea token into the clone URL for the mirror run (visible in the process list for those seconds) and strips it from the stored remote afterwards; no token VALUE is in any file, and the repo is private (anonymous API read = 404). A header-based fetch (`git -c http.extraHeader=…` from the token file) would remove it — one line in the decision list. **READY** — operator ruling 2026-10-12 (S20, `09` §3 decision 215): yes — the scripts into a repo + an install script; the operator present for the install. **READY** — owner Viktor | — | Owner: close the row (the install from git is proven) and file the same-disk / no-off-site facts as their own row if they still hold | CC |
@@ -182,7 +182,6 @@ stopping line that lies.
| **R-878** | Backup & restore | P4 | **A catch-up (R-871) runs the database-dump leg in the DAY, and that leg stops an app with a volume for its copy — the household may notice the stop, and a large volume makes it longer.** MEASURED 2026-10-05 on demo-felhom: the catch-up at 08:25:02 stopped opengist, copied 182.5 KB, started it again — about 1 s, then a few seconds of `health: starting`; the night does exactly the same, unseen. Nothing measured for a large volume. Fix direction (if it matters): skip the volume copy of a running app in a DAYTIME catch-up and leave it to the next night, or warn. `audits/catchup-2026-10-05/partA/live-demo-felhom.txt` | **READY — owner: CC****2026-10-05 (burn-down night): NEEDS A LIVE MEASUREMENT** — a timed catch-up on 9202 for an app with a multi-GB volume, then the skip-or-warn choice. | — | measure a large volume first | CC |
| **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged****2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.****2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC |
| **R-1087** | Backup & restore | P4 | **Whole-box copies left on the OLD local target are never pruned after the target moves.** MEASURED 2026-10-15 (storage-identity build B space check): the N100's host root (`local`) holds three copies from 2026-07-28 (5.5–5.7 GB each, ≈ 16.8 GB); its agent's `local_backup_target` is now `felhom-backup` (the USB disk, retention 3), and retention prunes only the current target. On a small-root box such leftovers would starve the local copies (R-548). Evidence: `audits/storage-identity-buildB-2026-10-15/space/felhom-pve-9201.txt`. | OPEN | — | Decide whether the agent prunes its own `vzdump-lxc-<vmid>-*` files on a former target (after a newer copy exists on the new one), or the target move asks; until then, the N100's three July files can be removed by hand with the operator's word. | CC |
| **R-1162** | Backup & restore | P3 | **ep0's whole-guest off-site prune was a count (keep-last 2), so a compromised controller could push out the pre-compromise copy, slowly.** Operator decision (STATUS D18, the brief's D1 = A, 2026-10-19 08:24): one copy per ISO week, 2 weeks. Before: PBS's own dry run and an offline model of its selection both showed `keep-weekly 2` removing nothing on demo-hp (2026-10-08, 10-15 kept) or demo-felhom (10-06, 10-13 kept); nothing deleted by the dry runs (8 snapshots before and after). On the operator's „yes": `prune-demo-hp` and `prune-demo-felhom` changed to `keep-weekly 2` (schedule 03:30 unchanged), read back 07:49 UTC. DooPlex's copy of ep0 already keeps `keep-weekly 8` (decision 71) — unchanged. Households without a job: R-1165. | NARROWED 2026-10-19 — changed and read back; the first scheduled run is 2026-10-20 03:30 UTC | `audits/day-2026-10-19b/P1/` | Read the 2026-10-20 03:30 UTC prune task on ep0 (`proxmox-backup-manager task list`, `prunejob` for both namespaces: OK, `keep-weekly 2`, two snapshots each), then close | CC |
| **R-1169** | Backup & restore | P3 | **After a failed off-site restore that is held (`restore_mixed`), the restore page says the database is back as it was and the app's files and other data come from the backup — measured false on both halves.** 2026-10-19 on 9202 (R-893's measurement): a table only the backup had stayed in the database (37 tables against 36), and Rallly had no files or other volumes from the backup. The sentence (`err.backup.db_restore_failed_held_mixed`) should say only what is certain: the restore stopped half-way, the app is held stopped, nothing is lost yet, support will help. A text change in the controller (HU + EN). | OPEN — filed 2026-10-19 | `audits/night-2026-10-19/P3/README.md` | Next controller release: the sentence rewritten in both bundles, a fixture for the held-mixed page | CC |
## Storage & devices — 9 rows (P3 3, P4 6)
@@ -252,7 +251,7 @@ stopping line that lies.
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC****2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent``acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
| **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice****REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc/<pid>/fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC |
| **R-388** | Monitoring & notifications | P3 | **PRODUCT DECISION (not a defect): the customer notification model is the wrong shape, and the settings page grows by one toggle per detector.** The operator's framing, recorded verbatim 2026-08-23: *"A customer should be notified only about things they can act on or are responsible for — the drive they unplugged, the storage they filled. **A failed backup is our incident, not theirs.** The intended shape is that we detect it, we tell them we noticed and are dealing with it, and they are not handed an error they cannot solve. The subscription should feel like being looked after, not like being on call."* Today's page is the opposite shape — one switch per detector, and it **grew from 12 to 15 in a single session** (one new alarm plus two compound toggles split into four). That growth is the argument, not an aside: a page that grows per detector keeps asking a household to make engineering decisions. | **READY** — **2026-10-12 evening: slice 1 delivered** (operator Q1/Q2 „as picked", `09` §3 decisions 234–235): the page shows three group switches; the per-detector switches and the cooldown sit under „Részletek"; an „egyedi" group is saved back unchanged; the default list fills groups 1 and 3 (households with no saved list), group 2 stays „egyedi" (`app_start_failed` off, R-329). Tests red-proved. controller 0.309.0, delivered 2026-10-12 18:08 to demo-hp, demo-felhom and Tester 1 (`controller-0.309.0-delivery.txt`). **READY** — **2026-10-12: the design is written** (`08` §8.1): three group switches („A gépem és a meghajtóim", „Az alkalmazásaim", „Amit mi intézünk") mapped onto today's 18 detectors; option A picked; slice 1 changes no wire. Two questions for the operator (group 3 default; the cooldown field). **READY** — operator ruling 2026-10-12 (S5, `09` §3 decision 200): design now (one page: options, a pick, the first slice), build in slices later. **OPEN — DIRECTION, operator's call** | a decision on scope; nothing here is a bug | Slice 2 (hub): group 3's mails say „we noticed X and are handling it" instead of the raw error | CC (+ operator: two answers) |
| **R-435** | Monitoring & notifications | P3 | **The snapshot-drop detector is blind to a single-app deletion — the exact shape the box can produce.**`snapshotDropFraction = 0.5` and `snapshotDropFloor = 5` (`hub/internal/monitor/offsite.go`) require a fall of MORE than half the previous count. demo-hp's baseline is **69** across 9 apps, so ~35 snapshots must go before it speaks; **one app's tag is ~9 and is invisible.**`offbox.go:1388` runs `forget --prune`**grouped by host,tags** — a per-tag wipe is precisely the shape a faulty retention or a targeted deletion produces. **This is deliberate, not accidental:** the constant's own comment argues the insensitivity, and *"a detector that cries wolf is switched off within a fortnight"* is a lesson this project paid for. **So this row is NOT a demand to lower the threshold.** It is a demand that the blind spot be written where the operator reads it, because "an unexplained fall is noticed within a day" (R-431, and `STATUS.md`) is true only of falls above half. **Discovered by arithmetic while planning the drill's Phase 2/3 pairing, which could not have worked:** Phase 2 deletes one app and Phase 3 expects the alarm to fire. **LIMITATION NOW WRITTEN INTO THE ALARM'S OWN DOCUMENTATION, hub v0.111.1** — the comment above `snapshotDropFraction` in `hub/internal/monitor/offsite.go` now states what the detector does NOT see, with the demo-hp arithmetic and the `forget --prune` grouping that makes the blind spot sit on the most likely single-app failure. **It also says explicitly that the numbers must NOT be lowered to "fix" this** and that per-app detection needs a SECOND signal keyed on the per-tag count. **The row stays OPEN because documenting a blind spot is not covering it** — and because `STATUS.md` and R-431 both still say "noticed within a day", which is true only of falls above half. **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-435.md` — the box's key can no longer delete (decisions 68–69); deletion through any other credential still needs >50 % to alarm. Question D7 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D7 (`09` §3 decision 191):** yes — option C, an error mail when even one snapshot disappears outside a hub-opened clean-up window (pinned tiers). | **VERIFY** — **2026-10-12 read: not provable today.** No mail for a snapshot drop reached the operator since 2026-10-11 (Gmail), but the window itself could not be read: no hub page shows `offsite_windows`, Tester 1's controller log restarted with 0.308.0 at 13:49Z, and the guest-agent path to its VM did not answer. Re-dated after the next weekly clean-up. **VERIFY — 2026-10-09: DELIVERED in hub 0.144.0. Live proof NOT run: a window is granted only when the box asks for one, and Tester 1's weekly clean-up is next due ~2026-10-12 (last window 2026-10-05); prove it then (no alarm inside the window; the hub's window row as control).****VERIFY — built on hub main 2026-10-08 (D7, decision 191), ships tomorrow; closes after the 9202 proof (a hand-granted window removes N, no alarm; the hub's window row as the control).****OPEN — documented in code v0.111.1; the coverage gap itself is unclosed****2026-10-05 (burn-down night): NEEDS A DESIGN.** Covering one-app deletions needs per-app snapshot counts from the controller and a per-tag threshold measured on real counts — two repos, a mechanism nobody has measured. | — | 2026-10-20: read Tester 1's controller log (window opened/closed, count before/after) and the operator mailbox (no drop alarm) the morning after its weekly clean-up | CC |
| **R-435** | Monitoring & notifications | P3 | **The snapshot-drop detector is blind to a single-app deletion — the exact shape the box can produce.**`snapshotDropFraction = 0.5` and `snapshotDropFloor = 5` (`hub/internal/monitor/offsite.go`) require a fall of MORE than half the previous count. demo-hp's baseline is **69** across 9 apps, so ~35 snapshots must go before it speaks; **one app's tag is ~9 and is invisible.**`offbox.go:1388` runs `forget --prune`**grouped by host,tags** — a per-tag wipe is precisely the shape a faulty retention or a targeted deletion produces. **This is deliberate, not accidental:** the constant's own comment argues the insensitivity, and *"a detector that cries wolf is switched off within a fortnight"* is a lesson this project paid for. **So this row is NOT a demand to lower the threshold.** It is a demand that the blind spot be written where the operator reads it, because "an unexplained fall is noticed within a day" (R-431, and `STATUS.md`) is true only of falls above half. **Discovered by arithmetic while planning the drill's Phase 2/3 pairing, which could not have worked:** Phase 2 deletes one app and Phase 3 expects the alarm to fire. **LIMITATION NOW WRITTEN INTO THE ALARM'S OWN DOCUMENTATION, hub v0.111.1** — the comment above `snapshotDropFraction` in `hub/internal/monitor/offsite.go` now states what the detector does NOT see, with the demo-hp arithmetic and the `forget --prune` grouping that makes the blind spot sit on the most likely single-app failure. **It also says explicitly that the numbers must NOT be lowered to "fix" this** and that per-app detection needs a SECOND signal keyed on the per-tag count. **The row stays OPEN because documenting a blind spot is not covering it** — and because `STATUS.md` and R-431 both still say "noticed within a day", which is true only of falls above half. **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-435.md` — the box's key can no longer delete (decisions 68–69); deletion through any other credential still needs >50 % to alarm. Question D7 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D7 (`09` §3 decision 191):** yes — option C, an error mail when even one snapshot disappears outside a hub-opened clean-up window (pinned tiers). | **VERIFY** — **2026-10-12 read: not provable today.** No mail for a snapshot drop reached the operator since 2026-10-11 (Gmail), but the window itself could not be read: no hub page shows `offsite_windows`, Tester 1's controller log restarted with 0.308.0 at 13:49Z, and the guest-agent path to its VM did not answer. Re-dated after the next weekly clean-up. **VERIFY — 2026-10-09: DELIVERED in hub 0.144.0. Live proof NOT run: a window is granted only when the box asks for one, and Tester 1's weekly clean-up is next due ~2026-10-12 (last window 2026-10-05); prove it then (no alarm inside the window; the hub's window row as control).****VERIFY — built on hub main 2026-10-08 (D7, decision 191), ships tomorrow; closes after the 9202 proof (a hand-granted window removes N, no alarm; the hub's window row as the control).****OPEN — documented in code v0.111.1; the coverage gap itself is unclosed****2026-10-05 (burn-down night): NEEDS A DESIGN.** Covering one-app deletions needs per-app snapshot counts from the controller and a per-tag threshold measured on real counts — two repos, a mechanism nobody has measured.**2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the night session that reached it was not briefed for this read (a Tester 1 log and the mailbox); the window row and the mails stay readable a day later. | — | 2026-10-20: read Tester 1's controller log (window opened/closed, count before/after) and the operator mailbox (no drop alarm) the morning after its weekly clean-up | CC |
| **R-724** | Monitoring & notifications | P3 | **[P3-LOW] The status pages disagree with each other in small ways a household notices.** MEASURED 2026-09-29 on a fresh box: Beállítások reads „Mentés ütemezés 02:30 / 03:00" while Biztonsági mentés says 02:30 / 03:30 / 04:15 / 04:30–08:30; Beállítások shows a raw `2026-09-29T19:22:30Z`; its „Helyi cím (LAN)" and „Átjáró" read „nem állapítható meg" on the page the household is asked to read out for remote help; the backups overview still says „Következő mentés — 0 órája" (the age of the LAST run, under the word "next" — noted 2026-09-14, never filed); the dashboard shows the backup at 19:33 where the backup pages say 21:33 (R-500, still reproducing). **FIXED 2026-09-30 (controller v0.283.0):** Beállítások shows the real schedule and a local time; the dashboard's last backup is local time (R-500); „az előző N órája készült" replaces „0 órája". **NARROWED — remaining:** „Helyi cím (LAN)" / „Átjáró" read through the file-sharing container, so a box without it says „nem állapítható meg" — not a text fix (the read needs another path). | **NARROWED — the LAN/gateway read only; owner: CC****2026-10-06 night: confirmed not a controller-only fix** — the only host-network container is samba (`infra/samba.go:123`) and the agent's local API has no guest-network route; the fix is a new agent route (e.g. the PVE lxc interfaces read) + a controller reader — a two-repo design. | — | — | CC |
| **R-285** | Monitoring & notifications | P4 | **A planned, supervised reinstall pages the operator as if the machine had died — there is no notion of expected downtime anywhere.** During the 2026-08-09 rehearsal the hub sent, all `status: sent` to the operator channel: `host_stale` 08:58 UTC, `node_stale` 09:00, **`host_down` 09:28 (error)**, **`node_down` 09:30 (error)**, `host_leaf_changed` 09:31, `host_recovered` 09:31, `node_recovered` 09:34, `offsite_delivery_stuck` 09:34 — eight operator mails for work that was deliberate, attended and announced. **This is the OPPOSITE gap from the one R-281 filed:** the alarms are not missing, they are indiscriminate. `host_stale` at 30 min and `host_down` at 60 min (`monitor/host_staleness.go:22-23`, `downAfter = 2 * threshold`) cannot distinguish a wiped-on-purpose box from a dead one, and `host_leaf_changed` firing on a reinstall is correct-but-expected. **Note the interaction with the mute used on 2026-08-09 evening:** blocking a customer silences everything, so today the only two settings are *page me for planned work* and *tell me nothing at all*. **What is owed is a middle:** a maintenance window, or an operator-set expected-downtime flag, that suppresses staleness and leaf-change while leaving genuine faults audible | **READY (M) — NEW 2026-08-09** | — | The evidence is the operator's mailbox plus `events`/`notification_log` for 2026-08-09 | CC |
| **R-1005** | Monitoring & notifications | P4 | **A recovery mail says how long the box was „stale", not how long it was gone: after a 51-minute power cut demo-hp's `host_recovered` read „was stale for 8m" and `node_recovered` „23m".** An operator reading only the recovery mail under-reads the outage (`audits/night-power-2026-10-13/` R6). Same kind, the other way: the off-site mail of 2026-10-14 02:18 was titled „3 of 3 apps failed to back up (offsite run)" while 11 other apps were backed up in that run — only the 3 flagged apps were counted. | **READY** — filed 2026-10-14 | — | CC: add the time since the last report before the gap to both recovery messages, and count every app of the run in the off-site subject, with a test that asserts each text; next hub release | CC |
@@ -340,8 +339,8 @@ stopping line that lies.
the R-row. Duplicating them here would create the second source this design avoids. -->
| item | due (UTC) | what to measure |
|---|---|---|
| R-895 | 2026-10-20 | the first verdict of the hub's own off-site snapshot check (hub 0.158.0; 2026-10-19 07:10 is the baseline): read the 07:10 log lines (count, newest upload vs the box's last run) and whether either alarm fired; detail in the R-895 row |
| R-435 | 2026-10-20 | Tester 1's weekly off-site clean-up ran inside a hub window with no drop alarm (box log + mailbox); detail in the R-435 row |
| R-895 | 2026-10-21 | the first verdict of the hub's own off-site snapshot check (hub 0.158.0; 2026-10-19 07:10 is the baseline): read the 07:10 log lines (count, newest upload vs the box's last run) and whether either alarm fired; detail in the R-895 row |
| R-435 | 2026-10-21 | Tester 1's weekly off-site clean-up ran inside a hub window with no drop alarm (box log + mailbox); detail in the R-435 row |
| R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row |
| R-1035 | 2026-11-11 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row |
| R-528 | 2026-11-12 | no failed oom_check / unexplained app_oom since 2026-10-12; detail in the R-528 row |
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
