diff --git a/CONTEXT.md b/CONTEXT.md index 68df1214..5a4dd1f9 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,17 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-05 (day) — the night's fixes (controller v0.294.0, agent v0.144.0 + v0.144.1, golden 0.294.0 vouched with agent +> 0.144.1, min_agent 0.131.0).** Rulings 100–103 (`09` §3: Tester 1's CF tokens NOT rotated → R-870; A1 by day on demo-hp +> only, with the operator's go; Tester 2 is a laptop off at night; re-sign Tester 2 only if online — it never was). +> CC decisions 104–108, *operator may reverse*: guard "young" = keep-daily CALENDAR days (`offbox_window.go`, built from +> `keepDaily`/`keepWeekly`/`keepMonthly`); `dockerexec.BeginImageWork`/`TryImageCleanup` for every pulling compose verb; +> the wrapper's kept report copy (`report---apply.json`) + `Leg.SendUnsentLoop` (5 min) + `pass.lock`; the +> selftest's saved block (`os-update-block.json`); a second agent release. Closed R-95, R-863..R-869. Opened R-870..R-876 +> (R-871 the operator's decision: a box off at night; **R-876 P2: after a mid-update crash the wrapper's repair misses +> dpkg's update journal — fix next**). Floors: demo-hp, demo-felhom, tester-1 at 0.294.0 (MinAgent 0.131.0); global +> unchanged. Report: `REPORT-night-fixes-2026-10-05.md`. + > **2026-10-04 (night) — R-840 BUILT (agent v0.143.0, hub v0.133.0, installer 1.31.0), R-859 + R-860 FIXED (hub v0.133.0, > controller v0.293.0), golden 0.293.0 vouched, drill-r50 removed.** The config bundle (`11` §5.4.2): one table of 22 > root paths in `felhom-os-apply`; signed `agent_config_update`; the trust root never a bundle path; installer installs the diff --git a/REPORT-night-fixes-2026-10-05.md b/REPORT-night-fixes-2026-10-05.md new file mode 100644 index 00000000..b29082c6 --- /dev/null +++ b/REPORT-night-fixes-2026-10-05.md @@ -0,0 +1,81 @@ +# REPORT — the night's fixes, the power cut by day, a box that is off at night (2026-10-05) + +Brief: "fix what the 2026-10-04 night found …" (operator, 2026-10-05). Evidence: `documentation/audits/night-fixes-2026-10-05/`. +Architecture read before the claims: `07-backup-architecture.md` (§6.1, the Tier-3 row), `11-os-updates.md` (§5.4.1, §8), +`09` §3 (decisions 53, 68–74, 81, 86), `audits/offsite-append-only-2026-10-03/DESIGN.md`. + +## 1. The Part table + +| Part | State | Note | +|---|---|---| +| §1 rulings recorded first | **done** | `09` decisions 100–103; R-870 (tokens, like R-831); Tester 2's page in `runbooks/target-selection.md` | +| A — clean-up guard (R-867) | **done** | controller v0.294.0; tests run restic 0.14.0's policy (proven identical to the binary); 5 red-proofs; live windows on both demo boxes with counts; R-867 and R-95 closed (DUE-CHECKS entry removed) | +| B — image clean-up race (R-863, R-864) | **done** | one lock for every pulling compose verb; the night's shape reproduced in a test and live on 9202 (fired at minute 3, skipped, install OK first time) | +| C — R8 real download (R-865) | **changed** | fix + tests + red-proof done. Live: the INSTALLED wrapper's measure read 12 802 456 B on demo-hp. **No R8 refusal line was produced**: it needs < 500 MB free on demo-hp's guest = 28.5 GB written into a thin pool with 18.7 GB free — it would have stopped every guest | +| D — R-868, R-869, R-866 | **done, with a second agent release** | R-868's first fix (v0.144.0) was measured NOT to work live (broken stderr pipe); v0.144.1 fixed it and the A5 shape then delivered one `applied` report. R-866 live with the hub blackholed. R-869 test + red-proof | +| E — power cut mid-update (A1 by day) | **done** | operator's go; crash 06:13:55 UTC, back by itself in 37 s; **the next pass failed** → R-876 (P2); by-hand repair; demo-hp left with every package current | +| F — a box off at night (spike) | **done** | read only; `partF/FINDINGS.md`; no architecture covers it → R-871; R-872..R-874; STATUS decision A/B | +| G — releases, golden, records | **done** | controller v0.294.0; agent v0.144.0 + v0.144.1 (decision 108); golden 0.294.0 baked + vouched (gate OK); `07`, `11`, `00`, `09` updated | + +## 2. Claims in the brief that turned out wrong (or only partly true) + +1. **"The policy's constants can drive the guard's line"** — true, and done — but with a cost the brief did not name: a + line derived from keep-daily (calendar days) can no longer catch a past-dated gap-fill that steers a 7–8-day-old + keep, which the 8-day line caught while refusing every honest window. Past-dated fakes can never make the policy drop + a snapshot inside the last keep-daily calendar days, so the line now catches only the skew-window shape. The rest is + R-822's residual, bounded by the cap and the hub's count check (decision 104). +2. **"The same race exists in the update and restore paths"** — the update path: NO, it was already guarded (no pass + while any app is updating, since v0.284). The restore and undo paths: exposed in principle (they pull inside + `compose up`), not measured; they now hold the same lock. +3. **"Dropping `-s` downloads nothing"** — TRUE, measured on 9202 (archive cache 10 → 10 files, versions unchanged). +4. **"A box off at W runs nothing when it comes back"** — mostly true: the database dumps, second copy, off-site copy and + app updates never catch up; but the whole-guest backup DOES catch up (48 h safety valve, about every 2 days for an + evening-only box), and the OS leg follows it by day under the `night` label. +5. **"The night A5 showed the wrapper finished"** (R-868's premise, and v0.144.0's design) — the packages were installed, + but the wrapper process itself died on its first log line after the agent's death. Found only by running it live. +6. Part E: "the next pass must repair dpkg first and finish" — it does NOT (R-876). + +## 3. What was proven, with numbers + +- **Part A.** Tests: the measured shape (7 d + 5 s → allowed), 60 nights × 3 apps with two same-day manual runs across a + month boundary (never refused, ≥ 45 removed), weekly windows, the skew-window fake (refused), the lab's 13 future fakes + (refused). Red-proofs: 8-day line back → 3 fail; each refusal dropped → its test fails. Lab: restic 0.14.0 binary vs + the test simulation over 92 snapshots: identical 72 removals. **Live:** demo-felhom 15 snapshots, predicted removals + `92e49f62`, `343d57a5` → window 5 removed exactly those (16 → 14 incl. the run's new one); demo-hp 136 snapshots, + predicted 18 → window 6 removed exactly those (145 → 127). Hub rows `pruned`, no event, no mail, key audit 0 findings. +- **Part B.** Live on 9202: controller start 05:28:35, install 05:31:15, clean-up 05:31:36 "skipped — … pulling images + now", BookStack deployed 05:31:54 (35.5 s), retry 05:33:36 ran (0 deleted). Teardown through the product verified. +- **Part C.** `download_bytes = 12802456 B` from the installed wrapper on demo-hp (was 0). +- **Part D.** R-866: `block=SAVED(2026-10-05T05:23:26Z; hub unreachable: … connect: invalid argument)`. R-868 (v0.144.1): + killed 06:04:00, wrapper `DONE upgraded=13`, kept copy, hub report id 63 `applied` (13) at 06:09:05, once. +- **Part E.** See `11` §8.4: back in 37 s; apps healthy within 4 min; household saw one timeline line; one operator mail + `os_update_failed` (true); next pass FAILED (R-876); after `dpkg --configure -a` the pass installed 12; the guest's + package list equals the pre-test one (279 lines). The first crash attempt did not fire (my watcher's pattern missed + apt's dpkg call; the pass installed normally) — rolled back again and repeated. +- **Part F.** `audits/night-fixes-2026-10-05/partF/FINDINGS.md`; two claims re-checked in source by me. + +## 4. Rows + +Register before **341**, after **340**. Closed (8): R-95, R-863, R-864, R-865, R-866, R-867, R-868, R-869. Opened (7): +R-870 (tokens, operator), R-871 (a box off at night — the operator's decision), R-872 (P2, no missed-backup alarm for a +box down at 05:00), R-873 (nightly "cannot be reached" mail), R-874 (restore-test never runs on short sessions), +R-875 (P4, kept-report reason text), **R-876 (P2, the repair misses dpkg's update journal)**. STATUS updated. +`unproven.py --summary`: unchanged — 55 claims, 35 not walked (no number moved). + +## 5. Teardown, three layers + +- **Machines:** 9202: BookStack removed through the product (no container, stack dir, volume); controller 0.294.0 (set by + hand, allowed there). demo-hp 9201: every package current, list identical to before; my scripts and logs in `/root` + removed after copying. The bake VM: CT 9100 destroyed, token shredded, reverted to `virgin`, qemu gone. +- **Hosts:** the blackhole routes on felhom-pve removed (in the same script). Nothing provisioned. +- **Hub:** two one-shot clean-up grants (consumed); floors for demo-hp, demo-felhom, tester-1 → 0.294.0; artifacts + vouched (agent 0.144.1, golden 0.294.0, min_agent 0.131.0); 12 signed jobs (3× agent 0.144.0, 3× bundle 0.144.0, + 3× agent 0.144.1, 3× bundle 0.144.1). Tester 2: read only, nothing sent (offline all session). +- Scratch secrets (hub password, hub key, controller password, deploy secrets) shredded at the end. + +## 6. Notes + +- The hub pod log prints a customer's e-mail address in "Customer email sent to …" lines (seen by the Part F helper; not + recorded anywhere). +- demo-hp's System page read "no running customer guest" / "unknown" for several minutes after the crash although the + guest ran — R-853 (facts late after a boot), not a new defect. diff --git a/STATUS.md b/STATUS.md index 7c8b4b81..f423ce6f 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,11 +1,62 @@ # STATUS — what works, what's broken, what's next -**Ready for the first real tester (Tester-2): yes. Tester 2 is installed, but offline since 2026-10-04 20:06 local.** +**Ready for the first real tester (Tester-2): yes. Tester 2 is a laptop that is switched off at night (your word, +2026-10-05) — it was offline all session; nothing was sent to it.** -**Updated 2026-10-05 (morning, after the OS-update night): every box of ours healthy; the ruled undo proven (48 s); -decision 78 proven on a fresh Tester 1 box; two P2 defects found (a new box's first app install can fail; the off-site -clean-up refuses normal retention and mails an error weekly). One test (a power cut mid-update) not run — needs your -permission. Morning note: `documentation/audits/night-2026-10-04/MORNING-NOTE.md`.** +**Updated 2026-10-05 (day, the night's fixes): every box of ours healthy. Fixed and proven live: the off-site clean-up +now deletes old copies (both demo boxes), a new box's first app install, the update's disk-space check, a killed +update's lost report. The power cut in the middle of an update was tested on demo-hp with your go: the box came back by +itself in 37 s, but the next update failed until I ran one command by hand — filed (R-876), fix next session. +One decision for you below (a box that is off at night). Report: `REPORT-night-fixes-2026-10-05.md`.** + +## Today (2026-10-05, day): the night's fixes, the power cut by day, a box that is off at night + +**Decisions I took myself (you may reverse each — `09` decisions 104–108):** +- The off-site clean-up's safety line is now "the last 7 calendar days" — the same number the clean-up keeps — not + "8 days old". It can never block an honest clean-up again. +- The image clean-up now waits while ANY app install, update, restore or undo is downloading — not only installs. +- A killed update keeps its report on the box until the hub has it; the agent looks for such reports every 5 minutes. +- **A second agent release today (0.144.1)**, against "one release per repo": the first fix for the lost report was + proven NOT to work on demo-hp, and shipping it as it was would have been worse. + +**What works now (proven live):** +- **The weekly off-site clean-up really deletes old backups.** One clean-up each by hand: demo-felhom 16 → 14, demo-hp + 145 → 127 — exactly the backups I predicted. No error mail, the hub's count check quiet, the key files clean. + This also closes R-95 (the box can no longer delete its own off-site history, and clean-up now works). +- **A new box's first app install works the first time:** the image clean-up met an install at minute 3 on the scratch + box, waited, and BookStack installed first try. A failed install now logs its real reason. +- **The update's disk-space check counts the real download** (12.8 MB for 13 packages; it counted 0 before). +- **A killed update still reports to the hub** (5 minutes later, once). The debug update runs with the hub away. +- Controller 0.294.0 on demo-hp, demo-felhom and Tester 1 (floor per customer; Tester 2 not moved). Agent 0.144.1 + and its root files on all three. New-install image 0.294.0 baked and approved. + +**Found today (filed, not fixed):** +- **After a power cut in the middle of an update, every later update fails until someone runs one command on the box** + (R-876, P2). The box itself comes back fine. Until the fix: `runbooks/crash-guard.md` has the command. I fix it next. +- A box that is off at night (below): no catch-up, no missed-backup alarm (R-872), a "server cannot be reached" mail to + the household every night (R-873), restore-tests never run (R-874). + +**Needs you:** + +1. **A box that is off every night (Tester 2) — what does the product promise?** Today such a box never gets its + nightly database backups, second copy or off-site copy; the whole-box backup runs only about every 2 days; no + alarm says so; the household is mailed "your server cannot be reached" every night. No design document covers it + (R-871). + - **A (my pick): a missed night runs once when the box comes back.** The database backups, the second copy and + the off-site copy run a few minutes after the box is on again (they take seconds to minutes; whether + any of them pauses an app is to be measured in the design — the whole-box backup, which does pause apps, already + has its own catch-up). App and system updates still + wait for a night. Costs: a design section and one controller release; the household may notice a busy disk for a + few minutes after switching on. + - **B: say plainly that the box must stay on at night.** The setup guide and the box's backup page say it; the + missed-night alarm fires after 2 nights off. Costs: wording + one alarm; a laptop household gets an alarm it + cannot fix except by changing habits. + - **If you do nothing:** Tester 2 keeps having no database or off-site backup, nobody is told, and the household + keeps getting the nightly "cannot be reached" mail. A restore-test alarm will fire around 2026-10-11. +2. **Tester 2's one-time step** — unchanged from yesterday (below). If you wait, it keeps working; it just cannot get + new root files. + +**Recorded, not decisions:** Tester 1's Cloudflare tokens are NOT rotated (your ruling; R-870 has the steps). ## Today (2026-10-04, night): root files for installed boxes, test approvals, Docker self-repair diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 245de38a..0d5d5609 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -232,7 +232,7 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis | **The hub reports LOSS OF VISIBILITY into either off-site store (not just how full it is)** | hub **v0.106.0** (R-339) | **IMPLEMENTED — deliberately NOT proven-live** | Both box checkers count consecutive failed fetch windows and emit `pbsdr_box_unreachable` / `offsite_box_unreachable` (severity `warning`) past a default 3 windows (≈30–45 min), each with a paired `*_recovered` all-clear routed via `recoveredPairedDownTypes` — required because the recoveries are severity `info`, which `severityNotifies` drops. Scopes stay customer-less (`pbsdr-box` / `pool-box`) → operator channel only. Fill logic untouched: a degraded read still drives no band transition. Evidence: `internal/monitor/box_reachability_test.go` + the cross-package wiring test in `internal/notify/`, which asserts an actual operator mail rather than a map entry. **Filed BECAUSE of a measured gap**, not a hypothesis: the 2026-08-18 ep0 outage ran 9 h 37 m with the hub silent | **The gap that remains is R-340**, and it is not small: the ep0 read is the `usage` op, which rides the LOCAL API daemon — the daemon that incident explicitly cleared — so this check would have shown GREEN for that entire outage. It closes "ep0 is unreachable as a host"; it does not close what actually happened. **No live or constructed outage has exercised the emit path**, and one cannot be manufactured against ep0 (Tier 2, protected) | | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | 07-13 closing bundle | | | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | 07-13 | | -| Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0 | **PARTIAL — the GUEST and HOST Debian fast lanes and the DOCKER engine slow lane are PROVEN-LIVE (2026-10-04), with the System page, the fleet view and the alarms; the KERNEL lane is MISSING** | Guest: `audits/os-guest-lane-2026-10-04/`. Host + fleet + alarms: `audits/os-host-lane-2026-10-04/`. Docker + System page + crash guard: `audits/os-docker-crash-2026-10-04/` — live-restore on with the same container ids on every box; Docker 29.8.2 on both demo boxes; operator-approved Docker release; a signed undo and a signed ring-1 step; a replay refused; the crash guard restarted demo-hp twice and kept it off the third time. Design `architecture/11-os-updates.md` §5.8, §5.9, §8 | **No automatic undo** (guest: last night's backup; host: by-hand runbook; Docker: a signed undo job); existing boxes get root-owned files by the signed config bundle since 2026-10-04 (R-840 CLOSED; a box from before agent 0.143.0 needs one by-hand bootstrap — Tester 2: R-862; `audits/r840-config-bundle-2026-10-04/`); test approvals now end with the test (R-859); the agent's sudoers is root-equivalent (R-861); the kernel lane (R-836); facts reach the hub late after a boot (R-853) | +| Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0 | **PARTIAL — the GUEST and HOST Debian fast lanes and the DOCKER engine slow lane are PROVEN-LIVE (2026-10-04), with the System page, the fleet view and the alarms; the KERNEL lane is MISSING** | Guest: `audits/os-guest-lane-2026-10-04/`. Host + fleet + alarms: `audits/os-host-lane-2026-10-04/`. Docker + System page + crash guard: `audits/os-docker-crash-2026-10-04/` — live-restore on with the same container ids on every box; Docker 29.8.2 on both demo boxes; operator-approved Docker release; a signed undo and a signed ring-1 step; a replay refused; the crash guard restarted demo-hp twice and kept it off the third time. Design `architecture/11-os-updates.md` §5.8, §5.9, §8 | **No automatic undo** (guest: last night's backup; host: by-hand runbook; Docker: a signed undo job); existing boxes get root-owned files by the signed config bundle since 2026-10-04 (R-840 CLOSED; a box from before agent 0.143.0 needs one by-hand bootstrap — Tester 2: R-862; `audits/r840-config-bundle-2026-10-04/`); test approvals now end with the test (R-859); the agent's sudoers is root-equivalent (R-861); the kernel lane (R-836); facts reach the hub late after a boot (R-853). **2026-10-05 (agent v0.144.1):** R8 measures the real download (R-865); a killed pass still reports (R-868, live); the debug pass runs with the hub away (R-866, live); **a power cut mid-update was proven by day on demo-hp — the box came back by itself in 37 s, but the next pass fails until `dpkg --configure -a` is run by hand (R-876, P2, open)** — `audits/night-fixes-2026-10-05/` | | **An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps** | controller **v0.259.0** + hub **v0.119.0** + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE on 0.258.0 with one blocker; THE BLOCKER IS FIXED AND PROVEN, THE WALK IS NOT REPEATED** | `audits/DRILL-first-hour-en-0258-2026-09-20.md` — a fresh install 2026-09-20, one intervention (R-494), stop rule not reached. Then `audits/i18n-closing-2026-09-21/live/` — the three blockers fixed and each proven on a live box or in the operator's inbox: the claim page answers English through the real cookie path; the Backup page's tier names follow the language; and the setup mail carries **four plain-ASCII English words** where the drill's carried `képző-szkítia-ásatás`, one day apart in the same inbox. | **R-596, R-597 and R-598 are CLOSED.** What this row still does NOT claim: **the fixed journey has not been walked end to end by a stranger on a fresh install.** Three fixes proven at the endpoint are not an hour proven by a person, and this project's own rule is that fixes are not a journey (see the recovery-journey row). **Also not walked:** the recovery code (needs ep0), backup/restore/remove/power-cut (proven 2026-09-14), and the two Backup-page *warnings* themselves — guest 9201 is healthy and a healthy box renders none, so they are covered by handler render tests, not live. **Verdict: nothing known now stands between an English-speaking tester and their box — and that is a different sentence from "the walk passed".** | | **A deletion of a customer's off-site history is NOTICED within a day** | hub **v0.111.0** (R-431) | **IMPLEMENTED — not yet PROVEN-LIVE** | 09-01 | `hub/internal/monitor/offsite.go` — third signal beside FILL and STALENESS. **On the hub deliberately:** a detector on the box is one the deletion can silence. Alarms when the reported count falls by more than HALF and by at least 5, guarded by `StatsKnown` (R-331), the declared `State` (R-204) and run success (R-100). **Threshold reasoned, not invented:** over 12 898 reports every decrease lands on ZERO and predates `stats_known`; in the 380-report `stats_known` window there are none. **ACCEPTANCE: 9 009 real points replayed → ZERO alarms** (`offsite_r431_test.go`, fixture committed). **What PROVEN-LIVE would need and this does NOT have:** a real drop observed on a live box producing a real mail — the live firing done at ship time was driven through the hub's own path with synthetic counts, which is an end-to-end delivery proof, not a proof that a genuine deletion is caught. | diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 5e3dc089..b8e3bc7f 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -1138,7 +1138,7 @@ crosses the line — **R-158**. | 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human | | 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** **`[BETA-DEFERRED]`** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) | | 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** **`[BETA-DEFERRED]`** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) **`[FACT]` 2026-10-03 (decision 70): ep0's datastore now has a nightly copy on DooPlex** (PBS pull-sync `ep0-felhom-offsite`, `remove-vanished false`, weekly verify, failures mailed to the operator); first pull 201 s, 12 GB, 4 of 4 snapshots, matching ep0 namespace for namespace. Restore route: `runbooks/ep0-datastore-copy.md` — never walked. `audits/offsite-lock-build-2026-10-03/partF/`. **`[FACT]` 2026-10-04: the restore route from the DooPlex copy was WALKED** on demo-hp (scratch VMID): listed in 2 s, restored in 186 s (15 GB logical), data read by `pct mount`; the restored config carries `onboot: 1` and the host's real drive binds — strip them on a restore beside the original (R-834). The copy now keeps 8 weekly copies (decision 71). `runbooks/ep0-datastore-copy.md` route 1. | -| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recovered, so nothing was timed.** `audits/evidence-drill-r95-recovery-2026-09-01/`. **`[FACT]` 2026-10-03 (decisions 68–69, hub v0.127.0, controller v0.289.1): THE BOX CAN NO LONGER DELETE ITS RESTIC HISTORY.** Both demo boxes run on an append-only key the hub pinned; a delete from the box is refused (`403 Forbidden`, measured on each, counts 13→13 and 100→100); the box never receives the sub-account password (the old endpoint answers 410, measured with demo-hp's own key); the hub reads every key file daily. Retention runs only in a hub-opened window behind a fake-snapshot guard — mechanics proven live (window 1 on demo-hp: opened, guard refused, closed in 3 s, operator mailed); **a real prune inside a window is NOT yet proven** (R-824). Residual: an add-only attacker can still fill the quota and plant past-dated snapshots (R-822). `audits/offsite-lock-build-2026-10-03/`. **`[FACT]` 2026-10-04 (controller v0.290.0, hub v0.128.0):** the guard now skips same-day-superseded young snapshots instead of refusing; window 2 on demo-hp ran without refusal and removed nothing (127→127 — every candidate was a young same-day copy); weekly windows are ON fleet-wide. **A window that actually removes snapshots has still not been observed** (first candidates age past 8 days around 2026-10-11). The household's set-aside deletion is the hub's after 7 days (decision 74), proven live on a planted scratch dir. | +| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recovered, so nothing was timed.** `audits/evidence-drill-r95-recovery-2026-09-01/`. **`[FACT]` 2026-10-03 (decisions 68–69, hub v0.127.0, controller v0.289.1): THE BOX CAN NO LONGER DELETE ITS RESTIC HISTORY.** Both demo boxes run on an append-only key the hub pinned; a delete from the box is refused (`403 Forbidden`, measured on each, counts 13→13 and 100→100); the box never receives the sub-account password (the old endpoint answers 410, measured with demo-hp's own key); the hub reads every key file daily. Retention runs only in a hub-opened window behind a fake-snapshot guard — mechanics proven live (window 1 on demo-hp: opened, guard refused, closed in 3 s, operator mailed); **a real prune inside a window is NOT yet proven** (R-824). Residual: an add-only attacker can still fill the quota and plant past-dated snapshots (R-822). `audits/offsite-lock-build-2026-10-03/`. **`[FACT]` 2026-10-04 (controller v0.290.0, hub v0.128.0):** the guard now skips same-day-superseded young snapshots instead of refusing; window 2 on demo-hp ran without refusal and removed nothing (127→127 — every candidate was a young same-day copy); weekly windows are ON fleet-wide. **A window that actually removes snapshots has still not been observed** (first candidates age past 8 days around 2026-10-11). The household's set-aside deletion is the hub's after 7 days (decision 74), proven live on a planted scratch dir. **`[FACT]` 2026-10-05 (controller v0.294.0, R-867 CLOSED): A WINDOW THAT REMOVES SNAPSHOTS IS OBSERVED, on both demo boxes.** The guard's line is now built from the policy's own constants (keep-daily 7 CALENDAR days, `09` decision 104) — the 8-day age line sat inside the keep window and refused every honest window (demo-felhom 2026-10-05 02:15 UTC, an error mail). One window each, opened by the operator's one-shot grant and run by the household's "run now": demo-felhom 16 → 14, demo-hp 145 → 127 — the removed snapshots are exactly those the policy predicted; the hub's rows read `pruned` with the same counts, no drop event, no mail, the key files clean. The UNATTENDED weekly window has not yet been observed removing (next due ~2026-10-11/12). Residual unchanged: R-822 (past-dated gap-fills steer older keeps). `audits/night-fixes-2026-10-05/partA/`. | | 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) | | 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** **`[BETA-DEFERRED]`** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) | | 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** **`[BETA-DEFERRED]`** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D **`[FACT]` 2026-10-03:** the PBS (whole-guest) off-site copy now also exists OFF Hetzner — DooPlex pulls ep0's datastore nightly (decision 70). The restic tier is still Hetzner-only. | diff --git a/documentation/architecture/09-update-architecture.md b/documentation/architecture/09-update-architecture.md index 5f568cde..fd356bea 100644 --- a/documentation/architecture/09-update-architecture.md +++ b/documentation/architecture/09-update-architecture.md @@ -741,6 +741,12 @@ its length, and both fixes cost something the household would notice — operato days from the box's request (R-823)**; the household (a recovery on the box) or the operator can cancel in that time; the hub deletes only `.orphaned-<…>`, never the live repository; every request, cancel and deletion is an operator event. *Operator brief 2026-10-04 (Part E).* Built hub v0.128.0 + controller v0.290.0. + **Note 2026-10-05 (R-867, controller v0.294.0):** the clean-up windows that feed this deletion path now remove + honest old copies: the guard's "young" line is keep-daily CALENDAR days (decision 104), not a fixed 8-day age that + sat inside the keep window. Proven live by hand the same morning: demo-felhom window 5 (16 → 14, the 2 snapshots + predicted), demo-hp window 6 (145 → 127, the 18 predicted); hub count check quiet, key files clean, no mail. + The reviewer's own error is recorded: the 2026-10-04 brief set the 8-day line and said real deletion would start + around 11 October; it would not have. `audits/night-fixes-2026-10-05/partA/`. ### 2026-10-04 (day) — three operator rulings (recorded before the work) @@ -809,6 +815,49 @@ its length, and both fixes cost something the household would notice — operato 1.30.0, agent 0.142.0 and the re-made golden; it lacks only agent 0.142.1's wrapper fix (R-858). *Operator ruling 2026-10-04 ~18:49.* +### 2026-10-05 (06:49) — four operator rulings (recorded before the work; the night-fixes brief) + +100. **Tester 1's Cloudflare tokens, shown in the 2026-10-04 night session's output, are NOT rotated** (option B) — + filed like R-831: R-870 holds what was shown, where, and the rotation steps. *Operator ruling 2026-10-05.* +101. **The power cut in the middle of an OS update (night A1) runs by day, on demo-hp only**, while the operator is + awake to switch the box on. The crash command is allowed for demo-hp only; CC asks the operator before the crash. + *Operator ruling 2026-10-05.* +102. **Tester 2's box is a laptop that is switched off at night.** It is not broken; its absence every night is the + input to the "box off at night" spike (R-871). Recorded on its page in `runbooks/target-selection.md`. + *Operator ruling 2026-10-05.* +103. **If Tester 2 is online during the session, CC re-signs its agent update to the vouched agent** (act 1 of + decision 97) and reads it back. Nothing else on Tester 2; its one-time bundle step stays with the operator + (R-862). *Operator ruling 2026-10-05.* + +### 2026-10-05 (day) — decided by CC — operator may reverse (night-fixes brief) + +104. **What "young" means in the off-site guard (R-867).** Options: (a) an age in hours derived from keep-daily + (7 × 24 h minus a margin) — the margin is a guess and DST/clock shapes move it; (b) CALENDAR days, as restic + itself buckets a snapshot's day, compared with keep-daily — exactly the policy's own boundary. **Chosen (b)**: + it is the only line that provably never refuses the honest policy (the newest snapshot of any of the last N + calendar days is always kept by keep-daily N). Cost, recorded: the line no longer catches past-dated gap-fills + that steer a 7–8-day-old keep (the 8-day line did, while refusing every honest window) — that shape is R-822's + residual, bounded by the cap and the hub's count check. Controller v0.294.0. +105. **Which commands hold off the image clean-up (R-863).** Options: (a) only the deploy path (the measured case); + (b) every compose command that can pull — `up`, `pull`, `create`, `run` — through one lock in `dockerexec`. + **Chosen (b)**: a restore, an undo and the FileBrowser sync pull the same way, and one lock is one place to + keep right. The one-time clean-up is retried every 2 minutes (at most 30 times) instead of waiting for the + next start, and writes its marker only after a pass that ran. Controller v0.294.0. +106. **Where a killed pass's report waits (R-868).** Options: (a) the agent writes a "pass started" record and + reconstructs the outcome from the journal — brittle; (b) the ROOT wrapper writes its own report beside the plan + before printing it, and the agent deletes it once sent. **Chosen (b)**, with the root write hardened (the + directory opened O_NOFOLLOW and checked to be the agent's own, the file created O_EXCL|O_NOFOLLOW) and a + cross-process pass lock so a sender never takes a running pass's copy. Agent v0.144.0. +107. **The debug OS pass with the hub away (R-866).** Options: (a) the selftest falls back to the block the daemon + saved, and says so; (b) a documented way to trigger the daemon's own leg. **Chosen (a).** The DAEMON does not + load the saved block at its own start (one line why: that would change what a box does after a restart with + the hub away — today it reports only, the safe direction — and no brief asked for it). Agent v0.144.0. +108. **A second agent release in one session (v0.144.1), against "one release per repo".** Options: (a) keep + v0.144.0 and leave R-868 open — its fix was MEASURED not to work live (the wrapper died on a broken stderr pipe + before saving the report); (b) release v0.144.1 with the two live findings fixed and red-proofed. **Chosen (b)**: + shipping a fix proven not to work is worse than a second release; precedent decision 86. Cost: a second signed + agent update and bundle per box. + ### 2026-10-04 (evening) — decided by CC unattended — operator may reverse (System page / Docker / crash-restart brief) 90. **How does a box tell a crash boot from a clean one?** Options: (a) `pstore` — measured on demo-hp: `efi_pstore` is on, diff --git a/documentation/architecture/11-os-updates.md b/documentation/architecture/11-os-updates.md index 755b4d3c..32a9e95a 100644 --- a/documentation/architecture/11-os-updates.md +++ b/documentation/architecture/11-os-updates.md @@ -280,7 +280,7 @@ longer there, from `snapshot.debian.org/archive//` with a tem | R5 | the simulation downgrades any package (the operator undo is a separate signed op, §5.6) | | R6 | the simulation installs a package that is neither installed nor in `allow_new` | | R7 | a listed version is not downloadable from the sources the installer set up or the named snapshot | -| R8 | free space on `/` (or the guest's rootfs) is below 3× the download size, minimum 500 MB (edge case 8) | +| R8 | free space on `/` (or the guest's rootfs) is below 3× the download size, minimum 500 MB (edge case 8). The download size is the `--print-uris` total WITHOUT `-s` (R-865, agent v0.144.0 — with `-s` it read 0 B) | | R9 | another apt/dpkg holds the lock, or the per-guest lane lock is held (a backup, a restore-test, C10) | | R10 | `layer` is `guest` and the vmid is not the box's own customer guest | | R11 | the plan names a package twice, or a version that is not a Debian version string | @@ -617,6 +617,28 @@ Evidence: `audits/os-host-lane-2026-10-04/` (parts A–G). two weekly golden cycles, both need a person anyway. - **The tunnel** (R-841): `running` / `not_running` / `unknown`; `tunnel_down` after two `not_running` reports. +### 8.4 The night's fixes as BUILT (2026-10-05) `[FACT]` + +Agent v0.144.0 + v0.144.1 (wrapper and agent; `09` decisions 106–108). Evidence `audits/night-fixes-2026-10-05/`. + +- **R8 measures the real download** (R-865): `--print-uris` without `-s`; the installed wrapper read 12 802 456 B for 13 + pending guest upgrades on demo-hp (it read 0 B before). `--print-uris` alone downloads nothing (measured on 9202). +- **A killed pass still reports** (R-868): the wrapper writes `report---apply.json` beside the plan before it + prints the report, and it survives a dead reader (a killed agent closes the stderr pipe — v0.144.0 died there, + measured); the agent sends a kept copy at start and every 5 minutes, then deletes it. Live: the A5 shape on demo-hp → + ONE `applied` report (13 packages) reached the hub 5 minutes later. A pass lock (flock) keeps the sender off a running + pass, also across the daemon and a selftest. +- **The debug pass with the hub away** (R-866): the daemon saves the hub's block (`os-update-block.json`); the selftest + uses it when the hub is unreachable and says `block=SAVED(…)` in its header. The daemon itself does not load the saved + block at start (decision 107). +- **A power cut in the middle of an update (night A1, run by day on demo-hp, decision 101):** crash at 06:13:55 UTC while + dpkg ran; the crash guard brought the host back in 37 s (1 unclean boot, still armed); every app healthy again within + 4 minutes; dpkg `--audit` clean, the 13 packages split 1 new / 12 old, all `ii`. **The next pass FAILED** (`E: dpkg was + interrupted, you must manually run 'dpkg --configure -a'`): the repair step runs only on a non-clean `--audit`, and a + crash can leave only dpkg's update journal behind — **R-876 (P2, open)**. By hand: `dpkg --configure -a` in the guest + (runbook `crash-guard.md`), then the next pass installed the 12. Operator mail `os_update_failed` (true); no + household mail; the household's timeline showed "Controller elindult". + ## 9. Where the rest lives - The finding: **R-812** (`backlog/OPEN-ITEMS.md`). The intention: **R-808** (`backlog/ROADMAP.md`). diff --git a/documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-snapshots-after.json b/documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-snapshots-after.json new file mode 100644 index 00000000..d1ac54eb --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-snapshots-after.json @@ -0,0 +1,142 @@ +[ + { + "id": "6ea854132efc6053ee5f7ca900a2db6f767dd3d89250efeb52fa30e319598ed1", + "short_id": "6ea85413", + "time": "2026-08-12T15:15:33.030477238Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "drill-retained-key-20260812" + ] + }, + { + "id": "b4d7cb79502a6c3f772254bc79463ec726f722f185a1c10571c389516e31746d", + "short_id": "b4d7cb79", + "time": "2026-08-31T02:15:04.935400697Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "6ce44d9de433625eff342dc0234d526a11336fb9b813aaa49cd737682c07387a", + "short_id": "6ce44d9d", + "time": "2026-09-20T02:15:05.018256188Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "c7000d7de36008707347c3b6dec9c79166feeae2a517137daa2c47004221e7ef", + "short_id": "c7000d7d", + "time": "2026-09-27T02:15:05.066652132Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "8ccc027e486d4f2e1b5415e2e602e29e8acbf0001a1596ec08b00c43b148ea68", + "short_id": "8ccc027e", + "time": "2026-09-29T02:15:05.16680869Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "40826801b49bdb9a249d0d0e7576081d5c96ea2bc7d4503d9a1bce6bfb19cb60", + "short_id": "40826801", + "time": "2026-09-30T02:15:05.173694815Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "c3a3b13b9832ba3d98cc3dc047b01b376ced2c824506e8f06001d8a3a7c5660e", + "short_id": "c3a3b13b", + "time": "2026-10-01T02:15:05.016873278Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "b2606da6d28cb24ce8d5c0616ced769b1a7f7b3d0b2923fca23b4364262b7a31", + "short_id": "b2606da6", + "time": "2026-10-02T02:15:05.138940278Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "64129ff5dcb75ea1ff766605e1a0b4dc36e6a4441a0be9433fe2519e5ddd16e0", + "short_id": "64129ff5", + "time": "2026-10-03T02:15:05.108133602Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "2bb8fb60447e6387d374d98d196807dd9e32187695ddab5df85ff2a1ac856404", + "short_id": "2bb8fb60", + "time": "2026-10-03T15:05:39.219021268Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "5dd1f0b99e8ed22d7fdcc9005501e880ddd600341fe35bd6897174acdd943857", + "short_id": "5dd1f0b9", + "time": "2026-10-03T15:17:27.992813997Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "89df02373ec2ffbe4a6b02d622c30c6f5001b922fc692bb1e1b84dfb0b8d2971", + "short_id": "89df0237", + "time": "2026-10-04T02:15:03.833138622Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "4e7867af43f024c8f712417bce982c5c8085524b51e7f37040c665d0dc1146de", + "short_id": "4e7867af", + "time": "2026-10-05T02:15:03.941403075Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "47bad696fe89450b2f5db11bfbc3a958edf2796a60d851dd9d92585eb36d53fc", + "short_id": "47bad696", + "time": "2026-10-05T05:21:04.032874428Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + } +] \ No newline at end of file diff --git a/documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-snapshots-before.json b/documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-snapshots-before.json new file mode 100644 index 00000000..b6d81bee --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-snapshots-before.json @@ -0,0 +1,152 @@ +[ + { + "id": "6ea854132efc6053ee5f7ca900a2db6f767dd3d89250efeb52fa30e319598ed1", + "short_id": "6ea85413", + "time": "2026-08-12T15:15:33.030477238Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "drill-retained-key-20260812" + ] + }, + { + "id": "b4d7cb79502a6c3f772254bc79463ec726f722f185a1c10571c389516e31746d", + "short_id": "b4d7cb79", + "time": "2026-08-31T02:15:04.935400697Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "92e49f62c017e103cf7a1f73f8982a7faa39d77eb9c7fd129b3910cc5734a262", + "short_id": "92e49f62", + "time": "2026-09-13T02:15:05.107149713Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "6ce44d9de433625eff342dc0234d526a11336fb9b813aaa49cd737682c07387a", + "short_id": "6ce44d9d", + "time": "2026-09-20T02:15:05.018256188Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "c7000d7de36008707347c3b6dec9c79166feeae2a517137daa2c47004221e7ef", + "short_id": "c7000d7d", + "time": "2026-09-27T02:15:05.066652132Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "343d57a5614f88b80a18b129b52753740c8391581073f406fe9878bf40716381", + "short_id": "343d57a5", + "time": "2026-09-28T02:15:05.279260454Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "8ccc027e486d4f2e1b5415e2e602e29e8acbf0001a1596ec08b00c43b148ea68", + "short_id": "8ccc027e", + "time": "2026-09-29T02:15:05.16680869Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "40826801b49bdb9a249d0d0e7576081d5c96ea2bc7d4503d9a1bce6bfb19cb60", + "short_id": "40826801", + "time": "2026-09-30T02:15:05.173694815Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "c3a3b13b9832ba3d98cc3dc047b01b376ced2c824506e8f06001d8a3a7c5660e", + "short_id": "c3a3b13b", + "time": "2026-10-01T02:15:05.016873278Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "b2606da6d28cb24ce8d5c0616ced769b1a7f7b3d0b2923fca23b4364262b7a31", + "short_id": "b2606da6", + "time": "2026-10-02T02:15:05.138940278Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "64129ff5dcb75ea1ff766605e1a0b4dc36e6a4441a0be9433fe2519e5ddd16e0", + "short_id": "64129ff5", + "time": "2026-10-03T02:15:05.108133602Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "2bb8fb60447e6387d374d98d196807dd9e32187695ddab5df85ff2a1ac856404", + "short_id": "2bb8fb60", + "time": "2026-10-03T15:05:39.219021268Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "5dd1f0b99e8ed22d7fdcc9005501e880ddd600341fe35bd6897174acdd943857", + "short_id": "5dd1f0b9", + "time": "2026-10-03T15:17:27.992813997Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "89df02373ec2ffbe4a6b02d622c30c6f5001b922fc692bb1e1b84dfb0b8d2971", + "short_id": "89df0237", + "time": "2026-10-04T02:15:03.833138622Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "4e7867af43f024c8f712417bce982c5c8085524b51e7f37040c665d0dc1146de", + "short_id": "4e7867af", + "time": "2026-10-05T02:15:03.941403075Z", + "hostname": "demo-felhom", + "tags": [ + "felhom-offbox", + "opengist" + ] + } +] \ No newline at end of file diff --git a/documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-window.txt b/documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-window.txt new file mode 100644 index 00000000..e61b9d4f --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-window.txt @@ -0,0 +1,26 @@ +2026-10-05T05:20:57Z +grant HTTP 200 + +HTTP 302 +before 15 after 14 +removed: [('343d57a5', '2026-09-28T02:15:05'), ('92e49f62', '2026-09-13T02:15:05')] +added by the run: [('47bad696', '2026-10-05T05:21:04')] +2026/10/05 05:16:20 [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-06 04:15 CEST +2026/10/05 05:16:35 [INFO] [bootrecon] boot window: fleet settled after 10s (3 identical samples 5s apart) — sweeping +2026/10/05 05:21:00 [INFO] [offbox] backup run started (1 app(s) toggled) +2026/10/05 05:21:01 [INFO] [offbox] pre-push dump leg completed in 892ms — snapshot pair is coherent +2026/10/05 05:21:06 [INFO] [offbox] backed up opengist (/mnt/sys_drive/felhom-data/backups/primary/opengist, 0 mandatory path(s)) +2026/10/05 05:21:14 [INFO] [offbox] clean-up window 5 (after-run): 2 of 5 planned snapshot(s) removed, 16 -> 14, in 6s +2026/10/05 05:21:18 [INFO] [offbox] backup OK: 1 app(s) backed up, 14 snapshot(s), 15s +2026/10/05 05:21:18 [INFO] [offbox] manual run progress reporting ended after 18s +Traceback (most recent call last): + File "", line 3, in +sqlite3.OperationalError: no such column: removed +Traceback (most recent call last): + File "", line 3, in +sqlite3.OperationalError: no such column: box_reason +--- hub offsite_windows (demo-felhom, newest 3) +(5, '2026-10-05 05:21:08', '2026-10-05 05:21:15', 16, 14, 8, 'pruned', 'box') +(3, '2026-10-05 02:15:08', '2026-10-05 02:15:11', 15, 15, 7, 'guard-refused', 'box') +--- hub events since 05:20 UTC +--- hub notification_log since 05:20 UTC diff --git a/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-prediction.txt b/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-prediction.txt new file mode 100644 index 00000000..9d47a46c --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-prediction.txt @@ -0,0 +1,46 @@ +PLAN 80ebd0d6 2026-09-13T02:16:27Z [felhom-offbox bentopdf] +PLAN 29516b0c 2026-09-13T02:16:31Z [felhom-offbox bookstack] +PLAN 21efaca4 2026-09-13T02:16:35Z [felhom-offbox calibre-web] +PLAN c8562d68 2026-09-13T02:16:39Z [felhom-offbox docmost] +PLAN b4350226 2026-09-13T02:16:44Z [felhom-offbox kimai] +PLAN 1ebc8069 2026-09-13T02:16:54Z [felhom-offbox paperless-ngx] +PLAN 6fac4ba8 2026-09-13T02:17:01Z [felhom-offbox romm] +PLAN 8cda9681 2026-09-13T02:17:08Z [felhom-offbox opengist] +PLAN 605bf9c7 2026-09-13T02:17:11Z [felhom-offbox privatebin] +PLAN 9efec80f 2026-09-28T13:42:12Z [felhom-offbox docmost] +PLAN ef800224 2026-09-28T13:42:20Z [felhom-offbox opengist] +PLAN c9c2414d 2026-09-28T13:42:24Z [felhom-offbox bentopdf] +PLAN 14981ce0 2026-09-28T13:42:27Z [felhom-offbox bookstack] +PLAN 693c93e1 2026-09-28T13:42:31Z [felhom-offbox calibre-web] +PLAN 90da3064 2026-09-28T13:42:36Z [felhom-offbox kimai] +PLAN 7db18829 2026-09-28T13:43:46Z [felhom-offbox paperless-ngx] +PLAN 57191440 2026-09-28T13:43:52Z [felhom-offbox privatebin] +PLAN 0febfb1d 2026-09-28T13:43:55Z [felhom-offbox romm] +PLAN b522e2f4 2026-10-03T02:16:49Z [felhom-offbox bentopdf] +PLAN c974cffe 2026-10-03T02:16:56Z [felhom-offbox calibre-web] +PLAN 0bffcb8f 2026-10-03T02:17:00Z [felhom-offbox paperless-ngx] +PLAN b5d27227 2026-10-03T02:17:08Z [felhom-offbox bookstack] +PLAN 5d1830e6 2026-10-03T02:17:13Z [felhom-offbox docmost] +PLAN 4a3c71f5 2026-10-03T02:17:19Z [felhom-offbox kimai] +PLAN 0a155533 2026-10-03T02:17:26Z [felhom-offbox opengist] +PLAN a5822405 2026-10-03T02:17:29Z [felhom-offbox privatebin] +PLAN 2cd36d95 2026-10-03T02:17:32Z [felhom-offbox romm] +PLAN bea276a7 2026-10-03T15:24:46Z [felhom-offbox bentopdf] +PLAN 59717090 2026-10-03T15:24:49Z [felhom-offbox calibre-web] +PLAN c6b5c67b 2026-10-03T15:24:51Z [felhom-offbox docmost] +PLAN cb83c862 2026-10-03T15:24:54Z [felhom-offbox paperless-ngx] +PLAN 38fbd3bc 2026-10-03T15:24:57Z [felhom-offbox privatebin] +PLAN 4ce84465 2026-10-03T15:24:59Z [felhom-offbox romm] +PLAN 2f72df90 2026-10-03T15:25:02Z [felhom-offbox bookstack] +PLAN 2c0385d3 2026-10-03T15:25:04Z [felhom-offbox kimai] +PLAN c43f2d0e 2026-10-03T15:25:07Z [felhom-offbox opengist] +PLAN 2d412cd5 2026-10-04T02:16:44Z [felhom-offbox bentopdf] +PLAN 93a66c3e 2026-10-04T02:16:46Z [felhom-offbox calibre-web] +PLAN 6dbbceac 2026-10-04T02:16:48Z [felhom-offbox docmost] +PLAN 84559526 2026-10-04T02:16:51Z [felhom-offbox paperless-ngx] +PLAN 2abceaa3 2026-10-04T02:16:54Z [felhom-offbox privatebin] +PLAN a86c8d7f 2026-10-04T02:16:56Z [felhom-offbox romm] +PLAN 40328365 2026-10-04T02:16:59Z [felhom-offbox bookstack] +PLAN f6c41de0 2026-10-04T02:17:01Z [felhom-offbox kimai] +PLAN d2b6725f 2026-10-04T02:17:04Z [felhom-offbox opengist] +GUARD at 2026-10-05T05:23:43Z: removes 18 [80ebd0d6 29516b0c 21efaca4 c8562d68 b4350226 1ebc8069 6fac4ba8 8cda9681 605bf9c7 9efec80f ef800224 c9c2414d 14981ce0 693c93e1 90da3064 7db18829 57191440 0febfb1d] refusal="" diff --git a/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-snapshots-after.json b/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-snapshots-after.json new file mode 100644 index 00000000..ab957022 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-snapshots-after.json @@ -0,0 +1,1272 @@ +[ + { + "id": "05c3346abcb6ec536fa17cfae36c0c40427612f93c192db946e5dbc01b8b0e72", + "short_id": "05c3346a", + "time": "2026-08-31T21:29:03.937755963Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "b6b7c205af0e99b9d7f17a2a66c252537a93344331381a0c63817cfb2175af57", + "short_id": "b6b7c205", + "time": "2026-08-31T21:29:08.116041068Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "0b5781f57e48d5763c69f5b3c1257edd763151b1881aab476bfdb34dd8bc77d4", + "short_id": "0b5781f5", + "time": "2026-08-31T21:29:12.001030704Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "6a2672b0246467608d33efdb211b8c9da4cc9c7d5dd555cfaa8ae72e0c2fc124", + "short_id": "6a2672b0", + "time": "2026-08-31T21:29:16.971205006Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "d3970ee37e652dfee708625c26a3ba7026a9a9820b67b053f417a632d1546371", + "short_id": "d3970ee3", + "time": "2026-08-31T21:29:21.639248203Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "5d5bee7f8cc6a73ec8d54c4fc7448a9c1eb913e080188787bea2fbe38414e14d", + "short_id": "5d5bee7f", + "time": "2026-08-31T21:29:31.670162316Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "7b0a7161ddccb6a1c17e1ba6af43d2957ae2a66333ff6e894c7cbd785ef8747c", + "short_id": "7b0a7161", + "time": "2026-08-31T21:29:34.900179928Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "1b8c7361bad9ba9e4264d9c4557535c79a35171dd4866123b2c77ae665a895f5", + "short_id": "1b8c7361", + "time": "2026-08-31T21:29:38.192674247Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "5dcee6e362b941af6e9e2738d2988f6b147732bfc049cfcfd9364b3c724f2936", + "short_id": "5dcee6e3", + "time": "2026-08-31T21:29:44.237386847Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "5d620a6e18ad087cceb1973a7ad7f9ef06fa00c7f44020dac02eb27674170074", + "short_id": "5d620a6e", + "time": "2026-09-20T02:16:42.508324795Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "b4e48dd193212d8b377cffd13fc159f1b0c9a4dab613f4d1d29fd163bd40fd3f", + "short_id": "b4e48dd1", + "time": "2026-09-20T02:16:53.12793248Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "8e840aa041119c0f145ed32e806b92e811a28fa7d2597d17380fbb650293c86e", + "short_id": "8e840aa0", + "time": "2026-09-20T02:17:00.48763028Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "05d5f5a6d696ecc5d4b243be2a138fc7a1f75b15a6a2eb7681465f1eec58d30c", + "short_id": "05d5f5a6", + "time": "2026-09-20T02:17:04.912612415Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "ad2d624aaf5000f3baccd5ef68c8e1ec25a3a6b41245af35e3e64ce4dda53e23", + "short_id": "ad2d624a", + "time": "2026-09-20T02:17:08.258668822Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "75b2e8f6eba74fb015ffd428f9f5a4a2feff6a5d5ddd3d8ca75d05d2073acb21", + "short_id": "75b2e8f6", + "time": "2026-09-20T02:17:11.527034232Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "a7b1c398cc679cf55df2c0d6e988fb894b1fd571c5ff5d217845d7ee1551cfd3", + "short_id": "a7b1c398", + "time": "2026-09-20T02:17:17.863387373Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "ac99cca985b6dc1092f42c945a6e1017edbabdfe13c5e040c6d9f4659f14cffd", + "short_id": "ac99cca9", + "time": "2026-09-20T02:17:21.118455764Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "6fb599cd275fc2276e9fe7a9ba321f0167a17618490334b41cbc3e79a533a1b1", + "short_id": "6fb599cd", + "time": "2026-09-20T02:17:24.988253922Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "27df03c3cda68f1ef116eb7e261b1129f34b2f399ef4e11442f98fbf087d9ddc", + "short_id": "27df03c3", + "time": "2026-09-27T02:16:47.978055374Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "e30300b7c0fd519cd8dead447a482348766a2051f2edb6aaea5b8c937ff5540a", + "short_id": "e30300b7", + "time": "2026-09-27T02:16:51.215507095Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "ee3a51b74dfa93c82f7eb93e279302054f37bd79ee0d517f4e9f48f2571564d6", + "short_id": "ee3a51b7", + "time": "2026-09-27T02:16:55.722014903Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "b5ed1c5635db0976b856e02dedc8395d68acee66a96f2b4e95262ff4d0d87e02", + "short_id": "b5ed1c56", + "time": "2026-09-27T02:16:59.556808493Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "ccec362a4177a1828f3487406ccc1a5ef6d51821e03e069346054efbb4003168", + "short_id": "ccec362a", + "time": "2026-09-27T02:17:12.451947753Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "1adf832d211c0ff87f2ff8c2fcb09249b87e920bb33bd16806c58333a9d1fd58", + "short_id": "1adf832d", + "time": "2026-09-27T02:17:22.454961328Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "37c2325cc8c2bef1176f13a6df61a345233a7ba9c78d2ea0b5363eaba535bc1e", + "short_id": "37c2325c", + "time": "2026-09-27T02:17:32.245321992Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "1a9c2d4c41bbd79115c4c3fec1c932a997cd0d3617a2d9b77cfb8342ba779874", + "short_id": "1a9c2d4c", + "time": "2026-09-27T02:17:35.42526792Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "ae0c1ee865ad2df2c06040224d7a62631cecc9fd7530463a7935e823e6191a1c", + "short_id": "ae0c1ee8", + "time": "2026-09-27T02:17:38.626408583Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "6cb379a8f8df1b8cd3fa2a56c33b1198e44e36f4be9d1fc093b2e392259347d6", + "short_id": "6cb379a8", + "time": "2026-09-28T13:42:46.558849032Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "nextcloud" + ] + }, + { + "id": "000a22015355baf71b2bc62e5304402559d6252cb2b1355d5cae34722c5a10d7", + "short_id": "000a2201", + "time": "2026-09-29T02:16:48.352396174Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "ddcbd8804492f51f99766ff4a94a9f2391a33b121d0fe04050378bfced55e8aa", + "short_id": "ddcbd880", + "time": "2026-09-29T02:16:55.110022545Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "8ccff452aaf5099d5357316b48f9fa2ffecf26dc7ef52722decb28643eb235cb", + "short_id": "8ccff452", + "time": "2026-09-29T02:16:59.097099692Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "c69ae789a5fa692307573873202cc87e4b70742878d45b71727ca0243c04febc", + "short_id": "c69ae789", + "time": "2026-09-29T02:17:09.274627189Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "599313e7ef693ce3dc11c20726b3e0f5800baa1fd1473b96660d92cceaabd368", + "short_id": "599313e7", + "time": "2026-09-29T02:17:19.953152125Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "75332c0a549033e88f5fda3639b779d2d267f4ba9478bf607900bb43b69bd782", + "short_id": "75332c0a", + "time": "2026-09-29T02:17:23.272859496Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "4d0098c1c74ad7e6372b48bee7167890cb64be192963bd399bda685f80df26d0", + "short_id": "4d0098c1", + "time": "2026-09-29T02:17:29.111837748Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "98db0ffc6793963d81976b111b53ac2e25fb2eab7d0fa8de6abbea958420e342", + "short_id": "98db0ffc", + "time": "2026-09-29T02:17:33.949692396Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "ec14270dc73dbe8e96fd46960248157912470555dab272f815c3cd040ac91136", + "short_id": "ec14270d", + "time": "2026-09-29T02:17:37.297731503Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "22e6c2e3b5087a4ddf5f2ac378120d33743abfd87835137252080af5f1314c3f", + "short_id": "22e6c2e3", + "time": "2026-09-30T11:32:34.478580985Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "e6613ab0ef9250e05b0b69d719b87a2c29d09af0c9d7f7cc49060e49d56ad4af", + "short_id": "e6613ab0", + "time": "2026-09-30T11:32:42.543834868Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "82592644bc9e520c3d0eecd7f4f88f164d378698900ea498f5d049bf3bba4f41", + "short_id": "82592644", + "time": "2026-09-30T11:32:52.955145104Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "31654cb429d9d460696e09ff9b11ab9353cf533e1eef793424e1f07802c4d60f", + "short_id": "31654cb4", + "time": "2026-09-30T11:33:00.394285874Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "7235832b3e34378b3ee02a32bd6ce151e70a50e8117d35a599860e22e0b92631", + "short_id": "7235832b", + "time": "2026-09-30T11:33:07.212828489Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "dea24011882a899cab3bb88b4767f820a0f6b704d464c27e0596f6a3b5b08904", + "short_id": "dea24011", + "time": "2026-09-30T11:33:11.86793512Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "e4d3896e1ee330d35cd714dd226fe8cb949bb283ee5a8965756092c85898ca7a", + "short_id": "e4d3896e", + "time": "2026-09-30T11:33:16.218517978Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "6bb83087ce0a30c5eeec5b91f41e8479a59af4e5272546562f03f166c51042ef", + "short_id": "6bb83087", + "time": "2026-09-30T11:33:25.72336732Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "140f7899bf2ed152795db08da526e1d50801081073b5fa26d7574b2e095d7a25", + "short_id": "140f7899", + "time": "2026-09-30T11:33:29.135930796Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "2e1c90508e299b33d6da9a75903cdd2a199b00a05b1df119ec40669aaf32c175", + "short_id": "2e1c9050", + "time": "2026-10-01T02:16:44.52764868Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "c54609b262525924d8b789c7de5897eaa4e18db574d70df8a23907a55de9d1bf", + "short_id": "c54609b2", + "time": "2026-10-01T02:16:51.929744344Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "15d79420478b0e392772e066203466c4a6c7799fc8b894e0342b308124df8671", + "short_id": "15d79420", + "time": "2026-10-01T02:16:56.910725774Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "e8d5b4527e468cfc7b9d5f58c9a67dd410eb2cd944ec9019dc9b9b510348165b", + "short_id": "e8d5b452", + "time": "2026-10-01T02:17:03.95424596Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "71c413f0ecd6c399704bb8163ad2b72ea09e50bf51a63b44583f4a4342b8399c", + "short_id": "71c413f0", + "time": "2026-10-01T02:17:07.469388005Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "86817a6c1f1f5bc812df28db2318bc60ecd603899f692d0060d11879695a077e", + "short_id": "86817a6c", + "time": "2026-10-01T02:17:15.13320776Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "a0770c2cd8c9807dab7dc1b2bca8270d2de131a04440f8d841f23e345f9743e7", + "short_id": "a0770c2c", + "time": "2026-10-01T02:17:18.614565966Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "7477ef8554ba7f92d9366ea635e265e2364b387251c92afefd44d5a7b7b863a2", + "short_id": "7477ef85", + "time": "2026-10-01T02:17:24.821106575Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "6348850c9f02a14eaf6b1002ad2edce3cbc89aa6a6cf63dc41c403e0dc09bbf2", + "short_id": "6348850c", + "time": "2026-10-01T02:17:28.082649988Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "c56e96a40128f24be2e45a1c3c78b589ee5088d36072cde5860bb988ca76d57f", + "short_id": "c56e96a4", + "time": "2026-10-02T02:16:51.117784535Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "67f96e9f2be6a7d7dea3affbd67c7202f8c39f4e66c5cc5300203b80a6af072d", + "short_id": "67f96e9f", + "time": "2026-10-02T02:16:58.844276425Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "b1c2c5ff49aadeef5124e1b4ee0aa6942d92c1cdb779bb913869387f1f6f75ee", + "short_id": "b1c2c5ff", + "time": "2026-10-02T02:17:02.750041714Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "b20723b07eb32ad54e49bc480971f3ffb689e2387ea43e163720a22b267b8c62", + "short_id": "b20723b0", + "time": "2026-10-02T02:17:06.049665942Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "696441381f966f17e0811bcecb1233132fee261c6c7889911828836c41f05752", + "short_id": "69644138", + "time": "2026-10-02T02:17:14.579068358Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "ba39115e7a5d72a31bcff4510d76493a7c6bf96b1f50815adb8ed31c0b77e48f", + "short_id": "ba39115e", + "time": "2026-10-02T02:17:18.008294157Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "b8b794f0b98e0f513e5be40bae369290ddcf5088ceffe1035eafc95de81606fa", + "short_id": "b8b794f0", + "time": "2026-10-02T02:17:24.914918672Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "705e90d776ccfa72aadef145023287d584b0bd5d964132b710b85d24de01ac59", + "short_id": "705e90d7", + "time": "2026-10-02T02:17:31.398812542Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "5915da33a84540956ba50ae01a8f7965acb14c0a28ff31bb95bcf24e6852be71", + "short_id": "5915da33", + "time": "2026-10-02T02:17:34.700367474Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "b522e2f455f79dce186326573f933810fe32ed12e5ff978f94ccca5e2f4ef9cf", + "short_id": "b522e2f4", + "time": "2026-10-03T02:16:49.879945674Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "c974cffe879c996d83bdb5b0d539b9c5b7ea8a9cd73548b639205e82b95d3349", + "short_id": "c974cffe", + "time": "2026-10-03T02:16:56.393882363Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "0bffcb8fb250f483d955c0c5d2b74d5581f3bb1a6788852480053de2b66ebb70", + "short_id": "0bffcb8f", + "time": "2026-10-03T02:17:00.315980736Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "b5d27227c4d2d89ac06bdd78066ef415135fd3389dd122e6685e3d052c5e20da", + "short_id": "b5d27227", + "time": "2026-10-03T02:17:08.916928673Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "5d1830e67837d3f2ecbcac716dfe6926b5e2eaff9d7204e4f744ffd290a53b0a", + "short_id": "5d1830e6", + "time": "2026-10-03T02:17:13.665128317Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "4a3c71f5e89dc1d99d5fc69a7ceb4e3caa30351d157b2a1f6a84f187497a27b3", + "short_id": "4a3c71f5", + "time": "2026-10-03T02:17:19.230120149Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "0a155533421525d9bb233b309fc151e9f8487e234e5b43ceeaecbc515d03d819", + "short_id": "0a155533", + "time": "2026-10-03T02:17:26.024547211Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "a5822405a635efb8d072313354d8dd7db1a52dcbaa90f1aa1b96469439b3673c", + "short_id": "a5822405", + "time": "2026-10-03T02:17:29.289383959Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "2cd36d952d1a73acff474f77d757ab36c2f2c499be320053521b28f82b198bbd", + "short_id": "2cd36d95", + "time": "2026-10-03T02:17:32.58096138Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "bea276a7afb1d6a9706c28c6e02dc717406ccffb5ad100e42707fc7c1ab0fbb1", + "short_id": "bea276a7", + "time": "2026-10-03T15:24:46.196687533Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "597170905424b94c44bfb1123602ac19e334356c39a5bc5467a759f1a7c231a4", + "short_id": "59717090", + "time": "2026-10-03T15:24:49.113532044Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "c6b5c67b90498708f5e22317db12b270cb8803f953f2c2d110b497581c32d825", + "short_id": "c6b5c67b", + "time": "2026-10-03T15:24:51.392171506Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "cb83c8620c7712086d1329ed028516c3994754814d88ffc1edc0e3c5da470870", + "short_id": "cb83c862", + "time": "2026-10-03T15:24:54.220252524Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "38fbd3bc942d53f1ad51d7688d9c1ff6151b2f42ac55d2903643db6d82c09f07", + "short_id": "38fbd3bc", + "time": "2026-10-03T15:24:57.338804736Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "4ce84465e83323c63fbafdf518854e6388866d649351cc7c14533dfc0af10ce5", + "short_id": "4ce84465", + "time": "2026-10-03T15:24:59.243526267Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "2f72df901c0586964d260c12939c62f2e37e7b53ebadff5ca112bc1269f87d8d", + "short_id": "2f72df90", + "time": "2026-10-03T15:25:02.5847505Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "2c0385d380cdf2ef7afc3dc632365dbf9b2482d07f9cfd740ea26bcf7f3eae59", + "short_id": "2c0385d3", + "time": "2026-10-03T15:25:04.980411947Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "c43f2d0e3545bc8e8cffc0f08e850ff177ceea1b99fa4e9c4be063f64eabec39", + "short_id": "c43f2d0e", + "time": "2026-10-03T15:25:07.97694046Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "5a56d51bb2261293ed20338db657a4014216ee343a4c2c5babb7baa42384453b", + "short_id": "5a56d51b", + "time": "2026-10-03T15:31:39.206098904Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "cf905e063e5a7f2ab4b8f13d71e49b3d381206049f9db380d1044eb774719ee1", + "short_id": "cf905e06", + "time": "2026-10-03T15:31:41.528439072Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "0824a70c01ffff21eda54bd0f17718d494a5698d38a22347b326f2f3eec45689", + "short_id": "0824a70c", + "time": "2026-10-03T15:31:44.135006336Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "e9eee3bd728c5d8d90d7bddff67a4fa3d8a4066eaf09f5e79462429c4e855605", + "short_id": "e9eee3bd", + "time": "2026-10-03T15:31:46.903924425Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "53a4bdf9732eaf6232f11adfe3a4e31239f636050e4ac11b952259e16c10fb16", + "short_id": "53a4bdf9", + "time": "2026-10-03T15:31:48.816807194Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "99c4ad053441005c3134c0adb0d219c46b3067aa3b4ad5800dc29af4e45d0087", + "short_id": "99c4ad05", + "time": "2026-10-03T15:31:51.673121035Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "e2db64cdb00e31b4456033c38dc0653dfb01760748cd302ea0079e9ce3096f94", + "short_id": "e2db64cd", + "time": "2026-10-03T15:31:54.069221581Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "5551bdd9779a6015c59e18831aabec43384a779f9f29e0732fad4479329c3986", + "short_id": "5551bdd9", + "time": "2026-10-03T15:31:56.90845268Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "e3d57107000233d627dddadf0ff2927ebfbe456c8b6236bc8cc31626a88f5185", + "short_id": "e3d57107", + "time": "2026-10-03T15:31:58.824425393Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "2d412cd5471cda5fe76a115426d08be3c183c0b6f0407ed56475dec2ed83e466", + "short_id": "2d412cd5", + "time": "2026-10-04T02:16:44.524065664Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "93a66c3eeb252d344a8f37be70bc8c5f72deec2d74dd9034ce00c993905c5da1", + "short_id": "93a66c3e", + "time": "2026-10-04T02:16:46.534034004Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "6dbbceaccd875c38a73ec105e54112655540d9b42af8e7054e7746328105f10b", + "short_id": "6dbbceac", + "time": "2026-10-04T02:16:48.856969168Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "845595263a5c44660a86ae6d4c37986f2345732dd3163e95309f0daea8b0bc64", + "short_id": "84559526", + "time": "2026-10-04T02:16:51.234048844Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "2abceaa322abcdec2fded16754223cea05290b839d6c2027f1b3bcd485049708", + "short_id": "2abceaa3", + "time": "2026-10-04T02:16:54.045587334Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "a86c8d7fbf9dfa7cf6f2940f54f0262052d40f0832143425cb712cf39f92f61e", + "short_id": "a86c8d7f", + "time": "2026-10-04T02:16:56.03786578Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "40328365ce85747f54d0320548cad5bfc5b78ca46aa7dada73b218d5d7cb5ff8", + "short_id": "40328365", + "time": "2026-10-04T02:16:59.09044838Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "f6c41de0c21de235bf5d79f7d9093da0dde7e3dc6755b307b9f8c25822591970", + "short_id": "f6c41de0", + "time": "2026-10-04T02:17:01.314326343Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "d2b6725ff83dc3e27c778fd449232dc606cc1558a6ae755fccfec10d80d263ea", + "short_id": "d2b6725f", + "time": "2026-10-04T02:17:04.567737509Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "5860f7fdb1582d05e7f37f3cbf9ef42661b8159ceb6d284e2ef057ee7ffffcc6", + "short_id": "5860f7fd", + "time": "2026-10-04T05:35:48.893065122Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "ca56106cb17f6cc89e4f3c4000283aa84971cddf70819beda248b4c076eefcb7", + "short_id": "ca56106c", + "time": "2026-10-04T05:35:51.882332467Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "bc0ff350d6f6726c71e374522fc9abf4305c3c393e10c36d780f0671022051e9", + "short_id": "bc0ff350", + "time": "2026-10-04T05:35:53.871253222Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "87c58189da991a9ddbb0693a8ce041de3256a10bd5d1e1825c742c37437cada6", + "short_id": "87c58189", + "time": "2026-10-04T05:35:55.753162092Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "9203e4296246f392b78ec30cb615a15f474954dd62b864b417ef0229127aa627", + "short_id": "9203e429", + "time": "2026-10-04T05:35:58.238852079Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "f4564d6144e581cf299db34b17345a776dd3665ff9ddd05310751ab6cf2269cb", + "short_id": "f4564d61", + "time": "2026-10-04T05:36:02.446280346Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "021da57387805b8e75cafc464a864593338ff1ca90865737fbd3680844ed9edc", + "short_id": "021da573", + "time": "2026-10-04T05:36:04.447050059Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "7971a80d8bdc1d6445216bd3f77ee192cef9da63d637de75e396beca2f61ae80", + "short_id": "7971a80d", + "time": "2026-10-04T05:36:07.004160186Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "1aa58dd03e49a9e3a7265d38e7fcbcef0d908d618783dc890d5278160ac08e23", + "short_id": "1aa58dd0", + "time": "2026-10-04T05:36:09.28929826Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "4fc5669ec2827ff88e1695c926c55208fc28865a5edbd1a6e7c69d39c2652fb4", + "short_id": "4fc5669e", + "time": "2026-10-05T02:16:23.746641999Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "fd47799c2549953365b4f99e59b15211d307a659700ea499baf09b3ddcd9f23a", + "short_id": "fd47799c", + "time": "2026-10-05T02:16:27.087572401Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "c5fe883d04bb89484faf58a18caee309f6142dab1e4e6400ee556f500d52fc6a", + "short_id": "c5fe883d", + "time": "2026-10-05T02:16:29.011227176Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "5e933e416390028db73e2bdd816b8a638fe25dacd5789dd38ef8f9c7e6f2f8c2", + "short_id": "5e933e41", + "time": "2026-10-05T02:16:30.984303284Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "5eea7fae4559e23aa073c3ca2e909d0933e14bb4323aaabf67c2b06f40ae565a", + "short_id": "5eea7fae", + "time": "2026-10-05T02:16:33.356524582Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "e5d4d77ac9dee0ed668372488b8ea193ff298950d9b42a1bf09d981d68de3b96", + "short_id": "e5d4d77a", + "time": "2026-10-05T02:16:35.969451814Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "b83743cfbebb690d02b66edc3280185463eabb19d1b1a4e804186e56abf16ce3", + "short_id": "b83743cf", + "time": "2026-10-05T02:16:38.825026042Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "1385727c6785e35a35e703635fdda6d9e36d53b3ccaefcc14ee9ca69643a8c13", + "short_id": "1385727c", + "time": "2026-10-05T02:16:42.022557074Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "c62d034a9cc76eeae303b494d906a7f2525584f243011991552e4aaef2cf2aa4", + "short_id": "c62d034a", + "time": "2026-10-05T02:16:44.050699823Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "4cfb15954bc0a65d4149546446a23183f0380b04ca7f392ebb30bf4ecef200e6", + "short_id": "4cfb1595", + "time": "2026-10-05T05:25:27.194993471Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "3e9cc1f63b440112d404c7a52ec2f0ee74ab37d99daa3a5ee94495315a766013", + "short_id": "3e9cc1f6", + "time": "2026-10-05T05:25:29.987605851Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "6e099f800dd5ce5c34512efb47e97d458df1eb5ec652d0d83a6d67c5410d0bbf", + "short_id": "6e099f80", + "time": "2026-10-05T05:25:32.263631348Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "33635aa36ab31e27c4571fa45b2e136ece7fc6c982aa700db1fd87161fb0a5ed", + "short_id": "33635aa3", + "time": "2026-10-05T05:25:34.821234745Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "52d049111af453f41ea50484197d4c76b7c6422ea4f09ecfc1cd2f3ac2f5b92c", + "short_id": "52d04911", + "time": "2026-10-05T05:25:37.818746767Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "025171e56a341f6c4cfc6c46440913b5ca03971c77c0cee70863cec22d10cd3f", + "short_id": "025171e5", + "time": "2026-10-05T05:25:39.756967802Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "843707e1f7ca0149b9b6f8265f84f22af0584a635445d3a379c6c88910302830", + "short_id": "843707e1", + "time": "2026-10-05T05:25:41.707587531Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "e5a50f3592bab52be10f64c27aae947643725097bcd6d3536ac843a079adeed3", + "short_id": "e5a50f35", + "time": "2026-10-05T05:25:44.526283921Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "e9e54d5a23c7687f7f7c3e4e09b89483c7913219102d57e113011102999b1e95", + "short_id": "e9e54d5a", + "time": "2026-10-05T05:25:48.343966404Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + } +] \ No newline at end of file diff --git a/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-snapshots-before.json b/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-snapshots-before.json new file mode 100644 index 00000000..60ae8a9f --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-snapshots-before.json @@ -0,0 +1,1362 @@ +[ + { + "id": "05c3346abcb6ec536fa17cfae36c0c40427612f93c192db946e5dbc01b8b0e72", + "short_id": "05c3346a", + "time": "2026-08-31T21:29:03.937755963Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "b6b7c205af0e99b9d7f17a2a66c252537a93344331381a0c63817cfb2175af57", + "short_id": "b6b7c205", + "time": "2026-08-31T21:29:08.116041068Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "0b5781f57e48d5763c69f5b3c1257edd763151b1881aab476bfdb34dd8bc77d4", + "short_id": "0b5781f5", + "time": "2026-08-31T21:29:12.001030704Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "6a2672b0246467608d33efdb211b8c9da4cc9c7d5dd555cfaa8ae72e0c2fc124", + "short_id": "6a2672b0", + "time": "2026-08-31T21:29:16.971205006Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "d3970ee37e652dfee708625c26a3ba7026a9a9820b67b053f417a632d1546371", + "short_id": "d3970ee3", + "time": "2026-08-31T21:29:21.639248203Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "5d5bee7f8cc6a73ec8d54c4fc7448a9c1eb913e080188787bea2fbe38414e14d", + "short_id": "5d5bee7f", + "time": "2026-08-31T21:29:31.670162316Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "7b0a7161ddccb6a1c17e1ba6af43d2957ae2a66333ff6e894c7cbd785ef8747c", + "short_id": "7b0a7161", + "time": "2026-08-31T21:29:34.900179928Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "1b8c7361bad9ba9e4264d9c4557535c79a35171dd4866123b2c77ae665a895f5", + "short_id": "1b8c7361", + "time": "2026-08-31T21:29:38.192674247Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "5dcee6e362b941af6e9e2738d2988f6b147732bfc049cfcfd9364b3c724f2936", + "short_id": "5dcee6e3", + "time": "2026-08-31T21:29:44.237386847Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "80ebd0d6dfad74eda917dde4adbd8d5e1aaeda6c4458ac108af6755d14496472", + "short_id": "80ebd0d6", + "time": "2026-09-13T02:16:27.703511646Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "29516b0ca93c6eac80c149d07956c382b8219824a8c51098c8916952fa28f5a4", + "short_id": "29516b0c", + "time": "2026-09-13T02:16:31.027664475Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "21efaca455d781bd284e80a375bf79958e4245c735299369667a965d13d112e1", + "short_id": "21efaca4", + "time": "2026-09-13T02:16:35.781806023Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "c8562d68bc20113953de1191138c6ea6a0adf4044bd4e64da58d920cb20094e4", + "short_id": "c8562d68", + "time": "2026-09-13T02:16:39.688779897Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "b43502261037116a48b48de918cf5290269481cc75e81a4c713382ee0fe8931d", + "short_id": "b4350226", + "time": "2026-09-13T02:16:44.380568437Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "1ebc806994b255f064f6346822e97b3d5d8a91ef9fdfafd31c6723bd1f3e4699", + "short_id": "1ebc8069", + "time": "2026-09-13T02:16:54.368550501Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "6fac4ba8791aafefecf9395c6003d60de1249dc84d20b4a72031223d9839924f", + "short_id": "6fac4ba8", + "time": "2026-09-13T02:17:01.835333487Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "8cda96813b5ae9a6e842b0259f79d26f37406ec105a4128bbe72909cdd0f5be3", + "short_id": "8cda9681", + "time": "2026-09-13T02:17:08.037512598Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "605bf9c77fa4e2e091665bcfe735121a4cf4b9492e1335fe92211f43603f0917", + "short_id": "605bf9c7", + "time": "2026-09-13T02:17:11.259193417Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "5d620a6e18ad087cceb1973a7ad7f9ef06fa00c7f44020dac02eb27674170074", + "short_id": "5d620a6e", + "time": "2026-09-20T02:16:42.508324795Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "b4e48dd193212d8b377cffd13fc159f1b0c9a4dab613f4d1d29fd163bd40fd3f", + "short_id": "b4e48dd1", + "time": "2026-09-20T02:16:53.12793248Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "8e840aa041119c0f145ed32e806b92e811a28fa7d2597d17380fbb650293c86e", + "short_id": "8e840aa0", + "time": "2026-09-20T02:17:00.48763028Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "05d5f5a6d696ecc5d4b243be2a138fc7a1f75b15a6a2eb7681465f1eec58d30c", + "short_id": "05d5f5a6", + "time": "2026-09-20T02:17:04.912612415Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "ad2d624aaf5000f3baccd5ef68c8e1ec25a3a6b41245af35e3e64ce4dda53e23", + "short_id": "ad2d624a", + "time": "2026-09-20T02:17:08.258668822Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "75b2e8f6eba74fb015ffd428f9f5a4a2feff6a5d5ddd3d8ca75d05d2073acb21", + "short_id": "75b2e8f6", + "time": "2026-09-20T02:17:11.527034232Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "a7b1c398cc679cf55df2c0d6e988fb894b1fd571c5ff5d217845d7ee1551cfd3", + "short_id": "a7b1c398", + "time": "2026-09-20T02:17:17.863387373Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "ac99cca985b6dc1092f42c945a6e1017edbabdfe13c5e040c6d9f4659f14cffd", + "short_id": "ac99cca9", + "time": "2026-09-20T02:17:21.118455764Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "6fb599cd275fc2276e9fe7a9ba321f0167a17618490334b41cbc3e79a533a1b1", + "short_id": "6fb599cd", + "time": "2026-09-20T02:17:24.988253922Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "27df03c3cda68f1ef116eb7e261b1129f34b2f399ef4e11442f98fbf087d9ddc", + "short_id": "27df03c3", + "time": "2026-09-27T02:16:47.978055374Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "e30300b7c0fd519cd8dead447a482348766a2051f2edb6aaea5b8c937ff5540a", + "short_id": "e30300b7", + "time": "2026-09-27T02:16:51.215507095Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "ee3a51b74dfa93c82f7eb93e279302054f37bd79ee0d517f4e9f48f2571564d6", + "short_id": "ee3a51b7", + "time": "2026-09-27T02:16:55.722014903Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "b5ed1c5635db0976b856e02dedc8395d68acee66a96f2b4e95262ff4d0d87e02", + "short_id": "b5ed1c56", + "time": "2026-09-27T02:16:59.556808493Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "ccec362a4177a1828f3487406ccc1a5ef6d51821e03e069346054efbb4003168", + "short_id": "ccec362a", + "time": "2026-09-27T02:17:12.451947753Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "1adf832d211c0ff87f2ff8c2fcb09249b87e920bb33bd16806c58333a9d1fd58", + "short_id": "1adf832d", + "time": "2026-09-27T02:17:22.454961328Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "37c2325cc8c2bef1176f13a6df61a345233a7ba9c78d2ea0b5363eaba535bc1e", + "short_id": "37c2325c", + "time": "2026-09-27T02:17:32.245321992Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "1a9c2d4c41bbd79115c4c3fec1c932a997cd0d3617a2d9b77cfb8342ba779874", + "short_id": "1a9c2d4c", + "time": "2026-09-27T02:17:35.42526792Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "ae0c1ee865ad2df2c06040224d7a62631cecc9fd7530463a7935e823e6191a1c", + "short_id": "ae0c1ee8", + "time": "2026-09-27T02:17:38.626408583Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "9efec80f7b80d975e17476192886db8da746f5defade7a2ad154c4fe65a9e6b9", + "short_id": "9efec80f", + "time": "2026-09-28T13:42:12.155371836Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "ef800224d3a28286e21d6dcd6a421a2ba70d4b17c33cef2a9c99574d98ebc8ba", + "short_id": "ef800224", + "time": "2026-09-28T13:42:20.678195963Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "c9c2414d66376984f0c1fdc0047aa36cb9148cd6e60366ae4b352dcd82ddc9b5", + "short_id": "c9c2414d", + "time": "2026-09-28T13:42:24.00598142Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "14981ce031d850cdb3265a37e062f75fd084cdc3ac0e86a122c4b185ae91fc44", + "short_id": "14981ce0", + "time": "2026-09-28T13:42:27.413685439Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "693c93e10fbb97414e5e891944467294e1b1486f8674ab2d5abfb27b0a8f20a9", + "short_id": "693c93e1", + "time": "2026-09-28T13:42:31.788196787Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "90da306418cb7d0e6fe57e1f69299f35138277f078f70bcb1fc83fcec55d4dbd", + "short_id": "90da3064", + "time": "2026-09-28T13:42:36.034012959Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "6cb379a8f8df1b8cd3fa2a56c33b1198e44e36f4be9d1fc093b2e392259347d6", + "short_id": "6cb379a8", + "time": "2026-09-28T13:42:46.558849032Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "nextcloud" + ] + }, + { + "id": "7db1882961593166f80acb2e266e19bb526ee57f6200d033157097ef4d71ca62", + "short_id": "7db18829", + "time": "2026-09-28T13:43:46.609571249Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "571914407d1247fc045af84fb32a3c85210f709b6c1a3ec6ee73e486dad45e02", + "short_id": "57191440", + "time": "2026-09-28T13:43:52.237162198Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "0febfb1d8fcb3c5f1cb96c7c5e8e8b0aaad8284502b496e405b40325977bf50f", + "short_id": "0febfb1d", + "time": "2026-09-28T13:43:55.636481766Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "000a22015355baf71b2bc62e5304402559d6252cb2b1355d5cae34722c5a10d7", + "short_id": "000a2201", + "time": "2026-09-29T02:16:48.352396174Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "ddcbd8804492f51f99766ff4a94a9f2391a33b121d0fe04050378bfced55e8aa", + "short_id": "ddcbd880", + "time": "2026-09-29T02:16:55.110022545Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "8ccff452aaf5099d5357316b48f9fa2ffecf26dc7ef52722decb28643eb235cb", + "short_id": "8ccff452", + "time": "2026-09-29T02:16:59.097099692Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "c69ae789a5fa692307573873202cc87e4b70742878d45b71727ca0243c04febc", + "short_id": "c69ae789", + "time": "2026-09-29T02:17:09.274627189Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "599313e7ef693ce3dc11c20726b3e0f5800baa1fd1473b96660d92cceaabd368", + "short_id": "599313e7", + "time": "2026-09-29T02:17:19.953152125Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "75332c0a549033e88f5fda3639b779d2d267f4ba9478bf607900bb43b69bd782", + "short_id": "75332c0a", + "time": "2026-09-29T02:17:23.272859496Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "4d0098c1c74ad7e6372b48bee7167890cb64be192963bd399bda685f80df26d0", + "short_id": "4d0098c1", + "time": "2026-09-29T02:17:29.111837748Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "98db0ffc6793963d81976b111b53ac2e25fb2eab7d0fa8de6abbea958420e342", + "short_id": "98db0ffc", + "time": "2026-09-29T02:17:33.949692396Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "ec14270dc73dbe8e96fd46960248157912470555dab272f815c3cd040ac91136", + "short_id": "ec14270d", + "time": "2026-09-29T02:17:37.297731503Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "22e6c2e3b5087a4ddf5f2ac378120d33743abfd87835137252080af5f1314c3f", + "short_id": "22e6c2e3", + "time": "2026-09-30T11:32:34.478580985Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "e6613ab0ef9250e05b0b69d719b87a2c29d09af0c9d7f7cc49060e49d56ad4af", + "short_id": "e6613ab0", + "time": "2026-09-30T11:32:42.543834868Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "82592644bc9e520c3d0eecd7f4f88f164d378698900ea498f5d049bf3bba4f41", + "short_id": "82592644", + "time": "2026-09-30T11:32:52.955145104Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "31654cb429d9d460696e09ff9b11ab9353cf533e1eef793424e1f07802c4d60f", + "short_id": "31654cb4", + "time": "2026-09-30T11:33:00.394285874Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "7235832b3e34378b3ee02a32bd6ce151e70a50e8117d35a599860e22e0b92631", + "short_id": "7235832b", + "time": "2026-09-30T11:33:07.212828489Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "dea24011882a899cab3bb88b4767f820a0f6b704d464c27e0596f6a3b5b08904", + "short_id": "dea24011", + "time": "2026-09-30T11:33:11.86793512Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "e4d3896e1ee330d35cd714dd226fe8cb949bb283ee5a8965756092c85898ca7a", + "short_id": "e4d3896e", + "time": "2026-09-30T11:33:16.218517978Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "6bb83087ce0a30c5eeec5b91f41e8479a59af4e5272546562f03f166c51042ef", + "short_id": "6bb83087", + "time": "2026-09-30T11:33:25.72336732Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "140f7899bf2ed152795db08da526e1d50801081073b5fa26d7574b2e095d7a25", + "short_id": "140f7899", + "time": "2026-09-30T11:33:29.135930796Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "2e1c90508e299b33d6da9a75903cdd2a199b00a05b1df119ec40669aaf32c175", + "short_id": "2e1c9050", + "time": "2026-10-01T02:16:44.52764868Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "c54609b262525924d8b789c7de5897eaa4e18db574d70df8a23907a55de9d1bf", + "short_id": "c54609b2", + "time": "2026-10-01T02:16:51.929744344Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "15d79420478b0e392772e066203466c4a6c7799fc8b894e0342b308124df8671", + "short_id": "15d79420", + "time": "2026-10-01T02:16:56.910725774Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "e8d5b4527e468cfc7b9d5f58c9a67dd410eb2cd944ec9019dc9b9b510348165b", + "short_id": "e8d5b452", + "time": "2026-10-01T02:17:03.95424596Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "71c413f0ecd6c399704bb8163ad2b72ea09e50bf51a63b44583f4a4342b8399c", + "short_id": "71c413f0", + "time": "2026-10-01T02:17:07.469388005Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "86817a6c1f1f5bc812df28db2318bc60ecd603899f692d0060d11879695a077e", + "short_id": "86817a6c", + "time": "2026-10-01T02:17:15.13320776Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "a0770c2cd8c9807dab7dc1b2bca8270d2de131a04440f8d841f23e345f9743e7", + "short_id": "a0770c2c", + "time": "2026-10-01T02:17:18.614565966Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "7477ef8554ba7f92d9366ea635e265e2364b387251c92afefd44d5a7b7b863a2", + "short_id": "7477ef85", + "time": "2026-10-01T02:17:24.821106575Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "6348850c9f02a14eaf6b1002ad2edce3cbc89aa6a6cf63dc41c403e0dc09bbf2", + "short_id": "6348850c", + "time": "2026-10-01T02:17:28.082649988Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "c56e96a40128f24be2e45a1c3c78b589ee5088d36072cde5860bb988ca76d57f", + "short_id": "c56e96a4", + "time": "2026-10-02T02:16:51.117784535Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "67f96e9f2be6a7d7dea3affbd67c7202f8c39f4e66c5cc5300203b80a6af072d", + "short_id": "67f96e9f", + "time": "2026-10-02T02:16:58.844276425Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "b1c2c5ff49aadeef5124e1b4ee0aa6942d92c1cdb779bb913869387f1f6f75ee", + "short_id": "b1c2c5ff", + "time": "2026-10-02T02:17:02.750041714Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "b20723b07eb32ad54e49bc480971f3ffb689e2387ea43e163720a22b267b8c62", + "short_id": "b20723b0", + "time": "2026-10-02T02:17:06.049665942Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "696441381f966f17e0811bcecb1233132fee261c6c7889911828836c41f05752", + "short_id": "69644138", + "time": "2026-10-02T02:17:14.579068358Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "ba39115e7a5d72a31bcff4510d76493a7c6bf96b1f50815adb8ed31c0b77e48f", + "short_id": "ba39115e", + "time": "2026-10-02T02:17:18.008294157Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "b8b794f0b98e0f513e5be40bae369290ddcf5088ceffe1035eafc95de81606fa", + "short_id": "b8b794f0", + "time": "2026-10-02T02:17:24.914918672Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "705e90d776ccfa72aadef145023287d584b0bd5d964132b710b85d24de01ac59", + "short_id": "705e90d7", + "time": "2026-10-02T02:17:31.398812542Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "5915da33a84540956ba50ae01a8f7965acb14c0a28ff31bb95bcf24e6852be71", + "short_id": "5915da33", + "time": "2026-10-02T02:17:34.700367474Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "b522e2f455f79dce186326573f933810fe32ed12e5ff978f94ccca5e2f4ef9cf", + "short_id": "b522e2f4", + "time": "2026-10-03T02:16:49.879945674Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "c974cffe879c996d83bdb5b0d539b9c5b7ea8a9cd73548b639205e82b95d3349", + "short_id": "c974cffe", + "time": "2026-10-03T02:16:56.393882363Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "0bffcb8fb250f483d955c0c5d2b74d5581f3bb1a6788852480053de2b66ebb70", + "short_id": "0bffcb8f", + "time": "2026-10-03T02:17:00.315980736Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "b5d27227c4d2d89ac06bdd78066ef415135fd3389dd122e6685e3d052c5e20da", + "short_id": "b5d27227", + "time": "2026-10-03T02:17:08.916928673Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "5d1830e67837d3f2ecbcac716dfe6926b5e2eaff9d7204e4f744ffd290a53b0a", + "short_id": "5d1830e6", + "time": "2026-10-03T02:17:13.665128317Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "4a3c71f5e89dc1d99d5fc69a7ceb4e3caa30351d157b2a1f6a84f187497a27b3", + "short_id": "4a3c71f5", + "time": "2026-10-03T02:17:19.230120149Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "0a155533421525d9bb233b309fc151e9f8487e234e5b43ceeaecbc515d03d819", + "short_id": "0a155533", + "time": "2026-10-03T02:17:26.024547211Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "a5822405a635efb8d072313354d8dd7db1a52dcbaa90f1aa1b96469439b3673c", + "short_id": "a5822405", + "time": "2026-10-03T02:17:29.289383959Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "2cd36d952d1a73acff474f77d757ab36c2f2c499be320053521b28f82b198bbd", + "short_id": "2cd36d95", + "time": "2026-10-03T02:17:32.58096138Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "bea276a7afb1d6a9706c28c6e02dc717406ccffb5ad100e42707fc7c1ab0fbb1", + "short_id": "bea276a7", + "time": "2026-10-03T15:24:46.196687533Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "597170905424b94c44bfb1123602ac19e334356c39a5bc5467a759f1a7c231a4", + "short_id": "59717090", + "time": "2026-10-03T15:24:49.113532044Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "c6b5c67b90498708f5e22317db12b270cb8803f953f2c2d110b497581c32d825", + "short_id": "c6b5c67b", + "time": "2026-10-03T15:24:51.392171506Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "cb83c8620c7712086d1329ed028516c3994754814d88ffc1edc0e3c5da470870", + "short_id": "cb83c862", + "time": "2026-10-03T15:24:54.220252524Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "38fbd3bc942d53f1ad51d7688d9c1ff6151b2f42ac55d2903643db6d82c09f07", + "short_id": "38fbd3bc", + "time": "2026-10-03T15:24:57.338804736Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "4ce84465e83323c63fbafdf518854e6388866d649351cc7c14533dfc0af10ce5", + "short_id": "4ce84465", + "time": "2026-10-03T15:24:59.243526267Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "2f72df901c0586964d260c12939c62f2e37e7b53ebadff5ca112bc1269f87d8d", + "short_id": "2f72df90", + "time": "2026-10-03T15:25:02.5847505Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "2c0385d380cdf2ef7afc3dc632365dbf9b2482d07f9cfd740ea26bcf7f3eae59", + "short_id": "2c0385d3", + "time": "2026-10-03T15:25:04.980411947Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "c43f2d0e3545bc8e8cffc0f08e850ff177ceea1b99fa4e9c4be063f64eabec39", + "short_id": "c43f2d0e", + "time": "2026-10-03T15:25:07.97694046Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "5a56d51bb2261293ed20338db657a4014216ee343a4c2c5babb7baa42384453b", + "short_id": "5a56d51b", + "time": "2026-10-03T15:31:39.206098904Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "cf905e063e5a7f2ab4b8f13d71e49b3d381206049f9db380d1044eb774719ee1", + "short_id": "cf905e06", + "time": "2026-10-03T15:31:41.528439072Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "0824a70c01ffff21eda54bd0f17718d494a5698d38a22347b326f2f3eec45689", + "short_id": "0824a70c", + "time": "2026-10-03T15:31:44.135006336Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "e9eee3bd728c5d8d90d7bddff67a4fa3d8a4066eaf09f5e79462429c4e855605", + "short_id": "e9eee3bd", + "time": "2026-10-03T15:31:46.903924425Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "53a4bdf9732eaf6232f11adfe3a4e31239f636050e4ac11b952259e16c10fb16", + "short_id": "53a4bdf9", + "time": "2026-10-03T15:31:48.816807194Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "99c4ad053441005c3134c0adb0d219c46b3067aa3b4ad5800dc29af4e45d0087", + "short_id": "99c4ad05", + "time": "2026-10-03T15:31:51.673121035Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "e2db64cdb00e31b4456033c38dc0653dfb01760748cd302ea0079e9ce3096f94", + "short_id": "e2db64cd", + "time": "2026-10-03T15:31:54.069221581Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "5551bdd9779a6015c59e18831aabec43384a779f9f29e0732fad4479329c3986", + "short_id": "5551bdd9", + "time": "2026-10-03T15:31:56.90845268Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "e3d57107000233d627dddadf0ff2927ebfbe456c8b6236bc8cc31626a88f5185", + "short_id": "e3d57107", + "time": "2026-10-03T15:31:58.824425393Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "2d412cd5471cda5fe76a115426d08be3c183c0b6f0407ed56475dec2ed83e466", + "short_id": "2d412cd5", + "time": "2026-10-04T02:16:44.524065664Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "93a66c3eeb252d344a8f37be70bc8c5f72deec2d74dd9034ce00c993905c5da1", + "short_id": "93a66c3e", + "time": "2026-10-04T02:16:46.534034004Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "6dbbceaccd875c38a73ec105e54112655540d9b42af8e7054e7746328105f10b", + "short_id": "6dbbceac", + "time": "2026-10-04T02:16:48.856969168Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "845595263a5c44660a86ae6d4c37986f2345732dd3163e95309f0daea8b0bc64", + "short_id": "84559526", + "time": "2026-10-04T02:16:51.234048844Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "2abceaa322abcdec2fded16754223cea05290b839d6c2027f1b3bcd485049708", + "short_id": "2abceaa3", + "time": "2026-10-04T02:16:54.045587334Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "a86c8d7fbf9dfa7cf6f2940f54f0262052d40f0832143425cb712cf39f92f61e", + "short_id": "a86c8d7f", + "time": "2026-10-04T02:16:56.03786578Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "40328365ce85747f54d0320548cad5bfc5b78ca46aa7dada73b218d5d7cb5ff8", + "short_id": "40328365", + "time": "2026-10-04T02:16:59.09044838Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "f6c41de0c21de235bf5d79f7d9093da0dde7e3dc6755b307b9f8c25822591970", + "short_id": "f6c41de0", + "time": "2026-10-04T02:17:01.314326343Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "d2b6725ff83dc3e27c778fd449232dc606cc1558a6ae755fccfec10d80d263ea", + "short_id": "d2b6725f", + "time": "2026-10-04T02:17:04.567737509Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "5860f7fdb1582d05e7f37f3cbf9ef42661b8159ceb6d284e2ef057ee7ffffcc6", + "short_id": "5860f7fd", + "time": "2026-10-04T05:35:48.893065122Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "ca56106cb17f6cc89e4f3c4000283aa84971cddf70819beda248b4c076eefcb7", + "short_id": "ca56106c", + "time": "2026-10-04T05:35:51.882332467Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "bc0ff350d6f6726c71e374522fc9abf4305c3c393e10c36d780f0671022051e9", + "short_id": "bc0ff350", + "time": "2026-10-04T05:35:53.871253222Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "87c58189da991a9ddbb0693a8ce041de3256a10bd5d1e1825c742c37437cada6", + "short_id": "87c58189", + "time": "2026-10-04T05:35:55.753162092Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "9203e4296246f392b78ec30cb615a15f474954dd62b864b417ef0229127aa627", + "short_id": "9203e429", + "time": "2026-10-04T05:35:58.238852079Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "f4564d6144e581cf299db34b17345a776dd3665ff9ddd05310751ab6cf2269cb", + "short_id": "f4564d61", + "time": "2026-10-04T05:36:02.446280346Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "021da57387805b8e75cafc464a864593338ff1ca90865737fbd3680844ed9edc", + "short_id": "021da573", + "time": "2026-10-04T05:36:04.447050059Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "7971a80d8bdc1d6445216bd3f77ee192cef9da63d637de75e396beca2f61ae80", + "short_id": "7971a80d", + "time": "2026-10-04T05:36:07.004160186Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "1aa58dd03e49a9e3a7265d38e7fcbcef0d908d618783dc890d5278160ac08e23", + "short_id": "1aa58dd0", + "time": "2026-10-04T05:36:09.28929826Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + }, + { + "id": "4fc5669ec2827ff88e1695c926c55208fc28865a5edbd1a6e7c69d39c2652fb4", + "short_id": "4fc5669e", + "time": "2026-10-05T02:16:23.746641999Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "opengist" + ] + }, + { + "id": "fd47799c2549953365b4f99e59b15211d307a659700ea499baf09b3ddcd9f23a", + "short_id": "fd47799c", + "time": "2026-10-05T02:16:27.087572401Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "privatebin" + ] + }, + { + "id": "c5fe883d04bb89484faf58a18caee309f6142dab1e4e6400ee556f500d52fc6a", + "short_id": "c5fe883d", + "time": "2026-10-05T02:16:29.011227176Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "romm" + ] + }, + { + "id": "5e933e416390028db73e2bdd816b8a638fe25dacd5789dd38ef8f9c7e6f2f8c2", + "short_id": "5e933e41", + "time": "2026-10-05T02:16:30.984303284Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bookstack" + ] + }, + { + "id": "5eea7fae4559e23aa073c3ca2e909d0933e14bb4323aaabf67c2b06f40ae565a", + "short_id": "5eea7fae", + "time": "2026-10-05T02:16:33.356524582Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "docmost" + ] + }, + { + "id": "e5d4d77ac9dee0ed668372488b8ea193ff298950d9b42a1bf09d981d68de3b96", + "short_id": "e5d4d77a", + "time": "2026-10-05T02:16:35.969451814Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "kimai" + ] + }, + { + "id": "b83743cfbebb690d02b66edc3280185463eabb19d1b1a4e804186e56abf16ce3", + "short_id": "b83743cf", + "time": "2026-10-05T02:16:38.825026042Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "paperless-ngx" + ] + }, + { + "id": "1385727c6785e35a35e703635fdda6d9e36d53b3ccaefcc14ee9ca69643a8c13", + "short_id": "1385727c", + "time": "2026-10-05T02:16:42.022557074Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "bentopdf" + ] + }, + { + "id": "c62d034a9cc76eeae303b494d906a7f2525584f243011991552e4aaef2cf2aa4", + "short_id": "c62d034a", + "time": "2026-10-05T02:16:44.050699823Z", + "hostname": "demo-hp", + "tags": [ + "felhom-offbox", + "calibre-web" + ] + } +] \ No newline at end of file diff --git a/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-window.txt b/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-window.txt new file mode 100644 index 00000000..3f64d0b1 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partA/demo-hp-window.txt @@ -0,0 +1,33 @@ +2026-10-05T05:23:54Z +grant HTTP 200 +HTTP 302 +2026/10/05 05:26:03 offbox_window.go:273: [INFO] [offbox] clean-up window 6 (after-run): 18 of 54 planned snapshot(s) removed, 145 -> 127, in 10s +2026/10/05 05:26:08 offbox.go:1244: [INFO] [offbox] backup OK: 9 app(s) backed up, 127 snapshot(s), 2m7s +before 136 after 127 removed 18 added by the run 9 +removed == predicted: True + removed 80ebd0d6 2026-09-13T02:16:27 ['bentopdf'] + removed 29516b0c 2026-09-13T02:16:31 ['bookstack'] + removed 21efaca4 2026-09-13T02:16:35 ['calibre-web'] + removed c8562d68 2026-09-13T02:16:39 ['docmost'] + removed b4350226 2026-09-13T02:16:44 ['kimai'] + removed 1ebc8069 2026-09-13T02:16:54 ['paperless-ngx'] + removed 6fac4ba8 2026-09-13T02:17:01 ['romm'] + removed 8cda9681 2026-09-13T02:17:08 ['opengist'] + removed 605bf9c7 2026-09-13T02:17:11 ['privatebin'] + removed 9efec80f 2026-09-28T13:42:12 ['docmost'] + removed ef800224 2026-09-28T13:42:20 ['opengist'] + removed c9c2414d 2026-09-28T13:42:24 ['bentopdf'] + removed 14981ce0 2026-09-28T13:42:27 ['bookstack'] + removed 693c93e1 2026-09-28T13:42:31 ['calibre-web'] + removed 90da3064 2026-09-28T13:42:36 ['kimai'] + removed 7db18829 2026-09-28T13:43:46 ['paperless-ngx'] + removed 57191440 2026-09-28T13:43:52 ['privatebin'] + removed 0febfb1d 2026-09-28T13:43:55 ['romm'] +--- hub offsite_windows (demo-hp, newest 2) +(6, '2026-10-05 05:25:53', '2026-10-05 05:26:04', 145, 127, 72, 'pruned', 'box') +(2, '2026-10-04 05:36:14', '2026-10-04 05:36:17', 127, 127, None, 'nothing', 'box') +--- hub events since 05:23 UTC +--- notification_log since 05:23 UTC +--- key audit +[{"customer":"Tester-2","lines":1,"pinned":1,"findings":null},{"customer":"demo-felhom","lines":1,"pinned":1,"findings":null},{"customer":"demo-hp","lines":1,"pinned":1,"findings":null},{"customer":"tester-1","lines":1,"pinned":1,"findings":null}] + diff --git a/documentation/audits/night-fixes-2026-10-05/partA/key-audit-after-felhom.json b/documentation/audits/night-fixes-2026-10-05/partA/key-audit-after-felhom.json new file mode 100644 index 00000000..2e80f360 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partA/key-audit-after-felhom.json @@ -0,0 +1 @@ +[{"customer":"Tester-2","lines":1,"pinned":1,"findings":null},{"customer":"demo-felhom","lines":1,"pinned":1,"findings":null},{"customer":"demo-hp","lines":1,"pinned":1,"findings":null},{"customer":"tester-1","lines":1,"pinned":1,"findings":null}] diff --git a/documentation/audits/night-fixes-2026-10-05/partA/lab-real-restic-vs-simulation.txt b/documentation/audits/night-fixes-2026-10-05/partA/lab-real-restic-vs-simulation.txt new file mode 100644 index 00000000..c4f041c9 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partA/lab-real-restic-vs-simulation.txt @@ -0,0 +1,17 @@ +R-867 lab, 2026-10-05, on DooPlex scratchpad (local repo, deleted after). restic binary copied out of +gitea.dooplex.hu/admin/felhom-controller:0.293.0: `restic 0.14.0 compiled with go1.19.8 on linux/amd64`. + +Snapshot set: 2 groups (host box, tags opengist / bookstack), one snapshot per night at 02:15:05 UTC from +2026-08-20 to 2026-10-05, box OFF 2026-09-15..17 (no snapshots), a same-day manual run at 15:00 on 2026-09-10 +and 2026-09-30. 92 snapshots, all UTC. + +Real policy: restic forget --group-by host,tags --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --dry-run --json + -> removes 72 snapshots +Test simulation restic0140Plan (offbox_window_test.go) over the same `restic snapshots --json`: + -> removes 72 snapshots +diff of the two sorted id lists: EMPTY (identical) + +The guard (v0.294.0 candidate) on that plan at now = 2026-10-05T02:20:00Z, maxRemove 500: + GUARD removes=70 refusal="" + (the 2 kept back are the 2026-09-30 02:15 copies superseded by the same-day 15:00 run — R-824's exclusion; + removed in a later window once 7 calendar days old) diff --git a/documentation/audits/night-fixes-2026-10-05/partA/red-proofs.txt b/documentation/audits/night-fixes-2026-10-05/partA/red-proofs.txt new file mode 100644 index 00000000..dcb20f39 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partA/red-proofs.txt @@ -0,0 +1,45 @@ +### RED-PROOF 1 the 8-day age line back: 'if calendarDaysBefore(s.Time, now) < keepDaily {' -> 'if now.Sub(s.Time) < 8*24*time.Hour {' + offbox_window_test.go:425: the honest drop of a 7 d + 5 s snapshot must pass: ids=[] why="the policy would remove snapshot n0 from 2026-09-28T02:15:05Z — within the last 7 days kept daily and not superseded the same day, which honest retention never does" +--- FAIL: TestOffsiteGuard_RealPolicyMeasuredShapeAllowed (0.00s) + offbox_window_test.go:456: night 7 (2026-08-27T02:17:00Z): the guard refused the honest policy: the policy would remove snapshot opengist0 from 2026-08-20T02:15:00Z — within the last 7 days kept daily and not superseded the same day, which honest retention never does +--- FAIL: TestOffsiteGuard_RealPolicySixtyNightsNeverRefuses (0.00s) + offbox_window_test.go:502: weekly window on 2026-09-14 refused: the policy would remove snapshot s6 from 2026-09-07T02:15:00Z — within the last 7 days kept daily and not superseded the same day, which honest retention never does +--- FAIL: TestOffsiteGuard_RealPolicyWeeklyWindows (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.020s +FAIL +rc=1 +### RED-PROOF 2 drop the future-date refusal: 's.Time.After(now.Add(offsiteGuardSkew))' -> 'false && s.Time.After(now.Add(offsiteGuardSkew))' + offbox_window_test.go:155: window close = [{ID:7 CountBefore:16 CountAfter:16 Removed:0 Outcome:guard-refused Reason:snapshot f1 (2027-01-01T03:00:00Z) is newer than the hub allows (2026-10-05T04:55:06Z)}] +--- FAIL: TestOffsiteGuard_LabThirteenFutureFakes_Refused (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.023s +FAIL +rc=1 +### RED-PROOF 3 drop the newer-than-hub refusal: '!newestAllowed.IsZero() && s.Time.After' -> 'false && s.Time.After' + offbox_window_test.go:169: newest-allowed bound not enforced: "" +--- FAIL: TestOffsiteGuard_RecentRemovalRefused (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.023s +FAIL +rc=1 +### RED-PROOF 4 drop the young-removal refusal: 'return nil, fmt.Sprintf("the policy would remove snapshot %s from %s — within' -> 'continue; _ = fmt.Sprintf("the policy would remove snapshot %s from %s — within' + offbox_window_test.go:165: why = "" +--- FAIL: TestOffsiteGuard_RecentRemovalRefused (0.00s) + offbox_window_test.go:232: why = "" +--- FAIL: TestOffsiteGuard_SameDaySupersededYoungExcluded (0.00s) + offbox_window_test.go:532: why = "" — a real snapshot inside the daily window must not be removed +--- FAIL: TestOffsiteGuard_RealPolicySkewWindowFakeRefused (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.024s +FAIL +rc=1 +### RED-PROOF 5 drop the weekly cap: 'if maxRemove >= 0 && len(keep) > maxRemove {' -> 'if false && len(keep) > maxRemove {' + offbox_window_test.go:202: ids=[o5-full o4-full o3-full o2-full o1-full o0-full] why="" +--- FAIL: TestOffsiteGuard_AboveWeeklyCapRefused (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.023s +FAIL +rc=1 +### restored +ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.022s diff --git a/documentation/audits/night-fixes-2026-10-05/partB/r863-live.txt b/documentation/audits/night-fixes-2026-10-05/partB/r863-live.txt new file mode 100644 index 00000000..1a3ca649 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partB/r863-live.txt @@ -0,0 +1,37 @@ +### 05:28:31 pull 0.294.0 in 9202 (outside the timing) +gitea.dooplex.hu/admin/felhom-controller:0.294.0 +0 +### 05:28:34 make the one-time clean-up due: remove its marker; switch the image; restart +gitea.dooplex.hu/admin/felhom-controller:0.294.0 started 2026-10-05T05:28:35.617220084Z +### 05:31:15 DEPLOY bookstack (POST /api/stacks/bookstack/deploy, the endpoint the install button calls) + +HTTP 202 +05:31:20 +05:31:36 +05:31:53 bookstack=Created bookstack-db=Up 9 seconds (health: starting) +05:32:09 bookstack=Up 15 seconds (health: starting) bookstack-db=Up 26 seconds (healthy) +### controller log since start (image retention + bookstack deploy lines) +2026/10/05 05:31:36 controller_image_retention.go:184: [INFO] [stacks] controller image retention: deleted gitea.dooplex.hu/admin/felhom-controller:0.291.0 (dde6b167c496, 410MB) — older than the previous controller and no container uses it (decision 56) +2026/10/05 05:31:36 controller_image_retention.go:115: [INFO] [stacks] controller image retention: pass over 3 controller image(s) — running 0.294.0, previous "0.293.0" (by version order (no swap record names one present)), 1 candidate(s), 1 deleted, the rest kept +2026/10/05 05:31:36 image_retention.go:211: [INFO] [stacks] image retention (one-time clean-up): skipped — an app install, update, restore or undo is pulling images now (R-863); tried again later +2026/10/05 05:31:54 deploy.go:611: [INFO] [stacks] Stack bookstack deployed successfully (took 35.5s) +2026/10/05 05:31:57 manager.go:1598: [INFO] [stacks] Stack bookstack post-start status: +### 05:33:54 after the retry +2026/10/05 05:31:36 image_retention.go:211: [INFO] [stacks] image retention (one-time clean-up): skipped — an app install, update, restore or undo is pulling images now (R-863); tried again later +2026/10/05 05:33:36 image_retention.go:389: [INFO] [stacks] image retention (one-time): deleted 0 image(s). docker disk before: Images 6.245GB 2.943GB (47%) | after: Images 6.245GB 2.943GB (47%) +deleted 0 + +bookstack lscr.io/linuxserver/bookstack:26.09.1 Up 2 minutes (healthy) +bookstack-db mariadb:12.3 Up 2 minutes (healthy) +### 05:34:05 teardown: stop + remove through the product +HTTP 200 + +HTTP 200 +### 05:34:49 verify +0 +ls: cannot access '/opt/docker/stacks/bookstack/app.yaml': No such file or directory +0 +2026/10/05 05:34:41 delete.go:804: [INFO] Stack bookstack removed successfully (took 26.7s) +2026/10/05 05:34:43 image_retention.go:265: [INFO] [stacks] image retention (remove of bookstack): deleted [lscr.io/linuxserver/bookstack:] (b274bee3f111, 416MB) — no container, installed app or undo names it (decision 53) +2026/10/05 05:34:43 image_retention.go:265: [INFO] [stacks] image retention (remove of bookstack): deleted [mariadb:] (dfff46ef3f9d, 334MB) — no container, installed app or undo names it (decision 53) +2026/10/05 05:34:43 image_retention.go:238: [INFO] [stacks] image retention (remove of bookstack): pass over 15 image(s) of [lscr.io/linuxserver/bookstack mariadb] — 2 candidate(s), 2 deleted, the rest kept diff --git a/documentation/audits/night-fixes-2026-10-05/partB/red-proofs.txt b/documentation/audits/night-fixes-2026-10-05/partB/red-proofs.txt new file mode 100644 index 00000000..69e44141 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partB/red-proofs.txt @@ -0,0 +1,28 @@ +### RED-PROOF R-863: TryImageCleanup check dropped (endCleanup, ok := func() {}, true) + image_retention_r863_test.go:70: the install failed: the clean-up deleted the image compose had just pulled (rmi [sha256:MDB]) +--- FAIL: TestR863_OneTimeCleanupDuringFirstInstallKeepsThePulledImage (0.01s) +--- PASS: TestR863_OnlyPullingVerbsHoldTheLock (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.014s +FAIL +rc=1 +### RED-PROOF R-863b: a busy pass reports done (the old behaviour: the one-time clean-up's marker ends it for good) + image_retention_r863_test.go:79: the clean-up reported done although it did not run — its marker would end it for good +--- FAIL: TestR863_OneTimeCleanupDuringFirstInstallKeepsThePulledImage (0.01s) +--- PASS: TestR863_OnlyPullingVerbsHoldTheLock (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.017s +FAIL +rc=1 +### restored +ok gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.018s +### RED-PROOF R-864: truncateStr (head) instead of tailStr for stderr + r864_stderr_tail_test.go:45: the reason was cut from the returned error: "exit code 1\nstderr: Image lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 Pulli +--- FAIL: TestR864_FailedComposeLogsTheReasonNotThePullProgress (0.01s) +--- PASS: TestR864_TailStrKeepsTheEndOnARuneBoundary (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.012s +FAIL +rc=1 +### restored +ok gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.015s diff --git a/documentation/audits/night-fixes-2026-10-05/partC/print-uris-real-apt.txt b/documentation/audits/night-fixes-2026-10-05/partC/print-uris-real-apt.txt new file mode 100644 index 00000000..4c4620b3 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partC/print-uris-real-apt.txt @@ -0,0 +1,28 @@ +R-865 probe, 2026-10-05, scratch guest 9202 on demo-hp (read only). Before/after: .deb files in /var/cache/apt/archives, and the versions. +status: running +10 +bash 5.2.37-2+b10 +libc6:amd64 2.41-12+deb13u4 +---WITH-S +Reading package lists... +Building dependency tree... +Reading state information... +0 upgraded, 0 newly installed, 2 reinstalled, 0 to remove and 1 not upgraded. +Inst libc6 [2.41-12+deb13u4] (2.41-12+deb13u4 Debian:13.7/stable [amd64]) +Conf libc6 (2.41-12+deb13u4 Debian:13.7/stable [amd64]) +Inst bash [5.2.37-2+b10] (5.2.37-2+b10 Debian:13.7/stable [amd64]) +Conf bash (5.2.37-2+b10 Debian:13.7/stable [amd64]) +---WITHOUT-S +Reading package lists... +Building dependency tree... +Reading state information... +0 upgraded, 0 newly installed, 2 reinstalled, 0 to remove and 1 not upgraded. +Need to get 4347 kB of archives. +After this operation, 0 B of additional disk space will be used. +'http://deb.debian.org/debian/pool/main/b/bash/bash_5.2.37-2%2bb10_amd64.deb' bash_5.2.37-2+b10_amd64.deb 1500792 MD5Sum:27b11721fea83d73b96e0f7023863771 +'http://deb.debian.org/debian/pool/main/g/glibc/libc6_2.41-12%2bdeb13u4_amd64.deb' libc6_2.41-12+deb13u4_amd64.deb 2846580 MD5Sum:5559581916477ef1f57ea9f82cecf22e +rc=0 +10 +bash 5.2.37-2+b10 +libc6:amd64 2.41-12+deb13u4 +(rc=0 above is head's; the cache count 10 -> 10 and the versions are unchanged: --print-uris without -s downloads nothing) diff --git a/documentation/audits/night-fixes-2026-10-05/partC/r8-live-installed-wrapper.txt b/documentation/audits/night-fixes-2026-10-05/partC/r8-live-installed-wrapper.txt new file mode 100644 index 00000000..ca58169e --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partC/r8-live-installed-wrapper.txt @@ -0,0 +1,6 @@ +### 05:44:30 demo-hp: the INSTALLED wrapper's download_bytes (R8's measure) for the 13 pending upgrades, as root, read-only (--print-uris) +36e16f7619cc556d +download_bytes = 12802456 B -> R8 needs max(500 MB, 3 x 12802456 B) = 524288000 B free +bind9-libs:amd64 1:9.20.26-1~deb13u1 +openssl 3.5.7-1~deb13u2 +13 diff --git a/documentation/audits/night-fixes-2026-10-05/partC/red-proof.txt b/documentation/audits/night-fixes-2026-10-05/partC/red-proof.txt new file mode 100644 index 00000000..6208a06a --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partC/red-proof.txt @@ -0,0 +1,8 @@ +### RED-PROOF R-865: -s put back into download_bytes +1 +FAIL: test_R8_counts_the_real_download (test_felhom_os_apply.Refusals.test_R8_counts_the_real_download) +FAIL: test_download_bytes_never_simulates (test_felhom_os_apply.Refusals.test_download_bytes_never_simulates) +Ran 79 tests in 0.499s +FAILED (failures=2) +### restored +OK diff --git a/documentation/audits/night-fixes-2026-10-05/partD/r866-kept-copies-sent-at-start.txt b/documentation/audits/night-fixes-2026-10-05/partD/r866-kept-copies-sent-at-start.txt new file mode 100644 index 00000000..93e920b8 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partD/r866-kept-copies-sent-at-start.txt @@ -0,0 +1,11 @@ +Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.454+02:00 level=INFO msg="felhom-agent daemon starting" version=0.144.1 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900 +Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.455+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T055135Z layer=docker vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T055135Z-docker-apply.json +Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.455+02:00 level=INFO msg="osupdate: DONE" run=20261005T055135Z layer=docker vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=0 pending=0 not_covered=0 restart_needed=7 reboot_needed=true wrapper_seconds=14.7 +Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.529+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T055135Z layer=guest vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T055135Z-guest-apply.json +Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.529+02:00 level=INFO msg="osupdate: DONE" run=20261005T055135Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=0 pending=0 not_covered=0 restart_needed=7 reboot_needed=true wrapper_seconds=13.3 +Oct 05 07:53:34 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:34.479+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T055135Z layer=host vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T055135Z-host-apply.json +Oct 05 07:53:34 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:34.479+02:00 level=INFO msg="osupdate: DONE" run=20261005T055135Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=0 pending=80 not_covered=80 restart_needed=28 reboot_needed=false wrapper_seconds=12.1 +Oct 05 07:53:34 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:34.498+02:00 level=INFO msg="osupdate: sent kept report(s)" count=3 +(62, 'demo-felhom-8363b5', '2026-10-05 05:53:34', 'debug', 'apply', 'nothing', 1, 'host') +(61, 'demo-felhom-8363b5', '2026-10-05 05:53:33', 'debug', 'apply', 'nothing', 1, 'guest') +(60, 'demo-felhom-8363b5', '2026-10-05 05:53:33', 'debug', 'apply', 'nothing', 1, 'docker') diff --git a/documentation/audits/night-fixes-2026-10-05/partD/r866-live-hub-away.txt b/documentation/audits/night-fixes-2026-10-05/partD/r866-live-hub-away.txt new file mode 100644 index 00000000..39a69a50 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partD/r866-live-hub-away.txt @@ -0,0 +1,11 @@ +05:51:35 agent felhom-agent 0.144.0; saved block: 2026-10-05 07:23:26 +05:51:35 hub blocked (blackhole routes) +=== felhom-agent 0.144.0 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=SAVED(2026-10-05T05:23:26Z; hub unreachable: hub: transport error: Get "https://hub.felhom.eu/api/v1/hosts/demo-felhom-8363b5/desired-state": dial tcp 37.191.56.193:443: connect: invalid argument) === + "healthy": true, + "outcome": "nothing", + "healthy": true, + "outcome": "nothing", + "healthy": true, + "outcome": "nothing", + pass took 42.8s +05:52:18 hub unblocked; kept copies now: report-20261005T055135Z-docker-apply.json report-20261005T055135Z-guest-apply.json report-20261005T055135Z-host-apply.json diff --git a/documentation/audits/night-fixes-2026-10-05/partD/r866-red-proofs.txt b/documentation/audits/night-fixes-2026-10-05/partD/r866-red-proofs.txt new file mode 100644 index 00000000..ed0fc5ff --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partD/r866-red-proofs.txt @@ -0,0 +1,19 @@ +### RED-PROOF R-866a: selftestOSBlock returns at once on a fetch error (v0.143.0 behaviour) + r866_selftest_block_test.go:37: the pass did not run from the saved block: ok=false block= src="desired state: hub: transport error: connect: invalid argument" +--- FAIL: TestR866_DebugPassUsesTheSavedBlockWhenTheHubIsAway (0.00s) + r866_selftest_block_test.go:53: ok=false why="desired state: down" +--- FAIL: TestR866_NoHubNoSavedBlockDoesNotRun (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent 0.011s +FAIL +rc=1 +### RED-PROOF R-866b: the daemon no longer saves the block + r866_selftest_block_test.go:37: the pass did not run from the saved block: ok=false block= src="desired state: hub: transport error: connect: invalid argument — and no saved block (the daemon saves one when the hub sends it)" +--- FAIL: TestR866_DebugPassUsesTheSavedBlockWhenTheHubIsAway (0.00s) +--- PASS: TestR866_NoHubNoSavedBlockDoesNotRun (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent 0.009s +FAIL +rc=1 +### restored +ok gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent 0.009s diff --git a/documentation/audits/night-fixes-2026-10-05/partD/r868-brokenpipe-red-proof.txt b/documentation/audits/night-fixes-2026-10-05/partD/r868-brokenpipe-red-proof.txt new file mode 100644 index 00000000..7b85bef3 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partD/r868-brokenpipe-red-proof.txt @@ -0,0 +1,14 @@ +### RED-PROOF R-868 (live finding, v0.144.1): log() writes to stderr unguarded (v0.144.0) + raise BrokenPipeError(32, "Broken pipe") +BrokenPipeError: [Errno 32] Broken pipe +FAILED (errors=1) +### restored +OK +### RED-PROOF R-868 loop (v0.144.1): SendUnsentLoop returns after the first look + unsent_test.go:138: the late copy was never sent +--- FAIL: TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop (3.05s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 3.063s +FAIL +### restored +ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 1.754s diff --git a/documentation/audits/night-fixes-2026-10-05/partD/r868-live-kill.txt b/documentation/audits/night-fixes-2026-10-05/partD/r868-live-kill.txt new file mode 100644 index 00000000..42cbdc10 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partD/r868-live-kill.txt @@ -0,0 +1,10 @@ +05:44:46 daemon pid before 2471595; kept copies before: 0 +05:44:46 debug pass started +KILLED at 05:45:04.880: the pass + the daemon (apt-get was running) +bash: line 11: 2525079 Killed nohup sudo -u felhom-agent /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=os-update -vmid 9201 > /root/r868-pass.log 2>&1 +05:45:16 wrapper finished; kept copy: +daemon now pid 2526755 active restarts 1 +audit rc=0 +bind9-libs:amd64 1:9.20.29-1~deb13u1 +openssl 3.5.7-1~deb13u3 +python3.13 3.13.5-2+deb13u5 diff --git a/documentation/audits/night-fixes-2026-10-05/partD/r868-live-retest.txt b/documentation/audits/night-fixes-2026-10-05/partD/r868-live-retest.txt new file mode 100644 index 00000000..95a50679 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partD/r868-live-retest.txt @@ -0,0 +1,24 @@ +### 06:03:27 rollback 2 (the same 13, simulated clean before) +0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded. +0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded. +audit rc=0 +06:03:42 agent felhom-agent 0.144.1; daemon pid 2571767; kept copies before: 0 +KILLED at 06:04:00.258: the pass + the daemon (apt-get was running) +bash: line 10: 2582951 Killed nohup sudo -u felhom-agent /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=os-update -vmid 9201 > /root/r868-pass2.log 2>&1 +06:04:20 wrapper finished; kept copy: report-20261005T060342Z-guest-apply.json +daemon now active restarts 1 +audit rc=0 +bind9-libs:amd64 1:9.20.29-1~deb13u1 +openssl 3.5.7-1~deb13u3 +python3.13 3.13.5-2+deb13u5 +Oct 05 08:02:59 demo-hp felhom-os-apply[2580370]: os-apply: BUNDLE DONE agent=0.144.1 written=1 same=20 self-check=ok signers-created=False +Oct 05 08:03:54 demo-hp felhom-os-apply[2583726]: os-apply: PLAN upgrade=13 already=0 not-installed=0 from-snapshot=0 +Oct 05 08:04:12 demo-hp felhom-os-apply[2585855]: os-apply: DONE rc=0 seconds=9.4 upgraded=13 restart-needed=containerd-shim,cron,dbus-daemon,sshd,systemd,systemd-journal,systemd-logind,systemd-network reboot-needed=yes +### 06:09:09 the copy is gone +Oct 05 08:04:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:04:05.487+02:00 level=INFO msg="felhom-agent daemon starting" version=0.144.1 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900 +Oct 05 08:09:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:09:05.492+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T060342Z layer=guest vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T060342Z +Oct 05 08:09:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:09:05.492+02:00 level=INFO msg="osupdate: DONE" run=20261005T060342Z layer=guest vmid=9201 ring=0 trigger=debug outcome=applied healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=13 pending=0 not_covered=0 re +Oct 05 08:09:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:09:05.558+02:00 level=INFO msg="osupdate: sent kept report(s)" count=1 +--- hub os_reports (demo-hp, newest 2) +(63, '2026-10-05 06:09:05', 'debug', 'apply', 'applied', 1, 'guest', '20261005T060342Z', 13) +(59, '2026-10-05 02:59:56', 'debug', 'apply', 'nothing', 1, 'docker', '20261005T025842Z', None) diff --git a/documentation/audits/night-fixes-2026-10-05/partD/r868-red-proofs.txt b/documentation/audits/night-fixes-2026-10-05/partD/r868-red-proofs.txt new file mode 100644 index 00000000..6b4a5ff4 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partD/r868-red-proofs.txt @@ -0,0 +1,33 @@ +### RED-PROOF R-868a: SendUnsent sends nothing (sendUnsentLocked returns 0 at once) + unsent_test.go:31: sent 0, want 1 +--- FAIL: TestR868_KilledPassIsReportedAtStart (0.00s) +--- PASS: TestR868_NormalPassLeavesNoCopyAndNoDuplicate (0.00s) + unsent_test.go:91: sent 0: [] +--- FAIL: TestR868_HubAwayKeepsTheCopy (0.00s) + unsent_test.go:111: not sent after the pass: 0 +--- FAIL: TestR868_RunningPassKeepsTheSenderOff (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.014s +FAIL +rc=1 +### RED-PROOF R-868b: finish no longer deletes the sent copy + unsent_test.go:31: sent 0, want 1 +--- FAIL: TestR868_KilledPassIsReportedAtStart (0.00s) + unsent_test.go:59: kept copies left after the hub got them: [/tmp/TestR868_NormalPassLeavesNoCopyAndNoDuplicate3889087443/001/report-20261004T040000Z-docker-apply.json /tmp/TestR868_NormalPassLeavesNoCopyAndNoDuplicate3889087443/001/report-20261004T040000Z-guest-apply.json /tmp/TestR868_NormalPassLeavesNoCopyAndNoDuplicate3889087443/001/report-20261004T040000Z-host-apply.json] +--- FAIL: TestR868_NormalPassLeavesNoCopyAndNoDuplicate (0.00s) + unsent_test.go:91: sent 0: [{RunID:20261004T040000Z Layer:docker Trigger:night Mode:apply Ring:0 ReleaseID: Outcome:nothing Healthy:true HealthReason:sent after the agent stopped mid-pass (R-868) VMID:0 Upgraded:[] Installed:[] Pending:[] NotCovered:[] RestartNeeded:[] DockerRestartNeeded:false RebootNeeded:false RebootScanned:false Refused:[] PassSeconds:0 DockerEngine: Authority: Undo:false unsent:} {RunID:20261004T040000Z Layer:guest Trigger:night Mode:apply Ring:0 ReleaseID: Outcome:applied Healthy:true HealthReason:sent after the agent stopped mid-pass (R-868) VMID:0 Upgraded:[{Name:libc6 Version:u4 Origin:}] Installed:[] Pending:[] NotCovered:[] RestartNeeded:[] DockerRestartNeeded:false RebootNeeded:false RebootScanned:false Refused:[] PassSeconds:0 DockerEngine: Authority: Undo:false unsent:}] +--- FAIL: TestR868_HubAwayKeepsTheCopy (0.00s) + unsent_test.go:111: not sent after the pass: 0 +--- FAIL: TestR868_RunningPassKeepsTheSenderOff (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.012s +FAIL +rc=1 +### RED-PROOF R-868c (wrapper): main() no longer saves the copy +ERROR: test_a_refusal_is_kept_too (test_felhom_os_apply.UnsentReport.test_a_refusal_is_kept_too) +FAIL: test_apply_keeps_a_copy_with_the_agents_ids (test_felhom_os_apply.UnsentReport.test_apply_keeps_a_copy_with_the_agents_ids) +Ran 86 tests in 0.520s +FAILED (failures=1, errors=1) +### restored +ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 2.328s +OK diff --git a/documentation/audits/night-fixes-2026-10-05/partD/r869-red-proof.txt b/documentation/audits/night-fixes-2026-10-05/partD/r869-red-proof.txt new file mode 100644 index 00000000..6f47ce6a --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partD/r869-red-proof.txt @@ -0,0 +1,9 @@ +### RED-PROOF R-869: newPath assigned after the log line (v0.293.0 order) + offbox_window_test.go:271: the line names no destination: +--- FAIL: TestR869_MoveAsideLineNamesTheDestination (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.009s +FAIL +rc=1 +### restored +ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.008s diff --git a/documentation/audits/night-fixes-2026-10-05/partD/raw/r868-pass.log b/documentation/audits/night-fixes-2026-10-05/partD/raw/r868-pass.log new file mode 100644 index 00000000..1ece4c39 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partD/raw/r868-pass.log @@ -0,0 +1,2 @@ +=== felhom-agent 0.144.0 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub === +time=2026-10-05T07:44:46.566+02:00 level=INFO msg="osupdate: START" run=20261005T054446Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261005T054446Z diff --git a/documentation/audits/night-fixes-2026-10-05/partD/raw/r868-pass2.log b/documentation/audits/night-fixes-2026-10-05/partD/raw/r868-pass2.log new file mode 100644 index 00000000..7a1741a2 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partD/raw/r868-pass2.log @@ -0,0 +1,2 @@ +=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub === +time=2026-10-05T08:03:42.262+02:00 level=INFO msg="osupdate: START" run=20261005T060342Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261005T060342Z diff --git a/documentation/audits/night-fixes-2026-10-05/partE/e0-before.txt b/documentation/audits/night-fixes-2026-10-05/partE/e0-before.txt new file mode 100644 index 00000000..64b5b088 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/e0-before.txt @@ -0,0 +1,39 @@ +### 06:09:35 demo-hp before the crash +{ + "armed": true, + "boot_id": "182af911-22a9-41ec-97ac-7ee247a4d470", + "config": { + "LIMIT": 3, + "PANIC_SECONDS": 10, + "REARM_HOURS": 24, + "WINDOW_MINUTES": 60 + }, + "kernel_panic": 10, + "last_boot_at": "2026-10-04T15:17:26Z", + "last_boot_unclean": true, + "last_trip": { + "at": "2026-10-04T14:54:43Z", + "reason": "2 unclean boots within 60 minutes \u2014 the next crash leaves the box off (limit 3)" + }, + "rearmed_at": "2026-10-04T15:33:25Z", + "rearmed_by": "operator", + "tripped": false, + "unclean_boots": [ + "2026-10-04T14:52:08Z", + "2026-10-04T14:54:33Z", + "2026-10-04T15:17:26Z" + ], + "unclean_boots_24h": 3, + "unclean_boots_in_window": 0, + "updated_at": "2026-10-05T05:34:05Z", + "version": 1 +} +kernel.panic = 10 +kernel.sysrq = 438 + 08:09:36 up 14:52, 1 user, load average: 0.91, 1.14, 1.10 + -2 cc0aa40bac534524a46f6c5983ce5ed9 Sun 2026-10-04 16:52:13 CEST Sun 2026-10-04 16:53:36 CEST + -1 53f69dfb93ef44c3a9ad803d86dfb0b3 Sun 2026-10-04 16:54:38 CEST Sun 2026-10-04 16:55:14 CEST + 0 182af91122a941ec97ac7ee247a4d470 Sun 2026-10-04 17:17:31 CEST Mon 2026-10-05 08:09:37 CEST +20 +21 +felhom-agent 0.144.1 diff --git a/documentation/audits/night-fixes-2026-10-05/partE/e1-rollback.txt b/documentation/audits/night-fixes-2026-10-05/partE/e1-rollback.txt new file mode 100644 index 00000000..704c0db9 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/e1-rollback.txt @@ -0,0 +1,22 @@ +### 06:09:53 rollback 3 (13 packages; simulated first) +0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded. +0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded. +audit rc=0 +bind9-dnsutils 1:9.20.26-1~deb13u1 +bind9-host 1:9.20.26-1~deb13u1 +bind9-libs:amd64 1:9.20.26-1~deb13u1 +libpcre2-8-0:amd64 10.46-1~deb13u2 +libpython3.13-minimal:amd64 3.13.5-2+deb13u4 +libpython3.13-stdlib:amd64 3.13.5-2+deb13u4 +libssh2-1t64:amd64 1.11.1-1+deb13u1 +libssl3t64:amd64 3.5.7-1~deb13u2 +libxml2:amd64 2.12.7+dfsg+really2.9.14-2.1+deb13u1 +openssl 3.5.7-1~deb13u2 +openssl-provider-legacy 3.5.7-1~deb13u2 +python3.13 3.13.5-2+deb13u4 +python3.13-minimal 3.13.5-2+deb13u4 +### 06:12:44 attempt 1 made no crash: the watcher pattern 'dpkg --status-fd' never matched (apt puts options before --status-fd); the pass installed the 13 normally. Pattern now 'bin/dpkg .*--status-fd'. +### 06:12:44 rollback 4 +0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded. +0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded. +audit rc=0 diff --git a/documentation/audits/night-fixes-2026-10-05/partE/e2-crash.txt b/documentation/audits/night-fixes-2026-10-05/partE/e2-crash.txt new file mode 100644 index 00000000..58634f57 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/e2-crash.txt @@ -0,0 +1,9 @@ +06:10:25.835 debug pass started +no dpkg seen +ssh ended 06:12:25 +### attempt 2 +06:13:33.983 debug pass started +dpkg running at 06:13:55.070: +CRASH at 06:13:55.904 +Timeout, server 192.168.0.104 not responding. +ssh ended 06:14:03 diff --git a/documentation/audits/night-fixes-2026-10-05/partE/e3-back.txt b/documentation/audits/night-fixes-2026-10-05/partE/e3-back.txt new file mode 100644 index 00000000..132abd6e --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/e3-back.txt @@ -0,0 +1,10 @@ +ssh back 06:14:48 + 08:14:47 up 0 min, 1 user, load average: 1.33, 0.29, 0.10 + "armed": true, + "last_boot_at": "2026-10-05T06:14:33Z", + "last_boot_unclean": true, + "rearmed_at": "2026-10-04T15:33:25Z", + "rearmed_by": "operator", + "tripped": false, + "unclean_boots_in_window": 1, +kernel.panic = 10 diff --git a/documentation/audits/night-fixes-2026-10-05/partE/e4-dpkg-at-boot.txt b/documentation/audits/night-fixes-2026-10-05/partE/e4-dpkg-at-boot.txt new file mode 100644 index 00000000..907229bb --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/e4-dpkg-at-boot.txt @@ -0,0 +1,20 @@ +### 06:14:56 guest state after the crash +status: running +audit rc=0 +bind9-dnsutils 1:9.20.26-1~deb13u1 ii +bind9-host 1:9.20.26-1~deb13u1 ii +bind9-libs 1:9.20.26-1~deb13u1 ii +libpcre2-8-0 10.46-1~deb13u2 ii +libpython3.13-minimal 3.13.5-2+deb13u4 ii +libpython3.13-stdlib 3.13.5-2+deb13u4 ii +libssh2-1t64 1.11.1-1+deb13u1 ii +libssl3t64 3.5.7-1~deb13u2 ii +libxml2 2.12.7+dfsg+really2.9.14-2.1+deb13u1 ii +openssl 3.5.7-1~deb13u2 ii +openssl-provider-legacy 3.5.7-1~deb13u3 ii +python3.13 3.13.5-2+deb13u4 ii +python3.13-minimal 3.13.5-2+deb13u4 ii +plan-20261005T025659Z-guest-apply.json +plan-20261005T054446Z-guest-apply.json +plan-20261005T061334Z-guest-apply.json +06:17:41 apps: 20 healthy of 21 diff --git a/documentation/audits/night-fixes-2026-10-05/partE/e5-next-pass.txt b/documentation/audits/night-fixes-2026-10-05/partE/e5-next-pass.txt new file mode 100644 index 00000000..659900bf --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/e5-next-pass.txt @@ -0,0 +1,14 @@ +### 06:17:43 the next pass (debug, by hand — the night leg's 20 h gap would hold the real one) +=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub === + "healthy": false, + "outcome": "failed", + "reboot_needed": false, + pass took 20.2s +Oct 05 08:13:35 demo-hp felhom-os-apply[2618444]: os-apply: START release=ring0-20261005T061334Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0 +Oct 05 08:13:40 demo-hp felhom-os-apply[2619128]: os-apply: REPAIR configured=0 fixed=0 +Oct 05 08:13:46 demo-hp felhom-os-apply[2619504]: os-apply: PLAN upgrade=13 already=0 not-installed=0 from-snapshot=0 +Oct 05 08:17:44 demo-hp felhom-os-apply[16225]: os-apply: START release=ring0-20261005T061743Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0 +Oct 05 08:17:49 demo-hp felhom-os-apply[16446]: os-apply: REPAIR configured=0 fixed=0 +Oct 05 08:17:55 demo-hp felhom-os-apply[16789]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0 +Oct 05 08:18:03 demo-hp felhom-os-apply[17194]: os-apply: FAILED rc=100 step=install — dpkg state: clean +audit rc=0 diff --git a/documentation/audits/night-fixes-2026-10-05/partE/e6-why-repair-missed.txt b/documentation/audits/night-fixes-2026-10-05/partE/e6-why-repair-missed.txt new file mode 100644 index 00000000..e1497c46 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/e6-why-repair-missed.txt @@ -0,0 +1,13 @@ +### 06:18:33 why: dpkg's update journal (apt's 'interrupted' check) vs dpkg --audit +total 20 +drwxr-xr-x 2 root root 4096 Oct 5 06:13 . +drwxr-xr-x 7 root root 4096 Oct 5 06:13 .. +-rw-r--r-- 1 root root 1048 Oct 5 06:13 0000 +-rw-r--r-- 1 root root 1055 Oct 5 06:13 0001 +-rw-r--r-- 1 root root 1017 Oct 5 06:13 0002 +audit rc=0 +Inst openssl [3.5.7-1~deb13u2] (3.5.7-1~deb13u3 Debian-Security:13/stable-security [amd64]) +Conf openssl (3.5.7-1~deb13u3 Debian-Security:13/stable-security [amd64]) +905: def repair(self): +911: self.x(APT_ENV + ["dpkg", "--configure", "-a", "--force-confold"]) +916: self.r.log(f"os-apply: REPAIR configured={configured} fixed={fixed}") diff --git a/documentation/audits/night-fixes-2026-10-05/partE/e7-repair-and-current.txt b/documentation/audits/night-fixes-2026-10-05/partE/e7-repair-and-current.txt new file mode 100644 index 00000000..89d49fc6 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/e7-repair-and-current.txt @@ -0,0 +1,27 @@ +### 06:18:55 by-hand repair: dpkg --configure -a in the guest (what the wrapper's REPAIR would do) +rc=0 +0 +### 06:18:58 the next pass after the repair +=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub === + "healthy": true, + "outcome": "applied", + "healthy": true, + "outcome": "nothing", + "healthy": true, + "outcome": "nothing", + pass took 1m16.9s +Oct 05 08:17:49 demo-hp felhom-os-apply[16446]: os-apply: REPAIR configured=0 fixed=0 +Oct 05 08:17:55 demo-hp felhom-os-apply[16789]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0 +Oct 05 08:18:03 demo-hp felhom-os-apply[17194]: os-apply: FAILED rc=100 step=install — dpkg state: clean +Oct 05 08:19:04 demo-hp felhom-os-apply[20081]: os-apply: REPAIR configured=0 fixed=0 +Oct 05 08:19:10 demo-hp felhom-os-apply[20406]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0 +Oct 05 08:19:28 demo-hp felhom-os-apply[21616]: os-apply: DONE rc=0 seconds=10.7 upgraded=12 restart-needed=cron,dbus-daemon,sshd,systemd,systemd-journal,systemd-logind,systemd-network reboot-needed=y +Oct 05 08:19:40 demo-hp felhom-os-apply[22557]: os-apply: REPAIR configured=0 fixed=0 +Oct 05 08:19:44 demo-hp felhom-os-apply[22730]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0 +Oct 05 08:19:44 demo-hp felhom-os-apply[22731]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +Oct 05 08:20:02 demo-hp felhom-os-apply[24349]: os-apply: REPAIR configured=0 fixed=0 +Oct 05 08:20:07 demo-hp felhom-os-apply[24637]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0 +Oct 05 08:20:07 demo-hp felhom-os-apply[24638]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +audit rc=0 +0 +package list == before rollback 1 (279 lines) diff --git a/documentation/audits/night-fixes-2026-10-05/partE/e8-household-mails-system.txt b/documentation/audits/night-fixes-2026-10-05/partE/e8-household-mails-system.txt new file mode 100644 index 00000000..024177b9 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/e8-household-mails-system.txt @@ -0,0 +1,9 @@ +--- hub events demo-hp since 06:13 UTC +('controller_started', 'info', 'Controller elindult (0.294.0)', 'controller', '2026-10-05 06:16:39') +('os_update_failed', 'error', 'OS update (guest) on demo-hp-bb76ea failed: {"dpkg_audit":"clean","rc":100,"tail":["E: dpkg was interrupted, you must manually run \'sudo dpkg --config', 'hub', '2026-10-05 06:18:03') +('os_update_applied', 'info', 'System security fixes installed on the box (12 package(s)).', 'hub', '2026-10-05 06:19:35') +--- notification_log since 06:13 UTC (all customers) +('demo-hp', 'os_update_failed', 'sent', 'operator', '2026-10-05 06:18:03') +('demo-hp', 'os_update_failed', 'skipped', 'customer', '2026-10-05 06:18:03') +--- System page line (demo-hp) +go · nothing · 12 s demo-hp-bb76ea Demo HP partial: no running customer guest ring 0 → normal updates ON switch off unknown 9.2.2 unknown unknown unknown ring0-20261005T061858Z 78 0 unknown no unknown no unknown not installed 0.144.1 unknown ring0-20261005T061858Z 0 7 unknown unknown unknown ring0-20261005T061858Z 0 min ago · nothing · 19 s tester-1-d70be4 Tester 1 ring 1 → demo updates ON switch off running 9.2.2 7.0.2-6-pve 7.0.2-6-pve 13.5 — 185 106 none no 10 s no 0 armed 0.144.1 13.6 — 49 0 29.8.2 2.3.6-1~debian.13~trixie on — 10 h ago · nothing · 16 s Amber: worth a look. Red: an operator alarm fires (`08` §6.3). "unknown": the box could not read the value — never a guess. Felhom Hub 0.133.0 diff --git a/documentation/audits/night-fixes-2026-10-05/partE/raw/9201-pkgs-after-parte.txt b/documentation/audits/night-fixes-2026-10-05/partE/raw/9201-pkgs-after-parte.txt new file mode 100644 index 00000000..5e5f045a --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/raw/9201-pkgs-after-parte.txt @@ -0,0 +1,279 @@ +adduser 3.152 +apparmor 4.1.0-1 +apt 3.0.3 +apt-listchanges 4.8 +apt-utils 3.0.3 +base-files 13.8+deb13u7 +base-passwd 3.6.7 +bash 5.2.37-2+b10 +bash-completion 1:2.16.0-7 +bind9-dnsutils 1:9.20.29-1~deb13u1 +bind9-host 1:9.20.29-1~deb13u1 +bind9-libs 1:9.20.29-1~deb13u1 +bsdextrautils 2.41.5-0+deb13u1 +bsdutils 1:2.41.5-0+deb13u1 +bzip2 1.0.8-6 +ca-certificates 20250419 +containerd.io 2.3.6-1~debian.13~trixie +coreutils 9.7-3 +cpio 2.15+dfsg-2 +cron 3.0pl1-197 +cron-daemon-common 3.0pl1-197 +curl 8.14.1-2+deb13u5 +dash 0.5.12-12 +dbus 1.16.2-2 +dbus-bin 1.16.2-2 +dbus-daemon 1.16.2-2 +dbus-session-bus-common 1.16.2-2 +dbus-system-bus-common 1.16.2-2 +dbus-user-session 1.16.2-2 +debconf 1.5.91 +debconf-i18n 1.5.91 +debian-archive-keyring 2025.1 +debian-faq 12.2 +debianutils 5.23.2 +dhcpcd-base 1:10.1.0-11+deb13u4 +diffutils 1:3.10-4 +distro-info-data 0.66+deb13u2 +dmidecode 3.6-2 +doc-debian 11.3+nmu1 +docker-buildx-plugin 0.37.1-1~debian.13~trixie +docker-ce 5:29.8.2-1~debian.13~trixie +docker-ce-cli 5:29.8.2-1~debian.13~trixie +docker-ce-rootless-extras 5:29.8.2-1~debian.13~trixie +docker-compose-plugin 5.6.0-1~debian.13~trixie +dpkg 1.22.22 +e2fsprogs 1.47.2-3+b12 +fdisk 2.41.5-0+deb13u1 +file 1:5.46-5 +findutils 4.10.0-3 +gcc-14-base 14.2.0-19 +gettext-base 0.23.1-2 +git 1:2.47.3-0+deb13u1 +git-man 1:2.47.3-0+deb13u1 +grep 3.11-4 +groff-base 1.23.0-9 +gzip 1.13-1+deb13u1 +hostname 3.25 +ifupdown 0.8.44+deb13u1 +ifupdown2 3.0.0-1.3 +inetutils-telnet 2:2.6-3+deb13u3 +init 1.69~deb13u1 +init-system-helpers 1.69~deb13u1 +iproute2 6.15.0-1 +iptables 1.8.11-2 +iputils-ping 3:20240905-3 +isc-dhcp-client 4.4.3-P1-8 +kmod 34.2-2 +krb5-locales 1.21.3-5+deb13u1 +less 668-1 +libacl1 2.3.2-2+b1 +libapparmor1 4.1.0-1 +libapt-pkg7.0 3.0.3 +libattr1 1:2.5.2-3 +libaudit-common 1:4.0.2-2+deb13u1 +libaudit1 1:4.0.2-2+deb13u1 +libblkid1 2.41.5-0+deb13u1 +libbpf1 1:1.5.0-3 +libbrotli1 1.1.0-2+b7 +libbsd0 0.12.2-2 +libbz2-1.0 1.0.8-6 +libc-bin 2.41-12+deb13u4 +libc-l10n 2.41-12+deb13u4 +libc6 2.41-12+deb13u4 +libcap-ng0 0.8.5-4+b1 +libcap2 1:2.75-10+deb13u1+b3 +libcap2-bin 1:2.75-10+deb13u1+b3 +libcbor0.10 0.10.2-2 +libcom-err2 1.47.2-3+b12 +libcrypt1 1:4.4.38-1 +libcurl3t64-gnutls 8.14.1-2+deb13u5 +libcurl4t64 8.14.1-2+deb13u5 +libdb5.3t64 5.3.28+dfsg2-9 +libdbus-1-3 1.16.2-2 +libdebconfclient0 0.280 +libedit2 3.1-20250104-1 +libelf1t64 0.192-4 +liberror-perl 0.17030-1 +libexpat1 2.8.3-1~deb13u1 +libext2fs2t64 1.47.2-3+b12 +libfdisk1 2.41.5-0+deb13u1 +libffi8 3.4.8-2 +libfido2-1 1.15.0-1+b1 +libfstrm0 0.6.1-1+b3 +libgcc-s1 14.2.0-19 +libgdbm-compat4t64 1.24-2 +libgdbm6t64 1.24-2 +libgmp10 2:6.3.0+dfsg-3 +libgnutls30t64 3.8.9-3+deb13u4 +libgssapi-krb5-2 1.21.3-5+deb13u1 +libhogweed6t64 3.10.1-1 +libicu76 76.1-4 +libidn2-0 2.3.8-2 +libip4tc2 1.8.11-2 +libip6tc2 1.8.11-2 +libjansson4 2.14-2+b3 +libjemalloc2 5.3.0-3 +libjson-c5 0.18+ds-1 +libk5crypto3 1.21.3-5+deb13u1 +libkeyutils1 1.6.3-6 +libkmod2 34.2-2 +libkrb5-3 1.21.3-5+deb13u1 +libkrb5support0 1.21.3-5+deb13u1 +liblastlog2-2 2.41.5-0+deb13u1 +libldap-common 2.6.10+dfsg-1 +libldap2 2.6.10+dfsg-1 +liblmdb0 0.9.31-1+b2 +liblocale-gettext-perl 1.07-7+b1 +liblockfile-bin 1.17-2 +liblsof0 4.99.4+dfsg-2 +liblz4-1 1.10.0-4 +liblzma5 5.8.1-1+deb13u1 +libmagic-mgc 1:5.46-5 +libmagic1t64 1:5.46-5 +libmaxminddb0 1.12.2-1 +libmd0 1.1.0-2+b1 +libmnl0 1.0.5-3 +libmount1 2.41.5-0+deb13u1 +libncursesw6 6.5+20250216-2 +libnetfilter-conntrack3 1.1.0-1 +libnettle8t64 3.10.1-1 +libnewt0.52 0.52.25-1 +libnfnetlink0 1.0.2-3 +libnftables1 1.1.3-1 +libnftnl11 1.2.9-1 +libnghttp2-14 1.64.0-1.1+deb13u1 +libnghttp3-9 1.8.0-1 +libngtcp2-16 1.11.0-1+deb13u1 +libngtcp2-crypto-gnutls8 1.11.0-1+deb13u1 +libnsl2 1.3.0-3+b3 +libnss-systemd 257.13-1~deb13u1 +libp11-kit0 0.25.5-3 +libpam-modules 1.7.0-5 +libpam-modules-bin 1.7.0-5 +libpam-runtime 1.7.0-5 +libpam-systemd 257.13-1~deb13u1 +libpam-wtmpdb 0.73.0-3+deb13u1 +libpam0g 1.7.0-5 +libpcre2-8-0 10.46-1~deb13u3 +libperl5.40 5.40.1-6+deb13u1 +libpipeline1 1.5.8-1 +libpopt0 1.19+dfsg-2 +libproc2-0 2:4.0.4-9 +libprotobuf-c1 1.5.1-1 +libpsl5t64 0.21.2-1.1+b1 +libpython3-stdlib 3.13.5-1 +libpython3.13-minimal 3.13.5-2+deb13u5 +libpython3.13-stdlib 3.13.5-2+deb13u5 +libreadline8t64 8.2-6 +librtmp1 2.4+20151223.gitfa8646d.1-2+b5 +libsasl2-2 2.1.28+dfsg1-9 +libsasl2-modules-db 2.1.28+dfsg1-9 +libseccomp2 2.6.0-2 +libselinux1 3.8.1-1 +libsemanage-common 3.8.1-1 +libsemanage2 3.8.1-1 +libsepol2 3.8.1-1 +libslang2 2.3.3-5+b2 +libsmartcols1 2.41.5-0+deb13u1 +libsqlite3-0 3.46.1-7+deb13u2 +libss2 1.47.2-3+b12 +libssh2-1t64 1.11.1-1+deb13u2 +libssl3t64 3.5.7-1~deb13u3 +libstdc++6 14.2.0-19 +libsystemd-shared 257.13-1~deb13u1 +libsystemd0 257.13-1~deb13u1 +libtasn1-6 4.20.0-2+deb13u1 +libtext-charwidth-perl 0.04-11+b4 +libtext-iconv-perl 1.7-8+b4 +libtext-wrapi18n-perl 0.06-10 +libtinfo6 6.5+20250216-2 +libtirpc-common 1.3.6+ds-1 +libtirpc3t64 1.3.6+ds-1 +libtlsrpt0 0.5.0rc1-2 +libuchardet0 0.0.8-1+b2 +libudev1 257.13-1~deb13u1 +libunistring5 1.3-2 +liburcu8t64 0.15.2-2 +libuuid1 2.41.5-0+deb13u1 +libuv1t64 1.50.0-2 +libwrap0 7.6.q-36 +libwtmpdb0 0.73.0-3+deb13u1 +libxml2 2.12.7+dfsg+really2.9.14-2.1+deb13u3 +libxtables12 1.8.11-2 +libxxhash0 0.8.3-2 +libzstd1 1.5.7+dfsg-1 +linux-sysctl-defaults 4.12.1 +locales 2.41-12+deb13u4 +login 1:4.16.0-2+really2.41.5-0+deb13u1 +login.defs 1:4.17.4-2 +logrotate 3.22.0-1 +logsave 1.47.2-3+b12 +lsof 4.99.4+dfsg-2 +man-db 2.13.1-1 +manpages 6.9.1-1 +mawk 1.3.4.20250131-1 +media-types 13.0.0 +mount 2.41.5-0+deb13u1 +nano 8.4-1+deb13u1 +ncurses-base 6.5+20250216-2 +ncurses-bin 6.5+20250216-2 +ncurses-term 6.5+20250216-2 +netbase 6.5 +netcat-traditional 1.10-50 +nftables 1.1.3-1 +openssh-client 1:10.0p1-7+deb13u4 +openssh-server 1:10.0p1-7+deb13u4 +openssh-sftp-server 1:10.0p1-7+deb13u4 +openssl 3.5.7-1~deb13u3 +openssl-provider-legacy 3.5.7-1~deb13u3 +passwd 1:4.17.4-2 +patch 2.8-2 +perl 5.40.1-6+deb13u1 +perl-base 5.40.1-6+deb13u1 +perl-modules-5.40 5.40.1-6+deb13u1 +pigz 2.8-1 +postfix 3.10.13-0+deb13u1 +procps 2:4.0.4-9 +python-apt-common 3.0.0 +python3 3.13.5-1 +python3-apt 3.0.0 +python3-certifi 2025.1.31+ds-1 +python3-chardet 5.2.0+dfsg-2 +python3-charset-normalizer 3.4.2-1 +python3-debconf 1.5.91 +python3-debian 1.0.1 +python3-debianbts 4.1.1 +python3-idna 3.10-1+deb13u1 +python3-minimal 3.13.5-1 +python3-reportbug 13.2.0 +python3-requests 2.32.3+dfsg-5+deb13u1 +python3-urllib3 2.3.0-3+deb13u2 +python3.13 3.13.5-2+deb13u5 +python3.13-minimal 3.13.5-2+deb13u5 +readline-common 8.2-6 +reportbug 13.2.0 +runit-helper 2.16.4 +sed 4.9-2+deb13u1 +sensible-utils 0.0.25 +sqv 1.3.0-3+b2 +ssh 1:10.0p1-7+deb13u4 +systemd 257.13-1~deb13u1 +systemd-sysv 257.13-1~deb13u1 +systemd-timesyncd 257.13-1~deb13u1 +sysvinit-utils 3.14-4 +tar 1.35+dfsg-3.1 +traceroute 1:2.1.6-1 +tzdata 2026c-0+deb13u1 +ucf 3.0052 +udev 257.13-1~deb13u1 +util-linux 2.41.5-0+deb13u1 +util-linux-extra 2.41.5-0+deb13u1 +vim-common 2:9.1.1230-2 +vim-tiny 2:9.1.1230-2 +wamerican 2020.12.07-4 +wget 1.25.0-2 +whiptail 0.52.25-1 +wtmpdb 0.73.0-3+deb13u1 +xz-utils 5.8.1-1+deb13u1 +zlib1g 1:1.3.dfsg+really1.3.1-1+b1 diff --git a/documentation/audits/night-fixes-2026-10-05/partE/raw/9201-pkgs-before-rollback1.txt b/documentation/audits/night-fixes-2026-10-05/partE/raw/9201-pkgs-before-rollback1.txt new file mode 100644 index 00000000..5e5f045a --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/raw/9201-pkgs-before-rollback1.txt @@ -0,0 +1,279 @@ +adduser 3.152 +apparmor 4.1.0-1 +apt 3.0.3 +apt-listchanges 4.8 +apt-utils 3.0.3 +base-files 13.8+deb13u7 +base-passwd 3.6.7 +bash 5.2.37-2+b10 +bash-completion 1:2.16.0-7 +bind9-dnsutils 1:9.20.29-1~deb13u1 +bind9-host 1:9.20.29-1~deb13u1 +bind9-libs 1:9.20.29-1~deb13u1 +bsdextrautils 2.41.5-0+deb13u1 +bsdutils 1:2.41.5-0+deb13u1 +bzip2 1.0.8-6 +ca-certificates 20250419 +containerd.io 2.3.6-1~debian.13~trixie +coreutils 9.7-3 +cpio 2.15+dfsg-2 +cron 3.0pl1-197 +cron-daemon-common 3.0pl1-197 +curl 8.14.1-2+deb13u5 +dash 0.5.12-12 +dbus 1.16.2-2 +dbus-bin 1.16.2-2 +dbus-daemon 1.16.2-2 +dbus-session-bus-common 1.16.2-2 +dbus-system-bus-common 1.16.2-2 +dbus-user-session 1.16.2-2 +debconf 1.5.91 +debconf-i18n 1.5.91 +debian-archive-keyring 2025.1 +debian-faq 12.2 +debianutils 5.23.2 +dhcpcd-base 1:10.1.0-11+deb13u4 +diffutils 1:3.10-4 +distro-info-data 0.66+deb13u2 +dmidecode 3.6-2 +doc-debian 11.3+nmu1 +docker-buildx-plugin 0.37.1-1~debian.13~trixie +docker-ce 5:29.8.2-1~debian.13~trixie +docker-ce-cli 5:29.8.2-1~debian.13~trixie +docker-ce-rootless-extras 5:29.8.2-1~debian.13~trixie +docker-compose-plugin 5.6.0-1~debian.13~trixie +dpkg 1.22.22 +e2fsprogs 1.47.2-3+b12 +fdisk 2.41.5-0+deb13u1 +file 1:5.46-5 +findutils 4.10.0-3 +gcc-14-base 14.2.0-19 +gettext-base 0.23.1-2 +git 1:2.47.3-0+deb13u1 +git-man 1:2.47.3-0+deb13u1 +grep 3.11-4 +groff-base 1.23.0-9 +gzip 1.13-1+deb13u1 +hostname 3.25 +ifupdown 0.8.44+deb13u1 +ifupdown2 3.0.0-1.3 +inetutils-telnet 2:2.6-3+deb13u3 +init 1.69~deb13u1 +init-system-helpers 1.69~deb13u1 +iproute2 6.15.0-1 +iptables 1.8.11-2 +iputils-ping 3:20240905-3 +isc-dhcp-client 4.4.3-P1-8 +kmod 34.2-2 +krb5-locales 1.21.3-5+deb13u1 +less 668-1 +libacl1 2.3.2-2+b1 +libapparmor1 4.1.0-1 +libapt-pkg7.0 3.0.3 +libattr1 1:2.5.2-3 +libaudit-common 1:4.0.2-2+deb13u1 +libaudit1 1:4.0.2-2+deb13u1 +libblkid1 2.41.5-0+deb13u1 +libbpf1 1:1.5.0-3 +libbrotli1 1.1.0-2+b7 +libbsd0 0.12.2-2 +libbz2-1.0 1.0.8-6 +libc-bin 2.41-12+deb13u4 +libc-l10n 2.41-12+deb13u4 +libc6 2.41-12+deb13u4 +libcap-ng0 0.8.5-4+b1 +libcap2 1:2.75-10+deb13u1+b3 +libcap2-bin 1:2.75-10+deb13u1+b3 +libcbor0.10 0.10.2-2 +libcom-err2 1.47.2-3+b12 +libcrypt1 1:4.4.38-1 +libcurl3t64-gnutls 8.14.1-2+deb13u5 +libcurl4t64 8.14.1-2+deb13u5 +libdb5.3t64 5.3.28+dfsg2-9 +libdbus-1-3 1.16.2-2 +libdebconfclient0 0.280 +libedit2 3.1-20250104-1 +libelf1t64 0.192-4 +liberror-perl 0.17030-1 +libexpat1 2.8.3-1~deb13u1 +libext2fs2t64 1.47.2-3+b12 +libfdisk1 2.41.5-0+deb13u1 +libffi8 3.4.8-2 +libfido2-1 1.15.0-1+b1 +libfstrm0 0.6.1-1+b3 +libgcc-s1 14.2.0-19 +libgdbm-compat4t64 1.24-2 +libgdbm6t64 1.24-2 +libgmp10 2:6.3.0+dfsg-3 +libgnutls30t64 3.8.9-3+deb13u4 +libgssapi-krb5-2 1.21.3-5+deb13u1 +libhogweed6t64 3.10.1-1 +libicu76 76.1-4 +libidn2-0 2.3.8-2 +libip4tc2 1.8.11-2 +libip6tc2 1.8.11-2 +libjansson4 2.14-2+b3 +libjemalloc2 5.3.0-3 +libjson-c5 0.18+ds-1 +libk5crypto3 1.21.3-5+deb13u1 +libkeyutils1 1.6.3-6 +libkmod2 34.2-2 +libkrb5-3 1.21.3-5+deb13u1 +libkrb5support0 1.21.3-5+deb13u1 +liblastlog2-2 2.41.5-0+deb13u1 +libldap-common 2.6.10+dfsg-1 +libldap2 2.6.10+dfsg-1 +liblmdb0 0.9.31-1+b2 +liblocale-gettext-perl 1.07-7+b1 +liblockfile-bin 1.17-2 +liblsof0 4.99.4+dfsg-2 +liblz4-1 1.10.0-4 +liblzma5 5.8.1-1+deb13u1 +libmagic-mgc 1:5.46-5 +libmagic1t64 1:5.46-5 +libmaxminddb0 1.12.2-1 +libmd0 1.1.0-2+b1 +libmnl0 1.0.5-3 +libmount1 2.41.5-0+deb13u1 +libncursesw6 6.5+20250216-2 +libnetfilter-conntrack3 1.1.0-1 +libnettle8t64 3.10.1-1 +libnewt0.52 0.52.25-1 +libnfnetlink0 1.0.2-3 +libnftables1 1.1.3-1 +libnftnl11 1.2.9-1 +libnghttp2-14 1.64.0-1.1+deb13u1 +libnghttp3-9 1.8.0-1 +libngtcp2-16 1.11.0-1+deb13u1 +libngtcp2-crypto-gnutls8 1.11.0-1+deb13u1 +libnsl2 1.3.0-3+b3 +libnss-systemd 257.13-1~deb13u1 +libp11-kit0 0.25.5-3 +libpam-modules 1.7.0-5 +libpam-modules-bin 1.7.0-5 +libpam-runtime 1.7.0-5 +libpam-systemd 257.13-1~deb13u1 +libpam-wtmpdb 0.73.0-3+deb13u1 +libpam0g 1.7.0-5 +libpcre2-8-0 10.46-1~deb13u3 +libperl5.40 5.40.1-6+deb13u1 +libpipeline1 1.5.8-1 +libpopt0 1.19+dfsg-2 +libproc2-0 2:4.0.4-9 +libprotobuf-c1 1.5.1-1 +libpsl5t64 0.21.2-1.1+b1 +libpython3-stdlib 3.13.5-1 +libpython3.13-minimal 3.13.5-2+deb13u5 +libpython3.13-stdlib 3.13.5-2+deb13u5 +libreadline8t64 8.2-6 +librtmp1 2.4+20151223.gitfa8646d.1-2+b5 +libsasl2-2 2.1.28+dfsg1-9 +libsasl2-modules-db 2.1.28+dfsg1-9 +libseccomp2 2.6.0-2 +libselinux1 3.8.1-1 +libsemanage-common 3.8.1-1 +libsemanage2 3.8.1-1 +libsepol2 3.8.1-1 +libslang2 2.3.3-5+b2 +libsmartcols1 2.41.5-0+deb13u1 +libsqlite3-0 3.46.1-7+deb13u2 +libss2 1.47.2-3+b12 +libssh2-1t64 1.11.1-1+deb13u2 +libssl3t64 3.5.7-1~deb13u3 +libstdc++6 14.2.0-19 +libsystemd-shared 257.13-1~deb13u1 +libsystemd0 257.13-1~deb13u1 +libtasn1-6 4.20.0-2+deb13u1 +libtext-charwidth-perl 0.04-11+b4 +libtext-iconv-perl 1.7-8+b4 +libtext-wrapi18n-perl 0.06-10 +libtinfo6 6.5+20250216-2 +libtirpc-common 1.3.6+ds-1 +libtirpc3t64 1.3.6+ds-1 +libtlsrpt0 0.5.0rc1-2 +libuchardet0 0.0.8-1+b2 +libudev1 257.13-1~deb13u1 +libunistring5 1.3-2 +liburcu8t64 0.15.2-2 +libuuid1 2.41.5-0+deb13u1 +libuv1t64 1.50.0-2 +libwrap0 7.6.q-36 +libwtmpdb0 0.73.0-3+deb13u1 +libxml2 2.12.7+dfsg+really2.9.14-2.1+deb13u3 +libxtables12 1.8.11-2 +libxxhash0 0.8.3-2 +libzstd1 1.5.7+dfsg-1 +linux-sysctl-defaults 4.12.1 +locales 2.41-12+deb13u4 +login 1:4.16.0-2+really2.41.5-0+deb13u1 +login.defs 1:4.17.4-2 +logrotate 3.22.0-1 +logsave 1.47.2-3+b12 +lsof 4.99.4+dfsg-2 +man-db 2.13.1-1 +manpages 6.9.1-1 +mawk 1.3.4.20250131-1 +media-types 13.0.0 +mount 2.41.5-0+deb13u1 +nano 8.4-1+deb13u1 +ncurses-base 6.5+20250216-2 +ncurses-bin 6.5+20250216-2 +ncurses-term 6.5+20250216-2 +netbase 6.5 +netcat-traditional 1.10-50 +nftables 1.1.3-1 +openssh-client 1:10.0p1-7+deb13u4 +openssh-server 1:10.0p1-7+deb13u4 +openssh-sftp-server 1:10.0p1-7+deb13u4 +openssl 3.5.7-1~deb13u3 +openssl-provider-legacy 3.5.7-1~deb13u3 +passwd 1:4.17.4-2 +patch 2.8-2 +perl 5.40.1-6+deb13u1 +perl-base 5.40.1-6+deb13u1 +perl-modules-5.40 5.40.1-6+deb13u1 +pigz 2.8-1 +postfix 3.10.13-0+deb13u1 +procps 2:4.0.4-9 +python-apt-common 3.0.0 +python3 3.13.5-1 +python3-apt 3.0.0 +python3-certifi 2025.1.31+ds-1 +python3-chardet 5.2.0+dfsg-2 +python3-charset-normalizer 3.4.2-1 +python3-debconf 1.5.91 +python3-debian 1.0.1 +python3-debianbts 4.1.1 +python3-idna 3.10-1+deb13u1 +python3-minimal 3.13.5-1 +python3-reportbug 13.2.0 +python3-requests 2.32.3+dfsg-5+deb13u1 +python3-urllib3 2.3.0-3+deb13u2 +python3.13 3.13.5-2+deb13u5 +python3.13-minimal 3.13.5-2+deb13u5 +readline-common 8.2-6 +reportbug 13.2.0 +runit-helper 2.16.4 +sed 4.9-2+deb13u1 +sensible-utils 0.0.25 +sqv 1.3.0-3+b2 +ssh 1:10.0p1-7+deb13u4 +systemd 257.13-1~deb13u1 +systemd-sysv 257.13-1~deb13u1 +systemd-timesyncd 257.13-1~deb13u1 +sysvinit-utils 3.14-4 +tar 1.35+dfsg-3.1 +traceroute 1:2.1.6-1 +tzdata 2026c-0+deb13u1 +ucf 3.0052 +udev 257.13-1~deb13u1 +util-linux 2.41.5-0+deb13u1 +util-linux-extra 2.41.5-0+deb13u1 +vim-common 2:9.1.1230-2 +vim-tiny 2:9.1.1230-2 +wamerican 2020.12.07-4 +wget 1.25.0-2 +whiptail 0.52.25-1 +wtmpdb 0.73.0-3+deb13u1 +xz-utils 5.8.1-1+deb13u1 +zlib1g 1:1.3.dfsg+really1.3.1-1+b1 diff --git a/documentation/audits/night-fixes-2026-10-05/partE/raw/e-pass.log b/documentation/audits/night-fixes-2026-10-05/partE/raw/e-pass.log new file mode 100644 index 00000000..fce346af --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/raw/e-pass.log @@ -0,0 +1,2 @@ +=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub === +time=2026-10-05T08:13:34.060+02:00 level=INFO msg="osupdate: START" run=20261005T061334Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261005T061334Z diff --git a/documentation/audits/night-fixes-2026-10-05/partE/rollback1.txt b/documentation/audits/night-fixes-2026-10-05/partE/rollback1.txt new file mode 100644 index 00000000..bd0fd1c2 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partE/rollback1.txt @@ -0,0 +1,31 @@ +### 05:44:03 before +bind9-dnsutils 1:9.20.29-1~deb13u1 +bind9-host 1:9.20.29-1~deb13u1 +bind9-libs:amd64 1:9.20.29-1~deb13u1 +libpcre2-8-0:amd64 10.46-1~deb13u3 +libpython3.13-minimal:amd64 3.13.5-2+deb13u5 +libpython3.13-stdlib:amd64 3.13.5-2+deb13u5 +libssh2-1t64:amd64 1.11.1-1+deb13u2 +libssl3t64:amd64 3.5.7-1~deb13u3 +libxml2:amd64 2.12.7+dfsg+really2.9.14-2.1+deb13u3 +openssl 3.5.7-1~deb13u3 +openssl-provider-legacy 3.5.7-1~deb13u3 +python3.13 3.13.5-2+deb13u5 +python3.13-minimal 3.13.5-2+deb13u5 +279 +### 05:44:05 rollback 1 (13 packages, simulated clean before) +0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded. +audit rc=0 +bind9-dnsutils 1:9.20.26-1~deb13u1 +bind9-host 1:9.20.26-1~deb13u1 +bind9-libs:amd64 1:9.20.26-1~deb13u1 +libpcre2-8-0:amd64 10.46-1~deb13u2 +libpython3.13-minimal:amd64 3.13.5-2+deb13u4 +libpython3.13-stdlib:amd64 3.13.5-2+deb13u4 +libssh2-1t64:amd64 1.11.1-1+deb13u1 +libssl3t64:amd64 3.5.7-1~deb13u2 +libxml2:amd64 2.12.7+dfsg+really2.9.14-2.1+deb13u1 +openssl 3.5.7-1~deb13u2 +openssl-provider-legacy 3.5.7-1~deb13u2 +python3.13 3.13.5-2+deb13u4 +python3.13-minimal 3.13.5-2+deb13u4 diff --git a/documentation/audits/night-fixes-2026-10-05/partF/FINDINGS.md b/documentation/audits/night-fixes-2026-10-05/partF/FINDINGS.md new file mode 100644 index 00000000..8bf6771f --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/partF/FINDINGS.md @@ -0,0 +1,73 @@ +# Part F — a box that is OFF every night (Tester 2) — read-only spike, 2026-10-05 + +Collected by a read-only helper session (hub DB copy at ~05:11 UTC, non-secret columns only, copy shredded; hub +log; source at felhom.eu 7221ee5c, controller 69e9145, agent c8d12f1). Two claims re-checked by the main session +against source: the 05:00 deadline check skips `down` customers (`hub/internal/monitor/deadline.go` ~360), and the +controller's daily scheduler always picks the NEXT future time (`controller/internal/scheduler/scheduler.go` +`nextDailyRun`, ~389-408). Hub DB times are UTC; the hub log prints Budapest time. + +## Q1 — Tester 2's record (customer `Tester-2`, host `Tester-2-be8404`) + +Customer created 2026-09-30 (language en); host created 2026-10-04 16:07 UTC (agent 0.142.0, controller 0.292.0); +no apps installed. Online twice on 2026-10-04: ~15:54–17:13 UTC, and 18:03:42–18:06:24 UTC (last controller report). +Nothing since. One evening of data: no daytime pattern can be established. + +| Leg | Result | Evidence | +|---|---|---| +| Whole-guest backup, local | ran once | 16:21:10 UTC, success | +| Whole-guest backup, PBS | ran once | 16:31:13 UTC, verify ok | +| OS update | ran once | `os_reports` 40/41, trigger `night`, 16:24/16:25 UTC, 49 + 106 packages, healthy | +| DB dump, tier-2 copy | never | — | +| Off-site copy | never | escrow `pending` | +| App updates | none | no apps | +| Restore-test | never | `restore_tests: []` in every report | + +Events: 17:58 `host_stale`, 18:01 `node_stale`, 18:05–06 recovered; 18:51 stale again; 19:36 `node_down` + +`host_down` (error) — the household was mailed "Your server cannot be reached." at 19:36:42 UTC. Hub log +2026-10-05 05:00 Budapest: "Deadline check: … 0 backup missed … 1 skipped (down)". + +**What fires if it stays off every night:** +- every evening `node_stale`/`host_stale` after 45 min; every night `node_down`/`host_down` after 90 min — the + household gets a `node_down` mail each night and a "reachable again" mail each morning (the 6 h cooldown does not + stop a daily repeat); +- **never** `expected_backup_missed` / `expected_dbdump_missed` while it is down at 05:00 (the deadline check skips + down customers); +- `restore_test_stale` ~2026-10-11 16:13 UTC (local tier, 7 d) and ~2026-10-16 (PBS, 12 d) — not skipped for down boxes; +- `os_update_stale` 2026-10-11 16:24 UTC unless a primary backup happens before; +- `offsite_stale` (48 h) once escrow is completed — the off-site leg runs only at night. + +**What the household sees:** the backups page shows "Esedékes / Due" or "Naprakész / Up to date" and "No successful +backup yet" per tier. No text says the server must stay on at night or that a night was missed because it was off. + +## Q2 — does anything catch up when the box comes back? + +- **DB dump, tier-2, off-site, the chained app-update leg, the controller's 04:30 self-update:** NO. The daily + scheduler always picks the next future time; `LastRun` is in memory only and never consulted. (A hub floor still + reaches the box by day, after any report.) +- **Whole-guest backup:** PARTIAL. The controller asks every 5 min; due = newest archive ≥ 24 h old (read from + storage, survives a reboot); runs only in the window [W+2h, W+6h) (04:30–08:30 for W 02:30), OR by the safety valve + (never backed up, or last > 48 h). A box on only in the afternoon/evening gets about one every 2 days. +- **OS leg:** 90 s after any successful primary backup, ≥ 20 h apart — so it follows the valve, by day, under the + `night` label. +- **Restore-test:** checks every 6 h, the timer restarts at each agent start, no check at start — a box never on for + 6 h in a row never runs one. +- Not verified: a SUSPENDED (not powered-off) laptop would probably run the daily jobs late rather than skip them + (reasoned from Go timers, not tested). + +## Q3 — architecture + +**No architecture document covers a box that is not always on.** Partial mentions: `11-os-updates.md` ("a box off for +a weekend does not alarm"); `07-backup-architecture.md` (the night chain and the [W+2h, W+6h) gate — no safety valve, +no catch-up); `08-alarm-ladder.md` (45/90 min); `where-felhom-stands.yaml` `fail.expected-downtime` (missing, R-285). +The intent "a box powered on only outside its window never starves" exists only as controller code comments +(`quiesce.go`). + +## Q4 — register + +R-285 (open, P4: no notion of expected downtime), R-30 (open, P3), R-195 / R-321 (closed: skips keyed off the +wrong fact), R-86 / R-100 (closed). **No row covers a box that is off every night.** + +## Not determined + +The daytime online pattern; Tester 2's actual W (on the box, which is off; 02:30 assumed); the type of signed job 53 +(queued 18:35 UTC, after the box went off — the 2026-10-04 agent update, per the night's records). diff --git a/documentation/audits/night-fixes-2026-10-05/system-page-end.txt b/documentation/audits/night-fixes-2026-10-05/system-page-end.txt new file mode 100644 index 00000000..6fd3e9e9 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/system-page-end.txt @@ -0,0 +1,6 @@ +demo-felhom-8363b5 Demo Ügyfél ring 0 → normal updates ON switch off running 9.2.2 7.0.2-6-pve 7.0.2-6-pve 13.7 ring0-20261005T055135Z 80 0 none no 10 s no 0 armed 0.144.1 13.7 ring0-20261005T055135Z 0 7 29.8.2 2.3.6-1~debian.13~trixie on ring0-20261005T055135Z 29 min ago · nothing · 12 s demo-hp-bb76ea Demo HP partial: no runni + +demo-hp-bb76ea Demo HP partial: no running customer guest ring 0 → normal updates ON switch off unknown 9.2.2 unknown unknown unknown ring0-20261005T061858Z 78 0 unknown no unknown no unknown not installed 0.144.1 unknown ring0-20261005T061858Z 0 7 unknown unknown unknown ring0-20261005T061858Z 3 min ago · nothing · 19 s tester- + +tester-1-d70be4 Tester 1 ring 1 → demo updates ON switch off running 9.2.2 7.0.2-6-pve 7.0.2-6-pve 13.5 — 185 106 none no 10 s no 0 armed 0.144.1 13.6 — 49 0 29.8.2 2.3.6-1~debian.13~trixie on — 10 h ago · nothing · 16 s Amber: worth a look. Red: an operator alarm fires (`08` §6.3). "unknown": the box could not read the value — + diff --git a/documentation/audits/night-fixes-2026-10-05/tester2-reads.txt b/documentation/audits/night-fixes-2026-10-05/tester2-reads.txt new file mode 100644 index 00000000..caf39863 --- /dev/null +++ b/documentation/audits/night-fixes-2026-10-05/tester2-reads.txt @@ -0,0 +1,2 @@ +2026-10-05T05:52:25Z ('Tester-2-be8404', '0.142.0', '2026-10-04 18:05:48') +2026-10-05T06:24:47Z ('Tester-2-be8404', '0.142.0', '2026-10-04 18:05:48') diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index e5ba16d7..fb0b2a8e 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,19 @@ --- +## 2026-10-05 (day) — the night's fixes: off-site clean-up guard, first-install image race, R8 download, a killed pass's report (controller v0.294.0, agent v0.144.0 + v0.144.1, golden 0.294.0; rulings 100–103, CC decisions 104–108) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-867** | **The off-site clean-up guard refused honest 7-day retention and mailed an error every window.** Controller v0.294.0: the guard's "young" line is keep-daily CALENDAR days, built from the same constants as the policy (decision 104); every other refusal kept. Tests run restic 0.14.0's policy itself (proven identical to the binary over 92 snapshots), 5 red-proofs. Live by hand: demo-felhom window 5 16 → 14, demo-hp window 6 145 → 127 — exactly the predicted snapshots; hub rows `pruned`, no event, no mail, key files clean. **Reasoning kept: a line that can sit inside the keep window is a line that refuses the honest case — derive it from the policy, never pick an age.** Full text: `git show 7221ee5c:documentation/backlog/OPEN-ITEMS.md`. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partA/` | +| **R-95** | **The box could delete its own off-site history.** Append-only key since 2026-10-03 (decisions 68–69); the last open item — a clean-up window that actually removes snapshots — was observed 2026-10-05 on both demo boxes (R-867's live windows, opened by the operator's one-shot grant; the dated check's four conditions all hold). The unattended weekly window is the same code path and is due ~2026-10-11/12; it is not separately re-checked (the DUE-CHECKS entry is removed with this row). Residual: R-822 (an add-only attacker steering older keeps). Full text: `git show 7221ee5c:documentation/backlog/OPEN-ITEMS.md`. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partA/`; `audits/offsite-lock-build-2026-10-03/` | +| **R-863** | **A new box's first app install could fail: the one-time image clean-up deleted the image compose had just pulled.** Controller v0.294.0: every compose command that can pull holds a shared lock (`dockerexec.BeginImageWork`); a clean-up pass takes it exclusively without waiting and otherwise does not run; the one-time pass is retried every 2 min and writes its marker only after a pass that ran (decision 105). Live on 9202: the clean-up fired at minute 3 (05:31:36 UTC) inside BookStack's 35.5 s install, skipped itself, BookStack installed first time; the retry at 05:33:36 ran and kept everything; teardown through the product. The update path was already guarded (`Updating`); restore/undo were exposed in principle and now hold the same lock. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partB/` | +| **R-864** | **A failed compose logged the head of stderr (pull progress) and cut the reason.** Controller v0.294.0 `tailStr` (rune-safe), pinned with the night's real first line. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partB/red-proofs.txt` | +| **R-869** | **The move-aside log line printed an empty destination.** Controller v0.294.0: assigned before the log line; test + red-proof. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partD/r869-red-proof.txt` | +| **R-865** | **R8 measured every download as 0 B (`--print-uris` with `-s` prints no URIs).** Agent v0.144.0: no `-s`; the fake answers like real apt (verbatim 9202 output), 2 tests, red-proof. Live: the installed wrapper's `download_bytes` on demo-hp read **12 802 456 B** for 13 pending upgrades (0 before), nothing installed. A live R8 REFUSAL line was not produced: it needs < 500 MB free on demo-hp's guest, i.e. 28.5 GB written into a thin pool with 18.7 GB free — it would have stopped every guest. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partC/` | +| **R-866** | **The debug OS pass could not run with the hub away.** Agent v0.144.0: the daemon saves the hub's block; the selftest falls back to it and says `block=SAVED(