night fixes 2026-10-05: R-867/R-95/R-863..R-869 closed, R-870..R-876 opened (R-876 P2: repair misses dpkg's journal); Part E crash record; Part F spike; golden 0.294.0; rulings 100-103, CC decisions 104-108; STATUS
gates / gates (push) Successful in 32s
gates / gates (push) Successful in 32s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
7221ee5cd6
commit
5fd2656021
55 files changed
+4856
-23
No files matched your search
+11
@@ -16,6 +16,17 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-05 (day) — the night's fixes (controller v0.294.0, agent v0.144.0 + v0.144.1, golden 0.294.0 vouched with agent
|
||||
> 0.144.1, min_agent 0.131.0).** Rulings 100–103 (`09` §3: Tester 1's CF tokens NOT rotated → R-870; A1 by day on demo-hp
|
||||
> only, with the operator's go; Tester 2 is a laptop off at night; re-sign Tester 2 only if online — it never was).
|
||||
> CC decisions 104–108, *operator may reverse*: guard "young" = keep-daily CALENDAR days (`offbox_window.go`, built from
|
||||
> `keepDaily`/`keepWeekly`/`keepMonthly`); `dockerexec.BeginImageWork`/`TryImageCleanup` for every pulling compose verb;
|
||||
> the wrapper's kept report copy (`report-<run>-<layer>-apply.json`) + `Leg.SendUnsentLoop` (5 min) + `pass.lock`; the
|
||||
> selftest's saved block (`os-update-block.json`); a second agent release. Closed R-95, R-863..R-869. Opened R-870..R-876
|
||||
> (R-871 the operator's decision: a box off at night; **R-876 P2: after a mid-update crash the wrapper's repair misses
|
||||
> dpkg's update journal — fix next**). Floors: demo-hp, demo-felhom, tester-1 at 0.294.0 (MinAgent 0.131.0); global
|
||||
> unchanged. Report: `REPORT-night-fixes-2026-10-05.md`.
|
||||
|
||||
> **2026-10-04 (night) — R-840 BUILT (agent v0.143.0, hub v0.133.0, installer 1.31.0), R-859 + R-860 FIXED (hub v0.133.0,
|
||||
> controller v0.293.0), golden 0.293.0 vouched, drill-r50 removed.** The config bundle (`11` §5.4.2): one table of 22
|
||||
> root paths in `felhom-os-apply`; signed `agent_config_update`; the trust root never a bundle path; installer installs the
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
# REPORT — the night's fixes, the power cut by day, a box that is off at night (2026-10-05)
|
||||
|
||||
Brief: "fix what the 2026-10-04 night found …" (operator, 2026-10-05). Evidence: `documentation/audits/night-fixes-2026-10-05/`.
|
||||
Architecture read before the claims: `07-backup-architecture.md` (§6.1, the Tier-3 row), `11-os-updates.md` (§5.4.1, §8),
|
||||
`09` §3 (decisions 53, 68–74, 81, 86), `audits/offsite-append-only-2026-10-03/DESIGN.md`.
|
||||
|
||||
## 1. The Part table
|
||||
|
||||
| Part | State | Note |
|
||||
|---|---|---|
|
||||
| §1 rulings recorded first | **done** | `09` decisions 100–103; R-870 (tokens, like R-831); Tester 2's page in `runbooks/target-selection.md` |
|
||||
| A — clean-up guard (R-867) | **done** | controller v0.294.0; tests run restic 0.14.0's policy (proven identical to the binary); 5 red-proofs; live windows on both demo boxes with counts; R-867 and R-95 closed (DUE-CHECKS entry removed) |
|
||||
| B — image clean-up race (R-863, R-864) | **done** | one lock for every pulling compose verb; the night's shape reproduced in a test and live on 9202 (fired at minute 3, skipped, install OK first time) |
|
||||
| C — R8 real download (R-865) | **changed** | fix + tests + red-proof done. Live: the INSTALLED wrapper's measure read 12 802 456 B on demo-hp. **No R8 refusal line was produced**: it needs < 500 MB free on demo-hp's guest = 28.5 GB written into a thin pool with 18.7 GB free — it would have stopped every guest |
|
||||
| D — R-868, R-869, R-866 | **done, with a second agent release** | R-868's first fix (v0.144.0) was measured NOT to work live (broken stderr pipe); v0.144.1 fixed it and the A5 shape then delivered one `applied` report. R-866 live with the hub blackholed. R-869 test + red-proof |
|
||||
| E — power cut mid-update (A1 by day) | **done** | operator's go; crash 06:13:55 UTC, back by itself in 37 s; **the next pass failed** → R-876 (P2); by-hand repair; demo-hp left with every package current |
|
||||
| F — a box off at night (spike) | **done** | read only; `partF/FINDINGS.md`; no architecture covers it → R-871; R-872..R-874; STATUS decision A/B |
|
||||
| G — releases, golden, records | **done** | controller v0.294.0; agent v0.144.0 + v0.144.1 (decision 108); golden 0.294.0 baked + vouched (gate OK); `07`, `11`, `00`, `09` updated |
|
||||
|
||||
## 2. Claims in the brief that turned out wrong (or only partly true)
|
||||
|
||||
1. **"The policy's constants can drive the guard's line"** — true, and done — but with a cost the brief did not name: a
|
||||
line derived from keep-daily (calendar days) can no longer catch a past-dated gap-fill that steers a 7–8-day-old
|
||||
keep, which the 8-day line caught while refusing every honest window. Past-dated fakes can never make the policy drop
|
||||
a snapshot inside the last keep-daily calendar days, so the line now catches only the skew-window shape. The rest is
|
||||
R-822's residual, bounded by the cap and the hub's count check (decision 104).
|
||||
2. **"The same race exists in the update and restore paths"** — the update path: NO, it was already guarded (no pass
|
||||
while any app is updating, since v0.284). The restore and undo paths: exposed in principle (they pull inside
|
||||
`compose up`), not measured; they now hold the same lock.
|
||||
3. **"Dropping `-s` downloads nothing"** — TRUE, measured on 9202 (archive cache 10 → 10 files, versions unchanged).
|
||||
4. **"A box off at W runs nothing when it comes back"** — mostly true: the database dumps, second copy, off-site copy and
|
||||
app updates never catch up; but the whole-guest backup DOES catch up (48 h safety valve, about every 2 days for an
|
||||
evening-only box), and the OS leg follows it by day under the `night` label.
|
||||
5. **"The night A5 showed the wrapper finished"** (R-868's premise, and v0.144.0's design) — the packages were installed,
|
||||
but the wrapper process itself died on its first log line after the agent's death. Found only by running it live.
|
||||
6. Part E: "the next pass must repair dpkg first and finish" — it does NOT (R-876).
|
||||
|
||||
## 3. What was proven, with numbers
|
||||
|
||||
- **Part A.** Tests: the measured shape (7 d + 5 s → allowed), 60 nights × 3 apps with two same-day manual runs across a
|
||||
month boundary (never refused, ≥ 45 removed), weekly windows, the skew-window fake (refused), the lab's 13 future fakes
|
||||
(refused). Red-proofs: 8-day line back → 3 fail; each refusal dropped → its test fails. Lab: restic 0.14.0 binary vs
|
||||
the test simulation over 92 snapshots: identical 72 removals. **Live:** demo-felhom 15 snapshots, predicted removals
|
||||
`92e49f62`, `343d57a5` → window 5 removed exactly those (16 → 14 incl. the run's new one); demo-hp 136 snapshots,
|
||||
predicted 18 → window 6 removed exactly those (145 → 127). Hub rows `pruned`, no event, no mail, key audit 0 findings.
|
||||
- **Part B.** Live on 9202: controller start 05:28:35, install 05:31:15, clean-up 05:31:36 "skipped — … pulling images
|
||||
now", BookStack deployed 05:31:54 (35.5 s), retry 05:33:36 ran (0 deleted). Teardown through the product verified.
|
||||
- **Part C.** `download_bytes = 12802456 B` from the installed wrapper on demo-hp (was 0).
|
||||
- **Part D.** R-866: `block=SAVED(2026-10-05T05:23:26Z; hub unreachable: … connect: invalid argument)`. R-868 (v0.144.1):
|
||||
killed 06:04:00, wrapper `DONE upgraded=13`, kept copy, hub report id 63 `applied` (13) at 06:09:05, once.
|
||||
- **Part E.** See `11` §8.4: back in 37 s; apps healthy within 4 min; household saw one timeline line; one operator mail
|
||||
`os_update_failed` (true); next pass FAILED (R-876); after `dpkg --configure -a` the pass installed 12; the guest's
|
||||
package list equals the pre-test one (279 lines). The first crash attempt did not fire (my watcher's pattern missed
|
||||
apt's dpkg call; the pass installed normally) — rolled back again and repeated.
|
||||
- **Part F.** `audits/night-fixes-2026-10-05/partF/FINDINGS.md`; two claims re-checked in source by me.
|
||||
|
||||
## 4. Rows
|
||||
|
||||
Register before **341**, after **340**. Closed (8): R-95, R-863, R-864, R-865, R-866, R-867, R-868, R-869. Opened (7):
|
||||
R-870 (tokens, operator), R-871 (a box off at night — the operator's decision), R-872 (P2, no missed-backup alarm for a
|
||||
box down at 05:00), R-873 (nightly "cannot be reached" mail), R-874 (restore-test never runs on short sessions),
|
||||
R-875 (P4, kept-report reason text), **R-876 (P2, the repair misses dpkg's update journal)**. STATUS updated.
|
||||
`unproven.py --summary`: unchanged — 55 claims, 35 not walked (no number moved).
|
||||
|
||||
## 5. Teardown, three layers
|
||||
|
||||
- **Machines:** 9202: BookStack removed through the product (no container, stack dir, volume); controller 0.294.0 (set by
|
||||
hand, allowed there). demo-hp 9201: every package current, list identical to before; my scripts and logs in `/root`
|
||||
removed after copying. The bake VM: CT 9100 destroyed, token shredded, reverted to `virgin`, qemu gone.
|
||||
- **Hosts:** the blackhole routes on felhom-pve removed (in the same script). Nothing provisioned.
|
||||
- **Hub:** two one-shot clean-up grants (consumed); floors for demo-hp, demo-felhom, tester-1 → 0.294.0; artifacts
|
||||
vouched (agent 0.144.1, golden 0.294.0, min_agent 0.131.0); 12 signed jobs (3× agent 0.144.0, 3× bundle 0.144.0,
|
||||
3× agent 0.144.1, 3× bundle 0.144.1). Tester 2: read only, nothing sent (offline all session).
|
||||
- Scratch secrets (hub password, hub key, controller password, deploy secrets) shredded at the end.
|
||||
|
||||
## 6. Notes
|
||||
|
||||
- The hub pod log prints a customer's e-mail address in "Customer email sent to …" lines (seen by the Part F helper; not
|
||||
recorded anywhere).
|
||||
- demo-hp's System page read "no running customer guest" / "unknown" for several minutes after the crash although the
|
||||
guest ran — R-853 (facts late after a boot), not a new defect.
|
||||
@@ -1,11 +1,62 @@
|
||||
# STATUS — what works, what's broken, what's next
|
||||
|
||||
**Ready for the first real tester (Tester-2): yes. Tester 2 is installed, but offline since 2026-10-04 20:06 local.**
|
||||
**Ready for the first real tester (Tester-2): yes. Tester 2 is a laptop that is switched off at night (your word,
|
||||
2026-10-05) — it was offline all session; nothing was sent to it.**
|
||||
|
||||
**Updated 2026-10-05 (morning, after the OS-update night): every box of ours healthy; the ruled undo proven (48 s);
|
||||
decision 78 proven on a fresh Tester 1 box; two P2 defects found (a new box's first app install can fail; the off-site
|
||||
clean-up refuses normal retention and mails an error weekly). One test (a power cut mid-update) not run — needs your
|
||||
permission. Morning note: `documentation/audits/night-2026-10-04/MORNING-NOTE.md`.**
|
||||
**Updated 2026-10-05 (day, the night's fixes): every box of ours healthy. Fixed and proven live: the off-site clean-up
|
||||
now deletes old copies (both demo boxes), a new box's first app install, the update's disk-space check, a killed
|
||||
update's lost report. The power cut in the middle of an update was tested on demo-hp with your go: the box came back by
|
||||
itself in 37 s, but the next update failed until I ran one command by hand — filed (R-876), fix next session.
|
||||
One decision for you below (a box that is off at night). Report: `REPORT-night-fixes-2026-10-05.md`.**
|
||||
|
||||
## Today (2026-10-05, day): the night's fixes, the power cut by day, a box that is off at night
|
||||
|
||||
**Decisions I took myself (you may reverse each — `09` decisions 104–108):**
|
||||
- The off-site clean-up's safety line is now "the last 7 calendar days" — the same number the clean-up keeps — not
|
||||
"8 days old". It can never block an honest clean-up again.
|
||||
- The image clean-up now waits while ANY app install, update, restore or undo is downloading — not only installs.
|
||||
- A killed update keeps its report on the box until the hub has it; the agent looks for such reports every 5 minutes.
|
||||
- **A second agent release today (0.144.1)**, against "one release per repo": the first fix for the lost report was
|
||||
proven NOT to work on demo-hp, and shipping it as it was would have been worse.
|
||||
|
||||
**What works now (proven live):**
|
||||
- **The weekly off-site clean-up really deletes old backups.** One clean-up each by hand: demo-felhom 16 → 14, demo-hp
|
||||
145 → 127 — exactly the backups I predicted. No error mail, the hub's count check quiet, the key files clean.
|
||||
This also closes R-95 (the box can no longer delete its own off-site history, and clean-up now works).
|
||||
- **A new box's first app install works the first time:** the image clean-up met an install at minute 3 on the scratch
|
||||
box, waited, and BookStack installed first try. A failed install now logs its real reason.
|
||||
- **The update's disk-space check counts the real download** (12.8 MB for 13 packages; it counted 0 before).
|
||||
- **A killed update still reports to the hub** (5 minutes later, once). The debug update runs with the hub away.
|
||||
- Controller 0.294.0 on demo-hp, demo-felhom and Tester 1 (floor per customer; Tester 2 not moved). Agent 0.144.1
|
||||
and its root files on all three. New-install image 0.294.0 baked and approved.
|
||||
|
||||
**Found today (filed, not fixed):**
|
||||
- **After a power cut in the middle of an update, every later update fails until someone runs one command on the box**
|
||||
(R-876, P2). The box itself comes back fine. Until the fix: `runbooks/crash-guard.md` has the command. I fix it next.
|
||||
- A box that is off at night (below): no catch-up, no missed-backup alarm (R-872), a "server cannot be reached" mail to
|
||||
the household every night (R-873), restore-tests never run (R-874).
|
||||
|
||||
**Needs you:**
|
||||
|
||||
1. **A box that is off every night (Tester 2) — what does the product promise?** Today such a box never gets its
|
||||
nightly database backups, second copy or off-site copy; the whole-box backup runs only about every 2 days; no
|
||||
alarm says so; the household is mailed "your server cannot be reached" every night. No design document covers it
|
||||
(R-871).
|
||||
- **A (my pick): a missed night runs once when the box comes back.** The database backups, the second copy and
|
||||
the off-site copy run a few minutes after the box is on again (they take seconds to minutes; whether
|
||||
any of them pauses an app is to be measured in the design — the whole-box backup, which does pause apps, already
|
||||
has its own catch-up). App and system updates still
|
||||
wait for a night. Costs: a design section and one controller release; the household may notice a busy disk for a
|
||||
few minutes after switching on.
|
||||
- **B: say plainly that the box must stay on at night.** The setup guide and the box's backup page say it; the
|
||||
missed-night alarm fires after 2 nights off. Costs: wording + one alarm; a laptop household gets an alarm it
|
||||
cannot fix except by changing habits.
|
||||
- **If you do nothing:** Tester 2 keeps having no database or off-site backup, nobody is told, and the household
|
||||
keeps getting the nightly "cannot be reached" mail. A restore-test alarm will fire around 2026-10-11.
|
||||
2. **Tester 2's one-time step** — unchanged from yesterday (below). If you wait, it keeps working; it just cannot get
|
||||
new root files.
|
||||
|
||||
**Recorded, not decisions:** Tester 1's Cloudflare tokens are NOT rotated (your ruling; R-870 has the steps).
|
||||
|
||||
## Today (2026-10-04, night): root files for installed boxes, test approvals, Docker self-repair
|
||||
|
||||
|
||||
@@ -232,7 +232,7 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis
|
||||
| **The hub reports LOSS OF VISIBILITY into either off-site store (not just how full it is)** | hub **v0.106.0** (R-339) | **IMPLEMENTED — deliberately NOT proven-live** | Both box checkers count consecutive failed fetch windows and emit `pbsdr_box_unreachable` / `offsite_box_unreachable` (severity `warning`) past a default 3 windows (≈30–45 min), each with a paired `*_recovered` all-clear routed via `recoveredPairedDownTypes` — required because the recoveries are severity `info`, which `severityNotifies` drops. Scopes stay customer-less (`pbsdr-box` / `pool-box`) → operator channel only. Fill logic untouched: a degraded read still drives no band transition. Evidence: `internal/monitor/box_reachability_test.go` + the cross-package wiring test in `internal/notify/`, which asserts an actual operator mail rather than a map entry. **Filed BECAUSE of a measured gap**, not a hypothesis: the 2026-08-18 ep0 outage ran 9 h 37 m with the hub silent | **The gap that remains is R-340**, and it is not small: the ep0 read is the `usage` op, which rides the LOCAL API daemon — the daemon that incident explicitly cleared — so this check would have shown GREEN for that entire outage. It closes "ep0 is unreachable as a host"; it does not close what actually happened. **No live or constructed outage has exercised the emit path**, and one cannot be manufactured against ep0 (Tier 2, protected) |
|
||||
| Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | 07-13 closing bundle | |
|
||||
| Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | 07-13 | |
|
||||
| Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0 | **PARTIAL — the GUEST and HOST Debian fast lanes and the DOCKER engine slow lane are PROVEN-LIVE (2026-10-04), with the System page, the fleet view and the alarms; the KERNEL lane is MISSING** | Guest: `audits/os-guest-lane-2026-10-04/`. Host + fleet + alarms: `audits/os-host-lane-2026-10-04/`. Docker + System page + crash guard: `audits/os-docker-crash-2026-10-04/` — live-restore on with the same container ids on every box; Docker 29.8.2 on both demo boxes; operator-approved Docker release; a signed undo and a signed ring-1 step; a replay refused; the crash guard restarted demo-hp twice and kept it off the third time. Design `architecture/11-os-updates.md` §5.8, §5.9, §8 | **No automatic undo** (guest: last night's backup; host: by-hand runbook; Docker: a signed undo job); existing boxes get root-owned files by the signed config bundle since 2026-10-04 (R-840 CLOSED; a box from before agent 0.143.0 needs one by-hand bootstrap — Tester 2: R-862; `audits/r840-config-bundle-2026-10-04/`); test approvals now end with the test (R-859); the agent's sudoers is root-equivalent (R-861); the kernel lane (R-836); facts reach the hub late after a boot (R-853) |
|
||||
| Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0 | **PARTIAL — the GUEST and HOST Debian fast lanes and the DOCKER engine slow lane are PROVEN-LIVE (2026-10-04), with the System page, the fleet view and the alarms; the KERNEL lane is MISSING** | Guest: `audits/os-guest-lane-2026-10-04/`. Host + fleet + alarms: `audits/os-host-lane-2026-10-04/`. Docker + System page + crash guard: `audits/os-docker-crash-2026-10-04/` — live-restore on with the same container ids on every box; Docker 29.8.2 on both demo boxes; operator-approved Docker release; a signed undo and a signed ring-1 step; a replay refused; the crash guard restarted demo-hp twice and kept it off the third time. Design `architecture/11-os-updates.md` §5.8, §5.9, §8 | **No automatic undo** (guest: last night's backup; host: by-hand runbook; Docker: a signed undo job); existing boxes get root-owned files by the signed config bundle since 2026-10-04 (R-840 CLOSED; a box from before agent 0.143.0 needs one by-hand bootstrap — Tester 2: R-862; `audits/r840-config-bundle-2026-10-04/`); test approvals now end with the test (R-859); the agent's sudoers is root-equivalent (R-861); the kernel lane (R-836); facts reach the hub late after a boot (R-853). **2026-10-05 (agent v0.144.1):** R8 measures the real download (R-865); a killed pass still reports (R-868, live); the debug pass runs with the hub away (R-866, live); **a power cut mid-update was proven by day on demo-hp — the box came back by itself in 37 s, but the next pass fails until `dpkg --configure -a` is run by hand (R-876, P2, open)** — `audits/night-fixes-2026-10-05/` |
|
||||
| **An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps** | controller **v0.259.0** + hub **v0.119.0** + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE on 0.258.0 with one blocker; THE BLOCKER IS FIXED AND PROVEN, THE WALK IS NOT REPEATED** | `audits/DRILL-first-hour-en-0258-2026-09-20.md` — a fresh install 2026-09-20, one intervention (R-494), stop rule not reached. Then `audits/i18n-closing-2026-09-21/live/` — the three blockers fixed and each proven on a live box or in the operator's inbox: the claim page answers English through the real cookie path; the Backup page's tier names follow the language; and the setup mail carries **four plain-ASCII English words** where the drill's carried `képző-szkítia-ásatás`, one day apart in the same inbox. | **R-596, R-597 and R-598 are CLOSED.** What this row still does NOT claim: **the fixed journey has not been walked end to end by a stranger on a fresh install.** Three fixes proven at the endpoint are not an hour proven by a person, and this project's own rule is that fixes are not a journey (see the recovery-journey row). **Also not walked:** the recovery code (needs ep0), backup/restore/remove/power-cut (proven 2026-09-14), and the two Backup-page *warnings* themselves — guest 9201 is healthy and a healthy box renders none, so they are covered by handler render tests, not live. **Verdict: nothing known now stands between an English-speaking tester and their box — and that is a different sentence from "the walk passed".** |
|
||||
| **A deletion of a customer's off-site history is NOTICED within a day** | hub **v0.111.0** (R-431) | **IMPLEMENTED — not yet PROVEN-LIVE** | 09-01 | `hub/internal/monitor/offsite.go` — third signal beside FILL and STALENESS. **On the hub deliberately:** a detector on the box is one the deletion can silence. Alarms when the reported count falls by more than HALF and by at least 5, guarded by `StatsKnown` (R-331), the declared `State` (R-204) and run success (R-100). **Threshold reasoned, not invented:** over 12 898 reports every decrease lands on ZERO and predates `stats_known`; in the 380-report `stats_known` window there are none. **ACCEPTANCE: 9 009 real points replayed → ZERO alarms** (`offsite_r431_test.go`, fixture committed). **What PROVEN-LIVE would need and this does NOT have:** a real drop observed on a live box producing a real mail — the live firing done at ship time was driven through the hub's own path with synthetic counts, which is an end-to-end delivery proof, not a proof that a genuine deletion is caught. |
|
||||
|
||||
|
||||
@@ -1138,7 +1138,7 @@ crosses the line — **R-158**.
|
||||
| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
|
||||
| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** **`[BETA-DEFERRED]`** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) |
|
||||
| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** **`[BETA-DEFERRED]`** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) **`[FACT]` 2026-10-03 (decision 70): ep0's datastore now has a nightly copy on DooPlex** (PBS pull-sync `ep0-felhom-offsite`, `remove-vanished false`, weekly verify, failures mailed to the operator); first pull 201 s, 12 GB, 4 of 4 snapshots, matching ep0 namespace for namespace. Restore route: `runbooks/ep0-datastore-copy.md` — never walked. `audits/offsite-lock-build-2026-10-03/partF/`. **`[FACT]` 2026-10-04: the restore route from the DooPlex copy was WALKED** on demo-hp (scratch VMID): listed in 2 s, restored in 186 s (15 GB logical), data read by `pct mount`; the restored config carries `onboot: 1` and the host's real drive binds — strip them on a restore beside the original (R-834). The copy now keeps 8 weekly copies (decision 71). `runbooks/ep0-datastore-copy.md` route 1. |
|
||||
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recoveredLine truncated
|
||||
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recoveredLine truncated
|
||||
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
|
||||
| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** **`[BETA-DEFERRED]`** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
|
||||
| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** **`[BETA-DEFERRED]`** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D **`[FACT]` 2026-10-03:** the PBS (whole-guest) off-site copy now also exists OFF Hetzner — DooPlex pulls ep0's datastore nightly (decision 70). The restic tier is still Hetzner-only. |
|
||||
|
||||
@@ -741,6 +741,12 @@ its length, and both fixes cost something the household would notice — operato
|
||||
days from the box's request (R-823)**; the household (a recovery on the box) or the operator can cancel in that
|
||||
time; the hub deletes only `<repo>.orphaned-<…>`, never the live repository; every request, cancel and deletion is
|
||||
an operator event. *Operator brief 2026-10-04 (Part E).* Built hub v0.128.0 + controller v0.290.0.
|
||||
**Note 2026-10-05 (R-867, controller v0.294.0):** the clean-up windows that feed this deletion path now remove
|
||||
honest old copies: the guard's "young" line is keep-daily CALENDAR days (decision 104), not a fixed 8-day age that
|
||||
sat inside the keep window. Proven live by hand the same morning: demo-felhom window 5 (16 → 14, the 2 snapshots
|
||||
predicted), demo-hp window 6 (145 → 127, the 18 predicted); hub count check quiet, key files clean, no mail.
|
||||
The reviewer's own error is recorded: the 2026-10-04 brief set the 8-day line and said real deletion would start
|
||||
around 11 October; it would not have. `audits/night-fixes-2026-10-05/partA/`.
|
||||
|
||||
### 2026-10-04 (day) — three operator rulings (recorded before the work)
|
||||
|
||||
@@ -809,6 +815,49 @@ its length, and both fixes cost something the household would notice — operato
|
||||
1.30.0, agent 0.142.0 and the re-made golden; it lacks only agent 0.142.1's wrapper fix (R-858). *Operator ruling
|
||||
2026-10-04 ~18:49.*
|
||||
|
||||
### 2026-10-05 (06:49) — four operator rulings (recorded before the work; the night-fixes brief)
|
||||
|
||||
100. **Tester 1's Cloudflare tokens, shown in the 2026-10-04 night session's output, are NOT rotated** (option B) —
|
||||
filed like R-831: R-870 holds what was shown, where, and the rotation steps. *Operator ruling 2026-10-05.*
|
||||
101. **The power cut in the middle of an OS update (night A1) runs by day, on demo-hp only**, while the operator is
|
||||
awake to switch the box on. The crash command is allowed for demo-hp only; CC asks the operator before the crash.
|
||||
*Operator ruling 2026-10-05.*
|
||||
102. **Tester 2's box is a laptop that is switched off at night.** It is not broken; its absence every night is the
|
||||
input to the "box off at night" spike (R-871). Recorded on its page in `runbooks/target-selection.md`.
|
||||
*Operator ruling 2026-10-05.*
|
||||
103. **If Tester 2 is online during the session, CC re-signs its agent update to the vouched agent** (act 1 of
|
||||
decision 97) and reads it back. Nothing else on Tester 2; its one-time bundle step stays with the operator
|
||||
(R-862). *Operator ruling 2026-10-05.*
|
||||
|
||||
### 2026-10-05 (day) — decided by CC — operator may reverse (night-fixes brief)
|
||||
|
||||
104. **What "young" means in the off-site guard (R-867).** Options: (a) an age in hours derived from keep-daily
|
||||
(7 × 24 h minus a margin) — the margin is a guess and DST/clock shapes move it; (b) CALENDAR days, as restic
|
||||
itself buckets a snapshot's day, compared with keep-daily — exactly the policy's own boundary. **Chosen (b)**:
|
||||
it is the only line that provably never refuses the honest policy (the newest snapshot of any of the last N
|
||||
calendar days is always kept by keep-daily N). Cost, recorded: the line no longer catches past-dated gap-fills
|
||||
that steer a 7–8-day-old keep (the 8-day line did, while refusing every honest window) — that shape is R-822's
|
||||
residual, bounded by the cap and the hub's count check. Controller v0.294.0.
|
||||
105. **Which commands hold off the image clean-up (R-863).** Options: (a) only the deploy path (the measured case);
|
||||
(b) every compose command that can pull — `up`, `pull`, `create`, `run` — through one lock in `dockerexec`.
|
||||
**Chosen (b)**: a restore, an undo and the FileBrowser sync pull the same way, and one lock is one place to
|
||||
keep right. The one-time clean-up is retried every 2 minutes (at most 30 times) instead of waiting for the
|
||||
next start, and writes its marker only after a pass that ran. Controller v0.294.0.
|
||||
106. **Where a killed pass's report waits (R-868).** Options: (a) the agent writes a "pass started" record and
|
||||
reconstructs the outcome from the journal — brittle; (b) the ROOT wrapper writes its own report beside the plan
|
||||
before printing it, and the agent deletes it once sent. **Chosen (b)**, with the root write hardened (the
|
||||
directory opened O_NOFOLLOW and checked to be the agent's own, the file created O_EXCL|O_NOFOLLOW) and a
|
||||
cross-process pass lock so a sender never takes a running pass's copy. Agent v0.144.0.
|
||||
107. **The debug OS pass with the hub away (R-866).** Options: (a) the selftest falls back to the block the daemon
|
||||
saved, and says so; (b) a documented way to trigger the daemon's own leg. **Chosen (a).** The DAEMON does not
|
||||
load the saved block at its own start (one line why: that would change what a box does after a restart with
|
||||
the hub away — today it reports only, the safe direction — and no brief asked for it). Agent v0.144.0.
|
||||
108. **A second agent release in one session (v0.144.1), against "one release per repo".** Options: (a) keep
|
||||
v0.144.0 and leave R-868 open — its fix was MEASURED not to work live (the wrapper died on a broken stderr pipe
|
||||
before saving the report); (b) release v0.144.1 with the two live findings fixed and red-proofed. **Chosen (b)**:
|
||||
shipping a fix proven not to work is worse than a second release; precedent decision 86. Cost: a second signed
|
||||
agent update and bundle per box.
|
||||
|
||||
### 2026-10-04 (evening) — decided by CC unattended — operator may reverse (System page / Docker / crash-restart brief)
|
||||
|
||||
90. **How does a box tell a crash boot from a clean one?** Options: (a) `pstore` — measured on demo-hp: `efi_pstore` is on,
|
||||
|
||||
@@ -280,7 +280,7 @@ longer there, from `snapshot.debian.org/archive/<archive>/<snapshot>` with a tem
|
||||
| R5 | the simulation downgrades any package (the operator undo is a separate signed op, §5.6) |
|
||||
| R6 | the simulation installs a package that is neither installed nor in `allow_new` |
|
||||
| R7 | a listed version is not downloadable from the sources the installer set up or the named snapshot |
|
||||
| R8 | free space on `/` (or the guest's rootfs) is below 3× the download size, minimum 500 MB (edge case 8) |
|
||||
| R8 | free space on `/` (or the guest's rootfs) is below 3× the download size, minimum 500 MB (edge case 8). The download size is the `--print-uris` total WITHOUT `-s` (R-865, agent v0.144.0 — with `-s` it read 0 B) |
|
||||
| R9 | another apt/dpkg holds the lock, or the per-guest lane lock is held (a backup, a restore-test, C10) |
|
||||
| R10 | `layer` is `guest` and the vmid is not the box's own customer guest |
|
||||
| R11 | the plan names a package twice, or a version that is not a Debian version string |
|
||||
@@ -617,6 +617,28 @@ Evidence: `audits/os-host-lane-2026-10-04/` (parts A–G).
|
||||
two weekly golden cycles, both need a person anyway.
|
||||
- **The tunnel** (R-841): `running` / `not_running` / `unknown`; `tunnel_down` after two `not_running` reports.
|
||||
|
||||
### 8.4 The night's fixes as BUILT (2026-10-05) `[FACT]`
|
||||
|
||||
Agent v0.144.0 + v0.144.1 (wrapper and agent; `09` decisions 106–108). Evidence `audits/night-fixes-2026-10-05/`.
|
||||
|
||||
- **R8 measures the real download** (R-865): `--print-uris` without `-s`; the installed wrapper read 12 802 456 B for 13
|
||||
pending guest upgrades on demo-hp (it read 0 B before). `--print-uris` alone downloads nothing (measured on 9202).
|
||||
- **A killed pass still reports** (R-868): the wrapper writes `report-<run>-<layer>-apply.json` beside the plan before it
|
||||
prints the report, and it survives a dead reader (a killed agent closes the stderr pipe — v0.144.0 died there,
|
||||
measured); the agent sends a kept copy at start and every 5 minutes, then deletes it. Live: the A5 shape on demo-hp →
|
||||
ONE `applied` report (13 packages) reached the hub 5 minutes later. A pass lock (flock) keeps the sender off a running
|
||||
pass, also across the daemon and a selftest.
|
||||
- **The debug pass with the hub away** (R-866): the daemon saves the hub's block (`os-update-block.json`); the selftest
|
||||
uses it when the hub is unreachable and says `block=SAVED(…)` in its header. The daemon itself does not load the saved
|
||||
block at start (decision 107).
|
||||
- **A power cut in the middle of an update (night A1, run by day on demo-hp, decision 101):** crash at 06:13:55 UTC while
|
||||
dpkg ran; the crash guard brought the host back in 37 s (1 unclean boot, still armed); every app healthy again within
|
||||
4 minutes; dpkg `--audit` clean, the 13 packages split 1 new / 12 old, all `ii`. **The next pass FAILED** (`E: dpkg was
|
||||
interrupted, you must manually run 'dpkg --configure -a'`): the repair step runs only on a non-clean `--audit`, and a
|
||||
crash can leave only dpkg's update journal behind — **R-876 (P2, open)**. By hand: `dpkg --configure -a` in the guest
|
||||
(runbook `crash-guard.md`), then the next pass installed the 12. Operator mail `os_update_failed` (true); no
|
||||
household mail; the household's timeline showed "Controller elindult".
|
||||
|
||||
## 9. Where the rest lives
|
||||
|
||||
- The finding: **R-812** (`backlog/OPEN-ITEMS.md`). The intention: **R-808** (`backlog/ROADMAP.md`).
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
[
|
||||
{
|
||||
"id": "6ea854132efc6053ee5f7ca900a2db6f767dd3d89250efeb52fa30e319598ed1",
|
||||
"short_id": "6ea85413",
|
||||
"time": "2026-08-12T15:15:33.030477238Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"drill-retained-key-20260812"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "b4d7cb79502a6c3f772254bc79463ec726f722f185a1c10571c389516e31746d",
|
||||
"short_id": "b4d7cb79",
|
||||
"time": "2026-08-31T02:15:04.935400697Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "6ce44d9de433625eff342dc0234d526a11336fb9b813aaa49cd737682c07387a",
|
||||
"short_id": "6ce44d9d",
|
||||
"time": "2026-09-20T02:15:05.018256188Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "c7000d7de36008707347c3b6dec9c79166feeae2a517137daa2c47004221e7ef",
|
||||
"short_id": "c7000d7d",
|
||||
"time": "2026-09-27T02:15:05.066652132Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "8ccc027e486d4f2e1b5415e2e602e29e8acbf0001a1596ec08b00c43b148ea68",
|
||||
"short_id": "8ccc027e",
|
||||
"time": "2026-09-29T02:15:05.16680869Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "40826801b49bdb9a249d0d0e7576081d5c96ea2bc7d4503d9a1bce6bfb19cb60",
|
||||
"short_id": "40826801",
|
||||
"time": "2026-09-30T02:15:05.173694815Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "c3a3b13b9832ba3d98cc3dc047b01b376ced2c824506e8f06001d8a3a7c5660e",
|
||||
"short_id": "c3a3b13b",
|
||||
"time": "2026-10-01T02:15:05.016873278Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "b2606da6d28cb24ce8d5c0616ced769b1a7f7b3d0b2923fca23b4364262b7a31",
|
||||
"short_id": "b2606da6",
|
||||
"time": "2026-10-02T02:15:05.138940278Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "64129ff5dcb75ea1ff766605e1a0b4dc36e6a4441a0be9433fe2519e5ddd16e0",
|
||||
"short_id": "64129ff5",
|
||||
"time": "2026-10-03T02:15:05.108133602Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "2bb8fb60447e6387d374d98d196807dd9e32187695ddab5df85ff2a1ac856404",
|
||||
"short_id": "2bb8fb60",
|
||||
"time": "2026-10-03T15:05:39.219021268Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "5dd1f0b99e8ed22d7fdcc9005501e880ddd600341fe35bd6897174acdd943857",
|
||||
"short_id": "5dd1f0b9",
|
||||
"time": "2026-10-03T15:17:27.992813997Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "89df02373ec2ffbe4a6b02d622c30c6f5001b922fc692bb1e1b84dfb0b8d2971",
|
||||
"short_id": "89df0237",
|
||||
"time": "2026-10-04T02:15:03.833138622Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "4e7867af43f024c8f712417bce982c5c8085524b51e7f37040c665d0dc1146de",
|
||||
"short_id": "4e7867af",
|
||||
"time": "2026-10-05T02:15:03.941403075Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "47bad696fe89450b2f5db11bfbc3a958edf2796a60d851dd9d92585eb36d53fc",
|
||||
"short_id": "47bad696",
|
||||
"time": "2026-10-05T05:21:04.032874428Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,152 @@
|
||||
[
|
||||
{
|
||||
"id": "6ea854132efc6053ee5f7ca900a2db6f767dd3d89250efeb52fa30e319598ed1",
|
||||
"short_id": "6ea85413",
|
||||
"time": "2026-08-12T15:15:33.030477238Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"drill-retained-key-20260812"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "b4d7cb79502a6c3f772254bc79463ec726f722f185a1c10571c389516e31746d",
|
||||
"short_id": "b4d7cb79",
|
||||
"time": "2026-08-31T02:15:04.935400697Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "92e49f62c017e103cf7a1f73f8982a7faa39d77eb9c7fd129b3910cc5734a262",
|
||||
"short_id": "92e49f62",
|
||||
"time": "2026-09-13T02:15:05.107149713Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "6ce44d9de433625eff342dc0234d526a11336fb9b813aaa49cd737682c07387a",
|
||||
"short_id": "6ce44d9d",
|
||||
"time": "2026-09-20T02:15:05.018256188Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "c7000d7de36008707347c3b6dec9c79166feeae2a517137daa2c47004221e7ef",
|
||||
"short_id": "c7000d7d",
|
||||
"time": "2026-09-27T02:15:05.066652132Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "343d57a5614f88b80a18b129b52753740c8391581073f406fe9878bf40716381",
|
||||
"short_id": "343d57a5",
|
||||
"time": "2026-09-28T02:15:05.279260454Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "8ccc027e486d4f2e1b5415e2e602e29e8acbf0001a1596ec08b00c43b148ea68",
|
||||
"short_id": "8ccc027e",
|
||||
"time": "2026-09-29T02:15:05.16680869Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "40826801b49bdb9a249d0d0e7576081d5c96ea2bc7d4503d9a1bce6bfb19cb60",
|
||||
"short_id": "40826801",
|
||||
"time": "2026-09-30T02:15:05.173694815Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "c3a3b13b9832ba3d98cc3dc047b01b376ced2c824506e8f06001d8a3a7c5660e",
|
||||
"short_id": "c3a3b13b",
|
||||
"time": "2026-10-01T02:15:05.016873278Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "b2606da6d28cb24ce8d5c0616ced769b1a7f7b3d0b2923fca23b4364262b7a31",
|
||||
"short_id": "b2606da6",
|
||||
"time": "2026-10-02T02:15:05.138940278Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "64129ff5dcb75ea1ff766605e1a0b4dc36e6a4441a0be9433fe2519e5ddd16e0",
|
||||
"short_id": "64129ff5",
|
||||
"time": "2026-10-03T02:15:05.108133602Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "2bb8fb60447e6387d374d98d196807dd9e32187695ddab5df85ff2a1ac856404",
|
||||
"short_id": "2bb8fb60",
|
||||
"time": "2026-10-03T15:05:39.219021268Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "5dd1f0b99e8ed22d7fdcc9005501e880ddd600341fe35bd6897174acdd943857",
|
||||
"short_id": "5dd1f0b9",
|
||||
"time": "2026-10-03T15:17:27.992813997Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "89df02373ec2ffbe4a6b02d622c30c6f5001b922fc692bb1e1b84dfb0b8d2971",
|
||||
"short_id": "89df0237",
|
||||
"time": "2026-10-04T02:15:03.833138622Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "4e7867af43f024c8f712417bce982c5c8085524b51e7f37040c665d0dc1146de",
|
||||
"short_id": "4e7867af",
|
||||
"time": "2026-10-05T02:15:03.941403075Z",
|
||||
"hostname": "demo-felhom",
|
||||
"tags": [
|
||||
"felhom-offbox",
|
||||
"opengist"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,26 @@
|
||||
2026-10-05T05:20:57Z
|
||||
grant HTTP 200
|
||||
|
||||
HTTP 302
|
||||
before 15 after 14
|
||||
removed: [('343d57a5', '2026-09-28T02:15:05'), ('92e49f62', '2026-09-13T02:15:05')]
|
||||
added by the run: [('47bad696', '2026-10-05T05:21:04')]
|
||||
2026/10/05 05:16:20 [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-06 04:15 CEST
|
||||
2026/10/05 05:16:35 [INFO] [bootrecon] boot window: fleet settled after 10s (3 identical samples 5s apart) — sweeping
|
||||
2026/10/05 05:21:00 [INFO] [offbox] backup run started (1 app(s) toggled)
|
||||
2026/10/05 05:21:01 [INFO] [offbox] pre-push dump leg completed in 892ms — snapshot pair is coherent
|
||||
2026/10/05 05:21:06 [INFO] [offbox] backed up opengist (/mnt/sys_drive/felhom-data/backups/primary/opengist, 0 mandatory path(s))
|
||||
2026/10/05 05:21:14 [INFO] [offbox] clean-up window 5 (after-run): 2 of 5 planned snapshot(s) removed, 16 -> 14, in 6s
|
||||
2026/10/05 05:21:18 [INFO] [offbox] backup OK: 1 app(s) backed up, 14 snapshot(s), 15s
|
||||
2026/10/05 05:21:18 [INFO] [offbox] manual run progress reporting ended after 18s
|
||||
Traceback (most recent call last):
|
||||
File "<stdin>", line 3, in <module>
|
||||
sqlite3.OperationalError: no such column: removed
|
||||
Traceback (most recent call last):
|
||||
File "<stdin>", line 3, in <module>
|
||||
sqlite3.OperationalError: no such column: box_reason
|
||||
--- hub offsite_windows (demo-felhom, newest 3)
|
||||
(5, '2026-10-05 05:21:08', '2026-10-05 05:21:15', 16, 14, 8, 'pruned', 'box')
|
||||
(3, '2026-10-05 02:15:08', '2026-10-05 02:15:11', 15, 15, 7, 'guard-refused', 'box')
|
||||
--- hub events since 05:20 UTC
|
||||
--- hub notification_log since 05:20 UTC
|
||||
@@ -0,0 +1,46 @@
|
||||
PLAN 80ebd0d6 2026-09-13T02:16:27Z [felhom-offbox bentopdf]
|
||||
PLAN 29516b0c 2026-09-13T02:16:31Z [felhom-offbox bookstack]
|
||||
PLAN 21efaca4 2026-09-13T02:16:35Z [felhom-offbox calibre-web]
|
||||
PLAN c8562d68 2026-09-13T02:16:39Z [felhom-offbox docmost]
|
||||
PLAN b4350226 2026-09-13T02:16:44Z [felhom-offbox kimai]
|
||||
PLAN 1ebc8069 2026-09-13T02:16:54Z [felhom-offbox paperless-ngx]
|
||||
PLAN 6fac4ba8 2026-09-13T02:17:01Z [felhom-offbox romm]
|
||||
PLAN 8cda9681 2026-09-13T02:17:08Z [felhom-offbox opengist]
|
||||
PLAN 605bf9c7 2026-09-13T02:17:11Z [felhom-offbox privatebin]
|
||||
PLAN 9efec80f 2026-09-28T13:42:12Z [felhom-offbox docmost]
|
||||
PLAN ef800224 2026-09-28T13:42:20Z [felhom-offbox opengist]
|
||||
PLAN c9c2414d 2026-09-28T13:42:24Z [felhom-offbox bentopdf]
|
||||
PLAN 14981ce0 2026-09-28T13:42:27Z [felhom-offbox bookstack]
|
||||
PLAN 693c93e1 2026-09-28T13:42:31Z [felhom-offbox calibre-web]
|
||||
PLAN 90da3064 2026-09-28T13:42:36Z [felhom-offbox kimai]
|
||||
PLAN 7db18829 2026-09-28T13:43:46Z [felhom-offbox paperless-ngx]
|
||||
PLAN 57191440 2026-09-28T13:43:52Z [felhom-offbox privatebin]
|
||||
PLAN 0febfb1d 2026-09-28T13:43:55Z [felhom-offbox romm]
|
||||
PLAN b522e2f4 2026-10-03T02:16:49Z [felhom-offbox bentopdf]
|
||||
PLAN c974cffe 2026-10-03T02:16:56Z [felhom-offbox calibre-web]
|
||||
PLAN 0bffcb8f 2026-10-03T02:17:00Z [felhom-offbox paperless-ngx]
|
||||
PLAN b5d27227 2026-10-03T02:17:08Z [felhom-offbox bookstack]
|
||||
PLAN 5d1830e6 2026-10-03T02:17:13Z [felhom-offbox docmost]
|
||||
PLAN 4a3c71f5 2026-10-03T02:17:19Z [felhom-offbox kimai]
|
||||
PLAN 0a155533 2026-10-03T02:17:26Z [felhom-offbox opengist]
|
||||
PLAN a5822405 2026-10-03T02:17:29Z [felhom-offbox privatebin]
|
||||
PLAN 2cd36d95 2026-10-03T02:17:32Z [felhom-offbox romm]
|
||||
PLAN bea276a7 2026-10-03T15:24:46Z [felhom-offbox bentopdf]
|
||||
PLAN 59717090 2026-10-03T15:24:49Z [felhom-offbox calibre-web]
|
||||
PLAN c6b5c67b 2026-10-03T15:24:51Z [felhom-offbox docmost]
|
||||
PLAN cb83c862 2026-10-03T15:24:54Z [felhom-offbox paperless-ngx]
|
||||
PLAN 38fbd3bc 2026-10-03T15:24:57Z [felhom-offbox privatebin]
|
||||
PLAN 4ce84465 2026-10-03T15:24:59Z [felhom-offbox romm]
|
||||
PLAN 2f72df90 2026-10-03T15:25:02Z [felhom-offbox bookstack]
|
||||
PLAN 2c0385d3 2026-10-03T15:25:04Z [felhom-offbox kimai]
|
||||
PLAN c43f2d0e 2026-10-03T15:25:07Z [felhom-offbox opengist]
|
||||
PLAN 2d412cd5 2026-10-04T02:16:44Z [felhom-offbox bentopdf]
|
||||
PLAN 93a66c3e 2026-10-04T02:16:46Z [felhom-offbox calibre-web]
|
||||
PLAN 6dbbceac 2026-10-04T02:16:48Z [felhom-offbox docmost]
|
||||
PLAN 84559526 2026-10-04T02:16:51Z [felhom-offbox paperless-ngx]
|
||||
PLAN 2abceaa3 2026-10-04T02:16:54Z [felhom-offbox privatebin]
|
||||
PLAN a86c8d7f 2026-10-04T02:16:56Z [felhom-offbox romm]
|
||||
PLAN 40328365 2026-10-04T02:16:59Z [felhom-offbox bookstack]
|
||||
PLAN f6c41de0 2026-10-04T02:17:01Z [felhom-offbox kimai]
|
||||
PLAN d2b6725f 2026-10-04T02:17:04Z [felhom-offbox opengist]
|
||||
GUARD at 2026-10-05T05:23:43Z: removes 18 [80ebd0d6 29516b0c 21efaca4 c8562d68 b4350226 1ebc8069 6fac4ba8 8cda9681 605bf9c7 9efec80f ef800224 c9c2414d 14981ce0 693c93e1 90da3064 7db18829 57191440 0febfb1d] refusal=""
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,33 @@
|
||||
2026-10-05T05:23:54Z
|
||||
grant HTTP 200
|
||||
HTTP 302
|
||||
2026/10/05 05:26:03 offbox_window.go:273: [INFO] [offbox] clean-up window 6 (after-run): 18 of 54 planned snapshot(s) removed, 145 -> 127, in 10s
|
||||
2026/10/05 05:26:08 offbox.go:1244: [INFO] [offbox] backup OK: 9 app(s) backed up, 127 snapshot(s), 2m7s
|
||||
before 136 after 127 removed 18 added by the run 9
|
||||
removed == predicted: True
|
||||
removed 80ebd0d6 2026-09-13T02:16:27 ['bentopdf']
|
||||
removed 29516b0c 2026-09-13T02:16:31 ['bookstack']
|
||||
removed 21efaca4 2026-09-13T02:16:35 ['calibre-web']
|
||||
removed c8562d68 2026-09-13T02:16:39 ['docmost']
|
||||
removed b4350226 2026-09-13T02:16:44 ['kimai']
|
||||
removed 1ebc8069 2026-09-13T02:16:54 ['paperless-ngx']
|
||||
removed 6fac4ba8 2026-09-13T02:17:01 ['romm']
|
||||
removed 8cda9681 2026-09-13T02:17:08 ['opengist']
|
||||
removed 605bf9c7 2026-09-13T02:17:11 ['privatebin']
|
||||
removed 9efec80f 2026-09-28T13:42:12 ['docmost']
|
||||
removed ef800224 2026-09-28T13:42:20 ['opengist']
|
||||
removed c9c2414d 2026-09-28T13:42:24 ['bentopdf']
|
||||
removed 14981ce0 2026-09-28T13:42:27 ['bookstack']
|
||||
removed 693c93e1 2026-09-28T13:42:31 ['calibre-web']
|
||||
removed 90da3064 2026-09-28T13:42:36 ['kimai']
|
||||
removed 7db18829 2026-09-28T13:43:46 ['paperless-ngx']
|
||||
removed 57191440 2026-09-28T13:43:52 ['privatebin']
|
||||
removed 0febfb1d 2026-09-28T13:43:55 ['romm']
|
||||
--- hub offsite_windows (demo-hp, newest 2)
|
||||
(6, '2026-10-05 05:25:53', '2026-10-05 05:26:04', 145, 127, 72, 'pruned', 'box')
|
||||
(2, '2026-10-04 05:36:14', '2026-10-04 05:36:17', 127, 127, None, 'nothing', 'box')
|
||||
--- hub events since 05:23 UTC
|
||||
--- notification_log since 05:23 UTC
|
||||
--- key audit
|
||||
[{"customer":"Tester-2","lines":1,"pinned":1,"findings":null},{"customer":"demo-felhom","lines":1,"pinned":1,"findings":null},{"customer":"demo-hp","lines":1,"pinned":1,"findings":null},{"customer":"tester-1","lines":1,"pinned":1,"findings":null}]
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
[{"customer":"Tester-2","lines":1,"pinned":1,"findings":null},{"customer":"demo-felhom","lines":1,"pinned":1,"findings":null},{"customer":"demo-hp","lines":1,"pinned":1,"findings":null},{"customer":"tester-1","lines":1,"pinned":1,"findings":null}]
|
||||
@@ -0,0 +1,17 @@
|
||||
R-867 lab, 2026-10-05, on DooPlex scratchpad (local repo, deleted after). restic binary copied out of
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.293.0: `restic 0.14.0 compiled with go1.19.8 on linux/amd64`.
|
||||
|
||||
Snapshot set: 2 groups (host box, tags opengist / bookstack), one snapshot per night at 02:15:05 UTC from
|
||||
2026-08-20 to 2026-10-05, box OFF 2026-09-15..17 (no snapshots), a same-day manual run at 15:00 on 2026-09-10
|
||||
and 2026-09-30. 92 snapshots, all UTC.
|
||||
|
||||
Real policy: restic forget --group-by host,tags --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --dry-run --json
|
||||
-> removes 72 snapshots
|
||||
Test simulation restic0140Plan (offbox_window_test.go) over the same `restic snapshots --json`:
|
||||
-> removes 72 snapshots
|
||||
diff of the two sorted id lists: EMPTY (identical)
|
||||
|
||||
The guard (v0.294.0 candidate) on that plan at now = 2026-10-05T02:20:00Z, maxRemove 500:
|
||||
GUARD removes=70 refusal=""
|
||||
(the 2 kept back are the 2026-09-30 02:15 copies superseded by the same-day 15:00 run — R-824's exclusion;
|
||||
removed in a later window once 7 calendar days old)
|
||||
@@ -0,0 +1,45 @@
|
||||
### RED-PROOF 1 the 8-day age line back: 'if calendarDaysBefore(s.Time, now) < keepDaily {' -> 'if now.Sub(s.Time) < 8*24*time.Hour {'
|
||||
offbox_window_test.go:425: the honest drop of a 7 d + 5 s snapshot must pass: ids=[] why="the policy would remove snapshot n0 from 2026-09-28T02:15:05Z — within the last 7 days kept daily and not superseded the same day, which honest retention never does"
|
||||
--- FAIL: TestOffsiteGuard_RealPolicyMeasuredShapeAllowed (0.00s)
|
||||
offbox_window_test.go:456: night 7 (2026-08-27T02:17:00Z): the guard refused the honest policy: the policy would remove snapshot opengist0 from 2026-08-20T02:15:00Z — within the last 7 days kept daily and not superseded the same day, which honest retention never does
|
||||
--- FAIL: TestOffsiteGuard_RealPolicySixtyNightsNeverRefuses (0.00s)
|
||||
offbox_window_test.go:502: weekly window on 2026-09-14 refused: the policy would remove snapshot s6 from 2026-09-07T02:15:00Z — within the last 7 days kept daily and not superseded the same day, which honest retention never does
|
||||
--- FAIL: TestOffsiteGuard_RealPolicyWeeklyWindows (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.020s
|
||||
FAIL
|
||||
rc=1
|
||||
### RED-PROOF 2 drop the future-date refusal: 's.Time.After(now.Add(offsiteGuardSkew))' -> 'false && s.Time.After(now.Add(offsiteGuardSkew))'
|
||||
offbox_window_test.go:155: window close = [{ID:7 CountBefore:16 CountAfter:16 Removed:0 Outcome:guard-refused Reason:snapshot f1 (2027-01-01T03:00:00Z) is newer than the hub allows (2026-10-05T04:55:06Z)}]
|
||||
--- FAIL: TestOffsiteGuard_LabThirteenFutureFakes_Refused (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.023s
|
||||
FAIL
|
||||
rc=1
|
||||
### RED-PROOF 3 drop the newer-than-hub refusal: '!newestAllowed.IsZero() && s.Time.After' -> 'false && s.Time.After'
|
||||
offbox_window_test.go:169: newest-allowed bound not enforced: ""
|
||||
--- FAIL: TestOffsiteGuard_RecentRemovalRefused (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.023s
|
||||
FAIL
|
||||
rc=1
|
||||
### RED-PROOF 4 drop the young-removal refusal: 'return nil, fmt.Sprintf("the policy would remove snapshot %s from %s — within' -> 'continue; _ = fmt.Sprintf("the policy would remove snapshot %s from %s — within'
|
||||
offbox_window_test.go:165: why = ""
|
||||
--- FAIL: TestOffsiteGuard_RecentRemovalRefused (0.00s)
|
||||
offbox_window_test.go:232: why = ""
|
||||
--- FAIL: TestOffsiteGuard_SameDaySupersededYoungExcluded (0.00s)
|
||||
offbox_window_test.go:532: why = "" — a real snapshot inside the daily window must not be removed
|
||||
--- FAIL: TestOffsiteGuard_RealPolicySkewWindowFakeRefused (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.024s
|
||||
FAIL
|
||||
rc=1
|
||||
### RED-PROOF 5 drop the weekly cap: 'if maxRemove >= 0 && len(keep) > maxRemove {' -> 'if false && len(keep) > maxRemove {'
|
||||
offbox_window_test.go:202: ids=[o5-full o4-full o3-full o2-full o1-full o0-full] why=""
|
||||
--- FAIL: TestOffsiteGuard_AboveWeeklyCapRefused (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.023s
|
||||
FAIL
|
||||
rc=1
|
||||
### restored
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.022s
|
||||
@@ -0,0 +1,37 @@
|
||||
### 05:28:31 pull 0.294.0 in 9202 (outside the timing)
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.294.0
|
||||
0
|
||||
### 05:28:34 make the one-time clean-up due: remove its marker; switch the image; restart
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.294.0 started 2026-10-05T05:28:35.617220084Z
|
||||
### 05:31:15 DEPLOY bookstack (POST /api/stacks/bookstack/deploy, the endpoint the install button calls)
|
||||
|
||||
HTTP 202
|
||||
05:31:20
|
||||
05:31:36
|
||||
05:31:53 bookstack=Created bookstack-db=Up 9 seconds (health: starting)
|
||||
05:32:09 bookstack=Up 15 seconds (health: starting) bookstack-db=Up 26 seconds (healthy)
|
||||
### controller log since start (image retention + bookstack deploy lines)
|
||||
2026/10/05 05:31:36 controller_image_retention.go:184: [INFO] [stacks] controller image retention: deleted gitea.dooplex.hu/admin/felhom-controller:0.291.0 (dde6b167c496, 410MB) — older than the previous controller and no container uses it (decision 56)
|
||||
2026/10/05 05:31:36 controller_image_retention.go:115: [INFO] [stacks] controller image retention: pass over 3 controller image(s) — running 0.294.0, previous "0.293.0" (by version order (no swap record names one present)), 1 candidate(s), 1 deleted, the rest kept
|
||||
2026/10/05 05:31:36 image_retention.go:211: [INFO] [stacks] image retention (one-time clean-up): skipped — an app install, update, restore or undo is pulling images now (R-863); tried again later
|
||||
2026/10/05 05:31:54 deploy.go:611: [INFO] [stacks] Stack bookstack deployed successfully (took 35.5s)
|
||||
2026/10/05 05:31:57 manager.go:1598: [INFO] [stacks] Stack bookstack post-start status:
|
||||
### 05:33:54 after the retry
|
||||
2026/10/05 05:31:36 image_retention.go:211: [INFO] [stacks] image retention (one-time clean-up): skipped — an app install, update, restore or undo is pulling images now (R-863); tried again later
|
||||
2026/10/05 05:33:36 image_retention.go:389: [INFO] [stacks] image retention (one-time): deleted 0 image(s). docker disk before: Images 6.245GB 2.943GB (47%) | after: Images 6.245GB 2.943GB (47%)
|
||||
deleted 0
|
||||
|
||||
bookstack lscr.io/linuxserver/bookstack:26.09.1 Up 2 minutes (healthy)
|
||||
bookstack-db mariadb:12.3 Up 2 minutes (healthy)
|
||||
### 05:34:05 teardown: stop + remove through the product
|
||||
HTTP 200
|
||||
|
||||
HTTP 200
|
||||
### 05:34:49 verify
|
||||
0
|
||||
ls: cannot access '/opt/docker/stacks/bookstack/app.yaml': No such file or directory
|
||||
0
|
||||
2026/10/05 05:34:41 delete.go:804: [INFO] Stack bookstack removed successfully (took 26.7s)
|
||||
2026/10/05 05:34:43 image_retention.go:265: [INFO] [stacks] image retention (remove of bookstack): deleted [lscr.io/linuxserver/bookstack:<none>] (b274bee3f111, 416MB) — no container, installed app or undo names it (decision 53)
|
||||
2026/10/05 05:34:43 image_retention.go:265: [INFO] [stacks] image retention (remove of bookstack): deleted [mariadb:<none>] (dfff46ef3f9d, 334MB) — no container, installed app or undo names it (decision 53)
|
||||
2026/10/05 05:34:43 image_retention.go:238: [INFO] [stacks] image retention (remove of bookstack): pass over 15 image(s) of [lscr.io/linuxserver/bookstack mariadb] — 2 candidate(s), 2 deleted, the rest kept
|
||||
@@ -0,0 +1,28 @@
|
||||
### RED-PROOF R-863: TryImageCleanup check dropped (endCleanup, ok := func() {}, true)
|
||||
image_retention_r863_test.go:70: the install failed: the clean-up deleted the image compose had just pulled (rmi [sha256:MDB])
|
||||
--- FAIL: TestR863_OneTimeCleanupDuringFirstInstallKeepsThePulledImage (0.01s)
|
||||
--- PASS: TestR863_OnlyPullingVerbsHoldTheLock (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.014s
|
||||
FAIL
|
||||
rc=1
|
||||
### RED-PROOF R-863b: a busy pass reports done (the old behaviour: the one-time clean-up's marker ends it for good)
|
||||
image_retention_r863_test.go:79: the clean-up reported done although it did not run — its marker would end it for good
|
||||
--- FAIL: TestR863_OneTimeCleanupDuringFirstInstallKeepsThePulledImage (0.01s)
|
||||
--- PASS: TestR863_OnlyPullingVerbsHoldTheLock (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.017s
|
||||
FAIL
|
||||
rc=1
|
||||
### restored
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.018s
|
||||
### RED-PROOF R-864: truncateStr (head) instead of tailStr for stderr
|
||||
r864_stderr_tail_test.go:45: the reason was cut from the returned error: "exit code 1\nstderr: Image lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 Pulli
|
||||
--- FAIL: TestR864_FailedComposeLogsTheReasonNotThePullProgress (0.01s)
|
||||
--- PASS: TestR864_TailStrKeepsTheEndOnARuneBoundary (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.012s
|
||||
FAIL
|
||||
rc=1
|
||||
### restored
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.015s
|
||||
@@ -0,0 +1,28 @@
|
||||
R-865 probe, 2026-10-05, scratch guest 9202 on demo-hp (read only). Before/after: .deb files in /var/cache/apt/archives, and the versions.
|
||||
status: running
|
||||
10
|
||||
bash 5.2.37-2+b10
|
||||
libc6:amd64 2.41-12+deb13u4
|
||||
---WITH-S
|
||||
Reading package lists...
|
||||
Building dependency tree...
|
||||
Reading state information...
|
||||
0 upgraded, 0 newly installed, 2 reinstalled, 0 to remove and 1 not upgraded.
|
||||
Inst libc6 [2.41-12+deb13u4] (2.41-12+deb13u4 Debian:13.7/stable [amd64])
|
||||
Conf libc6 (2.41-12+deb13u4 Debian:13.7/stable [amd64])
|
||||
Inst bash [5.2.37-2+b10] (5.2.37-2+b10 Debian:13.7/stable [amd64])
|
||||
Conf bash (5.2.37-2+b10 Debian:13.7/stable [amd64])
|
||||
---WITHOUT-S
|
||||
Reading package lists...
|
||||
Building dependency tree...
|
||||
Reading state information...
|
||||
0 upgraded, 0 newly installed, 2 reinstalled, 0 to remove and 1 not upgraded.
|
||||
Need to get 4347 kB of archives.
|
||||
After this operation, 0 B of additional disk space will be used.
|
||||
'http://deb.debian.org/debian/pool/main/b/bash/bash_5.2.37-2%2bb10_amd64.deb' bash_5.2.37-2+b10_amd64.deb 1500792 MD5Sum:27b11721fea83d73b96e0f7023863771
|
||||
'http://deb.debian.org/debian/pool/main/g/glibc/libc6_2.41-12%2bdeb13u4_amd64.deb' libc6_2.41-12+deb13u4_amd64.deb 2846580 MD5Sum:5559581916477ef1f57ea9f82cecf22e
|
||||
rc=0
|
||||
10
|
||||
bash 5.2.37-2+b10
|
||||
libc6:amd64 2.41-12+deb13u4
|
||||
(rc=0 above is head's; the cache count 10 -> 10 and the versions are unchanged: --print-uris without -s downloads nothing)
|
||||
@@ -0,0 +1,6 @@
|
||||
### 05:44:30 demo-hp: the INSTALLED wrapper's download_bytes (R8's measure) for the 13 pending upgrades, as root, read-only (--print-uris)
|
||||
36e16f7619cc556d
|
||||
download_bytes = 12802456 B -> R8 needs max(500 MB, 3 x 12802456 B) = 524288000 B free
|
||||
bind9-libs:amd64 1:9.20.26-1~deb13u1
|
||||
openssl 3.5.7-1~deb13u2
|
||||
13
|
||||
@@ -0,0 +1,8 @@
|
||||
### RED-PROOF R-865: -s put back into download_bytes
|
||||
1
|
||||
FAIL: test_R8_counts_the_real_download (test_felhom_os_apply.Refusals.test_R8_counts_the_real_download)
|
||||
FAIL: test_download_bytes_never_simulates (test_felhom_os_apply.Refusals.test_download_bytes_never_simulates)
|
||||
Ran 79 tests in 0.499s
|
||||
FAILED (failures=2)
|
||||
### restored
|
||||
OK
|
||||
@@ -0,0 +1,11 @@
|
||||
Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.454+02:00 level=INFO msg="felhom-agent daemon starting" version=0.144.1 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900
|
||||
Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.455+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T055135Z layer=docker vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T055135Z-docker-apply.json
|
||||
Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.455+02:00 level=INFO msg="osupdate: DONE" run=20261005T055135Z layer=docker vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=0 pending=0 not_covered=0 restart_needed=7 reboot_needed=true wrapper_seconds=14.7
|
||||
Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.529+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T055135Z layer=guest vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T055135Z-guest-apply.json
|
||||
Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.529+02:00 level=INFO msg="osupdate: DONE" run=20261005T055135Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=0 pending=0 not_covered=0 restart_needed=7 reboot_needed=true wrapper_seconds=13.3
|
||||
Oct 05 07:53:34 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:34.479+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T055135Z layer=host vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T055135Z-host-apply.json
|
||||
Oct 05 07:53:34 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:34.479+02:00 level=INFO msg="osupdate: DONE" run=20261005T055135Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=0 pending=80 not_covered=80 restart_needed=28 reboot_needed=false wrapper_seconds=12.1
|
||||
Oct 05 07:53:34 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:34.498+02:00 level=INFO msg="osupdate: sent kept report(s)" count=3
|
||||
(62, 'demo-felhom-8363b5', '2026-10-05 05:53:34', 'debug', 'apply', 'nothing', 1, 'host')
|
||||
(61, 'demo-felhom-8363b5', '2026-10-05 05:53:33', 'debug', 'apply', 'nothing', 1, 'guest')
|
||||
(60, 'demo-felhom-8363b5', '2026-10-05 05:53:33', 'debug', 'apply', 'nothing', 1, 'docker')
|
||||
@@ -0,0 +1,11 @@
|
||||
05:51:35 agent felhom-agent 0.144.0; saved block: 2026-10-05 07:23:26
|
||||
05:51:35 hub blocked (blackhole routes)
|
||||
=== felhom-agent 0.144.0 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=SAVED(2026-10-05T05:23:26Z; hub unreachable: hub: transport error: Get "https://hub.felhom.eu/api/v1/hosts/demo-felhom-8363b5/desired-state": dial tcp 37.191.56.193:443: connect: invalid argument) ===
|
||||
"healthy": true,
|
||||
"outcome": "nothing",
|
||||
"healthy": true,
|
||||
"outcome": "nothing",
|
||||
"healthy": true,
|
||||
"outcome": "nothing",
|
||||
pass took 42.8s
|
||||
05:52:18 hub unblocked; kept copies now: report-20261005T055135Z-docker-apply.json report-20261005T055135Z-guest-apply.json report-20261005T055135Z-host-apply.json
|
||||
@@ -0,0 +1,19 @@
|
||||
### RED-PROOF R-866a: selftestOSBlock returns at once on a fetch error (v0.143.0 behaviour)
|
||||
r866_selftest_block_test.go:37: the pass did not run from the saved block: ok=false block=<nil> src="desired state: hub: transport error: connect: invalid argument"
|
||||
--- FAIL: TestR866_DebugPassUsesTheSavedBlockWhenTheHubIsAway (0.00s)
|
||||
r866_selftest_block_test.go:53: ok=false why="desired state: down"
|
||||
--- FAIL: TestR866_NoHubNoSavedBlockDoesNotRun (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent 0.011s
|
||||
FAIL
|
||||
rc=1
|
||||
### RED-PROOF R-866b: the daemon no longer saves the block
|
||||
r866_selftest_block_test.go:37: the pass did not run from the saved block: ok=false block=<nil> src="desired state: hub: transport error: connect: invalid argument — and no saved block (the daemon saves one when the hub sends it)"
|
||||
--- FAIL: TestR866_DebugPassUsesTheSavedBlockWhenTheHubIsAway (0.00s)
|
||||
--- PASS: TestR866_NoHubNoSavedBlockDoesNotRun (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent 0.009s
|
||||
FAIL
|
||||
rc=1
|
||||
### restored
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent 0.009s
|
||||
@@ -0,0 +1,14 @@
|
||||
### RED-PROOF R-868 (live finding, v0.144.1): log() writes to stderr unguarded (v0.144.0)
|
||||
raise BrokenPipeError(32, "Broken pipe")
|
||||
BrokenPipeError: [Errno 32] Broken pipe
|
||||
FAILED (errors=1)
|
||||
### restored
|
||||
OK
|
||||
### RED-PROOF R-868 loop (v0.144.1): SendUnsentLoop returns after the first look
|
||||
unsent_test.go:138: the late copy was never sent
|
||||
--- FAIL: TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop (3.05s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 3.063s
|
||||
FAIL
|
||||
### restored
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 1.754s
|
||||
@@ -0,0 +1,10 @@
|
||||
05:44:46 daemon pid before 2471595; kept copies before: 0
|
||||
05:44:46 debug pass started
|
||||
KILLED at 05:45:04.880: the pass + the daemon (apt-get was running)
|
||||
bash: line 11: 2525079 Killed nohup sudo -u felhom-agent /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=os-update -vmid 9201 > /root/r868-pass.log 2>&1
|
||||
05:45:16 wrapper finished; kept copy:
|
||||
daemon now pid 2526755 active restarts 1
|
||||
audit rc=0
|
||||
bind9-libs:amd64 1:9.20.29-1~deb13u1
|
||||
openssl 3.5.7-1~deb13u3
|
||||
python3.13 3.13.5-2+deb13u5
|
||||
@@ -0,0 +1,24 @@
|
||||
### 06:03:27 rollback 2 (the same 13, simulated clean before)
|
||||
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
|
||||
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
|
||||
audit rc=0
|
||||
06:03:42 agent felhom-agent 0.144.1; daemon pid 2571767; kept copies before: 0
|
||||
KILLED at 06:04:00.258: the pass + the daemon (apt-get was running)
|
||||
bash: line 10: 2582951 Killed nohup sudo -u felhom-agent /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=os-update -vmid 9201 > /root/r868-pass2.log 2>&1
|
||||
06:04:20 wrapper finished; kept copy: report-20261005T060342Z-guest-apply.json
|
||||
daemon now active restarts 1
|
||||
audit rc=0
|
||||
bind9-libs:amd64 1:9.20.29-1~deb13u1
|
||||
openssl 3.5.7-1~deb13u3
|
||||
python3.13 3.13.5-2+deb13u5
|
||||
Oct 05 08:02:59 demo-hp felhom-os-apply[2580370]: os-apply: BUNDLE DONE agent=0.144.1 written=1 same=20 self-check=ok signers-created=False
|
||||
Oct 05 08:03:54 demo-hp felhom-os-apply[2583726]: os-apply: PLAN upgrade=13 already=0 not-installed=0 from-snapshot=0
|
||||
Oct 05 08:04:12 demo-hp felhom-os-apply[2585855]: os-apply: DONE rc=0 seconds=9.4 upgraded=13 restart-needed=containerd-shim,cron,dbus-daemon,sshd,systemd,systemd-journal,systemd-logind,systemd-network reboot-needed=yes
|
||||
### 06:09:09 the copy is gone
|
||||
Oct 05 08:04:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:04:05.487+02:00 level=INFO msg="felhom-agent daemon starting" version=0.144.1 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
|
||||
Oct 05 08:09:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:09:05.492+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T060342Z layer=guest vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T060342Z
|
||||
Oct 05 08:09:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:09:05.492+02:00 level=INFO msg="osupdate: DONE" run=20261005T060342Z layer=guest vmid=9201 ring=0 trigger=debug outcome=applied healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=13 pending=0 not_covered=0 re
|
||||
Oct 05 08:09:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:09:05.558+02:00 level=INFO msg="osupdate: sent kept report(s)" count=1
|
||||
--- hub os_reports (demo-hp, newest 2)
|
||||
(63, '2026-10-05 06:09:05', 'debug', 'apply', 'applied', 1, 'guest', '20261005T060342Z', 13)
|
||||
(59, '2026-10-05 02:59:56', 'debug', 'apply', 'nothing', 1, 'docker', '20261005T025842Z', None)
|
||||
@@ -0,0 +1,33 @@
|
||||
### RED-PROOF R-868a: SendUnsent sends nothing (sendUnsentLocked returns 0 at once)
|
||||
unsent_test.go:31: sent 0, want 1
|
||||
--- FAIL: TestR868_KilledPassIsReportedAtStart (0.00s)
|
||||
--- PASS: TestR868_NormalPassLeavesNoCopyAndNoDuplicate (0.00s)
|
||||
unsent_test.go:91: sent 0: []
|
||||
--- FAIL: TestR868_HubAwayKeepsTheCopy (0.00s)
|
||||
unsent_test.go:111: not sent after the pass: 0
|
||||
--- FAIL: TestR868_RunningPassKeepsTheSenderOff (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.014s
|
||||
FAIL
|
||||
rc=1
|
||||
### RED-PROOF R-868b: finish no longer deletes the sent copy
|
||||
unsent_test.go:31: sent 0, want 1
|
||||
--- FAIL: TestR868_KilledPassIsReportedAtStart (0.00s)
|
||||
unsent_test.go:59: kept copies left after the hub got them: [/tmp/TestR868_NormalPassLeavesNoCopyAndNoDuplicate3889087443/001/report-20261004T040000Z-docker-apply.json /tmp/TestR868_NormalPassLeavesNoCopyAndNoDuplicate3889087443/001/report-20261004T040000Z-guest-apply.json /tmp/TestR868_NormalPassLeavesNoCopyAndNoDuplicate3889087443/001/report-20261004T040000Z-host-apply.json]
|
||||
--- FAIL: TestR868_NormalPassLeavesNoCopyAndNoDuplicate (0.00s)
|
||||
unsent_test.go:91: sent 0: [{RunID:20261004T040000Z Layer:docker Trigger:night Mode:apply Ring:0 ReleaseID: Outcome:nothing Healthy:true HealthReason:sent after the agent stopped mid-pass (R-868) VMID:0 Upgraded:[] Installed:[] Pending:[] NotCovered:[] RestartNeeded:[] DockerRestartNeeded:false RebootNeeded:false RebootScanned:false Refused:[] PassSeconds:0 DockerEngine: Authority: Undo:false unsent:} {RunID:20261004T040000Z Layer:guest Trigger:night Mode:apply Ring:0 ReleaseID: Outcome:applied Healthy:true HealthReason:sent after the agent stopped mid-pass (R-868) VMID:0 Upgraded:[{Name:libc6 Version:u4 Origin:}] Installed:[] Pending:[] NotCovered:[] RestartNeeded:[] DockerRestartNeeded:false RebootNeeded:false RebootScanned:false Refused:[] PassSeconds:0 DockerEngine: Authority: Undo:false unsent:}]
|
||||
--- FAIL: TestR868_HubAwayKeepsTheCopy (0.00s)
|
||||
unsent_test.go:111: not sent after the pass: 0
|
||||
--- FAIL: TestR868_RunningPassKeepsTheSenderOff (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.012s
|
||||
FAIL
|
||||
rc=1
|
||||
### RED-PROOF R-868c (wrapper): main() no longer saves the copy
|
||||
ERROR: test_a_refusal_is_kept_too (test_felhom_os_apply.UnsentReport.test_a_refusal_is_kept_too)
|
||||
FAIL: test_apply_keeps_a_copy_with_the_agents_ids (test_felhom_os_apply.UnsentReport.test_apply_keeps_a_copy_with_the_agents_ids)
|
||||
Ran 86 tests in 0.520s
|
||||
FAILED (failures=1, errors=1)
|
||||
### restored
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 2.328s
|
||||
OK
|
||||
@@ -0,0 +1,9 @@
|
||||
### RED-PROOF R-869: newPath assigned after the log line (v0.293.0 order)
|
||||
offbox_window_test.go:271: the line names no destination:
|
||||
--- FAIL: TestR869_MoveAsideLineNamesTheDestination (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.009s
|
||||
FAIL
|
||||
rc=1
|
||||
### restored
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.008s
|
||||
@@ -0,0 +1,2 @@
|
||||
=== felhom-agent 0.144.0 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub ===
|
||||
time=2026-10-05T07:44:46.566+02:00 level=INFO msg="osupdate: START" run=20261005T054446Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261005T054446Z
|
||||
@@ -0,0 +1,2 @@
|
||||
=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub ===
|
||||
time=2026-10-05T08:03:42.262+02:00 level=INFO msg="osupdate: START" run=20261005T060342Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261005T060342Z
|
||||
@@ -0,0 +1,39 @@
|
||||
### 06:09:35 demo-hp before the crash
|
||||
{
|
||||
"armed": true,
|
||||
"boot_id": "182af911-22a9-41ec-97ac-7ee247a4d470",
|
||||
"config": {
|
||||
"LIMIT": 3,
|
||||
"PANIC_SECONDS": 10,
|
||||
"REARM_HOURS": 24,
|
||||
"WINDOW_MINUTES": 60
|
||||
},
|
||||
"kernel_panic": 10,
|
||||
"last_boot_at": "2026-10-04T15:17:26Z",
|
||||
"last_boot_unclean": true,
|
||||
"last_trip": {
|
||||
"at": "2026-10-04T14:54:43Z",
|
||||
"reason": "2 unclean boots within 60 minutes \u2014 the next crash leaves the box off (limit 3)"
|
||||
},
|
||||
"rearmed_at": "2026-10-04T15:33:25Z",
|
||||
"rearmed_by": "operator",
|
||||
"tripped": false,
|
||||
"unclean_boots": [
|
||||
"2026-10-04T14:52:08Z",
|
||||
"2026-10-04T14:54:33Z",
|
||||
"2026-10-04T15:17:26Z"
|
||||
],
|
||||
"unclean_boots_24h": 3,
|
||||
"unclean_boots_in_window": 0,
|
||||
"updated_at": "2026-10-05T05:34:05Z",
|
||||
"version": 1
|
||||
}
|
||||
kernel.panic = 10
|
||||
kernel.sysrq = 438
|
||||
08:09:36 up 14:52, 1 user, load average: 0.91, 1.14, 1.10
|
||||
-2 cc0aa40bac534524a46f6c5983ce5ed9 Sun 2026-10-04 16:52:13 CEST Sun 2026-10-04 16:53:36 CEST
|
||||
-1 53f69dfb93ef44c3a9ad803d86dfb0b3 Sun 2026-10-04 16:54:38 CEST Sun 2026-10-04 16:55:14 CEST
|
||||
0 182af91122a941ec97ac7ee247a4d470 Sun 2026-10-04 17:17:31 CEST Mon 2026-10-05 08:09:37 CEST
|
||||
20
|
||||
21
|
||||
felhom-agent 0.144.1
|
||||
@@ -0,0 +1,22 @@
|
||||
### 06:09:53 rollback 3 (13 packages; simulated first)
|
||||
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
|
||||
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
|
||||
audit rc=0
|
||||
bind9-dnsutils 1:9.20.26-1~deb13u1
|
||||
bind9-host 1:9.20.26-1~deb13u1
|
||||
bind9-libs:amd64 1:9.20.26-1~deb13u1
|
||||
libpcre2-8-0:amd64 10.46-1~deb13u2
|
||||
libpython3.13-minimal:amd64 3.13.5-2+deb13u4
|
||||
libpython3.13-stdlib:amd64 3.13.5-2+deb13u4
|
||||
libssh2-1t64:amd64 1.11.1-1+deb13u1
|
||||
libssl3t64:amd64 3.5.7-1~deb13u2
|
||||
libxml2:amd64 2.12.7+dfsg+really2.9.14-2.1+deb13u1
|
||||
openssl 3.5.7-1~deb13u2
|
||||
openssl-provider-legacy 3.5.7-1~deb13u2
|
||||
python3.13 3.13.5-2+deb13u4
|
||||
python3.13-minimal 3.13.5-2+deb13u4
|
||||
### 06:12:44 attempt 1 made no crash: the watcher pattern 'dpkg --status-fd' never matched (apt puts options before --status-fd); the pass installed the 13 normally. Pattern now 'bin/dpkg .*--status-fd'.
|
||||
### 06:12:44 rollback 4
|
||||
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
|
||||
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
|
||||
audit rc=0
|
||||
@@ -0,0 +1,9 @@
|
||||
06:10:25.835 debug pass started
|
||||
no dpkg seen
|
||||
ssh ended 06:12:25
|
||||
### attempt 2
|
||||
06:13:33.983 debug pass started
|
||||
dpkg running at 06:13:55.070:
|
||||
CRASH at 06:13:55.904
|
||||
Timeout, server 192.168.0.104 not responding.
|
||||
ssh ended 06:14:03
|
||||
@@ -0,0 +1,10 @@
|
||||
ssh back 06:14:48
|
||||
08:14:47 up 0 min, 1 user, load average: 1.33, 0.29, 0.10
|
||||
"armed": true,
|
||||
"last_boot_at": "2026-10-05T06:14:33Z",
|
||||
"last_boot_unclean": true,
|
||||
"rearmed_at": "2026-10-04T15:33:25Z",
|
||||
"rearmed_by": "operator",
|
||||
"tripped": false,
|
||||
"unclean_boots_in_window": 1,
|
||||
kernel.panic = 10
|
||||
@@ -0,0 +1,20 @@
|
||||
### 06:14:56 guest state after the crash
|
||||
status: running
|
||||
audit rc=0
|
||||
bind9-dnsutils 1:9.20.26-1~deb13u1 ii
|
||||
bind9-host 1:9.20.26-1~deb13u1 ii
|
||||
bind9-libs 1:9.20.26-1~deb13u1 ii
|
||||
libpcre2-8-0 10.46-1~deb13u2 ii
|
||||
libpython3.13-minimal 3.13.5-2+deb13u4 ii
|
||||
libpython3.13-stdlib 3.13.5-2+deb13u4 ii
|
||||
libssh2-1t64 1.11.1-1+deb13u1 ii
|
||||
libssl3t64 3.5.7-1~deb13u2 ii
|
||||
libxml2 2.12.7+dfsg+really2.9.14-2.1+deb13u1 ii
|
||||
openssl 3.5.7-1~deb13u2 ii
|
||||
openssl-provider-legacy 3.5.7-1~deb13u3 ii
|
||||
python3.13 3.13.5-2+deb13u4 ii
|
||||
python3.13-minimal 3.13.5-2+deb13u4 ii
|
||||
plan-20261005T025659Z-guest-apply.json
|
||||
plan-20261005T054446Z-guest-apply.json
|
||||
plan-20261005T061334Z-guest-apply.json
|
||||
06:17:41 apps: 20 healthy of 21
|
||||
@@ -0,0 +1,14 @@
|
||||
### 06:17:43 the next pass (debug, by hand — the night leg's 20 h gap would hold the real one)
|
||||
=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub ===
|
||||
"healthy": false,
|
||||
"outcome": "failed",
|
||||
"reboot_needed": false,
|
||||
pass took 20.2s
|
||||
Oct 05 08:13:35 demo-hp felhom-os-apply[2618444]: os-apply: START release=ring0-20261005T061334Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0
|
||||
Oct 05 08:13:40 demo-hp felhom-os-apply[2619128]: os-apply: REPAIR configured=0 fixed=0
|
||||
Oct 05 08:13:46 demo-hp felhom-os-apply[2619504]: os-apply: PLAN upgrade=13 already=0 not-installed=0 from-snapshot=0
|
||||
Oct 05 08:17:44 demo-hp felhom-os-apply[16225]: os-apply: START release=ring0-20261005T061743Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0
|
||||
Oct 05 08:17:49 demo-hp felhom-os-apply[16446]: os-apply: REPAIR configured=0 fixed=0
|
||||
Oct 05 08:17:55 demo-hp felhom-os-apply[16789]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0
|
||||
Oct 05 08:18:03 demo-hp felhom-os-apply[17194]: os-apply: FAILED rc=100 step=install — dpkg state: clean
|
||||
audit rc=0
|
||||
@@ -0,0 +1,13 @@
|
||||
### 06:18:33 why: dpkg's update journal (apt's 'interrupted' check) vs dpkg --audit
|
||||
total 20
|
||||
drwxr-xr-x 2 root root 4096 Oct 5 06:13 .
|
||||
drwxr-xr-x 7 root root 4096 Oct 5 06:13 ..
|
||||
-rw-r--r-- 1 root root 1048 Oct 5 06:13 0000
|
||||
-rw-r--r-- 1 root root 1055 Oct 5 06:13 0001
|
||||
-rw-r--r-- 1 root root 1017 Oct 5 06:13 0002
|
||||
audit rc=0
|
||||
Inst openssl [3.5.7-1~deb13u2] (3.5.7-1~deb13u3 Debian-Security:13/stable-security [amd64])
|
||||
Conf openssl (3.5.7-1~deb13u3 Debian-Security:13/stable-security [amd64])
|
||||
905: def repair(self):
|
||||
911: self.x(APT_ENV + ["dpkg", "--configure", "-a", "--force-confold"])
|
||||
916: self.r.log(f"os-apply: REPAIR configured={configured} fixed={fixed}")
|
||||
@@ -0,0 +1,27 @@
|
||||
### 06:18:55 by-hand repair: dpkg --configure -a in the guest (what the wrapper's REPAIR would do)
|
||||
rc=0
|
||||
0
|
||||
### 06:18:58 the next pass after the repair
|
||||
=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub ===
|
||||
"healthy": true,
|
||||
"outcome": "applied",
|
||||
"healthy": true,
|
||||
"outcome": "nothing",
|
||||
"healthy": true,
|
||||
"outcome": "nothing",
|
||||
pass took 1m16.9s
|
||||
Oct 05 08:17:49 demo-hp felhom-os-apply[16446]: os-apply: REPAIR configured=0 fixed=0
|
||||
Oct 05 08:17:55 demo-hp felhom-os-apply[16789]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0
|
||||
Oct 05 08:18:03 demo-hp felhom-os-apply[17194]: os-apply: FAILED rc=100 step=install — dpkg state: clean
|
||||
Oct 05 08:19:04 demo-hp felhom-os-apply[20081]: os-apply: REPAIR configured=0 fixed=0
|
||||
Oct 05 08:19:10 demo-hp felhom-os-apply[20406]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0
|
||||
Oct 05 08:19:28 demo-hp felhom-os-apply[21616]: os-apply: DONE rc=0 seconds=10.7 upgraded=12 restart-needed=cron,dbus-daemon,sshd,systemd,systemd-journal,systemd-logind,systemd-network reboot-needed=y
|
||||
Oct 05 08:19:40 demo-hp felhom-os-apply[22557]: os-apply: REPAIR configured=0 fixed=0
|
||||
Oct 05 08:19:44 demo-hp felhom-os-apply[22730]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
|
||||
Oct 05 08:19:44 demo-hp felhom-os-apply[22731]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
Oct 05 08:20:02 demo-hp felhom-os-apply[24349]: os-apply: REPAIR configured=0 fixed=0
|
||||
Oct 05 08:20:07 demo-hp felhom-os-apply[24637]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
|
||||
Oct 05 08:20:07 demo-hp felhom-os-apply[24638]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
audit rc=0
|
||||
0
|
||||
package list == before rollback 1 (279 lines)
|
||||
@@ -0,0 +1,9 @@
|
||||
--- hub events demo-hp since 06:13 UTC
|
||||
('controller_started', 'info', 'Controller elindult (0.294.0)', 'controller', '2026-10-05 06:16:39')
|
||||
('os_update_failed', 'error', 'OS update (guest) on demo-hp-bb76ea failed: {"dpkg_audit":"clean","rc":100,"tail":["E: dpkg was interrupted, you must manually run \'sudo dpkg --config', 'hub', '2026-10-05 06:18:03')
|
||||
('os_update_applied', 'info', 'System security fixes installed on the box (12 package(s)).', 'hub', '2026-10-05 06:19:35')
|
||||
--- notification_log since 06:13 UTC (all customers)
|
||||
('demo-hp', 'os_update_failed', 'sent', 'operator', '2026-10-05 06:18:03')
|
||||
('demo-hp', 'os_update_failed', 'skipped', 'customer', '2026-10-05 06:18:03')
|
||||
--- System page line (demo-hp)
|
||||
go · nothing · 12 s demo-hp-bb76ea Demo HP partial: no running customer guest ring 0 → normal updates ON switch off unknown 9.2.2 unknown unknown unknown ring0-20261005T061858Z 78 0 unknown no unknown no unknown not installed 0.144.1 unknown ring0-20261005T061858Z 0 7 unknown unknown unknown ring0-20261005T061858Z 0 min ago · nothing · 19 s tester-1-d70be4 Tester 1 ring 1 → demo updates ON switch off running 9.2.2 7.0.2-6-pve 7.0.2-6-pve 13.5 — 185 106 none no 10 s no 0 armed 0.144.1 13.6 — 49 0 29.8.2 2.3.6-1~debian.13~trixie on — 10 h ago · nothing · 16 s Amber: worth a look. Red: an operator alarm fires (`08` §6.3). "unknown": the box could not read the value — never a guess. Felhom Hub 0.133.0
|
||||
@@ -0,0 +1,279 @@
|
||||
adduser 3.152
|
||||
apparmor 4.1.0-1
|
||||
apt 3.0.3
|
||||
apt-listchanges 4.8
|
||||
apt-utils 3.0.3
|
||||
base-files 13.8+deb13u7
|
||||
base-passwd 3.6.7
|
||||
bash 5.2.37-2+b10
|
||||
bash-completion 1:2.16.0-7
|
||||
bind9-dnsutils 1:9.20.29-1~deb13u1
|
||||
bind9-host 1:9.20.29-1~deb13u1
|
||||
bind9-libs 1:9.20.29-1~deb13u1
|
||||
bsdextrautils 2.41.5-0+deb13u1
|
||||
bsdutils 1:2.41.5-0+deb13u1
|
||||
bzip2 1.0.8-6
|
||||
ca-certificates 20250419
|
||||
containerd.io 2.3.6-1~debian.13~trixie
|
||||
coreutils 9.7-3
|
||||
cpio 2.15+dfsg-2
|
||||
cron 3.0pl1-197
|
||||
cron-daemon-common 3.0pl1-197
|
||||
curl 8.14.1-2+deb13u5
|
||||
dash 0.5.12-12
|
||||
dbus 1.16.2-2
|
||||
dbus-bin 1.16.2-2
|
||||
dbus-daemon 1.16.2-2
|
||||
dbus-session-bus-common 1.16.2-2
|
||||
dbus-system-bus-common 1.16.2-2
|
||||
dbus-user-session 1.16.2-2
|
||||
debconf 1.5.91
|
||||
debconf-i18n 1.5.91
|
||||
debian-archive-keyring 2025.1
|
||||
debian-faq 12.2
|
||||
debianutils 5.23.2
|
||||
dhcpcd-base 1:10.1.0-11+deb13u4
|
||||
diffutils 1:3.10-4
|
||||
distro-info-data 0.66+deb13u2
|
||||
dmidecode 3.6-2
|
||||
doc-debian 11.3+nmu1
|
||||
docker-buildx-plugin 0.37.1-1~debian.13~trixie
|
||||
docker-ce 5:29.8.2-1~debian.13~trixie
|
||||
docker-ce-cli 5:29.8.2-1~debian.13~trixie
|
||||
docker-ce-rootless-extras 5:29.8.2-1~debian.13~trixie
|
||||
docker-compose-plugin 5.6.0-1~debian.13~trixie
|
||||
dpkg 1.22.22
|
||||
e2fsprogs 1.47.2-3+b12
|
||||
fdisk 2.41.5-0+deb13u1
|
||||
file 1:5.46-5
|
||||
findutils 4.10.0-3
|
||||
gcc-14-base 14.2.0-19
|
||||
gettext-base 0.23.1-2
|
||||
git 1:2.47.3-0+deb13u1
|
||||
git-man 1:2.47.3-0+deb13u1
|
||||
grep 3.11-4
|
||||
groff-base 1.23.0-9
|
||||
gzip 1.13-1+deb13u1
|
||||
hostname 3.25
|
||||
ifupdown 0.8.44+deb13u1
|
||||
ifupdown2 3.0.0-1.3
|
||||
inetutils-telnet 2:2.6-3+deb13u3
|
||||
init 1.69~deb13u1
|
||||
init-system-helpers 1.69~deb13u1
|
||||
iproute2 6.15.0-1
|
||||
iptables 1.8.11-2
|
||||
iputils-ping 3:20240905-3
|
||||
isc-dhcp-client 4.4.3-P1-8
|
||||
kmod 34.2-2
|
||||
krb5-locales 1.21.3-5+deb13u1
|
||||
less 668-1
|
||||
libacl1 2.3.2-2+b1
|
||||
libapparmor1 4.1.0-1
|
||||
libapt-pkg7.0 3.0.3
|
||||
libattr1 1:2.5.2-3
|
||||
libaudit-common 1:4.0.2-2+deb13u1
|
||||
libaudit1 1:4.0.2-2+deb13u1
|
||||
libblkid1 2.41.5-0+deb13u1
|
||||
libbpf1 1:1.5.0-3
|
||||
libbrotli1 1.1.0-2+b7
|
||||
libbsd0 0.12.2-2
|
||||
libbz2-1.0 1.0.8-6
|
||||
libc-bin 2.41-12+deb13u4
|
||||
libc-l10n 2.41-12+deb13u4
|
||||
libc6 2.41-12+deb13u4
|
||||
libcap-ng0 0.8.5-4+b1
|
||||
libcap2 1:2.75-10+deb13u1+b3
|
||||
libcap2-bin 1:2.75-10+deb13u1+b3
|
||||
libcbor0.10 0.10.2-2
|
||||
libcom-err2 1.47.2-3+b12
|
||||
libcrypt1 1:4.4.38-1
|
||||
libcurl3t64-gnutls 8.14.1-2+deb13u5
|
||||
libcurl4t64 8.14.1-2+deb13u5
|
||||
libdb5.3t64 5.3.28+dfsg2-9
|
||||
libdbus-1-3 1.16.2-2
|
||||
libdebconfclient0 0.280
|
||||
libedit2 3.1-20250104-1
|
||||
libelf1t64 0.192-4
|
||||
liberror-perl 0.17030-1
|
||||
libexpat1 2.8.3-1~deb13u1
|
||||
libext2fs2t64 1.47.2-3+b12
|
||||
libfdisk1 2.41.5-0+deb13u1
|
||||
libffi8 3.4.8-2
|
||||
libfido2-1 1.15.0-1+b1
|
||||
libfstrm0 0.6.1-1+b3
|
||||
libgcc-s1 14.2.0-19
|
||||
libgdbm-compat4t64 1.24-2
|
||||
libgdbm6t64 1.24-2
|
||||
libgmp10 2:6.3.0+dfsg-3
|
||||
libgnutls30t64 3.8.9-3+deb13u4
|
||||
libgssapi-krb5-2 1.21.3-5+deb13u1
|
||||
libhogweed6t64 3.10.1-1
|
||||
libicu76 76.1-4
|
||||
libidn2-0 2.3.8-2
|
||||
libip4tc2 1.8.11-2
|
||||
libip6tc2 1.8.11-2
|
||||
libjansson4 2.14-2+b3
|
||||
libjemalloc2 5.3.0-3
|
||||
libjson-c5 0.18+ds-1
|
||||
libk5crypto3 1.21.3-5+deb13u1
|
||||
libkeyutils1 1.6.3-6
|
||||
libkmod2 34.2-2
|
||||
libkrb5-3 1.21.3-5+deb13u1
|
||||
libkrb5support0 1.21.3-5+deb13u1
|
||||
liblastlog2-2 2.41.5-0+deb13u1
|
||||
libldap-common 2.6.10+dfsg-1
|
||||
libldap2 2.6.10+dfsg-1
|
||||
liblmdb0 0.9.31-1+b2
|
||||
liblocale-gettext-perl 1.07-7+b1
|
||||
liblockfile-bin 1.17-2
|
||||
liblsof0 4.99.4+dfsg-2
|
||||
liblz4-1 1.10.0-4
|
||||
liblzma5 5.8.1-1+deb13u1
|
||||
libmagic-mgc 1:5.46-5
|
||||
libmagic1t64 1:5.46-5
|
||||
libmaxminddb0 1.12.2-1
|
||||
libmd0 1.1.0-2+b1
|
||||
libmnl0 1.0.5-3
|
||||
libmount1 2.41.5-0+deb13u1
|
||||
libncursesw6 6.5+20250216-2
|
||||
libnetfilter-conntrack3 1.1.0-1
|
||||
libnettle8t64 3.10.1-1
|
||||
libnewt0.52 0.52.25-1
|
||||
libnfnetlink0 1.0.2-3
|
||||
libnftables1 1.1.3-1
|
||||
libnftnl11 1.2.9-1
|
||||
libnghttp2-14 1.64.0-1.1+deb13u1
|
||||
libnghttp3-9 1.8.0-1
|
||||
libngtcp2-16 1.11.0-1+deb13u1
|
||||
libngtcp2-crypto-gnutls8 1.11.0-1+deb13u1
|
||||
libnsl2 1.3.0-3+b3
|
||||
libnss-systemd 257.13-1~deb13u1
|
||||
libp11-kit0 0.25.5-3
|
||||
libpam-modules 1.7.0-5
|
||||
libpam-modules-bin 1.7.0-5
|
||||
libpam-runtime 1.7.0-5
|
||||
libpam-systemd 257.13-1~deb13u1
|
||||
libpam-wtmpdb 0.73.0-3+deb13u1
|
||||
libpam0g 1.7.0-5
|
||||
libpcre2-8-0 10.46-1~deb13u3
|
||||
libperl5.40 5.40.1-6+deb13u1
|
||||
libpipeline1 1.5.8-1
|
||||
libpopt0 1.19+dfsg-2
|
||||
libproc2-0 2:4.0.4-9
|
||||
libprotobuf-c1 1.5.1-1
|
||||
libpsl5t64 0.21.2-1.1+b1
|
||||
libpython3-stdlib 3.13.5-1
|
||||
libpython3.13-minimal 3.13.5-2+deb13u5
|
||||
libpython3.13-stdlib 3.13.5-2+deb13u5
|
||||
libreadline8t64 8.2-6
|
||||
librtmp1 2.4+20151223.gitfa8646d.1-2+b5
|
||||
libsasl2-2 2.1.28+dfsg1-9
|
||||
libsasl2-modules-db 2.1.28+dfsg1-9
|
||||
libseccomp2 2.6.0-2
|
||||
libselinux1 3.8.1-1
|
||||
libsemanage-common 3.8.1-1
|
||||
libsemanage2 3.8.1-1
|
||||
libsepol2 3.8.1-1
|
||||
libslang2 2.3.3-5+b2
|
||||
libsmartcols1 2.41.5-0+deb13u1
|
||||
libsqlite3-0 3.46.1-7+deb13u2
|
||||
libss2 1.47.2-3+b12
|
||||
libssh2-1t64 1.11.1-1+deb13u2
|
||||
libssl3t64 3.5.7-1~deb13u3
|
||||
libstdc++6 14.2.0-19
|
||||
libsystemd-shared 257.13-1~deb13u1
|
||||
libsystemd0 257.13-1~deb13u1
|
||||
libtasn1-6 4.20.0-2+deb13u1
|
||||
libtext-charwidth-perl 0.04-11+b4
|
||||
libtext-iconv-perl 1.7-8+b4
|
||||
libtext-wrapi18n-perl 0.06-10
|
||||
libtinfo6 6.5+20250216-2
|
||||
libtirpc-common 1.3.6+ds-1
|
||||
libtirpc3t64 1.3.6+ds-1
|
||||
libtlsrpt0 0.5.0rc1-2
|
||||
libuchardet0 0.0.8-1+b2
|
||||
libudev1 257.13-1~deb13u1
|
||||
libunistring5 1.3-2
|
||||
liburcu8t64 0.15.2-2
|
||||
libuuid1 2.41.5-0+deb13u1
|
||||
libuv1t64 1.50.0-2
|
||||
libwrap0 7.6.q-36
|
||||
libwtmpdb0 0.73.0-3+deb13u1
|
||||
libxml2 2.12.7+dfsg+really2.9.14-2.1+deb13u3
|
||||
libxtables12 1.8.11-2
|
||||
libxxhash0 0.8.3-2
|
||||
libzstd1 1.5.7+dfsg-1
|
||||
linux-sysctl-defaults 4.12.1
|
||||
locales 2.41-12+deb13u4
|
||||
login 1:4.16.0-2+really2.41.5-0+deb13u1
|
||||
login.defs 1:4.17.4-2
|
||||
logrotate 3.22.0-1
|
||||
logsave 1.47.2-3+b12
|
||||
lsof 4.99.4+dfsg-2
|
||||
man-db 2.13.1-1
|
||||
manpages 6.9.1-1
|
||||
mawk 1.3.4.20250131-1
|
||||
media-types 13.0.0
|
||||
mount 2.41.5-0+deb13u1
|
||||
nano 8.4-1+deb13u1
|
||||
ncurses-base 6.5+20250216-2
|
||||
ncurses-bin 6.5+20250216-2
|
||||
ncurses-term 6.5+20250216-2
|
||||
netbase 6.5
|
||||
netcat-traditional 1.10-50
|
||||
nftables 1.1.3-1
|
||||
openssh-client 1:10.0p1-7+deb13u4
|
||||
openssh-server 1:10.0p1-7+deb13u4
|
||||
openssh-sftp-server 1:10.0p1-7+deb13u4
|
||||
openssl 3.5.7-1~deb13u3
|
||||
openssl-provider-legacy 3.5.7-1~deb13u3
|
||||
passwd 1:4.17.4-2
|
||||
patch 2.8-2
|
||||
perl 5.40.1-6+deb13u1
|
||||
perl-base 5.40.1-6+deb13u1
|
||||
perl-modules-5.40 5.40.1-6+deb13u1
|
||||
pigz 2.8-1
|
||||
postfix 3.10.13-0+deb13u1
|
||||
procps 2:4.0.4-9
|
||||
python-apt-common 3.0.0
|
||||
python3 3.13.5-1
|
||||
python3-apt 3.0.0
|
||||
python3-certifi 2025.1.31+ds-1
|
||||
python3-chardet 5.2.0+dfsg-2
|
||||
python3-charset-normalizer 3.4.2-1
|
||||
python3-debconf 1.5.91
|
||||
python3-debian 1.0.1
|
||||
python3-debianbts 4.1.1
|
||||
python3-idna 3.10-1+deb13u1
|
||||
python3-minimal 3.13.5-1
|
||||
python3-reportbug 13.2.0
|
||||
python3-requests 2.32.3+dfsg-5+deb13u1
|
||||
python3-urllib3 2.3.0-3+deb13u2
|
||||
python3.13 3.13.5-2+deb13u5
|
||||
python3.13-minimal 3.13.5-2+deb13u5
|
||||
readline-common 8.2-6
|
||||
reportbug 13.2.0
|
||||
runit-helper 2.16.4
|
||||
sed 4.9-2+deb13u1
|
||||
sensible-utils 0.0.25
|
||||
sqv 1.3.0-3+b2
|
||||
ssh 1:10.0p1-7+deb13u4
|
||||
systemd 257.13-1~deb13u1
|
||||
systemd-sysv 257.13-1~deb13u1
|
||||
systemd-timesyncd 257.13-1~deb13u1
|
||||
sysvinit-utils 3.14-4
|
||||
tar 1.35+dfsg-3.1
|
||||
traceroute 1:2.1.6-1
|
||||
tzdata 2026c-0+deb13u1
|
||||
ucf 3.0052
|
||||
udev 257.13-1~deb13u1
|
||||
util-linux 2.41.5-0+deb13u1
|
||||
util-linux-extra 2.41.5-0+deb13u1
|
||||
vim-common 2:9.1.1230-2
|
||||
vim-tiny 2:9.1.1230-2
|
||||
wamerican 2020.12.07-4
|
||||
wget 1.25.0-2
|
||||
whiptail 0.52.25-1
|
||||
wtmpdb 0.73.0-3+deb13u1
|
||||
xz-utils 5.8.1-1+deb13u1
|
||||
zlib1g 1:1.3.dfsg+really1.3.1-1+b1
|
||||
@@ -0,0 +1,279 @@
|
||||
adduser 3.152
|
||||
apparmor 4.1.0-1
|
||||
apt 3.0.3
|
||||
apt-listchanges 4.8
|
||||
apt-utils 3.0.3
|
||||
base-files 13.8+deb13u7
|
||||
base-passwd 3.6.7
|
||||
bash 5.2.37-2+b10
|
||||
bash-completion 1:2.16.0-7
|
||||
bind9-dnsutils 1:9.20.29-1~deb13u1
|
||||
bind9-host 1:9.20.29-1~deb13u1
|
||||
bind9-libs 1:9.20.29-1~deb13u1
|
||||
bsdextrautils 2.41.5-0+deb13u1
|
||||
bsdutils 1:2.41.5-0+deb13u1
|
||||
bzip2 1.0.8-6
|
||||
ca-certificates 20250419
|
||||
containerd.io 2.3.6-1~debian.13~trixie
|
||||
coreutils 9.7-3
|
||||
cpio 2.15+dfsg-2
|
||||
cron 3.0pl1-197
|
||||
cron-daemon-common 3.0pl1-197
|
||||
curl 8.14.1-2+deb13u5
|
||||
dash 0.5.12-12
|
||||
dbus 1.16.2-2
|
||||
dbus-bin 1.16.2-2
|
||||
dbus-daemon 1.16.2-2
|
||||
dbus-session-bus-common 1.16.2-2
|
||||
dbus-system-bus-common 1.16.2-2
|
||||
dbus-user-session 1.16.2-2
|
||||
debconf 1.5.91
|
||||
debconf-i18n 1.5.91
|
||||
debian-archive-keyring 2025.1
|
||||
debian-faq 12.2
|
||||
debianutils 5.23.2
|
||||
dhcpcd-base 1:10.1.0-11+deb13u4
|
||||
diffutils 1:3.10-4
|
||||
distro-info-data 0.66+deb13u2
|
||||
dmidecode 3.6-2
|
||||
doc-debian 11.3+nmu1
|
||||
docker-buildx-plugin 0.37.1-1~debian.13~trixie
|
||||
docker-ce 5:29.8.2-1~debian.13~trixie
|
||||
docker-ce-cli 5:29.8.2-1~debian.13~trixie
|
||||
docker-ce-rootless-extras 5:29.8.2-1~debian.13~trixie
|
||||
docker-compose-plugin 5.6.0-1~debian.13~trixie
|
||||
dpkg 1.22.22
|
||||
e2fsprogs 1.47.2-3+b12
|
||||
fdisk 2.41.5-0+deb13u1
|
||||
file 1:5.46-5
|
||||
findutils 4.10.0-3
|
||||
gcc-14-base 14.2.0-19
|
||||
gettext-base 0.23.1-2
|
||||
git 1:2.47.3-0+deb13u1
|
||||
git-man 1:2.47.3-0+deb13u1
|
||||
grep 3.11-4
|
||||
groff-base 1.23.0-9
|
||||
gzip 1.13-1+deb13u1
|
||||
hostname 3.25
|
||||
ifupdown 0.8.44+deb13u1
|
||||
ifupdown2 3.0.0-1.3
|
||||
inetutils-telnet 2:2.6-3+deb13u3
|
||||
init 1.69~deb13u1
|
||||
init-system-helpers 1.69~deb13u1
|
||||
iproute2 6.15.0-1
|
||||
iptables 1.8.11-2
|
||||
iputils-ping 3:20240905-3
|
||||
isc-dhcp-client 4.4.3-P1-8
|
||||
kmod 34.2-2
|
||||
krb5-locales 1.21.3-5+deb13u1
|
||||
less 668-1
|
||||
libacl1 2.3.2-2+b1
|
||||
libapparmor1 4.1.0-1
|
||||
libapt-pkg7.0 3.0.3
|
||||
libattr1 1:2.5.2-3
|
||||
libaudit-common 1:4.0.2-2+deb13u1
|
||||
libaudit1 1:4.0.2-2+deb13u1
|
||||
libblkid1 2.41.5-0+deb13u1
|
||||
libbpf1 1:1.5.0-3
|
||||
libbrotli1 1.1.0-2+b7
|
||||
libbsd0 0.12.2-2
|
||||
libbz2-1.0 1.0.8-6
|
||||
libc-bin 2.41-12+deb13u4
|
||||
libc-l10n 2.41-12+deb13u4
|
||||
libc6 2.41-12+deb13u4
|
||||
libcap-ng0 0.8.5-4+b1
|
||||
libcap2 1:2.75-10+deb13u1+b3
|
||||
libcap2-bin 1:2.75-10+deb13u1+b3
|
||||
libcbor0.10 0.10.2-2
|
||||
libcom-err2 1.47.2-3+b12
|
||||
libcrypt1 1:4.4.38-1
|
||||
libcurl3t64-gnutls 8.14.1-2+deb13u5
|
||||
libcurl4t64 8.14.1-2+deb13u5
|
||||
libdb5.3t64 5.3.28+dfsg2-9
|
||||
libdbus-1-3 1.16.2-2
|
||||
libdebconfclient0 0.280
|
||||
libedit2 3.1-20250104-1
|
||||
libelf1t64 0.192-4
|
||||
liberror-perl 0.17030-1
|
||||
libexpat1 2.8.3-1~deb13u1
|
||||
libext2fs2t64 1.47.2-3+b12
|
||||
libfdisk1 2.41.5-0+deb13u1
|
||||
libffi8 3.4.8-2
|
||||
libfido2-1 1.15.0-1+b1
|
||||
libfstrm0 0.6.1-1+b3
|
||||
libgcc-s1 14.2.0-19
|
||||
libgdbm-compat4t64 1.24-2
|
||||
libgdbm6t64 1.24-2
|
||||
libgmp10 2:6.3.0+dfsg-3
|
||||
libgnutls30t64 3.8.9-3+deb13u4
|
||||
libgssapi-krb5-2 1.21.3-5+deb13u1
|
||||
libhogweed6t64 3.10.1-1
|
||||
libicu76 76.1-4
|
||||
libidn2-0 2.3.8-2
|
||||
libip4tc2 1.8.11-2
|
||||
libip6tc2 1.8.11-2
|
||||
libjansson4 2.14-2+b3
|
||||
libjemalloc2 5.3.0-3
|
||||
libjson-c5 0.18+ds-1
|
||||
libk5crypto3 1.21.3-5+deb13u1
|
||||
libkeyutils1 1.6.3-6
|
||||
libkmod2 34.2-2
|
||||
libkrb5-3 1.21.3-5+deb13u1
|
||||
libkrb5support0 1.21.3-5+deb13u1
|
||||
liblastlog2-2 2.41.5-0+deb13u1
|
||||
libldap-common 2.6.10+dfsg-1
|
||||
libldap2 2.6.10+dfsg-1
|
||||
liblmdb0 0.9.31-1+b2
|
||||
liblocale-gettext-perl 1.07-7+b1
|
||||
liblockfile-bin 1.17-2
|
||||
liblsof0 4.99.4+dfsg-2
|
||||
liblz4-1 1.10.0-4
|
||||
liblzma5 5.8.1-1+deb13u1
|
||||
libmagic-mgc 1:5.46-5
|
||||
libmagic1t64 1:5.46-5
|
||||
libmaxminddb0 1.12.2-1
|
||||
libmd0 1.1.0-2+b1
|
||||
libmnl0 1.0.5-3
|
||||
libmount1 2.41.5-0+deb13u1
|
||||
libncursesw6 6.5+20250216-2
|
||||
libnetfilter-conntrack3 1.1.0-1
|
||||
libnettle8t64 3.10.1-1
|
||||
libnewt0.52 0.52.25-1
|
||||
libnfnetlink0 1.0.2-3
|
||||
libnftables1 1.1.3-1
|
||||
libnftnl11 1.2.9-1
|
||||
libnghttp2-14 1.64.0-1.1+deb13u1
|
||||
libnghttp3-9 1.8.0-1
|
||||
libngtcp2-16 1.11.0-1+deb13u1
|
||||
libngtcp2-crypto-gnutls8 1.11.0-1+deb13u1
|
||||
libnsl2 1.3.0-3+b3
|
||||
libnss-systemd 257.13-1~deb13u1
|
||||
libp11-kit0 0.25.5-3
|
||||
libpam-modules 1.7.0-5
|
||||
libpam-modules-bin 1.7.0-5
|
||||
libpam-runtime 1.7.0-5
|
||||
libpam-systemd 257.13-1~deb13u1
|
||||
libpam-wtmpdb 0.73.0-3+deb13u1
|
||||
libpam0g 1.7.0-5
|
||||
libpcre2-8-0 10.46-1~deb13u3
|
||||
libperl5.40 5.40.1-6+deb13u1
|
||||
libpipeline1 1.5.8-1
|
||||
libpopt0 1.19+dfsg-2
|
||||
libproc2-0 2:4.0.4-9
|
||||
libprotobuf-c1 1.5.1-1
|
||||
libpsl5t64 0.21.2-1.1+b1
|
||||
libpython3-stdlib 3.13.5-1
|
||||
libpython3.13-minimal 3.13.5-2+deb13u5
|
||||
libpython3.13-stdlib 3.13.5-2+deb13u5
|
||||
libreadline8t64 8.2-6
|
||||
librtmp1 2.4+20151223.gitfa8646d.1-2+b5
|
||||
libsasl2-2 2.1.28+dfsg1-9
|
||||
libsasl2-modules-db 2.1.28+dfsg1-9
|
||||
libseccomp2 2.6.0-2
|
||||
libselinux1 3.8.1-1
|
||||
libsemanage-common 3.8.1-1
|
||||
libsemanage2 3.8.1-1
|
||||
libsepol2 3.8.1-1
|
||||
libslang2 2.3.3-5+b2
|
||||
libsmartcols1 2.41.5-0+deb13u1
|
||||
libsqlite3-0 3.46.1-7+deb13u2
|
||||
libss2 1.47.2-3+b12
|
||||
libssh2-1t64 1.11.1-1+deb13u2
|
||||
libssl3t64 3.5.7-1~deb13u3
|
||||
libstdc++6 14.2.0-19
|
||||
libsystemd-shared 257.13-1~deb13u1
|
||||
libsystemd0 257.13-1~deb13u1
|
||||
libtasn1-6 4.20.0-2+deb13u1
|
||||
libtext-charwidth-perl 0.04-11+b4
|
||||
libtext-iconv-perl 1.7-8+b4
|
||||
libtext-wrapi18n-perl 0.06-10
|
||||
libtinfo6 6.5+20250216-2
|
||||
libtirpc-common 1.3.6+ds-1
|
||||
libtirpc3t64 1.3.6+ds-1
|
||||
libtlsrpt0 0.5.0rc1-2
|
||||
libuchardet0 0.0.8-1+b2
|
||||
libudev1 257.13-1~deb13u1
|
||||
libunistring5 1.3-2
|
||||
liburcu8t64 0.15.2-2
|
||||
libuuid1 2.41.5-0+deb13u1
|
||||
libuv1t64 1.50.0-2
|
||||
libwrap0 7.6.q-36
|
||||
libwtmpdb0 0.73.0-3+deb13u1
|
||||
libxml2 2.12.7+dfsg+really2.9.14-2.1+deb13u3
|
||||
libxtables12 1.8.11-2
|
||||
libxxhash0 0.8.3-2
|
||||
libzstd1 1.5.7+dfsg-1
|
||||
linux-sysctl-defaults 4.12.1
|
||||
locales 2.41-12+deb13u4
|
||||
login 1:4.16.0-2+really2.41.5-0+deb13u1
|
||||
login.defs 1:4.17.4-2
|
||||
logrotate 3.22.0-1
|
||||
logsave 1.47.2-3+b12
|
||||
lsof 4.99.4+dfsg-2
|
||||
man-db 2.13.1-1
|
||||
manpages 6.9.1-1
|
||||
mawk 1.3.4.20250131-1
|
||||
media-types 13.0.0
|
||||
mount 2.41.5-0+deb13u1
|
||||
nano 8.4-1+deb13u1
|
||||
ncurses-base 6.5+20250216-2
|
||||
ncurses-bin 6.5+20250216-2
|
||||
ncurses-term 6.5+20250216-2
|
||||
netbase 6.5
|
||||
netcat-traditional 1.10-50
|
||||
nftables 1.1.3-1
|
||||
openssh-client 1:10.0p1-7+deb13u4
|
||||
openssh-server 1:10.0p1-7+deb13u4
|
||||
openssh-sftp-server 1:10.0p1-7+deb13u4
|
||||
openssl 3.5.7-1~deb13u3
|
||||
openssl-provider-legacy 3.5.7-1~deb13u3
|
||||
passwd 1:4.17.4-2
|
||||
patch 2.8-2
|
||||
perl 5.40.1-6+deb13u1
|
||||
perl-base 5.40.1-6+deb13u1
|
||||
perl-modules-5.40 5.40.1-6+deb13u1
|
||||
pigz 2.8-1
|
||||
postfix 3.10.13-0+deb13u1
|
||||
procps 2:4.0.4-9
|
||||
python-apt-common 3.0.0
|
||||
python3 3.13.5-1
|
||||
python3-apt 3.0.0
|
||||
python3-certifi 2025.1.31+ds-1
|
||||
python3-chardet 5.2.0+dfsg-2
|
||||
python3-charset-normalizer 3.4.2-1
|
||||
python3-debconf 1.5.91
|
||||
python3-debian 1.0.1
|
||||
python3-debianbts 4.1.1
|
||||
python3-idna 3.10-1+deb13u1
|
||||
python3-minimal 3.13.5-1
|
||||
python3-reportbug 13.2.0
|
||||
python3-requests 2.32.3+dfsg-5+deb13u1
|
||||
python3-urllib3 2.3.0-3+deb13u2
|
||||
python3.13 3.13.5-2+deb13u5
|
||||
python3.13-minimal 3.13.5-2+deb13u5
|
||||
readline-common 8.2-6
|
||||
reportbug 13.2.0
|
||||
runit-helper 2.16.4
|
||||
sed 4.9-2+deb13u1
|
||||
sensible-utils 0.0.25
|
||||
sqv 1.3.0-3+b2
|
||||
ssh 1:10.0p1-7+deb13u4
|
||||
systemd 257.13-1~deb13u1
|
||||
systemd-sysv 257.13-1~deb13u1
|
||||
systemd-timesyncd 257.13-1~deb13u1
|
||||
sysvinit-utils 3.14-4
|
||||
tar 1.35+dfsg-3.1
|
||||
traceroute 1:2.1.6-1
|
||||
tzdata 2026c-0+deb13u1
|
||||
ucf 3.0052
|
||||
udev 257.13-1~deb13u1
|
||||
util-linux 2.41.5-0+deb13u1
|
||||
util-linux-extra 2.41.5-0+deb13u1
|
||||
vim-common 2:9.1.1230-2
|
||||
vim-tiny 2:9.1.1230-2
|
||||
wamerican 2020.12.07-4
|
||||
wget 1.25.0-2
|
||||
whiptail 0.52.25-1
|
||||
wtmpdb 0.73.0-3+deb13u1
|
||||
xz-utils 5.8.1-1+deb13u1
|
||||
zlib1g 1:1.3.dfsg+really1.3.1-1+b1
|
||||
@@ -0,0 +1,2 @@
|
||||
=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub ===
|
||||
time=2026-10-05T08:13:34.060+02:00 level=INFO msg="osupdate: START" run=20261005T061334Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261005T061334Z
|
||||
@@ -0,0 +1,31 @@
|
||||
### 05:44:03 before
|
||||
bind9-dnsutils 1:9.20.29-1~deb13u1
|
||||
bind9-host 1:9.20.29-1~deb13u1
|
||||
bind9-libs:amd64 1:9.20.29-1~deb13u1
|
||||
libpcre2-8-0:amd64 10.46-1~deb13u3
|
||||
libpython3.13-minimal:amd64 3.13.5-2+deb13u5
|
||||
libpython3.13-stdlib:amd64 3.13.5-2+deb13u5
|
||||
libssh2-1t64:amd64 1.11.1-1+deb13u2
|
||||
libssl3t64:amd64 3.5.7-1~deb13u3
|
||||
libxml2:amd64 2.12.7+dfsg+really2.9.14-2.1+deb13u3
|
||||
openssl 3.5.7-1~deb13u3
|
||||
openssl-provider-legacy 3.5.7-1~deb13u3
|
||||
python3.13 3.13.5-2+deb13u5
|
||||
python3.13-minimal 3.13.5-2+deb13u5
|
||||
279
|
||||
### 05:44:05 rollback 1 (13 packages, simulated clean before)
|
||||
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
|
||||
audit rc=0
|
||||
bind9-dnsutils 1:9.20.26-1~deb13u1
|
||||
bind9-host 1:9.20.26-1~deb13u1
|
||||
bind9-libs:amd64 1:9.20.26-1~deb13u1
|
||||
libpcre2-8-0:amd64 10.46-1~deb13u2
|
||||
libpython3.13-minimal:amd64 3.13.5-2+deb13u4
|
||||
libpython3.13-stdlib:amd64 3.13.5-2+deb13u4
|
||||
libssh2-1t64:amd64 1.11.1-1+deb13u1
|
||||
libssl3t64:amd64 3.5.7-1~deb13u2
|
||||
libxml2:amd64 2.12.7+dfsg+really2.9.14-2.1+deb13u1
|
||||
openssl 3.5.7-1~deb13u2
|
||||
openssl-provider-legacy 3.5.7-1~deb13u2
|
||||
python3.13 3.13.5-2+deb13u4
|
||||
python3.13-minimal 3.13.5-2+deb13u4
|
||||
@@ -0,0 +1,73 @@
|
||||
# Part F — a box that is OFF every night (Tester 2) — read-only spike, 2026-10-05
|
||||
|
||||
Collected by a read-only helper session (hub DB copy at ~05:11 UTC, non-secret columns only, copy shredded; hub
|
||||
log; source at felhom.eu 7221ee5c, controller 69e9145, agent c8d12f1). Two claims re-checked by the main session
|
||||
against source: the 05:00 deadline check skips `down` customers (`hub/internal/monitor/deadline.go` ~360), and the
|
||||
controller's daily scheduler always picks the NEXT future time (`controller/internal/scheduler/scheduler.go`
|
||||
`nextDailyRun`, ~389-408). Hub DB times are UTC; the hub log prints Budapest time.
|
||||
|
||||
## Q1 — Tester 2's record (customer `Tester-2`, host `Tester-2-be8404`)
|
||||
|
||||
Customer created 2026-09-30 (language en); host created 2026-10-04 16:07 UTC (agent 0.142.0, controller 0.292.0);
|
||||
no apps installed. Online twice on 2026-10-04: ~15:54–17:13 UTC, and 18:03:42–18:06:24 UTC (last controller report).
|
||||
Nothing since. One evening of data: no daytime pattern can be established.
|
||||
|
||||
| Leg | Result | Evidence |
|
||||
|---|---|---|
|
||||
| Whole-guest backup, local | ran once | 16:21:10 UTC, success |
|
||||
| Whole-guest backup, PBS | ran once | 16:31:13 UTC, verify ok |
|
||||
| OS update | ran once | `os_reports` 40/41, trigger `night`, 16:24/16:25 UTC, 49 + 106 packages, healthy |
|
||||
| DB dump, tier-2 copy | never | — |
|
||||
| Off-site copy | never | escrow `pending` |
|
||||
| App updates | none | no apps |
|
||||
| Restore-test | never | `restore_tests: []` in every report |
|
||||
|
||||
Events: 17:58 `host_stale`, 18:01 `node_stale`, 18:05–06 recovered; 18:51 stale again; 19:36 `node_down` +
|
||||
`host_down` (error) — the household was mailed "Your server cannot be reached." at 19:36:42 UTC. Hub log
|
||||
2026-10-05 05:00 Budapest: "Deadline check: … 0 backup missed … 1 skipped (down)".
|
||||
|
||||
**What fires if it stays off every night:**
|
||||
- every evening `node_stale`/`host_stale` after 45 min; every night `node_down`/`host_down` after 90 min — the
|
||||
household gets a `node_down` mail each night and a "reachable again" mail each morning (the 6 h cooldown does not
|
||||
stop a daily repeat);
|
||||
- **never** `expected_backup_missed` / `expected_dbdump_missed` while it is down at 05:00 (the deadline check skips
|
||||
down customers);
|
||||
- `restore_test_stale` ~2026-10-11 16:13 UTC (local tier, 7 d) and ~2026-10-16 (PBS, 12 d) — not skipped for down boxes;
|
||||
- `os_update_stale` 2026-10-11 16:24 UTC unless a primary backup happens before;
|
||||
- `offsite_stale` (48 h) once escrow is completed — the off-site leg runs only at night.
|
||||
|
||||
**What the household sees:** the backups page shows "Esedékes / Due" or "Naprakész / Up to date" and "No successful
|
||||
backup yet" per tier. No text says the server must stay on at night or that a night was missed because it was off.
|
||||
|
||||
## Q2 — does anything catch up when the box comes back?
|
||||
|
||||
- **DB dump, tier-2, off-site, the chained app-update leg, the controller's 04:30 self-update:** NO. The daily
|
||||
scheduler always picks the next future time; `LastRun` is in memory only and never consulted. (A hub floor still
|
||||
reaches the box by day, after any report.)
|
||||
- **Whole-guest backup:** PARTIAL. The controller asks every 5 min; due = newest archive ≥ 24 h old (read from
|
||||
storage, survives a reboot); runs only in the window [W+2h, W+6h) (04:30–08:30 for W 02:30), OR by the safety valve
|
||||
(never backed up, or last > 48 h). A box on only in the afternoon/evening gets about one every 2 days.
|
||||
- **OS leg:** 90 s after any successful primary backup, ≥ 20 h apart — so it follows the valve, by day, under the
|
||||
`night` label.
|
||||
- **Restore-test:** checks every 6 h, the timer restarts at each agent start, no check at start — a box never on for
|
||||
6 h in a row never runs one.
|
||||
- Not verified: a SUSPENDED (not powered-off) laptop would probably run the daily jobs late rather than skip them
|
||||
(reasoned from Go timers, not tested).
|
||||
|
||||
## Q3 — architecture
|
||||
|
||||
**No architecture document covers a box that is not always on.** Partial mentions: `11-os-updates.md` ("a box off for
|
||||
a weekend does not alarm"); `07-backup-architecture.md` (the night chain and the [W+2h, W+6h) gate — no safety valve,
|
||||
no catch-up); `08-alarm-ladder.md` (45/90 min); `where-felhom-stands.yaml` `fail.expected-downtime` (missing, R-285).
|
||||
The intent "a box powered on only outside its window never starves" exists only as controller code comments
|
||||
(`quiesce.go`).
|
||||
|
||||
## Q4 — register
|
||||
|
||||
R-285 (open, P4: no notion of expected downtime), R-30 (open, P3), R-195 / R-321 (closed: skips keyed off the
|
||||
wrong fact), R-86 / R-100 (closed). **No row covers a box that is off every night.**
|
||||
|
||||
## Not determined
|
||||
|
||||
The daytime online pattern; Tester 2's actual W (on the box, which is off; 02:30 assumed); the type of signed job 53
|
||||
(queued 18:35 UTC, after the box went off — the 2026-10-04 agent update, per the night's records).
|
||||
@@ -0,0 +1,6 @@
|
||||
demo-felhom-8363b5 Demo Ügyfél ring 0 → normal updates ON switch off running 9.2.2 7.0.2-6-pve 7.0.2-6-pve 13.7 ring0-20261005T055135Z 80 0 none no 10 s no 0 armed 0.144.1 13.7 ring0-20261005T055135Z 0 7 29.8.2 2.3.6-1~debian.13~trixie on ring0-20261005T055135Z 29 min ago · nothing · 12 s demo-hp-bb76ea Demo HP partial: no runni
|
||||
|
||||
demo-hp-bb76ea Demo HP partial: no running customer guest ring 0 → normal updates ON switch off unknown 9.2.2 unknown unknown unknown ring0-20261005T061858Z 78 0 unknown no unknown no unknown not installed 0.144.1 unknown ring0-20261005T061858Z 0 7 unknown unknown unknown ring0-20261005T061858Z 3 min ago · nothing · 19 s tester-
|
||||
|
||||
tester-1-d70be4 Tester 1 ring 1 → demo updates ON switch off running 9.2.2 7.0.2-6-pve 7.0.2-6-pve 13.5 — 185 106 none no 10 s no 0 armed 0.144.1 13.6 — 49 0 29.8.2 2.3.6-1~debian.13~trixie on — 10 h ago · nothing · 16 s Amber: worth a look. Red: an operator alarm fires (`08` §6.3). "unknown": the box could not read the value —
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
2026-10-05T05:52:25Z ('Tester-2-be8404', '0.142.0', '2026-10-04 18:05:48')
|
||||
2026-10-05T06:24:47Z ('Tester-2-be8404', '0.142.0', '2026-10-04 18:05:48')
|
||||
@@ -26,6 +26,19 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-05 (day) — the night's fixes: off-site clean-up guard, first-install image race, R8 download, a killed pass's report (controller v0.294.0, agent v0.144.0 + v0.144.1, golden 0.294.0; rulings 100–103, CC decisions 104–108)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-867** | **The off-site clean-up guard refused honest 7-day retention and mailed an error every window.** Controller v0.294.0: the guard's "young" line is keep-daily CALENDAR days, built from the same constants as the policy (decision 104); every other refusal kept. Tests run restic 0.14.0's policy itself (proven identical to the binary over 92 snapshots), 5 red-proofs. Live by hand: demo-felhom window 5 16 → 14, demo-hp window 6 145 → 127 — exactly the predicted snapshots; hub rows `pruned`, no event, no mail, key files clean. **Reasoning kept: a line that can sit inside the keep window is a line that refuses the honest case — derive it from the policy, never pick an age.** Full text: `git show 7221ee5c:documentation/backlog/OPEN-ITEMS.md`. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partA/` |
|
||||
| **R-95** | **The box could delete its own off-site history.** Append-only key since 2026-10-03 (decisions 68–69); the last open item — a clean-up window that actually removes snapshots — was observed 2026-10-05 on both demo boxes (R-867's live windows, opened by the operator's one-shot grant; the dated check's four conditions all hold). The unattended weekly window is the same code path and is due ~2026-10-11/12; it is not separately re-checked (the DUE-CHECKS entry is removed with this row). Residual: R-822 (an add-only attacker steering older keeps). Full text: `git show 7221ee5c:documentation/backlog/OPEN-ITEMS.md`. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partA/`; `audits/offsite-lock-build-2026-10-03/` |
|
||||
| **R-863** | **A new box's first app install could fail: the one-time image clean-up deleted the image compose had just pulled.** Controller v0.294.0: every compose command that can pull holds a shared lock (`dockerexec.BeginImageWork`); a clean-up pass takes it exclusively without waiting and otherwise does not run; the one-time pass is retried every 2 min and writes its marker only after a pass that ran (decision 105). Live on 9202: the clean-up fired at minute 3 (05:31:36 UTC) inside BookStack's 35.5 s install, skipped itself, BookStack installed first time; the retry at 05:33:36 ran and kept everything; teardown through the product. The update path was already guarded (`Updating`); restore/undo were exposed in principle and now hold the same lock. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partB/` |
|
||||
| **R-864** | **A failed compose logged the head of stderr (pull progress) and cut the reason.** Controller v0.294.0 `tailStr` (rune-safe), pinned with the night's real first line. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partB/red-proofs.txt` |
|
||||
| **R-869** | **The move-aside log line printed an empty destination.** Controller v0.294.0: assigned before the log line; test + red-proof. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partD/r869-red-proof.txt` |
|
||||
| **R-865** | **R8 measured every download as 0 B (`--print-uris` with `-s` prints no URIs).** Agent v0.144.0: no `-s`; the fake answers like real apt (verbatim 9202 output), 2 tests, red-proof. Live: the installed wrapper's `download_bytes` on demo-hp read **12 802 456 B** for 13 pending upgrades (0 before), nothing installed. A live R8 REFUSAL line was not produced: it needs < 500 MB free on demo-hp's guest, i.e. 28.5 GB written into a thin pool with 18.7 GB free — it would have stopped every guest. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partC/` |
|
||||
| **R-866** | **The debug OS pass could not run with the hub away.** Agent v0.144.0: the daemon saves the hub's block; the selftest falls back to it and says `block=SAVED(<time>; hub unreachable: …)` in its header (decision 107). Live on demo-felhom with the hub blackholed: pass ran (`nothing` ×3); its reports, kept on disk, reached the hub at the next agent start. | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partD/r866-*` |
|
||||
| **R-868** | **An OS pass whose agent was killed installed but never reported.** Agent v0.144.0: the wrapper keeps its report beside the plan; the agent sends kept copies and deletes them (decision 106). **Measured live NOT to work** (05:45 UTC, demo-hp): the wrapper died on a broken stderr pipe before saving. Agent v0.144.1 (decision 108): the wrapper survives a dead reader; the agent looks again every 5 min. Live again (06:04 UTC, demo-hp, the A5 shape): wrapper `DONE upgraded=13`, copy kept, the hub got ONE `applied` report (13 packages) at 06:09:05 by the 5-minute look. **Reasoning kept: a root helper whose reader can die must never write unguarded to that reader — the log is not the outcome.** | CLOSED 2026-10-05 — FIXED | `audits/night-fixes-2026-10-05/partD/` |
|
||||
|
||||
## 2026-10-04 (night) — the config bundle, test approvals, the Docker-socket self-heal (agent v0.143.0, hub v0.133.0, controller v0.293.0, installer 1.31.0; rulings 96–99)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
@@ -126,11 +126,10 @@ stopping line that lies.
|
||||
| **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
|
||||
| **R-725** | Install & onboarding | P4 | **[P3-LOW] Small copy slips on the first-hour path.** MEASURED 2026-09-29: the self-bind PAGE says the passphrase was received „a beállításkor" while the mail and console say „a Felhom üzemeltetőjétől" (R-497 unified the mail and console, not the page); the recovery-code wizard addresses the household formally („Írja fel", „adja meg") while every other screen says „te"; the console's linked banner ends „a doboz össze van kötve. V" (a stray glyph); a gated app answers a phone app's API call with English JSON „this app is waiting for its first setup" (the browser gets the Hungarian gate page). **FIXED 2026-09-30:** the recovery wizard speaks „te" (controller v0.283.0; formal ceiling 18 → 17); the bind page says „a Felhom üzemeltetőjétől kaptál" (hub v0.126.0). **NARROWED — remaining:** the console's stray „V" (the installer/agent's banner, not these repos' text); the gate's English JSON to a phone app (the app shows its own error; left, deliberately); and the expired bind page still says „kérj újat az ügyfélszolgálattól" ABOVE the new „Új linket kérek" button (hub copy, next hub release). | **NARROWED — three small copy items; owner: CC** **Re-ranked 2026-10-03: P3→P4: three small copy slips left; nothing blocks the household.** | — | — | CC |
|
||||
|
||||
## Apps & catalog — 36 rows (P3 15, P4 21)
|
||||
## Apps & catalog — 34 rows (P3 15, P4 20)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-863** | Apps & catalog | P2 | **A brand-new box's first app install can fail because the controller's one-time image clean-up deletes the image the install is downloading.** MEASURED 2026-10-04 night drill on the fresh Tester 1 box (controller v0.293.0): BookStack deploy requested 19:44:39 UTC; `RunImageRetentionOnce` (3 min after the controller's first start, `cmd/controller/main.go` ~1730) deleted `mariadb:<none>` (dfff46ef3f9d, 334 MB, "no container, installed app or undo names it") at 19:45:04; `docker compose up -d` failed at 19:45:05 (exit 1); the household got „Alkalmazás telepítése nem sikerült: BookStack". A second press at 19:55:59 ran in 50 s. The window is exactly when a new household installs its first apps. Fix direction: the clean-up skips every image an install, update or restore in flight names (or waits for them), and a pull-by-digest image is not "unnamed". `audits/night-2026-10-04/tester1/t8-bookstack-failed-image-retention.txt` | **READY — owner: CC** | — | a fix + a test that runs the clean-up during a deploy | CC |
|
||||
| **R-489** | Apps & catalog | P3 | **[P3-LOW] `POST /api/stacks/{name}/remove` reports `volumes_removed: null` over named volumes it DID remove.** MEASURED 2026-09-13 on demo-hp five times (gokapi, actualbudget, adventurelog ×2, glance): `docker compose down --volumes` removed the app's named volumes (`docker volume ls` count 2 → 0) and the response carried `"volumes_removed":null`. The customer's confirmation dialog therefore cannot say what it deleted. Split out of R-474 (closed in v0.240.0 for the backups half). **Fix shape:** list the volumes before `down --volumes`, diff after, and report the difference (`[]` when none, never `null`). **PARTLY SHIPPED in v0.242.0 (`d698ce3`), measured live on 9202 the same night:** the difference is computed and a fresh compose-created volume IS reported (`["opengist_opengist_data"]`), but the listing filters on the compose project LABEL and a volume recreated by a unit restore (`docker volume create <name>`, `restore.go:154`) carries no labels — compose still removes it and the response says `[]` (`audits/v0242-2026-09-14/19-R489-cause.txt`). **Remaining fix:** list by the `<project>_` name prefix as well (union), or label the recreated volume as compose would. | **READY — rank P3-LOW; owner: CC (residual)** | — | — | CC |
|
||||
| **R-498** | Apps & catalog | P3 | **[P3-LOW] The „Első lépések" on 52 of 53 app pages tell a customer to open a literal `wiki.DOMAIN` — the placeholder is never filled in.** MEASURED 2026-09-14 on a fresh 0.242.0 box (drill step 5): BookStack's app page renders „Nyisd meg a **wiki.DOMAIN** címet a böngészőben", PrivateBin's „Nyisd meg a **paste.DOMAIN** címet". `grep -rl '\.DOMAIN c[íi]m' app-catalog-felhom.eu/templates/*/.felhom.yml` → **52 of 53** templates carry it in `first_steps`; the controller renders the string as text (`internal/stacks/metadata.go`). A stranger reading their first instruction meets a word that is not an address. **Fix shape:** substitute the stack's real `SUBDOMAIN.DOMAIN` at render time (one place in the controller), with a render test per template that fails on a literal `DOMAIN`. | **READY — rank P3-LOW; owner: CC** | — | — | CC |
|
||||
| **R-562** | Apps & catalog | P3 | **[P3-LOW] Dates and sizes are not formatted for any locale — and the Hungarian pages disagree with themselves.** FOUND 2026-09-17 by the i18n inventory §2.8: the two template date layouts differ (`2006. 01. 02. 15:04` Hungarian vs `2006-01-02 15:04` ISO); 10 layout literals in `internal/web` Go and 25 elsewhere pick formats ad hoc; sizes print a decimal POINT (`%.1f GB`, 4 helpers) where Hungarian uses a comma; `timeAgo`/`nextRunLabel`/`pruneLabel` produce Hungarian words outside the three converted pages. Not changed by v0.247.0 (Hungarian bytes are frozen by the parity rule). **Fix shape:** one date and one size formatter per language in `internal/i18n`, the Hungarian output deliberately changed in ONE reviewed release with the parity fixtures re-captured for that release only and the change named in its CHANGELOG. Needs an operator word on the Hungarian format (comma, date style). | **READY - rank P3-LOW; owner: CC** | — | — | CC |
|
||||
@@ -146,7 +145,6 @@ stopping line that lies.
|
||||
| **R-758** | Apps & catalog | P3 | **[P3-LOW] Eight templates declare a `mem_limit` that is not the sum of their compose limits, and no gate checks it.** FOUND 2026-10-01 by `scripts/onboarding_gaps.py` (the new-app checklist's gap page, row 5.3): adventurelog 384M vs 896M, bookstack 512M vs 768M, calcom 768M vs 1792M, claper 384M vs 640M, kimai 384M vs 640M, nextcloud 1024M vs 1664M, outline 768M vs 1152M, zipline 512M vs 768M — every one UNDER the sum, so the deploy screen and the box's capacity figure (`09` decision 22) read less memory than the app may take. REUSE.md §2 says `mem_limit` = the sum. The compose limits are what Docker enforces, so no app is starved by this; the figure the household and the capacity check read is wrong. **Needs:** the eight figures corrected (a template change: needs its own session, no image move), and a static gate (`--fast`) with a decoy, so a new app cannot repeat it. `app-catalog-felhom.eu/onboarding/EXISTING-APPS-GAPS.md` | **READY — rank P3-LOW; owner: CC (catalog)** | — | — | CC |
|
||||
| **R-762** | Apps & catalog | P3 | **[P2-MEDIUM] wger serves no CSS or JavaScript and no uploaded photo: every static file and every `/media/` file answers 404.** MEASURED 2026-10-01 on 9202 (drill catalog, the live template `82fff32`, wger 2.7), found by checklist rows 1.7 and 2.8: the login page links `/static/css/workout-manager.css`, `/static/bootstrap-compiled.css` — both 404 through traefik; the static root inside the container is empty (4 KB). A progress photo posted to `/api/v2/gallery/` answered 201 and the file is on the media volume, but `GET /media/gallery/…png` answers 404 signed in, without a session, and straight at the app inside the container. **Cause, read in the image:** the entrypoint runs `collectstatic` only when `DJANGO_DEBUG == "False"` and the template sets no `DJANGO_DEBUG`; and wger serves `/media/` only in development (`urls.py:393` „served like this during development only”) — upstream's production setup puts nginx in front for `/static` and `/media`. So the household gets an unstyled app and photos that never show. The same lines stand since the template was written (the 2026-09-29 template too). Not checked: whether any box runs wger (on 2026-09-30 none reported to the hub). **Needs:** `DJANGO_DEBUG=False` (collectstatic) and something that serves `/static` + `/media` (upstream's nginx sidecar, or the gunicorn switch of R-755 plus a static server), proven on the bench and on 9202 with a page that loads its CSS and a photo read back. Owner decides together with R-755 (same server question). `audits/new-app-checklist-2026-10-01/C/C8-signup-guest-media-static.txt`, `C/C4-seed-photo-size.txt` **-- 2026-10-01 (operator):** wger is `lifecycle: hidden` until this and its twin are fixed (catalog `55b8c8a`; read back on 9202: not on the app list, mealie control present). | **READY — rank P2-MEDIUM; owner: CC (catalog)** **Re-ranked 2026-10-03: P2→P3: wger is hidden from installs and no box runs it; needed only before it is offered again.** | — | — | CC |
|
||||
| **R-774** | Apps & catalog | P3 | **[P3-LOW] Two things the new apps' pages do not show yet: Karakeep's mail-ON path is unproven, and its official phone app reports crashes to its makers.** READ/MEASURED 2026-10-01: Karakeep's `smtp_mapping` (plaintext :2526, as Cal.com) was proven only with mail OFF (a fresh install boots) — 9202 has no hub, so the relay cannot be exercised there; the official mobile app ships Sentry crash reporting with a hard-coded DSN (`apps/mobile/app/_layout.tsx`, FIT.md). **Needs:** one password-reset mail from Karakeep on a hub-enabled box (demo-hp 9201, a throwaway install); and a sentence on the page about the phone app (copy, freeze). `audits/new-apps-2026-10-01/bench/karakeep-mail-off-boot.txt` | **READY — rank P3-LOW; owner: CC (catalog)** | — | — | CC |
|
||||
| **R-864** | Apps & catalog | P4 | **A failed `docker compose up` logs the FIRST lines of stderr (pull progress) and cuts the actual error.** 2026-10-04 night drill: BookStack's failure logged `stderr: Image … Pulling` + layer lines, then `5...` — the reason itself was lost; it had to be reconstructed from the image clean-up's log (R-863). Fix direction: keep the TAIL of stderr (the error is last). | **READY — owner: CC** | — | — | CC |
|
||||
| **R-76** | Apps & catalog | P4 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-26, size S, roadmap state `idea (surfaced by the R-75 spike, 2026-07-26)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say t | CC |
|
||||
| **R-284** | Apps & catalog | P4 | **„A kiválasztott tárhely majdnem megtelt." on a store that is 93 % FREE — an apparent inverted threshold.** Calibre-Web's deploy page rendered `<option value="/mnt/sys_drive" data-free-percent="93">` alongside „Tárhely (sys_drive) — **64.2 GB szabad**" and the warning „A kiválasztott tárhely majdnem megtelt." 93 % free read as 93 % used is the obvious candidate, and `checkStorageSpace(this)` is the function to look at. **Not confirmed by reading the code** — reported as measured output only. A capacity warning that cries wolf on an empty disk is one a customer learns to click past | **VERIFY** (2026-10-03 triage: Probably not a defect: the warning is hidden by default and shown only when free space is below 20%; the measurement likely read raw HTML without running scripts — felhom-controller/controller/internal/web/templates/deploy.html:622,808) — **READY (XS) — NEW 2026-08-09** | — | Check `checkStorageSpace`'s comparison against `data-free-percent`; add a render test per branch | CC |
|
||||
| **R-527** | Apps & catalog | P4 | **[P3-LOW] The catalog flag `locked_after_deploy` is read by no controller code — every setting is read-only after install whatever the catalog says.** FOUND 2026-09-15: `stacks/metadata.go` parses it; `grep -rn LockedAfterDeploy` finds no reader; `deploy.html` renders „Az alábbi beállítások csak olvashatók" for every field. Recorded as the design in `02-controller-module-map.md`; the flag is a seam never wired. **Fix shape:** remove the flag from the catalog, or wire an editable-after-install allow-list (a bigger change). | **VERIFY** (2026-10-03 triage: the row says nothing reads the flag, but deploy.go reads LockedAfterDeploy into LockedFields; the row may be stale or describe only the page — felhom-controller/controller/internal/stacks/deploy.go:400,1348-1371) — **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3->P4: every field is already read-only; no harm.** | — | — | CC |
|
||||
@@ -193,13 +191,11 @@ stopping line that lies.
|
||||
| **R-687** | App updates | P4 | **[P3-LOW] Part 7's live proof has four gaps a scratch box cannot close, and one observability gap.** (1) **W+5h reached with steps left** is proven by unit test only (`TestLeg_NoStepAtOrAfterW5h`) — the leg starts at W+105m and would need a 3-hour leg live; (2) **the off-site leg FAILING** before the update leg: 9202 has no off-site tier, so only the "no target" path ran live — failure and panic are `TestChainUpdateLeg_EveryPath`; (3) **a `files_may_change` step WITHOUT a whole copy**: both drill apps given the mark (wishlist, navidrome, romm) turned out whole on 9202 by the backup side's truth table (why, per app, is not logged — see the observability gap), so only "with a copy" ran live; (4) **the full-system gate waiting** cannot run on 9202 (no agent), and **did not occur on the demo boxes' real night either** (2026-09-25: both legs ended by 04:19, before the gate opened at 04:30, and no whole-box backup was due on either box) — unit + red-proof only (`TestD20_GateWaitsForTheLeg`). **Also cosmetic:** a leg with no steps reports `"steps": null` to the hub, not `[]`. **Observability:** when the leg TAKES a `files_may_change` step it does not log which whole copy allowed it (only the skip says why). `audits/night-2026-09-25/C/` **-- NARROWED 2026-09-25 (controller v0.273.0):** the cosmetic `"steps": null` → `[]` and the taken `files_may_change` step's missing log line are FIXED (red-proofed, `audits/night-2026-09-26/F/`). Items (1)–(4) stay; (4) did not occur on 2026-09-25 either (demo-felhom's whole-box backup ran at 07:29, three hours after its leg; demo-hp had none due). **-- 2026-09-28 (night 27/28):** (4) did not occur again — on demo-hp the leg ended 04:23:54 and the whole-guest backup began 04:37:06, after the gate opened at 04:30; demo-felhom's backup ran at 07:36 (`audits/evidence-golden-0276-2026-09-28/phaseD2-night-read.txt`). **-- 2026-09-30 (by day, demo-hp 9201): item (4) PROVEN LIVE.** The night chain pressed by hand, the window moved to W = now − 2h05m the moment the leg started, `quiesce.poll_interval` 1m: `[quiesce] full-system backup due and inside its window, but the automatic update leg is running … deferring` at 11:35:11 and 11:36:11 UTC while bookstack (55.1 s) and kimai (75.1 s) stepped; the leg's end line at 11:36:29; the backup quiesced at 11:37:11 (the first poll after), job done 11:47:19, the agent's `backup: completed` 9.98 GB. Config and window put back and read back (`audits/pg-last-six-2026-09-30/C/`). **Found, cosmetic, manual chain only:** the deferral names the moved window's W+5h (16:29) while the manual leg's own deadline was its start + the leg length (16:49). | **OPEN — P3, gaps (1)–(3) + the manual-chain deferral text; item (4) proven live 2026-09-30; owner: CC** **Re-ranked 2026-10-03: P3→P4: the gaps are covered by unit tests; left is live-proof completeness and one log text.** | — | — | CC |
|
||||
| **R-734** | App updates | P4 | **[P3-LOW] The harness marks immich `files_may_change` because immich rewrites six 13-byte `.immich` folder markers at every start.** MEASURED 2026-09-30 on the bench (v3.2.2 → v3.2.4): the bind-tree hash of `appdata/immich` changed; the only changed files were `{encoded-video,library,backups,profile,thumbs,upload}/.immich`, rewritten at each start — no household file. The mark is honest by the harness's rule and the ladder writer copies it (never edited by hand), so immich's v3.2.4 night step needs a fresh WHOLE copy (decision 13); on a box without one the night leg skips it and a person presses. The 2026-09-23 immich entry did not carry it (`files_changed []`). **Needs:** a decision whether app-owned marker files are excluded from the file hash (a per-template ignore list, or a size/name rule), or the mark stays. **-- 2026-09-30 (evening):** the harness now NAMES the files behind the mark (`files_changed_detail`, catalog `5b1972b`); on immich's step `0b82…` re-proof it named exactly the six `.immich` markers again. calibre-web's step (v4.0.6 → v4.0.8) carries the mark too, and the named files are its LIBRARY DATABASE: `media/books/metadata.db`, `metadata.db-shm`, `metadata.db-wal` changed; the book file did not (bench names re-run, `A/calibre-names/`). That is household data (the template's backup class `mandatory` holds DB and books as one unit), so the mark is right there and the night leg takes that step only with a fresh whole copy. On 9202 the same step changed no file in that folder (per-file hashes before/after) — not explained. `audits/more-night-apps-2026-09-30/` | **READY — rank P3-LOW; owner: CC (harness); the rule change needs a word** **Re-ranked 2026-10-03: P3→P4: the effect is an update that waits for a press; no data risk.** | — | — | CC + operator |
|
||||
|
||||
## Backup & restore — 54 rows (P2 10, P3 23, P4 21)
|
||||
## Backup & restore — 53 rows (P2 9, P3 24, P4 20)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-867** | Backup & restore | P2 | **The off-site clean-up's fake-snapshot guard refuses honest 7-day retention: every weekly window on a box with more than 7 nightly snapshots refuses, deletes nothing, and mails the operator an error.** MEASURED 2026-10-05 02:15 UTC on demo-felhom (controller v0.293.0): `clean-up window 3: the fake-snapshot guard REFUSED — nothing deleted: the policy would remove snapshot 343d57a5 from 2026-09-28T02:15:05Z — younger than 8 days and not superseded the same day, which honest retention never does (R-822)`; operator mail `offsite_prune_guard_refused` (error) sent. The snapshot was 7 days + 5 s old — exactly the one a keep-7-dailies policy drops at the same hour each night. So the guard's 8-day line sits inside honest retention: pruning never happens (the repository only grows) and a true-looking error mail arrives every window — the cry-wolf shape R-824 closed for same-day copies. Fix direction: the guard's age line must exceed the policy's own keep-window (e.g. keep-daily N → refuse only younger than N days minus a margin, derived from the same constant), pinned by a test that runs the real policy over 8 nightly snapshots. `audits/night-2026-10-04/night/n4-demo-felhom-prune-guard.txt` | **READY — owner: CC** | — | fix + the 8-night test | CC |
|
||||
| **R-32** | Backup & restore | P2 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC |
|
||||
| **R-95** | Backup & restore | P2 | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **NARROWED — the box can no longer delete (2026-10-03); weekly windows ON since 2026-10-04 with the fixed guard; LEFT: one window that actually removes snapshots, expected from ~2026-10-11 when the first candidates are older than 8 days** **DATED CHECK 2026-10-12 (DUE-CHECKS block), what to look at, on both demo boxes:** (1) the controller's window lines since 2026-10-10 — `pct exec 9201 -- docker logs felhom-controller 2>&1 | grep 'clean-up window'` on demo-hp (`ssh demo-hp`) and demo-felhom (`ssh felhom-pve`): a `clean-up window <id> (...): N of M planned snapshot(s) removed, A -> B` line with N > 0 and no `guard REFUSED`; (2) the snapshot count before and after — that line's `A -> B`, matching the hub's `offsite_windows` row (`count_before`, `count_after`, `box_result pruned`); (3) the hub's count check quiet — no `offsite_window_drop`, `offsite_prune_guard_refused` or `offsite_window_failed` event for either box since 2026-10-10; (4) the key file clean — operator `POST /offsite/key-audit` reads 0 findings for both. **All four hold → close R-95. Any one fails → the reason is a new row, and R-95 stays.** Since hub v0.129.0 a window refused only by the count cap after a gap can be cleared with one raised-cap grant (R-833). **MEASURED 2026-10-03 (R-436 CLOSED) — append-only HOLDS for a pinned key; it is NOT yet protection.** On the provider (`u629488-sub4`, scratch repo, removed): a key pinned to `command="rclone serve restic --stdio --append-only <dir>",restrict` backs up, restores (bytes identical) and checks; `forget <id> --prune`, `forget --keep-last 1` and a real `prune` are refused with `blob not removed, server response: 403 Forbidden (403)`; the unpinned control deletes. **What still defeats it:** the sub-account PASSWORD logs in on ports 22 and 23 and rewrote `authorized_keys` in this session, and a box can obtain that password from the hub at will (R-820). **Also found:** an add-only key can plant future-dated snapshots that make the box's own retention policy select every real snapshot (R-822); the hub holds every sub-account password in the clear (R-821); FOUR box features need a deleting login, not two — both `forget` sites, the orphan move-aside (`mv`) and the abandonment (`rm -rf`). **PROPOSAL (operator decides — STATUS):** the hub becomes the key registrar so the box never sees the password; switch boxes to the pinned key with no box-side retention first (quota headroom is months); then a weekly hub-opened clean-up window with a poisoning guard. `audits/offsite-append-only-2026-10-03/DESIGN.md`. The rank stays the operator's. **BUILT 2026-10-03 (decisions 68–69).** Both demo boxes migrated to a hub-pinned append-only key with history kept (11→13, 91→100); a delete from each box is refused (403); the old password endpoint answers 410 to demo-hp's own key; the daily key check is live; the clean-up window opened, refused by the guard and closed live on demo-hp. Left: R-824 (the guard refuses after any manual run, so no real prune has run), R-822 residual, R-823. `audits/offsite-lock-build-2026-10-03/` | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` **-- UNBLOCKED 2026-09-22, not closed.** The premise in this row's own title - *SFTP cannot express append-only* - is answered by Hetzner (ticket #2026090103040671, recorded in R-433 and R-436): on a Storage Box the `rclone serve restic --stdio --append-only` backend can be pinned to an SSH key as a FORCED COMMAND, so append-only IS expressible without a new machine. **The row stays open because nothing has been measured**: no forced-command key exists on `u629488`, no `forget --prune` has been refused through one, and the protection only holds if EVERY key that can reach the repository carries the prefix. The next step is R-436's, and it is small. | CC **SPIKE 2026-09-01 — `audits/SPIKE-r95-offsite-delete-2026-09-01.md`. THE WORD "ARMED" ABOVE IS NOT SUPPORTED AND IS WITHDRAWN PENDING R-429:** no `.snapshots` is visible to either box's sub-account (measured, both machines, with controls), so the seven-day bound is unverified and unverifiable from the product side. **Q3 (documented, `hub/internal/hetznerapi/hetznerapi.go:38-45`): the sub-account API has ONE permission axis, `readonly` — there is no append-only, so the PBS shape does NOT transfer** (PBS is a server that can refuse; a Storage Box is a filesystem that runs nothing). **Q5 (measured): withdrawing delete does NOT wedge the store** — restic treats a dead owner's lock as stale and proceeds — so the constraint everyone feared is not the blocker; but `unlock --remove-all` lies about success (R-430), and the crash-lock window is UNKNOWN. **Q6 (measured): restic 0.14.0 DOES speak `rest:`** (control: `banana:` → invalid backend), anLine truncated
|
||||
| **R-105** | Backup & restore | P2 | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-28, size M, roadmap state `READY — 2026-07-28`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. ****PARTLY FIXED BY ITS OWN UPDATE.** The `drives` third was traced and populated on both demo boxes on 2026-07-28 (the enrolled drives were never PVE storages, so `isUserDataDrive` never saw them). The other two thirds — `hosts.dr_record_json` and `host_escrow.directive_json` — were NOT re-verified this session and are carried as written.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. **UPDATE 2026-07-28 (vzdump-target move): the `drives` third is TRACED and now POPULATED on both demo boxes.** Cause: the enrolled data drives were never PVE storages at all — only agent-generated systemd mounts — so they never entered `report.StorageTargets` and `isUserDataDrive` never saw them. Giving each drive a `dir` storage at its own mountpoint supplied all three required fields at once (type `local-dir`, fs-UUID durable id, mount path), and the recipe now emits `uuid:91d2dc2d-…`/`/mnt/nvme-1tb` on demo-hp and `uuid:47a3361a-…`/`/mnt/hdd_1` o | CC |
|
||||
| **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | **READY** — owner Viktor | — | — | operator |
|
||||
| **R-304** | Backup & restore | P2 | **The retained escrow key works, and the customer is told their correct code is wrong.** DRILL 2026-08-12 answered the three questions separately, on `demo-felhom`, with planted data. **(a) retention: WORKS** — the first retained row in fleet history to carry material (`host_escrow_superseded` id 11, `identity_blob` 572 B), byte-identical (`sha256 a10032341c8584ed…`) to the pre-supersession `host_escrow` row. **(b) the material opens the old store: YES** — unsealed with the OLD recovery code it yielded a password byte-identical to the pre-change one (`sha c60c8bc737a6b7c6…`), and restored three planted files **byte-identical** from a store the box itself could no longer open (negative control first: `Fatal: wrong password or no key found`), **including a Hungarian accented filename verified as raw bytes**. **(c) the customer's route: DOES NOT EXIST, and misinforms.** `ListSupersededEscrow` (`store.go:2841`) is the only reader of a retained `identity_blob` and has **zero production callers** — five call sites, all `_test.go`; the product path (`POST /escrow/recover-offsite-password` → `FetchIdentityEscrow` → `GetHostDRBundle`, `store.go:3152`) selects `FROM host_escrow` — the CURRENT row only. Asked for the old password with the code that demonstrably opens the retained row, the product answered **"the recovery code did not open the sealed bundle — nothing was written"**. **This is the R-224 class again**: there an unreachable hub was reported as a bad code; here a VALID code for retained history is reported as a bad code, and the customer's attempt ends there. **Consequence:** the census answer stands (it was about retention); the countdown banner's promise is true in substance and false in practice; **any capability-map claim that the customer can recover the old history with their recovery code is false today and must move** | **READY (L) — NEW 2026-08-12, RANK 1** | R-198, R-199, R-224, R-241 | Decide the shape: serve retained rows on the recovery path (needs a "which package?" choice — a customer may have several), or stop promising retrieval anywhere the customer cannot perform it. **Until one of those, the honest position is that retention is an operator-only capability.** At minimum, the refusal must stop asserting the code is wrong when the hub simply never looked | operator + CC |
|
||||
@@ -231,7 +227,6 @@ stopping line that lies.
|
||||
| **R-698** | Backup & restore | P3 | **[P3-LOW] A backup stores the image's NAME, not the image — a restore of a version its maker has deleted cannot start.** `RecoveryManifest.image_pins` ("image NOT stored — re-pulled on restore"); since controller v0.275.0 each data file also records its running `ref@digest`, and a restore brings the data back AT ITS OWN VERSION (`07` §6.6) — so a restore asks for exactly the old image. **Measured 2026-09-26** (`audits/version-travel-2026-09-26/A7/`, registry HEADs, no pulls): the catalog's 42 ladder `ref@digest` pairs all resolve (200); an invented digest answers 404 on Docker Hub and ghcr.io (negative control). Not measured: the digests recorded on boxes (older than any ladder entry), how often makers delete versions, the catalog's 66 digest-less compose lines. **Options (decide nothing yet):** (a) keep — a restore of a deleted version fails at the pull and the household uses the next copy or a newer version; (b) mirror every INSTALLED image into the DooPlex registry, restore falls back to it — storage + bandwidth on DooPlex, a new part on the recovery path; (c) mirror only ladder-named versions — bounded, misses pre-ladder boxes; (d) `docker save` into the unit — hundreds of MB per app per copy on every tier. **-- 2026-09-30 late (decision 53):** a box now keeps only an app's running and previous image; a restore to an older version re-pulls it — as every restore already did. The limit above is unchanged. | **OPEN — P3; owner: operator (a decision), CC measures** | — | — | CC + operator |
|
||||
| **R-706** | Backup & restore | P3 | **[P3-LOW] Removing an app "with its backups" leaves its off-site verification copy on the drive.** Measured 2026-09-28 on demo-hp: after a full off-site restore of nextcloud (which leaves the downloaded copy in `backups/offsite-restore/nextcloud`, ~1 GB, by design, for the household to inspect), `POST /api/stacks/nextcloud/remove` with `remove_backups: true` removed the unit and listed `backup_paths_removed` WITHOUT the verification copy; it stayed until the restore page's own delete (`POST /backup/offbox/verify-copy/delete`, 302 `scratch_deleted`). A household that removes an app to free space keeps 1 GB it cannot see on the app list. **Fix direction:** the removal with backups also deletes the app's verification copy (the same `DeleteOffsiteRestoreCopy`). Evidence: `audits/kept-offsite-2026-09-28/E/E9-teardown.txt`. **-- 2026-09-28 evening: FIXED in controller v0.279.0** — a removal with its backups also deletes the verification copy and lists it among the removed paths (`TestR706_…`, red-proofed RP7). Not yet seen live (needs a full off-site restore then a removal). | **WATCHING — P3; owner: CC (live)** | — | — | CC |
|
||||
| **R-729** | Backup & restore | P3 | **[P3-LOW] An off-site target, once saved on the page, cannot be removed through the product.** MEASURED 2026-09-30 on 9202: `/backup/offbox/config` refuses an empty address and no route clears the target; the session removed its throwaway target from `settings.json` by hand, with the controller stopped (harness teardown on a scratch guest). A household that tries its own NAS and gives up keeps a disabled target forever. **Fix direction:** a „Távoli mentési cél törlése" press that clears the target (never the repository). | **READY — rank P3-LOW; owner: CC (controller)** | — | — | CC |
|
||||
| **R-869** | Backup & restore | P4 | **The controller's log line for decision 78's move-aside prints an empty destination: `the hub set the orphaned repo aside: /home/felhom-repo -> (nothing deleted)`.** 2026-10-05 03:08 UTC, Tester 1 box (controller v0.293.0); the next line names it (`/home/felhom-repo.orphaned-20261005`). Cosmetic, but it is the one line an operator reads to find where the old history went. Fix direction: log the hub's returned path, or the name the controller computes. `audits/night-2026-10-04/tester1/t14-d78-offsite-run.txt` | **READY — owner: CC** | — | — | CC |
|
||||
| **R-10** | Backup & restore | P4 | T-6E-1: DB-dump dir-fsync asymmetry (LOW, confirmed in 6E) **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-15, size XS, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **T-6E-1, confirmed in CAMPAIGN-6E.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | One-line hardening; batch with the next controller task | CC |
|
||||
| **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
|
||||
| **R-99** | Backup & restore | P4 | Server-side prune **never removes** a phantom snapshot. Confirmed it does NOT count them toward `keep-last` (dry-run kept 2 real + the phantom) so there is **no retention/data-loss bug** — but one accumulates per aborted upload, forever | READY (S) | — | Decide a cleanup path. Deletion on a **customer** datastore is a separate ruling — detection shipped, removal deliberately not automated | CC |
|
||||
@@ -253,6 +248,8 @@ stopping line that lies.
|
||||
| **R-815** | Backup & restore | P4 | First-ever **GC** on `felhom-offsite` (armed today 13:11 UTC, never run) | **VERIFY** (2026-10-03 triage: a July watch row with no id; given R-815. WATCHING — no completion record found; schedule `sun 04:30` still present 2026-09-30.) — WATCHING | schedule | **Sun 2026-08-02 04:30 UTC** — confirm it completes | CC |
|
||||
| **R-816** | Backup & restore | P4 | **No off-site failure class has ever been seen live.** F-DIAG (controller v0.182.0, 2026-07-28) split off-site failures into six causes — quota, orphaned, no_repo, no_units, transport, unknown — each with its own Hungarian message. None of the six has been exercised by a real failure on a box; the recovery inventory records it only as a known limit (`documentation/architecture/_recovery-inventory-2026-07-28.md:955`). Filed 2026-10-03 from the F-DIAG row's residue when that row moved to `CLOSED-ITEMS.md`. | **READY — filed 2026-10-03 (triage); owner: CC.** Exercise each class once on a scratch guest (a full quota, a missing repository, a blocked transport) and read the message the household sees. | — | — | CC |
|
||||
| **R-832** | Backup & restore | P4 | **ep0's copy in a place outside both Hetzner and the operator's home (roadmap).** Today DooPlex (the operator's home) holds it (decision 71). A Hetzner Storage Box would share a provider with ep0 and with every household's file backups, and cannot run PBS, so the copy could not be verified or restored from directly. | **DEFERRED — later, if the product grows** | — | — | operator |
|
||||
| **R-871** | Backup & restore | P2 | **No architecture document covers a box that is not always on, and the product has no catch-up for a missed night: a box that is OFF at its night window (Tester 2, a laptop switched off at night — operator 2026-10-05) never gets its database dumps, second copy, off-site copy or app updates.** FOUND 2026-10-05 (Part F spike, read only): the controller's daily jobs always schedule the NEXT future time (`controller/internal/scheduler/scheduler.go` `nextDailyRun`; `LastRun` in memory only, never consulted) — a missed 02:30/03:30/04:15 waits for the next night, for ever. Only the whole-guest backup catches up (outside its window only by the 48 h safety valve — about one every 2 days for an evening-only box), and the OS leg follows it by day under the `night` label. Nothing on the household's pages says the box must stay on at night. `07` §6.1 describes the night chain and the [W+2h, W+6h) gate but no catch-up and no safety valve; the intent lives only in controller comments (`quiesce.go`). **A promise to households — the operator's choice (STATUS, options A/B).** `audits/night-fixes-2026-10-05/partF/FINDINGS.md` | **WAITING-ON-OPERATOR — A (catch-up) or B (say plainly the box must be on at night)** | — | the decision, then a design section in `07` | operator |
|
||||
| **R-874** | Backup & restore | P3 | **A restore-test never runs on a box whose power-on sessions are all shorter than 6 hours.** FOUND 2026-10-05 (Part F spike, source): the agent checks every 6 h, the timer restarts at each agent start, and it does not check at start (`felhom-agent` restore-test schedule). Tester 2's sessions were ~1.5 h and ~5 min. `restore_test_stale` will then fire (~2026-10-11 for the local tier) with no visible cause for the household or the operator. Fix direction: check once shortly after start when the last test is overdue. `audits/night-fixes-2026-10-05/partF/FINDINGS.md` | **READY — owner: CC** (after R-871's decision) | R-871 | — | CC |
|
||||
|
||||
## Storage & devices — 12 rows (P3 7, P4 5)
|
||||
|
||||
@@ -271,7 +268,7 @@ stopping line that lies.
|
||||
| **R-368** | Storage & devices | P4 | **The storage default DOES apply at deploy time — the earlier claim that it never does was wrong, and the residual defect is smaller and different.** R-352 and `SPEC-app-data-placement-2026-08-21.md` §2.2 stated *"the deploy route never reads it"*, from `grep -nE 'GetDefaultStoragePath|primaryHDDPath|IsDefault' internal/stacks/deploy.go internal/stacks/manager.go` → nothing. **That grep searched Go files only and never the templates.** `internal/web/templates/deploy.html:612` reads `.IsDefault` directly off each `DeployStoragePath` (which embeds `settings.StoragePath`, `web/handlers.go:89-99`) and **pre-selects the default drive for a new deploy**: `{{else if and .IsDefault (not .NotAllowed)}}selected{{end}}`. So `// new apps use this by default` (`settings.go:453`) is **IMPRECISE ABOUT THE MECHANISM, NOT FALSE** — nobody calls `GetDefaultStoragePath()` on that route, but the value is honoured. The customer-facing label promises exactly this and no more: **„Legyen alapértelmezett új telepítéseknél"** (`storage.html:469`). **THE RESIDUAL, and it is the whole finding:** the default lives in the TEMPLATE, not in the server. `POST /api/stacks/<n>/deploy` accepts `values` verbatim; omit `HDD_PATH` and `withPathVars` (`stacks/deploy.go:584`) receives `""` and no default is applied. **That is why the invariant has no test — there is nothing server-side to test.** | **OPEN — LOW** | corrects R-352(2); supersedes SPEC §2.2 | Either move the default into the server so the API and the form agree and a test can pin it, or reword the comment to say the template owns it. Do not "fix" the behaviour: it is correct on the path customers use. | CC |
|
||||
| **R-568** | Storage & devices | P4 | **[P3-LOW] The dashboard's drive-health rows swap order between visits — the same two disks, listed in a different order a minute apart.** MEASURED 2026-09-17 on demo-hp 9201 during slice 1 release C's live proof: `/dashboard` fetched on 0.249.0 listed „KXG50PNV1T02 NVMe TOSHIBA 1024GB” then „SanDisk X600 M.2 2280 SATA 128GB”; fetched on 0.250.0 a minute later, the reverse (`audits/i18n-slice1-2026-09-17/C/live/hu-before-vs-after.txt`). `diskHealthRows` (`disk_health.go` L135–141) keeps the agent's response order and does not sort; the agent's order is therefore not stable. Cosmetic, but a household that reads „the second disk” finds a different one. **Fix shape:** sort the rows controller-side by a durable key (device path or serial), with a test that feeds two orders and expects one. | **READY - rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3->P4: cosmetic.** | — | — | CC |
|
||||
|
||||
## Security & access — 31 rows (P2 4, P3 24, P4 3)
|
||||
## Security & access — 32 rows (P2 4, P3 25, P4 3)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -303,18 +300,18 @@ stopping line that lies.
|
||||
| **R-782** | Security & access | P3 | **[P3-LOW] Two side observations of the R-753 sweep, inferred, not measured:** glance's seeded `glance.yml` has no `auth:` block (the dashboard is public to anyone with the address), and homepage's `/api/*` refuses a Host not in `HOMEPAGE_ALLOWED_HOSTS`, which the template does not set (widgets may 400). **Needs:** measure both on 9202; glance: decide whether a public link dashboard is intended (the setup gate does not cover it after setup). | **READY — rank P3-LOW; owner: CC (catalog)** | — | — | CC |
|
||||
| **R-783** | Security & access | P3 | **[P3-LOW] SparkyFitness: three wrong sign-ins by anyone shut EVERY visitor out of sign-in for ~10 s — a stranger retrying every 10 s keeps the household out.** MEASURED 2026-10-01 on 9202 through the simulated tunnel (`audits/visitors-2026-10-01/C/box/sparky-box.txt`): better-auth's sign-in limit (3 per 10 s) is keyed on one address — it reads the LEFTMOST X-Forwarded-For, whose chain the R-753 router reset removes, so its frontend nginx hands it traefik's address; the household from another address got 429 at 3 s and 10 s, in at 16 s. Not forgeable (a rotating forged address did not escape). better-auth's header setting is not exposed as an env by SparkyFitness. **Needs:** accept (10 s), or an upstream setting for better-auth's `ipAddressHeaders` + a right-walking reader. | **OPEN — rank P3-LOW; owner: CC** | — | — | CC |
|
||||
| **R-831** | Security & access | P3 | **The Hetzner storage API token (`HETZNER_TOKEN`, the storage project's token in `Secret/storagebox`) was printed into the 2026-10-03 session transcript** — CC read the gitignored `manifests/storagebox.secret.yaml` and its redaction pattern missed the quoted value. It can create, reset and delete Storage Box sub-accounts. Not rotated by the operator's choice (decision 73). **Rotation, whenever chosen (3 steps):** create a new token in the storage project in the Hetzner console → patch `Secret/storagebox` key `HETZNER_TOKEN` in `felhom-system` and `kubectl rollout restart deployment/hub` → delete the old token in the console. Rule for sessions: never print a file that holds secrets — read the one field needed. | **WAITING-ON-OPERATOR — rotation is his call** | — | rotate when chosen | operator |
|
||||
| **R-870** | Security & access | P3 | **Tester 1's two Cloudflare credentials — the zone API token (`infrastructure.cf_api_token`) and the tunnel token (`infrastructure.cf_tunnel_token`) of the hub's `customer_configs` row `tester-1` — were printed into the 2026-10-04 night session's transcript** (not into any file): a read-only query selected `substr(config_json,1,400)`, and both values sit in the first 400 characters. Tester 1 is CC's disposable test customer (`enkicsifelhom.hu`). **Not rotated, by the operator's ruling of 2026-10-05 06:49 (option B).** **Rotation, whenever chosen (3 steps):** in the Cloudflare dashboard create a new API token for the `enkicsifelhom.hu` zone with the same permissions and refresh the Tester 1 tunnel's token (Zero Trust → Networks → Tunnels → the tunnel → refresh token) → hub → Configs → `tester-1` → Edit → the two Cloudflare fields → Save, then confirm on the box that cloudflared reconnected (`docker ps` health `healthy`) → delete the old API token. Rule for sessions (as R-831): never select a whole config row — name the fields, and never `config_json` without `json_extract` of a non-secret field. | **WAITING-ON-OPERATOR — rotation is his call (ruled: not now)** | — | rotate when chosen | operator |
|
||||
| **R-134** | Security & access | P4 | **Two zone-resolvers disagree on depth.** The controller strips labels progressively (`controller/internal/cloudflare/zone.go:18`); the hub's `resolveZone` tries the exact name then `parentDomain`, which strips exactly ONE label (`hub/internal/cloudflare/unblock.go:115,136`) | READY (XS) | — | For a one-label Felhom-issued subdomain both work; for anything deeper the hub silently fails to find the zone while the controller succeeds — the geo-unblock would then no-op with a "no active zone found" error. One concept, two implementations. Same audit §2.6 | CC |
|
||||
| **R-525** | Security & access | P4 | **[P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured.** Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. **What it needs:** a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | **READY — rank P3-LOW; owner: CC (spike)** **Re-ranked 2026-10-03: P3->P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed.** | — | — | CC |
|
||||
| **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator |
|
||||
|
||||
## Box system & updates — 20 rows (P2 3, P3 14, P4 3)
|
||||
## Box system & updates — 20 rows (P2 4, P3 13, P4 3)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-530** | Box system & updates | P2 | **[P2-MEDIUM] A floor does not deliver an agent: agents update only by an operator-signed `agent_update` job per box, and nothing records which boxes still run 0.130.0.** MEASURED 2026-09-15: the hub HOLDS a floor whose declared MinAgent is above the box's agent (`api/handler.go` ResolveManagedFloor); the agent's only update path is `signedjobs` + `selfupdate.Executor`. demo-hp reached 0.131.0 by `felhom-opsign -op agent_update` (key `felhom-op-1`) at 08:44:16Z and its controller floor was then SERVED in 3 s. **demo-felhom (N100) and Peti's box still run 0.130.0** — not touched (Peti fenced; N100 not asked). **What it needs:** the operator signs per box, or rules a fleet rollout step. **NARROWED 2026-09-16 (operator ruling 1):** the keys stay on DooPlex owner-only and CC may sign `agent_update` until the first PAYING customer (testers excluded) — recorded in `CONTEXT.md` + `04-control-plane-authorization.md` §3.1. Both demo boxes now run agent 0.131.0 (demo-hp 2026-09-15, demo-felhom 2026-09-16, each by a per-box signed job; Peti's box untouched, still 0.130.0). **What remains:** a fleet rollout step — signing per box does not scale past a handful, and nothing lists which boxes are behind. **2026-09-25:** Peti's box was RETIRED (operator ruling) — it no longer counts as a box behind. | **WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: operator (signing)** | — | — | operator |
|
||||
| **R-604** | Box system & updates | P2 | **[P2-MEDIUM] A per-customer controller floor silently excludes that box from every global floor raise, and NOTHING says so — demo-hp missed four of them.** FOUND 2026-09-21 while raising the global floor to 0.259.0 at the operator's request. The raise logged `Global controller-version floor set to "0.259.0"` and then `managed floor SERVED for demo-felhom` — **and nothing at all for demo-hp**, which went on reporting every few minutes and stayed on 0.258.0. Cause: `customer_configs.min_controller_version` for demo-hp held **`0.243.0`**, a per-customer override that wins over the global. It is a **leftover from the 2026-09-16 drill**, whose golden was 0.243.0; **R-343 measured on 2026-08-18 that all five rows were EMPTY and recorded that as a safety property** — it stopped being true and nothing surfaced the change. demo-hp had therefore silently missed the raises to 0.253.0, 0.254.0, 0.257.0 and 0.259.0. **Why it is invisible rather than merely quiet:** `managed floor SERVED` fires **once per CHANGE** (`h.floorNotes`, `api/handler.go:600`), deliberately, because a box reports every few minutes — so a box whose override never changes is silent for ever, and its silence is indistinguishable from the silence of a box that already had the line. A session that raises the floor reads one SERVED line and reasonably concludes the fleet took it. **CLEARED** for demo-hp the same session (rollback line: POST `/customers/demo-hp/floor` with `min_controller_version=0.243.0`, `min_agent=0.131.0`); it then self-updated 0.258.0 → 0.259.0 in **under four minutes**, healthy, `settle-gate: GO — at/above floor 0.259.0`, and its claim page answers **"Wrong or expired code"** in English — the floor delivered the FIX, not a version string, to a box nobody hand-deployed. All five overrides are now empty. **Fix shape (smallest first):** the floor-raise page shows which customers carry an override and would NOT be moved, before the save; or the raise logs one line per customer naming the ones it skipped and why. A raise that quietly reaches half the fleet is worse than one that refuses. | **READY — rank P2-MEDIUM; owner: CC (hub)** | — | — | CC |
|
||||
| **R-812** | Box system & updates | P2 | **[P2] A box never receives operating-system security updates — not the Proxmox host, not the guest's Debian, not its Docker engine.** SEARCHED 2026-10-03 (read-only): `felhom-controller`, `felhom-agent`, `app-catalog-felhom.eu` and `felhom.eu` hold no `apt-get upgrade`, `apt full-upgrade`, `unattended-upgrades`, `pveupgrade` or `needrestart` that runs on a box. The installer aligns the host's Proxmox repositories to no-subscription *"so the box can pull security updates"* and then says plainly *"No upgrades are run"* (`scripts/felhom-host-install.sh:2133-2136`). The guest's Docker engine is installed when the golden is BAKED (`felhom-agent/configs/build-golden.sh:124-125`), so a fresh install gets that week's engine and an installed box keeps it forever. The only `apt full-upgrade` in the project is a by-hand step for the off-site endpoint ep0 (`documentation/runbooks/offsite-endpoint.md:41`), not a box. App images ARE updated (the update arc); the layer under them is not. The intention, with its scope, is **R-808** in `ROADMAP.md`. | **NARROWED 2026-10-04 (evening) — the guest's DOCKER engine slow lane is BUILT and proven live (agent v0.142.0, hub v0.132.0; `11` §5.8): live-restore on everywhere, ring 0 steps under a root-owned mark, ring 1 and undo only by a signed job the wrapper re-verifies; the operator approves each engine set on the System page. LEFT: the kernel lane (R-836); existing boxes (R-840).** Earlier: NARROWED 2026-10-04 (afternoon) — the HOST's Debian fast lane is BUILT and proven live too (agent v0.141.1, hub v0.131.1; `11` §8.2, `audits/os-host-lane-2026-10-04/`): appliances only, never kernel/boot/firmware, after a healthy guest step; fleet view and four alarms (§8.3). LEFT: the Docker and kernel slow lanes (R-836); existing boxes (R-840).** Earlier: NARROWED AGAIN 2026-10-04 (day) — the GUEST's Debian fast lane is BUILT and proven live (agent v0.140.0, hub v0.130.0, installer 1.29.0; `11` §8.1, `audits/os-guest-lane-2026-10-04/`). LEFT: the host, Docker and kernel lanes; the undo (R-842); existing boxes (R-840).** Earlier: NARROWED 2026-10-04 — the SPIKE is done (`11` §7.1, corrections C1–C12, `audits/os-updates-spike-2026-10-04/`); no product code yet. LEFT: the build steps of `11` §8, each with the operator's go; the §5.3 snapshot question is in STATUS; preconditions R-835, R-836, R-837.** | — | — | CC + operator |
|
||||
| **R-865** | Box system & updates | P3 | **The OS wrapper's free-space rule (R8) always measures the download as 0 bytes, so only its 500 MB floor ever applies.** MEASURED 2026-10-04 night drill (A4, Tester 1 box): with the guest's `/` at 400 MB free the wrapper refused correctly — `R8 free space 419430400 B is below max(500 MB, 3 x download 0 B)` — but the download was `0 B`. `download_bytes` runs `apt-get -s … --print-uris`; with `-s` apt prints no URI list (measured: 0 URIs with `-s`, 3 URIs / 1 471 812 B without). So a large update (a Docker set, a Proxmox minor) is checked only against 500 MB, not 3× its size as `11` §5.4.1 R8 says. Fix direction: drop `-s` from the `--print-uris` call (it downloads nothing by itself) and pin it with a test on real apt output. `audits/night-2026-10-04/accidents/a4-*.txt` | **READY — owner: CC** | — | fix + a test | CC |
|
||||
| **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
|
||||
| **R-35** | Box system & updates | P3 | **Config-apply should not end the customer's session.** The offsite config push bumped `config_version` 10→11 at 16:54:58 and the controller self-restarted (container `StartedAt` 16:54:59Z, back up 16:55:02); in-memory sessions died with it and **customer zero was force-logged-out mid-flow**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size S, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | Direction: **hot-apply the offbox target** (no restart for a config the running process can adopt), or **persist sessions** across restart. The restart itself is by design — the collateral is not. Evidence `controller-log-full.txt` | CC |
|
||||
| **R-50b** | Box system & updates | P3 | **[P2] A root-owned privileged host artifact is delivered unversioned from `main` — "which wrapper is on this host?" is unanswerable.** `configs/felhom-pbs-apply` installs to `/usr/local/sbin/felhom-pbs-apply` (0755 root:root) and is the pinned sudoers vector for `create\ |reconcile\|grant` against `/etc/pve/priv/storage`. It is fetched by `felhom-host-install.sh:1914` via `fetch_raw`, which hits `raw/branch/main/<path>` — **no tag, no pin, no checksum, and no record in the Day-0 artifact manifest**, unlike the agent binary (sha256-vouched) and the golden image. Three consequences: (1) two hosts installed a week apart can carry different privileged wrapper code while both reporting the same agent version; (2) a host hotfixed in place (felhom-pve, 2026-07-18) is indistinguishable from one that fetched the same content — the fleet has no inventory of it; (3) an accidental push to `main` reaches the next install of every host with no review gate between commit and root-owned deployment. | **NARROWED** — **(a) SHIPPED 2026-07-21; (b)/(c) open** — moved from `ROADMAP.md` 2026-10-03: it states a checkable fact about the shipped product, so it is a FINDING (the sorting rule). **Re-ranked 2026-10-03: [P2] → P3 — operator-only; leg (a) shipped.** | — | **Surfaced 2026-07-21 while stopping the R-39 v0.90.1 publish** (`felhom-controller/REPORT.md` §5): the publish was cancelled precisely because the version number would have claimed to carry a fix that in fact rides this unversioned channel. Candidate shapes, in increasing cost: (a) record the wrapper's sha256 in the Day-0 artifact manifest beside the agent binary and have the agent report the installed file's hash, so drift is at least *visible*; (b) `fetch_raw` takes a pinned ref (tag or commit) supplied by the manifest rather than `main`; (c) the wrapper becomes a published generic-registry artifact with the same gate ladder as the agent binary. **(a) is the cheap honest first step and would have caught this class already.** Pairs with R-39 (whose remaining fleet half is specced separately) **(a) SHIPPED 2026-07-21 — hub v0.68.0 + agent v0.91.2.** `ArtifactManifest.WrapperSHA256` + an operator field; agents report the installed wrapper's sha256 each cycle and the host page surfaces a mismatch. **An unknown on EITHER side reads as quiet, never as drift** — lighting every host amber on rollout day is how a warning becomes background noise. Live confirmation of exactly the problem: felhom-pve's July-18 in-place hotfix hashed `2888f2ea…`, matching **no commit anyone could name**; it now reports `104db0a4…` against a vouchable manifest value. **(b)/(c) REMAIN OPEN:** the wrapper is still fetched unversioned from `raw/branch/main` — this makes drift *visible*, it does not fix the channel. Also recorded: the 0440 sudoers file is not agent-readable, so its drift stays invisible. **Flips (2026-10-03):** `00` §A "The installer is PUBLISHED, not pushed" — the same discipline for the privileged wrappers. **Re-ranked 2026-10-03:** [P2] → P3: operator-only; (a) shipped 2026-07-21 (the report carries the wrapper sha256), (b)/(c) open. **Finding-shaped** — an R-424 instance; check against today's product before building. | CC |
|
||||
@@ -327,15 +324,16 @@ stopping line that lies.
|
||||
| **R-444** | Box system & updates | P3 | **[P3-LOW] Nothing runs `pct fstrim` on the fleet, and demo-hp's thin pool was carrying ~23.8 GB of blocks the guest had already freed.** MEASURED 2026-09-01 during this spike's teardown: the run itself added ~1.05 GiB that `local-lvm` did not reclaim on delete (68.97% -> 70.91%); `fstrim` INSIDE the unprivileged container is refused (`FITRIM ioctl failed: Operation not permitted`, all three mounts); `pct fstrim 9201` from the PVE host then trimmed **30.2 GiB + 57 GiB** and took `local-lvm` to **26.78%** — **23.8 GB BELOW this run's own starting point**, i.e. the surplus was long-standing, not ours. **Why it is not merely housekeeping:** a thin pool that only ever grows can reach 100% from DELETED data alone, and a full thin pool takes every guest on the host read-only. demo-hp had 16.4 GB free before the trim. **Not urgent, and the row says so** — but the appliance has no periodic trim and no operator surface reports the gap between guest-free and pool-used. Owner: **CC.** `audits/SPIKE-app-update-2026-09-01.md` | **OPEN — rank P3-LOW; owner: CC** | — | — | CC |
|
||||
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** | **WATCHING — rank P3-LOW; owner: CC (renew ≤ 14 days or bake); retire at the first external install** | — | — | CC |
|
||||
| **R-531** | Box system & updates | P3 | **[P3-LOW] Three supervisor facts measured live and not pinned: restart timing during a deploy was not measured; restarts before the hub first sees the stanza produce no `controller_restarted_by_agent`; deliberate operator kills spend the crash-loop budget.** MEASURED 2026-09-15 on 9201: after 3 test restarts in 13 minutes the 4th kill tripped the 30-minute pause and the dashboard stayed down (the guard as designed, `A4-kill-middeploy-9201.txt`). The hub checker seeds silently on first sight, so the three restarts before the first v0.131.0 report emitted nothing (only the crash-loop did). **What it needs:** a deploy-kill timing on a fresh budget; the operator's view whether a restart after minutes of uptime should count toward the budget **MEASURED 2026-09-16 on the drill box, both halves.** (1) **Timing during a deploy (F9'):** the controller was killed 5 s into a deploy on an EMPTY budget; the agent saw it on the next sweep, confirmed on the one after, and the dashboard answered 200 again **37 s** after the kill; the interrupted app ended `not_deployed`, not stuck. (2) **The budget's shape (F9''):** three further kills at idle, 20 minutes apart, recovered in **61 s / 41 s / 61 s** - and NONE of them accumulated, because the window is 15 minutes. Four restarts this session, zero pauses, zero crash-loop events. **So the brake catches a FAST loop and is blind to a SLOW one:** a controller dying every 20 minutes is restarted forever and the only trace is an `info` event that mails nobody. That is a design question for the operator (leave it / add a longer second counter / raise the severity of the Nth restart in a day), and this session deliberately measured it without changing it. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-f9prime.txt` and `phase2-f9dprime.txt`. | **READY — rank P3-LOW; owner: CC (measure) · operator (budget rule)** | — | — | CC + operator |
|
||||
| **R-868** | Box system & updates | P4 | **An OS pass whose agent is killed mid-install still installs (the root wrapper runs on under sudo), but its report never reaches the hub.** MEASURED 2026-10-05 02:57 UTC on demo-hp (night drill A5): the debug pass and the daemon `kill -9`-ed while `apt-get` ran; the wrapper finished all 6 packages, dpkg clean, systemd restarted the daemon in < 20 s (`NRestarts` 1, no rollback). The hub never got that pass's `applied` report; the next pass said `nothing`. For a NIGHT leg the hub would therefore miss which packages a ring-0 box installed that night (the approval set is still read from the next inventory). Fix direction: the wrapper keeps its last report on disk; the agent sends an unsent one at start. `audits/night-2026-10-04/accidents/a5-kill.txt` | **READY — owner: CC** | — | — | CC |
|
||||
| **R-184** | Box system & updates | P4 | **Nothing prevents the hub from vouching an agent version that was never released.** The R-115 gate proves every RELEASED version is installable, but it works from git tags — so a hub artifact-manifest entry naming a version with no tag and no package is invisible to it. The installer would then die at step 5 on a virgin machine, as root | **READY (S) — NEW 2026-08-03** | — | **Filed BECAUSE the R-115 gate deliberately does not cover it, rather than leaving the gap unstated.** CI cannot check it: the hub's `/api/v1/artifacts/<customer>` answers **401** without a per-customer retrieval passphrase and Gitea's package **listing** api answers **401** without a token (both measured 2026-08-03, P-C), so a credential-free gate can ask *"is this version installable"* but never *"which version is vouched"*. **Two shapes, and the second is better:** (a) give CI a hub credential — expands what CI can reach, and is the operator's call not a gate author's; (b) **validate at vouch time, in the hub**: the operator UI's Day-0 artifact form refuses a version whose package is not downloadable. (b) fails closed at the moment of the decision, needs no new credential anywhere, and puts the check where the mistake is actually made. **Exposure is low and should be said so:** vouching is a deliberate operator action against a version they have just released, and R-115's release path now makes released-but-unpublished nearly impossible. This is the residue, not the main risk | CC |
|
||||
| **R-194** | Box system & updates | P4 | **PVE's permission cache delays every grant-state verdict by an unknown amount, so "the agent can read it" and "the ACL exists" are not the same measurement.** Observed twice while validating R-190's self-repair on demo-felhom 2026-08-04: both ACL rows for `/storage/felhom-backup` were deleted, and `GET /access/permissions` continued to report `Datastore.AllocateSpace` present — for **~40 s** in one run and **~16 minutes** in another. During that window the capability probe reads healthy and the self-repair does not fire | **OPEN** | — | **Why it matters beyond the delay:** it puts a floor under how fast a lost grant can be noticed, it makes any single permission read a lagging indicator, and — the interesting part — **it is a candidate contributor to R-190's own timeline**: a grant removed at an unknown moment could keep working until a cache expiry, which is exactly the shape of *worked at 04:44, refused at 09:24*. That does not explain what removed it, but it may explain when the refusal SURFACED, and the two have been treated as the same instant. **Not a defect in our code** — it is PVE behaviour, and the mitigation already tolerates it (the repair fires on the next probe after the cache clears). **What is worth deciding:** whether the store-grant probe should ALSO consult the storage content listing as a second signal, since that appeared to reflect the loss immediately (`{"data":[]}` while the permission read still said present) — two signals disagreeing is itself information, and today only one of them is read | CC |
|
||||
| **R-373** | Box system & updates | P4 | **`SysDataGrowGB` is the intended lever for the system-data volume, it works, and nothing sets it.** Written down 2026-08-02 in `audits/SPIKE-recovery-unit-space-2026-08-02.md:230-232`, under an explicit *"### Not filed"* heading: the 20 G / 50 G mismatch was ruled a tier-sizing decision rather than a defect, *"`SysDataGrowGB` is the intended lever and it works; nothing sets it."* A lever with no caller is the same shape as R-368's comment — a setting that names a behaviour nothing invokes. **Age when filed: 20 days.** | **OPEN — LOW** | R-368 (same shape) | Either wire it to something an operator can reach, or remove it and record the sizing decision where a reader will meet it. | CC |
|
||||
| **R-836** | Box system & updates | P3 | **A new host kernel that hangs before userspace stays the GRUB default: `--next-boot` is not a one-shot on these hosts.** MEASURED 2026-10-04 on demo-hp (operator's word, 2 reboots): both demo hosts boot UEFI + GRUB without proxmox-boot-tool ESPs; installing a kernel makes it the default at once; `kernel pin <new> --next-boot` writes an ordinary `GRUB_DEFAULT`, and `proxmox-boot-cleanup.service` clears it only after a boot reaches userspace. With the old kernel pinned FIRST, the fallback after a good boot worked (new 60 s, old 76 s). READ FROM THE CODE, not measured: a hang leaves the new kernel default on every power cycle. Only `softdog` runs (useless before userspace); demo-hp's `sp5100_tco` ships unloaded, untested. Fix direction for the slow lane: GRUB's own one-shot (`GRUB_DEFAULT=saved` + `grub-reboot`) with the old kernel saved — to be measured, including Secure Boot (ON on demo-hp). `audits/os-updates-spike-2026-10-04/partH/` | **NARROWED 2026-10-04 (os-host-lane Part E, operator's word before each of 2 reboots) — GRUB's own one-shot is NOT a one-shot here either.** `GRUB_DEFAULT=saved` (old kernel saved) + `grub-reboot <new>`: boot 1 → new kernel, **Secure Boot ON and fine**; but GRUB could not clear `next_entry` (`grub-reboot` itself warns: *environment block on lvm device … will remain the default until manually cleared*; `/boot` is ext4 on LVM `pve-root`), so boot 2 (no command) → **the new kernel again**. `kernel.panic = 0`: a panic leaves the host stopped (R-851). `sp5100_tco` LOADS and answers (`SP5100 TCO timer`, 60 s, inactive, nowayout 0; read from sysfs, never opened, unloaded) — a hardware watchdog exists on demo-hp, but nothing arms it before userspace. demo-hp left on 7.0.14-20 with that as the saved default. **LEFT (fix direction):** a GRUB env block GRUB can write (on the ESP, vfat) or a userspace "boot good" step that rewrites the default, plus arming `sp5100_tco`; to be measured before the kernel slow lane. `audits/os-host-lane-2026-10-04/partE/` **READY — owner: CC + operator (reboots).** | — | — | CC |
|
||||
| **R-853** | Box system & updates | P3 | **After a boot the box's versions and crash facts reach the hub up to ~15 minutes late.** MEASURED 2026-10-04 on demo-hp (crash-guard test): the agent's first report after a boot has no `system.facts` — the facts read needs a RUNNING customer guest (`firstGuest`), the guest starts ~1–2 min after the agent, and the failed read is cached for 10 minutes; so the HOST half (the crash guard, the kernel) is lost too. The crash events arrived 15 min after the boot (17:17 → 17:32 CEST); nothing was lost (the guard keeps 7 days). Fix direction: the facts mode reads the host without a guest (guest fields `unknown`), and a failed read is not cached. `audits/os-docker-crash-2026-10-04/partC/c4-hub-events.txt` | **READY — owner: CC** | — | — | CC |
|
||||
| **R-839** | Box system & updates | P3 | **After a Docker restart that stopped every container, the boot sweep HELD an app whose `app.yaml` `HDD_PATH` names its user folder instead of the drive.** MEASURED 2026-10-04 on scratch 9202 (paperless-ngx): `bootrecon` logged `drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx is not a live mountpoint — NOT starting it`, although the drive `/mnt/felhom-drives/scratch_hdd` IS a mountpoint; the app stayed down until started by hand. The customer boxes' 9201s showed no hold. **Not diagnosed:** which writer put a per-app path in `HDD_PATH` on 9202, and whether a household box can get it. `audits/os-updates-spike-2026-10-04/partG/SUMMARY.md` | **READY — diagnose; owner: CC** | — | — | CC |
|
||||
| **R-875** | Box system & updates | P4 | **A kept OS report sent after the hub was away says "sent after the agent stopped mid-pass (R-868)" — wrong for that case: the agent did not stop, the hub was unreachable.** MEASURED 2026-10-05 05:53 UTC on demo-felhom (agent v0.144.1): the three reports of the R-866 hub-away pass reached the hub at the new agent's start with that reason text. The kept copy cannot tell the two causes apart. Fix direction: a neutral reason ("sent late — kept on the box until the hub could take it"), or the agent records which case it was. `audits/night-fixes-2026-10-05/partD/r866-kept-copies-sent-at-start.txt` | **READY — owner: CC** | — | — | CC |
|
||||
| **R-876** | Box system & updates | P2 | **After a power cut in the middle of an OS update, every later OS pass FAILS until a person runs `dpkg --configure -a`: the wrapper's repair step runs only when `dpkg --audit` shows something, but a crash can leave dpkg's update journal (`/var/lib/dpkg/updates/`) non-empty with `--audit` clean — and that journal is exactly what apt refuses on.** MEASURED 2026-10-05 on demo-hp (Part E, the night's A1 by day, agent v0.144.1): crash at 06:13:55 UTC while dpkg ran; back by itself in 37 s (crash guard armed, 1 unclean boot); at boot `dpkg --audit` clean, all 13 packages `ii` (1 new, 12 old), but `/var/lib/dpkg/updates/` held 3 files; the next pass logged `REPAIR configured=0 fixed=0`, then `FAILED rc=100 step=install` with `E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a'`; operator mail `os_update_failed` (true). A by-hand `dpkg --configure -a` (rc 0) and the next pass finished (12 installed). Cause: R-845's speed-up skips the repair on a clean audit (`configs/felhom-os-apply` `repair()`). Without the fix a box that loses power mid-update fails its OS leg every night and mails the operator every night. Fix direction: also repair when `/var/lib/dpkg/updates/` is non-empty (one `ls`, no cost on a clean pass), pinned by a test with that exact state. `audits/night-fixes-2026-10-05/partE/` | **READY — owner: CC** (next agent release) | — | fix + the crash-state test | CC |
|
||||
|
||||
## Monitoring & notifications — 25 rows (P2 2, P3 16, P4 7)
|
||||
## Monitoring & notifications — 27 rows (P2 3, P3 17, P4 7)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -364,6 +362,8 @@ stopping line that lies.
|
||||
| **R-348** | Monitoring & notifications | P4 | **Every agent restart blanks the reported backup list for up to ~18 hours, and the comment that covers it says "unaffected".** Observed 2026-08-20 while deploying R-344: the first host reports after `demo-hp`'s agent restart carry **`0 backups`** (11:15:50 and 11:30:52 CEST, two consecutive), while the box's own `pvesm list` shows archives present on **both** tiers. `internal/backup/store.go`'s `Store` is in-memory and `byTarget` is repopulated only when a backup **runs** — daily for the local tier, weekly for offsite — so the field reads 0 until the next run. `restore_tests` did **not** blank, because that half has a durable on-disk companion (`RestoreTestState`, R-189). **It blinds no alarm, and that was CHECKED rather than assumed.** `hub/internal/monitor/deadline.go` scans back over stored reports with a 7-day `backupEvidenceLookback` whose own comment names this exact case — *"when the LATEST report carries none... and against an agent that stayed restarted for days"* — and `pbs_snapshots` stayed populated at 2 regardless. So this is an observability wart, **not** a safety hole, and it is filed at that severity deliberately. **What is actually wrong is the comment.** The `Store` doc says *"Backups are unaffected — their freshness has a ground truth on the storage (R-84)"*. That is true of the **consequence** and false of the **field**, and it sits three lines below a paragraph explaining that the very same sentence about restore-tests *"used to be here and it is now FALSE"* — so the file already carries one correction of this shape and invites the next reader to trust the surviving half. | **READY (XS) — NEW 2026-08-20** | — | Say what is measured: the field IS lost on restart and repopulates only when a backup runs; the freshness VERDICT is unaffected because the hub looks back 7 days. **Name `backupEvidenceLookback` in the comment** so the cross-repo dependency is visible from the agent side — today the agent's claim of safety rests on a hub constant it does not mention. Per the workspace rule, a comment asserting an invariant needs a test pinning it: the pin belongs on the HUB side, asserting the verdict survives a report carrying `backups: []`. | CC |
|
||||
| **R-371** | Monitoring & notifications | P4 | **The off-site tier is the only backup tier that announces nothing on success.** Written down 2026-08-05 in `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md:508-513` and explicitly *"recorded, not filed"*: the off-site run emits **no hub event at all**, while both lesser tiers do (`db_dump_completed`, `crossdrive_completed`). Failures are covered by `backup_run_failures` and staleness by the hub's 8-day tier deadline, which is why it was judged a wrinkle. **Still true 2026-08-22** — the 2026-08-21 drill's own event dump shows `db_dump_completed` and six `crossdrive_completed` rows and no off-site success event. **Age when filed: 17 days.** | **OPEN — LOW** | — | Either emit one, or record deliberately that the highest-value tier is silent on success and say why. | CC |
|
||||
| **R-856** | Monitoring & notifications | P4 | **A crash restart reaches the household twice: the hub's "restarted after an unexpected stop" line AND the controller's app mails.** 2026-10-04 crash-guard test on demo-hp: after the third crash and the power-on, the controller sent `app_start_failed` (operator) and `app_stopped_unhealthy` (operator AND the household's address) for apps that were still coming up. Each is true on its own; the app ladder has no "the host just crashed" suppression like its boot grace for an ordinary restart (`08` §5). A design question for the operator, not a defect yet. `audits/os-docker-crash-2026-10-04/partC/c4-hub-events.txt` | **READY — operator decision** | — | — | operator |
|
||||
| **R-872** | Monitoring & notifications | P2 | **A box that is off every night never raises a missed-backup alarm: the 05:00 deadline check skips every customer whose node is `down`, so missing database dumps, second copies and off-site copies stay silent indefinitely; the only nightly signal is `node_down`.** MEASURED 2026-10-05 05:00 Budapest, hub log: `Deadline check: … 0 backup missed … 1 skipped (down)` — the skipped one is Tester 2, off since 18:06 UTC (`hub/internal/monitor/deadline.go` ~360: `if st == "down" \|\| st == StateDisabled { skipped++; continue }`). R-195 / R-321's shape again — a skip keyed off the wrong fact: "down now" was meant to avoid a double alarm, but a box down at every deadline is never checked at all. Fix direction (after R-871): count the days since the last success regardless of the node state, and alarm on N missed nights. `audits/night-fixes-2026-10-05/partF/FINDINGS.md` | **READY — owner: CC** (after R-871's decision) | R-871 | — | CC |
|
||||
| **R-873** | Monitoring & notifications | P3 | **A household whose box is off every night on purpose is mailed "Your server cannot be reached." every night and "reachable again" every morning; the operator gets about 6 mails a day for it.** FOUND 2026-10-05 (Part F spike): `node_down` after 90 min, mailed to the customer (Tester 2: 2026-10-04 19:36:42 UTC); the 6 h cooldown does not stop a daily repeat (`hub/internal/notify/dispatcher.go`). The R-285 gap (no notion of expected downtime) now reaching a real household. Fix direction depends on R-871: if B, a per-box "off at night" setting that quiets node_down within a nightly window; if A, the same plus the catch-up. `audits/night-fixes-2026-10-05/partF/FINDINGS.md` | **READY — owner: CC** (after R-871's decision) | R-871, R-285 | — | CC |
|
||||
|
||||
## Hub & operator — 23 rows (P2 1, P3 7, P4 15)
|
||||
|
||||
@@ -405,7 +405,7 @@ stopping line that lies.
|
||||
| **R-793** | Business & legal | P4 | **[P3-LOW] Enterprise / BUSL code ships inside four open images — Cal.com and Docmost (EE folders, off without a key), Outline (BUSL-1.1: no commercial "Document Service"), meilisearch v1.36 in Wanderer (EE modules).** READ 2026-10-02 (`audits/licences-2026-10-02/TABLE.md`). Each is fine as the catalog runs them: no EE key, the household's own Outline is not a Document Service, Wanderer uses plain search. **Watch:** never turn on an EE feature, never switch Karakeep's/Wanderer's meilisearch to the `-enterprise` image, and re-read on each major. | **WATCHING — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: a watch item; nothing is wrong as the catalog runs them.** | — | — | CC |
|
||||
| **R-794** | Business & legal | P4 | **[P3-LOW] redis 7.4 (RSALv2 / SSPL, not OSI) runs as a private cache in seven apps: dawarich, docmost, immich, nextcloud, outline, paperless-ngx, romm.** READ 2026-10-02 (`audits/licences-2026-10-02/TABLE.md`). Read as permitted (a private cache only its app uses is not Redis offered as a service — inferred). Valkey (BSD-3) or redis 8 (AGPL option) removes the question. **Needs:** a ladder step per app to valkey or redis 8, through the harness — no hurry. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: the row itself says no hurry; usage read as permitted.** | — | — | CC |
|
||||
|
||||
## Process & tooling — 87 rows (P3 4, P4 83)
|
||||
## Process & tooling — 86 rows (P3 4, P4 82)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -413,7 +413,6 @@ stopping line that lies.
|
||||
| **R-578** | Process & tooling | P3 | **[P3-LOW] A helper that takes the settings lock must never be called from inside a settings callback — there is no gate, only one test in one package.** FOUND 2026-09-18 the hard way, by localisation slice 2 release C introducing exactly that: `UpdateOffboxStatus` holds the settings WRITE lock while it runs its callback, `boxLang()` reads the language through the READ lock, and `sync.RWMutex` is not reentrant — so the off-site run's final status write DEADLOCKED, **holding the settings lock**, which would wedge everything else on that box that touches `settings.json`. The only symptom was `go test ./internal/backup/` going from 8 minutes to a 25-minute timeout. Fixed by hoisting the language resolution; `TestNoteHelpersAreNotCalledUnderTheSettingsLock` (internal/backup) now names the file and line in a second. **What is still open:** that test covers `internal/backup` only, and it knows only the `note`/`noteErr`/`boxLang` helpers. Any other settings-reading helper, in any other package, can make the same mistake with nothing to catch it but a hang. **Fix shape:** promote it to a gate over every package, keyed on "a call to a method that reads settings, inside a literal passed to a `settings.Update*` function"; or give `Settings` a re-entrant read path and remove the class. | **READY - rank P3-LOW; owner: CC** | — | — | CC |
|
||||
| **R-586** | Process & tooling | P3 | **[P2-MED] The ISO bootstrap harness captured the console to a FILE, so each banner erased the one before it — two checks were RED for two days and nobody saw.** FOUND 2026-09-18 starting slice 4 (R-559): running `scripts/iso/test/bootstrap-modes.sh` unchanged at `183727db9c44` reported `FAIL: R-496: banner painted to the console seam` and `FAIL: R-496: banner names the Tulajdonosi jelmondat`. **Cause:** the script paints each banner with `> "$CONSOLE_DEV"`. On a real console that is a character device and truncation is a no-op, so every paint appears; pointed at a plain file, as the harness did, each paint TRUNCATES. Commit `c033b3b` (ISO 1.28.0, R-535, 2026-09-16) added `print_bound_banner`, which paints immediately after the pairing banner in the same invocation — from that commit the pairing banner was wiped before the check read it. `c033b3b` did not touch the harness. **Why it survived: the harness is in NO gate and NO CI run** — not in `repo_gates.py`, not in `.gitea/`; it runs only when a person runs it, and between 09-16 and 09-18 nobody did. FIXED in the same session: the harness points `FELHOM_CONSOLE_DEV` at a FIFO with a background reader, which restores device semantics (opening a FIFO with `>` truncates nothing) and lets a test see EVERY paint — which slice 4's golden checks then needed anyway. Production code untouched. **What is still open: the harness remains outside every gate.** It needs a container, so it cannot join `repo_gates.py --fast`, which is what both the pre-push hook and CI run — meaning a non-fast entry would still never execute. **Fix shape:** either give CI a container-capable job that runs it, or make the ISO release gate's G16 the place it is required (done for G16 this session — so it now runs at least once per ISO release, which is better than never but later than a push). | **READY - rank P2-MED; owner: CC** **Re-ranked 2026-10-03: P2->P3: test harness gap; now required at each ISO release gate.** | — | — | CC |
|
||||
| **R-733** | Process & tooling | P3 | **[P3-LOW] The test bench has NO swap and the boxes have 512 MiB — so a box proof can pass on swap where the bench fails, and nobody records whether a customer guest has swap.** MEASURED 2026-09-30 (R-732): immich's first start was OOM-killed 61–104 times on the bench (swap 0) and passed on 9202 by swapping ~108 MB; the bench given 512 MiB swap passed too. demo-hp 9201, 9202 and demo-felhom 9201 all read `swap: 512`; the golden's guest config is not recorded in its bake evidence, so a customer guest's swap is NOT measured. The harness's memory watch judges `anon` against the limit and never reads `memory.swap.current`. **Needs:** the golden's `swap` read and recorded; the box walk and the harness report `memory.swap.peak` beside `anon`; a decision whether proofs run with swap off (the stricter venue, as R-732's fix was proven). | **READY — rank P3-LOW; owner: CC (harness + golden evidence)** | — | — | CC |
|
||||
| **R-866** | Process & tooling | P4 | **The debug OS pass (`felhom-agent --selftest=os-update`) cannot run while the hub is unreachable: it fetches the plan from the hub and stops, where the daemon's own leg uses its saved plan.** 2026-10-04 night drill A3 (Tester 1 box, hub blocked): `selftest=os-update: desired state: hub: transport error … connect: invalid argument`. So "an OS pass with the hub away" can be exercised only by the daemon's leg, which the 20-hour gap holds back on a box that just ran. Fix direction: the selftest falls back to the last saved block (and says so), or a documented way to trigger the daemon's leg. `audits/night-2026-10-04/accidents/a3-*.txt` | **READY — owner: CC** | — | — | CC |
|
||||
| **R-93** | Process & tooling | P4 | `drill-r50` is both a blocked customer and the only drift fixture **FACT 2026-09-13 (R-461): the fixture is GONE — `qm list` is empty on both demo boxes, so neither option is available and the row's premise no longer holds; the operator decides whether that closes it or reopens it as "build a drift fixture".** | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
|
||||
| **R-129** | Process & tooling | P4 | **Every doc says demo-hp has "no baked SSH key"** and needs the G1 break-glass password — but `ssh -o BatchMode=yes demo-hp` authenticated **by key**, first try, 2026-07-31 | READY (XS) | — | Stale in the expensive direction: a session that believes it sends itself to the hub vault for a credential it does not need. Verify who owns the key and when it landed, then correct `CLAUDE.md`, `runbooks/target-selection.md:41-42`, `runbooks/workspace-CLAUDE.md` and `felhom-agent/CLAUDE.md` together — or remove the key if it was not deliberate | CC |
|
||||
| **R-161** | Process & tooling | P4 | **The volume-persistence gate is enforced by CONVENTION, not automatically.** The catalog repo has no CI of any kind (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none exists). | **OPEN** — **REDUCED SCOPE — open** (operator ruling 2026-08-02) | a second person touching templates | **RULED. Both obvious enforcement points were rejected for measured reasons.** *Controller-side at template load:* rejected because such a check can only read the file, and a static audit of all 53 templates reports the catalog clean **including papra** — **it would pass on the exact defect it exists to catch**; the property is decidable only at runtime. *CI:* rejected for now — neither repo has any, and there are no users yet. **SHIPPED instead** (`app-catalog-felhom.eu` `fd7747d`): `scripts/catalog_gates.py`, ONE entry point running all three gates, non-zero exit on any failure, **mandated in the catalog's `CLAUDE.md`** the way `site_gates.py` is. Rationale for the record: of this project's gates, the only ones that ever get run are those with a single entry point named in a CLAUDE.md — `site_gates.py` is run, R-29's three orphans are named nowhere and have stopped nothing. **What remains open is only the automatic half:** this is convention, run by a person, and that is sufficient while one person touches templates. Revisit when a second does **UPDATE 2026-08-02:** `catalog_gates.py` gained `--fast` (gate 1 only — the network and runtime gates are deliberately NOT in a hook: a push that pulls images and starts containers gets bypassed within a week and the bypass becomes the habit) and `.githooks/pre-push` now runs it. **The automatic half now has a designated successor row: R-168** (Gitea Actions runner). This row stays open at its reduced scope — the runtime gate remains a deliberate periodic run **UPDATE 2026-08-02 (second):** the automatic half now EXISTS — R-168's runner executes `catalog_gates.py --fast` on every push to this repo (measured: run #1, `image-pin gate OK — 53 templates`, with the two runtime gates announced as skipped and their own output absent from the log). This row's *original* scope — the RUNTIME volume-persistence gate — is deliberately still NOT automatic and should stay that way: CI that pulls 53 images on every push gets disabled. It remains a periodic run | operator |
|
||||
@@ -507,5 +506,4 @@ stopping line that lies.
|
||||
the R-row. Duplicating them here would create the second source this design avoids. -->
|
||||
| item | due (UTC) | what to measure |
|
||||
|---|---|---|
|
||||
| R-95 | 2026-10-12 | the first weekly clean-up that removes snapshots, on demo-hp and demo-felhom: window log lines, the count before and after, the hub's count check quiet, the key file clean (detail in the R-95 row) |
|
||||
<!-- DUE-CHECKS-END -->
|
||||
@@ -17,3 +17,18 @@ normal running.
|
||||
4. **Check:** `sysctl kernel.panic` = 10; the System page shows "armed".
|
||||
|
||||
The numbers live in `/etc/felhom/crash-guard.conf` (LIMIT, WINDOW_MINUTES, PANIC_SECONDS, REARM_HOURS).
|
||||
|
||||
## After a crash in the middle of an OS update (R-876 — until the wrapper repairs this itself)
|
||||
|
||||
Measured 2026-10-05 on demo-hp (`audits/night-fixes-2026-10-05/partE/`): after the crash `dpkg --audit` is clean, but
|
||||
`/var/lib/dpkg/updates/` holds files, and every OS pass fails with `E: dpkg was interrupted, you must manually run
|
||||
'sudo dpkg --configure -a'` (operator mail `os_update_failed`). On the box's host, as root:
|
||||
|
||||
```bash
|
||||
pct exec 9201 -- ls /var/lib/dpkg/updates/ # non-empty = this case
|
||||
pct exec 9201 -- env DEBIAN_FRONTEND=noninteractive dpkg --configure -a --force-confold # rc 0
|
||||
pct exec 9201 -- dpkg --audit # empty
|
||||
```
|
||||
|
||||
The next pass (the night's, or `--selftest=os-update -vmid 9201` as `felhom-agent`) then installs the rest. The same
|
||||
applies to the host layer (run the commands on the host itself).
|
||||
@@ -170,6 +170,17 @@ return. Its hub customer, Storage Box sub-account (`u629488-sub2`, which never h
|
||||
removed through the hub's own customer delete; ep0 held nothing of it. The audit trail stays (events, the
|
||||
deletion and reset tombstones). Record: `audits/RETIRE-peti-2026-09-25.md`.
|
||||
|
||||
### `Tester 2` (`Tester-2-be8404`, hub customer `Tester-2`) — **Tier 2, a volunteer household** (operator rulings 2026-10-04, 2026-10-05)
|
||||
|
||||
- **What it is:** a volunteer household's box. **It is a LAPTOP that is switched OFF at night** (operator,
|
||||
2026-10-05) — it is not broken. Its absence every night is expected; a night of `DOWN` / stale on the hub is
|
||||
the normal state, not an incident. What such a box misses is the subject of `audits/night-fixes-2026-10-05/partF/`.
|
||||
- **Freely:** read its hub records.
|
||||
- **Care:** only the acts the operator ruled: the signed agent update to the vouched agent, the config bundle by the
|
||||
signed route (after the operator's one-time bootstrap, R-862), the live-restore reload.
|
||||
- **Forbidden:** any crash, reboot or package change; any drill. **Because it is a real household's machine.**
|
||||
CC has no route to it (its door admits only the operator's WireGuard peer).
|
||||
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2, PROTECTED** (operator ruling 2026-08-03)
|
||||
|
||||
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
06:01:28 anonymous round trip
|
||||
HTTP 200 648148917 bytes
|
||||
ff7a174ffe58360e4f655e5a48de17b6aa51c1521199ef3cddd81140ad79a405
|
||||
@@ -0,0 +1,62 @@
|
||||
# Golden 0.294.0 — bake + publish + vouch, 2026-10-05
|
||||
|
||||
Procedure: `documentation/runbooks/RUNBOOK-manual-build.md` §4.0 and §4.1 steps 1–5, in the drill VM on DooPlex.
|
||||
|
||||
| | Previous (`../golden-0.293.0-2026-10-04/`) | This bake |
|
||||
|---|---|---|
|
||||
| `build-golden.sh` | v3.2.0 (sha256 `645b3b659cba…`) | same file, unchanged (sha256 `645b3b659cba…`; VM copy matched) |
|
||||
| Controller | `felhom-controller:0.293.0` | **`felhom-controller:0.294.0`** (MinAgent 0.131.0, unchanged) |
|
||||
| Docker engine | the operator-approved set `os-docker-20261004-142842` | same pinned set (still in force) |
|
||||
| Guest packages | template | template — `GOLDEN_GUEST_PKGS` EMPTY: no guest release is in force (the earliest real approval is after ~11:07 UTC today, `audits/night-2026-10-04/PREDICTION.md`) |
|
||||
|
||||
## Launch
|
||||
|
||||
- Drill VM reverted to `virgin`, cold-booted per §4.0 (05:50:06 UTC); `pveversion` = `pve-manager/9.2.2`.
|
||||
- `pveam update` → `update successful`; template `debian-13-standard_13.6-1_amd64.tar.zst` (the only `_amd64`
|
||||
debian-13 entry), downloaded, `checksum verified`.
|
||||
- `/root/bake-run.sh` in the VM reads the token from the file; launched as transient unit `golden-bake`.
|
||||
- Token copied file → file (`scp`). `systemctl show golden-bake -p Environment -p ExecStart | grep -c -F <token>` =
|
||||
**0** (control with the token appended = **1**).
|
||||
|
||||
## Pass markers (from `bake.log`)
|
||||
|
||||
```
|
||||
[golden] Docker engine set PINNED to the approved release: containerd.io=2.3.6-1~debian.13~trixie … docker-ce=5:29.8.2-1~debian.13~trixie …
|
||||
[golden] no approved guest release given - the template versions stay; first-night count vs an approved release: n/a
|
||||
[golden] pending Debian upgrades in the baked guest (what a FUTURE approval may bring): 49
|
||||
docker OK (overlay2; data-root /var/lib/docker)
|
||||
live-restore: on
|
||||
INFO: including mount point rootfs ('/') in backup
|
||||
INFO: including mount point mp0 ('/var/lib/felhom') in backup
|
||||
[golden] pre-delete existing: HTTP 404 (404/204 expected)
|
||||
[golden] upload OK (HTTP 201)
|
||||
GOLDEN_VERSION=0.294.0
|
||||
GOLDEN_SHA256=ff7a174ffe58360e4f655e5a48de17b6aa51c1521199ef3cddd81140ad79a405
|
||||
```
|
||||
|
||||
No `excluding` and no `FATAL` in the log.
|
||||
|
||||
## Round trip
|
||||
|
||||
Anonymous GET of `…/generic/felhom-golden/0.294.0/golden.tar.zst`: HTTP 200, **648148917 bytes**, sha256
|
||||
`ff7a174ffe58360e4f655e5a48de17b6aa51c1521199ef3cddd81140ad79a405` = the printed sha (`02-round-trip.txt`).
|
||||
|
||||
## Secrets
|
||||
|
||||
Saved-log leak grep for the literal token: **0**; positive control (a throwaway copy with the token appended): **1**,
|
||||
copy shredded.
|
||||
|
||||
## Vouch (step 5)
|
||||
|
||||
`POST /configuration/artifacts` (operator Basic auth, the form the Configuration page posts): agent **0.144.1**
|
||||
(sha256 `6ccd521d…`), golden **0.294.0** (sha256 `ff7a174f…`), `min_agent` **0.131.0**, wrapper sha empty (as before).
|
||||
→ `303 /configuration?flash=artifacts_set`; hub log `Artifact manifest set: agent=0.144.1 golden=0.294.0
|
||||
min_agent="0.131.0" wrapper_sha=false bundle_sha="e89a9ddf…"` (the agent's config bundle, resolved by the hub).
|
||||
The controller floor stays per customer (demo-hp, demo-felhom, tester-1 at 0.294.0); the global floor is unchanged,
|
||||
so Tester 2's controller does not move.
|
||||
|
||||
## Teardown
|
||||
|
||||
`pct destroy 9100 --purge`; `shred -u` of the token, the runner script and the log in the VM (log copied off first);
|
||||
`poweroff`; qemu gone (`ps -eo comm | grep -c qemu-system-x86` = 0); `qemu-img snapshot -a virgin`. Host: nothing
|
||||
provisioned.
|
||||
@@ -0,0 +1,344 @@
|
||||
[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.294.0
|
||||
[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
|
||||
Logical volume "vm-9100-disk-0" created.
|
||||
Logical volume pve/vm-9100-disk-0 changed.
|
||||
Creating filesystem with 8388608 4k blocks and 2097152 inodes
|
||||
Filesystem UUID: 0467cbf1-4699-4396-9925-c8870a3ddca9
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
4096000, 7962624
|
||||
Logical volume "vm-9100-disk-1" created.
|
||||
Logical volume pve/vm-9100-disk-1 changed.
|
||||
Creating filesystem with 6291456 4k blocks and 1572864 inodes
|
||||
Filesystem UUID: 36159b8c-f711-4353-99f0-24922d38bffd
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
|
||||
Total bytes read: 553512960 (528MiB, 81MiB/s)
|
||||
Detected container architecture: amd64
|
||||
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
|
||||
done: SHA256:c4vE0ww5wxb3En7uuEI7tbUr4u06qjdbi+gOzBJIVJg root@felhom-golden
|
||||
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
|
||||
done: SHA256:0m2NetsaZUlBMNwRuMlBJRPm0PjoUs9FOrq8Jug8NeQ root@felhom-golden
|
||||
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
|
||||
done: SHA256:EUNeZpR9hthJ93H+RFsq9DUtLiHu1O25NRjuNm79Kog root@felhom-golden
|
||||
[golden] starting + installing Docker (official repo, trixie channel) …
|
||||
[golden] Docker engine set PINNED to the approved release: containerd.io=2.3.6-1~debian.13~trixie docker-buildx-plugin=0.37.1-1~debian.13~trixie docker-ce=5:29.8.2-1~debian.13~trixie docker-ce-cli=5:29.8.2-1~debian.13~trixie docker-ce-rootless-extras=5:29.8.2-1~debian.13~trixie docker-compose-plugin=5.6.0-1~debian.13~trixie
|
||||
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = (unset),
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to the standard locale ("C").
|
||||
locale: Cannot set LC_CTYPE to default locale: No such file or directory
|
||||
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
|
||||
locale: Cannot set LC_ALL to default locale: No such file or directory
|
||||
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = (unset),
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to the standard locale ("C").
|
||||
locale: Cannot set LC_CTYPE to default locale: No such file or directory
|
||||
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
|
||||
locale: Cannot set LC_ALL to default locale: No such file or directory
|
||||
installed: containerd.io 2.3.6-1~debian.13~trixie
|
||||
installed: docker-buildx-plugin 0.37.1-1~debian.13~trixie
|
||||
installed: docker-ce 5:29.8.2-1~debian.13~trixie
|
||||
installed: docker-ce-cli 5:29.8.2-1~debian.13~trixie
|
||||
installed: docker-ce-rootless-extras 5:29.8.2-1~debian.13~trixie
|
||||
installed: docker-compose-plugin 5.6.0-1~debian.13~trixie
|
||||
[golden] no approved guest release given - the template versions stay; first-night count vs an approved release: n/a
|
||||
[golden] pending Debian upgrades in the baked guest (what a FUTURE approval may bring): 49
|
||||
[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
|
||||
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
|
||||
[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
|
||||
Unable to find image 'hello-world:latest' locally
|
||||
latest: Pulling from library/hello-world
|
||||
4f55086f7dd0: Pulling fs layer
|
||||
4f55086f7dd0: Verifying Checksum
|
||||
4f55086f7dd0: Download complete
|
||||
4f55086f7dd0: Pull complete
|
||||
Digest: sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe895f9cc8f8
|
||||
Status: Downloaded newer image for hello-world:latest
|
||||
docker OK (overlay2; data-root /var/lib/docker)
|
||||
live-restore: on
|
||||
/var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
|
||||
/mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
|
||||
both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
|
||||
[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.294.0 (no registry cred at deploy) …
|
||||
|
||||
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
|
||||
Configure a credential helper to remove this warning. See
|
||||
https://docs.docker.com/go/credential-store/
|
||||
|
||||
0.294.0: Pulling from admin/felhom-controller
|
||||
774043ccc8cc: Pulling fs layer
|
||||
ab6b448d4be9: Pulling fs layer
|
||||
23a5bfa58353: Pulling fs layer
|
||||
862a57157567: Pulling fs layer
|
||||
60379d5ca023: Pulling fs layer
|
||||
9c190e04c57b: Pulling fs layer
|
||||
862a57157567: Waiting
|
||||
60379d5ca023: Waiting
|
||||
9c190e04c57b: Waiting
|
||||
774043ccc8cc: Verifying Checksum
|
||||
774043ccc8cc: Download complete
|
||||
862a57157567: Verifying Checksum
|
||||
862a57157567: Download complete
|
||||
23a5bfa58353: Verifying Checksum
|
||||
23a5bfa58353: Download complete
|
||||
60379d5ca023: Verifying Checksum
|
||||
60379d5ca023: Download complete
|
||||
9c190e04c57b: Verifying Checksum
|
||||
9c190e04c57b: Download complete
|
||||
ab6b448d4be9: Verifying Checksum
|
||||
ab6b448d4be9: Download complete
|
||||
774043ccc8cc: Pull complete
|
||||
ab6b448d4be9: Pull complete
|
||||
23a5bfa58353: Pull complete
|
||||
862a57157567: Pull complete
|
||||
60379d5ca023: Pull complete
|
||||
9c190e04c57b: Pull complete
|
||||
Digest: sha256:2e85d01377a6d7c5872ffcbdd5c173afc0adc53eade9fe38c12aa17b634ea589
|
||||
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.294.0
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.294.0
|
||||
[golden] asking the controller which infra images it manages …
|
||||
[golden] baking infra images (4): traefik:v3.7.13 cloudflare/cloudflared:2026.9.3 gtstef/filebrowser:1.5.6-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 …
|
||||
v3.7.13: Pulling from library/traefik
|
||||
e2de96513ba9: Pulling fs layer
|
||||
b686a4f73445: Pulling fs layer
|
||||
78cb21c375ca: Pulling fs layer
|
||||
acb2f33459b1: Pulling fs layer
|
||||
acb2f33459b1: Waiting
|
||||
b686a4f73445: Verifying Checksum
|
||||
b686a4f73445: Download complete
|
||||
acb2f33459b1: Verifying Checksum
|
||||
acb2f33459b1: Download complete
|
||||
e2de96513ba9: Verifying Checksum
|
||||
e2de96513ba9: Download complete
|
||||
78cb21c375ca: Verifying Checksum
|
||||
78cb21c375ca: Download complete
|
||||
e2de96513ba9: Pull complete
|
||||
b686a4f73445: Pull complete
|
||||
78cb21c375ca: Pull complete
|
||||
acb2f33459b1: Pull complete
|
||||
Digest: sha256:24841fe2de7304c149343d877d2923b4c8800a38ba015dea9174c23b20e344a0
|
||||
Status: Downloaded newer image for traefik:v3.7.13
|
||||
docker.io/library/traefik:v3.7.13
|
||||
2026.9.3: Pulling from cloudflare/cloudflared
|
||||
2cc7ee286bf3: Pulling fs layer
|
||||
c172f21841df: Pulling fs layer
|
||||
218cf840d0d9: Pulling fs layer
|
||||
f6069939f718: Pulling fs layer
|
||||
d6b1b89eccac: Pulling fs layer
|
||||
2780920e5dbf: Pulling fs layer
|
||||
7c12895b777b: Pulling fs layer
|
||||
3214acf345c0: Pulling fs layer
|
||||
52630fc75a18: Pulling fs layer
|
||||
dd64bf2dd177: Pulling fs layer
|
||||
b839dfae01f6: Pulling fs layer
|
||||
ebddc55facdc: Pulling fs layer
|
||||
c4bc6f35ff5e: Pulling fs layer
|
||||
b96fe2995f90: Pulling fs layer
|
||||
58c0c263dc73: Pulling fs layer
|
||||
bd8962e29291: Pulling fs layer
|
||||
cac2ae0193cb: Pulling fs layer
|
||||
f0383d5ebc47: Pulling fs layer
|
||||
f6069939f718: Waiting
|
||||
d6b1b89eccac: Waiting
|
||||
2780920e5dbf: Waiting
|
||||
7c12895b777b: Waiting
|
||||
3214acf345c0: Waiting
|
||||
52630fc75a18: Waiting
|
||||
dd64bf2dd177: Waiting
|
||||
b839dfae01f6: Waiting
|
||||
ebddc55facdc: Waiting
|
||||
c4bc6f35ff5e: Waiting
|
||||
b96fe2995f90: Waiting
|
||||
58c0c263dc73: Waiting
|
||||
bd8962e29291: Waiting
|
||||
cac2ae0193cb: Waiting
|
||||
f0383d5ebc47: Waiting
|
||||
2cc7ee286bf3: Verifying Checksum
|
||||
2cc7ee286bf3: Download complete
|
||||
218cf840d0d9: Verifying Checksum
|
||||
218cf840d0d9: Download complete
|
||||
c172f21841df: Verifying Checksum
|
||||
c172f21841df: Download complete
|
||||
f6069939f718: Verifying Checksum
|
||||
f6069939f718: Download complete
|
||||
2780920e5dbf: Verifying Checksum
|
||||
2780920e5dbf: Download complete
|
||||
d6b1b89eccac: Verifying Checksum
|
||||
d6b1b89eccac: Download complete
|
||||
2cc7ee286bf3: Pull complete
|
||||
3214acf345c0: Verifying Checksum
|
||||
3214acf345c0: Download complete
|
||||
7c12895b777b: Verifying Checksum
|
||||
7c12895b777b: Download complete
|
||||
52630fc75a18: Verifying Checksum
|
||||
52630fc75a18: Download complete
|
||||
dd64bf2dd177: Verifying Checksum
|
||||
dd64bf2dd177: Download complete
|
||||
b839dfae01f6: Verifying Checksum
|
||||
b839dfae01f6: Download complete
|
||||
c172f21841df: Pull complete
|
||||
ebddc55facdc: Verifying Checksum
|
||||
ebddc55facdc: Download complete
|
||||
c4bc6f35ff5e: Verifying Checksum
|
||||
c4bc6f35ff5e: Download complete
|
||||
b96fe2995f90: Verifying Checksum
|
||||
b96fe2995f90: Download complete
|
||||
bd8962e29291: Verifying Checksum
|
||||
bd8962e29291: Download complete
|
||||
cac2ae0193cb: Verifying Checksum
|
||||
cac2ae0193cb: Download complete
|
||||
58c0c263dc73: Verifying Checksum
|
||||
58c0c263dc73: Download complete
|
||||
218cf840d0d9: Pull complete
|
||||
f0383d5ebc47: Verifying Checksum
|
||||
f0383d5ebc47: Download complete
|
||||
f6069939f718: Pull complete
|
||||
d6b1b89eccac: Pull complete
|
||||
2780920e5dbf: Pull complete
|
||||
7c12895b777b: Pull complete
|
||||
3214acf345c0: Pull complete
|
||||
52630fc75a18: Pull complete
|
||||
dd64bf2dd177: Pull complete
|
||||
b839dfae01f6: Pull complete
|
||||
ebddc55facdc: Pull complete
|
||||
c4bc6f35ff5e: Pull complete
|
||||
b96fe2995f90: Pull complete
|
||||
58c0c263dc73: Pull complete
|
||||
bd8962e29291: Pull complete
|
||||
cac2ae0193cb: Pull complete
|
||||
f0383d5ebc47: Pull complete
|
||||
Digest: sha256:072c067d25ccbe61d46e18f0d0723255f2bb5304f7317caa95b27031520ff92c
|
||||
Status: Downloaded newer image for cloudflare/cloudflared:2026.9.3
|
||||
docker.io/cloudflare/cloudflared:2026.9.3
|
||||
1.5.6-stable: Pulling from gtstef/filebrowser
|
||||
55afa1ecc21d: Pulling fs layer
|
||||
8ed8f35f8d4f: Pulling fs layer
|
||||
989b226a579c: Pulling fs layer
|
||||
660aeead31d5: Pulling fs layer
|
||||
4f4fb700ef54: Pulling fs layer
|
||||
adce24567e4c: Pulling fs layer
|
||||
f17ea56b313b: Pulling fs layer
|
||||
6b6f3b3efe88: Pulling fs layer
|
||||
4ed1ca4f3fce: Pulling fs layer
|
||||
e6fc9c6a5757: Pulling fs layer
|
||||
d47782d1182a: Pulling fs layer
|
||||
660aeead31d5: Waiting
|
||||
4f4fb700ef54: Waiting
|
||||
adce24567e4c: Waiting
|
||||
f17ea56b313b: Waiting
|
||||
6b6f3b3efe88: Waiting
|
||||
4ed1ca4f3fce: Waiting
|
||||
e6fc9c6a5757: Waiting
|
||||
d47782d1182a: Waiting
|
||||
55afa1ecc21d: Verifying Checksum
|
||||
55afa1ecc21d: Download complete
|
||||
660aeead31d5: Verifying Checksum
|
||||
660aeead31d5: Download complete
|
||||
4f4fb700ef54: Verifying Checksum
|
||||
4f4fb700ef54: Download complete
|
||||
55afa1ecc21d: Pull complete
|
||||
989b226a579c: Verifying Checksum
|
||||
989b226a579c: Download complete
|
||||
8ed8f35f8d4f: Verifying Checksum
|
||||
8ed8f35f8d4f: Download complete
|
||||
f17ea56b313b: Verifying Checksum
|
||||
f17ea56b313b: Download complete
|
||||
4ed1ca4f3fce: Verifying Checksum
|
||||
4ed1ca4f3fce: Download complete
|
||||
6b6f3b3efe88: Verifying Checksum
|
||||
6b6f3b3efe88: Download complete
|
||||
d47782d1182a: Verifying Checksum
|
||||
d47782d1182a: Download complete
|
||||
adce24567e4c: Verifying Checksum
|
||||
adce24567e4c: Download complete
|
||||
e6fc9c6a5757: Verifying Checksum
|
||||
e6fc9c6a5757: Download complete
|
||||
8ed8f35f8d4f: Pull complete
|
||||
989b226a579c: Pull complete
|
||||
660aeead31d5: Pull complete
|
||||
4f4fb700ef54: Pull complete
|
||||
adce24567e4c: Pull complete
|
||||
f17ea56b313b: Pull complete
|
||||
6b6f3b3efe88: Pull complete
|
||||
4ed1ca4f3fce: Pull complete
|
||||
e6fc9c6a5757: Pull complete
|
||||
d47782d1182a: Pull complete
|
||||
Digest: sha256:7c5d7ac8ffda31294d278063cf9d2e04303b39e6dce1f4c691342240ca7703b8
|
||||
Status: Downloaded newer image for gtstef/filebrowser:1.5.6-stable
|
||||
docker.io/gtstef/filebrowser:1.5.6-stable
|
||||
1.1.0: Pulling from admin/felhom-samba
|
||||
897d797d2723: Pulling fs layer
|
||||
3051591aa250: Pulling fs layer
|
||||
ce57a3f93416: Pulling fs layer
|
||||
fb94eeec2fe1: Pulling fs layer
|
||||
fb94eeec2fe1: Waiting
|
||||
ce57a3f93416: Verifying Checksum
|
||||
ce57a3f93416: Download complete
|
||||
fb94eeec2fe1: Verifying Checksum
|
||||
fb94eeec2fe1: Download complete
|
||||
897d797d2723: Verifying Checksum
|
||||
897d797d2723: Download complete
|
||||
3051591aa250: Verifying Checksum
|
||||
3051591aa250: Download complete
|
||||
897d797d2723: Pull complete
|
||||
3051591aa250: Pull complete
|
||||
ce57a3f93416: Pull complete
|
||||
fb94eeec2fe1: Pull complete
|
||||
Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
|
||||
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0
|
||||
gitea.dooplex.hu/admin/felhom-samba:1.1.0
|
||||
[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
|
||||
[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
|
||||
[golden] baking the first-boot SSH host-key regeneration unit (F3) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
|
||||
[golden] identity-clean + minimize …
|
||||
[golden] stop + archive …
|
||||
INFO: including mount point rootfs ('/') in backup
|
||||
INFO: including mount point mp0 ('/var/lib/felhom') in backup
|
||||
INFO: archive file size: 618MB
|
||||
INFO: Finished Backup of VM 9100 (00:00:32)
|
||||
[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_10_05-07_55_26.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
|
||||
[golden] publishing golden (648148917 bytes, sha256 ff7a174ffe58360e…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.294.0/golden.tar.zst
|
||||
[golden] pre-delete existing: HTTP 404 (404/204 expected)
|
||||
[golden] upload OK (HTTP 201)
|
||||
GOLDEN_VERSION=0.294.0
|
||||
GOLDEN_SHA256=ff7a174ffe58360e4f655e5a48de17b6aa51c1521199ef3cddd81140ad79a405
|
||||
[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.294.0 / ff7a174ffe58360e4f655e5a48de17b6aa51c1521199ef3cddd81140ad79a405
|
||||
[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)
|
||||
Reference in new issue
Block a user