admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

night fixes 2026-10-05: R-867/R-95/R-863..R-869 closed, R-870..R-876 opened (R-876 P2: repair misses dpkg's journal); Part E crash record; Part F spike; golden 0.294.0; rulings 100-103, CC decisions 104-108; STATUS
gates / gates (push) Successful in 32s
Details

Browse Source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
authentik Default Admin
2026-10-05 08:26:23 +02:00
parent 7221ee5cd6
commit 5fd2656021
55 changed files with 4856 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-snapshots-after.json
+142
View File
@@ -0,0 +1,142 @@
[
{
"id": "6ea854132efc6053ee5f7ca900a2db6f767dd3d89250efeb52fa30e319598ed1",
"short_id": "6ea85413",
"time": "2026-08-12T15:15:33.030477238Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"drill-retained-key-20260812"
]
},
{
"id": "b4d7cb79502a6c3f772254bc79463ec726f722f185a1c10571c389516e31746d",
"short_id": "b4d7cb79",
"time": "2026-08-31T02:15:04.935400697Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "6ce44d9de433625eff342dc0234d526a11336fb9b813aaa49cd737682c07387a",
"short_id": "6ce44d9d",
"time": "2026-09-20T02:15:05.018256188Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "c7000d7de36008707347c3b6dec9c79166feeae2a517137daa2c47004221e7ef",
"short_id": "c7000d7d",
"time": "2026-09-27T02:15:05.066652132Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "8ccc027e486d4f2e1b5415e2e602e29e8acbf0001a1596ec08b00c43b148ea68",
"short_id": "8ccc027e",
"time": "2026-09-29T02:15:05.16680869Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "40826801b49bdb9a249d0d0e7576081d5c96ea2bc7d4503d9a1bce6bfb19cb60",
"short_id": "40826801",
"time": "2026-09-30T02:15:05.173694815Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "c3a3b13b9832ba3d98cc3dc047b01b376ced2c824506e8f06001d8a3a7c5660e",
"short_id": "c3a3b13b",
"time": "2026-10-01T02:15:05.016873278Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "b2606da6d28cb24ce8d5c0616ced769b1a7f7b3d0b2923fca23b4364262b7a31",
"short_id": "b2606da6",
"time": "2026-10-02T02:15:05.138940278Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "64129ff5dcb75ea1ff766605e1a0b4dc36e6a4441a0be9433fe2519e5ddd16e0",
"short_id": "64129ff5",
"time": "2026-10-03T02:15:05.108133602Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "2bb8fb60447e6387d374d98d196807dd9e32187695ddab5df85ff2a1ac856404",
"short_id": "2bb8fb60",
"time": "2026-10-03T15:05:39.219021268Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "5dd1f0b99e8ed22d7fdcc9005501e880ddd600341fe35bd6897174acdd943857",
"short_id": "5dd1f0b9",
"time": "2026-10-03T15:17:27.992813997Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "89df02373ec2ffbe4a6b02d622c30c6f5001b922fc692bb1e1b84dfb0b8d2971",
"short_id": "89df0237",
"time": "2026-10-04T02:15:03.833138622Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "4e7867af43f024c8f712417bce982c5c8085524b51e7f37040c665d0dc1146de",
"short_id": "4e7867af",
"time": "2026-10-05T02:15:03.941403075Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "47bad696fe89450b2f5db11bfbc3a958edf2796a60d851dd9d92585eb36d53fc",
"short_id": "47bad696",
"time": "2026-10-05T05:21:04.032874428Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
}
]
documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-snapshots-before.json
+152
View File
@@ -0,0 +1,152 @@
[
{
"id": "6ea854132efc6053ee5f7ca900a2db6f767dd3d89250efeb52fa30e319598ed1",
"short_id": "6ea85413",
"time": "2026-08-12T15:15:33.030477238Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"drill-retained-key-20260812"
]
},
{
"id": "b4d7cb79502a6c3f772254bc79463ec726f722f185a1c10571c389516e31746d",
"short_id": "b4d7cb79",
"time": "2026-08-31T02:15:04.935400697Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "92e49f62c017e103cf7a1f73f8982a7faa39d77eb9c7fd129b3910cc5734a262",
"short_id": "92e49f62",
"time": "2026-09-13T02:15:05.107149713Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "6ce44d9de433625eff342dc0234d526a11336fb9b813aaa49cd737682c07387a",
"short_id": "6ce44d9d",
"time": "2026-09-20T02:15:05.018256188Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "c7000d7de36008707347c3b6dec9c79166feeae2a517137daa2c47004221e7ef",
"short_id": "c7000d7d",
"time": "2026-09-27T02:15:05.066652132Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "343d57a5614f88b80a18b129b52753740c8391581073f406fe9878bf40716381",
"short_id": "343d57a5",
"time": "2026-09-28T02:15:05.279260454Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "8ccc027e486d4f2e1b5415e2e602e29e8acbf0001a1596ec08b00c43b148ea68",
"short_id": "8ccc027e",
"time": "2026-09-29T02:15:05.16680869Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "40826801b49bdb9a249d0d0e7576081d5c96ea2bc7d4503d9a1bce6bfb19cb60",
"short_id": "40826801",
"time": "2026-09-30T02:15:05.173694815Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "c3a3b13b9832ba3d98cc3dc047b01b376ced2c824506e8f06001d8a3a7c5660e",
"short_id": "c3a3b13b",
"time": "2026-10-01T02:15:05.016873278Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "b2606da6d28cb24ce8d5c0616ced769b1a7f7b3d0b2923fca23b4364262b7a31",
"short_id": "b2606da6",
"time": "2026-10-02T02:15:05.138940278Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "64129ff5dcb75ea1ff766605e1a0b4dc36e6a4441a0be9433fe2519e5ddd16e0",
"short_id": "64129ff5",
"time": "2026-10-03T02:15:05.108133602Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "2bb8fb60447e6387d374d98d196807dd9e32187695ddab5df85ff2a1ac856404",
"short_id": "2bb8fb60",
"time": "2026-10-03T15:05:39.219021268Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "5dd1f0b99e8ed22d7fdcc9005501e880ddd600341fe35bd6897174acdd943857",
"short_id": "5dd1f0b9",
"time": "2026-10-03T15:17:27.992813997Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "89df02373ec2ffbe4a6b02d622c30c6f5001b922fc692bb1e1b84dfb0b8d2971",
"short_id": "89df0237",
"time": "2026-10-04T02:15:03.833138622Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
},
{
"id": "4e7867af43f024c8f712417bce982c5c8085524b51e7f37040c665d0dc1146de",
"short_id": "4e7867af",
"time": "2026-10-05T02:15:03.941403075Z",
"hostname": "demo-felhom",
"tags": [
"felhom-offbox",
"opengist"
]
}
]
documentation/audits/night-fixes-2026-10-05/partA/demo-felhom-window.txt
+26
View File
@@ -0,0 +1,26 @@
2026-10-05T05:20:57Z
grant HTTP 200
HTTP 302
before 15 after 14
removed: [('343d57a5', '2026-09-28T02:15:05'), ('92e49f62', '2026-09-13T02:15:05')]
added by the run: [('47bad696', '2026-10-05T05:21:04')]
2026/10/05 05:16:20 [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-06 04:15 CEST
2026/10/05 05:16:35 [INFO] [bootrecon] boot window: fleet settled after 10s (3 identical samples 5s apart) — sweeping
2026/10/05 05:21:00 [INFO] [offbox] backup run started (1 app(s) toggled)
2026/10/05 05:21:01 [INFO] [offbox] pre-push dump leg completed in 892ms — snapshot pair is coherent
2026/10/05 05:21:06 [INFO] [offbox] backed up opengist (/mnt/sys_drive/felhom-data/backups/primary/opengist, 0 mandatory path(s))
2026/10/05 05:21:14 [INFO] [offbox] clean-up window 5 (after-run): 2 of 5 planned snapshot(s) removed, 16 -> 14, in 6s
2026/10/05 05:21:18 [INFO] [offbox] backup OK: 1 app(s) backed up, 14 snapshot(s), 15s
2026/10/05 05:21:18 [INFO] [offbox] manual run progress reporting ended after 18s
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
sqlite3.OperationalError: no such column: removed
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
sqlite3.OperationalError: no such column: box_reason
--- hub offsite_windows (demo-felhom, newest 3)
(5, '2026-10-05 05:21:08', '2026-10-05 05:21:15', 16, 14, 8, 'pruned', 'box')
(3, '2026-10-05 02:15:08', '2026-10-05 02:15:11', 15, 15, 7, 'guard-refused', 'box')
--- hub events since 05:20 UTC
--- hub notification_log since 05:20 UTC
documentation/audits/night-fixes-2026-10-05/partA/demo-hp-prediction.txt
+46
View File
@@ -0,0 +1,46 @@
PLAN 80ebd0d6 2026-09-13T02:16:27Z [felhom-offbox bentopdf]
PLAN 29516b0c 2026-09-13T02:16:31Z [felhom-offbox bookstack]
PLAN 21efaca4 2026-09-13T02:16:35Z [felhom-offbox calibre-web]
PLAN c8562d68 2026-09-13T02:16:39Z [felhom-offbox docmost]
PLAN b4350226 2026-09-13T02:16:44Z [felhom-offbox kimai]
PLAN 1ebc8069 2026-09-13T02:16:54Z [felhom-offbox paperless-ngx]
PLAN 6fac4ba8 2026-09-13T02:17:01Z [felhom-offbox romm]
PLAN 8cda9681 2026-09-13T02:17:08Z [felhom-offbox opengist]
PLAN 605bf9c7 2026-09-13T02:17:11Z [felhom-offbox privatebin]
PLAN 9efec80f 2026-09-28T13:42:12Z [felhom-offbox docmost]
PLAN ef800224 2026-09-28T13:42:20Z [felhom-offbox opengist]
PLAN c9c2414d 2026-09-28T13:42:24Z [felhom-offbox bentopdf]
PLAN 14981ce0 2026-09-28T13:42:27Z [felhom-offbox bookstack]
PLAN 693c93e1 2026-09-28T13:42:31Z [felhom-offbox calibre-web]
PLAN 90da3064 2026-09-28T13:42:36Z [felhom-offbox kimai]
PLAN 7db18829 2026-09-28T13:43:46Z [felhom-offbox paperless-ngx]
PLAN 57191440 2026-09-28T13:43:52Z [felhom-offbox privatebin]
PLAN 0febfb1d 2026-09-28T13:43:55Z [felhom-offbox romm]
PLAN b522e2f4 2026-10-03T02:16:49Z [felhom-offbox bentopdf]
PLAN c974cffe 2026-10-03T02:16:56Z [felhom-offbox calibre-web]
PLAN 0bffcb8f 2026-10-03T02:17:00Z [felhom-offbox paperless-ngx]
PLAN b5d27227 2026-10-03T02:17:08Z [felhom-offbox bookstack]
PLAN 5d1830e6 2026-10-03T02:17:13Z [felhom-offbox docmost]
PLAN 4a3c71f5 2026-10-03T02:17:19Z [felhom-offbox kimai]
PLAN 0a155533 2026-10-03T02:17:26Z [felhom-offbox opengist]
PLAN a5822405 2026-10-03T02:17:29Z [felhom-offbox privatebin]
PLAN 2cd36d95 2026-10-03T02:17:32Z [felhom-offbox romm]
PLAN bea276a7 2026-10-03T15:24:46Z [felhom-offbox bentopdf]
PLAN 59717090 2026-10-03T15:24:49Z [felhom-offbox calibre-web]
PLAN c6b5c67b 2026-10-03T15:24:51Z [felhom-offbox docmost]
PLAN cb83c862 2026-10-03T15:24:54Z [felhom-offbox paperless-ngx]
PLAN 38fbd3bc 2026-10-03T15:24:57Z [felhom-offbox privatebin]
PLAN 4ce84465 2026-10-03T15:24:59Z [felhom-offbox romm]
PLAN 2f72df90 2026-10-03T15:25:02Z [felhom-offbox bookstack]
PLAN 2c0385d3 2026-10-03T15:25:04Z [felhom-offbox kimai]
PLAN c43f2d0e 2026-10-03T15:25:07Z [felhom-offbox opengist]
PLAN 2d412cd5 2026-10-04T02:16:44Z [felhom-offbox bentopdf]
PLAN 93a66c3e 2026-10-04T02:16:46Z [felhom-offbox calibre-web]
PLAN 6dbbceac 2026-10-04T02:16:48Z [felhom-offbox docmost]
PLAN 84559526 2026-10-04T02:16:51Z [felhom-offbox paperless-ngx]
PLAN 2abceaa3 2026-10-04T02:16:54Z [felhom-offbox privatebin]
PLAN a86c8d7f 2026-10-04T02:16:56Z [felhom-offbox romm]
PLAN 40328365 2026-10-04T02:16:59Z [felhom-offbox bookstack]
PLAN f6c41de0 2026-10-04T02:17:01Z [felhom-offbox kimai]
PLAN d2b6725f 2026-10-04T02:17:04Z [felhom-offbox opengist]
GUARD at 2026-10-05T05:23:43Z: removes 18 [80ebd0d6 29516b0c 21efaca4 c8562d68 b4350226 1ebc8069 6fac4ba8 8cda9681 605bf9c7 9efec80f ef800224 c9c2414d 14981ce0 693c93e1 90da3064 7db18829 57191440 0febfb1d] refusal=""
documentation/audits/night-fixes-2026-10-05/partA/demo-hp-snapshots-after.json
+1272
View File
File diff suppressed because it is too large Load Diff
documentation/audits/night-fixes-2026-10-05/partA/demo-hp-snapshots-before.json
+1362
View File
File diff suppressed because it is too large Load Diff
documentation/audits/night-fixes-2026-10-05/partA/demo-hp-window.txt
+33
View File
@@ -0,0 +1,33 @@
2026-10-05T05:23:54Z
grant HTTP 200
HTTP 302
2026/10/05 05:26:03 offbox_window.go:273: [INFO] [offbox] clean-up window 6 (after-run): 18 of 54 planned snapshot(s) removed, 145 -> 127, in 10s
2026/10/05 05:26:08 offbox.go:1244: [INFO] [offbox] backup OK: 9 app(s) backed up, 127 snapshot(s), 2m7s
before 136 after 127 removed 18 added by the run 9
removed == predicted: True
removed 80ebd0d6 2026-09-13T02:16:27 ['bentopdf']
removed 29516b0c 2026-09-13T02:16:31 ['bookstack']
removed 21efaca4 2026-09-13T02:16:35 ['calibre-web']
removed c8562d68 2026-09-13T02:16:39 ['docmost']
removed b4350226 2026-09-13T02:16:44 ['kimai']
removed 1ebc8069 2026-09-13T02:16:54 ['paperless-ngx']
removed 6fac4ba8 2026-09-13T02:17:01 ['romm']
removed 8cda9681 2026-09-13T02:17:08 ['opengist']
removed 605bf9c7 2026-09-13T02:17:11 ['privatebin']
removed 9efec80f 2026-09-28T13:42:12 ['docmost']
removed ef800224 2026-09-28T13:42:20 ['opengist']
removed c9c2414d 2026-09-28T13:42:24 ['bentopdf']
removed 14981ce0 2026-09-28T13:42:27 ['bookstack']
removed 693c93e1 2026-09-28T13:42:31 ['calibre-web']
removed 90da3064 2026-09-28T13:42:36 ['kimai']
removed 7db18829 2026-09-28T13:43:46 ['paperless-ngx']
removed 57191440 2026-09-28T13:43:52 ['privatebin']
removed 0febfb1d 2026-09-28T13:43:55 ['romm']
--- hub offsite_windows (demo-hp, newest 2)
(6, '2026-10-05 05:25:53', '2026-10-05 05:26:04', 145, 127, 72, 'pruned', 'box')
(2, '2026-10-04 05:36:14', '2026-10-04 05:36:17', 127, 127, None, 'nothing', 'box')
--- hub events since 05:23 UTC
--- notification_log since 05:23 UTC
--- key audit
[{"customer":"Tester-2","lines":1,"pinned":1,"findings":null},{"customer":"demo-felhom","lines":1,"pinned":1,"findings":null},{"customer":"demo-hp","lines":1,"pinned":1,"findings":null},{"customer":"tester-1","lines":1,"pinned":1,"findings":null}]
documentation/audits/night-fixes-2026-10-05/partA/key-audit-after-felhom.json
+1
View File
@@ -0,0 +1 @@
[{"customer":"Tester-2","lines":1,"pinned":1,"findings":null},{"customer":"demo-felhom","lines":1,"pinned":1,"findings":null},{"customer":"demo-hp","lines":1,"pinned":1,"findings":null},{"customer":"tester-1","lines":1,"pinned":1,"findings":null}]
documentation/audits/night-fixes-2026-10-05/partA/lab-real-restic-vs-simulation.txt
+17
View File
@@ -0,0 +1,17 @@
R-867 lab, 2026-10-05, on DooPlex scratchpad (local repo, deleted after). restic binary copied out of
gitea.dooplex.hu/admin/felhom-controller:0.293.0: `restic 0.14.0 compiled with go1.19.8 on linux/amd64`.
Snapshot set: 2 groups (host box, tags opengist / bookstack), one snapshot per night at 02:15:05 UTC from
2026-08-20 to 2026-10-05, box OFF 2026-09-15..17 (no snapshots), a same-day manual run at 15:00 on 2026-09-10
and 2026-09-30. 92 snapshots, all UTC.
Real policy: restic forget --group-by host,tags --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --dry-run --json
-> removes 72 snapshots
Test simulation restic0140Plan (offbox_window_test.go) over the same `restic snapshots --json`:
-> removes 72 snapshots
diff of the two sorted id lists: EMPTY (identical)
The guard (v0.294.0 candidate) on that plan at now = 2026-10-05T02:20:00Z, maxRemove 500:
GUARD removes=70 refusal=""
(the 2 kept back are the 2026-09-30 02:15 copies superseded by the same-day 15:00 run — R-824's exclusion;
removed in a later window once 7 calendar days old)
documentation/audits/night-fixes-2026-10-05/partA/red-proofs.txt
+45
View File
@@ -0,0 +1,45 @@
### RED-PROOF 1 the 8-day age line back: 'if calendarDaysBefore(s.Time, now) < keepDaily {' -> 'if now.Sub(s.Time) < 8*24*time.Hour {'
offbox_window_test.go:425: the honest drop of a 7 d + 5 s snapshot must pass: ids=[] why="the policy would remove snapshot n0 from 2026-09-28T02:15:05Z — within the last 7 days kept daily and not superseded the same day, which honest retention never does"
--- FAIL: TestOffsiteGuard_RealPolicyMeasuredShapeAllowed (0.00s)
offbox_window_test.go:456: night 7 (2026-08-27T02:17:00Z): the guard refused the honest policy: the policy would remove snapshot opengist0 from 2026-08-20T02:15:00Z — within the last 7 days kept daily and not superseded the same day, which honest retention never does
--- FAIL: TestOffsiteGuard_RealPolicySixtyNightsNeverRefuses (0.00s)
offbox_window_test.go:502: weekly window on 2026-09-14 refused: the policy would remove snapshot s6 from 2026-09-07T02:15:00Z — within the last 7 days kept daily and not superseded the same day, which honest retention never does
--- FAIL: TestOffsiteGuard_RealPolicyWeeklyWindows (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.020s
FAIL
rc=1
### RED-PROOF 2 drop the future-date refusal: 's.Time.After(now.Add(offsiteGuardSkew))' -> 'false && s.Time.After(now.Add(offsiteGuardSkew))'
offbox_window_test.go:155: window close = [{ID:7 CountBefore:16 CountAfter:16 Removed:0 Outcome:guard-refused Reason:snapshot f1 (2027-01-01T03:00:00Z) is newer than the hub allows (2026-10-05T04:55:06Z)}]
--- FAIL: TestOffsiteGuard_LabThirteenFutureFakes_Refused (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.023s
FAIL
rc=1
### RED-PROOF 3 drop the newer-than-hub refusal: '!newestAllowed.IsZero() && s.Time.After' -> 'false && s.Time.After'
offbox_window_test.go:169: newest-allowed bound not enforced: ""
--- FAIL: TestOffsiteGuard_RecentRemovalRefused (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.023s
FAIL
rc=1
### RED-PROOF 4 drop the young-removal refusal: 'return nil, fmt.Sprintf("the policy would remove snapshot %s from %s — within' -> 'continue; _ = fmt.Sprintf("the policy would remove snapshot %s from %s — within'
offbox_window_test.go:165: why = ""
--- FAIL: TestOffsiteGuard_RecentRemovalRefused (0.00s)
offbox_window_test.go:232: why = ""
--- FAIL: TestOffsiteGuard_SameDaySupersededYoungExcluded (0.00s)
offbox_window_test.go:532: why = "" — a real snapshot inside the daily window must not be removed
--- FAIL: TestOffsiteGuard_RealPolicySkewWindowFakeRefused (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.024s
FAIL
rc=1
### RED-PROOF 5 drop the weekly cap: 'if maxRemove >= 0 && len(keep) > maxRemove {' -> 'if false && len(keep) > maxRemove {'
offbox_window_test.go:202: ids=[o5-full o4-full o3-full o2-full o1-full o0-full] why=""
--- FAIL: TestOffsiteGuard_AboveWeeklyCapRefused (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.023s
FAIL
rc=1
### restored
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.022s
documentation/audits/night-fixes-2026-10-05/partB/r863-live.txt
+37
View File
@@ -0,0 +1,37 @@
### 05:28:31 pull 0.294.0 in 9202 (outside the timing)
gitea.dooplex.hu/admin/felhom-controller:0.294.0
0
### 05:28:34 make the one-time clean-up due: remove its marker; switch the image; restart
gitea.dooplex.hu/admin/felhom-controller:0.294.0 started 2026-10-05T05:28:35.617220084Z
### 05:31:15 DEPLOY bookstack (POST /api/stacks/bookstack/deploy, the endpoint the install button calls)
HTTP 202
05:31:20
05:31:36
05:31:53 bookstack=Created bookstack-db=Up 9 seconds (health: starting)
05:32:09 bookstack=Up 15 seconds (health: starting) bookstack-db=Up 26 seconds (healthy)
### controller log since start (image retention + bookstack deploy lines)
2026/10/05 05:31:36 controller_image_retention.go:184: [INFO] [stacks] controller image retention: deleted gitea.dooplex.hu/admin/felhom-controller:0.291.0 (dde6b167c496, 410MB) — older than the previous controller and no container uses it (decision 56)
2026/10/05 05:31:36 controller_image_retention.go:115: [INFO] [stacks] controller image retention: pass over 3 controller image(s) — running 0.294.0, previous "0.293.0" (by version order (no swap record names one present)), 1 candidate(s), 1 deleted, the rest kept
2026/10/05 05:31:36 image_retention.go:211: [INFO] [stacks] image retention (one-time clean-up): skipped — an app install, update, restore or undo is pulling images now (R-863); tried again later
2026/10/05 05:31:54 deploy.go:611: [INFO] [stacks] Stack bookstack deployed successfully (took 35.5s)
2026/10/05 05:31:57 manager.go:1598: [INFO] [stacks] Stack bookstack post-start status:
### 05:33:54 after the retry
2026/10/05 05:31:36 image_retention.go:211: [INFO] [stacks] image retention (one-time clean-up): skipped — an app install, update, restore or undo is pulling images now (R-863); tried again later
2026/10/05 05:33:36 image_retention.go:389: [INFO] [stacks] image retention (one-time): deleted 0 image(s). docker disk before: Images 6.245GB 2.943GB (47%) | after: Images 6.245GB 2.943GB (47%)
deleted 0
bookstack lscr.io/linuxserver/bookstack:26.09.1 Up 2 minutes (healthy)
bookstack-db mariadb:12.3 Up 2 minutes (healthy)
### 05:34:05 teardown: stop + remove through the product
HTTP 200
HTTP 200
### 05:34:49 verify
0
ls: cannot access '/opt/docker/stacks/bookstack/app.yaml': No such file or directory
0
2026/10/05 05:34:41 delete.go:804: [INFO] Stack bookstack removed successfully (took 26.7s)
2026/10/05 05:34:43 image_retention.go:265: [INFO] [stacks] image retention (remove of bookstack): deleted [lscr.io/linuxserver/bookstack:<none>] (b274bee3f111, 416MB) — no container, installed app or undo names it (decision 53)
2026/10/05 05:34:43 image_retention.go:265: [INFO] [stacks] image retention (remove of bookstack): deleted [mariadb:<none>] (dfff46ef3f9d, 334MB) — no container, installed app or undo names it (decision 53)
2026/10/05 05:34:43 image_retention.go:238: [INFO] [stacks] image retention (remove of bookstack): pass over 15 image(s) of [lscr.io/linuxserver/bookstack mariadb] — 2 candidate(s), 2 deleted, the rest kept
documentation/audits/night-fixes-2026-10-05/partB/red-proofs.txt
+28
View File
@@ -0,0 +1,28 @@
### RED-PROOF R-863: TryImageCleanup check dropped (endCleanup, ok := func() {}, true)
image_retention_r863_test.go:70: the install failed: the clean-up deleted the image compose had just pulled (rmi [sha256:MDB])
--- FAIL: TestR863_OneTimeCleanupDuringFirstInstallKeepsThePulledImage (0.01s)
--- PASS: TestR863_OnlyPullingVerbsHoldTheLock (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.014s
FAIL
rc=1
### RED-PROOF R-863b: a busy pass reports done (the old behaviour: the one-time clean-up's marker ends it for good)
image_retention_r863_test.go:79: the clean-up reported done although it did not run — its marker would end it for good
--- FAIL: TestR863_OneTimeCleanupDuringFirstInstallKeepsThePulledImage (0.01s)
--- PASS: TestR863_OnlyPullingVerbsHoldTheLock (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.017s
FAIL
rc=1
### restored
ok gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.018s
### RED-PROOF R-864: truncateStr (head) instead of tailStr for stderr
r864_stderr_tail_test.go:45: the reason was cut from the returned error: "exit code 1\nstderr: Image lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 Pulli
--- FAIL: TestR864_FailedComposeLogsTheReasonNotThePullProgress (0.01s)
--- PASS: TestR864_TailStrKeepsTheEndOnARuneBoundary (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.012s
FAIL
rc=1
### restored
ok gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.015s
documentation/audits/night-fixes-2026-10-05/partC/print-uris-real-apt.txt
+28
View File
@@ -0,0 +1,28 @@
R-865 probe, 2026-10-05, scratch guest 9202 on demo-hp (read only). Before/after: .deb files in /var/cache/apt/archives, and the versions.
status: running
10
bash 5.2.37-2+b10
libc6:amd64 2.41-12+deb13u4
---WITH-S
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 2 reinstalled, 0 to remove and 1 not upgraded.
Inst libc6 [2.41-12+deb13u4] (2.41-12+deb13u4 Debian:13.7/stable [amd64])
Conf libc6 (2.41-12+deb13u4 Debian:13.7/stable [amd64])
Inst bash [5.2.37-2+b10] (5.2.37-2+b10 Debian:13.7/stable [amd64])
Conf bash (5.2.37-2+b10 Debian:13.7/stable [amd64])
---WITHOUT-S
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 2 reinstalled, 0 to remove and 1 not upgraded.
Need to get 4347 kB of archives.
After this operation, 0 B of additional disk space will be used.
'http://deb.debian.org/debian/pool/main/b/bash/bash_5.2.37-2%2bb10_amd64.deb' bash_5.2.37-2+b10_amd64.deb 1500792 MD5Sum:27b11721fea83d73b96e0f7023863771
'http://deb.debian.org/debian/pool/main/g/glibc/libc6_2.41-12%2bdeb13u4_amd64.deb' libc6_2.41-12+deb13u4_amd64.deb 2846580 MD5Sum:5559581916477ef1f57ea9f82cecf22e
rc=0
10
bash 5.2.37-2+b10
libc6:amd64 2.41-12+deb13u4
(rc=0 above is head's; the cache count 10 -> 10 and the versions are unchanged: --print-uris without -s downloads nothing)
documentation/audits/night-fixes-2026-10-05/partC/r8-live-installed-wrapper.txt
+6
View File
@@ -0,0 +1,6 @@
### 05:44:30 demo-hp: the INSTALLED wrapper's download_bytes (R8's measure) for the 13 pending upgrades, as root, read-only (--print-uris)
36e16f7619cc556d
download_bytes = 12802456 B -> R8 needs max(500 MB, 3 x 12802456 B) = 524288000 B free
bind9-libs:amd64 1:9.20.26-1~deb13u1
openssl 3.5.7-1~deb13u2
13
documentation/audits/night-fixes-2026-10-05/partC/red-proof.txt
+8
View File
@@ -0,0 +1,8 @@
### RED-PROOF R-865: -s put back into download_bytes
1
FAIL: test_R8_counts_the_real_download (test_felhom_os_apply.Refusals.test_R8_counts_the_real_download)
FAIL: test_download_bytes_never_simulates (test_felhom_os_apply.Refusals.test_download_bytes_never_simulates)
Ran 79 tests in 0.499s
FAILED (failures=2)
### restored
OK
documentation/audits/night-fixes-2026-10-05/partD/r866-kept-copies-sent-at-start.txt
+11
View File
@@ -0,0 +1,11 @@
Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.454+02:00 level=INFO msg="felhom-agent daemon starting" version=0.144.1 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900
Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.455+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T055135Z layer=docker vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T055135Z-docker-apply.json
Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.455+02:00 level=INFO msg="osupdate: DONE" run=20261005T055135Z layer=docker vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=0 pending=0 not_covered=0 restart_needed=7 reboot_needed=true wrapper_seconds=14.7
Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.529+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T055135Z layer=guest vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T055135Z-guest-apply.json
Oct 05 07:53:33 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:33.529+02:00 level=INFO msg="osupdate: DONE" run=20261005T055135Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=0 pending=0 not_covered=0 restart_needed=7 reboot_needed=true wrapper_seconds=13.3
Oct 05 07:53:34 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:34.479+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T055135Z layer=host vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T055135Z-host-apply.json
Oct 05 07:53:34 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:34.479+02:00 level=INFO msg="osupdate: DONE" run=20261005T055135Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=0 pending=80 not_covered=80 restart_needed=28 reboot_needed=false wrapper_seconds=12.1
Oct 05 07:53:34 demo-felhom felhom-agent[3840353]: time=2026-10-05T07:53:34.498+02:00 level=INFO msg="osupdate: sent kept report(s)" count=3
(62, 'demo-felhom-8363b5', '2026-10-05 05:53:34', 'debug', 'apply', 'nothing', 1, 'host')
(61, 'demo-felhom-8363b5', '2026-10-05 05:53:33', 'debug', 'apply', 'nothing', 1, 'guest')
(60, 'demo-felhom-8363b5', '2026-10-05 05:53:33', 'debug', 'apply', 'nothing', 1, 'docker')
documentation/audits/night-fixes-2026-10-05/partD/r866-live-hub-away.txt
+11
View File
@@ -0,0 +1,11 @@
05:51:35 agent felhom-agent 0.144.0; saved block: 2026-10-05 07:23:26
05:51:35 hub blocked (blackhole routes)
=== felhom-agent 0.144.0 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=SAVED(2026-10-05T05:23:26Z; hub unreachable: hub: transport error: Get "https://hub.felhom.eu/api/v1/hosts/demo-felhom-8363b5/desired-state": dial tcp 37.191.56.193:443: connect: invalid argument) ===
"healthy": true,
"outcome": "nothing",
"healthy": true,
"outcome": "nothing",
"healthy": true,
"outcome": "nothing",
pass took 42.8s
05:52:18 hub unblocked; kept copies now: report-20261005T055135Z-docker-apply.json report-20261005T055135Z-guest-apply.json report-20261005T055135Z-host-apply.json
documentation/audits/night-fixes-2026-10-05/partD/r866-red-proofs.txt
+19
View File
@@ -0,0 +1,19 @@
### RED-PROOF R-866a: selftestOSBlock returns at once on a fetch error (v0.143.0 behaviour)
r866_selftest_block_test.go:37: the pass did not run from the saved block: ok=false block=<nil> src="desired state: hub: transport error: connect: invalid argument"
--- FAIL: TestR866_DebugPassUsesTheSavedBlockWhenTheHubIsAway (0.00s)
r866_selftest_block_test.go:53: ok=false why="desired state: down"
--- FAIL: TestR866_NoHubNoSavedBlockDoesNotRun (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent 0.011s
FAIL
rc=1
### RED-PROOF R-866b: the daemon no longer saves the block
r866_selftest_block_test.go:37: the pass did not run from the saved block: ok=false block=<nil> src="desired state: hub: transport error: connect: invalid argument — and no saved block (the daemon saves one when the hub sends it)"
--- FAIL: TestR866_DebugPassUsesTheSavedBlockWhenTheHubIsAway (0.00s)
--- PASS: TestR866_NoHubNoSavedBlockDoesNotRun (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent 0.009s
FAIL
rc=1
### restored
ok gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent 0.009s
documentation/audits/night-fixes-2026-10-05/partD/r868-brokenpipe-red-proof.txt
+14
View File
@@ -0,0 +1,14 @@
### RED-PROOF R-868 (live finding, v0.144.1): log() writes to stderr unguarded (v0.144.0)
raise BrokenPipeError(32, "Broken pipe")
BrokenPipeError: [Errno 32] Broken pipe
FAILED (errors=1)
### restored
OK
### RED-PROOF R-868 loop (v0.144.1): SendUnsentLoop returns after the first look
unsent_test.go:138: the late copy was never sent
--- FAIL: TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop (3.05s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 3.063s
FAIL
### restored
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 1.754s
documentation/audits/night-fixes-2026-10-05/partD/r868-live-kill.txt
+10
View File
@@ -0,0 +1,10 @@
05:44:46 daemon pid before 2471595; kept copies before: 0
05:44:46 debug pass started
KILLED at 05:45:04.880: the pass + the daemon (apt-get was running)
bash: line 11: 2525079 Killed nohup sudo -u felhom-agent /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=os-update -vmid 9201 > /root/r868-pass.log 2>&1
05:45:16 wrapper finished; kept copy:
daemon now pid 2526755 active restarts 1
audit rc=0
bind9-libs:amd64 1:9.20.29-1~deb13u1
openssl 3.5.7-1~deb13u3
python3.13 3.13.5-2+deb13u5
documentation/audits/night-fixes-2026-10-05/partD/r868-live-retest.txt
+24
View File
@@ -0,0 +1,24 @@
### 06:03:27 rollback 2 (the same 13, simulated clean before)
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
audit rc=0
06:03:42 agent felhom-agent 0.144.1; daemon pid 2571767; kept copies before: 0
KILLED at 06:04:00.258: the pass + the daemon (apt-get was running)
bash: line 10: 2582951 Killed nohup sudo -u felhom-agent /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=os-update -vmid 9201 > /root/r868-pass2.log 2>&1
06:04:20 wrapper finished; kept copy: report-20261005T060342Z-guest-apply.json
daemon now active restarts 1
audit rc=0
bind9-libs:amd64 1:9.20.29-1~deb13u1
openssl 3.5.7-1~deb13u3
python3.13 3.13.5-2+deb13u5
Oct 05 08:02:59 demo-hp felhom-os-apply[2580370]: os-apply: BUNDLE DONE agent=0.144.1 written=1 same=20 self-check=ok signers-created=False
Oct 05 08:03:54 demo-hp felhom-os-apply[2583726]: os-apply: PLAN upgrade=13 already=0 not-installed=0 from-snapshot=0
Oct 05 08:04:12 demo-hp felhom-os-apply[2585855]: os-apply: DONE rc=0 seconds=9.4 upgraded=13 restart-needed=containerd-shim,cron,dbus-daemon,sshd,systemd,systemd-journal,systemd-logind,systemd-network reboot-needed=yes
### 06:09:09 the copy is gone
Oct 05 08:04:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:04:05.487+02:00 level=INFO msg="felhom-agent daemon starting" version=0.144.1 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
Oct 05 08:09:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:09:05.492+02:00 level=INFO msg="osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)" run=20261005T060342Z layer=guest vmid=9201 ring=0 trigger=debug path=/var/lib/felhom-agent/os/report-20261005T060342Z
Oct 05 08:09:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:09:05.492+02:00 level=INFO msg="osupdate: DONE" run=20261005T060342Z layer=guest vmid=9201 ring=0 trigger=debug outcome=applied healthy=true reason="sent after the agent stopped mid-pass (R-868)" upgraded=13 pending=0 not_covered=0 re
Oct 05 08:09:05 demo-hp felhom-agent[2584482]: time=2026-10-05T08:09:05.558+02:00 level=INFO msg="osupdate: sent kept report(s)" count=1
--- hub os_reports (demo-hp, newest 2)
(63, '2026-10-05 06:09:05', 'debug', 'apply', 'applied', 1, 'guest', '20261005T060342Z', 13)
(59, '2026-10-05 02:59:56', 'debug', 'apply', 'nothing', 1, 'docker', '20261005T025842Z', None)
documentation/audits/night-fixes-2026-10-05/partD/r868-red-proofs.txt
+33
View File
@@ -0,0 +1,33 @@
### RED-PROOF R-868a: SendUnsent sends nothing (sendUnsentLocked returns 0 at once)
unsent_test.go:31: sent 0, want 1
--- FAIL: TestR868_KilledPassIsReportedAtStart (0.00s)
--- PASS: TestR868_NormalPassLeavesNoCopyAndNoDuplicate (0.00s)
unsent_test.go:91: sent 0: []
--- FAIL: TestR868_HubAwayKeepsTheCopy (0.00s)
unsent_test.go:111: not sent after the pass: 0
--- FAIL: TestR868_RunningPassKeepsTheSenderOff (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.014s
FAIL
rc=1
### RED-PROOF R-868b: finish no longer deletes the sent copy
unsent_test.go:31: sent 0, want 1
--- FAIL: TestR868_KilledPassIsReportedAtStart (0.00s)
unsent_test.go:59: kept copies left after the hub got them: [/tmp/TestR868_NormalPassLeavesNoCopyAndNoDuplicate3889087443/001/report-20261004T040000Z-docker-apply.json /tmp/TestR868_NormalPassLeavesNoCopyAndNoDuplicate3889087443/001/report-20261004T040000Z-guest-apply.json /tmp/TestR868_NormalPassLeavesNoCopyAndNoDuplicate3889087443/001/report-20261004T040000Z-host-apply.json]
--- FAIL: TestR868_NormalPassLeavesNoCopyAndNoDuplicate (0.00s)
unsent_test.go:91: sent 0: [{RunID:20261004T040000Z Layer:docker Trigger:night Mode:apply Ring:0 ReleaseID: Outcome:nothing Healthy:true HealthReason:sent after the agent stopped mid-pass (R-868) VMID:0 Upgraded:[] Installed:[] Pending:[] NotCovered:[] RestartNeeded:[] DockerRestartNeeded:false RebootNeeded:false RebootScanned:false Refused:[] PassSeconds:0 DockerEngine: Authority: Undo:false unsent:} {RunID:20261004T040000Z Layer:guest Trigger:night Mode:apply Ring:0 ReleaseID: Outcome:applied Healthy:true HealthReason:sent after the agent stopped mid-pass (R-868) VMID:0 Upgraded:[{Name:libc6 Version:u4 Origin:}] Installed:[] Pending:[] NotCovered:[] RestartNeeded:[] DockerRestartNeeded:false RebootNeeded:false RebootScanned:false Refused:[] PassSeconds:0 DockerEngine: Authority: Undo:false unsent:}]
--- FAIL: TestR868_HubAwayKeepsTheCopy (0.00s)
unsent_test.go:111: not sent after the pass: 0
--- FAIL: TestR868_RunningPassKeepsTheSenderOff (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.012s
FAIL
rc=1
### RED-PROOF R-868c (wrapper): main() no longer saves the copy
ERROR: test_a_refusal_is_kept_too (test_felhom_os_apply.UnsentReport.test_a_refusal_is_kept_too)
FAIL: test_apply_keeps_a_copy_with_the_agents_ids (test_felhom_os_apply.UnsentReport.test_apply_keeps_a_copy_with_the_agents_ids)
Ran 86 tests in 0.520s
FAILED (failures=1, errors=1)
### restored
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 2.328s
OK
documentation/audits/night-fixes-2026-10-05/partD/r869-red-proof.txt
+9
View File
@@ -0,0 +1,9 @@
### RED-PROOF R-869: newPath assigned after the log line (v0.293.0 order)
offbox_window_test.go:271: the line names no destination:
--- FAIL: TestR869_MoveAsideLineNamesTheDestination (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.009s
FAIL
rc=1
### restored
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.008s
documentation/audits/night-fixes-2026-10-05/partD/raw/r868-pass.log
+2
View File
@@ -0,0 +1,2 @@
=== felhom-agent 0.144.0 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub ===
time=2026-10-05T07:44:46.566+02:00 level=INFO msg="osupdate: START" run=20261005T054446Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261005T054446Z
documentation/audits/night-fixes-2026-10-05/partD/raw/r868-pass2.log
+2
View File
@@ -0,0 +1,2 @@
=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub ===
time=2026-10-05T08:03:42.262+02:00 level=INFO msg="osupdate: START" run=20261005T060342Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261005T060342Z
documentation/audits/night-fixes-2026-10-05/partE/e0-before.txt
+39
View File
@@ -0,0 +1,39 @@
### 06:09:35 demo-hp before the crash
{
"armed": true,
"boot_id": "182af911-22a9-41ec-97ac-7ee247a4d470",
"config": {
"LIMIT": 3,
"PANIC_SECONDS": 10,
"REARM_HOURS": 24,
"WINDOW_MINUTES": 60
},
"kernel_panic": 10,
"last_boot_at": "2026-10-04T15:17:26Z",
"last_boot_unclean": true,
"last_trip": {
"at": "2026-10-04T14:54:43Z",
"reason": "2 unclean boots within 60 minutes \u2014 the next crash leaves the box off (limit 3)"
},
"rearmed_at": "2026-10-04T15:33:25Z",
"rearmed_by": "operator",
"tripped": false,
"unclean_boots": [
"2026-10-04T14:52:08Z",
"2026-10-04T14:54:33Z",
"2026-10-04T15:17:26Z"
],
"unclean_boots_24h": 3,
"unclean_boots_in_window": 0,
"updated_at": "2026-10-05T05:34:05Z",
"version": 1
}
kernel.panic = 10
kernel.sysrq = 438
08:09:36 up 14:52, 1 user, load average: 0.91, 1.14, 1.10
-2 cc0aa40bac534524a46f6c5983ce5ed9 Sun 2026-10-04 16:52:13 CEST Sun 2026-10-04 16:53:36 CEST
-1 53f69dfb93ef44c3a9ad803d86dfb0b3 Sun 2026-10-04 16:54:38 CEST Sun 2026-10-04 16:55:14 CEST
0 182af91122a941ec97ac7ee247a4d470 Sun 2026-10-04 17:17:31 CEST Mon 2026-10-05 08:09:37 CEST
20
21
felhom-agent 0.144.1
documentation/audits/night-fixes-2026-10-05/partE/e1-rollback.txt
+22
View File
@@ -0,0 +1,22 @@
### 06:09:53 rollback 3 (13 packages; simulated first)
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
audit rc=0
bind9-dnsutils 1:9.20.26-1~deb13u1
bind9-host 1:9.20.26-1~deb13u1
bind9-libs:amd64 1:9.20.26-1~deb13u1
libpcre2-8-0:amd64 10.46-1~deb13u2
libpython3.13-minimal:amd64 3.13.5-2+deb13u4
libpython3.13-stdlib:amd64 3.13.5-2+deb13u4
libssh2-1t64:amd64 1.11.1-1+deb13u1
libssl3t64:amd64 3.5.7-1~deb13u2
libxml2:amd64 2.12.7+dfsg+really2.9.14-2.1+deb13u1
openssl 3.5.7-1~deb13u2
openssl-provider-legacy 3.5.7-1~deb13u2
python3.13 3.13.5-2+deb13u4
python3.13-minimal 3.13.5-2+deb13u4
### 06:12:44 attempt 1 made no crash: the watcher pattern 'dpkg --status-fd' never matched (apt puts options before --status-fd); the pass installed the 13 normally. Pattern now 'bin/dpkg .*--status-fd'.
### 06:12:44 rollback 4
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
audit rc=0
documentation/audits/night-fixes-2026-10-05/partE/e2-crash.txt
+9
View File
@@ -0,0 +1,9 @@
06:10:25.835 debug pass started
no dpkg seen
ssh ended 06:12:25
### attempt 2
06:13:33.983 debug pass started
dpkg running at 06:13:55.070:
CRASH at 06:13:55.904
Timeout, server 192.168.0.104 not responding.
ssh ended 06:14:03
documentation/audits/night-fixes-2026-10-05/partE/e3-back.txt
+10
View File
@@ -0,0 +1,10 @@
ssh back 06:14:48
08:14:47 up 0 min, 1 user, load average: 1.33, 0.29, 0.10
"armed": true,
"last_boot_at": "2026-10-05T06:14:33Z",
"last_boot_unclean": true,
"rearmed_at": "2026-10-04T15:33:25Z",
"rearmed_by": "operator",
"tripped": false,
"unclean_boots_in_window": 1,
kernel.panic = 10
documentation/audits/night-fixes-2026-10-05/partE/e4-dpkg-at-boot.txt
+20
View File
@@ -0,0 +1,20 @@
### 06:14:56 guest state after the crash
status: running
audit rc=0
bind9-dnsutils 1:9.20.26-1~deb13u1 ii
bind9-host 1:9.20.26-1~deb13u1 ii
bind9-libs 1:9.20.26-1~deb13u1 ii
libpcre2-8-0 10.46-1~deb13u2 ii
libpython3.13-minimal 3.13.5-2+deb13u4 ii
libpython3.13-stdlib 3.13.5-2+deb13u4 ii
libssh2-1t64 1.11.1-1+deb13u1 ii
libssl3t64 3.5.7-1~deb13u2 ii
libxml2 2.12.7+dfsg+really2.9.14-2.1+deb13u1 ii
openssl 3.5.7-1~deb13u2 ii
openssl-provider-legacy 3.5.7-1~deb13u3 ii
python3.13 3.13.5-2+deb13u4 ii
python3.13-minimal 3.13.5-2+deb13u4 ii
plan-20261005T025659Z-guest-apply.json
plan-20261005T054446Z-guest-apply.json
plan-20261005T061334Z-guest-apply.json
06:17:41 apps: 20 healthy of 21
documentation/audits/night-fixes-2026-10-05/partE/e5-next-pass.txt
+14
View File
@@ -0,0 +1,14 @@
### 06:17:43 the next pass (debug, by hand — the night leg's 20 h gap would hold the real one)
=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub ===
"healthy": false,
"outcome": "failed",
"reboot_needed": false,
pass took 20.2s
Oct 05 08:13:35 demo-hp felhom-os-apply[2618444]: os-apply: START release=ring0-20261005T061334Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0
Oct 05 08:13:40 demo-hp felhom-os-apply[2619128]: os-apply: REPAIR configured=0 fixed=0
Oct 05 08:13:46 demo-hp felhom-os-apply[2619504]: os-apply: PLAN upgrade=13 already=0 not-installed=0 from-snapshot=0
Oct 05 08:17:44 demo-hp felhom-os-apply[16225]: os-apply: START release=ring0-20261005T061743Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0
Oct 05 08:17:49 demo-hp felhom-os-apply[16446]: os-apply: REPAIR configured=0 fixed=0
Oct 05 08:17:55 demo-hp felhom-os-apply[16789]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0
Oct 05 08:18:03 demo-hp felhom-os-apply[17194]: os-apply: FAILED rc=100 step=install — dpkg state: clean
audit rc=0
documentation/audits/night-fixes-2026-10-05/partE/e6-why-repair-missed.txt
+13
View File
@@ -0,0 +1,13 @@
### 06:18:33 why: dpkg's update journal (apt's 'interrupted' check) vs dpkg --audit
total 20
drwxr-xr-x 2 root root 4096 Oct 5 06:13 .
drwxr-xr-x 7 root root 4096 Oct 5 06:13 ..
-rw-r--r-- 1 root root 1048 Oct 5 06:13 0000
-rw-r--r-- 1 root root 1055 Oct 5 06:13 0001
-rw-r--r-- 1 root root 1017 Oct 5 06:13 0002
audit rc=0
Inst openssl [3.5.7-1~deb13u2] (3.5.7-1~deb13u3 Debian-Security:13/stable-security [amd64])
Conf openssl (3.5.7-1~deb13u3 Debian-Security:13/stable-security [amd64])
905: def repair(self):
911: self.x(APT_ENV + ["dpkg", "--configure", "-a", "--force-confold"])
916: self.r.log(f"os-apply: REPAIR configured={configured} fixed={fixed}")
documentation/audits/night-fixes-2026-10-05/partE/e7-repair-and-current.txt
+27
View File
@@ -0,0 +1,27 @@
### 06:18:55 by-hand repair: dpkg --configure -a in the guest (what the wrapper's REPAIR would do)
rc=0
0
### 06:18:58 the next pass after the repair
=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub ===
"healthy": true,
"outcome": "applied",
"healthy": true,
"outcome": "nothing",
"healthy": true,
"outcome": "nothing",
pass took 1m16.9s
Oct 05 08:17:49 demo-hp felhom-os-apply[16446]: os-apply: REPAIR configured=0 fixed=0
Oct 05 08:17:55 demo-hp felhom-os-apply[16789]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0
Oct 05 08:18:03 demo-hp felhom-os-apply[17194]: os-apply: FAILED rc=100 step=install — dpkg state: clean
Oct 05 08:19:04 demo-hp felhom-os-apply[20081]: os-apply: REPAIR configured=0 fixed=0
Oct 05 08:19:10 demo-hp felhom-os-apply[20406]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0
Oct 05 08:19:28 demo-hp felhom-os-apply[21616]: os-apply: DONE rc=0 seconds=10.7 upgraded=12 restart-needed=cron,dbus-daemon,sshd,systemd,systemd-journal,systemd-logind,systemd-network reboot-needed=y
Oct 05 08:19:40 demo-hp felhom-os-apply[22557]: os-apply: REPAIR configured=0 fixed=0
Oct 05 08:19:44 demo-hp felhom-os-apply[22730]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
Oct 05 08:19:44 demo-hp felhom-os-apply[22731]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
Oct 05 08:20:02 demo-hp felhom-os-apply[24349]: os-apply: REPAIR configured=0 fixed=0
Oct 05 08:20:07 demo-hp felhom-os-apply[24637]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
Oct 05 08:20:07 demo-hp felhom-os-apply[24638]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
audit rc=0
0
package list == before rollback 1 (279 lines)
documentation/audits/night-fixes-2026-10-05/partE/e8-household-mails-system.txt
+9
View File
@@ -0,0 +1,9 @@
--- hub events demo-hp since 06:13 UTC
('controller_started', 'info', 'Controller elindult (0.294.0)', 'controller', '2026-10-05 06:16:39')
('os_update_failed', 'error', 'OS update (guest) on demo-hp-bb76ea failed: {"dpkg_audit":"clean","rc":100,"tail":["E: dpkg was interrupted, you must manually run \'sudo dpkg --config', 'hub', '2026-10-05 06:18:03')
('os_update_applied', 'info', 'System security fixes installed on the box (12 package(s)).', 'hub', '2026-10-05 06:19:35')
--- notification_log since 06:13 UTC (all customers)
('demo-hp', 'os_update_failed', 'sent', 'operator', '2026-10-05 06:18:03')
('demo-hp', 'os_update_failed', 'skipped', 'customer', '2026-10-05 06:18:03')
--- System page line (demo-hp)
go · nothing · 12 s demo-hp-bb76ea Demo HP partial: no running customer guest ring 0 → normal updates ON switch off unknown 9.2.2 unknown unknown unknown ring0-20261005T061858Z 78 0 unknown no unknown no unknown not installed 0.144.1 unknown ring0-20261005T061858Z 0 7 unknown unknown unknown ring0-20261005T061858Z 0 min ago · nothing · 19 s tester-1-d70be4 Tester 1 ring 1 → demo updates ON switch off running 9.2.2 7.0.2-6-pve 7.0.2-6-pve 13.5 — 185 106 none no 10 s no 0 armed 0.144.1 13.6 — 49 0 29.8.2 2.3.6-1~debian.13~trixie on — 10 h ago · nothing · 16 s Amber: worth a look. Red: an operator alarm fires (`08` §6.3). "unknown": the box could not read the value — never a guess. Felhom Hub 0.133.0
documentation/audits/night-fixes-2026-10-05/partE/raw/9201-pkgs-after-parte.txt
+279
View File
@@ -0,0 +1,279 @@
adduser 3.152
apparmor 4.1.0-1
apt 3.0.3
apt-listchanges 4.8
apt-utils 3.0.3
base-files 13.8+deb13u7
base-passwd 3.6.7
bash 5.2.37-2+b10
bash-completion 1:2.16.0-7
bind9-dnsutils 1:9.20.29-1~deb13u1
bind9-host 1:9.20.29-1~deb13u1
bind9-libs 1:9.20.29-1~deb13u1
bsdextrautils 2.41.5-0+deb13u1
bsdutils 1:2.41.5-0+deb13u1
bzip2 1.0.8-6
ca-certificates 20250419
containerd.io 2.3.6-1~debian.13~trixie
coreutils 9.7-3
cpio 2.15+dfsg-2
cron 3.0pl1-197
cron-daemon-common 3.0pl1-197
curl 8.14.1-2+deb13u5
dash 0.5.12-12
dbus 1.16.2-2
dbus-bin 1.16.2-2
dbus-daemon 1.16.2-2
dbus-session-bus-common 1.16.2-2
dbus-system-bus-common 1.16.2-2
dbus-user-session 1.16.2-2
debconf 1.5.91
debconf-i18n 1.5.91
debian-archive-keyring 2025.1
debian-faq 12.2
debianutils 5.23.2
dhcpcd-base 1:10.1.0-11+deb13u4
diffutils 1:3.10-4
distro-info-data 0.66+deb13u2
dmidecode 3.6-2
doc-debian 11.3+nmu1
docker-buildx-plugin 0.37.1-1~debian.13~trixie
docker-ce 5:29.8.2-1~debian.13~trixie
docker-ce-cli 5:29.8.2-1~debian.13~trixie
docker-ce-rootless-extras 5:29.8.2-1~debian.13~trixie
docker-compose-plugin 5.6.0-1~debian.13~trixie
dpkg 1.22.22
e2fsprogs 1.47.2-3+b12
fdisk 2.41.5-0+deb13u1
file 1:5.46-5
findutils 4.10.0-3
gcc-14-base 14.2.0-19
gettext-base 0.23.1-2
git 1:2.47.3-0+deb13u1
git-man 1:2.47.3-0+deb13u1
grep 3.11-4
groff-base 1.23.0-9
gzip 1.13-1+deb13u1
hostname 3.25
ifupdown 0.8.44+deb13u1
ifupdown2 3.0.0-1.3
inetutils-telnet 2:2.6-3+deb13u3
init 1.69~deb13u1
init-system-helpers 1.69~deb13u1
iproute2 6.15.0-1
iptables 1.8.11-2
iputils-ping 3:20240905-3
isc-dhcp-client 4.4.3-P1-8
kmod 34.2-2
krb5-locales 1.21.3-5+deb13u1
less 668-1
libacl1 2.3.2-2+b1
libapparmor1 4.1.0-1
libapt-pkg7.0 3.0.3
libattr1 1:2.5.2-3
libaudit-common 1:4.0.2-2+deb13u1
libaudit1 1:4.0.2-2+deb13u1
libblkid1 2.41.5-0+deb13u1
libbpf1 1:1.5.0-3
libbrotli1 1.1.0-2+b7
libbsd0 0.12.2-2
libbz2-1.0 1.0.8-6
libc-bin 2.41-12+deb13u4
libc-l10n 2.41-12+deb13u4
libc6 2.41-12+deb13u4
libcap-ng0 0.8.5-4+b1
libcap2 1:2.75-10+deb13u1+b3
libcap2-bin 1:2.75-10+deb13u1+b3
libcbor0.10 0.10.2-2
libcom-err2 1.47.2-3+b12
libcrypt1 1:4.4.38-1
libcurl3t64-gnutls 8.14.1-2+deb13u5
libcurl4t64 8.14.1-2+deb13u5
libdb5.3t64 5.3.28+dfsg2-9
libdbus-1-3 1.16.2-2
libdebconfclient0 0.280
libedit2 3.1-20250104-1
libelf1t64 0.192-4
liberror-perl 0.17030-1
libexpat1 2.8.3-1~deb13u1
libext2fs2t64 1.47.2-3+b12
libfdisk1 2.41.5-0+deb13u1
libffi8 3.4.8-2
libfido2-1 1.15.0-1+b1
libfstrm0 0.6.1-1+b3
libgcc-s1 14.2.0-19
libgdbm-compat4t64 1.24-2
libgdbm6t64 1.24-2
libgmp10 2:6.3.0+dfsg-3
libgnutls30t64 3.8.9-3+deb13u4
libgssapi-krb5-2 1.21.3-5+deb13u1
libhogweed6t64 3.10.1-1
libicu76 76.1-4
libidn2-0 2.3.8-2
libip4tc2 1.8.11-2
libip6tc2 1.8.11-2
libjansson4 2.14-2+b3
libjemalloc2 5.3.0-3
libjson-c5 0.18+ds-1
libk5crypto3 1.21.3-5+deb13u1
libkeyutils1 1.6.3-6
libkmod2 34.2-2
libkrb5-3 1.21.3-5+deb13u1
libkrb5support0 1.21.3-5+deb13u1
liblastlog2-2 2.41.5-0+deb13u1
libldap-common 2.6.10+dfsg-1
libldap2 2.6.10+dfsg-1
liblmdb0 0.9.31-1+b2
liblocale-gettext-perl 1.07-7+b1
liblockfile-bin 1.17-2
liblsof0 4.99.4+dfsg-2
liblz4-1 1.10.0-4
liblzma5 5.8.1-1+deb13u1
libmagic-mgc 1:5.46-5
libmagic1t64 1:5.46-5
libmaxminddb0 1.12.2-1
libmd0 1.1.0-2+b1
libmnl0 1.0.5-3
libmount1 2.41.5-0+deb13u1
libncursesw6 6.5+20250216-2
libnetfilter-conntrack3 1.1.0-1
libnettle8t64 3.10.1-1
libnewt0.52 0.52.25-1
libnfnetlink0 1.0.2-3
libnftables1 1.1.3-1
libnftnl11 1.2.9-1
libnghttp2-14 1.64.0-1.1+deb13u1
libnghttp3-9 1.8.0-1
libngtcp2-16 1.11.0-1+deb13u1
libngtcp2-crypto-gnutls8 1.11.0-1+deb13u1
libnsl2 1.3.0-3+b3
libnss-systemd 257.13-1~deb13u1
libp11-kit0 0.25.5-3
libpam-modules 1.7.0-5
libpam-modules-bin 1.7.0-5
libpam-runtime 1.7.0-5
libpam-systemd 257.13-1~deb13u1
libpam-wtmpdb 0.73.0-3+deb13u1
libpam0g 1.7.0-5
libpcre2-8-0 10.46-1~deb13u3
libperl5.40 5.40.1-6+deb13u1
libpipeline1 1.5.8-1
libpopt0 1.19+dfsg-2
libproc2-0 2:4.0.4-9
libprotobuf-c1 1.5.1-1
libpsl5t64 0.21.2-1.1+b1
libpython3-stdlib 3.13.5-1
libpython3.13-minimal 3.13.5-2+deb13u5
libpython3.13-stdlib 3.13.5-2+deb13u5
libreadline8t64 8.2-6
librtmp1 2.4+20151223.gitfa8646d.1-2+b5
libsasl2-2 2.1.28+dfsg1-9
libsasl2-modules-db 2.1.28+dfsg1-9
libseccomp2 2.6.0-2
libselinux1 3.8.1-1
libsemanage-common 3.8.1-1
libsemanage2 3.8.1-1
libsepol2 3.8.1-1
libslang2 2.3.3-5+b2
libsmartcols1 2.41.5-0+deb13u1
libsqlite3-0 3.46.1-7+deb13u2
libss2 1.47.2-3+b12
libssh2-1t64 1.11.1-1+deb13u2
libssl3t64 3.5.7-1~deb13u3
libstdc++6 14.2.0-19
libsystemd-shared 257.13-1~deb13u1
libsystemd0 257.13-1~deb13u1
libtasn1-6 4.20.0-2+deb13u1
libtext-charwidth-perl 0.04-11+b4
libtext-iconv-perl 1.7-8+b4
libtext-wrapi18n-perl 0.06-10
libtinfo6 6.5+20250216-2
libtirpc-common 1.3.6+ds-1
libtirpc3t64 1.3.6+ds-1
libtlsrpt0 0.5.0rc1-2
libuchardet0 0.0.8-1+b2
libudev1 257.13-1~deb13u1
libunistring5 1.3-2
liburcu8t64 0.15.2-2
libuuid1 2.41.5-0+deb13u1
libuv1t64 1.50.0-2
libwrap0 7.6.q-36
libwtmpdb0 0.73.0-3+deb13u1
libxml2 2.12.7+dfsg+really2.9.14-2.1+deb13u3
libxtables12 1.8.11-2
libxxhash0 0.8.3-2
libzstd1 1.5.7+dfsg-1
linux-sysctl-defaults 4.12.1
locales 2.41-12+deb13u4
login 1:4.16.0-2+really2.41.5-0+deb13u1
login.defs 1:4.17.4-2
logrotate 3.22.0-1
logsave 1.47.2-3+b12
lsof 4.99.4+dfsg-2
man-db 2.13.1-1
manpages 6.9.1-1
mawk 1.3.4.20250131-1
media-types 13.0.0
mount 2.41.5-0+deb13u1
nano 8.4-1+deb13u1
ncurses-base 6.5+20250216-2
ncurses-bin 6.5+20250216-2
ncurses-term 6.5+20250216-2
netbase 6.5
netcat-traditional 1.10-50
nftables 1.1.3-1
openssh-client 1:10.0p1-7+deb13u4
openssh-server 1:10.0p1-7+deb13u4
openssh-sftp-server 1:10.0p1-7+deb13u4
openssl 3.5.7-1~deb13u3
openssl-provider-legacy 3.5.7-1~deb13u3
passwd 1:4.17.4-2
patch 2.8-2
perl 5.40.1-6+deb13u1
perl-base 5.40.1-6+deb13u1
perl-modules-5.40 5.40.1-6+deb13u1
pigz 2.8-1
postfix 3.10.13-0+deb13u1
procps 2:4.0.4-9
python-apt-common 3.0.0
python3 3.13.5-1
python3-apt 3.0.0
python3-certifi 2025.1.31+ds-1
python3-chardet 5.2.0+dfsg-2
python3-charset-normalizer 3.4.2-1
python3-debconf 1.5.91
python3-debian 1.0.1
python3-debianbts 4.1.1
python3-idna 3.10-1+deb13u1
python3-minimal 3.13.5-1
python3-reportbug 13.2.0
python3-requests 2.32.3+dfsg-5+deb13u1
python3-urllib3 2.3.0-3+deb13u2
python3.13 3.13.5-2+deb13u5
python3.13-minimal 3.13.5-2+deb13u5
readline-common 8.2-6
reportbug 13.2.0
runit-helper 2.16.4
sed 4.9-2+deb13u1
sensible-utils 0.0.25
sqv 1.3.0-3+b2
ssh 1:10.0p1-7+deb13u4
systemd 257.13-1~deb13u1
systemd-sysv 257.13-1~deb13u1
systemd-timesyncd 257.13-1~deb13u1
sysvinit-utils 3.14-4
tar 1.35+dfsg-3.1
traceroute 1:2.1.6-1
tzdata 2026c-0+deb13u1
ucf 3.0052
udev 257.13-1~deb13u1
util-linux 2.41.5-0+deb13u1
util-linux-extra 2.41.5-0+deb13u1
vim-common 2:9.1.1230-2
vim-tiny 2:9.1.1230-2
wamerican 2020.12.07-4
wget 1.25.0-2
whiptail 0.52.25-1
wtmpdb 0.73.0-3+deb13u1
xz-utils 5.8.1-1+deb13u1
zlib1g 1:1.3.dfsg+really1.3.1-1+b1
documentation/audits/night-fixes-2026-10-05/partE/raw/9201-pkgs-before-rollback1.txt
+279
View File
@@ -0,0 +1,279 @@
adduser 3.152
apparmor 4.1.0-1
apt 3.0.3
apt-listchanges 4.8
apt-utils 3.0.3
base-files 13.8+deb13u7
base-passwd 3.6.7
bash 5.2.37-2+b10
bash-completion 1:2.16.0-7
bind9-dnsutils 1:9.20.29-1~deb13u1
bind9-host 1:9.20.29-1~deb13u1
bind9-libs 1:9.20.29-1~deb13u1
bsdextrautils 2.41.5-0+deb13u1
bsdutils 1:2.41.5-0+deb13u1
bzip2 1.0.8-6
ca-certificates 20250419
containerd.io 2.3.6-1~debian.13~trixie
coreutils 9.7-3
cpio 2.15+dfsg-2
cron 3.0pl1-197
cron-daemon-common 3.0pl1-197
curl 8.14.1-2+deb13u5
dash 0.5.12-12
dbus 1.16.2-2
dbus-bin 1.16.2-2
dbus-daemon 1.16.2-2
dbus-session-bus-common 1.16.2-2
dbus-system-bus-common 1.16.2-2
dbus-user-session 1.16.2-2
debconf 1.5.91
debconf-i18n 1.5.91
debian-archive-keyring 2025.1
debian-faq 12.2
debianutils 5.23.2
dhcpcd-base 1:10.1.0-11+deb13u4
diffutils 1:3.10-4
distro-info-data 0.66+deb13u2
dmidecode 3.6-2
doc-debian 11.3+nmu1
docker-buildx-plugin 0.37.1-1~debian.13~trixie
docker-ce 5:29.8.2-1~debian.13~trixie
docker-ce-cli 5:29.8.2-1~debian.13~trixie
docker-ce-rootless-extras 5:29.8.2-1~debian.13~trixie
docker-compose-plugin 5.6.0-1~debian.13~trixie
dpkg 1.22.22
e2fsprogs 1.47.2-3+b12
fdisk 2.41.5-0+deb13u1
file 1:5.46-5
findutils 4.10.0-3
gcc-14-base 14.2.0-19
gettext-base 0.23.1-2
git 1:2.47.3-0+deb13u1
git-man 1:2.47.3-0+deb13u1
grep 3.11-4
groff-base 1.23.0-9
gzip 1.13-1+deb13u1
hostname 3.25
ifupdown 0.8.44+deb13u1
ifupdown2 3.0.0-1.3
inetutils-telnet 2:2.6-3+deb13u3
init 1.69~deb13u1
init-system-helpers 1.69~deb13u1
iproute2 6.15.0-1
iptables 1.8.11-2
iputils-ping 3:20240905-3
isc-dhcp-client 4.4.3-P1-8
kmod 34.2-2
krb5-locales 1.21.3-5+deb13u1
less 668-1
libacl1 2.3.2-2+b1
libapparmor1 4.1.0-1
libapt-pkg7.0 3.0.3
libattr1 1:2.5.2-3
libaudit-common 1:4.0.2-2+deb13u1
libaudit1 1:4.0.2-2+deb13u1
libblkid1 2.41.5-0+deb13u1
libbpf1 1:1.5.0-3
libbrotli1 1.1.0-2+b7
libbsd0 0.12.2-2
libbz2-1.0 1.0.8-6
libc-bin 2.41-12+deb13u4
libc-l10n 2.41-12+deb13u4
libc6 2.41-12+deb13u4
libcap-ng0 0.8.5-4+b1
libcap2 1:2.75-10+deb13u1+b3
libcap2-bin 1:2.75-10+deb13u1+b3
libcbor0.10 0.10.2-2
libcom-err2 1.47.2-3+b12
libcrypt1 1:4.4.38-1
libcurl3t64-gnutls 8.14.1-2+deb13u5
libcurl4t64 8.14.1-2+deb13u5
libdb5.3t64 5.3.28+dfsg2-9
libdbus-1-3 1.16.2-2
libdebconfclient0 0.280
libedit2 3.1-20250104-1
libelf1t64 0.192-4
liberror-perl 0.17030-1
libexpat1 2.8.3-1~deb13u1
libext2fs2t64 1.47.2-3+b12
libfdisk1 2.41.5-0+deb13u1
libffi8 3.4.8-2
libfido2-1 1.15.0-1+b1
libfstrm0 0.6.1-1+b3
libgcc-s1 14.2.0-19
libgdbm-compat4t64 1.24-2
libgdbm6t64 1.24-2
libgmp10 2:6.3.0+dfsg-3
libgnutls30t64 3.8.9-3+deb13u4
libgssapi-krb5-2 1.21.3-5+deb13u1
libhogweed6t64 3.10.1-1
libicu76 76.1-4
libidn2-0 2.3.8-2
libip4tc2 1.8.11-2
libip6tc2 1.8.11-2
libjansson4 2.14-2+b3
libjemalloc2 5.3.0-3
libjson-c5 0.18+ds-1
libk5crypto3 1.21.3-5+deb13u1
libkeyutils1 1.6.3-6
libkmod2 34.2-2
libkrb5-3 1.21.3-5+deb13u1
libkrb5support0 1.21.3-5+deb13u1
liblastlog2-2 2.41.5-0+deb13u1
libldap-common 2.6.10+dfsg-1
libldap2 2.6.10+dfsg-1
liblmdb0 0.9.31-1+b2
liblocale-gettext-perl 1.07-7+b1
liblockfile-bin 1.17-2
liblsof0 4.99.4+dfsg-2
liblz4-1 1.10.0-4
liblzma5 5.8.1-1+deb13u1
libmagic-mgc 1:5.46-5
libmagic1t64 1:5.46-5
libmaxminddb0 1.12.2-1
libmd0 1.1.0-2+b1
libmnl0 1.0.5-3
libmount1 2.41.5-0+deb13u1
libncursesw6 6.5+20250216-2
libnetfilter-conntrack3 1.1.0-1
libnettle8t64 3.10.1-1
libnewt0.52 0.52.25-1
libnfnetlink0 1.0.2-3
libnftables1 1.1.3-1
libnftnl11 1.2.9-1
libnghttp2-14 1.64.0-1.1+deb13u1
libnghttp3-9 1.8.0-1
libngtcp2-16 1.11.0-1+deb13u1
libngtcp2-crypto-gnutls8 1.11.0-1+deb13u1
libnsl2 1.3.0-3+b3
libnss-systemd 257.13-1~deb13u1
libp11-kit0 0.25.5-3
libpam-modules 1.7.0-5
libpam-modules-bin 1.7.0-5
libpam-runtime 1.7.0-5
libpam-systemd 257.13-1~deb13u1
libpam-wtmpdb 0.73.0-3+deb13u1
libpam0g 1.7.0-5
libpcre2-8-0 10.46-1~deb13u3
libperl5.40 5.40.1-6+deb13u1
libpipeline1 1.5.8-1
libpopt0 1.19+dfsg-2
libproc2-0 2:4.0.4-9
libprotobuf-c1 1.5.1-1
libpsl5t64 0.21.2-1.1+b1
libpython3-stdlib 3.13.5-1
libpython3.13-minimal 3.13.5-2+deb13u5
libpython3.13-stdlib 3.13.5-2+deb13u5
libreadline8t64 8.2-6
librtmp1 2.4+20151223.gitfa8646d.1-2+b5
libsasl2-2 2.1.28+dfsg1-9
libsasl2-modules-db 2.1.28+dfsg1-9
libseccomp2 2.6.0-2
libselinux1 3.8.1-1
libsemanage-common 3.8.1-1
libsemanage2 3.8.1-1
libsepol2 3.8.1-1
libslang2 2.3.3-5+b2
libsmartcols1 2.41.5-0+deb13u1
libsqlite3-0 3.46.1-7+deb13u2
libss2 1.47.2-3+b12
libssh2-1t64 1.11.1-1+deb13u2
libssl3t64 3.5.7-1~deb13u3
libstdc++6 14.2.0-19
libsystemd-shared 257.13-1~deb13u1
libsystemd0 257.13-1~deb13u1
libtasn1-6 4.20.0-2+deb13u1
libtext-charwidth-perl 0.04-11+b4
libtext-iconv-perl 1.7-8+b4
libtext-wrapi18n-perl 0.06-10
libtinfo6 6.5+20250216-2
libtirpc-common 1.3.6+ds-1
libtirpc3t64 1.3.6+ds-1
libtlsrpt0 0.5.0rc1-2
libuchardet0 0.0.8-1+b2
libudev1 257.13-1~deb13u1
libunistring5 1.3-2
liburcu8t64 0.15.2-2
libuuid1 2.41.5-0+deb13u1
libuv1t64 1.50.0-2
libwrap0 7.6.q-36
libwtmpdb0 0.73.0-3+deb13u1
libxml2 2.12.7+dfsg+really2.9.14-2.1+deb13u3
libxtables12 1.8.11-2
libxxhash0 0.8.3-2
libzstd1 1.5.7+dfsg-1
linux-sysctl-defaults 4.12.1
locales 2.41-12+deb13u4
login 1:4.16.0-2+really2.41.5-0+deb13u1
login.defs 1:4.17.4-2
logrotate 3.22.0-1
logsave 1.47.2-3+b12
lsof 4.99.4+dfsg-2
man-db 2.13.1-1
manpages 6.9.1-1
mawk 1.3.4.20250131-1
media-types 13.0.0
mount 2.41.5-0+deb13u1
nano 8.4-1+deb13u1
ncurses-base 6.5+20250216-2
ncurses-bin 6.5+20250216-2
ncurses-term 6.5+20250216-2
netbase 6.5
netcat-traditional 1.10-50
nftables 1.1.3-1
openssh-client 1:10.0p1-7+deb13u4
openssh-server 1:10.0p1-7+deb13u4
openssh-sftp-server 1:10.0p1-7+deb13u4
openssl 3.5.7-1~deb13u3
openssl-provider-legacy 3.5.7-1~deb13u3
passwd 1:4.17.4-2
patch 2.8-2
perl 5.40.1-6+deb13u1
perl-base 5.40.1-6+deb13u1
perl-modules-5.40 5.40.1-6+deb13u1
pigz 2.8-1
postfix 3.10.13-0+deb13u1
procps 2:4.0.4-9
python-apt-common 3.0.0
python3 3.13.5-1
python3-apt 3.0.0
python3-certifi 2025.1.31+ds-1
python3-chardet 5.2.0+dfsg-2
python3-charset-normalizer 3.4.2-1
python3-debconf 1.5.91
python3-debian 1.0.1
python3-debianbts 4.1.1
python3-idna 3.10-1+deb13u1
python3-minimal 3.13.5-1
python3-reportbug 13.2.0
python3-requests 2.32.3+dfsg-5+deb13u1
python3-urllib3 2.3.0-3+deb13u2
python3.13 3.13.5-2+deb13u5
python3.13-minimal 3.13.5-2+deb13u5
readline-common 8.2-6
reportbug 13.2.0
runit-helper 2.16.4
sed 4.9-2+deb13u1
sensible-utils 0.0.25
sqv 1.3.0-3+b2
ssh 1:10.0p1-7+deb13u4
systemd 257.13-1~deb13u1
systemd-sysv 257.13-1~deb13u1
systemd-timesyncd 257.13-1~deb13u1
sysvinit-utils 3.14-4
tar 1.35+dfsg-3.1
traceroute 1:2.1.6-1
tzdata 2026c-0+deb13u1
ucf 3.0052
udev 257.13-1~deb13u1
util-linux 2.41.5-0+deb13u1
util-linux-extra 2.41.5-0+deb13u1
vim-common 2:9.1.1230-2
vim-tiny 2:9.1.1230-2
wamerican 2020.12.07-4
wget 1.25.0-2
whiptail 0.52.25-1
wtmpdb 0.73.0-3+deb13u1
xz-utils 5.8.1-1+deb13u1
zlib1g 1:1.3.dfsg+really1.3.1-1+b1
documentation/audits/night-fixes-2026-10-05/partE/raw/e-pass.log
+2
View File
@@ -0,0 +1,2 @@
=== felhom-agent 0.144.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true block=hub ===
time=2026-10-05T08:13:34.060+02:00 level=INFO msg="osupdate: START" run=20261005T061334Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261005T061334Z
documentation/audits/night-fixes-2026-10-05/partE/rollback1.txt
+31
View File
@@ -0,0 +1,31 @@
### 05:44:03 before
bind9-dnsutils 1:9.20.29-1~deb13u1
bind9-host 1:9.20.29-1~deb13u1
bind9-libs:amd64 1:9.20.29-1~deb13u1
libpcre2-8-0:amd64 10.46-1~deb13u3
libpython3.13-minimal:amd64 3.13.5-2+deb13u5
libpython3.13-stdlib:amd64 3.13.5-2+deb13u5
libssh2-1t64:amd64 1.11.1-1+deb13u2
libssl3t64:amd64 3.5.7-1~deb13u3
libxml2:amd64 2.12.7+dfsg+really2.9.14-2.1+deb13u3
openssl 3.5.7-1~deb13u3
openssl-provider-legacy 3.5.7-1~deb13u3
python3.13 3.13.5-2+deb13u5
python3.13-minimal 3.13.5-2+deb13u5
279
### 05:44:05 rollback 1 (13 packages, simulated clean before)
0 upgraded, 0 newly installed, 13 downgraded, 0 to remove and 0 not upgraded.
audit rc=0
bind9-dnsutils 1:9.20.26-1~deb13u1
bind9-host 1:9.20.26-1~deb13u1
bind9-libs:amd64 1:9.20.26-1~deb13u1
libpcre2-8-0:amd64 10.46-1~deb13u2
libpython3.13-minimal:amd64 3.13.5-2+deb13u4
libpython3.13-stdlib:amd64 3.13.5-2+deb13u4
libssh2-1t64:amd64 1.11.1-1+deb13u1
libssl3t64:amd64 3.5.7-1~deb13u2
libxml2:amd64 2.12.7+dfsg+really2.9.14-2.1+deb13u1
openssl 3.5.7-1~deb13u2
openssl-provider-legacy 3.5.7-1~deb13u2
python3.13 3.13.5-2+deb13u4
python3.13-minimal 3.13.5-2+deb13u4
documentation/audits/night-fixes-2026-10-05/partF/FINDINGS.md
+73
View File
@@ -0,0 +1,73 @@
# Part F — a box that is OFF every night (Tester 2) — read-only spike, 2026-10-05
Collected by a read-only helper session (hub DB copy at ~05:11 UTC, non-secret columns only, copy shredded; hub
log; source at felhom.eu 7221ee5c, controller 69e9145, agent c8d12f1). Two claims re-checked by the main session
against source: the 05:00 deadline check skips `down` customers (`hub/internal/monitor/deadline.go` ~360), and the
controller's daily scheduler always picks the NEXT future time (`controller/internal/scheduler/scheduler.go`
`nextDailyRun`, ~389-408). Hub DB times are UTC; the hub log prints Budapest time.
## Q1 — Tester 2's record (customer `Tester-2`, host `Tester-2-be8404`)
Customer created 2026-09-30 (language en); host created 2026-10-04 16:07 UTC (agent 0.142.0, controller 0.292.0);
no apps installed. Online twice on 2026-10-04: ~15:54–17:13 UTC, and 18:03:42–18:06:24 UTC (last controller report).
Nothing since. One evening of data: no daytime pattern can be established.
| Leg | Result | Evidence |
|---|---|---|
| Whole-guest backup, local | ran once | 16:21:10 UTC, success |
| Whole-guest backup, PBS | ran once | 16:31:13 UTC, verify ok |
| OS update | ran once | `os_reports` 40/41, trigger `night`, 16:24/16:25 UTC, 49 + 106 packages, healthy |
| DB dump, tier-2 copy | never | — |
| Off-site copy | never | escrow `pending` |
| App updates | none | no apps |
| Restore-test | never | `restore_tests: []` in every report |
Events: 17:58 `host_stale`, 18:01 `node_stale`, 18:05–06 recovered; 18:51 stale again; 19:36 `node_down` +
`host_down` (error) — the household was mailed "Your server cannot be reached." at 19:36:42 UTC. Hub log
2026-10-05 05:00 Budapest: "Deadline check: … 0 backup missed … 1 skipped (down)".
**What fires if it stays off every night:**
- every evening `node_stale`/`host_stale` after 45 min; every night `node_down`/`host_down` after 90 min — the
household gets a `node_down` mail each night and a "reachable again" mail each morning (the 6 h cooldown does not
stop a daily repeat);
- **never** `expected_backup_missed` / `expected_dbdump_missed` while it is down at 05:00 (the deadline check skips
down customers);
- `restore_test_stale` ~2026-10-11 16:13 UTC (local tier, 7 d) and ~2026-10-16 (PBS, 12 d) — not skipped for down boxes;
- `os_update_stale` 2026-10-11 16:24 UTC unless a primary backup happens before;
- `offsite_stale` (48 h) once escrow is completed — the off-site leg runs only at night.
**What the household sees:** the backups page shows "Esedékes / Due" or "Naprakész / Up to date" and "No successful
backup yet" per tier. No text says the server must stay on at night or that a night was missed because it was off.
## Q2 — does anything catch up when the box comes back?
- **DB dump, tier-2, off-site, the chained app-update leg, the controller's 04:30 self-update:** NO. The daily
scheduler always picks the next future time; `LastRun` is in memory only and never consulted. (A hub floor still
reaches the box by day, after any report.)
- **Whole-guest backup:** PARTIAL. The controller asks every 5 min; due = newest archive ≥ 24 h old (read from
storage, survives a reboot); runs only in the window [W+2h, W+6h) (04:30–08:30 for W 02:30), OR by the safety valve
(never backed up, or last > 48 h). A box on only in the afternoon/evening gets about one every 2 days.
- **OS leg:** 90 s after any successful primary backup, ≥ 20 h apart — so it follows the valve, by day, under the
`night` label.
- **Restore-test:** checks every 6 h, the timer restarts at each agent start, no check at start — a box never on for
6 h in a row never runs one.
- Not verified: a SUSPENDED (not powered-off) laptop would probably run the daily jobs late rather than skip them
(reasoned from Go timers, not tested).
## Q3 — architecture
**No architecture document covers a box that is not always on.** Partial mentions: `11-os-updates.md` ("a box off for
a weekend does not alarm"); `07-backup-architecture.md` (the night chain and the [W+2h, W+6h) gate — no safety valve,
no catch-up); `08-alarm-ladder.md` (45/90 min); `where-felhom-stands.yaml` `fail.expected-downtime` (missing, R-285).
The intent "a box powered on only outside its window never starves" exists only as controller code comments
(`quiesce.go`).
## Q4 — register
R-285 (open, P4: no notion of expected downtime), R-30 (open, P3), R-195 / R-321 (closed: skips keyed off the
wrong fact), R-86 / R-100 (closed). **No row covers a box that is off every night.**
## Not determined
The daytime online pattern; Tester 2's actual W (on the box, which is off; 02:30 assumed); the type of signed job 53
(queued 18:35 UTC, after the box went off — the 2026-10-04 agent update, per the night's records).
documentation/audits/night-fixes-2026-10-05/system-page-end.txt
+6
View File
@@ -0,0 +1,6 @@
demo-felhom-8363b5 Demo Ügyfél ring 0 → normal updates ON switch off running 9.2.2 7.0.2-6-pve 7.0.2-6-pve 13.7 ring0-20261005T055135Z 80 0 none no 10 s no 0 armed 0.144.1 13.7 ring0-20261005T055135Z 0 7 29.8.2 2.3.6-1~debian.13~trixie on ring0-20261005T055135Z 29 min ago · nothing · 12 s demo-hp-bb76ea Demo HP partial: no runni
demo-hp-bb76ea Demo HP partial: no running customer guest ring 0 → normal updates ON switch off unknown 9.2.2 unknown unknown unknown ring0-20261005T061858Z 78 0 unknown no unknown no unknown not installed 0.144.1 unknown ring0-20261005T061858Z 0 7 unknown unknown unknown ring0-20261005T061858Z 3 min ago · nothing · 19 s tester-
tester-1-d70be4 Tester 1 ring 1 → demo updates ON switch off running 9.2.2 7.0.2-6-pve 7.0.2-6-pve 13.5 — 185 106 none no 10 s no 0 armed 0.144.1 13.6 — 49 0 29.8.2 2.3.6-1~debian.13~trixie on — 10 h ago · nothing · 16 s Amber: worth a look. Red: an operator alarm fires (`08` §6.3). "unknown": the box could not read the value —
documentation/audits/night-fixes-2026-10-05/tester2-reads.txt
+2
View File
@@ -0,0 +1,2 @@
2026-10-05T05:52:25Z ('Tester-2-be8404', '0.142.0', '2026-10-04 18:05:48')
2026-10-05T06:24:47Z ('Tester-2-be8404', '0.142.0', '2026-10-04 18:05:48')