night 2026-10-22: the seven 15:47 rulings in effect or ready (D6, D8 live; D21 waits for controller 0.340.0; D3/D5 website patch; D4 two patches); data-loss candidates proven/fixed/filed; R-1214 cause + capped copy proven on 9202; R-1204 alarm built; repos-private read; STATUS morning list + D29/D30/D31; CONTEXT; 03/05/07/09; rows 178 -> 180 (opened R-1218..R-1222, closed R-206, R-992, R-1207)
gates / gates (push) Successful in 16m7s
gates / gates (push) Successful in 16m7s
This commit is contained in:
@@ -129,6 +129,10 @@ the one trust root; the agent's API token is untouched; a box gets the new rule
|
||||
`agent_config_update` (the order: signed `agent_update` first, then the bundle — after the bundle, an agent below
|
||||
0.146.0 cannot update itself on that box).
|
||||
|
||||
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — built as a ready patch, NOT shipped] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
|
||||
|
||||
**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b). The repair is STATUS decision D30.
|
||||
|
||||
## 4. Control model — reconcile + signed destructive ops
|
||||
|
||||
Two channels, split by **reversibility**, not by transport.
|
||||
|
||||
@@ -454,6 +454,11 @@ warning mails the operator (anything lost, still broken, failed or skipped); inf
|
||||
The operator mail renders the counts as one `Repair:` line. The household's side is NOT a mail: the lost files (name,
|
||||
date taken) are shown on its own app page, on its box, until it dismisses them — file names never leave the box.
|
||||
|
||||
|
||||
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, built, NOT deployed; R-1204]
|
||||
|
||||
**[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.
|
||||
|
||||
### 15.5 How "the Hungarian did not change" is known
|
||||
|
||||
56 goldens captured from v0.117.0 before any string moved, in
|
||||
|
||||
@@ -647,6 +647,8 @@ leg** — LIVE on demo-felhom, `backups/secondary/bookstack/` and `.../docmost/`
|
||||
`recovery-unit` and nothing else, at 156 MB and 86 MB (INV Part B.2). Since v0.183.0 the button
|
||||
refuses **before** stopping the app and names the action that works.
|
||||
|
||||
**[DESIGN] A template may leave a named volume out of every backup (2026-10-22, operator D21 = B, R-49; controller 0.340.0).** `.felhom.yml` `backup.volumes: [{name: <volume>, class: excluded}]` — the name as the compose file declares it; the controller dumps every named volume of the app EXCEPT `<project>_<name>` (`appbackup.ExcludedDockerVolumes` → `backup.Manager.dumpableVolumes`), so the volume leaves the local unit, the second-drive copy and the off-site copy at once, and the next run's stale clean-up removes its old tar. Only class `excluded` acts; a name the compose file does not declare is ignored and logged; **a list that would exclude every named volume of the app is refused whole** (nothing excluded) — a template typo must not empty a backup (`TestR49_*`). Use it only for data the app rebuilds by itself: the first user is Immich's `immich_ml_cache` (model weights, re-downloaded at start) — 9202: unit 1.16 GB → 337 MB; with the volume removed, Immich started and its ML answered after 9.2 s (`audits/night-2026-10-22/P2-D21/`). A controller older than 0.340.0 ignores the key (inert). The raw database folder and Immich's own `upload/backups/` stay in until a restore from the dump alone is proven (D21's option A).
|
||||
|
||||
### 6.3 Where capture and restore disagree
|
||||
|
||||
**[FACT]** Three asymmetries, each source-cited:
|
||||
|
||||
@@ -2549,7 +2549,13 @@ well, or the box goes on following the live catalog and reports success. Filed a
|
||||
is fixed, the drill procedure is:
|
||||
|
||||
1. save `controller.yaml` as `controller.yaml.pre-update-night`;
|
||||
2. set `git.repo_url` (and `username`/`token` — the drill repo is private);
|
||||
2. set `git.repo_url` (and `username`/`token` — the drill repo is private). **The box's login is the Gitea user
|
||||
`drill-reader`** — not an admin, a read collaborator on `admin/app-catalog-drill` only, token scope
|
||||
`read:repository`; it cannot write or push and sees no other private repo (measured: 200 / 403 / 403 / 404,
|
||||
`audits/night-2026-10-22/P2-D6/01-token-limits.txt`). The token is `DRILL_CATALOG_TOKEN` and the account's password
|
||||
`DRILL_READER_PW` in CC's `sec` store — never printed. *Operator ruling 2026-10-22 15:47, D6: a read-only login, not
|
||||
a public drill repo (R-992, closed).* Since that night 9202 keeps `git.username`/`git.token` while it follows the
|
||||
live catalog (the public repo ignores them), so pointing it at the drill is the URL plus step 3;
|
||||
3. **remove `<data>/catalog-cache`**;
|
||||
4. restart the controller, sync, **rescan** (R-607: a sync can answer „nincs változás" while the
|
||||
catalog has moved, and the badge answers from the stale value until the rescan);
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
Hub appliance from the D4 proof VM 9343 (ISO 1.30.0 on demo-hp), read from a one-tar copy of hub.db + -wal + -shm at 18:04 CEST:
|
||||
id 49, status registered, first_seen 2026-10-22 14:59:13 UTC, not discarded. The operator presses Discard on the hub Hosts page.
|
||||
@@ -0,0 +1,12 @@
|
||||
# R-1187 (c) on 9202 — a .fab import into an installed app that is not running (throwaway uptime-kuma, its one named
|
||||
# volume uptime-kuma_uptime_kuma_data; markers written with a `docker run --rm -v <vol>:/v alpine touch` helper)
|
||||
## REPRODUCED on controller 0.340.0-rc.1 (before the fix), 2026-10-22 15:21–15:23 UTC
|
||||
15:21:47 before export: MARK-A ; export (stop_app true) → uptime-kuma_20261022-152152.fab (22 KB) ; then MARK-N written
|
||||
CONTROL (app RUNNING): before import MARK-A MARK-N → import ok → after: MARK-A (a clean replace)
|
||||
CASE (app STOPPED): before import MARK-A MARK-N → import ok → after: MARK-A MARK-N (MERGED — MARK-N survived)
|
||||
controller log 15:22:54: [DEBUG] [appexport] stack is deployed but not running — will overwrite config in place
|
||||
## FIXED on controller 0.340.0-rc.2 (d5f88a2), 15:30–15:32 UTC
|
||||
STOPPED: before import MARK-A MARK-N MARK-X → import REFUSED: „Az alkalmazás telepítve van, de nem fut, ezért a
|
||||
csomagot nem töltjük vissza: a régi adatai összekeveredhetnének a csomagéval. Az alkalmazás érintetlen.
|
||||
Indítsd el (vagy távolítsd el), és próbáld újra." → after: MARK-A MARK-N MARK-X (nothing changed)
|
||||
RUNNING (control): start → import ok → after: MARK-A (the running path still replaces)
|
||||
@@ -0,0 +1,31 @@
|
||||
15:37:41 full avg10=0.02 avg60=0.11 avg300=0.03 total=1426421 anon 2977931264 file 1118461952 565907456 workingset_refault_anon 50674 pswpin 50160 pswpout 305782
|
||||
15:37:44 full avg10=0.38 avg60=0.17 avg300=0.05 total=1482650 anon 2954280960 file 1159651328 605614080 workingset_refault_anon 50677 pswpin 50163 pswpout 314688
|
||||
15:37:46 full avg10=1.03 avg60=0.30 avg300=0.07 total=1592995 anon 2817572864 file 1305579520 746491904 workingset_refault_anon 50794 pswpin 50264 pswpout 347405
|
||||
15:37:48 full avg10=1.02 avg60=0.32 avg300=0.08 total=1596156 anon 2817511424 file 1297158144 747855872 workingset_refault_anon 50903 pswpin 50363 pswpout 347719
|
||||
15:37:50 full avg10=3.92 avg60=0.86 avg300=0.20 total=1954843 anon 2753028096 file 1359564800 863498240 workingset_refault_anon 51669 pswpin 51088 pswpout 366763
|
||||
15:37:52 full avg10=6.11 avg60=1.36 avg300=0.30 total=2463722 anon 2661449728 file 1457446912 969809920 workingset_refault_anon 51954 pswpin 51356 pswpout 390137
|
||||
15:37:54 full avg10=12.43 avg60=2.66 avg300=0.58 total=3197424 anon 2582831104 file 1535963136 1084706816 workingset_refault_anon 57081 pswpin 56378 pswpout 419473
|
||||
15:37:56 full avg10=13.80 avg60=3.22 avg300=0.72 total=3584025 anon 2508480512 file 1612091392 1217191936 workingset_refault_anon 77682 pswpin 76958 pswpout 451531
|
||||
15:37:58 full avg10=13.83 avg60=3.58 avg300=0.81 total=3943183 anon 2517655552 file 1601052672 1265401856 workingset_refault_anon 90983 pswpin 90258 pswpout 462834
|
||||
15:38:00 full avg10=16.76 avg60=4.44 avg300=1.01 total=4439979 anon 2479181824 file 1640677376 1326792704 workingset_refault_anon 94312 pswpin 93556 pswpout 476381
|
||||
15:38:02 full avg10=14.99 avg60=4.53 avg300=1.05 total=4574276 anon 2442424320 file 1675341824 1376956416 workingset_refault_anon 96385 pswpin 95562 pswpout 487861
|
||||
15:38:04 full avg10=13.72 avg60=4.64 avg300=1.10 total=4753844 anon 2352107520 file 1761906688 1479811072 workingset_refault_anon 98070 pswpin 97243 pswpout 512237
|
||||
15:38:06 full avg10=12.69 avg60=4.75 avg300=1.14 total=4916243 anon 2322202624 file 1789628416 1534160896 workingset_refault_anon 103396 pswpin 102483 pswpout 522846
|
||||
15:38:08 full avg10=13.29 avg60=5.12 avg300=1.24 total=5374360 anon 2285264896 file 1815957504 1609768960 workingset_refault_anon 106378 pswpin 105437 pswpout 540349
|
||||
15:38:10 full avg10=12.69 avg60=5.28 avg300=1.30 total=5458434 anon 2280771584 file 1828876288 1611206656 workingset_refault_anon 110996 pswpin 110030 pswpout 540986
|
||||
15:38:12 full avg10=10.93 avg60=5.20 avg300=1.32 total=5541954 anon 2255233024 file 1856950272 1634451456 workingset_refault_anon 111176 pswpin 110172 pswpout 546993
|
||||
15:38:14 full avg10=10.76 avg60=5.36 avg300=1.38 total=5867729 anon 2228326400 file 1883500544 1666342912 workingset_refault_anon 112159 pswpin 111119 pswpout 554054
|
||||
15:38:16 full avg10=10.26 avg60=5.45 avg300=1.42 total=5916591 anon 2217484288 file 1896869888 1674969088 workingset_refault_anon 112406 pswpin 111355 pswpout 556387
|
||||
15:38:18 full avg10=8.95 avg60=5.37 avg300=1.43 total=5994829 anon 2192748544 file 1920950272 1699532800 workingset_refault_anon 113145 pswpin 112061 pswpout 562876
|
||||
15:38:20 full avg10=8.59 avg60=5.42 avg300=1.47 total=6156025 anon 2187468800 file 1923219456 1716023296 workingset_refault_anon 116205 pswpin 114986 pswpout 566345
|
||||
15:38:22 full avg10=8.30 avg60=5.47 avg300=1.51 total=6347323 anon 2151600128 file 1957306368 1768026112 workingset_refault_anon 118566 pswpin 117299 pswpout 577430
|
||||
15:38:24 full avg10=8.25 avg60=5.55 avg300=1.55 total=6510370 anon 2122932224 file 1983873024 1821814784 workingset_refault_anon 121530 pswpin 120222 pswpout 588409
|
||||
15:38:26 full avg10=7.66 avg60=5.54 avg300=1.58 total=6575873 anon 2118803456 file 1980669952 1831043072 workingset_refault_anon 127295 pswpin 125968 pswpout 591207
|
||||
15:38:28 full avg10=7.17 avg60=5.52 avg300=1.60 total=6694348 anon 2081148928 file 2015191040 1862037504 workingset_refault_anon 128328 pswpin 127000 pswpout 600104
|
||||
15:38:30 full avg10=6.06 avg60=5.37 avg300=1.60 total=6722977 anon 2087165952 file 2009464832 1862000640 workingset_refault_anon 130120 pswpin 128710 pswpout 600153
|
||||
15:38:32 full avg10=5.50 avg60=5.29 avg300=1.61 total=6817784 anon 2100473856 file 2002825216 1868599296 workingset_refault_anon 135578 pswpin 134022 pswpout 601868
|
||||
15:38:34 full avg10=6.50 avg60=5.48 avg300=1.67 total=7031932 anon 2092240896 file 2014990336 1877950464 workingset_refault_anon 136858 pswpin 135295 pswpout 604035
|
||||
15:38:36 full avg10=6.22 avg60=5.46 avg300=1.69 total=7114873 anon 2097594368 file 2007240704 1894051840 workingset_refault_anon 140929 pswpin 139220 pswpout 606078
|
||||
15:38:38 full avg10=5.10 avg60=5.28 avg300=1.68 total=7124559 anon 2099490816 file 2005364736 1893933056 workingset_refault_anon 142157 pswpin 140407 pswpout 606105
|
||||
15:38:40 full avg10=4.53 avg60=5.17 avg300=1.68 total=7163770 anon 2103656448 file 1998692352 1884336128 workingset_refault_anon 144792 pswpin 142968 pswpout 606570
|
||||
15:38:42 full avg10=3.71 avg60=5.00 avg300=1.67 total=7168499 anon 2106265600 file 1985736704 1883377664 workingset_refault_anon 146085 pswpin 144169 pswpout 606614
|
||||
|
@@ -0,0 +1,21 @@
|
||||
15:40:12 full avg10=0.00 avg60=1.20 avg300=1.26 total=7601138 anon 2332147712 file 1179697152 1768947712 workingset_refault_anon 198199 pswpin 195091 pswpout 608043
|
||||
15:40:14 full avg10=0.00 avg60=1.16 avg300=1.25 total=7643545 anon 2338930688 file 1762066432 1769275392 workingset_refault_anon 198205 pswpin 195095 pswpout 609142
|
||||
15:40:16 full avg10=0.18 avg60=1.15 avg300=1.25 total=7663987 anon 2334179328 file 1770704896 1769304064 workingset_refault_anon 198240 pswpin 195115 pswpout 609747
|
||||
15:40:18 full avg10=1.78 avg60=1.41 avg300=1.30 total=7866238 anon 2262745088 file 1841442816 1820577792 workingset_refault_anon 199695 pswpin 196569 pswpout 623826
|
||||
15:40:20 full avg10=2.00 avg60=1.46 avg300=1.31 total=7895241 anon 2264272896 file 1831309312 1820860416 workingset_refault_anon 200428 pswpin 197274 pswpout 624060
|
||||
15:40:22 full avg10=2.54 avg60=1.58 avg300=1.34 total=8069533 anon 2179461120 file 1922334720 1873776640 workingset_refault_anon 202865 pswpin 199612 pswpout 639226
|
||||
15:40:24 full avg10=3.89 avg60=1.85 avg300=1.40 total=8208381 anon 2180362240 file 1919160320 1873920000 workingset_refault_anon 203286 pswpin 200020 pswpout 639312
|
||||
15:40:26 full avg10=3.91 avg60=1.92 avg300=1.41 total=8484990 anon 2177998848 file 1924026368 1892077568 workingset_refault_anon 210517 pswpin 207142 pswpout 645594
|
||||
15:40:28 full avg10=5.56 avg60=2.28 avg300=1.49 total=8656485 anon 2108825600 file 1989308416 1930575872 workingset_refault_anon 211932 pswpin 208548 pswpout 657459
|
||||
15:40:30 full avg10=5.64 avg60=2.41 avg300=1.52 total=8716832 anon 2122145792 file 1971916800 1930854400 workingset_refault_anon 213791 pswpin 210388 pswpout 657569
|
||||
15:40:32 full avg10=5.34 avg60=2.46 avg300=1.54 total=8921295 anon 2118180864 file 1980985344 1948692480 workingset_refault_anon 216028 pswpin 212529 pswpout 659784
|
||||
15:40:34 full avg10=6.91 avg60=2.84 avg300=1.63 total=9137580 anon 2107707392 file 2000109568 1974226944 workingset_refault_anon 219998 pswpin 216461 pswpout 665599
|
||||
15:40:36 full avg10=6.38 avg60=2.87 avg300=1.64 total=9213522 anon 2100957184 file 2004856832 1993576448 workingset_refault_anon 222847 pswpin 219234 pswpout 669447
|
||||
15:40:38 full avg10=6.67 avg60=3.04 avg300=1.69 total=9404607 anon 2075275264 file 2028175360 2015838208 workingset_refault_anon 228327 pswpin 224669 pswpout 677555
|
||||
15:40:40 full avg10=5.82 avg60=3.01 avg300=1.69 total=9421787 anon 2078093312 file 2014978048 2015526912 workingset_refault_anon 229181 pswpin 225503 pswpout 677663
|
||||
15:40:42 full avg10=5.49 avg60=3.04 avg300=1.71 total=9573729 anon 2025185280 file 2072969216 2064064512 workingset_refault_anon 231993 pswpin 228214 pswpout 692314
|
||||
15:40:44 full avg10=6.49 avg60=3.30 avg300=1.77 total=9736793 anon 2023051264 file 2059718656 2065530880 workingset_refault_anon 235355 pswpin 231508 pswpout 693161
|
||||
15:40:46 full avg10=5.49 avg60=3.22 avg300=1.76 total=9831708 anon 2017460224 file 2087301120 2074292224 workingset_refault_anon 237874 pswpin 233974 pswpout 695918
|
||||
15:40:48 full avg10=5.40 avg60=3.28 avg300=1.79 total=9941218 anon 2014801920 file 2091048960 2079084544 workingset_refault_anon 238418 pswpin 234518 pswpout 697134
|
||||
15:40:50 full avg10=6.05 avg60=3.47 avg300=1.84 total=10066148 anon 2031009792 file 2072170496 2081452032 workingset_refault_anon 247990 pswpin 243754 pswpout 701826
|
||||
15:40:52 full avg10=5.14 avg60=3.39 avg300=1.83 total=10075229 anon 2033516544 file 2061938688 2081267712 workingset_refault_anon 251428 pswpin 247066 pswpout 701871
|
||||
|
@@ -0,0 +1,27 @@
|
||||
15:38:59 full avg10=0.87 avg60=3.86 avg300=1.59 total=7220110 anon 2155155456 file 1933524992 1882132480 workingset_refault_anon 158542 pswpin 156128 pswpout 606649
|
||||
15:39:01 full avg10=0.71 avg60=3.74 avg300=1.58 total=7234341 anon 2176008192 file 1922453504 1882083328 workingset_refault_anon 159499 pswpin 157036 pswpout 606649
|
||||
15:39:03 full avg10=0.58 avg60=3.61 avg300=1.56 total=7266319 anon 2173059072 file 1928216576 1882087424 workingset_refault_anon 159601 pswpin 157135 pswpout 607242
|
||||
15:39:05 full avg10=0.65 avg60=3.53 avg300=1.56 total=7287150 anon 2167078912 file 1926488064 1882349568 workingset_refault_anon 159703 pswpin 157201 pswpout 607982
|
||||
15:39:07 full avg10=0.72 avg60=3.44 avg300=1.56 total=7336362 anon 2170904576 file 1930149888 1882324992 workingset_refault_anon 160569 pswpin 158047 pswpout 607992
|
||||
15:39:09 full avg10=0.77 avg60=3.36 avg300=1.55 total=7356199 anon 2175414272 file 1923297280 1882296320 workingset_refault_anon 160637 pswpin 158104 pswpout 607992
|
||||
15:39:11 full avg10=0.81 avg60=3.29 avg300=1.55 total=7372570 anon 2172841984 file 1903226880 1882152960 workingset_refault_anon 161145 pswpin 158560 pswpout 607993
|
||||
15:39:13 full avg10=0.66 avg60=3.18 avg300=1.54 total=7381638 anon 2172796928 file 1903489024 1882152960 workingset_refault_anon 161160 pswpin 158560 pswpout 607993
|
||||
15:39:15 full avg10=0.54 avg60=3.07 avg300=1.53 total=7417217 anon 2178797568 file 1916338176 1882107904 workingset_refault_anon 162494 pswpin 159862 pswpout 607993
|
||||
15:39:17 full avg10=0.62 avg60=3.01 avg300=1.52 total=7429109 anon 2178826240 file 1916338176 1882099712 workingset_refault_anon 162515 pswpin 159868 pswpout 607993
|
||||
15:39:19 full avg10=0.51 avg60=2.91 avg300=1.51 total=7442053 anon 2179309568 file 1909116928 1882083328 workingset_refault_anon 162585 pswpin 159919 pswpout 607993
|
||||
15:39:21 full avg10=0.42 avg60=2.81 avg300=1.50 total=7450768 anon 2179932160 file 1905176576 1882066944 workingset_refault_anon 162682 pswpin 159995 pswpout 607993
|
||||
15:39:23 full avg10=0.34 avg60=2.72 avg300=1.49 total=7459779 anon 2180591616 file 1905315840 1882066944 workingset_refault_anon 162866 pswpin 160179 pswpout 607993
|
||||
15:39:25 full avg10=0.28 avg60=2.63 avg300=1.48 total=7474379 anon 2190475264 file 1896730624 1882017792 workingset_refault_anon 163514 pswpin 160816 pswpout 608009
|
||||
15:39:27 full avg10=0.23 avg60=2.54 avg300=1.47 total=7483721 anon 2183577600 file 1897766912 1881993216 workingset_refault_anon 163591 pswpin 160890 pswpout 608009
|
||||
15:39:29 full avg10=0.18 avg60=2.46 avg300=1.46 total=7490992 anon 2183839744 file 1897914368 1881993216 workingset_refault_anon 163657 pswpin 160955 pswpout 608009
|
||||
15:39:31 full avg10=0.15 avg60=2.38 avg300=1.45 total=7498777 anon 2184044544 file 1898643456 1881985024 workingset_refault_anon 163757 pswpin 161032 pswpout 608009
|
||||
15:39:33 full avg10=0.12 avg60=2.30 avg300=1.44 total=7515110 anon 2187968512 file 1898991616 1881899008 workingset_refault_anon 164702 pswpin 161960 pswpout 608009
|
||||
15:39:35 full avg10=0.10 avg60=2.22 avg300=1.43 total=7520357 anon 2187526144 file 1899126784 1881862144 workingset_refault_anon 164780 pswpin 162024 pswpout 608009
|
||||
15:39:37 full avg10=0.08 avg60=2.15 avg300=1.42 total=7533498 anon 2188107776 file 1900191744 1881817088 workingset_refault_anon 164899 pswpin 162123 pswpout 608009
|
||||
15:39:39 full avg10=0.06 avg60=2.08 avg300=1.41 total=7559069 anon 2189148160 file 1897832448 1881800704 workingset_refault_anon 164999 pswpin 162209 pswpout 608014
|
||||
15:39:41 full avg10=0.23 avg60=2.05 avg300=1.41 total=7565482 anon 2188591104 file 1880342528 1881784320 workingset_refault_anon 165052 pswpin 162256 pswpout 608036
|
||||
15:39:43 full avg10=0.19 avg60=1.98 avg300=1.40 total=7583622 anon 2188668928 file 1880240128 1881767936 workingset_refault_anon 165126 pswpin 162316 pswpout 608036
|
||||
15:39:45 full avg10=0.15 avg60=1.91 avg300=1.39 total=7587346 anon 2189262848 file 1878487040 1881767936 workingset_refault_anon 165233 pswpin 162414 pswpout 608036
|
||||
15:39:47 full avg10=0.12 avg60=1.85 avg300=1.38 total=7594985 anon 2181861376 file 1875628032 1881759744 workingset_refault_anon 165254 pswpin 162421 pswpout 608036
|
||||
15:39:49 full avg10=0.10 avg60=1.79 avg300=1.37 total=7599902 anon 2183262208 file 1658728448 1881718784 workingset_refault_anon 165522 pswpin 162621 pswpout 608036
|
||||
15:39:51 full avg10=0.08 avg60=1.73 avg300=1.36 total=7599902 anon 2173874176 file 1727365120 1881690112 workingset_refault_anon 165594 pswpin 162659 pswpout 608036
|
||||
|
@@ -0,0 +1,19 @@
|
||||
15:41:12 full avg10=0.79 avg60=2.46 avg300=1.71 total=10149379 anon 2095677440 file 1990664192 2079645696 workingset_refault_anon 269199 pswpin 264419 pswpout 701950
|
||||
15:41:14 full avg10=0.83 avg60=2.41 avg300=1.71 total=10177124 anon 2121056256 file 1971712000 2079608832 workingset_refault_anon 271376 pswpin 266520 pswpout 701964
|
||||
15:41:16 full avg10=1.04 avg60=2.39 avg300=1.71 total=10226245 anon 2127192064 file 1965178880 2079539200 workingset_refault_anon 272795 pswpin 267881 pswpout 701969
|
||||
15:41:18 full avg10=0.85 avg60=2.32 avg300=1.70 total=10245591 anon 2128060416 file 1959108608 2079531008 workingset_refault_anon 272851 pswpin 267921 pswpout 701969
|
||||
15:41:20 full avg10=0.69 avg60=2.24 avg300=1.69 total=10264675 anon 2129645568 file 1951543296 2079518720 workingset_refault_anon 273146 pswpin 268182 pswpout 701969
|
||||
15:41:22 full avg10=0.57 avg60=2.17 avg300=1.67 total=10275758 anon 2130022400 file 1952010240 2079510528 workingset_refault_anon 273203 pswpin 268225 pswpout 701969
|
||||
15:41:24 full avg10=0.65 avg60=2.13 avg300=1.67 total=10304890 anon 2131419136 file 1958985728 2079481856 workingset_refault_anon 273444 pswpin 268422 pswpout 701970
|
||||
15:41:26 full avg10=0.71 avg60=2.09 avg300=1.66 total=10326741 anon 2135916544 file 1949175808 2079481856 workingset_refault_anon 274703 pswpin 269668 pswpout 701970
|
||||
15:41:28 full avg10=0.58 avg60=2.02 avg300=1.65 total=10346752 anon 2136711168 file 1938386944 2079481856 workingset_refault_anon 274779 pswpin 269737 pswpout 701970
|
||||
15:41:30 full avg10=0.47 avg60=1.95 avg300=1.64 total=10355602 anon 2136711168 file 1938386944 2079481856 workingset_refault_anon 274779 pswpin 269737 pswpout 701970
|
||||
15:41:32 full avg10=0.57 avg60=1.92 avg300=1.64 total=10376110 anon 2138894336 file 1950375936 2079481856 workingset_refault_anon 275388 pswpin 270320 pswpout 701970
|
||||
15:41:34 full avg10=0.46 avg60=1.86 avg300=1.63 total=10386198 anon 2143576064 file 1940713472 2079449088 workingset_refault_anon 276447 pswpin 271329 pswpout 701970
|
||||
15:41:36 full avg10=0.56 avg60=1.83 avg300=1.62 total=10409920 anon 2142969856 file 1941934080 2079395840 workingset_refault_anon 276569 pswpin 271431 pswpout 701970
|
||||
15:41:38 full avg10=0.64 avg60=1.80 avg300=1.62 total=10428296 anon 2143977472 file 1943191552 2079395840 workingset_refault_anon 276840 pswpin 271678 pswpout 701970
|
||||
15:41:40 full avg10=0.52 avg60=1.74 avg300=1.61 total=10438990 anon 2144059392 file 1937944576 2079395840 workingset_refault_anon 276858 pswpin 271695 pswpout 701970
|
||||
15:41:42 full avg10=0.43 avg60=1.69 avg300=1.59 total=10449658 anon 2143178752 file 1937944576 2079342592 workingset_refault_anon 276860 pswpin 271697 pswpout 701970
|
||||
15:41:44 full avg10=0.35 avg60=1.63 avg300=1.58 total=10462674 anon 2143178752 file 1936896000 2079309824 workingset_refault_anon 276860 pswpin 271697 pswpout 701970
|
||||
15:41:46 full avg10=0.28 avg60=1.58 avg300=1.57 total=10478707 anon 2126831616 file 1685921792 2079297536 workingset_refault_anon 277148 pswpin 271927 pswpout 701970
|
||||
15:41:48 full avg10=0.23 avg60=1.52 avg300=1.56 total=10478707 anon 2127224832 file 1685921792 2079289344 workingset_refault_anon 277152 pswpin 271931 pswpout 701970
|
||||
|
@@ -0,0 +1,9 @@
|
||||
2026/10/22 15:53:05 scheduler.go:150: [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-10-23 03:30 CEST
|
||||
2026/10/22 15:53:05 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="tier2-backup" schedule="03:30" nextRun=2026-10-23T03:30:00+02:00 totalJobs=14
|
||||
2026/10/22 15:53:05 scheduler.go:85: [DEBUG] [scheduler] daily job tier2-backup: next run at 2026-10-23 03:30:00 CEST (waiting 9h36m54s)
|
||||
2026/10/22 15:53:46 auth.go:144: [DEBUG] [web] auth: valid session for POST /api/backup/tier2
|
||||
2026/10/22 15:53:46 router.go:78: [DEBUG] [api] POST /api/backup/tier2 (path=/backup/tier2)
|
||||
2026/10/22 15:53:51 tier2.go:491: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (516.4 MB, 2 leg(s), 5s)
|
||||
2026/10/22 15:53:53 tier2.go:491: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (127.4 MB, 1 leg(s), 1s)
|
||||
2026/10/22 15:53:53 tier2.go:491: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 0s)
|
||||
2026/10/22 15:53:53 tier2.go:553: [INFO] [backup] Tier 2 run complete: 3 app(s) processed (incl. volume-only — F6)
|
||||
@@ -0,0 +1,3 @@
|
||||
R-1214 live tier2 proof, 9202, controller 0.340.0-rc.4, 2026-10-22 ~18:00 CEST
|
||||
rsync -ani --delete source vs tier2 copy (run inside felhom-controller; the guest has no rsync): differences: 0
|
||||
files: source 44, copy 44
|
||||
@@ -0,0 +1,62 @@
|
||||
15:53:45 full avg10=0.00 avg60=0.00 avg300=0.10 total=10551160 2
|
||||
15:53:49 full avg10=0.00 avg60=0.00 avg300=0.10 total=10563819 2
|
||||
15:53:52 full avg10=0.00 avg60=0.00 avg300=0.10 total=10565444 1
|
||||
15:53:55 full avg10=0.00 avg60=0.00 avg300=0.10 total=10565444 1
|
||||
15:53:58 full avg10=0.00 avg60=0.00 avg300=0.10 total=10565444 1
|
||||
15:54:01 full avg10=0.00 avg60=0.00 avg300=0.10 total=10565444 1
|
||||
15:54:04 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:07 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:10 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:13 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:16 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:19 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:22 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565444 1
|
||||
15:54:25 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565444 1
|
||||
15:54:28 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565549 1
|
||||
15:54:31 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565549 1
|
||||
15:54:34 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565549 1
|
||||
15:54:37 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565549 1
|
||||
15:54:40 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565549 1
|
||||
15:54:43 full avg10=0.00 avg60=0.00 avg300=0.07 total=10565549 1
|
||||
15:54:46 full avg10=0.00 avg60=0.00 avg300=0.07 total=10565549 1
|
||||
15:54:49 full avg10=0.00 avg60=0.00 avg300=0.07 total=10565549 1
|
||||
15:54:52 full avg10=0.00 avg60=0.00 avg300=0.07 total=10565549 1
|
||||
15:54:55 full avg10=0.00 avg60=0.00 avg300=0.07 total=10565549 1
|
||||
15:54:58 full avg10=0.00 avg60=0.00 avg300=0.07 total=10567212 1
|
||||
15:55:01 full avg10=0.00 avg60=0.00 avg300=0.07 total=10567212 1
|
||||
15:55:04 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:07 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:10 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:13 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:16 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:19 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:22 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:25 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:28 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:31 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:34 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:37 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:40 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567241 1
|
||||
15:55:43 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567241 1
|
||||
15:55:46 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:55:49 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:55:52 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:55:55 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:55:58 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:01 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:04 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:07 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:10 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:13 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:16 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:19 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:22 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:25 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:28 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:31 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:34 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:37 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:40 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:43 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:46 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:49 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
|
@@ -0,0 +1,9 @@
|
||||
#!/bin/bash
|
||||
# r1214run.sh LABEL CMD... — runs on demo-hp: samples 9202's cgroup every 2 s while CMD runs (in 9202), prints a summary
|
||||
L=$1; shift; CG=/sys/fs/cgroup/lxc/9202; OUT=/tmp/r1214-$L.tsv; : > $OUT
|
||||
( while :; do echo -e "$(date -u +%T)\t$(sed -n 2p $CG/memory.pressure)\t$(grep -E '^(anon|file) ' $CG/memory.stat | tr '\n' ' ')\t$(cat $CG/memory.swap.current)\t$(grep -E '^(pswpin|pswpout|workingset_refault_anon) ' $CG/memory.stat | tr '\n' ' ')" >> $OUT; sleep 2; done ) &
|
||||
SP=$!; F0=$(sed -n 2p $CG/memory.pressure | grep -o 'total=[0-9]*' | cut -d= -f2); T0=$(date +%s)
|
||||
LC_ALL=C pct exec 9202 -- "$@"; RC=$?
|
||||
T1=$(date +%s); F1=$(sed -n 2p $CG/memory.pressure | grep -o 'total=[0-9]*' | cut -d= -f2); sleep 4; kill $SP
|
||||
PK=$(cut -f2 $OUT | grep -o 'avg10=[0-9.]*' | cut -d= -f2 | sort -n | tail -1)
|
||||
echo "$L: rc=$RC seconds=$((T1-T0)) full-stall-total=$(( (F1-F0)/1000 )) ms peak full avg10=$PK % samples=$(wc -l < $OUT)"
|
||||
@@ -26,6 +26,14 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-22 night — the seven decisions put into effect (full text of the rows: `git show 1100a344:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-206** | **The build-cache cap and the weekly prune exist only as a hand-edited `/etc/docker/daemon.json` on DooPlex — not in Ansible, so a rebuild loses them.** (P4) | CLOSED 2026-10-22 night — D8 ruled (operator 15:47: DooPlex Docker settings from git plus a weekly build-cache prune) and installed: homelab-manifests `b35ba83`, role `docker_daemon` with a `docker-buildcache-prune` service + timer (Sun 23:30, `docker builder prune --force --filter until=168h`, never `-a`, with `docker system df` before and after). The live `daemon.json` was identical, so no Docker restart (dockerd up since 2026-08-05); timer next run 2026-10-25 23:30. Part (b) — loosening the housekeeping ban — was not needed: the prune lives in its own role and unit. | homelab-manifests `b35ba83` CHANGELOG |
|
||||
| **R-992** | **The drill-catalog route that `09` §6.5 documents cannot be followed: `admin/app-catalog-drill` is PRIVATE and no box has catalog credentials.** (P3) | CLOSED 2026-10-22 night — D6 ruled (operator 15:47: a read-only login for the scratch box's drill catalog). Gitea user `drill-reader` (not admin), read collaborator on `admin/app-catalog-drill`, token scope `read:repository`, kept in `sec`. Proven: the token reads the drill repo (200); write 403; push 403; the other private repos 404; 9202 cloned the drill catalog with it and returned to the live catalog with the credential kept. `09` §6.5 updated. | `audits/night-2026-10-22/P2-D6/01-token-limits.txt` |
|
||||
| **R-1207** | **A host with no (or too little) swap gets nothing from the guest swap fix (R-1192), and the agent never adds host swap.** (P2) | CLOSED 2026-10-22 night — DECIDED, D27 = report only (operator 15:47): the agent's WARN, the hub's `host_swap_short` (D28 = A) and the install-page sentence tell; the agent adds no swap and the installer refuses nobody. Reopens at the first BYO box with too little host swap. | `audits/night-2026-10-21/P2-noswap/` |
|
||||
|
||||
## 2026-10-22 day — the mail list and the swap rule (full text: R-1216 `git show 25dae2f5:documentation/backlog/OPEN-ITEMS.md`; R-1199's original row `git show 115f15c8:documentation/backlog/OPEN-ITEMS.md` — its reopened text of 2026-10-22 was never committed, and its substance is the Closed cell below)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -7,6 +7,20 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
## Moved from STATUS on 2026-10-22
|
||||
|
||||
**Updated 2026-10-22 (day).** A rebuilt box no longer overwrites the family's saved mail list (hub 0.165.0, controller 0.339.0); demo-felhom's own list, replaced this morning, is back. The swap warning now fires only when a machine has clearly too little swap (your D28 = A).
|
||||
|
||||
**Decided 2026-10-22 15:47 — „all picks”** (the operator; each ruling is in its row, and in CONTEXT.md): D3 = change the wording now (R-985); D4 = root SSH from the LAN by key only, the console password stays (R-982); D5 = approve the minimum machine (R-991); D6 = a read-only login for the scratch box's drill catalog (R-992, closed); D8 = DooPlex Docker settings from git plus a weekly build-cache prune (R-206, closed); D21 = B (R-49); D27 = report only (R-1207, closed). The rows as they stood:
|
||||
|
||||
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||
|---|---|---|---|---|---|---|
|
||||
| D27 | R-1207 | A host with no swap gets nothing from the swap fix, and the agent never adds swap. Add host swap automatically (BYO too), or refuse an 8 GB host without swap at install? | **Report only for now** (our installer gives 7.75 GB, and Tester 2's laptop used our installer; the alarm and the install-page sentence now tell you) — decide after the first BYO box | Add it: the agent changes a disk its owner set up; refuse: a BYO household cannot start | a no-swap host can still freeze; you get the alarm | — |
|
||||
| D21 | R-49 | Immich's off-site copy: leave out only the AI model files now (B), or also the raw database folder and Immich's own backups (A)? | **B now** (−62 %, no new risk); A's other parts after their proof | A: −92 %, but it needs a restore path we have never run | every Immich box sends about 12 times its real data off-site each night | — |
|
||||
| D8 | R-206 | Install DooPlex's Docker settings from git (Docker restarts once), with a weekly build-cache prune? | **Yes, in a quiet window** (a rebuilt DooPlex then keeps the settings) | No: the live settings stay hand-made, and a rebuild loses them | git and the host can drift apart | — |
|
||||
| D6 | R-992 | The scratch box cannot read the private drill catalog: give it a read-only login, or make that copy public? | **A read-only login** (the drill catalog stays private) | Public copy: anyone can read the drill catalog | every new-app test needs staging by hand | new-app tests |
|
||||
| D5 | R-991 | Approve the minimum machine: x86-64, 4 cores, 8 GB RAM, 256 GB SSD (recommended: 16 GB, 512 GB)? | **Approve** (measured on the N100 with 61 apps) — the 8 GB freeze of 2026-10-20 is fixed by agent 0.162.0 (11 h soak on a fresh 8 GB box, no freeze) | Change it: one edit | no hardware page; a buyer with a Raspberry Pi can still download the installer | the hardware page |
|
||||
| D4 | R-982 | Stop root SSH with a password from the household network (key only)? | **Yes — key only** (the password stays for the console) | Keep it: anyone on the household network can try the password | the host keeps accepting root password logins from the LAN | — |
|
||||
| D3 | R-985 | Our „zero-knowledge" wording is stronger than the product: change the wording, or remove the operator's root reach to the key? | **Change the wording now**; look again after the first sale | Remove the reach: a design change to break-glass | the website promises more than the product does | the website text |
|
||||
|
||||
**Updated 2026-10-22 (morning, your yeses).** Released: controller 0.338.0 (global floor), agent 0.162.0 (on the 3 test boxes), hub 0.164.0, installer 1.30.0 and the website download button. The new host-swap check warns on boxes bigger than 8 GB; that is too strict (decide D28).
|
||||
|
||||
**Decided 2026-10-22 11:42 — D28 = A** (the operator): | D28 | R-1216 | The host-swap check warns when host swap is below the guests' whole swap. A 16 GB box with the installer's 8 GB swap trips it, but the soak used 2.7 GB. Loosen it to „below what the installer gives"? | **Loosen it (A)** (no mail for a healthy box; 938 MB still warns) | Keep (B): you get a mail per big box, and the install page tells such a household to call us | demo-hp and demo-felhom stay flagged; one mail each, already sent | — |
|
||||
|
||||
Reference in New Issue
Block a user