night 2026-10-22: the seven 15:47 rulings in effect or ready (D6, D8 live; D21 waits for controller 0.340.0; D3/D5 website patch; D4 two patches); data-loss candidates proven/fixed/filed; R-1214 cause + capped copy proven on 9202; R-1204 alarm built; repos-private read; STATUS morning list + D29/D30/D31; CONTEXT; 03/05/07/09; rows 178 -> 180 (opened R-1218..R-1222, closed R-206, R-992, R-1207)
gates / gates (push) Successful in 16m7s
gates / gates (push) Successful in 16m7s
This commit is contained in:
+32
@@ -16,6 +16,38 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-22 15:47 — operator rulings, „all picks" (the decide table, rows now in `STATUS-HISTORY.md`).**
|
||||
> **D3** = change the zero-knowledge wording now, look again after the first sale (R-985). **D4** = root SSH from the
|
||||
> LAN by key only; the console password stays (R-982). **D5** = approve the minimum machine, `audits/sizing-2026-10-13/SIZING.md`
|
||||
> §5 (R-991). **D6** = a read-only login for the scratch box's drill catalog, not a public drill repo (R-992). **D8** =
|
||||
> DooPlex Docker settings from git plus a weekly build-cache prune (R-206). **D21 = B** — only Immich's model cache
|
||||
> leaves the backups now; the raw DB folder and Immich's own backups after their restore proof (R-49). **D27** = report
|
||||
> only: the agent adds no host swap and the installer refuses nobody; reopen at the first BYO box short of swap (R-1207).
|
||||
|
||||
> **2026-10-22 evening → night — the rulings put into effect (night brief; nothing released, deployed or published).**
|
||||
> **D8 live:** homelab-manifests `b35ba83`, role `docker_daemon` + `docker-buildcache-prune.{service,timer}` (Sun 23:30,
|
||||
> `docker builder prune --force --filter until=168h`, never `-a`); the live `daemon.json` was identical → no Docker
|
||||
> restart. **D6 live:** Gitea user `drill-reader` (not admin), read collaborator on `admin/app-catalog-drill`, token
|
||||
> `read:repository` in `sec`; proven (read 200, write/push 403, other private repos 404); `09` §6.5. **D21 built:**
|
||||
> controller `6a17c95` — `.felhom.yml` `backup.volumes: [{name, class: excluded}]` → `appbackup.ExcludedDockerVolumes`
|
||||
> (declared names only; refuses to exclude all of an app's named volumes) → `backup.Manager.dumpableVolumes`; catalog
|
||||
> `ce33e3b` (Immich, live, inert before controller 0.340.0); 9202: unit 1.16 GB → 337 MB, Immich ML answered after the
|
||||
> cache was removed. **D3/D5:** the volunteer guides committed (`0639cf07`); the website change is a ready patch
|
||||
> (`audits/night-2026-10-22/P2-D3-D5/website-D3-D5.patch`) — a push to `website/` publishes. **D4:** two ready patches
|
||||
> (`P2-D4/`): an agent config-bundle drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`
|
||||
> (`PermitRootLogin prohibit-password`; Debian's Include comes first and the first value wins; felhom-sshd :8822 has its
|
||||
> own config) and the installer's uninstall; proven on a fresh ISO 1.30.0 install (VM 9343, destroyed; its appliance 49
|
||||
> waits for the operator's Discard). **Data-loss candidates** (`audits/night-2026-10-22/P3-candidates/analysis.md`):
|
||||
> R-1187(c) fixed (`d5f88a2`: an import over an installed, not-running app is refused — 9202 reproduced the merge on
|
||||
> rc.1 and the refusal on rc.2); R-1189(d) fixed (`nightchain.CatchUp.After` runs the off-site proof + integrity check
|
||||
> once); new rows R-1218..R-1222 with their proposed fixes. **R-1214** (03:30 pressure): the copy's page cache is
|
||||
> charged to the guest; `backup/tier2.go` now runs rsync in a 512 MB memory-capped container of the controller's own
|
||||
> image for clean `/mnt/` paths with no symlink (`03c0a69`, `ae39467`; exit 125 → plain rsync); 9202 stall 5.7 s → 0.37 s.
|
||||
> **R-1204:** agent `a0c88e5` reports `handshake_future_s`; hub 0.166.0 (`e267c5e2`, built, not deployed) mails
|
||||
> `wg_tunnel_stale`/`wg_tunnel_ok` to the operator only. **Repos private (R-925):** only felhom-controller can go
|
||||
> private (after a CI read token); STATUS D29. Picks waiting: D29 (No), D30 (re-key through `RegisterWGPeerForHost`),
|
||||
> D31 (R-1205: skip an un-acked, key-less deletion). Record: `REPORT-night-2026-10-22.md`.
|
||||
|
||||
> **2026-10-22 ~14:50 — operator acts, checked by CC (read only).** Host `Tester-2-be8404` deleted (12:54:47 UTC);
|
||||
> customer `Tester-2` language set to Magyar; customer `Tester-2` RESET (12:55:56–12:56:04 UTC, journal #25, every leg
|
||||
> ok). The first live run of R-32's purge-first leg: logged in with the sub-account's own password, found no repository
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
# REPORT — night 2026-10-22: seven decisions, the data-loss candidates, the 03:30 pressure, the tunnel alarm, private repositories
|
||||
|
||||
Evening → night brief, started ~16:00 CEST. Nothing was released, deployed or published, except the catalog change and
|
||||
the scratch box's catalog login (both allowed by the brief). Design read: `07-backup-architecture.md` §6.1.1, §6.2,
|
||||
§10.2; `03-host-agent.md` (tunnel, bundle); `05-hub-architecture.md` §15; `06-offsite-connectivity.md`;
|
||||
`09-update-architecture.md` §6.5.
|
||||
|
||||
## 1. Five lines
|
||||
|
||||
1. **D8:** done before 18:00 — DooPlex's Docker settings come from git (no change, no restart), and a weekly build-cache
|
||||
prune timer runs Sunday 23:30.
|
||||
2. **The seven decisions:** live — D6, D8, D27 (report only). Ready, waiting for a release or publish — D21
|
||||
(controller 0.340.0), D3 + D5 (one website patch), D4 (agent bundle patch + installer patch).
|
||||
3. **Data-loss candidates:** 2 proven and fixed in code (R-1187 c, R-1189 d); 5 real in code, now rows with a written
|
||||
fix (R-1218, R-1219, R-1220, R-1221, R-1222); 1 closed half (R-1189 b, fixed in 0.304.0); 3 to the operator
|
||||
(R-1205 as D31, R-1189 f, R-1190 d); the rest stay as named candidates.
|
||||
4. **03:30 pressure:** the second-drive copy's file cache is charged to the guest and pushes the apps out of memory.
|
||||
The fix runs the copy in a 512 MB memory-capped container: full stall 5.7 s → 0.37 s on the scratch box.
|
||||
5. **Private repositories:** only felhom-controller can go private (after one CI change). felhom.eu, the agent and the
|
||||
catalog must stay public: the website, every install and every box read them with no login.
|
||||
|
||||
## 2. The morning list (also in STATUS)
|
||||
|
||||
1. Publish the website patch (D3 wording, D5 hardware page).
|
||||
2. D4: the agent bundle patch with the next agent release (your vouch) + the installer patch with a new installer tag.
|
||||
Discard appliance 49 on the hub's Hosts page (tonight's test install).
|
||||
3. Deploy hub 0.166.0 (tunnel alarm) and release the agent with `handshake_future_s` — one agent release with item 2.
|
||||
4. Release controller 0.340.0 (R-49, R-1187 c, R-1189 d, R-1214), MinAgent 0.158.1.
|
||||
5. Decide D29 — the controller repository private? CC: No, not now.
|
||||
6. Decide D30 — the tunnel repair. CC: the agent makes a new key and registers it on the path that exists.
|
||||
7. (D31, R-1205, when there is time.) Tester 2's „go" from yesterday stays on the list.
|
||||
|
||||
## 3. Per candidate
|
||||
|
||||
| Item | Result | Reason |
|
||||
|---|---|---|
|
||||
| R-1187 a | REAL in code → row R-1187 (now P2) | an export drops a folder with a symlink and still says done; capture code: needs a red-proof + 9202 proof |
|
||||
| R-1187 b | REAL in code → R-1187 | the import sets no owners and drops symlinks |
|
||||
| R-1187 c | **PROVEN and FIXED** (`d5f88a2`) | 9202 rc.1: an import over a stopped app merged old + new data; rc.2: refused, app untouched |
|
||||
| R-1187 d | PLAUSIBLE → R-1187 | no busy check; after (a) |
|
||||
| R-1187 f, g | REAL in code → R-1187 | HDD_PATH trusted; postgis missed, psql without ON_ERROR_STOP |
|
||||
| R-1188 a, c | already CLOSED (R-1195; 0.337.0) | — |
|
||||
| R-1188 b, d, e | R-1188, re-ranked P4 | unlikely or a sentence |
|
||||
| R-1189 a | REAL, half-live → R-1189 | agent code; rides an agent release |
|
||||
| R-1189 b | SPLIT: skipped nights CLOSED (R-899, 0.304.0); creep → **R-1221** (P4) | demo-hp journal: a night run after each of 3 off-cycle runs; creep 1.5–5 min/night measured |
|
||||
| R-1189 c, g, i | REAL in code → R-1189 (P4-shaped) | quiesce vs restore; spring forward 2027; double click |
|
||||
| R-1189 d | **FIXED** (`d5f88a2`) | the catch-up now runs the proof + integrity once; red-proved |
|
||||
| R-1189 e | REAL in code → **R-1222** (hub, P3) | an evening backup time gives a false „dump missed" every morning |
|
||||
| R-1189 f | operator → R-1189 | design question |
|
||||
| R-1190 a | REAL in code → **R-1218** (P2) | proposed fix, not built: key code needs a live proof, and 9202 has no hub |
|
||||
| R-1190 b | REAL in code → **R-1219** (P2) | the top bar links to a ceremony the page hides during an offer |
|
||||
| R-1190 c | PLAUSIBLE → R-1190 | a read-only fingerprint compare decides it |
|
||||
| R-1190 d | operator → R-1190 | design question |
|
||||
| R-1190 g | REAL in code → **R-1220** (P2) | an unreadable app.yaml lists a live folder as kept (deletable) |
|
||||
| R-1205 | operator → **D31** | the F-14 gate is deliberate (ruling 2026-07-13) |
|
||||
| R-1206 | NARROWED | Felhom tier: no loss, by design (R-822 guard). NAS tier: a unit test is owed |
|
||||
|
||||
Research input: `documentation/audits/night-2026-10-22/P3-candidates/analysis.md`.
|
||||
|
||||
## 4. Records
|
||||
|
||||
**Commits.**
|
||||
- felhom-controller: `6a17c95` (R-49), `d5f88a2` (R-1187 c, R-1189 d), `03c0a69` + `ae39467` (R-1214), `df78b6c` (docs).
|
||||
CI jobs 2214, 2220, 2221, 2222, 2223 — all success, attempt 1.
|
||||
- felhom-agent: `a0c88e5` (R-1204 + D4), `7651461` (D4 moved to a patch), `e803ad2` (docs, a correction). CI 2217,
|
||||
2218 success; 2224 in progress at the time of writing — see the final note below.
|
||||
- felhom.eu: `0639cf07`, `e267c5e2` + `1100a344` (one push), and this commit. CI 2215, 2219 success (2219 covers `e267c5e2`).
|
||||
- app-catalog-felhom.eu: `ce33e3b` (Immich `backup.volumes`). CI 2216 success.
|
||||
- homelab-manifests: `b35ba83` (D8), `c7506df` + `b0cce29` (REPORT). No CI in that repo.
|
||||
- Drill catalog (scratch, no CI by design): `2ba4b53`, `9ebdba6`, `b0350a3` (tree = live `ce33e3b` again).
|
||||
|
||||
**Images pushed.** `felhom-controller:0.340.0-rc.1`..`rc.4` (never offered to a household: not X.Y.Z).
|
||||
`felhom-hub:0.166.0` (built, NOT deployed: the live hub and `manifests/hub.yaml` are 0.165.0, read 18:2x).
|
||||
|
||||
**Teardown, three layers.**
|
||||
- **Machine (9202):** the throwaway uptime-kuma removed through the product (its `.fab` deleted); the hog container and
|
||||
19 GB of test files removed; memory 25898 MB / swap 512 MB again; Immich's `.felhom.yml` back to its pre-night file;
|
||||
controller back on 0.339.0 (healthy, every app up). Kept on purpose: the `drill-reader` catalog login (D6).
|
||||
Left: the four rc images in 9202's own Docker store, like earlier rc images there.
|
||||
- **Host (demo-hp):** VM 9343 destroyed (own id); `/tmp/r1214*` removed; two sampler loops stopped. The scratch
|
||||
storage and VM 341 / CT 9401 are standing items, not tonight's.
|
||||
- **Hub:** appliance 49 from VM 9343 is still „registered" — Discard is the operator's (session-only). Nothing else.
|
||||
- **DooPlex:** the D8 timer (intended). Nothing else changed.
|
||||
|
||||
**Rows:** before 178, after 180. Opened 5: R-1218, R-1219, R-1220, R-1221, R-1222. Closed 3: R-206 (D8), R-992 (D6),
|
||||
R-1207 (D27). Updated: R-49, R-925, R-982, R-985, R-991, R-1187 (P3→P2), R-1188 (P3→P4), R-1189, R-1190, R-1204,
|
||||
R-1205, R-1206, R-1214, R-1215.
|
||||
|
||||
**Unproven claims:** 35 of 55 not walked — unchanged. **System poster:** no listed fact changed (nothing that the
|
||||
poster draws shipped tonight) — none of the three actions was needed.
|
||||
|
||||
**Observations.**
|
||||
- 9202's sync refused Immich's template (R-971: Immich's data sits on a bench mount, not a household drive). —
|
||||
NOT-A-FINDING: R-971 by design; the file was placed by hand for the proof and put back.
|
||||
- The local unit restore of Immich was refused (R-538). — NOT-A-FINDING: by design; the proof used an empty-cache start.
|
||||
- A non-`--fast` catalog gate run started the runtime gate on DooPlex; its canary refused and left nothing. —
|
||||
NOT-A-FINDING: the canary is the control; the rule (`--fast` on DooPlex) exists.
|
||||
- Images and packages stay public when a repository goes private. — NOT-A-FINDING: they follow the user `admin`, and
|
||||
boxes must pull them; recorded in R-925 and D29.
|
||||
- A CI read token is a precondition of D29. — NOT-A-FINDING: a step of that decision, written in it.
|
||||
- R-925 said „32 source comments"; the served count is 61 on four pages. — fixed in the row.
|
||||
- **CC's own mistake 1:** the agent CHANGELOG and felhom.eu commit `1100a344`'s message say a push of the installer
|
||||
publishes it. Wrong: `/scripts/` is served from the tag `installer-v1.35.0` (R-110). Corrected in the agent CHANGELOG
|
||||
(`e803ad2`) and R-982; the commit message cannot change (no force-push). Keeping D4 as a patch still holds: both
|
||||
halves need a release that was forbidden tonight.
|
||||
- **CC's own mistake 2:** controller commit `df78b6c` carries a „Co-Authored-By" line, which the brief forbids. It stays
|
||||
(no force-push); no other commit tonight has one.
|
||||
- **CC's own mistake 3:** homelab-manifests' REPORT first claimed a separate daemon.json backup. There was none — the
|
||||
file did not change and is byte-identical to the role's copy in git (`cmp`). Corrected in `b0cce29`.
|
||||
- The guest has no rsync. — NOT-A-FINDING: rsync lives in the controller image; the capped copy uses that image.
|
||||
- R-1215's evidence is gone (Immich restarted at 02:16). — said in the row; it stays a watch.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-10-22 (day).** A rebuilt box no longer overwrites the family's saved mail list (hub 0.165.0, controller 0.339.0); demo-felhom's own list, replaced this morning, is back. The swap warning now fires only when a machine has clearly too little swap (your D28 = A).
|
||||
**Updated 2026-10-22 (night).** Your seven picks of 15:47 are in effect or ready: D6 and D8 are live, D21 waits for the controller release, and D3, D5 (the website) and D4 (root SSH) are ready patches. Four controller fixes are built and proven on the scratch box, among them a gentler night copy (the 03:30 memory pressure) and no import over a stopped app. Nothing was released or deployed; the morning list below has the order.
|
||||
|
||||
## What works
|
||||
- A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
|
||||
@@ -16,10 +16,9 @@
|
||||
|
||||
## What is broken
|
||||
- **P1:** about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
|
||||
- Two root-level weaknesses wait on your decisions D3 and D4 (R-985, R-982).
|
||||
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862).
|
||||
- DooPlex: the failed disk is still in the box, its alarms silenced (R-934); something sets the shared checkout to „bare" (R-935).
|
||||
- The scratch box cannot use agent features (R-1083) or read the drill catalog (R-992).
|
||||
- The scratch box cannot use agent features (R-1083).
|
||||
|
||||
## What we work on
|
||||
- Dated checks, run by CC. 2026-10-23: no app stop when the host is busy (R-1173). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts).
|
||||
@@ -30,15 +29,11 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
|
||||
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||
|---|---|---|---|---|---|---|
|
||||
| D27 | R-1207 | A host with no swap gets nothing from the swap fix, and the agent never adds swap. Add host swap automatically (BYO too), or refuse an 8 GB host without swap at install? | **Report only for now** (our installer gives 7.75 GB, and Tester 2's laptop used our installer; the alarm and the install-page sentence now tell you) — decide after the first BYO box | Add it: the agent changes a disk its owner set up; refuse: a BYO household cannot start | a no-swap host can still freeze; you get the alarm | — |
|
||||
| D21 | R-49 | Immich's off-site copy: leave out only the AI model files now (B), or also the raw database folder and Immich's own backups (A)? | **B now** (−62 %, no new risk); A's other parts after their proof | A: −92 %, but it needs a restore path we have never run | every Immich box sends about 12 times its real data off-site each night | — |
|
||||
| D31 | R-1205 | After a box death, a second install that is deleted before it makes a key stops the automatic re-issue of the whole-box copy. Let the hub skip such a deletion, or keep your manual press? | **Skip it** (that install held no key, so nothing is lost — the same reasoning the key carry uses) | Keep the press: you get a mail and press „Re-issue PBS credentials” each time | that household gets no whole-box copy until you press | — |
|
||||
| D30 | R-1204 | After a box ran with its clock ahead, how does its tunnel to ep0 heal? | **The agent makes a new key and registers it** (the peer path that exists; nothing new on ep0) | ep0 removes and re-adds the peer on a hub-signed request: a new write path on ep0 | with hub 0.166.0 you get the alarm, and the repair is by hand on ep0 | the tunnel fix |
|
||||
| D29 | R-925 | Make the controller's repository private now? (felhom.eu, the agent and the catalog must stay public: the website, every install and every box read them with no login.) | **No, not now** (its image stays public anyway, and the no-secret gate is the control; look again when felhom.eu can go private) | Yes: a read token in CI for every repo that clones it, one more secret to keep | nothing changes | — |
|
||||
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
|
||||
| D8 | R-206 | Install DooPlex's Docker settings from git (Docker restarts once), with a weekly build-cache prune? | **Yes, in a quiet window** (a rebuilt DooPlex then keeps the settings) | No: the live settings stay hand-made, and a rebuild loses them | git and the host can drift apart | — |
|
||||
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
|
||||
| D6 | R-992 | The scratch box cannot read the private drill catalog: give it a read-only login, or make that copy public? | **A read-only login** (the drill catalog stays private) | Public copy: anyone can read the drill catalog | every new-app test needs staging by hand | new-app tests |
|
||||
| D5 | R-991 | Approve the minimum machine: x86-64, 4 cores, 8 GB RAM, 256 GB SSD (recommended: 16 GB, 512 GB)? | **Approve** (measured on the N100 with 61 apps) — the 8 GB freeze of 2026-10-20 is fixed by agent 0.162.0 (11 h soak on a fresh 8 GB box, no freeze) | Change it: one edit | no hardware page; a buyer with a Raspberry Pi can still download the installer | the hardware page |
|
||||
| D4 | R-982 | Stop root SSH with a password from the household network (key only)? | **Yes — key only** (the password stays for the console) | Keep it: anyone on the household network can try the password | the host keeps accepting root password logins from the LAN | — |
|
||||
| D3 | R-985 | Our „zero-knowledge" wording is stronger than the product: change the wording, or remove the operator's root reach to the key? | **Change the wording now**; look again after the first sale | Remove the reach: a design change to break-glass | the website promises more than the product does | the website text |
|
||||
|
||||
## Waiting on you — do
|
||||
|
||||
@@ -51,8 +46,13 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
|
||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
||||
|
||||
**Today, 2026-10-22.**
|
||||
1. Give Tester 2's friend the owner passphrase and the one-page guide, and tell him „go". The reset worked: his old copies are gone, nobody else's changed. If not: nothing happens; the link mail expires in 7 days (he can ask for a new one).
|
||||
**Morning, 2026-10-23 — in this order.**
|
||||
1. Publish the website patch: the D3 wording and the D5 hardware page (R-985, R-991). Say „publish”; CC pushes it. If not: the site keeps the stronger promise and has no hardware page.
|
||||
2. D4 rides two releases, both ready patches: the agent's config bundle (the next agent release, your vouch) and a new installer tag (R-982). Also press Discard on the hub's Hosts page for appliance 49 (tonight's test install). If not: hosts keep taking root passwords from the LAN.
|
||||
3. Deploy hub 0.166.0 (the clock-ahead tunnel alarm, built) and release the agent with its tunnel field, in the same agent release as item 2 (R-1204). If not: a dead tunnel stays silent.
|
||||
4. Release controller 0.340.0: Immich's model cache out of backups, no import over a stopped app, the catch-up checks, the capped night copy (R-49, R-1187, R-1189, R-1214). If not: these four wait.
|
||||
5. Decide D29 (a private repository) and 6. D30 (the tunnel repair), in the table above; D31 when you have time.
|
||||
7. Tester 2 (from 2026-10-22): give the friend the owner passphrase and the one-page guide, and say „go”. If not: the link mail expires in 7 days.
|
||||
**The rest**
|
||||
- Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12.
|
||||
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
|
||||
|
||||
@@ -129,6 +129,10 @@ the one trust root; the agent's API token is untouched; a box gets the new rule
|
||||
`agent_config_update` (the order: signed `agent_update` first, then the bundle — after the bundle, an agent below
|
||||
0.146.0 cannot update itself on that box).
|
||||
|
||||
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — built as a ready patch, NOT shipped] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
|
||||
|
||||
**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b). The repair is STATUS decision D30.
|
||||
|
||||
## 4. Control model — reconcile + signed destructive ops
|
||||
|
||||
Two channels, split by **reversibility**, not by transport.
|
||||
|
||||
@@ -454,6 +454,11 @@ warning mails the operator (anything lost, still broken, failed or skipped); inf
|
||||
The operator mail renders the counts as one `Repair:` line. The household's side is NOT a mail: the lost files (name,
|
||||
date taken) are shown on its own app page, on its box, until it dismisses them — file names never leave the box.
|
||||
|
||||
|
||||
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, built, NOT deployed; R-1204]
|
||||
|
||||
**[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.
|
||||
|
||||
### 15.5 How "the Hungarian did not change" is known
|
||||
|
||||
56 goldens captured from v0.117.0 before any string moved, in
|
||||
|
||||
@@ -647,6 +647,8 @@ leg** — LIVE on demo-felhom, `backups/secondary/bookstack/` and `.../docmost/`
|
||||
`recovery-unit` and nothing else, at 156 MB and 86 MB (INV Part B.2). Since v0.183.0 the button
|
||||
refuses **before** stopping the app and names the action that works.
|
||||
|
||||
**[DESIGN] A template may leave a named volume out of every backup (2026-10-22, operator D21 = B, R-49; controller 0.340.0).** `.felhom.yml` `backup.volumes: [{name: <volume>, class: excluded}]` — the name as the compose file declares it; the controller dumps every named volume of the app EXCEPT `<project>_<name>` (`appbackup.ExcludedDockerVolumes` → `backup.Manager.dumpableVolumes`), so the volume leaves the local unit, the second-drive copy and the off-site copy at once, and the next run's stale clean-up removes its old tar. Only class `excluded` acts; a name the compose file does not declare is ignored and logged; **a list that would exclude every named volume of the app is refused whole** (nothing excluded) — a template typo must not empty a backup (`TestR49_*`). Use it only for data the app rebuilds by itself: the first user is Immich's `immich_ml_cache` (model weights, re-downloaded at start) — 9202: unit 1.16 GB → 337 MB; with the volume removed, Immich started and its ML answered after 9.2 s (`audits/night-2026-10-22/P2-D21/`). A controller older than 0.340.0 ignores the key (inert). The raw database folder and Immich's own `upload/backups/` stay in until a restore from the dump alone is proven (D21's option A).
|
||||
|
||||
### 6.3 Where capture and restore disagree
|
||||
|
||||
**[FACT]** Three asymmetries, each source-cited:
|
||||
|
||||
@@ -2549,7 +2549,13 @@ well, or the box goes on following the live catalog and reports success. Filed a
|
||||
is fixed, the drill procedure is:
|
||||
|
||||
1. save `controller.yaml` as `controller.yaml.pre-update-night`;
|
||||
2. set `git.repo_url` (and `username`/`token` — the drill repo is private);
|
||||
2. set `git.repo_url` (and `username`/`token` — the drill repo is private). **The box's login is the Gitea user
|
||||
`drill-reader`** — not an admin, a read collaborator on `admin/app-catalog-drill` only, token scope
|
||||
`read:repository`; it cannot write or push and sees no other private repo (measured: 200 / 403 / 403 / 404,
|
||||
`audits/night-2026-10-22/P2-D6/01-token-limits.txt`). The token is `DRILL_CATALOG_TOKEN` and the account's password
|
||||
`DRILL_READER_PW` in CC's `sec` store — never printed. *Operator ruling 2026-10-22 15:47, D6: a read-only login, not
|
||||
a public drill repo (R-992, closed).* Since that night 9202 keeps `git.username`/`git.token` while it follows the
|
||||
live catalog (the public repo ignores them), so pointing it at the drill is the URL plus step 3;
|
||||
3. **remove `<data>/catalog-cache`**;
|
||||
4. restart the controller, sync, **rescan** (R-607: a sync can answer „nincs változás" while the
|
||||
catalog has moved, and the badge answers from the stale value until the rescan);
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
Hub appliance from the D4 proof VM 9343 (ISO 1.30.0 on demo-hp), read from a one-tar copy of hub.db + -wal + -shm at 18:04 CEST:
|
||||
id 49, status registered, first_seen 2026-10-22 14:59:13 UTC, not discarded. The operator presses Discard on the hub Hosts page.
|
||||
@@ -0,0 +1,12 @@
|
||||
# R-1187 (c) on 9202 — a .fab import into an installed app that is not running (throwaway uptime-kuma, its one named
|
||||
# volume uptime-kuma_uptime_kuma_data; markers written with a `docker run --rm -v <vol>:/v alpine touch` helper)
|
||||
## REPRODUCED on controller 0.340.0-rc.1 (before the fix), 2026-10-22 15:21–15:23 UTC
|
||||
15:21:47 before export: MARK-A ; export (stop_app true) → uptime-kuma_20261022-152152.fab (22 KB) ; then MARK-N written
|
||||
CONTROL (app RUNNING): before import MARK-A MARK-N → import ok → after: MARK-A (a clean replace)
|
||||
CASE (app STOPPED): before import MARK-A MARK-N → import ok → after: MARK-A MARK-N (MERGED — MARK-N survived)
|
||||
controller log 15:22:54: [DEBUG] [appexport] stack is deployed but not running — will overwrite config in place
|
||||
## FIXED on controller 0.340.0-rc.2 (d5f88a2), 15:30–15:32 UTC
|
||||
STOPPED: before import MARK-A MARK-N MARK-X → import REFUSED: „Az alkalmazás telepítve van, de nem fut, ezért a
|
||||
csomagot nem töltjük vissza: a régi adatai összekeveredhetnének a csomagéval. Az alkalmazás érintetlen.
|
||||
Indítsd el (vagy távolítsd el), és próbáld újra." → after: MARK-A MARK-N MARK-X (nothing changed)
|
||||
RUNNING (control): start → import ok → after: MARK-A (the running path still replaces)
|
||||
@@ -0,0 +1,31 @@
|
||||
15:37:41 full avg10=0.02 avg60=0.11 avg300=0.03 total=1426421 anon 2977931264 file 1118461952 565907456 workingset_refault_anon 50674 pswpin 50160 pswpout 305782
|
||||
15:37:44 full avg10=0.38 avg60=0.17 avg300=0.05 total=1482650 anon 2954280960 file 1159651328 605614080 workingset_refault_anon 50677 pswpin 50163 pswpout 314688
|
||||
15:37:46 full avg10=1.03 avg60=0.30 avg300=0.07 total=1592995 anon 2817572864 file 1305579520 746491904 workingset_refault_anon 50794 pswpin 50264 pswpout 347405
|
||||
15:37:48 full avg10=1.02 avg60=0.32 avg300=0.08 total=1596156 anon 2817511424 file 1297158144 747855872 workingset_refault_anon 50903 pswpin 50363 pswpout 347719
|
||||
15:37:50 full avg10=3.92 avg60=0.86 avg300=0.20 total=1954843 anon 2753028096 file 1359564800 863498240 workingset_refault_anon 51669 pswpin 51088 pswpout 366763
|
||||
15:37:52 full avg10=6.11 avg60=1.36 avg300=0.30 total=2463722 anon 2661449728 file 1457446912 969809920 workingset_refault_anon 51954 pswpin 51356 pswpout 390137
|
||||
15:37:54 full avg10=12.43 avg60=2.66 avg300=0.58 total=3197424 anon 2582831104 file 1535963136 1084706816 workingset_refault_anon 57081 pswpin 56378 pswpout 419473
|
||||
15:37:56 full avg10=13.80 avg60=3.22 avg300=0.72 total=3584025 anon 2508480512 file 1612091392 1217191936 workingset_refault_anon 77682 pswpin 76958 pswpout 451531
|
||||
15:37:58 full avg10=13.83 avg60=3.58 avg300=0.81 total=3943183 anon 2517655552 file 1601052672 1265401856 workingset_refault_anon 90983 pswpin 90258 pswpout 462834
|
||||
15:38:00 full avg10=16.76 avg60=4.44 avg300=1.01 total=4439979 anon 2479181824 file 1640677376 1326792704 workingset_refault_anon 94312 pswpin 93556 pswpout 476381
|
||||
15:38:02 full avg10=14.99 avg60=4.53 avg300=1.05 total=4574276 anon 2442424320 file 1675341824 1376956416 workingset_refault_anon 96385 pswpin 95562 pswpout 487861
|
||||
15:38:04 full avg10=13.72 avg60=4.64 avg300=1.10 total=4753844 anon 2352107520 file 1761906688 1479811072 workingset_refault_anon 98070 pswpin 97243 pswpout 512237
|
||||
15:38:06 full avg10=12.69 avg60=4.75 avg300=1.14 total=4916243 anon 2322202624 file 1789628416 1534160896 workingset_refault_anon 103396 pswpin 102483 pswpout 522846
|
||||
15:38:08 full avg10=13.29 avg60=5.12 avg300=1.24 total=5374360 anon 2285264896 file 1815957504 1609768960 workingset_refault_anon 106378 pswpin 105437 pswpout 540349
|
||||
15:38:10 full avg10=12.69 avg60=5.28 avg300=1.30 total=5458434 anon 2280771584 file 1828876288 1611206656 workingset_refault_anon 110996 pswpin 110030 pswpout 540986
|
||||
15:38:12 full avg10=10.93 avg60=5.20 avg300=1.32 total=5541954 anon 2255233024 file 1856950272 1634451456 workingset_refault_anon 111176 pswpin 110172 pswpout 546993
|
||||
15:38:14 full avg10=10.76 avg60=5.36 avg300=1.38 total=5867729 anon 2228326400 file 1883500544 1666342912 workingset_refault_anon 112159 pswpin 111119 pswpout 554054
|
||||
15:38:16 full avg10=10.26 avg60=5.45 avg300=1.42 total=5916591 anon 2217484288 file 1896869888 1674969088 workingset_refault_anon 112406 pswpin 111355 pswpout 556387
|
||||
15:38:18 full avg10=8.95 avg60=5.37 avg300=1.43 total=5994829 anon 2192748544 file 1920950272 1699532800 workingset_refault_anon 113145 pswpin 112061 pswpout 562876
|
||||
15:38:20 full avg10=8.59 avg60=5.42 avg300=1.47 total=6156025 anon 2187468800 file 1923219456 1716023296 workingset_refault_anon 116205 pswpin 114986 pswpout 566345
|
||||
15:38:22 full avg10=8.30 avg60=5.47 avg300=1.51 total=6347323 anon 2151600128 file 1957306368 1768026112 workingset_refault_anon 118566 pswpin 117299 pswpout 577430
|
||||
15:38:24 full avg10=8.25 avg60=5.55 avg300=1.55 total=6510370 anon 2122932224 file 1983873024 1821814784 workingset_refault_anon 121530 pswpin 120222 pswpout 588409
|
||||
15:38:26 full avg10=7.66 avg60=5.54 avg300=1.58 total=6575873 anon 2118803456 file 1980669952 1831043072 workingset_refault_anon 127295 pswpin 125968 pswpout 591207
|
||||
15:38:28 full avg10=7.17 avg60=5.52 avg300=1.60 total=6694348 anon 2081148928 file 2015191040 1862037504 workingset_refault_anon 128328 pswpin 127000 pswpout 600104
|
||||
15:38:30 full avg10=6.06 avg60=5.37 avg300=1.60 total=6722977 anon 2087165952 file 2009464832 1862000640 workingset_refault_anon 130120 pswpin 128710 pswpout 600153
|
||||
15:38:32 full avg10=5.50 avg60=5.29 avg300=1.61 total=6817784 anon 2100473856 file 2002825216 1868599296 workingset_refault_anon 135578 pswpin 134022 pswpout 601868
|
||||
15:38:34 full avg10=6.50 avg60=5.48 avg300=1.67 total=7031932 anon 2092240896 file 2014990336 1877950464 workingset_refault_anon 136858 pswpin 135295 pswpout 604035
|
||||
15:38:36 full avg10=6.22 avg60=5.46 avg300=1.69 total=7114873 anon 2097594368 file 2007240704 1894051840 workingset_refault_anon 140929 pswpin 139220 pswpout 606078
|
||||
15:38:38 full avg10=5.10 avg60=5.28 avg300=1.68 total=7124559 anon 2099490816 file 2005364736 1893933056 workingset_refault_anon 142157 pswpin 140407 pswpout 606105
|
||||
15:38:40 full avg10=4.53 avg60=5.17 avg300=1.68 total=7163770 anon 2103656448 file 1998692352 1884336128 workingset_refault_anon 144792 pswpin 142968 pswpout 606570
|
||||
15:38:42 full avg10=3.71 avg60=5.00 avg300=1.67 total=7168499 anon 2106265600 file 1985736704 1883377664 workingset_refault_anon 146085 pswpin 144169 pswpout 606614
|
||||
|
@@ -0,0 +1,21 @@
|
||||
15:40:12 full avg10=0.00 avg60=1.20 avg300=1.26 total=7601138 anon 2332147712 file 1179697152 1768947712 workingset_refault_anon 198199 pswpin 195091 pswpout 608043
|
||||
15:40:14 full avg10=0.00 avg60=1.16 avg300=1.25 total=7643545 anon 2338930688 file 1762066432 1769275392 workingset_refault_anon 198205 pswpin 195095 pswpout 609142
|
||||
15:40:16 full avg10=0.18 avg60=1.15 avg300=1.25 total=7663987 anon 2334179328 file 1770704896 1769304064 workingset_refault_anon 198240 pswpin 195115 pswpout 609747
|
||||
15:40:18 full avg10=1.78 avg60=1.41 avg300=1.30 total=7866238 anon 2262745088 file 1841442816 1820577792 workingset_refault_anon 199695 pswpin 196569 pswpout 623826
|
||||
15:40:20 full avg10=2.00 avg60=1.46 avg300=1.31 total=7895241 anon 2264272896 file 1831309312 1820860416 workingset_refault_anon 200428 pswpin 197274 pswpout 624060
|
||||
15:40:22 full avg10=2.54 avg60=1.58 avg300=1.34 total=8069533 anon 2179461120 file 1922334720 1873776640 workingset_refault_anon 202865 pswpin 199612 pswpout 639226
|
||||
15:40:24 full avg10=3.89 avg60=1.85 avg300=1.40 total=8208381 anon 2180362240 file 1919160320 1873920000 workingset_refault_anon 203286 pswpin 200020 pswpout 639312
|
||||
15:40:26 full avg10=3.91 avg60=1.92 avg300=1.41 total=8484990 anon 2177998848 file 1924026368 1892077568 workingset_refault_anon 210517 pswpin 207142 pswpout 645594
|
||||
15:40:28 full avg10=5.56 avg60=2.28 avg300=1.49 total=8656485 anon 2108825600 file 1989308416 1930575872 workingset_refault_anon 211932 pswpin 208548 pswpout 657459
|
||||
15:40:30 full avg10=5.64 avg60=2.41 avg300=1.52 total=8716832 anon 2122145792 file 1971916800 1930854400 workingset_refault_anon 213791 pswpin 210388 pswpout 657569
|
||||
15:40:32 full avg10=5.34 avg60=2.46 avg300=1.54 total=8921295 anon 2118180864 file 1980985344 1948692480 workingset_refault_anon 216028 pswpin 212529 pswpout 659784
|
||||
15:40:34 full avg10=6.91 avg60=2.84 avg300=1.63 total=9137580 anon 2107707392 file 2000109568 1974226944 workingset_refault_anon 219998 pswpin 216461 pswpout 665599
|
||||
15:40:36 full avg10=6.38 avg60=2.87 avg300=1.64 total=9213522 anon 2100957184 file 2004856832 1993576448 workingset_refault_anon 222847 pswpin 219234 pswpout 669447
|
||||
15:40:38 full avg10=6.67 avg60=3.04 avg300=1.69 total=9404607 anon 2075275264 file 2028175360 2015838208 workingset_refault_anon 228327 pswpin 224669 pswpout 677555
|
||||
15:40:40 full avg10=5.82 avg60=3.01 avg300=1.69 total=9421787 anon 2078093312 file 2014978048 2015526912 workingset_refault_anon 229181 pswpin 225503 pswpout 677663
|
||||
15:40:42 full avg10=5.49 avg60=3.04 avg300=1.71 total=9573729 anon 2025185280 file 2072969216 2064064512 workingset_refault_anon 231993 pswpin 228214 pswpout 692314
|
||||
15:40:44 full avg10=6.49 avg60=3.30 avg300=1.77 total=9736793 anon 2023051264 file 2059718656 2065530880 workingset_refault_anon 235355 pswpin 231508 pswpout 693161
|
||||
15:40:46 full avg10=5.49 avg60=3.22 avg300=1.76 total=9831708 anon 2017460224 file 2087301120 2074292224 workingset_refault_anon 237874 pswpin 233974 pswpout 695918
|
||||
15:40:48 full avg10=5.40 avg60=3.28 avg300=1.79 total=9941218 anon 2014801920 file 2091048960 2079084544 workingset_refault_anon 238418 pswpin 234518 pswpout 697134
|
||||
15:40:50 full avg10=6.05 avg60=3.47 avg300=1.84 total=10066148 anon 2031009792 file 2072170496 2081452032 workingset_refault_anon 247990 pswpin 243754 pswpout 701826
|
||||
15:40:52 full avg10=5.14 avg60=3.39 avg300=1.83 total=10075229 anon 2033516544 file 2061938688 2081267712 workingset_refault_anon 251428 pswpin 247066 pswpout 701871
|
||||
|
@@ -0,0 +1,27 @@
|
||||
15:38:59 full avg10=0.87 avg60=3.86 avg300=1.59 total=7220110 anon 2155155456 file 1933524992 1882132480 workingset_refault_anon 158542 pswpin 156128 pswpout 606649
|
||||
15:39:01 full avg10=0.71 avg60=3.74 avg300=1.58 total=7234341 anon 2176008192 file 1922453504 1882083328 workingset_refault_anon 159499 pswpin 157036 pswpout 606649
|
||||
15:39:03 full avg10=0.58 avg60=3.61 avg300=1.56 total=7266319 anon 2173059072 file 1928216576 1882087424 workingset_refault_anon 159601 pswpin 157135 pswpout 607242
|
||||
15:39:05 full avg10=0.65 avg60=3.53 avg300=1.56 total=7287150 anon 2167078912 file 1926488064 1882349568 workingset_refault_anon 159703 pswpin 157201 pswpout 607982
|
||||
15:39:07 full avg10=0.72 avg60=3.44 avg300=1.56 total=7336362 anon 2170904576 file 1930149888 1882324992 workingset_refault_anon 160569 pswpin 158047 pswpout 607992
|
||||
15:39:09 full avg10=0.77 avg60=3.36 avg300=1.55 total=7356199 anon 2175414272 file 1923297280 1882296320 workingset_refault_anon 160637 pswpin 158104 pswpout 607992
|
||||
15:39:11 full avg10=0.81 avg60=3.29 avg300=1.55 total=7372570 anon 2172841984 file 1903226880 1882152960 workingset_refault_anon 161145 pswpin 158560 pswpout 607993
|
||||
15:39:13 full avg10=0.66 avg60=3.18 avg300=1.54 total=7381638 anon 2172796928 file 1903489024 1882152960 workingset_refault_anon 161160 pswpin 158560 pswpout 607993
|
||||
15:39:15 full avg10=0.54 avg60=3.07 avg300=1.53 total=7417217 anon 2178797568 file 1916338176 1882107904 workingset_refault_anon 162494 pswpin 159862 pswpout 607993
|
||||
15:39:17 full avg10=0.62 avg60=3.01 avg300=1.52 total=7429109 anon 2178826240 file 1916338176 1882099712 workingset_refault_anon 162515 pswpin 159868 pswpout 607993
|
||||
15:39:19 full avg10=0.51 avg60=2.91 avg300=1.51 total=7442053 anon 2179309568 file 1909116928 1882083328 workingset_refault_anon 162585 pswpin 159919 pswpout 607993
|
||||
15:39:21 full avg10=0.42 avg60=2.81 avg300=1.50 total=7450768 anon 2179932160 file 1905176576 1882066944 workingset_refault_anon 162682 pswpin 159995 pswpout 607993
|
||||
15:39:23 full avg10=0.34 avg60=2.72 avg300=1.49 total=7459779 anon 2180591616 file 1905315840 1882066944 workingset_refault_anon 162866 pswpin 160179 pswpout 607993
|
||||
15:39:25 full avg10=0.28 avg60=2.63 avg300=1.48 total=7474379 anon 2190475264 file 1896730624 1882017792 workingset_refault_anon 163514 pswpin 160816 pswpout 608009
|
||||
15:39:27 full avg10=0.23 avg60=2.54 avg300=1.47 total=7483721 anon 2183577600 file 1897766912 1881993216 workingset_refault_anon 163591 pswpin 160890 pswpout 608009
|
||||
15:39:29 full avg10=0.18 avg60=2.46 avg300=1.46 total=7490992 anon 2183839744 file 1897914368 1881993216 workingset_refault_anon 163657 pswpin 160955 pswpout 608009
|
||||
15:39:31 full avg10=0.15 avg60=2.38 avg300=1.45 total=7498777 anon 2184044544 file 1898643456 1881985024 workingset_refault_anon 163757 pswpin 161032 pswpout 608009
|
||||
15:39:33 full avg10=0.12 avg60=2.30 avg300=1.44 total=7515110 anon 2187968512 file 1898991616 1881899008 workingset_refault_anon 164702 pswpin 161960 pswpout 608009
|
||||
15:39:35 full avg10=0.10 avg60=2.22 avg300=1.43 total=7520357 anon 2187526144 file 1899126784 1881862144 workingset_refault_anon 164780 pswpin 162024 pswpout 608009
|
||||
15:39:37 full avg10=0.08 avg60=2.15 avg300=1.42 total=7533498 anon 2188107776 file 1900191744 1881817088 workingset_refault_anon 164899 pswpin 162123 pswpout 608009
|
||||
15:39:39 full avg10=0.06 avg60=2.08 avg300=1.41 total=7559069 anon 2189148160 file 1897832448 1881800704 workingset_refault_anon 164999 pswpin 162209 pswpout 608014
|
||||
15:39:41 full avg10=0.23 avg60=2.05 avg300=1.41 total=7565482 anon 2188591104 file 1880342528 1881784320 workingset_refault_anon 165052 pswpin 162256 pswpout 608036
|
||||
15:39:43 full avg10=0.19 avg60=1.98 avg300=1.40 total=7583622 anon 2188668928 file 1880240128 1881767936 workingset_refault_anon 165126 pswpin 162316 pswpout 608036
|
||||
15:39:45 full avg10=0.15 avg60=1.91 avg300=1.39 total=7587346 anon 2189262848 file 1878487040 1881767936 workingset_refault_anon 165233 pswpin 162414 pswpout 608036
|
||||
15:39:47 full avg10=0.12 avg60=1.85 avg300=1.38 total=7594985 anon 2181861376 file 1875628032 1881759744 workingset_refault_anon 165254 pswpin 162421 pswpout 608036
|
||||
15:39:49 full avg10=0.10 avg60=1.79 avg300=1.37 total=7599902 anon 2183262208 file 1658728448 1881718784 workingset_refault_anon 165522 pswpin 162621 pswpout 608036
|
||||
15:39:51 full avg10=0.08 avg60=1.73 avg300=1.36 total=7599902 anon 2173874176 file 1727365120 1881690112 workingset_refault_anon 165594 pswpin 162659 pswpout 608036
|
||||
|
@@ -0,0 +1,19 @@
|
||||
15:41:12 full avg10=0.79 avg60=2.46 avg300=1.71 total=10149379 anon 2095677440 file 1990664192 2079645696 workingset_refault_anon 269199 pswpin 264419 pswpout 701950
|
||||
15:41:14 full avg10=0.83 avg60=2.41 avg300=1.71 total=10177124 anon 2121056256 file 1971712000 2079608832 workingset_refault_anon 271376 pswpin 266520 pswpout 701964
|
||||
15:41:16 full avg10=1.04 avg60=2.39 avg300=1.71 total=10226245 anon 2127192064 file 1965178880 2079539200 workingset_refault_anon 272795 pswpin 267881 pswpout 701969
|
||||
15:41:18 full avg10=0.85 avg60=2.32 avg300=1.70 total=10245591 anon 2128060416 file 1959108608 2079531008 workingset_refault_anon 272851 pswpin 267921 pswpout 701969
|
||||
15:41:20 full avg10=0.69 avg60=2.24 avg300=1.69 total=10264675 anon 2129645568 file 1951543296 2079518720 workingset_refault_anon 273146 pswpin 268182 pswpout 701969
|
||||
15:41:22 full avg10=0.57 avg60=2.17 avg300=1.67 total=10275758 anon 2130022400 file 1952010240 2079510528 workingset_refault_anon 273203 pswpin 268225 pswpout 701969
|
||||
15:41:24 full avg10=0.65 avg60=2.13 avg300=1.67 total=10304890 anon 2131419136 file 1958985728 2079481856 workingset_refault_anon 273444 pswpin 268422 pswpout 701970
|
||||
15:41:26 full avg10=0.71 avg60=2.09 avg300=1.66 total=10326741 anon 2135916544 file 1949175808 2079481856 workingset_refault_anon 274703 pswpin 269668 pswpout 701970
|
||||
15:41:28 full avg10=0.58 avg60=2.02 avg300=1.65 total=10346752 anon 2136711168 file 1938386944 2079481856 workingset_refault_anon 274779 pswpin 269737 pswpout 701970
|
||||
15:41:30 full avg10=0.47 avg60=1.95 avg300=1.64 total=10355602 anon 2136711168 file 1938386944 2079481856 workingset_refault_anon 274779 pswpin 269737 pswpout 701970
|
||||
15:41:32 full avg10=0.57 avg60=1.92 avg300=1.64 total=10376110 anon 2138894336 file 1950375936 2079481856 workingset_refault_anon 275388 pswpin 270320 pswpout 701970
|
||||
15:41:34 full avg10=0.46 avg60=1.86 avg300=1.63 total=10386198 anon 2143576064 file 1940713472 2079449088 workingset_refault_anon 276447 pswpin 271329 pswpout 701970
|
||||
15:41:36 full avg10=0.56 avg60=1.83 avg300=1.62 total=10409920 anon 2142969856 file 1941934080 2079395840 workingset_refault_anon 276569 pswpin 271431 pswpout 701970
|
||||
15:41:38 full avg10=0.64 avg60=1.80 avg300=1.62 total=10428296 anon 2143977472 file 1943191552 2079395840 workingset_refault_anon 276840 pswpin 271678 pswpout 701970
|
||||
15:41:40 full avg10=0.52 avg60=1.74 avg300=1.61 total=10438990 anon 2144059392 file 1937944576 2079395840 workingset_refault_anon 276858 pswpin 271695 pswpout 701970
|
||||
15:41:42 full avg10=0.43 avg60=1.69 avg300=1.59 total=10449658 anon 2143178752 file 1937944576 2079342592 workingset_refault_anon 276860 pswpin 271697 pswpout 701970
|
||||
15:41:44 full avg10=0.35 avg60=1.63 avg300=1.58 total=10462674 anon 2143178752 file 1936896000 2079309824 workingset_refault_anon 276860 pswpin 271697 pswpout 701970
|
||||
15:41:46 full avg10=0.28 avg60=1.58 avg300=1.57 total=10478707 anon 2126831616 file 1685921792 2079297536 workingset_refault_anon 277148 pswpin 271927 pswpout 701970
|
||||
15:41:48 full avg10=0.23 avg60=1.52 avg300=1.56 total=10478707 anon 2127224832 file 1685921792 2079289344 workingset_refault_anon 277152 pswpin 271931 pswpout 701970
|
||||
|
@@ -0,0 +1,9 @@
|
||||
2026/10/22 15:53:05 scheduler.go:150: [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-10-23 03:30 CEST
|
||||
2026/10/22 15:53:05 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="tier2-backup" schedule="03:30" nextRun=2026-10-23T03:30:00+02:00 totalJobs=14
|
||||
2026/10/22 15:53:05 scheduler.go:85: [DEBUG] [scheduler] daily job tier2-backup: next run at 2026-10-23 03:30:00 CEST (waiting 9h36m54s)
|
||||
2026/10/22 15:53:46 auth.go:144: [DEBUG] [web] auth: valid session for POST /api/backup/tier2
|
||||
2026/10/22 15:53:46 router.go:78: [DEBUG] [api] POST /api/backup/tier2 (path=/backup/tier2)
|
||||
2026/10/22 15:53:51 tier2.go:491: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (516.4 MB, 2 leg(s), 5s)
|
||||
2026/10/22 15:53:53 tier2.go:491: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (127.4 MB, 1 leg(s), 1s)
|
||||
2026/10/22 15:53:53 tier2.go:491: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 0s)
|
||||
2026/10/22 15:53:53 tier2.go:553: [INFO] [backup] Tier 2 run complete: 3 app(s) processed (incl. volume-only — F6)
|
||||
@@ -0,0 +1,3 @@
|
||||
R-1214 live tier2 proof, 9202, controller 0.340.0-rc.4, 2026-10-22 ~18:00 CEST
|
||||
rsync -ani --delete source vs tier2 copy (run inside felhom-controller; the guest has no rsync): differences: 0
|
||||
files: source 44, copy 44
|
||||
@@ -0,0 +1,62 @@
|
||||
15:53:45 full avg10=0.00 avg60=0.00 avg300=0.10 total=10551160 2
|
||||
15:53:49 full avg10=0.00 avg60=0.00 avg300=0.10 total=10563819 2
|
||||
15:53:52 full avg10=0.00 avg60=0.00 avg300=0.10 total=10565444 1
|
||||
15:53:55 full avg10=0.00 avg60=0.00 avg300=0.10 total=10565444 1
|
||||
15:53:58 full avg10=0.00 avg60=0.00 avg300=0.10 total=10565444 1
|
||||
15:54:01 full avg10=0.00 avg60=0.00 avg300=0.10 total=10565444 1
|
||||
15:54:04 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:07 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:10 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:13 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:16 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:19 full avg10=0.00 avg60=0.00 avg300=0.09 total=10565444 1
|
||||
15:54:22 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565444 1
|
||||
15:54:25 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565444 1
|
||||
15:54:28 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565549 1
|
||||
15:54:31 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565549 1
|
||||
15:54:34 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565549 1
|
||||
15:54:37 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565549 1
|
||||
15:54:40 full avg10=0.00 avg60=0.00 avg300=0.08 total=10565549 1
|
||||
15:54:43 full avg10=0.00 avg60=0.00 avg300=0.07 total=10565549 1
|
||||
15:54:46 full avg10=0.00 avg60=0.00 avg300=0.07 total=10565549 1
|
||||
15:54:49 full avg10=0.00 avg60=0.00 avg300=0.07 total=10565549 1
|
||||
15:54:52 full avg10=0.00 avg60=0.00 avg300=0.07 total=10565549 1
|
||||
15:54:55 full avg10=0.00 avg60=0.00 avg300=0.07 total=10565549 1
|
||||
15:54:58 full avg10=0.00 avg60=0.00 avg300=0.07 total=10567212 1
|
||||
15:55:01 full avg10=0.00 avg60=0.00 avg300=0.07 total=10567212 1
|
||||
15:55:04 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:07 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:10 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:13 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:16 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:19 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:22 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:25 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:28 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:31 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:34 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:37 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567212 1
|
||||
15:55:40 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567241 1
|
||||
15:55:43 full avg10=0.00 avg60=0.00 avg300=0.06 total=10567241 1
|
||||
15:55:46 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:55:49 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:55:52 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:55:55 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:55:58 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:01 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:04 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:07 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:10 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:13 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:16 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:19 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:22 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:25 full avg10=0.00 avg60=0.00 avg300=0.05 total=10567241 1
|
||||
15:56:28 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:31 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:34 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:37 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:40 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:43 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:46 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
15:56:49 full avg10=0.00 avg60=0.00 avg300=0.04 total=10567241 1
|
||||
|
@@ -0,0 +1,9 @@
|
||||
#!/bin/bash
|
||||
# r1214run.sh LABEL CMD... — runs on demo-hp: samples 9202's cgroup every 2 s while CMD runs (in 9202), prints a summary
|
||||
L=$1; shift; CG=/sys/fs/cgroup/lxc/9202; OUT=/tmp/r1214-$L.tsv; : > $OUT
|
||||
( while :; do echo -e "$(date -u +%T)\t$(sed -n 2p $CG/memory.pressure)\t$(grep -E '^(anon|file) ' $CG/memory.stat | tr '\n' ' ')\t$(cat $CG/memory.swap.current)\t$(grep -E '^(pswpin|pswpout|workingset_refault_anon) ' $CG/memory.stat | tr '\n' ' ')" >> $OUT; sleep 2; done ) &
|
||||
SP=$!; F0=$(sed -n 2p $CG/memory.pressure | grep -o 'total=[0-9]*' | cut -d= -f2); T0=$(date +%s)
|
||||
LC_ALL=C pct exec 9202 -- "$@"; RC=$?
|
||||
T1=$(date +%s); F1=$(sed -n 2p $CG/memory.pressure | grep -o 'total=[0-9]*' | cut -d= -f2); sleep 4; kill $SP
|
||||
PK=$(cut -f2 $OUT | grep -o 'avg10=[0-9.]*' | cut -d= -f2 | sort -n | tail -1)
|
||||
echo "$L: rc=$RC seconds=$((T1-T0)) full-stall-total=$(( (F1-F0)/1000 )) ms peak full avg10=$PK % samples=$(wc -l < $OUT)"
|
||||
@@ -26,6 +26,14 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-22 night — the seven decisions put into effect (full text of the rows: `git show 1100a344:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-206** | **The build-cache cap and the weekly prune exist only as a hand-edited `/etc/docker/daemon.json` on DooPlex — not in Ansible, so a rebuild loses them.** (P4) | CLOSED 2026-10-22 night — D8 ruled (operator 15:47: DooPlex Docker settings from git plus a weekly build-cache prune) and installed: homelab-manifests `b35ba83`, role `docker_daemon` with a `docker-buildcache-prune` service + timer (Sun 23:30, `docker builder prune --force --filter until=168h`, never `-a`, with `docker system df` before and after). The live `daemon.json` was identical, so no Docker restart (dockerd up since 2026-08-05); timer next run 2026-10-25 23:30. Part (b) — loosening the housekeeping ban — was not needed: the prune lives in its own role and unit. | homelab-manifests `b35ba83` CHANGELOG |
|
||||
| **R-992** | **The drill-catalog route that `09` §6.5 documents cannot be followed: `admin/app-catalog-drill` is PRIVATE and no box has catalog credentials.** (P3) | CLOSED 2026-10-22 night — D6 ruled (operator 15:47: a read-only login for the scratch box's drill catalog). Gitea user `drill-reader` (not admin), read collaborator on `admin/app-catalog-drill`, token scope `read:repository`, kept in `sec`. Proven: the token reads the drill repo (200); write 403; push 403; the other private repos 404; 9202 cloned the drill catalog with it and returned to the live catalog with the credential kept. `09` §6.5 updated. | `audits/night-2026-10-22/P2-D6/01-token-limits.txt` |
|
||||
| **R-1207** | **A host with no (or too little) swap gets nothing from the guest swap fix (R-1192), and the agent never adds host swap.** (P2) | CLOSED 2026-10-22 night — DECIDED, D27 = report only (operator 15:47): the agent's WARN, the hub's `host_swap_short` (D28 = A) and the install-page sentence tell; the agent adds no swap and the installer refuses nobody. Reopens at the first BYO box with too little host swap. | `audits/night-2026-10-21/P2-noswap/` |
|
||||
|
||||
## 2026-10-22 day — the mail list and the swap rule (full text: R-1216 `git show 25dae2f5:documentation/backlog/OPEN-ITEMS.md`; R-1199's original row `git show 115f15c8:documentation/backlog/OPEN-ITEMS.md` — its reopened text of 2026-10-22 was never committed, and its substance is the Closed cell below)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -7,6 +7,20 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
## Moved from STATUS on 2026-10-22
|
||||
|
||||
**Updated 2026-10-22 (day).** A rebuilt box no longer overwrites the family's saved mail list (hub 0.165.0, controller 0.339.0); demo-felhom's own list, replaced this morning, is back. The swap warning now fires only when a machine has clearly too little swap (your D28 = A).
|
||||
|
||||
**Decided 2026-10-22 15:47 — „all picks”** (the operator; each ruling is in its row, and in CONTEXT.md): D3 = change the wording now (R-985); D4 = root SSH from the LAN by key only, the console password stays (R-982); D5 = approve the minimum machine (R-991); D6 = a read-only login for the scratch box's drill catalog (R-992, closed); D8 = DooPlex Docker settings from git plus a weekly build-cache prune (R-206, closed); D21 = B (R-49); D27 = report only (R-1207, closed). The rows as they stood:
|
||||
|
||||
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||
|---|---|---|---|---|---|---|
|
||||
| D27 | R-1207 | A host with no swap gets nothing from the swap fix, and the agent never adds swap. Add host swap automatically (BYO too), or refuse an 8 GB host without swap at install? | **Report only for now** (our installer gives 7.75 GB, and Tester 2's laptop used our installer; the alarm and the install-page sentence now tell you) — decide after the first BYO box | Add it: the agent changes a disk its owner set up; refuse: a BYO household cannot start | a no-swap host can still freeze; you get the alarm | — |
|
||||
| D21 | R-49 | Immich's off-site copy: leave out only the AI model files now (B), or also the raw database folder and Immich's own backups (A)? | **B now** (−62 %, no new risk); A's other parts after their proof | A: −92 %, but it needs a restore path we have never run | every Immich box sends about 12 times its real data off-site each night | — |
|
||||
| D8 | R-206 | Install DooPlex's Docker settings from git (Docker restarts once), with a weekly build-cache prune? | **Yes, in a quiet window** (a rebuilt DooPlex then keeps the settings) | No: the live settings stay hand-made, and a rebuild loses them | git and the host can drift apart | — |
|
||||
| D6 | R-992 | The scratch box cannot read the private drill catalog: give it a read-only login, or make that copy public? | **A read-only login** (the drill catalog stays private) | Public copy: anyone can read the drill catalog | every new-app test needs staging by hand | new-app tests |
|
||||
| D5 | R-991 | Approve the minimum machine: x86-64, 4 cores, 8 GB RAM, 256 GB SSD (recommended: 16 GB, 512 GB)? | **Approve** (measured on the N100 with 61 apps) — the 8 GB freeze of 2026-10-20 is fixed by agent 0.162.0 (11 h soak on a fresh 8 GB box, no freeze) | Change it: one edit | no hardware page; a buyer with a Raspberry Pi can still download the installer | the hardware page |
|
||||
| D4 | R-982 | Stop root SSH with a password from the household network (key only)? | **Yes — key only** (the password stays for the console) | Keep it: anyone on the household network can try the password | the host keeps accepting root password logins from the LAN | — |
|
||||
| D3 | R-985 | Our „zero-knowledge" wording is stronger than the product: change the wording, or remove the operator's root reach to the key? | **Change the wording now**; look again after the first sale | Remove the reach: a design change to break-glass | the website promises more than the product does | the website text |
|
||||
|
||||
**Updated 2026-10-22 (morning, your yeses).** Released: controller 0.338.0 (global floor), agent 0.162.0 (on the 3 test boxes), hub 0.164.0, installer 1.30.0 and the website download button. The new host-swap check warns on boxes bigger than 8 GB; that is too strict (decide D28).
|
||||
|
||||
**Decided 2026-10-22 11:42 — D28 = A** (the operator): | D28 | R-1216 | The host-swap check warns when host swap is below the guests' whole swap. A 16 GB box with the installer's 8 GB swap trips it, but the soak used 2.7 GB. Loosen it to „below what the installer gives"? | **Loosen it (A)** (no mail for a healthy box; 938 MB still warns) | Keep (B): you get a mail per big box, and the install page tells such a household to call us | demo-hp and demo-felhom stay flagged; one mail each, already sent | — |
|
||||
|
||||
Reference in New Issue
Block a user