night 2026-10-21: soak result (no freeze in 11 h; memory pressure above 20 % for 3 min in the tier2 job, R-1214; 1 Immich upload lost, R-1215), no-swap test (the freeze comes back without host swap, R-1207), teardown, the night report
gates / gates (push) Successful in 16m4s
gates / gates (push) Successful in 16m4s
Rows 182 -> 184. STATUS morning list item 2 carries the soak result; CONTEXT the two results.
This commit is contained in:
+4
-1
@@ -29,7 +29,10 @@
|
||||
> lacked disk warning, drive disconnected and second-drive failure, so the household got 1 alarm of 9. Built tonight:
|
||||
> controller `0.338.0-rc.2` (`sha256:ae52c93e…`, all of tonight's controller work, smoke-tested on 9202, 9202 back on
|
||||
> 0.337.0), hub `0.164.0` (`sha256:931594e3…`, manifest still 0.163.0), agent candidate `0.162.0` from `0da792e`
|
||||
> (sha256 `5fd7a23b…`, not published). The 8 GB soak with the swap fix: `audits/night-2026-10-21/P1-soak/`.
|
||||
> (sha256 `5fd7a23b…`, not published). The 8 GB soak with the swap fix (`audits/night-2026-10-21/P1-soak/`): 11 h, no
|
||||
> freeze, 0 failed health probes (largest interval 16 s), every night job ran; memory „full" above 20 % for ~3 min in the
|
||||
> 03:30 tier2 job (R-1214); 1 photo of ~26 000 not stored (R-1215). With `swapoff -a` on the host the freeze came back
|
||||
> within 20 s (dead up to 289 s; `P2-noswap/`) — the fix depends on host swap (R-1207).
|
||||
|
||||
> **2026-10-21 (day brief, before Tester 2) — hub 0.163.0 DEPLOYED (operator yes, 11:46 UTC); controller 0.337.0 released
|
||||
> (three test boxes 14:06 UTC, global floor after the freeze proof); agent `286a7e3` committed, NOT released.** Operator rulings
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
# REPORT — the night of 2026-10-21 → 22: make tomorrow's release safe
|
||||
|
||||
Unattended, 20:15 CEST → 08:00. Nothing released, deployed, published or vouched. Evidence:
|
||||
`documentation/audits/night-2026-10-21/`. Design read first: `07` §6.3/§8 (the restore ladder, R-893 option A),
|
||||
`03` §9 (guest sizing), `08` (notification groups), `10-localisation.md`, the controller CHANGELOG v0.336.0.
|
||||
|
||||
**Safest readings taken at the start (unattended):** (1) Tester4 is the soak customer — a new customer would provision on
|
||||
ep0 and the Storage Box, which are read-only tonight; its old host was already deleted. (2) „One hub image" — so the
|
||||
`node_down`-after-delete finding is a row, not a second image. (3) The website is published by a push, so W2 is a patch
|
||||
file, not a commit. (4) The no-swap test ran after the box's last night job, not at 07:00 sharp (see Part 2).
|
||||
|
||||
## 1. Five lines
|
||||
|
||||
1. **Soak (Part 1): NO FREEZE — 3 of the 4 pass lines met, 1 missed.** 11 h on a fresh 8 GB box: no `/api/health` gap (largest interval 16 s), every night job ran, uploads complete except 1 photo of ~26 000 (R-1215); memory „full" pressure passed 20 % for ~3 min in the 03:30 second-drive job (peak 39 % host / 45 % guest over 10 s — R-1214).
|
||||
2. **No-swap test (Part 2): THE FREEZE CAME BACK.** Host swap off, the same burst: the dashboard stopped answering within 20 s, dead up to 289 s at a time (24 of 125 probes failed), Immich lost 207 of 306 photos; it ended only when the Immich stream broke. Swap back on: normal at once. The agent warned once.
|
||||
3. **D22 built and proven: YES** — the put-back covers the local, second-drive and kept restores and a restore cut by a
|
||||
power loss; proven on 9202 (a damaged backup and a killed controller both kept the newest row and the app running).
|
||||
4. **First-hour words: 22 fixed** (controller W9–W14, W21, W22, W24, W25, W28, W29; hub W3–W5; catalog W16–W19; ISO
|
||||
source W1, W7); W6/W8 were already true; W2 is a ready patch (a publish).
|
||||
5. **R-1199:** the hub mailed the household by a list saved two days before the box existed; the box never sent its own
|
||||
— fixed in the controller (decided by CC, may be reversed). **R-1194:** a timing false alarm — the first report runs
|
||||
before the emergency-door setup; fixed in the agent.
|
||||
|
||||
## 2. The morning list (copied from STATUS)
|
||||
|
||||
**Morning list, 2026-10-22.** CC decided one thing (you may reverse it): a box tells the hub the notification list its page shows, even if never saved (R-1199; the rehearsal household got 1 alarm of 9).
|
||||
1. Release controller 0.338.0 (restore put-back incl. power cut, first-hour words, swap sentence)? On the scratch box a damaged backup and a killed controller both kept the newest data, app running (R-1177, R-1195, R-1178, R-1180).
|
||||
2. Release and vouch agent 0.162.0 (swap = RAM, host-swap check, no false door alarm — R-1192, R-1194)? Soak, 11 h on a fresh 8 GB box: no freeze, every check answered, every night job ran; memory pressure passed the 20 % line for 3 min in the 03:30 backup (R-1214). With no host swap the freeze comes back; our installer gives 7.75 GB. I'd release.
|
||||
3. Deploy hub 0.164.0 (host-swap alarm, bind-page words)? And delete the test host „Tester4-9aca27" (machine gone; my token cannot delete).
|
||||
4. How was Tester 2's laptop installed — our installer, or its own Proxmox? (Without host swap the fix does not help.)
|
||||
5. A publish: the website download button (patch ready), and with the next ISO the boot text and the cleared screen (R-1210). Secrets: keep or remove the `REHEARSAL_2026_10_20_*`, `FIXES_2026_10_21_*`, `NIGHT_2026_10_21_*` names in sec.
|
||||
|
||||
## 3. Part 1 — the soak (numbers)
|
||||
|
||||
Box: VM 9322 on demo-hp (4 vCPU, 8 GiB, 256 GB + 200 GB), installed from the public ISO, customer Tester4; agent fixture
|
||||
`0.162.0-night.1` then `night.2` by hand (provenance `P1-soak/00-provenance.txt`). Two channels: the box's own 10-s
|
||||
recorder (`03-rec.jsonl`, 3 632 samples) and a probe from DooPlex (`02-outside.txt`, 3 390 probes + demo-hp's view of the VM's
|
||||
disk reads).
|
||||
|
||||
| | Load window 18:52–20:53 UTC | Night 20:53–04:05 UTC |
|
||||
|---|---|---|
|
||||
| health probes / failed | 684 / 0 | 2 449 / 0 |
|
||||
| worst health gap (between answers) | 10–11 s (slowest answer 1.27 s) | **16 s** at 01:30:36 (slowest answer 4.07 s) |
|
||||
| worst memory „full" (10 s, host / guest) | 31.2 % / 35.2 % (18:53:38, Immich model load; 1 sample) | **39.4 % / 45.5 %** (01:30:42, tier2 job; 12 samples > 20 %, ~3 min) |
|
||||
| worst memory „full" (60 s) | — | 20.1 % / 25.0 % |
|
||||
| OOM kills / guest limit hits | 0 / 0 | 0 / 0 |
|
||||
| guest swap used (max) | 2 614 MB | 2 720 MB |
|
||||
|
||||
- **Guest swap raised:** 512 → 4096 MB by the fixture at 18:37:13 UTC (`04-agent-fixture.txt`).
|
||||
- **Uploads:** Paperless 99 cycles × 53 PDFs, all accepted; Immich 87 cycles × 305 + the first upload — every user holds
|
||||
303 photos (the set has 2 duplicates and 1 file Immich refuses by design), except cycle 36: 302 — **one photo hung 10 min
|
||||
and was not stored** (19:38 UTC, pressure 3.9 %, health 0.26 s; Immich's log lost at its 02:16 restart — R-1215);
|
||||
Nextcloud: both trees 3 031 of 3 033 files byte-identical to the manifest (sha256; the 2 are the uploader's by-design skips).
|
||||
- **Night jobs (box log `10-controller-log.txt`):** db-dump 00:30 (4 DBs, 4 m 14 s), tier2 01:30 (4 m 15 s), off-site
|
||||
02:15 (8 apps, 72 snapshots, 5 m 53 s), abandon-sweep 03:10, proof 03:30 (actualbudget PASSED), integrity 04:00
|
||||
(1 m 22 s) — all completed. The agent's whole-guest backup ran at 18:41 UTC (24 h cadence; next due after the night).
|
||||
- **Host swap the installer gives an 8 GB / 256 GB machine: 7.75 GB** (LVM `pve/swap`, SwapTotal 8 126 460 kB;
|
||||
`02-host-swap-after-install.txt`).
|
||||
- **Before (rehearsal, no fix):** „full" 96 %, dashboard dead for about an hour. **After:** no gap over 16 s.
|
||||
|
||||
## 4. Part 3 — D22, each scenario
|
||||
|
||||
| Scenario (9202, Rallly + Postgres, controller 0.338.0-rc.1) | Result |
|
||||
|---|---|
|
||||
| a normal local restore (the control) | unchanged: „1 adatkötet és az adatbázis visszaállítva", the expected rewind (A,B → A) |
|
||||
| the unit's database volume archive damaged (cut to half) | put back: the live volume poured back, markers **A,C** (C = the newest live row), Rallly running, „…visszaállítása sikertelen. Nem változott semmi…" — on 0.331.0 (2026-10-20) the same case crash-looped Postgres and lost C |
|
||||
| the controller killed (kill -9) right after the volume step, before the replay | on start: „interrupted restore is put back … 1 volume(s) poured back, definition=true", markers **A,C,D** (D newer than the backup), Rallly running, the record: „A visszaállítás félbeszakadt (áramszünet). Az alkalmazás adatai a visszaállítás előtti állapotban vannak." |
|
||||
| off-site restore cut the same way | by test (`TestR1195_AKilledOffsiteRestoreIsPutBackOnStart`) |
|
||||
| second drive: the unit refuses after the file half (R-1178) | by test: nothing moves, the app is never stopped |
|
||||
| the night mirror during a restore (R-1180) | by test with the production rsync: skipped, mirrored the next night |
|
||||
|
||||
**Red-proofs seen failing, then passing:** `TestR1177_*` (4 failed on the old `restore_unit.go`: the old „completed with
|
||||
data errors" text, no copy), `TestR1195_*` (the put-back at start switched off → „nothing poured back"; the off-site
|
||||
journal not written → no undo), `TestR1195_TheRecordSays…` (old „indítsd el újra"), `TestR1178_*` (1 restored +
|
||||
1 replaced), `TestR1180_*` ([a.jpg b.jpg]). Three older tests that pinned the old outcome were changed on purpose
|
||||
(the app started on a half state is now held or put back). 9202 put back as found: controller 0.337.0, the same apps.
|
||||
|
||||
## 5. Part 2 — machines with no swap
|
||||
|
||||
**The test** (`P2-noswap/`): the soak box after its night, `swapoff -a` on the host at 04:06 UTC (48 s; it held 1.9 GB),
|
||||
the burst from 04:08:25 to 04:38:27 (Immich machine-learning restarted so its models load again, as on the first afternoon),
|
||||
then `swapon -a`. The first attempt (04:06–04:08) is VOID and kept: the load script restarted its names at 1, so Immich got
|
||||
duplicates and Nextcloud skipped existing folders (`02-attempt1-VOID-duplicate-uploads.txt`); the rerun used fresh names.
|
||||
|
||||
| | No host swap (04:08–04:38) | Swap back on (04:38–04:44) |
|
||||
|---|---|---|
|
||||
| health probes / failed | 125 / **24** | 34 / 0 |
|
||||
| gaps between answers | **289 s**, 172 s, 68 s | none |
|
||||
| memory „full" (10 s, host / guest) | **36 % / 47 %** (94 of 180 samples > 20 %) | 1.2 % / 1.3 % |
|
||||
| VM disk reads (demo-hp's view) | ~40 GB per 25 s at the worst | quiet |
|
||||
| OOM kills | 0 | 0 |
|
||||
| uploads | Immich 99 of 306, then its stream broke (a login answer that was not JSON); Paperless 52 of 53 in the first cycle, then 22 × 53; Nextcloud cut at the time limit | — |
|
||||
|
||||
**Said plainly:** without host swap the freeze comes back — the same shape as the rehearsal (refaults, disk reads, no OOM
|
||||
kill), shorter only because the Immich stream died at 04:22 and took the load away. The agent logged its WARN once at the
|
||||
swapoff („host_swap_mb=938 guest_swap_mb=4096").
|
||||
|
||||
**Built (commit only):** agent `0da792e` reports `host.host_swap_mb` (Proxmox node status = the kernel's SwapTotal — the
|
||||
same number as `/proc/meminfo`; used instead of reading the file because the agent already holds the node status) and
|
||||
`host.guest_swap_mb`, logs one WARN when short; `GET /guest/memory` carries both. Hub 0.164.0: operator-only
|
||||
`host_swap_short` / `host_swap_ok`, edge-triggered from the host's newest swap event. Controller install page, when memory
|
||||
is tight and the host's swap is below the guest's RAM: **HU** „A gépen nincs elég cserehely (swap), ezért nagy
|
||||
feltöltéskor a doboz lelassulhat, és percekig sem válaszolhat. Szólj a Felhomnak, mi rendbe tesszük." **EN** „The machine
|
||||
has too little swap space, so during large uploads the box can slow down and not answer for minutes. Tell Felhom; we will
|
||||
fix it." Creating host swap on a BYO host is a decision → R-1207 (D27).
|
||||
|
||||
## 6. Part 5 — the two checks
|
||||
|
||||
**R-1199 (the household's mails went to the operator).** Hub DB read with `SELECT` on named non-secret columns only
|
||||
(`P5-checks/r1199-01-hub-db-read.txt`): Tester4's stored list was saved **2026-10-18 18:18** — before the rehearsal box
|
||||
existed — and holds `node_down, backup_failed, disk_critical, host_disk_critical, storage_fill_critical,
|
||||
offbox_repo_orphaned, app_update_undone, app_update_held, app_stopped_unhealthy`: no `disk_warning`, no
|
||||
`storage_disconnected`, no `crossdrive_failed`. The rehearsal day's log: the one household mail was `backup_failed`.
|
||||
**Mechanism:** the box shows `settings.DefaultEnabledEvents` ticked when nothing was saved
|
||||
(`controller/internal/settings/settings.go` `GetNotificationPrefs`), but the start-up sync pushes only when an address or
|
||||
a clear was saved (`settings.go` `SyncOnStartup`, `cmd/controller/main.go` start-up goroutine), and the hub mails the
|
||||
household only by its stored list (`hub/internal/notify/dispatcher.go` `processCustomer` → `isEventEnabled`). **Fix
|
||||
(controller `ba22f9b`):** a box with no saved choice pushes the default list at start, with no address (the hub's
|
||||
no-clobber guard keeps the stored one). The box's page is the source — `08` („the household decides"). *Decided by CC
|
||||
unattended — operator may reverse.* `TestR1199_*` red-proved.
|
||||
|
||||
**R-1194 (emergency SSH door „broken" on a new box).** On the fresh soak box: the agent ran `sshd -t -f
|
||||
/etc/felhom-sshd/sshd_config` **through sudo** at 20:33:59, the config was applied at 20:34:00.434, and the hub mailed
|
||||
`oob_degraded` („config invalid") at 20:34 CEST; as the agent's user the check passes now (rc 0, port 8822 listening);
|
||||
without sudo it fails on the host key — which the agent never does. **True candidate: the first report runs before the
|
||||
felhom-sshd loop's first pass** (port 0, config never rendered) — a false alarm on every fresh box, operator key or not.
|
||||
**Fix (agent `0da792e`):** no OOB stanza until the first pass has run (a failed pass counts). Red-proved.
|
||||
|
||||
## 7. Builds, commits, CI
|
||||
|
||||
| Artifact | Version | From | Digest / sha256 | State |
|
||||
|---|---|---|---|---|
|
||||
| controller | `0.338.0-rc.1` | `d582feb` | — | the D22 proof on 9202 |
|
||||
| controller | `0.338.0-rc.2` | `f0abdd0` | `sha256:ae52c93ebb83af1c646986119a1b74b07f992ffb03017624d052ece6266ec748` | smoke on 9202 (pages 200, 0 error lines); 9202 back on 0.337.0 |
|
||||
| hub | `0.164.0` | `6b0d23c2` | `sha256:931594e333a4e8a9784eff2fb144d022ab46f89dd6a865a93e6100dc28dbd9dc` | built, pushed, NOT deployed (manifest 0.163.0) |
|
||||
| agent | `0.162.0` candidate | `0da792e`, clean clone, release flags | `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` | not published; fixtures `night.1` (`286a7e3`) and `night.2` (`0da792e`) ran on the soak host |
|
||||
|
||||
**Commits.** controller: `d582feb`, `37b79ca`, `8c1e290`, `28b05d6`, `5776d54` (the helper's three: W9–W29), `ba22f9b`,
|
||||
`f0abdd0`, `1de8a55`. agent: `0da792e`, `b310773`. felhom.eu: `3c620f8d`, `f90e2229` (hub), `2fa5b8f3` (ISO source),
|
||||
`6b0d23c2`, `2293b252`, `9d3bf76d`, `57c06ba9`, `307d25b1`, `f576d7f7`, `c2c130ec`, `dee26da3` + this report.
|
||||
catalog: `13f9c9c` (live through the catalog sync). **CI:** controller 2175, 2178, 2181, 2184, 2186, 2187, 2188, 2191 —
|
||||
all success; agent 2176, 2190 — success; catalog 2180 — success; felhom.eu 2177–2192 **failed on one suite only**
|
||||
(`test_gate_decoys.py` hit the 300 s limit on the CI runner; R-1209, fixed in `c2c130ec`) — after the fix: `c2c130ec` 2193, `dee26da3` 2194, `97f088ec` 2195 — all success; the commit carrying this report is checked after its push (the final note)..
|
||||
|
||||
## 8. Teardown (three layers)
|
||||
|
||||
- **Machine:** VM 9322 `night1021-soak` and CT 9332 `night1021-loader` destroyed on demo-hp (names checked first) at
|
||||
~04:47 UTC; their disk folders on nvme-scratch are gone (used 301 → 208 GB). The thin pool `local-lvm` read the same before
|
||||
and after (26 159 563 KiB used) — never written. The public ISO on `local` was there before and stays. demo-hp's own clock
|
||||
and swap untouched. The night's evidence was copied off first (logs, recorder, Immich counts).
|
||||
- **Host (hub):** the host `Tester4-9aca27` stays on the hub (agent `0.162.0-night.2`); CC's token cannot delete a host —
|
||||
**the operator deletes it** (morning list 3). Until then it reads down. The customer Tester4 stays (the kept one).
|
||||
- **Off-site:** ep0 (read only) holds for Tester4 one copy, `ct/9201/2026-10-20T12:17:26Z`, from the rehearsal box —
|
||||
nothing from tonight; nothing removed. The Storage Box holds Tester4's off-site repository with tonight's 72 snapshots —
|
||||
left (a new Tester4 box sets it aside by design).
|
||||
- **9202:** back on controller 0.337.0 with the apps it had; the D22 test app removed through the product.
|
||||
|
||||
## 9. Rows
|
||||
|
||||
Before **176**; after **184**; opened **9** (R-1207 host swap on a no-swap host — D27; R-1208 `node_down` after an
|
||||
operator delete; R-1209 the CI time limit — closed the same night; R-1210 first-hour words waiting for a publish; R-1211
|
||||
the household `backup_failed` mail; R-1212 two texts ignoring the box language; R-1213 operator detail left on two pages; R-1214 memory pressure in the night's second-drive job; R-1215 one Immich upload hung);
|
||||
closed **1** (R-1209). Moved to WAITING-ON-OPERATOR (fixed on main, ship with the morning releases): R-1177, R-1178,
|
||||
R-1180, R-1195 (controller), R-1194 (agent). R-1188 (a) fixed. `unproven.py --summary`: unchanged, 35 of 55 not walked.
|
||||
|
||||
## 10. Observations
|
||||
|
||||
- The decoy suite timed out on CI on every felhom.eu push of the night — **FILED: R-1209** (fixed, closed).
|
||||
- `node_down` mailed for a customer whose host the operator had deleted — **FILED: R-1208**.
|
||||
- A household `backup_failed` mail with no app and no step — **FILED: R-1211** (the helper's).
|
||||
- Two backup texts ignore the box language — **FILED: R-1212**. Operator detail on two pages — **FILED: R-1213**.
|
||||
- `TestR603_*` was red on controller main since 0.337.0 (CI runs gates, not `go test`) — **NOT-A-FINDING:** fixed in
|
||||
the session (two English apostrophes), small.
|
||||
- The first `resend` press on a spent link before the new box registered sent nothing — **NOT-A-FINDING:** a live link
|
||||
existed (minted at the host delete, mailed 16:49); the page says „if it was a real link" by design.
|
||||
- The helper chose different words than the brief for W25 („a következő éjjel", the retry is the same night when pressed
|
||||
by day) and W29 („Leszerelve. Nem költöztettünk át adatot róla; ami rajta volt, a meghajtón maradt." — „adat nem volt
|
||||
rajta" would be false) — **NOT-A-FINDING:** the brief's texts were examples; the reasons are in the CHANGELOG.
|
||||
- On the installed app pages Mealie already showed the right address (R-498); W19 was wrong only on the install page —
|
||||
**NOT-A-FINDING** beyond the catalog fix.
|
||||
- An empty `backups/restore-undo/` folder stays after a put-back (only the per-app folder is removed) —
|
||||
**NOT-A-FINDING:** harmless, no data.
|
||||
- The soak load script's Nextcloud stream ran past the 2 hours (each stream finishes its current pass) —
|
||||
**NOT-A-FINDING:** harness.
|
||||
|
||||
- Memory „full" above 20 % in the night's second-drive job (~3 min) — **FILED: R-1214**. One Immich upload hung and was
|
||||
not stored — **FILED: R-1215**.
|
||||
- Without host swap the freeze comes back — **recorded on R-1207** (the operator's D27), not a new row.
|
||||
- One `execCommand failed: exit status 1` at 00:30:49 — **NOT-A-FINDING:** the memory check's `docker inspect` of all
|
||||
containers ran 1 s after actualbudget was stopped for its dump; the code keeps the other answers by design.
|
||||
- One Radicale health probe timed out at 01:30:24 (the second-drive job starting) — **NOT-A-FINDING:** the next probe passed,
|
||||
no alarm, the household's door answered throughout.
|
||||
- The guest hit its memory limit 101 times at 18:39 UTC, during the app installs, before the load — **NOT-A-FINDING:** page
|
||||
cache reclaim (0 OOM kills, swap 40 KB used), never again in 11 h.
|
||||
- A 637 MB `vzdump-lxc-9100-…` on the fresh box's `local` — **NOT-A-FINDING:** the installer's golden archive (9100 = the
|
||||
golden build guest, `felhom-host-install.sh`); the restore test skips it with a reason.
|
||||
- The byte-check tool `tools/nc_compare.py` turns TLS checking off — **NOT-A-FINDING:** a measurement script against a
|
||||
throwaway box with a self-signed certificate and a test domain that does not resolve; the box is destroyed.
|
||||
- The first no-swap attempt reused the soak's names (Immich duplicates, Nextcloud skips) — **NOT-A-FINDING:** harness;
|
||||
stopped after 2 min, kept as VOID, rerun with fresh names (`tools/load2.py` now takes a name tag).
|
||||
- CC ran an old one-off register script by mistake at 04:07 UTC; it added 16 duplicate rows to the register file —
|
||||
**NOT-A-FINDING:** caught at once by the row count, the file restored from git before anything was committed; the
|
||||
script renamed so it cannot run again.
|
||||
**Fixed without a row:** the R-603 test, the restore-record log line after a put-back, the CHANGELOG blank line.
|
||||
**Secrets:** names only — `NIGHT_2026_10_21_ROOTPW`, `NIGHT_2026_10_21_PAIRING_CODE`, `NIGHT_2026_10_21_BIND_TOKEN`,
|
||||
`NIGHT_2026_10_21_SETUP_CODE` (the last three spent). The bind link and the setup code were read from the mailbox through
|
||||
the mail connector, so both one-time values entered this session's record (spent within minutes). A scan of the evidence
|
||||
for every secret used tonight: 0 hits — all 34 secret names of the night, the rehearsal and the fixes run, across the 75 evidence files, this report, STATUS, CONTEXT and the register (positive control: a known word in the evidence was found). **Poster:** no fact in `felhom-system-poster.facts.md` changed (nothing
|
||||
released). **Instruction files:** none edited.
|
||||
@@ -1,8 +1,8 @@
|
||||
# STATUS — what works, what's broken, what's next
|
||||
|
||||
**Ready for the first real tester (Tester 2)? No — three morning yeses left:** release the controller (the restore put-back, the first-hour words), release and vouch the agent (the swap fix — the night soak says yes or no), deploy the hub. Tester 2 is off; nothing was sent to it.
|
||||
**Ready for the first real tester (Tester 2)? No — three morning yeses left:** release the controller (restore put-back, first-hour words), release and vouch the agent (the swap fix — the night soak says yes), deploy the hub. Tester 2 is off; nothing was sent to it.
|
||||
|
||||
**Updated 2026-10-22 (morning, after the night run).** A failed or cut restore now puts the household's data back and starts the app (proven on the scratch box); the first-hour texts are clearer; a host with too little swap is reported. All built and tested, nothing released.
|
||||
**Updated 2026-10-22 (morning, after the night run).** A failed or cut restore now puts the data back and starts the app (proven on the scratch box); clearer first-hour texts; a host short of swap is reported. Built and tested, nothing released.
|
||||
|
||||
## What works
|
||||
- A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
|
||||
@@ -15,7 +15,7 @@
|
||||
- The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15).
|
||||
|
||||
## What is broken
|
||||
- **P1:** the 8 GB box can freeze for minutes under big uploads until the agent with the swap fix is released (R-1192; the night soak result is in the morning list).
|
||||
- **P1:** the 8 GB box can freeze for minutes under big uploads until the agent with the swap fix is released (R-1192; the night soak says the fix works — morning list item 2).
|
||||
- **P1:** about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
|
||||
- Two root-level weaknesses wait on your decisions D3 and D4 (R-985, R-982).
|
||||
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862).
|
||||
@@ -23,7 +23,7 @@
|
||||
- The scratch box cannot use agent features (R-1083) or read the drill catalog (R-992).
|
||||
|
||||
## What we work on
|
||||
- Dated checks, run by CC. 2026-10-21 done: the hub's own off-site check found nothing wrong (R-895); downloads stay out of the second-drive copy (closed). 2026-10-23: no app stop when the host is busy (R-1173). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts).
|
||||
- Dated checks, run by CC. 2026-10-23: no app stop when the host is busy (R-1173). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts).
|
||||
|
||||
## Waiting on you — decide
|
||||
|
||||
@@ -46,21 +46,21 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
**Still needed? (yes/no)** — CC could not check these. Answer yes or no per line.
|
||||
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). Since 2026-10-16.
|
||||
2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources (`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). Since 2026-10-15.
|
||||
3. Look at these pages on demo-hp when you have time: Calibre-Web („Adatok és mentés"), the file manager, AdventureLog, Plex, the app page, the Plex and Sonarr guides („Útmutató"). Since 2026-10-16. Today's rehearsal screenshots: `documentation/audits/rehearsal-2026-10-20/screens/`.
|
||||
3. Look at these pages on demo-hp when you have time: Calibre-Web („Adatok és mentés"), the file manager, AdventureLog, Plex, the app page, the Plex and Sonarr guides („Útmutató"). Since 2026-10-16.
|
||||
4. Remove the test server „plex-k6" from your Plex account (plex.tv → Settings → Authorized Devices). Since 2026-10-15.
|
||||
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). Since 2026-10-14.
|
||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
|
||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
||||
|
||||
**Morning list, 2026-10-22.** CC decided one thing (you may reverse it): a box tells the hub the notification list its page shows, even if never saved (R-1199; the rehearsal household got 1 alarm of 9).
|
||||
1. Release controller 0.338.0 (restore put-back incl. power cut, first-hour words, swap sentence)? On the scratch box a damaged backup and a killed controller both kept the newest data and the app running (R-1177, R-1195, R-1178, R-1180).
|
||||
2. Release and vouch agent 0.162.0 (swap = RAM, host-swap check, no false door alarm — R-1192, R-1194)? The soak result is in the night report.
|
||||
3. Deploy hub 0.164.0 (host-swap alarm, bind-page words)?
|
||||
4. Tester 2's laptop: our installer or its own Proxmox? (Does it have host swap?)
|
||||
1. Release controller 0.338.0 (restore put-back incl. power cut, first-hour words, swap sentence)? On the scratch box a damaged backup and a killed controller both kept the newest data, app running (R-1177, R-1195, R-1178, R-1180).
|
||||
2. Release and vouch agent 0.162.0 (swap = RAM, host-swap check, no false door alarm — R-1192, R-1194)? Soak, 11 h on a fresh 8 GB box: no freeze, every check answered, every night job ran; memory pressure passed the 20 % line for 3 min in the 03:30 backup (R-1214). With no host swap the freeze comes back; our installer gives 7.75 GB. I'd release.
|
||||
3. Deploy hub 0.164.0 (host-swap alarm, bind-page words)? And delete the test host „Tester4-9aca27" (machine gone; my token cannot delete).
|
||||
4. How was Tester 2's laptop installed — our installer, or its own Proxmox? (Without host swap the fix does not help.)
|
||||
5. A publish: the website download button (patch ready), and with the next ISO the boot text and the cleared screen (R-1210). Secrets: keep or remove the `REHEARSAL_2026_10_20_*`, `FIXES_2026_10_21_*`, `NIGHT_2026_10_21_*` names in sec.
|
||||
**The rest**
|
||||
- Pull the failed disk Z6A07P2G out of DooPlex (R-934); it is still in. If not: its alarms return on 11 November. Since 2026-10-12.
|
||||
- Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12.
|
||||
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
|
||||
- Two leftovers on DooPlex: keep, commit or delete the Facebook `first-post/` folder in `/mnt/5_hdd/felhom.eu/leftovers/2026-10-14/`; commit or drop the `deny`-list edit in `~/git/homelab-manifests/.claude/settings.json`. If not: they stay. Since 2026-10-14.
|
||||
- Two leftovers on DooPlex: the Facebook `first-post/` folder in `leftovers/2026-10-14/` (keep, commit or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). If not: they stay. Since 2026-10-14.
|
||||
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784); print the break-glass sheet and the signing keys (R-923, R-924).
|
||||
- Later (yours, not urgent): R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,4 @@
|
||||
18:53:08Z LOAD START
|
||||
18:53:08Z nextcloud rclone START
|
||||
21:56:07Z nextcloud DONE rc 0
|
||||
21:56:07Z LOAD END
|
||||
@@ -0,0 +1,368 @@
|
||||
18:52:32Z LOAD START
|
||||
18:52:32Z nextcloud rclone START
|
||||
18:52:32Z nextcloud DONE rc 1
|
||||
18:52:32Z immich admin-sign-up 201
|
||||
/root/load.py:42: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
|
||||
st = os.stat(f); ts = dt.datetime.utcfromtimestamp(st.st_mtime).isoformat() + "Z"
|
||||
18:52:33Z immich IMG_20090000.jpg 201
|
||||
18:52:33Z immich IMG_20090018.jpg 201
|
||||
18:52:33Z immich IMG_20090036.jpg 201
|
||||
18:52:33Z immich IMG_20090054.jpg 201
|
||||
18:52:33Z immich IMG_20090072.jpg 201
|
||||
18:52:33Z immich IMG_20090090.jpg 201
|
||||
18:52:33Z immich IMG_20090108.jpg 201
|
||||
18:52:33Z immich IMG_20090126.jpg 201
|
||||
18:52:33Z immich IMG_20090144.jpg 201
|
||||
18:52:33Z immich IMG_20090162.jpg 201
|
||||
18:52:33Z immich IMG_20090180.jpg 201
|
||||
18:52:33Z immich IMG_20090198.jpg 201
|
||||
18:52:33Z immich IMG_20090216.jpg 201
|
||||
18:52:34Z immich IMG_20090234.jpg 201
|
||||
18:52:34Z immich IMG_20090252.jpg 201
|
||||
18:52:34Z immich IMG_20090270.jpg 201
|
||||
18:52:34Z immich IMG_20090288.jpg 201
|
||||
18:52:34Z immich IMG_20100001.jpg 201
|
||||
18:52:34Z immich IMG_20100019.jpg 201
|
||||
18:52:34Z immich IMG_20100037.jpg 201
|
||||
18:52:34Z immich IMG_20100055.jpg 201
|
||||
18:52:34Z immich IMG_20100073.jpg 201
|
||||
18:52:34Z immich IMG_20100091.jpg 201
|
||||
18:52:34Z paperless jelszavas_kivonat.pdf 200
|
||||
18:52:34Z immich IMG_20100109.jpg 201
|
||||
18:52:34Z immich IMG_20100127.jpg 201
|
||||
18:52:34Z immich IMG_20100145.jpg 201
|
||||
18:52:34Z immich IMG_20100163.jpg 201
|
||||
18:52:34Z immich IMG_20100181.jpg 201
|
||||
18:52:34Z immich IMG_20100199.jpg 201
|
||||
18:52:35Z immich IMG_20100217.jpg 201
|
||||
18:52:35Z paperless kezikonyv_100_oldal.pdf 200
|
||||
18:52:35Z immich IMG_20100235.jpg 201
|
||||
18:52:35Z immich IMG_20100253.jpg 201
|
||||
18:52:35Z immich IMG_20100271.jpg 201
|
||||
18:52:35Z immich IMG_20100289.jpg 201
|
||||
18:52:35Z immich IMG_20110002.jpg 201
|
||||
18:52:36Z paperless szamla_2024_000.pdf 200
|
||||
18:52:36Z immich IMG_20110020.jpg 201
|
||||
18:52:36Z immich IMG_20110038.jpg 201
|
||||
18:52:36Z immich IMG_20110056.jpg 201
|
||||
18:52:36Z immich IMG_20110074.jpg 201
|
||||
18:52:36Z immich IMG_20110092.jpg 201
|
||||
18:52:36Z paperless szamla_2024_003.pdf 200
|
||||
18:52:37Z immich IMG_20110110.jpg 201
|
||||
18:52:37Z paperless szamla_2024_006.pdf 200
|
||||
18:52:37Z immich IMG_20110128.jpg 201
|
||||
18:52:37Z immich IMG_20110146.jpg 201
|
||||
18:52:37Z immich IMG_20110164.jpg 201
|
||||
18:52:37Z immich IMG_20110182.jpg 201
|
||||
18:52:38Z paperless szamla_2024_009.pdf 200
|
||||
18:52:38Z immich IMG_20110200.jpg 201
|
||||
18:52:38Z immich IMG_20110218.jpg 201
|
||||
18:52:38Z immich IMG_20110236.jpg 201
|
||||
18:52:38Z paperless szamla_2024_012.pdf 200
|
||||
18:52:38Z immich IMG_20110254.jpg 201
|
||||
18:52:39Z immich IMG_20110272.jpg 201
|
||||
18:52:39Z immich IMG_20110290.jpg 201
|
||||
18:52:39Z immich IMG_20120003.jpg 201
|
||||
18:52:39Z immich IMG_20120021.jpg 201
|
||||
18:52:39Z paperless szamla_2024_015.pdf 200
|
||||
18:52:39Z immich IMG_20120039.jpg 201
|
||||
18:52:39Z immich IMG_20120057.jpg 201
|
||||
18:52:39Z immich IMG_20120075.jpg 201
|
||||
18:52:39Z immich IMG_20120093.jpg 201
|
||||
18:52:40Z immich IMG_20120111.jpg 201
|
||||
18:52:40Z paperless szamla_2024_018.pdf 200
|
||||
18:52:40Z immich IMG_20120129.jpg 201
|
||||
18:52:40Z immich IMG_20120147.jpg 201
|
||||
18:52:40Z immich IMG_20120165.jpg 201
|
||||
18:52:40Z immich IMG_20120183.jpg 201
|
||||
18:52:40Z paperless szamla_2024_021.pdf 200
|
||||
18:52:40Z immich IMG_20120201.jpg 201
|
||||
18:52:40Z immich IMG_20120219.jpg 201
|
||||
18:52:41Z immich IMG_20120237.jpg 201
|
||||
18:52:41Z paperless szamla_2024_024.pdf 200
|
||||
18:52:41Z immich IMG_20120255.jpg 201
|
||||
18:52:41Z immich IMG_20120273.jpg 201
|
||||
18:52:41Z immich IMG_20120291.jpg 201
|
||||
18:52:41Z immich IMG_20130004.jpg 201
|
||||
18:52:41Z paperless szamla_2024_027.pdf 200
|
||||
18:52:42Z immich IMG_20130022.jpg 201
|
||||
18:52:42Z immich IMG_20130040.jpg 201
|
||||
18:52:42Z paperless szamla_2024_030.pdf 200
|
||||
18:52:42Z immich IMG_20130058.jpg 201
|
||||
18:52:42Z immich IMG_20130076.jpg 201
|
||||
18:52:43Z immich IMG_20130094.jpg 201
|
||||
18:52:44Z paperless szamla_2024_033.pdf 200
|
||||
18:52:44Z immich IMG_20130112.jpg 201
|
||||
18:52:44Z immich IMG_20130130.jpg 201
|
||||
18:52:45Z immich IMG_20130148.jpg 201
|
||||
18:52:45Z paperless szamla_2024_036.pdf 200
|
||||
18:52:45Z immich IMG_20130166.jpg 201
|
||||
18:52:46Z paperless szamla_2024_039.pdf 200
|
||||
18:52:46Z immich IMG_20130184.jpg 201
|
||||
18:52:46Z immich IMG_20130202.jpg 201
|
||||
18:52:46Z immich IMG_20130220.jpg 201
|
||||
18:52:46Z immich IMG_20130238.jpg 201
|
||||
18:52:46Z immich IMG_20130256.jpg 201
|
||||
18:52:46Z immich IMG_20130274.jpg 201
|
||||
18:52:46Z paperless szamla_2024_042.pdf 200
|
||||
18:52:47Z immich IMG_20130292.jpg 201
|
||||
18:52:47Z immich IMG_20140005.jpg 201
|
||||
18:52:47Z immich IMG_20140023.jpg 201
|
||||
18:52:47Z immich IMG_20140041.jpg 201
|
||||
18:52:48Z paperless szamla_2024_045.pdf 200
|
||||
18:52:48Z immich IMG_20140059.jpg 201
|
||||
18:52:49Z paperless szamla_2024_048.pdf 200
|
||||
18:52:49Z immich IMG_20140077.jpg 201
|
||||
18:52:49Z immich IMG_20140095.jpg 201
|
||||
18:52:50Z immich IMG_20140113.jpg 201
|
||||
18:52:50Z immich IMG_20140131.jpg 201
|
||||
18:52:50Z immich IMG_20140149.jpg 201
|
||||
18:52:50Z immich IMG_20140167.jpg 201
|
||||
18:52:50Z paperless szamla_2025_001.pdf 200
|
||||
18:52:50Z immich IMG_20140185.jpg 201
|
||||
18:52:51Z paperless szamla_2025_004.pdf 200
|
||||
18:52:51Z immich IMG_20140203.jpg 201
|
||||
18:52:51Z immich IMG_20140221.jpg 201
|
||||
18:52:51Z paperless szamla_2025_007.pdf 200
|
||||
18:52:51Z immich IMG_20140239.jpg 201
|
||||
18:52:51Z immich IMG_20140257.jpg 201
|
||||
18:52:52Z immich IMG_20140275.jpg 201
|
||||
18:52:52Z immich IMG_20140293.jpg 201
|
||||
18:52:52Z immich IMG_20150006.jpg 201
|
||||
18:52:52Z immich IMG_20150024.jpg 201
|
||||
18:52:52Z immich IMG_20150042.jpg 201
|
||||
18:52:52Z immich IMG_20150060.jpg 201
|
||||
18:52:52Z immich IMG_20150078.jpg 201
|
||||
18:52:52Z paperless szamla_2025_010.pdf 200
|
||||
18:52:52Z immich IMG_20150096.jpg 201
|
||||
18:52:52Z immich IMG_20150114.jpg 201
|
||||
18:52:52Z immich IMG_20150132.jpg 201
|
||||
18:52:53Z immich IMG_20150150.jpg 201
|
||||
18:52:53Z immich IMG_20150168.jpg 201
|
||||
18:52:53Z paperless szamla_2025_013.pdf 200
|
||||
18:52:55Z paperless szamla_2025_016.pdf 200
|
||||
18:52:55Z immich IMG_20150186.jpg 201
|
||||
18:52:55Z immich IMG_20150204.jpg 201
|
||||
18:52:55Z immich IMG_20150222.jpg 201
|
||||
18:52:56Z immich IMG_20150240.jpg 201
|
||||
18:52:56Z immich IMG_20150258.jpg 201
|
||||
18:52:56Z immich IMG_20150276.jpg 201
|
||||
18:52:56Z paperless szamla_2025_019.pdf 200
|
||||
18:52:57Z immich IMG_20150294.jpg 201
|
||||
18:52:57Z immich IMG_20160007.jpg 201
|
||||
18:52:57Z immich IMG_20160025.jpg 201
|
||||
18:52:57Z immich IMG_20160043.jpg 201
|
||||
18:52:57Z paperless szamla_2025_022.pdf 200
|
||||
18:52:57Z immich IMG_20160061.jpg 201
|
||||
18:52:58Z immich IMG_20160079.jpg 201
|
||||
18:52:58Z immich IMG_20160097.jpg 201
|
||||
18:52:58Z immich IMG_20160115.jpg 201
|
||||
18:52:58Z immich IMG_20160133.jpg 201
|
||||
18:52:58Z paperless szamla_2025_025.pdf 200
|
||||
18:52:58Z immich IMG_20160151.jpg 201
|
||||
18:52:59Z immich IMG_20160169.jpg 201
|
||||
18:52:59Z paperless szamla_2025_028.pdf 200
|
||||
18:52:59Z immich IMG_20160187.jpg 201
|
||||
18:52:59Z immich IMG_20160205.jpg 201
|
||||
18:52:59Z immich IMG_20160223.jpg 201
|
||||
18:52:59Z immich IMG_20160241.jpg 201
|
||||
18:52:59Z immich IMG_20160259.jpg 201
|
||||
18:52:59Z paperless szamla_2025_031.pdf 200
|
||||
18:53:00Z immich IMG_20160277.jpg 201
|
||||
18:53:00Z immich IMG_20160295.jpg 201
|
||||
18:53:00Z immich IMG_20170008.jpg 201
|
||||
18:53:00Z immich IMG_20170026.jpg 201
|
||||
18:53:00Z immich IMG_20170044.jpg 201
|
||||
18:53:00Z immich IMG_20170062.jpg 201
|
||||
18:53:00Z paperless szamla_2025_034.pdf 200
|
||||
18:53:00Z immich IMG_20170080.jpg 201
|
||||
18:53:00Z immich IMG_20170098.jpg 201
|
||||
18:53:01Z immich IMG_20170116.jpg 201
|
||||
18:53:01Z paperless szamla_2025_037.pdf 200
|
||||
18:53:01Z immich IMG_20170134.jpg 201
|
||||
18:53:01Z immich IMG_20170152.jpg 201
|
||||
18:53:01Z immich IMG_20170170.jpg 201
|
||||
18:53:01Z immich IMG_20170188.jpg 201
|
||||
18:53:01Z immich IMG_20170206.jpg 201
|
||||
18:53:01Z paperless szamla_2025_040.pdf 200
|
||||
18:53:02Z immich IMG_20170224.jpg 201
|
||||
18:53:02Z immich IMG_20170242.jpg 201
|
||||
18:53:02Z immich IMG_20170260.jpg 201
|
||||
18:53:02Z immich IMG_20170278.jpg 201
|
||||
18:53:02Z immich IMG_20170296.jpg 201
|
||||
18:53:02Z immich IMG_20180009.jpg 201
|
||||
18:53:03Z immich IMG_20180027.jpg 201
|
||||
18:53:03Z paperless szamla_2025_043.pdf 200
|
||||
18:53:03Z immich IMG_20180045.jpg 201
|
||||
18:53:03Z immich IMG_20180063.jpg 201
|
||||
18:53:03Z immich IMG_20180081.jpg 201
|
||||
18:53:04Z paperless szamla_2025_046.pdf 200
|
||||
18:53:04Z immich IMG_20180099.jpg 201
|
||||
18:53:04Z immich IMG_20180117.jpg 201
|
||||
18:53:05Z immich IMG_20180135.jpg 201
|
||||
18:53:05Z immich IMG_20180153.jpg 201
|
||||
18:53:05Z paperless szamla_2025_049.pdf 200
|
||||
18:53:05Z immich IMG_20180171.jpg 201
|
||||
18:53:05Z immich IMG_20180189.jpg 201
|
||||
18:53:05Z immich IMG_20180207.jpg 201
|
||||
18:53:06Z paperless szamla_2026_002.pdf 200
|
||||
18:53:07Z paperless szamla_2026_005.pdf 200
|
||||
18:53:08Z paperless szamla_2026_008.pdf 200
|
||||
18:53:08Z immich IMG_20180225.jpg 201
|
||||
18:53:09Z immich IMG_20180243.jpg 201
|
||||
18:53:09Z paperless szamla_2026_011.pdf 200
|
||||
18:53:09Z immich IMG_20180261.jpg 201
|
||||
18:53:09Z immich IMG_20180279.jpg 201
|
||||
18:53:10Z immich IMG_20180297.jpg 201
|
||||
18:53:10Z paperless szamla_2026_014.pdf 200
|
||||
18:53:11Z paperless szamla_2026_017.pdf 200
|
||||
18:53:11Z immich IMG_20190010.jpg 201
|
||||
18:53:11Z immich IMG_20190028.jpg 201
|
||||
18:53:11Z immich IMG_20190046.jpg 201
|
||||
18:53:12Z immich IMG_20190064.jpg 201
|
||||
18:53:12Z immich IMG_20190082.jpg 201
|
||||
18:53:12Z immich IMG_20190100.jpg 201
|
||||
18:53:13Z paperless szamla_2026_020.pdf 200
|
||||
18:53:13Z immich IMG_20190118.jpg 201
|
||||
18:53:13Z immich IMG_20190136.jpg 201
|
||||
18:53:13Z immich IMG_20190154.jpg 201
|
||||
18:53:13Z immich IMG_20190172.jpg 201
|
||||
18:53:14Z immich IMG_20190190.jpg 201
|
||||
18:53:14Z immich IMG_20190208.jpg 201
|
||||
18:53:14Z immich IMG_20190226.jpg 201
|
||||
18:53:14Z immich IMG_20190244.jpg 201
|
||||
18:53:14Z paperless szamla_2026_023.pdf 200
|
||||
18:53:14Z immich IMG_20190262.jpg 201
|
||||
18:53:14Z immich IMG_20190280.jpg 201
|
||||
18:53:15Z immich IMG_20190298.jpg 201
|
||||
18:53:15Z immich IMG_20200011.jpg 201
|
||||
18:53:15Z immich IMG_20200029.jpg 201
|
||||
18:53:15Z immich IMG_20200047.jpg 201
|
||||
18:53:15Z immich IMG_20200065.jpg 201
|
||||
18:53:16Z paperless szamla_2026_026.pdf 200
|
||||
18:53:16Z immich IMG_20200083.jpg 201
|
||||
18:53:16Z immich IMG_20200101.jpg 201
|
||||
18:53:16Z immich IMG_20200119.jpg 201
|
||||
18:53:17Z immich IMG_20200137.jpg 201
|
||||
18:53:17Z immich IMG_20200155.jpg 201
|
||||
18:53:18Z immich IMG_20200173.jpg 201
|
||||
18:53:19Z immich IMG_20200191.jpg 201
|
||||
18:53:20Z immich IMG_20200209.jpg 201
|
||||
18:53:20Z immich IMG_20200227.jpg 201
|
||||
18:53:20Z paperless szamla_2026_029.pdf 200
|
||||
18:53:20Z immich IMG_20200245.jpg 201
|
||||
18:53:20Z immich IMG_20200263.jpg 201
|
||||
18:53:21Z immich IMG_20200281.jpg 201
|
||||
18:53:21Z immich IMG_20200299.jpg 201
|
||||
18:53:21Z immich IMG_20210012.jpg 201
|
||||
18:53:21Z immich IMG_20210030.jpg 201
|
||||
18:53:21Z immich IMG_20210048.jpg 201
|
||||
18:53:22Z immich IMG_20210066.jpg 201
|
||||
18:53:22Z paperless szamla_2026_032.pdf 200
|
||||
18:53:22Z immich IMG_20210084.jpg 201
|
||||
18:53:22Z immich IMG_20210102.jpg 201
|
||||
18:53:22Z immich IMG_20210120.jpg 201
|
||||
18:53:22Z immich IMG_20210138.jpg 201
|
||||
18:53:23Z immich IMG_20210156.jpg 201
|
||||
18:53:23Z paperless szamla_2026_035.pdf 200
|
||||
18:53:23Z immich IMG_20210174.jpg 201
|
||||
18:53:23Z immich IMG_20210192.jpg 201
|
||||
18:53:24Z immich IMG_20210210.jpg 201
|
||||
18:53:24Z paperless szamla_2026_038.pdf 200
|
||||
18:53:24Z immich IMG_20210228.jpg 201
|
||||
18:53:24Z immich IMG_20210246.jpg 201
|
||||
18:53:25Z immich IMG_20210264.jpg 201
|
||||
18:53:25Z immich IMG_20210282.jpg 201
|
||||
18:53:25Z immich IMG_20220013.jpg 201
|
||||
18:53:25Z immich IMG_20220031.jpg 201
|
||||
18:53:26Z paperless szamla_2026_041.pdf 200
|
||||
18:53:26Z immich IMG_20220049.jpg 201
|
||||
18:53:26Z immich IMG_20220067.jpg 201
|
||||
18:53:27Z immich IMG_20220085.jpg 201
|
||||
18:53:27Z paperless szamla_2026_044.pdf 200
|
||||
18:53:27Z immich IMG_20220103.jpg 201
|
||||
18:53:28Z paperless szamla_2026_047.pdf 200
|
||||
18:53:28Z immich IMG_20220121.jpg 201
|
||||
18:53:28Z immich IMG_20220139.jpg 201
|
||||
18:53:29Z immich IMG_20220157.jpg 201
|
||||
18:53:29Z paperless szkennelt_nyilatkozat.pdf 200
|
||||
18:53:29Z paperless DONE ok 53 bad 0
|
||||
18:53:29Z immich IMG_20220175.jpg 201
|
||||
18:53:29Z immich IMG_20220193.jpg 201
|
||||
18:53:30Z immich IMG_20220211.jpg 201
|
||||
18:53:30Z immich IMG_20220229.jpg 201
|
||||
18:53:30Z immich IMG_20220247.jpg 201
|
||||
18:53:30Z immich IMG_20220265.jpg 201
|
||||
18:53:31Z immich IMG_20220283.jpg 201
|
||||
18:53:31Z immich IMG_20230014.jpg 201
|
||||
18:53:32Z immich IMG_20230032.jpg 201
|
||||
18:53:32Z immich IMG_20230050.jpg 201
|
||||
18:53:32Z immich IMG_20230068.jpg 201
|
||||
18:53:33Z immich IMG_20230086.jpg 201
|
||||
18:53:33Z immich IMG_20230104.jpg 201
|
||||
18:53:33Z immich IMG_20230122.jpg 201
|
||||
18:53:34Z immich IMG_20230140.jpg 201
|
||||
18:53:34Z immich IMG_20230158.jpg 201
|
||||
18:53:35Z immich IMG_20230176.jpg 201
|
||||
18:53:35Z immich IMG_20230194.jpg 201
|
||||
18:53:36Z immich IMG_20230212.jpg 201
|
||||
18:53:36Z immich IMG_20230230.jpg 201
|
||||
18:53:37Z immich IMG_20230248.jpg 201
|
||||
18:53:37Z immich IMG_20230266.jpg 201
|
||||
18:53:37Z immich IMG_20230284.jpg 201
|
||||
18:53:38Z immich IMG_20240015.jpg 201
|
||||
18:53:39Z immich IMG_20240033.jpg 201
|
||||
18:53:39Z immich IMG_20240051.jpg 201
|
||||
18:53:40Z immich IMG_20240069.jpg 201
|
||||
18:53:40Z immich IMG_20240087.jpg 201
|
||||
18:53:40Z immich IMG_20240105.jpg 201
|
||||
18:53:41Z immich IMG_20240123.jpg 201
|
||||
18:53:41Z immich IMG_20240141.jpg 201
|
||||
18:53:41Z immich IMG_20240159.jpg 201
|
||||
18:53:42Z immich IMG_20240177.jpg 201
|
||||
18:53:42Z immich IMG_20240195.jpg 201
|
||||
18:53:42Z immich IMG_20240213.jpg 201
|
||||
18:53:43Z immich IMG_20240231.jpg 201
|
||||
18:53:43Z immich IMG_20240249.jpg 201
|
||||
18:53:43Z immich IMG_20240267.jpg 201
|
||||
18:53:44Z immich IMG_20240285.jpg 201
|
||||
18:53:44Z immich IMG_20250016.jpg 201
|
||||
18:53:44Z immich IMG_20250034.jpg 201
|
||||
18:53:45Z immich IMG_20250052.jpg 201
|
||||
18:53:45Z immich IMG_20250070.jpg 201
|
||||
18:53:45Z immich IMG_20250088.jpg 201
|
||||
18:53:45Z immich IMG_20250106.jpg 201
|
||||
18:53:45Z immich IMG_20250124.jpg 201
|
||||
18:53:47Z immich IMG_20250142.jpg 201
|
||||
18:53:47Z immich IMG_20250160.jpg 201
|
||||
18:53:47Z immich IMG_20250178.jpg 201
|
||||
18:53:48Z immich IMG_20250196.jpg 201
|
||||
18:53:48Z immich IMG_20250214.jpg 201
|
||||
18:53:49Z immich IMG_20250232.jpg 201
|
||||
18:53:49Z immich IMG_20250250.jpg 201
|
||||
18:53:49Z immich IMG_20250268.jpg 201
|
||||
18:53:49Z immich IMG_20250286.jpg 201
|
||||
18:53:50Z immich IMG_20260017.jpg 201
|
||||
18:53:50Z immich IMG_20260035.jpg 201
|
||||
18:53:50Z immich IMG_20260053.jpg 201
|
||||
18:53:50Z immich IMG_20260071.jpg 201
|
||||
18:53:50Z immich IMG_20260089.jpg 201
|
||||
18:53:50Z immich IMG_20260107.jpg 201
|
||||
18:53:51Z immich IMG_20260125.jpg 201
|
||||
18:53:51Z immich IMG_20260143.jpg 201
|
||||
18:53:51Z immich IMG_20260161.jpg 201
|
||||
18:53:52Z immich IMG_20260179.jpg 201
|
||||
18:53:52Z immich IMG_20260197.jpg 201
|
||||
18:53:53Z immich IMG_20260215.jpg 201
|
||||
18:53:53Z immich IMG_20260233.jpg 201
|
||||
18:53:53Z immich IMG_20260251.jpg 201
|
||||
18:53:53Z immich IMG_20260269.jpg 201
|
||||
18:53:54Z immich IMG_20260287.jpg 201
|
||||
18:53:54Z immich IMG_20160007_masolat.jpg 200
|
||||
18:53:54Z immich duplikatum_IMG_20160007.jpg 200
|
||||
18:54:03Z immich nyaralas_2025.mp4 201
|
||||
18:54:03Z immich rajz_Kata.png 201
|
||||
18:54:03Z immich torott_kep.jpg 201
|
||||
18:54:03Z immich ures.jpg 400
|
||||
18:54:03Z immich DONE ok 305 bad 1
|
||||
18:54:03Z LOAD END
|
||||
@@ -0,0 +1,478 @@
|
||||
18:54:04Z PROOF START 118 min — each stream repeats on its own until the end
|
||||
18:54:06Z c1 immich new user 201
|
||||
18:54:13Z c1 paperless delete 6 200
|
||||
18:54:14Z c1 paperless empty trash 200
|
||||
18:55:26Z c1 immich DONE ok 305 bad 1
|
||||
18:55:27Z c1 paperless DONE ok 53 bad 0
|
||||
18:55:27Z c2 immich new user 201
|
||||
18:55:29Z c2 paperless delete 5 200
|
||||
18:55:32Z c2 paperless empty trash 200
|
||||
18:56:36Z c2 paperless DONE ok 53 bad 0
|
||||
18:56:36Z c2 immich DONE ok 305 bad 1
|
||||
18:56:39Z c3 immich new user 201
|
||||
18:56:42Z c3 paperless delete 4 200
|
||||
18:56:44Z c3 paperless empty trash 200
|
||||
18:57:48Z c3 paperless DONE ok 53 bad 0
|
||||
18:57:52Z c4 paperless delete 4 200
|
||||
18:57:53Z c4 paperless empty trash 200
|
||||
18:57:56Z c3 immich DONE ok 305 bad 1
|
||||
18:57:58Z c4 immich new user 201
|
||||
18:58:53Z c4 paperless DONE ok 53 bad 0
|
||||
18:58:56Z c5 paperless delete 5 200
|
||||
18:58:58Z c5 paperless empty trash 200
|
||||
18:59:12Z c4 immich DONE ok 305 bad 1
|
||||
18:59:14Z c5 immich new user 201
|
||||
19:00:05Z c5 paperless DONE ok 53 bad 0
|
||||
19:00:13Z c6 paperless delete 4 200
|
||||
19:00:14Z c6 paperless empty trash 200
|
||||
19:00:32Z c5 immich DONE ok 305 bad 1
|
||||
19:00:34Z c6 immich new user 201
|
||||
19:01:20Z c6 paperless DONE ok 53 bad 0
|
||||
19:01:23Z c7 paperless delete 5 200
|
||||
19:01:24Z c7 paperless empty trash 200
|
||||
19:01:46Z c6 immich DONE ok 305 bad 1
|
||||
19:01:48Z c7 immich new user 201
|
||||
19:02:27Z c7 paperless DONE ok 53 bad 0
|
||||
19:02:31Z c8 paperless delete 5 200
|
||||
19:02:32Z c8 paperless empty trash 200
|
||||
19:02:58Z c7 immich DONE ok 305 bad 1
|
||||
19:03:00Z c8 immich new user 201
|
||||
19:03:39Z c8 paperless DONE ok 53 bad 0
|
||||
19:03:45Z c9 paperless delete 4 200
|
||||
19:03:47Z c9 paperless empty trash 200
|
||||
19:04:15Z c8 immich DONE ok 305 bad 1
|
||||
19:04:17Z c9 immich new user 201
|
||||
19:04:50Z c9 paperless DONE ok 53 bad 0
|
||||
19:04:53Z c10 paperless delete 4 200
|
||||
19:04:54Z c10 paperless empty trash 200
|
||||
19:05:49Z c9 immich DONE ok 305 bad 1
|
||||
19:05:55Z c10 immich new user 201
|
||||
19:06:03Z c10 paperless DONE ok 53 bad 0
|
||||
19:06:08Z c11 paperless delete 4 200
|
||||
19:06:09Z c11 paperless empty trash 200
|
||||
19:07:11Z c11 paperless DONE ok 53 bad 0
|
||||
19:07:14Z c12 paperless delete 4 200
|
||||
19:07:16Z c12 paperless empty trash 200
|
||||
19:07:23Z c10 immich DONE ok 305 bad 1
|
||||
19:07:24Z c11 immich new user 201
|
||||
19:08:19Z c12 paperless DONE ok 53 bad 0
|
||||
19:08:24Z c13 paperless delete 4 200
|
||||
19:08:26Z c13 paperless empty trash 200
|
||||
19:08:30Z c11 immich DONE ok 305 bad 1
|
||||
19:08:33Z c12 immich new user 201
|
||||
19:09:24Z c13 paperless DONE ok 53 bad 0
|
||||
19:09:26Z c14 paperless delete 4 200
|
||||
19:09:27Z c14 paperless empty trash 200
|
||||
19:09:37Z c12 immich DONE ok 305 bad 1
|
||||
19:09:38Z c13 immich new user 201
|
||||
19:10:34Z c14 paperless DONE ok 53 bad 0
|
||||
19:10:38Z c15 paperless delete 4 200
|
||||
19:10:40Z c15 paperless empty trash 200
|
||||
19:10:48Z c13 immich DONE ok 305 bad 1
|
||||
19:10:51Z c14 immich new user 201
|
||||
19:11:45Z c15 paperless DONE ok 53 bad 0
|
||||
19:11:49Z c16 paperless delete 4 200
|
||||
19:11:50Z c16 paperless empty trash 200
|
||||
19:12:17Z c14 immich DONE ok 305 bad 1
|
||||
19:12:18Z c15 immich new user 201
|
||||
19:12:57Z c16 paperless DONE ok 53 bad 0
|
||||
19:13:01Z c17 paperless delete 5 200
|
||||
19:13:04Z c17 paperless empty trash 200
|
||||
19:13:28Z c15 immich DONE ok 305 bad 1
|
||||
19:13:31Z c16 immich new user 201
|
||||
19:14:09Z c17 paperless DONE ok 53 bad 0
|
||||
19:14:13Z c18 paperless delete 4 200
|
||||
19:14:14Z c18 paperless empty trash 200
|
||||
19:14:36Z c16 immich DONE ok 305 bad 1
|
||||
19:14:38Z c17 immich new user 201
|
||||
19:15:22Z c18 paperless DONE ok 53 bad 0
|
||||
19:15:25Z c19 paperless delete 5 200
|
||||
19:15:27Z c19 paperless empty trash 200
|
||||
19:15:39Z c17 immich DONE ok 305 bad 1
|
||||
19:15:42Z c18 immich new user 201
|
||||
19:16:33Z c19 paperless DONE ok 53 bad 0
|
||||
19:16:36Z c20 paperless delete 4 200
|
||||
19:16:38Z c20 paperless empty trash 200
|
||||
19:16:48Z c18 immich DONE ok 305 bad 1
|
||||
19:16:50Z c19 immich new user 201
|
||||
19:17:42Z c20 paperless DONE ok 53 bad 0
|
||||
19:17:46Z c21 paperless delete 5 200
|
||||
19:17:47Z c21 paperless empty trash 200
|
||||
19:17:50Z c19 immich DONE ok 305 bad 1
|
||||
19:17:53Z c20 immich new user 201
|
||||
19:18:43Z c21 paperless DONE ok 53 bad 0
|
||||
19:18:45Z c22 paperless delete 4 200
|
||||
19:18:47Z c22 paperless empty trash 200
|
||||
19:18:59Z c20 immich DONE ok 305 bad 1
|
||||
19:19:00Z c21 immich new user 201
|
||||
19:19:51Z c22 paperless DONE ok 53 bad 0
|
||||
19:19:54Z c23 paperless delete 4 200
|
||||
19:19:56Z c23 paperless empty trash 200
|
||||
19:20:07Z c21 immich DONE ok 305 bad 1
|
||||
19:20:09Z c22 immich new user 201
|
||||
19:20:57Z c23 paperless DONE ok 53 bad 0
|
||||
19:21:00Z c24 paperless delete 4 200
|
||||
19:21:01Z c24 paperless empty trash 200
|
||||
19:21:26Z c22 immich DONE ok 305 bad 1
|
||||
19:21:28Z c23 immich new user 201
|
||||
19:22:14Z c24 paperless DONE ok 53 bad 0
|
||||
19:22:17Z c25 paperless delete 3 200
|
||||
19:22:17Z c25 paperless empty trash 200
|
||||
19:22:32Z c23 immich DONE ok 305 bad 1
|
||||
19:22:34Z c24 immich new user 201
|
||||
19:23:20Z c25 paperless DONE ok 53 bad 0
|
||||
19:23:25Z c26 paperless delete 4 200
|
||||
19:23:27Z c26 paperless empty trash 200
|
||||
19:23:43Z c24 immich DONE ok 305 bad 1
|
||||
19:23:45Z c25 immich new user 201
|
||||
19:24:36Z c26 paperless DONE ok 53 bad 0
|
||||
19:24:39Z c27 paperless delete 4 200
|
||||
19:24:40Z c27 paperless empty trash 200
|
||||
19:25:10Z c25 immich DONE ok 305 bad 1
|
||||
19:25:12Z c26 immich new user 201
|
||||
19:25:43Z c27 paperless DONE ok 53 bad 0
|
||||
19:25:52Z c28 paperless delete 4 200
|
||||
19:25:54Z c28 paperless empty trash 200
|
||||
19:26:36Z c26 immich DONE ok 305 bad 1
|
||||
19:26:39Z c27 immich new user 201
|
||||
19:27:21Z c28 paperless DONE ok 53 bad 0
|
||||
19:27:29Z c29 paperless delete 5 200
|
||||
19:27:30Z c29 paperless empty trash 200
|
||||
19:28:00Z c27 immich DONE ok 305 bad 1
|
||||
19:28:02Z c28 immich new user 201
|
||||
19:28:35Z c29 paperless DONE ok 53 bad 0
|
||||
19:28:37Z c30 paperless delete 4 200
|
||||
19:28:38Z c30 paperless empty trash 200
|
||||
19:29:09Z c28 immich DONE ok 305 bad 1
|
||||
19:29:13Z c29 immich new user 201
|
||||
19:29:46Z c30 paperless DONE ok 53 bad 0
|
||||
19:29:50Z c31 paperless delete 4 200
|
||||
19:29:51Z c31 paperless empty trash 200
|
||||
19:30:17Z c29 immich DONE ok 305 bad 1
|
||||
19:30:20Z c30 immich new user 201
|
||||
19:31:28Z c31 paperless DONE ok 53 bad 0
|
||||
19:31:30Z c32 paperless delete 6 200
|
||||
19:31:31Z c32 paperless empty trash 200
|
||||
19:31:54Z c30 immich DONE ok 305 bad 1
|
||||
19:31:56Z c31 immich new user 201
|
||||
19:32:55Z c32 paperless DONE ok 53 bad 0
|
||||
19:32:58Z c33 paperless delete 4 200
|
||||
19:32:59Z c33 paperless empty trash 200
|
||||
19:33:12Z c31 immich DONE ok 305 bad 1
|
||||
19:33:14Z c32 immich new user 201
|
||||
19:34:06Z c33 paperless DONE ok 53 bad 0
|
||||
19:34:11Z c34 paperless delete 5 200
|
||||
19:34:12Z c34 paperless empty trash 200
|
||||
19:34:32Z c32 immich DONE ok 305 bad 1
|
||||
19:34:33Z c33 immich new user 201
|
||||
19:35:18Z c34 paperless DONE ok 53 bad 0
|
||||
19:35:20Z c35 paperless delete 5 200
|
||||
19:35:21Z c35 paperless empty trash 200
|
||||
19:35:55Z c33 immich DONE ok 305 bad 1
|
||||
19:35:57Z c34 immich new user 201
|
||||
19:36:29Z c35 paperless DONE ok 53 bad 0
|
||||
19:36:32Z c36 paperless delete 3 200
|
||||
19:36:34Z c36 paperless empty trash 200
|
||||
19:37:05Z c34 immich DONE ok 305 bad 1
|
||||
19:37:09Z c35 immich new user 201
|
||||
19:37:41Z c36 paperless DONE ok 53 bad 0
|
||||
19:37:45Z c37 paperless delete 4 200
|
||||
19:37:47Z c37 paperless empty trash 200
|
||||
19:38:14Z c35 immich DONE ok 305 bad 1
|
||||
19:38:17Z c36 immich new user 201
|
||||
19:38:50Z c37 paperless DONE ok 53 bad 0
|
||||
19:38:53Z c38 paperless delete 3 200
|
||||
19:38:54Z c38 paperless empty trash 200
|
||||
19:39:54Z c38 paperless DONE ok 53 bad 0
|
||||
19:39:57Z c39 paperless delete 4 200
|
||||
19:39:59Z c39 paperless empty trash 200
|
||||
19:40:53Z c39 paperless DONE ok 53 bad 0
|
||||
19:40:55Z c40 paperless delete 4 200
|
||||
19:40:56Z c40 paperless empty trash 200
|
||||
19:41:47Z c40 paperless DONE ok 53 bad 0
|
||||
19:41:50Z c41 paperless delete 4 200
|
||||
19:41:50Z c41 paperless empty trash 200
|
||||
19:42:50Z c41 paperless DONE ok 53 bad 0
|
||||
19:42:51Z c42 paperless delete 5 200
|
||||
19:42:53Z c42 paperless empty trash 200
|
||||
19:43:50Z c42 paperless DONE ok 53 bad 0
|
||||
19:43:53Z c43 paperless delete 4 200
|
||||
19:43:54Z c43 paperless empty trash 200
|
||||
19:44:54Z c43 paperless DONE ok 53 bad 0
|
||||
19:44:57Z c44 paperless delete 5 200
|
||||
19:44:58Z c44 paperless empty trash 200
|
||||
19:45:56Z c44 paperless DONE ok 53 bad 0
|
||||
19:45:59Z c45 paperless delete 4 200
|
||||
19:46:01Z c45 paperless empty trash 200
|
||||
19:46:57Z c45 paperless DONE ok 53 bad 0
|
||||
19:47:01Z c46 paperless delete 5 200
|
||||
19:47:03Z c46 paperless empty trash 200
|
||||
19:48:01Z c46 paperless DONE ok 53 bad 0
|
||||
19:48:03Z c47 paperless delete 5 200
|
||||
19:48:05Z c47 paperless empty trash 200
|
||||
19:48:30Z c36 immich IMG_20120021.jpg ERR HTTPSConnectionPool(host='192.168.0.125', port=443): Read timed out. (read timeout=600)
|
||||
19:49:00Z c47 paperless DONE ok 53 bad 0
|
||||
19:49:02Z c48 paperless delete 4 200
|
||||
19:49:04Z c48 paperless empty trash 200
|
||||
19:49:36Z c36 immich DONE ok 304 bad 2
|
||||
19:49:38Z c37 immich new user 201
|
||||
19:50:14Z c48 paperless DONE ok 53 bad 0
|
||||
19:50:20Z c49 paperless delete 4 200
|
||||
19:50:21Z c49 paperless empty trash 200
|
||||
19:50:56Z c37 immich DONE ok 305 bad 1
|
||||
19:50:58Z c38 immich new user 201
|
||||
19:51:29Z c49 paperless DONE ok 53 bad 0
|
||||
19:51:32Z c50 paperless delete 3 200
|
||||
19:51:34Z c50 paperless empty trash 200
|
||||
19:52:16Z c38 immich DONE ok 305 bad 1
|
||||
19:52:18Z c39 immich new user 201
|
||||
19:52:43Z c50 paperless DONE ok 53 bad 0
|
||||
19:52:48Z c51 paperless delete 5 200
|
||||
19:52:51Z c51 paperless empty trash 200
|
||||
19:53:27Z c39 immich DONE ok 305 bad 1
|
||||
19:53:29Z c40 immich new user 201
|
||||
19:53:49Z c51 paperless DONE ok 53 bad 0
|
||||
19:53:55Z c52 paperless delete 4 200
|
||||
19:53:56Z c52 paperless empty trash 200
|
||||
19:54:58Z c40 immich DONE ok 305 bad 1
|
||||
19:55:00Z c41 immich new user 201
|
||||
19:55:00Z c52 paperless DONE ok 53 bad 0
|
||||
19:55:05Z c53 paperless delete 4 200
|
||||
19:55:06Z c53 paperless empty trash 200
|
||||
19:56:11Z c41 immich DONE ok 305 bad 1
|
||||
19:56:12Z c53 paperless DONE ok 53 bad 0
|
||||
19:56:14Z c42 immich new user 201
|
||||
19:56:16Z c54 paperless delete 5 200
|
||||
19:56:18Z c54 paperless empty trash 200
|
||||
19:57:25Z c42 immich DONE ok 305 bad 1
|
||||
19:57:27Z c43 immich new user 201
|
||||
19:57:28Z c54 paperless DONE ok 53 bad 0
|
||||
19:57:32Z c55 paperless delete 5 200
|
||||
19:57:33Z c55 paperless empty trash 200
|
||||
19:58:35Z c43 immich DONE ok 305 bad 1
|
||||
19:58:37Z c55 paperless DONE ok 53 bad 0
|
||||
19:58:37Z c44 immich new user 201
|
||||
19:58:42Z c56 paperless delete 4 200
|
||||
19:58:45Z c56 paperless empty trash 200
|
||||
19:59:49Z c44 immich DONE ok 305 bad 1
|
||||
19:59:52Z c45 immich new user 201
|
||||
19:59:52Z c56 paperless DONE ok 53 bad 0
|
||||
19:59:57Z c57 paperless delete 4 200
|
||||
19:59:59Z c57 paperless empty trash 200
|
||||
20:01:08Z c57 paperless DONE ok 53 bad 0
|
||||
20:01:09Z c45 immich DONE ok 305 bad 1
|
||||
20:01:11Z c46 immich new user 201
|
||||
20:01:15Z c58 paperless delete 5 200
|
||||
20:01:17Z c58 paperless empty trash 200
|
||||
20:02:23Z c46 immich DONE ok 305 bad 1
|
||||
20:02:24Z c58 paperless DONE ok 53 bad 0
|
||||
20:02:25Z c47 immich new user 201
|
||||
20:02:28Z c59 paperless delete 4 200
|
||||
20:02:29Z c59 paperless empty trash 200
|
||||
20:03:29Z c47 immich DONE ok 305 bad 1
|
||||
20:03:31Z c48 immich new user 201
|
||||
20:03:33Z c59 paperless DONE ok 53 bad 0
|
||||
20:03:36Z c60 paperless delete 4 200
|
||||
20:03:37Z c60 paperless empty trash 200
|
||||
20:04:40Z c60 paperless DONE ok 53 bad 0
|
||||
20:04:43Z c61 paperless delete 4 200
|
||||
20:04:44Z c61 paperless empty trash 200
|
||||
20:04:45Z c48 immich DONE ok 305 bad 1
|
||||
20:04:48Z c49 immich new user 201
|
||||
20:05:50Z c49 immich DONE ok 305 bad 1
|
||||
20:05:55Z c50 immich new user 201
|
||||
20:06:00Z c61 paperless DONE ok 53 bad 0
|
||||
20:06:05Z c62 paperless delete 4 200
|
||||
20:06:08Z c62 paperless empty trash 200
|
||||
20:07:03Z c50 immich DONE ok 305 bad 1
|
||||
20:07:05Z c51 immich new user 201
|
||||
20:07:17Z c62 paperless DONE ok 53 bad 0
|
||||
20:07:20Z c63 paperless delete 2 200
|
||||
20:07:21Z c63 paperless empty trash 200
|
||||
20:08:27Z c63 paperless DONE ok 53 bad 0
|
||||
20:08:31Z c51 immich DONE ok 305 bad 1
|
||||
20:08:33Z c52 immich new user 201
|
||||
20:08:33Z c64 paperless delete 5 200
|
||||
20:08:35Z c64 paperless empty trash 200
|
||||
20:09:34Z c64 paperless DONE ok 53 bad 0
|
||||
20:09:34Z c52 immich DONE ok 305 bad 1
|
||||
20:09:36Z c53 immich new user 201
|
||||
20:09:47Z c65 paperless delete 4 200
|
||||
20:09:49Z c65 paperless empty trash 200
|
||||
20:10:51Z c53 immich DONE ok 305 bad 1
|
||||
20:10:52Z c65 paperless DONE ok 53 bad 0
|
||||
20:10:54Z c54 immich new user 201
|
||||
20:10:56Z c66 paperless delete 4 200
|
||||
20:10:58Z c66 paperless empty trash 200
|
||||
20:12:04Z c54 immich DONE ok 305 bad 1
|
||||
20:12:06Z c55 immich new user 201
|
||||
20:12:13Z c66 paperless DONE ok 53 bad 0
|
||||
20:12:18Z c67 paperless delete 6 200
|
||||
20:12:20Z c67 paperless empty trash 200
|
||||
20:13:11Z c55 immich DONE ok 305 bad 1
|
||||
20:13:12Z c56 immich new user 201
|
||||
20:13:27Z c67 paperless DONE ok 53 bad 0
|
||||
20:13:30Z c68 paperless delete 5 200
|
||||
20:13:31Z c68 paperless empty trash 200
|
||||
20:14:14Z c56 immich DONE ok 305 bad 1
|
||||
20:14:17Z c57 immich new user 201
|
||||
20:14:33Z c68 paperless DONE ok 53 bad 0
|
||||
20:14:36Z c69 paperless delete 4 200
|
||||
20:14:38Z c69 paperless empty trash 200
|
||||
20:15:43Z c69 paperless DONE ok 53 bad 0
|
||||
20:15:46Z c57 immich DONE ok 305 bad 1
|
||||
20:15:48Z c70 paperless delete 4 200
|
||||
20:15:48Z c58 immich new user 201
|
||||
20:15:50Z c70 paperless empty trash 200
|
||||
20:16:57Z c58 immich DONE ok 305 bad 1
|
||||
20:16:58Z c59 immich new user 201
|
||||
20:17:03Z c70 paperless DONE ok 53 bad 0
|
||||
20:17:08Z c71 paperless delete 4 200
|
||||
20:17:10Z c71 paperless empty trash 200
|
||||
20:18:07Z c59 immich DONE ok 305 bad 1
|
||||
20:18:09Z c60 immich new user 201
|
||||
20:18:18Z c71 paperless DONE ok 53 bad 0
|
||||
20:18:24Z c72 paperless delete 5 200
|
||||
20:18:25Z c72 paperless empty trash 200
|
||||
20:19:17Z c60 immich DONE ok 305 bad 1
|
||||
20:19:18Z c61 immich new user 201
|
||||
20:19:29Z c72 paperless DONE ok 53 bad 0
|
||||
20:19:34Z c73 paperless delete 4 200
|
||||
20:19:35Z c73 paperless empty trash 200
|
||||
20:20:27Z c61 immich DONE ok 305 bad 1
|
||||
20:20:30Z c62 immich new user 201
|
||||
20:20:45Z c73 paperless DONE ok 53 bad 0
|
||||
20:20:51Z c74 paperless delete 4 200
|
||||
20:20:52Z c74 paperless empty trash 200
|
||||
20:21:54Z c62 immich DONE ok 305 bad 1
|
||||
20:21:56Z c63 immich new user 201
|
||||
20:22:11Z c74 paperless DONE ok 53 bad 0
|
||||
20:22:15Z c75 paperless delete 3 200
|
||||
20:22:16Z c75 paperless empty trash 200
|
||||
20:23:15Z c75 paperless DONE ok 53 bad 0
|
||||
20:23:18Z c63 immich DONE ok 305 bad 1
|
||||
20:23:18Z c76 paperless delete 4 200
|
||||
20:23:19Z c64 immich new user 201
|
||||
20:23:20Z c76 paperless empty trash 200
|
||||
20:24:26Z c76 paperless DONE ok 53 bad 0
|
||||
20:24:29Z c64 immich DONE ok 305 bad 1
|
||||
20:24:30Z c77 paperless delete 5 200
|
||||
20:24:31Z c65 immich new user 201
|
||||
20:24:31Z c77 paperless empty trash 200
|
||||
20:25:37Z c65 immich DONE ok 305 bad 1
|
||||
20:25:38Z c66 immich new user 201
|
||||
20:25:45Z c77 paperless DONE ok 53 bad 0
|
||||
20:25:52Z c78 paperless delete 5 200
|
||||
20:25:54Z c78 paperless empty trash 200
|
||||
20:26:52Z c66 immich DONE ok 305 bad 1
|
||||
20:26:54Z c67 immich new user 201
|
||||
20:26:59Z c78 paperless DONE ok 53 bad 0
|
||||
20:27:03Z c79 paperless delete 4 200
|
||||
20:27:05Z c79 paperless empty trash 200
|
||||
20:28:11Z c67 immich DONE ok 305 bad 1
|
||||
20:28:12Z c68 immich new user 201
|
||||
20:28:12Z c79 paperless DONE ok 53 bad 0
|
||||
20:28:18Z c80 paperless delete 5 200
|
||||
20:28:20Z c80 paperless empty trash 200
|
||||
20:29:19Z c68 immich DONE ok 305 bad 1
|
||||
20:29:22Z c69 immich new user 201
|
||||
20:29:28Z c80 paperless DONE ok 53 bad 0
|
||||
20:29:31Z c81 paperless delete 5 200
|
||||
20:29:33Z c81 paperless empty trash 200
|
||||
20:30:33Z c69 immich DONE ok 305 bad 1
|
||||
20:30:35Z c70 immich new user 201
|
||||
20:30:41Z c81 paperless DONE ok 53 bad 0
|
||||
20:30:45Z c82 paperless delete 5 200
|
||||
20:30:46Z c82 paperless empty trash 200
|
||||
20:31:48Z c70 immich DONE ok 305 bad 1
|
||||
20:31:49Z c71 immich new user 201
|
||||
20:31:54Z c82 paperless DONE ok 53 bad 0
|
||||
20:31:58Z c83 paperless delete 4 200
|
||||
20:32:00Z c83 paperless empty trash 200
|
||||
20:32:52Z c71 immich DONE ok 305 bad 1
|
||||
20:32:54Z c72 immich new user 201
|
||||
20:33:02Z c83 paperless DONE ok 53 bad 0
|
||||
20:33:08Z c84 paperless delete 4 200
|
||||
20:33:10Z c84 paperless empty trash 200
|
||||
20:34:03Z c72 immich DONE ok 305 bad 1
|
||||
20:34:05Z c73 immich new user 201
|
||||
20:34:23Z c84 paperless DONE ok 53 bad 0
|
||||
20:34:26Z c85 paperless delete 4 200
|
||||
20:34:27Z c85 paperless empty trash 200
|
||||
20:35:22Z c73 immich DONE ok 305 bad 1
|
||||
20:35:23Z c74 immich new user 201
|
||||
20:35:34Z c85 paperless DONE ok 53 bad 0
|
||||
20:35:39Z c86 paperless delete 4 200
|
||||
20:35:40Z c86 paperless empty trash 200
|
||||
20:36:34Z c74 immich DONE ok 305 bad 1
|
||||
20:36:40Z c75 immich new user 201
|
||||
20:36:56Z c86 paperless DONE ok 53 bad 0
|
||||
20:36:59Z c87 paperless delete 3 200
|
||||
20:37:00Z c87 paperless empty trash 200
|
||||
20:37:35Z c75 immich DONE ok 305 bad 1
|
||||
20:37:37Z c76 immich new user 201
|
||||
20:38:08Z c87 paperless DONE ok 53 bad 0
|
||||
20:38:12Z c88 paperless delete 5 200
|
||||
20:38:14Z c88 paperless empty trash 200
|
||||
20:39:00Z c76 immich DONE ok 305 bad 1
|
||||
20:39:03Z c77 immich new user 201
|
||||
20:39:21Z c88 paperless DONE ok 53 bad 0
|
||||
20:39:25Z c89 paperless delete 4 200
|
||||
20:39:27Z c89 paperless empty trash 200
|
||||
20:40:01Z c77 immich DONE ok 305 bad 1
|
||||
20:40:06Z c78 immich new user 201
|
||||
20:40:28Z c89 paperless DONE ok 53 bad 0
|
||||
20:40:32Z c90 paperless delete 4 200
|
||||
20:40:33Z c90 paperless empty trash 200
|
||||
20:41:26Z c78 immich DONE ok 305 bad 1
|
||||
20:41:28Z c79 immich new user 201
|
||||
20:41:40Z c90 paperless DONE ok 53 bad 0
|
||||
20:41:45Z c91 paperless delete 4 200
|
||||
20:41:47Z c91 paperless empty trash 200
|
||||
20:42:28Z c79 immich DONE ok 305 bad 1
|
||||
20:42:30Z c80 immich new user 201
|
||||
20:42:50Z c91 paperless DONE ok 53 bad 0
|
||||
20:42:54Z c92 paperless delete 4 200
|
||||
20:42:56Z c92 paperless empty trash 200
|
||||
20:43:36Z c80 immich DONE ok 305 bad 1
|
||||
20:43:38Z c81 immich new user 201
|
||||
20:43:59Z c92 paperless DONE ok 53 bad 0
|
||||
20:44:01Z c93 paperless delete 4 200
|
||||
20:44:02Z c93 paperless empty trash 200
|
||||
20:44:44Z c81 immich DONE ok 305 bad 1
|
||||
20:44:48Z c82 immich new user 201
|
||||
20:45:12Z c93 paperless DONE ok 53 bad 0
|
||||
20:45:16Z c94 paperless delete 5 200
|
||||
20:45:19Z c94 paperless empty trash 200
|
||||
20:46:20Z c82 immich DONE ok 305 bad 1
|
||||
20:46:23Z c83 immich new user 201
|
||||
20:46:29Z c94 paperless DONE ok 53 bad 0
|
||||
20:46:33Z c95 paperless delete 3 200
|
||||
20:46:34Z c95 paperless empty trash 200
|
||||
20:47:28Z c83 immich DONE ok 305 bad 1
|
||||
20:47:30Z c84 immich new user 201
|
||||
20:47:35Z c95 paperless DONE ok 53 bad 0
|
||||
20:47:39Z c96 paperless delete 4 200
|
||||
20:47:40Z c96 paperless empty trash 200
|
||||
20:48:40Z c84 immich DONE ok 305 bad 1
|
||||
20:48:42Z c85 immich new user 201
|
||||
20:48:55Z c96 paperless DONE ok 53 bad 0
|
||||
20:48:59Z c97 paperless delete 5 200
|
||||
20:49:00Z c97 paperless empty trash 200
|
||||
20:49:50Z c85 immich DONE ok 305 bad 1
|
||||
20:49:52Z c86 immich new user 201
|
||||
20:50:12Z c97 paperless DONE ok 53 bad 0
|
||||
20:50:15Z c98 paperless delete 5 200
|
||||
20:50:17Z c98 paperless empty trash 200
|
||||
20:51:07Z c86 immich DONE ok 305 bad 1
|
||||
20:51:10Z c87 immich new user 201
|
||||
20:51:37Z c98 paperless DONE ok 53 bad 0
|
||||
20:51:41Z c99 paperless delete 4 200
|
||||
20:51:43Z c99 paperless empty trash 200
|
||||
20:52:21Z c87 immich DONE ok 305 bad 1
|
||||
20:52:21Z immich cycles 87
|
||||
20:52:54Z c99 paperless DONE ok 53 bad 0
|
||||
20:52:54Z paperless cycles 99
|
||||
21:55:58Z c1 nextcloud DONE rc 0 error-lines 0
|
||||
21:55:58Z nextcloud cycles 1
|
||||
21:55:58Z PROOF END
|
||||
@@ -0,0 +1,3 @@
|
||||
18:52:32 SOAK START 120 min
|
||||
18:54:03 first uploads done; cycles for 118 min
|
||||
21:55:58 SOAK END
|
||||
@@ -0,0 +1,92 @@
|
||||
# 2026-10-22T04:41:08Z Immich DB, read-only SELECT (photos per soak user; the file that timed out in cycle 36)
|
||||
# email|photos|IMG_20120021.jpg present
|
||||
tester4+noswap1@felhom.eu|99|1
|
||||
tester4+proof36@felhom.eu|302|0
|
||||
tester4+odon@felhom.eu|303|1
|
||||
tester4+proof10@felhom.eu|303|1
|
||||
tester4+proof11@felhom.eu|303|1
|
||||
tester4+proof12@felhom.eu|303|1
|
||||
tester4+proof13@felhom.eu|303|1
|
||||
tester4+proof14@felhom.eu|303|1
|
||||
tester4+proof15@felhom.eu|303|1
|
||||
tester4+proof16@felhom.eu|303|1
|
||||
tester4+proof17@felhom.eu|303|1
|
||||
tester4+proof18@felhom.eu|303|1
|
||||
tester4+proof19@felhom.eu|303|1
|
||||
tester4+proof1@felhom.eu|303|1
|
||||
tester4+proof20@felhom.eu|303|1
|
||||
tester4+proof21@felhom.eu|303|1
|
||||
tester4+proof22@felhom.eu|303|1
|
||||
tester4+proof23@felhom.eu|303|1
|
||||
tester4+proof24@felhom.eu|303|1
|
||||
tester4+proof25@felhom.eu|303|1
|
||||
tester4+proof26@felhom.eu|303|1
|
||||
tester4+proof27@felhom.eu|303|1
|
||||
tester4+proof28@felhom.eu|303|1
|
||||
tester4+proof29@felhom.eu|303|1
|
||||
tester4+proof2@felhom.eu|303|1
|
||||
tester4+proof30@felhom.eu|303|1
|
||||
tester4+proof31@felhom.eu|303|1
|
||||
tester4+proof32@felhom.eu|303|1
|
||||
tester4+proof33@felhom.eu|303|1
|
||||
tester4+proof34@felhom.eu|303|1
|
||||
tester4+proof35@felhom.eu|303|1
|
||||
tester4+proof37@felhom.eu|303|1
|
||||
tester4+proof38@felhom.eu|303|1
|
||||
tester4+proof39@felhom.eu|303|1
|
||||
tester4+proof3@felhom.eu|303|1
|
||||
tester4+proof40@felhom.eu|303|1
|
||||
tester4+proof41@felhom.eu|303|1
|
||||
tester4+proof42@felhom.eu|303|1
|
||||
tester4+proof43@felhom.eu|303|1
|
||||
tester4+proof44@felhom.eu|303|1
|
||||
tester4+proof45@felhom.eu|303|1
|
||||
tester4+proof46@felhom.eu|303|1
|
||||
tester4+proof47@felhom.eu|303|1
|
||||
tester4+proof48@felhom.eu|303|1
|
||||
tester4+proof49@felhom.eu|303|1
|
||||
tester4+proof4@felhom.eu|303|1
|
||||
tester4+proof50@felhom.eu|303|1
|
||||
tester4+proof51@felhom.eu|303|1
|
||||
tester4+proof52@felhom.eu|303|1
|
||||
tester4+proof53@felhom.eu|303|1
|
||||
tester4+proof54@felhom.eu|303|1
|
||||
tester4+proof55@felhom.eu|303|1
|
||||
tester4+proof56@felhom.eu|303|1
|
||||
tester4+proof57@felhom.eu|303|1
|
||||
tester4+proof58@felhom.eu|303|1
|
||||
tester4+proof59@felhom.eu|303|1
|
||||
tester4+proof5@felhom.eu|303|1
|
||||
tester4+proof60@felhom.eu|303|1
|
||||
tester4+proof61@felhom.eu|303|1
|
||||
tester4+proof62@felhom.eu|303|1
|
||||
tester4+proof63@felhom.eu|303|1
|
||||
tester4+proof64@felhom.eu|303|1
|
||||
tester4+proof65@felhom.eu|303|1
|
||||
tester4+proof66@felhom.eu|303|1
|
||||
tester4+proof67@felhom.eu|303|1
|
||||
tester4+proof68@felhom.eu|303|1
|
||||
tester4+proof69@felhom.eu|303|1
|
||||
tester4+proof6@felhom.eu|303|1
|
||||
tester4+proof70@felhom.eu|303|1
|
||||
tester4+proof71@felhom.eu|303|1
|
||||
tester4+proof72@felhom.eu|303|1
|
||||
tester4+proof73@felhom.eu|303|1
|
||||
tester4+proof74@felhom.eu|303|1
|
||||
tester4+proof75@felhom.eu|303|1
|
||||
tester4+proof76@felhom.eu|303|1
|
||||
tester4+proof77@felhom.eu|303|1
|
||||
tester4+proof78@felhom.eu|303|1
|
||||
tester4+proof79@felhom.eu|303|1
|
||||
tester4+proof7@felhom.eu|303|1
|
||||
tester4+proof80@felhom.eu|303|1
|
||||
tester4+proof81@felhom.eu|303|1
|
||||
tester4+proof82@felhom.eu|303|1
|
||||
tester4+proof83@felhom.eu|303|1
|
||||
tester4+proof84@felhom.eu|303|1
|
||||
tester4+proof85@felhom.eu|303|1
|
||||
tester4+proof86@felhom.eu|303|1
|
||||
tester4+proof87@felhom.eu|303|1
|
||||
tester4+proof8@felhom.eu|303|1
|
||||
tester4+proof9@felhom.eu|303|1
|
||||
# immich-server started: 2026-10-22T02:16:05.052129172Z
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+39
@@ -0,0 +1,39 @@
|
||||
04:06:00 NOSWAP START (30 min)
|
||||
## before
|
||||
NAME TYPE SIZE USED PRIO
|
||||
/dev/dm-0 partition 7.7G 1.9G -1
|
||||
total used free shared buff/cache available
|
||||
Mem: 7939 2859 1830 100 3657 5080
|
||||
Swap: 7935 1935 6000
|
||||
memory: 4096
|
||||
swap: 4096
|
||||
4294967296
|
||||
04:06:01 swapoff -a
|
||||
|
||||
real 0m47.565s
|
||||
user 0m0.001s
|
||||
sys 0m7.070s
|
||||
Swap: 0 0 0
|
||||
04:06:49 restarting immich-machine-learning so its models load again under the burst (as on the first afternoon)
|
||||
immich-machine-learning
|
||||
04:06:50 burst start
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = "UTF-8",
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
|
||||
04:08:21 VOID: stopped at 04:08 UTC. load2.py restarted its cycle names at 1, so Immich got the soak users back (duplicates, ~10 s per 305 photos) and rclone skipped the soak folders. Only Paperless did real work. Rerun with TAG=noswap: 03-noswap-timeline.txt
|
||||
@@ -0,0 +1,73 @@
|
||||
04:08:21 NOSWAP START (30 min)
|
||||
## before
|
||||
total used free shared buff/cache available
|
||||
Mem: 7939 4610 1452 243 2426 3328
|
||||
Swap: 0 0 0
|
||||
memory: 4096
|
||||
swap: 4096
|
||||
4294967296
|
||||
04:08:22 swapoff -a
|
||||
|
||||
real 0m0.002s
|
||||
user 0m0.000s
|
||||
sys 0m0.003s
|
||||
Swap: 0 0 0
|
||||
04:08:23 restarting immich-machine-learning so its models load again under the burst (as on the first afternoon)
|
||||
immich-machine-learning
|
||||
04:08:25 burst start
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = "UTF-8",
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
|
||||
04:09:26 health health=000 15.002352
|
||||
04:10:26 health health=000 15.002167
|
||||
04:11:26 health health=000 15.002790
|
||||
04:12:26 health health=000 15.000452
|
||||
04:13:26 health health=000 15.002936
|
||||
04:14:26 health health=000 15.002567
|
||||
04:15:26 health health=000 15.002091
|
||||
04:16:26 health health=000 15.002821
|
||||
04:17:26 health health=000 15.002518
|
||||
04:18:26 health health=200 13.890303
|
||||
04:19:26 health health=200 6.679951
|
||||
04:20:26 health health=200 12.278351
|
||||
04:21:26 health health=200 6.391933
|
||||
04:22:26 health health=000 15.000930
|
||||
04:23:26 health health=200 2.343069
|
||||
04:24:26 health health=200 0.680254
|
||||
04:25:27 health health=200 0.610104
|
||||
04:26:27 health health=200 0.552664
|
||||
04:27:27 health health=200 0.432607
|
||||
04:28:27 health health=200 0.399314
|
||||
04:29:27 health health=200 0.097570
|
||||
04:30:27 health health=200 0.379978
|
||||
04:31:27 health health=200 0.098200
|
||||
04:32:27 health health=200 0.214732
|
||||
04:33:27 health health=200 0.069976
|
||||
04:34:27 health health=200 0.061167
|
||||
04:35:27 health health=200 0.013009
|
||||
04:36:27 health health=200 0.014631
|
||||
04:37:27 health health=200 0.012391
|
||||
04:38:27 health health=200 0.013004
|
||||
04:38:27 burst window over; swapon -a
|
||||
NAME TYPE SIZE USED PRIO
|
||||
/dev/dm-0 partition 7.7G 0B -1
|
||||
Swap: 7935 0 7935
|
||||
04:43:28 agent log (host-swap):
|
||||
Oct 22 06:06:35 felhom felhom-agent[334289]: time=2026-10-22T06:06:35.844+02:00 level=WARN msg="host-swap: the host's swap is below its guests' swap allowance — a memory burst can freeze the box (R-1192); the hub is told (host_swap_short)" host_swap_mb=938 g
|
||||
04:43:28 NOSWAP END
|
||||
@@ -0,0 +1,106 @@
|
||||
04:08:28Z PROOF START 30 min — each stream repeats on its own until the end
|
||||
04:08:29Z c1 immich new user 201
|
||||
04:08:30Z c1 paperless delete 11 200
|
||||
04:08:31Z c1 paperless empty trash 200
|
||||
04:22:47Z c1 immich DONE ok 99 bad 207
|
||||
Exception in thread Thread-2 (loop):
|
||||
Traceback (most recent call last):
|
||||
File "/usr/lib/python3/dist-packages/requests/models.py", line 974, in json
|
||||
return complexjson.loads(self.text, **kwargs)
|
||||
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.13/json/__init__.py", line 346, in loads
|
||||
return _default_decoder.decode(s)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~^^^
|
||||
File "/usr/lib/python3.13/json/decoder.py", line 348, in decode
|
||||
raise JSONDecodeError("Extra data", s, end)
|
||||
json.decoder.JSONDecodeError: Extra data: line 1 column 5 (char 4)
|
||||
|
||||
During handling of the above exception, another exception occurred:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/usr/lib/python3.13/threading.py", line 1043, in _bootstrap_inner
|
||||
self.run()
|
||||
~~~~~~~~^^
|
||||
File "/usr/lib/python3.13/threading.py", line 994, in run
|
||||
self._target(*self._args, **self._kwargs)
|
||||
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/root/load2.py", line 76, in loop
|
||||
fn(c)
|
||||
~~^^^
|
||||
File "/root/load2.py", line 45, in immich
|
||||
a.headers["Authorization"] = "Bearer " + r.json()["accessToken"]
|
||||
~~~~~~^^
|
||||
File "/usr/lib/python3/dist-packages/requests/models.py", line 978, in json
|
||||
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
|
||||
requests.exceptions.JSONDecodeError: Extra data: line 1 column 5 (char 4)
|
||||
04:23:51Z c1 paperless DONE ok 52 bad 1
|
||||
04:24:01Z c2 paperless delete 5 200
|
||||
04:24:03Z c2 paperless empty trash 200
|
||||
04:25:26Z c2 paperless DONE ok 53 bad 0
|
||||
04:25:29Z c3 paperless delete 6 200
|
||||
04:25:31Z c3 paperless empty trash 200
|
||||
04:26:48Z c3 paperless DONE ok 53 bad 0
|
||||
04:26:54Z c4 paperless delete 5 200
|
||||
04:26:57Z c4 paperless empty trash 200
|
||||
04:28:03Z c4 paperless DONE ok 53 bad 0
|
||||
04:28:06Z c5 paperless delete 4 200
|
||||
04:28:07Z c5 paperless empty trash 200
|
||||
04:29:06Z c5 paperless DONE ok 53 bad 0
|
||||
04:29:09Z c6 paperless delete 4 200
|
||||
04:29:10Z c6 paperless empty trash 200
|
||||
04:29:43Z c6 paperless DONE ok 53 bad 0
|
||||
04:29:44Z c7 paperless delete 4 200
|
||||
04:29:44Z c7 paperless empty trash 200
|
||||
04:30:15Z c7 paperless DONE ok 53 bad 0
|
||||
04:30:16Z c8 paperless delete 3 200
|
||||
04:30:16Z c8 paperless empty trash 200
|
||||
04:30:45Z c8 paperless DONE ok 53 bad 0
|
||||
04:30:46Z c9 paperless delete 2 200
|
||||
04:30:47Z c9 paperless empty trash 200
|
||||
04:31:23Z c9 paperless DONE ok 53 bad 0
|
||||
04:31:27Z c10 paperless delete 3 200
|
||||
04:31:27Z c10 paperless empty trash 200
|
||||
04:31:59Z c10 paperless DONE ok 53 bad 0
|
||||
04:32:01Z c11 paperless delete 4 200
|
||||
04:32:01Z c11 paperless empty trash 200
|
||||
04:32:34Z c11 paperless DONE ok 53 bad 0
|
||||
04:32:35Z c12 paperless delete 4 200
|
||||
04:32:36Z c12 paperless empty trash 200
|
||||
04:33:09Z c12 paperless DONE ok 53 bad 0
|
||||
04:33:10Z c13 paperless delete 4 200
|
||||
04:33:11Z c13 paperless empty trash 200
|
||||
04:33:41Z c13 paperless DONE ok 53 bad 0
|
||||
04:33:44Z c14 paperless delete 3 200
|
||||
04:33:44Z c14 paperless empty trash 200
|
||||
04:34:17Z c14 paperless DONE ok 53 bad 0
|
||||
04:34:18Z c15 paperless delete 4 200
|
||||
04:34:19Z c15 paperless empty trash 200
|
||||
04:34:46Z c15 paperless DONE ok 53 bad 0
|
||||
04:34:48Z c16 paperless delete 4 200
|
||||
04:34:49Z c16 paperless empty trash 200
|
||||
04:35:19Z c16 paperless DONE ok 53 bad 0
|
||||
04:35:20Z c17 paperless delete 7 200
|
||||
04:35:20Z c17 paperless empty trash 200
|
||||
04:35:52Z c17 paperless DONE ok 53 bad 0
|
||||
04:35:54Z c18 paperless delete 6 200
|
||||
04:35:55Z c18 paperless empty trash 200
|
||||
04:36:27Z c18 paperless DONE ok 53 bad 0
|
||||
04:36:28Z c19 paperless delete 6 200
|
||||
04:36:28Z c19 paperless empty trash 200
|
||||
04:36:56Z c19 paperless DONE ok 53 bad 0
|
||||
04:36:57Z c20 paperless delete 4 200
|
||||
04:36:57Z c20 paperless empty trash 200
|
||||
04:37:26Z c20 paperless DONE ok 53 bad 0
|
||||
04:37:27Z c21 paperless delete 4 200
|
||||
04:37:28Z c21 paperless empty trash 200
|
||||
04:37:57Z c21 paperless DONE ok 53 bad 0
|
||||
04:37:58Z c22 paperless delete 6 200
|
||||
04:37:59Z c22 paperless empty trash 200
|
||||
04:38:28Z c22 paperless DONE ok 53 bad 0
|
||||
04:38:29Z c23 paperless delete 6 200
|
||||
04:38:29Z c23 paperless empty trash 200
|
||||
04:38:59Z c23 paperless DONE ok 53 bad 0
|
||||
04:38:59Z paperless cycles 23
|
||||
04:06:53 BURST START 30 min
|
||||
04:08:28 BURST START 30 min
|
||||
04:43:28 BURST END rc=0
|
||||
@@ -0,0 +1,8 @@
|
||||
Oct 22 06:38:27 felhom kernel: Adding 8126460k swap on /dev/mapper/pve-swap. Priority:-1 extents:1 across:8126460k
|
||||
|
||||
06:43:55 up 10:16, 2 users, load average: 4.87, 6.76, 10.03
|
||||
total used free shared buff/cache available
|
||||
Mem: 7939 4129 1638 238 2717 3810
|
||||
Swap: 7935 551 7384
|
||||
NAME TYPE SIZE USED PRIO
|
||||
/dev/dm-0 partition 7.7G 551.9M -1
|
||||
@@ -0,0 +1,4 @@
|
||||
samples 180; max {'host_full': (36.1, '2026-10-22T04:09:52Z'), 'guest_full': (46.88, '2026-10-22T04:09:52Z'), 'load': (31.3, '2026-10-22T04:11:12Z'), 'swapMB': (0, ''), 'io_full': (26.91, '2026-10-22T04:16:22Z')}; samples over 20%: 94 [('2026-10-22T04:08:52Z', 24.44, 30.8), ('2026-10-22T04:09:02Z', 28.67, 36.95), ('2026-10-22T04:09:12Z', 32.85, 42.59), ('2026-10-22T04:09:22Z', 34.79, 45.09), ('2026-10-22T04:09:32Z', 34.64, 45.44)]; oom_kill +0; limit-hit events +0
|
||||
probes 125; non-200 24; slowest 200 15.00s; gaps>60s between 200s: [('2026-10-22T04:08:35+00:00', '2026-10-22T04:13:24Z', 289.0), ('2026-10-22T04:13:39+00:00', '2026-10-22T04:16:31Z', 172.0), ('2026-10-22T04:18:59+00:00', '2026-10-22T04:20:07Z', 68.0)]
|
||||
samples 35; max {'host_full': (1.16, '2026-10-22T04:38:52Z'), 'guest_full': (1.31, '2026-10-22T04:38:52Z'), 'load': (7.33, '2026-10-22T04:41:12Z'), 'swapMB': (553, '2026-10-22T04:42:32Z'), 'io_full': (0.29, '2026-10-22T04:38:52Z')}; samples over 20%: 0 []; oom_kill +0; limit-hit events +0
|
||||
probes 34; non-200 0; slowest 200 0.04s; gaps>60s between 200s: []
|
||||
@@ -10,6 +10,9 @@ urllib3.disable_warnings()
|
||||
IP = "192.168.0.125"; D = "tester4.test"
|
||||
env = dict(l.split("=", 1) for l in sys.stdin.read().splitlines() if "=" in l)
|
||||
end = time.time() + 60 * int(sys.argv[1])
|
||||
# TAG (argv 2, default "proof"): the name stem of each cycle's Immich user and Nextcloud folder. A second run needs
|
||||
# its own tag — with the old one, cycle 1.. reuse the soak's users and folders (Immich: duplicates; rclone: skips).
|
||||
TAG = sys.argv[2] if len(sys.argv) > 2 else "proof"
|
||||
lk = threading.Lock()
|
||||
def log(*a):
|
||||
with lk:
|
||||
@@ -40,8 +43,8 @@ def immich(c):
|
||||
a = s("photos")
|
||||
r = a.post(u("/api/auth/login"), json={"email": "tester4+odon@felhom.eu", "password": env["ODON"]}, timeout=60)
|
||||
a.headers["Authorization"] = "Bearer " + r.json()["accessToken"]
|
||||
em = f"tester4+proof{c}@felhom.eu"
|
||||
r = a.post(u("/api/admin/users"), json={"email": em, "password": env["ODON"], "name": f"proof{c}"}, timeout=60)
|
||||
em = f"tester4+{TAG}{c}@felhom.eu"
|
||||
r = a.post(u("/api/admin/users"), json={"email": em, "password": env["ODON"], "name": f"{TAG}{c}"}, timeout=60)
|
||||
log(f"c{c} immich new user", r.status_code)
|
||||
x = s("photos")
|
||||
r = x.post(u("/api/auth/login"), json={"email": em, "password": env["ODON"]}, timeout=60)
|
||||
@@ -53,7 +56,7 @@ def immich(c):
|
||||
continue
|
||||
ts = dt.datetime.fromtimestamp(os.stat(f).st_mtime, dt.timezone.utc).isoformat()
|
||||
try:
|
||||
r = x.post(u("/api/assets"), files={"assetData": open(f, "rb")}, data={"deviceAssetId": f, "deviceId": f"proof{c}", "fileCreatedAt": ts, "fileModifiedAt": ts}, timeout=600)
|
||||
r = x.post(u("/api/assets"), files={"assetData": open(f, "rb")}, data={"deviceAssetId": f, "deviceId": f"{TAG}{c}", "fileCreatedAt": ts, "fileModifiedAt": ts}, timeout=600)
|
||||
ok += r.status_code in (200, 201); bad += r.status_code not in (200, 201)
|
||||
except Exception as e:
|
||||
bad += 1; log(f"c{c} immich", os.path.basename(f), "ERR", str(e)[:100])
|
||||
@@ -63,7 +66,7 @@ def nextcloud(c):
|
||||
e = dict(os.environ, RCLONE_CONFIG_NC_TYPE="webdav", RCLONE_CONFIG_NC_URL=f"https://cloud.{D}/remote.php/dav/files/odon",
|
||||
RCLONE_CONFIG_NC_VENDOR="nextcloud", RCLONE_CONFIG_NC_USER="odon", RCLONE_CONFIG_NC_PASS=obs)
|
||||
p = subprocess.run(["rclone", "copy", "--no-check-certificate", "--transfers", "4", "--max-size", "1G", "--stats", "0", "--stats-log-level", "NOTICE",
|
||||
"/root/data/nextcloud/Csalad", f"nc:/Csalad_proof{c}"], env=e, capture_output=True, text=True)
|
||||
"/root/data/nextcloud/Csalad", f"nc:/Csalad_{TAG}{c}"], env=e, capture_output=True, text=True)
|
||||
errs = sum(1 for l in (p.stderr or "").splitlines() if "ERROR" in l)
|
||||
log(f"c{c} nextcloud DONE rc {p.returncode} error-lines {errs}")
|
||||
def loop(fn):
|
||||
|
||||
@@ -4,5 +4,5 @@
|
||||
cd /root
|
||||
ENVD=$(cat /root/.burstenv); shred -u /root/.burstenv
|
||||
echo "$(date -u +%T) BURST START $1 min" >> /root/noswap.log
|
||||
timeout $(( $1*60 + 300 )) python3 /root/load2.py $1 <<< "$ENVD" > /root/noswap-burst.log 2>&1
|
||||
timeout $(( $1*60 + 300 )) python3 /root/load2.py $1 noswap <<< "$ENVD" > /root/noswap-burst.log 2>&1
|
||||
echo "$(date -u +%T) BURST END rc=$?" >> /root/noswap.log
|
||||
|
||||
@@ -124,7 +124,7 @@ stopping line that lies.
|
||||
| **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC |
|
||||
| **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
|
||||
|
||||
## Apps & catalog — 14 rows (P3 6, P4 8)
|
||||
## Apps & catalog — 15 rows (P3 6, P4 9)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -142,6 +142,7 @@ stopping line that lies.
|
||||
| **R-1067** | Apps & catalog | P4 | **The catalog still carries `pi_compatible:` on every template, and nothing reads it any more.** controller v0.315.0 removed the „Pi kompatibilis" badge and the „csak x86" tag from every page (Raspberry Pi is no longer supported, so one was noise and the other was a warning on almost every app). The `ResourceHints.PiCompatible` struct field and the catalog key were deliberately LEFT: a box still on 0.314.0 reads the key, and removing it would make that box show „csak x86" on every app. **Gated on the floor:** once the fleet floor is at or above 0.315.0, drop `pi_compatible:` from the templates and the field from `stacks.ResourceHints` in the same pair of commits. Harmless until then — an unread key costs nothing. | **BLOCKED** — filed 2026-10-15. Checked 2026-10-16 (apps burndown): the floor is 0.321.0 and demo-hp, Tester 1 and the N100 run 0.321.0, but **Tester-2 still reports 0.292.0** (its newest report 2026-10-04; hub DB with WAL) — so nothing removed. | every box on ≥ 0.315.0 — today Tester-2 | Drop the catalog key and the struct field; grep the hub and the website for Pi wording at the same time (none found 2026-10-15) | CC |
|
||||
| **R-1043** | Apps & catalog | P4 | **Sonarr, Radarr and Prowlarr build their login redirect as an absolute `http://` URL.** They trust forwarded headers only from their „Trusted networks” (the household set `192.168.0.0/24`; traefik is `172.18.x`), so they ignore `X-Forwarded-Proto: https` and answer `Location: http://<name>/login` to an https visitor. Since controller v0.312.0 that hop is sent back to https by traefik (a GET, no password), so it is cosmetic; a template could seed traefik's network as trusted. (`audits/https-only-2026-10-14/``1A-3-4.txt`) | **OPEN** — filed 2026-10-14 | — | Seed the docker network as a trusted proxy in the *arr templates (with the R-1030 seed), measure the redirect | CC |
|
||||
| **R-1099** | Apps & catalog | P4 | **The N100 may hold RomM games in the old upload place, now shown under a fake „roms” platform.** Until 2026-10-16 RomM's own upload wrote to the drive's `roms/roms/<platform>/`; after the mapping fix (catalog `fb18471`, R-1089) those files are at RomM's `library/roms/roms/…`. Measured on 9201: such a game showed as missing in its platform and again under a platform „roms” (`audits/romm-ghost-2026-10-16/B/b-after.txt`). Files put in with the file manager are not affected — they are now found. The N100 runs RomM (hub `/apps/romm`); CC does not log in to the lived-in box. | OPEN — filed 2026-10-16 | Operator: look at `roms/roms/` on the N100's drive in the file manager | If it holds platform folders: move them up one level into `roms/`, then run „Scan” in RomM; if it is empty, close | Operator |
|
||||
| **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | OPEN | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC |
|
||||
|
||||
|
||||
## App updates — 4 rows (P3 4)
|
||||
@@ -240,17 +241,18 @@ stopping line that lies.
|
||||
| **R-1054** | Security & access | P3 | **The hub's operator/global API key is still one bearer that can do everything on `/api/v1`, including reading every box's break-glass root password (`GET /admin/hosts/{id}/recovery-credential`).** Hub v0.152.0 gave the web surface scoped tokens (`05` §17) but tokens are refused on `/api/v1` by design. Who uses the global key today is not listed in one place. Fix: list its consumers; move each operator use to a scoped token on the web side (or a token class on the API), and keep the global key only where a box path needs it — or retire it. | OPEN — filed 2026-10-15 by the token build | — | List the global key's consumers (pod env, scripts, runbooks) | CC |
|
||||
| **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f <conf>` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WAITING-ON-OPERATOR — fixed on main, unreleased; ships with the agent release (D26) | the agent release (morning list 2) | Release; the next fresh install must raise no oob_degraded; then close | operator |
|
||||
|
||||
## Box system & updates — 13 rows (P1 1, P2 2, P3 6, P4 4)
|
||||
## Box system & updates — 14 rows (P1 1, P2 2, P3 7, P4 4)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1192** | Box system & updates | P1 | **On the minimum machine (8 GB), the household's first afternoon froze the box for about an hour: the dashboard and every app stopped answering, and only pulling the plug brought it back.** Rehearsal 2026-10-20 on a fresh box (controller 0.336.0, agent 0.161.0): the install page let the household book 8 apps to 3638 of 3712 MB (98 %) — the guest has 4096 MB in all — and told it on every page „Az alkalmazások csúcsterhelése meghaladhatja a rendelkezésre álló memóriát. **Normál használat mellett ez nem okoz problémát.**" Then the first uploads (53 PDFs to Paperless, photos to Immich, a file tree to Nextcloud — sequential per app) → from ~13:05 UTC `/api/health` gave no answer in 15 s; the VM read ~4 GB/min from its disk (158 → 476 GB) with the network still; the box's own history shows load 139–152 and no samples for 36 min; it came back at 14:00 for minutes and froze again; the plug pulled at 14:11 → dashboard in 2 min 22 s, stable after. Lost on the way: 8 Paperless documents failed (5 ordinary invoices among them), the Immich upload cut at photo 150, Nextcloud at 251 of 3 393 files. Cause NOT diagnosed (no shell in the box; memory read ~2.3 GB while load was 150 — not plain „memory full"); the one hint: the hub's `app_oom` at 13:34 — immich-machine-learning killed by its memory limit. Reproduced a SECOND time at 15:27:55 with only one Immich upload + one chunked Nextcloud upload running (~35 min, then froze again; plug pulled 16:13); per-app memory now logged every 30 s (`B4-use/memwatch.txt`). Evidence `audits/rehearsal-2026-10-20/journal.md` (12:56–14:23), `B4-use/box-metrics-after-plug.txt`, `B4-use/vm-counters-before-plug.txt`, `B4-use/after-plug-watch.txt`. | WAITING-ON-OPERATOR **DIAGNOSED 2026-10-21 with numbers, fix committed (agent `286a7e3`), proven by applying the same setting by hand** (`audits/fixes-2026-10-21/P4-freeze/`). Reproduced on demo-hp: a nested 8 GB VM, Tester4 from the public ISO, the same 8 apps, the same uploads (generator seed 20261020) → the dashboard dead 13:28:18–13:34:48 UTC. **Mechanism:** the guest's swap is capped at Proxmox's LXC default of 512 MB (the agent never set `swap`); Immich's analysis loaded its models (+1.15 GB, `immich-machine-learning` 70 → 1 224 MB) and the server grew (+0.4 GB) on top of 8 apps' 1.75 GB, so the 4096 MB guest filled (anon 3 585 MB, swap 511/512 MB) and could evict only file pages: ~360 MB left for ~25 containers' code → 150 000–2 700 000 refaults and up to 12 GB of reads per 10 s, memory „full" pressure 96 %, IO 97 %, load 162, 0 OOM kills; the host had 2.2 GB RAM and 7.4 GB swap free. **Experiment:** `pct set 9201 --swap 4096` live at 13:34:06 → „full" pressure 96 % → 1.7 % in 2 min, dashboard back 42 s later, uploads running on. **Proof (2 h, 13:43–15:43, swap = RAM by hand):** the same uploads repeated as fresh work (Paperless 110 cycles × 53 PDFs, Immich 111 new users × 306 photos, two Nextcloud trees): 720 samples, max „full" pressure 12 %, load 33, 0 limit hits, every dashboard probe answered (slowest 1.26 s). Controller 0.337.0: the false „Normál használat mellett ez nem okoz problémát." replaced. Nextcloud's small-file WebDAV upload ran ~17 files/min under this load — slow, not lost. | the operator's agent release + vouch | Release agent with the swap fix (swap = RAM at bring-up, at RAM resize, once at start); after the update read one box's `pct config` (swap = memory) and its `guest-swap: raised` log line; then close | operator |
|
||||
| **R-1192** | Box system & updates | P1 | **On the minimum machine (8 GB), the household's first afternoon froze the box for about an hour: the dashboard and every app stopped answering, and only pulling the plug brought it back.** Rehearsal 2026-10-20 on a fresh box (controller 0.336.0, agent 0.161.0): the install page let the household book 8 apps to 3638 of 3712 MB (98 %) — the guest has 4096 MB in all — and told it on every page „Az alkalmazások csúcsterhelése meghaladhatja a rendelkezésre álló memóriát. **Normál használat mellett ez nem okoz problémát.**" Then the first uploads (53 PDFs to Paperless, photos to Immich, a file tree to Nextcloud — sequential per app) → from ~13:05 UTC `/api/health` gave no answer in 15 s; the VM read ~4 GB/min from its disk (158 → 476 GB) with the network still; the box's own history shows load 139–152 and no samples for 36 min; it came back at 14:00 for minutes and froze again; the plug pulled at 14:11 → dashboard in 2 min 22 s, stable after. Lost on the way: 8 Paperless documents failed (5 ordinary invoices among them), the Immich upload cut at photo 150, Nextcloud at 251 of 3 393 files. Cause NOT diagnosed (no shell in the box; memory read ~2.3 GB while load was 150 — not plain „memory full"); the one hint: the hub's `app_oom` at 13:34 — immich-machine-learning killed by its memory limit. Reproduced a SECOND time at 15:27:55 with only one Immich upload + one chunked Nextcloud upload running (~35 min, then froze again; plug pulled 16:13); per-app memory now logged every 30 s (`B4-use/memwatch.txt`). Evidence `audits/rehearsal-2026-10-20/journal.md` (12:56–14:23), `B4-use/box-metrics-after-plug.txt`, `B4-use/vm-counters-before-plug.txt`, `B4-use/after-plug-watch.txt`. | WAITING-ON-OPERATOR **DIAGNOSED 2026-10-21 with numbers, fix committed (agent `286a7e3`), proven by applying the same setting by hand** (`audits/fixes-2026-10-21/P4-freeze/`). Reproduced on demo-hp: a nested 8 GB VM, Tester4 from the public ISO, the same 8 apps, the same uploads (generator seed 20261020) → the dashboard dead 13:28:18–13:34:48 UTC. **Mechanism:** the guest's swap is capped at Proxmox's LXC default of 512 MB (the agent never set `swap`); Immich's analysis loaded its models (+1.15 GB, `immich-machine-learning` 70 → 1 224 MB) and the server grew (+0.4 GB) on top of 8 apps' 1.75 GB, so the 4096 MB guest filled (anon 3 585 MB, swap 511/512 MB) and could evict only file pages: ~360 MB left for ~25 containers' code → 150 000–2 700 000 refaults and up to 12 GB of reads per 10 s, memory „full" pressure 96 %, IO 97 %, load 162, 0 OOM kills; the host had 2.2 GB RAM and 7.4 GB swap free. **Experiment:** `pct set 9201 --swap 4096` live at 13:34:06 → „full" pressure 96 % → 1.7 % in 2 min, dashboard back 42 s later, uploads running on. **Proof (2 h, 13:43–15:43, swap = RAM by hand):** the same uploads repeated as fresh work (Paperless 110 cycles × 53 PDFs, Immich 111 new users × 306 photos, two Nextcloud trees): 720 samples, max „full" pressure 12 %, load 33, 0 limit hits, every dashboard probe answered (slowest 1.26 s). Controller 0.337.0: the false „Normál használat mellett ez nem okoz problémát." replaced. Nextcloud's small-file WebDAV upload ran ~17 files/min under this load — slow, not lost. **NIGHT SOAK 2026-10-21→22 (fresh box from the public ISO, the agent fix as a by-hand fixture, `audits/night-2026-10-21/P1-soak/`):** 2 h of the rehearsal uploads, then the night chain untouched — 3 389 probes, none failed, largest interval between answers 16 s, slowest 4.1 s; 0 OOM kills; every night job ran (dump, second drive, off-site 8 apps, sweep, proof, integrity). Memory „full" pressure went above 20 % twice: one 10-s sample at the Immich model load (31/35 %) and ~3 min in the 03:30 second-drive job (39/45 %) — R-1214. 1 photo of ~26 000 not stored (a 10-min hang at low pressure) — R-1215. The freeze did not come back. | the operator's agent release + vouch | Release agent with the swap fix (swap = RAM at bring-up, at RAM resize, once at start); after the update read one box's `pct config` (swap = memory) and its `guest-swap: raised` log line; then close | operator |
|
||||
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
|
||||
| **R-982** | Box system & updates | P3 | **The Proxmox host accepts root password login over SSH from the LAN (measured on demo-hp), so anyone on the household network can try the root password.** Design-shaped: the break-glass password exists for the console. Fix: key-only SSH for root; the password stays for the console. Evidence: `audits/security-review-2026-10-12/evidence/demo-hp-host-reads.txt`. | OPEN — design-proposal | — | Decide; then set it in the installer and the config bundle | CC |
|
||||
| **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC |
|
||||
| **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC |
|
||||
| **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC |
|
||||
| **R-1207** | Box system & updates | P2 | **A host with no (or too little) swap gets nothing from the guest swap fix (R-1192), and the agent never adds host swap.** The fix lets a guest swap as much as its RAM; that swap lives on the HOST. Measured 2026-10-21 night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap (LVM `pve/swap`), so the installer's boxes are covered; a BYO host (or one with swap turned off) may have none. Now reported and said (agent `0da792e`: `host_swap_mb` / `guest_swap_mb`, one WARN; hub 0.164.0 `host_swap_short` to the operator; the install page sentence) — but nothing adds swap. The no-swap test on the soak VM (`audits/night-2026-10-21/P2-noswap/`) says what happens without it. | WAITING-ON-OPERATOR | decision: should the agent (or the installer for BYO) create host swap when it is short, or should an 8 GB BYO host without swap be refused at install? | Operator decides; CC builds the pick | operator |
|
||||
| **R-1207** | Box system & updates | P2 | **A host with no (or too little) swap gets nothing from the guest swap fix (R-1192), and the agent never adds host swap.** The fix lets a guest swap as much as its RAM; that swap lives on the HOST. Measured 2026-10-21 night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap (LVM `pve/swap`), so the installer's boxes are covered; a BYO host (or one with swap turned off) may have none. Now reported and said (agent `0da792e`: `host_swap_mb` / `guest_swap_mb`, one WARN; hub 0.164.0 `host_swap_short` to the operator; the install page sentence) — but nothing adds swap. The no-swap test on the soak VM (`audits/night-2026-10-21/P2-noswap/`) says what happens without it. | WAITING-ON-OPERATOR **NO-SWAP TEST 2026-10-22 04:08–04:38 UTC (`audits/night-2026-10-21/P2-noswap/`):** the soak box, `swapoff -a` on the host, the same burst → the freeze CAME BACK within 20 s: `/api/health` dead for 289 s, then 172 s and 68 s (24 of 125 probes failed), memory „full" 36 % host / 47 % guest, the VM reading ~40 GB per 25 s from disk; Immich lost 207 of 306 photos and its stream broke; the box recovered only when the Immich stream stopped (04:23). After `swapon -a`: 0 failed probes, „full" ≤ 1.3 %. The agent logged its one WARN at the swapoff (host 938 MB < guests 4096 MB). | decision: should the agent (or the installer for BYO) create host swap when it is short, or should an 8 GB BYO host without swap be refused at install? | Operator decides; CC builds the pick | operator |
|
||||
| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | OPEN | — | Measure the tier2 job alone on the 8 GB box (per-step pressure); decide whether 20 % over 10 s is the right line, or limit the copy (ionice / cgroup IO weight) | CC |
|
||||
| **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
|
||||
| **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
|
||||
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement | — | — | CC |
|
||||
|
||||
Reference in New Issue
Block a user